CANVAS: Competency-Aware Navigation Assessment Suite for Pedestrian Trajectory Forecasting

We introduce CANVAS, a unified, Python-native framework for control-aware evaluation of pedestrian trajectory forecasting models in dataset-driven, open-loop simulations. Assessing such forecasting models requires metrics that not only capture the downstream effect of forecasting errors on control performance, but also systematically incorporate online feedback, since the models may encounter unfamiliar pedestrian behaviors. To address this need, we devise competency indices, calibrated measures that quantify how forecasting errors propagate to control, and respond to real-time distributional drift via adaptive conformal prediction (ACP). This integration combines control-aware competency evaluation with ACP, offering a principled and statistically guaranteed mechanism for estimating a model’s online competency. Owing to our dedicated out-of-distribution (OOD) dataset, CANVAS enables rigorous testing of whether forecasting models remain informative even under OOD regimes. As CANVAS standardizes the composition of datasets, forecasting models, and controllers, users can flexibly evaluate their custom prediction-control pipelines. Accordingly, CANVAS supports a reliable and reproducible online assessment of trajectory predictors, unlike existing toolkits that are restricted to offline predictive metrics, overlook control consequences, or disregard OOD conditions. By running evaluations across diverse models and controllers using CANVAS, we gain the key insight that low forecasting error does not necessarily translate into safer or more efficient control.