Observation Delay and Training Eligibility in Forecasting: A Matched Comparison of Chronological and Random Model Selection

Forecasting evaluations should consider both observation availability and training-label eligibility. We compare physical observation delay against an equally long additional fitting-use restriction in four public series: bicycle rentals, Beijing PM2.5, traffic volume and household power consumption. The six-hour-ahead forecasts used a fixed candidate set consisting of persistence, Ridge regression, Random Forest and histogram gradient boosting. The main comparison used conditions with identical training-label membership and evaluation origins, but different prediction information. Candidates were selected through historical validation, and the reserved final evaluation was performed only once under frozen rules. Chronological and random selectors picked Random Forest for three sources, producing zero primary selection contrasts. In Beijing, chronological selection picked histogram gradient boosting and had lower mean absolute error in both matched conditions, with a slightly larger advantage under physical delay. Physical delay was associated with higher selected-model errors for three sources, while household errors were nearly unchanged. These results suggest that equal training-label eligibility does not imply equal forecasting accuracy, and that zero selection contrasts can coexist with changes in prediction error. Evaluation protocols should distinguish observation availability from training eligibility and report absolute errors alongside selection regret.