Evaluation design decides the winner: horizon, leakage and metric effects in intermittent demand forecasting

Comparisons between machine-learning and classical demand forecasters are read as statements about methods. They are also statements about the evaluation. We show that three individually defensible design choices each reverse the ranking on the same data, using two public retail datasets chosen to bracket the intermittency spectrum: the M5 competition data (30,490 store-SKU series) and UCI Online Retail II (4,675 e-commerce series). Eight classical forecasters, a global gradient-boosted model, two deep global models and two pretrained zero-shot models are evaluated under one rolling-origin design. Lengthening the evaluation horizon from one week to a twenty-six-week lead time moves Croston's method from last of eight classical methods to first on the denser panel, its error against a naive benchmark falling from 1.005 to 0.568, while the sparse panel leaves Croston last at every lead time we test. Two things are therefore confounded in the received verdict on the Croston family: the one-step scoring that Teunter and Duncan (2009) warned about, and the family itself, since its third member TSB finishes third of the eight classical methods on the dense panel and fourth on the sparse one at one step with no change of horizon, while Croston and SBA finish in the bottom three of both. Restricting one price feature to strictly past values removes a silent target leak. A same-week transacted price identifies sale weeks perfectly and carries 42 percent of the model's gain; without it the gradient-boosted model falls from ahead of every classical method to behind tuned exponential smoothing. Replacing MASE with RMSSE turns the naive benchmark from a method that wins more than a fifth of all series on the sparse panel, more than every forecaster there except a short moving average, into the method with the worst mean RMSSE on both. Method selection is subject to the same instability: on Online Retail II the Syntetos-Boylan Approximation wins the most series in-sample yet loses to a naive random walk out-of-sample. A test using two pretrained models that need no per-series history finds that neither beats the best trained model in any sparse demand class: the gap favours the trained model in all four, and a paired bootstrap clears zero in all four. What bounds accuracy on the sparse tail is therefore data sufficiency and not model class. Scoring predictive quantiles rather than point forecasts, the two Chronos-Bolt checkpoints also beat an empirical training-quantile benchmark by about 20% in scaled pinball loss, and doing so exposes a second limitation: the two Chronos-Bolt checkpoints we score are trained on deciles, so a request for the 99th percentile returns the 90th, clamped. The ceiling is documented and the library logs a warning, but the call still returns a number rather than raising, so a pipeline that does not read its logs silently reads a relabelled 90th percentile. For practice, the error metric has to match the loss the decision carries and the evaluation horizon has to match the replenishment cycle. Feature timing and the quantiles a model can express both need auditing before a machine-learning gain is believed. All data and code are public.