Evaluation pitfalls in intermittent-demand forecasting and inventory policy: a fleet-scale study of how far accuracy, calibration, and cost disagree

Choosing a forecaster or an inventory policy for intermittent spare parts is usually settled on point accuracy, yet what an operation pays is holding cost plus the cost of stocking out. We benchmark eleven forecasters, from the operator's own deployed methods through gradient boosting and a zero-shot foundation model, on the full spare-parts catalogue of a large multi-brand maritime operator: 191,613 estimable stock items, evaluated one step ahead under an (s, S) cost simulator. The study is organised around four evaluation traps, because in each case an intuitive conclusion changes once the trap is removed. First, accuracy is a weak proxy for cost: the per-item rank correlation between point error and simulated cost is τb = 0.35 on demand-active items, and lower still, around 0.18 to 0.25, on the items that carry real signal, where an upward bias from scoring near-zero series and a downward bias from short-horizon noise both operate. Second, sizing safety stock from a model's own predictive quantiles does not beat a plain normal-approximation formula at matched service for any of the three models tested, and calibration does not predict the outcome: the best-calibrated model loses the most. Third, a deployed policy can appear dominated by a model-driven policy purely through an asymmetric simulator start, and under a like-for-like initialisation the operator's own policy is not dominated. Fourth, cost-aware evaluation on the most recent operational quarters collapses costs by about half and reorders methods, whether the recent data are censored or demand has genuinely declined, so the trap bites either way. The data are anonymised and available on request, and the full pipeline will be released as code on journal publication.