The Validation Crisis in AI Forecasting

Forecasts of when artificial general intelligence will arrive increasingly shape capital allocation, regulation, and where a generation of talent is placed — yet the confidence attached to them exceeds what the methods producing them can support. We argue this gap is structural, not careless: it is the predictable consequence of fitting a model to a measured window and projecting it forward without the validation discipline other quantitative fields learned to require. We import that discipline from quantitative finance, which developed it between 2014 and 2018 because capital was being lost to backtests that looked excellent and meant nothing. Applying three such tools — the deflated Sharpe ratio, the probability of backtest overfitting, and a walk-forward retrodiction protocol — we evaluate the major capability forecasts and introduce the Deflated Capability Forecast (DCF): a method that widens a forecast's stated interval by the amount its underlying methodology warrants, returning a distribution with explicit treatment of the tails in place of a point estimate carrying unearned precision. Across the forecasts where the method could be fully computed, deflation factors cluster between 1.3× and 2.0× — the stated intervals are systematically too narrow. We then turn the method on this work: before computing anything, we preregistered that the deflated Sharpe ratio applied to one landmark forecast would widen its interval by at least 2.3×. It produced 1.285×. We report the failure rather than revise the threshold — the failure lies not in the framework, which computed correctly, but in a prior set by intuition before the deflation it should have been derived from, which is the very error this work exists to identify. A discipline of honest validation is supposed to surface exactly this.