Survivor Bias in Learning-Curve Surrogates: Successive Halving Selects on Noise

Learning-curve surrogates for hyperparameter optimization — freeze-thaw GPs, power-law extrapolation, in-context learning-curve networks — are fit and deployed on curves that survived earlier rungsof successive halving (SH), Hyperband, or ASHA, yet no method in this line models the censoringmechanism. We state the honest crux up front: selection acts on observed prefixes, so if prefixes werenoiseless the censoring would be ignorable (missing at random). The bias enters because prefixes arenoisy observations of latent curve parameters: rung survival selects on noise, so survivors’ continuationssystematically disappoint a naive per-curve extrapolation — regression-to-the-mean survivor bias witha Tobit/Heckman structure on the latent parameters. In a controlled power-law family the effect isprecisely dose-responsive: survivor selection bias at the first rung grows from 0.0012 (accuracy units)at noise σ = 0.005 to 0.0539 at σ = 0.04 and grows with selection pressure η; at the median noisewe calibrate from real LCBench curves (σ ≈ 0.0004) it is indistinguishable from zero — an informativenull about SH robustness that we report rather than bury, with the caveat that noisier regimes (smallvalidation sets; PD1’s ImageNet task at σ ≈ 0.02) sit well inside the biased zone. A Heckman-type jointselection+outcome model — a hierarchical prior over latent curve parameters with rung-survival probitlikelihoods — attacks the survivor bias but exposes the same identification fragility econometrics warns of:with a single SH bracket it is weakly identified (it prefers the biased solution at realistic survivor counts),and pooling three Hyperband-style brackets — rung-assignment randomness as an exclusion restriction— cuts the survivor-only asymptote bias from +0.084 to −0.036 accuracy units (naive 0.864, corrected0.744, truth 0.780) but now overshoots, more so at higher noise. The robust practical estimator is theboring one: refit the prior on all observed prefixes, including those of killed configs (0.784, near-unbiased)— exactly the data a replaying tool already holds, reinforcing the crux that the fix is to use the prefixesone has, not to model survivor selection. The same posterior machinery cuts survivor-extrapolation biasat the first rung from 0.0192 (naive least squares) to 0.0057; a per-curve Tobit correction without thepopulation prior manages only 0.0144, and an Arellano–Bond-flavoured lag-instrument extrapolator isworse than the naive fit and is dropped. On LCBench and PD1 replays the story is honestly mixedand matches the calibrated-noise prediction: plain last-value SH is already near-optimal at decision level;naive power-law extrapolation hurts promotions and suffers a winner’s curse when picking from extrapolations (top-decile prediction bias 0.183 on LCBench); the corrected surrogate slashes that tail bias(∼7×, to 0.026) and ties last-value SH on LCBench decisions, but on PD1 it improves prediction whileworsening decisions, and a one-parameter variance-penalty hack captures most of the benefit for far lessmachinery — a prediction–decision gap we quantify rather than paper over.