Continuous Health Scoring for Hyperparameter Optimization: Addressing Hyperband's False Positive and Slow Starter Failure Modes

Hyperband (Li et al., 2018) improves hyperparameter optimization efficiency by eliminating underperforming configurations at fixed checkpoints based on comparative ranking. It has two well-documented failure modes: false positives (configurations that rank well at a checkpoint but deteriorate afterward) and slow starters (configurations that rank poorly early but improve later). Both occur because Hyperband makes binary decisions at fixed intervals based on current performance level, not on improvement trajectory or health trend. We propose complementing Hyperband with continuous health scoring via the stochastic power metric P(t) = E(t) × W(t), which measures improvement velocity relative to an adaptive expected trajectory. Applied to real Pythia scaling suite data (Biderman et al. 2023), treating 7 Pythia models (70M–12B) as HPO configurations (eta=3, rungs at checkpoints 3 and 6), the combined HB+PM approach at θ=0.5 recovers the 1.4B slow starter that Hyperband eliminates at Rung 1 while correctly eliminating genuinely weak configurations. The 1.4B model improves +7.1% after Rung 1, reaching a final score 0.111 above the elimination cutoff — the largest gap in the search. The power metric catches them because it tracks improvement rate, not current level. These findings are preliminary and invite collaboration on validation using standard HPO benchmarks.