When the Best Uplift Model Is Uncertain: Learner-Selection Credibility and Policy-Value Robustness

Heterogeneous treatment-effect (HTE) and uplift models prioritize individuals for interventions, but the apparent best learner may depend on the evaluation metric, sample, and deployment budget. We examine learner-selection credibility using two randomized uplift benchmarks and a factorial simulation with known conditional treatment effects. Protected-holdout analyses use paired treatment-stratified Poisson multiplier bootstraps, Holm-adjusted pairwise comparisons, calibration, retraining stability, and falsification tests. In the Criteo eight-model comparison (n=150,986), 2,000 bootstrap replicates yielded no Holm-significant fixed-depth comparisons across 5%, 10%, and 20% targeting budgets (0/84), but 13 of 28 Qini-style global-ranking comparisons were significant. The Hillstrom replication reported no Holm-significant fixed-depth (0/84) or Qini (0/28) comparisons. In 480 heterogeneous simulation replicates, observed Qini selected the true best evaluated top-10% policy in 17.50%, while deployment-matched observed top-10% uplift did so in 16.25%. Their selected policies retained 94.9% and 94.7% of the best candidate's true policy value on average. These results distinguish statistical evidence about global ranking from evidence of fixed-budget policy superiority, and exact winner identification from decision-value loss.