Honest ADMET: When the Generalization Gap Appears, Why, and Whether Selective Prediction Recovers It

Public ADMET leaderboards such as Therapeutics Data Commons (TDC) rank models by a single score under one fixed split. We ask three questions a deployer actually faces, across all 22 TDC ADMET endpoints: (i) how much does measured performance fall under realistic, chemistry-aware splits versus an optimistic random split; (ii) why the effect is heterogeneous and sometimes reverses; and (iii) whether selective prediction recovers the reliability that random-split numbers overstate. We evaluate two molecular representations (ECFP4 fingerprints and a frozen pretrained-transformer embedding) across all 22 endpoints. We make three contributions. First, a statistically honest gap atlas: signed, per-endpoint relative gaps with matched-replicate, split-seed block-bootstrap confidence intervals, paired Wilcoxon tests, and Benjamini-Hochberg FDR. Bemis-Murcko scaffold splits are much weaker stressors than cluster (sphere-exclusion) splits for ADMET — their median relative gap is about 2.3x smaller (+0.10 vs +0.23) and not established on 5 of 22 endpoints including hERG, consistent with Fooladi et al. (2025) — while cluster splits reveal a gap established on 21 of 22 endpoints (aggregate p < 1e-4). Second, error spikes roughly 8x in the most dissimilar nearest-neighbour similarity bin, so the gap is an applicability-domain effect. Third, we report an honest evaluation of uncertainty under shift, negatives included: split-conformal under-covers under cluster shift, covariate-shift weighting is a verified no-op that does not restore coverage, the best abstention signal is model-dependent, and per-endpoint abstention benefit does not track the optimism gap (Spearman rho = 0.07, n.s.). Frozen ChemBERTa embeddings shrink the gap modestly but significantly (p = 0.011) without closing it. All re-split numbers are anchored to TDC's official-split fingerprint baselines first; code and splits are released as a pip-installable, leakage-audited harness.