A leakage-controlled cold-start benchmark for drug-combination synergy classification

Only a small fraction of possible drug pairings can be screened experimentally. Most synergy-prediction models rely on rich molecular and omics inputs, and it is under-reported how much signal is recoverable from pre-treatment identity-and-context descriptors alone. We present a reproducible, leakage-controlled benchmark asking how well synergy class can be predicted from compound identities, cell line, tissue, study, clinical-development phase and target annotations, without using any measured dose-response readout. Using DrugComb v1.5 ( \(739,964\) experiments across \(26\) studies, \(17\) tissues, \(288\) cell lines and \(4,268\) compounds), we derive a three-class endpoint from the Zero Interaction Potency (ZIP) score at conventional \(± 10\) thresholds and exclude every response-derived variable. A logistic-regression baseline and a class-weighted gradient-boosting (LightGBM) model are evaluated under the predefined grouped split and under cold-start splits. On the standard test set, LightGBM attains \(0.782\) accuracy, \(0.805\) balanced accuracy, \(0.703\) macro-F1, \(0.545\) Cohen’s \(κ \) and \(0.922\) macro one-vs-rest AUC, outperforming logistic regression by paired McNemar testing ( \(p < 0.001\) ). Precision–recall analysis shows roughly six-fold enrichment of the rare antagonistic and synergistic classes over their priors. However, performance decreases sharply under distribution shift: balanced accuracy falls to \(0.694\) for unseen drugs and to \(0.482\) for unseen cell lines, and to the random floor for entirely unseen studies; a no-study ablation shows this is not driven by the study identifier. Permutation importance identifies cell-line identity as the dominant predictor, and a per-(cell line, drug) base-rate lookup recovers \(0.778\) balanced accuracy, within \(0.027\) of the full model, indicating that most of the standard-split performance reflects conditional memorization rather than learned pharmacology. Useful synergy triage is achievable for known entities without molecular features, but standard splits partly measure memorization of entity-specific base rates and can materially overstate cold-start deployment performance. The benchmark provides a leakage-controlled lower bound against which richer molecular and omics models can be measured, and argues that cold-start evaluation should be reported alongside conventional random or grouped splits.