Validation Strategy Dominates Model Choice in Chronological QSAR Evaluation

Quantitative structure–activity relationship (QSAR) models are commonly evaluated by random cross-validation or random train-test splits, but these retrospective procedures may overestimate performance when validation compounds are structurally or temporally close to the training data. In this study, we examined whether this optimism is primarily explained by structural redundancy, chronological data set evolution, or the choice of machine-learning model. Fourteen public ChEMBL bioactivity data sets were curated as binary classification tasks. Five validation conditions were compared: random validation, chronological validation by publication year, a size-matched randomly thinned control, structural filtering of the random validation set, and structural filtering of the chronological validation set. Model development was deliberately held fixed within the random-derived and year-derived comparisons, so that the effect of validation-set composition could be assessed without simultaneously changing the training cohort. Four standard QSAR model classes─random forest, support vector machine, XGBoost, and a feed-forward neural network─were evaluated under the same training-only model-selection workflow. Random validation produced closely aligned internal and external estimates, with mean MCC values of 0.75 and 0.72, respectively, while random thinning gave essentially unchanged validation performance (MCC 0.73). Structural filtering reduced validation MCC to 0.48 without changing the corresponding training-only CV estimate. Chronological validation produced a much larger generalization gap, with mean CV MCC of 0.73 and validation MCC of 0.19; additional structural filtering reduced validation MCC to 0.15. Model-class comparisons showed substantial overlap among classifiers, and no standard algorithm consistently overcame the chronological validation penalty. These results indicate that validation-set definition has a dominant influence on estimated QSAR generalization and that chronological data set shift is not reproduced by random or structure-only validation.