Answers are grounded in this paper and cite their sources.
Background Model validation is important to quantify the generalization capability of a predictive model or learning algorithm. In practice, data splitting is required to quantify the relevant out-of-sample performance reliably. For practitioners it is often unclear which of the available data splitting schemes is most suitable for their specific problem. This is particularly true when not only (in-distribution) reproducibility but also (out-of-distribution) transferability to another population or context needs to be assessed. Methods We performed a selective review to assess the documentation of external validations of clinical risk prediction models in the literature. We derived a novel estimand framework based on existing frameworks for treatment and diagnostic studies. To conduct statistical inference, we adopted different established methods for our purpose (exploratory visualization, three-level meta-analysis, Bayesian mixed model). Finally, we applied our framework in a hypothetical predictive modelling study based on real-world data. Results Our new framework allows a precise specification of the relevant performance estimand by defining constraints on inference and development context. To facilitate practical application, we provide an algorithm which allows estimand-aligned data splitting. We also propose a novel structured nested cross-validation scheme to enable an adequate empirical comparison of reproducibility and transferability. Our numerical experiments illustrate that the estimand can influence empirical validation results and should thus be carefully chosen. The meta-analysis approach offers a good compromise between simplicity and suitability for comparing the performance corresponding to different estimands. Conclusion Our framework facilitates the specification of the estimand in the predictive modelling context and thus helps to describe the validation gap between validation study and intended use more precisely. It can improve the current practice in the clinical risk prediction literature where the estimand is often only vaguely defined and the usefulness of the performance estimate(s) is thus unclear.
No takes yet. Share an insight, caveat, or question.
Alpers et al. (2026) studied this question.
Synapse has enriched 10 related papers. Consider them for comparative context: