Bayesian Sample Size Calculations for External Validation Studies of Risk Prediction Models

ABSTRACT Contemporary sample size calculations for external validation of risk prediction models require users to specify fixed values of assumed model performance metrics alongside target precision levels (e.g., 95% CI widths). However, due to the finite samples of previous studies, our knowledge of true model performance in the target population is uncertain, and so choosing fixed values represents an incomplete picture. As well, for net benefit (NB) as a measure of clinical utility, the relevance of conventional precision‐based inference is doubtful. In this work, we propose a general Bayesian framework for multi‐criteria sample size considerations for prediction models for binary outcomes. For statistical metrics of performance (e.g., discrimination and calibration), we propose sample size rules that target desired expected precision or desired assurance probability that the precision criteria will be satisfied. For NB, we propose rules based on Optimality Assurance (the probability that the planned study correctly identifies the optimal strategy) and Value of Information (VoI) analysis, which quantifies the expected gain in NB by learning about model performance from a validation study of a given size. We showcase these developments in a case study on the validation of a risk prediction model for deterioration among hospitalized COVID‐19 patients. Compared to conventional sample size calculation methods, a Bayesian approach requires explicit quantification of uncertainty around model performance, and thereby enables flexible sample size rules based on expected precision, assurance probabilities, and VoI. In our case study, calculations based on VoI for NB suggest considerably lower sample sizes are required than when focusing on the precision of calibration metrics. This approach is implemented in the accompanying software.