Discrimination stability and calibration of cardiovascular risk prediction models in the Framingham baseline cohort

In primary prevention of cardiovascular disease, risk prediction models require not only adequate discrimination but also robust stability and reliable probability calibration to support clinical decision-making. Using data from 4434 participants in the Framingham baseline cohort, this study systematically compared the performance of three representative prediction models for cardiovascular risk assessment: an L2-regularized logistic regression model, a lightly tuned XGBoost model, and an AutoML-XGBoost model with automated hyperparameter optimization. Model performance was evaluated using repeated stratified data splitting, with discrimination assessed by the area under the receiver operating characteristic curve (AUC), calibration evaluated using the Brier score and calibration curves, stability examined across repeated splits, and consistency of risk ranking analyzed between models. The results showed that the three models achieved comparable levels of discrimination, exhibiting similar AUC distributions across repeated validations, with no clear advantage of more complex models over the traditional regression approach. The logistic regression model demonstrated the smallest performance variability across data splits and achieved the lowest Brier score, indicating the most stable and well-calibrated probability estimates. The AutoML-XGBoost model outperformed the lightly tuned XGBoost model in terms of discrimination and stability, but its calibration performance remained slightly inferior to that of logistic regression. Risk ranking consistency analysis further revealed a high Spearman rank correlation of approximately 0.95 between logistic regression and AutoML-XGBoost, with an overlap of about 71% in individuals classified within the highest-risk decile, whereas the correlation between logistic regression and the lightly tuned XGBoost model was approximately 0.83, with only about 50% overlap in the highest-risk group. These findings indicate that increasing model complexity does not necessarily lead to improved overall predictive performance in classical cardiovascular risk prediction tasks, and that multidimensional performance evaluation is essential for the appropriate selection and clinical application of prediction models.