External validation of GDM risk prediction models using a machine learning reciprocal model-exchange framework

BACKGROUND: Although many risk prediction models have been developed, very few undergo external validation, primarily due to issues with data access. Therefore, we implemented a reciprocal model-exchange approach to facilitate external validation and demonstrate its use with gestational diabetes mellitus (GDM) prediction models. OBJECTIVE: To assess the robustness and generalisability of two independently developed GDM risk prediction models using a reciprocal model-exchange framework. METHODS: Two independently developed GDM risk prediction models were externally validated using a reciprocal model-exchange. The saved model's corresponding variable types and data pre-processor were exchanged. The Monash CatBoost model was validated using Irish data at Dublin City University (DCU), and the DCU logistic-regression GDM model was validated using Australian data at Monash University. Performance was assessed using discrimination, calibration and decision curve analysis. Model fairness was assessed. RESULTS: The prevalence of GDM was 21.1% in the Australian cohort and 11.7% in the Irish cohort. The Monash model's AUC dropped from 0.93 to 0.77, while the DCU model's AUC fell from 0.82 to 0.69. Calibration estimates confirmed systematic risk misestimation; each model tends to over or under-predict GDM probabilities outside its training domain, with calibration-in-the-large of -0.573 for the Monash model and 0.17 for the DCU model; slopes were 1.278 and 0.55 respectively. Both models showed performance variability across ethnic groups, with lower performance for Southeast/Northeast Asians and both performed better with increasing parity and among women without a prior GDM diagnosis. CONCLUSIONS: Each model's performance decreased upon external validation, and the fairness evaluations on the different sub-categories (ethnicities; parity and previous GDM) provided evidence on the areas to be addressed in model recalibration/updating before deployment can be progressed. This reciprocal model-exchange approach provides a solution to facilitating external validations, which are notably lacking in the current literature but are necessary to advance the risk prediction field.