A17-05 Comparing Cross-Site Validation, Transfer Learning, and Independent Training for ICU Mortality Prediction: A U.S. Multicenter Study

Rationale Deploying machine learning models across health systems remains challenging, with widely implemented algorithms such as commercial sepsis prediction demonstrating poor external validity. While intensive care unit (ICU) mortality prediction models can support clinical decision-making, optimal strategies for model transfer remain unclear. We evaluated three deployment strategies across hospitals to offer practical, evidence-based guidance for implementing prediction models in the ICU. Methods We conducted a federated analysis using the Common Longitudinal ICU data Format (CLIF) Consortium, a geographically diverse network of 9 academic health systems. We identified adult patients admitted to ICUs at 9 sites between 2018-2024 (2018-2023 for training and validation, 2024 for testing) who survived for at least 24 hours. Patient demographics, vital signs, laboratory results, and respiratory support data (52 variables) were extracted from electronic health records (EHR) 24 hours after ICU admission. The primary outcome was in-hospital mortality occurring at any time following the initial 24-hour period. We trained gradient boosted machine (XGBoost) and neural network binary classifiers to predict in-hospital mortality using three deployment strategies: 1) Cross-Site Validation, wherein a model trained at a single source institution was tested on all other sites as is; 2) Transfer Learning, wherein the source model was fine-tuned with a limited subset of local site data; and 3) Independent Training, wherein models were trained exclusively on target site data without leveraging prior knowledge. We evaluated discrimination using area under the receiver operating characteristic curve (AUC), calibration metrics, and decision curve analysis, following TRIPOD-AI guidelines. Results The overall cohort comprised 428,924 ICU hospital encounters (mean age 61 years, 45% female). Overall mortality was 9% (range 7-13% across sites). For XGBoost models, the cross-site validation AUC was 0.85 [95% CI 0.84-0.87], transfer learning AUC 0.86 [95% CI 0.84-0.87], and independent training AUC 0.87 [95% CI 0.85-0.89]. For neural networks, cross-site validation AUC was 0.81 [95% CI 0.75-0.87], transfer learning AUC 0.86 [95% CI 0.82-0.90], and independent training AUC 0.86 [95% CI 0.82-0.90]. Brier scores ranged from 0.05 to 0.13, improving with transfer learning and independent training (Figure 1). Conclusions Across nine health systems, transfer learning for neural networks matched independent training performance while XGBoost demonstrated consistent discrimination and calibration across all approaches. Based on these findings, optimal deployment strategies may differ by modeling technique. However, local validation in a specific patient population is essential, regardless of deployment approach. Future research should evaluate these patterns across other modeling techniques and ICU outcomes. This abstract is funded by: Institute for Translational Medicine Pilot Award