Abstract Rationale As AI clinical decision support models become increasingly integrated into hospital workflows, ongoing evaluation of deployed models is essential to detect performance drift. Despite growing deployment of clinical AI systems, few studies have reported real-world, longitudinal evaluation of model performance. We evaluated the temporal stability of MEWS ++, a clinical deterioration prediction model deployed in production to monitor patients admitted to step-down units in a large tertiary care medical center. Methods MEWS ++ was originally trained in 2017 and prospectively validated in 2020. In mid-2023, the model was re-deployed in step-down units to predict escalation of care, defined as transfer to an intensive care unit or in-unit death. We analyzed model performance across four six-month time periods (2023 Q3-4, 2024 Q1-2, 2024 Q3-4, 2025 Q1-2). For each step-down admission, a binary outcome label (escalation yes/no) was created, and the last model score before escalation was used for prediction. Performance was assessed using AUROC, AUPRC, and standard classification metrics at a pre-specified high-risk threshold. Differences in sensitivity, specificity, and precision across periods were compared using chi-square testing and Wilson confidence intervals. Results A total of 8,228 step-down admissions were included. From 2023 Q3-4 to 2025 Q1-2, composite escalation rate rose from 11.9% to 13.9%. Model AUROC rose from 0.77 (95% CI 0.73-0.81) in 2023 to 0.83 (95% CI 0.79-0.86) in 2025, and model AUPRC rose from 0.29 (95% CI 0.23-0.37) to 0.44 (95% CI 0.36-0.53) (Figure). At a fixed threshold of 0.64, model sensitivity rose from 40.3% (95% CI 34.4%-46.6%) to 53.4% (95% CI 47.8%-58.8%) (p = 0.002); model specificity remained stable at 88.4% (95% CI 86.8%-89.9%) to 89.5% (95% CI 88.0%-90.8%) (p = 0.14); and model precision rose from 31.9% (95% CI 27.0%-37.3%) to 45.0% (95% CI 40%-50.1%) (p 0.001). Conclusions Over two years of deployment, in the context of rising event prevalence in the underlying population, MEWS ++ maintained stable AUC with increased AUPRC and both higher sensitivity and precision with stable specificity. The direction of performance drift contrasts with expectations for static models, as performance increased rather than decreased with time. Further work will evaluate factors driving data drift including: (1) input drift with change in case-mix or input feature distribution, or (2) concept drift with change in clinical practice or the feedback effects of model predictions on care. Ongoing performance monitoring of deployed AI models is essential to ensure stable performance as clinical environments evolve. This abstract is funded by: None
Tandon et al. (Fri,) studied this question.