Answers are grounded in this paper and cite their sources.
Background Clinical decision support systems (CDSS) require an assessment of predictive performance and the pathway supplying inputs, routing alerts, and recording workflow responses. Aggregate monitoring may not identify where predictions fail, while recorded digital interactions do not establish clinical appropriateness or patient benefit. Objective To determine whether linking records for each prediction across input data, alert decisions, delivery, clinician interaction, and outcomes identifies reliability signals missed by model performance alone. Methods We retrospectively evaluated deidentified EHR records and operational logs for four CDSS workflows: sepsis warning, inpatient deterioration, 30-day readmission, and outpatient laboratory-risk follow-up. The systems used three tree-based classifiers and one L2-regularized logistic regression. Data included 4,200 patients, 6,008 encounters, 9,402 episodes, 17,768 predictions, and 2,950 routing records. Measures before delivery covered silent validation, early deployment, and mature deployment; delivery and interaction outcomes were restricted to clinician-facing phases. Predictions were linked through input availability and timing, risk estimation, alert decisions, routing or delivery, interactions, and structured outcomes. Record review included 427 episodes. Results AUROC changes showed no consistent direction across systems (0.692–0.789); System C declined from 0.774 during silent validation to 0.692 during mature deployment. Calibration slopes ranged from 0.511 to 0.823, calibration-in-the-large from −0.633 to 0.022, and observed-to-expected ratios from 0.674 to 1.013. Early deployment had lower feature availability and longer ingestion latency. Across combinations of workflow and phase, 67.2%–87.8% of predictions had at least one unavailable feature, 21.5%–40.6% had an ingestion event after prediction, and routing or delivery failure was 2.1%–6.3%. During clinical deployment, 95.6% of production alerts were delivered; opening, acknowledgment, override, and recorded action rates were 63.8%–75.2%, 48.0%–59.0%, 10.1%–18.5%, and 17.3%–29.0%. None of the 160 subgroup contrasts remained supported after multiplicity adjustment. Conclusion Linking records for each prediction located reliability signals across input data, routing or delivery, and recorded responses that were not apparent from discrimination or performance at the operating threshold alone. Evaluation of the complete workflow complements rather than replaces discrimination, calibration, reference outcome assessment, and clinical utility evaluation. This retrospective single-institution study did not estimate causal effects or patient benefit; its estimates are local, and the monitoring structure is not a validated universal framework.
No takes yet. Share an insight, caveat, or question.
Kopanitsa et al. (2026) studied this question.
Synapse has enriched 10 related papers. Consider them for comparative context: