What is the clinical evidence from this study?

Study design: Observational. Population: Pediatric ICU admission (n=8399). Intervention: Criticality Index-Mortality (CI-M) dynamic machine learning prediction algorithm. Primary outcome: Hospital survival or death during or following an ICU admission (AUROC 0.851, 95% CI 0.841-0.862).

December 1, 2022Open Access

The criticality Index-mortality: A dynamic machine learning prediction algorithm for mortality prediction in children cared for in an ICU

Key Result

The Criticality Index-Mortality dynamic machine learning algorithm predicted pediatric ICU mortality across all time periods with an AUROC of 0.851.

Study Design

Type

Observational (n=8,399)

Multicenter

Structured PICO

Population

8,399 pediatric and cardiac ICU admissions under 21 years of age, evaluated to develop and validate a dynamic machine learning mortality prediction algorithm.

Exposure

Dynamic mortality risk prediction every 3 hours using the Criticality Index-Mortality (CI-M) neural network model

Outcome

Hospital survival or death during or following an ICU admissionhard clinical

A dynamic machine learning model updating mortality risk every 3 hours in pediatric ICU patients demonstrated strong discrimination and calibration, potentially enhancing real-time clinical assessment of illness severity.

Main Result

Effect estimate: AUROC 0.851 (95% CI 0.841-0.862)

Limitations

Single site dataset with a relatively small sample size and potential for overfitting
Did not exhaust all possible modeling approaches available for machine learning
Data missingness has the potential to introduce bias
Unable to explore the details of risk trajectories for specific disease processes due to infrequent representation
Did not use diagnoses for risk assessments because they were not easily available in the EHR for individual time intervals
Relatively few deaths in the dataset
Dataset pre-dated the novel coronavirus pandemic

Abstract

Background The Criticality Index-Mortality uses physiology, therapy, and intensity of care to compute mortality risk for pediatric ICU patients. If the frequency of mortality risk computations were increased to every 3 h with model performance that could improve the assessment of severity of illness, it could be utilized to monitor patients for significant mortality risk change. Objectives To assess the performance of a dynamic method of updating mortality risk every 3 h using the Criticality Index-Mortality methodology and identify variables that are significant contributors to mortality risk predictions. Population There were 8,399 pediatric ICU admissions with 312 (3.7%) deaths from January 1, 2018 to February 29, 2020. We randomly selected 75% of patients for training, 13% for validation, and 12% for testing. Model A neural network was trained to predict hospital survival or death during or following an ICU admission. Variables included age, gender, laboratory tests, vital signs, medications categories, and mechanical ventilation variables. The neural network was calibrated to mortality risk using nonparametric logistic regression. Results Discrimination assessed across all time periods found an AUROC of 0.851 (0.841–0.862) and an AUPRC was 0.443 (0.417–0.467). When assessed for performance every 3 h, the AUROCs had a minimum value of 0.778 (0.689–0.867) and a maximum value of 0.885 (0.841,0.862); the AUPRCs had a minimum value 0.148 (0.058–0.328) and a maximum value of 0.499 (0.229–0.769). The calibration plot had an intercept of 0.011, a slope of 0.956, and the R 2 was 0.814. Comparison of observed vs. expected proportion of deaths revealed that 95.8% of the 543 risk intervals were not statistically significantly different. Construct validity assessed by death and survivor risk trajectories analyzed by mortality risk quartiles and 7 high and low risk diseases confirmed a priori clinical expectations about the trajectories of death and survivors. Conclusions The Criticality Index-Mortality computing mortality risk every 3 h for pediatric ICU patients has model performance that could enhance the clinical assessment of severity of illness. The overall Criticality Index-Mortality framework was effectively applied to develop an institutionally specific, and clinically relevant model for dynamic risk assessment of pediatric ICU patients.

Bookmark

View Full Paper