Key result
Targeting readmission interventions by predicted cost is linked to ~69% greater utility gain versus standard models.
Why the study?
Machine learning models for allocating readmission-mitigating interventions are typically selected by discriminative ability, which may not translate into utility in resource allocation.
Do machine learning models optimized for utility outperform those optimized for discriminative ability in allocating readmission-mitigating interventions?
Observational (n=513,495)
Yes
Do machine learning models optimized for utility outperform those optimized for discriminative ability in allocating readmission-mitigating interventions?
Absolute Event Rate: 175.94% vs 104%
Machine learning models for resource allocation should be optimized directly for utility rather than just discriminative ability, as higher AUC does not necessarily translate to greater cost savings.
Optimizing ML models for utility may improve readmission intervention allocation; hypothesis-generating and should not yet change practice.
OBJECTIVE: Machine learning (ML) models for allocating readmission-mitigating interventions are typically selected according to their discriminative ability, which may not necessarily translate into utility in allocation of resources. Our objective was to determine whether ML models for allocating readmission-mitigating interventions have different usefulness based on their overall utility and discriminative ability. MATERIALS AND METHODS: We conducted a retrospective utility analysis of ML models using claims data acquired from the Optum Clinformatics Data Mart, including 513,495 commercially-insured inpatients (mean [SD] age 69 [19] years; 294,895 [57%] Female) over the period January 2016 through January 2017 from all 50 states with mean 90 day cost of $11,552. Utility analysis estimates the cost, in dollars, of allocating interventions for lowering readmission risk based on the reduction in the 90-day cost. RESULTS: Allocating readmission-mitigating interventions based on a GBDT model trained to predict readmissions achieved an estimated utility gain of $104 per patient, and an AUC of 0.76 (95% CI 0.76, 0.77); allocating interventions based on a model trained to predict cost as a proxy achieved a higher utility of $175.94 per patient, and an AUC of 0.62 (95% CI 0.61, 0.62). A hybrid model combining both intervention strategies is comparable with the best models on either metric. Estimated utility varies by intervention cost and efficacy, with each model performing the best under different intervention settings. CONCLUSION: We demonstrate that machine learning models may be ranked differently based on overall utility and discriminative ability. Machine learning models for allocation of limited health resources should consider directly optimizing for utility.
No takes yet. Share an insight, caveat, or question.
Ko et al. (2021) conducted an observational in Hospital readmission (n=513,495). Machine learning model trained to predict cost vs. GBDT model trained to predict readmissions was evaluated on Estimated utility gain per patient. Allocating readmission interventions based on a model predicting cost achieved a higher utility gain ($175.94 per patient) compared to a standard readmission prediction model ($104 per patient).
Synapse has enriched 5 closely related papers on similar clinical questions. Consider them for comparative context: