What is the clinical evidence from this study?

Study design: Systematic Review. Population: Orthopedic surgical outcomes (n=18). Intervention: Machine learning prediction models. Primary outcome: Proportion, performance, and transparent reporting (TRIPOD guidelines) of externally validated ML prediction models.

April 18, 2021Open Access

Availability and reporting quality of external validations of machine-learning prediction models with orthopedic surgical outcomes: a systematic review

Key Result

Only 10 of 59 available machine learning prediction models in orthopedic surgery have been externally validated, and the 18 available validation studies demonstrated incomplete reporting of performance measures with a median TRIPOD completeness of 61%.

Study Design

Type

Systematic Review (n=18)

Structured PICO

What is the availability and reporting quality of external validations of machine-learning prediction models for orthopedic surgical outcomes?

Population

18 studies externally validating 10 different machine learning (ML) prediction models in orthopedic surgical outcomes

Intervention

External validation of machine learning prediction models

Outcome

Proportion, performance (discrimination, calibration, decision-curve analysis), and transparent reporting (using TRIPOD guidelines) of externally validated ML prediction models

Most predictive ML models in orthopedics lack external validation, and available validation studies suffer from incomplete reporting, limiting their clinical implementation.

Limitations

Studies meeting the selection criteria may have been missed.
Potential bias as 5 of the 18 included studies originated from the authors' institution.
Publication bias may have occurred as successful external validations may be published more often.
AUCs presented in 3 studies may have been too optimistic as they used ROC metrics on imbalanced datasets.
The presented low percentage of ML prediction models externally validated may have been unfair due to the recent surge in ML model publications.
TRIPOD guidelines may not be perfectly suited for ML models, though TRIPOD-AI is currently in development.

Abstract

Background and purpose - External validation of machine learning (ML) prediction models is an essential step before clinical application. We assessed the proportion, performance, and transparent reporting of externally validated ML prediction models in orthopedic surgery, using the Transparent Reporting for Individual Prognosis or Diagnosis (TRIPOD) guidelines.Material and methods - We performed a systematic search using synonyms for every orthopedic specialty, ML, and external validation. The proportion was determined by using 59 ML prediction models with only internal validation in orthopedic surgical outcome published up until June 18, 2020, previously identified by our group. Model performance was evaluated using discrimination, calibration, and decision-curve analysis. The TRIPOD guidelines assessed transparent reporting.Results - We included 18 studies externally validating 10 different ML prediction models of the 59 available ML models after screening 4,682 studies. All external validations identified in this review retained good discrimination. Other key performance measures were provided in only 3 studies, rendering overall performance evaluation difficult. The overall median TRIPOD completeness was 61% (IQR 43-89), with 6 items being reported in less than 4/18 of the studies.Interpretation - Most current predictive ML models are not externally validated. The 18 available external validation studies were characterized by incomplete reporting of performance measures, limiting a transparent examination of model performance. Further prospective studies are needed to validate or refute the myriad of predictive ML models in orthopedics while adhering to existing guidelines. This ensures clinicians can take full advantage of validated and clinically implementable ML decision tools.

KI fragen

Bookmark

View Full Paper