The use of interpretability methods in machine learning models for predicting current disease presence or future disease occurrence using structured data on potential risk factors: a systematic review of observational epidemiological studies in populations unselected for existing disease

Background Machine learning (ML) is increasingly used for predictive modelling of disease outcomes; however, real-world uses remain hampered by the limited interpretability of complex models. In this review, we provide a comprehensive synthesis of interpretability methods used for disease prediction in ML models focusing on studies which aim to identify risk-associated predictors across observational epidemiological designs. Methods We searched two biomedical databases (PubMed and MEDLINE) and two preprint repositories (bioRxiv and medRxiv) using combination of terms for ML interpretability and disease outcomes. We targeted primary studies developing/validating ML models to predict disease outcomes in non-clinical populations using structured data, excluding studies conducted in a care setting or in populations with pre-existing conditions. We also excluded studies which used unstructured data models, or which focused on intermediary outcomes, COVID‑19, mortality, prognosis, or hospital/surgical endpoints. We assessed the risk‑of‑bias (ROB) using a tool tailored to interpretability. The review protocol was registered in PROSPERO (CRD42024552595) and conducted according to PRISMA guidelines. Results We identified 2,123 records and after removing the duplicates, 1,345 studies were taken forward for title/abstract screening. Of the 100 potentially relevant studies, 82 were deemed eligible. The identified studies used 24 interpretability methods, with SHAP (SHapley Additive exPlanations) being the most common (44/82, 54%), followed by built‑in model feature importance (20/82, 24%), permutation feature importance (11/82, 13%), partial dependence plots (5/82, 6%), and LIME (Local Interpretable Model-agnostic Explanations) (4/82, 5%). Rule‑based approaches appeared in two studies and logistic regression coefficients/odds ratios in three. Most studies only provided global explanations (62/82, 76%). Presentation format was predominantly graphical (77/82, 94%) and tables were used in 32 studies (39%). Implementation of interpretability methods in prototype/real‑world tools was reported in seven studies (9%). Calibration of predicted probabilities was assessed in 26 (32%) studies. Only nine studies (11%) met all seven ROB criteria. Explanation stability was assessed in 32 of the 44 studies (73%) for which a clear assessment was possible; the remaining 38 studies were classified as unclear regarding explanation stability. Subgroup/bias‑related variation in explanations was assessed in 17 of the 63 studies (27%) for which clear assessment was possible and the remaining 19 studies were classified as unclear. Conclusions Interpretability methods are widely used in ML-based disease prediction in observational epidemiological studies using structured data on potential risk factors, but evaluation and reporting are inconsistent. We recommend standardised interpretability reporting, calibration of models when predicted probabilities are intended to represent absolute risk, application of multiple interpretability methods, routine subgroup/fairness analyses, and quantification of explanation variability.