A novel approach for meta-analysis of feature importance measures from machine-learning models in multi-cohort studies

Abstract Background Machine learning (ML) methods are increasingly used in observational epidemiological studies to identify predictive features. To synthesise results from analyses done across multiple cohorts, meta-analyses are usually applied. However, there are currently no methods to quantitatively synthesise results from multiple studies based on ML methods. Purpose Our aim was to describe a novel meta-analytical approach (MetaRanks) for assessing the association of multiple environmental exposures with Body Mass Index (BMI) across eight cohorts, as well as quantifying between-study heterogeneity and uncertainty. Meta-ranks offers a generic framework to effectively combine results obtained from different ML models applied to multiple cohorts. Methods MetaRanks is a meta-analytical model that summarises ranked effects. We applied it in a multi-cohort analysis of eight cohorts affiliated with the Dutch Geoscience and Health Cohort Consortium. Studies varied in population sizes (1,106–141,825). Associations with 69 residential neighbourhood environmental factors (air pollution, noise, temperature, neighbourhood socio-economic and demographic factors, food environment, drivability, and walkability) were explored in each study using random forest (RF) models. Meta-analytic models were fitted for each covariate using a two-stage approach: 1) beta-distributions were fitted to the bootstrapped RF ranks, thus estimating the per-study precision of ranks; and 2) a Bayesian model was fitted using the "rstan" R package with the parameters of the observed rank and its estimated precision from stage one to estimate the overall rank across the studies and heterogeneity. The heterogeneity was assessed as the ratio between the predicted interval (rank values in a new study) and the credible interval (rank values in the current meta-analysis). Results The meta-analytic models were used to estimate the overall importance of variables across studies. The top ten predictors were age, educational level, walkability, average job accessibility, average residence values in neighbourhood, the absorbance of particulate matter (PM) and the coarse fraction of PM. The heterogeneity was lower for the most important and the least important predictors. The highest degree of heterogeneity was found in indicators of built environment and neighbourhood-level socio-demographic characteristics. Conclusion To our knowledge MetaRanks is the first method for conducting covariate-oriented meta-analysis on data derived from a ML model applied to different datasets. It allows assessing the importance of different covariates across studies and their variability between studies. It can be employed with any kind of ML approach or feature importance scores.