Evaluating the properties of the fragility index of meta-analyses

Abstract Background The fragility index (FI) has become an increasingly popular supplementary measure for evaluating the robustness of a study’s conclusions. While initially developed for individual clinical trials, the FI has been extended to meta-analyses (MAs) of multiple studies. However, the existing literature provides limited insights into the properties of the FI in the context of MAs. This study aims to explore various statistical methods for MAs and assess the improvement in FI of MAs compared to the individual studies they comprise. Methods We investigated the empirical distributions of FI and fragility quotient (FQ) using a large database of Cochrane MAs with binary outcomes. The FI of MAs was calculated under different statistical methods, including fixed-effect and random-effects models, with between-study variance estimators (restricted maximum-likelihood and DerSimonian–Laird), alongside Hartung–Knapp–Sidik–Jonkman (HKSJ) confidence interval adjustments. Subgroup analyses were performed to explore the impact of heterogeneity, sample size, and effect measures on fragility. Furthermore, we employed a metric to evaluate the improvement in fragility by comparing the FI of MAs with the FIs of the individual studies they included. Results The median FI was 5 (IQR: 2–11) among 3, 758 MAs analyzed, with 29% reporting statistically significant results. Notably, 15% of MAs had an FI of 1, and 54% had an FI ≤ 5. MAs with larger sample sizes or higher I^2 values, tended to exhibit greater robustness. HKSJ adjustments introduced more uncertainty, yielding more fragile results compared to analyses without these adjustments. Fragility improvement was higher in MAs with considerable heterogeneity. Conclusions This study highlights the variability in fragility across MAs and underscores the influence of heterogeneity and statistical methods on FI. Further research is warranted to refine the assessment of fragility and incorporate clinical relevance into these evaluations.