Abstract Objective Systematic reviews and meta‐analyses (SRMAs) are critical for synthesizing evidence and guiding clinical and public health decision‐making. This study aims to evaluate the reliability, validity and reproducibility of the International League Against Epilepsy (ILAE) Commission on Epidemiology Risk of Bias Tool by comparing it against the Newcastle–Ottawa Scale (NOS) to inform whether the ILAE tool may serve as a valid alternative in epilepsy‐focused evidence syntheses. Methods This study was planned a priori on three consecutive SRMAs. We assessed 54 observational studies included in these SRMAs focused on psychiatric comorbidities in persons with epilepsy. Eligible studies had ≥30 participants per group and validated criteria for diagnosing epilepsy and psychiatric conditions. Two independent raters scored all studies using both tools. The ILAE tool comprises six specific domains: (1) Source of Study Population; (2) Completeness (Sensitivity) of Epilepsy Case‐Finding; (3) Sensitivity of Comorbidity Determination; (4) Accuracy of Epilepsy Diagnosis; (5) Accuracy of Comorbidity Diagnosis; and (6) Representativeness of Study Sample. Test–retest reproducibility used intraclass correlation coefficient (ICC). Correlation used Spearman's rho. Agreement used weighted kappa. Bland–Altman analysis reported mean difference. Results There was a strong positive correlation between NOS scores and ILAE ratings (Spearman's rho = 0.80, 95% confidence interval CI 0.68–89, p < 0.001). Cohen's weighted kappa was 0.68 (95% CI 0.37–0.92, p < 0.001). Bland–Altman mean difference was 0.09 with limits from −0.48 to 0.67, showing good agreement between tools. The ILAE tool showed excellent test–retest reproducibility (ICC 0.90, 95% CI 0.83–0.94). Significance The ILAE tool demonstrated strong reliability and substantial agreement with the NOS while offering epilepsy specific rigor in diagnostic accuracy, sensitivity, case finding and representativeness. The ILAE tool offers a reliable, conceptually relevant, field‐specific alternative for quality assessment in epilepsy SRMAs.
Kwon et al. (Sun,) studied this question.