Sources of Measurement Error in the Critical Appraisal Tool for Anatomical Meta-Analysis: An Item-Level and Generalizability-Theory Evaluation.

The Critical Appraisal Tool for Anatomical Meta-analysis (CATAM) was introduced to support anatomy-specific appraisal of the methodological quality and reporting of anatomical meta-analyses (AMAs). While prior work has supported its face and content validity and suggested high overall reliability under selected conditions, comprehensive evidence regarding its inter-rater reliability, item-level agreement, and sources of measurement error remains limited. This study aimed to provide an in-depth psychometric evaluation of CATAM by assessing (i) inter-rater reliability of the total score, (ii) item-level inter-rater agreement using robust ordinal multi-rater statistics, and (iii) sources of variance using Generalizability Theory, with the additional objective of identifying domains in which disagreement concentrated under independent, uncalibrated application. Fifty AMAs published in 2024 were identified through structured searches of PubMed, Scopus, and Google Scholar and independently evaluated by four blinded raters using the 14-item CATAM instrument. Inter-rater reliability of total scores was assessed using two-way random-effects intraclass correlation coefficients (ICC). Item-level agreement was evaluated using Gwet's AC2 with ordinal weights (primary) and Krippendorff's α (secondary). Generalizability Theory was applied under a fully crossed p × r × i design to partition variance attributable to meta-analyses, raters, items, and their interactions. Decision studies examined how reliability varies as a function of the number of raters and items. Single-rater reliability of the total CATAM score was low, whereas averaging scores across four raters yielded acceptable reliability under the present protocol (ICC(2,4)≈0.69). Item-level agreement varied substantially, ranging from perfect agreement to poor agreement across items. Five items (Abstract, Introduction, Methods/Searching Strategy, Results/Outcomes, and Discussion) fell below the pragmatic Gwet's AC2 screening threshold of 0.60, although two of these (Methods/Searching Strategy and Discussion) were near-threshold findings that should be interpreted cautiously. Generalizability analysis revealed that item-related variance dominated total score variability, while rater main effects and meta-analysis × rater interactions were negligible. The generalizability coefficient for relative decisions was moderate (G≈0.66), and the dependability coefficient for absolute decisions treating items as fixed was Φ≈0.59. Decision studies indicated that four raters provide an efficient balance between feasibility and reliability. Overall, CATAM demonstrated acceptable reliability when ratings were aggregated across multiple independent raters under the present protocol, although item-linked variance exceeded rater main-effect variance and these analyses cannot fully separate item characteristics from differences in how raters operationalized the published guidance. These findings support calibration with explicit decision rules and rescoring as the next methodological step; only if disagreement persists after that process should any future Delphi-based refinement be considered.