Performance metrics in systematic reviews and meta-analyses of artificial intelligence-based prediction models for children and pregnant women: protocol for an umbrella review

Artificial intelligence (AI) has shown significant promise in medical diagnostics. However, children and pregnant women represent “vulnerable populations” often underrepresented in large-scale training datasets. This paper summarizes a protocol for an umbrella review aiming to identify and summarize the performance metrics reported in systematic reviews and meta-analyses (SRMAs) of AI-based prediction tools for pediatric and obstetric conditions and to evaluate the methodological quality of these reviews. We will search the PubMed, EMBASE, Web of Science, and Cochrane Library from inception to identify eligible studies. Eligibility criteria include SRMAs of health conditions specific to children and pregnant women, and full-text articles published in English. Following screening and selection, two independent reviewers will perform data extraction in accordance with the Joanna Briggs Institute (JBI) Manual for Evidence Synthesis. Methodological quality will be assessed using the Assessing the Methodological Quality of Systematic Reviews 2 (AMSTAR2) tool. Our review will involve categorization by AI-based methodology, presence of external validation, and the type of health outcomes. The model types, performance metrics, and bias-mitigation strategies reported in the included systematic reviews will be reviewed. Following title and abstract screening, we will include only SRMAs that focus on AI-based prediction models for health conditions specific to children and pregnant women, or stratified analysis by children and pregnancy status. The results will be presented through narrative synthesis and summary tables. Our anticipated findings will emphasize the specific health conditions addressed by these SRMAs, along with the absence of consistent reporting standards for both performance and fairness metrics within these populations. This review will highlight the need for standardization of AI performance reporting, focusing on commonly researched domains in pediatric and obstetric care. This is critical for ensuring the fair and equitable adoption and implementation of these advanced technologies.