B – 65 Manual to Machine: Evaluating the Reliability of Artificial Intelligence for Systematic Review Data Extraction

Abstract Objective In an ongoing rapid systematic review of the use of behavior change techniques in cognitive rehabilitation interventions, Microsoft Copilot, a generative AI resource, will be utilized to support data extraction. The objective of this study is to examine data extraction reliability by Copilot vs. a human extractor. Method We searched PubMed, Scopus, WoS, and Embase databases (last updated 01/28/25) using the search terms “stroke” AND “cognition disorders” AND “cognitive training/ behavior change therapy” AND “single component/multicomponent intervention” with other variations. Review inclusion criteria included: randomized controlled trials of cognitive rehabilitation interventions applied to adults with stroke, published in English between 2004-2024, cognitive rehabilitation intervention, functional or cognitive outcome measures. 15 studies were reviewed. Data extraction was performed in parallel by one human extractor and Copilot using the same data extraction template. Data extracted included: metadata (author, location, etc.) and study outcomes. Interrater reliability was measured by assessing the agreement of extracted data between extractors with Cohen’s Kappa. Results Overall agreement between extractors was minimal (κ = 0.38; range: 0.29-0.46) There was greater agreement for metadata extraction (κ = 0.47; range: 0.38-1.0) which was minimal to almost perfect compared to outcome data (κ = 0.37; range: 0.27-0.44), which was minimal to weak. Conclusion The AI tool, Microsoft Copilot was not consistently reliable for data extraction. When utilizing an AI tool for data extraction for a systematic review, it should be restricted to the extraction of data with minimal complexity.