Performance Accuracy of ChatGPT for Data Extraction in Randomised Controlled Trials of Transdiagnostic Mental Health Apps: Proof of Concept Feasibility Study

Enthusiasm for digital mental health interventions is growing, with the number of trials evaluating their efficacy increasing rapidly each year. Synthesising this expanding evidence base will require significant time and resources. Large language models (LLMs) may streamline this process by automating key tasks such as data extraction. However, their accuracy in extracting data in digital mental health trials remains unclear. This study provides an evaluation of ChatGPT for data extraction from trials of digital mental health apps, examining accuracy, error patterns, and offering practical implications for use. Nineteen studies from a prior meta-analysis were included. Prompts instructing ChatGPT-4o and ChatGPT-5 to extract 15 data items from study PDFs were developed and tested on three studies before being applied to the remaining 16. Model outputs were compared with human extracted data to determine accuracy. ChatGPT-4o and 5 performed similarly, accurately extracting 83% and 85% of data, respectively. Several data items were extracted with 100% accuracy by both models (i.e., author, inclusion criteria, age, app name, CBT focus, outcome measurement tool). Analysed sample size yielded the lowest accuracy (56-62%). The most common error type was differences in interpretation of information between ChatGPT and human reviewers, particularly for items requiring expert judgement (e.g., provision of guidance). Findings indicate that ChatGPT may be useful for extracting straightforward information routinely reported in digital mental health trials, but careful human oversight remains necessary. Key challenges and directions for future work are discussed, including prompt optimisation, improved reporting standards, and the development of structured evaluation pipelines.