Development and Validation of an AI Tool for Automated PICO Scoping for the European Joint Clinical Assessment (JCA): A Proof of Concept

OBJECTIVE: The European Union Health Technology Assessment regulation (EU-HTAR) requires laborious PICO scoping (Patient population, Intervention, Comparator, Outcome) and consolidation across member states during joint clinical assessments (JCAs). We developed a tool to automate PICO extraction from HTA documents and clinical guidelines. METHODS: We developed a proof-of-concept AI pipeline using retrieval-augmented generation and large language models to process documents, translate to English, extract PICOs, and consolidate findings. The system was validated using two oncology cases from non-small cell lung cancer (NSCLC) and hepatocellular carcinoma (HCC). We curated datasets for both cases, comprising 49 reports totalling 5,463 pages from 16 unique countries. Performance was assessed using recall (sensitivity) & precision (positive predictive value) and compared against human data extraction. RESULTS: The AI system demonstrated strong recall performance, achieving 0.807 for NSCLC and 0.837 for HCC. The AI outperformed human extraction in the NSCLC case (0.81 versus 0.71 recall) while matching human performance in HCC (0.84 versus 0.81). Extraction of the comparator element showed high recall (0.911 NSCLC; 0.967 HCC). The AI pipeline required 96 minutes of computational time beyond translation, saving approximately 25 hours of human screening work. CONCLUSIONS: We showed that AI-based PICO extraction was feasible and valuable in two use-cases. The system showed good recall and had the potential to reduce workload in the JCA. It may be useful for the assessors and co-assessors in the EU-HTAR process, the industry in gaining insight into PICO expectations, and national HTA bodies in understanding the expected PICO wishes of other nations.