Abstract Systematic reviews are essential but labor-intensive. We evaluated LLM-assisted literature screening and drafting in two hepatology topics: carvedilol in compensated cirrhosis and anticoagulation in portal vein thrombosis. For each topic, we searched PubMed, Cochrane, and EMBASE. A few-shot prompt with explicit inclusion/exclusion criteria was used to screen titles and abstracts, with results compared to manual review. Included studies were then processed using a retrieval-augmented LLM to generate ten automated systematic review and meta-analysis drafts per topic, which were evaluated by a separate judge LLM for PRISMA 2020 compliance against human reviews. Screening performance: After deduplication (703 and 370 records), LLM-assisted screening showed high agreement with manual review (sensitivity 86–93%, specificity 96–99%) while reducing screening time to 3 and 2 h versus 62 and 30 h manually. Drafting performance: RAG-enabled LLMs generated structured manuscripts with variable PRISMA 2020 compliance: 100% for titles, 91% for introductions, 75–80% for methods, and 68–75% for results, with downstream weaknesses in abstracts and discussions (90%, but automated drafting was reliable mainly for titles and introductions, requiring human oversight to prevent errors and hallucinations.
Zou et al. (Mon,) studied this question.