Unsafe or potentially unsafe advice from patient-facing large language model chatbots in bench evaluations of primary-care question sets: a systematic review and meta-analysis

Systematic review and meta-analysis (Cochrane-level standard, PRISMA 2020). Across 95 model arms from 23 bench studies, 552 of 4,868 clinician-rated responses from patient-facing LLM chatbots fell in the studies' own worst safety tiers (labelled unsafe or potentially unsafe in most studies); GLMM pooled proportion 10.5% (95% CI 9.3 to 11.9; Hartung-Knapp 12.5%, 11.3 to 13.9), prediction interval 4.5 to 22.7%, GRADE certainty very low. Restricted to tiers whose label asserts unsafe with no qualifier, the pooled proportion is 9.5% (8.0 to 11.2) on 24 arms from 5 studies. No human-user study reported response-level safety ratings, so the human-user stratum could not be pooled. The problematic (intermediate) tier and undertriage are reported study by study. The deposit includes the full manuscript, the settled extraction dataset, the analysis code (including the revision 1 re-analysis and its changelog) and the supplementary logs.