A source-grounded benchmark for extracting causality-relevant pharmacovigilance evidence from liver injury case narratives

Liver injury case narratives contain causality-relevant information on exposure chronology, dechallenge, rechallenge, co-exposures, competing causes, phenotype, product characteristics, and outcome. Large language models (LLMs) may assist structured pharmacovigilance evidence abstraction, but parseable output alone does not establish agreement with source evidence or safe use. We conducted a descriptive single-run benchmark using 120 LiverTox-derived narratives held out from prompt and schema development: 42 herb-induced or botanical/herbal dietary supplement-associated liver injury cases and 78 conventional drug-induced liver injury comparators. A locked schema defined 40 fields per case. An initial single-expert reference was followed by a model-triggered, model-blinded human source re-audit of 240 unique case-field rows, yielding 184 corrections before final rescoring. GPT, Codex, and a heterogeneous Gemini mixed-endpoint condition were evaluated. Outcomes included status-schema adherence, agreement with the final curated reference, raw and normalized exact match, evidence-span provision and verbatim span match, and a six-rule targeted safety-discrepancy screen with blinded human adjudication. All 360 outputs were parseable. Valid-status rates were 100.0% for GPT, 99.9% for Codex, and 63.5% for the Gemini mixed-endpoint condition. Status agreement with the final curated reference was 81.5%, 81.5%, and 50.3%, respectively; normalized exact-match rates were 60.0%, 60.5%, and 30.8%. Verbatim span-match rates among nonblank model-provided spans were 100.0%, 99.8%, and 95.6%. The targeted screen generated 15, 25, and 40 automated flags, of which blinded adjudication confirmed 10, 19, and 38 potentially causality-distorting model errors. Unsupported absence of co-exposures was the most frequent confirmed category. Under the evaluated single-run conditions, the execution conditions produced parseable structured outputs, but schema adherence, agreement with the final curated reference, and targeted safety-relevant error profiles varied. Structured output and verbatim evidence spans did not remove the need for human verification. The study did not evaluate automated causality assessment, clinical readiness, or the effectiveness of a human-in-the-loop workflow. Not applicable.