Adjudicator sensitivity of safety estimates in large language models for open-ended cardiovascular reasoning: a blinded multi-adjudicator benchmark study running title: adjudicator variability in LLM safety

Large language models (LLMs) are increasingly evaluated for safety-critical clinical reasoning, but safety estimates in open-ended tasks may depend on expert adjudication thresholds. We evaluated 300 blinded responses generated by three API-accessed LLM configurations across 100 cardiovascular scenarios. Two cardiologists independently scored all responses, and the original workflow used predefined triggers for safety adjudication. First-adjudicator ascertainment was subsequently completed for all responses, after which a separate non-author cardiologist independently re-adjudicated all 300 responses while blinded to model identity, reviewer information, and prior decisions; 53 discordant classifications were then assessed by a third blinded cardiologist using a majority-resolution rule fixed before receipt of the third-adjudicator results. Continuous quality-score reliability was poor (ICC 0.028). The first adjudicator classified 53/300 responses (17.7%) as safety-positive, whereas the independent second adjudicator classified 0/300. First–second crude agreement was 82.3%, but positive agreement was 0%. Third-adjudicator resolution yielded 8/300 majority-classified safety-relevant errors (2.7%, 95% CI 1.4–5.2%). Safety estimates in open-ended cardiovascular LLM benchmarks were highly adjudicator-sensitive. Multi-rater reliability and transparent resolution procedures should accompany numerical safety estimates, which should not be interpreted as real-world clinical error prevalence.