Agreement Between a Large Language Model and the Revised Geneva Score Across Synthetic Patient Profiles: A Computational Concordance Study

BACKGROUND: This study examined how a frontier large language model (LLM) evaluating patient profiles at risk of pulmonary embolism categorizes risk compared with the revised Geneva score. The study also asked whether the LLM, given the score's nine components but not the score's name, would assign each component its published point value. METHODS: Claude Opus 4.8 (Anthropic, San Francisco, CA, USA) was compared with the revised Geneva score. All 384 possible patient profiles were enumerated. The LLM was asked to weight the score's components, then classify every profile as low, intermediate, or high probability. The revised Geneva score was never named. Risk classification ran twice under each of two wordings: all findings named present or absent, or only present findings listed. Responses were also examined for the score named and the total computed. RESULTS: The model gave every component the same point value as the revised Geneva score. Risk classification nonetheless diverged, and divergence tracked prompt wording. Agreement with the score averaged 41.7% under present-and-absent prompting and 62.6% under present-only. Of the 217 profiles the revised Geneva score rated high risk, the present-and-absent prompt called 24.2% low risk, versus 1.4% for the present-only prompt. When the LLM gave its reasoning for risk assignment, it named a Geneva score 91% of the time. Among responses that showed reasoning, named a Geneva score but not Wells, and stated a total, the total matched the revised Geneva score in 104/197 (52.8%) under present-and-absent and 89/97 (91.8%) under present-only. Most classifications had no assessable total. CONCLUSIONS: This study assessed agreement with the revised Geneva score in simulated patients; diagnostic accuracy and patient outcomes were not assessed. Claude Opus 4.8 reproduced the score's point values exactly, yet its risk classifications often disagreed with the score and varied depending on how the patient was described in the prompt.