What question did this study set out to answer?

The study aims to assess how reliably large language models reproduce decision-making guidelines in vital pulp therapy.

June 9, 2026Healthcare0 citationsOpen Access

How Reliably Do Large Language Models Reproduce Vital Pulp Therapy Guidelines? A Mixed-Effects Evaluation of Guideline-Concordance and Error Directionality

SUSine Güngör UsGazi University AMArzu Şahin MantıPsychiatric Association of Turkey AMArzu Kaya MumcuDumlupinar University

Key Points

The study aims to assess how reliably large language models reproduce decision-making guidelines in vital pulp therapy.
Twenty-six yes/no questions derived from evidence-based guidelines were analyzed.
Four LLMs (GPT-5, GPT-4o, DeepSeek-V3, Gemini 2.5 Flash) were tested under different prompting conditions.
Mixed-effects logistic regression was used to analyze repeated responses clustered by guideline questions.
High overall guideline-concordance accuracy was noted; Gemini excelled without prompts, while DeepSeek performed best with prompts.
Gemini demonstrated significantly higher odds of guideline-concordance than GPT-5 in non-prompted scenarios.
Error analysis revealed model-specific trends with Gemini showing low false positives but increased false negatives under prompts.

Abstract

Background: Large language models (LLMs) are increasingly consulted for clinical guidance, yet their reliability in protocol-sensitive domains remains insufficiently characterized. This study evaluated the ability of widely accessible LLMs to reproduce guideline-defined decision thresholds in vital pulp therapy (VPT), with emphasis on guideline-concordance accuracy, professional-role prompting, short-term response stability, and decision-level error directionality. Methods: Twenty-six binary yes/no questions were derived from an internationally recognized evidence-based guideline for VPT. Four LLMs—GPT-5, GPT-4o, DeepSeek-V3, and Gemini 2.5 Flash—were queried under non-prompted and professional-role-prompted conditions by two independent operators across three daily sessions over three consecutive days. Descriptive analyses were complemented by mixed-effects logistic regression in R to account for repeated responses clustered within guideline-derived questions. Results: Overall guideline-concordance accuracy was high across models. Gemini showed the highest observed accuracy under non-prompted conditions; DeepSeek showed the highest under prompted conditions. In the mixed-effects model, Gemini demonstrated significantly higher odds of guideline-concordant responses than GPT-5 under non-prompted conditions, whereas DeepSeek outperformed GPT-5 and GPT-4o under prompted conditions. The model × prompt interaction showed a trend toward significance but did not reach the conventional threshold. Day and within-day time point were not significantly associated with accuracy, supporting short-term response stability. Error-direction analysis revealed model-specific patterns: Gemini showed consistently low false-positive rates but increased false-negative responses under prompted conditions; DeepSeek showed reduced false-positive and no false-negative responses under prompted conditions. Conclusions: Average accuracy alone is insufficient to characterize the reliability of LLM-generated clinical guidance. Evaluation in protocol-sensitive domains should incorporate guideline-concordance, prompt responsiveness, short-term stability, and decision-level error directionality.

问 AI

Bookmark

View Full Paper