What question did this study set out to answer?

This study evaluates the performance and reliability of Large Language Models (LLMs) in answering patient inquiries based on radiology reports.

May 29, 2026Open Access

Patient-Facing Radiology Communication with LLMs: Calibration Deficit and the Metadata Paradox

Key Points

This study evaluates the performance and reliability of Large Language Models (LLMs) in answering patient inquiries based on radiology reports.
Analyzed 2000 simulated patient inquiries derived from 200 MIMIC-IV radiology reports.
Assessed responses from three LLMs (GPT-4o mini, Grok, Claude 3.5 Sonnet) using a scoring system validated by radiologists.
Evaluated performance calibration through Generalized Estimating Equations.
Overall agreement rate of the LLM judge with ground truth was 90.5%.
GPT-4o mini had a 2.8-fold higher risk of failure compared to Grok (adjusted OR 2.83; 95% CI: 2.28–3.49; p < 0.001).
Significantly higher errors in interpretive tasks were found compared to factual tasks, with a calibration deficit noted in safety-critical errors.

Abstract

Background/Objectives: Patients increasingly access radiology reports via online portals and frequently seek clarification. While Large Language Models (LLMs) may facilitate this communication, their clinical safety and reliability in this context remain largely uncharacterized. This study aimed to evaluate performance heterogeneity (the disparity between factual synthesis and interpretive reasoning), the Metadata Paradox (performance degradation triggered by demographic priors), and calibration characteristics in answering simulated patient questions derived from radiology reports. Methods: In this retrospective study, 2000 simulated inquiries were generated from 200 MIMIC-IV radiology reports based on an expert-refined 10-category taxonomy, categorized into factual tasks (e.g., terminology/anatomy) and interpretive tasks (e.g., diagnostic confidence/finding detail). Three LLMs (GPT-4o mini, Grok (v4-0709), Claude 3.5 Sonnet) generated 12,000 answers (with/without metadata). Quality was scored (1–3 scale) by Gemini 2.5 Flash, validated by three independent board-certified radiologists and finalized through four-specialist consensus adjudication (n = 1200). Performance and self-confidence calibration were assessed using Generalized Estimating Equations. Results: The LLM judge showed an overall agreement rate of 90.5% with the adjudicated ground truth. Grok and Claude 3.5 Sonnet significantly outperformed GPT-4o mini (p < 0.001); specifically, GPT-4o mini was associated with a 2.8-fold higher risk of failure compared to Grok (adjusted OR 2.83; 95% CI: 2.28–3.49; p < 0.001) and an absolute risk difference (ARD) of 8.4 percentage points. Accuracy reached its ceiling in factual tasks (Terminology: 98.1%) but was significantly lower in interpretive tasks (Diagnostic Confidence: 82.3%, p < 0.001). Metadata inclusion triggered the ‘Metadata Paradox,’ significantly increasing the risk of failure (OR 1.11; p = 0.044). A substantial calibration deficit (defined as the disconnect between self-confidence and accuracy) was observed; notably, the majority of safety-critical errors (Score 1: clinically significant misinformation; n = 131) were assigned high self-confidence (8/10; GPT-4o mini: 93.8%, Grok: 100%, Claude 3.5 Sonnet: 61.5%). Conclusions: Although LLMs accurately address factual queries, their consistent calibration deficit in safety-critical errors and susceptibility to stochastic stereotyping highlight the necessity of independent verification frameworks.

Read Full Paperexternally

Mark Helpful

Bookmark

Relay

View Full Paper