Key result
LLMs achieve near-perfect agreement on CAD-RADS scoring but struggle with clinical management recommendations.
Why the study?
Do Large Language Models accurately assign CAD-RADS scores and provide patient management recommendations based on synthetic CCTA reports compared to a human reference standard?
Do Large Language Models accurately assign CAD-RADS scores and provide patient management recommendations based on synthetic CCTA reports compared to a human reference standard?
Answers are grounded in this paper and cite their sources.
While LLMs demonstrate reliable CAD-RADS scoring for lower-grade categories, they struggle with higher-grade stenosis and currently have limited ability to generate dependable clinical management recommendations.
PURPOSE: To evaluate the capability of four publicly available Large language models (LLMs) to assign Coronary Artery Disease-Reporting and Data System (CAD-RADS) scores and provide patient management recommendations based on synthetic coronary CT angiography (CCTA) reports. METHODS: Four LLMs (ChatGPT 4o, Claude 3.7, DeepSeek, and Gemini 2.5 Pro) were tasked with analyzing reports and suggesting next steps. Prompts were framed from the perspective of both a cardiologist and a radiologist. Agreement with a human reference standard was assessed using weighted Cohen's kappa, Fleiss' kappa, and Krippendorff's alpha for CAD-RADS scoring, and unweighted Cohen's kappa for management recommendations. A Bayesian Wilcoxon signed-rank test was performed to assess directional bias. RESULTS: Performance variations were observed across LLMs and prompt identities. Claude-3.7 achieved almost perfect agreement for CAD-RADS scoring (κ = 0.997) regardless of prompt identity, Gemini similarly achieved almost perfect agreement (radiologist: κ = 0.962; cardiologist: κ = 0.990), ChatGPT demonstrated almost perfect agreement when prompted as a radiologist (κ = 0.896) but only substantial agreement when prompted as a cardiologist (κ = 0.715). DeepSeek showed the lowest overall performance (radiologist: κ = 0.637; cardiologist: κ = 0.768). By category, all LLMs correctly identified CAD-RADS 0, whereas higher-grade stenosis (4A/4B) remained the most challenging, with non-Claude models showing low-to-null agreement in some configurations. The LLMs' accuracy in proposing further management was considerably lower than their scoring accuracy, with CAD-RADS 3 showing the greatest variability in management recommendations across models and between human specialists. Furthermore, both CAD-RADS scoring and management recommendations varied depending on the professional identity specified in the prompt. CONCLUSION: While LLMs demonstrated reliable scoring performance for lower-grade CAD-RADS categories (0-2), agreement was substantially reduced for higher-grade stenosis categories (4A/4B) and non-diagnostic studies, which could pose risks to patients. Their current ability to generate dependable clinical management recommendations is limited.
No takes yet. Share an insight, caveat, or question.
Tarkowski et al. (2026) studied Coronary Artery Disease. Large language models (ChatGPT 4o, Claude 3.7, DeepSeek, Gemini 2.5 Pro) vs. Human reference standard was evaluated on Agreement with a human reference standard for CAD-RADS scoring and management recommendations. Large language models like Claude-3.7 achieved almost perfect agreement for CAD-RADS scoring, but all models struggled with higher-grade stenosis and clinical management recommendations.
Synapse has enriched 10 related papers. Consider them for comparative context: