Large language models, specifically GPT-4o and Llama3 70b, accurately generated full CAD-RADS 2.0 scores from cardiac CT reports with performance rates of 93% and 92.5%, respectively.
Observational (n=200)
Do large language models accurately generate CAD-RADS 2.0 scores from cardiac CT reports compared to expert radiologists?
Advanced large language models like GPT-4o and Llama3 70b can autonomously generate CAD-RADS 2.0 scores from cardiac CT reports with high accuracy, potentially improving reporting efficiency.
BACKGROUND: The Coronary Artery Disease-Reporting and Data System (CAD-RADS) 2.0 offers standardized guidelines for interpreting coronary artery disease in cardiac CT. Accurate and consistent CAD-RADS 2.0 scoring is crucial for comprehensive disease characterization and clinical decision-making. This study investigates the capability of large language models (LLMs) to autonomously generate CAD-RADS 2.0 scores from cardiac CT reports. METHODS: A dataset of cardiac CT reports was created to evaluate the performance of several state-of-the-art LLMs in generating CAD-RADS 2.0 scores via in-context learning. The tested models comprised GPT-3.5, GPT-4o, Mistral 7b, Mixtral 8 × 7b, Llama3 8b, Llama3 8b with a 64k context length, and Llama3 70b. The generated scores from each model were compared to the ground truth, which was provided by two board-certified cardiothoracic radiologists in consensus based on the reports. RESULTS: The final set comprised 200 cardiac CT reports. GPT-4o and Llama3 70b achieved the highest accuracy in generating full CAD-RADS 2.0 scores including all modifiers with a performance rate of 93 % and 92.5 %, respectively, followed by Mixtral 8 × 7b with 78 %. In contrast, older LLMs, such as Mistral 7b and GPT-3.5 performed poorly (16 %) and Llama3 8b demonstrated intermediate results with an accuracy of 41.5 %. CONCLUSION: LLMs enhanced with in-context learning are capable of autonomously generating CAD-RADS 2.0 scores for cardiac CT reports with excellent accuracy, potentially enhancing both the efficiency and consistency of cardiac CT reporting. Open-source models not only deliver competitive accuracy but also present the benefit of local hosting, mitigating concerns around data security.
Arnold et al. (Wed,) conducted a observational in Coronary artery disease (n=200). Large language models (GPT-4o, Llama3 70b, etc.) vs. Ground truth by board-certified cardiothoracic radiologists was evaluated on Accuracy in generating full CAD-RADS 2.0 scores including all modifiers. Large language models, specifically GPT-4o and Llama3 70b, accurately generated full CAD-RADS 2.0 scores from cardiac CT reports with performance rates of 93% and 92.5%, respectively.