Utilizing large language models and natural language processing to classify ischemia status from cardiac stress tests in a large multicenter healthcare system

Objective Documentation of myocardial ischemia prior to invasive coronary angiography is recommended to minimize patient risk. However, obtaining this information for quality-of-care assessment often requires extracting clinical information from unstructured electronic medical records text. To this end, we sought to evaluate multiple natural language processing (NLP) systems in their ability to classify cardiac stress test reports as documenting ischemia or no ischemia, implementing the one with the best combination of accuracy and feasibility. Results Four BERT large language models (LLMs) were fine-tuned, and a rules-based system was designed by training, validating, and testing on an annotated sample of 654 stress test reports from a multisite and multiyear dataset from the Veterans Health Administration (VHA). The LLM with the highest performance was a ClinicalBERT with precision, recall, and F1 of 86.4%, 100%, and 92.7%, respectively. The rules-based NLP system achieved similar results of 88.1%, 97.4%, and 92.5%, respectively. Stress test reports totaling 1,692,171 and representing 1,096,341 unique patients were classified using the rules-based system after ascertaining current technological limitations, and the system is presently operational for care quality evaluations. Utilizing NLP allows for accurate, high-throughput analysis of cardiac stress test text reports.