Leveraging large language models to automate the identification of healthcare access barriers for veterans

To develop and evaluate an automated system for identifying health care barriers focusing on transportation issues in veterans’ clinical notes using large language models (LLMs) and assess the impact of different prompting strategies on classification performance and explanation consistency. We developed a hybrid system combining pattern matching for templated notes with LLM analysis for free-text notes. Using 2000 manually annotated clinical notes, we compared four prompting strategies (dual-role short, dual-role long, analysis-first, analysis-only) across Mistral-7B and Llama-3.1 models. We evaluated classification performance using standard metrics and assessed explanation consistency through embedding similarity analysis. The analysis-first strategy achieved superior performance, with Mistral-7B reaching F1 score of 0.914, outperforming traditional machine learning approaches (GBM: 0.786, BERT: 0.811). LLMs demonstrated higher explanation consistency within models (mean cosine similarity 0.887–0.908) compared to cross-model similarities (0.767–0.872). Pattern matching successfully handled 6.7% of templated notes deterministically. Mistral-7B showed greater internal consistency but higher abstention rates compared to Llama-3.1. Requiring LLMs to analyze evidence before classification improves both accuracy and explanation consistency for identifying transportation barriers in clinical notes. This approach enables automated barrier detection at scale while providing clinically relevant explanations, supporting both population-level healthcare planning and individual patient care decisions. • Analysis-first LLM prompting achieves F1 score of 0.914 for identifying transportation barriers in veteran clinical notes. • Hybrid system combines pattern matching for templated notes with LLMs for free-text analysis. • LLMs outperform traditional machine learning approaches while providing clinically relevant explanations. • Mistral-7B demonstrates higher explanation consistency than Llama-3.1 across different prompting strategies. • System successfully deployed in VA’s secure infrastructure, processing over 100,000 clinical notes.