Identification of Risk Factors for Glaucoma Progression in Free-Text Clinical Notes using a Local Small Language Model

Objective: To evaluate the performance of a small language model (SLM) in identifying unstructured glaucoma risk factors in free-text clinical notes from the electronic health record (EHR), specifically medication non-adherence, visit non-adherence, and family history of glaucoma (FHoG). Design: Retrospective cross-sectional study. Participants: 1,242 subjects from the Bascom Palmer Ophthalmic Repository (BPOR). Methods: Clinical notes of glaucoma-related clinical encounters between 2014 and 2024 were extracted from BPOR. Two fellowship-trained glaucoma specialists labeled all notes for evidence of non-adherence and FHoG. In cases of disagreement, a third specialist served as an adjudicator. The structured EHR family history field was also extracted. We utilized MedGemma-27B-text-it, a specialized medical SLM, to identify medication non-adherence, visit non-adherence, and FHoG in clinical notes. A list of family members with glaucoma was also extracted by the SLM, which was subsequently converted to a numeric value. The model was run on a secure, local computer at the Bascom Palmer Eye Institute. Main Outcome Measures: Accuracy, sensitivity, specificity of SLM performance for each classification task. Jaccard index and mean squared error (MSE) of estimated number of family members with glaucoma. Results: The SLM analyzed all 1,242 notes with no mechanical error. The prevalence of medication and visit non-adherence documented in clinical notes were 10.7% and 6.7%, respectively. FHoG was noted in 28.7% of subjects. Compared with the reference standard, SLM accuracy was 94.7% in identifying medication non-adherence (sensitivity: 0.947; specificity: 0.947) and 96.4% accurate at identifying visit non-adherence (sensitivity: 0.940; specificity: 0.965). SLM accuracy in identifying FHoG was 98.4% (sensitivity: 0.984; specificity: 0.988) with Jaccard index of 0.988. In contrast, the structured EHR family history field was 49.2% accurate with Jaccard index of 0.756. The MSE of the SLM and the EHR field outputs in quantifying the number of relatives with glaucoma were 0.05 and 0.83, respectively (p<0.001). Conclusions: SLMs accurately evaluated clinical notes on a secure local network for all three extraction tasks. The structured EHR family history field was less accurate than the SLM. Implementation of an SLM pipeline in an ophthalmic institution can facilitate the identification of key risk factors for glaucoma progression in unstructured EHR data.