Bern Sleep-Wake Registry: temporal extraction of comorbidities from unstructured clinical text using a large language model

Introduction Large language models (LLMs) may enable the extraction of clinically relevant information from unstructured electronic health records, but most previous work focused on identifying diagnoses without preserving their temporal context. We evaluated the ability of an LLM to extract diagnoses and associated dates from longitudinal clinical reports. Methods In an observational study using the Bern Sleep-Wake Registry, diagnostic annotations from 248 patients comprising 2343 clinical entries were analyzed. Six common comorbidity groups, subdivided into 32 hierarchical diagnostic categories, were evaluated. Using gpt-oss-20b, we extracted diagnosis presence, the number of diagnostic instances, and dates per instance. A reference standard was established through independent review by two physicians, with disagreements adjudicated by a third physician. LLM performance was assessed using diagnostic classification and temporal agreement. Direct annotation costs were compared. Results At the patient level, the LLM achieved an overall F1 score of 0.88 for diagnosis presence and 0.96 for the presence of a reported date. For the earliest reported diagnosis date, agreement with the reference standard was 86.73% at the day level. When considering the cumulative set of dates for each diagnosis, date recall was 87.63%, the extra-date rate was 20.88%, and full agreement was achieved in 58.19% of cases. Human annotators showed substantial disagreement in temporal annotation, with a date recall of 75.49%, an extra-date rate of 17.89%, and full agreement in 58.64% of cases. LLM-based extraction reduced direct annotation costs by 94.65%. Conclusions LLMs can extract diagnoses and their dates from heterogeneous clinical text with high agreement with an adjudicated reference standard. These findings support the scalable use of LLMs for longitudinal clinical registries.