Information extraction (IE) from specialized clinical texts such as brain MRI reports is important for various clinical and population health contexts. However, this topic is under-explored due to privacy concerns limiting data availability and the inherent complexity and domain-specificity of clinical language. Common methods relying on substantial amounts of training data fail. The recent advances in large language model (LLM) research provide a promising solution to bridge the data scarcity gap, with improved ability to adapt to novel tasks with little supervision. We introduce a new, challenging dataset of 100 expert-annotated brain MRI reports, featuring 152 fine-grained entity types and 4 relation types, characterised by low inter-annotator agreement. This task reflects the inherent complexity and real-world ambiguity of medical text. We evaluate a small, open-weight LLM across span detection, named entity recognition, and relation extraction tasks. We compare few-shot prompting and parameter-efficient fine-tuning against specialized off-the-shelf biomedical IE systems. Our results demonstrate that both few-shot and fine-tuned LLM approaches substantially outperform off-the-shelf baselines. While LLMs show superiority, absolute performance, particularly for complex relations and fine-grained entities, remains modest, correlating with the dataset's inherent difficulty and the extreme low-resource setting.
Truong et al. (Tue,) studied this question.