Medical institutions produce large volumes of unstructured medical data that require the extraction of relevant medical information used for performing clinical studies and statistical analysis. This paper examines the application of large language models (LLMs) for named entity recognition (NER) in the medical domain, with a focus on their practical usefulness in clinical settings. We evaluate both prompt-based and fine-tuned approaches using the MACCROBAT2020 dataset, which includes clinical case reports annotated with biomedical entities. Furthermore, we extend our evaluation of the fine-tuning methodology to three biomedical NER datasets: QUAERO, NCBI, and E3C. The study compares the performance of several open-source LLMs against baseline models, using exact and relaxed F1 scores across multiple entity types. Fine-tuned LLMs achieved higher strict-match accuracy and produced more reliable structured outputs than prompt-based methods, GLiNER variants, and supervised BERT baselines, particularly for complex medical entities. However, BERT-based encoders remained substantially faster and competitive under relaxed matching. Their performance across multiple datasets, although uneven across languages and annotation schemas, together with efficient operation when quantized, indicates potential for clinical pilot studies. Our code is publicly available on GitHub (https://github.com/erikcalcina/llm-medical-ner) under the MIT license.
Calcina et al. (Sun,) studied this question.