Identifying and classifying software mentions in full text scholarly documents

Software is central to modern science, yet references to it in scholarly articles are often incomplete or inconsistent, hindering reproducibility and reuse. Existing methods for mining software mentions, such as rule-based and conventional NLP approaches, remain limited in scalability and robustness. The emergence of large language models (LLMs) offers new opportunities for improving this task. LLMs exhibit strong contextual reasoning and adaptability, making them well suited to extracting software mentions from heterogeneous academic texts. In this paper, we evaluate several LLM-based approaches using three gold-standard corpora, comparing prompting strategies and configurations against established baselines. Our contributions are threefold: (1) we provide the first systematic evaluation of LLMs for software mention extraction, (2) we analyse their strengths and weaknesses relative to prior techniques, and (3) we discuss implications for reproducibility and open science. Results show that LLMs significantly improve extraction accuracy and adaptability, advancing efforts to integrate software into the scholarly record.