Towards AI-Based Evaluation of Data Management Plans Compliance with Funder Requirements

Data management plans (DMPs) are essential documents outlining how research data will be collected, preserved, and shared. Compliance with funder guidelines is often assessed by checking free-text DMPs against a structured DMP template—a time-consuming and error-prone process. This paper lays the groundwork for automated evaluation of free-text DMPs by investigating methods for locating answers within DMPs. We employ Large Language Models (LLMs), including the closed-source GPT-4o and open-weight models such as Llama and RoBERTa, to perform extractive question answering, mapping each DMP template question to the relevant text in a DMP. We evaluate different inference settings and experiment with fine-tuning models for this specific task. Our experiments show that the most cost-effective and accurate approach is to use a general-purpose LLM accessed through a per-token billed API with structured JSON output. Fine-tuned versions of smaller models, such as Llama 3.1 3B, provide a self-hosted alternative suitable for privacy-sensitive scenarios, though they do not match the performance of larger models. We additionally present four publicly available datasets compiled in this study: (1) a human-annotated dataset of answers to questions in real DMPs, (2) a question-and-answer dataset gathered from DMPs, (3) a collection of free-text DMPs, and (4) a dataset with questions-and-answer and synthetic context.