Structuring medication safety narratives: development and evaluation of the medication-related incident reports annotation scheme

Introduction: Narrative reports of medication-related incidents contain valuable information about the causes and consequences of errors, but their unstructured format limits systematic analysis. Although natural language processing (NLP) can convert narrative reports into structured data, few annotation schemes have been developed specifically for medication safety and validated using real-world healthcare incident data. This study aimed to develop and evaluate the Medication-Related Incident Report Annotation (MRIRA) scheme, a multi-layer framework designed to structure narrative medication safety reports to support both qualitative analysis and automated text processing. Methods: Using narrative incident reports from the English National Health Service (NHS), a two-phase study design was implemented. In Phase 1, a purposive sample of 55 Controlled Drug incident reports was manually annotated to iteratively design the MRIRA scheme. The framework incorporated multiple annotation layers, including entities, events, attributes, and relations. The final scheme comprised 16 entity types, 11 event types, 5 attributes, 9 relation types, and 6 event argument roles. In Phase 2, two annotators independently applied the scheme to 30 incident reports, including 15 Controlled Drug reports and 15 reports from the National Reporting and Learning System/Learn from Patient Safety Events (NRLS/LFPSE). Inter-annotator agreement was evaluated using F1 scores under both strict and relaxed matching criteria. Results: Under strict evaluation, agreement was high for entity recognition (F1 = 0.85 and 0.91 across the two datasets) and entity-relation extraction (0.75 and 0.83). Agreement was moderate for event extraction (0.62 and 0.72) and acceptable for event attribute tagging (0.61 and 0.51). All metrics improved under relaxed matching criteria, indicating greater consistency when allowing minor boundary variation between annotations. Discussion: The MRIRA scheme provides a robust and reliable framework for structuring narrative medication safety reports. By enabling systematic extraction of entities, events, and contextual relationships from incident narratives, the scheme offers a high-quality annotated resource that can support the development of automated NLP tools and enhance organisational learning from medication-related incidents in healthcare systems.