Integrating Natural Language Processing in Child and Adolescent Mental Health Services Research Using Normalization Process Theory: Implementation Report

BACKGROUND Anxiety problems are prevalent among autistic children and adolescents and those with attention-deficit/hyperactivity disorder (ADHD). Natural language processing (NLP) is increasingly used to structure information from electronic health records (EHRs), but its implementation within secure child and adolescent mental health services (CAMHS) research environments is less frequently reported. OBJECTIVE This study documented the specification, governance, integration, and evaluation of an NLP-enabled information extraction workflow within the Oxford Health National Health Service (NHS) Foundation Trust CAMHS research setting for extracting documented anxiety treatment information and outcome-related text from the EHRs of autistic young people and those with ADHD and examined implementation using normalization process theory (NPT). METHODS A multidisciplinary team developed a locally deployed NLP pipeline to identify labeled text spans describing presenting problems, medication attributes, psychological therapies, and treatment outcomes in pseudonymized EHR notes. Implementation followed an iterative seven-phase road map: (1) defining research questions, (2) identifying key constructs, (3) codeveloping an annotation schema, (4) training human annotators, (5) resolving annotation discrepancies, (6) large-scale annotation of EHR data, and (7) developing and evaluating the NLP model. NPT was applied throughout to guide implementation and evaluate progress. Quantitative outcomes (data volume processed, annotation reliability, and model performance) and qualitative feedback were collected to assess implementation success. RESULTS The source cohort comprised 454,127 notes from 5906 patients; 697 (0.2%) randomly selected, manually annotated notes from 302 (5.1%) patients were used for model development and held-out evaluation. Category-specific position-based Krippendorff α values ranged from 0.325 to 0.579 for problem, psychological therapy, and outcome spans and from 0.665 to 0.967 for medication spans. Agreement was highest for medication names, doses, and frequencies and lowest for cognitive problems and treatment outcomes. Held-out macroaveraged F1-score was 0.87 (95% CI 0.86-0.89) across the 4 medication labels and 0.71 (95% CI 0.68-0.73) across the 5 problem labels, psychological therapy labels, and outcome labels; label-specific F1-score ranged from 0.60 to 0.98. Team-generated implementation records indicated that multidisciplinary schema development and clear protocols supported shared understanding and engagement, whereas annotation burden, documentation variability, data access delays, and computing constraints impeded implementation. No major adverse events or harms were reported. CONCLUSIONS This report demonstrates the feasibility of integrating NLP tools into NHS CAMHS research workflows to extract structured information on anxiety treatments from routine clinical records. The NPT framework provided a structured approach for documenting the organizational work, barriers, and adaptations involved. External validation, refinement of less reliable labels, and further evaluation are required before broader use. INTERNATIONAL REGISTERED REPORT RR2-10.31234/osf.io/r4gyw