Optimizing Entity Recognition in Psychiatric Treatment Data with Large Language Models

Extracting nuanced adverse drug reactions (ADRs) from patient self-reported messages using is pivotal but challenging, particularly given HIPAA constraints. We investigate locally deployable small LLMs-Mistral-7B, Llama-3-8B, and Gemma-7B-for ADR extraction using the PsyTAR dataset of self-reported messages. We implement in-context learning, demonstration selection, and fine-tuning with QLoRA. Results show Mistral-7B excels in few-shot settings, and Fine-tuning with F1 = 86%. These approaches safeguard data privacy and offer resource-efficient solutions for healthcare organizations. Our pipeline enables real-time ADR monitoring, helping clinicians address concerns more swiftly and enhance patient outcomes. Findings underscore that smaller LLMs can be effectively used under strict data privacy constraints, allowing healthcare teams to quickly identify and address patient-reported ADRs. Ultimately, these accessible solutions bolster patient safety.