Assessment of the Japanese national examination for pharmacists using locally fine tuned large language models

Locally deployable large language models (LLMs) offer advantages in data security and customization, making them particularly useful in domains that handle sensitive information. Although commercial cloud-based LLMs have shown strong performance on the Japanese National Examination for Pharmacists (JNEP), the effectiveness of locally deployable, open-weight models for such knowledge-based tasks has not been fully explored. Evaluating these models is important in contexts where data privacy and controlled model deployment are required. In this study, four locally deployable LLMs were evaluated: phi-4, DeepSeek R1 Distill Qwen (DSR1-Qwen32B), and two Japanese fine-tuned variants (CA-DSR1-32B and CA-DSR1-14B) based on DSR1-Qwen. Accuracies on the 165 text-based questions from the 109th JNEP ranged from 55.2% to 76.4%. Parameter-efficient fine-tuning using Low-Rank Adaptation (LoRA) improved the accuracy of phi-4 from 60.6% to 66.1%, suggesting that task-specific adaptation enhances performance. Although improvements varied across domains, the results indicate that fine-tuning can improve performance on examination-style questions requiring pharmaceutical knowledge. These results provide preliminary evidence that locally deployable LLMs, as well as resource-efficient adaptation techniques including LoRA, can be applied to structured, knowledge-based assessments such as the JNEP. Further evaluation across broader tasks is required, but locally deployable LLMs may offer a promising approach for knowledge-intensive tasks in pharmacy and related healthcare domains.