Answers are grounded in this paper and cite their sources.
Objective: Small language models (SLMs) suitable for local deployment may enable integration of generative AI into electronic health record (EHR) systems; however, their performance in non-English pharmaceutical domains remains unclear. Methods: Fifty-two open-source SLMs (≤20B parameters) were evaluated using 283 text-based questions from the 107th Japanese National License Examination for Pharmacists. Each model was tested in three independent runs. Performance was assessed using accuracy and response consistency metrics, including the Any Correct Rate, Full Correct Rate, and Consistency Gap. Results: Three models (GPT-OSS:20b, Qwen3:14b, and Qwen3:8b) achieved high accuracy comparable to frontier LLMs, whereas medical-specialized SLMs performed poorly. Consistency Gap showed no association with parameter size but demonstrated a nonlinear relationship with accuracy. Conclusions: Selected locally deployable SLMs achieved strong performance in a Japanese pharmaceutical benchmark, supporting their potential as components of EHR-compatible decisionsupport systems, provided that their reliability is further validated in clinical contexts.
No takes yet. Share an insight, caveat, or question.
Sato et al. (2026) studied this question.
Synapse has enriched 8 closely related papers on similar clinical questions. Consider them for comparative context: