Importance Delays in patient care and occupational burnout are potential consequences of difficulty in managing rising volumes of patient portal messages. The Smart Messaging Tool (SMT) was a custom-built natural language processing tool developed to address 4 key criteria: alignment with care pathways, adaptation to evolving conditions, demonstration of high accuracy, and accommodation of large volumes of messages. Objective To assess the viability of SMT in classifying patient messages in a large, integrated, value-based health care system. Design, Setting, and Participants This quality improvement study compared the first 2 years of SMT implementation (March 28, 2023, to March 27, 2025) with the legacy system it replaced. Patient messages sent via the Southern California Permanente Medical Group’s (SCPMG’s) patient portal were processed by SMT, which generated a label that populated the topic column in the receiving clinician’s inbox. Deployed broadly to primary care departments across SCPMG, SMT reached the inboxes of more than 1000 clinicians. Messages could come from any of SCPMG’s 4.9 million patients who opted to send secure messages. Main Outcomes and Measures Median time to first read by a clinician of high-acuity messages and SMT classification performance metrics were assessed. Results The SMT processed 3 030 247 messages sent by 1 042 418 unique patients (the majority 31.9% aged 30-49 years; 60.5% female). Median time-to-first read by a clinician for high-acuity messages fell from 22.03 hours (95% CI, 21.47-22.48 hours) to 5.02 hours (95% CI, 4.50-5.77 hours). The SMT achieved 81.0% (95% CI, 77.8%-83.6%) accuracy, which is higher than the 44.0% accuracy of the legacy system. The SMT achieved a top-3 accuracy of 88.5% (95% CI, 86.0%-90.8%) and a top-5 accuracy of 90.7% (95% CI, 88.5%-92.7%). Conclusions and Relevance This quality improvement study found that SMT reliably categorized patient messages to support improvements in the timely processing of high-acuity messages. The findings suggest that the SMT’s practical applicability underscores its relevance to organizations aiming to leverage natural language processing to address message management challenges.
Nguyen et al. (Thu,) studied this question.