Evaluation framework for conversational AI agents in pharmacy education: A scoping review of key characteristics and outcome measures

BACKGROUND: Innovative and scalable training solutions, such as conversational artificial intelligence agents (CAIAs), are essential to meet the complex demands of pharmacy education and practice. OBJECTIVES: To explore and synthesize the key characteristics and outcome measures of CAIAs in pharmacy education and propose an evaluation framework for their use in pharmacy education. METHODS: A scoping review identified studies published between 2020 and 2025. An evaluation framework was developed to capture key CAIA characteristics and outcome measures using the World Health Organization's digital health framework. RESULTS: Of 961 studies screened, six met inclusion criteria. Five studies originated from English-speaking countries. CAIAs supported training in communication skills (n = 3), human resource management (n = 2), and HIV care (n = 1). Learner interaction with CAIAs was variable and often poorly described. Common features included scenario-based learning, immediate, real-time, automated feedback, interactive-learning, and modalities. Frequently evaluated outcomes were functionality (n = 5), user experience (n = 4), cost-benefit (n = 3), and user characteristics (n = 3). Educational outcome measures of confidence, knowledge and skills were included in three studies. All studies used text-based interaction; two included audiovisual elements, one combined text and voice, and two relied solely on text. Most (n = 5) involved single-user formats. Three studies were in evaluation stage one (feasibility/usability), two were in stage two (effectiveness) and one in stage three (efficacy). Outcomes demonstrated a low uptake for CAIAs, but indicate increased learner confidence, knowledge and communication skills. Eleven educational features were added to our evaluation framework and three educational outcome categories. CONCLUSION: This review contributed to the development of a framework to guide the design and evaluation of CAIAs in pharmacy education. CAIAs have been introduced in pharmacy education but remain in the early stages of adoption. Further research is needed to validate their effectiveness and expand their use in pharmacy as well test the use of the framework in other healthcare disciplines.