Abstract B022: Evaluating multi-turn capability of large language models in cancer communication

Effective communication about breast and cervical cancers remains a persistent public health challenge, especially among Black African women, who often face culturally specific barriers including fear of stigma, religious beliefs, and mistrust of medical systems. Despite growing interest in using large language models (LLMs) for health communication, current evaluation methods fall short in three ways. First, they focus on single-turn accuracy, ignoring the multi-turn, back-and-forth nature of real patient conversations. Second, they fail to assess whether the AI can adapt to a patient's changing needs and emotional states across the dialogue. Third, they lack benchmarks or datasets that reflect the cultural and social contexts of underserved populations. Our work directly addresses these gaps by developing a framework for generating and evaluating culturally informed, multi-turn dialogues between patients and LLMs in cancer screening contexts.First, we develop a stage-based dialogue generation framework that produces realistic, culturally grounded patient–LLM conversations about breast and cervical cancer screening. The framework simulates both sides of the conversation by assigning roles to two separate large language models: a patient actor LLM, which generates responses based on culturally informed beliefs and personal barriers, and a provider actor LLM, which responds with clinically appropriate, empathetic guidance. Each conversation is aligned with the patient’s behavioral stage such as pre-contemplation, contemplation, preparation, or action, based on the Transtheoretical Model. At each stage, we use the Health Belief Model to guide the inclusion of beliefs such as perceived risk, perceived barriers, and cues to action. The 6-function model of medical communication structures the conversation flow (e.g., building relationships, information exchange, decision support). Using this approach, we generated a synthetic dataset of ∼2,000 multi-turn dialogues. Each dialogue includes culturally specific language, personal screening concerns, and localized medical advice tailored to the lived experiences of Black African women.Second, we present a multi-dimensional evaluation framework for assessing the quality of LLM-generated conversations in healthcare contexts. The framework evaluates five important dimensions: accuracy, completeness, context awareness, communication quality, and instruction following. Both quantitative scoring (GPT-4 based/model based automatic scoring) and qualitative assessments are employed to capture the richness and fidelity of simulated conversations. Based on one of our quantitative evaluations, the dataset demonstrates strong performance across key dialogue quality metrics, with average scores of 2.34 for Engagingness, 2.50 for Naturalness, 2.37 for Groundedness, and 2.36 for Coherence on a standardized 1–3 scale.This work lays the foundation for building trustworthy, equity-driven AI systems that can support personalized and effective healthcare communication for underserved populations. Citation Format: Agnik Saha, Victoria Churchill, Ugur Kursuncu, Muhammed Y. Idris. Evaluating multi-turn capability of large language models in cancer communication [abstract]. In: Proceedings of the 18th AACR Conference on the Science of Cancer Health Disparities; 2025 Sep 18-21; Baltimore, MD. Philadelphia (PA): AACR; Cancer Epidemiol Biomarkers Prev 2025;34(9 Suppl):Abstract nr B022.