A comparative evaluation of ChatGPT and Gemini for stroke caregiver support: accuracy, safety, and response characteristics

OBJECTIVE: The use of large language models is increasing in healthcare. The aim of this study was to evaluate and compare the responses provided by ChatGPT (OpenAI) and Gemini (Google) to frequently asked questions about stroke and its treatment by caregivers, in terms of clarity, empathy/tone, actionability, information risk/safety, and accuracy. METHODS: This study used a qualitative content analysis to evaluate responses generated by large language models to stroke-related caregiving questions. Ten frequently asked questions were identified through a review of the literature on stroke caregiving. These questions were presented to the free versions of ChatGPT and Gemini, and responses from each model were recorded. The recorded responses were independently evaluated by 10 experts from neurology, physiatry, and physiotherapy using a structured Likert-scale instrument across five predefined categories: clarity, empathy/tone, actionability, information risk/safety, and accuracy. RESULTS: Generalized estimating equation analyses indicated that the overall trends in response scores did not differ significantly between ChatGPT and Gemini across all evaluated categories (p>0.05). In contrast, response score trends varied primarily in clarity and empathy/tone, with limited variation in actionability (p<0.05). Intraclass correlation coefficient values were below 0.50 for both ChatGPT and Gemini. CONCLUSION: Although both models generally performed well, question-level variations and low inter-rater reliability limit generalizability, and even small shortcomings in information risk may have important implications for stroke caregivers and, indirectly, for patients with stroke.