A Comparative Analysis of Three Large Language Models in Answering Patient Queries on Otolaryngology Emergencies

Objectives: The use of large language models (LLMs) for advice by the public is rising. We aimed to compare the efficacy and safety of three LLMs in addressing otolaryngology emergencies from a patient-centred perspective as well as a clinician-oriented diagnostic standpoint. Methods: Thirty-two patient-framed questions were presented to Gemini, ChatGPT 3.5 and 4.0. Main outcome measures: Accuracy, comprehensiveness, empathy, and structure were scored on a 0–2 scale. Other metrics included word count, response time, Flesch-Kincaid readability scores, referencing and safety netting abilities. Results: Each model exhibited distinct strengths. ChatGPT-3.5 produced the longest (mean 366 words, p = 1.2 × 10 -5) but less empathetic (p = 0.0026) responses. ChatGPT-4.0 was the most comprehensive (p = 0.00019) but took the longest time (p = 2.6 × 10 -9). Gemini was the most empathetic but least comprehensive. All models ensured safety-netting, with similar answer structuring (p = 0.066). Conclusion: LLMs have significant potential to bridge gaps in patient education and support, particularly ChatGPT-4.0 which stands out for its detailed, structured responses. However, challenges such as high readability levels, occasional inaccuracies, and a lack of real-time adaptability must be addressed to meet the needs of diverse patient populations.