Key result
ChatGPT responses to hand surgery queries had inferior readability compared to Google, with no significant difference in accuracy between sources.
Why the study?
ChatGPT has the potential to increase accessibility of health information, but the usability of online medical information for hand surgery topics and the influence of medical consensus needed assessment.
Does ChatGPT provide more usable medical information compared to Google for hand surgery topics?
Cross-Sectional
Does ChatGPT provide more usable medical information compared to Google for hand surgery topics?
ChatGPT provides less readable medical information compared to Google for hand surgery topics, highlighting a potential health literacy barrier for patients using AI tools.
Answers are grounded in this paper and cite their sources.
ChatGPT yields less readable hand surgery information than Google at similar accuracy; leaves open whether consensus or prompting improves patient accessibility.
Background: ChatGPT is a natural language processing tool with potential to increase accessibility of health information. This study aimed to: (1) assess usability of online medical information for hand surgery topics; and (2) evaluate the influence of medical consensus. Methods: Three phrases were posed 20 times each to Google, ChatGPT-3.5, and ChatGPT–4.0: “What is the cause of carpal tunnel syndrome?” (high consensus), “What is the cause of tennis elbow?” (moderate consensus), and “Platelet-rich plasma for thumb arthritis?” (low consensus). Readability was assessed by grade level while reliability and accuracy were scored based on predetermined rubrics. Scores were compared via Mann–Whitney U tests with alpha set to .05. Results: Google responses had superior readability for moderate-high consensus topics ( P < .0001) with an average eighth-grade reading level compared to college sophomore level for ChatGPT. Low consensus topics had poor readability throughout. ChatGPT-4 responses had similar reliability but significantly inferior readability to ChatGPT-3.5 for low medical consensus topics ( P < .01). There was no significant difference in accuracy between sources. ChatGPT-4 and Google had differing coverage of cause of disease ( P < .05) and procedure details/efficacy/alternatives ( P < .05) with similar coverage of anatomy and pathophysiology. Conclusions: Compared to Google, ChatGPT does not provide readable responses when providing reliable medical information. While patients can modulate ChatGPT readability with prompt engineering, this requires insight into their health literacy and is an additional barrier to accessing medical information. Medical consensus influences usability of online medical information for both Google and ChatGPT. Providers should remain aware of ChatGPT limitations in distributing medical information.
No takes yet. Share an insight, caveat, or question.
Lavoie‐Gagne et al. (2025) conducted a cross-sectional in Hand surgery topics. ChatGPT (3.5 and 4.0) vs. Google was evaluated on Readability (grade level) (p=<.0001). ChatGPT responses to hand surgery queries had inferior readability compared to Google, with no significant difference in accuracy between sources.
Synapse has enriched 10 related papers. Consider them for comparative context: