Clarity Without Credibility? Human Versus AI Abstracts in Otolaryngology

ABSTRACT Objective This study evaluated whether otolaryngologists can distinguish between human‐ and machine‐written abstracts. The primary question was whether large language models (LLMs) produce abstracts comparable in clarity and usefulness to human‐authored work, and whether reviewers can identify authorship with accuracy. Methods A blinded cross‐sectional design was used. Forty‐eight abstracts were evaluated, consisting of twenty‐four human‐authored abstracts and 24 generated by four LLMs. Human abstracts were drawn from articles published after July 2025 to minimize overlap with LLM training data. Twenty otolaryngologists independently reviewed all abstracts. Using a structured rubric, raters classified authorship, rated clarity, usefulness, and confidence on 5‐point scales, and provided optional free‐text explanations. Group comparisons were performed using chi‐square and Mann–Whitney tests, with Kruskal–Wallis tests for model‐level analyses. Results Overall recognition accuracy was 44.7%. Human‐written abstracts were more often misclassified as AI than AI‐generated abstracts were mistaken for human. Human abstracts received significantly higher clarity and usefulness scores than LLM abstracts, though effect sizes were small. Confidence did not correlate with correctness, indicating miscalibration of rater judgments. Model‐level performance varied. Grok‐generated abstracts were most easily identified as AI, whereas GPT‐5 and Claude 3.5 more frequently resembled human writing. Free‐text rationales commonly referenced style, vagueness, or lack of detail when AI authorship was suspected. Conclusion LLMs generate abstracts that increasingly resemble human scientific writing, yet still lag in perceived usefulness and credibility. Clinicians were only moderately successful at detecting authorship and were frequently confident in incorrect classifications. These findings highlight both the promise and risks of AI‐assisted scientific communication.