Three-arm comparative audit quantifying scientific citation fabrication across contemporary large language models (LLMs) in sports and exercise medicine, with a deterministic retrieval engine (EvidenX) as control arm. Twelve published systematic reviews (2019–2025) served as gold standards (940 references via Europe PMC). Five LLMs were tested in memory mode (web search/RAG disabled) across three conditions: unaided generation, deterministic retrieval, and LLM grounded on verified references. Pooled citation fabrication in unaided generation was 49. 1% (613 citations; 95% CI 45. 2–53. 1), ranging from 33. 9% (Grok-4. 3) to 84. 1% (Groq Llama-3. 3). A frozen verifier with deterministic title-fallback correction revealed that naïve identifier-only verification over-counts fabrication by 17. 5 percentage points — a systematic super-counting bias affecting prior identifier-based audits. Fabrication rose monotonically with topic niche-ness. An expanded replication (v2, 2026-06-13) on a disjoint set of 42 PICO questions across 9 subdomains and 6 models reproduced the pooled rate at 48. 0% (564/1176 citations; 95% CI 45. 1–50. 8) and refined the subdomain gradient from supplements/ergogenics (38. 2%) to concussion/neurology (55. 6%). Deterministic retrieval (EvidenX) produced 0% fabrication by design; grounding LLMs on verified references eliminated added fabrication across all models, though evidence utilization varied two-fold and independently of standalone accuracy. Blind external validation by three independent reviewers confirmed the primary fabrication endpoint (Fleiss' κ=0. 855; human–machine κ=0. 95). This preprint is a pilot / proof-of-concept, JMIR-structured. Full v2 tables and provenance in V2RESULTS. md.
Mauricio Beitia Kraemer (Thu,) studied this question.
Synapse has enriched 5 closely related papers on similar clinical questions. Consider them for comparative context: