Large language models (LLMs) are increasingly used to assist with academic writing, yet their tendency to fabricate citations raises uncertainty about the accuracy of their interpretation of cited literature. We investigated citation-claim alignment by evaluating the accuracy with which claims from LLM-generated literature reviews correspond to the content of the scientific sources they cite, using mental health research as a case example. ChatGPT-5 was tasked with generating 12 literature reviews evaluating the evidence for three treatment modalities (psychotherapy, pharmacotherapy, and digital health) across four psychiatric disorders (major depressive disorder, generalized anxiety disorder, binge-eating disorder, and schizophrenia), each differing in the maturity of the underlying evidence base. Each citation-claim (N = 333) was extracted and independently evaluated against the cited source using a predefined accuracy scoring framework. Overall, 19.8% (n = 66) of citation-claims reflected fabrications, 18.3% (n = 61) contained major errors, 18.3% (n = 61) contained minor errors, and 43.5% (n = 145) were accurate. Despite the absence of clear disorder- or treatment-specific trends, error rates varied across individual reviews, with the proportion of citation-claims judged as fabricated ranging from 4 to 37% and those containing major inaccuracies ranging from 10 to 30% across the 12 study conditions. LLM-generated literature reviews on mental health topics are frequently undermined by serious citation-claim errors, demonstrating that they are not in a position to reliably replace human expertise in scholarly writing and interpretation.
Linardon et al. (Mon,) studied this question.