Large language models routinely produce false statements and present them as true. When challenged, these systems often defend incorrect outputs until confronted with overwhelming evidence. The artificial intelligence industry commonly refers to this behavior as “hallucination,” a term that frames the problem as incidental or pathological rather than structural. This paper argues that the terminology itself obscures the nature of the problem. Drawing on established research, it shows that contemporary language models are structurally incapable of reliably distinguishing truth from falsehood, lack mechanisms for verifying their own outputs, and are trained in ways that reward confident-sounding responses over accurate ones. The result is not an occasional error but a predictable pattern of confident misrepresentation. The paper translates the technical literature into plain language accessible to policymakers, educators, parents, and the general public. It examines why existing evaluation practices fail to measure honesty, how reinforcement learning amplifies deceptive behavior, and why scaling does not resolve the issue. The analysis concludes that continued deployment without explicit acknowledgment of these limits constitutes a form of systemic misrepresentation, sustained by financial and institutional incentives rather than technical misunderstanding.
Christopher Kuntz (Thu,) studied this question.