What question did this study set out to answer?

This work examines why large language models often give misleading information, attributing it to structural rather than intentional factors.

May 11, 2026Open Access

Plausibility, persuasion, and truth: why language models may appear designed to deceive

Key Points

This work examines why large language models often give misleading information, attributing it to structural rather than intentional factors.
Identified four structural features of large language models: optimization for plausibility, post-training incentives for persuasive responses, structural hallucination, and source bias.
Analyzed cognitive biases such as automation bias and fluency-based truth effects that affect user perception of information accuracy.
Proposed a framework for epistemic auditing to differentiate types of information distortion.
Highlighting that persuasive responses from language models can lead to decreased factual accuracy.
Suggesting that the governance focus should shift from inferring intent to identifying mechanisms of epistemic distortion.
Proposing a clearer framework for oversight on factual errors and biases in language model outputs.

Abstract

Large language models can produce fluent, coherent, and persuasive responses even when the information on which they rely is partial, contested, or false. In disputed domains, this may leave some users with the impression that they are being deliberately misled. This Comment argues that the phenomenon is better understood in structural than intentional terms. It results from the convergence of four features of current systems: optimization for plausibility rather than truth, post-training incentives that reward helpful and persuasive answers, structural hallucination, and source bias rooted in asymmetries of knowledge production and digitization. These structural tendencies are further reinforced at the point of uptake by cognitive vulnerabilities such as automation bias and fluency-based truth effects. Recent evidence on conversational persuasion suggests that gains in persuasive force may come at the expense of factual accuracy. The governance problem, then, is not primarily to infer intent, but to identify the mechanisms through which epistemic distortion is produced. This Comment therefore proposes a minimal framework for epistemic auditing that distinguishes factual error, systematic omission, corpus bias, and post-training- or prompt-induced distortion, with a view to more discriminating oversight and clearer lines of responsibility.

AI에게 질문

Bookmark

View Full Paper