Uncovering the Correlates of Hallucination in Large Language Models: A Regression Analysis Perspective

Large language models (LLMs) have become ubiquitous systems in research and society. Technical developments have made them more accurate in downstream tasks; and their use as a tool across domains such as education, medicine, or law has increased. However, they suffer from hallucinations. This phenomenon makes LLMs to produce nonfactual or unfaithful outputs to a given prompt, which translates to the inability to be fully trusted in sensitive domains. What variables correlate with LLMs' hallucinations? We study 15 variables across 30 LLMs using queries from 9 hallucination datasets. We find 7 variables to be associated with hallucinations. However, our most crucial result lies in uncovering that the phenomenon of hallucination is more complex than previously thought: Some of these variables do not only have systematic effects across LLMs but also differential effects across both LLMs and the input queries, as well as bivariate interactions, resulting in an intricate picture of how hallucinations are generated.