Learning the Phenotype of Medical Hallucinations

Abstract The clinical deployment of powerful general-purpose large language models (LLMs) is fundamentally limited by their propensity for unreliable generation ("hallucination"), posing a significant safety risk in high-stakes domains. Here, we introduce CHECK, a model-agnostic safety layer that learns a hallucination phenotype from first principles of information theory to proactively suppress unsafe outputs. Rather than relying on content-based heuristics, CHECK integrates structured, open clinical knowledge with an independent classifier that estimates hallucination probability from distributional signals, such as the uncertainty (entropy) of individual token predictions and the divergence (Kullback–Leibler divergence) of probability distributions across an ensemble of language models. In a high-stakes setting of physician question and answer (QA) about pivotal oncology trials, where misinformation can have severe consequences, CHECK reduced hallucinations in a state-of-the-art open model (Llama-3-70B) from a clinically untenable 31\% to just 0.3\%. This capability to discern hallucination generalized across diverse medical tasks (reasoning and education, patient and physician dialogue, radiology report summarization) and a suite of leading open-weight and commercial models (e.g., GPT-4o, DeepSeek-R1-Distill-Llama-70B, GPT-o3, GPT-5), achieving detection AUCs of 0.95–0.98. As a practical application, we demonstrate that the hallucination probability signal from CHECK can guide an iterative refinement process, improving GPT-4o’s USMLE pass rate by five percentage points to a new state-of-the-art of 92.1\% while significantly reducing compute. By systematically driving hallucination risk below accepted clinical error thresholds, CHECK provides a scalable and plug-and-play solution to enable the reliable deployment of LLMs in medicine and other mission-critical domains.