PolyTruth: Measuring Multilingual Hallucination Across Indian Languages

Language models exhibit dramatically different hallucination behaviors across languages, yet systematic measurement of multilingual hallucination remains scarce. We present PolyTruth, a research-grade multilingual hallucination benchmark spanning five low-resource languages—Hindi, Marathi, Tamil, Bengali, and Swahili—across 10 knowledge categories with adversarial coverage. We benchmark five 7B–9B parameter models, including two independent reasoning models (DeepSeekR1-Distill-Qwen-7B and DeepSeek-R1-DistillLlama-8B) alongside three non-reasoning models (Qwen 2.5 7B, Llama 3.1 8B, Gemma 2 9B), on 500 fact-checked questions per model, producing 2,500 total evaluation records. We additionally (i) validate our automatic correct/hallucinated/refused/incorrect classifier against a 300-item human-annotated sample (Fleiss-style pooled κ = 0.81, classifier accuracy 88.3% against adjudicated human labels), (ii) evaluate held-out unfiltered and paraphrased control items to test whether adversarial effects survive outside the items used to filter against our own development models, and (iii) diagnose and correct a bootstrap significancetesting artifact rather than reporting it as a finding. Our principal, human-validation-adjusted results are: (1) both reasoning models hallucinate at 9–10% versus 47–65% for nonreasoning models after correcting for a measured classifier boundary error—still an orderof-magnitude, non-overlapping-CI separation, though smaller than the uncorrected 0.6–1.2% vs. 52–71% gap our automatic classifier alone would suggest—while achieving only 24–27% accuracy, indicating reasoning improves hallucination avoidance far more than factual recall. (2) Models fall into three distinct failure profiles: hallucinators that fabricate when uncertain, a refuser that declines to answer, and attempters that produce non-hallucinated incorrect responses. (3) Hindi is the hardest language across all models (13.80% pooled accuracy) while Bengali is easiest (31.80%); both held-out and paraphrase controls reproduce this ordering, supporting that it is not an artifact of item selection. (4) Temporal ambiguity remains the attack type most associated with elevated hallucination, though with five models pooled its accuracy deficit is no longer cleanly separable from sparse-web-fact items at the per-cell sample sizes available (n = 21–105 per cell), and we report confidence intervals throughout rather than point estimates alone. (5) A tworeasoning-model, model-as-unit comparison (n = 2 vs. n = 3, d = 7.09, 3 df) supports a class-level reasoning/non-reasoning separation, but we flag the very low degrees of freedom and avoid the pseudoreplicated item-pooled test that would otherwise overstate certainty. We release PolyTruth, our human-annotation protocol, and our corrected evaluation code to support reproducible multilingual hallucination research.