Answers are grounded in this paper and cite their sources.
Large language models (LLMs) are increasingly used for information retrieval across diverse domains, including religious knowledge. However, these models are prone to "hallucination"—generating confident but factually incorrect outputs. When such errors involve sacred texts like the Qur'an, the consequences extend beyond factual inaccuracy to potential theological harm, as users may unknowingly propagate fabricated scripture. We conducted a pilot study evaluating three open-source LLMs (qwen2.5:0.5b, llama3:latest, aya:8b) on Qur'an citation queries, testing both authentic verse retrieval and responses to "trap questions" requesting non-existent verses. Our findings reveal severe safety failures: all three models fabricated non-existent Qur'anic verses with high confidence (100% fabrication rate on trap questions, 5/5 tests), while only one model (33%) correctly retrieved a well-known authentic verse. Explicit cautious prompting had zero measurable effect on reducing fabrication. Multilingual evaluation on Bangla input resulted in catastrophic failures (0% success rate), including gibberish output and infinite repetition loops, contradicting aya:8b's claimed 101-language support. Paradoxically, the smallest model (494M parameters) outperformed larger models (8B parameters), suggesting training data quality matters more than parameter count for domain-specific accuracy. Our zero-cost methodology using local LLMs via Ollama demonstrates that systematic hallucination measurement is accessible without API costs. These findings establish that current open-source LLMs are unsafe for unverified Islamic knowledge queries and motivate immediate deployment of verification systems (retrieval-augmented generation, post-generation validation) in production AI systems handling religious content. We openly share our methodology, evaluation criteria, and pilot data to enable replication and extension to other religious traditions.
No takes yet. Share an insight, caveat, or question.
Najmun Nahar Hira (2026) studied this question.
Synapse has enriched 8 closely related papers on similar clinical questions. Consider them for comparative context: