Building a Qur’an and Hadith research tool meant treating the general purpose LLM’s own memory as a liability, not a feature. And three production failures that taught me why every layer of the pipeline around it earns its keep.
Ask a general-purpose chat model to quote a hadith on a specific ruling, and it will usually hand you something that looks right: a plausible book name, a narrator chain, a quote in confident English. Ask it to check its own answer and it will often defend the fabrication just as confidently. That’s not a rare glitch, it’s the model doing exactly what it was trained to do, produce fluent, plausible continuations. The problem is that “plausible” and “real” are unrelated properties, and for a domain where the exact wording of a source is the evidence, a fluent misquote isn’t a stylistic slip. It’s fabricated primary-source material attributed to a real book and a real scholar.
That’s the premise behind Quran AI, a project I’ve spent the past several months building: a research tool over the Qur’an, classical tafsir, and four hadith collections, with one non-negotiable constraint threaded through every layer of it; the model that writes the final answer is never allowed to draw on what it remembers from training. It can only write from what was actually retrieved for this question, and nothing reaches the reader until that retrieval has been checked against the stored source text.
