How do you reduce LLM hallucinations in production?

Key facts

• NIST's Generative AI Profile (NIST AI 600-1, July 2024) calls hallucination confabulation and describes it as a natural result of the way generative models are designed. (Source: NIST AI 600-1, Generative Artificial Intelligence Profile)
• Xu, Jain and Kankanhalli (2024) argue formally that hallucination cannot be eliminated in large language models used as general problem solvers. (Source: Xu et al., Hallucination is Inevitable (arXiv 2401.11817))
• Kalai et al. (September 2025) argue that language models hallucinate partly because training and evaluation reward guessing over acknowledging uncertainty. (Source: Kalai et al., Why Language Models Hallucinate (arXiv 2509.04664))
• In Moffatt v. Air Canada (14 February 2024), British Columbia's Civil Resolution Tribunal held Air Canada responsible for wrong bereavement-fare information given by its website chatbot. (Source: Moffatt v. Air Canada, 2024 BCCRT 149)
• Liu et al. (TACL 2023) found that language models use relevant information best when it sits at the beginning or end of a long input, and significantly worse when it sits in the middle. (Source: Liu et al., Lost in the Middle (arXiv 2307.03172))
• Chain-of-Verification (Dhuliawala et al., 2023) reduced hallucinations by having a model draft an answer, plan fact-checking questions, answer them independently and then revise. (Source: Dhuliawala et al., Chain-of-Verification (arXiv 2309.11495))
• The OWASP Top 10 for LLM Applications (2025) lists Misinformation, including hallucination, as LLM09 and recommends retrieval from trusted sources, automatic validation, human oversight and clear communication of limits. (Source: OWASP: LLM09:2025 Misinformation)
• From Model to Product grows a prompt template that forbids invented rules and guessed details, and notes that this does not eliminate hallucination completely but makes behaviour far more predictable. (Source: From Model to Product, Chapter 11)

Definitions

Hallucination

Output that is fluent and confidently stated but false, unsupported by the input, or inconsistent with what the model said earlier. NIST's term for it is confabulation.

Grounding

Supplying the model with the facts it must use, retrieved or decided by the application, and instructing it to answer only from them.

Abstention

A designed response in which the system declines to answer, or says it does not know, when the evidence is missing or confidence is low.

Structured output

A mode in which generation is constrained to a supplied schema, such as JSON with fixed fields and allowed values, so the application can parse and check every field.

Why LLMs hallucinate, and why you cannot eliminate it

A language model generates the most plausible continuation of its input, and plausible is not the same as true. When the input lacks the facts, the model fills the gap with something that sounds right. NIST's Generative AI Profile calls this confabulation and describes it as a natural result of how generative models are designed.

Research explains why it persists. Kalai et al. argue that training and evaluation reward guessing over admitting uncertainty, so models learn to guess, and Xu, Jain and Kankanhalli argue formally that hallucination cannot be eliminated in models used as general problem solvers.

The practical conclusion is to stop aiming for zero and work in layers: make hallucinations rarer, make them detectable, and limit the damage when one gets through.

Ground the model in facts it is given

The biggest single reduction comes from not asking the model to remember. Retrieve the facts or decide them in code, put them in the prompt, and instruct the model to answer only from them. A model explaining a fact it was handed has far less room to invent than one recalling from training.

Chapter 11 of From Model to Product builds this up as a prompt template that locks the decisions the backend has made and forbids invented rules and guessed details, and is candid that this makes behaviour far more predictable without eliminating hallucination. LLM APIs in production walks through that template and the checks around it.

Grounding also means scope. Tell the model what it is for and what to do with questions outside it, because off-topic questions are where it has least to go on.

Retrieval quality sets the ceiling

When retrieval misses the passage that holds the answer, the model fills the gap, so a hallucination in a RAG system is often a retrieval failure in disguise. Measure retrieval on its own before tuning prompts.

Send less, not more. Liu et al. found models use information in the middle of a long context markedly worse than at its start or end, so a few relevant passages beat many loosely related ones.

Four retrieval habits do most of the work; pgvector or a vector database covers storage, and RAG vs fine-tuning covers when retrieval is the right fix at all.

• Filter before ranking, by tenant, jurisdiction, product version or date, so an answer cannot be built from a document that does not apply.
• Combine keyword and semantic search and rerank the candidates, so exact terms such as product codes are not lost.
• Chunk along the document's own structure, keeping headings with their text, so each passage carries its context.
• Retire stale documents from the index, since a correct quote from an outdated policy is still a wrong answer.

Constrain what the model can say

Every field the model does not write as free text is a field it cannot embellish. Use structured output modes that constrain generation to a JSON schema, enumerations for anything with a fixed set of values, and identifiers chosen from a supplied list rather than names or numbers typed out.

Then check the values in code. A schema guarantees shape, not truth: the model can return a well-formed date that is wrong. Compare every decided value with what the backend knows, confirm each cited document identifier is in the retrieved set, and recompute any figure the model reports.

Dr. Dulan Dias's write-up of building a production-ready prompt engine shows structured output and layered guardrails working together in one system.

Make every claim checkable

Ask for citations, then verify them. Have the model attach a quote or passage identifier to each claim, check in code that each quote appears in the source, and drop or flag any claim without support. Anthropic's guidance on reducing hallucinations recommends the same pattern of retracting claims that lack a supporting quote.

For long-form answers, add a verification pass. Chain-of-Verification has the model draft, plan fact-checking questions, answer them independently of the draft and revise. A cheaper signal is to run the same prompt several times and compare: inconsistent answers can indicate hallucination.

Each check adds latency and cost, so keep model-based verification for outputs where a wrong claim matters, and use code checks everywhere else.

Let the system say it does not know

A model told it may answer only when the evidence supports it, and given a defined way to decline, invents less. Anthropic lists explicit permission to say I don't know as a basic technique, and Dulan's AICON Belfast talk on Tellme AI, the museum guide live at three national museums, reports that giving the model that way out cut invention sharply.

Do not leave abstention to the model alone. When retrieval returns nothing relevant, return a fixed, honest response without calling the model. When a classifier is unsure which item the user means, ask a clarifying question rather than guess.

Track abstention as a metric. A rising rate points to a gap in retrieval or coverage; a rate near zero on hard questions suggests the model is still guessing.

Match the safeguards to the stakes

The right level of protection depends on what a wrong answer costs, not only on how often the model is wrong. In Moffatt v. Air Canada (February 2024), British Columbia's Civil Resolution Tribunal found the airline responsible for its chatbot's wrong fare advice, saying it made no difference whether information came from a static page or a chatbot.

OWASP's LLM09:2025 Misinformation entry recommends the same mix as below: trusted retrieval, automatic validation, human oversight and clear communication of limits.

Monitor hallucinations in production

Hallucination rates move whenever anything around the model moves: a new model version, a prompt edit, new documents or a new kind of question. Monitor them like any other quality metric.

• Run a faithfulness judge on a sample of live answers against the context each one was given.
• Chart citation-check failures, abstentions, user corrections and hand-offs to people, and alert on sudden shifts.
• Re-run the evaluation set after every model, prompt or index change; how to evaluate an LLM application covers building one.
• Log the sources, prompt version and model version behind every answer, so a reported error can be traced to its cause.
• Add each confirmed hallucination to the evaluation set so the fix stays fixed.

Frequently asked questions

Sources

1. NIST AI 600-1, Artificial Intelligence Risk Management Framework: Generative Artificial Intelligence Profile (July 2024)
2. Xu, Jain and Kankanhalli, Hallucination is Inevitable: An Innate Limitation of Large Language Models (arXiv 2401.11817)
3. Kalai et al., Why Language Models Hallucinate (arXiv 2509.04664)
4. Liu et al., Lost in the Middle: How Language Models Use Long Contexts (arXiv 2307.03172)
5. Dhuliawala et al., Chain-of-Verification Reduces Hallucination in Large Language Models (arXiv 2309.11495)
6. Anthropic: Reduce hallucinations
7. OWASP Top 10 for LLM Applications: LLM09:2025 Misinformation
8. Civil Resolution Tribunal of British Columbia: Moffatt v. Air Canada, 2024 BCCRT 149