What is an AI hallucination?

An AI hallucination is generated output that presents erroneous or false information as if it were reliable. NIST’s Generative AI Profile uses the term confabulation and also includes output that diverges from the prompt or contradicts earlier statements in the same context.

For a text draft, inspect both factual accuracy and fidelity to the supplied material. A made-up study is one problem; a summary that changes an existing policy is another. A polished explanation does not supply the evidence needed to support either.

Keep the claim separate from its presentation

NIST warns that generated logic or citations can appear to justify an answer while misleading a reader. An explanation of how an answer was reached is therefore something to inspect, rather than independent proof that it is correct.

Prioritize claims a reader may act on: a product’s availability, a stated deadline, a quoted person’s words or a comparison’s eligibility conditions. Record what the source actually supports before deciding whether the draft is ready.

Generation can produce plausible errors

NIST describes confabulation as a consequence of statistical generation. Language models predict tokens or words; such prediction can produce accurate, consistent output and can also produce factual errors or internal inconsistencies.

That explanation does not establish one error rate for every system or prompt. When reading an evaluation, identify the task, model version, input material, scoring rule and what counted as an error. Keep a summary benchmark separate from a product-policy draft or an open-ended question.

Look for the specific error

The following are editorial checks, not an exhaustive technical taxonomy. A single passage can fail more than one check.

Questions to ask about a generated claim
PatternWhat to inspectUseful evidence
Invented fact or citationDoes the named study, statement or source exist?The actual publication and relevant passage
Wrong attribution or scopeIs this about the right person, product, period and population?Source identity, version and stated limits
Context contradictionDid the draft change what the supplied material says?The exact input alongside the output
Unsupported inferenceDoes the conclusion follow from the cited facts?The premise and the reasoning, checked separately

A fictional product-policy review

Suppose a fictional shop’s supplied policy says: “Unopened items may be returned. Defective items may be exchanged.” A generated draft says: “Opened items can be returned,” and cites the policy page. The citation points to the right document, but the proposed sentence changes the eligibility condition.

In this hypothetical review, the editor compares the sentence with the passage, removes the unsupported return claim and keeps the distinction between a return and an exchange. No customer outcome or model test is reported.

The practical check is claim-level support. Opening a URL establishes that a page is accessible; it does not establish that the page says what the draft attributes to it. Keep the supporting passage with the corrected claim.

AI content detection concerns an estimate of content origin. Here the editorial question is whether the statement is supported. Do not treat a low detector score as a fact-check result or a high score as evidence that a particular claim is false.

A bias review asks a different question, such as whether a description unfairly stereotypes a group or omits relevant perspectives. An output can need both factual correction and a fairness review. Checking one does not complete the other.

Ground the draft, then verify its claims

Google’s grounding overview describes connecting output to verifiable information and says that access to specified sources reduces the chances of invented content. Its stated benefits include links that help audit the response.

Use that as a reason to inspect the supplied or retrieved material, not a guarantee that the final sentence is correct. Confirm the source is relevant and current for the claim, then check that the draft preserves its conditions. In the fictional example, the supplied source was useful but the output still needed correction.

NIST’s suggested action MS-2.5-003 calls for reviewing and verifying sources and citations in generated outputs during pre-deployment measurement and ongoing monitoring.

For an editorial review, save the draft and input, list the factual claims, attach their supporting passages, and mark unresolved claims for removal or further checking. Give a reviewer responsibility for the decision to publish; a request for the model to check itself is not your evidence record.

Correct the record without inventing a success rate

When a claim fails review, correct or remove it and check dependent statements, captions and structured data. If a published page needs correction, preserve enough context to explain what changed. Avoid replacing one unsupported statistic with another model-generated number.

Keep evaluation results tied to their tested conditions. A changed prompt, retrieved collection or model version may warrant a new check; none of the guidance here promises elimination of errors or a measured prevention percentage.

Generative AI covers the broader class of content-producing systems. AI content generation concerns the production workflow. This term focuses on a failure of factual or contextual support within the output.

Sources and scope

NIST supplies the confabulation definition, generation explanation and source-review action. Google supplies its grounding description. The product-policy review is fictional. This is an editorial and technical explanation, not medical, legal or other professional advice, and it makes no claim about a particular model’s measured accuracy.

Explore this topic: AI Hallucinations: Why They Happen and How to Stop Them.