Can I trust what AI tells me?

What a hallucination is

A hallucination is when an AI assistant gives an answer that sounds right but is wrong: an invented figure, a report that does not exist, a misread clause. The US National Institute of Standards and Technology (NIST) uses the word confabulation and describes it as confidently stated but erroneous or false content. One AI developer says the same: Anthropic's own documentation notes that even advanced models such as Claude can sometimes generate text that is factually incorrect or inconsistent with the given context.

Which decisions in your organisation would you want AI to support, and who should sign them off? Book a free AI Value Briefing and we will look at it with you.

Why it happens

An AI assistant writes by predicting the next likely word, based on patterns in the text it learned from. NIST explains that confabulation is a natural result of this design: the same prediction that often produces correct answers can also produce wrong or inconsistent ones. NIST notes the risk is higher for open-ended, long answers and for specialist subjects.

The real danger is the tone. Wrong answers look just as polished as right ones, and NIST points out that people may act on false content because the response sounds so confident.

Ground the answer in your own documents

You get more reliable answers when the AI works from material you trust rather than from its general memory. Vendor guidance suggests practical steps that anyone can use:

• Give the AI the approved document (a policy, contract or report) and ask it to answer only from that document.
• Ask it to quote the passages it relies on, then open the document and confirm the quotes.
• Tell it plainly that "I don't know" is an acceptable answer.

Anthropic's documentation is clear that such techniques reduce hallucinations but do not eliminate them, and advises validating critical information. Connecting AI to your own systems is covered in Field Guides 05 and 13.

Check in proportion to the stakes

A brainstorm needs a quick read. A figure in a board paper or a reply to a complaint needs careful checking. Field Guide 09 sets out a simple routine:

1. Decide who will see it and what happens if it is wrong.
2. Check facts, names and numbers against the original source, not the AI's summary.
3. Open every link and reference you plan to use.
4. On long work, check a few items in full; if one fails, check everything.
5. Ask a fresh chat to find errors and gaps in the draft, then check its findings yourself.

Watch for red flags: exact numbers with no source, references you cannot find, and answers that agree a little too neatly with what you hoped.

Test a setup before you rely on it

When a team uses AI for the same task every week, it pays to test it properly. An evaluation is a fixed set of 10 to 20 real examples with known good answers, rerun whenever you change the prompt, the tool or the model. It shows whether a change made things better or quietly broke something (Field Guide 27).

A person signs off

"The AI wrote it" is not an excuse. Before anything is sent, published or acted on, a named person who owns the decision reads it and signs it off. With that habit in place, teams can use AI with confidence instead of avoiding it.

Goes deeper in our free AI Field Guide: 9. Checking AI output before you use it, 27. Evaluations: testing AI before you trust a change, 5. Context engineering: giving AI what it needs to know, 13. Connectors and integrations: letting AI use your tools safely. Ask for these guides.