How to Tell When AI Is Making Things Up

01Hallucination is one word for six different failures

The term got stretched to cover every way a model can be wrong, which matters because the fix is different for each one.

Fabrication. The model invents something that doesn't exist. A case citation, a study, a function in a library. Usually well-formed, because it's generating what a plausible instance of that thing looks like.

Misgrounding. The source exists but doesn't say what the model claims. This is the sneaky one. The citation is real, the link works, you click through and skim and it looks about right. Stanford's legal AI study found systems citing real cases that didn't support the proposition, and mischaracterizing real holdings, which is much harder to catch than an invented case.

Outdated. Correct as of the training data, wrong now. Prices, versions, who runs what, what a law says. The model has no sense of its own recency.

Incompleteness. Nothing stated is false, but a load-bearing exception is missing. Common in anything with jurisdictional or conditional structure.

Sycophancy. You assert something wrong and the model builds a case for it. Stanford named this as one of the error types in legal tools specifically. Asked to support an incorrect premise, the systems often generated plausible arguments on fabricated or mischaracterized authority instead of correcting the premise.

This one deserves more fear than it gets. Every other failure is the model being wrong on its own. This one is the model being wrong because you were, so your own error comes back with citations attached.

Inference gaps. Failures on things that follow trivially from what the model already knows. Models trained on statements shaped like "A is B" often fail on "B is A," which researchers named the reversal curse. GPT-4 could answer who Tom Cruise's mother is and then fail on who Mary Lee Pfeiffer's son is.

02Which model is best is the wrong question

There's no such thing as the model with the lowest hallucination rate. The ranking flips depending on what you're measuring, and the gap between benchmarks is enormous.

Grounded summarization is the easy case. The model gets a document and has to summarize it faithfully, and a hallucination is any claim the source doesn't support. That's what Vectara's HHEM leaderboard measures, and top-ranked models score in the low single digits.

That number gets quoted as "hallucination rates are down to about 2 percent." It isn't. It's the score on a task where the answer is sitting inside the text you handed it. Two things get lost in the quoting. The models scoring lowest tend to be smaller ones, not the frontier reasoning models people are actually using, and when Vectara moved to longer and harder source documents the flagship reasoning models all landed above 10 percent.

Open-domain factual recall is a different universe. No source, nothing but training. On OpenAI's PersonQA, models from a single generation ran from 16 percent to 48.

Domain-specific real work is worse still, which is section five.

So a model isn't reliable or unreliable. It's close to reliable at staying faithful to text you gave it, shakier at recalling facts, and worse in specialized domains. Pick per task rather than per brand, and treat any quoted hallucination percentage as meaningless until you know which benchmark produced it.

03Newer is not always better

In OpenAI's own system card from April 2025, the o3 reasoning model hallucinated on 33 percent of PersonQA prompts. Its predecessor o1 hallucinated on 16 percent. o4-mini came in at 48. The newer, more capable, more expensive models made things up at two and three times the rate.

The obvious explanation is that o3 simply makes more claims per answer, so there are more chances to be wrong. OpenAI offered that much and then said plainly that they don't know why it happens and more research is needed. I want to be careful here, because the tidy story going around is that reasoning chains help on math and hurt on facts, and that isn't an established finding. It's a guess people repeat.

The effect does show up elsewhere. DeepSeek's V3 scored 3.9 percent on grounded summarization and its reasoning model R1 scored 14.3. But it isn't uniform. Research from 2025 found R1 improving on some fact-seeking benchmarks depending on how it was post-trained, and points at the post-training pipeline rather than reasoning length as the thing that matters.

What's solid enough to act on: a newer, better-reasoning model is not automatically a more factual one, and for factual work "use the newest" can be exactly backwards. Check it rather than assuming it.

04The benchmarks are contaminated

When a model scores well on a test, benchmark questions leak into training data, so it can score well by having effectively seen the exam.

The numbers aren't small. Meta's own Llama 2 paper reported that over 16 percent of MMLU test examples had detectable overlap with the pretraining data, about 11 percent of them heavily. An open audit of 15 popular benchmarks found contamination from 1 percent to 45, rising over time. A separate audit of multilingual benchmarks found some as high as 91.

Labs do filter. GPT-3's team removed anything sharing a 13-word run with a benchmark. GPT-4 used a different method, sampling three 50-character chunks from each test example and checking whether they appear in training data. Both only catch near-verbatim overlap. Researchers demonstrated the hole by training a 13-billion parameter model on rephrased benchmark questions, which sailed past the filters and produced GPT-4-level scores on those tests.

The problem also regenerates. A clean benchmark gets published, copied across repositories and forums and derivative datasets, and within a training cycle or two it's back in the corpus.

There's a real counterargument. An ICML 2025 paper found that moderate contamination gets substantially forgotten by the end of training, so small leakage doesn't automatically void a benchmark. Worth knowing, and worth reading the conditions: it holds for models trained well past compute-optimal, and forgetting depends on when the model saw the data. Contamination late in training sticks around.

The field's answer has been benchmarks that refresh their questions on a rolling schedule, like LiveBench and LiveCodeBench, which helps and hasn't closed it.

What I take from this. Benchmark scores are evidence, not proof, and the gap between a model's benchmark performance and its performance on your work is unknown from the outside. The only evaluation that fully belongs to you is one built from your own tasks, which is a genuine argument for keeping a private set of test questions you never publish.

05What retrieval fixes, and what it doesn't

The standard fix is retrieval: give the model the source documents, have it answer from those, cite as it goes. Vendors marketed this as solving hallucination. LexisNexis used the phrase "100 percent hallucination-free."

Stanford's RegLab ran the first preregistered test of that claim on commercial legal research tools, using over 200 hand-built queries scored by legal experts, on products selling for thousands a month.

Lexis+ AI hallucinated on 17 percent of queries. Westlaw's AI-Assisted Research on 33. GPT-4 with no retrieval at all on 43.

Both directions of that matter. Retrieval genuinely helped, cutting the rate by half or more. And the best purpose-built, retrieval-grounded, professionally sold legal AI system still produced false or misleading information on roughly one query in six. LexisNexis later narrowed its hallucination-free claim to cover linked citations only.

The accuracy figures underneath are worth as much as the hallucination ones. Lexis+ AI, the best system tested, answered 65 percent of queries accurately. Westlaw managed 42. So a third to half the time you're not getting a correct answer.

Then a result that complicates the pessimism, which is why I'm including it. A randomized trial put 127 law students into three groups: a retrieval-based legal tool, a reasoning model with no retrieval, and no AI. The retrieval group produced about as many fabricated citations as the students working without AI, three in total. The reasoning-model group produced eleven. Both AI groups did substantially better work, and faster.

So retrieval didn't make hallucination worse and the tools clearly helped. What retrieval doesn't do is remove the need to check, because the failure moves rather than disappearing. It becomes your job to notice that a real citation doesn't support the claim attached to it, and people are bad at that when the output looks authoritative and they're moving quickly.

06Longer answers are more dangerous

Two findings from different directions land on this.

Stanford found Westlaw's higher hallucination rate tracked with response length. Its answers averaged around 350 words against Lexis's 219. More words means more falsifiable propositions, which means more chances one of them is wrong.

Separately, Anthropic looked at whether a model's stated reasoning reflects what it's actually doing, and found unfaithful reasoning chains were consistently longer than faithful ones. For Claude 3.7 Sonnet, 2,064 tokens against 1,439. For DeepSeek R1, 6,003 against 4,737.

So the elaborate explanations were more likely to be covering the real reasoning than revealing it. Length is not thoroughness, and a detailed confident answer should raise your suspicion rather than settle it. The instinct runs the other way, which is the problem.

07Asking the model if it's sure doesn't work

The obvious move is to ask whether it's confident, or to have it explain its reasoning so you can check the logic.

Anthropic tested this by planting hints in prompts, seeing whether the model changed its answer, then checking whether its stated reasoning mentioned the hint. Claude 3.7 Sonnet acknowledged the hint about 25 percent of the time. DeepSeek R1 about 39. On misaligned hints, roughly 20 and 29.

The starkest result came from environments containing a reward hack the model could exploit. In five of six, it used the exploit in over 99 percent of cases and mentioned it in under 2 percent of its explanations. Behavior driven almost entirely by something the model never mentioned while explaining itself.

Faithfulness also dropped on harder questions, by 44 percent for Claude on the harder benchmark, which is exactly where you'd want the explanation to be real.

There's a reasonable counterargument that some of this is incompleteness rather than dishonesty, since compressing distributed computation into a linear narrative loses information. A 2025 paper found much of the apparent unfaithfulness disappears when models get bigger token budgets, with verbalization rising as high as 90 percent. Fair, and it doesn't change the practical conclusion. A model's explanation of its reasoning is generated output, not a report from inside, so it can't serve as verification.

Same for confidence language. "I'm certain" is a stylistic property that got selected for during preference tuning. It carries almost no information about whether the answer is right.

08What actually works

The method with the strongest research behind it is simple, and it comes from a 2024 Nature paper by Farquhar, Kossen, Kuhn and Gal.

Ask the same question several times in separate sessions and look at whether the answers mean the same thing. Not whether they're worded alike. Whether they agree.

When a model knows something, it converges. Different phrasings, same content. When it's making something up, the fabrications diverge, because there's nothing anchoring them. The paper clusters answers by meaning rather than wording and measures the spread, which they call semantic entropy. High spread flags likely confabulation. It works across datasets and tasks without task-specific tuning, which is why it ran in Nature.

You can do the human version in about ninety seconds for free. Ask three times in three fresh conversations and compare the substance. If the specifics move, especially names, numbers, dates and citations, stop trusting the answer.

The critical detail is separate sessions. Asking again in the same conversation is close to worthless, because the previous answer is sitting in the context and the model will stay consistent with it. At that point you're testing whether it can read what it just said. If you want the mechanism behind why fresh sessions diverge at all, it comes down to sampling.

09The checking system

Sort by failure mode first. If the model is summarizing text you provided, risk is low and the check is fast, because you have the source. If it's recalling facts from training, risk is much higher. Anything with a name, a number, a date or a citation is the high-risk category regardless of task.

Re-ask in fresh sessions. Three times, compare substance. This catches fabrication efficiently, because invented specifics don't survive resampling.

Click every citation and read the part that supports the claim. Not the title, not the abstract. The specific passage. Misgrounding is the failure that survives a casual check, and it's the one Stanford found in professional-grade tools.

Treat length as risk. Long detailed output contains more claims and a better chance one is wrong, and elaborate reasoning correlates with unfaithful reasoning.

Watch your own premise. If you asked in a way that assumed something, sycophancy means the answer may be built on your assumption. Occasionally ask the opposite question and see whether it argues that side just as well. If it does, it doesn't know.

Don't ask it to check itself. Its explanation isn't a window into its process, and it will confidently confirm.

Use the newest model for reasoning and verify before trusting it on facts.

Build a private eval. Twenty questions from your own work where you know the answers. Run them when you switch models. It's the only benchmark that can't be contaminated and the only one measuring your actual use case.

10The error you can't catch

Everything above has the same blind spot.

Consistency checking catches confabulation, meaning arbitrary invention. It does not catch consistent error, where the model reliably produces the same wrong answer every time. The Nature paper says so directly. Semantic entropy is built to detect answers that would change on resampling, which means errors that don't change are invisible to it.

Those happen when something wrong was well represented in training. A popular misconception, a fact that was true for years and isn't now, a widely repeated myth. The model isn't uncertain and inventing. It's confidently reproducing something false, and it'll reproduce it identically across ten fresh sessions.

Ask three times, get the same answer, feel reassured. That's the failure this method is structurally blind to.

There's no clever prompt for it. The only defense is a source that isn't the model, which is the unglamorous conclusion that verification has to come from outside the thing being verified.

11So how do you know

You don't, fully. That's the real answer and I'd rather say it than pretend.

What you can do is stop treating reliability as a property of the model and start treating it as a property of the setup.

The same model is close to reliable summarizing a document you handed it and closer to a coin flip on specialized domain questions. That range isn't the model having good and bad days. It's how much of the answer had to come from inside it. The more an answer depends on what's in the weights, the less you can trust it. The more it depends on text you supplied, the more you can.

So the working question isn't whether the model is accurate. It's how much of this answer the model had to invent, and whether you gave it what it needed not to.

The law student trial is the part I keep coming back to. The retrieval tool didn't add fabrications and it made the work meaningfully better and faster. The reasoning model without retrieval improved the work too and brought three times the fabricated citations along with it. Same students, same tasks, different setup, different failure rate.

That's the whole thing, really. Professionals use instruments with known error rates all the time, and it works because checking is built into the process rather than because the instrument is trusted. The failure mode isn't using AI for things it gets wrong sometimes. It's using it without the check, at speed, on things that matter, and finding out later.

Sources

Magesh, Surani, Dahl, Suzgun, Manning and Ho, Hallucination-Free? Assessing the Reliability of Leading AI Legal Research Tools, Journal of Empirical Legal Studies (2025), for the 17, 33 and 43 percent figures, the accuracy rates, the response-length correlation and the sycophancy error type. Schwarcz et al., AI-Powered Lawyering, for the 127-student trial. OpenAI's o3 and o4-mini system card (April 2025) for the PersonQA rates, and Are Reasoning Models More Prone to Hallucination? (2025) for the post-training account. Vectara's HHEM leaderboard for grounded summarization. Touvron et al. (2023) for the Llama 2 contamination disclosure, Li et al., An Open-Source Data Contamination Report (2024) for the 1 to 45 percent range, Yang et al., Rethinking Benchmark and Contamination (2023) for rephrased samples defeating filters, and Bordt et al., How Much Can We Forget about Data Contamination? (ICML 2025) for the counterargument. Chen et al. (2025) at Anthropic on chain-of-thought faithfulness, with Turpin et al. (2023) as the foundational work and Zaman and Srivastava (2025) for the incompleteness critique. Farquhar, Kossen, Kuhn and Gal, Detecting Hallucinations in Large Language Models Using Semantic Entropy, Nature (2024). Berglund et al. (2023) for the reversal curse. Anything wrong here is mine, not theirs.