The hidden AI hallucination: when right facts make a wrong decision

But most of what gets reported under the banner of "AI hallucinations" is the easy kind to spot: a fake citation, a case that doesn't exist, a book nobody wrote. Those should, at least in theory, be easy to spot. There's a harder version of this problem that barely gets covered at all: outputs where every individual fact is true, and the conclusion is still wrong. And for businesses - faulty reasoning costs real money.

For the purposes of this piece, let's call these fragile assemblies of truths that are structurally unsound - a house of cards hallucination.

What people actually measure

There's no shortage of AI benchmarks right now, both academic and commercial, and a lot of them are genuinely useful.

• BrowseComp grades whether a browsing agent finds the one correct source to answer a hard-to-find question.
• AA-Omniscience checks something different: whether a model knows when it doesn't know, rewarding an honest "I'm not sure" over a confident guess.
• HalluLens checks whether a model fabricates something ungrounded in its training data, or contradicts the material it was actually given.
• DataBench, DRACO and AA-Briefcase go further, grading whole analytical, deep research and business outputs, respectively, against detailed rubrics rather than a single right answer.

Every one of these is asking a version of the same question: is the output to a tightly bounded question accurate and / or are the facts going into that output well grounded and sourced. None of them ask the question that actually matters for a lot of knowledge work: was this correct-looking answer arrived at soundly, or did it just get lucky?

Why house of cards hallucinations are hard to spot

That's a big blind spot. A model can search for the right thing, cite something real, and still get to a wrong answer via a bad process: making sweeping claims from just 3 data points, quietly swapping in something it remembers from training instead of what it was actually given, or running the same shortcut on every item in a comparison without checking whether it still applies.

None of that shows up as a "wrong fact." None of it can be fact-checked.

It shows up as a wrong conclusion built out of true-sounding pieces, which is structurally the same trick a good conspiracy theory uses: nothing you check is false, and the argument is still broken. These are critical-thinking failures, and it's the thing the team at Count have spent the last few months trying to get a handle on. More on that soon. For now, here's what it looks like when it happens for real.

A real example

Recently we came across a great experiment by Prof. Adam Kucharski who ran a deliberately unfair test on Microsoft Copilot that shows exactly this type of hidden critical-thinking failure.

Adam gave Microsoft Copilot a dataset of a few thousand survey responses, split into a "US" group and a "UK" group, and asked it to describe how the two groups differed emotionally. Copilot reported that:

"[...] US and UK responses differ mainly in tone, intensity, and wording style, even though they express similar emotional states."

He ran it again with a different dataset for 5 countries and Copilot got more specific, stating:

"Italians are three times more likely to aspire to a career in the arts than the UK"

and:

"Americans are 1.5x more business focused than the French."

Everything looks good so far, right?

But here's the trick. Adam generated these datasets himself. And in the first dataset, both groups were actually identical. And in the second dataset, all five groups were identical!

Copilot didn't notice and provided precise-sounding numbers, invented wholesale, that described a difference that existed nowhere in the input.

Here's my interpretation on what happened, based on Adam's explanation but also my own experiments and experience:

1. Copilot assumed there would be differences between the groups, even when none existed. This makes sense - "why would somebody ask me to explain the differences if there are none?"
2. Copilot (probably - my guess here) looked at a sample of each dataset and not the full thing, which could have led it to see apparent differences that weren't actually representative of the entire dataset.
3. Copilot filled in gaps with its own internal-to-the-model assumptions / stereotypes about each nationality, rather than strictly adhering to the data it was given. This is why it could produce specific-sounding differences even when none existed.

Adam's own conclusion is direct: "If this had been a real dataset, groups with no discernible differences could easily have ended up being reported as wildly divergent, purely based on the underlying large language model's pre-existing notions of what different demographic groups are like."

The output read as a normal, plausible finding precisely because it matched stereotypes that a reader would already hold. A fake case citation is checkable in thirty seconds. A finding that confirms what you already expected doesn't feel like something you need to check at all, and that's what makes this failure mode more dangerous than a fabricated citation.

Why this isn't a one-off

If someone had pulled up a sample of the raw responses from each country side by side then there's a good chance they would have spotted some duplicates, gone to investigate further and then realised the datasets were identical.

Now, we all like to think we are diligent but I wonder how many of us actually would? I must admit, there are times when I've taken an AI's analysis at face value without digging into the raw data myself... that is, until much later when I start to see inconsistencies or results that don't quite add up.

Research on how people actually respond to AI reasoning suggests this is the default, not a rare lapse: readers judge an explanation as trustworthy mostly by whether they agree with where it lands, rarely by checking whether the logic holds and a confident tone actively suppresses error detection while keeping reliance high.

A separate study of nearly a thousand real AI-assisted work tasks found the same pattern from the other direction: the more confidence people had in the AI's ability, the less critical thinking they actually applied to its output.

Fluent and confident is exactly the profile a critical-thinking failure wears, and it's the profile that gets waved through without a second look.

A test you can run yourself

Last time, I proposed a short test you could try yourself about evidence sourcing: ask your favourite LLM where a stat came from and keep pushing until you get a real answer, a wrong answer, or nothing.

Today, let's ask the LLM about its reasoning - don't worry, this experiment takes about 2 minutes too.

Next time an AI tool hands you a comparison (countries, segments, competitors, time periods, it doesn't matter), ask it one follow-up before you trust the conclusion:

"Show me two or three concrete examples from each group, side by side, that actually support that difference."

If it can produce them, you've learned something real. If what comes back is vague, restates the summary instead of pointing at underlying evidence, or the examples don't actually look different from each other once you read them: that's the above Microsoft Copilot failure, live, in front of you.

It's a shortcut, not a proof, but it targets the exact thing that Kucharski's experiment does.

What this means for the rest of the business

None of this means AI tools can't be used in making business decisions. AI is still faster at producing a first pass than almost anything else. What's missing is a way to check the reasoning - the AI's own critical thinking process - before the conclusion goes anywhere near a decision, the same way a code review lets a developer catch a reasoning error before it ships. That check barely exists for knowledge work right now, and none of the benchmarks getting attention today are built to measure it either.

That's the actual question we've been building tools to answer at Count: can you see why the model thinks its answer is right?