The trust gap: why AI works for code but not (yet) for the rest of your work

And even in software, where everyone agrees AI is furthest along, trust in AI-generated code also fell from 40% to 29% in a single year (Stack Overflow's 2025 Developer Survey). And yet developers keep reaching for it anyway and usage has risen to 84%!

So if trust is slipping in the domain that built AI but usage is still on the up, we have to ask ourselves, why? Why is AI working for the software team but not for the rest of the business?

Why AI works for code…

In coding, developers find out something is wrong fast. I think coding absorbed AI more smoothly than almost any other kind of work for a fairly unglamorous reason: not because code is simple, or because AI is unusually good at it (plenty of developers will happily tell you it isn't), but because the software development cycle fits perfectly with the way AI works and catches issues early - before they reach a customer.

A developer writes on their own machine, then pushes the work into a shared repository, where someone else reviews it before it goes anywhere near production. Write, share, review. That habit didn't change when AI arrived. It just got a new author. The first draft might now come from a model instead of a person, but it still lands in the same shared space and goes through the same checks and reviews.

But beyond this - code isn't the result. The result is the new feature or that new webpage. The code... it's the reasoning behind the result. So in the standard software development cycle, the reasoning gets checked continuously, even if the final result is what ultimately matters.

… And why AI doesn’t work for knowledge work

Most knowledge work doesn't have that workflow and, honestly, it never has. A quarterly report, a strategy paper, a board deck: these usually get reviewed as finished documents, not as reasoning. What gets checked is the conclusion, the final report, not which sources it rests on, which assumptions got made along the way, or which alternatives got quietly dropped somewhere between the table of contents and the quarterly projections. If you wanted to review the latest strategy paper you asked your trusted colleague to come to a meeting room and explain their reasoning in person.

AI can now produce that finished document fast and fluently. But the thinking behind it is even more invisible. AI didn't get a shared, reviewable process to slot into - it didn’t get a way to show both reasoning and output. That's because most knowledge work relied on that internal human knowledge in the first place. Now if you want to review the latest strategy paper there’s nobody to invite to that meeting. There’s nobody to trust.

The gap that costs

This isn't a hypothetical gap, and it isn't cheap. In 2025, Deloitte had to refund part of a A$440,000 report it had produced for the Australian government, after an independent reviewer found fabricated references in it, including a citation to a book that was never written, attributed to a real, presumably bemused, academic (Fortune). Months later, an independent investigation found a similar pattern in a report Ernst & Young Canada had produced on loyalty-program fraud: citations that simply didn't hold up when anyone bothered to check (GPTZero's investigation).

In both cases, I'd guess the individual sentences read just fine. Nothing looked obviously wrong on the page. That's exactly the problem. These weren't sloppy, low-effort documents. They were confident, well-formatted ones that nobody had checked at the level of the claims underneath, because there is no equivalent of a code review for the reasoning that produced them.

These aren't isolated cases either. Even in law, a field built entirely on precedent and citation, general-purpose AI tools hallucinate case law in up to 88% of tested queries. Purpose-built legal research tools marketed, somewhat boldly, as "hallucination-free" still get it wrong up to 33% of the time (Stanford RegLab). A database tracking real court cases involving fabricated AI citations has logged over 1,300 entries and is, somewhat depressingly, still growing (Damien Charlotin's hallucination database).

Recently, OpenAI published a paper entitled "Why Language Models Hallucinate" (ArXiv preprint), which dives into the reasons behind these hallucinations and how they can’t be eliminated. Alex Banks has written a great explainer of this work here. TL;DR - the way LLMs are trained, evaluated and benchmarked all intrinsically reward an assured hallucination over an unsure admission that the model just does not know.

A two-minute test you can run yourself

You can see a version of this for yourself, and I'd genuinely recommend it. It takes about two minutes and it's oddly satisfying, in a horrifying sort of way.

Next time an AI tool gives you a confident answer leaning on a stat, a precedent, or a study, ask one simple follow-up:

"Where does that come from? Can you point me to the actual source?"

If the first answer is vague, keep pushing, the way you'd push a colleague who just said "I read it somewhere...".

You'll usually get one of three things back:

• a real, checkable source;
• a source that sounds plausible but doesn't actually say what was claimed when you go and look for it; or
• something invented outright, delivered with exactly the same confident tone as the parts that were true.

That confident tone is doing a lot of work. It's why these errors are so hard to catch just by reading: it's not obviously wrong and behind it isn’t malice - it's a critical-thinking failure. The AI isn’t intentionally telling a lie but using true-sounding facts assembled with faulty reasoning that nobody actually checked. It's the same pattern that makes a good conspiracy theory persuasive: every individual fact could check out but the argument can still be completely wrong.

What this means for the rest of the business

None of this means AI doesn't belong in knowledge work. It clearly does, the same way it belongs in code. What's missing is the equivalent of the habit developers already had: a shared space where the reasoning, not just the finished output, is visible enough for someone else to check it before it matters.

That's the problem I keep coming back to at Count, and one we're spending more and more of our time on. We think it's one of the biggest opportunities left in enterprise AI. Trust doesn't drop because AI gets things wrong sometimes - everyone does. It drops when people find out the hard way, after the fact, the way Deloitte's own numbers show it can. That's the trust gap this piece is actually about. This is the first in a series on what we're finding, including some experiments of our own.