AI data analysts: what they can do, and the part of the job they can't

Hand Claude a CSV of 40,000 orders and ask which regions are slowing down. Within a minute it has loaded the file, checked the columns, written a few dozen lines of Python, run them, drawn a chart and written three paragraphs about the north-east. Two years ago that was an afternoon.

Most of that speed comes from one change: assistants now calculate in code rather than in their heads. Anthropic's code execution tool and OpenAI's Code Interpreter both have the model write code and run it, and that matters, because models doing arithmetic from memory were never dependable. In one study, GPT-4 got only 59% of three-digit-by-three-digit multiplications right without a tool (Dziri et al., Faith and Fate, 2023). With code, the sums are right. What's left to go wrong is everything around the sums.

Here's what an AI analyst is good at today, in our experience and in the vendors' own documentation:

• Writing SQL and Python you'd otherwise type yourself, and explaining code someone else wrote.
• Profiling a new dataset: row counts, nulls, distributions, suspicious duplicates.
• First drafts of charts and summaries, for you to edit.
• Connected work. Through MCP servers, it can query Snowflake, BigQuery, Databricks or a dbt Semantic Layer directly, rather than waiting for an export. The AI for data teams guide has the setup.

If your week is mostly pulling numbers for other people, a lot of that week is now automatable. We won't pretend otherwise.

Benchmarks built from real analyst work tell a less cheerful story than the demos. DABstep, built by Adyen and Hugging Face from 450+ tasks on a real payments analytics platform, asks agents to combine code with reading the documentation. At launch, the best agent scored 14.55% on the hardest tasks (DABstep, June 2025). SpreadsheetBench V2, 321 end-to-end business workflows such as financial modelling and debugging, has a best published score of 34.89% (SpreadsheetBench, checked October 2026). On Spider 2.0's dbt track, where agents work inside a real dbt project, the top entry scores 65.6 (Spider 2.0 leaderboard, read October 2026).

Look at why they fail and a pattern shows up. It's rarely the arithmetic. It's the decisions:

Irrelevant detail hurts too. Adding a single irrelevant clause to maths problems cut GPT-4o's accuracy from 95.2% to 63.1% in Apple's GSM-Symbolic study (Mirzadeh et al., 2024). Real exports are full of irrelevant columns. The safer prompts in prompts for data analysis are built around exactly these failure points.

It replaces tasks, not the job. And the tasks it replaces were rarely the valuable part.

We've written about the three jobs that matter most for analytics teams when AI writes the queries: creating clarity about what's going on, building the case for action, and helping leaders think a decision through. None of those is typing. All of them need someone who knows the business, can be held accountable for a number, and can stand in a room when the number is challenged.

There's a harder truth underneath. An analysis doesn't succeed by being correct; it succeeds by being believed and acted on. As we put it in artifacts, not transcripts: "The graveyard of corporate life is filled with analyses that were right and didn't matter." An AI analyst can produce a correct analysis. Getting a sceptical operations director to change a plan because of it is a different skill.

The strongest counter-argument deserves a straight answer. If your team mostly runs a ticket queue of data pulls and recurring reports, much of that work will move to AI and self-serve tools, and some roles built around it will shrink. The analysts who do well will be the ones who own definitions, judge which questions matter, and check the machine's work.

Think of it as a capable new hire who's fast, tireless and has never met your company. You'd give them the definitions, check their first ten pieces of work closely, and loosen review as they earn it. Same here.

Two settings make this much safer. Give it standing context (a Claude project with your glossary and rules), and connect it to governed definitions where you have them, ideally a semantic layer, so it isn't inventing a meaning for "active customer" from a column name. For a comparison of the two main assistants on this work, see Claude vs ChatGPT for data analysis.

"AI data analyst" now covers four different things:

• A general assistant with code execution: Claude or ChatGPT with a file upload. Best for one-off files. See Claude data analysis and Claude vs ChatGPT for data analysis.
• A general assistant connected to your warehouse through MCP servers. Best for the data team, because it can reach live data and governed metrics.
• Your warehouse's own agent, such as Snowflake Cortex Analyst or Databricks Genie. Best when the definitions already live in the warehouse.
• Your BI tool's agent, such as Looker's Conversational Analytics or Copilot for Power BI. Best for business users who already live in that tool. More in conversational analytics.

For most data teams we'd pick the second as the analyst's own workbench and let business users use the fourth, because the same definitions can serve both. Whichever you choose, the reliability problem is the same one.

The dangerous output from an AI analyst isn't the obviously broken chart. It's the tidy analysis that confirms what you already suspected, which nobody checks because it sounds right. We've written about that failure in the hidden AI hallucination. Anthropic's own guidance concedes its anti-hallucination techniques "don't eliminate them entirely".

A two-minute review before anything ships:

1. Read the question back. Does the analysis answer the question asked, or an easier neighbour of it?
2. Find the definition. For the headline metric, what exactly was counted, over what window, with what excluded?
3. Read the joins. One fan-out join can double a revenue number without an error message.
4. Recompute one figure yourself, from the source.
5. Strike untested causes. Any "because" the data didn't test becomes a hypothesis, labelled as one.

Or let Count do the claim-by-claim part: connect it to Claude or ChatGPT, ask it to audit the analysis, and each claim comes back labelled as a citation, an assumption, model knowledge or a hallucination, with the reasoning chain that produced it.

Can AI replace data analysts?

What is an AI data analyst?

How accurate are AI data analysts?

Is ChatGPT or Claude a good data analyst?