Conversational analytics: what "chat with your data" gets right, and where it guesses

A sales director types "what was churn last quarter?" into a chat box and gets 4.2% back in eight seconds. That's conversational analytics, and the speed is real. Every major data vendor now ships a version: Looker's Conversational Analytics, Copilot for Power BI, Databricks Genie, Snowflake Cortex Analyst, ThoughtSpot's Spotter, Omni and Hex's Threads. Or you can connect Claude or ChatGPT straight to the warehouse through an MCP server and skip the BI tool entirely.

Under the chat box, there are two ways to get from a sentence to a number.

1. Semantic-layer first. The model doesn't write SQL against raw tables. It picks from metrics, dimensions and joins your data team has already defined, and the tool builds the query. Looker's version works this way: instead of querying your database directly, it selects fields, filters and sorts, and Looker composes the query from your LookML model (Google Cloud docs). Snowflake's Cortex Analyst reads semantic views for the same reason, saying plainly that "generic AI solutions often struggle with text-to-SQL conversions when given only a database schema".

2. Text-to-SQL against the schema. The model reads table and column names, writes SQL and runs it. This is what you get when you point Claude at a warehouse with no definitions attached. It's flexible, it can answer questions nobody modelled, and it's where most confident wrong numbers come from. See text to SQL for the mechanics.

Most real deployments mix the two: governed metrics where they exist, SQL where they don't. The question that matters is whether the answer tells you which one it used.

That 4.2% churn figure could be correct and still be wrong for the meeting. Logo churn or revenue churn? On ChartMogul's worked example, three customers paying $20, $30 and $50, losing the $50 one is 33% customer churn and 50% revenue churn (ChartMogul). Stripe's subscriber churn adds new subscribers to the denominator; ChartMogul's doesn't. A chat tool asked for "churn" picks one, and usually doesn't say which.

The research shows how often the meaning, not the syntax, is the problem. When a UIUC team audited two widely used text-to-SQL benchmarks, they found annotation errors in 52.8% of BIRD Mini-Dev items and 62.8% of Spider 2.0-Snow items, including questions ambiguous enough that the official "correct" SQL picked one reading of several. Re-scoring agents on corrected items moved rankings by up to nine places (Jin et al., January 2026). Real business questions are vaguer than benchmark questions, not less.

Three other things the vendors' own docs tell you:

• The fallback is general knowledge. Microsoft says that when a question relates to the semantic model, Copilot for Power BI uses it; "otherwise, it answers from the large language model's general knowledge" (Power BI docs). That's the answer most likely to be confidently wrong.
• Preparation decides quality. Without a prepared semantic model, Microsoft says "Copilot can misinterpret the data and return generic or inaccurate results." Databricks tells teams to add metrics, business rules and verified answers to improve Genie's responses.
• A plausible answer isn't a checked one. Google's Looker docs warn that Gemini "can generate output that seems plausible but is factually incorrect" and recommend validating it.

None of these vendors claims its chat box is always right. They tell you how to make it right more often, and leave the checking to you.

Our view: if your business users live in one BI tool and its semantic model is in good shape, its chat is the right front door for them. For the data team, and for questions that cross tools, we'd use Claude connected to the dbt MCP server and the warehouse, read-only, with project instructions that make it state its definition and show its SQL. The full setup, step by step, is in AI for data teams.

The setup we'd avoid is the most common one: an assistant connected to raw tables with no definitions and write access left on. It answers anything, and it answers anything confidently.

Requirements matter too. Copilot for Power BI needs a paid Fabric capacity (F2 or higher) or Power BI Premium (P1 or higher); a Pro licence alone isn't enough (Microsoft Learn). Looker's version returns at most 50,000 rows per query and doesn't yet handle forecasting or advanced statistics (Google Cloud docs).

We've been arguing about analytics interfaces for a long time. Count started because we thought dashboards were a crude way to understand a business, and we said so in dashboards are dead back in 2021. So we're wary when the whole industry settles on a single new interface again, this time the chat thread.

Chat is the right way to ask. It's the wrong place to keep what you found. A thread is linear: the definition the model chose sits ten messages above the number, the SQL is collapsed, and the chart that answered the follow-up has no link back to the question that framed it. When someone forwards a screenshot of the answer, all of that is gone. We made the long version of this case in chatbots aren't the answer and why AI analysis tools need to produce artifacts, not transcripts.

The fair counter-point: for a quick lookup ("how many orders yesterday?") chat is fine, and building anything heavier would be ceremony. The line we'd draw is whether the number will be repeated to someone else. If it will, it needs to travel with its definition, its query and its caveats, in something people can open and question.

One wrong number in front of an executive can undo a quarter of goodwill. The rollouts we've seen work, written up in how to deploy an analytics agent in two weeks, follow the same shape:

1. Scope narrowly. One domain (say, subscription revenue) and the ten questions people ask about it most.
2. Write the definitions down first, in your semantic layer or a glossary the tool reads. Most wrong answers trace back to a definition nobody wrote.
3. Build a test set. Twenty questions with known answers. Snowflake, Databricks and Microsoft all describe some form of verified queries or verified answers; use whichever your tool supports, and keep your own list regardless.
4. Let a few people try it and watch what they ask. Their phrasing will break things your test set didn't.
5. Widen access only when the test set passes, and rerun it whenever the model or the definitions change.

Ambiguity is the thing to design for. A good tool asks "which quarter, fiscal or calendar?" A poor one picks. If yours picks, put the answer in the definitions.

1. Ask what it counted. "Which metric definition, filters and time window did you use?" If it can't say, don't forward the number.
2. Ask where it came from. A governed metric, SQL the model wrote, or the model's general knowledge? Only the first two belong in a report.
3. Look at the query. Check the join keys and the date filter. Those two cause most wrong-but-plausible results.
4. Rephrase and ask again. "Customer churn in Q3" and "how many customers did we lose in Q3?" should give the same answer. If they don't, the question is ambiguous and the tool is guessing.
5. Recompute one number from a source you trust: the finance report, the billing system, last month's board pack.

That's five questions per answer, which is a lot of clicking for every lookup. It's the job Count does: connect it to Claude or ChatGPT, ask it to audit an answer, and it labels each claim as a citation, an assumption, model knowledge or a hallucination, with the reasoning chain under it. The broader method is in how to check AI answers.

What is conversational analytics?

Can I chat with my data in Claude or ChatGPT?

Is conversational analytics accurate?

What are the best tools to talk to your data?