Text-to-SQL: how AI writes SQL, how accurate it is, and how to check it

Type "what was churn last quarter?" and a text-to-SQL system has to make four decisions before it writes a line: which tables hold customers and subscriptions, how they join, what "churn" means here, and what "last quarter" covers. Writing the SQL itself is the easy part.

Modern systems do this in a loop, and with Claude connected to a warehouse over MCP you can watch each step in the tool calls:

1. Read the schema. List tables, read column names and types, sometimes sample rows. Tools such as BigQuery's get_table_info or Postgres MCP Pro's get_object_details exist for this.
2. Link the question to the schema. Map "customers" to a table, "churn" to a status or a missing renewal, "last quarter" to a date column and range.
3. Write the SQL.
4. Run it and repair it. If the query errors, read the message and try again. This step is why agents beat single-shot prompts on benchmarks.
5. Summarise the result in prose, often with a chart.

Steps 2 and 5 are where answers go wrong without anyone noticing. A query that errors gets fixed. A query that joins customers to invoices and counts rows runs perfectly and returns the number of invoices.

The vendors now offer two routes. Raw text-to-SQL gives the model your tables. Semantic text-to-SQL gives it a layer of named metrics and dimensions first (dbt's Semantic Layer, Snowflake's semantic views through Cortex Analyst, Databricks' Genie). The second is slower to set up and much harder to get wrong.

Text-to-SQL benchmarks tell three different stories depending on which date you read them.

November 2024: real schemas are hard. Spider 2.0 swapped the tidy academic databases of earlier benchmarks for enterprise ones: 632 real-world text-to-SQL workflow problems. The paper's best agent, built on o1-preview, "successfully solves only 21.3% of the tasks, compared with 91.2% on Spider 1.0 and 73.0% on BIRD" (Lei et al., arXiv v1). The project site now quotes 17.1% for o1-preview after later fixes to the evaluation suite, so quote one figure and say which.

October 2026: tuned agents close most of the gap. On the current Spider 2.0 leaderboards (read 22 October 2026), the top agent scores 96.70 on Spider 2.0-Snow, 76.23 on Spider 2.0-Lite and 65.6 on Spider 2.0-DBT. On BIRD, the top system scores 82.95 execution accuracy (GrainSQL, September 2026) against a human baseline of 92.96 from data engineers and database students.

January 2026: the yardstick is bent. An expert audit found annotation errors (wrong gold SQL, ambiguous questions) in 52.8% of BIRD Mini-Dev and 62.8% of Spider 2.0-Snow items. Re-scoring 16 agents on a corrected BIRD subset changed their performance by -7% to +31% in relative terms and moved rankings by up to nine places; the correlation between corrected and original rankings fell from 0.85 to 0.32 (Jin, Choi, Zhu, Kang, January 2026).

Our reading: "96%" means a heavily tuned agent agrees with a flawed answer key on a public set. It doesn't mean a model will answer your churn question correctly. The audit is the more useful finding, because its errors look exactly like yours: questions with more than one reasonable reading.

Most wrong text-to-SQL answers at work aren't syntax errors. They're definitions.

Churn can be logo or revenue churn: on ChartMogul's example of three customers paying $20, $30 and $50, losing the $50 customer is 33% customer churn and 50% revenue churn. Monthly active users can be a calendar month or a rolling 30 days (Google Analytics uses rolling). Return on ad spend computed as total value over total spend is not the same as the average of daily ROAS. In each case, both readings produce valid SQL and a confident number. See return on ad spend for one metric's competing definitions.

A semantic layer settles the definition once, in code someone reviewed, so the model picks a metric instead of inventing one. Over MCP that looks like:

• dbt: list_metrics and query_metrics on the dbt MCP server, with get_metrics_compiled_sql to show what ran.
• Snowflake: a CORTEX_ANALYST_MESSAGE tool over a semantic view on the Snowflake MCP server.
• Databricks: Genie, through the Databricks MCP server.

We'd put a semantic layer in front of anyone who isn't going to read the SQL. The fair objection: it only covers what you've modelled, and every team has fewer governed metrics than it thinks. For the long tail, raw text-to-SQL with a human reading the query is still the honest answer. Our semantic layers for AI guide covers the trade-offs.

These work in Claude or ChatGPT, with a warehouse connected over MCP or with a pasted schema (examples illustrative).

Make it read before it writes:

Before writing any SQL, describe the tables you'll use: what one row in each
represents (the grain), the join keys, and which columns hold dates and statuses.
Flag any join that could multiply rows.

Make it declare its definitions:

Question: what was churn by plan in Q3 2026?
Before you answer, list every choice the question leaves open (logo or revenue
churn, gross or net, how trials and downgrades count, which date column, time
zone, inclusive end date). Say which you chose and why. Then write the SQL.

Make it explain the query in plain words:

Explain the query line by line for someone who doesn't read SQL: what's included,
what's filtered out, what's counted. Then give the row count it returned and one
sanity check I can run myself, such as a total I'd know from another report.

Make it reconcile two routes:

Calculate the same number a second way, from a different table or with a
different method. If the two results differ, explain the difference before giving
me either one.

Make it stay on the governed path:

Use governed metrics (query_metrics) where one matches. If none does, say so and
show me the SQL you'd write instead, but don't run it until I agree.

The second prompt is the one we'd keep if we could keep only one. It turns a silent choice into a visible one.

You don't need to write SQL to check it. You need to know what to look for.

1. Find the FROM and every JOIN. Ask: after this join, is there still one row per customer (or whatever you're counting)? If not, counts and sums are inflated.
2. Read the WHERE. Are test accounts, refunds, soft-deleted rows and internal users excluded? Is anything excluded that shouldn't be?
3. Check the dates. Which column (created, billed, paid), which time zone, and is the end date included?
4. Check the aggregation. COUNT(*) or COUNT(DISTINCT id)? A ratio computed from totals, or an average of ratios?
5. Compare with a number you trust. Last month's figure from finance, a total from the source system. If it's off by more than rounding, stop.

Step 4 catches more than you'd expect. Averaging daily ratios instead of dividing totals is a classic analyst error, and models make it too.

For answers that go into a deck or a decision, Count turns these checks into something you can share. Connect Count's MCP server to Claude or ChatGPT and ask it to audit an answer. It replies with a link to a canvas where every claim is labelled (Citation, Assumption, Model knowledge or Hallucination) and sits on a reasoning chain: the prompt, each query and calculation, and the result it rests on. A churn number built on an unstated definition shows as an Assumption rather than hiding in fluent prose. The canvas runs SQL against your warehouse too, so you can fix the join and re-run it there. See also how to check AI answers.

Text-to-SQL is good enough now that refusing to use it is a cost. It's not good enough to use unread.

Use it freely for exploration ("what tables mention refunds?"), first drafts of queries an analyst will review, explaining an unfamiliar query, and questions answered by a governed metric through a semantic layer.

Use it with the check above for one-off business questions against modelled reporting tables, where a wrong answer costs a conversation, not a decision.

Don't use it unread for numbers going to a board, an investor, a regulator or a pricing decision; for anything against raw application tables; or for metrics your company defines differently from the textbook. In those cases the model is likely to be fluent, fast and confidently answering a slightly different question.

For connecting a specific warehouse safely, see Snowflake MCP, BigQuery MCP, Postgres MCP, Databricks MCP and dbt MCP. For prompts beyond SQL, see prompts for data analysis.

How accurate is text-to-SQL?

Can ChatGPT or Claude write SQL?

What is a semantic layer for text-to-SQL?

Will text-to-SQL replace data analysts?

What is the best AI for SQL?