AI benchmarks explained: what the scores measure, and what they don't

In February 2026 OpenAI stopped reporting its scores on SWE-bench Verified, one of the most quoted coding benchmarks. An audit of 138 hard tasks found 59.4% had flawed tests or descriptions, and models including GPT-5.2, Claude Opus 4.5 and Gemini 3 Flash could reproduce the reference fixes from memory (reported by Decrypt, February 2026). Models that scored around 70% on Verified scored about 23% on the harder SWE-bench Pro.

That's the whole problem with benchmarks in one story. A benchmark is a public exam. Once the questions are public, they can end up in training data. Once models have practised, the exam measures memory as much as skill.

Benchmarks are still the best shared evidence we have. They just need reading the way you'd read a company's own accounts: closely, with the footnotes.

These are the names that turn up in model launches and comparison articles, with the top scores we found on 22 October 2026.

Notice how the scores fall as the tasks get closer to real work. Multiple-choice science: 96%. Real finance tasks: 64%. Real spreadsheet workflows: 35%. That gradient is the most useful thing a table like this can tell you. For which of these to trust on accuracy, see what is the most accurate AI.

GPQA was hard when it was published. PhD experts in the relevant field scored 65%, and skilled non-experts with web access scored 34% (GPQA paper, 2023). Today the top models score above 96% on its Diamond subset. Artificial Analysis removed GPQA Diamond from its Intelligence Index in September 2026, after removing MMLU-Pro in January, because neither separated the leaders any more.

A saturated benchmark has two failure modes. Small differences between models become noise: on a few hundred questions, a couple of points is a handful of answers. And the remaining errors may be errors in the benchmark itself, not the model.

That second one is not hypothetical. FutureHouse found that about 29% of HLE's text-only chemistry and biology answers conflicted with peer-reviewed literature; the HLE team's own follow-up put the problematic share at about 18% (FutureHouse, 2025). In text-to-SQL, an expert audit found annotation errors in 52.8% of BIRD Mini-Dev items, and re-scoring agents on corrected questions moved them by up to nine places (Jin et al., January 2026). When the answer key is wrong, the leaderboard is partly ranking who agrees with the mistakes.

Most people meet benchmarks through a leaderboard, which blends several of them into one rank. Three kinds are worth knowing.

Composite indexes

The Artificial Analysis Intelligence Index (v4.3.2) weights ten evaluations, from Humanity's Last Exam to AA-Omniscience and Terminal-Bench. It changes its mix as tests saturate, so a score from January and one from October are on different scales. Its leader on our check date was Claude Opus 5.5 (max) at 58.

Preference arenas

LMArena, now called Arena, shows people two anonymous answers and asks which is better, then ranks models Elo-style. It's the best signal of what people like reading. It is weak evidence of correctness, because a confident, well-formatted wrong answer can win the vote.

The Open LLM Leaderboard

Hugging Face's Open LLM Leaderboard ranks open models (ones you can download) on six benchmarks run through the EleutherAI evaluation harness: IFEval, BBH, MATH Level 5, GPQA, MuSR and MMLU-Pro (Hugging Face docs). It's useful if you're choosing a model to host yourself. It doesn't rank Claude, ChatGPT or Gemini, and two of its six tests are the ones Artificial Analysis has since retired as saturated.

Two models answer 100 questions. Model A gets 60 right and 40 wrong. Model B gets 55 right, 5 wrong, and says "I don't know" to 40. Which is more accurate?

On plain accuracy, A wins. On a rule that penalises wrong answers, B wins easily. Both rules are in use.

• Accuracy counts correct answers and ignores the rest. It rewards guessing.
• Hallucination rate, as AA-Omniscience defines it, is wrong answers divided by everything that wasn't correct. It rewards declining.
• Net scores subtract wrong answers from right ones, so a guess has a cost.
• F1 blends precision and recall, and gives yet another order.

OpenAI's researchers argue that the dominance of accuracy-only scoring is one reason models hallucinate: they are trained "to be good test-takers" (Kalai et al., September 2025). Google DeepMind's SimpleQA Verified rebuilt OpenAI's SimpleQA to fix noisy labels and topical bias (arXiv, September 2025), which shows how much work goes into a fair answer key before any scoring starts.

When two articles disagree about which model is "more accurate", check whether they're quoting the same scoring rule. Often they aren't.

A launch post says a model "leads on accuracy". Before you believe it, or repeat it, run through five questions.

1. What exactly was measured? Recall, faithfulness to a document, coding, preference? Find the one-sentence definition.
2. Who ran it? Vendor-run, independent, or self-submitted? Self-submitted results need care: on DABstep, a data-analysis benchmark whose best agent scored 14.55% on hard tasks at launch, self-submitted entries on its public board have since claimed far higher scores, and nobody has independently checked them.
3. When, and which version? Benchmarks get revised and leaderboards go stale. Spider 2.0's headline figure changed between its paper and its site.
4. Is it saturated? If the top five are within two points near the ceiling, the ranking is noise.
5. Does declining count? If not, the leader may simply guess more.

Then build the only benchmark that's about you: twenty questions from your own work with known answers, run on the two models you're choosing between. Score right, wrong and declined separately.

Scoring your own set is where the time goes, because each answer is several claims. Count does that part: connected to Claude or ChatGPT, it splits an answer into claims and labels each as a citation, an assumption, model knowledge or a hallucination, with the reasoning chain behind it. For checking single answers by hand, see how to check AI answers.

What is an AI benchmark?

What are the most important LLM benchmarks in 2026?

What is the Open LLM Leaderboard?

Can AI benchmarks be trusted?

What does benchmark saturation mean?