A Frozen Benchmark Measures What a Leaderboard Hides

TL;DR

• The Agentic Analytics Benchmark is a public test from ClickHouse in which models answer 201 analytics questions over a synthetic data warehouse, each writing its own SQL inside an agentic loop.
• Round two, published on October 20, 2026, adds five models to the same board — GPT-6 Astra, GPT-6 Luna, GPT-6 Sol, Claude Opus 5.5 and Claude Sonnet 5.5 — on the same questions, warehouse and judge panel.
• Thirty-four models now sit on the board; Claude Fable 5.1 keeps the top spot at 76.6%, and all five additions land in a 68.7 to 69.7% band, statistically tied with each other.
• Sonnet 5.5 finishes every run inside the 60-turn budget where Sonnet 5 hit the cap on 16 questions, with about 5 cents per question against 7.
• The gain lands on the 83 multi-hop dimensional questions, where Sonnet 5.5 scores 73.5% against 62.7%, on the same table-discovery rate.
• Failures split into a wrong plan, wrong data selection and wrong implementation, and the wrong plan is the largest bucket for every new model, with no runtime errors.
• The harness pins the query engine and the timezone, checks every provider credential before the first paid call, and runs a contamination probe, which is what makes a frozen board worth reading.
• The board's measurement boundaries cover accuracy at one effort setting on one warehouse and exclude a reasoning-effort sweep, four cost rows with unreliable cache accounting, and any conclusion about your own schema.

A leaderboard that has stopped moving is a strange thing to publish. ClickHouse's second Agentic Analytics Benchmark adds five frontier models to a frozen board and reports that the top score barely shifted: Claude Fable 5.1 holds the lead at 76.6%, and the five new rows arrive between ranks 9 and 15. The interesting result sits in the columns beside the score, where turns, truncation and cost per question separate models that a single number makes look interchangeable.

The design is deliberately narrow. One warehouse, 201 curated questions with committee ground truth, a 60-turn budget per question, and a judge panel with one seat per provider where no model grades its own answers. A turn is one model call, and a turn can issue several SQL queries.

Thirty-four models answer the same 201 questions

The board ranks 34 models by pass rate on the same question set. Bright yellow marks the six rows in a statistical tie with the leader, and the post states the interval plainly: on 201 questions the 95% interval is about ±6 to 7 points. That interval is the most useful sentence in the article, because it converts a ranking into a set of overlapping bands.

All five additions conclude on every question, which the turn-limited column shows is rare enough to be worth reporting. GPT-6 Astra, Luna and Sol span 68.7 to 69.7% with overlapping intervals, below GPT-5.5 at 70.4% and above GPT-5.6 at 67.7%. Claude Opus 5.5 enters at 69.7%, two points above Opus 5 and level with GPT-6 Sol.

A caveat in the post deserves repeating, because it limits every GPT-6 row: all three ran at default reasoning effort. GPT-6 supports an "xhigh" setting that no model on the board has run with, and the first benchmark already showed how much a budget-shaped knob moves a Gemini number. Until that sweep runs, the label on those rows describes one configuration.

Claude Sonnet 5.5 finishes every run inside the turn budget

Sonnet 5.5's largest changes were in efficiency: turns, truncation, latency and cost. The pass rate is 68.7%, or 138 of 201 questions, against 62.2% for Sonnet 5. Paired on the same questions it wins 31 and loses 18, a net of 13 at a p of about 0.09, which the authors call directional and refuse to call significant. That restraint is the right call and it is rare in vendor benchmarking.

The efficiency numbers carry more weight. Sonnet 5 hit the 60-turn cap on 16 of 201 runs, which pushed its pass rate five points below its rate on the runs it concluded. Sonnet 5.5 hits the cap zero times, at a mean of 3.5 turns per question against 9.1, a median of 13.7 seconds per answer against 26, and $10.39 for the full run at list price against $14.40. Per question that is about 5 cents against 7.

Truncation is the column that a leaderboard habit hides. The accuracy figure records both incorrect answers and runs that exhaust the turn budget as failures. Buying on the headline score alone lands a team on the cheap seat of a family and then charges it in retries.

The gain lands on the multi-hop dimensional questions

The 201 questions are uneven by design. Eighty-three of them need the warehouse's normalized dimensional layer, and finding that layer was the main discriminator in the first round. Sonnet 5 scored 62.7% on those questions and Sonnet 5.5 scores 73.5%; on the remaining 118 questions the move is 61.9% to 65.3%.

The post explains what drives the delta. Both models discover the dimensional layer on about 89% of the questions that need it; Sonnet 5.5 uses the layer better once it is found. That is a statement about planning and joins, and it points a team at a different intervention: schema documentation and column comments help a model find a table, and clear grain and naming help it use the table correctly.

Failures split into plan, data selection and implementation

Of Sonnet 5.5's 63 failures, 75% are a wrong plan, 19% a wrong data selection and 6% a wrong implementation. There were no attempts skipped and no runtime errors. Sonnet 5 recorded 5% no-attempt and 15% wrong implementation, so the newer model always attempts and computes correctly more often; when it fails, it fails on the plan.

Across every new model the wrong plan is the largest bucket, from 51% for GPT-6 Astra to 75% for Sonnet 5.5, and none of the five produced a runtime error. The five new models produced no runtime errors on this warehouse. For these five models, deciding what to compute remained the largest failure category.

That failure shape changes how a team should test an analytics agent. A useful harness scores the final table and logs the plan, the tables read and the join order. Those traces give a team evidence it can use to improve the schema and the prompt, and they turn a failure into a category.

The harness pins the engine and checks credentials before it spends

The reproducibility controls in this round are the part worth copying into any internal evaluation. The board manifest records the query engine release and the session timezone, and a run on a different engine is refused, because a silent engine upgrade had already flipped one question's ground truth. Every provider credential that the candidates, the judges and the column linker will touch is checked before the first paid call, so a dead judge key stops the run instead of degrading the scoring one question at a time.

Two further controls deserve a mention. Numeric tolerance is a scoring section in the model config, with currency values as the default, so the comparator no longer assumes money. And the judge model carries uncertainty: a many-facet Rasch model over the cached panel votes gives candidate ability, judge severity and an own-family coefficient with bootstrap intervals, computed offline from the results file.

The post also reports a contamination probe on the five new models. None of them recovers an anonymized entity name from a question prefix, at 0 of 25 each, and their completion similarity to the question set sits in the same low band as the rest of the board. Sonnet 5.5, the one addition with a Spider control, recovers 81% of entity names on Spider dev against 7% on these questions, which is what makes the probe credible: it detects memorization where memorization exists.

The board's measurement boundaries

The board measures pass rate at one effort setting, on one synthetic warehouse, with one judge panel, inside a 60-turn budget. It leaves out a reasoning-effort sweep for the GPT-6 family, which the authors name as the next step, and it leaves out cost rows for four models whose gateway usage records do not split cached from uncached input reliably.

It also leaves out your schema. The post is explicit that nothing in the harness is specific to ClickHouse, and equally explicit that your warehouse will have its own discriminator: a slowly changing dimension, a semi-structured event table, a naming convention nobody wrote down. A benchmark answers which model behaves well on a class of questions. Testing on your own tables determines which model behaves well on them.

Run it on your own warehouse before choosing a model

The harness is public at github.com/ClickHouse/data-agent-mnist under Apache 2.0, and adding a model is a registry entry and one command. As an editorial recommendation, a small internal benchmark can start with questions drawn from real analyst requests, the same question set replayed every quarter, a frozen engine version, and a log of turns, cost and failure mode beside each score.

Teams building the analytics application themselves, through custom web and mobile development, inherit both the schema and the question set, which makes them the cheapest place to run that board. The discipline that pays off is the one this post demonstrates: publish the interval, name the configuration, and keep the failure categories visible next to the score.

Sources

Source: "The Agentic Analytics Benchmark, round two: GPT-6, Claude Opus 5.5, Claude Sonnet 5.5" — ClickHouse engineering blog, 2026-10-20. https://clickhouse.com/blog/agentic-analytics-benchmark-round-two

Benchmark harness: github.com/ClickHouse/data-agent-mnist, Apache 2.0, retrieved 2026-11-08. https://github.com/ClickHouse/data-agent-mnist

Charts: four images published in the ClickHouse post above, retrieved 2026-11-08, plus one capture of the post taken the same day.

Source: ClickHouse engineering blog, Agentic Analytics Benchmark round two, published 2026-10-20 and retrieved 2026-11-08.