Published: July 8, 2026
Databricks benchmarked AI coding agents on its own multi-million-line production codebase — spanning Python, Go, TypeScript, and Scala — and published the results on July 8, 2026, showing Claude Opus 4.8 completing 87% of real engineering tasks at $1.94 per task, open-source GLM 5.2 matching that quality tier at just $1.28 per task, and Claude Sonnet 5 costing more per completed task than Opus despite a lower token price. The benchmark, authored by Vinay Gaba, reaches a conclusion that inverts how most teams currently evaluate models: token price is a poor predictor of actual cost, and no single vendor occupies the entire performance frontier.
Why Databricks Built a Benchmark That Cannot Be Gamed
Public benchmarks such as SWE-Bench use synthetic tasks that models can be fine-tuned to score well on — a problem Databricks called out directly. Databricks’ benchmark evaluates agents on actual pull requests from its internal codebase, with correctness verified by real test suites rather than an LLM judge. The Databricks team stated: “We did not use an LLM judge to evaluate correctness, since we’ve found that this rewards sounding right over being right.”
The codebase is too large and proprietary for any model to have trained on, which eliminates the contamination problem that makes public benchmarks unreliable at the frontier. Tasks reflect what Databricks engineers actually do: roughly 25% low-complexity, 60% medium-complexity work. Solutions were reviewed by engineers before being counted as successful completions.
The Results: Cost Per Task Overturns Token Pricing Logic
The benchmark’s central finding is that token price per million and cost per completed task diverge significantly across models. Claude Sonnet 5 is approximately 1.7× cheaper per token than Claude Opus 4.8, but Databricks’ real-task measurement showed the reverse on a per-task basis: Sonnet 5 cost $2.09 per task versus Opus 4.8’s $1.94, while completing 6 percentage points fewer tasks (81% vs. 87%). Databricks attributed this to Sonnet 5 working longer and reading more context to reach the same quality bar — consuming 1.9× more tokens per task than Opus 4.8.
The three-tier capability structure Databricks observed:
- Top tier — Claude Opus 4.8 ($1.94/task, 87% completion) and GLM 5.2 ($1.28/task, statistically tied with Opus 4.8 on quality)
- Mid tier — Claude Sonnet 5 ($2.09/task, 81% completion)
- Routing tier — GPT 5.4 Mini and Claude Haiku, described as the class of models to push lower-complexity tasks toward
Databricks concluded: “The Pareto frontier for coding tasks (i.e. best quality for a given cost) includes models from OpenAI, Anthropic, and open source. This means today, only a mix of tools can provide frontier performance.”
GLM 5.2: The Open-Source Model That Matched Frontier Quality
GLM 5.2 is the benchmark’s standout finding. Databricks reported it landed in the top capability tier — “statistically tied with Opus 4.8 on quality” — at $1.28 per task, 34% cheaper than Opus 4.8 on a per-task basis. Databricks stated: “Open models, and GLM 5.2 in particular, are now able to handle even the highest level of task difficulty.”
The practical implication for engineering teams: “Because of its great performance for everyday coding tasks, we’ve been focused on serving GLM with the best performance, and the evidence shows it’s time to start deploying these as daily drivers for coding.” The gap between frontier commercial models and capable open-source alternatives, on real production code, has narrowed to a price differential rather than a quality differential.
The Harness Matters as Much as the Model
A secondary finding with immediate operational significance: the agent framework (harness) a model runs through changes its actual cost by more than 2× with no change in output quality. Databricks ran the same model at the same thinking effort through two harnesses — Claude Code/Codex and its own Pi framework — and observed cost per task “differed significantly (more than 2x in some cases), while quality remained the same.” Pi sent approximately 3× less context per turn than Claude Code or Codex.
For teams selecting AI coding tools, this means the model choice and the deployment framework are separate, equally important decisions. Selecting the most capable model and deploying it through a context-heavy framework can cost more than selecting a slightly less capable model with a leaner harness.
Frequently Asked Questions
What did Databricks test in its AI coding agent benchmark?
Databricks tested AI coding agents on its own multi-million-line production codebase, covering Python, Go, TypeScript, Scala, and other languages. Tasks were drawn from real Databricks pull requests, and correctness was verified by actual test suites — not an LLM judge. The benchmark published on July 8, 2026 measured task completion rate and cost per completed task for models including Claude Opus 4.8, Claude Sonnet 5, GLM 5.2, and GPT 5.4 Mini.
Which AI coding model performed best in the Databricks benchmark?
No single model ranked first on all dimensions. Claude Opus 4.8 achieved the highest task completion rate at 87% and $1.94 per task. GLM 5.2, an open-source model, matched Opus 4.8’s quality tier while costing $1.28 per task — 34% less. Databricks concluded that the Pareto frontier spans models from OpenAI, Anthropic, and open source, meaning no single vendor provides the best performance at every cost point.
Why did Claude Sonnet 5 cost more per task than Opus 4.8?
Claude Sonnet 5 is approximately 1.7× cheaper per token than Claude Opus 4.8, but consumed 1.9× more tokens per task on Databricks’ real codebase — making Sonnet’s per-task cost $2.09 versus Opus 4.8’s $1.94. Sonnet 5 also completed 6% fewer tasks (81% vs. 87%). Databricks attributed the higher token consumption to Sonnet reading more context and working longer per task to reach acceptable quality.
What is the “Pareto frontier” in AI coding agent evaluation?
The Pareto frontier in this context means the set of models that offer the best quality at each cost level — no model off the frontier gives you more quality or lower cost than a frontier model at the same trade-off point. Databricks found that the Pareto frontier spans models from multiple vendors, including open source, meaning a team optimising both quality and cost needs to use different models for different task types rather than standardising on one.
Databricks’ benchmark adds a real-production data point to the growing body of evidence that AI agents vary substantially by use case and deployment context. For a broader view of which AI agents perform across business workflows — not just coding — see our comparison of the best AI agents for business tasks. For AI coding assistant comparisons specifically, our AI coding assistant guide covers the tools most commonly deployed against production codebases.
For Context: The Databricks benchmark published on the same day as Microsoft’s TypeScript 7 release — see our coverage of TypeScript 7’s Go rewrite and 10x build speed improvement for the full picture of the July 8 developer tool landscape.
Last Updated: July 2026
