The harness, not the model, is your biggest cost lever
Databricks benchmarked coding agents on their multi-million line codebase. The same model cost 2x more on different harnesses with identical quality. The harness is your biggest cost lever.
TL;DR: Databricks published an internal coding agent benchmark on their multi-million line codebase. The headline finding was not which model won. It was that the same model cost 2x more on different harnesses with identical quality. The harness, not the model, is your biggest cost lever.
Key takeaways:
- Pi sent 3x less context per turn than Claude Code, producing 2x cost savings with the same task completion rate
- Sonnet 5 is 1.7x cheaper per token than Opus 4.8 but cost $2.09 per task versus Opus’s $1.94. It consumed 1.9x more tokens working through the same task
- GLM 5.2 statistically ties Opus 4.8 on quality at $1.28 per task versus $1.94
- The Pareto frontier for coding agents requires a mix of proprietary AND open source models. No single vendor covers it
- Every team with a merged PR backlog already has a coding agent benchmark. They have not built it yet
Why does the harness matter more than the model?
A coding agent harness is the software layer between you and the model. It decides what context to send each turn. It structures tool calls. It manages the conversation window and decides when to stop.
Developers treat the harness as a fixed cost. You pick Claude Code or Codex or Pi based on which model you want to use, and you assume the overhead is roughly the same across tools.
Databricks found it is not even close (Vinay Gaba, Ankit Mathur, Rishabh Singh, Patrick Wendell, Matei Zaharia, July 8 2026).
Running the same model through two different harnesses produced more than 2x cost differences while quality stayed identical. Pi sent about 3x less context per turn than Claude Code. It kept a tighter working set. It finished tasks in fewer runs.
Here’s the mechanism. Each turn, the harness sends the model a package of context: the system prompt, the conversation history, the current file contents, the tool definitions, and the output from the last tool call. A harness that sends 3x less context per turn burns fewer input tokens per turn. Over a 20-turn task, that difference compounds quickly.
The catch: Pi’s minimal system prompt means it performs better on well-specified prompts and worse on vague ones. One HN commenter from a company running an internal LLM proxy noted that Pi fails harder on poorly drafted prompts. Native harnesses like Claude Code are better at flailing along when the task is unclear. That’s a prompting discipline issue, not a tool limitation. But the right harness depends on how your team writes prompts.
Who benefits when your harness is inefficient?
Here’s the uncomfortable part. Native harnesses come from the companies that sell the models they call.
Claude Code generates more tokens per task than Pi for the same input. More tokens means more API revenue. Anthropic also recently changed their tokenizer so the same code input creates 30% more tokens.
This does not mean these companies are being malicious. It means their incentives align with making the harness and model work well together, not with minimizing your cost. When a company sells both the model and the harness, expect the harness to favor integration quality, not cost efficiency.
Why it works: separating the harness from the model provider breaks this incentive alignment. Pi comes from OpenClaw, which has no model to sell. Its only job is to call whatever model you choose as efficiently as possible. It sends 3x less context because it has nothing to gain from sending more.
The HN discussion picked up on this immediately. One commenter (Schlagbohrer) put it bluntly: “Anthropic is not beating the charges that they inflate token consumption with their own harness given these findings that Pi is 2.2x more efficient at token management. Big toothpaste ads tell you to use way too much toothpaste energy.”
What changes when you decouple the harness from the model?
The Databricks benchmark showed the Pareto frontier for coding tasks includes models from OpenAI, Anthropic, AND open source. No single vendor covers it. Developers who pick one vendor’s harness also lock themselves into that vendor’s model selection.
Decoupling gives you three levers instead of one.
First, model intelligence. Pick the right capability tier for each task type. Databricks found about 25% of engineering tasks are low complexity (flag flips, config updates) and about 60% are medium. Only about 15% need the frontier. Most teams default everything to the most expensive model because their harness makes switching painful.
Second, harness efficiency. Use a harness built for your prompting patterns and codebase structure. If your team writes precise prompts, Pi will save you money. If your team writes vague prompts, a more forgiving harness may be worth the overhead.
Third, routing. Send simple tasks to cheap models and complex ones to frontier models, all through the same harness. This is where the real savings live. Databricks is building this capability into Omnigent and Unity AI Gateway.
The catch: decoupling adds operational complexity. You manage multiple API keys. You track costs across providers. You test model-harness combinations to find which pairs work. That’s real work, and not every team has the infrastructure budget for it.
How do you build your own benchmark?
This is the reproducible part of the Databricks post. Any team with a merged PR backlog already has the dataset.
Step one: collect recent PRs from your team. Filter for human-written changes with good test suites. Keep the changes self-contained across a few modules. Include a distribution of your full stack.
Step two: rewrite each PR description as a prompt. State the problem and the constraints. Remove the solution details. If the original description explains why a bug fix is correct, strip that. You want the model to figure out the fix, not replicate it.
Step three: seal git history. The original commit is in the repo, and models with shell access will walk through git log to find it. Databricks discovered this when a few model scores looked too good to be true. The fix was cutting the working copy off from the repository entirely.
Step four: grade with real tests. The test suite the team wrote for that change is your ground truth. Do not use an LLM judge. Databricks found LLM judges reward sounding right over being right.
Step five: run every new model-harness combination through it. The pace of releases means last quarter’s ranking is stale. A benchmark built from your own PRs gives you a decision framework that no public leaderboard can match.
FAQ
What is the harness problem? Different agent harnesses send vastly different amounts of context per turn, creating 2x cost differences with the same model and task. The harness, not the model, is the primary cost variable.
Why did Sonnet 5 cost more per task than Opus 4.8? Sonnet 5 is cheaper per token but consumed 1.9x more tokens working through the same task. It needed more turns and more context per turn to reach the same answer.
Is GLM 5.2 as good as Opus 4.8 for coding? On Databricks’ benchmark, GLM 5.2 was statistically tied with Opus 4.8 on task completion rate at 66% of the cost ($1.28 per task versus $1.94). Internal developer feedback at Databricks confirmed the quality.
Should I switch everything to Pi? Not automatically. Pi is more efficient on well-specified prompts but struggles more with vague ones. The right harness depends on your team’s prompting habits and codebase structure.
How often should I re-benchmark? Every major model release and every harness update. The pace of improvement means last quarter’s ranking is stale.
Related Posts
- Cloudflare + Flue: the open agent harness stack. How Cloudflare turned Durable Objects into an agent platform and what that means for harness portability.
- DeepSeek V4’s hybrid attention changes your agent context budget. When the model compresses context more efficiently, the harness becomes the bottleneck.
- I tested 5 self-improving loop patterns. Only 2 survived production.. The agent harness design matters more than the model for production reliability too.
This article was published on Agentic Up (https://agenticup.dev): practical guides for developers and founders building with AI agents. Reach me at [email protected]
