Better Models, Worse Tools

TL;DR: Armin Ronacher (creator of Flask, building Pi) found that newer Claude models invent fake keys in non-Claude Code edit tools. Opus 4.8 and Sonnet 5 add fields like requireUnique, oldText2, and in_file to nested edit arrays. Even when the schema does not define them. The root cause: RL post-training optimized the model for Claude Code’s forgiving harness. The better the model gets at Claude Code, the worse it gets at every other tool shape.

Key takeaways:

• Opus 4.8 and Sonnet 5 invent fake keys in Pi’s nested edits[] array. Keys like requireUnique, oldText2, in_file, and cost. Even though the actual edit content is byte-correct.
• The regression comes from RL training that optimized the model for Claude Code’s flat edit schema. Claude Code’s harness silently filters unknown keys, so the model learned that extra keys are tolerated.
• OpenAI’s models avoid this with grammar-constrained sampling. The <|constrain|>json marker switches the sampler to mode where invalid keys cannot be generated.
• Anthropic has a strict mode that does the same, but complexity limits on tool definitions make it unusable for nested schemas.
• The implication: if you are building a non-Claude Code harness, newer Anthropic models are less reliable for your tool shapes, not more.

---

I spent last year assuming that a tool schema is an abstract contract and the model is a general reasoner that follows it lucumr.pocoo.org.

Armin Ronacher proved that assumption wrong.

He found it building Pi. An open-source agent harness. Pi’s edit tool supports multiple replacements in one call using a nested edits[] array. Newer Claude models started calling it with extra keys that do not exist in the schema. Not Haiku or a small model. Opus 4.8.

The edit content was byte-correct. The model had figured out exactly what to change. Then it appended garbage at the end.

How tool calls work

Tool calls are not objects passed between the model and the API. They are text.

When you send a tool schema to the Anthropic API, the server converts it into text that gets inserted into the prompt. That text happens to look like XML. The model reads it as text, understands what tools are available, and later writes text that looks like XML back. The API parses that text and returns structured data to you.

The model never sees JSON objects. It only sees tokens.

For a simple parameter like a file path, the model writes inline text inside an XML tag. For a nested array like Pi’s edits[], the model must write JSON inside XML inside a tool call. The highest-entropy point is after closing a multi-line string. The model must decide whether to write } or , "...".

Without grammar enforcement, the model can output any token it wants at that point. And its RL training has shaped what it will pick.

What the failure looks like

Pi’s edit tool expects this:

{
  "path": "some/file.py",
  "edits": [
    {"oldText": "text to replace", "newText": "replacement text"}
  ]
}

The model sometimes produces this instead:

{
  "path": "some/file.py",
  "edits": [
    {
      "oldText": "text to replace",
      "newText": "replacement text",
      "requireUnique": true
    }
  ]
}

requireUnique is not in Pi’s schema. The model invented it.

Across repeated trials, Armin saw a whole zoo of invented trailing keys: type, id, kind, unique, requireUnique, matchCase, in_file, forceMatchCount, children, notes, cost, oldText2, newText2, oldText_2, newText_2, and even event.0.additionalProperties inside the edit object itself.

The most frustrating part: the actual oldText and newText payloads were byte-correct in every invalid call inspected. The model wrote the right edit and then added nonsense at the end. Armin’s collaborator Petr Baudis found that in their sessions, Opus 4.8 failed around 20% of the time. Stripping thinking blocks from the history cut the failure rate in half simonwillison.net.

Why this is getting worse, not better

Opus 4.5 did not have this problem. It adapted to different tool schemas well. Armin wrote: “I was pretty convinced that we are on a good path where the models are more likely to adapt to any sort of tool shape.”

The regression appeared with Opus 4.8 and Sonnet 5. The newer models are worse at this specific failure mode, not better.

Here is the mechanism:

Claude Code’s edit schema is flat. file_path, old_string, new_string, and an optional replace_all flag. There is no nested array. Looking at Claude Code’s minified client code reveals retry paths for malformed tool use, parameter aliases, type coercions, Unicode repairs, and filtering of unknown keys. The client expects and accepts a fair amount of slop and repairs it silently.

When RL post-training optimizes the model in this environment, slightly malformed tool calls still complete the task and receive reward. The harness fully absorbs the error. There is little gradient against inventing an alias or adding a stray field.

Worse, the model becomes strongly adapted to Claude Code’s canonical edit tool shape. A different harness presents a tool with the same semantic intent but a different schema. That tool is now off-distribution. The better-trained model fights you harder because its prior is stronger.

Opus 4.8 and Sonnet 5 have learned that an edit operation may have one extra optional field. Under Pi’s nested oldText/newText shape, there is no trained name for that field. So the model samples a plausible name each time, producing dozens of random keys rather than one stable alias.

The catch: Anthropic has the fix but cannot use it

Anthropic offers a strict mode where the sampler refuses to generate keys not in the schema. This is the same mechanism OpenAI uses. Grammar-constrained sampling that blocks invalid tokens.

But strict mode has complexity limits on tool definitions. If your schema has nested arrays or too many parameters, the API errors out. Pi’s nested edits[] exceeds the limit. Claude Code does not use strict mode either.

So the constraint exists but is gated behind a limit that excludes the exact use case where you need it.

OpenAI’s harmony format avoids this entirely. They use a <|constrain|>json marker in the text that tells the sampler to switch to JSON grammar mode. The marker is part of the transcript, so the inference stack knows exactly when to constrain. The model physically cannot output requireUnique because the grammar blocks it. Codex models do not show this regression.

Why every harness pays the cost

The uncomfortable lesson: tool schemas are not neutral, at least on Anthropic models.

We like to pretend that a schema is an abstract contract and the model is a general reasoner that follows it. That was true for Opus 4.5. It is increasingly false for Opus 4.8 and Sonnet 5.

Tool schemas are somewhere in the distribution. Some shapes are close to what the model saw during post-training. Some are far. Some are easy for the provider’s hidden encoding (top-level attributes in the ANTML format). Some require the model to write large escaped JSON objects inside nested arrays after long multiline strings. The model can understand the schema and still be bad at sampling the exact shape under pressure.

Claude Code is closed source. The RL environment that shaped these priors is not documented. We cannot assume Claude-Code-trained behavior will transfer cleanly to your tools unless they are a close match. The more post-training happens inside one dominant harness, the more every other harness will have to inherit its quirks.

FAQ

What is the “Better Models: Worse Tools” problem? Newer Anthropic models (Opus 4.8, Sonnet 5) sometimes invent fake keys in tool calls when the tool schema differs from Claude Code’s. The edit content is correct, but the model appends extra keys that the schema does not define.

Why does this happen? RL post-training optimized the model for Claude Code’s specific tool schema. Claude Code’s harness silently filters unknown keys. So the model learned that extra keys are tolerated and keeps adding them.

Does this affect all tool calls? No. It only happens with tool schemas that differ from Claude Code’s flat edit format. Tools with nested arrays or non-standard parameter names are more likely to trigger it. Claude Code’s own tools are not affected because the harness cleans up the slop.

How does OpenAI handle this differently? OpenAI’s harmony format uses a <|constrain|>json marker that switches the sampler to grammar-constrained mode. The model physically cannot output tokens that violate the JSON schema.

How should harness builders handle this? Use strict mode if your schema is simple enough. For complex schemas, add a post-processing layer that filters unknown keys, or switch to grammar-constrained sampling on your end.

Related Posts

• Your agent needs policy gates, not just prompts. Why prompt-based guardrails fail for agent behavior and policy-based gates fix it.
• The 15 jobs of an agent harness. What a production agent harness needs to handle. Including tool validation and error recovery.
• Agents break in production and nobody talks about it. Real failure modes from production agent deployments, including silent tool errors.
• Cloudflare’s Flue: the agent harness that changed how I think about state. How different harnesses handle state, retries, and tool validation.

---

This article was published on Agentic Up (https://agenticup.dev): practical guides for developers and founders building with AI agents. Reach me at [email protected]