Your agent wastes money on three things. Here is how to fix each one.
Every AI agent pays the same computation three times: prompt KV cache, tool schemas, and redundant tool calls. Here is how to fix all three layers with concrete patterns.
TL;DR: Every agent turn pays the same computation three times. The KV cache for the system prompt. The full schemas of every MCP tool. Repeated calls to the same APIs. Three research papers from 2026 prove each layer is independently fixable. Together they cut costs by 70-90% and latency by 40-60%.
Key takeaways:
- KV/prompt caching (provider side) gives 40-80% cost reduction. Cache the system prompt only. Put dynamic content after the breakpoint.
- Schema-level caching (middleware) gives 95% reduction in tool schema tokens. Keep compact summaries in context. Load full schemas only for top-k matched tools.
- Tool result caching (application side) gives 11-34% latency reduction. Cache outputs of read-only tool calls. Skip redundant API execution.
- Pi (the coding agent) already implements layers 1 and 2 natively. You can copy the same patterns.
- All three layers stack. Each is independently useful. None requires changing your tools.
What is the problem you are paying for three times?
I spent a month watching my agent’s API bills grow without understanding why. The model responses were fast. The tool calls were working. But the token count kept climbing on every turn.
The reason became obvious once I traced it. Every turn pays for the same three things:
- The KV cache for the system prompt and tool definitions gets recomputed on every request (or busted by dynamic content)
- Every MCP tool schema gets re-serialized and re-injected into the prompt on every turn
- The same API calls (weather lookups, database queries, git status) execute over and over with identical parameters
Think of it like a restaurant kitchen. Every time a waiter puts in an order, the chef reads the entire recipe book from scratch, checks every ingredient in the pantry, and boils a fresh pot of water even if the last order asked for the same thing.
This post gives you three fixes. Each one targets one layer. Each one works independently. Together they cut the waste.
How does KV/prompt caching work and why does it fail in agents?
KV caching is built into every LLM provider. When you send a request, the model computes attention tensors for every token in your prompt. Those tensors are cached in the KV cache. On the next request, if the prompt prefix matches, the provider skips recomputing the cached tokens. You pay 10x less for cached input tokens.
The catch: The KV cache only works if the prompt prefix stays the same. In agentic sessions, every turn appends tool calls, tool results, and conversation history to the prompt. If your system prompt contains dynamic content like timestamps, session IDs, or user-specific values, the cache busts on every request.
A January 2026 paper from PWC called “Don’t Break the Cache” tested this across OpenAI, Anthropic, and Google on 500 agent sessions. The results:
| Model | Best cache mode | Cost reduction | TTFT improvement |
|---|---|---|---|
| GPT-5.2 | Exclude tool results | 79.6% | 13.0% |
| Claude Sonnet 4.5 | System prompt only | 78.5% | 22.9% |
| Gemini 2.5 Pro | System prompt only | 41.4% | 6.1% |
| GPT-4o | System prompt only | 45.9% | 30.9% |
The key finding was that full-context caching can make latency worse. On GPT-4o, naive full-context caching was 8.8% slower than no caching at all. The reason: caching dynamic tool results triggers cache writes that add overhead without any reuse benefit.
The fix: Cache the system prompt only. Put all stable content (tool definitions, persona, instructions) at the start of the prompt. Put all dynamic content (user messages, tool results, conversation history) after a cache breakpoint. Never include timestamps or session IDs in the system prompt.
Pi does this natively. Its anthropic-messages API adds cache_control: { type: "ephemeral" } to every system prompt block and to the last tool definition, creating a clean breakpoint between cached and dynamic content. You can set PI_CACHE_RETENTION=long to extend the TTL to 1 hour.
What is the MCP Tools Tax and how do you eliminate it?
Every MCP server re-serializes its full tool schemas on every conversational turn. The Model Context Protocol is stateless, so the client has to re-send every tool definition on every request.
Real measurements from production MCP deployments show the damage:
| Deployment | Tools | Tokens per turn for schemas only |
|---|---|---|
| Typical 4-server MCP host | 40-60 | 15,000-20,000 |
| GitHub MCP suite (full) | 93 | ~55,000 |
| Enterprise database catalog | 106 | ~54,600 |
That is 55,000 tokens per turn before the model does any reasoning. Over a 30-turn session, that is 1.42 million tokens spent on tool definitions that never change between turns.
An April 2026 paper called “Tool Attention Is All You Need” quantifies the problem and proposes a fix. The authors call it the Tools Tax.
The fix: Replace eager schema injection with a two-phase lazy loader.
Phase 1: Keep compact summaries (60 tokens each) of every tool in context. These give the model awareness that tools exist. They cost ~4,800 tokens total for 120 tools and they are stable across turns, so they earn full KV cache credit.
Phase 2: When the model needs a tool, it calls a search/loader tool. The middleware promotes the full JSON schema for only the top-k matched tools (k=10). The promoted schemas carry full type information and are placed after the cache breakpoint.
The paper reports 95% reduction in per-turn tool tokens (from 47,300 to 2,368) and context utilization going from 24% to 91%. Task success improved from 72% to 94% because the model was not drowning in irrelevant schemas.
Why this improves reasoning quality too: The paper found a reasoning-quality cliff when context utilization crosses 70%. At 24% utilization (naive MCP), the model had to process 3 tokens of noise for every 1 token of useful content. At 91% (Tool Attention), almost every token served the task.
Pi implements this pattern through its extension system. Tools registered via pi.registerTool() are not automatically active. The model gets a small initial set. It can call search_tools to find and activate more. On supported Anthropic models, activated tools use native deferred loading (tool_reference content blocks) that inject schemas at the tool-result position without touching the cached prefix.
What you would build: A middleware wrapper for MCP servers that precomputes tool embeddings, keeps summaries in context, and promotes full schemas only on demand. No changes needed to your tools or your agent framework. The reference implementation is open source at github.com/asadani/tool-attention.
How do you cache tool results instead of calling the same API twice?
Even after fixing layers 1 and 2, your agent still makes the same tool calls repeatedly. Over 40% of tool invocations in real workloads are duplicates. Identical calls with identical parameters.
A January 2026 paper called “ToolCaching” from Southeast University measured this and built a solution.
The fix: Cache the output of tool calls keyed by hash(tool_name + JSON.stringify(params)). On the next identical call within the TTL, return the cached result. No API call. No latency. No cost.
The hard part is deciding what to cache. Not all tool calls are cacheable:
- READ calls (weather lookup, database query, git status) can be cached
- WRITE calls (send email, delete file, transfer money) MUST NOT be cached
Why it works: The paper uses an LLM to classify each tool call. It achieves 98% accuracy on distinguishing READ vs WRITE calls and 86% accuracy on suggesting appropriate TTLs. Per-tool TTLs handle the freshness problem. 60 seconds for weather, 300 seconds for computations, 3600 seconds for static knowledge.
Simple caching (LRU with TTL) gives solid gains. The paper’s VAAC algorithm adds multi-armed bandits for admission control and value-weighted eviction for an extra 11% hit ratio and 34% latency reduction.
| Strategy | Hit ratio improvement | Latency reduction |
|---|---|---|
| Simple LRU + TTL | baseline | baseline |
| VAAC (bandit admission + value eviction) | +11% | +34% |
| Plus user-based grouping | +21% more | +7% |
What you would build: A thin proxy between your agent and its tools. On every tool call, check the cache. READ calls with cache hits return instantly. READ calls with cache misses execute the tool, save the result, and return it. WRITE calls pass through unconditionally. That is an afternoon of work for the first 80% of the gain.
How do these three layers stack?
Each layer targets a different source of waste. None requires changes to the others.
| Layer | What it caches | Typical reduction | Effort to implement |
|---|---|---|---|
| KV/prompt | Attention tensors for system prompt | 40-80% cost | Configure cache breakpoints |
| Tool schema | Tool definitions (summaries in, full on demand) | 95% token reduction | Middleware wrapper |
| Tool result | Outputs of READ tool calls | 11-34% latency | Proxy layer |
Combined on a typical MCP deployment with 40-60 tools across 4 servers, the expected outcome is 70-90% cost reduction and 40-60% latency improvement.
The cache breakpoint strategy from layer 1 makes layer 2 more effective (stable summaries earn KV cache credit). Layer 3 gives the most visible user-facing improvement (responses feel faster because duplicate tool calls skip the network).
What can you configure today without building anything?
If you use Pi, layers 1 and 2 are already active. Set PI_CACHE_RETENTION=long in your environment to extend the KV cache TTL to 1 hour. Pi already applies cache_control markers to the system prompt and the last tool definition. Its extension system with setActiveTools() already implements the lazy loading pattern from Tool Attention.
If you use another agent framework or direct API calls:
- Structure your system prompt so all stable content comes first. Put timestamps, session IDs, and user-specific values at the end or in the first user message.
- If you use Anthropic, add
cache_control: { type: "ephemeral" }to your system prompt and the last tool definition. The cost is zero. The savings start immediately. - If you use MCP, keep your server count under 4 and your tool count under 50. Above that, the Tools Tax starts degrading reasoning quality regardless of cost.
- Identify your most-called READ tools (weather, search, database queries) and add a simple in-memory cache with TTL. A Python
dictkeyed by(tool_name, frozenset(params.items()))withtime.time()expiry checks is enough for the first pass.
Three independent caching layers target the three sources of token waste in every agent turn. KV caching saves the provider prefix. Schema caching saves the tool overhead. Result caching saves redundant execution. Each layer is independently useful and they stack multiplicatively.
FAQ
What are the three caching layers for AI agents? KV/prompt caching caches the attention computation for the stable system prompt prefix. Schema-level caching keeps compact tool summaries in context and loads full schemas on demand. Tool result caching stores outputs of read-only tool calls to avoid redundant API execution.
Which caching layer gives the biggest cost savings? KV/prompt caching gives the largest single reduction at 40-80% cost savings. But the layers stack. Combining all three can bring per-turn costs down by 90% or more compared to naive MCP tool injection.
Is this only for Anthropic models? No. All three caching strategies work across providers. KV/prompt caching is available on OpenAI, Anthropic, and Google. Schema-level caching and tool result caching work at the application layer and are model-agnostic.
Do I need to modify my MCP servers to use these patterns? No. Schema-level caching is a middleware wrapper that sits between your agent and the MCP servers. Tool result caching is a proxy layer. Neither requires changes to the tool implementations.
How much can I realistically save? Combining all three layers on a typical MCP deployment with 40-60 tools across 4 servers, expect 70-90% cost reduction and 40-60% latency improvement based on published benchmarks.
Related Posts
- Your skill router is decomposing wrong. SkillWeaver proves granular decomposition beats retrieval. A different approach to the same context bloat problem from the skill library side.
- Context engineering for production AI agents. Four strategies for managing what your agent remembers, forgets, and never sees. KV cache management is one of them.
- Your MCP server is lying to your model. Tool poisoning attacks hide malicious instructions in tool descriptions. Reducing the number of in-context schemas has defensive benefits too.
This article was published on Agentic Up (https://agenticup.dev): practical guides for developers and founders building with AI agents. Reach me at [email protected]
