Cache your agent's reasoning, not its responses
Monolithic prompt caching caps at 38.7% hit rate. PMG's SemanticALLI caches intermediate reasoning instead, hitting 83.1% and cutting 78.4% of token costs.
TL;DR: PMG (a marketing intelligence platform) deployed SemanticALLI, a pipeline-aware caching system that caches intermediate reasoning artifacts instead of final responses. Monolithic prompt→output caching caps at 38.7% hit rate. Their structured approach hits 83.10% on the downstream synthesis stage, bypassing 4,023 LLM calls with 2.66ms median latency. Total token reduction: 78.4%. The key insight: even when users never repeat themselves, the pipeline often does.
Key takeaways:
- Monolithic prompt→output caching can’t recover from misses. 61.3% of requests regenerate from scratch
- Pipeline-aware caching at intermediate checkpoints hits 83.10% reuse on synthesis, 38.7% on intent resolution
- Hybrid retrieval (exact + dense + BM25) prevents entity-level collisions that pure semantic caching misses
- 78.4% total token reduction on a production workload of 1,000 prompts
- The principle transfers: cache where the pipeline repeats, not where the user does
The standard advice for reducing LLM costs is simple: cache the responses. Same prompt, same answer, skip the API call. Semantic caches like GPTCache, InstCache, and Asteria extend this to similar prompts via embedding similarity.
It’s good advice. It’s also incomplete.
PMG’s production system proves why. Their BI agent processes natural language into dashboards. Schema inspection, intent resolution, metric selection, chart code generation. Users ask for “executive summary with top KPIs” or “show media channel performance.” The prompts are novel. The pipeline underneath? It repeats constantly.
Monolithic caching hits a wall
PMG tested standard prompt→output caching on 500 production prompts. At a similarity threshold of 0.90:
| Metric | Value |
|---|---|
| Exact hits | 0 (0.0%) |
| Semantic hits | 194 (38.7%) |
| Misses | 306 (61.3%) |
The 38.7% hit rate is respectable for a semantic cache. But the 61.3% miss rate is the real story. Every miss regenerates the entire pipeline from scratch. No partial reuse. No amortization of the common sub-steps. Just a cold-start generation path.
The brittleness is structural. Two prompts like “Show DDA Revenue by channel” and “Show GA4 Revenue by channel” have a cosine similarity of 0.96, but they reference different attribution models with incompatible metric definitions. A semantic cache treats them as the same. A lexical-only cache treats them as completely different. Neither is right.
The fix: cache where the pipeline repeats
SemanticALLI decomposes the agent pipeline into two stages and caches each independently:
Stage 1: Analytic Intent Resolution (AIR). Maps natural language to a structured intent: metrics, dimensions, filters, temporal grain, chart type. This is the semantic bottleneck: novel prompts rarely match existing intents exactly.
Stage 2: Visualization Synthesis (VS). Generates chart code, dashboard directives, and rendering instructions from the structured intent. This is where reuse lives: once an intent is resolved, the same chart types, layout primitives, and encoding patterns recur across different users and queries.
The caching architecture uses three retrieval tiers:
- Exact hash (SHA-256): For deterministic duplicates (rare but free)
- Dense semantic (HNSW, text-embedding-3-large, 3072d): Broad recall
- BM25 lexical: Entity-level precision. Catches mismatches like DDA vs GA4 that pure embedding similarity misses
Reciprocal Rank Fusion (RRF) combines the dense and lexical scores. The result: recall when phrasing varies, precision when entities differ.
The numbers
| Stage | Invocations | Exact hits | Semantic hits | LLM calls | Avg tokens |
|---|---|---|---|---|---|
| AIR | 500 | 0 (0%) | 194 (38.7%) | 306 | 3,926 |
| VS | 4,841 | 4,023 (83.1%) | 0 (0%) | 818 | 934 |
| Total | 5,341 | 4,023 (75.3%) | 194 (38.7%) | 1,124 | 1,214 |
The asymmetry is the point. AIR is hard: mapping open-ended language to canonical intent. VS is easy: structured intent maps to finite chart templates.
The 4,023 exact VS hits bypassed LLM generation entirely at 2.66ms median latency per hit. Without SemanticALLI, generating those 4,023 invocations would cost 5,525 tokens each. With caching, they cost effectively zero.
End-to-end token projection: 59,906 tokens per user prompt without caching vs 12,964 with caching. That’s a 78.4% reduction.
The entity problem (and why it matters)
Semantic caching’s blind spot is business-critical entities. Two requests that differ by one metric name are semantically near-identical in embedding space but operationally incompatible. This is not a corner case. It’s the default in enterprise systems.
PMG’s hybrid retrieval solves this with BM25 scoring over schema tokens. The lexical layer requires matching mandatory terms (metric names, dimension names, attribution model identifiers). Even when the dense embedding suggests a strong match, the lexical gate rejects candidates that miss critical entities.
This is the pattern to steal: semantic recall for phrasing variance, lexical precision for entity safety.
When this doesn’t work
Pipeline-aware caching assumes stable intermediate representations. If your agent pipeline produces unstructured free text at every step with no structured intents or finite output templates, there’s nothing stable to cache.
It also requires you to decompose the pipeline into stages with well-defined input-output contracts. If your agent is a single monolithic prompt that does everything, you can’t insert cache checkpoints.
And the AIR stage remains the bottleneck. At τ=0.90, AIR still misses 61.3% of requests. Better intent canonicalization, schema-grounded representations, and domain-specific metric alias resolution could improve this. But intent resolution is genuinely hard.
What to steal from this
The authors identify their approach as “internal caching”: cache artifacts inside the workflow, not at the perimeter. The principle transfers to any multi-step agentic pipeline that produces structured intermediates:
- Coding agents: Cache compiled code, test outputs, dependency resolutions
- RAG pipelines: Cache retrieved chunks, reranking scores, fused results
- Planning agents: Cache subgoal decompositions, resource allocations
- Data pipelines: Cache converted schemas, intermediate DataFrames
The decision rule is simple: profile your pipeline. Find the step that produces the same output for different inputs. Cache at that step. Don’t wait for the user to repeat themselves. The pipeline already does.
FAQ
Is this semantic caching in disguise? No. Semantic caching still operates at the prompt→output boundary. SemanticALLI moves caching inside the pipeline, targeting intermediate artifacts. The difference is architectural: where the cache sits, not how retrieval works.
Doesn’t Agentic Plan Caching or InstCache already do this? Those systems cache execution plans or attention states. SemanticALLI caches structured intermediate reasoning artifacts: the outputs of specific pipeline stages. The closest prior work is plan caching, but SemanticALLI is more general (any structured intermediate) and operates at finer granularity (stage-level, not plan-level).
What similarity threshold should I use? PMG uses τ=0.90 for strict correctness and τ=0.85 for latency-sensitive workloads. The tradeoff is real: lowering τ from 0.90 to 0.85 cut median runtime from 57s to 25s, but increases collision risk on entity-level mismatches.
Is the code available? SemanticALLI is proprietary to PMG. But the architecture (two-stage decomposition, hybrid retrieval with RRF, stage-level caching) is straightforward to implement with open-source tools.
Related Posts
- Context engineering and memory architecture for AI agents. Complementary techniques for managing what your agent sees and reuses.
- The 2026 LLM architecture landscape. Understanding model design helps predict where caching is most effective.
- Loop engineering: production agent loops. When agents iterate, the redundancy compounds. That is exactly where internal caching pays off.
This article was published on Agentic Up (https://agenticup.dev): practical guides for developers and founders building with AI agents. Reach me at [email protected]
