Your agent memory has a blind spot
Every agent memory system assumes similar queries retrieve similar memories. The InMind benchmark proves this is wrong: 84% accuracy with memory in context drops to 3% to 16% via retrieval. Here is the fix: a two-tier architecture with a modify-versus-inform router.
TL;DR: The InMind benchmark proves every agent memory system has a structural blind spot: facts that an agent recalls on demand (up to 100%) get applied under indirect queries only 3% to 16% of the time. The fix is a two-tier architecture. A write-time router separates facts that modify answers from facts that inform them. Modifiers stay visible. Informers stay retrievable.
Key takeaways:
- Query-conditioned retrieval assumes a needed memory will resemble the query that needs it. Implicit associations violate this assumption.
- The InMind benchmark (125 tasks, 10 life domains) separates three confusions: storage failure, missing model knowledge, and retrieval failure.
- Six memory systems recall direct facts at 76% to 100% and none applies them indirectly beyond 16%. Even 8x dimensionality embeddings leave the gap intact.
- An always-in-state probe recovers 68.8%, proving the bottleneck is the retrieval interface, not storage or reasoning.
- The open problem is routing: deciding which facts must stay visible instead of retrieved on demand.
The moment it breaks
I asked the agent for a macaron recipe. It served me one with enthusiasm and almond flour. Seconds earlier I had asked it what I am allergic to. It answered correctly: tree nuts.
Nothing was forgotten. The system recalled the allergy on demand moments before. What failed is structural: the memory was never brought to bear at the one moment it mattered. Standard macarons use almond flour. Almond is a tree nut. The two texts share no token, no topic, no embedding-space neighborhood. The retriever had no signal to connect them.
This isn’t a hypothetical (InMind paper, arXiv 2607.24368). HippoRAG 2, A-Mem, xMemory, and A-RAG all fail exactly this task in the benchmark I’m about to show you.
How agent memory works today
Long-term memory systems store what a user says in an external store and retrieve it when a related query arrives. The dominant design is retrieval-based memory. A writer distills what the user said into records. When a new query arrives, the system embeds it, retrieves the nearest records, and pastes them into the prompt.
The design has real virtues. It scales past the context window. It updates modularly. It inherits well-tested machinery from retrieval-augmented generation. And it works on direct questions, because “what am I allergic to?” is itself a retrieval cue.
The architecture looks like this:
User says "I'm allergic to tree nuts"
→ Writer creates record: ["User is allergic to tree nuts"]
→ Embedder stores it at position P in vector space
User asks "give me a macaron recipe"
→ Embedder encodes the query → finds nearest neighbors
→ Nearest neighbors are about French patisserie, not allergies
→ Retriever returns recipe context, not the allergy fact
→ Model answers with almond flourEvery route to the decisive memory crosses a semantic-similarity link. An implicit association is a pair that gives this link no signal.
What is the implicit-association blind spot
The paper calls it the retrieval hypothesis: any memory needed to answer a query lies close to that query under some computable relevance score.
Explicit recall satisfies the hypothesis. “What am I allergic to?” finds “tree nut allergy” because the query names the fact.
Implicit associations violate it. The macaron recipe and the tree-nut allergy share no lexical, embedding, or topical cue. What connects them is a fact about French patisserie stated in neither turn.
The paper formalizes three conditions for an implicit association:
- Necessity: the memory is required to answer the query safely or correctly
- Semantic distance: the memory and query sit far apart under common relevance functions
- Knowledge bridge: there exists background knowledge that connects them, stated in neither turn
The structure recurs across domains: carbamazepine and grapefruit via a labeled drug interaction, hearing impairment and restaurant choice via noise exposure, Halal rules and medicinal capsules via gelatin origin.
Why existing benchmarks miss it
When a system fails an indirect query, three explanations compete:
- The fact was never stored
- The model lacks the knowledge that bridges fact and query
- The fact was stored and never surfaced
Existing benchmarks can’t tell them apart. Each explanation calls for an entirely different fix. Storage failure means fix the writer. Missing knowledge means train or prompt better. Retrieval failure means fix the interface.
InMind is built to separate them.
How the InMind benchmark works
The benchmark has 125 expert-verified tasks across 10 life domains. 113 tasks are grounded in citable public sources: FDA allergen guidance, OSHA regulations, USCIS travel rules, CPSC recalls. Each task pairs a personal fact with an indirect query whose correct answer changes because of that fact.
Every task has three measurements:
- Naive query tests storage: “What am I allergic to?” If the system answers correctly, the fact was stored.
- In-context control tests bridging knowledge: place the fact directly in the prompt alongside the indirect query. If the model answers correctly, it has the world knowledge to connect them.
- Indirect query tests application: the real query without the fact in context. If the system fails here, the retrieval interface failed.
A fourth measurement, target recall, is an answer-blind check of whether the retrieved context contains the fact.
The injection protocol matters. Each memory is inserted into the middle of a 47-session conversation trace. The fact must survive 38 further sessions of ordinary interaction before it is ever queried. This isn’t a toy test. This is real interference.
The numbers
Six memory systems spanning vector, graph, and agentic designs. Two embedding sizes. One answer-blind check. Here is what happens:
| System | Direct recall | Target recall (indirect) | Indirect application |
|---|---|---|---|
| Backbone (memory in context) | — | 100.0% | 84.0% |
| Naive RAG (emb3-large) | 97.6% | 6.4% | 16.0% |
| HippoRAG 2 | 93.6% | 0.8% | 8.8% |
| A-Mem | 99.2% | 2.4% | 6.4% |
| A-RAG | 97.6% | 5.6% | 4.8% |
| xMemory | 84.8% | 3.2% | 4.8% |
| Mem0 | 76.0% | 2.4% | 3.2% |
The pattern is unmistakable. Every system recalls direct facts at 76% to 100%. No system applies them indirectly beyond 16%. The spread among systems is smaller than the distance separating all of them from the 84% backbone.
The paper tried 8x dimensionality embeddings (text-embedding-3-large at 3,072 dimensions instead of MiniLM’s 384). Target recall went up for every system. The application gap stayed intact. A bigger embedding space captures more semantic nuance but it doesn’t bridge implicit associations. The bottleneck is architecture, not representation.
What it takes to close the gap
The paper ran one more experiment. They kept the decisive memory visible in the prompt before the query arrived. No retrieval. Just a maintained profile that stays in context.
The always-in-state probe recovered 68.8%.
Not 84% (the ceiling). But dramatically above the 3% to 16% range every retrieval system hit. The gap between 68.8% and 84% is retrieval noise and imperfect profile management. The important number is that most of the blind spot disappears when you stop retrieving and start pushing.
This locates the failure precisely: it isn’t storage (direct recall works). It isn’t reasoning (the in-context model works). It is the query-conditioned retrieval interface itself.
The fix: two-tier memory architecture
The logical solution follows from the diagnostic. Partition memory by how it is accessed, not by what it contains.
Facts that modify change what correct means. A user’s tree-nut allergy makes every food recommendation after that point contingent on this fact. A recipe that is objectively correct becomes wrong. Leaving it out isn’t a quality loss. It is a safety failure.
Facts that inform add useful context but don’t flip correctness. The user prefers dark mode, uses VS Code, is building a RAG pipeline. Making the answer better without these facts is a quality improvement. Leaving them out is harmless.
Every current memory system treats all facts as informers. That is the blind spot.
The first tier is a profile. A compact set of 10 to 20 modifier facts pre-pended to every prompt. The profile exists before the query arrives. It doesn’t need to be retrieved. It is always visible.
The second tier is the retrieval store. Everything else. Informer facts, conversation history, project context, reference material. Retrieved on demand as they are today.
How the write-time router works
A classifier runs once when a fact is first stated or updated. It decides which tier the fact belongs in.
Fast path: lexical classifier. A set of signal patterns scored by domain.
| Signal | Weight | Example |
|---|---|---|
| Negation + noun | 0.9 | ”I don’t eat pork,” “I can’t have dairy” |
| Medical/dietary keywords | 0.85 | ”allergic,” “diabetic,” “hypertension” |
| Medication names | 0.85 | ”on warfarin,” “prescribed metformin” |
| Obligation/modality | 0.7 | ”must avoid,” “never eat,” “required” |
| Legal/regulatory | 0.8 | ”visa,” “NDA,” “HIPAA,” “work permit” |
| Domain keywords | varies | Health/food domains lower the profile threshold to 0.4 |
A score above the domain-adjusted threshold routes to profile. A score below 0.3 routes to store. Scores in between escalate to the LLM router.
LLM router for ambiguous cases. A small prompt that receives the user statement and the current profile and decides modifier or informer. The call fires once every 5 to 10 turns on average. The cost is negligible.
The router doesn’t need to be perfect. A router that correctly classifies 70% of modifiers into the profile and 30% into the store already beats every current system, where 0% of modifiers reach the profile. The false positive cost is token overhead. The false negative cost is serving almond flour to a nut-allergy user. Bias toward profile for anything uncertain.
The catch
The lexical classifier has blind spots of its own. It doesn’t handle sarcasm (“yeah I love anaphylaxis, great idea”). It doesn’t handle reported speech that isn’t about the user (“my friend has a peanut allergy”). It has limited coverage of non-English patterns out of the box.
The router also can’t solve eviction. Profile facts change. Allergies resolve. Medications change. The profile needs a decay and confirmation mechanism. A fact with a permanent TTL requires explicit contradiction to remove. A session-scoped fact (“I have not eaten today”) expires after a timeout.
The paper doesn’t propose a production system. It diagnoses the blind spot and releases InMind as a benchmark so that proposed solutions can be scored. The always-in-state probe is deliberately minimal: it exists to isolate the bottleneck, not to ship.
Why routing is the open problem
The retrieval hypothesis has been assumed but never tested. InMind tests it and finds it false for the cases where being wrong is expensive.
The open problem is routing. Not storing more. Not embedding better. Not retrieving faster. Deciding which facts must stay visible at any given moment.
Some facts are latent modifiers of every decision. Allergies, medications, dietary restrictions, legal constraints. They need to be pushed into context, not pulled. Current systems treat all memories as equally dormant until queried. That is wrong for a class of facts whose signal isn’t in their similarity to the query but in their consequence if missed.
Every agent builder has hit this. They didn’t have a name for it. Now InMind gives you the benchmark to test against, the numbers to know where you stand, and a clear architectural direction for the fix.
FAQ
Can I dump all facts into the profile? No. The profile must stay small (10 to 20 items) because every fact in it consumes prompt tokens on every turn. A profile of 50 facts at 500 tokens per fact adds 25,000 tokens per conversation. The router must be selective.
Does a bigger embedding model fix this? No. The paper tested 8x dimensionality embeddings (3,072 vs 384). Target recall improved for every system. The application gap stayed intact. The bottleneck is structural, not representational.
How do I handle profile eviction? The simplest approach is domain-weighted LRU. Health facts don’t get evicted by entertainment facts. Each domain has a minimum slot allocation. Facts with permanent TTL require explicit user contradiction to remove.
What about graph-based memory systems like HippoRAG 2? They hit the same wall. HippoRAG 2 scored 0.8% target recall and 8.8% application on InMind. Knowledge graphs help when the bridge sits in the graph and the query activates it. They don’t help when the bridge is world knowledge that lives in neither the stored text nor the graph structure.
Is the two-tier architecture production-ready? Not yet. The paper doesn’t ship a router. It diagnoses the blind spot and releases InMind as a benchmark. The architecture described here is a logical conclusion from the diagnostic, not a shipped system. Building and validating a router is the next step.
Related Posts
- Context engineering is your memory architecture the broader pattern of managing what stays in context versus what gets retrieved, and why prompt design alone can’t fix structural retrieval failures.
- Your agent’s memory is making it a yes-man memory introduces a different failure mode: sycophancy. The agent agrees with the user’s prior beliefs regardless of facts recalled.
- DeepSeek Engram: memory as reasoning primitives DeepSeek treats memory as executable reasoning traces instead of retrievable text, a different architectural approach to the same underlying problem.
This article was published on Agentic Up (https://agenticup.dev): practical guides for developers and founders building with AI agents. Reach me at [email protected].
