Thinking is remembering: what Google's new paper reveals about reasoning in LLMs

TL;DR: Two Google papers prove that reasoning in LLMs is primarily a memory retrieval mechanism. Not a problem-solving tool. The computational buffer effect means extra tokens provide extended forward passes. Factual priming means surfacing related facts creates semantic stepping stones. Together, they explain why chain-of-thought helps even on simple factual questions , and why the reasoning tax is worth paying more often than you think.

I spent a year assuming reasoning tokens were for hard problems. Math. Code. Multi-hop logic. If a question was simple, I’d disable chain-of-thought to save tokens and latency. It felt efficient.

It was wrong.

Two papers from Google Research , published three months apart , prove that reasoning helps models recall facts they already know but cannot reach. Not through logic. Through two mechanisms that have nothing to do with problem decomposition.

This changes when and why you should use reasoning in your agents.

Key takeaways:

• Reasoning helps LLMs recall simple factual knowledge, not solve complex problems
• The computational buffer effect: extra reasoning tokens = extra forward passes to refine internal state
• Factual priming: generating related facts builds a semantic bridge before the target answer
• Hallucinated intermediate facts in reasoning traces degrade final answer accuracy
• Multi-trajectory selection with fact verification is the practical fix
• This contextualizes why Mixture of Agents works: reference models provide buffer + priming

The recall bottleneck

The first paper asks a simple question: when an LLM gets a fact wrong, does it not know the fact, or can it not reach it?

Calderon et al. call this “empty shelves or lost keys.” Empty shelves means the knowledge was never encoded in the weights. Lost keys means the knowledge is there but inaccessible.

Their findings are striking. Across 4 million responses from 13 LLMs on a new benchmark called WikiProfile, frontier models like GPT-5 and Gemini-3 encode 95 to 98 percent of facts. The error is not in encoding. It is in recall.

The failures are systematic. They disproportionately affect long-tail facts and reverse questions. The model knows the answer. It cannot find it in the moment.

This is the paper right before “Thinking to Recall” in the arXiv feed. It sets up the problem. The second paper provides the mechanism.

Tradeoff: The recall bottleneck matters for closed-book tasks where the agent must answer from memory. If you use RAG with robust retrieval, the bottleneck shifts to retrieval quality. This is not a universal problem , it is one you should know about when your model confidently misses an obvious fact.

Mechanism 1: The computational buffer

Gekhman et al. start with the observation that generating extra tokens gives the model more forward passes. Each forward pass is a chance to refine the internal state. This is the computational buffer hypothesis.

To test it, they design a brutal experiment. They intercept the model’s reasoning trace and replace it with the string “Let me think” repeated over and over until it matches the length of the original trace. No semantic content. No related facts. No logic. Just mechanical repetition.

The result: conditioning the model on this meaningless trace substantially improves recall compared to running with reasoning off.

The model does not need useful thoughts. It needs to think.

This is like a head chef who stands at the pass for 30 extra seconds before plating. They are not cooking during that time. They are scanning. Reorienting. Letting the mental map settle. When they finally plate, it is more accurate than if they rushed.

The computational buffer has limits. Pushing the dummy text to longer lengths produces diminishing returns. The natural reasoning trace still outperforms the dummy text. Content matters , but the compute itself is doing real work.

Kitchen analogy: The head chef at the pass. The head chef does not cook. They orchestrate. When the ticket comes in, they pause, scan the station, remember where everything is, then call the fire. The pause is not wasted time. It is retrieval time. The model’s reasoning tokens are that pause.

Mechanism 2: Factual priming

When the researchers analyze the natural reasoning traces, a different pattern emerges. The models are not writing logical proofs for simple factual questions. They are surfacing related facts.

Ask “Who was the 10th King of Nepal?” and the reasoning trace lists the first nine kings before naming the 10th. The first nine are not steps in a deduction. They are a semantic warm-up.

In human cognition, psychologists call this spreading activation. Processing one concept primes related concepts in semantic memory, making them easier to retrieve. Gekhman et al. show that LLMs exhibit a parallel mechanism, which they call factual priming.

To isolate this effect, they extract only the concrete facts from the reasoning traces , stripping filler text, search plans, and explicit mentions of the target answer. Then they condition the model on that short fact list. It recovers most of reasoning’s gains. And it helps even when reasoning turns off.

The facts themselves are the stepping stones.

Tradeoff: Factual priming is effective but fragile. The model generates intermediate facts without verification. If even one of those facts is incorrect, the final answer degrades significantly. The priming mechanism becomes a liability when the model builds a semantic bridge to the wrong destination.

The hallucination trap

Gekhman et al. build a large-scale auditing pipeline. They run a search-enabled verifier across hundreds of thousands of reasoning traces, checking every single intermediate fact for correctness.

The pattern is clear: if a reasoning trace contains even one hallucinated intermediate fact, the probability of a correct final answer drops sharply.

This explains why the content of reasoning still matters. The computational buffer gives the model more compute. But bad content poisons that compute. The buffer amplifies the signal, whether the signal is correct or hallucinated.

The fix the paper proposes is multi-trajectory selection. Generate multiple reasoning paths for the same question. Keep only the ones with factually verified intermediate steps. Prioritize those trajectories during decoding.

This is the same insight behind process reward models. Reward verifiable intermediate steps, not only final answers. The route matters as much as the destination.

What this means for agents

These two papers change how you should think about reasoning in agent systems.

First, chain-of-thought is not optional for factual recall, even on trivial questions. The reasoning tax (extra tokens, latency) buys you something real: access to knowledge the model already has. Disabling reasoning on “simple” queries is leaving recall on the table.

Second, the computational buffer effect provides a clean explanation for why Mixture of Agents works. Reference models in MoA generate their analysis traces. Those traces give the aggregator extra forward passes to refine its state , the buffer effect , plus additional facts that prime the correct answer , the priming effect. You do not need the reference models to be smarter than the aggregator. You need them to add compute and context.

Third, the hallucination trap means you should not blindly trust a single reasoning trajectory. If you can run multiple trajectories and select the factually clean one, you should. This is tractable for agent systems that can afford a second pass.

FAQ

Does this mean I should always enable reasoning? Not always. The gains are largest when the model needs to recall facts it has encoded but cannot reach on the first pass. For fully retrieved contexts (RAG with comprehensive results), the buffer effect adds latency without proportional benefit.

How many reasoning tokens are optimal? The paper shows diminishing returns beyond a certain point. The optimal length depends on the model and the question. A short reasoning trace (50 to 100 tokens) captures most of the buffer benefit. The priming benefit needs enough space to surface related facts but not so much that hallucination risk accumulates.

Does this apply to reasoning models like o3 or Gemini 2.5 Pro? Yes , these are the models tested in the paper. The effect is strongest for models that can toggle reasoning on and off, precisely because you can compare the same architecture with and without reasoning enabled.

How does this connect to the Vertical Agent Method? The Vertical Agent Method picks one workflow and ships one agent in 14 days. If that workflow involves fact recall , customer lookup, entity resolution, knowledge retrieval , enabling reasoning in the agent loop directly improves accuracy without changing the architecture.

Related Posts

• The Vertical Agent Method: ship AI agents in 14 days
• How to build model-agnostic agents that survive a provider shutdown

---

References

• Gekhman, Z., Aharoni, R., Ofek, E., Geva, M., Reichart, R., & Herzig, J. (2026). Thinking to Recall: How Reasoning Unlocks Parametric Knowledge in LLMs. arXiv:2603.09906.
• Calderon, N., Ben-David, E., Gekhman, Z., Ofek, E., & Yona, G. (2026). Empty Shelves or Lost Keys? Recall Is the Bottleneck for Parametric Factuality. arXiv:2602.14080.
• Wei, J., et al. (2022). Chain-of-Thought Prompting Elicits Reasoning in Large Language Models. The original CoT paper that established step-by-step reasoning for complex tasks.

---

This article was published on Agentic Up (https://agenticup.dev): practical guides for developers and founders building with AI agents. Reach me at [email protected]

---