--- title: Why your self-improving agent gets worse over time (and how to fix it) canonical: "https://agenticup.dev/posts/experience-internalization-collapse/" pubDate: "2026-07-03T00:00:00.000Z" description: "A new paper proves that most experience internalization methods collapse under repeated iteration. Three fixes: use abstract principles not specific logs, inject experience at the right decision step not all at once, and learn from successful demonstrations not corrected mistakes." tags: [experience-internalization, self-improving-agents, continual-learning, llm-agents, agent-memory] --- **TL;DR:** A new paper from Renmin University and Meituan proves what happens when self-improving agents try to learn from their own experience. They get worse. Each iteration degrades performance. The paper identifies three fixes (principle-level memory, step-wise injection, and off-policy training) and shows that together they sustain improvement across multiple iterations. I read the paper and broke down what each one means in practice. > **Key takeaways:** > - Instance-level experience (specific URLs, numbers, query strings) produces transient gains that collapse by iteration 3. Principle-level experience (reusable strategies) sustains improvement. > - Global injection causes the agent to answer without searching 63% of the time. Step-wise injection reduces this to 0% by aligning experience with the current decision state. > - On-policy training creates a negative spiral where the student generates bad trajectories and the teacher only applies local fixes. Off-policy training on successful demonstrations breaks this cycle. > - The three fixes are independent and complementary. Each one solves a failure mode the others do not address. > - The winning recipe: principle-level + step-wise + off-policy. Tested on Qwen3-4B and 8B across web reasoning benchmarks. --- I published a post earlier today about self-improving loop patterns. I tested five. Only two survived production. A reader wrote in and said: "You talked about why loops fail but not why they degrade over time. I have a loop that works on day one and fails on day thirty." That reader was right. The degradation problem is different from the convergence problem. A loop can converge on the right behavior initially and still get worse as it accumulates experience. This happens because the agent is learning from its own experience, and the way it learns introduces hidden failure modes. A new paper from Renmin University and Meituan ([arxiv 2606.04703](https://arxiv.org/abs/2606.04703)) explains exactly why this happens and how to fix it. ## What is experience internalization and why does it matter? Here is the core problem: your agent accumulates experience over time. It solves a task, learns from the solution, and uses that learning on the next task. This should compound. Each task should make the next one easier. In practice, it does not compound. It degrades. The paper calls this "progressive capability collapse." Figure 1 shows it clearly: every benchmark drops after iteration 1. By iteration 3, performance is below the baseline model that never learned anything. The authors identify three dimensions that cause this collapse. Each one is a design choice you make when building your agent's learning loop. Each one can either sustain or destroy your agent's ability to improve over time. ## Dimension 1: Experience Granularity Here is how it works: when your agent finishes a task, you capture what happened and store it for future use. But what exactly do you store? **Instance-level experience** preserves the specific details of the run. The exact query, the URLs visited, the numbers found, the sequence of tool calls. It is a detailed log of one specific event. The paper analyzed what instance-level experience actually contains. 74.4% of items contain specific URLs or domain names. 57.3% contain concrete numbers. 93.9% contain query-specific or entity-specific strings. **Principle-level experience** extracts the reusable strategy from the run. Not "I searched for 'USGS American Alligator' and found result #3" but "When searching for historical records, use government wildlife databases and cross-reference with known habitat patterns." The paper found that 84% of principle-level items contain reusable strategy-like statements. Only 3.7% of instance-level items do. **Why instance-level fails:** The agent memorizes specifics instead of strategies. When the next task involves a different animal, a different location, a different year, the specific log does not help. It actually hurts because the model wastes capacity on irrelevant details. The paper shows instance-level experience produces a gain in iteration 1 that collapses completely by iteration 3. **Why principle-level works:** Abstract strategies transfer across tasks. "Verify sources from authoritative domains" works whether the source is a wildlife database or a financial regulator. The model learns decision rules, not answer keys. **The catch:** Principle-level experience is harder to generate. You need a good model to extract the principle from the raw trajectory. The paper used DeepSeek-V4 for extraction. If your extraction model is weak, the principles will be low quality and the advantage disappears. **Who this is not for:** Agents that only ever see one type of task. If every query is structurally identical, instance-level experience is fine. The specificity stops being a disadvantage when there is no variation. ## Dimension 2: Experience Injection Pattern Here is how it works: you have your experience pool. Now you need to give it to the agent at the right time during a task. How do you do that? **Global injection** gives the agent all relevant experience at the start and leaves it in the context for the entire task. The same experience context is visible at every decision point. Search, verification, termination, answering. **Step-wise injection** selects experience based on the current state. At the search step, the agent sees principles about searching. At the verification step, it sees principles about verifying. At the answering step, it sees principles about when to stop. **Why global injection fails:** The paper discovered a specific failure mode they call "premature answering." When the model receives all experience at once, experience intended for later stages leaks into earlier decisions. A principle like "search broadly, do not settle for the first result" is useful during search. It is actively harmful when the model should stop searching and answer. The numbers are stark: global injection produced premature answers (the model outputs `` without any preceding tool call) in 63.82% of cases. Step-wise injection: 0%. Figure 4 in the paper illustrates this with a case study. The global-injection model terminates before searching at all. The step-wise model continues evidence-seeking tool use, finds the correct answer, and only then terminates. **Why step-wise works:** The agent needs different experience at different phases of a task. Search principles during exploration. Verification principles during fact-checking. Termination principles during the final decision. Each phase has its own information needs, and injecting the wrong information at the wrong time is worse than injecting nothing. **The catch:** Step-wise injection requires a selector model that can determine which experience is relevant to the current state. The paper used the same model for selection and acting. This works but adds latency to each decision step. The selector also introduces a new failure mode: if the selector picks the wrong experience, the agent acts on irrelevant information. **Who this is not for:** Short tasks with a single decision phase. If your agent typically completes tasks in 2-3 steps, global injection is fine because there are no distinct phases for experience to misalign with. ## Dimension 3: Internalization Regime Here is how it works: you have your experience. You have your injection pattern. Now you need to train the agent to internalize the experience so it does not need the context window anymore. How do you construct the training data? **On-policy distillation** works like this: the untrained student attempts the task without experience. It generates a trajectory. This trajectory is inevitably flawed because the student does not have experience yet. Then the teacher (the same model with access to experience) provides corrections on the student's states. The student learns from the corrections. The intuitive appeal is clear: the student learns from its own mistakes. The teacher shows up right when the student goes wrong and says "here is what you should have done instead." **Off-policy distillation** works differently: the teacher generates the full trajectory using experience. The student never sees the flawed path. It only sees successful, experience-guided demonstrations from start to finish. **Why on-policy fails:** The teacher can only provide local corrections on whatever state the student happens to reach. When the student has deviated from a useful search path, the teacher cannot fix the path. It can only fix the current step. The student learns "when you are in this specific broken state, do this specific fix" instead of "here is how to navigate the entire task correctly." This creates a negative spiral. The student generates broken trajectories → the teacher applies local fixes → the student learns local patches → the next iteration generates slightly different broken trajectories → the teacher applies different local fixes. The model never internalizes a coherent decision strategy. It accumulates patch after patch, and each patch interacts badly with the others. The paper also found a practical problem: trajectory inflation. After one on-policy update, the student's trajectories balloon from 2.5 assistant turns (baseline) to 21.9 turns. The student is flailing. Making more decisions, not better decisions. The teacher has to correct more and more states, and each correction is less useful because the states are further from the optimal path. **Why off-policy works:** The student learns from coherent, successful demonstrations. The experience-guided behavior is demonstrated from the first step to the last. The student internalizes "this is what a good trajectory looks like" not "this is how you escape from a bad state." The training signal is cleaner, more coherent, and produces models that generalize better across iterations. **The catch:** Off-policy requires rejection sampling. You generate trajectories from the teacher, filter out the failed ones, and train only on successes. This means you waste compute on trajectories that get rejected. The paper controlled for this by using the same query budget for both regimes, but off-policy still generates more total trajectories. **Who this is not for:** Applications where you cannot generate successful teacher trajectories. If your task is so difficult that even the teacher with experience fails most of the time, rejection sampling becomes prohibitively expensive. ## The winning recipe The paper combines all three fixes and tests the result. Principle-level experience. Step-wise injection. Off-policy distillation. The combined configuration sustains performance gains across three iterations on all benchmarks. More importantly, the model retains its ability to benefit from explicit experience at inference time. After internalization, if you give the model access to its experience pool, it still improves. The on-policy and global-injection models lose this ability. They get worse even when you give them the experience. This is the key metric for self-evolving systems. A model that cannot benefit from its own experience cannot serve as an effective teacher for the next iteration. The loop breaks. ## What this means for building agents The practical lesson is simple: be careful about how your agent learns from its own experience. Each design choice has hidden failure modes. If you store raw trajectories as memory, your agent will memorize URLs and query strings instead of learning strategies. Use principle extraction to convert logs into reusable lessons. If you dump all memories into the prompt at the start, your agent will answer before it searches. Use step-wise selection to give the right memory at the right decision point. If you train your agent by letting it attempt tasks and then correcting its mistakes, it will learn patches instead of principles. Train on successful demonstrations instead. The paper's recipe is not the only possible solution. But it demonstrates that the three dimensions are independent and complementary. Fixing one dimension helps. Fixing all three sustains improvement across iterations. ## FAQ > **What is experience internalization?** It is the process of converting contextual experience from past agent interactions into reusable model capability. Instead of stuffing memories into the context window, the model learns from them during training so it does not need them at inference time. > **Why do self-improving agents get worse over time?** Because most methods use on-policy distillation where the student generates trajectories first and the teacher corrects them. The student's trajectories are already broken, so the teacher can only give local fixes. The model accumulates patches instead of principles. > **What is principle-level vs instance-level experience?** Instance-level preserves specific details like URLs, numbers, and query strings. Principle-level extracts reusable strategies like "verify sources from authoritative domains." Instance-level produces gains in iteration 1 that collapse by iteration 3. Principle-level sustains improvement. > **What is step-wise injection?** It gives the agent only the experience relevant to the current decision step. Search principles during search. Verification principles during verification. Global injection gives all experience at once, which causes premature answering in 63% of cases. > **Should I use this for my agent?** If your agent runs more than a few hundred tasks and you want it to learn from its own experience, yes. The recipe is straightforward: use principle-level memory, inject it step-wise, and train on successful trajectories. The paper tested on Qwen3-4B and 8B, but the findings should transfer to any model with the same architecture. ## Related Posts - [I tested 5 self-improving loop patterns. Only 2 survived production.](/posts/self-improving-loop-patterns-production/). The companion post to this one. Covers five architectural patterns for self-improving loops and identifies the two that converge in production. - [Loop engineering: the production agent loop nobody talks about](/posts/loop-engineering-production-agent-loops/). The canonical ReAct loop, durable execution, and the gain-cost tradeoff that every loop faces. - [Your agent's memory is making it a yes-man](/posts/your-agents-memory-is-making-it-a-yes-man/). The sycophancy problem in agent memory systems. Relevant because every self-improving loop introduces bias toward its own past decisions. --- *This article was published on Agentic Up (https://agenticup.dev): practical guides for building AI agents that ship. Reach me at hello@agenticup.dev*