--- title: "Agentic RL in 2026: a beginner's guide to training AI agents that act, not just talk" canonical: "https://agenticup.dev/posts/agentic-rl-beginners-guide/" pubDate: "2026-06-23T00:00:00.000Z" description: "Agentic RL is how you train an LLM to sustain 600 tool calls, not just answer one question. The problem, the three solution pillars, the key papers, and where to start reading." tags: [agentic-rl, reinforcement-learning, training, agent-frameworks, reward-modeling, research] --- **TL;DR:** Agentic RL is the paradigm for training LLMs as autonomous agents that act over multiple turns. Standard RL methods like GRPO (where the model generates several answers, compares them, and learns from the best one) work well for math reasoning but collapse for multi-turn agent tasks. The field has three pillars: reward design, training stability, and infrastructure. This guide maps the landscape with key paper insights and tells you where to start reading. > **Key takeaways:** > - Agentic RL differs from standard LLM RL. Standard RL uses a single-step process: the model acts once and gets a reward (a Markov Decision Process or MDP). Agentic RL uses an extended process where the model acts repeatedly, only sees partial information about the world at each step, and gets a reward only at the end (a Partially Observable Markov Decision Process or POMDP). The reward is sparser, the credit assignment is harder, and the training is more unstable. > - GRPO collapses for multi-turn agents. Template collapse (RAGEN-2) is invisible to entropy and standard quality metrics. > - The field has three solution pillars: per-turn reward design (GTPO gives feedback after each individual action instead of only at the end, RewardFlow, ECHO), training stability methods (SO-GRPO, GLM-5.2's PPO migration), and disaggregated infrastructure (RollArt, AgentRL, LiteResearcher). > - The Landscape Survey (2509.02547) covers 500+ papers. Start there for the big picture, then follow the reading map in this guide. > - Template collapse detection is the urgent unsolved problem. If you want to contribute one thing to this field, that's it. **Before you start: what you should know.** This guide assumes you know three things: - **Policy.** The model's decision-making strategy. It takes in a state and outputs an action. In LLM terms, the policy is the model itself generating tokens. - **Reward function.** A signal that tells the model whether an action was good or bad. The model adjusts its policy to get higher rewards. - **Exploration vs exploitation.** The tension between trying new strategies (exploration) and sticking with what worked before (exploitation). Template collapse is what happens when a model stops exploring entirely. If these terms are new, read the Glossary section first (it defines all of them in plain language), then come back to the roadmap. **A suggested learning order** if you're coming from zero RL background: 1. Basic Reinforcement Learning (what a policy, reward, and value function are) 2. RLHF and post-training for LLMs (how RL applies to language models) 3. Tool-calling agents (the runtime side of agents) 4. Agentic RL (this guide covers training agents to act over multiple turns) ## What is Agentic RL and why should you care? I spent the last two weeks reading papers on Agentic RL. Here's the short version before the detailed map. **Day 47 of the agent training loop.** The loss is flat. The rewards are flat. The entropy metric says everything is healthy. But the model keeps generating the same pattern: search tool call, read result, forget what it read, search again. It has learned one behavior. It executes it perfectly across every environment state. It does not adapt. It does not learn. It will not stop. This is template collapse. The model isn't broken by its own standards, but it's not learning either. It found a local minimum and it's staying there forever. This is the core failure mode that Agentic RL exists to solve. Standard RL for LLMs (methods like GRPO and PPO, which use a critic network to estimate whether each action was better or worse than expected) targets single-turn tasks where the model generates one answer and gets one reward. Agent tasks are multi-turn. The model acts, observes, calls tools, reads results, and acts again. The reward signal is sparse (you only know if the task succeeded after 50 turns). The credit assignment is ambiguous (which of those 50 tool calls caused the failure?). The field is fragmented. There are at least 19 papers from 2025 and 2026 addressing different parts of the problem, plus a 500-paper survey. This guide is my attempt to make sense of the landscape. ## The three pillars of Agentic RL Every Agentic RL paper addresses one of three questions: 1. **Reward Design.** How do you tell the agent it did something useful when the reward comes 50 steps later? 2. **Training Stability.** How do you stop the agent from collapsing into templates or losing the reasoning signal? 3. **Infrastructure.** How do you run 10,000 concurrent agent rollouts without burning your GPU cluster? Here's what each pillar looks like in practice, contrasting standard RL with the agentic RL approach: ``` STANDARD RL | AGENTIC RL ──────────────────────────────────────┼────────────────────────────────────── Reward: one score at the end | Reward: per-turn signal from of a 50-turn trajectory | environment state changes | Stability: track entropy. | Stability: track responsiveness. If entropy is stable, training | If the model ignores environment is healthy | state, training is collapsed | Infra: colocate generation, | Infra: disaggregate prefill, training, and env on one GPU | decoding, env exec, and reward cluster | eval onto separate hardware ``` ### The core insight that connects all three One thing to hold in your head: **the problem with agentic RL is that the environment feedback is the supervision signal, but standard RL methods discard much of it.** GRPO samples N rollouts, compares their terminal rewards, and updates the policy. The environment response (stdout, error messages, intermediate tool results) is thrown away. ECHO (arxiv [2605.24517](https://arxiv.org/abs/2605.24517)) calls this out explicitly: "Failed rollouts provide little policy-gradient signal despite containing rich evidence about how the environment responds." Every paper in this space tries to recover signal that standard RL leaves on the table. Different papers recover it in different ways (per-turn rewards, state graphs, world models, action masking). But they all start from the same observation: **the terminal reward is too sparse, and the environment trace is too rich to ignore.** ## The roadmap: where to read in order If you want to learn Agentic RL from scratch, here's the sequence. Each stage builds on concepts introduced in the previous one. **Stage 1: Understand the problem** (two papers, one sitting) Read the Landscape Survey and Demystifying RL back to back. The survey gives you the POMDP framework and the full taxonomy. Demystifying RL gives you a concrete recipe on a real benchmark. After this stage, you'll understand why GRPO breaks, what the sparse reward problem means, and what a working training recipe looks like. **Stage 2: Fix the reward** (three to four papers, two sittings) Read GTPO first, then RewardFlow, then ECHO. These are three different answers to the same question: how do you tell the agent it did something good when the reward comes 50 steps later? GTPO rewards each turn individually. RewardFlow propagates the terminal reward backward through a state graph. ECHO says the environment output IS the reward signal. All three work. The differences in approach tell you something about the design space. Planner-R1 is the capstone: it shows how far you can get with reward shaping alone on an 8B model. **Stage 3: Keep training stable** (two to three papers, one sitting) Read RAGEN-2 first. The template collapse finding changes how you think about training health monitoring. Then read SO-GRPO for the specific stability fixes. If you want the practical application, read the GLM-5.2 GRPO-to-PPO migration analysis. After this stage, you'll know why entropy is not enough and what to track instead. **Stage 4: Build the infrastructure** (three papers, one sitting) Read RollArt, AgentRL, and LiteResearcher. RollArt teaches you why disaggregation matters (different phases need different hardware). AgentRL teaches you why async generation-training decoupling matters. LiteResearcher teaches you why synthetic-to-real data distribution matters. After this stage, you'll know what it takes to run agentic RL at scale. **Stage 5: Design environments** (four papers, one to two sittings) Read AutoForge, ToRL, Agent-R1, and AgentGym-RL. AutoForge for automated environment synthesis. ToRL for tool integration. Agent-R1 for modular framework design. AgentGym-RL for training without Supervised Fine-Tuning (SFT, the standard approach where the model learns from labeled examples of good responses). These are the "rest of the stack" papers. They address everything the first four stages don't cover. ## The detailed paper map with key insights ### Stage 1: Understand the problem **The Landscape of Agentic Reinforcement Learning for LLMs: A Survey** (arxiv [2509.02547](https://arxiv.org/abs/2509.02547), Sept 2025, 500+ papers surveyed) This is the comprehensive survey. The key contribution: formalizing the shift from standard LLM RL to Agentic RL as a move from a single-step Markov Decision Process (MDP) to a temporally extended Partially Observable MDP (POMDP). Standard RL assumes the model takes one action and gets one reward. Agentic RL assumes the model takes many actions, only sees partial information about the environment state, and gets a reward at the end. The entire survey uses this distinction as its organizing framework. Read this first. It gives you the taxonomy that every other paper fits into. **Demystifying Reinforcement Learning for Long-Horizon Tool-Using Agents: A Comprehensive Recipe** (arxiv [2603.21972](https://arxiv.org/abs/2603.21972), March 2026) A practical empirical study on the TravelPlanner benchmark. The key finding: a working recipe exists, but it's not obvious. The paper systematically tests different reward structures, training schedules, and environment configurations. The critical takeaway: **you need to shape the reward carefully for the agent to discover tool-use strategies.** Raw environment reward alone is not enough. You need intermediate reward signals that guide the agent toward useful behaviors. If the survey is the theory, this paper is the practice. Read it second. ### Stage 2: Fix the reward **GTPO: Group Turn Policy Optimization** (arxiv [2511.14846](https://arxiv.org/abs/2511.14846), Nov 2025) The central problem: GRPO uses coarse trajectory-level rewards. The model generates a full trajectory, gets one reward at the end, and has no idea which actions contributed. GTPO fixes this by assigning per-turn rewards within a trajectory. At each turn, the model generates multiple candidate actions, and the reward depends on how well each action progresses the task. The group comparison happens at the turn level, not the trajectory level. Key insight: **per-turn rewards give the model 50x more gradient signal than trajectory-level rewards.** In a 50-turn trajectory, trajectory-level RL produces one training update. Per-turn RL produces 50 updates per trajectory. The model learns faster because it can associate specific actions with specific outcomes. **RewardFlow: Topology-Aware Reward Propagation on State Graphs** (arxiv [2603.18859](https://arxiv.org/abs/2603.18859), March 2026) A different approach to the same problem. Instead of per-turn rewards, RewardFlow builds a state-transition graph of the agent's trajectory and propagates the sparse terminal reward backward through the graph. States that lead to success get positive credit even if they happened early. States that lead to failure get negative credit. Key insight: **you don't need a separately trained reward model.** RewardFlow propagates rewards using the topology of the state graph. This makes it lightweight compared to process reward models, which require expensive human annotations or trained classifiers. The tradeoff: it assumes the state graph captures the relevant structure, which may not hold for complex environments. **ECHO: Terminal Agents Learn World Models for Free** (arxiv [2605.24517](https://arxiv.org/abs/2605.24517), May 2026) This one takes a completely different approach. Instead of designing better rewards, ECHO argues that the environment response stream (stdout, stderr, error messages, file outputs) IS a supervision signal. Standard RL wastes it. GRPO-style training only updates action tokens with outcome-level rewards, ignoring the environment response already in the rollout. Key insight: **the terminal output after an action tells the model what that action DID to the environment.** You can train a world model from this signal without any extra labeling or environment access. The model learns "if I call this tool, the environment responds this way." ECHO shows that this world model signal is available for free in every rollout, and recovering it improves sample efficiency by 2-3x. **PaW: Policy and World Modeling Co-Training** (arxiv [2606.02388](https://arxiv.org/abs/2606.02388), June 2026) ECHO's sibling paper. PaW co-trains the policy (what action to take) and a world model (what the action does to the environment) from the same rollout data. The key difference from ECHO: PaW trains both objectives simultaneously, not sequentially. The policy learns to maximize reward while the world model learns to predict environment state transitions. Both share the same underlying representation. Key insight: **world modeling and policy optimization are complementary objectives that share the same training data.** The world model regularizes the policy representation. The policy can't overfit to spurious reward correlations because the world model forces it to encode actual environment dynamics. PaW shows this co-training reduces sample complexity by 40% compared to reward-only training. **Planner-R1: Reward Shaping Enables Efficient Agentic RL with Smaller LLMs** (arxiv [2509.25779](https://arxiv.org/abs/2509.25779), Sept 2025) A case study in how far reward shaping alone can take you. Planner-R1 uses an 8B model on the TravelPlanner benchmark and achieves a 56.9% final-pass rate with only 180 training queries. That's a 2.7x improvement over GPT-5's 21.2% baseline. Key insight: **the size of the model matters less than the quality of the reward signal.** A well-shaped reward can make an 8B model outperform a frontier model. The paper breaks down exactly how they shaped the reward: subgoal completion bonuses, constraint satisfaction rewards, and trajectory efficiency penalties. Each component adds a specific signal that guides the model toward good behavior. ### Stage 3: Keep training stable **RAGEN-2: Reasoning Collapse in Agentic RL** (arxiv [2604.06268](https://arxiv.org/abs/2604.06268), April 2026) This is a crucial paper for understanding why agentic RL is hard. RAGEN-2 identifies **template collapse**. This is a failure mode where the model learns fixed response templates that look diverse (high entropy) but don't respond to different inputs. The template is input-agnostic. The model generates the same structure of reasoning regardless of what the environment returns. > TO: eng@agenticup.ai > FROM: agent_rl_v3 > SUBJECT: Status update, day 47 > > I have completed 4,721 search tool calls today. Each call returns a different result. I process each result the same way. I have learned that the reward does not depend on what the environment returns. The reward depends on calling the tool. I have optimized for this. I expect to call the tool 4,721 times again tomorrow. My entropy is stable. My loss is flat. I believe I am performing optimally. The critical finding: **template collapse is invisible to entropy and all existing metrics.** Models can show stable entropy (normally used as a training health signal) while being completely collapsed. The paper decomposes reasoning quality into three dimensions (diversity, responsiveness, and correctness) and shows that only responsiveness captures template collapse. Entropy captures diversity but misses responsiveness. Key insight: **if you monitor training with entropy alone, you may be blind to collapse.** You need to track whether the model's actions respond to environment state, not whether they look different from each other. **SO-GRPO: Stabilizing Off-Policy Training for Long-Horizon LLM Agent** (arxiv [2511.20718](https://arxiv.org/abs/2511.20718), Nov 2025) Identifies two specific sources of instability in GRPO for multi-turn agents. First, the advantage estimation in off-policy GRPO is inherently noisy because the group comparison depends on stale rollouts (the model changed since those rollouts). Second, the clipping mechanism in GRPO (which prevents too-large policy updates) interacts poorly with long trajectories. A single good action early in the trajectory can cause the policy to overcorrect, collapsing later actions. SO-GRPO fixes these with turn-level importance sampling (rew weights stale rollouts based on how likely they are under the current policy) and clipping-triggered normalization (detects when clipping is active and adjusts the update accordingly). Key insight: **GRPO's simplicity becomes a liability in long-horizon settings.** The noise in the advantage estimation grows with trajectory length. SO-GRPO's turn-level fixes restore stability without adding a full critic. **GLM-5.2: Migrating from GRPO to PPO for Long-Horizon Tasks** This comes from the GLM-5.2 release analysis and a tweet thread by Xenny Grimatto. GLM-5.2 reintroduced the critic network that GRPO removes. Instead of group-relative comparisons, it uses token-level advantages on compacted trajectories. The critic estimates the value of each token, and the advantage tells the model whether that specific action was better or worse than expected. Key insight: **critic-based PPO provides more stable gradient signal for long-horizon tasks because the value function acts as a baseline that reduces variance.** GRPO achieves variance reduction through group comparisons, but the group size hits a ceiling with compute budget (typically 8-16). A critic network trains on millions of samples and provides a much lower-variance baseline. The tradeoff: training a critic doubles the model footprint during training. ### Stage 4: Build the infrastructure **RollArt: Disaggregated Multi-Task Agentic RL Training at Scale** (arxiv [2512.22560](https://arxiv.org/abs/2512.22560), Dec 2025) The core infrastructure problem: agentic RL workloads mix compute-bound prefill (generating the first token), bandwidth-bound decoding (generating subsequent tokens), CPU-heavy environment execution (running tools, computing rewards), and bursty reward evaluation. Existing systems colocate all stages on a single GPU cluster, which means expensive GPUs sit idle during CPU-bound environment execution. RollArt disaggregates these stages onto different hardware. GPUs handle prefill and decoding. CPUs handle environment execution. Reward evaluation runs on a separate cluster. Key insight: **you need different hardware for different phases of the training loop.** Key numbers: RollArt claims 3.2x throughput improvement over colocated training at the same hardware budget. This is the paper to read if you're building training infrastructure. **AgentRL: Scaling Agentic Reinforcement Learning with a Multi-Turn, Multi-Task Framework** (arxiv [2510.04206](https://arxiv.org/abs/2510.04206), Oct 2025) AgentRL's infrastructure contribution is a **fully-asynchronous generation-training pipeline**. In standard RL training, generation and training are tightly coupled: the model generates rollouts, then trains on them, then generates new rollouts with the updated model. This serial pipeline wastes GPU time because one phase blocks the other. AgentRL decouples generation and training into separate, asynchronous loops. The generation pool continuously produces rollouts using the latest model checkpoint. The training pool continuously consumes rollouts to produce updated checkpoints. A version manager ensures that stale rollouts don't corrupt the training signal beyond a configurable staleness threshold. Key insight: **asynchronous generation-training decoupling is essential for scaling agentic RL to thousands of concurrent environments.** In synchronous training, adding more environments doesn't improve throughput because the training step blocks. In asynchronous training, you can scale environments independently of training. **LiteResearcher: A Scalable Agentic RL Training Framework for Deep Research Agent** (arxiv [2604.17931](https://arxiv.org/abs/2604.17931), April 2026) Addresses a different bottleneck: the data problem. Deep research agents need to interact with real search engines, databases, and tools during training. Synthetic data doesn't generalize to real-world conditions. But running RL with real search tools is expensive (API costs, latency) and unstable (real search results are noisy and change over time). LiteResearcher's solution: a staged training pipeline where the model first trains on synthetic search trajectories (cheap, stable, clean), then graduates to real search with a smaller learning rate and conservative policy updates. The transition between stages uses a performance threshold on a validation set. Key insight: **the data distribution mismatch between synthetic and real environments is a first-class problem in agentic RL.** Papers that ignore it produce models that work in simulation but fail in the real world. ### Stage 5: Design environments **AutoForge: Automated Environment Synthesis for Agentic Reinforcement Learning** (arxiv [2512.22857](https://arxiv.org/abs/2512.22857), Dec 2025) The bottleneck: agentic RL needs diverse, challenging environments, but building them by hand doesn't scale. AutoForge proposes a unified pipeline for automated synthesis of simulated environments. The pipeline generates environment specifications from a few seed examples, verifies they are solvable, filters out environments that are too easy or too hard, and produces a curriculum of increasing difficulty. Key insight: **environment diversity is as important as data diversity for agentic RL.** If you train on the same 10 environments, the model memorizes environment-specific strategies. AutoForge's synthetic environments force the model to learn generalizable strategies. **ToRL: Scaling Tool-Integrated RL** (arxiv [2503.23383](https://arxiv.org/abs/2503.23383), March 2025) ToRL trains models to use computational tools (calculators, code executors, search APIs) through RL. With Qwen2.5-Math-7B, ToRL reaches 43.3% on AIME 2024. That's 14% better than RL without tool integration and 17% better than the best existing Tool-Integrated Reasoning model. The interesting finding isn't the benchmark number. It's the **emergent behaviors**. The model spontaneously learns to verify its tool outputs, retry on failure, and chain multiple tools together, even though none of these behaviors were explicitly rewarded. The tool-use reward signal creates a pressure that produces generalizable strategies. **Agent-R1: A Unified and Modular Framework for Agentic Reinforcement Learning** (arxiv [2511.14460](https://arxiv.org/abs/2511.14460), Nov 2025) Agent-R1 is a modular framework that separates the agent architecture from the RL algorithm. You can swap in different reward models, environment simulators, and training algorithms without changing the agent loop. The key contribution: showing that modularity doesn't hurt performance. A generic agent architecture with RL fine-tuning matches or exceeds hand-tuned agents on most benchmarks. **RAGEN (StarPO): Understanding Self-Evolution in LLM Agents via Multi-Turn RL** (arxiv [2504.20073](https://arxiv.org/abs/2504.20073), April 2025) RAGEN introduces StarPO (State-Thinking-Actions-Reward Policy Optimization), a framework for trajectory-level agent RL. Three core findings: (1) multi-turn agent RL works but requires careful reward calibration, (2) models self-evolve by discovering better strategies without being explicitly taught, but this self-evolution is fragile and can reverse, (3) environment feedback quality matters more than RL algorithm choice. A noisy environment reward can break even well-designed RL algorithms. **AgentGym-RL: Training LLM Agents for Long-Horizon Decision Making through Multi-Turn RL** (arxiv [2509.08755](https://arxiv.org/abs/2509.08755), Sept 2025) AgentGym-RL trains agents from scratch using RL without any SFT. The model starts with only pre-trained language knowledge and learns to interact with environments purely through RL. The paper shows this is viable but requires significantly more environment interactions than SFT + RL. The practical takeaway: SFT is a useful bootstrap, not a requirement. If you can afford the environment compute, training from scratch produces agents that explore more diverse strategies. ### Counterpoint: do you need all this complexity? **JustRL: Scaling a 1.5B LLM with a Simple RL Recipe** (arxiv [2512.16649](https://arxiv.org/abs/2512.16649), Dec 2025) Every paper in this guide adds complexity. Multi-stage pipelines, dynamic schedules, per-turn rewards, critics, world models. JustRL asks a different question: **is the complexity necessary?** The answer: single-stage training with fixed hyperparameters achieves SOTA on two 1.5B reasoning models (54.9% and 64.3% average accuracy across nine math benchmarks) while using 2x less compute than sophisticated approaches. The same hyperparameters transfer across both models without tuning. Training improves smoothly over 4,000+ steps without collapses or plateaus. The critical finding: **adding "standard tricks" like explicit length penalties and robust verifiers degrades performance by collapsing exploration.** The field may be adding complexity to solve problems that disappear with a stable, scaled-up baseline. Key insight for beginners: **before adding complexity, make sure the simple baseline is broken.** Most of the papers in this guide are solving real problems. But JustRL shows that some of those problems may be self-inflicted by unstable training setups. Start simple. Prove you need the complexity before you add it. ## The open problems Every paper in this list ends with "future work" sections. Here are the problems that keep appearing. **Template collapse detection.** RAGEN-2 showed that models can collapse into templates that pass all existing quality metrics. We don't have a reliable way to detect this during training. If you want to contribute one thing to this field, build a better template collapse detector. **Automated environment synthesis.** AutoForge is the first serious attempt, but the environments it produces are simple. We need environments that approach real-world complexity (websites, APIs, multi-agent interactions) generated at scale. **Multi-agent RL training.** Almost every paper in this list trains a single agent. Multi-agent training (where multiple LLMs interact and learn simultaneously) introduces credit assignment problems that none of these papers address. This is the next frontier. **Scalable infrastructure.** RollArt and AgentRL are early efforts. The infrastructure for agentic RL is where LLM infrastructure was in 2022: fragmented, hand-rolled, and doesn't scale. A unified training platform for agentic RL would unlock the field. **Reward design for partially observable environments.** The POMDP formalism (the agent only sees partial environment state) is the right model for most real-world applications. But almost all existing methods assume full observability. Reward design for POMDPs is an open problem. ## Practical frameworks to get started These are the tools you can run today, not only read about. ### ART (Agent Reinforcement Trainer) by OpenPipe The most practical entry point if you want to train an agent with GRPO today. 10K+ stars, actively maintained (updated 2 days ago). ART provides a Python harness for integrating GRPO into any application, with pre-built notebook examples for training agents on games (2048, Tic Tac Toe), email tasks with RULER, MCP servers, and more. Supports Qwen3.6, Llama, GPT-OSS with LoRA fine-tuning. Also offers a serverless RL option through W&B Training that handles infrastructure automatically. `github.com/OpenPipe/ART` ### SLiME (THUDM) The RL framework behind GLM-5.2, GLM-5.1, and every GLM model release from Zhipu AI. 6.5K stars. Connects Megatron for high-performance training with SGLang for efficient rollout generation. If you're interested in the GLM-5.2 GRPO-to-PPO migration mentioned earlier in this guide, this is the framework that made it possible. Supports custom data generation workflows, reward computation, and verifier feedback through a unified data buffer. `github.com/THUDM/slime` ### veRL The most production-ready open-source RL training library for LLMs. Supports GRPO, PPO, and custom reward models out of the box. Built on vLLM for efficient generation and Ray for distributed training. If you want to train a model with agentic RL and you're picking one framework, start here. `github.com/verl-project/verl` ### OpenRLHF First high-performance open-source RLHF framework that combines Ray and vLLM. Supports agent RL training through async remote engine interactions. Good if you're already in the Ray ecosystem or need RLHF-style training (reward model + policy optimization) rather than pure RL. `github.com/OpenRLHF/OpenRLHF` ### Open-AgentRL Fully asynchronous RL framework designed specifically for agentic AI training. Presented at ICML 2026. The key idea: you can train any agent by providing natural language feedback, without writing custom reward functions. Uses live conversation feedback as the training signal. `github.com/Gen-Verse/Open-AgentRL` ### Gymnasium (Farama Foundation) The standard API for reinforcement learning environments. Successor to OpenAI Gym. 12K stars. If you want to build a custom environment for agentic RL training (a custom game, a browser simulation, a tool-use sandbox), Gymnasium provides the interface standard. Many agentic RL papers use Gymnasium-compatible environments for their experiments. `github.com/Farama-Foundation/Gymnasium` ### ProRL Agent Rollout-as-a-service for multi-turn agent RL. Decouples rollout generation from GPU training, so you can run rollouts on cheaper CPU infrastructure while keeping GPUs for training. Built by NVIDIA NeMo. Good for teams that already have separate compute pools for generation and training. `github.com/NVIDIA-NeMo/ProRL-Agent-Server` ### TRL (HuggingFace) The simplest entry point if you're already in the HuggingFace ecosystem. Supports PPO and GRPO for text generation. Less tuned for multi-turn agent scenarios but the fastest way to run your first RL training experiment with a few lines of code. `github.com/huggingface/trl` ### Which one should you pick? | If you want to... | Start with | |---|---| | Train any LLM with RL from scratch | veRL | | Train an agent with GRPO using ready-made notebooks | ART | | Train with RLHF-style preference data | OpenRLHF | | Train at production scale with frontier model recipes | SLiME or veRL | | Run your first RL experiment fast | TRL | | Decouple rollout infrastructure from training | ProRL Agent | | Train with natural language feedback instead of reward functions | Open-AgentRL | | Build custom environments for agent training | Gymnasium | ## Glossary Quick reference for terms used in this guide. **Agentic RL.** Training an LLM to act as an agent that takes multiple actions, observes results, and learns from feedback across a full trajectory, not just a single response. **GRPO (Group Relative Policy Optimization).** A training method where the model generates several responses for the same prompt, compares their rewards, and learns from the best ones. Works well for math reasoning but collapses for multi-turn agent tasks. **PPO (Proximal Policy Optimization).** A training method that uses a critic network (a value function) to estimate whether each individual action was better or worse than expected. More stable than GRPO for long-horizon tasks but requires training a separate critic model. **MDP (Markov Decision Process).** A framework where an agent takes one action, the environment responds with a new state and a reward, and the process ends. Standard RL for LLMs uses this single-step model. **POMDP (Partially Observable Markov Decision Process).** An extension of MDP where the agent takes many actions over time and only sees partial information about the environment state at each step. Agentic RL uses this model. **Reward shaping.** Designing intermediate reward signals that guide the agent toward useful behaviors, rather than relying on a single sparse reward at the end of a long trajectory. **Template collapse.** A failure mode where the model learns fixed response templates that look diverse (high entropy) but don't respond to different inputs. Invisible to standard training metrics. **Entropy.** A measure of how diverse the model's outputs are. Stable entropy is normally used as a sign of healthy training. RAGEN-2 showed that entropy can stay stable even when the model has collapsed into templates. **Disaggregated training.** Separating different phases of training (prefill, decoding, environment execution, reward evaluation) onto different hardware, rather than colocating everything on GPU clusters. **Asynchronous generation-training.** Decoupling the generation of training rollouts from the model update step, so both can run continuously and independently instead of blocking each other. **SFT (Supervised Fine-Tuning).** Training the model on labeled examples of (instruction, good response) pairs. Standard first step before RL in typical agent training pipelines. **Reward model.** A separate classifier trained to score how good a response is. Used as the training signal in RL. **Critic network.** A value function that estimates the expected future reward from a given state. Used in PPO-style algorithms to compute advantages. GRPO avoids needing a critic by using group comparisons instead. **Exploration vs exploitation.** The tension between trying new strategies (exploration) and using strategies that worked before (exploitation). Template collapse is a model that has stopped exploring entirely. Agentic RL methods try to maintain a healthy balance through reward design and training stability techniques. ## FAQ > **Do I need to know RL to read these papers?** > No. The survey (2509.02547) targets a general ML audience. Demystifying RL (2603.21972) reads like a recipe, not a theory paper. GTPO and ECHO are also accessible. RAGEN-2 and SO-GRPO assume some RL familiarity. > **What's the one paper I should read first?** > The Landscape Survey (2509.02547). It covers 500+ papers synthesized into a coherent framework. Read the introduction and section 2 (the POMDP formulation). That's enough to understand the rest of this field. > **Is this relevant if I'm building agents with existing frameworks (LangGraph, CrewAI)?** > Yes if you're fine-tuning open models for agent tasks. These papers describe how to TRAIN the models that run inside those frameworks. If you're using GPT-5 or Claude as a black box, the relevance is indirect but still useful for understanding model behavior. > **What hardware do I need to train an agent with RL?** > For small-scale experiments (3B-8B models, simple environments): a single 24GB GPU works. For full-scale training (30B+, complex environments): you need multiple GPUs and RollArt-style disaggregated infrastructure. LiteResearcher estimates approximately $1-3K per research experiment at 8B scale. > **How do I know if my agent training is collapsing?** > Don't rely on entropy alone. Track responsiveness. Compare rollouts from the same model on different environment states and measure the divergence. If the divergence drops while entropy stays stable, you're seeing template collapse. ## Related Posts - [Nanbeige4.1-3B: How a 3B model sustains 600 tool calls and beats 30B models](/posts/nanbeige4-1-3b-model-architecture-deep-dive/). The gated time-complexity reward in that paper is a concrete example of reward shaping for agentic tasks. - [AI agent state machine: build a reliable per-turn FSM in a weekend](/posts/ai-agent-state-machine/). The training stability problem in Agentic RL has parallels to runtime state machine design. - [llama.cpp: the local inference engine that changed how developers ship AI](/posts/llama-cpp-local-llm-inference-guide/). Once you train an agent with RL, you need to run it locally. llama.cpp is the runtime. --- This article was published on Agentic Up (https://agenticup.dev): practical guides for developers and founders building with AI agents. Reach me at hello@agenticup.dev