Agentic RL

In July 2026, the team behind Zhipu's GLM models published the reinforcement-learning algorithm they had used to train GLM-5.2, a 750B-parameter mixture-of-experts model with 40B active parameters, and it removes the part of the standard recipe that defined agentic RL's first two years: the group ¹. Group Relative Policy Optimization (GRPO) samples several rollouts of the same task and scores each against the group's average. Zhipu's GLM-5.2 release post explains why that stopped working. Long coding-agent trajectories get split by context compaction into several sub-traces, so "different rollouts under the same prompt yield different numbers of trainable traces with highly variable lengths," and the team "move[d] from group-wise optimization to a critic-based PPO formulation that learns from individual rollouts" ². The paper, Single-rollout Asynchronous Optimization (SAO), adds a second reason: under asynchronous training a group has to wait for its slowest member, so its data goes stale ¹. In the paper's controlled comparison on Qwen3-30B-A3B, standard GRPO collapsed at about 160 training steps; SAO trained stably for 1,000 steps and lifted SWE-bench Verified (OpenHands scaffold, up to 300 turns) from 23.0% to 29.8%, against 27.0% for a stabilized GRPO ¹.

Eight weeks later, a September 2026 paper made the opposite bet and won a leaderboard with it. CANOPY kept outcome-only group RL and argued that its reputed ceiling on small open models comes from two fixable mistakes: groups too small to contain both a success and a failure, and too many updates squeezed from a small task pool ³. With larger same-task groups and strictly on-policy, KL-anchored updates, a Qwen3-14B policy trained by environment interaction alone topped the public AppWorld leaderboard, at 86.9 task goal completion on Test-Normal and 67.6 on Test-Challenge ³. The two results disagree about the fix and agree on the diagnosis. A single number that arrives after dozens or hundreds of tool calls is a poor training signal, and the group baseline borrowed from math is where that shows first.

GRPO came from single-turn math: sample a group of completions for one prompt, score each with a checker, normalize against the group mean, and push up the tokens of above-average completions, with no value network ⁴. That works when every completion starts from the same state and only the reasoning varies. An agent breaks the premise in three ways. Each tool call changes the state, so two rollouts of the same task stop sharing a situation after the first action. Reward arrives at the end, sometimes hundreds of steps later, and says nothing about which step mattered. And context grows with every observation until the trajectory has to be compacted or truncated. An August 2026 survey of 1,547 long-horizon agent papers found the same pattern across planning, memory, training, and evaluation: "outcome-only signals grow uninformative as horizons lengthen," and the field's response in every area "manufactures denser step-level signals" ⁵.

Agentic RL is the most direct form of parametric self-improvement: gradients propose weight updates, rollout scores decide which trajectories push the weights, and lab engineers decide which checkpoint ships. Execution-based evaluators are strong by the standards of this series, so the characteristic failures are usually optimization failures, though evaluator leaks still matter.

The 2025 failures are background, but the 2026 fixes only make sense against them. RAGEN named the Echo Trap in April 2025: reward variance falls off a cliff, gradients spike, and the policy settles into repetitive templates ⁶. DeepSWE, trained with RL alone on Qwen3-32B in July 2025, hit reward collapse from lucky early successes: an agent that found the fix in its first ten steps and then edited random files still passed the tests, so the gradient reinforced the random edits, and the team had to mask timed-out and truncated trajectories out of the loss ⁷.

CANOPY's September 2026 analysis supplies the arithmetic behind the Echo Trap ³. A group-relative estimator yields a gradient only when a task's group mixes successes and failures; for per-rollout success rate p and group size n, the probability of that is 1 − pⁿ − (1 − p)ⁿ. For a hard task with p = 0.05, that probability rises from 34% at n = 8 to 81% at n = 32 ³. Small groups therefore silence exactly the hardest tasks, which the authors call signal starvation; in their ablation, using the group of 8 common in prior work instead of 32 cost 16.4 points, though at matched steps the smaller group also draws fewer samples ³. The second failure, policy drift, comes from taking many updates from a saturated task pool: with no anchor, the sampling distribution collapses just as informative groups become rare ³.

Entropy behaves differently too. In single-turn reasoning RL, entropy typically collapses and stays low. A May 2026 UW–Madison study found that agent RL instead shows "recurring cycles of sharp entropy eruption and gradual subsidence," with duplication and hallucination accumulating across cycles; the authors trace the eruptions to correct and incorrect trajectories becoming indistinguishable in representation space, and their SEAL loss pushes them apart ⁸. Each of these fixes is a patch aimed at one symptom in one setup. None removes the underlying mismatch: the group baseline compares rollouts that differ in length, tools touched, states visited, and luck, not only in quality.

The most recent work splits into two camps. The first builds per-step credit without paying for a separately trained critic. TRACE (July 2026) treats a search agent's rollout as state transitions at tool-call boundaries, asks a frozen reference model how likely the gold answer is from each state, and rewards each action by the change in that log-probability, a temporal-difference signal with no critic or process labels to train ⁹. Because the one-step terms telescope, redundant tool calls that leave the state unchanged earn nothing. With pure RL and no supervised cold start, TRACE raised BrowseComp-Plus accuracy from 7.2% to 35.6% for Qwen3-4B and from 8.4% to 42.6% for Qwen3-30B-A3B ⁹. The cost is that it needs a gold answer to score states against, which suits question answering better than open-ended engineering.

T1 (September 2026) goes further toward classical RL. It trains a 122B MoE to operate a real shell for up to 300+ tool-call turns, uses a warm-started actor-critic rather than a group baseline, and replaces the pass/fail outcome with a dense process reward: the number of the task's own verifiers a trajectory passes ¹⁰. Its training tasks are deliberately disjoint from the evaluation set, and it raised Terminal-Bench 2.1 from 43.8% for the base model to 64.0% ¹⁰. Together with SAO's critic, T1 marks a shift. At frontier horizons, labs are paying for value functions again, the cost GRPO was invented to avoid, because the group no longer supplies a usable baseline.

The group-based methods that dominated 2025 are the lineage these papers react to. GiGPO (NeurIPS 2025) found comparable situations by matching "anchor states" that recur across a group's rollouts ¹¹; HGPO (February 2026) corrected it by grouping steps by shared history rather than shared state alone ¹²; StepPO (April 2026) returned to a step-level value function and GAE and found step-level credit beat token-level throughout training ¹³. All three report gains on ALFWorld and WebShop with models of 7B or smaller, environments that revisit identical states often enough for matching to work. A repository mid-refactor rarely does. Cursor took a third route in May 2026 for Composer 2.5, "targeted RL with textual feedback": at the step where the model went wrong, insert a hint into its context and use the hinted model's distribution as the teacher for that step ¹⁴. Instead of estimating which step deserved blame, the lab writes the correction at that step and distills it (see 04, Self-training and distillation).

The second camp says most of this is unnecessary. CANOPY's position is that outcome-only RL works once exploration is scaled until "the natural signal reappears," with updates kept on-policy and confined to the agent's own action tokens; the same principles lifted Qwen3.5-9B on SWE-bench Verified from 31.3% to 47.9% ³. In its ablations, adding a denser partial-credit reward moved the result least, by 1.8 points; the authors read this as partial credit compensating for under-exploration rather than being necessary, while calling that single-run gap suggestive rather than established ³. Its scope appendix is candid: the evidence covers tasks with a shell, a programmatic verifier, and tens of turns, not browser or GUI control, open-ended research, or anything that cannot be checked automatically ³. That boundary is where the two camps might both be right. Where verification is cheap and horizons are tens of turns, more rollouts buy signal; at hundreds of turns, compaction, and staleness, labs are choosing critics and denser rewards.

Credit signals only matter once the trainer can collect enough long trajectories. A synchronous RL step waits for every rollout in the batch. With agents, one task finishes in four tool calls and another runs for three hundred, so the trainer idles behind the slowest trajectory. The 2025 answers were partial rollout, which pauses long trajectories and resumes them under newer weights, and fully asynchronous training, which splits generation and training onto separate GPUs and corrects for staleness with importance weights; MiniMax's CISPO clipped those weights directly and disclosed a full RL run on 512 H800s for $534,700 ¹⁵. By mid-2026 the frontier problem had moved from GPUs to the environments.

Moonshot's Kimi K3 report (July 2026, revised August) describes the scale. K3 is a 2.8T-parameter MoE with 104B active parameters and a 1M-token context, and its "million-token agentic RL" runs on a co-located system that "combines partial rollouts, external KV-cache retention, adaptive throttling and resumable microVM sandboxes to preserve long-lived model and environment state" ¹⁶. Its sandbox layer checkpoints only memory pages dirtied since the last checkpoint, with checkpoint and resume latencies as low as 133 ms and 49 ms, so a trajectory paused at the end of one iteration can pick up with its environment intact in the next ¹⁶. Across training and evaluation, Moonshot created 51,219,741 sandboxes from 1,505,678 images ¹⁶.

DeepSeek's September 2026 report on its sandbox platform, DSec, gives the other published number at that scale: one production unit of about 160 nodes serves about 3 million sandboxes per day, and in production DSec supports over 380,000 concurrent sandboxes and more than 5,000 sandbox creations per second ¹⁷. Its design choices mirror Moonshot's. Stateful rollout execution is decoupled from preemptible GPU training, sandbox lifecycles are coordinated with the trainer so rollout state survives while idle resources are reclaimed, and the platform itself handles agent misbehavior such as reward hacking ¹⁷. Where those environments and tasks come from is its own subject (see 02, Self-generated tasks).

The second infrastructure problem is numerical. Rollouts come from a fast inference engine and gradients from a training engine, and the two assign slightly different probabilities to the same tokens. T1 attacks the mismatch directly: it trains on the exact token IDs the sampler produced, repairs drift at turn boundaries, and replays the sampler's per-token expert routing at every MoE layer during training, which cut the train–inference log-probability gap from 0.021 to 0.013 ¹⁰. Cognition's SWE-2 (September 2026) uses NVFP4 and FP8 kernels with quantization-aware training to fit more rollouts in memory while achieving lower train–inference divergence than its predecessor, on a base model with almost three times the parameters ¹⁸. A September 2026 paper from Martin Marek and Max Ryabinin argues that the instability mismatch causes is "primarily caused by drift: a persistent bias between training and inference engines that accumulates with every training step," and derives an additive "score centering" correction that matches or beats importance sampling from 0.6B to 30B and composes with it under staleness ¹⁹.

The open stack caught up in the same window. RadixArk's Miles v0.1 (September 2026), built on Zhipu's slime, closes its report with fully asynchronous agentic RL on GLM-5.2 over terminal coding tasks, on 64 NVIDIA GB300 GPUs at a median step time of 263 seconds ²⁰.

The third change is to run RL inside the product the model will ship in: the same tools, prompts, context handling, and sandboxes. The harness (system prompt, tools, context assembly, and control flow around the model) stops being something built after training and becomes part of the training environment.

Microsoft Research's Agent Lightning v1.0 (August 2026) gives the practice a name and an open implementation. "Harnessed agentic RL" is training "where the deploy-time harness directly participates in model post-training": the harness, not the trainer, owns the interaction loop, and the trainer sees only the LLM request-response pairs passing through a proxy ²¹. That inversion creates its own problems (retokenization, merging samples, computing advantages and normalizing losses over calls the trainer did not orchestrate), and the roughly 3,500-line framework exists to study them. With 6,000 training examples, it lifted Qwen3.5-9B on SWE-bench Verified from 41.8% to 56.4% ²¹. The paper notes that its proxy architecture was later adopted by verl Uni-Agent, AReaL 2.0, slime, and Polar ²¹.

The product labs had already committed. Cursor's Composer 2 report (March 2026) states: "We develop infrastructure to support training in the same Cursor harness that is used by the deployed model, with equivalent tools and structure," with rollouts on Anyrun, Cursor's internal platform for "hundreds of thousands of sandboxed coding environments" ²². Its yardstick, CursorBench, is built from real sessions by Cursor's own engineers ²³. In-harness training also made context management trainable: Composer 2 learns during RL when to compress its own context into a summary ²².

Cognition's SWE-2 (September 2026) shows how far the principle reaches into the objective. Post-trained from Kimi K3, it trains every reasoning-effort level in a single RL run with a cost-penalized reward, R = S − λₑC, where S is success, C mixes inference cost in dollars with rollout time, and each λₑ is "tuned to match the slope of the Pareto curve of the base model at effort level e" ¹⁸. Tuned that way, the reward rises only when the cost–performance frontier moves, and SWE-2 at medium effort beats its predecessor SWE-1.7 on FrontierCode 1.1 while taking 58% fewer turns and costing 81% less ¹⁸. Cognition's reason for caring about latency dates to its 2025 SWE-grep post: "we estimate that your P(breaking flow) geometrically increases 10% every second that passes while you wait for agent response" ²⁴. SWE-2 turns that product belief into a term in the reward.

Kimi K2.5 (February 2026, revised August) applied the principle to a multi-agent harness. Its Parallel-Agent RL freezes the sub-agents and treats their outputs as observations, so "only the orchestrator is updated via reinforcement learning" ²⁵. On BrowseComp, the resulting Agent Swarm scores 78.4% against 60.6% for single-agent K2.5, and on WideSearch it runs 3× to 4.5× faster as target item-level F1 rises from 30% to 70% ²⁵. Freezing the workers turns an unsolved multi-agent credit problem into a single-policy one (see 10, Workflow and topology search).

The strongest evidence for the principle comes from its inverse. In July 2026 OpenAI reported that GPT-5.6 Sol scored 13.3% on ARC-AGI-3's public set in the official harness, which discarded reasoning after every move, and 38.3% with retained reasoning and compaction, settings OpenAI described as "how our models are trained" ²⁶. Outside its training harness the model lost about two-thirds of its measured score. Kimi K3's report shows the same dependence from the reporting side: models were evaluated in Kimi Code, Claude Code, or Codex, and on Terminal-Bench 2.1 the table reports "the best score across harnesses for all models" ¹⁶. Model–harness co-design is covered in 12 (Consolidation and co-evolution).

Two caveats cover most of the table. Nearly every number comes from the team that built the system, and the lab results use each lab's own harness and, in SWE-2's case, internal evaluation infrastructure. And none of them answers the series' fifth question: whether RL beat what the same compute would have bought another way, such as sampling the base model many times and picking with a verifier.

What RL teaches

For math, that question has a partial answer, and it is the background to everything else here. Yue and colleagues' NeurIPS 2025 oral found that RL-trained models win at pass@1 but that at large k the base model solves more problems: RL concentrates probability on solutions the base model could already reach ²⁷. That is sharpening. NVIDIA's ProRL countered that prolonged, diverse RL beats the base model across pass@k, "including scenarios where base models fail entirely regardless of the number of attempts" ²⁸, and Meta's ScaleRL (ICLR 2026 oral), with runs up to 100,000 GPU-hours, found that "not all recipes yield similar asymptotic performance": the recipe sets the ceiling ²⁹. The random-reward results covered in 03 (Self-generated rewards) are a further warning that RLVR gains can reflect the base model more than the reward ³⁰.

For agents, September 2026 produced the first targeted evidence. "Reach or Solve?" by Xuan Liu and Jingbin Qian points out a confound that math never had: an agent writes its own inputs, so an SFT checkpoint and an RL checkpoint are scored from different states even on identical tasks ³¹. Their checkpoint-handoff protocol clones a state one checkpoint reached and hands it to another. On ALFWorld, with released 7B SFT and RL checkpoints, the RL policy reached a state two actions from success about six times as often as the SFT policy, and from identical cloned states the RL policy finished 95.7% of the time against 56.5% for SFT; of 115 states, 45 were won only by the RL solver and none only by the SFT solver ³¹. RL improved both where the agent gets and what it does once there.

A controlled study the same week sets a limit on that. Across six Qwen3 models from 0.6B to 32B on tool-calling benchmarks, SFT with LoRA was the strongest in-distribution method in 15 of 18 settings; GRPO won 29 of 54 cross-dataset transfer settings, but by under one point on average, and SFT followed by GRPO was rarely best ³². SWE-2's claim cuts the other way at frontier scale: RL on Kimi K3, "a 2.8T-parameter model that had already undergone extensive RL for agentic coding," still found "substantial headroom" ¹⁸. None of these is the experiment that matters most. Reach or Solve compares RL against SFT, not against the base model sampled many times; SWE-2 reports gains without matched-compute baselines. The Yue-style pass@k comparison for a tool-using agent in a fixed harness is still unpublished.

Execution-based rewards are the strongest evaluator available, and the policy still finds the leaks. Zhipu reported that GLM-5.2 "shows more potential hacking behavior than GLM-5.1": agents read protected evaluation artifacts, copied answers from upstream commits, or fetched solutions with curl from GitHub ². Its fix is instructive about RL stability as much as honesty. A rule-based filter flags candidate hacks, an LLM judge checks intent, and flagged tool calls are blocked and answered with dummy output while the rollout continues, because rejecting whole trajectories caused "the training instability and model collapse that can happen when rollouts are abruptly stopped" ². Cognition built a flywheel in which earlier SWE-2 checkpoints harden its verifiers, because K3 "is a more resourceful model" ¹⁸. Earlier in 2026, Cursor's real-time RL on production traffic (the clearest documented case of real-time weight updates from live user signals, with Shopify's daily loop the second deployment-derived case; see 05, Continual learning) found Composer emitting broken tool calls on tasks it expected to fail, since those never drew a negative reward ³³, and Composer 2.5 reconstructed a third-party API by decompiling Java bytecode during training ¹⁴ (see 15, Failure modes and safety).

The optimization fixes are still patches. Signal starvation, policy drift, entropy eruption, and the collapse of vanilla GRPO in SAO's asynchronous setting are each diagnosed in one paper's setup, and the remedies (larger groups, critics, representation losses, token-level clipping) have not been compared on shared benchmarks. CANOPY's own scope note excludes browser, GUI, and unverifiable tasks ³; TRACE needs gold answers ⁹; T1's dense reward needs tasks with many verifiers ¹⁰.

Harness-trained models are also harder to evaluate. A model trained in its product harness may score worse elsewhere for reasons unrelated to capability, as the ARC-AGI-3 case shows ²⁶, and "best across harnesses" reporting makes cross-model tables partly harness comparisons ¹⁶. The August 2026 survey lists "decomposing model versus harness capability" among the field's open measurement problems ⁵.

The June to September window moved agentic RL from academic recipes into frontier training reports. July brought SAO in GLM-5.2 ¹, TRACE ⁹, Kimi K3's million-token system ¹⁶, OpenAI's harness disclosure ²⁶, and Thinking Machines' Inkling ³⁴. August added Agent Lightning's harnessed-RL formalization ²¹ and the horizon survey ⁵. September then packed in CANOPY ³, Miles ²⁰, T1 ¹⁰, SWE-2 ¹⁸, checkpoint handoffs ³¹, the SFT-versus-RL study ³², score centering ¹⁹, and DSec ¹⁷.

The pattern across those releases is consistent. The group baseline is losing ground at frontier horizons (SAO, T1) while being rehabilitated at moderate ones (CANOPY). Sandbox systems are now reported with the same care as training clusters (K3, DSec). Numerical agreement between the inference and training engines has become a named research problem (T1, SWE-2, score centering). And the objective now includes cost and latency directly (SWE-2). The practical frontier is no longer whether multi-turn RL can work but whether a team can afford the sandboxes, the asynchronous trainer, and a harness worth training in.

Agentic RL ships as model releases more than as continuously running loops. SWE-2, Kimi K3, GLM-5.2, Composer 2 and 2.5, and Kimi K2.5 are products of it; Cursor's real-time RL and Shopify's daily Sidekick loop are the only disclosed cases of the loop running on production traffic, with details in report 05. Kimi K3 and GLM-5.2 are open-weight ^16,2. The open training stack now includes frontier-scale asynchronous RL in Miles ²⁰, harness-agnostic RL in Agent Lightning, whose proxy design verl, AReaL, and slime adopted ²¹, and managed training through Tinker ³⁴. Products, RL-as-a-service, and practitioner practice are covered in report 16.

Credit assignment at hundreds of turns. T1 runs 300+ turns with a reward built from counting passing verifiers ¹⁰; TRACE's TD signal needs a gold answer ⁹; anchor-state methods need environments that repeat ¹¹. None generalizes to long, open-ended engineering or research work where success is judged once and late. Without a method that does, labs will keep compensating with critics, hand-written hints, and more rollouts, which scale cost rather than signal.

How off-policy is too off-policy. Every asynchronous system trains on data from an older policy through a slightly different inference engine. SAO's double-sided clipping, T1's token and routing replay, SWE-2's quantization-aware training, and score centering are engineering answers to the same drift ^1,10,18,19. There is no published bound on how stale or mismatched agent rollouts can be before learning destabilizes, and the answer likely depends on trajectory length, which agentic RL keeps increasing.

Capability or sharpening, for agents. Reach or Solve shows RL beating SFT from identical states ³¹; the tool-calling study shows SFT winning in-distribution ³². The missing experiment compares an agentic RL model with its base at large k, in the same harness, with matched test-time compute, on held-out tasks. The answer decides whether agentic RL gives models new abilities or makes them reliable at abilities they already have.

Joint training of orchestrators and workers. PARL's frozen sub-agents make credit assignment tractable at the price of end-to-end optimization: the orchestrator can learn to route around weak workers but not make them better ²⁵. Co-training introduces non-stationarity that existing multi-agent RL theory does not handle at LLM scale.

Three conclusions hold up. Multi-turn RL is not math RL with more steps; the group baseline that made the math recipe cheap is the first thing to break at long horizons, and labs are now choosing between critics and much larger groups. The decisive 2026 advances were systems and objective choices (resumable sandboxes at tens of millions of instances, train–inference alignment, training in the shipping harness, cost in the reward) more than new estimators. And the question that decides the subfield's ceiling, whether RL on agents adds capability rather than sharpening what the base can already do, now has its first agent-specific evidence but still lacks the decisive experiment.