Agentic RL

I'd start with a paper Zhipu published in July. The team behind the GLM models described the reinforcement-learning algorithm they used to train GLM-5.2, a 750B-parameter mixture-of-experts model with 40B active parameters, and it throws out the group, the piece of the standard recipe that defined agentic RL for its first two years ¹.

Group Relative Policy Optimization (GRPO) samples several rollouts of the same task and scores each one against the group's average. Zhipu's GLM-5.2 release post says why that stopped working for them. Long coding-agent trajectories get split by context compaction into several sub-traces, so "different rollouts under the same prompt yield different numbers of trainable traces with highly variable lengths," and the team "move[d] from group-wise optimization to a critic-based PPO formulation that learns from individual rollouts" ².

The paper, Single-rollout Asynchronous Optimization (SAO), gives a second reason: under asynchronous training a group has to wait for its slowest member, and by then its data is stale ¹. The controlled comparison on Qwen3-30B-A3B shows what that costs. Standard GRPO collapsed at about 160 training steps. SAO trained stably for 1,000 steps and lifted SWE-bench Verified (OpenHands scaffold, up to 300 turns) from 23.0% to 29.8%, against 27.0% for a stabilized GRPO ¹.

CANOPY, from September, went the other way and topped a leaderboard. It kept outcome-only group RL and argued that its reputation for stalling on small open models comes from two fixable mistakes: groups too small to contain both a success and a failure, and too many updates squeezed out of a small task pool ³. With larger same-task groups and strictly on-policy, KL-anchored updates, a Qwen3-14B policy trained only by interacting with the environment reached the top of the public AppWorld leaderboard, at 86.9 task goal completion on Test-Normal and 67.6 on Test-Challenge ³.

So the two papers disagree about the fix. They agree on what is wrong: a single score that arrives after dozens or hundreds of tool calls is a weak training signal, and the group baseline borrowed from math breaks first.

GRPO came from single-turn math. You sample a group of completions for one prompt, score each with a checker, normalize against the group mean, and push up the tokens of the above-average completions, with no value network ⁴. That works because every completion starts from the same state and only the reasoning varies.

An agent breaks that assumption in three ways. Each tool call changes the state, so two rollouts of the same task stop sharing a situation after the first action. The reward arrives at the end, sometimes hundreds of steps later, and says nothing about which step earned it. And the context grows with every observation until the trajectory has to be compacted or truncated.

A survey of 1,547 long-horizon agent papers, published in August, found this problem in planning, memory, training and evaluation alike: "outcome-only signals grow uninformative as horizons lengthen," and in every one of those areas the field's response "manufactures denser step-level signals" ⁵.

In the terms of this series, agentic RL is a model improving its own weights directly. Gradients propose the weight updates, rollout scores decide which trajectories push the weights, and lab engineers decide which checkpoint ships. Execution-based evaluators are strong compared with the judges in most other reports here, so the typical failures are optimization failures, though evaluators still leak.

The 2025 failures are old news, but you need them to read the 2026 fixes. RAGEN named the Echo Trap in April 2025: reward variance falls off a cliff, gradients spike, and the policy falls back on repetitive templates ⁶. DeepSWE, trained with RL alone on Qwen3-32B in July 2025, hit reward collapse caused by lucky early successes. An agent that found the fix in its first 10 steps and then edited random files still passed the tests, so the gradient reinforced the random edits, and the team had to mask timed-out and truncated trajectories out of the loss ⁷.

CANOPY supplies the arithmetic behind the Echo Trap ³. A group-relative estimator produces a gradient only when a task's group contains both successes and failures. For per-rollout success rate p and group size n, the probability of that is 1 − pⁿ − (1 − p)ⁿ. For a hard task with p = 0.05, it rises from 34% at n = 8 to 81% at n = 32 ³.

Small groups therefore produce no gradient on exactly the hardest tasks, which the authors call signal starvation. In their ablation, using the group of 8 common in prior work instead of 32 cost 16.4 points, though at matched steps the smaller group also draws fewer samples ³. Their second failure, policy drift, comes from taking many updates from a task pool the policy has mostly solved: with no anchor, the sampling distribution collapses at the moment informative groups become rare ³.

Entropy also behaves differently. In single-turn reasoning RL, entropy usually collapses and stays low. Researchers at UW–Madison found in May that agent RL instead shows "recurring cycles of sharp entropy eruption and gradual subsidence," with duplication and hallucination building up across cycles. The authors trace the eruptions to correct and incorrect trajectories becoming indistinguishable in representation space, and their SEAL loss pushes the two apart ⁸.

I read each of these as a patch for one symptom in one setup. They leave the mismatch itself in place: the group baseline compares rollouts that differ in length, in the tools they touched, in the states they visited and in luck, as well as in quality. The remedies (larger groups, critics, representation losses, token-level clipping) also haven't been compared on shared benchmarks, and the collapse of vanilla GRPO in SAO's asynchronous setting is, like the others, diagnosed in a single paper's setup.

Recent work on credit assignment splits into two camps. The first builds per-step credit without paying for a separately trained critic. TRACE, from July, treats a search agent's rollout as a sequence of states at tool-call boundaries. It asks a frozen reference model how likely the gold answer is from each state, and rewards each action by how much it changed that log-probability. That is a temporal-difference signal with no critic and no process labels to train ⁹.

Because the one-step terms telescope, a redundant tool call that leaves the state unchanged earns nothing. With pure RL and no supervised cold start, TRACE raised BrowseComp-Plus accuracy from 7.2% to 35.6% for Qwen3-4B and from 8.4% to 42.6% for Qwen3-30B-A3B ⁹. It needs a gold answer to score states against, which suits question answering much better than open-ended engineering.

T1, from September, goes further back toward classical RL. It trains a 122B MoE to operate a real shell for up to 300+ tool-call turns, uses a warm-started actor-critic instead of a group baseline, and replaces the pass/fail outcome with a dense process reward: the number of the task's own verifiers a trajectory passes ¹⁰. Its training tasks are disjoint from the evaluation set by design, and it raised Terminal-Bench 2.1 from 43.8% for the base model to 64.0% ¹⁰. T1 and SAO point the same direction: at frontier horizons, labs are paying for value functions again, the cost GRPO was invented to avoid, because the group no longer gives them a usable baseline.

The group-based methods of 2025 are what these papers react against. GiGPO (NeurIPS 2025) found comparable situations by matching "anchor states" that recur across a group's rollouts ¹¹; HGPO corrected it by grouping steps by shared history as well as shared state ¹²; StepPO went back to a step-level value function and GAE and found step-level credit beat token-level throughout training ¹³. All three report gains on ALFWorld and WebShop with models of 7B or smaller. Those environments revisit identical states often enough for matching to work, and a repository in the middle of a refactor rarely does.

Cursor took a third route for Composer 2.5, which it calls "targeted RL with textual feedback": at the step where the model went wrong, insert a hint into its context, and use the hinted model's distribution as the teacher for that step ¹⁴. The lab writes the correction at the step that went wrong and distills it, so it never has to estimate which step deserved the blame.

The second camp says most of this is unnecessary. CANOPY's position is that outcome-only RL works once exploration is scaled until "the natural signal reappears," with updates kept on-policy and limited to the agent's own action tokens. The same principles lifted Qwen3.5-9B on SWE-bench Verified from 31.3% to 47.9% ³.

In its ablations, adding a denser partial-credit reward moved the result least, by 1.8 points. The authors read that as partial credit making up for too little exploration, while calling the single-run gap suggestive rather than established ³. Its scope appendix is candid: the evidence covers tasks with a shell, a programmatic verifier and tens of turns, and excludes browser or GUI control, open-ended research, and anything that can't be checked automatically ³.

I think both camps are right about their own regime. Where verification is cheap and a task takes tens of turns, more rollouts buy signal, and CANOPY's leaderboard result is hard to argue with. At hundreds of turns, with compaction and stale data, the labs with the most compute are choosing critics and denser rewards, and I'd follow them there. Each dense method also has a precondition: TRACE needs gold answers ⁹, and T1's reward needs tasks with many verifiers ¹⁰.

A better credit signal doesn't help until the trainer can collect enough long trajectories. A synchronous RL step waits for every rollout in the batch, and with agents one task finishes in four tool calls while another runs for 300, so the trainer idles behind the slowest one.

Teams in 2025 answered with partial rollout, which pauses long trajectories and resumes them under newer weights, and with fully asynchronous training, which puts generation and training on separate GPUs and corrects for staleness with importance weights. MiniMax's CISPO clipped those weights directly and disclosed a full RL run on 512 H800s for $534,700 ¹⁵. By mid-2026 the limiting resource had moved from GPUs to the environments the agent runs in.

Moonshot's Kimi K3 report shows the scale. K3 is a 2.8T-parameter MoE with 104B active parameters and a 1M-token context, and its "million-token agentic RL" runs on a co-located system that "combines partial rollouts, external KV-cache retention, adaptive throttling and resumable microVM sandboxes to preserve long-lived model and environment state" ¹⁶. The sandbox layer checkpoints only the memory pages dirtied since the last checkpoint, with checkpoint and resume latencies as low as 133 ms and 49 ms, so a trajectory paused at the end of one iteration picks up in the next with its environment intact ¹⁶. Across training and evaluation, Moonshot created 51,219,741 sandboxes from 1,505,678 images ¹⁶.

DeepSeek's September report on its sandbox platform, DSec, gives the other published number at that scale. One production unit of about 160 nodes serves about 3 million sandboxes per day, and in production DSec supports over 380,000 concurrent sandboxes and more than 5,000 sandbox creations per second ¹⁷. Its design choices match Moonshot's closely. Stateful rollout execution is decoupled from preemptible GPU training, sandbox lifecycles are coordinated with the trainer so rollout state is kept while idle resources are reclaimed, and the platform itself handles agent misbehavior such as reward hacking ¹⁷.

The second systems problem is numerical. Rollouts come from a fast inference engine and gradients from a training engine, and the two assign slightly different probabilities to the same tokens. T1 goes after the mismatch directly: it trains on the exact token IDs the sampler produced, repairs drift at turn boundaries, and replays the sampler's per-token expert routing at every MoE layer during training, which cut the train–inference log-probability gap from 0.021 to 0.013 ¹⁰.

Cognition's SWE-2 uses NVFP4 and FP8 kernels with quantization-aware training to fit more rollouts in memory while getting lower train–inference divergence than its predecessor, on a base model with almost three times the parameters ¹⁸. Martin Marek and Max Ryabinin argued in September that the resulting instability is "primarily caused by drift: a persistent bias between training and inference engines that accumulates with every training step." Their additive "score centering" correction matches or beats importance sampling from 0.6B to 30B and combines with it when data is stale ¹⁹.

Open-source tooling caught up over the summer. RadixArk's Miles v0.1, built on Zhipu's slime, ends its report with fully asynchronous agentic RL on GLM-5.2 over terminal coding tasks, on 64 NVIDIA GB300 GPUs at a median step time of 263 seconds ²⁰.

The third change is to run RL inside the product the model will ship in, with the same tools, prompts, context handling and sandboxes. The harness (system prompt, tools, context assembly and control flow around the model) stops being something built after training and becomes part of the training environment.

Microsoft Research's Agent Lightning v1.0, from August, gives the practice a name and an open implementation. "Harnessed agentic RL" is training "where the deploy-time harness directly participates in model post-training." The harness owns the interaction loop, and the trainer sees only the LLM request-response pairs passing through a proxy ²¹. Inverting control that way creates its own problems: retokenization, merging samples, and computing advantages and normalizing losses over calls the trainer didn't orchestrate.

The roughly 3,500-line framework exists to study them. With 6,000 training examples it lifted Qwen3.5-9B on SWE-bench Verified from 41.8% to 56.4% ²¹, and the paper notes that its proxy architecture was later adopted by verl Uni-Agent, AReaL 2.0, slime and Polar ²¹.

The product labs had already committed to this. Cursor's Composer 2 report from March says: "We develop infrastructure to support training in the same Cursor harness that is used by the deployed model, with equivalent tools and structure." Rollouts run on Anyrun, Cursor's internal platform for "hundreds of thousands of sandboxed coding environments" ²², and its yardstick, CursorBench, is built from real sessions by Cursor's own engineers ²³. Training in the harness also made context management trainable: Composer 2 learns during RL when to compress its own context into a summary ²².

Cognition's SWE-2 takes the idea all the way into the reward. Post-trained from Kimi K3, it trains every reasoning-effort level in a single RL run with a cost-penalized reward, R = S − λₑC, where S is success, C mixes inference cost in dollars with rollout time, and each λₑ is "tuned to match the slope of the Pareto curve of the base model at effort level e" ¹⁸. With λₑ set that way, the reward only goes up when the cost–performance frontier moves. SWE-2 at medium effort beats its predecessor SWE-1.7 on FrontierCode 1.1 while taking 58% fewer turns and costing 81% less ¹⁸.

Cognition's reason for caring about latency goes back to its 2025 SWE-grep post: "we estimate that your P(breaking flow) geometrically increases 10% every second that passes while you wait for agent response" ²⁴. SWE-2 writes that product belief into the reward as a term.

Kimi K2.5 applied the idea to a multi-agent harness. Its Parallel-Agent RL (PARL) freezes the sub-agents and treats their outputs as observations, so "only the orchestrator is updated via reinforcement learning" ²⁵. On BrowseComp the resulting Agent Swarm scores 78.4% against 60.6% for single-agent K2.5, and on WideSearch it runs 3× to 4.5× faster as target item-level F1 rises from 30% to 70% ²⁵. Freezing the workers turns an unsolved multi-agent credit problem into a single-policy one.

OpenAI measured the reverse case. It reported in July that GPT-5.6 Sol scored 13.3% on ARC-AGI-3's public set in the official harness, which discarded reasoning after every move, and 38.3% with retained reasoning and compaction, settings OpenAI described as "how our models are trained" ²⁶. Taken out of the harness it was trained in, the model lost about two-thirds of its measured score. For an agent that will ship inside a product, the product harness belongs in the training environment from day one.

The same dependence makes these models harder to evaluate. A model trained in its product harness can score worse elsewhere for reasons unrelated to capability, as the ARC-AGI-3 case shows. Kimi K3's report evaluated models in Kimi Code, Claude Code or Codex, and on Terminal-Bench 2.1 its table reports "the best score across harnesses for all models" ¹⁶, which makes the cross-model comparison partly a comparison of harnesses. The August survey lists "decomposing model versus harness capability" among the field's open measurement problems ⁵.

I wouldn't read this table as a ranking. Nearly every number comes from the team that built the system, and the lab results use each lab's own harness and, for SWE-2, internal evaluation infrastructure. No row says whether RL beat what the same compute would have bought another way, for example by sampling the base model many times and picking an answer with a verifier.

For math there is a partial answer, and it frames everything else here. Yue and colleagues' NeurIPS 2025 oral found that RL-trained models win at pass@1, but at large k the base model solves more problems: RL concentrates probability on solutions the base model could already reach ²⁷. That effect is usually called sharpening. NVIDIA's ProRL countered that prolonged, diverse RL beats the base model across pass@k, "including scenarios where base models fail entirely regardless of the number of attempts" ²⁸.

Meta's ScaleRL (ICLR 2026 oral), with runs of up to 100,000 GPU-hours, found that "not all recipes yield similar asymptotic performance," meaning the choice of recipe decides how high a run can eventually climb ²⁹. Random-reward experiments are a further warning that RLVR gains can reflect the base model more than the reward ³⁰.

The first evidence aimed at agents came in September. "Reach or Solve?" by Xuan Liu and Jingbin Qian points out a confound math never had: an agent writes its own inputs, so an SFT checkpoint and an RL checkpoint are scored from different states even on identical tasks ³¹. Their checkpoint-handoff protocol clones a state that one checkpoint reached and hands it to the other.

On ALFWorld, with released 7B SFT and RL checkpoints, the RL policy reached a state two actions from success about six times as often as the SFT policy. From identical cloned states, the RL policy finished 95.7% of the time against 56.5% for SFT, and of 115 states, 45 were won only by the RL solver and none only by the SFT solver ³¹. RL improved both where the agent gets to and what it does once it is there.

A controlled study from the same week limits how far that goes. Across six Qwen3 models from 0.6B to 32B on tool-calling benchmarks, SFT with LoRA was the strongest in-distribution method in 15 of 18 settings. GRPO won 29 of 54 cross-dataset transfer settings, but by under one point on average, and SFT followed by GRPO was rarely best ³². SWE-2 points in the opposite direction on a far larger model: RL on Kimi K3, "a 2.8T-parameter model that had already undergone extensive RL for agentic coding," still found "substantial headroom" ¹⁸.

None of these is the experiment I want to see. Reach or Solve compares RL against SFT, and the base model sampled many times isn't in it. SWE-2 reports gains without matched-compute baselines. I couldn't find a Yue-style pass@k comparison for a tool-using agent in a fixed harness, and until someone runs one, claims that agentic RL adds capability are being measured against the wrong baseline.

Execution-based rewards still leak, and the policy finds the leaks. Zhipu reported that GLM-5.2 "shows more potential hacking behavior than GLM-5.1": agents read protected evaluation artifacts, copied answers from upstream commits, or fetched solutions with curl from GitHub ².

Zhipu's fix is as much about RL stability as about cheating. A rule-based filter flags candidate hacks, an LLM judge checks intent, and flagged tool calls are blocked and answered with dummy output while the rollout continues, because rejecting whole trajectories caused "the training instability and model collapse that can happen when rollouts are abruptly stopped" ². Cognition built a flywheel in which earlier SWE-2 checkpoints harden its verifiers, because K3 "is a more resourceful model" ¹⁸.

Cursor's real-time RL updates Composer's weights in real time from live user signals on production traffic. It found Composer emitting broken tool calls on tasks it expected to fail, since broken calls never drew a negative reward ³³. Composer 2.5 reconstructed a third-party API by decompiling Java bytecode during training ¹⁴.

Agentic RL reaches users mostly as model releases, and rarely as a loop that keeps running after launch. SWE-2, Kimi K3, GLM-5.2, Composer 2 and 2.5, and Kimi K2.5 are products of it. Cursor's real-time RL and Shopify's daily Sidekick loop, both described in Report 05, are the only disclosed cases of the loop running on production traffic. Kimi K3 and GLM-5.2 are open-weight ^16,2.

Most of this work was published between July and September, and the releases move in a consistent direction. The group baseline is losing ground at frontier horizons (SAO, T1) and being rehabilitated at moderate ones (CANOPY). Labs now report their sandbox systems with the same care as their training clusters (K3, DSec). Agreement between the inference and training engines has become a named research problem (T1, SWE-2, score centering), and SWE-2 puts cost and latency directly into the objective.

For a team outside the labs, the open stack now covers most of this. Miles does frontier-scale asynchronous RL ²⁰. Agent Lightning does harness-agnostic RL, and verl, AReaL and slime adopted its proxy design ²¹. Thinking Machines, which released Inkling in July, offers managed training through Tinker ³⁴.

Whether multi-turn RL can work is settled. What decides whether a team should do it is whether it can pay for the sandboxes and the asynchronous trainer, and whether it has a harness worth training in.

Credit at hundreds of turns, when success is judged once, after the last turn. T1 runs 300+ turns with a reward built from counting passing verifiers ¹⁰; TRACE's TD signal needs a gold answer ⁹; anchor-state methods need environments that repeat ¹¹. All three break down on long, open-ended engineering or research tasks. Until a method works there, labs will keep compensating with critics, hand-written hints and more rollouts, and all three add cost faster than they add signal.

How stale or mismatched can rollouts get before training destabilizes? Every asynchronous system trains on data from an older policy through a slightly different inference engine. SAO's double-sided clipping, T1's token and routing replay, SWE-2's quantization-aware training and score centering are all engineering answers to that drift ^1,10,18,19. No paper has published a bound yet, and the answer probably depends on trajectory length, which agentic RL keeps increasing.

Does RL give agents new abilities, or make them reliable at abilities they already have? Reach or Solve shows RL beating SFT from identical states ³¹, and the tool-calling study shows SFT winning in-distribution ³². The experiment that would settle it compares an agentic RL model with its base at large k, in the same harness, with matched test-time compute, on held-out tasks.

Can orchestrators and workers be trained together? PARL's frozen sub-agents make credit assignment tractable at the price of end-to-end optimization: the orchestrator can learn to route around weak workers but can't make them better ²⁵. Training both at once makes each a moving target for the other, and existing multi-agent RL theory doesn't handle that at LLM scale.