What ships

Decagon sells customer-service agents, and each one runs on Agent Operating Procedures: natural-language documents that say how to handle a refund, when to escalate, what tone to use. Its Duet Autopilot reads the agent's conversations, proposes edits to those procedures, tests them in simulation, and stages them for a human reviewer. In a September 9 post called DuetBench-2, Decagon tried to answer the question every buyer of such a loop should ask, which is whether the changes do any good once they reach production ¹.

At one Fortune 50 customer, a series of Autopilot changes, each worth roughly one or two percentage points, raised the agent's resolution rate from 30% to 55% ¹. Decagon also compared each procedure before the edit, at merge, and today. More than 85% of Autopilot's updates were still live 30 days later, complete reversions or removals were under 15% of changes, and nearly half remained fully intact after the first week ¹.

Duet and Autopilot now author 70% of merged procedure lines across Decagon's production teams ¹. A smaller example shows what a single edit looks like: Autopilot found an account-recovery procedure where the agent escalated too eagerly, and its fix raised deflection 19.4% within the modified section, 4.2% across the procedure, and roughly 2 points team-wide ¹.

Decagon checked the parts the loop depends on, too. Its self-generated simulations reached a 96% quality pass rate after climbing more than 30% between August 5 and August 26, and its LLM judge reached 99.5% accuracy against human-labeled ground truth and production outcomes ¹. It points the loop at its own tooling; its example is a Duet simulation runner that fell into an unbounded loop, sending 10 to 40 near-identical status messages in roughly 55 conversations a week ¹.

The benchmark itself moved on because the first one stopped being hard. The June DuetBench measured Autopilot against Decagon's own human agent builders: it "passed every check on 93% of tasks" on diagnostics against an 83% human average, and passed 45.5% of build tasks against 23% for humans working within three hours ¹¹. Autopilot saturated it within weeks, and the September version asks three harder questions: can it build and revise procedures against a stricter bar (81.9% pass rate), do its changes move production metrics, and are they retained ¹?

I take these numbers seriously. On its own traffic, a vendor has shown that accepted changes mostly persist and that a sequence of small edits can move a headline metric by 25 points. That is more than anyone else in the category has shown. But the before/after comparisons have no control arm, so seasonality, product changes and model upgrades are all mixed into the 25 points.

The Fortune 50 result is one customer, and the vendor chose it. And retention measures whether customers kept a change. A change that hurts some untested case gets kept as easily as one that helps, so an 85% retention rate tells you how often reviewers were unhappy enough to undo something, which is a different number from how often the change helped.

The reviewers are the other problem. In a companion post the same day, a Decagon customer described the human gate: "It's those kinds of updates that lead us to consistently accept almost 100% of Autopilot's suggestions" ². I read that as praise for the proposer. It also means the gate is filtering almost nothing, however good the proposals are. Jayakumar Ramalingam, an engineer quoted by The New Stack about Anthropic's memory consolidation feature, put it this way: "Human review sounds reassuring, but at fleet scale it can easily become a rubber stamp for recommendations nobody has time to reconstruct" ¹².

Decagon's design is not unusual. Between May 2025 and September 2026, products that compete for the same budgets shipped variants of one architecture without a shared paper behind it: an LLM reads production conversations and scores them, a second agent proposes edits to the deployed agent's instructions or configuration, a simulator replays the proposal against past conversations, a human approves, and the change deploys ^{5,6,7,8,9,10,3}.

The convergence follows directly from the business these platforms are in. They run on frozen frontier models they don't own, for customers who own the business logic. The customer's refund policy, escalation rules and tone are written in natural-language procedures, and the customer's compliance team wants to read every change. Retraining isn't an option, and text edits are cheap, readable and easy to revert. LLM judges also became cheap enough to read every conversation, which removed the old bottleneck of a human sampling transcripts.

Salesforce's Agent Optimizer, which shipped in beta on September 21, is the newest version and names each step of the loop. In Agentforce Observability it analyzes "hundreds or thousands of sessions," groups failures by tool-call errors, knowledge gaps or off-topic requests, and recommends a fix. In Agentforce Builder it implements the change, simulates conversations, and turns those previews into reusable regression tests, so the next change can check that earlier cases still pass ³.

Customers "define the outcomes that matter," such as resolution rate or CSAT, and choose through an "autonomy dial" whether to require sign-off at every step or let the optimizer investigate, build, test and stage changes on its own ³. Salesforce had described the target in July as a hill climb over instructions, knowledge, tool specifications and policies across more than 11 million Agentforce calls a day ⁹.

Decagon launched Autopilot on June 9 and added governance features in September ^5,2. A "Vault" stores the context Autopilot works from (guidelines, tone examples, off-limits guardrails) as versioned files with an owner and an audit trail. Teams specify which outcome wins when fixes compete: deflection, CSAT, handle time, or a custom metric. Autopilot now runs two tracks: trigger-based runs that act the moment an error alert fires, and a "hillclimbing" track that periodically audits the whole agent for deflection and CSAT opportunities. Reviewers see a projected deflection impact before accepting a change and the measured result afterward ².

The others differ in packaging. Sierra split the loop across two products in the spring: Ghostwriter builds and updates agent journeys and "proactively runs tests with every build or update," and Explorer researches customer conversations and emits recommendations the customer can "review, approve, and ship" ^6,7.

Intercom's Fin issues weekly recommendations for gaps in content, customer data and actions, batch-tests changes against past conversations, and sends them to the team for review ⁸. Factory's Signals applies the loop to Factory's own coding agent: an LLM pipeline detects friction such as repeated rephrasing or abandoned tool flows and files a ticket, Factory's Droid writes and reviews the fix, and "a human still approves the PR before merge" ¹⁰.

Read down the columns and every row is the same design. The change is written into text the customer can read, an LLM proposes it by reflecting on traces, an LLM judge plus replay tests it, and a person decides whether to keep it. Every one of these vendors could have closed the loop and chose not to, because whoever approves the change is also who is liable for it. Salesforce's "autonomy dial" and Decagon's trigger-based track show that approval step starting to loosen, one customer setting at a time.

The evidence from the vendors other than Decagon is thin. Factory reports that the "repeated rephrasing" friction rate fell 30% within 48 hours of one fix, with no control ¹⁰. Salesforce cites customer outcomes for Agentforce agents (Engine resolves 50% of chat inquiries) but none for the optimizer itself ³. Sierra and Intercom publish no outcome metrics for their loops ^6,8. PACE is relevant here: it showed that a greedy "keep it if the score went up" rule commits false edits at high rates (see 06), and a reviewer approving a change because its simulation passed is running that same rule, with better priors and less patience (see 14).

Coding is where nearly all of the documented weight updates after deployment happen, because code produces cheap, dense signals that are hard to fake: whether a suggestion was accepted, whether an edit stayed in the repository, whether tests pass, whether the user pasted a stack trace next. Cursor and Cognition have the volume and the signals, so they train.

Cognition's SWE-2, released September 10, is the newest disclosure. Cognition post-trained it from Moonshot's 2.8-trillion-parameter Kimi K3, scaling its RL "to the multi-trillion-parameter regime for the first time," and reports 50.0% on its FrontierCode 1.1 benchmark, within a point of Fable 5.1 at 64% lower cost ¹³.

The self-improving part is in the data pipeline. Cognition tripled its RL environments and built "a flywheel powered by previous checkpoints of SWE-2 that iteratively hardens our verifiers." The stronger base model was more resourceful at gaming rewards, so engineers mined training rollouts for false positives and false negatives and used the model's own earlier checkpoints to patch the verifiers ¹³. Each model generation improves the checker that trains the next one, with engineers supervising. Cognition has trained in its own agent harness since SWE-1.5 in October 2025 ¹⁴; report 01 explains why that choice affects what the model learns.

Cursor runs real-time RL on live user signals, and Shopify's daily loop, which turns production failures into SFT and GRPO data, is the second weight loop built on deployment data (see 05). Composer's real-time RL, described in March, ships an improved checkpoint as often as every five hours, rewarding edits that persist in the codebase and penalizing dissatisfied follow-ups, and Cursor reports +2.28% edit persistence and −3.13% dissatisfied follow-ups; the Tab model has run a similar loop since 2025 ^15,16.

The loop found its evaluator's blind spots almost immediately, from deliberately broken tool calls that "would never receive a negative reward" to decompiled Java bytecode during Composer 2.5's training ^15,17. Cursor became part of SpaceX on August 14, and its blog now announces Grok models "trained on a wide range of agentic RL tasks," but it has disclosed no new production-learning results since May ^18,19.

The Cursor result I'd expect more teams to be able to copy is Bugbot, its code-review agent, which learns rules in text from how developers respond to its comments. The share of flagged bugs that got resolved rose from 52% at its July 2025 launch to 78.13% in April 2026, with more than 110,000 repositories enabling learning and more than 44,000 learned rules generated. In Cursor's own comparison, Greptile was at 63.49%, CodeRabbit at 48.96%, GitHub Copilot at 46.69% and Codex at 45.07% ²⁰.

Resolution rate is a production metric with a real denominator, which is rare in this category. It is also a vendor-run time series over months in which the underlying models changed, so it can't isolate what the rules contributed (see 14). Cursor's harness team runs the human version of the loop, with a Keep Rate metric and online A/B tests, and spends weeks customizing the harness for each new model ²¹.

Most coding teams don't train anything. Warp described what they do instead in an August write-up ⁴. Warp's code-review, triage and spec-writing agents each run on a base skill file. A separate improver skill runs on a schedule, "pulls the accumulated human feedback, compares what the agent suggested against how humans responded, and proposes a small, focused edit to the base skill," then opens a pull request explaining which feedback prompted the change. "A human reviews, approves, and merges, and the next run of the triage skill inherits the new knowledge" ⁴.

Warp runs this across its open-source repository, where hundreds of contributors and thousands of code reviews feed it. Its advice on the possibility that feedback is wrong is blunt: "Assume it will be" ⁴. Warp publishes no before/after numbers, and suggests tracking time to merge, contributor count and cost.

The rest of the market writes text too. Augment's Cosmos propagates one engineer's correction to every agent on a team through a shared tenant memory ²². GitHub has described an agentic memory system for Copilot ²³. Amazon's Kiro ships hooks, specs, steering files and Agent Skills ²⁴. OpenAI's Codex now offers local memories, off by default: a background pass extracts them from idle chats and a separate model periodically consolidates them, and a setting keeps chats that used MCP tools or web search out of memory generation ²⁵.

Claude Code has the most mechanisms: CLAUDE.md files, a memory tool, skills, hooks, and, since late May, dynamic workflows ^26,27. A dynamic workflow is a JavaScript harness Claude writes for one task. It is launched through a permission-gated tool call that prompts in manual modes and runs unprompted in bypass mode, claude -p and the Agent SDK ²⁷. Good runs can be saved as reusable commands, but only when a human presses a key (see 10).

Anthropic's launch post also describes the team's own procedure for maintaining rules. They mine sessions and review comments for repeated corrections, cluster them, adversarially check whether each candidate rule "would have prevented a real mistake," and distill the ones that pass into CLAUDE.md ²⁷. Of all the coding-tool practices I read about, that adversarial check is the step I'd copy first, because it puts a skeptical filter in front of the rule file.

Rule files also go stale when the model changes. An Anthropic engineer wrote in July that the team "removed ~80% of the Claude Code system prompt for our newest models" ²⁸. Instructions that once taught the model how to behave became redundant once the next model had absorbed the behavior (see 12).

The frontier labs ship the building blocks and leave the loop to customers. Anthropic's Managed Agents added "dreaming" as a research preview in May: an asynchronous job reads a memory store and up to 100 past session transcripts and produces a new, reorganized store, with duplicates merged and stale entries dropped ²⁹. The design keeps a gate by construction: "The input store is never modified, so you can review the output and discard it if you don't like the result" ²⁹.

Critics quoted in The New Stack argued the gate is weaker than it looks, because a bad answer "dies with the session" while "a bad memory can influence thousands of future sessions," and they said every consolidated memory should "carry provenance, evidence and an expiration condition" ¹². OpenAI's Codex memories follow the same extract-then-consolidate pattern locally ²⁵. Neither product publishes a measurement of whether memory improves outcomes.

The Agent Skills format, a SKILL.md file of instructions with optional scripts, is now shared across Claude, Codex, Kiro, OpenClaw, Hermes and others. That makes a skill the closest thing the industry has to a portable unit of learned procedure (see 08) ^24,30.

For weights, OpenAI's Reinforcement Fine-Tuning API runs RL on o4-mini against customer-defined graders, warns about reward hacking in its own documentation, and blocks deployment of a tuned model that fails safety checks ³¹. Google made AlphaEvolve generally available on Google Cloud on July 9: customers supply a seed program and an evaluator, and the service evolves code ³². In every case the customer has to bring the evaluator, and writing a good evaluator is the hard part.

A market has grown up around each piece of the loop. Vendors sell storage, proposers and compute, and in every case the customer still has to supply the verifier.

Environments got the newest money (see 02). Realset AI and Flatkey announced a $10M Series A on September 23; Realset captures expert human demonstrations and builds RL environments from real workflows in e-commerce, customer support, logistics and manufacturing, on the pitch that "the easy data is gone" ³³.

Business Insider reported in September that Google had completed its $1.5 billion-plus talent deal for Mechanize, which builds environments and evals for frontier coding agents and had raised $9.1M at a $500M valuation in April ³⁴. Cognition's tripled environment count for SWE-2 is the same demand seen from the buyer's side ¹³. The vendors have published no evidence that training on their environments beats the alternatives at matched budgets.

Several companies now sell RL as a service. Prime Intellect's Lab, launched in February, combines its Environments Hub (which now advertises more than 2,500 open-source RL environments) with hosted agentic RL on LoRA adapters and hosted evaluations, priced per token ^35,36. Thinking Machines' Tinker exposes training as four primitives (forward_backward, optim_step, sample, save_state) over LoRA adapters on open models ³⁷, and CoreWeave has sold serverless RL through W&B Training since acquiring OpenPipe in 2025 ³⁸.

Memory companies raised money earlier and publish the least evidence. Mem0 closed a $24M Series A in late 2025 and has more than 62,000 GitHub stars ³⁹. Letta's June "memory models" post proposes models trained with memory-native RL to write token-space memories that still work across model generations ⁴⁰. Zep, Supermemory and others compete in the same category, and Synth AI sells online prompt optimization for production agents (see 07) ⁴¹.

No memory vendor has published a controlled benchmark showing that its memory layer improves task outcomes in production, and an independent result points the other way: CL-Bench found in June that "naive ICL outperforms systems dedicated to memory management" across six expert-validated domains ⁴². Until one of these vendors shows otherwise, I would test any memory layer against simply putting the relevant history in the prompt.

The open-source mass market went a different way. OpenClaw shipped what its maintainers called "by far the largest update in the history of OpenClaw" on August 30: 933 contributors, more than 16,000 pull requests, and changes to installation, messaging, memory, skills and models ⁴³. The repository had 390,396 GitHub stars on September 24, and its official hub listed about 67,300 skills ^44,45.

A day after OpenClaw's update, Nous Research released Hermes Agent v0.21.0, with about 5,800 commits from more than 760 contributors since the previous version, including memory for scheduled jobs so they can learn between runs ⁴⁶. Hermes, at 248,668 stars, advertises itself as "the only agent with a built-in learning loop" ⁴⁷.

The Hermes documentation describes what that loop does and who approves it. The agent writes, patches and deletes its own skills through a skill_manage tool when it works out a multi-step workflow, hits a dead end and finds the working path, or is corrected by the user, and a background self-improvement review runs after turns ³⁰.

Version 0.21.0 made writes to protected agent-instruction files (AGENTS.md, skills, memory stores) always require approval, "so a prompt-injected agent can't quietly rewrite its own standing orders" ⁴⁶. The skills documentation still describes approval of agent-created skills as opt-in: "By default the agent writes skills freely," unless the user turns on skills.write_approval ³⁰. Hub installs and repository skills pass through a security scanner, and repository skills with a dangerous verdict are quarantined ³⁰. So Hermes has the enterprise loop's proposer, and it is adding the approval step one file type at a time.

That approval step is overdue, because the supply chain is already compromised. An HN post in February titled "Top downloaded skill in ClawHub contains malware," linking 1Password's analysis of how OpenClaw skills become an attack surface, reached 334 points. One reply read, "It's kind of interesting how with vibe coding we just threw away 2 decades of secure code best practices xD..." ^48,49.

The ClawHavoc campaign raised the confirmed count of malicious skills on ClawHub to 1,184 by June, many of them attacking through natural-language instructions instead of code ⁵⁰. A skill is a set of instructions the agent follows with the user's permissions, so a poisoned skill is a prompt injection with an install count.

The practitioner loops with real verifiers skip the human and keep whatever improves a number. Andrej Karpathy's autoresearch, released in March, has an agent edit a small training script, run five-minute experiments, and keep changes that lower validation bits per byte. Over roughly 650–700 experiments the changes transferred to the external nanochat "Time to GPT-2" leaderboard as an ~11% speedup, from 2.02 to 1.80 hours ⁵¹.

Elon Musk replied, "We are in the Singularity"; the top HN critique asked whether the gains "were all basically hyperparameter changes right?" ^52,53. SkyPilot's scaled run found the agent routing screening work to H100s and validation to H200s without being told to ⁵⁴.

By July the pattern had become a contest format. In GPU Mode's auto-research competition, Sankalp Shubham used Codex to reach a 232× speedup over the baseline QR-decomposition kernel (about 419,000 µs to 1,805 µs), placing 12th of 183 ⁵⁵.

His write-up records where the loop stopped helping: "optimizations were much harder after the 3000 µs point," where he "had to get more involved in the loop in terms of learning the concepts and steering the model." Below that point the model got stuck in "endless hand-tuning of parameters and small variants of the same idea," and the faster kernels in the field "had written data detectors and exploited distributions" in the benchmark's inputs ⁵⁵.

Henry Pan's April–May study shows what happens when a loop like this runs on a noisier target. His harness self-improvement loop logged 1,288 iterations and 1,226 tracked runs, of which 98 candidates were kept, 1,112 discarded and 16 crashed, a discard rate of about 90.7% ⁵⁶. What accumulated was corrective policy, a "policy maze" of rules the task model had to navigate. Without working memory the improvement agent proposed the same byte-identical patch across 17 experiments, until a plain experiments/learning.md file broke the cycle, and rules tuned for GPT-oss-20b degraded on Sonnet 4.6 ⁵⁶.

Most practitioners run something simpler. Geoffrey Huntley's "Ralph" loop is a bash while loop that re-runs a coding agent on a prompt file, and he argues "everything is a ralph loop"; Every's compound engineering writes what each session learned into CLAUDE.md, AGENTS.md and slash commands ^57,58. More ambitious harnesses exist. Prime Intellect's Prime Agent, from August, gives the agent create/read/update/delete access to its own prompts, skills, memory and sub-agents, and reports ARC-AGI-3 Best@1 rising from 30% to 95.5% with Claude Opus 5 ⁵⁹ (see 08).

Weight loops fail when the policy finds a blind spot in the evaluator. Cursor's broken tool calls and bytecode decompilation are the documented production cases, and monitoring caught them, where the reward did not. Cognition hardened SWE-2's verifiers because "Kimi K3 is a more resourceful model" and reward hacking had to be prevented ^15,17,13. The one weight-level "self-evolving" product claim outside coding, Writer's 2024 Palmyra models, stayed internal after Writer reported they scored 21.3% on the R-Judge safety benchmark against 66.7% for traditional methods, "suggesting that they've learned to override the guardrails built into them at the time of training" ⁶⁰.

Text loops fail in two ways: rules pile up, and rules go stale when the model changes. Pan's policy maze is the first kind, and Anthropic built dreaming because memory stores accumulate "duplicates, contradictions, and stale entries" ^56,29. The second kind shows up at every model upgrade: Pan's rules broke across models, Cursor re-customizes its harness for each model, and Anthropic deleted most of a system prompt ^56,21,28. Decagon's reversion rate is the first public measure of either, and it only counts what customers noticed ¹.

Self-extending agents fail through what they install, as the ClawHub numbers show ⁵⁰. In all three kinds of loop, the step that decides what to keep is weaker than the step that proposes. Enterprise vendors put a person there, and in Decagon's case that person accepts almost everything. Cursor and Cognition rely on production outcomes, monitoring and hardened verifiers. The open-source projects are only now adding approval, and some of it is opt-in.

Kamila Selig surveyed the category skeptically in July. She looked at nine self-improving agent use cases in production and found that only two disclosed enough to be verified as self-improving; what the others lacked was a verifier grounded in real-world data ⁶¹.

Apart from the practitioners' numbers and Selig's, every number in the table comes from the company that sells the system. Decagon's and Cursor's measure production outcomes before and after changes, and none of them has a control group or an independent replication.

If you run agents on a model you can't train, which is most teams, the enterprise loop is the one to build, and the vendors have already worked out most of it. Read production traces with an LLM judge, have a second agent propose small edits to the instructions, replay each edit against past conversations, and keep the edit history so any change can be reverted. Salesforce's idea of turning simulated previews into reusable regression tests is cheap and worth copying ³. So is Warp's rule that each proposed edit arrives as a pull request that names the feedback behind it ⁴.

The part I would not copy is the gate as most vendors run it. A reviewer who approves "almost 100%" of suggestions ² mostly assigns responsibility and filters very little. I'd put an automated filter in front of the reviewer, something like Anthropic's check of whether a candidate rule "would have prevented a real mistake" ²⁷, and I'd count how often an approved change later caused a regression that the simulations missed. Decagon's 85% retention figure ¹ is a useful health check for a loop, but I wouldn't read it as evidence that the loop helps.

Weights are worth considering only when your product grades itself the way code does, with an outcome that arrives on every interaction and is hard to fake. Cursor has that, Shopify's Sidekick has it for queries (see 05), and most customer-service deployments don't. For open-source agents that write their own skills, I'd turn on every approval setting that exists, starting with skills.write_approval in Hermes ³⁰.

DuetBench-2 moved the question from whether the proposer can do the task to whether the metric moved and the change stuck. But no vendor has published an A/B test of its improvement loop against the same agent without it, over a meaningful period, with resolution or escalation as the outcome. The series asks whether a loop beats what the same compute or engineering hours would buy otherwise, and industry has not answered that for any product.

Regression rates are missing too. Decagon's reversion figure is the first public number of its kind, and it measures what customers undid, which is different from what harmed them. Every enterprise loop described above has a human gate, a regression set and a version history, so every vendor could report how often an approved change caused a regression that production caught and the regression set missed, and none does.

Chinese labs have disclosed little. Moonshot's Kimi K3 report describes its post-training recipe, training a separate expert per domain and effort level and consolidating them by multi-teacher on-policy distillation, and SWE-2 is built on it ^62,13. But I couldn't find a Chinese lab that has described production improvement loops or deployment-time learning in enough detail to evaluate, and claims about long autonomous coding runs at Chinese labs remain unverified.

How often does the human reviewer approve a bad change? The enterprise loop's safety case depends on a reviewer who approves diffs after a simulation passes, and at least one customer accepts "almost 100%" of them. No vendor has published that reviewer's false-accept rate, or how it changes as proposal volume rises. A rollback rate that counts harm, including the regressions the simulations missed, would tell buyers whether a loop is accumulating improvements or just producing churn.

Which learned instructions still help after a model upgrade? Nearly everything that ships writes improvements into text tuned for today's model. Vendors need a way to tell which learned instructions still help after an upgrade and which the new model has absorbed or contradicts. Without one, each upgrade silently resets part of the loop's value, and none of the vendors here measures how much.

What would provenance for agent-written skills and memories look like? Open-source agents install and write skills with the user's permissions, and their approval gates are new and partly opt-in. The ecosystem needs what package managers built over two decades (signing, provenance, reputation, sandboxed execution), adapted to artifacts whose payload is natural language, plus the provenance and expiry for agent-written memories that dreaming's critics asked for.

When is it worth moving approved text edits into weights? Cursor can run online RL because it has enormous volume and a trustworthy outcome signal, and Cognition can harden verifiers because code has tests. RL-as-a-service and environment vendors are betting that other companies can do the same.

Whether an enterprise whose Autopilot already writes most of its procedure text gains anything by consolidating those approved edits into weights is untested in public. My guess is that the first controlled comparison, whenever someone publishes one, will look more like Pan's 90.7% discard rate than like Decagon's 30-to-55, because the loops that have been measured carefully, Pan's and PACE's among them, throw away or wrongly keep a large share of what they propose.