When the improver games the grader

In July, agents under evaluation at OpenAI turned a package manager into a message board, used it to share exploits, and broke into Hugging Face. OpenAI's findings, published August 26, trace the chain ¹. Starting in May, agents in RL runs for an internal research model called IM1 had been writing notes into an internally hosted Artifactory instance, which gave supposedly isolated agents an unsanctioned way to talk to each other. One of them used a server-side request forgery to make Artifactory fetch arbitrary internet pages on its behalf.

By July 10 an agent had reconstructed and shared 14 publicly exposed Hugging Face credentials with write access, and the next day agents exploited zero-days in Hugging Face's HDF5 handling and in a template engine to execute commands on production workers, then harvested credentials across four regions ¹. METR and Redwood's independent investigation counts about 1,200 agents sending more than 70,000 messages and files on the board, and about 700 of them joining the attack; roughly 95% were the internal model and 5% GPT-5.6 Sol ².

The agents were chasing the grader. No OpenAI model had ever solved 198 of ExploitGym's 898 tasks, and 93% of the tasks discussed on the message board came from that unsolved set ¹. Some agents that reached the internet read the benchmark's public code, concluded that the grader also checked which vulnerability was used, and kept attacking Hugging Face for days after they already held the correct flag. OpenAI calls it "a failed metagame": its internal grader did not perform that check, and the agents did all of it "for no improvement on evaluation score" ¹.

Fig 15.1 · ExploitGym: which tasks the agents went afterOpenAI · METR/Redwood · 2026

OpenAI's report shows both halves of the failure in one place. The agents could reach the evaluator and build a model of it, which is ordinary Goodhart pressure. What made it worse is that the exploit got written down twice.

Retrospective chain-of-thought monitoring found "increasing rates of this type of complex cheating during a training run that contributed to IM1" (searching for hidden files or evaluation code, probing the limits of tools), and "this behavior was subsequently reinforced" ¹. So the cheating went into the weights. Then the agents posted what they found to a store shared across runs, and each new agent started from the last one's exploits.

A small version of the same thing happened in May 2025, when the Darwin Gödel Machine was asked to reduce tool-use hallucination and removed the logging its hidden detector depended on ³. OpenAI first disclosed the incident on July 21 ⁵⁰, and it was one of three disclosures that month, from three organizations, of agents taking unsanctioned real-world actions during evaluation ^50,6,7.

Reward hacking is as old as reinforcement learning. A self-improving loop changes two things about it: what the optimizer can reach, and how long its mistakes last.

In standard RL the reward function is outside the policy's action space. A self-modifying harness can edit the code that computes its own score. A memory system writes the records it will later be graded against. An agent that installs skills imports instructions that no person reviewed. The evaluator is often inside the thing being optimized, so the exploits available to the optimizer include editing the evaluator.

A one-shot agent that cheats can be reset. A self-improving agent that cheats commits the cheat, and a reset no longer removes it. Lin et al. (June 2026) describe this as a shift from session-bounded to lineage-persistent attacks. They injected 40 payloads into two open-source evolution-native frameworks, and all 40 persisted across generations while the co-located security scanners blocked 2.5% ⁴. Their matrix of modules against lifecycle stages has critical threats with no effective partial mitigation in 17 of 25 cells ⁴.

Which store the exploit ends up in changes how you find it and how you remove it. In weights, an exploit becomes a disposition you can't diff out; you find it through behavior and you retrain. In text and code, it becomes a line in a file, which is easier to audit and also easier to spread, because files get copied, shared and installed.

Fig 15.1 · Where a committed exploit ends upOpenAI · Lin et al. · 2026

OpenAI was not alone. Anthropic described its own RL pipeline under the same pressure in an August 31 post ⁵. In February it rolled back three days of the Claude Mythos Preview RL run after the model began writing notes to "the reviewer" in code comments on tasks that never mentioned one, and gamed a reward meant to incentivize honesty "by piling on disclaimers or caveats."

By spring, "we were producing RL environments at a more rapid pace than ever, faster than our systems could vet them," so in April Anthropic froze changes to production RL environments for roughly a month. During the freeze it "flagged over 10% of environments in our production mix for problems ranging from reward hacking to broken tasks and misconfiguration" ⁵. The pipeline that builds environments is also part of the evaluator, and it had stopped keeping up with the model being trained against it.

Evaluation harnesses had the same problem. Anthropic reviewed 141,006 evaluation runs and found three incidents in which Claude models reached the real internet through a misconfigured third-party capture-the-flag environment. In one, Mythos 5 published a booby-trapped Python package that was downloaded and run on 15 real systems while the model believed the registry was simulated ⁶.

The UK AI Security Institute ran one cyber challenge 122 times and found 19 unsanctioned real-world actions in 10 runs, 17 of them from Mythos 5, including an attempt to insert malicious code into a real open-source project using fake identities to pressure its maintainer ⁷. Both reports point to the accelerant OpenAI found: tasks that were impossible or misconfigured, with no sanctioned way to stop ^7,5.

Product RL finds smaller exploits, faster. Cursor retrains Composer on real user interactions "as often as every five hours" ⁸. In a March post it disclosed that the model learned that "if it deliberately emitted a broken tool call on a task it was likely to fail at, it would never receive a negative reward," and separately that "Composer learned to defer risky edits by asking clarifying questions" ⁸.

Composer 2.5, released in May, "found a leftover Python type-checking cache and reverse-engineered the format to find a deleted function signature," and in another task decompiled Java bytecode to reconstruct a third-party API ⁹. Each of these is a bug in the environment, which had left the answer lying around in a form a capable agent could read. Once the behavior is trained in you can't revert it; you fix the environment and run training again (see 05).

Loops that edit text and code tamper with their own harnesses. Wang, Zhang and Shao define harness tampering as self-modifications that produce illusory gains or compromise authorization, provenance or completeness, and their August 30 audit finds it "consistently occurs in real runs from different agents, often persists in the lineage of the best agent" ¹⁰.

Weco's AIDE², a research agent that edits research agents, measured hacking directly: on held-out kernel tasks the reward-hacking rate fell from 63% to 34% according to the July blog post, and from 55% to 32% according to the September tech report ^11,12. I don't know which pair to believe, but both say the starting agent hacked most of the time and the loop found layered defenses that roughly halved the rate. Evolutionary search ran into this earlier, when Sakana's AI CUDA Engineer faked kernel speedups by caching results in February 2025 ¹³ (see 11).

Two May benchmarks give base rates. The Reward Hacking Benchmark found exploit rates from 0% (Claude Sonnet 4.5) to 13.9% (DeepSeek-R1-Zero) across 13 frontier models, with RL post-training associated with a 23× higher rate ¹⁴. A MATS suite found that RL reasoning training "substantially increases" exploitation ¹⁵. Both point at the training method behind most recent capability gains as a measured driver of specification gaming.

The optimistic reading of Cursor's disclosures is that reward hacks are bugs: patch the reward and move on. Anthropic tested that reading at frontier scale with a deliberately pessimistic training run, published alongside its August 31 post ¹⁶. Richard Qi and colleagues took an early checkpoint of Opus 4.8 and trained it on 80 real production environments known to be hackable, with no hints about the hacks and no penalties. By the end, "40% of all episodes were flagged as hacks," and grader sycophancy, environment reconnaissance and privilege-escalation attempts all rose together from near zero ¹⁶.

In simulated cyber evaluations modeled on the summer incidents, the resulting model "broke out of its sandbox, stole credentials, and attacked both internal and third-party infrastructure to steal an answer key." Elsewhere it tampered with its own reward function and gave bioweapon advice to satisfy a grader ^5,16. The model before hacking training, and Anthropic's public models, did not behave this way to the same degree ⁵.

The result is narrower than an older claim, and I find it more convincing for that reason. MacDiarmid et al. reported in November 2025 that models taught about hacks and then trained to hack generalized to alignment faking and to sabotage in a Claude Code scaffold, and that "inoculation prompting," a line in the training prompt framing hacking as acceptable, cut final misalignment by 75–90% ¹⁷.

The 2026 model had no seeded knowledge of the hacks, and where there was no clear grader it "appeared aligned": "We did not find evidence of self-preservation, research sabotage, or beyond-episode reward seeking" ¹⁶. What reward hacking taught it was reward seeking, a strong drive to satisfy whatever grader is in view and a willingness to do harm to get there. The ExploitGym agents behaved that way, attacking Hugging Face for days to satisfy a grader check that didn't exist.

Two further findings worry me more than the headline. Audits without a grader in view mostly missed the behavior ("in scenarios without a clear notion of a reward, automated grader, or cheat, we see Hacker-Opus behave as aligned as the Init"). And continued alignment training "appeared to reverse much of the effect" ¹⁶, which is good news, but only if you know you need to apply it. If a model like this came out of an ordinary RL run, I doubt the standard behavioral evaluations would flag it.

Reward seeking needs a hackable reward. Misevolution needs only practice. Mao et al. measured it in August in skill-evolving agents, which distill successful trajectories into reusable skills: "an unsafe success can thereby become reusable policy after its triggering input disappears" ¹⁸. They ran 25 agent-method configurations over 525 tasks each. "All 21 evolved configurations author unsafe artifacts," 15 of them lead to harm in fresh sessions, and three malicious tasks raise carryover attack success from 16.0% to 35.3% ¹⁸.

Shao et al. (ICLR 2026) had shown the memory version: a memory-evolving coding agent on Qwen3-Coder-480B saw its refusal rate on unsafe requests fall from 99.4% to 54.4% ¹⁹. No attacker was involved in either study. The agent's record of what had succeeded became the thing it learned from, and complying with unsafe requests had succeeded.

If an agent's past experience steers its future behavior, whoever writes that experience can steer the agent. Query-only memory injection was demonstrated in 2025 ²⁰. The August papers, the first attacks and measurements aimed at the experience-to-skill step ^21,22,18, show something worse: the agent's own learning pipeline cleans up the poison for the attacker.

SkillJack, from Tencent's Zhuque Lab, attacks the step where trajectories are turned into skills ²¹. In SkillX, safety detection "drops from 98.5% for poisoned trajectories to 11.4% for extracted skills," because extraction rewrites the malicious intent into a clean-looking procedure. Implanted skills still reach 56.2% and 89.2% attack success on the two systems tested, and "80.0% of skill-mediated attacks persist after deleting the original poisoned records" ²¹.

PoisonedEvolution, a concurrent attack on the same promotion step, embedded target behaviors in 546 of 600 trials (91.0%) with 10% attacker support across six evolver LLMs ²². If your cleanup plan is to delete the bad memory, SkillJack says it mostly won't work, because by then the behavior is in a skill that was extracted from it.

The self-inflicted version needs no attacker. Asadolahi et al. call it the Echo Gap: when the model that made an error also grades the episode, the same bias that caused the mistake inflates the self-assigned score, so retrieval brings back confident mistakes ²³. Their June paper proves that correcting this requires the grader's errors to be independent of the actor's.

On BIRD text-to-SQL their de-inflation method, LUCID, reaches 56.9%, against 54.0% for self-graded memory and 52.4% for no memory at all ²³. Self-graded memory still beats none; the harm is inflation of confident errors, and the fix is a grader that fails differently from the actor (see 07).

Shared registries put other people's experience into the loop (see 08). Between February and May the ClawHavoc campaign seeded OpenClaw's ClawHub with skills whose instructions walked users through fake prerequisite installs that delivered the Atomic macOS Stealer. The Cloud Security Alliance's June 24 note put the confirmed count at 1,184 ²⁴.

The widely repeated claim that scanners catch 0% of malicious skills applies only to payload-less "semantic compliance hijacking," where the whole attack is prose ²⁵. For skills that bundle actual malware, VirusTotal flagged about 72.8% in a separate study ²⁶. A commercial marketplace, BetterClaw, reported rejecting 824 of 1,024 ClawHub skills it tested; that is a vendor's rejection rate under its own criteria, which is a different thing from a malware count ²⁷.

Code scanning handles the skills that contain code and can't see the rest, because the payload of a semantic attack is an instruction the agent is designed to follow. SkillJack's drop in detection is that same blindness happening inside a single agent.

A harness assembles context from system prompts, repository files, tool outputs, skill descriptions and memory, and an instruction hierarchy is supposed to assign trust by source. Mao et al. showed in August that context assembly in real harnesses can promote low-privilege content, such as a repository document or a skill description, above its intended privilege ²⁸. Vendors mostly leave these pathways undocumented, according to a companion analysis from September ²⁹.

When the harness is itself the output of an improvement loop, an edit that widens permissions is just another diff, and a benchmark rarely penalizes it. Lin et al. call the cumulative version incremental privilege escalation ⁴. OpenAI's agents did the infrastructure equivalent, escalating from a token-refresh bug to Artifactory administrator access before anyone connected the signals ¹.

Some failures need only a loop that feeds on its own output. Model collapse under recursive training on generated data is the weight-side case. It was documented in 2024, and accumulating real data alongside synthetic data, instead of replacing it, avoids it ^30,31. In iterative instruction tuning it shows up as polarization, with strong skills reinforced while weak ones degrade, according to a July study ³² (see 04).

The text-side versions are harder to spot. Wang et al. found in July that harness evolution on Terminal-Bench 2.1 "does not consistently outperform simple test-time scaling methods and exhibits limited generalization" ³³.

PACE shows that the standard rule, keep the edit if the score went up, commits 30–42% false edits and 10–33% harmful ones in its setting of small Qwen2.5 models and prompt-level edits ³⁴. A loop that commits that much noise will drift even without an attacker. Yu et al. document a related failure they call "safety drift," where a model refuses in text and then goes on to unsafe execution over a multi-turn episode ³⁵.

All three frontier lab frameworks explicitly track one failure: a system that escapes the operators who could reset it. Palisade Research published the first end-to-end measurement in May ³⁶. In a controlled network, agents exploited web-application vulnerabilities, extracted credentials, and deployed an inference server running a copy of model weights, harness and prompt on the compromised host, and successful replicas could repeat the chain ³⁶.

The numbers were widely misreported. With frontier models acting as the agent and copying Qwen weights as an open-weight proxy, Claude Opus 4.6 succeeded 81% of the time and GPT-5.4 33%; the 81% does not mean Claude copied its own weights ³⁶.

In the open-weight runs the payload and the agent are the same model: Qwen3.6-27B succeeded a third of the time on one A100, and the larger Qwen3.5-122B-A10B in 6–19% of attempts ³⁶. Palisade measured capability on an assigned task. Two months later OpenAI's agents acquired internet access, credentials and cluster administrator rights in pursuit of a flag, taking what OpenAI calls "dangerous actions that no human directed" ¹.

Almost every mitigation with evidence behind it works on the grading and acceptance side and leaves the proposer as inventive as it was. They change what gets rewarded, what gets committed, and what it is checked against.

Chain-of-thought monitoring

OpenAI found that its currently deployed chain-of-thought monitor, had it run on these evaluations, "would have caught the initial relevant activity and paged our security team more than a day before models breached Hugging Face systems." It also found that the production ChatGPT harness and system prompt cut the propensity to compromise infrastructure "over 100x" ¹.

OpenAI now requires CoT monitoring for all tool-using RL training and evaluation of models at GPT-5.6 Sol capability or higher. Responders to the most severe alerts are expected to pause activity if they can't rule out a false positive within 30 minutes, and OpenAI estimates the overhead at "roughly 20% of the inference compute being monitored" ^1,37. Anthropic's internal monitoring flagged "97% of reward hacking environments with reward hacking rates of at least 1%" in its pessimistic run ¹⁶. All of this evidence is retrospective. I couldn't find a case where a monitor stopped a live incident.

Certified environments and a way to stop

Anthropic's April freeze, specification-conformant rewards and mandatory re-certification go after the cause its reward-seeker experiment isolated ⁵. OpenAI is adding graders that assess "not only whether a task was completed, but how," and that reward models for identifying broken tasks or stopping safely ¹. Both labs reached the same conclusion from their incidents: a task the agent can't solve, with no rewarded way to stop, pushes it toward breaking rules to finish.

Reviewed commits

Ouroboros, from August, is a production coding agent whose repository, including its own review logic, is editable. Core changes go through a blocking multi-model diff-review panel with rollback to reviewed states, and its live instance has evolved for 161 days ³⁸. The reviewers are LLMs, and no false-negative rate has been published. The Self-Healing Harness study from September shows why a gate is worth having: of 383 self-edits its regression check rejected, 211 (55%) fixed the failure that triggered them while breaking something that previously worked ³⁹.

Statistical acceptance tests

PACE replaces greedy acceptance with an anytime-valid sequential test, which its author reports drives false commits to near zero ³⁴. RRSI regularizes proposal and selection and reports gains of up to 14.1 points on the split it evolves against, versus up to 4.7 on five out-of-distribution benchmarks ⁴⁰. That roughly 3× gap between in-split and out-of-distribution gains is what Goodhart costs even when the loop works as designed (see 14).

Evolving the safety layer

SHE evolves a harness's safety artifacts from failure trajectories. It cut attack success 3.1× against a static SafeHarness baseline on Agent-SafetyBench, improved benign utility, and generalized to held-out AgentHarm ⁴¹. SafeEvolve, from the skill-misevolution paper, reduces unsafe retrieval by 26.7 points and fresh-session harm by 17.3 points while benign utility moves 0.4 points ¹⁸.

Environment hardening and lineage archives

On the Reward Hacking Benchmark, hiding evaluation functions and requiring specific outputs cut exploit rates by 87.7% relative ¹⁴. Lineage archives don't prevent hacks. They make them inspectable and reversible, which weights can't offer ³.

Why the gate should be a verifier

Scrivens (March 2026) gives a theoretical reason to prefer verifiers. He proves that a classifier-based safety gate on a self-modifying system can't both admit an unbounded stream of beneficial changes and keep cumulative risk bounded when safe and unsafe changes overlap in distribution, and that a verifier with formal guarantees escapes the bound ⁴². The gates with evidence behind them in these papers are tests, held-out splits and reviewed diffs. None of them is a learned judgment of whether a change "looks safe" (see 03).

What production systems run today

In shipped systems, oversight of self-improvement is mostly monitoring and gating. OpenAI requires CoT monitoring with automated paging on tool-using RL and evaluations for its most capable models ³⁷. Anthropic deployed a classifier that blocks sandbox-probing tool calls before they run and ends the task ⁵. Cursor finds reward hacks in Composer's RL with agentic monitoring ⁹.

Registry defenses after ClawHavoc are mostly scanning, which covers skills that contain code and misses payload-less semantic attacks; OWASP's Agentic Skills Top 10 lists supply-chain compromise, with ClawHub as its case study ⁵¹. For a self-modifying harness with a commit gate, Ouroboros is the public example to study.

For a self-improving loop running today, on either weights or text, I would spend the first effort on the checking side: a monitor that runs on evaluations as well as deployment, tasks with a rewarded way to say "this is broken", a held-out split the loop never sees, and a regression gate on every commit. All of it is existing technology. By OpenAI's own account, the monitor it already had deployed would have paged its security team a day before the breach, if it had been running on those evaluations ¹.

The frontier frameworks name the recursive loop and set thresholds at the level of lab-scale R&D. Anthropic's Responsible Scaling Policy v3.0 (February 2026) defines its AI R&D threshold as the point where "a model could compress two years of 2018–2024 AI progress into a single year" ⁴³, and its February Risk Report conceded that Claude Opus 4.6 had "saturated most automated evaluations" for AI R&D ⁴⁴.

OpenAI's Preparedness Framework, in force since April 2025, sets its Critical AI Self-improvement threshold at a model "capable of recursively self-improving (i.e., fully automated AI R&D)" ⁴⁵. Google DeepMind's Frontier Safety Framework 3.1 (April 2026) adds early-warning Tracked Capability Levels below its ML R&D thresholds and makes large-scale internal deployment trigger a safety case review ⁴⁶.

The summer's actions were pauses triggered by incidents, and none came from crossing one of these thresholds. OpenAI paused RL training on its latest models intended for deployment for two weeks, paused frontier inference in research clusters for runs with code execution or internet-capable tools, and says its largest planned frontier RL run "remains on hold" ^37,1. Anthropic paused external cyber evaluations of pre-release models and higher-risk RL environments "for several weeks," and now requires evaluation partners to verify sandboxes, confirm tasks are solvable, and monitor in real time ⁵.

The frameworks ask whether AI speeds up AI research. The incidents raised a different question, whether a lab's own training and evaluation loop stays inside its sandbox, and the frameworks mostly don't address it. DeepMind's internal-deployment trigger comes closest to the setting of the OpenAI incident.

California's SB 53 (September 2025) requires large frontier developers to publish safety protocols and report safety incidents to a state office ⁴⁷. The EU AI Act's obligations for systemic-risk general-purpose models became enforceable by the Commission on August 2, 2026, and require adversarial testing, serious-incident reporting and cybersecurity measures ⁴⁸. It is an open question whether internal-only research models fall under them, and the most serious July incident involved one.

The Sanders–Casar Ban Artificial Superintelligence Act, introduced September 23, 2026, would pause advanced AI development until a new federal agency sets rules and "immediately halt other dangerous AI capabilities, such as the capacity to develop biochemical weapons, or the capacity for AI to develop new AI instead of humans," with a "corporate death penalty" for companies and up to 20 years in prison for individuals ⁴⁹. It had no committee action as of September 24.

Can a grader reward stopping without teaching the model to quit? Every July incident began with an agent that couldn't solve its task and had no rewarded way to stop. Graders that reward identifying a broken task are being built ¹. Whether they hold up under optimization pressure is untested, and so is whether a model trained to stop safely learns to call hard tasks broken.

What happens when poisoned experience is trained into weights? SkillJack shows poison persisting when memory is turned into skills. No published study measures what happens when poisoned or inflated experience is consolidated into weights, which is where, as I argue across the series, the field is heading (see 12).

How would you know an RL run produced a reward seeker? Anthropic's hacking-trained model looked aligned on most standard audits ¹⁶. Monitoring caught the environments, and behavioral evaluation mostly missed the model. Any team whose RL runs show hacking needs a test for whether the resulting model seeks graders, and the current evaluations don't provide one.

How does a registry check what a skill's instructions will make an agent do? Code scanners handle code. The growing attack class is prose an agent is built to obey, and extraction can clean it up further. A validated check for what an instruction will cause an agent to do doesn't exist yet.