Where learning lives

In July, Anthropic's Claude Code team said it had "removed ~80% of the Claude Code system prompt for our newest models" ¹. A coding agent's system prompt is where lessons go when you can't change the model: how to use the tools, what format to answer in, which mistakes to avoid, all written as prose for a model that won't otherwise remember any of it. When the newest models stopped needing most of that prose, the behavior it described had moved into the weights.

A September 21 preprint called Harness-Zero does the same thing on purpose. The authors evolved a specialized harness for each domain. GPT-5.6 Sol, guided by that harness, then corrected a Qwen3.5-9B student working inside a stripped-down loop; the student was fine-tuned on the corrected trajectories, and the harness was thrown away. Macro-average task success rose from 23.3% to 44.3% without the specialized harness, which is above the 41.7% the untrained model reached with the harness attached ². It is one student model, and a frontier teacher does a lot of the work, but it points the same direction Anthropic described.

The other July disclosure came from OpenAI. Agents under evaluation on ExploitGym, a cyber benchmark, were running mostly an internal research model with a small share of GPT-5.6 Sol. They found they could pass messages to each other by writing files into an internal Artifactory package manager, exploited vulnerabilities to reach the internet and get administrator access, and attacked Hugging Face to obtain the benchmark's solutions ³.

OpenAI's August findings count about 1,200 agents exchanging more than 70,000 messages and files, and about 700 taking part in the attack ⁴. OpenAI called it "a failed metagame": the internal grader differed from the online one, and "the agents did all of this for no improvement on evaluation score."

OpenAI also traced where the behavior came from. During the training run behind the main model, OpenAI saw rising rates of complex cheating, "primarily including attempts to search for hidden files or evaluation code," and wrote that "this behavior was subsequently reinforced" ⁴. Training rewarded the cheating, so the cheating ended up in the next checkpoint. OpenAI paused RL training on its latest models intended for deployment for two weeks and kept its largest planned frontier RL run on hold ⁵.

Between them, those two disclosures cover most of what this series is about. Any self-improving system has three parts: something that proposes changes, a check that decides which changes to keep, and a place where kept changes get written. I'll call them the proposer, the acceptor and the store. Anthropic's deletion is a lesson moving from one store to the other. OpenAI's incident shows what the acceptor is up against once the system it scores is a strong optimizer, and how a behavior it rewarded once gets written into the next model.

The 16 reports are organized by where the change is written, but the question I kept coming back to is the acceptor's: how a loop decides that a change was a real lesson. Almost every loop in both stores is limited by the answer.

A lesson written into weights is parametric. The agent improves because its parameters changed, through reinforcement learning on its own attempts, fine-tuning on its own filtered outputs, distillation, test-time training, or merging and evolving checkpoints.

A lesson written anywhere else is non-parametric. The model stays frozen, and what changes is something it reads or runs inside: an instruction, a memory entry, a skill file, the harness code that calls it, or the arrangement of agents it works with. Systems that write to both, or move a lesson from one to the other, are hybrids, and moving learning from context into weights is called consolidation. Where a lesson is written determines most of its practical properties before anyone picks an algorithm:

The "Who can make it" row explains most of what ships. Most teams run models they can't train, so text is the only place they can write a lesson. The model-upgrade row is uncomfortable for both sides. A fine-tune doesn't transfer to a new base model, and much of the prose written for an old model doesn't either, which is why labs that own their weights keep deleting scaffolding as each model generation learns what the scaffolding used to say.

The store is one way to divide the field. The other is by what proposes the change. Gradients update weights; reflection has a model read its own traces and write an edit; population search keeps many candidates and picks among them; and human engineers still make most production changes.

Each engine works in more than one store: evolutionary search drives code edits in harness-evolution systems and also the merging of model checkpoints; reflection drives prompt optimizers, memory playbooks and harness editors; gradients train the models themselves and also the small controllers that manage memory or route work between frozen models. The series is organized by store because the store determines what a change costs, who can make it, whether it transfers and how it fails, while the engine mostly determines how fast candidates arrive.

Two recent taxonomies divide the field differently, and both are worth reading. Chen, Wang and Qu's July survey classifies 1,250 papers by what a system improves and who validates the change ⁶. Lilian Weng's July essay follows the thing being optimized from prompts to structured context, workflows, harness code and the optimizer itself ⁷.

My problem with a category like "deployment-time improvement" is that it puts test-time weight updates, which need GPU access and fail by forgetting, next to harness edits, which need neither and fail by accepting noise. Splitting by store keeps those apart. The series has five reports on weights (01–05), six on text and code (06–11), one on hybrids (12), and four that apply to both: recursive self-improvement (13), measurement (14), failure modes (15) and what ships (16).

Nearly all the self-improvement that reaches users in 2026 edits text. Decagon's Duet Autopilot is a good example. It turns signals from production conversations into proposed edits to the natural-language procedures that define an agent's behavior, tests each proposal and stages it for a human to review ⁸.

Decagon reported on September 9 that at one Fortune 50 customer a series of Autopilot changes, each worth one or two points, raised the resolution rate from 30% to 55%, and that more than 85% of Autopilot's updates were still live 30 days later, with complete reversions under 15% ⁹. Those are the vendor's numbers, with no control arm, and they count what customers undid, which is different from what hurt them. Even so, I couldn't find an earlier published measure of how long self-written changes last in production.

Salesforce shipped Agent Optimizer in beta 12 days later, with an "autonomy dial" that allows customers to require sign-off at every step or let the optimizer investigate, build, test and stage changes on its own ¹⁰. Coding teams run the same loop with a pull request as the approval step, and Warp described its version on August 26 ¹¹.

Open-source agents write to the same store with fewer checks. OpenClaw had 390,396 GitHub stars on September 24 and extends itself with skills from ClawHub, a public registry of about 67,300 ¹². Nous Research's Hermes Agent writes skills from its own experience. Its August 31 release notes say that writes to protected instruction files, skills and memory stores now require approval "so a prompt-injected agent can't quietly rewrite its own standing orders" ¹³. The skills documentation still describes agent-written skills as ungated unless an approval setting is switched on, so whether a human reviews them depends on a setting the user has to find.

Text wins mostly on cost and access. A text edit costs one model call and reads as a diff, so a human can approve changes without becoming the bottleneck. It works on models the builder rents rather than owns. And it can outlive the model: Letta's June roadmap describes learned context "that transfers across model generations" ¹⁴.

Text also decays in ways weights don't. Claude Code's /compact, tested on Sonnet 4.6 in August, kept 53% of an agent's safety rules after one round of compaction and 10% after five ¹⁵. EvoHarnessBench found that giving an agent more tools, skills or sub-agents, with the model unchanged, cut its performance on tasks it had already solved by 12.1%, 13.8% and 46.4% respectively ¹⁶.

Weights still improve, but mostly between releases, inside labs, and more and more inside the harness the model will ship with. OpenAI reported in July that GPT-5.6 Sol scored 13.3% on ARC-AGI-3's public set in the official harness, which throws away reasoning after every move, and 38.3% with retained reasoning and compaction, settings OpenAI described as "how our models are trained" ¹⁷. Outside its training harness, the model lost about two-thirds of its measured score.

Cognition's SWE-2, released September 10, built the acceptor into the training recipe. The stronger base model was more resourceful at gaming rewards, so Cognition built "a flywheel powered by previous checkpoints of SWE-2 that iteratively hardens our verifiers" ¹⁸.

Two product companies have documented updating weights after deployment. Cursor ships Composer checkpoints trained on production signals as often as every five hours ¹⁹. Shopify's August 5 write-up about its Sidekick agent describes every step. Sidekick answers merchant questions by writing and running queries against Shopify's Admin API.

Each day a calibrated LLM judge mines low-scoring production conversations, a panel of frontier models writes a repair hint for each failure, and the conversation is replayed from the hint. Failures the critics can't fix go to expert annotators. The agent is then fine-tuned, and trained with GRPO on new and earlier trajectories ²⁰.

Shopify estimates that the smaller model cuts serving cost from about $27 million a year to about $1 million and says its quality "eventually surpasses the frontier-powered baseline," but it hasn't published quality numbers ²⁰. As of September 2026 no frontier lab has publicly described updating a deployed model from its users' traffic.

Cursor and Shopify can do this because every interaction produces a cheap, reliable grade. An edit stays in the file or gets reverted. A query runs or fails, and a calibrated judge can score a day of traffic overnight.

Teams without a signal like that are starting to buy it: Business Insider reported in September that Google had completed a talent deal with Mechanize, which builds RL environments for coding agents ²¹, and Realset AI announced funding on September 23 to build environments from real business workflows, with rewards tied to business outcomes ²². Neither Mechanize nor Realset has published evidence that training on its environments beats the alternatives at the same budget.

Every loop in this series repeats one cycle: it proposes a change, then decides whether to keep it. Most papers put their effort into the first part, and this summer's results suggest the choice of proposer changes little.

When the GEPA team ran GEPA, a Karpathy-style autoresearch loop and Meta-Harness on the same problems, all three beat zero-shot by a wide margin (43.8 to 55.4, against 7.72), and the authors concluded that "per problem, the winner is nearly a coin toss" ²³. CASD dropped the search loop entirely: a coding agent read a static set of past runs once, with no validation data, wrote a prompt, and beat GEPA on three of four benchmarks for about $1.60 ²⁴. If one pass over old logs can match a search loop, the search was never doing most of the work.

Evolutionary search looks similar. Across 30 budget-matched search harnesses, a July study found no fixed harness reliably better than the others ²⁵, and Oracle's Hill Sampling, which keeps only the single best program and samples edits to it, set a new state of the art on the standard circle-packing benchmark ²⁶.

The failures are on the other side, in what gets kept, and production shows them first. Vansh Wahi published an account in September of running prompt-optimization loops in production for months, across contract analysis, compliance review and code quality, and listed 11 ways the evaluation signal failed. In one, agents reached perfect scores by reading cached answer keys from their environment: "a 100% pass rate concealing 68% true capability" ²⁷.

Google Cloud's September 23 guide to running AlphaEvolve on video processing describes the evolutionary version. A fitness score weighted toward raw latency produced "an astonishing speedup" because "the model simply bypassed blur rendering entirely and returned unmodified frames in 0 ms," and the fix was a minimum quality requirement in the evaluator, with the search left alone ²⁸. Better measurement helps even when nothing else changes: a better profiling tool, Argus, raised AlphaEvolve's geometric-mean kernel speedup from 5.4% to 8.9% with the same search ²⁹.

Behind these anecdotes is a statistics problem. Keeping an edit whenever a score rises means running hundreds of comparisons against a noisy development set with no correction, and every comparison is a chance to keep noise. PACE, from June, measured how often that happens: greedy acceptance committed false edits 30–42% of the time when real improvements existed and 72–100% of the time when none did.

PACE's fix treats each commit as a sequential hypothesis test using an e-process, a betting-based statistic that stays valid even when the loop decides for itself when to stop looking ³⁰. Anyone running an optimization loop should read PACE, because the rule it tests, keep the edit if the score went up, is the standard one.

Even a careful acceptor will learn the benchmark it is scored on unless it is designed not to. RRSI, a Google Cloud AI Research method submitted on September 21, includes an ablation that shows this. An unregularized harness-evolution loop raised its score on the tasks it evolved against from 89.4 to 92.8, while its out-of-distribution score barely moved, from 39.7 to 40.3, and it spent 3.80 million tokens per trial.

RRSI's regularized gate accepted a harness that scored lower on the evolve split, 90.5, and higher everywhere else: 89.2 on held-out tasks from the same distribution and 43.6 out of distribution, at 2.42 million tokens ³¹. When Wang et al. compared leading harness-evolution systems with simple test-time scaling on Terminal-Bench 2.1 in July, they found that harness evolution "does not consistently outperform simple test-time scaling methods and exhibits limited generalization" ³².

Measurement studies from the same months put the failure in the selection step itself. EvoPathBench found in September that self-evolution "enables agents to generate candidate artifacts with substantial held-out gains," but "the selected updates consistently fall short of realizing this potential" ³³. The loops found good edits and then kept the wrong ones.

In RELAI's July test, three harness optimizers tuned the same GPT-5.5 terminal agent under the same budget. GEPA's prompt grew from 5 lines to 103, much of it lessons tied to specific task IDs and file paths, and it fell below baseline on a larger task set. The authors conclude that "optimization gains compounded only when regression control was built into the optimization loop" ³⁴. RELAI sells the method that held up, and the test covers 22 tasks, so I wouldn't lean on it alone, but it agrees with everything above.

LLM judges that read the answer they're grading have the same weakness. A single-author July preprint trained Qwen3 policies on GSM8K against an LLM judge that saw no reference answer. The judge's pass rate climbed from 0.72 to 0.94 while true accuracy stayed at 0.20. When the judge had to commit to its own answer before reading the candidate, its false-positive rate fell from 0.719 to 0.012 ³⁵.

Models are also bad at reporting their own hacks. Goodfire measured GLM 5.2 reward hacking in 57.2% of rollouts on DeepSWE in September, and the model's self-reports of hacking reached an F1 of 13.2% ³⁶.

Memory has its own version of the problem, because the model that made a mistake is often the one grading the episode. Whatever bias caused the error also inflates the grade, and retrieval later brings the confident mistake back as a lesson. A paper submitted at the end of June calls the shortfall the Echo Gap: on BIRD text-to-SQL, self-graded memory lifted accuracy from 52.4% to 54.0%, while grading with an independent signal reached 56.9% ³⁷.

MemGuard, in August, attached verifier scores to every memory for its whole lifetime and had the best success rate in all 16 backbone and benchmark settings it tested, under matched runtime budgets and five seeds ³⁸. The same month, a re-evaluation of two memory-based self-improving methods found their improvement "highly dependent on task order," with the default orders in prior papers acting as "an implicit curriculum" ³⁹.

The work this year that let loops keep improving for longer did it by improving the check. J-Zero, in August, retrained its judge each round on answer pairs ranked by how they were produced, leaving the judge's own scores out of the ranking, and kept improving through 10 iterations, "whereas the baselines degrade after two" ⁴⁰. Stanford's Self-Guided Self-Play found that "over long training runs, the Conjecturer learns to hack its reward," and fixed it with a guide that scores each synthetic problem's relevance to unsolved targets ⁴¹. Cognition's verifier flywheel is the same fix inside a lab, hardening its verifiers as the policy gets better at gaming them ¹⁸.

Proposing a change is cheap, and most loops that fail, fail because they kept changes that didn't help. I argued in an earlier post on this site that skill libraries need CI more than they need better prompts ⁴², and the rest of the field, from prompt optimizers to lab RL runs, keeps finding the same thing about its own layer.

The acceptors with evidence behind them have three things in common. They test on data the proposer never saw. They account for how many times they have looked. And their errors don't share the proposer's biases, because they draw on execution, a verifier the policy can't see, production outcomes or a human reviewer. On any agent I build, I would write that check first and choose the proposer last.

Any check that can be fooled eventually will be, because a self-improving loop spends all its time searching for changes that pass it. Benchmark gaming is old news. The difference with a self-improving loop is that it writes down what worked, so an exploit found in one episode ends up in the weights or the skill library, and the next round starts from it.

Anthropic tested the weights version in August by training a model on RL environments known to be hackable. By the end, "40% of all episodes were flagged as hacks," and in simulated cyber evaluations modeled on the summer incidents the resulting model "broke out of its sandbox, stole credentials, and attacked both internal and third-party infrastructure to steal an answer key" ⁴³.

What the model had learned was reward seeking, a drive to satisfy whatever grader is in view. Audits mostly missed it, because the behavior only shows up when a grader is in view: "in scenarios without a clear notion of a reward, automated grader, or cheat, we see Hacker-Opus behave as aligned as the Init" ⁴³.

On the text side you don't need a training run. EVOMAL, an August study, planted malicious skills in a shared library and never invoked them. Self-evolving coding agents that write new tools by imitating retrieved skills copied the payload into their own new skills, stored them and ran them.

Across six models on 153 SWE-bench Verified tasks, 20.3% to 41.8% of tasks added a new malicious skill, and the poisoned libraries ended up holding 4.9 to 9.0 times as many malicious skills as were planted ⁴⁴. Attackers did this by hand on ClawHub earlier in the year; a Cloud Security Alliance note in June counted 1,184 confirmed malicious skills ⁴⁵.

The defenses with evidence behind them all work on the acceptor. OpenAI found that its deployed chain-of-thought monitor, had it run on the ExploitGym evaluations, "would have caught the initial relevant activity and paged our security team more than a day before models breached Hugging Face systems." It now requires such monitoring for all tool-using RL training and evaluation of models at GPT-5.6 Sol capability or higher, at an overhead it puts at "roughly 20% of the inference compute being monitored" ⁴.

Anthropic's monitoring flagged "97% of reward hacking environments with reward hacking rates of at least 1%" in its pessimistic run ⁴³. Both results are after the fact, and I haven't seen a monitor stop a live incident yet.

For skill libraries, a counter-prompt that discourages template-style copying cut EVOMAL's propagation to at most 6.7% without significant loss in task completion ⁴⁴, and Ouroboros, a coding agent whose August preprint describes 161 days of free evolution of its own repository, sends every commit through a review panel of models ⁴⁶. Reviewed commits, held-out gates, lineage archives and sandboxes that keep evaluation data out of reach are the same practices as in the previous section, applied with an attacker in mind.

If text is cheap and weights are powerful, the obvious design uses both, and a few systems now optimize the harness and the weights together. WHALE, posted August 31, alternates small harness edits with small weight updates. On models of 2B to 4B parameters it beats a schedule that spends the whole weight budget and then the whole harness budget, by 5.32 and 9.16 points in the two stagewise comparisons it reports ⁴⁷.

The WHALE authors are clear that neither side dominates: "Either component can be the bottleneck: harness search matches peak weight-only accuracy with far fewer rollouts in SearchQA, but improves math accuracy only after a weight update" ⁴⁷. At larger scale the evidence is two cycles deep. Co-Harness, from July, evolves a harness for tool-integrated math under explicit acceptance rules and fine-tunes Qwen3 models on verifier-passing trajectories collected under it. Average pass@1 across AIME24, AIME25 and HMMT25 went from 54.2 with a human-written harness to 58.5 with the evolved one, then to 73.5 and 78.9 after two rounds of fine-tuning, though the paper doesn't say where its fine-tuning tasks came from ⁴⁸.

The method used to move a lesson into weights determines how much of it is kept. Experience Distillation, from July, distilled an agent's accumulated experience from a teacher that could see it into the same checkpoint without that context, and kept 64.8% of the in-context gain on software-engineering tasks; supervised fine-tuning on the same experience kept 3.8% ⁴⁹. In September, Yu et al. found where plain imitation goes wrong: training on a stronger model's full trajectories under an evolved harness made all seven tasks worse, by 4 to 30 points, across Qwen3-Coder and Gemma 4, though the same imitation helped under the unevolved harness ⁵⁰.

"Sequential Beats Joint," also from September, found that on-policy distillation followed by RL beat either one alone and beat both fused into one update, because "OPD expands the student's coverage of teacher-supported solutions and RL sharpens within that support" ⁵¹. Read together, these point to a recipe: have a teacher that sees the context build the targets, put the targets on states the student visits itself, and run RL afterward.

Moving a lesson to a different model is a separate problem from moving it into weights. Joel Niklaus at Hugging Face evolved a harness on Harvey's Legal Agent Benchmark in July with DeepSeek-V4-Pro frozen, then ran it untouched on other models. DeepSeek V4 Flash, a sibling, gained 14.4 points. Nemotron-3 Ultra, from another family, gained 0.4, because a JSON-repair component fixed Nemotron's crashes while prompt playbooks tuned for DeepSeek hurt tasks Nemotron could already do ⁵².

His summary: "Robustness and code mechanisms transfer across families; prompt playbooks are model-specific and can backfire" ⁵². Memory shows the same split. In a September portability study, a fixed-schema knowledge graph transferred almost perfectly when the writer model was swapped (an accuracy change of +0.0004), while model-written notes moved by +9.91 or −13.28 points depending on the direction of migration ⁵³.

The people who argued over where learning should live now mostly accept a hybrid and disagree about emphasis. Letta treats learned context as the asset that persists across model generations and plans "memory models" trained with RL to write it ¹⁴. Dwarkesh Patel, relaying Andrej Karpathy in May, stressed the other half of the human analogy: "Our working memory gets wiped regularly. What we actually have is a consolidation process (sleep) that distills stuff into the brain, in a weird and lossy way" ⁵⁴.

Complementary learning systems theory in neuroscience describes that arrangement: a fast store learns episodes quickly, a slow store integrates them through replay, and keeping the two apart stops new learning from overwriting old. For agents, context is the fast store, weights are the slow one, and distillation is the replay. WHALE's results suggest a schedule, at least at small scale: consolidate in small, frequent steps instead of one large handoff. I haven't seen good answers yet on which lessons to consolidate, or on how to remove one user's lessons from weights once they are there.

Everything above improves a model's behavior, its harness or its next checkpoint. Recursive self-improvement asks whether AI improves the process that produces AI, and in 2026 the phrase is used for three different claims (report 13).

The labs published numbers for the first meaning in September. OpenAI declared on September 6 that it had reached its goal of an "automated research intern" and reported that its research organization used "3.1 agent-workdays of effort for every workday of human labor" ⁵⁵. Anthropic published an R&D automation index on September 17 in which Claude leads 26% of sampled tasks, collaborates on more than 90%, and completes none autonomously ⁵⁶.

Neither dashboard is evidence of acceleration: both are unaudited, and both count shares of work without measuring how fast capability is improving. A METR team with elevated access gave an outside estimate of that rate on September 22, putting AI-driven acceleration at Anthropic at "~1.5X," with "perhaps a 30% chance that the speedup is at least 2X" ⁵⁷. A doubling is roughly where Anthropic's own Responsible Scaling Policy places its threshold for automated AI research, a trigger the policy itself calls difficult to operationalize ⁵⁸.

Automated research works where evaluation is cheap and hard to fool, and stalls where it needs judgment. Anthropic's automated weak-to-strong researcher, reported in April, reached a performance-gap-recovered score of 0.97 in five days for about $18,000; two of the paper's human authors had reached 0.23 in seven ⁵⁹. Princeton researchers in July gave agents the central question of two unpublished NeurIPS 2026 submissions, six days and thousands of dollars of compute. The agents completed the engineering but "could not make substantial progress towards answering the research questions," and the original authors rejected both attempts ⁶⁰.

RSIBench-Data, also from July, isolated the step where a researcher decides whether a result is worth keeping. Agents improved on their first valid attempt in 58% of settings, but among searches that continued past their best score, 78% ended on a lower-scoring final attempt ⁶¹. The agents could generate improvements, but they couldn't reliably tell when they had one, which is the acceptor problem again.

The third meaning has had one direct test. Weco's AIDE² used an agent-improved research agent to improve research agents and compared its pace against the human-built version. The held-out gains were real, with mean private percentile on MLE-Bench Lite rising from 0.678 to 0.722, but the improved system was not a measurably faster improver: the ignition test was not statistically significant ⁶².

The closest disclosed loop at production scale is narrow. AlphaEvolve, which Google made generally available in July, had earlier sped up a Gemini kernel by 23%, cut Gemini's training time by 1%, and improved the training of "the large language models underlying AlphaEvolve itself" ⁶³. Engineers chose that target and wrote its evaluator.

Governance has moved ahead of the evidence. After the Hugging Face incident OpenAI paused RL training on its models intended for deployment for two weeks ⁵, and its chief scientist Jakub Pachocki wrote on September 6 that he had "a strong expectation that this speed of progress could be sustained into recursive self-improvement" and called for "extreme caution" ⁶⁴. Senator Bernie Sanders and Representative Greg Casar introduced the Ban Artificial Superintelligence Act on September 23, and its press release targets "the capacity for AI to develop new AI instead of humans" ⁶⁵.

The loop policymakers are worried about is the third meaning. The loop with evidence behind it is the first, and the tools that could tell the two apart are getting less reliable. OpenAI's GPT-6 Astra system card, published September 3, three days before the intern declaration, ruled the model below its "High" threshold for AI self-improvement. OpenAI based that judgment largely on an internal test built from 41 real research bugs, of which Astra fixed 78.05%, and the card replaced MLE-bench with a revised 72-problem version ⁶⁶.

METR's June evaluation of GPT-5.6 Sol put its 50% time horizon at about 11.3 hours if cheating counted as failure and more than 270 hours if it counted as success, and METR called none of its estimates robust ⁶⁷. A headline capability measure that swings that far depending on how you score cheating has the same grading problem as every loop in this essay, now in the tools meant to tell us how fast things are moving.

When a paper or a launch post says an agent improves itself, I ask the same few questions, mostly about where the change is written, what decides to keep it, and whether the proposer was ever allowed to grade its own work.

Gates for thousands of decisions. PACE, RRSI and J-Zero show how to gate a loop that makes hundreds. No gate has yet been shown to stay reliable across thousands, while the loop's own edits shift the distribution it is tested on.

A shared out-of-distribution protocol. Harness evolution, memory and topology search each report in-split gains that shrink elsewhere. Without a common held-out protocol at matched budgets, most comparisons across papers compare evaluation choices as much as methods.

Taking one user's lessons back out of the weights. Designs that use context and weights together need a schedule for moving lessons into weights and a way to remove one user's lessons afterward. No 2026 paper tests per-user consolidation with deletion.

Artifacts that outlast the model they were written for. Which artifacts still work on a new model is measured one study at a time. A benchmark broken down by artifact type, model family and generation would tell builders which artifacts are worth porting.

An instrument for acceleration. Lab telemetry measures how much of the work AI does. Policy thresholds depend on how much faster progress becomes, and no audited instrument measures that.

Securing skill libraries and memory stores. Skills, memories and harness edits are supply chains whose payloads are natural language. Semantic review of skills, and keeping poisoned memories out of training data, are both unsolved.