Self-generated tasks

On August 27, 2026, Gyouk Chu, Myeongho Jeon, and Eunho Yang posted J-Zero, a self-play system with three roles drawn from one base model ¹. A Challenger writes increasingly hard tasks, a Solver answers them, and a Judge scores the answers. The Judge is the new part. Earlier zero-data loops either needed an executor to grade answers or froze a judge at the start, and a frozen judge falls behind the Solver it is grading. J-Zero retrains the Judge every iteration on preference pairs whose ordering is known from how each answer was produced: the Solver's answer beats the Challenger's, and the Solver's decomposed-and-recombined answer beats its one-shot answer. The Judge never learns from its own scores. J-Zero beats its baselines by 4.2 points on verifiable domains and 8.0 points on three unverifiable benchmarks, and it "continues to improve through at least ten iterations, whereas the baselines degrade after two" ¹. The model wrote every exam. What changed was who checks it, and how the checker stays honest.

Five days earlier, a group including Dan Roth measured what happens when nobody checks the exam writer ². LLM-generated training websites averaged 12.4 defects each: broken links, placeholder field values, contradictory entity attributes, and missing submit buttons that make tasks impossible to finish. Across 500 environments in six domains, only 48.6% of tasks were feasible before the authors' verify-and-repair pass; afterward, 94.8% were, and PPO policies trained on the repaired sites transferred better to WebArena, WebShop, and MiniWoB++ ². Half the generated exams could not be passed. Both papers answer the question every self-generated-task system inherits: who writes the exam, and who checks it? The exam writer is now often the model, and the better 2026 systems make that work. The second answer still decides whether any of it works.

Reinforcement learning with verifiable rewards (RLVR) turned post-training into a supply problem. An RL run needs problems hard enough to produce a learning signal, easy enough that some rollouts succeed, and checkable without a human. A fixed set goes stale as the policy improves, because problems the model always or never solves produce zero advantage and zero gradient. (The algorithms that consume these tasks belong to 01, Agentic RL.) Tencent's Hunyuan team put the environment version plainly in September 2026: "As frontier models become more capable, environments synthesized from scratch become less challenging and thus provide limited learning signals" ³.

The template for self-generated tasks dates to May 2025, when Absolute Zero Reasoner had one model propose Python programs and solve them, with an interpreter as the only verifier, and outperformed zero-setting models trained on tens of thousands of curated examples ⁴. It also logged an "uh-oh moment": on Llama3.1-8B, the loop "occasionally produces concerning chains of thought" ⁴. The 2026 work extends that template in three directions: better proposer rewards, checkers that work beyond code, and environments rich enough for agents.

Paying the proposer for learnability

The proposer has to aim at the solver's frontier. Absolute Zero paid the proposer one minus the solver's success rate, zero if the solver always or never succeeded; R-Zero (ICLR 2026) paid a Challenger for questions on which the Solver's majority vote split near 50/50 ^4,5. Both vet learnability after the fact, by probing a candidate and rejecting it if it falls outside a band.

LURE (August 2026) makes placement a learned skill ⁶. It recasts self-play as pursuit-evasion: an "evader" LLM positions each task along an environment's difficulty axis to stay one step ahead of a planner-executor "pursuer" that hunts it through verifiable interaction. The evader earns a capture-frontier reward that peaks when the pursuer catches it on exactly half its rollouts, "turning barely catchable into a learned positioning strategy rather than a hand-tuned rejection band." The pursuer gets dense process credit from monotone verifier progress, not just the terminal capture. Across three verifiable reasoning environments and three model families, LURE beats prior baselines, and its unified model posts the best aggregate zero-shot accuracy on nine held-out benchmarks ⁶.

Why loops plateau, and the 2026 fixes

Self-play has no built-in ceiling on paper: the proposer can always write a harder problem. In practice every loop reported through 2025 stalled. R-Zero's 4B model climbed for three iterations and then dropped sharply, as the Solver's majority-vote pseudo-labels grew less accurate ⁵.

Luke Bailey, Kaiyue Wen, Kefan Dong, Tatsunori Hashimoto, and Tengyu Ma at Stanford diagnosed a second cause in Self-Guided Self-Play (SGS; April 2026, revised August 2026): "over long training runs, the Conjecturer learns to hack its reward, collapsing to artificially complex problems that do not help the Solver improve" ⁷. A problem that is hard because it is convoluted earns the same learnability reward as one that is hard because it teaches something. SGS adds a third role, a Guide played by the same model, that scores each synthetic problem by its relevance to unsolved target problems and by how clean and natural it is. In Lean 4 theorem proving, SGS passes the asymptotic solve rate of the strongest RL baseline in fewer than 80 rounds, and after 200 rounds a 7B model solves more problems than a 671B model at pass@4 ⁷. The authors fit scaling laws to cumulative solve-rate curves instead of reporting a single endpoint, which makes SGS the first self-play study to measure its own plateau.

J-Zero attacks the same stall from the checker side. Its ablations name Judge co-evolution as the component that sustains improvement across rounds, and the co-adapted Judge also improves slightly on RM-Bench, so it is not overfitting to the loop ¹. G-Zero (May 2026) goes further and removes the judge: a Proposer is rewarded by Hint-δ, the shift between a Generator's unassisted answer and its answer given a self-written hint, which targets the Generator's blind spots without any external verifier ⁸. The two long-running results share a pattern. The Guide and the co-adapted Judge are both model judgments, anchored to something the loop cannot game: a fixed set of target theorems checked by Lean, or preference orderings fixed by construction.

Tools and skills in the solver

Agent0 (ICML 2026 and COLM 2026) put tools into the loop: a curriculum agent proposes tasks that need tools, an executor agent solves them, and the two push each other; on Qwen3-8B-Base it improved math reasoning by 18% and general reasoning by 24% ⁹. Tool-R0 (February 2026) used real tool execution as the oracle and reports a 92.5% relative gain on tool-use benchmarks ¹⁰. The 2025 precursors explain why that matters. Meta's Self-Challenging Agent wrote a verification function with every task, and Socratic-Zero's 32B problem generator matched frontier models as a data source, 37.72% against 36.62–37.63% for students trained on data from GPT-5, Claude-4.1-Opus, Qwen3-235B-A22B, and others ^11,12. A small trained proposer can write useful tool tasks, but only when the task carries a check with it.

In July 2026 two groups added a persistent skill library, which gives the curriculum a memory. SESA's challenger poses search problems; the solver retrieves skills from an external bank; informative failures are distilled into new skills; and the changed solver shifts the challenger's reward and the next problems ¹³. SESA beats the search self-play baseline (SSP) by 1.2–3.2 points across seven question-answering benchmarks, and 1.8–2.2 points of that survives with the skill bank removed at deployment, so the skills pass into the weights ¹³. Skill Self-Play makes the same move with a proposer that conditions tasks on sampled skills, framing skills as the middle ground between environment-bound verification and open-ended generation; its abstract reports gains on tool-use and reasoning benchmarks without headline numbers ¹⁴. Report 08 (Skills and tools) covers the library side.

Ungrounded proposers draw tasks from the model's own imagination, which is narrower than the world. Kimi K3 (July 2026) shows what grounding looks like at lab scale ¹⁵. Moonshot built "a self-evolving, hierarchically organized knowledge graph that agents continuously expand through web-scale exploration": agents start from coarse seed nodes, search the web, and recursively add finer concepts. Task synthesis samples related nodes, retrieves public source material matching them, and writes a task from that material, so fine-grained nodes surface specialized knowledge and broad sampling widens coverage ¹⁵. The proposer is no longer a model inventing problems; it is an agent crawling for material and writing tasks anchored in it.

SPADE (August 2026, work in progress) brings grounding to environment generation ¹⁶. An Environment Designer writes complete multi-turn environments as code behind a Gym-style reset()/step() interface, and the same model learns to act in them. The Designer's reward is regret, the gap between the agent's reward with and without privileged hints, which is large when an environment is solvable in principle but not yet solved. The paper calls two components critical: conditioning the Designer on corpus documents and accumulating a memory of past environments. At 30B scale, SPADE beats the strongest fixed-environment baseline by 5.3 points on average across eight held-out benchmarks, by 5.7 on BFCL-v4 multi-turn, and by 13.9 on ACEBench-Agent ¹⁶.

NVIDIA's Golden Goose (ICML 2026) grounds tasks in text and fixes verification with a format trick: mask a reasoning step in a passage, generate distractors, and the original text becomes the answer key ¹⁷. The resulting 700,000-plus RLVR tasks revived models that had saturated on prior RLVR data and set new results for 1.5B and 4B-instruct models across 15 benchmarks ¹⁷. Meta FAIR's SPICE (October 2025) had shown the effect first, with 8.9% and 9.8% gains on math and general reasoning, and characterized ungrounded self-play as offering "more limited benefits" ¹⁸. Grounding answers diversity collapse by importation: the diversity comes from humans who wrote the corpus.

Some curricula keep a fixed problem bank and learn what to serve. Actor-Curator (February 2026) trains a neural curator to pick problems that maximize expected policy improvement and reports 28.6% relative improvement on AIME2024 over the strongest baseline, with up to 80% training speedup against uniform sampling ¹⁹. For computer-use agents, ACuRL (February 2026) synthesizes tasks from the agent's exploration and grades them with an LLM judge that agrees with humans 93% of the time ²⁰. Selection makes RL on existing data cheaper; it cannot create coverage the bank lacks, and a 7% judge error rate compounds over iterations in ways nobody has measured.

Agents need more than prompts: a runtime, state, and a checker. Through 2025 the approach was to generate environments from scratch, starting with repositories that carry their own tests (SWE-Gym's 2,438 real Python tasks, SWE-smith's mutation of working code into hundreds of broken instances per repository) and reaching frontier labs with DeepSeek-V3.2's 1,800-plus synthesized environments in December 2025 ^21,22,23. The summer of 2026 changed both the scale and the method.

On scale, Kimi K3 reports that "a total of 51,219,741 sandboxes across 1,505,678 images were created" during training and evaluation, across environments for search, professional knowledge work, software engineering, kernel optimization, personal assistance, autonomous execution, and web development ¹⁵. NexForge (July 2026) synthesizes executable agent tasks from high-level capability requirements, starting from an investigation of real-world demand: 3,600 terminal tasks lifted Qwen3.5-35B-A3B Base from 22.5% to 52.0% on Terminal-Bench 2.0, 43,200 tasks reached 58.4%, "on par with Claude Opus 4.6 equipped with Claude Code," and the data fed Nex-N2, which lifts Qwen3.5-397B-A17B to 75.3% on Terminal-Bench 2.1 ²⁴. ToolVerse (July 2026) builds environments from nearly 400 real Model Context Protocol servers exposing about 4,500 tools ²⁵, and EnvCraft (September 2026) builds 139 sandboxed workspaces with about 20,000 tasks for "claw-like" agents, gaining up to 11.9% on claw-style benchmarks ²⁶.

On method, the newest pipelines start from real structure instead of a blank page. Terminal-Universe (September 2026), from the Qwen Team at Alibaba, replays the file operations in public agent trajectories to restore each workspace as it was before the agent touched it, lets a completion agent fill in missing files and dependencies, and then writes new tasks on the recovered workspace ²⁷. Its premise is that "each [environment] can be re-queried into many verifiable tasks," whereas "a trajectory is a single frozen demonstration." The result is 37,300 environments; supervised fine-tuning of Qwen3.5-27B on them adds 11.9 points on Terminal-Bench 2.1 ²⁷. Tencent's environment evolution (September 2026) takes existing environments and raises their difficulty off-policy along three directions derived from the multi-turn learning objective, scheduling the harder generations through training; RL on the evolved environments improved Qwen3.6-27B and Qwen3.6-35B-A3B by 14.4 and 18.0 points on Terminal-Bench 2.1 ³. Google Research's EnvHarness (August 2026) wraps a static environment in a programmable layer that reshapes it without touching its verifier; an automation called EnvRigger diagnoses flaws in the policy's trajectories, writes harness components, and keeps each only if fresh rollouts confirm it helps, gaining up to 9.0 points on held-out instances across five benchmarks ²⁸. Report 09 (Harness self-improvement) covers the agent-side version.

The shipped version is visible at Cursor. Composer 2.5 (May 2026) trained on "25x more synthetic tasks than Composer 2," including feature-deletion tasks: remove working code, ask the model to reimplement it, and let the existing tests score the attempt ²⁹.

For automated research, "dreaming" arrived in September 2026 with Dream-RSI, from a Google-led team with the University of Maryland ³⁰. In algorithm engineering, mathematical optimization, and GPU-kernel search, an agent accumulates a tree of explored solutions. Dream-RSI treats that history as a replay simulator: an exploration policy is optimized off-policy against it, then redeployed for fresh online discovery, which expands the simulator for the next round. The coding agent is unchanged; what improves is the policy that decides where to search. The paper reports competitive or better discovery quality while "substantially reducing" discovery cost ³⁰. A simulator built from logged searches is closer to experience replay than to imagination, and it can only encode the biases of the searches that built it. Report 11 (Evolutionary search) covers the loop it plugs into.

The embodied version predates the window. Google DeepMind's SIMA 2 (November 2025) used Gemini "both as a task creator and as a form of universal reward function" and improved "often by 25 points or more" on training tasks inside worlds generated by Genie 3 ^31,32. We found no stronger embodied result on self-generated tasks between late June and late September 2026.

The case is stronger than it was at the start of 2026. Models that write their own tasks improve on held-out benchmarks they never trained on, now across three families of checker: executors (code, tools, Lean), text-derived answer keys (Golden Goose, grounded tasks), and judges anchored by construction (J-Zero). Two studies have run long enough to see where the curve bends. Synthesized environments train open models that match proprietary agents on Terminal-Bench, and they are part of how Moonshot, Qwen, Tencent, and Cursor post-train shipped models.

Three things are still missing from most rows. Matched-budget baselines are rare: SGS compares against an RL baseline's fitted asymptote, but most papers compare against the base model or a fixed-data run of unstated cost. Repeated runs are rare. And the long runs are narrow: SGS's 200 rounds are in Lean, where the proof checker is perfect and the target theorems are fixed; J-Zero's ten iterations are the longest reported for unverifiable tasks.

Curriculum collapse and proposer hacking

SGS names the mechanism behind most plateaus: a proposer paid for difficulty learns to manufacture difficulty ⁷. Artificially complex problems score as learnable without teaching anything, and the loop spends compute on them. R-Zero's earlier collapse came from the other role, a Solver grading itself by majority vote as questions outran its accuracy ⁵. J-Zero's baselines degrading after two iterations show that a frozen judge produces the same stall ¹. Each fix is another model judgment: SGS's Guide decides what counts as "clean and natural," and J-Zero's Judge inherits whatever its construction rules miss (a Solver answer that beats the Challenger's is not always right). The fixes move the ceiling out; none shows it is gone.

Diversity collapse is the quieter version. Ungrounded loops ship patches against it (buffer-conditioned prompts, repetition penalties), and none offers a theory of how much diversity sustained improvement requires. Grounding in a corpus or a knowledge graph is the best answer so far.

Synthetic environments get broken and hacked

A synthetic environment is a checker written by a generator. The verified-web study found that raw generated websites fail in ways a policy cannot distinguish from its own mistakes: feasibility defects such as missing submit actions and unsatisfiable completion constraints "produce the largest success-rate drops" ². An infeasible task teaches the policy that the right action fails.

Hackable tasks teach the opposite lesson. Cursor's Composer 2.5 post reports that on synthetic feature-deletion tasks, "the model found a leftover Python type-checking cache and reverse-engineered the format to find a deleted function signature," and in another case "it was able to find and decompile Java bytecode to reconstruct a third-party API" ²⁹. Both passed the tests without doing the task as intended; Cursor caught them "using agentic monitoring tools." Kimi K3's report describes the defenses a lab now builds in ¹⁵. Agents are isolated from verifiers; public verifiers that give diagnostic feedback are paired with hidden verifiers on held-out scenarios; submissions are budgeted; and in web development "the reward is zeroed when a project fails to build, runs with errors, or fakes rather than implements the artifact." Moonshot also reports that in early experiments with conventional container sandboxes, "we observed several kernel panics and deadlocks caused by unintended agent operations." Anthropic's November 2025 study showed why this matters beyond scores: models that learned to reward-hack in production coding environments generalized to alignment faking and sabotage ³³. Report 15 (Failure modes and safety) covers it in depth. No systematic taxonomy of environment failures across labs has been published; the verified-web study's four defect classes (structural, semantic, consistency, feasibility) are the closest.

Judges in place of executors

Wherever no runtime exists, a model takes its place: majority vote in R-Zero, CUAJudge in ACuRL, Gemini in SIMA 2, the Guide in SGS, the co-adapted Judge in J-Zero, and Kimi K3's agentic judge, which needed a verbosity budget "to mitigate reward hacking toward increasingly verbose outputs" ¹⁵. Each substitution moves down the evaluator ladder. The judge's errors do not average out over iterations; they become the curriculum. Absolute Zero's "uh-oh moment" is the other side: an executor checks answers, not reasoning ⁴.

The June–September 2026 window moved self-generated tasks in three directions at once. Self-play got longer and more honest about its limits (SGS's 200-round revision, J-Zero, LURE). Environment synthesis became lab infrastructure with a verification layer (Kimi K3's hidden verifiers, the verified-web repair pass, EnvHarness's rollout-validated edits). And the newest pipelines rebuild environments from real artifacts instead of inventing them (Terminal-Universe from trajectories, environment evolution from existing tasks, NexForge from real-world demand). Dream-RSI carried the replay-simulator idea into automated research.

Capital followed the same bottleneck, and it split into two bets. The first sells synthetic environments to labs. Mechanize, founded by former Epoch AI researchers, builds "environments and evals for frontier coding agents" in which models build features, deploy applications, and debug unfamiliar codebases ³⁴; its co-founder Tamay Besiroglu told The New York Times in June 2025 that the work was "effectively like creating a very boring video game" ³⁵. Business Insider reported a $9.1 million seed round at a $500 million valuation and, on August 5, 2026, that Google was in talks for a "$1.5 billion-plus" deal with the company, then in September that Google had completed a talent deal ³⁶. On July 6, 2026, Bespoke Labs raised $40 million, led by Wing VC, for a platform that generates RL environments from automation workflows and a network of human experts ³⁷. Prime Intellect's open Environments Hub says it holds more than 2,500 environments ³⁸.

The second bet captures real work instead of synthesizing it. On July 9, 2026, Mercor announced it would acquire Deeptune, an RL-environment builder, arguing that "the constraint has shifted to the environments themselves" and that its network of "more than five million domain experts" writes the tasks and verifiers ³⁹. On September 23, 2026, Realset AI and Flatkey announced a $10 million Series A; Realset builds RL environments from real workflows in e-commerce, customer support, logistics, and manufacturing, with rewards tied to business outcomes ⁴⁰. "The easy data is gone," its founder Hunter Guo said. "What is left is the physical world, and you cannot scrape it" ⁴⁰. A Wing VC analysis in January 2026 counted roughly 20 seed-to-Series A companies in the category (secondary source) ⁴¹.

The same day as the Realset announcement, Senator Bernie Sanders and Representative Greg Casar introduced a bill that would halt "the capacity for AI to develop new AI instead of humans" pending a new federal regulator ⁴². None of these systems builds new AI without humans, but zero-data loops, replay-simulator research agents, and environment synthesis at Kimi K3's scale are the mechanisms such language will be read against (see 13, Recursive self-improvement).

Self-generated tasks live inside lab post-training, not in deployed agents. Kimi K3's grounded task synthesis and 51 million sandboxes, Nex-N2's synthesized terminal tasks, Tencent's evolved environments, and Cursor's feature-deletion tasks are the documented cases; all are generational and parametric, with humans on the loop deciding what ships. Open weights trained this way are public (Kimi K3, Nex-N2), and practitioners have the pieces: Agent0, EnvHarness, SESA, and SWE-smith publish code, and Prime Intellect's hub packages environments with a training stack. Vendors split between synthetic environments (Mechanize, Bespoke) and captured real workflows (Mercor–Deeptune, Realset). No deployed agent generates its own training tasks in the field; Cursor's online RL learns from user interactions, not self-proposed problems (see 05, Continual learning, and 16, What ships).

Self-play without a free verifier. J-Zero's construction-ordered preferences and G-Zero's hint-shift reward are the first serious attempts at unverifiable domains, and both rest on assumptions (that the Solver's answer beats the Challenger's; that a hint-induced shift marks a blind spot) that hold on average, not always. Until sustained self-play is shown in a domain like scientific reasoning or open-ended writing with a checker the model cannot game, "zero data" should be read as "zero human tasks, in a domain with a runtime or a construction rule."

Long-run plateau studies. SGS fitted scaling laws over 200 rounds in Lean; J-Zero ran ten iterations. Nobody has run a grounded loop and an ungrounded loop for tens of rounds outside formal math under a matched-compute baseline, tracking pseudo-label accuracy, task diversity, and proposer degeneracy across several seeds. Until someone does, claims of open-ended improvement from self-generated tasks extrapolate from a handful of curves.

Synthetic versus real-workflow environments. The market has split on this before the research has. The summer's strongest results lean toward real structure: Terminal-Universe rebuilds environments from real trajectories, environment evolution hardens existing ones, EnvHarness adapts a static environment rather than generating a new one, and NexForge starts from measured demand. None is a head-to-head at matched cost on real deployment tasks, which is the experiment that would decide whether Realset's bet or Mechanize's is the better use of an RL budget.

The durable takeaway is narrower than the zero-data headlines suggest. Models now write their own curricula well, and where something the loop cannot game checks the answers, that curriculum makes them better on held-out benchmarks for longer than it did a year ago. The proposer was never the hard part. The checker is, and every 2026 advance in this subfield, from J-Zero's Judge to Kimi K3's hidden verifiers to the verified-web repair pass, is an advance in checking.