Self-generated tasks

J-Zero, posted by Gyouk Chu, Myeongho Jeon and Eunho Yang in late August, is a self-play system with three roles drawn from one base model ¹. A Challenger writes increasingly hard tasks, a Solver answers them, and a Judge scores the answers. The Judge is the new part. Earlier zero-data loops either needed an executor to grade answers or froze a judge at the start, and a frozen judge falls behind the Solver it is grading.

J-Zero retrains the Judge every iteration on preference pairs whose ordering is known from how each answer was produced: the Solver's answer beats the Challenger's, and the Solver's decomposed-and-recombined answer beats its one-shot answer. The Judge never learns from its own scores. J-Zero beats its baselines by 4.2 points on verifiable domains and 8.0 points on three unverifiable benchmarks, and it "continues to improve through at least ten iterations, whereas the baselines degrade after two" ¹. The model wrote every exam in that loop, and what kept it improving was a checker that keeps up with the Solver without ever grading itself.

Five days earlier, a group including Dan Roth measured the cost of leaving the exam writer unchecked ². LLM-generated training websites averaged 12.4 defects each: broken links, placeholder field values, contradictory entity attributes, and missing submit buttons that make tasks impossible to finish. Across 500 environments in six domains, only 48.6% of tasks were feasible before the authors' verify-and-repair pass. Afterward 94.8% were, and PPO policies trained on the repaired sites transferred better to WebArena, WebShop and MiniWoB++ ². Half the generated exams couldn't be passed at all.

Every system covered below has something that writes tasks and something that checks answers. In 2026 the task writer is usually the model itself, and it does that job reasonably well. In most of the failures below, from R-Zero's collapse to the unrepaired websites, the checker is the reason.

Reinforcement learning with verifiable rewards (RLVR) turned post-training into a supply problem. An RL run needs problems hard enough to produce a learning signal, easy enough that some rollouts succeed, and checkable without a human. A fixed set goes stale as the policy improves, because a problem the model always solves, or never solves, gives zero advantage and zero gradient. Tencent's Hunyuan team described the environment version of the problem in September: "As frontier models become more capable, environments synthesized from scratch become less challenging and thus provide limited learning signals" ³.

The template for self-generated tasks is Absolute Zero Reasoner from May 2025. One model proposed Python programs and solved them, with an interpreter as the only verifier, and it outperformed zero-setting models trained on tens of thousands of curated examples ⁴. It also logged an "uh-oh moment": on Llama3.1-8B, the loop "occasionally produces concerning chains of thought" ⁴. The 2026 work extends that template in three directions: better rewards for the proposer, checkers that work beyond code, and environments rich enough for agents.

Rewarding the proposer for learnable tasks

The proposer has to write problems right at the edge of what the solver can do. Absolute Zero paid the proposer one minus the solver's success rate, and zero if the solver always or never succeeded; R-Zero (ICLR 2026) paid a Challenger for questions on which the Solver's majority vote split near 50/50 ^4,5. Both check learnability after the fact, by probing a candidate problem and rejecting it if it falls outside a band.

LURE, from August, makes the placement a learned skill ⁶. It recasts self-play as pursuit and evasion. An "evader" LLM positions each task along an environment's difficulty axis to stay one step ahead of a planner-executor "pursuer" that hunts it through verifiable interaction.

The evader earns a capture-frontier reward that peaks when the pursuer catches it on exactly half its rollouts, "turning barely catchable into a learned positioning strategy rather than a hand-tuned rejection band." The pursuer gets dense process credit from monotone verifier progress as well as from the final capture. Across three verifiable reasoning environments and three model families, LURE beats prior baselines, and its unified model posts the best aggregate zero-shot accuracy on nine held-out benchmarks ⁶.

Why self-play loops stall

On paper self-play has no built-in limit, because the proposer can always write a harder problem. In practice every loop reported through 2025 stalled. R-Zero's 4B model climbed for three iterations and then dropped sharply, as the Solver's majority-vote pseudo-labels became less accurate ⁵.

Luke Bailey, Kaiyue Wen, Kefan Dong, Tatsunori Hashimoto and Tengyu Ma at Stanford found a second cause in Self-Guided Self-Play (SGS; April 2026, revised August 2026): "over long training runs, the Conjecturer learns to hack its reward, collapsing to artificially complex problems that do not help the Solver improve" ⁷. A problem that is hard because it is convoluted earns the same learnability reward as one that is hard because it teaches something, so a proposer paid for difficulty learns to manufacture difficulty and the loop spends compute on it.

SGS adds a third role, a Guide played by the same model, that scores each synthetic problem by its relevance to unsolved target problems and by how clean and natural it is. In Lean 4 theorem proving, SGS passes the asymptotic solve rate of the strongest RL baseline in fewer than 80 rounds, and after 200 rounds a 7B model solves more problems than a 671B model at pass@4 ⁷. The authors fit scaling laws to cumulative solve-rate curves instead of reporting one endpoint, which makes SGS the first self-play study to measure its own plateau.

J-Zero goes after the stall from the checker side. Its ablations name Judge co-evolution as the component that sustains improvement across rounds, and the co-adapted Judge also improves slightly on RM-Bench, which suggests it isn't just overfitting to the loop ¹. G-Zero, from May, removes the judge entirely. Its Proposer is rewarded by Hint-δ, the shift between a Generator's unassisted answer and its answer given a self-written hint, which targets the Generator's blind spots without any external verifier ⁸.

What the two long-running results have in common is specific. SGS's Guide and J-Zero's Judge are both model judgments, and both are tied to something the loop can't game: a fixed set of target theorems checked by Lean, or preference orderings fixed by how the answers were produced. They also inherit whatever those anchors miss. The Guide decides what counts as "clean and natural," and J-Zero's construction rules have gaps (a Solver answer that beats the Challenger's is not always right). I'd say these fixes push the plateau further out, and neither shows it is gone.

Diversity collapse is the slower failure. Ungrounded loops ship patches against it, such as buffer-conditioned prompts and repetition penalties, and the papers offer no theory of how much diversity sustained improvement needs. The answer so far has been to ground tasks in a corpus or a knowledge graph, which the next section covers.

Adding tools and skill libraries

Agent0 (ICML 2026 and COLM 2026) put tools into the loop. A curriculum agent proposes tasks that need tools, an executor agent solves them, and each pushes the other; on Qwen3-8B-Base it improved math reasoning by 18% and general reasoning by 24% ⁹. Tool-R0 used real tool execution as the oracle and reports a 92.5% relative gain on tool-use benchmarks ¹⁰.

Two 2025 papers show why the tools help. Meta's Self-Challenging Agent wrote a verification function with every task ¹¹, and Socratic-Zero's 32B problem generator matched frontier models as a data source, 37.72% against 36.62–37.63% for students trained on data from GPT-5, Claude-4.1-Opus, Qwen3-235B-A22B and others ¹². A small trained proposer can write useful tool tasks, but only when each task comes with its own check.

Two July papers gave the curriculum a memory by adding a persistent skill library. In SESA, a challenger poses search problems, the solver retrieves skills from an external bank, informative failures are distilled into new skills, and the changed solver shifts the challenger's reward and therefore the next problems ¹³.

SESA beats the search self-play baseline (SSP) by 1.2–3.2 points across seven question-answering benchmarks, and 1.8–2.2 points of that remains with the skill bank removed at deployment, so the solver had absorbed the skills ¹³. Skill Self-Play has a proposer that conditions tasks on sampled skills, and it frames skills as a middle ground between verification tied to an environment and open-ended generation; its abstract reports gains on tool-use and reasoning benchmarks without headline numbers ¹⁴.

A proposer that works only from its own imagination draws tasks from a narrower distribution than the world. Kimi K3 shows what grounding looks like at lab scale ¹⁵. Moonshot built "a self-evolving, hierarchically organized knowledge graph that agents continuously expand through web-scale exploration."

Agents start from coarse seed nodes, search the web, and recursively add finer concepts. Task synthesis samples related nodes, retrieves public source material that matches them, and writes a task from that material, so fine-grained nodes bring in specialized knowledge and broad sampling widens coverage ¹⁵. The proposer here is an agent crawling for material and writing tasks tied to what it found.

SPADE, a work-in-progress paper from August, applies grounding to environment generation ¹⁶. An Environment Designer writes complete multi-turn environments as code behind a Gym-style reset()/step() interface, and the same model learns to act in them. The Designer's reward is regret: the gap between the agent's reward with and without privileged hints, which is large when an environment is solvable in principle but not yet solved.

The paper calls two components critical, conditioning the Designer on corpus documents and keeping a memory of past environments. At 30B scale, SPADE beats the strongest fixed-environment baseline by 5.3 points on average across eight held-out benchmarks, by 5.7 on BFCL-v4 multi-turn, and by 13.9 on ACEBench-Agent ¹⁶.

NVIDIA's Golden Goose (ICML 2026) grounds tasks in text and solves verification with a format trick. Mask a reasoning step in a passage, generate distractors, and the original text becomes the answer key ¹⁷. The resulting 700,000-plus RLVR tasks revived models that had saturated on earlier RLVR data and set new results for 1.5B and 4B-instruct models across 15 benchmarks ¹⁷. Meta FAIR's SPICE had shown the effect first, in October 2025, with 8.9% and 9.8% gains on math and general reasoning, and described ungrounded self-play as offering "more limited benefits" ¹⁸. Grounding fixes diversity collapse by importing diversity from the humans who wrote the corpus.

Some curricula keep a fixed problem bank and learn what to serve from it. Actor-Curator trains a neural curator to pick problems that maximize expected policy improvement, and reports 28.6% relative improvement on AIME2024 over the strongest baseline, with up to 80% faster training than uniform sampling ¹⁹. For computer-use agents, ACuRL synthesizes tasks from the agent's exploration and grades them with an LLM judge that agrees with humans 93% of the time ²⁰.

Selection makes RL on existing data cheaper, but it can't add coverage the bank doesn't have. And a judge that is wrong 7% of the time is wrong in every iteration, and how those errors accumulate hasn't been measured.

An agent needs more than a prompt to practice on: it needs a runtime, state, and a checker. Through 2025 the approach was to generate environments from scratch. It started with repositories that come with their own tests: SWE-Gym's 2,438 real Python tasks, and SWE-smith, which mutates working code into hundreds of broken instances per repository ^21,22. It reached the frontier labs with DeepSeek-V3.2's 1,800-plus synthesized environments in December 2025 ²³. Over the summer of 2026 both the scale and the method changed.

Kimi K3 reports that "a total of 51,219,741 sandboxes across 1,505,678 images were created" during training and evaluation, in environments for search, professional knowledge work, software engineering, kernel optimization, personal assistance, autonomous execution and web development ¹⁵. NexForge synthesizes executable agent tasks from high-level capability requirements, starting from an investigation of real-world demand. Its 3,600 terminal tasks lifted Qwen3.5-35B-A3B Base from 22.5% to 52.0% on Terminal-Bench 2.0, 43,200 tasks reached 58.4%, "on par with Claude Opus 4.6 equipped with Claude Code," and the data fed Nex-N2, which lifts Qwen3.5-397B-A17B to 75.3% on Terminal-Bench 2.1 ²⁴.

ToolVerse builds environments from nearly 400 real Model Context Protocol servers exposing about 4,500 tools ²⁵, and EnvCraft builds 139 sandboxed workspaces with about 20,000 tasks for "claw-like" agents, gaining up to 11.9% on claw-style benchmarks ²⁶.

The change in method interests me more than the scale. The newest pipelines start from structure that already exists. Terminal-Universe, from the Qwen Team at Alibaba, replays the file operations in public agent trajectories to restore each workspace to the state it was in before the agent touched it. A completion agent fills in missing files and dependencies, and then the pipeline writes new tasks on the recovered workspace ²⁷.

The authors' argument is that "each [environment] can be re-queried into many verifiable tasks," whereas "a trajectory is a single frozen demonstration." They end up with 37,300 environments, and supervised fine-tuning of Qwen3.5-27B on them adds 11.9 points on Terminal-Bench 2.1 ²⁷.

Tencent and Google Research both start from environments that already exist. Tencent's environment evolution raises the difficulty of existing environments off-policy, along three directions derived from the multi-turn learning objective, and schedules the harder generations through training; RL on the evolved environments improved Qwen3.6-27B and Qwen3.6-35B-A3B by 14.4 and 18.0 points on Terminal-Bench 2.1 ³.

Google Research's EnvHarness wraps a static environment in a programmable layer that reshapes it without touching its verifier. An automation called EnvRigger diagnoses flaws in the policy's trajectories, writes harness components, and keeps each one only if fresh rollouts confirm it helps, which gained up to 9.0 points on held-out instances across five benchmarks ²⁸.

Cursor has published details of a shipped version. Composer 2.5 trained on "25x more synthetic tasks than Composer 2," including feature-deletion tasks: remove working code, ask the model to reimplement it, and let the existing tests score the attempt ²⁹.

A synthetic environment is a checker written by a generator, and it can be wrong in two directions. The first is infeasibility. The verified-web study found that raw generated websites fail in ways a policy can't tell apart from its own mistakes, and that feasibility defects such as missing submit actions and unsatisfiable completion constraints "produce the largest success-rate drops" ². An infeasible task teaches the policy that the right action fails.

A hackable task teaches the opposite. Cursor's Composer 2.5 post reports that on synthetic feature-deletion tasks, "the model found a leftover Python type-checking cache and reverse-engineered the format to find a deleted function signature," and in another case "it was able to find and decompile Java bytecode to reconstruct a third-party API" ²⁹. Both passed the tests without doing the task as intended, and Cursor caught them "using agentic monitoring tools."

Kimi K3's report describes the defenses a lab now builds in ¹⁵. Agents are isolated from verifiers. Public verifiers that give diagnostic feedback are paired with hidden verifiers on held-out scenarios. Submissions are budgeted, and in web development "the reward is zeroed when a project fails to build, runs with errors, or fakes rather than implements the artifact." Moonshot also reports that in early experiments with conventional container sandboxes, "we observed several kernel panics and deadlocks caused by unintended agent operations."

Anthropic's November 2025 study showed that the damage goes beyond inflated scores: models that learned to reward-hack in production coding environments generalized to alignment faking and sabotage ³³. A systematic taxonomy of environment failures across labs hasn't been published yet; the verified-web study's four defect classes (structural, semantic, consistency, feasibility) are a first version of one.

Where no runtime exists, a model does the checking: majority vote in R-Zero, CUAJudge in ACuRL, Gemini in SIMA 2, the Guide in SGS, the co-adapted Judge in J-Zero, and Kimi K3's agentic judge, which needed a verbosity budget "to mitigate reward hacking toward increasingly verbose outputs" ¹⁵. A model judge is a weaker check than an executor. A judge's errors don't average out over iterations, because the loop trains on what the judge accepted, and the next round's tasks are built on that. Absolute Zero's "uh-oh moment" shows the limit of executors too: an interpreter checks answers and never looks at the reasoning that produced them ⁴.

Dream-RSI, posted in September by a Google-led team with the University of Maryland, applies "dreaming" to automated research ³⁰. In algorithm engineering, mathematical optimization and GPU-kernel search, an agent builds up a tree of explored solutions. Dream-RSI treats that history as a replay simulator. An exploration policy is optimized off-policy against it and then redeployed for fresh online discovery, which expands the simulator for the next round.

The coding agent is unchanged; what improves is the policy that decides where to search. Its authors report competitive or better discovery quality while "substantially reducing" discovery cost ³⁰. I'd call a simulator built from logged searches experience replay more than imagination, and it can only encode the biases of the searches that built it. It plugs into the evolutionary search loop that Report 11 describes.

The embodied version came earlier. Google DeepMind's SIMA 2 (November 2025) used Gemini "both as a task creator and as a form of universal reward function" and improved "often by 25 points or more" on training tasks inside worlds generated by Genie 3 ^31,32. I couldn't find a stronger embodied result on self-generated tasks between late June and late September 2026.

The case for self-generated tasks is stronger than it was at the start of 2026. Models that write their own tasks improve on held-out benchmarks they never trained on, and that now holds for three kinds of checker: executors (code, tools, Lean), answer keys derived from text (Golden Goose, grounded tasks), and judges whose training labels are fixed by construction (J-Zero). SGS and J-Zero have run long enough to see where the curve bends. Synthesized environments train open models that match proprietary agents on Terminal-Bench, and Moonshot, Qwen, Tencent and Cursor use them to post-train models they ship.

Most rows are missing things I'd want before trusting them. Matched-budget baselines are rare: SGS compares against an RL baseline's fitted asymptote, but most papers compare against the base model or a fixed-data run of unstated cost. Repeated runs are rare too. And the long runs are narrow. SGS's 200 rounds are in Lean, where the proof checker is perfect and the target theorems are fixed, and J-Zero's 10 iterations are the longest reported for unverifiable tasks.

Money followed the shortage of environments, and investors are making two different bets. The first bet is selling synthetic environments to labs. Mechanize, founded by former Epoch AI researchers, builds "environments and evals for frontier coding agents" in which models build features, deploy applications and debug unfamiliar codebases ³⁴. Its co-founder Tamay Besiroglu told The New York Times in June 2025 that the work was "effectively like creating a very boring video game" ³⁵.

Business Insider reported a $9.1 million seed round at a $500 million valuation, then on August 5, 2026 that Google was in talks for a "$1.5 billion-plus" deal with the company, and in September that Google had completed a talent deal ³⁶. Bespoke Labs raised $40 million in July, led by Wing VC, for a platform that generates RL environments from automation workflows and a network of human experts ³⁷. Prime Intellect's open Environments Hub says it holds more than 2,500 environments ³⁸.

The second bet is capturing real work instead of synthesizing it. Mercor announced in July that it would acquire Deeptune, an RL-environment builder, arguing that "the constraint has shifted to the environments themselves" and that its network of "more than five million domain experts" writes the tasks and verifiers ³⁹.

Realset AI and Flatkey announced a $10 million Series A on September 23; Realset builds RL environments from real workflows in e-commerce, customer support, logistics and manufacturing, with rewards tied to business outcomes ⁴⁰. "The easy data is gone," its founder Hunter Guo said. "What is left is the physical world, and you cannot scrape it" ⁴⁰. A Wing VC analysis in January 2026 counted roughly 20 seed-to-Series A companies in the category (secondary source) ⁴¹.

On the same day as the Realset announcement, Senator Bernie Sanders and Representative Greg Casar introduced a bill that would halt "the capacity for AI to develop new AI instead of humans" pending a new federal regulator ⁴². Every system covered here still needs humans to build new AI. But zero-data loops, research agents trained on replay simulators, and environment synthesis at Kimi K3's scale are what language like that will be read against.

Self-generated tasks are used inside lab post-training, and I found no case of a deployed agent using them. Kimi K3's grounded task synthesis and 51 million sandboxes, Nex-N2's synthesized terminal tasks, Tencent's evolved environments and Cursor's feature-deletion tasks are the documented cases. They are all generational weight updates, with humans deciding what ships.

Open weights trained this way are public (Kimi K3, Nex-N2), and practitioners have the pieces: Agent0, EnvHarness, SESA and SWE-smith publish code, and Prime Intellect's hub packages environments with a training stack. Vendors are split between synthetic environments (Mechanize, Bespoke) and captured real workflows (Mercor–Deeptune, Realset). Cursor's online RL, the nearest case, learns from user interactions rather than from problems it proposes itself.

Can self-play work without a free verifier? J-Zero's construction-ordered preferences and G-Zero's hint-shift reward are the first serious attempts at unverifiable domains. Both depend on assumptions that hold on average and sometimes fail: that the Solver's answer beats the Challenger's, and that a shift caused by a hint marks a blind spot. Until someone shows sustained self-play in a domain like scientific reasoning or open-ended writing, with a checker the model can't game, I read "zero data" as "zero human tasks, in a domain with a runtime or a construction rule."

Where do long self-play runs plateau? SGS fitted scaling laws over 200 rounds in Lean, and J-Zero ran 10 iterations. No study yet runs a grounded loop and an ungrounded loop for tens of rounds outside formal math, against a matched-compute baseline, tracking pseudo-label accuracy, task diversity and proposer degeneracy across several seeds. Without one, claims of open-ended improvement from self-generated tasks extrapolate from a handful of curves.

Should a team buy synthetic environments or captured real workflows? The market has split on this before the research has. Four of the summer's environment results lean toward real structure: Terminal-Universe rebuilds environments from real trajectories, environment evolution hardens existing ones, EnvHarness adapts an existing static environment, and NexForge starts from measured demand. I'd bet on real structure, but these aren't head-to-head comparisons at matched cost on real deployment tasks, and that experiment is what would tell you whether Realset's bet or Mechanize's is the better use of an RL budget.