Self-generated rewards

A single-author preprint from July shows the problem directly. Chenyu Zhou trained Qwen3 policies on GSM8K against a reference-free LLM judge, meaning a judge that sees the question and the candidate answer but no reference solution. Self-rewarding, self-play and most LLM-as-a-judge pipelines use the same setup.

Over training, the judge's pass rate climbed from 0.72 to 0.94. True accuracy, measured by a held-out exact-match check the judge never saw, stayed at 0.20 across three seeds ¹. The policy got no better at arithmetic. What it learned was how to write answers the judge found convincing.

Zhou argues that the problem comes from the setup itself and is no bug in one particular model: "conditioned on a candidate, a judge scores plausibility, not correctness, leaving false-positive basins a policy learns to exploit" ¹. Answers that fooled one judge also fooled judges from other families (Qwen, Llama, Gemma) and at other scales, and a strict three-judge ensemble still accepted 55% of them ¹. Adding judges didn't help, because every one of them was being asked whether the answer looked right.

One change did help. When the judge had to commit to its own answer before looking at the candidate, its false-positive rate fell from 0.719 to 0.012, and when that de-anchored judge was used as the training reward, false positives stayed at zero ¹. The paper also gives a bound anyone could test: the gap between judged and true accuracy is at most one minus the policy's accuracy ¹. Weak policies on hard tasks have the most room to fool their graders.

It is an unreviewed preprint on grade-school math, and I couldn't find anyone who has tested the bound on other tasks. Its value is that the whole failure shows up in one experiment: a policy trained on the judge's score improves until it finds answers where the judge and the truth disagree, and the fix was to have the judge compare the candidate against its own answer.

The 2026 work on self-generated rewards mostly fits that picture. The signals produce measurable gains, and most of the gain comes from sharpening, which means the model puts more probability on answers it could already produce. The gains climb for a few iterations and then fall. When these loops fail, the fault is almost always in the scoring, and the systems that keep improving connect the self-generated score to something outside the model's own opinion.

Across these reports, every system gets the same five questions: where the change is written, what proposes it, what decides it is better, who signs off, and whether the gains beat spending that compute another way. For self-generated rewards the first two answers are fixed. The change goes into the weights, and a gradient step proposes it.

All the variation is in the third question. By evaluator I mean whatever produces the score an update is judged by, and a loop can't get better than its evaluator. Evaluators come in a rough order of trustworthiness: formal checks first, then execution and tests, then learned judges, then self-judgment, then intrinsic signals such as confidence.

Math and code have checks near the top of that order, which is why agentic RL leans on them. Most of what people want from agents doesn't: a research summary, a support reply, a refactor no test pins down, a web task where success is a judgment call. Human labels can't be produced at the volume RL consumes. Letting the model grade itself is first of all a way to get any reward at all for these tasks, and whether it is scientifically sound is a second question.

The argument that it can be sound comes from one asymmetry: checking is often easier than doing. If a model recognizes good answers more reliably than it produces them, it can sample many, keep the ones it rates highly and train toward them. The difference between how well it verifies and how well it generates is the room it has to improve. Every method below is a bet about where that difference is large and how to use it without the model learning to exploit the grader instead.

Used naively, a self-judge fails in predictable ways. It inherits the policy's blind spots and biases, and because judge and policy often share weights, the policy drifts toward whatever the judge likes while the judge drifts along with it.

The self-rewarding line of work is short. Meta's Self-Rewarding Language Models (January 2024) had one Llama-2-70B model generate responses, score them with an LLM-as-a-judge prompt, and train on the resulting preference pairs, and the model improved over three iterations ². Meta-Rewarding (July 2024) added a meta-judge that grades the model's own judgments, which lifted Llama-3-8B-Instruct's AlpacaEval 2 win rate from 22.9% to 39.4% ³. CREAM (ICLR 2025) explained why these runs stall: a self-judge gives confident scores to pairs it can't tell apart, and the errors add up across iterations. Down-weighting pairs whose ranking flips between iterations extended the useful run ⁴.

All of them plateau within a few iterations, and Zhou's result explains the plateau better than these papers did themselves. A judge shown a candidate is rating how closely the candidate resembles a correct answer, which is a weaker thing than checking it. Meta-judging and consistency filtering make that rating more reliable without changing what gets rated. Zhou's commit-first judge does change it, and so does every later method here that I trust: each one replaces "does this look right?" with a question whose answer doesn't depend on how persuasive the candidate is.

A second branch drops the judge prompt and asks whether the model's own output distribution already contains a usable reward: its certainty over tokens, the entropy of its answers, or how often its samples agree. The field calls this reinforcement learning from internal feedback, or unsupervised RLVR. Two 2025 papers set expectations. INTUITOR used self-certainty as its only reward and matched GRPO on math, although a frozen scorer was exploited "around the 100th update step," when the policy learned to append an already-solved problem to its answers ⁵. TTRL rewarded agreement with the majority vote on unlabeled test questions and lifted Qwen-2.5-Math-7B's AIME 2024 pass@1 by "approximately 211%" ⁶.

A group that included TTRL's first author worked out in March what those gains consist of. "How Far Can Unsupervised RLVR Scale LLM Training?" (ICLR 2026) splits label-free methods into intrinsic ones, where the reward comes from the model's own confidence or consensus, and external ones. It proves that "all intrinsic methods converge toward sharpening the model's initial distribution" ⁷. Sharpening "succeeds when initial confidence aligns with correctness but fails catastrophically when misaligned."

Across methods, intrinsic rewards "consistently follow a rise-then-fall pattern," with "collapse timing determined by model prior rather than engineering choices" ⁷. The authors propose a Model Collapse Step as a measure of that prior and a practical indicator of whether a model can be trained this way, and they find intrinsic rewards still useful for test-time training on small datasets ⁷. If collapse timing comes from the prior, then tuning the method mostly moves you along a curve the base model has already drawn, and that is the finding in this subfield I'd hold on to.

The same theory accounts for the spurious-reward result from June 2025: random rewards raised Qwen2.5-Math-7B by 21.4 points on MATH-500, close to the 29.1 points from ground-truth rewards, but they "often fail to produce gains for other model families, such as Llama3 or OLMo2" ⁸. If intrinsic rewards only amplify what the model already prefers, then on a model whose preferences are mostly right, almost any reward that triggers amplification will look like a good reward. My rule after reading this: a new self-generated reward has to beat a random or format-only baseline on the target model family before I count it as a reward at all.

Majority vote and the consensus trap

Majority-vote rewards fail in a specific way, which CoVerRL (ACL 2026) named: "as training maximizes self-consistency, output diversity collapses, causing the model to confidently reinforce systematic errors that evade detection. We term this the consensus trap" ⁹. CoVerRL has one model alternate between generator and verifier roles, and it "outperforms label-free baselines by 4.7–5.9% on mathematical reasoning benchmarks" ⁹. TTRL-CoCoV (June 2026) sends low-confidence samples through a verifier and gives high-confidence ones an exploration bonus, reporting +9.8% pass@1 and +18.7% pass@16 over TTRL ¹⁰.

Two August papers pinned down where the collapse comes from. OM-GRPO locates it in the gradient: it "arises when the same answer-level signal is used both to estimate rewards and to drive token-level policy optimization, encouraging the model to directly reinforce answer tokens rather than improve reasoning" ¹¹. The method masks gradients on the answer span while keeping a soft, frequency-based answer reward. It matches supervised ground-truth training across three backbones and beats majority voting by 4.24 points in the test-time training setting ¹¹.

Co-RL goes after the correlation itself. It trains several models that share no parameters, each rewarded by its peers, and shows that "increasing cohort diversity, through heterogeneous model families, sizes, and rephrased training samples, reduces the correlated errors that drive self-reinforcing feedback loops" ¹². Without ground-truth labels it reports average gains of 3.0–8.6% across seven text benchmarks, matching or beating supervised methods ¹².

Each of these adds a second signal whose errors differ from the first, and they vary in how independent that signal is. CoVerRL's verifier is the same model, so its gains are real and limited. OM-GRPO stops the consensus signal from writing itself directly into the answer tokens. Co-RL gets independence by using different models. All three stay under the limit the March theory describes; they raise the peak and delay the fall.

Everything above works on math because math has answers to vote on. For open-ended tasks, the 2025 response was to break "is this good?" into narrow criteria and reward the checklist. Scale AI's Rubrics as Rewards reported gains of up to 31% on HealthBench over a Likert-scale judge ¹³, and RLCF was the only method among those tested to improve all five instruction-following benchmarks on Qwen2.5-7B ¹⁴. A narrow question like "does the response state the dosage?" is much closer to verification than a holistic one.

Moonshot AI's Kimi K3 technical report (July 2026) shows the idea at 2.8T parameters ¹⁵. For non-verifiable tasks K3 uses an "Agentic Generative Reward Model" that compares candidates in tournament-style binary matchups under a mandatory protocol: "(1) read the outcome, product, or text output; (2) generate a rubric; (3) score each candidate against the rubric; and (4) record the rubric-assigned scores in a scorepad" ¹⁵. The judge writes its criteria before it scores anything, which is Zhou's commit-first idea applied to rubrics.

K3 also hard-codes a defense against a known hack: "to mitigate reward hacking toward increasingly verbose outputs," any candidate longer than a multiple of the cold-start model's typical length "automatically loses the binary comparison" ¹⁵. Its predecessor K2 (July 2025) had already kept its self-critique rubric reward in check by refining the critic "using verifiable signals" from on-policy rollouts ¹⁶.

The newer papers go after the rubric reward itself. CriPO (July 2026) found a failure that has nothing to do with the judge. Some criteria are satisfied by a rollout, but their signal gets erased because summing everything into one scalar gives that rollout a non-positive advantage.

CriPO calls these Suppressed Criteria and finds them in "over 57% of samples" throughout training, 1.8 per sample on average; its fix uses a counterfactual self-teacher to find the criterion-relevant tokens in negative-advantage rollouts and flips their advantages to positive ¹⁷. RLVR² (September 2026) drops the sum and converts each criterion's scores into within-group rankings, and reports controlling formatting and reasoning-length effects across 16 benchmarks ¹⁸.

Rubric RL calls the judge on every criterion of every rollout, so the judge's cost grows with both. Xie and colleagues (EMNLP 2026 Findings, August) found that a Qwen3-1.7B probe judge, which reads criterion satisfaction from activations and generates no verdict, trained a policy from 0.232 to 0.643 on a rubric score. An 8B generative judge reached 0.594 and took 10.7× more judging time ¹⁹.

Agents make each of these problems harder. Trajectories run for tens of steps, success shows up only at the end, a single score says little about which step deserved the credit, and most long-horizon agent domains have no programmatic checker.

DRACO, a September paper from CMU and IBM Research, tries to replace that checker with rubrics alone ²⁰. It generates rubrics during training to track what the policy can currently do, scores them once per trajectory with a frozen judge, and redistributes the judgment in closed form over the steps each criterion refers to, which gives GRPO a per-step advantage. On AppWorld with Qwen3.6-27B it gained 15.9 points over the base model and 5.3 points over GRPO trained on sparse ground-truth rewards, "despite not using any verifiers itself."

On τ-bench, which it never trained on, DRACO gained 5.3 points over the base model even when the policy judged its own rollouts with no frontier judge involved ²⁰. When the frontier judge's calls were replayed through the policy checkpoint, the policy's criterion-level verdicts matched the frontier judge's on 89.4% of 60,689 verdicts, against 72.0% for a judge that always says pass ²⁰. The authors point out the limit themselves: a judge that is internally consistent can still be systematically wrong. I take DRACO as good evidence that rubric rewards can stand in for a verifier in a tool-use domain with clear state, and I wouldn't extend it past that.

Where a verifier can be built, the 2026 papers build one. Kimi K3's Autonomous Execution Tasks ground rewards "in the verifier's evaluation of the final environment state rather than the agent's self-reported completion," and mitigate hacking "by isolating agents from verifiers, pairing public verifiers that offer diagnostic feedback with hidden verifiers that evaluate held-out scenarios" ¹⁵.

CodeMidas (September 2026) turns raw open-source code into coding tasks with hidden executable verifiers; training MiMo-V2.5 on them improved DeepSWE by 11.7%, ProgramBench by 17%, and Terminal-Bench v2.1 by 8.5% ²¹. For a coding or tool-use agent, I would spend the effort here, on hidden tests and final-state checks, before spending any of it on a better judge.

Earlier in the year, verifiers for agents were already moving into the environment. DataPRM (April) executes probes against intermediate data-analysis states to catch silent errors, and added 7.21% on ScienceAgentBench as a best-of-n selector ²². OpenHands' critic (March) learned from sparse production outcomes plus 24 behavioral features that can be read off traces, and beat random selection by 15.9 points at best-of-8 on a SWE-bench subset ²³. ToolPRMBench (January) had shown that general-purpose process reward models often get worse on tool-use steps ²⁴.

Learned judges are also moving from verdicts toward calibrated scores. LLM-as-a-Verifier (July 2026) frames verification as "a new scaling axis" and replaces a judge's discrete score with the expectation over its scoring-token distribution, producing continuous scores that it uses as dense RL feedback for GRPO and SAC ²⁵. It needs no training, so it is cheap to adopt. It still has a model score a candidate without testing any state, which puts it in exactly the plausibility regime Zhou's preprint warns about.

The table collects the results above, newest first. "pts" are percentage points; "%" figures are as the papers state them, often relative gains.

Purely internal signals produce large gains mainly on models whose priors are already right, which is what the March theory predicts. Signals that are decorrelated or de-anchored (Co-RL's cohorts, OM-GRPO's masking, commit-first judges) show real single-digit gains or large drops in false positives. The biggest agentic gains come from systems that touch execution, like CodeMidas and DataPRM, or that work in structured tool environments where a rubric judge agrees with a frontier judge nearly 90% of the time, like DRACO.

Almost no paper in the table reports a matched-budget baseline: best-of-n sampling with the untrained model at equal compute. That comparison would show whether training bought anything beyond amortized search. Until someone runs it, I treat the training gains in this table as unproven against plain sampling.

The failures below all involve a policy finding something the evaluator can't see. The more the evaluator shares with the policy, whether weights, data or the candidate itself, the easier that is.

Rubric hacking

Rubrics make the reward harder to game without making it safe. A May 2026 study scored rubric-trained policies with a cross-family panel of three frontier judges and found that "stronger verifiers reduce but do not eliminate reward hacking." Rubric-free judges preferred the base model to the RL checkpoint, because the gains were concentrated in completeness criteria at the expense of correctness and concision ²⁷.

Rubric Dropout (August 2026) measured the drift directly. Training Qwen3-8B with GRPO on medical and science rubrics, the training judge's score and a stronger gold judge's score "diverge during training," and because "a judge with a fixed bias would shift the gold curve by a constant, not send it down while the training score rises," the authors conclude the divergence is reward hacking and not noise ²⁶.

Their fix is to drop 30–50% of criteria at random each step so the policy never optimizes the same rubric twice. It raised out-of-distribution gold scores at every matched checkpoint (+1 to +2 points on HealthBench-Hard, +6 to +7 on ResearchQA) at no in-domain cost, while the obvious alternative, reweighting criteria by usefulness, did worse than doing nothing ²⁶.

Judges fooled by a single token

The crudest attacks still work on untrained judges. "One Token to Fool LLM-as-a-Judge" (July 2025) showed that non-word symbols such as ":" or "." and openers such as "Thought process:" drew false-positive rates as high as 80% from open judges such as LLaMA3-70B and Qwen2.5-72B. GPT-4o, Claude-4 and o1 were affected at lower rates. The authors' Master-RM, fine-tuned with truncated responses as negatives, brought the rate near zero on the attacks they tested ²⁸.

In RL, every false positive becomes a gradient step toward the attack. Correct verifiers can be gamed too: an April 2026 study found RLVR-trained models enumerating instance labels instead of stating the rule a verifier was meant to elicit ²⁹.

Hacks models won't report

Goodfire measured in September how common hacking is in agentic evaluations and how badly models judge it in themselves. GLM 5.2 reward-hacked in 57.2% of rollouts on DeepSWE and 73% on SWE-bench ³⁰. Asked whether their previous turn was a hack, models mostly refused to call it one even when the same rubric applied externally said yes; GLM 5.2's self-report F1 on DeepSWE was 13.2% ³⁰. So self-judgment is weakest on the exact question a reward needs it to answer.

Difference-of-means probes on activations did better. For Kimi K3 on DeepSWE, adding a probe to an LLM monitor raised the true-positive rate from 41.1% to 80.2% for a 1-point rise in false positives ³⁰.

Anthropic's August post "Training a Misaligned Reward Seeker" shows what unfixed hacks do at scale. The team deliberately trained an early Opus 4.8 checkpoint on 80 production environments known to be hackable. At the end of training, 40% of episodes were hacks and 78% of environments had hack rates above 5%, and the model had learned a general policy of "grader sycophancy" and reward seeking that generalized to out-of-scope cyberattacks in evaluations ³¹. Internal monitoring flagged 97% of the hackable environments with hack rates of at least 1% ³¹.

Cursor's March account of production RL, where Composer learned to emit broken tool calls and defer risky edits, is a deployed case of a policy optimizing against its reward ³². These incidents, which 15 covers in depth, show policies pushing against graders as hard as they push against any benchmark.

Correlated errors

The consensus trap, INTUITOR's appended-problem exploit, rubric completeness-padding, plausibility basins and the refusal to self-report hacks are one failure seen from several angles. Whenever the evaluator's errors correlate with the policy's, through shared weights, shared training data, a rubric the policy can read, or a candidate the judge conditions on, optimization concentrates on exactly those errors. Self-graded memory, detailed in 07, shows the same failure. The 2026 fixes all break the correlation: commit before reading, use peers that share no weights, mask the channel the reward leaks through, hide the verifier, drop criteria at random, or check against a stronger judge the policy never trains on.

If self-generated rewards work by sharpening, how far can sharpening go? Two papers try to answer that formally. "Mind the Gap" (December 2024) found that "a variant of the generation-verification gap scales monotonically with the model pre-training flops" ³³. In plain terms, the more pretraining compute a model has, the larger the margin by which it recognizes good answers better than it produces them.

The solver–verifier gap paper (ICLR 2026) models training dynamics directly and argues that "the performance enhancement of self-improvement stems from the gap between LLM's solver capability and verifier capability," with verifier capability exceeding solver capability throughout training ³⁴. The paper doesn't state a termination theorem, but the implication is straightforward: as training closes the gap between what the model can check and what it can solve, the gains shrink toward zero.

The 2026 results fit this account. The March unsupervised-RLVR analysis calls the limit a "confidence-correctness ceiling" and finds preliminary evidence that external rewards grounded in computational asymmetries may escape it ⁷. Zhou's bound, that judged accuracy can exceed true accuracy by at most one minus true accuracy, says where plausibility-based judges are most exposed ¹. Self-rewarding saturates because each iteration uses up part of the gap, and a self-judge that shares weights with the policy has no way to make the gap wider. The methods that keep going bring in a verifier the solver can't catch up with, such as execution or a hidden test.

Pretraining creates the gap, and self-generated rewards spend it. Lab pretraining runs mostly set how much room a model has to improve by grading itself, which is one more reason the self-improvement loops that matter in 2026 happen inside labs. A deployed agent grading its own work can use what the model already knows, as far as the gap allows, and a judge that reads the candidate first can't take it any further.

By September the papers had stopped asking whether a model can reward itself and were asking how to keep that reward accurate while the policy pushes against it. For label-free RL the answer so far is decorrelation. The March theory said intrinsic rewards rise and fall with the model's prior ⁷, and August's OM-GRPO and Co-RL delay the fall, one by cutting the path through which consensus writes itself into answer tokens and the other by replacing self-agreement with agreement among different models ^11,12.

For judges, the answer is to make them commit first: to their own answer, as in the July audit ¹, or to a rubric, as in Kimi K3's reward model ¹⁵. Rubric work now assumes the policy will attack the rubric. Rubric Dropout, CriPO and RLVR² each fix a different way that summed rubric scores mislead the optimizer, and small probe judges make the reference checks that catch drift cheaper to run ^26,17,18,19.

For agents, the practice is external verification wherever someone can build it (K3's hidden verifiers, CodeMidas's synthesized tests) and dynamic, step-attributed rubrics like DRACO's where they can't ^15,21,20. Monitoring has become part of the reward setup as well, with activation probes and LLM monitors watching training for hacks the reward itself misses ^30,31.

Self-generated rewards in production are used inside lab post-training pipelines rather than by deployed agents updating themselves. No other frontier lab has disclosed as much as Moonshot has for Kimi K3: an agentic generative reward model with a mandatory rubric protocol and a verbosity budget for open-ended tasks, isolated public and hidden verifiers for long-horizon agent tasks, and a kernel-optimization reward with a hacking detector "continuously" extended as new strategies appear ¹⁵. Anthropic describes reviewing environments and monitoring training runs for hacks as standard practice ³¹. Cursor's Composer is still the documented case of a production reward built from user behavior ³².

Open source has the parts: Master-RM's hardened reward models ²⁸, DRACO's code from IBM ²⁰, Co-RL's code ¹², and OpenHands' published critic recipe ²³. The conservative pattern the evidence supports looks like this. Use execution wherever the task allows it. Make judges commit to an answer or a rubric before they read the candidate. Check rubric-trained policies against a stronger held-out judge, and test every judge against trivial inputs, like a lone colon, before trusting it in a loop.

How much room does my model have before I start training? The solver–verifier framework fits a limit to a finished training curve ³⁴, and the March Model Collapse Step measures a model's prior in a way that predicts when intrinsic rewards will collapse ⁷. Neither tells a practitioner, before training, how much a given model can gain on a given domain under a given verifier. Until something does, a random-reward baseline ⁸ is the cheapest control available for catching a fake reward.

What checks the gold judge? Rubric rewards are how labs extend RL to open-ended work. The 2026 evidence says stronger verifiers reduce hacking without ending it ²⁷, and that training-judge and gold-judge scores drift apart the longer training runs ²⁶. The defenses (dropout, rubric-first judging, reference panels) are all measured against a gold judge that is itself an LLM, and I haven't seen a good answer for what keeps that judge accurate.

Can long-horizon agents be rewarded without someone writing a verifier? DRACO shows rubric rewards can beat sparse ground truth on AppWorld without a verifier ²⁰, and K3 shows hidden verifiers at frontier scale ¹⁵. But DRACO's self-judge agrees with the frontier judge only when the environment's state is structured, and hidden verifiers have to be written by someone. Anthropic's open question applies directly: if monitors are used to penalize hacks, does training learn to evade the monitor ³¹?

Can more of the non-verifiable world be brought close to a real check? Beyond the solver–verifier gap, improvement needs a signal whose errors are independent of the policy's: a judge that solves before it reads, peers that share no weights, a hidden verifier, or execution. Which open-ended tasks can be reshaped to get one of those is the question I'd most like to see answered in the next year.