Self-training and distillation

Give a model a math problem and let it reason. Give a second copy of the same model the problem plus the worked solution, and have it score, token by token, how likely it would have been to write what the first copy wrote. Train the first copy to close the gap. The setup is on-policy self-distillation with privileged context (OPSD), and it looks like a free lunch: no reward model, no external teacher, a dense signal at every token, and a grader that cannot be wrong about the final answer because it was handed the final answer ¹.

On July 6, 2026, Simran Kaur, Liam Fowl, Sanjeev Arora and colleagues at Princeton reported what happens when the student is a thinking model ². Across five Qwen3 and OLMo thinking models evaluated on AIME24, AIME25, and HMMT25, privileged-context distillation caused a relative drop of up to 17% in avg@16 accuracy. The damage grew with the amount of privileged context withheld from the student and was worst at long rollout budgets, exactly where thinking models otherwise gain most. The trained models produced fewer verification, backtracking, and hedging markers, even after length normalization. The diagnosis was local: privileged context lowers "fork rates", the frequency of high-entropy positions where several lines of reasoning remain open, and when the student begins a self-correction branch, "privileged OPD penalizes sampled reconsideration tokens that vanilla OPD supports" ². Distillation from the same teacher without the solution still improved these models, and the privileged variant still helped instruction-tuned models that do little search ².

The incident compresses the subfield into one failure. Self-training is a loop in which a model learns from its own outputs, filtered or graded by something, and it learns whatever that grader rewards. A grader with the answer in hand rewards confidence rather than the search that produced correct answers. Over the next eleven weeks, at least five groups took OPSD apart and reached a stronger conclusion: much of what the privileged teacher transfers is its behavior, not its information ^3,4,5,6,7.

The loop is old. STaR (2022) generated rationales, kept those that reached the correct answer, and fine-tuned on them; when the model failed, it was shown the answer and asked to rationalize backward, the first appearance of answer-conditioned supervision ⁸. ReST-EM (2023) framed the same filter-and-fine-tune loop as expectation-maximization with binary correctness feedback and beat fine-tuning on human solutions on MATH and APPS, then plateaued after two or three rounds, a shape that recurs across its successors ⁹. Agent pipelines replaced answer keys with unit tests; SWE-smith (April 2025) trained a 32B model to 40.2% on SWE-bench Verified from 50,000 synthesized tasks ¹⁰.

On-policy distillation (OPD) changed the grader's resolution. The student samples its own rollout, a teacher computes log-probabilities on every token, and the student minimizes per-token reverse KL to the teacher: RL's on-policy sampling with distillation's dense signal ¹¹. Thinking Machines' October 2025 write-up took a Qwen3-8B-Base student from 60% to 70% on AIME'24 in about 150 steps, 9 to 30 times cheaper than extrapolated SFT, and quoted the Qwen3 report's comparison of 74.4% for 1,800 GPU hours of OPD against 67.6% for 17,920 GPU hours of RL ^11,12. Because the student stays on its own distribution, the blog noted, OPD does not regress "in the self-distillation setting as SFT does" ¹¹. An earlier checkpoint could be the teacher of a later one, and in January 2026 the Self-Distilled Reasoner made the current model its own teacher by putting the gold solution in the teacher's context ¹.

The teacher is you, with a hint

By mid-2026 OPSD was a field of its own. A survey of recursive self-improvement that coded 1,250 papers (July 2026) counts it as a distinct thread of 56 papers that did not exist 18 months earlier ¹³. The variants differ mainly in what the teacher is shown. SDFT (January 2026) shows demonstrations and reports less forgetting than SFT ¹⁴; Skill-SD (April 2026) shows natural-language skills summarized from completed trajectories and reports gains of 14.0% and 10.9% over GRPO on AppWorld and Sokoban ¹⁵; SDAR (May 2026) gates the distillation loss where imperfect skill retrieval makes the teacher unreliable, improving over GRPO by 9.4% on ALFWorld and 10.2% on WebShop ¹⁶.

The August wave moved the method into multi-turn agents and ran into a problem math never posed: the student's own actions change the state. SMRC-SD (August 5, 2026) observed that a student acting differently from a reference trajectory reaches states the reference never covered, so the reference stops describing the situation the student is in and becomes "an unreliable source of guidance" ¹⁷. It distills only at turns where the student's state matches a state on the successful trajectory, and builds the teacher's context from that matched state. With Qwen3-1.7B, task success rose from 0.746 to 0.865 on ALFWorld and from 0.574 to 0.693 on WebShop against unconditional full-trajectory distillation ¹⁷. AgentOPSD (August 6) aggregates token-level teacher–student log-probability gaps into turn-level evidence and updates a Bayesian belief over which turns were pivotal, reaching 89.1% success on ALFWorld with Qwen2.5-7B, ahead of GRPO and other self-distillation baselines ¹⁸. LOPD (August) makes the privileged context a learned latent and reports beating RLVR and Skill-SD with under 30% of their rollout budget ¹⁹. On September 23, GUI-SD-v2 extended OPSD from GUI grounding to multi-turn phone-agent interaction, first training the model to follow privileged guidance, then selectively distilling step-specific reasoning and memory hints; it reports higher Pass@1 and Pass@3 than existing OPSD baselines on AndroidWorld and MobileWorld ²⁰.

Removing the teacher's information

A second strand asked how little the teacher needs. U-OPSD (August 2026) replaces the gold solution with a majority-vote pseudo-solution from the model's own rollouts and distills on sampled completions whose answers disagreed with that pseudo-solution; averaged over five math benchmarks, it improved Qwen3 non-thinking models by 8.5% at 4B and 10.7% at 8B over the base and beat ground-truth OPSD by 3.2% and 2.3%, while in thinking mode it only matched OPSD ²¹.

The same question, how little outside supervision a teacher needs, also appears inside RL. RISE (Salesforce, September 2026) builds the teacher from the model's own RLVR trajectory by extrapolating the displacement between the current checkpoint and a trailing anchor, turning a sparse outcome update into a dense token target, and reports gains over RLVR alone and over OPSD across math, code, and agent tasks ²².

The most radical result removes the teacher outright. On August 31, 2026, Yi Ding and Ruqi Zhang measured the supervision a teacher provides during OPD and found substantial noise that grows with teacher scale, to which the student was insensitive: it converged to the same performance whether the noisy supervision was kept or removed ²³. Learning concentrated on low-probability tokens, and "using a single fixed negative advantage matches the performance of teacher-provided ones", which led the authors to conclude that "OPD works largely by suppressing low log-probability tokens, which requires no teacher" ²³. Their teacher-free replacement, OPSA, raised Qwen3-1.7B's Avg@32 on AIME24 by 35.41 points over the base model and by 16.77 points over OPD ²³. A number that large on a 1.7B model needs replication, but the mechanism it points to is precise, and it is sharpening in its plainest form.

Data turned out to matter as little as the teacher. Training OPD on a single query recovers most of full-data OPD's gain; one query's rollouts reach 71.5% of the states full-data training visits, and 16 semantically distinct queries reach 98.9% and match full-data training (September 3) ²⁴. The authors call OPD "data-overfed but algorithm-starved". Data-free OPD (September 12) found that eight prompts match a 17,000-problem dataset, that swapping mathematics for competitive programming "still recovers over ninety percent of the in-domain gain", and that a teacher writing its own training questions matches or beats real data ²⁵. The paper's reading is that "OPD transfers the teacher's mode of reasoning rather than knowledge related to the data" ²⁵.

Self-training on verified agent trajectories

The filter-and-fine-tune loop did not disappear; it moved inside agent RL pipelines. Socratic-SWE (June 2026) distills its own solving traces into skills describing recurring failures, uses them to generate targeted repair tasks in real repositories, validates the tasks by execution, and trains on successful repairs; after three rounds it reached 50.40% on SWE-bench Verified, ahead of other self-evolving baselines under the same compute ²⁶. A September 20 paper on software-engineering agents found that pooled RL across task categories produces a "category see-saw", gains in some categories and regressions in others ²⁷. Its fix trains category experts that refresh which instances they have mastered, reuse "its own verified successful trajectories for Repair SFT", and reselect tasks for more RL, then consolidates the experts into one student with multi-teacher OPD. No external model supplies solutions; the final policy resolves 58.04% on the Pro-618 split and 59.00% on SWE-bench Multilingual, 5.39 and 2.78 points over its base ²⁷. In both systems the acceptor is execution: training data counts only if the tests pass.

If the teacher's noise can be dropped and its data can be eight prompts, what does the privileged context add? Five papers between August 10 and September 22, 2026 tested that directly, and they converge.

Ichihara and colleagues (August 10) replaced the paired reference solution with a problem and solution from a different example, keeping everything else fixed. Across three models and three math benchmarks, this "OP²SD" improved over the base model and stayed competitive with OPSD, implying "that OPSD gains do not necessarily come from access to the reference solution, and that the teacher's context-induced behavior is an important factor" ³. Shrestha and Tessier (August 18) found with Qwen3 models from 1.7B to 8B that the correct reference "does not provide a consistent performance benefit", that a solution from another problem can outperform the correct one on several benchmarks, and that the student's predictions align more with the base model's thinking behavior than with anything the reference supplies ⁴. Zhang and colleagues (September 17) built AMPLE-Math, 5,319 problems with six reasoning views sharing each answer, and compared each view with matched reference-free distillation: reference-free distillation accounted for much of Qwen3-1.7B's improvement, complete traces added about two points for SmolLM3-3B at step 50, and switching the student from short direct-response rollouts to long thinking rollouts turned gains into losses in both model families ⁵. Their conclusion is that a reference's value "is what it adds to this cross-mode transfer, not how much of the solution it reveals" ⁵.

Two papers measured the behavior itself. Baumann, Hübotter, Krause and colleagues (September 18) contrasted "attractive" self-distillation toward a privileged teacher with "repulsive" distillation away from one: attraction "suppresses exploratory reasoning and promotes shorter, more confident responses", repulsion lengthens responses and destabilizes, and combining attraction to a correct-solution teacher with repulsion from an incorrect-solution teacher cancels the shared behavioral shift, leaving a signal that tracks correctness with stable response lengths ⁶. That is the Princeton backfire's mechanism, isolated and reversed. Tian and colleagues (September 22) varied what a self-teacher sees and found that abstractions (a named strategy, a framing, a problem category) beat the full solution by 1.4 points at 4B and 1.6 at 8B while storing an order of magnitude fewer hint tokens; answer-only conditioning stayed within 0.2 points of the full solution, and initial teacher–student KL did not predict downstream performance ⁷. "What a self-teacher should see," they conclude, "is therefore not everything it could, but the level of abstraction its student can still act on" ⁷.

The picture is consistent across groups. The privileged context changes how the teacher behaves (more confident, shorter, in a different reasoning mode), and the student mostly learns that behavior. When the behavior is helpful, as with a direct-response model borrowing a thinking model's habits, OPSD works even with the wrong reference. When the behavior is "skip the check", as with a thinking model on long rollouts, it degrades the student regardless of how correct the reference is. An August 26 review of the literature names the resulting failure "collapse, the progressive narrowing of the set of reasoning paths the model can produce", says privileged information aggravates it, and organizes the fixes around three levers: which tokens get weight, what the teacher is shown, and how the guidance changes over training ²⁸.

The margins are real and conditional. Most 2026 rows beat GRPO or vanilla OPD on specific benchmarks in a single group's setup, and none has an independent replication yet. The rows that carry the most information are the ablations: the ones showing that the teacher's information, its noise, and its data can each be removed with little loss, because they say what the loop runs on.

A grader that knows the answer suppresses search

The mechanism behind the July backfire applies well beyond math ². Reverse-KL distillation pulls the student toward the teacher at every token the student generates. At a forking position, where a thinking model might continue or stop to verify, the unconditioned model spreads probability across both. A teacher conditioned on the answer has no reason to verify, so it puts low probability on the checking branch, and the student is penalized each time it forks. At test time the answer is gone and the confidence remains. The attractive-teacher result shows the same shift without thinking models in the picture: moving toward a privileged teacher makes responses shorter and more confident as a behavior in its own right ⁶.

Agent settings add a second failure. Whatever the teacher knows that the student will not know at inference, the student learns to act as if it knew, and in multi-turn tasks the teacher's knowledge can also be about the wrong state; SMRC-SD's gains came from refusing to distill where the reference no longer described the student's situation ¹⁷. Earlier in 2026, Li et al. found that OPD transfers little from a teacher whose thinking patterns the student cannot reach: where OPD worked, alignment concentrated on a shared set of high-probability tokens carrying 97% to 99% of the probability mass at student-visited states ³⁰. Teachers should be chosen by distributional closeness, not leaderboard rank. Mundane failures surface too: a September 17 study traced runaway response length in OPD to base students and post-trained teachers placing their stopping probability on different end-of-sequence tokens, even with identical declared stop sets ³¹.

What theory says the loop can and cannot do

The 2026 results fit a theory published in December 2024. Huang et al. showed that self-improvement cannot "create information that is not already in the model"; it sharpens, moving probability onto good sequences the model already generates. SFT on filtered samples is minimax optimal when the model already covers good answers, and an RL-style approach can improve on it by exploring online, bypassing the coverage requirement ³². The accurate reading is narrower than "self-training cannot create new capability": filter-and-fine-tune loops are capped by what the model already samples, and loops that explore can pass that cap, though not the information available to their verifier. A related result found that the generation–verification gap that bounds self-improvement grows with pretraining compute ³³.

September 2026 supplied the empirical version. "Sequential Beats Joint" (September 3) found that OPD followed by RL beats pure OPD, pure RLVR, and every tested way of fusing the two signals in one update, and explained it through pass@k and parameter updates: "OPD expands the student's coverage of teacher-supported solutions and RL sharpens within that support", while optimizing both at once makes them interfere ³⁴. OPD also made a better RL cold start than SFT, and its validation score was the signal for when to switch ³⁴. OPSA's finding that OPD's gains come from suppressing low-probability tokens is sharpening described at the token level ²³. The loop's ceiling is set by what the teacher's support and the verifier can see; OPD's sampling decides how efficiently the student reaches that support, then RL sharpens inside what the verifier can identify.

Collapse and the search alternative

The collapse the August review describes is a within-model narrowing of reasoning paths, not the classic model collapse from training generations of models on each other's outputs, which accumulating real data prevents ^28,35. Both share a cause: a signal that rewards the model for being more like its most confident self. Weight-space search does not escape the rule. Evolution strategies now match GRPO's accuracy with larger, off-task weight changes (COLM 2026) and, per an August 2026 study, avoid GRPO's entropy collapse and reach higher Pass@K ^36,37. They replace the proposer, not the acceptor: the fitness function plays the filter's role and has its blind spots.

Three things changed between June and September 2026.

The field's understanding of OPSD inverted. In the first half of the year, privileged context was treated as information the teacher passed down, and variants competed on richer context. By late September, five independent ablations had shown that wrong references, abstractions, or answer-only context often work as well, and that the damage and the benefit both come through behavior ^3,4,5,6,7. The practical rules that follow: keep answer-level context away from thinking models on long rollouts, distill only where the teacher's context describes the student's state, and prefer abstractions the student can act on.

OPD's role shifted from replacement to preparation. "RL Starts Before RL" (September 23) found that OPD-initialized students reach higher performance after RL than direct RL or SFT-then-RL, even "when OPD produces little immediate improvement in accuracy": on the text setting, OPD then RL averaged 59.18 across eight benchmarks, against 47.48 for SFT then RL and 51.90 for RL from the base, winning on all eight ²⁹. Together with "Sequential Beats Joint", two groups in one month converged on OPD first, RL second ^29,34. Both are preprints; if they hold, pre-RL accuracy is the wrong acceptor for the self-training stage.

The context-to-weights question sharpened, and on-policy sampling did not always win it. Experience Distillation (July 2026) distilled an agent's accumulated experience into the same checkpoint without that context: teacher-sampled forward-KL distillation kept 64.8% of the in-context gain on 749 SWE tasks, plain SFT kept 3.8%, and on-policy reverse KL at matched compute kept 9.1% and 0.4% on two text games, because the student without the experience rarely samples the trajectories where it helps ³⁸. Report 12 (Consolidation and co-evolution) owns that question. The relevant point here is the coverage condition again: the student's own rollouts are the right training distribution only when the improvement is reachable from them.

On-policy distillation now appears in shipped post-training, not just in papers. SEA-LION v4.8 (September 2026), built on NVIDIA Nemotron 3, was post-trained "with supervised fine-tuning and online on-policy distillation", raising its 120B model's SEA-HELM score from 49.30 to 63.44 ³⁹. The KwaiMind image editor (September 2026) trains specialized policies against separate rewards, "which are consolidated through on-policy distillation" ⁴⁰. The Data-free OPD authors describe OPD as "a standard component of frontier post-training pipelines" ²⁵.

Self-training on deployment traffic remains rare and early. NeoHorse-1 (September 8) records, for each user turn, the predicted capability demand, the model tier its router chose, and the interaction that followed, converts validated records into training examples, and adds routing-guided OPD; post-training lifted an eleven-benchmark macro-average from 58.94 to 64.87 at 4B ⁴¹. The closest production analogue to self-distillation remains Cursor's Composer 2.5 (May 2026), which inserts feedback at the point in a trajectory where the model could have done better and uses "the resulting model distribution as a teacher" ⁴²; report 05 covers Cursor's loop. Weight-level self-training after deployment, driven by an agent's own traces, is still the exception, consistent with the series' finding that shipped self-improvement mostly lives in text.

What the teacher should see. The August–September ablations show privileged context works mostly through induced behavior, and that the best context depends on student scale and task ⁷. Nobody can yet predict, before training, which context will teach a useful behavior and which will teach overconfidence; initial teacher–student KL does not ⁷. Nearly every self-distillation method depends on that choice.

When to stop distilling and start RL. If OPD's value is the coverage it prepares for RL, the self-training stage needs an acceptor that measures RL-readiness rather than accuracy ^29,34. "Sequential Beats Joint" suggests OPD validation score; a general answer would change how labs schedule post-training.

Self-training where no filter exists. Every strong result above has an answer key, a test suite, a deterministic environment, or a stronger checkpoint. Open-ended agent work has none, and the substitutes (majority votes, learned judges, self-written skills) inherit their grader's errors. Report 03's verifier problem is this report's ceiling.

Repeated self-writes into weights. SEAL (2025) showed a model can write its own fine-tuning data but forgot earlier tasks as edits accumulated ⁴³; on-policy distillation reduces forgetting ^11,14. Nobody has shown a self-directed loop writing into weights hundreds of times without drift, which is the gap between generational self-training in a lab and a model that improves from its own deployment.

A model learning from its own traces improves as far as its grader can see: to the edge of its coverage when the grader is a filter, somewhat further when the loop explores, and in the wrong direction when the grader's confidence comes from knowledge the student will not have. The 2026 work adds a harder lesson. The dense teacher transfers what it does more than what it knows, so the question for any self-distillation loop is what behavior the teacher's context induces, not how much the context contains.