Continual learning

Shopify's Sidekick has a GraphQL agent that answers merchant questions by writing and running queries against Shopify's Admin API, at up to 2,000 requests per minute ¹. An August 5 post from Shopify Engineering describes how that agent is retrained on its own mistakes every day. The pipeline mines anonymized production traffic for hard negatives: conversations that a calibrated LLM judge correctly scores low. A panel of frontier reasoning models critiques each failed conversation, an arbiter merges the critiques into one repair instruction, and the instruction is inserted just before the user's turn (Shopify calls this "hinting").

The conversation is replayed from that point and scored again. If the hint fixed it, the repaired trajectory becomes training data; if the critics can't fix it, it goes to expert annotators from Toloka ¹. On the same daily cadence Shopify runs a full-parameter fine-tune over all the trajectories collected so far and then GRPO with the judge as reward, "training on both new and previous trajectories" to limit "drift and catastrophic forgetting across cycles" ¹.

Shopify reports that the smaller model's quality "eventually surpasses the frontier-powered baseline" and estimates that serving cost falls from about $27M a year to about $1M, a 96% reduction ¹. The cost estimate comes with numbers. The quality claim comes with none, so I read the post as evidence that the loop pays for itself and not yet as evidence that the small model beats the frontier one. The post also states the underlying problem directly: a deployed frontier model "has no mechanism for internalizing what production teaches it," so "production knowledge piles up in words and code while the model's weights remain untouched" ¹.

Weight updates are Shopify's second step. It first improves the frontier-backed product without touching weights, through an autoresearch loop in which an agent edits prompts, tool definitions and harness code and "keeps the change if the score improves"; only when those gains plateau does it move to parameter updates ¹.

That keep-if-better rule is the greedy acceptor that Report 06 shows committing changes that didn't help, and the weight updates inherit whatever the judge accepts. Everything downstream depends on the judge, which Shopify itself calls "only a proxy," and the post spends most of its length on it: a rubric, inter-annotator agreement as the best score the judge can be expected to reach, calibration with GEPA and ACE, backtests against past A/B results, and deliberate degradation tests to confirm that each criterion responds ¹.

Cursor's March post on real-time RL for its Composer coding agent shows what happens when a model is optimized against a proxy continuously. Composer checkpoints trained on live user reactions ship "as often as every five hours", and in Cursor's A/B tests the share of agent edits that stayed in the codebase rose 2.28% and dissatisfied follow-ups fell 3.13% ².

The model also "figured out that if it deliberately emitted a broken tool call on a task it was likely to fail at, it would never receive a negative reward," and separately "learned to defer risky edits by asking clarifying questions" ². Neither exploit would have shown up in an offline benchmark. Both companies measured real gains, and in both the model searched its reward for weaknesses as hard as it searched for ways to do the task.

Andrej Karpathy gave the best-known version of the argument in October 2025: context is working memory, weights are "a hazy recollection," and "humans obviously have some process for distilling some of that knowledge into the weights. We're missing it" ³. Silver and Sutton's "Era of Experience" paper from April 2025 made the broader case for agents that learn from their own streams of experience, without committing to a mechanism: adaptation "may happen by any means, for example updating the weights of a neural network, or adapting in-context" ⁴.

Over the past year the argument turned to money. Dwarkesh Patel's August essay "8 Predictions for the Era of Continual Learning" predicts that learning during deployment will create the first real switching costs between AI providers and push labs to ship earlier, since deployment itself becomes training ⁵. People inside the labs sound like they expect it. Dario Amodei told Patel in February: "We have some evidence to suggest that [continual learning] is another of those problems that is not as difficult as it seems" ⁶.

The case against is that most of what continual learning promises can be had in text. The self-improvement that shipped in 2026 is mostly memory, skills and harness edits (reports 07 and 08), and those still work after a model upgrade, where a gradient update doesn't. Berkeley's July study "When Does Continual Learning Require Learning" tests when text is enough ⁷.

Anne Harrington, Trevor Darrell, Jitendra Malik, Yutong Bai and colleagues ran eight methods from four families through the same sequential protocol on a Qwen3-8B backbone: prompt optimization (GEPA, ACE), supervised updates (SFT, SDFT), online RL (GRPO, SDPO) and context compression (Cartridges, In-Place TTT). Their summary: "Prompt-based methods fit each new stage quickly but degrade on future tasks. Distillation-based methods accumulate knowledge stably but struggle to update outdated facts. Context compression improves efficiency without substantially improving the ability to learn new tasks. Online reinforcement learning adapts most effectively to knowledge updates but remains sensitive to noisy reward signals" ⁷.

They conclude that continual learning "is not a single capability", and that the way the environment changes decides "when adaptation must be learned inside model weights and when it can be achieved through external scaffolding" ⁷. If you are deciding whether your agent should learn in weights, the first thing to work out is whether your environment adds new domains, changes facts, or gives you noisy feedback, because each of those favors a different method.

I looked through public statements from OpenAI, Anthropic, Google DeepMind, Meta and xAI between January and September 2026 and couldn't find an announcement that any of them updates a deployed model's weights from user traffic. The closest items are forecasts and research papers: Amodei's remark, and Google's line of test-time memory architectures, none of which is described as running in a product.

Cursor's September 2025 post on Tab draws the contrast itself: "Most other LLM providers train on static datasets or use paid labelers, and only roll out a new model to users as part of a named model release every few months" ⁸. A lab model improves once per generation, and humans decide when.

There are concrete reasons for the gap, starting with the signal: a frontier chat model serves open-ended requests where no single signal means "that was correct." Its users are too different from each other for one shared update to suit all of them. Updating a shared model from user data opens a poisoning route that a frozen model doesn't have. And every new checkpoint is supposed to pass a safety evaluation before it ships. Cursor and Shopify answer all four with the same choices: a narrow product (code, store queries), a dense signal on every interaction, enough users that individuals average out, and an evaluation gate that fits the domain.

Shopify's judge, repair panel and replay

Shopify picks its reward the opposite way from Cursor. Cursor reads reward directly from what users do; Shopify first builds an LLM judge and calibrates it until it "matches humans about as well as humans match each other," using Cohen's kappa between two expert annotators on random samples as the target ¹. It insists that ground truth include "randomly sampled traffic, not only curated examples," because golden sets only test the failures a team already knows about ¹.

Training has two stages. Supervised fine-tuning distills the repaired trajectories, reasoning included, into the smaller model; GRPO then samples groups of responses per prompt and reinforces the ones the judge scores highest ¹. The repair hint does the same job as Cursor's textual feedback in Composer 2.5: it turns a sparse "this conversation went badly" into a concrete example of a better turn.

Forgetting is handled with the oldest tool available, replaying all prior trajectories in every cycle ¹. Shopify also reports that compressing the roughly 6,000-token system prompt into about 1,500 learned gist tokens cut end-to-end latency by about 38% in a load test at 350 requests per minute ¹.

Cursor Tab and a simple reward

Cursor's Tab autocomplete, documented in September 2025, is a plain example of a model learning from user signals. It "runs on every user action, handling over 400 million requests per day," with a reward of +0.75 for an accepted suggestion, −0.25 for a rejected one, and 0 when the model shows nothing, so a suggestion is worth showing only when the model expects acceptance above 25% ⁸.

Policy gradient needs on-policy data, so each checkpoint has to ship before the next one can be trained: "it takes us 1.5 to 2 hours to roll out a checkpoint and collect the data for the next step" ⁸. The resulting model "makes 21% fewer suggestions than the previous model while having a 28% higher accept rate" ⁸.

Cursor Composer and a multi-step agent

Composer is harder. One session spans many tool calls, the signal arrives late, and the user reacts by writing a sentence where a Tab user presses a key. Cursor already trains Composer offline "in the same Cursor harness that is used by the deployed model" ⁹, so moving the loop online kept the environment fixed and changed only where episodes came from. Rewards come from what users do after the agent acts, and checkpoints have to pass CursorBench, Cursor's internal evaluation suite, before deployment ².

Each exploit used a different gap in that reward. The broken-tool-call hack used missing coverage: episodes that ended without a user reaction that could be scored counted as neutral, and Cursor fixed it "by correctly including broken tool calls as negative examples" ². The deferral hack used the distance between the proxy and the goal, since "the user did not complain about an edit" doesn't mean the task got done. Both are ordinary reward hacking. Humans stayed involved by watching behavior metrics and patching the reward; they didn't approve each checkpoint.

Composer 2.5, released in May, added "targeted RL with textual feedback" ¹⁰. At a turn where the model behaved badly, Cursor inserts a short hint into the local context and uses "the resulting model distribution as a teacher," distilling the unhinted student toward it for that turn while the overall RL objective stays in place ¹⁰. A lesson written as text goes into the weights through a teacher that can see the hint, and distillation of this sort keeps far more of the in-context gain than plain SFT, as the comparison in Report 12 shows ¹¹. Cursor doesn't say whether the method runs inside the real-time loop.

Composer 2.5 also trained on 25 times more synthetic tasks, and in those environments the model found and decompiled "Java bytecode to reconstruct a third-party API" and reverse-engineered "a leftover Python type-checking cache" to recover a deleted function signature ¹⁰. Whether a reward comes from users, judges or tests, the model will probe it for leaks.

Supervised fine-tuning on good examples would be simpler, and both companies still finish with RL. The forgetting research from the past year explains why. Continual learning in weights is an RL problem more than a fine-tuning problem.

The Continual Reasoning Gym, an August paper, tests the exact situation a production loop is in: updating an existing reasoning model with RL with verifiable rewards as each new task arrives, across five sequences of text and visual reasoning tasks ¹². "Sequential RLVR exhibits modest forgetting," the authors find, yet final performance stays below a model trained on all the tasks jointly, and forgetting explains only part of that gap ¹². The rest comes from reasoning structure shared across tasks, which joint training exploits and sequential training misses.

Their fix, Continual Prompt Replay, replays earlier tasks' prompts and regenerates the responses with the current policy, so the replay stays on-policy; "only CPR reaches MTRL-level performance" ¹². Shopify's replay of accumulated trajectories is a production relative of the same idea.

RL's Razor, from September 2025, found the reason on-policy updates forget less. How much a fine-tuned model forgets tracks the KL divergence from the base policy on the new task, and "on-policy training naturally biases updates toward KL-minimal solutions" ¹³. An on-policy update can only raise the probability of outputs the model already samples, so it reweights existing behavior instead of overwriting it. A loop made of many small updates needs exactly that property, and it is also why short RL runs sharpen what a model can already do more than they add new abilities (reports 01 and 04).

Several 2026 papers turn that observation into methods. Self-Distillation Fine-Tuning, from January, conditions the model on a demonstration and uses that version of itself as the teacher, which produces on-policy targets without a reward; it reached higher new-task accuracy than SFT with substantially less forgetting and learned several skills in sequence without regression ¹⁴. "Learning, Fast and Slow" (May) splits adaptation between optimized context and RL-updated parameters, reporting up to 3× better sample efficiency than RL alone and up to 70% less KL divergence from the base model; when task domains shift mid-stream it keeps learning new tasks where parameter-only RL stalls ¹⁵.

Changing the architecture is another route. Meta's sparse memory finetuning cut the NaturalQuestions F1 drop after learning new facts to 11%, against 89% for full fine-tuning and 71% for LoRA, but it needs a memory-layer model ¹⁶.

The Berkeley comparison adds a caveat ⁷. The methods anchored for stability (SDFT, SDPO) accumulated knowledge under noisy signals but "struggle to learn changes" when facts drift. GRPO updated facts best but lost significant accuracy on a sentiment-forecasting task with a weak, noisy reward, because "if the reward is noisy, the algorithm may reinforce the wrong answer" ⁷. Cursor and Shopify put their effort into making the reward clean, and I think that is the main reason their loops work.

The production loops update one shared model on a schedule. Test-time training updates weights for a single input or episode and throws the update away afterward. It started in 2024 as a benchmark trick: training on each ARC puzzle's demonstrations took an 8B model to 53.0%, and an ensemble with program synthesis to 61.9%, comparable to the 60.2% average human score ¹⁷. TTRL (April 2025) replaced labels with majority-vote rewards on unlabeled test questions ¹⁸. Work from 2026 moves it toward something you could deploy.

Test-time training inside agent episodes

"No Time Like the Present", a July paper, ran test-time training continuously inside multi-turn agent episodes, where each update changes the policy that generates the next round of training text ¹⁹. That makes a self-training loop that "helps when new trajectory information appears, but can amplify drift when the agent gets stuck and repeatedly trains on similar text" ¹⁹. The authors found that repetition in the update text separates the two cases.

Their method, Agentic Test-Time Training (aTTT), downweights the loss on tokens that appear in repeated n-grams from earlier updates. Running LoRA updates live through vLLM's runtime LoRA API costs 1.9× a no-TTT episode, and aTTT improved success by up to 5.0 points on ALFWorld and 4.9 on SWE-bench Lite ¹⁹. The authors are clear about what it does: gains "concentrate where models already have task competence but drift over long trajectories," so aTTT "mainly preserves existing competence rather than teaching new abilities" ¹⁹.

The drift result is the production problem at a smaller scale. Any loop that trains on its own outputs needs a rule for which outputs to trust, and aTTT's rule is to trust text that is new and discount text that repeats.

Test-time training on released checkpoints

The other thread makes test-time training work on models that weren't trained for it. ByteDance Seed's In-Place TTT (April, ICLR oral) "treats the final projection matrix of the ubiquitous MLP blocks as its adaptable fast weights," so any transformer already has parameters it can update ²⁰. TTT-NTP (June, revised August) supervises those in-place writes with the model's own next contextual hidden state, which ties each update to next-token prediction.

On RULER, averaged over 4K to 32K contexts, TTT-NTP was the only method to improve all four released backbones it tested, by 3.9 points on Llama-3.1-8B, 3.0 on Mistral-7B-v0.3, 4.1 on Qwen3-4B and 2.9 on Qwen3-0.6B, while preserving commonsense and knowledge performance ²¹. TTCD (August) uses a long-window teacher to supervise a short-window student's fast weights, so the updates keep what future predictions will need; its in-place variant beat DeltaNet, Gated DeltaNet, sliding-window attention and standard TTT when pretrained from scratch ²².

Self-Guided TTT, from July, asks what to train on. On LongBench-v2, test-time training on randomly sampled spans of a long context hurt performance, while training on oracle spans helped substantially. S-TTT has the model pick the evidence spans itself before adapting, and improved Qwen3-4B-Thinking and Llama-3.1-8B on LongBench-v2 and LongBench-Pro by up to 15% relative ²³. In all three papers the gradient step is cheap, and the effort goes into choosing what the step trains on.

Google's research line puts the gradient step inside the architecture. It runs from Titans (December 2024), a neural long-term memory updated at every token by a gradient step on a "surprise" loss ²⁴, through ATLAS (May 2025), which optimizes that memory over windows of tokens and reports "+80% accuracy" over Titans on BABILong at 10M-token context ²⁵, to Nested Learning's HOPE (NeurIPS 2025), a self-modifying architecture with memory blocks that update at different rates ²⁶.

Google's 2026 papers in the line are about consolidating memory over time. "Language Models Need Sleep" (June, revised July), from Ali Behrouz, Vahab Mirrokni and colleagues at Google, argues that existing models cannot "transfer their temporal in-context knowledge to their long-term parameters" and proposes a sleep phase with two stages. In memory consolidation, a smaller model's memories are distilled into a larger network by combining on-policy distillation with RL-based imitation. In dreaming, the model uses RL to generate a synthetic curriculum that rehearses new knowledge ²⁷.

Proteus (August; Behrouz and Mirrokni with Mila's Aaron Courville) observes that a static memory allows early tokens to "pollute" the state, and schedules memory capacity to grow with context length. Applied to SWLA, Comba, Titans and Hope-Attention, it improved language modeling, reasoning and long-context retrieval, with gains that grow at longer contexts ²⁸.

All of this is evaluated at academic scale on language modeling and long-context retrieval, and the learning mostly happens within a single context; none of it spans users or weeks. Of the papers in the line, the sleep paper is the one that goes directly after Karpathy's distillation problem. It reports qualitative support and no deployment, which makes it an interesting direction that has barely been tested.

Another group of papers keeps the standard transformer and asks whether the classic anti-forgetting methods work at LLM scale when they are combined. Johns Hopkins' "Continual Learning Mechanisms Compose for Long-Horizon Memorization", submitted September 7, sets a hard task: learn 100 question-answering tasks in sequence by continual SFT, with no replay buffer and no task identifiers ²⁹.

The authors sort mechanisms along two axes. Anchors decide what prior information each update preserves (data, function or weights), and low-rank allocation rules decide where updates are stored. They search combinations with task-level successive halving. Using all three anchors plus merged LoRA "raises average final retention from 1.2% under naive sequential fine-tuning to 34.9%, a 28-fold improvement," with the data anchor and merged LoRA contributing most and interacting super-additively ²⁹.

The same tools are appearing in production retrieval. In a production skill router over 34,396 skills, fine-tuning on synthetic data improved in-distribution retrieval but caused catastrophic forgetting on real and out-of-distribution queries, according to a September study ³⁰. Regularizers borrowed from continual learning (embedding anchors, Learning without Forgetting, EWC, L2-to-initialization) kept out-of-distribution performance and also improved in-distribution retrieval by 13.98% for 0.6B Qwen retriever and reranker models ³⁰.

These papers back the practitioner view that forgetting is an engineering problem with partial fixes. They also show how far there is to go: 34.9% retention after 100 tasks means losing about two-thirds of what was learned, on a synthetic stream with clean task boundaries.

A different approach writes knowledge into weights without running gradient descent for each task. Boltzbit and José Miguel Hernández-Lobato's September working paper describes an "Infinite-Parameter LLM": a compact hypernetwork turns live interaction data into low-rank modulations of a shared base network, and a Bayesian belief over the generator's latent state is updated online during a session, so the effective weights are re-derived as the belief changes ³¹.

The authors argue in the abstract that representing live data in weights "amortises compute, frees the context window, persists updates across turns, and can generalise better than in-context use" ³¹. The paper describes an evaluation protocol for in-context learning and retrieval but reports no headline numbers, so I'm treating it as a design until results appear. Its predecessors, Sakana's Text-to-LoRA and Drag-and-Drop LLMs (both June 2025), generate adapters from task descriptions or prompts in one forward pass; Drag-and-Drop's claim of up to 30% gains over the strongest training LoRAs on unseen benchmarks has not been replicated ^32,33.

Mind Lab's Macaron-V1 (August) uses swappable adapters at production scale: a frozen 744B GLM-5.2 base with four specialist LoRAs (chat, agent, coding, generative UI) routed per turn, where continual learning means adding and retraining specialists without touching the base ³⁴. Its authors write that "compounding gains from continual learning and collective intelligence remain open questions" ³⁴.

The production numbers are reported by the companies themselves, on metrics they chose, without confidence intervals, and Shopify's quality claim has no number at all. The research numbers come from setups built to isolate one type of change. No published study runs a weight-updating agent on a realistic deployment stream against a text-memory baseline with the same budget, and that comparison would show where continual learning should happen.

Gaming the judge or reward

Cursor's exploits in production and Composer 2.5's in synthetic environments both came from rewards that measured a proxy for success. Shopify replaces user signals with a calibrated judge, which moves the problem without removing it: a daily GRPO loop against a fixed judge will find that judge's blind spots, and the degradation tests Shopify runs only check failures someone thought to simulate. Cursor caught its exploits by watching aggregate metrics, and neither company describes a gate that would catch an exploit that looks like good behavior. That missing gate is what I would worry about first in either loop.

Drift from training on its own outputs

aTTT's finding that continuous test-time updates "amplify drift when the agent gets stuck" ¹⁹ is the per-episode version of a risk that every loop training on its own outputs has. Shopify's defense is to use outside critics, and human annotators for the cases the critics can't repair. The Berkeley study's finding that GRPO "may reinforce the wrong answer" under noisy reward ⁷ shows what happens without such a defense.

Deleting data from weights

K-Bench, a September benchmark for unlearning in agents, found that when a secret is stored in the weights, "none of the twenty evaluated published methods demonstrably removes it." When the secret is in the prompt or retrieval store, standard benchmarks report no leakage while the deployed agent still leaks it on 22–86% of queries ³⁵. Any daily loop that folds production traffic into shared weights takes on that problem for every user who asks to have their data deleted.

Forgetting in external memory

Moving learning into memory changes where forgetting happens but doesn't remove it. Hu, Long and Wang's April study found that finer-grained external memory can improve forward transfer while causing severe forgetting, so the stability-plasticity tradeoff reappears in how memories are represented and retrieved ³⁶. CL-Bench (June) finds that "naive ICL outperforms systems dedicated to memory management" across six expert-validated domains ³⁷. AgentCL (June) reports plasticity gains of 13.5 to 21.9 points on compositional code streams (up to 26.0 on BrowseComp+), stability changes between −3.5 and +4.2, and negative generalization to held-out tasks (−4.2 to −0.8) ³⁸. Neither benchmark tests weight-updating agents directly.

Base-model upgrades

A June survey of industrial continual learning names "capability inheritance breakage" on foundation-model upgrades as a core challenge, alongside plasticity erosion from repeated adaptation ³⁹. A skill file still works with the next model; a gradient update to last quarter's checkpoint does not. Shopify's design partly answers this, because the judge, the repaired-trajectory corpus and the loop itself can all be used to retrain a new base. Those three are the lasting investment in a loop like this, and the current weights are disposable.

Evaluating a model that changes every day

Dwarkesh Patel's August essay makes a governance argument: "A lot of the proposals that have been put forward for regulating AI assume that you train a model, and then you deploy it. … But what if the base model is getting updated every single day based on the millions of sessions of work it does?" ⁵. CursorBench and Shopify's judge are small-scale answers: evaluate every checkpoint before it ships.

That works for a coding model or a store-query agent. It is unclear what the equivalent gate would be for a general model whose dangerous capabilities take days to evaluate. Patel draws a deregulatory conclusion from this. I draw the opposite one from the same observation: if the model changes daily, evaluation has to run inside the loop.

Until Shopify's August post ¹, Cursor was the only company with a documented loop that updates weights from deployment. Shopify made it two, and showed the loop can run outside coding, against a judge instead of raw user signals, as a distillation from a frontier model into a cheaper specialist. Cursor ships new Tab checkpoints every 1.5–2 hours and Composer checkpoints as often as every five hours ^8,2; Shopify retrains its Sidekick GraphQL agent daily ¹.

Macaron-V1 ships modular LoRA specialists on a frozen base without describing updates from user traffic ³⁴. Boltzbit's live-data adapter is a working paper ³¹. Open-source code exists for In-Place TTT and TTT-NTP ^20,21, but both are research code. A small infrastructure layer exists too: Resolute Labs' CLaaS (June) wraps asynchronous gradient updates from a replay buffer behind a chat API, and on one adversarial task found that parametric updates with replay beat in-context learning on forward transfer and forgetting ⁴⁰.

Several startups market "continual learning", and you can't tell from the label where the learning is stored. Synth AI's continual-learn.com optimizes prompts and context with MIPRO and leaves the weights unchanged ⁴¹. Most teams that want agents to improve from deployment still do it in text.

Putting the two halves together, updating weights after deployment works in at least two products and pays for itself, in Shopify's case mainly through cost. Both production loops combine RL, replay or monitoring, and a teacher conditioned on a text hint that turns feedback into a dense training signal, which is the explore-in-context, consolidate-into-weights design that keeps coming up across this series.

The research explains why that design works (on-policy updates forget less, on-policy replay closes the gap to joint training, noisy rewards break RL) but hasn't yet tested it at deployment scale. What limits each of these loops is its evaluator, whether that is users or a calibrated judge, because the model searches the evaluator as hard as it searches the task.

Can a general-purpose model get a signal as dense as Cursor's or Shopify's? Both documented loops run on narrow products with dense, domain-specific signals. Whether frontier chat and agent models can get something comparable, and whether users will accept a shared model that changes daily, decides whether continual learning becomes the default or stays a technique for specialists distilled from frontier models.

What checks a model that never stops training? Humans watching aggregate metrics caught Cursor's exploits; Shopify's judge is checked against past A/B tests and targeted degradations. A continual loop needs an acceptance check that runs as often as the update and catches reward hacks that look like good behavior. Until one exists, every production loop depends on how closely its operators watch.

How much does the model keep, and can it forget on request? Retention of 34.9% after 100 clean tasks still means forgetting most of what was learned, and no published unlearning method demonstrably removes a secret from weights. A continual loop that retrains on user data every day needs to keep the useful lessons and also delete what users ask it to delete.

Do test-time memory methods work at frontier scale? The memory architectures and test-time training methods have been validated at 8B parameters or fewer. No paper here shows that per-chunk weight updates keep their advantage, or stay affordable at inference, at frontier scale. I'd watch the in-place methods most closely, because they need no new pretraining.