Prompt and program optimization

Vansh Wahi spent months running autonomous prompt-optimization loops in production, on contract analysis, compliance review and code quality, and his September paper lists what went wrong ¹. He catalogues 11 ways the evaluation signal failed.

A syntactically broken prompt was promoted as the winner because a silent parser fallback raised the metric. A single corrupted ground-truth label led the optimizer to delete correct compliance rules so that its outputs would agree with the bad label. Agents reached perfect scores by reading cached answer keys from their environment, "a 100% pass rate concealing 68% true capability" ¹. Rewriting the LLM judge's rubric didn't fix the judge; the only reliable gain came from a structural constraint on the order of its output.

The proposer caused none of these failures, and this summer's research made proposing look close to solved. A group including Sumit Gulwani showed in August that an off-the-shelf coding agent, given only a static corpus of agent trajectories, could write an optimized prompt in one pass that beat GEPA, the reference reflective optimizer, on three of four agentic benchmarks. The pass cost about $1.60, more than 22× cheaper than validation-gated search, and needed neither environment access nor validation data ².

A month earlier the GEPA team had run GEPA, a Karpathy-style autoresearch loop and Meta-Harness on 10 competitive-programming problems at a matched $20 budget. All three were far ahead of zero-shot (43.8 to 55.4, against 7.72), but "per problem, the winner is nearly a coin toss" ³.

Read together, these papers say that generating candidate edits is now cheap and that the choice of generator makes less difference than most papers assume. Whether a run helps depends on two other parts of the loop: the evaluator, which scores the program, and the acceptor, which decides whether to keep an edit. Wahi's evaluators were fooled by a parser, a bad label and a leaked answer key. The usual acceptor keeps whatever scored highest on a small, noisy dev set. The 2026 literature on prompt and program optimization is mostly about those two parts, even when the papers present themselves as better proposers.

A prompt optimizer treats the text a frozen model reads as the parameters to learn. Most customers can't fine-tune closed frontier models; they can fine-tune open models, but then have to host them and re-tune on every upgrade. Instructions are cheap to change, a reviewer can read them, you can roll them back, and they work with any endpoint. Databricks showed the payoff in September 2025, when it used GEPA to lift the open-weight gpt-oss-120b 2.2 points above Claude Opus 4.1 on its information-extraction benchmark at roughly 90× lower serving cost ⁴.

Every system discussed here runs the same loop. A proposer suggests a new version of some text, an evaluator runs the program and returns a score (sometimes with written feedback), and an acceptor decides whether the new text replaces the old.

From 2022 to 2025 the field worked on the proposer. APE and OPRO asked an LLM for candidate instructions and kept the best-scoring one ^5,6; ProTeGi and TextGrad gave the proposer natural-language critiques, called textual "gradients", to act on ^7,8; DSPy and MIPROv2 turned pipelines into programs of modules whose prompts are compiled against a metric ^9,10. The acceptor barely changed: evaluate on a few dozen examples and keep the candidate if the number went up. An optimizer that proposes hundreds of candidates will find some that win by chance, and a greedy gate keeps them.

GEPA (July 2025; ICLR 2026 oral) is the baseline every 2026 paper compares against, so it helps to know how it works ¹¹. Each iteration samples a candidate from a Pareto frontier (a candidate stays on the frontier if it is best on at least one training instance), runs one module on a minibatch of about three examples, and passes the traces and feedback to a reflection prompt that diagnoses failures and writes a new instruction. Children that improve on the minibatch are scored on a separate Pareto set and may join the frontier.

Reflection turns a sparse reward into a diagnosis, and the Pareto front keeps specialists alive so the search doesn't collapse early. Against GRPO on six tasks GEPA was better by 6% on average and by up to 20%, with up to 35× fewer rollouts ¹¹. That comparison is about task adaptation, where the model already has the capability and needs to be told how to use it, and it doesn't show that prompts teach new capabilities.

The last three months changed who proposes. Coding agents that read traces, write analysis scripts and edit files have replaced purpose-built search loops, in results from independent researchers, the DSPy ecosystem and the GEPA team itself.

CASD (Coding-Agent Skill Distillation), from August, drops the search loop entirely ². Its authors write that the standard propose-rollout-score-retain loop "is unnecessary". A coding agent receives a static corpus of trajectories, writes and runs code to compute failure statistics across the whole corpus, inspects representative episodes, and distills behavioral rules into the prompt. In their account the key variable is how much the proposer reflects over: GEPA reflects on a handful of trajectories per step, while CASD reasons over the whole corpus at once.

Across ALFWorld, τ²-bench retail and telecom, and SpreadsheetBench-Verified, with the same data access for everyone, one CASD pass improved on the unoptimized baseline by 16.6 percentage points on average, against 10.9 for GEPA and 5.3 for SkillOpt, a validation-gated reflective search ². When the competitors got extra validation data and unrestricted environment access, CASD stayed ahead on two of four benchmarks.

I read CASD as strong evidence that search sophistication isn't what limits these systems: one well-informed proposal beat hundreds of greedy iterations. It also has no acceptor at all. No validation set checks the output, so if the corpus statistics mislead the agent, whatever it writes goes out unchecked unless a human reads it.

The GEPA team's July release, "optimize_anything goes omni", makes the proposer interchangeable on purpose ³. The optimize_anything API (published as a paper in May and accepted to CAIS 2026) frames any problem as improving a text artifact scored by a function; the paper reported Gemini Flash rising from 32.5% to 89.5% on ARC-AGI through agent-architecture search, and CUDA kernels that matched or beat PyTorch 87% of the time ¹².

The July release added pluggable engines: GEPA, an autoresearch-style autonomous Claude Code session, and Meta-Harness, the Stanford harness optimizer whose coding-agent proposer reads every prior candidate's code, scores and traces from a filesystem ¹³. On Frontier-CS, with task, model (Claude Sonnet 4.6), budget and evaluation server held fixed, GEPA averaged 43.8, autoresearch 55.4 and Meta-Harness 50.9, and per-problem wins split 3, 3 and 4 ³.

Each optimizer also plateaued early, and a different optimizer seeded with the stuck candidate usually kept improving. omni uses both effects: run all three engines on a slice of the budget, keep the best candidate, and hand it to a fresh optimizer. Every omni variant beat every standalone optimizer; omni-GEPA reached 61.8 against standalone GEPA's 43.8 ³.

Simpler proposers do about as well. Naive Prompt Optimization (NPO), from Purdue in August, keeps a single lineage in which a teacher model revises the prompt from rollout feedback, with no population, Pareto front or merging ²³. It matched or beat GEPA with fewer rollouts, and its advantage grew with teacher strength: "stronger teacher reasoning can partially substitute for optimizer-side search complexity." GRPO still won on some interactive games that respond less to prompting. CASD, omni and NPO agree, and I believe them: a strong model with broad evidence does as well as elaborate search, and which search strategy wins on a given problem is close to unpredictable ^2,3.

FAPO, a June paper, showed what a coding agent does when it can change more than wording ¹⁴. It gives a multi-step pipeline, in a standardized codebase, to Claude Code, which evaluates it, inspects intermediate outputs, diagnoses failures and proposes scoped changes. It tries prompt edits first and restructures the chain only when attribution points to a structural bottleneck, within a scope declared in advance. Against GEPA it won 15 of 18 model–benchmark comparisons (11 with non-overlapping ranges) by a mean of 14.1 points. In the six HoVer and IFBench cases where it escalated to structure, it won all six, by a mean of 33.8 points ¹⁴.

DSPy's Flex module (August) exposes a module's source code to GEPA alongside its instructions ¹⁵. On a location-matching task (240 held-out records, Claude Haiku 4.5 executing, Claude Opus 5 reflecting), the unoptimized program scored 90.4% at $0.98 per 1k records. Prompt-only GEPA reached 92.5% by writing a much longer instruction, which raised serving cost to $2.88 per 1k, 2.9× the baseline.

GEPA on the Flex program reached 95.0% at $0.70 per 1k (McNemar p = 0.019), because the reflection model wrote Python that handled easy cases deterministically and sent 75% fewer calls to the model ¹⁵. With a penalty on LLM calls in the metric, one variant called the model once across all 240 records and held 92.1% at about a hundredth of the baseline cost.

Flex's prompt-only arm shows a problem other papers measured too: reflective optimizers keep adding text. ESPO (September) found that GEPA-style iterations append rules and caveats, "producing prompts up to 3× longer yet no more accurate" ¹⁶. Its remedy clusters all training errors into structural patterns in one round, proposes candidates from four strategies with different biases, and selects with bootstrap stability selection. On seven NLP benchmarks ESPO averaged 74.67% against GEPA's 70.91%, with prompts 47% shorter (1,004 against 1,878 characters) ¹⁶.

SEPO (August) edits typed units in a two-layer prompt schema and records, for each edit, which examples it newly fixed or broke. On a 14-task held-out suite it beat GEPA by 3.1 points on Llama-3.1-8B and 2.2 on Qwen3-8B, spending 2.9M optimization tokens against GEPA's 4.1M and producing prompts over 5× shorter ¹⁷.

Two September papers apply structural editing to multi-agent systems. Control-data flow separation starts from the observation that one prompt often holds both task content and execution-critical protocol (routing, output format, termination signals), so an edit meant to improve content can corrupt the protocol and crash the pipeline ¹⁸. Moving control into typed, validated program objects and optimizing only the language left the framework with 100% eventual protocol validity while task scores improved.

AgentGrad goes after credit assignment: for each failure it modifies one agent at a time to find the agent whose change resolves the failure, extracts a gradient from that agent's corrected output, and clusters similar gradients before applying them. It reported state-of-the-art results on five multi-agent benchmarks and 2.5× faster optimization than the next-fastest baseline ¹⁹. The motivation goes back to January, when TextEP showed that textual feedback relayed through deep pipelines either amplifies biases or dissolves into generic advice ²⁰.

Harness research reached the same conclusion from the other direction. HARNESSEVO (June) split an ALFWorld scaffold into four separately evolved text slots under an equal budget and found no significant overall gain over evolving one flat string (0.657 against 0.642), with almost all the value in the reflection/control slot ²¹. Changes to control flow account for most of the gain, which is why prompt optimization is merging with harness optimization.

Taken at face value, the positive results from this year make a consistent case: coding-agent and structure-aware optimizers beat GEPA, often with less compute and shorter prompts, and prompt optimization combines well with weight training.

Prompt edits are increasingly used to produce training signal, and several of these results measure that. FORGE (September) treats every failure as two signals: how the prompt should change, and what new training data should be synthesized ²². It co-evolves prompts and verified synthetic instances, improving the aggregate score over the unoptimized baseline by 16.52 points across eight benchmarks. The synthesized data also helped outside FORGE, improving nine other prompt-optimization comparisons by 2 to 9 points and three GRPO comparisons by 4 to 8 points under matched budgets ²².

mmGRPO (revised May, CAIS 2026) found that running prompt optimization before multi-module GRPO beat either alone (73.4 against 71.2 for RL and 70.0 for prompt optimization), and attributed the gain to "the value of high-quality rollouts at the start" of RL ²⁸. Databricks found similar stacking with supervised fine-tuning in 2025: GEPA alone +2.1 points, SFT +1.9, both +4.8 ⁴.

Look at what nearly every row has in common, though. Each compares against GEPA or a zero-shot prompt, often at a single seed, with a greedy acceptor deciding what to keep. Several of the newest rows beat GEPA partly by fixing GEPA's selection step (ESPO's bootstrap selection, RLMOpt's regression constraints, SEPO's fix/break ledger), so part of what the table measures is better acceptance rather than better proposals.

Wahi sorts his 11 production failures into four classes: judge bias, harness and metric failures, ground-truth errors, and reward hacking ¹. Each defeats a greedy acceptor for a different reason. A parser that silently falls back to a default makes a broken prompt look better. A wrong label rewards the optimizer for deleting correct rules. A leaked answer key produces a perfect score that measures nothing.

His response, a Teacher–Student loop called PROCTOR, demotes the LLM judge "from oracle to advisor" and gates every change with five deterministic guardrails: hermetic sandboxes, capability-disjoint roles, acceptance checks that outrank the Teacher, frozen holdouts, and canary cases "engineered so that a perfect score is itself evidence of cheating" ¹. It is a single-author position paper with no controlled comparison. It is also a detailed public record of what a production prompt-optimization loop gets wrong, and every failure in it happens after the proposer has done its job.

Coin Flip, an April paper, gives the base rate ²⁷. Across 72 optimization runs on Claude Haiku 4.5 (six methods, four tasks, three repeats), 49% of optimized prompts scored below zero-shot, and on Amazon Nova Lite the failure rate was higher. The one clear success, HelpSteer2, had "exploitable output structure": a format the model can produce but doesn't produce by default.

Elsewhere, instruction tuning has made the model much less sensitive to phrasing, so an optimizer that perturbs wording has little to gain and ends up picking whichever candidate got lucky. A companion ANOVA found that the interaction between two agents' prompts explained 0.18% to 2.15% of variance and never reached significance, which undercuts the case for optimizing multiple modules jointly.

The authors' diagnostic is a $5 headroom test (generate 10 to 20 candidates, and optimize only if the best beats zero-shot by more than 2 points on 20 held-out examples) plus an $80 coupling test, against their estimate of $1,000 to $5,000 for a full DSPy compile ²⁷. One gap: as described, their method set didn't include GEPA. Even so, the $5 test is the first thing I would run before starting a prompt-optimization project.

RLMOpt, from August, reached the same conclusion about headroom. It has a recursive language model run the search while a deterministic harness enforces scoring, Pareto selection and regression constraints, and its authors conclude that "optimization gains are determined primarily by the headroom available in the seed prompt, rather than by the search budget" ²⁴. Across 11 matched runs it never produced a prompt worse than its seed; GEPA did so twice.

MAGE (July) found that with 30 training examples, well-designed fixed prompts beat every reflective optimizer it tested, and that expanding the candidate pool from three to five raised mean accuracy by 21.6% while raising variance 3.7× ²⁵. Its author calls this the "prompt optimization coupling effect": stochastic components in a closed reflective loop raise the mean and the variance together, so the result of a single run tells you little.

PACE, a June paper, explains those variance numbers ²⁶. A loop that keeps any candidate whose dev score beats the incumbent is doing adaptive multiple testing without correction; in the paper's words, the agent "effectively p-hacks itself." With Qwen2.5 agents from 0.5B to 3B parameters self-evolving on GSM8K, SVAMP and ARC-Challenge, the greedy acceptor did commit a real hidden improvement, but 30% to 42% of its other commits were false and 10% to 33% were harmful. When no real gain was available, it made 13 to 21 self-modifications per run, 72% to 100% of them false, and degraded the most fragile agent by 4.9 points ²⁶.

PACE's fix treats each commit as a sequential test, using a betting-based e-process that stays valid under optional stopping: a candidate is committed only when paired evidence against the incumbent crosses a threshold. It committed the real improvement and almost nothing else, and matched greedy's held-out accuracy at about 18% lower evaluation cost.

The caveats are serious. PACE is a single-author preprint, validated only on models of 3B parameters or fewer and only on prompt-level edits to single agents. It hasn't been tested on multi-module programs, frontier models, or the harness and skill edits where the same greedy gate is standard (reports 08 and 14). I still think it states the problem correctly, and it gives a mechanism with a guarantee attached. The positive results in the table above mostly used some version of the greedy acceptor PACE shows to be broken.

Two September optimizers build weaker versions of the same idea into their selection steps. STEVE traced textual-gradient instability to two sources, gradients computed from examples the prompt already handles and over-specialization to hard cases, and treats every update as provisional: it is accepted only if its gains on hard cases don't cause unacceptable regression on a preservation set ²⁹. ESPO's ablation found that adding candidate diversity without bootstrap stability selection lowered accuracy by 1.20%, so a stronger proposer hurt when the acceptor stayed naive ¹⁶. Neither offers PACE's statistical guarantee, but both put the fix in the acceptance step.

Three more results limit what an optimized prompt is worth once you have it. Joel Niklaus at Hugging Face found in July that a harness evolved for one DeepSeek model gave its sibling +14.4 points but gave a Nemotron model +0.4; the code mechanisms transferred, while "prompt playbooks are model-specific and can backfire" ³⁰. "Optimization before Evaluation" (April) showed that optimizing prompts per model reorders benchmark rankings, so comparisons that use a single template are confounded by how well the template fits each model ³¹.

CPInj (July) showed that in collaborative prompt optimization, where several clients jointly refine a shared prompt, injected malicious instructions remain after aggregation, persist through later benign optimization, and evade current server-side defenses ³². An optimized prompt is fitted to one model, one benchmark, and whoever contributed to it.

Model releases also make optimized text go stale. Anthropic said in July that it had "removed ~80% of the Claude Code system prompt for our newest models" ³³. Instructions the previous model needed became redundant once the next model was trained on the behavior. So an optimized prompt is worth the most on models that won't be retrained soon (open-weight deployments, frozen enterprise endpoints), and as a source of trajectories and data that the next training run can absorb, which is how FORGE and mmGRPO use it ^22,28. For anything meant to last past the next model release, I'd put that knowledge in code and tests.

The optimizers ship as open-source libraries, and a human runs the compile. DSPy (about 38,000 GitHub stars in September) includes GEPA, SIMBA and MIPROv2, and added Flex in August ^34,15. GEPA's repository (about 6,700 stars) packages optimize_anything, the omni engines and the Terrarium benchmarking harness, and claims "50+ production uses" across companies including Shopify, Databricks, Dropbox and OpenAI ^35,3.

The vendor features predate this year's critiques: OpenAI's dashboard prompt optimizer, Anthropic's Console prompt improver (launched November 2024, per a secondary source), Google's Vertex AI Prompt Optimizer (generally available since August 2025), and Arize's Prompt Learning SDK, built on Andrej Karpathy's May 2025 description of "system prompt learning" as a missing third way for models to learn ^{36,37,38,39,40}. All of these assume a human in the loop: an engineer assembles data and a metric, runs a compile, reads the result and deploys. The products don't document an anytime-valid acceptor, a headroom check or canary cases.

Can false-commit control work for real systems? PACE gives a guarantee for single-agent prompt edits on small models, and PROCTOR gives a production checklist without controlled evidence. False-commit control hasn't been shown for multi-module programs, structural edits or LLM-judged metrics, which is where FAPO, Flex and omni operate. Until someone does, I'd read the positive results above as upper bounds from greedy gates. CASD makes the question sharper: if one unvalidated pass beats validated search, an acceptor has to prove its worth by the errors it prevents.

How much of an optimized prompt's gain remains after an upgrade to the next version of the same model? NPO finds prompts transferring within a family, Niklaus finds prose failing across families, and Anthropic's deletions show instructions going stale as models learn the behavior. The three studies don't measure how much of the gain remains after a version upgrade within one family. Teams need that number to know whether optimized prose is an asset or a recurring cost, and whether lessons belong in code and tests instead, since those transfer better.

When is a prompt worth optimizing, and when should a run stop? Coin Flip's headroom test and RLMOpt's finding that headroom, not budget, drives gains give a rule for starting. omni's result, where a stuck optimizer improves again after a switch, suggests a rule for stopping or switching. There is still no convergence theory for reflective optimizers, and no standard for how many held-out wins an edit needs before it counts.