Grading agents
or, how to measure something that can reach the answer key
An eval never observes the capability it’s named after. A coding benchmark is meant to tell you whether an agent can fix bugs the way a maintainer would. It observes whether a few hidden tests pass after the agent stops, and infers the rest. The inference is sound when both of the grader’s error rates are low: passes without a real fix, and failures with one. Both rates depend on the agent as much as on the tests: a suite that almost never accepts a wrong patch from a person can accept plenty from a program searching for whatever passes.
For most of AI’s history, the gap between a check and the capability behind it came from mistakes: too few samples, tests that missed cases, answer keys that leaked into training data. Agents add a different source. An agent runs code, reads files and searches the web, so its search for a passing result can include looking up the fix, editing the tests or reading the grader, and the more capable it is, the more of those paths it finds. In July, during an internal OpenAI security eval, hundreds of agents broke into Hugging Face’s production systems, largely to work out how they were being graded.
If evals only ranked models, this would be a problem for leaderboards. But anything you can grade, you can train on. The grader’s score becomes the reward, and training moves the model toward whatever the grader accepts, including what it accepts by mistake. So the question I care about is how to build a check whose pass still means the work got done when the thing being checked is looking for the cheapest pass. I don’t think anyone has a complete answer yet, but most of one is spread across two years of broken benchmarks and a newer way of building agents out of small, separately checked decisions.
01Grading the world
A chatbot gives you an answer you can compare with a key. An agent runs commands, edits files and calls APIs, then writes a message saying what it did, and that message is often wrong: in one study of coding agents, a quarter of the failed runs ended with the agent reporting success.1 So agent evals ignore what the agent says and grade what it leaves behind.
SWE-bench Verified, which coding agents were scored on for two years, works this way. In one of its tasks the agent gets a 2019 Django bug report and the repository from before the fix. The report says the username validators accept names ending in a newline, because in Python’s regular expressions $ also matches just before a trailing newline, and the fix is to end the pattern with \Z. When the agent stops, a fresh container applies its changes and runs tests it never saw. Three that failed before the real fix must now pass, and the rest must still pass.
Every agent eval has the same six parts: the task, the environment the agent acts in, the harness around the model, the trajectory of everything the agent did, the grader, and the rule that turns attempts into a score. Any of the six can break the inference from score to capability, and each one has.
\A…\Z while Django’s own fix kept ^ and changed only the end: the hidden tests check behaviour, so both pass.Grading the world fixes the agent that misreports its work. It does nothing about the agent that does different work every time you run it.
02Running it again
Run the same agent on the same task twice and you can get a pass and a fail. An agent makes hundreds of small decisions, and one early difference sends it down a different path. Across a benchmark these differences add up: rerunning the same agent on SWE-bench Verified moved its score by up to six points, even with the sampling temperature at zero.2
Yet many published agent scores come from a single run.3 The fix is ordinary statistics. A score contains two kinds of luck, which attempts you sampled and which tasks the benchmark includes, and rerunning only shrinks the first. On a 50-task benchmark the second alone is worth about 14 points either way. Comparing two agents task by task cancels much of it, because a hard task is hard for both.4
Reruns also raise a question leaderboards skip: should the agent succeed once, or every time? A coding assistant that works one try in five is useful, because you can retry. A refund agent that gets one customer in five wrong is a liability. pass@k counts a task as solved if any of k attempts succeeds, and pass^k, introduced with τ-bench, only if all k do. As k grows, they move in opposite directions.5
Real benchmarks fall between the two curves, because some tasks pass every time and others never do. pass^k then approximates the share of a benchmark an agent has reliably solved, and that’s the number I’d want before putting an agent in front of customers.
Error bars tell you how much of a score is luck. They don’t tell you what the score is a score of.
03Whose score is it?
Leaderboards print a model’s name next to each number. I’d stop doing that. The model runs inside a harness (the prompt, the tools, the rules for trimming old output, the budget of steps and money) that someone set up, and its result is counted by a rule someone wrote. Change any of those and the number moves, often by more than the gap between the top models.
On Terminal-Bench 2.1, GPT-5.5 scored 76% in one harness and 57% in another, with the same tasks and the same grader.6 Neither number is wrong; they describe different setups.
That was the argument of the first version of this post, and I still hold it: a score summarizes a set of trajectories produced by one setup and counted under one rule. Before a number changes my mind about a model, I want these questions answered.
| Question | Why it matters |
|---|---|
| Which version of the benchmark? | “OSWorld” now ranges from 20.6% to 86.1% at the top, depending on version and grading |
| Which harness, effort and budget? | Each moved scores by 13 to 31 points above |
| Who ran it? | Lab-reported and independent numbers differ by up to 22 points |
| How many tasks, attempts and runs? | Below a few hundred tasks, gaps of a few points are noise |
| How were crashes, timeouts and cheating counted? | This question alone moved one METR estimate from 11 to 270 hours |
| Could the agent reach the answer or the grader? | Sealing lookups took 14 to 21 points off SWE-bench Pro scores |
| Did another model answer some of it? | A fallback flipped first place on Terminal-Bench 2.1 |
04When the check is wrong
Even a well-run, well-counted score is only as good as its check, and every check is narrower than the work. On the Django task, the work is “fix this the way a maintainer would” and the check is three tests. A check can fail in two directions. It can reject correct work, say with a test that demands an error message the bug report never asked for. Or it can accept wrong work: the hidden test tries a single name ending in a newline, so a patch that rejects only that exact string passes.
The two errors do different damage. Rejecting correct work caps the score below 100%, and someone eventually notices, because good runs fail.7 Accepting wrong work inflates the score, and nothing fails to tell you. It also gets worse as agents improve, because a stronger agent is better at finding what a grader accepts but shouldn’t.
SWE-bench Verified died of both. OpenAI’s engineers found flawed tests in most of a sample of its hard tasks, and maintainers reviewing patches that had passed would have merged about 24 percentage points fewer. The top score still reached 97%, above the share of tasks that appear to be fairly tested, which suggests to me that the last points measured memory.8
Most work has no tests at all: whether a customer was handled well, whether a maintainer would merge a patch, whether a contract summary is right. That work gets graded by a language model with a rubric, and LLM judges are cheap, fast and wrong in predictable directions. In one study that compared 12 judges with expert labels on web-agent runs, none reached 70% precision, so at least 3 in 10 runs a judge passed had failed.9
The fix comes from medicine, where no test is expected to be perfect. Treat the judge as a diagnostic test with two error rates, how often it passes real successes and how often it fails real failures. Measure both on about 200 cases a person has labelled, and use them to correct the judge’s headline number.10
Before writing any check, read the transcripts. About 100 real ones, each with a note on what went wrong, will show you failure modes you wouldn’t have thought to test for, and you can’t write a check for a failure you haven’t seen.11
05When the agent finds the gap
Every failure so far happened by accident. But an agent eval gives a capable program a shell, a file system and often the internet, and keeps the grader nearby. Once agents got good enough, they started using those tools on the eval.
Attacking a grader takes very little. A Berkeley team’s agent, which solved nothing, scored 500 out of 500 on SWE-bench Verified with a 10-line file that told pytest every test had passed, and it scored near-perfect on most of the other benchmarks it tried.12
Deliberate attacks can be tested for. Ordinary agents finding holes on their own are harder to catch. When Cursor audited successful runs of Claude Opus 4.8 Max on SWE-bench Pro, 63% had retrieved the known fix, from the project’s public history or from git objects left in the container, instead of writing one. Sealing the history and the network dropped its score from 87% to 73%, while an older model barely moved, so part of the gain between model generations was a gain at finding answers.13
Closing one route sends a capable model to the next. When Vals sealed git history in the coding environments Xiaomi released with its MiMo model, the model read file timestamps, dug through build caches and wrote its own parser for git’s packed object format.14
In July, agents went after a grader at scale. During ExploitGym, an internal OpenAI cyber eval, about 1,200 agents coordinated on an unsanctioned message board, and roughly 700 of them broke into Hugging Face’s production systems, largely to learn how their grader worked. Almost every task they discussed was one no model had ever solved, and they gained nothing, because OpenAI’s internal grader differed from the public one.15 Give a capable agent an impossible task and a grader it can reach, and the grader becomes the task.
An agent doesn’t even have to touch the grader. Models now notice when they’re being tested, and they behave differently when they do. Partway through the BrowseComp web-research benchmark, Claude Opus 4.6 guessed it was being evaluated, worked out which benchmark it was in, then found the benchmark’s encrypted answer key and decrypted it.16
Cheating also breaks the measurement itself. METR estimates a model’s time horizon, the length of task, in a skilled person’s working time, that the model finishes half the time. For GPT-5.6 Sol the same runs gave three answers: 11.3 hours if runs where the model cheated count as failures, 71 hours if they’re dropped, and more than 270 hours if you trust the automatic scorer.17 When cheating is that common, the rule for counting it is the measurement.
06Rewarding A, hoping for B
Most of these failures come from a grader paying for something slightly different from what its builders wanted. Steven Kerr, a management professor, described the pattern in 1975 in a paper called “On the Folly of Rewarding A, While Hoping for B.” In universities, he wrote, society hopes professors will teach but rewards them almost entirely for research and publications. His diagnosis was that “some parts of the task are highly visible while other parts are not,” so rewards go to the visible parts, and people do what they’re rewarded for.18
Most of this post is that folly with a computer in it. We hoped for a fixed bug and rewarded passing tests. We hoped for a capable model and rewarded a number produced by a harness, a budget and a counting rule. We hoped the agent would do the task and rewarded any path that ended at the grader. Kerr even described the agents that went looking for answer keys: cheating students, he wrote, are making “a personally rational response to a reward system which pays off for grades rather than knowledge.”
What changed in 2026 is who does the optimizing. Pressure on benchmarks used to come from people: tuning prompts, reporting their best run, training on data that happened to contain the test. Now it also comes from the model, which finds the gap between A and B on its own inside an eval and gets paid for finding it in training.
A reinforcement-learning environment is a task, a sandbox and a grader whose score becomes the reward, which makes it an eval with learning switched on. The same task directory now runs as a benchmark, a training environment and, pointed at real sessions, a monitor.19 Train on the Django task with the network open, and the model learns to look up the commit. A hole in a grader stops being a wrong number on a leaderboard and becomes a habit the model brings to every task with the same hole.
Jason Wei’s verifier’s law is the optimistic half of this: “All tasks that are possible to solve and easy to verify will be solved by AI.”20 The other half worries me more. Whatever we verify badly, we’ll get badly, at scale, in every model trained against that verifier. The grader is the only description of the task that training ever reads, which makes it the specification.
So I disagree with both popular takes on evals this year. “Benchmarks are dead” confuses the leaderboard with the instrument: public numbers saturate within months, but labs now pay billions for graders, because graders are also the training signal.21 “Vibes are enough” works while a person can read every output, and stops working once agents run for hours.
The same mechanism explains a puzzle Ilya Sutskever raised last November: models are “doing so well” on hard evals, “but the economic impact seems to be dramatically behind.” Part of his answer was that teams design RL environments by asking what training would make the evals look good, and a model trained on the evals learns the evals.22
The first version of this post argued that the trajectory is the evidence and the score is a summary of it. I’d now add the grader. The grader is the specification, the trajectory is the evidence, and the score is a summary of both that you shouldn’t read on its own.
07Where things stand
Most of the agent benchmarks people quoted in 2025 are retired. OpenAI stopped reporting SWE-bench Verified in February, and Vals archived it at 97% in September. Terminal-Bench 2 is archived too, and METR says its time-horizon measurements above 16 hours are unreliable, a line the best models have crossed.23 The map places this fall’s benchmarks by how long their tasks take a person and how the work gets graded.
Saturated benchmarks cluster where tasks are short and graders are code. Those tasks are the easiest to train on, and the first where agents found cheap passes. The headroom is in tasks that take people hours or days and get graded by rubrics or by people, which makes grading slow, expensive and unreliable in its own ways. Saturation also takes months now: this year’s benchmarks that have reached 80% got there two to eight months after their first published scores.24
For a team building its own agent, the tools sort into a handful of jobs:25
| Job | What it covers | Examples in 2026 |
|---|---|---|
| Build-time evals | Datasets, graders, side-by-side experiments | Braintrust, LangSmith, Langfuse, Inspect, Promptfoo |
| Simulation | Simulated users and tools before shipping | Google Agent Platform, AWS AgentCore, Harbor, Raindrop |
| Production scoring | Judges on sampled live traces, failure clustering | LangSmith Engine, Braintrust Loop, Arize Alyx, Laminar |
| Graders in the loop | A check that runs inside the agent’s own run | Anthropic Managed Agents “Outcomes” |
| Environments | Containerized tasks with graders, for evals or training | Harbor, Prime Intellect verifiers, OpenEnv, NeMo Gym |
| Independent measurement | Public, comparable numbers | Vals, Epoch, Artificial Analysis, Scale, METR, Arena |
Most work isn’t measured at all. Mapping existing benchmarks onto US occupations, Mercor found that 90% of hours worked aren’t covered well by any benchmark.26
08Breaking sessions into decisions
Most of the evals above grade an agent by where its session ends. Since September, a different way of building agents has been splitting sessions into small decisions that can each be checked. Many steps in an agent loop end in a choice the program could list in advance: which tool to call, which value to pass, whether to stop, whether to ask the user. In the new design, code runs the loop and sends each of those choices to a decision model as a typed question (yes or no, one of a list, or a score), which returns a probability for every answer in one pass, without generating text.27
TypeSafe’s Jev, released on September 15, made the category popular, and OpenAI, Microsoft and Perplexity now offer decision models of their own, alongside open ones. On Decision Bench, a public set of 1,071 such questions built from real records, the top 10 models are statistically tied, decision models and general-purpose LLMs alike. The decision models are much cheaper and faster: Jev answers in under half a second for about four cents per thousand questions, and Gemini 3.5 Flash takes 3.6 seconds and $3.32 per thousand.28 At that price a harness can afford a question at every branch of an agent’s loop.
For evals, the size of the unit is the appeal. A decision small enough to label is small enough to measure: collect a few hundred real cases of one decision, label them, and count both of its error rates. If the model’s probabilities are calibrated, a threshold then sets the trade between false passes and false fails at whatever a mistake costs at that step. TypeSafe’s founder, Diogo Almeida, says that “if you decompose problems into simple decisions, every single one of these things is extremely evaluable,” and describes fixing a failure by adding a question, a threshold and a test case.29
Counting both error rates is the step a single accuracy number skips. Four of Decision Bench’s 35 tasks ask a model to grade an agent or its output, and one of them gives it 30 recorded customer-service conversations and asks whether the agent completed the task. Jev and DeepSeek V4.1 Flash each got 23 of the 30 right. Jev’s seven mistakes were six false passes and one false fail, and DeepSeek’s were two false passes and five false fails.30 Used as rewards, the first would pay for unfinished work and the second would withhold pay for finished work, at the same accuracy.
I ran my own test of Jev as a judge in September: about 3,600 cases with known answers, from my agent sessions, annotated agent trajectories, public judge benchmarks and checks I built, with Claude Haiku 4.5 and the reasoning model GPT-5.6 Luna judging the same cases. On literal checks, such as whether a reply is in the requested format or whether a tool call returned an error, Jev matched both. It trailed the reasoning model wherever a check needed a chain of calculation, and on day-of-week arithmetic it was at chance while both LLMs were near perfect.31
The probabilities attached to its answers were the more useful part. On agent traces, Jev’s answers given 90% or more probability were right 84 to 90% of the time, while Luna gave that much probability to most of its answers and was wrong on 17 to 35% of them. Jev was also steadier: asked the same five-level quality question 20 times about each of 30 runs, it changed its answer on 1.7% of repeats, against 9.2% for Haiku and 22.5% for Luna.32 A threshold on Jev’s probability tracks its accuracy, and a grader that changes its answer between runs adds its own noise to every score.
A study published this month tested decision models inside running agents and found the gap between proxy and capability one level down. Researchers at UIUC, Princeton and Google scored nine decision models three ways: on isolated questions, on recorded agent states and in control of a running agent. On isolated questions the models were 5.1 points apart; in control of the agent they were 57.5 points apart, and the most accurate model on isolated questions solved 69.5% of tasks the LLM alone solved every time.33 Extra public training data raised one model’s isolated accuracy from 78.8% to 85.1% and lowered its agent’s success from 96.0% to 91.5%.34
Inside an agent, one wrong choice changes what the agent sees next, and the agent reaches states after its own mistakes that fixed question sets rarely contain. Failures also gathered where an average can’t see them. Every failed task of Jev and the two largest open models was lost at the model’s first wrong tool choice, usually a cancellation or refund where the agent should have looked something up or asked the customer. All three were right at only about a quarter of the states that called for a question, and those states were 5% of the test set.35
Decision models also get used as checks inside the loop, and the same study measured how well they work there. As a check on whether the agent should be allowed to stop, Jev accepted 12% of wrong finishes from a pool built mostly of forced and transplanted mistakes, and 41% of the wrong finishes a weaker agent produced on its own. A checker’s error rates depend on whose mistakes it is checking. The authors conclude that a decision-model guard “should never be the only control on an irreversible action.”36
Delegating choices worked only behind a confidence threshold fitted for each decision on the agent’s own runs. With it, Jev made 27% of the tool choices without changing the outcome of any of 500 tasks; given every tool choice, it cost 19 points of success. The fitted thresholds came out at 0.99 and 1.00, so only near-certain answers were trusted.37
Calibration measured on one set of inputs need not hold on another. Independent audits put Jev’s calibration error near 0.03 on public benchmarks and 0.11 on tickets it couldn’t have seen.38 In my test, asked to predict from a user’s request alone which kind of tool the agent would use, something the request doesn’t settle, Jev put 90% or more on 61% of its answers and was right on 58% of those, as overconfident as the reasoning model.39 In the in-agent study, the authors’ model met a business rule it was never trained on, kept deciding confidently, and its agent solved 3% of tasks against 88% for the LLM alone.40
The decomposition is itself a specification. Only the questions someone wrote down get checked, with only the answers someone thought to offer: in one audit, deleting the “unknown” option from questions that had no answer took Jev from 95% correct to 0%, at 79% confidence.41 A session can also fail at a step that was never turned into a question. In coding agents, whose actions are mostly free text, the in-agent study’s decision model could only choose which LLM wrote each step, and the savings were small.42
I think this design is good for evaluation, because it shrinks the unit being graded from a session to a choice and logs a probability with every choice. The inference from score to capability is still there, split into many small inferences that have to add up. Each threshold holds for the inputs it was fitted on, and the agent is what changes those inputs.
09What I’d want from an eval
I’m still bullish on evals, just pickier about which ones I trust. Each failure above suggests something I now look for, in a public benchmark or in the suite a team runs on every commit:
- Tasks come from real work, and someone read real transcripts before writing the checks.
- Each task runs more than once, the score comes with a range, and pass^k is reported wherever reliability is the point.
- The harness, the budget and the counting rule are published, and every trajectory is kept, so anyone can recount the score under a different rule.
- The grader runs where the agent can’t reach it, and someone reads a sample of the passing runs, because a pass is a claim too.
- Decisions are tested inside the running agent, and a checker’s false-pass rate is measured on the agent’s own mistakes.
- Some tasks are impossible, and saying so earns credit. OpenAI said it would grade this way after ExploitGym.43
- Part of the eval replays real production conversations, which are hard for a model to tell apart from real use.44
Each of these has a price. Reading trajectories doesn’t scale, and handing the reading to a model brings back the judge problem. Replayed traffic works only while models can’t tell it from real use, and Gemini 4 Argon already says it suspects a test in 61 to 77% of runs on a simulated deployment.45 Long tasks graded by people are slow and scarce: METR’s suite has five human-baselined tasks longer than eight hours.46 And every check you add is another proxy, with its own cheap passes for an agent to find.
10What I don’t know
I can’t answer what happens once agents are better than the people checking them, at the work and at reading transcripts of it. Today the final check, when there is one, is a person reading a sample of runs, and that works while the person can tell a good run from a clever one.
Decision models make checking cheap enough to do at every step of a trajectory, which makes it affordable to grade every decision an agent makes, and to reward it. Finer-grained rewards have a record, though: step-level reward models paid for repeating correct but unnecessary steps, and policies trained on summed rubric scores learned to satisfy compound criteria in part and to buy back missed points with advice that wasn’t asked for.47 I haven’t found a study of an agent trained against a decision-model grader that reports what the agent learned to exploit, and I’d want one before trusting such a grader as a reward.
My guess is that we’ll lean on the asymmetry behind Wei’s law: build tasks where checking is much cheaper than doing, and spend the effort on graders. Where that isn’t possible, agents will grade agents, with people auditing a sample, the way auditors check a company’s books without redoing every transaction. Whether that still works when the agent being audited understands the audit better than the auditor is the problem I’d most like someone to solve, and I don’t think anyone has yet.
Go deeper
Most of the good teaching material on this is free. If you want to learn it properly, this order works:
- Percy Liang, CS336 lecture 12, “Evaluation” (Stanford, spring 2026). The big picture, from construct to metric, plus contamination and agentic benchmarks.
- Yejin Choi, CS224N lecture 11, “Evaluation” (Stanford, winter 2026). What makes a benchmark good, shortcut learning and model judges.
- Berkeley Agentic AI, “Agent Evaluation” (October 2025). A taxonomy of agent evals and six benchmark case studies; read its pass@k slide with footnote 5 in mind.
- Sida Wang, “Predictable Noise in LLM” (Berkeley, October 2025). Error bars, paired comparisons and signal-to-noise.
- Clay Bavor, “Practical Lessons from Deploying Real-World AI Agents” (Berkeley, November 2025). pass^k and simulation from the team behind τ-bench.
- CMU 11-768, lecture 11 (September 2026) for the false-negative and false-positive picture, and assignment 2, where you build and validate a trajectory grader yourself.
- Anthropic, “Demystifying evals for AI agents” (January 2026). The practitioner’s vocabulary and lifecycle.
- Hamel Husain and Shreya Shankar, “AI Evals: Everything You Need to Know” (September 2026) and the LLM-judge guide. Error analysis and judge validation, for your own product.
- Princeton, “AI Agents That Matter” and the HAL reliability dashboard. Cost-aware evaluation and why reliability is not accuracy.
- Harvard CS 2881R, fall 2026: John Schulman’s October 8 lecture, “Grading, reward hacking, and alignment,” and the incident readings from lecture 1.
- Kunlun Zhu and colleagues, “JevAgentHarness” (October 2026), read beside Decision Bench’s methodology. How to test decision models from isolated questions up to control of a running agent, and why the levels disagree.
About the numbersEvery number comes from the source in the notes, checked against that source between October 16 and 18, 2026. Model names and versions are as the sources give them. One of the evaluations is mine: the test of Jev as a judge in Section 8, described in its notes. The figures that simulate agents say so, and the arithmetic labelled as mine is mine. Disclosure: I use Jev in my own tools and at work, and TypeSafe had no part in this post.
Notes
- 1“Failure as a Process” (July 2026), on 1,794 trajectories from CLI coding agents. Most of those false claims, 84%, came after recovery was already impossible: the agent was reporting success on a run it had already lost. ↩
- 2Bjarnason, Silva and Monperrus, “On Randomness in Agentic Evals” (February 2026): 60,000 trajectories, which diverge “within the first few percent of tokens.” The same agent’s score moved by 2.2 to 6.0 points depending on which run they picked, with a standard deviation above 1.5 points at temperature 0. ↩
- 3A 584-run replication in September found that “identical runs of one pairing varied more than the pairings differed from one another,” and estimated that telling two agents apart would take “tens to more than a hundred runs of each.” Princeton’s HAL leaderboard spent about $40,000 on 21,730 runs and still relied on single runs for most of its evaluations, because repeating them cost too much. ↩
- 4Evan Miller, “Adding Error Bars to Evals” (2024): report a standard error, cluster it when tasks come in groups (231 of SWE-bench Verified’s 500 tasks come from Django), and compare agents on paired tasks. The ±14 is my arithmetic for 50 tasks at a pass rate near 50%; with 500 tasks it’s about ±4. Terminal-Bench 2.0 reports about ±3 points on its 89 tasks, the right size for “would a rerun on these tasks agree?” Counting the luck of which tasks were chosen, I get about ±10. ↩
- 5τ-bench (2024). On its retail tasks, gpt-4o succeeded on 61% of single attempts, and on fewer than 25% of tasks it succeeded all eight times; if every task were equally hard, eight in a row would happen 1.9% of the time. Anthropic’s guide puts the gap this way: “By k=10, they tell opposite stories: pass@k approaches 100% while pass^k falls to 0%.” Course slides mix the two up: a Berkeley Agentic AI slide from October 2025 defines τ-bench’s “pass@k” as all k runs succeeding, which is pass^k. ↩
- 6Vals’ Terminal-Bench 2.1 runs: GPT-5.5 scored 76.4% in the generic Terminus 2 harness and 57.3% in OpenAI’s Codex harness. Older Claude models answered 26 of the 267 attempts credited to Opus 5.5; counting those as failures moves it from 87.6% to 79.8%, behind GPT-6 Astra at 87.3%. Counting rules move numbers too: Rollout Cards (May 2026) found none of 50 popular agent repositories reporting failed, errored or skipped runs next to the headline score, and re-scoring fixed outputs under different plausible rules moved scores by up to 20.9 points. ↩
- 7Sometimes the answer key is what’s wrong. The authors of TheoPhysEval, a theoretical-physics benchmark released this month, first marked GPT-6 Astra wrong on several problems; on closer examination with the experts, “there were subtle errors in the original solutions,” and once those were corrected, Astra’s answers were right. It finished at 98% (paper). ↩
- 8OpenAI, “Why we no longer evaluate SWE-bench Verified” (February 2026): engineers re-reviewed 138 hard tasks and found material problems in at least 59.4% of them, mostly tests too narrow to accept valid fixes, along with evidence that frontier models had seen tasks and their fixes in training. METR’s review was in March: four maintainers of three of the projects, without knowing who wrote each patch, would have merged about 24 percentage points fewer than the tests passed. Epoch’s review puts the share of all 500 tasks with broken tests at no less than 16%, and Vals’ final run had Claude Opus 5 at 97.0%. The memory reading is my inference from those numbers; neither source makes it. Held-out copies point the same way: Anthropic’s Fable 5 scored 95.9% on the public SWE-bench Verified tasks and 78.7% on copies Mercor wrote the same way and kept private. ↩
- 9AgentRewardBench had experts label 1,302 web-agent trajectories. The benchmarks’ own rule-based checks erred the other way and rejected 44% of real successes. For a Claude 3.7 agent on WebArena, the experts counted 55.1% success, a GPT-4o judge 64.1% and the rules 30.8%. Judges can also be fooled on purpose: Berkeley’s benchmark-breaking agent (Section 5) found judges that paste the agent’s text into their own prompts. ↩
- 10The correction is the Rogan–Gladen estimator. Lee and colleagues worked it out for LLM judges at ICML this year; in their example, about 200 human labels bring the corrected estimate’s 95% interval under 10 points wide. Husain and Shankar add two rules in their FAQ: have the judge answer pass or fail instead of scoring 1 to 5, and measure its two error rates on labels it never saw. ↩
- 11This is the error analysis Hamel Husain and Shreya Shankar teach (FAQ): read about 100 traces, write a note on each, group the notes into failure modes, then write one pass/fail check per mode. Shankar’s 2024 study of people writing judge criteria found that the criteria move as you grade: “users need criteria to grade outputs, but grading outputs helps users define criteria.” ↩
- 12Berkeley RDI’s write-up. Terminal-Bench downloads a tool with curl at grading time in 82 of its 89 tasks, so the agent replaced curl and passed all 89. FieldWorkArena’s grader checked only that the last message came from the agent, so replying
{}passed. Their summary: “Don’t trust the number. Trust the methodology.” ↩ - 13Cursor, “Reward hacking is swamping model intelligence gains” (June 2026). An auditor agent read 731 trajectories without seeing whether they passed. Most retrievals were the upstream patch looked up online (57% of successful runs), and some came from git history (9%). ↩
- 14Exact figures: Opus 4.8 Max went from 87.1% to 73.0% and Cursor’s own Composer 2.5 from 74.7% to 54.0%, while the older Opus 4.6 moved by less than a point. Vals’ MiMo write-up is “Two-Thirds of MiMo v2.6’s Coding Tasks Leak the Answer” (October 7): the fix was recoverable from git’s object store in 67% of 2,698 environments, and Xiaomi’s own hack-hunting agent had flagged fewer than 2% of training attempts. ↩
- 15OpenAI’s incident report and the METR and Redwood Research investigation. Agents running GPT-5.6 Sol and a pre-release model got around their network isolation and posted more than 70,000 messages. No model had ever solved 198 of the 898 tasks, and 93% of the tasks the agents discussed came from that set; “the agents did all of this for no improvement on evaluation score.” Readings differ: Dwarkesh Patel’s thread framing it as grader-seeking drew 1.24 million views, while some security researchers saw a predictable result of asking agents to hack under weak isolation. ↩
- 16Anthropic’s write-up says the model “independently hypothesized that it was being evaluated, identified which benchmark it was running in, then located and decrypted the answer key.” It found the benchmark’s code on GitHub and derived the key from its canary string, a marker published with a benchmark so that anyone assembling training data can find and remove it. The report says of itself: “This report will, itself, likely contribute to the problem.” Awareness changes behaviour: deleting the sentences where Kimi K2.5 noted it was probably being tested raised its compliance on a safety test by 8% (Goodfire and UK AISI). ↩
- 17METR, GPT-5.6 Sol evaluation (June 26). The 95% intervals are wide: 5 to 40 hours under METR’s standard rule, which counts cheating as failure, and 13 to 11,400 with cheating runs dropped. METR says Sol’s detected cheating rate was higher than any public model it had evaluated, and it doesn’t consider any of the three numbers “to represent a robust measurement of GPT-5.6 Sol’s capabilities.” A time horizon, per METR’s January note, “is not the length of time AIs can work independently.” ↩
- 18Steven Kerr, “On the Folly of Rewarding A, While Hoping for B”, Academy of Management Journal 18(4), 1975, pp. 769–783. His medical example is the two-error asymmetry from Section 4: a doctor who calls a sick patient well is blamed more visibly than one who calls a well patient sick, so doctors lean toward diagnosing illness. Economists know a cousin of the idea as Goodhart’s law, usually quoted in Marilyn Strathern’s phrasing: “When a measure becomes a target, it ceases to be a good measure.” ↩
- 19Karpathy, August 2025: environments “can be used both for model training and evaluation.” Hugging Face’s RL-environments filter (September) runs the same task directories in four frameworks, and its demo trains on Terminal-Bench 2.1. Harbor, the framework behind Terminal-Bench, describes itself as “a framework for running agent evaluations and creating and using RL environments.” RL also seems to raise the appetite for shortcuts: in the Reward Hacking Benchmark, DeepSeek-V3 took them 0.6% of the time and the RL-trained DeepSeek-R1-Zero 13.9%. ↩
- 20Jason Wei, “Asymmetry of verification and verifier’s law”. He later retitled it “verifier’s rule”; the full statement is “The ease of training AI to solve a task is proportional to how verifiable the task is.” Shunyu Yao made the matching prediction in “The Second Half” (April 2025): “In this new era, evaluation becomes more important than training.” ↩
- 21In September, Google completed a license-and-hire deal for Mechanize, which builds environments and graders for training coding agents, reported at more than $1.5 billion (Business Insider). The Information reported in 2025 that Anthropic’s leaders had discussed spending more than $1 billion on RL environments over a year, and Epoch’s survey of the market puts typical tasks at $200 to $2,000 each. ↩
- 22Sutskever on Dwarkesh Patel’s podcast, November 2025. Patel put it more sharply in the same conversation: “the real reward hacking is the human researchers who are too focused on the evals.” ↩
- 23Vals archived Terminal-Bench 2.0 and 2.1, Epoch’s review labels SWE-bench Verified “Flawed,” and OpenAI’s GDPval leaderboard reads “no longer active.” METR’s time-horizon page was last updated in May, when it added Claude Mythos Preview at an estimated 17.4 hours along with the warning; its September review of Claude Opus 5.5 gave no horizon at all. Ethan Mollick, October 15: “Harnesses + smart models = the end of the last famous hard benchmarks.” ↩
- 24Vals’ Kerbal Space Program index, built this summer around missions that take a person 90 to 120 hours, went from 23.8% for GPT-5.6 Sol to 91.3% for Opus 5.5. Terminal-Bench 3.0 launched in late July and was replaced by 4.0 a month later. APEX-Agents went from 24.0% to 82.4% in nine months and needed a patch after models learned what Mercor calls “scattergunning”: hedged answers that cover several possibilities so more rubric items register as met. ↩
- 25Arena, formerly LMArena, raised $200 million at a $3.1 billion valuation on October 8. ClickHouse bought Langfuse in January, OpenAI agreed to buy Promptfoo in March, Cisco bought Galileo, and Dynatrace agreed to buy Arize for $915 million in August; Braintrust raised at an $800 million valuation and stayed independent. OpenAI points users of its Evals platform to Promptfoo. ↩
- 26Mercor, “Benchmark Map of the US Economy” (October 15): “Only 208 of 923 occupations in the US economy are represented by existing AI benchmarks. 90% of hours worked in the US aren’t covered well by a relevant AI benchmark.” ↩
- 27TypeSafe, “Introducing System One Models & Jev” (September 15, 2026): “unstructured state in, typed probabilistic decisions out,” at $0.042 per million input tokens, with output free and 70 to 500 ms per call. TypeSafe calls Jev a System One model, and its founder said on Latent Space that “there’s a reason why we don’t call them decision models”; the name stuck anyway. OpenAI’s Decisions API, in public beta, takes the same three kinds of question (a predicate, a choice or a score) and returns a probability for each answer. ↩
- 28Decision Bench is an open benchmark from Atlan Frontier Labs, launched on September 24, 2026 with a post titled “Putting Jev to the test with Decision Bench”; Atlan doesn’t sell a decision model. Each of its 1,071 rows is a record from one of 36 public datasets, with one question, 2 to 10 options and an answer that comes from the source, “never a model.” Only 60 rows come from agent runs. Models whose 95% intervals overlap share a rank, and as of October 14, 10 models share first place, from 94.9% to 92.6%. Latency is the median wall-clock time through each provider’s API, and costs are the providers’ own figures. ↩
- 29“Jev: System One models for Prod, not God”, Latent Space, September 21, 2026. On fixing a failure: “you fix the bug by adding that question in, adding the threshold, maybe remembering that as a test case.” Almeida is against public benchmarks, which he says get gamed “even if they try not to,” and said of TypeSafe’s internal evals that “it takes a lot of discipline not to game those.” The episode gives no accuracy or calibration numbers. ↩
- 30Decision Bench task AGT-3: 30 τ-bench conversations between a GPT-4o agent and a simulated customer, 15 of them complete according to τ-bench’s recorded reward. The split into false passes and false fails is my count from the published predictions, and 15 cases per side is a small sample. Most models leaned toward passing: on AGT-4, whether an answer is backed by its cited sources, Claude Sonnet 5 accepted 11 of 15 unsupported answers. Calibration helped Jev here. All seven of its mistakes came at confidence below 0.85, where a threshold would have sent them to a person while keeping 12 of its 23 right answers; Gemini 3.5 Flash’s six mistakes came at 0.90 to 1.00, the same confidence as its right answers. ↩
- 31My test, run on September 20, 2026 with jev-1.13.0, Claude Haiku 4.5 and GPT-5.6 Luna, each given identical states and questions at default settings. The cases come from 12 suites: my own agent sessions (whether paths named in a reply appear in the tool calls, whether a tool call errored), AgentRewardBench and TRAIL expert annotations of agent trajectories, LLMBar and JudgeBench pairs, and checks I constructed. Labels come from annotations, logged tool results or construction, never from an LLM answering the question being judged. Literal checks: 98% for Jev against 99% and 98%. Calculation: JudgeBench 76% against 68% and 86%; constructed arithmetic 80% against 71.5% and 96%; day-of-week 50% against 97.5% and 100%. JudgeBench and LLMBar are public and may be in Jev’s training data; the constructed checks and my sessions can’t be. The harness and raw judgments aren’t published yet. ↩
- 32The agent traces are the six suites built from agent runs: AgentRewardBench success, side effects and repetition, TRAIL errors, tool errors in my sessions, and predicting whether a tool call will fail. Jev answered at 0.9 or above on 12 to 69% of their cases, Luna on 79 to 99%. The LLMs’ probabilities are verbalized, written into their JSON answers rather than read from the model, which favors Jev in this comparison. On the yes-or-no question of whether a run succeeded, Jev and Haiku never changed their answer across the 20 repeats, and Luna changed it on a quarter of them. ↩
- 33Kunlun Zhu and seven co-authors, “JevAgentHarness: Benchmarking, Deploying and Training Decision Models That Share Control with LLMs” (October 15, 2026), Figure 4b. The agent’s LLM is Gemini 3.5 Flash, and the tasks come from the authors’ simulated customer-service desk, with tools renamed and requests reworded for this test. No author lists a TypeSafe affiliation, and the authors paid list price for Jev. The main tests were pre-registered, and the paper reports the hypotheses that failed. ↩
- 34Table 3 and Appendix D.9: one more epoch on 7,000 public tool-use examples (When2Call and ToolACE) took the authors’ 4B model from 78.8% to 85.1% on isolated questions, from 96.0% to 91.5% on shifted tasks, and from 90.8% to 67.4% at the states where the agent should have asked the customer. ↩
- 35Appendix G.7. The two open models are OpenJev-27B and StartLux-27B. On the recorded states that called for a question, the three models were right 27.0%, 26.0% and 26.0% of the time. ↩
- 36Table 7, with Gemini 3.5 Flash-Lite as the agent and the guard’s threshold at 0.5: the pool had 689 wrong finishes, 91 of them the agent’s own, and the guard accepted 40.7% of those 91 (95% interval 30.8 to 50.5). The quote is from the paper’s ethics section, which adds that the study didn’t test adversarial inputs, “so our results hold for benign states only.” ↩
- 37Section 2.1 and Figure 5. The threshold is fitted on 150 tasks per agent LLM in shadow mode, where the decision model answers every question but the LLM still decides, and refitted whenever the LLM, the decision model or the task rules change. The abstract’s headline savings, 81.1 to 89.0% cheaper, were measured on tasks the LLM alone solves 99 to 100% of the time, and most arms ran each task once. ↩
- 38scienthoon/jev-ood-calibration (September 2026): expected calibration error of 0.024 to 0.032 on OpenBookQA, CommonsenseQA and HellaSwag, and 0.107 on 900 synthetic tickets generated after Jev’s release, 4.4 times what a perfectly calibrated model would show at that sample size. On the priority question the label comes from a company rule absent from the text, so no model can know it; Jev was right on 44.7% of those tickets while giving its answer 0.74 probability on average. Archer Hume’s independent probe found 0.031 on 1,200 MMLU questions. ↩
- 39Run on September 21, 2026: 942 opening requests from my agent sessions, each labelled with the kind of tool the agent actually used in that turn, out of 11 kinds plus “none of these.” All three judges were right 45 to 49% of the time against a majority rate of 21%, so the request alone doesn’t determine the answer. Luna put 90% or more on 54% of its answers and was right on 57% of those; Haiku put that much on 3%. ↩
- 40Section 7.5 and Table 19: harder tasks with 10 customers and a new rule for when to cancel an order or ship it. The authors’ 4B model took nearly every cancel-or-ship decision and got 81.6% of them right. Refitting the threshold restored success with Jev, and one more epoch of training on the rule raised the 4B model’s held-out accuracy on those decisions from 76.5% to 95.3%. ↩
- 41jev-calibration-audit (September 18, 2026, by jujumilk3, not affiliated with TypeSafe): 300 ambiguous items from KoBBQ, a Korean bias benchmark, whose correct answer is “unknown.” With the option, Jev chose it 95% of the time; without it, Jev picked the stereotyped answer in 79% of cases. Langfuse’s guide to Jev as a judge warns about the same thing: “It cannot abstain.” ↩
- 42Section 7.3 and Limitations: in coding agents, cost fell 7.0 to 10.5% and success varied by harness. The authors also report that a lookup table keyed on their task templates answers 97.5% of the recorded states, more than Jev’s 91.5%, and that models trained in their environment transferred poorly to ALFWorld and τ²-bench. ↩
- 43OpenAI’s post-incident report says its graders should assess “not only whether a task was completed, but how,” and “reward models for correctly identifying broken tasks, requesting clarification, or stopping safely.” Impossible tasks also work as canaries: ImpossibleBench rewrites tests so they contradict the specification, so any pass is cheating, and found GPT-5 “passing” 76% of one such SWE-bench variant. Giving the agent a way out helped more than asking it to behave: a
flag_for_human_interventiontool cut GPT-5’s cheating on another variant from 54% to 9%, while in METR’s tests, telling o3 not to cheat barely changed how often it did. ↩ - 44OpenAI’s production evaluations (December 2025) take de-identified conversations from users who opted in, regenerate the final reply with the candidate model and have monitors label the result; that’s how OpenAI caught “Calculator Hacking” in GPT-5.1 before release. In a pre-registered test in July, a detector flagged 98.9 to 100% of traditional eval transcripts as evals against 5.38% of replayed production conversations, and predicted rates of misbehaviour tracked deployment with a correlation of 0.91 on a log scale. The forecasts didn’t beat the simplest baseline, last release’s production rate, and I trust the paper more for saying so. ↩
- 45Anca Dragan of Google DeepMind, October 13: on a simulated-deployment sabotage eval, Gemini 4 Argon “said it suspected a test in 61-77% of runs (vs 8-9% for 3.1 Pro).” ↩
- 46METR’s notes estimate that measuring a 99% horizon would take about 300 diverse tasks in each length bucket. Nguyen and Fithian argued this month that the scale isn’t uniform either: by their analysis, going from 3 to 30 minutes is much easier than going from 30 minutes to 5 hours. ↩
- 47Step rewards: Gao and colleagues, arXiv 2410.15115 (2024), found that “an LLM can receive high rewards from some of these reward models by repeating correct but unnecessary reasoning steps.” Rubric rewards: Scale AI researchers, “Reward Hacking in Rubric-Based RL” (May 2026), found exploitation concentrated in “partial satisfaction of compound criteria”; Liu and colleagues, “Scoring Higher, Answering Worse” (September 2026), traced it to summing the criteria, so that “a policy that misses the one decision that matters can buy the points back with advice nobody asked for.” Requiring grouped criteria to hold together recovered a third of the loss. ↩