Somebody shows you a chart where the line goes up every quarter. The eval score was 71, then 78, then 84, and the team is pleased because the work is obviously paying off. Ask one question and the room changes: how many times did you look at this set while you were making those changes?
Usually the answer is hundreds. Which means the chart is not measuring the system getting better. It is measuring the system getting fitted to that particular set of examples, and those are different things that produce identical graphs.
Quantitative finance spent a couple of decades learning this expensively, under the name backtesting. You have a strategy, you run it against historical prices, and the returns look wonderful. Then you trade it with real money and it does not work, and the postmortem always finds the same handful of causes. Agent teams are now rediscovering every one of them, with eval sets instead of price series, mostly without knowing the failure has a literature.
The single sentence version: a backtest is a hypothesis test you have already peeked at. Everything below follows from that.
Your eval set is made of survivors
Where did the cases come from? Almost always the same place. Something went wrong, somebody noticed, somebody was annoyed enough to write it down, and it became a test case.
Every one of those steps is a filter. Failures nobody noticed are not in your set. Failures that were noticed but looked like user error are not in your set either. And whatever goes wrong in the workflows your quietest customers use is nowhere near it. You have assembled a collection of detectable, memorable, complained-about problems and you are treating it as a sample of reality.
Finance calls this survivorship, and the classic version is measuring a strategy against the companies that exist today, which quietly excludes everyone who went bankrupt. The bankruptcies were the whole risk. The agent version excludes silent failures, and those are the ones that cost you a customer without ever producing a support ticket.
This does not make the set useless. It makes it a floor rather than an estimate. Passing every case means you have fixed the problems you know how to see.
Leakage, and why it got worse
Look-ahead bias is using information at decision time that was not available at decision time. In a trading backtest it is subtle and lethal: a closing price that gets revised, a restated earnings figure, an index membership list as it looks now rather than as it looked then. The strategy appears to predict the future because you handed it the future.
The obvious modern version is that public benchmarks end up in training data, so a model can score well on a test it has effectively already seen. This is understood well enough that most teams have stopped trusting public benchmarks alone, which is the right instinct and roughly where How to Evaluate an AI Tool for Your Business leaves the argument.
The version that gets people is internal, and a private set does not protect you from it. You built a test case out of a production incident. You already know what the right answer was, because you saw how it resolved. So you write the expected output with hindsight baked in, encoding a judgement that nobody could have made at the moment of the decision. Now your agent is graded on whether it can be as smart as someone who already knows the ending.
The check is mechanical and slightly tedious. For each case, ask what was actually knowable at the moment the decision had to be made. Anything in your expected answer that depends on knowing the outcome is leakage, and it is inflating your score in the direction you would least like.
You can overfit by hand
Overfitting sounds like something that happens inside a training run, which makes people think they are safe from it when the only thing they are changing is a prompt. They are not safe from it. A person editing a system prompt, running the suite, reading the failures and editing again is performing gradient descent slowly, with worse instrumentation.
Twenty iterations against two hundred cases and you have fit those two hundred cases. The prompt now contains clauses that exist because of case 147 and would not survive contact with a case you have never seen. Nobody wrote “handle the situation from case 147” on purpose, and that is precisely what makes it hard to notice.
Statistics has a name for the underlying problem, multiple comparisons, and the useful way to hold it is that every look at a set spends a little of the information in it. A set you have consulted three times is close to honest. A set you have consulted four hundred times is a description of your own editing history.
Sets rot, and nobody retires them
An eval set has a half-life. Your product changes shape, your users bring different requests, the underlying model gets updated by a provider on a schedule you do not control. Cases written in March are answering a question from March.
Retiring cases feels like deleting coverage, so it does not happen, and suites accumulate. Some of the old cases are still load-bearing. Others now test behaviour you deliberately changed, and they either fail forever and get muted, or they pass for reasons unrelated to why they were written.
Muted tests are worth a specific mention because they are where suites go to die quietly. A case that has been failing since June and is on somebody’s list is not coverage. It is a comment.
The equity curve with no drawdown
Traders develop a specific reflex about backtests that look too good. A returns curve that rises smoothly with no losing stretches does not mean the strategy is excellent. It nearly always means there is a bug, usually a leak, and the experienced response to a beautiful backtest is suspicion rather than delight.
Borrow the reflex. If your agent passes every case in your suite, the finding is about your suite. Either it is too easy, or it has been fit, or the hard cases were never collected because hard cases are hard to write down. A suite worth having should fail sometimes, and you should be able to name which cases you expect to fail and why.
The related habit is refusing to celebrate a number without its denominator. “We fixed twelve of the fifteen failures” is a fact about fifteen examples and carries no information about the failures that were never in the set. That distinction is the whole argument in Why LLMs Are Bad at Big Data, arriving from a different direction.
What actually helps
Hold a set back and do not look at it. Not “look at it rarely.” Split your cases when you build them, put a portion somewhere inconvenient, and consult it when you are deciding whether to ship rather than while you are working. Its value is entirely a function of your not having seen it, and it is spent the first time you tune against it.
Write down what a pass means before you run anything. The threshold decided afterwards is not a threshold, it is a description of the result you got, and the pre-registration discipline is the same one that makes Shadow Mode work rather than being theatre.
Date every case and record where it came from: a real incident, a synthetic construction, a customer complaint, a hypothetical someone invented in a planning meeting. Provenance tells you what a case is evidence of, and a set where half the entries were invented in a room is a different instrument from one built out of production.
Count your looks. A single integer next to the suite, incremented whenever someone iterates against it, is crude and more honest than anything else on this list. When that number gets embarrassing, the set is done and needs replacing.
And prefer evidence collected after the change was designed. A case that already existed when you wrote the fix tells you the fix addresses a case you were looking at. A case that arrived afterwards tells you something about the world.
The ceiling, stated honestly
Do all of this properly and the problem does not disappear, it only slows down. A holdout set is out-of-sample exactly once. The moment you make a decision from it, it has entered your process, and the second decision you make from it is contaminated by the first.
So the realistic posture is rotation rather than purity. Sets get built, spent and replaced, the way a lab retires a reagent. Budget for producing new cases continuously instead of treating the suite as an asset that appreciates, because it does the opposite.
And treat every eval number as a lower bound on your uncertainty rather than a measurement of quality. It tells you how the system behaves on problems you already thought of. What it cannot tell you, ever, is how the system behaves on the problem that takes you down, because if you had thought of that one it would already be a case.
Start this week
Take your current eval suite and answer three questions about it in writing. Where did each case come from. When was it written. How many times has someone tuned against it.
Most teams cannot answer any of the three, and the exercise of not being able to is more persuasive than any argument here. Then pull twenty per cent of the cases out, put them somewhere annoying to reach, and do not look at them until the next ship decision.
Evaluation sits next to calibration, which is the question of whether a system’s confidence means anything, covered in What Horse Racing Taught Me About Model Confidence. The method for choosing a tool in the first place is How to Evaluate an AI Tool for Your Business. And recorded production incidents make excellent test fixtures, with one caveat that belongs here rather than there: a suite grown entirely from incidents you caught is a survivorship sample too, which is the honest footnote on Deterministic Replay for Non-Deterministic Systems.