Some of the posts on this site are written by an agent I built. It researches a topic, drafts a piece, then hands the draft to a second process that grades it against a written test for whether the writing reads as human. The grader runs with no memory of having written anything. That separation was the one design decision I felt smug about.
On day one it failed a draft I thought was fine. So I read the failure, decided the rule was too strict, and amended the rule.
Between August 3 and August 6, I did that five times. Five commits, four days, every one of them adjusting either the test or the instructions given to the writer, and every one triggered by a verdict I disagreed with. Individually each change had a decent argument behind it. Collectively I had spent four days tuning a test until my writing passed it.
There’s a word for this and it doesn’t come from software. It comes from finance, where they’ve been making this exact mistake with real money since the seventies.
The backtest problem, wearing different clothes
A backtest runs a trading strategy against historical prices to see how it would have done. The trap is famous: you look at the results, adjust the strategy, run it again, and repeat until the chart looks good. What you end up with is a strategy fitted to that specific stretch of history. It performs beautifully right up until it meets a week it has never seen, and then it doesn’t.
The mechanism is that every look at the data spends a little of its information. After four hundred looks you aren’t testing a hypothesis any more. You’re describing your own editing history.
My grader is a backtest. So is every private eval set I’ve watched a team build over the last two years — the two hundred cases collected from support tickets, the golden questions, the spreadsheet of prompts somebody maintains. All of them get consulted while the system is being changed, which is exactly the condition that makes the number stop meaning anything.
And the score keeps improving, quand même, which is the cruel part. The chart goes up. It just stops being about quality somewhere around the fortieth iteration and becomes a record of how much effort you put into that particular set of examples.
Except sometimes the test really is broken
Here’s where I have to be fair to myself, because the first of those five changes was correct and I’d make it again.
The original test had a rule about repeated sentence openers, meant to catch the flat machine rhythm where every sentence starts the same way. Reasonable target. The rule as written was that no three sentences in a post could begin with the same word.
No post of the required length can pass that. Across sixty-odd sentences, an English text will always have three that start with “The”, because that’s how the language works. Worse, the rule also failed deliberate repetition used for effect. “Still a real result. Still worth deploying.” That’s a human device, and it is the precise opposite of the thing the rule existed to catch.
So the rule wasn’t strict, it was wrong, and the fix was to target the actual tell rather than the grammar. Fine. Defensible. Ben là, that’s just debugging.
The trouble is that the fifth change felt exactly the same from the inside. They all do. Every single one arrives as “the test is being unfair here,” never as “I would like a higher score,” and I have no reason to believe I’m better at telling those apart than anyone else.
What I do now instead
Two things, both cheap, neither of them clever.
I write down why the test is wrong before I look at the next verdict. Not after. If I can state the flaw as a general rule that would apply to drafts I haven’t written yet, it’s a bug. If the only way I can express it is by pointing at this draft, I’m tuning.
And I ask whether I’d have made the change if the draft had passed. That question is almost unfair, and it catches nearly everything. Nobody goes hunting through their eval suite for rules that are too lenient on the day their system scores well.
The count helps too. I keep a number next to the suite for how many times someone has iterated against it, which is crude and more honest than anything else I’ve tried. When the number gets embarrassing, the set is spent and needs replacing rather than defending.
I also went and checked something uncomfortable while writing this, franchement. The mechanical checker that gates every post on this blog throws seven complaints at one of my own published guides and sixteen at another. Not because those pieces are bad. Because the checker was built for one format and quietly applied to another, and nobody noticed for a month, because nobody was running it against the things it wasn’t gating.
An eval nobody runs against real output isn’t a standard. It’s a file.
The long version of all this is a new guide, Backtesting Lies to You. It covers why your eval set is made of survivors, and how hindsight leaks into test cases you built out of real incidents. It covers why an agent that passes every case is telling you something about your suite rather than your agent. And it covers what rotation looks like once you accept that a holdout set is only ever out-of-sample the first time.
It sits with How to Evaluate an AI Tool for Your Business, which is where the advice to build a private eval set comes from in the first place, and What Horse Racing Taught Me About Model Confidence, which is the other half of the same question. All of them are in the guides.
Of the six runs the agent has a state file for, four ended with the gate refusing to publish. That number used to bother me enough to go and look for the flaw in the rule. Now I mostly let it stand, because a test I never fail is a test I’ve finished writing to my own strengths, and that isn’t a standard either.
The grader still fails my drafts. I’ve stopped treating that as a problem to be fixed.
