Years ago I built betting algorithms for horse racing, and the first thing that world teaches you is that the odds on the board are not a fact. They’re a summary of what everyone has bet so far, and the moment you place your own bet, they move. Bet big enough on a horse and you shorten its price yourself. You get paid less precisely because you were right.
Which makes testing a strategy genuinely awkward. You can’t take half your bankroll, run the new system on it, leave the other half on the old one, and compare. The first half changed the odds the second half was betting into. Both results are now measurements of a race that only existed because you were in it.
I’ve been watching engineering teams walk into that exact problem for two years without recognising it, because in their version it isn’t called parimutuel anything. It’s called an A/B test of an AI agent.
The test that’s broken before it runs
The setup is always sensible-sounding. You’ve deployed an agent on something that matters. Somebody asks whether it’s actually working, and the answer everyone reaches for is to split the population. Half the accounts get the agent, half don’t, compare after a quarter.
Every A/B test rests on an assumption that one group’s outcome doesn’t depend on what happened to the other group. For a green button versus a blue button, fine. For an agent, usually not, because agents don’t act on isolated units. They act on a pool that everyone is drawing from.
Say the agent manages half your campaigns and bids more aggressively. That money comes out of the same account budget the control campaigns are spending from, so the control gets starved. Your agent now looks good partly because you defunded its comparison group. If it’s bidding into the same auction the control competes in, worse: it raises the price the control pays. The control’s numbers degrade because of the treatment, not because of its own absence.
Now flip it. An agent that clears a support queue faster frees up humans, who then get to control-group tickets sooner than they otherwise would. The control improves because of the treatment. That bias runs toward finding nothing, so you conclude the agent is useless and switch off something that was working.
Two directions, same broken assumption. You can’t even guess the sign of the error without knowing which resource is shared, which means “we ran an A/B test” is not on its own an answer to anything.
There’s a second thing stacked underneath, and it showed up fast.
The old way to measure a tool was to compare people who use it with people who don’t. That was fine when hardly anybody used it. It’s much worse now that nearly everyone does, because whoever is left has selected themselves. They’re the ones who refused, or never got access, or work on the one system it doesn’t support. Those people differ from adopters in ways you can’t see, and therefore can’t correct for.
So the clean control group is evaporating, quand même, and it isn’t coming back.
What actually works, roughly
None of this is unsolved. The answers just live somewhere most agent teams haven’t read. Marketplace companies hit this wall a decade ago, because a delivery or rideshare network is nothing but shared state, and they wrote up their answers in public.
The main one is the switchback: stop splitting the population and split time instead. Agent on for everyone for an hour, off for everyone the next, randomise which is which. One state at a time means there’s no simultaneous contamination for the shared resource to carry.
The one I’d actually push, though, costs almost nothing and nobody does it. Deliberately randomise a small slice of your agent’s decisions and write the dice roll into the run record — not just what it did, but how likely it was to do that. Five percent down a different path is enough. That single extra field buys you a permanent ability to ask counterfactual questions from the logs, using methods that already exist for exactly this. Most agent deployments record the action and not the probability, which is what makes their logs useless for the only question anyone cares about later.
Where I’d argue against myself
Switchbacks are operationally miserable. Turning an autonomous system off and on for a fortnight is disruptive, and there is always someone who wants to skip a scheduled off-period because the quarter is closing. That pressure arrives at precisely the moment the design starts producing garbage, and I’ve never seen an experiment protocol win that argument on its own.
Geo tests dodge that but they’re weak. You get as many independent units as you have regions, which is not many, and regions are noisy relative to each other. If your agent’s real effect is a couple of percent, a geo test won’t find it, and running one for a quarter to learn nothing is an expensive way to feel rigorous.
And the uncomfortable one. Sometimes the answer is that you cannot measure this at your scale. Randall Lewis and Justin Rao made this case for advertising in a 2015 QJE paper, and the shape of it is brutal: when the per-unit effect is small and the outcome is noisy, the sample you’d need for an informative answer is far bigger than anything you’ll ever run. Agent deployments have that shape too. A lot of confident ROI numbers are noise wearing a suit.
That sounds defeatist and I don’t think it is. Knowing a question is unanswerable at your scale is a real finding, and it’s a much better position than a number you half-believe. It moves the decision onto different ground — cost, reversibility, whether the thing is pleasant to operate — instead of onto a fake measurement everyone quietly distrusts.
The long version is a new guide, How to Hold Out an Agent. It lays out the interference mechanisms one by one, prices what switchback and geo designs each cost you, and ends with the four things none of them can identify. It sits next to Backtesting Lies to You, which is about fooling yourself with data you’ve already tuned against, and Shadow Mode, which is about deciding to deploy in the first place.
The betting version of this lesson took me a lot longer to learn than it should have, and cost real money in the process. The engineering version is cheaper. You just have to write down what your agent is sharing with everything it isn’t running on.
