The agent has been running for a quarter. Throughput is up, the team likes it, and someone senior asks the only question that matters: did the agent do that?
The honest answer in most organisations is that nobody knows. There’s a number that moved and a change that shipped in roughly the same window, and the causal story connecting them was assembled afterwards by people who wanted it to be true. That isn’t measurement. It’s a narrative with a chart attached.
So you propose the obvious fix. Split the population, give half the agent, withhold it from the other half, compare. Clean, familiar, the thing you’d do for a pricing page or a checkout flow.
For an autonomous agent that test is usually invalid before you run it, and the reason is structural rather than a matter of sample size or duration. No amount of patience fixes it.
The assumption that quietly breaks
Every A/B test rests on an assumption with an ugly name: SUTVA, the stable unit treatment value assumption. Stripped of the jargon it says one thing. What happens to unit A must not depend on whether unit B got the treatment.
That holds for a checkout button. Showing me a green button does not change your experience of the blue one. It does not hold when the treatment is an agent, because agents almost never act on an isolated unit. They act on a pool of something that everyone is drawing from.
When SUTVA breaks, the control group stops being a measurement of what would have happened without the agent. It becomes a measurement of what happens to the people who didn’t get the agent while other people did — a different quantity, and one nobody wants.
What interference looks like when the treatment is an agent
A shared budget. The agent manages half your campaigns and bids more aggressively. That spend comes out of the same account budget the control campaigns draw from, so the control is starved by the treatment. The agent looks good partly because its comparison group was quietly defunded.
A shared auction. Worse than the budget case and more common than people think. If the agent raises bids in an auction the control group also competes in, it raises the price the control pays. In a parimutuel-flavoured market your own action moves the price you get, and the control’s degraded performance is caused by the treatment rather than by its own absence.
A shared queue. The agent clears tickets faster, which frees the humans, who then pick up control-group tickets sooner than they otherwise would. The control improves because of the treatment. This one is insidious because it biases toward finding no effect, so you conclude the agent did nothing and switch it off.
Shared inventory. The agent reprices its SKUs and sells more of them. There is now less stock, less warehouse capacity and less fulfilment headroom for everything else.
A shared model. The agent’s actions become tomorrow’s training data or tomorrow’s retrieved context. Treatment and control are reading from a corpus that the treatment is actively writing.
Notice that these push in different directions. Budget and auction interference flatter the agent; queue interference buries it. You cannot even sign the bias without knowing which resource is shared, which is why “we ran an A/B test” is not by itself an answer to anything.
The control group is also disappearing
There’s a second problem stacked on the first, and it has arrived quickly.
The classic design for measuring a tool is to compare people who use it against people who don’t. That worked when adoption was low. It works badly when adoption is very high, because the remaining non-users are no longer a random sample of anything. They are the people who refused, or who never got access, or who work on the one system it doesn’t support.
Those people differ from adopters in ways you cannot see and therefore cannot adjust for. Comparing them gives you a number that is contaminated by selection before interference even gets a chance. The population that would make a clean control group is evaporating, and it is not coming back.
Randomise time instead of units
The fix is not new, it just lives in a literature most agent teams have never read. Marketplace companies hit this wall a decade ago, because a rideshare or delivery network is nothing but shared state, and they cannot randomise riders without the riders competing for the same drivers. DoorDash and Lyft both published extensively on how they solved it.
The answer is the switchback. Rather than splitting the population, you split time. The agent is on for everyone for an hour, off for everyone for the next hour, and the assignment of which hour is which is randomised. Because the whole system is in one state at a time, there is no simultaneous contamination for the shared resource to transmit.
You pay for this in three ways, and they should be priced before you start.
Carryover is the first. Effects do not stop cleanly at the period boundary — a bid change keeps influencing delivery for a while after you revert it — so periods must be longer than the settling time of whatever you changed, and you usually need to discard a burn-in window at the start of each one.
The second is statistical. Consecutive periods are correlated, so your effective sample size is much smaller than the number of periods, and the naive standard error will lie to you cheerfully. The unit of randomisation is the period, not the event inside it, and the analysis has to respect that.
Third, switchbacks are operationally ugly. Turning an autonomous system off and on for weeks is disruptive, and there is always someone who wants to skip a scheduled off-period because the quarter is ending. That pressure is exactly when the design starts producing garbage.
Randomise geography instead
The other borrowed design comes from advertising, where geo experiments are standard practice. Split by market, run the agent in some regions and not others, compare regions rather than accounts.
Geo works when the shared resource is separable by geography and fails when it is not. Inventory in regional warehouses is often separable. A single national account budget is not, and neither is a global auction, so check that before you design around it.
The real cost of geo designs is power. You have as many independent units as you have regions, which is a small number, and regions differ from each other in ways that add variance. Small effects are simply undetectable this way. If your agent’s honest impact is a couple of percent, a geo test will not find it and you should know that before you spend a quarter running one.
Log the dice roll
The most useful idea here is also the least used, and it comes from the bandit literature rather than from experimentation.
Off-policy evaluation answers exactly the question you’re asking: what would have happened under a different decision policy? Given a log of decisions, it estimates the value of a policy you did not run, without running it. That is the formal version of the counterfactual everyone is guessing at in the incident review.
It comes with a real precondition. The system you logged has to have had some randomness in it, and you have to have recorded the probability with which each action was taken. Most agent deployments record what the agent did and not how likely it was to do that, which makes the log useless for this purpose.
Which suggests something worth doing on purpose. Deliberately randomise a small fraction of the agent’s decisions and record the probabilities. Send five percent of eligible decisions down an alternative path, or to a human, or to a deliberately more conservative policy, and write the dice roll into the run record alongside the outcome. You are buying a permanent ability to answer counterfactual questions for a small, bounded cost — and unlike a switchback, it runs continuously and does not require anyone to turn the system off. The recording discipline this needs is the one described in Deterministic Replay for Non-Deterministic Systems, with one extra field.
What none of this identifies
Equilibrium effects. An experiment tells you what happens when the agent runs on part of the system or part of the time. It does not tell you what happens when it runs everywhere, forever, and competitors adapt. Partial-deployment effects and full-deployment effects can differ in sign.
Anything slower than your window. Trust, brand, customer lifetime, the gradual erosion of a skill nobody practises any more. Two-week experiments cannot see any of it.
Changes to the humans. The agent’s largest effect is often that people stop checking. That shows up as a quiet rise in the cost of the failures it does produce, months later, and it is not in the metric you pre-registered.
Effects too small for the noise. This is the hard ceiling. Randall Lewis and Justin Rao made the case in advertising, in a 2015 Quarterly Journal of Economics paper on the economics of measuring returns to advertising: when per-unit effects are small and outcomes are wildly variable, the sample sizes required for an informative answer are far beyond what anyone will run. Agent deployments have the same shape. A great many confident ROI figures are underpowered noise dressed as a result, and knowing that a question is unanswerable at your scale is a legitimate and underrated finding.
Start this week
Write down what your agent shares with everything it isn’t running on. Budget, queue, auction, inventory, rate limit, training corpus. That list is the whole problem, and most teams have never written it down.
If the list is empty, run the ordinary A/B test with a clear conscience. If it isn’t, pick your design from what’s on it: switchback when the resource is global, geo when it separates cleanly, and randomised logging underneath either one because it costs almost nothing and keeps paying.
Then pre-register the effect size that would make you keep the agent, before you look at anything. A threshold chosen afterwards is not a threshold, it is a description of your result, which is the same discipline that makes Shadow Mode work rather than being theatre.
Measuring whether a system’s decisions were good in the first place is a different question, covered in What Horse Racing Taught Me About Model Confidence. The trap of grading yourself against data you have already tuned against is in Backtesting Lies to You. And the broader question of what you are entitled to conclude before handing an agent authority runs through The Agent Reliability Handbook.