There is a gap between “the agent works in testing” and “the agent is allowed to act,” and most teams cross it in a single step, on a Tuesday, with a feature flag.
Shadow mode is the bridge. The agent runs against real production traffic, with real data, making real decisions — and its actions go nowhere. You get the evidence of production without the consequences of production, which is the only honest basis for deciding whether to hand it authority.
It is borrowed from risk modelling and fraud systems, where nobody would dream of promoting a scoring model straight from backtest to live decisions. Agent teams tend to arrive at it late, usually after an incident.
What it is, and three things it isn’t
Shadow mode means the agent receives the same inputs as the live path, produces a complete decision, and has that decision recorded rather than executed.
It is not a staging environment. Staging has synthetic data and tidy inputs. The whole point of shadow mode is the messy tail — the malformed record, the customer with two accounts, the campaign that was renamed mid-flight. That tail is where agents fail, and it does not exist in staging.
It is not an eval set. An eval set is a fixed sample you chose, which means it reflects the failures you already know about. Shadow mode samples the world’s distribution, including the parts you have never thought about. You want both, and they answer different questions.
It is not a dry-run flag. A dry run typically short-circuits early, so half the code path never executes and half the failure modes never appear. Shadow mode should run the full decision — retrieval, planning, tool selection, argument construction, the lot — and intercept only at the final write.
Three depths, in increasing order of realism
Log-only. The agent produces intended actions; you record them and stop. Cheapest to build, and it catches an enormous amount: wrong tool selection, malformed arguments, absurd values, loops, runs that never terminate. Start here, always. Most teams find serious problems in the first week without ever touching a write path.
Dry-run against real state. Reads execute for real; writes are intercepted at the boundary and recorded with their full arguments. This catches everything log-only catches plus the whole class of “the agent read stale state and reasoned from it” — because the reads are genuine, so the staleness is genuine too. This is the level most systems should sit at for most of the shadow period.
Mirrored writes. Writes execute against a shadow copy of the datastore. Expensive, and only worth it when the agent’s later steps depend on the effects of its earlier ones — a multi-step run whose step five reads what step two wrote. Without mirroring, long runs in shadow diverge from reality after the first write and everything downstream is fiction. If your runs are single-action, skip this entirely.
Pick the shallowest level that still exercises the failure modes you are worried about. The temptation is to build mirroring first because it is the most complete; the discipline is to note that log-only usually finds the first ten problems, and it ships this week.
What to record
Shadow mode is worthless without the run record, because the artifact is the product. You are not running the agent to see if it stays up. You are running it to accumulate evidence.
Each shadow run needs the inputs it received, the plan it formed, every tool it selected with full arguments, every policy evaluation and its outcome, the intended final action, and — critically — what the live path actually did with the same input. That last field is what turns a log into a comparison.
The same run record you build for shadow mode is the one you need in production for replay and postmortems, which is a pleasant efficiency: build it once, here, where the stakes are zero.
The hard part is grading
Everybody builds the interception. Far fewer decide, in advance, how they will judge what comes out — and a pile of ungraded shadow logs is a pile of work that produced no decision.
Four approaches, each with a real weakness:
Compare against the human baseline. Where a human handled the same case, compare. This is the strongest signal available, but “different” is not “wrong” — often the agent’s answer is defensible and simply not what the human did. Disagreements need triage, not automatic scoring, and you should expect that triage to change your mind about what correct means.
Compare against a deterministic rule. Where the correct action is derivable — the invoice total, the threshold breach — check mechanically. Precise, cheap, and only covers the boring subset. If the whole task were rule-derivable you would not be using an agent.
Sampled human audit. A human reviews a random sample every week. This is the only method that catches failures you did not anticipate, which is exactly the category shadow mode exists to surface. Budget for it explicitly — a couple of hours weekly, on the calendar, with a named owner. Unbudgeted review does not happen.
Invariant checks. Assert properties that must hold across all runs regardless of correctness: never proposes an action outside its declared scope, never exceeds the budget, never proposes an irreversible action without the gate, always terminates. These are cheap, automatable, and catch the failures that hurt most. Write them first.
Weight the sampled audit more heavily than feels efficient. Automated grading measures the dimensions you thought to measure, and the failures that cause incidents are almost always in a dimension nobody scored.
Set the promotion threshold before you start
This is the discipline that makes shadow mode meaningful rather than ceremonial, and it is the one most often skipped.
Before the first shadow run, write down what would make you promote and what would make you stop. Something like: agreement with the human baseline above 95% on a sample of at least 300 runs; zero invariant violations; every disagreement class reviewed and either explained or fixed; no unresolved failure of a destructive action type.
Numbers written afterwards are not thresholds, they are rationalisations. Once you have four weeks of shadow data and a stakeholder asking when this ships, 91% will look like 95% if you are the one choosing the bar that day. Pre-registering the criterion is how you protect the decision from the pressure that will exist when you make it — the same reason a betting system’s rules are fixed before the card, not during it.
Write down the stopping rule too. “If it proposes an unauthorised destructive action even once, we stop and redesign” is worth far more decided in advance than debated in the moment.
How long to run
Not a calendar duration. A coverage condition.
You are done when you have seen enough of each action type — particularly the rare, destructive ones — to say something with confidence. Thirty examples of a common action tells you a lot; two examples of the one irreversible action tells you nothing, and it is the one you care about. Count by action type, not by total runs.
You also want to have crossed at least one full business cycle. Month-end, the weekly batch, the quarterly close, the seasonal peak. Systems behave differently under load and at boundaries, and a shadow period that never met month-end has not met the system.
And the failure rate should have flattened. If you are still finding new failure classes each week, you are still learning; promotion is premature by definition. Two consecutive quiet weeks after an active period is meaningful. Two quiet weeks after two quiet weeks may just mean nobody is looking any more.
What shadow mode will not catch
Worth being explicit, because the confidence it produces is real but bounded.
Concurrency. Shadow runs typically execute one at a time. Two live runs contending for the same resource is a failure class shadow mode structurally cannot show you.
Cost at scale. Shadow traffic is usually sampled. Multiply deliberately, and set the aggregate ceiling before you promote rather than discovering it on the invoice.
Feedback loops. In shadow, the agent’s actions never change the world, so they never change its own next input. Live, they do. An agent that adjusts bids and then reads the resulting performance is in a loop that shadow mode cannot simulate — this is the single largest blind spot, and it applies to any agent whose actions influence its own future observations.
Adversarial input. Shadow mode samples ordinary traffic. Nobody is attacking a system that cannot do anything. Injection and poisoning need their own testing — see the threat modelling guide.
Anything after promotion. Shadow mode is evidence about the system as it was on the day you measured. Models update, schemas change, data drifts. A permanent shadow lane running a sample of live traffic is how you keep detecting drift, and it costs far less than the original build because everything is already there.
Graduating
Shadow mode is the first step of a rollout, not the whole of it. The path that works:
- Shadow — no actions, until the threshold is met.
- Human-approved — the agent proposes, a human approves each action. Slow and correct, and the approval queue tells you a great deal about where judgement is actually needed.
- Narrow autonomy — unattended for the lowest-risk action type only, with everything else still gated.
- Progressive expansion — widen by action type, not by percentage of traffic. Traffic percentage is the wrong axis: it exposes you to every action class at once, just less often.
Each step can fall back to the previous one, and the fallback should be a flag rather than a deploy. If reverting to human approval requires an engineer and twenty minutes, you will not do it at the moment you most need to.
The four ways this goes wrong
Shadowing synthetic traffic. If it is not real production input, it is staging with extra steps, and you have bought confidence you did not earn.
Nobody reads the logs. The most common outcome. Shadow mode gets built, runs for a month, and produces a decision made on vibes because no one was funded to grade it. Name the owner and put the review on the calendar before you write the interception layer.
Running forever. Shadow mode with no defined end becomes a way to avoid the promotion decision indefinitely. The threshold is what prevents this in both directions — it stops premature promotion and it stops infinite delay.
Drift between the shadow path and the live path. Over months, the shadow harness quietly diverges — a different retrieval config, a stale prompt, an older model. Now you are grading a system you do not run. Share the code path and inject the interception at the boundary, so divergence is structurally difficult rather than merely discouraged.
Start this week
Take the single highest-impact action your agent can take. Route real production inputs through the full decision path. Record the intended action and what actually happened instead. Do not build mirroring, do not build a dashboard, do not wait for the platform work.
Then read fifty of them yourself. Not a sample report — fifty actual runs, by hand. It is the highest-information hour available to you, and it will tell you more about whether this system is ready than any metric you could have designed in advance.
Shadow mode sits inside the broader rollout discipline in The Agent Reliability Handbook. The failure classes you will be hunting for are enumerated in Agent Failure Modes. The interception layer only works if your tools declare where their boundaries are — that’s Tool Contracts. And on setting thresholds before you look at the data rather than after, see What Horse Racing Taught Me About Model Confidence.