The incident review is Thursday. Three weeks ago the agent issued a refund it had no business issuing, and the only question on the agenda is why. You have logs. You have the refund. What you do not have is any way to put the system back into the state it was in at the moment it decided.
So the review becomes archaeology. Someone reads a log line aloud, someone else proposes a theory, and everyone agrees the theory is plausible. Plausible is where these meetings end, because plausible is all the evidence supports. Then the same class of failure happens again in November and nobody can prove it is the same class.
Deterministic replay is the way out, and the name contains an apparent contradiction worth resolving immediately. You cannot make the model deterministic. Sampling is stochastic, providers move endpoints underneath you, and even a temperature of zero is not bit-exact once batching and floating-point accumulation are involved. That is not a problem to be solved. It is the material you are working with.
What you can make deterministic is everything else. Every input, every tool response, every clock read, every retrieved document, every policy evaluation. Fix all of it and the model becomes the only free variable in the system — which is exactly the condition under which you can finally reason about what it did. Determinism lives on the input side. That is the whole idea.
Replay is not re-running
These get conflated constantly, and the difference decides whether your investigation works.
Re-running takes the original request and sends it through the live system again. Fresh tool calls, current data, today’s model. It answers a real question — does this fail now? — and it is the right thing to do when you are checking whether a fix landed.
Replay takes the original request and feeds the system the recorded tool responses instead of calling anything. Nothing is queried, nothing is hit. The agent sees exactly the world it saw in August, because that world is coming off disk.
Re-running a three-week-old incident tells you almost nothing. The data has changed, the prices have changed, the account has been closed, and the run that made the bad decision cannot occur again because its preconditions are gone. Teams that only re-run conclude the bug is unreproducible and close the ticket. The bug was perfectly reproducible; the harness was just asking the wrong question.
A replay also has to run the full decision path to be worth anything. If it skips retrieval and injects the final prompt, you have excluded the stage where most agent failures are actually born.
Five things that move underneath you
Sampling. The obvious one, and the least interesting. Temperature and top-p make token selection stochastic. Recording the sampling parameters gets you close and never gets you exact.
The model itself. An alias like -latest is not a version, it is a subscription to someone else’s release schedule. If your run record says you called the alias, it does not say what you called. This is the single most common gap, and it is free to fix.
Tool responses. The largest source by volume and the one people forget, because it does not feel like nondeterminism — it feels like data. The same query against the same table returns different rows on a different day. That is the world moving, and if you do not capture what the world said, you cannot reconstruct why the agent believed it.
Your own code. now(), uuid4(), a shuffled candidate list, a dictionary iteration order that changed with a library upgrade. Boring, and it will ruin a replay quietly.
Everything upstream of retrieval. The vector index was rebuilt. A document was edited. A permission was revoked, so a chunk that was in context in August is not in context today. Retrieval is a moving target dressed as a fixed one.
Record the responses, not just the calls
Most run records log the tool call and stop — name, arguments, sometimes a status code. That is enough for an audit trail and useless for replay. The argument tells you what the agent asked. The response is what it learned, and what it learned is the thing that produced the next decision.
So capture the response body verbatim, as returned, before any parsing or truncation your code applies. Parsed representations are a lossy summary of the evidence, and the field your parser dropped is disproportionately likely to be the field that mattered.
Capture failures with the same care. A timeout, a partial result, a rate-limit response: those are inputs too. How the agent reacts to a degraded dependency is one of the more interesting things you will ever replay.
What a replayable record holds
Beyond the request and the plan, a record has to carry the fully-resolved context that actually went to the model: the prompt after templating, after retrieval, after tool-description assembly. Not the template. Not the retrieval query. The bytes that were sent.
Alongside that, the model identity and every sampling parameter, resolved to a concrete version rather than an alias. Each tool call paired with its verbatim response, in order, failures included. Every value your code generated that did not come from outside: timestamps, identifiers, random draws. And each policy evaluation with the inputs it saw and the decision it returned, because “the gate allowed this” and “the gate was never consulted” look identical downstream and have completely different fixes.
And a pointer to the state the run read against, where your systems can express one: a snapshot identifier, a table version, an index build number, an as-of timestamp. That pointer is what makes the data side of the run reconstructable, and it is the same discipline described in Making an Agent Explain What It Did to Your Data.
The test for whether a record is complete is mechanical. Take it to a machine with no network access and try to drive the system from it. Whatever the harness reaches for and cannot find is a missing field, and you will find three or four on the first attempt.
Pin the model, and be honest about what that buys
Pinning the version removes the largest and most confusing source of drift, the one where a run “changes behaviour” on a day you shipped nothing. Do it. It costs a config change.
It does not buy you a bit-exact rerun, and a harness built on the assumption that it does will fail constantly for reasons unrelated to your bug. Two replays of the same record will produce different wording, sometimes a different tool order, occasionally a different decision. That last one is not a broken harness. It is information: a decision that flips across identical inputs was never robust, and you have just learned that the original run could have gone either way.
Providers also retire versions. When yours is sunset, records that reference it become historical documents rather than executable ones — still readable as evidence, no longer runnable. Keep the record’s evidentiary value independent of whether it can still be driven, because eventually it cannot.
The payoff is the regression test
Investigation is the reason people build replay. Regression testing is the reason it pays for itself.
A run record from a real incident is a fixture that no one had to invent, describing a situation no one would have thought to write down. Promote it into the test suite and the postmortem stops being a document and becomes a check that runs on every prompt change.
The trick is what you assert on. Never assert on the text, which will not match and should not have to. Assert on the decision: which tool was selected, what arguments were constructed, whether the gate fired, whether the run terminated, whether the total spend stayed under its ceiling. The prose is presentation. The action is behaviour, and behaviour is what regressed.
Do this for a year and you accumulate something rare — a suite of adversarial cases drawn entirely from reality, which is a far better distribution than any set of examples your team would have designed in a room. Prompt edits are the highest-risk, least-tested change in most agent systems. This is the only practical way to make them safe to ship on a Friday.
What replay will not give you
Concurrency. Two runs racing for the same record produce a failure that lives in neither run’s record. A replay is single-threaded by construction, so this is the largest structural blind spot you inherit.
Anything you did not record. Coverage equals instrumentation, exactly. A field you chose not to capture in March is not recoverable in September, and the omission is invisible until the one incident that needed it.
Load-dependent behaviour. The timeout that only fires when the warehouse is busy, the retry storm that only forms under contention. Recorded responses arrive instantly and politely, which is precisely not the condition that caused the incident.
Anything that already happened. You will learn why the email went out. You will not unsend it. Reversal is a separate discipline, and it belongs in the design of the action rather than in the forensics afterwards.
The part nobody budgets for
Verbatim tool responses are large, and they contain whatever your tools returned — customer records, transaction detail, internal documents. You have created a second copy of your most sensitive data, in a system that was scoped as observability and is probably governed like observability.
Treat run records as production data. Same access controls, same retention limits, same redaction as the warehouse they came from, which is the argument made in Letting an Agent Touch the Warehouse Safely. Tiered retention handles most of it in practice: full fidelity for a short window, records attached to incidents kept indefinitely because they have become tests, and everything else reduced to a decision-level summary.
Decide this before you turn recording on. Retrofitting redaction onto eighteen months of accumulated records is a project nobody wants and somebody eventually gets.
Start this week
Pick the last incident you could not explain. Write down the fields you would have needed to replay it — you already know most of them, because you spent an afternoon wishing you had them. Add those fields to the record for one high-consequence action only. Not the platform. One action.
Then replay something that worked. A successful run, deliberately chosen, before you are under incident pressure. The harness will be broken in some small way and you want to discover that on a quiet afternoon rather than at hour three of an outage, when the ability to reconstruct the run is the only thing standing between you and a meeting where everyone agrees on a theory.
Run records and replay sit inside the broader reliability discipline in The Agent Reliability Handbook. The specific failure this capability defeats is catalogued as the unreproducible decision in Agent Failure Modes. Replay only reaches the boundary if your tools declare where their boundaries are, which is Tool Contracts. And the promoted-fixture habit pairs naturally with the pre-registered thresholds in Shadow Mode.