There’s a post on this site about agent design patterns that, for about five months, didn’t end. It stopped roughly where you’d expect a closing paragraph, then kept going into something else entirely: "role": "assistant"}], "parallel_tool_calls": true, followed by a tidy little block of token usage counts. The raw API response, written straight through and published.
It went live on March 16. I found it on August 14, while auditing all 26 posts on the site. That’s 151 days of a JSON payload sitting on a page about production reliability, franchement, which is the sort of joke you can only enjoy afterwards.
Nobody emailed me about it. I try not to think too hard about what that says about my traffic.
Fixing it took two minutes. Explaining it took much longer, and I never actually finished.
The theory everyone nods at
Here’s the best account I have. The generation hit a token cap partway through, and the code meant to pull the text out of the response wrote the whole response object instead. Clean story. It fits the evidence, it explains the truncation, and it explains the JSON.
I can’t prove a word of it.
What I had was the output and a timestamp. Missing: what the model actually returned, the parameters the call went out with, and any record of what my own code did in between. So the theory stays a theory, and plausible is where it stopped.
None of those would have been hard to keep. That’s the part that stings a little. A run record holding the verbatim response, the resolved model version and the arguments my code built would have settled the question in the time it takes to read a diff. Instead I have a story I believe and cannot support, about my own software, on my own laptop.
Me, I’ve sat through enough incident reviews to recognize the shape of that meeting. Someone reads a log line aloud. Someone else offers a mechanism. Everyone nods, because nobody has anything better, and the action item becomes “add more logging,” which is what you write down when you don’t know what happened.
I did try re-running it. Same prompt, same pipeline, current model. Out came a perfectly ordinary post with a perfectly ordinary ending, which told me precisely nothing.
This is a distinction I’d been sloppy about for years, and it turns out to carry most of the weight here. Re-running asks whether the thing fails now. Replay asks why it failed then. Different machinery, different questions, and if you only have the first one, every bug older than a few weeks looks unreproducible and quietly gets closed.
Replay means feeding the system its recorded responses instead of calling anything at all. No model call goes out. Tool results come off disk instead. The agent sees the world exactly as it was in March, because you kept a copy of what the world told it.
The catch is that this only works if you wrote it down at the time. Instrumentation isn’t retroactive. The field I didn’t capture in March was not available to me in September, and I didn’t learn which field I needed until the day I needed it.
What I actually changed
Not much, on purpose. I run a blog, not a trading desk, and building replay infrastructure for a one-person publishing pipeline is a fine way to spend a month shipping nothing.
So: record the model response verbatim, before anything parses it. Record the resolved model version instead of the alias I’d been passing. Two fields, both free, and either one on its own would have turned a five-month mystery into a two-minute read.
The version alias is the one I’d nag a stranger about. If your record says you called -latest, it doesn’t say what you called. You’ve written down a subscription rather than a fact, and the day behaviour shifts on a release you had nothing to do with, you’ll have no way to demonstrate it.
The second-order benefit surprised me. Once a run is recorded well enough to replay, the bad ones stop being embarrassments and start being test cases. I now have a small pile of saved runs where the pipeline did something stupid, and every prompt change I make gets pushed through them before it ships. Not to check that the wording matches, which it never will. To check that the thing it decided to do is still the thing I’d want it to do.
The argument against all of this
Recording verbatim responses isn’t free, and I’d be selling you something if I pretended it was. The records get large fast. They contain whatever your tools returned, which for most companies means customer data landing in a system that was scoped as observability and is governed like observability. You’ve made a second copy of your most sensitive material and filed it somewhere with looser access rules than the database it came from.
The more common failure isn’t cost, though. It’s neglect. Plenty of teams build the recording layer and run it for a year without opening a single record. Then the storage bill gets noticed and it goes off, roughly two weeks before the incident that needed it. Bon.
Which is why the recommendation isn’t “build the platform.” It’s much narrower. Take the one action in your system with the worst consequences when it goes wrong, record what it saw, and leave everything else alone until that pays for itself.
I’ve written the long version as a new guide, Deterministic Replay for Non-Deterministic Systems. It covers what a run record has to hold to be replayable at all, and why pinning the model version is necessary without being sufficient. It also covers the part I’ve come to find genuinely useful. A recorded incident makes a far better regression test than anything a team would invent in a meeting room, because nobody would have thought to invent that situation. It sits alongside The Agent Reliability Handbook and the taxonomy of agent failure modes in the guides.
The post about agent design patterns is cleaned up now. It ends with a sentence.
