Every agent demo works. That’s what a demo is for. The agent reads the ticket, writes the code, files the PR, and everyone in the room nods. Nobody asks what happens on the four hundredth run, at 2am, when the upstream API returns a 200 with an empty body.
This guide is about that four hundredth run. It collects what I’ve learned shipping agent features into systems where being wrong costs money — and where “the model hallucinated” is not an incident report anyone will accept.
The short version: reliability in agent systems is almost never a model problem. It’s a systems design problem wearing a model costume.
Why agent demos lie
A demo is a single run, observed by a human, on a happy path, with someone ready to hit Ctrl-C. Production is thousands of unobserved runs across inputs nobody anticipated, where the failure you get is not the failure you tested for.
The gap between those two things isn’t intelligence. Give the same model the same prompt and it will do roughly the same reasonable thing. What changes is everything around it: partial failures, retries, stale reads, concurrent runs stepping on each other, a tool that used to return a list and now returns a paginated envelope.
Agents are the first systems where we routinely deploy a component that is non-deterministic by design and then wire it directly into infrastructure that assumes determinism. We spent forty years learning to make distributed systems predictable. Then we put a probabilistic planner at the top of the call stack and acted surprised.
None of this is an argument against agents. It’s an argument for treating them as infrastructure rather than as a feature.
The failure mode that actually gets you
Ask most teams what they’re worried about and they’ll say hallucination. Wrong answers. Made-up citations.
That’s the chatbot threat model, and it stops applying the moment your agent can act. Once it holds a token and can call a tool, the thing that hurts you isn’t wrong text. It’s wrong action — and wrong action is quieter.
A hallucinated paragraph is visible. A hallucinated action looks exactly like a successful one: a 200 response, a green checkmark, a log line. You find out on Monday, from the numbers.
I wrote about this framing at more length in Your Agent Is a Junior Operator With Root Access, and the mental model there is the one I keep coming back to. You have hired someone fast, tireless, and confident, who is missing context you assume is obvious. You would not give that person unaudited production write access on day one. The question is not “how do we make it smarter?” It’s how do we make it safe when it’s wrong?
Because it will be wrong. Design for that and everything else follows.
Contracts before brains
An agent is not a model. An agent is a contract between a reasoning system and the systems it touches — and most production incidents live at that boundary, not inside the model.
By contract I mean the agreed shape of the interaction: what goes in, what comes out, what the error cases are, and who is responsible for recovery. This is the argument I made in Agents Need Contracts, Not More Brains, and I’ve become more convinced of it, not less.
Three guarantees are worth more than any amount of prompt tuning.
Idempotency
Every state-changing call an agent can make should be safe to run twice. Not “unlikely to run twice” — safe.
Agents retry. Frameworks retry. Networks partition and the agent, seeing no response, tries again. If create_invoice is not idempotent, you now have two invoices and a customer service problem.
The fix is unglamorous: accept an idempotency key on every write, and make the second call with the same key return the first call’s result rather than performing the work again. If an operation genuinely cannot be made idempotent, that fact belongs in the contract, and the agent should be required to get human confirmation before invoking it.
Here’s the retail version of this, which is where I first learned it the hard way: a bid adjustment that runs twice doesn’t double your bid, it doubles your spend. The API returned success both times. Nothing was “broken.” You just paid twice for the same decision.
Explicit side effects
A tool description that says "updates the record" is not a contract. It’s a rumour.
Does “update” append or overwrite? Does it cascade? Does it fire a webhook that emails a customer? An agent reading that description will pick the most plausible interpretation, which is not the same as the correct one.
Declare side effects explicitly and in the schema the agent actually reads — what gets written, what gets triggered downstream, whether the operation is reversible. If calling this tool sends an email to a human being, that must be impossible to miss.
Failure semantics
The worst tool response is an ambiguous one. {"status": "ok"} tells the agent nothing about whether the work happened, partially happened, or was queued.
Distinguish, in the response itself, between: succeeded, failed cleanly with no change, failed after partial application, and accepted-but-not-yet-applied. That fourth case is the one that produces the strangest bugs, because the agent moves on to the next step believing the world has already changed.
Vague success is worse than clear failure. A clear failure gets handled. A vague success gets built upon.
Separate the plan from the permission
This is the single highest-leverage structural decision in the whole stack, and it’s easy to get wrong because the wrong version looks like it works.
The agent decides what to do. Something else decides whether it’s allowed. Those must be different components, and the second one must not be reachable by the first.
If your permission checks live inside the prompt — “you are not allowed to delete production data” — you don’t have a control. You have a suggestion. Anything expressed in the prompt is subject to the same probabilistic process as everything else in the prompt, and it’s negotiable by any input the model reads afterward. That’s the core of why bolt-on audits don’t work: by the time the audit log records the action, the action has happened.
What you want is an enforcement layer the reasoning loop cannot see or influence. The agent proposes an action; the layer validates it against an allowlist, checks scope and quota, and either executes or refuses. The agent finds out the same way it finds out about any other tool failure — through the contract.
I’ve called this an action firewall, and the name is doing real work. You don’t put a firewall inside the process it’s protecting.
The practical test: if a cleverly-worded document that your agent retrieves could talk it into an action, your controls are in the wrong place. Prompt injection isn’t really a prompting problem — it’s privilege escalation through an untrusted input channel, and it’s solved the way privilege escalation is always solved.
Bound the blast radius
Assume the agent will do the wrong thing. Now ask: how much damage can that wrong thing cause?
If the honest answer is “unclear,” that’s the finding. Blast radius should be a number you can state, not a thing you reason about after an incident.
Capability tokens, not roles
Role-based access breaks down for autonomous systems, because roles are designed around a human’s job and agents don’t have jobs. They have runs.
Scope credentials to the run, not the service. Short-lived, narrow, tied to the specific task, revoked on completion. An agent handling a refund needs to refund that order — not to hold the refund capability indefinitely.
A single long-lived token shared across every tool an agent touches means the blast radius of any compromise is the union of everything that token can reach. That’s how a read-only integration becomes a write incident.
Circuit breakers and budgets
Agents fail in loops. That’s their characteristic failure shape, and it’s different from the systems most of us built before.
A human operator who tries something three times and fails will stop and ask someone. An agent will keep going, and each attempt costs money and may leave partial state behind. Give every run a retry budget, a cost ceiling, and a wall-clock timeout. When a budget is exhausted, stop and escalate rather than degrade quietly.
Circuit-break on the upstream too. If a service starts erroring, the agent should drop into a degraded read-only mode instead of hammering it with increasingly creative retries.
Run records and deterministic replay
When an agent does something strange, you need to answer one question fast: what did it see, and what did it decide?
Logs designed for humans are not sufficient here. You need a structured record per run: the triggering request, the plan the agent produced, every tool call with its arguments, every policy decision and its outcome, and the final state. Stored so you can reconstruct the run without re-reading a wall of text.
Deterministic replay is the payoff. Take a run record, feed the same inputs and the same recorded tool responses back through the system in a sandbox, and confirm you get the same decisions and the same guardrail triggers. This isn’t about proving the agent was right — it’s about making debugging tractable, and about turning any incident into a regression test you keep forever.
The teams I’ve seen do this well treat a run record the way a payments team treats a transaction log: not as debugging output, but as the authoritative account of what happened. That framing changes what you store and how long you keep it.
More patterns in this vein are in Agent design patterns for production.
Shadow mode
Before an agent takes an action in your business, let it propose the action and take none.
Run it against real production inputs, have it write its intended action to a queue, and compare against what actually happened — either what a human did, or what the previous automation did. You get a real measurement of how often it would have been right, on real traffic, at zero risk.
Two weeks of shadow mode will teach you more than any eval suite, because it surfaces the thing evals structurally cannot: the inputs you didn’t think to write a test for.
Watch specifically for near-misses — cases where the agent chose a defensible action that happened to be wrong. Outright failures are easy to spot and easy to fix. Plausible-but-wrong is the category that survives review and reaches production.
And be honest about the exit criterion before you start. “It looked good” is not a threshold. Decide in advance what agreement rate, on which action types, earns the right to go live.
Build the undo before the action
For every destructive capability you give an agent, build and test the reversal path first.
Not documented. Not planned. Built, and exercised, before the forward action ships.
This ordering feels backwards and is worth insisting on anyway, because the undo built after an incident is written by someone tired, at speed, under pressure, against a system already in a strange state. The undo built beforehand is written calmly by someone who understands the invariants.
Some actions have no undo. Money moved, emails sent, keys rotated. Those don’t get built last — they get a human gate, permanently, and you stop trying to automate the gate away. The goal was never full autonomy. The goal is more leverage without more risk, and a permanent gate on genuinely irreversible actions is a reasonable price.
Roll out progressively
Capability is not binary. The useful sequence:
- Read-only. The agent observes and reports. No writes at all.
- Propose. It writes intended actions to a queue for human approval.
- Write, reversible, scoped. Real writes, but only operations with a tested undo, on a bounded slice of traffic.
- Write, broad. Expand the slice as the record justifies it.
Most teams jump from step one to step four because steps two and three feel like theatre. They aren’t. They’re where you find out that your idempotency key isn’t actually being honoured by the downstream service.
Expand on evidence, not on enthusiasm. The run records tell you when you’ve earned the next step.
The readiness checklist
Before an agent touches anything that matters:
- Every write operation accepts an idempotency key, and the downstream service actually honours it.
- Every tool declares its side effects and whether it is reversible.
- Tool responses distinguish success, clean failure, partial application, and accepted-not-yet-applied.
- Policy enforcement runs outside the reasoning loop and cannot be influenced by model output or retrieved content.
- Credentials are scoped to the run and expire with it.
- Every run has a retry budget, a cost ceiling, and a timeout.
- A structured run record is written for every execution and retained.
- Any run can be replayed in a sandbox and produce the same decisions.
- Every destructive action has a tested reversal path, or a permanent human gate.
- The agent ran in shadow mode against real traffic, and you set the promotion threshold in advance.
- You can state the blast radius of a worst-case run as a bounded number.
If you can’t tick an item, that’s not a blocker on shipping — it’s a statement about which step of the rollout you’ve earned.
What to do this week
Pick the single highest-impact action your agent can take. Just one.
Write down its contract: inputs, outputs, side effects, failure modes, whether it’s reversible, and who recovers when it goes wrong. If you can’t fill in every field from the existing documentation, you’ve found your first gap — and it was there before the agent arrived. Agents don’t create these problems. They find them, at scale, at speed, at 2am.
Then run that one action in shadow for two weeks and count the near-misses.
Everything else on this list is easier once you’ve done that for one action, because you’ll stop thinking of the agent as something you’re configuring and start thinking of it as something you’re integrating. That shift is most of the work.