You are being asked to approve an agent deployment. You cannot read the code, you will not personally verify the tests, and the team asking is more expert in this system than you are. That is the normal condition of the job, and it is not an argument for rubber-stamping.
What you can do is ask questions whose answers are hard to fake. The Agent Reliability Handbook lists what a team should have in place. This is the other side of it: how to tell whether they actually have it.
The one technique
Everything below is an application of a single move: ask for the artifact, not the assurance.
“Do you handle retries safely?” gets you yes, sincerely, from a team that has not tested it. “Show me the test that fires the same idempotency key twice and asserts one row” gets you either the test or a pause. The pause is the finding.
This is not adversarial and should not feel it. Almost every gap surfaced this way is one the team would have wanted to know about. You are supplying the deadline that makes the invisible work legible — and if you have ever wondered why guardrail work never gets prioritised, this is the lever.
The nine questions
1. What is the worst a single run can do?
Good: a bounded number. “At most 40 records, at most £500, at most one customer email each, and here is the ceiling that enforces it.”
Hand-wave: “It’s sandboxed.” “It only has read access to most things.” “It can’t really do much damage.”
If the answer is an adjective rather than a quantity, the authority has not been scoped. This is the single most diagnostic question on the list, and a team that has done the work answers it immediately, because they had to compute it to build the limits.
2. Which actions are irreversible, and what gates them?
Good: an enumerated list, each item with a named reversal or an explicit human gate. “These four are irreversible. Three require approval. The fourth has a tested compensating transaction — here it is.”
Hand-wave: “Everything’s reversible, we have backups.” Backups are a disaster-recovery mechanism, not a compensation path. Restoring a database because an agent sent 200 wrong emails is not a reversal.
3. Show me a run record.
Not a description of the logging strategy — an actual record, for a specific run, on screen.
Good: they pull one up and it contains the inputs, the plan, every tool call with arguments, the policy checks, and the outcome.
Hand-wave: “Everything’s in the logs.” Logs tell you what happened. A run record lets you reconstruct why. If producing one takes a week of engineering, you do not have run records — you have the option of building them later, during an incident.
4. What did shadow mode tell you, and what threshold did you set beforehand?
Good: a promotion criterion written before the first shadow run, the observed result against it, and an account of the disagreements that were reviewed rather than averaged away.
Hand-wave: “It matched the human decision most of the time.” Most of the time is not a threshold, and a threshold produced after seeing the data is a justification. If the number appeared after the run, treat the whole exercise as observation rather than evidence — useful, but it did not test anything.
Ask when the criterion was written down. The answer is usually honest and immediately informative.
5. Who gets paged, and what do they do?
Good: a named rota, a runbook, and — the part that matters — a description of what that person is empowered to do at 3am without waking anyone else.
Hand-wave: “It alerts the team channel.” A channel is not an owner. If the recovery path requires the one engineer who built it, you have not deployed a system; you have deployed a dependency on a person.
6. What is the kill switch, and when was it last tested?
Good: one control, stops everything, no deploy required, tested on a schedule with a date they can give you.
Hand-wave: “We’d turn off the feature flag” — said with confidence, never rehearsed. An untested kill switch is a belief. The follow-up that finds the truth: who has used it, and when?
7. What are the spend ceilings, per run and in aggregate?
Good: both numbers, enforced automatically, with a stated behaviour when they are hit.
Hand-wave: a per-run limit only. Per-run ceilings do not catch ten thousand individually reasonable runs, and cost runaway rarely arrives as one expensive execution. It arrives as a retry loop nobody noticed for nine days.
8. What would make you turn this off?
Good: a stopping rule, written down, specific. “Two unauthorised destructive attempts, or accuracy below X on the weekly eval, and we revert to human approval.”
Hand-wave: “We’d look at it case by case.” Everything is case by case in the moment, under pressure, with the person who built it in the room. Deciding the rule in advance is the only version that survives contact with an actual incident.
9. What does it cost to be wrong once?
The question that determines how much of the above you should insist on. A wrong tag on a support ticket and a wrong refund are not the same deployment, and applying identical rigour to both is its own failure — it teaches the organisation that this process is theatre.
Good: they have thought in terms of the cost of a single bad outcome, and the controls are visibly proportionate to it.
Hand-wave: an accuracy percentage offered as though it answered the question. 97% accurate means nothing until you know what the 3% costs and who absorbs it.
The meta-question
After the session, ask yourself one thing: which answers did the team have to go away and find?
Anything they could not answer in the room is a gap, regardless of how good the answer is when it arrives a week later. Not because the follow-up is wrong, but because nobody was holding that fact — and if nobody was holding it before an approval meeting, nobody will be holding it during an incident.
Track that, not the eventual answers. It is the most reliable signal available to you and it requires no technical depth at all.
How to run it
Publish the questions in advance. There is no value in surprise. You want the team to prepare — the preparation is the intervention, and if these nine questions cause work to happen before the meeting, the meeting has already succeeded.
Frame it as a design review, not an audit. The failure mode of this process is that it becomes a ritual people learn to pass, at which point it produces confident answers and no information.
Ask what they are most worried about. Often the highest-value minute in the room. Teams generally know exactly where their system is weak, and are rarely asked in a setting where saying so is safe. Make it safe and you get the real risk register for free.
What “no” should look like
The decision is not binary and treating it as such is what makes approval meetings adversarial.
Almost always the right answer to a partially-ready system is narrower, not later. Approve the two lowest-consequence action types unattended, keep the rest behind human approval, and set the date to revisit. That gives the team real production signal, gives you a bounded downside, and moves the argument from “is this safe” to “which parts are safe now” — a question that has an answer.
A flat no is right when the blast radius is unbounded, when there is no kill switch, or when nobody can produce a run record. Those three are not process gaps. They mean that if this goes wrong, you will not be able to stop it, undo it, or explain it — and no amount of expected upside compensates for that combination.
What you are actually deciding
Not whether the system is correct. You cannot know that, and neither can the team; the honest position is that it will be wrong sometimes and the question is what happens then.
You are deciding how much authority to grant given your current confidence — which is a sizing decision, not a certification. That framing is more useful than it sounds, because it stops the conversation being about whether the technology works and makes it about consequence and reversibility, which is the thing you are actually qualified to judge.
Size the authority to what you can afford to be wrong about, and increase it as evidence accumulates. The details of how to think about confidence and its calibration are in What Horse Racing Taught Me About Model Confidence; the engineering behind every question above is in The Agent Reliability Handbook, and the failure modes each one is probing for are enumerated in Agent Failure Modes.