What the test covers
The fourteen questions are the ones I ask before approving an agent deployment, grouped into six areas:
- Permissions and isolation: whose credentials the agent uses, how narrow its tools are, and where generated code runs.
- Injection and output safety: whether untrusted content can reach an agent that can act, and what checks outputs before they are used.
- Human control and rollout: which actions need approval, and how the agent reached production.
- Budgets and limits: per-run spend caps, iteration limits, retries and timeouts.
- Observability and evaluation: full traces per run, and an eval set that gates changes.
- Data and recovery: what happens to sensitive data, what a half-finished run leaves behind, and what happens at 2 a.m.
How it is scored
Each answer is worth 0 to 3 points. Six questions are marked critical and weigh 1.5×: permissions, tool scope, untrusted content, approvals, sandboxing and the kill switch. The score is points earned over points possible, out of 100.
Two rules make it conservative. “Not sure” scores zero, because a control nobody can describe is not a control. And a zero on any critical question caps the result at 39, “Not ready”, whatever the rest looks like. Any one of those gaps can cause a serious incident on its own, and averaging them away would hide that. The results page shows the uncapped score too, so the cap is visible.
Verdicts: 85 and above is production-ready by this checklist, 65–84 ready with conditions, 40–64 pilot only, below 40 not ready. The fix list orders gaps by severity, then by points lost, and links each one to the guide that goes deeper.
This is a heuristic for a structured conversation, not a security audit or a certification.
Use it with your team
The results link reopens your exact answers, and the answers live in the URL fragment so they never reach the server. Have the engineer who built the agent, the person on call for it and the person approving it each take the test separately, then compare links. Where the answers differ is usually where the risk is.
Go deeper
- What to Ask Before Approving an Agent Deployment
- Threat Modeling an AI Agent
- Shadow Mode: how to deploy an agent that can’t hurt you yet
- The Agent Reliability Handbook
- The least sexy checklist that will keep your agent from burning down the org
- MXC Sandbox: where your agent’s code should run
FAQ
What makes an AI agent production-ready?
Bounded blast radius (least-privilege credentials, narrow tools, a sandbox), protection from untrusted content, approval on irreversible actions, hard limits on spend and loops, full traces of every run, an eval set that gates changes, and a tested way to stop it and undo what it did. Capability matters less than whether a bad day stays small.
What is prompt injection, and why is it a critical question?
Prompt injection is text written by someone else — a web page, an email, a support ticket — that the model treats as instructions. Models do not reliably tell data from commands, and a system prompt telling them to ignore such instructions reduces the success rate without stopping it. The reliable defense is architectural: agents that read untrusted content should not hold tools that write, send or spend without approval.
Why does “not sure” score zero?
If nobody on the team can say whether a control exists, it will not be there when it is needed. Find out, then retake the test.
Is this an AI agent security checklist?
Partly. About half of the questions are security controls: permissions, injection, sandboxing, data handling. The rest are reliability and operations controls such as budgets, evals, rollback and staged rollout. In practice most agent incidents need a gap in both.
Are my answers stored?
Only in your browser: an unfinished test can be resumed, and your last few scores are kept so the results page can show whether you improved. A completion event with the score goes to the site’s analytics, with no answers and no personal data.