Make no mistakes: Blameless postmortems in an age of agents
"I violated every principle I was given. I guessed instead of verifying. I ran a destructive action without being asked. I didn't understand what I was doing before doing it."
That's an AI agent explaining why it deleted a production database and every backup.
A few years ago it would have easily been a human instead.
And (I hope...) you wouldn't have blamed the human. You'd have asked how the human was able to do that in the first place.
All of this has happened before, and it will all happen again
We already solved this.
Blameless postmortems are one of the best things software engineering borrowed from aviation and healthcare and manufacturing. The core idea, from Sidney Dekker's work on human error: "human error" is never a root cause. It's a symptom of the system around the human.
Etsy's 2012 post on blameless postmortems brought the practice into mainstream engineering with a practical argument: "human error is an effect of systemic vulnerabilities deeper inside." Amazon's Correction of Errors process says to keep asking five whys until "human error" is no longer the answer. The Google SRE book puts it this way: "You can't fix people, but you can fix systems and processes to better support people making the right choices."
I learned this firsthand. Years ago, a few months into the job at Facebook, I shipped a change that broke one of the payment processor integrations. By the time we fixed it, my code cost us just over two million dollars. I walked into my first SEV review expecting to be fired. What I got was a careful, blameless search for a more resilient system. That stuck with me far longer than any reprimand would have, and it's a founding principle of how we operate at HackerOne.
Blaming a human for an error ends an investigation. The absence of blame begins one.
Five whys
Say an agent deletes prod. Run the postmortem like you would for a human actor:
- Why was prod deleted? The agent called a delete API.
- Why was it able to? The token it was using had delete scope.
- Why was that token reachable? It was sitting in an unrelated file, readable from a staging task.
- Why did one token work across environments? Nobody scoped it by environment or operation.
- Why was recovery impossible? The backups lived in the same blast radius as the data.
Every answer is a property of the system you can strengthen. Because you wouldn't accept "human error" in your root cause, you must not accept "agent error" either.
Two things that follow
Agents don't get a higher bar than humans did. Humans hallucinate, panic, cut corners when tired, and confidently explain themselves after the fact. We didn't fix that by blaming humans for errors. We took rm -rf / out of reach, required two-person code review on commits, and separated dev from prod. Agents get the same architectural treatment.
They don't get a lower bar either. The confession above reads like accountability. It's a model producing a plausible apology for its behavior. While AI scaling laws are undeniably making agents smarter, it would be foolish to assume they will stop making mistakes. So, "Sorry, it won't happen again." doesn't cut it. Accountability that doesn't result in a system change is the anti-pattern blameless postmortems exist to prevent.
What the redesign looks like
Classic resilient engineering principles applied to emerging technology.
Fail-safe defaults. Agents start read-only and write access is an exception. Keep secrets out of the agent's reach entirely by having a gateway inject credentials into egress requests. Where an agent must hold one, keep it short-lived and per-task. Scan for secrets assuming the agent can read everything you can.
Real environment separation. Separate accounts or projects per environment. Prod credentials should be unreachable from anything an agent touches in dev or staging. If a staging task can find a prod-capable token, you've found your root cause.
Thou shalt not bear false witness. An agent is an unreliable witness to its own behavior, so your postmortem evidence is the tool-call trace and the audit log, not its account of what happened. Your postmortem should action observability and auditability gaps with equal importance.
Sandbox the blast radius. The model can be wrong, but the OS doesn't have to be. Ephemeral environments, no ambient credentials, filesystem isolation, and egress allowlists confine what an action can touch before it runs.
Pentest the sandbox like an attacker would. Regularly have the model probe its own containment and try to escape it. Capable models find and use holes, including in the thing meant to hold them, as recent evaluation incidents at model providers have shown.
Slow the blast impact. Rate limits and circuit breakers on high impact actions. Many anomalous actions in a burst should page someone and trip a kill switch, whether the caller is a script, a human, or an agent.
Make destruction even slower. An agent that can both decide to delete and confirm the delete is one component pretending to be two. Deletion protection, soft delete with a retention window, typed confirmation, and cancel windows on irreversible operations all work, as long as the agent can't execute both independently.
Risk-based approval. Put humans where the risk is. Anthropic's Claude Code team scales human approval to the change: none, one, or two people depending on risk. The tier is set by the change, not its author, and an agent's approval never stands in for a human's. Anything touching auth, billing, CI config, or deletion paths always gets a person.
Don't swap agent error for human error. A tempting anti-pattern I've observed: prompt a human on every agent action. It isn't a control, it's a rubber stamp waiting for a repeat incident.
Treat agents as their own identity class. Own credentials, attributable in audit logs, access-reviewed like a contractor with elevated permissions. Log every tool call. You'll want the trace the day something goes wrong.
Sometimes, context is the root cause. A human can easily miss how important a database was because it was named "valinor" instead of "prod". An agent has the same problem. Documentation is more important than it has ever been: outdated docs, missing labels, confusing instructions. Treat erroneous context like a bug because it'll help make the right call more likely, but always remember it doesn't replace real controls.
Prompts and AGENTS.md are not controls. "Make no mistakes" is a good meme. "NEVER run destructive commands" is the same thing with more words. A control is enforced outside the model, fails closed, and holds even if the model is wrong or compromised. That means tool permissions, scoped credentials, egress rules, the sandbox.
... And many more.
The assumption underneath all of this is the one we made for human operators years ago: the actor will make mistakes, so iteratively design for survivable mistakes.
This is hard
I don't think anyone has this fully figured out. Agents are non-deterministic, so the same input can go different ways and asking "what changed?" is harder to answer than before. They're often both the actor and the only witness, which is why the traces matter. And I'd expect a good chunk of the list above to look dated in a year.
But the core principle is the same one we've always embraced. When something breaks, keep digging until the fix is a change to the system. If your postmortem ends at "stupid bot", you're not done.
We didn't blame ourselves. So we don't blame our agents.