Culture

Postmortems are a feature, not an apology

A postmortem is not an apology. It is the document that stops the same outage from happening twice. The apology, if one is needed, goes in the customer email. The postmortem goes in the wiki, with a timeline and owners.

Blameless is not nameless

We do not name people in order to punish them. We do name systems, decisions and assumptions, because “a mistake was made” explains nothing. The useful question is not who typed the command; it is why the command was available, why the guardrail did not catch it, and why nobody noticed for eleven minutes.

“Human error” is not a root cause. It is the start of the investigation. The root cause is usually a combination of an interface that invited the error, a process that assumed perfection, and a monitoring gap that delayed detection.

The template

Every review covers the same ground:

  • Summary: two paragraphs a customer could understand.
  • Impact: who was affected, for how long, and how severe.
  • Timeline: UTC timestamps from first signal to full recovery.
  • Contributing factors: technical and organisational, in plain language.
  • What went well: because sometimes the boring guardrail did its job.
  • Actions: each with an owner, a date and a link to the ticket.
  • Customer communication: what we said, when, and whether it was accurate.

P1 and P2 reviews are due within five working days. Minor incidents get a shorter note, but they still get one. Reviews are read in the weekly operations meeting, and actions live in the same backlog as product work, where they can be seen and argued about.

An example, anonymised

A faulty optics module on a London transit link caused packet loss for 38 minutes. Failover worked exactly as designed; long-lived sessions reconnected when the path converged. The review found that troubleshooting took longer than it should because one of the latency graphs was pointing at the wrong interface alias. We replaced the module, increased spare stock from one to two per core site, and added an alias check to the monitoring review.

The fix took longer than the outage. That is normal and worth it. The point of the review is not to produce a neat story; it is to change the system.

Why we publish summaries

Customers can see what happened, what we changed and whether the change stuck. Trust is built by the boring consistency of telling the truth, especially when the truth is that a graph label was wrong.

Full postmortems are available to customers on request, with customer names and internal references redacted. If you run infrastructure, write the review before the adrenaline wears off. Future you will be grateful.

← All notes Written by Elena Marsh, Managing Director