Operations
How we plan maintenance windows without surprises
The rules, the sign-off and what the numbers looked like last quarter.
Culture
A postmortem is not an apology. It is the document that stops the same outage from happening twice. The apology, if one is needed, goes in the customer email. The postmortem goes in the wiki, with a timeline and owners.
We do not name people in order to punish them. We do name systems, decisions and assumptions, because “a mistake was made” explains nothing. The useful question is not who typed the command; it is why the command was available, why the guardrail did not catch it, and why nobody noticed for eleven minutes.
“Human error” is not a root cause. It is the start of the investigation. The root cause is usually a combination of an interface that invited the error, a process that assumed perfection, and a monitoring gap that delayed detection.
Every review covers the same ground:
P1 and P2 reviews are due within five working days. Minor incidents get a shorter note, but they still get one. Reviews are read in the weekly operations meeting, and actions live in the same backlog as product work, where they can be seen and argued about.
A faulty optics module on a London transit link caused packet loss for 38 minutes. Failover worked exactly as designed; long-lived sessions reconnected when the path converged. The review found that troubleshooting took longer than it should because one of the latency graphs was pointing at the wrong interface alias. We replaced the module, increased spare stock from one to two per core site, and added an alias check to the monitoring review.
The fix took longer than the outage. That is normal and worth it. The point of the review is not to produce a neat story; it is to change the system.
Customers can see what happened, what we changed and whether the change stuck. Trust is built by the boring consistency of telling the truth, especially when the truth is that a graph label was wrong.
Full postmortems are available to customers on request, with customer names and internal references redacted. If you run infrastructure, write the review before the adrenaline wears off. Future you will be grateful.