I joined the on-call rotation in January with a fresh laptop and a strong opinion that alerts should be fixed at the source. A year later the opinion survives, but it has company.
The pager is a product
The first month I treated every page as a fact about the system. By March I understood that every page is a fact about a person’s decision, made months ago, about what they thought would matter. Half of ours were wrong, and nobody had been given permission to delete them.
So we gave permission. One rule: an alert that pages must have a runbook that says what to do, and if the answer is “look and see”, it is not a page, it is a dashboard.
Forty-one pages in a year, down from an estimated three hundred the year before. The three that mattered still paged.
Things I believed and no longer do
- “Alert on symptoms, not causes.” Half true. Alert on symptoms to page; alert on causes to annotate, so the person who is paged sees the likely cause next to the symptom instead of hunting for it at 3 a.m.
- “Automate the response.” Only after the third time. The first time you do not understand it. The second time you understand it but you are wrong. The third time, maybe.
- “On-call should be a burden shared equally.” Shared, yes. Equally, no. The person who built the thing should carry more of it for the first quarter. Nothing improves a design like being woken up by it.
The three real incidents
I will not relive them here, but each one left a rule behind.
- The certificate. Every expiring thing gets a calendar reminder thirty days out and an alert seven days out. Both. The alert alone was silenced during an unrelated incident and nobody unsilenced it.
- The autoscaler. A scale-down policy that was correct for the traffic pattern of the month it was written. Now every scaling policy has a comment with the date and the assumption, so the next person knows what it was solving for.
- The upgrade. We upgraded the control plane on a Friday because “it’s just a patch release”. It is never just a patch release. Upgrades go out Tuesday to Thursday, before noon, with someone watching who did not do the upgrade.
The system is not the thing in the cluster. The system is the cluster plus everyone who touches it, including the version of you that is asleep.
What I’d tell January me
Learn the runbooks before you need them. Fix one alert every week, not all of them at once. Write the incident review the same day, while it hurts. And keep a file of the pages that were right — on the bad weeks, that file is the reason you are still doing this.