Alert Rule

Origin & evolution

The canonical text is Rob Ewaschuk's "My Philosophy on Alerting", a Google doc that circulated internally around 2013 and later became the alerting chapter of the Site Reliability Engineering book. Its argument reframed the whole practice. Pages should be urgent, important, actionable, and real. Err on the side of removing noisy alerts, because over-monitoring is harder to recover from than under-monitoring. A page that a human cannot act on is a defect in the alert rule, not a feature⭐FeatureProduct SpecificationA product capability or featureView reference →.

From this came the symptom⚠️SymptomEngineeringA symptom of a problemView reference →-versus-cause distinction. Ewaschuk argued for alerting on symptoms, the conditions users actually feel, and demoting causes to dashboards🎛️DashboardData & AnalyticsAn analytics dashboardView reference → and context. Symptom-based alerts catch more real problems with fewer rules, since a single user-facing symptom can be produced by dozens of underlying causes you would otherwise have to enumerate one by one. As the SRE book puts it, in a layered system one person's symptom is another person's cause.

The newer refinement is SLO-based, burn-rate alerting, set out in the SRE Workbook. Here the rule fires on how fast a service is consuming its error budget⏳Error BudgetDevOps & PlatformAn error budget for a serviceView reference →, not on a raw threshold crossing. The recommended technique is multiwindow, multi-burn-rate: a fast short window and a slower long window must both breach before the rule fires. The long window resists transient spikes; the short window clears quickly once you fix the problem. Google's suggested starting points are paging when a service burns 2% of its monthly budget in an hour, or 5% in six hours.

How it works in practice

A team owns an API with a 99.9% availability SLO, giving roughly 43 minutes of allowed error per 30-day window. Their old rule paged whenever the error rate exceeded 1% for five minutes, and it fired most weeks on harmless blips. They replace it with a burn-rate rule: page only when the error budget burns at 14.4 times the sustainable rate across both a 5-minute and a 1-hour window.

The pager goes quiet on noise. A 90-second spike during a deploy no longer fires, because the 1-hour window stays calm. When a bad config change starts burning the budget for real, both windows trip within minutes and the on-call engineer is paged with the symptom already framed in budget terms. Every page now carries a linked runbook📘RunbookDevOps & PlatformA runbook for incident responseView reference →, so the responder opens a known procedure instead of starting from a blank terminal.

Alert rule vs. its neighbours

• Monitor. A monitor📺MonitorDevOps & PlatformA monitoring checkView reference → observes continuously and reports a health signal. An alert rule is the thin layer of judgement on top that decides the signal warrants human attention. The monitor never sleeps; the alert rule chooses when to wake someone.
• Incident🚨IncidentDevOps & PlatformA production incidentView reference →. An alert rule firing is a hypothesis🧪HypothesisValidationA testable belief about a solutionView reference → that something is wrong. An incident🚨IncidentDevOps & PlatformA production incidentView reference → is the declared, coordinated response once that hypothesis is confirmed worth handling. Many alerts auto-resolve and never become incidents; conversely a major incident may be opened by a customer report before any rule fires.
• Runbook. A runbook is the documented procedure a responder follows. The working rule, captured in the graph, is that every alert rule should point at one. An alert with no runbook is asking a tired human to improvise at 3am.

In the graph

In the Unified Product Graph an alert rule sits between observation👁️ObservationUser ResearchA specific behaviour or statement observedView reference → and response. It receives its signal through monitor_triggers_via_alert_ruleMonitortriggers viaAlert Rulehierarchy and dispatches action through alert_rule_triggers_runbookAlert RuletriggersRunbookcross-domain. Modelling the rule as its own entity, separate from the monitor that feeds it and the runbook it invokes, makes the "every alert needs a runbook" discipline queryable: any alert rule with no outgoing alert_rule_triggers_runbookAlert RuletriggersRunbookcross-domain edge is a page waiting to land on someone with no instructions.