Runbook

Origin & evolution

The runbook is one of the oldest artefacts in operations, older than the web. In the mainframe era of the 1960s and 1970s, operators kept physical binders, literally "run books", describing how to run scheduled jobs🔨JobUserJob To Be Done: what the user is trying to accomplishView reference → on systems like the IBM System/360, complete with the Job Control Language to invoke and the error messages to expect. The form absorbed influence from aviation and nuclear checklists: short, unambiguous steps with explicit verification and clear escalation.

The practice modernised as operations did. Network operations centres turned binders into wikis; the rise of Site Reliability Engineering reframed the runbook as a living, linked document, ideally reachable straight from the alert that needs🫀NeedUserA user need, pain, desire, or constraintView reference → it. The Google SRE book makes the case bluntly that a good runbook, combined with sensible alerting, cuts mean time to repair and reduces the cognitive load on a tired on-call engineer at 3am.

The frontier is automation. As runbook steps become precise enough to script, they graduate into runbook automation, where the document executes itself and the human approves rather than types. The open debate is how far to push that: an automated runbook that no one understands is a new failure mode, so the better teams keep the prose runbook as the source of truth even when a machine runs the steps.

How it works in practice

A team owns a Redis cache that occasionally fills to capacity. Rather than rediscover the fix each time, they write a runbook titled "Redis memory pressure". It opens with how to confirm the symptom⚠️SymptomEngineeringA symptom of a problemView reference →, the exact dashboard🎛️DashboardData & AnalyticsAn analytics dashboardView reference → panel and the threshold, then the safe remediation: scale the instance, flush a specific key prefix, and the explicit warning about which keys must never be flushed.

The alert that fires on memory pressure links directly to this runbook. When it triggers at 04:00, the on-call engineer, who has never touched Redis, follows the steps and resolves it in six minutes without paging the specialist. The procedure that lived in one person's head now lives where the alert can find it, which is the entire point.

Runbook vs. its neighbours

• Playbook📒PlaybookCustomer SuccessA standard operating procedure for CSView reference → tends to describe the broader strategy for a class of situation, the roles and decisions⚖️DecisionStrategyA recorded decision with context, rationale, and consequencesView reference →, while a runbook is the concrete step-by-step for one task. The playbook says who decides and when to escalate; the runbook says which command to run.
• Postmortem🩻PostmortemDevOps & PlatformA post-incident reviewView reference → is backward-looking analysis of one failure. A runbook is forward-looking instruction for handling the next one. A common pattern is that a postmortem produces a runbook, turning a hard-won lesson into a standing procedure.
• Monitor detects the condition; the runbook tells you what to do about it. The monitor fires the alert; the runbook is what the alert should link to.

In the graph

In the Unified Product Graph, runbook📘RunbookDevOps & PlatformA runbook for incident response is a leaf in the Operations and Quality region, with devops as its home domain. It is reached through hierarchy edges such as product_documented_in_runbookProductdocumented inRunbookhierarchy and infrastructure_component_documented_in_runbookInfrastructure Componentdocumented inRunbookhierarchy, which anchor it to whatever it documents. Two edges capture its operational life: alert_rule_triggers_runbookAlert RuletriggersRunbookcross-domain connects detection to procedure, and runbook_mitigates_incidentRunbookmitigatesIncidentcross-domain connects procedure to the live event it resolves. The causal edge postmortem_produces_runbookPostmortemproducesRunbookcausal closes the loop, recording that this procedure exists because something once went wrong and the team chose to remember.