Incident

Origin & evolution

Two lineages converge on the modern incident. The operational definition came from IT service management: ITIL framed an incident as an unplanned interruption to a service or a reduction in its quality, and built a lifecycle of detection, logging, categorisation, and resolution around it. That gave the word a precise, ticketed meaning inside enterprise IT.

The response discipline came from somewhere harder. The Incident Command System emerged from 1970s California wildfire response, where uncoordinated agencies fighting the same fire kept failing for organisational reasons rather than tactical ones. It defined a clear command hierarchy and a shared vocabulary so that strangers could coordinate under pressure. Google adapted it directly for software in Incident Management at Google (the Site Reliability Engineering book, O'Reilly, 2017), centred on the three Cs: coordinate, communicate, control, with a named incident commander running the response and engineers freed to fix the problem.

The thinking has moved from minimising incidents to learning🧠LearningValidationAn insight gained from an experimentView reference → from them. The mature view treats incidents as inevitable in complex systems and routine in well-run ones, valued for the signal they carry, with severity levels triggering proportionate response rather than uniform panic.

How it works in practice

At 02:14 a checkout error-rate monitor fires. An on-call engineer acknowledges, sees that errors crossed 5 percent, and declares a Sev2. They take the incident commander role, open a dedicated channel, and pull in a database specialist as operations lead. The commander does not debug; they coordinate, post updates every fifteen minutes, and decide.

Investigation🔍InvestigationEngineeringAn investigation into an issue or incidentView reference → points at a deployment📤DeploymentEngineeringA deployment eventView reference → from 01:50. The team rolls it back, error rate recovers by 02:31, and the incident is downgraded then closed. Seventeen minutes of elevated errors are recorded against the quarter's error budget⏳Error BudgetDevOps & PlatformAn error budget for a serviceView reference →. The next morning, because this crossed the severity bar, the incident triggers a postmortem🩻PostmortemDevOps & PlatformA post-incident reviewView reference →. The fix took minutes; the value is in what the postmortem extracts from those minutes.

Incident vs. its neighbours

• Bug🐛BugProduct SpecificationA defect or unexpected behaviourView reference → is a defect in the code, latent until something triggers it. An incident is the live, customer-affecting event. A bug can sit unnoticed for a year; the incident is the day it takes down checkout. One bug can cause many incidents.
• Root cause🪝Root CauseEngineeringAn identified root cause of an issueView reference → is the underlying condition the investigation lands on. The incident is the surface event; the root cause is the explanation beneath it. They are deliberately separate, because one incident can have several contributing causes.
• Postmortem is the written analysis produced after the event. The incident is what happened; the postmortem is what the team learned. A short incident can warrant a long postmortem, and a noisy one can warrant none.

In the graph

In the Unified Product Graph, incident🚨IncidentDevOps & PlatformA production incident is the anchor of the Operations and Quality region, chosen because it forces every other domain to pay attention: it references a service, may breach a commitment, and produces learning. Its edges encode the lifecycle directly: incident_caused_by_root_causeIncidentcaused byRoot Causecross-domain reaches into the investigation, incident_analysed_in_postmortemIncidentanalysed inPostmortemhierarchy and incident_triggers_postmortemIncidenttriggersPostmortemcross-domain capture the learning loop, incident_breaches_service_level_objectiveIncidentbreachesService Level Objectivecross-domain ties it to the reliability budget, and runbook_mitigates_incidentRunbookmitigatesIncidentcross-domain records how it was handled. That web is why an incident is queryable as more than a ticket: it is the join point between what broke, what it cost, and what changed because of it.