Investigation

Origin & evolution

Disciplined fault diagnosis grew up in two places at once. Manufacturing gave it the gemba walk and the Five Whys, Toyota's habit of going to where the problem happened and asking why until something systemic appears. Software gave it the scientific debugging tradition, codified in works such as Andreas Zeller's *Why Programs Fail*, which frames debugging as hypothesis🧪HypothesisValidationA testable belief about a solutionView reference → and experiment🧫ExperimentValidationA test designed to validate a hypothesisView reference →: observe the failure, form a theory of the cause, predict what you would see if the theory held, then test it.

The cultural shift that reshaped investigation was blamelessness. John Allspaw and the team at Etsy popularised the blameless postmortem, arguing that an engineer who fears punishment will give the account that protects them, not the account that explains the failure. Remove the threat☠️ThreatSecurityA specific security threatView reference → and you get the second story, the detailed how-and-why that teaches the organisation something. Allspaw's later Infinite Hows pushed further: an investigation does not uncover a pre-existing root cause🪝Root CauseEngineeringAn identified root cause of an issueView reference → so much as construct an explanation, so where it chooses to stop is a judgement, and a careful investigation keeps asking "how" after the first comfortable "why".

How it works in practice

A scheduled report stops arriving. The symptom is narrow: one daily email, missing. The investigation begins with a timeline. Logs show the report job🔨JobUserJob To Be Done: what the user is trying to accomplishView reference → started and exited cleanly, so the failure is downstream of generation. The engineer forms a hypothesis: the file uploaded but the notification did not send. A quick check of the storage bucket confirms the file is there, falsifying the "report never generated" theory and pointing at the delivery step. The next hypothesis, an expired API credential for the email provider, is confirmed by a 401 in the provider logs. The investigation has now produced two artefacts: a revealed bug🐛BugProduct SpecificationA defect or unexpected behaviourView reference →, the unhandled 401, and a candidate root cause, credentials with no rotation alert. Both feed the fix.

Investigation vs. its neighbours

• Incident🚨IncidentDevOps & PlatformA production incidentView reference → is the declared event that an investigation serves. The incident🚨IncidentDevOps & PlatformA production incidentView reference → is the organisational container with an owner and a clock; the investigation is the diagnostic work inside it. Not every investigation needs🫀NeedUserA user need, pain, desire, or constraintView reference → an incident, and a long incident may hold several investigations.
• Root cause is an output of the investigation, never a guaranteed one. The investigation🔍InvestigationEngineeringAn investigation into an issue or incident produces a candidate; whether it is the true underlying condition is the open question Allspaw warns about.
• Postmortem🩻PostmortemDevOps & PlatformA post-incident reviewView reference → is the written record and the learning🧠LearningValidationAn insight gained from an experimentView reference → ritual after the fact. The postmortem🩻PostmortemDevOps & PlatformA post-incident reviewView reference → documents what the investigation found and what changes follow; the investigation is the live diagnosis, the postmortem its durable account.

In the graph

In the Unified Product Graph, investigation🔍InvestigationEngineeringAn investigation into an issue or incident sits in the engineering and reliability region as the diagnostic spine of an incident chain. It points to its findings through investigation_revealed_bugInvestigationrevealedBugcausal and investigation_revealed_root_causeInvestigationrevealedRoot Causecausal, and the resulting change is tied to it by fix_derived_from_investigationFixderived fromInvestigationcausal. Its target is recorded by service_investigated_via_investigationServiceinvestigated viaInvestigationcausal, anchoring the diagnosis to the component under examination. Holding the investigation as its own node, between symptom and fix, preserves the chain of reasoning: a graph reader can see not just that a fault was fixed but how it was diagnosed and what evidence justified the conclusion.