Root Cause

Origin & evolution

The most famous root-cause technique came out of Toyota. Sakichi Toyoda devised the Five Whys in the 1930s, and Taiichi Ohno made it a pillar of the Toyota Production System, describing it as "the basis of Toyota's scientific approach" by which "repeating why five times" exposes both the problem and the fix. Ask why a machine stopped, then why the fuse blew, then why the bearing seized, and you walk from the symptom⚠️SymptomEngineeringA symptom of a problemView reference → back toward a maintenance policy that nobody had set.

A parallel tool arrived from quality engineering. Kaoru Ishikawa developed the cause-and-effect diagram while working with Kawasaki shipyards, formally presenting it in 1968. Its fishbone shape sorts candidate causes into families such as method, machine, material, and measurement, which guards against the Five Whys habit of marching down a single line and declaring victory early.

Both methods carry an assumption🤔AssumptionStrategyA belief taken as true that underpins a strategyView reference → that later thinkers challenged. The Five Whys traces one chain; reality branches. John Allspaw's 2014 essay The Infinite Hows, drawing on Dekker, Conklin, and Leveson, argues that in a retrospective🪞RetrospectiveTeam & OrganisationA team retrospectiveView reference → nothing is sitting there waiting to be found. Causes are constructed by where we choose to start and stop asking, and we declare a "root cause" at the point we get tired of looking. An outage in a distributed system is typically the joint product of several conditions, each necessary and none sufficient alone. Allspaw's repair is to ask "how", which calls for description and resists the tidy single-villain story.

How it works in practice

A payments API starts returning 500s at 09:14. The on-call engineer sees the symptom, a spike in error rate, and opens an investigation🔍InvestigationEngineeringAn investigation into an issue or incidentView reference →. The first answer is "the database connection pool was exhausted", which yields a workaround: raise the pool size. The Five Whys keeps going. Why was the pool exhausted? A slow query held connections open. Why was the query slow? A nightly index rebuild had not completed. Why not? The rebuild job🔨JobUserJob To Be Done: what the user is trying to accomplishView reference → had silently failed for three nights, because nobody monitored it. The named cause is now an absent alert, not a small pool, and the lasting fix targets that.

Root cause vs. its neighbours

• Symptom is what you observe: the error rate, the timeout, the angry support ticket🆘Support TicketCustomer SuccessCustomer support request or issueView reference →. The symptom⚠️SymptomEngineeringA symptom of a problemView reference → is the entry point of diagnosis; the root cause is its terminus. Treating the symptom buys time; removing the root cause buys recurrence-freedom.
• Trigger is the event that set this instance off, such as a traffic spike. The trigger explains now; the root cause explains why now mattered. The same trigger meets a sound system and nothing happens.
• Investigation is the work that connects the two. An investigation🔍InvestigationEngineeringAn investigation into an issue or incidentView reference → produces a candidate root cause; it does not guarantee a true one, which is exactly Allspaw's caution.

In the graph

In the Unified Product Graph, root_cause🪝Root CauseEngineeringAn identified root cause of an issue sits in the engineering and reliability region as the explanatory anchor of an incident🚨IncidentDevOps & PlatformA production incidentView reference → chain. It links downstream through root_cause_causes_symptomRoot CausecausesSymptomcausal and root_cause_causes_bugRoot CausecausesBugcausal, and an investigation_revealed_root_causeInvestigationrevealedRoot Causecausal edge records how it was identified. Closure runs the other way: fix_resolved_root_causeFixresolvedRoot Causecausal marks the condition as addressed. Modelling cause, symptom, investigation, and fix as separate connected nodes keeps a record that a fix targeting only the symptom leaves a root cause queryably unresolved.