Monitor

Origin & evolution

The deeper root is control theory. Rudolf Kálmán defined observability in 1960 as the degree to which a system's internal state can be reconstructed from its external outputs. Decades later that word migrated into software, and the distinction between monitoring and observability hardened into a real argument. Monitoring asks known questions of known signals; observability is the property that lets you ask questions you did not anticipate when you instrumented the thing.

Google's Site Reliability Engineering book (2016) gave the field its working vocabulary. It separated black-box monitoring, which probes a system from outside as a user would and reports active symptoms⚠️SymptomEngineeringA symptom of a problemView reference →, from white-box monitoring, which inspects internal state through logs and instrumented endpoints. The same chapter named the four golden signals: latency, traffic, errors, and saturation. The guidance was blunt. If you can measure only four things about a user-facing system, measure those.

In parallel, Peter Bourgon's 2017 post crystallised the three pillars of observability, metrics🔢MetricStrategyA unified metric that measures progress, health, or behaviour across the productView reference →, logs, and traces, as the telemetry types a monitor consumes. The framing stuck, partly because it described daily reality and partly because vendors already sold a product for each pillar. The current debate pushes past three discrete silos towards unified, high-cardinality event data, but the monitor's job🔨JobUserJob To Be Done: what the user is trying to accomplishView reference → under either model is the same: turn raw telemetry into a health judgement.

How it works in practice

A payments team runs a checkout service handling 4,000 requests per second. They wire a white-box monitor to the golden signals: p99 latency, request rate, the ratio of HTTP 5xx responses, and database connection-pool saturation. The latency monitor evaluates a rolling one-minute window and compares it to a threshold derived from the service level objective🥅Service Level ObjectiveDevOps & PlatformA service level objective (SLO)View reference →.

At 02:14, p99 latency climbs from 180ms to 1.9s while the error rate holds flat. No instance has crashed; nothing is technically down. The monitor catches the degradation precisely because it watches a symptom users feel, not just process liveness. A separate black-box monitor, probing the public endpoint every 30 seconds, confirms real requests are slow rather than a metrics artefact. Together they distinguish a genuine incident🚨IncidentDevOps & PlatformA production incidentView reference → from a noisy gauge.

Monitor vs. its neighbours

• Alert rule🛎️Alert RuleDevOps & PlatformAn alerting ruleView reference →. A monitor produces a continuous signal and a health verdict. An alert_rule🛎️Alert RuleDevOps & PlatformAn alerting ruleView reference → is the policy layered on top that decides when that verdict warrants waking a human. One monitor can feed several alert rules at different thresholds, which is why the graph models them as separate entities.
• Service level indicator🌡️Service Level IndicatorDevOps & PlatformA service level indicator (SLI)View reference →. An SLI is a precise, defined measurement of one aspect of service health, such as the fraction of requests served under 300ms. A monitor is the running mechanism that observes that measurement over time. The SLI defines what good means; the monitor reports where you currently stand against it.
• Observability tooling. Observability is a system property: how answerable your system is. A monitor is a concrete instrument that exercises that property to answer one recurring question. A highly observable system with no monitors tells you nothing until you go looking.

In the graph

In the Unified Product Graph a monitor lives in the operations region, bound to the things it observes. A product connects through product_monitored_by_monitorProductmonitored byMonitorhierarchy, an infrastructure component🖧Infrastructure ComponentDevOps & PlatformAn infrastructure component (server, CDN, etc.)View reference → through infrastructure_component_monitored_by_monitorInfrastructure Componentmonitored byMonitorhierarchy, and a running service through monitor_watches_serviceMonitorwatchesServicecross-domain. Detection flows outward through monitor_triggers_via_alert_ruleMonitortriggers viaAlert Rulehierarchy, which keeps observation👁️ObservationUser ResearchA specific behaviour or statement observedView reference → and notification as distinct, separately tunable steps. That separation matters: it lets you query which parts of a product have no monitor at all, the blind spots that surface only during an incident.