Metric Quality Assessment

Origin & evolution

The intellectual anchor is Goodhart's Law. Charles Goodhart, a British economist, wrote in 1975 that "any observed statistical regularity will tend to collapse once pressure is placed upon it for control purposes," drawing on how monetary targets stopped behaving once the Bank of England steered by them. The popular phrasing came later: the anthropologist Marilyn Strathern condensed it in 1997 to "when a measure becomes a target, it ceases to be a good measure." That sentence is the reason metric quality is a discipline and not an afterthought.

Adjacent ideas sharpened the practice. Donald Campbell's law made the same point about social indicators and corruption. The lean and analytics movement added the vanity-metric critique, popularised through Eric Ries and others, which warns against numbers that rise reassuringly (total registered users, raw page views) while telling you nothing you can act on. The working synthesis is a set of tests a healthy metric should pass: it moves in response to decisions⚖️DecisionStrategyA recorded decision with context, rationale, and consequencesView reference → you control, it resists gaming, it correlates with real value rather than activity, and someone owns the action it implies.

How it works in practice

A support team reports "tickets closed per day" and it climbs steadily, so leadership is pleased. An assessment runs four checks. Actionable: partly, the team can close faster. Gameable: badly, agents close-and-reopen, or shut tickets before resolution to hit the count. Vanity: high risk🎲RiskComplianceA risk to the product or businessView reference →, closed volume rises even when customers are angrier. Good proxy: weak, the goal is resolved problems, and closures correlate loosely with resolution. The metric fails three of four. The team keeps it as an operational signal but demotes it from a target, and pairs it with reopen rate and a resolution survey, the two numbers the close-count was quietly hiding. Within a quarter the gaming behaviour disappears, because no single number is worth gaming any more.

Metric quality assessment vs. its neighbours

• Metric is the thing being judged; the assessment is the judgement. A metric reports a value; the assessment reports whether that value can be trusted to mean what people assume.
• North star metric is a single chosen measure of core value, and it is the metric most exposed to Goodhart pressure precisely because the whole company steers by it. The assessment is what should vet a north star before it is anointed, and re-vet it as behaviour adapts.
• Data quality rule🧹Data Quality RuleData & AnalyticsA data quality validation ruleView reference → checks whether the underlying numbers are correct (complete, fresh, non-duplicated). The assessment checks whether a correct number is a good one to optimise. Clean data can still feed a terrible metric.

In the graph

In the Unified Product Graph, a metric quality assessment lives in the strategy region as an evaluative node attached to the metric it scrutinises, through metric_assessed_by_metric_quality_assessmentMetricassessed byMetric Quality Assessmenthierarchy. Keeping it as a distinct node, separate from the metric, records the judgement and its reasoning as first-class history: a team can see that a metric was once flagged as gameable, what was decided, and whether the concern was addressed. Neighbours such as north_star_metric🔢MetricStrategyA unified metric that measures progress, health, or behaviour across the productView reference → and data_quality_rule🧹Data Quality RuleData & AnalyticsA data quality validation ruleView reference → sit nearby, so the graph distinguishes "is this number correct?" from "is this number wise to chase?", the two questions teams most often blur into one.