Two dashboards read the same 4,000 orders and report different numbers. Neither is broken. growth_daily says the business has 3,119 buyers; finance says 870. One writes count(user_id), the other count(distinct user_id), and nobody noticed for six months. metric-guard surfaces 10 of 10 planted definition drifts — and attaches the number each definition actually produces.
A report that says "filter set differs" gets filed under technical debt and never touched. The same report that says net_revenue — contract says 697,691.42, dashboard says 755,387.65, +57,696.23 (+8.3%) gets fixed that week, because someone can finally say which number went into the board pack.
| metric | contract | dashboard | difference | cause |
|---|---|---|---|---|
| Buyers | 870 | 3,119 | +258.5% | count() where the contract says count(distinct) |
| Conversion rate | 1.60 | 3.59 | +124.0% | denominator is users, contract says sessions |
| Gross revenue | 760,938 | 834,751 | +9.7% | shipping and tax folded into the top line |
| Net revenue | 697,691 | 755,388 | +8.3% | pending orders included |
| Average order value | 223.69 | 240.10 | +7.3% | discount dropped from the numerator |
| Orders | 3,119 | 3,119 | latent | matches only because order_id is unique in this extract |
Plus three findings with no number attached: a timezone that differs from the contract, amounts summed across USD/EUR/GBP without conversion, and a metric on the dashboard (repeat_rate) that no contract entry owns. Checks that cannot be priced say so plainly instead of inventing a figure.
orders produces an identical number under both definitions today, purely because order_id happens to be unique in this extract. Calling that "fine" is how it ships. The report calls it latent and states what would separate the two — any join that fans out.
| # | what it catches |
|---|---|
| M1 | filter set differs — different population under the same metric name |
| M2 | dashboard timezone differs from the contract — daily buckets offset |
| M3 | currency-sensitive amounts summed without conversion |
| M4 | count() where the contract says count(distinct) |
| M5 | metric on a dashboard with no contract entry — nobody owns the definition |
| M6 | metric in the contract that no dashboard uses — dead, or computed by hand somewhere |
| M7 | expression differs — different arithmetic |
| M8 | different denominator on a rate |
M6 is judged across all dashboards at once, not per dashboard. Judged per dashboard it fires constantly — finance is not supposed to show a growth metric — and a check that always fires is a check people learn to ignore.
evidence/drift_report.html renders each disagreement as two dots joined by a line whose length is the gap. Magnitude is not the point — the gap is — and values span three orders of magnitude across metrics, so each row is scaled to its own pair with the actual numbers beside the dots.
Two series, light and dark modes, both validated for colour-vision separation (worst-pair ΔE 24.7 light / 26.8 dark against an ≥8 target). Legend plus direct values, so identity never rests on colour alone, and a table view underneath.
pip install pyyaml duckdb ./scripts/run_evidence.sh # runs everything, writes evidence/ open evidence/drift_report.html
The contract is a single file: metrics/metrics.yml. Six metrics, each with a definition written for a finance reader, the SQL expression, the filter set, the grain, whether it is currency-sensitive, an owner, and a refresh time. Two global keys — timezone and currency — apply to everything, because those are the two assumptions that silently differ between tools.
· Synthetic dataset (4,000 orders, seed 20260804) — a demonstrator of the method, not a benchmark. · This checks that definitions agree. Whether the underlying data is trustworthy is warehouse-quality-gate; whether the job that produced it actually ran is dag-guard. · Checks that cannot be priced are reported without a number rather than with a guessed one.