A golden-set harness that scores an LLM / extraction pipeline, judges the fields where exact match is unfair, and fails CI when quality regresses. Offline, reproducible, no keys required — docker compose up and go.
"It looks fine on a few examples" is how silent regressions ship. A pipeline that quietly drops from 100% to 48% passes every eyeball test. The fix is to turn "looks fine" into a number, and turn that number into a build gate.
data/goldenset.jsonl (25 labeled rows)
{"raw": "2024-03-14 STARBUCKS #1234 SEATTLE WA $6.75",
"date": "2024-03-14", "vendor": "STARBUCKS",
"amount": "6.75", "category": "Meals"}
System under test: an expense-line extractor (date · vendor · amount · category).
Swap in your own pipeline by replacing one function — the harness only sees inputs and outputs.
A defect we actually hit: the amount parser grabbed the first number in the line — a store #, ZIP, or order id — instead of the dollar amount.
g01: "2024-03-14 STARBUCKS #1234 ... $6.75"
got amount = 1234.00 (grabbed the store #) want 6.75
A human eyeballing five rows misses this. The harness sees it — and the CI gate (≥ 90%) blocks the buggy build from shipping.
The pipeline emits colloquial labels (Dining, Grocery) while the golden set uses canon (Meals, Groceries). Exact string match punishes a correct answer:
| category scored by… | accuracy |
|---|---|
| exact string match | 0.0% |
| LLM-as-judge (semantic) | 100.0% |
Default judge is a deterministic synonym table (zero deps, reproducible CI). Set EVAL_JUDGE=llm + an API key to use a real model for synonyms the table has never seen.
| CI step | What it proves | Result |
|---|---|---|
| pytest | the gate contract holds | 4/4 |
| gate on golden set | row-correct ≥ 90% | 100% |
| bug still caught | buggy run must fail the gate | ✓ |
The judge self-check is honestly < 100%: the offline judge misses Java=coffee. That miss is why the real LLM judge exists — shown, not hidden.
Scoring is plain code (regex + set logic), so results are reproducible and reviewable. docker compose up and CI run with no API key and no network. pytest encodes the gate as a contract — and if a buggy pipeline ever slips past the gate, CI itself fails, because a blind harness is worse than none.
· The golden set is small (25 rows) — a demonstrator, not a benchmark. Real coverage means dozens–hundreds of labeled rows per failure mode. · The offline judge only knows seeded synonyms; novel phrasings need EVAL_JUDGE=llm (tokens, latency, non-determinism — mitigate with temperature=0 + caching). · LLM-as-judge has known biases (verbosity, self-preference). Judged scores are a signal, not ground truth; structured fields stay on exact match. · The shipped bug is a realistic defect deliberately included so the before/after is reproducible — not scraped from a client repo.