Breakages planted
5
Data engineering · extraction monitoring
scraper-canary
A scraper that crashes gets attention. This one catches the costlier failure: a job that exits cleanly while the extracted data is wrong or incomplete.
Public · synthetic demoThe problem
A clean exit does not mean the extraction contract held.
Class names move, pagination shortens a list, currencies change meaning, and an upstream job can freeze yesterday's data. Every case can still produce a file. The canary checks the result before downstream systems treat that file as current and complete.
Input
Eight committed snapshots of a fictional catalogue
| Day | Extracted result | Expected check |
|---|---|---|
| day01–03 | 40 rows, USD, advancing timestamps | clean baseline |
| day04 | 40 rows; all 40 prices empty | field fill rate |
| day05 | 40 rows; every currency is EUR; selector recovered | enum stability; no recovery alarm |
| day06 | 20 rows | row-count band |
| day07 | 40 rows; timestamp unchanged from day06 | freshness |
| day08 | 35 rows; 5 cards moved from div to li | row count + selector hit rate |
The money shot
Every planted silent failure was stopped, without flagging the clean recovery.
Caught
5
Missed, silent
0
False positives
0
The zero false-positive count includes day01–03 and day05's recovered price selector. Day05 is still a real breakage because USD changed to EUR.
The seven invariants
Different silent failures leave different evidence.
| Invariant | What it checks | Result in this run |
|---|---|---|
selector_hit_rate | matched cards versus accepted history | day06 and day08 critical |
field_fill_rate | at least 95% present per field | day04 critical |
row_count_band | median+MAD accepted-count band | day06 and day08 critical |
type_and_format | decimal prices and integer stock | day04 critical |
enum_stability | new currency meaning | day05 critical |
structure_drift | tag-and-class skeleton change | WARN only |
freshness | source timestamp advances | day07 critical |
Why median + MAD
A zero-row outage widens a mean-and-standard-deviation band enough to hide the next short run. With the committed accepted history, the median stays 40 and the MAD band stays 37.0–43.0. Both 20 and 35 remain outside it.
Where it stops
The canary separates evidence from explanation.
A redesign can change the skeleton without damaging extraction. It stays WARN and appears under likely causes, never confirmed evidence by itself.
The first three accepted runs emit insufficient-baseline warnings. The canary does not manufacture a row-count band from less history.
Failed days do not teach the baseline
A run enters the statistical ledger only when it has no CRITICAL finding. Freshness alone compares the immediately preceding observed timestamp, so a rejected short run cannot hide the next day's stale timestamp.
How it differs from my other monitors
Each monitor guards a different contract.
| Sample | What it monitors | Failure case |
|---|---|---|
data-contract-guard | Schema, range, and enum of an acquired table batch | Firmware changes °C to °F and changes what values mean |
pipeline-heartbeat | Execution ledger: did the job run? | The scheduler dies and nobody notices for three days |
scraper-canary | Webpage-to-selector extraction contract | A class rename empties every price while the file still arrives |
How it's verified
The answer key is loaded only after every decision.
The canary package never reads or mentions planted_breakages.json. The standard-library runner evaluates all eight snapshots first, then loads the answer key for scoring. Seventeen tests cover parsing, median+MAD behavior, all seven invariants, severity, imports, and truth isolation.
Honest limitations
Five synthetic breakages are a test fixture, not a production error rate.
- The baseline has three accepted days. It does not model weekly or seasonal catalogue volume.
- All pages are committed snapshots of a fictional target. No live site or terms-of-service boundary is tested.
- Structure drift identifies a candidate cause, not which deployment changed the page.
- The run missed 0 planted breakages and produced 0 false positives in eight days. Those zeros do not estimate performance on unseen production drift.
- Freshness trusts the page timestamp. A source can advance that stamp while serving old records.
Synthetic snapshots, offline execution, and committed evidence. jigonyoo.com · Back to hub