Data engineering · pipeline monitoring

pipeline-heartbeat

A scheduler can die without throwing an exception. This monitor reads the run ledger and asks whether expected work actually happened.

Public · synthetic demo

The problem

An exit code can be green while the pipeline is empty, late, duplicated, or absent.

A run that exits 0 with 0 rows still looks successful in a scheduler. A disabled cron produces no exception because nothing starts. The ledger is checked for the work that should have occurred, not only for emitted errors.

Input

A deterministic run ledger with a healthy control.

Ledger	Scheduled runs	Actual runs	Role
`daily_orders_etl`	90	87	Eight planted fault patterns
`clean_daily_report`	90	90	Healthy control

{"exit_code":0,"rows_out":0}

Scheduler result: success
Heartbeat result: CRITICAL empty_success

The money shot

Every declared fault pattern was detected while the control stayed quiet.

Fault patterns caught

8 / 8

Missed

Healthy-control findings

Confirmed findings

The 17 findings comprise 5 CRITICAL and 12 WARN records. They are observations, not a production error-rate estimate.

The ten checks

Each silent failure leaves a different ledger trace.

Check	Measured evidence in this run
Liveness + schedule	97 hours since the last run against a 25.5-hour cadence-plus-grace bound; 3 scheduled runs missing
Empty success	`exit_code=0`, `rows_out=0`
Volume anomaly	400 rows against median 10000; band 9688.65–10311.35; n=30
Shape drift	Schema fingerprint transition named at its run
Freshness lag	48.02 hours against a 30-hour maximum
Stalled watermark	2 repeated-watermark findings
Partial sources	3 loaded of 5 expected
Duplicate run	2 distinct run IDs share one batch fingerprint
Runtime drift	2460 seconds against median 180; band 164.43–195.57; n=30

Cold start and state

The monitor refuses to invent a baseline.

Cold start

Volume and runtime require 14 usable historical runs. Until then they appear under Additional data required; the other checks remain active.

Repeat suppression

Finding fingerprints retain first_seen and increment consecutive_runs on a later monitor execution instead of presenting the same condition as new.

Median + MAD

The committed runtime band has median 180 and range 164.43–195.57 at n=30. A zero MAD uses an explicitly reported relative-tolerance fallback.

The period digest

The missing work stays visible beside the anomalies.

The committed digest compares 90 scheduled runs with 87 actual runs, lists the 3 missing UTC execution times, groups 17 findings by severity and check, records the schema transition, and names any evidence still required.

How it differs from my other monitors

Each monitor guards a separate contract.

Sample	What it monitors	Failure case
`data-contract-guard`	Content of an acquired batch	A unit change alters meaning while parsing still succeeds
`scraper-canary`	Page-to-selector extraction contract	A class rename empties prices while a file still arrives
`pipeline-heartbeat`	Execution ledger	The scheduler stops and no exception is emitted

A green heartbeat cannot prove the data is correct. Content validation belongs to the contract guard.

How it's verified

The checker cannot read the planted answer map.

The problem and control ledgers are evaluated before the scoring map is loaded. The checker package never imports or mentions that map. Repeated runs with the same ledger and evaluation time produce byte-identical report directories, and the healthy control exits 0 with 0 findings.

Honest limitations

Eight caught patterns are fixture coverage, not a field detection rate.

The generator places one planted pattern against each declared detector. Catching 8 of 8 shows those paths execute; it does not cover unknown production failures.
The control produced 0 findings because every declared field and threshold was constructed to pass. That is not an estimated false-positive rate.
The synthetic period does not model holidays, daylight-saving changes, or arbitrary cron expressions.
The ledger can show that a schema changed. It cannot prove which deployment or upstream event caused the change.
File-based finding state needs external coordination if multiple monitor processes write concurrently.

Offline ledger checks with committed evidence. jigonyoo.com · Back to hub