Data engineering · orchestration review

dag-guard

Reads your Airflow DAG files with astno Airflow install, no imports, no side effects — and reports what a green DAG in the UI will not. The sabotaged DAG here is valid Python that parses without warnings. It also queues 90,816 backfill runs the moment it deploys, writes revenue twice from two unordered tasks, and pages nobody when it fails at 3am.

Public · synthetic demo
The problem

The DAG renders perfectly. That is the problem.

Nothing about broken_pipeline.py looks wrong in review. It parses, it renders, the graph makes sense. The defects are all configuration — a missing max_active_runs, an append-only write with retries enabled, two writers to one table with no edge between them. They only become visible in production, usually at the worst moment.

The money shot

Both DAGs, checked

① 12 planted defects, 12 caught, 0 false alarms
planted defects      : 12
caught               : 12
missed               : none
false alarms on clean: 0

OK  12/12 caught, 0 false alarms
90,816backfill runs O1 computes
5 / 5 / 2critical / high / medium — exit 1
0findings on the clean DAG — exit 0

Exit code 1 on high or critical, so CI blocks the merge. The clean DAG exits 0 and the same job stays quiet.

② Three that cause incidents rather than annoyance

O1 computes the actual number. catchup=True is not a defect by itself — it is a defect when start_date is far in the past and the schedule is short. The check does the arithmetic: (now − start_date) / interval. Here that is 90,816 runs. A reviewer skimming the file sees catchup=True and moves on; the number is what makes it real.

O6 and O11 are a pair. O6 says the write is not idempotent. O11 says two of them run with no ordering. Either alone is survivable. Together they mean the revenue number depends on scheduler timing — the kind of bug that gets found in a board meeting.

O10 is about worker economics. A sensor in poke mode holds its slot while it waits. With a 15-minute schedule and a 24-hour timeout, every run parks a worker for up to a day. The pool fills with sensors waiting for upstreams that are themselves waiting for a free slot.

③ The 12 checks
#severitywhat it catches
O1criticalcatchup=True over a long backlog — 90,816 runs queue on deploy
O3criticaldependency cycle — the DAG can never complete
O5criticalno max_active_runs on a short schedule — two writers, one table
O6criticalappend-only write a rerun duplicates — silently double-counted revenue
O11criticaltwo writers to one target with no edge — row count depends on finish order
O2highretries not set — one 503 ends the whole run
O7highdepends_on_past with no retry budget — one failure blocks every future run
O9highno execution_timeout — a hung request holds a worker slot forever
O10highsensor waits longer than the schedule interval in poke mode
O12highno on_failure_callback — it fails at 3am, someone notices at 9
O4mediumtask with no dependencies either way — readers may get the stale version
O8mediumno SLA anywhere — late is not failed, so a stale dashboard looks live

Every finding carries three fields, not one: evidence (what is in the file), impact (what happens in production), fix (what to change). A finding that only says "missing retries" gets ignored; one that says what breaks does not.

Why parse instead of import

Importing a DAG is the obvious way to inspect it, and the wrong one.

Importing executes module-level code, so it needs the whole Airflow install, every connection the file touches, and any credential a top-level client happens to construct — a slow, fragile CI job that fails for reasons unrelated to the DAG. dag-guard parses with ast instead: it runs on a plain python:3.11-slim image in about a second and cannot have side effects.

The tradeoff is stated honestly: it sees what is written in the file, not what a dynamically generated DAG resolves to at runtime.

Reproduce it
python3 tests/test_checks.py      # the 12/12 assertion above
./scripts/run_evidence.sh         # full output for both DAGs into evidence/
python3 -m dagguard.cli dags/*.py --now 2026-08-04
Honest limitations
· Static analysis: dynamically generated DAGs resolve at runtime and are out of reach.
· Synthetic DAG pair (one clean, one with 12 planted defects) — a demonstrator, not a benchmark.
· It checks the orchestration. Whether the data itself is correct is warehouse-quality-gate;
  whether a job silently moved zero rows is pipeline-heartbeat.
This is a synthetic sample demonstrating the method. Inspect the checks, the fixtures, and the reproducible evidence ↗