An evidence completeness gate for SOC 2 / ISO 27001 control sets. It catches the pack where the readiness tracker reads 12 of 12 and every control has an attachment — while the artifacts are stale, cover the wrong population, prove the wrong thing, or are flatly contradicted by the access log. Offline, no keys, standard library only.
Public · synthetic demoThe author is an evidence engineer, not an auditor. This tool automates evidence collection and completeness checking. It issues no opinion, certifies nothing, and does not substitute for a licensed CPA firm's SOC 2 examination or an ISO 27001 certification body's audit. Its findings are inputs for that work, produced before fieldwork.
The tracker counts attachments: 12 controls, 24 artifacts, 12 green ticks, book the auditor. Nobody opens the files. Underneath, the quarterly access review was pulled from the HR system so contractors and service accounts were never in it; the backup control has a year of backup-succeeded logs and zero restore tests; four controls are evidenced by artifacts collected months outside their own refresh window; and five people who left still hold production grants. Each one is a finding raised in week one of fieldwork, and the check most teams ship — does this control have at least one attached artifact? — waves all four through.
| check | verdict on the shipped pack |
|---|---|
naive has an artifact? | 12/12 evidenced, 0 gaps → PASS (books the audit) |
| soc2-evidence gate | 14 findings across 7 of 12 controls → BLOCKED |
soc2-evidence · the readiness tracker says the pack is ready
==================================================================
clean pack · gate findings: 0 → PASS ✅
shipped pack · tracker 'has an artifact?': 12/12 evidenced, 0 gaps → PASS (books the audit) ❌
------------------------------------------------------------------
evidence gate on the SAME pack:
findings: 14 by kind: {'stale': 4, 'population': 1, 'artifact_type': 1, 'contradiction': 8}
controls still clean: 5/12
------------------------------------------------------------------
smoking gun 1 · RM-01: evidence is 501d old against a 365d window (136d past due)
smoking gun 2 · AC-03: review covers 34/53 of the roster — never in scope: contractor x11, service_account x8
smoking gun 3 · BC-01: 2x backup_job_log attached, 0x restore_test_report — the control requires the restore test
smoking gun 4 · 8 live grants held by 5 terminated identities, e.g. cara.iyer (write on prod-k8s, 23d after last day)
before (tracker): 12/12 evidenced, 0 issues, audit gets booked
after (gate): 14 findings across 7 controls, pack blocked
The clean pack passes untouched — 0 findings, exit 0. A gate that cries wolf on good evidence gets switched off in a week.
| kind | n | the question the count never asks |
|---|---|---|
| stale | 4 | When was it collected? Newest artifact per control aged against the control's own refresh_days window |
| population | 1 | Who does it cover? Reviewed identities diffed against the authoritative roster |
| artifact_type | 1 | What does it prove? required_artifact_type versus what is actually attached |
| contradiction | 8 | Does the rest of the estate agree? Terminated identities joined to the live grant log |
stale · VM-01: newest artifact EV-023 collected 2026-03-10 is 148d old against a 30d window (118d past due) population · AC-03: review covers 34/53 of the roster (64.2%); 19 identities never in scope: contractor x11, service_account x8 artifact_type · BC-01: 2 artifact(s) attached, all of type backup_job_log — the control requires restore_test_report, of which there are 0 contradiction · cara.iyer (engineer) terminated 2026-07-13 — 23d ago — still holds write on prod-k8s, last used 2026-07-05
Every finding names the number behind it, not just the control id — the window, the age, the days past due; covered over total plus who is missing and in what role; the type required against the types present; the person, their last day, the system, the privilege, the days elapsed. A finding that says "evidence is stale" gets argued with; one that says 118 days past a 30-day window does not.
The tests also assert the naive attachment count misses all four classes, so the finding the gate prevents is demonstrated rather than claimed. Fixture scale: 12 controls · 24 artifacts · 60 identities (53 active) · 90 grants, generated from a seed by scripts/make_data.py — CI regenerates them and fails if the result isn't byte-identical.
What breaks is the four dimensions no attachment count measures: when the artifact was collected, who it covers, what it actually proves, and whether the rest of the estate contradicts it. Those are data-engineering questions — date windows, population diffs, joins — not legal ones, and they are cheap to answer continuously against exports most organisations already produce from Okta, AWS, GitHub, Jira and their HRIS. Finding a 19-identity population gap in March costs a script run; finding it during fieldwork costs a re-test.
Run it as a gate: soc2_evidence.run exits non-zero on any finding, so it slots into a nightly job or CI and the pack is checked continuously instead of the week before fieldwork. --report out.json writes the findings as structured JSON for a ticket queue.
make test # 8 passed make demo # the tracker-vs-gate contrast above make gate # the shipped pack — exits non-zero, pack blocked make gate-clean # the no-false-alarms case — exits 0 python -m soc2_evidence.run --controls data/controls.json --evidence data/evidence.csv --as-of 2026-08-05
· Synthetic data (12 controls · 24 artifacts · 60 identities · 90 grants) — a demonstrator of the method, not a report on any real organisation. No client data appears here. · This checks whether the evidence is complete. It does not judge whether a control is designed adequately — that is a licensed CPA firm's call, and nothing here is an opinion or a certification. · Control definitions are hand-written JSON. Mapping a real framework, control versioning, and exceptions / compensating controls are out of scope. · It checks metadata and populations, not artifact contents: it cannot tell a genuine restore-test report from an empty file named like one. Chain of custody is the next layer, deliberately skipped. · Four checks are the common failure modes, not a complete taxonomy — sampling adequacy, segregation of duties and vendor evidence are not covered. · It reports findings; it does not remediate. Revoking the orphaned grants is still a human decision.