Data engineering · privacy & GDPR

pii-guard

A pre-release gate for "anonymized" exports. The customer table here passed its masking review and is still personal data: 652 PII instances in columns nobody labeled, masks that are 1:1 with the subject, 87.5% of rows unique at k=1, and an erasure request that only touched one table. Offline, standard library only, no keys.

Public · synthetic demo
The problem

"The PII columns are masked" is not "this file is anonymous."

A customer table is prepared for an analytics vendor. customer_name, email and phone are masked, someone regexes those three columns, finds nothing, signs off. The export ships. Nobody looked at support_notes, nobody counted how many distinct masked values there are, nobody computed k over birth_date + postal_code + gender, and nobody checked whether last month's erasure request removed the person from the order table. Each is a separate way the file is still identifiable, and a name-based review sees none of them.

The money shot

The same shipped export, checked two ways

① Naive masking review vs pii-guard
checkverdict on the shipped export
naive are the PII columns masked?0 findings → PASS (ships identifiable data)
pii-guard652 hidden PII · 799:799 masks · 567 ids reversed · 87.5% unique · 16 DSAR leftovers → BLOCKED
0 → 652PII instances (naive → guard)
87.5%rows unique (min k = 1)
567hashed ids reversed of 799
16DSAR leftovers in 3 tables

The declared columns really were masked — the naive check passes for a real reason. It is blind by construction, so the contrast is a computed number rather than a claim.

② Four separate ways it is still personal data
columnkindinstancesrows
support_notesemail201201
support_notesphone157157
shipping_instructionsphone106106
legacy_refnational_id7171
support_notesnational_id6565
shipping_instructionsemail5252

1 · Unlabeled columns. The scan is column-agnostic — every column, every cell, no name allow-listing. legacy_ref gets the same treatment as email, because PII does not live where the data dictionary says.

2 · Masking is not anonymization. 799 distinct masked emails for 799 subjects is 1:1 — a stable pseudonym the vendor can re-link across deliveries. And user_id is an unsalted sha256 of the address: the dictionary attack's wordlist is email_log, another table in the same delivery, and it recovers 567/799 (71.0%) from 568 addresses. The test re-hashes a recovered address and reproduces the stored id, so the inversion is real, not asserted.

3 · Quasi-identifiers. k over birth_date+postal_code+gender is min k=1, with 699/799 rows unique (87.5%) and a k-distribution of {1: 699, 2: 40, 3: 60}. No name is needed to re-identify a row that is alone in its group.

4 · Erasure completeness. DSAR-2026-0117 (CUST-000042) was erased from the primary table. 16 references survive in 3 tables — {'email_log': 4, 'orders': 6, 'support_tickets': 6} — matched by four identifier forms: {'email': 4, 'masked_email': 2, 'customer_id': 8, 'user_id': 2}. Keyed columns by equality, free text by containment.

③ The run, verbatim
  pii-guard · an 'anonymized' customer export, on its way to a vendor
  ====================================================================
  clean export      · PII findings: 0 · min k: 13 · unique rows: 0.0% · mask ratio: 0.001  → PASS ✅
  shipped export    · naive 'are the PII columns masked?' findings: 0  → PASS (ships it) ❌
  --------------------------------------------------------------------
  pii-guard on the SAME export:
    1 · unlabeled columns: 652 PII instances the name-based check never looks at
        support_notes          email         201 instances in 201 rows
        support_notes          phone         157 instances in 157 rows
        shipping_instructions  phone         106 instances in 106 rows
        legacy_ref             national_id    71 instances in  71 rows
        support_notes          national_id    65 instances in  65 rows
        shipping_instructions  email          52 instances in  52 rows
    2 · masking is not anonymization: 799 distinct masked emails for 799 subjects (1:1)
        user_id is an unsalted sha256 of the address — 567/799 recovered (71.0%) from 568 addresses in email_log
    3 · k-anonymity(birth_date+postal_code+gender): min k=1 · 699/799 rows unique (87.5%) · k-distribution {1: 699, 2: 40, 3: 60}
    4 · DSAR DSAR-2026-0117 (CUST-000042): erased from the primary table, 16 references survive in 3 tables {'email_log': 4, 'orders': 6, 'support_tickets': 6}
        matched by {'email': 4, 'masked_email': 2, 'customer_id': 8, 'user_id': 2}
  --------------------------------------------------------------------
  smoking gun · support_notes row 1: cyd.abernathy@globex.example.org  (a column nobody labeled)
  smoking gun · CUST-000001 user_id -> xan.gorman@acme.example.com  (one hash lookup)

  before (naive): 0 findings, the export ships
  after  (pii-guard): 652 hidden PII instances · 799:799 masks · 567 ids reversed · 87.5% of rows unique · 16 DSAR leftovers — blocked
④ Verified, not asserted
9 passedpolicy tests (pytest)
min k = 13clean export — 0 findings, exit 0
fixtures deterministic, CI diffs them

The policy is a test suite, not a promise: the anonymized export must pass with zero findings, each of the four sabotage classes must be caught on its own, and the naive review must miss all four. CI regenerates the fixtures from scripts/make_data.py and fails if they are not byte-identical. The clean export exits 0, the shipped one exits non-zero — a guard that lets identifiable data through fails its own CI.

Fixture safety

Nothing in the fixtures can collide with a real person.

Every identifier in the fixtures is deliberately non-resolvable: the RFC 2606 reserved domains (example.com / example.org / example.net) plus the .invalid TLD, phone numbers drawn only from the reserved 555-01xx fictional block, and national-ID strings built from ranges that are never issued (SSN area 000 / 666 / 9xx, resident-registration month fields 13–19). The 800 subjects and 4 related tables are generated from a seed. Nothing in the fixtures can collide with a real person.

Why it matters for GDPR / CCPA work

"We masked the name column" is an argument. min k=1 is a measurement.

Anonymous data is out of scope; pseudonymous data is not. Recital 26 turns on whether re-identification is reasonably likely. Three of the four layers produce numbers a DPIA actually needs — how many rows are unique on the quasi-identifiers, whether the pseudonym is stable across deliveries, whether the key is invertible with data the recipient already holds. The fourth is the Art. 17 problem nobody schedules time for: erasure is complete only when the subject is gone from every table, under every identifier form they ever had.

Reproduce it
docker compose up     # naive review vs the guard (offline)
make demo             # same, without Docker
make test             # the policy tests — 9 passed
make gate             # the release gate on the shipped export (non-zero = blocked)
make gate-clean       # the no-false-alarms case: the clean export must exit 0
Honest limitations
· Synthetic data (800 subjects + 4 related tables) — a demonstrator, not a benchmark. The numbers
  above come from the fixtures in data/, not from a client table.
· Detection is regex + shape. It misses obfuscated PII ("ada dot arden at example dot com"),
  free-text names and addresses, and anything in a non-Latin script. Every finding carries the
  cell excerpt so a human confirms it.
· k-anonymity is a floor, not a full privacy guarantee. It says nothing about attribute
  disclosure — l-diversity, t-closeness and differential privacy are out of scope.
· It guards one export boundary. It does not rewrite the data or fix the pipeline.
This is a synthetic sample demonstrating the method. Inspect the detectors, the re-identification math, and the reproducible fixtures ↗