"Where does personal data actually live, is it in a region our policy allows, who can read it, and is it encrypted the way we require?" Most teams answer that from memory. gcp-data-residency-pii-auditor answers it from the data. It finds personal data in BigQuery and Cloud Storage using validated value patterns plus column names, then checks residency, access, encryption and retention against a policy you write. The design rule is that raw values are counted and dropped: reports contain column names, PII types, counts and masked examples only, and models see redacted metadata.
Finding PII without a pile of false positives
Regexes alone flag every 16-digit order number as a card. So the detectors validate: card numbers must pass Luhn, IBANs must pass mod 97, US SSNs must satisfy area rules, and there are detectors for email, phone and public IPv4. Names, addresses, dates of birth and locations are recognised from column names only. It samples the first 50 rows of native BigQuery tables and the first 64 KB of CSV, JSONL, TSV and text objects in allow-listed buckets. Each column is classified as name+values, values or name_only, and a data class (public, internal, confidential, restricted) is assigned. A column that matches PII patterns under a name that does not say so raises rule P005.
The policy rules
| Rule | Checks |
|---|---|
| P001 residency | Default: restricted data US-only, confidential US or EU |
| P002 access | Public, domain-wide, or too many readers (default max 10 on restricted data) |
| P003 encryption | Customer-managed key required for restricted data |
| P004 retention | Retention settings against policy |
| P005 undocumented PII | PII in columns whose names don't say so |
What the live audit shows
I planted canary data in a real project: two BigQuery datasets (pii_canary_us, pii_canary_eu) and a bucket. The audit above covers 6 assets. The EU patients table holds date of birth, email and IBAN, which makes it restricted data outside the allowed US location: the critical finding, with the fix that BigQuery datasets cannot change location in place, so you create a replacement and copy. Three high findings are restricted data (the EU table, the US customers table and an exported people.csv) under Google-managed rather than customer-managed keys. The medium finding is a contact_info column whose values match email patterns while the name gives no hint, which is exactly the case name-based review and access controls would miss.
Validation against reality
| Check | Result |
|---|---|
| Offline tests | 60 pass: each validator on valid and invalid values, masking, classification, every finding rule, no-leak properties across report, API, CLI and model payload |
| Planted canaries in real BigQuery and GCS | 14 of 14 planted PII columns found across 3 assets; 0 false positives among other columns and benign assets; local and deployed results identical |
| Model eval (mock, gemini-2.5-flash, gpt-5-nano) | Findings identical to the model-free run; no personal value ever sent to a model (audited) |
| Deployed | Unauthenticated, other project, other dataset and openai provider all 403; response bodies contain no planted value |
The first run on a real dataset, my lakehouse one, produced a false positive: a column named is_credit_card is a payment-type flag, not a card number. I fixed it with type-aware classification and regression tests, and the dataset now audits as having no personal data.
Limits
Sampling is the first 50 rows or 64 KB, not a statistical sample, so rare PII deeper in a table is missed. Name-only classification (unsampled data, external tables, views) is low confidence. Free text is not searched beyond the listed patterns, and this does not replace Cloud DLP. Residency is the dataset or bucket location, not lineage, so copies elsewhere are not found. Access is read from dataset ACLs and bucket public access; it does not expand groups or evaluate project IAM, authorized views, policy tags or row-level security, so a table protected by policy tags can still be reported. The domain-wide access rule could only be unit-tested because BigQuery rejects invalid domains and a real one could belong to someone. Phone and SSN formats are US-centric.
Try it yourself
git clone https://github.com/soodrajesh/gcp-data-residency-pii-auditor
cd gcp-data-residency-pii-auditor
.venv/bin/pii audit my-project --no-sample --llm none # inventory by column names only; reads no data
cp deploy.env.example deploy.env && ./scripts/up.sh && ./validation/plant.sh plant
./scripts/down.sh📢 Have questions or feedback? Drop a comment below or connect with me on Twitter/X@spysood!