Menu

1 / 4

Sensitive Data Discovery in Python: Build a PII Scanner That Validates, Not Just Matches

DEV Community: database·Audrey Xu·10 days ago
#aEePDANm
#devtodatabase#devto#dev#column#values
Reading 0:00
15s threshold

Before you can encrypt, mask, or delete personal data, you need to know where it lives. That sounds obvious until someone asks “which of our 400 tables hold card numbers?” and the honest answer is a shrug. GDPR Article 30 expects a controller to keep a record of processing that describes “the categories of personal data”, and you cannot write that record from memory.

That is the job of sensitive data discovery : scan the stores you own, find columns that hold PII, and label them. The usual first attempt is a folder of regexes. It finds everything, including order numbers, timestamps, ZIP+4 codes and SKUs, and after a week nobody reads its report.

In this post we build a small scanner in Python that does the boring parts properly:

Walks a SQLite database and a folder of CSV exports.

Samples each column with reservoir sampling (bounded memory, reproducible).

Finds candidates with regexes, then validates them: Luhn and network prefixes for cards, ISO 7064 mod-97 for IBANs, SSA rules for SSNs,…

Continue reading — create a free account

Join HashtagPLUS to read full articles, follow hashtags, vote, and join the conversation.