gdpr-officer

gdpr-officer solves GDPR-compliant data erasure with a single operation, making PII permanently unreadable across all tables when the "right to be forgotten" is invoked — without modifying or deleting any data, preserving analytics integrity. It also protects sensitive data during breaches and leaks.

Erasing a customer's PII from a data lake/house across multiple tables is difficult. Finding and deleting every copy is cumbersome, error-prone, and breaks referential integrity.

gdpr-officer uses the crypto-shredding pattern. Delete a key, forget a customer. It:

• encrypts PII with a unique encryption key per customer before data is loaded into the platform, restricting access and protecting it during leaks;
• stores encryption keys in a separate key store outside the data platform, ensuring PII stays unreadable even when the lake/house is breached;
• when GDPR erasure is requested, forgets the customer by deleting their encryption key, rendering their PII permanently undecryptable across every table, while non-PII columns remain intact for analytics;
• enables authorised decryption for any customer whose key is maintained.

Install

pip install gdpr-officer              # Core library with local DuckDB backend
pip install gdpr-officer[gcp]         # Adds Google Cloud Firestore backend

How it works

Source → Extract → gdpr-officer → Load (PII encrypted) → Data Platform → dbt
                       ↕
               Separate Key Store
          (outside the data platform)

gdpr-officer sits between your extract and load steps. It encrypts PII columns using AES-256-GCM with a unique 32-byte key per customer. Each value gets its own random nonce, so identical plaintext produces a different ciphertext each time, to avoid pattern detection. The encrypted output is a base64 string containing the nonce and ciphertext.

The encryption key is stored separately outside the data platform. Even with full access to the data lake/house, PII cannot be decrypted without the key store.

Usage

Encrypt

Add between your extract and load steps. customer_id takes the identifier column. pii takes a list of columns to encrypt. Everything else passes through unchanged. For repeated use across multiple sources, these can be defined in a YAML configuration file.

from gdpr_officer import PiiEncryptor

officer = PiiEncryptor(
    key_backend="gcp_firestore",
    key_backend_config={"project": "my-gcp-project"},
)

df = extract_from_source()
df = officer.encrypt_df(
    df,
    customer_id="<your_customer_id_column>",
    pii=["<pii_column_1>", "<pii_column_2>", ...],
)
load_to_warehouse(df)

Lists of dicts and single rows are also supported:

rows = officer.encrypt_rows(rows, customer_id="...", pii=[...])
row = officer.encrypt_row(row, customer_id="...", pii=[...])

Generalise

Enables analytical use of sensitive values. Creates a generalised version of a column, next to the encrypted original. Such as: an age group from a birthdate, a region from a nationality.

!Consult your DPO whether your intended generalisation, and/or the combination of generalised columns pose a risk to re-identification, and whether it sufficiently meets compliance requirements!

from gdpr_officer import PiiEncryptor, age_group

df = officer.encrypt_df(
    df,
    customer_id="customer_id",
    pii=["email", "phone", "birthdate"],
    generalise={"birthdate": ("age_group", age_group(edges=[0, 18, 30, 40, 50, 65]))},
)
# birthdate is encrypted as usual; a new age_group column holds "18-29",
# "30-39", ... in the clear.

Each entry maps a source column to a (new_column, callable) pair. Two rules: the source column must also be listed in pii, and the target column must be a new name.

Built-in rules:

An unmapped or unparseable value returns the rule's default (None if unset), never the original value.

age_group is intended for events where age does not change, such as age at diagnosis. Age is computed at encryption time

Alternatively, birthdate could be generalised into birthyear, to enable age-based analytics. !Consult your DPO whether this generalisation, and/or the combination of generalised columns pose a risk to re-identification!

Configuring in YAML

Each source has two sections: pii_columns to encrypt, and generalise. Anything not listed passes through unchanged. See gdpr_officer/config_template.yaml for a fuller template.

sources:
  - name: customers
    customer_id_column: customer_id

    pii_columns:          # encrypted per customer
      - email
      - phone
      - birthdate         # generalised columns are encrypted too
      - nationality

    generalise:           # adds a generalised column next to the encrypted original
      birthdate:
        rule: age_group
        to: age_group     # required: name of the target column
        edges: [0, 18, 30, 40, 50, 65]

      nationality:
        rule: mapping
        to: region
        groups:
          EMEA: [NL, DE, ES]
          OCEANIA: [AU, NZ]
        default: Other

With a config file, encrypt_batch applies a source's rules automatically:

officer = PiiEncryptor.from_config("gdpr_officer.yaml")
result = officer.encrypt_batch(rows, "customers")

Forget

When a GDPR erasure request arrives, call forget() with the customer's identifier. This deletes their encryption key from the key store and writes an audit record. After deletion, every encrypted PII value for that customer across every table in the data lake/house is permanently undecryptable. Non-PII columns remain intact.

officer.forget("<customer_id>", reason="GDPR Article 17 request", requested_by="dpo@company.com")

Or via the CLI:

gdpr-officer forget <customer_id> --reason "GDPR Article 17 request" --by "dpo@company.com"

Decrypt

Decrypt PII columns when needed. Pass the same customer_id and pii parameters used during encryption.

decrypted_df = officer.decrypt_df(df, customer_id="...", pii=[...])

If a customer has been forgotten, decrypt_df leaves their PII columns as the encrypted base64 strings — the rest of the DataFrame comes back normally. decrypt_row raises a KeyError instead, so you can decide how to handle a missing key in your code.

Key store backends

Local (development)

The default backend stores keys in a local DuckDB file. No cloud setup needed. A warning is logged when this backend is active.

officer = PiiEncryptor()  # Defaults to local backend

You can inspect the key store with DuckDB:

import duckdb
conn = duckdb.connect("gdpr_officer_keys.duckdb")
print(conn.execute("SELECT * FROM customer_keys").fetchdf())
print(conn.execute("SELECT * FROM deletion_log").fetchdf())

GCP Firestore (production)

Create a Firestore database and configure IAM:

gcloud firestore databases create --location=<region>

# Pipeline service account — reads and writes keys during encryption
gcloud projects add-iam-policy-binding <project> \
    --member="serviceAccount:<pipeline-sa>@<project>.iam.gserviceaccount.com" \
    --role="roles/datastore.user"

# DPO / compliance — deletes keys for GDPR erasure
gcloud projects add-iam-policy-binding <project> \
    --member="user:<dpo-email>" \
    --role="roles/datastore.user"

officer = PiiEncryptor(
    key_backend="gcp_firestore",
    key_backend_config={"project": "<project>", "database": "(default)"},
)

Key migration

Copy keys from one backend to another. The exact key bytes are preserved, so data encrypted through the source backend can be decrypted through the target.

from gdpr_officer import PiiEncryptor, migrate_keys

source = PiiEncryptor(key_backend="local", key_backend_config={"db_path": "keys.duckdb"})
target = PiiEncryptor(key_backend="gcp_firestore", key_backend_config={"project": "<project>"})

result = migrate_keys(source=source, target=target)

Audit trail

Every forget() call writes an audit record to the key store.

officer.get_deletion_log()           # All erasure records
officer.is_forgotten("<customer_id>") # Whether a customer has been erased (no active key and an erasure record)
officer.list_active_customers()      # All customers with active keys

gdpr-officer audit-log
gdpr-officer audit-log --format json
gdpr-officer check <customer_id>
gdpr-officer list-customers

Development

git clone https://github.com/xiaohan-data/gdpr-officer
cd gdpr-officer
pip install -e ".[dev]"
pytest

The example scripts show the full encrypt → forget → decrypt lifecycle with sample data:

python examples/demo.py          # Minimal pipeline example
python examples/local_test.py    # Detailed inspection of encrypted output and key store

API reference

CLI reference

gdpr-officer forget <customer_id> --reason "..." --by "..."    # Delete a customer's key
gdpr-officer check <customer_id>                                # Check if a customer was forgotten
gdpr-officer list-customers                                     # List active customer keys
gdpr-officer audit-log [--format json]                          # Show erasure audit log

Roadmap

• AWS DynamoDB backend
• Azure Table Storage backend
• Key rotation with batch re-encryption
• Decryption utilities for controlled PII access workflows

License

Apache 2.0