cloakdf 0.0.4

Pseudonymisation/anonymisation engine with encrypted mapping storage

cloakdf

Install

pip install cloakdf

How to use

We’ll demonstrate cloakdf using the Northwind dataset. A classic sample database with customers, orders, and employees tables that share keys across them.

#| eval: false
import httpx
from pathlib import Path

base = "https://raw.githubusercontent.com/neo4j-contrib/northwind-neo4j/refs/heads/master/data"
files = ["customers.csv", "orders.csv", "employees.csv"]
data = Path("../data")
data.mkdir(exist_ok=True)

for f in files:
    (data/f).write_bytes(httpx.get(f"{base}/{f}").content)

1. Define your table configuration

Each table’s config specifies two kinds of columns:

• id groups (e.g. id1, id2) — key columns that are shared across tables. Columns in the same group get consistent pseudonyms: customerID in both customers and orders maps to the same UUID, preserving referential integrity.

Note: All id columns must be string dtype before encoding — cast with .astype(str) if needed. - mask — sensitive columns to replace with opaque hex tokens. These are stored in a shared vault, so identical values (e.g. the same address appearing twice) get the same token.

tables = {
    'customers': {
        'id1': 'customerID',
        'mask': ['contactName', 'address', 'phone', 'companyName', 'fax', 'city']
    },
    'orders': {
        'id1': 'customerID',
        'id2': 'employeeID',
        'mask': ['shipName', 'shipAddress']
    },
    'employees': {
        'id2': 'employeeID',
        'mask': ['firstName', 'lastName', 'address', 'homePhone', 'birthDate', 'notes']
    },
}

2. Encode your DataFrames

import pandas as pd
from pathlib import Path

data = Path("../data")
unk = CloakDF(tables)

originals, encoded = {}, {}
for name in tables:
    df = pd.read_csv(data/f"{name}.csv", on_bad_lines='skip')
    for k, v in tables[name].items():
        if k.startswith('id'): df[v] = df[v].astype(str)
    originals[name] = df
    encoded[name] = unk.encode(name, df)

encoded['customers'].head(3)

<style scoped> .dataframe tbody tr th:only-of-type { vertical-align: middle; } .dataframe tbody tr th { vertical-align: top; } .dataframe thead th { text-align: right; } </style>

	customerID	companyName	contactName	contactTitle	address	city	region	postalCode	country	phone	fax
0	9eb2c3d4-32d3-434c-863c-44d8ac4067d9	9288db060f26	5c1dfdfc8f62	Sales Representative	80e2676376a1	c1f7bd6a22f0	NaN	12209	Germany	715e53fda07f	cd0efa7df34e
1	82d494d5-a9e7-4690-b5ec-3104dad5b87d	18db1aa91f8c	3c4420341c1a	Owner	135d66291b39	42d81129a68e	NaN	05021	Mexico	a5611db2c349	25fabc23caa4
2	fa465d8d-ead0-4d03-ae91-8a65715c7fd1	9458f0d21f98	08b3087d3dda	Owner	a93f1f586e03	42d81129a68e	NaN	05023	Mexico	2a648576b9be	NaN

3. Save encrypted mappings

The mapping dictionaries (key_maps and vault) are the sensitive artefacts — they allow de-anonymisation. Generate a Fernet key and encrypt them at rest. You can store the key to a file or an environment variable:

⚠️ Never commit your Fernet key or encrypted mappings to version control. Ensure your .gitignore includes key files and *.enc files (e.g. *.key, *.enc, data/).

import os

key = CloakDF.generate_key()
unk.save(data/"mappings.enc", key)

# Optionally store the key in an environment variable
os.environ['CLOAKDF_KEY'] = key.decode()
print("✓ Mappings saved and key stored in env var")

✓ Mappings saved and key stored in env var

4. Load and decode

Load the encrypted mappings (using the key directly or from an environment variable) and reverse the encoding:

# Load key from env var (or pass `key` directly)
loaded_key = CloakDF.load_key(env_var='CLOAKDF_KEY')
# loaded_key = CloakDF.load_key(path="path/to/keyfile")  # alternative: from file

unk2 = CloakDF.load(data/"mappings.enc", loaded_key, tables)

for name in tables:
    decoded = unk2.decode(name, encoded[name])
    pd.testing.assert_frame_equal(decoded, originals[name])
    print(f"✓ {name} round-trip OK")

decoded.head(3)

✓ customers round-trip OK
✓ orders round-trip OK
✓ employees round-trip OK

<style scoped> .dataframe tbody tr th:only-of-type { vertical-align: middle; } .dataframe tbody tr th { vertical-align: top; } .dataframe thead th { text-align: right; } </style>

	employeeID	lastName	firstName	title	titleOfCourtesy	birthDate	hireDate	address	city	region	postalCode	country	homePhone	extension	photo	notes	reportsTo	photoPath
0	1	Davolio	Nancy	Sales Representative	Ms.	1948-12-08 00:00:00.000	1992-05-01 00:00:00.000	507 - 20th Ave. E. Apt. 2A	Seattle	WA	98122	USA	[phone]	5467	0x151C2F00020000000D000E0014002100FFFFFFFF4269...	Education includes a BA in psychology from Col...	2.0	http://accweb/emmployees/davolio.bmp
1	2	Fuller	Andrew	Vice President, Sales	Dr.	1952-02-19 00:00:00.000	1992-08-14 00:00:00.000	908 W. Capital Way	Tacoma	WA	98401	USA	[phone]	3457	0x151C2F00020000000D000E0014002100FFFFFFFF4269...	Andrew received his BTS commercial in 1974 and...	NaN	http://accweb/emmployees/fuller.bmp
2	3	Leverling	Janet	Sales Representative	Ms.	1963-08-30 00:00:00.000	1992-04-01 00:00:00.000	722 Moss Bay Blvd.	Kirkland	WA	98033	USA	[phone]	3355	0x151C2F00020000000D000E0014002100FFFFFFFF4269...	Janet has a BS degree in chemistry from Boston...	2.0	http://accweb/emmployees/leverling.bmp

5. Compare encoded vs decoded

Here’s what the employees table looks like — encoded (pseudonymised) vs decoded (original):

cols = ['employeeID', 'firstName', 'lastName', 'title', 'address', 'homePhone']
display(encoded['employees'][cols].head(3))
display(decoded[cols].head(3))

Text Redaction with cloakdf.redact

While CloakDF handles reversible pseudonymisation of structured tabular data, redact handles irreversible PII removal from free text. It’s designed for case notes, narratives, and other unstructured fields where there’s no mapping to reverse, you just need the sensitive information gone.

It combines a distilbert NER model for detecting person names with regex patterns for structured identifiers (emails, phone numbers, NHS numbers, UK postcodes, and standalone numbers). Each detected entity is replaced with a flat typed placeholder like [NAME], [EMAIL], [PHONE], etc., so downstream consumers can still see what kind of information was there without seeing the actual values.

The default model is Davlan/distilbert-base-multilingual-cased-ner-hrl, a lightweight multilingual NER model. In practice it captures around 98% of PII in typical case-note text, but it is not perfect unusual name formats, ambiguous identifiers, and non-English text can produce false negatives, while common words occasionally get flagged as false positives. Always review redacted output before sharing.

from cloakdf.redact import Redactor

r = Redactor()
sample = "Dr Jane Smith called from [e-mail] on 07123 456789. NHS: 123 456 7890, postcode SW1A 1AA."
r.redact(sample)

Loading weights:   0%|          | 0/102 [00:00<?, ?it/s]Loading weights: 100%|##########| 102/102 [00:00<00:00, 5849.88it/s]
Warning: You are sending unauthenticated requests to the HF Hub. Please set a HF_TOKEN to enable higher rate limits and faster downloads.

'Dr [NAME] called from [EMAIL] on [PHONE]. NHS: [NHS], postcode [POSTCODE].'