colgov

Deterministic, reversible tokenization and fail-closed access policy for tabular PII.

Tokenize a column and it stays joinable. Same plaintext, same token — every time, across tables and across runs — so JOIN, GROUP BY and COUNT(DISTINCT) keep working on data nobody can read.

Status: alpha (0.2.0). Every v0.1 and v0.2 feature is in place. The API may still change before 1.0. See CHANGELOG.md.

The problem

Encrypting a column normally destroys its analytical value. Standard authenticated encryption uses a random IV, so the same email address becomes different ciphertext every time it is encrypted — correct for protecting text, useless for a warehouse:

encrypt("ada@example.com")   # 'Yk2p...'
encrypt("ada@example.com")   # 'Qm9x...'  ← different every call

Join two tables on that column and you get zero rows back.

The approach

colgov uses AES-SIV (RFC 5297), a deterministic authenticated encryption mode. The same input always produces the same token, and the original value is recoverable with the key:

t.tokenize("ada@example.com", column="email")   # 'dEmr4UrbAT3YC8v2o0S4Uuf_L06ueUs51KdWaw0Y8YQ'
t.tokenize("ada@example.com", column="email")   # 'dEmr4UrbAT3YC8v2o0S4Uuf_L06ueUs51KdWaw0Y8YQ'  ← stable

Determinism is a deliberate trade, not a free win: it preserves the frequency distribution of a column, so low-cardinality fields stay re-identifiable even once tokenized. colgov treats that as a first-class concern and refuses, by default, to tokenize columns whose cardinality is too low to protect.

Usage

pip install colgov

from colgov import Tokenizer

key = Tokenizer.generate_key()        # 32 random bytes — store it in a KMS / secret manager
t = Tokenizer(key)

token = t.tokenize("ada@example.com", column="email")
t.detokenize(token, column="email")   # 'ada@example.com'

t.tokenize(None, column="email")      # None — NULL stays NULL

• One key per column. Each column's AES-256-SIV key is derived from the master key with HKDF-SHA256, using the column name as context. The same value in two different columns gives unrelated tokens, so you can't join tables on a column the policy didn't mean to link.
• Tokens are URL- and SQL-safe. They use unpadded base64url (A–Z a–z 0–9 - _).
• Tampering is detected. detokenize raises colgov.InvalidToken when a token was changed or was made with a different key or column.
• Lose the master key and the tokens can't be reversed. Anyone who has the key can reverse every token.

Refusing columns too predictable to protect

tokenize_column profiles a column before it tokenizes anything. If the column has fewer than 10 distinct non-null values, it raises LowCardinalityError and tokenizes nothing:

t.tokenize_column(["M", "F", "F", None], column="gender")
# LowCardinalityError: column 'gender' has 2 distinct values (minimum 10) ...

t.tokenize_column(values, column="gender", min_distinct=3)             # adjust the threshold
t.tokenize_column(values, column="gender", allow_low_cardinality=True) # opt out explicitly

Measuring re-identification risk

from colgov import column_risk, k_anonymity

column_risk(["a", "a", "a", "b", None])
# ColumnRisk(n_rows=5, n_null=1, n_distinct=2, min_frequency=1, top_share=0.75)

result = k_anonymity(rows, quasi_identifiers=["birth_year", "postcode", "gender"])
result.k               # size of the smallest group of rows sharing all three values
result.rows_below(5)   # how many rows sit in groups smaller than 5

k == 1 means at least one person is unique on those columns and can be singled out, even when every column is tokenized.

Classify, review, then govern

The full workflow has three steps: a rule pack suggests labels, a person decides, and a policy resolves what each role sees.

from colgov import PUBLIC, Catalog, Policy, RulePack, Tokenizer

table = {
    "customer_email": [...],
    "mobile_no": [...],
    "order_total": [...],
    "comments": [...],
}

# 1. Suggest: rule packs match column names and sampled values.
suggestions = RulePack.builtin("core").classify(table)
suggestions["customer_email"][0]
# Suggestion(column='customer_email', label='email', confidence=0.95,
#            rule_ids=('email-name', 'email-value'), ...)

# 2. Decide: only a named person turns a suggestion into a decision.
catalog = Catalog()
catalog.accept(suggestions["customer_email"][0], by="alice")
catalog.accept(suggestions["mobile_no"][0], by="alice")
catalog.decide("order_total", PUBLIC, by="alice", note="no personal data")
catalog.pending(table)     # ['comments']  — not reviewed yet
catalog.save("catalog.yaml")  # commit it; review decisions like code

# 3. Resolve: a fail-closed policy per role.
policy = Policy.from_yaml("""
roles:
  analyst:
    email: tokenize
    phone_number: deny
""")
view = policy.apply(table, role="analyst", catalog=catalog, tokenizer=Tokenizer(key))
list(view)                 # ['customer_email', 'order_total']

Policies fail closed at every step:

• An unknown role sees nothing.
• A column nobody has reviewed is denied, whatever the machine suggested.
• A label the role isn't granted is denied. That includes misspelt labels.
• Tokenized columns still go through the cardinality check, so a predictable column raises LowCardinalityError instead of leaking.

Treatments are clear, tokenize and deny. Columns reviewed as public are clear unless a role overrides it. Use policy.plan(role, columns, catalog) to see each column's treatment and the reason for it.

Writing a rule pack

pack: my-org
version: 1
labels:
  employee_id:
    description: Internal staff number
rules:
  - id: employee-id-name
    label: employee_id
    column_name: 'emp(loyee)?_?(id|no)'   # regex, case-insensitive, searched in the name
    confidence: 0.8
  - id: employee-id-value
    label: employee_id
    value_pattern: 'E[0-9]{6}'           # regex, must match the whole value
    min_match_ratio: 0.9                 # share of sampled non-null values (default 0.8)
    confidence: 0.9

Load it with RulePack.load("my-org.yaml"). Packs are validated strictly: unknown keys, undeclared labels, invalid regexes and duplicate rule ids are all rejected when the pack loads. The built-in core pack covers email, phone numbers, names, national IDs, dates of birth, postal codes, street addresses, IP addresses and payment cards.

Detokenization, with an audit trail

A role may turn tokens back into plaintext only for columns it could already see in clear. The rules are the same fail-closed ones as for views, so there is nothing extra to configure. Every attempt is recorded before any plaintext is returned, whether it was allowed, denied or failed. Each record names who asked, why, and how many values were involved.

from colgov import AccessDenied, JsonlAuditLog

audit = JsonlAuditLog("audit.jsonl")
policy.detokenize(
    tokens, column="customer_email", role="support", catalog=catalog,
    tokenizer=t, actor="carol", purpose="TICKET-4521", audit=audit,
)                          # ['ada@example.com', ...]

policy.detokenize(tokens, column="customer_email", role="analyst", ...)
# AccessDenied: role 'analyst' may not detokenize 'customer_email' ...

• Actor and purpose are required. If the audit log can't be written, no plaintext is returned.
• Audit records never contain data, neither plaintext nor tokens.
• The log is tamper-evident. JsonlAuditLog chains every line to the one before it with SHA-256. verify_audit_log("audit.jsonl") (or colgov audit verify) finds any line that was edited, deleted or reordered.
• Views can be audited too. Pass audit= and actor= to policy.apply.

pandas and PySpark

pip install "colgov[pandas]"   # or "colgov[spark]"

from colgov import pandas as cpd

suggestions = cpd.classify(df)
view = cpd.apply(df, policy, role="analyst", catalog=catalog, tokenizer=t)
plain = cpd.detokenize(view["customer_email"], policy, role="support", catalog=catalog,
                       tokenizer=t, actor="carol", purpose="TICKET-4521", audit=audit)

from colgov import spark as cspark

view = cspark.apply(sdf, policy, role="analyst", catalog=catalog, tokenizer=t)  # lazy DataFrame

• pandas: the index and the dtypes of clear columns are kept. None, NaN and pd.NA all count as null.
• Spark: tokenization runs in a UDF on the executors, and the cardinality check is a single aggregation. The master key is shipped to the executors, so colgov must be installed there, and you should only use a cluster you trust with the key.
• Both: tokenized columns must hold strings, so cast other types first.

Command line

colgov keygen                                   # new master key (base64)
export COLGOV_MASTER_KEY=...                    # or pass --key-file

colgov classify customers.csv                   # suggestions per column
colgov review customers.csv -c catalog.yaml --by alice
                                                # decide interactively; saved after every answer
colgov plan customers.csv -p policy.yaml -c catalog.yaml --role analyst
                                                # what the role would see, and why
colgov apply customers.csv -p policy.yaml -c catalog.yaml --role analyst -o analyst.csv \
             --audit audit.jsonl --actor bob
colgov detokenize -p policy.yaml -c catalog.yaml --role support --column customer_email \
                  --actor carol --purpose TICKET-4521 --audit audit.jsonl < tokens.txt
colgov audit verify audit.jsonl

review shows suggestions and the evidence for them, but never the column's values. In the CSV files, empty cells are treated as nulls.

Roadmap

v0.1

• Deterministic reversible tokenization with per-column key derivation (HKDF)
• Column classification from portable YAML rule packs
• Human review workflow — a machine suggests, a person decides
• Fail-closed policy resolution: an unclassified column is never visible
• Re-identification risk scoring (cardinality, k-anonymity)

v0.2

• Policy-governed detokenization with a tamper-evident audit log
• pandas and PySpark helpers
• colgov command-line tool

Scope

colgov governs columns in tabular data. It does not detect PII inside free-text prose — for that, use Presidio, which is excellent at it. An optional bridge is planned so Presidio can act as a value-shape detector feeding colgov's classification.

License

Apache-2.0