oblitus

Oblitus is Latin for "erased, forgotten". It is a Python library and command-line tool for tables that contain personal data. It:

1. detects which columns hold personal or sensitive data. It reads column names and values, and checks the check digits of about 30 document formats, Brazilian and international;
2. proposes a plan, a JSON file you can review and edit, that assigns one of 29 techniques to each flagged column;
3. applies the plan as pseudonymization (keyed, re-linkable by whoever holds the key) or true anonymization (no link left, and checked; the default), keeping either the statistics or the privacy first;
4. shows how noise deflects your data on an interactive page: distributions, per-record shifts and re-identification risk for every noise technique and strength;
5. reveals the original values of reversibly encrypted columns, for whoever holds the key;
6. generates synthetic data from a sample or from a schema. A schema can plant correlations, conditional rules, formulas, trends, seasonality, outliers and exact statistics. It can also generate data-science challenges with a known answer key.

Features and algorithms

Roadmap

Planned, not yet implemented:

• Differential privacy: AIM synthesis. Its reference package pulls in PyTorch and JAX, learns column domains from the data and keeps its budget accounting inside the library (see docs/differential-privacy.md).

Install

Python 3.12 or later:

pip install oblitus
# with DuckDB support:
pip install "oblitus[duckdb]"
# microphone and camera entropy sources for `oblitus keygen`:
pip install "oblitus[audio]" "oblitus[camera]"
# with PySpark, for Spark DataFrames (needs Java to run Spark itself):
pip install "oblitus[spark]"
# with OpenDP, for differentially private aggregates:
pip install "oblitus[opendp]"
# with attack-based risk checks (Anonymeter):
pip install "oblitus[anonymeter]"
# more countries' identifiers and international phone numbers:
pip install "oblitus[identifiers]"
# names and addresses inside free text, with a named-entity model:
pip install "oblitus[ner]"

uv add oblitus works too. The package imports as oblitus and installs the oblitus command. Each GitHub release also carries the wheel and the source archive. Versions up to 0.2.0 were called data-anonymizer and were published on GitHub only.

Quick start

From a clone of the repository, this writes a sample customer table from the bundled example schema, then scans, plans, anonymizes it and writes a noise explorer page:

git clone https://github.com/rmonteiro-pereira/oblitus.git
cd oblitus
uv sync
export OBLITUS_SECRET_KEY="$(openssl rand -hex 32)"

uv run oblitus synthesize --schema examples/customers_schema.json --rows 2000 -o customers.csv
uv run oblitus scan customers.csv
uv run oblitus propose customers.csv --protection anonymization -o plan.json
# review and edit plan.json, then:
uv run oblitus apply customers.csv --plan plan.json -o customers_anon.csv --report report.json
uv run oblitus explore-noise customers.csv --column monthly_spend -o monthly_spend_noise.html

apply ends with a line such as wrote 2000 rows to customers_anon.csv; k-anonymity 1 -> 5; result is anonymized. Open the explorer page in a browser; docs/noise-explorer.md gives the command for each system, and warns that the page holds real values from the column.

How it works, on one row

scan classifies every column. On the sample table it finds five direct identifiers (customer_id, full_name, email, cpf, and voucher, whose few filled values are almost all distinct) and age as a quasi-identifier. propose turns that into a plan, and apply runs it. One row before and after; your values will differ, and timestamps and amounts are shortened here:

• The identifiers are replaced: fresh codes for the ID and voucher, random fakes for name, email and CPF.
• Timestamps are cut to the day and amounts rounded to three significant figures, because a value distinct in every row would single the row out.
• Ages are averaged in groups of five, and local suppression blanks the rare cells that would still make a row unique.
• The rows come back in a fresh random order, so the "after" row is a different customer: nothing in the release lines it up with the original.

apply then checks the released table and labels it anonymized, at_risk, pseudonymized or identifiable, with the reason for anything short of the target.

The same flow in Python. scan, propose, validate, apply and reveal take pandas, Polars, DuckDB and Spark tables and return the same kind. A DuckDB or Spark table needs the plan's row_key (see docs/protection.md).

import pandas as pd
from oblitus import Anonymizer, Fidelity, Protection

# Parse dates, so the plan treats signup_date as a timestamp rather than text.
customers = pd.read_csv("customers.csv", parse_dates=["signup_date"])
anonymizer = Anonymizer(secret_key="replace-with-a-long-random-secret", seed=42, locale="pt_BR")

for finding in anonymizer.scan(customers):
    print(finding.column, finding.category, finding.confidence, finding.evidence)

plan = anonymizer.propose(
    customers, protection=Protection.ANONYMIZATION, fidelity=Fidelity.STATISTICAL
)
result = anonymizer.apply(customers, plan)
result.data  # the anonymized DataFrame, of the kind you passed in
result.protection.level  # anonymized, at_risk, pseudonymized or identifiable
result.protection.problems  # why a level was not reached
result.utility.max_correlation_change

Configuration

In Python, pass secret_key= to Anonymizer; the library never reads the environment. Treat the key like a password. Anyone who holds it can decrypt the reversible columns, and can test whether a guessed real value produces a given pseudonym.

Generate a key with oblitus keygen, which mixes os.urandom (always) with the extra sources you name: hardware (/dev/hwrng), microphone and camera (low-order bits of ambient noise, extras audio and camera), timing (CPU jitter) and beacon (drand and NIST, opt-in, public). A source that is absent or fails its health check is skipped and reported on stderr. See docs/protection.md. In Python, generate_key(sources) returns the key and a report, and Anonymizer(throwaway_key_source=...) draws the anonymization throwaway key from the same mix.

Command line

All commands take --seed N (reproducible noise and synthesis; with a secret key the noise is reproducible only for the key's holder, with a seed alone anyone who knows the seed can recompute it) and --locale (Faker locale for fake values, such as pt_BR; default en_US). Tables can be .csv or .parquet, read and written with Polars.

P is anonymization (the default) or pseudonymization. F is statistical (the default) or privacy. -o is short for --output.

Exit codes:

• 0: success.
• 2: an error.
• 3: apply wrote its output, but the result does not reach the plan's protection target.

Documentation

• docs/detection.md: categories, recognized identifiers, and how a column is classified
• docs/protection.md: pseudonymization against anonymization, the default proposals, plan files and the protection checks
• docs/techniques.md: every technique, its parameters and its effect
• docs/business-confidentiality.md: sharing company data, such as tenant revenue and rent, without revealing any one company's figures
• docs/noise-explorer.md: writing and opening the noise explorer page, and reading its metrics
• docs/synthesis.md: schemas, planted patterns, synthesis from a sample, and data-science challenges
• docs/differential-privacy.md: what epsilon means, public bounds and domains, DP aggregates, MST synthesis and the budget report
• ARCHITECTURE.md: C4-style system views, how the code is organised and where to change things
• CONTRIBUTING.md: the project's standards for code, design, tests, docs, privacy review, commits and releases
• CHANGELOG.md: released changes

Development

uv sync
uv run pytest
uv run ruff format .
uv run ruff check .
uv run mypy src

GitHub Actions runs the same checks on Python 3.12 and 3.13 for every push to main and every pull request (.github/workflows/ci.yml); there, formatting is checked with ruff format --check instead of rewritten.

The standards every change follows are in CONTRIBUTING.md.