PAMOLA.CORE

Privacy Engineering for Python. Finally.

PAMOLA.CORE is the open-source foundation of the PAMOLA platform — a Python library for privacy-preserving data operations with reproducible pipelines, structured artifacts, and an audit trail per operation.

Developed by Realm Inveo Inc.

The Problem

You need to anonymize sensitive data. You've tried:

• ARX: Powerful, but Java, GUI-focused, opaque operations.
• Faker + Presidio + custom scripts: Fragmented, no pipeline, no proof.
• DP libraries: Great math, but narrow scope.

You're still missing:

• Direct operations (mask, generalize, pseudonymize, suppress) — not just "achieve k-anonymity".
• Risk measurement (privacy/fidelity/utility metrics on real outputs).
• Reproducibility (config + metrics + artifacts written to disk per task).

The Solution

PAMOLA.CORE: operations-first privacy engineering with a per-operation lifecycle (validate → load → process → save → metrics → visualize → cache).

from pathlib import Path

import pandas as pd

from pamola_core import DataSource, FullMaskingOperation, HashBasedPseudonymizationOperation

df = pd.DataFrame({
    "customer_id": [101, 102, 103],
    "email": ["ann@example.com", "bob@example.com", "cy@example.com"],
    "phone": ["+1 416 555 0101", "+1 416 555 0102", "+1 416 555 0103"],
})
task_dir = Path("anonymize_customers")


def output_of(result):
    """Each operation writes its own output file; read it back to feed the next step."""
    return pd.read_csv(result.get_artifacts_by_category("output")[0].path)


# 1) Keyed, irreversible hash of the e-mail column
hashed = HashBasedPseudonymizationOperation(field_name="email").execute(
    DataSource(dataframes={"main": df}), task_dir / "01_hash_email", reporter=None
)

# 2) Mask the phone column of the hashed output
masked = FullMaskingOperation(field_name="phone").execute(
    DataSource(dataframes={"main": output_of(hashed)}), task_dir / "02_mask_phone", reporter=None
)

print(masked.status)        # OperationStatus.SUCCESS
print(output_of(masked))    # e-mail hashed, phone masked, customer_id unchanged

One operation, one field, one output file. An operation reads the dataset you pass in its DataSource and writes a complete copy with its field transformed. It does not see what another operation did to the same dataset. To protect several fields, feed each operation the output of the previous one, as above — otherwise every output file still contains the other fields in clear text. Give each operation its own directory so its config.json is kept.

The hash uses a fresh random pepper per run (use_pepper=True by default), so it cannot be recomputed from the e-mail alone; on three e-mails the operation also warns that hashing a low-entropy column is reversible by enumeration.

Output structure (per operation directory):

anonymize_customers/01_hash_email/
├── config.json           # Operation configuration (secrets redacted)
├── output/               # The transformed dataset (CSV by default, Parquet with output_format="parquet")
└── metrics/              # Operation metrics (JSON)

Charts (visualizations/) are written only with generate_visualization=True and the viz extra; the consistent-mapping operation adds an encrypted maps/ directory.

Security note: config.json redacts sensitive parameters (salts, mapping encryption keys, fake-data keys) declared in OperationConfig.SENSITIVE_KEYS. See CHANGELOG.md for the full release notes.

PAMOLA Ecosystem

PAMOLA.CORE is part of a comprehensive privacy engineering stack:

Scope of CORE: This package provides anonymization and pseudonymization primitives, classical privacy metrics, and reproducibility plumbing. It does not implement formal differential-privacy synthetic-data generation — that lives in PAMOLA.SYNT / PAMOLA.BEST.

CORE also does not search for a k-anonymous generalization the way ARX or Mondrian-style tools do. You choose the generalization levels, apply them with the generalization and suppression operations, and verify the result with KAnonymityProfilerOperation (its ENRICH mode adds a per-record k that RecordSuppressionOperation(suppression_condition="risk") can act on).

What's In CORE

All classes below are exported from the top-level pamola_core package.

Note on attack simulation: pamola_core/attacks/ is an internal package with no stability guarantee, and it is deliberately narrow — it carries only the attacks that measure anonymization quality: LinkageAttack (its TF-IDF / SVD "CVPLA" variant, which is not the CVPL metric below) and AttributeInference. The CVPL linkage metric lives in pamola_core.metrics.privacy.cvpl and is run through CVPLLinkageOperation. There are no public, registered Attack*Operation classes.

Two corrections to earlier releases of this note. These modules are not used by PrivacyMetricOperation — that operation imports pamola_core.metrics.privacy. And DCR/NNDR live in pamola_core.metrics.privacy.distance and .neighbor; duplicate copies that had accumulated under attacks/ were removed, as was membership inference, which measures model privacy rather than anonymization.

Singling-out is not implemented. It is the third anonymization-facing criterion, and no report produced from this package covers it.

Pseudonymization Spotlight

from pamola_core import ConsistentMappingPseudonymizationOperation

op = ConsistentMappingPseudonymizationOperation(
    field_name="customer_id",
    mapping_encryption_key="ab" * 32,         # 256-bit hex key - keep it out of source control
    pseudonym_type="uuid",                    # or "sequential" / "random_string"
    mapping_format="csv",                     # encrypted at rest with AES-256-GCM
    persist_frequency=1000,
)

# Outputs:
#  - {task_dir}/output/         pseudonymized data
#  - {task_dir}/maps/           encrypted mapping file (*.enc) - the re-identification key
#  - {task_dir}/metrics/        operation metrics
#  - {task_dir}/config.json     mapping_encryption_key is replaced with "*REDACTED*"

Highlights:

• Mapping encryption keys are never persisted to disk. OperationConfig.SENSITIVE_KEYS + to_safe_dict() redact secrets before any save_config() call.
• No plain-text mapping file is ever written: mappings and their backups are encrypted in memory and replaced atomically.
• Hash-based op rejects weak salts (all-zero or empty) when use_pepper=False, and warns when the source column has so few distinct values that the hash is reversible by enumeration.
• Per-run session id invalidates stale disk cache when use_pepper=True, so previous-run pseudonyms cannot be served back.
• Compound identifiers, ENRICH/REPLACE modes, reverse mapping, and Dask pickle safety have dedicated tests.

Metrics

Metric operations compare an original and a transformed dataset and write JSON artifacts under {task_dir}/metrics/:

from pathlib import Path

import numpy as np
import pandas as pd

from pamola_core import DataSource, FidelityOperation, PrivacyMetricOperation

rng = np.random.default_rng(0)
original_df = pd.DataFrame({
    "age": rng.integers(18, 90, 1000),
    "gender": rng.choice(["F", "M"], 1000),
    "income": rng.normal(50_000, 15_000, 1000).round(),
})
anonymized_df = original_df.assign(age=(original_df["age"] // 10) * 10)  # 10-year bands

pair = DataSource(dataframes={"original": original_df, "transformed": anonymized_df})
names = {"original_dataset_name": "original", "transformed_dataset_name": "transformed"}
task_dir = Path("metrics_task")

fidelity = FidelityOperation(fidelity_metrics=["ks", "kl"], columns=["age", "income"]).execute(
    pair, task_dir / "fidelity", reporter=None, **names
)
privacy = PrivacyMetricOperation(
    privacy_metrics=["dcr", "nndr", "uniqueness"],
    columns=["age", "gender", "income"],
    metric_params={"uniqueness": {"quasi_identifiers": ["age", "gender"], "sensitives": ["income"]}},
).execute(pair, task_dir / "privacy", reporter=None, **names)

DCR/NNDR run an exact, block-wise search with bounded memory; above 10K × 10K records FAISS is selected automatically where it returns the same result as the exact search. A comparison that would exceed max_comparisons fails with a typed ResourceError instead of exhausting memory.

For dataset-level utility scoring (classification / regression downstream models) use UtilityMetricOperation.

Installation

From PyPI:

pip install pamola-core

The base install is large (about 0.9 GB of dependencies: pandas, scikit-learn, Dask, spaCy, FAISS, PyArrow and others). Install it into a virtual environment.

From source:

git clone https://github.com/realmdata-ai/PAMOLA.git
cd PAMOLA
pip install -e .

Visualization is optional:

pip install "pamola-core[viz]"

The base install writes JSON, CSV and Parquet, and never draws. Charts need matplotlib, plotly, seaborn, kaleido, wordcloud and matplotlib-venn, which are large, so they ship in the viz extra instead of the base install. Everything else — profiling, anonymization, transformations, metrics — works without them.

Call a chart without the extra and you get a message naming what to install, not a ModuleNotFoundError about a package you never asked for:

DependencyMissingError: no visualization backend is installed ...
They ship with the viz extra: pip install 'pamola-core[viz]'

Visualization is off by default: operations only draw when you pass generate_visualization=True. A default run starts no render thread, writes no plot files and logs nothing about charts, with or without the viz extra. With generate_visualization=True the charts are written and registered as artifacts before execute() returns; on a base install that request returns a single ERROR result naming pip install 'pamola-core[viz]' instead of running and logging a traceback from every layer of the plotting stack.

Behaviour change before 1.0.0: generate_visualization used to default to True. Pass generate_visualization=True to keep the old behaviour.

Test extras:

pip install -e ".[test]"   # adds pytest, pytest-cov
pip install -e ".[viz]"    # adds the plotting stack

Heads-up: Apart from viz, all scientific dependencies (numpy, pandas, scikit-learn, scipy, dask, spacy, faker, cryptography, etc.) are pinned in the main [project.dependencies] table — no separate [fast]/[ner]/[dp] extras in this release. Splitting the remaining heavy stacks into [nlp] and [bigdata] is planned; see FP-02b.

As of 1.0.0.dev4, torch and sdv are no longer installed: nothing in the library imported them. If you relied on them being pulled in as a side effect, install them explicitly. Splitting the remaining heavy dependencies (spacy, nltk, faiss, dask, the plotting stack) into optional extras is planned for a later release.

Supported Python Versions

PAMOLA.CORE supports Python 3.11 and 3.12 (requires-python = ">=3.11,<3.13"). Python 3.10 is not supported: the fixed filelock 4.2.1 requires Python 3.11 or newer.

Core Dependencies

A non-exhaustive view of the heaviest third-party packages (full list in pyproject.toml):

Versioning

PAMOLA.CORE follows Semantic Versioning and PEP 440.

import pamola_core
print(pamola_core.__version__)   # e.g. "1.0.0"

The badge at the top shows the newest version published on PyPI; the version on develop can be ahead of it.

• Source of truth: pyproject.toml → version
• Changelog: CHANGELOG.md
• CI/CD: GitHub Actions — lint (ruff), test (3.11/3.12, pytest), build (sdist+wheel), PyPI publish on tag v*
• Release rules: Dev tags (v*dev*) must be on develop; stable tags on main.

CLI

The pamola-core console script is installed automatically:

pamola-core --version
pamola-core list-ops                              # discover registered operations
pamola-core run --task task.json                  # run a task definition
pamola-core run --op FullMaskingOperation --config config.json --input data.csv
pamola-core schema FullMaskingOperation           # show parameter schema
pamola-core validate-config --config config.json  # validate a config file

Run pamola-core --help for the full command list.

Examples

Hands-on notebooks live under examples/:

• examples/anonymization/pseudonymization/ — simple + advanced for hash-based and consistent-mapping pseudonymization
• examples/anonymization/ — masking, generalization, noise, suppression
• examples/profiling/ — field-level and dataset-level profilers
• examples/transformations/ — merge, split, aggregate, clean, impute
• examples/fake_data/ — fake-value substitution for names, email, phone, organization (replacement values, reproducible when a key is given; not synthetic data)
• examples/metrics/ — fidelity, privacy, utility metrics
• examples/data_examples/sample.csv — non-PII synthetic sample used by the notebooks

No real personal data is included in this repository. All sample records are artificially generated; the origin and licence of every data asset is recorded in DATA_PROVENANCE.md.

Philosophy

• Operations-first: Direct transforms with a well-defined 7-step lifecycle, not constraint optimization.
• Measure everything: Privacy, fidelity, and utility metrics persisted as JSON artifacts.
• Reproducibility by default: Each operation writes a config.json (with secrets redacted) alongside its output.
• Secret hygiene: OperationConfig.SENSITIVE_KEYS provides a single place to declare parameters that must never reach disk.

API Documentation

The project uses Sphinx to generate API reference documentation from Python docstrings.

pip install sphinx sphinx-autodoc-typehints
sphinx-build -b html docs docs/_build    # output: docs/_build/index.html

See docs/README.md for build options.

Documentation

Use Cases

• Data Engineering: Prepare pseudonymized or masked datasets for ML training pipelines.
• Healthcare: Building blocks (masking, generalization, suppression) for de-identification workflows; whether a result meets HIPAA Safe Harbor or Expert Determination is for the data owner to establish.
• Finance: Privacy-engineering building blocks that can support PCI/GDPR programs.
• Compliance: Structured per-operation artifacts that can serve as technical evidence in a compliance process.
• Data Sharing: Risk-assessed data exchange between organizations.

What the terms mean here

Privacy-engineering vocabulary is used loosely in the industry. This is what PAMOLA.CORE 1.0 means by each term, what it provides for it, and what it does not claim. The definitions follow the 1.0 specification.

Privacy metrics are evidence, not determinations

The metrics in CORE (k-anonymity and uniqueness profilers, distance to closest record, nearest-neighbour distance ratio, fidelity and utility metrics, and the CVPL linkability measure) report measurements. None of them decides that data is legally anonymous, compliant with a regulation, or safe to release, and none of their wording should be read that way. See the regulatory evidence map for what each measurement does and does not establish.

Regulatory Context

PAMOLA.CORE provides technical building blocks for privacy compliance programs:

Important: PAMOLA.CORE provides technical capabilities only. It never certifies that a dataset is anonymous, de-identified, or compliant with GDPR, HIPAA, CCPA/CPRA or any other regime, and a successful run is not such a determination. Privacy metrics are evidence inputs to a regulated process; legal compliance requires organizational policies, procedures, and legal guidance beyond software tools. Pseudonymized data remains personal data under GDPR.

Contributing

git clone https://github.com/realmdata-ai/PAMOLA.git
cd PAMOLA
pip install -e ".[test]"
pytest tests/ -v

See CONTRIBUTING.md for guidelines.

Ownership & Licensing

PAMOLA.CORE is developed and maintained by Realm Inveo Inc. (REALMDATA) and distributed under the MIT License.

Contact

_{Built by Realm Inveo Inc.}