carduus 0.4.1

PySpark implementation of the Open Privacy Preserving Record Linkage protocol.

Carduus

The open source implementation of the Open Privacy Preserving Record Linkage (OPPRL) protocol build on Spark.

:warning: Carduus has been superseded by spindle-token which includes a superset of carduus's functionality. Carduus remains available on pypi and Github but future development on OPPRL tokenization will be focused on spindle-token. You can read more here.

Rationale

Privacy Preserving Record Linkage (PPRL) is crucial component to data de-identification systems. PPRL obfuscate identifying attributes or other sensitive information about the subjects described in the records of a dataset while still preserving the ability to link records pertaining to the same subject through the use of an encrypted token. This practice is sometimes referred to as "tokenization" and is one of the components of data de-identification.

The task of PPRL is to replace the attributes of a every record denoting Personally Identifiable Information (PII) with a token produced by a one-way cryptographic function. This prevents observers of the tokenized data from obtaining the PII. The tokens are produced deterministically such that input records with the same, or similar, PII attributes will produce an identical token. This allows practitioners to associate records across datasets that are highly likely to belong to the same data subject without having access to PII.

Tokenization is also used when data is shared between organizations to limit, or in some cases fully mitigate, the risk of subject re-identification in the event that an untrusted third party gains access to a dataset containing sensitive data. Each party produced encrypted tokens using a different secret key so that any compromised data asset is, at worst, only matchable to other datasets maintained by the same party. During data sharing transactions, a specific "transcryption" data flow is used to first re-encrypt the sender's tokens into ephemeral tokens that do not match tokens in any other dataset and can only be ingested using the recipients secret key. At no point in the "transcryption" data flow is the original PII used.

Carduus is the first (and canonical) implementation of the Open Privacy Preserving Record Linkage (OPPRL) protocol. This protocol presents a standardized methodology for tokenization that can be implemented in any data system to increase interoperability. The carduus implementation is a python library that distributes the tokenization workload using apache Spark across multiple cores or multiple machines in a high performance computing cluster for efficient tokenization of any scale datasets.

Why the name "Carduus"? The carduus is a genus of thistle plants that was used to brush fibrous materials so that individual fibres align in preparation for spinning the material into thread or yarn. Today this process is known as "carding" and is done by specialized machines.

Getting Started

See the getting started guide on the project's web page for an detailed explanation of how carduus is used including example code snippets.

The full API and an example usage on Databricks are also provided on the project's web page.

Security Audit

This project has received a security audit from Echelon Risk + Cyber who provided the following statement. More details on this security audit can be obtained from Echelon Risk + Cyber at this link.

Echelon Risk + Cyber certifies that as of May 30, 2025, The Spindle Token implementation and Open Privacy Preserving Record Linkage (OPPRL) Protocol exhibit a high degree of alignment with secure cryptographic standards and secure development practices. The use of FIPS -compliant algorithms (AES-G CM-SIV, RSA-OAEP, SHA2 family), layered encryption, and privacy preserving design patterns indicate strong foundational security. Note: This certification is issued in good faith, based on the materials available to the Echelon team at the time of the review.

Contributing

Please refer to the carduus contributing guide for information on how to get started contributing to the project.

Organizations that have contributed to Carduus