Lightspeed Data Anonymizer

A PII and sensitive-data redaction library for Lightspeed assistants and the Lightspeed AI observability pipeline, maintained by the lightspeed-core team.

Derived from data-anonymizer. This project is seeded from data-anonymizer, built by Red Hat's UXE Data & AI Solutions team for the Ask Red Hat agent, and reused with the authors' permission. The lightspeed-core team maintains this fork and owns its tuning, testing, accuracy, and support for Lightspeed workloads. See NOTICE for attribution.

Introduction

Lightspeed Data Anonymizer is a standalone PII/sensitive-data redaction library for Red Hat AI use cases, implemented as a wrapper around the Presidio SDK.

PII entities that can be processed using data-anonymizer:

• Email Address
• Hostname
• IP Address (IPv4, IPv6)
• Location
• Organization (off by default)
• Person
• Phone Number
• URL

Organization names are exempt from anonymization and are left in the text. Detecting them also flags product and technology names such as OpenShift, RHEL and Sosreport, so enable it only if you need it:

anonymizer = PIIAnonymizer(organization=True, allow_list=["RHEL", "OpenShift"])

Locations written in all caps are detected by a second pass over a title cased copy of the text. What it costs depends on the text: roughly 2% on prose with no all caps words, 14% on a typical mixed sentence, and up to 80% on an address block, which is the case that needs it. Turn it off to trade that recall for latency:

anonymizer = PIIAnonymizer(case_normalization=False)

Upgrading from 0.2.0

organization now defaults to False, so organization names that 0.2.0 redacted are left in the text. This is deliberate, for the reason above, but it changes what leaves the library. Callers that relied on the old behaviour have to ask for it:

anonymizer = PIIAnonymizer(organization=True)

Choosing a model

Detection quality is set by the spaCy model behind it. Install en_core_web_lg as the default; en_core_web_trf detects substantially more locations and people, but depends on torch, which adds roughly 500 MB to an image:

Memory is usually the deciding factor rather than latency. Check that the environment has the headroom for the transformer before enabling it.

Detection of emails, hostnames, IP addresses and phone numbers is by regex and is identical on both models.

Install models with spaCy (they are not PyPI packages, so they are not pulled in automatically):

python -m spacy download en_core_web_lg      # default
python -m spacy download en_core_web_trf     # also: pip install torch

Both can be installed side by side, so an environment can switch between them without reinstalling. Select the model with the DATA_ANONYMIZER_SPACY_MODEL environment variable, so a deployment can switch without a code change in the calling service:

export DATA_ANONYMIZER_SPACY_MODEL=en_core_web_trf

The spacy_model argument overrides the environment variable:

anonymizer = PIIAnonymizer(spacy_model="en_core_web_trf")

Naming a model that is not installed raises on construction, so a misconfigured environment fails at startup rather than silently anonymizing less than it should.

To always redact specific terms, regardless of what the model detects, pass a deny list. Naming a term here enables its entity type even when that type is otherwise off:

anonymizer = PIIAnonymizer(deny_list={"ORGANIZATION": ["Acme Corp"], "LOCATION": ["Novo mesto"]})

Getting Started

1. Install the library and a spaCy model.

pip install lightspeed-data-anonymizer
python -m spacy download en_core_web_lg

(From source instead: pip install "git+https://github.com/anik120/lightspeed-data-anonymizer.git".)

2. Import the PIIAnonymizer

from lightspeed_data_anonymizer.anonymizer import PIIAnonymizer

3. Instantiate and use the anonymizer.

anonymizer = PIIAnonymizer()

To anonymize a list of strings:
anonymized_list = anonymizer.anonymize_list([str1, str2, str3])

To anonymize a text/string:
anonymized_text = anonymizer.anonymize_text(text)