Nerpii

Nerpii is a Python library developed to perform Named Entity Recognition (NER) on structured datasets and synthesize Personal Identifiable Information (PII).

NER is performed with Presidio and with a NLP model available on HuggingFace, while the PII generation is based on Faker.

Installation

Nerpii requires Python 3.10 or later. You can install it by using pip:

pip install nerpii

Quickstart

Named Entity Recognition

You can import the NamedEntityRecognizer using

from nerpii.named_entity_recognizer import NamedEntityRecognizer

You can create a recognizer passing as parameter a path to a csv file or a Pandas Dataframe

recognizer = NamedEntityRecognizer('./csv_path.csv', lang='en')

The lang parameter is used to define the language of the dataset. The default value is en (english), but it can also be set to it (italian).

NER runs on a random sample of rows (500 by default, set with data_sample). Pass random_state to get the same sample every time. The Presidio analyzer and the Hugging Face model are loaded the first time they are needed, and are then reused.

Please note that if there are columns in the dataset containing names of people consisting of first and last names (e.g. John Smith), before creating a recognizer, it is necessary to split the name into two different columns called first_name and last_name using the function split_name().

from nerpii.named_entity_recognizer import split_name

df = split_name('./csv_path.csv', name_of_column_to_split)

The first word of each name becomes the first name and the rest becomes the last name (e.g. Alexis R. Graves gives Alexis and R. Graves). Missing names stay missing, and the dataframe you pass in is not modified.

The NamedEntityRecognizer class contains three methods to perform NER on a dataset:

recognizer.assign_entities_with_presidio()

which assigns Presidio entities, listed here

recognizer.assign_entities_manually()

which assigns manually ZIPCODE and CREDIT_CARD_NUMBER entities

recognizer.assign_organization_entity_with_model()

which assigns ORGANIZATION entity using a NLP model available on HuggingFace.

To perform NER, you have to run these three methods sequentially, as reported below:

recognizer.assign_entities_with_presidio()
recognizer.assign_entities_manually()
recognizer.assign_organization_entity_with_model()

The final output is a dictionary in which column names are given as keys and assigned entities and a confidence score as values.

This dictionary can be accessed using

recognizer.dict_global_entities

PII generation

After performing NER on a dataset, you can generate new PII using Faker.

You can import the FakerGenerator using

from nerpii.faker_generator import FakerGenerator

You can create a generator using

generator = FakerGenerator(dataset, recognizer.dict_global_entities)

If you want to generate Italian PII, add lang = "it" as parameter to the previous object (default: lang = "en")

First and last names are generated for PERSON columns whose names mark them as name columns: e.g. first_name, FirstName or nome for first names, and last_name, surname or cognome for last names.

To generate first names of the same gender as the original ones, add a gender column with get_gender() before creating the generator. The generator drops the column from its output.

from nerpii.named_entity_recognizer import get_gender

dataset = get_gender(dataset)
generator = FakerGenerator(dataset, recognizer.dict_global_entities)

To generate new PII you can run

synthetic_dataset = generator.get_faker_generation()

The generator works on a copy of the dataset, so the original dataframe is left unchanged. The synthesized dataframe is returned and is also available as generator.dataset.

By default every non-null value of a recognized column is replaced. To replace only some values, mark them in the dataset (e.g. with *) and pass the mark as generation_mark:

generator = FakerGenerator(dataset, recognizer.dict_global_entities, generation_mark="*")

The method above can generate the following PII:

• address
• phone number
• email naddress
• first name
• last name
• city
• state
• url
• zipcode
• credit card
• ssn
• country

City, state, zipcode and country are generated together for each row, so they are consistent with each other. They come from the generator's locale: the country is always United States (or Italy with lang = "it"), and for Italian data the state is the province.

Examples

You can find a notebook example in the notebook folder.

Development

Nerpii uses uv to manage dependencies. To install them and run the checks that CI runs:

uv sync
uv run flake8 nerpii tests
uv run pytest

License

Nerpii is licensed under the GNU General Public License v3.0.