idscrub 🧽✨

• Names and other personally identifying information are often present in text, even if they are not clearly visible or requested.
• This information may need to be removed prior to further analysis in many cases.
• idscrub identifies and removes (✨scrubs✨) personal data from text using regular expressions and named-entity recognition.

Installation

idscrub can be installed using pip into a Python >=3.12 environment.

We recommend installing with the SpaCy transformer model (en_core_web_trf) as a dependency:

pip install idscrub[trf]

If you do not need SpaCy:

pip install idscrub

How to use the code

Basic usage example (see basic_usage.ipynb for further examples):

from idscrub import IDScrub

scrub = IDScrub(['Our names are Hamish McDonald, L. Salah, and Elena Suárez.', 'My number is +441111111111 and I live at AA11 1AA.'])
scrubbed_texts = scrub.scrub(
    pipeline=[
        {"method": "spacy_entities", "entity_types": ["PERSON"]},
        {"method": "uk_phone_numbers"},
        {"method": "uk_postcodes"},
    ]
)

print(scrubbed_texts)

# Output: ['Our names are [PERSON], [PERSON], and [PERSON].', 'My number is [PHONENO] and I live at [POSTCODE].']

Scrubbed data can be identified using the following methods (see the usage example notebook for further information):

import pandas as pd
from idscrub import IDScrub

# From lists of text:
scrub = IDScrub(['Our names are Hamish McDonald, L. Salah, and Elena Suárez.', 'My number is +441111111111 and I live at AA11 1AA.'])
scrubbed_texts = scrub.scrub(
    pipeline=[
        {"method": "spacy_entities", "entity_types": ["PERSON"]},
        {"method": "uk_phone_numbers"},
        {"method": "uk_postcodes"},
    ]
)
scrubbed_df = scrub.get_scrubbed_data()
print(scrubbed_df)

# From a Pandas DataFrame:
scrubbed_df, scrubbed_data = IDScrub.dataframe(
    df=pd.read_csv('path/to/csv'), 
    id_col="ID", 
    pipeline=[
        {"method": "spacy_entities", "entity_types": ["PERSON"]},
        {"method": "uk_phone_numbers"},
        {"method": "uk_postcodes"},
    ]
)
print(scrubbed_data)

Method key-value pairs for further customisation, e.g. "entity_types", can be viewed by viewing the docstring e.g. ?IDScrub.spacy_entities.

Key-value pairs represent method arguments:

IDScrub.spacy_entities(entity_types=["PERSON"]) is equivalent to pipeline=[{"method": "spacy_entities", "entity_types": ["PERSON"]}].

Personal data types supported

Considerations before use

• You must follow GDPR guidance when processing personal data using this package.
• This package has been designed as a first pass for standardised personal data removal.
• Users are encouraged to check and confirm outputs and conduct manual reviews where necessary, e.g. when cleaning high risk datasets.
• It is up to the user to assess whether this removal process needs to be supplemented by other methods for their given dataset and security requirements.

Input data

• This package is designed for text-based documents structured as a list of strings.
• It performs best when contextual meaning can be inferred from the text.
• For best results, input text should therefore resemble natural language.
• Highly fragmented, informal, technical, or syntactically broken text may reduce detection accuracy and lead to incomplete or incorrect name detection.

Biases and evaluation

• idscrub supports integration with SpaCy and Hugging Face models for name cleaning.
• These models are state-of-the-art, capable of identifying approximately 90% of named entities, but may not remove all names.
• Biases present in these models due to their training data may affect performance. For example:
  • English names may be more reliably identified than names common in other languages.
  • Uncommon or non-Western naming conventions may be missed or misclassified.

Models

• Only Spacy's en_core_web_trf and no Hugging Face models have been formally evaluated.
• We therefore recommend that the current default en_core_web_trf is used for name scrubbing. Other models need to be evaluated by the user.

Similar Python packages

• Similar packages exist for undertaking this task, such as Presidio, Scrubadub and Sanityze.

• Development of idscrub was undertaken to:

  • Bring together different scrubbing methods across the Department for Business and Trade.
  • Adhere to infrastructure requirements.
  • Guarantee future stability and maintainability.
  • Encourage future scrubbing methods to be added collaboratively and transparently.
  • Allow for full flexibility depending on the use case and required outputs.

• To leverage the power of other packages, we have added methods that allow you to interact with them. These include: IDScrub.presidio() and IDScrub.google_phone_numbers(). See the usage example notebook and method docstrings for further information.

AI declaration

AI has been used in the development of idscrub, primarily to develop regular expressions, suggest code refinements and draft documentation.

Development setup

This project is managed by uv.

To install all dependencies for this project, run:

uv sync

If you do not have Python 3.12, run:

uv python install 3.12

To run tests:

uv run pytest

or

make test

Author

Analytical Data Science, Department for Business and Trade