RedactifyAI

RedactifyAI is a Python package for detecting and anonymizing sensitive Personally Identifiable Information (PII) in textual data using Microsoft's Presidio and Apache Spark.

Key Features

• Integration with Presidio: Detects and anonymizes PII such as names, emails, phone numbers, and more.
• Spark-powered processing: Handle large-scale data anonymization with PySpark.
• Custom Recognizers: Extend PII detection with custom logic for your specific needs.

Models

• en_core_web_lg

Installation

You can install RedactifyAI from PyPI or by building the wheel file locally.

Install from PyPI (if published)

pip install redactify-ai

Build Locally

1. Clone the repository:

   git clone https://github.com/your-repo/redactify-ai.git
   cd redactify-ai
   

2. Build the wheel:

   rm -rf build dist *.egg-info
   python setup.py sdist bdist_wheel
   

3. Install the wheel:

   pip install dist/redactify_ai-0.0.1-py3-none-any.whl
   

4. Upload the Package to PyPi
   1. Install Twine pip install twine
   2. Generate token for PyPi account
   3. Upload the Package twine upload dist/*

Usage

Step 1: Configuration

Prepare a config.yaml file for Presidio configuration (e.g., recognizers, anonymization rules). Example:

presidio:
   entities:
      - PERSON
      - PHONE_NUMBER
      - EMAIL_ADDRESS
      - LOCATION
      - DATE_TIME
      - CREDIT_CARD
   language: en
   score_threshold: 0.6
   mask_character: "*"
   spacy_model: en_core_web_lg # download the model of your choice, e.g. en_core_web_sm
   spacy_model_dir: /path/to/model/   # Custom model storage path

Step 2: Create a Processor

from redactify_ai.config import load_config
from redactify_ai.processor import PresidioDLPProcessor

# Load configuration
config = load_config("config.yaml")
processor = PresidioDLPProcessor(config)

Step 3: Anonymize DataFrame with PySpark

from redactify_ai.utils import anonymize_text_udf
from pyspark.sql import SparkSession

# Initialize Spark session
spark = SparkSession.builder.appName("PresidioDLP").getOrCreate()

# Create mock DataFrame
data = [("Hi, I'm John Doe. Email me at john.doe@gmail.com.",)]
df = spark.createDataFrame(data, ["transcripts"])

# Apply anonymization
anonymize_udf = anonymize_text_udf(processor)
df_redacted = df.withColumn("transcripts_redacted", anonymize_udf(df["transcripts"]))
df_redacted.show(truncate=False)

Running the Pipeline

To run the pipeline script provided in this repository:

python run_pipeline.py

End-to-End Integration Testing

If you want to verify that the RedactifyAI pipeline correctly redacts PII over a full environment—including Spark and real NLP models—an end-to-end test is provided.

Prerequisites

• Docker installed on your system
• test_config.yaml, the end-to-end test script (e.g., test_pipeline_integration.py), and the project source code in your working directory

Note: The test requires the en_core_web_lg SpaCy model to be present. The provided Dockerfile ensures this model is pre-installed and ready for use.

Running the End-to-End Test

1. Build the Docker Image

   In your project directory, run:

   docker build -t redactify-test .
   

2. Run the Test Suite

   Spawn a container from your image and execute the integration test:

   docker run --rm -v "$PWD:/app" -w /app redactify-test bash -c "
    python setup.py sdist bdist_wheel &&
    pip install dist/redactify_ai-0.0.1-py3-none-any.whl &&
    pytest tests/test_pipeline_integration.py --maxfail=1 --disable-warnings --tb=short
    "
   

   This will build/wheel/install/testing using the real Spark instance, SpaCy language model, and all necessary dependencies.

What This Test Does

• Runs the pipeline using a sample DataFrame with mock PII.
• Uses the actual PresidioDLPProcessor and redaction logic.
• Asserts that PII is redacted and replaced with mask characters as defined in your configuration.

How to Interpret Results

• The test will pass if the sensitive information is successfully redacted from the output.
• The script will print the redacted text, which should show mask characters (e.g., *) in place of PII.

If you make changes to the model configuration or the pipeline logic, rerunning this end-to-end test will ensure your modifications continue to correctly anonymize sensitive data.

For any issues, please check that your Docker build completes successfully and that your working directory contains all the necessary files (test_config.yaml, test script, and source code).

Contributing

Contributions are welcome! Please create issues or pull requests if you find bugs or would like to add new features.

License

This project is licensed under the MIT License. See the LICENSE file for details.