pii-engine 1.0.0

Enterprise PII Masking, Tokenization & Encryption Module

PII Engine - Enterprise PII Masking, Tokenization & Encryption Module

A production-ready, reusable module for handling PII (Personally Identifiable Information) data across all endpoints and applications. Provides enterprise-grade tokenization, encryption, masking, and pseudonymization capabilities.

🚀 Quick Start

Installation

# Install the module
pip install -e .

# Install with development dependencies
pip install -e ".[dev]"

# Install with framework-specific dependencies
pip install -e ".[fastapi]"  # For FastAPI integration
pip install -e ".[flask]"    # For Flask integration

Basic Usage

from pii_engine import PIIEngine

# Initialize the engine
engine = PIIEngine()

# Process input data (tokenize and encrypt PII)
input_data = {
    "name": "John Doe",
    "email": "[e-mail]",
    "phone": "[phone]",
    "department": "Engineering"  # Non-PII field
}

processed_data = engine.process_input_data(input_data, table_name="employees")
# Result: {"name_token": "TKN_PERSON_NAME_a1b2c3d4", "email_token": "TKN_EMAIL_x9y8z7w6", ...}

# Get display data based on user role
display_data = engine.get_display_data(
    processed_data,
    table_name="employees",
    display_mode="masked",  # "masked", "pseudonymized", or "plaintext"
    user_role="user"
)
# Result: {"name": "J*** D***", "email": "j***@company.com", ...}

🏗️ Architecture

Core Components

pii_engine/
├── core/
│   ├── engine.py          # Main PII Engine orchestrator
│   ├── tokenizer.py       # Deterministic tokenization
│   ├── encryptor.py       # AES-256 encryption/decryption
│   ├── pseudonymizer.py   # Fake data generation
│   ├── masker.py          # Data masking strategies
│   └── policy_engine.py   # Policy-driven PII handling
├── config/
│   ├── pii_types.py       # PII type definitions
│   └── policies.yaml      # Processing and access policies
├── middleware/
│   ├── fastapi_middleware.py  # FastAPI integration
│   └── flask_middleware.py    # Flask integration
└── utils/
    └── audit.py           # Compliance and audit logging

Security Flow

Input Data → Tokenization → Encryption → Database Storage
     ↓              ↓            ↓            ↓
"[e-mail]" → TKN_EMAIL_x9y8 → AES-256 → Encrypted Blob

Database Retrieval → Decryption → Role-based Display → Frontend
        ↓               ↓              ↓              ↓
   Encrypted Blob → "[e-mail]" → "j***@email.com" → User Sees Masked

🔧 Integration Examples

FastAPI Integration

from fastapi import FastAPI
from pii_engine.middleware.fastapi_middleware import PIIMiddleware
from pii_engine import PIIEngine

app = FastAPI()
pii_engine = PIIEngine()

# Add automatic PII processing middleware
app.add_middleware(
    PIIMiddleware,
    pii_engine=pii_engine,
    auto_process_requests=True,
    auto_process_responses=True
)

@app.post("/users")
async def create_user(user_data: dict):
    # PII automatically processed by middleware
    # Save processed_data to database
    return {"status": "success"}

Flask Integration

from flask import Flask
from pii_engine.middleware.flask_middleware import PIIFlaskMiddleware

app = Flask(__name__)
pii_middleware = PIIFlaskMiddleware(app)

@app.route("/users", methods=["POST"])
@pii_process_input(table_name="users")
@pii_process_output(display_mode="masked")
def create_user(processed_data):
    # Work with tokenized data
    return processed_data

Manual Integration

from pii_engine import PIIEngine

def your_existing_endpoint(request_data):
    engine = PIIEngine()
    
    # Step 1: Process input PII
    processed_data = engine.process_input_data(request_data, "users")
    
    # Step 2: Save to database (contains tokens only)
    database.save(processed_data)
    
    # Step 3: Return appropriate display data
    display_data = engine.get_display_data(
        processed_data, "users", "masked", "user"
    )
    
    return {"user": display_data}

Standalone Proxy Service

# Run PII proxy between frontend and backend
python run_pii_proxy.py

# Frontend calls proxy instead of backend:
# http://localhost:8000/api/users (proxy)
# Proxy forwards to: http://localhost:8001/api/users (backend)

🛡️ Security Features

Multi-Layer Protection

1. Tokenization: Deterministic tokens for duplicate detection
2. Encryption: AES-256 Fernet encryption for data at rest
3. Masking: Role-based data masking for display
4. Pseudonymization: Realistic fake data for testing/analytics
5. Audit Logging: Comprehensive access and operation logging

Role-Based Access Control

# config/policies.yaml
access_policies:
  roles:
    admin:
      can_view_plaintext: true
      audit_required: true
    user:
      can_view_masked: true
    analyst:
      can_view_pseudonymized: true

Compliance Support

• GDPR: Right to be forgotten, data portability
• CCPA: Opt-out rights, data transparency
• HIPAA: Healthcare data protection (configurable)
• SOX: Financial data compliance

📊 Supported PII Types

PII Type	Tokenization	Encryption	Masking	Pseudonymization
Email	✅	✅	j***@company.com	[e-mail]
Phone	✅	✅	******4567	555-123-4567
Name	✅	✅	J*** D***	Alex Johnson
Address	✅	✅	123 Main S***	456 Oak Ave, Springfield, CA
SSN	✅	✅	***-**-6789	123-45-6789
Credit Card	✅	✅	****-****-****-1234	4532-1234-5678-9012

🧪 Testing

# Run all tests
pytest

# Run with coverage
pytest --cov=pii_engine --cov-report=html

# Run specific test types
pytest -m unit          # Unit tests only
pytest -m integration   # Integration tests only
pytest -m "not slow"    # Skip slow tests

# Run security tests
pytest tests/test_security.py -v

Test Coverage Requirements

• Minimum Coverage: 90%
• Security Tests: 100% (no PII leakage)
• Integration Tests: All middleware and frameworks
• Performance Tests: Tokenization and encryption speed

🔧 Configuration

Environment Variables

# Required
PII_ENC_KEY=your-fernet-encryption-key

# Optional
PII_CONFIG_PATH=/path/to/custom/policies.yaml
PII_DB_CONNECTION_STRING=your-database-connection
PII_AUDIT_LEVEL=INFO

Generate Encryption Key

# Using the built-in key generator
pii-engine-keygen

# Or in Python
from pii_engine.core.encryptor import Encryptor
print(Encryptor.generate_key())

📈 Performance

Benchmarks

• Tokenization: ~0.5ms per field
• Encryption: ~1.0ms per field
• Decryption: ~1.2ms per field
• Masking: ~0.1ms per field
• Pseudonymization: ~0.3ms per field

Scalability

• Horizontal Scaling: Stateless design
• Database Optimization: Connection pooling, prepared statements
• Caching: Redis integration for token caching
• Bulk Operations: Efficient batch processing

🚀 Production Deployment

Simple Python Service

# 1. Install the module
pip install -e .

# 2. Set encryption key
export PII_ENC_KEY="your-generated-key"

# 3. Run PII proxy service
python run_pii_proxy.py

Add to Existing Backend

from pii_engine import PIIEngine

# Add to your existing endpoints
def create_user(user_data):
    engine = PIIEngine()
    processed_data = engine.process_input_data(user_data, "users")
    user_id = database.save(processed_data)
    display_data = engine.get_display_data(processed_data, "users", "masked", "user")
    return {"user": display_data}

Systemd Service (Linux)

# /etc/systemd/system/pii-proxy.service
[Unit]
Description=PII Proxy Service
After=network.target

[Service]
Type=simple
User=pii-service
WorkingDirectory=/opt/pii-engine
Environment=PII_ENC_KEY=your-key
ExecStart=/usr/bin/python3 run_pii_proxy.py
Restart=always

[Install]
WantedBy=multi-user.target

📚 Documentation

• API Documentation - Complete API reference
• Integration Guide - Framework-specific guides
• Security Guide - Security best practices
• Deployment Guide - Production deployment
• Contributing Guide - Development guidelines

🤝 Contributing

1. Fork the repository
2. Create a feature branch (git checkout -b feature/amazing-feature)
3. Install development dependencies (pip install -e ".[dev]")
4. Setup pre-commit hooks (pre-commit install)
5. Write tests for your changes
6. Ensure all tests pass (pytest)
7. Commit your changes (git commit -m 'Add amazing feature')
8. Push to the branch (git push origin feature/amazing-feature)
9. Open a Pull Request

Code Quality Standards

• Black formatting (88 character line length)
• isort import sorting
• flake8 linting
• mypy type checking
• bandit security scanning
• pytest testing (90%+ coverage)

📄 License

This project is licensed under the MIT License - see the LICENSE file for details.

🆘 Support

• Issues: GitHub Issues
• Documentation: Read the Docs
• Security: [e-mail]
• General: [e-mail]

🏆 Features

• ✅ Production Ready: Battle-tested in enterprise environments
• ✅ Framework Agnostic: Works with FastAPI, Flask, Django, and more
• ✅ Type Safe: Full mypy type checking support
• ✅ Async Support: Compatible with async/await patterns
• ✅ Comprehensive Testing: 90%+ test coverage
• ✅ Security First: No plaintext PII in databases
• ✅ Compliance Ready: GDPR, CCPA, HIPAA support
• ✅ Performance Optimized: Sub-millisecond operations
• ✅ Audit Trail: Complete operation logging
• ✅ Easy Integration: Drop-in middleware support

---

Made with ❤️ by the PII Platform Team