zero-harm-ai-detectors 0.2.3

AI-powered privacy and harmful content detection library

Zero Harm AI Detectors

AI-powered detection of PII, secrets, and harmful content in text.

Now with transformer-based models for 85-95% accuracy in person name detection!

🚀 What's New in v0.2.x

• 🤖 AI-Powered Detection: Uses BERT/RoBERTa for accurate PII detection
• 🎯 Unified Pipeline: Single ZeroHarmPipeline class for everything
• 📊 Confidence Scores: Every detection includes confidence (0-1)
• 🔄 Backward Compatible: Drop-in replacement - works with old API
• ⚡ Smart Detection: AI accuracy + regex speed where appropriate
• 🌍 Better Support: International names, locations, organizations

📦 Installation

Quick Start (CPU)

pip install zero_harm_ai_detectors transformers torch

For GPU (Faster)

pip install zero_harm_ai_detectors transformers
pip install torch --extra-index-url https://download.pytorch.org/whl/cu118

Lightweight (Regex Only)

pip install zero_harm_ai_detectors

🎯 Quick Start

One-Line Detection (Easiest)

from zero_harm_ai_detectors import detect_all_threats

result = detect_all_threats(
    "Contact John Smith at [e-mail]. API key: sk-abc123."
)

print(result['redacted'])
# Output: Contact [REDACTED_PERSON] at [REDACTED_EMAIL]. API key: [REDACTED_SECRET].

print(result['detections'])
# Output: {'PERSON': [...], 'EMAIL': [...], 'API_KEY': [...]}

Full Pipeline (Recommended)

from zero_harm_ai_detectors import ZeroHarmPipeline, RedactionStrategy

# Initialize once (loads models)
pipeline = ZeroHarmPipeline()

# Use many times
text = "Email John Smith at [e-mail] or call 555-123-4567"
result = pipeline.detect(text, redaction_strategy=RedactionStrategy.TOKEN)

print(f"Original: {result.original_text}")
print(f"Redacted: {result.redacted_text}")

for det in result.detections:
    print(f"  {det.type}: {det.text} (confidence: {det.confidence:.0%})")

Legacy API (Still Works!)

# Old code works unchanged!
from zero_harm_ai_detectors import detect_pii, detect_secrets, redact_text

text = "Contact [e-mail] with API key sk-abc123"

pii = detect_pii(text)  # Now uses AI automatically!
secrets = detect_secrets(text)

redacted = redact_text(text, {**pii, **secrets})

🎨 Features

Detectable Content

PII (Personally Identifiable Information)

• ✉️ Emails: [e-mail]
• 📞 Phone Numbers: 555-123-4567
• 🆔 SSN: 123-45-6789
• 💳 Credit Cards: 4532-0151-1283-0366
• 👤 Person Names: AI-powered, 85-95% accuracy (NEW!)
• 📍 Locations: Cities, states, countries (NEW!)
• 🏢 Organizations: Companies, institutions (NEW!)
• 🏠 Addresses: Street addresses, P.O. boxes
• 🏥 Medical Records: MRN detection
• 🚗 Driver's Licenses: US state formats
• 📅 Dates of Birth: Multiple formats

Secrets & Credentials

• 🔑 API Keys: OpenAI, AWS, Google, etc.
• 🎫 Tokens: GitHub, Slack, Stripe, JWT
• 🔐 Passwords: Pattern-based detection

Harmful Content

• ☠️ Toxic Language
• ⚔️ Threats
• 😡 Insults
• 🔞 Obscene Content
• 👿 Identity Hate

Redaction Strategies

# TOKEN: [REDACTED_EMAIL]
RedactionStrategy.TOKEN

# MASK_ALL: ********************
RedactionStrategy.MASK_ALL

# MASK_LAST4: ****************.com
RedactionStrategy.MASK_LAST4

# HASH: 8d969eef6ecad3c29a3a...
RedactionStrategy.HASH

📚 Advanced Usage

Custom Configuration

from zero_harm_ai_detectors import ZeroHarmPipeline, PipelineConfig

config = PipelineConfig(
    pii_threshold=0.8,  # Higher confidence threshold
    pii_model="Jean-Baptiste/roberta-large-ner-english",  # Better model
    harmful_threshold_per_label=0.6,
    device="cuda"  # Use GPU
)

pipeline = ZeroHarmPipeline(config)

Selective Detection

# Only detect PII
result = pipeline.detect(
    text,
    detect_pii=True,
    detect_secrets=False,
    detect_harmful=False
)

Filtering by Confidence

result = pipeline.detect(text)

# Only high-confidence detections
high_conf = [d for d in result.detections if d.confidence >= 0.9]

for det in high_conf:
    print(f"{det.type}: {det.text} ({det.confidence:.2%})")

Batch Processing

texts = [
    "Email: [e-mail]",
    "Phone: [phone]",
    "Meet Jane at Microsoft"
]

for text in texts:
    result = pipeline.detect(text)
    print(f"Text: {text}")
    print(f"Redacted: {result.redacted_text}")

API Integration

from flask import Flask, request, jsonify
from zero_harm_ai_detectors import ZeroHarmPipeline

app = Flask(__name__)

# Load once at startup
pipeline = ZeroHarmPipeline()

@app.route("/api/check_privacy", methods=["POST"])
def check_privacy():
    data = request.json
    text = data.get("text", "")
    
    result = pipeline.detect(text)
    
    return jsonify({
        "original": result.original_text,
        "redacted": result.redacted_text,
        "detections": result.to_dict()["detections"],
        "harmful": result.harmful,
        "severity": result.severity
    })

🔬 Comparison: Regex vs AI

Feature	Regex (Old)	AI (New)	Winner
Person Names	30-40%	85-95%	🏆 AI
Locations	❌	80-90%	🏆 AI
Organizations	❌	75-85%	🏆 AI
Context Understanding	❌	✅	🏆 AI
Email Detection	99%+	99%+	🤝 Tie
Phone Detection	95%+	95%+	🤝 Tie
Speed (single)	1-5ms	50-200ms	🏆 Regex
False Positives	High	Low	🏆 AI

⚡ Performance

Speed Benchmarks

Operation	Time	Notes
Pipeline loading	5-10s	One-time at startup
Email detection	50ms	AI + regex
Person name	150ms	AI (transformer)
Full detection	200ms	All types

Best Practices

# ✅ Good: Load once, reuse
PIPELINE = ZeroHarmPipeline()

def process(text):
    return PIPELINE.detect(text)  # Fast!

# ❌ Bad: Load every time
def process(text):
    pipeline = ZeroHarmPipeline()  # Slow!
    return pipeline.detect(text)

🔄 Migration from v0.1.x

Your old code works without changes:

# Old code (v0.1.x)
from zero_harm_ai_detectors import detect_pii, detect_secrets

pii = detect_pii("Contact [e-mail]")  # Now uses AI!
secrets = detect_secrets("API key sk-abc123")

# Force old regex behavior if needed
pii = detect_pii(text, use_ai=False)

See MIGRATION_GUIDE.md for detailed instructions.

🧪 Testing

# Install dev dependencies
pip install -e ".[dev]"

# Run tests
pytest

# With coverage
pytest --cov=zero_harm_ai_detectors --cov-report=html

# Run specific test
pytest tests/test_ai_detectors.py -v

📊 Model Information

PII Detection Models

Model	Size	Languages	Accuracy
dslim/bert-base-NER	420MB	English	85%
Jean-Baptiste/roberta-large-ner-english	1.3GB	English	92%

Harmful Content Model

Model	Size	Languages	Categories
unitary/multilingual-toxic-xlm-roberta	1.1GB	100+	6 labels

💡 Use Cases

• API Gateways: Scan requests/responses for sensitive data
• Chat Applications: Prevent PII leakage
• Data Pipelines: Clean datasets before sharing
• Content Moderation: Filter harmful content
• Compliance: GDPR, HIPAA, PCI-DSS
• Security: Detect leaked credentials

🐛 Troubleshooting

Models not loading

pip install transformers torch

Out of memory

config = PipelineConfig(device="cpu")  # Use CPU

Slow performance

# Skip unnecessary detection
result = pipeline.detect(text, detect_harmful=False)

📞 Support

• Email: [e-mail]
• GitHub Issues: Create an issue

📄 License

MIT License - see LICENSE file for details.

🙏 Acknowledgments

• Hugging Face for transformer models
• PyTorch team for the ML framework

---

Made with ❤️ by Zero Harm AI LLC

Protecting privacy, one detection at a time.