sentinel-rs 0.1.1

High-performance pattern matching engine for Python, powered by Rust

Sentinel-RS 🦀

High-performance pattern matching engine for Python, powered by Rust.

Process millions of log lines per second with parallel regex matching. Perfect for log anonymization, data sanitization, and any large-scale text transformation task.

Why Sentinel-RS?

✅ 10-50x faster than pure Python regex processing
✅ True parallelism - uses all CPU cores, bypasses Python's GIL
✅ Pattern agnostic - define any regex patterns you need
✅ Memory efficient - buffered I/O and memory-mapped file support
✅ Zero overhead - native Rust speed with Pythonic API

Installation

pip install sentinel-rs

Quick Start

Basic Usage

import sentinel_rs

# Define your patterns
rules = {
    r'\b[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+\.[A-Z|a-z]{2,}\b': '[EMAIL]',
    r'\b(?:\d{1,3}\.){3}\d{1,3}\b': '[IP]',
    r'password=\S+': 'password=***',
}

# Process a single string
text = "User [e-mail] logged in from 192.168.1.1"
result = sentinel_rs.scrub_text(text, rules)
print(result)
# Output: "User [EMAIL] logged in from [IP]"

# Process a file (uses all CPU cores automatically)
lines = sentinel_rs.scrub_logs_parallel(
    'application.log',
    'application_scrubbed.log',
    rules
)
print(f"Processed {lines:,} lines")

For Large Files (> 1GB)

import sentinel_rs

# Use memory-mapped I/O for better performance on huge files
lines = sentinel_rs.scrub_logs_mmap(
    'huge_logfile.log',
    'huge_logfile_scrubbed.log',
    rules
)

Use Cases

Log Anonymization (PII Scrubbing)

import sentinel_rs

# Remove personally identifiable information from logs
pii_rules = {
    r'\b[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+\.[A-Z|a-z]{2,}\b': '[EMAIL]',
    r'\b(?:\d{1,3}\.){3}\d{1,3}\b': '[IP]',
    r'\b(?:\d{4}[-\s]?){3}\d{4}\b': '[CREDIT_CARD]',
    r'\b\d{3}-\d{2}-\d{4}\b': '[SSN]',
    r'\+?1[-.]?\d{3}[-.]?\d{3}[-.]?\d{4}': '[PHONE]',
}

sentinel_rs.scrub_logs_parallel('logs/app.log', 'logs/app_clean.log', pii_rules)

Custom Business Data Redaction

import sentinel_rs

# Mask your internal identifiers and codes
custom_rules = {
    r'EMPLOYEE-\d{6}': '[EMP_ID]',
    r'PROJECT-[A-Z]{3}-\d{4}': '[PROJECT]',
    r'INTERNAL-KEY-[A-Z0-9]{16}': '[SECRET]',
}

sentinel_rs.scrub_logs_parallel('internal.log', 'redacted.log', custom_rules)

API Response Sanitization

import sentinel_rs

# Remove sensitive data from API responses before logging
api_rules = {
    r'"api_key":\s*"[^"]+': '"api_key": "[REDACTED]',
    r'"token":\s*"[^"]+': '"token": "[REDACTED]',
    r'"password":\s*"[^"]+': '"password": "[REDACTED]',
}

clean_response = sentinel_rs.scrub_text(api_response, api_rules)

Format Conversion

import sentinel_rs

# Transform date formats, standardize patterns, etc.
conversion_rules = {
    r'\d{2}/\d{2}/\d{4}': '[DATE]',
    r'\$\d+\.\d{2}': '[AMOUNT]',
}

sentinel_rs.scrub_logs_parallel('raw.log', 'normalized.log', conversion_rules)

API Reference

Core Functions

scrub_text(text: str, rules: dict) -> str

Process a single string in memory.

Parameters:

• text: Input string to process
• rules: Dictionary mapping regex patterns to replacement strings

Returns: Transformed string

Example:

result = sentinel_rs.scrub_text(
    "Contact: [e-mail]",
    {r'\S+@\S+': '[EMAIL]'}
)

scrub_logs_parallel(input_path: str, output_path: str, rules: dict) -> int

Process a file using parallel execution across all CPU cores.

Parameters:

• input_path: Path to input file
• output_path: Path to output file
• rules: Dictionary mapping regex patterns to replacement strings

Returns: Number of lines processed

Best for: Files < 1GB, general use

scrub_logs_mmap(input_path: str, output_path: str, rules: dict) -> int

Process a file using memory-mapped I/O for maximum performance.

Parameters:

• input_path: Path to input file
• output_path: Path to output file
• rules: Dictionary mapping regex patterns to replacement strings

Returns: Number of lines processed

Best for: Files > 1GB, memory-constrained environments

Benchmarking

from sentinel_rs import Benchmark

rules = {r'@\S+': '@[HIDDEN]', r'\d+\.\d+\.\d+\.\d+': '[IP]'}
benchmark = Benchmark(rules)
results = benchmark.run('test.log')

print(f"Rust:   {results['rust_time']:.3f}s")
print(f"Python: {results['python_time']:.3f}s") 
print(f"Speedup: {results['speedup']:.2f}x")

Performance

Typical performance on modern hardware (M1/Ryzen/Intel i7+):

File Size	Lines	Pure Python	Sentinel-RS	Speedup
10 MB	100K	2.5s	0.15s	16x
100 MB	1M	25s	1.2s	20x
1 GB	10M	250s	11s	22x

Performance scales linearly with CPU core count

Pattern Examples

Common PII Patterns

# Email addresses
r'\b[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+\.[A-Z|a-z]{2,}\b'

# IPv4 addresses  
r'\b(?:\d{1,3}\.){3}\d{1,3}\b'

# IPv6 addresses
r'\b(?:[A-Fa-f0-9]{1,4}:){7}[A-Fa-f0-9]{1,4}\b'

# Credit cards (basic)
r'\b(?:\d{4}[-\s]?){3}\d{4}\b'

# US Social Security Numbers
r'\b\d{3}-\d{2}-\d{4}\b'

# US Phone numbers
r'\+?1[-.]?\d{3}[-.]?\d{3}[-.]?\d{4}'

# API Keys (32+ alphanumeric)
r'\b[A-Za-z0-9]{32,}\b'

# AWS Access Keys
r'AKIA[0-9A-Z]{16}'

# Bearer tokens
r'Bearer\s+[A-Za-z0-9\-._~+/]+=*'

# JWT tokens
r'eyJ[A-Za-z0-9-_=]+\.eyJ[A-Za-z0-9-_=]+\.[A-Za-z0-9-_.+/=]+'

URL Sanitization

# Remove query parameters with sensitive keys
r'(https?://[^?]+)\?.*(?:token|key|password|secret)=[^&\s]+'

# Mask credentials in URLs
r'://[^:]+:[^@]+@'  # Replace with '://[CREDENTIALS]@'

Demo & Testing

Run the Demo

python demo.py

The demo showcases:

1. In-memory text scrubbing
2. File processing
3. Performance benchmarking (Rust vs Python)
4. Custom pattern examples

Generate Test Data

# Generate 1 million diverse log lines
python scripts/generate_logs.py -n 1000000 -o test.log

# Generate smaller test file
python scripts/generate_logs.py -n 10000 -o small.log

Run Tests

pytest tests/ -v

How It Works

Sentinel-RS is built on three key technologies:

1. PyO3 - Rust bindings for Python (zero-copy data transfer)
2. Rayon - Data parallelism library (automatic work distribution)
3. Regex - Rust's optimized regex engine

Flow:

Python defines patterns → PyO3 bridge → Rust compiles regex → 
Rayon parallelizes across cores → Process millions of lines → 
Return results to Python

The engine is completely pattern-agnostic - it doesn't know what PII is or what you're matching. You define all the logic in Python, and Rust provides the speed.

Requirements

• Python 3.8+
• Any platform (Linux, macOS, Windows)
• Multi-core CPU recommended for maximum performance

Development

Build from Source

# Install Rust (if not already installed)
curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh

# Install maturin
pip install maturin

# Build in development mode
maturin develop

# Build for release (optimized)
maturin develop --release

Project Structure

sentinel-rs/
├── src/lib.rs           # Rust core (pattern matching engine)
├── sentinel_rs/         # Python package
│   └── __init__.py      # Python wrapper & utilities
├── tests/               # Test suite
├── scripts/             # Utility scripts
└── demo.py             # Interactive demo

Contributing

Contributions welcome! Areas for improvement:

• Additional optimization techniques
• Support for more file formats
• Better error messages
• Documentation improvements

License

MIT License - see LICENSE file for details.

Acknowledgments

Built with:

• PyO3 - Rust-Python bindings
• Maturin - Build tool
• Rayon - Data parallelism
• Regex - Rust regex engine

Security Note

⚠️ Important: This library performs pattern matching based on the regex rules YOU provide. It's your responsibility to:

• Test patterns thoroughly before production use
• Ensure patterns match your specific data formats
• Validate that scrubbed data meets your compliance requirements
• Handle false positives/negatives appropriately

Always test with non-production data first!

---

Made with ❤️ and 🦀 Rust

For questions, issues, or feature requests, please open an issue on GitHub.