allyanonimiser 3.5.1

Australian-focused PII detection and anonymization for the insurance industry

Allyanonimiser

Australian-focused PII detection and anonymization for the insurance industry with support for stream processing of very large files.

📖 Read the full documentation

Version 3.5.1 — Label-Context Disambiguation + Per-Call Option Fixes

Bug-fix and precision release. AU bench unchanged from v3.5.0 (still beats openai/privacy-filter on 5 of 6 categories).

What's New

• Per-call options no longer leak — analyze(active_entity_types=...) / min_score_threshold=... used to permanently restrict the shared analyzer; they are now true per-call parameters (also fixed in DataFrameProcessor and StreamProcessor). The explicit set_active_entity_types() / set_min_score_threshold() methods remain the persistent knobs.
• analyze_batch() now matches analyze() exactly — the batch path previously skipped PERSON/LOCATION/ORG false-positive filtering and entity-type mapping, so DataFrame/CSV processing missed the v3.5.0 precision work.
• Label-context disambiguation — an explicit label immediately before a value ("TFN is 123 456 789", "Medicare number: ...") now beats bare-shape competitors and checksum validation, so labelled identifiers resolve to the labelled type instead of whatever 9-digit pattern also fired. Labelled regexes tolerate filler words (TFN is, ABN number:).
• Span-containment absorption — fragments fully contained in a wider, pattern-derived span are absorbed (lg-model DATE fragments inside DOB spans, stray NUMBER triplets inside TFNs, the bare CRN pattern matching the tail of an ABN). Postcodes inside addresses are never absorbed (keep_postcode depends on them).
• Internal consolidation — AU identifier regexes and false-positive word lists each now live in one module (patterns/shared_regex.py, core/false_positives.py) instead of two drifting copies.

Version 3.5.0 — International PII + Precision Overhaul

Adds out-of-the-box detection for international PII shapes that show up in AU-insurance data (expat customers, business travellers, system audit logs), and overhauls the conflict resolver so permissive patterns can no longer silently drop valid runner-ups. Beats openai/privacy-filter on 5 of 6 categories of the enriched AU bench. See Benchmarks.

What's New

• 5 new entity types loaded by default — PHONE_INTL (with +CC, 00 IDD prefix, and parenthesised area-code variants), US_SSN (with SSA reservation rules), CREDIT_CARD (Luhn-validated 13-19 digits), ISO_DATETIME (2024-05-22T14:32:00), TIME (12/24h, with/without seconds). All anchored on structural features that don't collide with AU patterns. See docs/patterns/international.md.
• Validate-then-pick conflict resolution — when multiple patterns match the same span, the resolver now walks candidates from highest priority down and returns the first that passes per-type validation. Previously a permissive pattern (e.g. CREDIT_CARD on a 13-digit phone) could win by priority, fail its checksum, and silently drop the valid runner-up.
• PERSON precision overhauled — added city/state-postcode/date-shape/acronym/label-word rejection, iterative trim of trailing label tokens (Joe Smith\nDOB → Joe Smith), and applied the FP check to single-candidate spans (previously bypassed). AU bench PERSON F1 0.836 → 0.954.
• CREDIT_CARD Luhn validator wired into EntityValidator.validate_credit_card.
• VEHICLE_REGISTRATION tightened — added SSN/TIN/NIN to the label deny-list plus an SSN-shape negative lookahead so bad SSN 999-04-7100 no longer absorbs SSN 999-04 as a plate.
• DATE_OF_BIRTH / INCIDENT_DATE no longer eat their prefix — capture-group rewrite so spans equal just the date (was 'DOB: 04/01/1959', now '04/01/1959').
• common_formats.py identifier patterns require a digit — prevents Claim Note → Note, Policy Number → Number, claim-mail → mail false positives.
• Enriched AU bench — bench/data/au_insurance.jsonl extended from 100 to 160 docs with three new templates (expat claim, payment record, business travel claim) covering international shapes.
• 40 new regression tests in tests/test_general_intl_patterns.py covering PHONE_INTL / US_SSN / CREDIT_CARD / ISO_DATETIME / TIME and the SSN/VR collision regression.
• CI fix — pyright workflow step had an invalid --level basic flag making it a no-op; now reads strictness from [tool.pyright] in pyproject.toml.

Installation

# Basic installation
pip install allyanonimiser==3.5.1

# With stream processing support for large files
pip install "allyanonimiser[stream]==3.5.1"

Prerequisites:

• Python 3.12 or 3.13 (spaCy does not yet ship cp314 wheels)

• A spaCy language model for Named Entity Recognition (NER). The library defaults to the small model — install it with:

  python -m spacy download en_core_web_sm   # 44 MB, fast — the default
  

  Switch to the large model if you need higher NER recall on names, places, and organisations (see the table below):

  python -m spacy download en_core_web_lg   # 587 MB, higher accuracy
  

  Then pass it explicitly:

  from allyanonimiser import create_allyanonimiser, SPACY_MODEL_ACCURATE
  ally = create_allyanonimiser(spacy_model=SPACY_MODEL_ACCURATE)
  

Choosing a spaCy model

	SPACY_MODEL_FAST (en_core_web_sm)	SPACY_MODEL_ACCURATE (en_core_web_lg)
Default in v3.3+? (unchanged in v3.4)	yes	no
Download size	44 MB	587 MB
Resident memory	~200 MB	~1.5 GB
Cold start	~0.5s	2 – 5s
Pattern detection (TFN, ABN, MEDICARE, AU_PHONE, EMAIL, dates, etc.)	identical	identical
PERSON recall on insurance text	medium (~80%)	high (~92%)
LOCATION and ORG recall	noticeably worse	high
Serverless / Azure Functions friendliness	good	poor (size + cold start)

TL;DR: the default works for most pipelines. Switch to SPACY_MODEL_ACCURATE only when missing a person's name or a city is a real cost in your downstream workflow.

Note: Pass spacy_model=None to disable spaCy entirely — pattern detection (emails, phones, IDs, dates, etc.) keeps working.

Verify Your Setup

After installation, verify that everything is properly configured:

python verify_setup.py

Or check spaCy status programmatically:

from allyanonimiser import create_allyanonimiser

ally = create_allyanonimiser()
status = ally.check_spacy_status()

if not status['is_loaded']:
    print(status['recommendation'])
else:
    print(f"✓ spaCy model loaded: {status['model_name']}")

Quick Start

from allyanonimiser import create_allyanonimiser

# Create the Allyanonimiser instance with default settings
ally = create_allyanonimiser()

# Analyze text
results = ally.analyze(
    text="Please reference your policy AU-12345678 for claims related to your vehicle rego XYZ123."
)

# Print results
for result in results:
    print(f"Entity: {result.entity_type}, Text: {result.text}, Score: {result.score}")

# Anonymize text
anonymized = ally.anonymize(
    text="Please reference your policy AU-12345678 for claims related to your vehicle rego XYZ123.",
    operators={
        "POLICY_NUMBER": "mask",  # Replace with asterisks
        "VEHICLE_REGISTRATION": "replace"  # Replace with <VEHICLE_REGISTRATION>
    }
)

print(anonymized["text"])
# Output: "Please reference your policy ********** for claims related to your vehicle rego <VEHICLE_REGISTRATION>."

Adding Custom Patterns

from allyanonimiser import create_allyanonimiser

# Create an Allyanonimiser instance
ally = create_allyanonimiser()

# Add a custom pattern with regex
ally.add_pattern({
    "entity_type": "REFERENCE_CODE",
    "patterns": [r"REF-\d{6}-[A-Z]{2}", r"Reference\s+#\d{6}"],
    "context": ["reference", "code", "ref"],
    "name": "Reference Code"
})

# Generate a pattern from examples
ally.create_pattern_from_examples(
    entity_type="EMPLOYEE_ID",
    examples=["EMP00123", "EMP45678", "EMP98765"],
    context=["employee", "staff", "id"],
    generalization_level="medium"  # Options: none, low, medium, high
)

# Test custom patterns
text = "Employee EMP12345 created REF-123456-AB for the project."
results = ally.analyze(text)
for result in results:
    print(f"Found {result.entity_type}: {result.text}")

Built-in Pattern Reference

Australian Patterns

Entity Type	Description	Example Pattern	Pattern File
AU_TFN	Australian Tax File Number	\b\d{3}\s*\d{3}\s*\d{3}\b	au_patterns.py
AU_PHONE	Australian Phone Number	\b(?:\+?61|0)4\d{2}\s*\d{3}\s*\d{3}\b	au_patterns.py
AU_MEDICARE	Australian Medicare Number	\b\d{4}\s*\d{5}\s*\d{1}\b	au_patterns.py
AU_DRIVERS_LICENSE	Australian Driver's License	Various formats including \b\d{8}\b (NSW) \b\d{4}[a-zA-Z]{2}\b (NSW legacy)	au_patterns.py
AU_ADDRESS	Australian Address	Address patterns with street names	au_patterns.py
AU_POSTCODE	Australian Postcode	\b\d{4}\b	au_patterns.py
AU_BSB_ACCOUNT	Australian BSB and Account Number	\b\d{3}-\d{3}\s*\d{6,10}\b	au_patterns.py
AU_ABN	Australian Business Number	\b\d{2}\s*\d{3}\s*\d{3}\s*\d{3}\b	au_patterns.py
AU_PASSPORT	Australian Passport Number	\b[A-Za-z]\d{8}\b	au_patterns.py

General Patterns

Entity Type	Description	Example Pattern	Pattern File
CREDIT_CARD	Credit Card Number	\b\d{4}[\s-]\d{4}[\s-]\d{4}[\s-]\d{4}\b	general_patterns.py
PERSON	Person Name	Name patterns with context	general_patterns.py
EMAIL_ADDRESS	Email Address	\b[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+\.[A-Z|a-z]{2,}\b	general_patterns.py
DATE_OF_BIRTH	Date of Birth	\bDOB:\s*\d{1,2}[/.-]\d{1,2}[/.-]\d{2,4}\b	general_patterns.py
LOCATION	Location	City and location patterns	general_patterns.py
DATE	Date	\b\d{1,2}[/.-]\d{1,2}[/.-]\d{2,4}\b	general_patterns.py
MONETARY_VALUE	Money Amount	\$\d{1,3}(?:,\d{3})*(?:\.\d{2})?\b	general_patterns.py
ORGANIZATION	Organization	Organization name patterns	general_patterns.py

Insurance Patterns

Entity Type	Description	Example Pattern	Pattern File
INSURANCE_POLICY_NUMBER	Insurance Policy Number	\b(?:POL|P|Policy)[- ]?\d{6,9}\b	insurance_patterns.py
INSURANCE_CLAIM_NUMBER	Insurance Claim Number	\b(?:CLM|CL|C)[- ]?\d{6,9}\b	insurance_patterns.py
VEHICLE_VIN	Vehicle Identification Number	\b[A-HJ-NPR-Z0-9]{17}\b	insurance_patterns.py
INVOICE_NUMBER	Invoice / Quote Number	\bINV-\d{4,10}\b, \bQ-\d{4}\b	insurance_patterns.py
BROKER_CODE	Broker Code	\bBRK-\d{4}\b	insurance_patterns.py
VEHICLE_DETAILS	Make / model patterns	(?:Toyota|Honda|...) <model> <year>	insurance_patterns.py
INCIDENT_DATE	Date of Incident	\bon\s+(\d{1,2}[/.-]\d{1,2}[/.-]\d{2,4})\b	insurance_patterns.py
NAME_CONSULTANT	Assigned consultant / agent	Assigned\s+To\s*:\s*<Name>	insurance_patterns.py

International / System Patterns (new in v3.5)

Non-AU PII shapes loaded by default — anchored on structural features that don't collide with AU patterns. See docs/patterns/international.md.

Entity Type	Description	Example Pattern	Pattern File
PHONE_INTL	International phone (+CC, 00 IDD, or parenthesised 3-4 digit area)	(?<!\d)\+\d{1,3}[\s.\-()]*\d[\d\s.\-()]{6,18}\d(?!\d)	general_intl_patterns.py
US_SSN	US Social Security Number (SSA reservation rules enforced)	\b(?!000|666|9\d{2})\d{3}-(?!00)\d{2}-(?!0000)\d{4}\b	general_intl_patterns.py
CREDIT_CARD	Luhn-validated 13-19 digit card	\b(?:\d[ -]?){12,18}\d\b + validate_credit_card	general_intl_patterns.py
ISO_DATETIME	ISO 8601 datetime (T-separated)	\b\d{4}-\d{2}-\d{2}T\d{2}:\d{2}(?::\d{2})?(?:Z|[+-]\d{2}:?\d{2})?\b	general_intl_patterns.py
TIME	12/24h time	(?<![/\d])\b(?:[01]?\d|2[0-3]):[0-5]\d(?::[0-5]\d)?(?:\s*[AaPp]\.?[Mm]\.?)?\b	general_intl_patterns.py

Features

• Australian-Focused PII Detection: Specialized patterns for TFNs, Medicare numbers, vehicle registrations, addresses, and more
• Insurance Industry Specialization: Detect policy numbers, claim references, and other industry-specific identifiers
• Multiple Entity Types: Comprehensive detection of general and specialized PII
• Flexible Anonymization: Multiple anonymization operators (replace, mask, redact, hash, and more)
• Stream Processing: Memory-efficient processing of large files with chunking support
• Reporting System: Comprehensive tracking and visualization of anonymization activities
• Jupyter Integration: Rich visualization capabilities in notebook environments
• DataFrame Support: Process pandas DataFrames with batch processing and multi-processing support
• Robust Detection: Enhanced validation and context-aware detection to reduce false positives
• Overlapping Entity Handling: Smart resolution of overlapping entities for clean anonymization
• Pattern Loading: Automatic loading of all default patterns (Australian, Insurance, General)
• Improved Medicare Detection: Fixed detection and validation for Australian Medicare numbers
• Multiple Entity Masking: Simultaneous masking of multiple entity types with different operators

Benchmarks

Head-to-head against openai/privacy-filter (a 1.5B-parameter MoE token classifier, Apache 2.0) on three datasets. Scoring is character-level binary masking (P/R/F1 per category).

AU insurance (synthetic, 100 documents, ~12 PII spans/doc)

Our primary use case. Faker en_AU data plus test-valid TFN/Medicare/ABN/ACN across claim notes, emails, underwriting forms, and medical reports.

Category	Allyanonimiser F1	openai/privacy-filter F1
PERSON	0.836	0.855
ADDRESS	0.980	0.918
EMAIL	1.000	0.982
PHONE	1.000	0.837
DATE	0.863	0.991
Account-like IDs (TFN, Medicare, ABN, policy, VIN, etc.)	0.877	0.846
Overall (any PII)	0.920	0.943

Allyanonimiser wins 4 of 6 categories. openai wins overall by a small margin on precision, but Allyanonimiser runs ~25× faster per document on CPU (1.4s vs 36s for the full eval) and has perfect recall on AU-formatted phones and emails.

AI4Privacy open-pii-500k (English validation, 1,000 rows)

General-purpose multilingual PII. Allyanonimiser's AU-specific phone and account patterns don't match US/EU formats, so it loses those categories by design — the table below explains why each category falls where it does.

Category	Allyanonimiser F1	openai/privacy-filter F1
PERSON	0.653	0.836
EMAIL	0.990	0.915
ADDRESS	0.216	0.464
PHONE	0.011	0.829
DATE	0.490	0.642
ACCOUNT	0.074	0.700
Overall (any PII)	0.729	0.781

Text Anonymization Benchmark (TAB, ECHR legal text, 127 docs)

Real multi-annotator court cases from the European Court of Human Rights. Allyanonimiser doesn't target legal quasi-identifiers like case numbers (TAB CODE) or geographic-entity masking (TAB LOC, ~cities/countries), so those categories are expected low. DATE includes fuzzy temporal expressions ("14 days ago", "summer 2005").

Category	Allyanonimiser F1	openai/privacy-filter F1
PERSON	0.761	0.805
DATE	0.904	0.459
LOCATION	0.424	0.000*
CODE	0.012	0.077
Overall (any PII)	0.560	0.378

*openai's private_address semantics don't cover TAB's LOC (city/country names).

How to read this

• On Australian insurance data, Allyanonimiser is competitive with or beats a state-of-the-art 1.5B-parameter model on 4 out of 6 categories, at ~25× the throughput, and runs on CPU without torch.
• On general multilingual PII, openai/privacy-filter is stronger — its training data covers phone/address/account formats we don't target.
• The methodology is deliberately transparent: all scoring is char-level, and the eval scripts + synthetic AU data are in bench/. Run uv pip install -e ".[bench]" && python bench/run_au_insurance_eval.py to reproduce.

Package Structure (v3.0)

allyanonimiser/
├── __init__.py          # Public API and factory functions
├── allyanonimiser.py    # Allyanonimiser facade
├── core/                # Detection and anonymization engine
│   ├── analyzer.py      # EnhancedAnalyzer, RecognizerResult
│   ├── anonymizer.py    # EnhancedAnonymizer, DEFAULT_ENTITY_PRIORITY
│   ├── pattern_manager.py
│   ├── pattern_registry.py
│   ├── context_analyzer.py
│   └── validators.py
├── io/                  # File and DataFrame adapters
│   ├── csv_processor.py
│   ├── dataframe_processor.py
│   └── stream_processor.py
├── patterns/            # Built-in pattern definitions
│   ├── au_patterns.py
│   ├── insurance_patterns.py
│   └── general_patterns.py
├── utils/               # Helpers (spaCy, settings, text processing)
├── insurance/           # Domain-specific analyzers
└── reporting.py         # Report generation

Migrating from v2.x

Import path changes

# v2.x
from allyanonimiser.enhanced_analyzer import EnhancedAnalyzer
from allyanonimiser.dataframe_processor import DataFrameProcessor
from allyanonimiser.validators import validate_regex

# v3.0
from allyanonimiser.core.analyzer import EnhancedAnalyzer
from allyanonimiser.io.dataframe_processor import DataFrameProcessor
from allyanonimiser.core.validators import validate_regex

# Top-level imports still work for the main API:
from allyanonimiser import create_allyanonimiser, EnhancedAnalyzer, DataFrameProcessor

API changes

# v2.x — stringly-typed
ally.manage_acronyms(action="add", data={"TPD": "Total and Permanent Disability"})
ally.manage_acronyms(action="get")
ally.manage_acronyms(action="remove", data=["TPD"])
ally.manage_patterns(action="create_from_examples", entity_type="MY_ID", examples=[...])
ally.manage_patterns(action="load", filepath="patterns.json")

# v3.0 — explicit methods
ally.add_acronyms({"TPD": "Total and Permanent Disability"})
ally.get_acronyms()
ally.remove_acronyms(["TPD"])
ally.create_pattern_from_examples(entity_type="MY_ID", examples=[...])
ally.load_patterns("patterns.json")

Removed

• setup.py — use pyproject.toml
• generators/ module (was stubs only)
• InsuranceEmailAnalyzer, MedicalReportAnalyzer (were stubs only)
• All deprecated wrapper methods (set_acronym_dictionary, create_dataframe_processor, etc.)

New: Configurable entity priority

from allyanonimiser.core.anonymizer import EnhancedAnonymizer, DEFAULT_ENTITY_PRIORITY

# Override priorities for your use case
custom_priority = {**DEFAULT_ENTITY_PRIORITY, "MY_CUSTOM_TYPE": 95}
anonymizer = EnhancedAnonymizer(analyzer=analyzer, entity_priority=custom_priority)

Supported Entity Types

Allyanonimiser supports 43 different entity types across five categories:

Complete Entity Type Reference

🇦🇺 Australian-Specific Entities (13 types)

Entity Type	Description	Example
AU_TFN	Tax File Number	123 456 789
AU_PHONE	Phone Number	0412 345 678, [phone]
AU_MEDICARE	Medicare Number	2123 45678 1
AU_DRIVERS_LICENSE	Driver's License	VIC1234567, NSW98765
AU_ADDRESS	Street Address	123 Collins St, Melbourne VIC 3000
AU_POSTCODE	Postcode	2000, 3000, 4000
AU_BSB	Bank State Branch	062-000, 123-456
AU_ACCOUNT_NUMBER	Bank Account	1234567890
AU_ABN	Business Number	11 222 333 444
AU_ACN	Company Number	123 456 789
AU_PASSPORT	Passport Number	PA1234567, AB9876543
AU_CENTRELINK_CRN	Centrelink Reference	123 456 789A
VEHICLE_REGISTRATION	Vehicle Rego	ABC123, 1ABC23

🏢 Insurance-Specific Entities (8 types)

Entity Type	Description	Example
INSURANCE_POLICY_NUMBER	Policy Number	POL-12345678, AU-98765432
INSURANCE_CLAIM_NUMBER	Claim Number	CL-23456789, C-987654
VEHICLE_VIN	Vehicle ID Number	1HGCM82633A123456
INVOICE_NUMBER	Invoice/Quote	INV-2024001, Q-5678
BROKER_CODE	Broker Code	BRK-1234
VEHICLE_DETAILS	Vehicle Description	2022 Toyota Camry
INCIDENT_DATE	Date of Incident	on 15/03/2024
NAME_CONSULTANT	Consultant Name	Assigned To: Sarah Johnson

👤 General PII Entities (8 types)

Entity Type	Description	Example
CREDIT_CARD	Credit Card (Luhn-validated 13-19 digits)	4111 1111 1111 1111
PERSON	Person Name	John Smith, Dr. Sarah O'Connor
EMAIL_ADDRESS	Email	[e-mail]
DATE_OF_BIRTH	Date of Birth	DOB: 01/01/1990
LOCATION	Location	Sydney, New South Wales
DATE	General Date	15/03/2024, March 15, 2024
MONEY_AMOUNT	Money Amount	$1,234.56
ORGANIZATION	Organization	ABC Pty Ltd, XYZ Limited

🌐 International / System PII Entities (5 types, added in v3.5)

Non-AU shapes that show up in AU-insurance data when the customer or counterparty is overseas (expats, business travellers), or when system audit logs leak ISO timestamps into claim notes. All anchored on structural features that don't collide with AU patterns. See docs/patterns/international.md.

Entity Type	Description	Example
PHONE_INTL	International phone (+CC, 00 IDD, or 3-4 digit parenthesised area code)	[phone], [phone], [phone]
US_SSN	US Social Security Number (with SSA reservation rules)	818-04-7100
ISO_DATETIME	ISO 8601 datetime (T-separated, optional timezone)	2024-05-22T14:32:00, 2024-05-22T14:32:00Z
TIME	12/24h time	14:32, 6:52 PM, 09:15:42

CREDIT_CARD (in the General section above) was upgraded in v3.5 with Luhn validation so random 16-digit blocks like policy numbers no longer match.

🤖 Additional spaCy NER Entities (9 types)

Entity Type	Description	Example
NUMBER	Numeric Value	42, 1234
TIME	Time Expression	3:30 PM, 14:45
PERCENT	Percentage	25%, 99.9%
PRODUCT	Product Name	iPhone 15, Windows 11
EVENT	Event Name	Olympic Games
WORK_OF_ART	Title	"The Great Gatsby"
LAW	Legal Document	Privacy Act 1988
LANGUAGE	Language	English, Spanish
FACILITY	Building/Airport	Sydney Opera House

Quick Entity Detection Test

from allyanonimiser import create_allyanonimiser

ally = create_allyanonimiser()

# Test text with multiple entity types
test_text = """
Customer: John Smith (TFN: 123 456 789)
Phone: [phone]
Email: [e-mail]
Policy: POL-12345678
Address: 123 Collins St, Melbourne VIC 3000
Payment: $1,234.56 via Credit Card 4111 1111 1111 1111
"""

results = ally.analyze(test_text)

# Display detected entities by type
from collections import Counter
entity_types = Counter(r.entity_type for r in results)
print(f"Found {len(results)} entities across {len(entity_types)} types:")
for entity_type, count in entity_types.most_common():
    print(f"  {entity_type}: {count}")

Usage Examples

Analyze Text for PII Entities

from allyanonimiser import create_allyanonimiser

ally = create_allyanonimiser()

# Analyze text
results = ally.analyze(
    text="Customer John Smith (TFN: 123 456 789) reported an incident on 15/06/2023 at his residence in Sydney NSW 2000."
)

# Print detected entities
for result in results:
    print(f"Entity: {result.entity_type}, Text: {result.text}, Score: {result.score}")

Anonymize Text with Different Operators

from allyanonimiser import create_allyanonimiser, AnonymizationConfig

ally = create_allyanonimiser()

# Using configuration object
config = AnonymizationConfig(
    operators={
        "PERSON": "replace",           # Replace with <PERSON>
        "AU_TFN": "hash",              # Replace with SHA-256 hash
        "DATE": "redact",              # Replace with [REDACTED]
        "AU_ADDRESS": "mask",          # Replace with *****
        "DATE_OF_BIRTH": "age_bracket" # Replace with age bracket (e.g., <40-45>)
    },
    age_bracket_size=5  # Size of age brackets
)

# Anonymize text
anonymized = ally.anonymize(
    text="Customer John Smith (TFN: 123 456 789) reported an incident on 15/06/2023. He was born on 05/08/1982 and lives at 42 Main St, Sydney NSW 2000.",
    config=config
)

print(anonymized["text"])

Process Text with Analysis and Anonymization

from allyanonimiser import create_allyanonimiser, AnalysisConfig, AnonymizationConfig

ally = create_allyanonimiser()

# Configure analysis
analysis_config = AnalysisConfig(
    active_entity_types=["PERSON", "EMAIL_ADDRESS", "PHONE_NUMBER", "DATE_OF_BIRTH"],
    min_score_threshold=0.7
)

# Configure anonymization
anonymization_config = AnonymizationConfig(
    operators={
        "PERSON": "replace",
        "EMAIL_ADDRESS": "mask",
        "PHONE_NUMBER": "redact",
        "DATE_OF_BIRTH": "age_bracket"
    }
)

# Process text (analyze + anonymize)
result = ally.process(
    text="Customer Jane Doe ([e-mail]) called on 0412-345-678 regarding her DOB: 22/05/1990.",
    analysis_config=analysis_config,
    anonymization_config=anonymization_config
)

# Access different parts of the result
print("Anonymized text:")
print(result["anonymized"])

print("\nDetected entities:")
for entity in result["analysis"]["entities"]:
    print(f"{entity['entity_type']}: {entity['text']} (score: {entity['score']:.2f})")

print("\nPII-rich segments:")
for segment in result["segments"]:
    print(f"Original: {segment['text']}")
    print(f"Anonymized: {segment['anonymized']}")

Working with DataFrames

import pandas as pd
from allyanonimiser import create_allyanonimiser

# Create DataFrame
df = pd.DataFrame({
    "id": [1, 2, 3],
    "notes": [
        "Customer John Smith (DOB: 15/6/1980) called about policy POL-123456.",
        "Jane Doe (email: [e-mail]) requested a refund.",
        "Alex Johnson from Sydney NSW 2000 reported an incident."
    ]
})

# Create Allyanonimiser
ally = create_allyanonimiser()

# Anonymize a specific column
anonymized_df = ally.process_dataframe(
    df, 
    column="notes", 
    operation="anonymize",
    output_column="anonymized_notes",  # New column for anonymized text
    operators={
        "PERSON": "replace",
        "EMAIL_ADDRESS": "mask",
        "PHONE_NUMBER": "redact"
    }
)

# Display result
print(anonymized_df[["id", "notes", "anonymized_notes"]])

CSV File Processing

Allyanonimiser provides comprehensive CSV processing capabilities with automatic PII detection, preview modes, and streaming support for large files.

Basic CSV Processing

Process CSV files directly without manual DataFrame operations:

from allyanonimiser import create_allyanonimiser

ally = create_allyanonimiser()

# Auto-detect and process PII columns
result = ally.process_csv_file(
    input_file="customer_data.csv",
    output_file="customer_data_clean.csv"  # Auto-generated if not specified
)

print(f"Processed {result['rows_processed']} rows")
print(f"PII columns found: {result.get('auto_detected_columns', [])}")
print(f"Entities anonymized: {result['entities_found']}")

Auto-Detect PII Columns

Automatically identify which columns contain PII before processing:

# Identify which columns contain PII
pii_columns = ally.detect_pii_columns("data.csv")
print(f"Columns with PII: {pii_columns}")
# Output: ['name', 'email', 'phone', 'notes']

# Process only the detected PII columns
result = ally.process_csv_file(
    input_file="data.csv",
    columns=pii_columns,  # Use detected columns
    output_file="data_anonymized.csv"
)

Preview Changes Before Processing

Preview what will be changed before processing the entire file:

# Preview what will be changed
preview = ally.preview_csv_changes(
    "customer_data.csv",
    sample_rows=5  # Number of rows to preview
)

# Display preview
for _, row in preview.iterrows():
    print(f"{row['column']}: {row['original'][:30]}... → {row['anonymized'][:30]}...")

# If satisfied with preview, process the full file
if input("Proceed with anonymization? (y/n): ").lower() == 'y':
    ally.process_csv_file("customer_data.csv")

Stream Large CSV Files

Process multi-GB files efficiently using streaming:

# Process multi-GB files in chunks
result = ally.stream_process_csv(
    input_file="huge_dataset.csv",  # 10GB file
    output_file="huge_dataset_clean.csv",
    columns=["customer_notes", "comments"],  # Specific columns to process
    chunk_size=10000,  # Process 10k rows at a time
    operators={
        "PERSON": "replace",
        "EMAIL_ADDRESS": "mask",
        "AU_TFN": "hash"
    }
)

print(f"Processed {result['total_rows']} rows in {result['chunks_processed']} chunks")
print(f"Processing time: {result['processing_time']:.2f} seconds")

Process All CSV Files in Directory

Batch process multiple CSV files with a single command:

# Batch process multiple CSV files
results = ally.process_csv_directory(
    input_dir="./raw_data/",
    output_dir="./anonymized_data/",
    columns_to_anonymize=["notes", "description"],  # Or None for auto-detect
    recursive=True,  # Include subdirectories
    generate_report=True  # Create processing report
)

print(f"Processed {len(results['files_processed'])} files")
print(f"Total entities found: {results['total_entities_found']}")
print(f"Report saved to: {results['report_file']}")

Advanced CSV Processing with Custom Configuration

from allyanonimiser import create_allyanonimiser, AnonymizationConfig

ally = create_allyanonimiser()

# Configure anonymization for different entity types
config = AnonymizationConfig(
    operators={
        "PERSON": "replace",
        "EMAIL_ADDRESS": "mask",
        "AU_TFN": "hash",
        "AU_MEDICARE": "redact",
        "CREDIT_CARD": "mask",
        "DATE_OF_BIRTH": "age_bracket"
    },
    age_bracket_size=5
)

# Process with custom configuration
result = ally.process_csv_file(
    input_file="sensitive_data.csv",
    output_file="processed_data.csv",
    config=config,
    columns=None,  # Auto-detect all PII columns
    sample_before_processing=True,  # Show preview first
    generate_stats=True  # Generate statistics report
)

# Access detailed statistics
if result.get('statistics'):
    stats = result['statistics']
    print(f"Entity type distribution:")
    for entity_type, count in stats['entity_counts'].items():
        print(f"  {entity_type}: {count}")

CSV Processing Tutorial

This comprehensive tutorial demonstrates how to use Allyanonimiser's CSV processing features for various scenarios.

Tutorial 1: Basic CSV Anonymization

from allyanonimiser import create_allyanonimiser
import pandas as pd

# Create sample CSV data
data = pd.DataFrame({
    'customer_id': [1001, 1002, 1003],
    'name': ['John Smith', 'Jane Doe', 'Robert Johnson'],
    'email': ['[e-mail]', '[e-mail]', '[e-mail]'],
    'phone': ['0412 345 678', '[phone]', '0499 123 456'],
    'notes': [
        'TFN: 123 456 789, Lives at 123 Collins St Melbourne',
        'Medicare: 2123 45678 1, DOB: 15/06/1980',
        'Policy POL-12345678, Claim CL-987654'
    ]
})

# Save to CSV
data.to_csv('sample_customers.csv', index=False)

# Create Allyanonimiser instance
ally = create_allyanonimiser()

# Process the CSV file
result = ally.process_csv_file(
    input_file='sample_customers.csv',
    output_file='customers_anonymized.csv'
)

# Display results
print(f"Processing complete!")
print(f"Rows processed: {result['rows_processed']}")
print(f"Columns with PII: {result.get('pii_columns', [])}")
print(f"Total entities anonymized: {result['entities_found']}")

# Read and display anonymized data
anonymized_df = pd.read_csv('customers_anonymized.csv')
print("\nAnonymized data:")
print(anonymized_df.to_string())

Tutorial 2: Selective Column Processing

from allyanonimiser import create_allyanonimiser

ally = create_allyanonimiser()

# First, detect which columns contain PII
pii_columns = ally.detect_pii_columns('sample_customers.csv')
print(f"Detected PII columns: {pii_columns}")

# Let user select which columns to anonymize
print("\nSelect columns to anonymize:")
selected_columns = []
for col in pii_columns:
    if input(f"Anonymize '{col}'? (y/n): ").lower() == 'y':
        selected_columns.append(col)

# Process only selected columns
if selected_columns:
    result = ally.process_csv_file(
        input_file='sample_customers.csv',
        output_file='selective_anonymized.csv',
        columns=selected_columns,
        operators={
            "PERSON": "replace",
            "EMAIL_ADDRESS": "mask",
            "AU_TFN": "hash"
        }
    )
    print(f"\nAnonymized {len(selected_columns)} columns")
else:
    print("No columns selected for anonymization")

Tutorial 3: Handling Large Files with Streaming

from allyanonimiser import create_allyanonimiser
import time

ally = create_allyanonimiser()

# For large files, use streaming to process in chunks
print("Processing large CSV file...")
start_time = time.time()

result = ally.stream_process_csv(
    input_file='large_dataset.csv',  # Assume this is a multi-GB file
    output_file='large_dataset_clean.csv',
    chunk_size=5000,  # Process 5000 rows at a time
    columns=['customer_notes', 'description', 'comments'],  # Focus on text columns
    operators={
        "PERSON": "replace",
        "EMAIL_ADDRESS": "mask",
        "AU_TFN": "hash",
        "AU_MEDICARE": "redact"
    },
    progress_callback=lambda chunk_num, total_chunks: 
        print(f"Processing chunk {chunk_num}/{total_chunks}...")
)

elapsed_time = time.time() - start_time
print(f"\nProcessing complete!")
print(f"Total rows: {result['total_rows']}")
print(f"Processing time: {elapsed_time:.2f} seconds")
print(f"Rows per second: {result['total_rows']/elapsed_time:.0f}")

Tutorial 4: Batch Processing Directory

from allyanonimiser import create_allyanonimiser
import os
from pathlib import Path

ally = create_allyanonimiser()

# Setup directories
input_dir = Path('./customer_data')
output_dir = Path('./anonymized_data')
output_dir.mkdir(exist_ok=True)

# Process all CSV files in directory
print(f"Processing all CSV files in {input_dir}...")

results = ally.process_csv_directory(
    input_dir=str(input_dir),
    output_dir=str(output_dir),
    columns_to_anonymize=None,  # Auto-detect for each file
    recursive=True,  # Include subdirectories
    file_pattern='*.csv',  # Process only CSV files
    generate_report=True,
    operators={
        "PERSON": "replace",
        "EMAIL_ADDRESS": "mask",
        "AU_TFN": "hash",
        "AU_MEDICARE": "redact",
        "CREDIT_CARD": "mask"
    }
)

# Display summary
print(f"\nBatch processing complete!")
print(f"Files processed: {len(results['files_processed'])}")
print(f"Files skipped: {len(results.get('files_skipped', []))}")
print(f"Total entities found: {results['total_entities_found']}")

# Display per-file statistics
print("\nPer-file statistics:")
for file_info in results['file_statistics']:
    print(f"  {file_info['filename']}: {file_info['entities_found']} entities")

# Report location
if results.get('report_file'):
    print(f"\nDetailed report saved to: {results['report_file']}")

Tutorial 5: Custom Processing with Validation

from allyanonimiser import create_allyanonimiser
import pandas as pd

ally = create_allyanonimiser()

# Define custom validation function
def validate_anonymization(original_df, anonymized_df):
    """Validate that anonymization was successful"""
    issues = []
    
    # Check that no original PII remains
    pii_patterns = {
        'email': r'[a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\.[a-zA-Z]{2,}',
        'tfn': r'\b\d{3}\s*\d{3}\s*\d{3}\b',
        'medicare': r'\b\d{4}\s*\d{5}\s*\d{1}\b'
    }
    
    for col in anonymized_df.columns:
        for row_idx, value in enumerate(anonymized_df[col]):
            if pd.notna(value):
                value_str = str(value)
                for pii_type, pattern in pii_patterns.items():
                    import re
                    if re.search(pattern, value_str):
                        issues.append(f"Found {pii_type} in row {row_idx}, column '{col}'")
    
    return issues

# Load original data
original_df = pd.read_csv('sample_customers.csv')

# Process with preview first
print("Previewing changes...")
preview = ally.preview_csv_changes('sample_customers.csv', sample_rows=3)
print(preview)

# Get user confirmation
if input("\nProceed with full anonymization? (y/n): ").lower() == 'y':
    # Process the file
    result = ally.process_csv_file(
        input_file='sample_customers.csv',
        output_file='validated_output.csv',
        operators={
            "DEFAULT": "replace"  # Use replace for all entity types
        }
    )
    
    # Load and validate anonymized data
    anonymized_df = pd.read_csv('validated_output.csv')
    validation_issues = validate_anonymization(original_df, anonymized_df)
    
    if validation_issues:
        print("\nValidation issues found:")
        for issue in validation_issues:
            print(f"  - {issue}")
    else:
        print("\n✓ Validation successful: No PII detected in output")
        print(f"✓ {result['entities_found']} entities successfully anonymized")

Tutorial 6: Generating Processing Reports

from allyanonimiser import create_allyanonimiser
import json

ally = create_allyanonimiser()

# Enable detailed reporting
ally.start_new_report("csv_processing_session")

# Process multiple files with reporting
files_to_process = [
    'customers.csv',
    'transactions.csv',
    'support_tickets.csv'
]

for file_path in files_to_process:
    print(f"\nProcessing {file_path}...")
    result = ally.process_csv_file(
        input_file=file_path,
        output_file=f"anonymized_{file_path}",
        generate_stats=True
    )
    
    # Add to report
    ally.add_to_report({
        'file': file_path,
        'rows': result['rows_processed'],
        'entities': result['entities_found'],
        'columns_processed': result.get('pii_columns', [])
    })

# Generate comprehensive report
report = ally.get_report()
summary = report.get_summary()

print("\n" + "="*50)
print("PROCESSING REPORT SUMMARY")
print("="*50)
print(f"Total files processed: {len(files_to_process)}")
print(f"Total entities anonymized: {summary['total_entities']}")
print(f"Average processing time: {summary['avg_processing_time']*1000:.2f} ms")

# Export detailed report
report.export_report("csv_processing_report.html", "html")
report.export_report("csv_processing_report.json", "json")

print(f"\nReports saved:")
print(f"  - csv_processing_report.html (visual report)")
print(f"  - csv_processing_report.json (detailed data)")

Pattern Generalization Levels

When creating patterns from examples using create_pattern_from_examples(), you can specify a generalization level that controls how flexible the resulting pattern will be:

Level	Description	Example Input	Generated Pattern	Will Match
none	Exact matching only	"EMP12345"	\bEMP12345\b	Only "EMP12345"
low	Basic generalization	"EMP12345"	\bEMP\d{5}\b	"EMP12345", "EMP67890", but not "EMP123"
medium	Moderate flexibility	"EMP12345"	\bEMP\d+\b	"EMP12345", "EMP123", "EMP9", but not "EMPLOYEE12345"
high	Maximum flexibility	"EMP12345"	\b[A-Z]{3}\d+\b	"EMP12345", "ABC123", etc.

Higher generalization levels detect more variants but may increase false positives. Choose the appropriate level based on your needs for precision vs. recall.

Anonymization Operators

The package supports several anonymization operators that control how detected entities are transformed:

Operator	Description	Example	Result
replace	Replace with entity type	"John Smith"	<PERSON>
redact	Fully redact the entity	"[e-mail]"	[REDACTED]
mask	Partially mask while preserving structure	"[e-mail]"	j***.s****@e******.com
hash	Replace with consistent hash	"John Smith"	7f9d6a... (same for all "John Smith")
encrypt	Encrypt with a key (recoverable)	"John Smith"	AES256:a7f9c...
age_bracket	Convert dates to age brackets	"DOB: 15/03/1980"	DOB: 40-44
custom	User-defined function	(depends on function)	(custom output)

Example usage:

result = ally.anonymize(
    text="Please contact John Smith at [e-mail]",
    operators={
        "PERSON": "replace",       # Replace with entity type
        "EMAIL_ADDRESS": "mask",   # Partially mask the email
        "PHONE_NUMBER": "redact",  # Fully redact phone numbers
        "DATE_OF_BIRTH": "age_bracket"  # Convert DOB to age bracket
    }
)

Custom Operator Example

The custom operator allows you to define your own transformation function:

from allyanonimiser import create_allyanonimiser

# Create a custom transformation function
def randomize_names(entity_text, entity_type):
    """Replace person names with random names from a predefined list."""
    if entity_type != "PERSON":
        return entity_text
        
    # Simple list of random replacement names
    replacements = ["Alex Taylor", "Sam Johnson", "Jordan Lee", "Casey Brown"]
    
    # Use hash of original name to consistently select the same replacement
    import hashlib
    hash_val = int(hashlib.md5(entity_text.encode()).hexdigest(), 16)
    index = hash_val % len(replacements)
    
    return replacements[index]

# Create an Allyanonimiser instance
ally = create_allyanonimiser()

# Use the custom operator
result = ally.anonymize(
    text="Customer John Smith sent an email to Mary Johnson about policy POL-123456.",
    operators={
        "PERSON": randomize_names,  # Pass the function directly
        "POLICY_NUMBER": "mask"     # Other operators work as usual
    }
)

print(result["text"])
# Output: "Customer Alex Taylor sent an email to Sam Johnson about policy ***-******."

Custom operators are powerful for specialized anonymization needs like:

• Generating synthetic but realistic replacements
• Contextual anonymization based on entity values
• Domain-specific transformations (e.g., preserving data distributions)
• Implementing differential privacy mechanisms

Pattern Management

from allyanonimiser import create_allyanonimiser

# Create an instance
ally = create_allyanonimiser()

# 1. Adding pattern to an existing group in pattern files
# If you want to contribute a new pattern to the codebase,
# edit the appropriate file in patterns/ directory:
#  - patterns/au_patterns.py: For Australian-specific patterns
#  - patterns/general_patterns.py: For general PII patterns 
#  - patterns/insurance_patterns.py: For insurance-specific patterns

# 2. Using custom patterns without modifying code
# Add a custom pattern with detailed options
ally.add_pattern({
    "entity_type": "COMPANY_PROJECT_ID",
    "patterns": [
        r"PRJ-\d{4}-[A-Z]{3}",       # Format: PRJ-1234-ABC
        r"Project\s+ID:\s*(\d{4})"   # Format: Project ID: 1234
    ],
    "context": ["project", "id", "identifier", "code"],
    "name": "Company Project ID",
    "score": 0.85,                   # Confidence score (0-1)
    "language": "en",                # Language code
    "description": "Internal project identifier format"
})

# 3. Save patterns for reuse
ally.export_config("company_patterns.json")

# 4. Load saved patterns in another session
new_ally = create_allyanonimiser(settings_path="company_patterns.json")

# 5. Pattern testing and validation
from allyanonimiser.validators import test_pattern_against_examples

# Test if a pattern works against examples
results = test_pattern_against_examples(
    pattern=r"PRJ-\d{4}-[A-Z]{3}",
    positive_examples=["PRJ-1234-ABC", "PRJ-5678-XYZ"],
    negative_examples=["PRJ-123-AB", "PROJECT-1234"]
)
print(f"Pattern is valid: {results['is_valid']}")
print(f"Diagnostic info: {results['message']}")

Advanced Features

Generating Reports

from allyanonimiser import create_allyanonimiser
import os

# Create output directory
os.makedirs("output", exist_ok=True)

# Create an Allyanonimiser instance
ally = create_allyanonimiser()

# Start a new report session
ally.start_new_report(session_id="example_session")

# Configure anonymization
anonymization_config = {
    "operators": {
        "PERSON": "replace",
        "EMAIL_ADDRESS": "mask",
        "PHONE_NUMBER": "redact",
        "AU_ADDRESS": "replace",
        "DATE_OF_BIRTH": "age_bracket",
        "AU_TFN": "hash",
        "AU_MEDICARE": "mask"
    },
    "age_bracket_size": 10
}

# Process a batch of files
result = ally.process_files(
    file_paths=["data/sample1.txt", "data/sample2.txt", "data/sample3.txt"],
    output_dir="output/anonymized",
    anonymize=True,
    operators=anonymization_config["operators"],
    report=True,
    report_output="output/report.html",
    report_format="html"
)

# Display summary
print(f"Processed {result['total_files']} files")
print(f"Detected {result['report']['total_entities']} entities")
print(f"Average processing time: {result['report']['report']['avg_processing_time']*1000:.2f} ms")

Comprehensive Reporting System

Allyanonimiser includes a comprehensive reporting system that allows you to track, analyze, and visualize anonymization activities.

from allyanonimiser import create_allyanonimiser

# Create instance
ally = create_allyanonimiser()

# Start a new report session
ally.start_new_report("my_session")

# Process multiple texts
texts = [
    "Customer John Smith (DOB: 15/06/1980) called about claim CL-12345.",
    "Jane Doe (email: [e-mail]) requested policy information.",
    "Claims assessor reviewed case for Robert Johnson (ID: 987654321)."
]

for i, text in enumerate(texts):
    ally.anonymize(
        text=text,
        operators={
            "PERSON": "replace",
            "EMAIL_ADDRESS": "mask",
            "DATE_OF_BIRTH": "age_bracket"
        },
        document_id=f"doc_{i+1}"
    )

# Get report summary
report = ally.get_report()
summary = report.get_summary()

print(f"Total documents processed: {summary['total_documents']}")
print(f"Total entities detected: {summary['total_entities']}")
print(f"Entities per document: {summary['entities_per_document']:.2f}")
print(f"Anonymization rate: {summary['anonymization_rate']*100:.2f}%")
print(f"Average processing time: {summary['avg_processing_time']*1000:.2f} ms")

# Export report to different formats
report.export_report("report.html", "html")  # Rich HTML visualization
report.export_report("report.json", "json")  # Detailed JSON data
report.export_report("report.csv", "csv")    # CSV statistics

In Jupyter Notebooks

from allyanonimiser import create_allyanonimiser
import pandas as pd
import matplotlib.pyplot as plt

# Create an Allyanonimiser instance
ally = create_allyanonimiser()

# Start a report session and process some texts
# ... processing code ...

# Display rich interactive report
ally.display_report_in_notebook()

# Access report data programmatically
report = ally.get_report()
summary = report.get_summary()

# Create custom visualizations
entity_counts = summary['entity_counts']
plt.figure(figsize=(10, 6))
plt.bar(entity_counts.keys(), entity_counts.values())
plt.title('Entity Type Distribution')
plt.xlabel('Entity Type')
plt.ylabel('Count')
plt.xticks(rotation=45)
plt.tight_layout()
plt.show()

What's New in Version 2.1.0

• Added support for NSW legacy driver's license pattern
• Improved pattern recognition for Australian TFNs
• Enhanced handling of date formats
• Fixed issues with BSB recognition
• Added more comprehensive test suite for Australian patterns
• Performance improvements for large file processing

For older versions and detailed change history, see the CHANGELOG.md file.

Documentation

For complete documentation, examples, and advanced usage, visit our documentation site:

• Installation Guide
• Quick Start Tutorial
• Pattern Reference
• Anonymization Operators
• API Reference

License

This project is licensed under the MIT License - see the LICENSE file for details.