Using an AI coding assistant? This repo includes an llms.txt with the full API surface, config reference, and Q&A - optimised for Claude, Cursor, Copilot, and ChatGPT.

In the world of machine learning and natural language processing, clean and well-structured text data is crucial for building effective downstream models and managing token limits in language models.

SqueakyCleanText simplifies the process by automatically addressing common text issues - removing PII, anonymizing named entities (persons, organisations, locations), and ensuring your data is clean and well-structured for language models and classical ML pipelines with minimal effort on your part.

Key Features

• Named Entity Recognition (NER):
  • Multi-backend: ONNX (default, torch-free), PyTorch, GLiNER, and ensemble modes
  • Zero-shot custom entities via GLiNER (e.g., PRODUCT, EVENT, SKILL)
  • Multi-language support (English, Dutch, German, Spanish, French, Portuguese, Italian)
  • Ensemble voting across backends for improved accuracy
  • Configurable confidence thresholds
  • Lazy model loading (models load on demand per language)
  • Shared ONNX sessions across same-model languages (~600 MB RAM saved)
  • Automatic text chunking for long documents (CJK/Arabic safe)
  • GPU acceleration support (CUDA for ONNX and PyTorch)
  • Model warm-up API to pre-load on startup
• Text Normalization:
  • Corrects text encoding problems and handles bad Unicode characters
  • Removes or replaces HTML tags and URLs with configurable tokens
  • Handles emails, phone numbers, and other contact details
  • Multilingual date detection and replacement (ISO 8601, month names, common formats)
  • Fuzzy date matching for misspelled months (requires [fuzzy] extra)
  • Year and number standardization
  • Configurable emoji removal
  • Configurable bracket/brace content removal
  • Removes isolated letters and symbols
  • Normalizes whitespace and handles currency symbols
  • Smart case folding (preserves NER tokens like <PERSON>)
• Language Support:
  • Automatic language detection (English, Dutch, German, Spanish)
  • Language-specific NER models; French, Portuguese, Italian via multilingual model
  • Language-aware stopword removal
  • Extensible: add custom languages with stopwords, month names, and NER models
• Dual Output Formats:
  • Language Model format (preserves structure with tokens)
  • Statistical Model format (optimized for classical ML)
• Performance:
  • ONNX Runtime inference (torch-free base install, ~3-5x faster than PyTorch)
  • Thread-parallel batch processing via ThreadPoolExecutor
  • Async batch processing (aprocess_batch) for FastAPI / aiohttp
  • Lazy model loading (only loads models as needed)
  • Shared ONNX sessions for same-model languages (saves ~600 MB for FR/PT/IT)
  • Memory-efficient processing of large texts
  • GPU acceleration (CUDA) for both ONNX and PyTorch backends

Benefits

For Language Models

• Maintains text structure while anonymizing sensitive information
• Configurable token replacements
• Preserves context while removing noise
• Handles long documents through intelligent chunking

For Statistical Models

• Removes stopwords and punctuation
• Case normalization
• Special symbol removal
• Optimized for classification tasks

Advanced NER Processing

• Ensemble approach reduces missed entities
• Language-specific models improve accuracy
• Confidence thresholds for precision control
• Efficient batch processing for large datasets
• Automatic handling of long documents

Installation

pip install SqueakyCleanText

The base install uses ONNX Runtime for NER inference - no PyTorch or Transformers required.

Optional Extras

You can combine extras: pip install SqueakyCleanText[gpu,fuzzy,gliner]

Usage

Basic Usage

from sct import TextCleaner

# Initialize the TextCleaner
cleaner = TextCleaner()

# Input text
text = "Contact John Doe at john.doe@company.com. Meeting on 2023-10-01."

# Process the text
lm_text, stat_text, lang = cleaner.process(text)

print(f"Language Model format:    {lm_text}")
# Output: "Contact <PERSON> at <EMAIL>. Meeting on <YEAR>."

print(f"Statistical Model format: {stat_text}")
# Output: "contact meeting"

print(f"Detected Language: {lang}")
# Output: "ENGLISH"

Using TextCleanerConfig

from sct import TextCleaner, TextCleanerConfig

# Create an immutable configuration
cfg = TextCleanerConfig(
    check_ner_process=True,
    ner_confidence_threshold=0.85,
    positional_tags=('PER', 'LOC', 'ORG', 'MISC'),
    replace_with_url="<URL>",
    replace_with_email="<EMAIL>",
    replace_with_phone_numbers="<PHONE>",
    language="en",  # Pin to English (also accepts 'ENGLISH', 'eng')
)

# Initialize with config
cleaner = TextCleaner(cfg=cfg)

Language Specification

All language parameters accept Lingua names ('ENGLISH'), ISO 639-1 ('en'), or ISO 639-3 ('eng') codes:

# Pin to one language (skip auto-detection)
cfg = TextCleanerConfig(language='de', check_ner_process=False)

# Restrict detection to specific languages (auto-detect among them)
cfg = TextCleanerConfig(language=('en', 'nl', 'de'), check_ner_process=False)

# Add extra languages for detection
cfg = TextCleanerConfig(extra_languages=('fr', 'pt'), check_ner_process=False)

GLiNER: Zero-Shot Custom NER

Use GLiNER to recognize any entity type without retraining:

from sct import TextCleaner, TextCleanerConfig

cfg = TextCleanerConfig(
    ner_backend='gliner',
    gliner_model='urchade/gliner_large-v2.1',
    gliner_labels=('person', 'organization', 'location', 'product', 'event'),
    gliner_label_map={
        'person': 'PER', 'organization': 'ORG', 'location': 'LOC',
        # 'product' and 'event' are unmapped - they become <PRODUCT>, <EVENT> tokens
    },
    gliner_threshold=0.4,
)

cleaner = TextCleaner(cfg=cfg)
lm_text, stat_text, lang = cleaner.process(
    "John bought an iPhone at the Apple Store in Berlin during CES 2025."
)
# lm_text: "<PERSON> bought an <PRODUCT> at the <ORGANISATION> in <LOCATION> during <EVENT>."

Ensemble NER

Combine ONNX/Torch models with GLiNER for improved recall via ensemble voting:

from sct import TextCleaner, TextCleanerConfig

cfg = TextCleanerConfig(
    ner_backend='ensemble_onnx',  # or 'ensemble_torch'
    gliner_model='urchade/gliner_large-v2.1',
    gliner_labels=('person', 'organization', 'location'),
    gliner_label_map={'person': 'PER', 'organization': 'ORG', 'location': 'LOC'},
)

cleaner = TextCleaner(cfg=cfg)
lm_text, stat_text, lang = cleaner.process("Angela Merkel visited the Bundestag in Berlin.")

PII Detection Mode

Automatically configure GLiNER for comprehensive PII detection with 60+ entity types (personal, financial, healthcare, identity, digital):

from sct import TextCleaner, TextCleanerConfig

cfg = TextCleanerConfig(ner_mode='pii')

cleaner = TextCleaner(cfg=cfg)
lm_text, stat_text, lang = cleaner.process(
    "John Smith's SSN is 123-45-6789, email john@example.com, DOB 1990-01-15"
)
# Entities are anonymized: names, SSNs, emails, dates of birth, and 50+ more PII types

PII mode auto-configures: ner_backend='gliner', uses knowledgator/gliner-pii-base-v1.0, sets threshold to 0.3 (recall-focused), and expands positional tags. User-provided values always take priority.

Alternative PII models (pass as gliner_model):

Synthetic Replacement

Replace detected entities with realistic fake values (via Faker) instead of <TAG> placeholder tokens:

from sct import TextCleaner, TextCleanerConfig

cfg = TextCleanerConfig(
    ner_mode='pii',
    replacement_mode='synthetic',  # pip install squeakycleantext[synthetic]
)

cleaner = TextCleaner(cfg=cfg)
lm_text, stat_text, lang = cleaner.process(
    "Contact John Smith at john.smith@company.com or +1-555-0123"
)
# Output: "Contact Jennifer Williams at lisa45@example.net or +1-555-0198"
# Same entity always maps to same fake value within a document

Note: Synthetic replacement preserves data utility for downstream ML tasks but is NOT GDPR-compliant anonymization. Same-document consistency is maintained (same entity text always maps to the same fake value).

Reversible Anonymization

Replace entities with indexed placeholders (<PERSON_0>, <LOCATION_1>) and get a mapping for round-trip deanonymization:

from sct import TextCleaner, TextCleanerConfig

cfg = TextCleanerConfig(
    ner_mode='pii',
    replacement_mode='reversible',
)

cleaner = TextCleaner(cfg=cfg)
result = cleaner.process("John Smith works at Google in London.")

print(result.lm_text)
# "<PERSON_0> works at <ORGANISATION_0> in <LOCATION_0>."

# Access the anonymization map via metadata
anon_map = result.metadata['anon_map']
restored = anon_map.deanonymize(result.lm_text)
# "John Smith works at Google in London."

# Serialize the map for storage
import json
json.dumps(anon_map.to_dict())

Note: ProcessResult from process() unpacks as a 3-tuple (lm_text, stat_text, language) for backward compatibility, but also exposes .metadata for reversible maps and document classification.

Document Classification (GLiClass)

Classify documents before processing using zero-shot classification with GLiClass:

from sct import TextCleaner, TextCleanerConfig

cfg = TextCleanerConfig(
    check_classify_document=True,
    gliclass_labels=('email', 'code', 'legal', 'medical'),
    # gliclass_model defaults to 'knowledgator/gliclass-edge-v3.0' (32.7M params)
)

cleaner = TextCleaner(cfg=cfg)  # pip install squeakycleantext[classify]
result = cleaner.process("Dear Sir, please find attached the contract...")

# Classification results in metadata
print(result.metadata['classes'])
# [{"label": "email", "score": 0.92}, {"label": "legal", "score": 0.78}]

Bi-Encoder GLiNER Models

Bi-encoder models (ModernBERT, etc.) are auto-detected and leverage pre-computed label embeddings for faster inference with larger context windows:

from sct import TextCleaner, TextCleanerConfig

cfg = TextCleanerConfig(
    ner_backend='gliner',
    gliner_model='knowledgator/gliner-bi-base-v2.0',
    gliner_labels=('person', 'organization', 'location'),
)

cleaner = TextCleaner(cfg=cfg)
# Auto-detects bi-encoder → caches label embeddings → uses 2048+ token context window

Entity Description Labels (ZERONER-Style)

Provide natural-language descriptions for labels to improve zero-shot recognition accuracy:

from sct import TextCleaner, TextCleanerConfig

cfg = TextCleanerConfig(
    ner_backend='gliner',
    gliner_model='knowledgator/gliner-bi-base-v2.0',
    gliner_label_descriptions={
        'person': "a person's full legal name",
        'location': "a geographical place or address",
        'organization': "a company, institution, or government body",
    },
)

cleaner = TextCleaner(cfg=cfg)
# Descriptions are used for inference, results are mapped back to original label names

Batch Processing

from sct import TextCleaner, TextCleanerConfig

cfg = TextCleanerConfig(
    check_remove_stopwords=True,
    check_remove_punctuation=True,
    check_ner_process=True,
    positional_tags=('PER', 'ORG', 'LOC'),
    ner_confidence_threshold=0.90,
)

cleaner = TextCleaner(cfg=cfg)

# Sample texts
texts = [
    "Email maria.garcia@example.es for more info.",  # Spanish
    "Besuchen Sie uns im Büro in Berlin.",           # German
    "Voor vragen, bel +31 20 123 4567.",             # Dutch
]

# Process texts in batch (uses ThreadPoolExecutor for parallel processing)
results = cleaner.process_batch(texts, batch_size=2)

for lm_text, stat_text, lang in results:
    print(f"Language: {lang}")
    print(f"LM Format:    {lm_text}")
    print(f"Stat Format:  {stat_text}")
    print("-" * 40)

from sct import sct, config

# Customize settings via module-level variables
config.CHECK_NER_PROCESS = True
config.NER_CONFIDENCE_THRESHOLD = 0.85
config.POSITIONAL_TAGS = ['PER', 'LOC', 'ORG']
config.REPLACE_WITH_URL = "<URL>"
config.REPLACE_WITH_EMAIL = "<EMAIL>"
config.LANGUAGE = "ENGLISH"

# Initialize (reads from module-level config)
cleaner = sct.TextCleaner()

NER Backends

SqueakyCleanText supports six NER backends, selectable via the ner_backend config field:

Default NER Models (ONNX)

GLiNER Model Recommendations

GLiNER2 note: pip install squeakycleantext[gliner2] installs Knowledgator's gliner2 package, not Fastino AI's GLiNER2 from EMNLP 2025 (different API).

GLiNER Label Mapping

GLiNER uses lowercase free-text labels (e.g., 'person', 'product'). To map them to standard NER tags used by the anonymizer, use gliner_label_map:

gliner_label_map={
    'person': 'PER',          # → <PERSON>
    'organization': 'ORG',    # → <ORGANISATION>
    'location': 'LOC',        # → <LOCATION>
}
# Unmapped labels are uppercased automatically:
# 'product' → <PRODUCT>, 'event' → <EVENT>, 'skill' → <SKILL>

API

TextCleaner

process(text: str) -> Tuple[str, Optional[str], Optional[str]]

Processes the input text and returns a tuple containing:

• Cleaned text formatted for language models.
• Cleaned text formatted for statistical models (None if check_statistical_model_processing is False).
• Detected language of the text (None if language detection is disabled).

process_batch(texts: List[str], batch_size: int = None) -> List[Tuple[str, Optional[str], Optional[str]]]

Processes multiple texts using thread-parallel execution. Each result follows the same format as process().

aprocess_batch(texts: List[str], batch_size: int = None) -> List[Tuple[str, Optional[str], Optional[str]]]

Async version of process_batch for use with asyncio-based frameworks (FastAPI, aiohttp). Runs the batch in a thread-pool executor so it does not block the event loop:

from sct import TextCleaner

cleaner = TextCleaner()

# In an async context (FastAPI route, aiohttp handler, etc.)
results = await cleaner.aprocess_batch(texts)

warmup(languages: Optional[List[str]] = None) -> None

Pre-loads NER models to avoid first-request latency. Call once during application startup:

cleaner = TextCleaner()
cleaner.warmup(['ENGLISH', 'DUTCH'])  # or warmup() for all supported languages

TextCleanerConfig

Immutable (frozen) dataclass. Create modified copies with dataclasses.replace():

import dataclasses
new_cfg = dataclasses.replace(cfg, check_ner_process=False)

Privacy Engine

Applications can apply versioned policy rules and inspect each decision. This makes privacy behavior testable across services, datasets, and model upgrades.

from sct import EntityRule, PrivacyPolicy, TextCleaner, TextCleanerConfig

policy = PrivacyPolicy(
    name="support-export",
    version="1",
    rules={
        "PERSON": EntityRule(action="redact", min_score=0.8),
        "ORGANISATION": EntityRule(action="keep"),
    },
    allowlist=frozenset({"Jane Public"}),
)

cleaner = TextCleaner(
    cfg=TextCleanerConfig(ner_mode="pii"),
    privacy_policy=policy,
    include_entities=True,
)
result = cleaner.process("Contact Maria Chen for help.")

print(result.metadata["findings"])
print(result.metadata["detector"])

Reversible output is pseudonymized personal data. Anyone with the token map can restore the source values. Keep token maps outside logs and cleaned datasets.

Use a TokenStore implementation to keep maps behind opaque references:

from sct import InMemoryTokenStore, TextCleaner, TextCleanerConfig

store = InMemoryTokenStore()
cleaner = TextCleaner(
    cfg=TextCleanerConfig(replacement_mode="reversible"),
    token_store=store,
)
result = cleaner.process("Maria lives in Utrecht.")
reference = result.metadata["anon_map_reference"]

InMemoryTokenStore supports tests and short sessions. Production systems should implement TokenStore with encryption, access control, expiry, and audit logs.

Boundary adapters

The base package includes JSON, DataFrame, retrieval, command-line, and streaming adapters:

from sct import RAGChunk, process_json, process_rag_chunks

cleaned_json = process_json(cleaner, {"message": "Email a@example.com"})
cleaned_chunks = process_rag_chunks(
    cleaner,
    [RAGChunk(id="1", text="Call +1-202-555-0147", metadata={})],
)

printf 'Email maria@example.com' | sct-clean --no-ner
printf 'Email maria@example.com' | sct-clean --no-ner \
  --replacement-mode reversible --token-map tokens.json
pip install 'SqueakyCleanText[mcp]'
sct-mcp

BufferedStreamingRedactor holds a full document for cross-chunk NER safety. WindowedStreamingRedactor provides bounded latency for contact tokens.

The OCR adapter accepts any OCRProvider. The included Tesseract provider also needs a local Tesseract executable. Browser export writes an ONNX Runtime Web manifest, but it does not copy model files or Python language detection.

Corpus owners can call assess_reidentification_risk to measure k-anonymity for selected quasi-identifiers. Reports contain counts only, not source values.

PII Evaluation

Model changes need ground-truth privacy evaluation, not only backend agreement. The evaluation harness reports exact entity metrics by label, language, domain, and named slice.

python -m scripts.evaluate_pii \
  --dataset evaluation/pii_starter.jsonl \
  --predictions path/to/model-predictions.jsonl \
  --model model-name-and-revision \
  --output evaluation/results/model-name.json

The included starter corpus is synthetic and only checks the evaluation path. See evaluation/README.md for schemas and release rules.

Pinned candidates live in sct.model_candidates. Run one candidate with:

python -m scripts.benchmark_models \
  --dataset evaluation/pii_starter.jsonl \
  --candidate gliner2-pii \
  --output evaluation/results/gliner2-pii.json

Published third-party results are reference data only. assess_candidate requires local recall, exact-span F1, latency, and ONNX parity before adoption.

Architecture

SqueakyCleanText processes text through a configurable pipeline of sequential steps:

Input Text
  │
  ├─ Fix Unicode (ftfy)
  ├─ ASCII transliteration (unidecode)
  ├─ Emoji removal
  ├─ HTML replacement
  ├─ URL / Email / Phone replacement
  ├─ Date & Year replacement
  ├─ Number & Currency replacement
  ├─ Isolated letter/symbol removal
  ├─ Whitespace normalization
  │
  ├─ NER Processing (ONNX / Torch / GLiNER / Ensemble)
  │   ├─ Language detection (Lingua)
  │   ├─ Text chunking (token-bounded)
  │   ├─ Entity recognition (per-chunk)
  │   ├─ Ensemble voting (cross-model)
  │   └─ Entity anonymization (Presidio)
  │
  └─ Statistical Model Output
      ├─ Case folding
      ├─ Stopword removal
      └─ Punctuation removal

  ▼
(lm_text, stat_text, language)

Each step is toggled by a TextCleanerConfig field. The pipeline is built once at initialization; disabled steps are skipped entirely (zero overhead).

What's New

v0.7.0

• Privacy policies: Apply versioned entity rules, allowlists, thresholds, and redaction actions.
• Structured audit findings: Inspect entity decisions, detector backends, and pinned model revisions.
• Safer reversible processing: Restore contact values and keep token maps behind an optional TokenStore.
• Boundary adapters: Clean JSON, DataFrame-like objects, RAG chunks, standard input, and MCP tool requests.
• Privacy evaluation: Score exact PII spans by label, language, domain, and custom slice.
• Model adoption gates: Compare pinned candidates using recall, F1, latency, and ONNX parity.
• Extended media support: Redact OCR regions, process bounded streams, and export browser deployment manifests.
• Corpus risk indicators: Measure k-anonymity for user-selected quasi-identifiers.
• Supply-chain improvements: Pin model revisions, publish through PyPI Trusted Publishing, and ship standard lockfiles.
• Python 3.14: Added CI coverage alongside Python 3.11 through 3.13.

Migration notes:

• gliclass_onnx=True now raises an error because the upstream package has no native ONNX loader.
• HTML processing now honors replace_with_html instead of silently stripping tags.
• Structured adapters reject reversible mode unless each token map can be retained safely.

v0.6.0

• PII detection mode (ner_mode='pii'): auto-configures GLiNER with 60+ PII entity labels (personal, financial, healthcare, identity, digital)
• Synthetic replacement (replacement_mode='synthetic'): Faker-generated realistic values instead of <TAG> placeholders, with per-document consistency
• Reversible anonymization (replacement_mode='reversible'): indexed placeholders (<PERSON_0>) with AnonymizationMap for round-trip deanonymization
• Document classification (check_classify_document=True): zero-shot GLiClass pre-classification before text processing
• ProcessResult: process() returns ProcessResult (backward-compatible 3-tuple) with .metadata for anonymization maps and classification results
• GLiNER ONNX mode (gliner_onnx=True): load GLiNER with pre-built ONNX weights from HuggingFace Hub (auto-set for PII + ONNX backend)
• Bi-encoder support: auto-detects ModernBERT and other bi-encoder GLiNER models, caches label embeddings, dynamic context windows (2048-8192 tokens)
• Entity description labels: ZERONER-style natural-language descriptions for improved zero-shot accuracy
• Presidio GLiNER backend (beta): opt-in ner_backend='presidio_gliner' for Presidio's context-aware recognition pipeline
• ModernBERT ONNX export: updated export script with ModernBERT support (English, 8192 token context)
• Dynamic chunk sizing: GLiNER chunk size adapts to model's actual context window instead of hardcoded 384

v0.5.x

• aprocess_batch(): async batch processing for FastAPI / aiohttp integrations
• warmup(languages): pre-load NER models at startup to eliminate first-request latency
• custom_pipeline_steps: attach arbitrary (text: str) -> str callables after the built-in pipeline
• French, Portuguese, and Italian NER support via a shared multilingual ONNX session
• Improved NER sentence boundary detection with abbreviation guard

v0.4.5

• Frozen TextCleanerConfig dataclass: immutable, thread-safe, per-instance configuration
• ONNX-first NER inference: torch-free base install (~400 MB models vs ~7 GB)
• Thread-parallel batch processing via ThreadPoolExecutor
• Five NER backends: onnx, torch, gliner, ensemble_onnx, ensemble_torch
• GLiNER zero-shot NER for custom entity types (PRODUCT, EVENT, SKILL, etc.)
• Ensemble voting across backends for improved recall
• Lazy per-language model loading
• Multilingual date detection and fuzzy date matching
• Configurable emoji removal, bracket/brace content removal, and smart case folding
• stop-words replaces NLTK (50 KB bundled vs 30 MB download)
• PyTorch and Transformers moved to optional extras
• Migrated to pyproject.toml (PEP 517), Python 3.11-3.13, ruff linter

Contributing

Contributions are welcome! Please feel free to submit a Pull Request or open an issue.

License

This project is licensed under the MIT License - see the LICENSE file for details.

Acknowledgements

The package took inspirations from the following repo:

• clean-text