SqueakyCleanText

Clean, privacy-safe text datasets for LLM training, fine-tuning and RAG, in 75 languages.

SqueakyCleanText (SCT) is a text processing pipeline for datasets. A JSONL file, a Hugging Face dataset or a DataTrove job goes in. Cleaned text comes out, plus sct_* columns and a data card that describe each record without storing personal data. It runs on CPU or GPU.

in:   I’m Anna Berg & my mail is anna@example.se. Order 4711 ships on 2024-05-01.
out:  I'm <PERSON> & my mail is <EMAIL>. Order 4711 ships on 2024-05-01.
      sct_language=ENGLISH  sct_pii_count=2  sct_pii_types=['EMAIL', 'PERSON']

in:   Bel +31 20 123 4567 voor Jan de Vries.
out:  Bel <PHONE> voor <PERSON>.
      sct_language=DUTCH  sct_pii_count=2  sct_pii_types=['PERSON', 'PHONE']

Documentation: https://rhnfzl.github.io/SqueakyCleanText/

What it does

One call runs these stages. Each one is a config field with a safe default, so you can switch it off.

1. Detect the language. One of 75 languages, with a confidence score. Pin it or limit it when you know.

2. Repair the text. Fixes mojibake and HTML entities (ftfy), folds styled characters, tidies whitespace, and can replace HTML tags, numbers and dates with tags.

3. Remove personal data. Regex rules (emails, phones, URLs, IPs, IBAN and card numbers) and an ensemble of multilingual NER models (names, places, organisations). Found values become:

   • tags such as <PERSON> (default),
   • realistic fake values (synthetic), or
   • reversible placeholders <PERSON_0> plus a map you store apart.

   SCT ships no word lists; pass your own terms to always mask (custom_deny_terms), never mask (custom_allow_terms), or your own labelled regex (custom_patterns).

4. Score and document. Every result has value-free signals. Turn them into sct_* columns (language, PII share, warnings; optional quality score and MinHash for near-duplicate removal) and a data card with sct-report (docs/SCORE.md).

5. Scale. process_batch runs documents in threads and batches the NER calls, on CPU (ONNX Runtime, no PyTorch needed) or an NVIDIA GPU. Adapters for Hugging Face datasets, DataTrove, LangChain, Haystack and MCP.

SCT finds most personal data, not all of it. Check its output before you publish sensitive data.

Install

pip install squeakycleantext

Python 3.12 or newer. Add extras for what you use, for example pip install "squeakycleantext[datasets,gpu]":

All extras: docs/INSTALL.md.

Quickstart

For training and RAG data, use the training preset. It repairs the text and replaces personal data. Numbers, dates, HTML, case, line breaks and code stay as written.

from sct import TextCleaner, preset_config
from sct.score import score_record

cleaner = TextCleaner(cfg=preset_config("training"))
for r in cleaner.process_batch([
    "I’m Anna Berg & my mail is anna@example.se. Order 4711 ships on 2024-05-01.",
    "Bel +31 20 123 4567 voor Jan de Vries.",
]):
    row = score_record(r)
    print(r.lm_text)
    print(row["sct_language"], row["sct_pii_count"], row["sct_pii_types"])
# -> I'm <PERSON> & my mail is <EMAIL>. Order 4711 ships on 2024-05-01.
# -> ENGLISH 2 ['EMAIL', 'PERSON']
# -> Bel <PHONE> voor <PERSON>.
# -> DUTCH 2 ['PERSON', 'PHONE']

The first run downloads the NER models.

A whole Hugging Face dataset (datasets extra). clean_dataset wraps Dataset.map: batch_size, num_proc and other map arguments pass through. output="new_column" keeps the original text next to the cleaned text_clean, and score=True adds the sct_* columns.

from datasets import Dataset
from sct.integrations.hf import clean_dataset

if __name__ == "__main__":
    ds = Dataset.from_dict({"text": ["Mail jan@example.nl", "Call Maria Chen on +44 20 7946 0958."]})
    ds = clean_dataset(ds, columns=["text"], output="new_column", score=True, num_proc=2)
    print(ds[1]["text_clean"], ds[1]["sct_pii_types"])
    # -> Call <PERSON> on <PHONE>. ['PERSON', 'PHONE']

Full cleanup for classic ML. The default config (TextCleaner()) also replaces ID-shaped numbers, dates and HTML, and stat_text gives a lower-cased version without stopwords and punctuation:

from sct import TextCleaner

result = TextCleaner().process("I’m Anna Berg & my mail is anna@example.se. Order 4711 ships on 2024-05-01.")
print(result.lm_text)    # -> I'm <PERSON> & my mail is <EMAIL>. Order <NUMBER> ships on <DATE>.
print(result.stat_text)  # -> person mail email number ships date

Use it in your data pipeline

DataTrove (datatrove extra). Put SCT after your language and quality filters, so the NER step only sees documents you keep:

from datatrove.executor import LocalPipelineExecutor
from datatrove.pipeline.readers import JsonlReader
from datatrove.pipeline.writers import JsonlWriter
from sct.integrations.datatrove import SCTCleaner

if __name__ == "__main__":
    LocalPipelineExecutor(
        pipeline=[JsonlReader("raw/"), SCTCleaner(), JsonlWriter("clean/")],
        tasks=4,
        workers=2,
    ).run()

Everything else:

The framework adapters use the training preset unless you pass a config, so their output keeps numbers, dates, HTML and case. They build one cleaner per worker process and reject reversible mode. Plan for 5 to 7 GB of RAM per worker with the default NER ensemble.

How it works

Every step is a TextCleanerConfig field (docs/CONFIG.md). The diagrams are built by scripts/diagrams/; open the .excalidraw files in resources/ to view them in Excalidraw.

What you get back

process and process_batch return a ProcessResult:

• lm_text: the cleaned text, sentences intact, for LLMs, RAG and training.
• stat_text: lower-cased, without stopwords and punctuation, for classic ML models.
• language: the detected language.
• signals: a value-free view with the same keys for every input: language and confidence, pii_count, pii_share, pii_types, sensitive, warnings and the NER models that ran. Safe to log or store next to the text.
• metadata: the same counts, plus raw values only when you ask for them (the reversible map, or findings with include_entities=True). Keep those out of logs.

score_record turns a result into flat sct_* columns for filtering and data cards. ScoreConfig adds an optional quality score and a MinHash signature. Fields: docs/API.md, columns: docs/SCORE.md.

Modes, privacy policies and more PII types (ner_mode="pii"): docs/REPLACEMENT_MODES.md, docs/PRIVACY.md, docs/NER_BACKENDS.md.

Speed and hardware

Default config (full multilingual NER ensemble), release benchmark documents (up to about 900 characters), SCT 0.8.0, AMD Threadripper PRO 7985WX and RTX 4090, median of 3 runs. 0.8.1 runs at the same speed.

The full ensemble needs about 12 to 16 GB of free memory on one GPU; English only needs 3 to 5 GB. As a rough estimate from this table, 1 million short documents take about 26 minutes English-only on one RTX 4090, about 2 hours with the full ensemble on 3 GPUs, and about 10 hours on 64 CPU cores.

NER is the slow stage. Turn it off (check_ner_process=False) for regex-only cleaning, or limit language= to load fewer models. Hardware advice and GPU setup: docs/PERFORMANCE.md.

Quality

Measured on open datasets with benchmarks/.

• Cleaning (FineWeb-2 with injected noise, default config): 100% of HTML, 99.6% of mojibake and 100% of URLs removed.
• Language detection (WiLI): 95.1% accuracy over 7,200 documents.
• PII (12 open PII datasets, about 40 languages): the 0.8.1 release branch leaked fewer values than 0.8.0 on every dataset, for example 18.1% to 8.2% on REDACT (comparison, pre-release commit 2078e8c4).
• Multilingual PII eval (220 records, 11 languages): 0.93 recall, 0.95 precision (docs/MULTILINGUAL_EVAL.md).

Test your own data or model with the evaluation harness (docs/EVALUATION.md).

Languages

Language detection covers 75 languages. Full NER models cover English, Dutch, German and Spanish. The other languages use the multilingual WikiNEural model with the English model, or a model you bring with ner_models. Full list: docs/LANGUAGES.md.

Limits and design principles

• Safe defaults, every choice configurable. Each behaviour is a config field; nothing is forced.
• No guarantee that all PII is found. Models miss entities, mostly organisations and non-Latin scripts. Review output before you publish it.
• Free text, not tables. For tables with known PII columns, drop or hash those columns first.
• Reversible maps are personal data. Anyone with the map can restore the values. Store maps apart from the cleaned data, with access control.
• Synthetic data is not anonymisation. Fake values keep the data useful; they do not make it anonymous in a legal sense.
• Offsets are post-regex. Finding offsets point into the text after the regex step, not into the raw input.

Detailed known limits of 0.8.2: docs/limits.md.

Documentation

New in 0.8.1: fewer leaks at the same speed, <IBAN> and <CARD> tags, and JSON and longer bracketed text survive cleanup. Some defaults changed; each has a setting to switch it back (CHANGELOG.md).

Contributing

Issues and pull requests are welcome. Read AGENTS.md first: model mistakes are fixed with model-driven rules, never with word lists, and every behaviour stays configurable.

Citation

@software{fazal_squeakycleantext,
  author = {Fazal, Rehan},
  title = {SqueakyCleanText},
  url = {https://github.com/rhnfzl/SqueakyCleanText},
  version = {0.8.1},
  year = {2026}
}

License and acknowledgements

MIT. See LICENSE. The package took inspiration from clean-text. It builds on ftfy, Lingua, ONNX Runtime and the NER models listed in docs/NER_BACKENDS.md.