LLM Dataset Sanitizer PRO

Production-grade, modular dataset sanitization, PII redaction, and curation pipeline for LLM training and fine-tuning.

Documentation: https://yog-sotho.github.io/LLM-Sanitizer-Pro/ (CLI reference, provenance, performance, verifying releases) · Changelog

🚀 Features

• Multi-Format Streaming: JSONL, JSON (ijson streaming), CSV/TSV, TXT, Parquet, Excel, and gzip variants — from files or stdin/stdout.

• Hugging Face Hub Input (--input hf://owner/dataset[/config[/split]]): sanitize a Hub dataset directly — shards are fetched via the Hub's parquet API (only pyarrow needed, no datasets library), cached locally, and streamed through the pipeline. Set HF_TOKEN for private/gated datasets.

• Advanced PII Redaction: Email, URL, phone (NANP + international), payment card (Luhn-checked), IBAN (mod-97-checked), SSN (validity-checked), and IPv4/IPv6 detection — validators reject order IDs, timestamps, version strings and loopback addresses — with three modes: token replacement, partial masking (--pii-mask), and stable pseudonymization (--pii-pseudonymize + exportable mapping).

• NER-Backed PII Detection (--pii-ner): names, places and organizations that regexes cannot catch, using spaCy, transformers or GLiNER2-PII (--pii-ner-backend gliner; Apache-2.0, 42 PII types, 7 languages; pip install "llm-sanitizer-pro[gliner]"). GLiNER also finds structured PII by context: address, date_of_birth, id_number (passport, licence, tax, national ID), financial, username and credential. Select them with --pii-ner-entities or all; --pii-ner-threshold sets the confidence cut. All three redaction modes apply: "Sarah Connor" becomes [PII_PERSON], S*** C***, or a stable Person_0001.

• Measured PII Accuracy (python -m sanitizer_pro.evaluation --backend regex+gliner): span-level precision, recall and F1 per kind on a labeled set (bundled, or yours as JSONL). The bundled set is synthetic: 600 records, 919 spans, 200 hard negatives such as order IDs, timestamps, versions, UUIDs and loopback IPs. On it, the regex layer scores precision and recall of 1.00 with zero false positives. Adding GLiNER2-PII gives person F1 0.94 and micro F1 0.96 (overlap matching). These are in-distribution numbers for a set this project generated, so evaluate on your own data before relying on them. CI fails if the regex layer's accuracy on this set regresses.

• Secrets Detection (--redact-secrets): finds and redacts credentials — AWS (access and secret keys), Google, Stripe, Slack, GitHub, GitLab, Hugging Face, npm, PyPI, Azure storage, OpenAI and Anthropic keys, private keys in any PEM/PGP armor (truncated ones too), JWTs, bearer tokens, DB connection strings, and entropy-checked api_key = "…" assignments (the variable name is kept) — with high-precision provider-shaped patterns. Works with or without --remove-pii, and honors masking/pseudonymization.

• Profiles (--profile fine-tune|pretrain|rag): one flag applies a curated bundle of defaults for a common job; your explicit flags and --config always win over the preset. --profile list shows what each sets.

• High-Performance Deduplication (three tiers):

  • Exact SHA-256 dedup (in-memory or disk-backed SQLite for huge datasets).
  • Fuzzy near-dedup (--fuzzy-dedup, --fuzzy-threshold): MinHash over word 3-shingles.
    • Signatures: computed by rensa (Rust, pip install "llm-sanitizer-pro[fuzzy]") or datasketch.
    • Matching: LSH finds candidates and each one is verified against its stored signature.
    • Measured accuracy: at thresholds 0.7–0.85, 97–98% of pairs at or above the threshold are caught, and no pair 0.15 or more below it.
    • Disk-backed index: with --dedup-backend sqlite the index lives on disk. Memory stayed flat at about 140 MB from 100k to 500k records, while the in-memory index reached 1.1 GB at 500k.
    • Resume: with --dedup-db-path the index survives --resume.
  • Semantic near-dedup (--semantic-dedup): static embeddings (model2vec, ~30MB, no torch) catch paraphrases that share no n-grams (--semantic-threshold).
    • Index: a usearch HNSW index (--semantic-index, part of the [semantic] extra) keeps lookups fast as the index grows.
    • Measured accuracy: on 20k real embeddings it matched exact search on 99.9% of decisions, at ~3k rec/s.
    • Fallback: without usearch, an LSH index is used. It slows down on large inputs.

• LLM-Native Formatting: Direct export to ChatML (--format-chatml) and Alpaca/Instruct (--format-instruct) schemas, with automatic key mapping (prompt/question/response/completion/…). ShareGPT conversations ({"from": "human", "value": …}) are converted to OpenAI messages.

• Chat Dataset Validation (--validate-chat): lint conversations before they reach a trainer — role alternation, empty turns, missing assistant replies, multiple/misplaced system messages, unknown roles, and per-conversation token budgets (--chat-max-tokens), with a per-reason rejection breakdown in the report and stats file. Understands OpenAI tool calling (function name and JSON arguments checked; tool results must answer an open tool_call_id, and every call must be answered), multimodal content parts ([{"type": "text"}, {"type": "image_url"}]), and ShareGPT records. With an HF --tokenizer that has a chat template, budgets count the rendered conversation (role markup and special tokens included) — what the trainer actually sees.

• Quality & Content Filtering: Length/word/uniqueness/ASCII gates, all-caps rejection, code detection, profanity filtering, and pluggable Python quality scripts.

• Language Filtering (--lang-filter en,zh): GlotLID v3 (fastText, 2,000+ language varieties, the identifier used by FineWeb-2) by default when installed (pip install "llm-sanitizer-pro[lang]"; the 1.7 GB model downloads once), OpenLID-v2 (--lang-backend openlid, GPL-3.0, downloaded on demand), or langdetect. Filters accept ISO 639-1 or 639-3 codes — zh matches cmn_Hani/yue_Hani, ar matches Arabic dialects — with --lang-confidence gating. Each --jobs worker loads its own model copy (~1.7 GB RAM for GlotLID).

• Quality Scoring (--quality-min-score, --keep-top-percent, --quality-score-field): every record gets a [0, 1] quality score from one of five backends:

  • heuristic (default): no dependencies; C4/Gopher-style prose signals with a repetition penalty.
  • fineweb-edu: the FineWeb-Edu classifier, the model behind FineWeb-Edu. Its 0–5 score is mapped to 0–1, so the authors' recommended cut of 3 is about 0.5.
  • dclm: the DCLM-Baseline fastText filter; the score is P(high quality).
  • fasttext: any fastText classifier you supply, via --quality-model PATH and --quality-label.
  • perplexity: causal-LM perplexity.

  Filter by an absolute bar, keep only the best P%, or just annotate records. With --jobs N, every worker runs the scorer. Model scorers: pip install "llm-sanitizer-pro[quality]".

• Pretraining Rule Filters (--quality-rules gopher,gopher-repetition,c4,fineweb or all): the document heuristics of Gopher, C4 and FineWeb, with datatrove's thresholds and reason names. They need no dependencies and run on the full document. Accept/reject decisions agree with datatrove on 98–100% of documents, measured on 3,000 C4 documents including deliberately perturbed ones; the small remainder comes from tokenization at the threshold boundaries. Failures are reported per rule in the stats, the summary and the report. The rules are tuned for English, so run them after --lang-filter.

• Benchmark Decontamination (--decontaminate): removes records whose word 8-grams overlap an eval set. Matching covers question and solution/choice text.

  • Benchmarks: MMLU, MMLU-Pro, GPQA, GSM8K, GSM-Plus, MATH, MATH-500, AIME 2024/2025, IFEval, BBH, MuSR, HLE, SimpleQA, HumanEval(+), MBPP(+), ARC, HellaSwag, TruthfulQA and WinoGrande.
  • Groups: open-llm-v2 is the Open LLM Leaderboard v2 set. all means every benchmark that isn't gated.
  • Downloads: benchmarks come from the Hugging Face Hub and are cached. GPQA and HLE are gated: accept their terms on the Hub and set HF_TOKEN. You can also add your own reference files with --decontam-refs.
  • Reporting: removed records are counted per benchmark in the stats, the summary and the report. When benchmarks overlap (MATH-500 is a subset of MATH), the first one listed gets the credit.
  • Size: indexing all 20 ungated benchmarks gives 4.1M n-grams, about 300 MB of RAM.

• Dataset Splitting & Sharding: --split train=0.9,val=0.05,test=0.05 or fixed-size shards with --shard-size. Splits and --sample are decided by a hash of each record's content (salted by --seed), so a record always lands in the same split — across runs and filter changes — and exact duplicates never straddle train and test. (--split and --shard-size can't be combined with --resume.)

• Crash-Safe I/O: Atomic JSON writes (.tmp + os.replace()), safe HTML stripping via html.parser, structure-preserving text normalization (newlines kept for code/markdown data).

• Resumable Runs (--resume): progress is checkpointed to <output>.checkpoint.json every --checkpoint-interval records; after a crash or Ctrl-C, rerun the same command and the pipeline skips already-processed input, restores statistics, pseudonym and sampling state, and appends to the output. Rows written after the last checkpoint by a hard crash (OOM-kill, SIGKILL) are truncated away and re-processed, so the result matches an uninterrupted run. Pair with --dedup-backend sqlite --dedup-db-path for dedup state that also survives the restart (rolled back to the checkpoint on resume).

• Parallel Processing (--jobs N): output, statistics and report samples are identical to a single-process run; pseudonyms too when a pseudonym key is set (see below).

  • JSONL input: workers read and parse byte-range chunks of the files themselves.
  • Measured speed: 100k records with PII + secrets redaction and dedup ran at 20.6k rec/s with 4 jobs, against 6.1k rec/s for one process.
  • Pseudonyms: with --pii-pseudonymize, set SANITIZE_PSEUDO_KEY (or --pseudo-key) for keyed pseudonyms. The same value gets the same pseudonym in every worker and every run that uses the key.

• Many Input Files: --input takes a directory (searched recursively) or a glob ('data/**/*.jsonl'). Files are read in sorted order, may mix formats, and work with --resume.

• Provenance: --manifest run.json records exactly how a dataset was produced: versions, models, full config and its hash, SHA-256 of every input and output, and per-stage counts. --dataset-card README.md writes a Hugging Face dataset card from it, and sanitize diff a.json b.json compares two runs (--fail-on-change for CI).

• Audit Report (--report audit.html): a self-contained HTML artifact per run — removal funnel, PII redaction counts by type, quality-score distribution, chat-failure breakdown, and before/after redaction samples. Light/dark aware, no external assets; archive it next to the dataset or attach it to a compliance ticket. Also available from the Python API via s.write_report(path).

📦 Installation

pip install llm-sanitizer-pro          # core (stdlib + tqdm)
pip install "llm-sanitizer-pro[all]"   # formats, fuzzy dedup, langdetect, transformers; NER, GLiNER,
                                       # semantic, GlotLID and model scorers have their own extras

🔨 Usage

# Basic PII redaction + exact dedup
sanitize --input data.jsonl --output clean.jsonl --remove-pii --deduplicate

# Fuzzy dedup + ChatML formatting
sanitize --input data.jsonl --output chatml.jsonl --fuzzy-dedup --fuzzy-threshold 0.7 --format-chatml

# Parallel processing + SQLite dedup backend for datasets bigger than RAM
sanitize --input huge.jsonl --output clean.jsonl --jobs 8 --deduplicate --dedup-backend sqlite

# A whole directory of shards, near-dedup on disk, 8 workers
sanitize --input 'shards/**/*.jsonl' --output clean.jsonl --jobs 8 \
    --fuzzy-dedup --dedup-backend sqlite --dedup-db-path minhash.db

# Train/val/test split with reproducible sampling
sanitize --input data.jsonl --output out.jsonl --split train=0.9,val=0.05,test=0.05 --seed 42

# Preview effects without writing output
sanitize --input data.jsonl --output out.jsonl --dry-run --stats-file report.json

# Remove records that overlap with benchmark test sets (auto-downloads + caches them)
sanitize --input data.jsonl --output clean.jsonl --decontaminate gsm8k,mmlu,humaneval

# Decontaminate against your own held-out eval set
sanitize --input data.jsonl --output clean.jsonl --decontam-refs my_eval_set.jsonl

# List available built-in benchmarks
sanitize --decontaminate list

# NER-backed PII: redact person names too (see install note below)
sanitize --input data.jsonl --output clean.jsonl --remove-pii --pii-ner

# Pseudonymize everything consistently (names, emails, …) and keep the mapping.
# With a key, pseudonyms are the same across runs and --jobs workers.
SANITIZE_PSEUDO_KEY=… sanitize --input data.jsonl --output clean.jsonl --remove-pii --pii-ner \
    --pii-pseudonymize --pseudo-map-file mapping.json --jobs 4

# Convert instruction data to ChatML, then reject structurally invalid conversations
sanitize --input data.jsonl --output chat.jsonl --format-chatml --validate-chat

# Enforce a context-window budget per conversation, counted through the
# model's chat template
sanitize --input chat.jsonl --output fit.jsonl --validate-chat --chat-max-tokens 4096 \
    --tokenizer Qwen/Qwen3-8B

# Tool-calling data (OpenAI tool_calls / tool results, or ShareGPT
# function_call / observation turns)
sanitize --input tools.jsonl --output clean.jsonl --validate-chat \
    --chat-roles system,user,assistant,tool

# Multi-agent traces: keep structural and tool-call checks, relax ordering rules
sanitize --input traces.jsonl --output clean.jsonl --validate-chat --chat-lenient \
    --chat-roles system,user,assistant,tool

# Quality scoring: drop junk below an absolute bar, or keep only the best 30%
sanitize --input data.jsonl --output clean.jsonl --quality-min-score 0.5
sanitize --input data.jsonl --output best.jsonl --keep-top-percent 30

# Annotate records with their score instead of filtering (sort downstream)
sanitize --input data.jsonl --output scored.jsonl --quality-score-field quality

# Perplexity-based scoring with a causal LM (pip install transformers torch)
sanitize --input data.jsonl --output clean.jsonl --quality-scorer perplexity \
    --quality-model distilgpt2 --quality-min-score 0.4

# Full pipeline with an HTML audit report artifact
sanitize --input data.jsonl --output clean.jsonl --remove-pii --deduplicate \
    --quality-min-score 0.5 --report audit.html

# Sanitize a Hugging Face Hub dataset directly (cached under ~/.cache)
sanitize --input hf://openai/gsm8k/main/train --output clean.jsonl \
    --remove-pii --deduplicate --report audit.html

# Long job that survives crashes: checkpoint + durable dedup, rerun to continue
sanitize --input huge.jsonl --output clean.jsonl --resume \
    --deduplicate --dedup-backend sqlite --dedup-db-path dedup.db

# Semantic dedup: drop paraphrased near-duplicates (pip install model2vec)
sanitize --input data.jsonl --output clean.jsonl --semantic-dedup --semantic-threshold 0.85

# One-flag preset for SFT data (PII + secrets + dedup + cleanup), report attached
sanitize --input data.jsonl --output clean.jsonl --profile fine-tune --report audit.html

# Redact leaked credentials from scraped code/logs
sanitize --input data.jsonl --output clean.jsonl --redact-secrets

# Structured JSON logs for a service pipeline
sanitize --input data.jsonl --output clean.jsonl --log-format json

NER-backed PII install

--pii-ner auto-selects an installed backend (spaCy preferred, transformers fallback):

pip install "llm-sanitizer-pro[ner]"
pip install "en_core_web_sm @ https://huggingface.co/spacy/en_core_web_sm/resolve/main/en_core_web_sm-any-py3-none-any.whl"

Use --pii-ner-entities person,location,org to widen coverage (default: person) and --pii-ner-model to swap in another spaCy model or HF token-classification model.

How decontamination works

A record is flagged as contaminated when at least --decontam-min-hits (default 1) of its normalized word 8-grams (--decontam-ngram) also appear in the reference index built from benchmark test sets — the same n-gram collision approach used to decontaminate GPT-3 and Llama training data. Benchmark items shorter than the n-gram size are matched whole. Named benchmarks require pyarrow; --decontam-refs files work with any supported input format and no extra dependencies.

Run sanitize --help for the full option reference, or sanitize --generate-config yaml to print a config template usable with --config.

🐍 Python API

The CLI is a thin layer over a first-class library — services should embed it directly instead of shelling out:

from sanitizer_pro import Sanitizer, SanitizerConfig

config = SanitizerConfig(
    remove_pii=True, deduplicate=True,
    quality_min_score=0.5, quality_score_field="_q",
    decontaminate=["gsm8k", "mmlu"], validate_chat=False,
)

with Sanitizer(config) as s:
    clean = list(s.process(records))     # iterable of dicts in → clean dicts out
    report = s.stats.to_dict()           # same schema as --stats-file

    # Or inspect records one at a time (audit-UI building block):
    result = s.process_record({"text": "..."})
    result.kept        # bool
    result.reason      # 'quality' | 'duplicate' | 'contaminated' | 'chat:<detail>' | ...
    result.score       # quality score in [0, 1] when scoring is enabled

SanitizerConfig is the single configuration object: the CLI flags have the same names, take their defaults from it, and a CLI run builds one and drives the very same Sanitizer engine — so library and command-line results are identical. Dedup and pseudonym state persist across calls on one Sanitizer instance. For streaming, s.feed(record) returns the records to write now and s.finish() releases the keep_top_percent buffer at the end (process() combines both). Export pseudonym mappings with s.export_pseudonym_map(path).

🧪 Development

pip install -e .[dev]
pytest            # run the test suite
ruff check sanitizer_pro tests benchmarks scripts
mypy              # strict
pip install -e .[docs] && mkdocs serve           # docs site at localhost:8000
python scripts/gen_cli_reference.py              # after changing CLI options

# Throughput (rec/s, MB/s, peak RSS per scenario, --jobs scaling); nightly in CI
python -m benchmarks.bench --records 100000
python -m benchmarks.fuzzy_recall --threshold 0.85     # fuzzy dedup vs exact Jaccard
python -m benchmarks.semantic_recall --size 100000     # semantic index vs exact search

License

MIT

Author

Yogsotho