🧼 LLM Sanitizer PRO

Your AI is only as clean as the data you feed it.
Point it at a messy dataset → it scrubs out personal info, leaked passwords, duplicates and junk → you get training data you're not embarrassed to use.

🤔 What is this?

So you want to train or fine-tune an AI on your own data: support chats, documents, scraped web pages, old forum posts. Great idea! Tiny problem: real-world data is filthy.

It's full of people's email addresses and phone numbers. Someone pasted an API key into a support ticket back in 2019. Half the rows are copy-pastes of the other half. And a good chunk of it is just… garbage.

Feed that to a model and it learns all of it, including your customers' phone numbers. 😬

LLM Sanitizer PRO is the washing machine for training data. One command, and out comes a dataset that is:

• 🕵️ Private. Emails, phone numbers, credit cards, bank accounts (IBAN), US social security numbers and IP addresses are swapped for safe placeholders.
• 🔑 Secret-free. Leaked API keys, tokens, passwords and private keys (AWS, GitHub, OpenAI, Stripe, Slack and many more) get scrubbed.
• 👯 Duplicate-free. Exact copies, near-copies, and even reworded copies get thrown out.
• 🗑️ Junk-free. Too-short, gibberish, ALL-CAPS-SCREAMING and low-quality text is filtered out.
• 🎓 Cheat-proof. Questions from famous AI tests (MMLU, GSM8K, HumanEval and friends) are removed, so your model can't just memorize the answers.
• 🍽️ Ready to serve. Exported in the formats training tools expect (ChatML, Alpaca), split into train/validation/test if you like.

No PhD required. If you can copy-paste one command, you can do this.

🖼️ How it works

Dirty data goes in the left, clean data comes out the right. Every record takes the same trip through the machine.

🔍 See it in action

Here's one record before and after. This is real output, not a mock-up.

Before:

Hi, I'm Jane. Email me at jane.doe@example.com or call +1 415-555-0132. My card is 4111 1111 1111 1111. The deploy key is ghp_A1b2C3d4E5f6G7h8I9j0K1l2M3n4O5p6Q7r8 so please keep it safe.

After:

Hi, I'm Jane. Email me at [PII_EMAIL] or call [PII_PHONE]. My card is [PII_CARD]. The deploy key is [SECRET_GITHUB_TOKEN] so please keep it safe.

And the identical copy of that record further down the file? Gone. Duplicates removed. ✨

Not a fan of placeholders? Pick your style:

💡 Did you notice "Jane" survived? Names, places and company names need a smarter, AI-based detector. Switch it on with --pii-ner (one extra install, explained here).

🚀 Get started in 2 minutes

1. Install it

pip install "llm-sanitizer-pro[all]"

You need Python 3.10 or newer. Don't want to install anything? Use the container instead (it already has everything inside). Put your data in the current folder and run (the --user part lets it save the result into your folder):

docker run --rm --user "$(id -u):$(id -g)" -v "$PWD:/data" ghcr.io/yog-sotho/llm-sanitizer-pro \
    --input my_data.jsonl --output clean.jsonl --profile fine-tune --report audit.html

2. Clean your data

sanitize --input my_data.jsonl --output clean.jsonl --profile fine-tune --report audit.html

That's it. clean.jsonl is your sparkling new dataset.

3. Check what happened

Open audit.html in your browser. It shows what got removed and why, what kind of personal info was hidden, and before/after samples. It's a single file, so you can email it to your boss or your privacy team. 📎

🎛️ Not sure which settings? Pick a profile

Tell it what the data is for, and it picks sensible settings for you:

Anything you add yourself always wins over the profile. Run sanitize --profile list to see exactly what each one switches on.

🧰 "I want to…" cheat sheet

Mix and match freely. More copy-paste examples live in the recipes.

📂 What can I feed it?

JSONL, JSON, CSV/TSV, plain text, Parquet and Excel, zipped (.gz) or not. Give it a single file, a whole folder, or a dataset straight from the Hugging Face Hub (--input hf://owner/dataset).

🛡️ Your data stays yours

Everything runs on your own computer. Your data is never uploaded anywhere. The only downloads are optional extras that you switch on yourself: AI models for name detection or quality scoring, and the public test sets used for --decontaminate.

🤓 For the nerds

The full technical story lives in the documentation:

• 📖 Features in detail: every feature, its flag, and its measured accuracy
• 🍳 Recipes: 26 copy-paste commands for common jobs
• ⌨️ CLI reference: every option and its default
• 🐍 Python API: use it as a library, plus how to develop on it
• ⚡ Performance: about 20,000 records per second with 4 jobs
• 🧾 Provenance: run manifests and Hugging Face dataset cards
• 🔏 Verifying releases: signatures and build provenance
• 📜 Changelog

License

MIT

Author

Yogsotho