scrubkit

Clean scraped data safely. Fixes broken encoding, leftover HTML and invisible characters — and never destroys a legitimate value.

You scraped 40,000 rows. Some titles say Café. Some still have <b> tags in them. Some rows say "Please wait, verifying you are human". And some cells contain a zero-width character that quietly breaks every join you run.

scrubkit fixes all of that in one call, tells you what it refused to touch and why, and gives you a reversible record of every change it made.

pip install scrubkit

from scrubkit import clean

clean("Café <b>Noir</b>&nbsp;").data
# 'Café Noir'

No dependencies. No LLM. No network. Same input always gives the same bytes out.

---

Clean a scrape in one line

Pass whatever your scraper gave you — a list of rows, a single object, a JSON string, a CSV string, or a plain string. You get the same shape back, cleaned.

from scrubkit import clean

rows = [
    {"title": "Café <b>Noir</b>&nbsp;", "price": "US $59.99", "stock": "N/A"},
    {"title": "Thé Vert​",         "price": "US $9.00",  "stock": "12"},
]

result = clean(rows)

result.data
# [{'title': 'Café Noir', 'price': 'US $59.99', 'stock': 'N/A'},
#  {'title': 'Thé Vert',  'price': 'US $9.00',  'stock': '12'}]

Nested lists and dicts are walked to any depth, so scraped specs and variants arrays are cleaned too.

What it fixes

Input	Output
Café	Café	mojibake (UTF-8 read as cp1252)
Café	Café	double- and triple-encoded mojibake
<p>Warm light.</p><p>Fits any room.</p>	Warm light. Fits any room.	HTML tags, with the block break kept
Solid oak &mdash; seats&nbsp;two.	Solid oak — seats two.	HTML entities, decoded once
Tokyo​ Ltd­	Tokyo Ltd	zero-width space, BOM, soft hyphen
Bar Stool	Bar Stool	non-breaking and exotic spaces
" Lumina "	"Lumina"	leading and trailing whitespace

What it refuses to do

This is the part that matters, and it is why scrubkit exists.

Input	Left exactly as it was, because
"None"	it is a real surname
"NA"	it is Namibia's ISO country code
"captcha"	it is a real word in real product data
"Please wait, verifying you are human"	your scrape failed — deleting the row would hide that
می‌تواند (Persian ZWNJ)	the zero-width non-joiner is a required letter-shaping character
👨‍👩‍👧 (emoji ZWJ)	removing the joiner splits one family into three people
AT&T, 5 < 7	a real HTML entity ends in ; — these are not markup
&amp;amp;	decoding once gives &amp;, which is ambiguous — so it is reported, not guessed
3 m², ½ cm	Unicode NFKC would rewrite these to 3 m2 and 1/2
<div class='cta'>…</div> in a description_html field	the field's name says it holds markup on purpose

A cleaner that tells you "row 42 says 'Access denied', your scrape failed" is worth more than one that silently deletes row 42 and reports a 100% clean dataset.

The boundary: AUTO / opt-in / flag

Every rule sits in exactly one of three tiers, and the tier is part of the API, not a promise in a README.

Tier	Meaning	Rules
AUTO	The removed bytes carry no meaning the surviving bytes do not already carry. Applied silently, always recorded.	mojibake repair, HTML tag stripping, entity decoding, invisible-character removal, whitespace normalising, trimming
OPT-IN	Correct and useful, but it changes something you may depend on — your row count, a value's type, your schema. Never applied unless you ask by name.	nulling placeholders, dropping exact duplicates, coercing numeric text, repairing keys
FLAG	There is a realistic dataset where applying this destroys real data, and no deterministic rule can tell it apart from the one where the fix is right. Reported with a concrete proposal. No option turns these on.	failed-scrape detection, near-duplicates, ambiguous placeholders, NFKC normalisation, HTML-bearing fields, impossible values

With no options at all, scrubkit cannot change your row count, any value's type, or your schema. Read the full ruleset — every rule with the argument for its placement — straight from the package:

from scrubkit import RULES

for rule in RULES:
    print(rule.action, rule.rule_id, "—", rule.what)

Reading the flags

result = clean(rows)

for flag in result.flags:
    print(f"{flag['path']}  [{flag['rule']}]")
    print(f"   {flag['reason']}")

[0].desc  [boilerplate.flag]
   value contains site furniture ('please wait'), which means the extractor
   missed the real content. Nothing here can be repaired -- the content was
   never captured. Re-scrape this record.

[0].stock  [placeholder.null_high_confidence]
   'N/A' is an unambiguous masked-missing marker that every other check counts
   as a present, valid value. Nulling it would change this field's type, so it
   is proposed rather than applied.
   enable_with: placeholder_policy='null_high_confidence'

Every flag names the option that would act on it, so you can decide once and switch it on.

Opt-ins

result = clean(
    rows,
    coerce_numeric_text=True,                    # "$12.50" -> 12.5
    drop_exact_duplicates=True,                  # remove byte-identical rows
    placeholder_policy="null_high_confidence",   # "N/A" -> None, "None" stays
    repair_keys=True,                            # clean the dict keys too
)

Option	Default	What it changes
placeholder_policy	"flag"	"null_high_confidence" nulls only unambiguous placeholders; "null_all" also nulls "None", "NA", "-" — correct only if you know your data has no such legitimate values
drop_exact_duplicates	False	Removes rows byte-identical to an earlier row after cleaning. Off by default: dropping rows changes every count and sum downstream
coerce_numeric_text	False	Turns numeric strings into numbers, all-or-nothing per field
repair_keys	False	Cleans dict keys as well as values (changes your schema)
trim_whitespace	True	Trims the edges only — interior line breaks are kept
detect_duplicates	True	Scans for duplicate and near-duplicate rows

Coercion refuses rather than guesses:

clean([{"price": "12,50"}, {"price": "9,00"}, {"price": "7,25"}],
      coerce_numeric_text=True).data
# unchanged — a lone ',' is a decimal separator in fr/de/es and a malformed
# thousands group in en. The refusal, with that reason, is in .flags

Is it actually deterministic?

Don't take our word for it. The test suite ships inside the installed package:

pip install "scrubkit[test]"
python -m pytest --pyargs scrubkit

It asserts, on realistic scraper fixtures:

• byte-identical output — ten runs of the same input produce the same bytes, same output_sha256, same replay_id
• idempotence — cleaning the output again changes nothing
• the control set is untouched — ten rows of legitimate data that a careless cleaner would damage come back byte-for-byte identical, with zero changes
• reversibility — the change record alone rebuilds your exact input
• no new defects — a repair that would introduce a defect is reverted
• the boundary matches the code — every rule in RULES is implemented, no FLAG rule can be switched on by any combination of options

In your own code, the whole claim is two lines:

assert clean(data).output_sha256 == clean(data).output_sha256   # deterministic
assert clean(clean(data).data).data == clean(data).data         # idempotent

Undo anything

result = clean(rows)

result.changes[0]
# {'path': '[0].title', 'path_steps': [0, 'title'],
#  'rule': 'mojibake.roundtrip', 'action': 'AUTO',
#  'before': 'Café <b>Noir</b>&nbsp;', 'after': 'Café <b>Noir</b>&nbsp;',
#  'detail': {'iterations': 1, 'residual_mojibake': False}}

result.revert() == rows     # True — reconstructed from the record alone

The change list is a complete inverse patch, not a log. Store it as JSON and undo later with scrubkit.revert(data, changes).

scrubkit vs ftfy

ftfy is excellent, and if your only problem is broken Unicode, use ftfy — it is the reference implementation for mojibake repair and it goes deeper on that one job than scrubkit does.

scrubkit solves the next problem up. Scraped data is not only mis-encoded: it has HTML left in it, invisible characters, error pages that look like content, placeholder strings you must not confuse with real values, and rows that are duplicates of each other. scrubkit handles that whole pipeline, tells you what it refused to touch and why, and hands you a reversible record.

	ftfy	scrubkit
Mojibake / broken encoding	✅ deeper	✅
HTML entities	✅	✅ decoded once, refused when ambiguous
Control & invisible characters	✅	✅ context-gated for Persian, Arabic, Indic, emoji
Unicode normalisation, curly quotes, ligatures	✅ by default	reported, never applied automatically
Stripping HTML tags	—	✅
Detecting a failed scrape (captcha / cookie wall / error page)	—	✅ flagged
Placeholder handling that knows None is a surname	—	✅
Duplicate and near-duplicate rows	—	✅
Structured rows, nested objects, CSV	—	✅
Report of what was refused, and why	—	✅
Reversible change record	—	✅
Dependencies	1	0

They compose. Nothing stops you running ftfy for the Unicode layer and scrubkit for the pipeline around it.

Also works on

Not just scraping — anything where text arrived from somewhere you don't control:

• Apify / Scrapy / Playwright output — clean(dataset_items)
• CSV exports from legacy systems — clean(open("export.csv").read()) returns parsed, cleaned rows
• Before loading into a database — invisible characters are what make two "identical" keys fail to join
• Before sending text to an LLM — mojibake and markup burn tokens and confuse extraction

API

clean(
    data,                              # rows, object, JSON text, CSV text, or str
    *,
    placeholder_policy="flag",         # "flag" | "null_high_confidence" | "null_all"
    drop_exact_duplicates=False,
    coerce_numeric_text=False,
    repair_keys=False,
    trim_whitespace=True,
    detect_duplicates=True,
) -> CleanResult

CleanResult carries .data, .flags, .changes, .summary, .duplicates, .replay_id, .input_sha256, .output_sha256, .dropped_rows, plus .revert() and .to_dict().

Bad options raise ValueError — that is a typo in your code. Bad data never raises: a malformed row comes back with a warning in .summary["input"]["warnings"], because you wanted your data cleaned, not an exception in the middle of a scraping run.

Optional extra

Near-duplicate detection needs pip install "scrubkit[duplicates]". Without it, exact duplicates still work and the near-duplicate scan degrades to a stated reason in .duplicates — nothing else changes.

clean(rows).duplicates["near"]
# {'skipped': True, 'reason': "near-duplicate matching unavailable in this
#  environment (ModuleNotFoundError: No module named 'rapidfuzz'); exact-
#  duplicate detection above is unaffected"}

A scan that quietly found nothing is indistinguishable from a clean dataset, so it says so instead.

Known limitation

Combining placeholder_policy="null_*" with coerce_numeric_text=True is the one case where cleaning twice differs from cleaning once: nulling placeholders removes the non-numeric values that were correctly blocking coercion, so a field refused on the first pass can qualify on the second. It settles immediately, it cannot happen with the default options, and it is asserted in the test suite so it cannot get worse silently. Clean once — which is what a pipeline does anyway.

Hosted API

scrubkit is the library. The same engine runs as a hosted service at aidatatools.dev — no install, callable from any language, plus dataset-level quality scoring that the library does not include. There is an agent-payable endpoint (x402) and a subscription for teams. The library is free and stays free.

License

Apache-2.0. See LICENSE.