cleanmonkey

One-call text cleanup for invisible characters, smart quotes, and whitespace normalization.

Install

pip install cleanmonkey

Quick Start

from cleanmonkey import clean

# Sensible defaults handle the common garbage
clean("hello\u00a0world\u2019s \u2014 test")
# → "hello world's - test"

# Idempotent — safe to call twice
clean(clean(text)) == clean(text)

What It Cleans (by default)

Granular Control

Override any default:

clean(text, smart_quotes=False)       # keep curly quotes
clean(text, dashes=False)             # keep em/en dashes
clean(text, fullwidth=True)           # also normalize fullwidth digits/letters
clean(text, collapse_spaces=False)    # keep multiple spaces
clean(text, strip=False)              # keep leading/trailing whitespace

Profiles

clean(text, profile="default")     # all normalizations (the default)
clean(text, profile="csv")         # default + fullwidth normalization
clean(text, profile="sql")         # default + fullwidth normalization
clean(text, profile="display")     # keep smart quotes & dashes; still clean invisible, control, whitespace, line endings
clean(text, profile="minimal")     # invisible chars only, no collapsing or stripping
clean(text, profile="aggressive")  # everything including fullwidth

Batch Helpers

from cleanmonkey import clean_column, clean_dict

# Clean a list (non-strings pass through)
clean_column(["hello\u00a0world", 42, None])
# → ["hello world", 42, None]

# Recursively clean dict values
clean_dict({"name": "John\u00a0Doe", "nested": {"val": "test\u200b"}})
# → {"name": "John Doe", "nested": {"val": "test"}}

# Also clean keys
clean_dict({"key\u00a0name": "val"}, keys=True)
# → {"key name": "val"}

Inspect

Find out what's lurking in your text:

from cleanmonkey import inspect

for info in inspect("hello\u00a0world\u200b"):
    print(f"{info.codepoint} {info.name} count={info.count} at {info.positions}")
# U+00A0 NO-BREAK SPACE count=1 at [5]
# U+200B ZERO WIDTH SPACE count=1 at [11]

CLI

# Clean a file
cleanmonkey input.txt -o output.txt

# Pipe through stdin
cat dirty.csv | cleanmonkey > clean.csv

# Use a profile
cleanmonkey --profile csv input.txt

# Inspect mode — report what's in a file
cleanmonkey --inspect input.txt

# Machine-readable JSON inspect output
cleanmonkey --json input.txt

# Selective overrides
cleanmonkey --no-smart-quotes --fullwidth input.txt

# Preserve whitespace structure
cleanmonkey --no-strip --no-collapse-spaces input.txt

# Preserve line endings (CR/CRLF)
cleanmonkey --no-line-endings input.txt

Built for LLMs

cleanmonkey is designed to work well as a tool for large language models. Invisible character cleanup is a constant source of silent bugs in LLM-driven data pipelines — non-breaking spaces break splits, zero-width characters corrupt comparisons, and smart quotes fail exact matches. Without cleanmonkey, LLMs end up generating repetitive .replace() chains that miss edge cases and waste tokens. A single clean() call handles all of it with a structured, idempotent result — no multi-step prompting or character-by-character debugging required. Fewer tokens in, clean data out.

A note on whitespace

By default clean() strips each line and collapses runs of spaces, which is correct for prose but destroys indentation in structured text (YAML, Python, Markdown). Use clean(text, strip=False, collapse_spaces=False) or profile="minimal" for those. See LIMITATIONS.md for this and other deliberate design tradeoffs.

Quality & review

cleanmonkey is hardened with a competitive multi-model review methodology and a measurable release gate:

• CONTRIBUTING.md — testing philosophy and the review-panel process.
• LIMITATIONS.md — intentional tradeoffs reviewers should not re-litigate.
• RELEASE_READINESS.md + release_readiness.json + scripts/readiness.py — the release rubric and convergence metric (python scripts/readiness.py).
• REVIEW_HISTORY.md — how the library was hardened.

License

MIT