dsvmonkey

Detect, profile, normalize and repair delimiter-separated-values files.

CSV is a polite lie. Real files are tab-separated, pipe-separated, or semicolon-separated; start with decorative title rows; carry BOMs and mixed encodings; include ragged rows and quoted newlines. dsvmonkey reads them anyway, tells you what it found, and hands you a clean stream of rows.

Status

Alpha. API is not yet stable.

Install

pip install dsvmonkey

For development (editable install with test tooling):

pip install -e .[dev]
# or equivalently:
pip install -r requirements-dev.txt

Both requirements.txt and requirements-dev.txt are thin pointers to pyproject.toml — the single source of truth for dependency lists. Edit dependencies in pyproject.toml; the requirements files need no maintenance.

What it does

• Detect encoding, delimiter, quote char, header row and line endings — each with a confidence score, runner-up alternatives and the reasoning behind the choice.
• Normalize cells on read using cleanmonkey (BOMs, NBSPs, zero-width spaces, smart quotes, stray control chars).
• Profile date columns via datemonkey.
• Repair ragged rows, stray BOMs and inconsistent line endings.
• Stream row-by-row; large files are fine.
• Chain cleanly into pgmonkey (DB import), xlfilldown (Excel output) and typemonkey (type inference).

CLI

dsvmonkey inspect   file.csv                       # human-readable detection report
dsvmonkey normalize file.csv -o clean.csv          # strip BOM, fix ragged rows, normalize endings
dsvmonkey convert   file.csv -o out.jsonl --to jsonl

Run dsvmonkey --help or dsvmonkey <command> --help for the full list. Flags are command-specific:

• inspect: -v/--verbose, --no-columns, --sample-rows, --excel-serial-min, --no-deep-scan, --clean-sample, --strict (exit 3 instead of 0 when the profile recommends human review — the unattended-pipeline gate).
• normalize: --encoding, --line-ending lf|crlf|cr, --delimiter, --field-count, --no-clean, --no-deep-scan, --keep-empty-rows, --sanitize-formulas, --strict (same gate semantics as inspect --strict: profile first, exit 3 with no output written when detection isn't confident enough).
• convert: --to {csv,tsv,jsonl}, --no-clean, --no-deep-scan, --keep-empty-rows, --sanitize-formulas (applies on every output format, including jsonl — JSONL output is commonly transformed back to CSV/Excel later, where formula payloads surviving as JSON string values become live formulas), --strict (gate as above).

Python API

import dsvmonkey

# Profile a file — encoding, delimiter, headers, etc.
profile = dsvmonkey.profile_file("file.csv")

# Stream cleaned rows as dicts
for row in dsvmonkey.read("file.csv"):
    ...

# Write a cleaned version
report = dsvmonkey.repair("messy.csv", "clean.csv")

# Convert to JSON Lines
dsvmonkey.to_jsonl("file.csv", "file.jsonl")

# Per-column profiling (date-format detection via datemonkey)
columns = dsvmonkey.profile_columns("file.csv")

Limitations

Some behaviours are deliberate design tradeoffs rather than bugs (e.g. mixed-encoding detection requires UTF-8 multi-byte evidence to avoid false-positives on cp1252 files; duplicate header names in dict mode warn-and-collapse rather than raise). See LIMITATIONS.md for the full list with rationale and escape hatches.

Using with AI assistants

SKILL.md at the repo root is a drop-in Claude Code / agent skill that teaches LLMs how to call dsvmonkey correctly — decision tree, failure modes it already handles, worked examples, and a "don't" list so agents stop reinventing broken CSV parsing. Copy it to ~/.claude/skills/ or include it in a project's AGENTS.md / CLAUDE.md for automatic discovery.

Contributing & review methodology

CONTRIBUTING.md documents the testing philosophy and the competitive multi-model review panel used to harden this package. The release decision is made against an explicit rubric (RELEASE_READINESS.md, scored by scripts/readiness.py); the panel-by-panel record lives in REVIEW_HISTORY.md and deliberate tradeoffs in LIMITATIONS.md.

License

MIT. See LICENSE.