csvnorm
A command-line utility to validate and normalize CSV files for initial exploration.
Version 1.0 Breaking Change
If upgrading from v0.x: The default output has changed from file to stdout for better Unix composability.
# v0.x behavior
csvnorm data.csv # Created data.csv in current directory
# v1.0 behavior (NEW)
csvnorm data.csv # Outputs to stdout
csvnorm data.csv -o clean.csv # Explicitly save to file
csvnorm data.csv > clean.csv # Or use shell redirect
Never redirect to the input file (csvnorm data.csv > data.csv): the shell empties data.csv before csvnorm starts, and the data is lost. csvnorm refuses to overwrite its input only when the output path is given with -o.
This follows the Unix philosophy and matches tools like jq, csvkit, and xsv.
Installation
Recommended (uv):
uv tool install csvnorm
Or with pip:
pip install csvnorm
Purpose
This tool prepares CSV files for basic exploratory data analysis (EDA), not for complex transformations. It focuses on achieving a clean, standardized baseline format that allows you to quickly assess data quality and structure before designing more sophisticated ETL pipelines.
What it does:
- Validates CSV structure and reports errors
- Normalizes encoding to UTF-8 when needed
- Normalizes delimiters and field names
- Creates a consistent starting point for data exploration
What it doesn't do:
- Complex data transformations or business logic
- Type inference or data validation beyond structure
- Heavy processing or aggregations
Features
- CSV Validation: Checks for common CSV errors and inconsistencies using DuckDB
- Delimiter Normalization: Converts all field separators to standard commas (
,) - Field Name Normalization: Converts column headers to snake_case format
- Encoding Normalization: Auto-detects encoding and converts to UTF-8 when needed (ASCII is already UTF-8 compatible)
- Processing Summary: Displays comprehensive statistics (rows, columns, file sizes) and error details
- Error Reporting: Exports detailed error file for invalid rows with summary panel
- Remote URL Support: Process CSV files directly from HTTP/HTTPS URLs without downloading (unless
--fix-mojibakeis used)
Usage
csvnorm input.csv [options]
csvnorm - # read from stdin
By default, csvnorm writes to stdout for easy piping and composability with other Unix tools. Use - as input to read from stdin.
Options
| Option | Description |
|---|---|
-o, --output-file PATH |
Write to file instead of stdout |
-f, --force |
Force overwrite of existing output file (when -o is specified) |
-k, --keep-names |
Keep original column names (disable snake_case) |
-d, --delimiter CHAR |
Set custom output delimiter (default: ,) |
-s, --skip-rows N |
Skip first N rows of input file (useful for metadata/comments) |
--fix-mojibake [N] |
Fix mojibake using ftfy (optional sample size N; use 0 to force repair) |
--strict |
Write no output (stdout or file) if any validation errors occur; an existing output file is left untouched, even with --force. Exit code is 1 on validation errors with or without --strict |
--check |
Validate CSV without processing or normalizing (exit code 0=valid, 1=invalid) |
--download-remote |
Download remote CSV locally before processing (needed for remote .zip/.gz) |
-V, --verbose |
Enable verbose output for debugging |
-v, --version |
Show version number |
-h, --help |
Show help message |
Examples
# Default: output to stdout
csvnorm data.csv
# Read from stdin
cat data.csv | csvnorm -
curl -s https://example.com/data.csv | csvnorm - -o clean.csv
csvnorm - --check < data.csv
# Preview first rows
csvnorm data.csv | head -20
# Pipe to other tools
csvnorm data.csv | csvcut -c name,age | csvstat
# Save to file
csvnorm data.csv -o output.csv
# Shell redirect
csvnorm data.csv > output.csv
# Process remote CSV from URL
csvnorm "https://raw.githubusercontent.com/aborruso/csvnorm/refs/heads/main/test/Trasporto%20Pubblico%20Locale%20Settore%20Pubblico%20Allargato%20-%20Indicatore%202000-2020%20Trasferimenti%20Correnti%20su%20Entrate%20Correnti.csv" -o output.csv
# Process remote compressed CSV (download first, then handle gzip/zip locally)
csvnorm "https://example.com/data.csv.gz" --download-remote -o output.csv
# Custom delimiter
csvnorm data.csv -d ';' -o output.csv
# Keep original headers
csvnorm data.csv --keep-names -o output.csv
# Skip first 2 rows (metadata or comments)
csvnorm data.csv --skip-rows 2 -o output.csv
# Force overwrite with verbose output
csvnorm data.csv -f -V -o processed.csv
# Fix mojibake using ftfy (default sample size)
csvnorm data.csv --fix-mojibake -o fixed.csv
# Fix mojibake with custom sample size
csvnorm data.csv --fix-mojibake 4000 -o fixed.csv
# Force mojibake repair even with low badness score
csvnorm data.csv --fix-mojibake 0 -o fixed.csv
# Fail-fast mode: exit with error if validation errors occur
csvnorm data.csv --strict > output.csv || echo "Validation failed!"
# Use in pipelines where data quality is critical
csvnorm remote_data.csv --strict | other_tool || handle_error
# Quick validation check (no processing or output)
csvnorm data.csv --check && echo "Valid CSV" || echo "Invalid CSV"
# Check remote CSV for validity
csvnorm https://example.com/data.csv --check
# Use in CI/CD pipelines for validation
csvnorm raw_data.csv --check || exit 1
Output
Default behavior (stdout):
- Writes normalized CSV to stdout
- Progress and errors go to stderr
- Validation errors (if any) are shown to stderr before the output data
- Reject file saved in the system temp directory as
csvnorm_reject_errors_<id>.csv; its full path is shown on stderr (Details:). Nothing is written in the current directory - Perfect for piping to other tools or shell redirection
File output (with -o):
- Creates a normalized CSV file at the specified path with:
- UTF-8 encoding
- Consistent field delimiters
- Normalized column names (unless
--keep-namesis specified)
- Error report if any invalid rows are found (saved as
{output_name}_reject_errors.csvin the same directory) - Shows success table with statistics (rows, columns, file sizes)
- Supports absolute and relative paths
- Any file extension is allowed (not limited to
.csv)
Input file protection:
- csvnorm will never overwrite the input file, even with
--force - If you try to use the same path for input and output, you'll get an error
- Use
-oto specify a different output path
Remote URLs:
- Encoding is handled automatically by DuckDB
- If
--fix-mojibakeis enabled, the URL is downloaded to a temp file first
Encoding detection limits:
- Non-UTF-8 encodings are detected automatically with charset-normalizer, which can confuse single-byte encodings that share most characters (for example cp1252 with cp1250 or cp1258, or cp850 with cp1250). When scores tie, csvnorm prefers Western European encodings (cp1252).
- Check accented and typographic characters (
à,ì,“ ”,€) in the output of non-UTF-8 files. If they are wrong, convert the file to UTF-8 yourself before running csvnorm, for exampleiconv -f cp1252 -t utf-8 data.csv > data_utf8.csv.
Mojibake repair (--fix-mojibake [N]):
- Mojibake is garbled text produced by decoding bytes with the wrong character encoding (e.g.,
CittÃinstead ofCittà). - Enables optional mojibake repair using ftfy (for already-misdecoded text).
Nis the sample size (number of characters) used by the detector; default is 5000.- The repair runs only when ftfy's badness heuristic flags the sample as "bad."
- Use
N=0to force repair without detection (useful for files with low badness scores but visible mojibake). - Note: ftfy cannot recover bytes that were irreversibly lost in the original encoding. Replacement characters (
�) may remain where data was corrupted beyond repair. - HTTP timeout is set to 30 seconds
- Only public URLs are supported (no authentication)
The tool provides modern terminal output (shown only when using -o to write to a file) with:
- Progress indicators for multi-step processing
- Color-coded error messages with panels
- Success summary table with statistics (rows, columns, file sizes)
- Encoding conversion status (converted/no conversion/remote; ASCII is already UTF-8 compatible)
- Error summary panel with reject count and error types when validation fails
- ASCII art banner with
--versionand-Vverbose mode
Success Example: (shown only when using -o)
✓ Success
Input: test/utf8_basic.csv
Output: output/utf8_basic.csv
Encoding: ascii (ASCII is UTF-8 compatible; no conversion needed)
Rows: 2
Columns: 3
Input size: 42 B
Output size: 43 B
Headers: normalized to snake_case
Error Example: (shown only when using -o)
✓ Success
Input: test/malformed_rows.csv
Output: output/malformed_rows.csv
Encoding: ascii (ASCII is UTF-8 compatible; no conversion needed)
Rows: 1
Columns: 4
Input size: 24 B
Output size: 40 B
Headers: normalized to snake_case
╭──────────────────────────── ! Validation Failed ─────────────────────────────╮
│ Validation Errors: │
│ │
│ Rejected rows: 2 │
│ │
│ Error types: │
│ • Expected Number of Columns: 3 Found: 2 │
│ • Expected Number of Columns: 3 Found: 4 │
│ │
│ Details: output/malformed_rows_reject_errors.csv │
╰──────────────────────────────────────────────────────────────────────────────╯
Exit Codes
| Code | Meaning |
|---|---|
| 0 | Success, no validation errors |
| 1 | Validation errors (with or without --strict; without it the accepted rows are still written), or a fatal error (file not found, download failure, etc.) |
| 2 | Invalid command line (unknown option, missing argument) |
Requirements
- Python 3.11+
- Dependencies (automatically installed):
charset-normalizer>=3.0.0- Encoding detectionduckdb>=2.0.0- CSV validation and normalizationftfy>=6.3.1- Mojibake repairrich>=13.0.0- Modern terminal output formattingrich-argparse>=1.0.0- Enhanced CLI help formatting
Optional extras:
[dev]- Development dependencies (pytest>=7.0.0,pytest-cov>=4.0.0,ruff>=0.1.0)
Development
Setup
git clone https://github.com/aborruso/csvnorm
cd csvnorm
# Create and activate venv with uv (recommended)
uv venv
source .venv/bin/activate
uv pip install -e ".[dev]"
# Or with pip
pip install -e ".[dev]"
Testing
pytest tests/ -v
Project Structure
csvnorm/
├── src/csvnorm/
│ ├── __init__.py # Package version
│ ├── __main__.py # python -m support
│ ├── cli.py # CLI argument parsing
│ ├── core.py # Main processing pipeline
│ ├── encoding.py # Encoding detection/conversion
│ ├── validation.py # DuckDB validation
│ └── utils.py # Helper functions
├── tests/ # Test suite
├── test/ # CSV fixtures
└── pyproject.toml # Package configuration
Stay Updated
Get notified of new releases
Watch → Custom → ✓ Releases to receive notifications for all new versions.
Get notified of breaking changes only
Subscribe to Announcements to be notified only about:
- Breaking changes (major version bumps)
- Security updates
- Important deprecation notices
We follow Semantic Versioning:
- MAJOR (e.g., 1.0.0 → 2.0.0): Breaking changes
- MINOR (e.g., 1.0.0 → 1.1.0): New features, backward compatible
- PATCH (e.g., 1.0.0 → 1.0.1): Bug fixes only
See docs/COMMUNICATION.md for details.
License
MIT License (c) 2026 aborruso@gmail.com - See LICENSE file for details
Acknowledgments
csvnorm is built on top of excellent open-source libraries:
- charset-normalizer - Universal encoding detection
- DuckDB - Fast in-process analytical database for CSV processing
- ftfy - Fixes mojibake and other text encoding issues
- Rich - Beautiful terminal output formatting
- rich-argparse - Enhanced CLI help formatting
We are grateful to the creators and maintainers of these libraries, without whom csvnorm would not exist and function.
Metadata
Release files for csvnorm 1.3.0
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | Actions |
|---|---|---|---|---|
| csvnorm-1.3.0.tar.gz | 53.7 kB | With provenance | Details for csvnorm-1.3.0.tar.gz |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Actions Reset |
|---|---|---|---|---|
| csvnorm-1.3.0-py3-none-any.whl | Python 3 | none | any | Details for csvnorm-1.3.0-py3-none-any.whl |
Total release size: 87.5 kB
Release files / csvnorm-1.3.0.tar.gz
| Download URL | csvnorm-1.3.0.tar.gz |
|---|---|
| Size | 53.7 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
53b7250f0b57221b0c143c543e00ad3ad4ac437d8990e5b3aff97a628b9f5d15
|
|
BLAKE2b-256 checksum How to use checksums |
8478f66be87fb25f5e3e85235cbea96db0b28f7a7d3cf4853d8fa0183c6c3ba8
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.16
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Nov 16, 2026.
Transparency logRelease files / csvnorm-1.3.0-py3-none-any.whl
| Download URL | csvnorm-1.3.0-py3-none-any.whl |
|---|---|
| Size | 33.8 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
f1f210003728751e5c7e819c64ab1b54a9d0ca2501d376a0934dc14e3c47c0d2
|
|
BLAKE2b-256 checksum How to use checksums |
d77a35d7ba5efe521e0039c30ca405017b35a57ba474c62d5c1507a6ca74d45c
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.16
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Nov 16, 2026.
Transparency logSecurity for csvnorm 1.3.0
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
Full provenance
Both files in this release have been uploaded with provenance metadata.
Attestation for csvnorm-1.3.0.tar.gz
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Nov 16, 2026.
Transparency logA note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.