csv-salvage
Read the full documentation at csv-salvage.readthedocs.io.
csv-salvage is a high-performance, fault-tolerant CSV salvage and reconstruction engine written in Rust with native Python bindings (via PyO3 & Maturin).
It rescues corrupted, malformed, and irregular CSV data in scenarios where Polars, DuckDB, and Pandas fail and throw ParserError.
💥 The Pain Point It Solves
Real-world datasets from legacy systems, mainframe exports, and unvalidated third-party vendors often generate severely corrupted CSV files:
- Unescaped newlines inside text cells: Line breaks breaking rows in half without proper quote enclosures.
- Unbalanced or orphan quotes: Stray quotes inside text fields (e.g.
101,"15" Monitor",49.99) throwing parsers completely out of sync. - Variable delimiters: Inconsistent separators across files or mixed comma/semicolon/tab dialects.
- Inconsistent column counts (ragged rows): Missing trailing columns, phantom extra delimiters, or unexpected column drift.
The Current State of the Art (And Why It Fails)
Today, data engineers and analysts resort to writing slow, brittle Python regular expression scripts to clean and patch corrupted files line-by-line before feeding them into Pandas, DuckDB, or Polars. These scripts take hours on multi-gigabyte datasets, consume massive RAM, and break on unseen edge cases.
⚡ Core Architecture in Rust
- DFA & State-Machine Parser: Custom finite state automaton (DFA via manual state machine or
regex-automata) engineered specifically for error-recovery, token resynchronization, and zero-panic resilience. - Schema Inference Heuristics: Adaptive detection of expected column counts, delimiters, and dialect conventions even in chaotic files.
- Parallel Row Normalization: Parsed rows are normalized with Rayon in parallel for larger inputs (1,024 or more rows); quote-aware parsing remains sequential to preserve multiline CSV boundaries.
🌟 Value Proposition
- "Lifesaver" Mode (Zero Panic Guarantee):
- The parser never crashes.
- Automatically repairs orphan quotes and equalizes ragged rows.
- If a row is mathematically unrecoverable, it cleanly routes the corrupted record into a dedicated discard / quarantine stream (with line numbers and diagnostic reasons) rather than halting execution.
- Parallel Batch Normalization:
- Uses
rayonto normalize large batches of parsed rows in parallel while preserving row order. - Exports directly to cleaned, RFC-4180-compliant CSV files ready for instant loading into DuckDB, Polars, or Pandas.
- Provides a Python iterator for salvaged rows.
- Uses
📦 Installation
# Using pip
pip install csv-salvage
# Using uv
uv add csv-salvage
(Pre-built wheels are compiled for Linux, musllinux, macOS, and Windows across major architectures).
🚀 Quickstart & Proposed API
1. Rescuing Corrupted Files for DuckDB / Polars Ingestion
import csv_salvage as cs
# Repair a corrupt CSV file and route unrecoverable rows to quarantine
summary = cs.salvage_file(
input_path="legacy_export_corrupted.csv",
output_path="clean_data.csv",
quarantine_path="discarded_rows.log",
repair_quotes=True, # Auto-fix orphan/unbalanced quotes
fill_ragged_rows=True, # Pad short rows with empty strings
)
print(f"Total Rows: {summary.total_rows}")
print(f"Salvaged Rows: {summary.salvaged_rows}")
print(f"Quarantined: {summary.quarantined_rows}")
# Now load clean_data.csv into Polars or DuckDB with zero errors!
import duckdb
df = duckdb.read_csv("clean_data.csv")
2. Native Python Streaming
import csv_salvage as cs
# Iterate over salvaged rows
with cs.open("broken_records.csv") as stream:
for record in stream:
process_row(record)
3. Command-Line Usage
csv-salvage fix broken.csv --output clean.csv --quarantine discarded.csv
Use --delimiter ';' or --expected-columns 5 to override inference. The
--no-fill-ragged-rows, --no-merge-overflow, and --no-repair-quotes options
disable the corresponding repairs. python -m csv_salvage provides the same
interface.
4. Rust Library Usage
use csv_salvage::{SalvageOptions, rust_salvage_file};
let options = SalvageOptions::default();
let summary = rust_salvage_file(
"corrupted.csv",
"clean.csv",
Some("quarantine.csv"),
&options,
)?;
println!("Successfully salvaged {} rows!", summary.salvaged_rows);
🗺️ Roadmap
- v0.1: resilient parser, dialect inference, row repair, and Python bindings.
- v0.2: release automation and cross-platform CI.
- v0.3: parallel normalization of large row sets and accelerated quote lookahead.
- v0.4: CLI for file repair.
- v0.5: optional Arrow-compatible integration.
- v1.0: benchmark suite, stable API/support contract, and production-readiness review.
🛠️ Development & Contributing
See CONTRIBUTING.md for local setup, development commands, and coding guidelines.
# Synchronize environment
uv sync
# Build & install locally in editable development mode
uv run maturin develop --uv
# Run Rust unit tests
cargo test
# Run Python integration and release tests
uv run python -m unittest discover -s tests -v
# Check formatting and lints
cargo fmt --check
cargo clippy --all-targets --all-features -- -D warnings
# Build documentation strictly
uv run --group docs mkdocs build --strict
📄 License
Licensed under the Apache License, Version 2.0.
Metadata
Release files for csv-salvage 0.4.0
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Built distributions (wheels)
Total release size: 5.0 MB
Release files / csv_salvage-0.4.0-cp310-cp310-win_amd64.whl
| Download URL | csv_salvage-0.4.0-cp310-cp310-win_amd64.whl |
|---|---|
| Size | 214.0 kB |
| Tags | CPython 3.10 Windows x86-64 |
|
SHA-256 checksum How to use checksums |
5b49a539f46a717071c55a3c135d0a152bd46b782e3d315d308c9a727db8060a
|
|
BLAKE2b-256 checksum How to use checksums |
3613efbc9c8b1f6ce54b5f865872078ecde03d2d456649a2521c54d1d098cfd6
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
uv/0.13.3 {"installer":{"name":"uv","version":"0.13.3","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"Ubuntu","version":"24.04","id":"noble","libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":true}
|
Release files / csv_salvage-0.4.0-cp310-cp310-musllinux_1_2_x86_64.whl
| Download URL | csv_salvage-0.4.0-cp310-cp310-musllinux_1_2_x86_64.whl |
|---|---|
| Size | 582.5 kB |
| Tags | CPython 3.10 Linux musl 1.2+ x86-64 |
|
SHA-256 checksum How to use checksums |
7633f429c5b4285489b6b2fee1284bf999b0f2fb56355d05961e5604c9dc6b7a
|
|
BLAKE2b-256 checksum How to use checksums |
84ac859584a90a81a6d085e94ecf30d372926970f744106a6159414d90dc2978
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
uv/0.13.3 {"installer":{"name":"uv","version":"0.13.3","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"Ubuntu","version":"24.04","id":"noble","libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":true}
|
Release files / csv_salvage-0.4.0-cp310-cp310-musllinux_1_2_i686.whl
| Download URL | csv_salvage-0.4.0-cp310-cp310-musllinux_1_2_i686.whl |
|---|---|
| Size | 614.8 kB |
| Tags | CPython 3.10 Linux musl 1.2+ x86-32 |
|
SHA-256 checksum How to use checksums |
738df2678311ab32a6d4c2f24598a02151914187c8e6a92cfb00c89ec797666e
|
|
BLAKE2b-256 checksum How to use checksums |
0e43db6703abba97db588d3d28db213e08eb3d8b5d03de631d7de7864552cae6
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
uv/0.13.3 {"installer":{"name":"uv","version":"0.13.3","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"Ubuntu","version":"24.04","id":"noble","libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":true}
|
Release files / csv_salvage-0.4.0-cp310-cp310-musllinux_1_2_armv7l.whl
| Download URL | csv_salvage-0.4.0-cp310-cp310-musllinux_1_2_armv7l.whl |
|---|---|
| Size | 646.1 kB |
| Tags | CPython 3.10 Linux musl 1.2+ ARMv7l |
|
SHA-256 checksum How to use checksums |
51c2e0308351224ba423a9037d0f8675257c0c323875ee5500c3048a3c2d4cc0
|
|
BLAKE2b-256 checksum How to use checksums |
58766a95386de0345b0d333965bdf920f11a5c051735f6a16934b8bb4f74efe4
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
uv/0.13.3 {"installer":{"name":"uv","version":"0.13.3","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"Ubuntu","version":"24.04","id":"noble","libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":true}
|
Release files / csv_salvage-0.4.0-cp310-cp310-musllinux_1_2_aarch64.whl
| Download URL | csv_salvage-0.4.0-cp310-cp310-musllinux_1_2_aarch64.whl |
|---|---|
| Size | 549.0 kB |
| Tags | CPython 3.10 Linux musl 1.2+ ARM64 |
|
SHA-256 checksum How to use checksums |
0402634d5274f514436064d821f21c0dddc627c2bb7b10332c7309ff1d4d15b3
|
|
BLAKE2b-256 checksum How to use checksums |
516f057f857ad48ee732c29bb865cca21c2f74147daf87f9212870023484887b
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
uv/0.13.3 {"installer":{"name":"uv","version":"0.13.3","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"Ubuntu","version":"24.04","id":"noble","libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":true}
|
Release files / csv_salvage-0.4.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
| Download URL | csv_salvage-0.4.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl |
|---|---|
| Size | 377.7 kB |
| Tags | CPython 3.10 Linux glibc 2.17+ x86-64 |
|
SHA-256 checksum How to use checksums |
1779125dfe84fbb2f29a204f48f613c032a14be44071c8275750ddd60c5eaea2
|
|
BLAKE2b-256 checksum How to use checksums |
bc34ff6fb8fb495b8c0aeaf9ac6b9d698b96f47f8c679538885a35d4fe2455bf
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
uv/0.13.3 {"installer":{"name":"uv","version":"0.13.3","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"Ubuntu","version":"24.04","id":"noble","libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":true}
|
Release files / csv_salvage-0.4.0-cp310-cp310-manylinux_2_17_s390x.manylinux2014_s390x.whl
| Download URL | csv_salvage-0.4.0-cp310-cp310-manylinux_2_17_s390x.manylinux2014_s390x.whl |
|---|---|
| Size | 395.9 kB |
| Tags | CPython 3.10 Linux glibc 2.17+ IBM System/390x |
|
SHA-256 checksum How to use checksums |
53bfc449fad6c44d7abe4d367ccfbbbb1c0e1cbd2f19c422d0afec7e30a8ae9d
|
|
BLAKE2b-256 checksum How to use checksums |
c621c154c05eade0d3852667588d9b38d2caebdb933758da87c5c5f00d727735
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
uv/0.13.3 {"installer":{"name":"uv","version":"0.13.3","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"Ubuntu","version":"24.04","id":"noble","libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":true}
|
Release files / csv_salvage-0.4.0-cp310-cp310-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl
| Download URL | csv_salvage-0.4.0-cp310-cp310-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl |
|---|---|
| Size | 489.6 kB |
| Tags | CPython 3.10 Linux glibc 2.17+ PowerPC 64-le |
|
SHA-256 checksum How to use checksums |
541806834e9fc6427e35f4f1f3d4965af6112a00b128f6e5e8ec321b482668ec
|
|
BLAKE2b-256 checksum How to use checksums |
27267f8285dd9cfec00a3ce4808b289deec51a44086b697b083cd3654d7a84ba
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
uv/0.13.3 {"installer":{"name":"uv","version":"0.13.3","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"Ubuntu","version":"24.04","id":"noble","libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":true}
|
Release files / csv_salvage-0.4.0-cp310-cp310-manylinux_2_17_i686.manylinux2014_i686.whl
| Download URL | csv_salvage-0.4.0-cp310-cp310-manylinux_2_17_i686.manylinux2014_i686.whl |
|---|---|
| Size | 398.7 kB |
| Tags | CPython 3.10 Linux glibc 2.17+ x86-32 |
|
SHA-256 checksum How to use checksums |
bb8690b4f818af4dd4783fc3d3c4a9907862ee0988572f8a00a4995b7983dab2
|
|
BLAKE2b-256 checksum How to use checksums |
75892ea8454119c0d23592ffdb9e1a0bc10b9d328bdc9f36dff4605d1900fa37
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
uv/0.13.3 {"installer":{"name":"uv","version":"0.13.3","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"Ubuntu","version":"24.04","id":"noble","libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":true}
|
Release files / csv_salvage-0.4.0-cp310-cp310-manylinux_2_17_armv7l.manylinux2014_armv7l.whl
| Download URL | csv_salvage-0.4.0-cp310-cp310-manylinux_2_17_armv7l.manylinux2014_armv7l.whl |
|---|---|
| Size | 371.2 kB |
| Tags | CPython 3.10 Linux glibc 2.17+ ARMv7l |
|
SHA-256 checksum How to use checksums |
6055d035df1da5b8bfd0d0c8749e1a9a91ae3ddf22bd9907d501c21258cc6687
|
|
BLAKE2b-256 checksum How to use checksums |
4596f2905fbc45128747511126af157babaec823b3d7a694c6986a89e05209a2
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
uv/0.13.3 {"installer":{"name":"uv","version":"0.13.3","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"Ubuntu","version":"24.04","id":"noble","libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":true}
|
Release files / csv_salvage-0.4.0-cp310-cp310-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
| Download URL | csv_salvage-0.4.0-cp310-cp310-manylinux_2_17_aarch64.manylinux2014_aarch64.whl |
|---|---|
| Size | 372.5 kB |
| Tags | CPython 3.10 Linux glibc 2.17+ ARM64 |
|
SHA-256 checksum How to use checksums |
72ced38957330dde7024078c836ab8351befcdd9d641d32406d29e94e1763644
|
|
BLAKE2b-256 checksum How to use checksums |
4a22dc9e15def9c51d7e440152482cc2b92e3e0a4cb9da6e57b1699f549806b2
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
uv/0.13.3 {"installer":{"name":"uv","version":"0.13.3","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"Ubuntu","version":"24.04","id":"noble","libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":true}
|
Security for csv-salvage 0.4.0
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
No provenance found
None of the files in this release were uploaded with provenance metadata.
A note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.