Skip to main content

csv-salvage

Read the full documentation at csv-salvage.readthedocs.io.

CI Publish codecov PyPI License Python Versions

csv-salvage is a high-performance, fault-tolerant CSV salvage and reconstruction engine written in Rust with native Python bindings (via PyO3 & Maturin).

It rescues corrupted, malformed, and irregular CSV data in scenarios where Polars, DuckDB, and Pandas fail and throw ParserError.


💥 The Pain Point It Solves

Real-world datasets from legacy systems, mainframe exports, and unvalidated third-party vendors often generate severely corrupted CSV files:

  • Unescaped newlines inside text cells: Line breaks breaking rows in half without proper quote enclosures.
  • Unbalanced or orphan quotes: Stray quotes inside text fields (e.g. 101,"15" Monitor",49.99) throwing parsers completely out of sync.
  • Variable delimiters: Inconsistent separators across files or mixed comma/semicolon/tab dialects.
  • Inconsistent column counts (ragged rows): Missing trailing columns, phantom extra delimiters, or unexpected column drift.

The Current State of the Art (And Why It Fails)

Today, data engineers and analysts resort to writing slow, brittle Python regular expression scripts to clean and patch corrupted files line-by-line before feeding them into Pandas, DuckDB, or Polars. These scripts take hours on multi-gigabyte datasets, consume massive RAM, and break on unseen edge cases.


⚡ Core Architecture in Rust

  • DFA & State-Machine Parser: Custom finite state automaton (DFA via manual state machine or regex-automata) engineered specifically for error-recovery, token resynchronization, and zero-panic resilience.
  • Schema Inference Heuristics: Adaptive detection of expected column counts, delimiters, and dialect conventions even in chaotic files.
  • Parallel Row Normalization: Parsed rows are normalized with Rayon in parallel for larger inputs (1,024 or more rows); quote-aware parsing remains sequential to preserve multiline CSV boundaries.

🌟 Value Proposition

  1. "Lifesaver" Mode (Zero Panic Guarantee):
    • The parser never crashes.
    • Automatically repairs orphan quotes and equalizes ragged rows.
    • If a row is mathematically unrecoverable, it cleanly routes the corrupted record into a dedicated discard / quarantine stream (with line numbers and diagnostic reasons) rather than halting execution.
  2. Parallel Batch Normalization:
    • Uses rayon to normalize large batches of parsed rows in parallel while preserving row order.
    • Exports directly to cleaned, RFC-4180-compliant CSV files ready for instant loading into DuckDB, Polars, or Pandas.
    • Provides a Python iterator for salvaged rows.

📦 Installation

# Using pip
pip install csv-salvage

# Using uv
uv add csv-salvage

(Pre-built wheels are compiled for Linux, musllinux, macOS, and Windows across major architectures).


🚀 Quickstart & Proposed API

1. Rescuing Corrupted Files for DuckDB / Polars Ingestion

import csv_salvage as cs

# Repair a corrupt CSV file and route unrecoverable rows to quarantine
summary = cs.salvage_file(
    input_path="legacy_export_corrupted.csv",
    output_path="clean_data.csv",
    quarantine_path="discarded_rows.log",
    repair_quotes=True,       # Auto-fix orphan/unbalanced quotes
    fill_ragged_rows=True,    # Pad short rows with empty strings
)

print(f"Total Rows:     {summary.total_rows}")
print(f"Salvaged Rows:  {summary.salvaged_rows}")
print(f"Quarantined:    {summary.quarantined_rows}")

# Now load clean_data.csv into Polars or DuckDB with zero errors!
import duckdb
df = duckdb.read_csv("clean_data.csv")

2. Native Python Streaming

import csv_salvage as cs

# Iterate over salvaged rows
with cs.open("broken_records.csv") as stream:
    for record in stream:
        process_row(record)

3. Command-Line Usage

csv-salvage fix broken.csv --output clean.csv --quarantine discarded.csv

Use --delimiter ';' or --expected-columns 5 to override inference. The --no-fill-ragged-rows, --no-merge-overflow, and --no-repair-quotes options disable the corresponding repairs. python -m csv_salvage provides the same interface.

4. Rust Library Usage

use csv_salvage::{SalvageOptions, rust_salvage_file};

let options = SalvageOptions::default();

let summary = rust_salvage_file(
    "corrupted.csv",
    "clean.csv",
    Some("quarantine.csv"),
    &options,
)?;

println!("Successfully salvaged {} rows!", summary.salvaged_rows);

🗺️ Roadmap

  • v0.1: resilient parser, dialect inference, row repair, and Python bindings.
  • v0.2: release automation and cross-platform CI.
  • v0.3: parallel normalization of large row sets and accelerated quote lookahead.
  • v0.4: CLI for file repair.
  • v0.5: optional Arrow-compatible integration.
  • v1.0: benchmark suite, stable API/support contract, and production-readiness review.

🛠️ Development & Contributing

See CONTRIBUTING.md for local setup, development commands, and coding guidelines.

# Synchronize environment
uv sync

# Build & install locally in editable development mode
uv run maturin develop --uv

# Run Rust unit tests
cargo test

# Run Python integration and release tests
uv run python -m unittest discover -s tests -v

# Check formatting and lints
cargo fmt --check
cargo clippy --all-targets --all-features -- -D warnings

# Build documentation strictly
uv run --group docs mkdocs build --strict

📄 License

Licensed under the Apache License, Version 2.0.

Metadata

Release files for csv-salvage 0.4.0

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Built distributions (wheels)

Table of built distributions (wheels) for csv-salvage 0.4.0
File Actions
csv_salvage-0.4.0-cp310-cp310-win_amd64.whl CPython 3.10 CPython 3.10 Windows x86-64 Details for csv_salvage-0.4.0-cp310-cp310-win_amd64.whl
csv_salvage-0.4.0-cp310-cp310-musllinux_1_2_x86_64.whl CPython 3.10 CPython 3.10 Linux musl 1.2+ x86-64 Details for csv_salvage-0.4.0-cp310-cp310-musllinux_1_2_x86_64.whl
csv_salvage-0.4.0-cp310-cp310-musllinux_1_2_i686.whl CPython 3.10 CPython 3.10 Linux musl 1.2+ x86-32 Details for csv_salvage-0.4.0-cp310-cp310-musllinux_1_2_i686.whl
csv_salvage-0.4.0-cp310-cp310-musllinux_1_2_armv7l.whl CPython 3.10 CPython 3.10 Linux musl 1.2+ ARMv7l Details for csv_salvage-0.4.0-cp310-cp310-musllinux_1_2_armv7l.whl
csv_salvage-0.4.0-cp310-cp310-musllinux_1_2_aarch64.whl CPython 3.10 CPython 3.10 Linux musl 1.2+ ARM64 Details for csv_salvage-0.4.0-cp310-cp310-musllinux_1_2_aarch64.whl
csv_salvage-0.4.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl CPython 3.10 CPython 3.10 Linux glibc 2.17+ x86-64 Details for csv_salvage-0.4.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
csv_salvage-0.4.0-cp310-cp310-manylinux_2_17_s390x.manylinux2014_s390x.whl CPython 3.10 CPython 3.10 Linux glibc 2.17+ IBM System/390x Details for csv_salvage-0.4.0-cp310-cp310-manylinux_2_17_s390x.manylinux2014_s390x.whl
csv_salvage-0.4.0-cp310-cp310-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl CPython 3.10 CPython 3.10 Linux glibc 2.17+ PowerPC 64-le Details for csv_salvage-0.4.0-cp310-cp310-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl
csv_salvage-0.4.0-cp310-cp310-manylinux_2_17_i686.manylinux2014_i686.whl CPython 3.10 CPython 3.10 Linux glibc 2.17+ x86-32 Details for csv_salvage-0.4.0-cp310-cp310-manylinux_2_17_i686.manylinux2014_i686.whl
csv_salvage-0.4.0-cp310-cp310-manylinux_2_17_armv7l.manylinux2014_armv7l.whl CPython 3.10 CPython 3.10 Linux glibc 2.17+ ARMv7l Details for csv_salvage-0.4.0-cp310-cp310-manylinux_2_17_armv7l.manylinux2014_armv7l.whl
csv_salvage-0.4.0-cp310-cp310-manylinux_2_17_aarch64.manylinux2014_aarch64.whl CPython 3.10 CPython 3.10 Linux glibc 2.17+ ARM64 Details for csv_salvage-0.4.0-cp310-cp310-manylinux_2_17_aarch64.manylinux2014_aarch64.whl

Total release size: 5.0 MB

Release files / csv_salvage-0.4.0-cp310-cp310-win_amd64.whl

Download URL csv_salvage-0.4.0-cp310-cp310-win_amd64.whl
Size 214.0 kB
Tags CPython 3.10 Windows x86-64
SHA-256 checksum
How to use checksums
5b49a539f46a717071c55a3c135d0a152bd46b782e3d315d308c9a727db8060a
BLAKE2b-256 checksum
How to use checksums
3613efbc9c8b1f6ce54b5f865872078ecde03d2d456649a2521c54d1d098cfd6
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via uv/0.13.3 {"installer":{"name":"uv","version":"0.13.3","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"Ubuntu","version":"24.04","id":"noble","libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":true}

Release files / csv_salvage-0.4.0-cp310-cp310-musllinux_1_2_x86_64.whl

Download URL csv_salvage-0.4.0-cp310-cp310-musllinux_1_2_x86_64.whl
Size 582.5 kB
Tags CPython 3.10 Linux musl 1.2+ x86-64
SHA-256 checksum
How to use checksums
7633f429c5b4285489b6b2fee1284bf999b0f2fb56355d05961e5604c9dc6b7a
BLAKE2b-256 checksum
How to use checksums
84ac859584a90a81a6d085e94ecf30d372926970f744106a6159414d90dc2978
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via uv/0.13.3 {"installer":{"name":"uv","version":"0.13.3","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"Ubuntu","version":"24.04","id":"noble","libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":true}

Release files / csv_salvage-0.4.0-cp310-cp310-musllinux_1_2_i686.whl

Download URL csv_salvage-0.4.0-cp310-cp310-musllinux_1_2_i686.whl
Size 614.8 kB
Tags CPython 3.10 Linux musl 1.2+ x86-32
SHA-256 checksum
How to use checksums
738df2678311ab32a6d4c2f24598a02151914187c8e6a92cfb00c89ec797666e
BLAKE2b-256 checksum
How to use checksums
0e43db6703abba97db588d3d28db213e08eb3d8b5d03de631d7de7864552cae6
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via uv/0.13.3 {"installer":{"name":"uv","version":"0.13.3","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"Ubuntu","version":"24.04","id":"noble","libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":true}

Release files / csv_salvage-0.4.0-cp310-cp310-musllinux_1_2_armv7l.whl

Download URL csv_salvage-0.4.0-cp310-cp310-musllinux_1_2_armv7l.whl
Size 646.1 kB
Tags CPython 3.10 Linux musl 1.2+ ARMv7l
SHA-256 checksum
How to use checksums
51c2e0308351224ba423a9037d0f8675257c0c323875ee5500c3048a3c2d4cc0
BLAKE2b-256 checksum
How to use checksums
58766a95386de0345b0d333965bdf920f11a5c051735f6a16934b8bb4f74efe4
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via uv/0.13.3 {"installer":{"name":"uv","version":"0.13.3","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"Ubuntu","version":"24.04","id":"noble","libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":true}

Release files / csv_salvage-0.4.0-cp310-cp310-musllinux_1_2_aarch64.whl

Download URL csv_salvage-0.4.0-cp310-cp310-musllinux_1_2_aarch64.whl
Size 549.0 kB
Tags CPython 3.10 Linux musl 1.2+ ARM64
SHA-256 checksum
How to use checksums
0402634d5274f514436064d821f21c0dddc627c2bb7b10332c7309ff1d4d15b3
BLAKE2b-256 checksum
How to use checksums
516f057f857ad48ee732c29bb865cca21c2f74147daf87f9212870023484887b
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via uv/0.13.3 {"installer":{"name":"uv","version":"0.13.3","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"Ubuntu","version":"24.04","id":"noble","libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":true}

Release files / csv_salvage-0.4.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl

Download URL csv_salvage-0.4.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Size 377.7 kB
Tags CPython 3.10 Linux glibc 2.17+ x86-64
SHA-256 checksum
How to use checksums
1779125dfe84fbb2f29a204f48f613c032a14be44071c8275750ddd60c5eaea2
BLAKE2b-256 checksum
How to use checksums
bc34ff6fb8fb495b8c0aeaf9ac6b9d698b96f47f8c679538885a35d4fe2455bf
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via uv/0.13.3 {"installer":{"name":"uv","version":"0.13.3","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"Ubuntu","version":"24.04","id":"noble","libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":true}

Release files / csv_salvage-0.4.0-cp310-cp310-manylinux_2_17_s390x.manylinux2014_s390x.whl

Download URL csv_salvage-0.4.0-cp310-cp310-manylinux_2_17_s390x.manylinux2014_s390x.whl
Size 395.9 kB
Tags CPython 3.10 Linux glibc 2.17+ IBM System/390x
SHA-256 checksum
How to use checksums
53bfc449fad6c44d7abe4d367ccfbbbb1c0e1cbd2f19c422d0afec7e30a8ae9d
BLAKE2b-256 checksum
How to use checksums
c621c154c05eade0d3852667588d9b38d2caebdb933758da87c5c5f00d727735
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via uv/0.13.3 {"installer":{"name":"uv","version":"0.13.3","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"Ubuntu","version":"24.04","id":"noble","libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":true}

Release files / csv_salvage-0.4.0-cp310-cp310-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl

Download URL csv_salvage-0.4.0-cp310-cp310-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl
Size 489.6 kB
Tags CPython 3.10 Linux glibc 2.17+ PowerPC 64-le
SHA-256 checksum
How to use checksums
541806834e9fc6427e35f4f1f3d4965af6112a00b128f6e5e8ec321b482668ec
BLAKE2b-256 checksum
How to use checksums
27267f8285dd9cfec00a3ce4808b289deec51a44086b697b083cd3654d7a84ba
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via uv/0.13.3 {"installer":{"name":"uv","version":"0.13.3","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"Ubuntu","version":"24.04","id":"noble","libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":true}

Release files / csv_salvage-0.4.0-cp310-cp310-manylinux_2_17_i686.manylinux2014_i686.whl

Download URL csv_salvage-0.4.0-cp310-cp310-manylinux_2_17_i686.manylinux2014_i686.whl
Size 398.7 kB
Tags CPython 3.10 Linux glibc 2.17+ x86-32
SHA-256 checksum
How to use checksums
bb8690b4f818af4dd4783fc3d3c4a9907862ee0988572f8a00a4995b7983dab2
BLAKE2b-256 checksum
How to use checksums
75892ea8454119c0d23592ffdb9e1a0bc10b9d328bdc9f36dff4605d1900fa37
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via uv/0.13.3 {"installer":{"name":"uv","version":"0.13.3","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"Ubuntu","version":"24.04","id":"noble","libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":true}

Release files / csv_salvage-0.4.0-cp310-cp310-manylinux_2_17_armv7l.manylinux2014_armv7l.whl

Download URL csv_salvage-0.4.0-cp310-cp310-manylinux_2_17_armv7l.manylinux2014_armv7l.whl
Size 371.2 kB
Tags CPython 3.10 Linux glibc 2.17+ ARMv7l
SHA-256 checksum
How to use checksums
6055d035df1da5b8bfd0d0c8749e1a9a91ae3ddf22bd9907d501c21258cc6687
BLAKE2b-256 checksum
How to use checksums
4596f2905fbc45128747511126af157babaec823b3d7a694c6986a89e05209a2
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via uv/0.13.3 {"installer":{"name":"uv","version":"0.13.3","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"Ubuntu","version":"24.04","id":"noble","libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":true}

Release files / csv_salvage-0.4.0-cp310-cp310-manylinux_2_17_aarch64.manylinux2014_aarch64.whl

Download URL csv_salvage-0.4.0-cp310-cp310-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Size 372.5 kB
Tags CPython 3.10 Linux glibc 2.17+ ARM64
SHA-256 checksum
How to use checksums
72ced38957330dde7024078c836ab8351befcdd9d641d32406d29e94e1763644
BLAKE2b-256 checksum
How to use checksums
4a22dc9e15def9c51d7e440152482cc2b92e3e0a4cb9da6e57b1699f549806b2
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via uv/0.13.3 {"installer":{"name":"uv","version":"0.13.3","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"Ubuntu","version":"24.04","id":"noble","libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":true}

Security for csv-salvage 0.4.0

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

No provenance found

None of the files in this release were uploaded with provenance metadata.

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page