compara

Compare two tables on a composite key and get the counts, the per-column statistics and a self-contained HTML report. Key columns, compared columns and normalisation rules are parameters, so one tool serves every recurring comparison: a nightly export against yesterday's, a CSV against the Parquet a warehouse emits, a migration's source against its target.

• Reads CSV, newline-delimited JSON and Parquet, and the two sides need not be in the same format.
• Ten million rows × 20 columns in about a second on four cores (numbers below).
• One static binary; the report is a single HTML file with nothing to host.
• Exit codes made for pipelines: 0 identical, 1 differences, 2 error, 3 duplicate keys (with --fail-on-dups).

Install

Prebuilt binaries for Linux (x86_64, aarch64), macOS (arm64, x86_64) and Windows (x64) are on the releases page, and the same binaries through package managers:

Use

compara compare a.csv b.csv -k account_id,txn_id -i updated_at
compara compare a.csv b.csv -k account_id,txn_id --trim --tolerance 0.005
compara compare export.csv warehouse.parquet -k id --summary
compara compare a.csv b.csv --profile orders --json summary.json --export-dir out/
compara columns data.parquet          # column names (reads only the footer)
compara head data.ndjson -n 5         # first rows, aligned

compare writes <a>__vs__<b>.html (or --out PATH): counts, per-column statistics, and the changed, added, removed and duplicate rows, each section capped at --max-rows (counts are always exact). examples/report.html is one, made from the two files next to it:

compara compare examples/orders_2026-08.csv examples/orders_2026-09.csv -k order_id,line_no -i updated_at

Duplicate keys are counted and listed per file; the first occurrence of each key takes part in the join.

Profiles. A compara.toml in the working directory (or ~/.config/compara/compara.toml, or --config PATH) names recurring comparisons: key, compared and ignored columns, normalisation, engine. See compara.example.toml. The command line overrides the profile.

Speed

Ten million rows × 20 columns, keyed on (account_id, txn_id), --ignore updated_at, one GitHub Actions runner (AMD EPYC 7763, 4 vCPU / 16 GB), best of five (run 36672430118). Each cell is wall time · CPU time · memory above the mapped input.

compara started as the Rust port in data-comparison-polyglot, where the same comparison is implemented in C, C++, Rust and Zig and measured side by side; that repository has the full tables and how they are produced.

Engines

Three backends, one result contract: every engine returns identical counts and column statistics for the same input, and the test suite asserts it cell for cell. --engine auto takes the first one that can load the input, in this order.

All three read values as text, with no type inference, so 1.0 and 1 stay different unless a tolerance is set, and all three treat an empty field as absent whether or not it is quoted.

Input formats

turbo decides what a file is by its content rather than its name: a Parquet magic number, a leading {, or a CSV header.

The Parquet reader is written here rather than taken from a crate, to keep that representation instead of converting to Arrow arrays. It reads PLAIN, dictionary (PLAIN_DICTIONARY, RLE_DICTIONARY), RLE booleans and the version-2 delta encodings; uncompressed, snappy, gzip, zstd and LZ4-raw pages; data pages v1 and v2; definition levels for optional columns. It refuses, by name, nested or repeated schemas, BYTE_STREAM_SPLIT, LZO and Brotli.

A typed Parquet value is rendered to text by a fixed rule: byte arrays as their bytes, booleans true/false, integers and decimals in full, floats in the shortest round-trip form, dates YYYY-MM-DD, timestamps ISO 8601. tests/fixtures/formats/typed.csv is that rule written down.

Contributing

See CONTRIBUTING.md.

License

MIT