compara

Extremely fast table comparison: two tables on a composite key, with counts, per-column statistics and a self-contained HTML report. Key columns, compared columns and normalisation rules are parameters, so one tool serves every recurring comparison: a nightly export against yesterday's, a CSV against the Parquet a warehouse emits, a migration's source against its target.

Faster software for a better planet — less energy, less compute, more time for life.

• Reads CSV, newline-delimited JSON and Parquet, and the two sides need not be in the same format.
• Ten million rows × 20 columns in about a second on four cores (numbers below).
• One static binary; the report is a single HTML file with nothing to host.
• Exit codes made for pipelines: 0 identical, 1 differences, 2 error, 3 duplicate keys (with --fail-on-dups).

Install

Prebuilt binaries for Linux (x86_64, aarch64), macOS (arm64, x86_64) and Windows (x64) are on the releases page, and the same binaries through package managers:

Use

compara compare a.csv b.csv -k account_id,txn_id -i updated_at
compara compare a.csv b.csv -k account_id,txn_id -i updated_at -o report.html
compara compare a.csv b.csv -k account_id,txn_id --trim --tolerance 0.005
compara compare export.csv warehouse.parquet -k id
compara compare a.csv b.csv --profile orders --json summary.json --export-dir out/
compara columns data.parquet          # column names (reads only the footer)
compara head data.ndjson -n 5         # first rows, aligned

compare prints the counts and writes nothing else unless asked. -o PATH writes the HTML report: counts, per-column statistics, and the changed, added, removed and duplicate rows, each section capped at --max-rows (counts are always exact). examples/report.html is one, made from the two files next to it:

compara compare examples/orders_2026-08.csv examples/orders_2026-09.csv -k order_id,line_no -i updated_at -o examples/report.html

Duplicate keys are counted and listed per file; the first occurrence of each key takes part in the join.

Profiles. A compara.toml in the working directory (or ~/.config/compara/compara.toml, or --config PATH) names recurring comparisons: key, compared and ignored columns, normalisation, engine. See compara.example.toml. The command line overrides the profile.

Speed

Ten million rows × 20 columns, keyed on (account_id, txn_id), --ignore updated_at, one GitHub Actions runner (AMD EPYC 7763, 4 vCPU / 16 GB), best of five. Each cell is wall time · CPU time · memory above the mapped input.

Contender benchmarks rerun weekly (bench-contenders.yml) against the latest DuckDB and pandas.

Engines

Three backends, one result contract: every engine returns identical counts and column statistics for the same input, and the test suite asserts it cell for cell. --engine auto takes the first one that can load the input, in this order.

All three read values as text, with no type inference, so 1.0 and 1 stay different unless a tolerance is set, and all three treat an empty field as absent whether or not it is quoted. A Parquet column is the exception, on either side: it stores typed values, so a column it holds as numbers compares by value with the other file's text (31.40 against the DOUBLE 31.4 is no change), and a date, time or timestamp column by instant (T or a space, a zero fraction or none, Z or nothing).

Input formats

turbo decides what a file is by its content rather than its name: a Parquet magic number, a leading {, or a CSV header.

The Parquet reader is written here rather than taken from a crate, to keep that representation instead of converting to Arrow arrays. It reads PLAIN, dictionary (PLAIN_DICTIONARY, RLE_DICTIONARY), RLE booleans and the version-2 delta encodings; uncompressed, snappy, gzip, zstd and LZ4-raw pages; data pages v1 and v2; definition levels for optional columns. It refuses, by name, nested or repeated schemas, BYTE_STREAM_SPLIT, LZO and Brotli.

A typed Parquet value is rendered to text by a fixed rule: byte arrays as their bytes, booleans true/false, integers and decimals in full, floats in the shortest round-trip form, dates YYYY-MM-DD, timestamps ISO 8601. tests/fixtures/formats/typed.csv is that rule written down.

Two Parquet files take a columnar path instead, which never builds a row: it joins on the key columns, then compares one column at a time, string columns by dictionary id or bytes and booleans, integers, floats, dates, times, timestamps and decimals of up to eight bytes as one word per value. A cell is rendered by the rule above only when the report shows it. A column the two files store as different types, or with a codec or encoding outside PLAIN, dictionary, snappy and v1 pages, goes the row-by-row way instead, with the same answer.

Contributing

See CONTRIBUTING.md.

License

MIT