compara

Extremely fast table comparison: two tables on a composite key, with counts, per-column statistics and a self-contained HTML report. Key columns, compared columns and normalisation rules are parameters, so one tool serves every recurring comparison: a nightly export against yesterday's, a CSV against the Parquet a warehouse emits, a migration's source against its target.

Faster software for a better planet — less energy, less compute, more time for life.

• Reads CSV, TSV, JSON (newline-delimited or one array of objects) and Parquet, and the two sides need not be in the same format.
• Ten million rows × 20 columns in about a second on four cores (numbers below).
• One static binary; the report is a single HTML file with nothing to host.
• Exit codes made for pipelines: 0 identical, 1 differences, 2 error, 3 duplicate keys (with --fail-on-dups).

Install

Prebuilt binaries for Linux (x86_64, aarch64), macOS (arm64, x86_64) and Windows (x64) are on the releases page, and the same binaries through package managers:

Use

compara compare a.csv b.csv -k account_id,txn_id -i updated_at
compara compare a.csv b.csv -k account_id,txn_id -i updated_at -o report.html
compara compare a.csv b.csv -k account_id,txn_id --trim --tolerance 0.005
compara compare export.csv warehouse.parquet -k id
compara compare a.csv b.csv --profile orders --json summary.json --export-dir out/
compara columns data.parquet          # column names (reads only the footer)
compara head data.ndjson -n 5         # first rows, aligned

compare prints the counts and writes nothing else unless asked. -o PATH writes the HTML report: counts, per-column statistics, and the changed, added, removed and duplicate rows, each section capped at --max-rows (counts are always exact). examples/report.html is one, made from the two files next to it:

compara compare examples/orders_2026-08.csv examples/orders_2026-09.csv -k order_id,line_no -i updated_at -o examples/report.html

Duplicate keys are counted and listed per file; the first occurrence of each key takes part in the join.

Profiles. A compara.toml in the working directory (or ~/.config/compara/compara.toml, or --config PATH) names recurring comparisons: key, compared and ignored columns, normalisation, engine. See compara.example.toml. The command line overrides the profile.

Speed

Ten million rows × 20 columns, keyed on (account_id, txn_id), --ignore updated_at, one GitHub Actions runner (AMD EPYC 7763, 4 vCPU / 16 GB), best of five. Each cell is wall time · CPU time · memory above the mapped input.

Two Parquet files against DuckDB and Polars, on the same kind of runner (CI run 39468615515: a 5M-row pair written by pyarrow, counts only, median of three): compara 1.26 s at one thread and 0.56 s at four, DuckDB 6.73 s and 3.13 s, Polars 4.98 s and 2.73 s. With the changed, added and removed rows written out, compara 0.72 s at four threads, DuckDB 4.00 s, Polars 2.74 s.

Contender benchmarks rerun weekly (bench-contenders.yml) against the latest DuckDB.

Engines

Three backends, one result contract: every engine returns identical counts and column statistics for the same input, and the test suite asserts it cell for cell. --engine auto takes the first one that can load the input, in this order.

All three read values as text, with no type inference, so 1.0 and 1 stay different unless a tolerance is set, and all three treat an empty field as absent whether or not it is quoted. A Parquet column is the exception, on either side: it stores typed values, so a column it holds as numbers compares by value with the other file's text (31.40 against the DOUBLE 31.4 is no change), and a date, time or timestamp column by instant (T or a space, a zero fraction or none, Z or nothing).

Input formats

turbo decides what a file is by its content rather than its name: a Parquet magic number, a leading { or [, or a CSV header.

Nested JSON. A value that is an object or an array is compared as its JSON text, as written: any difference inside it is a change to that column, and the report and export show the text. The same object written with its keys in another order, or with other spacing, is a change too, so JSON compares best flat, one scalar per key. For two files written by different tools, flatten nested values into top-level keys first (jq, or the writer's own option), and each nested field becomes a column compared on its own.

The Parquet reader is written here rather than taken from a crate, to keep that representation instead of converting to Arrow arrays. It reads PLAIN, dictionary (PLAIN_DICTIONARY, RLE_DICTIONARY), RLE booleans and the version-2 delta encodings; uncompressed, snappy, gzip, zstd and LZ4-raw pages; data pages v1 and v2; definition levels for optional columns. It refuses, by name, nested or repeated schemas, BYTE_STREAM_SPLIT, LZO and Brotli.

A typed Parquet value is rendered to text by a fixed rule: byte arrays as their bytes, booleans true/false, integers and decimals in full, floats in the shortest round-trip form, dates YYYY-MM-DD, timestamps ISO 8601. tests/fixtures/formats/typed.csv is that rule written down.

Two Parquet files take a columnar path instead, which never builds a row: it joins on the key columns, then compares one column at a time, string columns by dictionary id or bytes and booleans, integers, floats, dates, times, timestamps and decimals of up to eight bytes as one word per value. A cell is rendered by the rule above only when the report shows it or --export-dir lists it. Added and removed rows are known from the join, so their cells are written out during that pass; which rows changed is known only after the last column, so an export reads the compared columns once more and writes out just those rows. It costs that second pass and the text it writes, rather than holding every column of both files at once. A column the two files store as different types, or with an encoding or page outside PLAIN, dictionary and v1 pages (or LZO or Brotli compression), goes the row-by-row way instead, with the same answer.

Contributing

See CONTRIBUTING.md.

License

MIT