deepdiff-rs

onix is a Rust rewrite of Python DeepDiff's core: byte-compatible output, 37-4245x faster, with ignore_order support included. Install it as deepdiff-rs, a drop-in DeepDiff class for Python, or run the diff engine as the onix command-line tool.

deepdiff-rs reads live Python objects (or JSON) and produces the exact same report DeepDiff does at verbose_level=2, so it slots into code that already parses DeepDiff output while running dramatically faster on large or deeply nested inputs.

Status (September 2026): deepdiff-rs 0.x is live on PyPI (Python 3.9+, wheels for Linux x86_64/aarch64, macOS arm64/x86_64, and Windows x64, plus an sdist); the onix CLI builds from source, and nothing is on crates.io yet. Ordered and ignore_order diffing are complete, differentially tested against real DeepDiff 9.1.0, and benchmarked. It is 0.x, not stable or 1.0: the API may still change before 1.0.

Table of contents

• Install
• Quickstart
• Diffing tables
• Performance
• Reference
• Layout
• Known limitations
• Contributing
• License

Install

Python (the deepdiff-rs package, import name deepdiff_rs):

pip install deepdiff-rs

From source:

cd crates/onix-py
uv tool install maturin              # the build tool (skip if already installed)
uv sync --group test                 # creates .venv, installs pytest, pinned deepdiff, and pyarrow/polars/duckdb/pandas for the table-diff tests
uv run --group test maturin develop --release

CLI (the onix binary), from a clean clone:

cargo install --path crates/onix-cli

Library crate (onix-core), a path dependency only (it sets publish = false):

[dependencies]
onix-core = { path = "crates/onix-core" }

Quickstart

The drop-in DeepDiff class, on live Python objects:

from deepdiff_rs import DeepDiff

diff = DeepDiff({"a": 1}, {"a": 2})
if diff:
    print(diff.to_json())   # byte-compatible with DeepDiff(...).to_json() at verbose_level=2
    print(diff.to_dict())   # the same report as a native Python dict

{"values_changed":{"root['a']":{"new_value":2,"old_value":1}}}
{'values_changed': {"root['a']": {'new_value': 2, 'old_value': 1}}}

diff_json, the fast path when you already have JSON text (it parses, diffs, and serializes entirely in Rust, with no Python-object conversion):

from deepdiff_rs import diff_json

print(diff_json('{"a": 1}', '{"a": 2}'))

{"values_changed":{"root['a']":{"new_value":2,"old_value":1}}}

The onix CLI, diffing two JSON files (compact JSON to stdout, {} for no differences):

$ echo '{"a": 1}' > left.json
$ echo '{"a": 2}' > right.json
$ onix diff left.json right.json
{"values_changed":{"root['a']":{"new_value":2,"old_value":1}}}

Pass --ignore-order to compare every list by value instead of by position, mirroring DeepDiff(..., ignore_order=True).

Diffing tables

diff_tables compares two tables the way DeepDiff compares two objects. It takes any object implementing the Arrow PyCapsule interface — a pyarrow Table or RecordBatch, a polars DataFrame, a DuckDB relation — and imports it with no Python round trip. The two tables are matched on a required, non-empty set of key columns (the table's primary key).

It reports the schema diff (which columns were added, removed, or changed type), the keyed row diff (which rows were added, removed, or changed, and which keys are duplicated), and the per-cell diff (cells_changed): one row per changed cell, carrying the key columns, column, old_value/new_value, and change (became_null/became_non_null, type_changed, or value_changed), ordered by the canonical string rendering of the key columns (nulls first), then left-schema column order — the exact rendering and change-classification rules are in the module doc of crates/onix-arrow/src/row_diff.rs. Rows are matched by the key columns; rows_added, rows_removed, cells_changed, and duplicate_keys return Arrow tables, and summary() counts each outcome.

import pyarrow as pa
from deepdiff_rs import diff_tables

left = pa.table({
    "id": pa.array([1, 2, 3, 9, 9], pa.int64()),  # 9 is a duplicate key
    "amount": pa.array([10, 20, 30, 90, 91], pa.int32()),
})
right = pa.table({
    "id": pa.array([2, 3, 4], pa.int64()),
    "amount": pa.array([20, 31, 40], pa.int64()),
    "note": pa.array(["a", "b", "c"], pa.string()),
})

diff = diff_tables(left, right, key=["id"])
print(diff.summary(), "added ids:", pa.table(diff.rows_added()).column("id").to_pylist(), "removed ids:", pa.table(diff.rows_removed()).column("id").to_pylist())
print("cells changed:", pa.table(diff.cells_changed()).to_pylist(), "duplicate keys:", pa.table(diff.duplicate_keys()).to_pylist())

{'columns_added': 1, 'columns_removed': 0, 'columns_type_changed': 1, 'rows_added': 1, 'rows_removed': 1, 'rows_changed': 1, 'duplicate_keys': 1, 'null_keys': 0, 'cells_changed': 1} added ids: [4] removed ids: [1]
cells changed: [{'id': 3, 'column': 'amount', 'old_value': '30', 'new_value': '31', 'change': 'value_changed'}] duplicate keys: [{'id': 9, 'left_count': 2, 'right_count': 0}]

A key appearing more than once on either side is reported in duplicate_keys (with left_count and right_count) and excluded from the added/removed/changed sets; a null key matches its counterpart and is counted in null_keys. Rows are compared by the non-key columns present on both sides, with onix's value semantics (integers and integral floats fold together, all NaNs compare equal, 1.00 equals 1.0000, a timestamp compares by its instant and a time or duration by its value across units, dictionary-encoded values equal their plain form, and null equals null); a nested non-key column is out of scope and is skipped rather than compared. The exact value-comparison rules are documented on the hashing functions in crates/onix-arrow/src/row_diff.rs.

Type comparison uses the full logical Arrow type (timestamp unit and timezone, decimal precision and scale, and so on), but physical encodings that carry the same logical type compare equal — a dictionary-encoded string equals a plain string, polars' Utf8View equals pyarrow's Utf8, the list variants normalize together, and a map compares equal however a library spells it — so the same table read through pyarrow, polars, or DuckDB reports no spurious type changes. The full normalization rules are documented on normalized_type (and map_entries) in crates/onix-arrow/src/schema.rs; nullability is ignored but reported in each record. Column names must be unique on each side; a repeated name raises ValueError. diff.schema_arrow is the same result as an Arrow table: it implements __arrow_c_stream__, so polars.DataFrame(diff.schema_arrow) consumes it with no pyarrow needed, and diff.schema_arrow.to_pyarrow() returns a pyarrow.Table; pandas.api.interchange.from_dataframe(diff.schema_arrow) also works, but pandas' own implementation of that protocol needs pyarrow installed regardless of which path you take.

pyarrow is optional: install it with pip install deepdiff-rs[arrow]. It is needed only for to_pyarrow() and for passing pyarrow objects in — importing deepdiff_rs and diffing polars or DuckDB tables need it not at all. Passing an object that implements neither Arrow protocol raises TypeError; calling to_pyarrow() without pyarrow installed raises ImportError naming the extra. diff.to_json() gives the whole diff — schema, summary, and rows_added/rows_removed/cells_changed/duplicate_keys in full, one JSON object per row — as a single string with no pyarrow, polars, or pandas needed at all; see Known limitations for its row cap.

Performance

Two committed, regenerable reports back the numbers below; every figure here is copied verbatim from them.

The Python bindings against real deepdiff on live Python objects, the number a real caller pays (source: crates/onix-py/benchmarks/bench_bindings.py, macOS 26.5.1, Apple M5 Max, median of 11 isolated subprocess runs per side, run on 2026-09-04):

The engine's own diff-only time and peak resident memory against pinned deepdiff 9.1.0 (source: perf/RESULTS.md, same machine, median over tier-appropriate runs, diff time excluding process startup and JSON parsing on both sides):

Both reports carry their full methodology, fairness rules, and the reproduce command. perf/RESULTS.md is an upper bound (JSON parsed straight into the engine, no Python-object conversion); the bindings table is the product-surface number. Regenerate them with perf/run_bench.sh and crates/onix-py/benchmarks/bench_bindings.py (see CONTRIBUTING.md).

Reference

Python API. The public surface is DeepDiff, diff_json, diff_tables (returning a TableDiff), MaxDepthError, and MAX_DEPTH_CEILING.

• DeepDiff(t1, t2, ignore_order=False, max_depth=None): diffs two live Python objects of supported value types — None, bool, int, float, str, dict, list, tuple, set, frozenset, datetime.datetime, datetime.date, datetime.time, and datetime.timedelta (see Known limitations for the exact restrictions and exclusions, including which types a dict key may be); .to_json() returns the DeepDiff-compatible JSON string, .to_dict() the same report as a dict — with Python types preserved, so a value the diff found in a tuple, set or frozenset comes back as one and a datetime/date/time/timedelta comes back as a real one of those — and the instance is falsy when there is no difference. The set_item_added/set_item_removed categories are lists of path strings, each ending in the item itself (root['a'][2], root['x'], root[(1, 2)]).
• diff_json(a, b, ignore_order=False, max_depth=None) -> str: diffs two JSON strings entirely in Rust and returns the report as a JSON string.
• MaxDepthError (a ValueError subclass) is raised when input exceeds max_depth; MAX_DEPTH_CEILING (20,000) is the hard upper bound on max_depth.
• diff_tables(left, right, key=[...]) -> TableDiff: diffs two Arrow tables (see Diffing tables). TableDiff.schema is the list of changed columns, .schema_arrow the same as an Arrow table, .summary() the schema and row change counts, .to_json() the full diff (schema, summary, and every row-level member, capped at 10,000 embedded rows); .rows_added(), .rows_removed(), .cells_changed(), and .duplicate_keys() return Arrow tables.

CLI. onix diff <a.json> <b.json> [--max-depth N] [--ignore-order] [--timing] reads both files as JSON and prints a compact, single-line DeepDiff-compatible report to stdout ({} when there is no difference).

• --max-depth N overrides the recursion-depth bound (default: the ONIX_MAX_DEPTH environment variable if set, else 512).
• --ignore-order compares every list by hash-based matching instead of by position, mirroring DeepDiff(..., ignore_order=True).
• --timing prints one line of JSON ({"parse_ns": N, "diff_ns": N}) to stderr.

Exit codes:

Layout

crates/onix-core   # the diff engine (library, no I/O)
crates/onix-cli    # the `onix` binary (thin CLI over the core)
crates/onix-arrow  # Arrow table diffing (schema diff and keyed row diff)
crates/onix-py     # PyO3 bindings, published as `deepdiff-rs`
scripts/           # gen_goldens.py: regenerates tests/golden/ from real DeepDiff
tests/golden       # DeepDiff-generated expected outputs (the compatibility corpus)
perf/              # cross-language benchmark harness and RESULTS.md

Known limitations

• Only the core diff is implemented: exclude_paths, significant_digits, custom operators, verbose_level != 2, and delta/patch are not (yet) supported.
• Supported value types are None, bool, int, float, str, dict, list, tuple, set, frozenset, datetime.datetime, datetime.date, datetime.time, and datetime.timedelta; a set/frozenset member may be any of these except a list, dict or set, matching Python's own hashability rule, transitively through whatever the member nests, and a dict key may be str, None, bool, int, float, datetime.datetime, datetime.date, or a tuple of those (never nested). ints must fit in i64/u64, floats must be finite, and anything else — a custom object, an arbitrary-precision int, or a non-finite float — raises TypeError/ValueError naming the exact path it was found at. A non-str key's path renders via Python's own repr(), except a tuple key, which splits into one bracket group per element (root[1][2] for (1, 2), never root[(1, 2)], with one deliberate exception for a real DeepDiff bug on the empty tuple — see tests/golden/README.md). A nested dict value's own bool/None/int/float key stringifies the way json.dumps does, but its datetime/date/tuple key renders that same repr() text where DeepDiff's own to_json() raises TypeError on one — a superset, not a difference in the findings (see tests/golden/README.md). 1/1.0/True match as the same key between two dicts (Python dict/set equality). The Datetimes and Sets bullets below cover the deliberate divergences for those types. See crates/onix-py/src/convert.rs and tests/golden/README.md.
• A subclass of a supported type (a tuple, set or frozenset subclass including namedtuple, a datetime/date/time/timedelta subclass such as pandas' Timestamp) raises TypeError rather than being diffed as its base type, because DeepDiff reports each value's own type name. A type_changes entry's old_type/new_type are type names in to_dict(), where DeepDiff returns the type objects. Both are described in tests/golden/README.md.
• Datetimes compare by instant, with a naive value read as UTC, matching DeepDiff. A changed pair is reported normalized to UTC (to_json() renders ...+00:00, to_dict() returns UTC-aware datetimes); everywhere else a datetime keeps its raw value. Three deliberate departures: to_json() renders a date as YYYY-MM-DD where DeepDiff's own to_json() raises TypeError (a documented superset); a zoneinfo/pytz tzinfo comes back from to_dict() as a fixed-offset datetime.timezone carrying the offset it was in force at, not the original zone object; and a set holding both a naive and an aware value at one instant reports both as members, where DeepDiff's own digest cache can report only one (see crates/onix-py/src/convert.rs). Comparing two datetimes whose UTC form would leave year 1..=9999 raises ValueError naming the path, where DeepDiff raises OverflowError; under ignore_order DeepDiff's hasher normalizes every datetime and so raises for such a value even when it is only added, removed, or shuffled, where onix hashes by instant and reports it normally (see tests/golden/README.md). truncate_datetime is not supported; the normalized-versus-raw split is documented there too. A time/timedelta, unlike a datetime, is never normalized for report (DeepDiff compares time/date/timedelta with a plain !=), and a naive time is never equal to an aware one; to_json() renders a time as time.isoformat()'s bytes and a timedelta as str(timedelta)'s, both supersets. Under ignore_order, DeepHash hashes a time by whole seconds-of-day only — dropping the microsecond and any offset, a confirmed upstream quirk — while a timedelta hashes exactly; see tests/golden/README.md's "Known DeepDiff quirks" section.
• Sets are diffed deterministically, where DeepDiff's own answers depend on the order the running process happens to iterate a set in (hash order, and PYTHONHASHSEED-dependent for str members) or on how its digest cache/computation handles a tuple, frozenset, or calendar member independently of Python's own ==. Each consequence — entry order, which member of an equality class is reported, set-versus-sequence coercion, and a tuple/frozenset member's own (positional, not order-/repetition-insensitive) matching rule — is shown with both tools' output in tests/golden/README.md's "Set iteration order" section. A report holding a frozenset value also serializes to JSON here, where DeepDiff's own to_json() raises TypeError — a superset, not a difference in the findings.
• A str containing a lone (unpaired) surrogate code point (e.g. '\udc80', legal in Python but not encodable as UTF-8) raises ValueError naming the exact path on either side, before the two values are ever compared — including a pair DeepDiff would call equal and report as no change, since DeepDiff's scalar equality is plain Python == and never hits the encoding problem; DeepDiff does report a plain change for a differing pair, and crashes with an unhandled UnicodeEncodeError if such a string is ever hashed (a set/frozenset member). See tests/golden/README.md's "Known DeepDiff quirks" section.
• A str inside a tuple or frozenset set item is escaped exactly as Python's repr() escapes it, against Unicode 16.0.0; on a Python older than 3.14 (an older unicodedata table), a code point assigned to Unicode after that Python's own version is escaped by DeepDiff and rendered literally by onix. See tests/golden/README.md's "Pinned versions" section.
• Adversarially deep input raises MaxDepthError instead of crashing: the default max_depth is 512 and the hard ceiling is MAX_DEPTH_CEILING (20,000). See crates/onix-py/src/guard.rs.
• ignore_order pairing is O(N^2) in unpaired elements per side and carries a polynomial cost in both time and memory with input depth; it has no max_passes/max_diffs cutoff, so bound the size and depth of untrusted input yourself. See crates/onix-core/src/ignore_order/mod.rs.
• A values_changed between two multi-line strings runs a difflib-style O(N*M) line diff on the default path with no opt-out, worst when changes are spread evenly through the text (about 35 s for a heavily edited 1 MB string and growing quadratically, so a few megabytes is minutes), so bound the size of untrusted strings yourself. See crates/onix-core/src/unified_diff.rs.
• Ordered sequences (list or tuple) of scalars (null, bool, number, string, datetime, date, time, timedelta) run a difflib-style O(N*M) matcher on the default path with its popular-element (autojunk) purge disabled for DeepDiff parity, worst for sequences of a few repeated values with dense edits (about 620 s for two 8,000-element lists of two repeated values with every other element changed, growing faster than quadratically), so bound the size of untrusted sequences yourself. See crates/onix-core/src/lcs.rs.
• diff_tables inherits some Arrow-interchange quirks: a column name with an embedded NUL byte (\0) arrives truncated at the NUL through the C Data Interface (the report shows the truncated name; rare in practice); a list of structs named exactly key/value with a nullable key is not distinguished from a real map, so a migration between the two is not reported as a type change (polars exports both as the same Arrow type — see map_entries in crates/onix-arrow/src/schema.rs); and a polars all-null (Null-typed) column fails at Arrow C import with a ValueError (the datatype "Null" doesn't expect buffer at index 0), so give such a column a concrete type first (a pyarrow all-None column, inferred as null, works and compares as all-null).
• In diff_tables, DuckDB labels a TIMESTAMP WITH TIME ZONE column with the connection's session time zone when it exports to Arrow (a UTC session as Timestamp(µs, "UTC"), an America/New_York session as Timestamp(µs, "America/New_York")), so on a non-UTC machine such a column can be reported as a type change against a UTC column from another library. Run SET TimeZone='UTC' on the DuckDB connection first for a deterministic, machine-independent result.
• diff_tables refuses a column whose Arrow type is nested deeper than MAX_NESTING_DEPTH (128) with a MaxDepthError, because comparing arbitrarily deep nesting would overflow the native stack; 128 is far beyond any real schema. Importing a schema nested many thousands of levels deep is also slow regardless, a cost of the Arrow C Data Interface itself.
• diff_tables's row diff costs, per call: RAM for a 32-byte hash per row per side (sorted), so peak memory is linear in row count (about 75 MB at 1M rows/side, 660 MB at 10M) and time is N·log N (about 2 s for a 10M-row pair), plus a second term for the duplicate-key report, which holds the key values of every distinct duplicated key — an all-duplicate 200k-rows/side table (100k distinct duplicated keys) peaks at about 37 MB with 16-byte keys and 1.05 GB with 1 KB keys, 1M rows/side (500k distinct) at about 165 MB with 16-byte keys; a third term for the per-cell diff, which re-reads both inputs, holds the changed rows of both sides, and renders every changed cell to a string held once in the output — so its cost is the number of changed cells times the cell width, not the changed-row count alone (every value is rendered in full). Measured, same method: with two narrow int64 columns, 85 MB at 1M rows/side with 2% changed and 629 MB with every row changed; with a 1 KB string cell changed on every row, 1.18 GB at 100k rows/side and 2.34 GB at 200k (about 11.6 KB per changed row, linear, so on the order of 11.6 GB at 1M). Then temp disk, because each input is re-read several times and so is spooled to an anonymous file (tempfile: unlinked at once, mode 0600, no predictable name — nothing is left on disk even on abnormal exit), both spools resident at once, peaking at the decoded size of both inputs (about 315 MB — 161 + 154 MB uncompressed Arrow IPC — for the 1M-row fixture pair; on the order of 10 GB for the full 5 GB-per-side fixture pair, which on Linux may be a RAM-backed tmpfs), a full temp filesystem raising ValueError naming TMPDIR. None of these has a built-in cap: for untrusted input, bound the row count, the changed fraction, and the column widths (key columns for duplicate-heavy data, changed cells — rendered in full — for wide value columns). Figures are the peak resident set of cargo run -p onix-arrow --release --example row_diff_rss, single run, macOS on an Apple M-series laptop, 2026-09-05, same method as Performance. See crates/onix-arrow/src/row_diff.rs.
• TableDiff.to_json() is the one member with a built-in cap: it embeds rows_added, rows_removed, cells_changed, and duplicate_keys in full, one JSON object per row, so it refuses with ValueError — naming the row count and the cap — once those four together hold more than 10,000 rows. The cap bounds row count only, the same way the row-diff cost bullet above states its per-cell term as changed cells times cell width, not column count or cell width — so a table under the row cap but with wide or large cells can still be large; use the Arrow-returning accessors instead. See crates/onix-arrow/src/json_rows.rs.
• diff_tables compares scalar columns by value (hashed: null, booleans, every integer, float and decimal width, strings and binary in every encoding, timestamps, dates, times, durations, intervals, and dictionaries of these; refused with ValueError: run-end encoded columns and any type-and-unit combination Arrow itself cannot build; nested non-key columns skipped, nested key columns refused), with the exact enumeration in the module doc of crates/onix-arrow/src/row_diff.rs. It also refuses a key column whose type differs across the two inputs after encoding normalization (the conservative choice: a primary key that changed type is refused rather than guessed, not coerced). A row whose only difference is a lossless type change — an Int32 widened to Int64, a timestamp unit change at the same instant — hashes equal on both sides, so it is in neither rows_changed nor cells_changed; the column's type change is still reported in schema.
• Output is byte-identical to DeepDiff except for the cases listed above and two path-rendering quirks; tests/golden/README.md enumerates every accepted exception, including integers past 2^53 (the limit of exact f64 representation) inside ordered scalar lists and ignore_order pairing among naive datetimes, which DeepDiff ranks using the process's local timezone while onix reads a naive value as UTC everywhere.

Contributing

Issues and pull requests are welcome. Open an issue to report a bug, a DeepDiff divergence (include both inputs and the report each engine produces), or a question. Building from source, the quality gates, the golden corpus, benchmarking, mutation testing, and publishing are all in CONTRIBUTING.md.

License

MIT: see LICENSE.

onix reimplements algorithms from CPython's difflib (PSF License) and reproduces the behavior of DeepDiff (MIT); their notices and license texts are in THIRD-PARTY-NOTICES.md.