dference 0.1.0

Compare two DataFrames by key and explore every difference – in an interactive widget for marimo and Jupyter, or as plain polars tables.

Rows are matched on a key (one or several columns) and classified as

The widget is modelled on marimo.ui.table: paging, sorting, search, a status filter, marimo-style column filters, per-column match rates, a side-by-side detail view, invisible-character highlighting and CSV export. It stays fast on millions of rows because filtering, sorting and paging run in polars on the Python side – the browser only ever receives the visible page.

Installation

uv add dference            # or: pip install dference
uv add "dference[pandas]"  # if your inputs are pandas DataFrames

dference depends on polars, anywidget and traitlets. Inputs can be polars (eager or lazy), pandas, pyarrow, or anything implementing the Arrow PyCapsule interface or to_polars() (DuckDB relations, narwhals, …).

Quick start

marimo

import marimo as mo
from dference import DataFrameDiff

diff = DataFrameDiff(crm, erp, key="customer_id", left_name="CRM", right_name="ERP")
view = mo.ui.anywidget(diff)
view

In another cell, work with the rows checked in the widget. The cell re-runs when the selection changes – and only then; paging, sorting and filtering never trigger re-execution:

view.value["selected_ids"]  # reactive dependency
diff.selected_frame()  # the checked rows as a polars DataFrame

Jupyter

from dference import DataFrameDiff

DataFrameDiff(crm, erp, key=["region", "customer_id"])

Without a UI

import dference

result = dference.compare(crm, erp, key="customer_id", left_name="CRM", right_name="ERP")

result.summary  # Summary(equal=158, mismatch=78, missing_left=10, missing_right=8, ...)
result.frame()  # wide: key, status, differing, "<col> [CRM]", "<col> [ERP]", ...
result.frame("mismatch")
result.mismatches()  # long: one row per (key, column) that differs
result.column_stats()  # per column: dtypes, mismatches, equal/mismatch share

How values are compared

• Exact equality. There is deliberately no numeric tolerance. If small float differences should not count, round both frames first, e.g. df.with_columns(cs.float().round(2)).
• Missing values are equal to each other: null == null, and in float columns NaN counts as missing too (pandas uses NaN as its null marker, so a frame from pandas and one from Arrow would otherwise disagree).
• Same column, same dtype. By default (strict=True) a column must have the identical dtype on both sides.
• strict=False aligns differences that lose nothing – the same values stored differently: integers of different width, Float32 vs. Float64, decimals of different precision, Datetime / Duration in different time units (same time zone), text vs. Categorical / Enum, and an all-null column vs. anything. ColumnInfo.compare_dtype tells you which dtype was used.
• Everything else is always your decision. Int vs. float, Date vs. Datetime, naive vs. time-zone aware, different time zones, text vs. numbers, different nested types: compare() raises one ValueError that lists every such column (keys included) with a cast to fix it, e.g. right = right.with_columns(pl.col("qty").cast(pl.Int64)). Note that pandas has no date dtype – a Date column from pandas arrives as a Datetime.
• Keys must be unique on each side; null keys match null keys.
• Columns present on one side only are shown but not compared. Use ignore_columns=[...] to leave columns out entirely.

Widget features

• Status overview on a shared scale (equal / mismatch / only left / only right, plus found vs. not found); filter by status from the menu in the status column header.
• Each side has one colour and one chip – the first letter of its name (left_short= / right_short= to override) – used everywhere: differences show both values with equal weight, each marked by its side's chip, and rows on one side only carry the chip of the side they exist on.
• Per-column match rate in the header; click ≠ or = to show only rows that differ in, or are equal in, that column.
• Column menu per header: sort and type-aware filters (text, number range, date range, boolean, null checks). For compared columns a row matches if the left or the right value matches; a side that does not exist in a row never matches.
• Invisible characters are made visible: ␣ leading/trailing/repeated spaces, → tab, ↵ line feed, ⍽ no-break spaces, ZWSP, BOM, … Cells that differ only in invisible characters are flagged.
• Side-by-side detail view with numeric deltas; step through the filtered rows across page boundaries.
• CSV export of all rows matching the current filters.

Python API

Performance

The comparison is one hash join plus vectorised expressions; the widget keeps the ordered row ids of the last few views in a small cache, so paging through a view is a slice. Timings from benchmarks/bench.py on a single CPU core (polars parallelises, so more cores are faster):

Memory: the joined frame holds both sides once, plus one boolean flag per compared column.

Limitations

• The widget needs a running Python kernel; in a static HTML export it shows no rows.
• CSV export sends all matching rows to the browser in one message – filter first for very large exports, or use view_frame().write_csv(...).
• view_frame() reflects the filters last sent by the widget but is not a reactive value in marimo.

Development

git clone https://github.com/oberbichler/dference && cd dference
uv sync                                # creates .venv with all dev tools

uv run ruff format .                   # format
uv run ruff check .                    # lint
uv run ty check                        # type check
uv run pytest --cov                    # tests with coverage
uv run playwright install chromium     # once: browser for the widget tests
uv run pytest tests/e2e                # widget in a real browser (skipped without Chromium)

uv run marimo edit examples/demo.py    # interactive demo
uv run python benchmarks/bench.py 1_000_000

The frontend is a single dependency-free ES module (src/dference/static/widget.js) with its stylesheet next to it – no build step. anywidget reloads both on change while you develop.

Node.js is only needed for frontend tooling, never at runtime:

npm ci                                 # Biome + TypeScript (dev only)

npm run fix                            # format + auto-fix JS, CSS and JSON (Biome)
npm run check                          # lint (Biome) and type-check the JSDoc (tsc)

Types live in JSDoc comments and are checked with tsc --checkJs in strict mode (see tsconfig.json); the widget API is typed via @anywidget/types. Nothing is compiled – the file that ships is the file you edit.

Releasing

The version is derived from the git tag (v1.2.3) by hatch-vcs – there is no version to bump in pyproject.toml or uv.lock. A checkout between releases reports a dev version such as 0.1.1.dev3+g94570d5.

For every release:

1. Rename ## [Unreleased] in CHANGELOG.md to ## [x.y.z] - <date> (add a new empty Unreleased section above it) and merge that to main.
2. Run Actions → Release → Run workflow on main with the version x.y.z, or gh workflow run release.yml -f version=x.y.z.

The workflow refuses versions that already exist as tag, release or on PyPI, runs the full CI on the commit, builds with that version, publishes to PyPI via trusted publishing, and only then tags the commit vx.y.z and creates the GitHub release with the changelog section as notes and the wheel and sdist attached. If a step fails, Re-run failed jobs continues from there.

Once, before the first release: on PyPI add a pending trusted publisher for the project dference – owner oberbichler, repository dference, workflow release.yml, environment pypi – and create the environment pypi in the GitHub repository settings (optionally with required reviewers).

License

ISC © Thomas Oberbichler