Skip to main content

diffolars

A small Polars-based toolkit for comparing two versions of a dataframe and generating randomized test data to exercise that comparison.

Ideally used to compare dataloads in the day-to-day of a database analyst.

GitHub: https://github.com/ko222uky/diffolars

Installation

uv add diffolars

Generating test data

diffolars.demo generates a random dataframe and a mutated copy of it, useful for testing diff logic without hand-crafting fixtures.

from diffolars.demo import get_df_pair

pair = get_df_pair(
    n_rows=100,
    n_cols=10,
    n_new_rows=5,    # rows added in the mutated copy
    n_new_cols=2,    # columns added in the mutated copy
    coverage=0.1,    # fraction of existing cells randomly changed
    seed=42,
)

original = pair["original"]
mutated = pair["mutated"]

Every generated row gets a record_id UUID column, used to match rows between the original and mutated dataframes. get_random_data and get_mutated_data are also available individually if you want to generate or mutate a dataframe on its own.

Diffing

diffolars.diff provides the comparison API for two dataloads (e.g. a previous load vs. the latest load), and is under active development.

Inputs to these functions can be a list, polars.DataFrame, or polars.LazyFrame.

  • column_intercept / column_symmetric_diff — shared vs. exclusive columns between the two tables
  • row_intercept / row_symmetric_diff — shared vs. exclusive rows, based on a record ID column
  • prune_rows — returns the rows exclusive to each table (i.e. dropped by the join), tagged with which table they came from
  • report_prune — summarizes the pruned rows/columns as a single dict entry, suitable for logging
  • get_core — prunes each table down to their shared rows and columns, and returns them as a pair of _A/_B-suffixed DataFrames ready for a field-to-field comparison
  • bitdiff — joins the two core tables and computes a per-row diff_bitarray (pl.UInt64), with each bit flagging whether a given column matched between the previous and latest load. Since each row's diff is packed into a single 64-bit integer, the core tables can have at most 64 non-ID columns; compute_bitarray64 (the per-row helper bitdiff calls) raises a ValueError if that limit is exceeded
  • bitdiff_summary — reads back a bitdiff result and reports, per core column, how many rows were modified vs. not modified
  • bitarray_upset_plot / bitdiff_plot — builds an upset plot (matplotlib) showing which columns tend to be modified together, with an optional top_n to limit the plot to the most-frequently-modified columns and a left-hand histogram of each column's total modification count

Currently only the Windows build is available.

Command-line interface

diffolars.cli exposes diff_cli, a Click command that runs the diff pipeline (report_prune, pruned_rows, bitdiff) over two parquet dataloads and prints the three result tables. It's registered as the diffolars console script.

From within this project:

uv run diffolars \
  --prev-load original.parquet \
  --latest-load mutated.parquet \
  --id-col record_id

Or by using uvx:

uvx diffolars \
  --prev-load original.parquet \
  --latest-load mutated.parquet \
  --id-col record_id
Option Default Description
--prev-load original.parquet Path to the previous/original data load.
--latest-load mutated.parquet Path to the latest/mutated data load.
--id-col record_id Name of the record identifier column.
--scan / --no-scan --scan Read with pl.scan_parquet (lazy) instead of pl.read_parquet (eager).
--write / --no-write --write Write the resulting diff tables to parquet.
--bitarray-summary / --no-bitarray-summary --bitarray-summary Produce a per-column modified/not-modified summary and upset plot after the bitdiff is computed.
--top-n 20 Limit the upset plot to the top N most-frequently-modified columns.

When --write is set (the default), results are saved under data/<prev-stem>-<latest-stem>/<today's date>/, as diff_activity_log_record.parquet, diff_record_differences.parquet, and diff_bitarray_results.parquet. If --bitarray-summary is also set, this directory additionally gets bitarray_summary.parquet and bitarray_summary_upsetplot.png.

API reference

Static API docs are generated with pdoc from the package's docstrings, and are hosted at ko222uky.github.io/diffolars (source lives under docs). Regenerate them after docstring changes with:

uv run --group dev pdoc diffolars -o docs

Testing

Tests performed with pytest and coverage:

 uv run coverage run -m pytest --junitxml=report.xml ;
 uv run coverage html

Test result data is created in data/prev-latest.

License

Dual-licensed under either of

at your option.

Metadata

Release files for diffolars 1.1.2

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for diffolars 1.1.2
File Size Uploaded Provenance Actions
diffolars-1.1.2.tar.gz 18.6 kB Without provenance Details for diffolars-1.1.2.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for diffolars 1.1.2
File Interpreter ABI Platform Actions
diffolars-1.1.2-py3-none-any.whl Python 3 none any Details for diffolars-1.1.2-py3-none-any.whl

Total release size: 40.1 kB

Release files / diffolars-1.1.2.tar.gz

Download URL diffolars-1.1.2.tar.gz
Size 18.6 kB
Tags Source
SHA-256 checksum
How to use checksums
ccc0e10d555b83418708e78a21dc9b1b3f443d15c31b2792b98e3548d826dc77
BLAKE2b-256 checksum
How to use checksums
1f34fe06f18b38af3ed90b6acecfe191b6262e7e3d75f3a86648ad42a59d9e67
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via uv/0.11.31 {"installer":{"name":"uv","version":"0.11.31","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"Ubuntu","version":"24.04","id":"noble","libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":true}

Release files / diffolars-1.1.2-py3-none-any.whl

Download URL diffolars-1.1.2-py3-none-any.whl
Size 21.5 kB
Tags Python 3
SHA-256 checksum
How to use checksums
3a6c8bde8ad2347c8abab1359caa96963a46bb5f0742c424d6796fffa85ea86b
BLAKE2b-256 checksum
How to use checksums
6a797b4a0e62668f399c5a50bbc6d2b15b9547540a8bddad74041b3ba991503a
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via uv/0.11.31 {"installer":{"name":"uv","version":"0.11.31","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"Ubuntu","version":"24.04","id":"noble","libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":true}

Security for diffolars 1.1.2

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

No provenance found

None of the files in this release were uploaded with provenance metadata.

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page