diffolars
A small Polars-based toolkit for comparing two versions of a dataframe and generating randomized test data to exercise that comparison.
Ideally used to compare dataloads in the day-to-day of a database analyst.
GitHub: https://github.com/ko222uky/diffolars
Installation
uv add diffolars
Generating test data
diffolars.demo generates a random dataframe and a mutated copy of it, useful
for testing diff logic without hand-crafting fixtures.
from diffolars.demo import get_df_pair
pair = get_df_pair(
n_rows=100,
n_cols=10,
n_new_rows=5, # rows added in the mutated copy
n_new_cols=2, # columns added in the mutated copy
coverage=0.1, # fraction of existing cells randomly changed
seed=42,
)
original = pair["original"]
mutated = pair["mutated"]
Every generated row gets a record_id UUID column, used to match rows
between the original and mutated dataframes. get_random_data and
get_mutated_data are also available individually if you want to generate or
mutate a dataframe on its own.
Diffing
diffolars.diff provides the comparison API for two dataloads (e.g. a
previous load vs. the latest load), and is under active development.
Inputs to these functions can be a list, polars.DataFrame, or
polars.LazyFrame.
column_intercept/column_symmetric_diff— shared vs. exclusive columns between the two tablesrow_intercept/row_symmetric_diff— shared vs. exclusive rows, based on a record ID columnprune_rows— returns the rows exclusive to each table (i.e. dropped by the join), tagged with which table they came fromreport_prune— summarizes the pruned rows/columns as a single dict entry, suitable for loggingget_core— prunes each table down to their shared rows and columns, and returns them as a pair of_A/_B-suffixed DataFrames ready for a field-to-field comparisonbitdiff— joins the two core tables and computes a per-rowdiff_bitarray(pl.UInt64), with each bit flagging whether a given column matched between the previous and latest load. Since each row's diff is packed into a single 64-bit integer, the core tables can have at most 64 non-ID columns;compute_bitarray64(the per-row helperbitdiffcalls) raises aValueErrorif that limit is exceededbitdiff_summary— reads back abitdiffresult and reports, per core column, how many rows were modified vs. not modifiedbitarray_upset_plot/bitdiff_plot— builds an upset plot (matplotlib) showing which columns tend to be modified together, with an optionaltop_nto limit the plot to the most-frequently-modified columns and a left-hand histogram of each column's total modification count
Currently only the Windows build is available.
Command-line interface
diffolars.cli exposes diff_cli, a Click command that runs the diff
pipeline (report_prune, pruned_rows, bitdiff) over two parquet dataloads
and prints the three result tables. It's registered as the diffolars
console script.
From within this project:
uv run diffolars \
--prev-load original.parquet \
--latest-load mutated.parquet \
--id-col record_id
Or by using uvx:
uvx diffolars \
--prev-load original.parquet \
--latest-load mutated.parquet \
--id-col record_id
| Option | Default | Description |
|---|---|---|
--prev-load |
original.parquet |
Path to the previous/original data load. |
--latest-load |
mutated.parquet |
Path to the latest/mutated data load. |
--id-col |
record_id |
Name of the record identifier column. |
--scan / --no-scan |
--scan |
Read with pl.scan_parquet (lazy) instead of pl.read_parquet (eager). |
--write / --no-write |
--write |
Write the resulting diff tables to parquet. |
--bitarray-summary / --no-bitarray-summary |
--bitarray-summary |
Produce a per-column modified/not-modified summary and upset plot after the bitdiff is computed. |
--top-n |
20 |
Limit the upset plot to the top N most-frequently-modified columns. |
When --write is set (the default), results are saved under
data/<prev-stem>-<latest-stem>/<today's date>/, as
diff_activity_log_record.parquet, diff_record_differences.parquet, and
diff_bitarray_results.parquet. If --bitarray-summary is also set, this
directory additionally gets bitarray_summary.parquet and
bitarray_summary_upsetplot.png.
API reference
Static API docs are generated with pdoc from the package's docstrings, and are hosted at ko222uky.github.io/diffolars (source lives under docs). Regenerate them after docstring changes with:
uv run --group dev pdoc diffolars -o docs
Testing
Tests performed with pytest and coverage:
uv run coverage run -m pytest --junitxml=report.xml ;
uv run coverage html
Test result data is created in data/prev-latest.
License
Dual-licensed under either of
- MIT license (LICENSE-MIT)
- Apache License, Version 2.0 (LICENSE-APACHE)
at your option.
Metadata
Release files for diffolars 1.1.2
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | Actions |
|---|---|---|---|---|
| diffolars-1.1.2.tar.gz | 18.6 kB | Without provenance | Details for diffolars-1.1.2.tar.gz |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Actions Reset |
|---|---|---|---|---|
| diffolars-1.1.2-py3-none-any.whl | Python 3 | none | any | Details for diffolars-1.1.2-py3-none-any.whl |
Total release size: 40.1 kB
Release files / diffolars-1.1.2.tar.gz
| Download URL | diffolars-1.1.2.tar.gz |
|---|---|
| Size | 18.6 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
ccc0e10d555b83418708e78a21dc9b1b3f443d15c31b2792b98e3548d826dc77
|
|
BLAKE2b-256 checksum How to use checksums |
1f34fe06f18b38af3ed90b6acecfe191b6262e7e3d75f3a86648ad42a59d9e67
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
uv/0.11.31 {"installer":{"name":"uv","version":"0.11.31","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"Ubuntu","version":"24.04","id":"noble","libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":true}
|
Release files / diffolars-1.1.2-py3-none-any.whl
| Download URL | diffolars-1.1.2-py3-none-any.whl |
|---|---|
| Size | 21.5 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
3a6c8bde8ad2347c8abab1359caa96963a46bb5f0742c424d6796fffa85ea86b
|
|
BLAKE2b-256 checksum How to use checksums |
6a797b4a0e62668f399c5a50bbc6d2b15b9547540a8bddad74041b3ba991503a
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
uv/0.11.31 {"installer":{"name":"uv","version":"0.11.31","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"Ubuntu","version":"24.04","id":"noble","libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":true}
|
Security for diffolars 1.1.2
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
No provenance found
None of the files in this release were uploaded with provenance metadata.
A note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.