PUDL Diff: Compare Two Sets of PUDL Outputs

pudl_diff compares two sets of PUDL Parquet outputs, table by table, and reports on what changed: the schema (columns and dtypes), the row counts, and the rows themselves, matched on primary keys where a table has them. It's useful for confirming that a change to PUDL's code, its dependencies, or its raw inputs did, or didn't, change the data, and for seeing how nightly builds and stable releases differ. It reads local directories or remote buckets, and works on tables too big for memory.

It produces:

• a colorized summary in the terminal, one line per table plus totals,
• a JSON report of the whole comparison, described by a published JSON Schema, and
• Parquet files holding the rows that differ, for tables that do.

An existing report can be shown again as the same terminal summary, without comparing anything, with pudl_diff --from-report.

Installation

uv pip install catalystcoop.pudl_diff
# or
pip install catalystcoop.pudl_diff
# or
conda install -c conda-forge catalystcoop.pudl_diff
# or
pixi add catalystcoop.pudl_diff

Python 3.14 or newer is required.

pudl_diff doesn't depend on PUDL itself. If PUDL is installed in the same environment it is used for its metadata and defaults, and otherwise pudl_diff falls back on PUDL's published outputs.

Usage

Compare a table from the last nightly build against your local outputs in $PUDL_OUTPUT/parquet:

pudl_diff out_eia__yearly_generators

Compare every table in two datasets, local or remote, writing the reports to diffs:

pudl_diff --left s3://pudl.catalyst.coop/stable --right ~/my_outputs --output-path diffs

See the documentation for more, or run pudl_diff --help.

Related Projects

• PUDL, the Public Utility Data Liberation Project, whose outputs pudl_diff compares, and whose nightly builds and stable releases are its default inputs.
• PUDL Data Viewer for browsing and querying PUDL's published data.
• PUDL Examples, notebooks that work with PUDL data.
• Catalyst Agent Skills, for helping LLM-based agents work with PUDL data and code.
• FERC XBRL Extractor, which produces some of the data that goes into PUDL.
• All of Catalyst's documentation.

Contributing

Bug reports, questions and pull requests are welcome in the issue tracker. Please follow our Code of Conduct. The changelog lists what has changed, and why.

Development

• Install pixi if you don't already have it.
• Run pixi install to create the development environment, and pixi run prek install to install the pre-commit hooks defined in .pre-commit-config.yaml, using prek as the runner.
• Run git config merge.ours.driver true so the merge=ours rule in .gitattributes (which keeps your side of pixi.lock on conflict) takes effect.
• Run pixi run test, pixi run lint and pixi run format. See AGENTS.md for the rest of the tasks. The repository follows the layout of Catalyst's Python template, cheshire.

About Catalyst Cooperative

Catalyst Cooperative is a small group of data wranglers and policy wonks organized as a worker-owned cooperative consultancy. Our goal is a more just, livable, and sustainable world. We integrate public data and perform custom analyses to inform public policy (Hire us!). Our focus is primarily on mitigating climate change and improving electric utility regulation in the United States.

Contact Us

• For general support, questions, or other conversations around the project that might be of interest to others, check out the GitHub Discussions.
• If you'd like to get occasional updates about our projects sign up for our email list.
• Want to schedule a time to chat with us one-on-one? Join us for Office Hours.
• More info on our website: https://catalyst.coop
• For private communication about the project or to hire us to provide customized data extraction and analysis, you can email the maintainers: pudl@catalyst.coop.