DataComPy

DataComPy is a package to compare two DataFrames (or tables) such as Pandas, Spark, Polars, and even Snowflake. Originally it was created to be something of a replacement for SAS's PROC COMPARE for Pandas DataFrames with some more functionality than just Pandas.DataFrame.equals(Pandas.DataFrame) (in that it prints out some stats, and lets you tweak how accurate matches have to be). Supported types include:

• Pandas
• Polars
• Spark (classic and Spark Connect)
• Snowflake

Quick Installation

pip install datacompy

or

conda install datacompy

Installing extras

If you would like to use Spark or any other backends please make sure you install via extras:

pip install datacompy[spark]
pip install datacompy[snowflake]

Supported backends

• Pandas: (See documentation)
• Spark: (See documentation)
• Polars: (See documentation)
• Snowflake/Snowpark: (See documentation)

Command Line Interface

DataComPy ships a datacompy command, so ad hoc checks and CI pipelines do not need a throw-away script (see documentation):

# Compare two files and print a report
datacompy compare --left before.csv --right after.csv --on id

# Fail a build on any difference, with a JSON report saved as an artifact
datacompy compare \
    --left before.parquet --right after.parquet \
    --on account_id,as_of_date \
    --abs-tol balance=0.01 \
    --max-unequal-rows 0 \
    --report-format json --output reports/diff.json --quiet

It exits 0 when the datasets match, 1 when they differ, and 2 on error. CSV, Parquet, and JSON inputs are supported on the pandas, polars, and Spark backends, including tab separated CSV via a .tsv extension, and Snowflake tables can be compared in place.

Programmatic Report Access

Every compare object exposes build_report_data() which returns a typed ReportData object — useful for dashboards, JSON export, or custom rendering without relying on the string report:

import pandas as pd
from datacompy import PandasCompare

df1 = pd.DataFrame({"id": [1, 2, 3], "val": [10, 20, 30]})
df2 = pd.DataFrame({"id": [1, 2, 3], "val": [10, 99, 30]})

compare = PandasCompare(df1, df2, join_columns="id")

# Access structured data directly
data = compare.build_report_data()
print(data.row_summary.unequal_rows)  # 1
print(data.mismatch_stats.stats[0].column)  # 'val'

# Render / export — methods live on ReportData itself
print(data.render())  # same text as compare.report()
data.save("report.html")  # HTML file
data.to_dict()  # JSON-serializable dict

See the Report API documentation for the full reference.

Contributors

We welcome and appreciate your contributions! Before we can accept any contributions, we ask that you please be sure to sign the Contributor License Agreement (CLA).

To run the CI test matrix locally, including the Spark and Spark Connect tracks without a hand-managed JDK, see the Developer Instructions.

This project adheres to the Open Source Code of Conduct. By participating, you are expected to honor this code.