DataComPy
DataComPy is a package to compare two DataFrames (or tables) such as Pandas, Spark, Polars, and
even Snowflake. Originally it was created to be something of a replacement
for SAS's PROC COMPARE for Pandas DataFrames with some more functionality than
just Pandas.DataFrame.equals(Pandas.DataFrame) (in that it prints out some stats,
and lets you tweak how accurate matches have to be). Supported types include:
- Pandas
- Polars
- Spark
- Snowflake (via snowpark)
- Dask (via Fugue)
- DuckDB (via Fugue)
[!IMPORTANT] datacompy is progressing towards a
v1release. During this transition, asupport/0.19.xbranch will be maintained solely forv0.19.xusers. This branch will only receive dependency updates and critical bug fixes; no new features will be added. All new feature development should target thev1branches (developand eventuallymain).
Quick Installation
pip install datacompy
or
conda install datacompy
Installing extras
If you would like to use Spark or any other backends please make sure you install via extras:
pip install datacompy[spark]
pip install datacompy[fugue]
pip install datacompy[snowflake]
LegacySparkCompare and SparkPandasCompare removal
Starting with v0.17.0, both LegacySparkCompare and SparkPandasCompare have been removed.
Supported versions and dependencies
Different versions of Spark, Pandas, and Python interact differently. Below is a matrix of what we test with. With the move to Pandas on Spark API and compatability issues with Pandas 2+ we will for the mean time note support Pandas 2 with the Pandas on Spark implementation. Spark plans to support Pandas 2 in Spark 4
| Spark 3.4.4 | Spark 3.5.6 | |
|---|---|---|
| Python 3.10 | ✅ | ✅ |
| Python 3.11 | ✅ | ✅ |
| Python 3.12 | ❌ | ❌ |
| Pandas < 1.5.3 | Pandas >=2.0.0 | |
|---|---|---|
Compare |
✅ | ✅ |
SparkSQLCompare |
✅ | ✅ |
| Fugue | ✅ | ✅ |
[!NOTE] At the current time Python
3.12is not supported by Spark and also Ray within Fugue. If you are using Python3.12and above, please note that not all functioanlity will be supported. Pandas and Polars support should work fine and are tested.
Supported backends
- Pandas: (See documentation)
- Spark: (See documentation)
- Polars: (See documentation)
- Snowflake/Snowpark: (See documentation)
- Fugue is a Python library that provides a unified interface for data processing on Pandas, DuckDB, Polars, Arrow, Spark, Dask, Ray, and many other backends. DataComPy integrates with Fugue to provide a simple way to compare data across these backends. Please note that Fugue will use the Pandas (Native) logic at its lowest level (See documentation)
Contributors
We welcome and appreciate your contributions! Before we can accept any contributions, we ask that you please be sure to sign the Contributor License Agreement (CLA).
This project adheres to the Open Source Code of Conduct. By participating, you are expected to honor this code.
Roadmap
Roadmap details can be found here
1 maintainer
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Filter files by name, interpreter, ABI, and platform.
If you're not sure about the file name format, learn more about wheel file names.
Copy a direct link to the current filters
File details
Details for the file datacompy-0.19.4.tar.gz.
File metadata
- Download URL: datacompy-0.19.4.tar.gz
- Upload date:
- Size: 93.2 kB
- Tags: Source
- Uploaded using Trusted Publishing? Yes
- Uploaded via: twine/6.1.0 CPython/3.13.7
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 |
687bac4e3ea385e150288a387a0d190f6d3278410491f186bf2ef4547cfe40b4
|
|
| MD5 |
a33ee645ed27883eedcfc90d54ed7592
|
|
| BLAKE2b-256 |
d5500e90d602aa0bbfb69997d694e5eac573bb161c29f3cbc21531a6454e466a
|
Provenance
The following attestation bundles were made for datacompy-0.19.4.tar.gz:
Publisher:
publish-package.yml on capitalone/datacompy
-
Statement:
-
Statement type:
https://in-toto.io/Statement/v1 -
Predicate type:
https://docs.pypi.org/attestations/publish/v1 -
Subject name:
datacompy-0.19.4.tar.gz -
Subject digest:
687bac4e3ea385e150288a387a0d190f6d3278410491f186bf2ef4547cfe40b4 - Sigstore transparency entry: 1092871867
- Sigstore integration time:
-
Permalink:
capitalone/datacompy@2bd2088f28786bcce32b4c57ca7474c2a126b0fb -
Branch / Tag:
refs/heads/support/0.19.x - Owner: https://github.com/capitalone
-
Access:
public
-
Token Issuer:
https://token.actions.githubusercontent.com -
Runner Environment:
github-hosted -
Publication workflow:
publish-package.yml@2bd2088f28786bcce32b4c57ca7474c2a126b0fb -
Trigger Event:
workflow_dispatch
-
Statement type:
File details
Details for the file datacompy-0.19.4-py3-none-any.whl.
File metadata
- Download URL: datacompy-0.19.4-py3-none-any.whl
- Upload date:
- Size: 66.8 kB
- Tags: Python 3
- Uploaded using Trusted Publishing? Yes
- Uploaded via: twine/6.1.0 CPython/3.13.7
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 |
dbecefd882019e650bc363d102cdb232d9df39afc717e933098ac2cdc22aa264
|
|
| MD5 |
53a5fad9c075876d939ebcb1a3ab32e6
|
|
| BLAKE2b-256 |
1236a06cbea52c30081549c68597dd60519b96a3ad0b98cb9bcbb74288e7fa52
|
Provenance
The following attestation bundles were made for datacompy-0.19.4-py3-none-any.whl:
Publisher:
publish-package.yml on capitalone/datacompy
-
Statement:
-
Statement type:
https://in-toto.io/Statement/v1 -
Predicate type:
https://docs.pypi.org/attestations/publish/v1 -
Subject name:
datacompy-0.19.4-py3-none-any.whl -
Subject digest:
dbecefd882019e650bc363d102cdb232d9df39afc717e933098ac2cdc22aa264 - Sigstore transparency entry: 1092871895
- Sigstore integration time:
-
Permalink:
capitalone/datacompy@2bd2088f28786bcce32b4c57ca7474c2a126b0fb -
Branch / Tag:
refs/heads/support/0.19.x - Owner: https://github.com/capitalone
-
Access:
public
-
Token Issuer:
https://token.actions.githubusercontent.com -
Runner Environment:
github-hosted -
Publication workflow:
publish-package.yml@2bd2088f28786bcce32b4c57ca7474c2a126b0fb -
Trigger Event:
workflow_dispatch
-
Statement type:
