DataComPy
DataComPy is a package to compare two Pandas DataFrames. Originally started to
be something of a replacement for SAS's PROC COMPARE for Pandas DataFrames
with some more functionality than just Pandas.DataFrame.equals(Pandas.DataFrame)
(in that it prints out some stats, and lets you tweak how accurate matches have to be).
Then extended to carry that functionality over to Spark Dataframes.
Quick Installation
pip install datacompy
or
conda install datacompy
Installing extras
If you would like to use Spark or any other backends please make sure you install via extras:
pip install datacompy[spark]
pip install datacompy[dask]
pip install datacompy[duckdb]
pip install datacompy[polars]
pip install datacompy[ray]
In-scope Spark versions
Different versions of Spark play nicely with only certain versions of Python below is a matrix of what we test with
| Spark 3.1.3 | Spark 3.2.3 | Spark 3.3.4 | Spark 3.4.2 | Spark 3.5.0 | |
|---|---|---|---|---|---|
| Python 3.8 | ✅ | ✅ | ✅ | ✅ | ✅ |
| Python 3.9 | ✅ | ✅ | ✅ | ✅ | ✅ |
| Python 3.10 | ✅ | ✅ | ✅ | ✅ | ✅ |
| Python 3.11 | ❌ | ❌ | ❌ | ✅ | ✅ |
| Python 3.12 | ❌ | ❌ | ❌ | ❌ | ❌ |
:::{note}
At the current time Python 3.12 is not supported by Spark and also Ray within Fugue.
:::
Supported backends
- Pandas: (See documentation)
- Spark: (See documentation)
- Polars (Experimental): (See documentation)
- Fugue is a Python library that provides a unified interface for data processing on Pandas, DuckDB, Polars, Arrow, Spark, Dask, Ray, and many other backends. DataComPy integrates with Fugue to provide a simple way to compare data across these backends. Please note that Fugue will use the Pandas (Native) logic at its lowest level (See documentation)
Contributors
We welcome and appreciate your contributions! Before we can accept any contributions, we ask that you please be sure to sign the Contributor License Agreement (CLA).
This project adheres to the Open Source Code of Conduct. By participating, you are expected to honor this code.
Roadmap
Roadmap details can be found here
Metadata
Release files for datacompy 0.11.1
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | Actions |
|---|---|---|---|---|
| datacompy-0.11.1.tar.gz | 56.8 kB | Without provenance | Details for datacompy-0.11.1.tar.gz |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Actions Reset |
|---|---|---|---|---|
| datacompy-0.11.1-py3-none-any.whl | Python 3 | none | any | Details for datacompy-0.11.1-py3-none-any.whl |
Total release size: 98.4 kB
Release files / datacompy-0.11.1.tar.gz
| Download URL | datacompy-0.11.1.tar.gz |
|---|---|
| Size | 56.8 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
cfb5e871a7f4a8d9dec70f8e9e0f331d13b16183624b4960dd425ac6f1a0338e
|
|
BLAKE2b-256 checksum How to use checksums |
5b2d70fc642bc93b557b5265d8ba4b2336bf03e22550da54dd6289820d4c9a8f
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/5.0.0 CPython/3.9.18
|
Release files / datacompy-0.11.1-py3-none-any.whl
| Download URL | datacompy-0.11.1-py3-none-any.whl |
|---|---|
| Size | 41.6 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
0ec8c7272cec4c2731e7a502936cad6a7ae611b28d3454dd57696ba6e9926215
|
|
BLAKE2b-256 checksum How to use checksums |
d41879adc5c43cd4cb49471c122b5504019bd1f50077cd578eb7ec4ad3e363a2
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/5.0.0 CPython/3.9.18
|
Security for datacompy 0.11.1
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
No provenance found
None of the files in this release were uploaded with provenance metadata.
A note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.