Dataframe comparison in Python
Project description
DataComPy
DataComPy is a package to compare two Pandas DataFrames. Originally started to
be something of a replacement for SAS's PROC COMPARE
for Pandas DataFrames
with some more functionality than just Pandas.DataFrame.equals(Pandas.DataFrame)
(in that it prints out some stats, and lets you tweak how accurate matches have to be).
Then extended to carry that functionality over to Spark Dataframes.
Quick Installation
pip install datacompy
or
conda install datacompy
Installing extras
If you would like to use Spark or any other backends please make sure you install via extras:
pip install datacompy[spark]
pip install datacompy[dask]
pip install datacompy[duckdb]
pip install datacompy[polars]
pip install datacompy[ray]
Legacy Spark Deprecation
Starting with version 0.12.0
The original SparkCompare
implementation differs from all the other native implementations. To align the API better, and keep behaviour consistent we are deprecating SparkCompare
into a new module LegacySparkCompare
If you wish to use the old SparkCompare moving forward you can
import datacompy.legacy.LegacySparkCompare
Supported versions and dependncies
Different versions of Spark, Pandas, and Python interact differently. Below is a matrix of what we test with. With the move to Pandas on Spark API and compatability issues with Pandas 2+ we will for the mean time note support Pandas 2 with the Pandas on Spark implementation. Spark plans to support Pandas 2 in Spark 4
With version 0.12.0
:
- Not support Pandas
2.0.0
For the native Spark implemention - Spark
3.1
support will be dropped - Python
3.8
support is dropped
Spark 3.2.4 | Spark 3.3.4 | Spark 3.4.2 | Spark 3.5.1 | |
---|---|---|---|---|
Python 3.9 | ✅ | ✅ | ✅ | ✅ |
Python 3.10 | ✅ | ✅ | ✅ | ✅ |
Python 3.11 | ❌ | ❌ | ✅ | ✅ |
Python 3.12 | ❌ | ❌ | ❌ | ❌ |
Pandas < 1.5.3 | Pandas >=2.0.0 | |
---|---|---|
Native Pandas | ✅ | ✅ |
Native Spark | ✅ | ❌ |
Fugue | ✅ | ✅ |
[!NOTE] At the current time Python
3.12
is not supported by Spark and also Ray within Fugue.
Supported backends
- Pandas: (See documentation)
- Spark (Pandas on Spark API): (See documentation)
- Polars (Experimental): (See documentation)
- Fugue is a Python library that provides a unified interface for data processing on Pandas, DuckDB, Polars, Arrow, Spark, Dask, Ray, and many other backends. DataComPy integrates with Fugue to provide a simple way to compare data across these backends. Please note that Fugue will use the Pandas (Native) logic at its lowest level (See documentation)
Contributors
We welcome and appreciate your contributions! Before we can accept any contributions, we ask that you please be sure to sign the Contributor License Agreement (CLA).
This project adheres to the Open Source Code of Conduct. By participating, you are expected to honor this code.
Roadmap
Roadmap details can be found here
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Hashes for datacompy-0.12.0-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 5a7bb313bcb273bfb6b71d036d140de0006af0cda11344fc2d9b6349ef2e262c |
|
MD5 | 2568844d3696841759c88e2ef1769102 |
|
BLAKE2b-256 | 89ea808a9e8f61f2be97fe14967af5834c23cdb6dda87df69c6f26921a78130f |