Skip to main content

Dataframe comparison in Python

Project description


PyPI - Python Version Code style: black PyPI version Anaconda-Server Badge PyPI - Downloads

DataComPy is a package to compare two Pandas DataFrames. Originally started to be something of a replacement for SAS's PROC COMPARE for Pandas DataFrames with some more functionality than just Pandas.DataFrame.equals(Pandas.DataFrame) (in that it prints out some stats, and lets you tweak how accurate matches have to be). Then extended to carry that functionality over to Spark Dataframes.

Quick Installation

pip install datacompy


conda install datacompy

Installing extras

If you would like to use Spark or any other backends please make sure you install via extras:

pip install datacompy[spark]
pip install datacompy[dask]
pip install datacompy[duckdb]
pip install datacompy[polars]
pip install datacompy[ray]

In-scope Spark versions

Different versions of Spark play nicely with only certain versions of Python below is a matrix of what we test with

Spark 3.1.3 Spark 3.2.3 Spark 3.3.4 Spark 3.4.2 Spark 3.5.0
Python 3.8
Python 3.9
Python 3.10
Python 3.11
Python 3.12

:::{note} At the current time Python 3.12 is not supported by Spark and also Ray within Fugue. :::

Supported backends

  • Pandas: (See documentation)
  • Spark: (See documentation)
  • Polars (Experimental): (See documentation)
  • Fugue is a Python library that provides a unified interface for data processing on Pandas, DuckDB, Polars, Arrow, Spark, Dask, Ray, and many other backends. DataComPy integrates with Fugue to provide a simple way to compare data across these backends. Please note that Fugue will use the Pandas (Native) logic at its lowest level (See documentation)


We welcome and appreciate your contributions! Before we can accept any contributions, we ask that you please be sure to sign the Contributor License Agreement (CLA).

This project adheres to the Open Source Code of Conduct. By participating, you are expected to honor this code.


Roadmap details can be found here

Project details

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

datacompy-0.11.1.tar.gz (56.8 kB view hashes)

Uploaded Source

Built Distribution

datacompy-0.11.1-py3-none-any.whl (41.6 kB view hashes)

Uploaded Python 3

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page