The Open-source Framework for Precision Data Testing

📊 🔎 ✅

Data validation for scientists, engineers, and analysts seeking correctness.

pandera is a Union.ai open source project that provides a flexible and expressive API for performing data validation on dataframe-like objects to make data processing pipelines more readable and robust.

Dataframes contain information that pandera explicitly validates at runtime. This is useful in production-critical or reproducible research settings. With pandera, you can:

Define a schema once and use it to validate different dataframe types including pandas, polars, dask, modin, and pyspark.
Check the types and properties of columns in a DataFrame or values in a Series.
Perform more complex statistical validation like hypothesis testing.
Parse data to standardize the preprocessing steps needed to produce valid data.
Seamlessly integrate with existing data analysis/processing pipelines via function decorators.
Define dataframe models with the class-based API with pydantic-style syntax and validate dataframes using the typing syntax.
Synthesize data from schema objects for property-based testing with pandas data structures.
Lazily Validate dataframes so that all validation checks are executed before raising an error.
Integrate with a rich ecosystem of python tools like pydantic, fastapi, and mypy.

Documentation

The official documentation is hosted here: https://pandera.readthedocs.io

Install

Using pip:

pip install pandera

Using conda:

conda install -c conda-forge pandera

Extras

Installing additional functionality:

pip

pip install 'pandera[hypotheses]' # hypothesis checks
pip install 'pandera[io]'         # yaml/script schema io utilities
pip install 'pandera[strategies]' # data synthesis strategies
pip install 'pandera[mypy]'       # enable static type-linting of pandas
pip install 'pandera[fastapi]'    # fastapi integration
pip install 'pandera[dask]'       # validate dask dataframes
pip install 'pandera[pyspark]'    # validate pyspark dataframes
pip install 'pandera[modin]'      # validate modin dataframes
pip install 'pandera[modin-ray]'  # validate modin dataframes with ray
pip install 'pandera[modin-dask]' # validate modin dataframes with dask
pip install 'pandera[geopandas]'  # validate geopandas geodataframes
pip install 'pandera[polars]'     # validate polars dataframes

conda

conda install -c conda-forge pandera-hypotheses  # hypothesis checks
conda install -c conda-forge pandera-io          # yaml/script schema io utilities
conda install -c conda-forge pandera-strategies  # data synthesis strategies
conda install -c conda-forge pandera-mypy        # enable static type-linting of pandas
conda install -c conda-forge pandera-fastapi     # fastapi integration
conda install -c conda-forge pandera-dask        # validate dask dataframes
conda install -c conda-forge pandera-pyspark     # validate pyspark dataframes
conda install -c conda-forge pandera-modin       # validate modin dataframes
conda install -c conda-forge pandera-modin-ray   # validate modin dataframes with ray
conda install -c conda-forge pandera-modin-dask  # validate modin dataframes with dask
conda install -c conda-forge pandera-geopandas   # validate geopandas geodataframes
conda install -c conda-forge pandera-polars      # validate polars dataframes

Quick Start

import pandas as pd
import pandera as pa


# data to validate
df = pd.DataFrame({
    "column1": [1, 4, 0, 10, 9],
    "column2": [-1.3, -1.4, -2.9, -10.1, -20.4],
    "column3": ["value_1", "value_2", "value_3", "value_2", "value_1"]
})

# define schema
schema = pa.DataFrameSchema({
    "column1": pa.Column(int, checks=pa.Check.le(10)),
    "column2": pa.Column(float, checks=pa.Check.lt(-1.2)),
    "column3": pa.Column(str, checks=[
        pa.Check.str_startswith("value_"),
        # define custom checks as functions that take a series as input and
        # outputs a boolean or boolean Series
        pa.Check(lambda s: s.str.split("_", expand=True).shape[1] == 2)
    ]),
})

validated_df = schema(df)
print(validated_df)

#     column1  column2  column3
#  0        1     -1.3  value_1
#  1        4     -1.4  value_2
#  2        0     -2.9  value_3
#  3       10    -10.1  value_2
#  4        9    -20.4  value_1

DataFrame Model

pandera also provides an alternative API for expressing schemas inspired by dataclasses and pydantic. The equivalent DataFrameModel for the above DataFrameSchema would be:

from pandera.typing import Series

class Schema(pa.DataFrameModel):

    column1: int = pa.Field(le=10)
    column2: float = pa.Field(lt=-1.2)
    column3: str = pa.Field(str_startswith="value_")

    @pa.check("column3")
    def column_3_check(cls, series: Series[str]) -> Series[bool]:
        """Check that values have two elements after being split with '_'"""
        return series.str.split("_", expand=True).shape[1] == 2

Schema.validate(df)

Development Installation

git clone https://github.com/pandera-dev/pandera.git
cd pandera
export PYTHON_VERSION=...  # specify desired python version
pip install -r dev/requirements-${PYTHON_VERSION}.txt
pip install -e .

Tests

pip install pytest
pytest tests

Contributing to pandera

All contributions, bug reports, bug fixes, documentation improvements, enhancements and ideas are welcome.

A detailed overview on how to contribute can be found in the contributing guide on GitHub.

Issues

Go here to submit feature requests or bugfixes.

Need Help?

There are many ways of getting help with your questions. You can ask a question on Github Discussions page or reach out to the maintainers and pandera community on Discord

Why `pandera`?

dataframe-centric data types, column nullability, and uniqueness are first-class concepts.
Define dataframe models with the class-based API with pydantic-style syntax and validate dataframes using the typing syntax.
check_input and check_output decorators enable seamless integration with existing code.
Checks provide flexibility and performance by providing access to pandas API by design and offers built-in checks for common data tests.
Hypothesis class provides a tidy-first interface for statistical hypothesis testing.
Checks and Hypothesis objects support both tidy and wide data validation.
Use schemas as generative contracts to synthesize data for unit testing.
Schema inference allows you to bootstrap schemas from data.

How to Cite

If you use pandera in the context of academic or industry research, please consider citing the paper and/or software package.

Paper

@InProceedings{ niels_bantilan-proc-scipy-2020,
  author    = { {N}iels {B}antilan },
  title     = { pandera: {S}tatistical {D}ata {V}alidation of {P}andas {D}ataframes },
  booktitle = { {P}roceedings of the 19th {P}ython in {S}cience {C}onference },
  pages     = { 116 - 124 },
  year      = { 2020 },
  editor    = { {M}eghann {A}garwal and {C}hris {C}alloway and {D}illon {N}iederhut and {D}avid {S}hupe },
  doi       = { 10.25080/Majora-342d178e-010 }
}

Software Package

License and Credits

pandera is licensed under the MIT license and is written and maintained by Niels Bantilan (niels@pandera.ci)

Release files for pandera 0.20.0

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for pandera 0.20.0
File	Size	Uploaded
pandera-0.20.0.tar.gz	196.7 kB	Jun 26, 2024	Details

Built distribution (wheel)

Table of built distributions (wheels) for pandera 0.20.0
File	Interpreter	ABI	Platform	Reset
pandera-0.20.0-py3-none-any.whl 253.9 kB Jun 26, 2024	Python 3	none	any	Details

Total release size: 450.5 kB

Release files / pandera-0.20.0.tar.gz

Download URL	pandera-0.20.0.tar.gz
Size	196.7 kB
Tags	Source
SHA-256 checksum How to use checksums	`ee2f53c3f8b8f618d961846abd7c7760ca71df32c1cdeeedefaebc13480a8027`
BLAKE2b-256 checksum How to use checksums	`f1d33eb4a8053db37397cc65bfc1453f6ea5789acc175e41dd0b16e683c92cb8`
Upload date	Jun 26, 2024
Uploaded using Trusted Publishing? What is trusted publishing?	No
Uploaded via	`twine/5.1.1 CPython/3.12.4`

Release files / pandera-0.20.0-py3-none-any.whl

Download URL	pandera-0.20.0-py3-none-any.whl
Size	253.9 kB
Tags	Python 3
SHA-256 checksum How to use checksums	`72ddcfa06945d42140c9c9639aa32bb8f05e3c27c897293613e55b911b485529`
BLAKE2b-256 checksum How to use checksums	`189134bdf22c22b42bd664ccf75aeec8d69adf114720d38e88374d9369799c7c`
Upload date	Jun 26, 2024
Uploaded using Trusted Publishing? What is trusted publishing?	No
Uploaded via	`twine/5.1.1 CPython/3.12.4`