Skip to main content
Join the official 2019 Python Developers SurveyStart the survey!

A package to validate the schema of a pandas dataframe

Project description

Build Status PyPI version

Opulent-Pandas

Opulent-Pandas is a schema validation packages aimed specifically at validating the schema of pandas dataframes. It takes heavy inspiration from voluptuous, and tries to stay as close as possible to the API defined in this package. Opulent-Pandas is different from voluptuous in that it heavily relies on Pandas to perform the validation. This makes Opulent-Pandas considerably faster than voluptuous on larger datasets. It does, however, mean that the input format is also a Pandas DataFrame, rather than a dict (as is the case for voluptuous) A performance comparison of voluptuous and Opulent-Pandas will be added to this readme soon!

Example

Defining a schema in Opulent-Pandas is very similar to how you would in voluptuous. To make the similarities and differences clear, let's walk through the same example as is done in the voluptuous readme.

Twitter's user search API accepts query URLs like:

$ curl 'https://api.twitter.com/1.1/users/search.json?q=python&per_page=20&page=1'

To validate this we might use a schema like:

>>> from opulent_pandas import Schema, TypeValidator, Required
>>> schema = Schema({
...   Required('q'): [TypeValidator(str)],
...   Required('per_page'): [TypeValidator(int)],
...   Required('page'): [TypeValidator(int)],
... })

Comparing with voluptuous, you'll notice that the validators per field are always specified as a list. Other than that, it's very similar to how you would define the schema with voluptuous

If we look at the more complex schema, as defined in the readme of voluptuous, we see very similar schemas:

>>> from opulent_pandas.validator import Required, RangeValidator, TypeValidator, ValueLengthValidator 
>>> schema = Schema({
...   Required('q'): [TypeValidator(str), ValueLengthValidator(min_length=1)],
...   Required('per_page'): [TypeValidator(int), RangeValidator(min=1, max=20)],
...   Required('page'): [TypeValidator(int), RangeValidator(min=0)],
... })

One difference between Opulent-Pandas and voluptuous is that Opulent-Pandas has a validate function that can be used to validate a given data structure rather tha voluptuous' approach of passing the data directly to your schema as a parameter.

If you pass data in that does not satisfy the requirements specified in your Opulent-Pandas schema, you'll get a corresponding error message. Walking through the examples provided in the voluptuous readme:

There are 3 required fields: TODO: this example should also tell you which columns are missing. Seems to be a bug.

>>> from opulent_pandas import MissingColumnError
>>> try:
...   schema.validate({})
...   raise AssertionError('MissingColumnError not raised')
... except MissingColumnError as e:
...   exc = e
>>> str(exc) == "Columns missing"
True

q must be a string:

>>> from opulent_pandas import InvalidTypeError
>>> try:
...   schema.validate(pd.DataFrame({'q': [123], 'per_page':[10], 'page': [1]})
...   raise AssertionError('InvalidTypeError not raised')
... except InvalidTypeError as e:
...   exc = e
>>> str(exc) == "Invalid data type found for column: q. Required: <class 'str'>"
True

...and must be at least one character in length:

>>> from opulent_pandas import ValueLengthError
>>> try:
...   schema.validate(pd.DataFrame({'q': [''], 'per_page': 5, 'page': 12}))
...   raise AssertionError('ValueLengthError not raised')
... except ValueLengthError as e:
...   exc = e
>>> str(exc) == "Value found with length smaller than enforced minimum length for column: q. Minimum Length: 1"
True

"per_page" is a positive integer no greater than 20:

>>> from opulent_pandas import RangeError
>>> try:
...    schema.validate(pd.DataFrame({'q': ['#topic'], 'per_page': [900], 'page': [12]}))
...    raise AssertionError('RangeError not raised')
... except RangeError as e:
...    exc = e
>>> str(exc) == "Value found larger than enforced maximum for column: per_page. Required maximum: 20"
True

>>> try:
...    schema.validate(pd.DataFrame({'q': ['#topic'], 'per_page': [-10], 'page': [12]}))
...    raise AssertionError('RangeError not raised')
... except RangeError as e:
...    exc = e
>>> str(exc) == "Value found larger than enforced minimum for column: per_page. Required minimum: 1"
True

"page" is an integer >= 0:

>>> try:
...   schema.validate(pd.DataFrame({'q': ['#topic'], 'per_page': ['one']})
...   raise AssertionError('InvalidTypeError not raised')
... except InvalidTypeError as e:
...   exc = e
>>> str(exc) == "Invalid data type found for column: page. Required type: <class 'int'>"
True

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Files for opulent-pandas, version 0.0.4
Filename, size File type Python version Upload date Hashes
Filename, size opulent_pandas-0.0.4-py3-none-any.whl (10.0 kB) File type Wheel Python version py3 Upload date Hashes View hashes
Filename, size opulent-pandas-0.0.4.tar.gz (4.8 kB) File type Source Python version None Upload date Hashes View hashes

Supported by

Elastic Elastic Search Pingdom Pingdom Monitoring Google Google BigQuery Sentry Sentry Error logging AWS AWS Cloud computing DataDog DataDog Monitoring Fastly Fastly CDN SignalFx SignalFx Supporter DigiCert DigiCert EV certificate StatusPage StatusPage Status page