Skip to main content

Test driven data-wrangling and data validation.

Project description

Current Build Status Development Status Apache 2.0 License Supported Python Versions

Datatest helps speed up and formalize data-wrangling and data validation tasks. It repurposes software testing practices for data preparation and quality assurance projects. Datatest can help you:

  • Clean and wrangle data faster and more accurately.

  • Maintain a record of checks and decisions regarding important data sets.

  • Distinguish between ideal criteria and acceptible deviation.

  • Measure progress of data preparation tasks.

  • On-board new team members with an explicit and structured process.

  • Test data pipeline components and end-to-end behavior.

Datatest supports both pytest and unittest style testing. It implements a system of validation methods, difference classes, and acceptance context managers.

Datatest has no hard dependencies; supports Python 2.6, 2.7, 3.1 through 3.8, PyPy, and PyPy3; and is freely available under the Apache License, version 2.

Documentation:
Official:

Installation

The easiest way to install datatest is to use pip:

pip install datatest

To upgrade an existing installation, use the “--upgrade” option:

pip install --upgrade datatest

Stuntman Mike

If you need bug-fixes or features that are not available in the current stable release, you can “pip install” the development version directly from GitHub:

pip install --upgrade https://github.com/shawnbrown/datatest/archive/master.zip

All of the usual caveats for a development install should apply—only use this version if you can risk some instability or if you know exactly what you’re doing. While care is taken to never break the build, it can happen.

Safety-first Clyde

If you need to review and test packages before installing, you can install datatest manually.

Download the latest source distribution from the Python Package Index (PyPI):

https://pypi.org/project/datatest/ (navigate to “Download files”)

Unpack the file (replacing X.Y.Z with the appropriate version number) and review the source code:

tar xvfz datatest-X.Y.Z.tar.gz

Change to the unpacked directory and run the tests:

cd datatest-X.Y.Z
python setup.py test

Don’t worry if some of the tests are skipped. Tests for optional data sources (like pandas DataFrames or MS Excel files) are skipped when the related third-party packages are not installed.

If the source code and test results are satisfactory, install the package:

python setup.py install

Supported Versions

Tested on Python 2.6, 2.7, 3.1 through 3.8, PyPy, and PyPy3. Datatest is pure Python and may also run on other implementations as well (check using “setup.py test” before installing).

Backward Compatibility

If you have existing tests that use API features which have changed since 0.8.0, you can still run your old code by adding the following import to the beginning of each file:

from datatest.__past__ import api08

To maintain existing test code, this project makes a best-effort attempt to provide backward compatibility support for older features. The API will be improved in the future but only in measured and sustainable ways.

All of the data used at the National Committee for an Effective Congress has been checked with datatest for several years so there is, already, a large and growing codebase that relies on current features and must be maintained into the future.

Soft Dependencies

There are no hard, third-party dependencies. But if you want to interface with pandas DataFrames, MS Excel workbooks, or other optional data sources, you will need to install the relevant packages (pandas, xlrd, etc.).

Older Pythons (3.1 and 2.6)

While datatest supports Python 3.1 and 2.6, some earlier builds of these versions were bundled with an older version of SQLite that is not compatible with datatest. The sqlite3 package is part of the Python Standard Library and some features of datatest use it for internal data handling—though users never need to use the package directly.

If you must use one of these older Python versions and you are experiencing issues, it is recommended that you upgrade to the latest patch release (currently Python 3.1.5 or Python 2.6.9).

Development Repository

The development repository for datatest is hosted on GitHub.


Freely licensed under the Apache License, Version 2.0

Copyright 2014 - 2019 National Committee for an Effective Congress, et al.

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

datatest-0.9.6.tar.gz (102.2 kB view details)

Uploaded Source

Built Distribution

datatest-0.9.6-py2.py3-none-any.whl (115.0 kB view details)

Uploaded Python 2Python 3

File details

Details for the file datatest-0.9.6.tar.gz.

File metadata

  • Download URL: datatest-0.9.6.tar.gz
  • Upload date:
  • Size: 102.2 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/1.13.0 pkginfo/1.5.0.1 requests/2.22.0 setuptools/40.8.0 requests-toolbelt/0.9.1 tqdm/4.32.1 CPython/3.7.3

File hashes

Hashes for datatest-0.9.6.tar.gz
Algorithm Hash digest
SHA256 866006f1e869ba3806d852cb0722583f1eea04f79e7732d021059a1be423c9f8
MD5 f4f9a9783aa61dd93bbecf0f1d553ea9
BLAKE2b-256 879ffb5c754b818bc974189df2460de265eab96c50e3ca820a58174aedda30c3

See more details on using hashes here.

File details

Details for the file datatest-0.9.6-py2.py3-none-any.whl.

File metadata

  • Download URL: datatest-0.9.6-py2.py3-none-any.whl
  • Upload date:
  • Size: 115.0 kB
  • Tags: Python 2, Python 3
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/1.13.0 pkginfo/1.5.0.1 requests/2.22.0 setuptools/40.8.0 requests-toolbelt/0.9.1 tqdm/4.32.1 CPython/3.7.3

File hashes

Hashes for datatest-0.9.6-py2.py3-none-any.whl
Algorithm Hash digest
SHA256 bfcd89c434c319c2b0ec2d1157ef352ba0ec77b3482e087903e9bb88dbf82bc0
MD5 034de9ce6c71e9ab9e657492b03d674a
BLAKE2b-256 009a90c9581f16827009ba36385142fd73bf179e16966aeaf75e7ef59ed72e1f

See more details on using hashes here.

Supported by

AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Pingdom Monitoring Sentry Error logging StatusPage Status page