vizdataquality

This is a Python package for visualizing data quality, and has two main parts. One is software that helps you comprehensively profile and investigate data quality using this workflow from the 6-step Data Quality Method:

1. Is anything obviously wrong (look at your data and any documentation)?
2. Watch out for special values
3. Is any data missing?
4. Check each variable
5. Check combinations of variables
6. Profile the cleaned data

The other is software for investigating patterns and structures of missing values in your data. When a given pattern of missing values has been found to be associated with other factors or attributes of the data then it becomes a "structure of missingness". Patterns and structures of missing values are part of Step 5 of the workflow, because they involve combinations of variables.

Documentation

The vizdataquality documentation is hosted on Read the Docs.

Installation

We recommend installing vizdataquality in a python virtual environment or Conda environment.

To install vizdataquality, most users should run:

pip install 'vizdataquality'

Tutorials

The package includes notebooks that show you how to:

• Calculate a set of data quality attributes and output them to a file
• Use each type of plot, e.g., datetime value distribution
• Create a report while you investigate data quality and profile a dataset
• Apply the six-step workflow to an open parking fines dataset
• Investigate missing data patternsin the UCI diabetes dataset and read the corresponding PLOS One paper
• Apply the 6-step Data Quality Method:
  • YouTube video that provides an overview of the method
  • Step 1 notebook and corresponding Step 1 video
  • Step 2 notebook and corresponding Step 2 video
  • Step 3 notebook and corresponding Step 3 video
  • Step 4 notebook and corresponding Step 4 video
  • Step 5 notebook and corresponding Step 5 video
  • Step 6 notebook and corresponding Step 6 video

After installing vizdataquality, to follow theses tutorials interactively you will need to clone or download this repository. Then start jupyter from within it:

python -m jupyter notebook notebooks

Development

• Documentation is built on readthedocs.com from main branch
• PyPi pulls on creating a release on project repository on GitHub.

Notice

The vizdataquality software is released under the Apache Licence, version 2.0. See LICENCE for details.

The file missing_data_functions.py contains some code that has been derived from setvis, which uses the same licence as vizdataquality. The same person leads the development of both packages.

Acknowledgements

The development of the vizdataquality software was supported by funding from the Engineering and Physical Sciences Research Council (EP/N013980/1; EP/R511717/1; EP/X029689/1), the Alan Turing Institute and Higher Education Innovation Funding (HEIF) from UK Research and Innovation (UKRI).