Data Validation Engine

The Data Validation Engine (DVE) is a configuration driven data validation library built and utilised by NHS England. Currently the package has been reverted from v1.0.0 release to a 0.x as we feel the package is not yet mature enough to be considered a 1.0.0 release. So please bear this in mind if reading through the commits and references to a v1+ release when on v0.x.

As mentioned above, the DVE is "configuration driven" which means the majority of development for you as a user will be building a JSON document to describe how the data will be validated. The JSON document is known as a dischema file and example files can be accessed here. If you'd like to learn more about JSON document and how to build one from scratch, then please read the documentation here.

Once a dischema file has been defined, you are ready to use the DVE. The DVE is typically orchestrated based on four key "services". These are...

If you'd like more detailed documentation around these services the please read the extended documentation here.

The DVE has been designed in a way that's modular and can support users who just want to utilise specific "services" from the DVE (i.e. just the file transformation + data contract). Additionally, the DVE is designed to support different backend implementations. As part of the base installation of DVE, you will find backend support for Spark and DuckDB. So, if you need a MySQL backend implementation, you can implement this yourself. Given our organisations requirements, it will be unlikely that we add anymore specific backend implementations into the base package beyond Spark and DuckDB. So, if you are unable to implement this yourself, I would recommend reading the guidance on requesting new features and raising bug reports here.

Additionally, if you'd like to contribute a new backend implementation into the base DVE package, then please look at the Contributing section.

Installation and usage

The DVE is a Python package and can be installed using package managers such as pip. As of the latest release we support Python 3.10 & 3.11, with Spark v3.4 and DuckDB v1.1. In the future we will be looking to upgrade the DVE to working on a higher versions of Python, DuckDB and Spark.

If you're planning to use the Spark backend implementation, you will also need OpenJDK 11 installed.

Python dependencies are listed in pyproject.toml.

To install the DVE package you can simply install using a package manager such as pip.

pip install data-validation-engine

Note - Only versions >=0.6.2 are available on PyPi. For older versions please install directly from the git repo or build from source.

Once you have installed the DVE you are ready to use it. For guidance on how to create your dischema JSON document (configuration), please read the documentation.

Version 0.0.1 does support a working Python 3.7 installation. However, we will not be supporting any issues with that version of the DVE if you choose to use it. Use at your own risk.

Requesting new features and raising bug reports

Before creating new issues, please check to see if the same bug/feature has been created already. Where a duplicate is created, the ticket will be closed and referenced to an existing issue.

If you have spotted a bug with the DVE then please raise an issue here using the "bug template".

If you have feature request then please follow the same process whilst using the "Feature request template".

Upcoming features

Below is a list of features that we would like to implement or have been requested.

If you are interested in getting any of the unreleased features listed above available, then please read the Contributing section and then submit us a pull request.

Contributing

Please see guidance here.

Legal

This codebase is released under the MIT License. This covers both the codebase and any sample code in the documentation.

Any HTML or Markdown documentation is © Crown copyright and available under the terms of the Open Government 3.0 licence.