Skip to main content

Engine for creating and running validation suites for general purposes

Project description

Validations Engine

Engine for creating and running validations for general purposes.

Release Python Version License Code style: black

Source Downloads Page Installation Command
PyPi PyPi Downloads Link pip install validations-engine

Build status

Develop Stable Documentation Sonar
Test Publish Documentation Status Quality Gate Status

Purpose

This automation engine was conceived to simulate key communication/integration lines between services that compose the core jobs of the data engineering pipeline at QuintoAndar, to guarantee there is no failures in the pipeline (and early catch eventual failures). But it can execute validations for general purposes.

Structure and the engine core concepts

This engine is composed by the Executors and the ValidationSuites.

Executors

Executors are responsible for executing all the respective validation suites that inherit them. A suite can inherit a custom executor or the BaseValidationSuitesExecutor. The executor may contain default and generic code and tests for a group of suites. For example, we may have one executor with features for validating a group of Databases and another for validating some APIs.

Validation Suites

A validation suite works like a Python unit test class. Once you define the validation methods (following the name pattern), these methods will be run by the engine.

Inside the validation suite class, you may implement into the validation methods the validations you want to perform. You can implement a custom validation inside your suite or if it is a common validation you may want to generalize it and use a custom executor like DatabaseValidationSuitesExecutor for sharing the same validation among other suites that inherits it. Every suite must inherit from an executor, in order to be parsed and run. So it should inherit from the BaseValidationSuitesExecutor or from a custom one (that inherits from the base executor).

The validation_* methods inside the suite and inside the executor classes will run automatically by the validation engine.

To check library main features and full documentation you can check the Validations Engine's Documentation, which is hosted by Read the Docs.

License

Apache License 2.0

Contributing

All contributions are welcome! Feel free to open Pull Requests. Check the development and contributing guidelines described in CONTRIBUTING.md

Made with :heart: by the Data Engineering team from QuintoAndar

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

validations_engine-2.0.0.tar.gz (6.8 kB view hashes)

Uploaded Source

Built Distribution

validations_engine-2.0.0-py2.py3-none-any.whl (8.1 kB view hashes)

Uploaded Python 2 Python 3

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page