Package used in validation tabular data
Project description
Validata Table
Validata Table is python package used as a tabular data validation service.
It includes four subpackages, where you can find their respective documentations :
validata_core
validata_ui
validata_api
tests
used for testing the project (in development environment or in continuous integration)
It offers also a command line tool validata
used to validate tabular data.
See validata_core/README.md for more details.
Using validata-table
package
You can use locally this package validata-table
, doing:
pip install validata-table
This allows you to use validata
command line tool to validate tabular data:
validata --help
Development
This project is based on Docker to use a local environement developement.
This project includes a Makefile, which allows you to run predefined actions by running specific commands.
Dependency management tool used : Poetry version 1.6.1
Requirements
First install Docker Engine and docker-compose >= version 2 on your machine if not already done.
Then you may clone source code in your deployment directory:
git clone https://gitlab.com/validata-table/validata-table.git
cd validata-table
Run on development local environment
Configuration is done by editing environment variables in .env
,
(see .env.example
file to set .env
file)
Warning: Docker env files do not support using quotes around variable values!
Launch the development local environment, thanks to the makefile command:
# in validata-table/
make serve_dev
This launches two docker containers:
- validata-table-api-dev
- validata-table-ui-dev
Validata Table API (using validata-table-api-dev
docker container)
To access to the API of Validata Table click on http://localhost:5000/
Validata Table UI (using validata-table-ui-dev
docker container)
To access to the API of Validata Table click on http://localhost:5001/
Validata Table command line tool (using validata-table-api-dev
docker container)
To use validata command line tool in the docker development environment, run:
docker exec -it validata-table-api-dev bash
validata --help
Test
To launch tests in the development environment, run:
make test
Linting
Some code linting tools are configured for this project:
-
black: to format code, run
make black
-
isort: to format import, run
make isort
-
flake8: to enforce style coding, run
make flake8
-
pyright: to check static types.
pyright
will be executed in local virtual environment withpoetry
:First you need to create a virtual environment .venv at the root of the project and configure it:
# At /validata-table/ root python3.9 -m venv .venv # install virtual environement locally poetry config virtualenvs.in-project true poetry config --list # Check if correctly configured >>> ... virtualenvs.in-project = true ... poetry install # install project dependencies
Then execute locally
pyright
withpoetry
:poetry run pyright .
Continuous Integration
The continuous integration is configured in .gitlab-ci.yml
file
Release a new version
On master branch :
- Update version in pyproject.toml and CHANGELOG.md files
- Update version Docker images used in docker-compose.yml file:
- registry.gitlab.com/validata-table/validata-table/validata-table-api:vX.X.X
- registry.gitlab.com/validata-table/validata-table/validata-table-ui:vX.X.X
- Commit changes using
Release
as commit message - Create git tag (starting with "v" for the release)
git tag -a
- Git push:
git push && git push --tags
- Check that pypi package is created and container images for validata_ui and validata_api are well-built (validata-table pipelines)
Creating and pushing a new release will trigger the pipeline in order to automatically:
- publish a new version of
validata-table
package on PyPI - build a new tag of the Docker image
validata-table-ui
, based on the new version just created of the packagevalidata-table
, and publish it on the gitlab container registry validata-table-ui, used to run user interface Validata - build a new tag of the Docker image
validata-table-api
, based on the new version just created of the packagevalidata-table
and publish it on the gitlab container registry validata-table-api, used to run the API of Validata
This pipeline runs when a new tag under the format 'vX.X.X' is pushed.
Deploy to production
You can deploy this project on your own production server by using Docker.
Production environment is based on Docker images validata-table-ui
and validata-table-api
hosted on gitlab container registries validata-table-ui
and validata-table-api
To deploy in production, you can follow these steps described below.
First you may clone source code in your deployment directory:
git clone https://gitlab.com/validata-table/validata-table.git
cd validata-table
Configuration is done by editing environment variables in .env
,
(see .env.example
file to set .env
file).
Warning: Docker env files do not support using quotes around variable values !
Launch the docker production environment with makefile:
make serve_prod
OR launch the docker production environment with docker compose
command:
docker compose -f docker-compose.yml up --build -d
Then:
- To access to the API of Validata Table click on http://localhost:<PORT_NUMBER_API>/ (replacing PORT_NUMBER_API with the value you choose)
- To access to the UI of Validata Table click on http://localhost:<PORT_NUMBER_UI>/ (replacing PORT_NUMBER_UI with the value you choose)
- To access to the
validata
command lines tool:docker exec -it validata-table-api bash validata --help
History
To keep track of the project's history, Validata Table comes from the merge of four gitlab repositories :
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Hashes for validata_table-0.10.3-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 5d17fa19dff005f640b2c9473ff223f170988cb0de9cb24bf1189a60b17301b0 |
|
MD5 | ccf9eb3ed86c3a6daed71364bd0225fd |
|
BLAKE2b-256 | 3bb64541aecfc5fc14fe5b63871461384f586e7f21329fe24e9fb1897a3d75d3 |