Versatile Data Kit Heartbeat tool
Heartbeat tool for verifying deployed SDK and Control Service are functional and working correctly.
It checks that a job can be created, deployed, run and deleted.
- Health monitoring - one can schedule this run every few minutes and get heartbeat of their installation
- Regression testing after change in Versatile Data Kit, as clients would be able to customize vdk, they should have a way to check for regressions.
- Installation/Upgrade of VDK Control Service acceptance test.
What it does ?
It simulates Data Engineer workflow:
- Creates a data job, downloads keytab
- Deploys the data job with pre-defined scripts to run on a scheduled basis (every minute)
- Different data jobs and run tests can be run depending on the configuration.
- This way it can be run in different modes. See config.py and the DATAJOB_DIRECTORY_* and JOB_RUN_TEST_* configuration options.
- Undeploys and deletes the data job.
Prerequisites
See heartbeat_config_example.ini and complete the TODOs inside
Installation
# TODO: Change to public PYPI index
pip install -i https://test.pypi.org/simple/ vdk-heartbeat
Configuration
See config.py for details on what can be configured.
Running
You can run the test locally, part of your CICD or schedule it to run periodically.
The test is passed or fail test.
If it fails it returns non-zero error code and prints the error.
It also produces a tests.xml file in junit xml format.
- Specify configuration in environment variables or in a file (use the file for things that can be in source control)
- Example:
export DATABASE_PASS=xxx
vdk-heartbeat -f heartbeat_config.ini
Extensibility
Users can replace the data job being deployed and executed and the run test which is used to verify the job run/execution.
See config.py DATAJOB_DIRECTORY_* and JOB_RUN_TEST_* configuration options.
Build, test, and release
See or run cicd/build.sh to build and test the project locally.
Release
Releases are made to PyPI.
Versioning follows https://semver.org.
- A release step in Gitlab CI is automatically triggered after merging changes if build/tests are successful.
- Update major or minor version when necessary only.
1 maintainer
Metadata
1 maintainer
Release files for vdk-heartbeat 0.4.380166752
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | |
|---|---|---|---|
| vdk-heartbeat-0.4.380166752.tar.gz | 20.5 kB | Details |
Release files / vdk-heartbeat-0.4.380166752.tar.gz
| Download URL | vdk-heartbeat-0.4.380166752.tar.gz |
|---|---|
| Size | 20.5 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
549f012078c556affe944a2cf681ba2469bfbb56da22c07318fbe62520aebc6f
|
|
BLAKE2b-256 checksum How to use checksums |
3336ea303f6210b2abe7b78fedc136a5e8da01c867f0bc5398ff7b93a11e5df2
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/3.4.2 importlib_metadata/4.8.1 pkginfo/1.7.1 requests/2.26.0 requests-toolbelt/0.9.1 tqdm/4.62.3 CPython/3.7.11
|
