Overview
Deployv addon: Gitlab tools. Generated by cookiecutter and cookiecutter-deploy-addon.
A small set of tools to automate repetitive tasks for testing gitlab repositories.
It is important to say that this is not by any mean a replacement for the MQT tools, these are tools that we need in order to make gitlab-ci work properly with the MQT tools.
Free software: BSD license
Installation
pip install vxci
Documentation
Development
To run the all tests run:
tox
Note, to combine the coverage data from all the tox environments run:
Windows |
set PYTEST_ADDOPTS=--cov-append tox |
|---|---|
Other |
PYTEST_ADDOPTS=--cov-append tox |
Changelog
Unreleased
push_coverage_combine: publish_coverage prints the line and the branch rate of coverage.xml, coverage_line_rate: 93.44% and coverage_branch_rate: 84.34%, one per line. GitLab keeps one coverage per job and publish_coverage’s is the total of lines and branches, so a job of its own can report each rate with a coverage: regex on these lines, for a badge per rate (badges/<branch>/coverage.svg?job=<job>). Nothing is printed when there is no coverage.xml.
staggered_install: the check that every install left a coverage data file counted “Combined N files”, a line coverage 7 does not print: it writes “Combined data file <path>” for each file. The check then saw 0 files and failed the job after every module had passed. Both outputs are counted now.
build_image: a push the registry refuses for a moment is tried again, 3 times, 15s, 30s and 45s apart, before the job fails. Only a read timeout was retried; quay.io answering “failed to fetch oauth token … 503 Service Unavailable” failed build_docker on the first attempt, after the image was already built.
A pull the registry refuses for a moment, a 5xx such as quay.io answering “503 Service Unavailable”, is tried again the same way, in every command that pulls the instance and postgres images (staggered_install, test_repo, clickbot…). A pull of an image that does not exist still fails at once.
staggered_install: --staggered_odoo_args (STAGGERED_ODOO_ARGS) adds odoo-bin options to every install, the template databases included, with and without --isolated and --parallel. STAGGERED_ODOO_ARGS: "--with-demo" runs the tests of a repository that stand on demo data: test_repo loads it on 19.0, staggered_install did not, and those tests failed in setUpClass with “External ID not found in the system: base.user_demo”.
staggered_install and test jobs: no coverage report any more. A job reports only its own share, and its total and its “Coverage failure” read as the coverage of the project; publish_coverage, which combines the files of every job, is the only one that reports. The data files are still combined and copied out as before.
staggered_install: the job fails when fewer coverage data files are combined than installs were measured, naming how many were lost. With --pytest, pytest-cov combined and deleted the files of the installs running next to it: vauxoo/ircodoo 19.0 combined 29 files for 74 installs and published 27 % instead of about 92 %, while the job stayed green.
With PGHOST set, the jobs run on that PostgreSQL server instead of starting a container of their own: no postgres image is pulled and Odoo connects to PGHOST, PGPORT, PGUSER and PGPASSWORD. Every database the job creates is named vxci_<CI_JOB_ID>_<name> and dropped when the job ends, cancelled included, unless VXCI_NO_CLEAR_CONTAINERS is set. vxci pg_sweep drops the ones a killed job left behind, never those of a job with a session open. Without PGHOST nothing changes but the next point.
The postgres container of a job keeps its data on disk, not on a tmpfs: a dozen jobs at once held every database in RAM and swapped ircmig to a halt.
staggered_install: a job that sets neither STAGGERED_KEEP_DB nor --parallel runs again. It failed before starting with “Some environment variables were not found: STAGGERED_KEEP_DB, PARALLEL”.
clickbot: the asset bundles are built and committed once before the suite, as Odoo’s own runner does before its HTTP tests. Each browser test built them again inside its transaction, and with four workers doing it at once the webclient missed the minute Odoo waits for it: apps failed with “The ready “odoo.isReady === true” code was always falsy” before the clickbot clicked anything.
check_log: on Odoo 17.0 and later the “0 failed, 0 error(s) of 0 tests when loading database” warning is ignored again, as on 16.0: Odoo logs it from odoo.tests.result now, not odoo.tests.runner. With --pytest a module whose tests are all post_install logs it at install.
check_log: the <module>.pytest.log of a --pytest run, the console output of pytest, is no longer read as an install log. It logged “Modules loaded not found in logs” as an ERROR for each module.
staggered_install: a job that does not pass --pytest runs again. vxci 0.0.64 failed before starting with “Some environment variables were not found: STAGGERED_PYTEST”, since a flag left off counted as a missing variable; no flag is required any more.
staggered_install: --pytest (STAGGERED_PYTEST), with --isolated --parallel N, runs the post_install tests of each module under pytest-odoo with N pytest-xdist workers on copies of its installed database, instead of one class after another inside the install. The pytest versions clickbot pins, plus pytest-cov, are installed in the instance container.
clickbot: on Odoo 20.0 each bucket walks only its own apps again. 20.0 starts the clickbot with an object of options, { xmlId: ... }, and the bare xmlid the suite passed left it empty, which walks every app: each bucket walked the whole database and the job ran out of its hour.
clickbot: pytest and its plugins are installed as root. In the Odoo 16.0 image /home/odoo/.local belongs to root, and the install as odoo failed with “Permission denied: ‘/home/odoo/.local/lib’”.
clickbot: the suite also fails when an integer id, a foreign key or a sequence reached CLICKBOT_ID_CEILING_PCT percent (60 by default) of the largest value it can take, and logs the 20 fullest ones. The ceiling of an integer column fed by a bigint sequence is the integer one. CLICKBOT_ID_CEILING_ALLOWED accepts table.column items above the threshold. It reads only the catalog, the sequences and indexed maximums, so it takes seconds on large databases.
clickbot: Odoo 16.0 or later, no longer 19.0 only. The suite starts the clickbot the way Odoo’s own test of each version does (the debug services in 16.0, the module loader since 17.0), on /web or /odoo (18.0), passes success_signal only where browser_js takes it, posts the JSON-RPC call without build_rpc_payload before 19.0, reads failure_count only where ir.cron has it, and blocks the external requests of the crons in 16.0, which does not block them in tests.
check_log: when the log path holds the staggered_logs folder of a staggered_install --isolated --parallel run, every warning is reported with the install that logged it, templates included, and that install’s log, and a table of those installs closes the output. The joined odoo.log.gz_* still decides the exit status, and a warning no install log holds, such as one from another job’s log, is still reported as it is.
clickbot: new command. Restores the database a test job kept (db.pgc and filestore.tgz in CLICKBOT_DIR, $CI_COMMIT_REF_SLUG/clickbot by default) and runs the vxci_clickbot suite shipped in this package on it, under pytest-odoo with CLICKBOT_WORKERS pytest-xdist workers (4) in one job: Odoo’s clickbot over every app admin and demo see, and “Run Manually” on every scheduled action. The apps are packed into CLICKBOT_APP_BUCKETS tests (12) on the run itself, by how many menus each one has, never from a table of times; the crons into CLICKBOT_CRON_BUCKETS (8). A project tunes it with CLICKBOT_USERS, CLICKBOT_SKIP_APPS, CLICKBOT_SKIP_MENUS, CLICKBOT_SKIP_CRONS, CLICKBOT_SETUP (package.module:function hooks called with env), CLICKBOT_TIMEOUT, CLICKBOT_CRON_TIMEOUT and CLICKBOT_CURSOR_LOCK_TIMEOUT (120 s, Odoo’s 20 s is too short on a loaded runner). The verdict is pytest’s exit code and the JUnit report, written to CLICKBOT_DIR with odoo.log; a skipped test turns it red. Odoo 19.0 or later. It replaces the .gitlab/clickbot.sh and the click_all test files each project carried.
staggered_install –isolated –parallel N (STAGGERED_PARALLEL): runs the whole plan in one job, N installs at a time off one queue, instead of a share of a parallel: job fixed beforehand by staggered_costs.json. The work is done by odoo-bin-server-staggered (vauxoo/odoo-staggered) inside the instance container, the same command a developer runs locally, installed from the wheel quay.io/vauxoo/dockerv carries in ODOO_STAGGERED_WHEEL_DIR. vxci passes it the modules, ODOO_TEST_TAGS, the language, coverage around each module install, STAGGERED_KEEP_DB and --allow_deprecated, and collects the saved log, the log of every install and the coverage as before. Measured on vauxoo/ircodoo: 2868 s in one job, against 3150 s to 3364 s for the slowest of five nodes.
build_image: with ORCHESTSH=True, the image is looked up by the name the orchestsh CLI gave it (the image_name.env it writes). A main branch whose name is not VERSION (e.g. 20.0 building saas-19.4) can now set MAIN_BRANCH to push the <MAIN_APP>-<VERSION>-latest tag instead of failing with “Unable to find the recently built image”, with the orchestsh CLI before and after orchestsh/executor!366.
test_repo/test_image: with ORCHESTSH=True, instance_image is IMAGE_REPO:IMAGE_TAG when both are set (the image_name.env sourced from the build job), and is computed only as a fallback.
common: new optional ORCHESTSH_REGISTRY and ORCHESTSH_TOKEN environment variables. When the registry is set, the image notification sent to ORCHEST_REGISTRY after a push is also sent to OrchestSh, so both systems can run side by side while projects migrate. A failed OrchestSh notification is logged and does not fail the job.
staggered_install: new command that installs the modules of a repository one at a time on a single database, dependencies first and the main app last, running each module’s own tests as it is installed. A module that fails does not stop the rest. Every install runs under coverage run -p and the run combines them into .coverage_<job name>. No run asks for demo data.
staggered_install –isolated: installs each module alone on a clone of a template database holding exactly the closure of what it declares, so a module using another one it never declared fails instead of passing on what an earlier module brought in. Splits itself across the nodes of a GitLab parallel: job (CI_NODE_INDEX/CI_NODE_TOTAL), balanced with the per-module seconds of staggered_costs.json when the repository commits it. STAGGERED_KEEP_DB keeps the main app’s database. Demo data that Odoo failed to load ends the run red.
coverage: new COVERAGE_EXCLUDE_LINES environment variable to append regexes to the exclude_lines section of the generated .coveragerc. A repository can now skip a whole family of lines from its variables.sh instead of spreading # pragma: no cover comments over the source. Use \n to separate several regexes, since variables.sh only accepts single-line exports.
tools_test: fall back to Chrome for Testing when a pinned Chrome version is no longer published in the Google apt pool, which purges old builds. Keeps the older Chrome that Odoo’s legacy --headless tour driver needs installable, so browser tours stop being silently skipped with “Chrome executable not found”.
0.1.0 (2017-03-03)
First release on PyPI.
Metadata
Release files for vxci 0.0.70
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | |
|---|---|---|---|---|
| vxci-0.0.70.tar.gz | 112.9 kB | Without provenance | Details |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Reset |
|---|---|---|---|---|
| vxci-0.0.70-py2.py3-none-any.whl | Python 3, Python 2 | none | any | Details |
Total release size: 196.1 kB
Release files / vxci-0.0.70.tar.gz
| Download URL | vxci-0.0.70.tar.gz |
|---|---|
| Size | 112.9 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
c79773d4e5904335048a2ffa9a8d4357014b48994d8ac6afc53851452b0d9778
|
|
BLAKE2b-256 checksum How to use checksums |
eeb510bdeda3d5410db8c2000ad9065f899b47add5467bd04d3a0ab22eea58cb
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/7.0.0 CPython/3.10.16
|
Release files / vxci-0.0.70-py2.py3-none-any.whl
| Download URL | vxci-0.0.70-py2.py3-none-any.whl |
|---|---|
| Size | 83.3 kB |
| Tags | Python 2 Python 3 |
|
SHA-256 checksum How to use checksums |
4c521c61b2f23743ea933e484d7be7f9aae25d8bcca672dd1f64078a7ca01cf2
|
|
BLAKE2b-256 checksum How to use checksums |
34b7ba2d1231fb57c4e8d96fa4f6f80b6f51e846d4a06ddfd32a86a32b97b92a
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/7.0.0 CPython/3.10.16
|
Security for vxci 0.0.70
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
No provenance found
None of the files in this release were uploaded with provenance metadata.
A note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.