Execute pipeline stages and steps
Project description
Staging
About
Automate your commands - be it for environment setup or in CI/CD pipeline. Parallelize execution, and make it easier to handle dependent commands.
Requirements
This package requires tomlkit
and dataclass-wizard
packages.
Usage
Full description
Staging is configurable in pyproject.toml
file and is accessible through CLI interface.
In order to use staging, you have to define steps and then stages. Stages are built from steps.
Let's see example below (which uses all available options):
[tool.staging.steps]
# Test
get_test_packages = {execute = "command to get test packages", output="test_packages"}
coverage = { prepare = "coverage run -m pytest -xv {packages}", execute = "coverage report {flags}", cleanup = "coverage erase", output = "coverage_result", format = {flags = "flags", packages= "test_packages"} }
# Lint
clean = {execute = "command to execute", success_codes=[0,1,2,3,4,5]}
isort = { execute = "isort {flags} . tests", format = {flags = "flags"}}
black = { execute = "black {flags} . tests", format = {flags = "flags"}}
pylint = { execute = "pylint {package} tests", format = {package = "package"}}
noprint = { execute = "noprint -ve {package} tests", format = {package = "package"}, error_codes=[1,2]}
finish = { execute = "finishing command" }
[tool.staging.stages.format]
description = "Format code"
format = {package="staging"}
steps = [
{step = "isort"},
{step = "black"},
]
[tool.staging.stages.lint]
description = "Check linting"
format = {flags="--check", package="staging"}
steps = [
{step = "clean", continue_on_failure=true},
{parallel = {steps = ["isort", "black", "pylint", "noprint"]}, continue_on_failure=true},
{step = "finish"},
]
[tool.staging.stages.test]
description = "Test the package"
format = {flags="-m --fail-under=30"}
steps = [
{step = "get_test_packages"},
{step = "coverage"},
]
Here we have defined 7 different steps: get_test_packages
, coverage
, clean
, and so on.
These 7 steps are later used in 3 stages: test
, lint
, and format
.
Let's say we execute staging format
. What happens?
- Stage context is updated with formatter {key=
package
, value=staging
}. - Step
isort
is executed. Commandisort {flags} . tests
is formatted using stage formatter. Since flags is empty, the final command is simplyisort . tests
. - Step
black
is executed. This works the same asisort
above.
Now let's say we execute staging lint
.
- Context has formatter for value
package
, but also forflags
. - Step
clean
is executed. It might fail, but we don't care and instead the process is continued. - Steps
isort
,black
,pylint
andnoprint
are all executed in a thread pool due to being specified in aparallel
block. This time however, even though some of the steps are the same as in thestaging format
, we now have aflags
formatter defined. Thanks to this, we now have executedisort --check . tests
command that would verify if the formatting was properly applied. Due to how steps are configured - specified custom success and error codes - the parallel block would fail (one command will crash others). However, the parallel block allows to specifycontinue_on_error
as well, just like step block. - As such,
finish
block is also executed.
Last but not least, staging test
.
- Similarly to previous stages, we set up the stage formatter context.
- Step
get_test_packages
outputs a package name to stdout. This value will be saved undertest_packages
key in the staging context. - Step
coverage
has a bit of a different structure, it hasprepare
andcleanup
blocks on top of typicalexecute
. Prepare is executed beforeexecute
. However, it's not taken into account when writing to output. Moreover, it will always fail when status code is different to 0. Block calledcleanup
is just like what it says it is. It's a cleanup process executed afterexecute
. It will be executed no matter what is the final status of theexecute
command.
You can also chain your stages in one command. Keep in mind however that formatting contexts are defined on per stage basis.
So if you execute staging format lint
- formatting context from stage format
won't be propagated to stage lint
.
Running stages in one execution is also performed in serial manner - no parallelism.
Example
For real-life examples, see pyproject.toml
file for this package.
Development
Installation
Install virtual environment and check_bump package in editable mode with dev dependencies.
python -m venv venv
source venv/bin/activate
pip install -e .[dev]
How to?
Automate as much as we can, see configuration in pyproject.toml
file to see what are the flags used.
staging format # Reformat the code
staging lint # Check for linting issues
staging test # Run unit tests and coverage report
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.