pipeforge 1.0.0

A pipeline jobs orchestrator

1. One line description

A pipeline jobs orchestrator library and stand-alone script (which can be used directly from the command line)

2. Information for users

2.1. Installation

You don't want to use this repository directly. Instead you should install the latest released version by running this command:

$ python -m pip install pipeforge

2.2. User guide

Check this file for the full guide.

You can also access this same information from python itself:

import pipeforge

help(pipeforge)

All public classes and functions are fully documented:

import pipeforge

help(pipeforge.log_configure)
help(pipeforge.ScriptManager)
help(pipeforge.Pipeline)

help(pipeforge.JobParams)

3. Information for developers

3.1. Development environment

First of all you need a mongoDB server running somewhere. You can deploy one locally and make it listen on port 27017 by running this command:

$ podman run -p 27017:27017 docker.io/library/mongo

Leave it running as "pipeforge" needs it in order to work.

Next, make sure you have all python modules listed in "dependencies" section found inside the pyproject.toml file already installed, if not, use your distro package manager to install them:

$ cat pyproject.toml | awk '/dependencies/,/]/' 
$ pacman -S ...

NOTE: This (installing dependencies through the global package manager) is
preferred to creating a virtual environment and using "python -m pip
install" inside of it because it makes it simpler to run the examples.
If you insist on using a virtual environment, this is what you need to do:

    $ python -m venv .venv
    $ source .venv/bin/activate
    $ python -m pip install .

Next, update the value of PYTHONPATH to make sure it searches for "pipeforce" in the current folder (where the source code is) instead of in system paths (in case you had already "pip install"ed it in the past):

$ export PYTHONPATH=`pwd`:$PYTHONPATH

You can now run the different demos like this:

DEMO #1:
Run (locally) a few python scripts that share inputs and outputs one with
each other. It takes ~10 seconds to complete.
    
    $ export PIPEFORGERUNNER_HEARTBEAT_SECONDS=1
    $ python pipeforge run mongodb://localhost:27017/pipelines__test pipeforge/examples/hello_world.toml LocalScriptManager

DEMO #2:
Simulate running a more complex pipeline with many steps without actually
executing anything. It takes ~1 minute to complete.

    $ export PIPEFORGE__TARGET_BRANCH=origin/master
    $ export PIPEFORGE__FEATURE_BRANCH=origin/my_new_feature
    $ python pipeforge run mongodb://localhost:27017/pipelines__test pipeforge/examples/merge_pull_request.toml DummyScriptManager

While any of the previous demos is running you can monitor its real-time execution progress by opening another terminal and running this command:

$ python pipeforge inspector mongodb://localhost:27017/pipelines__test

3.2. Running built-in tests

In addition to running "demo pipelines" (such as those in the "pipeforge/examples" folder that we used in the previous section) you can also run a full suite of pre-built test cases:

$ ./tests/run.py

Note you must make sure this command returns "OK" after making any change to the source code!

If one test fail you can obtain a detailed log of what is going on by re-running it in isolation like this:

$ ./tests/run.py --test=A:7  # (or A:4, or B:3, ... depending on the test
                             # template and number)

3.3. Running the code linter

Before merging new changes you must also check that the following command returns an empty list of warnings:

$ ruff check .

NOTE: You might need to install "ruff" first. If so, use you distro's package manager.

3.4. Distribution

3.4.1. As a python package

The source code in this repository is meant to be distributed as a python package that can be "pip install"ed.

Once you are ready to make a release:

1. Increase the "version" number in file "pyproject.toml"

2. Run the next command:

   $ ./release.sh package

3. Publish the contents of the "dist" folder to a remote package repository such as PyPi (see here)

4. Tell your users that a new version is available and that they can install it by running this command:

   $ python -m pip install pipeforge

3.4.2. As a python standalone script

You can also create a "single file" script that can be run directly from the command line without the need to install it as a package.

In order to do that, run this command:

$ ./release.sh script

The resulting binary ("pipeforge.pyz") can be copied to a different computer and run like this:

$ python pipeforge.pyz --help

3.4.3. As a standalone executable

Finally, you can create a single binary which does not even require python to be installed in the system to work.

$ ./release.sh exe

The resulting binary ("pipeforge.exe") can be copied to a different computer and run like this:

$ ./pipeforge.exe --help

NOTE: This will generate a Linux ELF executable if you run "release.sh exe" from a Linux system, and a Windows EXE file if you run it from a Windows system.