pypipegraph
| Build status: |  |
|---|---|
| Documentation | https://pypipegraph.readthedocs.io/en/latest/ |
| Code style |  ](https://github.com/ambv/black) |
Introduction
pypipegraph: is an MIT-licensed library for constructing a workflow piece by piece and executing just the parts of it that need to be (re-)done. It supports using multiple cores (SMP) and (eventually, alpha code right now) machines (cluster) and is a hybrid between a dependency tracker (think 'make') and a cluster engine.
More specifically, you construct Jobs, which encapsulate output (i.e. stuff that needs to be done), invariants (which force re-evaluation of output jobs if they change), and stuff inbetween (e.g. load data from disk).
From your point of view, you create a pypipegraph, you create jobs, chain them together, then ask the pypipegraph to run. It examines all jobs for their need to run (either because the have not been finished, or because they have been invalidated), distributes them across multiple python instances, and get's them executed in a sensible order.
It is robust against jobs dying for whatever reason (only the failed job and everything 'downstream' will be affected, independend jobs will continue running), allows you to resume at any point 'in between' jobs, and isolates jobs against each other.
pypipegraph supports Python 3 only.
30 second summary
pypipegraph.new_pipeline()
output_filenameA = 'sampleA.txt'
def do_the_work():
op = open(output_filename, 'wb').write("hello world")
jobA = pypipegraph.FileGeneratingJob(output_filenameA, do_the_work)
output_filenameB = 'sampleB.txt'
def do_the_work():
op = open(output_filenameB, 'wb').write(open(output_filenameA, 'rb').read() + ", once again")
jobB = pypipegraph.FileGeneratingJob(output_filenameB, do_the_work)
jobB.depends_on(jobA)
pypipegraph.run()
print('the pipegraph is done and has returned control to you.')
print('sampleA.txt contains "hello world"')
print('sampleB.txt contains "hello world, once again")
1 maintainer
Release files for pypipegraph 0.197
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | |
|---|---|---|---|
| pypipegraph-0.197.tar.gz | 51.0 kB | Details |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Reset |
|---|---|---|---|---|
| pypipegraph-0.197-py2.py3-none-any.whl | Python 3, Python 2 | none | any | Details |
Total release size: 110.9 kB
Release files / pypipegraph-0.197.tar.gz
| Download URL | pypipegraph-0.197.tar.gz |
|---|---|
| Size | 51.0 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
c7b52289b597702b0bc0934a71858c5c7b2ab77a4221bbf936841dc7a883fb6a
|
|
BLAKE2b-256 checksum How to use checksums |
6681a6392bd9698f995f35c11386e87e93e5c073ea94d6439bca1ab60828de9a
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/3.4.1 importlib_metadata/4.6.1 pkginfo/1.7.1 requests/2.25.1 requests-toolbelt/0.9.1 tqdm/4.61.2 CPython/3.9.6
|
Release files / pypipegraph-0.197-py2.py3-none-any.whl
| Download URL | pypipegraph-0.197-py2.py3-none-any.whl |
|---|---|
| Size | 59.9 kB |
| Tags | Python 2 Python 3 |
|
SHA-256 checksum How to use checksums |
289b257b7afb83c5da38b6467403a76139696d4f5cd03a23b1b50d091e4b47a9
|
|
BLAKE2b-256 checksum How to use checksums |
25811475601d682a572df173a89529255910c545c79ff54030c722e9ed2966ce
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/3.2.0 pkginfo/1.5.0.1 requests/2.24.0 setuptools/44.0.0 requests-toolbelt/0.9.1 tqdm/4.50.1 CPython/3.8.10
|
