parproc 0.4.0

Process Parallelization

PARPROC

A library for parallelizing scripts, by allowing different script elements to be grouped in processes, and allowing each group to depend on the completion of one or more other groups

Installation and Usage

Installation

pip install parproc

Usage

See examples/success.py for an example. Different jobs can be defined and dependencies between them established, e.g.:

import parproc as pp

@pp.Proc(now=True)
def func0(context):
    time.sleep(1)


@pp.Proc(now=True)
def func1(context):
    time.sleep(3)


@pp.Proc(now=True)
def func2(context, deps=['func0', 'func1']):
    time.sleep(2)

The now=True argument kicks off the jobs as soon as they are defined, which is optional. The example then waits for all jobs to finish

pp.wait_for_all()

Argument choices and glob expansion

Protos can define allowed values for arguments via arg_choices. When set, only those values are accepted when creating procs, and you can use glob patterns (*, ?) in filled-out names to create multiple procs at once:

@pp.Proto(name='build::[env]', arg_choices={'env': ['dev', 'prod']})
def build(context: pp.ProcContext, env: str) -> str:
    return f'built_{env}'

# Create one proc
pp.create('build::dev')

# Create all matching procs: build::dev and build::prod
names = pp.create('build::*')
pp.start(*names)
pp.wait(*names)

Using * or ? in an argument when the proto has no arg_choices for that argument raises an error.

Contributing

Feel free to send me PRs

Setting up the development environment

1. Install uv if you haven't already
2. Clone the repository
3. Initialize the development environment:

   uv sync --dev
   

   This will install all dependencies and development tools.

Running tests

Run all tests using the test script:

./scripts/test.sh

Or run specific test modules:

./scripts/test.sh tests.simple tests.proto

You can also run tests directly with uv:

uv run python -m unittest tests.simple tests.proto tests.errorformat

Running examples

Run any of the example scripts:

uv run python examples/success.py
uv run python examples/error.py
uv run python examples/failed_deps.py

Change Log