philiprehberger-run-parallel

Run multiple functions in parallel and collect results with the simplest possible API.

Installation

pip install philiprehberger-run-parallel

Usage

Run functions in parallel

from philiprehberger_run_parallel import parallel

results = parallel(
    lambda: 1 + 1,
    lambda: 2 + 2,
    lambda: 3 + 3,
)
# [2, 4, 6]

Pass arguments with tuples

from philiprehberger_run_parallel import parallel
import time

results = parallel(
    (time.sleep, 0.1),
    (pow, 2, 10),
)
# [None, 1024]

Track progress

from philiprehberger_run_parallel import parallel

def on_progress(completed, total):
    print(f"{completed}/{total} done")

results = parallel(
    lambda: 1 + 1,
    lambda: 2 + 2,
    lambda: 3 + 3,
    on_progress=on_progress,
)
# 1/3 done
# 2/3 done
# 3/3 done

Process-based parallelism

from philiprehberger_run_parallel import parallel_process

def cpu_work():
    return sum(range(10_000_000))

results = parallel_process([cpu_work, cpu_work, cpu_work], max_workers=3)

Map a function over items

from philiprehberger_run_parallel import parallel_map

results = parallel_map(str.upper, ["hello", "world"])
# ["HELLO", "WORLD"]

# Control the number of workers
results = parallel_map(fetch_url, urls, workers=8, timeout=30)

Async parallel

import asyncio
from philiprehberger_run_parallel import aparallel

async def main():
    results = await aparallel(
        fetch("https://example.com/a"),
        fetch("https://example.com/b"),
    )
    print(results)

asyncio.run(main())

Error handling

from philiprehberger_run_parallel import parallel, ParallelError

try:
    results = parallel(
        lambda: 1,
        lambda: 1 / 0,
    )
except ParallelError as e:
    print(e.errors)   # [None, ZeroDivisionError(...)]
    print(e.results)  # [1, None]

API

Development

pip install -e .
python -m pytest tests/ -v

Support

If you find this project useful:

⭐ Star the repo

🐛 Report issues

💡 Suggest features

❤️ Sponsor development

🌐 All Open Source Projects

💻 GitHub Profile

🔗 LinkedIn Profile

License

MIT