scitex-parallel

Thread/process pool parallel execution utilities — `map` with tqdm, auto CPU count.

Full Documentation · uv pip install scitex-parallel[all]

Problem and Solution

Quick Start

from scitex_parallel import run

results = run(my_func, [(x,) for x in items], n_jobs=4)

Demo

flowchart LR
    A[items list] --> B[run func, items, n_jobs=8]
    B --> T1[thread 1]
    B --> T2[thread 2]
    B --> Tn[thread N]
    T1 & T2 & Tn --> C[tqdm progress bar]
    C --> D[results list]

_{Figure 1. Fan-out/fan-in: one call maps items across a thread pool with a live progress bar, then gathers results in order.}

from scitex_parallel import run

urls = ["https://api.example.com/{}".format(i) for i in range(100)]
results = run(fetch_url, [(u,) for u in urls], n_jobs=8, desc="Fetching")

Fetching: 100%|████████████| 100/100 [00:04<00:00, 22.3 it/s]

Installation

uv pip install "scitex-parallel[all]"

Through the umbrella: uv pip install "scitex[parallel]". Requires Python ≥ 3.10.

Architecture

flowchart LR
    U["user code"] --> R["run(): public API in __init__.py"]
    R --> P["ThreadPoolExecutor (n_jobs)"]
    P --> T["_run.py worker: func(*args)"]
    T --> Q["tqdm progress bar"]
    Q --> L["ordered results list"]

_{Figure 2. Module layout: the public run() fans tuple-args out to pool workers and streams progress back through tqdm.}

1 Interfaces

from scitex_parallel import run

# I/O-bound parallel map with tqdm progress bar.
results = run(fetch_url, [(u,) for u in urls], n_jobs=8, desc="Fetching")

# Tuple-arg form for multi-arg functions.
results = run(my_func, [(a, b) for a, b in zip(xs, ys)], n_jobs=4)

Part of SciTeX

scitex-parallel is part of SciTeX. Install via the umbrella with pip install scitex[parallel] to use as scitex.parallel (Python) or scitex parallel ... (CLI).

Four Freedoms for Research

0. The freedom to run your research anywhere — your machine, your terms.
1. The freedom to study how every step works — from raw data to final manuscript.
2. The freedom to redistribute your workflows, not just your papers.
3. The freedom to modify any module and share improvements with the community.

AGPL-3.0 — because we believe research infrastructure deserves the same freedoms as the software it runs on.

License

AGPL-3.0