pygolem

Spawn tiny little servants, running things in parallel.

Before

import threading
import time
from concurrent.futures import ThreadPoolExecutor, as_completed

def fetch_all(fn, items, workers=10, max_retries=3, rate_limit=5):
    values = [None] * len(items)
    errors = {}
    remaining = list(enumerate(items))
    timestamps = []
    lock = threading.Lock()

    def rate_limited(item):
        while True:
            with lock:
                now = time.monotonic()
                timestamps[:] = [t for t in timestamps if now - t < 1]
                if len(timestamps) < rate_limit:
                    timestamps.append(now)
                    break
            time.sleep(0.05)
        return fn(item)

    for attempt in range(max_retries + 1):
        with ThreadPoolExecutor(max_workers=workers) as executor:
            futures = {executor.submit(rate_limited, item): i for i, item in remaining}
            remaining = []
            for future in as_completed(futures):
                index = futures[future]
                try:
                    values[index] = future.result()
                except Exception as e:
                    remaining.append((index, items[index]))
                    errors[index] = e
        if not remaining or attempt == max_retries:
            break
        time.sleep(2 ** attempt)

    return values, errors

results, errors = fetch_all(fetch_url, urls)

Roughly 40 lines, and this version doesn't even validate its own inputs, guard against the bool-is-int trap, or cap thread count safely at scale.

After

from pygolem import threads

results = threads.map_parallel(fetch_url, urls, workers=10, retries=3, rate_limit=("token", 5))

results.values   # successful outputs, in original input order
results.errors   # {index: {"type": ..., "message": ...}} for items that failed permanently

Four lines. Same results. Ordered output, isolated per-item failures, retries with backoff, and a global rate limit.

What it is

pygolem wraps concurrent.futures.ThreadPoolExecutor with the things a real parallel workload eventually needs:

1. Ordered results: .values always matches your input order, even though threads finish in whatever order they finish in.
2. Per-item error isolation: one failing item never crashes the batch.
3. Retries with backoff, plus retry_if to skip retrying failures you already know are permanent.
4. Global rate limiting: cap total throughput across the entire pool, not per-worker, with a choice of token bucket, leaky bucket, or sliding window.

Dependency-free, just the standard library.

Installation

pip install pygolem

Quick example

from pygolem import threads
import requests

def fetch(url):
    return requests.get(url, timeout=5).status_code

urls = [
    "https://example.com",
    "https://httpbin.org/status/500",   # will fail, then retry
    "https://github.com",
]

result = threads.map_parallel(
    fetch, urls,
    workers=3,
    retries=2,
    backoff=("exponential", 0.5),
    rate_limit=("token", 5),
)

print(result.values)   # [200, 500, 200]
print(result.errors)   # {1: {"type": "ConnectionError", "message": "..."}}

API

threads.map_parallel(fn, items, workers=None, retries=None, retry_if=None, backoff=("linear", 1), rate_limit=None)

Returns a Data object: .values (results in input order, None for permanent failures) and .errors ({index: {"type": str, "message": str}}).

Known limitations

• A stuck fn hangs the whole call. There's no timeout yet, and Python can't force-kill a thread, so one item that never returns means map_parallel never returns either.
• Retries can repeat side effects. If fn does something real (like a request) and then fails afterward, retrying does that thing again. Use retry_if to skip retrying failures you know already went through.
• One slow retry delays everyone. The backoff wait happens between rounds, so already-successful items still sit and wait for it.
• More workers won't speed up CPU-heavy code. Threads share one GIL, extra workers mainly help waiting on network/disk, not crunching numbers.

Benchmarks

I/O-bound workload (time.sleep(0.01) per call), measured with time.perf_counter(). Full breakdown in tests/02_speed.

Overhead over a raw ThreadPoolExecutor stays under 1% at scale. The retry/error/ordering machinery is close to free.

The workers default is capped at 32 for a reason: an earlier version that defaulted to workers=len(items) crashed at 100,000 items (RuntimeError: can't start new thread). The capped default handles the same load in 33s without crashing.

Tested

125 pytest tests, 0 failures — input validation, ordering, error isolation, retries, all three rate-limit algorithms, and a dedicated suite that proves out the known limitations. Full breakdown in tests/TEST_REPORTS.md.

pytest tests/ -v --tb=short

Roadmap

• Token-bucket and leaky-bucket rate limiting (alongside sliding-window)
• Conditional retries via retry_if
• pygolem.aio — asyncio engine, with item-level backoff
• pygolem.processes — multiprocessing engine, with real cancellation of hung workers
• Per-item timeout support
• Optional progress callback

About the name

Inspired by Minecraft's copper golem. A small automaton that potters around doing small repetitive tasks on its own, tirelessly, until it needs a little maintenance. Not a bad description of a background worker.

License

Apache License 2.0. see LICENSE.