scissiparity 1.0.0

Zero-input-serialization, GIL-free parallel map.

scissiparity 🦠

Zero-input-serialization, GIL-free parallel map.

multiprocessing forces you to serialize data in both directions, so large or unpicklable inputs crash or blow up memory. scissiparity clones the interpreter with os.fork(), reads inputs for free through Copy-On-Write, and serializes only the results you yield.

uv add scissiparity

Usage

Write ordinary sequential logic over a chunk. The @fission decorator splits the iterable across your CPU cores, runs a forked process per chunk, and merges the yielded results into one lazy stream.

from collections.abc import Iterator

from scissiparity import fission

MASSIVE_UNPICKLABLE_STATE = load_everything_into_memory()


@fission
def process_users(user_ids: list[str]) -> Iterator[float]:
    for user_id in user_ids:
        state = MASSIVE_UNPICKLABLE_STATE[user_id]
        yield heavy_cpu_computation(state)


results = list(process_users(["user_1", "user_2", "user_3"]))

Properties

• See-through types. @fission preserves the wrapped signature. Element and keyword-argument types survive the decoration, so type checkers infer process_users(...) -> Iterator[float] with no annotations lost.
• Zero input serialization. Children read the parent's memory natively via Copy-On-Write; only yielded outputs are piped back (via dill). Unpicklable inputs are fine.
• Exception teleportation. An exception in a child is serialized, piped back, and re-raised in the parent. One failure aborts the run.
• Composes without fork-bombing. A @fission function called from inside another runs sequentially, so simple cells combine into larger pipelines.
• Zero configuration. Parallelism is inferred from os.sched_getaffinity(0) (falling back to os.cpu_count()). There are no workers= knobs.

Results are streamed back in completion order, not input order.