scissiparity 2.0.0

Zero-input-serialization, GIL-free parallel map.

scissiparity 🦠

Zero-input-serialization, GIL-free parallel map.

multiprocessing forces you to serialize data in both directions, so large or unpicklable inputs crash or blow up memory. scissiparity clones the interpreter with os.fork(), reads inputs for free through Copy-On-Write, and serializes only the results you yield.

uv add scissiparity

Usage

Write ordinary sequential logic over a chunk. The @fission decorator splits the iterable across your CPU cores, runs a forked process per chunk, and merges the yielded results into one lazy stream.

from collections.abc import Iterator

from scissiparity import fission

MASSIVE_UNPICKLABLE_STATE = load_everything_into_memory()


@fission
def process_users(user_ids: list[str]) -> Iterator[float]:
    for user_id in user_ids:
        state = MASSIVE_UNPICKLABLE_STATE[user_id]
        yield heavy_cpu_computation(state)


results = list(process_users(["user_1", "user_2", "user_3"]))

Properties

• See-through types. @fission preserves the wrapped signature. Element and keyword-argument types survive the decoration, so type checkers infer process_users(...) -> Iterator[float] with no annotations lost.
• Zero input serialization. Children read the parent's memory natively via Copy-On-Write; only yielded outputs are piped back (via dill). Unpicklable inputs are fine.
• Exception teleportation. An exception in a child is serialized, piped back, and re-raised in the parent with its original traceback and chained causes intact (via tblib), so failures read exactly as they would in-process. One failure aborts the run.
• Composes without fork-bombing. A @fission function called from inside another runs sequentially, so simple cells combine into larger pipelines.
• Zero configuration. Parallelism is inferred from os.sched_getaffinity(0) (falling back to os.cpu_count()). There are no workers= knobs.

Results are streamed back in completion order, not input order.

PyTorch & transformer models

scissiparity is an excellent fit for CPU inference: os.fork() lets every worker share the loaded model weights through Copy-On-Write, so a multi-gigabyte model is loaded into memory once — not re-serialized or re-loaded per worker.

• Load the model before applying @fission, and don't run tensor ops in the parent first. fork() duplicates only the calling thread, so a live BLAS/OpenMP thread pool can deadlock the children.
• Set torch.set_num_threads(1) and TOKENIZERS_PARALLELISM=false so process-level and library-level parallelism don't oversubscribe your cores.

CUDA / GPU is not supported. A CUDA context cannot survive os.fork() (it raises Cannot re-initialize CUDA in forked subprocess), and GPU memory is not Copy-On-Write shareable, so the core advantage doesn't apply. Use spawn-based tooling such as torch.multiprocessing for GPU workloads.