scissiparity 0.1.1

Zero-input-serialization, GIL-free parallel map-reduce.

scissiparity 🦠

Zero-input-serialization, GIL-free parallel map-reduce.

Standard Python multiprocessing is a nightmare of PicklingErrors, explicit worker pools, and massive memory duplication because it forces you to serialize data in both directions. scissiparity solves this through asymmetric serialization: it natively clones the Python interpreter's memory space via OS-level Copy-On-Write to read inputs for free, and only serializes the strictly necessary outputs.

uv add scissiparity

The Difference

Concept	multiprocessing.Pool	scissiparity
Input Propagation	Serializes/Pickles inputs across IPC queues.	Zero-serialization. Reads parent memory natively via os.fork().
Output Extraction	Pickles results via IPC queues.	Pickles results via native os.pipe() (dill).
Memory Overhead	Duplicates all inputs.	Zero-copy for inputs. OS-level Copy-On-Write mechanics.
Configuration	Requires manual chunking and worker sizing.	Zero config. Inferred purely from OS CPU topography.

Usage

You write standard $N=1$ domain logic. The @fission decorator mathematically divides the incoming iterable, fractures the interpreter, and natively routes the yielded results backward via pipe multiplexing.

from scissiparity import fission

# 1. A massive or unpicklable object lives in the parent process.
# A standard multiprocessing.Pool would crash trying to serialize this across IPC.
MASSIVE_UNPICKLABLE_STATE = {"user_1": <OpenDatabaseConnection>, "user_2": ...}

@fission
def process_users(user_ids: list[str]):
    # Protect the Linear Illusion: The user writes simple, sequential logic.
    for uid in user_ids:

        # 2. Downward Flow (Zero-serialization):
        # Child processes read the parent's memory directly via OS-level Copy-On-Write.
        state = MASSIVE_UNPICKLABLE_STATE[uid]
        computed_value = heavy_cpu_computation(state)

        # 3. Upward Flow (Serialization):
        # Only the minimal resulting data is serialized (via dill) and piped back.
        yield computed_value

# Execution naturally fissions into N processes (where N = os.cpu_count()).
# Results are multiplexed back to the parent natively as a linear stream.
results = list(process_users(["user_1", "user_2", "user_3", ...]))

Core Mechanics

• Asymmetric Serialization (Copy-On-Write): Because it leverages standard UNIX os.fork(), child processes can read the parent's massive in-memory variables instantly. You can process objects that are entirely unpicklable. Serialization (dill) is invoked only to pipe the final yielded results back to the parent.
• Exception Teleportation: If a child process raises an Exception, the error is natively serialized, piped back to the parent, and re-raised in the main thread. If one child dies, the workflow aborts gracefully. Zero "ghost compute."
• Zero Configuration: There are no workers=8 arguments. The decorator strictly aligns to the native physics of your machine by inspecting os.sched_getaffinity(0) or os.cpu_count().