zeropickle 0.1.0

Move numpy arrays and metadata between processes blazingly fast!

ZeroPickleQueue 🚀

ZeroPickleQueue is a high-performance, near drop-in replacement for Python's multiprocessing.Queue, optimized for NumPy arrays.

By leveraging Python's shared_memory module, it eliminates the massive overhead of pickling and unpickling large datasets between processes.

---

🏎️ Why ZeroPickleQueue?

In standard Python multiprocessing, sending a large NumPy array through a Queue involves:

1. Serialization: Pickling the array in the producer process.
2. IPC Transfer: Moving the serialized bytes through a pipe.
3. Deserialization: Unpickling the array in the consumer process.

For high-resolution video frames (e.g., 4K images), this process can take 50ms–200ms per frame, creating a massive bottleneck. ZeroPickleQueue writes the data to a shared memory block and only passes compact metadata through the queue, reducing transfer latency substantially.

---

✨ Features

• Near Drop-in Compatibility: API follows multiprocessing.Queue (put, get, empty, full) with NumPy-first constraints.
• Metadata Support: Pass extra info (timestamps, labels) along with your arrays.
• Automatic Resource Management: Includes atexit handlers to clean up shared memory segments and prevent memory leaks.
• Thread-Safe Slot Recycling: Uses a background acknowledgment thread to manage memory slot reuse automatically.

---

📦 Installation

pip install zeropickle

---

🛠 Usage

Basic Example

The interface is designed to be familiar to anyone who has used standard Python multiprocessing.

import numpy as np
from multiprocessing import Process
from zeropickle import ZeroPickleQueue

def producer(queue):
    # Create a large dummy frame
    frame = np.random.randint(0, 255, (2160, 3840, 3), dtype=np.uint8)
    queue.put(frame)
    queue.put(None)  # Sentinel to signal completion

def consumer(queue):
    while True:
        item = queue.get()
        if item is None:
            break
        print(f"Received array with shape: {item.shape}")

if __name__ == "__main__":
    # Define the shape and dtype beforehand
    q = ZeroPickleQueue(
        data_shape=(2160, 3840, 3),
        data_dtype=np.uint8,
        max_size=10
    )

    p1 = Process(target=producer, args=(q,))
    p2 = Process(target=consumer, args=(q,))

    p1.start()
    p2.start()
    p1.join()
    p2.join()

Passing Metadata

You can pass a tuple where the first element is the NumPy array, and subsequent elements are your metadata. None is also supported as a sentinel value for graceful consumer shutdown. Other non-array payloads are not supported.

Compatibility Notes

ZeroPickleQueue is a near drop-in replacement, not a full drop-in replacement. It keeps the familiar queue methods, but requires fixed data_shape/data_dtype at initialization and only accepts NumPy-array payloads (plus None sentinel).

# Producer
queue.put((image_array, {"frame_id": 42, "detected_objects": 3}))

# Consumer
image, meta = queue.get()
print(meta["frame_id"])

---

⚙️ How It Works

1. Pre-allocation: On initialization, a shared memory segment is created to hold max_size arrays of the specified shape.
2. Slot Tracking: An internal queue manages "slots" (indices) in that memory block.
3. The Put Operation: The producer claims a slot, copies the NumPy data into it using np.copyto, and sends the slot index via a standard queue.
4. The Get Operation: The consumer reads the slot index, accesses the shared memory, and immediately notifies an "acknowledgment queue."
5. Recycling: A background thread monitors acknowledgments and returns used slots back to the producer for reuse.

---

⚠️ Limitations & Tips

• Fixed Dimensions: Because memory is pre-allocated, every array must match the data_shape and data_dtype provided at initialization.
• Copy on Get: Currently, .get() returns a copy of the data. This is safer as it prevents the producer from overwriting memory that the consumer is still reading.
• Cleanup: Always call .close() or rely on the atexit handler to ensure /dev/shm (on Linux) doesn't fill up with orphaned memory segments.

---

🤝 Contributing

Contributions are welcome! If you have ideas for non-copying reads or support for variable-length data, feel free to open an issue or a PR.