zerobuffer-ipc 1.1.7

High-performance zero-copy inter-process communication library for video streaming

ZeroBuffer Python Implementation

Native Python implementation of the ZeroBuffer protocol for high-performance zero-copy inter-process communication.

Features

• True Zero-Copy: Uses memoryview and buffer protocol to avoid data copies
• Cross-Platform: Supports Linux, Windows, and macOS
• Protocol Compatible: Binary compatible with C++ and C# implementations
• Pythonic API: Clean, idiomatic Python interface
• Type Safe: Full type hints for better IDE support
• Thread Safe: Built-in synchronization for multi-threaded applications

Requirements

• Python 3.8 or later
• posix-ipc (Linux/macOS only): pip install posix-ipc
• pywin32 (Windows only): pip install pywin32

Installation

pip install -e .

Quick Start

Reader Example

from zerobuffer import Reader, BufferConfig

# Create a buffer
config = BufferConfig(metadata_size=1024, payload_size=1024*1024)
with Reader("my-buffer", config) as reader:
    # Wait for writer to connect
    print("Waiting for writer...")
    
    # Read frames
    while True:
        frame = reader.read_frame(timeout=5.0)
        if frame:
            # Access data without copying
            data = frame.data  # This is a memoryview
            print(f"Received frame {frame.sequence}: {len(data)} bytes")
            
            # Process the frame...
            
            # Must release frame to free buffer space
            reader.release_frame(frame)

Writer Example

from zerobuffer import Writer

# Connect to existing buffer
with Writer("my-buffer") as writer:
    # Write some metadata
    metadata = b"{'format': 'raw', 'version': 1}"
    writer.set_metadata(metadata)
    
    # Write frames
    for i in range(100):
        data = f"Frame {i}".encode()
        writer.write_frame(data)
        print(f"Sent frame {i}")

Zero-Copy Advanced Usage

# True zero-copy writing with memoryview
import numpy as np

# Create numpy array
arr = np.arange(1000, dtype=np.float32)

# Get memoryview of array (no copy)
view = memoryview(arr)

# Write with zero-copy
writer.write_frame_zero_copy(view)

# Or use direct buffer access
buffer = writer.get_frame_buffer(size=4000)
# Write directly into shared memory
buffer[:] = arr.tobytes()
writer.commit_frame()

API Reference

Reader Class

Reader(name: str, config: Optional[BufferConfig] = None)

Creates a new buffer and prepares for reading.

Methods:

• get_metadata() -> Optional[memoryview]: Get metadata as zero-copy memoryview
• read_frame(timeout: Optional[float] = 5.0) -> Optional[Frame]: Read next frame
• release_frame(frame: Frame) -> None: Release frame and free buffer space
• is_writer_connected() -> bool: Check if writer is connected
• close() -> None: Close reader and clean up resources

Writer Class

Writer(name: str)

Connects to an existing buffer for writing.

Methods:

• set_metadata(data: Union[bytes, bytearray, memoryview]) -> None: Write metadata (once only)
• write_frame(data: Union[bytes, bytearray, memoryview]) -> None: Write a frame
• write_frame_zero_copy(data: memoryview) -> None: Write frame with zero-copy
• get_frame_buffer(size: int) -> memoryview: Get buffer for direct writing
• commit_frame() -> None: Commit frame after direct writing
• is_reader_connected() -> bool: Check if reader is connected
• close() -> None: Close writer

Frame Class

Represents a zero-copy reference to frame data.

Properties:

• data -> memoryview: Zero-copy view of frame data
• size -> int: Size of frame data
• sequence -> int: Sequence number

BufferConfig Class

BufferConfig(metadata_size: int = 1024, payload_size: int = 1024*1024)

Configuration for creating a buffer.

Zero-Copy Guarantees

The Python implementation provides true zero-copy access through:

1. memoryview objects: No data copying when accessing frame data
2. Buffer protocol: Direct memory access for compatible objects
3. Shared memory: Direct mapping of shared memory into process space

When Copies Occur

• Converting memoryview to bytes: bytes(frame.data)
• Using non-buffer protocol objects with write_frame()
• String encoding: "text".encode()

Avoiding Copies

• Use memoryview objects whenever possible
• Use write_frame_zero_copy() for memoryview data
• Use numpy arrays or other buffer protocol objects
• Access frame data directly via frame.data memoryview

Performance Considerations

1. Pre-allocate buffers: Reuse buffers instead of creating new ones
2. Batch operations: Process multiple frames before releasing
3. Use appropriate buffer sizes: See capacity planning in main README
4. Monitor buffer utilization: Avoid buffer full conditions

Error Handling

All operations may raise exceptions from zerobuffer.exceptions:

• WriterDeadException: Writer process died
• ReaderDeadException: Reader process died
• BufferFullException: Buffer is full
• FrameTooLargeException: Frame exceeds buffer capacity
• SequenceError: Frame sequence validation failed

Thread Safety

The Reader and Writer classes are thread-safe. Multiple threads can:

• Call methods on the same Reader/Writer instance
• Read frames concurrently (with proper frame release)
• Write frames concurrently

However, only one Reader and one Writer can connect to a buffer at a time.

Platform Notes

Linux

• Uses POSIX shared memory (/dev/shm)
• Requires posix-ipc package
• File locks in /tmp/zerobuffer/

Windows

• Uses Windows named shared memory
• Requires pywin32 package
• File locks in temp directory

macOS

• Similar to Linux with BSD-specific handling
• Requires posix-ipc package

Testing Utilities

The Python implementation includes testing utilities for cross-platform compatibility with C# and C++ implementations:

BufferNamingService

Ensures unique buffer names across test runs to prevent conflicts:

from zerobuffer_serve.services import BufferNamingService

# Creates unique buffer names for test isolation
naming_service = BufferNamingService(logger)
actual_name = naming_service.get_buffer_name("test-buffer")
# Returns: "test-buffer_<pid>_<timestamp>" or uses Harmony environment variables

TestDataPatterns

Provides consistent test data generation across all language implementations:

from zerobuffer_serve.test_data_patterns import TestDataPatterns

# Generate deterministic frame data
data = TestDataPatterns.generate_frame_data(size=1024, sequence=1)

# Generate simple pattern data
simple_data = TestDataPatterns.generate_simple_frame_data(size=1024)

# Verify data matches pattern
is_valid = TestDataPatterns.verify_simple_frame_data(data)

# Generate test metadata
metadata = TestDataPatterns.generate_metadata(size=256)

These utilities ensure that Python, C#, and C++ implementations can exchange data correctly in cross-platform tests.

Duplex Channel Support

The Python implementation includes full support for duplex channels (bidirectional request-response communication):

from zerobuffer.duplex import DuplexChannelFactory, ProcessingMode
from zerobuffer import BufferConfig

# Server side
factory = DuplexChannelFactory()
server = factory.create_immutable_server("my-service", BufferConfig(4096, 10*1024*1024))

def process_request(frame):
    """Process request and return response"""
    # Frame is automatically disposed via RAII
    data = bytes(frame.data)
    result = process_data(data)
    return result

server.start(process_request, ProcessingMode.SINGLE_THREAD)

# Client side
client = factory.create_client("my-service")
sequence = client.send_request(b"request data")
response = client.receive_response(timeout_ms=5000)

if response.is_valid and response.sequence == sequence:
    with response:  # Context manager for RAII
        print(f"Response: {bytes(response.data)}")

See DUPLEX_CHANNEL.md for detailed documentation.