pymembus 1.0.2

Python shared memory library

pymembus

Python bindings for libmembus, a small shared-memory IPC library.

pymembus is useful when multiple local processes need to exchange data with low overhead:

• raw shared memory blocks with memmap
• broadcast message queues with memmsg
• command channels with memcmd
• fixed-schema shared key/value state with memkv
• video ring buffers with memvid
• audio ring buffers with memaud
• simple readiness polling with select()

All shared-memory names in the examples use POSIX-style names such as "/myshare". On Linux, stale objects can remain after a crash; each type has a remove(name) helper for cleanup.

Contents

• Install
• Build From Source
• Run Tests
• Quick Start
• API Guide
• Diagnostics
• Command Line Helpers
• Troubleshooting
• Development Notes

Install

From PyPI:

python3 -m pip install pymembus

If you build from source, install system build dependencies first. On Debian or Ubuntu:

sudo apt-get update
sudo apt-get install -y build-essential git cmake libboost-all-dev
sudo apt-get install -y python3 python3-pip

Optional tools used by this repository's CMake documentation targets:

sudo apt-get install -y doxygen graphviz go-md2man

Build From Source

Install this checkout into your active Python environment:

python3 -m pip install .

Or build with CMake directly:

cmake -S . -B ./bld -DCMAKE_BUILD_TYPE=Release
cmake --build ./bld -j

The source build fetches pinned third-party dependencies when needed:

• libmembus v1.2.0
• pybind11 v2.13.6

libmembus 1.2.0 requires C++20 and Boost stacktrace support. The Python build requires CMake 3.30 or newer.

To uninstall a pip install:

python3 -m pip uninstall -y pymembus

To build distribution artifacts:

python3 setup.py sdist
python3 setup.py bdist_wheel

Run Tests

The test suite uses pytest. If you built with CMake:

cmake -S . -B ./bld -DCMAKE_BUILD_TYPE=Release
cmake --build ./bld --target pymembus-test

You can also run pytest from the repository root:

python3 -m pytest -v

Wheel builds made through scikit-build do not run the CMake test target during install; run tests explicitly with one of the commands above.

The pytest configuration in pyproject.toml limits collection to src/pytest/py and adds bld/lib to PYTHONPATH, so it will not try to run vendored tests under bld/_deps.

The suite covers maps, messages, commands, key/value stores, video/audio formats, NumPy buffer sharing, diagnostics, and select(). NumPy-specific tests are skipped when NumPy is not installed.

Quick Start

import pymembus

if hasattr(pymembus, "pymembus"):
    pymembus = pymembus.pymembus

name = "/quickstart"
pymembus.memmsg.remove(name)

tx = pymembus.memmsg()
rx = pymembus.memmsg()

assert tx.open(name, 1024, True, True)      # writer, create
assert rx.open(name, 1024, False, False)    # reader, attach

assert tx.write("hello")
message, overrun = rx.read_with_overrun(0)

assert message == "hello"
assert not overrun

rx.close()
tx.close()
pymembus.memmsg.remove(name)

API Guide

The snippets below assume the normalized import pattern from the quick start:

import pymembus

if hasattr(pymembus, "pymembus"):
    pymembus = pymembus.pymembus

Raw Shared Memory: memmap

memmap gives direct access to a named shared-memory block.

import pymembus

name = "/my_map"
pymembus.memmap.remove(name)

writer = pymembus.memmap()
reader = pymembus.memmap()

assert writer.open(name, 1024, True, True)
assert writer.write("hello") == 5

assert reader.open(name, 0, False, False, True)  # read-only attach
assert reader.read(5) == "hello"

view = memoryview(writer)
assert view.shape == (1024,)
assert not view.readonly

readonly_view = memoryview(reader)
assert readonly_view.readonly

del readonly_view
del view
reader.close()
writer.close()
pymembus.memmap.remove(name)

Parameters for memmap.open(name, size, create=False, new=False, read_only=False):

• create: create the object if it does not exist
• new: remove any existing object first
• read_only: attach without write permissions

Broadcast Messages: memmsg

memmsg is a single-writer, multi-reader message queue. Every reader receives every message independently.

name = "/my_messages"
pymembus.memmsg.remove(name)

tx = pymembus.memmsg()
rx1 = pymembus.memmsg()
rx2 = pymembus.memmsg()

assert tx.open(name, 4096, True, True)
assert rx1.open(name, 4096, False, False)
assert rx2.open(name, 4096, False, False)

assert tx.write("frame-ready")
assert rx1.read_with_overrun(0) == ("frame-ready", False)
assert rx2.read_with_overrun(0) == ("frame-ready", False)

tx.close()
rx1.close()
rx2.close()

Use poll() for non-blocking readiness checks:

if rx1.poll():
    msg = rx1.read(0)

When a reader falls behind far enough that the writer overwrites unread messages, read_with_overrun() returns ("", True).

Command Channels: memcmd

memcmd is a multi-writer command channel. It is useful for control paths, for example sending commands from a UI process to a capture process.

name = "/camera_commands"
pymembus.memcmd.remove(name)

receiver = pymembus.memcmd()
sender = pymembus.memcmd()

assert receiver.open(name, 1024, True, True)  # bReader=True, bCreate=True
assert sender.open(name, 1024)

assert sender.write("pan-left")
cmd, overrun = receiver.read_with_overrun(0)
assert cmd == "pan-left"
assert not overrun

sender.close()
receiver.close()

Shared State: memkv

memkv is a fixed-schema key/value store. The owner creates the store and sets slot names; any process can then read or write values.

name = "/camera_state"
pymembus.memkv.remove(name)

owner = pymembus.memkv()
assert owner.create(name, 3, 16, 64, True)
assert owner.setName(0, "mode")
assert owner.setName(1, "count")
assert owner.setName(2, "status")

peer = pymembus.memkv()
assert peer.open(name)

assert peer.setValue("mode", "auto")
value, stale = peer.getValue("mode")
assert value == "auto"
assert not stale

epoch = peer.getEpoch()
assert owner.setValue("status", "ready")
changed, epoch = peer.getChanged(epoch)
assert changed == {"status": "ready"}

peer.close()
owner.close()

getValue() returns (value, stale). stale is true if the lock-free read did not settle before its retry limit.

Video Ring Buffers: memvid

memvid stores packed video frames in a shared ring buffer. Use video_format values instead of old numeric bits-per-pixel values.

name = "/video"
pymembus.memvid.remove(name)

video = pymembus.memvid()
assert video.open(
    name,
    True,
    640,
    480,
    pymembus.video_format.rgb24,
    30,
    4,
)

slot = video.getPtr(0)
assert video.setVpts(slot, 123456)
assert video.setApts(slot, 123000)
assert video.next(1) == 1

assert video.getSeq() == 1
assert video.getFrameSeq(slot) == 1
assert video.getFormatName() == "RGB24"

frame = memoryview(video[slot])
assert frame.shape == (480, 640, 3)

del frame
video.close()
pymembus.memvid.remove(name)

Supported video formats:

• gray8
• rgb24
• bgr24
• rgba32
• bgra32
• yuyv422
• uyvy422

NumPy can view frame buffers without copying:

import numpy as np

frame = np.array(video[slot], copy=False)
frame[10, 10] = [255, 0, 0]

del frame
video.close()
pymembus.memvid.remove(name)

Audio Ring Buffers: memaud

memaud stores PCM audio buffers in a shared ring buffer. Use audio_format values instead of old numeric bits-per-sample values.

name = "/audio"
pymembus.memaud.remove(name)

audio = pymembus.memaud()
assert audio.open(
    name,
    True,
    2,
    pymembus.audio_format.s16le,
    48000,
    50,
    4,
)

slot = audio.getPtr(0)
assert audio.setPts(slot, 123456)
assert audio.next(1) == 1

assert audio.getChannels() == 2
assert audio.getSampleRate() == 48000
assert audio.getFormatName() == "S16LE"

buf = memoryview(audio[slot])
assert buf.shape == (960, 2)

del buf
audio.close()
pymembus.memaud.remove(name)

Supported audio formats:

• u8
• s16le
• s24le
• s32le
• f32le
• f64le

s24le is exposed as raw bytes because Python and NumPy do not have a native 24-bit integer scalar type.

Buffer Lifetime And Read-Only Views

memmap, memvid, and memaud expose Python's buffer protocol. A writer that created a share exports writable buffers. A reader opened read-only, or with open_existing() for video/audio, exports read-only buffers.

Keep one lifetime rule in mind for video and audio buffers: release memoryview or NumPy arrays before calling close() on the owning object. pymembus intentionally raises RuntimeError if a video or audio mapping is closed while exported frame buffers still exist, because those buffers would otherwise point at unmapped shared memory.

Waiting On Multiple Sources: select()

select(wait_ms, conditions) polls a list of Python callables and returns the zero-based index of the first ready condition, or -1 on timeout.

idx = pymembus.select(100, [
    lambda: video.getSeq() > last_video_seq,
    lambda: commands.poll(),
])

if idx == 0:
    read_video_frame()
elif idx == 1:
    handle_command()

The conditions should be cheap, non-consuming readiness checks.

Diagnostics

Most API calls return False, -1, or an empty string on failure. Check last_error() or last_error_message() immediately after a failed call:

missing = pymembus.memmap()

if not missing.open("/does-not-exist", 0, False):
    assert pymembus.last_error() == pymembus.errc.open_failed
    print(pymembus.last_error_message())

Common error codes include:

• open_failed
• create_failed
• map_failed
• size_mismatch
• invalid_layout
• not_open
• access_denied
• message_too_large
• lock_timeout
• timeout
• overrun

Command Line Helpers

After installation, the package may install a pymembus helper command on Linux:

pymembus help
pymembus files
pymembus info version
sudo pymembus uninstall

pymembus info <variable> accepts values such as name, description, url, version, build, company, author, lib, include, bin, and share.

Troubleshooting

ModuleNotFoundError: No module named 'pymembus'

Build the extension first:

cmake -S . -B ./bld -DCMAKE_BUILD_TYPE=Release
cmake --build ./bld -j

Then run tests from the repository root:

python3 -m pytest -v

The repository's pytest config adds bld/lib to PYTHONPATH.

Pytest tries to run pybind11 tests

This happens when pytest recursively scans build artifacts. The repository config sets:

norecursedirs = ["bld", "_skbuild", "dist", "build", ".git", "*.egg-info"]

If you use a custom pytest command, prefer:

python3 -m pytest -v src/pytest/py

A share already exists or has an invalid layout

Remove stale shared-memory objects before creating a fresh one:

pymembus.memmsg.remove("/my_messages")
pymembus.memvid.remove("/video")
pymembus.memaud.remove("/audio")

The 1.2.0 libmembus wire formats validate shared-memory headers. Old shares created by earlier library versions may be rejected with invalid_layout.

Doxygen or Graphviz warnings during build

The CMake build may emit documentation warnings from Doxygen or Graphviz. They do not affect the Python extension or pytest suite.

Development Notes

• Project metadata lives in PROJECT.txt.
• The Python extension source is in src/py/cpp/main.cpp.
• Tests live in src/pytest/py/test.py.
• The fallback libmembus dependency is configured in src/libmembus.cmake.
• See UPDATE.md for the libmembus 1.2.0 migration report.

References

• Python
• CMake
• pip
• git
• Boost
• pybind11
• libmembus