atomicshm

Access shared memory in Python with atomic operations, for cases where the other sharing party requires atomicity.

Python gives you shared memory but no way to touch it atomically. If another process — C, Rust, Go, C++ — is running a lock-free protocol over those bytes, memoryview assignment is not good enough: it is not atomic, it carries no memory ordering, and the peer has no idea the GIL exists. This gives you the handful of operations that are actually needed, and nothing else.

from multiprocessing.shared_memory import SharedMemory
import atomicshm

shm = SharedMemory(create=True, size=4096)
with atomicshm.AtomicView(shm.buf) as view:
    view.store_u64(0, 0)
    previous = view.fetch_add_u64(0, 1)      # returns the value it replaced
    if view.cas_u32(64, 0, 1) == 0:          # compare-and-swap succeeded
        ...

• 8, 16, 32, and 64-bit load, store, exchange, compare-and-swap, and fetch-add/sub/and/or/xor.
• Explicit memory ordering — relaxed, acquire, release, acq_rel, seq_cst.
• 14–28 ns per operation, which is less than calling an empty Python function.
• No dependencies, no runtime configuration, ~1300 lines of C (mostly one macro expanded four times).
• One abi3 wheel per platform, working on CPython 3.11 and every version after.

Install

pip install atomicshm

Wheels are published for Linux (manylinux and musllinux, x86-64 and aarch64), macOS (Intel and Apple silicon), and Windows (x64 and ARM64). Anywhere else, the sdist builds with any C compiler and no other tooling.

The operations

Create an AtomicView over any writable, C-contiguous buffer — SharedMemory.buf, an mmap, a bytearray, a numpy array. Then, for each width N in 8, 16, 32, 64:

Plus view.nbytes, view.address, view.closed, view.close(), and atomicshm.fence(order).

Arithmetic wraps at the width. All arguments are positional-only.

cas returns the previous value, not a bool

That is what the hardware gives you, and it means a retry loop never needs a reload:

current = view.load_u64(off)
while (previous := view.cas_u64(off, current, current + 1)) != current:
    current = previous

The swap succeeded if and only if the return value equals expected.

Cells, for hot fixed offsets

Control fields — a queue head, a lock word — live at a known offset. Binding one once moves the bounds and alignment checks out of the loop:

counter = view.cell_u64(0)
bump = counter.fetch_add           # hoist the attribute lookup too
for _ in range(1_000_000):
    bump(1)

Cells have the same operations without the offset argument and without the width suffix: load, store, exchange, cas, fetch_add, …. A cell holds a reference to its view, so the mapping stays alive as long as the cell does.

Values are unsigned coming out, either way going in

Arguments may be signed or unsigned — anything in [-2**(N-1), 2**N) — and are stored as the low N bits. Results are always unsigned, because that is the one interpretation that is always well defined. Wrapping arithmetic is identical for both, so a signed counter only needs converting when you look at it:

atomicshm.as_signed(view.load_u32(0), 32)     # 0xFFFFFFFF -> -1

Memory ordering

SEQ_CST by default: correct for any protocol, and the right thing to reach for unless you have a specific reason not to. Weaker orders are available where they matter:

Passing an order an operation cannot use — RELEASE to a load, say — raises ValueError rather than being silently reinterpreted. cas takes a second, optional failure order (RELAXED, ACQUIRE, or SEQ_CST), defaulting to the success order with its release half removed.

On MSVC every operation carries a full barrier regardless of the order you ask for. Stronger than requested is always sound, and it keeps the x64/ARM64 differences out of the code entirely — see DESIGN.md.

Two things that will bite you

Alignment is required

An access at offset must be N/8-byte aligned. x86 would tolerate a misaligned atomic; AArch64 will fault or silently stop being atomic. So misalignment raises ValueError on every platform, including the ones that would have let you get away with it locally. SharedMemory and mmap are page-aligned, so in practice this only constrains how you lay out your struct.

The view holds the buffer exported

An AtomicView keeps its target's buffer exported for its whole life, so the mapping cannot be unmapped out from under it. The flip side is that SharedMemory.close() raises BufferError until the view lets go:

view = atomicshm.AtomicView(shm.buf)
...
view.close()        # or use the view as a context manager
shm.close()
shm.unlink()

This is also what makes it safe to resolve an address once and reuse it. A view resolves its base pointer at construction and a cell resolves its slot address at creation; neither is re-derived per call. Three rules keep that sound:

• The export pins the memory. While a buffer is exported, its owner may not move or free it — bytearray.append, mmap.close, and memoryview.release all raise BufferError until the view lets go.
• The owner cannot be collected. A view holds a strong reference to its target and a cell holds one to its view, so nothing a cell depends on can be deallocated while the cell is alive — including under the cycle collector, which can only reclaim a view when the cells pointing at it are unreachable too.
• A view's base is write-once. close() can clear it, nothing can re-point it, and AtomicView.__init__ refuses a second call. So while a view is open, its base is the value every cell resolved against.

An address is therefore never used without first confirming, through an owned reference, that the export is still live. The failure mode when a view is closed is a ValueError, never a stale pointer.

Talking to a peer process

Lay the peer's struct out with natural alignment and address the fields by offset. Given:

struct control {                        // C
    _Atomic uint64_t head;              // offset 0
    _Atomic uint64_t tail;              // offset 8
    _Atomic uint32_t lock;              // offset 16
};

#[repr(C)]                              // Rust
struct Control {
    head: AtomicU64,                    // offset 0
    tail: AtomicU64,                    // offset 8
    lock: AtomicU32,                    // offset 16
}

head = view.cell_u64(0)                 # Python
tail = view.cell_u64(8)
lock = view.cell_u32(16)

The orders map one-to-one onto C11's memory_order_* and Rust's Ordering::*, and the operations onto atomic_fetch_add / fetch_add, atomic_compare_exchange_strong / compare_exchange, and so on. cas is always the strong form; there is no weak variant, because spurious failure has no upside from Python.

A spinlock

lock = view.cell_u32(16)

def acquire():
    while lock.cas(0, 1, atomicshm.ACQUIRE) != 0:
        pass

def release():
    lock.store(0, atomicshm.RELEASE)

A single-producer ring buffer index

head, tail = view.cell_u64(0), view.cell_u64(8)

def push(payload):                                   # producer side
    t = tail.load(atomicshm.RELAXED)                 # only we write it
    if t - head.load(atomicshm.ACQUIRE) == CAPACITY:
        return False
    buf[slot(t)] = payload                           # publish the payload...
    tail.store(t + 1, atomicshm.RELEASE)             # ...then the index
    return True

The release store is what makes the payload visible to the consumer before the index that points at it.

Performance

On a Linux x86-64 desktop, CPython 3.12 (python benchmarks/bench.py):

empty python function call            38.3 ns     <- the floor for any Python call
AtomicView.load_u64                   18.5 ns
AtomicView.fetch_add_u64              27.7 ns
AtomicView.cas_u64                    24.2 ns
AtomicU64.load                        14.0 ns
AtomicU64.fetch_add                   24.5 ns
AtomicU64.cas                         22.4 ns

Every operation costs less than calling an empty Python function, and most of what is left is vectorcall dispatch rather than the atomic instruction — a lock cmpxchg is a handful of nanoseconds. There is no PyArg_ParseTuple, no tuple allocated for arguments, no Python frame pushed, and the GIL is never released.

In throughput terms that is roughly 35–70 million operations per second from a single Python thread, loop overhead included. The practical consequence is that you can stop designing around the cost: a shared counter, a lock word, or a ring buffer index can be touched on the hot path without the access itself becoming the thing you have to justify.

What this is not

No mutexes, condition variables, or any blocking primitive; no futex / WaitOnAddress parking; no 128-bit atomics; no weak compare-exchange; no fetch_min/fetch_max; no atomic access to Python objects. The scope is the complete set of operations needed to implement a lock-free protocol against a peer process, and stopping there is what lets the library be finished.

If you need a lock, build it out of cas — the recipe above is the whole thing.

Compared with atomics

atomics solves the same problem and covers more ground: it wraps the patomic C library through CFFI, supports Python 3.6+, and offers a richer type model (AtomicInt, AtomicUint, AtomicBytes, alignment introspection). It is good software, and if you need Python 3.6–3.10 it is your option.

The difference that will decide it for most projects is cost at the call site. Same 64-bit slot, same mapping, Linux x86-64, CPython 3.12:

A gap that size deserves an explanation rather than a benchmark table, and it has a simple one. atomics makes 65 Python function calls per load, 114 per fetch_add, and 142 per compare-and-swap. This library makes none. Profiling one load shows _released twelve times, _assert_not_released eight times, address three times, release three times, plus cffi.cast and from_buffer — every call re-acquires the buffer, re-derives the pointer through CFFI, and re-validates. Multiply those call counts by the ~38 ns an empty Python call costs and you predict 2.5 / 4.4 / 5.4 µs against 2.8 / 4.6 / 5.9 µs measured. The whole gap is Python call overhead. Neither library is spending meaningful time on the atomic instruction.

atomicshm resolves the address once, at AtomicView construction and again at cell_uN(), and then does nothing per call but a bounds check, an alignment check, and the instruction. It can cache that pointer safely for exactly the reason described in the export note above: holding the buffer exported is what guarantees the mapping cannot move or vanish. The safety property and the speed are the same mechanism.

This matters because latency is usually the entire reason to reach for shared memory. At microseconds per operation, atomic access is something you budget for, ration, and design around — batching to amortise it, or accepting it and justifying the cost. At tens of nanoseconds it stops being a design constraint. A million compare-and-swaps is ~17 ms rather than ~6 seconds.

The second difference is the license: atomics is GPL-3.0, this is MIT. These are single hardware instructions with an argument check in front of them, and a copyleft obligation across an entire application is a steep price for lock xadd. Worth noting that patomic itself is LGPL-3.0-or-later with a linking exception — the copyleft is a choice made at the Python wrapper layer, not inherited from the atomics implementation. atomicshm also has no runtime dependency at all, where atomics requires cffi.

The two do interoperate: both perform genuine hardware atomics on the same bytes, so a process using one and a process using the other coordinate correctly.

Measured with timeit, best of 5 runs of 20,000 iterations, bound methods hoisted out of the loop for both; call counts from cProfile. No comparison script ships here — running one would mean importing a GPL library from an MIT package, which is one of the things this library exists to avoid.

Provenance

atomicshm was written from the GCC/Clang and MSVC intrinsic documentation and the CPython C API. No source from atomics or patomic was read, copied, or adapted, and the API here was designed and fully implemented before either was known to be relevant.

Requirements

CPython 3.11+ on a 64-bit platform. Free-threaded builds are supported and do not re-enable the GIL. The build fails rather than produce a binary whose "atomics" are a process-private lock, which would be silently useless across processes.

License

MIT