lazily

Lazy reactive primitives for Python — the Cell kernel (Source / Computed cells, plus Effect) with automatic dependency tracking and cache invalidation, plus the language-agnostic lazily-spec wire protocol for mirroring graph state across processes and languages.

Overview

lazily is the Python port of the Cell kernel (#lzcellkernel): two value kinds — Source and Computed — plus the value-less Effect sink. Cell is the value-node concept; Source is its native writable handle.

• Source — a value written from outside (set / merge); the writable kind. Construct with source / cell (native class Source, slot SourceSlot; Cell / CellSlot remain identity-preserving migration aliases). A MergeCell is a Source whose write folds under a non-KeepLatest policy (Cell ≡ Source(KeepLatest)).
• Computed — a value computed from upstream, via a compute function. Construct with computed(ctx, f). Guarded by default and lazy by default.
• Effect — a value-less sink outside the hierarchy; nothing can depend on it.

A Computed is lazy by default: dependents are marked dirty on invalidation but only recompute when accessed. When you need eager push-style semantics — recompute immediately, observe v1 → v2 with no unset window — make it eager: computed(ctx, f).eager(). Going eager attaches a scheduled puller Effect over the backing memo (eagerness is graph state — an _eager bit plus a side table — not a distinct type), so N writes inside one batch re-materialize the computed once, at the flush. Every cell is guarded — an equal recompute is suppressed by the PartialEq guard (matching TC39 Signal.Computed), so unchanged values never cascade downstream work. There is no unguarded derived mode: computed is the guarded derived constructor, and the former separate memo construction is retired.

Migration note (v2 Cell kernel, #lzcellkernel). The v1 value vocabulary is removed: Signal / signal / signal_def, formula / formula_def / FormulaCell, SourceCell / SourceCellSlot, and the .drive() / .undrive() / is_driven / is_active methods are gone. Use the v2 spelling instead — an eager Computed is computed(ctx, f).eager(); the lifecycle is .eager() / .lazy() / .is_eager(). The construction sugar cell / cell_def is deprecated in favour of source / source_def, and the derived-value slot decorator is deprecated in favour of the guarded computed (slot_def remains as the storage-sense factory). Python has no compile-time read/write split (see the design's §4): the split is a convention — a Source has set / merge, a Computed does not — not a runtime gate.

There is no dedicated Context class — a plain dict is the context, so the Rust reference's ctx.computed(f) is spelled computed(ctx, f) here. Slot is retained as the storage position that holds a node: a Slot uses itself as the dictionary key that caches its value, so any dict works as the reactive "world" (lazily-spec §5.0 "Slot-as-storage"). It is the Python analog of lazily-rs's surviving storage-sense Slot; construct it directly with Slot(callable=…) when a raw storage node is genuinely needed.

Feature coverage

Coverage by feature family across every binding, generated from coverage.json in lazily-spec. Legend: ✅ shipped · ~ partial · — absent · ⊘ not applicable. The canonical matrix with per-cell notes and platform carve-outs lives in lazily-spec § Cross-Language Coverage.

Summary — family × language

Roll-up rule: a family cell is ✅ only when every required row in that family is ✅; ~ when the family is mixed (some shipped or partial); — when no required row is shipped or partial; ⊘ only when every required row in the family is not applicable. Rows the spec marks MAY (optional) are excluded from the roll-up — declining an optional feature is not a gap.

A family cell summarises 74 feature rows. For row-level marks, per-cell notes, and platform carve-outs see the canonical coverage matrix in lazily-spec.

Installation

pip install lazily

Example usage

from lazily import SourceSlot, source, slot

# Sources hold a value that can be updated.
name = SourceSlot[dict, dict, str]()


# Slots are functions that depend on sources and other slots.
@slot
def greeting(ctx: dict) -> str:
    print("Calculating greeting...")
    return f"Hello, {name(ctx).value}!"


# A SourceSlot can also have a default value.
@source
def response(ctx: dict) -> str:
    return "How are you?"


@slot
def greeting_and_response(ctx: dict) -> str:
    print("Calculating greeting_and_response...")
    return f"{greeting(ctx)} {response(ctx).value}"


ctx = {}

name(ctx).value = "World"

# First access: runs the function
print(greeting(ctx))
# Calculating greeting...
# 'Hello, World!'

# Second access: uses cache (no print)
print(greeting(ctx))
# 'Hello, World!'

# Dependencies also access cached values
print(greeting_and_response(ctx))
# Calculating greeting_and_response...
# 'Hello, World! How are you?'

# Dependencies also cached
print(greeting_and_response(ctx))
# 'Hello, World! How are you?'

# Update cell: invalidates cache
name(ctx).value = "Lazily"

# Access again: re-runs the function
print(greeting_and_response(ctx))
# Calculating greeting_and_response...
# Calculating greeting...
# 'Hello, Lazily! How are you?'

# Another access: uses cache
print(greeting_and_response(ctx))
# 'Hello, Lazily! How are you?'

Core Concepts

Context

A plain dict is the context. It owns all cached Slot values; Slots store their cache under themselves as keys. The current implementation is single-threaded — create one dict per reactive graph.

Slot

A Slot wraps a compute function (ctx) -> T; the result is cached after first access. Dependencies are discovered automatically via a global slot_stack — any Slot or Cell read during computation becomes a dependency and re-subscribes on every recompute, so conditional branches update the dependency graph with no manual cleanup. When a dependency invalidates, the Slot only marks its cache dirty; it does not recompute until called again.

Source cell

A Source cell holds a mutable value. Reading cell.value inside a Computed or Effect auto-subscribes that reader; assigning cell.value = x (or cell.set(x)) compares old and new via != and, only if changed, cascades invalidation to dependents. Construct with source (the v1 cell name is deprecated).

Eager Computed

An eager Computed — computed(ctx, f).eager() — is the counterpart to a lazy Computed. Where a lazy cell marks itself dirty on invalidation and recomputes on the next read, an eager one recomputes the instant a dependency is invalidated, before the mutating call returns. The value is always materialized, so observers never see an intermediate unset value.

from lazily import SourceSlot, computed

n = SourceSlot[dict, dict, int]()

ctx: dict = {}
n(ctx).value = 1

doubled = computed(ctx, lambda c: n(c).value * 2).eager()  # eager: materialized now
print(doubled.value)   # 2

n(ctx).value = 5       # doubled recomputes immediately
print(doubled.value)   # 10 — already current, no lazy read needed

An eager Computed is composed from existing primitives, not a parallel engine: a memoized backing Slot supplies glitch-free, guarded recomputation, and a small puller Effect re-materializes it after every invalidation to supply the eagerness. It inherits the guard (an equal recompute suppresses the downstream cascade). .lazy() (or .dispose()) removes the eager puller — the value stays readable but reverts to lazy (recompute-on-read) behavior.

StateMachine

StateMachine[S, E] is a finite state machine backed by a reactive Cell, so its state participates in dependency tracking like any other reactive value. Construct it with StateMachine(ctx, initial, transition) where transition is a pure (state, event) -> next_state | None (returning None rejects the event). send(event) returns whether the transition was accepted; a self-transition to an equal state is accepted but suppressed by the Cell's PartialEq guard.

Reactive collections, async, and thread-safe contexts

lazily-py also implements the lazily-spec compute-layer MUSTs, each ported from its Lean formal model in lazily-formal:

• ReactiveMap / SourceMap / ComputedMap / SourceTree — keyed reactive collections (#reactivemap) with independent value/membership/order signals and atomic move. One generic ReactiveMap over a handle kind; SourceMap (input cells, adds set + eager entry) and ComputedMap (derived slots, lazy get_or_insert_with + eager materialize_all) are its specializations.
• QueueCell — a reactive FIFO queue (SPSC primitive with an MPSC-via-batch usage rule) with a pluggable QueueStorage backend. Reader-kind invalidation (head/len/is_empty/is_full/closed), bounded reactive backpressure via is_full, and the closure lifecycle (drain / Closed-distinct-from-Empty / idempotent).
• LatestDurableProjectionCore / LatestDurableProjection — keyed durable egress that converges each sink key to its latest desired epoch, permits one in-flight attempt per key, and generation-fences stale acknowledgements.
• reconcile_ops — move-minimized keyed reconciliation (LIS kernel).
• AsyncSlot / AsyncEffect — the async slot lifecycle with stale-completion discard, and cleanup-before-body effect scheduling.
• ThreadSafeContext — a lock-serialized batch boundary that coalesces writes into one invalidation pass.
• lazily.ffi — the C-ABI FFI boundary (LazilyFfiStatus, LazilyFfiMessageKind incl. CrdtSync = 3, LazilyFfiBytes).

The test suite gates on lazily-formal's lake build (every theorem checks) and mirrors the named Lean theorems as property tests. See SPEC.md for the full compliance surface.

Latest-durable projection — lazily.latest_durable_projection

Use LatestDurableProjection when the durable side effect is a projection of current state (for example, saving the latest document image), rather than a command log where every intermediate value must execute. upsert_desired keeps only the newest pending epoch per key; claim allows at most one sink attempt per key; ack_applied and fail_retryable must match that attempt's generation and epoch. reconnect advances the sink generation and safely requeues claimed work, so a stale actor can never clear newer intent.

from lazily import LatestDurableProjection

projection = LatestDurableProjection[str, str]({}, generation=1)
projection.upsert_desired("document", epoch=41, value="latest text")
attempt = projection.claim("document", generation=1).envelope
assert attempt is not None

# Await the external write in the caller's driver, then acknowledge its exact
# token. A concurrent epoch 42 remains pending even when epoch 41 succeeds.
projection.ack_applied(attempt.key, attempt.generation, attempt.epoch)

The graph-agnostic LatestDurableProjectionCore and the reactive LatestDurableProjection, ThreadSafeLatestDurableProjection, and AsyncLatestDurableProjection shells implement the same contract. The async shell's transitions are intentionally synchronous; only the external sink driver awaits I/O. All three shells replay conformance/egress/latest_durable_projection.json from lazily-spec v0.38.0 and correspond to the corrected lazily-formal v0.38.1 model.

Reactive queue — lazily.queue

QueueCell is a FIFO collection composed of reactive cells — not a new cell kind — that adds queue semantics (push to tail, pop from head) to the reactive graph. ThreadSafeQueueCell and AsyncQueueCell carry the same Core surface; the topic and work-queue families likewise ship ThreadSafe* and Async* shells. Nothing in the async family is async-coloured: storage reads, cursor advances, and lease decisions return plain values.

The reactive shell wraps a pluggable QueueStorage backend (default VecDequeStorage) and owns demand-driven reader-kind handles. It invalidates exactly the handles whose values changed — a push to a non-empty queue does NOT invalidate the head reader, while a pop does. Thread-safe shells serialize their whole public operation through a ThreadSafeContext; the same canonical fixture corpus and invalidation matrices replay against all three flavors.

from lazily import QueueCell, QueuePopError, batch

ctx = {}
q: QueueCell[str] = QueueCell(ctx)

q.try_push("a")
q.try_push("b")
assert q.head() == "a"
assert q.len() == 2
assert q.try_pop() == "a"

# Bounded queue → reactive backpressure via is_full.
bq = QueueCell[int].with_capacity(ctx, 2)
bq.try_push(1)
bq.try_push(2)
assert bq.is_full()
assert bq.try_push(3).label == "Full"   # reject at capacity
assert bq.try_pop() == 1
assert not bq.is_full()                  # pop freed a slot → is_full reader invalidated

# MPSC: multiple producers push inside one batch → one invalidation pass.
batch(lambda: (q.try_push("p1"), q.try_push("p2")))

# Closure: pop on closed+empty returns Closed (distinct from Empty).
q.close()
assert q.is_closed()
assert q.try_push("x").label == "Closed"

Competing-consumer work queue

WorkQueueCell provides exclusive FIFO claims with visibility deadlines, worker-scoped acknowledgements, tail retries, and bounded dead-letter handling. Item ids survive retries while each claim gets a fresh delivery id.

from lazily import WorkQueueCell

work = WorkQueueCell[str](ctx, visibility_timeout=10, max_deliveries=3)
work.push("job")
delivery = work.claim("worker-a", 100)
assert delivery is not None
assert work.ack("worker-a", delivery.delivery_id)

The reader-kind independence law is explicit: every successful operation derives its changed-reader set from the before/after state and resets those memoized handles in one batch. Unchanged reader kinds remain warm.

Transport-agnostic reactive ingress — lazily.ingress

A client consuming a remote stream usually grows four accidental mechanisms: a refresh() loop that re-reads whether the connection is healthy, a hand-rolled "is this message still relevant?" check, a reconnect path that forgets what was already applied, and transport-shaped consumer code that disagrees with itself per transport. Every one of those is a derive being simulated with a call. IngressCell makes them derives, and makes the transport a value the primitive never touches.

An envelope carries its own provenance (generation / sequence / stamped_at), so a WebSocket frame, an RPC response, and a polled page are the same input once decoded. Admission applies a normative order — lifecycle → generation fence → freshness → generation handoff → dedupe → ordering → backpressure → merge — and each keyed scope exposes four independent reader kinds plus three receipt channels.

from lazily import IngressCell, IngressEnvelope, IngressPolicy, Sum

ctx: dict = {}
ingress = IngressCell[str, int](ctx, IngressPolicy(reorder_window=4), Sum)

ingress.admit(IngressEnvelope("alpha", 1, 0, 0, 5))
assert ingress.value("alpha") == 5
assert ingress.readiness("alpha") == "ready"     # a derive, not a poll

# Out of order: buffered, so nothing a reader can observe moved.
ingress.admit(IngressEnvelope("alpha", 1, 2, 0, 4))
assert ingress.value_is_valid("alpha")            # the value reader stays warm

# The delivery that closes the gap flushes the run as ONE coalesced window.
ingress.admit(IngressEnvelope("alpha", 1, 1, 0, 2))
assert ingress.value("alpha") == 11
assert ingress.drain("alpha") == 11               # an egress, never an ack
assert ingress.suspend("alpha").from_sequence == 3

ThreadSafeIngressCell and AsyncIngressCell are the other two flavors of the same contract; all three replay the canonical lazily-spec/conformance/ingress/*.json corpus. Nothing in the family is async-coloured — an admission decision is a function of the fence, the watermark, the reorder buffer, and the observed clock, so there is nothing to await. Awaiting belongs to the transport, and the transport is outside the primitive by construction.

IPC — the lazily-spec wire protocol

lazily.ipc implements the language-agnostic lazily-spec wire protocol, so a Python graph's state can be mirrored to remote observers across processes and languages. The JSON encoding is byte-compatible with the Rust (lazily-rs), Zig (lazily-zig), and TypeScript (@lazily/signaling) bindings, and is validated against the canonical lazily-spec/conformance fixtures (vendored under tests/conformance/).

Two message kinds flow over any transport (WebSocket text, WebRTC data, FFI buffer):

• Snapshot — the full graph state at an epoch (nodes, edges, roots).
• Delta — an ordered batch of the 7 DeltaOp variants (CellSet, SlotValue, Invalidate, NodeAdd, NodeRemove, EdgeAdd, EdgeRemove) applied with epoch sequencing and fail-closed resync.

from lazily import (
    Snapshot, NodeSnapshot, EdgeSnapshot, ShmBlobRef,
    Delta, DeltaOp, IpcMessage,
)

# Build and serialize a snapshot — encode_json() returns transport-agnostic bytes.
snap = Snapshot(
    epoch=7,
    nodes=[
        NodeSnapshot.payload(1, "i32", bytes([1, 2, 3])),
        NodeSnapshot.opaque(2, "opaque-type"),
        NodeSnapshot.shared_blob(3, "text/plain", ShmBlobRef(0, 16, 1, 7, 999)),
    ],
    edges=[EdgeSnapshot(2, 1), EdgeSnapshot(3, 1)],
    roots=[1, 2],
)
wire = IpcMessage.of_snapshot(snap).encode_json()
assert IpcMessage.decode_json(wire).snapshot == snap

# An incremental delta carrying mutations.
delta = Delta.next(40, [
    DeltaOp.cell_set(1, bytes([10])),
    DeltaOp.invalidate(3),
])
IpcMessage.of_delta(delta).encode_json()

# The same frames over `msgpack`, the cross-language binary default: an
# externally tagged envelope over MessagePack maps keyed by the *json* field
# names. Map key order is encoder-defined, so conformance is a semantic round
# trip — `decode(encode(m)) == m` — never a golden byte string.
packed = IpcMessage.of_snapshot(snap).encode_msgpack()
assert IpcMessage.decode_msgpack(packed).snapshot == snap

A PeerPermissions boundary gates what is shared: it is default-deny, so only nodes a peer is explicitly allowed to read are serialized into a snapshot or delta — non-allowlisted nodes are omitted entirely.

Shared-memory blobs — ShmBlobArena

ShmBlobArena lets a Python process host blob payloads (not just carry ShmBlobRef descriptors). It is a bytearray-backed, append-only arena with a 40-byte header (LZSH magic + FNV-1a-64 checksum) and wraparound, ported from the lazily-rs ShmBlobArena<B> and byte-compatible with the Rust and Zig arenas. The module exports ShmBlobArena, ShmBlobArenaError (with its variant subclasses), and SHM_BLOB_HEADER_LEN.

Lossless tree CRDT — lazily.lossless_tree_crdt

LosslessTreeCrdt (#lzlosstree) is a single rooted concrete-syntax tree whose leaves own every rendered byte — render(tree) == source_text for valid, invalid, and unknown source alike. Where TextCrdt is a flat lossless floor, this is the structured tree that can itself be the wire authority. Element nodes own structure only; all text lives in leaf nodes tagged Token / Trivia / Raw / Error, so unknown/invalid spans round-trip exactly as Raw/Error leaves rather than being discarded.

from lazily import LeafKind, LosslessTreeCrdt, SeedElement, SeedLeaf
from lazily.lossless_tree_crdt import ROOT

tree = LosslessTreeCrdt(peer=1)
heading = tree.create_node(ROOT, None, SeedElement("heading"))
tree.create_node(heading, None, SeedLeaf(LeafKind.TOKEN, "# "))
title = tree.create_node(heading, None, SeedLeaf(LeafKind.RAW, "Título"))
assert tree.render() == "# Título"

# Op-based delta sync: fork, diverge, converge through a dotted frontier.
other = tree.fork(peer=2)
other.edit_leaf(title, 0, 0, "X")
tree.apply_update(other.diff(tree.frontier()))
assert tree.render() == other.render()

Leaf text embeds TextCrdt wholesale; child order is a fractional index (key_between); the clock is a Lamport TreeOpId. Anti-entropy is op-based over a dotted, non-contiguous version frontier (TreeVersionFrontier) — a dot set (contiguous prefix + sparse holes), never a per-peer max, so a missing interior op stays representable and re-requestable. Leaf-local wire offsets are UTF-8 bytes (byte_to_char). The wire codec (tree_update_to_wire / tree_update_from_wire) validates against lazily-spec's lossless-tree-delta.json, and all nine conformance/lossless-tree/ fixtures replay.

Command / RPC message plane — lazily.command

command-plane-v1 is an additive sibling to Snapshot / Delta / CrdtSync: four evented frames (CommandSubmit / CommandCancel / CommandEvents / CommandProjection) that carry command traffic, not cell state. lazily owns the envelope; the namespace owns the IpcValue payload, which lazily never decodes.

The single hard rule: terminal authority is the causal receipt. A command is terminal only when a terminal CausalReceipt for its command_id folds in (applied, or rejected — including the cancelled / superseded / timed_out reasons). observed / accepted / started events are progress only; a transport ACK is never terminal.

from lazily import (
    CommandPolicy, CommandRpcClient, CommandSubmit, DedupePolicy,
    applied_receipt,
)
from lazily.ipc import IpcValue

class Transport:
    def __init__(self):
        self.sent = []

    def send(self, message):
        self.sent.append(message)

client = CommandRpcClient(Transport())
cmd_id = client.submit(CommandSubmit(
    command_id="cmd-1", causation_id="cmd-1", source="plugin",
    target="controller", namespace="agent-doc", name="editor_route",
    authority_generation=1, idempotency_key="doc:run", deadline_ms=0,
    policy=CommandPolicy(DedupePolicy.SAME_IDEMPOTENCY_KEY, False, True),
    payload_type="agent-doc.editor_route.v1", payload_hash="sha256:…",
    payload=IpcValue.of(b"{…}"),
    required_features=["command-plane-v1"],
))
# `call` resolves ONLY on a terminal receipt — never an ACK or `accepted`.
client.ingest_receipt(applied_receipt("rcpt-1", cmd_id, "controller", 1))
assert client.poll_call(cmd_id).kind.value == "resolved"

CommandProjection is the pure reducer (generation guards, idempotency, cancel-before-terminal-only, terminal-conflict-fails-closed, reconnect equivalence); CommandRpcClient is the derived RPC facade. The wire codec validates against lazily-spec's message-passing.json, and all eight conformance/message-passing/ fixtures replay.

Benchmarks

Wall-clock benchmarks live in BENCHMARKS.md, covering both the in-library micro-suite (reactive core, keyed reconciliation, SourceMap, TextCrdt, CRDT plane) and a large spreadsheet-shaped scale suite that mirrors the lazily-rs / lazily-go scale groups (N input cells + N formula slots, formula[i] = input[i] + input[i-1]). The scale suite is measured up to a full 10,000,000-cell Google Sheets workbook (N = 5,000,000); a one-cell edit plus a 1,000-cell viewport read stays in the ~75 µs range regardless of sheet size, because the lazy pull model recomputes only the ~2 formulas that read the edited input.

make bench          # micro-suite
make bench-scale    # scale suite (default N = 1,000,000)

# or directly, with a custom size:
uv run python -m lazily.benchmarks
LAZILY_SCALE_N=5000000 uv run python -m lazily.scale_bench   # 10M-cell workbook

See BENCHMARKS.md for the full results, hardware, and honest notes on CPython's per-node overhead.

The lazily family

lazily-py is one binding in a cross-language reactive family — the same cell kernel, the same keyed collections and CRDTs, and the same lazily-spec wire protocol — so peers written in different languages talk to each other without a translation layer.

See lazily-spec for the canonical Snapshot/Delta schemas, the IPC Lean proofs of the epoch/memo/batch invariants, and the conformance fixtures every IPC-capable binding validates against. The language-agnostic formal model — the flat FSM kernel and the full Harel state chart — lives in lazily-formal.

Per-binding parity, with per-cell notes and platform carve-outs, lives in lazily-spec § Cross-Language Coverage — it is generated from coverage.json, so it does not rot the way a hand-copied table would.

Development

This project uses uv. Run the local CI-equivalent suite — type-check (ty), lint (ruff), the runnable README example, and the test suite — with:

uv run poe precommit

SPEC.md is the authoritative specification for the Python primitives and the lazily-spec compliance notes.