meshlogd 0.3.0

Event-sourced, eventually-consistent dataset that converges over a partition-prone Reticulum (LoRa) mesh.

meshlog

An event-sourced, eventually-consistent dataset that converges over a frequently-partitioning, low-bandwidth mesh. Built for the "trusted collaborators, no internet, private radio" case: a base camp and field teams (humanitarian assessment, SAR, expeditions) collecting a shared dataset with no server and no connectivity, reconciling whenever radios come into range.

The transport is Reticulum on real hardware (LoRa/packet radio/anything), but the engine is transport-agnostic and ships with an in-memory transport so you can run, test, and watch it converge with no radios at all.

Install

pip install meshlogd            # core engine (stdlib-only)
pip install "meshlogd[radio]"   # + Reticulum, for the real LoRa/packet-radio transport

Naming: the distribution and daemon command are meshlogd; the importable library is meshlog (import meshlog) — the trailing d follows the Unix daemon convention (cf. ssh/sshd), like how pyyaml installs but imports as yaml.

The idea in one breath

KoboToolbox already solves offline data entry but syncs to a central server over the internet. meshlog replaces that central sync with peer-to-peer anti-entropy gossip over a radio mesh, so the shared dataset converges with no server and no internet. Kobo (or any form UI) owns the forms; meshlog owns the replication layer Kobo doesn't have.

Why event sourcing makes this easy

The whole database is a grow-only set of immutable, per-author events. That single decision hands you the hard distributed-systems properties almost for free:

• Merging is set union. Union is commutative, associative, and idempotent, so the dataset is a CRDT: it converges regardless of the order or grouping in which events arrive across a partitioning network.
• Per-author contiguous feeds (the Secure-Scuttlebutt trick) collapse "what do you have?" down to one integer per author — a high-water-mark frontier. That digest is tiny enough to ride inside a Reticulum announce.
• Hash-chained feeds make history tamper-evident (we're trusted-collaborator, so we want integrity — just not anonymity).
• The materialized view is a pure fold over the event set, so two nodes with the same events compute byte-identical state. Conflicts on mutable scalar fields resolve by Last-Writer-Wins keyed on a Hybrid Logical Clock, not wall time and not arrival order — so concurrent edits from two islands pick the same winner everywhere.

Anti-entropy protocol (the entire thing)

when two peers can talk:
    each sends its FRONTIER  = {author: high_water_seq}
    on receiving a peer's frontier:
        push the slice of every feed above the peer's marks   (chunked)
    ingest is idempotent + gap-buffering, so overlaps/reorders are harmless

Multi-hop convergence is automatic: a node relays every author's events it holds, not just its own, so A's events reach C through B even if A and C never meet. That's the store-and-forward property, achieved without special-casing it.

Layout

meshlog/
  hlc.py            Hybrid Logical Clock
  event.py          immutable, hash-chained, content-addressed Event
  feedstore.py      append-only per-author log; frontier + delta computation
  reducer.py        deterministic fold -> materialized records (Kobo-style)
  transport.py      Transport interface + in-memory SimTransport (partitions)
  node.py           Node + the anti-entropy protocol
  rns_transport.py  real Reticulum transport (Link + Resource + announce)
demo_sim.py         base camp + 4 teams converge across staged partitions
test_convergence.py 200 randomized partition trials + fold-determinism check
meshlog/cli.py      CLI to run a node over a live Reticulum mesh (the `meshlogd` command)

Run it (no radios needed)

python -m demo_sim          # scripted scenario, prints the converged dataset
python -m test_convergence  # 200 randomized partition/merge trials

The demo islands three groups, has each collect independently, stages a conflicting concurrent edit, then heals the network in stages and verifies every node ends with identical frontiers, identical views, and the same LWW winner.

Run it for real (over Reticulum)

pip install "meshlogd[radio]"          # pulls in Reticulum (rns)
# Machine A:
meshlogd --name base   --data base.jsonl   --create R-100 site_assessment
# Machine B (same LAN to start; add a LoRa interface in ~/.reticulum/config later):
meshlogd --name team_a --data team_a.jsonl --create R-200 water_point
# type `view` at either prompt to watch the dataset converge

Verification status

• feedstore, reducer, hlc, node, transport, the demo, and the 200 randomized trials run and pass as committed.
• rns_transport.py / meshlog/cli.py are written against the Reticulum 1.3.x API (Identity, Destination, Link, Resource, Packet, announce handlers) and are faithful to the docs/examples, but have not been exercised on a live mesh here. Stand up two instances to validate end-to-end before field use.

Deliberately left as extension points

• Attachments / blobs. Events carry a content hash + size; the bytes are not inlined. Add a content-addressed blob store and transfer blobs lazily over RNS Resources, after structured events, so photos never starve assessment data on a slow link.
• Signatures. Event.is_authentic() currently checks the content hash. With RNS you already hold each author's Ed25519 key — sign the event id and verify on ingest to close the trusted-but-verify loop.
• Wire codec. Prototype uses JSON for readability. Swap _encode/_decode for MessagePack or CBOR to cut bytes on the radio.
• Compaction / snapshots. Grow-only logs grow. Add periodic signed snapshots
  • feed truncation once a quorum has acknowledged a watermark.
• Kobo bridge. Poll the KoboToolbox/ODK submission API (or read KoboCollect instances) and translate each submission into record.create + field.set events. That's the actual on-ramp to the HHI use case.

Reading list (steal from these)

• Secure Scuttlebutt — append-only per-author feeds + gossip replication for intermittent offline networks. This design is a close cousin.
• CRDTs — why union-based / LWW-register merges converge.
• Kulkarni et al. 2014, Hybrid Logical Clocks — the clock used here.
• Reticulum manual, Understanding + API Reference chapters — Link, Resource, and the announce mechanism.