snakemake-logger-plugin-dart

A Snakemake 9+ logger plugin that streams workflow events — scheduled jobs, progress, the rule DAG, logs and errors — in real time over WebSocket to a desktop application that hosts the server.

The app is the server and this plugin is the client, so there is nothing in between: no polling, no intermediate service, and the endpoint stays on loopback. The other half of the integration is the Dart package snakemake_bridge, which provides the embedded server, typed event models and a state reducer — but any server implementing the protocol specification works just as well. For the design, the security model and the failure modes, see ARCHITECTURE.md.

Requirements

• Python >= 3.11
• Snakemake >= 9.0

Installation

pip install snakemake-logger-plugin-dart

Install it in the same Python environment as Snakemake (same venv/conda). Snakemake discovers plugins through the entry points of the active environment, so a plugin installed next door is simply invisible.

Verify that Snakemake found it:

snakemake --help | grep logger-dart
#   --logger-dart-address VALUE
#   --logger-dart-token VALUE
#   --logger-dart-flush-timeout VALUE

If that prints nothing, run pip show snakemake snakemake-logger-plugin-dart and check that both report the same Location.

Usage

Point the plugin at the address the app is listening on:

export SNAKEMAKE_LOGGER_DART_TOKEN=<token-generated-by-the-app>

snakemake --cores 4 \
    --logger dart \
    --logger-dart-address ws://127.0.0.1:8765

Passing the token through the environment rather than --logger-dart-token keeps it out of ps output, where any local user could read it.

Settings

How it behaves

The guiding rule is that the workflow never pays for the telemetry. A closed app, a dropped connection or a stalled consumer degrade what the app sees, never the run itself.

• emit() never blocks Snakemake. Events go onto an in-memory queue (10,000 entries) drained by a dedicated thread; if the queue fills, new events are dropped with a warning rather than applying backpressure to the run.
• The worker reconnects on its own with exponential backoff (0.5 s up to 10 s), for as long as the run is active.
• A replay buffer keeps the last 100,000 events. On every (re)connection the plugin sends hello, the app answers with replay, and the gap is refilled — so an app opened halfway through a run still catches up. Events dropped from the send queue remain in this buffer and are recoverable.
• Delivery is at-least-once: after a replay some events arrive twice, and the consumer deduplicates by the envelope's seq.
• On shutdown the queue is flushed (bounded by flush-timeout) and a bye event closes the run.

If nothing shows up in the app

In this order:

1. the address passed to --logger-dart-address carries the port the app actually printed;
2. the app and the plugin use exactly the same token — a mismatch is refused at the handshake with 401, before any event is sent;
3. snakemake --help | grep logger-dart finds the plugin;
4. the plugin lives in the same Python environment as the snakemake executable in use;
5. the app's server was already listening before the run started (or, if not, that it reconnects — the replay buffer covers a late start).

Development

pip install -e '.[dev]'
pytest

With the package also installed in the environment, pytest imports the installed copy and local edits are silently ignored. Run PYTHONPATH=src pytest to test the working tree.

Full guide — building the wheel, the Dart tests and end-to-end validation with a real workflow — in INSTALL.md.