groom

A local, single-process web dashboard for workhorse agent-workflow operator gates. Run groom serve on your host while author/coder (or any other workhorse-based workflow) runs in the background. groom shows every running workflow, pages you the moment one blocks on an operator gate, and lets you answer the gate from the browser. The shared await_operator node blocks in place rather than exiting, so the run keeps running. The answer travels over the run's own workhorse control socket and the run wakes at once. The gate file stays the durable record, and the fallback channel when nothing is listening.

Beyond the gates, the same dashboard is the fleet's telemetry reader — every run's spans, per-node timings and error status, filterable without leaving the browser:

How it works

• Each run pushes its telemetry to groom over OTLP. A run on groom's own host also advertises its run directory, workspace and pid. groom reads both from local disk to serve the run's Files and Diff panels and to answer its gates. A run with no groom listening behaves the same.
• groom holds live fleet and browser state in memory and pushes JSON state to open browser tabs over a websocket; the browser renders it with Preact + htm (the vendored htm/preact standalone build — no build step, no node_modules). Every shape the socket pushes is also fetchable over HTTP, so a tab whose socket has gone quiet resyncs instead of going stale. Those pushes are edge-triggered — something changed, so tell the tabs — which covers everything except the half of a run row that is derived from the clock: the liveness dot, silent 4m, in node 12m. The event that should turn a row dead is the run ceasing to emit, and an absence cannot be pushed, so the run list is additionally re-rendered and broadcast every GROOM_LIVE_TICK_S (5s), skipped entirely when no tab is connected. That is what makes the dashboard safe to leave open and read without refreshing. A selected run loads its span summary, recent logs, and metrics from SQLite once when its watch subscription opens. A loading indicator covers that initial read. The history lists timestamped log records, completed spans, and metric observations, including runs with no script logs. The type filter shows up to 60 recent records per type; an explicit run selection includes older spans beyond the fleet’s activity window. Committed OTLP batches update the history in memory and push it immediately to the watching tabs; the clock tick only refreshes elapsed labels, with no database reads. Tabs share the watched history, which is released when the last tab leaves. Reopening the run loads a fresh snapshot, and reconnecting restores the subscription automatically.
• Fleet rows and the detail pane project the same received telemetry. Operator and machine wait gauges carry their gate path and question; node, activity, elapsed time and terminal state come from the run's exports. The freshness label shows how old that evidence is. Silence leaves the last reported state visible; groom does not infer completion by reading checkpoints, run.json, or the host PID. A resumed process exports a new session identity, replacing the prior session's live values as its telemetry arrives. The gate pane shows only the latest question set, in full, including nested headings. Questions render as Markdown (marked, sanitized with DOMPurify before insertion since the content is LLM-authored). The automatic attendant receives that same latest question set. Earlier exchanges stay in the gate file; the pane has no additional context disclosure. Each workflow row can expand a git diff of its working tree (rendered with diff2html). All front-end assets are vendored locally; nothing is loaded from a CDN at runtime.
• Gates travel over the run's control socket, with the file as the record. An answer goes socket-first — groom asks the run what it is waiting on and delivers the answer on the run's own path spelling; the run persists it into the gate file before acknowledging — and falls back to writing the gate file from outside only when nothing answers on the control socket. Wait telemetry updates native gate discovery, including after groom reconnects. Socket questions are queried when submitting an answer so the write uses the waiting process's exact path. Machine waits show their pending result without an operator answer form.

Install

groom will publish to PyPI as stablemate-groom, because the name groom belongs to an unrelated project. Until that release, install it from a checkout of the stablemate workspace:

pipx install ./groom        # isolated CLI on your PATH

Requires Python ≥ 3.12. groom is an optional add-on — no base workflow requires it — and nothing has to be configured on the workflow side: workhorse probes for a collector at the default port when a run starts.

Usage

groom serve                       # binds 127.0.0.1:8787 — loopback only, the default
groom serve --host 0.0.0.0       # all interfaces: for runs on another machine (see note)

From a checkout of this workspace, prefix each command with uv run (uv run groom serve) to use the working tree instead of the installed copy.

Binding. groom binds loopback by default because it has no authentication, and it answers operator gates. A run on another machine needs --host 0.0.0.0. That choice prints a one-line exposure warning, and --allow-non-loopback acknowledges it. Only do that on a trusted network.

Telemetry collector (OTLP) + AFK alerting

groom is also the default local OpenTelemetry collector for workhorse runs. The same uvicorn process and port expose standard OTLP/HTTP receivers — POST /v1/traces, POST /v1/metrics and POST /v1/logs — so an ordinary run

workhorse-coder run                # probes the endpoint at start; exports if groom answers

(No env var is needed while groom serve is up on the default port — workhorse enables telemetry when it finds a collector listening. Point elsewhere with OTEL_EXPORTER_OTLP_ENDPOINT, or opt out with WORKHORSE_OTEL=0.)

It streams node/agent-turn spans, gas/heartbeat metrics, and the log records of the engine and every node it runs in-process into groom. Because a pushed span carries its own identity, every run appears with no discovery step. Spans and metrics persist in an embedded SQLite file (groom.db in the platform data dir; override with GROOM_DB), searchable from the dashboard's Telemetry pane, via GET /traces?run=…&node=…&status=…& slower_than=…, or with raw sqlite3 queries. Rows older than GROOM_RETENTION_DAYS (30) are pruned at startup and on a periodic tick — but only for a run whose telemetry is already archived to disk (below), so the window bounds the database, not the evidence.

A batch the store cannot take is answered 503 with a Retry-After, not 500: the exporter re-sends it, where an unhandled error would have said "stored" by omission. That is a deliberate trade of duplicates for losses — spans and turns are keyed and absorb a re-send, metrics and logs are plain appends and a retried batch can land twice. Duplicate rows are cosmetic in tables read by recency; a dropped batch is gone. The connection behind it recycles itself on a SQLite error and retries once, so a wedged handle heals on the next request instead of at the next restart. Whether any of that has happened is in GET /api/state under store — reopens, failures, last_error, and ok, which is false only while a failure is newer than the last statement that worked.

The liveness counters — workhorse.run.heartbeat, workhorse.turn.heartbeat, workhorse.cap_wait.heartbeat — are never stored at all: insert_metrics drops them at the door. They tick every ~10s for every open node, so on long runs they outgrow everything else combined: in one real store the run heartbeat alone was 1.77M of 2.21M metric rows, and a later one hit 2.8 GB with heartbeats ~80% of 18M rows. Nothing reads their history — the alert rules fold them into memory at ingest, and groom status asks the running server for that in-memory picture over HTTP (/api/live) — so persistence bought nothing and cost most of the file. Rows written before this change drain on age alone: the never-archived series are swept by the ordinary prune without waiting for an archive behind them, because there is none. The seven liveness gauges (turn.active, turn.idle_s, turn.elapsed_s, wait.active, wait.elapsed_s, node.active, node.elapsed_s) are stored and read live but not archived, on the same reasoning: climbing idle on an active turn diagnoses a wedged agent now, while an explicit wait is expected parked work, and neither question is asked of a run that ended a month ago. models.UNARCHIVED_DELETABLE_METRICS names them, in one place, because what may be deleted without a copy is a rule that has to be readable rather than inferred.

Archival: SQLite is the live tier, disk is the permanent one

A run's spans and logs are not lost when they age out. Once a run has been quiet longer than GROOM_RETENTION_DAYS, an archival pass writes everything it holds — every span, every log, and the four budget metric series — as one telemetry.jsonl beside that run's transcripts, and only then does prune delete the rows. Write, move, delete, in that order: prune is fail-closed and refuses to touch a run the archive has not reported back.

<data dir>/transcripts/<run-id>/…      live runs, the harvester's only target
<data dir>/archives/<run-id>/          frozen: transcripts + telemetry.jsonl

Two roots on one filesystem, so promotion is a single os.rename. An archived run is frozen — never re-archived, never resumed, never written again. A second run claiming a name already under archives/ is archived under a derived name (R-a1b2c3d4) rather than overwriting it.

The file is one JSON object per line, ordered by ts, discriminated by kind (span / log / metric), with line 1 a manifest naming the run and its row counts. Records carry run_id, node, generation and seq, which is exactly the key the per-node transcripts are classified by — so the telemetry joins the transcripts with no correlation step. The heartbeats and the liveness gauges never reach it; they are for the in-memory picture, not for history.

The archive has no retention of its own and no index table: the set of archived runs is what is on disk under archives/, read by listing it. A sweep interrupted halfway resumes from the manifest rather than rewriting the file.

groom archive status         # root, frozen runs, pending, held, last sweep
groom archive ls             # what is frozen (--long for the manifests)
groom archive show RUN       # stream one run's telemetry.jsonl
groom archive now --dry-run  # force a pass; --dry-run writes nothing

The pass runs in its own task at groom serve launch, seeded already-expired so a restart archives immediately, then every GROOM_ARCHIVE_EVERY_S (6h) for at most GROOM_ARCHIVE_RUNS_PER_PASS (25) runs — oldest first, the rest reported as pending. Nothing in it may raise into groom's tick.

The pane shows the runs connected right now. A month of retention means the unfiltered strip is mostly runs that ended days ago, and a dashboard is for watching, so a run card (and its spans — a span table listing a hidden run's nodes is telemetry from nowhere) appears only while the run is live by the same heartbeat predicate the fleet rows use. History is one tick of show ended away (GET /traces?show_ended=1), and searching an explicit run= always finds it: naming a run is asking for that run, finished or not.

Native runs are first-class dashboard rows, not just telemetry. A run on groom's own host advertises its run_dir, workspace path, pid, and per-node activity label on the OTLP resource; groom materializes a fleet row from that (keyed by run_id), shows what it is doing ("coder · reviewing ACME-A2JX"), and — because it shares the host — serves the row's Files/Diff panels and answers its operator gates straight from the local filesystem (groom.localfs). The native test is self-validating. groom draws the row only for a run whose run_dir it can read locally, so a run whose paths don't resolve on this host never double-lists.

Alert rules run on every ingest plus a periodic tick, and page you through browser notifications and an away-from-keyboard push — configure GROOM_NTFY_TOPIC (posts to https://ntfy.sh/<topic>; override the server with GROOM_NTFY_URL) and/or GROOM_WEBHOOK_URL (JSON {"title","message"}):

Groom deliberately does not alert on total run age: resumptions reuse a run identity and multi-day runs are normal. Use workhorse's WORKHORSE_MAX_RUNTIME_S when a run needs a hard wall-clock limit.

The crux: workhorse heartbeats for as long as its process lives, so silence and slowness are different observations. STALL means the run stopped emitting (dead/killed/frozen); STUCK means it is alive and parked. Alerts dedupe per (run, rule).

ENDED covers the case neither absence rule can, because a terminal is precisely what retires a run from both of them: the run finished. That is the loudest event on a queue — nothing is executing until someone launches the next run — and until it existed it was the only one that paged nobody. It fires on every ending, naming the terminal and the error class, because from outside "it crashed" and "it succeeded" are the same silence. A resume reuses the run id and clears the fired set, so the next session's ending pages on its own.

A process killed before it exports its terminal remains in its last reported state with an aging freshness label. STALL reports the missing telemetry; completion requires the run's terminal export.

BLOCKED and WAITING cover the mirror image: a run parked on an operator gate is behaving correctly, so no rule described it, and STUCK skips an open wait by design. Yet it is the only alert whose subject can end it — the human reading the page is the missing input. Answering the gate closes the wait and retires both. Run groom serve under nohup/systemd as the always-on collector (the consuming repo's make groom-serve target does exactly that).

Where is my run right now?

Spans export on completion, so a run's current node — the only one that matters when it will not finish — has no row in spans and never will while it hangs. The live picture therefore comes from metrics, which ship on a timer regardless of span state:

uv run groom status              # every live run: open node, node age, agent idleness
uv run groom status --run <id>   # one run
uv run groom status --json       # same data, machine-readable

Read it like this:

Wait until a run needs attention

uv run --all-packages groom wait --run RUN --json
uv run --all-packages groom wait --run RUN --until blocked,ended

wait holds one WebSocket connection to groom serve and stays silent during normal progress. It returns one event and exits when the named run is blocked, ended, stuck, stalled, or gave up. --until selects a comma-separated set from blocked,ended,stuck,stalled,gave-up,churn,watchdog,died,waiting. There is no timeout; Ctrl+C cancels the monitor without stopping the workflow.

Current conditions are checked in the initial server snapshot, including runs hidden from the dashboard. A run not yet known to the server is watched until it reports a matching condition. This uses the server's current cache, not an archive search: an ending evicted from that cache is no longer available. Machine, capacity and retry waits do not count as operator gates. Starting a new waiter while the same condition remains active returns that condition again.

--run takes the exact telemetry run ID. GROOM_URL selects the collector, as for groom status. Both CLI and server must support the wait protocol; an older server is reported as a monitoring error. JSON output contains run_id, event, node, question, gate_path, terminal, and message; unavailable context is an empty string. Exit 0 means a matching event, even if the workflow failed; 1 means the connection or protocol failed, 2 means invalid arguments, and 130 means cancellation. Disconnections are reported instead of silently reconnecting across a gap in monitoring.

What was it saying? (groom logs)

status says where a run is; logs say what it was doing on the way there.

uv run groom logs --run <id>                 # everything, oldest-first
uv run groom logs --run <id> --node select_item
uv run groom logs --level WARNING            # a FLOOR: WARNING + ERROR + FATAL
uv run groom logs --contains "over budget"

Deterministic (non-agent) nodes appear here because workhorse runs them in-process and hands each one a per-node logger, preserving the node's decisions alongside its spans and artifacts.

Records carry the same run_id/run_dir resource as the spans, so a log line joins to its node span and its on-disk artifacts with no correlation step. They are correlated by an explicit node attribute rather than trace_id, which is zeroes: workhorse never makes its node spans current, so there is no ambient context for the SDK to attach.

No alert rule fires on logs, deliberately — liveness is already answered by the heartbeat metrics, and paging on log content would mean guessing which strings are worth waking someone for, per workflow. Logs are for reading once a metric has told you where to look. They expire on the same window as everything else and are archived with it — one signal, one clock, one file.

Test runs are not telemetry

A test suite is not a run anyone comes back to, and on a machine where groom serve is up it is by far the loudest producer: one make test of the workflows suite wrote a six-figure number of spans, burying every real run in the fleet view. So test telemetry is kept out at both ends — workhorse declines to auto-enable export from a test process (WORKHORSE_OTEL=1 still forces it on), and the receivers drop any record whose run_dir is certainly a test dir — one containing pytest-of- or .workhorse-test/. Neither undoes what an older producer already wrote:

uv run groom purge-tests --dry-run   # what would go
uv run groom purge-tests             # delete it, then VACUUM to shrink the file

purge-tests casts the wider net: it also evicts runs whose dir is a tempfile.mkdtemp scratch dir (/tmp/tmpXXXXXXXX/…), which is how the largest single junk run in a real store got there. That one is a guess — a genuine run launched from a mkdtemp directory matches too — so it is confined to a command you run deliberately and can preview with --dry-run, rather than to the ingest path, where the same guess would discard evidence unasked. Runs are identified by their run dir in both cases, never by name.

Reading the store afterwards

The same file the dashboard reads answers the after-the-fact questions — what a run cost and where it looped, what it would have cost on another model set, what a node actually said, and where the wall clock went:

uv run groom cost --run <id>          # money and rework per node, with /work turns-per-item
uv run groom prices --run <id>        # re-price the same turns against another model set
uv run groom loops --run <id>         # which review→rework loops converge, and which do not
uv run groom transcript --run <id>    # the reasoning and tool trace prompt.md omits
uv run groom profile --run <id>       # what occupied the wall clock
uv run groom db-path                  # …and the SQLite file, for anything the CLI does not ask

Cost coverage is not uniform — it is harness × provider, and the two ways of not pricing a turn (nothing reported, versus a reported 0) differ in how visible they are. Every command above says how much of itself is real rather than presenting a partial sum as a complete one.

Those commands, the schema and its one footgun, and worked SQL are in docs/STORE.md.

See docs/features/groom/ at the repo root for the full design — groom's own OKF book, queryable with ostler graph --surface groom.

Map

• groom/: the package behind the groom command: the server, the collector, the store and the dashboard assets.