diwata-pulse-kit 0.1.1

Local-first telemetry collector + store for the Diwata stack.

Pulse

Status: v0.1.0 in progress — collector + store + query CLI/JSON API + read-only dashboard are built (Phases 1–4); see docs/working/backlog.md. Role (architecture): Event topology backbone / "System events + telemetry" source of truth for the Diwata stack. Stack: Python 3.12+, FastAPI, SQLite (v0.1.0) → Postgres + Redis (later). Dashboard is a static vanilla-JS page served by Pulse (no node/build step).

---

Thesis

Pulse is the telemetry and observability layer for the Diwata stack. Every product already emits, or will emit, small typed events at meaningful moments — Grain emits phase.close, task.close, suggest.accept, and workflow.next.stop_reason today; Assay will emit verification.completed and check-level results next. Pulse is the place those events land, are stored durably, and can be queried as metrics — so the stack can answer "how is the workflow actually performing, across products, over time?" without each product re-inventing storage and aggregation. Like the rest of the stack, Pulse is local-first, opt-in, and provider-neutral: nothing is collected unless explicitly enabled, everything runs against a local SQLite file before any hosted backend exists, and the ingest contract is a plain versioned JSON event that any producer (or agent/familiar) can POST or drop on disk — no vendor SDK, no proprietary wire format.

Pulse does not change how producers emit. Grain's telemetry_service.emit already POSTs to a configured endpoint and falls back to .grain/telemetry_queue.jsonl when no endpoint is reachable. Pulse simply is that endpoint (and the drain for those queue files). The contract Grain shipped in v0.4.0 — TelemetryEvent {event_type, version, timestamp, payload} — is the contract Pulse accepts, unchanged.

---

What Pulse is

• The collector — an HTTP endpoint (and a file-drop drain) that accepts versioned TelemetryEvent JSON from any Diwata producer, exactly in Grain's shipped shape.
• The store — an append-only, durable record of every accepted event. Starts as a single local SQLite file; promotes to Postgres when a hosted deployment needs it.
• The metrics surface — a CLI (pulse ...) and JSON API that read events back as aggregates: event counts, per-type/per-product/per-phase rollups, stop-reason frequency, verification pass rates — so grain metrics and Assay reports can read cross-run history from Pulse instead of only local files.
• Agent-usable / agent-agnostic — a familiar can push events and query metrics headlessly over HTTP or the CLI with --format json. No interactive step, no browser, no SDK lock-in.
• The dashboard — a read-only, local-first human view over that same read API, served by Pulse itself at GET / (pulse serve serves it). A self-contained static page (vanilla JS, no build step) that renders the metrics as plain tables + inline bars — see below.
• Opt-in and local-first — default off at the producer (Grain's gate) and self-hostable on a developer's own machine. Nothing leaves the machine unless an endpoint is configured.

What Pulse is NOT

• Not a producer. Pulse never instruments anyone's code. Producers own emission; Pulse owns transport, storage, and aggregation. (This is the boundary Grain's v0.4.0 contract states explicitly: "Grain only emits.")
• Not a tracer / APM. No distributed tracing, spans, flame graphs, or per-request latency profiling. Pulse ingests discrete domain events, not call stacks.
• Not an error tracker. It is not Sentry. Crash/exception capture is out of scope; events are intentional, typed, no-PII domain signals.
• Not Chronicle. Chronicle is the immutable decision + rationale + high-risk action audit trail. Pulse is operational telemetry (counts, durations, frequencies). They are distinct source-of-truth owners in the architecture.
• Not Lore. Pulse does not store normalized entity data (persons, companies, signals). It stores events about how the systems run.
• Not a vendor-locked telemetry SaaS. No proprietary wire format, no required client library. The contract is plain versioned JSON over HTTP or a .jsonl file drop.

---

Where Pulse fits

Layer	Owner
Workflow events (phase.close, task.close, suggest.accept, workflow.next.stop_reason)	Grain emits
Verification events (verification.completed, check results)	Assay emits (planned)
Transport · storage · aggregation · query	Pulse owns
System events + telemetry (source of truth)	Pulse (per docs/canonical/architecture.md)

See docs/working/v0.1.0_plan.md for the v0.1.0 scope, docs/working/integration_grain_assay.md for the producer-side contract, and docs/working/backlog.md for the phased build order.

---

Dashboard

Pulse ships a read-only dashboard — the human view over the exact same read API that agents/familiars consume as JSON. It is a self-contained static page (vanilla JS + CSS, no node, no build step, no framework); it only fetches the existing read endpoints and renders them as plain tables with simple inline bars (no chart library). It adds no business logic — every number comes from GET /v1/metrics/* and GET /v1/events.

It is served by Pulse itself, so running the collector serves the dashboard:

pulse serve                 # collector + dashboard at http://127.0.0.1:8006/
pulse dashboard             # print the dashboard URL (convenience over `pulse serve`)
pulse dashboard --open      # ...and open it in the default browser

Open http://127.0.0.1:8006/ to see:

• Overview — event counts by type and by source (/v1/metrics/counts).
• Phases — per-phase rollup: closes, tasks done, task closes, last-closed (/v1/metrics/phases).
• Stop reasons — workflow.next.stop_reason frequency (/v1/metrics/stop-reasons).
• Verification — Assay pass-rate, per-target trends, and over-time (/v1/metrics/verification). Lights up automatically once Assay emits verification.completed; shows an empty state until then.
• Recent events — the newest events (/v1/events?limit=).

The dashboard is read-only and, like the other read paths, stays open even when an ingest token is set (PULSE_INGEST_TOKEN gates only ingest). The JSON API is unchanged — the dashboard mounts at / (+ /dashboard.css, /dashboard.js, /static/) without touching any /v1/* route. Empty/zero states render gracefully against a fresh store; an optional auto-refresh toggle is off by default.