tkati-dashboard — dataflow graph viewer

Reads a serialized tkati dataflow directory (see docs/dataflow-serialization.md) and serves a local web page rendering it as a graph — no live runtime process required, no manifest to maintain, just a directory of *.json/*.yaml/*.yml fragment files (freely mixable — both encodings merge into the same graph identically).

Usage

tkati-dashboard path/to/dataflow-dir

Then open http://127.0.0.1:8000/ in a browser. Pass more than one directory (or --flows-root, see Multiple flows below) to observe several dataflows from one dashboard instead of just this one. The page fetches the list of flows from /api/flows, then the selected one's graph from /api/flows/{id}/graph, and renders it with React Flow, laid out by dagre using each node's real measured box (from its label text, via canvas measurement — no DOM mount needed) rather than a flat grid, so a node with a long broker/topic string pushes its neighbors aside instead of overlapping them. The graph always lays out left-to-right.

A node consuming a stream with a consumer group doesn't get that edge's group/lag as a floating label — it gets its own stacked row inside the consuming node instead, right below the node's own header, one row per such input (so a fan-in node like a sessionizer joining two topics shows two rows, each fed by its own arrow landing directly on its row). An edge with nothing extra to show beyond its kind (no consumer group) stays a plain, unlabeled-beyond-kind line into the node's header. Source/sink nodes (kafka-topic, clickhouse-table) and processing nodes are colored differently. Click a node to fill the always-visible inspector panel on the right — the whole node box glows blue while selected, and every edge into or out of it (in the graph itself) turns the same blue and draws on top of any edge it crosses, so what a node actually talks to is easy to trace at a glance — with its full connection/config/schema details, plus every stream edge touching it and that edge's live consumer lag; drag the panel's left edge to resize it (the width is remembered across reloads).

The page follows your OS/browser light/dark preference (prefers-color-scheme) and switches live when it changes — no reload, and the graph keeps its layout and selection. There's no in-page toggle; the browser's setting (or DevTools' "Emulate CSS prefers-color-scheme") decides.

Try it with the bundled example

examples/simple-pipeline is the smallest interesting dataflow: two Kafka topics and one processing node (raw-events → dedup → deduped-events). Its connection blocks point at localhost:9092, so if you have a broker there, seed it with sample events first:

uv run python packages/tkati-dashboard/examples/simple-pipeline/seed_kafka.py
uv run tkati-dashboard packages/tkati-dashboard/examples/simple-pipeline

Then open http://127.0.0.1:8000/, click the raw-events or deduped-events node, and click "Load latest events" in the side panel to see the real messages the seed script just produced (raw-events includes two intentional duplicate event_ids so you can see what the dedup node in between is for).

For a bigger graph exercising fragment merging across topics.json, tables.json, nodes.json, and edges.json (two raw topics → dedup → a sessionize/enrich node → a ClickHouse table, plus a side branch straight to another table), see examples/analytics-pipeline — its topics aren't seeded with data, so "Latest events" there will error unless you produce to them yourself.

Since examples/ itself contains both as sibling directories, it also doubles as a ready-made --flows-root demo — tkati-dashboard --flows-root packages/tkati-dashboard/examples serves both as separate flows, switchable from the ☰ menu (see Multiple flows).

Options:

• --host (default 127.0.0.1)
• --port (default 8000)
• --flows-root DIR — auto-discover flows (see below); repeatable

A dataflow directory is re-read on every request to its /api/flows/{id}/graph, so editing the fragments and refreshing the browser picks up the change without restarting the server.

Multiple flows

A real deployment usually runs more than one dataflow per environment, and one dashboard instance can observe all of them at once instead of one directory per instance:

# name each flow directory explicitly (flow id = each directory's own basename)
tkati-dashboard path/to/orders-flow path/to/clicks-flow

# or point at a parent directory and let every fragment-containing subdirectory become a flow
tkati-dashboard --flows-root path/to/env/flows

The two forms can be combined, and --flows-root is repeatable. Either way, a flow's id is its directory's own basename — passing two directories with the same name is a startup error asking you to rename one of them. Unlike the explicit directories, a --flows-root's subdirectories are rescanned on every request to /api/flows, so adding or removing a flow directory there shows up without restarting the server; a subdirectory with no fragments in it (a stray README, say) is silently skipped rather than treated as an error.

With more than one flow configured, the graph page gets a ☰ menu in its top-left corner naming the currently selected flow — click it to switch to another one. It stays collapsed to that single button otherwise, so the canvas keeps its space once a flow is picked. The current flow is also reflected in the page's URL as ?flow=<id>, so a link to a specific flow's view can be bookmarked or shared. With only one flow configured (the common case, and the default if you pass a single directory as before), the menu doesn't appear at all — the page looks exactly as it always has.

Node panel

The panel on the right is always there, like an inspector rather than a popup — it shows a placeholder until you click a node, then shows that node's full metadata: connection settings, config, and schema (field → type). Clicking empty canvas clears the selection back to the placeholder. Every section has a clickable header to collapse/expand it — a collapsed section stays mounted, just hidden, so collapsing a live-fetching section and reopening it doesn't re-fetch. For a kafka-topic node, the panel also fetches two more, live views:

• GET /api/flows/{flow_id}/nodes/{id}/snapshot connects to connection.broker/connection.topic and shows the most recent messages on that topic (newest last), parsed as JSON, using a throwaway consumer group that never commits offsets. It only fetches on demand — click "Load latest events" — and a "↻ Refresh" button afterward pulls a fresh batch on request rather than automatically. Each message renders as its own pretty-printed JSON block (fields reordered to match schema, when there is one) rather than a table — real messages commonly carry 5-20 fields, too many to lay out sensibly as table columns in a side panel — with a small header showing that message's Kafka-level partition, offset, and timestamp, alongside its parsed body.
• GET /api/flows/{flow_id}/nodes/{id}/topic-stats shows the topic's partitioning and replication (per-partition leader/replicas/in-sync-replicas, flagging any under-replicated partition) and its topic-level config — retention.ms, retention.bytes, cleanup.policy, segment.bytes, compression.type, max.message.bytes — via tkati_dashboard.topic_stats, marking each value that differs from the broker default. Partition/replica info comes from the same topic metadata lookup as the snapshot/lag features (_kafka_metadata.py); the config comes from a separate read-only AdminClient.describe_configs() call.

These are two of the places tkati-dashboard talks to a live broker rather than just the serialized directory — best-effort conveniences for the panel, not something the graph view itself depends on: a broker that's unreachable, or a topic that doesn't exist, shows an inline error in that section instead of breaking the page.

Consumer lag

For every stream edge whose consumer.group_id is set and whose source is a kafka-topic, the page also fetches GET /api/flows/{flow_id}/nodes/{topic_id}/consumer-lag?group_id=... and shows the result in two places: the group/lag lines in that edge's stacked row inside the consuming node (see above), and, for either node that edge touches, the inspector panel's "Consumer lag" section (← other-node (group_id) for an edge consumed by the selected node, → other-node (group_id) for one where it's the topic being consumed). This is another place tkati-dashboard talks to a live broker: it looks up group_id's committed offset with Consumer.committed() and compares it to the topic's high watermark — it never subscribes or polls as that group, so it can't join it, trigger a rebalance, or otherwise disturb a real pipeline's consumer. A group that has never committed an offset is reported as fully behind (lag = the topic's full size); an unreachable broker shows lag: n/a instead of failing the page.

The number on the edge in the graph is the group's total lag summed over every partition. The inspector panel's section shows that same total and, beneath each one, a partition/lag table breaking it down per partition — the endpoint already returns both, so the breakdown costs no extra broker traffic, and it's what distinguishes a backlog spread evenly across partitions from one stuck partition holding all of it while the rest are caught up. The table appears once that edge's lag resolves (a loading or errored edge shows just its summary row), and scrolls within a fixed height so a high-partition-count topic doesn't crowd out the sections below it. The graph's own labels stay aggregate-only: their line count feeds the layout's node sizing, so per-partition lines there would reshape the graph on every refresh.

Lag is time-sensitive, so it doesn't just get fetched once. A ☰-style control in the canvas's top-right corner (shown whenever the graph has at least one such edge) works like Grafana's refresh picker: a "↻" button re-fetches every visible edge's lag immediately, and a dropdown next to it sets an auto-refresh interval (Off/5s/15s/30s/1m/5m, persisted across reloads, paused while the browser tab isn't visible). The inspector's "Consumer lag" section additionally has its own per-row "↻" to refresh just the one edge you're looking at, without waiting for the next tick or refreshing every other edge in the graph.

Validation

tkati_dashboard.dataflow.load_dataflow enforces the rules from the serialization doc: the directory must contain at least one *.json/*.yaml/*.yml fragment, node ids must be unique (or identically redefined) across fragments, edges must reference existing nodes, and a node's schema, when present, must use field types known to tkati_core.type_mapping — schema itself is always optional, since it isn't always on hand for a real-world node. A validation failure surfaces as an HTTP 422 with the error message, shown inline on the page instead of a blank graph.