kestrel-feature-observability

The single observability package for Kestrel Sovereign — one repo, one version, one source of truth for the whole observability domain — with two roles selected by the two entry-point groups (package boundaries need not match process boundaries):

• Base install (pip install kestrel-feature-observability) — the lightweight per-agent emitter Feature (the kestrel_sovereign.features entry point). It attaches an ObservabilityHook to the agent's hook system; every lifecycle event is emitted as an OpenTelemetry span (a session run_span with child tool_spans) via KestrelTracer, exported over OTLP/HTTP to whatever OTEL_EXPORTER_OTLP_ENDPOINT points at (e.g. a host-supervised Phoenix). No DB — this is what every agent gets. Prometheus metrics emit through the SDK's shared registry, so a single /metrics scrape stays coherent across the framework + every feature package.
• Host extra (kestrel-feature-observability[fleet]) — enables the FleetObservabilityHostFeature (the kestrel_sovereign.host_features entry point), which ships the single "Observability" console panel with a three-view sub-nav: Timeline (default) answers when activity happened with a temporal overview and compact span detail; Navigator answers where it fits with the hierarchical Tenant → Fleet → Agent → Subagent → Session → Turn → Events tree and a persistent inspector for the selected Turn/Event span; and Phoenix provides exhaustive trace forensics in the curated thin embed. Timeline and Navigator links preserve the exact OTel span ID in both directions, visibly highlight it, and report an honest containing-Turn or no-highlight fallback when that exact span is unavailable. All three are pure read-models over Phoenix's GraphQL through the same-origin /phoenix/graphql proxy (no store or new host routes). The HostFeature lives in the kestrel_feature_observability.fleet subpackage. Since the custom store/entities were retired, fleet/feature.py imports only the HostFeature/UIContributions contract from kestrel_sdk, so the host role is gated by the SDK version, not by an extra-only importable module: the [fleet] extra tightens the SDK pin (>=0.32.0,<0.33) to the range that exports that contract. The import/entry point stays guarded — if the resolved SDK is too old to export the contract, it degrades to None (with a warning logged) and the host skips the panel instead of crashing the feature scan.

Embed note: the browser console may log No HydrateFallback element provided to render during initial hydration on the Phoenix subtab — this comes from Phoenix's own React Router bundle (vendor-streamdown chunk) during SPA hydration and is expected upstream noise (cosmetic, no functional impact); the arize-phoenix bump that resolves it is blocked by kestrel-sovereign's fastapi pin.

This package supersedes the separate kestrel-feature-observability-fleet package, which is deprecated.

Installation

uv pip install kestrel-feature-observability

For real Prometheus output:

uv pip install 'kestrel-feature-observability[metrics]'

For the fleet host role (the Phoenix-embed console panel):

uv pip install 'kestrel-feature-observability[fleet]'

Both features are auto-discovered by Kestrel Sovereign via their entry-point groups — install the base package alongside kestrel-sovereign and ObservabilityFeature registers itself into every agent; install with [fleet] on the host and FleetObservabilityHostFeature registers at host scope.

Emitter transport

The hook emits OpenTelemetry spans via KestrelTracer (kestrel_feature_observability.tracing), exported over OTLP/HTTP. Endpoint discovery is OTel-standard:

• OTEL_EXPORTER_OTLP_TRACES_ENDPOINT — a full traces endpoint (used as-is), or
• OTEL_EXPORTER_OTLP_ENDPOINT — a base endpoint (the exporter appends /v1/traces), e.g. the host-supervised local Phoenix.
• OTEL_EXPORTER_OTLP_HEADERS — honored for auth.
• KESTREL_OTEL_PROJECT — the Phoenix project spans land in, stamped as the openinference.project.name Resource attribute. Defaults to kestrel-fleet, which the fleet console's curated Observability panel deep-links to — so per-agent hook traces show up there instead of Phoenix's "default" project.

When no OTLP endpoint is configured the tracer is a no-op — no provider, no exporter, no network — so the emit path costs nothing and the agent runs unaffected (Prometheus counters still fire locally). The span shape is the nested doll session ⊃ turn ⊃ tool ⊃ tool-start markers, one trace per turn (the session band is an attribute grouping — OpenInference session.id and the backward-compatible kestrel.session_id are stamped with the same value on every span — not a trace). A session-marker root is exported lazily on the first lifecycle event; each UserPromptSubmit starts a turn (a new trace root <agent> turn <n>, tagged kestrel.turn_id/kestrel.turn_index); each PreToolUse emits an instant <tool> (started) marker and each PostToolUse a child tool_span (tool name, real duration, success) parented to the current turn; Stop emits a turn <n> summary (the session stays live), and AgentTerminate/teardown emits the true session summary aggregating turns. orchestrator is the agent itself when self-driven, else inherited.

The turn <n> summary carries the per-turn stats — kestrel.tool_count, kestrel.error_count, kestrel.success_ratio, and kestrel.duration_ms (the go-forward unified key; the legacy kestrel.turn_duration_ms is still emitted alongside for back-compat) — and the session summary aggregates them across the session (kestrel.turn_count plus the same totals, with kestrel.duration_ms alongside the legacy kestrel.session_duration_ms). By default the turn root carries no prompt text; setting KESTREL_OTEL_CAPTURE_PROMPTS=1 opts in to stamping the turn's user prompt on its root span as input.value, truncated to KESTREL_OTEL_MAX_IO_CHARS (default 20000) characters.

Claude Code hook emitter

The same package ships a kestrel-obs-claude-hook console script so that Claude Code sessions land in the fleet Observability Timeline exactly like kestrel agents and talon runs. Claude Code's hooks system runs a shell command per lifecycle event with a JSON payload on stdin; this script turns those events into the identical span shape as the in-process emitter above — session ⊃ turn ⊃ tool ⊃ tool-start markers, one trace per turn — posted over OTLP/HTTP:

• SessionStart → an immediately-ended AGENT session-marker root (session.id and kestrel.session_id = the Claude session id, kestrel.agent_name = claude-code, kestrel.orchestrator = $KESTREL_OBSERVABILITY_ORCHESTRATOR else Direct). Both session attributes are retained on every span the hook emits.
• UserPromptSubmit → a labeled claude-code turn <n> root (a new trace). By default the root carries no prompt text; setting KESTREL_OTEL_CAPTURE_PROMPTS=1 opts in to stamping the payload's prompt on the turn root as input.value, truncated to KESTREL_OTEL_MAX_IO_CHARS (default 20000) characters.
• PreToolUse → an instant <tool> (started) marker under the current turn.
• PostToolUse / PostToolUseFailure → a completed tool_span under the current turn. PostToolUse fires after a tool succeeds; failed tools fire the separate PostToolUseFailure event (top-level error / duration_ms), which is recorded as a failed span. Duration prefers the payload's own duration_ms, else the gap to the paired PreToolUse.
• Stop → a turn <n> summary (the session stays live); SessionEnd (and a defensive staleness sweep) → the true session summary and state cleanup. The turn summary carries kestrel.tool_count, kestrel.error_count, kestrel.success_ratio, and kestrel.duration_ms (with the legacy kestrel.turn_duration_ms alongside); the session summary aggregates them (kestrel.turn_count plus the same totals, kestrel.duration_ms alongside the legacy kestrel.session_duration_ms).

A SessionStart with source compact/resume/fork preserves the live session (Claude Code reuses the session_id), so compaction never resets the turn counter or duplicates turn ids.

Each hook invocation is its own process, so a tiny per-session state file ($KESTREL_OBS_CLAUDE_STATE_DIR, else $XDG_STATE_HOME/kestrel-obs-claude, else $TMPDIR/kestrel-obs-claude/<session_id>.json, written atomically) carries the session/turn trace + span ids so spans across invocations share traces with no daemon. The openinference.project.name (project = repo) is resolved from the payload cwd's git remote (owner/repo), else $KESTREL_OTEL_PROJECT, else omitted, and cached per session. The script always exits 0, prints nothing to stdout (Claude Code interprets PreToolUse/Stop stdout for gating), never records the user prompt, and is an instant no-op — OpenTelemetry is never even imported — when neither OTEL_EXPORTER_OTLP_ENDPOINT nor OTEL_EXPORTER_OTLP_TRACES_ENDPOINT is set.

Wiring it into Claude Code

Hooks live in ~/.claude/hooks/ by convention; add a thin wrapper that pins the endpoint and execs the console script (keeping the endpoint out of your global env), matching the existing hook-directory layout:

# ~/.claude/hooks/obs-emit.sh
#!/usr/bin/env bash
exec env OTEL_EXPORTER_OTLP_ENDPOINT=http://127.0.0.1:6006 kestrel-obs-claude-hook

chmod +x ~/.claude/hooks/obs-emit.sh

Then register the wrapper on the lifecycle events in ~/.claude/settings.json. settings.json supports multiple hooks per event, so these are added alongside any existing entries (e.g. a PreToolUse guard) — never replace or reorder them. Note PostToolUseFailure alongside PostToolUse: failed tools fire a separate event, so without it errored tool calls would go unrecorded:

{
  "hooks": {
    "SessionStart":       [{ "hooks": [{ "type": "command", "command": "~/.claude/hooks/obs-emit.sh" }] }],
    "UserPromptSubmit":   [{ "hooks": [{ "type": "command", "command": "~/.claude/hooks/obs-emit.sh" }] }],
    "PreToolUse":         [{ "hooks": [{ "type": "command", "command": "~/.claude/hooks/obs-emit.sh" }] }],
    "PostToolUse":        [{ "hooks": [{ "type": "command", "command": "~/.claude/hooks/obs-emit.sh" }] }],
    "PostToolUseFailure": [{ "hooks": [{ "type": "command", "command": "~/.claude/hooks/obs-emit.sh" }] }],
    "Stop":               [{ "hooks": [{ "type": "command", "command": "~/.claude/hooks/obs-emit.sh" }] }],
    "SubagentStop":       [{ "hooks": [{ "type": "command", "command": "~/.claude/hooks/obs-emit.sh" }] }],
    "SessionEnd":         [{ "hooks": [{ "type": "command", "command": "~/.claude/hooks/obs-emit.sh" }] }]
  }
}

Making the console script reachable

The script must be on PATH (or invoked by absolute path) from any cwd, since Claude Code runs hooks from the project directory. Two options:

• Host venv (absolute path). If the host installs this package into a venv, point the wrapper at the absolute console-script path, e.g. exec env OTEL_EXPORTER_OTLP_ENDPOINT=http://127.0.0.1:6006 /opt/kestrel/venv/bin/kestrel-obs-claude-hook.
• uv tool install. uv tool install kestrel-feature-observability puts kestrel-obs-claude-hook on the uv tools PATH (~/.local/bin). Note that a uv tool venv is isolated: editing this package's source (or bumping a dependency) does not update an already-installed tool — re-run uv tool install --reinstall kestrel-feature-observability (or, for local development, uv tool install --editable . and reinstall after dependency changes) to pick up changes.

Privacy

The hook is observational — it never blocks, denies, or modifies. User-message content is not recorded (never stamped on any span); tool errors are truncated to 200 chars; exceptions in the hook are swallowed so they cannot affect agent operation.

Prompt capture is strictly opt-in: setting KESTREL_OTEL_CAPTURE_PROMPTS=1 (off by default) stamps the turn's user prompt on the turn-root span as input.value, truncated to KESTREL_OTEL_MAX_IO_CHARS (default 20000) characters — nothing changes unless an operator explicitly enables it at their own wiring point.

Dependencies

• kestrel-sovereign-sdk>=0.14.1,<1 — base Feature, Hook, and shared metrics module
• httpx>=0.27.0 — lightweight HTTP client (OTLP/HTTP export transport)
• opentelemetry-sdk + opentelemetry-exporter-otlp-proto-http + openinference-semantic-conventions — the OTel span builders + OTLP export
• Optional [metrics] extra → kestrel-sovereign-sdk[metrics] → prometheus-client
• Optional [fleet] extra → kestrel-sovereign-sdk>=0.32.0,<0.33 (the HostFeature contract for the Phoenix-embed console panel). No DB.

The base emitter has no runtime dependency on kestrel-sovereign (or any fleet package); it emits OTel spans over OTLP/HTTP. The [fleet] extra adds only the host SDK contract for the embed panel, so agents stay lightweight.

Development

uv pip install -e '.[test]'
uv run pytest

License

Apache-2.0