windsock 0.1.0

Product analytics for AI agents — aggregate action-graph analytics (tool sequences, decision paths, completion funnels) across runs. Open-core.

Windsock

Product analytics for AI agents. Aggregate behavior analytics — tool-call sequences, decision-path funnels, completion rates — across thousands of agent runs. Open-core.

Per-trace tools (Langfuse, LangSmith, PostHog AI Analytics) let you debug one run. Windsock answers the questions you can only ask across all runs: which tool sequences lead to failure? where do agents get stuck? did my last prompt change improve completion rate? The unit of analysis is the agent action graph, not a single conversation.

Status: early build. Shipped so far: radar/ — the dogfood agent (below), the span/ingestion contract, the shared parser, and the windsock report CLI. Ingestion (ClickHouse), dashboard (React + Kea), and MCP server follow. See tracker: #1 (epic) · #2 (Radar).

---

Install

pip install windsock              # SDK + CLI — deps: opentelemetry + click, nothing else
pip install "windsock[server]"    # + the ingest server (Django, ClickHouse)
pip install "windsock[radar]"     # + the Radar demo agent (LangGraph, Ollama)

Instrument an agent in five lines:

import windsock.sdk as ws

ws.configure("spans.jsonl")                          # once at startup
with ws.session("research:acme", channel="web"):     # one session per run
    with ws.llm("draft", model="gpt-5.6-terra"):
        ...                                          # your provider call
        ws.usage(prompt=812, completion=344)
    ws.gate("relevance", "keep")                     # each decision, as made
    ws.outcome("complete")

Then windsock report spans.jsonl answers "is it working?" across every run.

Quickstart — windsock report

Point it at an OpenInference/OTel span file (JSONL, one span per line — see the contract) and see in one screen whether your agent is working. No new instrumentation, no server.

uvx --from git+https://github.com/Ref34t/windsock windsock report spans.jsonl

# or in a clone:
uv run windsock report data/spans.jsonl --by channel

Real output over Radar's own 1,911 runs:

Flags:

Flag	Meaning
--window 7d	Rolling window for rates + trend (24h, 7d, 2w, …). Rates are windowed, never all-time-only; the all-time line is printed alongside.
--by channel	Slice the class mix by any dimension your spans carry (channel, task, …)
--by version	Release-over-release drift: class mix per producer-stamped version — did the outcome mix move after a deploy? Unversioned runs group under (none)
--json	Machine-readable output — the stable report schema below
--session <run_id>	Single-run drill-down: outcome, gates (with judged item titles), tokens
--cost-per-mtok 2.50	Add an estimated cost line (USD per 1M total tokens)
--fail-over 'failure_rate>20'	CI/agent gate: exit 3 when the windowed rate breaches (failure_rate/completion_rate, >/<). Output unchanged; breach reason on stderr. Exit codes: 0 ok · 1 error · 2 usage · 3 breach
--min-runs 5	Liveness gate: exit 3 when the window holds fewer than N runs. A dead agent (zero runs) passes every rate gate — this catches it. Pair with --fail-over

There's also windsock audit spans.jsonl — a second-opinion pass over the agent's gate decisions: re-judges every unique titled keep/discard with an adversarial prompt (configurable LLM, default local Ollama) and prints flagged disagreements as a copy-paste-ready gold-candidate block. Report-only: it never writes anything — agents nominate, humans canonize. Cost: one LLM call per unique item in --window (default 24h); cap with --limit.

--json emits one object, stable under report_schema_version (currently 0): {report_schema_version, runs, window, window_end, windowed, all_time, trend, gates, tokens[, by, dimensions]} — windowed/all_time/each dimensions[] entry share one rates shape (runs, classified, unclassified, class_mix, completion_rate, failure_rate). Rates follow the contract's §5 definition: unclassified runs are excluded from numerator and denominator and reported separately. Exit code is 0 on success.

---

Instrument your own agent — the trace() SDK

No framework required. windsock.sdk emits the same windsock.* span schema Radar produces natively, so windsock report/audit and ingestion read it through the same parser. Two equivalent styles:

import windsock.sdk as ws
ws.configure("spans.jsonl")

# A) explicit — you own the session boundary
def my_agent(question: str) -> dict:
    with ws.session(intent=question, channel="cli"):
        with ws.tool("search"):
            hits = search(question)
        ws.gate("relevance", "keep" if hits else "discard", title=hits[0] if hits else None)
        ws.outcome("complete" if hits else "no_results")
        return {"answer": summarize(hits) if hits else None}

# B) one-line wrap — for an UN-instrumented body
agent = ws.trace(my_agent_without_session)   # each call == one windsock.run

session() vs trace() — pick one, don't stack them. trace() opens the run's root span for you; use it when the wrapped body has no session() of its own. If the body already calls session(), call it directly — you don't need trace(). Wrapping an already-instrumented body still produces exactly one run per call (the inner session() folds into the outer root, and the inner intent/dimensions win so the outcome row reflects the real intent, not the wrapper's function name), but it emits a one-time WindsockUsageWarning nudging you to drop the redundant wrap. Nesting two session()s directly behaves the same way: one root, inner intent wins.

---

radar/ — the dogfood agent

Radar is a research agent that tracks the AI agent-tooling / observability space. It exists to generate real, messy agent-behavior data to build Windsock against — and its OpenInference span schema is Windsock's ingestion contract. It's also genuinely useful: its output is ongoing competitive intel.

• Stack: LangGraph + openinference-instrumentation-langchain → OpenInference/OTel spans → JSONL (data/).
• Inference: self-hosted Ollama + Qwen2.5-3B-Instruct on a 4-core/8GB/no-GPU VPS — zero-cost, self-host-first. See docs/agdr/AgDR-0001.
• 7 tools: web_search, search_hn, search_reddit, fetch_url, classify_relevance, dedupe, summarize, store.
• 2 decision gates: relevance (keep/discard), dedupe (new/seen) — tagged on spans for the decision-tree view.
• Failures are real, not simulated: fetch_url fails naturally on JS/paywalled/timeout pages; junk searches get discarded — producing the 20–40% failure rate that makes "where agents get stuck" meaningful.

Run it (on the VPS)

# 1. Ollama with the model (AgDR-0001)
ollama pull qwen2.5:3b-instruct-q4_K_M

# 2. Deps + config
uv sync --all-extras
cp .env.example .env        # set OLLAMA_BASE_URL if not localhost

# 3. Single run
uv run python -m radar.run --topic "Voker agent analytics" --channel web

# 4. Batch backfill toward ≥1,000 runs
uv run python -m radar.run --backfill            # every (target × channel) seed
uv run python -m radar.run --backfill --limit 50 # first 50

Spans land in data/spans.jsonl — one trace per run, tool/gate/LLM spans nested. That file is the input the ingestion layer (build-order step 3) will consume.

Definition of done (#2)

• Runs end-to-end on the real space, on a schedule
• Emits OpenInference spans covering all 7 tools + both gates + session boundary
• Inference against the self-hosted OSS model (no external LLM API)
• ≥1,000 runs with an observed 20–40% failure rate
• Span schema documented → proposed as Windsock's ingestion contract
• Raw material for blog post #1 ("where my agent kept failing")

Layout

radar/
  config.py           # env-driven config
  instrumentation.py  # OpenInference/OTel → JSONL exporter (the ingestion contract)
  tools.py            # 7 tools + 2 gates
  agent.py            # LangGraph action graph
  seeds.py            # ~30 targets × 3 channels
  run.py              # single run + batch backfill
docs/agdr/            # decision records
data/                 # span JSONL (gitignored)