Witdem

Analytics for AI agents and multi-step AI applications.

Tracing tells you what executed. Witdem connects those executions to application outcomes: which paths ran, what they cost, where they failed, and whether they achieved the product goal you defined.

Website · Documentation · Get started · YAML contracts · Frameworks · Providers · Contributing · Security

See Witdem in action

From one-line setup and two-line instrumentation to a measured run, its execution graph, business outcome, cost, and contract evidence.

▶ Watch the 1:48 installation-to-analysis walkthrough

First run

Choose one backend launcher. Applications use witdem-sdk with either path.

Docker-managed (requires Docker Compose):

npx -y witdem@latest up

Native Python (requires Python and pipx):

pipx install witdem-analytics
witdem up

Both start the receiver, ELT worker, and dashboard, wait for health, open http://localhost:8501, and preserve data across down and upgrades. Their lifecycle vocabulary is the same:

npx -y witdem@latest status   # Docker path
witdem status                     # pipx path
npx -y witdem@latest logs receiver
witdem down

Check releases and compatibility without changing packages or data:

npx -y witdem@latest update --check
witdem update --check

See the upgrade guide for exact NPX, pipx, and SDK commands.

Add the SDK to an existing Haystack 3 project:

python -m pip install "witdem-sdk[haystack]"
export WITDEM_ENDPOINT=http://localhost:4318

From the application project, create the YAML contract and then edit it to describe that application's result:

witdem-sdk init

This creates a small .witdem/witdem.yaml index, a separate contract file, and a basic coding-agent skill at .witdem/skills/witdem/SKILL.md. It does not inspect or rewrite application code, and it will not replace an existing project unless --force is used.

To make the canonical skill discoverable through the common project skill location, initialize with witdem-sdk init --expose-agent-skill. This creates .agents/skills/witdem as a link; the skill remains owned by .witdem.

Wrap the pipeline once. Its existing run, run_async, and run_async_generator calls stay unchanged:

from witdem_sdk.integrations.haystack import instrument

pipeline = instrument(build_pipeline())
result = pipeline.run(data)

The project file only identifies the service and references definitions:

version: 2
service:
  name: support-agent
telemetry:
  capture_content: false
contracts: [contracts/answer.yml]

The contract names the result and independently reportable goal requirements. Application code reports their values with Witdem.report(...); Witdem never extracts business meaning from a framework-specific return object.

Run the application normally. The execution and its business result appear at http://localhost:8501.

What you get

• Actual execution paths, branches, loops, retries, tools, model calls, and failures
• Per-run latency, token usage, and cost when provider/model/usage evidence is sufficient
• Application outcomes and product-goal success defined by your application
• Provider and model comparisons using attributable time, usage, cost, and quality
• Workflow/path analytics and run-linked issue investigation
• Local DuckDB storage and a self-hosted dashboard; prompt and response capture is off by default

Witdem does not infer business success from a completed LLM call. Framework instrumentation records what happened technically; .witdem/witdem.yaml explains what the final result means.

Integration status

Statuses reflect current implementation and test evidence, not roadmap intent.

Witdem also normalizes retrieval, reranking, search, embeddings, OCR, tools, evaluations, image, audio, video, and extension operations without inferring vendors. See AI operations and evaluations.

See compatibility.json for machine-readable version constraints and Providers for the difference between native, framework-observed, and generic support.

Providers

Witdem currently has verified paths for:

• OpenAI and Azure OpenAI
• Anthropic, including Claude Agent SDK telemetry
• DeepSeek
• Mistral
• Amazon Bedrock
• Google Vertex AI
• Ollama
• Cohere (structurally tested, not live-validated)
• OpenRouter, including selected upstream provider and fallback metadata
• Hugging Face inference through smolagents or LiteLLM

OpenAI and Anthropic have dedicated SDK integrations. Haystack observes the generator actually used at each model boundary. The remaining provider examples use standard GenAI OpenTelemetry attributes or witdem_sdk.integrations.generic; this is explicit in every provider guide.

Cost is not assumed from a provider name. Witdem uses provider-reported money when present, or the versioned server catalog when provider, model, and token usage match a catalog entry. Unknown prices remain Not measured with a diagnostic reason.

Parallel and asynchronous execution

When sibling spans overlap, Witdem preserves the observed fan-out and fan-in rather than flattening them into a false sequence:

             ┌─ semantic retriever ─┐
execution ───┤                      ├─ answer
             └─ keyword retriever ──┘

The runnable Haystack parallel pipeline demonstrates this with two concurrent retrievers and an OpenAI answer component.

Self-hosting

NPX is the Docker-managed path; pipx is a genuinely native, Node-free and Docker-free path. The npm package never runs a postinstall script.

npx -y witdem@latest status
curl http://localhost:4318/readiness
curl http://localhost:8501/health

See Operations for lifecycle, ports, logs, data, backup/recovery, and Development only when contributing to Witdem itself.

Documentation

• Getting started
• Running Witdem with npx
• Concepts: tracing and business meaning
• Tutorial: defining a YAML contract
• YAML configuration
• Workflow definitions, compilation, and replay
• Haystack
• LangGraph
• LangChain
• Native/custom Python
• OpenAI Agents
• Anthropic and Claude Agent
• Provider support
• Pricing catalog and automated refresh
• Dashboard
• Troubleshooting
• Examples
• Operations
• CLI reference
• Upgrade and compatibility
• Architecture and data flow
• Evidence bundles
• Release notes
• Performance and lifecycle implementation report
• Development and contributing

Community

Witdem is built in the open. Bug reports, integration examples, documentation improvements, and focused code contributions are welcome.

• Read Contributing before opening a pull request.
• Participate under the Code of Conduct.
• Report vulnerabilities privately according to the Security policy.

License

Witdem is licensed under the Apache License 2.0.

Built with

Local analytics storage

Raw-to-serving transformation

Dashboard routing, queries, and tables

Witdem is independent of these projects. Their names and unmodified marks identify the technologies used and do not imply endorsement.