Witdem

Analytics for AI agents and multi-step AI applications.

Tracing tells you what executed. Witdem connects those executions to application outcomes: which paths ran, what they cost, where they failed, and whether they achieved the product goal you defined.

Get started · YAML contracts · Frameworks · Providers · Contributing · Security

See Witdem in action

From one-line setup and two-line instrumentation to a measured run, its execution graph, business outcome, cost, and contract evidence.

▶ Watch the 1:48 installation-to-analysis walkthrough

First run

Choose one backend launcher. Applications use witdem-sdk with either path.

Docker-managed (requires Docker Compose):

npx -y witdem@stable-0-1 up

Native Python (requires Python and pipx):

pipx install witdem-analytics
witdem up

Both start the receiver, ELT worker, and dashboard, wait for health, open http://localhost:8501, and preserve data across down and upgrades. Their lifecycle vocabulary is the same:

npx -y witdem@stable-0-1 status   # Docker path
witdem status                     # pipx path
npx -y witdem@stable-0-1 logs receiver
witdem down

Check releases and compatibility without changing packages or data:

npx -y witdem@stable-0-1 update --check
witdem update --check

See the upgrade guide for exact NPX, pipx, and SDK commands.

Add the SDK to an existing Haystack 3 project:

python -m pip install "witdem-sdk[haystack]"
export WITDEM_ENDPOINT=http://localhost:4318

From the application project, create the YAML contract and then edit it to describe that application's result:

witdem-sdk init --runtime haystack

This creates .witdem/witdem.yaml. It does not inspect or rewrite application code, and it will not replace an existing contract unless --force is used.

Wrap the pipeline once. Its existing run, run_async, and run_async_generator calls stay unchanged:

from witdem_sdk.integrations.haystack import instrument

pipeline = instrument(build_pipeline())
result = pipeline.run(data)

Add .witdem/witdem.yaml to describe what a useful result means:

version: 1
service:
  name: support-agent
  runtime: haystack
telemetry:
  capture_content: false
contracts:
  - name: answer
    mode: expression
    artifact:
      name: Support answer
      valid:
        non_empty: $.answer
    decision:
      name: Result validity
      expected: true
      observed: $.witdem.artifact_valid
    product_goal:
      name: Useful answer returned
      achieved: $.witdem.artifact_valid

Run the application normally. The execution and its business result appear at http://localhost:8501.

What you get

• Actual execution paths, branches, loops, retries, tools, model calls, and failures
• Per-run latency, token usage, and cost when provider/model/usage evidence is sufficient
• Application outcomes and product-goal success defined by your application
• Provider and model comparisons using attributable time, usage, cost, and quality
• Workflow/path analytics and run-linked issue investigation
• Local DuckDB storage and a self-hosted dashboard; prompt and response capture is off by default

Witdem does not infer business success from a completed LLM call. Framework instrumentation records what happened technically; .witdem/witdem.yaml explains what the final result means.

Integration status

Statuses reflect current implementation and test evidence, not roadmap intent.

Witdem also normalizes retrieval, reranking, search, embeddings, OCR, tools, evaluations, image, audio, video, and extension operations without inferring vendors. See AI operations and evaluations.

See compatibility.json for machine-readable version constraints and Providers for the difference between native, framework-observed, and generic support.

Providers

Witdem currently has verified paths for:

• OpenAI and Azure OpenAI
• Anthropic, including Claude Agent SDK telemetry
• DeepSeek
• Mistral
• Amazon Bedrock
• Google Vertex AI
• Ollama
• Cohere (structurally tested, not live-validated)
• OpenRouter, including selected upstream provider and fallback metadata
• Hugging Face inference through smolagents or LiteLLM

OpenAI and Anthropic have dedicated SDK integrations. Haystack observes the generator actually used at each model boundary. The remaining provider examples use standard GenAI OpenTelemetry attributes or witdem_sdk.integrations.generic; this is explicit in every provider guide.

Cost is not assumed from a provider name. Witdem uses provider-reported money when present, or the versioned server catalog when provider, model, and token usage match a catalog entry. Unknown prices remain Not measured with a diagnostic reason.

Parallel and asynchronous execution

When sibling spans overlap, Witdem preserves the observed fan-out and fan-in rather than flattening them into a false sequence:

             ┌─ semantic retriever ─┐
execution ───┤                      ├─ answer
             └─ keyword retriever ──┘

The runnable Haystack parallel pipeline demonstrates this with two concurrent retrievers and an OpenAI answer component.

Self-hosting

NPX is the Docker-managed path; pipx is a genuinely native, Node-free and Docker-free path. The npm package never runs a postinstall script.

npx -y witdem@stable-0-1 status
curl http://localhost:4318/readiness
curl http://localhost:8501/health

See Operations for lifecycle, ports, logs, data, backup/recovery, and Development only when contributing to Witdem itself.

Documentation

• Getting started
• Running Witdem with npx
• Concepts: tracing and business meaning
• Tutorial: defining a YAML contract
• YAML configuration
• Workflow definitions, compilation, and replay
• Haystack
• LangGraph
• LangChain
• Native/custom Python
• OpenAI Agents
• Anthropic and Claude Agent
• Provider support
• Pricing catalog and automated refresh
• Dashboard
• Troubleshooting
• Examples
• Operations
• CLI reference
• Upgrade and compatibility
• Architecture and data flow
• Release notes
• Performance and lifecycle implementation report
• Development and contributing

Community

Witdem is built in the open. Bug reports, integration examples, documentation improvements, and focused code contributions are welcome.

• Read Contributing before opening a pull request.
• Participate under the Code of Conduct.
• Report vulnerabilities privately according to the Security policy.

License

Witdem is licensed under the Apache License 2.0.

Built with

Local analytics storage

Raw-to-serving transformation

Dashboard routing, queries, and tables

Witdem is independent of these projects. Their names and unmodified marks identify the technologies used and do not imply endorsement.