WeHarness tracing for Python

Send agent, model and tool spans to WeHarness using OpenTelemetry and OTLP/HTTP. Requires Python 3.10 or newer and a project ingest key from the WeHarness Tracing page.

import weharness_tracing as wh

# Set WEHARNESS_INGEST_KEY in your server environment, never in browser code.
wh.init(service="my-agent")
with wh.agent_run("support"):
    with wh.tool_call("lookup_order"):
        pass  # Call your tool here.
wh.shutdown()

The default endpoint is https://api.weharness.io/otlp/v1/traces. Override it with WEHARNESS_OTLP_ENDPOINT if needed. Set WEHARNESS_REVISION to correlate traces with a source revision.

Known prompt, output, tool-payload and stacktrace attributes are filtered by default. This filter is not a general secret detector: do not put secrets or personal data in span names, error messages or custom metadata. Content capture requires explicit configuration and appropriate project settings. An ingest key sends traces; it does not authorize reading them.

An existing OpenTelemetry tracer provider is reused. Background batches are bounded; call shutdown() to flush pending spans before the process exits.

Local evaluation runner

The Python package includes weharness-run. Save the cases returned by the WeHarness MCP evaluate_local_start operation as cases.json, then run your agent in the shape it has:

weharness-run --cases cases.json --command "python agent.py" --timeout 300 --out report.json
weharness-run --cases cases.json --function my_agent:graph --out report.json
weharness-run --cases cases.json --url http://127.0.0.1:8000/invoke \
  --start "uvicorn app:app --port 8000" --out report.json
weharness-run --cases cases.json --acp "opencode acp" --out report.json

• --command runs a CLI entry point. Each case input goes to stdin, or replaces {input} in the command. The output is what it prints.
• --function module:attr (or path/to/agent.py:attr) imports your code from the working directory and calls it in a separate Python process. It accepts plain and async functions, generators, objects with invoke, ainvoke or run (LangGraph graphs, LangChain runnables, runner objects), OpenAI Agents SDK Agent objects (run with Runner.run), and classes that take no arguments. A case input that is a JSON object is passed parsed, a LangGraph graph receives {"messages": [{"role": "user", "content": ...}]}, anything else gets the text; --input-as text|json|messages overrides this. The output is the return value: a string as is, an OpenAI Agents SDK result's final_output, a LangGraph state's last message content, anything else as JSON text. What the function prints is not part of the output. --python chooses the interpreter (default: the active virtualenv's, else the one running weharness-run).
• --url POSTs {"input": "<case input>"} to the endpoint (change it with --body '{"message": {input}}', add headers with --header) and uses the reply as the output: JSON as JSON text, anything else as text. A non-2xx reply gives exit code 1 and http_status. --start starts the server under the runner, waits until it accepts connections, runs the attempts one at a time and stops it afterwards. Without --start the runner cannot observe or isolate the server's own calls (effects is null).
• --acp "<command>" runs an agent that speaks the Agent Client Protocol over stdio, such as opencode acp. Install the extra first: pip install "weharness-tracing[acp]" (the official ACP Python SDK). Each attempt starts the command, opens one session in the working directory and sends the case input as the prompt. The output is the agent's reply text, and the attempt adds acp: the stop reason, the tool calls, the usage and cost the agent reported, and each permission decision. The agent gets no file system or terminal capability from the runner and uses its own tools. Nobody answers permission requests during a run: a request that names a path outside the working directory is rejected, and any other is allowed. The agent keeps its own MCP configuration and model. It is not a Python process, so the side effect interceptor does not cover it (see below). --python chooses the interpreter that has the ACP SDK (default: the one running weharness-run).

For each repetition (--repeats, default what WeHarness asked, else 3; --parallel N runs N attempts at a time) the report records the output, exit code, duration, the trace ids the agent sent and the side effects it attempted. Each attempt's spans carry eval.run.id, eval.case.id and eval.attempt resource attributes. Give the report to evaluate_local_report through your coding host. The runner itself uploads nothing. Its exit code indicates report generation, not that the agent passed the behavior checks; inspect the evaluation verdict.

Side effects

Isolation is on by default. Every Python process the agent starts (the command and its Python children, the function's process, a server started with --start) loads an interceptor for outbound HTTP through http.client and urllib.request, urllib3 and requests, and httpx (which the OpenAI, Anthropic and Google SDKs use). Before anything is sent:

• Model providers pass: OpenAI, Anthropic, Gemini and Vertex AI, Azure OpenAI, Bedrock, Mistral, Cohere, Groq, DeepSeek, Together, Fireworks, xAI, OpenRouter, Perplexity, the WeHarness model proxy, the hosts in OPENAI_BASE_URL, ANTHROPIC_BASE_URL, GOOGLE_GEMINI_BASE_URL, AZURE_OPENAI_ENDPOINT and WEHARNESS_MODELS_URL, and any --model-host.
• WeHarness trace endpoints and the endpoints in WEHARNESS_OTLP_ENDPOINT, OTEL_EXPORTER_OTLP_ENDPOINT and OTEL_EXPORTER_OTLP_TRACES_ENDPOINT pass; the trace ids in those exports become the attempt's trace_ids.
• localhost and loopback addresses pass. Private network addresses do not.
• Any other host is answered from --mocks, or refused: the agent's HTTP client raises a connection error that names the call and how to allow it. --allow-host api.example.com (or *.example.com) lets real calls through.

A mocks file is a JSON list, or {"mocks": [...]}. The first match wins:

[
  {"method": "POST", "url": "https://api.stripe.com/v1/refunds", "status": 200,
   "body": {"id": "re_123", "status": "succeeded"}},
  {"url": "api.crm.example/customers/*", "headers": {"X-Mock": "1"}, "body": {"name": "Ada"}}
]

In url, * matches any characters; a pattern matches the URL without its query unless it contains ?, and a pattern without a scheme matches both http and https. method defaults to any, status to 200, and a body that is not a string is sent as JSON.

Each attempt's effects lists what it tried, grouped with a count:

{"method": "POST", "host": "api.stripe.com", "path": "/v1/refunds",
 "action": "mocked", "kind": "external", "status": 200, "count": 1}

action is allowed, mocked, refused or replayed; kind is model, trace, local, allowed_host or external. Query strings are never reported; paths are. effects is null when the runner could not observe the agent: a non-Python agent, a server it did not start, or --no-isolation.

What this does not cover: raw sockets, other HTTP clients (aiohttp, pycurl, gRPC), processes that are not Python (Node, shell tools, containers), and database or queue drivers. Python tools the agent runs are covered too, so install dependencies before the run. The interceptor prevents accidental side effects during an evaluation; it is not a security sandbox, and the agent's code can undo it. If it cannot load, the agent's Python process stops rather than run without it.

Record and replay

--record DIR stores each model call's request (without credential headers or query keys) and response, per case and attempt, in DIR/<case_id>/attempt-<n>.jsonl. --replay DIR answers identical model requests (same method, URL and JSON body) from that recording, in recorded order, without the network, and refuses a request the recording does not hold. Trace exports, localhost and allowed hosts behave as in any run; mocks answer as usual. Attempts carry "recorded": true or "replayed": true. Use replay to re-run a failure without new model calls while you change the code around the model. Recordings contain prompts and model outputs; keep them out of version control unless you mean to share them.

Credentials

Your agent's provider credentials remain available, and so does WEHARNESS_INGEST_KEY: it only sends your project's traces, and your agent needs it so each attempt carries its traces (WeHarness checks tool calls and links failures to them). WeHarness credentials named WEHARNESS_TOKEN, WEHARNESS_API_KEY, WEHARNESS_MCP_TOKEN, and variables starting with WEHARNESS_SECRET are removed from the child environment. This is a local process runner, not a security sandbox; run only code you trust. On POSIX, timed-out attempts receive a process-group termination; Windows currently terminates the direct process only.

Model access during a run

By default your agent calls its model with your own provider key, read on your machine and never sent to WeHarness. If you ask your coding host to use WeHarness's models instead, evaluate_local_start takes a model and a credit cap, reserves the cap in credits, and returns model_access.env once: a model token for that run (WEHARNESS_MODELS_TOKEN, WEHARNESS_MODELS_URL, WEHARNESS_MODELS_ALLOWED) and the chosen provider's own variables pointed at WeHarness's model proxy (for example ANTHROPIC_API_KEY and ANTHROPIC_BASE_URL). Set them in the environment you start weharness-run in; it passes them on to your agent unchanged. Save the cases file without model_access, and keep the token out of files you commit.

The token allows only that model, within the credit cap, and stops working when the run closes (every attempt reported, the run failed, or left unreported) or after 12 hours. The credits its calls used are then debited and the rest of the cap is released. Your coding host's model can see the token.