evidpath 0.4.0

Deterministic interaction-testing CLI for auditing recommender, search, and agent systems with seeded scenarios, trace-based judging, and reproducible regression workflows.

Evidpath

Source | Full docs | Releases | Issues

Evidpath helps teams check recommender systems, search rankers, and agent targets before launch. Point it at an endpoint, local Python target, or driver config, run an audit, and open a report that shows what happened.

Public domains:

• recommender: deepest domain, including native HTTP, schema-mapped HTTP, in-process Python, local reference service, compare, generated scenarios, generated populations, and run-swarm.
• search: real public domain with native HTTP, schema-mapped HTTP, in-process Python, reference audits, check-target, compare, and plan-first audit/compare workflows.
• agents: real public domain for trajectory audits with built-in reference, in-process Python/LangGraph-style targets, OpenAI-compatible Chat Completions, Anthropic Messages, MCP stdio, and HTTP-session targets. Agent audits record observe-only boundary evidence for normalized tool calls when tool calls are present. Harness-owned or fixture-backed paths can also add mediated boundary events, local allow/warn/block policy outcomes, and post-run boundary check artifacts when explicitly configured.

Generated scenario/population workflows and run-swarm are currently recommender-only. Default HTTP-session agents can use --target-url; custom agent HTTP/session targets use --driver-config-path. Agent serve-reference paths are deferred.

Install

python -m pip install evidpath

Requirements:

• Python 3.11+

First Run

Generate a starter driver config when your target does not speak the native HTTP contract:

evidpath init --domain recommender --driver-kind http_schema_mapped --base-url http://127.0.0.1:8051 --non-interactive

Check that your recommender endpoint responds in the expected shape:

evidpath check-target --domain recommender --target-url http://127.0.0.1:8051
evidpath check-target --domain recommender --driver-config-path ./evidpath-driver-config.json

Run one audit:

evidpath audit --domain recommender --target-url http://127.0.0.1:8051 --scenario returning-user-home-feed --seed 7

For search rankers, use --domain search:

evidpath audit --domain search --scenario navigational-query --seed 7
evidpath compare --domain search --baseline-url http://127.0.0.1:8051 --candidate-url http://127.0.0.1:8052 --scenario navigational-query --rerun-count 2
evidpath plan-run --workflow audit --domain search --scenario navigational-query --output-dir ./planned-search-audit

For agents, use --domain agents with an in-process callable, protocol driver, default HTTP-session URL, or the built-in reference target:

evidpath audit --domain agents --scenario current-info-tool-use --seed 7
evidpath audit --domain agents --target-url http://127.0.0.1:8065 --scenario current-info-tool-use --seed 7

For a custom deployed HTTP-session agent, generate a JSON driver config:

evidpath init --domain agents --driver-kind http_session --base-url https://agent.example.com --non-interactive
evidpath check-target --domain agents --driver-config-path ./evidpath-driver-config.json

Then audit:

evidpath audit --domain agents --driver-config-path ./evidpath-driver-config.json --scenario current-info-tool-use --seed 7

For reviewable handoff, validate and compile a JSON source spec into the same run-plan lifecycle:

evidpath validate-spec --spec-path ./evidpath-spec.json
evidpath plan-from-spec --spec-path ./evidpath-spec.json --output-dir ./planned-from-spec
evidpath execute-plan --run-plan-path ./planned-from-spec/run_plan.json

Source spec v1 is JSON-only and compiles into run_plan.json; it can preserve the supported local subset of boundary policy/check declarations for mediated runtime paths and post-run verification. Executed agent audits can write observe-only boundary_events.jsonl records from normalized tool calls. Harness-owned or fixture-backed agent paths may additionally write runtime boundary events when they explicitly route interactions through the boundary gateway. When supported checks are configured, runs can write boundary_checks.json with deterministic verifier results over available boundary evidence. These records describe local evidence; they do not create branch fanout, use hosted workers, provide broad replay, or isolate execution.

By default, Evidpath writes an evidpath-output/ folder in your current directory. The most useful files are:

• report.md
• results.json
• traces.jsonl
• HTTP-session agent audits also include boundary_summary.json and boundary_events.jsonl observe-mode sandbox evidence sidecars.
• optional boundary_events.jsonl for observe-mode boundary events
• optional boundary_checks.json for local boundary verifier results

Compare two versions before launch:

evidpath compare --domain recommender --baseline-url http://127.0.0.1:8051 --candidate-url http://127.0.0.1:8052 --rerun-count 2

What You Need

• a recommender or search HTTP endpoint, a supported agent driver config, or a Python callable/class
• for native HTTP targets, the request and response shape described in the domain-specific external target contract

If your HTTP service has a different shape, use a schema-mapped driver config with dot paths, JSONPath items_path, or small Python transforms. If your ranker or agent is local Python code, use evidpath.audit(callable=..., ...).

Agent targets support in-process Python/LangGraph-style objects, OpenAI-compatible Chat Completions, Anthropic Messages, MCP stdio, and session-aware HTTP services. Default HTTP-session services can use --target-url; custom session paths, auth headers, provider drivers, in-process drivers, and MCP stdio use driver configs. HTTP-session agent configs support check-target --driver-config-path; provider, in-process, and MCP agent configs can be generated and audited but currently return unsupported preflight messages. Boundary gateway metadata distinguishes final-output-only, observed, proxied, fixture-backed, mediated, instrumented, and harness-owned evidence labels without treating those labels as hosted sandbox isolation. Only explicitly mediated runtime paths can carry local policy outcomes, and post-run boundary checks are verifier evidence rather than proof that an earlier side effect was blocked.

For repository examples of each agent driver family, see the full product docs linked below.

Optional extras provide adapters for common in-process objects:

• evidpath[huggingface]
• evidpath[mlflow]
• evidpath[sklearn]

The reference target remains available in the repo for demos and local onboarding.

Provider credentials are only needed for AI-backed generation, advisory semantic features, or provider-backed agent drivers. A normal check-target, audit, or compare run against your own endpoint does not need an Evidpath provider API key.

Semantic advisory artifacts are explanation sidecars. Deterministic checks, package validation, and smoke commands remain the release gate.

Links

• Full product docs: https://github.com/NDETERMINA/limitation/tree/main/products/evidpath
• Demo guide: https://github.com/NDETERMINA/limitation/blob/main/products/evidpath/DEMO.md
• Releases: https://github.com/NDETERMINA/limitation/releases
• Issues: https://github.com/NDETERMINA/limitation/issues