evidpath 0.5.0

Deterministic interaction-testing CLI for auditing recommender, search, and agent systems with seeded scenarios, trace-based judging, and reproducible regression workflows.

Evidpath

Source | Full docs | Releases | Issues

Evidpath helps teams check recommender systems, search rankers, and agent targets before launch. Point it at an endpoint, local Python target, or driver config, run an audit, and open a report that shows what happened.

Public domains:

• recommender: deepest domain, including native HTTP, schema-mapped HTTP, in-process Python, local reference service, compare, generated scenarios, generated populations, and run-swarm.
• search: real public domain with native HTTP, schema-mapped HTTP, in-process Python, reference audits, check-target, compare, generated scenarios, generated populations, and plan-first audit/compare workflows. Search does not support run-swarm yet.
• agents: real public domain for trajectory audits with built-in reference, in-process Python/LangGraph-style targets, OpenAI-compatible Chat Completions, Anthropic Messages, MCP stdio, and HTTP-session targets. Agent audits record observe-only boundary evidence for normalized tool calls when tool calls are present. Harness-owned or fixture-backed paths can also add mediated boundary events, local allow/warn/block policy outcomes, and post-run boundary check artifacts when explicitly configured. Supported local boundary evidence can additionally produce bounded branch manifests for alternate outcomes such as timeouts and permission denials. Agent source specs can also reference local task/environment packs that bundle one task, local fixture declarations, boundary expectations, setup/reset state, and safe exports without claiming hosted isolation. Agents also support the first bounded local run-swarm path over task/environment pack cases, with an agent_swarm_manifest.json, per-case run roots, workspace diffs, command evidence, verifier output, target-brain refs, deterministic advisory helper sidecars, task-by-brain summaries, and local corpus review.

Generated scenario/population workflows are supported for recommender and search. Agent generated coverage is supported as generated local task/environment packs with generated coverage sidecars that run against a user-provided agent. Agent run-swarm is supported for the bounded local task/environment-pack matrix path with executable brain variants from source specs. Metadata-only or unsupported variants are preserved with reason codes. Agent generation-selection source specs can now write an agent_generation_selection_bundle.json before execution, recording generated task/environment/boundary/verifier candidates, selected candidate ids, lifecycle counts, provider/cache status, seeds, hashes, and manual-review promotion mode. Run reports, runs show, static viewers, and corpus review surface this generation quality separately from deterministic target behavior. Generated customer brain candidate specs may be recorded as deferred evidence, but they are not executable target brains.

Generated customer agent brains, generic agent scenario/population generation, judge-swarm release gating, hosted approval, managed sandbox isolation, managed secrets/egress, and search run-swarm are deferred. Default HTTP-session agents can use --target-url; custom agent HTTP/session targets use --driver-config-path. Agent serve-reference paths are deferred.

Install

python -m pip install evidpath

Requirements:

• Python 3.11+

First Run

Generate a starter driver config when your target does not speak the native HTTP contract:

evidpath init --domain recommender --driver-kind http_schema_mapped --base-url http://127.0.0.1:8051 --non-interactive

Check that your recommender endpoint responds in the expected shape:

evidpath check-target --domain recommender --target-url http://127.0.0.1:8051
evidpath check-target --domain recommender --driver-config-path ./evidpath-driver-config.json

Run one audit:

evidpath audit --domain recommender --target-url http://127.0.0.1:8051 --scenario returning-user-home-feed --seed 7

For search rankers, use --domain search:

evidpath audit --domain search --scenario navigational-query --seed 7
evidpath compare --domain search --baseline-url http://127.0.0.1:8051 --candidate-url http://127.0.0.1:8052 --scenario navigational-query --rerun-count 2
evidpath plan-run --workflow audit --domain search --scenario navigational-query --output-dir ./planned-search-audit
evidpath generate-scenarios --domain search --mode fixture --brief "fresh typo and ambiguous query behavior" --output-dir ./search-generated
evidpath generate-population --domain search --mode fixture --brief "freshness-sensitive search users" --population-size 4 --population-candidate-count 8 --output-dir ./search-generated
evidpath audit --domain search --scenario-pack-path ./search-generated/scenario-packs/fresh-typo-and-ambiguous-query-behavior-fixture.json --population-pack-path ./search-generated/population-packs/freshness-sensitive-search-users-fixture.json --output-dir ./search-generated-audit
evidpath compare --domain search --baseline-url http://127.0.0.1:8051 --candidate-url http://127.0.0.1:8052 --scenario-pack-path ./search-generated/scenario-packs/fresh-typo-and-ambiguous-query-behavior-fixture.json --population-pack-path ./search-generated/population-packs/freshness-sensitive-search-users-fixture.json --rerun-count 1 --output-dir ./search-generated-compare

Generated search and recommender packs include local generated_coverage_plan.json and generated_coverage_manifest.json sidecars that record provenance, validator status, selected/rejected coverage, local promotion mode, and hashes. These sidecars explain generated coverage; the deterministic judge and compare policy still decide outcomes.

For agents, generate a reviewable local task/environment pack:

evidpath generate-agent-pack --brief "support agent current-info and refusal coverage" --output-dir ./agent-generated
evidpath audit --domain agents --scenario-pack-path ./agent-generated/agent-packs/support-agent-current-info-and-refusal-coverage-fixture.json --output-dir ./agent-generated-audit

This writes an evidpath.agent_task_environment_pack/v1 pack plus generated coverage sidecars. It can describe supported local tools, fixtures, boundary checks, deterministic fault profiles, and safe exports. It does not generate an agent brain, target implementation, hosted sandbox, managed secret/egress backend, SQL/browser/email/queue/MCP state, or release approval workflow.

Run a bounded local agent swarm over a task/environment pack and driver config:

evidpath run-swarm \
  --domain agents \
  --brief "local coding proof" \
  --agent-pack-path products/evidpath/examples/agent_coding_swarm/coding-agent-pack.json \
  --driver-config-path products/evidpath/examples/agent_coding_swarm/driver_config.json \
  --agent-max-cases 1 \
  --semantic-mode off \
  --output-dir ./agent-coding-swarm

This writes agent_swarm_manifest.json and per-case run roots with local workspace diffs, bounded command evidence, verifier output, and standard reports/results/traces where available. It also records target-brain identity, brain visibility, unsupported target seed status where applicable, provider cache status, and advisory reporter/critic artifacts. Advisory artifacts are visible review aids; they do not turn deterministic failed or needs-attention evidence into a pass. The first agent swarm path is local and fixture-backed; it does not claim hosted isolation, generated customer agent brains, managed egress/secrets, browser/computer-use automation, deploys, purchases, deletes, or externally visible side effects.

To exercise generation-selection before the swarm executes:

evidpath plan-from-spec --spec-path products/evidpath/examples/source_specs/agent_run_swarm_generation_selection_fixture.json --output-dir ./agent-generation-selection
evidpath execute-plan --run-plan-path ./agent-generation-selection/run_plan.json
evidpath runs show ./agent-generation-selection --root ./agent-generation-selection

For agents, use --domain agents with an in-process callable, protocol driver, default HTTP-session URL, or the built-in reference target:

evidpath audit --domain agents --scenario current-info-tool-use --seed 7
evidpath audit --domain agents --target-url http://127.0.0.1:8065 --scenario current-info-tool-use --seed 7

For a custom deployed HTTP-session agent, generate a JSON driver config:

evidpath init --domain agents --driver-kind http_session --base-url https://agent.example.com --non-interactive
evidpath check-target --domain agents --driver-config-path ./evidpath-driver-config.json

Then audit:

evidpath audit --domain agents --driver-config-path ./evidpath-driver-config.json --scenario current-info-tool-use --seed 7

For reviewable handoff, validate and compile a JSON source spec into the same run-plan lifecycle:

evidpath validate-spec --spec-path ./evidpath-spec.json
evidpath plan-from-spec --spec-path ./evidpath-spec.json --output-dir ./planned-from-spec
evidpath execute-plan --run-plan-path ./planned-from-spec/run_plan.json

Saved local evidence can be indexed and inspected without executing another run:

evidpath runs index --root ./evidpath-output
evidpath runs list --root ./evidpath-output
evidpath runs show run-agent-1 --root ./evidpath-output
evidpath runs promote-baseline run-agent-1 --root ./evidpath-output
evidpath runs open run-agent-1 --root ./evidpath-output

This writes local run_index.json, baseline_registry.json, and bundle-local index.html viewer artifacts. It is local-first run management, not hosted dashboards, team approval, scheduling, automatic reruns, managed storage, or managed sandbox isolation.

For a cross-artifact local review pass, build a corpus over saved runs, trace imports, generated coverage sidecars, packs, boundary evidence, and baselines:

evidpath corpus index --root ./evidpath-output --with-health --with-review-summary
evidpath corpus list --index ./evidpath-output/corpus_index.json --domain agents
evidpath corpus promote-candidates --index ./evidpath-output/corpus_index.json
evidpath corpus open --index ./evidpath-output/corpus_index.json

This writes corpus_index.json, corpus_health.json, review_manifest.json, review_summary.json, and review_workbench/index.html as local files. Corpus commands do not execute audits, compares, trace imports, generated coverage, branch tests, saved plans, hosted calls, or pack mutation. They are local artifact review, not hosted/team review, managed sandboxing, scheduled replay, judge-swarm gating, automatic release approval, or generated agent brains.

Source spec v1 is JSON-only and compiles into run_plan.json; it can preserve the supported local subset of boundary policy/check/branch declarations for mediated runtime paths, post-run verification, and bounded branch matrices. For agents, source specs can also reference a local task/environment pack:

PYTHONPATH=products/evidpath evidpath validate-spec --spec-path products/evidpath/examples/source_specs/agent_audit_task_environment_pack.json
PYTHONPATH=products/evidpath evidpath plan-from-spec --spec-path products/evidpath/examples/source_specs/agent_audit_task_environment_pack.json --output-dir /tmp/evidpath-agent-pack-demo
PYTHONPATH=products/evidpath evidpath execute-plan --run-plan-path /tmp/evidpath-agent-pack-demo/run_plan.json

Agent run-swarm source specs can also execute multiple local brain variants in one swarm and preserve advisory-helper intent without hidden provider calls:

PYTHONPATH=products/evidpath evidpath validate-spec --spec-path products/evidpath/examples/source_specs/agent_run_swarm_brain_advisory.json
PYTHONPATH=products/evidpath evidpath plan-from-spec --spec-path products/evidpath/examples/source_specs/agent_run_swarm_brain_advisory.json --output-dir /tmp/evidpath-agent-brain-swarm
PYTHONPATH=products/evidpath evidpath execute-plan --run-plan-path /tmp/evidpath-agent-brain-swarm/run_plan.json

For a two-brain synthetic example that produces a clear compliant versus faulty target delta, use products/evidpath/examples/source_specs/agent_run_swarm_multi_brain_synthetic.json. Provider helper mode is explicit and cache-aware: set agent_brains.helper_swarms.mode to provider and choose replay, live, or refresh. Missing, mismatched, malformed, or unsafe helper provider output is recorded as advisory-only helper metadata and does not change deterministic case status.

Pack v1 is local and declarative: fixture files, local HTTP fixture routes, deterministic fault profiles, boundary configs, and safe export artifacts. It does not provide generated agent brains, arbitrary commands, Docker/VM or hosted workers, managed egress, managed secrets, replay stores, hosted production trace connectors, browser state, SQL fixtures, email sinks, queues, or MCP server state. Native agent run artifacts can be imported into a review-required draft pack:

PYTHONPATH=products/evidpath evidpath import-traces --domain agents --run-dir /tmp/evidpath-agent-pack-demo --output-dir /tmp/evidpath-agent-trace-import

This writes trace_import.json, trace_import_report.md, and draft-agent-pack.json. The draft must be reviewed before baseline or release-gating use. Executed agent audits can write observe-only boundary_events.jsonl records from normalized tool calls. Harness-owned or fixture-backed agent paths may additionally write runtime boundary events when they explicitly route interactions through the boundary gateway. When supported checks are configured, runs can write boundary_checks.json with deterministic verifier results over available boundary evidence. When local branch testing is configured, runs can write boundary_branch_manifest.json with bounded simulated outcomes such as timeout, auth denied, malformed result, rate limit, stale result, adversarial sensor content, partial write, permission denied, ambiguous status, or success override. These records describe local evidence; they do not use hosted workers, provide broad replay, isolate execution, or prove recovery for hidden final-output-only boundaries.

By default, Evidpath writes an evidpath-output/ folder in your current directory. The most useful files are:

• report.md
• results.json
• traces.jsonl
• HTTP-session agent audits also include boundary_summary.json and boundary_events.jsonl observe-mode sandbox evidence sidecars.
• optional boundary_events.jsonl for observe-mode boundary events
• optional boundary_checks.json for local boundary verifier results
• optional boundary_branch_manifest.json for explicitly configured local boundary branch testing

Compare two versions before launch:

evidpath compare --domain recommender --baseline-url http://127.0.0.1:8051 --candidate-url http://127.0.0.1:8052 --rerun-count 2

What You Need

• a recommender or search HTTP endpoint, a supported agent driver config, or a Python callable/class
• for native HTTP targets, the request and response shape described in the domain-specific external target contract

If your HTTP service has a different shape, use a schema-mapped driver config with dot paths, JSONPath items_path, or small Python transforms. If your ranker or agent is local Python code, use evidpath.audit(callable=..., ...).

Agent targets support in-process Python/LangGraph-style objects, OpenAI-compatible Chat Completions, Anthropic Messages, MCP stdio, and session-aware HTTP services. Default HTTP-session services can use --target-url; custom session paths, auth headers, provider drivers, in-process drivers, and MCP stdio use driver configs. HTTP-session agent configs support check-target --driver-config-path; provider, in-process, and MCP agent configs can be generated and audited but currently return unsupported preflight messages. Boundary gateway metadata distinguishes final-output-only, observed, proxied, fixture-backed, mediated, instrumented, and harness-owned evidence labels without treating those labels as hosted sandbox isolation. Only explicitly mediated runtime paths can carry local policy outcomes, and post-run boundary checks are verifier evidence rather than proof that an earlier side effect was blocked.

For repository examples of each agent driver family, see the full product docs linked below.

Optional extras provide adapters for common in-process objects:

• evidpath[huggingface]
• evidpath[mlflow]
• evidpath[sklearn]

The reference target remains available in the repo for demos and local onboarding.

Provider credentials are only needed for AI-backed generation, advisory semantic features, or provider-backed agent drivers. A normal check-target, audit, or compare run against your own endpoint does not need an Evidpath provider API key.

Provider-backed generation supports local provider-cache modes:

• live: call the provider and write safe replay evidence.
• replay: read a matching cache record without a live provider call.
• refresh: call the provider again and record refreshed evidence.

Fixture mode stays deterministic and cache-free. Cache sidecars store prompt digests, seed-vector status, provider/cache status, and safe response payloads only after redaction checks. They are local evidence artifacts, not hosted replay, managed sandboxing, or managed secrets.

Semantic advisory artifacts are explanation sidecars. Deterministic checks, package validation, and smoke commands remain the release gate.

Links

• Full product docs: https://github.com/NDETERMINA/limitation/tree/main/products/evidpath
• Demo guide: https://github.com/NDETERMINA/limitation/blob/main/products/evidpath/DEMO.md
• Releases: https://github.com/NDETERMINA/limitation/releases
• Issues: https://github.com/NDETERMINA/limitation/issues