getvda-evaluator

Governance evaluation for AI agents, in Python — an entry point into the existing getvda.ai framework, not a parallel one.

It calls the same services an @getvda/evaluator-sdk user calls: ACP for the signed bundle, Witness for the seal, HITL for escalation. An agent governed through this path is governed by identical machinery. Nothing here re-implements policy.

LangChain DeepAgents middleware

from deepagents import create_deep_agent
from getvda_evaluator.middleware import build

agent = create_deep_agent(
    tools=[...],
    middleware=[build(
        environment="ref-env",
        bundle_source=fetch_signed_bundle,     # ACP
        capability_map={"send_email": "notify_guest"},
        sealer=seal_to_witness,                # Witness
    )],
)

It binds wrap_tool_call, not wrap_model_call. Governance gates actions: a model deciding to do something is not yet doing it, and an evidence trail of intentions is not an evidence trail of conduct.

Two behaviours worth knowing

Fail-static. If governance cannot be fetched or verified, the middleware denies. An agent that keeps acting when its rules are unreachable is ungoverned while appearing governed, which is worse than being stopped.

An unmapped tool is denied, not exempt. Tools with no capability_map entry use their own name, fail to match a capability, and are refused. A tool does not escape governance because nobody wrote a mapping line. Use ungoverned_tools to exempt deliberately, in writing.

Divergence is the risk this package carries

There are now two implementations of the same decision. Both are tested against packages/evaluator-conformance/cases.json — one contract, one set of verdicts, checked in CI on every change.

Change the contract before changing either implementation, never after. If they disagree you have two answers to "was this action allowed", and an evidence product with two answers has none.

That gate earned its place on its first run: this port had invented a SKILL.md format (markdown headings) that the real parser rejects. The suite caught it before it could grant permissions nobody had written down.

Upstream drift

The middleware binds to LangChain's wrap_tool_call / awrap_tool_call, which live in libs/langchain_v1 — a v1 rewrite still in flight. tests/test_api_drift.py pins that surface and a weekly CI job runs it, because a middleware that stops intercepting still looks installed.

The drift test skips when LangChain is absent: the evaluation core is deliberately testable without an agent framework, since it is the part that must not drift.

pytest packages/evaluator-py                       # core + conformance
pip install -e '.[deepagents]' && pytest           # + upstream API check