At a glance

Architecture

flowchart LR
    accTitle: Evidence-bound agent evaluation architecture
    accDescr: An evaluated agent system interacts with tools and protocols. Controlled adapters normalize observations into evidence. Deterministic evaluators verify preconditions, policy, side effects, outcomes, reliability, and release readiness. Semantic judging is subordinate and cannot rescue deterministic failure.

    O[Evaluation objective]

    subgraph SUBJECT[Evaluated subject · untrusted for grading authority]
      direction TB
      A[Agent runtime + model]
      T[Tools + application state]
      H[Handoffs + approvals + memory]
    end

    subgraph INTEGRATIONS[Observed integration boundaries]
      direction TB
      OA[OpenAI Agents SDK]
      MCP[MCP protocol + auth labs]
    end

    subgraph CONTROL[Trusted evaluation control plane]
      direction LR
      C[Scenario + authority contracts]
      D[Controlled delivery + adapters]
      E[Evidence + typed receipts]
      V[Deterministic verification]
      S[Subordinate semantic judging]
      R[Reliability + release gate]
    end

    O --> C
    C --> A
    A <--> T
    A <--> H
    A <--> OA
    OA <--> MCP
    T --> D
    H --> D
    OA --> D
    MCP --> D
    D --> E
    E --> V
    V -->|deterministic closure| S
    V -->|failure / blocked| R
    S -->|may narrow success| R

    classDef neutral fill:#f6f8fa,stroke:#57606a,color:#24292f,stroke-width:1.5px
    classDef advisory fill:#fbefff,stroke:#8250df,color:#24292f,stroke-width:2px
    classDef authority fill:#ddf4ff,stroke:#0969da,color:#24292f,stroke-width:2px
    classDef evidence fill:#dafbe1,stroke:#1a7f37,color:#24292f,stroke-width:2px
    classDef untrusted fill:#ffebe9,stroke:#cf222e,color:#24292f,stroke-width:2px,stroke-dasharray:5 3
    classDef terminal fill:#dafbe1,stroke:#1a7f37,color:#24292f,stroke-width:3px

    class O neutral
    class A,T,H untrusted
    class OA,MCP advisory
    class C,D,V authority
    class E evidence
    class S advisory
    class R terminal

    style SUBJECT stroke:#cf222e,stroke-width:2px,stroke-dasharray:6 4
    style INTEGRATIONS stroke:#8250df,stroke-width:2px,stroke-dasharray:6 4
    style CONTROL stroke:#0969da,stroke-width:2px,stroke-dasharray:6 4
    linkStyle default stroke:#57606a,stroke-width:1.5px

Diagram key: red dashed = evaluated/untrusted subject · purple = provider/protocol or semantic boundary · blue = deterministic evaluator authority · green = evidence and terminal release truth. Color is never the only signal.

The deeper trust model, identity domains, OpenAI/MCP bridges, authorization boundaries, replay semantics, and release authority live in Architecture.

Engineering thesis

Agents act. Evaluators observe. Evidence binds. Policy constrains. State proves. Replay regrades. Statistics quantify. Release gates decide.

The evaluated component must not be able to manufacture the authority or evidence that certifies its own success.

Core invariants

• Outcome before rhetoric: independently observed state outranks the agent's claim about state.
• Safety is non-compensatory: critical authorization or policy violations cannot be averaged away.
• Unknown is not green: missing or unverifiable evidence becomes BLOCKED, not PASS.
• Identity is canonical: behavior-bearing subject, scenario, attack, approval, retrieval, protocol, evidence, and report material participates in content-addressed identity.
• Delivery is a precondition: adversarial/protocol conditions are graded only after the required controlled relation is verified.
• Delegation never expands: valid handoff paths preserve or reduce effective authority.
• Approval is relational: evidence must bind the exact pending invocation and accepted authority context.
• Replay is historical: replay revalidates persisted evidence without pretending to rerun provider behavior or side effects.
• Semantic judging is subordinate: it may narrow deterministic success but never rescue deterministic failure.
• Release authority is deterministic: reliability and release decisions derive from verified trial evidence and explicit policy.

See Evaluation Lifecycle, Evaluation Model, and Evidence Hierarchy for the detailed grading/evidence flows moved out of this README.

Executable assurance surfaces

Quick start

The deterministic core runs without model credentials.

python -m venv .venv
source .venv/bin/activate
python -m pip install -e '.[dev]'
pytest
agent-evals doctor

Optional integration lanes are declared in pyproject.toml:

python -m pip install -e '.[dev,openai,mcp]'

Project manifests and repository-owned requirement files are authoritative for interpreter and dependency requirements; the README intentionally avoids duplicating executable version truth.

Documentation

Use the documentation hub for role-based review paths. Key entry points:

Repository map

.github/
artifacts/
docs/
src/
tests/

Only top-level ownership boundaries are shown; individual files are documented in the technical pages.

Scope and non-claims

Local deterministic evidence does not by itself prove live-provider correctness, hosted MCP behavior, production IAM, human identity, external target state, distributed exactly-once execution, arbitrary cache coherence/schema migration, universal prompt-injection resistance, or cryptographic publisher attestation.

Those boundaries are intentional. See Security and Limitations.

License

MIT — see LICENSE.