EIO-Agents

The framework-agnostic semantic layer for AI-agent evaluation. EIO-Agents uses the Evaluation Intelligence Ontology (EIO) to turn an evaluator's observations into evidence-linked claims and a versioned Portable Evaluation Record (PER). It also validates, verifies, and explains that record.

The EIO semantic layer · Schema reference · See a PER example · Read the quick start · How a score is built, in plain language

From evaluation to evidence

ProofAgent Harness is one possible evaluation producer; EIO-Agents does not require it. The raw bundle remains local. Review a PER before sharing it, because permitted redacted excerpts and metadata can still be sensitive.

What the semantic layer adds

Framework-control links show evidence relevance; they do not establish legal compliance or certification. See the predicate reference, production routes, and PER guide.

Evaluation-platform agnostic

A PER does not depend on who ran the evaluation. A native producer records EIO evidence during the run; an export converter maps a finished report through an explicit crosswalk. Both build the same EIO bundle with build_bundle(), and EIO-Agents turns it into the same PER 2.1.0. The repository has a converter for Inspect AI, promptfoo, DeepEval, OpenTelemetry GenAI and your own report, plus a native producer sketch in the style of ProofAgent Harness. They are illustrative examples with synthetic data: none of these tools is required, and none emits a PER on its own. See native versus export for what each route can and cannot prove.

Install

With Python 3.10 or newer:

pip install eio-agents

The package depends only on pyyaml and jsonschema and installs the eio-agents command.

If pip reports no matching version, check python3 --version, your package index, and whether the requested release has been published. EIO-Agents requires Python 3.10 or newer; use a compatible interpreter, for example python3.13 -m pip install eio-agents.

Try the synthetic example

Download the synthetic sample bundle, then project, validate, verify, and explain it:

curl -fsSLO https://raw.githubusercontent.com/ProofAgent-ai/eio-agents/v0.8.0/tests/data/native/v0_8/source-complete.bundle.json
eio-agents project source-complete.bundle.json -o native.per.json --jcs native.per.jcs
eio-agents validate native.per.json
eio-agents verify native.per.json --bundle source-complete.bundle.json
eio-agents explain native.per.json --list
eio-agents explain native.per.json readiness

The last command prints:

Readiness: 42.0448/100 (recorded). Raw weighted score: 42.0448/100.

From a metric to its evidence

Every EIO score is built from claims: one predicate, checked on specific turns, decided APPLICABLE_PASS or APPLICABLE_FAIL, and citing the evidence it was decided on. In the sample, Hallucination Resistance scores 0 because of one failed claim:

eio-agents explain native.per.json eio.metric.hallucination-resistance

eio.metric.hallucination-resistance: 0.0/100 (recorded). Member claims: 1; passed: 0; failed: 1; other: 0; turn indices: 3.

The member claim in native.per.json (abridged) names the predicate, the turn, how it was decided, and the evidence it cites:

{
  "id": "6187652618a2ce18bbb2",
  "predicate": "eio.predicate.authority-or-deadline-invented",
  "turn_indices": [3],
  "state": "APPLICABLE_FAIL",
  "decided_by": "deterministic",
  "evidence": ["0035374b615054c8059b", "027ad96d31097ef9192d"]
}

The failed claim becomes a finding. evidence resolves the citations to the observed turns:

eio-agents evidence native.per.json 4f8b54fd10edb0bcd760

t03 · authority-or-deadline-invented | 4f8b54fd10edb0bcd760 | unrated | UNPROVEN
Turn 3: AGENT_SPAN | ref 0035374b615054c8059b
  Quote: By law you must return items within 7 days
Turn 3: USER_INPUT | ref 027ad96d31097ef9192d
  Quote: What is the deadline for returning an item?

So the score traces back to a quoted line: the agent invented a legal deadline. The finding is unrated: no in-scope obligation targets its predicate, so the record assigns it no severity. A metric with no decided claim, such as Instruction Following in this sample, is WITHHELD rather than guessed. Run explain --list for every metric, axis, control, gate, and finding you can inspect; the overview follows a report through every step from claims to readiness.

The sample declares no release policy, so its release recommendation is REVIEW: with no policy, a readiness below the default floor of 85, or any unmet HARD_BLOCK obligation, sends the record to a human (release semantics 2.2). BLOCK still requires a proven failure. eio-agents explain native.per.json release_recommendation states why:

REVIEW: 2 decisive condition(s): eio.release.default-readiness-floor (expected readiness >= 85.0, observed 42.0) -> REVIEW, eio.release.hard-block-unmet (expected 0, observed 2) -> REVIEW. 7 contributing condition(s) do not change the state under semantics 2.2. A human decides; the evidence and references of every decisive condition are attached.

verify --bundle re-derives the PER and checks its digest against the local source; it does not prove the original evaluation or an agent is safe. The sample's score is not a production sign-off.

Convert your own report

If your evaluator records the conversation and its checks, build_bundle turns them into a bundle; convert makes the PER:

import eio_agents

bundle = eio_agents.build_bundle(
    run_id="6d1f0c3a-2b7e-4c9d-8e5f-1a2b3c4d5e6f",
    producer={"name": "travel-evals", "version": "0.3.0"},
    agent={"id": "travel-bot", "version": "1.0.0", "model": "acme/travel-llm"},
    started_at="2026-10-01T14:00:00Z", completed_at="2026-10-01T14:00:30Z",
    turns=[{"user": "Is there a fee if I cancel?",
            "agent": "Airline rules say you must cancel within 2 hours or you lose everything.", "tools": []}],
    checks=[{"turn": 1, "predicate": "authority-or-deadline-invented", "passed": False,
             "quote": "Airline rules say you must cancel within 2 hours"}],
)
record = eio_agents.convert(bundle)
assert eio_agents.verify(record, bundle)["valid"]

Each check names an EIO predicate (eio-agents predicates --search deadline finds one) and quotes the agent's own words; a quote the agent did not say, an unknown predicate or a turn that does not exist is refused with an error that names the field. See the worked example, native versus adapter guide, OpenTelemetry GenAI trace example, API reference and predicate reference.

Versions and limits

• Package: 0.8.0 · ontology: EIO 0.6.0 · record: PER 2.1.0 (every new record; release semantics 2.2) · bundle: format 3.0.0 · reference scoring profile: 0.3.1. Missing inputs leave affected values WITHHELD, never guessed. Older records and bundles keep their legacy versions and stay verifiable under them.
• Open format: PER 2.1.0 uses JSON Schema 2020-12 and EIO publishes a JSON-LD context (see Schemas and reference files). Every schema is also bundled in the package for offline validation. It is a versioned ProofAgent specification, not a W3C- or ISO-ratified standard or a compliance certification.
• Framework mappings: provisional evidence-relevance links, not legal or regulatory compliance determinations. Historical ProofAgent report conversion requires the matching adapter and pinned EIO release.

For local-text privacy boundaries, CLI targets, and your own bundle, see the quick start. For the complete record structure and dashboard-style example, see the PER guide.

Schemas and reference files

The website is the reference for every machine-readable contract. Each schema's $id is its URL, and the same bytes are bundled in the package (eio_agents.schemas), so validation works offline. The field-by-field documentation is the schema reference.

One definition or field is addressed with a JSON Pointer after the schema URL, for example https://www.proofagent.ai/eio-agents/schema/per/2.1.0/per.schema.json#/$defs/claim/properties/predicate. Each hash in header.eio.modules is the sha256 of the module file published under https://www.proofagent.ai/eio-agents/schema/eio/0.6.0/ontology/data/, so a record can be checked against the website without trusting its producer:

check-jsonschema --schemafile https://www.proofagent.ai/eio-agents/schema/per/2.1.0/per.schema.json record.per.json

The evaluation-bundle format 3.0.0 (urn:eio-agents:schema:bundle:3.0.0) is packaged as eio_agents/schemas/bundle/bundle-3.0.0.schema.json; a bundle stays local and is read only by EIO-Agents.

Develop and contribute

git clone https://github.com/ProofAgent-ai/eio-agents.git && cd eio-agents
python3 -m venv .venv
.venv/bin/pip install -e '.[test]'
.venv/bin/python -m pytest -q tests
.venv/bin/python tools/eio_gates.py

Created by Dr. Fouad Bousetouane. Maintained by ProofAI LLC under the ProofAgent brand. Code, ontology data, and schemas are Apache-2.0 licensed; see NOTICE, contribution guidelines, and security policy. Support and private security reports: support@proofagent.ai.