Skip to main content

EIO-Agents — Evaluation Intelligence Ontology for AI Agents (Standard Semantic Layer)

PyPI Python License

Evaluation stack: agent infrastructure, evaluation framework, EIO-Agents semantic layer, and Portable Evaluation Record

The semantic layer for AI-agent evaluation: the Evaluation Intelligence Ontology (EIO) and the Portable Evaluation Record (PER 2.1). Convert, validate, verify, explain.

The EIO semantic layer · Schema reference · See a PER example · Read the quick start · How a score is built, in plain language

From evaluation to evidence

You provide EIO-Agents produces You can check
A native evaluation bundle with observed turns, sources, and decisions A PER with cited claims, findings, measured or withheld scores, provenance, and a content digest Schema and semantic validity, source consistency, and a human-readable explanation

ProofAgent Harness is one possible evaluation producer; EIO-Agents does not require it. The raw bundle remains local. Review a PER before sharing it, because permitted redacted excerpts and metadata can still be sensitive.

What the semantic layer adds

Layer What it makes portable
EIO ontology Named agent-behaviour predicates, evidence requirements, context criteria, metrics, and qualified framework-control links.
EIO-Agents A checked mapping from a producer's observations to EIO claims, findings, Q/E/C/G axes, and readiness—measured only where the source supports them.
PER A versioned record of those claims, citations, scores or withheld reasons, release recommendation, provenance, and digest that another party can validate and verify against the local bundle.

Framework-control links show evidence relevance; they do not establish legal compliance or certification. See the predicate reference, production routes, and PER guide.

Evaluation-platform agnostic

Many producers, one portable record: ProofAgent Harness as a native producer, and Inspect AI, promptfoo, DeepEval, OpenTelemetry GenAI and your own report through export converters, all build the same EIO bundle, which EIO-Agents turns into one PER

A PER does not depend on who ran the evaluation. A native producer records EIO evidence during the run; an export converter maps a finished report through an explicit crosswalk. Both build the same EIO bundle with build_bundle(), and EIO-Agents turns it into the same PER. The repository has a converter for Inspect AI, promptfoo, DeepEval, OpenTelemetry GenAI and your own report, plus a native producer sketch in the style of ProofAgent Harness. They are illustrative examples with synthetic data: none of these tools is required, and none emits a PER on its own. See native versus export for what each route can and cannot prove.

Install

With Python 3.10 or newer:

pip install eio-agents

The package depends only on pyyaml and jsonschema and installs the eio-agents command.

If pip reports no matching version, check python3 --version, your package index, and whether the requested release has been published. EIO-Agents requires Python 3.10 or newer; use a compatible interpreter, for example python3.13 -m pip install eio-agents.

Try the synthetic example

Download the synthetic sample bundle, then project, validate, verify, and explain it:

curl -fsSLO https://raw.githubusercontent.com/ProofAgent-ai/eio-agents/v0.8.0/tests/data/native/v0_8/source-complete.bundle.json
eio-agents project source-complete.bundle.json -o native.per.json --jcs native.per.jcs
eio-agents validate native.per.json
eio-agents verify native.per.json --bundle source-complete.bundle.json
eio-agents explain native.per.json --list
eio-agents explain native.per.json readiness

The last command prints:

Readiness: 42.0448/100 (recorded). Raw weighted score: 42.0448/100.

From a metric to its evidence

Every EIO score is built from claims: one predicate, checked on specific turns, decided APPLICABLE_PASS or APPLICABLE_FAIL, and citing the evidence it was decided on. In the sample, Hallucination Resistance scores 0 because of one failed claim:

eio-agents explain native.per.json eio.metric.hallucination-resistance
eio.metric.hallucination-resistance: 0.0/100 (recorded). Member claims: 1; passed: 0; failed: 1; other: 0; turn indices: 3.

The member claim in native.per.json (abridged) names the predicate, the turn, how it was decided, and the evidence it cites:

{
  "id": "6187652618a2ce18bbb2",
  "predicate": "eio.predicate.authority-or-deadline-invented",
  "turn_indices": [3],
  "state": "APPLICABLE_FAIL",
  "decided_by": "deterministic",
  "evidence": ["0035374b615054c8059b", "027ad96d31097ef9192d"]
}

The failed claim becomes a finding. evidence resolves the citations to the observed turns:

eio-agents evidence native.per.json 4f8b54fd10edb0bcd760
t03 · authority-or-deadline-invented | 4f8b54fd10edb0bcd760 | unrated | UNPROVEN
Turn 3: AGENT_SPAN | ref 0035374b615054c8059b
  Quote: By law you must return items within 7 days
Turn 3: USER_INPUT | ref 027ad96d31097ef9192d
  Quote: What is the deadline for returning an item?

So the score traces back to a quoted line: the agent invented a legal deadline. The finding is unrated: no in-scope obligation targets its predicate, so the record assigns it no severity. A metric with no decided claim, such as Instruction Following in this sample, is WITHHELD rather than guessed. Run explain --list for every metric, axis, control, gate, and finding you can inspect; the overview follows a report through every step from claims to readiness.

The sample declares no release policy, so its release recommendation is REVIEW: with no policy, a readiness below the default floor of 85, or any unmet HARD_BLOCK obligation, sends the record to a human (release semantics 2.2). BLOCK still requires a proven failure. eio-agents explain native.per.json release_recommendation states why:

REVIEW: 2 decisive condition(s): eio.release.default-readiness-floor (expected readiness >= 85.0, observed 42.0) -> REVIEW, eio.release.hard-block-unmet (expected 0, observed 2) -> REVIEW. 7 contributing condition(s) do not change the state under semantics 2.2. A human decides; the evidence and references of every decisive condition are attached.

verify --bundle re-derives the PER and checks its digest against the local source; it does not prove the original evaluation or an agent is safe. The sample's score is not a production sign-off.

Convert your own report

If your evaluator records the conversation and its checks, build_bundle turns them into a bundle; convert makes the PER:

import eio_agents

bundle = eio_agents.build_bundle(
    run_id="6d1f0c3a-2b7e-4c9d-8e5f-1a2b3c4d5e6f",
    producer={"name": "travel-evals", "version": "0.3.0"},
    agent={"id": "travel-bot", "version": "1.0.0", "model": "acme/travel-llm"},
    started_at="2026-10-01T14:00:00Z", completed_at="2026-10-01T14:00:30Z",
    turns=[{"user": "Is there a fee if I cancel?",
            "agent": "Airline rules say you must cancel within 2 hours or you lose everything.", "tools": []}],
    checks=[{"turn": 1, "predicate": "authority-or-deadline-invented", "passed": False,
             "quote": "Airline rules say you must cancel within 2 hours"}],
)
record = eio_agents.convert(bundle)
assert eio_agents.verify(record, bundle)["valid"]

Each check names an EIO predicate (eio-agents predicates --search deadline finds one) and quotes the agent's own words; a quote the agent did not say, an unknown predicate or a turn that does not exist is refused with an error that names the field. See the worked example, native versus adapter guide, OpenTelemetry GenAI trace example, API reference and predicate reference.

Versions and limits

  • Package: 0.8.5 · ontology: EIO 0.6.0 · record: PER 2.1.0 (default; release semantics 2.2), PER 2.1.1 when a record holds a jury-proven finding, PER 2.1.2 when it carries evaluator usage telemetry · bundle: format 3.0.0 · reference scoring profile: 0.3.1. Missing inputs leave affected values WITHHELD, never guessed. Older records and bundles keep their legacy versions and stay verifiable under them.
  • Open format: PER 2.1.x uses JSON Schema 2020-12 and EIO publishes a JSON-LD context (see Schemas and reference files). Every schema is also bundled in the package for offline validation. It is a versioned ProofAgent specification, not a W3C- or ISO-ratified standard or a compliance certification.
  • Jury proof (0.8.4+): a finding decided by an LLM jury is PROVEN only when at least 2 of 3 jurors state the failure, their quotes are located in the transcript and the evidence contract is met; a re-test that does not reproduce it withdraws the proof. A jury-proven CRITICAL triggers the critical-breach cap.
  • Evaluator telemetry (0.8.5+): optional measured LLM usage of the evaluator itself; see Evaluator usage telemetry.
  • Framework mappings: provisional evidence-relevance links, not legal or regulatory compliance determinations. Historical ProofAgent report conversion requires the matching adapter and pinned EIO release.

For local-text privacy boundaries, CLI targets, and your own bundle, see the quick start. For the complete record structure and dashboard-style example, see the PER guide.

Evaluator usage telemetry

From 0.8.5 a producer can record how much its own evaluation cost to run, alongside the agent's telemetry. The agent under test is a black box whose internal token use is usually unknown, so this block measures the evaluator (planner, jury, re-tests, assessors) instead. Fields follow the OpenTelemetry GenAI semantic conventions and carry no cost or price fields:

"telemetry": {
  "evaluator_usage": {
    "conventions": "otel-gen-ai",
    "provenance": "MEASURED",
    "wall_clock_seconds": 388.4,
    "llm_calls": 142,
    "tokens": {"input": 410233, "output": 61877},
    "duration_ms": {"p50": 1840, "p95": 6210, "max": 11902},
    "errors": {"count": 1, "types": ["timeout"]},
    "retries": 3,
    "by_role": [
      {"role": "jury", "model": "gpt-4.1-mini", "llm_calls": 84,
       "tokens": {"input": 250112, "output": 40210},
       "duration_ms": {"p50": 1720, "p95": 5100, "max": 9050}, "errors": 0}
    ]
  }
}
  • provenance is MEASURED, PARTIAL (some calls reported no token usage) or UNAVAILABLE.
  • Latencies are whole milliseconds, nearest-rank percentiles, with p50 ≤ p95 ≤ max enforced by the verifier.
  • Standard roles (planner, conductor, jury, retest_jury, context_assessor, compliance, confirmation, scoring_observer, other) and error types (timeout, rate_limit, provider_error, invalid_output) stay readable; any other label is fingerprinted by the privacy boundary.
  • The block is optional. A record that carries it is issued as PER 2.1.2; every other record keeps its exact bytes.

Schemas and reference files

The website is the reference for every machine-readable contract. Each schema's $id is its URL, and the same bytes are bundled in the package (eio_agents.schemas), so validation works offline. The field-by-field documentation is the schema reference.

File URL Where a record uses it
PER 2.1.2 JSON Schema https://www.proofagent.ai/eio-agents/schema/per/2.1.2/per.schema.json header.schema_uri of records with evaluator usage telemetry
PER 2.1.1 JSON Schema https://www.proofagent.ai/eio-agents/schema/per/2.1.1/per.schema.json header.schema_uri of records with a jury-proven finding
PER 2.1.0 JSON Schema https://www.proofagent.ai/eio-agents/schema/per/2.1.0/per.schema.json header.schema_uri of every other new record
PER 2.0.0 JSON Schema https://www.proofagent.ai/eio-agents/schema/per/2.0.0/per.schema.json header.schema_uri of earlier records
EIO 0.6.0 JSON-LD context https://www.proofagent.ai/eio-agents/schema/eio/0.6.0/eio-context-0.6.0.jsonld maps record fields to EIO terms
EIO 0.6.0 manifest https://www.proofagent.ai/eio-agents/schema/eio/0.6.0/ontology/data/manifest.yaml header.eio.release
EIO 0.6.0 release digests https://www.proofagent.ai/eio-agents/schema/eio/0.6.0/ontology/data/RELEASE-DIGESTS.json header.eio.ontology_sha256 and header.eio.modules
Evaluation claim schema https://www.proofagent.ai/eio-agents/schema/eio/0.6.0/evaluation-claim.schema.json one entry of claims[]
Evidence graph schema https://www.proofagent.ai/eio-agents/schema/eio/0.6.0/evidence-graph.schema.json the local evidence graph that claims cite
Ontology module schema https://www.proofagent.ai/eio-agents/schema/eio/0.6.0/module.schema.json each ontology module file

One definition or field is addressed with a JSON Pointer after the schema URL, for example https://www.proofagent.ai/eio-agents/schema/per/2.1.0/per.schema.json#/$defs/claim/properties/predicate. Each hash in header.eio.modules is the sha256 of the module file published under https://www.proofagent.ai/eio-agents/schema/eio/0.6.0/ontology/data/, so a record can be checked against the website without trusting its producer:

check-jsonschema --schemafile https://www.proofagent.ai/eio-agents/schema/per/2.1.0/per.schema.json record.per.json

The evaluation-bundle format 3.0.0 (urn:eio-agents:schema:bundle:3.0.0) is packaged as eio_agents/schemas/bundle/bundle-3.0.0.schema.json; a bundle stays local and is read only by EIO-Agents.

Develop and contribute

git clone https://github.com/ProofAgent-ai/eio-agents.git && cd eio-agents
python3 -m venv .venv
.venv/bin/pip install -e '.[test]'
.venv/bin/python -m pytest -q tests
.venv/bin/python tools/eio_gates.py

Created by Dr. Fouad Bousetouane. Maintained by ProofAI LLC under the ProofAgent brand. Code, ontology data, and schemas are Apache-2.0 licensed; see NOTICE, contribution guidelines, and security policy. Support and private security reports: support@proofagent.ai.

Metadata

Release files for eio-agents 0.9.0

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for eio-agents 0.9.0
File Size Uploaded Provenance Actions
eio_agents-0.9.0.tar.gz 2.1 MB With provenance Details for eio_agents-0.9.0.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for eio-agents 0.9.0
File Interpreter ABI Platform Actions
eio_agents-0.9.0-py3-none-any.whl Python 3 none any Details for eio_agents-0.9.0-py3-none-any.whl

Total release size: 3.4 MB

Release files / eio_agents-0.9.0.tar.gz

Download URL eio_agents-0.9.0.tar.gz
Size 2.1 MB
Tags Source
SHA-256 checksum
How to use checksums
5afd043a2ab9903725a36e06f0ae98f9492f9b213cc825fec009ccfc5c6420b5
BLAKE2b-256 checksum
How to use checksums
d6230d33439b99bf202b28b862af41e7389d73b79b7c569584b888a1de1288b3
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.14

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 12, 2026.

Transparency log

Release files / eio_agents-0.9.0-py3-none-any.whl

Download URL eio_agents-0.9.0-py3-none-any.whl
Size 1.3 MB
Tags Python 3
SHA-256 checksum
How to use checksums
a050c68dc4130b42bd67f6c11fdbfcbbdc32cea73dac76ea1d0c3b7322a74ac2
BLAKE2b-256 checksum
How to use checksums
649e517c4e24b10d3f5e3952ad78464727d5a5e69e87b5d3a247e7dcb97a2788
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.14

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 12, 2026.

Transparency log

Security for eio-agents 0.9.0

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

Full provenance

Both files in this release have been uploaded with provenance metadata.

Select from 2 files

Attestation for eio_agents-0.9.0.tar.gz

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 12, 2026.

Transparency log

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page