Skip to main content

ECP Runtime

Reference runtime and CLI for the Evaluation Context Protocol (ECP).

ECP is a vendor-neutral protocol for testing agent outputs, tool calls, and evaluator-visible audit context across frameworks, models, eval platforms, and CI systems.

Install

pip install "ecp-runtime==0.10.0"

Usage

ecp init
ecp validate ecp_eval/manifest.yaml
ecp run --manifest ecp_eval/manifest.yaml --json

Run the flagship example:

ecp run --manifest examples/customer_support_demo/manifest.yaml --report report.html

Useful commands:

ecp validate examples/customer_support_demo/manifest.yaml
ecp conformance --target "python examples/customer_support_demo/agent.py"
ecp doctor

The runtime negotiates protocol compatibility during agent/initialize. It sends protocol 1.0, accepts lower minor versions within major 1, and continues to support versionless agents as legacy protocol 0.1 with one warning per run.

Manifest target values may be either a command for the default stdio transport or an ECP Streamable HTTP endpoint:

target: "http://127.0.0.1:8765/ecp"

Execution Boundaries

A hung agent must never pin a CI job. Two independent limits apply:

ecp run --manifest manifest.yaml --timeout 30 --max-duration 900
  • --timeout (ECP_RPC_TIMEOUT, default 30s) bounds a single RPC.
  • --max-duration (ECP_MAX_DURATION, unset by default) bounds the whole run. This is not redundant: an agent that answers just inside the per-RPC timeout on every step can still run for hours.

When a limit is breached, or the agent crashes or breaks the protocol, the run degrades instead of aborting. The failing step is recorded as failed, the rest of that scenario is marked skipped, and the next scenario starts with a fresh agent. Steps that never produced a result still count as failed checks, so a timeout can never be mistaken for a pass.

Audit Record

Every run produces a structured audit payload, embedded under audit in the JSON report and writable standalone:

ecp run --manifest manifest.yaml --audit-out ecp_audit.json

It records the run id, timestamps, the manifest path and its SHA-256 digest, the resolved target, agent metadata, the configured limits, per-step latency and exit_reason, aggregated token usage, and pass/fail totals. Compare totals.steps_planned against totals.steps_executed to spot a run that degraded rather than one that genuinely passed. The schema is schema/audit.schema.json.

Agents can populate the token figures by returning usage from a step:

Result(public_output="...", usage={"input_tokens": 1204, "output_tokens": 88})

The llm_judge grader needs the optional judge extra, which pulls in litellm:

pip install "ecp-runtime[judge]"

Then set an API key and optional judge model:

$env:OPENAI_API_KEY="your_key_here"
$env:ECP_LLM_JUDGE_MODEL="gpt-4o-mini"

Metadata

Release files for ecp-runtime 0.10.0

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for ecp-runtime 0.10.0
File Size Uploaded Provenance Actions
ecp_runtime-0.10.0.tar.gz 39.7 kB With provenance Details for ecp_runtime-0.10.0.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for ecp-runtime 0.10.0
File Interpreter ABI Platform Actions
ecp_runtime-0.10.0-py3-none-any.whl Python 3 none any Details for ecp_runtime-0.10.0-py3-none-any.whl

Total release size: 70.3 kB

Release files / ecp_runtime-0.10.0.tar.gz

Download URL ecp_runtime-0.10.0.tar.gz
Size 39.7 kB
Tags Source
SHA-256 checksum
How to use checksums
9b351d387a48ddc9575be45621284c09902a59ac6f1f43ca3ff7c2ca02b82d1e
BLAKE2b-256 checksum
How to use checksums
d3c9492b507bfd3db0aadbfbe417544195b97ccc1aa69974df98cde91f31772d
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.14

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Oct 18, 2026.

Transparency log

Release files / ecp_runtime-0.10.0-py3-none-any.whl

Download URL ecp_runtime-0.10.0-py3-none-any.whl
Size 30.7 kB
Tags Python 3
SHA-256 checksum
How to use checksums
38ac5ee32a241e6159c9013ae28ce03051dc0f8bcede20d22aa7a927b2227d82
BLAKE2b-256 checksum
How to use checksums
9daf4a3589d7fefc4d22124555dfe3c44d7d7bba6fd165a3693ea8a46d9f0410
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.14

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Oct 18, 2026.

Transparency log

Security for ecp-runtime 0.10.0

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

Full provenance

Both files in this release have been uploaded with provenance metadata.

Select from 2 files

Attestation for ecp_runtime-0.10.0.tar.gz

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Oct 18, 2026.

Transparency log

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page