agent-eval-rpc Python Client
agent-eval-rpc lets Python programs call the judging and ingestion APIs implemented by @tangle-network/agent-eval.
The Python package validates requests and responses with Pydantic.
The Node package owns rubric execution, model calls, and scoring.
Install
Python 3.10 or newer and Node.js 20 or newer are required. Install matching package versions:
pip install agent-eval-rpc
npm install --global @tangle-network/agent-eval
Configure an OpenAI-compatible model endpoint for judge calls:
export AGENT_EVAL_LLM_BASE_URL=https://api.openai.com/v1
export AGENT_EVAL_LLM_API_KEY="$YOUR_API_KEY"
export AGENT_EVAL_LLM_MODEL=gpt-4.1-mini
OPENAI_BASE_URL, OPENAI_API_KEY, and OPENAI_MODEL are also accepted.
The endpoint receives the content, rubric, and context passed to client.judge().
Judge Content
from agent_eval_rpc import Client
client = Client()
result = client.judge(
content="The retry budget is checked before each provider call.",
rubric_name="anti-slop",
)
print(result.composite)
print(result.dimensions)
print(result.failure_modes)
print(result.rationale)
Client() first checks for an HTTP server at http://127.0.0.1:5005.
If none is running, it invokes agent-eval rpc as a subprocess.
Inspect client.transport to see which path was selected.
For repeated or concurrent calls, start the server once:
agent-eval serve --port 5005
Then force HTTP from Python when desired:
client = Client(transport="http", base_url="http://127.0.0.1:5005")
Define A Rubric
Use a built-in rubric by name or pass an inline rubric. Exactly one is required.
from agent_eval_rpc import Client, FailureMode, Rubric, RubricDimension
rubric = Rubric(
name="commit-message",
description="Checks whether a commit message explains why the change exists.",
systemPrompt="Score the commit message using the supplied response schema.",
dimensions=[
RubricDimension(
id="explains_why",
description="The message states the reason for the change.",
weight=1.0,
),
],
failureModes=[
FailureMode(
id="what-only",
description="The message states the edit without its reason.",
),
],
)
result = Client().judge(content="fix retry accounting", rubric=rubric)
List the built-in rubrics and their version hashes:
for rubric in Client().list_rubrics().rubrics:
print(rubric.name, rubric.rubric_version)
Client Options
Client(
base_url: str | None = None,
cli_path: str | None = None,
transport: "auto" | "http" | "subprocess" = "auto",
timeout_s: float = 200.0,
)
client.judge() returns:
| Field | Meaning |
|---|---|
composite |
Weighted score from 0 to 1 |
dimensions |
Score for each rubric dimension |
failure_modes |
Detected negative-pattern IDs |
wins |
Detected positive-pattern IDs |
rationale |
Model explanation |
rubric_version |
Stable rubric hash used for comparison |
model |
Model reported by the provider |
duration_ms |
Total call duration |
Hosted Event Ingestion
HostedClient sends evaluation events and trace spans to a server that implements the hosted ingest format.
This is separate from Client, which calls the local judging API.
from agent_eval_rpc import HostedClient
with HostedClient(
endpoint="https://your-ingest.example",
api_key="tenant-token",
tenant_id="acme",
) as client:
response = client.ingest_eval_run(event)
assert response.accepted == 1
Review hosted.py for the typed event fields and retry behavior.
The event payload can include run paths, scenario IDs, candidate values, scores, errors, costs, summaries, and trace attributes.
Errors
| Exception | Meaning |
|---|---|
ValidationError |
The request does not match the Python or server schema |
RubricNotFoundError |
The named built-in rubric does not exist |
TransportError |
The HTTP server or subprocess could not be reached |
AgentEvalError |
Base class for client errors |
Errors include .code and .details when the server returned structured error data.
Versions
The Python and npm packages are released with the same version.
Use client.version() to check the running Node package and wire-format version:
version = Client().version()
print(version.version, version.wire_version)
Development
cd clients/python
pip install -e ".[dev]"
pytest
Run the cross-language tests after building the Node package:
cd ../..
pnpm build
cd clients/python
pytest
The runnable Python example is examples/judge_anti_slop.py.
1 maintainer
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Filter files by name, interpreter, ABI, and platform.
If you're not sure about the file name format, learn more about wheel file names.
Copy a direct link to the current filters
File details
Details for the file agent_eval_rpc-0.123.3.tar.gz.
File metadata
- Download URL: agent_eval_rpc-0.123.3.tar.gz
- Upload date:
- Size: 40.4 kB
- Tags: Source
- Uploaded using Trusted Publishing? Yes
- Uploaded via: twine/6.1.0 CPython/3.13.14
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 |
e78b022d5879d0ec44f4db98bfdcf4daadbf3bad241daec65cab2d0fa029ccc2
|
|
| MD5 |
77a31503b29e3549ef663ccbebddb7da
|
|
| BLAKE2b-256 |
078a23c26d826420a046b73d7e2cdd274da69a0b8135d67c69352d9e6535f602
|
Provenance
The following attestation bundles were made for agent_eval_rpc-0.123.3.tar.gz:
Publisher:
publish.yml on tangle-network/agent-eval
-
Statement:
-
Statement type:
https://in-toto.io/Statement/v1 -
Predicate type:
https://docs.pypi.org/attestations/publish/v1 -
Subject name:
agent_eval_rpc-0.123.3.tar.gz -
Subject digest:
e78b022d5879d0ec44f4db98bfdcf4daadbf3bad241daec65cab2d0fa029ccc2 - Sigstore transparency entry: 2221412589
- Sigstore integration time:
-
Permalink:
tangle-network/agent-eval@5389e684736ef75e435da60cf40f09967dc9d9c3 -
Branch / Tag:
refs/tags/v0.123.3 - Owner: https://github.com/tangle-network
-
Access:
public
-
Token Issuer:
https://token.actions.githubusercontent.com -
Runner Environment:
github-hosted -
Publication workflow:
publish.yml@5389e684736ef75e435da60cf40f09967dc9d9c3 -
Trigger Event:
push
-
Statement type:
File details
Details for the file agent_eval_rpc-0.123.3-py3-none-any.whl.
File metadata
- Download URL: agent_eval_rpc-0.123.3-py3-none-any.whl
- Upload date:
- Size: 14.3 kB
- Tags: Python 3
- Uploaded using Trusted Publishing? Yes
- Uploaded via: twine/6.1.0 CPython/3.13.14
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 |
81a03cd963f9beb319a2dac9442a042156ef01d79353e19698f3dd24150fe327
|
|
| MD5 |
74675a0d22e26320a2ad200175757bef
|
|
| BLAKE2b-256 |
954ecbd63a870e22269c40eb1b76820652f84616be3923da4f393521d3dec8f0
|
Provenance
The following attestation bundles were made for agent_eval_rpc-0.123.3-py3-none-any.whl:
Publisher:
publish.yml on tangle-network/agent-eval
-
Statement:
-
Statement type:
https://in-toto.io/Statement/v1 -
Predicate type:
https://docs.pypi.org/attestations/publish/v1 -
Subject name:
agent_eval_rpc-0.123.3-py3-none-any.whl -
Subject digest:
81a03cd963f9beb319a2dac9442a042156ef01d79353e19698f3dd24150fe327 - Sigstore transparency entry: 2221412924
- Sigstore integration time:
-
Permalink:
tangle-network/agent-eval@5389e684736ef75e435da60cf40f09967dc9d9c3 -
Branch / Tag:
refs/tags/v0.123.3 - Owner: https://github.com/tangle-network
-
Access:
public
-
Token Issuer:
https://token.actions.githubusercontent.com -
Runner Environment:
github-hosted -
Publication workflow:
publish.yml@5389e684736ef75e435da60cf40f09967dc9d9c3 -
Trigger Event:
push
-
Statement type:
