agent-eval-rpc — Python client
Python client for @tangle-network/agent-eval — a content/code judging framework written in TypeScript. This package is a thin transport adapter: every judgement runs in the Node runtime, marshalled over HTTP or stdio RPC. Two languages, one implementation. No drift.
What you get
A function-call interface to score any string against a rubric:
from agent_eval_rpc import Client
client = Client() # auto-detects HTTP server, falls back to subprocess
result = client.judge(
content="We just launched zero-copy IO between agents and their workdir",
rubric_name="anti-slop",
)
print(result.composite) # 0.0..1.0 — single number to gate on
print(result.dimensions) # {"buyer_quality": 0.7, "voice": 0.8, "signal": 0.9}
print(result.failure_modes) # [] or ["ai-cadence", "marketing-tone", ...]
print(result.wins) # ["specific-component", "earned-detail", ...]
print(result.rationale) # "The post names a real architectural detail..."
That's the entire surface for content judging. A self-contained runnable
example with pytest invariants lives at
examples/judge_anti_slop.py.
Hosted-tier ingest
Ship eval-run events + OTel-shape trace spans to any orchestrator that
speaks the wire format frozen at HOSTED_WIRE_VERSION = '2026-05-26.v1'.
Same contract as the TypeScript client at src/hosted/client.ts; same
reference receiver at examples/hosted-ingest-server/.
from agent_eval_rpc import (
HostedClient, EvalRunEvent, EvalRunGenerationSnapshot,
EvalRunCellScore, make_trace_span,
)
with HostedClient(
endpoint="http://localhost:8080",
api_key="dev-token",
tenant_id="acme",
) as client:
res = client.ingest_eval_run(EvalRunEvent(
runId="run-py-1",
runDir="/runs/run-py-1",
timestamp="2026-05-27T00:00:00Z",
status="finished",
labels={"env": "test"},
baseline=EvalRunGenerationSnapshot(
index=0, surfaceHash="h-base",
cells=[EvalRunCellScore(scenarioId="s1", rep=0, compositeMean=0.5,
dimensions={"llm": {"accuracy": 0.5}})],
compositeMean=0.5, costUsd=0.1, durationMs=1000,
),
generations=[],
gateDecision="hold",
totalCostUsd=0.1, totalDurationMs=1000,
))
assert res.accepted == 1
spans = [make_trace_span(
trace_id="t", span_id="s-0", name="dispatch",
start_time_unix_nano=1_700_000_000_000_000_000,
end_time_unix_nano=1_700_000_001_000_000_000,
attributes={"step": 0},
tangle_run_id="run-py-1", tangle_scenario_id="s1", tangle_generation=0,
)]
client.ingest_traces(spans)
Bearer auth, per-tenant idempotency, capped exponential backoff on
5xx/408/429. The tangle.* pivot keys are first-class on make_trace_span
so the dashboard can stitch traces back to their owning run.
End-to-end tests at tests/test_hosted.py spawn the TypeScript reference
receiver and prove binary compat across languages; if either side drifts,
both test suites fail.
Install
cd clients/python
pip install -e .
To use it, one of:
npm install -g @tangle-network/agent-eval— gives you theagent-evalbinary, used by the subprocess transport (works offline, slower per call due to Node startup ~500ms).- Run a server:
agent-eval serve --port 5005— gives you HTTP transport (~10ms per call once up).
The Python client picks whichever is available. Force one with Client(transport="http") or Client(transport="subprocess").
Why the architecture works this way
The TypeScript package is the source of truth for evaluation logic. We don't reimplement rubrics, scoring, or judges in Python — we marshal JSON to the canonical runtime over a versioned wire protocol (defined as Zod schemas, exported as OpenAPI, mirrored in this package as pydantic models).
Adding a new method to the API means: define a Zod schema in src/wire/schemas.ts, write the handler in src/wire/handlers.ts, and the Python client picks it up on the next regeneration. There is no separate Python implementation to maintain.
This is the same pattern as the Anthropic SDK, Stripe SDK, and gRPC: one canonical implementation, language-specific transport clients.
API
Client
Client(
base_url: str | None = None, # AGENT_EVAL_URL or http://127.0.0.1:5005
cli_path: str | None = None, # AGENT_EVAL_CLI or 'agent-eval'
transport: "auto" | "http" | "subprocess" = "auto",
timeout_s: float = 120.0,
)
client.judge(...)
Score a piece of content against a rubric.
def judge(
*,
content: str, # the text being judged
rubric_name: str | None = None, # OR
rubric: Rubric | dict | None = None, # an inline rubric definition
context: dict | None = None, # free-form metadata for the judge
model: str | None = None, # override the judge LLM
) -> JudgeResult
Either rubric_name (use a built-in like "anti-slop") or rubric (an inline definition with your own dimensions/prompt). Not both.
Returns JudgeResult:
composite: float— weighted score in 0..1. The single number to gate on.dimensions: dict[str, float]— per-axis scores (e.g.{"buyer_quality": 0.7}).failure_modes: list[str]— ids of negative patterns detected.wins: list[str]— ids of positive patterns detected.rationale: str— plain-English explanation.rubric_version: str— stable hash of the rubric used. Compare scores only when this matches.model: str— LLM that produced the judgement.duration_ms: int— wall-clock latency.
client.list_rubrics()
Return every rubric the server has registered, with their dimensions and stable rubric_version.
rubrics = client.list_rubrics()
for r in rubrics.rubrics:
print(r.name, r.description, r.rubric_version)
client.version()
Return server + wire-protocol version. Match your pip install version to version; check wire_version for compatibility.
v = client.version()
assert v.version.startswith("0.20")
assert v.wire_version == "1.0.0"
Defining a custom rubric
Built-in anti-slop is tuned for technical-buyer audiences. For different scoring, pass a Rubric inline:
from agent_eval_rpc import Client, Rubric, RubricDimension, FailureMode
rubric = Rubric(
name="my-rubric",
description="Does this commit message explain WHY, not just what?",
systemPrompt="You score commit messages. Score 0..1 on whether the WHY is clear...",
dimensions=[
RubricDimension(id="explains_why", description="Does the message say *why*?", weight=1.0),
],
failureModes=[
FailureMode(id="what-not-why", description="States the change but not the reason"),
],
)
result = client.judge(content="bumped the version", rubric=rubric)
Errors
| Exception | When |
|---|---|
ValidationError |
Server (or pydantic) rejected the request as malformed. Fix your inputs. |
RubricNotFoundError |
Unknown rubric_name. Call list_rubrics() to see what's registered. |
TransportError |
HTTP unreachable or subprocess failed. Retry or check the server. |
AgentEvalError |
Base class — catches everything above. |
All errors carry .code and .details (the structured payload from the server).
Versioning
This package is version-locked to the npm package. agent-eval-rpc==0.21.0 ↔ @tangle-network/agent-eval@0.21.0. CI verifies the npm package, Python package, runtime __version__, and release tag all agree before publish. If one registry publish fails after the other succeeds, retry the failed publish from the same tag or supersede with the next patch release.
wire_version is separate. It bumps only on breaking schema changes. Package versions can differ across releases as long as wire_version is the same.
Development
# install in editable mode
pip install -e ".[dev]"
# unit tests (no Node required)
pytest tests/test_models.py
# integration tests against the bundled CLI
cd ../.. && pnpm build # build the agent-eval CLI in repo root
cd clients/python && pytest # runs subprocess tests against dist/cli.js
Adding a new method
When the TS side adds a new endpoint (say evaluateScenario):
- Update
src/wire/schemas.tswithEvaluateScenarioRequestSchemaandEvaluateScenarioResponseSchema. - Add a handler in
src/wire/handlers.ts, route insrc/wire/server.ts, and case insrc/wire/rpc.ts. - In this client, add the matching pydantic model in
models.pyand method onClient. The pattern is mechanical — copy the shape fromjudge. - Test in both languages. Bump versions together.
A future iteration moves step 3 to datamodel-code-generator -i openapi.json so it's mechanical-and-automatic instead of mechanical-by-hand. Until the surface grows past ~10 endpoints, hand-written models are more readable.
1 maintainer
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Filter files by name, interpreter, ABI, and platform.
If you're not sure about the file name format, learn more about wheel file names.
Copy a direct link to the current filters
File details
Details for the file agent_eval_rpc-0.121.0.tar.gz.
File metadata
- Download URL: agent_eval_rpc-0.121.0.tar.gz
- Upload date:
- Size: 40.3 kB
- Tags: Source
- Uploaded using Trusted Publishing? Yes
- Uploaded via: twine/6.1.0 CPython/3.13.12
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 |
de0c8ad5b57b4e39807778dc49111e62d45d2118ed0691899fb3f61a7511d4d4
|
|
| MD5 |
f5a9c879b4e22bd5ba8ebbda7fd180fc
|
|
| BLAKE2b-256 |
699bc5512a22d20f92555cdc2864d67b45fef2429e97a3ef322031fedde5a901
|
Provenance
The following attestation bundles were made for agent_eval_rpc-0.121.0.tar.gz:
Publisher:
publish.yml on tangle-network/agent-eval
-
Statement:
-
Statement type:
https://in-toto.io/Statement/v1 -
Predicate type:
https://docs.pypi.org/attestations/publish/v1 -
Subject name:
agent_eval_rpc-0.121.0.tar.gz -
Subject digest:
de0c8ad5b57b4e39807778dc49111e62d45d2118ed0691899fb3f61a7511d4d4 - Sigstore transparency entry: 2173970824
- Sigstore integration time:
-
Permalink:
tangle-network/agent-eval@5454763324d7e5d3b8c897b69a79bee93debbdb6 -
Branch / Tag:
refs/tags/v0.121.0 - Owner: https://github.com/tangle-network
-
Access:
public
-
Token Issuer:
https://token.actions.githubusercontent.com -
Runner Environment:
github-hosted -
Publication workflow:
publish.yml@5454763324d7e5d3b8c897b69a79bee93debbdb6 -
Trigger Event:
push
-
Statement type:
File details
Details for the file agent_eval_rpc-0.121.0-py3-none-any.whl.
File metadata
- Download URL: agent_eval_rpc-0.121.0-py3-none-any.whl
- Upload date:
- Size: 15.9 kB
- Tags: Python 3
- Uploaded using Trusted Publishing? Yes
- Uploaded via: twine/6.1.0 CPython/3.13.12
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 |
1b88daa97ff752ad8f8611e886db0c607f11040e262682ececf93f5c48acc245
|
|
| MD5 |
b9d9eece23ff8291cd2a12e32f94049f
|
|
| BLAKE2b-256 |
90bdbde5bab00983eb7eeda000fd29ab73bf1cff707572a71bd5a39c2d67aca8
|
Provenance
The following attestation bundles were made for agent_eval_rpc-0.121.0-py3-none-any.whl:
Publisher:
publish.yml on tangle-network/agent-eval
-
Statement:
-
Statement type:
https://in-toto.io/Statement/v1 -
Predicate type:
https://docs.pypi.org/attestations/publish/v1 -
Subject name:
agent_eval_rpc-0.121.0-py3-none-any.whl -
Subject digest:
1b88daa97ff752ad8f8611e886db0c607f11040e262682ececf93f5c48acc245 - Sigstore transparency entry: 2173970868
- Sigstore integration time:
-
Permalink:
tangle-network/agent-eval@5454763324d7e5d3b8c897b69a79bee93debbdb6 -
Branch / Tag:
refs/tags/v0.121.0 - Owner: https://github.com/tangle-network
-
Access:
public
-
Token Issuer:
https://token.actions.githubusercontent.com -
Runner Environment:
github-hosted -
Publication workflow:
publish.yml@5454763324d7e5d3b8c897b69a79bee93debbdb6 -
Trigger Event:
push
-
Statement type:
