agentic-evals
A standalone, framework-agnostic evaluation and scoring engine for LLM and agent outputs.
Extracted from AgenticLens's
proven evaluation module — the same engine, usable on its own. It scores
whatever trace-shaped data you give it (see EvalTrace/EvalSpan below);
it has no dependency on any specific tracing/observability tool, and no
dependency on AgenticLens itself.
Status
Published on PyPI. The engine (deterministic checks, LLM-as-judge and
custom evaluators, a release gate, live Python/HTTP targets, the Eval()
quickstart API, and the agentic-evals CLI) is real, tested code — see the
changelog for what shipped in each release. The API may
still move between minor versions ahead of a 1.0; pin a version in
production and read the changelog before upgrading.
Judge reliability and live resilience (0.9.0)
LLM-judged and live evaluations now say what they cost and whether they actually happened:
from agentic_evals import FACTUALITY, JudgeResponse, LLMRubricEvaluator, RetryPolicy
judge = LLMRubricEvaluator(
"factuality",
FACTUALITY,
complete_fn, # may return JudgeResponse(text=..., cost_usd=...)
retry=RetryPolicy(max_attempts=3, backoff_seconds=0.5),
samples=3, # judge three times, report agreement
)
- Judge cost, tokens and latency are recorded on each judge
Scoreunder thejudge_*keys AgenticLens already reads, and totalled insummary.judge. - A judge that fails after its retries yields an explicit unavailable score that never passes; the default gate fails and names the cases.
samples=Nreports per-case agreement;GateConfig(min_judge_agreement=...)catches a flaky judge.- Live targets retry with backoff and report
timed_outseparately from failed (CaseEvaluation.target_status,summary.live, JUnit<error>). aevaluate_suite/arun_live_suiteawaitasync defevaluators and targets.
See the judge reliability guide and the offline judge and async target examples. Report schema 1.0 gains optional fields only.
Declarative CLI (0.8.0)
Run saved suites and samples, then apply an explicit release gate:
agentic-evals evaluate examples/ci/suite.yaml examples/ci/samples.jsonl --out report.json
agentic-evals gate report.json --config examples/ci/gate.yaml
agentic-evals pack list
agentic-evals live examples/ci/suite.yaml --target my_agent:run --max-workers 4 --out live.json
evaluate and live exit 0 when a report is produced, including failing cases.
gate exits 0 for pass, 1 for a failed gate, and 2 for input or usage errors.
Use --format json|junit|markdown for native evidence, JUnit or job summaries.
See the CLI guide, CI workflow, and
voice-agent CLI example.
The native report and gate contracts remain compatible with 0.7.2 consumers.
Why a separate package
AgenticLens's evaluation module was never actually AgenticLens-specific — scoring an LLM/agent output against expectations doesn't need AgenticLens's full trace schema, CLI, or dashboards. Pulling it out means:
agentic-sidecar,agentic-chaos, or any other project can score outputs without depending on all of AgenticLens.- Anyone with any trace-shaped data — not just AgenticLens users — can use it; scoring a string doesn't need to care what produced it.
Install
pip install agentic-evals
Quickstart in under a minute
The fast path in -- no TestSuite, no TestCase, no trace object. Write a
file, run it:
# capitals_eval.py
from agentic_evals import Eval, equals
def my_agent(country: str) -> str:
return {"France": "Paris", "Japan": "Tokyo"}[country]
Eval(
"capitals",
data=[
{"input": "France", "expected": "Paris"},
{"input": "Japan", "expected": "Tokyo"},
],
task=my_agent,
scores=[equals],
)
$ python capitals_eval.py
agentic-evals :: capitals
--------------------------
[PASS] France equals=1.00
[PASS] Japan equals=1.00
equals avg 1.000
2/2 passed (100.0%) in 0ms
Or let the CLI find every eval file in a directory and roll them into one
CI-friendly exit code -- 0 if every case in every file passed, 1
otherwise, no config file required:
pip install agentic-evals
agentic-evals run # discovers *_eval.py / eval_*.py / *.eval.py
A scorer is any function that returns a float/bool (0-1), a Score, or
{"score": ..., "name": ...} -- called with whichever of input, output,
expected it declares as parameters:
def contains_the_total(output: str, expected: str) -> bool:
return expected in output
Six ready-made scorers ship for this API: equals, contains,
icontains, levenshtein, matches (regex), numeric_close (tolerance-
based). Eval() also accepts threshold= (default
1.0) -- a case passes when every one of its scores clears it -- and
data= may be a zero-argument callable for data you'd rather build lazily.
This is deliberately the simple surface. Everything below -- trace-aware
scorers, JSON Schema/tool-call expectations, eval packs, cost/latency
release gates -- is the same engine underneath, for when a plain
input/output/expected row isn't enough.
The declarative API
For trace-aware expectations (tool calls, latency/cost thresholds, JSON
Schema) and CI release gates, use TestSuite/TestCase/evaluate_suite
directly -- the engine Eval() above is a thin, opinionated front end for.
Core concepts
EvalTrace/EvalSpan— the minimal trace shape the engine inspects: a trace id, spans (each optionally naming atool_nameand carrying arbitraryattributes), total latency, estimated cost, and metadata. Deliberately not tied to any specific instrumentation format — build one from whatever you already have.Score— a single named judgment (0-1 value, pass/fail, explanation), with a stablemetrickey that reports group by.Score.skip(...)records a check that did not apply to a case.Evaluator— anything with a.nameand an.evaluate(context) -> list[Score].CallableEvaluatoradapts a plain Python function;LLMJudgeEvaluatorandBusinessRuleEvaluatorare named convenience subclasses for readability/reporting.TestCase/TestSuite— declarative expectations (exact match, substring, JSON Schema, required fields, required/forbidden tool calls, tool arguments and their values, tool call order, latency/cost/turn-count thresholds, or a named custom evaluator) plus the cases that make up a suite.evaluate_suite— runs a suite against suppliedEvaluationSamples and returns anEvaluationReport(per-case scores plus a pass-rate/cost/ latency summary, broken down by metric and by tag).GateConfig/evaluate_gate— turn anEvaluationReportinto a pass/fail release decision on configurable thresholds, for the whole suite or for a single tag or metric.
TestSuite quickstart
from agentic_evals import (
EvalSpan,
EvalTrace,
EvaluationSample,
TestCase,
TestSuite,
evaluate_suite,
)
suite = TestSuite(
name="support-answers",
version="1",
cases=[
TestCase(
id="case-1",
name="Answer contains the right total",
expected_contains=["42"],
required_tools=["calculator"],
max_latency_ms=2000,
)
],
)
sample = EvaluationSample(
case_id="case-1",
output="The combined total is 42.",
trace=EvalTrace(
trace_id="trace-1",
total_latency_ms=350,
spans=[EvalSpan(tool_name="calculator")],
),
)
report = evaluate_suite(suite, [sample])
print(report.summary.pass_rate) # 1.0
Breakdowns by metric and tag
An overall pass rate hides where the failures are. Tag your cases, and the report's summary breaks results down two ways:
suite = TestSuite(
name="support-answers",
version="1",
cases=[
TestCase(
id="r1",
name="Refund status",
tags=["refunds"],
expected_contains=["refund"],
required_tools=["lookup_order"],
),
TestCase(id="t1", name="Order tracking", tags=["tracking"], expected_contains=["shipped"]),
],
)
report = evaluate_suite(suite, samples)
for tag, stats in report.summary.tags.items():
print(f"{tag}: {stats.passed_cases}/{stats.total_cases} cases passed")
for metric, stats in report.summary.metrics.items():
print(f"{metric}: {stats.passed}/{stats.total} checks passed")
summary.tags— oneTagSummaryper tag: case counts, pass rate and average score for the cases carrying that tag.summary.metrics— oneMetricSummaryper metric: how many checks passed, failed or were skipped, plus pass rate and average score. Checks such ascontains:refundandcontains:shippedroll up under the single metriccontains; the detailed name stays onScore.name.- Each
CaseEvaluationcarries its case'stagsandmetadata, so a report can be filtered or regrouped without the original suite.
A custom evaluator can mark a check as not applicable instead of passing or
failing it. A skipped score never fails the case and stays out of averages
and pass rates; it is counted separately in MetricSummary.skipped:
def cites_order_id(context: EvaluationContext) -> Score:
order_id = context.case.metadata.get("order_id")
if order_id is None:
return Score.skip("cites_order_id", "Case has no order id to cite.")
cited = order_id in context.sample.output
return Score(
name="cites_order_id",
value=float(cited),
passed=cited,
explanation=f"Order id {order_id} cited: {cited}.",
)
Tool call expectations
Beyond which tools were called, a case can state what they were called
with and in what order. Tool arguments are read from each span's
attributes["tool_args"]:
TestCase(
id="publish-draft",
name="Saves the draft before publishing it",
required_tools=["save_draft", "publish"],
forbidden_tools=["delete_document"],
required_tool_arguments={"publish": ["document_id"]}, # keys present
expected_tool_arguments={"publish": {"visibility": "internal"}}, # exact values
required_tool_order=["save_draft", "publish"], # first calls in order
)
expected_tool_argumentspasses when at least one call to the tool carries every listed argument with an equal value. Values are compared as given, without type coercion (5is not"5").required_tool_orderpasses when each listed tool is first called before the next one in the list. Calls to other tools in between are fine; a listed tool that is never called fails the check.
LLM-as-judge
from agentic_evals import (
EvaluationContext,
EvaluatorConfig,
EvaluatorRegistry,
LLMJudgeEvaluator,
Score,
TestCase,
)
def judge(context: EvaluationContext) -> Score:
# Call whatever model/provider you like here.
correct = "42" in context.sample.output
return Score(
name="answer_quality",
value=0.95 if correct else 0.1,
passed=correct,
explanation="Judged against the rubric in context.config.config.",
)
registry = EvaluatorRegistry()
registry.register(LLMJudgeEvaluator("answer_quality_judge", judge, version="quality-prompt-v2"))
case = TestCase(
id="case-1",
name="Answer quality",
evaluators=[EvaluatorConfig(name="answer_quality_judge", threshold=0.8)],
)
Which evaluator produced each score
Every Score in a report names what produced it. evaluate_suite stamps
Score.evaluator and Score.evaluator_version:
| Score source | evaluator |
evaluator_version |
|---|---|---|
| Built-in checks (exact match, contains, tools, thresholds) | "agentic_evals.builtin" (BUILTIN_EVALUATOR) |
this package's __version__ |
| A registered evaluator | its registry name | its version attribute (version= on CallableEvaluator, LLMJudgeEvaluator, BusinessRuleEvaluator, LLMRubricEvaluator) |
A None version means the evaluator did not declare one. Version a judge
whenever its prompt or model changes, so two reports can be compared only
when they were graded by the same judge.
Judge cost, retries and agreement
LLMJudgeEvaluator and LLMRubricEvaluator take retry=RetryPolicy(...) and
samples=N. Every judge score records judge_status (ok or
unavailable), judge_attempts and judge_latency_ms, plus any usage the
judge reported (judge_cost_usd, judge_cost_source, tokens, model,
provider). A rubric complete_fn reports usage by returning a
JudgeResponse. A judge that raises, or a rubric completion that cannot be parsed,
no longer aborts the suite. After the last attempt it becomes an unavailable score:
value 0 and never passing, with the error recorded.
summary.judge totals judge cost apart from the agent's cost, and lists
unavailable cases and per-case agreement. See the
judge reliability guide.
Release gates
from agentic_evals import GateConfig, evaluate_gate
decision = evaluate_gate(
report,
GateConfig(min_pass_rate=0.95, max_average_latency_ms=1500, max_total_cost_usd=0.25),
)
if not decision.passed:
raise SystemExit(f"Release gate failed: {decision.reasons}")
A gate can hold one slice of the report to a stricter bar than the suite
as a whole, keyed by case tag or by Score.metric:
GateConfig(
min_pass_rate=0.95,
min_tag_pass_rate={"safety": 1.0}, # every safety-tagged case must pass
min_metric_pass_rate={"forbidden_tool": 1.0}, # no forbidden tool call, anywhere
)
A tag or metric named in the config but missing from the report fails the gate, so a renamed tag cannot quietly switch a check off.
Judge and live-target checks (0.9.0):
GateConfig(
max_judge_unavailable_scores=0, # the default: an unavailable judge fails the gate
min_judge_agreement=0.8, # every case judged with samples=N
max_judge_cost_usd=2.50, # fails when any judge score has no cost
max_timed_out_cases=0, # live runs: timeouts, apart from wrong answers
max_target_error_cases=0,
)
Never fabricates a value it can't back up: total_cost_usd on a summary or
gate decision stays None unless every case in scope has a known cost —
an incomplete cost picture is reported as unavailable, not $0.00.
Cost provenance travels with the numbers. Set EvalTrace.cost_source to
"measured" (billed or provider-reported) or "estimated" (computed from
token counts and a price table); each CaseEvaluation.cost_source carries
it, and the summary reports priced_cases and cost_sources (priced cases
per provenance, with "unspecified" for traces that did not say). A case
with no sample at all has sample_available=False, so its empty trace_id
reads as "no trace exists" rather than "trace ID lost".
Built-in scorers
agentic_evals.scorers ships ready-to-use scorers so you don't have to
hand-write a CallableEvaluator for common checks:
scorers.text(deterministic, no LLM):exact_match,contains_all,contains_any,levenshtein_similarity,embedding_similarity,valid_json,json_diff,numeric_diff,regex_match,starts_with,ends_with,numeric_range.scorers.rubric(LLM-graded, provider-neutral):RubricTemplate+LLMRubricEvaluator, with built-in templatesFACTUALITY,CLOSED_QA,SUMMARY_QUALITY,BATTLE(pairwise A/B),MODERATION,TRANSLATION,SECURITY,SQL_CORRECTNESS,POSSIBLE,PII_LEAKAGE, plusmake_rubric()to build one from your own criteria. LikeLLMJudgeEvaluator, this package never calls a model itself -- you pass acomplete_fn: Callable[[str], str].scorers.trajectory(reads the trace, not just the output text -- the part a plain text-scoring library has no equivalent for):tool_call_precision,tool_call_recall,no_redundant_tool_calls,trajectory_efficiency.
from agentic_evals import (
EvalTrace,
EvaluationSample,
EvaluatorConfig,
TestCase,
TestSuite,
default_registry,
evaluate_suite,
)
suite = TestSuite(
name="support-answers",
version="1",
cases=[
TestCase(
id="case-1",
name="Answer is close to the reference",
expected_output="The combined total is 42.",
evaluators=[EvaluatorConfig(name="levenshtein_similarity", threshold=0.9)],
)
],
)
sample = EvaluationSample(case_id="case-1", output="The combined total is 42.", trace=EvalTrace())
report = evaluate_suite(suite, [sample], registry=default_registry())
default_registry() covers every text/trajectory scorer under a
stable name. Rubric scorers need a complete_fn, so register an
LLMRubricEvaluator instance yourself:
from agentic_evals import FACTUALITY, LLMRubricEvaluator, default_registry
registry = default_registry()
registry.register(LLMRubricEvaluator("factuality", FACTUALITY, complete_fn=call_your_model))
When no built-in template fits, describe the criterion in plain language
and make_rubric() builds the template — the prompt, the verdict letters
and their scores:
from agentic_evals import LLMRubricEvaluator, make_rubric
concise = make_rubric("concise", "The answer is at most two sentences and has no preamble.")
tone = make_rubric(
"tone",
"The reply is courteous and does not blame the reader.",
levels=[ # best first; scores in [0, 1]
("Courteous throughout.", 1.0),
("Neutral: neither courteous nor rude.", 0.5),
("Rude, dismissive, or blames the reader.", 0.0),
],
)
registry.register(LLMRubricEvaluator("concise", concise, complete_fn=call_your_model))
registry.register(LLMRubricEvaluator("tone", tone, complete_fn=call_your_model))
The default scale is pass/fail. Pass with_reference=True to show the
judge a reference next to the output; it is read from
EvaluatorConfig.config["reference"], as with the built-in templates.
Live targets
Point a suite at a real running system (a trusted Python callable, or an HTTP endpoint) instead of pre-recorded samples:
from agentic_evals import PythonTarget, run_live_suite
report = run_live_suite(suite, PythonTarget(callable_path="my_module:run_case"))
run_live_suite(..., max_workers=4) calls the target for several cases at
once on a thread pool. The report keeps suite case order, so it matches a
sequential run that produced the same outputs. Only raise max_workers for
targets that are safe to call concurrently. Python and HTTP targets go
through the same scoring path; the test suite checks that both produce the
same report for the same outputs.
retry=RetryPolicy(max_attempts=3, backoff_seconds=1) retries a call that
raises or times out. A case whose calls all fail is reported on the case
(target_status timed_out or error, target_attempts, target_error)
instead of aborting the run, and summary.live counts each outcome. HTTP
targets always apply timeout_seconds; Python targets apply it when it is set
explicitly, cancelling async def targets. arun_live_suite(..., max_concurrency=N) awaits async targets and evaluators; aevaluate_suite
does the same for saved samples.
Live targets are intentionally powerful developer-facing integrations — Python targets execute local code and HTTP targets can reach arbitrary URLs. Only point them at trusted suite files and trusted target definitions.
Eval packs
A pack bundles a TestSuite with the scorer names it needs into one
shareable YAML/JSON file:
from agentic_evals import (
EvalSpan,
EvalTrace,
EvaluationSample,
default_registry,
evaluate_suite,
load_builtin_pack,
)
pack = load_builtin_pack("tool-use-correctness") # validates required_scorers up front
sample = EvaluationSample(
case_id="refund-status-lookup",
output="Your refund is on its way.",
trace=EvalTrace(spans=[EvalSpan(tool_name="lookup_refund")]),
)
report = evaluate_suite(pack.to_suite(), [sample], registry=default_registry())
Ships 3 built-in packs (list_builtin_packs()): tool-use-correctness,
json-output-contract (both runnable with default_registry()), and
factual-qa (needs a registry with an LLMRubricEvaluator registered,
since it uses the FACTUALITY rubric). Load your own with load_pack(path).
Skills
agentic_evals/skills/ ships methodology playbooks (SKILL.md cards --
Trigger/Do/Avoid/Check/Risk), not runnable code, scoped to this package's
own API. 17 skills cover the eval lifecycle end to end:
- Frame:
define-an-eval-objective,elicit-eval-criteria - Build data:
build-an-eval-dataset,size-a-test-suite - Score:
write-a-scorer,choose-a-rubric-template,validate-a-scorer - Run:
instrument-a-trace,run-a-live-suite - Experiment:
design-an-eval-experiment,analyze-an-eval-experiment - Investigate:
discover-failure-modes,red-team-an-agent-suite,debug-a-flaky-llm-judge - Operate:
define-a-release-gate,report-eval-results,monitor-evals-in-production
These document how to use this package well and are meant to be read directly, or picked up by a coding agent's own skill mechanism -- distinct from "packs" above, which are runnable configuration.
Using it with AgenticLens's own traces
If you already have an AgenticLens Run (from its instrumentation API or
OTLP ingestion), AgenticLens itself provides the adapter —
agenticlens.evaluation.to_eval_trace(run) — so you don't have to hand-build
an EvalTrace. This package has no dependency in the other direction.
What's deliberately not here
Dataset versioning/splitting, judge calibration, and HTML report rendering
stay in AgenticLens for now — those are product features built on top of
this engine, not the engine. So do trace-aware evaluators (tool trajectory,
handoffs, retrieval groundedness), failure diagnosis, and the analysis of
judge cost and evaluator disagreement that AgenticLens 0.7.0 runs over this
package's EvaluationReport.
License
MIT
Control Tower integration boundary
AgenticOps Control Tower owns deployment registration, heartbeat, fleet inventory and operator workflows. AgenticLens owns observability and analysis; Agentic Evals owns scoring and release-gate computation; Agentic Sidecar owns decision supervision (SUPERVISE); Agentic Chaos owns fault injection and resilience experiments. AI Operations Specification owns shared semantics and is currently draft.
Tower's native artifact readers consume existing producer outputs using an explicit deployment/evidence link. They do not change producer behavior or convert a failed run, failed gate, blocked action or injected fault into fleet health. This link is a Tower-local contract, not a normative AIOS schema. See Control Tower's evidence contract. Tower 0.5 introduced pushed evidence posture; 0.6.0 adds cited investigation and approved deterministic staging recovery. Pull collection and production runtime adapters remain planned; native interoperability does not establish stable AIOS conformance.
Operational safety roadmap
See planned safety acceptance criteria for responsibilities and dependencies. These additions do not claim implemented end-to-end containment or integrity protection.
Sidecar 0.8 interoperability
Evals 0.7.1 retains standalone scoring and release-gate ownership. Sidecar supervision outcomes and private-context lifecycle validation are not scores; broader Agent Alone vs Agent + Sidecar evidence remains Sidecar v0.9 work.
Sidecar 0.8 provides Shared/Exclusive context-only, guardrails-only or combined configuration, explicit host context delivery and mandatory guardrail composition. Exclusive rules cannot weaken shared restrictions. Cross-framework privacy, revocation and delegated authority remain v0.9 validation; stable schemas remain v1.0. Native correlated Lens readback requires Lens 0.7.1. All sibling packages stay optional; no dependency floor is increased without a runtime requirement. See the Sidecar integration contract.
Control Tower 0.6.0 integration
Tower Phase 6 and 6A ship together in 0.6.0 with MCP 0.3.6: evidence-cited investigation, exact-preview human approval and verified deterministic staging recovery. Native producer outcomes and package ownership remain unchanged. See compatibility, examples and dependency matrix. This is documentation alignment; no package release or dependency bump is needed.
Published report and gate contract
As of 0.7.2, native EvaluationReport (schema version 1.0) and
GateDecision JSON Schemas ship in the package and are available offline:
from jsonschema import Draft202012Validator
from agentic_evals import report_schema, gate_decision_schema
# report and decision are the results of evaluate_suite/evaluate_gate.
Draft202012Validator(report_schema()).validate(report.model_dump(mode="json"))
Draft202012Validator(gate_decision_schema()).validate(decision.model_dump(mode="json"))
Contract 1.x changes are additive only; readers must tolerate unknown fields.
See contract policy and sibling compatibility and
validated JSON examples. The quickstart EvalResult
is a separate format. Native schema validation does not establish AIOS conformance.
Native consumer validation
MCP 0.3.8 adds evals.validate_artifact and validates Lens report inputs and
produced SLO gates. Control Tower 0.6.1 validates complete reports and supplied
gates before evidence projection. Both bundle exact Evals 0.7.2 schemas for
offline, sibling-independent validation. See the consumer matrix.
AgenticLens 0.8.0 suite management
AgenticLens 0.8.0 delivers its v0.6 milestone: agenticlens.suites and
agenticlens suite build regression suites from CSV/JSON imports, production
traces and user feedback. It masks PII by default before storage (--no-mask
opts out), detects duplicates and
conflicting labels, adds synthetic variations and stable splits, records
reviews, and approves suites into immutable, hash-verified versions that CI
verifies, exports and evaluates. Suites stay plain agentic-evals TestSuite
files.
Evals runs these suites unchanged with evaluate_suite and run_live_suite.
Lens keeps management state (lineage, review status, domain, split) in
TestCase.metadata and tags, and an exported approved version adds
metadata.agenticlens_suite_version (name, version, SHA-256, approver).
Suite approval is not a release gate: scoring and gate computation stay in
Evals. No Evals change or release is needed.
See the suite management guide.
AgenticLens 0.9.0 ModelFit
AgenticLens 0.9.0 delivers its v0.7 milestone: agenticlens.modelfit and
agenticlens modelfit recommend a model per task category from recorded
experiment reports, under hard quality, cost, latency, safety, provider and
deployment constraints that fail closed. Each recommendation carries a
confidence level with reasons, an explanation, alternatives, a fallback
(on a different provider when one is feasible), savings against a reference variant tied to the
experiment evidence, and quality risks, plus a task-level model matrix.
Evals is unchanged. ModelFit reads the native evaluation reports that Lens
experiments embed per trial (cases[].passed, latency_ms, cost_usd,
tags, metadata) and computes no release gate; scoring and gate
computation stay in Evals. Lens keeps its agentic-evals>=0.7.1,<1
dependency. No Evals change or release is needed.
See the ModelFit guide.
Chaos 0.7.0 interoperability
Chaos 0.7.0 adds sequential fault cascades, deterministic monotonic breaking-point search, synthetic response poisoning (including prompt/tool-response instruction and policy-bypass payloads), and explicit-price retry cost estimates. Native child event IDs and fault types remain intact; cascade IDs/stages and poisoning strategy/payload/seed are additive event detail. Existing report readers and AIOS exports remain compatible.
See the v0.7 guide and offline synthetic example. The example includes observed synthetic dispatches and a bypassed negative control. Injection or recovery alone does not prove containment. No new shared schema, automatic adapter, hosted execution capability or containment guarantee is introduced.
Scores and release gates remain owned by Evals; Chaos contracts and threshold trials are per-campaign evidence. Convert observations to native EvaluationSamples explicitly. No runtime dependency or patch release is needed.
Control Tower 0.7.0 fleet operations
Tower 0.7.0 adds frozen environment/capability targeting, bulk desired settings, separate approval, durable per-target results and bounded stop/MCP-revocation acknowledgement through Python, HTTP and CLI. No fleet atomicity or automatic rollback/resume is claimed. Timed-out/requested commands need host reconciliation.
Evals retains scoring and native report/gate decisions. Bulk execution does not recompute gates or equate passing evaluation with deployment health or containment.
Existing optional dependency ranges remain sufficient. No sibling runtime or native evidence schema changes are required; this documentation alignment needs no patch release. See the operator guide.
See the local Tower compatibility guide for the operator workflow and package boundaries.
AgenticLens 0.10.0 routing compatibility
See the local integration boundary guide.
AgenticLens 0.10.0 implements roadmap v0.8: evidence-based runtime model
selection, fail-closed routing constraints, bounded provider failover,
confidence/escalation handoffs, shadow/canary modes and a local routing journal.
The default is advisory; execution uses explicit caller-owned callbacks.
MCP 0.3.12 exposes read-only lens.routing_decide and pins hosted Lens 0.10.0.
See the routing guide
and offline example.
Lens retains agentic-evals>=0.7.1,<1, which covers current Evals 0.8.0.
Standalone scoring and release-gate computation remain Evals-owned. Routing
outcomes do not redefine native evaluation reports or gate decisions. No dependency
change or Evals patch release is required.
Chaos 0.8.0 interoperability
Chaos 0.8.0 adds:
- multi-stage chaos workflows: steps, steady-state health checks, parallel branches, approval gates that can be resumed, and reusable templates
- declarative YAML/JSON/TOML experiments, with validation that never executes targets and cron schedule previews
Scopeexplosion-radius control (steps, providers and exclusions) on every built-in fault andcascade(); custom faults get it by callingBaseFault.__init__, settingscopeor overridingin_scope()- a topology fuzzer that weights fault placement by fan-out
Native event IDs and fault types are unchanged. Workflow provenance
(detail.workflow) and an optional ChaosReport.workflow summary are
additive. A workflow's steps merge into one native report for contracts and
AIOS export.
See the v0.8 guide and the offline examples. Workflow statuses, health checks and fuzz outcomes are resilience evidence. They are not scores, release gates or proof of containment. No new shared schema, automatic adapter or hosted execution capability is introduced.
Workflow statuses and per-step contracts are per-run resilience checks, not Evals scores or release gates. Convert observed outputs to native EvaluationSamples explicitly. No adapter, runtime dependency or Evals patch release is needed.
Metadata
Release files for agentic-evals 0.9.0
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | Actions |
|---|---|---|---|---|
| agentic_evals-0.9.0.tar.gz | 365.7 kB | With provenance | Details for agentic_evals-0.9.0.tar.gz |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Actions Reset |
|---|---|---|---|---|
| agentic_evals-0.9.0-py3-none-any.whl | Python 3 | none | any | Details for agentic_evals-0.9.0-py3-none-any.whl |
Total release size: 471.9 kB
Release files / agentic_evals-0.9.0.tar.gz
| Download URL | agentic_evals-0.9.0.tar.gz |
|---|---|
| Size | 365.7 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
20cbf206f914539c0295a0998f4baa373d0c2f06e516ef3da4d9c4d34bccd914
|
|
BLAKE2b-256 checksum How to use checksums |
73a05251e272e2cd86f94e5ba8a18a7be384e9d0ac6f7b8461314ca3577949c4
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.14
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Oct 19, 2026.
Transparency logRelease files / agentic_evals-0.9.0-py3-none-any.whl
| Download URL | agentic_evals-0.9.0-py3-none-any.whl |
|---|---|
| Size | 106.2 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
fcc10dfbf359515a32d62f26bb07e6185455c247a84b48c01c2038040c2630d9
|
|
BLAKE2b-256 checksum How to use checksums |
8f855a42fa47244fef7ecc6327a9d4222f12a892762d4f1d6dd37a1aea9ec1fd
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.14
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Oct 19, 2026.
Transparency logSecurity for agentic-evals 0.9.0
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
Full provenance
Both files in this release have been uploaded with provenance metadata.
Attestation for agentic_evals-0.9.0.tar.gz
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Oct 19, 2026.
Transparency logA note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.