EvalBridge

Connect agents to rigorous evaluations effortlessly

EvalBridge is a production-grade Python library for contract-first evaluation of LLM agents. Run contract-first eval scenarios against a user-supplied Python agent while capturing tool traces, validating structured outputs, scoring with typed rubrics, clustering failures by cause, and producing a release scorecard with actionable diffs. You define scenarios with Pydantic models, plug in any agent that implements a small turn protocol, run under fixed budgets and guardrails, and export JUnit/JSON/Markdown/HTML for CI and review.

• PyPI distribution: set via PYPI_PACKAGE_NAME (default build name in this repo: kaisen-evalbridge)
• Import package: kaisen_evalbridge
• CLI: kaisen-evalbridge (entry point from kaisen_evalbridge.cli:main)
• Python: 3.11+

Features

Scenario DSL (code-first)

Define eval cases in Python with typed inputs, expected structured outputs (Pydantic BaseModel), and attached rubrics.

What it does: Each Scenario carries an id, name, optional description, inputs, expected, plus lists of DeterministicRubric and LLMRubric instances. Output typing is inferred from the expected model for harness validation.

How to use it: Call Scenario.define(...) with your input/expected model instances and rubrics. Use the scenario as the first argument to ContractHarness.run(scenario, agent).

from pydantic import BaseModel
from kaisen_evalbridge import Scenario, DeterministicRubric

class TaskIn(BaseModel):
    task: str

class TaskOut(BaseModel):
    result: str

scenario = Scenario.define(
    id="task-1",
    name="Complete task",
    inputs=TaskIn(task="run"),
    expected=TaskOut(result="done"),
    deterministic_rubrics=[
        DeterministicRubric(
            id="match",
            description="Exact result",
            evaluate=lambda output, expected: output.result == expected.result,
        ),
    ],
)

Contract execution harness

Run agents under comparable limits: max turns, retries, and tool guardrails.

What it does: ContractHarness loops agent turns until the agent marks done=True, validates each turn’s structured_output against the scenario output type, enforces GuardrailConfig (allowed/blocked tools, max tool calls per turn), scores rubrics on success, and returns a HarnessResult with status passed, failed, guardrail, validation, or budget_exhausted. Traces are recorded automatically.

How to use it: Implement the Agent protocol (run_turn(inputs, turn_index, history) -> AgentTurnResult). Construct ContractHarness(budget=ExecutionBudget(...), guardrails=GuardrailConfig(...)) and call run(scenario, agent).

from kaisen_evalbridge import AgentTurnResult, ContractHarness, ExecutionBudget

class MyAgent:
    def run_turn(self, *, inputs, turn_index, history):
        return AgentTurnResult(
            structured_output=TaskOut(result="done"),
            done=True,
        )

result = ContractHarness(budget=ExecutionBudget(max_turns=5, max_retries=1)).run(
    scenario, MyAgent()
)

Trace capture

Record prompts, messages, tool calls, structured outputs, and per-step validation.

What it does: The harness uses TraceRecorder internally. Each step stores optional prompt, ChatMessage list, ToolCallRecord list, serialized structured output, and ValidationResult. The finished Trace is on HarnessResult.trace. You can also use TraceRecorder directly for custom instrumentation.

How to use it: Inspect result.trace.steps after a harness run, or build traces manually with TraceRecorder(scenario_id).record_step(...) and finish().

Typed scoring pipeline

Aggregate schema compliance, rubric outcomes, and optional custom validators.

What it does: ScoringPipeline.score(result) / score_many(results) produce ScenarioScoreReport with passed, schema_compliance, mean_rubric_score, and rubric_scores. DeterministicValidator runs pure functions over HarnessResult. rubric_scores_to_json serializes rubric scores for artifacts.

How to use it: Pass harness results to ScoringPipeline().score_many(results) or rely on build_release_scorecard, which uses the pipeline internally.

Rubrics (deterministic and LLM-graded)

Attach checkers that emit typed RubricScore JSON (rubric_id, kind, passed, score, details).

What it does: DeterministicRubric runs a Python callable (evaluate(output, expected)). LLMRubric delegates to your LLMGrader.grade(...) implementation. The harness runs all scenario rubrics after a valid structured output is produced.

How to use it: Add rubrics on Scenario.define(..., deterministic_rubrics=[...], llm_rubrics=[...]), or materialize them from golden manifests via GoldenSetRegistry.

Failure clustering

Group failed runs into root-cause categories using rules and trace-derived features.

What it does: extract_trace_features(result) builds compact features (status, step counts, validation failures, tool usage, guardrail flag). cluster_failures(results, rules) returns ordered FailureCluster lists with counts and example scenario IDs. default_cluster_rules() covers guardrail, schema, budget, and rubric failures.

How to use it: Pass custom ClusterRule lists to cluster_failures or to build_release_scorecard(..., cluster_rules=...).

Release scorecard

Compact per-scenario and aggregate metrics plus top failure clusters.

What it does: build_release_scorecard(results) returns ReleaseScorecard with per_scenario entries (pass flag, schema compliance, rubric scores), aggregate metrics (pass rate, mean schema compliance, mean rubric score, rubric pass rate), and top_failure_clusters.

How to use it: After a batch harness run, scorecard = build_release_scorecard(results); read scorecard.aggregate.pass_rate and scorecard.top_failure_clusters for release review.

Regression gating

Configurable CI thresholds with structured violations.

What it does: evaluate_regression_gate(results=..., thresholds=RegressionThresholds(...)) checks minimum pass rate, mean schema compliance, per-rubric minimum scores, and optionally blocks new high-severity clusters compared to a baseline set. Returns RegressionGateResult with passed and violations (GateViolation with codes such as pass_rate, schema_compliance, rubric_score, new_high_severity_cluster).

How to use it: In CI, fail the job when not gate.passed; attach gate.violations to logs. Include the gate in exports via build_eval_artifacts(..., regression_gate=gate).

Golden-set management as code

Version eval cases as JSON manifests plus a runtime registry.

What it does: GoldenSetManifest / GoldenCaseSpec / RubricConfigSpec describe cases (model names, input/expected dicts, rubric configs). load_golden_manifest(path) and save_golden_manifest(manifest, path) round-trip JSON. GoldenSetRegistry maps model names and checker/grader refs to Python types and callables. materialize_scenario / materialize_golden_set produce runnable Scenario objects.

How to use it: Commit manifests beside tests (see tests/fixtures/golden_manifest.json), register Pydantic models and checker functions, then scenarios = materialize_golden_set(manifest, registry) and run the harness on each.

Near-miss explorer

Find similar past failures using embeddings over run summaries (local, optional store).

What it does: summarize_run(result) builds a text summary from scenario id, status, and trace features. NearMissExplorer with LocalNearMissStore indexes runs via HashedEmbeddingProvider (deterministic local vectors, no external API). find_near_misses(result, top_k=...) returns NearMissMatch rows with cosine similarity. Optional persistence: LocalNearMissStore(directory=...) loads/saves cases on disk.

How to use it: After failures, explorer.index_result(past_result) for historical runs; on a new failure, explorer.find_near_misses(failed_result) to surface comparable cases.

Artifact export

JUnit for CI, JSON for machines, Markdown/HTML for humans.

What it does: build_eval_artifacts(results, ...) builds an EvalArtifactDocument (scorecard, per-scenario summaries, optional regression gate) plus json_text, junit_xml, markdown, and html. write_eval_artifacts(artifacts, directory) writes {prefix}.json, .xml, .md, and .html files.

How to use it: artifacts = build_eval_artifacts(results, regression_gate=gate) then write_eval_artifacts(artifacts, "./reports") or publish artifacts.junit_xml in your CI test reporter.

CLI

Minimal command-line entry for version and help.

What it does: Prints distribution name and version, or usage help.

How to use it: kaisen-evalbridge --version · kaisen-evalbridge (help)

Package metadata

Resolve distribution name from the environment at runtime and build time.

What it does: get_pypi_package_name() / get_import_package_name() read PYPI_PACKAGE_NAME. package_info() returns PackageInfo (distribution name, import name, version).

How to use it: from kaisen_evalbridge import package_info for tooling and the CLI prog name.

Pages

EvalBridge is delivered as a Python library, not a hosted web application. There are no frontend routes or authenticated UI in this repository.

Location	What you see
(no web app)	All product workflows are code-first: define scenarios, run ContractHarness, export artifacts from your scripts or CI.
frontend/public/logo.svg	Brand logo asset (primary #0056a6, accent #ff8c00).
frontend/public/favicon.svg	Favicon asset for future or external docs sites.

API

There is no HTTP backend in this repo—no REST paths under /api. Integration is through the public Python API exported from kaisen_evalbridge (and the CLI below).

Configuration and metadata

Entry point	Purpose
get_pypi_package_name(required=True)	Read PyPI/wheel distribution name from PYPI_PACKAGE_NAME.
get_import_package_name(required=True)	Derive Python import name (dashes → underscores).
package_info()	Return PackageInfo (distribution name, import name, version).
__version__	Package version string.

Scenarios and rubrics

Entry point	Purpose
Scenario.define(...)	Create a typed scenario with inputs, expected output, and rubrics.
DeterministicRubric / .run(output=, expected=)	Deterministic rubric definition and execution.
LLMRubric / .run(output=, expected=)	LLM rubric using your LLMGrader implementation.
RubricScore	Typed rubric result model (JSON-serializable).

Agent and harness

Entry point	Purpose
Agent (protocol)	Contract for user agents (run_turn).
AgentTurnResult	Per-turn prompt, messages, tool calls, structured output, done.
ContractHarness.run(scenario, agent)	Execute scenario with trace capture and rubric scoring.
ExecutionBudget	max_turns, max_retries.
GuardrailConfig	Allowed/blocked tools, max tool calls per turn.
HarnessResult	Run outcome, status, trace, rubric scores, errors.

Trace

Entry point	Purpose
TraceRecorder	Record steps and finish a Trace.
Trace, TraceStep	Scenario-level trace container and step records.
ChatMessage, MessageRole	Message model for traces and turns.
ToolCallRecord	Tool name and arguments on a step.
ValidationResult	Per-step schema validation pass/fail and errors.

Scoring

Entry point	Purpose
ScoringPipeline.score(result)	Score one harness result.
ScoringPipeline.score_many(results)	Score a batch.
DeterministicValidator / .run(result)	Custom deterministic validator over a run.
rubric_scores_to_json(scores)	Serialize rubric scores to JSON-friendly dicts.

Clustering and scorecard

Entry point	Purpose
extract_trace_features(result)	Features for clustering rules.
cluster_failures(results, rules)	Group failures into FailureCluster list.
ClusterRule, FailureCluster, TraceFeatures	Rule and result models.
default_cluster_rules()	Built-in guardrail/schema/budget/rubric rules.
build_release_scorecard(results, ...)	Build ReleaseScorecard with aggregates and top clusters.

Regression gating

Entry point	Purpose
RegressionThresholds	Min pass rate, schema compliance, per-rubric mins, new high-severity cluster blocking.
evaluate_regression_gate(...)	Evaluate thresholds; return RegressionGateResult.
GateViolation, RegressionGateResult	Violation detail and overall pass/fail.

Golden set

Entry point	Purpose
load_golden_manifest(path)	Load JSON manifest.
save_golden_manifest(manifest, path)	Write JSON manifest.
GoldenSetRegistry	Register Pydantic models, checkers, graders by name.
materialize_scenario(spec, registry)	One Scenario from a case spec.
materialize_golden_set(manifest, registry)	All scenarios from a manifest.
scenario_to_golden_spec(scenario, ...)	Export a scenario to a serializable spec.

Near-miss explorer

Entry point	Purpose
summarize_run(result)	Text summary for embedding.
cosine_similarity(a, b)	Similarity helper for vectors.
HashedEmbeddingProvider	Local deterministic embeddings.
LocalNearMissStore	In-memory or on-disk case store.
NearMissExplorer.index_result(result)	Index a run for later search.
NearMissExplorer.find_near_misses(result, top_k=5)	Retrieve similar indexed cases.

Artifacts

Entry point	Purpose
build_eval_artifacts(results, ...)	Build full EvalArtifacts bundle.
export_json(document)	JSON string from EvalArtifactDocument.
export_junit_xml(results, suite_name=...)	JUnit XML string.
export_markdown_report(document, ...)	Markdown report string.
export_html_report(document, ...)	HTML report string.
write_eval_artifacts(artifacts, directory, prefix=...)	Write all formats to disk.

Command-line interface

Invocation	Purpose
kaisen-evalbridge --version	Print {distribution_name} {version} and exit 0.
kaisen-evalbridge	Print help and exit 0.

Configuration

Variable	Required	Purpose
PYPI_PACKAGE_NAME	Yes (runtime and build)	PyPI/wheel distribution name. Must match [project].name in pyproject.toml when building. If unset, get_pypi_package_name() and package builds raise a clear error. Copy from .env.example; do not commit .env.

Third-party API keys: None are required by EvalBridge itself. LLM rubrics use your own LLMGrader implementation (you supply credentials in your agent or grader code). The near-miss store uses HashedEmbeddingProvider by default (offline); swap in another EmbeddingProvider if you connect an external embedding API from your application code.

Run it locally

From a clone, using uv:

export PYPI_PACKAGE_NAME=kaisen-evalbridge
uv run python scripts/ensure_pyproject_name.py
uv sync

Use the library in Python (set PYPI_PACKAGE_NAME in the environment or before import):

import os
os.environ.setdefault("PYPI_PACKAGE_NAME", "kaisen-evalbridge")

from kaisen_evalbridge import ContractHarness, Scenario
# ... define models, scenario, agent, then harness.run(...)

CLI:

export PYPI_PACKAGE_NAME=kaisen-evalbridge
uv run kaisen-evalbridge --version

Test:

export PYPI_PACKAGE_NAME=kaisen-evalbridge
uv run pytest

Lint, typecheck, and build (as in project automation):

export PYPI_PACKAGE_NAME=kaisen-evalbridge
uv run ruff check .
uv run mypy src
uv build

Install from PyPI (when published) with pip install <your PYPI_PACKAGE_NAME value>; development installs use uv sync as above.

Architecture

Layer	Technology
Frontend	— (brand SVG assets only under frontend/public/)
Backend / API server	— (library-only; no HTTP service)
Database	— (optional local JSON files for LocalNearMissStore when a directory is configured)

Repository layout

Path	Role
src/kaisen_evalbridge/	Library source: harness, scenarios, rubrics, trace, scoring, clustering, scorecard, regression, golden manifests, near-miss store, artifacts, config, CLI.
tests/	Pytest suite and golden manifest fixture.
scripts/ensure_pyproject_name.py	Syncs pyproject.toml project name with PYPI_PACKAGE_NAME.
hatch_build.py	Hatch metadata hook enforcing name match at build time.
pyproject.toml / uv.lock	Project metadata and locked dependencies (pydantic runtime; pytest, ruff, mypy for dev).
frontend/public/	Static logo and favicon.

How components connect

Your application (or CI job) imports kaisen_evalbridge, defines Pydantic contracts and scenarios, and passes an Agent instance into ContractHarness.run. The harness calls your agent each turn, validates structured outputs, enforces guardrails, and fills a Trace. Rubric scores attach to HarnessResult. Downstream modules consume the same results: ScoringPipeline and build_release_scorecard for metrics, cluster_failures for root-cause groups, evaluate_regression_gate for CI policy, build_eval_artifacts / write_eval_artifacts for reports, and NearMissExplorer for similarity over past runs. There is no separate frontend or backend process—everything runs in-process in Python.

flowchart LR
  subgraph your_code [Your code]
    Agent[Agent.run_turn]
    Scenarios[Scenario / golden manifest]
  end
  subgraph evalbridge [kaisen_evalbridge]
    Harness[ContractHarness]
    Trace[TraceRecorder]
    Score[ScoringPipeline / scorecard]
    Gate[evaluate_regression_gate]
    Artifacts[build_eval_artifacts]
    NearMiss[NearMissExplorer]
  end
  Scenarios --> Harness
  Agent --> Harness
  Harness --> Trace
  Harness --> Score
  Score --> Gate
  Score --> Artifacts
  Harness --> NearMiss

License

MIT