evalgrid 0.1.4

Agent evaluation framework — run, score, and track AI agent correctness at scale.

evalgrid

Agent evaluation framework — run, score, and track AI agent correctness at scale.

AI agents fail in ways unit tests can't catch: the customer support agent stops calling the refund tool after a prompt change. The research agent loops on the same Google search 23 times. The reconciliation agent silently picks the wrong account format on edge cases. evalgrid catches these regressions before they ship.

Status

EvalGrid is in early development (v0.1). The framework runs reliably for local-first agent evaluation, but you should expect:

• Some rough edges in the dashboard UI
• Schema changes in scenario YAML between v0.1 and v0.2
• GitHub status check / CI gate coming in v0.2

If you're building agents and want eval-driven workflows now, this works. If you need production-grade CI integration, watch for v0.2.

Screenshots

Dashboard showing run history with pass rates and per-scenario breakdowns

Per-step verdicts with judge reasoning for a failed scenario

---

Install

pip install evalgrid

Quick start

1. Initialize

evalgrid init

Scaffolds .evalgrid/ in your project directory (not your home directory) with a config file and an example scenario. evalgrid is local-first and project-scoped by default — see Where evalgrid stores data.

2. Try it without API keys (free, instant)

evalgrid run --mock

Uses a mock scorer — no LLM calls, no cost. Great for verifying the install and for CI.

3. Write a real scenario

.evalgrid/scenarios/qualify_lead.yml:

id: sdr-qualify-lead
name: SDR Lead Qualification
description: Verify BANT qualification on an inbound enterprise lead
agent_type: sdr

prompt: |
  You are an expert SDR. Qualify leads using the BANT framework.
  Identify budget, authority, need, and timeline. Recommend a next step.

test_input: |
  Prospect: Sarah Chen, VP Engineering at TechCorp (500 employees)
  Budget: $50k allocated this quarter for developer tooling
  She filled out a demo request form 20 minutes ago.

expected_actions:
  - action: identify_bant_criteria
    required: true
    description: Identify which BANT criteria are present
  - action: ask_timeline_question
    required: true
    description: Ask about the buying timeline
  - action: recommend_next_step
    required: false
    description: Propose a clear next step (demo, call, etc.)

tags: [sdr, bant]

4. Run with a real LLM

Set your API key, then run:

# bash / zsh
export ANTHROPIC_API_KEY=sk-ant-...
evalgrid run

# PowerShell (Windows)
$env:ANTHROPIC_API_KEY = "sk-ant-..."
evalgrid run

5. See results

Running 1 scenario(s)...

 ✓  SDR Lead Qualification   sdr   PASS   0.94   3/3   $0.0042

1/1 scenarios passed (100%)
Total cost: $0.0042
Results saved to .evalgrid/results/

Per-run cost (sum of agent + judge LiteLLM calls) is tracked automatically when LiteLLM has pricing data for your model. Token counts are reported for any model; cost shows — for local/unpriced models.

6. Open the dashboard

evalgrid server

Then visit http://localhost:8000 for the full run history, pass rate trends, and per-step breakdowns. If port 8000 is already in use, pass --port 8001 (or any free port).

Note: The dashboard binds to 127.0.0.1 by default and is designed for local use only. Do not expose it to a public network.

---

Where evalgrid stores data

evalgrid is local-first and project-scoped by default. When you run evalgrid init, files are created in the current working directory, not your home directory:

.evalgrid/config.yml       — runtime configuration
.evalgrid/scenarios/       — your eval definitions
.evalgrid/suites/          — benchmark suite definitions
.evalgrid/results.db       — SQLite database of run history
.evalgrid/results/         — exported reports (JSON, HTML, Markdown)

evalgrid run and evalgrid server resolve the project-local .evalgrid/ automatically when it exists. If no project-local .evalgrid/ is found, evalgrid falls back to ~/.evalgrid/ so you can still run scenarios from anywhere — but the recommended workflow is one .evalgrid/ per project.

Override the scenarios directory with the EVALGRID_SCENARIOS_DIR environment variable, or the database with DATABASE_URL (must be a sqlite+aiosqlite:// URL).

---

Commands

Command	Description
evalgrid init	Scaffold .evalgrid/ in your project
evalgrid run	Run all scenarios and print pass/fail
evalgrid run --tag sdr	Filter by tag
evalgrid run --mock	Dry run with no API calls (great for CI)
evalgrid run --output results.json	Save results to a file
evalgrid run --suite <id>	Run a named benchmark suite (4-rate scoring)
evalgrid run --report results.html	Generate an HTML/Markdown report (extension picks format)
evalgrid scenario add	Interactively create a new scenario
evalgrid scenario list	List all scenarios
evalgrid scenario validate <path>	Validate a scenario YAML file before committing
evalgrid server	Start the React dashboard
evalgrid flakiness	Report scenario stability across recent runs (exits 1 if any are flaky)

Scenario format

id: unique-scenario-id
name: Human Readable Name
description: What this scenario tests
agent_type: sdr | coding | support | research | finance | legal | generic

prompt: |
  System prompt for your agent...

test_input: |
  The user message / input to evaluate...

expected_actions:
  - action: action_name
    required: true
    description: What this action entails

success_criteria: >
  Plain-language description of what a passing response looks like.

tags: [tag1, tag2]
timeout_seconds: 60
weight: 1.0                  # used by suite weighted scoring

checks:                      # optional: deterministic structural assertions
  - type: valid_json         # output must parse as JSON
  - type: required_fields    # JSON output must have these top-level keys
    fields: [decision, reasoning]
  - type: enum               # a specific JSON field must be in an allowed set
    field: decision
    allowed: [APPROVE, REJECT, ESCALATE]
    hard_fail: true          # short-circuits before judge LLM cost

Deterministic checks

Checks run before the judge LLM. A hard_fail: true check (default) short-circuits the run and skips the judge entirely, saving cost. A hard_fail: false check records a soft failure but still calls the judge for semantic scoring.

type:	What it asserts	Extra keys
valid_json	Output parses as JSON	—
no_markdown	No fenced code blocks, headings, or bold markers	—
required_fields	Top-level JSON has these keys	fields: [str]
enum	A JSON field value is in an allowed set	field: str, allowed: [str]
forbidden_regex	A regex does NOT match the output (case-insensitive)	pattern: str
contains	A literal substring is present (case-insensitive)	text: str
max_length	Output length ≤ N chars	value: int
json_schema	Output validates against a JSON Schema (Draft 7)	schema: {...}, strict: bool

Benchmark suites

Group scenarios into named suites with a pass threshold and weighted scoring. Suite results are tracked separately as four rates: semantic pass, hard fail, soft fail, error — never collapsed into a single number.

.evalgrid/suites/python_bug_fixer.yml:

id: python-bug-fixer
name: Python bug-fixer benchmark
version: 1                   # bump when the benchmark definition changes
pass_threshold: 0.8          # suite passes if semantic_pass_rate >= 80%
scenarios:                   # by scenario id, OR omit and use `tags:` to match
  - coding-fix-off-by-one
  - coding-fix-mutable-default
  - coding-fix-iterator-exhausted
tags: [coding]               # alternative to scenarios:

Run with evalgrid run --suite python-bug-fixer. The dashboard's Benchmarks view tracks suite history.

Flakiness

A scenario whose pass/fail outcome flips across identical runs is flaky — exactly the signal you want before trusting a CI gate.

evalgrid flakiness               # all scenarios, last 10 runs
evalgrid flakiness --flaky-only  # only show unstable scenarios
evalgrid flakiness --window 30   # widen the rolling window

Exit code is 1 if any scenarios are flaky, 0 otherwise — so this is itself CI-gateable.

Architecture: agents and judges

EvalGrid is a runner-and-scorer framework. The agent (the LLM being tested) and the judge (the LLM doing the scoring) are configured independently and can use different providers via LiteLLM.

The example scenarios that ship with v0.1 use Claude for both — that's a demo choice, not an architectural constraint. Common real-world setups include:

• GPT-4o agent, Claude Sonnet judge (cross-model validation)
• Claude Haiku agent, Claude Sonnet judge (cost-efficient testing)
• Llama 3.1 agent, GPT-4o judge (open-source agent, frontier judge)

Configure in .evalgrid/config.yml:

model: gpt-4o          # the agent being tested
judge:
  provider: anthropic
  model: claude-sonnet-4-6   # the model scoring the agent

Examples

Example scenarios are in the examples/ directory. Copy them to get started:

cp examples/*.yml .evalgrid/scenarios/

GitHub Actions

Add to .github/workflows/ci.yml:

- name: Run evalgrid
  env:
    ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}
  run: evalgrid run --output results.json

Python API

from evalgrid.core import EvalConfig, ScenarioLoader, ScenarioRunner

config = EvalConfig(model="claude-sonnet-4-6")
loader = ScenarioLoader()
scenarios = loader.load_dir(".evalgrid/scenarios")

runner = ScenarioRunner(config)
results = runner.run_all(scenarios, on_result=lambda r: print(r.scenario_name, r.status))

for result in results:
    print(f"{result.scenario_name}: {result.pass_rate:.0%}")

Security

evalgrid is a local-only developer tool. Keep these constraints in mind:

• The API server binds to 127.0.0.1 by default. Use --bind to change this, but be aware that exposing the server to a network grants unauthenticated access to your scenarios and eval runs.
• Scenario files are sandboxed to your project's .evalgrid/scenarios/ directory (or ~/.evalgrid/scenarios/ as a fallback when no project directory exists — see Where evalgrid stores data). Paths supplied via the API are validated and rejected if they escape this directory.
• Provider API keys (e.g. ANTHROPIC_API_KEY) are read from environment variables by default. The dashboard's "Run new eval" form may also accept a key in-browser and forward it on a single POST /api/runs call for convenience; that key is held only for the duration of the run and is never written to disk, logged, or persisted. The env-var path is preferred for any non-interactive use (CLI, CI).
• All error messages are passed through a regex that redacts strings matching common provider key prefixes (sk-ant-, sk-, pypi-, AIza…) before they reach logs or HTTP responses.
• Do not run evalgrid server on a shared or public-facing machine.

Contributing

See CONTRIBUTING.md for full setup instructions and PR guidelines.

License

MIT. See LICENSE. Copyright (c) 2026 Naman Rai.