evalgrid 0.1.0

Agent evaluation framework — run, score, and track AI agent correctness at scale.

evalgrid

Agent evaluation framework — run, score, and track AI agent correctness at scale.

AI agents fail in ways unit tests can't catch: the customer support agent stops calling the refund tool after a prompt change. The research agent loops on the same Google search 23 times. The reconciliation agent silently picks the wrong account format on edge cases. evalgrid catches these regressions before they ship.

Status

EvalGrid is in early development (v0.1). The framework runs reliably for local-first agent evaluation, but you should expect:

• Some rough edges in the dashboard UI
• Schema changes in scenario YAML between v0.1 and v0.2
• GitHub status check / CI gate coming in v0.2

If you're building agents and want eval-driven workflows now, this works. If you need production-grade CI integration, watch for v0.2.

Screenshots

Dashboard showing run history with pass rates and per-scenario breakdowns

Per-step verdicts with judge reasoning for a failed scenario

---

Install

pip install evalgrid

Quick start

1. Initialize

evalgrid init

Scaffolds .evalgrid/ with a config file and an example scenario.

2. Write a scenario

.evalgrid/scenarios/qualify_lead.yml:

id: sdr-qualify-lead
name: SDR Lead Qualification
description: Verify BANT qualification on an inbound enterprise lead
agent_type: sdr

prompt: |
  You are an expert SDR. Qualify leads using the BANT framework.
  Identify budget, authority, need, and timeline. Recommend a next step.

test_input: |
  Prospect: Sarah Chen, VP Engineering at TechCorp (500 employees)
  Budget: $50k allocated this quarter for developer tooling
  She filled out a demo request form 20 minutes ago.

expected_actions:
  - action: identify_bant_criteria
    required: true
    description: Identify which BANT criteria are present
  - action: ask_timeline_question
    required: true
    description: Ask about the buying timeline
  - action: recommend_next_step
    required: false
    description: Propose a clear next step (demo, call, etc.)

tags: [sdr, bant]

3. Run

export ANTHROPIC_API_KEY=sk-ant-...
evalgrid run

4. See results

Running 1 scenario(s)...

 ✓  SDR Lead Qualification   sdr   PASS   0.94   3/3

1/1 scenarios passed (100%)
Results saved to .evalgrid/results/

5. Open the dashboard

evalgrid server

Then visit http://localhost:8000 for the full run history, pass rate trends, and per-step breakdowns.

Note: The dashboard binds to 127.0.0.1 by default and is designed for local use only. Do not expose it to a public network.

---

Commands

Command	Description
evalgrid init	Scaffold .evalgrid/ in your project
evalgrid run	Run all scenarios and print pass/fail
evalgrid run --tag sdr	Filter by tag
evalgrid run --mock	Dry run with no API calls (great for CI)
evalgrid run --output results.json	Save results to a file
evalgrid scenario add	Interactively create a new scenario
evalgrid scenario list	List all scenarios
evalgrid server	Start the React dashboard

Scenario format

id: unique-scenario-id
name: Human Readable Name
description: What this scenario tests
agent_type: sdr | coding | support | research | finance | legal | generic

prompt: |
  System prompt for your agent...

test_input: |
  The user message / input to evaluate...

expected_actions:
  - action: action_name
    required: true
    description: What this action entails

success_criteria: >
  Plain-language description of what a passing response looks like.

tags: [tag1, tag2]
timeout_seconds: 60

Architecture: agents and judges

EvalGrid is a runner-and-scorer framework. The agent (the LLM being tested) and the judge (the LLM doing the scoring) are configured independently and can use different providers via LiteLLM.

The example scenarios that ship with v0.1 use Claude for both — that's a demo choice, not an architectural constraint. Common real-world setups include:

• GPT-4o agent, Claude Sonnet judge (cross-model validation)
• Claude Haiku agent, Claude Sonnet judge (cost-efficient testing)
• Llama 3.1 agent, GPT-4o judge (open-source agent, frontier judge)

Configure in .evalgrid/config.yml:

model: gpt-4o          # the agent being tested
judge:
  provider: anthropic
  model: claude-sonnet-4-6   # the model scoring the agent

Examples

Example scenarios are in the examples/ directory. Copy them to get started:

cp examples/*.yml .evalgrid/scenarios/

GitHub Actions

Add to .github/workflows/ci.yml:

- name: Run evalgrid
  env:
    ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}
  run: evalgrid run --output results.json

Python API

from evalgrid.core import EvalConfig, ScenarioLoader, ScenarioRunner

config = EvalConfig(model="claude-sonnet-4-6")
loader = ScenarioLoader()
scenarios = loader.load_dir(".evalgrid/scenarios")

runner = ScenarioRunner(config)
results = runner.run_all(scenarios, on_result=lambda r: print(r.scenario_name, r.status))

for result in results:
    print(f"{result.scenario_name}: {result.pass_rate:.0%}")

Security

evalgrid is a local-only developer tool. Keep these constraints in mind:

• The API server binds to 127.0.0.1 by default. Use --bind to change this, but be aware that exposing the server to a network grants unauthenticated access to your scenarios and eval runs.
• Scenario files are sandboxed to ~/.evalgrid/scenarios/. Paths supplied via the API are validated and rejected if they escape this directory.
• Provider API keys (e.g. ANTHROPIC_API_KEY) are read from environment variables only — never pass them through the API.
• Do not run evalgrid server on a shared or public-facing machine.

Contributing

See CONTRIBUTING.md for full setup instructions and PR guidelines.

License

MIT. See LICENSE. Copyright (c) 2026 Naman Rai.