EvalForge — CI/CD for AI Agents

EvalForge is an infrastructure-first evaluation harness for tool-using, multi-step AI agents. It turns agent behavior into repeatable tests that can run locally or in CI and block a deployment when quality regresses.

Change a prompt, model, tool schema, or RAG pipeline → run scenarios → capture trajectories → grade outcomes → compare against a baseline → enforce gates.

Why this exists

Normal unit tests are necessary but insufficient for agents. Agents are non-deterministic, make multiple model calls, use tools, retrieve context, and mutate environments. EvalForge treats the trajectory and outcome as first-class test artifacts instead of evaluating only the final string.

v0.1 capabilities

• YAML evaluation suites and reusable scenarios
• Multiple trials per task + concurrent execution
• Language-agnostic command adapter (JSON over stdin/stdout)
• HTTP adapter for already-running agents
• Full trajectory/tool-call capture
• Deterministic graders: exact match, contains, regex, similarity, JSON contract
• Tool sequence and tool argument correctness
• Citation-based RAG groundedness proxy
• Known-hallucination regression guard
• Optional OpenAI-compatible LLM-as-judge grader
• Human-score ingestion that participates in the same pass/fail pipeline
• Task success, tool correctness, hallucination rate, latency, token and cost aggregation
• Baseline-vs-candidate regression comparison
• Prompt/model A/B runs over the exact same suite
• Absolute and regression-based CI gates
• Trajectory replay / re-grading without paying for another agent run
• FastAPI run-inspection API
• Docker + GitHub Actions example

Architecture

                   ┌────────────────────────┐
                   │  YAML Evaluation Suite │
                   │ tasks · trials · gates │
                   └────────────┬───────────┘
                                │
                    ┌───────────▼───────────┐
                    │      Eval Runner      │
                    │ concurrency · retries │
                    └───────┬────────┬──────┘
                            │        │
             ┌──────────────▼─┐    ┌─▼──────────────┐
             │ Agent Adapter   │    │ Recorded Replay │
             │ command / HTTP  │    │ no model call   │
             └────────┬────────┘    └──────┬─────────┘
                      │                    │
                trajectory + output + usage
                      │                    │
                ┌─────▼────────────────────▼──┐
                │          Graders            │
                │ code · tool · RAG · LLM     │
                └──────────────┬──────────────┘
                               │
                       ┌───────▼────────┐
                       │ Metrics + Runs │
                       └───────┬────────┘
                               │
                    ┌──────────▼──────────┐
                    │ Regression CI Gates │
                    │  PASS / BLOCK       │
                    └─────────────────────┘

Run it

python -m venv .venv
source .venv/bin/activate
make install      # hash-pinned deps from requirements/dev.lock, then the package itself
pytest

evalforge run \
  --suite examples/suites/customer_support.yaml \
  --agent "python examples/agents/demo_agent.py" \
  --out runs/demo.json

A failing gate exits with code 2, which is what blocks a CI job. Invalid input — a malformed suite, an unknown gate metric, an unreadable run file — exits with code 1 and a one-line error, before any agent is called where possible.

Dependencies are pinned with hashes in requirements/*.lock (runtime, dev, dev+postgres), used by CI and the Docker image. After changing dependencies in pyproject.toml, regenerate them with make lock (needs uv).

Agent protocol

EvalForge deliberately does not force a LangChain/LlamaIndex/custom-agent dependency. For the command adapter, your executable receives one TaskSpec JSON object on stdin and returns one AgentOutput JSON object on stdout:

{
  "output": "Refund submitted. Source: KB-REFUND-1",
  "trajectory": [
    {"type": "message", "role": "user", "content": "Refund order R200"},
    {"type": "tool_call", "tool_call": {"name": "lookup_order", "arguments": {"order_id": "R200"}}}
  ],
  "tool_calls": [
    {"name": "lookup_order", "arguments": {"order_id": "R200"}}
  ],
  "usage": {
    "input_tokens": 100,
    "output_tokens": 35,
    "cost_usd": 0.012
  },
  "model": "your-model"
}

That adapter boundary lets the same suite evaluate Python, TypeScript, Java, Go, MCP-based agents, or remote services.

Suite example

name: checkout-agent
trials_per_task: 3
concurrency: 8

gates:
  - metric: task_success
    op: ">="
    value: 0.90
  - metric: avg_cost_usd
    op: "<="
    value: 0.15
  - metric: task_success
    op: ">="
    max_regression: 0.03

tasks:
  - id: refund-order
    input: Refund order R200.
    expected_tools: [lookup_order, issue_refund]
    graders:
      - type: tool_sequence
        name: tool_correctness
        pass_threshold: 1.0

For a regression gate, provide a baseline. A quality metric with op: ">=" is allowed to drop by at most max_regression; a cost/latency metric with op: "<=" is allowed to rise by at most that amount. A regression gate with no baseline to compare against fails (shown as no baseline in the gates table) — it can't pass without one.

evalforge run --suite examples/suites/customer_support.yaml --agent "..." \
  --out runs/pr-184.json --baseline-branch main

--baseline-branch main auto-resolves the baseline to the latest run of this suite on that branch — no more hand-tracking runs/main.json yourself. It looks in Postgres if EVALFORGE_DATABASE_URL is set, otherwise in --out's directory. GITHUB_HEAD_REF/GITHUB_REF_NAME set this automatically in GitHub Actions; set GIT_BRANCH yourself elsewhere. Use --baseline <path> instead for a fixed file, same as before — the two are mutually exclusive. Either way, the run prints which baseline it resolved (run id, branch, commit, timestamp), and regression gates show the baseline value and delta next to each result. Run files in that directory that can't be read (corrupt, or from an older schema_version) are skipped with a logged warning rather than silently. If the baseline came from a suite definition that's since changed (a task, grader, or gate edited — not just its description or concurrency), you'll get a warning that the comparison may not be meaningful, since evalforge tracks a content hash (suite_version) of every run's suite on the EvalRun itself.

In CI, --baseline-branch needs storage that outlives the job. Each CI job starts with an empty workspace, so without Postgres the --out directory never contains an earlier run and every regression gate fails with no baseline. Either set EVALFORGE_DATABASE_URL (as a secret) to a Postgres instance your CI can reach, or restore previous run files into the --out directory before running (e.g. actions/cache or downloading the main branch's last evalforge-run artifact).

Replay a trajectory

When you improve a grader or rubric, re-score an existing run without spending tokens again:

evalforge replay \
  --suite examples/suites/customer_support.yaml \
  --run runs/demo.json \
  --out runs/replayed.json

Compare two runs

evalforge compare --baseline runs/main.json --candidate runs/pr-184.json

Prompt/model A/B test

Run the same task bank against two commands:

evalforge ab \
  --suite examples/suites/customer_support.yaml \
  --agent-a "python agent_prompt_v11.py" \
  --agent-b "python agent_prompt_v12.py"

EvalForge stores both run artifacts and prints metric deltas. The two commands can represent different prompts, models, tool schemas, retrieval strategies, or entire agent implementations.

Human grading

Human scores can be imported without changing the run pipeline:

- type: human
  name: human_quality
  pass_threshold: 0.8
  config:
    file: reviews/human_scores.json

{
  "refund": {"score": 0.95, "reason": "Correct and policy-compliant"},
  "order-status": 1.0
}

Inspect runs over HTTP

evalforge serve --host 0.0.0.0 --port 8000
# GET /health
# GET /runs?limit=20&offset=0&suite=customer-support-regression
# GET /runs/{key}

Interactive FastAPI docs are available at /docs.

By default runs are read back from the local runs/ directory ({key} is a filename). Set EVALFORGE_DATABASE_URL to persist runs to Postgres instead — evalforge run/ab/replay write to it in addition to the local JSON file, and /runs//runs/{key} ({key} becomes the run id) query it directly with real pagination and suite filtering instead of parsing every file in runs/ on every request:

export EVALFORGE_DATABASE_URL=postgresql://evalforge:evalforge@localhost:5432/evalforge
pip install -e ".[postgres]"

Database connections are pooled (EVALFORGE_DB_POOL_MAX_SIZE, default 10). Runs stored under a different schema_version are treated the same way in both backends: skipped (with a logged warning) when listing or resolving a baseline, and a 409 with the reason from /runs/{key}.

docker-compose.yml runs a Postgres container alongside the API and wires this automatically. It requires a .env file next to it:

cp .env.example .env    # then set POSTGRES_PASSWORD and EVALFORGE_API_KEY (e.g. openssl rand -hex 32)
docker compose up --build

Compose refuses to start without both values, and publishes the API on 127.0.0.1:8000 only; put a TLS-terminating reverse proxy in front of it to expose it further. (If a evalforge-db volume already exists from an older setup, Postgres keeps the password it was initialized with — change it in the database, or remove the volume, to match .env.)

Securing the API

The API is open by default — fine for localhost, not for anywhere else. Before exposing it beyond your own machine:

export EVALFORGE_API_KEY=some-long-random-value      # required Authorization: Bearer <key> on /runs*
export EVALFORGE_CORS_ORIGINS=https://your-dashboard.example  # comma-separated allowlist; unset = no cross-origin reads
export EVALFORGE_RATE_LIMIT_PER_MINUTE=60             # per client IP; see below

The API key is compared in constant time. The rate limit is counted in Postgres when EVALFORGE_DATABASE_URL is set (a fixed one-minute window shared by every replica); without a database it falls back to an in-memory counter per process. Behind a reverse proxy every request arrives from the proxy's IP, so rate-limit there (or by forwarded client IP at the proxy) rather than relying on this per-IP limit alone.

/health never requires the API key, so container/orchestrator health checks keep working unauthenticated. Every request gets an X-Request-ID (echoed back if the caller sends one), and the API and runner emit one JSON log line per request/run to stderr — evalforge run keeps its human-readable table on stdout, so scripts piping stdout are unaffected.

LLM-as-judge

Add this grader to a task or suite:

- type: llm_judge
  name: policy_quality
  pass_threshold: 0.8
  config:
    model: your-judge-model
    rubric: >-
      Score whether the answer follows the refund policy, cites evidence,
      and avoids claiming actions that did not occur.

Then set:

export EVALFORGE_JUDGE_API_KEY=...
export EVALFORGE_JUDGE_BASE_URL=https://api.openai.com/v1
export EVALFORGE_JUDGE_MODEL=...

The implementation is OpenAI-compatible so it can also point to compatible gateways/local inference servers.

Releasing

The package is published to PyPI as evalforge-agents (import it as evalforge_agents; the CLI command is evalforge). Releases go through .github/workflows/release.yml with PyPI Trusted Publishing, so no API token is stored anywhere.

One-time setup:

1. On PyPI → Your account → Publishing → Add a new pending publisher: project evalforge-agents, owner AbhilashSomigari, repository evalforge, workflow release.yml, environment pypi.
2. Optional but recommended: in the GitHub repo → Settings → Environments → pypi, add yourself as a required reviewer so every release waits for a manual approval.

Each release: bump version in pyproject.toml, commit, then push a matching tag — the workflow refuses a tag that doesn't match:

git tag v0.1.0 && git push origin v0.1.0

It re-runs lint, type checks and tests, builds the sdist and wheel, checks them with twine check --strict, then uploads.

What comes next

The next platform milestones are: distributed workers; human review queues; richer LLM/NLI groundedness graders; environment snapshots and state-based outcome grading; provider-native OpenAI/Anthropic adapters; statistical confidence intervals; flaky-eval detection; prompt/model matrix experiments; GitHub PR annotations; and an observability UI built on the same run schema.

Design principle

The run artifact is the product. The UI is only one consumer. Every trial stores the task, outcome-facing output, tool calls, trajectory, grader results, latency, usage and cost so the same evidence can drive CI, debugging, dashboards, human review, and future graders.