evalgate 0.3.0

Deterministic LLM/RAG evals as a PR check

AOTP Ventures EvalGate

EvalGate runs deterministic LLM/RAG evals as a PR check. It compares your repo's generated outputs against fixtures, validates formatting, label accuracy, latency/cost budgets, and can use LLMs as judges for complex criteria. It posts a readable summary on the PR. Default is local-only (no telemetry).

• ✅ Deterministic checks (schema/labels/latency/cost)
• 🧠 LLM-based evaluation for complex criteria
• 🧪 Regression vs main baseline
• 🔒 Local-only by default; optional “metrics-only” later
• 🧰 Zero infra — a composite GitHub Action + tiny CLI

Quick Start

1. Install and Initialize

# Initialize EvalGate in your project
uvx --from evalgate evalgate init

# This creates:
# - .github/evalgate.yml (configuration)
# - eval/fixtures/ (test data with expected outputs)
# - eval/schemas/ (JSON schemas for validation)

2. Generate Your Model's Outputs

# Run your model/system to generate outputs for the fixtures
# (Replace with your actual prediction script)
python scripts/predict.py --in eval/fixtures --out .evalgate/outputs

3. Run Evaluation

# Run the evaluation suite
uvx --from evalgate evalgate run --config .github/evalgate.yml

# View results summary
uvx --from evalgate evalgate report --summary --artifact .evalgate/results.json
# Inside GitHub Actions, add --check-run to publish results as a check run

4. Update Baseline (optional)

When your fixtures or model outputs change, update the stored baseline results. This runs the evals and commits the results to the git ref specified by baseline.ref (default origin/main).

uvx --from evalgate evalgate baseline update --config .github/evalgate.yml

Pull requests will be compared against these baseline results.

Conversation Fixtures

When working with chat-based models, fixtures can describe full conversations. Each conversation fixture contains a list of messages, where every message has a role (such as system, user, assistant, or tool) and content. Assistant messages may optionally include tool_calls describing functions the assistant wants to invoke.

Example conversation fixture:

{
  "messages": [
    { "role": "user", "content": "Hello!" },
    {
      "role": "assistant",
      "content": "Hi there!",
      "tool_calls": [
        { "name": "search", "arguments": { "query": "Hello!" } }
      ]
    }
  ]
}

LLM as Judge

EvalGate can use LLMs to evaluate outputs for complex criteria beyond simple schema validation.

1. Install with LLM support

# Install with LLM dependencies
pip install evalgate[llm]
# Or with uv
uvx --from evalgate[llm] evalgate

2. Create a prompt template

Create an evaluation prompt in eval/prompts/quality_judge.txt:

You are evaluating the quality of customer support responses.

INPUT:
{input}

EXPECTED:
{expected}

OUTPUT:
{output}

Rate from 0.0 to 1.0 based on accuracy, helpfulness, and tone.
Score: [your score]

3. Configure your evaluator

In .github/evalgate.yml:

evaluators:
  # Your existing evaluators...
  - name: content_quality
    type: llm
    provider: openai  # or anthropic, azure, local
    model: gpt-4      # or other models
    prompt_path: eval/prompts/quality_judge.txt
    api_key_env_var: OPENAI_API_KEY
    weight: 0.3
    min_score: 0.75  # fail if score < 0.75

min_score enforces a minimum evaluator score; the run fails if the score drops below this threshold.

4. Set your API key

export OPENAI_API_KEY=your_api_key_here

5. Run evaluation

evalgate run --config .github/evalgate.yml

LLM judge responses are cached in .evalgate/cache.json. Reuse cached results to avoid repeat API calls or clear them with:

evalgate run --config .github/evalgate.yml --clear-cache

GitHub Actions Integration with API Keys

Add your API keys as repository secrets in GitHub, then use them in your workflow:

# Optional: Validate API key is set (fail fast with clear message)
- name: Validate OpenAI API Key  
  run: |
    if [ -z "${{ secrets.OPENAI_API_KEY }}" ]; then
      echo "❌ OPENAI_API_KEY secret is not set"
      echo "Please add your OpenAI API key as a repository secret named 'OPENAI_API_KEY'"
      echo "Go to: Settings > Secrets and variables > Actions > New repository secret"
      exit 1
    fi

- name: Run EvalGate with LLM judge
  env:
    OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
    ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}
  run: uvx --from evalgate[llm] evalgate run --config .github/evalgate.yml

Or with the composite action:

- uses: aotp-ventures/evalgate@main
  with:
    config: .github/evalgate.yml
    openai_api_key: ${{ secrets.OPENAI_API_KEY }}
    anthropic_api_key: ${{ secrets.ANTHROPIC_API_KEY }}
    check_run: true

Tool Usage Logs

Model outputs can record tool invocations to enable deterministic evaluation of agent behavior. Each output may include a tool_calls array with call name and args in the order executed:

{
  "output": "...",
  "tool_calls": [
    {"name": "search", "args": {"query": "foo"}},
    {"name": "lookup", "args": {"id": 1}}
  ]
}

The tool_usage evaluator compares these logs against expected sequences via the expected_tool_calls config field.

GitHub Actions Integration

Option 1: Use the Composite Action

Add this to your .github/workflows/ directory:

name: EvalGate
on: [pull_request]

jobs:
  evalgate:
    runs-on: ubuntu-latest
    outputs:
      total_score: ${{ steps.evalgate.outputs.total_score }}
      passed: ${{ steps.evalgate.outputs.passed }}
    permissions:
      contents: read
      checks: write
    steps:
      - uses: actions/checkout@v4
        with: { fetch-depth: 0 }

      # Generate your model outputs
      - name: Generate outputs
        run: python scripts/predict.py --in eval/fixtures --out .evalgate/outputs

      # Run EvalGate
      - uses: aotp-ventures/evalgate@main
        id: evalgate
        with:
          config: .github/evalgate.yml
          check_run: true
  build:
    needs: evalgate
    if: ${{ needs.evalgate.outputs.passed == 'true' }}
    runs-on: ubuntu-latest
    steps:
      - run: echo "EvalGate score ${{ needs.evalgate.outputs.total_score }}"
          

Note: The workflow requires checks: write permission to publish the check run.

Option 2: Direct Integration

Or integrate directly in your existing workflow:

- name: Install uv
  run: |
    curl -LsSf https://astral.sh/uv/install.sh | sh
    echo "$HOME/.local/bin" >> $GITHUB_PATH

- name: Run EvalGate
  id: evalgate
  run: |
    uvx --from evalgate evalgate run --config .github/evalgate.yml
    total_score=$(jq -r '.overall' .evalgate/results.json)
    passed=$(jq -r '.gate.passed' .evalgate/results.json)
    echo "total_score=$total_score" >> "$GITHUB_OUTPUT"
    echo "passed=$passed" >> "$GITHUB_OUTPUT"

- name: EvalGate Summary
  if: always()
  run: uvx --from evalgate evalgate report --summary --artifact ./.evalgate/results.json

- name: Continue if passed
  if: ${{ steps.evalgate.outputs.passed == 'true' }}
  run: echo "EvalGate score ${{ steps.evalgate.outputs.total_score }}"

# Optional: Upload detailed results for debugging
- name: Upload EvalGate Results
  if: always()
  uses: actions/upload-artifact@v4
  with:
    name: evalgate-results
    path: .evalgate/results.json
    retention-days: 30

## Conversation Flow Evaluator

Validate multi-turn conversations by checking the final message and turn count.

```yaml
evaluators:
  - name: convo_flow
    type: conversation
    expected_final_field: content
    max_turns: 5
    weight: 0.2

Each output must provide a messages array. The evaluator compares the last message's content against the fixture's expected.content and fails if the conversation exceeds max_turns.

Writing a custom evaluator

EvalGate supports custom evaluators through the plugin registry introduced in Issue 1. This lets you package and share evaluation logic as reusable plugins.

from evalgate.plugins import registry
from evalgate.evaluators import BaseEvaluator

@registry.evaluator("my_custom")
class MyCustomEvaluator(BaseEvaluator):
    def evaluate(self, outputs, fixtures, **kwargs):
        score = 1.0  # your scoring logic here
        violations: list[str] = []
        return score, violations

After registering, reference the evaluator in your configuration:

evaluators:
  - name: my_custom
    type: my_custom
    weight: 0.5

Refreshing your baseline

EvalGate compares pull requests against a baseline stored on your main branch. When your model's expected outputs change, refresh the baseline so future PRs compare against the new results:

# Generate fresh outputs and update baseline
python scripts/predict.py --in eval/fixtures --out .evalgate/outputs
uvx --from evalgate evalgate run --config .github/evalgate.yml
git add .evalgate/results.json
git commit -m "Refresh eval baseline"

Merge the commit into main, and subsequent runs will use the updated baseline for regression checks.