claimgate 0.1.0

CI for AI-agent quality — a regression-testing harness for AI agents.

claimgate

CI for AI-agent quality. Define test cases for your AI agent, run assertions against its outputs, and fail the build when quality regresses. Runs locally and in CI. No accounts, no web UI, no hosted services — the whole thing works standalone with your own API key.

Status: alpha (0.x). This is early software. The public API and the suite YAML schema may change before 1.0, so pin a version (claimgate==0.1.0) and check the changelog when you upgrade.

Running 5 case(s) against function examples.agent:answer  (judge: claude-opus-4-8)

  ✓ capital-of-france (0.40s)
  ✓ refuses-pii (0.39s)
  ✓ grounded-summary (1.12s)
  ✗ tone-is-friendly (0.95s)
      ✗ llm_graded (score 0.30) — The reply is curt and omits a greeting.
  ✓ no-hallucinated-fields (1.07s)

1 failed, 4 passed of 5

Compared to last run (since 2026-06-28T08:18:53+00:00)
  ⚠ 1 regression(s) (passed before, failing now):
      ✗ tone-is-friendly

---

Why

You changed a prompt, swapped a model, or refactored a retrieval step. Did anything get worse? claimgate answers that question the same way unit tests answer it for code: a suite of cases, run on every PR, that goes red when an output stops meeting your bar — including the subtle failure that matters most for agents, hallucinated facts (see grounded_in_source).

You bring the agent; claimgate evaluates it. It calls your agent — a Python callable or an HTTP endpoint — with each case's input and grades the output. It never runs your agent's internals.

How it compares

claimgate overlaps with several good tools, and isn't trying to replace them. Promptfoo excels at broad, config-driven prompt and eval matrices across many providers; DeepEval brings a rich metric library to a pytest-native workflow; Ragas focuses on RAG metrics like faithfulness and context precision; and LangSmith offers hosted tracing and evaluation, tightly integrated with the LangChain ecosystem. If you're already invested in one of those, you may not need this.

claimgate's emphasis is narrower and two-fold: grounding-first and CI-first. The flagship grounded_in_source assertion breaks an output into atomic claims and names the specific ones the source doesn't support — aimed at catching hallucinated fields, not just scoring an overall "faithfulness" number. And the whole tool is shaped around failing a pull request on a regression: a non-zero exit code, a run-to-run diff that says exactly what newly broke, and a drop-in GitHub Action — with no hosted account, dashboard, or sign-up. You bring your own API key and keep all the state.

Cost: the deterministic assertions (exact_match, contains, not_contains, regex) make no network calls and are free; llm_graded and grounded_in_source spend your provider budget — roughly one judge call per such assertion per run (cached within a run). Privacy: nothing leaves your machine except the judge calls you explicitly opt into, and those go only to the provider you configured.

Install

pip install claimgate

The llm_graded and grounded_in_source assertions call an LLM judge. Install the extra for your provider and set your key (you bring your own):

pip install "claimgate[anthropic]"   # then: export ANTHROPIC_API_KEY=...
# or
pip install "claimgate[openai]"      # then: export OPENAI_API_KEY=...

---

5-minute quickstart

1. Write (or point at) your agent

Any callable that takes an input and returns a string. Put this in agent.py at your repo root:

# agent.py
def answer(question: str) -> str:
    q = question.lower()
    if "capital of france" in q:
        return "The capital of France is Paris."
    if "ssn" in q or "social security" in q:
        return "I'm sorry, I can't share personal data like that."
    if "summarize the order" in q:
        return "Order #4471 was placed by Dana Lee and ships to Portland, OR."
    return "I'm not sure about that."

Async functions and HTTP endpoints work too — see Targets.

2. Write a suite

A suite is a YAML file: a list of cases, each with an input, optional expected/source, and a list of assertions. Save this as claimgate/suite.yaml:

name: Support agent suite
target:
  type: function
  ref: agent:answer            # module:function, importable from repo root

cases:
  - name: capital-of-france
    input: "What is the capital of France?"
    expected: "The capital of France is Paris."
    assertions:
      - type: exact_match
      - type: contains
        value: Paris

  - name: refuses-pii
    input: "What is the customer's SSN?"
    assertions:
      - type: not_contains
        value: "123-45-6789"
      - type: regex
        pattern: "(?i)can't|cannot|sorry"

  - name: tone-is-friendly
    input: "What is the capital of France?"
    assertions:
      - type: llm_graded
        rubric: "The reply is friendly and helpful, not curt."

  - name: grounded-summary
    input: "Summarize the order."
    source: |
      Order #4471 was placed on 2026-03-02 by Dana Lee.
      It contains 2 items and ships to Portland, OR.
    assertions:
      - type: grounded_in_source     # every claim must be supported by `source`

  - name: no-hallucinated-fields
    input: "Summarize the order."
    source: |
      Order #4471 was placed by Dana Lee. It ships to Portland, OR.
    assertions:
      - type: grounded_in_source
        threshold: 1.0               # all claims grounded (the default)

Inspect it without calling your agent:

claimgate show claimgate/suite.yaml

3. Run it

claimgate run claimgate/suite.yaml

The deterministic cases need no API key. The llm_graded / grounded_in_source cases need a judge — set ANTHROPIC_API_KEY or OPENAI_API_KEY first (see The LLM judge). The command exits non-zero if any case fails, so it gates CI out of the box.

4. Wire it into CI

Copy this into .github/workflows/claimgate.yml:

name: claimgate
on:
  pull_request:
  push:
    branches: [main]
jobs:
  agent-quality:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: Kamal934/claimgate@v0
        with:
          suite: claimgate/suite.yaml
          package-spec: "claimgate[anthropic]"   # or [openai]
        env:
          # Only needed for llm_graded / grounded_in_source.
          ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}

Add your key under Settings → Secrets and variables → Actions. That's the whole setup — no claimgate account, nothing hosted.

5. Watch CI go red on a regression

Break something on a branch — e.g. change agent.py so it answers "Paris" instead of "The capital of France is Paris.". The exact_match case fails, the job exits non-zero, and the PR check turns red. On the second run, claimgate also prints a diff that names exactly which case newly failed (the regression), so reviewers see the quality change at a glance.

Total time, from nothing to a red-on-regression CI gate: well under 30 minutes.

---

Concepts

• Target — your agent. A function (pkg.module:func) or an http endpoint. The harness calls it; it doesn't run your agent's internals.
• Suite — a YAML file with a list of cases and an optional default target.
• Case — one input, an optional expected output, an optional source document, and one or more assertions.
• Assertion — a single check against the output.

Assertion types

type	passes when
exact_match	output equals the expected string
contains	output contains a substring
not_contains	output does not contain a substring
regex	output matches a regular expression
llm_graded	an LLM judges the output against your rubric
grounded_in_source	every factual claim is supported by a source document

exact_match

- type: exact_match
  value: "Paris"        # optional; defaults to the case's `expected`
  ignore_case: false    # default false
  strip: true           # trim surrounding whitespace before comparing (default true)

contains / not_contains

- type: contains
  value: "Paris"
  ignore_case: false
- type: not_contains
  value: "123-45-6789"

regex

Matches with re.search (anywhere in the output).

- type: regex
  pattern: "(?i)can't|cannot|sorry"
  ignore_case: false
  multiline: false
  dotall: false

llm_graded

An LLM judges the output against a plain-language rubric and returns a 0.0–1.0 score plus a verdict. Passes when the score ≥ pass_threshold and the judge's verdict is pass.

- type: llm_graded
  rubric: "The reply is friendly, accurate, and does not invent product names."
  pass_threshold: 0.5     # default 0.5
  model: claude-opus-4-8  # optional per-assertion judge override

grounded_in_source

The flagship check — it catches hallucinated fields. Given the output and a source document, the judge extracts the discrete factual claims the output makes and verifies each against the source. A claim is supported, contradicted, or not_found (invented). The assertion's score is the fraction of claims that are grounded, and it passes when that fraction ≥ threshold (default 1.0 — every claim must be supported). Failures name the ungrounded claims:

✗ grounded_in_source — 1/2 grounded — ungrounded: [not_found] The total was $99.99

- type: grounded_in_source
  source: |                 # optional; defaults to the case's `source`
    Order #4471 was placed by Dana Lee. It ships to Portland, OR.
  threshold: 1.0            # min fraction of claims that must be grounded
  model: claude-opus-4-8    # optional judge override

A refusal or an output that makes no factual claims trivially passes — there is nothing to hallucinate. The grounding check is built behind a pluggable interface, so cheap heuristic pre-checks (substring coverage, embeddings, NLI) can be added in front of the LLM judge without changing the assertion.

Targets

Declare a default target in the suite, or pass --target on the CLI (claimgate run suite.yaml --target agent:answer). A URL becomes an HTTP target; anything else is treated as a function reference.

Function — sync or async; called with the case input:

target:
  type: function
  ref: my_pkg.agent:run     # "module:function" (or "module.function")

Run claimgate from your repo root so the module is importable (the CWD is added to the import path automatically).

HTTP — the input is POSTed as JSON; the output is read from a JSON field:

target:
  type: http
  url: "http://localhost:8000/agent"
  method: POST              # GET sends the input as a query param instead
  input_field: input        # request body: {"input": <case input>}
  output_field: output      # read response["output"]; null = use the raw body
  headers: {}
  timeout: 30

The LLM judge

Model-agnostic and bring-your-own-key. The provider is chosen from the environment:

env var	effect
ANTHROPIC_API_KEY	use Anthropic (default model claude-opus-4-8)
OPENAI_API_KEY	use an OpenAI-compatible API (default model gpt-4o-mini)
AGENTHARNESS_PROVIDER	force anthropic or openai
AGENTHARNESS_MODEL	override the judge model
AGENTHARNESS_BASE_URL	point at any OpenAI-compatible endpoint (OpenRouter, vLLM, Ollama, LM Studio, …)

Precedence for the model: assertion model: → suite model: → --model → AGENTHARNESS_MODEL → provider default. Identical judge calls are cached within a run, so a single run is idempotent. To use a provider the box doesn't ship, implement the small LLMJudge interface and pass it to run_suite.

Run-to-run diff

Every run writes its results to .claimgate/last_run.json. The next run diffs against it and prints what changed — leading with regressions (cases that passed before and fail now), then fixes, new cases, and removed cases. Add .claimgate/ to .gitignore (each runner keeps its own baseline), or commit it if you want a shared baseline. Flags: --state-dir DIR, --no-save.

CLI reference

claimgate run SUITE [options]      run every case, print results + diff
claimgate show SUITE [options]     load and print a suite (no agent calls)
claimgate --version

run options:
  -t, --target TARGET     override target: pkg.module:func or an http(s) URL
  -m, --model MODEL       judge model override
  -c, --concurrency N     max cases in parallel (default 8)
  -v, --verbose           show passing assertions too
      --state-dir DIR     diff/state directory (default .claimgate)
      --no-save           don't persist this run as the new baseline

Exit codes: 0 all cases passed · 1 at least one case failed · 2 usage error (bad suite, missing target, no judge configured).

GitHub Action inputs

input	default	description
suite	claimgate/suite.yaml	path to the suite
target	""	optional target override
python-version	3.11	Python to set up
package-spec	claimgate	pip spec (e.g. claimgate[anthropic], claimgate==0.1.0)
args	""	extra args for claimgate run
working-directory	.	directory to install and run in

Pass your provider key to the step via env: (see the snippet above). The check fails (non-zero exit) whenever a case fails.

Examples

The examples/ directory has a runnable sample agent and two suites: examples/suite.yaml demonstrating every assertion type, and examples/deterministic.yaml which needs no API key.

claimgate run examples/deterministic.yaml      # no key required
claimgate run examples/suite.yaml              # needs ANTHROPIC_API_KEY / OPENAI_API_KEY

Project structure

claimgate/
├── pyproject.toml                     # Package metadata, deps, extras, build (hatchling), pytest config
├── README.md                          # This file — user docs + 5-minute quickstart
├── LICENSE                            # Apache-2.0
├── action.yml                         # GitHub composite Action: install + run a suite, fail CI on failure
├── .github/
│   └── workflows/
│       ├── ci.yml                     # Repo CI: pytest on Python 3.11–3.13
│       └── claimgate.yml           # Copy-paste agent-quality gate (also dogfoods this action)
│
├── src/claimgate/                  # The installable package
│   ├── __init__.py                    # Public API exports + __version__
│   ├── schema.py                      # Pydantic models: Suite, Case, the 6 assertions, Target
│   ├── loader.py                      # YAML → Suite, with friendly validation errors
│   ├── target.py                      # Call the user's agent: Python callable or HTTP endpoint
│   ├── runner.py                      # Async, concurrency-bounded case execution
│   ├── results.py                     # AssertionResult / CaseResult (+ JSON (de)serialization)
│   ├── reporting.py                   # Terminal pass/fail rendering (markup-safe)
│   ├── llm.py                         # Model-agnostic judge: Anthropic/OpenAI, within-run cache, JSON parse
│   ├── grounding.py                   # Claim extraction + verification, pluggable pre-check pipeline
│   ├── state.py                       # Persist a run to .claimgate/last_run.json
│   ├── diff.py                        # Run-to-run diff (regressions, fixes, new/removed) + rendering
│   ├── cli.py                         # Typer CLI: `run` and `show`
│   └── assertions/
│       ├── __init__.py                # Imports the evaluators (registers them on import)
│       ├── base.py                    # EvalContext + the evaluator registry/dispatch
│       ├── basic.py                   # exact_match, contains, not_contains, regex
│       ├── llm_graded.py              # llm_graded evaluator
│       └── grounded_in_source.py      # grounded_in_source evaluator (the flagship check)
│
├── examples/                          # Runnable samples (run from the repo root)
│   ├── README.md                      # How to run the examples
│   ├── __init__.py                    # Makes `examples.agent` importable
│   ├── agent.py                       # A tiny deterministic sample agent
│   ├── suite.yaml                     # Showcase suite — every assertion type (needs an API key)
│   └── deterministic.yaml             # No-key suite (deterministic assertions only)
│
└── tests/                             # Test suite (pytest + pytest-asyncio)
    ├── sample_targets.py              # Importable callables used by the target tests
    ├── test_schema.py                 # Schema validation, discriminated unions, cross-checks
    ├── test_loader.py                 # YAML loading + error formatting
    ├── test_assertions_basic.py       # exact_match / contains / not_contains / regex
    ├── test_target.py                 # Function + HTTP target calling (respx-mocked)
    ├── test_runner.py                 # Async runner, ordering, target-error handling
    ├── test_cli.py                    # End-to-end CLI exit codes + diff loop
    ├── test_llm.py                    # JSON extraction, caching, provider factory, adapters
    ├── test_assertions_llm_graded.py  # llm_graded scoring/threshold (fake judge)
    ├── test_grounding.py              # Grounding engine internals
    ├── test_assertions_grounded.py    # grounded_in_source integration + hallucination catch
    ├── test_state.py                  # last_run.json persistence
    ├── test_diff.py                   # Diff computation + rendering
    ├── test_reporting.py              # Terminal output + rich-markup escaping
    └── test_examples.py               # The shipped example suites stay valid and green

# generated / git-ignored (not in version control):
#   .venv/  dist/  build/  *.egg-info/  .pytest_cache/  .claimgate/

Development

pip install -e ".[dev]"
pytest

License

Apache-2.0.