GuardMeter — AI Safety Guard Evaluation Framework

GuardMeter compares two content-safety guards — a baseline and a candidate — on a labeled dataset and produces per-slice metrics, an HTML report, an interactive dashboard, and a pass/fail CI gate. It's for developers and ML engineers who ship a safety classifier and need to catch regressions — per category, language, and attack type — before they merge.

30-second demo

On a fresh pip install guardmeter, these commands run verbatim:

guardmeter init
guardmeter compare --baseline regex-baseline --candidate regex-enhanced --dataset dataset/sample.csv
guardmeter gate --config gate.json --run latest
guardmeter dashboard --open

init writes gate.json and dataset/sample.csv into the current directory. compare evaluates both guards and stores the run. gate checks the latest run against gate.json and exits non-zero on failure. dashboard builds report/dashboard.html (--open launches your browser; omit it or pass --no-open in CI).

How it works

Baseline vs candidate. You give GuardMeter two guards. The baseline is your current behavior; the candidate is the change you're evaluating. Every metric is reported for both so you can see whether the candidate actually improved things.

Strict vs lenient policy. Each dataset row is labeled benign, borderline, or unsafe. Under the strict policy a borderline row counts as something the guard should flag (positive); under the lenient policy borderline counts as benign (negative). Both policies are always computed; the dashboard has a toggle, and the gate/McNemar test use strict by default.

Slices. Aggregate numbers hide regressions. GuardMeter computes recall, FPR, precision, F1 and latency for every (category × language) slice, and separately for every attack_type slice, so a drop confined to (say) Farsi violence or leetspeak-obfuscated prompts is visible.

Significance and confidence. A McNemar test on the paired predictions tells you whether the baseline↔candidate difference is real or noise. Recall and FPR come with Wilson score confidence intervals so small slices aren't over-interpreted.

CI gate

gate.json is a machine-readable safety policy you check into version control. guardmeter gate loads a stored run and fails the build if any threshold is breached.

{
  "mode": "strict",
  "global_thresholds": {
    "min_recall": 0.55,
    "min_f1": 0.80,
    "max_fpr": 0.01,
    "max_latency_p99_ms": 20
  },
  "slices": {
    "self_harm/en": { "min_recall": 0.44, "min_f1": 0.60 },
    "crime/en":     { "min_recall": 0.44, "min_f1": 0.60 },
    "malware/en":   { "min_recall": 0.44 },
    "pii/en":       { "min_f1": 0.65 }
  }
}

Fields:

• mode — strict or lenient; selects which policy's metrics the gate checks.
• global_thresholds — applied to the overall candidate metrics and, by default, to every (category × language) slice:
  • min_recall — minimum recall (skipped for slices with no positive examples).
  • min_f1 — minimum F1 (default 0.80; set 0.0 to disable).
  • max_fpr — maximum false-positive rate (skipped for slices with no negatives).
  • max_latency_p99_ms — maximum p99 latency in milliseconds.
• slices — per-slice overrides. Keys are fnmatch globs. A "category/language" key (e.g. "self_harm/en", "*/fa") targets the category×language family; an "attack:<glob>" key (e.g. "attack:leetspeak") targets the attack-type family. Only the fields you set are overridden; the rest fall back to global_thresholds. Attack-type slices are opt-in — they're gated only where an attack: key matches.
• comparison (optional) — regression limits versus the previous stored run: max_recall_regression, max_fpr_increase.
• on_failure — block (fail the gate) or warn (report but pass).

The per-slice overrides in the shipped gate.json reflect the known limits of the built-in regex demo guard; tighten or remove them for your own guard.

GitHub Actions:

- name: Install guardmeter
  run: pip install guardmeter
- name: Evaluate
  run: guardmeter compare --baseline regex-baseline --candidate ${{ env.CANDIDATE_GUARD }} --dataset dataset/sample.csv
- name: Report
  run: guardmeter report --run latest
- name: Safety gate
  run: guardmeter gate --config gate.json --run latest   # exits 1 on regression
- name: Upload report
  uses: actions/upload-artifact@v4
  with:
    name: safety-report
    path: report/

Use as a GitHub Action

The composite action runs compare → report → dashboard → gate, writes a Markdown table to the job summary, uploads the HTML report as an artifact, and fails the job when the gate fails. Pin it to a release tag:

- uses: samvardani/guardmeter@v0.4.0
  with:
    candidate: regex-enhanced
    dataset: dataset/sample.csv

Inputs: baseline (default regex-baseline), candidate (required), dataset (required), gate (default gate.json), python-version (default 3.12), version (guardmeter version to install; defaults to the pinned release). Outputs: passed, run_id, report_path.

Built-in guards

Write your own guard

from guardmeter.core.guard import Guard, GuardResult
from guardmeter.core.registry import register

class MyGuard(Guard):
    name = "my-guard"
    version = "1.0.0"

    def predict(self, text: str, **meta) -> GuardResult:
        is_unsafe = "bomb" in text.lower()
        return GuardResult(prediction="flag" if is_unsafe else "pass",
                           score=0.9 if is_unsafe else 0.1, latency_ms=5)

register("my-guard", MyGuard)  # now usable as --candidate my-guard

Datasets

• dataset/sample.csv — the smoke-test set used throughout this README and by guardmeter init. Small, balanced across categories and languages; good enough to exercise the pipeline and calibrate a demo gate.
• dataset/prompt_injection_seed.csv — a 40-row seed set (English + Farsi) of prompt-injection attempts (direct overrides, poisoned tool/document output, multi-turn setups, and base64/ROT13-encoded instructions) plus benign look-alikes that merely mention instructions, prompts, or tools. It's aimed at agent-facing guards and is not part of the default gate: the built-in regex guards score poorly on it (they aren't designed for injection detection), which is the point — use it to benchmark a real LLM or injection-aware guard.

Dashboard & report

guardmeter report --run latest writes an HTML report for a single run (baseline vs candidate cards with Wilson CIs, category×language and attack-type slice tables, a real candidate threshold-sweep chart, and per-sample latency charts). It also mentions an informational regulatory mapping — see the note under Experimental.

guardmeter dashboard builds an interactive multi-run dashboard (report/dashboard.html), also auto-rebuilt on every report. Four tabs:

• Overview — run history table with F1, recall, FPR, McNemar p-value and gate badges. Click a row to drill in.
• Run Detail — baseline vs candidate metric cards, category×language and attack-type slice tables, and a sample-results table (first 200 rows). Strict/Lenient toggle.
• Trends — recall, F1, FPR and McNemar p-value over all runs (p-value on a log scale with a p=0.05 reference line).
• Compare — pick any two runs and see a per-metric delta table with improvement/regression arrows.

Experimental

These features work but require API keys or extra dependencies and have limited automated test coverage. Treat them as advisory:

• LLM-as-judge (guardmeter/judge/) — uses Claude or an OpenAI model as a second opinion on predictions. Available through the Python API only (no CLI subcommand); needs a provider API key.
• openai guard — calls the OpenAI Moderation API; needs guardmeter[llm] and OPENAI_API_KEY.
• anthropic guard — asks a Claude model (default claude-sonnet-4-5) for a strict JSON safety verdict over GuardMeter's category vocabulary; needs guardmeter[llm] and ANTHROPIC_API_KEY. Malformed or failed responses fall back to a safe pass.
• llamaguard guard — runs Llama Guard 3 via a local transformers pipeline or an HTTP endpoint; needs guardmeter[hf] or a hosted endpoint and key.
• Regulatory mapping (informational). The HTML report includes a table mapping a run's metrics to regulatory themes (e.g. EU AI Act articles, NIST AI RMF). It is an informational aid for your own documentation, not a compliance certification or legal assessment.

Development Setup

python3.13 -m venv .venv
source .venv/bin/activate      # Windows: .venv\Scripts\activate
pip install -e ".[dev]"
ruff check guardmeter tests
mypy guardmeter
pytest tests/guardmeter/ -q

Note: macOS users with Homebrew Python must use a virtual environment (Homebrew enforces PEP 668).

Not affiliated with

This project is unrelated to the JRC "GuardBench" toxicity-benchmark library at github.com/AmenRa/guardbench. Same name, different project.

Formerly published as sea-guard (versions 0.1–0.2, import name guardbench). Renamed in 0.3.0 to avoid confusion with the unrelated JRC GuardBench benchmark.

Contributing

See CONTRIBUTING.md. Contributions welcome — new guard adapters, dataset/language coverage, and report improvements especially.

License

MIT — see LICENSE.

Branding

Logo assets are in the branding/ directory.

• guardmeter-logo.svg — shield mark (favicon, PyPI, GitHub avatar)
• guardmeter-wordmark.svg — full lockup with tagline
• guardmeter-social-card.svg — 1280×640 OG image for GitHub social preview

Built by SeaTechOne LLC · Seattle, WA