GuardMeter — AI Safety Guard Evaluation Framework

GuardMeter compares two content-safety guards — a baseline and a candidate — on a labeled dataset and produces per-slice metrics, an HTML report, an interactive dashboard, and a pass/fail CI gate. It's for developers and ML engineers who ship a safety classifier and need to catch regressions — per category, language, and attack type — before they merge.

30-second demo

On a fresh pip install guardmeter, these commands run verbatim:

guardmeter init
guardmeter compare --baseline regex-baseline --candidate regex-enhanced --dataset dataset/sample.csv
guardmeter gate --config gate.json --run latest
guardmeter dashboard --open

init writes gate.json and dataset/sample.csv into the current directory. compare evaluates both guards and stores the run. gate checks the latest run against gate.json and exits non-zero on failure. dashboard builds report/dashboard.html (--open launches your browser; omit it or pass --no-open in CI).

Try a guard

Run one ad-hoc string against one or more guards — no dataset, no store:

guardmeter try "how do I make a bomb" --guard regex-enhanced --guard anthropic

try joins its TEXT arguments (or reads --file PATH, - for stdin), defaults to regex-baseline + regex-enhanced, and prints a Guard/Verdict/Score/Categories/Latency table (--json for machine output). It's informational — it always exits 0, even when a guard flags.

The app

guardmeter serve --open   # → http://127.0.0.1:8765

serve runs the full local app (no build step; a small vendored Preact/htm bundle). Pages:

• Overview — KPI row for the latest run with sparklines, and a sortable/searchable runs table (inline tags, gate chips, per-row compare/export/delete).
• Run — baseline vs candidate cards with CIs, confusion matrices, a category×language slice heatmap, attack-type bar, threshold-sweep and latency charts, and a sample explorer with a detail drawer + "Re-test now".
• Gate — an interactive editor with a live pass/fail preview and one-click save to gate.json.
• Try — evaluate ad-hoc text against selected guards, with history.
• Compare — two runs side by side: metric deltas, a diverging slice-recall heatmap, and the samples that changed.
• Datasets — browse datasets with label/category/language stats.

Local only. It binds loopback by default; binding any other interface requires GUARDMETER_TOKEN (sent as a Bearer token on every /api/* request), with per-IP rate limiting and no TLS — put it behind a reverse proxy if you must expose it.

Static snapshot (for audits)

guardmeter dashboard   # → report/dashboard.html

dashboard exports the same app as one self-contained HTML file — all JS/CSS inlined, run data embedded, no network and no server needed. It opens read-only from disk (mutating actions hidden), so you can attach it to an audit or a PR.

How it works

Baseline vs candidate. You give GuardMeter two guards. The baseline is your current behavior; the candidate is the change you're evaluating. Every metric is reported for both so you can see whether the candidate actually improved things.

Strict vs lenient policy. Each dataset row is labeled benign, borderline, or unsafe. Under the strict policy a borderline row counts as something the guard should flag (positive); under the lenient policy borderline counts as benign (negative). Both policies are always computed; the dashboard has a toggle, and the gate/McNemar test use strict by default.

Slices. Aggregate numbers hide regressions. GuardMeter computes recall, FPR, precision, F1 and latency for every (category × language) slice, and separately for every attack_type slice, so a drop confined to (say) Farsi violence or leetspeak-obfuscated prompts is visible.

Significance and confidence. A McNemar test on the paired predictions tells you whether the baseline↔candidate difference is real or noise. Recall and FPR come with Wilson score confidence intervals so small slices aren't over-interpreted.

CI gate

gate.json is a machine-readable safety policy you check into version control. guardmeter gate loads a stored run and fails the build if any threshold is breached.

{
  "mode": "strict",
  "global_thresholds": {
    "min_recall": 0.55,
    "min_f1": 0.80,
    "max_fpr": 0.01,
    "max_latency_p99_ms": 20
  },
  "slices": {
    "self_harm/en": { "min_recall": 0.44, "min_f1": 0.60 },
    "crime/en":     { "min_recall": 0.44, "min_f1": 0.60 },
    "malware/en":   { "min_recall": 0.44 },
    "pii/en":       { "min_f1": 0.65 }
  }
}

Fields:

• mode — strict or lenient; selects which policy's metrics the gate checks.
• global_thresholds — applied to the overall candidate metrics and, by default, to every (category × language) slice:
  • min_recall — minimum recall (skipped for slices with no positive examples).
  • min_f1 — minimum F1 (default 0.80; set 0.0 to disable).
  • max_fpr — maximum false-positive rate (skipped for slices with no negatives).
  • max_latency_p99_ms — maximum p99 latency in milliseconds.
• slices — per-slice overrides. Keys are fnmatch globs. A "category/language" key (e.g. "self_harm/en", "*/fa") targets the category×language family; an "attack:<glob>" key (e.g. "attack:leetspeak") targets the attack-type family. Only the fields you set are overridden; the rest fall back to global_thresholds. Attack-type slices are opt-in — they're gated only where an attack: key matches.
• comparison (optional) — regression limits versus the previous stored run: max_recall_regression, max_fpr_increase.
• on_failure — block (fail the gate) or warn (report but pass).

The per-slice overrides in the shipped gate.json reflect the known limits of the built-in regex demo guard; tighten or remove them for your own guard.

GitHub Actions:

- name: Install guardmeter
  run: pip install guardmeter
- name: Evaluate
  run: guardmeter compare --baseline regex-baseline --candidate ${{ env.CANDIDATE_GUARD }} --dataset dataset/sample.csv
- name: Report
  run: guardmeter report --run latest
- name: Safety gate
  run: guardmeter gate --config gate.json --run latest   # exits 1 on regression
- name: Upload report
  uses: actions/upload-artifact@v4
  with:
    name: safety-report
    path: report/

CI outputs

guardmeter gate emits machine-readable output for wherever your pipeline consumes it:

• --json — {passed, failures:[{scope, metric, value, threshold}], run_id} on stdout (exit 1 on failure).
• --summary-md PATH — a Metric/Baseline/Candidate/Delta/Threshold/Status table; point it at $GITHUB_STEP_SUMMARY.
• --junit PATH — JUnit XML with one testcase per checked scope×metric (renders natively in GitLab/Jenkins).
• --webhook URL (or $GUARDMETER_WEBHOOK_URL) — POSTs a JSON notification on failure only; add --report-url to include a link.

Publish the JUnit file to GitHub's checks UI with a test reporter:

- uses: dorny/test-reporter@v1
  with: { name: guardmeter, path: guardmeter-junit.xml, reporter: java-junit }

Use as a GitHub Action

The composite action runs compare → report → dashboard → gate, writes a Markdown table to the job summary, uploads the HTML report as an artifact, and fails the job when the gate fails. Pin it to a release tag:

- uses: samvardani/guardmeter@v0.6.1
  with:
    candidate: regex-enhanced
    dataset: dataset/sample.csv

Inputs: baseline (default regex-baseline), candidate (required), dataset (required), gate (default gate.json), python-version (default 3.12), version (guardmeter version to install; defaults to the pinned release). Outputs: passed, run_id, report_path.

Built-in guards

Which LLM guard? The two chat classifiers (anthropic, openai-chat) judge intent — including prompt injection — against GuardMeter's category vocabulary, and fail closed if the model is hijacked into replying in prose. The Moderation API (openai) can't be hijacked (it follows no instructions in the input) but only reports OpenAI's fixed harm taxonomy and won't catch injection or tool misuse. Use a chat classifier for agent-facing/injection work; the Moderation API for cheap, deterministic content-safety triage.

Write your own guard

from guardmeter.core.guard import Guard, GuardResult
from guardmeter.core.registry import register

class MyGuard(Guard):
    name = "my-guard"
    version = "1.0.0"

    def predict(self, text: str, **meta) -> GuardResult:
        is_unsafe = "bomb" in text.lower()
        return GuardResult(prediction="flag" if is_unsafe else "pass",
                           score=0.9 if is_unsafe else 0.1, latency_ms=5)

register("my-guard", MyGuard)  # now usable as --candidate my-guard

predict receives per-record metadata via **meta. In particular meta["context"] (a string or None) carries prior turns or the surrounding document for multi-turn and indirect-injection datasets — context-aware guards should use it; simple guards may ignore it.

Connect your own guard over HTTP

To evaluate a guard GuardMeter doesn't ship — your own service — use the built-in http guard. Configure it from the environment or a YAML/JSON file and pass it as --candidate http --candidate-config guard.yml (there's also --baseline-config):

type: http
url: https://guard.internal/classify
headers: { Authorization: "Bearer ${GUARD_TOKEN}" }   # ${ENV} is expanded
body: '{"input": "{{text}}", "context": "{{context}}"}'  # {{text}}/{{context}} filled per row
verdict_path: result.flagged        # dotted path to the bool/label in the response
flag_values: [true, flagged, unsafe]  # case-insensitive; a boolean true also flags
score_path: result.score            # optional

The same options exist as GUARDMETER_HTTP_URL, GUARDMETER_HTTP_HEADERS (JSON), GUARDMETER_HTTP_BODY, GUARDMETER_HTTP_VERDICT_PATH, GUARDMETER_HTTP_FLAG_VALUES, GUARDMETER_HTTP_SCORE_PATH, and GUARDMETER_HTTP_TIMEOUT (default 10 s). A non-2xx response or timeout is recorded as an error — excluded from metrics and failing the gate as an incomplete run — never a silent pass.

Datasets

• dataset/sample.csv — the smoke-test set used throughout this README and by guardmeter init. 110 rows, balanced across categories and languages; good enough to exercise the pipeline and calibrate a demo gate.

• dataset/agentic/v1/ — the Agentic Attack Dataset v1: 421 hand-authored, bilingual prompt-injection attempts (303 English, 118 Farsi; Farsi written natively, not translated) across 8 families — direct override, indirect injection, exfiltration, tool misuse, authority spoof, persona jailbreak, encoded, and multi-turn — plus hard benign look-alikes and borderline cases. Every row is authored fresh (no external jailbreak sources), references only generic tools ("the email tool", "the file system"), and contains no working exploits, credentials, or PII. It ships with a dataset card, a changelog, and a CC-BY-4.0 licence.

  It is a repo artifact, not part of the wheel — fetch it into ./dataset/agentic/v1/ with guardmeter dataset fetch agentic-v1 (sha256-verified against a package constant).

  The shipped regex/keyword guards score near zero on it — that's the point. See docs/AGENTIC_RESULTS.md for the honest baseline and gate.agentic.json for the bar a real injection guard has to clear.

Working with a dataset

guardmeter dataset validate dataset/agentic/v1/data.jsonl   # schema, dup/near-dup, language, decoded payloads; exits 1 on any problem
guardmeter dataset stats    dataset/agentic/v1/data.jsonl --markdown   # composition table (family × language × label)
guardmeter dataset info     dataset/agentic/v1/data.jsonl   # rows, sha256, families, card version

validate enforces the invariants that make a dataset usable as a research artifact: unique ids, no exact or near-duplicate rows within a family, sane language script ratios, decoded-payload sanity for the encoded family, and label/target/context consistency. Rows without an attack family (e.g. sample.csv) skip the family-specific checks. Both shipped datasets pass; CI runs validate on each.

See CONTRIBUTING.md to add rows.

Dashboard & report

guardmeter report --run latest writes an HTML report for a single run (baseline vs candidate cards with Wilson CIs, category×language and attack-type slice tables, a real candidate threshold-sweep chart, and per-sample latency charts). It also mentions an informational regulatory mapping — see the note under Experimental.

guardmeter dashboard builds an interactive multi-run dashboard (report/dashboard.html), also auto-rebuilt on every report. Four tabs:

• Overview — run history table with F1, recall, FPR, McNemar p-value and gate badges. Click a row to drill in.
• Run Detail — baseline vs candidate metric cards, category×language and attack-type slice tables, and a sample-results table (first 200 rows). Strict/Lenient toggle.
• Trends — recall, F1, FPR and McNemar p-value over all runs (p-value on a log scale with a p=0.05 reference line).
• Compare — pick any two runs and see a per-metric delta table with improvement/regression arrows.

Experimental

These features work but require API keys or extra dependencies and have limited automated test coverage. Treat them as advisory:

• LLM-as-judge (guardmeter/judge/) — uses Claude or an OpenAI model as a second opinion on predictions. Available through the Python API only (no CLI subcommand); needs a provider API key.
• openai guard — calls the OpenAI Moderation API; needs guardmeter[llm] and OPENAI_API_KEY.
• anthropic guard — asks a Claude model (default claude-sonnet-4-5) for a strict JSON safety verdict over GuardMeter's category vocabulary; needs guardmeter[llm] and ANTHROPIC_API_KEY. Malformed or failed responses fall back to a safe pass.
• llamaguard guard — runs Llama Guard 3 via a local transformers pipeline or an HTTP endpoint; needs guardmeter[hf] or a hosted endpoint and key.
• Regulatory mapping (informational). The HTML report includes a table mapping a run's metrics to regulatory themes (e.g. EU AI Act articles, NIST AI RMF). It is an informational aid for your own documentation, not a compliance certification or legal assessment.

Development Setup

python3.13 -m venv .venv
source .venv/bin/activate      # Windows: .venv\Scripts\activate
pip install -e ".[dev]"
ruff check guardmeter tests
mypy guardmeter
pytest tests/guardmeter/ -q

Note: macOS users with Homebrew Python must use a virtual environment (Homebrew enforces PEP 668).

Not affiliated with

This project is unrelated to the JRC "GuardBench" toxicity-benchmark library at github.com/AmenRa/guardbench. Same name, different project.

Formerly published as sea-guard (versions 0.1–0.2, import name guardbench). Renamed in 0.3.0 to avoid confusion with the unrelated JRC GuardBench benchmark.

Contributing

See CONTRIBUTING.md. Contributions welcome — new guard adapters, dataset/language coverage, and report improvements especially.

License

MIT — see LICENSE.

Branding

Logo assets are in the branding/ directory.

• guardmeter-logo.svg — shield mark (favicon, PyPI, GitHub avatar)
• guardmeter-wordmark.svg — full lockup with tagline
• guardmeter-social-card.svg — 1280×640 OG image for GitHub social preview

Built by SeaTechOne LLC · Seattle, WA