agent-diff-guard

Catches risky changes to an AI agent's config — system prompt, tool permissions, model choice — before they merge. A narrow, single-purpose CI check, not a full observability platform: point it at a repo and it scans every changed file against your base branch automatically.

Get a Pro license → — full per-file detail, model cost-tier detection, SARIF output for GitHub code scanning.

What it catches

• Safety instructions removed — a phrase like "must never" / "do not" / "refuse" present before a change and missing after it.
• Dangerous tool permissions added — shell_exec, sudo, rm -rf, exec(, eval(, admin_override appearing for the first time.
• Model swapped to a pricier tier (Pro) — e.g. haiku → opus, gpt-4o-mini → gpt-4o.

Install

pip install agent-diff-guard

Usage

check — scans changed files against a git ref (recommended)

agent-diff-guard check                          # all files changed vs. HEAD, in the current repo
agent-diff-guard check --against origin/main     # all files changed vs. another ref
agent-diff-guard check path/to/agent_config.yaml # restrict to one file
agent-diff-guard check --repo /path/to/repo      # run against a repo other than cwd
agent-diff-guard check --license-key "$LICENSE"  # unlock Pro (or set AGENT_DIFF_GUARD_LICENSE)

Exit code is 1 if any file has a HIGH severity finding, 0 otherwise — safe to use as a CI gate. The exit code always reflects every scanned file, on the free tier too.

Free tier shows full detail (path + findings) for the first 3 files with findings; beyond that it collapses into a one-line summary — the CI gate still fails correctly either way, Pro just gives the full per-file breakdown.

agent-diff-guard check --format sarif --license-key "$LICENSE"   # Pro: SARIF output

A Pro license key is verified live against Gumroad on every run. If it's invalid or unreachable, the tool falls back to the free tier rather than erroring out, so a network hiccup never breaks your CI.

diff — compare two file versions directly

For comparing arbitrary snapshots outside a git checkout:

agent-diff-guard diff path/to/before.txt path/to/after.txt
agent-diff-guard diff path/to/before.txt path/to/after.txt --license-key "$LICENSE"

GitHub Action

- uses: jwa-wa/agent-diff-guard@v1
  with:
    license-key: ${{ secrets.AGENT_DIFF_GUARD_LICENSE }}   # optional, omit for free tier

That's the whole setup for a pull_request workflow — it reads the PR's base branch automatically and works against a standard (shallow) checkout, no extra config needed.

For a push-triggered workflow (no PR context), it compares against the previous commit instead; your checkout step needs fetch-depth: 2 (or 0) for that:

- uses: actions/checkout@v4
  with:
    fetch-depth: 2
- uses: jwa-wa/agent-diff-guard@v1

Store your license key as a repo/org secret — the Action passes it through as an environment variable, never a command-line argument, so it doesn't end up in job logs.

Pricing

Free — the CI-gate result (pass/fail) always covers every scanned file. Full per-file detail for the first 3 files with findings each run.

Pro — get a license:

• Full per-file detail for every file, not just the first 3.
• Model cost-tier change detection.
• SARIF output — findings show up natively in GitHub's Security tab.

License

MIT