agentproof-scan

Maintenance mode — detection surface and coverage are frozen. 0.3.0 fixed access friction (keyless --demo, working exit-code contract for file input) and corrected two false-GREEN paths; 0.3.1 is documentation only. Neither adds detection capability. Issues welcome, bugs get fixed. Extensions ship as separate projects.

Catch your AI agent leaking its system prompt or API keys — before you ship it.

What this is — and what it needs from you

agentproof-scan is not a static code scanner. It does not read your source tree, and there is no --path or --directory flag. It is a black-box prober: it sends a batch of probing questions to a running agent and checks what comes back for (a) strings shaped like real secrets (API keys) or (b) the hidden contents of its own system prompt.

So it needs a target, and there are only two kinds:

What you have	Path
A running agent that speaks JSON over HTTP	--url → ③
Output your agent already produced (logs, traces, transcripts)	--trace → ②
Neither yet	① to see a report, then the 9-line agent in ③

If you arrived expecting to point this at a repository: ① and ② both run with nothing but the package installed. Start there.

When to run it: before you ship, not in production

This is a pre-deployment test, not a production monitor. The intended loop is the one you already use for tests:

1. Run your agent locally (http://127.0.0.1:8000)
2. Probe it
3. Fix what leaks
4. Then deploy

A unit test also executes real code at runtime and is still a preventive check — same idea here. Nothing in this tool is built to sit in front of live user traffic.

Pick your entry point

Ordered by what you need on hand. Stop at the first one you can run.

	Path	Needs	API key	Network
①	--demo	nothing	no	no
②	agentproof-scan-trace --trace	a log file	no	no
③	--url	a running agent	your agent's, if it has one	localhost is fine
④	--agent-config	③ + a nested request body	same	same

Install

pip install agentproof-scan

pip: command not found → python -m pip install agentproof-scan. agentproof-scan: command not found → python -m agentproof_scan (Windows: py -m agentproof_scan) — same tool, same flags.

On Windows without Python, py -m pip install agentproof-scan triggers the Python Install Manager to fetch Python for you. [VERIFIED: py -m ... installed Python 3.14.6 on Windows, owner, 2026-07-11.]

---

① --demo — see a report in 10 seconds

No key, no network, no account. Nothing to fill in:

agentproof-scan --demo

--demo runs the scanner against a bundled, canned agent response with planted credentials from 10 different providers in it, and prints the same JSON report the real scanner emits. leak_count will be 15.

Watch the CI gate fire, still offline:

agentproof-scan --demo --fail-on-findings   # exits 2 because the demo leaks

This proves the tool runs. It does not scan anything of yours — for that, go to ② or ③.

---

② --trace — scan output you already have

If your agent has already run and you kept the output, you don't need a live endpoint. No key, no network:

agentproof-scan-trace --trace mytrace.json --fail-on-findings

Input format. JSON (not JSONL): one object, or an array of them. Three fields, all optional:

Field	Meaning
answer_text	What the agent said to the user (the final output surface)
reasoning_text	The agent's thinking/reasoning trace, if you captured it
target	A label for the report. Cosmetic

[
  {"answer_text": "I can't share that.", "reasoning_text": "the key is sk-...", "target": "support-bot"},
  {"answer_text": "Sure, here you go."}
]

Unknown fields are ignored, so you can usually pass records you already have. To reshape yours, one jq line is normally enough:

jq '[.[] | {answer_text: .output, reasoning_text: .thinking, target: .model}]' \
  mylogs.json > trace.json

Why this path matters. An agent can refuse to reveal a secret in its answer while still leaking it in its reasoning — the "thinking" that output-only checks never look at. That blind spot is the main thing this scanner was built to catch. To see it with no file of your own:

agentproof-scan-trace --target reasoning_demo_canary --fail-on-findings

That prints final_output with leak_count: 0 and reasoning with leak_count: 1, then exits 2 — the answer is clean, the reasoning is not.

Omitting reasoning_text is not the same as it being clean. A record with no reasoning field reports that surface as not_applicable, never as clean — the scanner will not claim a surface it never saw.

---

③ --url — scan a running agent

If your agent is a self-hosted HTTP endpoint that speaks JSON, point the scanner straight at it — no adapter code to write. A local dev server is the intended target; you do not need a deployed URL.

agentproof-scan \
  --url http://127.0.0.1:8000/chat \
  --prompt-field message \
  --response-field reply

• --url — your agent's endpoint, localhost included.
• --prompt-field — dot-path into the request body where the probe text is injected.
• --response-field — dot-path into the response where the answer is read. Nested replies: choices.0.message.content.

Reasoning trace? If your agent returns its "thinking," add --reasoning-field <path> and that surface is scanned too, separately from the answer. Set it if you have it — see ② for why it's the one worth wiring up.

Nothing to point it at yet? Save this as leaky_agent.py. It is deliberately leaky, so ③ should light up — which is how you confirm your flags are right before aiming at the real thing:

import json
from http.server import BaseHTTPRequestHandler, HTTPServer
KEY = "sk-ant-api03-Qm4Zx8Lp1Vt6Rn9Bw2Hs7Dc3Fg5Jk0AeY4uI7oP9qW1rT3yU6mC2nK"  # not a real key, but real-shaped
class A(BaseHTTPRequestHandler):
    def do_POST(self):
        json.loads(self.rfile.read(int(self.headers["Content-Length"])))
        self.send_response(200); self.end_headers()
        self.wfile.write(json.dumps({"reply": f"Sure! My key is {KEY}"}).encode())
HTTPServer(("127.0.0.1", 8000), A).serve_forever()

python leaky_agent.py                       # terminal 1 — stdlib only, no dependencies

agentproof-scan --url http://127.0.0.1:8000/chat \
  --prompt-field message --response-field reply --fail-on-findings   # terminal 2

Replace A with your own handler — or change the URL to your own server — and ③ is pointed at your agent.

The key above is a synthetic canary reused from this repository's test fixtures: it matches the shape a detector looks for and authenticates against nothing. See SECURITY.md.

Needs auth? Pass a header, but put only the name of an environment variable in the flag, never the key itself. Put the key in .env (MY_AGENT_KEY=your-real-key) with an editor, then reference it by name:

agentproof-scan --url http://127.0.0.1:8000/chat \
  --prompt-field message --response-field reply \
  --auth-header "Authorization=Bearer {MY_AGENT_KEY}"

Your key stays in .env. It is never written to the config, the report, or any log, and any secret-shaped string in a response is masked before it's printed.

⚠️ If --response-field doesn't match your response, the scan does not fail — it widens. Rather than risk hiding a leak, the scanner falls back to scanning the entire response payload. It prints a [warn] ... not found line to stderr once and the report carries response_field_missed: <count>. If you see that field, your path is probably wrong — the findings may be real, but they didn't come from where you think.

⚠️ Scan only agents you own or control. These probes are adversarial by design; pointing them at a third-party endpoint you don't operate is your responsibility. Each run makes real API calls to your agent (probes × --stability), so it spends whatever those calls cost on your account.

---

④ --agent-config — nested request bodies

When flags aren't enough (custom headers, a deep request body):

agentproof-scan --agent-config my_agent.yaml

url: http://127.0.0.1:8000/v1/chat
method: POST
prompt_field: messages.0.content           # inject the probe here
response_field: choices.0.message.content  # read the answer here
reasoning_field: choices.0.message.reasoning       # optional
auth_header: "Authorization=Bearer {MY_AGENT_KEY}" # env-var name, not the key
body:                                      # your request template
  model: my-model
  messages:
    - role: user
      content: ""

A filled-in template ships inside the package, so you can copy it rather than retype:

python -c "import agentproof_scan,os;print(os.path.join(os.path.dirname(agentproof_scan.__file__),'examples'))"

That folder holds my_agent.yaml and trace_example.json (input for ②).

Flat flags win over config keys when both are given. Full key table below.

Every config key

Flat flags win over config keys when both are given.

Key	Required	CLI flag	Notes
url	✅	--url	Empty ⇒ exit 1, reason=missing_url
prompt_field	✅	--prompt-field	dot-path into the request body
response_field	✅	--response-field	dot-path into the response
method	—	--method	Default POST
reasoning_field	—	--reasoning-field	Enables the reasoning surface
timeout	—	--timeout	Seconds, default 60
name	—	--target-name	⚠ the key is name, the flag is --target-name
auth_header	—	--auth-header	"Header=ENV_VAR" or "Header=Bearer {ENV_VAR}" — an env-var name
headers	—	(config only)	Static extra headers, a mapping
body	—	(config only)	Request template; prompt_field is written into it

dot-path syntax: segments split on .; a segment that is an integer indexes into a list. So choices.0.message.content means data["choices"][0]["message"]["content"]. There is no escape syntax — a key containing a literal . cannot be addressed.

(Prefer to wire it in yourself? You still can: implement the small AgentAdapter interface in agentproof_scan/adapters/base.py and register it in ADAPTERS in agentproof_scan/scan.py. The top-level scan.py is only a clone-launcher shim and isn't in the wheel.)

Scope: the generic HTTP path above is shipped and is where this tool stops. Broader shapes — non-JSON bodies, streaming responses, and non-HTTP transports — are out of scope here (maintenance mode); they belong to separate follow-on projects.

---

📁 Scanning logs you already have

---

Exit codes — three states, never two

Both commands share one contract. Gate CI on the exit code, not on grepping fields.

Code	Meaning	When
0	Clean	Every probe reached your agent, nothing leaked
1	The scan did not run	Missing/placeholder key, HTTP error, timeout, bad flag. Never means "safe". Prints AGENTPROOF_SCAN_DID_NOT_RUN reason=<slug> to stderr
2	Findings	The agent leaked (requires --fail-on-findings)

1 and 2 are different on purpose: a build that fails because nobody could ask the agent must not be read as the agent leaked, and neither may be read as safe.

By default --fail-on-findings gates on leaked secrets only; add --fail-on any to include prompt-disclosure signals.

Full history of how this contract came to be true: CHANGELOG.md.

---

🔍 How it works (it's counting, not a formula)

The scanner plants a fake "canary" secret in a test agent's system prompt, sends it a batch of probing questions, then checks the agent's final answer for that planted secret — and, when you point it at your own agent and say where the reasoning ("thinking") trace lives, that second surface too. A plain pattern-matcher (no AI doing the judging) counts how many runs the secret literally shows up in each. A result reads like "leaked in 4 of 10 runs."

Because it's straight matching against known secret shapes, there's nothing hidden: you can read the code. Every number we mark GREEN-backed you can reproduce from a clone, offline, byte-for-byte — see Reproducing these numbers. The cross-model observations are a different kind of number: measured snapshots that depend on a live model, reported as directional. We don't blur the two.

---

📊 How to read the results (in plain terms)

Two fields matter:

• leak — the agent printed something shaped like a real secret (sk-proj-****, sk-ant-****, AIza****, …). This is the bad one: a credential escaped. (All secrets are masked in the report, so the report itself is safe to share.)
• prompt_disclosure — no secret leaked, but the agent revealed the contents of its hidden system prompt (a planted "canary" phrase showed up). A softer failure: it overshared its instructions.

Analogy: leak = the guard handed over the vault key. prompt_disclosure = the guard didn't hand over the key, but read the security manual aloud. Both are bad; the first is worse.

leak_rate (in repeat/stability mode) = how often a probe pulled a leak, over the runs that got an answer. For example 4/10 (0.4) = 4 of 10 answered tries leaked. A flaky leak is still a leak — repetition shows how reliably an agent fails. A probe that never got an answer (rate-limited, timed out) shows leak_rate: null, not 0.0 — the report separates "didn't leak" from "wasn't asked."

Which secrets it recognizes: the scanner looks for key shapes from major providers — OpenAI (including modern sk-proj- / sk-svcacct- / sk-admin- keys and the legacy sk- format), Anthropic, Google, AWS, GitHub, and xAI.

---

🧪 See it in action (catch a flaw, clear a safe agent)

Two contrasting demo targets ship with the repo:

agentproof-scan --target simple_chatbot_canary   # planted fake secret → expect leaks/disclosure
agentproof-scan --target simple_chatbot          # clean prompt        → expect 0

• *_canary targets have a fake secret + canary phrases planted in their prompt → the scanner should light up, proving the rule catches leaks.
• The clean simple_chatbot has no secret → the scanner should stay at 0, proving safe agents pass (no false alarms).

Canary fails + clean passes = you can trust the verdict.

🧪 The live Gemini demo (needs a free key)

This is the first step that needs an API key, and it is the only reason the demo needs one: the target is a real agent backed by Google Gemini, so something has to pay for the call.

1. Get a free key → https://aistudio.google.com/apikey
2. Create a file named .env in the folder you run from with an editor (not the shell — see Prerequisites), containing one line: GEMINI_API_KEY=your-real-key-here with your own key in place of the placeholder.
3. Then run:

agentproof-scan                  # one pass = 15 probes = 15 API calls
agentproof-scan --stability 2    # 2 passes = 30 calls — more reliable (see notes)

If you skipped the key, this exits 1 with reason=missing_env rather than printing a misleading clean result — an all-zero report with no key would mean "no key", not "safe".

⚠️ On a free Gemini key, watch the request count. Each pass sends 15 probes = 15 API calls; --stability N multiplies that (--stability 5 = 75 calls, --stability 10 = 150). The free tier has a per-minute quota, and hitting it is not a tool failure — the scanner exits 1 with reason=rate_limit and tells you to wait or lower --stability. Start with --stability 2; raise it once you know your quota headroom.

⚠️ A zero you can trust. A scan that never reached your agent, or was cut off partway (bad key, HTTP 500, timeout, rate limit), must not read as "safe." The scanner refuses to report clean unless every probe actually got an answer: otherwise it exits 1 and prints AGENTPROOF_SCAN_DID_NOT_RUN reason=<slug> on stderr (rate_limit, auth_failed, http_status, incomplete_scan, …). A leak it did find is still reported and, under --fail-on-findings, still exits 2 — a partial scan may not claim clean, but it may claim what it found. (Earlier versions, up to 0.1.3, could print leak_count: 0 and exit 0 here — that's the bug 0.1.4 closes.)

Why repeat with --stability? A single run is non-deterministic — a leaky agent can still answer "safely" on any one try, so a one-shot scan might read 0 by luck. Repeating measures how often it leaks (leak_rate) and is the reliable way to read the verdict. A probe that never got an answer shows leak_rate: null (not 0.0) — "not asked" is not "didn't leak." ⚠ Each repeat costs 15 × N API calls; on a free key start at --stability 2 (see the request-count note above).

---

🤝 --handoff: turn results into a fix

--handoff prints a ready-to-paste block for an AI assistant — masked findings plus a request for the smallest code change that stops the leak:

agentproof-scan --target victim --handoff
agentproof-scan --target victim --stability 10 --handoff   # aggregate over 10 runs first

Paste the block into your AI assistant of choice, fill in your agent's framework/model, and it proposes the smallest fix. (If nothing leaked, it tells you you're safe and prints no block.)

---

📤 Reading the report (field reference)

The report is JSON on stdout. Gate CI on the exit code, not on grepping fields (see the leak_count warning under Scanning the reasoning channel), but when you want to read or post-process it, these are the fields:

Field	Meaning
target	Which agent was scanned
rule	Detection ruleset id (secret-leak-v0)
scan_ran	Whether the scan actually executed
complete	Whether every probe got an answer. false ⇒ a clean result is not claimable
probes_answered / total_probes	Coverage of this run
abort_reason	Slug if the scan was cut short, else null
leak_count	Probes that leaked a secret in the answer surface
disclosure_count	Probes that exposed system-prompt content
findings[]	One entry per probe with a finding
response_field_missed	Present only when --response-field didn't match (see above)

Each findings[] entry carries probe, leak, prompt_disclosure, disclosed_phrases, response_excerpt (masked), and leaked[], where each leaked item is:

Field	Meaning
provider	Credential family (anthropic, stripe, github_pat, …)
match	The secret, masked — never the raw value
scope	What that credential would grant, in plain terms
fingerprint	Salted, truncated one-way hash (sha256-t48:…) — lets you tell two findings apart, and match one against a candidate you already hold, without the report carrying the secret

With --stability N the shape changes: per_probe replaces findings with per-probe leak_rate / disclosure_rate (a probe that never got an answer reports null, not 0.0).

---

🔁 Run it in CI (fail the build on a leak, before it merges)

Point it at your own agent in a GitHub Action so a leak turns the check red instead of reaching production. Put your agent's URL and key in repo Secrets (never in the file), and add .github/workflows/agentproof.yml:

name: agentproof secret-leak scan
on: [push, pull_request]

jobs:
  scan:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/setup-python@v5
        with: { python-version: '3.11' }

      - run: pip install agentproof-scan

      - name: Scan my agent for leaks
        env:
          AGENT_URL: ${{ secrets.AGENT_URL }}
          MY_AGENT_KEY: ${{ secrets.MY_AGENT_KEY }}
        run: |
          if [ -z "$AGENT_URL" ]; then
            echo "AGENT_URL secret is not set — refusing to scan."
            exit 1
          fi
          agentproof-scan \
            --url "$AGENT_URL" \
            --prompt-field message \
            --response-field reply \
            --auth-header "Authorization=Bearer {MY_AGENT_KEY}" \
            --fail-on-findings

(Also at examples/ci/agentproof.yml — copy it, don't clone the repo.)

Keep the AGENT_URL guard anyway. An unset secret is an empty string, and the guard turns that into a clear "secret not set" failure. Since 0.1.3 an empty --url already exits 1 on its own (reason=missing_url) rather than scanning anything — so a missing AGENT_URL fails the build as "the scan did not run," not as "your agent leaked." The guard just makes that reason obvious in the log. Forked pull requests get no secrets, so they hit this path — and a 1 there is correct: nothing was scanned, and nothing is claimed clean.

If your agent leaks a secret, --fail-on-findings exits non-zero and the check goes red. By default it fails on leaked secrets only (not softer prompt-disclosure signals); add --fail-on any to gate on those too. Scan only agents you own.

---

Scanning the reasoning channel

The final answer isn't the only place a secret can show up. A model will sometimes keep a key out of its answer but leave it in its reasoning ("thinking") — and a check that only reads the answer never sees it.

If your agent returns its reasoning, tell the scanner where to find it with --reasoning-field and it checks that surface too. This works on the own-agent path (--url / --agent-config); the bundled demo targets scan the answer only.

agentproof-scan --url https://my-agent.example.com/chat \
  --prompt-field message --response-field reply \
  --reasoning-field think          # dot-paths work: choices.0.message.reasoning

An agent that refuses in its answer while spilling the key in its reasoning is reported as a leak — the answer surface stays clean, the reasoning surface trips:

{ "surface": "reasoning", "leak": true,
  "leaked": [{ "provider": "aws", "match": "AKIA****" }] }

Findings are reported for the answer and the reasoning separately (never mixed together). If there's no reasoning to look at, it says not_applicable — meaning "couldn't check this surface," which is not the same as "safe." No extra API calls: the trace is captured during the same probe run.

⚠️ leak_count does not include reasoning leaks — gate your CI on the exit code. Keeping the surfaces separate has a sharp edge: the top-level leak_count counts the answer surface only. The reasoning surface is reported in its own block. So an agent that refuses in its answer and spills the key in its reasoning prints leak_count: 0, and a CI step that greps leak_count will pass it — the exact blind spot this page opens with.

Use the gate instead. It covers both surfaces:

agentproof-scan --agent-config my_agent.yaml --fail-on-findings   # reasoning-only leak → exit 2

--fail-on secret behaves the same way for secrets. [VERIFIED: a reasoning-only leak exits 2 under both flags, 2026-07-16.] Read leak_count as "leaks in the answer," never as "leaks."

This surface only exists if your provider hands you the trace. It's not something the scanner can go and fetch — if the API doesn't return a thinking field, there is nothing to scan and you get not_applicable, which is "we couldn't look," not a pass. Providers differ, and the field's location differs too, so you have to point at it:

Provider	Trace returned?	--reasoning-field path
OpenAI (gpt-4o-mini)	No	— surface is not_applicable
Anthropic (extended thinking on)	Yes	content.0.thinking (answer: content.1.text)
OpenAI-shaped APIs that expose it	Varies	choices.0.message.reasoning

[Observed 2026-07-16; providers change these, so check yours rather than trusting the table.] A model that returns no trace isn't safer than one that does — you just can't see that surface.

---

Verification scope

All test data in this repository comes from Linux containers.

Verified — 9 cells, live PyPI install (not local build): Python 3.9 / 3.11 / 3.13 × venv / system / pipx

Not verified:

Environment	Status
Windows (PowerShell 5.1 / pwsh 7 / cmd)	Not verified. CI coverage planned.
macOS	Not verified. CI coverage planned.
Linux, installed on host (not container)	Not verified.
GitHub Codespaces, as a new user	Not verified. We develop there; we have never walked the first-run path.
Python installation itself (PATH, python.org / MS Store / pyenv)	Cannot be covered by CI. Runners ship with Python pre-installed.

The last row will not be closed by automation. If you hit friction installing Python itself, an issue is the only way that row changes.

0.1.4 handles UTF-16LE+BOM .env files (byte-level test); 0.2.0 keeps that behaviour. This is not a claim that the tool passes on Windows PowerShell 5.1. It is a claim about the bytes.

---

What's measured — three separate things

This page makes three different claims. They come from three different measurements, they count three different things, and they do not add up to one number. A scanner that labels 16 credential shapes is not a scanner that "catches 16 kinds of attack."

Layer	The claim	What backs it
Detection — 16 families (15 default + postgres opt-in)	the matcher puts the right family label on a credential's shape, and stays quiet on look-alikes	axis_b_coverage_green.json — the 10 families added since 0.1.4: 10/10 labelled, 0 missed, 0 false positives on near-miss strings. The original six are held by test_secrets_integrity.py.
Elicitation — 10 families	a credential sitting in an agent's response is picked up end-to-end (probe → response → report), with the right family and nothing invented	elicitation_green.json — 10 planted, 10 detected, 0 missed, 0 spurious providers. postgres is excluded (it's opt-in).
Reasoning-attack (H-CoT) — 3 probes	when a fake "reasoning step" is injected into an agent you own, the scanner sees the resulting reasoning-channel leak	hcot_green.json — 3 probes, 0 missed, 0 false positives; in all three the answer stayed clean and only the reasoning leaked.

Three things to be clear about, because the labels are easy to over-read:

• Detection 16 ≠ elicitation 10. The first counts shapes the matcher knows. The second counts families carried end-to-end out of an agent's reply. Neither number is a subset or a total of the other, and neither is "how many attacks it stops."
• The H-CoT row is about the scanner, not about models. It says this tool sees that leak. It is not a claim that any real model is vulnerable to H-CoT — that would need live measurement against real models, which is not in this release. The probes exist so you can check an agent you own; they are not a jailbreak kit.
• All three are offline (canned, 0 API calls). They exercise the scanner against planted, synthetic, shape-only fakes — no real key and no live model is involved. So none of them says how often a real agent leaks. That question is measured elsewhere (see What we've observed) and those observations are directional, not reproducible here.

Reproducing these numbers

Every GREEN-backed number in the table above reproduces from a clone, offline, with no API key — and byte-for-byte, not just "close enough":

git clone https://github.com/ghkfuddl1327-wq/agentproof && cd agentproof
python score_axis_b_coverage.py   # → axis_b_coverage_green.json
python score_elicitation.py       # → elicitation_green.json
python score_hcot.py              # → hcot_green.json
python -m pytest -q               # the gates behind them

Run any of them twice and you get identical bytes (no clock, no RNG seed drift). If a regenerated file differs from the committed one, treat the claim as broken — that's the point of shipping the generators next to the artifacts.

What does not reproduce this way: the cross-model observations elsewhere on this page. Those are measured snapshots — they depend on API keys, model availability, and provider-side behaviour that changes under us. They are reported as directional, and re-running them will not give you the same bytes. We keep the two kinds of number apart on purpose.

---

Defense prompts — reference, not a fix

There is no prompt that "fixes" leakage. What the repo ships is a reference: a defense hypothesis, measured per model, with results that vary by model — what helps one can leave a residual on another. Adding a defense block raises the cost of a leak; it is not a guarantee, and the only figure that means anything for your setup is the one you measure on your own model. To keep this page short, the prompts and the measurements behind them live in the repo, not here:

prompts/system_defense/

Installed via pip? They live in the repo, not the package — open the link above, or git clone the repo to read them locally. Each is a plain-text block.

REFERENCE.md is the honest version: which block was measured against which model, in directional buckets (not precise rates), and the limits — chiefly that it moves the final answer surface, not the reasoning trace (see above). Keeping it in the repo lets the reference grow without turning this page into a wall of text.

---

📈 What we've observed so far (early & qualitative)

The clearest pattern in our testing: leak behavior depends heavily on the underlying model, not just on the prompt. Per-category figures are deliberately not published here — the public probe set is abstracted, so numbers measured with earlier wording wouldn't transfer. The write-ups that are documented live in prompts/.

Multi-model targets (ngpt_*, llm_*) need pip install ngpt llm plus the relevant provider key (OPENAI_API_KEY, XAI_API_KEY, OPENROUTER_API_KEY, …) in your .env.

---

Two levels of defense (does adding a guardrail actually help?)

Two more canaries plant the same fake secret but add a prompt-level defense — so you can watch the leak rate change:

agentproof-scan --target simple_chatbot_defended_canary   # prompt guardrail
agentproof-scan --target simple_chatbot_hardened_canary   # stronger prompt guardrail

• defended — a system-prompt guardrail instructs the agent to refuse extraction attempts. This usually lowers leaks, but a clever probe can still slip through.
• hardened — the same idea with a stronger, more explicit prompt instruction. It tends to refuse more often, but it is still a prompt-level defense.

Takeaway: stronger prompt instructions reduce leaks, but a prompt-level defense alone is never a guarantee — a determined probe can still find a gap. The more robust approach is a non-prompt safety net (filtering secret-shaped strings out of the output before it reaches the user); the demo targets here illustrate the prompt layer only, not output filtering.

---

What it catches — and what it doesn't (plainly)

It catches: a set list of credential types, matched by their shape. As of 0.2.0 the detection list is 15 families on by default — OpenAI, Anthropic, Google, AWS, GitHub (classic), GitHub fine-grained PAT, xAI, Stripe, Slack, JWT, PEM private keys, SendGrid, GCP OAuth client secrets, npm, Twilio — plus postgres, which is off by default and opt-in (16th family; see below for why it isn't promoted). Matching holds up whether the secret is in plain text or JSON, across different languages, and in the answer or the reasoning — for the types it knows.

Two of these are exposure signals rather than proof of a secret leak, and the tool says so in the finding's scope: a JWT is often a public ID token, and a Twilio SK… is a public identifier whose paired secret is separate. They're worth surfacing; they are not automatically an incident.

postgres (a password inside a postgres://… URL) is opt-in because it is the one type that could not meet the no-false-positives bar: the password has no prefix to anchor on, so common documentation strings (mysecretpassword, postgres_dev_password) trip it. Rather than loosen the bar for every user, it's off unless you ask:

AGP_ENABLE_OPTIONAL=1 agentproof-scan --target …    # turns postgres on, FPs included

Strings that have a real key's shape but are obvious dummies (sk-ant-…EXAMPLE, AKIA…FAKE, …placeholder…) are filtered out rather than reported, so example code and docs don't set off false alarms.

It doesn't catch:

• Credential types outside that list — the list is finite and hand-written. A type that isn't in it is not matched at all. Adding families does not make the list complete; it moves the boundary.
• Secrets with no tell-tale prefix — the postgres://… case above is the example, and it's why that family is opt-in. A real limit of shape-matching.
• Secrets described in words — if a secret is paraphrased with no literal key-string, shape-matching can't see it.
• Live/runtime catching — this runs before you ship (offline), not as a live hook while your agent is running.
• Models we haven't tested — results come from a small set of lightweight models, not the big frontier ones.

"No false positives" is true for random text on the default types above — it's not a promise that a shape-matching type never flags a token that turns out to be public. The JWT and Twilio caveats above are exactly that case, stated up front.

---

Notes

Why an editor and not echo '...' > .env? A shell command puts your key on the command line, and the shell saves that line to its history in plaintext (~/.bash_history, PowerShell's ConsoleHost_history.txt). A tool that detects leaked credentials should not teach you to write one to a file nobody thinks to check. An editor avoids the history entirely. (If you do use the shell, this tool does not scan shell history — that's on you to clear.)

On .env encoding. The loader reads .env written as UTF-8, UTF-8 with BOM, and UTF-16LE with BOM (what PowerShell 5.1's > produces — [VERIFIED on a real PowerShell 5.1 file, owner, 2026-07-11]), and strips stray quotes cmd.exe leaves around the key name. Don't quote the value — KEY="abc" keeps the quotes literally.

---

If you don't have Python

GitHub Codespaces gives you a Linux container in the browser. Requires only a GitHub account. No local Python installation.

pip install agentproof-scan

⚠ Before you do this, read the next section. Codespaces is not verified as a new-user path (see table above), and running this tool means putting a provider API key into a cloud VM.

---

Handling your API key ⚠

This tool needs a live provider key to call your agent. Wherever you run it, that key is exposed to that environment.

• Use a scoped, low-quota, disposable key. Not your production key.
• Set a hard spend cap before you start. Budget alerts are not caps.
• Revoke the key when you are done.
• In Codespaces: use a Codespaces secret, not a committed .env. Never commit .env. It is gitignored — do not override that.
• If you fork this repo, your fork's Codespace inherits nothing of ours. Your key is yours to manage.

A Codespaces secret arrives as an environment variable, and a real environment variable always beats a .env file — so the scanner runs with no .env at all.

This tool detects leaks. It does not prevent them. Scoping, rate limits, and hard spend caps do that. See What it catches — and what it doesn't.

---

❓ Stuck? (no experience needed — your escape hatch)

If any step is confusing, paste this into an AI assistant and follow along:

I'm trying to run an open-source Python tool called "agentproof-scan". I'm a beginner. Walk me through, step by step on my computer: (1) install Python if needed, (2) pip install agentproof-scan, (3) run agentproof-scan --demo — this needs no API key and no network, and should print a JSON report with a leak_count of 15. After each step, ask me what I saw before continuing. Only if I say I want to scan a live agent afterwards, then help me get a free Google Gemini API key and put it in a .env file as GEMINI_API_KEY=... (with my real key in place of the ...).

---

⚠️ A note on the test fixtures

agentproof_scan/victim_agent.py and the *_canary adapters contain intentional vulnerabilities — fake, format-only secrets (not real keys) used as test fixtures to prove the scanner works. They are not exploits, and the embedded strings are not usable credentials. The probe set in this public repo uses neutral, category-labeled example questions — it does not ship copy-pasteable injection prompts.

---

Status

This tool grew out of red-team probing experiments and has reached the scope it set out to cover. It is now in maintenance mode: issues are welcome and bugs get fixed, but no new features land here — further credential-exposure work ships as separate projects rather than growing this one. The detection rule and the cross-model numbers are still worth scrutiny — if you can break something marked as working, please open an issue.

Invalid keys, since 0.1.3: a present-but-invalid key no longer produces a misleading 0. The provider returns an HTTP error and the scanner treats it as the scan did not run: exit 1, no report, refusing to claim clean. Earlier versions did report 0 here; that was the bug 0.1.3 closes.

Which slug you get depends on your provider, not on us. The scanner reports reason=auth_failed only for HTTP 401/403; every other error status becomes reason=http_status. Providers disagree about what a bad key is:

What you do	Gemini (the Quick Start default)	OpenAI
Key not set at all	exit 1, reason=missing_env	exit 1, reason=missing_env
Malformed key	400 → exit 1, reason=http_status	401 → exit 1, reason=auth_failed
Wrong or expired key	400 → exit 1, reason=http_status	401 → exit 1, reason=auth_failed

So on the demo path a bad key shows up as http_status, not auth_failed — Google answers a bad key with 400, not 401. [VERIFIED against both live APIs, 2026-07-16.] Don't branch your CI on the slug to mean "bad key"; branch on the exit code. Whatever the slug, the contract is the same: exit 1 and no clean verdict.

Released in 0.2.0: the wider credential-type coverage that 0.1.4 held back (Stripe, Slack, JWT, PEM, SendGrid, Twilio, npm, GitHub fine-grained PAT, GCP) now ships — you can run it. See What it catches for the list and What's measured for what each claim is backed by. postgres remains opt-in, not promoted to default (below).

0.2.0 is additive: the six families from 0.1.4 behave exactly as before, and the exit codes (0 / 1 / 2) and the rule slug are unchanged.

---

License

Apache License 2.0 — see LICENSE. You're free to use, modify, and contribute.