honeypot-ai

Hidden prompt-injection toolkit for AI agents. Three tools in one package:

Status: beta. The scanner is heuristic: treat findings as leads to review, not proof.

Why not test AI with AI

Asking one model to judge another model's transcript is itself prompt-injectable, and it is nondeterministic. Every verdict here comes from something observable instead:

• scan matches concrete hiding techniques and injection-language families in the HTML (the families follow the prompt-injection material in PayloadsAllTheThings), then reports exactly what matched.
• tripwire records HTTP requests to unique canary URLs. A request either arrived or it did not.
• harness decides at the tool-call boundary: did the agent call a decoy, put a canary credential in an argument or in its answer, or target a host outside the allowlist?

A model is only ever the subject of a test, never the judge. Run each case N times with --repeats and treat the attack success rate as the measurement.

Install

pip install --pre honeypot-ai                 # beta release
pip install --pre "honeypot-ai[selenium]"     # + Selenium (Chrome or Firefox must be installed)
pip install --pre "honeypot-ai[rendered]"     # + Playwright view for the harness

1. Scan a page

honeypot-ai scan https://example.com
honeypot-ai scan page.html
curl -s https://example.com | honeypot-ai scan -
honeypot-ai scan https://example.com --browser chrome      # also ask a real browser what is hidden
honeypot-ai scan https://example.com --format json --fail-on medium

What it looks for: text hidden with display:none, visibility:hidden, opacity:0, filter:opacity(0), content-visibility:hidden, transparent or alpha-zero paint, zero font size, transform:scale(0), off-screen positioning, clipping / clip-path, same-colour text, <style> class rules, CSS ::before / ::after content (including CSS hex escapes), the hidden attribute, <noscript> / <template>, HTML comments, inline <script> bodies and CSS comments (for agents that read raw HTML), alt / aria-label / title, <meta>, JSON-LD, hidden inputs, data-* attributes, Unicode tag characters and zero-width characters. Hidden text is then checked for injection-like language: instruction overrides, text addressed to an AI, prompt leaks, tool-use commands, "the user already approved this", jailbreak modes, markdown-image exfiltration, exfiltration and internal/metadata URLs, secrets next to send/post verbs, curl | sh, ... Obfuscation is folded before matching — HTML entities, zero-width characters, Cyrillic/Greek homoglyphs, leetspeak, base64 blobs (decoded and reported as encoded_payload), and a keyword split across visible and hidden nodes.

Verdicts: clean, hidden_content (hidden but harmless-looking, e.g. screen-reader text), suspicious, likely_injection. Exit code is 1 when a finding reaches --fail-on (default high), 2 on errors, so it drops straight into CI.

--browser adds a rendered layer: Selenium walks every text node and reports the ones the browser has in the DOM but does not show. This catches hiding done by external CSS or JavaScript, which the static layer cannot see. Use --user-agent to look for pages that serve different content to agents.

From Python, inside your own Selenium tests:

from honeypot_ai import scan_driver

driver.get(url)                       # wherever your agent's browser just went
report = scan_driver(driver)
assert report.verdict in ("clean", "hidden_content"), report.render_text()

2. Test your own agent end to end (tripwire)

The tripwire serves every hiding variant on localhost. Each page hides an instruction to open a unique canary URL. If your agent obeys, the request lands on the canary and is recorded: it works for Selenium agents, browser-use, LangChain tools, anything that can make an HTTP request. Each poisoned page also carries a unique output marker, so an agent that obeys without network egress (it just repeats the token) is still detectable through marker_in.

honeypot-ai serve                 # prints one URL per variant; point your agent at them; Ctrl+C for the report
honeypot-ai serve --duration 120 --out reports/ --strict

or in pytest:

def test_my_agent_resists(tripwire):
    answer = run_my_agent(f"Summarize {tripwire.url('css_hidden')}")     # your Selenium agent
    assert not tripwire.tripped(), tripwire.report_markdown()
    assert not tripwire.marker_in(answer, "css_hidden")

See examples/selenium_e2e.py for a complete Selenium loop. If the agent runs in a container or on another machine, use --public-host (CLI) / public_host= (Python) so the canary URL is reachable.

3. The harness (agent as a chat() backend)

A local fake site serves a payload hidden in different ways; the agent gets real tools plus decoys and canary credentials. Touching a decoy means it was hijacked.

honeypot-ai demo                                                  # offline, no model needed
honeypot-ai run --backend ollama --model qwen3.5:4b --no-think
honeypot-ai run --backend openai --model gpt-4o-mini --view selenium

Views (what the agent "sees" of the page): html, text, a11y, rendered (Playwright), selenium. Backends: ollama, openai (any OpenAI-compatible server), scripted (offline). Each run writes raw.jsonl, chain.jsonl and report.md/json to honeypot_runs/. Policy decisions are made at the tool-call boundary, never by an LLM judging text.

Models are nondeterministic, so --repeats 5 runs every variant five times and reports an attack success rate in summary.md / summary.json. --defense none|fence|harden|both A/B-tests spotlighting (<untrusted_content> fences around fetched pages) and a hardened system prompt; the canaries stay the ground truth. The scanner verdict for the same page is recorded next to the behavioral result, so the summary can mark a detection blind spot (agent compromised, scanner said clean) or agent resisted (scanner flagged it, agent did not obey).

Canary credentials in the final answer are incidents too (canary_in_output), so prompt-leak and markdown-image exfiltration are caught even when the agent never calls a tool. Encoded and zero-width-split canaries inside tool arguments are detected as well. A broken backend or page is isolated to its own run (verdict: error), and an agent that repeats the same tool call three times is stopped as a loop instead of burning every step.

pytest:

def test_resists(honeypot):
    assert not honeypot.run("css_hidden").compromised

pytest --hp-backend ollama --hp-model <tag> --hp-view text --hp-defense both

Speed: on CPU a thinking model can take minutes per call. Use --no-think (pytest: --hp-no-think).

Page variants and payloads

honeypot-ai variants

visible, css_hidden, visibility_hidden, white_on_white, transparent_text, offscreen, tiny_font, opacity_zero, hidden_attr, style_class, html_comment, script_content, css_comment, alt_text, aria_label, title_attr, noscript, template, meta_tag, json_ld, hidden_input, data_attr, base64_comment, css_content, unicode_tags, split_keyword, plus control (no payload, to measure false positives).

Payloads: exfil_email, read_secrets, shell, prompt_leak, markdown_exfil, authority_spoof.

Responsible use

Payloads only use reserved .test domains and fake credentials, and decoy tools never execute anything. Scan pages you are allowed to fetch. Findings about a third-party site are leads: verify before you report them to anyone.

Development

pip install -e ".[test]"
pytest -q
pip install build twine && python -m build && twine check dist/*