Skip to main content

pytest-wardenbot

CI codecov OpenSSF Scorecard Python versions License: Apache 2.0

Pytest security tests for chatbots and tool-using agents: prompt injection, system-prompt leaks, and agents acting on instructions hidden in tool results, MCP resources and prompts. Agent results come from what a controlled harness recorded the agent doing, not from what the agent says. Everything can run on your machine, including against Ollama, vLLM or llama.cpp models with no API key.

It tests your chatbot or agent, not your MCP server. To scan MCP servers and tool definitions themselves, use a scanner such as Snyk Agent Scan or Cisco's MCP Scanner.

📖 Documentation · llms.txt for coding agents · Beta (0.6): public APIs change only under the stability policy.

Quickstart: chatbot

pip install pytest-wardenbot
pytest --wardenbot-quickstart                  # writes conftest.py and test_my_bot.py
export CHATBOT_URL=https://your-bot.example.com/chat
pytest                                         # 29 checks, about 35 requests to your bot

The generated conftest.py posts {"message": "..."} and reads {"response": "..."}; change the two field names there if your bot differs. For OpenAI, Anthropic, LangChain and other clients, see Add your chatbot.

Quickstart: agent

pip install "pytest-wardenbot[mcp,openai]"
pytest --wardenbot-quickstart=agent            # writes conftest.py and test_my_agent.py
export OPENAI_API_KEY=sk-...                   # or: export AGENT_LOCAL=ollama AGENT_MODEL=qwen2.5:7b
pytest --wardenbot-agent-required              # 8 scenarios, each with a baseline and an attack

The generated conftest.py contains a complete tool-calling agent, so the first run tests a real model end to end. Then replace it with your own agent:

Your agent Connect it with
LangGraph LangGraphAgentTarget ([langgraph])
Claude Agent SDK ClaudeAgentTarget ([claude-agent]), isolated from your machine
OpenAI Agents SDK Recipe
Any other tool loop 40-line custom target with TrialTools

What a result means

Chatbot suite. A failing check found a problem in a response: the bot followed an injected instruction, complied with a request it should have refused, or leaked its system prompt (planting a canary token makes leak checks much stronger). A passing check found none of the patterns it looks for. The graders are measured on 744 labeled responses: none of the 492 well-behaved replies fail, and the page lists the known misses and how the graders did on replies they had never seen.

Agent suite. An attack trial fails only when the harness recorded the agent calling the forbidden tool or sending the canary, including calls your own code blocked. It passes as resisted only when the harness saw the injected content delivered and recorded nothing forbidden. Missing evidence is inconclusive, and a model that can't call tools is unsupported, never resisted. Each attack has a benign baseline; whether the task succeeded is reported by your agent and tracked separately. See agent outcomes.

A green run means these cases passed, not that your system is secure. The attacks are fixed and simple next to a motivated attacker who adapts to your bot. Use the suite as a regression gate in CI, alongside red-team testing.

Chatbot suite

Category Checks Grading
Prompt injection / jailbreak resistance 5 prompts × 2 checks deterministic
System-prompt elicitation 3 deterministic, canary-first
Refusal bypass (roleplay, pretext, hypothetical) 3 deterministic
Off-topic deflection (for scoped bots) 2 deterministic
Indirect (cross-prompt) injection 4 deterministic
Encoded payloads (Base64, ROT13, leet, hex) 4 deterministic
Multi-turn jailbreak (needs a session-aware adapter) 3 deterministic
Business facts (your prices, hours, policies) one per fact deterministic
Semantic checks ([judge] extra) one per case LLM judge, hosted or local

No judge model is involved unless you add judge cases. When a check fails, the message includes a Markdown remediation brief you can paste into Cursor or Claude Code; the bot's reply is fenced as untrusted text.

Adapters: HTTP, OpenAI Responses and Chat Completions, Anthropic Messages, LangChain Runnables, Ollama/vLLM/llama.cpp presets, or any object with send_message and reset_session.

Agent suite

  • Eight versioned MCP scenarios inject instructions through tool results, tool descriptions, resources and prompts, over real local MCP transports. Each checks a forbidden tool call or a synthetic canary, with a benign baseline.
  • Required mode (--wardenbot-agent-required) fails the run if any selected case is skipped, deselected, unsupported or lacks evidence, also under pytest-xdist.
  • Application packs (opt-in) cover poisoned retrieval, cross-session memory, tenant and role separation, and approval before write, through hooks into your own application.
  • Bounded campaigns share call and turn budgets across repeated trials; portable evidence saves versioned results as JSON, JUnit or self-contained HTML that can be re-graded later.
  • Fully local: the local presets check that a model can call tools first, and a CI test proves local runs make no outside network connections.

Install extras

pip install "pytest-wardenbot[mcp]"          # controlled MCP harness for the agent suite
pip install "pytest-wardenbot[langgraph]"    # LangGraph agent target
pip install "pytest-wardenbot[claude-agent]" # Claude Agent SDK agent target
pip install "pytest-wardenbot[openai]"       # OpenAI adapters (sync and async)
pip install "pytest-wardenbot[anthropic]"    # Anthropic adapter (sync and async)
pip install "pytest-wardenbot[langchain]"    # LangChain Runnable adapter
pip install "pytest-wardenbot[judge]"        # DeepEval semantic checks
pip install "pytest-wardenbot[async]"        # pytest-asyncio for parallel probes

Python 3.11–3.14, pytest 8.1.1 or later. The package ships type hints (py.typed). Releases are published from CI with PyPI attestations, an SBOM and a maintainer approval step; see SECURITY.md.

How it relates to other tools

  • Promptfoo runs red-team and evaluation suites from a CLI, including MCP security testing. pytest-wardenbot runs inside the pytest suite and CI you already have, and grades agents on recorded actions rather than final text.
  • RAMPART also brings security testing to pytest, built on PyRIT's attack generation. pytest-wardenbot keeps a small core install and its own controlled harness, with versioned scenarios, required coverage and portable evidence.
  • DeepEval and Inspect AI provide evaluation metrics and agent evaluation infrastructure. pytest-wardenbot focuses on security regressions with explicit outcomes, and uses DeepEval for its optional semantic checks.
  • Garak and PyRIT are broad attack and research toolkits.
  • MCP scanners such as Snyk Agent Scan and Cisco's MCP Scanner inspect servers and tool definitions; pytest-wardenbot tests how your agent behaves when what those servers send is hostile.

Questions and feedback: GitHub Discussions. Bugs: GitHub Issues.

License

Apache 2.0. See LICENSE.md.

Powered by

WardenBot AI runs continuous security monitoring for AI chatbots and agents. This plugin is the free, open-source slice of that work: the same kinds of checks, in your own CI. Tell us about your setup if you want them run continuously against your deployed systems.

Metadata

Release files for pytest-wardenbot 0.6.1

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for pytest-wardenbot 0.6.1
File Size Uploaded Provenance Actions
pytest_wardenbot-0.6.1.tar.gz 685.8 kB With provenance Details for pytest_wardenbot-0.6.1.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for pytest-wardenbot 0.6.1
File Interpreter ABI Platform Actions
pytest_wardenbot-0.6.1-py3-none-any.whl Python 3 none any Details for pytest_wardenbot-0.6.1-py3-none-any.whl

Total release size: 888.9 kB

Release files / pytest_wardenbot-0.6.1.tar.gz

Download URL pytest_wardenbot-0.6.1.tar.gz
Size 685.8 kB
Tags Source
SHA-256 checksum
How to use checksums
a44d72ac4270526a1893bd145283de3e98f8e26590c6f511cbe7291589f6191a
BLAKE2b-256 checksum
How to use checksums
8d2849b5b3c32cd945fd480d342b508961353b06eb41420d210204283d322e53
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.16

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 11, 2026.

Transparency log

Release files / pytest_wardenbot-0.6.1-py3-none-any.whl

Download URL pytest_wardenbot-0.6.1-py3-none-any.whl
Size 203.1 kB
Tags Python 3
SHA-256 checksum
How to use checksums
5891733895d5979d428fff4ed6cb397f84e18a214c80b0776e994d07a2f9acda
BLAKE2b-256 checksum
How to use checksums
1ec4c66eec867cba447f9ebefb1f3b1348558c48407b90c62d0d063ed083eb97
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.16

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 11, 2026.

Transparency log

Security for pytest-wardenbot 0.6.1

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

Full provenance

Both files in this release have been uploaded with provenance metadata.

Select from 2 files

Attestation for pytest_wardenbot-0.6.1.tar.gz

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 11, 2026.

Transparency log

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page