pytest-wardenbot

Pytest security tests for chatbots and tool-using agents — prompt injection, MCP and application boundary checks, data leaks and semantic regressions.

📖 Documentation: pytest-wardenbot.wardenbot.ai

Status: pre-release. The 0.3 series is in active development. APIs may change before the first stable release. Follow ongoing work in GitHub Issues.

What it does

Run pytest against chatbot responses and instrumented agent operations. Check prompt injection, data disclosure, task correctness, and unauthorized tool effects.

• Black-box. Tests run against your live chatbot via HTTP, OpenAI API, Anthropic API, or any object you write a small adapter for.
• Deterministic-first. The chatbot suite ships 29 tests graded with regex, substring, and schema checks, without a separate judge model. Your chatbot's API usage may still cost money. Optional LLM-judge tests (DeepEval) ship as an extra for semantic checks.
• Agent-ready failures. When a test fails, the failure message includes a structured Markdown remediation prompt you can paste into Cursor or Claude Code.
• Public adapter checks. The bundled OpenAI and Anthropic adapters have mock-based tests and a scheduled live API smoke workflow. Check the latest workflow results for the current live status.
• Instrumented agent scenarios. Eight versioned cases cover MCP tool results, descriptions, resources and prompts, each with a benign baseline. A recording sink proves effects; required-coverage mode prevents missing evidence from becoming a successful CI run.
• Application boundary packs. Opt-in retrieval, memory, synthetic tenant/role and approval checks exercise controlled local operations. Each includes benign, vulnerable, resistant and no-exposure controls. See the application integration guide.

What "passing" means (and doesn't)

A passing chatbot test means its configured response check did not detect a failure. A passing required agent suite additionally needs observed exposure, complete declared-boundary evidence, and successful benign task controls. Neither result establishes security beyond the tested cases. See the versioned coverage matrix and evaluation limits.

A green run does not mean your chatbot is secure. Frontier-grade attacks are multi-turn, novel, and adapted to your specific bot — no fixed corpus catches all of them. Treat the shipped suite as a starter set: pair it with periodic red-team exercises (or our Continuous Monitoring service) for the always-on adversarial coverage CI alone can't provide.

Install

Requires Python 3.11+. The package is maintained by the WardenBot AI organization; the GitHub transfer does not change its PyPI name or Python imports.

pip install pytest-wardenbot

Optional extras for LLM-judge tests or vendor-native adapters:

pip install "pytest-wardenbot[judge]"        # adds DeepEval for semantic checks
pip install "pytest-wardenbot[openai]"       # adds OpenAI Responses + Chat adapters (sync + async)
pip install "pytest-wardenbot[anthropic]"    # adds Anthropic Messages adapter (sync + async)
pip install "pytest-wardenbot[langchain]"    # adds LangChainAdapter for any Runnable (sync + async)
pip install "pytest-wardenbot[async]"        # adds pytest-asyncio for parallel async probing (run_probes)
pip install "pytest-wardenbot[mcp]"          # adds the controlled local MCP harness

Assistants retirement: OpenAI retired the Assistants API on 26 August 2026; it is unavailable. Use OpenAIResponsesAdapter for new OpenAI text integrations. Legacy Assistants class names remain importable for explicitly injected test clients or compatible proxies; their default constructors fail before SDK or network calls. See OpenAI migration.

See the dependency support matrix for validated SDK combinations and adapter setup.

Instrumented agent trials

For larger runs, bounded campaigns share call and turn budgets across isolated trials, drain cooperative cancellation, and retain every planned case in the report. Retries require an explicit safety declaration.

Portable evidence keeps versioned findings, fingerprints and provenance in JSON, JUnit and standalone HTML. Saved observations can be re-graded separately from new executions; redacted evidence cannot imply a passing grade. A fixed synthetic fixture demonstrates replay from a clean install.

The additive pytest_wardenbot.agents API records explicit outcomes: violation, observed resistance, inconclusive evidence, unsupported capabilities and execution errors. A trusted recording sink establishes effects; model claims do not. Task success is tracked separately, and diagnostic content is hidden by default.

The optional MCP harness exercises tool results, tool descriptions, resources and prompts over real local MCP transports, with scripted vulnerable/resistant/unexposed controls. It needs no provider credentials. See agent trials for the contract and application integration responsibilities.

After configuring your agent_target and agent_observer fixtures:

pytest --pyargs pytest_wardenbot.agent_tests --wardenbot-agent-required
# Select one versioned scenario and its benign baseline:
pytest --pyargs pytest_wardenbot.agent_tests \
  --wardenbot-agent-scenario=mcp.tool-result.forbidden-tool@1 \
  --wardenbot-agent-required

The scenario guide includes a complete local fixture setup. The reference agent and pilot methodology exercise a bounded model/tool loop with paired benign and attack trials. Existing chatbot tests remain in the separate pytest_wardenbot.tests suite.

A second support application example uses direct local ticket APIs, scoped approvals, and a real model tool loop without MCP. Both examples default to offline SDK controls. The support pilot records repeated trials, held-out cases, benign quoted/encoded text, utility, incomplete results, and sample-size limits; it does not rank models.

Quickstart (under 60 seconds)

pip install pytest-wardenbot
pytest --wardenbot-quickstart           # generates conftest.py + test_my_bot.py
export CHATBOT_URL=https://your-chatbot.example.com/chat
export CHATBOT_TOKEN=sk-...              # optional
pytest                                   # runs all shipped tests against your bot

--wardenbot-quickstart accepts an industry template:

pytest --wardenbot-quickstart=ecommerce       # adds refund/shipping fact placeholders
pytest --wardenbot-quickstart=saas-support    # adds plan/trial fact placeholders
pytest --wardenbot-quickstart=generic         # default; minimal placeholders

Then edit conftest.py to replace the TODO placeholders with your real business facts and re-run pytest. Worked examples in examples/ cover the basic HTTP setup, a custom OpenAI adapter, and a GitHub Actions workflow.

Manual setup (if you prefer)

Add this to your project's conftest.py:

import os
import pytest
from pytest_wardenbot.adapters.http import HTTPChatbotAdapter

@pytest.fixture
def chatbot():
    return HTTPChatbotAdapter(
        url="https://your-chatbot.example.com/chat",
        headers={"Authorization": f"Bearer {os.environ['CHATBOT_TOKEN']}"},
        request_field="message",      # the JSON key your bot reads the prompt from
        response_field="response",     # the JSON key your bot returns the text in
    )

Then run the shipped tests with pytest --pyargs pytest_wardenbot.tests.

When a test fails, read the failure message, paste the agent-ready Markdown into Cursor / Claude Code, ship the fix.

Bundled chatbot checks

That's 29 deterministic tests out-of-the-box (plus the opt-in canary leak test, plus your business-truth and judge lists). Runtime and chatbot API costs depend on your endpoint; the deterministic checks do not call a separate judge model.

LangChain adapters, ensemble judging, and parallel async probes are available in the 0.1 series. Follow upcoming work in GitHub Issues.

How it's different from related tools

• vs Promptfoo: Promptfoo is a developer testing CLI. We're a pytest plugin — same tool your existing test suite uses, same CI integration you already have.
• vs DeepEval: DeepEval focuses on evaluation metrics (faithfulness, relevancy). We focus on adversarial security probes (jailbreak, system-prompt leak, refusal-bypass) — different problem, complementary tool. (We use DeepEval under the hood for our optional semantic checks.)
• vs Garak / PyRIT: Garak and PyRIT are research-grade attack libraries. We package a curated subset as everyday pytest tests with clear failure messages.

License

Apache 2.0. See LICENSE.md.

Powered by

WardenBot AI — continuous external monitoring for AI chatbots.

The pytest plugin is the free, open-source slice of our test corpus. Want continuous monitoring across all your bots with daily probes and a dashboard? Tell us about your setup — we open invites in small batches.