watchllm 0.1.1

AI agent reliability testing platform

AI agent reliability testing infrastructure. Stress test your agents against 8 attack categories—prompt injection, tool abuse, jailbreaks, data exfiltration, and more—before they hit production.

Observability, evaluation, and adversarial testing platform for LLM agents.
Ship reliable, secure, and debuggable agentic systems.

Website • Documentation

---

Installation

Install the official Python SDK via pip:

pip install watchllm

Authentication

WatchLLM requires an API key to securely log your simulations to your dashboard.

Option A: Interactive CLI Login (Recommended for Local Dev)

watchllm auth login

This will prompt you for your API key and securely save it to ~/.watchllm/config.

Option B: Environment Variables (Recommended for CI/CD) Set the following environment variable in your pipeline or .env file:

export WATCHLLM_API_KEY="your_api_key_here"

Running Simulations via CLI

The fastest way to test an agent is using the WatchLLM CLI.

# Run a specific attack category
watchllm simulate --agent my_module.my_agent --categories prompt_injection

# Run multiple categories simultaneously
watchllm simulate --agent my_module.my_agent --categories prompt_injection,tool_abuse,hallucination

# Run all 8 attack categories
watchllm simulate --agent my_module.my_agent --categories all

Note: Replace my_module.my_agent with the Python import path to your agent function. Your agent function must accept a string and return a string.

Native Python Integration (CI/CD)

For automated testing inside your test suite or CI/CD pipelines, wrap your agent with the @watchllm.test decorator. This exposes your agent's signature to the WatchLLM runner.

import watchllm

@watchllm.test(
    categories=["prompt_injection", "data_exfiltration"],
    threshold=0.3,   # Raises WatchLLMThresholdError if severity >= 0.3
)
def my_agent(user_input: str) -> str:
    # Your agent logic here
    return response

If the agent exhibits a vulnerability during the test, the SDK will automatically fail the pipeline and exit with a non-zero status code.

Analyzing Results

Once a simulation completes, the CLI will output the terminal state:

Simulation ID: sim_abc123
Status: completed
Severity: 0.90
Verdict: Critical vulnerability detected

To view the complete execution graph, scrub through the node lineage, and replay specific forks, visit the dashboard: https://dashboard.watchllm.dev

Command Reference

Command	Description
watchllm doctor	Verifies your setup, API key validity, and agent reachability.
watchllm simulate	Launches a new attack simulation.
watchllm replay	Prints a text-based tree of the execution graph directly in your terminal.
watchllm status	Returns the current completion percentage and live severity scores of a running simulation.

Requirements

• Python 3.9 or higher
• The requests library (installed automatically)