aiandme-cli 0.2.0

AIandMe CLI - command line interface for AI agent security testing.

AIandMe CLI (Beta)

CLI-first security testing for AI agents and chatbots. Adversarial attacks, behavioral QA, posture scoring, and guardrails export — from your terminal to your CI/CD pipeline.

pip install aiandme-cli

---

Overview

AIandMe runs automated adversarial attacks against your bot's live endpoint, evaluates responses using LLM-as-a-judge, and produces structured findings aligned with the OWASP Top 10 for LLM Applications and the OWASP Agentic AI Threats.

Platform Services

Service	Description
CLI Tool	Full-featured command line interface. Initialize projects, run tests, check posture, export guardrails.
pytest Plugin	Native pytest integration with markers, fixtures, and baseline comparison. Run security tests alongside unit tests.
Adversarial Testing	OWASP-aligned attack scenarios: single-turn, multi-turn, adaptive, and agentic.
Behavioral Testing	Validate intent boundaries, response quality, and functional correctness.
Posture Scoring	Quantified 0–100 security score with breakdown by findings, coverage, and resilience. Track over time.
Guardrails Export	Generate protection rules from test findings. Export to OpenAI, Azure AI Content Safety, AWS Bedrock, or AIandMe format.

Why AIandMe?

Manual red-teaming doesn't scale. Static analysis can't catch runtime behavior. Generic pentesting tools don't understand LLM-specific attack vectors like prompt injection, jailbreaks, or tool abuse.

AIandMe is built for this. Point it at your bot's endpoint, define the scope (or let it extract one from your system prompt), and get a structured security report with actionable findings — all mapped to OWASP LLM and Agentic AI categories.

Testing feeds into hardening: export guardrails, track posture across releases, and catch regressions before they reach production. Works with any chatbot or agent, cloud or on-prem.

---

Get Started

1. Install & authenticate

pip install aiandme-cli
aiandme login

2. Scan your bot & create a project

aiandme init scans your bot, extracts its scope and risk profile, and creates a project — all in one step. Point it at one or more sources:

# From a system prompt file
aiandme init -n "My Bot" --prompt ./system_prompt.txt

# From a live bot endpoint (API probing)
aiandme init -n "My Bot" -e ./bot-config.json

# From a live URL (browser discovery)
aiandme init -n "My Bot" -u https://my-bot.example.com

# Combine sources for better analysis
aiandme init -n "My Bot" --prompt ./system.txt -e ./bot-config.json

The --endpoint/-e flag accepts a JSON config (file or inline string) matching the experiment integration shape:

{
  "streaming": false,
  "thread_auth": {"endpoint": "", "headers": {}, "payload": {}},
  "thread_init": {"endpoint": "https://bot.com/threads", "headers": {}, "payload": {}},
  "chat_completion": {"endpoint": "https://bot.com/chat", "headers": {"Authorization": "Bearer token"}, "payload": {"content": "$PROMPT"}}
}

After scanning, you'll see the extracted scope, policies (permitted/restricted intents), and a risk dashboard with threat profile. Confirm to create the project.

3. Run a security test

# Run against your bot (uses project's default integration if configured during init)
aiandme test

# Or specify an endpoint directly
aiandme test -e ./bot-config.json

# Choose test category and depth
aiandme test -t owasp_multi_turn -l system

4. Review results

# Watch experiment progress
aiandme status --watch

# View logs
aiandme logs

# Check posture score
aiandme posture

# Export guardrails
aiandme guardrails --vendor openai -o guardrails.json

---

Test Categories

Category	Mode	Description
owasp_single_turn	Adversarial	Single-prompt attacks: prompt injection, jailbreaks, data exfiltration. Fast coverage of basic vulnerabilities.
owasp_multi_turn	Adversarial	Conversational attacks that build context over multiple turns. Tests context manipulation and gradual escalation.
owasp_agentic_multi_turn	Adversarial	Targets tool-using agents. Tests goal hijacking, tool misuse, and privilege escalation.
behavioral	QA	Intent boundary validation and response quality testing. Ensures agent behaves within defined scope.

Adaptive mode: Both owasp_multi_turn and owasp_agentic_multi_turn support an adaptive flag that enables evolutionary search — the attack strategy adapts based on bot responses instead of following scripted prompts.

Testing Levels

Level	Description
unit	Standard coverage (~20 min) — default
system	Deep testing (~45 min)
acceptance	Full coverage (~90 min)

---

pytest Integration

Run security tests alongside your existing test suite with native pytest markers and fixtures.

# test_security.py
import pytest

@pytest.mark.aiandme
def test_prompt_injection(aiandme):
    """Test prompt injection defenses."""
    result = aiandme.test("llm001")
    assert result.passed, f"Failed: {result.findings}"

@pytest.mark.aiandme
def test_posture_threshold(aiandme_posture):
    """Ensure posture meets minimum."""
    assert aiandme_posture["score"] >= 70

@pytest.mark.aiandme
def test_no_regressions(aiandme, aiandme_baseline):
    """Compare against baseline."""
    result = aiandme.test("llm001")
    if aiandme_baseline:
        regressions = result.compare(aiandme_baseline)
        assert not regressions

# Run with AIandMe enabled
pytest --aiandme tests/

# Filter by category
pytest --aiandme --aiandme-category=adversarial

# Set failure threshold
pytest --aiandme --aiandme-fail-on=high

# Compare to baseline
pytest --aiandme --aiandme-baseline=baseline.json

# Save new baseline
pytest --aiandme --aiandme-save-baseline=baseline.json

---

CI/CD Integration

Block insecure deployments automatically with exit codes.

Build → Unit Tests → AI Security (aiandme test) → Deploy

# .github/workflows/security.yml
name: AI Security Tests
on: [push, pull_request]

jobs:
  security:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - run: pip install aiandme
      - name: Run Security Tests
        env:
          AIANDME_API_KEY: ${{ secrets.AIANDME_API_KEY }}
        run: |
          aiandme test --wait --fail-on=high

---

Usage

aiandme [--base-url URL] COMMAND [OPTIONS] [ARGS]

Authentication

Command	Description
login	Authenticate via browser (OAuth PKCE)
logout	Clear stored credentials
whoami	Show current authentication status

Organisation Management

Command	Description
orgs list	List available organisations
orgs current	Show current organisation
switch <id>	Switch to organisation

Provider Management

Providers are LLM configurations used for running security tests.

Command	Description
providers list	List configured providers
providers add	Add new provider
providers update <id>	Update provider config
providers remove <id>	Remove provider

providers add options

--name, -n        Provider name: openai, claude, azureopenai, gemini, grok, custom
--api-key, -k     API key
--endpoint, -e    Endpoint URL (required for azureopenai, custom)
--model, -m       Model name (optional)
--default         Set as default provider
--interactive     Interactive configuration mode

Project Management

Command	Description
projects list	List projects
projects use <id>	Select project
projects current	Show current project
projects show [id]	Show project details

init — scan bot & create project

aiandme init --name NAME [OPTIONS]

Sources (at least one required):
  --prompt, -p PATH       System prompt file (text source)
  --url, -u URL           Live bot URL for browser discovery (url source)
  --endpoint, -e CONFIG   Bot integration config — JSON string or file path (endpoint source)
  --repo, -r PATH         Repository path to scan (agentic or text source)
  --openapi, -o PATH      OpenAPI spec file (text source)

Options:
  --description, -d       Project description
  --timeout, -t SECONDS   Scan timeout (default: 180)
  --yes, -y               Auto-confirm project creation (no interactive prompts)

Test Execution

test — run security tests on current project

aiandme test [OPTIONS]

Test Category:
  --test-category, -t   Test to run (default: owasp_multi_turn)
                        Values: owasp_single_turn, owasp_multi_turn,
                                owasp_agentic_multi_turn, behavioral

Testing Level:
  --testing-level, -l   Depth of testing (default: unit)
                        unit | system | acceptance

Chat Endpoint (required):
  --chat-endpoint       Chat completion URL of the bot to test
  --chat-header         Header for chat endpoint (repeatable)
  --chat-payload        JSON payload template for chat

Init Endpoint (optional):
  --init-endpoint       Thread initialization URL
  --init-header         Header for init endpoint (repeatable)
  --init-payload        JSON payload for init

Auth Endpoint (optional):
  --auth-endpoint       Auth/token endpoint URL
  --auth-header         Header for auth endpoint (repeatable)
  --auth-payload        JSON payload for auth

Other:
  --provider-id         Provider to use (default: first available)
  --name, -n            Experiment name (auto-generated if omitted)
  --lang                Language (default: english). Accepts codes: en, de, es...
  --adaptive            Enable adaptive mode (evolutionary attack strategy)
  --streaming           Enable streaming mode (requires wss:// endpoint)
  --no-auto-start       Create without starting (manual mode)
  --wait, -w            Wait for completion
  --fail-on SEVERITY    Exit non-zero if findings >= severity
                        Values: critical, high, medium, low, any

Experiment Management

Command	Description
experiments list	List experiments
experiments show <id>	Show experiment details
experiments status <id>	Check status
experiments status <id> --watch	Watch until completion
experiments wait <id>	Wait with progressive backoff (30s → 60s → 120s → 300s)
experiments logs <id>	List experiment logs
experiments report <id>	Download HTML report

status is also available as a top-level alias — without an ID it shows the most recent experiment:

aiandme status [experiment_id] [--watch]

Results & Export

# View experiment results (table, json, or csv)
aiandme logs [experiment_id] [--format table] [--verdict pass|fail] [--page N] [--size N]

# Security posture score
aiandme posture [--json]

# Export guardrails configuration
aiandme guardrails [--vendor aiandme|openai|azure|bedrock] [--format json|yaml] [-o FILE]

Documentation

aiandme docs

Opens documentation in browser.

---

Examples

End-to-end: scan, create project, test, review

aiandme login
aiandme switch abc123

# Scan bot & create project (uses endpoint config file)
aiandme init -n "Support Bot" -e ./bot-config.json

# Run adversarial test (uses project's default integration)
aiandme test -t owasp_multi_turn -l unit

# Watch and review
aiandme status --watch
aiandme logs
aiandme posture

Multi-source project init

# Combine system prompt + live endpoint for best scope extraction
aiandme init \
  --name "Support Bot" \
  --prompt ./prompts/system.txt \
  --endpoint ./bot-config.json

# From repository + OpenAPI spec
aiandme init \
  --name "API Agent" \
  --repo ./my-agent \
  --openapi ./openapi.yaml

Bot config with auth + thread init

{
  "streaming": false,
  "thread_auth": {
    "endpoint": "https://bot.com/oauth/token",
    "headers": {},
    "payload": {"client_id": "x", "client_secret": "y"}
  },
  "thread_init": {
    "endpoint": "https://bot.com/threads",
    "headers": {"Content-Type": "application/json"},
    "payload": {}
  },
  "chat_completion": {
    "endpoint": "https://bot.com/chat",
    "headers": {"Content-Type": "application/json"},
    "payload": {"messages": [{"role": "user", "content": "$PROMPT"}]}
  }
}

# Use with init or test
aiandme init -n "My Bot" -e ./bot-config.json
aiandme test -e ./bot-config.json

Export guardrails

aiandme guardrails --vendor openai --format json -o guardrails.json

---

Configuration

Environment Variables

Variable	Description	Default
AIANDME_BASE_URL	API base URL	https://api.aiandme.io
AIANDME_AUTH0_DOMAIN	Auth0 domain (on-prem)	aiandme.eu.auth0.com
AIANDME_AUTH0_CLIENT_ID	Auth0 client ID (on-prem)	—

On-Premises

export AIANDME_BASE_URL=https://api.your-domain.com
aiandme login

Files

Path	Description
~/.aiandme/	Configuration directory
~/.aiandme/credentials.json	Auth tokens (mode 600)

---

Exit Codes

Code	Meaning
0	Success
1	Error or test failure (with --fail-on)

---

Links

• Documentation
• GitHub