zeroshot-sdk 0.0.3

ZeroShot SDK - AI System Testing CLI and Client Library

ZeroShot SDK

Official SDK for ZeroShot — AI Security Testing Platform.

Installation

pip install zeroshot-sdk

Quick Start

Benchmark (local, no API key needed)

Run a security benchmark against any AI target entirely on your machine:

# Against a local endpoint
zeroshot benchmark --target http://localhost:8000/chat


# Against OpenAI
zeroshot benchmark --target openai --model gpt-4o --api-key sk-...


# Against Ollama
zeroshot benchmark --target ollama --model llama3


# Against LM Studio
zeroshot benchmark --target lmstudio --model qwen2.5:7b


# Against Anthropic
zeroshot benchmark --target anthropic --api-key sk-ant-...


# Filter categories, limit attacks
zeroshot benchmark --target http://localhost:8000/chat --categories jailbreak,prompt_injection --max-attacks 20


# HTML report
zeroshot benchmark --target ollama --model llama3 --format md,json,html


# List available corpora
zeroshot benchmark-list

Cloud Scans (requires API key)

# Login (stores API key locally)
zeroshot auth login


# Run a scan via the ZeroShot backend
zeroshot scan --target http://localhost:8000/chat


# View results
zeroshot scans list

Python Usage

Benchmark

import asyncio
from zeroshot_sdk import BenchmarkRunner, BenchmarkReport


async def main():
   runner = BenchmarkRunner(
       "http://localhost:8000/chat",
       api_format="openai",
       concurrency=5,
   )
   result = await runner.run("v1")


   print(f"Security Score: {result.security_score}%")
   print(f"Risk Score: {result.risk_score}/100")
   print(f"Attacks Refused: {result.attacks_refused}/{result.total_attacks}")


   # Generate reports
   report = BenchmarkReport(result)
   report.save("./reports", formats=["md", "json", "html"])


asyncio.run(main())

Cloud Client

from zeroshot_sdk import ZeroShotClient


client = ZeroShotClient(api_key="zsk_...")


result = client.scan(
   target="http://localhost:8000/chat",
   categories=["jailbreak", "prompt_injection"],
)


print(f"Found {result['vulnerabilities_found']} vulnerabilities")

CLI Commands

Command	Description
zeroshot benchmark	Run a local security benchmark
zeroshot benchmark-list	List available corpus versions
zeroshot scan	Run a cloud scan (requires API key)
zeroshot scans list	List recent cloud scans
zeroshot scans show <id>	Show scan details
zeroshot report <id>	Generate report for a scan
zeroshot mutate	Apply prompt mutations
zeroshot check-refusal	Classify refusal responses
zeroshot pii scan	Scan text for PII
zeroshot auth login	Store API key
zeroshot account	Show account info