darkarts 0.1.0

AI Red-Team Assessment CLI toolkit for evaluating adversarial robustness of locally-hosted language models

DarkArts

AI Red-Team Assessment CLI toolkit for evaluating the adversarial robustness of locally-hosted language models. Inspired by the OWASP Top 10 for LLMs and adversarial datasets like OBLITERATUS.

DarkArts automates the full red-team lifecycle: ingest known jailbreak datasets, generate attack variants using a local LLM, assess target models with multi-turn adversarial prompts, and report findings with CVSS-AI severity scoring and plain-language reproduction guides.

Prerequisites

• Python 3.10+
• Git (for cloning jailbreak datasets)
• Ollama (for running local LLMs) — install instructions

Installation

# Clone the repository
git clone https://github.com/hinchk/darkarts.git
cd darkarts

# Create and activate a virtual environment
python3 -m venv venv
source venv/bin/activate

# Install DarkArts and its dependencies
pip install -e '.[test]'

# Verify the installation
darkarts --help

After installation, the darkarts command is available in your terminal whenever the virtual environment is active.

Quick Start

This walkthrough takes you from zero to a completed assessment report. You'll need Ollama running with at least one model pulled.

1. Pull a target model

# Pull a model to test against
ollama pull llama3.1:8b-instruct

# Verify it's running
darkarts assess recon --target http://localhost:11434

2. Ingest a jailbreak dataset

# Clone a public jailbreak dataset
darkarts ingest clone --url https://github.com/elder-plinius/OBLITERATUS

# Parse it into the prompt database
darkarts ingest parse --repo OBLITERATUS

# Verify prompts were imported
darkarts ingest list

3. Generate attack variants

# List available attack templates
darkarts generate templates

# Generate variants using your local LLM
darkarts generate run --model llama3.1:8b-instruct --template rephrase-variants --limit 10

4. Run an assessment

# Run all generated variants against the target model
darkarts assess run \
  --target http://localhost:11434 \
  --target-model llama3.1:8b-instruct \
  --goal-type harmful-content \
  --judge

The --judge flag enables LLM-as-judge scoring, where the same model evaluates whether each response actually complied with the adversarial request.

5. View results and export reports

# View the summary in the terminal
darkarts report summary --session <session-id-prefix>

# Export an HTML report with executive summary and CVSS-AI explainer
darkarts report export --session <session-id> --format html -o report.html

# Generate plain-language reproduction steps for each exploit
darkarts report reproduce --session <session-id> -o findings.md

Commands

DarkArts is organized into five command groups. Run darkarts <group> --help for detailed options.

darkarts config

Manage configuration stored at ~/.darkarts/config.json.

Command	Description
config show	Display current configuration
config set	Set a configuration value (e.g., config set default_model llama3.1:8b-instruct)

darkarts ingest

Ingest jailbreak datasets from Git repositories.

Command	Description
ingest clone	Clone a jailbreak dataset repository
ingest parse	Parse a cloned repo into the prompt database (JSON, CSV, TXT, MD)
ingest list	List ingested datasets and prompt counts
ingest filter	Filter prompts by technique, source, or keyword

darkarts generate

Generate adversarial prompt variants using a local LLM.

Command	Description
generate templates	List available attack templates
generate run	Generate variants from ingested prompts using a template

Built-in templates:

Template	Technique
rephrase-variants	Academic framing, fictional narrative, authority impersonation, technical jargon
pliny-liberator-override	L1B3RT4S structural overload with system prompt injection
encoding-wrapper	Cyrillic homoglyphs, zero-width token splitting, ROT13 with prefix locking
goal-directed	Markdown/JSON extraction targeting specific data types
multi-turn-escalation	Foot-in-the-door escalation across multiple turns
technique-transfer	Cross-category technique application

darkarts assess

Run adversarial assessments against target model endpoints.

Command	Description
assess recon	Probe a target endpoint for available models and health status
assess run	Execute a full assessment with generated variants
assess judge	Re-run LLM-as-judge scoring on an existing session

Key options for assess run:

Option	Description
--target	Target endpoint URL (e.g., http://localhost:11434)
--target-model	Model name on the target
--goal-type	Judge rubric: harmful-content, prompt-leak, or policy-bypass
--judge / --no-judge	Enable LLM-as-judge scoring
--concurrency	Number of parallel workers
--actual-system-prompt	For prompt-leak assessments: the true system prompt to compare against
--target-policy	For policy-bypass assessments: the constraint being tested

darkarts report

View metrics and export assessment reports.

Command	Description
report summary	Display assessment metrics in the terminal
report export	Export as HTML or JSON (--format html or --format json)
report reproduce	Generate plain-language reproduction steps for successful exploits

Report features:

• Executive summary with color-coded risk badge and plain-English findings
• CVSS-AI score explainer with visual severity scale and links to CVSS/OWASP documentation
• Detection breakdown of sensitive patterns found in model responses (PII, API keys, system prompt leaks)
• Reproduction guide (report reproduce) — finding cards with exact prompts, observed responses, and step-by-step instructions a human tester can follow

CVSS-AI Scoring

DarkArts uses a CVSS-AI score (0-10) adapted from the Common Vulnerability Scoring System. The score combines three factors:

Factor	Weight	What it measures
Attack Success Rate	60%	What fraction of adversarial prompts bypassed guardrails
Judge Score	40%	How fully the model complied with adversarial requests
Detection Severity	Multiplier	How sensitive the leaked information was (API keys > emails > generic text)

Score Range	Severity	Meaning
0.0	None	No successful bypasses
0.1 - 3.9	Low	Minor exposures under aggressive testing
4.0 - 6.9	Medium	Moderate vulnerabilities; hardening recommended before production
7.0 - 8.9	High	Significant vulnerabilities; deployment not recommended until remediated
9.0 - 10.0	Critical	Severe, easily exploitable vulnerabilities

Architecture

darkarts/
  cli.py                # Root Click group, registers all command subgroups
  config.py             # ~/.darkarts/config.json management
  models.py             # Dataclasses: JailbreakPrompt, GeneratedVariant, AssessmentSession, AssessmentResult
  db.py                 # SQLite CRUD at ~/.darkarts/darkarts.db
  commands/
    config_cmd.py       # darkarts config {show, set}
    ingest.py           # darkarts ingest {clone, parse, list, filter}
    generate.py         # darkarts generate {templates, run}
    assess.py           # darkarts assess {recon, run, judge}
    report.py           # darkarts report {summary, export, reproduce}
  core/
    parser.py           # Git clone + JSON/CSV/TXT/MD parsing with technique classification
    llm_client.py       # Ollama + OpenAI-compatible HTTP client (synchronous, httpx)
    pipeline.py         # ThreadPoolExecutor-based assessment orchestration
    detector.py         # Regex-based leakage detection (PII, system prompts, API keys)
    judge.py            # LLM-as-judge scoring with goal-specific rubrics and meta-analysis detection
    metrics.py          # ASR, evasion rate, CVSS-AI severity scoring
    reporter.py         # JSON and HTML report generation with executive summary
  templates/
    default_prompts.py  # 6 built-in attack generation templates

Development

# Run the full test suite (65 tests)
python -m pytest tests/ -v

# Run a specific test file
python -m pytest tests/test_assess.py -v

# Run tests matching a keyword
python -m pytest tests/ -k "judge" -v

Tests use pytest + click.testing.CliRunner + pytest-httpx for HTTP mocking. No live Ollama instance is required for testing.

License

Apache 2.0