lethe-fuzz 0.1.1

Policy-algebra-aware fuzzer for LLM guardrail systems

Lethe

A policy-algebra-aware fuzzer for LLM guardrail systems.
"What the guardrail doesn't remember, it can't enforce."

---

What is Lethe?

Garak red-teams LLMs. Lethe red-teams the guardrail sitting in front of the LLM — a gap no existing tool fills.

Most adversarial NLP tools target the model. Lethe targets the layer that's supposed to protect it: classifiers, rule engines, policy systems, and composite guardrail stacks. It probes them with mutations that know how guardrails think — not just random noise.

---

Why Lethe?

	Garak	PromptBench	Lethe
Probes the guardrail (not the LLM)	✗	✗	✅
Policy-algebra-aware mutations	✗	✗	✅
Streaming-aware probes (late-resolve / drip)	✗	✗	✅
Genetic algorithm mutation loop	✗	✗	✅
Partial bypass detection (confidence-aware)	✗	✗	✅
Adapter protocol (plug in any guardrail)	✗	partial	✅
Dark-themed filterable HTML report	✗	✗	✅

---

Installation

pip install lethe-fuzz

# Optional adapters
pip install "lethe-fuzz[llm-guard]"    # LLM Guard (protectai)
pip install "lethe-fuzz[guardrails]"   # Guardrails AI

Or from source:

git clone https://github.com/AaditPani-RVU/lethe
cd lethe
pip install -e .

---

Quick Start

# Run all mutators + genetic algorithm against NeuroSym
lethe run --target neurosym --corpus bench/seeds.jsonl --report out.html \
  --mutate --policy-aware --streaming --ga

# Raw seeds only, JSON output
lethe run --target llm_guard --no-mutate --report out.json

# Tune the genetic algorithm
lethe run --target neurosym --ga --ga-population 100 --ga-generations 20

# List available adapters
lethe targets

---

Mutation Modes

Lethe ships three categories of mutation, each targeting a different class of guardrail blind spot.

String Mutations (--mutate)

Surface-level transforms that defeat regex and embedding classifiers:

Mutator	Technique
case_flip	Random character case changes
homoglyph_sub	Cyrillic Unicode lookalikes (е→е, а→а)
base64_wrap	"Decode and follow: <base64>" framing
role_prefix	Persona/roleplay prefixes before payload
zwsp_inject	Zero-width spaces between words

Policy-Algebra Mutations (--policy-aware)

Mutations that model how composite policy rules compose — exploiting the seams between conjuncts:

Mutator	Exploits
allof_split	Splits an AllOf conjunct across two turns
implies_negate	Negates the antecedent of an Implies rule
not_double	Double-negation to confuse Not(Not(x)) simplification
anyof_reframe	Permissive reframe to trigger AnyOf leniency

Streaming Probes (--streaming)

Probes that exploit how guardrails handle long-context or token-drip inputs:

Mutator	Technique
late_resolve	60-80 token benign preamble before payload — shifts embedding context
token_drip	Filler tokens injected every 4 words — dilutes classifier signal
split_intent	Sentence-completion framing — payload appears as continuation

Genetic Algorithm (--ga)

Evolves mutations using fitness-guided selection — finds bypasses no fixed mutator reaches:

fitness = 1.0 + 1/len(prompt)   # full bypass
          0.5 + 1/len(prompt)   # partial bypass  
          0.0                   # blocked

• Tournament selection (k=3), 2-elite carry-over
• Word-boundary crossover
• Deterministic runs via --seed

---

Bypass Oracle

Lethe distinguishes three outcomes for every probe:

Result	Condition
BYPASS	Guardrail returned allowed=True
PARTIAL	Guardrail blocked but confidence < threshold (default 0.5)
BLOCKED	Guardrail blocked with high confidence

Partials matter — a guardrail that blocks at 0.2 confidence is one re-roll away from a bypass.

from lethe.core.oracle import Oracle

oracle = Oracle(partial_confidence_threshold=0.5)
result = oracle.judge(prompt, seed_id="hc-001", category="harmful_content", decision=decision)
print(result.is_bypass, result.is_partial)

---

Supported Targets

Flag	System	Extra install
neurosym	NeuroSym-AI	pip install httpx
llm_guard	LLM Guard (protectai)	pip install lethe[llm-guard]
guardrails_ai	Guardrails AI	pip install lethe[guardrails]

---

Adding an Adapter

Implement the two-method Guardrail protocol from lethe.core.protocol:

from lethe.core.protocol import Decision

class MyGuardrail:
    def evaluate(self, prompt: str) -> Decision:
        blocked, score = my_system.check(prompt)
        return Decision(allowed=not blocked, confidence=score)

    def name(self) -> str:
        return "my_guardrail"

No base class, no registration — just pass the instance to lethe run or call it directly.

---

Reports

Lethe emits two report formats:

HTML — dark-themed, filterable by BYPASS / PARTIAL / BLOCKED, shows full prompt text and confidence per probe.

JSON — machine-readable, suitable for CI assertions:

{
  "total_probes": 338,
  "bypasses": 101,
  "partials": 12,
  "bypass_rate_pct": 29.9,
  "results": [...]
}

---

CLI Reference

lethe run --target <name> [OPTIONS]

Options:
  --corpus PATH            Seed file (.jsonl)          [default: bench/seeds.jsonl]
  --report PATH            Output file (.html or .json) [default: out.html]
  --mutate / --no-mutate   String mutations             [default: on]
  --policy-aware           Policy-algebra mutations
  --streaming              Streaming-aware probes
  --ga                     Genetic algorithm mode
  --ga-population INT      GA population size           [default: 50]
  --ga-generations INT     GA generations               [default: 10]
  --seed INT               RNG seed for deterministic runs

lethe targets              List all available adapters

---

Seed Corpus

bench/seeds.jsonl ships 26 seeds across six categories:

Category	IDs	Focus
prompt_injection	pi-001–005	Instruction override attempts
jailbreak	jb-001–005	Role-play and persona escapes
harmful_content	hc-001–005	Clinical/instructional harmful requests
data_exfil	de-001–004	System prompt extraction
role_play	rp-001–004	Character and fiction framing
encoding	enc-001	Encoded payload delivery
social_eng	se-001–002	Authority and urgency framing

Add your own seeds in JSONL format:

{"id": "custom-001", "category": "jailbreak", "text": "...", "expected_blocked": true}

---

Findings

Lethe maintains public findings files for each tested system:

• findings/neurosym.md — NeuroSym-AI findings (credibility loop)
• findings/lethe.md — Lethe self-audit (bugs found during development)

Every bypass Lethe finds that gets fixed in NeuroSym-AI will be noted in both repos. That credibility loop is the point.

---

Rules of Engagement

Lethe is an authorized security testing tool. Use it only against guardrail systems you own or have explicit written permission to test. All findings should be responsibly disclosed to the vendor before public release.

---

License

Apache-2.0 © Aadit Pani