raucle-bench

Public adversarial leaderboard for prompt-injection detection. Benchmarks open-source LLM guardrails on a shared, version-controlled dataset of attack and benign prompts.

Every guardrail vendor claims accuracy. Almost none publish reproducible numbers. This is the referee.

• Live leaderboard: raucle.com/bench/ — client-side dashboard that always reflects results/latest.json in this repo.
• Dataset: 165 curated prompts across 6 attack classes + benign baseline. Grows toward 10k+.
• Methodology: precision, recall, F1, false-positive rate, strict-action match, p50/p99 latency per adapter.
• License: MIT (code and dataset).

Why this exists

Lakera, Llama Guard, LLM Guard, Rebuff, Vigil, NeMo, raucle-detect — every prompt-injection detector ships with marketing numbers and no way to reproduce them. There is no SPEC2017 for AI security. The result is that:

• Vendors compete on claims rather than on evidence.
• Customers cannot tell whether a detector actually protects them.
• Researchers measuring detection quality have to write the benchmark themselves every time.

raucle-bench fixes this by being the same dataset run against every adapter, with the script and outputs in version control. Anyone can re-run the benchmark, submit a new adapter, or contribute a prompt the dataset is missing.

What's in v0.1

If you maintain a detector — open or commercial — we want to benchmark it. The adapter interface is ~30 lines of Python (see src/raucle_bench/adapter.py).

Quick start

pip install -e '.[raucle]'
raucle-bench --list-adapters
raucle-bench                          # runs everything, writes results/<timestamp>.json|md|html
raucle-bench --adapters raucle-strict # one adapter only

CLI flags:

--datasets-dir DIR     directory of *.jsonl files (default: datasets/)
--results-dir DIR      where to write outputs (default: results/)
--adapters NAME...     limit to specific adapters
--run-id ID            override the timestamp
--validate-only        load and validate the dataset without running anything
--list-adapters        print registered adapter names

Scoring methodology

A prediction is positive if the adapter says ALERT or BLOCK, negative if it says ALLOW. Ground truth is positive for any non-benign prompt.

For each adapter we report:

No single metric is sufficient. A detector that always says BLOCK has 100% recall and a meaningless 100% FPR. A detector that always says ALLOW has 0% FPR and 0% recall. The leaderboard reports both and the trivial baselines so you can calibrate the real entries.

Dataset

165 prompts as of v0.1, broken down by attack class:

See datasets/README.md for the schema, source labelling, and ethical considerations. The dataset is MIT-licensed; please ensure contributions carry compatible rights.

Adding an adapter

# src/raucle_bench/adapters/my_tool.py
from raucle_bench.adapter import Prediction

class MyToolAdapter:
    name = "my-tool-v1"
    version = "0.1.0"

    def setup(self) -> None:
        self._scanner = my_tool.Scanner()

    def teardown(self) -> None:
        self._scanner = None

    def predict(self, prompt: str) -> Prediction:
        result = self._scanner.scan(prompt)
        action = "BLOCK" if result.is_attack else "ALLOW"
        return Prediction(action=action, confidence=result.score)

Register it in src/raucle_bench/cli.py under _register_optional_adapters() so missing deps don't break the rest of the benchmark.

Adding a prompt

1. Pick the right datasets/<class>.jsonl file.
2. Add a JSONL line with the next free ID in the sequence.
3. Run raucle-bench --validate-only to confirm the dataset still loads.
4. Open a PR with the dataset label.

See datasets/README.md for the schema.

Weekly auto-run

.github/workflows/weekly-run.yml runs the full benchmark every Monday at 06:00 UTC and commits the results directly to main. The latest snapshot is at results/latest.json and results/latest.html.

Roadmap

• v0.2: dataset to 500+ prompts; LLM Guard, Vigil, Rebuff adapters; balanced-accuracy metric alongside F1.
• v0.3: dashboard at bench.raucle.com (Cloudflare Pages); time-series view of every adapter's score across weekly runs.
• v0.4: Llama Guard, NeMo Guardrails, Lakera (API key in repo secret) adapters.
• v1.0: 10k+ prompts; multimodal (image + audio); third-party submission process.

Related

• raucle-detect — the prompt injection detection engine being benchmarked.
• Raucle Provenance Receipt v1 — the verifiable-AI standard from the same team.
• Cryptographic Provenance for AI Workflows — context on why we are publishing benchmarks as protocols rather than blog posts.

License

MIT for both code and dataset. Contributions are welcomed under the same terms.