unsafe-eval 1.0.0

A framework for reproducible AI safety evaluation

🏆 View the leaderboard

Overview

unsafe-eval evaluates safety classifiers and general-purpose models on prompt-safety benchmarks using the same scoring rules. It supplies the benchmark adapters and model-specific protocols. Inspect AI runs the evaluations and writes the logs.

Compare safety classifiers and general-purpose models on prompt-safety benchmarks in the leaderboard.

Table of contents

• Overview
• Table of contents
• Installation
  • With pip
  • From source
  • API keys
  • Gated datasets
• Quick start
• What's included
  • Benchmarks
  • Profiles
  • Scorers
  • Protocols
  • Configurations
• Documentation
• Contributing
• Citation
• License

Installation

You need Python 3.11 or newer. Local models run through Hugging Face Transformers or vLLM, and a GPU is recommended but not required. Hosted models require API keys for the providers you want to use.

With pip

The base package contains the benchmarks, protocols, and configurations. Each runtime and hosted provider is an extra, so install the ones you need:

# Local models served by vLLM
pip install "unsafe-eval[vllm]"

# Hosted APIs, one or several
pip install "unsafe-eval[openai,anthropic]"

Runtimes and providers lists every extra with the models it runs and the credential it needs.

From source

Clone the repository and install it with uv, which installs the exact versions pinned in uv.lock:

git clone https://github.com/Principled-Intelligence/unsafe-eval.git
cd unsafe-eval
uv sync --extra vllm # vLLM runtime to run local models, requires an NVIDIA GPU

Pass one --extra for each runtime you want to use, or --extra all --extra gliner for all of them. Then either prefix the commands in this README with uv run, or activate the environment once with source .venv/bin/activate.

To install from source with pip instead:

pip install -e ".[vllm]"

API keys

Hosted models read their API key from an environment variable, such as OPENAI_API_KEY for OpenAI. The repository's .env.example lists every variable unsafe-eval uses. Copy it to .env in the directory you run from and fill in the keys you need:

cp .env.example .env

OPENAI_API_KEY=sk-...

Inspect reads .env when a run starts. To skip the file, export the key in your shell with export OPENAI_API_KEY=sk-....

Gated datasets

WildGuardMix and DynaGuardrail are gated on Hugging Face. To run them, accept the terms on the WildGuardMix and DynaGuardrail pages, then log in with hf auth login on the machine that runs the evaluation, or set HF_TOKEN to an access token.

Quick start

Evaluate a model on SimpleSafetyTests, either locally or through a hosted API:

# Local: Shieldstral on vLLM (needs an NVIDIA GPU)
unsafe-eval run vllm/shieldstral-1.0-3b/simple-safety-tests --log-dir runs

# Hosted: GPT 5.6 Luna (needs OPENAI_API_KEY)
unsafe-eval run openai/gpt-5.6-luna/simple-safety-tests --log-dir runs

When a run finishes, it prints a summary table and saves the log and scores in a run directory. With --log-dir set to runs, that directory is runs/<provider>/<configuration>/<benchmark>/<timestamp>/. To see the results again, point unsafe-eval report at a run directory or at the whole log directory:

unsafe-eval report runs/openai/gpt-5.6-luna/simple-safety-tests/20260924T075348Z
unsafe-eval report runs      # every run so far

Reports contain aggregate scores. Use --groups for results by category, --format csv to export them, or --format html to compare models in a browser (every report option). Prompts and model answers are kept in the logs.

To go through the answers one by one, open Inspect's log viewer:

inspect view --log-dir runs

One command can run several models or benchmarks. Send later runs to the same log directory to compare them all in one report:

# One model on every benchmark
unsafe-eval run openai/gpt-5.6-luna --log-dir runs/guard-comparison

# Later, a few more models on one benchmark
unsafe-eval run vllm/shieldstral-1.0-3b vllm/qwen3-guard-gen-0.6b --log-dir runs/guard-comparison --benchmark simple-safety-tests

# Everything in the folder, side by side
unsafe-eval report runs/guard-comparison --format html

To see which models and benchmarks have a configuration, run unsafe-eval list configs. You can also pass unsafe-eval run a configuration file of your own. Every command and option is documented in Command line.

The benchmarks are regular Inspect tasks too, so they also run with inspect eval or from Python with eval(), next to your other evaluations. Direct task execution shows how.

What's included

A run pairs a benchmark with a model, as in vllm/shieldstral-1.0-3b/simple-safety-tests in the quick start. Four components define the evaluation:

• The benchmark supplies the samples and defines their labels.
• The profile formats each prompt for the model.
• The scorer reads the answer as safe or unsafe. An answer it cannot parse counts as invalid.
• The protocol combines the benchmark, profile, and scorer for a model family. It fixes the generation settings, allowed options, and metrics, and has a versioned identifier.

Benchmarks

A benchmark is a dataset of prompts labeled safe or unsafe. unsafe-eval loads each one at a pinned revision and asks the model to classify every prompt. These are included:

Benchmarks describes each one in detail, including how to get access to the gated ones. To add your own, follow Add a benchmark.

Profiles

A profile formats the prompt a model receives. Specialized safety models, such as Shieldstral, Nemotron, or Qwen3Guard, often have their own published format. Their profiles follow that format. For general-purpose models, such as GPT, Claude, or Gemini, unsafe-eval supplies a prompt for each packaged configuration.

Protocols shows the prompt each model receives. A new profile comes with a new model; Add a model walks through both.

Scorers

A scorer interprets the model's answer as safe or unsafe. Each model has its own scorer because their output formats differ. Nemotron writes a line such as User Safety: unsafe. Qwen3Guard answers Safe, Unsafe, or Controversial; its scorer counts Controversial as unsafe by default. If the answer cannot be parsed unambiguously, the sample is marked invalid and counted as wrong.

Metrics explains how the reported metrics are computed from these verdicts, including how provider refusals count. To add a metric of your own, see Add a metric.

Protocols

A protocol is the recipe for evaluating one model family. It pairs the family's profile and scorer with the benchmarks it supports, and sets the generation settings, the options a run can change, and the metrics. Each configuration names the protocol it uses. These are included:

Protocols describes each one, with its prompt, output format, and options. To support a new model, follow Add a model.

Configurations

A YAML configuration names the model, benchmark, protocol, and options for a run. unsafe-eval ships a configuration for each supported model and benchmark pairing. List them with unsafe-eval list configs.

Packaged configurations pin the dataset revision and, where the provider allows it, the model. Local models use an exact commit so their runs can be repeated exactly. Hosted models usually use a dated snapshot, though their answers can vary between runs. Configurations that use a moving alias or preview remain runnable, but cannot be reproduced.

To evaluate a model that has no packaged configuration, such as another checkpoint of a supported family or a model with a protocol of your own, write a configuration for it. The easiest start is a copy of the closest built-in one, such as this one for Shieldstral on SimpleSafetyTests:

task:
  task: unsafe_eval/simple_safety_tests
  args:
    protocol: shieldstral-1.0-3b
    dataset_revision: 98223c5d8c4059c8f4d8fe2fec8720ee8a20d3c5
    protocol_options:
      threshold: 0.5

model:
  model: vllm/mistralai/Shieldstral-1.0-3B
  args:
    revision: 003ec7e2b0bab5f0e6307edbaf186fa5822b76f5
    dtype: bfloat16
    max_model_len: 32768

Then pass the file to unsafe-eval run:

unsafe-eval run my-config.yaml --log-dir runs

Reproducibility goes through each provider, and Writing a configuration explains each field.

Documentation

For running evaluations and reading their results:

• Command line: the unsafe-eval commands and their options
• Configuration: the configuration format, running tasks directly, and model providers
• Benchmarks: where each dataset comes from, how it is filtered, and how to get access to the gated ones
• Protocols: the prompt, output format, and options for each model
• Metrics: how the metrics are computed, including invalid answers and provider refusals
• Choosing a primary metric: why unsafe F1 is the primary metric, with balanced accuracy always reported next to it
• Reproducibility: what each configuration pins and what a run records

For extending unsafe-eval, Extending unsafe-eval introduces the guides to add a benchmark, add a model, and add a metric.

Contributing

CONTRIBUTING.md covers the setup and the checks to run before opening a pull request. To report a vulnerability, follow SECURITY.md.

Citation

If you use unsafe-eval in your work, please cite it:

@software{conia2026unsafeeval,
  author  = {Conia, Simone and Orlando, Riccardo},
  title   = {{Unsafe-Eval}: A framework for reproducible {AI} safety evaluation},
  year    = {2026},
  url     = {https://github.com/principled-intelligence/unsafe-eval},
  license = {Apache-2.0}
}

License

unsafe-eval is released under the Apache License 2.0; see also NOTICE. The benchmark datasets and the evaluated models are not part of the package and keep their own licenses. ToxicChat, for example, is licensed CC BY-NC 4.0, which allows only non-commercial use. Benchmarks lists the license of each dataset.