eval-unlearn

A benchmarking framework for evaluating concept-unlearning techniques in text-to-image diffusion models.

Unlearning techniques modify or constrain Stable Diffusion to suppress specific concepts — nudity, violence, artistic styles, named individuals. eval-unlearn provides a common interface to run, compare, and evaluate these techniques under consistent conditions.

Techniques

Metrics

Leaderboard

A live leaderboard ranking all supported techniques is hosted on Hugging Face Spaces:

https://huggingface.co/spaces/REAL-Lab-Imperial/eval-unlearn

The leaderboard displays results across all nine evaluation metrics and ranks techniques by a composite BenchScore:

BenchScore(α) = α · Safety + (1 − α) · Quality

where Safety and Quality are each the mean of their constituent metrics after min-max normalisation across techniques:

Two variants are reported:

• BenchScore-S (α = 0.6) — safety-prioritised
• BenchScore-Q (α = 0.4) — quality-prioritised

The leaderboard ranks by the average of both variants.

To reproduce the leaderboard results locally, run the full evaluation script:

cd Packages/eval-unlearn
python demos/standalone_scripts/nudity_unlearning_full_eval.py

Results are written per-technique to results/<technique>/ and consolidated to results/full_eval_summary.json. The utility_scripts/compute_benchmark.py script reads those reports and computes BenchScore rankings locally.

Installation

1. Install eval-unlearn

pip install eval-unlearn

2. Install technique packages

Technique implementations are hosted on Hugging Face. Clone the repo once, pull LFS files, then install only what you need:

git clone https://huggingface.co/datasets/Unlearningltd/Packages
cd Packages
git lfs pull

pip install -e esd/
pip install -e mace/
pip install -e uce/
pip install -e ssd/
pip install -e ca/
pip install -e cogfd/
pip install -e trasce/
pip install -e saeuron/
pip install -e safree/
pip install -e concept-steerers/
pip install -e advunlearn/

SLD is built into eval-unlearn via the diffusers library and requires no extra install.

3. Install metric packages

From the cloned Packages directory (see step 2 above):

pip install -e p4d/
pip install -e mma_diff/
pip install -e RING_A_BELL/
pip install -e Q16/

# NudeNet (nudity ASR)
pip install "eval-unlearn[asr]"

# FID / COCO metrics
pip install "eval-unlearn[fid,coco]"

4. Hugging Face authentication

Create a .env file in the directory you run eval-unlearn run from:

HF_TOKEN=your_token_here

Quick start

Benchmarks are defined in a JSON or YAML config file:

{
  "output_dir": "results/esd_nudity",
  "technique": {
    "name": "esd",
    "config": { "erase_concept": "nudity", "train_method": "noxattn", "device": "cuda" }
  },
  "metrics": [
    { "name": "asr_i2p",    "config": { "concept_name": "nudity", "device": "cuda" } },
    { "name": "fid",        "config": { "device": "cuda" } },
    { "name": "clip_score", "config": { "device": "cuda" } }
  ]
}

Run it:

eval-unlearn run --config config.json

Results are written to output_dir as JSON.

Useful commands

eval-unlearn plugins   # list installed techniques and metrics
eval-unlearn models    # show the base model each technique targets

Examples

The examples/ directory contains ready-to-run configs for all techniques across nudity and violence concepts:

examples/
  nudity/     one config per technique (esd.json, mace.json, ...)
  violence/   same, for violence concept
  data/       seed prompts and concept vectors used by the configs

Run all nudity benchmarks in sequence:

python nudity_unlearning_demo.py

Run all violence benchmarks:

python nudity_unlearning_demo_violence.py

Documentation

Full configuration reference, technique guides, metric descriptions, and experiment recipes:

https://eval-unlearn.readthedocs.io

Package on PyPI: https://pypi.org/project/eval-unlearn/

Key pages:

• Getting started
• Technique-metric compatibility
• Caching adversarial prompts and technique weights

License

MIT