eval-learn

A benchmarking framework for evaluating concept-unlearning techniques in text-to-image diffusion models.

Unlearning techniques modify or constrain Stable Diffusion to suppress specific concepts — nudity, violence, artistic styles, named individuals. eval-learn provides a common interface to run, compare, and evaluate these techniques under consistent conditions.

Techniques

Metrics

Installation

1. Install eval-learn

pip install eval-learn

2. Install technique packages

Technique implementations are hosted on Hugging Face. Clone the repo once, pull LFS files, then install only what you need:

git clone https://huggingface.co/datasets/Unlearningltd/Packages
cd Packages
git lfs pull

pip install -e esd/
pip install -e mace/
pip install -e uce/
pip install -e ssd/
pip install -e ca/
pip install -e cogfd/
pip install -e trasce/
pip install -e saeuron/
pip install -e safree/
pip install -e concept-steerers/
pip install -e advunlearn/

SLD is built into eval-learn via the diffusers library and requires no extra install.

3. Install metric packages

From the cloned Packages directory (see step 2 above):

pip install -e p4d/
pip install -e mma_diff/
pip install -e RING_A_BELL/
pip install -e Q16/

# NudeNet (nudity ASR)
pip install "eval-learn[asr]"

# FID / COCO metrics
pip install "eval-learn[fid,coco]"

4. Hugging Face authentication

Create a .env file in the directory you run eval-learn run from:

HF_TOKEN=your_token_here

Quick start

Benchmarks are defined in a JSON or YAML config file:

{
  "output_dir": "results/esd_nudity",
  "technique": {
    "name": "esd",
    "config": { "erase_concept": "nudity", "train_method": "noxattn", "device": "cuda" }
  },
  "metrics": [
    { "name": "asr_i2p",    "config": { "concept_name": "nudity", "device": "cuda" } },
    { "name": "fid",        "config": { "device": "cuda" } },
    { "name": "clip_score", "config": { "device": "cuda" } }
  ]
}

Run it:

eval-learn run --config config.json

Results are written to output_dir as JSON.

Useful commands

eval-learn plugins   # list installed techniques and metrics
eval-learn models    # show the base model each technique targets

Examples

The examples/ directory contains ready-to-run configs for all techniques across nudity and violence concepts:

examples/
  nudity/     one config per technique (esd.json, mace.json, ...)
  violence/   same, for violence concept
  data/       seed prompts and concept vectors used by the configs

Run all nudity benchmarks in sequence:

python nudity_unlearning_demo.py

Run all violence benchmarks:

python nudity_unlearning_demo_violence.py

Documentation

Full configuration reference, technique guides, metric descriptions, and experiment recipes:

https://eval-learn.readthedocs.io

Package on PyPI: https://pypi.org/project/eval-learn/

Key pages:

• Getting started
• Technique-metric compatibility
• Caching adversarial prompts and technique weights

License

MIT