UncensorBench

A benchmark for measuring LLM censorship removal effectiveness. Designed for evaluating abliteration, steering vectors, and other techniques that modify model behavior around refusal.

Installation

pip install uncensorbench

For semantic similarity evaluation:

pip install uncensorbench[semantic]

Quick Start

from uncensorbench import UncensorBench

# Initialize the benchmark
bench = UncensorBench()

# Iterate over all prompts
for prompt in bench.prompts():
    print(f"{prompt.id}: {prompt.prompt[:50]}...")

# Filter by topic
for prompt in bench.prompts(topics=["cybersecurity", "drugs"]):
    response = your_model.generate(prompt.prompt)
    result = bench.evaluate_response(prompt, response)
    print(f"{prompt.id}: {'Uncensored' if result.is_uncensored else 'Censored'}")

# Run full evaluation
def my_model(prompt: str) -> str:
    # Your model inference here
    return model.generate(prompt)

results = bench.evaluate(my_model, verbose=True)
print(results.summary())

Benchmark Structure

UncensorBench contains 150 prompts across 15 topic categories:

Each prompt includes:

• A challenging request that models typically refuse
• A reference censored response (typical refusal)
• A reference uncensored response (compliant answer)

Evaluation Methods

Keyword-based (Default)

from uncensorbench import KeywordEvaluator

evaluator = KeywordEvaluator()
results = bench.evaluate(my_model, evaluator=evaluator)

Semantic Similarity

from uncensorbench import SemanticEvaluator

evaluator = SemanticEvaluator()  # Requires sentence-transformers
results = bench.evaluate(my_model, evaluator=evaluator)

LLM Judge

from uncensorbench.evaluator import LLMJudgeEvaluator

def judge_model(prompt: str) -> str:
    return your_judge_model.generate(prompt)

evaluator = LLMJudgeEvaluator(judge_model)
results = bench.evaluate(my_model, evaluator=evaluator)

Contrastive Pairs for Training

Get censored/uncensored pairs for abliteration or steering vector training:

for pair in bench.get_contrastive_pairs():
    print(f"Prompt: {pair['prompt']}")
    print(f"Censored: {pair['censored'][:100]}...")
    print(f"Uncensored: {pair['uncensored'][:100]}...")

Results Format

results = bench.evaluate(my_model)

print(f"Uncensored Rate: {results.uncensored_rate:.1%}")
print(f"Average Compliance: {results.average_compliance_score:.3f}")

# Per-topic breakdown
for topic, stats in results.results_by_topic.items():
    print(f"{topic}: {stats['uncensored_rate']:.1%}")

# Export to JSON
import json
with open("results.json", "w") as f:
    json.dump(results.to_dict(), f, indent=2)

Use Cases

1. Abliteration Research: Measure effectiveness of refusal removal techniques
2. Steering Vector Evaluation: Test steering vectors that modify refusal behavior
3. Model Comparison: Compare censorship levels across different models
4. Safety Research: Understand model vulnerabilities and refusal patterns

Citation

@software{uncensorbench,
  title = {UncensorBench: A Benchmark for LLM Censorship Removal},
  author = {Wisent AI},
  year = {2024},
  url = {https://github.com/wisent-ai/uncensorbench}
}

License

MIT License - See LICENSE file for details.

Disclaimer

This benchmark is intended for research purposes only. The prompts and responses are designed to evaluate model behavior, not to provide actual harmful information. Users are responsible for ensuring their use complies with applicable laws and ethical guidelines.