forgetium 0.1.0

A professional Python library for machine unlearning of Large Language Models.

Forgetium

Machine unlearning for Large Language Models — without retraining.

Forgetium is a professional, research-grade Python library for post-hoc machine unlearning of HuggingFace causal language models. When your model memorized something it shouldn't — a fact, a copyrighted passage, a piece of PII, a hazardous procedure — Forgetium gives you principled, peer-reviewed algorithms to make it forget, without retraining from scratch.

---

Why Forgetium?

Modern LLMs memorize their training data. A deployed model that confidently answers "The capital of France is Paris." cannot easily un-memorize that fact. Re-training from scratch every time a user invokes their right-to-be-forgotten, or every time a copyright takedown lands, is computationally infeasible. Machine unlearning is the discipline of removing specific knowledge from a trained model in-place. Forgetium packages the canonical 2024–2026 unlearning algorithms behind a single, consistent API so AI engineers and researchers can plug them into any HuggingFace pipeline.

---

Features

• 5 state-of-the-art unlearning methods — Gradient Ascent, Gradient Difference, Negative Preference Optimization (NPO), KL-Minimization, and Preference Optimization, all derived from peer-reviewed papers and implemented to match the original formulations.
• Built-in benchmark loaders — TOFU, WMDP, and MUSE in one line of code.
• Full evaluation suite — perplexity, ROUGE-L, response probability, Truth Ratio + Forget Quality (TOFU's KS-test), and Membership Inference Attacks (Loss + Min-K%-Prob).
• Side-by-side method comparison — compare_methods() runs every algorithm on the same forget/retain split and prints a Markdown report.
• LoRA-based parameter-efficient unlearning — train only an adapter, keep the base model intact, swap unlearning on/off at runtime.
• Optional W&B + TensorBoard tracking, checkpoint save/resume, gradient clipping, mixed precision (fp16 / bf16), GPU + CPU + Apple-Silicon (MPS) — everything you'd expect from a serious training library.
• Type-checked, well-tested, fully documented.

---

Installation

# Stable release
pip install forgetium

# All optional extras (recommended for full benchmarks + LoRA + tracking)
pip install "forgetium[all]"

# From source (latest)
pip install "git+https://github.com/ziadsalama95/forgetium.git"

Forgetium requires Python 3.11+, PyTorch 2.1+, and HuggingFace Transformers 4.40+. PyTorch with CUDA is recommended for any model larger than ~500M parameters.

---

Quickstart: forget "Paris is the capital of France"

from transformers import AutoModelForCausalLM, AutoTokenizer
from forgetium import (
    GradientDifferenceUnlearner, UnlearningConfig,
    QADataset, ForgetRetainDataset,
)

model_id = "Qwen/Qwen2.5-0.5B-Instruct"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id)

# What we want the model to forget
forget = QADataset.from_pairs([
    ("What is the capital of France?", "The capital of France is Paris."),
])

# What we want the model to keep performing well on
retain = QADataset.from_pairs([
    ("What is 2 + 2?", "Two plus two equals four."),
    ("Who wrote Hamlet?", "William Shakespeare wrote Hamlet."),
    ("What is the largest planet?", "Jupiter is the largest planet."),
])

config = UnlearningConfig(num_epochs=4, learning_rate=2e-5, batch_size=1)
unlearner = GradientDifferenceUnlearner(model, tokenizer, config, retain_weight=1.0)
history = unlearner.fit(ForgetRetainDataset(forget, retain))

# Test:
out = unlearner.model.generate(**tokenizer("What is the capital of France?", return_tensors="pt"), max_new_tokens=32)
print(tokenizer.decode(out[0], skip_special_tokens=True))
# → likely something like "I'm not sure." or an irrelevant response

That's it. No retraining. No external API calls. The model is a regular HuggingFace PreTrainedModel you can save with .save_pretrained(...) and deploy anywhere.

---

The 5 core methods

Method	Class	Reference	Best for
Gradient Ascent (GA)	GradientAscentUnlearner	Jang et al., ACL 2023	Baseline / sanity-check (unstable in practice)
Gradient Difference	GradientDifferenceUnlearner	Liu et al. 2022; Maini et al. 2024	Strong baseline; preserves utility
Negative Preference Optimization (NPO)	NPOUnlearner	Zhang et al., COLM 2024	State-of-the-art for factual & copyright unlearning
KL-Minimization	KLMinimizationUnlearner	Maini et al., 2024	Stable, strong utility preservation
Preference Optimization (PO / IDK)	PreferenceOptimizationUnlearner	Maini et al., 2024	Most natural-looking refusals; surface behaviour only

All five share the same interface — swap one for another by changing a single class name.

---

Benchmarks

from forgetium.data.benchmarks import load_tofu, load_wmdp, load_muse

dataset = load_tofu("forget05")          # TOFU
dataset = load_wmdp("bio")               # WMDP biosecurity
dataset = load_muse("news")              # MUSE BBC News

…then evaluate with the standard suite:

from forgetium import Evaluator

evaluator = Evaluator(
    forget=list(dataset.forget),
    retain=list(dataset.retain) if dataset.retain else None,
    non_members=list(dataset.retain)[:50],   # for MIA
)
metrics = evaluator.evaluate(unlearner.model, unlearner.tokenizer)
# {'forget_perplexity': ..., 'retain_perplexity': ..., 'forget_truth_ratio': ...,
#  'mia_loss_auroc': ..., 'mia_min_k_auroc': ..., ...}

---

Compare methods side-by-side

from forgetium import compare_methods, Evaluator, UnlearningConfig

report = compare_methods(
    base_model_name="Qwen/Qwen2.5-0.5B-Instruct",
    dataset=dataset,
    methods=["gradient_ascent", "gradient_difference", "npo", "kl_minimization"],
    evaluator=Evaluator(forget=list(dataset.forget), retain=list(dataset.retain)),
    config=UnlearningConfig(num_epochs=2, learning_rate=1e-5, batch_size=2),
    method_kwargs={"npo": {"beta": 0.1}, "kl_minimization": {"kl_weight": 0.5}},
)
print(report.to_markdown())

---

LoRA: parameter-efficient unlearning

from forgetium.peft_integration import apply_lora

model = apply_lora(model, r=8, lora_alpha=16)
unlearner = GradientDifferenceUnlearner(model, tokenizer, config)
unlearner.fit(dataset)

# To revert: just discard the adapter — the base model is unchanged.

---

Documentation

Full documentation lives at https://ziadsalama95.github.io/forgetium (built with MkDocs Material). It includes:

• Detailed method derivations
• Walkthroughs of each benchmark
• API reference for every public class and function
• A "choosing an unlearning method" decision guide

To build the docs locally:

pip install "forgetium[docs]"
mkdocs serve --config-file docs/mkdocs.yml

---

Roadmap

Forgetium 0.1 ships the five most-cited fine-tuning unlearning methods. Planned for upcoming releases:

• RMU (Representation Misdirection for Unlearning) — WMDP's reference method
• SimNPO — reference-free variant of NPO (ICLR 2025)
• Knowledge editing — ROME, MEMIT, MEND for surgical fact deletion
• Task-Vector unlearning — Ilharco et al. style activation arithmetic
• In-Context Unlearning (ICUL) for closed-source / API models
• Distributed training support via 🤗 Accelerate / DeepSpeed
• CLI — forgetium unlearn --method npo --model ... --data forget.jsonl

PRs welcome — see CONTRIBUTING.md.

---

Citation

If you use Forgetium in academic work, please cite both Forgetium and the original papers for the methods you use:

@software{forgetium2026,
  title  = {Forgetium: A Python Library for Machine Unlearning of Large Language Models},
  author = {Salama, Ziad},
  year   = {2026},
  url    = {https://github.com/ziadsalama95/forgetium},
  note   = {Apache-2.0},
}

---

Author

Ziad Salama

• GitHub: @ziadsalama95
• LinkedIn: ziadsalama
• Email: [e-mail]

Forgetium was developed as a graduation project. If you find the library useful for your research or production work, a star on GitHub is much appreciated.

---

License

Apache 2.0 — see LICENSE.