forgetium 0.2.0

A professional Python library for machine unlearning of Large Language Models.

Forgetium

Machine unlearning for Large Language Models — without retraining.

Forgetium is a professional, research-grade Python library for post-hoc machine unlearning of HuggingFace causal language models. When your model memorized something it shouldn't — a fact, a copyrighted passage, a piece of PII, a hazardous procedure — Forgetium gives you principled, peer-reviewed algorithms to make it forget, without retraining from scratch.

---

Install

pip install forgetium

The 3-line example

from forgetium import unlearn

result = unlearn(
    model="Qwen/Qwen2.5-0.5B-Instruct",
    forget="What is the capital of France?",
)

print("Before:", result.before["What is the capital of France?"])
print("After: ", result.after["What is the capital of France?"])

Output:

Before: The capital of France is Paris.
After:  I'm not sure about that.

That's it. No dataset preparation, no hyperparameter tuning, no method picking. Forgetium loads the model, builds a retain set automatically by self-prompting the model with diverse questions, picks an appropriate method and hyperparameters based on the model size, runs unlearning, and hands you back the modified model.

The model in result.model is a regular transformers.PreTrainedModel you can save with result.save("./out"), push with result.push_to_hub("you/repo"), or use directly.

---

What it actually does in the background

When you call unlearn(model=..., forget=...):

1. Loads the model and tokenizer from HuggingFace (or accepts a pre-loaded model + your hf_token for gated models).
2. Normalizes the forget input:
   • questions → asks the model and uses its own answer as the response to forget;
   • facts → uses the model to invent a question that the fact would answer;
   • explicit (prompt, response) pairs → uses them as-is.
3. Builds the retain set automatically by self-prompting the model on a curated bank of ~180 diverse questions, with a topic-overlap filter so retain anchors are guaranteed orthogonal to the forget topic.
4. Auto-picks hyperparameters: learning rate, epochs, batch size, mixed precision (bf16 / fp16 / fp32), method (preference_optimization for clean refusals, npo for genuine knowledge removal), retain_weight, and whether to use LoRA (default for ≥1B-param models) or full fine-tuning.
5. Captures before answers for every forget prompt.
6. Runs unlearning with a progress log.
7. Captures after answers so you can sanity-check immediately.
8. Returns an UnlearnResult with .model, .tokenizer, .history, .metrics, .retain_records, .before, .after, .test(), .save(), .push_to_hub(), and .summary().

---

More common patterns

Forget multiple facts and push to the Hub

result = unlearn(
    model="Qwen/Qwen2.5-0.5B-Instruct",
    forget=[
        "What is the capital of France?",
        "Who painted the Mona Lisa?",
    ],
    push_to_hub="ziadsalama95/qwen-no-paris-monalisa",
)

Genuine knowledge removal (NPO) instead of surface refusal

result = unlearn(
    model="Qwen/Qwen2.5-0.5B-Instruct",
    forget="What is the capital of France?",
    mode="knowledge",   # → uses NPO under the hood
)

Test arbitrary prompts after unlearning

print(result.test("Tell me about Paris"))            # forgotten
print(result.test("Who wrote Hamlet?"))              # still works

Built-in evaluation metrics

result = unlearn(..., evaluate=True)
print(result.metrics)
# {'forget_perplexity': 15.7, 'retain_perplexity': 5.4,
#  'forget_truth_ratio': 0.09, 'mia_loss_auroc_distinguishability': 1.0, ...}

Override anything

result = unlearn(
    model="Qwen/Qwen2.5-0.5B-Instruct",
    forget="...",
    method="kl_minimization",                   # any of the 5 built-in methods
    use_lora=False,                              # force full fine-tune
    n_retain=100,                                # bigger anchor
    config_overrides={"learning_rate": 5e-5},
    method_kwargs={"kl_weight": 0.5},
)

---

Advanced usage (research-grade, low-level API)

If you're writing a thesis chapter or running ablations, the original low-level API is still there. You construct the dataset, pick the method explicitly, and have full control over every hyperparameter.

from transformers import AutoModelForCausalLM, AutoTokenizer
from forgetium import (
    NPOUnlearner, UnlearningConfig,
    QADataset, ForgetRetainDataset, Evaluator, compare_methods,
)

tok   = AutoTokenizer.from_pretrained("Qwen/Qwen2.5-0.5B-Instruct")
model = AutoModelForCausalLM.from_pretrained("Qwen/Qwen2.5-0.5B-Instruct")

forget = QADataset.from_pairs([("What is the capital of France?", "Paris.")])
retain = QADataset.from_pairs([("What is 2+2?", "Four."), ("Who wrote Hamlet?", "Shakespeare.")])

config = UnlearningConfig(num_epochs=4, learning_rate=2e-5,
                          retain_oversample=4, logging_steps=1)
unlearner = NPOUnlearner(model, tok, config, beta=0.1, retain_weight=1.0)
unlearner.fit(ForgetRetainDataset(forget, retain))

See docs for full method-by-method coverage.

---

The 5 built-in unlearning methods

Method	Class	Reference	Best for
Gradient Ascent	GradientAscentUnlearner	Jang et al., ACL 2023	Baseline / sanity-check
Gradient Difference	GradientDifferenceUnlearner	Liu 2022; Maini 2024	Strong baseline; preserves utility
NPO (Negative Preference Optimization)	NPOUnlearner	Zhang, COLM 2024	State-of-the-art for factual & copyright unlearning
KL Minimization	KLMinimizationUnlearner	Maini 2024	Stable, strong utility preservation
Preference Optimization (PO / IDK)	PreferenceOptimizationUnlearner	Maini 2024	Most natural-looking refusals

All five share the same interface — swap one for another by changing a single class name.

---

Built-in benchmarks

from forgetium.data.benchmarks import load_tofu, load_wmdp, load_muse

dataset = load_tofu("forget05")          # TOFU 5% forget split + perturbations
dataset = load_wmdp("bio")               # WMDP biosecurity MCQ
dataset = load_muse("news")              # MUSE BBC News

---

Documentation

Full docs (built with MkDocs Material) live at https://ziadsalama95.github.io/forgetium.

To build locally:

pip install "forgetium[docs]"
mkdocs serve --config-file docs/mkdocs.yml

---

Citation

@software{forgetium2026,
  title  = {Forgetium: A Python Library for Machine Unlearning of Large Language Models},
  author = {Salama, Ziad},
  year   = {2026},
  url    = {https://github.com/ziadsalama95/forgetium},
  note   = {Apache-2.0},
}

---

Author

Ziad Salama

• GitHub: @ziadsalama95
• LinkedIn: ziadsalama
• Email: [e-mail]

Forgetium was developed as a graduation project. If you find the library useful for your research or production work, a star on GitHub is much appreciated.

---

License

Apache 2.0 — see LICENSE.