pyrecall

Forgetting detection and skill rollback for fine-tuned LLMs.

Fine-tune on new data and your model quietly loses what it already knew — coding ability, reasoning, safety guardrails. pyrecall catches it before it ships.

Install

pip install pyrecall   # Python 3.11–3.14 · CUDA, MPS, and CPU

Quickstart

from pyrecall import Model

model = Model("Qwen/Qwen2.5-1.5b-Instruct")
model.snapshot("before")
model.learn("data.jsonl", epochs=3)

if not model.check().is_healthy:
    model.rollback(to="before")

pyrecall init                                      # default model used is Qwen/Qwen2.5-1.5B-Instruct
pyrecall snapshot before_v1                        # baseline before training
pyrecall learn train.jsonl --snapshot-after after_v1  # train + snapshot in one step
pyrecall check                                     # compares last two snapshots
# exit 0 → ship   exit 2 → pyrecall rollback before_v1

To continuously monitor for new snapshots

pyrecall check --watch

How it works

Benchmarks 180 prompts across 9 skill categories (reasoning, coding, safety, math, multilingual, and more) using log-likelihood scoring. After training, check diffs the scores and flags any category that drops past your threshold (default 10%). Only the LoRA adapter is stored per snapshot — a few hundred MB, not the full model.

Any causal LM on HuggingFace Hub is supported — Llama, Mistral, Phi, Gemma, Qwen, Falcon, GPT-2/Neo/J, and more. LoRA targets are auto-detected.

Docs

Full CLI reference, Python API, experiment tracker integrations (W&B, MLflow, Neptune), custom benchmarks, per-category thresholds, and more:

pyrecall.github.io/Pyrecall

Contributing

git clone https://github.com/Arths17/Pyrecall
pip install -e ".[dev]"
pytest

Open an issue before large changes. MIT — LICENSE.