rlprobe 0.1.0

Flight recorder + doctor for RL fine-tuning runs (GRPO/PPO): record everything, diagnose failures early.

rlprobe

Flight recorder + doctor for RL fine-tuning runs (GRPO / PPO and variants).

RL fine-tuning fails in ways that are slow, silent, and expensive: a run can look healthy on its loss curves for hours while the policy quietly collapses (entropy → 0), reward-hacks (padding answers for length), or escapes its reference model (KL blowup). rlprobe records a complete, replayable account of a training run at near-zero overhead, then diagnoses it against a catalog of 14 known failure signatures — live during training (with optional auto-halt) or post-hoc.

from rlprobe.integrations.trl import attach

trainer = GRPOTrainer(...)        # your existing setup
attach(trainer, "runs/my-run")    # ← one line
trainer.train()

$ rlprobe doctor runs/my-run
CRITICAL ENTROPY_COLLAPSE | onset ~step 212 | confidence 90%
entropy fell from ~2.31 to ~0.048 — the policy is (near-)deterministic and
exploration is dead, while reward has plateaued (classic collapse pair)

try:
  • add/increase an entropy bonus
  • raise the sampling temperature during rollouts
  • check whether the reward is saturated (nothing left to explore for)

Install

pip install rlprobe[trl]      # with the TRL integration
pip install rlprobe           # core only (manual API + doctor; no torch needed)

What it does

• Flight recorder — per-step scalar metrics, per-sample forensics (prompts, completions, rewards, advantages, GRPO group ids), discrete events, and full run config, written to a self-contained run directory. Async writer, never blocks training, flushes on crash/SIGTERM, <2% step-time overhead, verified at 100k-step scale.
• Doctor — deterministic detectors (trend tests, sustained thresholds, changepoints — no ML magic) for 14 failure signatures: KL blowup, entropy collapse, length reward-hacking, format gaming, clip saturation, degenerate GRPO groups, advantage collapse, value divergence, NaN/grad pathology, truncation starvation, repetition spirals, reward-std death, positional KL asymmetry, throughput rot. Ranked findings with onset step, evidence, and remediations.
• Live guardian — rlprobe watch tails a growing run and alerts (webhook support); attach(trainer, run_dir, halt=True) stops training automatically on critical, high-confidence findings.
• Post-mortem suite — rlprobe report (self-contained HTML), rlprobe diff (where did run B diverge from A?), rlprobe samples / rlprobe show (forensic queries down to individual completions).
• Framework-agnostic core — the TRL callback is a thin layer over a manual API (Recorder.log_step / log_samples / log_event) that any training loop can call.

No GPU needed to try it — the synthetic run generator fabricates realistic pathological recordings:

rlprobe synth runs/demo --pathology entropy_collapse
rlprobe doctor runs/demo
rlprobe report runs/demo

Documentation

Full docs live in docs/ (mkdocs; pip install -e ".[docs]" && mkdocs serve): quickstart, the failure-signature catalog — each signature explained as RL-debugging documentation in its own right — the recording format, and the API & CLI reference.

Development

pip install -e ".[dev,trl]"
pytest
ruff check src tests examples benchmarks

Design history and phase-by-phase implementation notes: HANDOFF.md.

License

Apache-2.0