closed-loop-training 0.4.0

ReMiT + RLPT closed-loop flywheel: continuous self-improving LLM training

Closed-Loop Training System — ReMiT + RLPT Flywheel

╔══════════════════════════════════════════════════════════╗
║     ██████╗██╗     ██████╗ ███████╗██████╗ ██╗          ║
║    ██╔════╝██║    ██╔═══██╗██╔════╝██╔══██╗██║          ║
║    ██║     ██║    ██║   ██║█████╗  ██║  ██║██║          ║
║    ██║     ██║    ██║   ██║██╔══╝  ██║  ██║██║          ║
║    ╚██████╗██████╗╚██████╔╝███████╗██████╔╝███████╗     ║
║     ╚═════╝╚═════╝ ╚═════╝ ╚══════╝╚═════╝ ╚══════╝     ║
║                                                          ║
║         L  O  O  P        T  R  A  I  N  I  N  G         ║
║                                                          ║
║         ReMiT + RLPT  |  Continuous Self-Improvement    ║
╚══════════════════════════════════════════════════════════╝

Author: KingLabsA
GitHub: https://github.com/KingLabsA/llm-training-platform
HuggingFace: https://huggingface.co/King3Djbl | https://huggingface.co/fableforge-ai
Ollama: https://ollama.com/FableForge-AI
License: Apache 2.0

Continuous self-improving training: ReMiT token reweighting + RLPT next-segment reasoning, chained in iterative flywheel.

---

Quickstart

pip install closed-loop-training  # or: pip install -e .
closedloop train --model meta-llama/Llama-3.2-1B --corpus data.jsonl
# Try dummy mode first:
closedloop train --model dummy --corpus data.json --iterations 3 --tui

---

Features

• ReMiT Mid-Training — Dynamic token reweighting using frozen RL reference. Tokens where the RL model assigns higher probability get upweighted. Soft sigmoid weighting clipped to [0.8, 1.2]. No token discarding.
• RLPT Post-Training — Next-segment reasoning on pre-training data. GRPO-style optimization with generative reward model (GRM). No human annotations needed.
• Closed-Loop Flywheel — Post-trained model becomes the RL reference for the next cycle. Each iteration bootstraps from the previous. Convergence detection stops when improvement < 0.5%.
• Live TUI — Rich terminal dashboard with real-time ReMiT/RLPT metrics, progress bars, and iteration history table.
• Dummy Mode — Test the full pipeline without GPU or real models. Use --model dummy.
• Checkpoint & Resume — Save & resume from any iteration. History saved as JSON.
• Convergence Detection — Automatic early stopping when per-iteration improvement falls below threshold.

---

Architecture: Closed-Loop Flywheel

                    ┌─────────────────────────────────────┐
                    │         Closed-Loop Iteration        │
                    │                                     │
                    │  ┌──────────┐    ┌──────────────┐   │
                    │  │  ReMiT   │    │     RLPT     │   │
                    │  │  Mid-    │───>│  Post-       │   │
                    │  │ Training │    │  Training    │   │
                    │  └────┬─────┘    └──────┬───────┘   │
                    │       │                 │           │
                    │       v                 v           │
                    │  ┌─────────────────────────────┐    │
                    │  │     RL Reference Model      │    │
                    │  │  (becomes teacher next cycle)│    │
                    │  └──────────────┬──────────────┘    │
                    │                 │                   │
                    │                 v                   │
                    │  ┌─────────────────────────────┐    │
                    │  │     Gap Analysis +          │    │
                    │  │     Domain Targeting        │    │
                    │  └─────────────────────────────┘    │
                    │                 │                   │
                    └─────────────────┼───────────────────┘
                                      │
                                      v
                          ┌─────────────────────┐
                          │  Improved Base Model │
                          │  (fed to next cycle) │
                          └─────────────────────┘

---

How It Works

1. Initialize — Load a base model and training corpus. If no RL reference exists yet, skip ReMiT weighting on the first iteration.
2. ReMiT Mid-Training — For each batch, compute per-token loss discrepancy between the base model and the frozen RL reference. Generate importance weights via centered sigmoid clipped to [0.8, 1.2]. Train on weighted NLL.
3. RLPT Post-Training — Segment corpus text into sentence units. For each segment, prompt the model to predict the next segment. Score predictions with a Generative Reward Model (GRM). Optimize via GRPO (group-relative advantages with KL penalty).
4. Reference Update — Set the post-trained model as the new RL reference (frozen). The next ReMiT cycle will compare against this improved reference.
5. Convergence Check — Measure ReMiT loss improvement. If below threshold, stop.
6. Iterate — Repeat. Each cycle the model learns from its own improvements.

---

CLI Reference

closedloop train

Flag	Default	Description
--model	meta-llama/Llama-3.2-1B	HuggingFace model name or dummy
--corpus	(required)	Path to JSON, JSONL, or text corpus
--iterations	3	Number of closed-loop flywheel iterations
--mid-train-steps	500	ReMiT steps per iteration
--rl-train-steps	500	RLPT steps per iteration
--remit-batch-size	8	Batch size for ReMiT
--rlpt-batch-size	4	Batch size for RLPT
--max-samples	10000	Max corpus samples to load
--domain	None	Target domain name
--max-seq-length	2048	Maximum sequence length
--lr	3e-4	Base learning rate
--remit-lr	3e-4	ReMiT learning rate
--rlpt-lr	1e-6	RLPT learning rate
--output-dir	./closed_loop_output	Output directory
--log-interval	10	Steps between logging
--device	auto	auto, cuda, or cpu
--tui	(flag)	Enable live TUI display

closedloop resume

Flag	Default	Description
--checkpoint	(required)	Path to .pt checkpoint
--output-dir	checkpoint dir	Output directory
--iterations	from checkpoint	Override remaining iterations
--tui	(flag)	Enable live TUI display

TUI Mode vs Non-TUI Mode

• TUI mode (--tui): Rich live dashboard with real-time ReMiT/RLPT metrics, progress bars, iteration history table, phase indicators, and elapsed time. Ideal for demos, monitoring, and debugging.
• Non-TUI mode (default): Print-based logging. Each step logs loss/metrics. Compact output suitable for scripts, CI/CD, and headless environments.

Environment Variables

Variable	Purpose
PYTHONPATH	Should include project root for imports

---

Citation

@misc{kinglabsa2026closedloop,
  author = {KingLabsA},
  title = {Closed-Loop Training System: ReMiT + RLPT Flywheel},
  year = {2026},
  publisher = {GitHub},
  url = {https://github.com/KingLabsA/llm-training-platform}
}

@article{huang2026remit,
  title={ReMiT: Reinforcement Learning Guided Mid-Training for Large Language Models},
  author={Huang et al.},
  journal={arXiv preprint arXiv:2602.03075},
  year={2026}
}

@inproceedings{li2026rlpt,
  title={RLPT: Reinforcement Learning on Pre-Training Data for Next-Segment Reasoning},
  author={Li et al.},
  booktitle={ACL 2026},
  year={2026}
}

---

License

Apache 2.0 — See LICENSE file.

---

Built by KingLabsA. ReMiT paper: arXiv 2602.03075. RLPT paper: ACL 2026.