miniverl 0.16.0

PyPI · Stable docs · Development docs · 中文

Run common verl PPO/GRPO configs directly on one NVIDIA GPU. Hand miniVERL your upstream config tree, config name and Hydra overrides; bind local data or reward code. Its versioned compiler preserves the experiment while scheduling actor, critic, reference and reward roles in phases. No second configuration language to maintain.

The current development line covers PPO/GAE, GRPO, Dr.GRPO, RLOO and REINFORCE++ against official verl v0.9.0 (483b8a00). PPO uses an independent trainable critic with its own optimizer and checkpoint state; actor KL, entropy regularization, grouped rollouts, task rewards and a pinned sequence-classifier reward role share the same provenance model. The established verl v0.8.0 OPD profiles remain available for direct GKD and sampled-k1 distillation.

PyPI v0.16.0 is stable; main is development.

Your first local experiment

Install the CUDA-enabled PyTorch build that matches your machine, then:

python -m pip install "miniverl[train,hydra]"
miniverl data sample --reward-profile target-length --rows 8 --out data/rl-prompts.parquet
miniverl run --example hydra-ppo --bind reward.provider=target_length --dry-run
miniverl run --example hydra-ppo --bind reward.provider=target_length --run-id local-ppo

The packaged Hydra examples compose official verl defaults plus bounded Qwen3-0.6B launch overrides. The binding selects a length-reward exercise. Bring your own config directly:

miniverl run --verl-config-path /path/to/verl/trainer/config \
  --verl-config-name ppo_trainer --dry-run \
  algorithm.adv_estimator=grpo actor_rollout_ref.rollout.n=8 trainer.n_gpus_per_node=8

Each run records the config tree, ordered overrides, resolved bytes, field decisions, bindings, snapshots and execution plan. Inspect, resume and export:

miniverl inspect runs/local-ppo
miniverl run --example hydra-ppo --bind reward.provider=target_length \
  --resume-from runs/local-ppo/checkpoints/step-000002
miniverl export-adapter --run runs/local-ppo --out runs/local-ppo/model
miniverl export-verl --run runs/local-ppo --target-verl v0.9.0 --out ppo-handoff
miniverl bridge doctor ppo-handoff --json

The five-minute workflow explains each artifact and offers the matching GRPO commands. These are small length-reward exercises; their reward is directly inspectable in rewards.jsonl.

[train] supplies the ML stack, [hydra] pins composition, and [cuda] adds quantization. Select PyTorch's CUDA build with the PyTorch installer. The single-GPU guide covers memory planning and the maintainer-measured RTX 4080 environment.

What a run gives you

• A field-by-field compiler report. Experiment fields retain their meaning; physical distribution fields receive an explicit one-GPU lowering.
• Policy-bound trajectories. Group/sample identity, generated-token spans, behavior log-probabilities and policy version travel together.
• Auditable rewards and objectives. Reward components, group advantages, reference KL, clipping, entropy and update metrics are structured data.
• Exact recovery. Transactional manifests and checkpoints restore policy, optimizer, cursor, RNG and reward identities before another rollout.
• Portable outputs. PEFT adapters, safetensors, Parquet, resolved config and typed provenance can move into a larger workflow.

How it works

One GPU is treated as a temporal scheduler for logical roles. RL generates complete prompt groups, scores outcomes, evaluates optional reference and reward-model roles, computes the pinned advantage estimator, and updates the actor. PPO adds a separate critic phase with clipped value updates. OPD uses the same trajectory and checkpoint foundations, with teacher targets supplying the learning signal.

Choose your path

Native recipes also support SFT, DPO and offline KD, plus calculator, JSON navigation, read-only SQLite and custom tool environments.

Current capability matrix

The upstream compatibility corpus resolves real verl examples and records every field's outcome. Native Hydra composition, shuffled actor/critic epochs and group filtering/refill use the new v5 profile. Composition, semantics and hardware remain separate statuses. Resolved YAML and historical v1/v2/v3/v4 profiles remain available.

Measured systems evidence

The published Qwen3-0.6B/1.7B OPD workload consumed 32 distinct prompts and completed eight current-policy updates at 3.1914 GiB peak reserved VRAM on an RTX 4080. A matched interruption after update four reproduced byte-identical trajectories, adapter and optimizer tensors. The SmolLM2-360M/1.7B workload completed the same shape at 1.4961 GiB. System records contain configs, hashes and phase timings.

Rollout Runtime v2 measured hf_cached and managed vLLM across 24 RTX 4080 cells spanning response length, sampling mode and n=1/4. See the runtime report. Evidence for the new RL family is published through the exact-wheel release qualification rather than presented as a task-quality comparison.

Research record

The scientific reports preserve their original outcomes and frozen inputs: calculator protocol, RecoveryBench, Alignment Lab, and the External Alignment Gate. They are scoped studies, separate from runtime qualification.

Compatibility boundary

miniVERL targets one local process and one NVIDIA CUDA GPU. The compiler distinguishes exact or conformant semantics, local physical lowering, distributed-only settings, feasible work not yet implemented, and technically unsupported inputs. The compatibility policy is the complete matrix; limitations collects architecture, measurement, security and generalization boundaries in one place. miniVERL is an independent Apache-2.0 project.

Development

git clone https://github.com/DaoyuanLi2816/mini-verl.git
cd mini-verl
python -m pip install -e ".[dev,train]"
pytest -q -m "not gpu and not network"

See CONTRIBUTING.md, CHANGELOG.md, CITATION.cff, the guide for verl users, SECURITY.md and the Apache-2.0 license.