areno 0.0.6

👋 Hi, everyone! AReno is a fast, effortless, and self-contained toolkit that scales RL post-training up locally, initiated by the inclusionAI ASystem team and maintained by the AReno community.

AReno: ASystem Reinforcement Learning Nano

AReno is a local LLM post-training toolkit for RL, SFT/DPO-style training, serving, and agentic RL. It was originally developed by engineers from the ASystem Team at Ant Group.

Built on a self-contained, full-stack design, AReno is optimized to extract maximum performance from a single node, making it well-suited for fast, local post-training with no external training or inference backend in the loop.

AReno's mission is to make LLM RL accessible for a broad community of researchers and developers — so you can go from a base checkpoint to a trained, served model on a single node, without standing up a cluster or wiring together a training framework, an inference server, and a kernel library.

Small but complete, like its name — nano in footprint, full-stack in capability. We hope AReno makes scaling up your ideas locally both fast and delightful. Enjoy!

Highlights

• ✨ Plug-and-play: various post-training methods are easily accessible via the --algo flag or the same Trainer class from Python, no cluster or launcher to set up.
• 🪶 Lightweight: single self-contained package, no external training/inference backend, just PyTorch, FlashAttention, and a handful of other libraries.
• 🧰 Agentic RL ready: run an agent function against AReno's local OpenAI-compatible proxy, return explicit trajectories, and train from tokens, logprobs, rewards, and loss masks derived by the trainer.
• 🧩 Extensible: easily register new algorithms, model adapters, reward functions, and hardware backends without changing the core.

Installation

Clone AReno and run the installer:

git clone https://github.com/inclusionAI/AReno.git
cd AReno
bash scripts/install.sh

No installation options or prior setup decisions are required before running the script. Before changing Python, the script checks required system tools, rejects WSL1, and verifies that nvidia-smi can see a GPU. It then uses the active Python environment when appropriate. In preconfigured IDE environments where activation metadata is unavailable, it detects and reuses a Python interpreter that already provides PyTorch instead of creating an empty .venv; otherwise it prepares .venv. It then checks for a CUDA-enabled PyTorch 2.6 or newer build, installs the remaining dependencies in the required order, detects the CUDA build environment, builds AReno, and finishes by running areno check. The installer never installs or upgrades PyTorch because the correct build depends on the machine's CUDA platform. If PyTorch is missing or incompatible, it stops with guidance for the selected Python environment. Other installed packages are reused when they satisfy AReno's requirements; only missing or incompatible packages are installed or updated. If a step fails, it reports the failed stage, preserves the complete output in the user state directory (usually ~/.local/state/areno/install.log), and prints suggestions specific to that failure.

AReno training and serving target Linux with NVIDIA CUDA. The installer checks platform support before changing the environment. Windows users should run it inside WSL2. The installer does not support macOS or CPU-only environments.

To preview what the installer will do without making changes:

bash scripts/install.sh --dry-run

For setup or support reports:

areno check
areno env --json  # attach this to setup/support reports

Docker setup escape hatch (recommended when you want to verify AReno before debugging local build state):

docker build -t areno .
docker run --gpus all --rm -it areno areno check

If you need local project files, model files, or a Hugging Face cache inside the container:

docker run --gpus all --rm -it \
  -v $PWD:/workspace \
  -v $HOME/.cache/huggingface:/root/.cache/huggingface \
  areno \
  areno check

Host checklist before blaming AReno setup:

nvidia-smi
docker run --gpus all --rm nvidia/cuda:12.4.1-base-ubuntu22.04 nvidia-smi
docker run --gpus all --rm areno areno check

Docker gives you a known-good Python/PyTorch/CUDA user-space install path and reuses the same areno check diagnostic flow. It does not replace host requirements: the host still needs a working NVIDIA driver, NVIDIA Container Toolkit support for --gpus all, and a driver new enough for the container CUDA runtime. Docker also does not solve model downloads, Hugging Face tokens, cache paths, network access, disk space, or multi-node networking; those remain user environment concerns.

The installer selects the attention setup automatically. It installs FlashAttention for supported GPUs and leaves older GPUs on AReno's native compatibility backend. Build helpers, visible GPU architectures, and CUDA variables are handled by the script.

Quick Start

With the SDK, RL loop is a short cycle of Trainer calls. Each step below maps a concept to the SDK call that performs it:

flowchart LR
    A["Trainer<br/>init()"] -->
    B["rollout_batch<br/>on-policy samples"] -->
    C["reward fn<br/>score"] -->
    D["train<br/>optimizer step"] -->|repeat| B

1. Create the trainer — construct a Trainer on the AReno backend and init() it to load the tokenizer and start workers.
2. Roll out — inside rollout_session(...), rollout_batch(...) generates on-policy completions for each prompt.
3. Score — reward each completion and turn rewards into advantages (your reward function, not AReno's).
4. Train — pack the rollout into TrainSequence objects and call train(batch, loss_fn) to run one optimizer step.
5. Repeat — new weights produce new rollouts; loop until done, then close().

import asyncio
from functools import partial

from datasets import load_dataset

from areno.api import (
    Areno,
    ArenoConfig,
    SamplingParams,
    Trainer,
    TrainSequence,
    gspo_loss_fn,
)
from examples.math.math_verify_reward import reward_fn


def to_advantages(rewards):
    mean = sum(rewards) / len(rewards)
    var = sum((r - mean) ** 2 for r in rewards) / max(len(rewards), 1)
    std = max(var**0.5, 1e-6)
    return [(r - mean) / std for r in rewards]


async def main():
    # 1. Create the trainer
    trainer = Trainer(
        world_size=1,
        model_path="Qwen/Qwen3-0.6B",
        backend_type=Areno,
        custom_config=ArenoConfig(tp_size=1),
    )
    trainer.init()

    try:
        # 2. Roll out on-policy completions for one GSM8K prompt
        row = load_dataset("gsm8k", "main", split="train[0:1]")[0]
        prompt = (
            "Solve the problem and put the final answer in \\boxed{}.\n\n"
            f"Problem: {row['question']}\nSolution:"
        )
        prompt_tokens = trainer.get_tokenizer().encode(prompt)
        sampling = SamplingParams(max_new_tokens=512, temperature=1.0)

        async with trainer.rollout_session(sampling_params=sampling, proxy=False):
            rollout = trainer.rollout_batch(
                [prompt],
                n_samples=8,
                sampling_params=sampling,
            )[0]

        # 3. Score with the same reward function the CLI uses, then form advantages
        completions = [trainer.get_tokenizer().decode(seq.resp_tokens) for seq in rollout.sequences]
        rewards = reward_fn(row, completions)
        advantages = to_advantages(rewards)

        batch = []
        for seq, reward, advantage in zip(rollout.sequences, rewards, advantages, strict=True):
            response_len = len(seq.resp_tokens)
            batch.append(
                TrainSequence(
                    prompt_mask=[True] * len(prompt_tokens) + [False] * response_len,
                    tokens=prompt_tokens + seq.resp_tokens,
                    logprobs=[0.0] * len(prompt_tokens) + seq.resp_logprobs,
                    advantages=[0.0] * len(prompt_tokens) + [advantage] * response_len,
                    reward=reward,
                    eos_token_id=trainer.get_tokenizer().eos_token_id,
                )
            )

        # 4. Train one step
        stats = trainer.train(batch, partial(gspo_loss_fn, clip_eps=3.0e-4), mini_bs=4)

        # 5. Repeat the loop over more prompts
    finally:
        trainer.close()


asyncio.run(main())

See the documentation for the full Trainer API.

Command Line Interface (CLI)

You can use the AReno Command Line Interface (CLI) to quickly get started with post-training without writing any Python.

Operations agent

areno agent is a local operations assistant for AReno train and serve tasks. It uses an OpenAI-compatible model to inspect the current checkout, read command help, run diagnostics, and produce or execute AReno commands for the current machine. Configure the agent once with your endpoint, model, and API key:

areno agent --set \
  --base-url http://127.0.0.1:8000/v1 \
  --model deepseek-v4-flash \
  --api-key "$OPENAI_API_KEY"

Then describe the job in natural language:

areno agent "Give me a complete command to run the math demo with n-samples=8, fitting the current GPU and using as much GPU memory as practical."

From a source checkout, you can run the same agent without installing AReno:

./agent.sh "Give me a complete command to run the math demo with n-samples=8, fitting the current GPU and using as much GPU memory as practical."

The agent can ask follow-up questions through the terminal when a required parameter is missing, and it streams command output while it works.

Diagnostics

Check whether the current machine is ready to run AReno:

areno check

areno check prints OK, WARN, and FAIL statuses with concrete next steps for common setup issues such as missing CUDA, CPU-only PyTorch, missing CUDA_HOME, unavailable nvcc, missing optional runtime dependencies, or a missing areno_accel extension.

For issue reports, collect a descriptive environment report:

areno env --json

The report includes AReno, Python, platform, PyTorch/CUDA, GPU, nvcc, dependency import status, and relevant environment variables.

Training

Tiny training smoke test

Use this command when you only want to check that a machine can run one small official training task end to end:

areno train \
  --ckpt Qwen/Qwen3-0.6B \
  --dataset-path gsm8k:main \
  --dataset-loader-fn examples/math/dataset_loader.py \
  --reward-fn-path examples/math/math_verify_reward.py \
  --algo gspo \
  --tp-size 1 \
  --world-size 1 \
  --batch-size 1

This is a smoke/sanity task for the CLI, dataset loader, reward function, rollout, and training-step wiring. It is not a quality benchmark. It requires a CUDA-capable NVIDIA GPU; CPU-only machines can install the package for docs and metadata checks, but cannot run the AReno training engine. A successful run should reach rollout logs and a train_stats=... line without raising an exception.

Run GSPO on a GSM8K-style dataset with a reward function:

areno train \
  --ckpt Qwen/Qwen3-0.6B \
  --dataset-path gsm8k:main \
  --dataset-loader-fn examples/math/dataset_loader.py \
  --reward-fn-path examples/math/math_verify_reward.py \
  --algo gspo \
  --tp-size 4

--ckpt and --dataset-path accept either local paths or remote repo IDs. By default, remote refs use ModelScope. Use --model-hub hf to pull non-local model and dataset refs from Hugging Face. Switch algorithms by changing --algo (e.g. --algo grpo, --algo sft).

For models whose tokenizer chat template supports a thinking-mode switch, add --disable-thinking to pass enable_thinking=False during training prompt rendering. Tokenizers that do not support this argument automatically use their normal chat-template path.

For rollout-based algorithms, add --tune-params when you want AReno to probe rollout and train memory before the real run and fill conservative values for --max-running-prompts, --batch-size, and --mini-bs:

areno train \
  --ckpt Qwen/Qwen3-0.6B \
  --dataset-path gsm8k:main \
  --dataset-loader-fn examples/math/dataset_loader.py \
  --reward-fn-path examples/math/math_verify_reward.py \
  --algo gspo \
  --tp-size 1 \
  --world-size 1 \
  --n-samples 8 \
  --tune-params \
  --mem-frac 0.9 \
  --tune-max-samples 256

The tuner uses dummy-loaded model weights and synthetic token rows, respects the configured sequence limits and tensor-parallel size, and enables --drop-rollout-state for the tuned run. See docs/cli/training.rst for the full tuning rules.

For Agentic RL, add --agent-fn to supply an agent function. The agent calls the local OpenAI-compatible endpoint, including tools and tool_choice when needed, and returns explicit AgentTrajectoryTurn objects. AReno converts those turns into trainable assistant outputs and masks tool results by default:

python examples/agentic/tictactoe/dataset_generator.py \
  --output /tmp/areno-tictactoe.jsonl \
  --count 2048 \
  --seed 2026

areno train \
  --ckpt Qwen/Qwen3-0.6B \
  --dataset-path /tmp/areno-tictactoe.jsonl \
  --dataset-loader-fn examples/agentic/tictactoe/dataset_loader.py \
  --reward-fn-path examples/agentic/tictactoe/reward.py \
  --agent-fn examples/agentic/tictactoe/run_agent.py \
  --algo gspo \
  --tp-size 1 \
  --world-size 1

For a more realistic multi-turn software-engineering loop, see examples/agentic/coding. It uses a 100-record SWE-bench-style local dataset that ranges from easy to hard with most records marked hard, constrained Codex-style tools (inspect_tree, read_file, rg, apply_patch, run_command, submit), and an explicit trajectory-returning run_agent.py. The records are self-contained local Python tasks and do not require sandbox services, package installation, or network access.

DuelGrid is a larger agentic demo with a browser game UI and multi-action turns. Before GSPO/RLVR post-training, Gemma-E2B-it performs poorly in this game and often moves back and forth without progress. After training, it learns to collect pickups, chase the user, attack when in range, and avoid trap tiles.

Train before	Reward	Train after

See examples/agentic/duelgrid for the rule engine, fixed-path dataset loader, reward function, OpenAI-compatible agent, and browser UI.

For the full list of training options, run areno train --help.

Serving

Serve a trained checkpoint as an OpenAI-compatible endpoint with continuous batching:

areno serve \
  --model-path /path/to/model \
  --tp-size 1 \
  --world-size 1 \
  --port 8000

Point any OpenAI client at http://localhost:8000/v1/chat/completions to start generating. For the full list of serving options, run areno serve --help.

Use --disable-thinking with areno serve to pass enable_thinking=False to compatible tokenizer chat templates for request rendering.

Development

If you want to contribute to AReno or customize it for your own needs, read the contribution guide and make a development install:

git clone https://github.com/inclusionAI/AReno.git
cd AReno
bash scripts/install.sh

# Set up pre-commit hooks (formatting, linting, commit message checks)
pip install pre-commit
pre-commit install --install-hooks

New algorithms, model adapters, kernels, reward functions, and hardware backends all have first-class extension points, so most contributions land without forking the core.

Citation and Acknowledgement

If you find the project helpful, please cite:

@misc{areno2026,
  title        = {AReno: A Self-Contained, Full-Stack Toolkit for Single-Node LLM RL Post-Training},
  author       = {Zibo He and Le Su and Zongyu Li and Xiaowei Zhu and Cheng Wang and Zhenxuan Pan},
  year         = {2026},
  url          = {https://github.com/inclusionAI/AReno},
  license      = {Apache-2.0}
}

AReno's API design is inspired by Tinker from ThinkingMachines. We would like to express our gratitude for their pioneering work.

License

This repository's source code is available under the Apache 2.0 License.