license: other license_name: polyform-noncommercial-1.0.0 license_link: https://polyformproject.org/licenses/noncommercial/1.0.0 library_name: pytorch tags:

• reinforcement-learning
• gymnasium
• mujoco
• unity
• ml-agents
• causal-gpt-rl

Causal GPT-RL

GPT-style transformers (Llama) running as RL policies in continuous and discrete control environments.

Here to run a policy? → Quick Start. Evaluating the product? → Available Policies for what is published and how it scores.

No expert demonstrations required: trained only on simple and medium offline trajectories, several bundles outperform both. Horizon is adjustable at inference, with stable retention beyond the training window. Try the bundles on Hugging Face.

Both LLM generation and RL interaction are autoregressive:

token           → next token                           (LLM generation)
(state, action) → (next state from env, next action)   (RL rollout)

Causal GPT-RL policies act stably under their own rollouts — long-horizon control without the drift that has historically kept transformers from being usable as RL agents.

A single autoregressive model drives full-episode rollouts via a KV cache, the same mechanism an LLM uses to hold its context while generating — no separate critic, no auxiliary networks at inference. It computes a value head on every forward pass, but a rollout never reads it: the action alone carries the loop.

The calling contract that follows from this — why the output is one step ahead of the observation you just passed, and why actions keep coming with no environment attached — is Transformer Model Integrating Environment Dynamics for RL.

This repository is the public inference runtime: it loads policy bundles, runs Gymnasium/MuJoCo rollouts, and provides small evaluation helpers. Training is not here — it runs on AWS Marketplace as the CCNets Causal GPT-RL Training Algorithm.

• Code (GitHub): ccnets-team/causal-gpt-rl
• Hugging Face org: https://huggingface.co/ccnets
• MuJoCo runs (W&B): https://wandb.ai/causal-gpt-rl/mujoco
• Website: https://ccnets.org
• LinkedIn: https://www.linkedin.com/company/ccnets

Released under PolyForm Noncommercial 1.0.0. Commercial use is licensed separately — see License.

Install

pip install "causal-gpt-rl[hub,mujoco]"

From a clone, python -m pip install -e ".[hub,mujoco]". For private bundles, run hf auth login first. On Windows the Hub cache uses symlinks, which fail with OSError: [WinError 1314] unless Developer Mode is on — enable it, or set HF_HUB_DISABLE_SYMLINKS=1.

Turning a bundle into a self-contained ONNX policy is Export a delivered bundle to ONNX.

Quick Start

import gymnasium as gym

from causal_gpt_rl.inference import load_runner_from_hub, run_episodes

env = gym.make("Ant-v5")
runner = load_runner_from_hub(
    repo_id="ccnets/causal-gpt-rl",
    subfolder="ant-v5",
)

stats = run_episodes(env, runner, num_episodes=5, seed=0)
env.close()
print(stats["return_mean"], stats["return_std"])

Notebook version: examples/hub_quickstart.ipynb

Five episodes off one seed is a smoke test, not the protocol the published scores were measured under. To measure a bundle that way, see Reproduce a published score.

Available Policies

Policy bundles, the environments they run in, and the trajectory datasets are published on the Hugging Face org:

Returns, the evaluation protocol, and the runtime each score was measured on are on the corresponding model card; the MuJoCo training runs are public at wandb.ai/causal-gpt-rl/mujoco, and worked runs — including the Unity download-and-measure walkthroughs — are in examples/.

Observation & Action Spaces

Every fixed-shape Gymnasium space, and any Dict / Tuple nesting of them. Variable-length and structural spaces — raw images, Text, Sequence, Graph, OneOf — are out of scope; encode those into vectors on your side. A bundle carries its own spaces, so you pass observations exactly as your env produces them and get back a valid sample of its action space.

See docs/ for the per-space contract, the calling contract, the API reference, and ONNX export.

Rollout History

A bundle's context_length is the length used in training, and not a limit at inference.

kv_cache_max_len — how much past a rollout retains — is a load-time knob. It defaults to the bundle's own context_length, which keeps a rollout inside the window the policy was measured on:

runner = load_runner("path/to/bundle", kv_cache_max_len=64)

A long horizon becomes a matter of context length — conditioning on context offers a general way to address RL problems, using a quantity language models already expose: how much past the model retains, in steps rather than words. Unlike an MLP policy, that history can be changed at load time without changing the model. See Transformer Model Integrating Environment Dynamics for RL.

Larger values run. Whether more past helps a particular environment is a separate question — across the Hugging Face bundles, it helps some and hurts others.

On long rollouts, a mean return can average early failures and full-length runs into a number that describes neither. See Measuring a Long Horizon for what to report instead.

Bundle Format

A bundle is a directory of two files, and it is the same directory whether it came from Hugging Face, from local export, or from a training run. On the Hub each bundle is one subfolder of a repo, which is what subfolder= selects; locally, pass the directory to load_runner("path/to/bundle").

bundle/
  model.safetensors    # inference weights, state normalization embedded
  config.json          # model config, observation/action specs, context length

Public bundles are bundle_format_version=2. Runtime releases keep backward compatibility, so the bundles on Hugging Face today and the bundles produced through AWS load on any package version.

Development Checks

python -m compileall -q causal_gpt_rl
python -m unittest discover -s tests
python -m build
python -m twine check dist/*

License

Released under PolyForm Noncommercial License 1.0.0. See LICENSE for details. Commercial use is licensed separately — through a CCNets Causal GPT-RL Training Algorithm subscription on AWS Marketplace, or by contacting the maintainers via ccnets.org.