Causal GPT-RL

GPT-style transformers (GPT-2, Llama) running as RL policies in continuous-control environments.

Both LLM generation and RL interaction are autoregressive:

token           → next token                           (LLM generation)
(state, action) → (next state from env, next action)   (RL rollout)

Causal GPT-RL policies act stably under their own rollouts — long-horizon control without the drift that has historically kept transformers from being usable as RL agents.

A single autoregressive model drives full-episode rollouts via KV cache — no critic, no auxiliary networks at inference.

This repository is the public inference runtime. It loads policy bundles, runs Gymnasium/MuJoCo rollouts, and provides small evaluation helpers.

Code (GitHub): ccnets-team/causal-gpt-rl
Run logs (W&B, public): wandb.ai/junhopark/Causal GPT-RL
Hugging Face org: https://huggingface.co/ccnets
Website: https://ccnets.org
LinkedIn: https://www.linkedin.com/company/ccnets

Install

For Hub loading and MuJoCo environments:

pip install "causal-gpt-rl[hub,mujoco]"

For local development:

git clone https://github.com/ccnets-team/causal-gpt-rl.git
cd causal-gpt-rl
python -m pip install -e ".[hub,mujoco]"

For private bundles, authenticate first:

hf auth login

Quick Start

import gymnasium as gym

from causal_gpt_rl.inference import load_runner_from_hub, run_episodes

env = gym.make("Ant-v5")
runner = load_runner_from_hub(
    repo_id="ccnets/causal-gpt-rl",
    subfolder="ant-v5",
)

stats = run_episodes(env, runner, num_episodes=5, seed=0)
env.close()
print(stats["return_mean"], stats["return_std"])

Notebook version: examples/hub_quickstart.ipynb

Supported Environments

Env	Bundle	Ctx	Return	Norm.	Simple Ref.	Medium Ref.
`Ant-v5`	`ant-v5`	32	5126.03±1568.43	77.05±23.26	59.99 ✓	86.54 ✗
`HalfCheetah-v5`	`halfcheetah-v5`	32	6816.48±3135.53	42.87±19.01	43.54 ✗	74.83 ✗
`Hopper-v5`	`hopper-v5`	32	2713.66±1075.57	70.21±28.01	42.65 ✓	72.91 ✗
`Walker2d-v5`	`walker2d-v5`	32	3899.88±706.57	56.93±10.32	59.51 ✗	83.26 ✗
`Humanoid-v5`	`humanoid-v5`	32	7892.65±1018.11	91.63±11.99	63.29 ✓	81.30 ✓

Training data is expert-free: bundles are trained using Minari simple and medium datasets only; expert trajectories are not used for training.

Return and Norm. are mean±std over 50 episodes with seeds 0..49. Ctx is context length. max_steps=1000, and KV cache max length is capped to Ctx.

Normalized scores use random=0 and expert=100:

100 * (return - random_ref) / (expert_ref - random_ref)

Simple Ref. and Medium Ref. are the normalized means of the Minari simple-v0 and medium-v0 datasets. They are shown for context and are not the normalization baseline. ✓ marks a reference the bundle's Norm. exceeds, ✗ one it does not.

KV cache retention sweep

Ctx above is the bundle's context_length — the model's context window, fixed at 32 and not changeable at inference. kv_cache_max_len (how much past the rollout retains) is a load-time knob; the main table caps it to Ctx (1×). Sweeping it to 0.5×, 1×, and 2× the window, with the same protocol (50 episodes, seeds 0..49, max_steps=1000):

Env	`kv=16` (0.5×)	`kv=32` (1×)	`kv=64` (2×)	Trend
`Ant-v5`	5109.97±1711.18	5126.03±1568.43	5256.33±1429.58	2× highest mean (≈)
`HalfCheetah-v5`	6793.17±2939.17	6816.48±3135.53	6468.21±3234.51	≈ flat
`Hopper-v5`	3361.71±103.69	2713.66±1075.57	992.92±445.63	shorter ↑, 2× collapses
`Walker2d-v5`	3950.09±459.01	3899.88±706.57	3842.19±718.60	≈ flat
`Humanoid-v5`	7431.52±2024.95	7892.65±1018.11	8040.41±38.02	longer ↑, steadiest

For Ant-v5, the recommended serving setting is kv_cache_max_len=16. The model context remains 32; only the retained rollout history is shortened. The recommended setting scores 5467.20±1153.95 (82.11±17.11 normalized) over the same 50 seeds.

The kv=32 column matches the main table. At kv=64 the rollout attends over more history than the model's 32-token window — positions outside its native range. This stays within the backbone's position capacity (Llama/RoPE, max_position_embeddings=256), so it is an extrapolation regime, not a hard cap.

Best retention is environment-dependent, but for most envs the difference across 0.5×/1×/2× is within run-to-run noise (Trend marks these ≈). The real exceptions: Hopper-v5 clearly prefers a shorter window (0.5× is +24% with much lower variance; 2× collapses to roughly a third of return, episodes ending early), while Humanoid-v5 is best at 2× (highest return and its steadiest — std 38 across all 50 episodes). The context window (1×) is a safe default; deviating from it helps only in specific environments.

Evaluation runtime — every row above is measured on this one:

causal-gpt-rl 0.13.0
torch 2.8.0+cu129
gymnasium 1.2.3
mujoco 3.2.3
minari 0.5.3

mujoco is pinned to 3.2.3 because that is the version the Minari datasets were recorded with (requirements: ['mujoco==3.2.3', 'gymnasium>=1.0.0']). The Norm. and Medium Ref. columns are derived from those recorded trajectories, so returns are only comparable to them when measured on the same physics.

Bundle Format

Public bundles use bundle_format_version=2:

bundle/
  model.safetensors
  config.json

model.safetensors — model state dict for inference, with state normalization statistics embedded in the weights.
config.json — model config, observation specs, action specs, context length, a state_normalization block, and optional env_id.

Older bundles (bundle_format_version=1) shipped a separate state_normalizer.safetensors sidecar. They still load with current releases. If you are pinned to causal-gpt-rl <= 0.2.x, use the sidecar bundles preserved at the bundles-v1 tag:

runner = load_runner_from_hub(
    repo_id="ccnets/causal-gpt-rl",
    subfolder="ant-v5",
    revision="bundles-v1",
)

Hugging Face Layout

Recommended layout:

ccnets/causal-gpt-rl/
  ant-v5/
    model.safetensors
    config.json
    README.md

For local bundles, use load_runner("path/to/bundle").

API

from causal_gpt_rl.inference import (
    PolicyRunner,                          # step-wise rollout policy with KV cache
    load_runner,                           # load runner from a local bundle directory
    load_runner_from_hub,                  # load runner from a Hugging Face Hub repo
    run_episodes,                          # evaluate over N episodes; returns stats dict
    export_bundle,                         # write a bundle directory from a runner
    convert_legacy_bundle_to_safetensors,  # migrate legacy bundles to the safetensors format
)

Development Checks

python -m compileall -q causal_gpt_rl
python -m unittest discover -s tests
python -m build
python -m twine check dist/*

License

Released under PolyForm Noncommercial License 1.0.0. See LICENSE for details. For commercial licensing, contact the maintainers via ccnets.org.

Downloads last month: 178

Video Preview

Reinforcement Learning