racingmania-generalist-seed1

Generalist (PPO), seed 1: Generalist PPO agent trained on the whole generated training pool

Status: evaluated (evaluated; scores available); physics version v1; card generated 2026-11-15 from the result files.

RacingMania is an original 3D time-attack racing game for the browser (three.js + cannon-es RaycastVehicle) with reinforcement-learning drivers trained on a batched PyTorch port of the same car physics. Code, paper and full results: https://github.com/BurnyCoder/reinforcement-learning-in-trackmania-like-racing-game.

Videos

Official evaluation runs of this model replayed in the browser game ("Watch replay"): the car states the evaluator recorded while scoring the run, so each clip is exactly the run its caption describes. More clips, including the controls: https://github.com/BurnyCoder/reinforcement-learning-in-trackmania-like-racing-game#videos.

Best lap of the second generalist seed

• What you see: The generalist of seed 1, trained by reinforcement learning on generated tracks only, as in the preregistered recipe. This hand-made track was never part of its training, so the lap is generalization: it reads the road ahead and picks a speed for each bend.
• Model: generalist/seed1: PPO, trained by reinforcement learning for 1 billion agent steps on 48,949 generated training tracks (seed 1, 1.9 h on one GPU).
• Map: Technical (campaign-04), a hand-made campaign track, 666 m long with no jumps and up to 5° of banking; this model never saw it during training.
• Evaluation: official evaluation in results/generalist/seed1/eval_campaign.csv (standing start, most probable action, final checkpoint): finished in 21.60 s; the scripted driver needs 42.52 s, so this is 1.97× its speed. The clip replays this run.
• Why this clip: chosen by a fixed rule as the highest-scoring finished track of the 5 campaign tracks it finished in its official evaluation.

Model

Item	Value
Algorithm	PPO
Architecture	MLP 2x256 tanh (actor and critic), two 3-way heads
Objective	centreline progress
Observation	car state + 20 look-ahead points
Training data	48,949 generated tracks
Start states	standing + random centreline (p=0.5)
Agent steps trained	1,000,341,504
Wall-clock hours	1.92
Git commit	a861208dbcd896c0f7fe896657c5ee2ca2f45920

Evaluation

Success-weighted speed (SWS): mean over tracks of finished x (scripted driver's time / agent's time); 1.0 means finishing every track as fast as the scripted reference driver. One deterministic run per track from a standing start with the most probable action; intervals are 95 % percentile bootstrap intervals over tracks (one seed).

Track set	Tracks	SWS [95 % CI]	Finished
Campaign (hand-made)	5	1.590 [1.451, 1.783]	100.0 %
Out-of-range test	187	0.875 [0.769, 0.982]	57.8 %
Test (unseen generated)	971	1.214 [1.177, 1.250]	81.5 %
Unfiltered test	200	1.231 [1.147, 1.311]	82.0 %
Own training tracks	1000	1.250 [1.215, 1.284]	83.9 %
Training sample	1000	1.250 [1.215, 1.284]	83.9 %
Validation	197	1.236 [1.153, 1.315]	82.7 %

How to use

Files: final.pt (PyTorch checkpoint: agent and norm state dicts, ppo and env_cfg settings, obs_dim), policy.onnx (observation function + frozen normaliser + actor in one graph, as run by the browser game), run.json, training and validation curves, and per-track evaluation CSVs under eval/.

PyTorch, inside a clone of the repository (uv sync):

import torch
from racingmania.env.obs import obs_size
from racingmania.rl.networks import ActorCritic
from racingmania.rl.normalizer import RunningNorm

sd = torch.load("final.pt", map_location="cpu", weights_only=False)
k = sd["env_cfg"]["window_k"]
agent = ActorCritic(obs_size(k), tuple(sd["ppo"]["hidden"]), sd["ppo"]["action_mode"])
agent.load_state_dict(sd["agent"]); agent.eval()
norm = RunningNorm(obs_size(k)); norm.load_state_dict(sd["norm"]); norm.frozen = True
# action = agent.act(norm(env.observe().float()), deterministic=True)[1]   (see racingmania.runs.load_policy)

ONNX (inputs and outputs are specified in docs/formats.md, section "Policy model"):

import numpy as np, onnxruntime as ort
sess = ort.InferenceSession("policy.onnx")
inputs = {i.name: np.zeros([d if isinstance(d, int) else 1 for d in i.shape], np.float32) for i in sess.get_inputs()}
steer_logits, pedal_logits = sess.run(None, inputs)
steer, pedal = int(steer_logits.argmax()), int(pedal_logits.argmax())  # 0/1/2 = left/none/right, gas/coast/brake

To watch it drive, open the game (web/index.html, or the GitHub Pages site), choose it in a track's AI drivers table and press "Watch AI" (live in the browser) or "Watch replay" (its official evaluation run).

Training configuration (from run.json)

Setting	Value
total_steps	1000000000
num_envs	16384
num_steps	32
lr	0.001
adaptive_lr	True
desired_kl	0.01
lr_min	1e-05
lr_max	0.01
gamma	0.99
gae_lambda	0.95
num_minibatches	4
update_epochs	5
clip_coef	0.2
ent_coef	0.01
vf_coef	1.0
max_grad_norm	1.0
weight_decay	0.0
hidden	[256, 256]
action_mode	factored
eval_every_minutes	10.0
checkpoint_every_minutes	10.0
rotate_tracks_every	20
loaded_tracks	4096
seed	1
compile	True
env.random_start_prob	0.5
env.random_start_speed	[0.0, 30.0]
env.random_start_lateral	0.5
env.max_steps	1500
env.reward_scale	0.1
env.window_k	20
env.window_stride	3
env.blind	False
env.time_cost	0.0
env.fall_penalty	0.0
env.hint_weight	0.0
env.scripted_starts	False

Licence and citation

MIT. Author: Libor "Burny" Burian. Repository: https://github.com/BurnyCoder/reinforcement-learning-in-trackmania-like-racing-game.