racingmania-generalist-seed0

Generalist (PPO) [pilot, seed 0], seed 0: Generalist PPO agent trained on the whole generated training pool

Status: evaluated (evaluated; scores available); physics version v1; card generated 2026-11-15 from the result files.

RacingMania is an original 3D time-attack racing game for the browser (three.js + cannon-es RaycastVehicle) with reinforcement-learning drivers trained on a batched PyTorch port of the same car physics. Code, paper and full results: https://github.com/BurnyCoder/reinforcement-learning-in-trackmania-like-racing-game.

Videos

Official evaluation runs of this model replayed in the browser game ("Watch replay"): the car states the evaluator recorded while scoring the run, so each clip is exactly the run its caption describes. More clips, including the controls: https://github.com/BurnyCoder/reinforcement-learning-in-trackmania-like-racing-game#videos.

Best generalist lap on a hand-made track it never saw in training: it generalized

• What you see: This is generalization: driving well on a track the model never saw in training. The generalist learned on tens of thousands of generated tracks and never on these hand-made ones, which were designed separately. It reads the road ahead, picks a speed for each bend, and laps this track in about half the scripted driver's time. On the specialist's own track it is only slightly slower than the specialist (see the results). Compare clip 7 among the negative results: the specialist on this same track.
• Model: generalist/seed0: PPO, trained by reinforcement learning for 1 billion agent steps on 48,949 generated training tracks (seed 0, 1.3 h on one GPU).
• Map: Technical (campaign-04), a hand-made campaign track, 666 m long with no jumps and up to 5° of banking; this model never saw it during training.
• Evaluation: official evaluation in results/generalist/seed0/eval_campaign.csv (standing start, most probable action, final checkpoint): finished in 21.27 s; the scripted driver needs 42.52 s, so this is 2.00× its speed. The clip replays this run.
• Why this clip: chosen by a fixed rule as the highest-scoring finished track of the 5 campaign tracks it finished in its official evaluation.

Best generalist lap on a held-out generated track it never saw in training: it generalized

• What you see: Generalization again, now on a generated track from the held-out test seeds: the same kind of track as in training, but never this one, driven from the standing start to the finish. Watch it slow down for each bend, hug the inside edge through it and accelerate out again, using only what it sees of the road ahead. Compare it with the next clip, the scripted driver on the same track.
• Model: generalist/seed0: PPO, trained by reinforcement learning for 1 billion agent steps on 48,949 generated training tracks (seed 0, 1.3 h on one GPU).
• Map: Unseen test track 2000747 (test-2000747), a held-out generated test track, 1,461 m long with no jumps and up to 11° of banking; this model never saw it during training.
• Evaluation: official evaluation in results/generalist/seed0/eval_test.csv (standing start, most probable action, final checkpoint): finished in 43.35 s; the scripted driver needs 79.77 s, so this is 1.84× its speed. The clip replays this run.
• Why this clip: chosen by a fixed rule as the highest-scoring finished track of the 779 test tracks it finished in its official evaluation.

Typical generalist lap on a held-out generated track it never saw in training

• What you see: Not the best lap but a typical one: the median of the generalist's finished test tracks that have a jump. It shows what most of its successful laps on unseen tracks look like: a clean run with a jump, clearly faster than the scripted driver but less spectacular than the best lap.
• Model: generalist/seed0: PPO, trained by reinforcement learning for 1 billion agent steps on 48,949 generated training tracks (seed 0, 1.3 h on one GPU).
• Map: Unseen test track 2000386 (test-2000386), a held-out generated test track, 1,097 m long with 1 jump and up to 10° of banking; this model never saw it during training.
• Evaluation: official evaluation in results/generalist/seed0/eval_test.csv (standing start, most probable action, final checkpoint): finished in 33.01 s; the scripted driver needs 49.36 s, so this is 1.50× its speed. The clip replays this run.
• Why this clip: chosen by a fixed rule as the median-scoring finished track among those with at least one jump (397 test tracks with a jump that it finished) in its official evaluation.

Generalization failure on a held-out track: the generalist falls off halfway round

• What you see: Generalization does not always work. About one unseen track in five ends like this: the generalist enters a bend too fast, cannot hold the line and leaves the road on the outside. Most of its falls come right after tight corners rather than at jumps (failure analysis in results/generalist/seed*/failures_*.json): it trades safety for speed.
• Model: generalist/seed0: PPO, trained by reinforcement learning for 1 billion agent steps on 48,949 generated training tracks (seed 0, 1.3 h on one GPU).
• Map: Unseen test track 2000430 (test-2000430), a held-out generated test track, 1,559 m long with no jumps and up to 14° of banking; this model never saw it during training.
• Evaluation: official evaluation in results/generalist/seed0/eval_test.csv (standing start, most probable action, final checkpoint): did not finish: fell after 24.43 s and 776 m (the scripted driver finishes in 66.30 s). The clip replays this run.
• Why this clip: chosen by a fixed rule as the fall whose position is closest to the middle of its track among its 190 falls on test tracks in its official evaluation.

Generalization failure beyond the training range: a corner tighter than any training track

• What you see: This track was generated with tighter corners than any track the generalist trained on. It approaches the corner at a speed that would suit the wider corners it knows and flies off the outside. Learning from many tracks lets it drive new combinations of familiar features, but not features outside what it has seen: on tracks like this it almost always falls, while longer jumps than in training, for example, are handled well (see the results).
• Model: generalist/seed0: PPO, trained by reinforcement learning for 1 billion agent steps on 48,949 generated training tracks (seed 0, 1.3 h on one GPU).
• Map: Out-of-range track 3000123 (oor-3000123), a held-out out-of-range test track with one factor beyond the training range, arc radius 14–22 m (training: 22–120 m), 855 m long with 2 jumps and up to 9° of banking; this model never saw it during training.
• Evaluation: official evaluation in results/generalist/seed0/eval_oor.csv (standing start, most probable action, final checkpoint): did not finish: fell after 17.76 s and 449 m (the scripted driver finishes in 52.16 s). The clip replays this run.
• Why this clip: chosen by a fixed rule as the fall whose position is closest to the middle of its track among its 43 falls on tight-turn out-of-range tracks in its official evaluation.

Model

Item	Value
Algorithm	PPO
Architecture	MLP 2x256 tanh (actor and critic), two 3-way heads
Objective	centreline progress
Observation	car state + 20 look-ahead points
Training data	48,949 generated tracks
Start states	standing + random centreline (p=0.5)
Agent steps trained	1,000,341,504
Wall-clock hours	1.27
Git commit	1c8e972820d695e7e9f0e589e3e475dc9b7b7a16

Evaluation

Success-weighted speed (SWS): mean over tracks of finished x (scripted driver's time / agent's time); 1.0 means finishing every track as fast as the scripted reference driver. One deterministic run per track from a standing start with the most probable action; intervals are 95 % percentile bootstrap intervals over tracks (one seed).

Track set	Tracks	SWS [95 % CI]	Finished
Campaign (hand-made)	5	1.596 [1.447, 1.799]	100.0 %
Out-of-range test	187	0.788 [0.678, 0.899]	52.4 %
Test (unseen generated)	971	1.199 [1.160, 1.237]	80.2 %
Unfiltered test	200	1.227 [1.143, 1.306]	81.5 %
Training sample	1000	1.234 [1.198, 1.269]	82.6 %
Validation	197	1.242 [1.159, 1.319]	82.7 %

How to use

Files: final.pt (PyTorch checkpoint: agent and norm state dicts, ppo and env_cfg settings, obs_dim), policy.onnx (observation function + frozen normaliser + actor in one graph, as run by the browser game), run.json, training and validation curves, and per-track evaluation CSVs under eval/.

PyTorch, inside a clone of the repository (uv sync):

import torch
from racingmania.env.obs import obs_size
from racingmania.rl.networks import ActorCritic
from racingmania.rl.normalizer import RunningNorm

sd = torch.load("final.pt", map_location="cpu", weights_only=False)
k = sd["env_cfg"]["window_k"]
agent = ActorCritic(obs_size(k), tuple(sd["ppo"]["hidden"]), sd["ppo"]["action_mode"])
agent.load_state_dict(sd["agent"]); agent.eval()
norm = RunningNorm(obs_size(k)); norm.load_state_dict(sd["norm"]); norm.frozen = True
# action = agent.act(norm(env.observe().float()), deterministic=True)[1]   (see racingmania.runs.load_policy)

ONNX (inputs and outputs are specified in docs/formats.md, section "Policy model"):

import numpy as np, onnxruntime as ort
sess = ort.InferenceSession("policy.onnx")
inputs = {i.name: np.zeros([d if isinstance(d, int) else 1 for d in i.shape], np.float32) for i in sess.get_inputs()}
steer_logits, pedal_logits = sess.run(None, inputs)
steer, pedal = int(steer_logits.argmax()), int(pedal_logits.argmax())  # 0/1/2 = left/none/right, gas/coast/brake

To watch it drive, open the game (web/index.html, or the GitHub Pages site), choose it in a track's AI drivers table and press "Watch AI" (live in the browser) or "Watch replay" (its official evaluation run).

Training configuration (from run.json)

Setting	Value
total_steps	1000000000
num_envs	16384
num_steps	32
lr	0.001
adaptive_lr	True
desired_kl	0.01
lr_min	1e-05
lr_max	0.01
gamma	0.99
gae_lambda	0.95
num_minibatches	4
update_epochs	5
clip_coef	0.2
ent_coef	0.01
vf_coef	1.0
max_grad_norm	1.0
weight_decay	0.0
hidden	[256, 256]
action_mode	factored
eval_every_minutes	10.0
checkpoint_every_minutes	10.0
rotate_tracks_every	20
loaded_tracks	4096
seed	0
compile	True
env.random_start_prob	0.5
env.random_start_speed	[0.0, 30.0]
env.random_start_lateral	0.5
env.max_steps	1500
env.reward_scale	0.1
env.window_k	20
env.window_stride	3
env.blind	False
env.time_cost	0.0
env.fall_penalty	0.0
env.hint_weight	0.0
env.scripted_starts	False

Licence and citation

MIT. Author: Libor "Burny" Burian. Repository: https://github.com/BurnyCoder/reinforcement-learning-in-trackmania-like-racing-game.