racingmania-pqn-seed1

PQN, seed 1: PQN baseline

Status: evaluated (evaluated; scores available); physics version v1; card generated 2026-11-15 from the result files.

RacingMania is an original 3D time-attack racing game for the browser (three.js + cannon-es RaycastVehicle) with reinforcement-learning drivers trained on a batched PyTorch port of the same car physics. Code, paper and full results: https://github.com/BurnyCoder/reinforcement-learning-in-trackmania-like-racing-game.

Videos

Official evaluation runs of this model replayed in the browser game ("Watch replay"): the car states the evaluator recorded while scoring the run, so each clip is exactly the run its caption describes. More clips, including the controls: https://github.com/BurnyCoder/reinforcement-learning-in-trackmania-like-racing-game#videos.

Best lap of the PQN baseline

• What you see: The baseline algorithm, PQN (parallelised Q-learning), trained with the same inputs, actions, reward and step budget as the PPO generalist on generated tracks. The track was never part of its training.
• Model: pqn/seed1: PQN, trained by reinforcement learning for 1 billion agent steps on 48,949 generated training tracks (seed 1, 1.1 h on one GPU).
• Map: Technical (campaign-04), a hand-made campaign track, 666 m long with no jumps and up to 5° of banking; this model never saw it during training.
• Evaluation: official evaluation in results/pqn/seed1/eval_campaign.csv (standing start, most probable action, final checkpoint): finished in 26.04 s; the scripted driver needs 42.52 s, so this is 1.63× its speed. The clip replays this run.
• Why this clip: chosen by a fixed rule as the highest-scoring finished track of the 5 campaign tracks it finished in its official evaluation.

Model

Item	Value
Algorithm	PQN
Architecture	Q-network 2x256, LayerNorm, one 9-way head
Objective	centreline progress
Observation	car state + 20 look-ahead points
Training data	48,949 generated tracks
Start states	standing + random centreline (p=0.5)
Agent steps trained	1,000,341,504
Wall-clock hours	1.10
Git commit	70a35809b494c89e7fa074064a9c8dc961ef0360

Evaluation

Success-weighted speed (SWS): mean over tracks of finished x (scripted driver's time / agent's time); 1.0 means finishing every track as fast as the scripted reference driver. One deterministic run per track from a standing start with the most probable action; intervals are 95 % percentile bootstrap intervals over tracks (one seed).

Track set	Tracks	SWS [95 % CI]	Finished
Campaign (hand-made)	5	1.391 [1.270, 1.517]	100.0 %
Out-of-range test	187	0.725 [0.631, 0.820]	55.1 %
Test (unseen generated)	971	1.054 [1.020, 1.086]	79.8 %
Unfiltered test	200	1.105 [1.035, 1.173]	83.0 %
Own training tracks	1000	1.052 [1.019, 1.085]	79.7 %
Training sample	1000	1.052 [1.019, 1.085]	79.7 %
Validation	197	1.034 [0.956, 1.109]	78.2 %

How to use

Files: final.pt (PyTorch checkpoint: agent and norm state dicts, ppo and env_cfg settings, obs_dim), policy.onnx (observation function + frozen normaliser + actor in one graph, as run by the browser game), run.json, training and validation curves, and per-track evaluation CSVs under eval/.

PyTorch, inside a clone of the repository (uv sync):

PQN checkpoints hold a Q-network; load them with racingmania.rl.pqn (see the repository).

ONNX (inputs and outputs are specified in docs/formats.md, section "Policy model"):

import numpy as np, onnxruntime as ort
sess = ort.InferenceSession("policy.onnx")
inputs = {i.name: np.zeros([d if isinstance(d, int) else 1 for d in i.shape], np.float32) for i in sess.get_inputs()}
steer_logits, pedal_logits = sess.run(None, inputs)
steer, pedal = int(steer_logits.argmax()), int(pedal_logits.argmax())  # 0/1/2 = left/none/right, gas/coast/brake

To watch it drive, open the game (web/index.html, or the GitHub Pages site), choose it in a track's AI drivers table and press "Watch AI" (live in the browser) or "Watch replay" (its official evaluation run).

Training configuration (from run.json)

Setting	Value
total_steps	1000000000
num_envs	16384
num_steps	32
lr	0.00025
adaptive_lr	False
desired_kl	0.01
lr_min	1e-05
lr_max	0.01
gamma	0.99
gae_lambda	0.95
num_minibatches	4
update_epochs	4
clip_coef	0.2
ent_coef	0.01
vf_coef	1.0
max_grad_norm	10.0
weight_decay	0.0
hidden	[256, 256]
action_mode	factored
eval_every_minutes	10.0
checkpoint_every_minutes	10.0
rotate_tracks_every	20
loaded_tracks	4096
seed	1
compile	True
q_lambda	0.65
start_e	1.0
end_e	0.05
exploration_fraction	0.5
env.random_start_prob	0.5
env.random_start_speed	[0.0, 30.0]
env.random_start_lateral	0.5
env.max_steps	1500
env.reward_scale	0.1
env.window_k	20
env.window_stride	3
env.blind	False
env.time_cost	0.0
env.fall_penalty	0.0
env.hint_weight	0.0
env.scripted_starts	False

Licence and citation

MIT. Author: Libor "Burny" Burian. Repository: https://github.com/BurnyCoder/reinforcement-learning-in-trackmania-like-racing-game.