CONTROL Resonant IDM

An inverse dynamics model for the game CONTROL Resonant. It watches gameplay video and predicts what the gamepad was doing on every frame: the 21 buttons and both analog sticks. The purpose is to label gameplay footage that has no recorded input, so it can be used to train a playing agent.

This is a single-game model. Models for other games live next to it in the games_idm collection and are kept separate for now.

Results

Three recording sessions that the model never saw during training (88 minutes in total), scored against the real gamepad input recorded alongside the video:

session length stick error stick error, sticks at rest button F1 button AP
A 13.6 min 0.121 0.180 0.36 0.42
B 51.8 min 0.086 0.142 0.29 0.30
C 21.8 min 0.108 0.178 0.27 0.27

Stick error is the mean absolute error over the four stick axes, each in [-1, 1]. "Sticks at rest" is the error of always predicting centred sticks, which is the baseline to beat. Button F1 and AP are averaged over the buttons that were pressed at least once in that session, so the three rows are not averaged over the same set of buttons.

Per-button F1 on the same sessions:

button A B C
Left trigger 0.68 0.58 0.64
X (west) 0.59 0.62 0.55
A (south) 0.57 0.61 0.50
B (east) 0.52 0.50 0.51
Left bumper 0.32 0.46 0.47
Right bumper 0.28 0.20 0.21
Y (north) 0.20 0.17 0.19
D-pad down โ€“ 0.15 0.20
Left stick click 0.06 0.10 0.09
Start 0.00 0.05 0.07

What this means in practice:

  • Stick labels are useful: the error is 35 to 40 percent below the at-rest baseline.
  • The main action buttons are right about half of the time or better.
  • Rare buttons (D-pad, stick clicks, Start, right trigger) are not learned yet. They are almost absent from the training data.
  • A press that changes nothing on screen cannot be recovered from video by any model.

Usage

pip install torch transformers safetensors numpy
# ffmpeg must be on PATH
from huggingface_hub import snapshot_download
import sys

folder = snapshot_download("XvKuoMing/control-resonant-idm")
sys.path.insert(0, folder)
from idm import load, label_video, BUTTONS

model = load(folder)                     # cuda by default, load(folder, "cpu") also works
out = label_video(model, "gameplay.mp4")

out["buttons"]   # (frames, 21) uint8, column order = BUTTONS
out["probs"]     # (frames, 21) float32, probability of each button being held
out["sticks"]    # (frames, 4) float32: left x, left y, right x, right y

Frame k of the output is video time k / 60 seconds. Sticks are in [-1, 1] with x to the right and y down. label_video decodes the whole file into memory (about 12 GB per hour of video), so split long recordings into pieces of a few minutes, or decode in chunks and call label_frames.

From the command line, python idm.py gameplay.mp4 writes gameplay.actions.npz.

Input

  • Video resampled to 60 fps.
  • The full 16:9 frame squashed to 256ร—256 (no crop), RGB.
  • The model was trained on 1080p captures with the in-game HUD visible. Other resolutions work as long as the whole frame is passed in.

Architecture

part detail
frame encoder SigLIP 2 large, patch 16, 256 px (fine-tuned)
tokens per frame 16ร—16 patches projected to 512 dimensions, average-pooled to 4ร—4
temporal model 4-layer transformer, 8 heads, over all tokens of the window
window 8 frames, every 2nd frame at 60 fps (267 ms)
outputs per frame 21 button logits, 4 stick values through tanh
parameters 329 M

Button order: BACK, DPAD_DOWN, DPAD_LEFT, DPAD_RIGHT, DPAD_UP, EAST, GUIDE, LEFT_SHOULDER, LEFT_THUMB, LEFT_TRIGGER, NORTH, RIGHT_BOTTOM, RIGHT_LEFT, RIGHT_RIGHT, RIGHT_SHOULDER, RIGHT_THUMB, RIGHT_TRIGGER, RIGHT_UP, SOUTH, START, WEST. EAST/SOUTH/WEST/NORTH are B/A/X/Y on an Xbox pad. The triggers are treated as buttons (held or not).

Training

  • Data: 8 recording sessions of one player, 6.4 hours, with gamepad input logged at 1000 Hz and aligned to the video. The recordings are published as the control-resonant dataset: sessions s01 to s08 are the training data, s09 to s11 are the three held-out sessions in the results above.
  • Validation: a contiguous block of each session (from 30% to 40% of its length), never used for training.
  • 8000 steps, batch 16, AdamW, learning rate 1e-4 (1e-5 for the frame encoder), weight decay 0.01.
  • Loss: binary cross-entropy on buttons, with positives of rare buttons weighted up to 10ร—, plus L1 on sticks.
  • Frames with no input at all were thinned to 30% so that idle time does not dominate.
  • This checkpoint is the one with the lowest validation loss (step 7000). Validation scores were still improving when training stopped, so more data and more steps should help.

Limitations

  • One game, one player, one controller layout, one graphics setup.
  • Trained on 6.4 hours. Expect it to fail on scenes it has rarely seen, such as menus and cutscenes.
  • Button labels are too noisy to be used as ground truth without filtering by probability.
Downloads last month
-
Safetensors
Model size
0.3B params
Tensor type
F32
ยท
Inference Providers NEW
This model isn't deployed by any Inference Provider. ๐Ÿ™‹ Ask for provider support

Model tree for XvKuoMing/control-resonant-idm

Finetuned
(7)
this model

Dataset used to train XvKuoMing/control-resonant-idm

Collection including XvKuoMing/control-resonant-idm