sebastian-stapf/action-progress-video-checkpoints

Action-progress control for LTX-2.5: checkpoints

LoRA and conditioning modules that let LTX-2.5 (22B, image-to-video) follow a requested action-progress curve: the same first frame and caption can be generated with the action happening earlier or later than it naturally would. Project: Whose Time Is It? Instance-Specific Action-Progress Control for Video Editing.

Each dataset folder holds the run's best and last checkpoint. "Best" is the evaluated checkpoint with the highest early/late gain G; "last" is the final training step.

Runs

dataset	folder	training data	run
DROID	droid/	3,193 clips, 960x544, 24 FPS	droid_s31_ltxcap_20261023-080331
Pouring	pouring/	181 clips, 512x928, 24 FPS	pouring_s31_20261024-125034
Penn Action	penn_action/	308 windows from 179 clips, 928x512, 24 FPS	penn_s31_20261024-124917
Ego4D (6,032 clips)	ego4d_6k/	6,032 clips, 896x672, 30 FPS	ego4d896_s31_ltxcap_20261023-081248
Ego4D (16,750 clips)	ego4d_17k/	16,750 clips, 896x672, 30 FPS; stopped at 20,000 steps	ego4d896_s31_ltxcap_20261024-185921
HOI multi-actor	hoi_multi/	1,479 HOIGen clips with 2-4 actors (steps 1-1,840), then 1,479 single-actor clips added (to 3,690); 960x544	multi_s31_ltxcap_20261023-081149

All runs: LTX-2.5 22B dev transformer (bf16) with a LoRA, a progress encoder (AdaLN) that conditions on the requested curve, and an actor branch tied to the actor's frame-0 mask (multi-actor: one curve per actor, routed by its mask). Training uses each natural clip plus RIFE-retimed variants (forward, rewind, zigzag). 2 nodes x 4 GH200 (Ego4D 16,750: 4 nodes), batch 1 per GPU, learning rate 1e-4.

Checkpoints and evaluation

dataset	folder	step	kind	gain G [95% CI]	E_p	FVD
DROID	droid/	3,192	best	1.60 [1.20, 1.97]	0.251	554
DROID	droid/	3,990	last	1.10 [0.71, 1.55]	0.250	638
Pouring	pouring/	1,320	best and last	1.25 [1.02, 1.48]	0.172	1989
Penn Action	penn_action/	500	best	0.30 [0.17, 0.42]	0.185	967
Penn Action	penn_action/	1,520	last	0.25 [0.06, 0.43]	0.208	879
Ego4D (6,032 clips)	ego4d_6k/	7,540	best and last	0.19 [0.11, 0.27]	0.183	794
Ego4D (16,750 clips)	ego4d_17k/	20,000	best and last	0.17 [0.09, 0.26]	0.188	815
HOI multi-actor	hoi_multi/	3,690	best and last	0.18 [0.08, 0.28]	0.247	1383

Evaluation: each model regenerates held-out clips (48 per dataset, Pouring 10) from their first frame and caption under the native curve and the curve retimed early and late (h(u) = u + d sin(pi u) / pi, d = +/-0.6). Gemma-4-31B reads the progress of each video at 16 points. Gain G is the early-minus-late response over the requested difference (1 = follows the request, 0 = ignores it). E_p is the mean distance to the requested curve. FVD (I3D) compares against the real clips; on 48 videos compare it only within one dataset. The Ego4D 16,750 run is scored on clips it trained on: every labelled Ego4D window is in its training set. All checkpoints' rows are in <folder>/eval.json.

Ego4D with ground-truth progress (ego4d_gt/)

Ego4D clips whose progress curve comes from the FHO critical frames instead of a VLM: the pre frame maps to 0, the point of no return to 0.5 and the post frame to 1, linear in between and flat outside. 13,379 clips (the 26 verbs with more than 100 clips, 896x672, 30 FPS); each mixed epoch shows the versions 40% natural, 30% forward, 20% rewind, 10% zigzag after one natural epoch. Same model and training setup as the other folders (2 nodes x 4 GH200, batch 1 per GPU). Run ego4d896_s31_gt_20261027-165650.

step	kind
5,000	intermediate
12,250	intermediate
16,720	last (final training step, 2026-10-28)

These checkpoints are not evaluated yet. They load like the other single-actor folders.

Loading

The configs name the base weights (LTX-2.5 22B dev transformer, Gemma text encoder, video VAE) by their paths on the training cluster; point model.model_path, model.text_encoder_path and the VAE paths to your copies. Code: the ProgGen fork (github.com/gerardDonahue/ProgGen-NEU-BERN, branch rife16-todi). Single-actor folders load with external_scripts.actor_progress.evaluate.load_model (LoRA, progress encoder and actor branch of one step). The multi-actor folder attaches multi_actor_at_NNNNN.safetensors with external_scripts.multi_actor_rife16.model.attach.

Licence

Derived from LTX-2 and subject to its licence. The weights were trained on DROID, Ego4D, Penn Action, a pouring dataset and HOIGen-1M clips; their terms apply to any use.