EgomodoV2 training runs on NymeriaPlus (8x H200)

All runs train EgomodoModel with the two-stage denoiser (latent 1024, 4 blocks per stage, 8 heads) on NymeriaPlus, with 8 GPUs, bf16, weight decay 0.01, gradient clipping 1.0, 1,000 warm-up steps, learning-rate decay to 10% of the peak and EMA 0.999. Validation uses 64 clips from the training sequences every 2,500 steps. Test uses 773 clips from 4 unseen sequences and was run only at the end of the finished LR-sweep runs. Errors are in metres; semantic similarity is TMR (higher is better).

Learning-rate sweep (20k steps, batch 128 per GPU, QK-norm off)

lr 1e-3 is best on every test metric and its validation error was still falling at 20k steps.

Long runs (planned 200k steps)

The long runs without QK-norm diverged near step 56k at both lr 5e-4 and 2.5e-4. In the saved weights the largest attention logit scale, max over heads of ||Wq_h^T Wk_h||_2 / sqrt(128), of the last body block grew from 1.0 (10k) to 39.5 (30k) and 391 (55k). The QK-norm option (RMSNorm on per-head queries and keys) is meant to stop this; its run was stopped at 5.4k steps, so it has not yet been tested past the divergence point.

Notes on each run

• egomodo_lrsweep20k_lr5e-4: LR sweep, lr 5e-4, 20k steps, no QK-norm. Finished.
• egomodo_lrsweep20k_lr1e-3: LR sweep, lr 1e-3, 20k steps, no QK-norm. Finished; best run of the sweep.
• egomodo_lrsweep20k_lr2e-3: LR sweep, lr 2e-3, 20k steps, no QK-norm. Finished; learns much more slowly.
• egomodo_lrsweep20k_lr5e-3: LR sweep, lr 5e-3, no QK-norm. Loss became NaN at step 1,250 and the run crashed at the 2.5k validation. No checkpoint.
• egomodo_nymeria_h200_b128x8: Main run, lr 5e-4, planned 200k steps. Diverged (NaN from step 56,350) and crashed at the 57.5k validation. Last checkpoint before divergence: 50k.
• egomodo_nymeria_h200_b128x8_r50k: Resumed from the main run's 50k checkpoint with lr 2.5e-4. Diverged again (NaN from step 59,400). Kept checkpoint 55k has the best validation MPJPE of all long runs.
• egomodo_nymeria_h200_b128x8_qknorm: Main-run settings from scratch with QK-norm on. Stopped by hand at 5.4k steps to investigate a slowdown.
• egomodo_nymeria_h200_b64x8: First attempt with batch 64 per GPU. Stopped at 13.5k steps in favour of batch 128 per GPU.

Files

• summary.csv: one row per run with the numbers above.
• <run>/metrics.jsonl: training logs every 50 steps, validation every 2,500 steps and, for finished sweep runs, the test metrics.
• <run>/config.json: the full training configuration.
• <run>/test/: test metrics (metrics.json, metrics_summary.json) and the generated clips per test sequence (safetensors).
• <run>/checkpoints/step_XXXXXXX.pt: one checkpoint per run, as saved by the trainer.

The QK-norm run needs the qk_norm option of TwostageDenoiserConfig, which is not in upstream EgomodoV2 yet.

EgomodoV2 text experiments (2026-10-30 to 10-31)

The folder egomodov2_text/ holds four EgomodoV2 runs with and without the motion description text: two 20k-step runs with test evaluation (with text: test MPJPE 0.0840, semantic similarity 0.891, FID 0.075; without text: 0.0904, 0.855, 0.093) and two 100k-step runs that diverged. See egomodov2_text/README.md.