seungukdo's picture
|
download
raw
4.5 kB
# EgomodoV2 training runs on NymeriaPlus (8x H200)
All runs train `EgomodoModel` with the two-stage denoiser (latent 1024, 4 blocks per stage, 8 heads) on NymeriaPlus,
with 8 GPUs, bf16, weight decay 0.01, gradient clipping 1.0, 1,000 warm-up steps, learning-rate decay to 10% of the peak and EMA 0.999.
Validation uses 64 clips from the training sequences every 2,500 steps. Test uses 773 clips from 4 unseen sequences
and was run only at the end of the finished LR-sweep runs. Errors are in metres; semantic similarity is TMR (higher is better).
## Learning-rate sweep (20k steps, batch 128 per GPU, QK-norm off)
| lr | best val MPJPE | test MPJPE | test MPJPE-PA | test hand MPJPE | test root traj. | test semantic sim. | test FID | result |
|---|---|---|---|---|---|---|---|---|
| 0.0005 | 0.0739 | 0.104 | 0.0681 | 0.2348 | 0.0699 | 0.8517 | 0.099 | finished |
| 0.001 | 0.0712 | 0.0981 | 0.0642 | 0.2233 | 0.0665 | 0.8584 | 0.0862 | finished |
| 0.002 | 0.1318 | 0.1303 | 0.0808 | 0.287 | 0.0864 | 0.8048 | 0.1289 | finished |
| 0.005 | - | - | - | - | - | - | - | NaN at step 1,250, crashed |
lr 1e-3 is best on every test metric and its validation error was still falling at 20k steps.
## Long runs (planned 200k steps)
| run | lr | batch per GPU | QK-norm | steps | first NaN step | best val MPJPE (step) | kept checkpoint |
|---|---|---|---|---|---|---|---|
| egomodo_nymeria_h200_b128x8 | 0.0005 | 128 | False | 50-57500 | 56350 | 0.0775 (55000) | egomodo_nymeria_h200_b128x8/checkpoints/step_0050000.pt |
| egomodo_nymeria_h200_b128x8_r50k | 0.00025 | 128 | False | 50050-60000 | 59400 | 0.0774 (55000) | egomodo_nymeria_h200_b128x8_r50k/checkpoints/step_0055000.pt |
| egomodo_nymeria_h200_b128x8_qknorm | 0.0005 | 128 | True | 50-5400 | - | 0.1317 (5000) | egomodo_nymeria_h200_b128x8_qknorm/checkpoints/step_0005000.pt |
| egomodo_nymeria_h200_b64x8 | 0.0005 | 64 | False | 50-13500 | - | 0.0984 (12500) | egomodo_nymeria_h200_b64x8/checkpoints/step_0010000.pt |
The long runs without QK-norm diverged near step 56k at both lr 5e-4 and 2.5e-4. In the saved weights the largest
attention logit scale, max over heads of ||Wq_h^T Wk_h||_2 / sqrt(128), of the last body block grew from 1.0 (10k) to 39.5 (30k)
and 391 (55k). The QK-norm option (RMSNorm on per-head queries and keys) is meant to stop this; its run was stopped
at 5.4k steps, so it has not yet been tested past the divergence point.
## Notes on each run
- `egomodo_lrsweep20k_lr5e-4`: LR sweep, lr 5e-4, 20k steps, no QK-norm. Finished.
- `egomodo_lrsweep20k_lr1e-3`: LR sweep, lr 1e-3, 20k steps, no QK-norm. Finished; best run of the sweep.
- `egomodo_lrsweep20k_lr2e-3`: LR sweep, lr 2e-3, 20k steps, no QK-norm. Finished; learns much more slowly.
- `egomodo_lrsweep20k_lr5e-3`: LR sweep, lr 5e-3, no QK-norm. Loss became NaN at step 1,250 and the run crashed at the 2.5k validation. No checkpoint.
- `egomodo_nymeria_h200_b128x8`: Main run, lr 5e-4, planned 200k steps. Diverged (NaN from step 56,350) and crashed at the 57.5k validation. Last checkpoint before divergence: 50k.
- `egomodo_nymeria_h200_b128x8_r50k`: Resumed from the main run's 50k checkpoint with lr 2.5e-4. Diverged again (NaN from step 59,400). Kept checkpoint 55k has the best validation MPJPE of all long runs.
- `egomodo_nymeria_h200_b128x8_qknorm`: Main-run settings from scratch with QK-norm on. Stopped by hand at 5.4k steps to investigate a slowdown.
- `egomodo_nymeria_h200_b64x8`: First attempt with batch 64 per GPU. Stopped at 13.5k steps in favour of batch 128 per GPU.
## Files
- `summary.csv`: one row per run with the numbers above.
- `<run>/metrics.jsonl`: training logs every 50 steps, validation every 2,500 steps and, for finished sweep runs, the test metrics.
- `<run>/config.json`: the full training configuration.
- `<run>/test/`: test metrics (`metrics.json`, `metrics_summary.json`) and the generated clips per test sequence (safetensors).
- `<run>/checkpoints/step_XXXXXXX.pt`: one checkpoint per run, as saved by the trainer.
The QK-norm run needs the `qk_norm` option of `TwostageDenoiserConfig`, which is not in upstream EgomodoV2 yet.
## EgomodoV2 text experiments (2026-10-30 to 10-31)
The folder `egomodov2_text/` holds four EgomodoV2 runs with and without the motion description text: two 20k-step runs with
test evaluation (with text: test MPJPE 0.0840, semantic similarity 0.891, FID 0.075; without text: 0.0904, 0.855, 0.093) and
two 100k-step runs that diverged. See `egomodov2_text/README.md`.

Xet Storage Details

Size:
4.5 kB
·
Xet hash:
2af5150118ff5d7e28932a4d6875f8c8e7a2cdb21ea89d11462e7dc9dd7f8aeb

Xet efficiently stores files, intelligently splitting them into unique chunks and accelerating uploads and downloads. More info.