Buckets:

seungukdo
/

motion_trainings

Files

xet

seungukdo/motion_trainings / summary.csv

seungukdo

10 days ago

download

raw

2.17 kB

run,learning_rate,batch_per_gpu,qk_norm,first_step,last_step,first_nan_step,best_val_step,best_val_mpjpe,test_mpjpe,test_mpjpe_pa,test_mpjpe_hand,test_traj_trans_error,test_semantic_similarity,test_fid,checkpoint,note
egomodo_lrsweep20k_lr5e-4,0.0005,128,False,50,20000,,20000,0.0739,0.104,0.0681,0.2348,0.0699,0.8517,0.099,egomodo_lrsweep20k_lr5e-4/checkpoints/step_0020000.pt,"LR sweep, lr 5e-4, 20k steps, no QK-norm. Finished."
egomodo_lrsweep20k_lr1e-3,0.001,128,False,50,20000,,20000,0.0712,0.0981,0.0642,0.2233,0.0665,0.8584,0.0862,egomodo_lrsweep20k_lr1e-3/checkpoints/step_0020000.pt,"LR sweep, lr 1e-3, 20k steps, no QK-norm. Finished; best run of the sweep."
egomodo_lrsweep20k_lr2e-3,0.002,128,False,50,20000,,20000,0.1318,0.1303,0.0808,0.287,0.0864,0.8048,0.1289,egomodo_lrsweep20k_lr2e-3/checkpoints/step_0020000.pt,"LR sweep, lr 2e-3, 20k steps, no QK-norm. Finished; learns much more slowly."
egomodo_lrsweep20k_lr5e-3,0.005,128,False,50,2500,1250,,,,,,,,,,"LR sweep, lr 5e-3, no QK-norm. Loss became NaN at step 1,250 and the run crashed at the 2.5k validation. No checkpoint."
egomodo_nymeria_h200_b128x8,0.0005,128,False,50,57500,56350,55000,0.0775,,,,,,,egomodo_nymeria_h200_b128x8/checkpoints/step_0050000.pt,"Main run, lr 5e-4, planned 200k steps. Diverged (NaN from step 56,350) and crashed at the 57.5k validation. Last checkpoint before divergence: 50k."
egomodo_nymeria_h200_b128x8_r50k,0.00025,128,False,50050,60000,59400,55000,0.0774,,,,,,,egomodo_nymeria_h200_b128x8_r50k/checkpoints/step_0055000.pt,"Resumed from the main run's 50k checkpoint with lr 2.5e-4. Diverged again (NaN from step 59,400). Kept checkpoint 55k has the best validation MPJPE of all long runs."
egomodo_nymeria_h200_b128x8_qknorm,0.0005,128,True,50,5400,,5000,0.1317,,,,,,,egomodo_nymeria_h200_b128x8_qknorm/checkpoints/step_0005000.pt,Main-run settings from scratch with QK-norm on. Stopped by hand at 5.4k steps to investigate a slowdown.
egomodo_nymeria_h200_b64x8,0.0005,64,False,50,13500,,12500,0.0984,,,,,,,egomodo_nymeria_h200_b64x8/checkpoints/step_0010000.pt,First attempt with batch 64 per GPU. Stopped at 13.5k steps in favour of batch 128 per GPU.

Xet Storage Details

Size:: 2.17 kB
Xet hash:: b33a57cc6c55a7ef93baf0f6711def2d1c4468027e775d57aff6f8d9bcabae22

Xet efficiently stores files, intelligently splitting them into unique chunks and accelerating uploads and downloads. More info.

	run,learning_rate,batch_per_gpu,qk_norm,first_step,last_step,first_nan_step,best_val_step,best_val_mpjpe,test_mpjpe,test_mpjpe_pa,test_mpjpe_hand,test_traj_trans_error,test_semantic_similarity,test_fid,checkpoint,note
	egomodo_lrsweep20k_lr5e-4,0.0005,128,False,50,20000,,20000,0.0739,0.104,0.0681,0.2348,0.0699,0.8517,0.099,egomodo_lrsweep20k_lr5e-4/checkpoints/step_0020000.pt,"LR sweep, lr 5e-4, 20k steps, no QK-norm. Finished."
	egomodo_lrsweep20k_lr1e-3,0.001,128,False,50,20000,,20000,0.0712,0.0981,0.0642,0.2233,0.0665,0.8584,0.0862,egomodo_lrsweep20k_lr1e-3/checkpoints/step_0020000.pt,"LR sweep, lr 1e-3, 20k steps, no QK-norm. Finished; best run of the sweep."
	egomodo_lrsweep20k_lr2e-3,0.002,128,False,50,20000,,20000,0.1318,0.1303,0.0808,0.287,0.0864,0.8048,0.1289,egomodo_lrsweep20k_lr2e-3/checkpoints/step_0020000.pt,"LR sweep, lr 2e-3, 20k steps, no QK-norm. Finished; learns much more slowly."
	egomodo_lrsweep20k_lr5e-3,0.005,128,False,50,2500,1250,,,,,,,,,,"LR sweep, lr 5e-3, no QK-norm. Loss became NaN at step 1,250 and the run crashed at the 2.5k validation. No checkpoint."
	egomodo_nymeria_h200_b128x8,0.0005,128,False,50,57500,56350,55000,0.0775,,,,,,,egomodo_nymeria_h200_b128x8/checkpoints/step_0050000.pt,"Main run, lr 5e-4, planned 200k steps. Diverged (NaN from step 56,350) and crashed at the 57.5k validation. Last checkpoint before divergence: 50k."
	egomodo_nymeria_h200_b128x8_r50k,0.00025,128,False,50050,60000,59400,55000,0.0774,,,,,,,egomodo_nymeria_h200_b128x8_r50k/checkpoints/step_0055000.pt,"Resumed from the main run's 50k checkpoint with lr 2.5e-4. Diverged again (NaN from step 59,400). Kept checkpoint 55k has the best validation MPJPE of all long runs."
	egomodo_nymeria_h200_b128x8_qknorm,0.0005,128,True,50,5400,,5000,0.1317,,,,,,,egomodo_nymeria_h200_b128x8_qknorm/checkpoints/step_0005000.pt,Main-run settings from scratch with QK-norm on. Stopped by hand at 5.4k steps to investigate a slowdown.
	egomodo_nymeria_h200_b64x8,0.0005,64,False,50,13500,,12500,0.0984,,,,,,,egomodo_nymeria_h200_b64x8/checkpoints/step_0010000.pt,First attempt with batch 64 per GPU. Stopped at 13.5k steps in favour of batch 128 per GPU.