Prahaladha commited on
Commit
c51b2b8
·
verified ·
1 Parent(s): c1053db

sync main: 6 files from /dev/shm/mwm/logs

Browse files
logs/export_tf_nomas4_600.log ADDED
@@ -0,0 +1,5 @@
 
 
 
 
 
 
1
+ consolidating DCP shards from /dev/shm/mwm/runs/tf_nomas4/ckpts/checkpoint_600 (key='model') …
2
+ /dev/shm/mwm/venv/lib/python3.12/site-packages/torch/distributed/checkpoint/state_dict_loader.py:362: UserWarning: torch.distributed is unavailable or uninitialized, assuming the intent is to load in a single process.
3
+ warnings.warn(
4
+ loaded 885 parameter tensors
5
+ saved .pt → /dev/shm/mwm/exports/tf_nomas4/step_600.pt (5683 MB)
logs/fix_q001b.log ADDED
@@ -0,0 +1,6 @@
 
 
 
 
 
 
 
1
+ 01:25:59 q001b real copy: 7.9G; dirs: shard_q000 shard_q001 shard_q001b shard_q002 shard_rehearse
2
+ 01:25:59 relaunched; waiting for the first step
3
+ [10/31 01:27:15 minwm.engine.checkpoint.checkpointer]: loaded checkpoint /dev/shm/mwm/runs/tf_nomas4/ckpts/checkpoint_600
4
+ [10/31 01:28:58 minwm.engine.monitor]: step 610 time/data(ms): 1.719 time/forward(ms): 2271 time/backward(ms): 6601 grad_norm: 0.1057 loss: 0.1473 steps/s
5
+ 01:28:59 checkpoint tester still running
6
+ 01:28:59 FIX_Q001B_DONE
logs/fold_events.log CHANGED
@@ -1,2 +1,4 @@
1
  01:12:50 quality dirs: shard_q000 shard_q001 shard_q001b shard_q002 shard_rehearse
2
  01:12:50 waiting for the next tf_nomas4 checkpoint
 
 
 
1
  01:12:50 quality dirs: shard_q000 shard_q001 shard_q001b shard_q002 shard_rehearse
2
  01:12:50 waiting for the next tf_nomas4 checkpoint
3
+ 01:23:50 fold-in at checkpoint_600
4
+ 01:23:55 old trainer processes left: 0
logs/play_tf_nomas4_t3_600.log ADDED
File without changes
logs/prune.log CHANGED
@@ -13,3 +13,5 @@ pruned export (on HF) /dev/shm/mwm/exports/tf_nomas2/step_1000.pt
13
  22:23:37 pruned tf_nomas3 checkpoint_500 (export kept)
14
  pruned export (on HF) /dev/shm/mwm/exports/tf_nomas3/step_250.pt
15
  pruned export (on HF) /dev/shm/mwm/exports/tf_nomas3/step_500.pt
 
 
 
13
  22:23:37 pruned tf_nomas3 checkpoint_500 (export kept)
14
  pruned export (on HF) /dev/shm/mwm/exports/tf_nomas3/step_250.pt
15
  pruned export (on HF) /dev/shm/mwm/exports/tf_nomas3/step_500.pt
16
+ 01:23:42 pruned tf_nomas4 checkpoint_200 (export kept)
17
+ pruned export (on HF) /dev/shm/mwm/exports/tf_nomas4/step_200.pt
logs/train_tf_nomas4.until600.log ADDED
@@ -0,0 +1,517 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ MWM_TRAIN3 repo /dev/shm/mwm/repo data /dev/shm/mwm/data/quality char_labels none char_k 200 drop 0.1
2
+ MWM_TRAIN config configs/wan21/action2v/train/stage1_ar_tf_aug.py
3
+ MWM_TRAIN start 23:38:58 stage tf name tf_nomas4 gpus 0,1,2,3,4,5 n 6 steps 1200 lr 5e-6 init /dev/shm/mwm/exports/tf_nomas3/step_1000.pt teacher same extra
4
+ [10/30 23:39:03 minwm.distributed.sp.parallel_state]: Initializing distributed environment with world_size=6, device=cuda:0
5
+ [10/30 23:39:03 minwm.distributed.sp.parallel_state]: world_size=6 rank=0 local_rank=0 distributed_init_method=env:// backend=nccl
6
+ [Gloo] Rank [Gloo] Rank [Gloo] Rank 30 is connected to is connected to 545 peer ranks. is connected to peer ranks. Expected number of connected peer ranks is : 5Expected number of connected peer ranks is : 5 peer ranks. 5Expected number of connected peer ranks is :
7
+
8
+ 5
9
+ [Gloo] Rank 1 is connected to 5 peer ranks. Expected number of connected peer ranks is : 5
10
+ [Gloo] Rank 2 is connected to 5 peer ranks. Expected number of connected peer ranks is : 5
11
+ [Gloo] Rank 5 is connected to 5 peer ranks. Expected number of connected peer ranks is : 5
12
+ [10/30 23:39:07 minwm.distributed.sp.nccl]: Found nccl from library libnccl.so.2
13
+ [10/30 23:39:07 minwm.distributed.sp.device_communicators.pynccl]: Trainer is using nccl==2.27.5
14
+ [Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
15
+ [Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
16
+ [Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
17
+ [Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
18
+ [Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
19
+ [Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
20
+ [Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
21
+ [Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
22
+ [Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
23
+ [Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
24
+ [Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
25
+ [Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
26
+ [Gloo] Rank 0 is connected to 5 peer ranks. Expected number of connected peer ranks is : 5
27
+ [Gloo] Rank 2 is connected to 5 peer ranks. Expected number of connected peer ranks is : 5
28
+ [Gloo] Rank 5 is connected to 5 peer ranks. Expected number of connected peer ranks is : 5
29
+ [Gloo] Rank 1 is connected to 5 peer ranks. Expected number of connected peer ranks is : 5
30
+ [Gloo] Rank 4 is connected to 5 peer ranks. [Gloo] Rank Expected number of connected peer ranks is : 53
31
+ is connected to 5 peer ranks. Expected number of connected peer ranks is : 5
32
+ [10/30 23:39:08 minwm.distributed.sp.nccl]: Found nccl from library libnccl.so.2
33
+ [10/30 23:39:08 minwm.distributed.sp.device_communicators.pynccl]: Trainer is using nccl==2.27.5
34
+ [10/30 23:39:10 minwm.engine.trainer]: base RNG seed: 1348092628 (+ dp_rank)
35
+ Some weights of CausalWan21Model were not initialized from the model checkpoint at ./ckpts/Wan2.1-T2V-1.3B and are newly initialized: ['blocks.4.self_attn.prope_o.weight', 'blocks.23.self_attn.prope_o.bias', 'blocks.14.self_attn.prope_o.weight', 'blocks.20.self_attn.prope_o.weight', 'blocks.27.self_attn.prope_o.bias', 'blocks.20.self_attn.prope_o.bias', 'blocks.26.self_attn.prope_o.weight', 'blocks.5.self_attn.prope_o.bias', 'blocks.24.self_attn.prope_o.bias', 'blocks.28.self_attn.prope_o.weight', 'blocks.15.self_attn.prope_o.weight', 'blocks.11.self_attn.prope_o.bias', 'blocks.1.self_attn.prope_o.weight', 'blocks.8.self_attn.prope_o.bias', 'blocks.8.self_attn.prope_o.weight', 'blocks.17.self_attn.prope_o.bias', 'blocks.15.self_attn.prope_o.bias', 'blocks.18.self_attn.prope_o.weight', 'blocks.9.self_attn.prope_o.weight', 'blocks.24.self_attn.prope_o.weight', 'blocks.21.self_attn.prope_o.weight', 'blocks.0.self_attn.prope_o.weight', 'blocks.13.self_attn.prope_o.bias', 'blocks.14.self_attn.prope_o.bias', 'blocks.3.self_attn.prope_o.bias', 'blocks.7.self_attn.prope_o.weight', 'blocks.12.self_attn.prope_o.weight', 'blocks.12.self_attn.prope_o.bias', 'blocks.16.self_attn.prope_o.weight', 'blocks.27.self_attn.prope_o.weight', 'blocks.1.self_attn.prope_o.bias', 'blocks.5.self_attn.prope_o.weight', 'blocks.17.self_attn.prope_o.weight', 'blocks.0.self_attn.prope_o.bias', 'blocks.19.self_attn.prope_o.bias', 'blocks.28.self_attn.prope_o.bias', 'blocks.10.self_attn.prope_o.bias', 'blocks.16.self_attn.prope_o.bias', 'blocks.10.self_attn.prope_o.weight', 'blocks.3.self_attn.prope_o.weight', 'blocks.18.self_attn.prope_o.bias', 'blocks.22.self_attn.prope_o.bias', 'blocks.9.self_attn.prope_o.bias', 'blocks.6.self_attn.prope_o.weight', 'blocks.4.self_attn.prope_o.bias', 'blocks.22.self_attn.prope_o.weight', 'blocks.29.self_attn.prope_o.weight', 'blocks.6.self_attn.prope_o.bias', 'blocks.21.self_attn.prope_o.bias', 'blocks.23.self_attn.prope_o.weight', 'blocks.25.self_attn.prope_o.weight', 'blocks.19.self_attn.prope_o.weight', 'blocks.25.self_attn.prope_o.bias', 'blocks.11.self_attn.prope_o.weight', 'blocks.2.self_attn.prope_o.weight', 'blocks.2.self_attn.prope_o.bias', 'blocks.26.self_attn.prope_o.bias', 'blocks.7.self_attn.prope_o.bias', 'blocks.29.self_attn.prope_o.bias', 'blocks.13.self_attn.prope_o.weight']
36
+ You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.
37
+ Some weights of CausalWan21Model were not initialized from the model checkpoint at ./ckpts/Wan2.1-T2V-1.3B and are newly initialized: ['blocks.3.self_attn.prope_o.weight', 'blocks.7.self_attn.prope_o.bias', 'blocks.15.self_attn.prope_o.weight', 'blocks.25.self_attn.prope_o.bias', 'blocks.24.self_attn.prope_o.bias', 'blocks.17.self_attn.prope_o.bias', 'blocks.12.self_attn.prope_o.weight', 'blocks.9.self_attn.prope_o.bias', 'blocks.2.self_attn.prope_o.bias', 'blocks.13.self_attn.prope_o.bias', 'blocks.4.self_attn.prope_o.weight', 'blocks.25.self_attn.prope_o.weight', 'blocks.28.self_attn.prope_o.bias', 'blocks.1.self_attn.prope_o.weight', 'blocks.12.self_attn.prope_o.bias', 'blocks.4.self_attn.prope_o.bias', 'blocks.27.self_attn.prope_o.bias', 'blocks.19.self_attn.prope_o.weight', 'blocks.1.self_attn.prope_o.bias', 'blocks.26.self_attn.prope_o.bias', 'blocks.22.self_attn.prope_o.bias', 'blocks.2.self_attn.prope_o.weight', 'blocks.16.self_attn.prope_o.bias', 'blocks.27.self_attn.prope_o.weight', 'blocks.5.self_attn.prope_o.weight', 'blocks.18.self_attn.prope_o.bias', 'blocks.10.self_attn.prope_o.weight', 'blocks.17.self_attn.prope_o.weight', 'blocks.20.self_attn.prope_o.bias', 'blocks.9.self_attn.prope_o.weight', 'blocks.21.self_attn.prope_o.bias', 'blocks.6.self_attn.prope_o.bias', 'blocks.24.self_attn.prope_o.weight', 'blocks.14.self_attn.prope_o.weight', 'blocks.15.self_attn.prope_o.bias', 'blocks.16.self_attn.prope_o.weight', 'blocks.8.self_attn.prope_o.weight', 'blocks.23.self_attn.prope_o.bias', 'blocks.11.self_attn.prope_o.bias', 'blocks.18.self_attn.prope_o.weight', 'blocks.21.self_attn.prope_o.weight', 'blocks.6.self_attn.prope_o.weight', 'blocks.3.self_attn.prope_o.bias', 'blocks.11.self_attn.prope_o.weight', 'blocks.8.self_attn.prope_o.bias', 'blocks.19.self_attn.prope_o.bias', 'blocks.10.self_attn.prope_o.bias', 'blocks.29.self_attn.prope_o.bias', 'blocks.23.self_attn.prope_o.weight', 'blocks.14.self_attn.prope_o.bias', 'blocks.26.self_attn.prope_o.weight', 'blocks.28.self_attn.prope_o.weight', 'blocks.0.self_attn.prope_o.bias', 'blocks.29.self_attn.prope_o.weight', 'blocks.20.self_attn.prope_o.weight', 'blocks.22.self_attn.prope_o.weight', 'blocks.7.self_attn.prope_o.weight', 'blocks.13.self_attn.prope_o.weight', 'blocks.5.self_attn.prope_o.bias', 'blocks.0.self_attn.prope_o.weight']
38
+ You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.
39
+ Some weights of CausalWan21Model were not initialized from the model checkpoint at ./ckpts/Wan2.1-T2V-1.3B and are newly initialized: ['blocks.8.self_attn.prope_o.weight', 'blocks.9.self_attn.prope_o.weight', 'blocks.27.self_attn.prope_o.weight', 'blocks.14.self_attn.prope_o.bias', 'blocks.19.self_attn.prope_o.weight', 'blocks.10.self_attn.prope_o.bias', 'blocks.5.self_attn.prope_o.bias', 'blocks.25.self_attn.prope_o.bias', 'blocks.9.self_attn.prope_o.bias', 'blocks.2.self_attn.prope_o.weight', 'blocks.17.self_attn.prope_o.weight', 'blocks.7.self_attn.prope_o.weight', 'blocks.22.self_attn.prope_o.weight', 'blocks.21.self_attn.prope_o.bias', 'blocks.8.self_attn.prope_o.bias', 'blocks.11.self_attn.prope_o.bias', 'blocks.6.self_attn.prope_o.bias', 'blocks.25.self_attn.prope_o.weight', 'blocks.29.self_attn.prope_o.bias', 'blocks.15.self_attn.prope_o.bias', 'blocks.6.self_attn.prope_o.weight', 'blocks.19.self_attn.prope_o.bias', 'blocks.4.self_attn.prope_o.weight', 'blocks.2.self_attn.prope_o.bias', 'blocks.10.self_attn.prope_o.weight', 'blocks.28.self_attn.prope_o.bias', 'blocks.3.self_attn.prope_o.weight', 'blocks.13.self_attn.prope_o.bias', 'blocks.27.self_attn.prope_o.bias', 'blocks.5.self_attn.prope_o.weight', 'blocks.1.self_attn.prope_o.weight', 'blocks.28.self_attn.prope_o.weight', 'blocks.17.self_attn.prope_o.bias', 'blocks.12.self_attn.prope_o.weight', 'blocks.1.self_attn.prope_o.bias', 'blocks.21.self_attn.prope_o.weight', 'blocks.16.self_attn.prope_o.bias', 'blocks.12.self_attn.prope_o.bias', 'blocks.26.self_attn.prope_o.weight', 'blocks.7.self_attn.prope_o.bias', 'blocks.18.self_attn.prope_o.bias', 'blocks.14.self_attn.prope_o.weight', 'blocks.4.self_attn.prope_o.bias', 'blocks.11.self_attn.prope_o.weight', 'blocks.29.self_attn.prope_o.weight', 'blocks.0.self_attn.prope_o.weight', 'blocks.0.self_attn.prope_o.bias', 'blocks.20.self_attn.prope_o.bias', 'blocks.24.self_attn.prope_o.weight', 'blocks.15.self_attn.prope_o.weight', 'blocks.16.self_attn.prope_o.weight', 'blocks.18.self_attn.prope_o.weight', 'blocks.24.self_attn.prope_o.bias', 'blocks.23.self_attn.prope_o.weight', 'blocks.3.self_attn.prope_o.bias', 'blocks.13.self_attn.prope_o.weight', 'blocks.26.self_attn.prope_o.bias', 'blocks.22.self_attn.prope_o.bias', 'blocks.20.self_attn.prope_o.weight', 'blocks.23.self_attn.prope_o.bias']
40
+ You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.
41
+ Some weights of CausalWan21Model were not initialized from the model checkpoint at ./ckpts/Wan2.1-T2V-1.3B and are newly initialized: ['blocks.5.self_attn.prope_o.bias', 'blocks.0.self_attn.prope_o.bias', 'blocks.22.self_attn.prope_o.bias', 'blocks.2.self_attn.prope_o.weight', 'blocks.13.self_attn.prope_o.weight', 'blocks.10.self_attn.prope_o.bias', 'blocks.22.self_attn.prope_o.weight', 'blocks.20.self_attn.prope_o.weight', 'blocks.9.self_attn.prope_o.weight', 'blocks.27.self_attn.prope_o.bias', 'blocks.23.self_attn.prope_o.weight', 'blocks.24.self_attn.prope_o.weight', 'blocks.25.self_attn.prope_o.weight', 'blocks.14.self_attn.prope_o.bias', 'blocks.15.self_attn.prope_o.weight', 'blocks.21.self_attn.prope_o.bias', 'blocks.19.self_attn.prope_o.weight', 'blocks.12.self_attn.prope_o.bias', 'blocks.11.self_attn.prope_o.weight', 'blocks.21.self_attn.prope_o.weight', 'blocks.5.self_attn.prope_o.weight', 'blocks.28.self_attn.prope_o.bias', 'blocks.7.self_attn.prope_o.bias', 'blocks.12.self_attn.prope_o.weight', 'blocks.28.self_attn.prope_o.weight', 'blocks.29.self_attn.prope_o.weight', 'blocks.16.self_attn.prope_o.weight', 'blocks.25.self_attn.prope_o.bias', 'blocks.10.self_attn.prope_o.weight', 'blocks.11.self_attn.prope_o.bias', 'blocks.2.self_attn.prope_o.bias', 'blocks.17.self_attn.prope_o.bias', 'blocks.19.self_attn.prope_o.bias', 'blocks.18.self_attn.prope_o.weight', 'blocks.23.self_attn.prope_o.bias', 'blocks.24.self_attn.prope_o.bias', 'blocks.7.self_attn.prope_o.weight', 'blocks.9.self_attn.prope_o.bias', 'blocks.8.self_attn.prope_o.weight', 'blocks.29.self_attn.prope_o.bias', 'blocks.20.self_attn.prope_o.bias', 'blocks.0.self_attn.prope_o.weight', 'blocks.27.self_attn.prope_o.weight', 'blocks.18.self_attn.prope_o.bias', 'blocks.14.self_attn.prope_o.weight', 'blocks.3.self_attn.prope_o.weight', 'blocks.6.self_attn.prope_o.bias', 'blocks.15.self_attn.prope_o.bias', 'blocks.13.self_attn.prope_o.bias', 'blocks.4.self_attn.prope_o.bias', 'blocks.17.self_attn.prope_o.weight', 'blocks.26.self_attn.prope_o.bias', 'blocks.4.self_attn.prope_o.weight', 'blocks.6.self_attn.prope_o.weight', 'blocks.1.self_attn.prope_o.bias', 'blocks.1.self_attn.prope_o.weight', 'blocks.26.self_attn.prope_o.weight', 'blocks.8.self_attn.prope_o.bias', 'blocks.16.self_attn.prope_o.bias', 'blocks.3.self_attn.prope_o.bias']
42
+ You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.
43
+ Some weights of CausalWan21Model were not initialized from the model checkpoint at ./ckpts/Wan2.1-T2V-1.3B and are newly initialized: ['blocks.26.self_attn.prope_o.bias', 'blocks.23.self_attn.prope_o.weight', 'blocks.9.self_attn.prope_o.weight', 'blocks.15.self_attn.prope_o.bias', 'blocks.13.self_attn.prope_o.bias', 'blocks.2.self_attn.prope_o.bias', 'blocks.1.self_attn.prope_o.bias', 'blocks.11.self_attn.prope_o.weight', 'blocks.29.self_attn.prope_o.weight', 'blocks.23.self_attn.prope_o.bias', 'blocks.5.self_attn.prope_o.bias', 'blocks.16.self_attn.prope_o.bias', 'blocks.0.self_attn.prope_o.bias', 'blocks.7.self_attn.prope_o.weight', 'blocks.19.self_attn.prope_o.weight', 'blocks.27.self_attn.prope_o.weight', 'blocks.18.self_attn.prope_o.bias', 'blocks.17.self_attn.prope_o.bias', 'blocks.20.self_attn.prope_o.bias', 'blocks.10.self_attn.prope_o.bias', 'blocks.28.self_attn.prope_o.bias', 'blocks.18.self_attn.prope_o.weight', 'blocks.3.self_attn.prope_o.weight', 'blocks.25.self_attn.prope_o.weight', 'blocks.20.self_attn.prope_o.weight', 'blocks.22.self_attn.prope_o.weight', 'blocks.3.self_attn.prope_o.bias', 'blocks.24.self_attn.prope_o.bias', 'blocks.25.self_attn.prope_o.bias', 'blocks.28.self_attn.prope_o.weight', 'blocks.21.self_attn.prope_o.weight', 'blocks.8.self_attn.prope_o.bias', 'blocks.5.self_attn.prope_o.weight', 'blocks.19.self_attn.prope_o.bias', 'blocks.6.self_attn.prope_o.bias', 'blocks.16.self_attn.prope_o.weight', 'blocks.15.self_attn.prope_o.weight', 'blocks.22.self_attn.prope_o.bias', 'blocks.7.self_attn.prope_o.bias', 'blocks.17.self_attn.prope_o.weight', 'blocks.9.self_attn.prope_o.bias', 'blocks.11.self_attn.prope_o.bias', 'blocks.14.self_attn.prope_o.weight', 'blocks.12.self_attn.prope_o.bias', 'blocks.2.self_attn.prope_o.weight', 'blocks.8.self_attn.prope_o.weight', 'blocks.4.self_attn.prope_o.bias', 'blocks.13.self_attn.prope_o.weight', 'blocks.4.self_attn.prope_o.weight', 'blocks.29.self_attn.prope_o.bias', 'blocks.14.self_attn.prope_o.bias', 'blocks.24.self_attn.prope_o.weight', 'blocks.0.self_attn.prope_o.weight', 'blocks.26.self_attn.prope_o.weight', 'blocks.6.self_attn.prope_o.weight', 'blocks.21.self_attn.prope_o.bias', 'blocks.12.self_attn.prope_o.weight', 'blocks.27.self_attn.prope_o.bias', 'blocks.1.self_attn.prope_o.weight', 'blocks.10.self_attn.prope_o.weight']
44
+ You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.
45
+ Some weights of CausalWan21Model were not initialized from the model checkpoint at ./ckpts/Wan2.1-T2V-1.3B and are newly initialized: ['blocks.13.self_attn.prope_o.weight', 'blocks.14.self_attn.prope_o.bias', 'blocks.7.self_attn.prope_o.bias', 'blocks.18.self_attn.prope_o.bias', 'blocks.15.self_attn.prope_o.bias', 'blocks.28.self_attn.prope_o.bias', 'blocks.2.self_attn.prope_o.weight', 'blocks.17.self_attn.prope_o.bias', 'blocks.25.self_attn.prope_o.bias', 'blocks.20.self_attn.prope_o.bias', 'blocks.0.self_attn.prope_o.bias', 'blocks.28.self_attn.prope_o.weight', 'blocks.27.self_attn.prope_o.weight', 'blocks.29.self_attn.prope_o.bias', 'blocks.12.self_attn.prope_o.bias', 'blocks.26.self_attn.prope_o.bias', 'blocks.8.self_attn.prope_o.bias', 'blocks.29.self_attn.prope_o.weight', 'blocks.11.self_attn.prope_o.weight', 'blocks.15.self_attn.prope_o.weight', 'blocks.21.self_attn.prope_o.bias', 'blocks.25.self_attn.prope_o.weight', 'blocks.2.self_attn.prope_o.bias', 'blocks.27.self_attn.prope_o.bias', 'blocks.5.self_attn.prope_o.weight', 'blocks.4.self_attn.prope_o.bias', 'blocks.24.self_attn.prope_o.weight', 'blocks.17.self_attn.prope_o.weight', 'blocks.13.self_attn.prope_o.bias', 'blocks.20.self_attn.prope_o.weight', 'blocks.9.self_attn.prope_o.bias', 'blocks.23.self_attn.prope_o.bias', 'blocks.12.self_attn.prope_o.weight', 'blocks.4.self_attn.prope_o.weight', 'blocks.6.self_attn.prope_o.bias', 'blocks.7.self_attn.prope_o.weight', 'blocks.16.self_attn.prope_o.weight', 'blocks.18.self_attn.prope_o.weight', 'blocks.3.self_attn.prope_o.bias', 'blocks.6.self_attn.prope_o.weight', 'blocks.1.self_attn.prope_o.bias', 'blocks.22.self_attn.prope_o.bias', 'blocks.0.self_attn.prope_o.weight', 'blocks.10.self_attn.prope_o.weight', 'blocks.19.self_attn.prope_o.bias', 'blocks.23.self_attn.prope_o.weight', 'blocks.1.self_attn.prope_o.weight', 'blocks.16.self_attn.prope_o.bias', 'blocks.11.self_attn.prope_o.bias', 'blocks.19.self_attn.prope_o.weight', 'blocks.10.self_attn.prope_o.bias', 'blocks.26.self_attn.prope_o.weight', 'blocks.24.self_attn.prope_o.bias', 'blocks.21.self_attn.prope_o.weight', 'blocks.5.self_attn.prope_o.bias', 'blocks.8.self_attn.prope_o.weight', 'blocks.3.self_attn.prope_o.weight', 'blocks.22.self_attn.prope_o.weight', 'blocks.14.self_attn.prope_o.weight', 'blocks.9.self_attn.prope_o.weight']
46
+ You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.
47
+ [10/30 23:39:14 minwm.distributed.fsdp]: FSDP2: activation-checkpointed 30 layer module(s)
48
+ [10/30 23:39:14 minwm.distributed.fsdp]: FSDP2: sharded 30 layer module(s) in FSDPCausalWan21Model
49
+ [10/30 23:40:12 minwm.engine.trainer]: loaded pretrained weights from /dev/shm/mwm/exports/tf_nomas3/step_1000.pt
50
+ [10/30 23:41:56 minwm.engine.monitor]: step 10 time/data(ms): 2.218 time/forward(ms): 2128 time/backward(ms): 6758 grad_norm: 2.78 loss: 0.1019 steps/sec: 0.09618 ms/step: 1.04e+04 memory/max_reserved(GiB): 30.32 memory/max_reserved(%): 38.29 memory/num_ooms: 0
51
+ [10/30 23:43:26 minwm.engine.monitor]: step 20 time/data(ms): 2.248 time/forward(ms): 2081 time/backward(ms): 6580 grad_norm: 0.114 loss: 0.05678 steps/sec: 0.1114 ms/step: 8981 memory/max_reserved(GiB): 30.32 memory/max_reserved(%): 38.29 memory/num_ooms: 0
52
+ [10/30 23:44:56 minwm.engine.monitor]: step 30 time/data(ms): 2.305 time/forward(ms): 2099 time/backward(ms): 6584 grad_norm: 0.07225 loss: 0.1151 steps/sec: 0.1107 ms/step: 9032 memory/max_reserved(GiB): 30.32 memory/max_reserved(%): 38.29 memory/num_ooms: 0
53
+ [10/30 23:46:27 minwm.engine.monitor]: step 40 time/data(ms): 2.265 time/forward(ms): 2117 time/backward(ms): 6815 grad_norm: 0.1075 loss: 0.06289 steps/sec: 0.1102 ms/step: 9071 memory/max_reserved(GiB): 30.32 memory/max_reserved(%): 38.29 memory/num_ooms: 0
54
+ [10/30 23:47:57 minwm.engine.monitor]: step 50 time/data(ms): 1.841 time/forward(ms): 2270 time/backward(ms): 6595 grad_norm: 0.05821 loss: 0.02314 steps/sec: 0.111 ms/step: 9008 memory/max_reserved(GiB): 30.32 memory/max_reserved(%): 38.29 memory/num_ooms: 0
55
+ [10/30 23:49:27 minwm.engine.monitor]: step 60 time/data(ms): 2.08 time/forward(ms): 2085 time/backward(ms): 6589 grad_norm: 0.063 loss: 0.09621 steps/sec: 0.1111 ms/step: 9001 memory/max_reserved(GiB): 30.32 memory/max_reserved(%): 38.29 memory/num_ooms: 0
56
+ [10/30 23:50:57 minwm.engine.monitor]: step 70 time/data(ms): 2.021 time/forward(ms): 2263 time/backward(ms): 6599 grad_norm: 0.06582 loss: 0.04823 steps/sec: 0.1108 ms/step: 9023 memory/max_reserved(GiB): 30.32 memory/max_reserved(%): 38.29 memory/num_ooms: 0
57
+ [10/30 23:52:28 minwm.engine.monitor]: step 80 time/data(ms): 2.722 time/forward(ms): 2267 time/backward(ms): 6606 grad_norm: 0.0775 loss: 0.1047 steps/sec: 0.1105 ms/step: 9053 memory/max_reserved(GiB): 30.32 memory/max_reserved(%): 38.29 memory/num_ooms: 0
58
+ [10/30 23:53:58 minwm.engine.monitor]: step 90 time/data(ms): 2.202 time/forward(ms): 2292 time/backward(ms): 6609 grad_norm: 0.08042 loss: 0.07051 steps/sec: 0.1105 ms/step: 9046 memory/max_reserved(GiB): 30.32 memory/max_reserved(%): 38.29 memory/num_ooms: 0
59
+ [10/30 23:55:29 minwm.engine.monitor]: step 100 time/data(ms): 2.19 time/forward(ms): 2294 time/backward(ms): 6621 grad_norm: 0.07171 loss: 0.07699 steps/sec: 0.11 ms/step: 9094 memory/max_reserved(GiB): 30.32 memory/max_reserved(%): 38.29 memory/num_ooms: 0
60
+ [10/30 23:56:59 minwm.engine.monitor]: step 110 time/data(ms): 1.979 time/forward(ms): 2112 time/backward(ms): 6818 grad_norm: 0.06937 loss: 0.08037 steps/sec: 0.1109 ms/step: 9014 memory/max_reserved(GiB): 30.32 memory/max_reserved(%): 38.29 memory/num_ooms: 0
61
+ [10/30 23:58:30 minwm.engine.monitor]: step 120 time/data(ms): 1.512 time/forward(ms): 2110 time/backward(ms): 6796 grad_norm: 0.07279 loss: 0.06516 steps/sec: 0.1104 ms/step: 9060 memory/max_reserved(GiB): 30.32 memory/max_reserved(%): 38.29 memory/num_ooms: 0
62
+ [10/31 00:00:00 minwm.engine.monitor]: step 130 time/data(ms): 2.095 time/forward(ms): 2429 time/backward(ms): 6615 grad_norm: 0.07834 loss: 0.08304 steps/sec: 0.1102 ms/step: 9075 memory/max_reserved(GiB): 30.32 memory/max_reserved(%): 38.29 memory/num_ooms: 0
63
+ [10/31 00:01:31 minwm.engine.monitor]: step 140 time/data(ms): 1.497 time/forward(ms): 2270 time/backward(ms): 6606 grad_norm: 0.07661 loss: 0.0269 steps/sec: 0.1106 ms/step: 9045 memory/max_reserved(GiB): 30.32 memory/max_reserved(%): 38.29 memory/num_ooms: 0
64
+ [10/31 00:03:01 minwm.engine.monitor]: step 150 time/data(ms): 1.517 time/forward(ms): 2273 time/backward(ms): 6614 grad_norm: 0.08383 loss: 0.09444 steps/sec: 0.1114 ms/step: 8979 memory/max_reserved(GiB): 30.32 memory/max_reserved(%): 38.29 memory/num_ooms: 0
65
+ [10/31 00:04:30 minwm.engine.monitor]: step 160 time/data(ms): 1.633 time/forward(ms): 2093 time/backward(ms): 6595 grad_norm: 0.0483 loss: 0.1145 steps/sec: 0.112 ms/step: 8932 memory/max_reserved(GiB): 30.32 memory/max_reserved(%): 38.29 memory/num_ooms: 0
66
+ [10/31 00:06:00 minwm.engine.monitor]: step 170 time/data(ms): 2.199 time/forward(ms): 2091 time/backward(ms): 6594 grad_norm: 0.07617 loss: 0.07862 steps/sec: 0.1107 ms/step: 9034 memory/max_reserved(GiB): 30.32 memory/max_reserved(%): 38.29 memory/num_ooms: 0
67
+ [10/31 00:07:31 minwm.engine.monitor]: step 180 time/data(ms): 2.003 time/forward(ms): 2087 time/backward(ms): 6591 grad_norm: 0.08448 loss: 0.09374 steps/sec: 0.1108 ms/step: 9024 memory/max_reserved(GiB): 30.32 memory/max_reserved(%): 38.29 memory/num_ooms: 0
68
+ [10/31 00:09:01 minwm.engine.monitor]: step 190 time/data(ms): 2.061 time/forward(ms): 2264 time/backward(ms): 6611 grad_norm: 0.05914 loss: 0.1611 steps/sec: 0.1106 ms/step: 9041 memory/max_reserved(GiB): 30.32 memory/max_reserved(%): 38.29 memory/num_ooms: 0
69
+ [10/31 00:10:31 minwm.engine.monitor]: step 200 time/data(ms): 1.953 time/forward(ms): 2114 time/backward(ms): 6789 grad_norm: 0.06499 loss: 0.06016 steps/sec: 0.1105 ms/step: 9046 memory/max_reserved(GiB): 30.32 memory/max_reserved(%): 38.29 memory/num_ooms: 0
70
+ [10/31 00:10:39 minwm.engine.checkpoint.checkpointer]: saved checkpoint /dev/shm/mwm/runs/tf_nomas4/ckpts/checkpoint_200
71
+ [10/31 00:12:09 minwm.engine.monitor]: step 210 time/data(ms): 1.814 time/forward(ms): 2268 time/backward(ms): 6600 grad_norm: 69.79 loss: 0.09043 steps/sec: 0.1024 ms/step: 9767 memory/max_reserved(GiB): 30.32 memory/max_reserved(%): 38.29 memory/num_ooms: 0
72
+ [10/31 00:13:40 minwm.engine.monitor]: step 220 time/data(ms): 1.781 time/forward(ms): 2294 time/backward(ms): 6601 grad_norm: 659.1 loss: 0.07217 steps/sec: 0.1103 ms/step: 9067 memory/max_reserved(GiB): 30.32 memory/max_reserved(%): 38.29 memory/num_ooms: 0
73
+ [10/31 00:15:10 minwm.engine.monitor]: step 230 time/data(ms): 1.706 time/forward(ms): 2111 time/backward(ms): 6813 grad_norm: 2.096 loss: 0.05185 steps/sec: 0.1105 ms/step: 9052 memory/max_reserved(GiB): 30.32 memory/max_reserved(%): 38.29 memory/num_ooms: 0
74
+ [10/31 00:16:41 minwm.engine.monitor]: step 240 time/data(ms): 1.456 time/forward(ms): 2087 time/backward(ms): 6586 grad_norm: 0.0808 loss: 0.1243 steps/sec: 0.1107 ms/step: 9029 memory/max_reserved(GiB): 30.32 memory/max_reserved(%): 38.29 memory/num_ooms: 0
75
+ [10/31 00:18:10 minwm.engine.monitor]: step 250 time/data(ms): 2.028 time/forward(ms): 2263 time/backward(ms): 6603 grad_norm: 0.08702 loss: 0.106 steps/sec: 0.1115 ms/step: 8966 memory/max_reserved(GiB): 30.32 memory/max_reserved(%): 38.29 memory/num_ooms: 0
76
+ W1031 00:19:18.777000 3094248 torch/distributed/elastic/agent/server/api.py:725] Received 15 death signal, shutting down workers
77
+ W1031 00:19:18.779000 3094248 torch/distributed/elastic/multiprocessing/api.py:908] Sending process 3094276 closing signal SIGTERM
78
+ W1031 00:19:18.781000 3094248 torch/distributed/elastic/multiprocessing/api.py:908] Sending process 3094277 closing signal SIGTERM
79
+ W1031 00:19:18.783000 3094248 torch/distributed/elastic/multiprocessing/api.py:908] Sending process 3094278 closing signal SIGTERM
80
+ W1031 00:19:18.787000 3094248 torch/distributed/elastic/multiprocessing/api.py:908] Sending process 3094279 closing signal SIGTERM
81
+ W1031 00:19:18.791000 3094248 torch/distributed/elastic/multiprocessing/api.py:908] Sending process 3094280 closing signal SIGTERM
82
+ W1031 00:19:18.794000 3094248 torch/distributed/elastic/multiprocessing/api.py:908] Sending process 3094281 closing signal SIGTERM
83
+ Traceback (most recent call last):
84
+ File "/dev/shm/mwm/venv/bin/torchrun", line 10, in <module>
85
+ sys.exit(main())
86
+ ^^^^^^
87
+ File "/dev/shm/mwm/venv/lib/python3.12/site-packages/torch/distributed/elastic/multiprocessing/errors/__init__.py", line 357, in wrapper
88
+ return f(*args, **kwargs)
89
+ ^^^^^^^^^^^^^^^^^^
90
+ File "/dev/shm/mwm/venv/lib/python3.12/site-packages/torch/distributed/run.py", line 936, in main
91
+ run(args)
92
+ File "/dev/shm/mwm/venv/lib/python3.12/site-packages/torch/distributed/run.py", line 927, in run
93
+ elastic_launch(
94
+ File "/dev/shm/mwm/venv/lib/python3.12/site-packages/torch/distributed/launcher/api.py", line 156, in __call__
95
+ return launch_agent(self._config, self._entrypoint, list(args))
96
+ ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
97
+ File "/dev/shm/mwm/venv/lib/python3.12/site-packages/torch/distributed/launcher/api.py", line 284, in launch_agent
98
+ result = agent.run()
99
+ ^^^^^^^^^^^
100
+ File "/dev/shm/mwm/venv/lib/python3.12/site-packages/torch/distributed/elastic/metrics/api.py", line 138, in wrapper
101
+ result = f(*args, **kwargs)
102
+ ^^^^^^^^^^^^^^^^^^
103
+ File "/dev/shm/mwm/venv/lib/python3.12/site-packages/torch/distributed/elastic/agent/server/api.py", line 717, in run
104
+ result = self._invoke_run(role)
105
+ ^^^^^^^^^^^^^^^^^^^^^^
106
+ File "/dev/shm/mwm/venv/lib/python3.12/site-packages/torch/distributed/elastic/agent/server/api.py", line 881, in _invoke_run
107
+ time.sleep(monitor_interval)
108
+ File "/dev/shm/mwm/venv/lib/python3.12/site-packages/torch/distributed/elastic/multiprocessing/api.py", line 85, in _terminate_process_handler
109
+ raise SignalException(f"Process {os.getpid()} got signal: {sigval}", sigval=sigval)
110
+ torch.distributed.elastic.multiprocessing.api.SignalException: Process 3094248 got signal: 15
111
+ [W1031 00:19:19.659307039 AllocatorConfig.cpp:28] Warning: PYTORCH_CUDA_ALLOC_CONF is deprecated, use PYTORCH_ALLOC_CONF instead (function operator())
112
+ MWM_TRAIN3 repo /dev/shm/mwm/repo data /dev/shm/mwm/data/quality char_labels none char_k 200 drop 0.1
113
+ MWM_TRAIN config configs/wan21/action2v/train/stage1_ar_tf_aug.py
114
+ MWM_TRAIN start 00:19:24 stage tf name tf_nomas4 gpus 0,1,2,3,4,5 n 6 steps 1200 lr 5e-6 init /dev/shm/mwm/exports/tf_nomas3/step_1000.pt teacher same extra checkpoint.resume=True
115
+ [10/31 00:19:28 minwm.distributed.sp.parallel_state]: Initializing distributed environment with world_size=6, device=cuda:0
116
+ [10/31 00:19:28 minwm.distributed.sp.parallel_state]: world_size=6 rank=0 local_rank=0 distributed_init_method=env:// backend=nccl
117
+ [Gloo] Rank 0 is connected to 5 peer ranks. Expected number of connected peer ranks is : 5
118
+ [Gloo] Rank 1 is connected to 5 peer ranks. Expected number of connected peer ranks is : 5
119
+ [Gloo] Rank [Gloo] Rank 5 is connected to 54 peer ranks. is connected to Expected number of connected peer ranks is : 55 peer ranks. Expected number of connected peer ranks is :
120
+ 5
121
+ [Gloo] Rank [Gloo] Rank 3 is connected to 52 peer ranks. is connected to Expected number of connected peer ranks is : 55 peer ranks. Expected number of connected peer ranks is :
122
+ 5
123
+ [10/31 00:19:32 minwm.distributed.sp.nccl]: Found nccl from library libnccl.so.2
124
+ [10/31 00:19:32 minwm.distributed.sp.device_communicators.pynccl]: Trainer is using nccl==2.27.5
125
+ [Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
126
+ [Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
127
+ [Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
128
+ [Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
129
+ [Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
130
+ [Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
131
+ [Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
132
+ [Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
133
+ [Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
134
+ [Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
135
+ [Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
136
+ [Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
137
+ [Gloo] Rank 1 is connected to 5 peer ranks. Expected number of connected peer ranks is : 5
138
+ [Gloo] Rank 4 is connected to 5 peer ranks. Expected number of connected peer ranks is : 5
139
+ [Gloo] Rank 0 is connected to 5 peer ranks. Expected number of connected peer ranks is : 5
140
+ [Gloo] Rank 3 is connected to 5 peer ranks. Expected number of connected peer ranks is : 5
141
+ [Gloo] Rank 2 is connected to 5 peer ranks. Expected number of connected peer ranks is : 5
142
+ [Gloo] Rank 5 is connected to 5 peer ranks. Expected number of connected peer ranks is : 5
143
+ [10/31 00:19:34 minwm.distributed.sp.nccl]: Found nccl from library libnccl.so.2
144
+ [10/31 00:19:34 minwm.distributed.sp.device_communicators.pynccl]: Trainer is using nccl==2.27.5
145
+ [10/31 00:19:35 minwm.engine.trainer]: base RNG seed: 1097413646 (+ dp_rank)
146
+ Some weights of CausalWan21Model were not initialized from the model checkpoint at ./ckpts/Wan2.1-T2V-1.3B and are newly initialized: ['blocks.16.self_attn.prope_o.weight', 'blocks.25.self_attn.prope_o.bias', 'blocks.7.self_attn.prope_o.weight', 'blocks.23.self_attn.prope_o.weight', 'blocks.29.self_attn.prope_o.weight', 'blocks.25.self_attn.prope_o.weight', 'blocks.8.self_attn.prope_o.weight', 'blocks.6.self_attn.prope_o.bias', 'blocks.3.self_attn.prope_o.bias', 'blocks.1.self_attn.prope_o.weight', 'blocks.15.self_attn.prope_o.weight', 'blocks.4.self_attn.prope_o.weight', 'blocks.18.self_attn.prope_o.weight', 'blocks.10.self_attn.prope_o.weight', 'blocks.5.self_attn.prope_o.weight', 'blocks.21.self_attn.prope_o.bias', 'blocks.17.self_attn.prope_o.weight', 'blocks.18.self_attn.prope_o.bias', 'blocks.2.self_attn.prope_o.weight', 'blocks.11.self_attn.prope_o.bias', 'blocks.4.self_attn.prope_o.bias', 'blocks.6.self_attn.prope_o.weight', 'blocks.9.self_attn.prope_o.bias', 'blocks.19.self_attn.prope_o.weight', 'blocks.11.self_attn.prope_o.weight', 'blocks.26.self_attn.prope_o.weight', 'blocks.0.self_attn.prope_o.weight', 'blocks.12.self_attn.prope_o.weight', 'blocks.23.self_attn.prope_o.bias', 'blocks.16.self_attn.prope_o.bias', 'blocks.21.self_attn.prope_o.weight', 'blocks.7.self_attn.prope_o.bias', 'blocks.20.self_attn.prope_o.bias', 'blocks.22.self_attn.prope_o.bias', 'blocks.13.self_attn.prope_o.bias', 'blocks.2.self_attn.prope_o.bias', 'blocks.29.self_attn.prope_o.bias', 'blocks.20.self_attn.prope_o.weight', 'blocks.13.self_attn.prope_o.weight', 'blocks.10.self_attn.prope_o.bias', 'blocks.19.self_attn.prope_o.bias', 'blocks.24.self_attn.prope_o.weight', 'blocks.3.self_attn.prope_o.weight', 'blocks.14.self_attn.prope_o.bias', 'blocks.8.self_attn.prope_o.bias', 'blocks.28.self_attn.prope_o.weight', 'blocks.15.self_attn.prope_o.bias', 'blocks.1.self_attn.prope_o.bias', 'blocks.28.self_attn.prope_o.bias', 'blocks.0.self_attn.prope_o.bias', 'blocks.17.self_attn.prope_o.bias', 'blocks.27.self_attn.prope_o.weight', 'blocks.22.self_attn.prope_o.weight', 'blocks.5.self_attn.prope_o.bias', 'blocks.9.self_attn.prope_o.weight', 'blocks.24.self_attn.prope_o.bias', 'blocks.26.self_attn.prope_o.bias', 'blocks.14.self_attn.prope_o.weight', 'blocks.27.self_attn.prope_o.bias', 'blocks.12.self_attn.prope_o.bias']
147
+ You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.
148
+ Some weights of CausalWan21Model were not initialized from the model checkpoint at ./ckpts/Wan2.1-T2V-1.3B and are newly initialized: ['blocks.15.self_attn.prope_o.bias', 'blocks.22.self_attn.prope_o.bias', 'blocks.28.self_attn.prope_o.bias', 'blocks.27.self_attn.prope_o.weight', 'blocks.11.self_attn.prope_o.weight', 'blocks.11.self_attn.prope_o.bias', 'blocks.5.self_attn.prope_o.bias', 'blocks.3.self_attn.prope_o.weight', 'blocks.9.self_attn.prope_o.bias', 'blocks.2.self_attn.prope_o.weight', 'blocks.0.self_attn.prope_o.weight', 'blocks.13.self_attn.prope_o.weight', 'blocks.10.self_attn.prope_o.weight', 'blocks.1.self_attn.prope_o.bias', 'blocks.29.self_attn.prope_o.weight', 'blocks.0.self_attn.prope_o.bias', 'blocks.12.self_attn.prope_o.weight', 'blocks.19.self_attn.prope_o.weight', 'blocks.26.self_attn.prope_o.bias', 'blocks.17.self_attn.prope_o.bias', 'blocks.20.self_attn.prope_o.bias', 'blocks.14.self_attn.prope_o.bias', 'blocks.16.self_attn.prope_o.weight', 'blocks.6.self_attn.prope_o.bias', 'blocks.26.self_attn.prope_o.weight', 'blocks.14.self_attn.prope_o.weight', 'blocks.20.self_attn.prope_o.weight', 'blocks.15.self_attn.prope_o.weight', 'blocks.28.self_attn.prope_o.weight', 'blocks.7.self_attn.prope_o.weight', 'blocks.25.self_attn.prope_o.bias', 'blocks.7.self_attn.prope_o.bias', 'blocks.23.self_attn.prope_o.weight', 'blocks.10.self_attn.prope_o.bias', 'blocks.23.self_attn.prope_o.bias', 'blocks.21.self_attn.prope_o.bias', 'blocks.18.self_attn.prope_o.bias', 'blocks.8.self_attn.prope_o.bias', 'blocks.19.self_attn.prope_o.bias', 'blocks.3.self_attn.prope_o.bias', 'blocks.24.self_attn.prope_o.bias', 'blocks.27.self_attn.prope_o.bias', 'blocks.16.self_attn.prope_o.bias', 'blocks.12.self_attn.prope_o.bias', 'blocks.22.self_attn.prope_o.weight', 'blocks.24.self_attn.prope_o.weight', 'blocks.1.self_attn.prope_o.weight', 'blocks.5.self_attn.prope_o.weight', 'blocks.21.self_attn.prope_o.weight', 'blocks.4.self_attn.prope_o.bias', 'blocks.18.self_attn.prope_o.weight', 'blocks.6.self_attn.prope_o.weight', 'blocks.29.self_attn.prope_o.bias', 'blocks.17.self_attn.prope_o.weight', 'blocks.9.self_attn.prope_o.weight', 'blocks.2.self_attn.prope_o.bias', 'blocks.25.self_attn.prope_o.weight', 'blocks.8.self_attn.prope_o.weight', 'blocks.4.self_attn.prope_o.weight', 'blocks.13.self_attn.prope_o.bias']
149
+ You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.
150
+ Some weights of CausalWan21Model were not initialized from the model checkpoint at ./ckpts/Wan2.1-T2V-1.3B and are newly initialized: ['blocks.16.self_attn.prope_o.bias', 'blocks.3.self_attn.prope_o.weight', 'blocks.5.self_attn.prope_o.bias', 'blocks.27.self_attn.prope_o.weight', 'blocks.23.self_attn.prope_o.weight', 'blocks.8.self_attn.prope_o.bias', 'blocks.4.self_attn.prope_o.bias', 'blocks.29.self_attn.prope_o.weight', 'blocks.13.self_attn.prope_o.weight', 'blocks.0.self_attn.prope_o.bias', 'blocks.1.self_attn.prope_o.bias', 'blocks.12.self_attn.prope_o.weight', 'blocks.13.self_attn.prope_o.bias', 'blocks.14.self_attn.prope_o.weight', 'blocks.6.self_attn.prope_o.bias', 'blocks.10.self_attn.prope_o.bias', 'blocks.8.self_attn.prope_o.weight', 'blocks.7.self_attn.prope_o.weight', 'blocks.18.self_attn.prope_o.bias', 'blocks.25.self_attn.prope_o.bias', 'blocks.18.self_attn.prope_o.weight', 'blocks.20.self_attn.prope_o.bias', 'blocks.28.self_attn.prope_o.bias', 'blocks.15.self_attn.prope_o.weight', 'blocks.12.self_attn.prope_o.bias', 'blocks.4.self_attn.prope_o.weight', 'blocks.19.self_attn.prope_o.weight', 'blocks.17.self_attn.prope_o.weight', 'blocks.21.self_attn.prope_o.bias', 'blocks.2.self_attn.prope_o.bias', 'blocks.2.self_attn.prope_o.weight', 'blocks.5.self_attn.prope_o.weight', 'blocks.21.self_attn.prope_o.weight', 'blocks.29.self_attn.prope_o.bias', 'blocks.26.self_attn.prope_o.bias', 'blocks.10.self_attn.prope_o.weight', 'blocks.19.self_attn.prope_o.bias', 'blocks.17.self_attn.prope_o.bias', 'blocks.1.self_attn.prope_o.weight', 'blocks.9.self_attn.prope_o.weight', 'blocks.11.self_attn.prope_o.weight', 'blocks.24.self_attn.prope_o.bias', 'blocks.23.self_attn.prope_o.bias', 'blocks.22.self_attn.prope_o.bias', 'blocks.3.self_attn.prope_o.bias', 'blocks.27.self_attn.prope_o.bias', 'blocks.26.self_attn.prope_o.weight', 'blocks.15.self_attn.prope_o.bias', 'blocks.24.self_attn.prope_o.weight', 'blocks.14.self_attn.prope_o.bias', 'blocks.11.self_attn.prope_o.bias', 'blocks.22.self_attn.prope_o.weight', 'blocks.9.self_attn.prope_o.bias', 'blocks.0.self_attn.prope_o.weight', 'blocks.28.self_attn.prope_o.weight', 'blocks.6.self_attn.prope_o.weight', 'blocks.7.self_attn.prope_o.bias', 'blocks.16.self_attn.prope_o.weight', 'blocks.20.self_attn.prope_o.weight', 'blocks.25.self_attn.prope_o.weight']
151
+ You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.
152
+ Some weights of CausalWan21Model were not initialized from the model checkpoint at ./ckpts/Wan2.1-T2V-1.3B and are newly initialized: ['blocks.25.self_attn.prope_o.bias', 'blocks.20.self_attn.prope_o.weight', 'blocks.23.self_attn.prope_o.bias', 'blocks.16.self_attn.prope_o.bias', 'blocks.8.self_attn.prope_o.bias', 'blocks.1.self_attn.prope_o.weight', 'blocks.6.self_attn.prope_o.bias', 'blocks.2.self_attn.prope_o.weight', 'blocks.11.self_attn.prope_o.bias', 'blocks.19.self_attn.prope_o.weight', 'blocks.10.self_attn.prope_o.weight', 'blocks.6.self_attn.prope_o.weight', 'blocks.28.self_attn.prope_o.weight', 'blocks.5.self_attn.prope_o.bias', 'blocks.3.self_attn.prope_o.weight', 'blocks.3.self_attn.prope_o.bias', 'blocks.21.self_attn.prope_o.weight', 'blocks.23.self_attn.prope_o.weight', 'blocks.22.self_attn.prope_o.weight', 'blocks.0.self_attn.prope_o.weight', 'blocks.13.self_attn.prope_o.weight', 'blocks.29.self_attn.prope_o.weight', 'blocks.9.self_attn.prope_o.weight', 'blocks.15.self_attn.prope_o.weight', 'blocks.26.self_attn.prope_o.weight', 'blocks.24.self_attn.prope_o.weight', 'blocks.14.self_attn.prope_o.weight', 'blocks.4.self_attn.prope_o.weight', 'blocks.18.self_attn.prope_o.bias', 'blocks.4.self_attn.prope_o.bias', 'blocks.26.self_attn.prope_o.bias', 'blocks.1.self_attn.prope_o.bias', 'blocks.16.self_attn.prope_o.weight', 'blocks.12.self_attn.prope_o.bias', 'blocks.21.self_attn.prope_o.bias', 'blocks.15.self_attn.prope_o.bias', 'blocks.7.self_attn.prope_o.bias', 'blocks.13.self_attn.prope_o.bias', 'blocks.25.self_attn.prope_o.weight', 'blocks.11.self_attn.prope_o.weight', 'blocks.28.self_attn.prope_o.bias', 'blocks.8.self_attn.prope_o.weight', 'blocks.7.self_attn.prope_o.weight', 'blocks.9.self_attn.prope_o.bias', 'blocks.17.self_attn.prope_o.bias', 'blocks.17.self_attn.prope_o.weight', 'blocks.27.self_attn.prope_o.weight', 'blocks.22.self_attn.prope_o.bias', 'blocks.0.self_attn.prope_o.bias', 'blocks.10.self_attn.prope_o.bias', 'blocks.29.self_attn.prope_o.bias', 'blocks.5.self_attn.prope_o.weight', 'blocks.24.self_attn.prope_o.bias', 'blocks.18.self_attn.prope_o.weight', 'blocks.19.self_attn.prope_o.bias', 'blocks.2.self_attn.prope_o.bias', 'blocks.27.self_attn.prope_o.bias', 'blocks.14.self_attn.prope_o.bias', 'blocks.20.self_attn.prope_o.bias', 'blocks.12.self_attn.prope_o.weight']
153
+ You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.
154
+ Some weights of CausalWan21Model were not initialized from the model checkpoint at ./ckpts/Wan2.1-T2V-1.3B and are newly initialized: ['blocks.27.self_attn.prope_o.bias', 'blocks.0.self_attn.prope_o.weight', 'blocks.23.self_attn.prope_o.bias', 'blocks.29.self_attn.prope_o.bias', 'blocks.6.self_attn.prope_o.bias', 'blocks.17.self_attn.prope_o.bias', 'blocks.0.self_attn.prope_o.bias', 'blocks.25.self_attn.prope_o.bias', 'blocks.16.self_attn.prope_o.bias', 'blocks.19.self_attn.prope_o.weight', 'blocks.1.self_attn.prope_o.bias', 'blocks.28.self_attn.prope_o.weight', 'blocks.18.self_attn.prope_o.bias', 'blocks.8.self_attn.prope_o.bias', 'blocks.22.self_attn.prope_o.weight', 'blocks.17.self_attn.prope_o.weight', 'blocks.18.self_attn.prope_o.weight', 'blocks.10.self_attn.prope_o.bias', 'blocks.19.self_attn.prope_o.bias', 'blocks.11.self_attn.prope_o.bias', 'blocks.2.self_attn.prope_o.bias', 'blocks.13.self_attn.prope_o.bias', 'blocks.16.self_attn.prope_o.weight', 'blocks.27.self_attn.prope_o.weight', 'blocks.15.self_attn.prope_o.bias', 'blocks.4.self_attn.prope_o.bias', 'blocks.12.self_attn.prope_o.weight', 'blocks.4.self_attn.prope_o.weight', 'blocks.29.self_attn.prope_o.weight', 'blocks.26.self_attn.prope_o.bias', 'blocks.24.self_attn.prope_o.bias', 'blocks.8.self_attn.prope_o.weight', 'blocks.1.self_attn.prope_o.weight', 'blocks.23.self_attn.prope_o.weight', 'blocks.26.self_attn.prope_o.weight', 'blocks.25.self_attn.prope_o.weight', 'blocks.20.self_attn.prope_o.weight', 'blocks.9.self_attn.prope_o.weight', 'blocks.28.self_attn.prope_o.bias', 'blocks.13.self_attn.prope_o.weight', 'blocks.7.self_attn.prope_o.weight', 'blocks.12.self_attn.prope_o.bias', 'blocks.6.self_attn.prope_o.weight', 'blocks.11.self_attn.prope_o.weight', 'blocks.3.self_attn.prope_o.weight', 'blocks.7.self_attn.prope_o.bias', 'blocks.15.self_attn.prope_o.weight', 'blocks.21.self_attn.prope_o.weight', 'blocks.3.self_attn.prope_o.bias', 'blocks.9.self_attn.prope_o.bias', 'blocks.2.self_attn.prope_o.weight', 'blocks.14.self_attn.prope_o.weight', 'blocks.10.self_attn.prope_o.weight', 'blocks.5.self_attn.prope_o.weight', 'blocks.21.self_attn.prope_o.bias', 'blocks.22.self_attn.prope_o.bias', 'blocks.24.self_attn.prope_o.weight', 'blocks.14.self_attn.prope_o.bias', 'blocks.20.self_attn.prope_o.bias', 'blocks.5.self_attn.prope_o.bias']
155
+ You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.
156
+ Some weights of CausalWan21Model were not initialized from the model checkpoint at ./ckpts/Wan2.1-T2V-1.3B and are newly initialized: ['blocks.6.self_attn.prope_o.bias', 'blocks.13.self_attn.prope_o.weight', 'blocks.23.self_attn.prope_o.weight', 'blocks.4.self_attn.prope_o.bias', 'blocks.6.self_attn.prope_o.weight', 'blocks.18.self_attn.prope_o.bias', 'blocks.7.self_attn.prope_o.bias', 'blocks.5.self_attn.prope_o.weight', 'blocks.15.self_attn.prope_o.bias', 'blocks.0.self_attn.prope_o.weight', 'blocks.21.self_attn.prope_o.weight', 'blocks.25.self_attn.prope_o.bias', 'blocks.1.self_attn.prope_o.weight', 'blocks.3.self_attn.prope_o.weight', 'blocks.21.self_attn.prope_o.bias', 'blocks.22.self_attn.prope_o.bias', 'blocks.17.self_attn.prope_o.bias', 'blocks.10.self_attn.prope_o.bias', 'blocks.22.self_attn.prope_o.weight', 'blocks.23.self_attn.prope_o.bias', 'blocks.17.self_attn.prope_o.weight', 'blocks.27.self_attn.prope_o.weight', 'blocks.18.self_attn.prope_o.weight', 'blocks.3.self_attn.prope_o.bias', 'blocks.8.self_attn.prope_o.bias', 'blocks.4.self_attn.prope_o.weight', 'blocks.9.self_attn.prope_o.weight', 'blocks.11.self_attn.prope_o.weight', 'blocks.15.self_attn.prope_o.weight', 'blocks.14.self_attn.prope_o.bias', 'blocks.16.self_attn.prope_o.weight', 'blocks.20.self_attn.prope_o.bias', 'blocks.5.self_attn.prope_o.bias', 'blocks.28.self_attn.prope_o.bias', 'blocks.24.self_attn.prope_o.bias', 'blocks.19.self_attn.prope_o.bias', 'blocks.12.self_attn.prope_o.bias', 'blocks.14.self_attn.prope_o.weight', 'blocks.12.self_attn.prope_o.weight', 'blocks.16.self_attn.prope_o.bias', 'blocks.11.self_attn.prope_o.bias', 'blocks.19.self_attn.prope_o.weight', 'blocks.29.self_attn.prope_o.weight', 'blocks.10.self_attn.prope_o.weight', 'blocks.8.self_attn.prope_o.weight', 'blocks.9.self_attn.prope_o.bias', 'blocks.1.self_attn.prope_o.bias', 'blocks.26.self_attn.prope_o.weight', 'blocks.29.self_attn.prope_o.bias', 'blocks.28.self_attn.prope_o.weight', 'blocks.25.self_attn.prope_o.weight', 'blocks.2.self_attn.prope_o.bias', 'blocks.24.self_attn.prope_o.weight', 'blocks.20.self_attn.prope_o.weight', 'blocks.26.self_attn.prope_o.bias', 'blocks.0.self_attn.prope_o.bias', 'blocks.7.self_attn.prope_o.weight', 'blocks.2.self_attn.prope_o.weight', 'blocks.13.self_attn.prope_o.bias', 'blocks.27.self_attn.prope_o.bias']
157
+ You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.
158
+ [10/31 00:19:39 minwm.distributed.fsdp]: FSDP2: activation-checkpointed 30 layer module(s)
159
+ [10/31 00:19:39 minwm.distributed.fsdp]: FSDP2: sharded 30 layer module(s) in FSDPCausalWan21Model
160
+ [10/31 00:20:40 minwm.engine.checkpoint.checkpointer]: loaded checkpoint /dev/shm/mwm/runs/tf_nomas4/ckpts/checkpoint_200
161
+ [10/31 00:22:23 minwm.engine.monitor]: step 210 time/data(ms): 1.902 time/forward(ms): 2300 time/backward(ms): 6643 grad_norm: 0.07516 loss: 0.05791 steps/sec: 0.09719 ms/step: 1.029e+04 memory/max_reserved(GiB): 30.28 memory/max_reserved(%): 38.24 memory/num_ooms: 0
162
+ [10/31 00:23:53 minwm.engine.monitor]: step 220 time/data(ms): 1.631 time/forward(ms): 2269 time/backward(ms): 6602 grad_norm: 0.06266 loss: 0.07548 steps/sec: 0.1105 ms/step: 9047 memory/max_reserved(GiB): 30.28 memory/max_reserved(%): 38.24 memory/num_ooms: 0
163
+ [10/31 00:25:24 minwm.engine.monitor]: step 230 time/data(ms): 2.209 time/forward(ms): 2299 time/backward(ms): 6610 grad_norm: 0.06698 loss: 0.08214 steps/sec: 0.1102 ms/step: 9075 memory/max_reserved(GiB): 30.28 memory/max_reserved(%): 38.24 memory/num_ooms: 0
164
+ [10/31 00:26:55 minwm.engine.monitor]: step 240 time/data(ms): 1.785 time/forward(ms): 2112 time/backward(ms): 6783 grad_norm: 0.0766 loss: 0.1359 steps/sec: 0.11 ms/step: 9093 memory/max_reserved(GiB): 30.28 memory/max_reserved(%): 38.24 memory/num_ooms: 0
165
+ [10/31 00:28:25 minwm.engine.monitor]: step 250 time/data(ms): 1.852 time/forward(ms): 2258 time/backward(ms): 6780 grad_norm: 0.06664 loss: 0.05994 steps/sec: 0.1104 ms/step: 9054 memory/max_reserved(GiB): 30.28 memory/max_reserved(%): 38.24 memory/num_ooms: 0
166
+ [10/31 00:29:56 minwm.engine.monitor]: step 260 time/data(ms): 1.803 time/forward(ms): 2259 time/backward(ms): 6589 grad_norm: 0.4421 loss: 0.07811 steps/sec: 0.1103 ms/step: 9065 memory/max_reserved(GiB): 30.28 memory/max_reserved(%): 38.24 memory/num_ooms: 0
167
+ [10/31 00:31:27 minwm.engine.monitor]: step 270 time/data(ms): 1.894 time/forward(ms): 2273 time/backward(ms): 6606 grad_norm: 0.09639 loss: 0.07427 steps/sec: 0.1101 ms/step: 9086 memory/max_reserved(GiB): 30.28 memory/max_reserved(%): 38.24 memory/num_ooms: 0
168
+ [10/31 00:32:58 minwm.engine.monitor]: step 280 time/data(ms): 2.037 time/forward(ms): 2320 time/backward(ms): 6771 grad_norm: 0.07914 loss: 0.1117 steps/sec: 0.1103 ms/step: 9062 memory/max_reserved(GiB): 30.28 memory/max_reserved(%): 38.24 memory/num_ooms: 0
169
+ [10/31 00:34:28 minwm.engine.monitor]: step 290 time/data(ms): 2.011 time/forward(ms): 2369 time/backward(ms): 6618 grad_norm: 0.164 loss: 0.07447 steps/sec: 0.1105 ms/step: 9050 memory/max_reserved(GiB): 30.28 memory/max_reserved(%): 38.24 memory/num_ooms: 0
170
+ [10/31 00:35:58 minwm.engine.monitor]: step 300 time/data(ms): 1.757 time/forward(ms): 2270 time/backward(ms): 6598 grad_norm: 0.0953 loss: 0.0505 steps/sec: 0.111 ms/step: 9009 memory/max_reserved(GiB): 30.28 memory/max_reserved(%): 38.24 memory/num_ooms: 0
171
+ [10/31 00:37:29 minwm.engine.monitor]: step 310 time/data(ms): 2.123 time/forward(ms): 2266 time/backward(ms): 6594 grad_norm: 0.08634 loss: 0.06449 steps/sec: 0.1105 ms/step: 9051 memory/max_reserved(GiB): 30.28 memory/max_reserved(%): 38.24 memory/num_ooms: 0
172
+ [10/31 00:38:59 minwm.engine.monitor]: step 320 time/data(ms): 2.328 time/forward(ms): 2105 time/backward(ms): 6774 grad_norm: 0.07031 loss: 0.03594 steps/sec: 0.1102 ms/step: 9074 memory/max_reserved(GiB): 30.28 memory/max_reserved(%): 38.24 memory/num_ooms: 0
173
+ [10/31 00:40:31 minwm.engine.monitor]: step 330 time/data(ms): 2.232 time/forward(ms): 2110 time/backward(ms): 6778 grad_norm: 27.23 loss: 0.08933 steps/sec: 0.1098 ms/step: 9107 memory/max_reserved(GiB): 30.28 memory/max_reserved(%): 38.24 memory/num_ooms: 0
174
+ [10/31 00:42:01 minwm.engine.monitor]: step 340 time/data(ms): 2.1 time/forward(ms): 2085 time/backward(ms): 6592 grad_norm: 0.07648 loss: 0.06752 steps/sec: 0.1107 ms/step: 9032 memory/max_reserved(GiB): 30.28 memory/max_reserved(%): 38.24 memory/num_ooms: 0
175
+ [10/31 00:43:32 minwm.engine.monitor]: step 350 time/data(ms): 1.864 time/forward(ms): 2267 time/backward(ms): 6607 grad_norm: 0.08208 loss: 0.1899 steps/sec: 0.1103 ms/step: 9069 memory/max_reserved(GiB): 30.28 memory/max_reserved(%): 38.24 memory/num_ooms: 0
176
+ [10/31 00:45:02 minwm.engine.monitor]: step 360 time/data(ms): 2.151 time/forward(ms): 2109 time/backward(ms): 6786 grad_norm: 3.156 loss: 0.07607 steps/sec: 0.1102 ms/step: 9073 memory/max_reserved(GiB): 30.28 memory/max_reserved(%): 38.24 memory/num_ooms: 0
177
+ [10/31 00:46:33 minwm.engine.monitor]: step 370 time/data(ms): 2.449 time/forward(ms): 2128 time/backward(ms): 6771 grad_norm: 0.05692 loss: 0.06076 steps/sec: 0.1107 ms/step: 9034 memory/max_reserved(GiB): 30.28 memory/max_reserved(%): 38.24 memory/num_ooms: 0
178
+ [10/31 00:48:03 minwm.engine.monitor]: step 380 time/data(ms): 2.013 time/forward(ms): 2441 time/backward(ms): 6606 grad_norm: 0.07225 loss: 0.08602 steps/sec: 0.1107 ms/step: 9032 memory/max_reserved(GiB): 30.28 memory/max_reserved(%): 38.24 memory/num_ooms: 0
179
+ [10/31 00:49:33 minwm.engine.monitor]: step 390 time/data(ms): 2.283 time/forward(ms): 2267 time/backward(ms): 6607 grad_norm: 9.927 loss: 0.07774 steps/sec: 0.1106 ms/step: 9044 memory/max_reserved(GiB): 30.28 memory/max_reserved(%): 38.24 memory/num_ooms: 0
180
+ [10/31 00:51:03 minwm.engine.monitor]: step 400 time/data(ms): 2.241 time/forward(ms): 2277 time/backward(ms): 6627 grad_norm: 0.06757 loss: 0.1373 steps/sec: 0.1113 ms/step: 8984 memory/max_reserved(GiB): 30.28 memory/max_reserved(%): 38.24 memory/num_ooms: 0
181
+ [10/31 00:51:09 minwm.engine.checkpoint.checkpointer]: saved checkpoint /dev/shm/mwm/runs/tf_nomas4/ckpts/checkpoint_400
182
+ [10/31 00:52:40 minwm.engine.monitor]: step 410 time/data(ms): 1.911 time/forward(ms): 2277 time/backward(ms): 6606 grad_norm: 0.07318 loss: 0.09068 steps/sec: 0.1034 ms/step: 9669 memory/max_reserved(GiB): 30.28 memory/max_reserved(%): 38.24 memory/num_ooms: 0
183
+ [10/31 00:54:11 minwm.engine.monitor]: step 420 time/data(ms): 1.872 time/forward(ms): 2270 time/backward(ms): 6600 grad_norm: 0.0689 loss: 0.09363 steps/sec: 0.1102 ms/step: 9075 memory/max_reserved(GiB): 30.28 memory/max_reserved(%): 38.24 memory/num_ooms: 0
184
+ [10/31 00:55:42 minwm.engine.monitor]: step 430 time/data(ms): 2.042 time/forward(ms): 2307 time/backward(ms): 6742 grad_norm: 0.05472 loss: 0.04118 steps/sec: 0.11 ms/step: 9090 memory/max_reserved(GiB): 30.34 memory/max_reserved(%): 38.32 memory/num_ooms: 0
185
+ [10/31 00:57:12 minwm.engine.monitor]: step 440 time/data(ms): 1.715 time/forward(ms): 2271 time/backward(ms): 6606 grad_norm: 377.2 loss: 0.06712 steps/sec: 0.1104 ms/step: 9055 memory/max_reserved(GiB): 30.34 memory/max_reserved(%): 38.32 memory/num_ooms: 0
186
+ [10/31 00:58:42 minwm.engine.monitor]: step 450 time/data(ms): 1.999 time/forward(ms): 2118 time/backward(ms): 6801 grad_norm: 0.2745 loss: 0.09503 steps/sec: 0.1106 ms/step: 9038 memory/max_reserved(GiB): 30.34 memory/max_reserved(%): 38.32 memory/num_ooms: 0
187
+ [10/31 01:00:13 minwm.engine.monitor]: step 460 time/data(ms): 1.846 time/forward(ms): 2112 time/backward(ms): 6772 grad_norm: 731.8 loss: 0.1038 steps/sec: 0.1107 ms/step: 9030 memory/max_reserved(GiB): 30.34 memory/max_reserved(%): 38.32 memory/num_ooms: 0
188
+ [10/31 01:01:43 minwm.engine.monitor]: step 470 time/data(ms): 1.855 time/forward(ms): 2083 time/backward(ms): 6581 grad_norm: 0.07778 loss: 0.1285 steps/sec: 0.1105 ms/step: 9049 memory/max_reserved(GiB): 30.34 memory/max_reserved(%): 38.32 memory/num_ooms: 0
189
+ [10/31 01:03:14 minwm.engine.monitor]: step 480 time/data(ms): 1.655 time/forward(ms): 2106 time/backward(ms): 6770 grad_norm: 0.1432 loss: 0.09995 steps/sec: 0.1104 ms/step: 9058 memory/max_reserved(GiB): 30.34 memory/max_reserved(%): 38.32 memory/num_ooms: 0
190
+ [10/31 01:04:45 minwm.engine.monitor]: step 490 time/data(ms): 1.85 time/forward(ms): 2273 time/backward(ms): 6601 grad_norm: 12.71 loss: 0.08873 steps/sec: 0.1099 ms/step: 9102 memory/max_reserved(GiB): 30.34 memory/max_reserved(%): 38.32 memory/num_ooms: 0
191
+ [10/31 01:06:16 minwm.engine.monitor]: step 500 time/data(ms): 2.108 time/forward(ms): 2263 time/backward(ms): 6614 grad_norm: 0.08869 loss: 0.08407 steps/sec: 0.1099 ms/step: 9100 memory/max_reserved(GiB): 30.34 memory/max_reserved(%): 38.32 memory/num_ooms: 0
192
+ [10/31 01:07:47 minwm.engine.monitor]: step 510 time/data(ms): 1.998 time/forward(ms): 2133 time/backward(ms): 6785 grad_norm: 0.07441 loss: 0.1426 steps/sec: 0.1102 ms/step: 9075 memory/max_reserved(GiB): 30.34 memory/max_reserved(%): 38.32 memory/num_ooms: 0
193
+ [10/31 01:09:17 minwm.engine.monitor]: step 520 time/data(ms): 1.793 time/forward(ms): 2266 time/backward(ms): 6594 grad_norm: 0.6371 loss: 0.1089 steps/sec: 0.1105 ms/step: 9049 memory/max_reserved(GiB): 30.34 memory/max_reserved(%): 38.32 memory/num_ooms: 0
194
+ [10/31 01:10:48 minwm.engine.monitor]: step 530 time/data(ms): 1.577 time/forward(ms): 2271 time/backward(ms): 6612 grad_norm: 0.07826 loss: 0.03975 steps/sec: 0.1105 ms/step: 9048 memory/max_reserved(GiB): 30.34 memory/max_reserved(%): 38.32 memory/num_ooms: 0
195
+ [10/31 01:12:18 minwm.engine.monitor]: step 540 time/data(ms): 2.19 time/forward(ms): 2257 time/backward(ms): 6728 grad_norm: 0.07122 loss: 0.075 steps/sec: 0.1108 ms/step: 9025 memory/max_reserved(GiB): 30.34 memory/max_reserved(%): 38.32 memory/num_ooms: 0
196
+ [10/31 01:13:48 minwm.engine.monitor]: step 550 time/data(ms): 1.617 time/forward(ms): 2267 time/backward(ms): 6602 grad_norm: 0.1086 loss: 0.09965 steps/sec: 0.1103 ms/step: 9067 memory/max_reserved(GiB): 30.34 memory/max_reserved(%): 38.32 memory/num_ooms: 0
197
+ [10/31 01:15:19 minwm.engine.monitor]: step 560 time/data(ms): 2.204 time/forward(ms): 2110 time/backward(ms): 6776 grad_norm: 0.08409 loss: 0.09653 steps/sec: 0.1103 ms/step: 9070 memory/max_reserved(GiB): 30.34 memory/max_reserved(%): 38.32 memory/num_ooms: 0
198
+ [10/31 01:16:50 minwm.engine.monitor]: step 570 time/data(ms): 2.623 time/forward(ms): 2268 time/backward(ms): 6602 grad_norm: 0.3704 loss: 0.07062 steps/sec: 0.1103 ms/step: 9063 memory/max_reserved(GiB): 30.34 memory/max_reserved(%): 38.32 memory/num_ooms: 0
199
+ [10/31 01:18:21 minwm.engine.monitor]: step 580 time/data(ms): 2.214 time/forward(ms): 2297 time/backward(ms): 6617 grad_norm: 27.64 loss: 0.07063 steps/sec: 0.1101 ms/step: 9086 memory/max_reserved(GiB): 30.34 memory/max_reserved(%): 38.32 memory/num_ooms: 0
200
+ [10/31 01:19:52 minwm.engine.monitor]: step 590 time/data(ms): 1.577 time/forward(ms): 2265 time/backward(ms): 6606 grad_norm: 0.07944 loss: 0.1037 steps/sec: 0.11 ms/step: 9094 memory/max_reserved(GiB): 30.34 memory/max_reserved(%): 38.32 memory/num_ooms: 0
201
+ [10/31 01:21:22 minwm.engine.monitor]: step 600 time/data(ms): 2.043 time/forward(ms): 2272 time/backward(ms): 6607 grad_norm: 0.05966 loss: 0.1165 steps/sec: 0.1103 ms/step: 9066 memory/max_reserved(GiB): 30.34 memory/max_reserved(%): 38.32 memory/num_ooms: 0
202
+ [10/31 01:21:27 minwm.engine.checkpoint.checkpointer]: saved checkpoint /dev/shm/mwm/runs/tf_nomas4/ckpts/checkpoint_600
203
+ [10/31 01:22:57 minwm.engine.monitor]: step 610 time/data(ms): 2.148 time/forward(ms): 2089 time/backward(ms): 6586 grad_norm: 0.1183 loss: 0.1152 steps/sec: 0.1052 ms/step: 9510 memory/max_reserved(GiB): 30.34 memory/max_reserved(%): 38.32 memory/num_ooms: 0
204
+ W1031 01:23:50.877000 3209799 torch/distributed/elastic/agent/server/api.py:725] Received 15 death signal, shutting down workers
205
+ W1031 01:23:50.878000 3209799 torch/distributed/elastic/multiprocessing/api.py:908] Sending process 3209809 closing signal SIGTERM
206
+ W1031 01:23:50.881000 3209799 torch/distributed/elastic/multiprocessing/api.py:908] Sending process 3209810 closing signal SIGTERM
207
+ W1031 01:23:50.884000 3209799 torch/distributed/elastic/multiprocessing/api.py:908] Sending process 3209811 closing signal SIGTERM
208
+ W1031 01:23:50.888000 3209799 torch/distributed/elastic/multiprocessing/api.py:908] Sending process 3209812 closing signal SIGTERM
209
+ W1031 01:23:50.891000 3209799 torch/distributed/elastic/multiprocessing/api.py:908] Sending process 3209813 closing signal SIGTERM
210
+ W1031 01:23:50.895000 3209799 torch/distributed/elastic/multiprocessing/api.py:908] Sending process 3209814 closing signal SIGTERM
211
+ Traceback (most recent call last):
212
+ File "/dev/shm/mwm/venv/bin/torchrun", line 10, in <module>
213
+ sys.exit(main())
214
+ ^^^^^^
215
+ File "/dev/shm/mwm/venv/lib/python3.12/site-packages/torch/distributed/elastic/multiprocessing/errors/__init__.py", line 357, in wrapper
216
+ return f(*args, **kwargs)
217
+ ^^^^^^^^^^^^^^^^^^
218
+ File "/dev/shm/mwm/venv/lib/python3.12/site-packages/torch/distributed/run.py", line 936, in main
219
+ run(args)
220
+ File "/dev/shm/mwm/venv/lib/python3.12/site-packages/torch/distributed/run.py", line 927, in run
221
+ elastic_launch(
222
+ File "/dev/shm/mwm/venv/lib/python3.12/site-packages/torch/distributed/launcher/api.py", line 156, in __call__
223
+ return launch_agent(self._config, self._entrypoint, list(args))
224
+ ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
225
+ File "/dev/shm/mwm/venv/lib/python3.12/site-packages/torch/distributed/launcher/api.py", line 284, in launch_agent
226
+ result = agent.run()
227
+ ^^^^^^^^^^^
228
+ File "/dev/shm/mwm/venv/lib/python3.12/site-packages/torch/distributed/elastic/metrics/api.py", line 138, in wrapper
229
+ result = f(*args, **kwargs)
230
+ ^^^^^^^^^^^^^^^^^^
231
+ File "/dev/shm/mwm/venv/lib/python3.12/site-packages/torch/distributed/elastic/agent/server/api.py", line 717, in run
232
+ result = self._invoke_run(role)
233
+ ^^^^^^^^^^^^^^^^^^^^^^
234
+ File "/dev/shm/mwm/venv/lib/python3.12/site-packages/torch/distributed/elastic/agent/server/api.py", line 881, in _invoke_run
235
+ time.sleep(monitor_interval)
236
+ File "/dev/shm/mwm/venv/lib/python3.12/site-packages/torch/distributed/elastic/multiprocessing/api.py", line 85, in _terminate_process_handler
237
+ raise SignalException(f"Process {os.getpid()} got signal: {sigval}", sigval=sigval)
238
+ torch.distributed.elastic.multiprocessing.api.SignalException: Process 3209799 got signal: 15
239
+ [W1031 01:23:51.450972437 AllocatorConfig.cpp:28] Warning: PYTORCH_CUDA_ALLOC_CONF is deprecated, use PYTORCH_ALLOC_CONF instead (function operator())
240
+ MWM_TRAIN3 repo /dev/shm/mwm/repo data /dev/shm/mwm/data/quality char_labels none char_k 200 drop 0.1
241
+ MWM_TRAIN config configs/wan21/action2v/train/stage1_ar_tf_aug.py
242
+ MWM_TRAIN start 01:23:56 stage tf name tf_nomas4 gpus 0,1,2,3,4,5 n 6 steps 1200 lr 5e-6 init /dev/shm/mwm/exports/tf_nomas3/step_1000.pt teacher same extra checkpoint.resume=True
243
+ [10/31 01:24:00 minwm.distributed.sp.parallel_state]: Initializing distributed environment with world_size=6, device=cuda:0
244
+ [10/31 01:24:00 minwm.distributed.sp.parallel_state]: world_size=6 rank=0 local_rank=0 distributed_init_method=env:// backend=nccl
245
+ [Gloo] Rank 2 is connected to 5 peer ranks. Expected number of connected peer ranks is : 5
246
+ [Gloo] Rank 0 is connected to [Gloo] Rank 5 peer ranks. 1Expected number of connected peer ranks is : is connected to 55 peer ranks. Expected number of connected peer ranks is :
247
+ 5
248
+ [Gloo] Rank 5 is connected to 5 peer ranks. Expected number of connected peer ranks is : 5
249
+ [Gloo] Rank [Gloo] Rank 34 is connected to is connected to 55 peer ranks. peer ranks. Expected number of connected peer ranks is : Expected number of connected peer ranks is : 55
250
+
251
+ [10/31 01:24:05 minwm.distributed.sp.nccl]: Found nccl from library libnccl.so.2
252
+ [10/31 01:24:05 minwm.distributed.sp.device_communicators.pynccl]: Trainer is using nccl==2.27.5
253
+ [Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
254
+ [Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
255
+ [Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
256
+ [Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
257
+ [Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
258
+ [Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
259
+ [Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
260
+ [Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
261
+ [Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
262
+ [Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
263
+ [Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
264
+ [Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
265
+ [Gloo] Rank 0 is connected to 5 peer ranks. Expected number of connected peer ranks is : 5
266
+ [Gloo] Rank 5 is connected to 5 peer ranks. Expected number of connected peer ranks is : 5
267
+ [Gloo] Rank 1 is connected to 5 peer ranks. Expected number of connected peer ranks is : 5
268
+ [Gloo] Rank 2 is connected to 5 peer ranks. Expected number of connected peer ranks is : 5
269
+ [Gloo] Rank 3 is connected to 5 peer ranks. Expected number of connected peer ranks is : 5
270
+ [Gloo] Rank 4 is connected to 5 peer ranks. Expected number of connected peer ranks is : 5
271
+ [10/31 01:24:06 minwm.distributed.sp.nccl]: Found nccl from library libnccl.so.2
272
+ [10/31 01:24:06 minwm.distributed.sp.device_communicators.pynccl]: Trainer is using nccl==2.27.5
273
+ [10/31 01:24:08 minwm.engine.trainer]: base RNG seed: 2050329764 (+ dp_rank)
274
+ Some weights of CausalWan21Model were not initialized from the model checkpoint at ./ckpts/Wan2.1-T2V-1.3B and are newly initialized: ['blocks.12.self_attn.prope_o.weight', 'blocks.23.self_attn.prope_o.bias', 'blocks.24.self_attn.prope_o.bias', 'blocks.7.self_attn.prope_o.bias', 'blocks.12.self_attn.prope_o.bias', 'blocks.19.self_attn.prope_o.weight', 'blocks.7.self_attn.prope_o.weight', 'blocks.28.self_attn.prope_o.bias', 'blocks.25.self_attn.prope_o.weight', 'blocks.22.self_attn.prope_o.bias', 'blocks.15.self_attn.prope_o.weight', 'blocks.11.self_attn.prope_o.bias', 'blocks.29.self_attn.prope_o.bias', 'blocks.23.self_attn.prope_o.weight', 'blocks.18.self_attn.prope_o.bias', 'blocks.17.self_attn.prope_o.weight', 'blocks.26.self_attn.prope_o.bias', 'blocks.13.self_attn.prope_o.bias', 'blocks.9.self_attn.prope_o.weight', 'blocks.4.self_attn.prope_o.bias', 'blocks.16.self_attn.prope_o.bias', 'blocks.6.self_attn.prope_o.weight', 'blocks.0.self_attn.prope_o.weight', 'blocks.20.self_attn.prope_o.weight', 'blocks.2.self_attn.prope_o.weight', 'blocks.22.self_attn.prope_o.weight', 'blocks.14.self_attn.prope_o.bias', 'blocks.3.self_attn.prope_o.weight', 'blocks.10.self_attn.prope_o.bias', 'blocks.24.self_attn.prope_o.weight', 'blocks.1.self_attn.prope_o.bias', 'blocks.14.self_attn.prope_o.weight', 'blocks.0.self_attn.prope_o.bias', 'blocks.19.self_attn.prope_o.bias', 'blocks.8.self_attn.prope_o.weight', 'blocks.27.self_attn.prope_o.weight', 'blocks.5.self_attn.prope_o.weight', 'blocks.9.self_attn.prope_o.bias', 'blocks.10.self_attn.prope_o.weight', 'blocks.5.self_attn.prope_o.bias', 'blocks.3.self_attn.prope_o.bias', 'blocks.29.self_attn.prope_o.weight', 'blocks.4.self_attn.prope_o.weight', 'blocks.26.self_attn.prope_o.weight', 'blocks.20.self_attn.prope_o.bias', 'blocks.17.self_attn.prope_o.bias', 'blocks.25.self_attn.prope_o.bias', 'blocks.28.self_attn.prope_o.weight', 'blocks.21.self_attn.prope_o.weight', 'blocks.18.self_attn.prope_o.weight', 'blocks.2.self_attn.prope_o.bias', 'blocks.21.self_attn.prope_o.bias', 'blocks.27.self_attn.prope_o.bias', 'blocks.1.self_attn.prope_o.weight', 'blocks.8.self_attn.prope_o.bias', 'blocks.6.self_attn.prope_o.bias', 'blocks.11.self_attn.prope_o.weight', 'blocks.16.self_attn.prope_o.weight', 'blocks.15.self_attn.prope_o.bias', 'blocks.13.self_attn.prope_o.weight']
275
+ You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.
276
+ Some weights of CausalWan21Model were not initialized from the model checkpoint at ./ckpts/Wan2.1-T2V-1.3B and are newly initialized: ['blocks.14.self_attn.prope_o.bias', 'blocks.22.self_attn.prope_o.bias', 'blocks.23.self_attn.prope_o.bias', 'blocks.15.self_attn.prope_o.bias', 'blocks.10.self_attn.prope_o.bias', 'blocks.9.self_attn.prope_o.weight', 'blocks.12.self_attn.prope_o.bias', 'blocks.1.self_attn.prope_o.weight', 'blocks.4.self_attn.prope_o.bias', 'blocks.21.self_attn.prope_o.weight', 'blocks.16.self_attn.prope_o.weight', 'blocks.19.self_attn.prope_o.weight', 'blocks.7.self_attn.prope_o.bias', 'blocks.7.self_attn.prope_o.weight', 'blocks.21.self_attn.prope_o.bias', 'blocks.24.self_attn.prope_o.bias', 'blocks.4.self_attn.prope_o.weight', 'blocks.6.self_attn.prope_o.bias', 'blocks.29.self_attn.prope_o.weight', 'blocks.22.self_attn.prope_o.weight', 'blocks.20.self_attn.prope_o.bias', 'blocks.17.self_attn.prope_o.bias', 'blocks.20.self_attn.prope_o.weight', 'blocks.3.self_attn.prope_o.bias', 'blocks.9.self_attn.prope_o.bias', 'blocks.11.self_attn.prope_o.weight', 'blocks.11.self_attn.prope_o.bias', 'blocks.19.self_attn.prope_o.bias', 'blocks.27.self_attn.prope_o.weight', 'blocks.13.self_attn.prope_o.bias', 'blocks.29.self_attn.prope_o.bias', 'blocks.6.self_attn.prope_o.weight', 'blocks.0.self_attn.prope_o.bias', 'blocks.14.self_attn.prope_o.weight', 'blocks.10.self_attn.prope_o.weight', 'blocks.25.self_attn.prope_o.bias', 'blocks.26.self_attn.prope_o.weight', 'blocks.0.self_attn.prope_o.weight', 'blocks.8.self_attn.prope_o.bias', 'blocks.27.self_attn.prope_o.bias', 'blocks.18.self_attn.prope_o.weight', 'blocks.26.self_attn.prope_o.bias', 'blocks.3.self_attn.prope_o.weight', 'blocks.8.self_attn.prope_o.weight', 'blocks.18.self_attn.prope_o.bias', 'blocks.2.self_attn.prope_o.bias', 'blocks.2.self_attn.prope_o.weight', 'blocks.25.self_attn.prope_o.weight', 'blocks.1.self_attn.prope_o.bias', 'blocks.23.self_attn.prope_o.weight', 'blocks.24.self_attn.prope_o.weight', 'blocks.13.self_attn.prope_o.weight', 'blocks.17.self_attn.prope_o.weight', 'blocks.5.self_attn.prope_o.weight', 'blocks.12.self_attn.prope_o.weight', 'blocks.28.self_attn.prope_o.weight', 'blocks.5.self_attn.prope_o.bias', 'blocks.15.self_attn.prope_o.weight', 'blocks.28.self_attn.prope_o.bias', 'blocks.16.self_attn.prope_o.bias']
277
+ You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.
278
+ Some weights of CausalWan21Model were not initialized from the model checkpoint at ./ckpts/Wan2.1-T2V-1.3B and are newly initialized: ['blocks.20.self_attn.prope_o.bias', 'blocks.1.self_attn.prope_o.bias', 'blocks.25.self_attn.prope_o.weight', 'blocks.29.self_attn.prope_o.bias', 'blocks.24.self_attn.prope_o.bias', 'blocks.9.self_attn.prope_o.bias', 'blocks.15.self_attn.prope_o.weight', 'blocks.1.self_attn.prope_o.weight', 'blocks.23.self_attn.prope_o.weight', 'blocks.24.self_attn.prope_o.weight', 'blocks.10.self_attn.prope_o.bias', 'blocks.0.self_attn.prope_o.weight', 'blocks.2.self_attn.prope_o.weight', 'blocks.25.self_attn.prope_o.bias', 'blocks.5.self_attn.prope_o.weight', 'blocks.3.self_attn.prope_o.weight', 'blocks.20.self_attn.prope_o.weight', 'blocks.13.self_attn.prope_o.weight', 'blocks.22.self_attn.prope_o.weight', 'blocks.11.self_attn.prope_o.weight', 'blocks.2.self_attn.prope_o.bias', 'blocks.11.self_attn.prope_o.bias', 'blocks.19.self_attn.prope_o.bias', 'blocks.17.self_attn.prope_o.weight', 'blocks.29.self_attn.prope_o.weight', 'blocks.21.self_attn.prope_o.weight', 'blocks.27.self_attn.prope_o.bias', 'blocks.4.self_attn.prope_o.weight', 'blocks.28.self_attn.prope_o.weight', 'blocks.4.self_attn.prope_o.bias', 'blocks.16.self_attn.prope_o.weight', 'blocks.8.self_attn.prope_o.weight', 'blocks.15.self_attn.prope_o.bias', 'blocks.14.self_attn.prope_o.bias', 'blocks.26.self_attn.prope_o.weight', 'blocks.10.self_attn.prope_o.weight', 'blocks.6.self_attn.prope_o.bias', 'blocks.18.self_attn.prope_o.bias', 'blocks.8.self_attn.prope_o.bias', 'blocks.28.self_attn.prope_o.bias', 'blocks.16.self_attn.prope_o.bias', 'blocks.9.self_attn.prope_o.weight', 'blocks.7.self_attn.prope_o.bias', 'blocks.6.self_attn.prope_o.weight', 'blocks.27.self_attn.prope_o.weight', 'blocks.21.self_attn.prope_o.bias', 'blocks.7.self_attn.prope_o.weight', 'blocks.22.self_attn.prope_o.bias', 'blocks.12.self_attn.prope_o.weight', 'blocks.0.self_attn.prope_o.bias', 'blocks.12.self_attn.prope_o.bias', 'blocks.17.self_attn.prope_o.bias', 'blocks.3.self_attn.prope_o.bias', 'blocks.5.self_attn.prope_o.bias', 'blocks.26.self_attn.prope_o.bias', 'blocks.13.self_attn.prope_o.bias', 'blocks.19.self_attn.prope_o.weight', 'blocks.14.self_attn.prope_o.weight', 'blocks.18.self_attn.prope_o.weight', 'blocks.23.self_attn.prope_o.bias']
279
+ You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.
280
+ Some weights of CausalWan21Model were not initialized from the model checkpoint at ./ckpts/Wan2.1-T2V-1.3B and are newly initialized: ['blocks.0.self_attn.prope_o.weight', 'blocks.29.self_attn.prope_o.bias', 'blocks.16.self_attn.prope_o.bias', 'blocks.21.self_attn.prope_o.weight', 'blocks.2.self_attn.prope_o.bias', 'blocks.19.self_attn.prope_o.weight', 'blocks.24.self_attn.prope_o.weight', 'blocks.24.self_attn.prope_o.bias', 'blocks.5.self_attn.prope_o.bias', 'blocks.18.self_attn.prope_o.weight', 'blocks.9.self_attn.prope_o.bias', 'blocks.3.self_attn.prope_o.bias', 'blocks.26.self_attn.prope_o.weight', 'blocks.12.self_attn.prope_o.bias', 'blocks.29.self_attn.prope_o.weight', 'blocks.27.self_attn.prope_o.bias', 'blocks.14.self_attn.prope_o.bias', 'blocks.10.self_attn.prope_o.weight', 'blocks.17.self_attn.prope_o.bias', 'blocks.4.self_attn.prope_o.weight', 'blocks.27.self_attn.prope_o.weight', 'blocks.15.self_attn.prope_o.bias', 'blocks.4.self_attn.prope_o.bias', 'blocks.9.self_attn.prope_o.weight', 'blocks.19.self_attn.prope_o.bias', 'blocks.22.self_attn.prope_o.bias', 'blocks.3.self_attn.prope_o.weight', 'blocks.12.self_attn.prope_o.weight', 'blocks.28.self_attn.prope_o.bias', 'blocks.15.self_attn.prope_o.weight', 'blocks.7.self_attn.prope_o.bias', 'blocks.11.self_attn.prope_o.weight', 'blocks.8.self_attn.prope_o.weight', 'blocks.5.self_attn.prope_o.weight', 'blocks.7.self_attn.prope_o.weight', 'blocks.28.self_attn.prope_o.weight', 'blocks.23.self_attn.prope_o.bias', 'blocks.10.self_attn.prope_o.bias', 'blocks.18.self_attn.prope_o.bias', 'blocks.14.self_attn.prope_o.weight', 'blocks.0.self_attn.prope_o.bias', 'blocks.23.self_attn.prope_o.weight', 'blocks.6.self_attn.prope_o.bias', 'blocks.13.self_attn.prope_o.bias', 'blocks.22.self_attn.prope_o.weight', 'blocks.2.self_attn.prope_o.weight', 'blocks.25.self_attn.prope_o.bias', 'blocks.21.self_attn.prope_o.bias', 'blocks.1.self_attn.prope_o.weight', 'blocks.26.self_attn.prope_o.bias', 'blocks.6.self_attn.prope_o.weight', 'blocks.20.self_attn.prope_o.weight', 'blocks.17.self_attn.prope_o.weight', 'blocks.16.self_attn.prope_o.weight', 'blocks.1.self_attn.prope_o.bias', 'blocks.20.self_attn.prope_o.bias', 'blocks.13.self_attn.prope_o.weight', 'blocks.11.self_attn.prope_o.bias', 'blocks.8.self_attn.prope_o.bias', 'blocks.25.self_attn.prope_o.weight']
281
+ You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.
282
+ Some weights of CausalWan21Model were not initialized from the model checkpoint at ./ckpts/Wan2.1-T2V-1.3B and are newly initialized: ['blocks.0.self_attn.prope_o.bias', 'blocks.3.self_attn.prope_o.weight', 'blocks.11.self_attn.prope_o.bias', 'blocks.11.self_attn.prope_o.weight', 'blocks.8.self_attn.prope_o.bias', 'blocks.17.self_attn.prope_o.weight', 'blocks.13.self_attn.prope_o.weight', 'blocks.28.self_attn.prope_o.weight', 'blocks.24.self_attn.prope_o.weight', 'blocks.18.self_attn.prope_o.weight', 'blocks.1.self_attn.prope_o.bias', 'blocks.15.self_attn.prope_o.bias', 'blocks.23.self_attn.prope_o.bias', 'blocks.5.self_attn.prope_o.bias', 'blocks.20.self_attn.prope_o.weight', 'blocks.7.self_attn.prope_o.weight', 'blocks.0.self_attn.prope_o.weight', 'blocks.29.self_attn.prope_o.weight', 'blocks.12.self_attn.prope_o.bias', 'blocks.22.self_attn.prope_o.bias', 'blocks.24.self_attn.prope_o.bias', 'blocks.23.self_attn.prope_o.weight', 'blocks.16.self_attn.prope_o.bias', 'blocks.21.self_attn.prope_o.bias', 'blocks.9.self_attn.prope_o.weight', 'blocks.21.self_attn.prope_o.weight', 'blocks.1.self_attn.prope_o.weight', 'blocks.16.self_attn.prope_o.weight', 'blocks.27.self_attn.prope_o.bias', 'blocks.9.self_attn.prope_o.bias', 'blocks.10.self_attn.prope_o.weight', 'blocks.20.self_attn.prope_o.bias', 'blocks.4.self_attn.prope_o.weight', 'blocks.5.self_attn.prope_o.weight', 'blocks.17.self_attn.prope_o.bias', 'blocks.19.self_attn.prope_o.weight', 'blocks.10.self_attn.prope_o.bias', 'blocks.7.self_attn.prope_o.bias', 'blocks.19.self_attn.prope_o.bias', 'blocks.18.self_attn.prope_o.bias', 'blocks.22.self_attn.prope_o.weight', 'blocks.2.self_attn.prope_o.weight', 'blocks.29.self_attn.prope_o.bias', 'blocks.14.self_attn.prope_o.weight', 'blocks.6.self_attn.prope_o.weight', 'blocks.6.self_attn.prope_o.bias', 'blocks.4.self_attn.prope_o.bias', 'blocks.13.self_attn.prope_o.bias', 'blocks.15.self_attn.prope_o.weight', 'blocks.12.self_attn.prope_o.weight', 'blocks.2.self_attn.prope_o.bias', 'blocks.25.self_attn.prope_o.weight', 'blocks.27.self_attn.prope_o.weight', 'blocks.14.self_attn.prope_o.bias', 'blocks.26.self_attn.prope_o.bias', 'blocks.26.self_attn.prope_o.weight', 'blocks.3.self_attn.prope_o.bias', 'blocks.25.self_attn.prope_o.bias', 'blocks.28.self_attn.prope_o.bias', 'blocks.8.self_attn.prope_o.weight']
283
+ You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.
284
+ Some weights of CausalWan21Model were not initialized from the model checkpoint at ./ckpts/Wan2.1-T2V-1.3B and are newly initialized: ['blocks.0.self_attn.prope_o.bias', 'blocks.5.self_attn.prope_o.bias', 'blocks.24.self_attn.prope_o.bias', 'blocks.10.self_attn.prope_o.bias', 'blocks.24.self_attn.prope_o.weight', 'blocks.20.self_attn.prope_o.bias', 'blocks.7.self_attn.prope_o.weight', 'blocks.1.self_attn.prope_o.weight', 'blocks.11.self_attn.prope_o.weight', 'blocks.2.self_attn.prope_o.weight', 'blocks.12.self_attn.prope_o.weight', 'blocks.9.self_attn.prope_o.bias', 'blocks.26.self_attn.prope_o.weight', 'blocks.20.self_attn.prope_o.weight', 'blocks.29.self_attn.prope_o.weight', 'blocks.3.self_attn.prope_o.weight', 'blocks.8.self_attn.prope_o.weight', 'blocks.23.self_attn.prope_o.weight', 'blocks.26.self_attn.prope_o.bias', 'blocks.12.self_attn.prope_o.bias', 'blocks.0.self_attn.prope_o.weight', 'blocks.19.self_attn.prope_o.bias', 'blocks.27.self_attn.prope_o.bias', 'blocks.23.self_attn.prope_o.bias', 'blocks.16.self_attn.prope_o.bias', 'blocks.6.self_attn.prope_o.weight', 'blocks.18.self_attn.prope_o.bias', 'blocks.13.self_attn.prope_o.bias', 'blocks.1.self_attn.prope_o.bias', 'blocks.25.self_attn.prope_o.weight', 'blocks.8.self_attn.prope_o.bias', 'blocks.21.self_attn.prope_o.weight', 'blocks.4.self_attn.prope_o.weight', 'blocks.27.self_attn.prope_o.weight', 'blocks.16.self_attn.prope_o.weight', 'blocks.17.self_attn.prope_o.weight', 'blocks.2.self_attn.prope_o.bias', 'blocks.22.self_attn.prope_o.weight', 'blocks.10.self_attn.prope_o.weight', 'blocks.29.self_attn.prope_o.bias', 'blocks.15.self_attn.prope_o.bias', 'blocks.17.self_attn.prope_o.bias', 'blocks.4.self_attn.prope_o.bias', 'blocks.22.self_attn.prope_o.bias', 'blocks.6.self_attn.prope_o.bias', 'blocks.21.self_attn.prope_o.bias', 'blocks.15.self_attn.prope_o.weight', 'blocks.28.self_attn.prope_o.bias', 'blocks.28.self_attn.prope_o.weight', 'blocks.13.self_attn.prope_o.weight', 'blocks.9.self_attn.prope_o.weight', 'blocks.14.self_attn.prope_o.weight', 'blocks.11.self_attn.prope_o.bias', 'blocks.19.self_attn.prope_o.weight', 'blocks.18.self_attn.prope_o.weight', 'blocks.5.self_attn.prope_o.weight', 'blocks.3.self_attn.prope_o.bias', 'blocks.7.self_attn.prope_o.bias', 'blocks.25.self_attn.prope_o.bias', 'blocks.14.self_attn.prope_o.bias']
285
+ You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.
286
+ [10/31 01:24:11 minwm.distributed.fsdp]: FSDP2: activation-checkpointed 30 layer module(s)
287
+ [10/31 01:24:12 minwm.distributed.fsdp]: FSDP2: sharded 30 layer module(s) in FSDPCausalWan21Model
288
+ [rank5]: Traceback (most recent call last):
289
+ [rank5]: File "/dev/shm/mwm/minWM-main/tools/train_mwm.py", line 74, in <module>
290
+ [rank5]: main()
291
+ [rank5]: File "/dev/shm/mwm/minWM-main/tools/train_mwm.py", line 69, in main
292
+ [rank5]: trainer = build_trainer(cfg)
293
+ [rank5]: ^^^^^^^^^^^^^^^^^^
294
+ [rank5]: File "/dev/shm/mwm/minWM-main/tools/train_mwm.py", line 50, in build_trainer
295
+ [rank5]: return trainer_cls(MWMConfig.from_dict(cfg))
296
+ [rank5]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
297
+ [rank5]: File "/dev/shm/mwm/repo/minwm/engine/trainer.py", line 167, in __init__
298
+ [rank5]: self.dataloader = self.build_dataloader(self.cfg.data)
299
+ [rank5]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
300
+ [rank5]: File "/dev/shm/mwm/repo/minwm/engine/trainer.py", line 386, in build_dataloader
301
+ [rank5]: return _build_dataloader_cfg(data)
302
+ [rank5]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^
303
+ [rank5]: File "/dev/shm/mwm/repo/minwm/data/build.py", line 68, in build_dataloader
304
+ [rank5]: dataset = build(data.dataset)
305
+ [rank5]: ^^^^^^^^^^^^^^^^^^^
306
+ [rank5]: File "/dev/shm/mwm/repo/minwm/config/lazy.py", line 126, in build
307
+ [rank5]: return cls(**kwargs)
308
+ [rank5]: ^^^^^^^^^^^^^
309
+ [rank5]: File "/dev/shm/mwm/repo/minwm/data/datasets/lmdb.py", line 204, in __init__
310
+ [rank5]: self._sds = _open_shards(data_path)
311
+ [rank5]: ^^^^^^^^^^^^^^^^^^^^^^^
312
+ [rank5]: File "/dev/shm/mwm/repo/minwm/data/datasets/lmdb.py", line 63, in _open_shards
313
+ [rank5]: envs.append(_open(sub))
314
+ [rank5]: ^^^^^^^^^^
315
+ [rank5]: File "/dev/shm/mwm/repo/minwm/data/datasets/lmdb.py", line 27, in _open
316
+ [rank5]: return lmdb.open(path, readonly=True, lock=False, readahead=False, meminit=False)
317
+ [rank5]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
318
+ [rank5]: lmdb.Error: The environment '/dev/shm/mwm/data/quality/shard_q001b' is already open in this process.
319
+ [rank4]: Traceback (most recent call last):
320
+ [rank4]: File "/dev/shm/mwm/minWM-main/tools/train_mwm.py", line 74, in <module>
321
+ [rank4]: main()
322
+ [rank4]: File "/dev/shm/mwm/minWM-main/tools/train_mwm.py", line 69, in main
323
+ [rank4]: trainer = build_trainer(cfg)
324
+ [rank4]: ^^^^^^^^^^^^^^^^^^
325
+ [rank4]: File "/dev/shm/mwm/minWM-main/tools/train_mwm.py", line 50, in build_trainer
326
+ [rank4]: return trainer_cls(MWMConfig.from_dict(cfg))
327
+ [rank4]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
328
+ [rank4]: File "/dev/shm/mwm/repo/minwm/engine/trainer.py", line 167, in __init__
329
+ [rank4]: self.dataloader = self.build_dataloader(self.cfg.data)
330
+ [rank4]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
331
+ [rank4]: File "/dev/shm/mwm/repo/minwm/engine/trainer.py", line 386, in build_dataloader
332
+ [rank4]: return _build_dataloader_cfg(data)
333
+ [rank4]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^
334
+ [rank4]: File "/dev/shm/mwm/repo/minwm/data/build.py", line 68, in build_dataloader
335
+ [rank4]: dataset = build(data.dataset)
336
+ [rank4]: ^^^^^^^^^^^^^^^^^^^
337
+ [rank4]: File "/dev/shm/mwm/repo/minwm/config/lazy.py", line 126, in build
338
+ [rank4]: return cls(**kwargs)
339
+ [rank4]: ^^^^^^^^^^^^^
340
+ [rank4]: File "/dev/shm/mwm/repo/minwm/data/datasets/lmdb.py", line 204, in __init__
341
+ [rank4]: self._sds = _open_shards(data_path)
342
+ [rank4]: ^^^^^^^^^^^^^^^^^^^^^^^
343
+ [rank4]: File "/dev/shm/mwm/repo/minwm/data/datasets/lmdb.py", line 63, in _open_shards
344
+ [rank4]: envs.append(_open(sub))
345
+ [rank4]: ^^^^^^^^^^
346
+ [rank4]: File "/dev/shm/mwm/repo/minwm/data/datasets/lmdb.py", line 27, in _open
347
+ [rank4]: return lmdb.open(path, readonly=True, lock=False, readahead=False, meminit=False)
348
+ [rank4]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
349
+ [rank4]: lmdb.Error: The environment '/dev/shm/mwm/data/quality/shard_q001b' is already open in this process.
350
+ [rank4]:[W1031 01:25:04.127750030 ProcessGroupNCCL.cpp:1524] Warning: WARNING: destroy_process_group() was not called before program exit, which can leak resources. For more info, please see https://pytorch.org/docs/stable/distributed.html#shutdown (function operator())
351
+ [rank5]:[W1031 01:25:04.130113754 ProcessGroupNCCL.cpp:1524] Warning: WARNING: destroy_process_group() was not called before program exit, which can leak resources. For more info, please see https://pytorch.org/docs/stable/distributed.html#shutdown (function operator())
352
+ [rank2]: Traceback (most recent call last):
353
+ [rank2]: File "/dev/shm/mwm/minWM-main/tools/train_mwm.py", line 74, in <module>
354
+ [rank2]: main()
355
+ [rank2]: File "/dev/shm/mwm/minWM-main/tools/train_mwm.py", line 69, in main
356
+ [rank2]: trainer = build_trainer(cfg)
357
+ [rank2]: ^^^^^^^^^^^^^^^^^^
358
+ [rank2]: File "/dev/shm/mwm/minWM-main/tools/train_mwm.py", line 50, in build_trainer
359
+ [rank2]: return trainer_cls(MWMConfig.from_dict(cfg))
360
+ [rank2]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
361
+ [rank2]: File "/dev/shm/mwm/repo/minwm/engine/trainer.py", line 167, in __init__
362
+ [rank2]: self.dataloader = self.build_dataloader(self.cfg.data)
363
+ [rank2]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
364
+ [rank2]: File "/dev/shm/mwm/repo/minwm/engine/trainer.py", line 386, in build_dataloader
365
+ [rank2]: return _build_dataloader_cfg(data)
366
+ [rank2]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^
367
+ [rank2]: File "/dev/shm/mwm/repo/minwm/data/build.py", line 68, in build_dataloader
368
+ [rank2]: dataset = build(data.dataset)
369
+ [rank2]: ^^^^^^^^^^^^^^^^^^^
370
+ [rank2]: File "/dev/shm/mwm/repo/minwm/config/lazy.py", line 126, in build
371
+ [rank2]: return cls(**kwargs)
372
+ [rank2]: ^^^^^^^^^^^^^
373
+ [rank2]: File "/dev/shm/mwm/repo/minwm/data/datasets/lmdb.py", line 204, in __init__
374
+ [rank2]: self._sds = _open_shards(data_path)
375
+ [rank2]: ^^^^^^^^^^^^^^^^^^^^^^^
376
+ [rank2]: File "/dev/shm/mwm/repo/minwm/data/datasets/lmdb.py", line 63, in _open_shards
377
+ [rank2]: envs.append(_open(sub))
378
+ [rank2]: ^^^^^^^^^^
379
+ [rank2]: File "/dev/shm/mwm/repo/minwm/data/datasets/lmdb.py", line 27, in _open
380
+ [rank2]: return lmdb.open(path, readonly=True, lock=False, readahead=False, meminit=False)
381
+ [rank2]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
382
+ [rank2]: lmdb.Error: The environment '/dev/shm/mwm/data/quality/shard_q001b' is already open in this process.
383
+ [rank3]: Traceback (most recent call last):
384
+ [rank3]: File "/dev/shm/mwm/minWM-main/tools/train_mwm.py", line 74, in <module>
385
+ [rank3]: main()
386
+ [rank3]: File "/dev/shm/mwm/minWM-main/tools/train_mwm.py", line 69, in main
387
+ [rank3]: trainer = build_trainer(cfg)
388
+ [rank3]: ^^^^^^^^^^^^^^^^^^
389
+ [rank3]: File "/dev/shm/mwm/minWM-main/tools/train_mwm.py", line 50, in build_trainer
390
+ [rank3]: return trainer_cls(MWMConfig.from_dict(cfg))
391
+ [rank3]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
392
+ [rank3]: File "/dev/shm/mwm/repo/minwm/engine/trainer.py", line 167, in __init__
393
+ [rank3]: self.dataloader = self.build_dataloader(self.cfg.data)
394
+ [rank3]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
395
+ [rank3]: File "/dev/shm/mwm/repo/minwm/engine/trainer.py", line 386, in build_dataloader
396
+ [rank3]: return _build_dataloader_cfg(data)
397
+ [rank3]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^
398
+ [rank3]: File "/dev/shm/mwm/repo/minwm/data/build.py", line 68, in build_dataloader
399
+ [rank3]: dataset = build(data.dataset)
400
+ [rank3]: ^^^^^^^^^^^^^^^^^^^
401
+ [rank3]: File "/dev/shm/mwm/repo/minwm/config/lazy.py", line 126, in build
402
+ [rank3]: return cls(**kwargs)
403
+ [rank3]: ^^^^^^^^^^^^^
404
+ [rank3]: File "/dev/shm/mwm/repo/minwm/data/datasets/lmdb.py", line 204, in __init__
405
+ [rank3]: self._sds = _open_shards(data_path)
406
+ [rank3]: ^^^^^^^^^^^^^^^^^^^^^^^
407
+ [rank3]: File "/dev/shm/mwm/repo/minwm/data/datasets/lmdb.py", line 63, in _open_shards
408
+ [rank3]: envs.append(_open(sub))
409
+ [rank3]: ^^^^^^^^^^
410
+ [rank3]: File "/dev/shm/mwm/repo/minwm/data/datasets/lmdb.py", line 27, in _open
411
+ [rank3]: return lmdb.open(path, readonly=True, lock=False, readahead=False, meminit=False)
412
+ [rank3]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
413
+ [rank3]: lmdb.Error: The environment '/dev/shm/mwm/data/quality/shard_q001b' is already open in this process.
414
+ [rank5]:[W1031 01:25:05.054446080 AllocatorConfig.cpp:28] Warning: PYTORCH_CUDA_ALLOC_CONF is deprecated, use PYTORCH_ALLOC_CONF instead (function operator())
415
+ [rank4]:[W1031 01:25:05.073169814 AllocatorConfig.cpp:28] Warning: PYTORCH_CUDA_ALLOC_CONF is deprecated, use PYTORCH_ALLOC_CONF instead (function operator())
416
+ [rank0]: Traceback (most recent call last):
417
+ [rank0]: File "/dev/shm/mwm/minWM-main/tools/train_mwm.py", line 74, in <module>
418
+ [rank0]: main()
419
+ [rank0]: File "/dev/shm/mwm/minWM-main/tools/train_mwm.py", line 69, in main
420
+ [rank0]: trainer = build_trainer(cfg)
421
+ [rank0]: ^^^^^^^^^^^^^^^^^^
422
+ [rank0]: File "/dev/shm/mwm/minWM-main/tools/train_mwm.py", line 50, in build_trainer
423
+ [rank0]: return trainer_cls(MWMConfig.from_dict(cfg))
424
+ [rank0]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
425
+ [rank0]: File "/dev/shm/mwm/repo/minwm/engine/trainer.py", line 167, in __init__
426
+ [rank0]: self.dataloader = self.build_dataloader(self.cfg.data)
427
+ [rank0]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
428
+ [rank0]: File "/dev/shm/mwm/repo/minwm/engine/trainer.py", line 386, in build_dataloader
429
+ [rank0]: return _build_dataloader_cfg(data)
430
+ [rank0]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^
431
+ [rank0]: File "/dev/shm/mwm/repo/minwm/data/build.py", line 68, in build_dataloader
432
+ [rank0]: dataset = build(data.dataset)
433
+ [rank0]: ^^^^^^^^^^^^^^^^^^^
434
+ [rank0]: File "/dev/shm/mwm/repo/minwm/config/lazy.py", line 126, in build
435
+ [rank0]: return cls(**kwargs)
436
+ [rank0]: ^^^^^^^^^^^^^
437
+ [rank0]: File "/dev/shm/mwm/repo/minwm/data/datasets/lmdb.py", line 204, in __init__
438
+ [rank0]: self._sds = _open_shards(data_path)
439
+ [rank0]: ^^^^^^^^^^^^^^^^^^^^^^^
440
+ [rank0]: File "/dev/shm/mwm/repo/minwm/data/datasets/lmdb.py", line 63, in _open_shards
441
+ [rank0]: envs.append(_open(sub))
442
+ [rank0]: ^^^^^^^^^^
443
+ [rank0]: File "/dev/shm/mwm/repo/minwm/data/datasets/lmdb.py", line 27, in _open
444
+ [rank0]: return lmdb.open(path, readonly=True, lock=False, readahead=False, meminit=False)
445
+ [rank0]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
446
+ [rank0]: lmdb.Error: The environment '/dev/shm/mwm/data/quality/shard_q001b' is already open in this process.
447
+ [rank1]: Traceback (most recent call last):
448
+ [rank1]: File "/dev/shm/mwm/minWM-main/tools/train_mwm.py", line 74, in <module>
449
+ [rank1]: main()
450
+ [rank1]: File "/dev/shm/mwm/minWM-main/tools/train_mwm.py", line 69, in main
451
+ [rank1]: trainer = build_trainer(cfg)
452
+ [rank1]: ^^^^^^^^^^^^^^^^^^
453
+ [rank1]: File "/dev/shm/mwm/minWM-main/tools/train_mwm.py", line 50, in build_trainer
454
+ [rank1]: return trainer_cls(MWMConfig.from_dict(cfg))
455
+ [rank1]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
456
+ [rank1]: File "/dev/shm/mwm/repo/minwm/engine/trainer.py", line 167, in __init__
457
+ [rank1]: self.dataloader = self.build_dataloader(self.cfg.data)
458
+ [rank1]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
459
+ [rank1]: File "/dev/shm/mwm/repo/minwm/engine/trainer.py", line 386, in build_dataloader
460
+ [rank1]: return _build_dataloader_cfg(data)
461
+ [rank1]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^
462
+ [rank1]: File "/dev/shm/mwm/repo/minwm/data/build.py", line 68, in build_dataloader
463
+ [rank1]: dataset = build(data.dataset)
464
+ [rank1]: ^^^^^^^^^^^^^^^^^^^
465
+ [rank1]: File "/dev/shm/mwm/repo/minwm/config/lazy.py", line 126, in build
466
+ [rank1]: return cls(**kwargs)
467
+ [rank1]: ^^^^^^^^^^^^^
468
+ [rank1]: File "/dev/shm/mwm/repo/minwm/data/datasets/lmdb.py", line 204, in __init__
469
+ [rank1]: self._sds = _open_shards(data_path)
470
+ [rank1]: ^^^^^^^^^^^^^^^^^^^^^^^
471
+ [rank1]: File "/dev/shm/mwm/repo/minwm/data/datasets/lmdb.py", line 63, in _open_shards
472
+ [rank1]: envs.append(_open(sub))
473
+ [rank1]: ^^^^^^^^^^
474
+ [rank1]: File "/dev/shm/mwm/repo/minwm/data/datasets/lmdb.py", line 27, in _open
475
+ [rank1]: return lmdb.open(path, readonly=True, lock=False, readahead=False, meminit=False)
476
+ [rank1]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
477
+ [rank1]: lmdb.Error: The environment '/dev/shm/mwm/data/quality/shard_q001b' is already open in this process.
478
+ W1031 01:25:05.906000 3353220 torch/distributed/elastic/multiprocessing/api.py:908] Sending process 3353230 closing signal SIGTERM
479
+ W1031 01:25:05.907000 3353220 torch/distributed/elastic/multiprocessing/api.py:908] Sending process 3353231 closing signal SIGTERM
480
+ W1031 01:25:05.908000 3353220 torch/distributed/elastic/multiprocessing/api.py:908] Sending process 3353232 closing signal SIGTERM
481
+ W1031 01:25:05.909000 3353220 torch/distributed/elastic/multiprocessing/api.py:908] Sending process 3353233 closing signal SIGTERM
482
+ W1031 01:25:05.910000 3353220 torch/distributed/elastic/multiprocessing/api.py:908] Sending process 3353234 closing signal SIGTERM
483
+ E1031 01:25:07.140000 3353220 torch/distributed/elastic/multiprocessing/api.py:882] failed (exitcode: 1) local_rank: 5 (pid: 3353235) of binary: /dev/shm/mwm/venv/bin/python
484
+ Traceback (most recent call last):
485
+ File "/dev/shm/mwm/venv/bin/torchrun", line 10, in <module>
486
+ sys.exit(main())
487
+ ^^^^^^
488
+ File "/dev/shm/mwm/venv/lib/python3.12/site-packages/torch/distributed/elastic/multiprocessing/errors/__init__.py", line 357, in wrapper
489
+ return f(*args, **kwargs)
490
+ ^^^^^^^^^^^^^^^^^^
491
+ File "/dev/shm/mwm/venv/lib/python3.12/site-packages/torch/distributed/run.py", line 936, in main
492
+ run(args)
493
+ File "/dev/shm/mwm/venv/lib/python3.12/site-packages/torch/distributed/run.py", line 927, in run
494
+ elastic_launch(
495
+ File "/dev/shm/mwm/venv/lib/python3.12/site-packages/torch/distributed/launcher/api.py", line 156, in __call__
496
+ return launch_agent(self._config, self._entrypoint, list(args))
497
+ ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
498
+ File "/dev/shm/mwm/venv/lib/python3.12/site-packages/torch/distributed/launcher/api.py", line 293, in launch_agent
499
+ raise ChildFailedError(
500
+ torch.distributed.elastic.multiprocessing.errors.ChildFailedError:
501
+ ============================================================
502
+ tools/train_mwm.py FAILED
503
+ ------------------------------------------------------------
504
+ Failures:
505
+ <NO_OTHER_FAILURES>
506
+ ------------------------------------------------------------
507
+ Root Cause (first observed failure):
508
+ [0]:
509
+ time : 2026-10-31_01:25:05
510
+ host : fsxznwc8ls-simpli-server-864dcff478-4tj44
511
+ rank : 5 (local_rank: 5)
512
+ exitcode : 1 (pid: 3353235)
513
+ error_file: <N/A>
514
+ traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html
515
+ ============================================================
516
+ [W1031 01:25:07.004007112 AllocatorConfig.cpp:28] Warning: PYTORCH_CUDA_ALLOC_CONF is deprecated, use PYTORCH_ALLOC_CONF instead (function operator())
517
+ rc=1 MWM_TRAIN_END 01:25:07 tf_nomas4