AbhishekGhosh1998 commited on
Commit
536163a
·
verified ·
1 Parent(s): 133db3c

granularity results tierb-fixed-moonep-full/moonep/tierb @ 2026-08-30T01:38:56Z from sp-aiinfra-interactive-aghosh-7jtjl-8xb200-moe-cg-0

Browse files
Files changed (25) hide show
  1. tierb-fixed-moonep-full/moonep/tierb/manifest.txt +2 -2
  2. tierb-fixed-moonep-full/moonep/tierb/pip_freeze.txt +1 -1
  3. tierb-fixed-moonep-full/moonep/tierb/provenance.txt +63 -0
  4. tierb-fixed-moonep-full/moonep/tierb/tierb_gbs8/DSV2_LITE_L24__moonep__matched__seq8192__gbs8__capnone__async/rep1/mem_samples.csv +3 -179
  5. tierb-fixed-moonep-full/moonep/tierb/tierb_gbs8/DSV2_LITE_L24__moonep__matched__seq8192__gbs8__capnone__async/rep1/run.env +1 -1
  6. tierb-fixed-moonep-full/moonep/tierb/tierb_gbs8/DSV2_LITE_L24__moonep__matched__seq8192__gbs8__capnone__async/rep1/train.log +0 -0
  7. tierb-fixed-moonep-full/moonep/tierb/tierb_gbs8/QWEN3_235B__moonep__matched__seq8192__gbs8__capnone__async/rep1/mem_samples.csv +137 -137
  8. tierb-fixed-moonep-full/moonep/tierb/tierb_gbs8/QWEN3_235B__moonep__matched__seq8192__gbs8__capnone__async/rep1/run.env +1 -1
  9. tierb-fixed-moonep-full/moonep/tierb/tierb_gbs8/QWEN3_235B__moonep__matched__seq8192__gbs8__capnone__async/rep1/train.log +198 -199
  10. tierb-fixed-moonep-full/moonep/tierb/tierb_gbs8/QWEN3_235B__moonep__matched__seq8192__gbs8__capnone__async/rep2/mem_samples.csv +119 -119
  11. tierb-fixed-moonep-full/moonep/tierb/tierb_gbs8/QWEN3_235B__moonep__matched__seq8192__gbs8__capnone__async/rep2/run.env +1 -1
  12. tierb-fixed-moonep-full/moonep/tierb/tierb_gbs8/QWEN3_235B__moonep__matched__seq8192__gbs8__capnone__async/rep2/train.log +213 -215
  13. tierb-fixed-moonep-full/moonep/tierb/tierb_gbs8/QWEN3_235B__moonep__matched__seq8192__gbs8__capnone__async/rep3/mem_samples.csv +128 -136
  14. tierb-fixed-moonep-full/moonep/tierb/tierb_gbs8/QWEN3_235B__moonep__matched__seq8192__gbs8__capnone__async/rep3/run.env +1 -1
  15. tierb-fixed-moonep-full/moonep/tierb/tierb_gbs8/QWEN3_235B__moonep__matched__seq8192__gbs8__capnone__async/rep3/train.log +219 -221
  16. tierb-fixed-moonep-full/moonep/tierb/tierb_gbs8/QWEN3_30B__moonep__matched__seq4096__gbs8__capnone__async/rep1/mem_samples.csv +129 -137
  17. tierb-fixed-moonep-full/moonep/tierb/tierb_gbs8/QWEN3_30B__moonep__matched__seq4096__gbs8__capnone__async/rep1/run.env +1 -1
  18. tierb-fixed-moonep-full/moonep/tierb/tierb_gbs8/QWEN3_30B__moonep__matched__seq4096__gbs8__capnone__async/rep1/train.log +0 -0
  19. tierb-fixed-moonep-full/moonep/tierb/tierb_gbs8/QWEN3_30B__moonep__matched__seq4096__gbs8__capnone__async/rep2/mem_samples.csv +133 -133
  20. tierb-fixed-moonep-full/moonep/tierb/tierb_gbs8/QWEN3_30B__moonep__matched__seq4096__gbs8__capnone__async/rep2/run.env +1 -1
  21. tierb-fixed-moonep-full/moonep/tierb/tierb_gbs8/QWEN3_30B__moonep__matched__seq4096__gbs8__capnone__async/rep2/train.log +0 -0
  22. tierb-fixed-moonep-full/moonep/tierb/tierb_gbs8/QWEN3_30B__moonep__matched__seq4096__gbs8__capnone__async/rep3/mem_samples.csv +118 -118
  23. tierb-fixed-moonep-full/moonep/tierb/tierb_gbs8/QWEN3_30B__moonep__matched__seq4096__gbs8__capnone__async/rep3/run.env +1 -1
  24. tierb-fixed-moonep-full/moonep/tierb/tierb_gbs8/QWEN3_30B__moonep__matched__seq4096__gbs8__capnone__async/rep3/train.log +0 -0
  25. tierb-fixed-moonep-full/moonep/tierb/tierb_gbs8/results.tsv +6 -12
tierb-fixed-moonep-full/moonep/tierb/manifest.txt CHANGED
@@ -1,4 +1,4 @@
1
- timestamp: 2026-08-29T23:44:54+00:00
2
  host: sp-aiinfra-interactive-aghosh-7jtjl-8xb200-moe-cg-0
3
  gpu: NVIDIA B200 x8
4
  driver: 580.95.05
@@ -16,4 +16,4 @@ nsys: 1
16
  megatron_repo: /workspace/Megatron-MoE-Echo
17
  megatron_branch_required: moonep_backend_async_grad_reduce
18
  megatron_branch_actual: moonep_backend_async_grad_reduce
19
- megatron_commit: 7bc235bf
 
1
+ timestamp: 2026-08-30T01:31:08+00:00
2
  host: sp-aiinfra-interactive-aghosh-7jtjl-8xb200-moe-cg-0
3
  gpu: NVIDIA B200 x8
4
  driver: 580.95.05
 
16
  megatron_repo: /workspace/Megatron-MoE-Echo
17
  megatron_branch_required: moonep_backend_async_grad_reduce
18
  megatron_branch_actual: moonep_backend_async_grad_reduce
19
+ megatron_commit: 2e68c8ad
tierb-fixed-moonep-full/moonep/tierb/pip_freeze.txt CHANGED
@@ -134,7 +134,7 @@ matplotlib==3.10.3
134
  matplotlib-inline==0.1.7
135
  mdit-py-plugins==0.4.2
136
  mdurl==0.1.2
137
- -e git+ssh://git@github.com/sriram-17-17/my-Megatron-MoE-Echo.git@7bc235bfc2a6d28129bf8df798ac4e667113af85#egg=megatron_core
138
  mistune==3.1.3
139
  mkl==2021.1.1
140
  mkl-devel==2021.1.1
 
134
  matplotlib-inline==0.1.7
135
  mdit-py-plugins==0.4.2
136
  mdurl==0.1.2
137
+ -e git+ssh://git@github.com/sriram-17-17/my-Megatron-MoE-Echo.git@2e68c8ad6bfb5bac5d6eb962fa7d4fbf0a5fb095#egg=megatron_core
138
  mistune==3.1.3
139
  mkl==2021.1.1
140
  mkl-devel==2021.1.1
tierb-fixed-moonep-full/moonep/tierb/provenance.txt CHANGED
@@ -27,3 +27,66 @@ command : run_ladder.sh --phase tierb --gbs 8 --out /workspace/tierb-fixed
27
  7, 120 MHz, 1965 MHz, 1000.00 W, 28
28
 
29
  NVLink topology:
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
27
  7, 120 MHz, 1965 MHz, 1000.00 W, 28
28
 
29
  NVLink topology:
30
+ GPU0 GPU1 GPU2 GPU3 GPU4 GPU5 GPU6 GPU7 NIC0 NIC1 NIC2 NIC3 NIC4 NIC5 NIC6 NIC7 NIC8 CPU Affinity NUMA Affinity GPU NUMA ID
31
+ GPU0 X NV18 NV18 NV18 NV18 NV18 NV18 NV18 PIX NODE SYS SYS SYS SYS SYS SYS NODE 0-27 0 N/A
32
+ GPU1 NV18 X NV18 NV18 NV18 NV18 NV18 NV18 NODE PIX SYS SYS SYS SYS SYS SYS NODE 0-27 0 N/A
33
+ GPU2 NV18 NV18 X NV18 NV18 NV18 NV18 NV18 SYS SYS PIX NODE SYS SYS SYS SYS SYS 28-55 1 N/A
34
+ GPU3 NV18 NV18 NV18 X NV18 NV18 NV18 NV18 SYS SYS NODE PIX SYS SYS SYS SYS SYS 28-55 1 N/A
35
+ GPU4 NV18 NV18 NV18 NV18 X NV18 NV18 NV18 SYS SYS SYS SYS PIX NODE SYS SYS SYS 56-83 2 N/A
36
+ GPU5 NV18 NV18 NV18 NV18 NV18 X NV18 NV18 SYS SYS SYS SYS NODE PIX SYS SYS SYS 56-83 2 N/A
37
+ GPU6 NV18 NV18 NV18 NV18 NV18 NV18 X NV18 SYS SYS SYS SYS SYS SYS PIX NODE SYS 84-111 3 N/A
38
+ GPU7 NV18 NV18 NV18 NV18 NV18 NV18 NV18 X SYS SYS SYS SYS SYS SYS NODE PIX SYS 84-111 3 N/A
39
+ NIC0 PIX NODE SYS SYS SYS SYS SYS SYS X NODE SYS SYS SYS SYS SYS SYS NODE
40
+ NIC1 NODE PIX SYS SYS SYS SYS SYS SYS NODE X SYS SYS SYS SYS SYS SYS NODE
41
+ NIC2 SYS SYS PIX NODE SYS SYS SYS SYS SYS SYS X NODE SYS SYS SYS SYS SYS
42
+ NIC3 SYS SYS NODE PIX SYS SYS SYS SYS SYS SYS NODE X SYS SYS SYS SYS SYS
43
+ NIC4 SYS SYS SYS SYS PIX NODE SYS SYS SYS SYS SYS SYS X NODE SYS SYS SYS
44
+ unavailable
45
+
46
+ ---- toolchain -----------------------------------------------
47
+ nvcc : release 12.9, V12.9.86
48
+ gcc : gcc (Ubuntu 13.3.0-6ubuntu2~24.04) 13.3.0
49
+ python : 3.12.3
50
+ torch : 2.8.0a0+5228986c39.nv25.06 cuda=12.9 nccl=2.27.3
51
+ torch arch : ['sm_75', 'sm_80', 'sm_86', 'sm_90', 'sm_100', 'sm_120', 'compute_120']
52
+ transformer_engine: 2.10.0.dev0+10f6128d [/usr/local/lib/python3.12/dist-packages/transformer_engine/__init__.py]
53
+ deep_ep : 1.2.1+34152ae [/workspace/DeepEP/deep_ep/__init__.py]
54
+ ultra_ep : 1.0.0 [/workspace/UltraEP/ultra_ep/__init__.py]
55
+ moonep : 0.0.1 [/workspace/MoonEP/moonep/__init__.py]
56
+ megatron : 0.14.0rc5 [None]
57
+ flash_attn : 2.7.4.post1 [/usr/local/lib/python3.12/dist-packages/flash_attn/__init__.py]
58
+ numpy : 1.26.4 [/usr/local/lib/python3.12/dist-packages/numpy/__init__.py]
59
+ huggingface_hub: 1.29.0 [/usr/local/lib/python3.12/dist-packages/huggingface_hub/__init__.py]
60
+
61
+ ---- repositories --------------------------------------------
62
+ Megatron-MoE-Echo moonep_backend_async_grad_reduce 2e68c8ad6bfb
63
+ Megatron-UltraEP ultraep-bracing-regions-through-kernels 03d57ef7d476
64
+ UltraEP main 94cab099b44f
65
+ DeepEP HEAD 34152ae28f80 (+3 untracked: build output)
66
+ MoonEP megatron-ddp-symmetric-params d8bc6667fad7
67
+ blackwell_run_scripts granularity-moonep eec9aee807fa
68
+
69
+ ---- environment ---------------------------------------------
70
+ CUDA_ARCH_LIST=7.5 8.0 8.6 9.0 10.0 12.0
71
+ CUDA_DRIVER_VERSION=575.57.08
72
+ CUDA_HOME=/usr/local/cuda
73
+ CUDA_MODULE_LOADING=LAZY
74
+ CUDA_VERSION=12.9.1.010
75
+ HF_REMOTE_ROOT=tierb-fixed-moonep-full
76
+ MOONEP_ASSERT_NO_PENDING_REDUCE=1
77
+ MOONEP_ASYNC_GRAD_REDUCE=1
78
+ NCCL_VERSION=2.27.3
79
+ NVSHMEM_VERSION=3.2.5
80
+ PYTORCH_BUILD_NUMBER=0
81
+ PYTORCH_BUILD_VERSION=2.8.0a0+5228986
82
+ PYTORCH_HOME=/opt/pytorch/pytorch
83
+ PYTORCH_VERSION=2.8.0a0+5228986
84
+ TORCH_ALLOW_TF32_CUBLAS_OVERRIDE=1
85
+ TORCH_CUDA_ARCH_LIST=7.5 8.0 8.6 9.0 10.0 12.0+PTX
86
+ TORCH_NCCL_USE_COMM_NONBLOCKING=0
87
+
88
+ ---- shared library resolution -------------------------------
89
+ libnvshmem_host.so: not in ldconfig
90
+ libnccl.so: /lib/x86_64-linux-gnu/libnccl.so.2
91
+ lrwxrwxrwx 1 root root 20 Aug 29 16:41 /usr/local/lib/python3.12/dist-packages/nvidia/nvshmem/lib/libnvshmem_host.so -> libnvshmem_host.so.3
92
+ -rw-r--r-- 1 root root 179045744 Aug 29 16:37 /usr/local/lib/python3.12/dist-packages/nvidia/nvshmem/lib/libnvshmem_host.so.3
tierb-fixed-moonep-full/moonep/tierb/tierb_gbs8/DSV2_LITE_L24__moonep__matched__seq8192__gbs8__capnone__async/rep1/mem_samples.csv CHANGED
@@ -7,7 +7,7 @@
7
  6, 0
8
  7, 0
9
  0, 4
10
- 1, 1
11
  2, 1
12
  3, 1
13
  4, 1
@@ -20,13 +20,13 @@
20
  3, 4
21
  4, 4
22
  5, 4
23
- 6, 1
24
  7, 1
25
  0, 1
26
  1, 1
27
  2, 1
28
  3, 1
29
- 4, 3
30
  5, 4
31
  6, 4
32
  7, 4
@@ -54,179 +54,3 @@
54
  5, 1
55
  6, 1
56
  7, 1
57
- 0, 4
58
- 1, 4
59
- 2, 4
60
- 3, 4
61
- 4, 4
62
- 5, 1
63
- 6, 1
64
- 7, 1
65
- 0, 4
66
- 1, 4
67
- 2, 4
68
- 3, 4
69
- 4, 4
70
- 5, 4
71
- 6, 4
72
- 7, 4
73
- 0, 4
74
- 1, 4
75
- 2, 4
76
- 3, 4
77
- 4, 4
78
- 5, 4
79
- 6, 35
80
- 7, 4
81
- 0, 821
82
- 1, 821
83
- 2, 819
84
- 3, 821
85
- 4, 819
86
- 5, 819
87
- 6, 821
88
- 7, 821
89
- 0, 1537
90
- 1, 1537
91
- 2, 1537
92
- 3, 1537
93
- 4, 1537
94
- 5, 1537
95
- 6, 1537
96
- 7, 1537
97
- 0, 2471
98
- 1, 2471
99
- 2, 2471
100
- 3, 2471
101
- 4, 2471
102
- 5, 2471
103
- 6, 2471
104
- 7, 2471
105
- 0, 26195
106
- 1, 26193
107
- 2, 26199
108
- 3, 26187
109
- 4, 26235
110
- 5, 26211
111
- 6, 26207
112
- 7, 26203
113
- 0, 30319
114
- 1, 30407
115
- 2, 30253
116
- 3, 30253
117
- 4, 30253
118
- 5, 30253
119
- 6, 30385
120
- 7, 30407
121
- 0, 36177
122
- 1, 36263
123
- 2, 36113
124
- 3, 36111
125
- 4, 36111
126
- 5, 36109
127
- 6, 36243
128
- 7, 36263
129
- 0, 40669
130
- 1, 40739
131
- 2, 40611
132
- 3, 40613
133
- 4, 40591
134
- 5, 40641
135
- 6, 40727
136
- 7, 40805
137
- 0, 41133
138
- 1, 41217
139
- 2, 41067
140
- 3, 41065
141
- 4, 41065
142
- 5, 41063
143
- 6, 41199
144
- 7, 41217
145
- 0, 41137
146
- 1, 41221
147
- 2, 41071
148
- 3, 41071
149
- 4, 41071
150
- 5, 41067
151
- 6, 41203
152
- 7, 41221
153
- 0, 108757
154
- 1, 108839
155
- 2, 108697
156
- 3, 108689
157
- 4, 108699
158
- 5, 108695
159
- 6, 108829
160
- 7, 108843
161
- 0, 109391
162
- 1, 109475
163
- 2, 109325
164
- 3, 109325
165
- 4, 109325
166
- 5, 109321
167
- 6, 109457
168
- 7, 109475
169
- 0, 109665
170
- 1, 109763
171
- 2, 109639
172
- 3, 109613
173
- 4, 109633
174
- 5, 109651
175
- 6, 109833
176
- 7, 109763
177
- 0, 110429
178
- 1, 110513
179
- 2, 110363
180
- 3, 110363
181
- 4, 110363
182
- 5, 110359
183
- 6, 110495
184
- 7, 110513
185
- 0, 111079
186
- 1, 111169
187
- 2, 111003
188
- 3, 110985
189
- 4, 111047
190
- 5, 111029
191
- 6, 111149
192
- 7, 111201
193
- 0, 119291
194
- 1, 119363
195
- 2, 127427
196
- 3, 121263
197
- 4, 127429
198
- 5, 117155
199
- 6, 119357
200
- 7, 119365
201
- 0, 133237
202
- 1, 133311
203
- 2, 133157
204
- 3, 133155
205
- 4, 133155
206
- 5, 133157
207
- 6, 133303
208
- 7, 133311
209
- 0, 133237
210
- 1, 133311
211
- 2, 133157
212
- 3, 133155
213
- 4, 133155
214
- 5, 133157
215
- 6, 133303
216
- 7, 133311
217
- 0, 133237
218
- 1, 133311
219
- 2, 133157
220
- 3, 133155
221
- 4, 133155
222
- 5, 133157
223
- 6, 133303
224
- 7, 133311
225
- 0, 9575
226
- 1, 9578
227
- 2, 9578
228
- 3, 9578
229
- 4, 9898
230
- 5, 133155
231
- 6, 11178
232
- 7, 9578
 
7
  6, 0
8
  7, 0
9
  0, 4
10
+ 1, 4
11
  2, 1
12
  3, 1
13
  4, 1
 
20
  3, 4
21
  4, 4
22
  5, 4
23
+ 6, 4
24
  7, 1
25
  0, 1
26
  1, 1
27
  2, 1
28
  3, 1
29
+ 4, 4
30
  5, 4
31
  6, 4
32
  7, 4
 
54
  5, 1
55
  6, 1
56
  7, 1
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
tierb-fixed-moonep-full/moonep/tierb/tierb_gbs8/DSV2_LITE_L24__moonep__matched__seq8192__gbs8__capnone__async/rep1/run.env CHANGED
@@ -3,7 +3,7 @@ config=DSV2_LITE_L24
3
  family=tierB
4
  repo=/workspace/Megatron-MoE-Echo
5
  branch=moonep_backend_async_grad_reduce
6
- commit=7bc235bf
7
  moonep_source_dir=/workspace/MoonEP
8
  moonep_source_branch=megatron-ddp-symmetric-params
9
  moonep_source_commit=d8bc666
 
3
  family=tierB
4
  repo=/workspace/Megatron-MoE-Echo
5
  branch=moonep_backend_async_grad_reduce
6
+ commit=2e68c8ad
7
  moonep_source_dir=/workspace/MoonEP
8
  moonep_source_branch=megatron-ddp-symmetric-params
9
  moonep_source_commit=d8bc666
tierb-fixed-moonep-full/moonep/tierb/tierb_gbs8/DSV2_LITE_L24__moonep__matched__seq8192__gbs8__capnone__async/rep1/train.log CHANGED
The diff for this file is too large to render. See raw diff
 
tierb-fixed-moonep-full/moonep/tierb/tierb_gbs8/QWEN3_235B__moonep__matched__seq8192__gbs8__capnone__async/rep1/mem_samples.csv CHANGED
@@ -26,7 +26,7 @@
26
  1, 1
27
  2, 1
28
  3, 1
29
- 4, 3
30
  5, 4
31
  6, 4
32
  7, 4
@@ -62,86 +62,86 @@
62
  5, 4
63
  6, 4
64
  7, 4
65
- 0, 4
66
- 1, 4
67
- 2, 4
68
- 3, 4
69
- 4, 4
70
- 5, 4
71
- 6, 4
72
- 7, 4
73
- 0, 821
74
- 1, 821
75
- 2, 821
76
- 3, 819
77
- 4, 821
78
- 5, 821
79
- 6, 821
80
- 7, 821
81
- 0, 1537
82
- 1, 1537
83
- 2, 1537
84
- 3, 1537
85
- 4, 1537
86
- 5, 1537
87
- 6, 1537
88
- 7, 1537
89
- 0, 2551
90
- 1, 2551
91
- 2, 2551
92
- 3, 2551
93
- 4, 2551
94
- 5, 2551
95
- 6, 2551
96
- 7, 2551
97
- 0, 46603
98
- 1, 46619
99
- 2, 46599
100
- 3, 46619
101
- 4, 46635
102
- 5, 46639
103
- 6, 46631
104
- 7, 46637
105
- 0, 56813
106
- 1, 56831
107
- 2, 56771
108
- 3, 56791
109
- 4, 56519
110
- 5, 56847
111
- 6, 56779
112
- 7, 56849
113
- 0, 59169
114
- 1, 59167
115
- 2, 59167
116
- 3, 59161
117
- 4, 58875
118
- 5, 59167
119
- 6, 59159
120
- 7, 59181
121
- 0, 59733
122
  1, 59731
123
- 2, 59731
124
- 3, 59733
125
  4, 59443
126
  5, 59731
127
  6, 59733
128
- 7, 59731
129
  0, 59733
130
- 1, 59731
131
  2, 59733
132
  3, 59733
133
  4, 59445
134
  5, 59733
135
  6, 59733
136
  7, 59733
137
- 0, 86921
138
- 1, 86961
139
- 2, 86961
140
- 3, 86961
141
- 4, 86921
142
- 5, 86961
143
- 6, 86961
144
- 7, 86961
145
  0, 87633
146
  1, 87673
147
  2, 87673
@@ -150,75 +150,75 @@
150
  5, 87673
151
  6, 87673
152
  7, 87673
153
- 0, 87833
154
- 1, 87869
155
- 2, 87873
156
- 3, 87873
157
- 4, 87869
158
- 5, 87939
159
- 6, 87899
160
- 7, 87937
161
- 0, 94181
162
- 1, 95013
163
- 2, 95133
164
- 3, 95229
165
- 4, 94733
166
- 5, 95325
167
- 6, 94797
168
- 7, 94821
169
- 0, 97481
170
- 1, 97527
171
- 2, 97499
172
- 3, 97465
173
- 4, 97433
174
- 5, 97525
175
- 6, 97527
176
- 7, 97523
177
- 0, 120267
178
- 1, 120303
179
- 2, 120303
180
- 3, 120303
181
- 4, 120303
182
- 5, 120303
183
- 6, 120303
184
- 7, 120303
185
- 0, 120267
186
- 1, 120303
187
- 2, 120303
188
- 3, 120303
189
- 4, 120303
190
- 5, 120303
191
- 6, 120303
192
- 7, 120303
193
- 0, 119621
194
- 1, 119657
195
- 2, 119657
196
- 3, 119657
197
- 4, 119657
198
- 5, 119657
199
- 6, 119657
200
- 7, 119657
201
- 0, 117395
202
- 1, 117431
203
- 2, 117431
204
- 3, 117431
205
- 4, 117431
206
- 5, 117431
207
- 6, 117431
208
- 7, 117431
209
- 0, 92863
210
- 1, 92899
211
- 2, 92897
212
- 3, 115751
213
- 4, 92899
214
- 5, 115751
215
- 6, 115751
216
- 7, 92899
217
  0, 1
218
- 1, 4
219
  2, 4
220
- 3, 114021
221
- 4, 4
222
  5, 4
223
  6, 4
224
  7, 4
 
26
  1, 1
27
  2, 1
28
  3, 1
29
+ 4, 1
30
  5, 4
31
  6, 4
32
  7, 4
 
62
  5, 4
63
  6, 4
64
  7, 4
65
+ 0, 777
66
+ 1, 751
67
+ 2, 725
68
+ 3, 767
69
+ 4, 725
70
+ 5, 731
71
+ 6, 753
72
+ 7, 725
73
+ 0, 945
74
+ 1, 943
75
+ 2, 945
76
+ 3, 939
77
+ 4, 949
78
+ 5, 963
79
+ 6, 961
80
+ 7, 957
81
+ 0, 1975
82
+ 1, 1975
83
+ 2, 1975
84
+ 3, 1975
85
+ 4, 1975
86
+ 5, 1975
87
+ 6, 1975
88
+ 7, 1975
89
+ 0, 32245
90
+ 1, 31861
91
+ 2, 35701
92
+ 3, 43765
93
+ 4, 35701
94
+ 5, 41461
95
+ 6, 30517
96
+ 7, 30517
97
+ 0, 48447
98
+ 1, 48449
99
+ 2, 48449
100
+ 3, 48449
101
+ 4, 48449
102
+ 5, 48449
103
+ 6, 48449
104
+ 7, 48449
105
+ 0, 56867
106
+ 1, 56821
107
+ 2, 56921
108
+ 3, 56933
109
+ 4, 56617
110
+ 5, 56883
111
+ 6, 56835
112
+ 7, 56885
113
+ 0, 58733
114
+ 1, 58755
115
+ 2, 58741
116
+ 3, 58729
117
+ 4, 58447
118
+ 5, 58749
119
+ 6, 58733
120
+ 7, 58759
121
+ 0, 59731
122
  1, 59731
123
+ 2, 59733
124
+ 3, 59731
125
  4, 59443
126
  5, 59731
127
  6, 59733
128
+ 7, 59733
129
  0, 59733
130
+ 1, 59733
131
  2, 59733
132
  3, 59733
133
  4, 59445
134
  5, 59733
135
  6, 59733
136
  7, 59733
137
+ 0, 85347
138
+ 1, 85387
139
+ 2, 85387
140
+ 3, 85387
141
+ 4, 85347
142
+ 5, 85387
143
+ 6, 85387
144
+ 7, 85387
145
  0, 87633
146
  1, 87673
147
  2, 87673
 
150
  5, 87673
151
  6, 87673
152
  7, 87673
153
+ 0, 87633
154
+ 1, 87673
155
+ 2, 87673
156
+ 3, 87673
157
+ 4, 87633
158
+ 5, 87673
159
+ 6, 87673
160
+ 7, 87673
161
+ 0, 88183
162
+ 1, 88229
163
+ 2, 88215
164
+ 3, 88229
165
+ 4, 88177
166
+ 5, 88267
167
+ 6, 88267
168
+ 7, 88259
169
+ 0, 97425
170
+ 1, 97499
171
+ 2, 97511
172
+ 3, 97499
173
+ 4, 97395
174
+ 5, 97541
175
+ 6, 97511
176
+ 7, 97625
177
+ 0, 120273
178
+ 1, 120309
179
+ 2, 120309
180
+ 3, 120309
181
+ 4, 120309
182
+ 5, 120309
183
+ 6, 120309
184
+ 7, 120309
185
+ 0, 120273
186
+ 1, 120309
187
+ 2, 120309
188
+ 3, 120309
189
+ 4, 120309
190
+ 5, 120309
191
+ 6, 120309
192
+ 7, 120309
193
+ 0, 119365
194
+ 1, 119663
195
+ 2, 119663
196
+ 3, 119663
197
+ 4, 119663
198
+ 5, 119401
199
+ 6, 119663
200
+ 7, 119663
201
+ 0, 117333
202
+ 1, 117437
203
+ 2, 117049
204
+ 3, 117501
205
+ 4, 117843
206
+ 5, 117369
207
+ 6, 117501
208
+ 7, 118083
209
+ 0, 4
210
+ 1, 115541
211
+ 2, 92905
212
+ 3, 109993
213
+ 4, 115801
214
+ 5, 92905
215
+ 6, 103465
216
+ 7, 115757
217
  0, 1
218
+ 1, 3
219
  2, 4
220
+ 3, 4
221
+ 4, 114027
222
  5, 4
223
  6, 4
224
  7, 4
tierb-fixed-moonep-full/moonep/tierb/tierb_gbs8/QWEN3_235B__moonep__matched__seq8192__gbs8__capnone__async/rep1/run.env CHANGED
@@ -3,7 +3,7 @@ config=QWEN3_235B
3
  family=tierB
4
  repo=/workspace/Megatron-MoE-Echo
5
  branch=moonep_backend_async_grad_reduce
6
- commit=7bc235bf
7
  moonep_source_dir=/workspace/MoonEP
8
  moonep_source_branch=megatron-ddp-symmetric-params
9
  moonep_source_commit=d8bc666
 
3
  family=tierB
4
  repo=/workspace/Megatron-MoE-Echo
5
  branch=moonep_backend_async_grad_reduce
6
+ commit=2e68c8ad
7
  moonep_source_dir=/workspace/MoonEP
8
  moonep_source_branch=megatron-ddp-symmetric-params
9
  moonep_source_commit=d8bc666
tierb-fixed-moonep-full/moonep/tierb/tierb_gbs8/QWEN3_235B__moonep__matched__seq8192__gbs8__capnone__async/rep1/train.log CHANGED
@@ -1,23 +1,16 @@
1
- [23:45:03] WARN: moonep_backend_async_grad_reduce lacks --moe-router-fusion; QWEN3_235B runs unfused (timing only, routing identical)
2
  ==============================================================
3
  QWEN3_235B / moonep (slots/rank=16 policy=matched)
4
  E=128 K=8 H'=1536 m=16
5
  L=12 seq=8192 gbs=8 (1 microbatches)
6
  cap_mode=none cap=none cg=1 recompute=none
7
  memory_dump=none snapshots=/workspace/tierb-fixed-moonep-full/moonep/tierb/tierb_gbs8/QWEN3_235B__moonep__matched__seq8192__gbs8__capnone__async/rep1/memory_snapshots/snapshot.pickle
8
- repo=/workspace/Megatron-MoE-Echo @ 7bc235bf
9
  ==============================================================
10
- W0829 23:45:04.937000 237461 torch/distributed/run.py:766]
11
- W0829 23:45:04.937000 237461 torch/distributed/run.py:766] *****************************************
12
- W0829 23:45:04.937000 237461 torch/distributed/run.py:766] Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed.
13
- W0829 23:45:04.937000 237461 torch/distributed/run.py:766] *****************************************
14
- /workspace/Megatron-MoE-Echo/megatron/training/arguments.py:831: UserWarning: Disabling sequence parallelism because tensor model parallelism is disabled
15
- warnings.warn("Disabling sequence parallelism because tensor model parallelism is disabled")
16
- WARNING: one_logger package is required to enable e2e metrics tracking. please go to https://confluence.nvidia.com/display/MLWFO/Package+Repositories for details to install it
17
- /workspace/Megatron-MoE-Echo/megatron/training/arguments.py:831: UserWarning: Disabling sequence parallelism because tensor model parallelism is disabled
18
- warnings.warn("Disabling sequence parallelism because tensor model parallelism is disabled")
19
- /workspace/Megatron-MoE-Echo/megatron/training/arguments.py:831: UserWarning: Disabling sequence parallelism because tensor model parallelism is disabled
20
- warnings.warn("Disabling sequence parallelism because tensor model parallelism is disabled")
21
  /workspace/Megatron-MoE-Echo/megatron/training/arguments.py:831: UserWarning: Disabling sequence parallelism because tensor model parallelism is disabled
22
  warnings.warn("Disabling sequence parallelism because tensor model parallelism is disabled")
23
  using world size: 8, data-parallel size: 8, context-parallel size: 1, hierarchical context-parallel sizes: None, tensor-model-parallel size: 1, pipeline-model-parallel size: 1
@@ -575,53 +568,60 @@ WARNING:megatron.core.rerun_state_machine:RerunStateMachine initialized in mode
575
  warnings.warn("Disabling sequence parallelism because tensor model parallelism is disabled")
576
  /workspace/Megatron-MoE-Echo/megatron/training/arguments.py:831: UserWarning: Disabling sequence parallelism because tensor model parallelism is disabled
577
  warnings.warn("Disabling sequence parallelism because tensor model parallelism is disabled")
 
578
  /workspace/Megatron-MoE-Echo/megatron/training/arguments.py:831: UserWarning: Disabling sequence parallelism because tensor model parallelism is disabled
579
  warnings.warn("Disabling sequence parallelism because tensor model parallelism is disabled")
580
- [Gloo] Rank [Gloo] Rank 0 is connected to 77 is connected to peer ranks. 7Expected number of connected peer ranks is : peer ranks. 7Expected number of connected peer ranks is : 7
581
-
 
 
 
 
 
 
582
  [Gloo] Rank 4 is connected to 7 peer ranks. Expected number of connected peer ranks is : 7
583
  [Gloo] Rank 1 is connected to 7 peer ranks. Expected number of connected peer ranks is : 7
584
- [Gloo] Rank 3 is connected to 7 peer ranks. Expected number of connected peer ranks is : 7
585
- [Gloo] Rank [Gloo] Rank 5 is connected to 72 peer ranks. is connected to Expected number of connected peer ranks is : 77 peer ranks. Expected number of connected peer ranks is :
586
- 7
587
  [Gloo] Rank 6 is connected to 7 peer ranks. Expected number of connected peer ranks is : 7
588
- [Gloo] Rank 3 is connected to [Gloo] Rank 7 peer ranks. 2Expected number of connected peer ranks is : is connected to 77
589
- peer ranks. Expected number of connected peer ranks is : 7
590
- [Gloo] Rank 5 is connected to 7 peer ranks. Expected number of connected peer ranks is : 7
591
  [Gloo] Rank 4 is connected to 7 peer ranks. Expected number of connected peer ranks is : 7
592
  [Gloo] Rank 0 is connected to 7 peer ranks. Expected number of connected peer ranks is : 7
 
593
  [Gloo] Rank 1 is connected to 7 peer ranks. Expected number of connected peer ranks is : 7
594
- [Gloo] Rank 6 is connected to 7 peer ranks. Expected number of connected peer ranks is : 7
595
- [Gloo] Rank 7 is connected to 7 peer ranks. Expected number of connected peer ranks is : 7
 
 
596
  [Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
597
  > initialized tensor model parallel with size 1
598
  > initialized pipeline model parallel with size 1
599
  > setting random seeds to 1234 ...
600
- [Gloo] Rank [Gloo] Rank 00 is connected to is connected to 00 peer ranks. peer ranks. Expected number of connected peer ranks is : Expected number of connected peer ranks is : 00
601
 
602
- [Gloo] Rank [Gloo] Rank 00 is connected to is connected to 00 peer ranks. peer ranks. Expected number of connected peer ranks is : Expected number of connected peer ranks is : 00
603
 
 
 
 
604
  [Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
605
- [Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
606
- [Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
607
- [rank2]:[W829 23:45:23.527949468 ProcessGroupNCCL.cpp:4789] [PG ID 0 PG GUID 0 Rank 2] using GPU 2 as device used by this process is currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect. You can specify device_id in init_process_group() to force use of a particular device.
608
- [rank5]:[W829 23:45:23.528057503 ProcessGroupNCCL.cpp:4789] [PG ID 0 PG GUID 0 Rank 5] using GPU 5 as device used by this process is currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect. You can specify device_id in init_process_group() to force use of a particular device.
609
- [rank1]:[W829 23:45:23.528113700 ProcessGroupNCCL.cpp:4789] [PG ID 0 PG GUID 0 Rank 1] using GPU 1 as device used by this process is currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect. You can specify device_id in init_process_group() to force use of a particular device.
610
- [rank4]:[W829 23:45:23.528140272 ProcessGroupNCCL.cpp:4789] [PG ID 0 PG GUID 0 Rank 4] using GPU 4 as device used by this process is currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect. You can specify device_id in init_process_group() to force use of a particular device.
611
- [rank6]:[W829 23:45:23.528167906 ProcessGroupNCCL.cpp:4789] [PG ID 0 PG GUID 0 Rank 6] using GPU 6 as device used by this process is currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect. You can specify device_id in init_process_group() to force use of a particular device.
612
- [rank3]:[W829 23:45:23.529332491 ProcessGroupNCCL.cpp:4789] [PG ID 0 PG GUID 0 Rank 3] using GPU 3 as device used by this process is currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect. You can specify device_id in init_process_group() to force use of a particular device.
613
- [rank7]:[W829 23:45:23.529419740 ProcessGroupNCCL.cpp:4789] [PG ID 0 PG GUID 0 Rank 7] using GPU 7 as device used by this process is currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect. You can specify device_id in init_process_group() to force use of a particular device.
614
  > compiling dataset index builder ...
615
  make: Entering directory '/workspace/Megatron-MoE-Echo/megatron/core/datasets'
616
  make: Nothing to be done for 'default'.
617
  make: Leaving directory '/workspace/Megatron-MoE-Echo/megatron/core/datasets'
618
- >>> done with dataset index builder. Compilation time: 0.033 seconds
619
  WARNING: constraints for invoking optimized fused softmax kernel are not met. We default back to unfused kernel invocations.
620
  > compiling and loading fused kernels ...
621
- [rank0]:[W829 23:45:23.696190153 ProcessGroupNCCL.cpp:4789] [PG ID 0 PG GUID 0 Rank 0] using GPU 0 as device used by this process is currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect. You can specify device_id in init_process_group() to force use of a particular device.
622
- >>> done with compiling and loading fused kernels. Compilation time: 3.725 seconds
623
- time to initialize megatron (seconds): 7.357
624
- [after megatron is initialized] datetime: 2026-08-29 23:45:29
625
  building GPT model ...
626
  /usr/local/lib/python3.12/dist-packages/transformer_engine/pytorch/cpu_offload.py:708: DeprecationWarning: Offloading weights is deprecated. Using offload_weights=True does not have any effect.
627
  warnings.warn(
@@ -646,100 +646,100 @@ Params for bucket 1 (206045184 elements, 206045184 padded size):
646
  module.output_layer.weight
647
  Params for bucket 2 (71840000 elements, 71840000 padded size):
648
  module.decoder.layers.11.pre_mlp_layernorm.weight
 
649
  module.decoder.layers.11.mlp.router.weight
650
  module.decoder.layers.11.self_attention.q_layernorm.weight
651
  module.decoder.layers.11.self_attention.linear_qkv.weight
652
  module.decoder.layers.11.self_attention.linear_qkv.layer_norm_weight
653
  module.decoder.final_layernorm.weight
654
- module.decoder.layers.11.self_attention.k_layernorm.weight
655
  module.decoder.layers.11.self_attention.linear_proj.weight
656
  Params for bucket 3 (71835904 elements, 71835904 padded size):
657
  module.decoder.layers.10.mlp.router.weight
658
- module.decoder.layers.10.pre_mlp_layernorm.weight
659
- module.decoder.layers.10.self_attention.k_layernorm.weight
660
- module.decoder.layers.10.self_attention.q_layernorm.weight
661
  module.decoder.layers.10.self_attention.linear_qkv.weight
 
 
662
  module.decoder.layers.10.self_attention.linear_qkv.layer_norm_weight
 
663
  module.decoder.layers.10.self_attention.linear_proj.weight
664
  Params for bucket 4 (71835904 elements, 71835904 padded size):
665
  module.decoder.layers.9.mlp.router.weight
666
- module.decoder.layers.9.pre_mlp_layernorm.weight
667
- module.decoder.layers.9.self_attention.linear_proj.weight
668
- module.decoder.layers.9.self_attention.q_layernorm.weight
669
  module.decoder.layers.9.self_attention.k_layernorm.weight
 
 
670
  module.decoder.layers.9.self_attention.linear_qkv.weight
671
  module.decoder.layers.9.self_attention.linear_qkv.layer_norm_weight
 
672
  Params for bucket 5 (71835904 elements, 71835904 padded size):
673
  module.decoder.layers.8.mlp.router.weight
674
- module.decoder.layers.8.pre_mlp_layernorm.weight
675
  module.decoder.layers.8.self_attention.k_layernorm.weight
 
676
  module.decoder.layers.8.self_attention.linear_qkv.weight
677
  module.decoder.layers.8.self_attention.q_layernorm.weight
678
  module.decoder.layers.8.self_attention.linear_qkv.layer_norm_weight
679
  module.decoder.layers.8.self_attention.linear_proj.weight
680
  Params for bucket 6 (71835904 elements, 71835904 padded size):
681
  module.decoder.layers.7.mlp.router.weight
682
- module.decoder.layers.7.pre_mlp_layernorm.weight
683
  module.decoder.layers.7.self_attention.q_layernorm.weight
684
  module.decoder.layers.7.self_attention.k_layernorm.weight
685
  module.decoder.layers.7.self_attention.linear_qkv.layer_norm_weight
686
  module.decoder.layers.7.self_attention.linear_qkv.weight
 
687
  module.decoder.layers.7.self_attention.linear_proj.weight
688
  Params for bucket 7 (71835904 elements, 71835904 padded size):
689
  module.decoder.layers.6.mlp.router.weight
690
  module.decoder.layers.6.pre_mlp_layernorm.weight
691
  module.decoder.layers.6.self_attention.k_layernorm.weight
692
- module.decoder.layers.6.self_attention.linear_qkv.weight
693
  module.decoder.layers.6.self_attention.q_layernorm.weight
 
694
  module.decoder.layers.6.self_attention.linear_qkv.layer_norm_weight
695
  module.decoder.layers.6.self_attention.linear_proj.weight
696
  Params for bucket 8 (71835904 elements, 71835904 padded size):
697
- module.decoder.layers.5.mlp.router.weight
698
  module.decoder.layers.5.pre_mlp_layernorm.weight
699
- module.decoder.layers.5.self_attention.linear_qkv.weight
700
  module.decoder.layers.5.self_attention.q_layernorm.weight
701
  module.decoder.layers.5.self_attention.linear_qkv.layer_norm_weight
 
 
702
  module.decoder.layers.5.self_attention.k_layernorm.weight
703
  module.decoder.layers.5.self_attention.linear_proj.weight
704
  Params for bucket 9 (71835904 elements, 71835904 padded size):
705
- module.decoder.layers.4.mlp.router.weight
706
  module.decoder.layers.4.pre_mlp_layernorm.weight
707
- module.decoder.layers.4.self_attention.k_layernorm.weight
708
  module.decoder.layers.4.self_attention.q_layernorm.weight
709
  module.decoder.layers.4.self_attention.linear_qkv.weight
710
  module.decoder.layers.4.self_attention.linear_qkv.layer_norm_weight
 
 
711
  module.decoder.layers.4.self_attention.linear_proj.weight
712
  Params for bucket 10 (71835904 elements, 71835904 padded size):
713
  module.decoder.layers.3.mlp.router.weight
714
- module.decoder.layers.3.pre_mlp_layernorm.weight
715
  module.decoder.layers.3.self_attention.linear_qkv.weight
716
  module.decoder.layers.3.self_attention.q_layernorm.weight
 
717
  module.decoder.layers.3.self_attention.k_layernorm.weight
718
  module.decoder.layers.3.self_attention.linear_qkv.layer_norm_weight
719
  module.decoder.layers.3.self_attention.linear_proj.weight
720
  Params for bucket 11 (71835904 elements, 71835904 padded size):
721
- module.decoder.layers.2.mlp.router.weight
722
- module.decoder.layers.2.self_attention.k_layernorm.weight
723
  module.decoder.layers.2.self_attention.q_layernorm.weight
 
 
724
  module.decoder.layers.2.self_attention.linear_qkv.weight
 
725
  module.decoder.layers.2.self_attention.linear_qkv.layer_norm_weight
726
  module.decoder.layers.2.self_attention.linear_proj.weight
727
- module.decoder.layers.2.pre_mlp_layernorm.weight
728
  Params for bucket 12 (71835904 elements, 71835904 padded size):
729
  module.decoder.layers.1.mlp.router.weight
 
730
  module.decoder.layers.1.self_attention.linear_qkv.weight
731
- module.decoder.layers.1.self_attention.q_layernorm.weight
732
  module.decoder.layers.1.pre_mlp_layernorm.weight
733
- module.decoder.layers.1.self_attention.k_layernorm.weight
734
  module.decoder.layers.1.self_attention.linear_qkv.layer_norm_weight
735
  module.decoder.layers.1.self_attention.linear_proj.weight
736
  Params for bucket 13 (71835904 elements, 71835904 padded size):
737
- module.decoder.layers.0.pre_mlp_layernorm.weight
738
- module.decoder.layers.0.mlp.router.weight
739
- module.decoder.layers.0.self_attention.q_layernorm.weight
740
  module.decoder.layers.0.self_attention.linear_qkv.weight
741
- module.decoder.layers.0.self_attention.k_layernorm.weight
742
  module.decoder.layers.0.self_attention.linear_qkv.layer_norm_weight
 
 
 
 
743
  module.decoder.layers.0.self_attention.linear_proj.weight
744
  Params for bucket 14 (206045184 elements, 206045184 padded size):
745
  module.embedding.word_embeddings.weight
@@ -749,8 +749,8 @@ Params for bucket 1 (44040192 elements, 44040192 padded size):
749
  module.decoder.layers.11.mlp.experts.linear_fc2.weight2
750
  module.decoder.layers.11.mlp.experts.linear_fc2.weight4
751
  module.decoder.layers.11.mlp.experts.linear_fc2.weight6
752
- module.decoder.layers.11.mlp.experts.linear_fc2.weight3
753
  module.decoder.layers.11.mlp.experts.linear_fc2.weight1
 
754
  module.decoder.layers.11.mlp.experts.linear_fc2.weight5
755
  Params for bucket 2 (44040192 elements, 44040192 padded size):
756
  module.decoder.layers.11.mlp.experts.linear_fc2.weight10
@@ -764,12 +764,12 @@ Params for bucket 3 (50331648 elements, 50331648 padded size):
764
  module.decoder.layers.11.mlp.experts.linear_fc2.weight14
765
  module.decoder.layers.11.mlp.experts.linear_fc1.weight1
766
  module.decoder.layers.11.mlp.experts.linear_fc1.weight0
767
- module.decoder.layers.11.mlp.experts.linear_fc2.weight15
768
  module.decoder.layers.11.mlp.experts.linear_fc1.weight2
 
769
  Params for bucket 4 (50331648 elements, 50331648 padded size):
770
  module.decoder.layers.11.mlp.experts.linear_fc1.weight3
771
- module.decoder.layers.11.mlp.experts.linear_fc1.weight6
772
  module.decoder.layers.11.mlp.experts.linear_fc1.weight4
 
773
  module.decoder.layers.11.mlp.experts.linear_fc1.weight5
774
  Params for bucket 5 (50331648 elements, 50331648 padded size):
775
  module.decoder.layers.11.mlp.experts.linear_fc1.weight7
@@ -783,63 +783,63 @@ Params for bucket 6 (50331648 elements, 50331648 padded size):
783
  module.decoder.layers.11.mlp.experts.linear_fc1.weight12
784
  Params for bucket 7 (44040192 elements, 44040192 padded size):
785
  module.decoder.layers.11.mlp.experts.linear_fc1.weight15
786
- module.decoder.layers.10.mlp.experts.linear_fc2.weight3
787
- module.decoder.layers.10.mlp.experts.linear_fc2.weight1
788
- module.decoder.layers.10.mlp.experts.linear_fc2.weight0
789
  module.decoder.layers.10.mlp.experts.linear_fc2.weight2
 
790
  module.decoder.layers.10.mlp.experts.linear_fc2.weight4
 
 
791
  Params for bucket 8 (44040192 elements, 44040192 padded size):
 
 
 
792
  module.decoder.layers.10.mlp.experts.linear_fc2.weight5
793
- module.decoder.layers.10.mlp.experts.linear_fc2.weight7
794
  module.decoder.layers.10.mlp.experts.linear_fc2.weight9
 
795
  module.decoder.layers.10.mlp.experts.linear_fc2.weight11
796
- module.decoder.layers.10.mlp.experts.linear_fc2.weight6
797
- module.decoder.layers.10.mlp.experts.linear_fc2.weight8
798
- module.decoder.layers.10.mlp.experts.linear_fc2.weight10
799
  Params for bucket 9 (50331648 elements, 50331648 padded size):
800
- module.decoder.layers.10.mlp.experts.linear_fc2.weight13
801
- module.decoder.layers.10.mlp.experts.linear_fc2.weight15
802
- module.decoder.layers.10.mlp.experts.linear_fc1.weight0
803
  module.decoder.layers.10.mlp.experts.linear_fc2.weight12
804
  module.decoder.layers.10.mlp.experts.linear_fc2.weight14
805
  module.decoder.layers.10.mlp.experts.linear_fc1.weight1
 
 
 
806
  Params for bucket 10 (50331648 elements, 50331648 padded size):
807
  module.decoder.layers.10.mlp.experts.linear_fc1.weight2
 
808
  module.decoder.layers.10.mlp.experts.linear_fc1.weight3
809
  module.decoder.layers.10.mlp.experts.linear_fc1.weight4
810
- module.decoder.layers.10.mlp.experts.linear_fc1.weight5
811
  Params for bucket 11 (50331648 elements, 50331648 padded size):
812
  module.decoder.layers.10.mlp.experts.linear_fc1.weight6
 
813
  module.decoder.layers.10.mlp.experts.linear_fc1.weight9
814
  module.decoder.layers.10.mlp.experts.linear_fc1.weight7
815
- module.decoder.layers.10.mlp.experts.linear_fc1.weight8
816
  Params for bucket 12 (50331648 elements, 50331648 padded size):
817
  module.decoder.layers.10.mlp.experts.linear_fc1.weight10
 
818
  module.decoder.layers.10.mlp.experts.linear_fc1.weight12
819
  module.decoder.layers.10.mlp.experts.linear_fc1.weight13
820
- module.decoder.layers.10.mlp.experts.linear_fc1.weight11
821
  Params for bucket 13 (44040192 elements, 44040192 padded size):
 
 
822
  module.decoder.layers.10.mlp.experts.linear_fc1.weight14
823
  module.decoder.layers.9.mlp.experts.linear_fc2.weight2
824
  module.decoder.layers.9.mlp.experts.linear_fc2.weight0
825
- module.decoder.layers.9.mlp.experts.linear_fc2.weight1
826
- module.decoder.layers.10.mlp.experts.linear_fc1.weight15
827
  Params for bucket 14 (44040192 elements, 44040192 padded size):
828
- module.decoder.layers.9.mlp.experts.linear_fc2.weight6
829
- module.decoder.layers.9.mlp.experts.linear_fc2.weight4
830
- module.decoder.layers.9.mlp.experts.linear_fc2.weight8
831
  module.decoder.layers.9.mlp.experts.linear_fc2.weight3
832
- module.decoder.layers.9.mlp.experts.linear_fc2.weight7
833
  module.decoder.layers.9.mlp.experts.linear_fc2.weight5
 
834
  module.decoder.layers.9.mlp.experts.linear_fc2.weight9
 
 
 
835
  Params for bucket 15 (50331648 elements, 50331648 padded size):
836
- module.decoder.layers.9.mlp.experts.linear_fc2.weight10
837
- module.decoder.layers.9.mlp.experts.linear_fc2.weight12
838
- module.decoder.layers.9.mlp.experts.linear_fc2.weight14
839
- module.decoder.layers.9.mlp.experts.linear_fc1.weight0
840
- module.decoder.layers.9.mlp.experts.linear_fc2.weight13
841
  module.decoder.layers.9.mlp.experts.linear_fc2.weight11
 
842
  module.decoder.layers.9.mlp.experts.linear_fc2.weight15
 
 
 
 
843
  Params for bucket 16 (50331648 elements, 50331648 padded size):
844
  module.decoder.layers.9.mlp.experts.linear_fc1.weight1
845
  module.decoder.layers.9.mlp.experts.linear_fc1.weight4
@@ -870,122 +870,122 @@ Params for bucket 20 (44040192 elements, 44040192 padded size):
870
  module.decoder.layers.8.mlp.experts.linear_fc2.weight7
871
  Params for bucket 21 (44040192 elements, 44040192 padded size):
872
  module.decoder.layers.8.mlp.experts.linear_fc2.weight8
873
- module.decoder.layers.8.mlp.experts.linear_fc2.weight10
874
  module.decoder.layers.8.mlp.experts.linear_fc2.weight12
 
875
  module.decoder.layers.8.mlp.experts.linear_fc2.weight14
876
  module.decoder.layers.8.mlp.experts.linear_fc2.weight9
877
  module.decoder.layers.8.mlp.experts.linear_fc2.weight11
878
  module.decoder.layers.8.mlp.experts.linear_fc2.weight13
879
  Params for bucket 22 (44040192 elements, 44040192 padded size):
880
  module.decoder.layers.8.mlp.experts.linear_fc2.weight15
881
- module.decoder.layers.8.mlp.experts.linear_fc1.weight1
882
  module.decoder.layers.8.mlp.experts.linear_fc1.weight0
883
  module.decoder.layers.8.mlp.experts.linear_fc1.weight2
 
884
  Params for bucket 23 (50331648 elements, 50331648 padded size):
885
  module.decoder.layers.8.mlp.experts.linear_fc1.weight3
 
886
  module.decoder.layers.8.mlp.experts.linear_fc1.weight4
887
  module.decoder.layers.8.mlp.experts.linear_fc1.weight5
888
- module.decoder.layers.8.mlp.experts.linear_fc1.weight6
889
  Params for bucket 24 (50331648 elements, 50331648 padded size):
890
  module.decoder.layers.8.mlp.experts.linear_fc1.weight7
 
891
  module.decoder.layers.8.mlp.experts.linear_fc1.weight10
892
  module.decoder.layers.8.mlp.experts.linear_fc1.weight8
893
- module.decoder.layers.8.mlp.experts.linear_fc1.weight9
894
  Params for bucket 25 (50331648 elements, 50331648 padded size):
895
  module.decoder.layers.8.mlp.experts.linear_fc1.weight11
 
896
  module.decoder.layers.8.mlp.experts.linear_fc1.weight13
897
  module.decoder.layers.8.mlp.experts.linear_fc1.weight14
898
- module.decoder.layers.8.mlp.experts.linear_fc1.weight12
899
  Params for bucket 26 (44040192 elements, 44040192 padded size):
900
- module.decoder.layers.8.mlp.experts.linear_fc1.weight15
901
- module.decoder.layers.7.mlp.experts.linear_fc2.weight3
902
- module.decoder.layers.7.mlp.experts.linear_fc2.weight1
903
- module.decoder.layers.7.mlp.experts.linear_fc2.weight0
904
  module.decoder.layers.7.mlp.experts.linear_fc2.weight2
 
905
  module.decoder.layers.7.mlp.experts.linear_fc2.weight4
 
 
 
906
  Params for bucket 27 (44040192 elements, 44040192 padded size):
 
 
 
907
  module.decoder.layers.7.mlp.experts.linear_fc2.weight5
908
  module.decoder.layers.7.mlp.experts.linear_fc2.weight9
909
  module.decoder.layers.7.mlp.experts.linear_fc2.weight7
910
  module.decoder.layers.7.mlp.experts.linear_fc2.weight11
911
- module.decoder.layers.7.mlp.experts.linear_fc2.weight6
912
- module.decoder.layers.7.mlp.experts.linear_fc2.weight8
913
- module.decoder.layers.7.mlp.experts.linear_fc2.weight10
914
  Params for bucket 28 (50331648 elements, 50331648 padded size):
915
- module.decoder.layers.7.mlp.experts.linear_fc2.weight15
916
- module.decoder.layers.7.mlp.experts.linear_fc1.weight0
917
- module.decoder.layers.7.mlp.experts.linear_fc2.weight13
918
  module.decoder.layers.7.mlp.experts.linear_fc2.weight12
919
  module.decoder.layers.7.mlp.experts.linear_fc2.weight14
920
  module.decoder.layers.7.mlp.experts.linear_fc1.weight1
 
 
 
921
  Params for bucket 29 (50331648 elements, 50331648 padded size):
922
  module.decoder.layers.7.mlp.experts.linear_fc1.weight2
 
923
  module.decoder.layers.7.mlp.experts.linear_fc1.weight3
924
  module.decoder.layers.7.mlp.experts.linear_fc1.weight4
925
- module.decoder.layers.7.mlp.experts.linear_fc1.weight5
926
  Params for bucket 30 (50331648 elements, 50331648 padded size):
927
  module.decoder.layers.7.mlp.experts.linear_fc1.weight6
 
928
  module.decoder.layers.7.mlp.experts.linear_fc1.weight7
929
  module.decoder.layers.7.mlp.experts.linear_fc1.weight9
930
- module.decoder.layers.7.mlp.experts.linear_fc1.weight8
931
  Params for bucket 31 (50331648 elements, 50331648 padded size):
932
  module.decoder.layers.7.mlp.experts.linear_fc1.weight10
933
- module.decoder.layers.7.mlp.experts.linear_fc1.weight13
934
  module.decoder.layers.7.mlp.experts.linear_fc1.weight12
935
  module.decoder.layers.7.mlp.experts.linear_fc1.weight11
 
936
  Params for bucket 32 (44040192 elements, 44040192 padded size):
937
- module.decoder.layers.7.mlp.experts.linear_fc1.weight14
938
- module.decoder.layers.6.mlp.experts.linear_fc2.weight2
939
  module.decoder.layers.6.mlp.experts.linear_fc2.weight0
 
 
940
  module.decoder.layers.6.mlp.experts.linear_fc2.weight1
941
- module.decoder.layers.7.mlp.experts.linear_fc1.weight15
942
  Params for bucket 33 (44040192 elements, 44040192 padded size):
943
  module.decoder.layers.6.mlp.experts.linear_fc2.weight6
944
  module.decoder.layers.6.mlp.experts.linear_fc2.weight4
945
  module.decoder.layers.6.mlp.experts.linear_fc2.weight8
946
  module.decoder.layers.6.mlp.experts.linear_fc2.weight3
947
- module.decoder.layers.6.mlp.experts.linear_fc2.weight5
948
  module.decoder.layers.6.mlp.experts.linear_fc2.weight7
 
949
  module.decoder.layers.6.mlp.experts.linear_fc2.weight9
950
  Params for bucket 34 (50331648 elements, 50331648 padded size):
951
  module.decoder.layers.6.mlp.experts.linear_fc2.weight10
952
  module.decoder.layers.6.mlp.experts.linear_fc2.weight14
953
  module.decoder.layers.6.mlp.experts.linear_fc2.weight12
954
- module.decoder.layers.6.mlp.experts.linear_fc2.weight11
955
  module.decoder.layers.6.mlp.experts.linear_fc2.weight13
 
956
  module.decoder.layers.6.mlp.experts.linear_fc2.weight15
957
  module.decoder.layers.6.mlp.experts.linear_fc1.weight0
958
  Params for bucket 35 (50331648 elements, 50331648 padded size):
959
  module.decoder.layers.6.mlp.experts.linear_fc1.weight1
960
- module.decoder.layers.6.mlp.experts.linear_fc1.weight2
961
  module.decoder.layers.6.mlp.experts.linear_fc1.weight3
962
  module.decoder.layers.6.mlp.experts.linear_fc1.weight4
 
963
  Params for bucket 36 (50331648 elements, 50331648 padded size):
964
  module.decoder.layers.6.mlp.experts.linear_fc1.weight5
965
- module.decoder.layers.6.mlp.experts.linear_fc1.weight8
966
  module.decoder.layers.6.mlp.experts.linear_fc1.weight6
967
  module.decoder.layers.6.mlp.experts.linear_fc1.weight7
 
968
  Params for bucket 37 (50331648 elements, 50331648 padded size):
969
  module.decoder.layers.6.mlp.experts.linear_fc1.weight9
970
- module.decoder.layers.6.mlp.experts.linear_fc1.weight11
971
  module.decoder.layers.6.mlp.experts.linear_fc1.weight12
972
  module.decoder.layers.6.mlp.experts.linear_fc1.weight10
 
973
  Params for bucket 38 (44040192 elements, 44040192 padded size):
974
  module.decoder.layers.6.mlp.experts.linear_fc1.weight13
 
975
  module.decoder.layers.6.mlp.experts.linear_fc1.weight15
976
  module.decoder.layers.6.mlp.experts.linear_fc1.weight14
977
- module.decoder.layers.5.mlp.experts.linear_fc2.weight0
978
  Params for bucket 39 (44040192 elements, 44040192 padded size):
979
  module.decoder.layers.5.mlp.experts.linear_fc2.weight1
980
- module.decoder.layers.5.mlp.experts.linear_fc2.weight5
981
  module.decoder.layers.5.mlp.experts.linear_fc2.weight3
 
982
  module.decoder.layers.5.mlp.experts.linear_fc2.weight7
983
  module.decoder.layers.5.mlp.experts.linear_fc2.weight2
984
  module.decoder.layers.5.mlp.experts.linear_fc2.weight4
985
  module.decoder.layers.5.mlp.experts.linear_fc2.weight6
986
  Params for bucket 40 (44040192 elements, 44040192 padded size):
987
- module.decoder.layers.5.mlp.experts.linear_fc2.weight11
988
  module.decoder.layers.5.mlp.experts.linear_fc2.weight9
 
989
  module.decoder.layers.5.mlp.experts.linear_fc2.weight13
990
  module.decoder.layers.5.mlp.experts.linear_fc2.weight8
991
  module.decoder.layers.5.mlp.experts.linear_fc2.weight12
@@ -993,45 +993,45 @@ Params for bucket 40 (44040192 elements, 44040192 padded size):
993
  module.decoder.layers.5.mlp.experts.linear_fc2.weight14
994
  Params for bucket 41 (44040192 elements, 44040192 padded size):
995
  module.decoder.layers.5.mlp.experts.linear_fc2.weight15
996
- module.decoder.layers.5.mlp.experts.linear_fc1.weight0
997
  module.decoder.layers.5.mlp.experts.linear_fc1.weight1
998
  module.decoder.layers.5.mlp.experts.linear_fc1.weight2
 
999
  Params for bucket 42 (50331648 elements, 50331648 padded size):
1000
  module.decoder.layers.5.mlp.experts.linear_fc1.weight3
 
1001
  module.decoder.layers.5.mlp.experts.linear_fc1.weight6
1002
  module.decoder.layers.5.mlp.experts.linear_fc1.weight4
1003
- module.decoder.layers.5.mlp.experts.linear_fc1.weight5
1004
  Params for bucket 43 (50331648 elements, 50331648 padded size):
1005
  module.decoder.layers.5.mlp.experts.linear_fc1.weight7
1006
- module.decoder.layers.5.mlp.experts.linear_fc1.weight10
1007
  module.decoder.layers.5.mlp.experts.linear_fc1.weight8
1008
  module.decoder.layers.5.mlp.experts.linear_fc1.weight9
 
1009
  Params for bucket 44 (50331648 elements, 50331648 padded size):
1010
  module.decoder.layers.5.mlp.experts.linear_fc1.weight11
1011
- module.decoder.layers.5.mlp.experts.linear_fc1.weight13
1012
  module.decoder.layers.5.mlp.experts.linear_fc1.weight14
1013
  module.decoder.layers.5.mlp.experts.linear_fc1.weight12
 
1014
  Params for bucket 45 (44040192 elements, 44040192 padded size):
1015
  module.decoder.layers.5.mlp.experts.linear_fc1.weight15
 
 
1016
  module.decoder.layers.4.mlp.experts.linear_fc2.weight0
1017
  module.decoder.layers.4.mlp.experts.linear_fc2.weight2
1018
  module.decoder.layers.4.mlp.experts.linear_fc2.weight4
1019
- module.decoder.layers.4.mlp.experts.linear_fc2.weight3
1020
- module.decoder.layers.4.mlp.experts.linear_fc2.weight1
1021
  Params for bucket 46 (44040192 elements, 44040192 padded size):
1022
- module.decoder.layers.4.mlp.experts.linear_fc2.weight8
1023
- module.decoder.layers.4.mlp.experts.linear_fc2.weight6
1024
- module.decoder.layers.4.mlp.experts.linear_fc2.weight10
1025
  module.decoder.layers.4.mlp.experts.linear_fc2.weight5
1026
- module.decoder.layers.4.mlp.experts.linear_fc2.weight7
1027
  module.decoder.layers.4.mlp.experts.linear_fc2.weight9
 
1028
  module.decoder.layers.4.mlp.experts.linear_fc2.weight11
 
 
 
1029
  Params for bucket 47 (50331648 elements, 50331648 padded size):
1030
- module.decoder.layers.4.mlp.experts.linear_fc2.weight12
1031
- module.decoder.layers.4.mlp.experts.linear_fc2.weight14
1032
  module.decoder.layers.4.mlp.experts.linear_fc1.weight0
1033
  module.decoder.layers.4.mlp.experts.linear_fc2.weight15
1034
- module.decoder.layers.4.mlp.experts.linear_fc2.weight13
 
1035
  module.decoder.layers.4.mlp.experts.linear_fc1.weight1
1036
  Params for bucket 48 (50331648 elements, 50331648 padded size):
1037
  module.decoder.layers.4.mlp.experts.linear_fc1.weight2
@@ -1091,21 +1091,21 @@ Params for bucket 57 (44040192 elements, 44040192 padded size):
1091
  module.decoder.layers.2.mlp.experts.linear_fc2.weight0
1092
  module.decoder.layers.3.mlp.experts.linear_fc1.weight14
1093
  Params for bucket 58 (44040192 elements, 44040192 padded size):
1094
- module.decoder.layers.2.mlp.experts.linear_fc2.weight4
1095
- module.decoder.layers.2.mlp.experts.linear_fc2.weight2
1096
- module.decoder.layers.2.mlp.experts.linear_fc2.weight6
1097
  module.decoder.layers.2.mlp.experts.linear_fc2.weight1
1098
- module.decoder.layers.2.mlp.experts.linear_fc2.weight3
1099
  module.decoder.layers.2.mlp.experts.linear_fc2.weight5
 
1100
  module.decoder.layers.2.mlp.experts.linear_fc2.weight7
 
 
 
1101
  Params for bucket 59 (44040192 elements, 44040192 padded size):
 
 
 
1102
  module.decoder.layers.2.mlp.experts.linear_fc2.weight8
1103
  module.decoder.layers.2.mlp.experts.linear_fc2.weight10
1104
  module.decoder.layers.2.mlp.experts.linear_fc2.weight12
1105
  module.decoder.layers.2.mlp.experts.linear_fc2.weight14
1106
- module.decoder.layers.2.mlp.experts.linear_fc2.weight9
1107
- module.decoder.layers.2.mlp.experts.linear_fc2.weight11
1108
- module.decoder.layers.2.mlp.experts.linear_fc2.weight13
1109
  Params for bucket 60 (44040192 elements, 44040192 padded size):
1110
  module.decoder.layers.2.mlp.experts.linear_fc2.weight15
1111
  module.decoder.layers.2.mlp.experts.linear_fc1.weight1
@@ -1128,14 +1128,14 @@ Params for bucket 63 (50331648 elements, 50331648 padded size):
1128
  module.decoder.layers.2.mlp.experts.linear_fc1.weight12
1129
  Params for bucket 64 (44040192 elements, 44040192 padded size):
1130
  module.decoder.layers.2.mlp.experts.linear_fc1.weight15
1131
- module.decoder.layers.1.mlp.experts.linear_fc2.weight2
1132
  module.decoder.layers.1.mlp.experts.linear_fc2.weight0
 
1133
  module.decoder.layers.1.mlp.experts.linear_fc2.weight4
1134
  module.decoder.layers.1.mlp.experts.linear_fc2.weight3
1135
  module.decoder.layers.1.mlp.experts.linear_fc2.weight1
1136
  Params for bucket 65 (44040192 elements, 44040192 padded size):
1137
- module.decoder.layers.1.mlp.experts.linear_fc2.weight8
1138
  module.decoder.layers.1.mlp.experts.linear_fc2.weight6
 
1139
  module.decoder.layers.1.mlp.experts.linear_fc2.weight10
1140
  module.decoder.layers.1.mlp.experts.linear_fc2.weight5
1141
  module.decoder.layers.1.mlp.experts.linear_fc2.weight9
@@ -1145,14 +1145,14 @@ Params for bucket 66 (50331648 elements, 50331648 padded size):
1145
  module.decoder.layers.1.mlp.experts.linear_fc2.weight12
1146
  module.decoder.layers.1.mlp.experts.linear_fc2.weight14
1147
  module.decoder.layers.1.mlp.experts.linear_fc1.weight1
1148
- module.decoder.layers.1.mlp.experts.linear_fc2.weight15
1149
- module.decoder.layers.1.mlp.experts.linear_fc2.weight13
1150
  module.decoder.layers.1.mlp.experts.linear_fc1.weight0
 
 
1151
  Params for bucket 67 (50331648 elements, 50331648 padded size):
1152
  module.decoder.layers.1.mlp.experts.linear_fc1.weight2
 
1153
  module.decoder.layers.1.mlp.experts.linear_fc1.weight4
1154
  module.decoder.layers.1.mlp.experts.linear_fc1.weight5
1155
- module.decoder.layers.1.mlp.experts.linear_fc1.weight3
1156
  Params for bucket 68 (50331648 elements, 50331648 padded size):
1157
  module.decoder.layers.1.mlp.experts.linear_fc1.weight6
1158
  module.decoder.layers.1.mlp.experts.linear_fc1.weight7
@@ -1160,53 +1160,53 @@ Params for bucket 68 (50331648 elements, 50331648 padded size):
1160
  module.decoder.layers.1.mlp.experts.linear_fc1.weight9
1161
  Params for bucket 69 (50331648 elements, 50331648 padded size):
1162
  module.decoder.layers.1.mlp.experts.linear_fc1.weight10
 
1163
  module.decoder.layers.1.mlp.experts.linear_fc1.weight11
1164
  module.decoder.layers.1.mlp.experts.linear_fc1.weight12
1165
- module.decoder.layers.1.mlp.experts.linear_fc1.weight13
1166
  Params for bucket 70 (44040192 elements, 44040192 padded size):
 
1167
  module.decoder.layers.1.mlp.experts.linear_fc1.weight15
1168
- module.decoder.layers.0.mlp.experts.linear_fc2.weight2
1169
- module.decoder.layers.0.mlp.experts.linear_fc2.weight0
1170
  module.decoder.layers.1.mlp.experts.linear_fc1.weight14
1171
- module.decoder.layers.0.mlp.experts.linear_fc2.weight1
 
1172
  Params for bucket 71 (44040192 elements, 44040192 padded size):
1173
- module.decoder.layers.0.mlp.experts.linear_fc2.weight4
1174
- module.decoder.layers.0.mlp.experts.linear_fc2.weight6
1175
- module.decoder.layers.0.mlp.experts.linear_fc2.weight8
1176
  module.decoder.layers.0.mlp.experts.linear_fc2.weight3
1177
- module.decoder.layers.0.mlp.experts.linear_fc2.weight5
1178
  module.decoder.layers.0.mlp.experts.linear_fc2.weight7
 
1179
  module.decoder.layers.0.mlp.experts.linear_fc2.weight9
 
 
 
1180
  Params for bucket 72 (50331648 elements, 50331648 padded size):
 
 
 
 
1181
  module.decoder.layers.0.mlp.experts.linear_fc2.weight10
1182
  module.decoder.layers.0.mlp.experts.linear_fc2.weight14
1183
  module.decoder.layers.0.mlp.experts.linear_fc2.weight12
1184
- module.decoder.layers.0.mlp.experts.linear_fc1.weight0
1185
- module.decoder.layers.0.mlp.experts.linear_fc2.weight11
1186
- module.decoder.layers.0.mlp.experts.linear_fc2.weight13
1187
- module.decoder.layers.0.mlp.experts.linear_fc2.weight15
1188
  Params for bucket 73 (50331648 elements, 50331648 padded size):
1189
  module.decoder.layers.0.mlp.experts.linear_fc1.weight1
1190
- module.decoder.layers.0.mlp.experts.linear_fc1.weight4
1191
  module.decoder.layers.0.mlp.experts.linear_fc1.weight3
1192
  module.decoder.layers.0.mlp.experts.linear_fc1.weight2
 
1193
  Params for bucket 74 (50331648 elements, 50331648 padded size):
1194
  module.decoder.layers.0.mlp.experts.linear_fc1.weight5
1195
  module.decoder.layers.0.mlp.experts.linear_fc1.weight8
1196
- module.decoder.layers.0.mlp.experts.linear_fc1.weight7
1197
  module.decoder.layers.0.mlp.experts.linear_fc1.weight6
 
1198
  Params for bucket 75 (50331648 elements, 50331648 padded size):
1199
  module.decoder.layers.0.mlp.experts.linear_fc1.weight9
1200
- module.decoder.layers.0.mlp.experts.linear_fc1.weight11
1201
- module.decoder.layers.0.mlp.experts.linear_fc1.weight10
1202
  module.decoder.layers.0.mlp.experts.linear_fc1.weight12
 
 
1203
  Params for bucket 76 (37748736 elements, 37748736 padded size):
1204
  module.decoder.layers.0.mlp.experts.linear_fc1.weight13
1205
  module.decoder.layers.0.mlp.experts.linear_fc1.weight14
1206
  module.decoder.layers.0.mlp.experts.linear_fc1.weight15
1207
- INFO:megatron.core.optimizer:Setting up optimizer with config OptimizerConfig(optimizer='adam', lr=0.0001, min_lr=1e-05, decoupled_lr=None, decoupled_min_lr=None, weight_decay=0.1, fp8_recipe='mxfp8', fp16=False, bf16=True, reuse_grad_buf_for_mxfp8_param_ag=False, params_dtype=torch.bfloat16, use_precision_aware_optimizer=False, store_param_remainders=True, main_grads_dtype=torch.float32, main_params_dtype=torch.float32, exp_avg_dtype=torch.float32, exp_avg_sq_dtype=torch.float32, loss_scale=None, initial_loss_scale=4294967296, min_loss_scale=1.0, loss_scale_window=1000, hysteresis=2, adam_beta1=0.9, adam_beta2=0.999, adam_eps=1e-08, sgd_momentum=0.9, use_distributed_optimizer=True, overlap_param_gather_with_optimizer_step=False, optimizer_cpu_offload=False, optimizer_offload_fraction=1.0, use_torch_optimizer_for_cpu_offload=False, overlap_cpu_optimizer_d2h_h2d=False, pin_cpu_grads=True, pin_cpu_params=True, clip_grad=1.0, log_num_zeros_in_grad=False, barrier_with_L1_time=True, timers=<megatron.core.timers.Timers object at 0x751c290313d0>, config_logger_dir='')
1208
  INFO:megatron.core.optimizer_param_scheduler:> learning rate decay style: cosine
1209
- [after model, optimizer, and learning rate scheduler are built] datetime: 2026-08-29 23:45:32
1210
  > building train, validation, and test datasets ...
1211
  > datasets target sizes (minimum size):
1212
  train: 96
@@ -1214,7 +1214,7 @@ INFO:megatron.core.optimizer_param_scheduler:> learning rate decay style: cosine
1214
  test: 0
1215
  INFO:megatron.core.datasets.blended_megatron_dataset_config:Let split_matrix = [(0, 1.0), None, None]
1216
  > building train, validation, and test datasets for GPT ...
1217
- INFO:megatron.core.datasets.blended_megatron_dataset_builder:Building GPTDataset splits with sizes=(96, 0, 0) and config=GPTDatasetConfig(random_seed=1234, sequence_length=8192, blend=(['/workspace/data/wikitext/wikitext103_gpt2_text_document'], None), blend_per_split=None, split='100,0,0', split_matrix=[(0, 1.0), None, None], num_dataset_builder_threads=1, path_to_cache=None, mmap_bin_files=True, mock=False, tokenizer=<megatron.training.tokenizer.tokenizer._GPT2BPETokenizer object at 0x751982c299a0>, mid_level_dataset_surplus=0.005, reset_position_ids=False, reset_attention_mask=False, eod_mask_loss=False, create_attention_mask=True, drop_last_partial_validation_sequence=True, add_extra_token_to_sequence=True, object_storage_cache_path=None)
1218
  INFO:megatron.core.datasets.indexed_dataset:Load the _IndexReader from /workspace/data/wikitext/wikitext103_gpt2_text_document.idx
1219
  INFO:megatron.core.datasets.indexed_dataset: Extract the sequence lengths
1220
  INFO:megatron.core.datasets.indexed_dataset: Extract the sequence pointers
@@ -1227,14 +1227,14 @@ INFO:megatron.core.datasets.gpt_dataset: Load the sample index from eadc3756447e
1227
  INFO:megatron.core.datasets.gpt_dataset: Load the shuffle index from eadc3756447e39d609e8aa728a91f451-GPTDataset-train-shuffle_index.npy
1228
  INFO:megatron.core.datasets.gpt_dataset:> total number of samples: 14536
1229
  > finished creating GPT datasets ...
1230
- [after dataloaders are built] datetime: 2026-08-29 23:45:32
1231
  done with setup ...
1232
- (min, max) time across ranks (ms):
1233
- model-and-optimizer-setup ......................: (3261.11, 3293.81)
1234
- train/valid/test-data-iterators-setup ..........: (63.05, 68.76)
1235
  training ...
1236
  Overwriting rerun_state_machine.current_iteration from -1 to 0...
1237
- [before the start of training step] datetime: 2026-08-29 23:45:32
 
 
 
1238
  INFO:moonep.api:MoonEP comm buffer sizes:
1239
  config : R=8, S=8192, H=4096, E=128, K=8, NvS=69600, NvS_padded=69632
1240
  sm_count : num_sms=32, num_sms_dedup=148
@@ -1246,44 +1246,43 @@ INFO:moonep.api:MoonEP comm buffer sizes:
1246
  [MoonEP selfcheck 2/4] dropped groups empty OK | covered rows [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 128, 129, 130, 131, 132, 133, 134, 135, 136, 137, 138, 139, 140, 141, 142, 143] sizes [9344, 2688, 128, 24192, 128, 0, 128, 256, 128, 768, 10496, 128, 128, 2432, 128, 1536, 14208, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0] | duplicated experts [[69, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1], [88, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1], [55, 52, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1], [-1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1], [63, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1], [124, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1], [10, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1], [58, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1]]
1247
  [MoonEP selfcheck 3/4] dropped groups empty OK | covered rows [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 128, 129, 130, 131, 132, 133, 134, 135, 136, 137, 138, 139, 140, 141, 142, 143] sizes [128, 128, 128, 0, 128, 44416, 128, 128, 128, 6912, 128, 256, 128, 128, 128, 128, 13824, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0] | duplicated experts [[78, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1], [111, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1], [62, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1], [-1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1], [57, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1], [106, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1], [121, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1], [57, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1]]
1248
  [MoonEP selfcheck 4/4] dropped groups empty OK | covered rows [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 128, 129, 130, 131, 132, 133, 134, 135, 136, 137, 138, 139, 140, 141, 142, 143] sizes [2304, 128, 128, 512, 128, 12416, 128, 1152, 1280, 16768, 640, 29056, 128, 128, 128, 1664, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0] | duplicated experts [[-1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1], [118, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1], [11, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1], [118, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1], [39, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1], [11, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1], [73, 79, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1], [86, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1]]
 
1249
  Number of parameters in transformer block in billions: 29.85
1250
  Number of parameters in embedding layers in billions: 0.41
1251
  Total number of parameters in billions: 30.26
1252
  Number of parameters in most loaded shard in billions: 30.2590
1253
  Theoretical memory footprints: weight and optimizer=216428.96 MB
1254
- [2026-08-29 23:45:53] iteration 1/ 12 | consumed samples: 8 | elapsed time per iteration (ms): 21086.4 | throughput per GPU (TFLOP/s/GPU): 8.6 | learning rate: 2.000000E-05 | global batch size: 8 | lm loss: 1.115993E+01 | loss scale: 1.0 | grad norm: 44.807 | number of skipped iterations: 0 | number of nan iterations: 0 |
1255
  [Rank 0] (after 1 iterations) memory (MB) | allocated: 67239.548828125 | max allocated: 67239.54931640625 | reserved: 68548.0 | max reserved: 68548.0
1256
- [2026-08-29 23:45:54] iteration 2/ 12 | consumed samples: 16 | elapsed time per iteration (ms): 434.3 | throughput per GPU (TFLOP/s/GPU): 416.4 | learning rate: 4.000000E-05 | global batch size: 8 | lm loss: 1.117600E+01 | loss scale: 1.0 | grad norm: 44.335 | number of skipped iterations: 0 | number of nan iterations: 0 |
1257
- [2026-08-29 23:45:54] iteration 3/ 12 | consumed samples: 24 | elapsed time per iteration (ms): 331.5 | throughput per GPU (TFLOP/s/GPU): 545.5 | learning rate: 6.000000E-05 | global batch size: 8 | lm loss: 9.686145E+00 | loss scale: 1.0 | grad norm: 62.918 | number of skipped iterations: 0 | number of nan iterations: 0 |
1258
  Capture CUDA graph for training!!!
1259
  CUDA graph capture done!!!
1260
- [2026-08-29 23:45:55] iteration 4/ 12 | consumed samples: 32 | elapsed time per iteration (ms): 982.5 | throughput per GPU (TFLOP/s/GPU): 184.1 | learning rate: 8.000000E-05 | global batch size: 8 | lm loss: 1.338087E+01 | loss scale: 1.0 | grad norm: 20.295 | number of skipped iterations: 0 | number of nan iterations: 0 |
1261
- [2026-08-29 23:45:56] iteration 5/ 12 | consumed samples: 40 | elapsed time per iteration (ms): 324.9 | throughput per GPU (TFLOP/s/GPU): 556.6 | learning rate: 1.000000E-04 | global batch size: 8 | lm loss: 1.049608E+01 | loss scale: 1.0 | grad norm: 95.117 | number of skipped iterations: 0 | number of nan iterations: 0 |
1262
- [2026-08-29 23:45:56] iteration 6/ 12 | consumed samples: 48 | elapsed time per iteration (ms): 321.3 | throughput per GPU (TFLOP/s/GPU): 562.7 | learning rate: 7.750000E-05 | global batch size: 8 | lm loss: 1.044566E+01 | loss scale: 1.0 | grad norm: 52.048 | number of skipped iterations: 0 | number of nan iterations: 0 |
1263
- [2026-08-29 23:45:56] iteration 7/ 12 | consumed samples: 56 | elapsed time per iteration (ms): 321.2 | throughput per GPU (TFLOP/s/GPU): 562.9 | learning rate: 3.250000E-05 | global batch size: 8 | lm loss: 1.027068E+01 | loss scale: 1.0 | grad norm: 6.324 | number of skipped iterations: 0 | number of nan iterations: 0 |
1264
- [2026-08-29 23:45:56] iteration 8/ 12 | consumed samples: 64 | elapsed time per iteration (ms): 321.6 | throughput per GPU (TFLOP/s/GPU): 562.2 | learning rate: 1.000000E-05 | global batch size: 8 | lm loss: 9.808611E+00 | loss scale: 1.0 | grad norm: 5.543 | number of skipped iterations: 0 | number of nan iterations: 0 |
1265
- [2026-08-29 23:45:57] iteration 9/ 12 | consumed samples: 72 | elapsed time per iteration (ms): 320.4 | throughput per GPU (TFLOP/s/GPU): 564.5 | learning rate: 1.000000E-05 | global batch size: 8 | lm loss: 9.545443E+00 | loss scale: 1.0 | grad norm: 5.290 | number of skipped iterations: 0 | number of nan iterations: 0 |
1266
- [2026-08-29 23:45:57] iteration 10/ 12 | consumed samples: 80 | elapsed time per iteration (ms): 323.8 | throughput per GPU (TFLOP/s/GPU): 558.4 | learning rate: 1.000000E-05 | global batch size: 8 | lm loss: 9.446625E+00 | loss scale: 1.0 | grad norm: 5.018 | number of skipped iterations: 0 | number of nan iterations: 0 |
1267
- [2026-08-29 23:45:57] iteration 11/ 12 | consumed samples: 88 | elapsed time per iteration (ms): 321.9 | throughput per GPU (TFLOP/s/GPU): 561.8 | learning rate: 1.000000E-05 | global batch size: 8 | lm loss: 9.324770E+00 | loss scale: 1.0 | grad norm: 4.700 | number of skipped iterations: 0 | number of nan iterations: 0 |
1268
- [2026-08-29 23:45:58] iteration 12/ 12 | consumed samples: 96 | elapsed time per iteration (ms): 321.0 | throughput per GPU (TFLOP/s/GPU): 563.4 | learning rate: 1.000000E-05 | global batch size: 8 | lm loss: 9.210297E+00 | loss scale: 1.0 | grad norm: 4.466 | number of skipped iterations: 0 | number of nan iterations: 0 |
1269
- [after training is done] datetime: 2026-08-29 23:45:58
1270
- [rank0]:[W829 23:45:59.838424178 ProcessGroupNCCL.cpp:1505] Warning: WARNING: destroy_process_group() was not called before program exit, which can leak resources. For more info, please see https://pytorch.org/docs/stable/distributed.html#shutdown (function operator())
1271
- [rank4]:[W829 23:46:00.089107186 ProcessGroupNCCL.cpp:1505] Warning: WARNING: destroy_process_group() was not called before program exit, which can leak resources. For more info, please see https://pytorch.org/docs/stable/distributed.html#shutdown (function operator())
1272
- [rank6]:[W829 23:46:00.110120316 ProcessGroupNCCL.cpp:1505] Warning: WARNING: destroy_process_group() was not called before program exit, which can leak resources. For more info, please see https://pytorch.org/docs/stable/distributed.html#shutdown (function operator())
1273
- [rank2]:[W829 23:46:00.110384142 ProcessGroupNCCL.cpp:1505] Warning: WARNING: destroy_process_group() was not called before program exit, which can leak resources. For more info, please see https://pytorch.org/docs/stable/distributed.html#shutdown (function operator())
1274
- [rank7]:[W829 23:46:00.117783475 ProcessGroupNCCL.cpp:1505] Warning: WARNING: destroy_process_group() was not called before program exit, which can leak resources. For more info, please see https://pytorch.org/docs/stable/distributed.html#shutdown (function operator())
1275
- [rank5]:[W829 23:46:00.195426955 ProcessGroupNCCL.cpp:1505] Warning: WARNING: destroy_process_group() was not called before program exit, which can leak resources. For more info, please see https://pytorch.org/docs/stable/distributed.html#shutdown (function operator())
1276
- [rank3]:[W829 23:46:01.738688058 ProcessGroupNCCL.cpp:1505] Warning: WARNING: destroy_process_group() was not called before program exit, which can leak resources. For more info, please see https://pytorch.org/docs/stable/distributed.html#shutdown (function operator())
1277
- [rank1]:[W829 23:46:01.749461216 ProcessGroupNCCL.cpp:1505] Warning: WARNING: destroy_process_group() was not called before program exit, which can leak resources. For more info, please see https://pytorch.org/docs/stable/distributed.html#shutdown (function operator())
1278
- W0829 23:46:04.688000 237461 torch/distributed/elastic/agent/server/api.py:719] Received 15 death signal, shutting down workers
1279
- W0829 23:46:04.689000 237461 torch/distributed/elastic/multiprocessing/api.py:900] Sending process 237651 closing signal SIGTERM
1280
- W0829 23:46:04.689000 237461 torch/distributed/elastic/multiprocessing/api.py:900] Sending process 237652 closing signal SIGTERM
1281
- W0829 23:46:04.690000 237461 torch/distributed/elastic/multiprocessing/api.py:900] Sending process 237653 closing signal SIGTERM
1282
- W0829 23:46:04.690000 237461 torch/distributed/elastic/multiprocessing/api.py:900] Sending process 237654 closing signal SIGTERM
1283
- W0829 23:46:04.690000 237461 torch/distributed/elastic/multiprocessing/api.py:900] Sending process 237655 closing signal SIGTERM
1284
- W0829 23:46:04.691000 237461 torch/distributed/elastic/multiprocessing/api.py:900] Sending process 237656 closing signal SIGTERM
1285
- W0829 23:46:04.691000 237461 torch/distributed/elastic/multiprocessing/api.py:900] Sending process 237657 closing signal SIGTERM
1286
- W0829 23:46:04.692000 237461 torch/distributed/elastic/multiprocessing/api.py:900] Sending process 237658 closing signal SIGTERM
1287
  Traceback (most recent call last):
1288
  File "/usr/local/bin/torchrun", line 8, in <module>
1289
  sys.exit(main())
@@ -1311,4 +1310,4 @@ Traceback (most recent call last):
1311
  time.sleep(monitor_interval)
1312
  File "/usr/local/lib/python3.12/dist-packages/torch/distributed/elastic/multiprocessing/api.py", line 84, in _terminate_process_handler
1313
  raise SignalException(f"Process {os.getpid()} got signal: {sigval}", sigval=sigval)
1314
- torch.distributed.elastic.multiprocessing.api.SignalException: Process 237461 got signal: 15
 
1
+ [01:31:18] WARN: moonep_backend_async_grad_reduce lacks --moe-router-fusion; QWEN3_235B runs unfused (timing only, routing identical)
2
  ==============================================================
3
  QWEN3_235B / moonep (slots/rank=16 policy=matched)
4
  E=128 K=8 H'=1536 m=16
5
  L=12 seq=8192 gbs=8 (1 microbatches)
6
  cap_mode=none cap=none cg=1 recompute=none
7
  memory_dump=none snapshots=/workspace/tierb-fixed-moonep-full/moonep/tierb/tierb_gbs8/QWEN3_235B__moonep__matched__seq8192__gbs8__capnone__async/rep1/memory_snapshots/snapshot.pickle
8
+ repo=/workspace/Megatron-MoE-Echo @ 2e68c8ad
9
  ==============================================================
10
+ W0830 01:31:19.434000 380220 torch/distributed/run.py:766]
11
+ W0830 01:31:19.434000 380220 torch/distributed/run.py:766] *****************************************
12
+ W0830 01:31:19.434000 380220 torch/distributed/run.py:766] Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed.
13
+ W0830 01:31:19.434000 380220 torch/distributed/run.py:766] *****************************************
 
 
 
 
 
 
 
14
  /workspace/Megatron-MoE-Echo/megatron/training/arguments.py:831: UserWarning: Disabling sequence parallelism because tensor model parallelism is disabled
15
  warnings.warn("Disabling sequence parallelism because tensor model parallelism is disabled")
16
  using world size: 8, data-parallel size: 8, context-parallel size: 1, hierarchical context-parallel sizes: None, tensor-model-parallel size: 1, pipeline-model-parallel size: 1
 
568
  warnings.warn("Disabling sequence parallelism because tensor model parallelism is disabled")
569
  /workspace/Megatron-MoE-Echo/megatron/training/arguments.py:831: UserWarning: Disabling sequence parallelism because tensor model parallelism is disabled
570
  warnings.warn("Disabling sequence parallelism because tensor model parallelism is disabled")
571
+ WARNING: one_logger package is required to enable e2e metrics tracking. please go to https://confluence.nvidia.com/display/MLWFO/Package+Repositories for details to install it
572
  /workspace/Megatron-MoE-Echo/megatron/training/arguments.py:831: UserWarning: Disabling sequence parallelism because tensor model parallelism is disabled
573
  warnings.warn("Disabling sequence parallelism because tensor model parallelism is disabled")
574
+ /workspace/Megatron-MoE-Echo/megatron/training/arguments.py:831: UserWarning: Disabling sequence parallelism because tensor model parallelism is disabled
575
+ warnings.warn("Disabling sequence parallelism because tensor model parallelism is disabled")
576
+ /workspace/Megatron-MoE-Echo/megatron/training/arguments.py:831: UserWarning: Disabling sequence parallelism because tensor model parallelism is disabled
577
+ warnings.warn("Disabling sequence parallelism because tensor model parallelism is disabled")
578
+ /workspace/Megatron-MoE-Echo/megatron/training/arguments.py:831: UserWarning: Disabling sequence parallelism because tensor model parallelism is disabled
579
+ warnings.warn("Disabling sequence parallelism because tensor model parallelism is disabled")
580
+ [Gloo] Rank [Gloo] Rank 0 is connected to 72 peer ranks. is connected to Expected number of connected peer ranks is : 77 peer ranks. Expected number of connected peer ranks is :
581
+ 7
582
  [Gloo] Rank 4 is connected to 7 peer ranks. Expected number of connected peer ranks is : 7
583
  [Gloo] Rank 1 is connected to 7 peer ranks. Expected number of connected peer ranks is : 7
584
+ [Gloo] Rank 5 is connected to 7 peer ranks. Expected number of connected peer ranks is : 7[Gloo] Rank
585
+ 3 is connected to 7 peer ranks. Expected number of connected peer ranks is : 7
 
586
  [Gloo] Rank 6 is connected to 7 peer ranks. Expected number of connected peer ranks is : 7
587
+ [Gloo] Rank 7 is connected to 7 peer ranks. Expected number of connected peer ranks is : 7
 
 
588
  [Gloo] Rank 4 is connected to 7 peer ranks. Expected number of connected peer ranks is : 7
589
  [Gloo] Rank 0 is connected to 7 peer ranks. Expected number of connected peer ranks is : 7
590
+ [Gloo] Rank 2 is connected to 7 peer ranks. Expected number of connected peer ranks is : 7
591
  [Gloo] Rank 1 is connected to 7 peer ranks. Expected number of connected peer ranks is : 7
592
+ [Gloo] Rank 3 is connected to 7 peer ranks. Expected number of connected peer ranks is : 7
593
+ [Gloo] Rank 5 is connected to 7 peer ranks. Expected number of connected peer ranks is : 7
594
+ [Gloo] Rank [Gloo] Rank 6 is connected to 77 is connected to peer ranks. 7Expected number of connected peer ranks is : peer ranks. 7Expected number of connected peer ranks is :
595
+ 7
596
  [Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
597
  > initialized tensor model parallel with size 1
598
  > initialized pipeline model parallel with size 1
599
  > setting random seeds to 1234 ...
600
+ [Gloo] Rank [Gloo] Rank 0[Gloo] Rank 00 is connected to is connected to is connected to 000 peer ranks. peer ranks. peer ranks. Expected number of connected peer ranks is : Expected number of connected peer ranks is : Expected number of connected peer ranks is : 000
601
 
 
602
 
603
+ [Gloo] Rank [Gloo] Rank 00 is connected to is connected to 00 peer ranks. [Gloo] Rank peer ranks. Expected number of connected peer ranks is : Expected number of connected peer ranks is : 000 is connected to
604
+
605
+ 0 peer ranks. Expected number of connected peer ranks is : 0
606
  [Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
607
+ [rank7]:[W830 01:31:37.479166448 ProcessGroupNCCL.cpp:4789] [PG ID 0 PG GUID 0 Rank 7] using GPU 7 as device used by this process is currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect. You can specify device_id in init_process_group() to force use of a particular device.
608
+ [rank6]:[W830 01:31:37.479214134 ProcessGroupNCCL.cpp:4789] [PG ID 0 PG GUID 0 Rank 6] using GPU 6 as device used by this process is currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect. You can specify device_id in init_process_group() to force use of a particular device.
609
+ [rank1]:[W830 01:31:37.479251837 ProcessGroupNCCL.cpp:4789] [PG ID 0 PG GUID 0 Rank 1] using GPU 1 as device used by this process is currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect. You can specify device_id in init_process_group() to force use of a particular device.
610
+ [rank4]:[W830 01:31:37.481705539 ProcessGroupNCCL.cpp:4789] [PG ID 0 PG GUID 0 Rank 4] using GPU 4 as device used by this process is currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect. You can specify device_id in init_process_group() to force use of a particular device.
611
+ [rank3]:[W830 01:31:37.481736938 ProcessGroupNCCL.cpp:4789] [PG ID 0 PG GUID 0 Rank 3] using GPU 3 as device used by this process is currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect. You can specify device_id in init_process_group() to force use of a particular device.
612
+ [rank2]:[W830 01:31:37.481915707 ProcessGroupNCCL.cpp:4789] [PG ID 0 PG GUID 0 Rank 2] using GPU 2 as device used by this process is currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect. You can specify device_id in init_process_group() to force use of a particular device.
613
+ [rank5]:[W830 01:31:37.481926780 ProcessGroupNCCL.cpp:4789] [PG ID 0 PG GUID 0 Rank 5] using GPU 5 as device used by this process is currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect. You can specify device_id in init_process_group() to force use of a particular device.
 
 
614
  > compiling dataset index builder ...
615
  make: Entering directory '/workspace/Megatron-MoE-Echo/megatron/core/datasets'
616
  make: Nothing to be done for 'default'.
617
  make: Leaving directory '/workspace/Megatron-MoE-Echo/megatron/core/datasets'
618
+ >>> done with dataset index builder. Compilation time: 0.036 seconds
619
  WARNING: constraints for invoking optimized fused softmax kernel are not met. We default back to unfused kernel invocations.
620
  > compiling and loading fused kernels ...
621
+ [rank0]:[W830 01:31:37.645502642 ProcessGroupNCCL.cpp:4789] [PG ID 0 PG GUID 0 Rank 0] using GPU 0 as device used by this process is currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect. You can specify device_id in init_process_group() to force use of a particular device.
622
+ >>> done with compiling and loading fused kernels. Compilation time: 3.763 seconds
623
+ time to initialize megatron (seconds): 7.462
624
+ [after megatron is initialized] datetime: 2026-08-30 01:31:43
625
  building GPT model ...
626
  /usr/local/lib/python3.12/dist-packages/transformer_engine/pytorch/cpu_offload.py:708: DeprecationWarning: Offloading weights is deprecated. Using offload_weights=True does not have any effect.
627
  warnings.warn(
 
646
  module.output_layer.weight
647
  Params for bucket 2 (71840000 elements, 71840000 padded size):
648
  module.decoder.layers.11.pre_mlp_layernorm.weight
649
+ module.decoder.layers.11.self_attention.k_layernorm.weight
650
  module.decoder.layers.11.mlp.router.weight
651
  module.decoder.layers.11.self_attention.q_layernorm.weight
652
  module.decoder.layers.11.self_attention.linear_qkv.weight
653
  module.decoder.layers.11.self_attention.linear_qkv.layer_norm_weight
654
  module.decoder.final_layernorm.weight
 
655
  module.decoder.layers.11.self_attention.linear_proj.weight
656
  Params for bucket 3 (71835904 elements, 71835904 padded size):
657
  module.decoder.layers.10.mlp.router.weight
 
 
 
658
  module.decoder.layers.10.self_attention.linear_qkv.weight
659
+ module.decoder.layers.10.self_attention.q_layernorm.weight
660
+ module.decoder.layers.10.pre_mlp_layernorm.weight
661
  module.decoder.layers.10.self_attention.linear_qkv.layer_norm_weight
662
+ module.decoder.layers.10.self_attention.k_layernorm.weight
663
  module.decoder.layers.10.self_attention.linear_proj.weight
664
  Params for bucket 4 (71835904 elements, 71835904 padded size):
665
  module.decoder.layers.9.mlp.router.weight
 
 
 
666
  module.decoder.layers.9.self_attention.k_layernorm.weight
667
+ module.decoder.layers.9.self_attention.q_layernorm.weight
668
+ module.decoder.layers.9.pre_mlp_layernorm.weight
669
  module.decoder.layers.9.self_attention.linear_qkv.weight
670
  module.decoder.layers.9.self_attention.linear_qkv.layer_norm_weight
671
+ module.decoder.layers.9.self_attention.linear_proj.weight
672
  Params for bucket 5 (71835904 elements, 71835904 padded size):
673
  module.decoder.layers.8.mlp.router.weight
 
674
  module.decoder.layers.8.self_attention.k_layernorm.weight
675
+ module.decoder.layers.8.pre_mlp_layernorm.weight
676
  module.decoder.layers.8.self_attention.linear_qkv.weight
677
  module.decoder.layers.8.self_attention.q_layernorm.weight
678
  module.decoder.layers.8.self_attention.linear_qkv.layer_norm_weight
679
  module.decoder.layers.8.self_attention.linear_proj.weight
680
  Params for bucket 6 (71835904 elements, 71835904 padded size):
681
  module.decoder.layers.7.mlp.router.weight
 
682
  module.decoder.layers.7.self_attention.q_layernorm.weight
683
  module.decoder.layers.7.self_attention.k_layernorm.weight
684
  module.decoder.layers.7.self_attention.linear_qkv.layer_norm_weight
685
  module.decoder.layers.7.self_attention.linear_qkv.weight
686
+ module.decoder.layers.7.pre_mlp_layernorm.weight
687
  module.decoder.layers.7.self_attention.linear_proj.weight
688
  Params for bucket 7 (71835904 elements, 71835904 padded size):
689
  module.decoder.layers.6.mlp.router.weight
690
  module.decoder.layers.6.pre_mlp_layernorm.weight
691
  module.decoder.layers.6.self_attention.k_layernorm.weight
 
692
  module.decoder.layers.6.self_attention.q_layernorm.weight
693
+ module.decoder.layers.6.self_attention.linear_qkv.weight
694
  module.decoder.layers.6.self_attention.linear_qkv.layer_norm_weight
695
  module.decoder.layers.6.self_attention.linear_proj.weight
696
  Params for bucket 8 (71835904 elements, 71835904 padded size):
 
697
  module.decoder.layers.5.pre_mlp_layernorm.weight
 
698
  module.decoder.layers.5.self_attention.q_layernorm.weight
699
  module.decoder.layers.5.self_attention.linear_qkv.layer_norm_weight
700
+ module.decoder.layers.5.mlp.router.weight
701
+ module.decoder.layers.5.self_attention.linear_qkv.weight
702
  module.decoder.layers.5.self_attention.k_layernorm.weight
703
  module.decoder.layers.5.self_attention.linear_proj.weight
704
  Params for bucket 9 (71835904 elements, 71835904 padded size):
 
705
  module.decoder.layers.4.pre_mlp_layernorm.weight
 
706
  module.decoder.layers.4.self_attention.q_layernorm.weight
707
  module.decoder.layers.4.self_attention.linear_qkv.weight
708
  module.decoder.layers.4.self_attention.linear_qkv.layer_norm_weight
709
+ module.decoder.layers.4.mlp.router.weight
710
+ module.decoder.layers.4.self_attention.k_layernorm.weight
711
  module.decoder.layers.4.self_attention.linear_proj.weight
712
  Params for bucket 10 (71835904 elements, 71835904 padded size):
713
  module.decoder.layers.3.mlp.router.weight
 
714
  module.decoder.layers.3.self_attention.linear_qkv.weight
715
  module.decoder.layers.3.self_attention.q_layernorm.weight
716
+ module.decoder.layers.3.pre_mlp_layernorm.weight
717
  module.decoder.layers.3.self_attention.k_layernorm.weight
718
  module.decoder.layers.3.self_attention.linear_qkv.layer_norm_weight
719
  module.decoder.layers.3.self_attention.linear_proj.weight
720
  Params for bucket 11 (71835904 elements, 71835904 padded size):
 
 
721
  module.decoder.layers.2.self_attention.q_layernorm.weight
722
+ module.decoder.layers.2.mlp.router.weight
723
+ module.decoder.layers.2.pre_mlp_layernorm.weight
724
  module.decoder.layers.2.self_attention.linear_qkv.weight
725
+ module.decoder.layers.2.self_attention.k_layernorm.weight
726
  module.decoder.layers.2.self_attention.linear_qkv.layer_norm_weight
727
  module.decoder.layers.2.self_attention.linear_proj.weight
 
728
  Params for bucket 12 (71835904 elements, 71835904 padded size):
729
  module.decoder.layers.1.mlp.router.weight
730
+ module.decoder.layers.1.self_attention.k_layernorm.weight
731
  module.decoder.layers.1.self_attention.linear_qkv.weight
 
732
  module.decoder.layers.1.pre_mlp_layernorm.weight
733
+ module.decoder.layers.1.self_attention.q_layernorm.weight
734
  module.decoder.layers.1.self_attention.linear_qkv.layer_norm_weight
735
  module.decoder.layers.1.self_attention.linear_proj.weight
736
  Params for bucket 13 (71835904 elements, 71835904 padded size):
 
 
 
737
  module.decoder.layers.0.self_attention.linear_qkv.weight
 
738
  module.decoder.layers.0.self_attention.linear_qkv.layer_norm_weight
739
+ module.decoder.layers.0.mlp.router.weight
740
+ module.decoder.layers.0.pre_mlp_layernorm.weight
741
+ module.decoder.layers.0.self_attention.k_layernorm.weight
742
+ module.decoder.layers.0.self_attention.q_layernorm.weight
743
  module.decoder.layers.0.self_attention.linear_proj.weight
744
  Params for bucket 14 (206045184 elements, 206045184 padded size):
745
  module.embedding.word_embeddings.weight
 
749
  module.decoder.layers.11.mlp.experts.linear_fc2.weight2
750
  module.decoder.layers.11.mlp.experts.linear_fc2.weight4
751
  module.decoder.layers.11.mlp.experts.linear_fc2.weight6
 
752
  module.decoder.layers.11.mlp.experts.linear_fc2.weight1
753
+ module.decoder.layers.11.mlp.experts.linear_fc2.weight3
754
  module.decoder.layers.11.mlp.experts.linear_fc2.weight5
755
  Params for bucket 2 (44040192 elements, 44040192 padded size):
756
  module.decoder.layers.11.mlp.experts.linear_fc2.weight10
 
764
  module.decoder.layers.11.mlp.experts.linear_fc2.weight14
765
  module.decoder.layers.11.mlp.experts.linear_fc1.weight1
766
  module.decoder.layers.11.mlp.experts.linear_fc1.weight0
 
767
  module.decoder.layers.11.mlp.experts.linear_fc1.weight2
768
+ module.decoder.layers.11.mlp.experts.linear_fc2.weight15
769
  Params for bucket 4 (50331648 elements, 50331648 padded size):
770
  module.decoder.layers.11.mlp.experts.linear_fc1.weight3
 
771
  module.decoder.layers.11.mlp.experts.linear_fc1.weight4
772
+ module.decoder.layers.11.mlp.experts.linear_fc1.weight6
773
  module.decoder.layers.11.mlp.experts.linear_fc1.weight5
774
  Params for bucket 5 (50331648 elements, 50331648 padded size):
775
  module.decoder.layers.11.mlp.experts.linear_fc1.weight7
 
783
  module.decoder.layers.11.mlp.experts.linear_fc1.weight12
784
  Params for bucket 7 (44040192 elements, 44040192 padded size):
785
  module.decoder.layers.11.mlp.experts.linear_fc1.weight15
 
 
 
786
  module.decoder.layers.10.mlp.experts.linear_fc2.weight2
787
+ module.decoder.layers.10.mlp.experts.linear_fc2.weight0
788
  module.decoder.layers.10.mlp.experts.linear_fc2.weight4
789
+ module.decoder.layers.10.mlp.experts.linear_fc2.weight3
790
+ module.decoder.layers.10.mlp.experts.linear_fc2.weight1
791
  Params for bucket 8 (44040192 elements, 44040192 padded size):
792
+ module.decoder.layers.10.mlp.experts.linear_fc2.weight8
793
+ module.decoder.layers.10.mlp.experts.linear_fc2.weight6
794
+ module.decoder.layers.10.mlp.experts.linear_fc2.weight10
795
  module.decoder.layers.10.mlp.experts.linear_fc2.weight5
 
796
  module.decoder.layers.10.mlp.experts.linear_fc2.weight9
797
+ module.decoder.layers.10.mlp.experts.linear_fc2.weight7
798
  module.decoder.layers.10.mlp.experts.linear_fc2.weight11
 
 
 
799
  Params for bucket 9 (50331648 elements, 50331648 padded size):
 
 
 
800
  module.decoder.layers.10.mlp.experts.linear_fc2.weight12
801
  module.decoder.layers.10.mlp.experts.linear_fc2.weight14
802
  module.decoder.layers.10.mlp.experts.linear_fc1.weight1
803
+ module.decoder.layers.10.mlp.experts.linear_fc2.weight15
804
+ module.decoder.layers.10.mlp.experts.linear_fc2.weight13
805
+ module.decoder.layers.10.mlp.experts.linear_fc1.weight0
806
  Params for bucket 10 (50331648 elements, 50331648 padded size):
807
  module.decoder.layers.10.mlp.experts.linear_fc1.weight2
808
+ module.decoder.layers.10.mlp.experts.linear_fc1.weight5
809
  module.decoder.layers.10.mlp.experts.linear_fc1.weight3
810
  module.decoder.layers.10.mlp.experts.linear_fc1.weight4
 
811
  Params for bucket 11 (50331648 elements, 50331648 padded size):
812
  module.decoder.layers.10.mlp.experts.linear_fc1.weight6
813
+ module.decoder.layers.10.mlp.experts.linear_fc1.weight8
814
  module.decoder.layers.10.mlp.experts.linear_fc1.weight9
815
  module.decoder.layers.10.mlp.experts.linear_fc1.weight7
 
816
  Params for bucket 12 (50331648 elements, 50331648 padded size):
817
  module.decoder.layers.10.mlp.experts.linear_fc1.weight10
818
+ module.decoder.layers.10.mlp.experts.linear_fc1.weight11
819
  module.decoder.layers.10.mlp.experts.linear_fc1.weight12
820
  module.decoder.layers.10.mlp.experts.linear_fc1.weight13
 
821
  Params for bucket 13 (44040192 elements, 44040192 padded size):
822
+ module.decoder.layers.9.mlp.experts.linear_fc2.weight1
823
+ module.decoder.layers.10.mlp.experts.linear_fc1.weight15
824
  module.decoder.layers.10.mlp.experts.linear_fc1.weight14
825
  module.decoder.layers.9.mlp.experts.linear_fc2.weight2
826
  module.decoder.layers.9.mlp.experts.linear_fc2.weight0
 
 
827
  Params for bucket 14 (44040192 elements, 44040192 padded size):
 
 
 
828
  module.decoder.layers.9.mlp.experts.linear_fc2.weight3
 
829
  module.decoder.layers.9.mlp.experts.linear_fc2.weight5
830
+ module.decoder.layers.9.mlp.experts.linear_fc2.weight7
831
  module.decoder.layers.9.mlp.experts.linear_fc2.weight9
832
+ module.decoder.layers.9.mlp.experts.linear_fc2.weight6
833
+ module.decoder.layers.9.mlp.experts.linear_fc2.weight4
834
+ module.decoder.layers.9.mlp.experts.linear_fc2.weight8
835
  Params for bucket 15 (50331648 elements, 50331648 padded size):
 
 
 
 
 
836
  module.decoder.layers.9.mlp.experts.linear_fc2.weight11
837
+ module.decoder.layers.9.mlp.experts.linear_fc2.weight13
838
  module.decoder.layers.9.mlp.experts.linear_fc2.weight15
839
+ module.decoder.layers.9.mlp.experts.linear_fc1.weight0
840
+ module.decoder.layers.9.mlp.experts.linear_fc2.weight10
841
+ module.decoder.layers.9.mlp.experts.linear_fc2.weight14
842
+ module.decoder.layers.9.mlp.experts.linear_fc2.weight12
843
  Params for bucket 16 (50331648 elements, 50331648 padded size):
844
  module.decoder.layers.9.mlp.experts.linear_fc1.weight1
845
  module.decoder.layers.9.mlp.experts.linear_fc1.weight4
 
870
  module.decoder.layers.8.mlp.experts.linear_fc2.weight7
871
  Params for bucket 21 (44040192 elements, 44040192 padded size):
872
  module.decoder.layers.8.mlp.experts.linear_fc2.weight8
 
873
  module.decoder.layers.8.mlp.experts.linear_fc2.weight12
874
+ module.decoder.layers.8.mlp.experts.linear_fc2.weight10
875
  module.decoder.layers.8.mlp.experts.linear_fc2.weight14
876
  module.decoder.layers.8.mlp.experts.linear_fc2.weight9
877
  module.decoder.layers.8.mlp.experts.linear_fc2.weight11
878
  module.decoder.layers.8.mlp.experts.linear_fc2.weight13
879
  Params for bucket 22 (44040192 elements, 44040192 padded size):
880
  module.decoder.layers.8.mlp.experts.linear_fc2.weight15
 
881
  module.decoder.layers.8.mlp.experts.linear_fc1.weight0
882
  module.decoder.layers.8.mlp.experts.linear_fc1.weight2
883
+ module.decoder.layers.8.mlp.experts.linear_fc1.weight1
884
  Params for bucket 23 (50331648 elements, 50331648 padded size):
885
  module.decoder.layers.8.mlp.experts.linear_fc1.weight3
886
+ module.decoder.layers.8.mlp.experts.linear_fc1.weight6
887
  module.decoder.layers.8.mlp.experts.linear_fc1.weight4
888
  module.decoder.layers.8.mlp.experts.linear_fc1.weight5
 
889
  Params for bucket 24 (50331648 elements, 50331648 padded size):
890
  module.decoder.layers.8.mlp.experts.linear_fc1.weight7
891
+ module.decoder.layers.8.mlp.experts.linear_fc1.weight9
892
  module.decoder.layers.8.mlp.experts.linear_fc1.weight10
893
  module.decoder.layers.8.mlp.experts.linear_fc1.weight8
 
894
  Params for bucket 25 (50331648 elements, 50331648 padded size):
895
  module.decoder.layers.8.mlp.experts.linear_fc1.weight11
896
+ module.decoder.layers.8.mlp.experts.linear_fc1.weight12
897
  module.decoder.layers.8.mlp.experts.linear_fc1.weight13
898
  module.decoder.layers.8.mlp.experts.linear_fc1.weight14
 
899
  Params for bucket 26 (44040192 elements, 44040192 padded size):
 
 
 
 
900
  module.decoder.layers.7.mlp.experts.linear_fc2.weight2
901
+ module.decoder.layers.7.mlp.experts.linear_fc2.weight0
902
  module.decoder.layers.7.mlp.experts.linear_fc2.weight4
903
+ module.decoder.layers.8.mlp.experts.linear_fc1.weight15
904
+ module.decoder.layers.7.mlp.experts.linear_fc2.weight1
905
+ module.decoder.layers.7.mlp.experts.linear_fc2.weight3
906
  Params for bucket 27 (44040192 elements, 44040192 padded size):
907
+ module.decoder.layers.7.mlp.experts.linear_fc2.weight8
908
+ module.decoder.layers.7.mlp.experts.linear_fc2.weight6
909
+ module.decoder.layers.7.mlp.experts.linear_fc2.weight10
910
  module.decoder.layers.7.mlp.experts.linear_fc2.weight5
911
  module.decoder.layers.7.mlp.experts.linear_fc2.weight9
912
  module.decoder.layers.7.mlp.experts.linear_fc2.weight7
913
  module.decoder.layers.7.mlp.experts.linear_fc2.weight11
 
 
 
914
  Params for bucket 28 (50331648 elements, 50331648 padded size):
 
 
 
915
  module.decoder.layers.7.mlp.experts.linear_fc2.weight12
916
  module.decoder.layers.7.mlp.experts.linear_fc2.weight14
917
  module.decoder.layers.7.mlp.experts.linear_fc1.weight1
918
+ module.decoder.layers.7.mlp.experts.linear_fc2.weight15
919
+ module.decoder.layers.7.mlp.experts.linear_fc1.weight0
920
+ module.decoder.layers.7.mlp.experts.linear_fc2.weight13
921
  Params for bucket 29 (50331648 elements, 50331648 padded size):
922
  module.decoder.layers.7.mlp.experts.linear_fc1.weight2
923
+ module.decoder.layers.7.mlp.experts.linear_fc1.weight5
924
  module.decoder.layers.7.mlp.experts.linear_fc1.weight3
925
  module.decoder.layers.7.mlp.experts.linear_fc1.weight4
 
926
  Params for bucket 30 (50331648 elements, 50331648 padded size):
927
  module.decoder.layers.7.mlp.experts.linear_fc1.weight6
928
+ module.decoder.layers.7.mlp.experts.linear_fc1.weight8
929
  module.decoder.layers.7.mlp.experts.linear_fc1.weight7
930
  module.decoder.layers.7.mlp.experts.linear_fc1.weight9
 
931
  Params for bucket 31 (50331648 elements, 50331648 padded size):
932
  module.decoder.layers.7.mlp.experts.linear_fc1.weight10
 
933
  module.decoder.layers.7.mlp.experts.linear_fc1.weight12
934
  module.decoder.layers.7.mlp.experts.linear_fc1.weight11
935
+ module.decoder.layers.7.mlp.experts.linear_fc1.weight13
936
  Params for bucket 32 (44040192 elements, 44040192 padded size):
937
+ module.decoder.layers.7.mlp.experts.linear_fc1.weight15
 
938
  module.decoder.layers.6.mlp.experts.linear_fc2.weight0
939
+ module.decoder.layers.6.mlp.experts.linear_fc2.weight2
940
+ module.decoder.layers.7.mlp.experts.linear_fc1.weight14
941
  module.decoder.layers.6.mlp.experts.linear_fc2.weight1
 
942
  Params for bucket 33 (44040192 elements, 44040192 padded size):
943
  module.decoder.layers.6.mlp.experts.linear_fc2.weight6
944
  module.decoder.layers.6.mlp.experts.linear_fc2.weight4
945
  module.decoder.layers.6.mlp.experts.linear_fc2.weight8
946
  module.decoder.layers.6.mlp.experts.linear_fc2.weight3
 
947
  module.decoder.layers.6.mlp.experts.linear_fc2.weight7
948
+ module.decoder.layers.6.mlp.experts.linear_fc2.weight5
949
  module.decoder.layers.6.mlp.experts.linear_fc2.weight9
950
  Params for bucket 34 (50331648 elements, 50331648 padded size):
951
  module.decoder.layers.6.mlp.experts.linear_fc2.weight10
952
  module.decoder.layers.6.mlp.experts.linear_fc2.weight14
953
  module.decoder.layers.6.mlp.experts.linear_fc2.weight12
 
954
  module.decoder.layers.6.mlp.experts.linear_fc2.weight13
955
+ module.decoder.layers.6.mlp.experts.linear_fc2.weight11
956
  module.decoder.layers.6.mlp.experts.linear_fc2.weight15
957
  module.decoder.layers.6.mlp.experts.linear_fc1.weight0
958
  Params for bucket 35 (50331648 elements, 50331648 padded size):
959
  module.decoder.layers.6.mlp.experts.linear_fc1.weight1
 
960
  module.decoder.layers.6.mlp.experts.linear_fc1.weight3
961
  module.decoder.layers.6.mlp.experts.linear_fc1.weight4
962
+ module.decoder.layers.6.mlp.experts.linear_fc1.weight2
963
  Params for bucket 36 (50331648 elements, 50331648 padded size):
964
  module.decoder.layers.6.mlp.experts.linear_fc1.weight5
 
965
  module.decoder.layers.6.mlp.experts.linear_fc1.weight6
966
  module.decoder.layers.6.mlp.experts.linear_fc1.weight7
967
+ module.decoder.layers.6.mlp.experts.linear_fc1.weight8
968
  Params for bucket 37 (50331648 elements, 50331648 padded size):
969
  module.decoder.layers.6.mlp.experts.linear_fc1.weight9
 
970
  module.decoder.layers.6.mlp.experts.linear_fc1.weight12
971
  module.decoder.layers.6.mlp.experts.linear_fc1.weight10
972
+ module.decoder.layers.6.mlp.experts.linear_fc1.weight11
973
  Params for bucket 38 (44040192 elements, 44040192 padded size):
974
  module.decoder.layers.6.mlp.experts.linear_fc1.weight13
975
+ module.decoder.layers.5.mlp.experts.linear_fc2.weight0
976
  module.decoder.layers.6.mlp.experts.linear_fc1.weight15
977
  module.decoder.layers.6.mlp.experts.linear_fc1.weight14
 
978
  Params for bucket 39 (44040192 elements, 44040192 padded size):
979
  module.decoder.layers.5.mlp.experts.linear_fc2.weight1
 
980
  module.decoder.layers.5.mlp.experts.linear_fc2.weight3
981
+ module.decoder.layers.5.mlp.experts.linear_fc2.weight5
982
  module.decoder.layers.5.mlp.experts.linear_fc2.weight7
983
  module.decoder.layers.5.mlp.experts.linear_fc2.weight2
984
  module.decoder.layers.5.mlp.experts.linear_fc2.weight4
985
  module.decoder.layers.5.mlp.experts.linear_fc2.weight6
986
  Params for bucket 40 (44040192 elements, 44040192 padded size):
 
987
  module.decoder.layers.5.mlp.experts.linear_fc2.weight9
988
+ module.decoder.layers.5.mlp.experts.linear_fc2.weight11
989
  module.decoder.layers.5.mlp.experts.linear_fc2.weight13
990
  module.decoder.layers.5.mlp.experts.linear_fc2.weight8
991
  module.decoder.layers.5.mlp.experts.linear_fc2.weight12
 
993
  module.decoder.layers.5.mlp.experts.linear_fc2.weight14
994
  Params for bucket 41 (44040192 elements, 44040192 padded size):
995
  module.decoder.layers.5.mlp.experts.linear_fc2.weight15
 
996
  module.decoder.layers.5.mlp.experts.linear_fc1.weight1
997
  module.decoder.layers.5.mlp.experts.linear_fc1.weight2
998
+ module.decoder.layers.5.mlp.experts.linear_fc1.weight0
999
  Params for bucket 42 (50331648 elements, 50331648 padded size):
1000
  module.decoder.layers.5.mlp.experts.linear_fc1.weight3
1001
+ module.decoder.layers.5.mlp.experts.linear_fc1.weight5
1002
  module.decoder.layers.5.mlp.experts.linear_fc1.weight6
1003
  module.decoder.layers.5.mlp.experts.linear_fc1.weight4
 
1004
  Params for bucket 43 (50331648 elements, 50331648 padded size):
1005
  module.decoder.layers.5.mlp.experts.linear_fc1.weight7
 
1006
  module.decoder.layers.5.mlp.experts.linear_fc1.weight8
1007
  module.decoder.layers.5.mlp.experts.linear_fc1.weight9
1008
+ module.decoder.layers.5.mlp.experts.linear_fc1.weight10
1009
  Params for bucket 44 (50331648 elements, 50331648 padded size):
1010
  module.decoder.layers.5.mlp.experts.linear_fc1.weight11
 
1011
  module.decoder.layers.5.mlp.experts.linear_fc1.weight14
1012
  module.decoder.layers.5.mlp.experts.linear_fc1.weight12
1013
+ module.decoder.layers.5.mlp.experts.linear_fc1.weight13
1014
  Params for bucket 45 (44040192 elements, 44040192 padded size):
1015
  module.decoder.layers.5.mlp.experts.linear_fc1.weight15
1016
+ module.decoder.layers.4.mlp.experts.linear_fc2.weight3
1017
+ module.decoder.layers.4.mlp.experts.linear_fc2.weight1
1018
  module.decoder.layers.4.mlp.experts.linear_fc2.weight0
1019
  module.decoder.layers.4.mlp.experts.linear_fc2.weight2
1020
  module.decoder.layers.4.mlp.experts.linear_fc2.weight4
 
 
1021
  Params for bucket 46 (44040192 elements, 44040192 padded size):
 
 
 
1022
  module.decoder.layers.4.mlp.experts.linear_fc2.weight5
 
1023
  module.decoder.layers.4.mlp.experts.linear_fc2.weight9
1024
+ module.decoder.layers.4.mlp.experts.linear_fc2.weight7
1025
  module.decoder.layers.4.mlp.experts.linear_fc2.weight11
1026
+ module.decoder.layers.4.mlp.experts.linear_fc2.weight6
1027
+ module.decoder.layers.4.mlp.experts.linear_fc2.weight8
1028
+ module.decoder.layers.4.mlp.experts.linear_fc2.weight10
1029
  Params for bucket 47 (50331648 elements, 50331648 padded size):
1030
+ module.decoder.layers.4.mlp.experts.linear_fc2.weight13
 
1031
  module.decoder.layers.4.mlp.experts.linear_fc1.weight0
1032
  module.decoder.layers.4.mlp.experts.linear_fc2.weight15
1033
+ module.decoder.layers.4.mlp.experts.linear_fc2.weight12
1034
+ module.decoder.layers.4.mlp.experts.linear_fc2.weight14
1035
  module.decoder.layers.4.mlp.experts.linear_fc1.weight1
1036
  Params for bucket 48 (50331648 elements, 50331648 padded size):
1037
  module.decoder.layers.4.mlp.experts.linear_fc1.weight2
 
1091
  module.decoder.layers.2.mlp.experts.linear_fc2.weight0
1092
  module.decoder.layers.3.mlp.experts.linear_fc1.weight14
1093
  Params for bucket 58 (44040192 elements, 44040192 padded size):
 
 
 
1094
  module.decoder.layers.2.mlp.experts.linear_fc2.weight1
 
1095
  module.decoder.layers.2.mlp.experts.linear_fc2.weight5
1096
+ module.decoder.layers.2.mlp.experts.linear_fc2.weight3
1097
  module.decoder.layers.2.mlp.experts.linear_fc2.weight7
1098
+ module.decoder.layers.2.mlp.experts.linear_fc2.weight4
1099
+ module.decoder.layers.2.mlp.experts.linear_fc2.weight2
1100
+ module.decoder.layers.2.mlp.experts.linear_fc2.weight6
1101
  Params for bucket 59 (44040192 elements, 44040192 padded size):
1102
+ module.decoder.layers.2.mlp.experts.linear_fc2.weight11
1103
+ module.decoder.layers.2.mlp.experts.linear_fc2.weight9
1104
+ module.decoder.layers.2.mlp.experts.linear_fc2.weight13
1105
  module.decoder.layers.2.mlp.experts.linear_fc2.weight8
1106
  module.decoder.layers.2.mlp.experts.linear_fc2.weight10
1107
  module.decoder.layers.2.mlp.experts.linear_fc2.weight12
1108
  module.decoder.layers.2.mlp.experts.linear_fc2.weight14
 
 
 
1109
  Params for bucket 60 (44040192 elements, 44040192 padded size):
1110
  module.decoder.layers.2.mlp.experts.linear_fc2.weight15
1111
  module.decoder.layers.2.mlp.experts.linear_fc1.weight1
 
1128
  module.decoder.layers.2.mlp.experts.linear_fc1.weight12
1129
  Params for bucket 64 (44040192 elements, 44040192 padded size):
1130
  module.decoder.layers.2.mlp.experts.linear_fc1.weight15
 
1131
  module.decoder.layers.1.mlp.experts.linear_fc2.weight0
1132
+ module.decoder.layers.1.mlp.experts.linear_fc2.weight2
1133
  module.decoder.layers.1.mlp.experts.linear_fc2.weight4
1134
  module.decoder.layers.1.mlp.experts.linear_fc2.weight3
1135
  module.decoder.layers.1.mlp.experts.linear_fc2.weight1
1136
  Params for bucket 65 (44040192 elements, 44040192 padded size):
 
1137
  module.decoder.layers.1.mlp.experts.linear_fc2.weight6
1138
+ module.decoder.layers.1.mlp.experts.linear_fc2.weight8
1139
  module.decoder.layers.1.mlp.experts.linear_fc2.weight10
1140
  module.decoder.layers.1.mlp.experts.linear_fc2.weight5
1141
  module.decoder.layers.1.mlp.experts.linear_fc2.weight9
 
1145
  module.decoder.layers.1.mlp.experts.linear_fc2.weight12
1146
  module.decoder.layers.1.mlp.experts.linear_fc2.weight14
1147
  module.decoder.layers.1.mlp.experts.linear_fc1.weight1
 
 
1148
  module.decoder.layers.1.mlp.experts.linear_fc1.weight0
1149
+ module.decoder.layers.1.mlp.experts.linear_fc2.weight13
1150
+ module.decoder.layers.1.mlp.experts.linear_fc2.weight15
1151
  Params for bucket 67 (50331648 elements, 50331648 padded size):
1152
  module.decoder.layers.1.mlp.experts.linear_fc1.weight2
1153
+ module.decoder.layers.1.mlp.experts.linear_fc1.weight3
1154
  module.decoder.layers.1.mlp.experts.linear_fc1.weight4
1155
  module.decoder.layers.1.mlp.experts.linear_fc1.weight5
 
1156
  Params for bucket 68 (50331648 elements, 50331648 padded size):
1157
  module.decoder.layers.1.mlp.experts.linear_fc1.weight6
1158
  module.decoder.layers.1.mlp.experts.linear_fc1.weight7
 
1160
  module.decoder.layers.1.mlp.experts.linear_fc1.weight9
1161
  Params for bucket 69 (50331648 elements, 50331648 padded size):
1162
  module.decoder.layers.1.mlp.experts.linear_fc1.weight10
1163
+ module.decoder.layers.1.mlp.experts.linear_fc1.weight13
1164
  module.decoder.layers.1.mlp.experts.linear_fc1.weight11
1165
  module.decoder.layers.1.mlp.experts.linear_fc1.weight12
 
1166
  Params for bucket 70 (44040192 elements, 44040192 padded size):
1167
+ module.decoder.layers.0.mlp.experts.linear_fc2.weight1
1168
  module.decoder.layers.1.mlp.experts.linear_fc1.weight15
 
 
1169
  module.decoder.layers.1.mlp.experts.linear_fc1.weight14
1170
+ module.decoder.layers.0.mlp.experts.linear_fc2.weight0
1171
+ module.decoder.layers.0.mlp.experts.linear_fc2.weight2
1172
  Params for bucket 71 (44040192 elements, 44040192 padded size):
 
 
 
1173
  module.decoder.layers.0.mlp.experts.linear_fc2.weight3
 
1174
  module.decoder.layers.0.mlp.experts.linear_fc2.weight7
1175
+ module.decoder.layers.0.mlp.experts.linear_fc2.weight5
1176
  module.decoder.layers.0.mlp.experts.linear_fc2.weight9
1177
+ module.decoder.layers.0.mlp.experts.linear_fc2.weight6
1178
+ module.decoder.layers.0.mlp.experts.linear_fc2.weight4
1179
+ module.decoder.layers.0.mlp.experts.linear_fc2.weight8
1180
  Params for bucket 72 (50331648 elements, 50331648 padded size):
1181
+ module.decoder.layers.0.mlp.experts.linear_fc2.weight13
1182
+ module.decoder.layers.0.mlp.experts.linear_fc2.weight11
1183
+ module.decoder.layers.0.mlp.experts.linear_fc2.weight15
1184
+ module.decoder.layers.0.mlp.experts.linear_fc1.weight0
1185
  module.decoder.layers.0.mlp.experts.linear_fc2.weight10
1186
  module.decoder.layers.0.mlp.experts.linear_fc2.weight14
1187
  module.decoder.layers.0.mlp.experts.linear_fc2.weight12
 
 
 
 
1188
  Params for bucket 73 (50331648 elements, 50331648 padded size):
1189
  module.decoder.layers.0.mlp.experts.linear_fc1.weight1
 
1190
  module.decoder.layers.0.mlp.experts.linear_fc1.weight3
1191
  module.decoder.layers.0.mlp.experts.linear_fc1.weight2
1192
+ module.decoder.layers.0.mlp.experts.linear_fc1.weight4
1193
  Params for bucket 74 (50331648 elements, 50331648 padded size):
1194
  module.decoder.layers.0.mlp.experts.linear_fc1.weight5
1195
  module.decoder.layers.0.mlp.experts.linear_fc1.weight8
 
1196
  module.decoder.layers.0.mlp.experts.linear_fc1.weight6
1197
+ module.decoder.layers.0.mlp.experts.linear_fc1.weight7
1198
  Params for bucket 75 (50331648 elements, 50331648 padded size):
1199
  module.decoder.layers.0.mlp.experts.linear_fc1.weight9
 
 
1200
  module.decoder.layers.0.mlp.experts.linear_fc1.weight12
1201
+ module.decoder.layers.0.mlp.experts.linear_fc1.weight10
1202
+ module.decoder.layers.0.mlp.experts.linear_fc1.weight11
1203
  Params for bucket 76 (37748736 elements, 37748736 padded size):
1204
  module.decoder.layers.0.mlp.experts.linear_fc1.weight13
1205
  module.decoder.layers.0.mlp.experts.linear_fc1.weight14
1206
  module.decoder.layers.0.mlp.experts.linear_fc1.weight15
1207
+ INFO:megatron.core.optimizer:Setting up optimizer with config OptimizerConfig(optimizer='adam', lr=0.0001, min_lr=1e-05, decoupled_lr=None, decoupled_min_lr=None, weight_decay=0.1, fp8_recipe='mxfp8', fp16=False, bf16=True, reuse_grad_buf_for_mxfp8_param_ag=False, params_dtype=torch.bfloat16, use_precision_aware_optimizer=False, store_param_remainders=True, main_grads_dtype=torch.float32, main_params_dtype=torch.float32, exp_avg_dtype=torch.float32, exp_avg_sq_dtype=torch.float32, loss_scale=None, initial_loss_scale=4294967296, min_loss_scale=1.0, loss_scale_window=1000, hysteresis=2, adam_beta1=0.9, adam_beta2=0.999, adam_eps=1e-08, sgd_momentum=0.9, use_distributed_optimizer=True, overlap_param_gather_with_optimizer_step=False, optimizer_cpu_offload=False, optimizer_offload_fraction=1.0, use_torch_optimizer_for_cpu_offload=False, overlap_cpu_optimizer_d2h_h2d=False, pin_cpu_grads=True, pin_cpu_params=True, clip_grad=1.0, log_num_zeros_in_grad=False, barrier_with_L1_time=True, timers=<megatron.core.timers.Timers object at 0x790345189280>, config_logger_dir='')
1208
  INFO:megatron.core.optimizer_param_scheduler:> learning rate decay style: cosine
1209
+ [after model, optimizer, and learning rate scheduler are built] datetime: 2026-08-30 01:31:46
1210
  > building train, validation, and test datasets ...
1211
  > datasets target sizes (minimum size):
1212
  train: 96
 
1214
  test: 0
1215
  INFO:megatron.core.datasets.blended_megatron_dataset_config:Let split_matrix = [(0, 1.0), None, None]
1216
  > building train, validation, and test datasets for GPT ...
1217
+ INFO:megatron.core.datasets.blended_megatron_dataset_builder:Building GPTDataset splits with sizes=(96, 0, 0) and config=GPTDatasetConfig(random_seed=1234, sequence_length=8192, blend=(['/workspace/data/wikitext/wikitext103_gpt2_text_document'], None), blend_per_split=None, split='100,0,0', split_matrix=[(0, 1.0), None, None], num_dataset_builder_threads=1, path_to_cache=None, mmap_bin_files=True, mock=False, tokenizer=<megatron.training.tokenizer.tokenizer._GPT2BPETokenizer object at 0x79009ea156a0>, mid_level_dataset_surplus=0.005, reset_position_ids=False, reset_attention_mask=False, eod_mask_loss=False, create_attention_mask=True, drop_last_partial_validation_sequence=True, add_extra_token_to_sequence=True, object_storage_cache_path=None)
1218
  INFO:megatron.core.datasets.indexed_dataset:Load the _IndexReader from /workspace/data/wikitext/wikitext103_gpt2_text_document.idx
1219
  INFO:megatron.core.datasets.indexed_dataset: Extract the sequence lengths
1220
  INFO:megatron.core.datasets.indexed_dataset: Extract the sequence pointers
 
1227
  INFO:megatron.core.datasets.gpt_dataset: Load the shuffle index from eadc3756447e39d609e8aa728a91f451-GPTDataset-train-shuffle_index.npy
1228
  INFO:megatron.core.datasets.gpt_dataset:> total number of samples: 14536
1229
  > finished creating GPT datasets ...
1230
+ [after dataloaders are built] datetime: 2026-08-30 01:31:46
1231
  done with setup ...
 
 
 
1232
  training ...
1233
  Overwriting rerun_state_machine.current_iteration from -1 to 0...
1234
+ (min, max) time across ranks (ms):
1235
+ model-and-optimizer-setup ......................: (3305.74, 3311.41)
1236
+ train/valid/test-data-iterators-setup ..........: (66.88, 70.10)
1237
+ [before the start of training step] datetime: 2026-08-30 01:31:46
1238
  INFO:moonep.api:MoonEP comm buffer sizes:
1239
  config : R=8, S=8192, H=4096, E=128, K=8, NvS=69600, NvS_padded=69632
1240
  sm_count : num_sms=32, num_sms_dedup=148
 
1246
  [MoonEP selfcheck 2/4] dropped groups empty OK | covered rows [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 128, 129, 130, 131, 132, 133, 134, 135, 136, 137, 138, 139, 140, 141, 142, 143] sizes [9344, 2688, 128, 24192, 128, 0, 128, 256, 128, 768, 10496, 128, 128, 2432, 128, 1536, 14208, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0] | duplicated experts [[69, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1], [88, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1], [55, 52, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1], [-1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1], [63, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1], [124, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1], [10, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1], [58, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1]]
1247
  [MoonEP selfcheck 3/4] dropped groups empty OK | covered rows [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 128, 129, 130, 131, 132, 133, 134, 135, 136, 137, 138, 139, 140, 141, 142, 143] sizes [128, 128, 128, 0, 128, 44416, 128, 128, 128, 6912, 128, 256, 128, 128, 128, 128, 13824, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0] | duplicated experts [[78, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1], [111, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1], [62, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1], [-1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1], [57, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1], [106, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1], [121, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1], [57, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1]]
1248
  [MoonEP selfcheck 4/4] dropped groups empty OK | covered rows [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 128, 129, 130, 131, 132, 133, 134, 135, 136, 137, 138, 139, 140, 141, 142, 143] sizes [2304, 128, 128, 512, 128, 12416, 128, 1152, 1280, 16768, 640, 29056, 128, 128, 128, 1664, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0] | duplicated experts [[-1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1], [118, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1], [11, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1], [118, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1], [39, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1], [11, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1], [73, 79, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1], [86, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1]]
1249
+ [2026-08-30 01:32:08] iteration 1/ 12 | consumed samples: 8 | elapsed time per iteration (ms): 21390.2 | throughput per GPU (TFLOP/s/GPU): 8.5 | learning rate: 2.000000E-05 | global batch size: 8 | lm loss: 1.115993E+01 | loss scale: 1.0 | grad norm: 44.807 | number of skipped iterations: 0 | number of nan iterations: 0 |
1250
  Number of parameters in transformer block in billions: 29.85
1251
  Number of parameters in embedding layers in billions: 0.41
1252
  Total number of parameters in billions: 30.26
1253
  Number of parameters in most loaded shard in billions: 30.2590
1254
  Theoretical memory footprints: weight and optimizer=216428.96 MB
 
1255
  [Rank 0] (after 1 iterations) memory (MB) | allocated: 67239.548828125 | max allocated: 67239.54931640625 | reserved: 68548.0 | max reserved: 68548.0
1256
+ [2026-08-30 01:32:08] iteration 2/ 12 | consumed samples: 16 | elapsed time per iteration (ms): 361.6 | throughput per GPU (TFLOP/s/GPU): 500.1 | learning rate: 4.000000E-05 | global batch size: 8 | lm loss: 1.117600E+01 | loss scale: 1.0 | grad norm: 44.337 | number of skipped iterations: 0 | number of nan iterations: 0 |
1257
+ [2026-08-30 01:32:08] iteration 3/ 12 | consumed samples: 24 | elapsed time per iteration (ms): 332.0 | throughput per GPU (TFLOP/s/GPU): 544.7 | learning rate: 6.000000E-05 | global batch size: 8 | lm loss: 9.685843E+00 | loss scale: 1.0 | grad norm: 62.891 | number of skipped iterations: 0 | number of nan iterations: 0 |
1258
  Capture CUDA graph for training!!!
1259
  CUDA graph capture done!!!
1260
+ [2026-08-30 01:32:09] iteration 4/ 12 | consumed samples: 32 | elapsed time per iteration (ms): 945.4 | throughput per GPU (TFLOP/s/GPU): 191.3 | learning rate: 8.000000E-05 | global batch size: 8 | lm loss: 1.338080E+01 | loss scale: 1.0 | grad norm: 20.297 | number of skipped iterations: 0 | number of nan iterations: 0 |
1261
+ [2026-08-30 01:32:10] iteration 5/ 12 | consumed samples: 40 | elapsed time per iteration (ms): 319.2 | throughput per GPU (TFLOP/s/GPU): 566.6 | learning rate: 1.000000E-04 | global batch size: 8 | lm loss: 1.049032E+01 | loss scale: 1.0 | grad norm: 95.133 | number of skipped iterations: 0 | number of nan iterations: 0 |
1262
+ [2026-08-30 01:32:10] iteration 6/ 12 | consumed samples: 48 | elapsed time per iteration (ms): 317.9 | throughput per GPU (TFLOP/s/GPU): 568.7 | learning rate: 7.750000E-05 | global batch size: 8 | lm loss: 1.049771E+01 | loss scale: 1.0 | grad norm: 55.144 | number of skipped iterations: 0 | number of nan iterations: 0 |
1263
+ [2026-08-30 01:32:10] iteration 7/ 12 | consumed samples: 56 | elapsed time per iteration (ms): 316.7 | throughput per GPU (TFLOP/s/GPU): 571.0 | learning rate: 3.250000E-05 | global batch size: 8 | lm loss: 1.027093E+01 | loss scale: 1.0 | grad norm: 6.340 | number of skipped iterations: 0 | number of nan iterations: 0 |
1264
+ [2026-08-30 01:32:11] iteration 8/ 12 | consumed samples: 64 | elapsed time per iteration (ms): 317.5 | throughput per GPU (TFLOP/s/GPU): 569.6 | learning rate: 1.000000E-05 | global batch size: 8 | lm loss: 9.810117E+00 | loss scale: 1.0 | grad norm: 5.519 | number of skipped iterations: 0 | number of nan iterations: 0 |
1265
+ [2026-08-30 01:32:11] iteration 9/ 12 | consumed samples: 72 | elapsed time per iteration (ms): 316.4 | throughput per GPU (TFLOP/s/GPU): 571.5 | learning rate: 1.000000E-05 | global batch size: 8 | lm loss: 9.549583E+00 | loss scale: 1.0 | grad norm: 5.211 | number of skipped iterations: 0 | number of nan iterations: 0 |
1266
+ [2026-08-30 01:32:11] iteration 10/ 12 | consumed samples: 80 | elapsed time per iteration (ms): 318.1 | throughput per GPU (TFLOP/s/GPU): 568.4 | learning rate: 1.000000E-05 | global batch size: 8 | lm loss: 9.450918E+00 | loss scale: 1.0 | grad norm: 4.911 | number of skipped iterations: 0 | number of nan iterations: 0 |
1267
+ [2026-08-30 01:32:12] iteration 11/ 12 | consumed samples: 88 | elapsed time per iteration (ms): 317.1 | throughput per GPU (TFLOP/s/GPU): 570.2 | learning rate: 1.000000E-05 | global batch size: 8 | lm loss: 9.332552E+00 | loss scale: 1.0 | grad norm: 4.618 | number of skipped iterations: 0 | number of nan iterations: 0 |
1268
+ [2026-08-30 01:32:12] iteration 12/ 12 | consumed samples: 96 | elapsed time per iteration (ms): 316.9 | throughput per GPU (TFLOP/s/GPU): 570.6 | learning rate: 1.000000E-05 | global batch size: 8 | lm loss: 9.219189E+00 | loss scale: 1.0 | grad norm: 4.492 | number of skipped iterations: 0 | number of nan iterations: 0 |
1269
+ [after training is done] datetime: 2026-08-30 01:32:12
1270
+ [rank0]:[W830 01:32:13.801912097 ProcessGroupNCCL.cpp:1505] Warning: WARNING: destroy_process_group() was not called before program exit, which can leak resources. For more info, please see https://pytorch.org/docs/stable/distributed.html#shutdown (function operator())
1271
+ [rank3]:[W830 01:32:14.885010634 ProcessGroupNCCL.cpp:1505] Warning: WARNING: destroy_process_group() was not called before program exit, which can leak resources. For more info, please see https://pytorch.org/docs/stable/distributed.html#shutdown (function operator())
1272
+ [rank6]:[W830 01:32:14.949664138 ProcessGroupNCCL.cpp:1505] Warning: WARNING: destroy_process_group() was not called before program exit, which can leak resources. For more info, please see https://pytorch.org/docs/stable/distributed.html#shutdown (function operator())
1273
+ [rank5]:[W830 01:32:14.003043426 ProcessGroupNCCL.cpp:1505] Warning: WARNING: destroy_process_group() was not called before program exit, which can leak resources. For more info, please see https://pytorch.org/docs/stable/distributed.html#shutdown (function operator())
1274
+ [rank2]:[W830 01:32:14.056089290 ProcessGroupNCCL.cpp:1505] Warning: WARNING: destroy_process_group() was not called before program exit, which can leak resources. For more info, please see https://pytorch.org/docs/stable/distributed.html#shutdown (function operator())
1275
+ [rank7]:[W830 01:32:15.487751536 ProcessGroupNCCL.cpp:1505] Warning: WARNING: destroy_process_group() was not called before program exit, which can leak resources. For more info, please see https://pytorch.org/docs/stable/distributed.html#shutdown (function operator())
1276
+ [rank4]:[W830 01:32:15.505877593 ProcessGroupNCCL.cpp:1505] Warning: WARNING: destroy_process_group() was not called before program exit, which can leak resources. For more info, please see https://pytorch.org/docs/stable/distributed.html#shutdown (function operator())
1277
+ [rank1]:[W830 01:32:15.558402976 ProcessGroupNCCL.cpp:1505] Warning: WARNING: destroy_process_group() was not called before program exit, which can leak resources. For more info, please see https://pytorch.org/docs/stable/distributed.html#shutdown (function operator())
1278
+ W0830 01:32:19.433000 380220 torch/distributed/elastic/agent/server/api.py:719] Received 15 death signal, shutting down workers
1279
+ W0830 01:32:19.433000 380220 torch/distributed/elastic/multiprocessing/api.py:900] Sending process 380405 closing signal SIGTERM
1280
+ W0830 01:32:19.434000 380220 torch/distributed/elastic/multiprocessing/api.py:900] Sending process 380406 closing signal SIGTERM
1281
+ W0830 01:32:19.434000 380220 torch/distributed/elastic/multiprocessing/api.py:900] Sending process 380407 closing signal SIGTERM
1282
+ W0830 01:32:19.434000 380220 torch/distributed/elastic/multiprocessing/api.py:900] Sending process 380408 closing signal SIGTERM
1283
+ W0830 01:32:19.434000 380220 torch/distributed/elastic/multiprocessing/api.py:900] Sending process 380409 closing signal SIGTERM
1284
+ W0830 01:32:19.435000 380220 torch/distributed/elastic/multiprocessing/api.py:900] Sending process 380410 closing signal SIGTERM
1285
+ W0830 01:32:19.435000 380220 torch/distributed/elastic/multiprocessing/api.py:900] Sending process 380411 closing signal SIGTERM
 
1286
  Traceback (most recent call last):
1287
  File "/usr/local/bin/torchrun", line 8, in <module>
1288
  sys.exit(main())
 
1310
  time.sleep(monitor_interval)
1311
  File "/usr/local/lib/python3.12/dist-packages/torch/distributed/elastic/multiprocessing/api.py", line 84, in _terminate_process_handler
1312
  raise SignalException(f"Process {os.getpid()} got signal: {sigval}", sigval=sigval)
1313
+ torch.distributed.elastic.multiprocessing.api.SignalException: Process 380220 got signal: 15
tierb-fixed-moonep-full/moonep/tierb/tierb_gbs8/QWEN3_235B__moonep__matched__seq8192__gbs8__capnone__async/rep2/mem_samples.csv CHANGED
@@ -27,7 +27,7 @@
27
  2, 1
28
  3, 1
29
  4, 1
30
- 5, 4
31
  6, 4
32
  7, 4
33
  0, 0
@@ -78,52 +78,52 @@
78
  5, 821
79
  6, 821
80
  7, 821
81
- 0, 1975
82
- 1, 1975
83
- 2, 1975
84
- 3, 1975
85
- 4, 1975
86
- 5, 1975
87
- 6, 1975
88
- 7, 1975
89
- 0, 5201
90
- 1, 4985
91
- 2, 4913
92
- 3, 5249
93
- 4, 5297
94
- 5, 5417
95
- 6, 4913
96
- 7, 5153
97
- 0, 47099
98
- 1, 47099
99
- 2, 47099
100
- 3, 47099
101
- 4, 47099
102
- 5, 47099
103
- 6, 47099
104
- 7, 47099
105
- 0, 56813
106
- 1, 56811
107
- 2, 56807
108
- 3, 56801
109
- 4, 56569
110
- 5, 56851
111
- 6, 56843
112
- 7, 56875
113
- 0, 58799
114
- 1, 58797
115
- 2, 58799
116
- 3, 58797
117
- 4, 58511
118
- 5, 58797
119
- 6, 58797
120
- 7, 58799
121
  0, 59733
122
- 1, 59731
123
- 2, 59733
124
  3, 59731
125
  4, 59445
126
- 5, 59731
127
  6, 59731
128
  7, 59733
129
  0, 59733
@@ -132,16 +132,24 @@
132
  3, 59733
133
  4, 59445
134
  5, 59733
135
- 6, 59731
136
  7, 59733
137
- 0, 86985
138
- 1, 87025
139
- 2, 86961
140
- 3, 87025
141
- 4, 86985
142
- 5, 87025
143
  6, 86961
144
- 7, 87025
 
 
 
 
 
 
 
 
145
  0, 87633
146
  1, 87673
147
  2, 87673
@@ -150,74 +158,66 @@
150
  5, 87673
151
  6, 87673
152
  7, 87673
153
- 0, 87635
154
- 1, 87675
155
- 2, 87675
156
- 3, 87677
157
- 4, 87635
158
- 5, 87675
159
- 6, 87675
160
- 7, 87675
161
- 0, 88685
162
- 1, 88725
163
- 2, 88725
164
- 3, 88725
165
- 4, 88685
166
- 5, 88725
167
- 6, 88725
168
- 7, 88725
169
- 0, 97521
170
- 1, 97547
171
- 2, 97525
172
- 3, 97525
173
- 4, 97439
174
- 5, 97651
175
- 6, 97549
176
- 7, 97607
177
- 0, 120267
178
- 1, 120303
179
- 2, 120303
180
- 3, 120303
181
- 4, 120303
182
- 5, 120303
183
- 6, 120303
184
- 7, 120303
185
- 0, 120267
186
- 1, 120303
187
- 2, 120303
188
- 3, 120303
189
- 4, 120303
190
- 5, 120303
191
- 6, 120303
192
- 7, 120303
193
- 0, 119621
194
- 1, 119657
195
- 2, 119657
196
- 3, 119657
197
- 4, 119657
198
- 5, 119657
199
- 6, 119657
200
- 7, 119657
201
- 0, 117395
202
- 1, 117431
203
- 2, 117819
204
- 3, 117431
205
- 4, 117431
206
- 5, 117819
207
- 6, 117431
208
- 7, 117819
209
- 0, 114307
210
- 1, 114087
211
- 2, 114669
212
- 3, 114343
213
- 4, 114087
214
- 5, 114669
215
- 6, 114407
216
- 7, 114669
217
  0, 1
218
- 1, 68
219
- 2, 114667
220
- 3, 324
221
  4, 4
222
  5, 4
223
  6, 4
 
27
  2, 1
28
  3, 1
29
  4, 1
30
+ 5, 1
31
  6, 4
32
  7, 4
33
  0, 0
 
78
  5, 821
79
  6, 821
80
  7, 821
81
+ 0, 1537
82
+ 1, 1537
83
+ 2, 1537
84
+ 3, 1537
85
+ 4, 1537
86
+ 5, 1537
87
+ 6, 1537
88
+ 7, 1537
89
+ 0, 2551
90
+ 1, 2551
91
+ 2, 2551
92
+ 3, 1783
93
+ 4, 2551
94
+ 5, 1783
95
+ 6, 2551
96
+ 7, 2551
97
+ 0, 46595
98
+ 1, 46609
99
+ 2, 46605
100
+ 3, 46633
101
+ 4, 46595
102
+ 5, 46603
103
+ 6, 46605
104
+ 7, 46571
105
+ 0, 56769
106
+ 1, 56737
107
+ 2, 56805
108
+ 3, 56785
109
+ 4, 56561
110
+ 5, 56801
111
+ 6, 56851
112
+ 7, 56831
113
+ 0, 58731
114
+ 1, 58731
115
+ 2, 58727
116
+ 3, 58747
117
+ 4, 58469
118
+ 5, 58751
119
+ 6, 58751
120
+ 7, 58747
121
  0, 59733
122
+ 1, 59733
123
+ 2, 59731
124
  3, 59731
125
  4, 59445
126
+ 5, 59733
127
  6, 59731
128
  7, 59733
129
  0, 59733
 
132
  3, 59733
133
  4, 59445
134
  5, 59733
135
+ 6, 59733
136
  7, 59733
137
+ 0, 85347
138
+ 1, 85387
139
+ 2, 85387
140
+ 3, 85387
141
+ 4, 85347
142
+ 5, 85387
143
  6, 86961
144
+ 7, 86961
145
+ 0, 87633
146
+ 1, 87673
147
+ 2, 87673
148
+ 3, 87673
149
+ 4, 87633
150
+ 5, 87673
151
+ 6, 87673
152
+ 7, 87673
153
  0, 87633
154
  1, 87673
155
  2, 87673
 
158
  5, 87673
159
  6, 87673
160
  7, 87673
161
+ 0, 88091
162
+ 1, 88131
163
+ 2, 88129
164
+ 3, 88143
165
+ 4, 88091
166
+ 5, 88153
167
+ 6, 88149
168
+ 7, 88145
169
+ 0, 97523
170
+ 1, 97533
171
+ 2, 97555
172
+ 3, 97541
173
+ 4, 97557
174
+ 5, 97583
175
+ 6, 97567
176
+ 7, 97643
177
+ 0, 120273
178
+ 1, 120309
179
+ 2, 120309
180
+ 3, 120309
181
+ 4, 120309
182
+ 5, 120309
183
+ 6, 120309
184
+ 7, 120309
185
+ 0, 120273
186
+ 1, 120309
187
+ 2, 120309
188
+ 3, 120309
189
+ 4, 120309
190
+ 5, 120309
191
+ 6, 120309
192
+ 7, 120309
193
+ 0, 119627
194
+ 1, 119663
195
+ 2, 119663
196
+ 3, 119663
197
+ 4, 119663
198
+ 5, 119663
199
+ 6, 119663
200
+ 7, 119663
201
+ 0, 117401
202
+ 1, 117825
203
+ 2, 117437
204
+ 3, 117437
205
+ 4, 117437
206
+ 5, 117437
207
+ 6, 117437
208
+ 7, 117825
209
+ 0, 92869
210
+ 1, 115757
211
+ 2, 115757
212
+ 3, 92905
213
+ 4, 115757
214
+ 5, 115757
215
+ 6, 94633
216
+ 7, 115757
 
 
 
 
 
 
 
 
217
  0, 1
218
+ 1, 113926
219
+ 2, 4
220
+ 3, 4
221
  4, 4
222
  5, 4
223
  6, 4
tierb-fixed-moonep-full/moonep/tierb/tierb_gbs8/QWEN3_235B__moonep__matched__seq8192__gbs8__capnone__async/rep2/run.env CHANGED
@@ -3,7 +3,7 @@ config=QWEN3_235B
3
  family=tierB
4
  repo=/workspace/Megatron-MoE-Echo
5
  branch=moonep_backend_async_grad_reduce
6
- commit=7bc235bf
7
  moonep_source_dir=/workspace/MoonEP
8
  moonep_source_branch=megatron-ddp-symmetric-params
9
  moonep_source_commit=d8bc666
 
3
  family=tierB
4
  repo=/workspace/Megatron-MoE-Echo
5
  branch=moonep_backend_async_grad_reduce
6
+ commit=2e68c8ad
7
  moonep_source_dir=/workspace/MoonEP
8
  moonep_source_branch=megatron-ddp-symmetric-params
9
  moonep_source_commit=d8bc666
tierb-fixed-moonep-full/moonep/tierb/tierb_gbs8/QWEN3_235B__moonep__matched__seq8192__gbs8__capnone__async/rep2/train.log CHANGED
@@ -1,25 +1,23 @@
1
- [23:46:17] WARN: moonep_backend_async_grad_reduce lacks --moe-router-fusion; QWEN3_235B runs unfused (timing only, routing identical)
2
  ==============================================================
3
  QWEN3_235B / moonep (slots/rank=16 policy=matched)
4
  E=128 K=8 H'=1536 m=16
5
  L=12 seq=8192 gbs=8 (1 microbatches)
6
  cap_mode=none cap=none cg=1 recompute=none
7
  memory_dump=none snapshots=/workspace/tierb-fixed-moonep-full/moonep/tierb/tierb_gbs8/QWEN3_235B__moonep__matched__seq8192__gbs8__capnone__async/rep2/memory_snapshots/snapshot.pickle
8
- repo=/workspace/Megatron-MoE-Echo @ 7bc235bf
9
  ==============================================================
10
- W0829 23:46:19.086000 240032 torch/distributed/run.py:766]
11
- W0829 23:46:19.086000 240032 torch/distributed/run.py:766] *****************************************
12
- W0829 23:46:19.086000 240032 torch/distributed/run.py:766] Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed.
13
- W0829 23:46:19.086000 240032 torch/distributed/run.py:766] *****************************************
14
  /workspace/Megatron-MoE-Echo/megatron/training/arguments.py:831: UserWarning: Disabling sequence parallelism because tensor model parallelism is disabled
15
  warnings.warn("Disabling sequence parallelism because tensor model parallelism is disabled")
16
- /workspace/Megatron-MoE-Echo/megatron/training/arguments.py:831: UserWarning: Disabling sequence parallelism because tensor model parallelism is disabled
17
- warnings.warn("Disabling sequence parallelism because tensor model parallelism is disabled")
18
- WARNING: one_logger package is required to enable e2e metrics tracking. please go to https://confluence.nvidia.com/display/MLWFO/Package+Repositories for details to install it
19
  /workspace/Megatron-MoE-Echo/megatron/training/arguments.py:831: UserWarning: Disabling sequence parallelism because tensor model parallelism is disabled
20
  warnings.warn("Disabling sequence parallelism because tensor model parallelism is disabled")
21
  /workspace/Megatron-MoE-Echo/megatron/training/arguments.py:831: UserWarning: Disabling sequence parallelism because tensor model parallelism is disabled
22
  warnings.warn("Disabling sequence parallelism because tensor model parallelism is disabled")
 
23
  /workspace/Megatron-MoE-Echo/megatron/training/arguments.py:831: UserWarning: Disabling sequence parallelism because tensor model parallelism is disabled
24
  warnings.warn("Disabling sequence parallelism because tensor model parallelism is disabled")
25
  using world size: 8, data-parallel size: 8, context-parallel size: 1, hierarchical context-parallel sizes: None, tensor-model-parallel size: 1, pipeline-model-parallel size: 1
@@ -577,51 +575,53 @@ WARNING:megatron.core.rerun_state_machine:RerunStateMachine initialized in mode
577
  warnings.warn("Disabling sequence parallelism because tensor model parallelism is disabled")
578
  /workspace/Megatron-MoE-Echo/megatron/training/arguments.py:831: UserWarning: Disabling sequence parallelism because tensor model parallelism is disabled
579
  warnings.warn("Disabling sequence parallelism because tensor model parallelism is disabled")
580
- [Gloo] Rank [Gloo] Rank 20 is connected to is connected to 77 peer ranks. peer ranks. Expected number of connected peer ranks is : Expected number of connected peer ranks is : 77
581
-
582
- [Gloo] Rank 1 is connected to 7 peer ranks. Expected number of connected peer ranks is : 7
583
  [Gloo] Rank 4 is connected to 7 peer ranks. Expected number of connected peer ranks is : 7
584
- [Gloo] Rank [Gloo] Rank [Gloo] Rank [Gloo] Rank 3657 is connected to is connected to is connected to is connected to 7777 peer ranks. peer ranks. peer ranks. peer ranks. Expected number of connected peer ranks is : Expected number of connected peer ranks is : Expected number of connected peer ranks is : Expected number of connected peer ranks is : 7777
585
-
586
-
587
-
588
  [Gloo] Rank 0 is connected to 7 peer ranks. Expected number of connected peer ranks is : 7
 
589
  [Gloo] Rank 2 is connected to 7 peer ranks. Expected number of connected peer ranks is : 7
 
 
590
  [Gloo] Rank 5 is connected to 7 peer ranks. Expected number of connected peer ranks is : 7
 
591
  [Gloo] Rank 3 is connected to 7 peer ranks. Expected number of connected peer ranks is : 7
 
592
  [Gloo] Rank 1 is connected to 7 peer ranks. Expected number of connected peer ranks is : 7
593
- [Gloo] Rank 4 is connected to 7 peer ranks. Expected number of connected peer ranks is : 7
594
- [Gloo] Rank [Gloo] Rank 7 is connected to 67 is connected to peer ranks. 7Expected number of connected peer ranks is : peer ranks. 7Expected number of connected peer ranks is :
595
- 7
 
 
596
  [Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
597
  > initialized tensor model parallel with size 1
598
  > initialized pipeline model parallel with size 1
599
  > setting random seeds to 1234 ...
 
 
 
 
600
  [Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
601
  [Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
602
  [Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
603
- [Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
604
- [Gloo] Rank [Gloo] Rank 00 is connected to is connected to 00 peer ranks. peer ranks. Expected number of connected peer ranks is : Expected number of connected peer ranks is : 00
605
-
606
- [Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
607
- [rank4]:[W829 23:46:37.521085384 ProcessGroupNCCL.cpp:4789] [PG ID 0 PG GUID 0 Rank 4] using GPU 4 as device used by this process is currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect. You can specify device_id in init_process_group() to force use of a particular device.
608
- [rank5]:[W829 23:46:37.521095894 ProcessGroupNCCL.cpp:4789] [PG ID 0 PG GUID 0 Rank 5] using GPU 5 as device used by this process is currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect. You can specify device_id in init_process_group() to force use of a particular device.
609
- [rank2]:[W829 23:46:37.521112608 ProcessGroupNCCL.cpp:4789] [PG ID 0 PG GUID 0 Rank 2] using GPU 2 as device used by this process is currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect. You can specify device_id in init_process_group() to force use of a particular device.
610
- [rank1]:[W829 23:46:37.521143928 ProcessGroupNCCL.cpp:4789] [PG ID 0 PG GUID 0 Rank 1] using GPU 1 as device used by this process is currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect. You can specify device_id in init_process_group() to force use of a particular device.
611
- [rank6]:[W829 23:46:37.522800573 ProcessGroupNCCL.cpp:4789] [PG ID 0 PG GUID 0 Rank 6] using GPU 6 as device used by this process is currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect. You can specify device_id in init_process_group() to force use of a particular device.
612
- [rank7]:[W829 23:46:37.522847520 ProcessGroupNCCL.cpp:4789] [PG ID 0 PG GUID 0 Rank 7] using GPU 7 as device used by this process is currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect. You can specify device_id in init_process_group() to force use of a particular device.
613
- [rank3]:[W829 23:46:37.522872548 ProcessGroupNCCL.cpp:4789] [PG ID 0 PG GUID 0 Rank 3] using GPU 3 as device used by this process is currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect. You can specify device_id in init_process_group() to force use of a particular device.
614
  > compiling dataset index builder ...
615
  make: Entering directory '/workspace/Megatron-MoE-Echo/megatron/core/datasets'
616
  make: Nothing to be done for 'default'.
617
  make: Leaving directory '/workspace/Megatron-MoE-Echo/megatron/core/datasets'
618
- >>> done with dataset index builder. Compilation time: 0.035 seconds
619
  WARNING: constraints for invoking optimized fused softmax kernel are not met. We default back to unfused kernel invocations.
620
  > compiling and loading fused kernels ...
621
- [rank0]:[W829 23:46:37.680689063 ProcessGroupNCCL.cpp:4789] [PG ID 0 PG GUID 0 Rank 0] using GPU 0 as device used by this process is currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect. You can specify device_id in init_process_group() to force use of a particular device.
622
- >>> done with compiling and loading fused kernels. Compilation time: 3.980 seconds
623
- time to initialize megatron (seconds): 7.571
624
- [after megatron is initialized] datetime: 2026-08-29 23:46:43
625
  building GPT model ...
626
  /usr/local/lib/python3.12/dist-packages/transformer_engine/pytorch/cpu_offload.py:708: DeprecationWarning: Offloading weights is deprecated. Using offload_weights=True does not have any effect.
627
  warnings.warn(
@@ -645,43 +645,43 @@ INFO:megatron.core.distributed.param_and_grad_buffer:Number of buckets for gradi
645
  Params for bucket 1 (206045184 elements, 206045184 padded size):
646
  module.output_layer.weight
647
  Params for bucket 2 (71840000 elements, 71840000 padded size):
 
 
648
  module.decoder.layers.11.pre_mlp_layernorm.weight
649
  module.decoder.layers.11.self_attention.q_layernorm.weight
650
- module.decoder.layers.11.mlp.router.weight
651
  module.decoder.layers.11.self_attention.linear_qkv.weight
652
  module.decoder.layers.11.self_attention.linear_qkv.layer_norm_weight
653
- module.decoder.final_layernorm.weight
654
- module.decoder.layers.11.self_attention.k_layernorm.weight
655
  module.decoder.layers.11.self_attention.linear_proj.weight
656
  Params for bucket 3 (71835904 elements, 71835904 padded size):
657
- module.decoder.layers.10.mlp.router.weight
658
  module.decoder.layers.10.pre_mlp_layernorm.weight
659
  module.decoder.layers.10.self_attention.q_layernorm.weight
660
- module.decoder.layers.10.self_attention.linear_qkv.weight
661
  module.decoder.layers.10.self_attention.k_layernorm.weight
 
662
  module.decoder.layers.10.self_attention.linear_qkv.layer_norm_weight
 
663
  module.decoder.layers.10.self_attention.linear_proj.weight
664
  Params for bucket 4 (71835904 elements, 71835904 padded size):
665
- module.decoder.layers.9.mlp.router.weight
666
- module.decoder.layers.9.pre_mlp_layernorm.weight
667
- module.decoder.layers.9.self_attention.q_layernorm.weight
668
- module.decoder.layers.9.self_attention.linear_qkv.layer_norm_weight
669
  module.decoder.layers.9.self_attention.linear_qkv.weight
670
  module.decoder.layers.9.self_attention.k_layernorm.weight
 
671
  module.decoder.layers.9.self_attention.linear_proj.weight
 
 
 
672
  Params for bucket 5 (71835904 elements, 71835904 padded size):
673
  module.decoder.layers.8.mlp.router.weight
 
674
  module.decoder.layers.8.self_attention.k_layernorm.weight
675
  module.decoder.layers.8.self_attention.q_layernorm.weight
676
- module.decoder.layers.8.pre_mlp_layernorm.weight
677
  module.decoder.layers.8.self_attention.linear_qkv.weight
678
  module.decoder.layers.8.self_attention.linear_qkv.layer_norm_weight
679
  module.decoder.layers.8.self_attention.linear_proj.weight
680
  Params for bucket 6 (71835904 elements, 71835904 padded size):
681
  module.decoder.layers.7.mlp.router.weight
 
682
  module.decoder.layers.7.pre_mlp_layernorm.weight
683
  module.decoder.layers.7.self_attention.k_layernorm.weight
684
- module.decoder.layers.7.self_attention.q_layernorm.weight
685
  module.decoder.layers.7.self_attention.linear_qkv.layer_norm_weight
686
  module.decoder.layers.7.self_attention.linear_qkv.weight
687
  module.decoder.layers.7.self_attention.linear_proj.weight
@@ -689,167 +689,167 @@ Params for bucket 7 (71835904 elements, 71835904 padded size):
689
  module.decoder.layers.6.mlp.router.weight
690
  module.decoder.layers.6.self_attention.linear_qkv.weight
691
  module.decoder.layers.6.self_attention.k_layernorm.weight
692
- module.decoder.layers.6.self_attention.linear_qkv.layer_norm_weight
693
  module.decoder.layers.6.pre_mlp_layernorm.weight
694
  module.decoder.layers.6.self_attention.q_layernorm.weight
695
- module.decoder.layers.6.self_attention.linear_proj.weight
696
  Params for bucket 8 (71835904 elements, 71835904 padded size):
697
  module.decoder.layers.5.mlp.router.weight
 
698
  module.decoder.layers.5.self_attention.linear_qkv.weight
699
- module.decoder.layers.5.self_attention.q_layernorm.weight
700
  module.decoder.layers.5.pre_mlp_layernorm.weight
 
701
  module.decoder.layers.5.self_attention.linear_qkv.layer_norm_weight
702
- module.decoder.layers.5.self_attention.k_layernorm.weight
703
  module.decoder.layers.5.self_attention.linear_proj.weight
704
  Params for bucket 9 (71835904 elements, 71835904 padded size):
705
- module.decoder.layers.4.pre_mlp_layernorm.weight
706
- module.decoder.layers.4.self_attention.linear_qkv.weight
707
  module.decoder.layers.4.self_attention.q_layernorm.weight
 
708
  module.decoder.layers.4.self_attention.linear_qkv.layer_norm_weight
709
- module.decoder.layers.4.mlp.router.weight
710
  module.decoder.layers.4.self_attention.k_layernorm.weight
 
711
  module.decoder.layers.4.self_attention.linear_proj.weight
712
  Params for bucket 10 (71835904 elements, 71835904 padded size):
713
  module.decoder.layers.3.mlp.router.weight
714
- module.decoder.layers.3.pre_mlp_layernorm.weight
715
  module.decoder.layers.3.self_attention.q_layernorm.weight
716
  module.decoder.layers.3.self_attention.linear_qkv.weight
 
717
  module.decoder.layers.3.self_attention.k_layernorm.weight
718
  module.decoder.layers.3.self_attention.linear_qkv.layer_norm_weight
719
  module.decoder.layers.3.self_attention.linear_proj.weight
720
  Params for bucket 11 (71835904 elements, 71835904 padded size):
721
  module.decoder.layers.2.mlp.router.weight
 
 
722
  module.decoder.layers.2.pre_mlp_layernorm.weight
723
  module.decoder.layers.2.self_attention.linear_qkv.weight
724
- module.decoder.layers.2.self_attention.q_layernorm.weight
725
- module.decoder.layers.2.self_attention.k_layernorm.weight
726
  module.decoder.layers.2.self_attention.linear_qkv.layer_norm_weight
727
  module.decoder.layers.2.self_attention.linear_proj.weight
728
  Params for bucket 12 (71835904 elements, 71835904 padded size):
729
  module.decoder.layers.1.mlp.router.weight
730
  module.decoder.layers.1.pre_mlp_layernorm.weight
731
- module.decoder.layers.1.self_attention.q_layernorm.weight
732
- module.decoder.layers.1.self_attention.linear_qkv.weight
733
  module.decoder.layers.1.self_attention.k_layernorm.weight
 
 
734
  module.decoder.layers.1.self_attention.linear_qkv.layer_norm_weight
735
  module.decoder.layers.1.self_attention.linear_proj.weight
736
  Params for bucket 13 (71835904 elements, 71835904 padded size):
 
 
737
  module.decoder.layers.0.self_attention.q_layernorm.weight
738
  module.decoder.layers.0.pre_mlp_layernorm.weight
739
- module.decoder.layers.0.self_attention.linear_qkv.layer_norm_weight
740
- module.decoder.layers.0.mlp.router.weight
741
- module.decoder.layers.0.self_attention.linear_qkv.weight
742
- module.decoder.layers.0.self_attention.k_layernorm.weight
743
  module.decoder.layers.0.self_attention.linear_proj.weight
 
 
744
  Params for bucket 14 (206045184 elements, 206045184 padded size):
745
  module.embedding.word_embeddings.weight
746
  INFO:megatron.core.distributed.param_and_grad_buffer:Number of buckets for gradient all-reduce / reduce-scatter: 76
747
  Params for bucket 1 (44040192 elements, 44040192 padded size):
748
- module.decoder.layers.11.mlp.experts.linear_fc2.weight0
749
- module.decoder.layers.11.mlp.experts.linear_fc2.weight2
750
- module.decoder.layers.11.mlp.experts.linear_fc2.weight4
751
- module.decoder.layers.11.mlp.experts.linear_fc2.weight6
752
  module.decoder.layers.11.mlp.experts.linear_fc2.weight1
753
  module.decoder.layers.11.mlp.experts.linear_fc2.weight3
754
  module.decoder.layers.11.mlp.experts.linear_fc2.weight5
 
 
 
 
755
  Params for bucket 2 (44040192 elements, 44040192 padded size):
756
- module.decoder.layers.11.mlp.experts.linear_fc2.weight8
757
- module.decoder.layers.11.mlp.experts.linear_fc2.weight10
758
- module.decoder.layers.11.mlp.experts.linear_fc2.weight12
759
  module.decoder.layers.11.mlp.experts.linear_fc2.weight7
760
  module.decoder.layers.11.mlp.experts.linear_fc2.weight11
761
  module.decoder.layers.11.mlp.experts.linear_fc2.weight9
762
  module.decoder.layers.11.mlp.experts.linear_fc2.weight13
 
 
 
763
  Params for bucket 3 (50331648 elements, 50331648 padded size):
764
- module.decoder.layers.11.mlp.experts.linear_fc2.weight14
765
- module.decoder.layers.11.mlp.experts.linear_fc1.weight1
766
  module.decoder.layers.11.mlp.experts.linear_fc1.weight2
767
  module.decoder.layers.11.mlp.experts.linear_fc2.weight15
768
- module.decoder.layers.11.mlp.experts.linear_fc1.weight0
 
769
  Params for bucket 4 (50331648 elements, 50331648 padded size):
770
  module.decoder.layers.11.mlp.experts.linear_fc1.weight3
771
  module.decoder.layers.11.mlp.experts.linear_fc1.weight4
772
- module.decoder.layers.11.mlp.experts.linear_fc1.weight5
773
  module.decoder.layers.11.mlp.experts.linear_fc1.weight6
 
774
  Params for bucket 5 (50331648 elements, 50331648 padded size):
775
  module.decoder.layers.11.mlp.experts.linear_fc1.weight7
776
- module.decoder.layers.11.mlp.experts.linear_fc1.weight8
777
  module.decoder.layers.11.mlp.experts.linear_fc1.weight10
 
778
  module.decoder.layers.11.mlp.experts.linear_fc1.weight9
779
  Params for bucket 6 (50331648 elements, 50331648 padded size):
780
  module.decoder.layers.11.mlp.experts.linear_fc1.weight11
781
- module.decoder.layers.11.mlp.experts.linear_fc1.weight14
782
  module.decoder.layers.11.mlp.experts.linear_fc1.weight13
 
783
  module.decoder.layers.11.mlp.experts.linear_fc1.weight12
784
  Params for bucket 7 (44040192 elements, 44040192 padded size):
785
- module.decoder.layers.11.mlp.experts.linear_fc1.weight15
786
  module.decoder.layers.10.mlp.experts.linear_fc2.weight3
787
  module.decoder.layers.10.mlp.experts.linear_fc2.weight1
788
- module.decoder.layers.10.mlp.experts.linear_fc2.weight0
789
  module.decoder.layers.10.mlp.experts.linear_fc2.weight2
 
790
  module.decoder.layers.10.mlp.experts.linear_fc2.weight4
791
  Params for bucket 8 (44040192 elements, 44040192 padded size):
792
  module.decoder.layers.10.mlp.experts.linear_fc2.weight5
793
- module.decoder.layers.10.mlp.experts.linear_fc2.weight7
794
  module.decoder.layers.10.mlp.experts.linear_fc2.weight9
 
795
  module.decoder.layers.10.mlp.experts.linear_fc2.weight11
796
- module.decoder.layers.10.mlp.experts.linear_fc2.weight8
797
  module.decoder.layers.10.mlp.experts.linear_fc2.weight6
 
798
  module.decoder.layers.10.mlp.experts.linear_fc2.weight10
799
  Params for bucket 9 (50331648 elements, 50331648 padded size):
800
- module.decoder.layers.10.mlp.experts.linear_fc2.weight13
801
  module.decoder.layers.10.mlp.experts.linear_fc1.weight0
802
  module.decoder.layers.10.mlp.experts.linear_fc2.weight15
 
803
  module.decoder.layers.10.mlp.experts.linear_fc2.weight12
804
  module.decoder.layers.10.mlp.experts.linear_fc2.weight14
805
  module.decoder.layers.10.mlp.experts.linear_fc1.weight1
806
  Params for bucket 10 (50331648 elements, 50331648 padded size):
807
  module.decoder.layers.10.mlp.experts.linear_fc1.weight2
808
- module.decoder.layers.10.mlp.experts.linear_fc1.weight4
809
  module.decoder.layers.10.mlp.experts.linear_fc1.weight5
810
  module.decoder.layers.10.mlp.experts.linear_fc1.weight3
 
811
  Params for bucket 11 (50331648 elements, 50331648 padded size):
812
  module.decoder.layers.10.mlp.experts.linear_fc1.weight6
813
- module.decoder.layers.10.mlp.experts.linear_fc1.weight7
814
  module.decoder.layers.10.mlp.experts.linear_fc1.weight8
815
  module.decoder.layers.10.mlp.experts.linear_fc1.weight9
 
816
  Params for bucket 12 (50331648 elements, 50331648 padded size):
817
  module.decoder.layers.10.mlp.experts.linear_fc1.weight10
 
818
  module.decoder.layers.10.mlp.experts.linear_fc1.weight13
819
  module.decoder.layers.10.mlp.experts.linear_fc1.weight11
820
- module.decoder.layers.10.mlp.experts.linear_fc1.weight12
821
  Params for bucket 13 (44040192 elements, 44040192 padded size):
822
  module.decoder.layers.10.mlp.experts.linear_fc1.weight14
823
- module.decoder.layers.9.mlp.experts.linear_fc2.weight1
824
  module.decoder.layers.9.mlp.experts.linear_fc2.weight2
825
  module.decoder.layers.9.mlp.experts.linear_fc2.weight0
 
826
  module.decoder.layers.10.mlp.experts.linear_fc1.weight15
827
  Params for bucket 14 (44040192 elements, 44040192 padded size):
 
 
 
828
  module.decoder.layers.9.mlp.experts.linear_fc2.weight3
829
  module.decoder.layers.9.mlp.experts.linear_fc2.weight5
830
  module.decoder.layers.9.mlp.experts.linear_fc2.weight7
831
  module.decoder.layers.9.mlp.experts.linear_fc2.weight9
832
- module.decoder.layers.9.mlp.experts.linear_fc2.weight6
833
- module.decoder.layers.9.mlp.experts.linear_fc2.weight4
834
- module.decoder.layers.9.mlp.experts.linear_fc2.weight8
835
  Params for bucket 15 (50331648 elements, 50331648 padded size):
836
- module.decoder.layers.9.mlp.experts.linear_fc2.weight13
 
 
837
  module.decoder.layers.9.mlp.experts.linear_fc2.weight11
 
838
  module.decoder.layers.9.mlp.experts.linear_fc2.weight15
839
  module.decoder.layers.9.mlp.experts.linear_fc1.weight0
840
- module.decoder.layers.9.mlp.experts.linear_fc2.weight10
841
- module.decoder.layers.9.mlp.experts.linear_fc2.weight14
842
- module.decoder.layers.9.mlp.experts.linear_fc2.weight12
843
  Params for bucket 16 (50331648 elements, 50331648 padded size):
844
  module.decoder.layers.9.mlp.experts.linear_fc1.weight1
 
845
  module.decoder.layers.9.mlp.experts.linear_fc1.weight2
846
  module.decoder.layers.9.mlp.experts.linear_fc1.weight3
847
- module.decoder.layers.9.mlp.experts.linear_fc1.weight4
848
  Params for bucket 17 (50331648 elements, 50331648 padded size):
849
  module.decoder.layers.9.mlp.experts.linear_fc1.weight5
 
850
  module.decoder.layers.9.mlp.experts.linear_fc1.weight8
851
  module.decoder.layers.9.mlp.experts.linear_fc1.weight6
852
- module.decoder.layers.9.mlp.experts.linear_fc1.weight7
853
  Params for bucket 18 (50331648 elements, 50331648 padded size):
854
  module.decoder.layers.9.mlp.experts.linear_fc1.weight9
855
  module.decoder.layers.9.mlp.experts.linear_fc1.weight11
@@ -857,81 +857,81 @@ Params for bucket 18 (50331648 elements, 50331648 padded size):
857
  module.decoder.layers.9.mlp.experts.linear_fc1.weight10
858
  Params for bucket 19 (44040192 elements, 44040192 padded size):
859
  module.decoder.layers.9.mlp.experts.linear_fc1.weight13
860
- module.decoder.layers.8.mlp.experts.linear_fc2.weight0
861
  module.decoder.layers.9.mlp.experts.linear_fc1.weight14
862
  module.decoder.layers.9.mlp.experts.linear_fc1.weight15
 
863
  Params for bucket 20 (44040192 elements, 44040192 padded size):
864
- module.decoder.layers.8.mlp.experts.linear_fc2.weight4
865
- module.decoder.layers.8.mlp.experts.linear_fc2.weight2
866
- module.decoder.layers.8.mlp.experts.linear_fc2.weight6
867
  module.decoder.layers.8.mlp.experts.linear_fc2.weight1
868
- module.decoder.layers.8.mlp.experts.linear_fc2.weight5
869
  module.decoder.layers.8.mlp.experts.linear_fc2.weight3
 
870
  module.decoder.layers.8.mlp.experts.linear_fc2.weight7
 
 
 
871
  Params for bucket 21 (44040192 elements, 44040192 padded size):
 
 
 
872
  module.decoder.layers.8.mlp.experts.linear_fc2.weight8
873
  module.decoder.layers.8.mlp.experts.linear_fc2.weight10
874
  module.decoder.layers.8.mlp.experts.linear_fc2.weight12
875
  module.decoder.layers.8.mlp.experts.linear_fc2.weight14
876
- module.decoder.layers.8.mlp.experts.linear_fc2.weight11
877
- module.decoder.layers.8.mlp.experts.linear_fc2.weight9
878
- module.decoder.layers.8.mlp.experts.linear_fc2.weight13
879
  Params for bucket 22 (44040192 elements, 44040192 padded size):
880
  module.decoder.layers.8.mlp.experts.linear_fc2.weight15
881
  module.decoder.layers.8.mlp.experts.linear_fc1.weight1
882
- module.decoder.layers.8.mlp.experts.linear_fc1.weight2
883
  module.decoder.layers.8.mlp.experts.linear_fc1.weight0
 
884
  Params for bucket 23 (50331648 elements, 50331648 padded size):
885
  module.decoder.layers.8.mlp.experts.linear_fc1.weight3
886
- module.decoder.layers.8.mlp.experts.linear_fc1.weight5
887
  module.decoder.layers.8.mlp.experts.linear_fc1.weight4
 
888
  module.decoder.layers.8.mlp.experts.linear_fc1.weight6
889
  Params for bucket 24 (50331648 elements, 50331648 padded size):
890
  module.decoder.layers.8.mlp.experts.linear_fc1.weight7
 
891
  module.decoder.layers.8.mlp.experts.linear_fc1.weight8
892
  module.decoder.layers.8.mlp.experts.linear_fc1.weight9
893
- module.decoder.layers.8.mlp.experts.linear_fc1.weight10
894
  Params for bucket 25 (50331648 elements, 50331648 padded size):
895
  module.decoder.layers.8.mlp.experts.linear_fc1.weight11
896
  module.decoder.layers.8.mlp.experts.linear_fc1.weight14
897
- module.decoder.layers.8.mlp.experts.linear_fc1.weight12
898
  module.decoder.layers.8.mlp.experts.linear_fc1.weight13
 
899
  Params for bucket 26 (44040192 elements, 44040192 padded size):
 
 
900
  module.decoder.layers.8.mlp.experts.linear_fc1.weight15
901
  module.decoder.layers.7.mlp.experts.linear_fc2.weight2
902
  module.decoder.layers.7.mlp.experts.linear_fc2.weight0
903
  module.decoder.layers.7.mlp.experts.linear_fc2.weight4
904
- module.decoder.layers.7.mlp.experts.linear_fc2.weight3
905
- module.decoder.layers.7.mlp.experts.linear_fc2.weight1
906
  Params for bucket 27 (44040192 elements, 44040192 padded size):
907
- module.decoder.layers.7.mlp.experts.linear_fc2.weight8
908
- module.decoder.layers.7.mlp.experts.linear_fc2.weight6
909
- module.decoder.layers.7.mlp.experts.linear_fc2.weight10
910
  module.decoder.layers.7.mlp.experts.linear_fc2.weight5
911
- module.decoder.layers.7.mlp.experts.linear_fc2.weight7
912
  module.decoder.layers.7.mlp.experts.linear_fc2.weight9
 
913
  module.decoder.layers.7.mlp.experts.linear_fc2.weight11
 
 
 
914
  Params for bucket 28 (50331648 elements, 50331648 padded size):
915
- module.decoder.layers.7.mlp.experts.linear_fc2.weight12
916
  module.decoder.layers.7.mlp.experts.linear_fc1.weight0
917
- module.decoder.layers.7.mlp.experts.linear_fc2.weight14
918
  module.decoder.layers.7.mlp.experts.linear_fc2.weight15
919
  module.decoder.layers.7.mlp.experts.linear_fc2.weight13
 
 
920
  module.decoder.layers.7.mlp.experts.linear_fc1.weight1
921
  Params for bucket 29 (50331648 elements, 50331648 padded size):
922
  module.decoder.layers.7.mlp.experts.linear_fc1.weight2
923
- module.decoder.layers.7.mlp.experts.linear_fc1.weight5
924
  module.decoder.layers.7.mlp.experts.linear_fc1.weight3
 
925
  module.decoder.layers.7.mlp.experts.linear_fc1.weight4
926
  Params for bucket 30 (50331648 elements, 50331648 padded size):
927
  module.decoder.layers.7.mlp.experts.linear_fc1.weight6
928
- module.decoder.layers.7.mlp.experts.linear_fc1.weight8
929
  module.decoder.layers.7.mlp.experts.linear_fc1.weight9
 
930
  module.decoder.layers.7.mlp.experts.linear_fc1.weight7
931
  Params for bucket 31 (50331648 elements, 50331648 padded size):
932
  module.decoder.layers.7.mlp.experts.linear_fc1.weight10
933
- module.decoder.layers.7.mlp.experts.linear_fc1.weight13
934
  module.decoder.layers.7.mlp.experts.linear_fc1.weight12
 
935
  module.decoder.layers.7.mlp.experts.linear_fc1.weight11
936
  Params for bucket 32 (44040192 elements, 44040192 padded size):
937
  module.decoder.layers.7.mlp.experts.linear_fc1.weight14
@@ -944,37 +944,37 @@ Params for bucket 33 (44040192 elements, 44040192 padded size):
944
  module.decoder.layers.6.mlp.experts.linear_fc2.weight6
945
  module.decoder.layers.6.mlp.experts.linear_fc2.weight8
946
  module.decoder.layers.6.mlp.experts.linear_fc2.weight3
947
- module.decoder.layers.6.mlp.experts.linear_fc2.weight7
948
  module.decoder.layers.6.mlp.experts.linear_fc2.weight5
 
949
  module.decoder.layers.6.mlp.experts.linear_fc2.weight9
950
  Params for bucket 34 (50331648 elements, 50331648 padded size):
951
  module.decoder.layers.6.mlp.experts.linear_fc2.weight10
952
- module.decoder.layers.6.mlp.experts.linear_fc2.weight12
953
  module.decoder.layers.6.mlp.experts.linear_fc2.weight14
 
954
  module.decoder.layers.6.mlp.experts.linear_fc2.weight11
955
  module.decoder.layers.6.mlp.experts.linear_fc2.weight13
956
  module.decoder.layers.6.mlp.experts.linear_fc2.weight15
957
  module.decoder.layers.6.mlp.experts.linear_fc1.weight0
958
  Params for bucket 35 (50331648 elements, 50331648 padded size):
959
  module.decoder.layers.6.mlp.experts.linear_fc1.weight1
 
960
  module.decoder.layers.6.mlp.experts.linear_fc1.weight3
961
  module.decoder.layers.6.mlp.experts.linear_fc1.weight4
962
- module.decoder.layers.6.mlp.experts.linear_fc1.weight2
963
  Params for bucket 36 (50331648 elements, 50331648 padded size):
964
  module.decoder.layers.6.mlp.experts.linear_fc1.weight5
965
- module.decoder.layers.6.mlp.experts.linear_fc1.weight7
966
  module.decoder.layers.6.mlp.experts.linear_fc1.weight8
967
  module.decoder.layers.6.mlp.experts.linear_fc1.weight6
 
968
  Params for bucket 37 (50331648 elements, 50331648 padded size):
969
  module.decoder.layers.6.mlp.experts.linear_fc1.weight9
970
- module.decoder.layers.6.mlp.experts.linear_fc1.weight10
971
- module.decoder.layers.6.mlp.experts.linear_fc1.weight11
972
  module.decoder.layers.6.mlp.experts.linear_fc1.weight12
 
 
973
  Params for bucket 38 (44040192 elements, 44040192 padded size):
974
  module.decoder.layers.6.mlp.experts.linear_fc1.weight13
 
975
  module.decoder.layers.5.mlp.experts.linear_fc2.weight0
976
  module.decoder.layers.6.mlp.experts.linear_fc1.weight14
977
- module.decoder.layers.6.mlp.experts.linear_fc1.weight15
978
  Params for bucket 39 (44040192 elements, 44040192 padded size):
979
  module.decoder.layers.5.mlp.experts.linear_fc2.weight1
980
  module.decoder.layers.5.mlp.experts.linear_fc2.weight3
@@ -1003,78 +1003,78 @@ Params for bucket 42 (50331648 elements, 50331648 padded size):
1003
  module.decoder.layers.5.mlp.experts.linear_fc1.weight5
1004
  Params for bucket 43 (50331648 elements, 50331648 padded size):
1005
  module.decoder.layers.5.mlp.experts.linear_fc1.weight7
 
1006
  module.decoder.layers.5.mlp.experts.linear_fc1.weight10
1007
  module.decoder.layers.5.mlp.experts.linear_fc1.weight8
1008
- module.decoder.layers.5.mlp.experts.linear_fc1.weight9
1009
  Params for bucket 44 (50331648 elements, 50331648 padded size):
1010
  module.decoder.layers.5.mlp.experts.linear_fc1.weight11
 
1011
  module.decoder.layers.5.mlp.experts.linear_fc1.weight13
1012
  module.decoder.layers.5.mlp.experts.linear_fc1.weight14
1013
- module.decoder.layers.5.mlp.experts.linear_fc1.weight12
1014
  Params for bucket 45 (44040192 elements, 44040192 padded size):
1015
- module.decoder.layers.4.mlp.experts.linear_fc2.weight2
1016
- module.decoder.layers.4.mlp.experts.linear_fc2.weight0
1017
- module.decoder.layers.4.mlp.experts.linear_fc2.weight4
1018
- module.decoder.layers.5.mlp.experts.linear_fc1.weight15
1019
  module.decoder.layers.4.mlp.experts.linear_fc2.weight3
1020
  module.decoder.layers.4.mlp.experts.linear_fc2.weight1
 
 
 
 
1021
  Params for bucket 46 (44040192 elements, 44040192 padded size):
1022
- module.decoder.layers.4.mlp.experts.linear_fc2.weight8
1023
- module.decoder.layers.4.mlp.experts.linear_fc2.weight6
1024
- module.decoder.layers.4.mlp.experts.linear_fc2.weight10
1025
  module.decoder.layers.4.mlp.experts.linear_fc2.weight5
1026
- module.decoder.layers.4.mlp.experts.linear_fc2.weight9
1027
  module.decoder.layers.4.mlp.experts.linear_fc2.weight7
 
1028
  module.decoder.layers.4.mlp.experts.linear_fc2.weight11
 
 
 
1029
  Params for bucket 47 (50331648 elements, 50331648 padded size):
 
 
 
1030
  module.decoder.layers.4.mlp.experts.linear_fc2.weight12
1031
  module.decoder.layers.4.mlp.experts.linear_fc2.weight14
1032
  module.decoder.layers.4.mlp.experts.linear_fc1.weight1
1033
- module.decoder.layers.4.mlp.experts.linear_fc1.weight0
1034
- module.decoder.layers.4.mlp.experts.linear_fc2.weight15
1035
- module.decoder.layers.4.mlp.experts.linear_fc2.weight13
1036
  Params for bucket 48 (50331648 elements, 50331648 padded size):
1037
  module.decoder.layers.4.mlp.experts.linear_fc1.weight2
 
1038
  module.decoder.layers.4.mlp.experts.linear_fc1.weight4
1039
  module.decoder.layers.4.mlp.experts.linear_fc1.weight5
1040
- module.decoder.layers.4.mlp.experts.linear_fc1.weight3
1041
  Params for bucket 49 (50331648 elements, 50331648 padded size):
1042
  module.decoder.layers.4.mlp.experts.linear_fc1.weight6
 
1043
  module.decoder.layers.4.mlp.experts.linear_fc1.weight7
1044
  module.decoder.layers.4.mlp.experts.linear_fc1.weight8
1045
- module.decoder.layers.4.mlp.experts.linear_fc1.weight9
1046
  Params for bucket 50 (50331648 elements, 50331648 padded size):
1047
  module.decoder.layers.4.mlp.experts.linear_fc1.weight10
1048
- module.decoder.layers.4.mlp.experts.linear_fc1.weight11
1049
  module.decoder.layers.4.mlp.experts.linear_fc1.weight12
1050
  module.decoder.layers.4.mlp.experts.linear_fc1.weight13
 
1051
  Params for bucket 51 (44040192 elements, 44040192 padded size):
1052
- module.decoder.layers.3.mlp.experts.linear_fc2.weight1
1053
- module.decoder.layers.4.mlp.experts.linear_fc1.weight15
1054
  module.decoder.layers.4.mlp.experts.linear_fc1.weight14
1055
- module.decoder.layers.3.mlp.experts.linear_fc2.weight2
1056
  module.decoder.layers.3.mlp.experts.linear_fc2.weight0
 
 
 
1057
  Params for bucket 52 (44040192 elements, 44040192 padded size):
 
 
 
1058
  module.decoder.layers.3.mlp.experts.linear_fc2.weight3
1059
  module.decoder.layers.3.mlp.experts.linear_fc2.weight7
1060
  module.decoder.layers.3.mlp.experts.linear_fc2.weight5
1061
  module.decoder.layers.3.mlp.experts.linear_fc2.weight9
1062
- module.decoder.layers.3.mlp.experts.linear_fc2.weight6
1063
- module.decoder.layers.3.mlp.experts.linear_fc2.weight4
1064
- module.decoder.layers.3.mlp.experts.linear_fc2.weight8
1065
  Params for bucket 53 (50331648 elements, 50331648 padded size):
 
 
 
1066
  module.decoder.layers.3.mlp.experts.linear_fc2.weight13
1067
  module.decoder.layers.3.mlp.experts.linear_fc2.weight11
1068
  module.decoder.layers.3.mlp.experts.linear_fc2.weight15
1069
  module.decoder.layers.3.mlp.experts.linear_fc1.weight0
1070
- module.decoder.layers.3.mlp.experts.linear_fc2.weight10
1071
- module.decoder.layers.3.mlp.experts.linear_fc2.weight14
1072
- module.decoder.layers.3.mlp.experts.linear_fc2.weight12
1073
  Params for bucket 54 (50331648 elements, 50331648 padded size):
1074
  module.decoder.layers.3.mlp.experts.linear_fc1.weight1
1075
  module.decoder.layers.3.mlp.experts.linear_fc1.weight4
1076
- module.decoder.layers.3.mlp.experts.linear_fc1.weight2
1077
  module.decoder.layers.3.mlp.experts.linear_fc1.weight3
 
1078
  Params for bucket 55 (50331648 elements, 50331648 padded size):
1079
  module.decoder.layers.3.mlp.experts.linear_fc1.weight5
1080
  module.decoder.layers.3.mlp.experts.linear_fc1.weight7
@@ -1087,126 +1087,126 @@ Params for bucket 56 (50331648 elements, 50331648 padded size):
1087
  module.decoder.layers.3.mlp.experts.linear_fc1.weight12
1088
  Params for bucket 57 (44040192 elements, 44040192 padded size):
1089
  module.decoder.layers.3.mlp.experts.linear_fc1.weight13
1090
- module.decoder.layers.3.mlp.experts.linear_fc1.weight14
1091
  module.decoder.layers.2.mlp.experts.linear_fc2.weight0
 
1092
  module.decoder.layers.3.mlp.experts.linear_fc1.weight15
1093
  Params for bucket 58 (44040192 elements, 44040192 padded size):
1094
- module.decoder.layers.2.mlp.experts.linear_fc2.weight1
1095
- module.decoder.layers.2.mlp.experts.linear_fc2.weight3
1096
- module.decoder.layers.2.mlp.experts.linear_fc2.weight5
1097
- module.decoder.layers.2.mlp.experts.linear_fc2.weight7
1098
  module.decoder.layers.2.mlp.experts.linear_fc2.weight2
1099
  module.decoder.layers.2.mlp.experts.linear_fc2.weight4
1100
  module.decoder.layers.2.mlp.experts.linear_fc2.weight6
 
 
 
 
1101
  Params for bucket 59 (44040192 elements, 44040192 padded size):
1102
- module.decoder.layers.2.mlp.experts.linear_fc2.weight11
1103
- module.decoder.layers.2.mlp.experts.linear_fc2.weight9
1104
- module.decoder.layers.2.mlp.experts.linear_fc2.weight13
1105
  module.decoder.layers.2.mlp.experts.linear_fc2.weight8
1106
  module.decoder.layers.2.mlp.experts.linear_fc2.weight12
1107
  module.decoder.layers.2.mlp.experts.linear_fc2.weight10
1108
  module.decoder.layers.2.mlp.experts.linear_fc2.weight14
 
 
 
1109
  Params for bucket 60 (44040192 elements, 44040192 padded size):
1110
  module.decoder.layers.2.mlp.experts.linear_fc2.weight15
 
1111
  module.decoder.layers.2.mlp.experts.linear_fc1.weight2
1112
  module.decoder.layers.2.mlp.experts.linear_fc1.weight0
1113
- module.decoder.layers.2.mlp.experts.linear_fc1.weight1
1114
  Params for bucket 61 (50331648 elements, 50331648 padded size):
1115
  module.decoder.layers.2.mlp.experts.linear_fc1.weight3
 
1116
  module.decoder.layers.2.mlp.experts.linear_fc1.weight5
1117
  module.decoder.layers.2.mlp.experts.linear_fc1.weight6
1118
- module.decoder.layers.2.mlp.experts.linear_fc1.weight4
1119
  Params for bucket 62 (50331648 elements, 50331648 padded size):
1120
  module.decoder.layers.2.mlp.experts.linear_fc1.weight7
1121
- module.decoder.layers.2.mlp.experts.linear_fc1.weight9
1122
  module.decoder.layers.2.mlp.experts.linear_fc1.weight10
1123
  module.decoder.layers.2.mlp.experts.linear_fc1.weight8
 
1124
  Params for bucket 63 (50331648 elements, 50331648 padded size):
1125
  module.decoder.layers.2.mlp.experts.linear_fc1.weight11
 
1126
  module.decoder.layers.2.mlp.experts.linear_fc1.weight12
1127
  module.decoder.layers.2.mlp.experts.linear_fc1.weight13
1128
- module.decoder.layers.2.mlp.experts.linear_fc1.weight14
1129
  Params for bucket 64 (44040192 elements, 44040192 padded size):
1130
- module.decoder.layers.1.mlp.experts.linear_fc2.weight2
1131
  module.decoder.layers.1.mlp.experts.linear_fc2.weight0
 
1132
  module.decoder.layers.1.mlp.experts.linear_fc2.weight4
1133
  module.decoder.layers.2.mlp.experts.linear_fc1.weight15
1134
- module.decoder.layers.1.mlp.experts.linear_fc2.weight1
1135
  module.decoder.layers.1.mlp.experts.linear_fc2.weight3
 
1136
  Params for bucket 65 (44040192 elements, 44040192 padded size):
1137
- module.decoder.layers.1.mlp.experts.linear_fc2.weight8
1138
  module.decoder.layers.1.mlp.experts.linear_fc2.weight6
 
1139
  module.decoder.layers.1.mlp.experts.linear_fc2.weight10
1140
  module.decoder.layers.1.mlp.experts.linear_fc2.weight5
1141
- module.decoder.layers.1.mlp.experts.linear_fc2.weight9
1142
  module.decoder.layers.1.mlp.experts.linear_fc2.weight7
 
1143
  module.decoder.layers.1.mlp.experts.linear_fc2.weight11
1144
  Params for bucket 66 (50331648 elements, 50331648 padded size):
1145
  module.decoder.layers.1.mlp.experts.linear_fc2.weight12
1146
- module.decoder.layers.1.mlp.experts.linear_fc2.weight14
1147
  module.decoder.layers.1.mlp.experts.linear_fc1.weight0
1148
- module.decoder.layers.1.mlp.experts.linear_fc2.weight15
1149
  module.decoder.layers.1.mlp.experts.linear_fc2.weight13
 
1150
  module.decoder.layers.1.mlp.experts.linear_fc1.weight1
1151
  Params for bucket 67 (50331648 elements, 50331648 padded size):
1152
  module.decoder.layers.1.mlp.experts.linear_fc1.weight2
 
1153
  module.decoder.layers.1.mlp.experts.linear_fc1.weight3
1154
  module.decoder.layers.1.mlp.experts.linear_fc1.weight4
1155
- module.decoder.layers.1.mlp.experts.linear_fc1.weight5
1156
  Params for bucket 68 (50331648 elements, 50331648 padded size):
1157
  module.decoder.layers.1.mlp.experts.linear_fc1.weight6
 
1158
  module.decoder.layers.1.mlp.experts.linear_fc1.weight9
1159
  module.decoder.layers.1.mlp.experts.linear_fc1.weight7
1160
- module.decoder.layers.1.mlp.experts.linear_fc1.weight8
1161
  Params for bucket 69 (50331648 elements, 50331648 padded size):
1162
  module.decoder.layers.1.mlp.experts.linear_fc1.weight10
1163
  module.decoder.layers.1.mlp.experts.linear_fc1.weight12
1164
- module.decoder.layers.1.mlp.experts.linear_fc1.weight13
1165
  module.decoder.layers.1.mlp.experts.linear_fc1.weight11
 
1166
  Params for bucket 70 (44040192 elements, 44040192 padded size):
1167
  module.decoder.layers.1.mlp.experts.linear_fc1.weight14
1168
- module.decoder.layers.0.mlp.experts.linear_fc2.weight0
1169
  module.decoder.layers.0.mlp.experts.linear_fc2.weight2
1170
- module.decoder.layers.0.mlp.experts.linear_fc2.weight1
1171
  module.decoder.layers.1.mlp.experts.linear_fc1.weight15
 
1172
  Params for bucket 71 (44040192 elements, 44040192 padded size):
1173
  module.decoder.layers.0.mlp.experts.linear_fc2.weight4
1174
  module.decoder.layers.0.mlp.experts.linear_fc2.weight6
1175
  module.decoder.layers.0.mlp.experts.linear_fc2.weight8
1176
  module.decoder.layers.0.mlp.experts.linear_fc2.weight3
1177
- module.decoder.layers.0.mlp.experts.linear_fc2.weight7
1178
  module.decoder.layers.0.mlp.experts.linear_fc2.weight5
 
1179
  module.decoder.layers.0.mlp.experts.linear_fc2.weight9
1180
  Params for bucket 72 (50331648 elements, 50331648 padded size):
1181
  module.decoder.layers.0.mlp.experts.linear_fc2.weight10
1182
  module.decoder.layers.0.mlp.experts.linear_fc2.weight14
1183
  module.decoder.layers.0.mlp.experts.linear_fc2.weight12
1184
- module.decoder.layers.0.mlp.experts.linear_fc1.weight0
1185
  module.decoder.layers.0.mlp.experts.linear_fc2.weight11
1186
  module.decoder.layers.0.mlp.experts.linear_fc2.weight13
1187
  module.decoder.layers.0.mlp.experts.linear_fc2.weight15
 
1188
  Params for bucket 73 (50331648 elements, 50331648 padded size):
1189
  module.decoder.layers.0.mlp.experts.linear_fc1.weight1
 
1190
  module.decoder.layers.0.mlp.experts.linear_fc1.weight3
1191
  module.decoder.layers.0.mlp.experts.linear_fc1.weight4
1192
- module.decoder.layers.0.mlp.experts.linear_fc1.weight2
1193
  Params for bucket 74 (50331648 elements, 50331648 padded size):
1194
  module.decoder.layers.0.mlp.experts.linear_fc1.weight5
1195
  module.decoder.layers.0.mlp.experts.linear_fc1.weight6
1196
- module.decoder.layers.0.mlp.experts.linear_fc1.weight7
1197
  module.decoder.layers.0.mlp.experts.linear_fc1.weight8
 
1198
  Params for bucket 75 (50331648 elements, 50331648 padded size):
1199
  module.decoder.layers.0.mlp.experts.linear_fc1.weight9
1200
- module.decoder.layers.0.mlp.experts.linear_fc1.weight12
1201
  module.decoder.layers.0.mlp.experts.linear_fc1.weight10
1202
  module.decoder.layers.0.mlp.experts.linear_fc1.weight11
 
1203
  Params for bucket 76 (37748736 elements, 37748736 padded size):
1204
  module.decoder.layers.0.mlp.experts.linear_fc1.weight13
1205
  module.decoder.layers.0.mlp.experts.linear_fc1.weight14
1206
  module.decoder.layers.0.mlp.experts.linear_fc1.weight15
1207
- INFO:megatron.core.optimizer:Setting up optimizer with config OptimizerConfig(optimizer='adam', lr=0.0001, min_lr=1e-05, decoupled_lr=None, decoupled_min_lr=None, weight_decay=0.1, fp8_recipe='mxfp8', fp16=False, bf16=True, reuse_grad_buf_for_mxfp8_param_ag=False, params_dtype=torch.bfloat16, use_precision_aware_optimizer=False, store_param_remainders=True, main_grads_dtype=torch.float32, main_params_dtype=torch.float32, exp_avg_dtype=torch.float32, exp_avg_sq_dtype=torch.float32, loss_scale=None, initial_loss_scale=4294967296, min_loss_scale=1.0, loss_scale_window=1000, hysteresis=2, adam_beta1=0.9, adam_beta2=0.999, adam_eps=1e-08, sgd_momentum=0.9, use_distributed_optimizer=True, overlap_param_gather_with_optimizer_step=False, optimizer_cpu_offload=False, optimizer_offload_fraction=1.0, use_torch_optimizer_for_cpu_offload=False, overlap_cpu_optimizer_d2h_h2d=False, pin_cpu_grads=True, pin_cpu_params=True, clip_grad=1.0, log_num_zeros_in_grad=False, barrier_with_L1_time=True, timers=<megatron.core.timers.Timers object at 0x70bf386b5010>, config_logger_dir='')
1208
  INFO:megatron.core.optimizer_param_scheduler:> learning rate decay style: cosine
1209
- [after model, optimizer, and learning rate scheduler are built] datetime: 2026-08-29 23:46:46
1210
  > building train, validation, and test datasets ...
1211
  > datasets target sizes (minimum size):
1212
  train: 96
@@ -1214,7 +1214,7 @@ INFO:megatron.core.optimizer_param_scheduler:> learning rate decay style: cosine
1214
  test: 0
1215
  INFO:megatron.core.datasets.blended_megatron_dataset_config:Let split_matrix = [(0, 1.0), None, None]
1216
  > building train, validation, and test datasets for GPT ...
1217
- INFO:megatron.core.datasets.blended_megatron_dataset_builder:Building GPTDataset splits with sizes=(96, 0, 0) and config=GPTDatasetConfig(random_seed=1234, sequence_length=8192, blend=(['/workspace/data/wikitext/wikitext103_gpt2_text_document'], None), blend_per_split=None, split='100,0,0', split_matrix=[(0, 1.0), None, None], num_dataset_builder_threads=1, path_to_cache=None, mmap_bin_files=True, mock=False, tokenizer=<megatron.training.tokenizer.tokenizer._GPT2BPETokenizer object at 0x70bf389001a0>, mid_level_dataset_surplus=0.005, reset_position_ids=False, reset_attention_mask=False, eod_mask_loss=False, create_attention_mask=True, drop_last_partial_validation_sequence=True, add_extra_token_to_sequence=True, object_storage_cache_path=None)
1218
  INFO:megatron.core.datasets.indexed_dataset:Load the _IndexReader from /workspace/data/wikitext/wikitext103_gpt2_text_document.idx
1219
  INFO:megatron.core.datasets.indexed_dataset: Extract the sequence lengths
1220
  INFO:megatron.core.datasets.indexed_dataset: Extract the sequence pointers
@@ -1227,14 +1227,14 @@ INFO:megatron.core.datasets.gpt_dataset: Load the sample index from eadc3756447e
1227
  INFO:megatron.core.datasets.gpt_dataset: Load the shuffle index from eadc3756447e39d609e8aa728a91f451-GPTDataset-train-shuffle_index.npy
1228
  INFO:megatron.core.datasets.gpt_dataset:> total number of samples: 14536
1229
  > finished creating GPT datasets ...
1230
- [after dataloaders are built] datetime: 2026-08-29 23:46:47
1231
  done with setup ...
1232
  (min, max) time across ranks (ms):
1233
- model-and-optimizer-setup ......................: (3290.03, 3304.28)
1234
- train/valid/test-data-iterators-setup ..........: (63.09, 81.05)
1235
  training ...
1236
  Overwriting rerun_state_machine.current_iteration from -1 to 0...
1237
- [before the start of training step] datetime: 2026-08-29 23:46:47
1238
  INFO:moonep.api:MoonEP comm buffer sizes:
1239
  config : R=8, S=8192, H=4096, E=128, K=8, NvS=69600, NvS_padded=69632
1240
  sm_count : num_sms=32, num_sms_dedup=148
@@ -1246,44 +1246,42 @@ INFO:moonep.api:MoonEP comm buffer sizes:
1246
  [MoonEP selfcheck 2/4] dropped groups empty OK | covered rows [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 128, 129, 130, 131, 132, 133, 134, 135, 136, 137, 138, 139, 140, 141, 142, 143] sizes [9344, 2688, 128, 24192, 128, 0, 128, 256, 128, 768, 10496, 128, 128, 2432, 128, 1536, 14208, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0] | duplicated experts [[69, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1], [88, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1], [55, 52, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1], [-1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1], [63, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1], [124, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1], [10, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1], [58, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1]]
1247
  [MoonEP selfcheck 3/4] dropped groups empty OK | covered rows [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 128, 129, 130, 131, 132, 133, 134, 135, 136, 137, 138, 139, 140, 141, 142, 143] sizes [128, 128, 128, 0, 128, 44416, 128, 128, 128, 6912, 128, 256, 128, 128, 128, 128, 13824, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0] | duplicated experts [[78, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1], [111, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1], [62, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1], [-1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1], [57, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1], [106, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1], [121, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1], [57, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1]]
1248
  [MoonEP selfcheck 4/4] dropped groups empty OK | covered rows [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 128, 129, 130, 131, 132, 133, 134, 135, 136, 137, 138, 139, 140, 141, 142, 143] sizes [2304, 128, 128, 512, 128, 12416, 128, 1152, 1280, 16768, 640, 29056, 128, 128, 128, 1664, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0] | duplicated experts [[-1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1], [118, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1], [11, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1], [118, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1], [39, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1], [11, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1], [73, 79, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1], [86, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1]]
1249
- [2026-08-29 23:47:08] iteration 1/ 12 | consumed samples: 8 | elapsed time per iteration (ms): 21385.7 | throughput per GPU (TFLOP/s/GPU): 8.5 | learning rate: 2.000000E-05 | global batch size: 8 | lm loss: 1.115993E+01 | loss scale: 1.0 | grad norm: 44.806 | number of skipped iterations: 0 | number of nan iterations: 0 |
1250
  Number of parameters in transformer block in billions: 29.85
1251
  Number of parameters in embedding layers in billions: 0.41
1252
  Total number of parameters in billions: 30.26
1253
  Number of parameters in most loaded shard in billions: 30.2590
1254
  Theoretical memory footprints: weight and optimizer=216428.96 MB
1255
  [Rank 0] (after 1 iterations) memory (MB) | allocated: 67239.548828125 | max allocated: 67239.54931640625 | reserved: 68548.0 | max reserved: 68548.0
1256
- [2026-08-29 23:47:08] iteration 2/ 12 | consumed samples: 16 | elapsed time per iteration (ms): 360.7 | throughput per GPU (TFLOP/s/GPU): 501.3 | learning rate: 4.000000E-05 | global batch size: 8 | lm loss: 1.117600E+01 | loss scale: 1.0 | grad norm: 44.336 | number of skipped iterations: 0 | number of nan iterations: 0 |
1257
- [2026-08-29 23:47:09] iteration 3/ 12 | consumed samples: 24 | elapsed time per iteration (ms): 332.9 | throughput per GPU (TFLOP/s/GPU): 543.2 | learning rate: 6.000000E-05 | global batch size: 8 | lm loss: 9.685959E+00 | loss scale: 1.0 | grad norm: 62.848 | number of skipped iterations: 0 | number of nan iterations: 0 |
1258
  Capture CUDA graph for training!!!
1259
  CUDA graph capture done!!!
1260
- [2026-08-29 23:47:10] iteration 4/ 12 | consumed samples: 32 | elapsed time per iteration (ms): 945.9 | throughput per GPU (TFLOP/s/GPU): 191.2 | learning rate: 8.000000E-05 | global batch size: 8 | lm loss: 1.338090E+01 | loss scale: 1.0 | grad norm: 20.295 | number of skipped iterations: 0 | number of nan iterations: 0 |
1261
- [2026-08-29 23:47:10] iteration 5/ 12 | consumed samples: 40 | elapsed time per iteration (ms): 324.3 | throughput per GPU (TFLOP/s/GPU): 557.7 | learning rate: 1.000000E-04 | global batch size: 8 | lm loss: 1.049949E+01 | loss scale: 1.0 | grad norm: 95.128 | number of skipped iterations: 0 | number of nan iterations: 0 |
1262
- [2026-08-29 23:47:10] iteration 6/ 12 | consumed samples: 48 | elapsed time per iteration (ms): 322.0 | throughput per GPU (TFLOP/s/GPU): 561.6 | learning rate: 7.750000E-05 | global batch size: 8 | lm loss: 1.041523E+01 | loss scale: 1.0 | grad norm: 50.280 | number of skipped iterations: 0 | number of nan iterations: 0 |
1263
- [2026-08-29 23:47:11] iteration 7/ 12 | consumed samples: 56 | elapsed time per iteration (ms): 322.0 | throughput per GPU (TFLOP/s/GPU): 561.6 | learning rate: 3.250000E-05 | global batch size: 8 | lm loss: 1.026906E+01 | loss scale: 1.0 | grad norm: 6.318 | number of skipped iterations: 0 | number of nan iterations: 0 |
1264
- [2026-08-29 23:47:11] iteration 8/ 12 | consumed samples: 64 | elapsed time per iteration (ms): 321.4 | throughput per GPU (TFLOP/s/GPU): 562.7 | learning rate: 1.000000E-05 | global batch size: 8 | lm loss: 9.806656E+00 | loss scale: 1.0 | grad norm: 5.554 | number of skipped iterations: 0 | number of nan iterations: 0 |
1265
- [2026-08-29 23:47:11] iteration 9/ 12 | consumed samples: 72 | elapsed time per iteration (ms): 323.5 | throughput per GPU (TFLOP/s/GPU): 559.0 | learning rate: 1.000000E-05 | global batch size: 8 | lm loss: 9.544450E+00 | loss scale: 1.0 | grad norm: 5.321 | number of skipped iterations: 0 | number of nan iterations: 0 |
1266
- [2026-08-29 23:47:12] iteration 10/ 12 | consumed samples: 80 | elapsed time per iteration (ms): 322.7 | throughput per GPU (TFLOP/s/GPU): 560.3 | learning rate: 1.000000E-05 | global batch size: 8 | lm loss: 9.444943E+00 | loss scale: 1.0 | grad norm: 5.068 | number of skipped iterations: 0 | number of nan iterations: 0 |
1267
- [2026-08-29 23:47:12] iteration 11/ 12 | consumed samples: 88 | elapsed time per iteration (ms): 324.4 | throughput per GPU (TFLOP/s/GPU): 557.5 | learning rate: 1.000000E-05 | global batch size: 8 | lm loss: 9.321287E+00 | loss scale: 1.0 | grad norm: 4.742 | number of skipped iterations: 0 | number of nan iterations: 0 |
1268
- [2026-08-29 23:47:12] iteration 12/ 12 | consumed samples: 96 | elapsed time per iteration (ms): 321.5 | throughput per GPU (TFLOP/s/GPU): 562.5 | learning rate: 1.000000E-05 | global batch size: 8 | lm loss: 9.205903E+00 | loss scale: 1.0 | grad norm: 4.468 | number of skipped iterations: 0 | number of nan iterations: 0 |
1269
- [after training is done] datetime: 2026-08-29 23:47:12
1270
- [rank0]:[W829 23:47:14.239671143 ProcessGroupNCCL.cpp:1505] Warning: WARNING: destroy_process_group() was not called before program exit, which can leak resources. For more info, please see https://pytorch.org/docs/stable/distributed.html#shutdown (function operator())
1271
- [rank4]:[W829 23:47:15.875156992 ProcessGroupNCCL.cpp:1505] Warning: WARNING: destroy_process_group() was not called before program exit, which can leak resources. For more info, please see https://pytorch.org/docs/stable/distributed.html#shutdown (function operator())
1272
- [rank6]:[W829 23:47:15.875453613 ProcessGroupNCCL.cpp:1505] Warning: WARNING: destroy_process_group() was not called before program exit, which can leak resources. For more info, please see https://pytorch.org/docs/stable/distributed.html#shutdown (function operator())
1273
- [rank1]:[W829 23:47:15.884984177 ProcessGroupNCCL.cpp:1505] Warning: WARNING: destroy_process_group() was not called before program exit, which can leak resources. For more info, please see https://pytorch.org/docs/stable/distributed.html#shutdown (function operator())
1274
- [rank3]:[W829 23:47:15.886398543 ProcessGroupNCCL.cpp:1505] Warning: WARNING: destroy_process_group() was not called before program exit, which can leak resources. For more info, please see https://pytorch.org/docs/stable/distributed.html#shutdown (function operator())
1275
- [rank5]:[W829 23:47:15.949105908 ProcessGroupNCCL.cpp:1505] Warning: WARNING: destroy_process_group() was not called before program exit, which can leak resources. For more info, please see https://pytorch.org/docs/stable/distributed.html#shutdown (function operator())
1276
- [rank2]:[W829 23:47:15.956457205 ProcessGroupNCCL.cpp:1505] Warning: WARNING: destroy_process_group() was not called before program exit, which can leak resources. For more info, please see https://pytorch.org/docs/stable/distributed.html#shutdown (function operator())
1277
- [rank7]:[W829 23:47:15.975350193 ProcessGroupNCCL.cpp:1505] Warning: WARNING: destroy_process_group() was not called before program exit, which can leak resources. For more info, please see https://pytorch.org/docs/stable/distributed.html#shutdown (function operator())
1278
- W0829 23:47:18.883000 240032 torch/distributed/elastic/agent/server/api.py:719] Received 15 death signal, shutting down workers
1279
- W0829 23:47:18.884000 240032 torch/distributed/elastic/multiprocessing/api.py:900] Sending process 240216 closing signal SIGTERM
1280
- W0829 23:47:18.884000 240032 torch/distributed/elastic/multiprocessing/api.py:900] Sending process 240217 closing signal SIGTERM
1281
- W0829 23:47:18.885000 240032 torch/distributed/elastic/multiprocessing/api.py:900] Sending process 240218 closing signal SIGTERM
1282
- W0829 23:47:18.886000 240032 torch/distributed/elastic/multiprocessing/api.py:900] Sending process 240219 closing signal SIGTERM
1283
- W0829 23:47:18.887000 240032 torch/distributed/elastic/multiprocessing/api.py:900] Sending process 240220 closing signal SIGTERM
1284
- W0829 23:47:18.887000 240032 torch/distributed/elastic/multiprocessing/api.py:900] Sending process 240221 closing signal SIGTERM
1285
- W0829 23:47:18.888000 240032 torch/distributed/elastic/multiprocessing/api.py:900] Sending process 240222 closing signal SIGTERM
1286
- W0829 23:47:18.888000 240032 torch/distributed/elastic/multiprocessing/api.py:900] Sending process 240223 closing signal SIGTERM
1287
  Traceback (most recent call last):
1288
  File "/usr/local/bin/torchrun", line 8, in <module>
1289
  sys.exit(main())
@@ -1311,4 +1309,4 @@ Traceback (most recent call last):
1311
  time.sleep(monitor_interval)
1312
  File "/usr/local/lib/python3.12/dist-packages/torch/distributed/elastic/multiprocessing/api.py", line 84, in _terminate_process_handler
1313
  raise SignalException(f"Process {os.getpid()} got signal: {sigval}", sigval=sigval)
1314
- torch.distributed.elastic.multiprocessing.api.SignalException: Process 240032 got signal: 15
 
1
+ [01:32:32] WARN: moonep_backend_async_grad_reduce lacks --moe-router-fusion; QWEN3_235B runs unfused (timing only, routing identical)
2
  ==============================================================
3
  QWEN3_235B / moonep (slots/rank=16 policy=matched)
4
  E=128 K=8 H'=1536 m=16
5
  L=12 seq=8192 gbs=8 (1 microbatches)
6
  cap_mode=none cap=none cg=1 recompute=none
7
  memory_dump=none snapshots=/workspace/tierb-fixed-moonep-full/moonep/tierb/tierb_gbs8/QWEN3_235B__moonep__matched__seq8192__gbs8__capnone__async/rep2/memory_snapshots/snapshot.pickle
8
+ repo=/workspace/Megatron-MoE-Echo @ 2e68c8ad
9
  ==============================================================
10
+ W0830 01:32:33.428000 382773 torch/distributed/run.py:766]
11
+ W0830 01:32:33.428000 382773 torch/distributed/run.py:766] *****************************************
12
+ W0830 01:32:33.428000 382773 torch/distributed/run.py:766] Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed.
13
+ W0830 01:32:33.428000 382773 torch/distributed/run.py:766] *****************************************
14
  /workspace/Megatron-MoE-Echo/megatron/training/arguments.py:831: UserWarning: Disabling sequence parallelism because tensor model parallelism is disabled
15
  warnings.warn("Disabling sequence parallelism because tensor model parallelism is disabled")
 
 
 
16
  /workspace/Megatron-MoE-Echo/megatron/training/arguments.py:831: UserWarning: Disabling sequence parallelism because tensor model parallelism is disabled
17
  warnings.warn("Disabling sequence parallelism because tensor model parallelism is disabled")
18
  /workspace/Megatron-MoE-Echo/megatron/training/arguments.py:831: UserWarning: Disabling sequence parallelism because tensor model parallelism is disabled
19
  warnings.warn("Disabling sequence parallelism because tensor model parallelism is disabled")
20
+ WARNING: one_logger package is required to enable e2e metrics tracking. please go to https://confluence.nvidia.com/display/MLWFO/Package+Repositories for details to install it
21
  /workspace/Megatron-MoE-Echo/megatron/training/arguments.py:831: UserWarning: Disabling sequence parallelism because tensor model parallelism is disabled
22
  warnings.warn("Disabling sequence parallelism because tensor model parallelism is disabled")
23
  using world size: 8, data-parallel size: 8, context-parallel size: 1, hierarchical context-parallel sizes: None, tensor-model-parallel size: 1, pipeline-model-parallel size: 1
 
575
  warnings.warn("Disabling sequence parallelism because tensor model parallelism is disabled")
576
  /workspace/Megatron-MoE-Echo/megatron/training/arguments.py:831: UserWarning: Disabling sequence parallelism because tensor model parallelism is disabled
577
  warnings.warn("Disabling sequence parallelism because tensor model parallelism is disabled")
578
+ /workspace/Megatron-MoE-Echo/megatron/training/arguments.py:831: UserWarning: Disabling sequence parallelism because tensor model parallelism is disabled
579
+ warnings.warn("Disabling sequence parallelism because tensor model parallelism is disabled")
 
580
  [Gloo] Rank 4 is connected to 7 peer ranks. Expected number of connected peer ranks is : 7
 
 
 
 
581
  [Gloo] Rank 0 is connected to 7 peer ranks. Expected number of connected peer ranks is : 7
582
+ [Gloo] Rank 1 is connected to 7 peer ranks. Expected number of connected peer ranks is : 7
583
  [Gloo] Rank 2 is connected to 7 peer ranks. Expected number of connected peer ranks is : 7
584
+ [Gloo] Rank 3 is connected to 7 peer ranks. Expected number of connected peer ranks is : 7
585
+ [Gloo] Rank 6 is connected to 7 peer ranks. Expected number of connected peer ranks is : 7
586
  [Gloo] Rank 5 is connected to 7 peer ranks. Expected number of connected peer ranks is : 7
587
+ [Gloo] Rank 7 is connected to 7 peer ranks. Expected number of connected peer ranks is : 7
588
  [Gloo] Rank 3 is connected to 7 peer ranks. Expected number of connected peer ranks is : 7
589
+ [Gloo] Rank 0 is connected to 7 peer ranks. Expected number of connected peer ranks is : 7
590
  [Gloo] Rank 1 is connected to 7 peer ranks. Expected number of connected peer ranks is : 7
591
+ [Gloo] Rank 2 is connected to 7 peer ranks. Expected number of connected peer ranks is : 7
592
+ [Gloo] Rank [Gloo] Rank [Gloo] Rank [Gloo] Rank 4567 is connected to is connected to is connected to is connected to 7777 peer ranks. peer ranks. peer ranks. peer ranks. Expected number of connected peer ranks is : Expected number of connected peer ranks is : Expected number of connected peer ranks is : Expected number of connected peer ranks is : 7777
593
+
594
+
595
+
596
  [Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
597
  > initialized tensor model parallel with size 1
598
  > initialized pipeline model parallel with size 1
599
  > setting random seeds to 1234 ...
600
+ [Gloo] Rank [Gloo] Rank 0[Gloo] Rank [Gloo] Rank 0 is connected to 00 is connected to 0 is connected to is connected to 0 peer ranks. 00 peer ranks. Expected number of connected peer ranks is : peer ranks. peer ranks. Expected number of connected peer ranks is : 0Expected number of connected peer ranks is :
601
+ Expected number of connected peer ranks is : 000
602
+
603
+
604
  [Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
605
  [Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
606
  [Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
607
+ [rank7]:[W830 01:32:51.378489561 ProcessGroupNCCL.cpp:4789] [PG ID 0 PG GUID 0 Rank 7] using GPU 7 as device used by this process is currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect. You can specify device_id in init_process_group() to force use of a particular device.
608
+ [rank1]:[W830 01:32:51.378526948 ProcessGroupNCCL.cpp:4789] [PG ID 0 PG GUID 0 Rank 1] using GPU 1 as device used by this process is currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect. You can specify device_id in init_process_group() to force use of a particular device.
609
+ [rank4]:[W830 01:32:51.378555820 ProcessGroupNCCL.cpp:4789] [PG ID 0 PG GUID 0 Rank 4] using GPU 4 as device used by this process is currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect. You can specify device_id in init_process_group() to force use of a particular device.
610
+ [rank3]:[W830 01:32:51.378565188 ProcessGroupNCCL.cpp:4789] [PG ID 0 PG GUID 0 Rank 3] using GPU 3 as device used by this process is currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect. You can specify device_id in init_process_group() to force use of a particular device.
611
+ [rank2]:[W830 01:32:51.380630702 ProcessGroupNCCL.cpp:4789] [PG ID 0 PG GUID 0 Rank 2] using GPU 2 as device used by this process is currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect. You can specify device_id in init_process_group() to force use of a particular device.
612
+ [rank5]:[W830 01:32:51.381181551 ProcessGroupNCCL.cpp:4789] [PG ID 0 PG GUID 0 Rank 5] using GPU 5 as device used by this process is currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect. You can specify device_id in init_process_group() to force use of a particular device.
613
+ [rank6]:[W830 01:32:51.381335334 ProcessGroupNCCL.cpp:4789] [PG ID 0 PG GUID 0 Rank 6] using GPU 6 as device used by this process is currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect. You can specify device_id in init_process_group() to force use of a particular device.
 
 
 
 
614
  > compiling dataset index builder ...
615
  make: Entering directory '/workspace/Megatron-MoE-Echo/megatron/core/datasets'
616
  make: Nothing to be done for 'default'.
617
  make: Leaving directory '/workspace/Megatron-MoE-Echo/megatron/core/datasets'
618
+ >>> done with dataset index builder. Compilation time: 0.033 seconds
619
  WARNING: constraints for invoking optimized fused softmax kernel are not met. We default back to unfused kernel invocations.
620
  > compiling and loading fused kernels ...
621
+ [rank0]:[W830 01:32:51.542543761 ProcessGroupNCCL.cpp:4789] [PG ID 0 PG GUID 0 Rank 0] using GPU 0 as device used by this process is currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect. You can specify device_id in init_process_group() to force use of a particular device.
622
+ >>> done with compiling and loading fused kernels. Compilation time: 4.053 seconds
623
+ time to initialize megatron (seconds): 7.444
624
+ [after megatron is initialized] datetime: 2026-08-30 01:32:57
625
  building GPT model ...
626
  /usr/local/lib/python3.12/dist-packages/transformer_engine/pytorch/cpu_offload.py:708: DeprecationWarning: Offloading weights is deprecated. Using offload_weights=True does not have any effect.
627
  warnings.warn(
 
645
  Params for bucket 1 (206045184 elements, 206045184 padded size):
646
  module.output_layer.weight
647
  Params for bucket 2 (71840000 elements, 71840000 padded size):
648
+ module.decoder.final_layernorm.weight
649
+ module.decoder.layers.11.mlp.router.weight
650
  module.decoder.layers.11.pre_mlp_layernorm.weight
651
  module.decoder.layers.11.self_attention.q_layernorm.weight
652
+ module.decoder.layers.11.self_attention.k_layernorm.weight
653
  module.decoder.layers.11.self_attention.linear_qkv.weight
654
  module.decoder.layers.11.self_attention.linear_qkv.layer_norm_weight
 
 
655
  module.decoder.layers.11.self_attention.linear_proj.weight
656
  Params for bucket 3 (71835904 elements, 71835904 padded size):
 
657
  module.decoder.layers.10.pre_mlp_layernorm.weight
658
  module.decoder.layers.10.self_attention.q_layernorm.weight
 
659
  module.decoder.layers.10.self_attention.k_layernorm.weight
660
+ module.decoder.layers.10.self_attention.linear_qkv.weight
661
  module.decoder.layers.10.self_attention.linear_qkv.layer_norm_weight
662
+ module.decoder.layers.10.mlp.router.weight
663
  module.decoder.layers.10.self_attention.linear_proj.weight
664
  Params for bucket 4 (71835904 elements, 71835904 padded size):
 
 
 
 
665
  module.decoder.layers.9.self_attention.linear_qkv.weight
666
  module.decoder.layers.9.self_attention.k_layernorm.weight
667
+ module.decoder.layers.9.self_attention.linear_qkv.layer_norm_weight
668
  module.decoder.layers.9.self_attention.linear_proj.weight
669
+ module.decoder.layers.9.mlp.router.weight
670
+ module.decoder.layers.9.self_attention.q_layernorm.weight
671
+ module.decoder.layers.9.pre_mlp_layernorm.weight
672
  Params for bucket 5 (71835904 elements, 71835904 padded size):
673
  module.decoder.layers.8.mlp.router.weight
674
+ module.decoder.layers.8.pre_mlp_layernorm.weight
675
  module.decoder.layers.8.self_attention.k_layernorm.weight
676
  module.decoder.layers.8.self_attention.q_layernorm.weight
 
677
  module.decoder.layers.8.self_attention.linear_qkv.weight
678
  module.decoder.layers.8.self_attention.linear_qkv.layer_norm_weight
679
  module.decoder.layers.8.self_attention.linear_proj.weight
680
  Params for bucket 6 (71835904 elements, 71835904 padded size):
681
  module.decoder.layers.7.mlp.router.weight
682
+ module.decoder.layers.7.self_attention.q_layernorm.weight
683
  module.decoder.layers.7.pre_mlp_layernorm.weight
684
  module.decoder.layers.7.self_attention.k_layernorm.weight
 
685
  module.decoder.layers.7.self_attention.linear_qkv.layer_norm_weight
686
  module.decoder.layers.7.self_attention.linear_qkv.weight
687
  module.decoder.layers.7.self_attention.linear_proj.weight
 
689
  module.decoder.layers.6.mlp.router.weight
690
  module.decoder.layers.6.self_attention.linear_qkv.weight
691
  module.decoder.layers.6.self_attention.k_layernorm.weight
692
+ module.decoder.layers.6.self_attention.linear_proj.weight
693
  module.decoder.layers.6.pre_mlp_layernorm.weight
694
  module.decoder.layers.6.self_attention.q_layernorm.weight
695
+ module.decoder.layers.6.self_attention.linear_qkv.layer_norm_weight
696
  Params for bucket 8 (71835904 elements, 71835904 padded size):
697
  module.decoder.layers.5.mlp.router.weight
698
+ module.decoder.layers.5.self_attention.k_layernorm.weight
699
  module.decoder.layers.5.self_attention.linear_qkv.weight
 
700
  module.decoder.layers.5.pre_mlp_layernorm.weight
701
+ module.decoder.layers.5.self_attention.q_layernorm.weight
702
  module.decoder.layers.5.self_attention.linear_qkv.layer_norm_weight
 
703
  module.decoder.layers.5.self_attention.linear_proj.weight
704
  Params for bucket 9 (71835904 elements, 71835904 padded size):
705
+ module.decoder.layers.4.mlp.router.weight
 
706
  module.decoder.layers.4.self_attention.q_layernorm.weight
707
+ module.decoder.layers.4.pre_mlp_layernorm.weight
708
  module.decoder.layers.4.self_attention.linear_qkv.layer_norm_weight
 
709
  module.decoder.layers.4.self_attention.k_layernorm.weight
710
+ module.decoder.layers.4.self_attention.linear_qkv.weight
711
  module.decoder.layers.4.self_attention.linear_proj.weight
712
  Params for bucket 10 (71835904 elements, 71835904 padded size):
713
  module.decoder.layers.3.mlp.router.weight
 
714
  module.decoder.layers.3.self_attention.q_layernorm.weight
715
  module.decoder.layers.3.self_attention.linear_qkv.weight
716
+ module.decoder.layers.3.pre_mlp_layernorm.weight
717
  module.decoder.layers.3.self_attention.k_layernorm.weight
718
  module.decoder.layers.3.self_attention.linear_qkv.layer_norm_weight
719
  module.decoder.layers.3.self_attention.linear_proj.weight
720
  Params for bucket 11 (71835904 elements, 71835904 padded size):
721
  module.decoder.layers.2.mlp.router.weight
722
+ module.decoder.layers.2.self_attention.k_layernorm.weight
723
+ module.decoder.layers.2.self_attention.q_layernorm.weight
724
  module.decoder.layers.2.pre_mlp_layernorm.weight
725
  module.decoder.layers.2.self_attention.linear_qkv.weight
 
 
726
  module.decoder.layers.2.self_attention.linear_qkv.layer_norm_weight
727
  module.decoder.layers.2.self_attention.linear_proj.weight
728
  Params for bucket 12 (71835904 elements, 71835904 padded size):
729
  module.decoder.layers.1.mlp.router.weight
730
  module.decoder.layers.1.pre_mlp_layernorm.weight
 
 
731
  module.decoder.layers.1.self_attention.k_layernorm.weight
732
+ module.decoder.layers.1.self_attention.linear_qkv.weight
733
+ module.decoder.layers.1.self_attention.q_layernorm.weight
734
  module.decoder.layers.1.self_attention.linear_qkv.layer_norm_weight
735
  module.decoder.layers.1.self_attention.linear_proj.weight
736
  Params for bucket 13 (71835904 elements, 71835904 padded size):
737
+ module.decoder.layers.0.self_attention.k_layernorm.weight
738
+ module.decoder.layers.0.self_attention.linear_qkv.weight
739
  module.decoder.layers.0.self_attention.q_layernorm.weight
740
  module.decoder.layers.0.pre_mlp_layernorm.weight
 
 
 
 
741
  module.decoder.layers.0.self_attention.linear_proj.weight
742
+ module.decoder.layers.0.mlp.router.weight
743
+ module.decoder.layers.0.self_attention.linear_qkv.layer_norm_weight
744
  Params for bucket 14 (206045184 elements, 206045184 padded size):
745
  module.embedding.word_embeddings.weight
746
  INFO:megatron.core.distributed.param_and_grad_buffer:Number of buckets for gradient all-reduce / reduce-scatter: 76
747
  Params for bucket 1 (44040192 elements, 44040192 padded size):
 
 
 
 
748
  module.decoder.layers.11.mlp.experts.linear_fc2.weight1
749
  module.decoder.layers.11.mlp.experts.linear_fc2.weight3
750
  module.decoder.layers.11.mlp.experts.linear_fc2.weight5
751
+ module.decoder.layers.11.mlp.experts.linear_fc2.weight0
752
+ module.decoder.layers.11.mlp.experts.linear_fc2.weight4
753
+ module.decoder.layers.11.mlp.experts.linear_fc2.weight2
754
+ module.decoder.layers.11.mlp.experts.linear_fc2.weight6
755
  Params for bucket 2 (44040192 elements, 44040192 padded size):
 
 
 
756
  module.decoder.layers.11.mlp.experts.linear_fc2.weight7
757
  module.decoder.layers.11.mlp.experts.linear_fc2.weight11
758
  module.decoder.layers.11.mlp.experts.linear_fc2.weight9
759
  module.decoder.layers.11.mlp.experts.linear_fc2.weight13
760
+ module.decoder.layers.11.mlp.experts.linear_fc2.weight10
761
+ module.decoder.layers.11.mlp.experts.linear_fc2.weight8
762
+ module.decoder.layers.11.mlp.experts.linear_fc2.weight12
763
  Params for bucket 3 (50331648 elements, 50331648 padded size):
764
+ module.decoder.layers.11.mlp.experts.linear_fc1.weight0
 
765
  module.decoder.layers.11.mlp.experts.linear_fc1.weight2
766
  module.decoder.layers.11.mlp.experts.linear_fc2.weight15
767
+ module.decoder.layers.11.mlp.experts.linear_fc2.weight14
768
+ module.decoder.layers.11.mlp.experts.linear_fc1.weight1
769
  Params for bucket 4 (50331648 elements, 50331648 padded size):
770
  module.decoder.layers.11.mlp.experts.linear_fc1.weight3
771
  module.decoder.layers.11.mlp.experts.linear_fc1.weight4
 
772
  module.decoder.layers.11.mlp.experts.linear_fc1.weight6
773
+ module.decoder.layers.11.mlp.experts.linear_fc1.weight5
774
  Params for bucket 5 (50331648 elements, 50331648 padded size):
775
  module.decoder.layers.11.mlp.experts.linear_fc1.weight7
 
776
  module.decoder.layers.11.mlp.experts.linear_fc1.weight10
777
+ module.decoder.layers.11.mlp.experts.linear_fc1.weight8
778
  module.decoder.layers.11.mlp.experts.linear_fc1.weight9
779
  Params for bucket 6 (50331648 elements, 50331648 padded size):
780
  module.decoder.layers.11.mlp.experts.linear_fc1.weight11
 
781
  module.decoder.layers.11.mlp.experts.linear_fc1.weight13
782
+ module.decoder.layers.11.mlp.experts.linear_fc1.weight14
783
  module.decoder.layers.11.mlp.experts.linear_fc1.weight12
784
  Params for bucket 7 (44040192 elements, 44040192 padded size):
 
785
  module.decoder.layers.10.mlp.experts.linear_fc2.weight3
786
  module.decoder.layers.10.mlp.experts.linear_fc2.weight1
787
+ module.decoder.layers.11.mlp.experts.linear_fc1.weight15
788
  module.decoder.layers.10.mlp.experts.linear_fc2.weight2
789
+ module.decoder.layers.10.mlp.experts.linear_fc2.weight0
790
  module.decoder.layers.10.mlp.experts.linear_fc2.weight4
791
  Params for bucket 8 (44040192 elements, 44040192 padded size):
792
  module.decoder.layers.10.mlp.experts.linear_fc2.weight5
 
793
  module.decoder.layers.10.mlp.experts.linear_fc2.weight9
794
+ module.decoder.layers.10.mlp.experts.linear_fc2.weight7
795
  module.decoder.layers.10.mlp.experts.linear_fc2.weight11
 
796
  module.decoder.layers.10.mlp.experts.linear_fc2.weight6
797
+ module.decoder.layers.10.mlp.experts.linear_fc2.weight8
798
  module.decoder.layers.10.mlp.experts.linear_fc2.weight10
799
  Params for bucket 9 (50331648 elements, 50331648 padded size):
 
800
  module.decoder.layers.10.mlp.experts.linear_fc1.weight0
801
  module.decoder.layers.10.mlp.experts.linear_fc2.weight15
802
+ module.decoder.layers.10.mlp.experts.linear_fc2.weight13
803
  module.decoder.layers.10.mlp.experts.linear_fc2.weight12
804
  module.decoder.layers.10.mlp.experts.linear_fc2.weight14
805
  module.decoder.layers.10.mlp.experts.linear_fc1.weight1
806
  Params for bucket 10 (50331648 elements, 50331648 padded size):
807
  module.decoder.layers.10.mlp.experts.linear_fc1.weight2
 
808
  module.decoder.layers.10.mlp.experts.linear_fc1.weight5
809
  module.decoder.layers.10.mlp.experts.linear_fc1.weight3
810
+ module.decoder.layers.10.mlp.experts.linear_fc1.weight4
811
  Params for bucket 11 (50331648 elements, 50331648 padded size):
812
  module.decoder.layers.10.mlp.experts.linear_fc1.weight6
 
813
  module.decoder.layers.10.mlp.experts.linear_fc1.weight8
814
  module.decoder.layers.10.mlp.experts.linear_fc1.weight9
815
+ module.decoder.layers.10.mlp.experts.linear_fc1.weight7
816
  Params for bucket 12 (50331648 elements, 50331648 padded size):
817
  module.decoder.layers.10.mlp.experts.linear_fc1.weight10
818
+ module.decoder.layers.10.mlp.experts.linear_fc1.weight12
819
  module.decoder.layers.10.mlp.experts.linear_fc1.weight13
820
  module.decoder.layers.10.mlp.experts.linear_fc1.weight11
 
821
  Params for bucket 13 (44040192 elements, 44040192 padded size):
822
  module.decoder.layers.10.mlp.experts.linear_fc1.weight14
 
823
  module.decoder.layers.9.mlp.experts.linear_fc2.weight2
824
  module.decoder.layers.9.mlp.experts.linear_fc2.weight0
825
+ module.decoder.layers.9.mlp.experts.linear_fc2.weight1
826
  module.decoder.layers.10.mlp.experts.linear_fc1.weight15
827
  Params for bucket 14 (44040192 elements, 44040192 padded size):
828
+ module.decoder.layers.9.mlp.experts.linear_fc2.weight6
829
+ module.decoder.layers.9.mlp.experts.linear_fc2.weight4
830
+ module.decoder.layers.9.mlp.experts.linear_fc2.weight8
831
  module.decoder.layers.9.mlp.experts.linear_fc2.weight3
832
  module.decoder.layers.9.mlp.experts.linear_fc2.weight5
833
  module.decoder.layers.9.mlp.experts.linear_fc2.weight7
834
  module.decoder.layers.9.mlp.experts.linear_fc2.weight9
 
 
 
835
  Params for bucket 15 (50331648 elements, 50331648 padded size):
836
+ module.decoder.layers.9.mlp.experts.linear_fc2.weight10
837
+ module.decoder.layers.9.mlp.experts.linear_fc2.weight12
838
+ module.decoder.layers.9.mlp.experts.linear_fc2.weight14
839
  module.decoder.layers.9.mlp.experts.linear_fc2.weight11
840
+ module.decoder.layers.9.mlp.experts.linear_fc2.weight13
841
  module.decoder.layers.9.mlp.experts.linear_fc2.weight15
842
  module.decoder.layers.9.mlp.experts.linear_fc1.weight0
 
 
 
843
  Params for bucket 16 (50331648 elements, 50331648 padded size):
844
  module.decoder.layers.9.mlp.experts.linear_fc1.weight1
845
+ module.decoder.layers.9.mlp.experts.linear_fc1.weight4
846
  module.decoder.layers.9.mlp.experts.linear_fc1.weight2
847
  module.decoder.layers.9.mlp.experts.linear_fc1.weight3
 
848
  Params for bucket 17 (50331648 elements, 50331648 padded size):
849
  module.decoder.layers.9.mlp.experts.linear_fc1.weight5
850
+ module.decoder.layers.9.mlp.experts.linear_fc1.weight7
851
  module.decoder.layers.9.mlp.experts.linear_fc1.weight8
852
  module.decoder.layers.9.mlp.experts.linear_fc1.weight6
 
853
  Params for bucket 18 (50331648 elements, 50331648 padded size):
854
  module.decoder.layers.9.mlp.experts.linear_fc1.weight9
855
  module.decoder.layers.9.mlp.experts.linear_fc1.weight11
 
857
  module.decoder.layers.9.mlp.experts.linear_fc1.weight10
858
  Params for bucket 19 (44040192 elements, 44040192 padded size):
859
  module.decoder.layers.9.mlp.experts.linear_fc1.weight13
 
860
  module.decoder.layers.9.mlp.experts.linear_fc1.weight14
861
  module.decoder.layers.9.mlp.experts.linear_fc1.weight15
862
+ module.decoder.layers.8.mlp.experts.linear_fc2.weight0
863
  Params for bucket 20 (44040192 elements, 44040192 padded size):
 
 
 
864
  module.decoder.layers.8.mlp.experts.linear_fc2.weight1
 
865
  module.decoder.layers.8.mlp.experts.linear_fc2.weight3
866
+ module.decoder.layers.8.mlp.experts.linear_fc2.weight5
867
  module.decoder.layers.8.mlp.experts.linear_fc2.weight7
868
+ module.decoder.layers.8.mlp.experts.linear_fc2.weight4
869
+ module.decoder.layers.8.mlp.experts.linear_fc2.weight2
870
+ module.decoder.layers.8.mlp.experts.linear_fc2.weight6
871
  Params for bucket 21 (44040192 elements, 44040192 padded size):
872
+ module.decoder.layers.8.mlp.experts.linear_fc2.weight9
873
+ module.decoder.layers.8.mlp.experts.linear_fc2.weight11
874
+ module.decoder.layers.8.mlp.experts.linear_fc2.weight13
875
  module.decoder.layers.8.mlp.experts.linear_fc2.weight8
876
  module.decoder.layers.8.mlp.experts.linear_fc2.weight10
877
  module.decoder.layers.8.mlp.experts.linear_fc2.weight12
878
  module.decoder.layers.8.mlp.experts.linear_fc2.weight14
 
 
 
879
  Params for bucket 22 (44040192 elements, 44040192 padded size):
880
  module.decoder.layers.8.mlp.experts.linear_fc2.weight15
881
  module.decoder.layers.8.mlp.experts.linear_fc1.weight1
 
882
  module.decoder.layers.8.mlp.experts.linear_fc1.weight0
883
+ module.decoder.layers.8.mlp.experts.linear_fc1.weight2
884
  Params for bucket 23 (50331648 elements, 50331648 padded size):
885
  module.decoder.layers.8.mlp.experts.linear_fc1.weight3
 
886
  module.decoder.layers.8.mlp.experts.linear_fc1.weight4
887
+ module.decoder.layers.8.mlp.experts.linear_fc1.weight5
888
  module.decoder.layers.8.mlp.experts.linear_fc1.weight6
889
  Params for bucket 24 (50331648 elements, 50331648 padded size):
890
  module.decoder.layers.8.mlp.experts.linear_fc1.weight7
891
+ module.decoder.layers.8.mlp.experts.linear_fc1.weight10
892
  module.decoder.layers.8.mlp.experts.linear_fc1.weight8
893
  module.decoder.layers.8.mlp.experts.linear_fc1.weight9
 
894
  Params for bucket 25 (50331648 elements, 50331648 padded size):
895
  module.decoder.layers.8.mlp.experts.linear_fc1.weight11
896
  module.decoder.layers.8.mlp.experts.linear_fc1.weight14
 
897
  module.decoder.layers.8.mlp.experts.linear_fc1.weight13
898
+ module.decoder.layers.8.mlp.experts.linear_fc1.weight12
899
  Params for bucket 26 (44040192 elements, 44040192 padded size):
900
+ module.decoder.layers.7.mlp.experts.linear_fc2.weight3
901
+ module.decoder.layers.7.mlp.experts.linear_fc2.weight1
902
  module.decoder.layers.8.mlp.experts.linear_fc1.weight15
903
  module.decoder.layers.7.mlp.experts.linear_fc2.weight2
904
  module.decoder.layers.7.mlp.experts.linear_fc2.weight0
905
  module.decoder.layers.7.mlp.experts.linear_fc2.weight4
 
 
906
  Params for bucket 27 (44040192 elements, 44040192 padded size):
 
 
 
907
  module.decoder.layers.7.mlp.experts.linear_fc2.weight5
 
908
  module.decoder.layers.7.mlp.experts.linear_fc2.weight9
909
+ module.decoder.layers.7.mlp.experts.linear_fc2.weight7
910
  module.decoder.layers.7.mlp.experts.linear_fc2.weight11
911
+ module.decoder.layers.7.mlp.experts.linear_fc2.weight8
912
+ module.decoder.layers.7.mlp.experts.linear_fc2.weight6
913
+ module.decoder.layers.7.mlp.experts.linear_fc2.weight10
914
  Params for bucket 28 (50331648 elements, 50331648 padded size):
 
915
  module.decoder.layers.7.mlp.experts.linear_fc1.weight0
 
916
  module.decoder.layers.7.mlp.experts.linear_fc2.weight15
917
  module.decoder.layers.7.mlp.experts.linear_fc2.weight13
918
+ module.decoder.layers.7.mlp.experts.linear_fc2.weight12
919
+ module.decoder.layers.7.mlp.experts.linear_fc2.weight14
920
  module.decoder.layers.7.mlp.experts.linear_fc1.weight1
921
  Params for bucket 29 (50331648 elements, 50331648 padded size):
922
  module.decoder.layers.7.mlp.experts.linear_fc1.weight2
 
923
  module.decoder.layers.7.mlp.experts.linear_fc1.weight3
924
+ module.decoder.layers.7.mlp.experts.linear_fc1.weight5
925
  module.decoder.layers.7.mlp.experts.linear_fc1.weight4
926
  Params for bucket 30 (50331648 elements, 50331648 padded size):
927
  module.decoder.layers.7.mlp.experts.linear_fc1.weight6
 
928
  module.decoder.layers.7.mlp.experts.linear_fc1.weight9
929
+ module.decoder.layers.7.mlp.experts.linear_fc1.weight8
930
  module.decoder.layers.7.mlp.experts.linear_fc1.weight7
931
  Params for bucket 31 (50331648 elements, 50331648 padded size):
932
  module.decoder.layers.7.mlp.experts.linear_fc1.weight10
 
933
  module.decoder.layers.7.mlp.experts.linear_fc1.weight12
934
+ module.decoder.layers.7.mlp.experts.linear_fc1.weight13
935
  module.decoder.layers.7.mlp.experts.linear_fc1.weight11
936
  Params for bucket 32 (44040192 elements, 44040192 padded size):
937
  module.decoder.layers.7.mlp.experts.linear_fc1.weight14
 
944
  module.decoder.layers.6.mlp.experts.linear_fc2.weight6
945
  module.decoder.layers.6.mlp.experts.linear_fc2.weight8
946
  module.decoder.layers.6.mlp.experts.linear_fc2.weight3
 
947
  module.decoder.layers.6.mlp.experts.linear_fc2.weight5
948
+ module.decoder.layers.6.mlp.experts.linear_fc2.weight7
949
  module.decoder.layers.6.mlp.experts.linear_fc2.weight9
950
  Params for bucket 34 (50331648 elements, 50331648 padded size):
951
  module.decoder.layers.6.mlp.experts.linear_fc2.weight10
 
952
  module.decoder.layers.6.mlp.experts.linear_fc2.weight14
953
+ module.decoder.layers.6.mlp.experts.linear_fc2.weight12
954
  module.decoder.layers.6.mlp.experts.linear_fc2.weight11
955
  module.decoder.layers.6.mlp.experts.linear_fc2.weight13
956
  module.decoder.layers.6.mlp.experts.linear_fc2.weight15
957
  module.decoder.layers.6.mlp.experts.linear_fc1.weight0
958
  Params for bucket 35 (50331648 elements, 50331648 padded size):
959
  module.decoder.layers.6.mlp.experts.linear_fc1.weight1
960
+ module.decoder.layers.6.mlp.experts.linear_fc1.weight2
961
  module.decoder.layers.6.mlp.experts.linear_fc1.weight3
962
  module.decoder.layers.6.mlp.experts.linear_fc1.weight4
 
963
  Params for bucket 36 (50331648 elements, 50331648 padded size):
964
  module.decoder.layers.6.mlp.experts.linear_fc1.weight5
 
965
  module.decoder.layers.6.mlp.experts.linear_fc1.weight8
966
  module.decoder.layers.6.mlp.experts.linear_fc1.weight6
967
+ module.decoder.layers.6.mlp.experts.linear_fc1.weight7
968
  Params for bucket 37 (50331648 elements, 50331648 padded size):
969
  module.decoder.layers.6.mlp.experts.linear_fc1.weight9
 
 
970
  module.decoder.layers.6.mlp.experts.linear_fc1.weight12
971
+ module.decoder.layers.6.mlp.experts.linear_fc1.weight11
972
+ module.decoder.layers.6.mlp.experts.linear_fc1.weight10
973
  Params for bucket 38 (44040192 elements, 44040192 padded size):
974
  module.decoder.layers.6.mlp.experts.linear_fc1.weight13
975
+ module.decoder.layers.6.mlp.experts.linear_fc1.weight15
976
  module.decoder.layers.5.mlp.experts.linear_fc2.weight0
977
  module.decoder.layers.6.mlp.experts.linear_fc1.weight14
 
978
  Params for bucket 39 (44040192 elements, 44040192 padded size):
979
  module.decoder.layers.5.mlp.experts.linear_fc2.weight1
980
  module.decoder.layers.5.mlp.experts.linear_fc2.weight3
 
1003
  module.decoder.layers.5.mlp.experts.linear_fc1.weight5
1004
  Params for bucket 43 (50331648 elements, 50331648 padded size):
1005
  module.decoder.layers.5.mlp.experts.linear_fc1.weight7
1006
+ module.decoder.layers.5.mlp.experts.linear_fc1.weight9
1007
  module.decoder.layers.5.mlp.experts.linear_fc1.weight10
1008
  module.decoder.layers.5.mlp.experts.linear_fc1.weight8
 
1009
  Params for bucket 44 (50331648 elements, 50331648 padded size):
1010
  module.decoder.layers.5.mlp.experts.linear_fc1.weight11
1011
+ module.decoder.layers.5.mlp.experts.linear_fc1.weight12
1012
  module.decoder.layers.5.mlp.experts.linear_fc1.weight13
1013
  module.decoder.layers.5.mlp.experts.linear_fc1.weight14
 
1014
  Params for bucket 45 (44040192 elements, 44040192 padded size):
 
 
 
 
1015
  module.decoder.layers.4.mlp.experts.linear_fc2.weight3
1016
  module.decoder.layers.4.mlp.experts.linear_fc2.weight1
1017
+ module.decoder.layers.5.mlp.experts.linear_fc1.weight15
1018
+ module.decoder.layers.4.mlp.experts.linear_fc2.weight0
1019
+ module.decoder.layers.4.mlp.experts.linear_fc2.weight2
1020
+ module.decoder.layers.4.mlp.experts.linear_fc2.weight4
1021
  Params for bucket 46 (44040192 elements, 44040192 padded size):
 
 
 
1022
  module.decoder.layers.4.mlp.experts.linear_fc2.weight5
 
1023
  module.decoder.layers.4.mlp.experts.linear_fc2.weight7
1024
+ module.decoder.layers.4.mlp.experts.linear_fc2.weight9
1025
  module.decoder.layers.4.mlp.experts.linear_fc2.weight11
1026
+ module.decoder.layers.4.mlp.experts.linear_fc2.weight8
1027
+ module.decoder.layers.4.mlp.experts.linear_fc2.weight6
1028
+ module.decoder.layers.4.mlp.experts.linear_fc2.weight10
1029
  Params for bucket 47 (50331648 elements, 50331648 padded size):
1030
+ module.decoder.layers.4.mlp.experts.linear_fc2.weight13
1031
+ module.decoder.layers.4.mlp.experts.linear_fc1.weight0
1032
+ module.decoder.layers.4.mlp.experts.linear_fc2.weight15
1033
  module.decoder.layers.4.mlp.experts.linear_fc2.weight12
1034
  module.decoder.layers.4.mlp.experts.linear_fc2.weight14
1035
  module.decoder.layers.4.mlp.experts.linear_fc1.weight1
 
 
 
1036
  Params for bucket 48 (50331648 elements, 50331648 padded size):
1037
  module.decoder.layers.4.mlp.experts.linear_fc1.weight2
1038
+ module.decoder.layers.4.mlp.experts.linear_fc1.weight3
1039
  module.decoder.layers.4.mlp.experts.linear_fc1.weight4
1040
  module.decoder.layers.4.mlp.experts.linear_fc1.weight5
 
1041
  Params for bucket 49 (50331648 elements, 50331648 padded size):
1042
  module.decoder.layers.4.mlp.experts.linear_fc1.weight6
1043
+ module.decoder.layers.4.mlp.experts.linear_fc1.weight9
1044
  module.decoder.layers.4.mlp.experts.linear_fc1.weight7
1045
  module.decoder.layers.4.mlp.experts.linear_fc1.weight8
 
1046
  Params for bucket 50 (50331648 elements, 50331648 padded size):
1047
  module.decoder.layers.4.mlp.experts.linear_fc1.weight10
 
1048
  module.decoder.layers.4.mlp.experts.linear_fc1.weight12
1049
  module.decoder.layers.4.mlp.experts.linear_fc1.weight13
1050
+ module.decoder.layers.4.mlp.experts.linear_fc1.weight11
1051
  Params for bucket 51 (44040192 elements, 44040192 padded size):
 
 
1052
  module.decoder.layers.4.mlp.experts.linear_fc1.weight14
 
1053
  module.decoder.layers.3.mlp.experts.linear_fc2.weight0
1054
+ module.decoder.layers.3.mlp.experts.linear_fc2.weight2
1055
+ module.decoder.layers.3.mlp.experts.linear_fc2.weight1
1056
+ module.decoder.layers.4.mlp.experts.linear_fc1.weight15
1057
  Params for bucket 52 (44040192 elements, 44040192 padded size):
1058
+ module.decoder.layers.3.mlp.experts.linear_fc2.weight6
1059
+ module.decoder.layers.3.mlp.experts.linear_fc2.weight4
1060
+ module.decoder.layers.3.mlp.experts.linear_fc2.weight8
1061
  module.decoder.layers.3.mlp.experts.linear_fc2.weight3
1062
  module.decoder.layers.3.mlp.experts.linear_fc2.weight7
1063
  module.decoder.layers.3.mlp.experts.linear_fc2.weight5
1064
  module.decoder.layers.3.mlp.experts.linear_fc2.weight9
 
 
 
1065
  Params for bucket 53 (50331648 elements, 50331648 padded size):
1066
+ module.decoder.layers.3.mlp.experts.linear_fc2.weight10
1067
+ module.decoder.layers.3.mlp.experts.linear_fc2.weight12
1068
+ module.decoder.layers.3.mlp.experts.linear_fc2.weight14
1069
  module.decoder.layers.3.mlp.experts.linear_fc2.weight13
1070
  module.decoder.layers.3.mlp.experts.linear_fc2.weight11
1071
  module.decoder.layers.3.mlp.experts.linear_fc2.weight15
1072
  module.decoder.layers.3.mlp.experts.linear_fc1.weight0
 
 
 
1073
  Params for bucket 54 (50331648 elements, 50331648 padded size):
1074
  module.decoder.layers.3.mlp.experts.linear_fc1.weight1
1075
  module.decoder.layers.3.mlp.experts.linear_fc1.weight4
 
1076
  module.decoder.layers.3.mlp.experts.linear_fc1.weight3
1077
+ module.decoder.layers.3.mlp.experts.linear_fc1.weight2
1078
  Params for bucket 55 (50331648 elements, 50331648 padded size):
1079
  module.decoder.layers.3.mlp.experts.linear_fc1.weight5
1080
  module.decoder.layers.3.mlp.experts.linear_fc1.weight7
 
1087
  module.decoder.layers.3.mlp.experts.linear_fc1.weight12
1088
  Params for bucket 57 (44040192 elements, 44040192 padded size):
1089
  module.decoder.layers.3.mlp.experts.linear_fc1.weight13
 
1090
  module.decoder.layers.2.mlp.experts.linear_fc2.weight0
1091
+ module.decoder.layers.3.mlp.experts.linear_fc1.weight14
1092
  module.decoder.layers.3.mlp.experts.linear_fc1.weight15
1093
  Params for bucket 58 (44040192 elements, 44040192 padded size):
 
 
 
 
1094
  module.decoder.layers.2.mlp.experts.linear_fc2.weight2
1095
  module.decoder.layers.2.mlp.experts.linear_fc2.weight4
1096
  module.decoder.layers.2.mlp.experts.linear_fc2.weight6
1097
+ module.decoder.layers.2.mlp.experts.linear_fc2.weight1
1098
+ module.decoder.layers.2.mlp.experts.linear_fc2.weight5
1099
+ module.decoder.layers.2.mlp.experts.linear_fc2.weight3
1100
+ module.decoder.layers.2.mlp.experts.linear_fc2.weight7
1101
  Params for bucket 59 (44040192 elements, 44040192 padded size):
 
 
 
1102
  module.decoder.layers.2.mlp.experts.linear_fc2.weight8
1103
  module.decoder.layers.2.mlp.experts.linear_fc2.weight12
1104
  module.decoder.layers.2.mlp.experts.linear_fc2.weight10
1105
  module.decoder.layers.2.mlp.experts.linear_fc2.weight14
1106
+ module.decoder.layers.2.mlp.experts.linear_fc2.weight11
1107
+ module.decoder.layers.2.mlp.experts.linear_fc2.weight9
1108
+ module.decoder.layers.2.mlp.experts.linear_fc2.weight13
1109
  Params for bucket 60 (44040192 elements, 44040192 padded size):
1110
  module.decoder.layers.2.mlp.experts.linear_fc2.weight15
1111
+ module.decoder.layers.2.mlp.experts.linear_fc1.weight1
1112
  module.decoder.layers.2.mlp.experts.linear_fc1.weight2
1113
  module.decoder.layers.2.mlp.experts.linear_fc1.weight0
 
1114
  Params for bucket 61 (50331648 elements, 50331648 padded size):
1115
  module.decoder.layers.2.mlp.experts.linear_fc1.weight3
1116
+ module.decoder.layers.2.mlp.experts.linear_fc1.weight4
1117
  module.decoder.layers.2.mlp.experts.linear_fc1.weight5
1118
  module.decoder.layers.2.mlp.experts.linear_fc1.weight6
 
1119
  Params for bucket 62 (50331648 elements, 50331648 padded size):
1120
  module.decoder.layers.2.mlp.experts.linear_fc1.weight7
 
1121
  module.decoder.layers.2.mlp.experts.linear_fc1.weight10
1122
  module.decoder.layers.2.mlp.experts.linear_fc1.weight8
1123
+ module.decoder.layers.2.mlp.experts.linear_fc1.weight9
1124
  Params for bucket 63 (50331648 elements, 50331648 padded size):
1125
  module.decoder.layers.2.mlp.experts.linear_fc1.weight11
1126
+ module.decoder.layers.2.mlp.experts.linear_fc1.weight14
1127
  module.decoder.layers.2.mlp.experts.linear_fc1.weight12
1128
  module.decoder.layers.2.mlp.experts.linear_fc1.weight13
 
1129
  Params for bucket 64 (44040192 elements, 44040192 padded size):
 
1130
  module.decoder.layers.1.mlp.experts.linear_fc2.weight0
1131
+ module.decoder.layers.1.mlp.experts.linear_fc2.weight2
1132
  module.decoder.layers.1.mlp.experts.linear_fc2.weight4
1133
  module.decoder.layers.2.mlp.experts.linear_fc1.weight15
 
1134
  module.decoder.layers.1.mlp.experts.linear_fc2.weight3
1135
+ module.decoder.layers.1.mlp.experts.linear_fc2.weight1
1136
  Params for bucket 65 (44040192 elements, 44040192 padded size):
 
1137
  module.decoder.layers.1.mlp.experts.linear_fc2.weight6
1138
+ module.decoder.layers.1.mlp.experts.linear_fc2.weight8
1139
  module.decoder.layers.1.mlp.experts.linear_fc2.weight10
1140
  module.decoder.layers.1.mlp.experts.linear_fc2.weight5
 
1141
  module.decoder.layers.1.mlp.experts.linear_fc2.weight7
1142
+ module.decoder.layers.1.mlp.experts.linear_fc2.weight9
1143
  module.decoder.layers.1.mlp.experts.linear_fc2.weight11
1144
  Params for bucket 66 (50331648 elements, 50331648 padded size):
1145
  module.decoder.layers.1.mlp.experts.linear_fc2.weight12
 
1146
  module.decoder.layers.1.mlp.experts.linear_fc1.weight0
1147
+ module.decoder.layers.1.mlp.experts.linear_fc2.weight14
1148
  module.decoder.layers.1.mlp.experts.linear_fc2.weight13
1149
+ module.decoder.layers.1.mlp.experts.linear_fc2.weight15
1150
  module.decoder.layers.1.mlp.experts.linear_fc1.weight1
1151
  Params for bucket 67 (50331648 elements, 50331648 padded size):
1152
  module.decoder.layers.1.mlp.experts.linear_fc1.weight2
1153
+ module.decoder.layers.1.mlp.experts.linear_fc1.weight5
1154
  module.decoder.layers.1.mlp.experts.linear_fc1.weight3
1155
  module.decoder.layers.1.mlp.experts.linear_fc1.weight4
 
1156
  Params for bucket 68 (50331648 elements, 50331648 padded size):
1157
  module.decoder.layers.1.mlp.experts.linear_fc1.weight6
1158
+ module.decoder.layers.1.mlp.experts.linear_fc1.weight8
1159
  module.decoder.layers.1.mlp.experts.linear_fc1.weight9
1160
  module.decoder.layers.1.mlp.experts.linear_fc1.weight7
 
1161
  Params for bucket 69 (50331648 elements, 50331648 padded size):
1162
  module.decoder.layers.1.mlp.experts.linear_fc1.weight10
1163
  module.decoder.layers.1.mlp.experts.linear_fc1.weight12
 
1164
  module.decoder.layers.1.mlp.experts.linear_fc1.weight11
1165
+ module.decoder.layers.1.mlp.experts.linear_fc1.weight13
1166
  Params for bucket 70 (44040192 elements, 44040192 padded size):
1167
  module.decoder.layers.1.mlp.experts.linear_fc1.weight14
 
1168
  module.decoder.layers.0.mlp.experts.linear_fc2.weight2
1169
+ module.decoder.layers.0.mlp.experts.linear_fc2.weight0
1170
  module.decoder.layers.1.mlp.experts.linear_fc1.weight15
1171
+ module.decoder.layers.0.mlp.experts.linear_fc2.weight1
1172
  Params for bucket 71 (44040192 elements, 44040192 padded size):
1173
  module.decoder.layers.0.mlp.experts.linear_fc2.weight4
1174
  module.decoder.layers.0.mlp.experts.linear_fc2.weight6
1175
  module.decoder.layers.0.mlp.experts.linear_fc2.weight8
1176
  module.decoder.layers.0.mlp.experts.linear_fc2.weight3
 
1177
  module.decoder.layers.0.mlp.experts.linear_fc2.weight5
1178
+ module.decoder.layers.0.mlp.experts.linear_fc2.weight7
1179
  module.decoder.layers.0.mlp.experts.linear_fc2.weight9
1180
  Params for bucket 72 (50331648 elements, 50331648 padded size):
1181
  module.decoder.layers.0.mlp.experts.linear_fc2.weight10
1182
  module.decoder.layers.0.mlp.experts.linear_fc2.weight14
1183
  module.decoder.layers.0.mlp.experts.linear_fc2.weight12
 
1184
  module.decoder.layers.0.mlp.experts.linear_fc2.weight11
1185
  module.decoder.layers.0.mlp.experts.linear_fc2.weight13
1186
  module.decoder.layers.0.mlp.experts.linear_fc2.weight15
1187
+ module.decoder.layers.0.mlp.experts.linear_fc1.weight0
1188
  Params for bucket 73 (50331648 elements, 50331648 padded size):
1189
  module.decoder.layers.0.mlp.experts.linear_fc1.weight1
1190
+ module.decoder.layers.0.mlp.experts.linear_fc1.weight2
1191
  module.decoder.layers.0.mlp.experts.linear_fc1.weight3
1192
  module.decoder.layers.0.mlp.experts.linear_fc1.weight4
 
1193
  Params for bucket 74 (50331648 elements, 50331648 padded size):
1194
  module.decoder.layers.0.mlp.experts.linear_fc1.weight5
1195
  module.decoder.layers.0.mlp.experts.linear_fc1.weight6
 
1196
  module.decoder.layers.0.mlp.experts.linear_fc1.weight8
1197
+ module.decoder.layers.0.mlp.experts.linear_fc1.weight7
1198
  Params for bucket 75 (50331648 elements, 50331648 padded size):
1199
  module.decoder.layers.0.mlp.experts.linear_fc1.weight9
 
1200
  module.decoder.layers.0.mlp.experts.linear_fc1.weight10
1201
  module.decoder.layers.0.mlp.experts.linear_fc1.weight11
1202
+ module.decoder.layers.0.mlp.experts.linear_fc1.weight12
1203
  Params for bucket 76 (37748736 elements, 37748736 padded size):
1204
  module.decoder.layers.0.mlp.experts.linear_fc1.weight13
1205
  module.decoder.layers.0.mlp.experts.linear_fc1.weight14
1206
  module.decoder.layers.0.mlp.experts.linear_fc1.weight15
1207
+ INFO:megatron.core.optimizer:Setting up optimizer with config OptimizerConfig(optimizer='adam', lr=0.0001, min_lr=1e-05, decoupled_lr=None, decoupled_min_lr=None, weight_decay=0.1, fp8_recipe='mxfp8', fp16=False, bf16=True, reuse_grad_buf_for_mxfp8_param_ag=False, params_dtype=torch.bfloat16, use_precision_aware_optimizer=False, store_param_remainders=True, main_grads_dtype=torch.float32, main_params_dtype=torch.float32, exp_avg_dtype=torch.float32, exp_avg_sq_dtype=torch.float32, loss_scale=None, initial_loss_scale=4294967296, min_loss_scale=1.0, loss_scale_window=1000, hysteresis=2, adam_beta1=0.9, adam_beta2=0.999, adam_eps=1e-08, sgd_momentum=0.9, use_distributed_optimizer=True, overlap_param_gather_with_optimizer_step=False, optimizer_cpu_offload=False, optimizer_offload_fraction=1.0, use_torch_optimizer_for_cpu_offload=False, overlap_cpu_optimizer_d2h_h2d=False, pin_cpu_grads=True, pin_cpu_params=True, clip_grad=1.0, log_num_zeros_in_grad=False, barrier_with_L1_time=True, timers=<megatron.core.timers.Timers object at 0x719122c89400>, config_logger_dir='')
1208
  INFO:megatron.core.optimizer_param_scheduler:> learning rate decay style: cosine
1209
+ [after model, optimizer, and learning rate scheduler are built] datetime: 2026-08-30 01:33:00
1210
  > building train, validation, and test datasets ...
1211
  > datasets target sizes (minimum size):
1212
  train: 96
 
1214
  test: 0
1215
  INFO:megatron.core.datasets.blended_megatron_dataset_config:Let split_matrix = [(0, 1.0), None, None]
1216
  > building train, validation, and test datasets for GPT ...
1217
+ INFO:megatron.core.datasets.blended_megatron_dataset_builder:Building GPTDataset splits with sizes=(96, 0, 0) and config=GPTDatasetConfig(random_seed=1234, sequence_length=8192, blend=(['/workspace/data/wikitext/wikitext103_gpt2_text_document'], None), blend_per_split=None, split='100,0,0', split_matrix=[(0, 1.0), None, None], num_dataset_builder_threads=1, path_to_cache=None, mmap_bin_files=True, mock=False, tokenizer=<megatron.training.tokenizer.tokenizer._GPT2BPETokenizer object at 0x718e7c70d6a0>, mid_level_dataset_surplus=0.005, reset_position_ids=False, reset_attention_mask=False, eod_mask_loss=False, create_attention_mask=True, drop_last_partial_validation_sequence=True, add_extra_token_to_sequence=True, object_storage_cache_path=None)
1218
  INFO:megatron.core.datasets.indexed_dataset:Load the _IndexReader from /workspace/data/wikitext/wikitext103_gpt2_text_document.idx
1219
  INFO:megatron.core.datasets.indexed_dataset: Extract the sequence lengths
1220
  INFO:megatron.core.datasets.indexed_dataset: Extract the sequence pointers
 
1227
  INFO:megatron.core.datasets.gpt_dataset: Load the shuffle index from eadc3756447e39d609e8aa728a91f451-GPTDataset-train-shuffle_index.npy
1228
  INFO:megatron.core.datasets.gpt_dataset:> total number of samples: 14536
1229
  > finished creating GPT datasets ...
1230
+ [after dataloaders are built] datetime: 2026-08-30 01:33:01
1231
  done with setup ...
1232
  (min, max) time across ranks (ms):
1233
+ model-and-optimizer-setup ......................: (3407.31, 3431.39)
1234
+ train/valid/test-data-iterators-setup ..........: (66.04, 78.01)
1235
  training ...
1236
  Overwriting rerun_state_machine.current_iteration from -1 to 0...
1237
+ [before the start of training step] datetime: 2026-08-30 01:33:01
1238
  INFO:moonep.api:MoonEP comm buffer sizes:
1239
  config : R=8, S=8192, H=4096, E=128, K=8, NvS=69600, NvS_padded=69632
1240
  sm_count : num_sms=32, num_sms_dedup=148
 
1246
  [MoonEP selfcheck 2/4] dropped groups empty OK | covered rows [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 128, 129, 130, 131, 132, 133, 134, 135, 136, 137, 138, 139, 140, 141, 142, 143] sizes [9344, 2688, 128, 24192, 128, 0, 128, 256, 128, 768, 10496, 128, 128, 2432, 128, 1536, 14208, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0] | duplicated experts [[69, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1], [88, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1], [55, 52, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1], [-1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1], [63, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1], [124, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1], [10, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1], [58, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1]]
1247
  [MoonEP selfcheck 3/4] dropped groups empty OK | covered rows [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 128, 129, 130, 131, 132, 133, 134, 135, 136, 137, 138, 139, 140, 141, 142, 143] sizes [128, 128, 128, 0, 128, 44416, 128, 128, 128, 6912, 128, 256, 128, 128, 128, 128, 13824, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0] | duplicated experts [[78, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1], [111, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1], [62, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1], [-1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1], [57, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1], [106, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1], [121, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1], [57, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1]]
1248
  [MoonEP selfcheck 4/4] dropped groups empty OK | covered rows [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 128, 129, 130, 131, 132, 133, 134, 135, 136, 137, 138, 139, 140, 141, 142, 143] sizes [2304, 128, 128, 512, 128, 12416, 128, 1152, 1280, 16768, 640, 29056, 128, 128, 128, 1664, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0] | duplicated experts [[-1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1], [118, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1], [11, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1], [118, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1], [39, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1], [11, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1], [73, 79, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1], [86, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1]]
1249
+ [2026-08-30 01:33:22] iteration 1/ 12 | consumed samples: 8 | elapsed time per iteration (ms): 20983.8 | throughput per GPU (TFLOP/s/GPU): 8.6 | learning rate: 2.000000E-05 | global batch size: 8 | lm loss: 1.115993E+01 | loss scale: 1.0 | grad norm: 44.807 | number of skipped iterations: 0 | number of nan iterations: 0 |
1250
  Number of parameters in transformer block in billions: 29.85
1251
  Number of parameters in embedding layers in billions: 0.41
1252
  Total number of parameters in billions: 30.26
1253
  Number of parameters in most loaded shard in billions: 30.2590
1254
  Theoretical memory footprints: weight and optimizer=216428.96 MB
1255
  [Rank 0] (after 1 iterations) memory (MB) | allocated: 67239.548828125 | max allocated: 67239.54931640625 | reserved: 68548.0 | max reserved: 68548.0
1256
+ [2026-08-30 01:33:22] iteration 2/ 12 | consumed samples: 16 | elapsed time per iteration (ms): 359.1 | throughput per GPU (TFLOP/s/GPU): 503.6 | learning rate: 4.000000E-05 | global batch size: 8 | lm loss: 1.117600E+01 | loss scale: 1.0 | grad norm: 44.337 | number of skipped iterations: 0 | number of nan iterations: 0 |
1257
+ [2026-08-30 01:33:22] iteration 3/ 12 | consumed samples: 24 | elapsed time per iteration (ms): 330.3 | throughput per GPU (TFLOP/s/GPU): 547.5 | learning rate: 6.000000E-05 | global batch size: 8 | lm loss: 9.685940E+00 | loss scale: 1.0 | grad norm: 62.876 | number of skipped iterations: 0 | number of nan iterations: 0 |
1258
  Capture CUDA graph for training!!!
1259
  CUDA graph capture done!!!
1260
+ [2026-08-30 01:33:23] iteration 4/ 12 | consumed samples: 32 | elapsed time per iteration (ms): 958.6 | throughput per GPU (TFLOP/s/GPU): 188.6 | learning rate: 8.000000E-05 | global batch size: 8 | lm loss: 1.338060E+01 | loss scale: 1.0 | grad norm: 20.309 | number of skipped iterations: 0 | number of nan iterations: 0 |
1261
+ [2026-08-30 01:33:24] iteration 5/ 12 | consumed samples: 40 | elapsed time per iteration (ms): 318.8 | throughput per GPU (TFLOP/s/GPU): 567.1 | learning rate: 1.000000E-04 | global batch size: 8 | lm loss: 1.050054E+01 | loss scale: 1.0 | grad norm: 95.222 | number of skipped iterations: 0 | number of nan iterations: 0 |
1262
+ [2026-08-30 01:33:24] iteration 6/ 12 | consumed samples: 48 | elapsed time per iteration (ms): 318.0 | throughput per GPU (TFLOP/s/GPU): 568.6 | learning rate: 7.750000E-05 | global batch size: 8 | lm loss: 1.042314E+01 | loss scale: 1.0 | grad norm: 50.787 | number of skipped iterations: 0 | number of nan iterations: 0 |
1263
+ [2026-08-30 01:33:24] iteration 7/ 12 | consumed samples: 56 | elapsed time per iteration (ms): 316.6 | throughput per GPU (TFLOP/s/GPU): 571.1 | learning rate: 3.250000E-05 | global batch size: 8 | lm loss: 1.026939E+01 | loss scale: 1.0 | grad norm: 6.314 | number of skipped iterations: 0 | number of nan iterations: 0 |
1264
+ [2026-08-30 01:33:25] iteration 8/ 12 | consumed samples: 64 | elapsed time per iteration (ms): 317.1 | throughput per GPU (TFLOP/s/GPU): 570.2 | learning rate: 1.000000E-05 | global batch size: 8 | lm loss: 9.807120E+00 | loss scale: 1.0 | grad norm: 5.549 | number of skipped iterations: 0 | number of nan iterations: 0 |
1265
+ [2026-08-30 01:33:25] iteration 9/ 12 | consumed samples: 72 | elapsed time per iteration (ms): 316.5 | throughput per GPU (TFLOP/s/GPU): 571.4 | learning rate: 1.000000E-05 | global batch size: 8 | lm loss: 9.543880E+00 | loss scale: 1.0 | grad norm: 5.309 | number of skipped iterations: 0 | number of nan iterations: 0 |
1266
+ [2026-08-30 01:33:25] iteration 10/ 12 | consumed samples: 80 | elapsed time per iteration (ms): 319.1 | throughput per GPU (TFLOP/s/GPU): 566.7 | learning rate: 1.000000E-05 | global batch size: 8 | lm loss: 9.444735E+00 | loss scale: 1.0 | grad norm: 5.055 | number of skipped iterations: 0 | number of nan iterations: 0 |
1267
+ [2026-08-30 01:33:25] iteration 11/ 12 | consumed samples: 88 | elapsed time per iteration (ms): 319.8 | throughput per GPU (TFLOP/s/GPU): 565.5 | learning rate: 1.000000E-05 | global batch size: 8 | lm loss: 9.322056E+00 | loss scale: 1.0 | grad norm: 4.732 | number of skipped iterations: 0 | number of nan iterations: 0 |
1268
+ [2026-08-30 01:33:26] iteration 12/ 12 | consumed samples: 96 | elapsed time per iteration (ms): 317.7 | throughput per GPU (TFLOP/s/GPU): 569.1 | learning rate: 1.000000E-05 | global batch size: 8 | lm loss: 9.206485E+00 | loss scale: 1.0 | grad norm: 4.466 | number of skipped iterations: 0 | number of nan iterations: 0 |
1269
+ [after training is done] datetime: 2026-08-30 01:33:26
1270
+ [rank0]:[W830 01:33:27.749790371 ProcessGroupNCCL.cpp:1505] Warning: WARNING: destroy_process_group() was not called before program exit, which can leak resources. For more info, please see https://pytorch.org/docs/stable/distributed.html#shutdown (function operator())
1271
+ [rank6]:[W830 01:33:28.189206623 ProcessGroupNCCL.cpp:1505] Warning: WARNING: destroy_process_group() was not called before program exit, which can leak resources. For more info, please see https://pytorch.org/docs/stable/distributed.html#shutdown (function operator())
1272
+ [rank3]:[W830 01:33:28.191561772 ProcessGroupNCCL.cpp:1505] Warning: WARNING: destroy_process_group() was not called before program exit, which can leak resources. For more info, please see https://pytorch.org/docs/stable/distributed.html#shutdown (function operator())
1273
+ [rank2]:[W830 01:33:28.192020147 ProcessGroupNCCL.cpp:1505] Warning: WARNING: destroy_process_group() was not called before program exit, which can leak resources. For more info, please see https://pytorch.org/docs/stable/distributed.html#shutdown (function operator())
1274
+ [rank5]:[W830 01:33:28.201417052 ProcessGroupNCCL.cpp:1505] Warning: WARNING: destroy_process_group() was not called before program exit, which can leak resources. For more info, please see https://pytorch.org/docs/stable/distributed.html#shutdown (function operator())
1275
+ [rank4]:[W830 01:33:29.333803578 ProcessGroupNCCL.cpp:1505] Warning: WARNING: destroy_process_group() was not called before program exit, which can leak resources. For more info, please see https://pytorch.org/docs/stable/distributed.html#shutdown (function operator())
1276
+ [rank7]:[W830 01:33:29.413464158 ProcessGroupNCCL.cpp:1505] Warning: WARNING: destroy_process_group() was not called before program exit, which can leak resources. For more info, please see https://pytorch.org/docs/stable/distributed.html#shutdown (function operator())
1277
+ [rank1]:[W830 01:33:29.627573333 ProcessGroupNCCL.cpp:1505] Warning: WARNING: destroy_process_group() was not called before program exit, which can leak resources. For more info, please see https://pytorch.org/docs/stable/distributed.html#shutdown (function operator())
1278
+ W0830 01:33:33.630000 382773 torch/distributed/elastic/agent/server/api.py:719] Received 15 death signal, shutting down workers
1279
+ W0830 01:33:33.631000 382773 torch/distributed/elastic/multiprocessing/api.py:900] Sending process 382951 closing signal SIGTERM
1280
+ W0830 01:33:33.631000 382773 torch/distributed/elastic/multiprocessing/api.py:900] Sending process 382952 closing signal SIGTERM
1281
+ W0830 01:33:33.631000 382773 torch/distributed/elastic/multiprocessing/api.py:900] Sending process 382953 closing signal SIGTERM
1282
+ W0830 01:33:33.632000 382773 torch/distributed/elastic/multiprocessing/api.py:900] Sending process 382955 closing signal SIGTERM
1283
+ W0830 01:33:33.632000 382773 torch/distributed/elastic/multiprocessing/api.py:900] Sending process 382956 closing signal SIGTERM
1284
+ W0830 01:33:33.634000 382773 torch/distributed/elastic/multiprocessing/api.py:900] Sending process 382958 closing signal SIGTERM
 
 
1285
  Traceback (most recent call last):
1286
  File "/usr/local/bin/torchrun", line 8, in <module>
1287
  sys.exit(main())
 
1309
  time.sleep(monitor_interval)
1310
  File "/usr/local/lib/python3.12/dist-packages/torch/distributed/elastic/multiprocessing/api.py", line 84, in _terminate_process_handler
1311
  raise SignalException(f"Process {os.getpid()} got signal: {sigval}", sigval=sigval)
1312
+ torch.distributed.elastic.multiprocessing.api.SignalException: Process 382773 got signal: 15
tierb-fixed-moonep-full/moonep/tierb/tierb_gbs8/QWEN3_235B__moonep__matched__seq8192__gbs8__capnone__async/rep3/mem_samples.csv CHANGED
@@ -50,7 +50,7 @@
50
  1, 4
51
  2, 4
52
  3, 4
53
- 4, 1
54
  5, 1
55
  6, 1
56
  7, 1
@@ -62,86 +62,78 @@
62
  5, 4
63
  6, 4
64
  7, 4
65
- 0, 4
66
- 1, 629
67
- 2, 47
68
- 3, 629
69
- 4, 45
70
- 5, 629
71
- 6, 629
72
- 7, 629
73
- 0, 909
74
- 1, 911
75
- 2, 931
76
- 3, 917
77
- 4, 937
78
- 5, 919
79
- 6, 951
80
- 7, 941
81
- 0, 1975
82
- 1, 1975
83
- 2, 1975
84
- 3, 1975
85
- 4, 1975
86
- 5, 1975
87
- 6, 1975
88
- 7, 1975
89
- 0, 36853
90
- 1, 38389
91
- 2, 32245
92
- 3, 40309
93
- 4, 46069
94
- 5, 46453
95
- 6, 33781
96
- 7, 42997
97
- 0, 48447
98
- 1, 48449
99
- 2, 48449
100
- 3, 48449
101
- 4, 48449
102
- 5, 48449
103
- 6, 48449
104
- 7, 48449
105
- 0, 56841
106
- 1, 56871
107
- 2, 56837
108
- 3, 56893
109
- 4, 56535
110
- 5, 56877
111
- 6, 56897
112
- 7, 56837
113
- 0, 59185
114
- 1, 59185
115
- 2, 59185
116
- 3, 59185
117
- 4, 58897
118
- 5, 59185
119
- 6, 59185
120
- 7, 59187
121
  0, 59731
122
  1, 59731
123
- 2, 59731
124
  3, 59731
125
  4, 59443
126
  5, 59731
127
  6, 59731
128
  7, 59733
129
  0, 59733
130
- 1, 59731
131
  2, 59733
132
  3, 59733
133
  4, 59445
134
  5, 59733
135
- 6, 59731
136
  7, 59733
137
- 0, 86987
138
- 1, 86961
139
- 2, 86961
140
- 3, 86961
141
- 4, 86921
142
- 5, 86961
143
- 6, 87025
144
- 7, 86961
145
  0, 87633
146
  1, 87673
147
  2, 87673
@@ -150,75 +142,75 @@
150
  5, 87673
151
  6, 87673
152
  7, 87673
153
- 0, 87839
154
- 1, 87901
155
- 2, 87905
156
- 3, 87879
157
- 4, 87905
158
- 5, 87921
159
- 6, 87937
160
- 7, 87941
161
- 0, 88685
162
- 1, 88725
163
- 2, 88725
164
- 3, 88725
165
- 4, 88685
166
- 5, 88725
167
- 6, 88725
168
- 7, 88725
169
- 0, 97521
170
- 1, 97571
171
- 2, 97513
172
- 3, 97545
173
- 4, 97461
174
- 5, 97629
175
- 6, 97583
176
- 7, 97555
177
- 0, 120267
178
- 1, 120303
179
- 2, 120303
180
- 3, 120303
181
- 4, 120303
182
- 5, 120303
183
- 6, 120303
184
- 7, 120303
185
- 0, 120267
186
- 1, 120303
187
- 2, 120303
188
- 3, 120303
189
- 4, 120303
190
- 5, 120303
191
- 6, 120303
192
- 7, 120303
193
- 0, 119621
194
- 1, 119657
195
- 2, 119657
196
- 3, 119657
197
- 4, 119657
198
- 5, 119395
199
- 6, 119657
200
- 7, 119657
201
- 0, 117327
202
- 1, 117819
203
- 2, 117495
204
- 3, 118077
205
- 4, 117751
206
- 5, 117431
207
- 6, 117495
208
- 7, 118077
209
- 0, 388
210
- 1, 114667
211
- 2, 114405
212
- 3, 114667
213
- 4, 114341
214
- 5, 92897
215
- 6, 114085
216
- 7, 114667
217
  0, 1
218
- 1, 1
219
- 2, 3
220
- 3, 4
221
- 4, 114021
222
  5, 4
223
  6, 4
224
  7, 4
 
50
  1, 4
51
  2, 4
52
  3, 4
53
+ 4, 2
54
  5, 1
55
  6, 1
56
  7, 1
 
62
  5, 4
63
  6, 4
64
  7, 4
65
+ 0, 819
66
+ 1, 819
67
+ 2, 819
68
+ 3, 819
69
+ 4, 819
70
+ 5, 819
71
+ 6, 819
72
+ 7, 819
73
+ 0, 1537
74
+ 1, 1537
75
+ 2, 1537
76
+ 3, 1537
77
+ 4, 1537
78
+ 5, 1537
79
+ 6, 1537
80
+ 7, 1537
81
+ 0, 2551
82
+ 1, 2551
83
+ 2, 2551
84
+ 3, 2551
85
+ 4, 2551
86
+ 5, 2551
87
+ 6, 2551
88
+ 7, 2551
89
+ 0, 46711
90
+ 1, 46701
91
+ 2, 46709
92
+ 3, 46689
93
+ 4, 46711
94
+ 5, 46711
95
+ 6, 46711
96
+ 7, 46711
97
+ 0, 56781
98
+ 1, 56813
99
+ 2, 56805
100
+ 3, 56843
101
+ 4, 56537
102
+ 5, 56875
103
+ 6, 56909
104
+ 7, 56871
105
+ 0, 58753
106
+ 1, 58717
107
+ 2, 58755
108
+ 3, 58787
109
+ 4, 58473
110
+ 5, 58753
111
+ 6, 58781
112
+ 7, 58741
 
 
 
 
 
 
 
 
113
  0, 59731
114
  1, 59731
115
+ 2, 59733
116
  3, 59731
117
  4, 59443
118
  5, 59731
119
  6, 59731
120
  7, 59733
121
  0, 59733
122
+ 1, 59733
123
  2, 59733
124
  3, 59733
125
  4, 59445
126
  5, 59733
127
+ 6, 59733
128
  7, 59733
129
+ 0, 85347
130
+ 1, 85387
131
+ 2, 85387
132
+ 3, 85387
133
+ 4, 85347
134
+ 5, 85387
135
+ 6, 85387
136
+ 7, 85387
137
  0, 87633
138
  1, 87673
139
  2, 87673
 
142
  5, 87673
143
  6, 87673
144
  7, 87673
145
+ 0, 87633
146
+ 1, 87673
147
+ 2, 87673
148
+ 3, 87673
149
+ 4, 87633
150
+ 5, 87673
151
+ 6, 87673
152
+ 7, 87673
153
+ 0, 88039
154
+ 1, 88079
155
+ 2, 88079
156
+ 3, 88079
157
+ 4, 88039
158
+ 5, 88079
159
+ 6, 88079
160
+ 7, 88079
161
+ 0, 97463
162
+ 1, 97489
163
+ 2, 97505
164
+ 3, 97503
165
+ 4, 97421
166
+ 5, 97551
167
+ 6, 97573
168
+ 7, 97533
169
+ 0, 120273
170
+ 1, 120309
171
+ 2, 120309
172
+ 3, 120309
173
+ 4, 120309
174
+ 5, 120309
175
+ 6, 120309
176
+ 7, 120309
177
+ 0, 120273
178
+ 1, 120309
179
+ 2, 120309
180
+ 3, 120309
181
+ 4, 120309
182
+ 5, 120309
183
+ 6, 120309
184
+ 7, 120309
185
+ 0, 119045
186
+ 1, 119663
187
+ 2, 119663
188
+ 3, 119663
189
+ 4, 119663
190
+ 5, 119663
191
+ 6, 119337
192
+ 7, 119017
193
+ 0, 117077
194
+ 1, 118083
195
+ 2, 117433
196
+ 3, 118083
197
+ 4, 117757
198
+ 5, 117437
199
+ 6, 117437
200
+ 7, 116787
201
+ 0, 92867
202
+ 1, 115757
203
+ 2, 94633
204
+ 3, 115757
205
+ 4, 115757
206
+ 5, 115757
207
+ 6, 110761
208
+ 7, 4
209
  0, 1
210
+ 1, 4
211
+ 2, 4
212
+ 3, 114027
213
+ 4, 4
214
  5, 4
215
  6, 4
216
  7, 4
tierb-fixed-moonep-full/moonep/tierb/tierb_gbs8/QWEN3_235B__moonep__matched__seq8192__gbs8__capnone__async/rep3/run.env CHANGED
@@ -3,7 +3,7 @@ config=QWEN3_235B
3
  family=tierB
4
  repo=/workspace/Megatron-MoE-Echo
5
  branch=moonep_backend_async_grad_reduce
6
- commit=7bc235bf
7
  moonep_source_dir=/workspace/MoonEP
8
  moonep_source_branch=megatron-ddp-symmetric-params
9
  moonep_source_commit=d8bc666
 
3
  family=tierB
4
  repo=/workspace/Megatron-MoE-Echo
5
  branch=moonep_backend_async_grad_reduce
6
+ commit=2e68c8ad
7
  moonep_source_dir=/workspace/MoonEP
8
  moonep_source_branch=megatron-ddp-symmetric-params
9
  moonep_source_commit=d8bc666
tierb-fixed-moonep-full/moonep/tierb/tierb_gbs8/QWEN3_235B__moonep__matched__seq8192__gbs8__capnone__async/rep3/train.log CHANGED
@@ -1,25 +1,23 @@
1
- [23:47:32] WARN: moonep_backend_async_grad_reduce lacks --moe-router-fusion; QWEN3_235B runs unfused (timing only, routing identical)
2
  ==============================================================
3
  QWEN3_235B / moonep (slots/rank=16 policy=matched)
4
  E=128 K=8 H'=1536 m=16
5
  L=12 seq=8192 gbs=8 (1 microbatches)
6
  cap_mode=none cap=none cg=1 recompute=none
7
  memory_dump=none snapshots=/workspace/tierb-fixed-moonep-full/moonep/tierb/tierb_gbs8/QWEN3_235B__moonep__matched__seq8192__gbs8__capnone__async/rep3/memory_snapshots/snapshot.pickle
8
- repo=/workspace/Megatron-MoE-Echo @ 7bc235bf
9
  ==============================================================
10
- W0829 23:47:34.311000 242741 torch/distributed/run.py:766]
11
- W0829 23:47:34.311000 242741 torch/distributed/run.py:766] *****************************************
12
- W0829 23:47:34.311000 242741 torch/distributed/run.py:766] Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed.
13
- W0829 23:47:34.311000 242741 torch/distributed/run.py:766] *****************************************
14
  /workspace/Megatron-MoE-Echo/megatron/training/arguments.py:831: UserWarning: Disabling sequence parallelism because tensor model parallelism is disabled
15
  warnings.warn("Disabling sequence parallelism because tensor model parallelism is disabled")
 
16
  /workspace/Megatron-MoE-Echo/megatron/training/arguments.py:831: UserWarning: Disabling sequence parallelism because tensor model parallelism is disabled
17
  warnings.warn("Disabling sequence parallelism because tensor model parallelism is disabled")
18
  /workspace/Megatron-MoE-Echo/megatron/training/arguments.py:831: UserWarning: Disabling sequence parallelism because tensor model parallelism is disabled
19
  warnings.warn("Disabling sequence parallelism because tensor model parallelism is disabled")
20
- /workspace/Megatron-MoE-Echo/megatron/training/arguments.py:831: UserWarning: Disabling sequence parallelism because tensor model parallelism is disabled
21
- warnings.warn("Disabling sequence parallelism because tensor model parallelism is disabled")
22
- WARNING: one_logger package is required to enable e2e metrics tracking. please go to https://confluence.nvidia.com/display/MLWFO/Package+Repositories for details to install it
23
  /workspace/Megatron-MoE-Echo/megatron/training/arguments.py:831: UserWarning: Disabling sequence parallelism because tensor model parallelism is disabled
24
  warnings.warn("Disabling sequence parallelism because tensor model parallelism is disabled")
25
  /workspace/Megatron-MoE-Echo/megatron/training/arguments.py:831: UserWarning: Disabling sequence parallelism because tensor model parallelism is disabled
@@ -574,54 +572,56 @@ using torch.bfloat16 for parameters ...
574
  -------------------- end of arguments ---------------------
575
  INFO:megatron.core.num_microbatches_calculator:setting number of microbatches to constant 1
576
  > building GPT2BPETokenizer tokenizer ...
 
 
577
  > padded vocab (size: 50257) with 47 dummy tokens (new size: 50304)
578
  WARNING:megatron.core.rerun_state_machine:RerunStateMachine initialized in mode RerunMode.VALIDATE_RESULTS
579
  > initializing torch distributed ...
580
  [Gloo] Rank 0 is connected to 7 peer ranks. Expected number of connected peer ranks is : 7
 
581
  [Gloo] Rank 1 is connected to 7 peer ranks. Expected number of connected peer ranks is : 7
582
  [Gloo] Rank 2 is connected to 7 peer ranks. Expected number of connected peer ranks is : 7
583
- [Gloo] Rank 3 is connected to 7 peer ranks. Expected number of connected peer ranks is : 7
584
  [Gloo] Rank 4 is connected to 7 peer ranks. Expected number of connected peer ranks is : 7
585
- [Gloo] Rank 5 is connected to 7 peer ranks. Expected number of connected peer ranks is : 7
586
- [Gloo] Rank [Gloo] Rank 7 is connected to 76 peer ranks. is connected to Expected number of connected peer ranks is : 77 peer ranks. Expected number of connected peer ranks is :
587
  7
588
- [Gloo] Rank [Gloo] Rank 0 is connected to 17 is connected to peer ranks. 7Expected number of connected peer ranks is : peer ranks. 7Expected number of connected peer ranks is : [Gloo] Rank
589
- 72 is connected to
590
- 7 peer ranks. Expected number of connected peer ranks is : 7
591
- [Gloo] Rank [Gloo] Rank 54 is connected to is connected to 77 peer ranks. peer ranks. Expected number of connected peer ranks is : Expected number of connected peer ranks is : 77
592
-
593
  [Gloo] Rank 3 is connected to 7 peer ranks. Expected number of connected peer ranks is : 7
594
- [Gloo] Rank [Gloo] Rank 7 is connected to 67 is connected to 7 peer ranks. peer ranks. Expected number of connected peer ranks is : 7Expected number of connected peer ranks is :
595
  7
596
  [Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
597
  > initialized tensor model parallel with size 1
598
  > initialized pipeline model parallel with size 1
599
  > setting random seeds to 1234 ...
600
- [Gloo] Rank [Gloo] Rank 0[Gloo] Rank 0 is connected to 0 is connected to 0 is connected to 0 peer ranks. 0 peer ranks. Expected number of connected peer ranks is : peer ranks. Expected number of connected peer ranks is : 0Expected number of connected peer ranks is : 0
601
- 0
602
 
 
 
 
603
  [Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
604
  [Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
605
- [Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
606
- [Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
607
- [rank2]:[W829 23:47:52.047774281 ProcessGroupNCCL.cpp:4789] [PG ID 0 PG GUID 0 Rank 2] using GPU 2 as device used by this process is currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect. You can specify device_id in init_process_group() to force use of a particular device.
608
- [rank4]:[W829 23:47:52.064582906 ProcessGroupNCCL.cpp:4789] [PG ID 0 PG GUID 0 Rank 4] using GPU 4 as device used by this process is currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect. You can specify device_id in init_process_group() to force use of a particular device.
609
- [rank6]:[W829 23:47:52.065910975 ProcessGroupNCCL.cpp:4789] [PG ID 0 PG GUID 0 Rank 6] using GPU 6 as device used by this process is currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect. You can specify device_id in init_process_group() to force use of a particular device.
610
- [rank5]:[W829 23:47:52.065942810 ProcessGroupNCCL.cpp:4789] [PG ID 0 PG GUID 0 Rank 5] using GPU 5 as device used by this process is currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect. You can specify device_id in init_process_group() to force use of a particular device.
611
- [rank3]:[W829 23:47:52.066506087 ProcessGroupNCCL.cpp:4789] [PG ID 0 PG GUID 0 Rank 3] using GPU 3 as device used by this process is currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect. You can specify device_id in init_process_group() to force use of a particular device.
612
- [rank1]:[W829 23:47:52.066710093 ProcessGroupNCCL.cpp:4789] [PG ID 0 PG GUID 0 Rank 1] using GPU 1 as device used by this process is currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect. You can specify device_id in init_process_group() to force use of a particular device.
613
- [rank7]:[W829 23:47:52.068582353 ProcessGroupNCCL.cpp:4789] [PG ID 0 PG GUID 0 Rank 7] using GPU 7 as device used by this process is currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect. You can specify device_id in init_process_group() to force use of a particular device.
614
  > compiling dataset index builder ...
615
  make: Entering directory '/workspace/Megatron-MoE-Echo/megatron/core/datasets'
616
  make: Nothing to be done for 'default'.
617
  make: Leaving directory '/workspace/Megatron-MoE-Echo/megatron/core/datasets'
618
- >>> done with dataset index builder. Compilation time: 0.035 seconds
619
  WARNING: constraints for invoking optimized fused softmax kernel are not met. We default back to unfused kernel invocations.
620
  > compiling and loading fused kernels ...
621
- [rank0]:[W829 23:47:53.247994951 ProcessGroupNCCL.cpp:4789] [PG ID 0 PG GUID 0 Rank 0] using GPU 0 as device used by this process is currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect. You can specify device_id in init_process_group() to force use of a particular device.
622
- >>> done with compiling and loading fused kernels. Compilation time: 3.776 seconds
623
- time to initialize megatron (seconds): 7.432
624
- [after megatron is initialized] datetime: 2026-08-29 23:47:58
625
  building GPT model ...
626
  /usr/local/lib/python3.12/dist-packages/transformer_engine/pytorch/cpu_offload.py:708: DeprecationWarning: Offloading weights is deprecated. Using offload_weights=True does not have any effect.
627
  warnings.warn(
@@ -645,36 +645,36 @@ INFO:megatron.core.distributed.param_and_grad_buffer:Number of buckets for gradi
645
  Params for bucket 1 (206045184 elements, 206045184 padded size):
646
  module.output_layer.weight
647
  Params for bucket 2 (71840000 elements, 71840000 padded size):
648
- module.decoder.layers.11.pre_mlp_layernorm.weight
649
  module.decoder.layers.11.self_attention.q_layernorm.weight
 
650
  module.decoder.layers.11.mlp.router.weight
651
  module.decoder.layers.11.self_attention.k_layernorm.weight
652
  module.decoder.layers.11.self_attention.linear_qkv.weight
653
  module.decoder.layers.11.self_attention.linear_qkv.layer_norm_weight
654
- module.decoder.final_layernorm.weight
655
  module.decoder.layers.11.self_attention.linear_proj.weight
 
656
  Params for bucket 3 (71835904 elements, 71835904 padded size):
 
657
  module.decoder.layers.10.self_attention.linear_qkv.weight
 
658
  module.decoder.layers.10.self_attention.linear_qkv.layer_norm_weight
659
  module.decoder.layers.10.mlp.router.weight
660
- module.decoder.layers.10.pre_mlp_layernorm.weight
661
  module.decoder.layers.10.self_attention.k_layernorm.weight
662
- module.decoder.layers.10.self_attention.q_layernorm.weight
663
  module.decoder.layers.10.self_attention.linear_proj.weight
664
  Params for bucket 4 (71835904 elements, 71835904 padded size):
665
  module.decoder.layers.9.self_attention.k_layernorm.weight
 
666
  module.decoder.layers.9.self_attention.linear_proj.weight
667
  module.decoder.layers.9.mlp.router.weight
668
  module.decoder.layers.9.pre_mlp_layernorm.weight
669
- module.decoder.layers.9.self_attention.linear_qkv.weight
670
  module.decoder.layers.9.self_attention.q_layernorm.weight
671
  module.decoder.layers.9.self_attention.linear_qkv.layer_norm_weight
672
  Params for bucket 5 (71835904 elements, 71835904 padded size):
673
  module.decoder.layers.8.mlp.router.weight
674
  module.decoder.layers.8.pre_mlp_layernorm.weight
 
675
  module.decoder.layers.8.self_attention.linear_qkv.weight
676
  module.decoder.layers.8.self_attention.q_layernorm.weight
677
- module.decoder.layers.8.self_attention.k_layernorm.weight
678
  module.decoder.layers.8.self_attention.linear_qkv.layer_norm_weight
679
  module.decoder.layers.8.self_attention.linear_proj.weight
680
  Params for bucket 6 (71835904 elements, 71835904 padded size):
@@ -683,62 +683,62 @@ Params for bucket 6 (71835904 elements, 71835904 padded size):
683
  module.decoder.layers.7.self_attention.k_layernorm.weight
684
  module.decoder.layers.7.self_attention.linear_qkv.weight
685
  module.decoder.layers.7.self_attention.q_layernorm.weight
686
- module.decoder.layers.7.self_attention.linear_proj.weight
687
  module.decoder.layers.7.self_attention.linear_qkv.layer_norm_weight
 
688
  Params for bucket 7 (71835904 elements, 71835904 padded size):
689
  module.decoder.layers.6.mlp.router.weight
690
- module.decoder.layers.6.self_attention.q_layernorm.weight
691
- module.decoder.layers.6.self_attention.linear_qkv.weight
692
  module.decoder.layers.6.pre_mlp_layernorm.weight
 
693
  module.decoder.layers.6.self_attention.k_layernorm.weight
694
  module.decoder.layers.6.self_attention.linear_qkv.layer_norm_weight
 
695
  module.decoder.layers.6.self_attention.linear_proj.weight
696
  Params for bucket 8 (71835904 elements, 71835904 padded size):
697
- module.decoder.layers.5.self_attention.q_layernorm.weight
698
  module.decoder.layers.5.pre_mlp_layernorm.weight
 
699
  module.decoder.layers.5.self_attention.linear_qkv.layer_norm_weight
700
- module.decoder.layers.5.mlp.router.weight
701
  module.decoder.layers.5.self_attention.linear_qkv.weight
702
  module.decoder.layers.5.self_attention.k_layernorm.weight
703
  module.decoder.layers.5.self_attention.linear_proj.weight
704
  Params for bucket 9 (71835904 elements, 71835904 padded size):
705
- module.decoder.layers.4.mlp.router.weight
706
  module.decoder.layers.4.pre_mlp_layernorm.weight
707
  module.decoder.layers.4.self_attention.q_layernorm.weight
 
 
708
  module.decoder.layers.4.self_attention.k_layernorm.weight
709
  module.decoder.layers.4.self_attention.linear_qkv.weight
710
- module.decoder.layers.4.self_attention.linear_qkv.layer_norm_weight
711
  module.decoder.layers.4.self_attention.linear_proj.weight
712
  Params for bucket 10 (71835904 elements, 71835904 padded size):
713
  module.decoder.layers.3.mlp.router.weight
714
- module.decoder.layers.3.pre_mlp_layernorm.weight
715
  module.decoder.layers.3.self_attention.q_layernorm.weight
716
  module.decoder.layers.3.self_attention.linear_qkv.weight
 
717
  module.decoder.layers.3.self_attention.k_layernorm.weight
718
  module.decoder.layers.3.self_attention.linear_qkv.layer_norm_weight
719
  module.decoder.layers.3.self_attention.linear_proj.weight
720
  Params for bucket 11 (71835904 elements, 71835904 padded size):
721
  module.decoder.layers.2.mlp.router.weight
 
722
  module.decoder.layers.2.pre_mlp_layernorm.weight
723
- module.decoder.layers.2.self_attention.k_layernorm.weight
724
  module.decoder.layers.2.self_attention.linear_qkv.weight
725
- module.decoder.layers.2.self_attention.q_layernorm.weight
726
  module.decoder.layers.2.self_attention.linear_qkv.layer_norm_weight
727
  module.decoder.layers.2.self_attention.linear_proj.weight
728
  Params for bucket 12 (71835904 elements, 71835904 padded size):
729
  module.decoder.layers.1.mlp.router.weight
730
  module.decoder.layers.1.pre_mlp_layernorm.weight
731
  module.decoder.layers.1.self_attention.k_layernorm.weight
732
- module.decoder.layers.1.self_attention.q_layernorm.weight
733
  module.decoder.layers.1.self_attention.linear_qkv.weight
 
734
  module.decoder.layers.1.self_attention.linear_qkv.layer_norm_weight
735
  module.decoder.layers.1.self_attention.linear_proj.weight
736
  Params for bucket 13 (71835904 elements, 71835904 padded size):
737
- module.decoder.layers.0.self_attention.k_layernorm.weight
738
  module.decoder.layers.0.self_attention.q_layernorm.weight
739
- module.decoder.layers.0.pre_mlp_layernorm.weight
740
  module.decoder.layers.0.self_attention.linear_proj.weight
741
- module.decoder.layers.0.mlp.router.weight
 
742
  module.decoder.layers.0.self_attention.linear_qkv.weight
743
  module.decoder.layers.0.self_attention.linear_qkv.layer_norm_weight
744
  Params for bucket 14 (206045184 elements, 206045184 padded size):
@@ -746,8 +746,8 @@ Params for bucket 14 (206045184 elements, 206045184 padded size):
746
  INFO:megatron.core.distributed.param_and_grad_buffer:Number of buckets for gradient all-reduce / reduce-scatter: 76
747
  Params for bucket 1 (44040192 elements, 44040192 padded size):
748
  module.decoder.layers.11.mlp.experts.linear_fc2.weight0
749
- module.decoder.layers.11.mlp.experts.linear_fc2.weight2
750
  module.decoder.layers.11.mlp.experts.linear_fc2.weight4
 
751
  module.decoder.layers.11.mlp.experts.linear_fc2.weight6
752
  module.decoder.layers.11.mlp.experts.linear_fc2.weight3
753
  module.decoder.layers.11.mlp.experts.linear_fc2.weight1
@@ -757,40 +757,40 @@ Params for bucket 2 (44040192 elements, 44040192 padded size):
757
  module.decoder.layers.11.mlp.experts.linear_fc2.weight10
758
  module.decoder.layers.11.mlp.experts.linear_fc2.weight12
759
  module.decoder.layers.11.mlp.experts.linear_fc2.weight7
760
- module.decoder.layers.11.mlp.experts.linear_fc2.weight9
761
  module.decoder.layers.11.mlp.experts.linear_fc2.weight11
 
762
  module.decoder.layers.11.mlp.experts.linear_fc2.weight13
763
  Params for bucket 3 (50331648 elements, 50331648 padded size):
764
  module.decoder.layers.11.mlp.experts.linear_fc2.weight14
765
  module.decoder.layers.11.mlp.experts.linear_fc1.weight1
766
- module.decoder.layers.11.mlp.experts.linear_fc2.weight15
767
  module.decoder.layers.11.mlp.experts.linear_fc1.weight2
 
768
  module.decoder.layers.11.mlp.experts.linear_fc1.weight0
769
  Params for bucket 4 (50331648 elements, 50331648 padded size):
770
  module.decoder.layers.11.mlp.experts.linear_fc1.weight3
771
  module.decoder.layers.11.mlp.experts.linear_fc1.weight5
772
- module.decoder.layers.11.mlp.experts.linear_fc1.weight6
773
  module.decoder.layers.11.mlp.experts.linear_fc1.weight4
 
774
  Params for bucket 5 (50331648 elements, 50331648 padded size):
775
  module.decoder.layers.11.mlp.experts.linear_fc1.weight7
776
- module.decoder.layers.11.mlp.experts.linear_fc1.weight9
777
  module.decoder.layers.11.mlp.experts.linear_fc1.weight8
 
778
  module.decoder.layers.11.mlp.experts.linear_fc1.weight10
779
  Params for bucket 6 (50331648 elements, 50331648 padded size):
780
  module.decoder.layers.11.mlp.experts.linear_fc1.weight11
 
781
  module.decoder.layers.11.mlp.experts.linear_fc1.weight12
782
  module.decoder.layers.11.mlp.experts.linear_fc1.weight13
783
- module.decoder.layers.11.mlp.experts.linear_fc1.weight14
784
  Params for bucket 7 (44040192 elements, 44040192 padded size):
785
  module.decoder.layers.11.mlp.experts.linear_fc1.weight15
786
  module.decoder.layers.10.mlp.experts.linear_fc2.weight0
787
  module.decoder.layers.10.mlp.experts.linear_fc2.weight2
788
  module.decoder.layers.10.mlp.experts.linear_fc2.weight4
789
- module.decoder.layers.10.mlp.experts.linear_fc2.weight1
790
  module.decoder.layers.10.mlp.experts.linear_fc2.weight3
 
791
  Params for bucket 8 (44040192 elements, 44040192 padded size):
792
- module.decoder.layers.10.mlp.experts.linear_fc2.weight8
793
  module.decoder.layers.10.mlp.experts.linear_fc2.weight6
 
794
  module.decoder.layers.10.mlp.experts.linear_fc2.weight10
795
  module.decoder.layers.10.mlp.experts.linear_fc2.weight5
796
  module.decoder.layers.10.mlp.experts.linear_fc2.weight7
@@ -798,68 +798,68 @@ Params for bucket 8 (44040192 elements, 44040192 padded size):
798
  module.decoder.layers.10.mlp.experts.linear_fc2.weight11
799
  Params for bucket 9 (50331648 elements, 50331648 padded size):
800
  module.decoder.layers.10.mlp.experts.linear_fc2.weight12
 
801
  module.decoder.layers.10.mlp.experts.linear_fc2.weight14
802
- module.decoder.layers.10.mlp.experts.linear_fc1.weight1
803
  module.decoder.layers.10.mlp.experts.linear_fc2.weight13
804
  module.decoder.layers.10.mlp.experts.linear_fc2.weight15
805
- module.decoder.layers.10.mlp.experts.linear_fc1.weight0
806
  Params for bucket 10 (50331648 elements, 50331648 padded size):
807
  module.decoder.layers.10.mlp.experts.linear_fc1.weight2
808
- module.decoder.layers.10.mlp.experts.linear_fc1.weight4
809
  module.decoder.layers.10.mlp.experts.linear_fc1.weight5
810
  module.decoder.layers.10.mlp.experts.linear_fc1.weight3
 
811
  Params for bucket 11 (50331648 elements, 50331648 padded size):
812
  module.decoder.layers.10.mlp.experts.linear_fc1.weight6
813
- module.decoder.layers.10.mlp.experts.linear_fc1.weight7
814
  module.decoder.layers.10.mlp.experts.linear_fc1.weight8
815
  module.decoder.layers.10.mlp.experts.linear_fc1.weight9
 
816
  Params for bucket 12 (50331648 elements, 50331648 padded size):
817
  module.decoder.layers.10.mlp.experts.linear_fc1.weight10
 
818
  module.decoder.layers.10.mlp.experts.linear_fc1.weight13
819
  module.decoder.layers.10.mlp.experts.linear_fc1.weight11
820
- module.decoder.layers.10.mlp.experts.linear_fc1.weight12
821
  Params for bucket 13 (44040192 elements, 44040192 padded size):
822
- module.decoder.layers.9.mlp.experts.linear_fc2.weight1
823
- module.decoder.layers.10.mlp.experts.linear_fc1.weight15
824
  module.decoder.layers.10.mlp.experts.linear_fc1.weight14
825
- module.decoder.layers.9.mlp.experts.linear_fc2.weight0
826
  module.decoder.layers.9.mlp.experts.linear_fc2.weight2
 
 
 
827
  Params for bucket 14 (44040192 elements, 44040192 padded size):
828
- module.decoder.layers.9.mlp.experts.linear_fc2.weight3
829
- module.decoder.layers.9.mlp.experts.linear_fc2.weight7
830
- module.decoder.layers.9.mlp.experts.linear_fc2.weight5
831
- module.decoder.layers.9.mlp.experts.linear_fc2.weight9
832
  module.decoder.layers.9.mlp.experts.linear_fc2.weight6
833
  module.decoder.layers.9.mlp.experts.linear_fc2.weight4
834
  module.decoder.layers.9.mlp.experts.linear_fc2.weight8
 
 
 
 
835
  Params for bucket 15 (50331648 elements, 50331648 padded size):
836
- module.decoder.layers.9.mlp.experts.linear_fc2.weight13
 
 
837
  module.decoder.layers.9.mlp.experts.linear_fc2.weight11
 
838
  module.decoder.layers.9.mlp.experts.linear_fc2.weight15
839
  module.decoder.layers.9.mlp.experts.linear_fc1.weight0
840
- module.decoder.layers.9.mlp.experts.linear_fc2.weight10
841
- module.decoder.layers.9.mlp.experts.linear_fc2.weight14
842
- module.decoder.layers.9.mlp.experts.linear_fc2.weight12
843
  Params for bucket 16 (50331648 elements, 50331648 padded size):
844
  module.decoder.layers.9.mlp.experts.linear_fc1.weight1
845
- module.decoder.layers.9.mlp.experts.linear_fc1.weight3
846
  module.decoder.layers.9.mlp.experts.linear_fc1.weight4
847
  module.decoder.layers.9.mlp.experts.linear_fc1.weight2
 
848
  Params for bucket 17 (50331648 elements, 50331648 padded size):
849
  module.decoder.layers.9.mlp.experts.linear_fc1.weight5
850
- module.decoder.layers.9.mlp.experts.linear_fc1.weight6
851
  module.decoder.layers.9.mlp.experts.linear_fc1.weight7
852
  module.decoder.layers.9.mlp.experts.linear_fc1.weight8
 
853
  Params for bucket 18 (50331648 elements, 50331648 padded size):
854
  module.decoder.layers.9.mlp.experts.linear_fc1.weight9
 
855
  module.decoder.layers.9.mlp.experts.linear_fc1.weight12
856
  module.decoder.layers.9.mlp.experts.linear_fc1.weight10
857
- module.decoder.layers.9.mlp.experts.linear_fc1.weight11
858
  Params for bucket 19 (44040192 elements, 44040192 padded size):
859
  module.decoder.layers.9.mlp.experts.linear_fc1.weight13
860
- module.decoder.layers.8.mlp.experts.linear_fc2.weight0
861
  module.decoder.layers.9.mlp.experts.linear_fc1.weight14
862
  module.decoder.layers.9.mlp.experts.linear_fc1.weight15
 
863
  Params for bucket 20 (44040192 elements, 44040192 padded size):
864
  module.decoder.layers.8.mlp.experts.linear_fc2.weight1
865
  module.decoder.layers.8.mlp.experts.linear_fc2.weight5
@@ -873,286 +873,286 @@ Params for bucket 21 (44040192 elements, 44040192 padded size):
873
  module.decoder.layers.8.mlp.experts.linear_fc2.weight9
874
  module.decoder.layers.8.mlp.experts.linear_fc2.weight13
875
  module.decoder.layers.8.mlp.experts.linear_fc2.weight8
876
- module.decoder.layers.8.mlp.experts.linear_fc2.weight10
877
  module.decoder.layers.8.mlp.experts.linear_fc2.weight12
 
878
  module.decoder.layers.8.mlp.experts.linear_fc2.weight14
879
  Params for bucket 22 (44040192 elements, 44040192 padded size):
880
  module.decoder.layers.8.mlp.experts.linear_fc2.weight15
881
- module.decoder.layers.8.mlp.experts.linear_fc1.weight0
882
  module.decoder.layers.8.mlp.experts.linear_fc1.weight2
 
883
  module.decoder.layers.8.mlp.experts.linear_fc1.weight1
884
  Params for bucket 23 (50331648 elements, 50331648 padded size):
885
  module.decoder.layers.8.mlp.experts.linear_fc1.weight3
 
886
  module.decoder.layers.8.mlp.experts.linear_fc1.weight6
887
  module.decoder.layers.8.mlp.experts.linear_fc1.weight4
888
- module.decoder.layers.8.mlp.experts.linear_fc1.weight5
889
  Params for bucket 24 (50331648 elements, 50331648 padded size):
890
  module.decoder.layers.8.mlp.experts.linear_fc1.weight7
 
891
  module.decoder.layers.8.mlp.experts.linear_fc1.weight9
892
  module.decoder.layers.8.mlp.experts.linear_fc1.weight10
893
- module.decoder.layers.8.mlp.experts.linear_fc1.weight8
894
  Params for bucket 25 (50331648 elements, 50331648 padded size):
895
  module.decoder.layers.8.mlp.experts.linear_fc1.weight11
896
  module.decoder.layers.8.mlp.experts.linear_fc1.weight12
897
  module.decoder.layers.8.mlp.experts.linear_fc1.weight13
898
  module.decoder.layers.8.mlp.experts.linear_fc1.weight14
899
  Params for bucket 26 (44040192 elements, 44040192 padded size):
900
- module.decoder.layers.7.mlp.experts.linear_fc2.weight3
901
- module.decoder.layers.7.mlp.experts.linear_fc2.weight1
902
- module.decoder.layers.8.mlp.experts.linear_fc1.weight15
903
  module.decoder.layers.7.mlp.experts.linear_fc2.weight0
904
  module.decoder.layers.7.mlp.experts.linear_fc2.weight2
905
  module.decoder.layers.7.mlp.experts.linear_fc2.weight4
 
 
 
906
  Params for bucket 27 (44040192 elements, 44040192 padded size):
 
 
 
907
  module.decoder.layers.7.mlp.experts.linear_fc2.weight5
908
  module.decoder.layers.7.mlp.experts.linear_fc2.weight7
909
  module.decoder.layers.7.mlp.experts.linear_fc2.weight9
910
  module.decoder.layers.7.mlp.experts.linear_fc2.weight11
911
- module.decoder.layers.7.mlp.experts.linear_fc2.weight8
912
- module.decoder.layers.7.mlp.experts.linear_fc2.weight6
913
- module.decoder.layers.7.mlp.experts.linear_fc2.weight10
914
  Params for bucket 28 (50331648 elements, 50331648 padded size):
915
- module.decoder.layers.7.mlp.experts.linear_fc2.weight13
916
- module.decoder.layers.7.mlp.experts.linear_fc2.weight15
917
- module.decoder.layers.7.mlp.experts.linear_fc1.weight0
918
  module.decoder.layers.7.mlp.experts.linear_fc2.weight12
919
  module.decoder.layers.7.mlp.experts.linear_fc2.weight14
920
  module.decoder.layers.7.mlp.experts.linear_fc1.weight1
 
 
 
921
  Params for bucket 29 (50331648 elements, 50331648 padded size):
922
  module.decoder.layers.7.mlp.experts.linear_fc1.weight2
923
- module.decoder.layers.7.mlp.experts.linear_fc1.weight4
924
  module.decoder.layers.7.mlp.experts.linear_fc1.weight5
 
925
  module.decoder.layers.7.mlp.experts.linear_fc1.weight3
926
  Params for bucket 30 (50331648 elements, 50331648 padded size):
927
  module.decoder.layers.7.mlp.experts.linear_fc1.weight6
928
- module.decoder.layers.7.mlp.experts.linear_fc1.weight7
929
  module.decoder.layers.7.mlp.experts.linear_fc1.weight9
930
  module.decoder.layers.7.mlp.experts.linear_fc1.weight8
 
931
  Params for bucket 31 (50331648 elements, 50331648 padded size):
932
  module.decoder.layers.7.mlp.experts.linear_fc1.weight10
933
  module.decoder.layers.7.mlp.experts.linear_fc1.weight11
934
  module.decoder.layers.7.mlp.experts.linear_fc1.weight12
935
  module.decoder.layers.7.mlp.experts.linear_fc1.weight13
936
  Params for bucket 32 (44040192 elements, 44040192 padded size):
 
 
937
  module.decoder.layers.7.mlp.experts.linear_fc1.weight14
938
  module.decoder.layers.6.mlp.experts.linear_fc2.weight0
939
  module.decoder.layers.6.mlp.experts.linear_fc2.weight2
940
- module.decoder.layers.7.mlp.experts.linear_fc1.weight15
941
- module.decoder.layers.6.mlp.experts.linear_fc2.weight1
942
  Params for bucket 33 (44040192 elements, 44040192 padded size):
943
- module.decoder.layers.6.mlp.experts.linear_fc2.weight6
944
- module.decoder.layers.6.mlp.experts.linear_fc2.weight4
945
- module.decoder.layers.6.mlp.experts.linear_fc2.weight8
946
  module.decoder.layers.6.mlp.experts.linear_fc2.weight3
947
- module.decoder.layers.6.mlp.experts.linear_fc2.weight7
948
  module.decoder.layers.6.mlp.experts.linear_fc2.weight5
 
949
  module.decoder.layers.6.mlp.experts.linear_fc2.weight9
 
 
 
950
  Params for bucket 34 (50331648 elements, 50331648 padded size):
951
- module.decoder.layers.6.mlp.experts.linear_fc2.weight10
952
- module.decoder.layers.6.mlp.experts.linear_fc2.weight12
953
- module.decoder.layers.6.mlp.experts.linear_fc2.weight14
954
  module.decoder.layers.6.mlp.experts.linear_fc2.weight13
955
  module.decoder.layers.6.mlp.experts.linear_fc2.weight11
956
  module.decoder.layers.6.mlp.experts.linear_fc2.weight15
 
 
 
957
  module.decoder.layers.6.mlp.experts.linear_fc1.weight0
958
  Params for bucket 35 (50331648 elements, 50331648 padded size):
959
  module.decoder.layers.6.mlp.experts.linear_fc1.weight1
 
960
  module.decoder.layers.6.mlp.experts.linear_fc1.weight3
961
  module.decoder.layers.6.mlp.experts.linear_fc1.weight4
962
- module.decoder.layers.6.mlp.experts.linear_fc1.weight2
963
  Params for bucket 36 (50331648 elements, 50331648 padded size):
964
  module.decoder.layers.6.mlp.experts.linear_fc1.weight5
965
  module.decoder.layers.6.mlp.experts.linear_fc1.weight6
966
- module.decoder.layers.6.mlp.experts.linear_fc1.weight7
967
  module.decoder.layers.6.mlp.experts.linear_fc1.weight8
 
968
  Params for bucket 37 (50331648 elements, 50331648 padded size):
969
  module.decoder.layers.6.mlp.experts.linear_fc1.weight9
970
- module.decoder.layers.6.mlp.experts.linear_fc1.weight10
971
- module.decoder.layers.6.mlp.experts.linear_fc1.weight11
972
  module.decoder.layers.6.mlp.experts.linear_fc1.weight12
 
 
973
  Params for bucket 38 (44040192 elements, 44040192 padded size):
974
  module.decoder.layers.6.mlp.experts.linear_fc1.weight13
975
  module.decoder.layers.6.mlp.experts.linear_fc1.weight15
976
- module.decoder.layers.6.mlp.experts.linear_fc1.weight14
977
  module.decoder.layers.5.mlp.experts.linear_fc2.weight0
 
978
  Params for bucket 39 (44040192 elements, 44040192 padded size):
979
- module.decoder.layers.5.mlp.experts.linear_fc2.weight4
980
- module.decoder.layers.5.mlp.experts.linear_fc2.weight2
981
- module.decoder.layers.5.mlp.experts.linear_fc2.weight6
982
  module.decoder.layers.5.mlp.experts.linear_fc2.weight1
983
- module.decoder.layers.5.mlp.experts.linear_fc2.weight5
984
  module.decoder.layers.5.mlp.experts.linear_fc2.weight3
 
985
  module.decoder.layers.5.mlp.experts.linear_fc2.weight7
 
 
 
986
  Params for bucket 40 (44040192 elements, 44040192 padded size):
 
 
 
987
  module.decoder.layers.5.mlp.experts.linear_fc2.weight8
988
- module.decoder.layers.5.mlp.experts.linear_fc2.weight12
989
  module.decoder.layers.5.mlp.experts.linear_fc2.weight10
 
990
  module.decoder.layers.5.mlp.experts.linear_fc2.weight14
991
- module.decoder.layers.5.mlp.experts.linear_fc2.weight11
992
- module.decoder.layers.5.mlp.experts.linear_fc2.weight9
993
- module.decoder.layers.5.mlp.experts.linear_fc2.weight13
994
  Params for bucket 41 (44040192 elements, 44040192 padded size):
995
  module.decoder.layers.5.mlp.experts.linear_fc2.weight15
 
996
  module.decoder.layers.5.mlp.experts.linear_fc1.weight1
997
  module.decoder.layers.5.mlp.experts.linear_fc1.weight2
998
- module.decoder.layers.5.mlp.experts.linear_fc1.weight0
999
  Params for bucket 42 (50331648 elements, 50331648 padded size):
1000
  module.decoder.layers.5.mlp.experts.linear_fc1.weight3
1001
- module.decoder.layers.5.mlp.experts.linear_fc1.weight5
1002
  module.decoder.layers.5.mlp.experts.linear_fc1.weight6
1003
  module.decoder.layers.5.mlp.experts.linear_fc1.weight4
 
1004
  Params for bucket 43 (50331648 elements, 50331648 padded size):
1005
  module.decoder.layers.5.mlp.experts.linear_fc1.weight7
1006
- module.decoder.layers.5.mlp.experts.linear_fc1.weight8
1007
  module.decoder.layers.5.mlp.experts.linear_fc1.weight9
1008
  module.decoder.layers.5.mlp.experts.linear_fc1.weight10
 
1009
  Params for bucket 44 (50331648 elements, 50331648 padded size):
1010
  module.decoder.layers.5.mlp.experts.linear_fc1.weight11
1011
- module.decoder.layers.5.mlp.experts.linear_fc1.weight14
1012
  module.decoder.layers.5.mlp.experts.linear_fc1.weight12
1013
  module.decoder.layers.5.mlp.experts.linear_fc1.weight13
 
1014
  Params for bucket 45 (44040192 elements, 44040192 padded size):
1015
- module.decoder.layers.5.mlp.experts.linear_fc1.weight15
1016
- module.decoder.layers.4.mlp.experts.linear_fc2.weight3
1017
- module.decoder.layers.4.mlp.experts.linear_fc2.weight1
1018
- module.decoder.layers.4.mlp.experts.linear_fc2.weight0
1019
  module.decoder.layers.4.mlp.experts.linear_fc2.weight2
 
1020
  module.decoder.layers.4.mlp.experts.linear_fc2.weight4
 
 
 
1021
  Params for bucket 46 (44040192 elements, 44040192 padded size):
 
 
 
1022
  module.decoder.layers.4.mlp.experts.linear_fc2.weight5
1023
- module.decoder.layers.4.mlp.experts.linear_fc2.weight9
1024
  module.decoder.layers.4.mlp.experts.linear_fc2.weight7
 
1025
  module.decoder.layers.4.mlp.experts.linear_fc2.weight11
1026
- module.decoder.layers.4.mlp.experts.linear_fc2.weight6
1027
- module.decoder.layers.4.mlp.experts.linear_fc2.weight8
1028
- module.decoder.layers.4.mlp.experts.linear_fc2.weight10
1029
  Params for bucket 47 (50331648 elements, 50331648 padded size):
1030
- module.decoder.layers.4.mlp.experts.linear_fc2.weight13
1031
- module.decoder.layers.4.mlp.experts.linear_fc1.weight0
1032
- module.decoder.layers.4.mlp.experts.linear_fc2.weight15
1033
  module.decoder.layers.4.mlp.experts.linear_fc2.weight12
1034
  module.decoder.layers.4.mlp.experts.linear_fc2.weight14
1035
  module.decoder.layers.4.mlp.experts.linear_fc1.weight1
 
 
 
1036
  Params for bucket 48 (50331648 elements, 50331648 padded size):
1037
  module.decoder.layers.4.mlp.experts.linear_fc1.weight2
1038
- module.decoder.layers.4.mlp.experts.linear_fc1.weight3
1039
  module.decoder.layers.4.mlp.experts.linear_fc1.weight4
1040
  module.decoder.layers.4.mlp.experts.linear_fc1.weight5
 
1041
  Params for bucket 49 (50331648 elements, 50331648 padded size):
1042
  module.decoder.layers.4.mlp.experts.linear_fc1.weight6
1043
- module.decoder.layers.4.mlp.experts.linear_fc1.weight9
1044
  module.decoder.layers.4.mlp.experts.linear_fc1.weight7
1045
  module.decoder.layers.4.mlp.experts.linear_fc1.weight8
 
1046
  Params for bucket 50 (50331648 elements, 50331648 padded size):
1047
  module.decoder.layers.4.mlp.experts.linear_fc1.weight10
1048
- module.decoder.layers.4.mlp.experts.linear_fc1.weight12
1049
  module.decoder.layers.4.mlp.experts.linear_fc1.weight13
1050
  module.decoder.layers.4.mlp.experts.linear_fc1.weight11
 
1051
  Params for bucket 51 (44040192 elements, 44040192 padded size):
 
 
1052
  module.decoder.layers.4.mlp.experts.linear_fc1.weight14
1053
  module.decoder.layers.3.mlp.experts.linear_fc2.weight2
1054
  module.decoder.layers.3.mlp.experts.linear_fc2.weight0
1055
- module.decoder.layers.4.mlp.experts.linear_fc1.weight15
1056
- module.decoder.layers.3.mlp.experts.linear_fc2.weight1
1057
  Params for bucket 52 (44040192 elements, 44040192 padded size):
1058
- module.decoder.layers.3.mlp.experts.linear_fc2.weight4
1059
- module.decoder.layers.3.mlp.experts.linear_fc2.weight6
1060
- module.decoder.layers.3.mlp.experts.linear_fc2.weight8
1061
  module.decoder.layers.3.mlp.experts.linear_fc2.weight3
1062
  module.decoder.layers.3.mlp.experts.linear_fc2.weight5
1063
  module.decoder.layers.3.mlp.experts.linear_fc2.weight7
1064
  module.decoder.layers.3.mlp.experts.linear_fc2.weight9
 
 
 
1065
  Params for bucket 53 (50331648 elements, 50331648 padded size):
1066
- module.decoder.layers.3.mlp.experts.linear_fc2.weight10
1067
- module.decoder.layers.3.mlp.experts.linear_fc2.weight12
1068
- module.decoder.layers.3.mlp.experts.linear_fc2.weight14
1069
- module.decoder.layers.3.mlp.experts.linear_fc1.weight0
1070
- module.decoder.layers.3.mlp.experts.linear_fc2.weight11
1071
  module.decoder.layers.3.mlp.experts.linear_fc2.weight13
 
1072
  module.decoder.layers.3.mlp.experts.linear_fc2.weight15
 
 
 
 
1073
  Params for bucket 54 (50331648 elements, 50331648 padded size):
1074
  module.decoder.layers.3.mlp.experts.linear_fc1.weight1
1075
- module.decoder.layers.3.mlp.experts.linear_fc1.weight4
1076
  module.decoder.layers.3.mlp.experts.linear_fc1.weight2
 
1077
  module.decoder.layers.3.mlp.experts.linear_fc1.weight3
1078
  Params for bucket 55 (50331648 elements, 50331648 padded size):
1079
  module.decoder.layers.3.mlp.experts.linear_fc1.weight5
1080
  module.decoder.layers.3.mlp.experts.linear_fc1.weight8
1081
- module.decoder.layers.3.mlp.experts.linear_fc1.weight7
1082
  module.decoder.layers.3.mlp.experts.linear_fc1.weight6
 
1083
  Params for bucket 56 (50331648 elements, 50331648 padded size):
1084
  module.decoder.layers.3.mlp.experts.linear_fc1.weight9
1085
  module.decoder.layers.3.mlp.experts.linear_fc1.weight11
1086
- module.decoder.layers.3.mlp.experts.linear_fc1.weight10
1087
  module.decoder.layers.3.mlp.experts.linear_fc1.weight12
 
1088
  Params for bucket 57 (44040192 elements, 44040192 padded size):
1089
  module.decoder.layers.3.mlp.experts.linear_fc1.weight13
1090
  module.decoder.layers.3.mlp.experts.linear_fc1.weight14
1091
  module.decoder.layers.3.mlp.experts.linear_fc1.weight15
1092
  module.decoder.layers.2.mlp.experts.linear_fc2.weight0
1093
  Params for bucket 58 (44040192 elements, 44040192 padded size):
1094
- module.decoder.layers.2.mlp.experts.linear_fc2.weight4
1095
- module.decoder.layers.2.mlp.experts.linear_fc2.weight2
1096
- module.decoder.layers.2.mlp.experts.linear_fc2.weight6
1097
  module.decoder.layers.2.mlp.experts.linear_fc2.weight1
1098
  module.decoder.layers.2.mlp.experts.linear_fc2.weight3
1099
  module.decoder.layers.2.mlp.experts.linear_fc2.weight5
1100
  module.decoder.layers.2.mlp.experts.linear_fc2.weight7
 
 
 
1101
  Params for bucket 59 (44040192 elements, 44040192 padded size):
 
 
 
1102
  module.decoder.layers.2.mlp.experts.linear_fc2.weight8
1103
  module.decoder.layers.2.mlp.experts.linear_fc2.weight12
1104
  module.decoder.layers.2.mlp.experts.linear_fc2.weight10
1105
  module.decoder.layers.2.mlp.experts.linear_fc2.weight14
1106
- module.decoder.layers.2.mlp.experts.linear_fc2.weight9
1107
- module.decoder.layers.2.mlp.experts.linear_fc2.weight11
1108
- module.decoder.layers.2.mlp.experts.linear_fc2.weight13
1109
  Params for bucket 60 (44040192 elements, 44040192 padded size):
1110
  module.decoder.layers.2.mlp.experts.linear_fc2.weight15
1111
- module.decoder.layers.2.mlp.experts.linear_fc1.weight1
1112
  module.decoder.layers.2.mlp.experts.linear_fc1.weight2
1113
  module.decoder.layers.2.mlp.experts.linear_fc1.weight0
 
1114
  Params for bucket 61 (50331648 elements, 50331648 padded size):
1115
  module.decoder.layers.2.mlp.experts.linear_fc1.weight3
1116
- module.decoder.layers.2.mlp.experts.linear_fc1.weight4
1117
  module.decoder.layers.2.mlp.experts.linear_fc1.weight5
1118
  module.decoder.layers.2.mlp.experts.linear_fc1.weight6
 
1119
  Params for bucket 62 (50331648 elements, 50331648 padded size):
1120
  module.decoder.layers.2.mlp.experts.linear_fc1.weight7
1121
- module.decoder.layers.2.mlp.experts.linear_fc1.weight10
1122
- module.decoder.layers.2.mlp.experts.linear_fc1.weight8
1123
  module.decoder.layers.2.mlp.experts.linear_fc1.weight9
 
 
1124
  Params for bucket 63 (50331648 elements, 50331648 padded size):
1125
  module.decoder.layers.2.mlp.experts.linear_fc1.weight11
 
1126
  module.decoder.layers.2.mlp.experts.linear_fc1.weight13
1127
  module.decoder.layers.2.mlp.experts.linear_fc1.weight14
1128
- module.decoder.layers.2.mlp.experts.linear_fc1.weight12
1129
  Params for bucket 64 (44040192 elements, 44040192 padded size):
1130
  module.decoder.layers.2.mlp.experts.linear_fc1.weight15
 
 
1131
  module.decoder.layers.1.mlp.experts.linear_fc2.weight2
1132
  module.decoder.layers.1.mlp.experts.linear_fc2.weight0
1133
  module.decoder.layers.1.mlp.experts.linear_fc2.weight4
1134
- module.decoder.layers.1.mlp.experts.linear_fc2.weight3
1135
- module.decoder.layers.1.mlp.experts.linear_fc2.weight1
1136
  Params for bucket 65 (44040192 elements, 44040192 padded size):
1137
- module.decoder.layers.1.mlp.experts.linear_fc2.weight8
1138
- module.decoder.layers.1.mlp.experts.linear_fc2.weight6
1139
- module.decoder.layers.1.mlp.experts.linear_fc2.weight10
1140
  module.decoder.layers.1.mlp.experts.linear_fc2.weight5
1141
- module.decoder.layers.1.mlp.experts.linear_fc2.weight9
1142
  module.decoder.layers.1.mlp.experts.linear_fc2.weight7
 
1143
  module.decoder.layers.1.mlp.experts.linear_fc2.weight11
 
 
 
1144
  Params for bucket 66 (50331648 elements, 50331648 padded size):
 
 
 
1145
  module.decoder.layers.1.mlp.experts.linear_fc2.weight12
1146
  module.decoder.layers.1.mlp.experts.linear_fc2.weight14
1147
- module.decoder.layers.1.mlp.experts.linear_fc1.weight0
1148
- module.decoder.layers.1.mlp.experts.linear_fc2.weight15
1149
- module.decoder.layers.1.mlp.experts.linear_fc2.weight13
1150
  module.decoder.layers.1.mlp.experts.linear_fc1.weight1
1151
  Params for bucket 67 (50331648 elements, 50331648 padded size):
1152
  module.decoder.layers.1.mlp.experts.linear_fc1.weight2
1153
- module.decoder.layers.1.mlp.experts.linear_fc1.weight3
1154
  module.decoder.layers.1.mlp.experts.linear_fc1.weight4
1155
  module.decoder.layers.1.mlp.experts.linear_fc1.weight5
 
1156
  Params for bucket 68 (50331648 elements, 50331648 padded size):
1157
  module.decoder.layers.1.mlp.experts.linear_fc1.weight6
1158
  module.decoder.layers.1.mlp.experts.linear_fc1.weight7
@@ -1160,53 +1160,53 @@ Params for bucket 68 (50331648 elements, 50331648 padded size):
1160
  module.decoder.layers.1.mlp.experts.linear_fc1.weight9
1161
  Params for bucket 69 (50331648 elements, 50331648 padded size):
1162
  module.decoder.layers.1.mlp.experts.linear_fc1.weight10
1163
- module.decoder.layers.1.mlp.experts.linear_fc1.weight13
1164
  module.decoder.layers.1.mlp.experts.linear_fc1.weight11
1165
  module.decoder.layers.1.mlp.experts.linear_fc1.weight12
 
1166
  Params for bucket 70 (44040192 elements, 44040192 padded size):
1167
  module.decoder.layers.1.mlp.experts.linear_fc1.weight14
1168
  module.decoder.layers.0.mlp.experts.linear_fc2.weight1
1169
- module.decoder.layers.0.mlp.experts.linear_fc2.weight2
1170
  module.decoder.layers.0.mlp.experts.linear_fc2.weight0
1171
  module.decoder.layers.1.mlp.experts.linear_fc1.weight15
 
1172
  Params for bucket 71 (44040192 elements, 44040192 padded size):
1173
  module.decoder.layers.0.mlp.experts.linear_fc2.weight3
1174
  module.decoder.layers.0.mlp.experts.linear_fc2.weight5
1175
  module.decoder.layers.0.mlp.experts.linear_fc2.weight7
1176
- module.decoder.layers.0.mlp.experts.linear_fc2.weight9
1177
- module.decoder.layers.0.mlp.experts.linear_fc2.weight4
1178
- module.decoder.layers.0.mlp.experts.linear_fc2.weight6
1179
  module.decoder.layers.0.mlp.experts.linear_fc2.weight8
 
 
 
1180
  Params for bucket 72 (50331648 elements, 50331648 padded size):
1181
- module.decoder.layers.0.mlp.experts.linear_fc2.weight11
1182
- module.decoder.layers.0.mlp.experts.linear_fc2.weight13
1183
- module.decoder.layers.0.mlp.experts.linear_fc2.weight15
1184
  module.decoder.layers.0.mlp.experts.linear_fc2.weight10
1185
  module.decoder.layers.0.mlp.experts.linear_fc2.weight14
1186
  module.decoder.layers.0.mlp.experts.linear_fc2.weight12
1187
  module.decoder.layers.0.mlp.experts.linear_fc1.weight0
 
 
 
1188
  Params for bucket 73 (50331648 elements, 50331648 padded size):
1189
  module.decoder.layers.0.mlp.experts.linear_fc1.weight1
1190
- module.decoder.layers.0.mlp.experts.linear_fc1.weight4
1191
- module.decoder.layers.0.mlp.experts.linear_fc1.weight3
1192
  module.decoder.layers.0.mlp.experts.linear_fc1.weight2
 
 
1193
  Params for bucket 74 (50331648 elements, 50331648 padded size):
1194
  module.decoder.layers.0.mlp.experts.linear_fc1.weight5
1195
- module.decoder.layers.0.mlp.experts.linear_fc1.weight7
1196
  module.decoder.layers.0.mlp.experts.linear_fc1.weight8
1197
  module.decoder.layers.0.mlp.experts.linear_fc1.weight6
 
1198
  Params for bucket 75 (50331648 elements, 50331648 padded size):
1199
  module.decoder.layers.0.mlp.experts.linear_fc1.weight9
1200
- module.decoder.layers.0.mlp.experts.linear_fc1.weight11
1201
  module.decoder.layers.0.mlp.experts.linear_fc1.weight12
1202
  module.decoder.layers.0.mlp.experts.linear_fc1.weight10
 
1203
  Params for bucket 76 (37748736 elements, 37748736 padded size):
1204
  module.decoder.layers.0.mlp.experts.linear_fc1.weight13
1205
  module.decoder.layers.0.mlp.experts.linear_fc1.weight15
1206
  module.decoder.layers.0.mlp.experts.linear_fc1.weight14
1207
- INFO:megatron.core.optimizer:Setting up optimizer with config OptimizerConfig(optimizer='adam', lr=0.0001, min_lr=1e-05, decoupled_lr=None, decoupled_min_lr=None, weight_decay=0.1, fp8_recipe='mxfp8', fp16=False, bf16=True, reuse_grad_buf_for_mxfp8_param_ag=False, params_dtype=torch.bfloat16, use_precision_aware_optimizer=False, store_param_remainders=True, main_grads_dtype=torch.float32, main_params_dtype=torch.float32, exp_avg_dtype=torch.float32, exp_avg_sq_dtype=torch.float32, loss_scale=None, initial_loss_scale=4294967296, min_loss_scale=1.0, loss_scale_window=1000, hysteresis=2, adam_beta1=0.9, adam_beta2=0.999, adam_eps=1e-08, sgd_momentum=0.9, use_distributed_optimizer=True, overlap_param_gather_with_optimizer_step=False, optimizer_cpu_offload=False, optimizer_offload_fraction=1.0, use_torch_optimizer_for_cpu_offload=False, overlap_cpu_optimizer_d2h_h2d=False, pin_cpu_grads=True, pin_cpu_params=True, clip_grad=1.0, log_num_zeros_in_grad=False, barrier_with_L1_time=True, timers=<megatron.core.timers.Timers object at 0x7be9dac4d3a0>, config_logger_dir='')
1208
  INFO:megatron.core.optimizer_param_scheduler:> learning rate decay style: cosine
1209
- [after model, optimizer, and learning rate scheduler are built] datetime: 2026-08-29 23:48:02
1210
  > building train, validation, and test datasets ...
1211
  > datasets target sizes (minimum size):
1212
  train: 96
@@ -1214,7 +1214,7 @@ INFO:megatron.core.optimizer_param_scheduler:> learning rate decay style: cosine
1214
  test: 0
1215
  INFO:megatron.core.datasets.blended_megatron_dataset_config:Let split_matrix = [(0, 1.0), None, None]
1216
  > building train, validation, and test datasets for GPT ...
1217
- INFO:megatron.core.datasets.blended_megatron_dataset_builder:Building GPTDataset splits with sizes=(96, 0, 0) and config=GPTDatasetConfig(random_seed=1234, sequence_length=8192, blend=(['/workspace/data/wikitext/wikitext103_gpt2_text_document'], None), blend_per_split=None, split='100,0,0', split_matrix=[(0, 1.0), None, None], num_dataset_builder_threads=1, path_to_cache=None, mmap_bin_files=True, mock=False, tokenizer=<megatron.training.tokenizer.tokenizer._GPT2BPETokenizer object at 0x7be9dceba480>, mid_level_dataset_surplus=0.005, reset_position_ids=False, reset_attention_mask=False, eod_mask_loss=False, create_attention_mask=True, drop_last_partial_validation_sequence=True, add_extra_token_to_sequence=True, object_storage_cache_path=None)
1218
  INFO:megatron.core.datasets.indexed_dataset:Load the _IndexReader from /workspace/data/wikitext/wikitext103_gpt2_text_document.idx
1219
  INFO:megatron.core.datasets.indexed_dataset: Extract the sequence lengths
1220
  INFO:megatron.core.datasets.indexed_dataset: Extract the sequence pointers
@@ -1227,14 +1227,14 @@ INFO:megatron.core.datasets.gpt_dataset: Load the sample index from eadc3756447e
1227
  INFO:megatron.core.datasets.gpt_dataset: Load the shuffle index from eadc3756447e39d609e8aa728a91f451-GPTDataset-train-shuffle_index.npy
1228
  INFO:megatron.core.datasets.gpt_dataset:> total number of samples: 14536
1229
  > finished creating GPT datasets ...
1230
- [after dataloaders are built] datetime: 2026-08-29 23:48:02
1231
  done with setup ...
1232
  training ...
1233
  Overwriting rerun_state_machine.current_iteration from -1 to 0...
1234
  (min, max) time across ranks (ms):
1235
- model-and-optimizer-setup ......................: (3473.33, 3497.43)
1236
- train/valid/test-data-iterators-setup ..........: (64.41, 69.66)
1237
- [before the start of training step] datetime: 2026-08-29 23:48:02
1238
  INFO:moonep.api:MoonEP comm buffer sizes:
1239
  config : R=8, S=8192, H=4096, E=128, K=8, NvS=69600, NvS_padded=69632
1240
  sm_count : num_sms=32, num_sms_dedup=148
@@ -1246,44 +1246,42 @@ INFO:moonep.api:MoonEP comm buffer sizes:
1246
  [MoonEP selfcheck 2/4] dropped groups empty OK | covered rows [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 128, 129, 130, 131, 132, 133, 134, 135, 136, 137, 138, 139, 140, 141, 142, 143] sizes [9344, 2688, 128, 24192, 128, 0, 128, 256, 128, 768, 10496, 128, 128, 2432, 128, 1536, 14208, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0] | duplicated experts [[69, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1], [88, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1], [55, 52, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1], [-1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1], [63, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1], [124, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1], [10, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1], [58, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1]]
1247
  [MoonEP selfcheck 3/4] dropped groups empty OK | covered rows [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 128, 129, 130, 131, 132, 133, 134, 135, 136, 137, 138, 139, 140, 141, 142, 143] sizes [128, 128, 128, 0, 128, 44416, 128, 128, 128, 6912, 128, 256, 128, 128, 128, 128, 13824, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0] | duplicated experts [[78, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1], [111, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1], [62, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1], [-1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1], [57, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1], [106, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1], [121, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1], [57, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1]]
1248
  [MoonEP selfcheck 4/4] dropped groups empty OK | covered rows [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 128, 129, 130, 131, 132, 133, 134, 135, 136, 137, 138, 139, 140, 141, 142, 143] sizes [2304, 128, 128, 512, 128, 12416, 128, 1152, 1280, 16768, 640, 29056, 128, 128, 128, 1664, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0] | duplicated experts [[-1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1], [118, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1], [11, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1], [118, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1], [39, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1], [11, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1], [73, 79, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1], [86, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1]]
1249
- [2026-08-29 23:48:23] iteration 1/ 12 | consumed samples: 8 | elapsed time per iteration (ms): 21162.6 | throughput per GPU (TFLOP/s/GPU): 8.5 | learning rate: 2.000000E-05 | global batch size: 8 | lm loss: 1.115993E+01 | loss scale: 1.0 | grad norm: 44.807 | number of skipped iterations: 0 | number of nan iterations: 0 |
1250
  Number of parameters in transformer block in billions: 29.85
1251
  Number of parameters in embedding layers in billions: 0.41
1252
  Total number of parameters in billions: 30.26
1253
  Number of parameters in most loaded shard in billions: 30.2590
1254
  Theoretical memory footprints: weight and optimizer=216428.96 MB
1255
  [Rank 0] (after 1 iterations) memory (MB) | allocated: 67239.548828125 | max allocated: 67239.54931640625 | reserved: 68548.0 | max reserved: 68548.0
1256
- [2026-08-29 23:48:24] iteration 2/ 12 | consumed samples: 16 | elapsed time per iteration (ms): 436.6 | throughput per GPU (TFLOP/s/GPU): 414.1 | learning rate: 4.000000E-05 | global batch size: 8 | lm loss: 1.117600E+01 | loss scale: 1.0 | grad norm: 44.335 | number of skipped iterations: 0 | number of nan iterations: 0 |
1257
- [2026-08-29 23:48:24] iteration 3/ 12 | consumed samples: 24 | elapsed time per iteration (ms): 331.5 | throughput per GPU (TFLOP/s/GPU): 545.5 | learning rate: 6.000000E-05 | global batch size: 8 | lm loss: 9.685947E+00 | loss scale: 1.0 | grad norm: 62.878 | number of skipped iterations: 0 | number of nan iterations: 0 |
1258
  Capture CUDA graph for training!!!
1259
  CUDA graph capture done!!!
1260
- [2026-08-29 23:48:25] iteration 4/ 12 | consumed samples: 32 | elapsed time per iteration (ms): 944.5 | throughput per GPU (TFLOP/s/GPU): 191.5 | learning rate: 8.000000E-05 | global batch size: 8 | lm loss: 1.338058E+01 | loss scale: 1.0 | grad norm: 20.306 | number of skipped iterations: 0 | number of nan iterations: 0 |
1261
- [2026-08-29 23:48:25] iteration 5/ 12 | consumed samples: 40 | elapsed time per iteration (ms): 324.9 | throughput per GPU (TFLOP/s/GPU): 556.6 | learning rate: 1.000000E-04 | global batch size: 8 | lm loss: 1.049307E+01 | loss scale: 1.0 | grad norm: 95.187 | number of skipped iterations: 0 | number of nan iterations: 0 |
1262
- [2026-08-29 23:48:26] iteration 6/ 12 | consumed samples: 48 | elapsed time per iteration (ms): 322.5 | throughput per GPU (TFLOP/s/GPU): 560.7 | learning rate: 7.750000E-05 | global batch size: 8 | lm loss: 1.048625E+01 | loss scale: 1.0 | grad norm: 54.475 | number of skipped iterations: 0 | number of nan iterations: 0 |
1263
- [2026-08-29 23:48:26] iteration 7/ 12 | consumed samples: 56 | elapsed time per iteration (ms): 323.2 | throughput per GPU (TFLOP/s/GPU): 559.5 | learning rate: 3.250000E-05 | global batch size: 8 | lm loss: 1.027090E+01 | loss scale: 1.0 | grad norm: 6.334 | number of skipped iterations: 0 | number of nan iterations: 0 |
1264
- [2026-08-29 23:48:26] iteration 8/ 12 | consumed samples: 64 | elapsed time per iteration (ms): 322.7 | throughput per GPU (TFLOP/s/GPU): 560.3 | learning rate: 1.000000E-05 | global batch size: 8 | lm loss: 9.809750E+00 | loss scale: 1.0 | grad norm: 5.526 | number of skipped iterations: 0 | number of nan iterations: 0 |
1265
- [2026-08-29 23:48:27] iteration 9/ 12 | consumed samples: 72 | elapsed time per iteration (ms): 321.8 | throughput per GPU (TFLOP/s/GPU): 562.0 | learning rate: 1.000000E-05 | global batch size: 8 | lm loss: 9.547915E+00 | loss scale: 1.0 | grad norm: 5.230 | number of skipped iterations: 0 | number of nan iterations: 0 |
1266
- [2026-08-29 23:48:27] iteration 10/ 12 | consumed samples: 80 | elapsed time per iteration (ms): 322.3 | throughput per GPU (TFLOP/s/GPU): 561.1 | learning rate: 1.000000E-05 | global batch size: 8 | lm loss: 9.449049E+00 | loss scale: 1.0 | grad norm: 4.934 | number of skipped iterations: 0 | number of nan iterations: 0 |
1267
- [2026-08-29 23:48:27] iteration 11/ 12 | consumed samples: 88 | elapsed time per iteration (ms): 320.9 | throughput per GPU (TFLOP/s/GPU): 563.5 | learning rate: 1.000000E-05 | global batch size: 8 | lm loss: 9.330864E+00 | loss scale: 1.0 | grad norm: 4.634 | number of skipped iterations: 0 | number of nan iterations: 0 |
1268
- [2026-08-29 23:48:28] iteration 12/ 12 | consumed samples: 96 | elapsed time per iteration (ms): 321.9 | throughput per GPU (TFLOP/s/GPU): 561.8 | learning rate: 1.000000E-05 | global batch size: 8 | lm loss: 9.217180E+00 | loss scale: 1.0 | grad norm: 4.483 | number of skipped iterations: 0 | number of nan iterations: 0 |
1269
- [after training is done] datetime: 2026-08-29 23:48:28
1270
- [rank0]:[W829 23:48:29.704910691 ProcessGroupNCCL.cpp:1505] Warning: WARNING: destroy_process_group() was not called before program exit, which can leak resources. For more info, please see https://pytorch.org/docs/stable/distributed.html#shutdown (function operator())
1271
- [rank5]:[W829 23:48:30.362260925 ProcessGroupNCCL.cpp:1505] Warning: WARNING: destroy_process_group() was not called before program exit, which can leak resources. For more info, please see https://pytorch.org/docs/stable/distributed.html#shutdown (function operator())
1272
- [rank4]:[W829 23:48:30.756421216 ProcessGroupNCCL.cpp:1505] Warning: WARNING: destroy_process_group() was not called before program exit, which can leak resources. For more info, please see https://pytorch.org/docs/stable/distributed.html#shutdown (function operator())
1273
- [rank3]:[W829 23:48:31.400079860 ProcessGroupNCCL.cpp:1505] Warning: WARNING: destroy_process_group() was not called before program exit, which can leak resources. For more info, please see https://pytorch.org/docs/stable/distributed.html#shutdown (function operator())
1274
- [rank7]:[W829 23:48:31.439735903 ProcessGroupNCCL.cpp:1505] Warning: WARNING: destroy_process_group() was not called before program exit, which can leak resources. For more info, please see https://pytorch.org/docs/stable/distributed.html#shutdown (function operator())
1275
- [rank1]:[W829 23:48:31.441838729 ProcessGroupNCCL.cpp:1505] Warning: WARNING: destroy_process_group() was not called before program exit, which can leak resources. For more info, please see https://pytorch.org/docs/stable/distributed.html#shutdown (function operator())
1276
- [rank2]:[W829 23:48:31.444486514 ProcessGroupNCCL.cpp:1505] Warning: WARNING: destroy_process_group() was not called before program exit, which can leak resources. For more info, please see https://pytorch.org/docs/stable/distributed.html#shutdown (function operator())
1277
- [rank6]:[W829 23:48:31.448760172 ProcessGroupNCCL.cpp:1505] Warning: WARNING: destroy_process_group() was not called before program exit, which can leak resources. For more info, please see https://pytorch.org/docs/stable/distributed.html#shutdown (function operator())
1278
- W0829 23:48:34.082000 242741 torch/distributed/elastic/agent/server/api.py:719] Received 15 death signal, shutting down workers
1279
- W0829 23:48:34.082000 242741 torch/distributed/elastic/multiprocessing/api.py:900] Sending process 242931 closing signal SIGTERM
1280
- W0829 23:48:34.083000 242741 torch/distributed/elastic/multiprocessing/api.py:900] Sending process 242932 closing signal SIGTERM
1281
- W0829 23:48:34.083000 242741 torch/distributed/elastic/multiprocessing/api.py:900] Sending process 242933 closing signal SIGTERM
1282
- W0829 23:48:34.084000 242741 torch/distributed/elastic/multiprocessing/api.py:900] Sending process 242934 closing signal SIGTERM
1283
- W0829 23:48:34.084000 242741 torch/distributed/elastic/multiprocessing/api.py:900] Sending process 242935 closing signal SIGTERM
1284
- W0829 23:48:34.085000 242741 torch/distributed/elastic/multiprocessing/api.py:900] Sending process 242936 closing signal SIGTERM
1285
- W0829 23:48:34.085000 242741 torch/distributed/elastic/multiprocessing/api.py:900] Sending process 242937 closing signal SIGTERM
1286
- W0829 23:48:34.086000 242741 torch/distributed/elastic/multiprocessing/api.py:900] Sending process 242938 closing signal SIGTERM
1287
  Traceback (most recent call last):
1288
  File "/usr/local/bin/torchrun", line 8, in <module>
1289
  sys.exit(main())
@@ -1311,4 +1309,4 @@ Traceback (most recent call last):
1311
  time.sleep(monitor_interval)
1312
  File "/usr/local/lib/python3.12/dist-packages/torch/distributed/elastic/multiprocessing/api.py", line 84, in _terminate_process_handler
1313
  raise SignalException(f"Process {os.getpid()} got signal: {sigval}", sigval=sigval)
1314
- torch.distributed.elastic.multiprocessing.api.SignalException: Process 242741 got signal: 15
 
1
+ [01:33:45] WARN: moonep_backend_async_grad_reduce lacks --moe-router-fusion; QWEN3_235B runs unfused (timing only, routing identical)
2
  ==============================================================
3
  QWEN3_235B / moonep (slots/rank=16 policy=matched)
4
  E=128 K=8 H'=1536 m=16
5
  L=12 seq=8192 gbs=8 (1 microbatches)
6
  cap_mode=none cap=none cg=1 recompute=none
7
  memory_dump=none snapshots=/workspace/tierb-fixed-moonep-full/moonep/tierb/tierb_gbs8/QWEN3_235B__moonep__matched__seq8192__gbs8__capnone__async/rep3/memory_snapshots/snapshot.pickle
8
+ repo=/workspace/Megatron-MoE-Echo @ 2e68c8ad
9
  ==============================================================
10
+ W0830 01:33:47.017000 385325 torch/distributed/run.py:766]
11
+ W0830 01:33:47.017000 385325 torch/distributed/run.py:766] *****************************************
12
+ W0830 01:33:47.017000 385325 torch/distributed/run.py:766] Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed.
13
+ W0830 01:33:47.017000 385325 torch/distributed/run.py:766] *****************************************
14
  /workspace/Megatron-MoE-Echo/megatron/training/arguments.py:831: UserWarning: Disabling sequence parallelism because tensor model parallelism is disabled
15
  warnings.warn("Disabling sequence parallelism because tensor model parallelism is disabled")
16
+ WARNING: one_logger package is required to enable e2e metrics tracking. please go to https://confluence.nvidia.com/display/MLWFO/Package+Repositories for details to install it
17
  /workspace/Megatron-MoE-Echo/megatron/training/arguments.py:831: UserWarning: Disabling sequence parallelism because tensor model parallelism is disabled
18
  warnings.warn("Disabling sequence parallelism because tensor model parallelism is disabled")
19
  /workspace/Megatron-MoE-Echo/megatron/training/arguments.py:831: UserWarning: Disabling sequence parallelism because tensor model parallelism is disabled
20
  warnings.warn("Disabling sequence parallelism because tensor model parallelism is disabled")
 
 
 
21
  /workspace/Megatron-MoE-Echo/megatron/training/arguments.py:831: UserWarning: Disabling sequence parallelism because tensor model parallelism is disabled
22
  warnings.warn("Disabling sequence parallelism because tensor model parallelism is disabled")
23
  /workspace/Megatron-MoE-Echo/megatron/training/arguments.py:831: UserWarning: Disabling sequence parallelism because tensor model parallelism is disabled
 
572
  -------------------- end of arguments ---------------------
573
  INFO:megatron.core.num_microbatches_calculator:setting number of microbatches to constant 1
574
  > building GPT2BPETokenizer tokenizer ...
575
+ /workspace/Megatron-MoE-Echo/megatron/training/arguments.py:831: UserWarning: Disabling sequence parallelism because tensor model parallelism is disabled
576
+ warnings.warn("Disabling sequence parallelism because tensor model parallelism is disabled")
577
  > padded vocab (size: 50257) with 47 dummy tokens (new size: 50304)
578
  WARNING:megatron.core.rerun_state_machine:RerunStateMachine initialized in mode RerunMode.VALIDATE_RESULTS
579
  > initializing torch distributed ...
580
  [Gloo] Rank 0 is connected to 7 peer ranks. Expected number of connected peer ranks is : 7
581
+ [Gloo] Rank 3 is connected to 7 peer ranks. Expected number of connected peer ranks is : 7
582
  [Gloo] Rank 1 is connected to 7 peer ranks. Expected number of connected peer ranks is : 7
583
  [Gloo] Rank 2 is connected to 7 peer ranks. Expected number of connected peer ranks is : 7
 
584
  [Gloo] Rank 4 is connected to 7 peer ranks. Expected number of connected peer ranks is : 7
585
+ [Gloo] Rank 5 is connected to 7 peer ranks. Expected number of connected peer ranks is : 7[Gloo] Rank
586
+ [Gloo] Rank 6 is connected to 77 is connected to peer ranks. 7Expected number of connected peer ranks is : peer ranks. 7Expected number of connected peer ranks is :
587
  7
588
+ [Gloo] Rank 0 is connected to 7 peer ranks. Expected number of connected peer ranks is : 7
589
+ [Gloo] Rank 4 is connected to 7 peer ranks. Expected number of connected peer ranks is : 7
590
+ [Gloo] Rank 5 is connected to 7 peer ranks. Expected number of connected peer ranks is : 7
591
+ [Gloo] Rank 1 is connected to 7 peer ranks. Expected number of connected peer ranks is : 7
592
+ [Gloo] Rank 2 is connected to 7 peer ranks. Expected number of connected peer ranks is : 7
593
  [Gloo] Rank 3 is connected to 7 peer ranks. Expected number of connected peer ranks is : 7
594
+ [Gloo] Rank [Gloo] Rank 7 is connected to 67 is connected to peer ranks. 7Expected number of connected peer ranks is : peer ranks. 7Expected number of connected peer ranks is :
595
  7
596
  [Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
597
  > initialized tensor model parallel with size 1
598
  > initialized pipeline model parallel with size 1
599
  > setting random seeds to 1234 ...
600
+ [Gloo] Rank [Gloo] Rank 00 is connected to is connected to 00 peer ranks. peer ranks. Expected number of connected peer ranks is : Expected number of connected peer ranks is : 00
 
601
 
602
+ [Gloo] Rank 0[Gloo] Rank is connected to 00[Gloo] Rank is connected to peer ranks. 00Expected number of connected peer ranks is : is connected to peer ranks. 00Expected number of connected peer ranks is :
603
+ peer ranks. 0Expected number of connected peer ranks is :
604
+ 0
605
  [Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
606
  [Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
607
+ [rank5]:[W830 01:34:04.105198961 ProcessGroupNCCL.cpp:4789] [PG ID 0 PG GUID 0 Rank 5] using GPU 5 as device used by this process is currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect. You can specify device_id in init_process_group() to force use of a particular device.
608
+ [rank1]:[W830 01:34:04.105212291 ProcessGroupNCCL.cpp:4789] [PG ID 0 PG GUID 0 Rank 1] using GPU 1 as device used by this process is currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect. You can specify device_id in init_process_group() to force use of a particular device.
609
+ [rank7]:[W830 01:34:04.105213047 ProcessGroupNCCL.cpp:4789] [PG ID 0 PG GUID 0 Rank 7] using GPU 7 as device used by this process is currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect. You can specify device_id in init_process_group() to force use of a particular device.
610
+ [rank4]:[W830 01:34:04.105279090 ProcessGroupNCCL.cpp:4789] [PG ID 0 PG GUID 0 Rank 4] using GPU 4 as device used by this process is currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect. You can specify device_id in init_process_group() to force use of a particular device.
611
+ [rank3]:[W830 01:34:04.105289555 ProcessGroupNCCL.cpp:4789] [PG ID 0 PG GUID 0 Rank 3] using GPU 3 as device used by this process is currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect. You can specify device_id in init_process_group() to force use of a particular device.
612
+ [rank6]:[W830 01:34:04.105317293 ProcessGroupNCCL.cpp:4789] [PG ID 0 PG GUID 0 Rank 6] using GPU 6 as device used by this process is currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect. You can specify device_id in init_process_group() to force use of a particular device.
613
+ [rank2]:[W830 01:34:04.105351538 ProcessGroupNCCL.cpp:4789] [PG ID 0 PG GUID 0 Rank 2] using GPU 2 as device used by this process is currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect. You can specify device_id in init_process_group() to force use of a particular device.
 
 
614
  > compiling dataset index builder ...
615
  make: Entering directory '/workspace/Megatron-MoE-Echo/megatron/core/datasets'
616
  make: Nothing to be done for 'default'.
617
  make: Leaving directory '/workspace/Megatron-MoE-Echo/megatron/core/datasets'
618
+ >>> done with dataset index builder. Compilation time: 0.033 seconds
619
  WARNING: constraints for invoking optimized fused softmax kernel are not met. We default back to unfused kernel invocations.
620
  > compiling and loading fused kernels ...
621
+ [rank0]:[W830 01:34:05.270035787 ProcessGroupNCCL.cpp:4789] [PG ID 0 PG GUID 0 Rank 0] using GPU 0 as device used by this process is currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect. You can specify device_id in init_process_group() to force use of a particular device.
622
+ >>> done with compiling and loading fused kernels. Compilation time: 3.693 seconds
623
+ time to initialize megatron (seconds): 7.525
624
+ [after megatron is initialized] datetime: 2026-08-30 01:34:10
625
  building GPT model ...
626
  /usr/local/lib/python3.12/dist-packages/transformer_engine/pytorch/cpu_offload.py:708: DeprecationWarning: Offloading weights is deprecated. Using offload_weights=True does not have any effect.
627
  warnings.warn(
 
645
  Params for bucket 1 (206045184 elements, 206045184 padded size):
646
  module.output_layer.weight
647
  Params for bucket 2 (71840000 elements, 71840000 padded size):
 
648
  module.decoder.layers.11.self_attention.q_layernorm.weight
649
+ module.decoder.layers.11.pre_mlp_layernorm.weight
650
  module.decoder.layers.11.mlp.router.weight
651
  module.decoder.layers.11.self_attention.k_layernorm.weight
652
  module.decoder.layers.11.self_attention.linear_qkv.weight
653
  module.decoder.layers.11.self_attention.linear_qkv.layer_norm_weight
 
654
  module.decoder.layers.11.self_attention.linear_proj.weight
655
+ module.decoder.final_layernorm.weight
656
  Params for bucket 3 (71835904 elements, 71835904 padded size):
657
+ module.decoder.layers.10.pre_mlp_layernorm.weight
658
  module.decoder.layers.10.self_attention.linear_qkv.weight
659
+ module.decoder.layers.10.self_attention.q_layernorm.weight
660
  module.decoder.layers.10.self_attention.linear_qkv.layer_norm_weight
661
  module.decoder.layers.10.mlp.router.weight
 
662
  module.decoder.layers.10.self_attention.k_layernorm.weight
 
663
  module.decoder.layers.10.self_attention.linear_proj.weight
664
  Params for bucket 4 (71835904 elements, 71835904 padded size):
665
  module.decoder.layers.9.self_attention.k_layernorm.weight
666
+ module.decoder.layers.9.self_attention.linear_qkv.weight
667
  module.decoder.layers.9.self_attention.linear_proj.weight
668
  module.decoder.layers.9.mlp.router.weight
669
  module.decoder.layers.9.pre_mlp_layernorm.weight
 
670
  module.decoder.layers.9.self_attention.q_layernorm.weight
671
  module.decoder.layers.9.self_attention.linear_qkv.layer_norm_weight
672
  Params for bucket 5 (71835904 elements, 71835904 padded size):
673
  module.decoder.layers.8.mlp.router.weight
674
  module.decoder.layers.8.pre_mlp_layernorm.weight
675
+ module.decoder.layers.8.self_attention.k_layernorm.weight
676
  module.decoder.layers.8.self_attention.linear_qkv.weight
677
  module.decoder.layers.8.self_attention.q_layernorm.weight
 
678
  module.decoder.layers.8.self_attention.linear_qkv.layer_norm_weight
679
  module.decoder.layers.8.self_attention.linear_proj.weight
680
  Params for bucket 6 (71835904 elements, 71835904 padded size):
 
683
  module.decoder.layers.7.self_attention.k_layernorm.weight
684
  module.decoder.layers.7.self_attention.linear_qkv.weight
685
  module.decoder.layers.7.self_attention.q_layernorm.weight
 
686
  module.decoder.layers.7.self_attention.linear_qkv.layer_norm_weight
687
+ module.decoder.layers.7.self_attention.linear_proj.weight
688
  Params for bucket 7 (71835904 elements, 71835904 padded size):
689
  module.decoder.layers.6.mlp.router.weight
 
 
690
  module.decoder.layers.6.pre_mlp_layernorm.weight
691
+ module.decoder.layers.6.self_attention.linear_qkv.weight
692
  module.decoder.layers.6.self_attention.k_layernorm.weight
693
  module.decoder.layers.6.self_attention.linear_qkv.layer_norm_weight
694
+ module.decoder.layers.6.self_attention.q_layernorm.weight
695
  module.decoder.layers.6.self_attention.linear_proj.weight
696
  Params for bucket 8 (71835904 elements, 71835904 padded size):
697
+ module.decoder.layers.5.mlp.router.weight
698
  module.decoder.layers.5.pre_mlp_layernorm.weight
699
+ module.decoder.layers.5.self_attention.q_layernorm.weight
700
  module.decoder.layers.5.self_attention.linear_qkv.layer_norm_weight
 
701
  module.decoder.layers.5.self_attention.linear_qkv.weight
702
  module.decoder.layers.5.self_attention.k_layernorm.weight
703
  module.decoder.layers.5.self_attention.linear_proj.weight
704
  Params for bucket 9 (71835904 elements, 71835904 padded size):
 
705
  module.decoder.layers.4.pre_mlp_layernorm.weight
706
  module.decoder.layers.4.self_attention.q_layernorm.weight
707
+ module.decoder.layers.4.self_attention.linear_qkv.layer_norm_weight
708
+ module.decoder.layers.4.mlp.router.weight
709
  module.decoder.layers.4.self_attention.k_layernorm.weight
710
  module.decoder.layers.4.self_attention.linear_qkv.weight
 
711
  module.decoder.layers.4.self_attention.linear_proj.weight
712
  Params for bucket 10 (71835904 elements, 71835904 padded size):
713
  module.decoder.layers.3.mlp.router.weight
 
714
  module.decoder.layers.3.self_attention.q_layernorm.weight
715
  module.decoder.layers.3.self_attention.linear_qkv.weight
716
+ module.decoder.layers.3.pre_mlp_layernorm.weight
717
  module.decoder.layers.3.self_attention.k_layernorm.weight
718
  module.decoder.layers.3.self_attention.linear_qkv.layer_norm_weight
719
  module.decoder.layers.3.self_attention.linear_proj.weight
720
  Params for bucket 11 (71835904 elements, 71835904 padded size):
721
  module.decoder.layers.2.mlp.router.weight
722
+ module.decoder.layers.2.self_attention.q_layernorm.weight
723
  module.decoder.layers.2.pre_mlp_layernorm.weight
 
724
  module.decoder.layers.2.self_attention.linear_qkv.weight
725
+ module.decoder.layers.2.self_attention.k_layernorm.weight
726
  module.decoder.layers.2.self_attention.linear_qkv.layer_norm_weight
727
  module.decoder.layers.2.self_attention.linear_proj.weight
728
  Params for bucket 12 (71835904 elements, 71835904 padded size):
729
  module.decoder.layers.1.mlp.router.weight
730
  module.decoder.layers.1.pre_mlp_layernorm.weight
731
  module.decoder.layers.1.self_attention.k_layernorm.weight
 
732
  module.decoder.layers.1.self_attention.linear_qkv.weight
733
+ module.decoder.layers.1.self_attention.q_layernorm.weight
734
  module.decoder.layers.1.self_attention.linear_qkv.layer_norm_weight
735
  module.decoder.layers.1.self_attention.linear_proj.weight
736
  Params for bucket 13 (71835904 elements, 71835904 padded size):
737
+ module.decoder.layers.0.mlp.router.weight
738
  module.decoder.layers.0.self_attention.q_layernorm.weight
 
739
  module.decoder.layers.0.self_attention.linear_proj.weight
740
+ module.decoder.layers.0.pre_mlp_layernorm.weight
741
+ module.decoder.layers.0.self_attention.k_layernorm.weight
742
  module.decoder.layers.0.self_attention.linear_qkv.weight
743
  module.decoder.layers.0.self_attention.linear_qkv.layer_norm_weight
744
  Params for bucket 14 (206045184 elements, 206045184 padded size):
 
746
  INFO:megatron.core.distributed.param_and_grad_buffer:Number of buckets for gradient all-reduce / reduce-scatter: 76
747
  Params for bucket 1 (44040192 elements, 44040192 padded size):
748
  module.decoder.layers.11.mlp.experts.linear_fc2.weight0
 
749
  module.decoder.layers.11.mlp.experts.linear_fc2.weight4
750
+ module.decoder.layers.11.mlp.experts.linear_fc2.weight2
751
  module.decoder.layers.11.mlp.experts.linear_fc2.weight6
752
  module.decoder.layers.11.mlp.experts.linear_fc2.weight3
753
  module.decoder.layers.11.mlp.experts.linear_fc2.weight1
 
757
  module.decoder.layers.11.mlp.experts.linear_fc2.weight10
758
  module.decoder.layers.11.mlp.experts.linear_fc2.weight12
759
  module.decoder.layers.11.mlp.experts.linear_fc2.weight7
 
760
  module.decoder.layers.11.mlp.experts.linear_fc2.weight11
761
+ module.decoder.layers.11.mlp.experts.linear_fc2.weight9
762
  module.decoder.layers.11.mlp.experts.linear_fc2.weight13
763
  Params for bucket 3 (50331648 elements, 50331648 padded size):
764
  module.decoder.layers.11.mlp.experts.linear_fc2.weight14
765
  module.decoder.layers.11.mlp.experts.linear_fc1.weight1
 
766
  module.decoder.layers.11.mlp.experts.linear_fc1.weight2
767
+ module.decoder.layers.11.mlp.experts.linear_fc2.weight15
768
  module.decoder.layers.11.mlp.experts.linear_fc1.weight0
769
  Params for bucket 4 (50331648 elements, 50331648 padded size):
770
  module.decoder.layers.11.mlp.experts.linear_fc1.weight3
771
  module.decoder.layers.11.mlp.experts.linear_fc1.weight5
 
772
  module.decoder.layers.11.mlp.experts.linear_fc1.weight4
773
+ module.decoder.layers.11.mlp.experts.linear_fc1.weight6
774
  Params for bucket 5 (50331648 elements, 50331648 padded size):
775
  module.decoder.layers.11.mlp.experts.linear_fc1.weight7
 
776
  module.decoder.layers.11.mlp.experts.linear_fc1.weight8
777
+ module.decoder.layers.11.mlp.experts.linear_fc1.weight9
778
  module.decoder.layers.11.mlp.experts.linear_fc1.weight10
779
  Params for bucket 6 (50331648 elements, 50331648 padded size):
780
  module.decoder.layers.11.mlp.experts.linear_fc1.weight11
781
+ module.decoder.layers.11.mlp.experts.linear_fc1.weight14
782
  module.decoder.layers.11.mlp.experts.linear_fc1.weight12
783
  module.decoder.layers.11.mlp.experts.linear_fc1.weight13
 
784
  Params for bucket 7 (44040192 elements, 44040192 padded size):
785
  module.decoder.layers.11.mlp.experts.linear_fc1.weight15
786
  module.decoder.layers.10.mlp.experts.linear_fc2.weight0
787
  module.decoder.layers.10.mlp.experts.linear_fc2.weight2
788
  module.decoder.layers.10.mlp.experts.linear_fc2.weight4
 
789
  module.decoder.layers.10.mlp.experts.linear_fc2.weight3
790
+ module.decoder.layers.10.mlp.experts.linear_fc2.weight1
791
  Params for bucket 8 (44040192 elements, 44040192 padded size):
 
792
  module.decoder.layers.10.mlp.experts.linear_fc2.weight6
793
+ module.decoder.layers.10.mlp.experts.linear_fc2.weight8
794
  module.decoder.layers.10.mlp.experts.linear_fc2.weight10
795
  module.decoder.layers.10.mlp.experts.linear_fc2.weight5
796
  module.decoder.layers.10.mlp.experts.linear_fc2.weight7
 
798
  module.decoder.layers.10.mlp.experts.linear_fc2.weight11
799
  Params for bucket 9 (50331648 elements, 50331648 padded size):
800
  module.decoder.layers.10.mlp.experts.linear_fc2.weight12
801
+ module.decoder.layers.10.mlp.experts.linear_fc1.weight0
802
  module.decoder.layers.10.mlp.experts.linear_fc2.weight14
 
803
  module.decoder.layers.10.mlp.experts.linear_fc2.weight13
804
  module.decoder.layers.10.mlp.experts.linear_fc2.weight15
805
+ module.decoder.layers.10.mlp.experts.linear_fc1.weight1
806
  Params for bucket 10 (50331648 elements, 50331648 padded size):
807
  module.decoder.layers.10.mlp.experts.linear_fc1.weight2
 
808
  module.decoder.layers.10.mlp.experts.linear_fc1.weight5
809
  module.decoder.layers.10.mlp.experts.linear_fc1.weight3
810
+ module.decoder.layers.10.mlp.experts.linear_fc1.weight4
811
  Params for bucket 11 (50331648 elements, 50331648 padded size):
812
  module.decoder.layers.10.mlp.experts.linear_fc1.weight6
 
813
  module.decoder.layers.10.mlp.experts.linear_fc1.weight8
814
  module.decoder.layers.10.mlp.experts.linear_fc1.weight9
815
+ module.decoder.layers.10.mlp.experts.linear_fc1.weight7
816
  Params for bucket 12 (50331648 elements, 50331648 padded size):
817
  module.decoder.layers.10.mlp.experts.linear_fc1.weight10
818
+ module.decoder.layers.10.mlp.experts.linear_fc1.weight12
819
  module.decoder.layers.10.mlp.experts.linear_fc1.weight13
820
  module.decoder.layers.10.mlp.experts.linear_fc1.weight11
 
821
  Params for bucket 13 (44040192 elements, 44040192 padded size):
 
 
822
  module.decoder.layers.10.mlp.experts.linear_fc1.weight14
 
823
  module.decoder.layers.9.mlp.experts.linear_fc2.weight2
824
+ module.decoder.layers.9.mlp.experts.linear_fc2.weight0
825
+ module.decoder.layers.10.mlp.experts.linear_fc1.weight15
826
+ module.decoder.layers.9.mlp.experts.linear_fc2.weight1
827
  Params for bucket 14 (44040192 elements, 44040192 padded size):
 
 
 
 
828
  module.decoder.layers.9.mlp.experts.linear_fc2.weight6
829
  module.decoder.layers.9.mlp.experts.linear_fc2.weight4
830
  module.decoder.layers.9.mlp.experts.linear_fc2.weight8
831
+ module.decoder.layers.9.mlp.experts.linear_fc2.weight3
832
+ module.decoder.layers.9.mlp.experts.linear_fc2.weight5
833
+ module.decoder.layers.9.mlp.experts.linear_fc2.weight7
834
+ module.decoder.layers.9.mlp.experts.linear_fc2.weight9
835
  Params for bucket 15 (50331648 elements, 50331648 padded size):
836
+ module.decoder.layers.9.mlp.experts.linear_fc2.weight10
837
+ module.decoder.layers.9.mlp.experts.linear_fc2.weight12
838
+ module.decoder.layers.9.mlp.experts.linear_fc2.weight14
839
  module.decoder.layers.9.mlp.experts.linear_fc2.weight11
840
+ module.decoder.layers.9.mlp.experts.linear_fc2.weight13
841
  module.decoder.layers.9.mlp.experts.linear_fc2.weight15
842
  module.decoder.layers.9.mlp.experts.linear_fc1.weight0
 
 
 
843
  Params for bucket 16 (50331648 elements, 50331648 padded size):
844
  module.decoder.layers.9.mlp.experts.linear_fc1.weight1
 
845
  module.decoder.layers.9.mlp.experts.linear_fc1.weight4
846
  module.decoder.layers.9.mlp.experts.linear_fc1.weight2
847
+ module.decoder.layers.9.mlp.experts.linear_fc1.weight3
848
  Params for bucket 17 (50331648 elements, 50331648 padded size):
849
  module.decoder.layers.9.mlp.experts.linear_fc1.weight5
 
850
  module.decoder.layers.9.mlp.experts.linear_fc1.weight7
851
  module.decoder.layers.9.mlp.experts.linear_fc1.weight8
852
+ module.decoder.layers.9.mlp.experts.linear_fc1.weight6
853
  Params for bucket 18 (50331648 elements, 50331648 padded size):
854
  module.decoder.layers.9.mlp.experts.linear_fc1.weight9
855
+ module.decoder.layers.9.mlp.experts.linear_fc1.weight11
856
  module.decoder.layers.9.mlp.experts.linear_fc1.weight12
857
  module.decoder.layers.9.mlp.experts.linear_fc1.weight10
 
858
  Params for bucket 19 (44040192 elements, 44040192 padded size):
859
  module.decoder.layers.9.mlp.experts.linear_fc1.weight13
 
860
  module.decoder.layers.9.mlp.experts.linear_fc1.weight14
861
  module.decoder.layers.9.mlp.experts.linear_fc1.weight15
862
+ module.decoder.layers.8.mlp.experts.linear_fc2.weight0
863
  Params for bucket 20 (44040192 elements, 44040192 padded size):
864
  module.decoder.layers.8.mlp.experts.linear_fc2.weight1
865
  module.decoder.layers.8.mlp.experts.linear_fc2.weight5
 
873
  module.decoder.layers.8.mlp.experts.linear_fc2.weight9
874
  module.decoder.layers.8.mlp.experts.linear_fc2.weight13
875
  module.decoder.layers.8.mlp.experts.linear_fc2.weight8
 
876
  module.decoder.layers.8.mlp.experts.linear_fc2.weight12
877
+ module.decoder.layers.8.mlp.experts.linear_fc2.weight10
878
  module.decoder.layers.8.mlp.experts.linear_fc2.weight14
879
  Params for bucket 22 (44040192 elements, 44040192 padded size):
880
  module.decoder.layers.8.mlp.experts.linear_fc2.weight15
 
881
  module.decoder.layers.8.mlp.experts.linear_fc1.weight2
882
+ module.decoder.layers.8.mlp.experts.linear_fc1.weight0
883
  module.decoder.layers.8.mlp.experts.linear_fc1.weight1
884
  Params for bucket 23 (50331648 elements, 50331648 padded size):
885
  module.decoder.layers.8.mlp.experts.linear_fc1.weight3
886
+ module.decoder.layers.8.mlp.experts.linear_fc1.weight5
887
  module.decoder.layers.8.mlp.experts.linear_fc1.weight6
888
  module.decoder.layers.8.mlp.experts.linear_fc1.weight4
 
889
  Params for bucket 24 (50331648 elements, 50331648 padded size):
890
  module.decoder.layers.8.mlp.experts.linear_fc1.weight7
891
+ module.decoder.layers.8.mlp.experts.linear_fc1.weight8
892
  module.decoder.layers.8.mlp.experts.linear_fc1.weight9
893
  module.decoder.layers.8.mlp.experts.linear_fc1.weight10
 
894
  Params for bucket 25 (50331648 elements, 50331648 padded size):
895
  module.decoder.layers.8.mlp.experts.linear_fc1.weight11
896
  module.decoder.layers.8.mlp.experts.linear_fc1.weight12
897
  module.decoder.layers.8.mlp.experts.linear_fc1.weight13
898
  module.decoder.layers.8.mlp.experts.linear_fc1.weight14
899
  Params for bucket 26 (44040192 elements, 44040192 padded size):
 
 
 
900
  module.decoder.layers.7.mlp.experts.linear_fc2.weight0
901
  module.decoder.layers.7.mlp.experts.linear_fc2.weight2
902
  module.decoder.layers.7.mlp.experts.linear_fc2.weight4
903
+ module.decoder.layers.8.mlp.experts.linear_fc1.weight15
904
+ module.decoder.layers.7.mlp.experts.linear_fc2.weight1
905
+ module.decoder.layers.7.mlp.experts.linear_fc2.weight3
906
  Params for bucket 27 (44040192 elements, 44040192 padded size):
907
+ module.decoder.layers.7.mlp.experts.linear_fc2.weight8
908
+ module.decoder.layers.7.mlp.experts.linear_fc2.weight6
909
+ module.decoder.layers.7.mlp.experts.linear_fc2.weight10
910
  module.decoder.layers.7.mlp.experts.linear_fc2.weight5
911
  module.decoder.layers.7.mlp.experts.linear_fc2.weight7
912
  module.decoder.layers.7.mlp.experts.linear_fc2.weight9
913
  module.decoder.layers.7.mlp.experts.linear_fc2.weight11
 
 
 
914
  Params for bucket 28 (50331648 elements, 50331648 padded size):
 
 
 
915
  module.decoder.layers.7.mlp.experts.linear_fc2.weight12
916
  module.decoder.layers.7.mlp.experts.linear_fc2.weight14
917
  module.decoder.layers.7.mlp.experts.linear_fc1.weight1
918
+ module.decoder.layers.7.mlp.experts.linear_fc1.weight0
919
+ module.decoder.layers.7.mlp.experts.linear_fc2.weight15
920
+ module.decoder.layers.7.mlp.experts.linear_fc2.weight13
921
  Params for bucket 29 (50331648 elements, 50331648 padded size):
922
  module.decoder.layers.7.mlp.experts.linear_fc1.weight2
 
923
  module.decoder.layers.7.mlp.experts.linear_fc1.weight5
924
+ module.decoder.layers.7.mlp.experts.linear_fc1.weight4
925
  module.decoder.layers.7.mlp.experts.linear_fc1.weight3
926
  Params for bucket 30 (50331648 elements, 50331648 padded size):
927
  module.decoder.layers.7.mlp.experts.linear_fc1.weight6
 
928
  module.decoder.layers.7.mlp.experts.linear_fc1.weight9
929
  module.decoder.layers.7.mlp.experts.linear_fc1.weight8
930
+ module.decoder.layers.7.mlp.experts.linear_fc1.weight7
931
  Params for bucket 31 (50331648 elements, 50331648 padded size):
932
  module.decoder.layers.7.mlp.experts.linear_fc1.weight10
933
  module.decoder.layers.7.mlp.experts.linear_fc1.weight11
934
  module.decoder.layers.7.mlp.experts.linear_fc1.weight12
935
  module.decoder.layers.7.mlp.experts.linear_fc1.weight13
936
  Params for bucket 32 (44040192 elements, 44040192 padded size):
937
+ module.decoder.layers.6.mlp.experts.linear_fc2.weight1
938
+ module.decoder.layers.7.mlp.experts.linear_fc1.weight15
939
  module.decoder.layers.7.mlp.experts.linear_fc1.weight14
940
  module.decoder.layers.6.mlp.experts.linear_fc2.weight0
941
  module.decoder.layers.6.mlp.experts.linear_fc2.weight2
 
 
942
  Params for bucket 33 (44040192 elements, 44040192 padded size):
 
 
 
943
  module.decoder.layers.6.mlp.experts.linear_fc2.weight3
 
944
  module.decoder.layers.6.mlp.experts.linear_fc2.weight5
945
+ module.decoder.layers.6.mlp.experts.linear_fc2.weight7
946
  module.decoder.layers.6.mlp.experts.linear_fc2.weight9
947
+ module.decoder.layers.6.mlp.experts.linear_fc2.weight4
948
+ module.decoder.layers.6.mlp.experts.linear_fc2.weight6
949
+ module.decoder.layers.6.mlp.experts.linear_fc2.weight8
950
  Params for bucket 34 (50331648 elements, 50331648 padded size):
 
 
 
951
  module.decoder.layers.6.mlp.experts.linear_fc2.weight13
952
  module.decoder.layers.6.mlp.experts.linear_fc2.weight11
953
  module.decoder.layers.6.mlp.experts.linear_fc2.weight15
954
+ module.decoder.layers.6.mlp.experts.linear_fc2.weight10
955
+ module.decoder.layers.6.mlp.experts.linear_fc2.weight14
956
+ module.decoder.layers.6.mlp.experts.linear_fc2.weight12
957
  module.decoder.layers.6.mlp.experts.linear_fc1.weight0
958
  Params for bucket 35 (50331648 elements, 50331648 padded size):
959
  module.decoder.layers.6.mlp.experts.linear_fc1.weight1
960
+ module.decoder.layers.6.mlp.experts.linear_fc1.weight2
961
  module.decoder.layers.6.mlp.experts.linear_fc1.weight3
962
  module.decoder.layers.6.mlp.experts.linear_fc1.weight4
 
963
  Params for bucket 36 (50331648 elements, 50331648 padded size):
964
  module.decoder.layers.6.mlp.experts.linear_fc1.weight5
965
  module.decoder.layers.6.mlp.experts.linear_fc1.weight6
 
966
  module.decoder.layers.6.mlp.experts.linear_fc1.weight8
967
+ module.decoder.layers.6.mlp.experts.linear_fc1.weight7
968
  Params for bucket 37 (50331648 elements, 50331648 padded size):
969
  module.decoder.layers.6.mlp.experts.linear_fc1.weight9
 
 
970
  module.decoder.layers.6.mlp.experts.linear_fc1.weight12
971
+ module.decoder.layers.6.mlp.experts.linear_fc1.weight11
972
+ module.decoder.layers.6.mlp.experts.linear_fc1.weight10
973
  Params for bucket 38 (44040192 elements, 44040192 padded size):
974
  module.decoder.layers.6.mlp.experts.linear_fc1.weight13
975
  module.decoder.layers.6.mlp.experts.linear_fc1.weight15
 
976
  module.decoder.layers.5.mlp.experts.linear_fc2.weight0
977
+ module.decoder.layers.6.mlp.experts.linear_fc1.weight14
978
  Params for bucket 39 (44040192 elements, 44040192 padded size):
 
 
 
979
  module.decoder.layers.5.mlp.experts.linear_fc2.weight1
 
980
  module.decoder.layers.5.mlp.experts.linear_fc2.weight3
981
+ module.decoder.layers.5.mlp.experts.linear_fc2.weight5
982
  module.decoder.layers.5.mlp.experts.linear_fc2.weight7
983
+ module.decoder.layers.5.mlp.experts.linear_fc2.weight4
984
+ module.decoder.layers.5.mlp.experts.linear_fc2.weight2
985
+ module.decoder.layers.5.mlp.experts.linear_fc2.weight6
986
  Params for bucket 40 (44040192 elements, 44040192 padded size):
987
+ module.decoder.layers.5.mlp.experts.linear_fc2.weight9
988
+ module.decoder.layers.5.mlp.experts.linear_fc2.weight11
989
+ module.decoder.layers.5.mlp.experts.linear_fc2.weight13
990
  module.decoder.layers.5.mlp.experts.linear_fc2.weight8
 
991
  module.decoder.layers.5.mlp.experts.linear_fc2.weight10
992
+ module.decoder.layers.5.mlp.experts.linear_fc2.weight12
993
  module.decoder.layers.5.mlp.experts.linear_fc2.weight14
 
 
 
994
  Params for bucket 41 (44040192 elements, 44040192 padded size):
995
  module.decoder.layers.5.mlp.experts.linear_fc2.weight15
996
+ module.decoder.layers.5.mlp.experts.linear_fc1.weight0
997
  module.decoder.layers.5.mlp.experts.linear_fc1.weight1
998
  module.decoder.layers.5.mlp.experts.linear_fc1.weight2
 
999
  Params for bucket 42 (50331648 elements, 50331648 padded size):
1000
  module.decoder.layers.5.mlp.experts.linear_fc1.weight3
 
1001
  module.decoder.layers.5.mlp.experts.linear_fc1.weight6
1002
  module.decoder.layers.5.mlp.experts.linear_fc1.weight4
1003
+ module.decoder.layers.5.mlp.experts.linear_fc1.weight5
1004
  Params for bucket 43 (50331648 elements, 50331648 padded size):
1005
  module.decoder.layers.5.mlp.experts.linear_fc1.weight7
 
1006
  module.decoder.layers.5.mlp.experts.linear_fc1.weight9
1007
  module.decoder.layers.5.mlp.experts.linear_fc1.weight10
1008
+ module.decoder.layers.5.mlp.experts.linear_fc1.weight8
1009
  Params for bucket 44 (50331648 elements, 50331648 padded size):
1010
  module.decoder.layers.5.mlp.experts.linear_fc1.weight11
 
1011
  module.decoder.layers.5.mlp.experts.linear_fc1.weight12
1012
  module.decoder.layers.5.mlp.experts.linear_fc1.weight13
1013
+ module.decoder.layers.5.mlp.experts.linear_fc1.weight14
1014
  Params for bucket 45 (44040192 elements, 44040192 padded size):
 
 
 
 
1015
  module.decoder.layers.4.mlp.experts.linear_fc2.weight2
1016
+ module.decoder.layers.4.mlp.experts.linear_fc2.weight0
1017
  module.decoder.layers.4.mlp.experts.linear_fc2.weight4
1018
+ module.decoder.layers.5.mlp.experts.linear_fc1.weight15
1019
+ module.decoder.layers.4.mlp.experts.linear_fc2.weight1
1020
+ module.decoder.layers.4.mlp.experts.linear_fc2.weight3
1021
  Params for bucket 46 (44040192 elements, 44040192 padded size):
1022
+ module.decoder.layers.4.mlp.experts.linear_fc2.weight8
1023
+ module.decoder.layers.4.mlp.experts.linear_fc2.weight6
1024
+ module.decoder.layers.4.mlp.experts.linear_fc2.weight10
1025
  module.decoder.layers.4.mlp.experts.linear_fc2.weight5
 
1026
  module.decoder.layers.4.mlp.experts.linear_fc2.weight7
1027
+ module.decoder.layers.4.mlp.experts.linear_fc2.weight9
1028
  module.decoder.layers.4.mlp.experts.linear_fc2.weight11
 
 
 
1029
  Params for bucket 47 (50331648 elements, 50331648 padded size):
 
 
 
1030
  module.decoder.layers.4.mlp.experts.linear_fc2.weight12
1031
  module.decoder.layers.4.mlp.experts.linear_fc2.weight14
1032
  module.decoder.layers.4.mlp.experts.linear_fc1.weight1
1033
+ module.decoder.layers.4.mlp.experts.linear_fc1.weight0
1034
+ module.decoder.layers.4.mlp.experts.linear_fc2.weight15
1035
+ module.decoder.layers.4.mlp.experts.linear_fc2.weight13
1036
  Params for bucket 48 (50331648 elements, 50331648 padded size):
1037
  module.decoder.layers.4.mlp.experts.linear_fc1.weight2
 
1038
  module.decoder.layers.4.mlp.experts.linear_fc1.weight4
1039
  module.decoder.layers.4.mlp.experts.linear_fc1.weight5
1040
+ module.decoder.layers.4.mlp.experts.linear_fc1.weight3
1041
  Params for bucket 49 (50331648 elements, 50331648 padded size):
1042
  module.decoder.layers.4.mlp.experts.linear_fc1.weight6
 
1043
  module.decoder.layers.4.mlp.experts.linear_fc1.weight7
1044
  module.decoder.layers.4.mlp.experts.linear_fc1.weight8
1045
+ module.decoder.layers.4.mlp.experts.linear_fc1.weight9
1046
  Params for bucket 50 (50331648 elements, 50331648 padded size):
1047
  module.decoder.layers.4.mlp.experts.linear_fc1.weight10
 
1048
  module.decoder.layers.4.mlp.experts.linear_fc1.weight13
1049
  module.decoder.layers.4.mlp.experts.linear_fc1.weight11
1050
+ module.decoder.layers.4.mlp.experts.linear_fc1.weight12
1051
  Params for bucket 51 (44040192 elements, 44040192 padded size):
1052
+ module.decoder.layers.3.mlp.experts.linear_fc2.weight1
1053
+ module.decoder.layers.4.mlp.experts.linear_fc1.weight15
1054
  module.decoder.layers.4.mlp.experts.linear_fc1.weight14
1055
  module.decoder.layers.3.mlp.experts.linear_fc2.weight2
1056
  module.decoder.layers.3.mlp.experts.linear_fc2.weight0
 
 
1057
  Params for bucket 52 (44040192 elements, 44040192 padded size):
 
 
 
1058
  module.decoder.layers.3.mlp.experts.linear_fc2.weight3
1059
  module.decoder.layers.3.mlp.experts.linear_fc2.weight5
1060
  module.decoder.layers.3.mlp.experts.linear_fc2.weight7
1061
  module.decoder.layers.3.mlp.experts.linear_fc2.weight9
1062
+ module.decoder.layers.3.mlp.experts.linear_fc2.weight4
1063
+ module.decoder.layers.3.mlp.experts.linear_fc2.weight6
1064
+ module.decoder.layers.3.mlp.experts.linear_fc2.weight8
1065
  Params for bucket 53 (50331648 elements, 50331648 padded size):
 
 
 
 
 
1066
  module.decoder.layers.3.mlp.experts.linear_fc2.weight13
1067
+ module.decoder.layers.3.mlp.experts.linear_fc2.weight11
1068
  module.decoder.layers.3.mlp.experts.linear_fc2.weight15
1069
+ module.decoder.layers.3.mlp.experts.linear_fc1.weight0
1070
+ module.decoder.layers.3.mlp.experts.linear_fc2.weight10
1071
+ module.decoder.layers.3.mlp.experts.linear_fc2.weight14
1072
+ module.decoder.layers.3.mlp.experts.linear_fc2.weight12
1073
  Params for bucket 54 (50331648 elements, 50331648 padded size):
1074
  module.decoder.layers.3.mlp.experts.linear_fc1.weight1
 
1075
  module.decoder.layers.3.mlp.experts.linear_fc1.weight2
1076
+ module.decoder.layers.3.mlp.experts.linear_fc1.weight4
1077
  module.decoder.layers.3.mlp.experts.linear_fc1.weight3
1078
  Params for bucket 55 (50331648 elements, 50331648 padded size):
1079
  module.decoder.layers.3.mlp.experts.linear_fc1.weight5
1080
  module.decoder.layers.3.mlp.experts.linear_fc1.weight8
 
1081
  module.decoder.layers.3.mlp.experts.linear_fc1.weight6
1082
+ module.decoder.layers.3.mlp.experts.linear_fc1.weight7
1083
  Params for bucket 56 (50331648 elements, 50331648 padded size):
1084
  module.decoder.layers.3.mlp.experts.linear_fc1.weight9
1085
  module.decoder.layers.3.mlp.experts.linear_fc1.weight11
 
1086
  module.decoder.layers.3.mlp.experts.linear_fc1.weight12
1087
+ module.decoder.layers.3.mlp.experts.linear_fc1.weight10
1088
  Params for bucket 57 (44040192 elements, 44040192 padded size):
1089
  module.decoder.layers.3.mlp.experts.linear_fc1.weight13
1090
  module.decoder.layers.3.mlp.experts.linear_fc1.weight14
1091
  module.decoder.layers.3.mlp.experts.linear_fc1.weight15
1092
  module.decoder.layers.2.mlp.experts.linear_fc2.weight0
1093
  Params for bucket 58 (44040192 elements, 44040192 padded size):
 
 
 
1094
  module.decoder.layers.2.mlp.experts.linear_fc2.weight1
1095
  module.decoder.layers.2.mlp.experts.linear_fc2.weight3
1096
  module.decoder.layers.2.mlp.experts.linear_fc2.weight5
1097
  module.decoder.layers.2.mlp.experts.linear_fc2.weight7
1098
+ module.decoder.layers.2.mlp.experts.linear_fc2.weight2
1099
+ module.decoder.layers.2.mlp.experts.linear_fc2.weight4
1100
+ module.decoder.layers.2.mlp.experts.linear_fc2.weight6
1101
  Params for bucket 59 (44040192 elements, 44040192 padded size):
1102
+ module.decoder.layers.2.mlp.experts.linear_fc2.weight9
1103
+ module.decoder.layers.2.mlp.experts.linear_fc2.weight11
1104
+ module.decoder.layers.2.mlp.experts.linear_fc2.weight13
1105
  module.decoder.layers.2.mlp.experts.linear_fc2.weight8
1106
  module.decoder.layers.2.mlp.experts.linear_fc2.weight12
1107
  module.decoder.layers.2.mlp.experts.linear_fc2.weight10
1108
  module.decoder.layers.2.mlp.experts.linear_fc2.weight14
 
 
 
1109
  Params for bucket 60 (44040192 elements, 44040192 padded size):
1110
  module.decoder.layers.2.mlp.experts.linear_fc2.weight15
 
1111
  module.decoder.layers.2.mlp.experts.linear_fc1.weight2
1112
  module.decoder.layers.2.mlp.experts.linear_fc1.weight0
1113
+ module.decoder.layers.2.mlp.experts.linear_fc1.weight1
1114
  Params for bucket 61 (50331648 elements, 50331648 padded size):
1115
  module.decoder.layers.2.mlp.experts.linear_fc1.weight3
 
1116
  module.decoder.layers.2.mlp.experts.linear_fc1.weight5
1117
  module.decoder.layers.2.mlp.experts.linear_fc1.weight6
1118
+ module.decoder.layers.2.mlp.experts.linear_fc1.weight4
1119
  Params for bucket 62 (50331648 elements, 50331648 padded size):
1120
  module.decoder.layers.2.mlp.experts.linear_fc1.weight7
 
 
1121
  module.decoder.layers.2.mlp.experts.linear_fc1.weight9
1122
+ module.decoder.layers.2.mlp.experts.linear_fc1.weight8
1123
+ module.decoder.layers.2.mlp.experts.linear_fc1.weight10
1124
  Params for bucket 63 (50331648 elements, 50331648 padded size):
1125
  module.decoder.layers.2.mlp.experts.linear_fc1.weight11
1126
+ module.decoder.layers.2.mlp.experts.linear_fc1.weight12
1127
  module.decoder.layers.2.mlp.experts.linear_fc1.weight13
1128
  module.decoder.layers.2.mlp.experts.linear_fc1.weight14
 
1129
  Params for bucket 64 (44040192 elements, 44040192 padded size):
1130
  module.decoder.layers.2.mlp.experts.linear_fc1.weight15
1131
+ module.decoder.layers.1.mlp.experts.linear_fc2.weight1
1132
+ module.decoder.layers.1.mlp.experts.linear_fc2.weight3
1133
  module.decoder.layers.1.mlp.experts.linear_fc2.weight2
1134
  module.decoder.layers.1.mlp.experts.linear_fc2.weight0
1135
  module.decoder.layers.1.mlp.experts.linear_fc2.weight4
 
 
1136
  Params for bucket 65 (44040192 elements, 44040192 padded size):
 
 
 
1137
  module.decoder.layers.1.mlp.experts.linear_fc2.weight5
 
1138
  module.decoder.layers.1.mlp.experts.linear_fc2.weight7
1139
+ module.decoder.layers.1.mlp.experts.linear_fc2.weight9
1140
  module.decoder.layers.1.mlp.experts.linear_fc2.weight11
1141
+ module.decoder.layers.1.mlp.experts.linear_fc2.weight8
1142
+ module.decoder.layers.1.mlp.experts.linear_fc2.weight6
1143
+ module.decoder.layers.1.mlp.experts.linear_fc2.weight10
1144
  Params for bucket 66 (50331648 elements, 50331648 padded size):
1145
+ module.decoder.layers.1.mlp.experts.linear_fc2.weight13
1146
+ module.decoder.layers.1.mlp.experts.linear_fc2.weight15
1147
+ module.decoder.layers.1.mlp.experts.linear_fc1.weight0
1148
  module.decoder.layers.1.mlp.experts.linear_fc2.weight12
1149
  module.decoder.layers.1.mlp.experts.linear_fc2.weight14
 
 
 
1150
  module.decoder.layers.1.mlp.experts.linear_fc1.weight1
1151
  Params for bucket 67 (50331648 elements, 50331648 padded size):
1152
  module.decoder.layers.1.mlp.experts.linear_fc1.weight2
 
1153
  module.decoder.layers.1.mlp.experts.linear_fc1.weight4
1154
  module.decoder.layers.1.mlp.experts.linear_fc1.weight5
1155
+ module.decoder.layers.1.mlp.experts.linear_fc1.weight3
1156
  Params for bucket 68 (50331648 elements, 50331648 padded size):
1157
  module.decoder.layers.1.mlp.experts.linear_fc1.weight6
1158
  module.decoder.layers.1.mlp.experts.linear_fc1.weight7
 
1160
  module.decoder.layers.1.mlp.experts.linear_fc1.weight9
1161
  Params for bucket 69 (50331648 elements, 50331648 padded size):
1162
  module.decoder.layers.1.mlp.experts.linear_fc1.weight10
 
1163
  module.decoder.layers.1.mlp.experts.linear_fc1.weight11
1164
  module.decoder.layers.1.mlp.experts.linear_fc1.weight12
1165
+ module.decoder.layers.1.mlp.experts.linear_fc1.weight13
1166
  Params for bucket 70 (44040192 elements, 44040192 padded size):
1167
  module.decoder.layers.1.mlp.experts.linear_fc1.weight14
1168
  module.decoder.layers.0.mlp.experts.linear_fc2.weight1
 
1169
  module.decoder.layers.0.mlp.experts.linear_fc2.weight0
1170
  module.decoder.layers.1.mlp.experts.linear_fc1.weight15
1171
+ module.decoder.layers.0.mlp.experts.linear_fc2.weight2
1172
  Params for bucket 71 (44040192 elements, 44040192 padded size):
1173
  module.decoder.layers.0.mlp.experts.linear_fc2.weight3
1174
  module.decoder.layers.0.mlp.experts.linear_fc2.weight5
1175
  module.decoder.layers.0.mlp.experts.linear_fc2.weight7
 
 
 
1176
  module.decoder.layers.0.mlp.experts.linear_fc2.weight8
1177
+ module.decoder.layers.0.mlp.experts.linear_fc2.weight6
1178
+ module.decoder.layers.0.mlp.experts.linear_fc2.weight4
1179
+ module.decoder.layers.0.mlp.experts.linear_fc2.weight9
1180
  Params for bucket 72 (50331648 elements, 50331648 padded size):
 
 
 
1181
  module.decoder.layers.0.mlp.experts.linear_fc2.weight10
1182
  module.decoder.layers.0.mlp.experts.linear_fc2.weight14
1183
  module.decoder.layers.0.mlp.experts.linear_fc2.weight12
1184
  module.decoder.layers.0.mlp.experts.linear_fc1.weight0
1185
+ module.decoder.layers.0.mlp.experts.linear_fc2.weight13
1186
+ module.decoder.layers.0.mlp.experts.linear_fc2.weight11
1187
+ module.decoder.layers.0.mlp.experts.linear_fc2.weight15
1188
  Params for bucket 73 (50331648 elements, 50331648 padded size):
1189
  module.decoder.layers.0.mlp.experts.linear_fc1.weight1
 
 
1190
  module.decoder.layers.0.mlp.experts.linear_fc1.weight2
1191
+ module.decoder.layers.0.mlp.experts.linear_fc1.weight3
1192
+ module.decoder.layers.0.mlp.experts.linear_fc1.weight4
1193
  Params for bucket 74 (50331648 elements, 50331648 padded size):
1194
  module.decoder.layers.0.mlp.experts.linear_fc1.weight5
 
1195
  module.decoder.layers.0.mlp.experts.linear_fc1.weight8
1196
  module.decoder.layers.0.mlp.experts.linear_fc1.weight6
1197
+ module.decoder.layers.0.mlp.experts.linear_fc1.weight7
1198
  Params for bucket 75 (50331648 elements, 50331648 padded size):
1199
  module.decoder.layers.0.mlp.experts.linear_fc1.weight9
 
1200
  module.decoder.layers.0.mlp.experts.linear_fc1.weight12
1201
  module.decoder.layers.0.mlp.experts.linear_fc1.weight10
1202
+ module.decoder.layers.0.mlp.experts.linear_fc1.weight11
1203
  Params for bucket 76 (37748736 elements, 37748736 padded size):
1204
  module.decoder.layers.0.mlp.experts.linear_fc1.weight13
1205
  module.decoder.layers.0.mlp.experts.linear_fc1.weight15
1206
  module.decoder.layers.0.mlp.experts.linear_fc1.weight14
1207
+ INFO:megatron.core.optimizer:Setting up optimizer with config OptimizerConfig(optimizer='adam', lr=0.0001, min_lr=1e-05, decoupled_lr=None, decoupled_min_lr=None, weight_decay=0.1, fp8_recipe='mxfp8', fp16=False, bf16=True, reuse_grad_buf_for_mxfp8_param_ag=False, params_dtype=torch.bfloat16, use_precision_aware_optimizer=False, store_param_remainders=True, main_grads_dtype=torch.float32, main_params_dtype=torch.float32, exp_avg_dtype=torch.float32, exp_avg_sq_dtype=torch.float32, loss_scale=None, initial_loss_scale=4294967296, min_loss_scale=1.0, loss_scale_window=1000, hysteresis=2, adam_beta1=0.9, adam_beta2=0.999, adam_eps=1e-08, sgd_momentum=0.9, use_distributed_optimizer=True, overlap_param_gather_with_optimizer_step=False, optimizer_cpu_offload=False, optimizer_offload_fraction=1.0, use_torch_optimizer_for_cpu_offload=False, overlap_cpu_optimizer_d2h_h2d=False, pin_cpu_grads=True, pin_cpu_params=True, clip_grad=1.0, log_num_zeros_in_grad=False, barrier_with_L1_time=True, timers=<megatron.core.timers.Timers object at 0x7741db81d250>, config_logger_dir='')
1208
  INFO:megatron.core.optimizer_param_scheduler:> learning rate decay style: cosine
1209
+ [after model, optimizer, and learning rate scheduler are built] datetime: 2026-08-30 01:34:14
1210
  > building train, validation, and test datasets ...
1211
  > datasets target sizes (minimum size):
1212
  train: 96
 
1214
  test: 0
1215
  INFO:megatron.core.datasets.blended_megatron_dataset_config:Let split_matrix = [(0, 1.0), None, None]
1216
  > building train, validation, and test datasets for GPT ...
1217
+ INFO:megatron.core.datasets.blended_megatron_dataset_builder:Building GPTDataset splits with sizes=(96, 0, 0) and config=GPTDatasetConfig(random_seed=1234, sequence_length=8192, blend=(['/workspace/data/wikitext/wikitext103_gpt2_text_document'], None), blend_per_split=None, split='100,0,0', split_matrix=[(0, 1.0), None, None], num_dataset_builder_threads=1, path_to_cache=None, mmap_bin_files=True, mock=False, tokenizer=<megatron.training.tokenizer.tokenizer._GPT2BPETokenizer object at 0x773f352a9970>, mid_level_dataset_surplus=0.005, reset_position_ids=False, reset_attention_mask=False, eod_mask_loss=False, create_attention_mask=True, drop_last_partial_validation_sequence=True, add_extra_token_to_sequence=True, object_storage_cache_path=None)
1218
  INFO:megatron.core.datasets.indexed_dataset:Load the _IndexReader from /workspace/data/wikitext/wikitext103_gpt2_text_document.idx
1219
  INFO:megatron.core.datasets.indexed_dataset: Extract the sequence lengths
1220
  INFO:megatron.core.datasets.indexed_dataset: Extract the sequence pointers
 
1227
  INFO:megatron.core.datasets.gpt_dataset: Load the shuffle index from eadc3756447e39d609e8aa728a91f451-GPTDataset-train-shuffle_index.npy
1228
  INFO:megatron.core.datasets.gpt_dataset:> total number of samples: 14536
1229
  > finished creating GPT datasets ...
1230
+ [after dataloaders are built] datetime: 2026-08-30 01:34:14
1231
  done with setup ...
1232
  training ...
1233
  Overwriting rerun_state_machine.current_iteration from -1 to 0...
1234
  (min, max) time across ranks (ms):
1235
+ model-and-optimizer-setup ......................: (3338.00, 3360.91)
1236
+ train/valid/test-data-iterators-setup ..........: (63.69, 75.16)
1237
+ [before the start of training step] datetime: 2026-08-30 01:34:14
1238
  INFO:moonep.api:MoonEP comm buffer sizes:
1239
  config : R=8, S=8192, H=4096, E=128, K=8, NvS=69600, NvS_padded=69632
1240
  sm_count : num_sms=32, num_sms_dedup=148
 
1246
  [MoonEP selfcheck 2/4] dropped groups empty OK | covered rows [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 128, 129, 130, 131, 132, 133, 134, 135, 136, 137, 138, 139, 140, 141, 142, 143] sizes [9344, 2688, 128, 24192, 128, 0, 128, 256, 128, 768, 10496, 128, 128, 2432, 128, 1536, 14208, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0] | duplicated experts [[69, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1], [88, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1], [55, 52, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1], [-1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1], [63, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1], [124, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1], [10, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1], [58, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1]]
1247
  [MoonEP selfcheck 3/4] dropped groups empty OK | covered rows [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 128, 129, 130, 131, 132, 133, 134, 135, 136, 137, 138, 139, 140, 141, 142, 143] sizes [128, 128, 128, 0, 128, 44416, 128, 128, 128, 6912, 128, 256, 128, 128, 128, 128, 13824, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0] | duplicated experts [[78, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1], [111, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1], [62, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1], [-1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1], [57, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1], [106, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1], [121, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1], [57, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1]]
1248
  [MoonEP selfcheck 4/4] dropped groups empty OK | covered rows [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 128, 129, 130, 131, 132, 133, 134, 135, 136, 137, 138, 139, 140, 141, 142, 143] sizes [2304, 128, 128, 512, 128, 12416, 128, 1152, 1280, 16768, 640, 29056, 128, 128, 128, 1664, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0] | duplicated experts [[-1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1], [118, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1], [11, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1], [118, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1], [39, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1], [11, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1], [73, 79, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1], [86, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1]]
1249
+ [2026-08-30 01:34:35] iteration 1/ 12 | consumed samples: 8 | elapsed time per iteration (ms): 21275.7 | throughput per GPU (TFLOP/s/GPU): 8.5 | learning rate: 2.000000E-05 | global batch size: 8 | lm loss: 1.115993E+01 | loss scale: 1.0 | grad norm: 44.806 | number of skipped iterations: 0 | number of nan iterations: 0 |
1250
  Number of parameters in transformer block in billions: 29.85
1251
  Number of parameters in embedding layers in billions: 0.41
1252
  Total number of parameters in billions: 30.26
1253
  Number of parameters in most loaded shard in billions: 30.2590
1254
  Theoretical memory footprints: weight and optimizer=216428.96 MB
1255
  [Rank 0] (after 1 iterations) memory (MB) | allocated: 67239.548828125 | max allocated: 67239.54931640625 | reserved: 68548.0 | max reserved: 68548.0
1256
+ [2026-08-30 01:34:36] iteration 2/ 12 | consumed samples: 16 | elapsed time per iteration (ms): 364.0 | throughput per GPU (TFLOP/s/GPU): 496.8 | learning rate: 4.000000E-05 | global batch size: 8 | lm loss: 1.117600E+01 | loss scale: 1.0 | grad norm: 44.336 | number of skipped iterations: 0 | number of nan iterations: 0 |
1257
+ [2026-08-30 01:34:36] iteration 3/ 12 | consumed samples: 24 | elapsed time per iteration (ms): 331.4 | throughput per GPU (TFLOP/s/GPU): 545.7 | learning rate: 6.000000E-05 | global batch size: 8 | lm loss: 9.686085E+00 | loss scale: 1.0 | grad norm: 62.885 | number of skipped iterations: 0 | number of nan iterations: 0 |
1258
  Capture CUDA graph for training!!!
1259
  CUDA graph capture done!!!
1260
+ [2026-08-30 01:34:37] iteration 4/ 12 | consumed samples: 32 | elapsed time per iteration (ms): 979.0 | throughput per GPU (TFLOP/s/GPU): 184.7 | learning rate: 8.000000E-05 | global batch size: 8 | lm loss: 1.338064E+01 | loss scale: 1.0 | grad norm: 20.299 | number of skipped iterations: 0 | number of nan iterations: 0 |
1261
+ [2026-08-30 01:34:37] iteration 5/ 12 | consumed samples: 40 | elapsed time per iteration (ms): 319.7 | throughput per GPU (TFLOP/s/GPU): 565.7 | learning rate: 1.000000E-04 | global batch size: 8 | lm loss: 1.049333E+01 | loss scale: 1.0 | grad norm: 95.200 | number of skipped iterations: 0 | number of nan iterations: 0 |
1262
+ [2026-08-30 01:34:38] iteration 6/ 12 | consumed samples: 48 | elapsed time per iteration (ms): 318.9 | throughput per GPU (TFLOP/s/GPU): 567.0 | learning rate: 7.750000E-05 | global batch size: 8 | lm loss: 1.048383E+01 | loss scale: 1.0 | grad norm: 54.327 | number of skipped iterations: 0 | number of nan iterations: 0 |
1263
+ [2026-08-30 01:34:38] iteration 7/ 12 | consumed samples: 56 | elapsed time per iteration (ms): 318.4 | throughput per GPU (TFLOP/s/GPU): 568.0 | learning rate: 3.250000E-05 | global batch size: 8 | lm loss: 1.027052E+01 | loss scale: 1.0 | grad norm: 6.338 | number of skipped iterations: 0 | number of nan iterations: 0 |
1264
+ [2026-08-30 01:34:38] iteration 8/ 12 | consumed samples: 64 | elapsed time per iteration (ms): 318.4 | throughput per GPU (TFLOP/s/GPU): 567.8 | learning rate: 1.000000E-05 | global batch size: 8 | lm loss: 9.809936E+00 | loss scale: 1.0 | grad norm: 5.526 | number of skipped iterations: 0 | number of nan iterations: 0 |
1265
+ [2026-08-30 01:34:38] iteration 9/ 12 | consumed samples: 72 | elapsed time per iteration (ms): 317.7 | throughput per GPU (TFLOP/s/GPU): 569.2 | learning rate: 1.000000E-05 | global batch size: 8 | lm loss: 9.548426E+00 | loss scale: 1.0 | grad norm: 5.232 | number of skipped iterations: 0 | number of nan iterations: 0 |
1266
+ [2026-08-30 01:34:39] iteration 10/ 12 | consumed samples: 80 | elapsed time per iteration (ms): 318.2 | throughput per GPU (TFLOP/s/GPU): 568.2 | learning rate: 1.000000E-05 | global batch size: 8 | lm loss: 9.449459E+00 | loss scale: 1.0 | grad norm: 4.938 | number of skipped iterations: 0 | number of nan iterations: 0 |
1267
+ [2026-08-30 01:34:39] iteration 11/ 12 | consumed samples: 88 | elapsed time per iteration (ms): 318.6 | throughput per GPU (TFLOP/s/GPU): 567.6 | learning rate: 1.000000E-05 | global batch size: 8 | lm loss: 9.331503E+00 | loss scale: 1.0 | grad norm: 4.636 | number of skipped iterations: 0 | number of nan iterations: 0 |
1268
+ [2026-08-30 01:34:39] iteration 12/ 12 | consumed samples: 96 | elapsed time per iteration (ms): 318.2 | throughput per GPU (TFLOP/s/GPU): 568.4 | learning rate: 1.000000E-05 | global batch size: 8 | lm loss: 9.217281E+00 | loss scale: 1.0 | grad norm: 4.482 | number of skipped iterations: 0 | number of nan iterations: 0 |
1269
+ [after training is done] datetime: 2026-08-30 01:34:39
1270
+ [rank0]:[W830 01:34:41.434365646 ProcessGroupNCCL.cpp:1505] Warning: WARNING: destroy_process_group() was not called before program exit, which can leak resources. For more info, please see https://pytorch.org/docs/stable/distributed.html#shutdown (function operator())
1271
+ [rank6]:[W830 01:34:42.271159288 ProcessGroupNCCL.cpp:1505] Warning: WARNING: destroy_process_group() was not called before program exit, which can leak resources. For more info, please see https://pytorch.org/docs/stable/distributed.html#shutdown (function operator())
1272
+ [rank4]:[W830 01:34:42.512519329 ProcessGroupNCCL.cpp:1505] Warning: WARNING: destroy_process_group() was not called before program exit, which can leak resources. For more info, please see https://pytorch.org/docs/stable/distributed.html#shutdown (function operator())
1273
+ [rank2]:[W830 01:34:42.516537505 ProcessGroupNCCL.cpp:1505] Warning: WARNING: destroy_process_group() was not called before program exit, which can leak resources. For more info, please see https://pytorch.org/docs/stable/distributed.html#shutdown (function operator())
1274
+ [rank7]:[W830 01:34:42.636432904 ProcessGroupNCCL.cpp:1505] Warning: WARNING: destroy_process_group() was not called before program exit, which can leak resources. For more info, please see https://pytorch.org/docs/stable/distributed.html#shutdown (function operator())
1275
+ [rank3]:[W830 01:34:42.082170425 ProcessGroupNCCL.cpp:1505] Warning: WARNING: destroy_process_group() was not called before program exit, which can leak resources. For more info, please see https://pytorch.org/docs/stable/distributed.html#shutdown (function operator())
1276
+ [rank5]:[W830 01:34:42.086366756 ProcessGroupNCCL.cpp:1505] Warning: WARNING: destroy_process_group() was not called before program exit, which can leak resources. For more info, please see https://pytorch.org/docs/stable/distributed.html#shutdown (function operator())
1277
+ [rank1]:[W830 01:34:42.088352473 ProcessGroupNCCL.cpp:1505] Warning: WARNING: destroy_process_group() was not called before program exit, which can leak resources. For more info, please see https://pytorch.org/docs/stable/distributed.html#shutdown (function operator())
1278
+ W0830 01:34:46.823000 385325 torch/distributed/elastic/agent/server/api.py:719] Received 15 death signal, shutting down workers
1279
+ W0830 01:34:46.823000 385325 torch/distributed/elastic/multiprocessing/api.py:900] Sending process 385510 closing signal SIGTERM
1280
+ W0830 01:34:46.824000 385325 torch/distributed/elastic/multiprocessing/api.py:900] Sending process 385511 closing signal SIGTERM
1281
+ W0830 01:34:46.824000 385325 torch/distributed/elastic/multiprocessing/api.py:900] Sending process 385512 closing signal SIGTERM
1282
+ W0830 01:34:46.825000 385325 torch/distributed/elastic/multiprocessing/api.py:900] Sending process 385513 closing signal SIGTERM
1283
+ W0830 01:34:46.825000 385325 torch/distributed/elastic/multiprocessing/api.py:900] Sending process 385514 closing signal SIGTERM
1284
+ W0830 01:34:46.826000 385325 torch/distributed/elastic/multiprocessing/api.py:900] Sending process 385515 closing signal SIGTERM
 
 
1285
  Traceback (most recent call last):
1286
  File "/usr/local/bin/torchrun", line 8, in <module>
1287
  sys.exit(main())
 
1309
  time.sleep(monitor_interval)
1310
  File "/usr/local/lib/python3.12/dist-packages/torch/distributed/elastic/multiprocessing/api.py", line 84, in _terminate_process_handler
1311
  raise SignalException(f"Process {os.getpid()} got signal: {sigval}", sigval=sigval)
1312
+ torch.distributed.elastic.multiprocessing.api.SignalException: Process 385325 got signal: 15
tierb-fixed-moonep-full/moonep/tierb/tierb_gbs8/QWEN3_30B__moonep__matched__seq4096__gbs8__capnone__async/rep1/mem_samples.csv CHANGED
@@ -20,13 +20,13 @@
20
  3, 4
21
  4, 4
22
  5, 4
23
- 6, 2
24
  7, 1
25
  0, 1
26
  1, 1
27
  2, 1
28
  3, 1
29
- 4, 3
30
  5, 4
31
  6, 4
32
  7, 4
@@ -45,20 +45,12 @@
45
  4, 1
46
  5, 1
47
  6, 1
48
- 7, 0
49
- 0, 4
50
- 1, 1
51
- 2, 1
52
- 3, 1
53
- 4, 1
54
- 5, 1
55
- 6, 1
56
  7, 1
57
  0, 4
58
  1, 4
59
  2, 4
60
  3, 4
61
- 4, 4
62
  5, 1
63
  6, 1
64
  7, 1
@@ -70,22 +62,22 @@
70
  5, 4
71
  6, 4
72
  7, 4
73
- 0, 4
74
- 1, 47
75
- 2, 47
76
- 3, 47
77
- 4, 629
78
- 5, 629
79
- 6, 629
80
- 7, 629
81
- 0, 911
82
- 1, 919
83
- 2, 905
84
- 3, 929
85
- 4, 909
86
- 5, 929
87
- 6, 925
88
- 7, 919
89
  0, 1831
90
  1, 1831
91
  2, 1831
@@ -94,14 +86,14 @@
94
  5, 1831
95
  6, 1831
96
  7, 1831
97
- 0, 12661
98
- 1, 12677
99
- 2, 12461
100
- 3, 13213
101
- 4, 13017
102
- 5, 12697
103
- 6, 12877
104
- 7, 13133
105
  0, 47207
106
  1, 47207
107
  2, 47207
@@ -110,7 +102,7 @@
110
  5, 47207
111
  6, 47207
112
  7, 47207
113
- 0, 47547
114
  1, 47551
115
  2, 47551
116
  3, 47551
@@ -118,54 +110,54 @@
118
  5, 47551
119
  6, 47551
120
  7, 47551
121
- 0, 48133
122
- 1, 47959
123
- 2, 48161
124
- 3, 47959
125
- 4, 47959
126
- 5, 48161
127
- 6, 47959
128
- 7, 48161
129
- 0, 48721
130
- 1, 48375
131
- 2, 48749
132
- 3, 48375
133
- 4, 48511
134
- 5, 48749
135
- 6, 48375
136
- 7, 48749
137
- 0, 49251
138
  1, 48905
139
- 2, 49279
140
  3, 48905
141
  4, 49041
142
- 5, 49279
143
- 6, 48905
144
- 7, 49279
145
  0, 49251
146
- 1, 48905
147
  2, 49279
148
- 3, 48907
149
  4, 49043
150
- 5, 49279
151
  6, 48907
152
  7, 49279
153
- 0, 62095
154
- 1, 62075
155
- 2, 62089
156
- 3, 62079
157
- 4, 62107
158
- 5, 62089
159
- 6, 62079
160
- 7, 62089
161
- 0, 83419
162
- 1, 83473
163
- 2, 83455
164
- 3, 83423
165
- 4, 83495
166
- 5, 83463
167
- 6, 83419
168
- 7, 83453
169
  0, 83947
170
  1, 83951
171
  2, 83941
@@ -174,67 +166,67 @@
174
  5, 83941
175
  6, 83931
176
  7, 83941
177
- 0, 84229
178
- 1, 84255
179
- 2, 84241
180
- 3, 84233
181
- 4, 84349
182
- 5, 84291
183
- 6, 84251
184
- 7, 84331
185
- 0, 84613
186
- 1, 84617
187
- 2, 84607
188
- 3, 84597
189
- 4, 84625
190
- 5, 84607
191
- 6, 84597
192
- 7, 84607
193
- 0, 95193
194
- 1, 95129
195
- 2, 95197
196
- 3, 94989
197
- 4, 95123
198
- 5, 95213
199
- 6, 95089
200
- 7, 95221
201
- 0, 116439
202
- 1, 116343
203
- 2, 116411
204
- 3, 116193
205
- 4, 116315
206
- 5, 116411
207
- 6, 116289
208
- 7, 116411
209
- 0, 116703
210
- 1, 116611
211
- 2, 116701
212
- 3, 116459
213
- 4, 116583
214
- 5, 116701
215
- 6, 116551
216
- 7, 116701
217
- 0, 116703
218
- 1, 116611
219
- 2, 116701
220
- 3, 116459
221
- 4, 116583
222
- 5, 116701
223
- 6, 116555
224
- 7, 116701
225
- 0, 116703
226
- 1, 116611
227
- 2, 116701
228
- 3, 116459
229
- 4, 116583
230
- 5, 116701
231
- 6, 116555
232
- 7, 116701
233
- 0, 21171
234
- 1, 21174
235
- 2, 21494
236
- 3, 116457
237
- 4, 22774
238
  5, 21174
239
  6, 21174
240
  7, 21174
 
20
  3, 4
21
  4, 4
22
  5, 4
23
+ 6, 4
24
  7, 1
25
  0, 1
26
  1, 1
27
  2, 1
28
  3, 1
29
+ 4, 1
30
  5, 4
31
  6, 4
32
  7, 4
 
45
  4, 1
46
  5, 1
47
  6, 1
 
 
 
 
 
 
 
 
48
  7, 1
49
  0, 4
50
  1, 4
51
  2, 4
52
  3, 4
53
+ 4, 1
54
  5, 1
55
  6, 1
56
  7, 1
 
62
  5, 4
63
  6, 4
64
  7, 4
65
+ 0, 725
66
+ 1, 725
67
+ 2, 725
68
+ 3, 725
69
+ 4, 725
70
+ 5, 725
71
+ 6, 725
72
+ 7, 725
73
+ 0, 1079
74
+ 1, 1077
75
+ 2, 1079
76
+ 3, 1079
77
+ 4, 1079
78
+ 5, 1079
79
+ 6, 1079
80
+ 7, 1079
81
  0, 1831
82
  1, 1831
83
  2, 1831
 
86
  5, 1831
87
  6, 1831
88
  7, 1831
89
+ 0, 19271
90
+ 1, 19269
91
+ 2, 19277
92
+ 3, 19283
93
+ 4, 19291
94
+ 5, 19287
95
+ 6, 19291
96
+ 7, 19277
97
  0, 47207
98
  1, 47207
99
  2, 47207
 
102
  5, 47207
103
  6, 47207
104
  7, 47207
105
+ 0, 47549
106
  1, 47551
107
  2, 47551
108
  3, 47551
 
110
  5, 47551
111
  6, 47551
112
  7, 47551
113
+ 0, 48313
114
+ 1, 47969
115
+ 2, 48341
116
+ 3, 47969
117
+ 4, 48105
118
+ 5, 48341
119
+ 6, 47969
120
+ 7, 48341
121
+ 0, 49103
122
+ 1, 48759
123
+ 2, 49131
124
+ 3, 48759
125
+ 4, 48895
126
+ 5, 49131
127
+ 6, 48761
128
+ 7, 49131
129
+ 0, 49249
130
  1, 48905
131
+ 2, 49277
132
  3, 48905
133
  4, 49041
134
+ 5, 49277
135
+ 6, 48907
136
+ 7, 49277
137
  0, 49251
138
+ 1, 48907
139
  2, 49279
140
+ 3, 48905
141
  4, 49043
142
+ 5, 49277
143
  6, 48907
144
  7, 49279
145
+ 0, 62969
146
+ 1, 62867
147
+ 2, 62963
148
+ 3, 62953
149
+ 4, 62981
150
+ 5, 62963
151
+ 6, 62953
152
+ 7, 62963
153
+ 0, 83461
154
+ 1, 83465
155
+ 2, 83463
156
+ 3, 83461
157
+ 4, 83517
158
+ 5, 83475
159
+ 6, 83467
160
+ 7, 83469
161
  0, 83947
162
  1, 83951
163
  2, 83941
 
166
  5, 83941
167
  6, 83931
168
  7, 83941
169
+ 0, 84269
170
+ 1, 84223
171
+ 2, 84199
172
+ 3, 84223
173
+ 4, 84339
174
+ 5, 84293
175
+ 6, 84327
176
+ 7, 84265
177
+ 0, 89737
178
+ 1, 89989
179
+ 2, 89763
180
+ 3, 89657
181
+ 4, 89661
182
+ 5, 89907
183
+ 6, 89813
184
+ 7, 90003
185
+ 0, 95559
186
+ 1, 95447
187
+ 2, 95535
188
+ 3, 95295
189
+ 4, 95449
190
+ 5, 95557
191
+ 6, 95403
192
+ 7, 95569
193
+ 0, 105447
194
+ 1, 106009
195
+ 2, 105845
196
+ 3, 107203
197
+ 4, 106127
198
+ 5, 106645
199
+ 6, 105807
200
+ 7, 111847
201
+ 0, 116707
202
+ 1, 116615
203
+ 2, 116705
204
+ 3, 116463
205
+ 4, 116587
206
+ 5, 116705
207
+ 6, 116559
208
+ 7, 116705
209
+ 0, 116707
210
+ 1, 116615
211
+ 2, 116705
212
+ 3, 116463
213
+ 4, 116587
214
+ 5, 116705
215
+ 6, 116559
216
+ 7, 116705
217
+ 0, 115739
218
+ 1, 115967
219
+ 2, 116057
220
+ 3, 115815
221
+ 4, 115939
222
+ 5, 116057
223
+ 6, 115911
224
+ 7, 115795
225
+ 0, 21173
226
+ 1, 21430
227
+ 2, 116057
228
+ 3, 22454
229
+ 4, 21174
230
  5, 21174
231
  6, 21174
232
  7, 21174
tierb-fixed-moonep-full/moonep/tierb/tierb_gbs8/QWEN3_30B__moonep__matched__seq4096__gbs8__capnone__async/rep1/run.env CHANGED
@@ -3,7 +3,7 @@ config=QWEN3_30B
3
  family=tierB
4
  repo=/workspace/Megatron-MoE-Echo
5
  branch=moonep_backend_async_grad_reduce
6
- commit=7bc235bf
7
  moonep_source_dir=/workspace/MoonEP
8
  moonep_source_branch=megatron-ddp-symmetric-params
9
  moonep_source_commit=d8bc666
 
3
  family=tierB
4
  repo=/workspace/Megatron-MoE-Echo
5
  branch=moonep_backend_async_grad_reduce
6
+ commit=2e68c8ad
7
  moonep_source_dir=/workspace/MoonEP
8
  moonep_source_branch=megatron-ddp-symmetric-params
9
  moonep_source_commit=d8bc666
tierb-fixed-moonep-full/moonep/tierb/tierb_gbs8/QWEN3_30B__moonep__matched__seq4096__gbs8__capnone__async/rep1/train.log CHANGED
The diff for this file is too large to render. See raw diff
 
tierb-fixed-moonep-full/moonep/tierb/tierb_gbs8/QWEN3_30B__moonep__matched__seq4096__gbs8__capnone__async/rep2/mem_samples.csv CHANGED
@@ -26,8 +26,8 @@
26
  1, 1
27
  2, 1
28
  3, 1
29
- 4, 4
30
- 5, 4
31
  6, 4
32
  7, 4
33
  0, 0
@@ -38,7 +38,7 @@
38
  5, 0
39
  6, 0
40
  7, 0
41
- 0, 1
42
  1, 1
43
  2, 1
44
  3, 1
@@ -71,37 +71,37 @@
71
  6, 4
72
  7, 4
73
  0, 821
74
- 1, 821
75
- 2, 821
76
  3, 821
77
- 4, 821
78
- 5, 821
79
- 6, 821
80
- 7, 821
81
- 0, 1831
82
- 1, 1831
83
- 2, 1831
84
- 3, 1831
85
- 4, 1831
86
- 5, 1831
87
- 6, 1831
88
- 7, 1831
89
- 0, 4865
90
- 1, 4629
91
- 2, 4609
92
- 3, 4809
93
- 4, 4549
94
- 5, 4965
95
- 6, 4749
96
- 7, 4689
97
- 0, 46757
98
- 1, 46765
99
- 2, 46755
100
- 3, 46739
101
- 4, 46767
102
- 5, 46729
103
- 6, 46751
104
- 7, 46785
105
  0, 47547
106
  1, 47549
107
  2, 47549
@@ -110,54 +110,54 @@
110
  5, 47549
111
  6, 47549
112
  7, 47549
113
- 0, 48081
114
- 1, 47875
115
- 2, 48085
116
- 3, 47895
117
- 4, 47895
118
- 5, 48117
119
- 6, 47927
120
- 7, 48115
121
- 0, 48651
122
- 1, 48311
123
- 2, 48695
124
- 3, 48307
125
- 4, 48475
126
- 5, 48693
127
- 6, 48343
128
- 7, 48723
129
- 0, 49249
130
  1, 48905
131
  2, 49279
132
- 3, 48907
133
  4, 49041
134
  5, 49277
135
  6, 48905
136
- 7, 49279
137
  0, 49251
138
- 1, 48905
139
  2, 49279
140
  3, 48907
141
  4, 49043
142
  5, 49279
143
  6, 48907
144
  7, 49279
145
- 0, 59887
146
- 1, 59871
147
- 2, 59861
148
- 3, 59851
149
- 4, 59879
150
- 5, 59861
151
- 6, 59851
152
- 7, 59861
153
- 0, 83441
154
- 1, 83427
155
- 2, 83429
156
- 3, 83449
157
- 4, 83467
158
- 5, 83461
159
- 6, 83421
160
- 7, 83451
161
  0, 83947
162
  1, 83951
163
  2, 83941
@@ -166,67 +166,67 @@
166
  5, 83941
167
  6, 83931
168
  7, 83941
169
- 0, 84279
170
- 1, 84219
171
- 2, 84293
172
- 3, 84269
173
- 4, 84339
174
- 5, 84307
175
- 6, 84321
176
- 7, 84273
177
- 0, 84997
178
- 1, 85001
179
- 2, 84991
180
- 3, 84981
181
- 4, 85009
182
- 5, 84991
183
- 6, 84981
184
- 7, 84991
185
- 0, 95579
186
- 1, 95473
187
- 2, 95577
188
- 3, 95317
189
- 4, 95471
190
- 5, 95567
191
- 6, 95411
192
- 7, 95607
193
- 0, 103699
194
- 1, 100805
195
- 2, 102497
196
- 3, 105057
197
- 4, 102123
198
- 5, 102643
199
- 6, 101297
200
- 7, 107699
201
- 0, 116703
202
- 1, 116611
203
- 2, 116701
204
- 3, 116459
205
- 4, 116583
206
- 5, 116701
207
- 6, 116555
208
- 7, 116701
209
- 0, 116703
210
- 1, 116611
211
- 2, 116701
212
- 3, 116459
213
- 4, 116583
214
- 5, 116701
215
- 6, 116555
216
- 7, 116701
217
- 0, 23094
218
- 1, 116609
219
- 2, 116699
220
- 3, 116457
221
- 4, 116581
222
- 5, 116699
223
- 6, 116553
224
- 7, 116699
225
- 0, 22771
226
- 1, 21174
227
- 2, 21174
228
- 3, 21174
229
- 4, 21174
230
- 5, 21174
231
- 6, 21494
232
- 7, 116699
 
26
  1, 1
27
  2, 1
28
  3, 1
29
+ 4, 1
30
+ 5, 3
31
  6, 4
32
  7, 4
33
  0, 0
 
38
  5, 0
39
  6, 0
40
  7, 0
41
+ 0, 4
42
  1, 1
43
  2, 1
44
  3, 1
 
71
  6, 4
72
  7, 4
73
  0, 821
74
+ 1, 819
75
+ 2, 819
76
  3, 821
77
+ 4, 819
78
+ 5, 819
79
+ 6, 819
80
+ 7, 819
81
+ 0, 1537
82
+ 1, 1537
83
+ 2, 1537
84
+ 3, 1537
85
+ 4, 1537
86
+ 5, 1537
87
+ 6, 1537
88
+ 7, 1537
89
+ 0, 1975
90
+ 1, 1975
91
+ 2, 1975
92
+ 3, 1975
93
+ 4, 1975
94
+ 5, 1975
95
+ 6, 1975
96
+ 7, 1975
97
+ 0, 46653
98
+ 1, 46653
99
+ 2, 46645
100
+ 3, 46629
101
+ 4, 46653
102
+ 5, 46657
103
+ 6, 46665
104
+ 7, 46651
105
  0, 47547
106
  1, 47549
107
  2, 47549
 
110
  5, 47549
111
  6, 47549
112
  7, 47549
113
+ 0, 48127
114
+ 1, 47943
115
+ 2, 48147
116
+ 3, 47929
117
+ 4, 47959
118
+ 5, 48155
119
+ 6, 47929
120
+ 7, 48161
121
+ 0, 48611
122
+ 1, 48293
123
+ 2, 48643
124
+ 3, 48267
125
+ 4, 48369
126
+ 5, 48631
127
+ 6, 48267
128
+ 7, 48655
129
+ 0, 49251
130
  1, 48905
131
  2, 49279
132
+ 3, 48905
133
  4, 49041
134
  5, 49277
135
  6, 48905
136
+ 7, 49277
137
  0, 49251
138
+ 1, 48907
139
  2, 49279
140
  3, 48907
141
  4, 49043
142
  5, 49279
143
  6, 48907
144
  7, 49279
145
+ 0, 61515
146
+ 1, 61473
147
+ 2, 61549
148
+ 3, 61647
149
+ 4, 61675
150
+ 5, 61549
151
+ 6, 61519
152
+ 7, 61657
153
+ 0, 83447
154
+ 1, 83453
155
+ 2, 83441
156
+ 3, 83439
157
+ 4, 83495
158
+ 5, 83441
159
+ 6, 83447
160
+ 7, 83459
161
  0, 83947
162
  1, 83951
163
  2, 83941
 
166
  5, 83941
167
  6, 83931
168
  7, 83941
169
+ 0, 84243
170
+ 1, 84235
171
+ 2, 84193
172
+ 3, 84217
173
+ 4, 84255
174
+ 5, 84271
175
+ 6, 84265
176
+ 7, 84283
177
+ 0, 90373
178
+ 1, 90401
179
+ 2, 91611
180
+ 3, 90369
181
+ 4, 91097
182
+ 5, 90391
183
+ 6, 90001
184
+ 7, 90243
185
+ 0, 95551
186
+ 1, 95415
187
+ 2, 95557
188
+ 3, 95319
189
+ 4, 95455
190
+ 5, 95591
191
+ 6, 95427
192
+ 7, 95569
193
+ 0, 102643
194
+ 1, 102153
195
+ 2, 103697
196
+ 3, 102399
197
+ 4, 102125
198
+ 5, 102751
199
+ 6, 105409
200
+ 7, 104899
201
+ 0, 116707
202
+ 1, 116615
203
+ 2, 116705
204
+ 3, 116463
205
+ 4, 116587
206
+ 5, 116705
207
+ 6, 116559
208
+ 7, 116705
209
+ 0, 116707
210
+ 1, 116615
211
+ 2, 116705
212
+ 3, 116463
213
+ 4, 116587
214
+ 5, 116705
215
+ 6, 116559
216
+ 7, 116705
217
+ 0, 115653
218
+ 1, 115969
219
+ 2, 115651
220
+ 3, 115817
221
+ 4, 115533
222
+ 5, 116059
223
+ 6, 115505
224
+ 7, 116059
225
+ 0, 16945
226
+ 1, 16948
227
+ 2, 16948
228
+ 3, 16948
229
+ 4, 17140
230
+ 5, 17332
231
+ 6, 115119
232
+ 7, 18100
tierb-fixed-moonep-full/moonep/tierb/tierb_gbs8/QWEN3_30B__moonep__matched__seq4096__gbs8__capnone__async/rep2/run.env CHANGED
@@ -3,7 +3,7 @@ config=QWEN3_30B
3
  family=tierB
4
  repo=/workspace/Megatron-MoE-Echo
5
  branch=moonep_backend_async_grad_reduce
6
- commit=7bc235bf
7
  moonep_source_dir=/workspace/MoonEP
8
  moonep_source_branch=megatron-ddp-symmetric-params
9
  moonep_source_commit=d8bc666
 
3
  family=tierB
4
  repo=/workspace/Megatron-MoE-Echo
5
  branch=moonep_backend_async_grad_reduce
6
+ commit=2e68c8ad
7
  moonep_source_dir=/workspace/MoonEP
8
  moonep_source_branch=megatron-ddp-symmetric-params
9
  moonep_source_commit=d8bc666
tierb-fixed-moonep-full/moonep/tierb/tierb_gbs8/QWEN3_30B__moonep__matched__seq4096__gbs8__capnone__async/rep2/train.log CHANGED
The diff for this file is too large to render. See raw diff
 
tierb-fixed-moonep-full/moonep/tierb/tierb_gbs8/QWEN3_30B__moonep__matched__seq4096__gbs8__capnone__async/rep3/mem_samples.csv CHANGED
@@ -26,7 +26,7 @@
26
  1, 1
27
  2, 1
28
  3, 1
29
- 4, 3
30
  5, 4
31
  6, 4
32
  7, 4
@@ -64,20 +64,20 @@
64
  7, 4
65
  0, 4
66
  1, 629
67
- 2, 629
68
- 3, 629
69
  4, 629
70
  5, 629
71
  6, 629
72
  7, 629
73
- 0, 929
74
- 1, 935
75
- 2, 907
76
- 3, 923
77
- 4, 943
78
- 5, 921
79
- 6, 921
80
- 7, 947
81
  0, 1831
82
  1, 1831
83
  2, 1831
@@ -86,14 +86,14 @@
86
  5, 1831
87
  6, 1831
88
  7, 1831
89
- 0, 15033
90
- 1, 15189
91
- 2, 15209
92
- 3, 15169
93
- 4, 15309
94
- 5, 14441
95
- 6, 14657
96
- 7, 15229
97
  0, 47207
98
  1, 47207
99
  2, 47207
@@ -110,54 +110,54 @@
110
  5, 47551
111
  6, 47551
112
  7, 47551
113
- 0, 48133
114
- 1, 47957
115
- 2, 48151
116
  3, 47959
117
- 4, 47959
118
- 5, 48161
119
  6, 47959
120
- 7, 48161
121
  0, 49103
122
- 1, 48761
123
- 2, 49131
124
  3, 48759
125
- 4, 48897
126
- 5, 49133
127
- 6, 48761
128
- 7, 49131
129
  0, 49249
130
  1, 48907
131
- 2, 49277
132
  3, 48905
133
- 4, 49043
134
- 5, 49279
135
- 6, 48907
136
- 7, 49277
137
  0, 49251
138
  1, 48907
139
- 2, 49277
140
  3, 48907
141
- 4, 49043
142
  5, 49279
143
  6, 48907
144
  7, 49279
145
- 0, 63573
146
- 1, 63551
147
- 2, 63547
148
- 3, 63557
149
- 4, 63563
150
- 5, 63547
151
- 6, 63557
152
- 7, 63547
153
- 0, 83465
154
- 1, 83459
155
- 2, 83473
156
- 3, 83455
157
- 4, 83517
158
- 5, 83465
159
- 6, 83477
160
- 7, 83473
161
  0, 83947
162
  1, 83951
163
  2, 83941
@@ -166,67 +166,67 @@
166
  5, 83941
167
  6, 83931
168
  7, 83941
169
- 0, 84337
170
- 1, 84341
171
- 2, 84331
172
- 3, 84321
173
- 4, 84349
174
- 5, 84331
175
- 6, 84321
176
- 7, 84331
177
- 0, 84997
178
- 1, 85001
179
- 2, 84991
180
- 3, 84981
181
- 4, 85009
182
- 5, 84991
183
- 6, 84981
184
- 7, 84991
185
- 0, 95527
186
- 1, 95431
187
- 2, 95503
188
- 3, 95295
189
- 4, 95471
190
- 5, 95577
191
- 6, 95423
192
- 7, 95537
193
- 0, 103155
194
- 1, 106155
195
- 2, 103043
196
- 3, 102399
197
- 4, 102523
198
- 5, 103443
199
- 6, 102351
200
- 7, 102643
201
- 0, 116703
202
- 1, 116611
203
- 2, 116701
204
- 3, 116459
205
- 4, 116583
206
- 5, 116701
207
- 6, 116555
208
- 7, 116701
209
- 0, 116703
210
- 1, 116611
211
- 2, 116701
212
- 3, 116459
213
- 4, 116583
214
- 5, 116701
215
- 6, 116555
216
- 7, 116701
217
- 0, 23094
218
- 1, 116609
219
- 2, 116699
220
- 3, 116457
221
- 4, 116581
222
- 5, 116699
223
- 6, 116553
224
- 7, 116699
225
- 0, 21491
226
- 1, 116609
227
- 2, 22774
228
- 3, 21174
229
- 4, 21174
230
- 5, 21174
231
- 6, 21174
232
- 7, 21174
 
26
  1, 1
27
  2, 1
28
  3, 1
29
+ 4, 1
30
  5, 4
31
  6, 4
32
  7, 4
 
64
  7, 4
65
  0, 4
66
  1, 629
67
+ 2, 45
68
+ 3, 35
69
  4, 629
70
  5, 629
71
  6, 629
72
  7, 629
73
+ 0, 875
74
+ 1, 875
75
+ 2, 877
76
+ 3, 893
77
+ 4, 903
78
+ 5, 905
79
+ 6, 899
80
+ 7, 909
81
  0, 1831
82
  1, 1831
83
  2, 1831
 
86
  5, 1831
87
  6, 1831
88
  7, 1831
89
+ 0, 16477
90
+ 1, 16833
91
+ 2, 17345
92
+ 3, 16517
93
+ 4, 17209
94
+ 5, 16913
95
+ 6, 16773
96
+ 7, 16577
97
  0, 47207
98
  1, 47207
99
  2, 47207
 
110
  5, 47551
111
  6, 47551
112
  7, 47551
113
+ 0, 48207
114
+ 1, 47959
115
+ 2, 48235
116
  3, 47959
117
+ 4, 47999
118
+ 5, 48235
119
  6, 47959
120
+ 7, 48331
121
  0, 49103
122
+ 1, 48759
123
+ 2, 49133
124
  3, 48759
125
+ 4, 48895
126
+ 5, 49131
127
+ 6, 48759
128
+ 7, 49133
129
  0, 49249
130
  1, 48907
131
+ 2, 49279
132
  3, 48905
133
+ 4, 49041
134
+ 5, 49277
135
+ 6, 48905
136
+ 7, 49279
137
  0, 49251
138
  1, 48907
139
+ 2, 49279
140
  3, 48907
141
+ 4, 49041
142
  5, 49279
143
  6, 48907
144
  7, 49279
145
+ 0, 66095
146
+ 1, 66077
147
+ 2, 65899
148
+ 3, 65887
149
+ 4, 65957
150
+ 5, 65919
151
+ 6, 65909
152
+ 7, 65899
153
+ 0, 83463
154
+ 1, 83509
155
+ 2, 83483
156
+ 3, 83467
157
+ 4, 83535
158
+ 5, 83455
159
+ 6, 83487
160
+ 7, 83515
161
  0, 83947
162
  1, 83951
163
  2, 83941
 
166
  5, 83941
167
  6, 83931
168
  7, 83941
169
+ 0, 84249
170
+ 1, 84237
171
+ 2, 84221
172
+ 3, 84217
173
+ 4, 84273
174
+ 5, 84265
175
+ 6, 84201
176
+ 7, 84257
177
+ 0, 89637
178
+ 1, 89977
179
+ 2, 90223
180
+ 3, 90153
181
+ 4, 89697
182
+ 5, 90063
183
+ 6, 89849
184
+ 7, 89919
185
+ 0, 95507
186
+ 1, 95411
187
+ 2, 95461
188
+ 3, 95249
189
+ 4, 95409
190
+ 5, 95503
191
+ 6, 95353
192
+ 7, 95529
193
+ 0, 107557
194
+ 1, 108155
195
+ 2, 111849
196
+ 3, 106913
197
+ 4, 107327
198
+ 5, 107955
199
+ 6, 114103
200
+ 7, 107845
201
+ 0, 116707
202
+ 1, 116615
203
+ 2, 116705
204
+ 3, 116463
205
+ 4, 116587
206
+ 5, 116705
207
+ 6, 116559
208
+ 7, 116705
209
+ 0, 116707
210
+ 1, 116615
211
+ 2, 116705
212
+ 3, 116463
213
+ 4, 116587
214
+ 5, 116705
215
+ 6, 116559
216
+ 7, 116705
217
+ 0, 116061
218
+ 1, 115969
219
+ 2, 116059
220
+ 3, 115817
221
+ 4, 115941
222
+ 5, 116059
223
+ 6, 115913
224
+ 7, 116059
225
+ 0, 20737
226
+ 1, 20740
227
+ 2, 20740
228
+ 3, 20740
229
+ 4, 20996
230
+ 5, 116057
231
+ 6, 22020
232
+ 7, 20740
tierb-fixed-moonep-full/moonep/tierb/tierb_gbs8/QWEN3_30B__moonep__matched__seq4096__gbs8__capnone__async/rep3/run.env CHANGED
@@ -3,7 +3,7 @@ config=QWEN3_30B
3
  family=tierB
4
  repo=/workspace/Megatron-MoE-Echo
5
  branch=moonep_backend_async_grad_reduce
6
- commit=7bc235bf
7
  moonep_source_dir=/workspace/MoonEP
8
  moonep_source_branch=megatron-ddp-symmetric-params
9
  moonep_source_commit=d8bc666
 
3
  family=tierB
4
  repo=/workspace/Megatron-MoE-Echo
5
  branch=moonep_backend_async_grad_reduce
6
+ commit=2e68c8ad
7
  moonep_source_dir=/workspace/MoonEP
8
  moonep_source_branch=megatron-ddp-symmetric-params
9
  moonep_source_commit=d8bc666
tierb-fixed-moonep-full/moonep/tierb/tierb_gbs8/QWEN3_30B__moonep__matched__seq4096__gbs8__capnone__async/rep3/train.log CHANGED
The diff for this file is too large to render. See raw diff
 
tierb-fixed-moonep-full/moonep/tierb/tierb_gbs8/results.tsv CHANGED
@@ -1,13 +1,7 @@
1
  config backend policy slots seq gbs cap recompute rep status iter_ms_mean iter_ms_min peak_mib run_dir
2
- QWEN3_235B moonep matched 16 8192 8 none none 1 OK 321.60 320.40 120303 /workspace/tierb-fixed-moonep-full/moonep/tierb/tierb_gbs8/QWEN3_235B__moonep__matched__seq8192__gbs8__capnone__async/rep1
3
- QWEN3_235B moonep matched 16 8192 8 none none 2 OK 322.05 320.40 120303 /workspace/tierb-fixed-moonep-full/moonep/tierb/tierb_gbs8/QWEN3_235B__moonep__matched__seq8192__gbs8__capnone__async/rep2
4
- QWEN3_235B moonep matched 16 8192 8 none none 3 OK 322.10 320.40 120303 /workspace/tierb-fixed-moonep-full/moonep/tierb/tierb_gbs8/QWEN3_235B__moonep__matched__seq8192__gbs8__capnone__async/rep3
5
- QWEN3_30B moonep matched 16 4096 8 none none 1 OK 361.53 358.70 116703 /workspace/tierb-fixed-moonep-full/moonep/tierb/tierb_gbs8/QWEN3_30B__moonep__matched__seq4096__gbs8__capnone__async/rep1
6
- QWEN3_30B moonep matched 16 4096 8 none none 2 OK 361.17 358.70 116703 /workspace/tierb-fixed-moonep-full/moonep/tierb/tierb_gbs8/QWEN3_30B__moonep__matched__seq4096__gbs8__capnone__async/rep2
7
- QWEN3_30B moonep matched 16 4096 8 none none 3 OK 360.76 357.60 116703 /workspace/tierb-fixed-moonep-full/moonep/tierb/tierb_gbs8/QWEN3_30B__moonep__matched__seq4096__gbs8__capnone__async/rep3
8
- DSV2_LITE_L24 moonep matched 8 8192 8 none none 1 OK 596.03 595.70 133311 /workspace/tierb-fixed-moonep-full/moonep/tierb/tierb_gbs8/DSV2_LITE_L24__moonep__matched__seq8192__gbs8__capnone__async/rep1
9
- DSV2_LITE_L24 moonep matched 8 8192 8 none none 2 OK 596.18 594.10 133311 /workspace/tierb-fixed-moonep-full/moonep/tierb/tierb_gbs8/DSV2_LITE_L24__moonep__matched__seq8192__gbs8__capnone__async/rep2
10
- DSV2_LITE_L24 moonep matched 8 8192 8 none none 3 OK 595.97 594.00 133311 /workspace/tierb-fixed-moonep-full/moonep/tierb/tierb_gbs8/DSV2_LITE_L24__moonep__matched__seq8192__gbs8__capnone__async/rep3
11
- QWEN2_57B moonep matched 8 4096 8 none none 1 OK 183.70 182.60 96737 /workspace/tierb-fixed-moonep-full/moonep/tierb/tierb_gbs8/QWEN2_57B__moonep__matched__seq4096__gbs8__capnone__async/rep1
12
- QWEN2_57B moonep matched 8 4096 8 none none 2 OK 184.22 182.60 96737 /workspace/tierb-fixed-moonep-full/moonep/tierb/tierb_gbs8/QWEN2_57B__moonep__matched__seq4096__gbs8__capnone__async/rep2
13
- QWEN2_57B moonep matched 8 4096 8 none none 3 OK 184.20 182.60 96737 /workspace/tierb-fixed-moonep-full/moonep/tierb/tierb_gbs8/QWEN2_57B__moonep__matched__seq4096__gbs8__capnone__async/rep3
 
1
  config backend policy slots seq gbs cap recompute rep status iter_ms_mean iter_ms_min peak_mib run_dir
2
+ QWEN3_235B moonep matched 16 8192 8 none none 1 OK 317.23 316.40 120309 /workspace/tierb-fixed-moonep-full/moonep/tierb/tierb_gbs8/QWEN3_235B__moonep__matched__seq8192__gbs8__capnone__async/rep1
3
+ QWEN3_235B moonep matched 16 8192 8 none none 2 OK 317.53 316.40 120309 /workspace/tierb-fixed-moonep-full/moonep/tierb/tierb_gbs8/QWEN3_235B__moonep__matched__seq8192__gbs8__capnone__async/rep2
4
+ QWEN3_235B moonep matched 16 8192 8 none none 3 OK 317.80 316.40 120309 /workspace/tierb-fixed-moonep-full/moonep/tierb/tierb_gbs8/QWEN3_235B__moonep__matched__seq8192__gbs8__capnone__async/rep3
5
+ QWEN3_30B moonep matched 16 4096 8 none none 1 OK 354.41 352.30 116707 /workspace/tierb-fixed-moonep-full/moonep/tierb/tierb_gbs8/QWEN3_30B__moonep__matched__seq4096__gbs8__capnone__async/rep1
6
+ QWEN3_30B moonep matched 16 4096 8 none none 2 OK 354.46 352.20 116707 /workspace/tierb-fixed-moonep-full/moonep/tierb/tierb_gbs8/QWEN3_30B__moonep__matched__seq4096__gbs8__capnone__async/rep2
7
+ QWEN3_30B moonep matched 16 4096 8 none none 3 OK 355.16 352.20 116707 /workspace/tierb-fixed-moonep-full/moonep/tierb/tierb_gbs8/QWEN3_30B__moonep__matched__seq4096__gbs8__capnone__async/rep3