AbhishekGhosh1998 commited on
Commit
4a6b6df
·
verified ·
1 Parent(s): 05d3ca0

granularity results tiera-rerun-fixed-moonep @ 2026-08-30T06:50:39Z from sp-aiinfra-interactive-aghosh-7jtjl-8xb200-moe-cg-0

Browse files
Files changed (24) hide show
  1. tiera-rerun-fixed-moonep/.completed/echo__ponly__gbs16 +1 -0
  2. tiera-rerun-fixed-moonep/STATUS.txt +3 -2
  3. tiera-rerun-fixed-moonep/echo/a2/a2_matched_gbs16/G1__echo__matched__seq4096__gbs16__cap2p40625/rep1/mem_samples.csv +232 -0
  4. tiera-rerun-fixed-moonep/echo/a2/a2_matched_gbs16/G1__echo__matched__seq4096__gbs16__cap2p40625/rep1/run.env +70 -0
  5. tiera-rerun-fixed-moonep/echo/a2/a2_matched_gbs16/G1__echo__matched__seq4096__gbs16__cap2p40625/rep1/train.log +1023 -0
  6. tiera-rerun-fixed-moonep/echo/a2/a2_matched_gbs16/G1__echo__matched__seq4096__gbs16__cap2p40625/rep2/mem_samples.csv +264 -0
  7. tiera-rerun-fixed-moonep/echo/a2/a2_matched_gbs16/G1__echo__matched__seq4096__gbs16__cap2p40625/rep2/run.env +70 -0
  8. tiera-rerun-fixed-moonep/echo/a2/a2_matched_gbs16/G1__echo__matched__seq4096__gbs16__cap2p40625/rep2/train.log +987 -0
  9. tiera-rerun-fixed-moonep/echo/a2/a2_matched_gbs16/G1__echo__matched__seq4096__gbs16__cap2p40625/rep3/mem_samples.csv +232 -0
  10. tiera-rerun-fixed-moonep/echo/a2/a2_matched_gbs16/G1__echo__matched__seq4096__gbs16__cap2p40625/rep3/run.env +70 -0
  11. tiera-rerun-fixed-moonep/echo/a2/a2_matched_gbs16/G1__echo__matched__seq4096__gbs16__cap2p40625/rep3/train.log +1023 -0
  12. tiera-rerun-fixed-moonep/echo/a2/a2_matched_gbs16/G2__echo__matched__seq4096__gbs16__cap1p08203125/rep1/mem_samples.csv +256 -0
  13. tiera-rerun-fixed-moonep/echo/a2/a2_matched_gbs16/G2__echo__matched__seq4096__gbs16__cap1p08203125/rep1/run.env +70 -0
  14. tiera-rerun-fixed-moonep/echo/a2/a2_matched_gbs16/G2__echo__matched__seq4096__gbs16__cap1p08203125/rep1/train.log +1107 -0
  15. tiera-rerun-fixed-moonep/echo/a2/a2_matched_gbs16/G2__echo__matched__seq4096__gbs16__cap1p08203125/rep2/mem_samples.csv +184 -0
  16. tiera-rerun-fixed-moonep/echo/a2/a2_matched_gbs16/G2__echo__matched__seq4096__gbs16__cap1p08203125/rep2/run.env +70 -0
  17. tiera-rerun-fixed-moonep/echo/a2/a2_matched_gbs16/G2__echo__matched__seq4096__gbs16__cap1p08203125/rep2/train.log +1085 -0
  18. tiera-rerun-fixed-moonep/echo/a2/a2_matched_gbs16/results.tsv +5 -0
  19. tiera-rerun-fixed-moonep/echo/a2/manifest.txt +19 -0
  20. tiera-rerun-fixed-moonep/echo/a2/pip_freeze.txt +317 -0
  21. tiera-rerun-fixed-moonep/echo/a2/provenance.txt +90 -0
  22. tiera-rerun-fixed-moonep/hf_watcher.log +3 -0
  23. tiera-rerun-fixed-moonep/steps.tsv +1 -0
  24. tiera-rerun-fixed-moonep/study.log +48 -0
tiera-rerun-fixed-moonep/.completed/echo__ponly__gbs16 ADDED
@@ -0,0 +1 @@
 
 
1
+ 2026-08-30T06:44:11+00:00
tiera-rerun-fixed-moonep/STATUS.txt CHANGED
@@ -1,10 +1,11 @@
1
  study_root: /workspace/tiera-rerun-fixed-moonep
2
  gbs: 16
3
- updated: 2026-08-30T06:36:31+00:00
4
- current: echo__ponly__gbs16
5
 
6
  completed steps:
7
  echo__a1__gbs16
 
8
 
9
  failed steps:
10
  FAIL echo tierbprobe 16 2026-08-30T04:52:38+00:00 2026-08-30T05:06:38+00:00 /workspace/tiera-rerun-fixed-moonep/echo/tierbprobe
 
1
  study_root: /workspace/tiera-rerun-fixed-moonep
2
  gbs: 16
3
+ updated: 2026-08-30T06:44:13+00:00
4
+ current: echo__a2__gbs16
5
 
6
  completed steps:
7
  echo__a1__gbs16
8
+ echo__ponly__gbs16
9
 
10
  failed steps:
11
  FAIL echo tierbprobe 16 2026-08-30T04:52:38+00:00 2026-08-30T05:06:38+00:00 /workspace/tiera-rerun-fixed-moonep/echo/tierbprobe
tiera-rerun-fixed-moonep/echo/a2/a2_matched_gbs16/G1__echo__matched__seq4096__gbs16__cap2p40625/rep1/mem_samples.csv ADDED
@@ -0,0 +1,232 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ 0, 0
2
+ 1, 0
3
+ 2, 0
4
+ 3, 0
5
+ 4, 0
6
+ 5, 0
7
+ 6, 0
8
+ 7, 0
9
+ 0, 0
10
+ 1, 0
11
+ 2, 0
12
+ 3, 0
13
+ 4, 0
14
+ 5, 0
15
+ 6, 0
16
+ 7, 0
17
+ 0, 2
18
+ 1, 1
19
+ 2, 1
20
+ 3, 1
21
+ 4, 1
22
+ 5, 1
23
+ 6, 1
24
+ 7, 1
25
+ 0, 4
26
+ 1, 4
27
+ 2, 4
28
+ 3, 4
29
+ 4, 1
30
+ 5, 1
31
+ 6, 1
32
+ 7, 1
33
+ 0, 4
34
+ 1, 4
35
+ 2, 4
36
+ 3, 4
37
+ 4, 4
38
+ 5, 4
39
+ 6, 4
40
+ 7, 4
41
+ 0, 4
42
+ 1, 4
43
+ 2, 41
44
+ 3, 4
45
+ 4, 41
46
+ 5, 4
47
+ 6, 35
48
+ 7, 4
49
+ 0, 887
50
+ 1, 897
51
+ 2, 905
52
+ 3, 895
53
+ 4, 893
54
+ 5, 899
55
+ 6, 915
56
+ 7, 911
57
+ 0, 1895
58
+ 1, 1895
59
+ 2, 1895
60
+ 3, 1895
61
+ 4, 1895
62
+ 5, 1895
63
+ 6, 1895
64
+ 7, 1895
65
+ 0, 39467
66
+ 1, 39493
67
+ 2, 39463
68
+ 3, 39483
69
+ 4, 39529
70
+ 5, 39505
71
+ 6, 39503
72
+ 7, 39527
73
+ 0, 39907
74
+ 1, 39907
75
+ 2, 39883
76
+ 3, 39907
77
+ 4, 39907
78
+ 5, 39907
79
+ 6, 39907
80
+ 7, 39907
81
+ 0, 42357
82
+ 1, 42355
83
+ 2, 42355
84
+ 3, 42355
85
+ 4, 42355
86
+ 5, 42355
87
+ 6, 42355
88
+ 7, 42355
89
+ 0, 42583
90
+ 1, 42581
91
+ 2, 42581
92
+ 3, 42581
93
+ 4, 42581
94
+ 5, 42581
95
+ 6, 42581
96
+ 7, 42581
97
+ 0, 42769
98
+ 1, 42759
99
+ 2, 42745
100
+ 3, 42779
101
+ 4, 42763
102
+ 5, 42759
103
+ 6, 42755
104
+ 7, 42769
105
+ 0, 43493
106
+ 1, 43491
107
+ 2, 43491
108
+ 3, 43491
109
+ 4, 43491
110
+ 5, 43491
111
+ 6, 43491
112
+ 7, 43491
113
+ 0, 43493
114
+ 1, 43491
115
+ 2, 43491
116
+ 3, 43491
117
+ 4, 43491
118
+ 5, 43491
119
+ 6, 43491
120
+ 7, 43491
121
+ 0, 45265
122
+ 1, 45263
123
+ 2, 45263
124
+ 3, 45263
125
+ 4, 45263
126
+ 5, 45263
127
+ 6, 45263
128
+ 7, 45263
129
+ 0, 45265
130
+ 1, 45263
131
+ 2, 45263
132
+ 3, 45263
133
+ 4, 45263
134
+ 5, 45263
135
+ 6, 45263
136
+ 7, 45263
137
+ 0, 85827
138
+ 1, 85825
139
+ 2, 85825
140
+ 3, 85825
141
+ 4, 85825
142
+ 5, 85825
143
+ 6, 85825
144
+ 7, 85825
145
+ 0, 86905
146
+ 1, 86903
147
+ 2, 86903
148
+ 3, 86903
149
+ 4, 86903
150
+ 5, 86903
151
+ 6, 86903
152
+ 7, 86903
153
+ 0, 86905
154
+ 1, 86903
155
+ 2, 86903
156
+ 3, 86903
157
+ 4, 86903
158
+ 5, 86903
159
+ 6, 86903
160
+ 7, 86903
161
+ 0, 86905
162
+ 1, 86903
163
+ 2, 86903
164
+ 3, 86903
165
+ 4, 86903
166
+ 5, 86903
167
+ 6, 86903
168
+ 7, 86903
169
+ 0, 104407
170
+ 1, 104349
171
+ 2, 104349
172
+ 3, 104349
173
+ 4, 104405
174
+ 5, 104349
175
+ 6, 104349
176
+ 7, 104349
177
+ 0, 105875
178
+ 1, 105817
179
+ 2, 105817
180
+ 3, 105817
181
+ 4, 105873
182
+ 5, 105817
183
+ 6, 105817
184
+ 7, 105817
185
+ 0, 106523
186
+ 1, 106465
187
+ 2, 106465
188
+ 3, 106465
189
+ 4, 106521
190
+ 5, 106465
191
+ 6, 106465
192
+ 7, 106465
193
+ 0, 107183
194
+ 1, 107131
195
+ 2, 107075
196
+ 3, 107119
197
+ 4, 107157
198
+ 5, 107169
199
+ 6, 107147
200
+ 7, 107183
201
+ 0, 138613
202
+ 1, 139203
203
+ 2, 137959
204
+ 3, 136499
205
+ 4, 136849
206
+ 5, 136653
207
+ 6, 159453
208
+ 7, 146361
209
+ 0, 161641
210
+ 1, 161525
211
+ 2, 161525
212
+ 3, 161525
213
+ 4, 161723
214
+ 5, 161525
215
+ 6, 161525
216
+ 7, 161525
217
+ 0, 161641
218
+ 1, 161525
219
+ 2, 161525
220
+ 3, 161525
221
+ 4, 161723
222
+ 5, 161525
223
+ 6, 161525
224
+ 7, 161525
225
+ 0, 3695
226
+ 1, 2098
227
+ 2, 2098
228
+ 3, 2098
229
+ 4, 2098
230
+ 5, 2098
231
+ 6, 2418
232
+ 7, 161525
tiera-rerun-fixed-moonep/echo/a2/a2_matched_gbs16/G1__echo__matched__seq4096__gbs16__cap2p40625/rep1/run.env ADDED
@@ -0,0 +1,70 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ backend=echo
2
+ config=G1
3
+ family=tierA
4
+ repo=/workspace/Megatron-MoE-Echo
5
+ branch=bracing-regions-through-kernels
6
+ commit=9a236793
7
+ num_layers=16
8
+ hidden=4096
9
+ ffn_hidden=14336
10
+ moe_ffn_hidden=14336
11
+ num_experts=8
12
+ topk=2
13
+ experts_per_rank=1
14
+ expert_params=176160768
15
+ heads=32
16
+ kv_groups=8
17
+ kv_channels=
18
+ qk_layernorm=0
19
+ shared_ffn=0
20
+ shared_expert_overlap=0
21
+ moe_token_dispatcher=flex
22
+ mla=0
23
+ dense_first_layers=0
24
+ moe_layers=16
25
+ rotary_base=1000000
26
+ rotary_scaling=
27
+ qkv_bias=0
28
+ router_balancing=aux_loss
29
+ aux_loss_coeff=1e-2
30
+ router_pre_softmax=0
31
+ router_fusion=0
32
+ router_dtype=fp32
33
+ device_initiated_grouped_gemm=1
34
+ norm_epsilon=1e-5
35
+ seq_length=4096
36
+ micro_batch_size=1
37
+ global_batch_size=16
38
+ num_microbatches=2
39
+ assignments_per_rank=8192
40
+ mean_tokens_per_expert=8192
41
+ slot_policy=matched
42
+ slots_per_rank=1
43
+ capacity_mode=static
44
+ capacity=2.40625
45
+ cuda_graph=1
46
+ probe_mode=0
47
+ async_grad_reduce=0
48
+ branch_role=timing
49
+ memory_dump_mode=none
50
+ memory_snapshot_path=
51
+ cuda_graph_warmup_steps=3
52
+ recompute=none
53
+ force_balance=0
54
+ train_iters=12
55
+ ep_size=8
56
+ echo_dispatch_overlap=1
57
+ echo_dispatch_recompute=0
58
+ ultraep_quota_min_tokens=
59
+ ultraep_weight_sync_join=
60
+ profile_enable=0
61
+ kernel_trace=0
62
+ fp8_mode=mxfp8
63
+ fp8_recipe=mxfp8
64
+ training_seed=1234
65
+ learning_rate=1e-4
66
+ min_learning_rate=1.0e-5
67
+ lr_warmup_iters=5
68
+ state_dump_enable=0
69
+ state_dump_dir=
70
+ moonep_assert_no_pending_reduce=0
tiera-rerun-fixed-moonep/echo/a2/a2_matched_gbs16/G1__echo__matched__seq4096__gbs16__cap2p40625/rep1/train.log ADDED
@@ -0,0 +1,1023 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ==============================================================
2
+ G1 / echo (slots/rank=1 policy=matched)
3
+ E=8 K=2 H'=14336 m=1
4
+ L=16 seq=4096 gbs=16 (2 microbatches)
5
+ cap_mode=static cap=2.40625 cg=1 recompute=none
6
+ memory_dump=none snapshots=/workspace/tiera-rerun-fixed-moonep/echo/a2/a2_matched_gbs16/G1__echo__matched__seq4096__gbs16__cap2p40625/rep1/memory_snapshots/snapshot.pickle
7
+ repo=/workspace/Megatron-MoE-Echo @ 9a236793
8
+ ==============================================================
9
+ W0830 06:44:30.036000 651947 torch/distributed/run.py:766]
10
+ W0830 06:44:30.036000 651947 torch/distributed/run.py:766] *****************************************
11
+ W0830 06:44:30.036000 651947 torch/distributed/run.py:766] Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed.
12
+ W0830 06:44:30.036000 651947 torch/distributed/run.py:766] *****************************************
13
+ /workspace/Megatron-MoE-Echo/megatron/training/arguments.py:831: UserWarning: Disabling sequence parallelism because tensor model parallelism is disabled
14
+ warnings.warn("Disabling sequence parallelism because tensor model parallelism is disabled")
15
+ /workspace/Megatron-MoE-Echo/megatron/training/arguments.py:831: UserWarning: Disabling sequence parallelism because tensor model parallelism is disabled
16
+ warnings.warn("Disabling sequence parallelism because tensor model parallelism is disabled")
17
+ /workspace/Megatron-MoE-Echo/megatron/training/arguments.py:831: UserWarning: Disabling sequence parallelism because tensor model parallelism is disabled
18
+ warnings.warn("Disabling sequence parallelism because tensor model parallelism is disabled")
19
+ using world size: 8, data-parallel size: 8, context-parallel size: 1, hierarchical context-parallel sizes: None, tensor-model-parallel size: 1, pipeline-model-parallel size: 1
20
+ WARNING: overriding default arguments for tokenizer_type:GPT2BPETokenizer with tokenizer_type:GPT2BPETokenizer
21
+ Number of virtual stages per pipeline stage: None
22
+ accumulate and all-reduce gradients in fp32 for bfloat16 data type.
23
+ using torch.bfloat16 for parameters ...
24
+ /workspace/Megatron-MoE-Echo/megatron/training/arguments.py:831: UserWarning: Disabling sequence parallelism because tensor model parallelism is disabled
25
+ warnings.warn("Disabling sequence parallelism because tensor model parallelism is disabled")
26
+ ------------------------ arguments ------------------------
27
+ account_for_embedding_in_pipeline_split ......... False
28
+ account_for_loss_in_pipeline_split .............. False
29
+ accumulate_allreduce_grads_in_fp32 .............. True
30
+ adam_beta1 ...................................... 0.9
31
+ adam_beta2 ...................................... 0.999
32
+ adam_eps ........................................ 1e-08
33
+ add_bias_linear ................................. False
34
+ add_position_embedding .......................... False
35
+ add_qkv_bias .................................... False
36
+ adlr_autoresume ................................. False
37
+ adlr_autoresume_interval ........................ 1000
38
+ align_grad_reduce ............................... True
39
+ align_param_gather .............................. False
40
+ app_tag_run_name ................................ None
41
+ app_tag_run_version ............................. 0.0.0
42
+ apply_layernorm_1p .............................. False
43
+ apply_query_key_layer_scaling ................... False
44
+ apply_residual_connection_post_layernorm ........ False
45
+ apply_rope_fusion ............................... True
46
+ async_save ...................................... None
47
+ async_tensor_model_parallel_allreduce ........... True
48
+ attention_backend ............................... AttnBackend.auto
49
+ attention_dropout ............................... 0.0
50
+ attention_softmax_in_fp32 ....................... False
51
+ auto_detect_ckpt_format ......................... False
52
+ barrier_with_L1_time ............................ True
53
+ bert_binary_head ................................ True
54
+ bert_embedder_type .............................. megatron
55
+ bert_load ....................................... None
56
+ bf16 ............................................ True
57
+ bias_dropout_fusion ............................. True
58
+ bias_gelu_fusion ................................ False
59
+ bias_swiglu_fusion .............................. True
60
+ biencoder_projection_dim ........................ 0
61
+ biencoder_shared_query_context_model ............ False
62
+ block_data_path ................................. None
63
+ calc_ft_timeouts ................................ False
64
+ calculate_per_token_loss ........................ False
65
+ check_for_large_grads ........................... False
66
+ check_for_nan_in_loss_and_grad .................. False
67
+ check_for_spiky_loss ............................ False
68
+ check_weight_hash_across_dp_replicas_interval ... None
69
+ ckpt_assume_constant_structure .................. False
70
+ ckpt_convert_format ............................. None
71
+ ckpt_convert_save ............................... None
72
+ ckpt_convert_update_legacy_dist_opt_format ...... False
73
+ ckpt_format ..................................... torch_dist
74
+ ckpt_fully_parallel_load ........................ False
75
+ ckpt_fully_parallel_save ........................ True
76
+ ckpt_fully_parallel_save_deprecated ............. False
77
+ ckpt_step ....................................... None
78
+ classes_fraction ................................ 1.0
79
+ clip_grad ....................................... 1.0
80
+ clone_scatter_output_in_embedding ............... True
81
+ config_logger_dir ...............................
82
+ consumed_train_samples .......................... 0
83
+ consumed_valid_samples .......................... 0
84
+ context_parallel_size ........................... 1
85
+ cp_comm_type .................................... ['p2p']
86
+ create_attention_mask_in_dataloader ............. True
87
+ cross_entropy_fusion_impl ....................... native
88
+ cross_entropy_loss_fusion ....................... False
89
+ cuda_graph_scope ................................ full_iteration
90
+ cuda_graph_warmup_steps ......................... 3
91
+ data_args_path .................................. None
92
+ data_cache_path ................................. None
93
+ data_parallel_random_init ....................... False
94
+ data_parallel_sharding_strategy ................. no_shard
95
+ data_parallel_size .............................. 8
96
+ data_path ....................................... ['/workspace/data/wikitext/wikitext103_gpt2_text_document']
97
+ data_per_class_fraction ......................... 1.0
98
+ data_sharding ................................... True
99
+ dataloader_type ................................. single
100
+ ddp_average_in_collective ....................... False
101
+ ddp_bucket_size ................................. None
102
+ ddp_num_buckets ................................. None
103
+ ddp_pad_buckets_for_high_nccl_busbw ............. False
104
+ decoder_first_pipeline_num_layers ............... None
105
+ decoder_last_pipeline_num_layers ................ None
106
+ decoder_num_layers .............................. None
107
+ decoder_seq_length .............................. None
108
+ decoupled_lr .................................... None
109
+ decoupled_min_lr ................................ None
110
+ decrease_batch_size_if_needed ................... False
111
+ defer_embedding_wgrad_compute ................... False
112
+ delay_wgrad_compute ............................. False
113
+ deprecated_use_mcore_models ..................... True
114
+ deterministic_mode .............................. False
115
+ dino_bottleneck_size ............................ 256
116
+ dino_freeze_last_layer .......................... 1
117
+ dino_head_hidden_size ........................... 2048
118
+ dino_local_crops_number ......................... 10
119
+ dino_local_img_size ............................. 96
120
+ dino_norm_last_layer ............................ False
121
+ dino_teacher_temp ............................... 0.07
122
+ dino_warmup_teacher_temp ........................ 0.04
123
+ dino_warmup_teacher_temp_epochs ................. 30
124
+ disable_bf16_reduced_precision_matmul ........... False
125
+ disable_mamba_mem_eff_path ...................... False
126
+ disable_straggler_on_startup .................... False
127
+ dist_ckpt_format_deprecated ..................... None
128
+ dist_ckpt_strictness ............................ assume_ok_unexpected
129
+ distribute_saved_activations .................... False
130
+ distributed_backend ............................. nccl
131
+ distributed_timeout_minutes ..................... 10
132
+ embedding_init_method_std ....................... None
133
+ embedding_path .................................. None
134
+ empty_unused_memory_level ....................... 0
135
+ enable_cuda_graph ............................... True
136
+ enable_experimental ............................. False
137
+ enable_ft_package ............................... False
138
+ enable_gloo_process_groups ...................... True
139
+ enable_msc ...................................... True
140
+ enable_one_logger ............................... True
141
+ encoder_num_layers .............................. 16
142
+ encoder_seq_length .............................. 4096
143
+ end_weight_decay ................................ 0.1
144
+ eod_mask_loss ................................... False
145
+ error_injection_rate ............................ 0
146
+ error_injection_type ............................ transient_error
147
+ eval_interval ................................... 1000000
148
+ eval_iters ...................................... 0
149
+ evidence_data_path .............................. None
150
+ exit_duration_in_mins ........................... None
151
+ exit_interval ................................... None
152
+ exit_on_missing_checkpoint ...................... False
153
+ exit_signal_handler ............................. False
154
+ exp_avg_dtype ................................... torch.float32
155
+ exp_avg_sq_dtype ................................ torch.float32
156
+ expert_model_parallel_size ...................... 8
157
+ expert_tensor_parallel_size ..................... 1
158
+ export_force_local_attention .................... False
159
+ export_kd_cfg ................................... None
160
+ export_kd_teacher_ckpt_format ................... None
161
+ export_kd_teacher_load .......................... None
162
+ export_kv_cache_quant ........................... False
163
+ export_legacy_megatron .......................... False
164
+ export_model_type ............................... GPTModel
165
+ export_moe_apply_probs_on_input ................. False
166
+ export_qk_l2_norm ............................... False
167
+ export_quant_cfg ................................ None
168
+ export_real_quant_cfg ........................... None
169
+ export_te_mcore_model ........................... False
170
+ external_cuda_graph ............................. False
171
+ ffn_hidden_size ................................. 14336
172
+ finetune ........................................ False
173
+ finetune_data_split ............................. train
174
+ finetune_hf_dataset ............................. None
175
+ first_last_layers_bf16 .......................... False
176
+ flash_decode .................................... False
177
+ fp16 ............................................ False
178
+ fp16_lm_cross_entropy ........................... False
179
+ fp32_residual_connection ........................ False
180
+ fp8 ............................................. e4m3
181
+ fp8_amax_compute_algo ........................... most_recent
182
+ fp8_amax_history_len ............................ 1
183
+ fp8_interval .................................... 1
184
+ fp8_margin ...................................... 0
185
+ fp8_param_gather ................................ False
186
+ fp8_recipe ...................................... mxfp8
187
+ fp8_wgrad ....................................... True
188
+ fsdp_double_buffer .............................. False
189
+ global_batch_size ............................... 16
190
+ grad_reduce_in_bf16 ............................. False
191
+ gradient_accumulation_fusion .................... True
192
+ gradient_reduce_div_fusion ...................... True
193
+ group_query_attention ........................... True
194
+ head_lr_mult .................................... 1.0
195
+ heterogeneous_layers_config_encoded_json ........ None
196
+ heterogeneous_layers_config_path ................ None
197
+ hidden_dropout .................................. 0.0
198
+ hidden_size ..................................... 4096
199
+ hierarchical_context_parallel_sizes ............. None
200
+ high_priority_stream_groups ..................... []
201
+ hybrid_attention_ratio .......................... 0.0
202
+ hybrid_mlp_ratio ................................ 0.0
203
+ hybrid_override_pattern ......................... None
204
+ hysteresis ...................................... 2
205
+ ict_head_size ................................... None
206
+ ict_load ........................................ None
207
+ img_h ........................................... 224
208
+ img_w ........................................... 224
209
+ indexer_batch_size .............................. 128
210
+ indexer_log_interval ............................ 1000
211
+ inference_batch_times_seqlen_threshold .......... -1
212
+ inference_dynamic_batching ...................... False
213
+ inference_dynamic_batching_buffer_guaranteed_fraction 0.2
214
+ inference_dynamic_batching_buffer_overflow_factor None
215
+ inference_dynamic_batching_buffer_size_gb ....... 40.0
216
+ inference_dynamic_batching_chunk_size ........... 256
217
+ inference_dynamic_batching_max_requests_override None
218
+ inference_dynamic_batching_max_tokens_override .. None
219
+ inference_dynamic_batching_num_cuda_graphs ...... 16
220
+ inference_max_batch_size ........................ 8
221
+ inference_max_seq_length ........................ 2560
222
+ inference_rng_tracker ........................... False
223
+ init_method_std ................................. 0.01
224
+ init_method_xavier_uniform ...................... False
225
+ init_model_with_meta_device ..................... False
226
+ initial_loss_scale .............................. 4294967296
227
+ inprocess_active_world_size ..................... 8
228
+ inprocess_barrier_timeout ....................... 120
229
+ inprocess_completion_timeout .................... 120
230
+ inprocess_empty_cuda_cache ...................... False
231
+ inprocess_granularity ........................... node
232
+ inprocess_hard_timeout .......................... 90
233
+ inprocess_heartbeat_interval .................... 30
234
+ inprocess_heartbeat_timeout ..................... 60
235
+ inprocess_last_call_wait ........................ 1
236
+ inprocess_max_iterations ........................ None
237
+ inprocess_monitor_process_interval .............. 1.0
238
+ inprocess_monitor_thread_interval ............... 1.0
239
+ inprocess_progress_watchdog_interval ............ 1.0
240
+ inprocess_restart ............................... False
241
+ inprocess_soft_timeout .......................... 60
242
+ inprocess_termination_grace_time ................ 1
243
+ is_hybrid_model ................................. False
244
+ iter_per_epoch .................................. 1250
245
+ iterations_to_skip .............................. []
246
+ keep_fp8_transpose_cache_when_using_custom_fsdp . False
247
+ kitchen_config_file ............................. None
248
+ kitchen_recipe_number ........................... None
249
+ kv_channels ..................................... 128
250
+ kv_lora_rank .................................... 32
251
+ lazy_mpu_init ................................... None
252
+ load ............................................ None
253
+ load_main_params_from_ckpt ...................... None
254
+ load_model_opt_format ........................... False
255
+ local_rank ...................................... 0
256
+ log_energy ...................................... False
257
+ log_interval .................................... 1
258
+ log_loss_scale_to_tensorboard ................... True
259
+ log_memory_to_tensorboard ....................... False
260
+ log_num_zeros_in_grad ........................... False
261
+ log_params_norm ................................. False
262
+ log_progress .................................... False
263
+ log_straggler ................................... False
264
+ log_throughput .................................. True
265
+ log_timers_to_tensorboard ....................... False
266
+ log_validation_ppl_to_tensorboard ............... False
267
+ log_world_size_to_tensorboard ................... False
268
+ logging_level ................................... None
269
+ loss_scale ...................................... None
270
+ loss_scale_window ............................... 1000
271
+ lr .............................................. 0.0001
272
+ lr_decay_iters .................................. 8
273
+ lr_decay_samples ................................ None
274
+ lr_decay_style .................................. cosine
275
+ lr_warmup_fraction .............................. None
276
+ lr_warmup_init .................................. 0.0
277
+ lr_warmup_iters ................................. 5
278
+ lr_warmup_samples ............................... 0
279
+ lr_wsd_decay_iters .............................. None
280
+ lr_wsd_decay_samples ............................ None
281
+ lr_wsd_decay_style .............................. exponential
282
+ main_grads_dtype ................................ torch.float32
283
+ main_params_dtype ............................... torch.float32
284
+ make_vocab_size_divisible_by .................... 128
285
+ mamba_head_dim .................................. 64
286
+ mamba_num_groups ................................ 8
287
+ mamba_num_heads ................................. None
288
+ mamba_state_dim ................................. 128
289
+ manual_gc ....................................... False
290
+ manual_gc_eval .................................. True
291
+ manual_gc_interval .............................. 0
292
+ mask_factor ..................................... 1.0
293
+ mask_prob ....................................... 0.15
294
+ mask_type ....................................... random
295
+ masked_softmax_fusion ........................... False
296
+ max_position_embeddings ......................... 32768
297
+ max_tokens_to_oom ............................... 12000
298
+ memory_snapshot_path ............................ snapshot.pickle
299
+ merge_file ...................................... /workspace/data/wikitext/gpt2-merges.txt
300
+ micro_batch_size ................................ 1
301
+ microbatch_group_size_per_vp_stage .............. None
302
+ mid_level_dataset_surplus ....................... 0.005
303
+ min_loss_scale .................................. 1.0
304
+ min_lr .......................................... 1e-05
305
+ mlp_chunks_for_prefill .......................... 1
306
+ mmap_bin_files .................................. True
307
+ mock_data ....................................... False
308
+ moe_apply_probs_on_input ........................ False
309
+ moe_aux_loss_coeff .............................. 0.01
310
+ moe_deepep_num_sms .............................. 20
311
+ moe_echo_enable_random_offloading ............... False
312
+ moe_echo_expert_dispatch_overlap ................ True
313
+ moe_echo_expert_dispatcher_type ................. hybridep
314
+ moe_echo_recompute_expert_dispatch .............. False
315
+ moe_enable_deepep ............................... False
316
+ moe_enable_echo ................................. True
317
+ moe_enable_hybridep ............................. True
318
+ moe_expert_capacity_factor ...................... None
319
+ moe_extended_tp ................................. False
320
+ moe_ffn_hidden_size ............................. 14336
321
+ moe_grouped_gemm ................................ True
322
+ moe_hybridep_num_sms ............................ 16
323
+ moe_input_jitter_eps ............................ None
324
+ moe_layer_freq .................................. 1
325
+ moe_layer_recompute ............................. False
326
+ moe_num_echo_experts ............................ 8
327
+ moe_pad_expert_input_to_capacity ................ False
328
+ moe_per_layer_logging ........................... False
329
+ moe_permute_fusion .............................. False
330
+ moe_received_token_capacity ..................... 2.40625
331
+ moe_router_bias_update_rate ..................... 0.001
332
+ moe_router_dtype ................................ fp32
333
+ moe_router_enable_expert_bias ................... False
334
+ moe_router_force_load_balancing ................. False
335
+ moe_router_group_topk ........................... None
336
+ moe_router_load_balancing_type .................. aux_loss
337
+ moe_router_num_groups ........................... None
338
+ moe_router_padding_for_fp8 ...................... False
339
+ moe_router_pre_softmax .......................... False
340
+ moe_router_score_function ....................... softmax
341
+ moe_router_topk ................................. 2
342
+ moe_router_topk_scaling_factor .................. None
343
+ moe_shared_expert_intermediate_size ............. None
344
+ moe_shared_expert_overlap ....................... False
345
+ moe_token_dispatcher_type ....................... flex
346
+ moe_token_drop_policy ........................... probs
347
+ moe_upcycling_granularity ....................... 1
348
+ moe_use_device_initiated_grouped_gemm ........... True
349
+ moe_use_legacy_grouped_gemm ..................... False
350
+ moe_use_upcycling ............................... False
351
+ moe_z_loss_coeff ................................ None
352
+ mrope_section ................................... None
353
+ mscale .......................................... 1.0
354
+ mscale_all_dim .................................. 0.0
355
+ mtp_loss_scaling_factor ......................... 0.1
356
+ mtp_num_layers .................................. None
357
+ multi_latent_attention .......................... False
358
+ nccl_all_reduce_for_prefill ..................... False
359
+ nccl_communicator_config_path ................... None
360
+ nccl_ub ......................................... False
361
+ no_load_optim ................................... None
362
+ no_load_rng ..................................... None
363
+ no_persist_layer_norm ........................... False
364
+ no_rope_freq .................................... None
365
+ no_save_optim ................................... None
366
+ no_save_rng ..................................... None
367
+ non_persistent_ckpt_type ........................ None
368
+ non_persistent_global_ckpt_dir .................. None
369
+ non_persistent_local_ckpt_algo .................. fully_parallel
370
+ non_persistent_local_ckpt_dir ................... None
371
+ non_persistent_save_interval .................... None
372
+ norm_epsilon .................................... 1e-05
373
+ normalization ................................... RMSNorm
374
+ num_attention_heads ............................. 32
375
+ num_channels .................................... 3
376
+ num_classes ..................................... 1000
377
+ num_dataset_builder_threads ..................... 1
378
+ num_distributed_optimizer_instances ............. 1
379
+ num_experts ..................................... 8
380
+ num_layers ...................................... 16
381
+ num_layers_at_end_in_bf16 ....................... 1
382
+ num_layers_at_start_in_bf16 ..................... 1
383
+ num_layers_per_virtual_pipeline_stage ........... None
384
+ num_query_groups ................................ 8
385
+ num_virtual_stages_per_pipeline_rank ............ None
386
+ num_workers ..................................... 2
387
+ object_storage_cache_path ....................... None
388
+ one_logger_async ................................ False
389
+ one_logger_project .............................. megatron-lm
390
+ one_logger_run_name ............................. None
391
+ onnx_safe ....................................... None
392
+ openai_gelu ..................................... False
393
+ optimizer ....................................... adam
394
+ optimizer_cpu_offload ........................... False
395
+ optimizer_offload_fraction ...................... 1.0
396
+ output_bert_embeddings .......................... False
397
+ overlap_cpu_optimizer_d2h_h2d ................... False
398
+ overlap_grad_reduce ............................. True
399
+ overlap_moe_expert_parallel_comm ................ False
400
+ overlap_p2p_comm ................................ False
401
+ overlap_p2p_comm_warmup_flush ................... False
402
+ overlap_param_gather ............................ True
403
+ overlap_param_gather_with_optimizer_step ........ False
404
+ override_opt_param_scheduler .................... False
405
+ params_dtype .................................... torch.bfloat16
406
+ patch_dim ....................................... 16
407
+ per_split_data_args_path ........................ None
408
+ perform_initialization .......................... True
409
+ pin_cpu_grads ................................... True
410
+ pin_cpu_params .................................. True
411
+ pipeline_model_parallel_comm_backend ............ None
412
+ pipeline_model_parallel_layout .................. None
413
+ pipeline_model_parallel_size .................... 1
414
+ position_embedding_type ......................... rope
415
+ pretrained_checkpoint ........................... None
416
+ profile ......................................... False
417
+ profile_ranks ................................... [0]
418
+ profile_step_end ................................ 12
419
+ profile_step_start .............................. 10
420
+ q_lora_rank ..................................... None
421
+ qk_head_dim ..................................... 128
422
+ qk_l2_norm ...................................... False
423
+ qk_layernorm .................................... False
424
+ qk_pos_emb_head_dim ............................. 64
425
+ query_in_block_prob ............................. 0.1
426
+ rampup_batch_size ............................... None
427
+ rank ............................................ 0
428
+ recompute_granularity ........................... None
429
+ recompute_method ................................ None
430
+ recompute_modules ............................... None
431
+ recompute_num_layers ............................ None
432
+ record_memory_history ........................... False
433
+ relative_attention_max_distance ................. 128
434
+ relative_attention_num_buckets .................. 32
435
+ replication ..................................... False
436
+ replication_factor .............................. 2
437
+ replication_jump ................................ None
438
+ rerun_mode ...................................... validate_results
439
+ reset_attention_mask ............................ False
440
+ reset_position_ids .............................. False
441
+ result_rejected_tracker_filename ................ None
442
+ retriever_report_topk_accuracies ................ []
443
+ retriever_score_scaling ......................... False
444
+ retriever_seq_length ............................ 256
445
+ retro_add_retriever ............................. False
446
+ retro_attention_gate ............................ 1
447
+ retro_cyclic_train_iters ........................ None
448
+ retro_encoder_attention_dropout ................. 0.1
449
+ retro_encoder_hidden_dropout .................... 0.1
450
+ retro_encoder_layers ............................ 2
451
+ retro_num_neighbors ............................. 2
452
+ retro_num_retrieved_chunks ...................... 2
453
+ retro_project_dir ............................... None
454
+ retro_verify_neighbor_count ..................... True
455
+ reuse_grad_buf_for_mxfp8_param_ag ............... False
456
+ rope_scaling_factor ............................. 8.0
457
+ rope_type ....................................... None
458
+ rotary_base ..................................... 1000000
459
+ rotary_interleaved .............................. False
460
+ rotary_percent .................................. 1.0
461
+ rotary_scaling_factor ........................... 1.0
462
+ rotary_seq_len_interpolation_factor ............. None
463
+ run_workload_inspector_server ................... False
464
+ sample_rate ..................................... 1.0
465
+ save ............................................ None
466
+ save_interval ................................... None
467
+ save_retain_interval ............................ None
468
+ scatter_gather_tensors_in_pipeline .............. True
469
+ seed ............................................ 1234
470
+ seq_length ...................................... 4096
471
+ sequence_parallel ............................... False
472
+ sft ............................................. False
473
+ sft_tokenizer_prompt_format ..................... nemotron-h-aligned
474
+ sgd_momentum .................................... 0.9
475
+ short_seq_prob .................................. 0.1
476
+ skip_train ...................................... False
477
+ skipped_train_samples ........................... 0
478
+ spec ............................................ None
479
+ split ........................................... 100,0,0
480
+ squared_relu .................................... False
481
+ start_weight_decay .............................. 0.1
482
+ straggler_ctrlr_port ............................ 65535
483
+ straggler_minmax_count .......................... 1
484
+ suggested_communication_unit_size ............... None
485
+ swiglu .......................................... True
486
+ swin_backbone_type .............................. tiny
487
+ symmetric_ar_type ............................... None
488
+ te_rng_tracker .................................. True
489
+ tensor_model_parallel_size ...................... 1
490
+ tensorboard_dir ................................. None
491
+ tensorboard_log_interval ........................ 1
492
+ tensorboard_queue_size .......................... 1000
493
+ test_data_path .................................. None
494
+ test_mode ....................................... False
495
+ tiktoken_num_special_tokens ..................... 1000
496
+ tiktoken_pattern ................................ None
497
+ tiktoken_special_tokens ......................... None
498
+ timing_log_level ................................ 0
499
+ timing_log_option ............................... minmax
500
+ titles_data_path ................................ None
501
+ tokenizer_model ................................. None
502
+ tokenizer_type .................................. GPT2BPETokenizer
503
+ torch_fsdp2_reshard_after_forward ............... True
504
+ tp_comm_bootstrap_backend ....................... nccl
505
+ tp_comm_bulk_dgrad .............................. True
506
+ tp_comm_bulk_wgrad .............................. True
507
+ tp_comm_overlap ................................. False
508
+ tp_comm_overlap_ag .............................. True
509
+ tp_comm_overlap_cfg ............................. None
510
+ tp_comm_overlap_rs .............................. True
511
+ tp_comm_overlap_rs_dgrad ........................ False
512
+ tp_comm_split_ag ................................ True
513
+ tp_comm_split_rs ................................ True
514
+ train_data_path ................................. None
515
+ train_iters ..................................... 12
516
+ train_samples ................................... None
517
+ train_sync_interval ............................. None
518
+ transformer_impl ................................ transformer_engine
519
+ transformer_pipeline_model_parallel_size ........ 1
520
+ untie_embeddings_and_output_weights ............. True
521
+ use_checkpoint_args ............................. False
522
+ use_checkpoint_opt_param_scheduler .............. False
523
+ use_cpu_initialization .......................... None
524
+ use_custom_fsdp ................................. False
525
+ use_dist_ckpt ................................... True
526
+ use_dist_ckpt_deprecated ........................ False
527
+ use_distributed_optimizer ....................... True
528
+ use_flash_attn .................................. False
529
+ use_legacy_models ............................... False
530
+ use_mp_args_from_checkpoint_args ................ False
531
+ use_one_sent_docs ............................... False
532
+ use_persistent_ckpt_worker ...................... False
533
+ use_precision_aware_optimizer ................... False
534
+ use_pytorch_profiler ............................ False
535
+ use_ring_exchange_p2p ........................... False
536
+ use_rope_scaling ................................ False
537
+ use_rotary_position_embeddings .................. False
538
+ use_sharp ....................................... False
539
+ use_tokenizer_model_from_checkpoint_args ........ True
540
+ use_torch_fsdp2 ................................. False
541
+ use_torch_optimizer_for_cpu_offload ............. False
542
+ use_tp_pp_dp_mapping ............................ False
543
+ v_head_dim ...................................... 128
544
+ valid_data_path ................................. None
545
+ variable_seq_lengths ............................ False
546
+ virtual_pipeline_model_parallel_size ............ None
547
+ vision_backbone_type ............................ vit
548
+ vision_pretraining .............................. False
549
+ vision_pretraining_type ......................... classify
550
+ vocab_extra_ids ................................. 0
551
+ vocab_file ...................................... /workspace/data/wikitext/gpt2-vocab.json
552
+ vocab_size ...................................... None
553
+ wandb_exp_name ..................................
554
+ wandb_project ...................................
555
+ wandb_save_dir ..................................
556
+ weight_decay .................................... 0.1
557
+ weight_decay_incr_style ......................... constant
558
+ wgrad_deferral_limit ............................ 0
559
+ world_size ...................................... 8
560
+ yaml_cfg ........................................ None
561
+ -------------------- end of arguments ---------------------
562
+ INFO:megatron.core.num_microbatches_calculator:setting number of microbatches to constant 2
563
+ > building GPT2BPETokenizer tokenizer ...
564
+ /workspace/Megatron-MoE-Echo/megatron/training/arguments.py:831: UserWarning: Disabling sequence parallelism because tensor model parallelism is disabled
565
+ warnings.warn("Disabling sequence parallelism because tensor model parallelism is disabled")
566
+ /workspace/Megatron-MoE-Echo/megatron/training/arguments.py:831: UserWarning: Disabling sequence parallelism because tensor model parallelism is disabled
567
+ warnings.warn("Disabling sequence parallelism because tensor model parallelism is disabled")
568
+ /workspace/Megatron-MoE-Echo/megatron/training/arguments.py:831: UserWarning: Disabling sequence parallelism because tensor model parallelism is disabled
569
+ warnings.warn("Disabling sequence parallelism because tensor model parallelism is disabled")
570
+ > padded vocab (size: 50257) with 47 dummy tokens (new size: 50304)
571
+ WARNING:megatron.core.rerun_state_machine:RerunStateMachine initialized in mode RerunMode.VALIDATE_RESULTS
572
+ > initializing torch distributed ...
573
+ /workspace/Megatron-MoE-Echo/megatron/training/arguments.py:831: UserWarning: Disabling sequence parallelism because tensor model parallelism is disabled
574
+ warnings.warn("Disabling sequence parallelism because tensor model parallelism is disabled")
575
+ WARNING: one_logger package is required to enable e2e metrics tracking. please go to https://confluence.nvidia.com/display/MLWFO/Package+Repositories for details to install it
576
+ [Gloo] Rank 0 is connected to 7 peer ranks. Expected number of connected peer ranks is : 7
577
+ [Gloo] Rank [Gloo] Rank 1[Gloo] Rank [Gloo] Rank is connected to 2473 is connected to is connected to peer ranks. is connected to 77Expected number of connected peer ranks is : 7 peer ranks. peer ranks. [Gloo] Rank 7 peer ranks.
578
+ Expected number of connected peer ranks is : Expected number of connected peer ranks is : 5Expected number of connected peer ranks is : 77 is connected to 7
579
+
580
+
581
+ 7[Gloo] Rank [Gloo] Rank peer ranks. 7Expected number of connected peer ranks is : is connected to 776
582
+ peer ranks. is connected to Expected number of connected peer ranks is : 77 peer ranks.
583
+ Expected number of connected peer ranks is : 7
584
+ [Gloo] Rank 0 is connected to 7 peer ranks. Expected number of connected peer ranks is : 7
585
+ [Gloo] Rank 1 is connected to 7 peer ranks. Expected number of connected peer ranks is : 7
586
+ [Gloo] Rank 2 is connected to 7 peer ranks. Expected number of connected peer ranks is : 7
587
+ [Gloo] Rank 3 is connected to 7 peer ranks. Expected number of connected peer ranks is : 7
588
+ [Gloo] Rank 4 is connected to 7 peer ranks. Expected number of connected peer ranks is : 7
589
+ [Gloo] Rank 5 is connected to 7 peer ranks. Expected number of connected peer ranks is : 7
590
+ [Gloo] Rank [Gloo] Rank 7 is connected to 67 is connected to peer ranks. 7Expected number of connected peer ranks is : peer ranks. 7Expected number of connected peer ranks is : 7
591
+
592
+ [Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
593
+ > initialized tensor model parallel with size 1
594
+ > initialized pipeline model parallel with size 1
595
+ > setting random seeds to 1234 ...
596
+ [Gloo] Rank [Gloo] Rank 00 is connected to is connected to 00 peer ranks. peer ranks. Expected number of connected peer ranks is : Expected number of connected peer ranks is : 00
597
+
598
+ [Gloo] Rank [Gloo] Rank 00 is connected to is connected to 00 peer ranks. peer ranks. Expected number of connected peer ranks is : Expected number of connected peer ranks is : 00
599
+
600
+ [Gloo] Rank [Gloo] Rank 00 is connected to is connected to 00 peer ranks. peer ranks. Expected number of connected peer ranks is : Expected number of connected peer ranks is : 00
601
+
602
+ [Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
603
+ [rank3]:[W830 06:44:47.018891615 ProcessGroupNCCL.cpp:4789] [PG ID 0 PG GUID 0 Rank 3] using GPU 3 as device used by this process is currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect. You can specify device_id in init_process_group() to force use of a particular device.
604
+ [rank1]:[W830 06:44:47.018917720 ProcessGroupNCCL.cpp:4789] [PG ID 0 PG GUID 0 Rank 1] using GPU 1 as device used by this process is currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect. You can specify device_id in init_process_group() to force use of a particular device.
605
+ [rank7]:[W830 06:44:47.018941474 ProcessGroupNCCL.cpp:4789] [PG ID 0 PG GUID 0 Rank 7] using GPU 7 as device used by this process is currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect. You can specify device_id in init_process_group() to force use of a particular device.
606
+ [rank4]:[W830 06:44:47.018962476 ProcessGroupNCCL.cpp:4789] [PG ID 0 PG GUID 0 Rank 4] using GPU 4 as device used by this process is currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect. You can specify device_id in init_process_group() to force use of a particular device.
607
+ [rank5]:[W830 06:44:47.018992957 ProcessGroupNCCL.cpp:4789] [PG ID 0 PG GUID 0 Rank 5] using GPU 5 as device used by this process is currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect. You can specify device_id in init_process_group() to force use of a particular device.
608
+ [rank2]:[W830 06:44:47.019052212 ProcessGroupNCCL.cpp:4789] [PG ID 0 PG GUID 0 Rank 2] using GPU 2 as device used by this process is currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect. You can specify device_id in init_process_group() to force use of a particular device.
609
+ [rank6]:[W830 06:44:47.020946712 ProcessGroupNCCL.cpp:4789] [PG ID 0 PG GUID 0 Rank 6] using GPU 6 as device used by this process is currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect. You can specify device_id in init_process_group() to force use of a particular device.
610
+ > compiling dataset index builder ...
611
+ make: Entering directory '/workspace/Megatron-MoE-Echo/megatron/core/datasets'
612
+ make: Nothing to be done for 'default'.
613
+ make: Leaving directory '/workspace/Megatron-MoE-Echo/megatron/core/datasets'
614
+ >>> done with dataset index builder. Compilation time: 0.035 seconds
615
+ WARNING: constraints for invoking optimized fused softmax kernel are not met. We default back to unfused kernel invocations.
616
+ > compiling and loading fused kernels ...
617
+ [rank0]:[W830 06:44:47.177209335 ProcessGroupNCCL.cpp:4789] [PG ID 0 PG GUID 0 Rank 0] using GPU 0 as device used by this process is currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect. You can specify device_id in init_process_group() to force use of a particular device.
618
+ >>> done with compiling and loading fused kernels. Compilation time: 3.726 seconds
619
+ time to initialize megatron (seconds): 6.816
620
+ [after megatron is initialized] datetime: 2026-08-30 06:44:53
621
+ building GPT model ...
622
+ /usr/local/lib/python3.12/dist-packages/transformer_engine/pytorch/cpu_offload.py:708: DeprecationWarning: Offloading weights is deprecated. Using offload_weights=True does not have any effect.
623
+ warnings.warn(
624
+ /usr/local/lib/python3.12/dist-packages/transformer_engine/pytorch/cpu_offload.py:708: DeprecationWarning: Offloading weights is deprecated. Using offload_weights=True does not have any effect.
625
+ warnings.warn(
626
+ /usr/local/lib/python3.12/dist-packages/transformer_engine/pytorch/cpu_offload.py:708: DeprecationWarning: Offloading weights is deprecated. Using offload_weights=True does not have any effect.
627
+ warnings.warn(
628
+ /usr/local/lib/python3.12/dist-packages/transformer_engine/pytorch/cpu_offload.py:708: DeprecationWarning: Offloading weights is deprecated. Using offload_weights=True does not have any effect.
629
+ warnings.warn(
630
+ /usr/local/lib/python3.12/dist-packages/transformer_engine/pytorch/cpu_offload.py:708: DeprecationWarning: Offloading weights is deprecated. Using offload_weights=True does not have any effect.
631
+ warnings.warn(
632
+ /usr/local/lib/python3.12/dist-packages/transformer_engine/pytorch/cpu_offload.py:708: DeprecationWarning: Offloading weights is deprecated. Using offload_weights=True does not have any effect.
633
+ warnings.warn(
634
+ /usr/local/lib/python3.12/dist-packages/transformer_engine/pytorch/cpu_offload.py:708: DeprecationWarning: Offloading weights is deprecated. Using offload_weights=True does not have any effect.
635
+ warnings.warn(
636
+ /usr/local/lib/python3.12/dist-packages/transformer_engine/pytorch/cpu_offload.py:708: DeprecationWarning: Offloading weights is deprecated. Using offload_weights=True does not have any effect.
637
+ warnings.warn(
638
+ > number of parameters on (tensor, pipeline) model parallel rank (0, 0): 3902410752
639
+ INFO:megatron.core.distributed.distributed_data_parallel:Setting up DistributedDataParallel with config DistributedDataParallelConfig(grad_reduce_in_fp32=True, overlap_grad_reduce=True, overlap_param_gather=True, align_param_gather=False, use_distributed_optimizer=True, num_distributed_optimizer_instances=1, check_for_nan_in_grad=False, check_for_large_grads=False, bucket_size=40000000, pad_buckets_for_high_nccl_busbw=False, average_in_collective=False, fp8_param_gather=False, reuse_grad_buf_for_mxfp8_param_ag=False, use_custom_fsdp=False, data_parallel_sharding_strategy='no_shard', gradient_reduce_div_fusion=True, suggested_communication_unit_size=None, preserve_fp32_weights=True, keep_fp8_transpose_cache_when_using_custom_fsdp=False, nccl_ub=False, fsdp_double_buffer=False)
640
+ INFO:megatron.core.distributed.param_and_grad_buffer:Number of buckets for gradient all-reduce / reduce-scatter: 18
641
+ Params for bucket 1 (206045184 elements, 206045184 padded size):
642
+ module.output_layer.weight
643
+ Params for bucket 2 (41988096 elements, 41988096 padded size):
644
+ module.decoder.final_layernorm.weight
645
+ module.decoder.layers.15.self_attention.linear_qkv.weight
646
+ module.decoder.layers.15.pre_mlp_layernorm.weight
647
+ module.decoder.layers.15.mlp.router.weight
648
+ module.decoder.layers.15.self_attention.linear_qkv.layer_norm_weight
649
+ module.decoder.layers.15.self_attention.linear_proj.weight
650
+ Params for bucket 3 (41984000 elements, 41984000 padded size):
651
+ module.decoder.layers.14.mlp.router.weight
652
+ module.decoder.layers.14.self_attention.linear_proj.weight
653
+ module.decoder.layers.14.self_attention.linear_qkv.layer_norm_weight
654
+ module.decoder.layers.14.pre_mlp_layernorm.weight
655
+ module.decoder.layers.14.self_attention.linear_qkv.weight
656
+ Params for bucket 4 (41984000 elements, 41984000 padded size):
657
+ module.decoder.layers.13.self_attention.linear_qkv.weight
658
+ module.decoder.layers.13.pre_mlp_layernorm.weight
659
+ module.decoder.layers.13.mlp.router.weight
660
+ module.decoder.layers.13.self_attention.linear_proj.weight
661
+ module.decoder.layers.13.self_attention.linear_qkv.layer_norm_weight
662
+ Params for bucket 5 (41984000 elements, 41984000 padded size):
663
+ module.decoder.layers.12.mlp.router.weight
664
+ module.decoder.layers.12.self_attention.linear_proj.weight
665
+ module.decoder.layers.12.self_attention.linear_qkv.layer_norm_weight
666
+ module.decoder.layers.12.pre_mlp_layernorm.weight
667
+ module.decoder.layers.12.self_attention.linear_qkv.weight
668
+ Params for bucket 6 (41984000 elements, 41984000 padded size):
669
+ module.decoder.layers.11.self_attention.linear_qkv.weight
670
+ module.decoder.layers.11.pre_mlp_layernorm.weight
671
+ module.decoder.layers.11.mlp.router.weight
672
+ module.decoder.layers.11.self_attention.linear_proj.weight
673
+ module.decoder.layers.11.self_attention.linear_qkv.layer_norm_weight
674
+ Params for bucket 7 (41984000 elements, 41984000 padded size):
675
+ module.decoder.layers.10.mlp.router.weight
676
+ module.decoder.layers.10.self_attention.linear_qkv.weight
677
+ module.decoder.layers.10.pre_mlp_layernorm.weight
678
+ module.decoder.layers.10.self_attention.linear_proj.weight
679
+ module.decoder.layers.10.self_attention.linear_qkv.layer_norm_weight
680
+ Params for bucket 8 (41984000 elements, 41984000 padded size):
681
+ module.decoder.layers.9.mlp.router.weight
682
+ module.decoder.layers.9.self_attention.linear_proj.weight
683
+ module.decoder.layers.9.self_attention.linear_qkv.layer_norm_weight
684
+ module.decoder.layers.9.self_attention.linear_qkv.weight
685
+ module.decoder.layers.9.pre_mlp_layernorm.weight
686
+ Params for bucket 9 (41984000 elements, 41984000 padded size):
687
+ module.decoder.layers.8.self_attention.linear_qkv.layer_norm_weight
688
+ module.decoder.layers.8.self_attention.linear_proj.weight
689
+ module.decoder.layers.8.mlp.router.weight
690
+ module.decoder.layers.8.self_attention.linear_qkv.weight
691
+ module.decoder.layers.8.pre_mlp_layernorm.weight
692
+ Params for bucket 10 (41984000 elements, 41984000 padded size):
693
+ module.decoder.layers.7.mlp.router.weight
694
+ module.decoder.layers.7.pre_mlp_layernorm.weight
695
+ module.decoder.layers.7.self_attention.linear_qkv.weight
696
+ module.decoder.layers.7.self_attention.linear_proj.weight
697
+ module.decoder.layers.7.self_attention.linear_qkv.layer_norm_weight
698
+ Params for bucket 11 (41984000 elements, 41984000 padded size):
699
+ module.decoder.layers.6.mlp.router.weight
700
+ module.decoder.layers.6.self_attention.linear_qkv.layer_norm_weight
701
+ module.decoder.layers.6.self_attention.linear_proj.weight
702
+ module.decoder.layers.6.pre_mlp_layernorm.weight
703
+ module.decoder.layers.6.self_attention.linear_qkv.weight
704
+ Params for bucket 12 (41984000 elements, 41984000 padded size):
705
+ module.decoder.layers.5.mlp.router.weight
706
+ module.decoder.layers.5.self_attention.linear_qkv.layer_norm_weight
707
+ module.decoder.layers.5.self_attention.linear_qkv.weight
708
+ module.decoder.layers.5.pre_mlp_layernorm.weight
709
+ module.decoder.layers.5.self_attention.linear_proj.weight
710
+ Params for bucket 13 (41984000 elements, 41984000 padded size):
711
+ module.decoder.layers.4.mlp.router.weight
712
+ module.decoder.layers.4.self_attention.linear_qkv.layer_norm_weight
713
+ module.decoder.layers.4.self_attention.linear_proj.weight
714
+ module.decoder.layers.4.self_attention.linear_qkv.weight
715
+ module.decoder.layers.4.pre_mlp_layernorm.weight
716
+ Params for bucket 14 (41984000 elements, 41984000 padded size):
717
+ module.decoder.layers.3.self_attention.linear_qkv.weight
718
+ module.decoder.layers.3.pre_mlp_layernorm.weight
719
+ module.decoder.layers.3.mlp.router.weight
720
+ module.decoder.layers.3.self_attention.linear_proj.weight
721
+ module.decoder.layers.3.self_attention.linear_qkv.layer_norm_weight
722
+ Params for bucket 15 (41984000 elements, 41984000 padded size):
723
+ module.decoder.layers.2.mlp.router.weight
724
+ module.decoder.layers.2.self_attention.linear_qkv.layer_norm_weight
725
+ module.decoder.layers.2.self_attention.linear_proj.weight
726
+ module.decoder.layers.2.self_attention.linear_qkv.weight
727
+ module.decoder.layers.2.pre_mlp_layernorm.weight
728
+ Params for bucket 16 (41984000 elements, 41984000 padded size):
729
+ module.decoder.layers.1.self_attention.linear_qkv.layer_norm_weight
730
+ module.decoder.layers.1.self_attention.linear_proj.weight
731
+ module.decoder.layers.1.self_attention.linear_qkv.weight
732
+ module.decoder.layers.1.mlp.router.weight
733
+ module.decoder.layers.1.pre_mlp_layernorm.weight
734
+ Params for bucket 17 (41984000 elements, 41984000 padded size):
735
+ module.decoder.layers.0.mlp.router.weight
736
+ module.decoder.layers.0.pre_mlp_layernorm.weight
737
+ module.decoder.layers.0.self_attention.linear_qkv.layer_norm_weight
738
+ module.decoder.layers.0.self_attention.linear_qkv.weight
739
+ module.decoder.layers.0.self_attention.linear_proj.weight
740
+ Params for bucket 18 (206045184 elements, 206045184 padded size):
741
+ module.embedding.word_embeddings.weight
742
+ INFO:megatron.core.distributed.param_and_grad_buffer:Number of buckets for gradient all-reduce / reduce-scatter: 32
743
+ Params for bucket 1 (58720256 elements, 58720256 padded size):
744
+ module.decoder.layers.15.mlp.experts.linear_fc2.weight0
745
+ Params for bucket 2 (117440512 elements, 117440512 padded size):
746
+ module.decoder.layers.15.mlp.experts.linear_fc1.weight0
747
+ Params for bucket 3 (58720256 elements, 58720256 padded size):
748
+ module.decoder.layers.14.mlp.experts.linear_fc2.weight0
749
+ Params for bucket 4 (117440512 elements, 117440512 padded size):
750
+ module.decoder.layers.14.mlp.experts.linear_fc1.weight0
751
+ Params for bucket 5 (58720256 elements, 58720256 padded size):
752
+ module.decoder.layers.13.mlp.experts.linear_fc2.weight0
753
+ Params for bucket 6 (117440512 elements, 117440512 padded size):
754
+ module.decoder.layers.13.mlp.experts.linear_fc1.weight0
755
+ Params for bucket 7 (58720256 elements, 58720256 padded size):
756
+ module.decoder.layers.12.mlp.experts.linear_fc2.weight0
757
+ Params for bucket 8 (117440512 elements, 117440512 padded size):
758
+ module.decoder.layers.12.mlp.experts.linear_fc1.weight0
759
+ Params for bucket 9 (58720256 elements, 58720256 padded size):
760
+ module.decoder.layers.11.mlp.experts.linear_fc2.weight0
761
+ Params for bucket 10 (117440512 elements, 117440512 padded size):
762
+ module.decoder.layers.11.mlp.experts.linear_fc1.weight0
763
+ Params for bucket 11 (58720256 elements, 58720256 padded size):
764
+ module.decoder.layers.10.mlp.experts.linear_fc2.weight0
765
+ Params for bucket 12 (117440512 elements, 117440512 padded size):
766
+ module.decoder.layers.10.mlp.experts.linear_fc1.weight0
767
+ Params for bucket 13 (58720256 elements, 58720256 padded size):
768
+ module.decoder.layers.9.mlp.experts.linear_fc2.weight0
769
+ Params for bucket 14 (117440512 elements, 117440512 padded size):
770
+ module.decoder.layers.9.mlp.experts.linear_fc1.weight0
771
+ Params for bucket 15 (58720256 elements, 58720256 padded size):
772
+ module.decoder.layers.8.mlp.experts.linear_fc2.weight0
773
+ Params for bucket 16 (117440512 elements, 117440512 padded size):
774
+ module.decoder.layers.8.mlp.experts.linear_fc1.weight0
775
+ Params for bucket 17 (58720256 elements, 58720256 padded size):
776
+ module.decoder.layers.7.mlp.experts.linear_fc2.weight0
777
+ Params for bucket 18 (117440512 elements, 117440512 padded size):
778
+ module.decoder.layers.7.mlp.experts.linear_fc1.weight0
779
+ Params for bucket 19 (58720256 elements, 58720256 padded size):
780
+ module.decoder.layers.6.mlp.experts.linear_fc2.weight0
781
+ Params for bucket 20 (117440512 elements, 117440512 padded size):
782
+ module.decoder.layers.6.mlp.experts.linear_fc1.weight0
783
+ Params for bucket 21 (58720256 elements, 58720256 padded size):
784
+ module.decoder.layers.5.mlp.experts.linear_fc2.weight0
785
+ Params for bucket 22 (117440512 elements, 117440512 padded size):
786
+ module.decoder.layers.5.mlp.experts.linear_fc1.weight0
787
+ Params for bucket 23 (58720256 elements, 58720256 padded size):
788
+ module.decoder.layers.4.mlp.experts.linear_fc2.weight0
789
+ Params for bucket 24 (117440512 elements, 117440512 padded size):
790
+ module.decoder.layers.4.mlp.experts.linear_fc1.weight0
791
+ Params for bucket 25 (58720256 elements, 58720256 padded size):
792
+ module.decoder.layers.3.mlp.experts.linear_fc2.weight0
793
+ Params for bucket 26 (117440512 elements, 117440512 padded size):
794
+ module.decoder.layers.3.mlp.experts.linear_fc1.weight0
795
+ Params for bucket 27 (58720256 elements, 58720256 padded size):
796
+ module.decoder.layers.2.mlp.experts.linear_fc2.weight0
797
+ Params for bucket 28 (117440512 elements, 117440512 padded size):
798
+ module.decoder.layers.2.mlp.experts.linear_fc1.weight0
799
+ Params for bucket 29 (58720256 elements, 58720256 padded size):
800
+ module.decoder.layers.1.mlp.experts.linear_fc2.weight0
801
+ Params for bucket 30 (117440512 elements, 117440512 padded size):
802
+ module.decoder.layers.1.mlp.experts.linear_fc1.weight0
803
+ Params for bucket 31 (58720256 elements, 58720256 padded size):
804
+ module.decoder.layers.0.mlp.experts.linear_fc2.weight0
805
+ Params for bucket 32 (117440512 elements, 117440512 padded size):
806
+ module.decoder.layers.0.mlp.experts.linear_fc1.weight0
807
+ INFO:megatron.core.optimizer:Setting up optimizer with config OptimizerConfig(optimizer='adam', lr=0.0001, min_lr=1e-05, decoupled_lr=None, decoupled_min_lr=None, weight_decay=0.1, fp8_recipe='mxfp8', fp16=False, bf16=True, reuse_grad_buf_for_mxfp8_param_ag=False, params_dtype=torch.bfloat16, use_precision_aware_optimizer=False, store_param_remainders=True, main_grads_dtype=torch.float32, main_params_dtype=torch.float32, exp_avg_dtype=torch.float32, exp_avg_sq_dtype=torch.float32, loss_scale=None, initial_loss_scale=4294967296, min_loss_scale=1.0, loss_scale_window=1000, hysteresis=2, adam_beta1=0.9, adam_beta2=0.999, adam_eps=1e-08, sgd_momentum=0.9, use_distributed_optimizer=True, overlap_param_gather_with_optimizer_step=False, optimizer_cpu_offload=False, optimizer_offload_fraction=1.0, use_torch_optimizer_for_cpu_offload=False, overlap_cpu_optimizer_d2h_h2d=False, pin_cpu_grads=True, pin_cpu_params=True, clip_grad=1.0, log_num_zeros_in_grad=False, barrier_with_L1_time=True, timers=<megatron.core.timers.Timers object at 0x72cef00f86e0>, config_logger_dir='')
808
+ INFO:megatron.core.optimizer_param_scheduler:> learning rate decay style: cosine
809
+ [after model, optimizer, and learning rate scheduler are built] datetime: 2026-08-30 06:44:53
810
+ > building train, validation, and test datasets ...
811
+ > datasets target sizes (minimum size):
812
+ train: 192
813
+ validation: 0
814
+ test: 0
815
+ INFO:megatron.core.datasets.blended_megatron_dataset_config:Let split_matrix = [(0, 1.0), None, None]
816
+ > building train, validation, and test datasets for GPT ...
817
+ INFO:megatron.core.datasets.blended_megatron_dataset_builder:Building GPTDataset splits with sizes=(192, 0, 0) and config=GPTDatasetConfig(random_seed=1234, sequence_length=4096, blend=(['/workspace/data/wikitext/wikitext103_gpt2_text_document'], None), blend_per_split=None, split='100,0,0', split_matrix=[(0, 1.0), None, None], num_dataset_builder_threads=1, path_to_cache=None, mmap_bin_files=True, mock=False, tokenizer=<megatron.training.tokenizer.tokenizer._GPT2BPETokenizer object at 0x72cef03b4ef0>, mid_level_dataset_surplus=0.005, reset_position_ids=False, reset_attention_mask=False, eod_mask_loss=False, create_attention_mask=True, drop_last_partial_validation_sequence=True, add_extra_token_to_sequence=True, object_storage_cache_path=None)
818
+ INFO:megatron.core.datasets.indexed_dataset:Load the _IndexReader from /workspace/data/wikitext/wikitext103_gpt2_text_document.idx
819
+ INFO:megatron.core.datasets.indexed_dataset: Extract the sequence lengths
820
+ INFO:megatron.core.datasets.indexed_dataset: Extract the sequence pointers
821
+ INFO:megatron.core.datasets.indexed_dataset: Extract the document indices
822
+ INFO:megatron.core.datasets.indexed_dataset:> total number of sequences: 1165029
823
+ INFO:megatron.core.datasets.indexed_dataset:> total number of documents: 1801350
824
+ INFO:megatron.core.datasets.gpt_dataset:Load the GPTDataset train indices
825
+ INFO:megatron.core.datasets.gpt_dataset: Load the document index from 44ed1ee5bcb9e5b030387b8d27a6c406-GPTDataset-train-document_index.npy
826
+ INFO:megatron.core.datasets.gpt_dataset: Load the sample index from 44ed1ee5bcb9e5b030387b8d27a6c406-GPTDataset-train-sample_index.npy
827
+ INFO:megatron.core.datasets.gpt_dataset: Load the shuffle index from 44ed1ee5bcb9e5b030387b8d27a6c406-GPTDataset-train-shuffle_index.npy
828
+ INFO:megatron.core.datasets.gpt_dataset:> total number of samples: 29073
829
+ > finished creating GPT datasets ...
830
+ [after dataloaders are built] datetime: 2026-08-30 06:44:53
831
+ done with setup ...
832
+ training ...
833
+ Overwriting rerun_state_machine.current_iteration from -1 to 0...
834
+ (min, max) time across ranks (ms):
835
+ model-and-optimizer-setup ......................: (207.42, 209.76)
836
+ train/valid/test-data-iterators-setup ..........: (63.21, 71.94)
837
+ [before the start of training step] datetime: 2026-08-30 06:44:53
838
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:334: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
839
+ warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
840
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:334: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
841
+ warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
842
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:334: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
843
+ warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
844
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:334: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
845
+ warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
846
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:334: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
847
+ warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
848
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:336: UserWarning: The use_host_meta is deprecated, it will be removed in the future.
849
+ warnings.warn("The use_host_meta is deprecated, it will be removed in the future.")
850
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:336: UserWarning: The use_host_meta is deprecated, it will be removed in the future.
851
+ warnings.warn("The use_host_meta is deprecated, it will be removed in the future.")
852
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:334: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
853
+ warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
854
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:336: UserWarning: The use_host_meta is deprecated, it will be removed in the future.
855
+ warnings.warn("The use_host_meta is deprecated, it will be removed in the future.")
856
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:336: UserWarning: The use_host_meta is deprecated, it will be removed in the future.
857
+ warnings.warn("The use_host_meta is deprecated, it will be removed in the future.")
858
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:336: UserWarning: The use_host_meta is deprecated, it will be removed in the future.
859
+ warnings.warn("The use_host_meta is deprecated, it will be removed in the future.")
860
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:334: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
861
+ warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
862
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:334: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
863
+ warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
864
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:336: UserWarning: The use_host_meta is deprecated, it will be removed in the future.
865
+ warnings.warn("The use_host_meta is deprecated, it will be removed in the future.")
866
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:336: UserWarning: The use_host_meta is deprecated, it will be removed in the future.
867
+ warnings.warn("The use_host_meta is deprecated, it will be removed in the future.")
868
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:336: UserWarning: The use_host_meta is deprecated, it will be removed in the future.
869
+ warnings.warn("The use_host_meta is deprecated, it will be removed in the future.")
870
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:482: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
871
+ warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
872
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:482: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
873
+ warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
874
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:482: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
875
+ warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
876
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:482: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
877
+ warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
878
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:482: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
879
+ warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
880
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:482: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
881
+ warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
882
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:482: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
883
+ warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
884
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:482: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
885
+ warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
886
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:334: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
887
+ warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
888
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:334: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
889
+ warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
890
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:336: UserWarning: The use_host_meta is deprecated, it will be removed in the future.
891
+ warnings.warn("The use_host_meta is deprecated, it will be removed in the future.")
892
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:336: UserWarning: The use_host_meta is deprecated, it will be removed in the future.
893
+ warnings.warn("The use_host_meta is deprecated, it will be removed in the future.")
894
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:334: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
895
+ warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
896
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:336: UserWarning: The use_host_meta is deprecated, it will be removed in the future.
897
+ warnings.warn("The use_host_meta is deprecated, it will be removed in the future.")
898
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:334: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
899
+ warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
900
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:336: UserWarning: The use_host_meta is deprecated, it will be removed in the future.
901
+ warnings.warn("The use_host_meta is deprecated, it will be removed in the future.")
902
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:334: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
903
+ warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
904
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:336: UserWarning: The use_host_meta is deprecated, it will be removed in the future.
905
+ warnings.warn("The use_host_meta is deprecated, it will be removed in the future.")
906
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:334: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
907
+ warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
908
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:336: UserWarning: The use_host_meta is deprecated, it will be removed in the future.
909
+ warnings.warn("The use_host_meta is deprecated, it will be removed in the future.")
910
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:334: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
911
+ warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
912
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:336: UserWarning: The use_host_meta is deprecated, it will be removed in the future.
913
+ warnings.warn("The use_host_meta is deprecated, it will be removed in the future.")
914
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:334: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
915
+ warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
916
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:336: UserWarning: The use_host_meta is deprecated, it will be removed in the future.
917
+ warnings.warn("The use_host_meta is deprecated, it will be removed in the future.")
918
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:482: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
919
+ warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
920
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:482: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
921
+ warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
922
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:482: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
923
+ warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
924
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:482: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
925
+ warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
926
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:482: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
927
+ warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
928
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:482: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
929
+ warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
930
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:482: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
931
+ warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
932
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:482: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
933
+ warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
934
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:334: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
935
+ warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
936
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:334: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
937
+ warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
938
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:334: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
939
+ warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
940
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:334: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
941
+ warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
942
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:334: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
943
+ warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
944
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:334: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
945
+ warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
946
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:334: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
947
+ warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
948
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:334: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
949
+ warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
950
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:336: UserWarning: The use_host_meta is deprecated, it will be removed in the future.
951
+ warnings.warn("The use_host_meta is deprecated, it will be removed in the future.")
952
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:336: UserWarning: The use_host_meta is deprecated, it will be removed in the future.
953
+ warnings.warn("The use_host_meta is deprecated, it will be removed in the future.")
954
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:336: UserWarning: The use_host_meta is deprecated, it will be removed in the future.
955
+ warnings.warn("The use_host_meta is deprecated, it will be removed in the future.")
956
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:336: UserWarning: The use_host_meta is deprecated, it will be removed in the future.
957
+ warnings.warn("The use_host_meta is deprecated, it will be removed in the future.")
958
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:336: UserWarning: The use_host_meta is deprecated, it will be removed in the future.
959
+ warnings.warn("The use_host_meta is deprecated, it will be removed in the future.")
960
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:336: UserWarning: The use_host_meta is deprecated, it will be removed in the future.
961
+ warnings.warn("The use_host_meta is deprecated, it will be removed in the future.")
962
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:336: UserWarning: The use_host_meta is deprecated, it will be removed in the future.
963
+ warnings.warn("The use_host_meta is deprecated, it will be removed in the future.")
964
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:336: UserWarning: The use_host_meta is deprecated, it will be removed in the future.
965
+ warnings.warn("The use_host_meta is deprecated, it will be removed in the future.")
966
+ [2026-08-30 06:45:31] iteration 1/ 12 | consumed samples: 16 | elapsed time per iteration (ms): 37891.4 | throughput per GPU (TFLOP/s/GPU): 8.8 | learning rate: 2.000000E-05 | global batch size: 16 | lm loss: 1.104376E+01 | loss scale: 1.0 | grad norm: 34.298 | number of skipped iterations: 0 | number of nan iterations: 0 |Number of parameters in transformer block in billions: 23.22
967
+
968
+ Number of parameters in embedding layers in billions: 0.41
969
+ Total number of parameters in billions: 23.63
970
+ Number of parameters in most loaded shard in billions: 23.6320
971
+ Theoretical memory footprints: weight and optimizer=169029.43 MB
972
+ [Rank 0] (after 1 iterations) memory (MB) | allocated: 63795.404296875 | max allocated: 96519.67578125 | reserved: 99122.0 | max reserved: 99122.0
973
+ [2026-08-30 06:45:32] iteration 2/ 12 | consumed samples: 32 | elapsed time per iteration (ms): 579.4 | throughput per GPU (TFLOP/s/GPU): 575.4 | learning rate: 4.000000E-05 | global batch size: 16 | lm loss: 1.104935E+01 | loss scale: 1.0 | grad norm: 35.989 | number of skipped iterations: 0 | number of nan iterations: 0 |
974
+ [2026-08-30 06:45:33] iteration 3/ 12 | consumed samples: 48 | elapsed time per iteration (ms): 559.7 | throughput per GPU (TFLOP/s/GPU): 595.7 | learning rate: 6.000000E-05 | global batch size: 16 | lm loss: 9.657161E+00 | loss scale: 1.0 | grad norm: 40.337 | number of skipped iterations: 0 | number of nan iterations: 0 |
975
+ Capture CUDA graph for training!!!
976
+ CUDA graph capture done!!!
977
+ [2026-08-30 06:45:34] iteration 4/ 12 | consumed samples: 64 | elapsed time per iteration (ms): 1645.4 | throughput per GPU (TFLOP/s/GPU): 202.6 | learning rate: 8.000000E-05 | global batch size: 16 | lm loss: 1.080619E+01 | loss scale: 1.0 | grad norm: 93.469 | number of skipped iterations: 0 | number of nan iterations: 0 |
978
+ [2026-08-30 06:45:35] iteration 5/ 12 | consumed samples: 80 | elapsed time per iteration (ms): 470.5 | throughput per GPU (TFLOP/s/GPU): 708.5 | learning rate: 1.000000E-04 | global batch size: 16 | lm loss: 1.068921E+01 | loss scale: 1.0 | grad norm: 17.369 | number of skipped iterations: 0 | number of nan iterations: 0 |
979
+ [2026-08-30 06:45:35] iteration 6/ 12 | consumed samples: 96 | elapsed time per iteration (ms): 467.0 | throughput per GPU (TFLOP/s/GPU): 713.9 | learning rate: 7.750000E-05 | global batch size: 16 | lm loss: 1.001069E+01 | loss scale: 1.0 | grad norm: 45.636 | number of skipped iterations: 0 | number of nan iterations: 0 |
980
+ [2026-08-30 06:45:36] iteration 7/ 12 | consumed samples: 112 | elapsed time per iteration (ms): 469.0 | throughput per GPU (TFLOP/s/GPU): 710.8 | learning rate: 3.250000E-05 | global batch size: 16 | lm loss: 9.327785E+00 | loss scale: 1.0 | grad norm: 7.423 | number of skipped iterations: 0 | number of nan iterations: 0 |
981
+ [2026-08-30 06:45:36] iteration 8/ 12 | consumed samples: 128 | elapsed time per iteration (ms): 465.5 | throughput per GPU (TFLOP/s/GPU): 716.2 | learning rate: 1.000000E-05 | global batch size: 16 | lm loss: 9.224079E+00 | loss scale: 1.0 | grad norm: 29.331 | number of skipped iterations: 0 | number of nan iterations: 0 |
982
+ [2026-08-30 06:45:36] iteration 9/ 12 | consumed samples: 144 | elapsed time per iteration (ms): 468.3 | throughput per GPU (TFLOP/s/GPU): 711.9 | learning rate: 1.000000E-05 | global batch size: 16 | lm loss: 8.703784E+00 | loss scale: 1.0 | grad norm: 5.921 | number of skipped iterations: 0 | number of nan iterations: 0 |
983
+ [2026-08-30 06:45:37] iteration 10/ 12 | consumed samples: 160 | elapsed time per iteration (ms): 462.1 | throughput per GPU (TFLOP/s/GPU): 721.5 | learning rate: 1.000000E-05 | global batch size: 16 | lm loss: 8.598076E+00 | loss scale: 1.0 | grad norm: 5.870 | number of skipped iterations: 0 | number of nan iterations: 0 |
984
+ [2026-08-30 06:45:37] iteration 11/ 12 | consumed samples: 176 | elapsed time per iteration (ms): 468.0 | throughput per GPU (TFLOP/s/GPU): 712.3 | learning rate: 1.000000E-05 | global batch size: 16 | lm loss: 8.479810E+00 | loss scale: 1.0 | grad norm: 4.778 | number of skipped iterations: 0 | number of nan iterations: 0 |
985
+ [2026-08-30 06:45:38] iteration 12/ 12 | consumed samples: 192 | elapsed time per iteration (ms): 468.1 | throughput per GPU (TFLOP/s/GPU): 712.3 | learning rate: 1.000000E-05 | global batch size: 16 | lm loss: 8.394531E+00 | loss scale: 1.0 | grad norm: 4.727 | number of skipped iterations: 0 | number of nan iterations: 0 |
986
+ [after training is done] datetime: 2026-08-30 06:45:38
987
+ W0830 06:45:38.726000 651947 torch/distributed/elastic/agent/server/api.py:719] Received 15 death signal, shutting down workers
988
+ W0830 06:45:38.727000 651947 torch/distributed/elastic/multiprocessing/api.py:900] Sending process 652036 closing signal SIGTERM
989
+ W0830 06:45:38.728000 651947 torch/distributed/elastic/multiprocessing/api.py:900] Sending process 652037 closing signal SIGTERM
990
+ W0830 06:45:38.730000 651947 torch/distributed/elastic/multiprocessing/api.py:900] Sending process 652038 closing signal SIGTERM
991
+ W0830 06:45:38.732000 651947 torch/distributed/elastic/multiprocessing/api.py:900] Sending process 652039 closing signal SIGTERM
992
+ W0830 06:45:38.732000 651947 torch/distributed/elastic/multiprocessing/api.py:900] Sending process 652040 closing signal SIGTERM
993
+ W0830 06:45:38.735000 651947 torch/distributed/elastic/multiprocessing/api.py:900] Sending process 652041 closing signal SIGTERM
994
+ W0830 06:45:38.736000 651947 torch/distributed/elastic/multiprocessing/api.py:900] Sending process 652042 closing signal SIGTERM
995
+ W0830 06:45:38.738000 651947 torch/distributed/elastic/multiprocessing/api.py:900] Sending process 652043 closing signal SIGTERM
996
+ Traceback (most recent call last):
997
+ File "/usr/local/bin/torchrun", line 8, in <module>
998
+ sys.exit(main())
999
+ ^^^^^^
1000
+ File "/usr/local/lib/python3.12/dist-packages/torch/distributed/elastic/multiprocessing/errors/__init__.py", line 355, in wrapper
1001
+ return f(*args, **kwargs)
1002
+ ^^^^^^^^^^^^^^^^^^
1003
+ File "/usr/local/lib/python3.12/dist-packages/torch/distributed/run.py", line 892, in main
1004
+ run(args)
1005
+ File "/usr/local/lib/python3.12/dist-packages/torch/distributed/run.py", line 883, in run
1006
+ elastic_launch(
1007
+ File "/usr/local/lib/python3.12/dist-packages/torch/distributed/launcher/api.py", line 139, in __call__
1008
+ return launch_agent(self._config, self._entrypoint, list(args))
1009
+ ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
1010
+ File "/usr/local/lib/python3.12/dist-packages/torch/distributed/launcher/api.py", line 261, in launch_agent
1011
+ result = agent.run()
1012
+ ^^^^^^^^^^^
1013
+ File "/usr/local/lib/python3.12/dist-packages/torch/distributed/elastic/metrics/api.py", line 138, in wrapper
1014
+ result = f(*args, **kwargs)
1015
+ ^^^^^^^^^^^^^^^^^^
1016
+ File "/usr/local/lib/python3.12/dist-packages/torch/distributed/elastic/agent/server/api.py", line 711, in run
1017
+ result = self._invoke_run(role)
1018
+ ^^^^^^^^^^^^^^^^^^^^^^
1019
+ File "/usr/local/lib/python3.12/dist-packages/torch/distributed/elastic/agent/server/api.py", line 870, in _invoke_run
1020
+ time.sleep(monitor_interval)
1021
+ File "/usr/local/lib/python3.12/dist-packages/torch/distributed/elastic/multiprocessing/api.py", line 84, in _terminate_process_handler
1022
+ raise SignalException(f"Process {os.getpid()} got signal: {sigval}", sigval=sigval)
1023
+ torch.distributed.elastic.multiprocessing.api.SignalException: Process 651947 got signal: 15
tiera-rerun-fixed-moonep/echo/a2/a2_matched_gbs16/G1__echo__matched__seq4096__gbs16__cap2p40625/rep2/mem_samples.csv ADDED
@@ -0,0 +1,264 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ 0, 0
2
+ 1, 0
3
+ 2, 0
4
+ 3, 0
5
+ 4, 0
6
+ 5, 0
7
+ 6, 0
8
+ 7, 0
9
+ 0, 0
10
+ 1, 0
11
+ 2, 0
12
+ 3, 0
13
+ 4, 0
14
+ 5, 0
15
+ 6, 0
16
+ 7, 0
17
+ 0, 4
18
+ 1, 1
19
+ 2, 1
20
+ 3, 1
21
+ 4, 1
22
+ 5, 1
23
+ 6, 1
24
+ 7, 1
25
+ 0, 4
26
+ 1, 4
27
+ 2, 4
28
+ 3, 4
29
+ 4, 4
30
+ 5, 1
31
+ 6, 1
32
+ 7, 1
33
+ 0, 4
34
+ 1, 4
35
+ 2, 4
36
+ 3, 4
37
+ 4, 4
38
+ 5, 4
39
+ 6, 4
40
+ 7, 4
41
+ 0, 4
42
+ 1, 4
43
+ 2, 4
44
+ 3, 4
45
+ 4, 4
46
+ 5, 4
47
+ 6, 4
48
+ 7, 4
49
+ 0, 821
50
+ 1, 821
51
+ 2, 821
52
+ 3, 821
53
+ 4, 821
54
+ 5, 821
55
+ 6, 821
56
+ 7, 821
57
+ 0, 1537
58
+ 1, 1537
59
+ 2, 1537
60
+ 3, 1537
61
+ 4, 1537
62
+ 5, 1537
63
+ 6, 1537
64
+ 7, 1537
65
+ 0, 2231
66
+ 1, 2231
67
+ 2, 2231
68
+ 3, 2231
69
+ 4, 2231
70
+ 5, 2231
71
+ 6, 2231
72
+ 7, 2231
73
+ 0, 39545
74
+ 1, 39537
75
+ 2, 39569
76
+ 3, 39577
77
+ 4, 39587
78
+ 5, 39599
79
+ 6, 39571
80
+ 7, 39565
81
+ 0, 39823
82
+ 1, 39847
83
+ 2, 39845
84
+ 3, 39847
85
+ 4, 39829
86
+ 5, 39849
87
+ 6, 39867
88
+ 7, 39843
89
+ 0, 42357
90
+ 1, 42355
91
+ 2, 42355
92
+ 3, 42355
93
+ 4, 42355
94
+ 5, 42355
95
+ 6, 42355
96
+ 7, 42355
97
+ 0, 42583
98
+ 1, 42581
99
+ 2, 42581
100
+ 3, 42581
101
+ 4, 42581
102
+ 5, 42581
103
+ 6, 42581
104
+ 7, 42581
105
+ 0, 42635
106
+ 1, 42631
107
+ 2, 42631
108
+ 3, 42623
109
+ 4, 42635
110
+ 5, 42641
111
+ 6, 42643
112
+ 7, 42633
113
+ 0, 43493
114
+ 1, 43491
115
+ 2, 43491
116
+ 3, 43491
117
+ 4, 43491
118
+ 5, 43491
119
+ 6, 43491
120
+ 7, 43491
121
+ 0, 43493
122
+ 1, 43491
123
+ 2, 43491
124
+ 3, 43491
125
+ 4, 43491
126
+ 5, 43491
127
+ 6, 43491
128
+ 7, 43491
129
+ 0, 44571
130
+ 1, 44569
131
+ 2, 44569
132
+ 3, 44569
133
+ 4, 44569
134
+ 5, 44569
135
+ 6, 44569
136
+ 7, 44569
137
+ 0, 45265
138
+ 1, 45263
139
+ 2, 45263
140
+ 3, 45263
141
+ 4, 45263
142
+ 5, 45263
143
+ 6, 45263
144
+ 7, 45263
145
+ 0, 85827
146
+ 1, 85825
147
+ 2, 85825
148
+ 3, 85825
149
+ 4, 85825
150
+ 5, 85825
151
+ 6, 85825
152
+ 7, 85825
153
+ 0, 86905
154
+ 1, 86903
155
+ 2, 86903
156
+ 3, 86903
157
+ 4, 86903
158
+ 5, 86903
159
+ 6, 86903
160
+ 7, 86903
161
+ 0, 86905
162
+ 1, 86903
163
+ 2, 86903
164
+ 3, 86903
165
+ 4, 86903
166
+ 5, 86903
167
+ 6, 86903
168
+ 7, 86903
169
+ 0, 86905
170
+ 1, 86903
171
+ 2, 86903
172
+ 3, 86903
173
+ 4, 86903
174
+ 5, 86903
175
+ 6, 86903
176
+ 7, 86903
177
+ 0, 103929
178
+ 1, 103889
179
+ 2, 103877
180
+ 3, 103893
181
+ 4, 103947
182
+ 5, 103931
183
+ 6, 103877
184
+ 7, 103883
185
+ 0, 105801
186
+ 1, 105771
187
+ 2, 105791
188
+ 3, 105781
189
+ 4, 105871
190
+ 5, 105759
191
+ 6, 105815
192
+ 7, 105815
193
+ 0, 106523
194
+ 1, 106465
195
+ 2, 106465
196
+ 3, 106465
197
+ 4, 106521
198
+ 5, 106465
199
+ 6, 106465
200
+ 7, 106465
201
+ 0, 107149
202
+ 1, 107111
203
+ 2, 107041
204
+ 3, 107139
205
+ 4, 107171
206
+ 5, 107171
207
+ 6, 107185
208
+ 7, 107139
209
+ 0, 147989
210
+ 1, 135369
211
+ 2, 141587
212
+ 3, 146035
213
+ 4, 140705
214
+ 5, 137961
215
+ 6, 139429
216
+ 7, 141585
217
+ 0, 161639
218
+ 1, 161525
219
+ 2, 161527
220
+ 3, 161525
221
+ 4, 161723
222
+ 5, 161529
223
+ 6, 161525
224
+ 7, 161525
225
+ 0, 161639
226
+ 1, 161525
227
+ 2, 161527
228
+ 3, 161525
229
+ 4, 161723
230
+ 5, 161529
231
+ 6, 161525
232
+ 7, 161525
233
+ 0, 160731
234
+ 1, 160617
235
+ 2, 160619
236
+ 3, 160617
237
+ 4, 160815
238
+ 5, 160621
239
+ 6, 160617
240
+ 7, 160617
241
+ 0, 2090
242
+ 1, 160617
243
+ 2, 3114
244
+ 3, 1834
245
+ 4, 1834
246
+ 5, 1834
247
+ 6, 1834
248
+ 7, 1834
249
+ 0, 0
250
+ 1, 0
251
+ 2, 0
252
+ 3, 0
253
+ 4, 0
254
+ 5, 0
255
+ 6, 0
256
+ 7, 0
257
+ 0, 0
258
+ 1, 0
259
+ 2, 0
260
+ 3, 0
261
+ 4, 0
262
+ 5, 0
263
+ 6, 0
264
+ 7, 0
tiera-rerun-fixed-moonep/echo/a2/a2_matched_gbs16/G1__echo__matched__seq4096__gbs16__cap2p40625/rep2/run.env ADDED
@@ -0,0 +1,70 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ backend=echo
2
+ config=G1
3
+ family=tierA
4
+ repo=/workspace/Megatron-MoE-Echo
5
+ branch=bracing-regions-through-kernels
6
+ commit=9a236793
7
+ num_layers=16
8
+ hidden=4096
9
+ ffn_hidden=14336
10
+ moe_ffn_hidden=14336
11
+ num_experts=8
12
+ topk=2
13
+ experts_per_rank=1
14
+ expert_params=176160768
15
+ heads=32
16
+ kv_groups=8
17
+ kv_channels=
18
+ qk_layernorm=0
19
+ shared_ffn=0
20
+ shared_expert_overlap=0
21
+ moe_token_dispatcher=flex
22
+ mla=0
23
+ dense_first_layers=0
24
+ moe_layers=16
25
+ rotary_base=1000000
26
+ rotary_scaling=
27
+ qkv_bias=0
28
+ router_balancing=aux_loss
29
+ aux_loss_coeff=1e-2
30
+ router_pre_softmax=0
31
+ router_fusion=0
32
+ router_dtype=fp32
33
+ device_initiated_grouped_gemm=1
34
+ norm_epsilon=1e-5
35
+ seq_length=4096
36
+ micro_batch_size=1
37
+ global_batch_size=16
38
+ num_microbatches=2
39
+ assignments_per_rank=8192
40
+ mean_tokens_per_expert=8192
41
+ slot_policy=matched
42
+ slots_per_rank=1
43
+ capacity_mode=static
44
+ capacity=2.40625
45
+ cuda_graph=1
46
+ probe_mode=0
47
+ async_grad_reduce=0
48
+ branch_role=timing
49
+ memory_dump_mode=none
50
+ memory_snapshot_path=
51
+ cuda_graph_warmup_steps=3
52
+ recompute=none
53
+ force_balance=0
54
+ train_iters=12
55
+ ep_size=8
56
+ echo_dispatch_overlap=1
57
+ echo_dispatch_recompute=0
58
+ ultraep_quota_min_tokens=
59
+ ultraep_weight_sync_join=
60
+ profile_enable=0
61
+ kernel_trace=0
62
+ fp8_mode=mxfp8
63
+ fp8_recipe=mxfp8
64
+ training_seed=1234
65
+ learning_rate=1e-4
66
+ min_learning_rate=1.0e-5
67
+ lr_warmup_iters=5
68
+ state_dump_enable=0
69
+ state_dump_dir=
70
+ moonep_assert_no_pending_reduce=0
tiera-rerun-fixed-moonep/echo/a2/a2_matched_gbs16/G1__echo__matched__seq4096__gbs16__cap2p40625/rep2/train.log ADDED
@@ -0,0 +1,987 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ==============================================================
2
+ G1 / echo (slots/rank=1 policy=matched)
3
+ E=8 K=2 H'=14336 m=1
4
+ L=16 seq=4096 gbs=16 (2 microbatches)
5
+ cap_mode=static cap=2.40625 cg=1 recompute=none
6
+ memory_dump=none snapshots=/workspace/tiera-rerun-fixed-moonep/echo/a2/a2_matched_gbs16/G1__echo__matched__seq4096__gbs16__cap2p40625/rep2/memory_snapshots/snapshot.pickle
7
+ repo=/workspace/Megatron-MoE-Echo @ 9a236793
8
+ ==============================================================
9
+ W0830 06:45:45.250000 655335 torch/distributed/run.py:766]
10
+ W0830 06:45:45.250000 655335 torch/distributed/run.py:766] *****************************************
11
+ W0830 06:45:45.250000 655335 torch/distributed/run.py:766] Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed.
12
+ W0830 06:45:45.250000 655335 torch/distributed/run.py:766] *****************************************
13
+ using world size: 8, data-parallel size: 8, context-parallel size: 1, hierarchical context-parallel sizes: None, tensor-model-parallel size: 1, pipeline-model-parallel size: 1
14
+ WARNING: overriding default arguments for tokenizer_type:GPT2BPETokenizer with tokenizer_type:GPT2BPETokenizer
15
+ Number of virtual stages per pipeline stage: None
16
+ accumulate and all-reduce gradients in fp32 for bfloat16 data type.
17
+ using torch.bfloat16 for parameters ...
18
+ /workspace/Megatron-MoE-Echo/megatron/training/arguments.py:831: UserWarning: Disabling sequence parallelism because tensor model parallelism is disabled
19
+ warnings.warn("Disabling sequence parallelism because tensor model parallelism is disabled")
20
+ ------------------------ arguments ------------------------
21
+ account_for_embedding_in_pipeline_split ......... False
22
+ account_for_loss_in_pipeline_split .............. False
23
+ accumulate_allreduce_grads_in_fp32 .............. True
24
+ adam_beta1 ...................................... 0.9
25
+ adam_beta2 ...................................... 0.999
26
+ adam_eps ........................................ 1e-08
27
+ add_bias_linear ................................. False
28
+ add_position_embedding .......................... False
29
+ add_qkv_bias .................................... False
30
+ adlr_autoresume ................................. False
31
+ adlr_autoresume_interval ........................ 1000
32
+ align_grad_reduce ............................... True
33
+ align_param_gather .............................. False
34
+ app_tag_run_name ................................ None
35
+ app_tag_run_version ............................. 0.0.0
36
+ apply_layernorm_1p .............................. False
37
+ apply_query_key_layer_scaling ................... False
38
+ apply_residual_connection_post_layernorm ........ False
39
+ apply_rope_fusion ............................... True
40
+ async_save ...................................... None
41
+ async_tensor_model_parallel_allreduce ........... True
42
+ attention_backend ............................... AttnBackend.auto
43
+ attention_dropout ............................... 0.0
44
+ attention_softmax_in_fp32 ....................... False
45
+ auto_detect_ckpt_format ......................... False
46
+ barrier_with_L1_time ............................ True
47
+ bert_binary_head ................................ True
48
+ bert_embedder_type .............................. megatron
49
+ bert_load ....................................... None
50
+ bf16 ............................................ True
51
+ bias_dropout_fusion ............................. True
52
+ bias_gelu_fusion ................................ False
53
+ bias_swiglu_fusion .............................. True
54
+ biencoder_projection_dim ........................ 0
55
+ biencoder_shared_query_context_model ............ False
56
+ block_data_path ................................. None
57
+ calc_ft_timeouts ................................ False
58
+ calculate_per_token_loss ........................ False
59
+ check_for_large_grads ........................... False
60
+ check_for_nan_in_loss_and_grad .................. False
61
+ check_for_spiky_loss ............................ False
62
+ check_weight_hash_across_dp_replicas_interval ... None
63
+ ckpt_assume_constant_structure .................. False
64
+ ckpt_convert_format ............................. None
65
+ ckpt_convert_save ............................... None
66
+ ckpt_convert_update_legacy_dist_opt_format ...... False
67
+ ckpt_format ..................................... torch_dist
68
+ ckpt_fully_parallel_load ........................ False
69
+ ckpt_fully_parallel_save ........................ True
70
+ ckpt_fully_parallel_save_deprecated ............. False
71
+ ckpt_step ....................................... None
72
+ classes_fraction ................................ 1.0
73
+ clip_grad ....................................... 1.0
74
+ clone_scatter_output_in_embedding ............... True
75
+ config_logger_dir ...............................
76
+ consumed_train_samples .......................... 0
77
+ consumed_valid_samples .......................... 0
78
+ context_parallel_size ........................... 1
79
+ cp_comm_type .................................... ['p2p']
80
+ create_attention_mask_in_dataloader ............. True
81
+ cross_entropy_fusion_impl ....................... native
82
+ cross_entropy_loss_fusion ....................... False
83
+ cuda_graph_scope ................................ full_iteration
84
+ cuda_graph_warmup_steps ......................... 3
85
+ data_args_path .................................. None
86
+ data_cache_path ................................. None
87
+ data_parallel_random_init ....................... False
88
+ data_parallel_sharding_strategy ................. no_shard
89
+ data_parallel_size .............................. 8
90
+ data_path ....................................... ['/workspace/data/wikitext/wikitext103_gpt2_text_document']
91
+ data_per_class_fraction ......................... 1.0
92
+ data_sharding ................................... True
93
+ dataloader_type ................................. single
94
+ ddp_average_in_collective ....................... False
95
+ ddp_bucket_size ................................. None
96
+ ddp_num_buckets ................................. None
97
+ ddp_pad_buckets_for_high_nccl_busbw ............. False
98
+ decoder_first_pipeline_num_layers ............... None
99
+ decoder_last_pipeline_num_layers ................ None
100
+ decoder_num_layers .............................. None
101
+ decoder_seq_length .............................. None
102
+ decoupled_lr .................................... None
103
+ decoupled_min_lr ................................ None
104
+ decrease_batch_size_if_needed ................... False
105
+ defer_embedding_wgrad_compute ................... False
106
+ delay_wgrad_compute ............................. False
107
+ deprecated_use_mcore_models ..................... True
108
+ deterministic_mode .............................. False
109
+ dino_bottleneck_size ............................ 256
110
+ dino_freeze_last_layer .......................... 1
111
+ dino_head_hidden_size ........................... 2048
112
+ dino_local_crops_number ......................... 10
113
+ dino_local_img_size ............................. 96
114
+ dino_norm_last_layer ............................ False
115
+ dino_teacher_temp ............................... 0.07
116
+ dino_warmup_teacher_temp ........................ 0.04
117
+ dino_warmup_teacher_temp_epochs ................. 30
118
+ disable_bf16_reduced_precision_matmul ........... False
119
+ disable_mamba_mem_eff_path ...................... False
120
+ disable_straggler_on_startup .................... False
121
+ dist_ckpt_format_deprecated ..................... None
122
+ dist_ckpt_strictness ............................ assume_ok_unexpected
123
+ distribute_saved_activations .................... False
124
+ distributed_backend ............................. nccl
125
+ distributed_timeout_minutes ..................... 10
126
+ embedding_init_method_std ....................... None
127
+ embedding_path .................................. None
128
+ empty_unused_memory_level ....................... 0
129
+ enable_cuda_graph ............................... True
130
+ enable_experimental ............................. False
131
+ enable_ft_package ............................... False
132
+ enable_gloo_process_groups ...................... True
133
+ enable_msc ...................................... True
134
+ enable_one_logger ............................... True
135
+ encoder_num_layers .............................. 16
136
+ encoder_seq_length .............................. 4096
137
+ end_weight_decay ................................ 0.1
138
+ eod_mask_loss ................................... False
139
+ error_injection_rate ............................ 0
140
+ error_injection_type ............................ transient_error
141
+ eval_interval ................................... 1000000
142
+ eval_iters ...................................... 0
143
+ evidence_data_path .............................. None
144
+ exit_duration_in_mins ........................... None
145
+ exit_interval ................................... None
146
+ exit_on_missing_checkpoint ...................... False
147
+ exit_signal_handler ............................. False
148
+ exp_avg_dtype ................................... torch.float32
149
+ exp_avg_sq_dtype ................................ torch.float32
150
+ expert_model_parallel_size ...................... 8
151
+ expert_tensor_parallel_size ..................... 1
152
+ export_force_local_attention .................... False
153
+ export_kd_cfg ................................... None
154
+ export_kd_teacher_ckpt_format ................... None
155
+ export_kd_teacher_load .......................... None
156
+ export_kv_cache_quant ........................... False
157
+ export_legacy_megatron .......................... False
158
+ export_model_type ............................... GPTModel
159
+ export_moe_apply_probs_on_input ................. False
160
+ export_qk_l2_norm ............................... False
161
+ export_quant_cfg ................................ None
162
+ export_real_quant_cfg ........................... None
163
+ export_te_mcore_model ........................... False
164
+ external_cuda_graph ............................. False
165
+ ffn_hidden_size ................................. 14336
166
+ finetune ........................................ False
167
+ finetune_data_split ............................. train
168
+ finetune_hf_dataset ............................. None
169
+ first_last_layers_bf16 .......................... False
170
+ flash_decode .................................... False
171
+ fp16 ............................................ False
172
+ fp16_lm_cross_entropy ........................... False
173
+ fp32_residual_connection ........................ False
174
+ fp8 ............................................. e4m3
175
+ fp8_amax_compute_algo ........................... most_recent
176
+ fp8_amax_history_len ............................ 1
177
+ fp8_interval .................................... 1
178
+ fp8_margin ...................................... 0
179
+ fp8_param_gather ................................ False
180
+ fp8_recipe ...................................... mxfp8
181
+ fp8_wgrad ....................................... True
182
+ fsdp_double_buffer .............................. False
183
+ global_batch_size ............................... 16
184
+ grad_reduce_in_bf16 ............................. False
185
+ gradient_accumulation_fusion .................... True
186
+ gradient_reduce_div_fusion ...................... True
187
+ group_query_attention ........................... True
188
+ head_lr_mult .................................... 1.0
189
+ heterogeneous_layers_config_encoded_json ........ None
190
+ heterogeneous_layers_config_path ................ None
191
+ hidden_dropout .................................. 0.0
192
+ hidden_size ..................................... 4096
193
+ hierarchical_context_parallel_sizes ............. None
194
+ high_priority_stream_groups ..................... []
195
+ hybrid_attention_ratio .......................... 0.0
196
+ hybrid_mlp_ratio ................................ 0.0
197
+ hybrid_override_pattern ......................... None
198
+ hysteresis ...................................... 2
199
+ ict_head_size ................................... None
200
+ ict_load ........................................ None
201
+ img_h ........................................... 224
202
+ img_w ........................................... 224
203
+ indexer_batch_size .............................. 128
204
+ indexer_log_interval ............................ 1000
205
+ inference_batch_times_seqlen_threshold .......... -1
206
+ inference_dynamic_batching ...................... False
207
+ inference_dynamic_batching_buffer_guaranteed_fraction 0.2
208
+ inference_dynamic_batching_buffer_overflow_factor None
209
+ inference_dynamic_batching_buffer_size_gb ....... 40.0
210
+ inference_dynamic_batching_chunk_size ........... 256
211
+ inference_dynamic_batching_max_requests_override None
212
+ inference_dynamic_batching_max_tokens_override .. None
213
+ inference_dynamic_batching_num_cuda_graphs ...... 16
214
+ inference_max_batch_size ........................ 8
215
+ inference_max_seq_length ........................ 2560
216
+ inference_rng_tracker ........................... False
217
+ init_method_std ................................. 0.01
218
+ init_method_xavier_uniform ...................... False
219
+ init_model_with_meta_device ..................... False
220
+ initial_loss_scale .............................. 4294967296
221
+ inprocess_active_world_size ..................... 8
222
+ inprocess_barrier_timeout ....................... 120
223
+ inprocess_completion_timeout .................... 120
224
+ inprocess_empty_cuda_cache ...................... False
225
+ inprocess_granularity ........................... node
226
+ inprocess_hard_timeout .......................... 90
227
+ inprocess_heartbeat_interval .................... 30
228
+ inprocess_heartbeat_timeout ..................... 60
229
+ inprocess_last_call_wait ........................ 1
230
+ inprocess_max_iterations ........................ None
231
+ inprocess_monitor_process_interval .............. 1.0
232
+ inprocess_monitor_thread_interval ............... 1.0
233
+ inprocess_progress_watchdog_interval ............ 1.0
234
+ inprocess_restart ............................... False
235
+ inprocess_soft_timeout .......................... 60
236
+ inprocess_termination_grace_time ................ 1
237
+ is_hybrid_model ................................. False
238
+ iter_per_epoch .................................. 1250
239
+ iterations_to_skip .............................. []
240
+ keep_fp8_transpose_cache_when_using_custom_fsdp . False
241
+ kitchen_config_file ............................. None
242
+ kitchen_recipe_number ........................... None
243
+ kv_channels ..................................... 128
244
+ kv_lora_rank .................................... 32
245
+ lazy_mpu_init ................................... None
246
+ load ............................................ None
247
+ load_main_params_from_ckpt ...................... None
248
+ load_model_opt_format ........................... False
249
+ local_rank ...................................... 0
250
+ log_energy ...................................... False
251
+ log_interval .................................... 1
252
+ log_loss_scale_to_tensorboard ................... True
253
+ log_memory_to_tensorboard ....................... False
254
+ log_num_zeros_in_grad ........................... False
255
+ log_params_norm ................................. False
256
+ log_progress .................................... False
257
+ log_straggler ................................... False
258
+ log_throughput .................................. True
259
+ log_timers_to_tensorboard ....................... False
260
+ log_validation_ppl_to_tensorboard ............... False
261
+ log_world_size_to_tensorboard ................... False
262
+ logging_level ................................... None
263
+ loss_scale ...................................... None
264
+ loss_scale_window ............................... 1000
265
+ lr .............................................. 0.0001
266
+ lr_decay_iters .................................. 8
267
+ lr_decay_samples ................................ None
268
+ lr_decay_style .................................. cosine
269
+ lr_warmup_fraction .............................. None
270
+ lr_warmup_init .................................. 0.0
271
+ lr_warmup_iters ................................. 5
272
+ lr_warmup_samples ............................... 0
273
+ lr_wsd_decay_iters .............................. None
274
+ lr_wsd_decay_samples ............................ None
275
+ lr_wsd_decay_style .............................. exponential
276
+ main_grads_dtype ................................ torch.float32
277
+ main_params_dtype ............................... torch.float32
278
+ make_vocab_size_divisible_by .................... 128
279
+ mamba_head_dim .................................. 64
280
+ mamba_num_groups ................................ 8
281
+ mamba_num_heads ................................. None
282
+ mamba_state_dim ................................. 128
283
+ manual_gc ....................................... False
284
+ manual_gc_eval .................................. True
285
+ manual_gc_interval .............................. 0
286
+ mask_factor ..................................... 1.0
287
+ mask_prob ....................................... 0.15
288
+ mask_type ....................................... random
289
+ masked_softmax_fusion ........................... False
290
+ max_position_embeddings ......................... 32768
291
+ max_tokens_to_oom ............................... 12000
292
+ memory_snapshot_path ............................ snapshot.pickle
293
+ merge_file ...................................... /workspace/data/wikitext/gpt2-merges.txt
294
+ micro_batch_size ................................ 1
295
+ microbatch_group_size_per_vp_stage .............. None
296
+ mid_level_dataset_surplus ....................... 0.005
297
+ min_loss_scale .................................. 1.0
298
+ min_lr .......................................... 1e-05
299
+ mlp_chunks_for_prefill .......................... 1
300
+ mmap_bin_files .................................. True
301
+ mock_data ....................................... False
302
+ moe_apply_probs_on_input ........................ False
303
+ moe_aux_loss_coeff .............................. 0.01
304
+ moe_deepep_num_sms .............................. 20
305
+ moe_echo_enable_random_offloading ............... False
306
+ moe_echo_expert_dispatch_overlap ................ True
307
+ moe_echo_expert_dispatcher_type ................. hybridep
308
+ moe_echo_recompute_expert_dispatch .............. False
309
+ moe_enable_deepep ............................... False
310
+ moe_enable_echo ................................. True
311
+ moe_enable_hybridep ............................. True
312
+ moe_expert_capacity_factor ...................... None
313
+ moe_extended_tp ................................. False
314
+ moe_ffn_hidden_size ............................. 14336
315
+ moe_grouped_gemm ................................ True
316
+ moe_hybridep_num_sms ............................ 16
317
+ moe_input_jitter_eps ............................ None
318
+ moe_layer_freq .................................. 1
319
+ moe_layer_recompute ............................. False
320
+ moe_num_echo_experts ............................ 8
321
+ moe_pad_expert_input_to_capacity ................ False
322
+ moe_per_layer_logging ........................... False
323
+ moe_permute_fusion .............................. False
324
+ moe_received_token_capacity ..................... 2.40625
325
+ moe_router_bias_update_rate ..................... 0.001
326
+ moe_router_dtype ................................ fp32
327
+ moe_router_enable_expert_bias ................... False
328
+ moe_router_force_load_balancing ................. False
329
+ moe_router_group_topk ........................... None
330
+ moe_router_load_balancing_type .................. aux_loss
331
+ moe_router_num_groups ........................... None
332
+ moe_router_padding_for_fp8 ...................... False
333
+ moe_router_pre_softmax .......................... False
334
+ moe_router_score_function ....................... softmax
335
+ moe_router_topk ................................. 2
336
+ moe_router_topk_scaling_factor .................. None
337
+ moe_shared_expert_intermediate_size ............. None
338
+ moe_shared_expert_overlap ....................... False
339
+ moe_token_dispatcher_type ....................... flex
340
+ moe_token_drop_policy ........................... probs
341
+ moe_upcycling_granularity ....................... 1
342
+ moe_use_device_initiated_grouped_gemm ........... True
343
+ moe_use_legacy_grouped_gemm ..................... False
344
+ moe_use_upcycling ............................... False
345
+ moe_z_loss_coeff ................................ None
346
+ mrope_section ................................... None
347
+ mscale .......................................... 1.0
348
+ mscale_all_dim .................................. 0.0
349
+ mtp_loss_scaling_factor ......................... 0.1
350
+ mtp_num_layers .................................. None
351
+ multi_latent_attention .......................... False
352
+ nccl_all_reduce_for_prefill ..................... False
353
+ nccl_communicator_config_path ................... None
354
+ nccl_ub ......................................... False
355
+ no_load_optim ................................... None
356
+ no_load_rng ..................................... None
357
+ no_persist_layer_norm ........................... False
358
+ no_rope_freq .................................... None
359
+ no_save_optim ................................... None
360
+ no_save_rng ..................................... None
361
+ non_persistent_ckpt_type ........................ None
362
+ non_persistent_global_ckpt_dir .................. None
363
+ non_persistent_local_ckpt_algo .................. fully_parallel
364
+ non_persistent_local_ckpt_dir ................... None
365
+ non_persistent_save_interval .................... None
366
+ norm_epsilon .................................... 1e-05
367
+ normalization ................................... RMSNorm
368
+ num_attention_heads ............................. 32
369
+ num_channels .................................... 3
370
+ num_classes ..................................... 1000
371
+ num_dataset_builder_threads ..................... 1
372
+ num_distributed_optimizer_instances ............. 1
373
+ num_experts ..................................... 8
374
+ num_layers ...................................... 16
375
+ num_layers_at_end_in_bf16 ....................... 1
376
+ num_layers_at_start_in_bf16 ..................... 1
377
+ num_layers_per_virtual_pipeline_stage ........... None
378
+ num_query_groups ................................ 8
379
+ num_virtual_stages_per_pipeline_rank ............ None
380
+ num_workers ..................................... 2
381
+ object_storage_cache_path ....................... None
382
+ one_logger_async ................................ False
383
+ one_logger_project .............................. megatron-lm
384
+ one_logger_run_name ............................. None
385
+ onnx_safe ....................................... None
386
+ openai_gelu ..................................... False
387
+ optimizer ....................................... adam
388
+ optimizer_cpu_offload ........................... False
389
+ optimizer_offload_fraction ...................... 1.0
390
+ output_bert_embeddings .......................... False
391
+ overlap_cpu_optimizer_d2h_h2d ................... False
392
+ overlap_grad_reduce ............................. True
393
+ overlap_moe_expert_parallel_comm ................ False
394
+ overlap_p2p_comm ................................ False
395
+ overlap_p2p_comm_warmup_flush ................... False
396
+ overlap_param_gather ............................ True
397
+ overlap_param_gather_with_optimizer_step ........ False
398
+ override_opt_param_scheduler .................... False
399
+ params_dtype .................................... torch.bfloat16
400
+ patch_dim ....................................... 16
401
+ per_split_data_args_path ........................ None
402
+ perform_initialization .......................... True
403
+ pin_cpu_grads ................................... True
404
+ pin_cpu_params .................................. True
405
+ pipeline_model_parallel_comm_backend ............ None
406
+ pipeline_model_parallel_layout .................. None
407
+ pipeline_model_parallel_size .................... 1
408
+ position_embedding_type ......................... rope
409
+ pretrained_checkpoint ........................... None
410
+ profile ......................................... False
411
+ profile_ranks ................................... [0]
412
+ profile_step_end ................................ 12
413
+ profile_step_start .............................. 10
414
+ q_lora_rank ..................................... None
415
+ qk_head_dim ..................................... 128
416
+ qk_l2_norm ...................................... False
417
+ qk_layernorm .................................... False
418
+ qk_pos_emb_head_dim ............................. 64
419
+ query_in_block_prob ............................. 0.1
420
+ rampup_batch_size ............................... None
421
+ rank ............................................ 0
422
+ recompute_granularity ........................... None
423
+ recompute_method ................................ None
424
+ recompute_modules ............................... None
425
+ recompute_num_layers ............................ None
426
+ record_memory_history ........................... False
427
+ relative_attention_max_distance ................. 128
428
+ relative_attention_num_buckets .................. 32
429
+ replication ..................................... False
430
+ replication_factor .............................. 2
431
+ replication_jump ................................ None
432
+ rerun_mode ...................................... validate_results
433
+ reset_attention_mask ............................ False
434
+ reset_position_ids .............................. False
435
+ result_rejected_tracker_filename ................ None
436
+ retriever_report_topk_accuracies ................ []
437
+ retriever_score_scaling ......................... False
438
+ retriever_seq_length ............................ 256
439
+ retro_add_retriever ............................. False
440
+ retro_attention_gate ............................ 1
441
+ retro_cyclic_train_iters ........................ None
442
+ retro_encoder_attention_dropout ................. 0.1
443
+ retro_encoder_hidden_dropout .................... 0.1
444
+ retro_encoder_layers ............................ 2
445
+ retro_num_neighbors ............................. 2
446
+ retro_num_retrieved_chunks ...................... 2
447
+ retro_project_dir ............................... None
448
+ retro_verify_neighbor_count ..................... True
449
+ reuse_grad_buf_for_mxfp8_param_ag ............... False
450
+ rope_scaling_factor ............................. 8.0
451
+ rope_type ....................................... None
452
+ rotary_base ..................................... 1000000
453
+ rotary_interleaved .............................. False
454
+ rotary_percent .................................. 1.0
455
+ rotary_scaling_factor ........................... 1.0
456
+ rotary_seq_len_interpolation_factor ............. None
457
+ run_workload_inspector_server ................... False
458
+ sample_rate ..................................... 1.0
459
+ save ............................................ None
460
+ save_interval ................................... None
461
+ save_retain_interval ............................ None
462
+ scatter_gather_tensors_in_pipeline .............. True
463
+ seed ............................................ 1234
464
+ seq_length ...................................... 4096
465
+ sequence_parallel ............................... False
466
+ sft ............................................. False
467
+ sft_tokenizer_prompt_format ..................... nemotron-h-aligned
468
+ sgd_momentum .................................... 0.9
469
+ short_seq_prob .................................. 0.1
470
+ skip_train ...................................... False
471
+ skipped_train_samples ........................... 0
472
+ spec ............................................ None
473
+ split ........................................... 100,0,0
474
+ squared_relu .................................... False
475
+ start_weight_decay .............................. 0.1
476
+ straggler_ctrlr_port ............................ 65535
477
+ straggler_minmax_count .......................... 1
478
+ suggested_communication_unit_size ............... None
479
+ swiglu .......................................... True
480
+ swin_backbone_type .............................. tiny
481
+ symmetric_ar_type ............................... None
482
+ te_rng_tracker .................................. True
483
+ tensor_model_parallel_size ...................... 1
484
+ tensorboard_dir ................................. None
485
+ tensorboard_log_interval ........................ 1
486
+ tensorboard_queue_size .......................... 1000
487
+ test_data_path .................................. None
488
+ test_mode ....................................... False
489
+ tiktoken_num_special_tokens ..................... 1000
490
+ tiktoken_pattern ................................ None
491
+ tiktoken_special_tokens ......................... None
492
+ timing_log_level ................................ 0
493
+ timing_log_option ............................... minmax
494
+ titles_data_path ................................ None
495
+ tokenizer_model ................................. None
496
+ tokenizer_type .................................. GPT2BPETokenizer
497
+ torch_fsdp2_reshard_after_forward ............... True
498
+ tp_comm_bootstrap_backend ....................... nccl
499
+ tp_comm_bulk_dgrad .............................. True
500
+ tp_comm_bulk_wgrad .............................. True
501
+ tp_comm_overlap ................................. False
502
+ tp_comm_overlap_ag .............................. True
503
+ tp_comm_overlap_cfg ............................. None
504
+ tp_comm_overlap_rs .............................. True
505
+ tp_comm_overlap_rs_dgrad ........................ False
506
+ tp_comm_split_ag ................................ True
507
+ tp_comm_split_rs ................................ True
508
+ train_data_path ................................. None
509
+ train_iters ..................................... 12
510
+ train_samples ................................... None
511
+ train_sync_interval ............................. None
512
+ transformer_impl ................................ transformer_engine
513
+ transformer_pipeline_model_parallel_size ........ 1
514
+ untie_embeddings_and_output_weights ............. True
515
+ use_checkpoint_args ............................. False
516
+ use_checkpoint_opt_param_scheduler .............. False
517
+ use_cpu_initialization .......................... None
518
+ use_custom_fsdp ................................. False
519
+ use_dist_ckpt ................................... True
520
+ use_dist_ckpt_deprecated ........................ False
521
+ use_distributed_optimizer ....................... True
522
+ use_flash_attn .................................. False
523
+ use_legacy_models ............................... False
524
+ use_mp_args_from_checkpoint_args ................ False
525
+ use_one_sent_docs ............................... False
526
+ use_persistent_ckpt_worker ...................... False
527
+ use_precision_aware_optimizer ................... False
528
+ use_pytorch_profiler ............................ False
529
+ use_ring_exchange_p2p ........................... False
530
+ use_rope_scaling ................................ False
531
+ use_rotary_position_embeddings .................. False
532
+ use_sharp ....................................... False
533
+ use_tokenizer_model_from_checkpoint_args ........ True
534
+ use_torch_fsdp2 ................................. False
535
+ use_torch_optimizer_for_cpu_offload ............. False
536
+ use_tp_pp_dp_mapping ............................ False
537
+ v_head_dim ...................................... 128
538
+ valid_data_path ................................. None
539
+ variable_seq_lengths ............................ False
540
+ virtual_pipeline_model_parallel_size ............ None
541
+ vision_backbone_type ............................ vit
542
+ vision_pretraining .............................. False
543
+ vision_pretraining_type ......................... classify
544
+ vocab_extra_ids ................................. 0
545
+ vocab_file ...................................... /workspace/data/wikitext/gpt2-vocab.json
546
+ vocab_size ...................................... None
547
+ wandb_exp_name ..................................
548
+ wandb_project ...................................
549
+ wandb_save_dir ..................................
550
+ weight_decay .................................... 0.1
551
+ weight_decay_incr_style ......................... constant
552
+ wgrad_deferral_limit ............................ 0
553
+ world_size ...................................... 8
554
+ yaml_cfg ........................................ None
555
+ -------------------- end of arguments ---------------------
556
+ INFO:megatron.core.num_microbatches_calculator:setting number of microbatches to constant 2
557
+ > building GPT2BPETokenizer tokenizer ...
558
+ > padded vocab (size: 50257) with 47 dummy tokens (new size: 50304)
559
+ WARNING:megatron.core.rerun_state_machine:RerunStateMachine initialized in mode RerunMode.VALIDATE_RESULTS
560
+ > initializing torch distributed ...
561
+ /workspace/Megatron-MoE-Echo/megatron/training/arguments.py:831: UserWarning: Disabling sequence parallelism because tensor model parallelism is disabled
562
+ warnings.warn("Disabling sequence parallelism because tensor model parallelism is disabled")
563
+ /workspace/Megatron-MoE-Echo/megatron/training/arguments.py:831: UserWarning: Disabling sequence parallelism because tensor model parallelism is disabled
564
+ warnings.warn("Disabling sequence parallelism because tensor model parallelism is disabled")
565
+ /workspace/Megatron-MoE-Echo/megatron/training/arguments.py:831: UserWarning: Disabling sequence parallelism because tensor model parallelism is disabled
566
+ warnings.warn("Disabling sequence parallelism because tensor model parallelism is disabled")
567
+ /workspace/Megatron-MoE-Echo/megatron/training/arguments.py:831: UserWarning: Disabling sequence parallelism because tensor model parallelism is disabled
568
+ warnings.warn("Disabling sequence parallelism because tensor model parallelism is disabled")
569
+ WARNING: one_logger package is required to enable e2e metrics tracking. please go to https://confluence.nvidia.com/display/MLWFO/Package+Repositories for details to install it
570
+ /workspace/Megatron-MoE-Echo/megatron/training/arguments.py:831: UserWarning: Disabling sequence parallelism because tensor model parallelism is disabled
571
+ warnings.warn("Disabling sequence parallelism because tensor model parallelism is disabled")
572
+ /workspace/Megatron-MoE-Echo/megatron/training/arguments.py:831: UserWarning: Disabling sequence parallelism because tensor model parallelism is disabled
573
+ warnings.warn("Disabling sequence parallelism because tensor model parallelism is disabled")
574
+ /workspace/Megatron-MoE-Echo/megatron/training/arguments.py:831: UserWarning: Disabling sequence parallelism because tensor model parallelism is disabled
575
+ warnings.warn("Disabling sequence parallelism because tensor model parallelism is disabled")
576
+ [Gloo] Rank [Gloo] Rank 2[Gloo] Rank [Gloo] Rank [Gloo] Rank is connected to 01475 is connected to is connected to is connected to peer ranks. is connected to 777Expected number of connected peer ranks is : 7 peer ranks. peer ranks. peer ranks. 7 peer ranks. Expected number of connected peer ranks is : Expected number of connected peer ranks is : Expected number of connected peer ranks is :
577
+ Expected number of connected peer ranks is : 7777
578
+
579
+
580
+
581
+ [Gloo] Rank 3 is connected to 7 peer ranks. Expected number of connected peer ranks is : 7
582
+ [Gloo] Rank [Gloo] Rank 7 is connected to 67 is connected to peer ranks. 7Expected number of connected peer ranks is : peer ranks. 7Expected number of connected peer ranks is :
583
+ 7
584
+ [Gloo] Rank 0 is connected to 7 peer ranks. Expected number of connected peer ranks is : 7
585
+ [Gloo] Rank 1 is connected to 7 peer ranks. Expected number of connected peer ranks is : 7
586
+ [Gloo] Rank 2 is connected to 7 peer ranks. Expected number of connected peer ranks is : 7
587
+ [Gloo] Rank 3 is connected to 7 peer ranks. Expected number of connected peer ranks is : 7
588
+ [Gloo] Rank 4 is connected to 7 peer ranks. Expected number of connected peer ranks is : 7
589
+ [Gloo] Rank 5 is connected to 7 peer ranks. Expected number of connected peer ranks is : 7
590
+ [Gloo] Rank 7 is connected to 7 peer ranks. Expected number of connected peer ranks is : 7
591
+ [Gloo] Rank 6 is connected to 7 peer ranks. Expected number of connected peer ranks is : 7
592
+ [Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
593
+ > initialized tensor model parallel with size 1
594
+ > initialized pipeline model parallel with size 1
595
+ > setting random seeds to 1234 ...
596
+ [Gloo] Rank [Gloo] Rank 0 is connected to 00 is connected to peer ranks. 0Expected number of connected peer ranks is : peer ranks. 0Expected number of connected peer ranks is :
597
+ 0
598
+ [Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
599
+ [Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
600
+ [Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
601
+ [Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
602
+ [Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
603
+ [rank7]:[W830 06:46:03.578790501 ProcessGroupNCCL.cpp:4789] [PG ID 0 PG GUID 0 Rank 7] using GPU 7 as device used by this process is currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect. You can specify device_id in init_process_group() to force use of a particular device.
604
+ [rank2]:[W830 06:46:03.578807856 ProcessGroupNCCL.cpp:4789] [PG ID 0 PG GUID 0 Rank 2] using GPU 2 as device used by this process is currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect. You can specify device_id in init_process_group() to force use of a particular device.
605
+ [rank4]:[W830 06:46:03.578834248 ProcessGroupNCCL.cpp:4789] [PG ID 0 PG GUID 0 Rank 4] using GPU 4 as device used by this process is currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect. You can specify device_id in init_process_group() to force use of a particular device.
606
+ [rank1]:[W830 06:46:03.578850380 ProcessGroupNCCL.cpp:4789] [PG ID 0 PG GUID 0 Rank 1] using GPU 1 as device used by this process is currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect. You can specify device_id in init_process_group() to force use of a particular device.
607
+ [rank5]:[W830 06:46:03.578865697 ProcessGroupNCCL.cpp:4789] [PG ID 0 PG GUID 0 Rank 5] using GPU 5 as device used by this process is currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect. You can specify device_id in init_process_group() to force use of a particular device.
608
+ [rank3]:[W830 06:46:03.578912375 ProcessGroupNCCL.cpp:4789] [PG ID 0 PG GUID 0 Rank 3] using GPU 3 as device used by this process is currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect. You can specify device_id in init_process_group() to force use of a particular device.
609
+ [rank6]:[W830 06:46:03.578911167 ProcessGroupNCCL.cpp:4789] [PG ID 0 PG GUID 0 Rank 6] using GPU 6 as device used by this process is currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect. You can specify device_id in init_process_group() to force use of a particular device.
610
+ > compiling dataset index builder ...
611
+ make: Entering directory '/workspace/Megatron-MoE-Echo/megatron/core/datasets'
612
+ make: Nothing to be done for 'default'.
613
+ make: Leaving directory '/workspace/Megatron-MoE-Echo/megatron/core/datasets'
614
+ >>> done with dataset index builder. Compilation time: 0.034 seconds
615
+ WARNING: constraints for invoking optimized fused softmax kernel are not met. We default back to unfused kernel invocations.
616
+ > compiling and loading fused kernels ...
617
+ [rank0]:[W830 06:46:03.730419083 ProcessGroupNCCL.cpp:4789] [PG ID 0 PG GUID 0 Rank 0] using GPU 0 as device used by this process is currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect. You can specify device_id in init_process_group() to force use of a particular device.
618
+ >>> done with compiling and loading fused kernels. Compilation time: 3.842 seconds
619
+ time to initialize megatron (seconds): 7.199
620
+ [after megatron is initialized] datetime: 2026-08-30 06:46:09
621
+ building GPT model ...
622
+ /usr/local/lib/python3.12/dist-packages/transformer_engine/pytorch/cpu_offload.py:708: DeprecationWarning: Offloading weights is deprecated. Using offload_weights=True does not have any effect.
623
+ warnings.warn(
624
+ /usr/local/lib/python3.12/dist-packages/transformer_engine/pytorch/cpu_offload.py:708: DeprecationWarning: Offloading weights is deprecated. Using offload_weights=True does not have any effect.
625
+ warnings.warn(
626
+ /usr/local/lib/python3.12/dist-packages/transformer_engine/pytorch/cpu_offload.py:708: DeprecationWarning: Offloading weights is deprecated. Using offload_weights=True does not have any effect.
627
+ warnings.warn(
628
+ /usr/local/lib/python3.12/dist-packages/transformer_engine/pytorch/cpu_offload.py:708: DeprecationWarning: Offloading weights is deprecated. Using offload_weights=True does not have any effect.
629
+ warnings.warn(
630
+ /usr/local/lib/python3.12/dist-packages/transformer_engine/pytorch/cpu_offload.py:708: DeprecationWarning: Offloading weights is deprecated. Using offload_weights=True does not have any effect.
631
+ warnings.warn(
632
+ /usr/local/lib/python3.12/dist-packages/transformer_engine/pytorch/cpu_offload.py:708: DeprecationWarning: Offloading weights is deprecated. Using offload_weights=True does not have any effect.
633
+ warnings.warn(
634
+ /usr/local/lib/python3.12/dist-packages/transformer_engine/pytorch/cpu_offload.py:708: DeprecationWarning: Offloading weights is deprecated. Using offload_weights=True does not have any effect.
635
+ warnings.warn(
636
+ /usr/local/lib/python3.12/dist-packages/transformer_engine/pytorch/cpu_offload.py:708: DeprecationWarning: Offloading weights is deprecated. Using offload_weights=True does not have any effect.
637
+ warnings.warn(
638
+ > number of parameters on (tensor, pipeline) model parallel rank (0, 0): 3902410752
639
+ INFO:megatron.core.distributed.distributed_data_parallel:Setting up DistributedDataParallel with config DistributedDataParallelConfig(grad_reduce_in_fp32=True, overlap_grad_reduce=True, overlap_param_gather=True, align_param_gather=False, use_distributed_optimizer=True, num_distributed_optimizer_instances=1, check_for_nan_in_grad=False, check_for_large_grads=False, bucket_size=40000000, pad_buckets_for_high_nccl_busbw=False, average_in_collective=False, fp8_param_gather=False, reuse_grad_buf_for_mxfp8_param_ag=False, use_custom_fsdp=False, data_parallel_sharding_strategy='no_shard', gradient_reduce_div_fusion=True, suggested_communication_unit_size=None, preserve_fp32_weights=True, keep_fp8_transpose_cache_when_using_custom_fsdp=False, nccl_ub=False, fsdp_double_buffer=False)
640
+ INFO:megatron.core.distributed.param_and_grad_buffer:Number of buckets for gradient all-reduce / reduce-scatter: 18
641
+ Params for bucket 1 (206045184 elements, 206045184 padded size):
642
+ module.output_layer.weight
643
+ Params for bucket 2 (41988096 elements, 41988096 padded size):
644
+ module.decoder.final_layernorm.weight
645
+ module.decoder.layers.15.mlp.router.weight
646
+ module.decoder.layers.15.self_attention.linear_qkv.layer_norm_weight
647
+ module.decoder.layers.15.self_attention.linear_proj.weight
648
+ module.decoder.layers.15.pre_mlp_layernorm.weight
649
+ module.decoder.layers.15.self_attention.linear_qkv.weight
650
+ Params for bucket 3 (41984000 elements, 41984000 padded size):
651
+ module.decoder.layers.14.pre_mlp_layernorm.weight
652
+ module.decoder.layers.14.self_attention.linear_qkv.weight
653
+ module.decoder.layers.14.mlp.router.weight
654
+ module.decoder.layers.14.self_attention.linear_proj.weight
655
+ module.decoder.layers.14.self_attention.linear_qkv.layer_norm_weight
656
+ Params for bucket 4 (41984000 elements, 41984000 padded size):
657
+ module.decoder.layers.13.self_attention.linear_qkv.weight
658
+ module.decoder.layers.13.pre_mlp_layernorm.weight
659
+ module.decoder.layers.13.mlp.router.weight
660
+ module.decoder.layers.13.self_attention.linear_proj.weight
661
+ module.decoder.layers.13.self_attention.linear_qkv.layer_norm_weight
662
+ Params for bucket 5 (41984000 elements, 41984000 padded size):
663
+ module.decoder.layers.12.mlp.router.weight
664
+ module.decoder.layers.12.self_attention.linear_proj.weight
665
+ module.decoder.layers.12.self_attention.linear_qkv.layer_norm_weight
666
+ module.decoder.layers.12.self_attention.linear_qkv.weight
667
+ module.decoder.layers.12.pre_mlp_layernorm.weight
668
+ Params for bucket 6 (41984000 elements, 41984000 padded size):
669
+ module.decoder.layers.11.self_attention.linear_qkv.weight
670
+ module.decoder.layers.11.pre_mlp_layernorm.weight
671
+ module.decoder.layers.11.mlp.router.weight
672
+ module.decoder.layers.11.self_attention.linear_qkv.layer_norm_weight
673
+ module.decoder.layers.11.self_attention.linear_proj.weight
674
+ Params for bucket 7 (41984000 elements, 41984000 padded size):
675
+ module.decoder.layers.10.mlp.router.weight
676
+ module.decoder.layers.10.self_attention.linear_proj.weight
677
+ module.decoder.layers.10.self_attention.linear_qkv.layer_norm_weight
678
+ module.decoder.layers.10.self_attention.linear_qkv.weight
679
+ module.decoder.layers.10.pre_mlp_layernorm.weight
680
+ Params for bucket 8 (41984000 elements, 41984000 padded size):
681
+ module.decoder.layers.9.self_attention.linear_qkv.weight
682
+ module.decoder.layers.9.pre_mlp_layernorm.weight
683
+ module.decoder.layers.9.mlp.router.weight
684
+ module.decoder.layers.9.self_attention.linear_proj.weight
685
+ module.decoder.layers.9.self_attention.linear_qkv.layer_norm_weight
686
+ Params for bucket 9 (41984000 elements, 41984000 padded size):
687
+ module.decoder.layers.8.self_attention.linear_qkv.weight
688
+ module.decoder.layers.8.pre_mlp_layernorm.weight
689
+ module.decoder.layers.8.mlp.router.weight
690
+ module.decoder.layers.8.self_attention.linear_qkv.layer_norm_weight
691
+ module.decoder.layers.8.self_attention.linear_proj.weight
692
+ Params for bucket 10 (41984000 elements, 41984000 padded size):
693
+ module.decoder.layers.7.self_attention.linear_qkv.weight
694
+ module.decoder.layers.7.pre_mlp_layernorm.weight
695
+ module.decoder.layers.7.mlp.router.weight
696
+ module.decoder.layers.7.self_attention.linear_proj.weight
697
+ module.decoder.layers.7.self_attention.linear_qkv.layer_norm_weight
698
+ Params for bucket 11 (41984000 elements, 41984000 padded size):
699
+ module.decoder.layers.6.mlp.router.weight
700
+ module.decoder.layers.6.self_attention.linear_qkv.layer_norm_weight
701
+ module.decoder.layers.6.self_attention.linear_proj.weight
702
+ module.decoder.layers.6.self_attention.linear_qkv.weight
703
+ module.decoder.layers.6.pre_mlp_layernorm.weight
704
+ Params for bucket 12 (41984000 elements, 41984000 padded size):
705
+ module.decoder.layers.5.pre_mlp_layernorm.weight
706
+ module.decoder.layers.5.self_attention.linear_qkv.layer_norm_weight
707
+ module.decoder.layers.5.mlp.router.weight
708
+ module.decoder.layers.5.self_attention.linear_qkv.weight
709
+ module.decoder.layers.5.self_attention.linear_proj.weight
710
+ Params for bucket 13 (41984000 elements, 41984000 padded size):
711
+ module.decoder.layers.4.pre_mlp_layernorm.weight
712
+ module.decoder.layers.4.self_attention.linear_qkv.weight
713
+ module.decoder.layers.4.mlp.router.weight
714
+ module.decoder.layers.4.self_attention.linear_proj.weight
715
+ module.decoder.layers.4.self_attention.linear_qkv.layer_norm_weight
716
+ Params for bucket 14 (41984000 elements, 41984000 padded size):
717
+ module.decoder.layers.3.self_attention.linear_proj.weight
718
+ module.decoder.layers.3.self_attention.linear_qkv.layer_norm_weight
719
+ module.decoder.layers.3.mlp.router.weight
720
+ module.decoder.layers.3.pre_mlp_layernorm.weight
721
+ module.decoder.layers.3.self_attention.linear_qkv.weight
722
+ Params for bucket 15 (41984000 elements, 41984000 padded size):
723
+ module.decoder.layers.2.mlp.router.weight
724
+ module.decoder.layers.2.self_attention.linear_qkv.layer_norm_weight
725
+ module.decoder.layers.2.self_attention.linear_qkv.weight
726
+ module.decoder.layers.2.pre_mlp_layernorm.weight
727
+ module.decoder.layers.2.self_attention.linear_proj.weight
728
+ Params for bucket 16 (41984000 elements, 41984000 padded size):
729
+ module.decoder.layers.1.mlp.router.weight
730
+ module.decoder.layers.1.self_attention.linear_qkv.layer_norm_weight
731
+ module.decoder.layers.1.self_attention.linear_proj.weight
732
+ module.decoder.layers.1.pre_mlp_layernorm.weight
733
+ module.decoder.layers.1.self_attention.linear_qkv.weight
734
+ Params for bucket 17 (41984000 elements, 41984000 padded size):
735
+ module.decoder.layers.0.mlp.router.weight
736
+ module.decoder.layers.0.pre_mlp_layernorm.weight
737
+ module.decoder.layers.0.self_attention.linear_proj.weight
738
+ module.decoder.layers.0.self_attention.linear_qkv.weight
739
+ module.decoder.layers.0.self_attention.linear_qkv.layer_norm_weight
740
+ Params for bucket 18 (206045184 elements, 206045184 padded size):
741
+ module.embedding.word_embeddings.weight
742
+ INFO:megatron.core.distributed.param_and_grad_buffer:Number of buckets for gradient all-reduce / reduce-scatter: 32
743
+ Params for bucket 1 (58720256 elements, 58720256 padded size):
744
+ module.decoder.layers.15.mlp.experts.linear_fc2.weight0
745
+ Params for bucket 2 (117440512 elements, 117440512 padded size):
746
+ module.decoder.layers.15.mlp.experts.linear_fc1.weight0
747
+ Params for bucket 3 (58720256 elements, 58720256 padded size):
748
+ module.decoder.layers.14.mlp.experts.linear_fc2.weight0
749
+ Params for bucket 4 (117440512 elements, 117440512 padded size):
750
+ module.decoder.layers.14.mlp.experts.linear_fc1.weight0
751
+ Params for bucket 5 (58720256 elements, 58720256 padded size):
752
+ module.decoder.layers.13.mlp.experts.linear_fc2.weight0
753
+ Params for bucket 6 (117440512 elements, 117440512 padded size):
754
+ module.decoder.layers.13.mlp.experts.linear_fc1.weight0
755
+ Params for bucket 7 (58720256 elements, 58720256 padded size):
756
+ module.decoder.layers.12.mlp.experts.linear_fc2.weight0
757
+ Params for bucket 8 (117440512 elements, 117440512 padded size):
758
+ module.decoder.layers.12.mlp.experts.linear_fc1.weight0
759
+ Params for bucket 9 (58720256 elements, 58720256 padded size):
760
+ module.decoder.layers.11.mlp.experts.linear_fc2.weight0
761
+ Params for bucket 10 (117440512 elements, 117440512 padded size):
762
+ module.decoder.layers.11.mlp.experts.linear_fc1.weight0
763
+ Params for bucket 11 (58720256 elements, 58720256 padded size):
764
+ module.decoder.layers.10.mlp.experts.linear_fc2.weight0
765
+ Params for bucket 12 (117440512 elements, 117440512 padded size):
766
+ module.decoder.layers.10.mlp.experts.linear_fc1.weight0
767
+ Params for bucket 13 (58720256 elements, 58720256 padded size):
768
+ module.decoder.layers.9.mlp.experts.linear_fc2.weight0
769
+ Params for bucket 14 (117440512 elements, 117440512 padded size):
770
+ module.decoder.layers.9.mlp.experts.linear_fc1.weight0
771
+ Params for bucket 15 (58720256 elements, 58720256 padded size):
772
+ module.decoder.layers.8.mlp.experts.linear_fc2.weight0
773
+ Params for bucket 16 (117440512 elements, 117440512 padded size):
774
+ module.decoder.layers.8.mlp.experts.linear_fc1.weight0
775
+ Params for bucket 17 (58720256 elements, 58720256 padded size):
776
+ module.decoder.layers.7.mlp.experts.linear_fc2.weight0
777
+ Params for bucket 18 (117440512 elements, 117440512 padded size):
778
+ module.decoder.layers.7.mlp.experts.linear_fc1.weight0
779
+ Params for bucket 19 (58720256 elements, 58720256 padded size):
780
+ module.decoder.layers.6.mlp.experts.linear_fc2.weight0
781
+ Params for bucket 20 (117440512 elements, 117440512 padded size):
782
+ module.decoder.layers.6.mlp.experts.linear_fc1.weight0
783
+ Params for bucket 21 (58720256 elements, 58720256 padded size):
784
+ module.decoder.layers.5.mlp.experts.linear_fc2.weight0
785
+ Params for bucket 22 (117440512 elements, 117440512 padded size):
786
+ module.decoder.layers.5.mlp.experts.linear_fc1.weight0
787
+ Params for bucket 23 (58720256 elements, 58720256 padded size):
788
+ module.decoder.layers.4.mlp.experts.linear_fc2.weight0
789
+ Params for bucket 24 (117440512 elements, 117440512 padded size):
790
+ module.decoder.layers.4.mlp.experts.linear_fc1.weight0
791
+ Params for bucket 25 (58720256 elements, 58720256 padded size):
792
+ module.decoder.layers.3.mlp.experts.linear_fc2.weight0
793
+ Params for bucket 26 (117440512 elements, 117440512 padded size):
794
+ module.decoder.layers.3.mlp.experts.linear_fc1.weight0
795
+ Params for bucket 27 (58720256 elements, 58720256 padded size):
796
+ module.decoder.layers.2.mlp.experts.linear_fc2.weight0
797
+ Params for bucket 28 (117440512 elements, 117440512 padded size):
798
+ module.decoder.layers.2.mlp.experts.linear_fc1.weight0
799
+ Params for bucket 29 (58720256 elements, 58720256 padded size):
800
+ module.decoder.layers.1.mlp.experts.linear_fc2.weight0
801
+ Params for bucket 30 (117440512 elements, 117440512 padded size):
802
+ module.decoder.layers.1.mlp.experts.linear_fc1.weight0
803
+ Params for bucket 31 (58720256 elements, 58720256 padded size):
804
+ module.decoder.layers.0.mlp.experts.linear_fc2.weight0
805
+ Params for bucket 32 (117440512 elements, 117440512 padded size):
806
+ module.decoder.layers.0.mlp.experts.linear_fc1.weight0
807
+ INFO:megatron.core.optimizer:Setting up optimizer with config OptimizerConfig(optimizer='adam', lr=0.0001, min_lr=1e-05, decoupled_lr=None, decoupled_min_lr=None, weight_decay=0.1, fp8_recipe='mxfp8', fp16=False, bf16=True, reuse_grad_buf_for_mxfp8_param_ag=False, params_dtype=torch.bfloat16, use_precision_aware_optimizer=False, store_param_remainders=True, main_grads_dtype=torch.float32, main_params_dtype=torch.float32, exp_avg_dtype=torch.float32, exp_avg_sq_dtype=torch.float32, loss_scale=None, initial_loss_scale=4294967296, min_loss_scale=1.0, loss_scale_window=1000, hysteresis=2, adam_beta1=0.9, adam_beta2=0.999, adam_eps=1e-08, sgd_momentum=0.9, use_distributed_optimizer=True, overlap_param_gather_with_optimizer_step=False, optimizer_cpu_offload=False, optimizer_offload_fraction=1.0, use_torch_optimizer_for_cpu_offload=False, overlap_cpu_optimizer_d2h_h2d=False, pin_cpu_grads=True, pin_cpu_params=True, clip_grad=1.0, log_num_zeros_in_grad=False, barrier_with_L1_time=True, timers=<megatron.core.timers.Timers object at 0x73a55704c740>, config_logger_dir='')
808
+ INFO:megatron.core.optimizer_param_scheduler:> learning rate decay style: cosine
809
+ [after model, optimizer, and learning rate scheduler are built] datetime: 2026-08-30 06:46:09
810
+ > building train, validation, and test datasets ...
811
+ > datasets target sizes (minimum size):
812
+ train: 192
813
+ validation: 0
814
+ test: 0
815
+ INFO:megatron.core.datasets.blended_megatron_dataset_config:Let split_matrix = [(0, 1.0), None, None]
816
+ > building train, validation, and test datasets for GPT ...
817
+ INFO:megatron.core.datasets.blended_megatron_dataset_builder:Building GPTDataset splits with sizes=(192, 0, 0) and config=GPTDatasetConfig(random_seed=1234, sequence_length=4096, blend=(['/workspace/data/wikitext/wikitext103_gpt2_text_document'], None), blend_per_split=None, split='100,0,0', split_matrix=[(0, 1.0), None, None], num_dataset_builder_threads=1, path_to_cache=None, mmap_bin_files=True, mock=False, tokenizer=<megatron.training.tokenizer.tokenizer._GPT2BPETokenizer object at 0x73a55892f440>, mid_level_dataset_surplus=0.005, reset_position_ids=False, reset_attention_mask=False, eod_mask_loss=False, create_attention_mask=True, drop_last_partial_validation_sequence=True, add_extra_token_to_sequence=True, object_storage_cache_path=None)
818
+ INFO:megatron.core.datasets.indexed_dataset:Load the _IndexReader from /workspace/data/wikitext/wikitext103_gpt2_text_document.idx
819
+ INFO:megatron.core.datasets.indexed_dataset: Extract the sequence lengths
820
+ INFO:megatron.core.datasets.indexed_dataset: Extract the sequence pointers
821
+ INFO:megatron.core.datasets.indexed_dataset: Extract the document indices
822
+ INFO:megatron.core.datasets.indexed_dataset:> total number of sequences: 1165029
823
+ INFO:megatron.core.datasets.indexed_dataset:> total number of documents: 1801350
824
+ INFO:megatron.core.datasets.gpt_dataset:Load the GPTDataset train indices
825
+ INFO:megatron.core.datasets.gpt_dataset: Load the document index from 44ed1ee5bcb9e5b030387b8d27a6c406-GPTDataset-train-document_index.npy
826
+ INFO:megatron.core.datasets.gpt_dataset: Load the sample index from 44ed1ee5bcb9e5b030387b8d27a6c406-GPTDataset-train-sample_index.npy
827
+ INFO:megatron.core.datasets.gpt_dataset: Load the shuffle index from 44ed1ee5bcb9e5b030387b8d27a6c406-GPTDataset-train-shuffle_index.npy
828
+ INFO:megatron.core.datasets.gpt_dataset:> total number of samples: 29073
829
+ > finished creating GPT datasets ...
830
+ [after dataloaders are built] datetime: 2026-08-30 06:46:09
831
+ done with setup ...
832
+ (min, max) time across ranks (ms):
833
+ model-and-optimizer-setup ......................: (204.86, 207.92)
834
+ train/valid/test-data-iterators-setup ..........: (65.32, 74.72)
835
+ training ...
836
+ Overwriting rerun_state_machine.current_iteration from -1 to 0...
837
+ [before the start of training step] datetime: 2026-08-30 06:46:09
838
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:334: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
839
+ warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
840
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:334: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
841
+ warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
842
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:334: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
843
+ warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
844
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:334: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
845
+ warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
846
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:334: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
847
+ warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
848
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:336: UserWarning: The use_host_meta is deprecated, it will be removed in the future.
849
+ warnings.warn("The use_host_meta is deprecated, it will be removed in the future.")
850
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:334: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
851
+ warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
852
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:336: UserWarning: The use_host_meta is deprecated, it will be removed in the future.
853
+ warnings.warn("The use_host_meta is deprecated, it will be removed in the future.")
854
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:336: UserWarning: The use_host_meta is deprecated, it will be removed in the future.
855
+ warnings.warn("The use_host_meta is deprecated, it will be removed in the future.")
856
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:334: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
857
+ warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
858
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:334: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
859
+ warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
860
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:336: UserWarning: The use_host_meta is deprecated, it will be removed in the future.
861
+ warnings.warn("The use_host_meta is deprecated, it will be removed in the future.")
862
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:336: UserWarning: The use_host_meta is deprecated, it will be removed in the future.
863
+ warnings.warn("The use_host_meta is deprecated, it will be removed in the future.")
864
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:336: UserWarning: The use_host_meta is deprecated, it will be removed in the future.
865
+ warnings.warn("The use_host_meta is deprecated, it will be removed in the future.")
866
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:336: UserWarning: The use_host_meta is deprecated, it will be removed in the future.
867
+ warnings.warn("The use_host_meta is deprecated, it will be removed in the future.")
868
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:336: UserWarning: The use_host_meta is deprecated, it will be removed in the future.
869
+ warnings.warn("The use_host_meta is deprecated, it will be removed in the future.")
870
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:482: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
871
+ warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
872
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:482: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
873
+ warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
874
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:482: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
875
+ warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
876
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:482: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
877
+ warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
878
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:482: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
879
+ warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
880
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:482: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
881
+ warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
882
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:482: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
883
+ warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
884
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:482: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
885
+ warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
886
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:334: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
887
+ warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
888
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:336: UserWarning: The use_host_meta is deprecated, it will be removed in the future.
889
+ warnings.warn("The use_host_meta is deprecated, it will be removed in the future.")
890
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:334: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
891
+ warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
892
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:336: UserWarning: The use_host_meta is deprecated, it will be removed in the future.
893
+ warnings.warn("The use_host_meta is deprecated, it will be removed in the future.")
894
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:334: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
895
+ warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
896
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:336: UserWarning: The use_host_meta is deprecated, it will be removed in the future.
897
+ warnings.warn("The use_host_meta is deprecated, it will be removed in the future.")
898
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:334: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
899
+ warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
900
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:336: UserWarning: The use_host_meta is deprecated, it will be removed in the future.
901
+ warnings.warn("The use_host_meta is deprecated, it will be removed in the future.")
902
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:334: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
903
+ warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
904
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:336: UserWarning: The use_host_meta is deprecated, it will be removed in the future.
905
+ warnings.warn("The use_host_meta is deprecated, it will be removed in the future.")
906
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:334: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
907
+ warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
908
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:336: UserWarning: The use_host_meta is deprecated, it will be removed in the future.
909
+ warnings.warn("The use_host_meta is deprecated, it will be removed in the future.")
910
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:334: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
911
+ warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
912
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:336: UserWarning: The use_host_meta is deprecated, it will be removed in the future.
913
+ warnings.warn("The use_host_meta is deprecated, it will be removed in the future.")
914
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:334: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
915
+ warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
916
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:336: UserWarning: The use_host_meta is deprecated, it will be removed in the future.
917
+ warnings.warn("The use_host_meta is deprecated, it will be removed in the future.")
918
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:482: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
919
+ warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
920
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:482: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
921
+ warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
922
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:482: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
923
+ warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
924
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:482: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
925
+ warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
926
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:482: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
927
+ warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
928
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:482: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
929
+ warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
930
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:482: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
931
+ warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
932
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:482: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
933
+ warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
934
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:334: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
935
+ warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
936
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:334: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
937
+ warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
938
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:334: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
939
+ warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
940
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:334: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
941
+ warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
942
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:334: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
943
+ warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
944
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:334: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
945
+ warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
946
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:334: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
947
+ warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
948
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:334: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
949
+ warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
950
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:336: UserWarning: The use_host_meta is deprecated, it will be removed in the future.
951
+ warnings.warn("The use_host_meta is deprecated, it will be removed in the future.")
952
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:336: UserWarning: The use_host_meta is deprecated, it will be removed in the future.
953
+ warnings.warn("The use_host_meta is deprecated, it will be removed in the future.")
954
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:336: UserWarning: The use_host_meta is deprecated, it will be removed in the future.
955
+ warnings.warn("The use_host_meta is deprecated, it will be removed in the future.")
956
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:336: UserWarning: The use_host_meta is deprecated, it will be removed in the future.
957
+ warnings.warn("The use_host_meta is deprecated, it will be removed in the future.")
958
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:336: UserWarning: The use_host_meta is deprecated, it will be removed in the future.
959
+ warnings.warn("The use_host_meta is deprecated, it will be removed in the future.")
960
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:336: UserWarning: The use_host_meta is deprecated, it will be removed in the future.
961
+ warnings.warn("The use_host_meta is deprecated, it will be removed in the future.")
962
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:336: UserWarning: The use_host_meta is deprecated, it will be removed in the future.
963
+ warnings.warn("The use_host_meta is deprecated, it will be removed in the future.")
964
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:336: UserWarning: The use_host_meta is deprecated, it will be removed in the future.
965
+ warnings.warn("The use_host_meta is deprecated, it will be removed in the future.")
966
+ [2026-08-30 06:46:47] iteration 1/ 12 | consumed samples: 16 | elapsed time per iteration (ms): 38131.4 | throughput per GPU (TFLOP/s/GPU): 8.7 | learning rate: 2.000000E-05 | global batch size: 16 | lm loss: 1.104376E+01 | loss scale: 1.0 | grad norm: 34.299 | number of skipped iterations: 0 | number of nan iterations: 0 |Number of parameters in transformer block in billions: 23.22
967
+
968
+ Number of parameters in embedding layers in billions: 0.41
969
+ Total number of parameters in billions: 23.63
970
+ Number of parameters in most loaded shard in billions: 23.6320
971
+ Theoretical memory footprints: weight and optimizer=169029.43 MB
972
+ [Rank 0] (after 1 iterations) memory (MB) | allocated: 63795.404296875 | max allocated: 96519.67578125 | reserved: 99122.0 | max reserved: 99122.0
973
+ [2026-08-30 06:46:48] iteration 2/ 12 | consumed samples: 32 | elapsed time per iteration (ms): 573.2 | throughput per GPU (TFLOP/s/GPU): 581.6 | learning rate: 4.000000E-05 | global batch size: 16 | lm loss: 1.104935E+01 | loss scale: 1.0 | grad norm: 35.988 | number of skipped iterations: 0 | number of nan iterations: 0 |
974
+ [2026-08-30 06:46:48] iteration 3/ 12 | consumed samples: 48 | elapsed time per iteration (ms): 559.6 | throughput per GPU (TFLOP/s/GPU): 595.8 | learning rate: 6.000000E-05 | global batch size: 16 | lm loss: 9.657887E+00 | loss scale: 1.0 | grad norm: 40.365 | number of skipped iterations: 0 | number of nan iterations: 0 |
975
+ Capture CUDA graph for training!!!
976
+ CUDA graph capture done!!!
977
+ [2026-08-30 06:46:50] iteration 4/ 12 | consumed samples: 64 | elapsed time per iteration (ms): 1666.0 | throughput per GPU (TFLOP/s/GPU): 200.1 | learning rate: 8.000000E-05 | global batch size: 16 | lm loss: 1.081011E+01 | loss scale: 1.0 | grad norm: 92.418 | number of skipped iterations: 0 | number of nan iterations: 0 |
978
+ [2026-08-30 06:46:51] iteration 5/ 12 | consumed samples: 80 | elapsed time per iteration (ms): 469.9 | throughput per GPU (TFLOP/s/GPU): 709.4 | learning rate: 1.000000E-04 | global batch size: 16 | lm loss: 1.069354E+01 | loss scale: 1.0 | grad norm: 17.104 | number of skipped iterations: 0 | number of nan iterations: 0 |
979
+ [2026-08-30 06:46:51] iteration 6/ 12 | consumed samples: 96 | elapsed time per iteration (ms): 469.0 | throughput per GPU (TFLOP/s/GPU): 710.9 | learning rate: 7.750000E-05 | global batch size: 16 | lm loss: 9.820256E+00 | loss scale: 1.0 | grad norm: 38.716 | number of skipped iterations: 0 | number of nan iterations: 0 |
980
+ [2026-08-30 06:46:52] iteration 7/ 12 | consumed samples: 112 | elapsed time per iteration (ms): 466.1 | throughput per GPU (TFLOP/s/GPU): 715.2 | learning rate: 3.250000E-05 | global batch size: 16 | lm loss: 9.326885E+00 | loss scale: 1.0 | grad norm: 6.996 | number of skipped iterations: 0 | number of nan iterations: 0 |
981
+ [2026-08-30 06:46:52] iteration 8/ 12 | consumed samples: 128 | elapsed time per iteration (ms): 466.8 | throughput per GPU (TFLOP/s/GPU): 714.2 | learning rate: 1.000000E-05 | global batch size: 16 | lm loss: 9.158098E+00 | loss scale: 1.0 | grad norm: 28.651 | number of skipped iterations: 0 | number of nan iterations: 0 |
982
+ [2026-08-30 06:46:52] iteration 9/ 12 | consumed samples: 144 | elapsed time per iteration (ms): 463.5 | throughput per GPU (TFLOP/s/GPU): 719.3 | learning rate: 1.000000E-05 | global batch size: 16 | lm loss: 8.723837E+00 | loss scale: 1.0 | grad norm: 5.718 | number of skipped iterations: 0 | number of nan iterations: 0 |
983
+ [2026-08-30 06:46:53] iteration 10/ 12 | consumed samples: 160 | elapsed time per iteration (ms): 462.6 | throughput per GPU (TFLOP/s/GPU): 720.6 | learning rate: 1.000000E-05 | global batch size: 16 | lm loss: 8.580932E+00 | loss scale: 1.0 | grad norm: 5.040 | number of skipped iterations: 0 | number of nan iterations: 0 |
984
+ [2026-08-30 06:46:53] iteration 11/ 12 | consumed samples: 176 | elapsed time per iteration (ms): 470.7 | throughput per GPU (TFLOP/s/GPU): 708.2 | learning rate: 1.000000E-05 | global batch size: 16 | lm loss: 8.497838E+00 | loss scale: 1.0 | grad norm: 4.420 | number of skipped iterations: 0 | number of nan iterations: 0 |
985
+ [2026-08-30 06:46:54] iteration 12/ 12 | consumed samples: 192 | elapsed time per iteration (ms): 466.7 | throughput per GPU (TFLOP/s/GPU): 714.3 | learning rate: 1.000000E-05 | global batch size: 16 | lm loss: 8.392807E+00 | loss scale: 1.0 | grad norm: 4.866 | number of skipped iterations: 0 | number of nan iterations: 0 |
986
+ [after training is done] datetime: 2026-08-30 06:46:54
987
+ [rank0]:[W830 06:46:55.659787361 ProcessGroupNCCL.cpp:1505] Warning: WARNING: destroy_process_group() was not called before program exit, which can leak resources. For more info, please see https://pytorch.org/docs/stable/distributed.html#shutdown (function operator())
tiera-rerun-fixed-moonep/echo/a2/a2_matched_gbs16/G1__echo__matched__seq4096__gbs16__cap2p40625/rep3/mem_samples.csv ADDED
@@ -0,0 +1,232 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ 0, 0
2
+ 1, 0
3
+ 2, 0
4
+ 3, 0
5
+ 4, 0
6
+ 5, 0
7
+ 6, 0
8
+ 7, 0
9
+ 0, 0
10
+ 1, 0
11
+ 2, 0
12
+ 3, 0
13
+ 4, 0
14
+ 5, 0
15
+ 6, 0
16
+ 7, 0
17
+ 0, 4
18
+ 1, 1
19
+ 2, 1
20
+ 3, 1
21
+ 4, 1
22
+ 5, 1
23
+ 6, 1
24
+ 7, 1
25
+ 0, 4
26
+ 1, 4
27
+ 2, 4
28
+ 3, 4
29
+ 4, 2
30
+ 5, 1
31
+ 6, 1
32
+ 7, 1
33
+ 0, 4
34
+ 1, 4
35
+ 2, 4
36
+ 3, 4
37
+ 4, 4
38
+ 5, 4
39
+ 6, 4
40
+ 7, 4
41
+ 0, 821
42
+ 1, 819
43
+ 2, 821
44
+ 3, 819
45
+ 4, 819
46
+ 5, 819
47
+ 6, 821
48
+ 7, 821
49
+ 0, 1537
50
+ 1, 1537
51
+ 2, 1537
52
+ 3, 1537
53
+ 4, 1537
54
+ 5, 1537
55
+ 6, 1537
56
+ 7, 1537
57
+ 0, 2231
58
+ 1, 2231
59
+ 2, 2231
60
+ 3, 2231
61
+ 4, 2231
62
+ 5, 2231
63
+ 6, 2231
64
+ 7, 2231
65
+ 0, 39533
66
+ 1, 39545
67
+ 2, 39523
68
+ 3, 39531
69
+ 4, 39541
70
+ 5, 39549
71
+ 6, 39557
72
+ 7, 39595
73
+ 0, 39821
74
+ 1, 39803
75
+ 2, 39823
76
+ 3, 39823
77
+ 4, 39827
78
+ 5, 39845
79
+ 6, 39833
80
+ 7, 39831
81
+ 0, 42357
82
+ 1, 42355
83
+ 2, 42355
84
+ 3, 42355
85
+ 4, 42355
86
+ 5, 42355
87
+ 6, 42355
88
+ 7, 42355
89
+ 0, 42583
90
+ 1, 42581
91
+ 2, 42581
92
+ 3, 42581
93
+ 4, 42581
94
+ 5, 42581
95
+ 6, 42581
96
+ 7, 42581
97
+ 0, 42713
98
+ 1, 42739
99
+ 2, 42723
100
+ 3, 42697
101
+ 4, 42723
102
+ 5, 42759
103
+ 6, 42735
104
+ 7, 42721
105
+ 0, 43493
106
+ 1, 43491
107
+ 2, 43491
108
+ 3, 43491
109
+ 4, 43491
110
+ 5, 43491
111
+ 6, 43491
112
+ 7, 43491
113
+ 0, 43493
114
+ 1, 43491
115
+ 2, 43491
116
+ 3, 43491
117
+ 4, 43491
118
+ 5, 43491
119
+ 6, 43491
120
+ 7, 43491
121
+ 0, 45265
122
+ 1, 45263
123
+ 2, 45263
124
+ 3, 45263
125
+ 4, 45263
126
+ 5, 45263
127
+ 6, 45263
128
+ 7, 45263
129
+ 0, 45265
130
+ 1, 45263
131
+ 2, 45263
132
+ 3, 45263
133
+ 4, 45263
134
+ 5, 45263
135
+ 6, 45263
136
+ 7, 45263
137
+ 0, 85827
138
+ 1, 85825
139
+ 2, 85825
140
+ 3, 85825
141
+ 4, 85825
142
+ 5, 85825
143
+ 6, 85825
144
+ 7, 85825
145
+ 0, 86905
146
+ 1, 86903
147
+ 2, 86903
148
+ 3, 86903
149
+ 4, 86903
150
+ 5, 86903
151
+ 6, 86903
152
+ 7, 86903
153
+ 0, 86905
154
+ 1, 86903
155
+ 2, 86903
156
+ 3, 86903
157
+ 4, 86903
158
+ 5, 86903
159
+ 6, 86903
160
+ 7, 86903
161
+ 0, 86905
162
+ 1, 86903
163
+ 2, 86903
164
+ 3, 86903
165
+ 4, 86903
166
+ 5, 86903
167
+ 6, 86903
168
+ 7, 86903
169
+ 0, 104113
170
+ 1, 104035
171
+ 2, 104045
172
+ 3, 104045
173
+ 4, 104121
174
+ 5, 104075
175
+ 6, 104065
176
+ 7, 104095
177
+ 0, 105837
178
+ 1, 105815
179
+ 2, 105815
180
+ 3, 105801
181
+ 4, 105871
182
+ 5, 105815
183
+ 6, 105815
184
+ 7, 105815
185
+ 0, 106523
186
+ 1, 106465
187
+ 2, 106465
188
+ 3, 106465
189
+ 4, 106521
190
+ 5, 106465
191
+ 6, 106465
192
+ 7, 106465
193
+ 0, 107185
194
+ 1, 107049
195
+ 2, 107065
196
+ 3, 107077
197
+ 4, 107101
198
+ 5, 107067
199
+ 6, 107061
200
+ 7, 107085
201
+ 0, 136161
202
+ 1, 138979
203
+ 2, 156317
204
+ 3, 136653
205
+ 4, 135565
206
+ 5, 146571
207
+ 6, 135821
208
+ 7, 135821
209
+ 0, 161639
210
+ 1, 161527
211
+ 2, 161525
212
+ 3, 161525
213
+ 4, 161723
214
+ 5, 161525
215
+ 6, 161525
216
+ 7, 161525
217
+ 0, 161639
218
+ 1, 161527
219
+ 2, 161525
220
+ 3, 161525
221
+ 4, 161723
222
+ 5, 161525
223
+ 6, 161525
224
+ 7, 161525
225
+ 0, 2098
226
+ 1, 2098
227
+ 2, 2418
228
+ 3, 161525
229
+ 4, 3698
230
+ 5, 2098
231
+ 6, 2098
232
+ 7, 2098
tiera-rerun-fixed-moonep/echo/a2/a2_matched_gbs16/G1__echo__matched__seq4096__gbs16__cap2p40625/rep3/run.env ADDED
@@ -0,0 +1,70 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ backend=echo
2
+ config=G1
3
+ family=tierA
4
+ repo=/workspace/Megatron-MoE-Echo
5
+ branch=bracing-regions-through-kernels
6
+ commit=9a236793
7
+ num_layers=16
8
+ hidden=4096
9
+ ffn_hidden=14336
10
+ moe_ffn_hidden=14336
11
+ num_experts=8
12
+ topk=2
13
+ experts_per_rank=1
14
+ expert_params=176160768
15
+ heads=32
16
+ kv_groups=8
17
+ kv_channels=
18
+ qk_layernorm=0
19
+ shared_ffn=0
20
+ shared_expert_overlap=0
21
+ moe_token_dispatcher=flex
22
+ mla=0
23
+ dense_first_layers=0
24
+ moe_layers=16
25
+ rotary_base=1000000
26
+ rotary_scaling=
27
+ qkv_bias=0
28
+ router_balancing=aux_loss
29
+ aux_loss_coeff=1e-2
30
+ router_pre_softmax=0
31
+ router_fusion=0
32
+ router_dtype=fp32
33
+ device_initiated_grouped_gemm=1
34
+ norm_epsilon=1e-5
35
+ seq_length=4096
36
+ micro_batch_size=1
37
+ global_batch_size=16
38
+ num_microbatches=2
39
+ assignments_per_rank=8192
40
+ mean_tokens_per_expert=8192
41
+ slot_policy=matched
42
+ slots_per_rank=1
43
+ capacity_mode=static
44
+ capacity=2.40625
45
+ cuda_graph=1
46
+ probe_mode=0
47
+ async_grad_reduce=0
48
+ branch_role=timing
49
+ memory_dump_mode=none
50
+ memory_snapshot_path=
51
+ cuda_graph_warmup_steps=3
52
+ recompute=none
53
+ force_balance=0
54
+ train_iters=12
55
+ ep_size=8
56
+ echo_dispatch_overlap=1
57
+ echo_dispatch_recompute=0
58
+ ultraep_quota_min_tokens=
59
+ ultraep_weight_sync_join=
60
+ profile_enable=0
61
+ kernel_trace=0
62
+ fp8_mode=mxfp8
63
+ fp8_recipe=mxfp8
64
+ training_seed=1234
65
+ learning_rate=1e-4
66
+ min_learning_rate=1.0e-5
67
+ lr_warmup_iters=5
68
+ state_dump_enable=0
69
+ state_dump_dir=
70
+ moonep_assert_no_pending_reduce=0
tiera-rerun-fixed-moonep/echo/a2/a2_matched_gbs16/G1__echo__matched__seq4096__gbs16__cap2p40625/rep3/train.log ADDED
@@ -0,0 +1,1023 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ==============================================================
2
+ G1 / echo (slots/rank=1 policy=matched)
3
+ E=8 K=2 H'=14336 m=1
4
+ L=16 seq=4096 gbs=16 (2 microbatches)
5
+ cap_mode=static cap=2.40625 cg=1 recompute=none
6
+ memory_dump=none snapshots=/workspace/tiera-rerun-fixed-moonep/echo/a2/a2_matched_gbs16/G1__echo__matched__seq4096__gbs16__cap2p40625/rep3/memory_snapshots/snapshot.pickle
7
+ repo=/workspace/Megatron-MoE-Echo @ 9a236793
8
+ ==============================================================
9
+ W0830 06:47:05.447000 658758 torch/distributed/run.py:766]
10
+ W0830 06:47:05.447000 658758 torch/distributed/run.py:766] *****************************************
11
+ W0830 06:47:05.447000 658758 torch/distributed/run.py:766] Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed.
12
+ W0830 06:47:05.447000 658758 torch/distributed/run.py:766] *****************************************
13
+ /workspace/Megatron-MoE-Echo/megatron/training/arguments.py:831: UserWarning: Disabling sequence parallelism because tensor model parallelism is disabled
14
+ warnings.warn("Disabling sequence parallelism because tensor model parallelism is disabled")
15
+ using world size: 8, data-parallel size: 8, context-parallel size: 1, hierarchical context-parallel sizes: None, tensor-model-parallel size: 1, pipeline-model-parallel size: 1
16
+ WARNING: overriding default arguments for tokenizer_type:GPT2BPETokenizer with tokenizer_type:GPT2BPETokenizer
17
+ Number of virtual stages per pipeline stage: None
18
+ accumulate and all-reduce gradients in fp32 for bfloat16 data type.
19
+ using torch.bfloat16 for parameters ...
20
+ /workspace/Megatron-MoE-Echo/megatron/training/arguments.py:831: UserWarning: Disabling sequence parallelism because tensor model parallelism is disabled
21
+ warnings.warn("Disabling sequence parallelism because tensor model parallelism is disabled")
22
+ ------------------------ arguments ------------------------
23
+ account_for_embedding_in_pipeline_split ......... False
24
+ account_for_loss_in_pipeline_split .............. False
25
+ accumulate_allreduce_grads_in_fp32 .............. True
26
+ adam_beta1 ...................................... 0.9
27
+ adam_beta2 ...................................... 0.999
28
+ adam_eps ........................................ 1e-08
29
+ add_bias_linear ................................. False
30
+ add_position_embedding .......................... False
31
+ add_qkv_bias .................................... False
32
+ adlr_autoresume ................................. False
33
+ adlr_autoresume_interval ........................ 1000
34
+ align_grad_reduce ............................... True
35
+ align_param_gather .............................. False
36
+ app_tag_run_name ................................ None
37
+ app_tag_run_version ............................. 0.0.0
38
+ apply_layernorm_1p .............................. False
39
+ apply_query_key_layer_scaling ................... False
40
+ apply_residual_connection_post_layernorm ........ False
41
+ apply_rope_fusion ............................... True
42
+ async_save ...................................... None
43
+ async_tensor_model_parallel_allreduce ........... True
44
+ attention_backend ............................... AttnBackend.auto
45
+ attention_dropout ............................... 0.0
46
+ attention_softmax_in_fp32 ....................... False
47
+ auto_detect_ckpt_format ......................... False
48
+ barrier_with_L1_time ............................ True
49
+ bert_binary_head ................................ True
50
+ bert_embedder_type .............................. megatron
51
+ bert_load ....................................... None
52
+ bf16 ............................................ True
53
+ bias_dropout_fusion ............................. True
54
+ bias_gelu_fusion ................................ False
55
+ bias_swiglu_fusion .............................. True
56
+ biencoder_projection_dim ........................ 0
57
+ biencoder_shared_query_context_model ............ False
58
+ block_data_path ................................. None
59
+ calc_ft_timeouts ................................ False
60
+ calculate_per_token_loss ........................ False
61
+ check_for_large_grads ........................... False
62
+ check_for_nan_in_loss_and_grad .................. False
63
+ check_for_spiky_loss ............................ False
64
+ check_weight_hash_across_dp_replicas_interval ... None
65
+ ckpt_assume_constant_structure .................. False
66
+ ckpt_convert_format ............................. None
67
+ ckpt_convert_save ............................... None
68
+ ckpt_convert_update_legacy_dist_opt_format ...... False
69
+ ckpt_format ..................................... torch_dist
70
+ ckpt_fully_parallel_load ........................ False
71
+ ckpt_fully_parallel_save ........................ True
72
+ ckpt_fully_parallel_save_deprecated ............. False
73
+ ckpt_step ....................................... None
74
+ classes_fraction ................................ 1.0
75
+ clip_grad ....................................... 1.0
76
+ clone_scatter_output_in_embedding ............... True
77
+ config_logger_dir ...............................
78
+ consumed_train_samples .......................... 0
79
+ consumed_valid_samples .......................... 0
80
+ context_parallel_size ........................... 1
81
+ cp_comm_type .................................... ['p2p']
82
+ create_attention_mask_in_dataloader ............. True
83
+ cross_entropy_fusion_impl ....................... native
84
+ cross_entropy_loss_fusion ....................... False
85
+ cuda_graph_scope ................................ full_iteration
86
+ cuda_graph_warmup_steps ......................... 3
87
+ data_args_path .................................. None
88
+ data_cache_path ................................. None
89
+ data_parallel_random_init ....................... False
90
+ data_parallel_sharding_strategy ................. no_shard
91
+ data_parallel_size .............................. 8
92
+ data_path ....................................... ['/workspace/data/wikitext/wikitext103_gpt2_text_document']
93
+ data_per_class_fraction ......................... 1.0
94
+ data_sharding ................................... True
95
+ dataloader_type ................................. single
96
+ ddp_average_in_collective ....................... False
97
+ ddp_bucket_size ................................. None
98
+ ddp_num_buckets ................................. None
99
+ ddp_pad_buckets_for_high_nccl_busbw ............. False
100
+ decoder_first_pipeline_num_layers ............... None
101
+ decoder_last_pipeline_num_layers ................ None
102
+ decoder_num_layers .............................. None
103
+ decoder_seq_length .............................. None
104
+ decoupled_lr .................................... None
105
+ decoupled_min_lr ................................ None
106
+ decrease_batch_size_if_needed ................... False
107
+ defer_embedding_wgrad_compute ................... False
108
+ delay_wgrad_compute ............................. False
109
+ deprecated_use_mcore_models ..................... True
110
+ deterministic_mode .............................. False
111
+ dino_bottleneck_size ............................ 256
112
+ dino_freeze_last_layer .......................... 1
113
+ dino_head_hidden_size ........................... 2048
114
+ dino_local_crops_number ......................... 10
115
+ dino_local_img_size ............................. 96
116
+ dino_norm_last_layer ............................ False
117
+ dino_teacher_temp ............................... 0.07
118
+ dino_warmup_teacher_temp ........................ 0.04
119
+ dino_warmup_teacher_temp_epochs ................. 30
120
+ disable_bf16_reduced_precision_matmul ........... False
121
+ disable_mamba_mem_eff_path ...................... False
122
+ disable_straggler_on_startup .................... False
123
+ dist_ckpt_format_deprecated ..................... None
124
+ dist_ckpt_strictness ............................ assume_ok_unexpected
125
+ distribute_saved_activations .................... False
126
+ distributed_backend ............................. nccl
127
+ distributed_timeout_minutes ..................... 10
128
+ embedding_init_method_std ....................... None
129
+ embedding_path .................................. None
130
+ empty_unused_memory_level ....................... 0
131
+ enable_cuda_graph ............................... True
132
+ enable_experimental ............................. False
133
+ enable_ft_package ............................... False
134
+ enable_gloo_process_groups ...................... True
135
+ enable_msc ...................................... True
136
+ enable_one_logger ............................... True
137
+ encoder_num_layers .............................. 16
138
+ encoder_seq_length .............................. 4096
139
+ end_weight_decay ................................ 0.1
140
+ eod_mask_loss ................................... False
141
+ error_injection_rate ............................ 0
142
+ error_injection_type ............................ transient_error
143
+ eval_interval ................................... 1000000
144
+ eval_iters ...................................... 0
145
+ evidence_data_path .............................. None
146
+ exit_duration_in_mins ........................... None
147
+ exit_interval ................................... None
148
+ exit_on_missing_checkpoint ...................... False
149
+ exit_signal_handler ............................. False
150
+ exp_avg_dtype ................................... torch.float32
151
+ exp_avg_sq_dtype ................................ torch.float32
152
+ expert_model_parallel_size ...................... 8
153
+ expert_tensor_parallel_size ..................... 1
154
+ export_force_local_attention .................... False
155
+ export_kd_cfg ................................... None
156
+ export_kd_teacher_ckpt_format ................... None
157
+ export_kd_teacher_load .......................... None
158
+ export_kv_cache_quant ........................... False
159
+ export_legacy_megatron .......................... False
160
+ export_model_type ............................... GPTModel
161
+ export_moe_apply_probs_on_input ................. False
162
+ export_qk_l2_norm ............................... False
163
+ export_quant_cfg ................................ None
164
+ export_real_quant_cfg ........................... None
165
+ export_te_mcore_model ........................... False
166
+ external_cuda_graph ............................. False
167
+ ffn_hidden_size ................................. 14336
168
+ finetune ........................................ False
169
+ finetune_data_split ............................. train
170
+ finetune_hf_dataset ............................. None
171
+ first_last_layers_bf16 .......................... False
172
+ flash_decode .................................... False
173
+ fp16 ............................................ False
174
+ fp16_lm_cross_entropy ........................... False
175
+ fp32_residual_connection ........................ False
176
+ fp8 ............................................. e4m3
177
+ fp8_amax_compute_algo ........................... most_recent
178
+ fp8_amax_history_len ............................ 1
179
+ fp8_interval .................................... 1
180
+ fp8_margin ...................................... 0
181
+ fp8_param_gather ................................ False
182
+ fp8_recipe ...................................... mxfp8
183
+ fp8_wgrad ....................................... True
184
+ fsdp_double_buffer .............................. False
185
+ global_batch_size ............................... 16
186
+ grad_reduce_in_bf16 ............................. False
187
+ gradient_accumulation_fusion .................... True
188
+ gradient_reduce_div_fusion ...................... True
189
+ group_query_attention ........................... True
190
+ head_lr_mult .................................... 1.0
191
+ heterogeneous_layers_config_encoded_json ........ None
192
+ heterogeneous_layers_config_path ................ None
193
+ hidden_dropout .................................. 0.0
194
+ hidden_size ..................................... 4096
195
+ hierarchical_context_parallel_sizes ............. None
196
+ high_priority_stream_groups ..................... []
197
+ hybrid_attention_ratio .......................... 0.0
198
+ hybrid_mlp_ratio ................................ 0.0
199
+ hybrid_override_pattern ......................... None
200
+ hysteresis ...................................... 2
201
+ ict_head_size ................................... None
202
+ ict_load ........................................ None
203
+ img_h ........................................... 224
204
+ img_w ........................................... 224
205
+ indexer_batch_size .............................. 128
206
+ indexer_log_interval ............................ 1000
207
+ inference_batch_times_seqlen_threshold .......... -1
208
+ inference_dynamic_batching ...................... False
209
+ inference_dynamic_batching_buffer_guaranteed_fraction 0.2
210
+ inference_dynamic_batching_buffer_overflow_factor None
211
+ inference_dynamic_batching_buffer_size_gb ....... 40.0
212
+ inference_dynamic_batching_chunk_size ........... 256
213
+ inference_dynamic_batching_max_requests_override None
214
+ inference_dynamic_batching_max_tokens_override .. None
215
+ inference_dynamic_batching_num_cuda_graphs ...... 16
216
+ inference_max_batch_size ........................ 8
217
+ inference_max_seq_length ........................ 2560
218
+ inference_rng_tracker ........................... False
219
+ init_method_std ................................. 0.01
220
+ init_method_xavier_uniform ...................... False
221
+ init_model_with_meta_device ..................... False
222
+ initial_loss_scale .............................. 4294967296
223
+ inprocess_active_world_size ..................... 8
224
+ inprocess_barrier_timeout ....................... 120
225
+ inprocess_completion_timeout .................... 120
226
+ inprocess_empty_cuda_cache ...................... False
227
+ inprocess_granularity ........................... node
228
+ inprocess_hard_timeout .......................... 90
229
+ inprocess_heartbeat_interval .................... 30
230
+ inprocess_heartbeat_timeout ..................... 60
231
+ inprocess_last_call_wait ........................ 1
232
+ inprocess_max_iterations ........................ None
233
+ inprocess_monitor_process_interval .............. 1.0
234
+ inprocess_monitor_thread_interval ............... 1.0
235
+ inprocess_progress_watchdog_interval ............ 1.0
236
+ inprocess_restart ............................... False
237
+ inprocess_soft_timeout .......................... 60
238
+ inprocess_termination_grace_time ................ 1
239
+ is_hybrid_model ................................. False
240
+ iter_per_epoch .................................. 1250
241
+ iterations_to_skip .............................. []
242
+ keep_fp8_transpose_cache_when_using_custom_fsdp . False
243
+ kitchen_config_file ............................. None
244
+ kitchen_recipe_number ........................... None
245
+ kv_channels ..................................... 128
246
+ kv_lora_rank .................................... 32
247
+ lazy_mpu_init ................................... None
248
+ load ............................................ None
249
+ load_main_params_from_ckpt ...................... None
250
+ load_model_opt_format ........................... False
251
+ local_rank ...................................... 0
252
+ log_energy ...................................... False
253
+ log_interval .................................... 1
254
+ log_loss_scale_to_tensorboard ................... True
255
+ log_memory_to_tensorboard ....................... False
256
+ log_num_zeros_in_grad ........................... False
257
+ log_params_norm ................................. False
258
+ log_progress .................................... False
259
+ log_straggler ................................... False
260
+ log_throughput .................................. True
261
+ log_timers_to_tensorboard ....................... False
262
+ log_validation_ppl_to_tensorboard ............... False
263
+ log_world_size_to_tensorboard ................... False
264
+ logging_level ................................... None
265
+ loss_scale ...................................... None
266
+ loss_scale_window ............................... 1000
267
+ lr .............................................. 0.0001
268
+ lr_decay_iters .................................. 8
269
+ lr_decay_samples ................................ None
270
+ lr_decay_style .................................. cosine
271
+ lr_warmup_fraction .............................. None
272
+ lr_warmup_init .................................. 0.0
273
+ lr_warmup_iters ................................. 5
274
+ lr_warmup_samples ............................... 0
275
+ lr_wsd_decay_iters .............................. None
276
+ lr_wsd_decay_samples ............................ None
277
+ lr_wsd_decay_style .............................. exponential
278
+ main_grads_dtype ................................ torch.float32
279
+ main_params_dtype ............................... torch.float32
280
+ make_vocab_size_divisible_by .................... 128
281
+ mamba_head_dim .................................. 64
282
+ mamba_num_groups ................................ 8
283
+ mamba_num_heads ................................. None
284
+ mamba_state_dim ................................. 128
285
+ manual_gc ....................................... False
286
+ manual_gc_eval .................................. True
287
+ manual_gc_interval .............................. 0
288
+ mask_factor ..................................... 1.0
289
+ mask_prob ....................................... 0.15
290
+ mask_type ....................................... random
291
+ masked_softmax_fusion ........................... False
292
+ max_position_embeddings ......................... 32768
293
+ max_tokens_to_oom ............................... 12000
294
+ memory_snapshot_path ............................ snapshot.pickle
295
+ merge_file ...................................... /workspace/data/wikitext/gpt2-merges.txt
296
+ micro_batch_size ................................ 1
297
+ microbatch_group_size_per_vp_stage .............. None
298
+ mid_level_dataset_surplus ....................... 0.005
299
+ min_loss_scale .................................. 1.0
300
+ min_lr .......................................... 1e-05
301
+ mlp_chunks_for_prefill .......................... 1
302
+ mmap_bin_files .................................. True
303
+ mock_data ....................................... False
304
+ moe_apply_probs_on_input ........................ False
305
+ moe_aux_loss_coeff .............................. 0.01
306
+ moe_deepep_num_sms .............................. 20
307
+ moe_echo_enable_random_offloading ............... False
308
+ moe_echo_expert_dispatch_overlap ................ True
309
+ moe_echo_expert_dispatcher_type ................. hybridep
310
+ moe_echo_recompute_expert_dispatch .............. False
311
+ moe_enable_deepep ............................... False
312
+ moe_enable_echo ................................. True
313
+ moe_enable_hybridep ............................. True
314
+ moe_expert_capacity_factor ...................... None
315
+ moe_extended_tp ................................. False
316
+ moe_ffn_hidden_size ............................. 14336
317
+ moe_grouped_gemm ................................ True
318
+ moe_hybridep_num_sms ............................ 16
319
+ moe_input_jitter_eps ............................ None
320
+ moe_layer_freq .................................. 1
321
+ moe_layer_recompute ............................. False
322
+ moe_num_echo_experts ............................ 8
323
+ moe_pad_expert_input_to_capacity ................ False
324
+ moe_per_layer_logging ........................... False
325
+ moe_permute_fusion .............................. False
326
+ moe_received_token_capacity ..................... 2.40625
327
+ moe_router_bias_update_rate ..................... 0.001
328
+ moe_router_dtype ................................ fp32
329
+ moe_router_enable_expert_bias ................... False
330
+ moe_router_force_load_balancing ................. False
331
+ moe_router_group_topk ........................... None
332
+ moe_router_load_balancing_type .................. aux_loss
333
+ moe_router_num_groups ........................... None
334
+ moe_router_padding_for_fp8 ...................... False
335
+ moe_router_pre_softmax .......................... False
336
+ moe_router_score_function ....................... softmax
337
+ moe_router_topk ................................. 2
338
+ moe_router_topk_scaling_factor .................. None
339
+ moe_shared_expert_intermediate_size ............. None
340
+ moe_shared_expert_overlap ....................... False
341
+ moe_token_dispatcher_type ....................... flex
342
+ moe_token_drop_policy ........................... probs
343
+ moe_upcycling_granularity ....................... 1
344
+ moe_use_device_initiated_grouped_gemm ........... True
345
+ moe_use_legacy_grouped_gemm ..................... False
346
+ moe_use_upcycling ............................... False
347
+ moe_z_loss_coeff ................................ None
348
+ mrope_section ................................... None
349
+ mscale .......................................... 1.0
350
+ mscale_all_dim .................................. 0.0
351
+ mtp_loss_scaling_factor ......................... 0.1
352
+ mtp_num_layers .................................. None
353
+ multi_latent_attention .......................... False
354
+ nccl_all_reduce_for_prefill ..................... False
355
+ nccl_communicator_config_path ................... None
356
+ nccl_ub ......................................... False
357
+ no_load_optim ................................... None
358
+ no_load_rng ..................................... None
359
+ no_persist_layer_norm ........................... False
360
+ no_rope_freq .................................... None
361
+ no_save_optim ................................... None
362
+ no_save_rng ..................................... None
363
+ non_persistent_ckpt_type ........................ None
364
+ non_persistent_global_ckpt_dir .................. None
365
+ non_persistent_local_ckpt_algo .................. fully_parallel
366
+ non_persistent_local_ckpt_dir ................... None
367
+ non_persistent_save_interval .................... None
368
+ norm_epsilon .................................... 1e-05
369
+ normalization ................................... RMSNorm
370
+ num_attention_heads ............................. 32
371
+ num_channels .................................... 3
372
+ num_classes ..................................... 1000
373
+ num_dataset_builder_threads ..................... 1
374
+ num_distributed_optimizer_instances ............. 1
375
+ num_experts ..................................... 8
376
+ num_layers ...................................... 16
377
+ num_layers_at_end_in_bf16 ....................... 1
378
+ num_layers_at_start_in_bf16 ..................... 1
379
+ num_layers_per_virtual_pipeline_stage ........... None
380
+ num_query_groups ................................ 8
381
+ num_virtual_stages_per_pipeline_rank ............ None
382
+ num_workers ..................................... 2
383
+ object_storage_cache_path ....................... None
384
+ one_logger_async ................................ False
385
+ one_logger_project .............................. megatron-lm
386
+ one_logger_run_name ............................. None
387
+ onnx_safe ....................................... None
388
+ openai_gelu ..................................... False
389
+ optimizer ....................................... adam
390
+ optimizer_cpu_offload ........................... False
391
+ optimizer_offload_fraction ...................... 1.0
392
+ output_bert_embeddings .......................... False
393
+ overlap_cpu_optimizer_d2h_h2d ................... False
394
+ overlap_grad_reduce ............................. True
395
+ overlap_moe_expert_parallel_comm ................ False
396
+ overlap_p2p_comm ................................ False
397
+ overlap_p2p_comm_warmup_flush ................... False
398
+ overlap_param_gather ............................ True
399
+ overlap_param_gather_with_optimizer_step ........ False
400
+ override_opt_param_scheduler .................... False
401
+ params_dtype .................................... torch.bfloat16
402
+ patch_dim ....................................... 16
403
+ per_split_data_args_path ........................ None
404
+ perform_initialization .......................... True
405
+ pin_cpu_grads ................................... True
406
+ pin_cpu_params .................................. True
407
+ pipeline_model_parallel_comm_backend ............ None
408
+ pipeline_model_parallel_layout .................. None
409
+ pipeline_model_parallel_size .................... 1
410
+ position_embedding_type ......................... rope
411
+ pretrained_checkpoint ........................... None
412
+ profile ......................................... False
413
+ profile_ranks ................................... [0]
414
+ profile_step_end ................................ 12
415
+ profile_step_start .............................. 10
416
+ q_lora_rank ..................................... None
417
+ qk_head_dim ..................................... 128
418
+ qk_l2_norm ...................................... False
419
+ qk_layernorm .................................... False
420
+ qk_pos_emb_head_dim ............................. 64
421
+ query_in_block_prob ............................. 0.1
422
+ rampup_batch_size ............................... None
423
+ rank ............................................ 0
424
+ recompute_granularity ........................... None
425
+ recompute_method ................................ None
426
+ recompute_modules ............................... None
427
+ recompute_num_layers ............................ None
428
+ record_memory_history ........................... False
429
+ relative_attention_max_distance ................. 128
430
+ relative_attention_num_buckets .................. 32
431
+ replication ..................................... False
432
+ replication_factor .............................. 2
433
+ replication_jump ................................ None
434
+ rerun_mode ...................................... validate_results
435
+ reset_attention_mask ............................ False
436
+ reset_position_ids .............................. False
437
+ result_rejected_tracker_filename ................ None
438
+ retriever_report_topk_accuracies ................ []
439
+ retriever_score_scaling ......................... False
440
+ retriever_seq_length ............................ 256
441
+ retro_add_retriever ............................. False
442
+ retro_attention_gate ............................ 1
443
+ retro_cyclic_train_iters ........................ None
444
+ retro_encoder_attention_dropout ................. 0.1
445
+ retro_encoder_hidden_dropout .................... 0.1
446
+ retro_encoder_layers ............................ 2
447
+ retro_num_neighbors ............................. 2
448
+ retro_num_retrieved_chunks ...................... 2
449
+ retro_project_dir ............................... None
450
+ retro_verify_neighbor_count ..................... True
451
+ reuse_grad_buf_for_mxfp8_param_ag ............... False
452
+ rope_scaling_factor ............................. 8.0
453
+ rope_type ....................................... None
454
+ rotary_base ..................................... 1000000
455
+ rotary_interleaved .............................. False
456
+ rotary_percent .................................. 1.0
457
+ rotary_scaling_factor ........................... 1.0
458
+ rotary_seq_len_interpolation_factor ............. None
459
+ run_workload_inspector_server ................... False
460
+ sample_rate ..................................... 1.0
461
+ save ............................................ None
462
+ save_interval ................................... None
463
+ save_retain_interval ............................ None
464
+ scatter_gather_tensors_in_pipeline .............. True
465
+ seed ............................................ 1234
466
+ seq_length ...................................... 4096
467
+ sequence_parallel ............................... False
468
+ sft ............................................. False
469
+ sft_tokenizer_prompt_format ..................... nemotron-h-aligned
470
+ sgd_momentum .................................... 0.9
471
+ short_seq_prob .................................. 0.1
472
+ skip_train ...................................... False
473
+ skipped_train_samples ........................... 0
474
+ spec ............................................ None
475
+ split ........................................... 100,0,0
476
+ squared_relu .................................... False
477
+ start_weight_decay .............................. 0.1
478
+ straggler_ctrlr_port ............................ 65535
479
+ straggler_minmax_count .......................... 1
480
+ suggested_communication_unit_size ............... None
481
+ swiglu .......................................... True
482
+ swin_backbone_type .............................. tiny
483
+ symmetric_ar_type ............................... None
484
+ te_rng_tracker .................................. True
485
+ tensor_model_parallel_size ...................... 1
486
+ tensorboard_dir ................................. None
487
+ tensorboard_log_interval ........................ 1
488
+ tensorboard_queue_size .......................... 1000
489
+ test_data_path .................................. None
490
+ test_mode ....................................... False
491
+ tiktoken_num_special_tokens ..................... 1000
492
+ tiktoken_pattern ................................ None
493
+ tiktoken_special_tokens ......................... None
494
+ timing_log_level ................................ 0
495
+ timing_log_option ............................... minmax
496
+ titles_data_path ................................ None
497
+ tokenizer_model ................................. None
498
+ tokenizer_type .................................. GPT2BPETokenizer
499
+ torch_fsdp2_reshard_after_forward ............... True
500
+ tp_comm_bootstrap_backend ....................... nccl
501
+ tp_comm_bulk_dgrad .............................. True
502
+ tp_comm_bulk_wgrad .............................. True
503
+ tp_comm_overlap ................................. False
504
+ tp_comm_overlap_ag .............................. True
505
+ tp_comm_overlap_cfg ............................. None
506
+ tp_comm_overlap_rs .............................. True
507
+ tp_comm_overlap_rs_dgrad ........................ False
508
+ tp_comm_split_ag ................................ True
509
+ tp_comm_split_rs ................................ True
510
+ train_data_path ................................. None
511
+ train_iters ..................................... 12
512
+ train_samples ................................... None
513
+ train_sync_interval ............................. None
514
+ transformer_impl ................................ transformer_engine
515
+ transformer_pipeline_model_parallel_size ........ 1
516
+ untie_embeddings_and_output_weights ............. True
517
+ use_checkpoint_args ............................. False
518
+ use_checkpoint_opt_param_scheduler .............. False
519
+ use_cpu_initialization .......................... None
520
+ use_custom_fsdp ................................. False
521
+ use_dist_ckpt ................................... True
522
+ use_dist_ckpt_deprecated ........................ False
523
+ use_distributed_optimizer ....................... True
524
+ use_flash_attn .................................. False
525
+ use_legacy_models ............................... False
526
+ use_mp_args_from_checkpoint_args ................ False
527
+ use_one_sent_docs ............................... False
528
+ use_persistent_ckpt_worker ...................... False
529
+ use_precision_aware_optimizer ................... False
530
+ use_pytorch_profiler ............................ False
531
+ use_ring_exchange_p2p ........................... False
532
+ use_rope_scaling ................................ False
533
+ use_rotary_position_embeddings .................. False
534
+ use_sharp ....................................... False
535
+ use_tokenizer_model_from_checkpoint_args ........ True
536
+ use_torch_fsdp2 ................................. False
537
+ use_torch_optimizer_for_cpu_offload ............. False
538
+ use_tp_pp_dp_mapping ............................ False
539
+ v_head_dim ...................................... 128
540
+ valid_data_path ................................. None
541
+ variable_seq_lengths ............................ False
542
+ virtual_pipeline_model_parallel_size ............ None
543
+ vision_backbone_type ............................ vit
544
+ vision_pretraining .............................. False
545
+ vision_pretraining_type ......................... classify
546
+ vocab_extra_ids ................................. 0
547
+ vocab_file ...................................... /workspace/data/wikitext/gpt2-vocab.json
548
+ vocab_size ...................................... None
549
+ wandb_exp_name ..................................
550
+ wandb_project ...................................
551
+ wandb_save_dir ..................................
552
+ weight_decay .................................... 0.1
553
+ weight_decay_incr_style ......................... constant
554
+ wgrad_deferral_limit ............................ 0
555
+ world_size ...................................... 8
556
+ yaml_cfg ........................................ None
557
+ -------------------- end of arguments ---------------------
558
+ INFO:megatron.core.num_microbatches_calculator:setting number of microbatches to constant 2
559
+ > building GPT2BPETokenizer tokenizer ...
560
+ /workspace/Megatron-MoE-Echo/megatron/training/arguments.py:831: UserWarning: Disabling sequence parallelism because tensor model parallelism is disabled
561
+ warnings.warn("Disabling sequence parallelism because tensor model parallelism is disabled")
562
+ > padded vocab (size: 50257) with 47 dummy tokens (new size: 50304)
563
+ WARNING:megatron.core.rerun_state_machine:RerunStateMachine initialized in mode RerunMode.VALIDATE_RESULTS
564
+ > initializing torch distributed ...
565
+ /workspace/Megatron-MoE-Echo/megatron/training/arguments.py:831: UserWarning: Disabling sequence parallelism because tensor model parallelism is disabled
566
+ warnings.warn("Disabling sequence parallelism because tensor model parallelism is disabled")
567
+ /workspace/Megatron-MoE-Echo/megatron/training/arguments.py:831: UserWarning: Disabling sequence parallelism because tensor model parallelism is disabled
568
+ warnings.warn("Disabling sequence parallelism because tensor model parallelism is disabled")
569
+ /workspace/Megatron-MoE-Echo/megatron/training/arguments.py:831: UserWarning: Disabling sequence parallelism because tensor model parallelism is disabled
570
+ warnings.warn("Disabling sequence parallelism because tensor model parallelism is disabled")
571
+ /workspace/Megatron-MoE-Echo/megatron/training/arguments.py:831: UserWarning: Disabling sequence parallelism because tensor model parallelism is disabled
572
+ warnings.warn("Disabling sequence parallelism because tensor model parallelism is disabled")
573
+ /workspace/Megatron-MoE-Echo/megatron/training/arguments.py:831: UserWarning: Disabling sequence parallelism because tensor model parallelism is disabled
574
+ warnings.warn("Disabling sequence parallelism because tensor model parallelism is disabled")
575
+ WARNING: one_logger package is required to enable e2e metrics tracking. please go to https://confluence.nvidia.com/display/MLWFO/Package+Repositories for details to install it
576
+ [Gloo] Rank 4 is connected to 7 peer ranks. Expected number of connected peer ranks is : [Gloo] Rank 70
577
+ is connected to 7 peer ranks. Expected number of connected peer ranks is : 7
578
+ [Gloo] Rank 7 is connected to 7 peer ranks. Expected number of connected peer ranks is : 7
579
+ [Gloo] Rank [Gloo] Rank [Gloo] Rank [Gloo] Rank [Gloo] Rank 16235 is connected to is connected to is connected to is connected to is connected to 77777 peer ranks. peer ranks. peer ranks. peer ranks. peer ranks. Expected number of connected peer ranks is : Expected number of connected peer ranks is : Expected number of connected peer ranks is : Expected number of connected peer ranks is : Expected number of connected peer ranks is : 77777
580
+
581
+
582
+
583
+
584
+ [Gloo] Rank 2 is connected to 7 peer ranks. Expected number of connected peer ranks is : 7
585
+ [Gloo] Rank 3 is connected to 7 peer ranks. Expected number of connected peer ranks is : 7
586
+ [Gloo] Rank 0 is connected to 7 peer ranks. Expected number of connected peer ranks is : 7
587
+ [Gloo] Rank 1 is connected to 7 peer ranks. Expected number of connected peer ranks is : 7
588
+ [Gloo] Rank 4 is connected to 7 peer ranks. Expected number of connected peer ranks is : 7
589
+ [Gloo] Rank 6 is connected to 7 peer ranks. Expected number of connected peer ranks is : 7
590
+ [Gloo] Rank 5 is connected to 7 peer ranks. Expected number of connected peer ranks is : 7
591
+ [Gloo] Rank 7 is connected to 7 peer ranks. Expected number of connected peer ranks is : 7
592
+ [Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
593
+ > initialized tensor model parallel with size 1
594
+ > initialized pipeline model parallel with size 1
595
+ > setting random seeds to 1234 ...
596
+ [Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
597
+ [Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
598
+ [Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
599
+ [Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
600
+ [Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
601
+ [Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
602
+ [Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
603
+ [rank5]:[W830 06:47:23.364184986 ProcessGroupNCCL.cpp:4789] [PG ID 0 PG GUID 0 Rank 5] using GPU 5 as device used by this process is currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect. You can specify device_id in init_process_group() to force use of a particular device.
604
+ [rank3]:[W830 06:47:23.364342895 ProcessGroupNCCL.cpp:4789] [PG ID 0 PG GUID 0 Rank 3] using GPU 3 as device used by this process is currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect. You can specify device_id in init_process_group() to force use of a particular device.
605
+ [rank7]:[W830 06:47:23.374833410 ProcessGroupNCCL.cpp:4789] [PG ID 0 PG GUID 0 Rank 7] using GPU 7 as device used by this process is currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect. You can specify device_id in init_process_group() to force use of a particular device.
606
+ [rank6]:[W830 06:47:23.374883654 ProcessGroupNCCL.cpp:4789] [PG ID 0 PG GUID 0 Rank 6] using GPU 6 as device used by this process is currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect. You can specify device_id in init_process_group() to force use of a particular device.
607
+ [rank1]:[W830 06:47:23.374911183 ProcessGroupNCCL.cpp:4789] [PG ID 0 PG GUID 0 Rank 1] using GPU 1 as device used by this process is currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect. You can specify device_id in init_process_group() to force use of a particular device.
608
+ [rank4]:[W830 06:47:23.374918935 ProcessGroupNCCL.cpp:4789] [PG ID 0 PG GUID 0 Rank 4] using GPU 4 as device used by this process is currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect. You can specify device_id in init_process_group() to force use of a particular device.
609
+ [rank2]:[W830 06:47:23.374927356 ProcessGroupNCCL.cpp:4789] [PG ID 0 PG GUID 0 Rank 2] using GPU 2 as device used by this process is currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect. You can specify device_id in init_process_group() to force use of a particular device.
610
+ > compiling dataset index builder ...
611
+ make: Entering directory '/workspace/Megatron-MoE-Echo/megatron/core/datasets'
612
+ make: Nothing to be done for 'default'.
613
+ make: Leaving directory '/workspace/Megatron-MoE-Echo/megatron/core/datasets'
614
+ >>> done with dataset index builder. Compilation time: 0.036 seconds
615
+ WARNING: constraints for invoking optimized fused softmax kernel are not met. We default back to unfused kernel invocations.
616
+ > compiling and loading fused kernels ...
617
+ [rank0]:[W830 06:47:23.534087244 ProcessGroupNCCL.cpp:4789] [PG ID 0 PG GUID 0 Rank 0] using GPU 0 as device used by this process is currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect. You can specify device_id in init_process_group() to force use of a particular device.
618
+ >>> done with compiling and loading fused kernels. Compilation time: 3.783 seconds
619
+ time to initialize megatron (seconds): 7.176
620
+ [after megatron is initialized] datetime: 2026-08-30 06:47:29
621
+ building GPT model ...
622
+ /usr/local/lib/python3.12/dist-packages/transformer_engine/pytorch/cpu_offload.py:708: DeprecationWarning: Offloading weights is deprecated. Using offload_weights=True does not have any effect.
623
+ warnings.warn(
624
+ /usr/local/lib/python3.12/dist-packages/transformer_engine/pytorch/cpu_offload.py:708: DeprecationWarning: Offloading weights is deprecated. Using offload_weights=True does not have any effect.
625
+ warnings.warn(
626
+ /usr/local/lib/python3.12/dist-packages/transformer_engine/pytorch/cpu_offload.py:708: DeprecationWarning: Offloading weights is deprecated. Using offload_weights=True does not have any effect.
627
+ warnings.warn(
628
+ /usr/local/lib/python3.12/dist-packages/transformer_engine/pytorch/cpu_offload.py:708: DeprecationWarning: Offloading weights is deprecated. Using offload_weights=True does not have any effect.
629
+ warnings.warn(
630
+ /usr/local/lib/python3.12/dist-packages/transformer_engine/pytorch/cpu_offload.py:708: DeprecationWarning: Offloading weights is deprecated. Using offload_weights=True does not have any effect.
631
+ warnings.warn(
632
+ /usr/local/lib/python3.12/dist-packages/transformer_engine/pytorch/cpu_offload.py:708: DeprecationWarning: Offloading weights is deprecated. Using offload_weights=True does not have any effect.
633
+ warnings.warn(
634
+ /usr/local/lib/python3.12/dist-packages/transformer_engine/pytorch/cpu_offload.py:708: DeprecationWarning: Offloading weights is deprecated. Using offload_weights=True does not have any effect.
635
+ warnings.warn(
636
+ /usr/local/lib/python3.12/dist-packages/transformer_engine/pytorch/cpu_offload.py:708: DeprecationWarning: Offloading weights is deprecated. Using offload_weights=True does not have any effect.
637
+ warnings.warn(
638
+ > number of parameters on (tensor, pipeline) model parallel rank (0, 0): 3902410752
639
+ INFO:megatron.core.distributed.distributed_data_parallel:Setting up DistributedDataParallel with config DistributedDataParallelConfig(grad_reduce_in_fp32=True, overlap_grad_reduce=True, overlap_param_gather=True, align_param_gather=False, use_distributed_optimizer=True, num_distributed_optimizer_instances=1, check_for_nan_in_grad=False, check_for_large_grads=False, bucket_size=40000000, pad_buckets_for_high_nccl_busbw=False, average_in_collective=False, fp8_param_gather=False, reuse_grad_buf_for_mxfp8_param_ag=False, use_custom_fsdp=False, data_parallel_sharding_strategy='no_shard', gradient_reduce_div_fusion=True, suggested_communication_unit_size=None, preserve_fp32_weights=True, keep_fp8_transpose_cache_when_using_custom_fsdp=False, nccl_ub=False, fsdp_double_buffer=False)
640
+ INFO:megatron.core.distributed.param_and_grad_buffer:Number of buckets for gradient all-reduce / reduce-scatter: 18
641
+ Params for bucket 1 (206045184 elements, 206045184 padded size):
642
+ module.output_layer.weight
643
+ Params for bucket 2 (41988096 elements, 41988096 padded size):
644
+ module.decoder.final_layernorm.weight
645
+ module.decoder.layers.15.pre_mlp_layernorm.weight
646
+ module.decoder.layers.15.self_attention.linear_qkv.weight
647
+ module.decoder.layers.15.mlp.router.weight
648
+ module.decoder.layers.15.self_attention.linear_qkv.layer_norm_weight
649
+ module.decoder.layers.15.self_attention.linear_proj.weight
650
+ Params for bucket 3 (41984000 elements, 41984000 padded size):
651
+ module.decoder.layers.14.mlp.router.weight
652
+ module.decoder.layers.14.self_attention.linear_qkv.layer_norm_weight
653
+ module.decoder.layers.14.self_attention.linear_proj.weight
654
+ module.decoder.layers.14.self_attention.linear_qkv.weight
655
+ module.decoder.layers.14.pre_mlp_layernorm.weight
656
+ Params for bucket 4 (41984000 elements, 41984000 padded size):
657
+ module.decoder.layers.13.self_attention.linear_qkv.weight
658
+ module.decoder.layers.13.pre_mlp_layernorm.weight
659
+ module.decoder.layers.13.mlp.router.weight
660
+ module.decoder.layers.13.self_attention.linear_proj.weight
661
+ module.decoder.layers.13.self_attention.linear_qkv.layer_norm_weight
662
+ Params for bucket 5 (41984000 elements, 41984000 padded size):
663
+ module.decoder.layers.12.mlp.router.weight
664
+ module.decoder.layers.12.self_attention.linear_qkv.weight
665
+ module.decoder.layers.12.self_attention.linear_proj.weight
666
+ module.decoder.layers.12.self_attention.linear_qkv.layer_norm_weight
667
+ module.decoder.layers.12.pre_mlp_layernorm.weight
668
+ Params for bucket 6 (41984000 elements, 41984000 padded size):
669
+ module.decoder.layers.11.mlp.router.weight
670
+ module.decoder.layers.11.self_attention.linear_proj.weight
671
+ module.decoder.layers.11.self_attention.linear_qkv.layer_norm_weight
672
+ module.decoder.layers.11.self_attention.linear_qkv.weight
673
+ module.decoder.layers.11.pre_mlp_layernorm.weight
674
+ Params for bucket 7 (41984000 elements, 41984000 padded size):
675
+ module.decoder.layers.10.mlp.router.weight
676
+ module.decoder.layers.10.self_attention.linear_proj.weight
677
+ module.decoder.layers.10.self_attention.linear_qkv.layer_norm_weight
678
+ module.decoder.layers.10.self_attention.linear_qkv.weight
679
+ module.decoder.layers.10.pre_mlp_layernorm.weight
680
+ Params for bucket 8 (41984000 elements, 41984000 padded size):
681
+ module.decoder.layers.9.mlp.router.weight
682
+ module.decoder.layers.9.self_attention.linear_proj.weight
683
+ module.decoder.layers.9.self_attention.linear_qkv.layer_norm_weight
684
+ module.decoder.layers.9.pre_mlp_layernorm.weight
685
+ module.decoder.layers.9.self_attention.linear_qkv.weight
686
+ Params for bucket 9 (41984000 elements, 41984000 padded size):
687
+ module.decoder.layers.8.mlp.router.weight
688
+ module.decoder.layers.8.self_attention.linear_proj.weight
689
+ module.decoder.layers.8.self_attention.linear_qkv.layer_norm_weight
690
+ module.decoder.layers.8.pre_mlp_layernorm.weight
691
+ module.decoder.layers.8.self_attention.linear_qkv.weight
692
+ Params for bucket 10 (41984000 elements, 41984000 padded size):
693
+ module.decoder.layers.7.mlp.router.weight
694
+ module.decoder.layers.7.self_attention.linear_proj.weight
695
+ module.decoder.layers.7.self_attention.linear_qkv.layer_norm_weight
696
+ module.decoder.layers.7.pre_mlp_layernorm.weight
697
+ module.decoder.layers.7.self_attention.linear_qkv.weight
698
+ Params for bucket 11 (41984000 elements, 41984000 padded size):
699
+ module.decoder.layers.6.self_attention.linear_qkv.layer_norm_weight
700
+ module.decoder.layers.6.self_attention.linear_proj.weight
701
+ module.decoder.layers.6.mlp.router.weight
702
+ module.decoder.layers.6.self_attention.linear_qkv.weight
703
+ module.decoder.layers.6.pre_mlp_layernorm.weight
704
+ Params for bucket 12 (41984000 elements, 41984000 padded size):
705
+ module.decoder.layers.5.self_attention.linear_proj.weight
706
+ module.decoder.layers.5.pre_mlp_layernorm.weight
707
+ module.decoder.layers.5.self_attention.linear_qkv.weight
708
+ module.decoder.layers.5.mlp.router.weight
709
+ module.decoder.layers.5.self_attention.linear_qkv.layer_norm_weight
710
+ Params for bucket 13 (41984000 elements, 41984000 padded size):
711
+ module.decoder.layers.4.self_attention.linear_proj.weight
712
+ module.decoder.layers.4.self_attention.linear_qkv.layer_norm_weight
713
+ module.decoder.layers.4.mlp.router.weight
714
+ module.decoder.layers.4.self_attention.linear_qkv.weight
715
+ module.decoder.layers.4.pre_mlp_layernorm.weight
716
+ Params for bucket 14 (41984000 elements, 41984000 padded size):
717
+ module.decoder.layers.3.pre_mlp_layernorm.weight
718
+ module.decoder.layers.3.self_attention.linear_qkv.weight
719
+ module.decoder.layers.3.mlp.router.weight
720
+ module.decoder.layers.3.self_attention.linear_proj.weight
721
+ module.decoder.layers.3.self_attention.linear_qkv.layer_norm_weight
722
+ Params for bucket 15 (41984000 elements, 41984000 padded size):
723
+ module.decoder.layers.2.self_attention.linear_qkv.weight
724
+ module.decoder.layers.2.mlp.router.weight
725
+ module.decoder.layers.2.pre_mlp_layernorm.weight
726
+ module.decoder.layers.2.self_attention.linear_qkv.layer_norm_weight
727
+ module.decoder.layers.2.self_attention.linear_proj.weight
728
+ Params for bucket 16 (41984000 elements, 41984000 padded size):
729
+ module.decoder.layers.1.self_attention.linear_qkv.weight
730
+ module.decoder.layers.1.pre_mlp_layernorm.weight
731
+ module.decoder.layers.1.mlp.router.weight
732
+ module.decoder.layers.1.self_attention.linear_proj.weight
733
+ module.decoder.layers.1.self_attention.linear_qkv.layer_norm_weight
734
+ Params for bucket 17 (41984000 elements, 41984000 padded size):
735
+ module.decoder.layers.0.self_attention.linear_qkv.layer_norm_weight
736
+ module.decoder.layers.0.pre_mlp_layernorm.weight
737
+ module.decoder.layers.0.mlp.router.weight
738
+ module.decoder.layers.0.self_attention.linear_qkv.weight
739
+ module.decoder.layers.0.self_attention.linear_proj.weight
740
+ Params for bucket 18 (206045184 elements, 206045184 padded size):
741
+ module.embedding.word_embeddings.weight
742
+ INFO:megatron.core.distributed.param_and_grad_buffer:Number of buckets for gradient all-reduce / reduce-scatter: 32
743
+ Params for bucket 1 (58720256 elements, 58720256 padded size):
744
+ module.decoder.layers.15.mlp.experts.linear_fc2.weight0
745
+ Params for bucket 2 (117440512 elements, 117440512 padded size):
746
+ module.decoder.layers.15.mlp.experts.linear_fc1.weight0
747
+ Params for bucket 3 (58720256 elements, 58720256 padded size):
748
+ module.decoder.layers.14.mlp.experts.linear_fc2.weight0
749
+ Params for bucket 4 (117440512 elements, 117440512 padded size):
750
+ module.decoder.layers.14.mlp.experts.linear_fc1.weight0
751
+ Params for bucket 5 (58720256 elements, 58720256 padded size):
752
+ module.decoder.layers.13.mlp.experts.linear_fc2.weight0
753
+ Params for bucket 6 (117440512 elements, 117440512 padded size):
754
+ module.decoder.layers.13.mlp.experts.linear_fc1.weight0
755
+ Params for bucket 7 (58720256 elements, 58720256 padded size):
756
+ module.decoder.layers.12.mlp.experts.linear_fc2.weight0
757
+ Params for bucket 8 (117440512 elements, 117440512 padded size):
758
+ module.decoder.layers.12.mlp.experts.linear_fc1.weight0
759
+ Params for bucket 9 (58720256 elements, 58720256 padded size):
760
+ module.decoder.layers.11.mlp.experts.linear_fc2.weight0
761
+ Params for bucket 10 (117440512 elements, 117440512 padded size):
762
+ module.decoder.layers.11.mlp.experts.linear_fc1.weight0
763
+ Params for bucket 11 (58720256 elements, 58720256 padded size):
764
+ module.decoder.layers.10.mlp.experts.linear_fc2.weight0
765
+ Params for bucket 12 (117440512 elements, 117440512 padded size):
766
+ module.decoder.layers.10.mlp.experts.linear_fc1.weight0
767
+ Params for bucket 13 (58720256 elements, 58720256 padded size):
768
+ module.decoder.layers.9.mlp.experts.linear_fc2.weight0
769
+ Params for bucket 14 (117440512 elements, 117440512 padded size):
770
+ module.decoder.layers.9.mlp.experts.linear_fc1.weight0
771
+ Params for bucket 15 (58720256 elements, 58720256 padded size):
772
+ module.decoder.layers.8.mlp.experts.linear_fc2.weight0
773
+ Params for bucket 16 (117440512 elements, 117440512 padded size):
774
+ module.decoder.layers.8.mlp.experts.linear_fc1.weight0
775
+ Params for bucket 17 (58720256 elements, 58720256 padded size):
776
+ module.decoder.layers.7.mlp.experts.linear_fc2.weight0
777
+ Params for bucket 18 (117440512 elements, 117440512 padded size):
778
+ module.decoder.layers.7.mlp.experts.linear_fc1.weight0
779
+ Params for bucket 19 (58720256 elements, 58720256 padded size):
780
+ module.decoder.layers.6.mlp.experts.linear_fc2.weight0
781
+ Params for bucket 20 (117440512 elements, 117440512 padded size):
782
+ module.decoder.layers.6.mlp.experts.linear_fc1.weight0
783
+ Params for bucket 21 (58720256 elements, 58720256 padded size):
784
+ module.decoder.layers.5.mlp.experts.linear_fc2.weight0
785
+ Params for bucket 22 (117440512 elements, 117440512 padded size):
786
+ module.decoder.layers.5.mlp.experts.linear_fc1.weight0
787
+ Params for bucket 23 (58720256 elements, 58720256 padded size):
788
+ module.decoder.layers.4.mlp.experts.linear_fc2.weight0
789
+ Params for bucket 24 (117440512 elements, 117440512 padded size):
790
+ module.decoder.layers.4.mlp.experts.linear_fc1.weight0
791
+ Params for bucket 25 (58720256 elements, 58720256 padded size):
792
+ module.decoder.layers.3.mlp.experts.linear_fc2.weight0
793
+ Params for bucket 26 (117440512 elements, 117440512 padded size):
794
+ module.decoder.layers.3.mlp.experts.linear_fc1.weight0
795
+ Params for bucket 27 (58720256 elements, 58720256 padded size):
796
+ module.decoder.layers.2.mlp.experts.linear_fc2.weight0
797
+ Params for bucket 28 (117440512 elements, 117440512 padded size):
798
+ module.decoder.layers.2.mlp.experts.linear_fc1.weight0
799
+ Params for bucket 29 (58720256 elements, 58720256 padded size):
800
+ module.decoder.layers.1.mlp.experts.linear_fc2.weight0
801
+ Params for bucket 30 (117440512 elements, 117440512 padded size):
802
+ module.decoder.layers.1.mlp.experts.linear_fc1.weight0
803
+ Params for bucket 31 (58720256 elements, 58720256 padded size):
804
+ module.decoder.layers.0.mlp.experts.linear_fc2.weight0
805
+ Params for bucket 32 (117440512 elements, 117440512 padded size):
806
+ module.decoder.layers.0.mlp.experts.linear_fc1.weight0
807
+ INFO:megatron.core.optimizer:Setting up optimizer with config OptimizerConfig(optimizer='adam', lr=0.0001, min_lr=1e-05, decoupled_lr=None, decoupled_min_lr=None, weight_decay=0.1, fp8_recipe='mxfp8', fp16=False, bf16=True, reuse_grad_buf_for_mxfp8_param_ag=False, params_dtype=torch.bfloat16, use_precision_aware_optimizer=False, store_param_remainders=True, main_grads_dtype=torch.float32, main_params_dtype=torch.float32, exp_avg_dtype=torch.float32, exp_avg_sq_dtype=torch.float32, loss_scale=None, initial_loss_scale=4294967296, min_loss_scale=1.0, loss_scale_window=1000, hysteresis=2, adam_beta1=0.9, adam_beta2=0.999, adam_eps=1e-08, sgd_momentum=0.9, use_distributed_optimizer=True, overlap_param_gather_with_optimizer_step=False, optimizer_cpu_offload=False, optimizer_offload_fraction=1.0, use_torch_optimizer_for_cpu_offload=False, overlap_cpu_optimizer_d2h_h2d=False, pin_cpu_grads=True, pin_cpu_params=True, clip_grad=1.0, log_num_zeros_in_grad=False, barrier_with_L1_time=True, timers=<megatron.core.timers.Timers object at 0x7d521f580590>, config_logger_dir='')
808
+ INFO:megatron.core.optimizer_param_scheduler:> learning rate decay style: cosine
809
+ [after model, optimizer, and learning rate scheduler are built] datetime: 2026-08-30 06:47:29
810
+ > building train, validation, and test datasets ...
811
+ > datasets target sizes (minimum size):
812
+ train: 192
813
+ validation: 0
814
+ test: 0
815
+ INFO:megatron.core.datasets.blended_megatron_dataset_config:Let split_matrix = [(0, 1.0), None, None]
816
+ > building train, validation, and test datasets for GPT ...
817
+ INFO:megatron.core.datasets.blended_megatron_dataset_builder:Building GPTDataset splits with sizes=(192, 0, 0) and config=GPTDatasetConfig(random_seed=1234, sequence_length=4096, blend=(['/workspace/data/wikitext/wikitext103_gpt2_text_document'], None), blend_per_split=None, split='100,0,0', split_matrix=[(0, 1.0), None, None], num_dataset_builder_threads=1, path_to_cache=None, mmap_bin_files=True, mock=False, tokenizer=<megatron.training.tokenizer.tokenizer._GPT2BPETokenizer object at 0x7d5222159eb0>, mid_level_dataset_surplus=0.005, reset_position_ids=False, reset_attention_mask=False, eod_mask_loss=False, create_attention_mask=True, drop_last_partial_validation_sequence=True, add_extra_token_to_sequence=True, object_storage_cache_path=None)
818
+ INFO:megatron.core.datasets.indexed_dataset:Load the _IndexReader from /workspace/data/wikitext/wikitext103_gpt2_text_document.idx
819
+ INFO:megatron.core.datasets.indexed_dataset: Extract the sequence lengths
820
+ INFO:megatron.core.datasets.indexed_dataset: Extract the sequence pointers
821
+ INFO:megatron.core.datasets.indexed_dataset: Extract the document indices
822
+ INFO:megatron.core.datasets.indexed_dataset:> total number of sequences: 1165029
823
+ INFO:megatron.core.datasets.indexed_dataset:> total number of documents: 1801350
824
+ INFO:megatron.core.datasets.gpt_dataset:Load the GPTDataset train indices
825
+ INFO:megatron.core.datasets.gpt_dataset: Load the document index from 44ed1ee5bcb9e5b030387b8d27a6c406-GPTDataset-train-document_index.npy
826
+ INFO:megatron.core.datasets.gpt_dataset: Load the sample index from 44ed1ee5bcb9e5b030387b8d27a6c406-GPTDataset-train-sample_index.npy
827
+ INFO:megatron.core.datasets.gpt_dataset: Load the shuffle index from 44ed1ee5bcb9e5b030387b8d27a6c406-GPTDataset-train-shuffle_index.npy
828
+ INFO:megatron.core.datasets.gpt_dataset:> total number of samples: 29073
829
+ > finished creating GPT datasets ...
830
+ [after dataloaders are built] datetime: 2026-08-30 06:47:29
831
+ done with setup ...
832
+ (min, max) time across ranks (ms):
833
+ model-and-optimizer-setup ......................: (203.77, 206.93)
834
+ train/valid/test-data-iterators-setup ..........: (59.74, 71.63)
835
+ training ...
836
+ Overwriting rerun_state_machine.current_iteration from -1 to 0...
837
+ [before the start of training step] datetime: 2026-08-30 06:47:29
838
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:334: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
839
+ warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
840
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:334: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
841
+ warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
842
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:334: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
843
+ warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
844
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:334: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
845
+ warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
846
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:336: UserWarning: The use_host_meta is deprecated, it will be removed in the future.
847
+ warnings.warn("The use_host_meta is deprecated, it will be removed in the future.")
848
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:336: UserWarning: The use_host_meta is deprecated, it will be removed in the future.
849
+ warnings.warn("The use_host_meta is deprecated, it will be removed in the future.")
850
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:334: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
851
+ warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
852
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:334: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
853
+ warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
854
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:336: UserWarning: The use_host_meta is deprecated, it will be removed in the future.
855
+ warnings.warn("The use_host_meta is deprecated, it will be removed in the future.")
856
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:334: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
857
+ warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
858
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:336: UserWarning: The use_host_meta is deprecated, it will be removed in the future.
859
+ warnings.warn("The use_host_meta is deprecated, it will be removed in the future.")
860
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:334: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
861
+ warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
862
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:336: UserWarning: The use_host_meta is deprecated, it will be removed in the future.
863
+ warnings.warn("The use_host_meta is deprecated, it will be removed in the future.")
864
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:336: UserWarning: The use_host_meta is deprecated, it will be removed in the future.
865
+ warnings.warn("The use_host_meta is deprecated, it will be removed in the future.")
866
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:336: UserWarning: The use_host_meta is deprecated, it will be removed in the future.
867
+ warnings.warn("The use_host_meta is deprecated, it will be removed in the future.")
868
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:336: UserWarning: The use_host_meta is deprecated, it will be removed in the future.
869
+ warnings.warn("The use_host_meta is deprecated, it will be removed in the future.")
870
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:482: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
871
+ warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
872
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:482: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
873
+ warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
874
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:482: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
875
+ warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
876
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:482: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
877
+ warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
878
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:482: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
879
+ warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
880
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:482: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
881
+ warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
882
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:482: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
883
+ warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
884
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:482: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
885
+ warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
886
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:334: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
887
+ warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
888
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:336: UserWarning: The use_host_meta is deprecated, it will be removed in the future.
889
+ warnings.warn("The use_host_meta is deprecated, it will be removed in the future.")
890
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:334: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
891
+ warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
892
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:336: UserWarning: The use_host_meta is deprecated, it will be removed in the future.
893
+ warnings.warn("The use_host_meta is deprecated, it will be removed in the future.")
894
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:334: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
895
+ warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
896
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:336: UserWarning: The use_host_meta is deprecated, it will be removed in the future.
897
+ warnings.warn("The use_host_meta is deprecated, it will be removed in the future.")
898
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:334: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
899
+ warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
900
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:336: UserWarning: The use_host_meta is deprecated, it will be removed in the future.
901
+ warnings.warn("The use_host_meta is deprecated, it will be removed in the future.")
902
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:334: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
903
+ warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
904
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:336: UserWarning: The use_host_meta is deprecated, it will be removed in the future.
905
+ warnings.warn("The use_host_meta is deprecated, it will be removed in the future.")
906
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:334: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
907
+ warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
908
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:336: UserWarning: The use_host_meta is deprecated, it will be removed in the future.
909
+ warnings.warn("The use_host_meta is deprecated, it will be removed in the future.")
910
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:334: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
911
+ warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
912
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:336: UserWarning: The use_host_meta is deprecated, it will be removed in the future.
913
+ warnings.warn("The use_host_meta is deprecated, it will be removed in the future.")
914
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:334: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
915
+ warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
916
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:336: UserWarning: The use_host_meta is deprecated, it will be removed in the future.
917
+ warnings.warn("The use_host_meta is deprecated, it will be removed in the future.")
918
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:482: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
919
+ warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
920
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:482: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
921
+ warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
922
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:482: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
923
+ warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
924
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:482: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
925
+ warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
926
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:482: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
927
+ warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
928
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:482: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
929
+ warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
930
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:482: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
931
+ warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
932
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:482: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
933
+ warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
934
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:334: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
935
+ warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
936
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:334: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
937
+ warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
938
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:334: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
939
+ warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
940
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:334: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
941
+ warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
942
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:334: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
943
+ warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
944
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:334: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
945
+ warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
946
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:334: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
947
+ warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
948
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:334: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
949
+ warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
950
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:336: UserWarning: The use_host_meta is deprecated, it will be removed in the future.
951
+ warnings.warn("The use_host_meta is deprecated, it will be removed in the future.")
952
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:336: UserWarning: The use_host_meta is deprecated, it will be removed in the future.
953
+ warnings.warn("The use_host_meta is deprecated, it will be removed in the future.")
954
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:336: UserWarning: The use_host_meta is deprecated, it will be removed in the future.
955
+ warnings.warn("The use_host_meta is deprecated, it will be removed in the future.")
956
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:336: UserWarning: The use_host_meta is deprecated, it will be removed in the future.
957
+ warnings.warn("The use_host_meta is deprecated, it will be removed in the future.")
958
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:336: UserWarning: The use_host_meta is deprecated, it will be removed in the future.
959
+ warnings.warn("The use_host_meta is deprecated, it will be removed in the future.")
960
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:336: UserWarning: The use_host_meta is deprecated, it will be removed in the future.
961
+ warnings.warn("The use_host_meta is deprecated, it will be removed in the future.")
962
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:336: UserWarning: The use_host_meta is deprecated, it will be removed in the future.
963
+ warnings.warn("The use_host_meta is deprecated, it will be removed in the future.")
964
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:336: UserWarning: The use_host_meta is deprecated, it will be removed in the future.
965
+ warnings.warn("The use_host_meta is deprecated, it will be removed in the future.")
966
+ [2026-08-30 06:48:07] iteration 1/ 12 | consumed samples: 16 | elapsed time per iteration (ms): 37805.9 | throughput per GPU (TFLOP/s/GPU): 8.8 | learning rate: 2.000000E-05 | global batch size: 16 | lm loss: 1.104376E+01 | loss scale: 1.0 | grad norm: 34.296 | number of skipped iterations: 0 | number of nan iterations: 0 |
967
+ Number of parameters in transformer block in billions: 23.22
968
+ Number of parameters in embedding layers in billions: 0.41
969
+ Total number of parameters in billions: 23.63
970
+ Number of parameters in most loaded shard in billions: 23.6320
971
+ Theoretical memory footprints: weight and optimizer=169029.43 MB
972
+ [Rank 0] (after 1 iterations) memory (MB) | allocated: 63795.404296875 | max allocated: 96519.67578125 | reserved: 99122.0 | max reserved: 99122.0
973
+ [2026-08-30 06:48:07] iteration 2/ 12 | consumed samples: 32 | elapsed time per iteration (ms): 570.6 | throughput per GPU (TFLOP/s/GPU): 584.3 | learning rate: 4.000000E-05 | global batch size: 16 | lm loss: 1.104935E+01 | loss scale: 1.0 | grad norm: 35.986 | number of skipped iterations: 0 | number of nan iterations: 0 |
974
+ [2026-08-30 06:48:08] iteration 3/ 12 | consumed samples: 48 | elapsed time per iteration (ms): 557.9 | throughput per GPU (TFLOP/s/GPU): 597.6 | learning rate: 6.000000E-05 | global batch size: 16 | lm loss: 9.657968E+00 | loss scale: 1.0 | grad norm: 40.332 | number of skipped iterations: 0 | number of nan iterations: 0 |
975
+ Capture CUDA graph for training!!!
976
+ CUDA graph capture done!!!
977
+ [2026-08-30 06:48:09] iteration 4/ 12 | consumed samples: 64 | elapsed time per iteration (ms): 1653.3 | throughput per GPU (TFLOP/s/GPU): 201.6 | learning rate: 8.000000E-05 | global batch size: 16 | lm loss: 1.081336E+01 | loss scale: 1.0 | grad norm: 93.363 | number of skipped iterations: 0 | number of nan iterations: 0 |
978
+ [2026-08-30 06:48:10] iteration 5/ 12 | consumed samples: 80 | elapsed time per iteration (ms): 469.0 | throughput per GPU (TFLOP/s/GPU): 710.9 | learning rate: 1.000000E-04 | global batch size: 16 | lm loss: 1.068684E+01 | loss scale: 1.0 | grad norm: 17.191 | number of skipped iterations: 0 | number of nan iterations: 0 |
979
+ [2026-08-30 06:48:10] iteration 6/ 12 | consumed samples: 96 | elapsed time per iteration (ms): 467.1 | throughput per GPU (TFLOP/s/GPU): 713.7 | learning rate: 7.750000E-05 | global batch size: 16 | lm loss: 9.917974E+00 | loss scale: 1.0 | grad norm: 42.077 | number of skipped iterations: 0 | number of nan iterations: 0 |
980
+ [2026-08-30 06:48:11] iteration 7/ 12 | consumed samples: 112 | elapsed time per iteration (ms): 472.2 | throughput per GPU (TFLOP/s/GPU): 706.0 | learning rate: 3.250000E-05 | global batch size: 16 | lm loss: 9.323465E+00 | loss scale: 1.0 | grad norm: 7.381 | number of skipped iterations: 0 | number of nan iterations: 0 |
981
+ [2026-08-30 06:48:11] iteration 8/ 12 | consumed samples: 128 | elapsed time per iteration (ms): 466.8 | throughput per GPU (TFLOP/s/GPU): 714.2 | learning rate: 1.000000E-05 | global batch size: 16 | lm loss: 9.001214E+00 | loss scale: 1.0 | grad norm: 19.429 | number of skipped iterations: 0 | number of nan iterations: 0 |
982
+ [2026-08-30 06:48:12] iteration 9/ 12 | consumed samples: 144 | elapsed time per iteration (ms): 465.3 | throughput per GPU (TFLOP/s/GPU): 716.4 | learning rate: 1.000000E-05 | global batch size: 16 | lm loss: 8.649192E+00 | loss scale: 1.0 | grad norm: 5.915 | number of skipped iterations: 0 | number of nan iterations: 0 |
983
+ [2026-08-30 06:48:12] iteration 10/ 12 | consumed samples: 160 | elapsed time per iteration (ms): 461.8 | throughput per GPU (TFLOP/s/GPU): 721.9 | learning rate: 1.000000E-05 | global batch size: 16 | lm loss: 8.543521E+00 | loss scale: 1.0 | grad norm: 5.254 | number of skipped iterations: 0 | number of nan iterations: 0 |
984
+ [2026-08-30 06:48:13] iteration 11/ 12 | consumed samples: 176 | elapsed time per iteration (ms): 467.3 | throughput per GPU (TFLOP/s/GPU): 713.4 | learning rate: 1.000000E-05 | global batch size: 16 | lm loss: 8.433981E+00 | loss scale: 1.0 | grad norm: 4.838 | number of skipped iterations: 0 | number of nan iterations: 0 |
985
+ [2026-08-30 06:48:13] iteration 12/ 12 | consumed samples: 192 | elapsed time per iteration (ms): 471.2 | throughput per GPU (TFLOP/s/GPU): 707.4 | learning rate: 1.000000E-05 | global batch size: 16 | lm loss: 8.369435E+00 | loss scale: 1.0 | grad norm: 4.479 | number of skipped iterations: 0 | number of nan iterations: 0 |
986
+ [after training is done] datetime: 2026-08-30 06:48:13
987
+ W0830 06:48:14.145000 658758 torch/distributed/elastic/agent/server/api.py:719] Received 15 death signal, shutting down workers
988
+ W0830 06:48:14.146000 658758 torch/distributed/elastic/multiprocessing/api.py:900] Sending process 658847 closing signal SIGTERM
989
+ W0830 06:48:14.147000 658758 torch/distributed/elastic/multiprocessing/api.py:900] Sending process 658848 closing signal SIGTERM
990
+ W0830 06:48:14.149000 658758 torch/distributed/elastic/multiprocessing/api.py:900] Sending process 658849 closing signal SIGTERM
991
+ W0830 06:48:14.151000 658758 torch/distributed/elastic/multiprocessing/api.py:900] Sending process 658850 closing signal SIGTERM
992
+ W0830 06:48:14.153000 658758 torch/distributed/elastic/multiprocessing/api.py:900] Sending process 658851 closing signal SIGTERM
993
+ W0830 06:48:14.155000 658758 torch/distributed/elastic/multiprocessing/api.py:900] Sending process 658852 closing signal SIGTERM
994
+ W0830 06:48:14.157000 658758 torch/distributed/elastic/multiprocessing/api.py:900] Sending process 658853 closing signal SIGTERM
995
+ W0830 06:48:14.159000 658758 torch/distributed/elastic/multiprocessing/api.py:900] Sending process 658854 closing signal SIGTERM
996
+ Traceback (most recent call last):
997
+ File "/usr/local/bin/torchrun", line 8, in <module>
998
+ sys.exit(main())
999
+ ^^^^^^
1000
+ File "/usr/local/lib/python3.12/dist-packages/torch/distributed/elastic/multiprocessing/errors/__init__.py", line 355, in wrapper
1001
+ return f(*args, **kwargs)
1002
+ ^^^^^^^^^^^^^^^^^^
1003
+ File "/usr/local/lib/python3.12/dist-packages/torch/distributed/run.py", line 892, in main
1004
+ run(args)
1005
+ File "/usr/local/lib/python3.12/dist-packages/torch/distributed/run.py", line 883, in run
1006
+ elastic_launch(
1007
+ File "/usr/local/lib/python3.12/dist-packages/torch/distributed/launcher/api.py", line 139, in __call__
1008
+ return launch_agent(self._config, self._entrypoint, list(args))
1009
+ ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
1010
+ File "/usr/local/lib/python3.12/dist-packages/torch/distributed/launcher/api.py", line 261, in launch_agent
1011
+ result = agent.run()
1012
+ ^^^^^^^^^^^
1013
+ File "/usr/local/lib/python3.12/dist-packages/torch/distributed/elastic/metrics/api.py", line 138, in wrapper
1014
+ result = f(*args, **kwargs)
1015
+ ^^^^^^^^^^^^^^^^^^
1016
+ File "/usr/local/lib/python3.12/dist-packages/torch/distributed/elastic/agent/server/api.py", line 711, in run
1017
+ result = self._invoke_run(role)
1018
+ ^^^^^^^^^^^^^^^^^^^^^^
1019
+ File "/usr/local/lib/python3.12/dist-packages/torch/distributed/elastic/agent/server/api.py", line 870, in _invoke_run
1020
+ time.sleep(monitor_interval)
1021
+ File "/usr/local/lib/python3.12/dist-packages/torch/distributed/elastic/multiprocessing/api.py", line 84, in _terminate_process_handler
1022
+ raise SignalException(f"Process {os.getpid()} got signal: {sigval}", sigval=sigval)
1023
+ torch.distributed.elastic.multiprocessing.api.SignalException: Process 658758 got signal: 15
tiera-rerun-fixed-moonep/echo/a2/a2_matched_gbs16/G2__echo__matched__seq4096__gbs16__cap1p08203125/rep1/mem_samples.csv ADDED
@@ -0,0 +1,256 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ 0, 0
2
+ 1, 0
3
+ 2, 0
4
+ 3, 0
5
+ 4, 0
6
+ 5, 0
7
+ 6, 0
8
+ 7, 0
9
+ 0, 0
10
+ 1, 0
11
+ 2, 0
12
+ 3, 0
13
+ 4, 0
14
+ 5, 0
15
+ 6, 0
16
+ 7, 0
17
+ 0, 1
18
+ 1, 1
19
+ 2, 1
20
+ 3, 1
21
+ 4, 1
22
+ 5, 1
23
+ 6, 1
24
+ 7, 1
25
+ 0, 4
26
+ 1, 4
27
+ 2, 4
28
+ 3, 4
29
+ 4, 1
30
+ 5, 1
31
+ 6, 1
32
+ 7, 1
33
+ 0, 4
34
+ 1, 4
35
+ 2, 4
36
+ 3, 4
37
+ 4, 4
38
+ 5, 4
39
+ 6, 4
40
+ 7, 4
41
+ 0, 4
42
+ 1, 629
43
+ 2, 629
44
+ 3, 629
45
+ 4, 629
46
+ 5, 629
47
+ 6, 629
48
+ 7, 629
49
+ 0, 891
50
+ 1, 899
51
+ 2, 887
52
+ 3, 887
53
+ 4, 885
54
+ 5, 885
55
+ 6, 899
56
+ 7, 891
57
+ 0, 1895
58
+ 1, 1895
59
+ 2, 1895
60
+ 3, 1895
61
+ 4, 1895
62
+ 5, 1895
63
+ 6, 1895
64
+ 7, 1895
65
+ 0, 39401
66
+ 1, 39365
67
+ 2, 39379
68
+ 3, 39417
69
+ 4, 39345
70
+ 5, 39397
71
+ 6, 39439
72
+ 7, 39409
73
+ 0, 39785
74
+ 1, 39741
75
+ 2, 39767
76
+ 3, 39773
77
+ 4, 39757
78
+ 5, 39803
79
+ 6, 39789
80
+ 7, 39779
81
+ 0, 42297
82
+ 1, 42289
83
+ 2, 42289
84
+ 3, 42289
85
+ 4, 42289
86
+ 5, 42289
87
+ 6, 42289
88
+ 7, 42289
89
+ 0, 42541
90
+ 1, 42533
91
+ 2, 42533
92
+ 3, 42533
93
+ 4, 42533
94
+ 5, 42533
95
+ 6, 42533
96
+ 7, 42533
97
+ 0, 42709
98
+ 1, 42695
99
+ 2, 42699
100
+ 3, 42703
101
+ 4, 42735
102
+ 5, 42715
103
+ 6, 42725
104
+ 7, 42713
105
+ 0, 43457
106
+ 1, 43449
107
+ 2, 43449
108
+ 3, 43449
109
+ 4, 43449
110
+ 5, 43449
111
+ 6, 43449
112
+ 7, 43449
113
+ 0, 43735
114
+ 1, 43727
115
+ 2, 43727
116
+ 3, 43727
117
+ 4, 43727
118
+ 5, 43727
119
+ 6, 43727
120
+ 7, 43727
121
+ 0, 45167
122
+ 1, 45159
123
+ 2, 45159
124
+ 3, 45159
125
+ 4, 45159
126
+ 5, 45159
127
+ 6, 45159
128
+ 7, 45159
129
+ 0, 45167
130
+ 1, 45159
131
+ 2, 45159
132
+ 3, 45159
133
+ 4, 45159
134
+ 5, 45159
135
+ 6, 45159
136
+ 7, 45159
137
+ 0, 75747
138
+ 1, 75739
139
+ 2, 75739
140
+ 3, 75739
141
+ 4, 75739
142
+ 5, 75739
143
+ 6, 75739
144
+ 7, 75739
145
+ 0, 76233
146
+ 1, 76225
147
+ 2, 76225
148
+ 3, 76225
149
+ 4, 76225
150
+ 5, 76225
151
+ 6, 76225
152
+ 7, 76225
153
+ 0, 76233
154
+ 1, 76225
155
+ 2, 76225
156
+ 3, 76225
157
+ 4, 76225
158
+ 5, 76225
159
+ 6, 76225
160
+ 7, 76225
161
+ 0, 76233
162
+ 1, 76225
163
+ 2, 76225
164
+ 3, 76225
165
+ 4, 76225
166
+ 5, 76225
167
+ 6, 76225
168
+ 7, 76225
169
+ 0, 94137
170
+ 1, 94155
171
+ 2, 94127
172
+ 3, 94147
173
+ 4, 94155
174
+ 5, 94159
175
+ 6, 94175
176
+ 7, 94165
177
+ 0, 94853
178
+ 1, 94833
179
+ 2, 94887
180
+ 3, 94899
181
+ 4, 94925
182
+ 5, 94879
183
+ 6, 94923
184
+ 7, 94955
185
+ 0, 95613
186
+ 1, 95605
187
+ 2, 95605
188
+ 3, 95605
189
+ 4, 95605
190
+ 5, 95605
191
+ 6, 95605
192
+ 7, 95605
193
+ 0, 96223
194
+ 1, 96235
195
+ 2, 96225
196
+ 3, 96233
197
+ 4, 96237
198
+ 5, 96223
199
+ 6, 96227
200
+ 7, 96249
201
+ 0, 77247
202
+ 1, 106731
203
+ 2, 77251
204
+ 3, 98447
205
+ 4, 76989
206
+ 5, 77251
207
+ 6, 77235
208
+ 7, 77035
209
+ 0, 145689
210
+ 1, 145681
211
+ 2, 145683
212
+ 3, 145459
213
+ 4, 145461
214
+ 5, 145683
215
+ 6, 145681
216
+ 7, 145455
217
+ 0, 145689
218
+ 1, 145681
219
+ 2, 145683
220
+ 3, 145459
221
+ 4, 145461
222
+ 5, 145683
223
+ 6, 145681
224
+ 7, 145455
225
+ 0, 145689
226
+ 1, 145681
227
+ 2, 145683
228
+ 3, 145459
229
+ 4, 145461
230
+ 5, 145683
231
+ 6, 145681
232
+ 7, 145455
233
+ 0, 144773
234
+ 1, 144765
235
+ 2, 144767
236
+ 3, 144543
237
+ 4, 144545
238
+ 5, 144767
239
+ 6, 3398
240
+ 7, 144539
241
+ 0, 2115
242
+ 1, 144762
243
+ 2, 3139
244
+ 3, 1862
245
+ 4, 1862
246
+ 5, 1862
247
+ 6, 1862
248
+ 7, 1862
249
+ 0, 0
250
+ 1, 0
251
+ 2, 0
252
+ 3, 0
253
+ 4, 0
254
+ 5, 0
255
+ 6, 0
256
+ 7, 0
tiera-rerun-fixed-moonep/echo/a2/a2_matched_gbs16/G2__echo__matched__seq4096__gbs16__cap1p08203125/rep1/run.env ADDED
@@ -0,0 +1,70 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ backend=echo
2
+ config=G2
3
+ family=tierA
4
+ repo=/workspace/Megatron-MoE-Echo
5
+ branch=bracing-regions-through-kernels
6
+ commit=9a236793
7
+ num_layers=16
8
+ hidden=4096
9
+ ffn_hidden=14336
10
+ moe_ffn_hidden=3584
11
+ num_experts=32
12
+ topk=8
13
+ experts_per_rank=4
14
+ expert_params=44040192
15
+ heads=32
16
+ kv_groups=8
17
+ kv_channels=
18
+ qk_layernorm=0
19
+ shared_ffn=0
20
+ shared_expert_overlap=0
21
+ moe_token_dispatcher=flex
22
+ mla=0
23
+ dense_first_layers=0
24
+ moe_layers=16
25
+ rotary_base=1000000
26
+ rotary_scaling=
27
+ qkv_bias=0
28
+ router_balancing=aux_loss
29
+ aux_loss_coeff=1e-2
30
+ router_pre_softmax=0
31
+ router_fusion=0
32
+ router_dtype=fp32
33
+ device_initiated_grouped_gemm=1
34
+ norm_epsilon=1e-5
35
+ seq_length=4096
36
+ micro_batch_size=1
37
+ global_batch_size=16
38
+ num_microbatches=2
39
+ assignments_per_rank=32768
40
+ mean_tokens_per_expert=8192
41
+ slot_policy=matched
42
+ slots_per_rank=4
43
+ capacity_mode=static
44
+ capacity=1.08203125
45
+ cuda_graph=1
46
+ probe_mode=0
47
+ async_grad_reduce=0
48
+ branch_role=timing
49
+ memory_dump_mode=none
50
+ memory_snapshot_path=
51
+ cuda_graph_warmup_steps=3
52
+ recompute=none
53
+ force_balance=0
54
+ train_iters=12
55
+ ep_size=8
56
+ echo_dispatch_overlap=1
57
+ echo_dispatch_recompute=0
58
+ ultraep_quota_min_tokens=
59
+ ultraep_weight_sync_join=
60
+ profile_enable=0
61
+ kernel_trace=0
62
+ fp8_mode=mxfp8
63
+ fp8_recipe=mxfp8
64
+ training_seed=1234
65
+ learning_rate=1e-4
66
+ min_learning_rate=1.0e-5
67
+ lr_warmup_iters=5
68
+ state_dump_enable=0
69
+ state_dump_dir=
70
+ moonep_assert_no_pending_reduce=0
tiera-rerun-fixed-moonep/echo/a2/a2_matched_gbs16/G2__echo__matched__seq4096__gbs16__cap1p08203125/rep1/train.log ADDED
@@ -0,0 +1,1107 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ==============================================================
2
+ G2 / echo (slots/rank=4 policy=matched)
3
+ E=32 K=8 H'=3584 m=4
4
+ L=16 seq=4096 gbs=16 (2 microbatches)
5
+ cap_mode=static cap=1.08203125 cg=1 recompute=none
6
+ memory_dump=none snapshots=/workspace/tiera-rerun-fixed-moonep/echo/a2/a2_matched_gbs16/G2__echo__matched__seq4096__gbs16__cap1p08203125/rep1/memory_snapshots/snapshot.pickle
7
+ repo=/workspace/Megatron-MoE-Echo @ 9a236793
8
+ ==============================================================
9
+ W0830 06:48:20.670000 662151 torch/distributed/run.py:766]
10
+ W0830 06:48:20.670000 662151 torch/distributed/run.py:766] *****************************************
11
+ W0830 06:48:20.670000 662151 torch/distributed/run.py:766] Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed.
12
+ W0830 06:48:20.670000 662151 torch/distributed/run.py:766] *****************************************
13
+ /workspace/Megatron-MoE-Echo/megatron/training/arguments.py:831: UserWarning: Disabling sequence parallelism because tensor model parallelism is disabled
14
+ warnings.warn("Disabling sequence parallelism because tensor model parallelism is disabled")
15
+ /workspace/Megatron-MoE-Echo/megatron/training/arguments.py:831: UserWarning: Disabling sequence parallelism because tensor model parallelism is disabled
16
+ warnings.warn("Disabling sequence parallelism because tensor model parallelism is disabled")
17
+ /workspace/Megatron-MoE-Echo/megatron/training/arguments.py:831: UserWarning: Disabling sequence parallelism because tensor model parallelism is disabled
18
+ warnings.warn("Disabling sequence parallelism because tensor model parallelism is disabled")
19
+ /workspace/Megatron-MoE-Echo/megatron/training/arguments.py:831: UserWarning: Disabling sequence parallelism because tensor model parallelism is disabled
20
+ warnings.warn("Disabling sequence parallelism because tensor model parallelism is disabled")
21
+ /workspace/Megatron-MoE-Echo/megatron/training/arguments.py:831: UserWarning: Disabling sequence parallelism because tensor model parallelism is disabled
22
+ warnings.warn("Disabling sequence parallelism because tensor model parallelism is disabled")
23
+ /workspace/Megatron-MoE-Echo/megatron/training/arguments.py:831: UserWarning: Disabling sequence parallelism because tensor model parallelism is disabled
24
+ warnings.warn("Disabling sequence parallelism because tensor model parallelism is disabled")
25
+ /workspace/Megatron-MoE-Echo/megatron/training/arguments.py:831: UserWarning: Disabling sequence parallelism because tensor model parallelism is disabled
26
+ warnings.warn("Disabling sequence parallelism because tensor model parallelism is disabled")
27
+ using world size: 8, data-parallel size: 8, context-parallel size: 1, hierarchical context-parallel sizes: None, tensor-model-parallel size: 1, pipeline-model-parallel size: 1
28
+ WARNING: overriding default arguments for tokenizer_type:GPT2BPETokenizer with tokenizer_type:GPT2BPETokenizer
29
+ Number of virtual stages per pipeline stage: None
30
+ accumulate and all-reduce gradients in fp32 for bfloat16 data type.
31
+ using torch.bfloat16 for parameters ...
32
+ /workspace/Megatron-MoE-Echo/megatron/training/arguments.py:831: UserWarning: Disabling sequence parallelism because tensor model parallelism is disabled
33
+ warnings.warn("Disabling sequence parallelism because tensor model parallelism is disabled")
34
+ ------------------------ arguments ------------------------
35
+ account_for_embedding_in_pipeline_split ......... False
36
+ account_for_loss_in_pipeline_split .............. False
37
+ accumulate_allreduce_grads_in_fp32 .............. True
38
+ adam_beta1 ...................................... 0.9
39
+ adam_beta2 ...................................... 0.999
40
+ adam_eps ........................................ 1e-08
41
+ add_bias_linear ................................. False
42
+ add_position_embedding .......................... False
43
+ add_qkv_bias .................................... False
44
+ adlr_autoresume ................................. False
45
+ adlr_autoresume_interval ........................ 1000
46
+ align_grad_reduce ............................... True
47
+ align_param_gather .............................. False
48
+ app_tag_run_name ................................ None
49
+ app_tag_run_version ............................. 0.0.0
50
+ apply_layernorm_1p .............................. False
51
+ apply_query_key_layer_scaling ................... False
52
+ apply_residual_connection_post_layernorm ........ False
53
+ apply_rope_fusion ............................... True
54
+ async_save ...................................... None
55
+ async_tensor_model_parallel_allreduce ........... True
56
+ attention_backend ............................... AttnBackend.auto
57
+ attention_dropout ............................... 0.0
58
+ attention_softmax_in_fp32 ....................... False
59
+ auto_detect_ckpt_format ......................... False
60
+ barrier_with_L1_time ............................ True
61
+ bert_binary_head ................................ True
62
+ bert_embedder_type .............................. megatron
63
+ bert_load ....................................... None
64
+ bf16 ............................................ True
65
+ bias_dropout_fusion ............................. True
66
+ bias_gelu_fusion ................................ False
67
+ bias_swiglu_fusion .............................. True
68
+ biencoder_projection_dim ........................ 0
69
+ biencoder_shared_query_context_model ............ False
70
+ block_data_path ................................. None
71
+ calc_ft_timeouts ................................ False
72
+ calculate_per_token_loss ........................ False
73
+ check_for_large_grads ........................... False
74
+ check_for_nan_in_loss_and_grad .................. False
75
+ check_for_spiky_loss ............................ False
76
+ check_weight_hash_across_dp_replicas_interval ... None
77
+ ckpt_assume_constant_structure .................. False
78
+ ckpt_convert_format ............................. None
79
+ ckpt_convert_save ............................... None
80
+ ckpt_convert_update_legacy_dist_opt_format ...... False
81
+ ckpt_format ..................................... torch_dist
82
+ ckpt_fully_parallel_load ........................ False
83
+ ckpt_fully_parallel_save ........................ True
84
+ ckpt_fully_parallel_save_deprecated ............. False
85
+ ckpt_step ....................................... None
86
+ classes_fraction ................................ 1.0
87
+ clip_grad ....................................... 1.0
88
+ clone_scatter_output_in_embedding ............... True
89
+ config_logger_dir ...............................
90
+ consumed_train_samples .......................... 0
91
+ consumed_valid_samples .......................... 0
92
+ context_parallel_size ........................... 1
93
+ cp_comm_type .................................... ['p2p']
94
+ create_attention_mask_in_dataloader ............. True
95
+ cross_entropy_fusion_impl ....................... native
96
+ cross_entropy_loss_fusion ....................... False
97
+ cuda_graph_scope ................................ full_iteration
98
+ cuda_graph_warmup_steps ......................... 3
99
+ data_args_path .................................. None
100
+ data_cache_path ................................. None
101
+ data_parallel_random_init ....................... False
102
+ data_parallel_sharding_strategy ................. no_shard
103
+ data_parallel_size .............................. 8
104
+ data_path ....................................... ['/workspace/data/wikitext/wikitext103_gpt2_text_document']
105
+ data_per_class_fraction ......................... 1.0
106
+ data_sharding ................................... True
107
+ dataloader_type ................................. single
108
+ ddp_average_in_collective ....................... False
109
+ ddp_bucket_size ................................. None
110
+ ddp_num_buckets ................................. None
111
+ ddp_pad_buckets_for_high_nccl_busbw ............. False
112
+ decoder_first_pipeline_num_layers ............... None
113
+ decoder_last_pipeline_num_layers ................ None
114
+ decoder_num_layers .............................. None
115
+ decoder_seq_length .............................. None
116
+ decoupled_lr .................................... None
117
+ decoupled_min_lr ................................ None
118
+ decrease_batch_size_if_needed ................... False
119
+ defer_embedding_wgrad_compute ................... False
120
+ delay_wgrad_compute ............................. False
121
+ deprecated_use_mcore_models ..................... True
122
+ deterministic_mode .............................. False
123
+ dino_bottleneck_size ............................ 256
124
+ dino_freeze_last_layer .......................... 1
125
+ dino_head_hidden_size ........................... 2048
126
+ dino_local_crops_number ......................... 10
127
+ dino_local_img_size ............................. 96
128
+ dino_norm_last_layer ............................ False
129
+ dino_teacher_temp ............................... 0.07
130
+ dino_warmup_teacher_temp ........................ 0.04
131
+ dino_warmup_teacher_temp_epochs ................. 30
132
+ disable_bf16_reduced_precision_matmul ........... False
133
+ disable_mamba_mem_eff_path ...................... False
134
+ disable_straggler_on_startup .................... False
135
+ dist_ckpt_format_deprecated ..................... None
136
+ dist_ckpt_strictness ............................ assume_ok_unexpected
137
+ distribute_saved_activations .................... False
138
+ distributed_backend ............................. nccl
139
+ distributed_timeout_minutes ..................... 10
140
+ embedding_init_method_std ....................... None
141
+ embedding_path .................................. None
142
+ empty_unused_memory_level ....................... 0
143
+ enable_cuda_graph ............................... True
144
+ enable_experimental ............................. False
145
+ enable_ft_package ............................... False
146
+ enable_gloo_process_groups ...................... True
147
+ enable_msc ...................................... True
148
+ enable_one_logger ............................... True
149
+ encoder_num_layers .............................. 16
150
+ encoder_seq_length .............................. 4096
151
+ end_weight_decay ................................ 0.1
152
+ eod_mask_loss ................................... False
153
+ error_injection_rate ............................ 0
154
+ error_injection_type ............................ transient_error
155
+ eval_interval ................................... 1000000
156
+ eval_iters ...................................... 0
157
+ evidence_data_path .............................. None
158
+ exit_duration_in_mins ........................... None
159
+ exit_interval ................................... None
160
+ exit_on_missing_checkpoint ...................... False
161
+ exit_signal_handler ............................. False
162
+ exp_avg_dtype ................................... torch.float32
163
+ exp_avg_sq_dtype ................................ torch.float32
164
+ expert_model_parallel_size ...................... 8
165
+ expert_tensor_parallel_size ..................... 1
166
+ export_force_local_attention .................... False
167
+ export_kd_cfg ................................... None
168
+ export_kd_teacher_ckpt_format ................... None
169
+ export_kd_teacher_load .......................... None
170
+ export_kv_cache_quant ........................... False
171
+ export_legacy_megatron .......................... False
172
+ export_model_type ............................... GPTModel
173
+ export_moe_apply_probs_on_input ................. False
174
+ export_qk_l2_norm ............................... False
175
+ export_quant_cfg ................................ None
176
+ export_real_quant_cfg ........................... None
177
+ export_te_mcore_model ........................... False
178
+ external_cuda_graph ............................. False
179
+ ffn_hidden_size ................................. 14336
180
+ finetune ........................................ False
181
+ finetune_data_split ............................. train
182
+ finetune_hf_dataset ............................. None
183
+ first_last_layers_bf16 .......................... False
184
+ flash_decode .................................... False
185
+ fp16 ............................................ False
186
+ fp16_lm_cross_entropy ........................... False
187
+ fp32_residual_connection ........................ False
188
+ fp8 ............................................. e4m3
189
+ fp8_amax_compute_algo ........................... most_recent
190
+ fp8_amax_history_len ............................ 1
191
+ fp8_interval .................................... 1
192
+ fp8_margin ...................................... 0
193
+ fp8_param_gather ................................ False
194
+ fp8_recipe ...................................... mxfp8
195
+ fp8_wgrad ....................................... True
196
+ fsdp_double_buffer .............................. False
197
+ global_batch_size ............................... 16
198
+ grad_reduce_in_bf16 ............................. False
199
+ gradient_accumulation_fusion .................... True
200
+ gradient_reduce_div_fusion ...................... True
201
+ group_query_attention ........................... True
202
+ head_lr_mult .................................... 1.0
203
+ heterogeneous_layers_config_encoded_json ........ None
204
+ heterogeneous_layers_config_path ................ None
205
+ hidden_dropout .................................. 0.0
206
+ hidden_size ..................................... 4096
207
+ hierarchical_context_parallel_sizes ............. None
208
+ high_priority_stream_groups ..................... []
209
+ hybrid_attention_ratio .......................... 0.0
210
+ hybrid_mlp_ratio ................................ 0.0
211
+ hybrid_override_pattern ......................... None
212
+ hysteresis ...................................... 2
213
+ ict_head_size ................................... None
214
+ ict_load ........................................ None
215
+ img_h ........................................... 224
216
+ img_w ........................................... 224
217
+ indexer_batch_size .............................. 128
218
+ indexer_log_interval ............................ 1000
219
+ inference_batch_times_seqlen_threshold .......... -1
220
+ inference_dynamic_batching ...................... False
221
+ inference_dynamic_batching_buffer_guaranteed_fraction 0.2
222
+ inference_dynamic_batching_buffer_overflow_factor None
223
+ inference_dynamic_batching_buffer_size_gb ....... 40.0
224
+ inference_dynamic_batching_chunk_size ........... 256
225
+ inference_dynamic_batching_max_requests_override None
226
+ inference_dynamic_batching_max_tokens_override .. None
227
+ inference_dynamic_batching_num_cuda_graphs ...... 16
228
+ inference_max_batch_size ........................ 8
229
+ inference_max_seq_length ........................ 2560
230
+ inference_rng_tracker ........................... False
231
+ init_method_std ................................. 0.01
232
+ init_method_xavier_uniform ...................... False
233
+ init_model_with_meta_device ..................... False
234
+ initial_loss_scale .............................. 4294967296
235
+ inprocess_active_world_size ..................... 8
236
+ inprocess_barrier_timeout ....................... 120
237
+ inprocess_completion_timeout .................... 120
238
+ inprocess_empty_cuda_cache ...................... False
239
+ inprocess_granularity ........................... node
240
+ inprocess_hard_timeout .......................... 90
241
+ inprocess_heartbeat_interval .................... 30
242
+ inprocess_heartbeat_timeout ..................... 60
243
+ inprocess_last_call_wait ........................ 1
244
+ inprocess_max_iterations ........................ None
245
+ inprocess_monitor_process_interval .............. 1.0
246
+ inprocess_monitor_thread_interval ............... 1.0
247
+ inprocess_progress_watchdog_interval ............ 1.0
248
+ inprocess_restart ............................... False
249
+ inprocess_soft_timeout .......................... 60
250
+ inprocess_termination_grace_time ................ 1
251
+ is_hybrid_model ................................. False
252
+ iter_per_epoch .................................. 1250
253
+ iterations_to_skip .............................. []
254
+ keep_fp8_transpose_cache_when_using_custom_fsdp . False
255
+ kitchen_config_file ............................. None
256
+ kitchen_recipe_number ........................... None
257
+ kv_channels ..................................... 128
258
+ kv_lora_rank .................................... 32
259
+ lazy_mpu_init ................................... None
260
+ load ............................................ None
261
+ load_main_params_from_ckpt ...................... None
262
+ load_model_opt_format ........................... False
263
+ local_rank ...................................... 0
264
+ log_energy ...................................... False
265
+ log_interval .................................... 1
266
+ log_loss_scale_to_tensorboard ................... True
267
+ log_memory_to_tensorboard ....................... False
268
+ log_num_zeros_in_grad ........................... False
269
+ log_params_norm ................................. False
270
+ log_progress .................................... False
271
+ log_straggler ................................... False
272
+ log_throughput .................................. True
273
+ log_timers_to_tensorboard ....................... False
274
+ log_validation_ppl_to_tensorboard ............... False
275
+ log_world_size_to_tensorboard ................... False
276
+ logging_level ................................... None
277
+ loss_scale ...................................... None
278
+ loss_scale_window ............................... 1000
279
+ lr .............................................. 0.0001
280
+ lr_decay_iters .................................. 8
281
+ lr_decay_samples ................................ None
282
+ lr_decay_style .................................. cosine
283
+ lr_warmup_fraction .............................. None
284
+ lr_warmup_init .................................. 0.0
285
+ lr_warmup_iters ................................. 5
286
+ lr_warmup_samples ............................... 0
287
+ lr_wsd_decay_iters .............................. None
288
+ lr_wsd_decay_samples ............................ None
289
+ lr_wsd_decay_style .............................. exponential
290
+ main_grads_dtype ................................ torch.float32
291
+ main_params_dtype ............................... torch.float32
292
+ make_vocab_size_divisible_by .................... 128
293
+ mamba_head_dim .................................. 64
294
+ mamba_num_groups ................................ 8
295
+ mamba_num_heads ................................. None
296
+ mamba_state_dim ................................. 128
297
+ manual_gc ....................................... False
298
+ manual_gc_eval .................................. True
299
+ manual_gc_interval .............................. 0
300
+ mask_factor ..................................... 1.0
301
+ mask_prob ....................................... 0.15
302
+ mask_type ....................................... random
303
+ masked_softmax_fusion ........................... False
304
+ max_position_embeddings ......................... 32768
305
+ max_tokens_to_oom ............................... 12000
306
+ memory_snapshot_path ............................ snapshot.pickle
307
+ merge_file ...................................... /workspace/data/wikitext/gpt2-merges.txt
308
+ micro_batch_size ................................ 1
309
+ microbatch_group_size_per_vp_stage .............. None
310
+ mid_level_dataset_surplus ....................... 0.005
311
+ min_loss_scale .................................. 1.0
312
+ min_lr .......................................... 1e-05
313
+ mlp_chunks_for_prefill .......................... 1
314
+ mmap_bin_files .................................. True
315
+ mock_data ....................................... False
316
+ moe_apply_probs_on_input ........................ False
317
+ moe_aux_loss_coeff .............................. 0.01
318
+ moe_deepep_num_sms .............................. 20
319
+ moe_echo_enable_random_offloading ............... False
320
+ moe_echo_expert_dispatch_overlap ................ True
321
+ moe_echo_expert_dispatcher_type ................. hybridep
322
+ moe_echo_recompute_expert_dispatch .............. False
323
+ moe_enable_deepep ............................... False
324
+ moe_enable_echo ................................. True
325
+ moe_enable_hybridep ............................. True
326
+ moe_expert_capacity_factor ...................... None
327
+ moe_extended_tp ................................. False
328
+ moe_ffn_hidden_size ............................. 3584
329
+ moe_grouped_gemm ................................ True
330
+ moe_hybridep_num_sms ............................ 16
331
+ moe_input_jitter_eps ............................ None
332
+ moe_layer_freq .................................. 1
333
+ moe_layer_recompute ............................. False
334
+ moe_num_echo_experts ............................ 32
335
+ moe_pad_expert_input_to_capacity ................ False
336
+ moe_per_layer_logging ........................... False
337
+ moe_permute_fusion .............................. False
338
+ moe_received_token_capacity ..................... 1.08203125
339
+ moe_router_bias_update_rate ..................... 0.001
340
+ moe_router_dtype ................................ fp32
341
+ moe_router_enable_expert_bias ................... False
342
+ moe_router_force_load_balancing ................. False
343
+ moe_router_group_topk ........................... None
344
+ moe_router_load_balancing_type .................. aux_loss
345
+ moe_router_num_groups ........................... None
346
+ moe_router_padding_for_fp8 ...................... False
347
+ moe_router_pre_softmax .......................... False
348
+ moe_router_score_function ....................... softmax
349
+ moe_router_topk ................................. 8
350
+ moe_router_topk_scaling_factor .................. None
351
+ moe_shared_expert_intermediate_size ............. None
352
+ moe_shared_expert_overlap ....................... False
353
+ moe_token_dispatcher_type ....................... flex
354
+ moe_token_drop_policy ........................... probs
355
+ moe_upcycling_granularity ....................... 1
356
+ moe_use_device_initiated_grouped_gemm ........... True
357
+ moe_use_legacy_grouped_gemm ..................... False
358
+ moe_use_upcycling ............................... False
359
+ moe_z_loss_coeff ................................ None
360
+ mrope_section ................................... None
361
+ mscale .......................................... 1.0
362
+ mscale_all_dim .................................. 0.0
363
+ mtp_loss_scaling_factor ......................... 0.1
364
+ mtp_num_layers .................................. None
365
+ multi_latent_attention .......................... False
366
+ nccl_all_reduce_for_prefill ..................... False
367
+ nccl_communicator_config_path ................... None
368
+ nccl_ub ......................................... False
369
+ no_load_optim ................................... None
370
+ no_load_rng ..................................... None
371
+ no_persist_layer_norm ........................... False
372
+ no_rope_freq .................................... None
373
+ no_save_optim ................................... None
374
+ no_save_rng ..................................... None
375
+ non_persistent_ckpt_type ........................ None
376
+ non_persistent_global_ckpt_dir .................. None
377
+ non_persistent_local_ckpt_algo .................. fully_parallel
378
+ non_persistent_local_ckpt_dir ................... None
379
+ non_persistent_save_interval .................... None
380
+ norm_epsilon .................................... 1e-05
381
+ normalization ................................... RMSNorm
382
+ num_attention_heads ............................. 32
383
+ num_channels .................................... 3
384
+ num_classes ..................................... 1000
385
+ num_dataset_builder_threads ..................... 1
386
+ num_distributed_optimizer_instances ............. 1
387
+ num_experts ..................................... 32
388
+ num_layers ...................................... 16
389
+ num_layers_at_end_in_bf16 ....................... 1
390
+ num_layers_at_start_in_bf16 ..................... 1
391
+ num_layers_per_virtual_pipeline_stage ........... None
392
+ num_query_groups ................................ 8
393
+ num_virtual_stages_per_pipeline_rank ............ None
394
+ num_workers ..................................... 2
395
+ object_storage_cache_path ....................... None
396
+ one_logger_async ................................ False
397
+ one_logger_project .............................. megatron-lm
398
+ one_logger_run_name ............................. None
399
+ onnx_safe ....................................... None
400
+ openai_gelu ..................................... False
401
+ optimizer ....................................... adam
402
+ optimizer_cpu_offload ........................... False
403
+ optimizer_offload_fraction ...................... 1.0
404
+ output_bert_embeddings .......................... False
405
+ overlap_cpu_optimizer_d2h_h2d ................... False
406
+ overlap_grad_reduce ............................. True
407
+ overlap_moe_expert_parallel_comm ................ False
408
+ overlap_p2p_comm ................................ False
409
+ overlap_p2p_comm_warmup_flush ................... False
410
+ overlap_param_gather ............................ True
411
+ overlap_param_gather_with_optimizer_step ........ False
412
+ override_opt_param_scheduler .................... False
413
+ params_dtype .................................... torch.bfloat16
414
+ patch_dim ....................................... 16
415
+ per_split_data_args_path ........................ None
416
+ perform_initialization .......................... True
417
+ pin_cpu_grads ................................... True
418
+ pin_cpu_params .................................. True
419
+ pipeline_model_parallel_comm_backend ............ None
420
+ pipeline_model_parallel_layout .................. None
421
+ pipeline_model_parallel_size .................... 1
422
+ position_embedding_type ......................... rope
423
+ pretrained_checkpoint ........................... None
424
+ profile ......................................... False
425
+ profile_ranks ................................... [0]
426
+ profile_step_end ................................ 12
427
+ profile_step_start .............................. 10
428
+ q_lora_rank ..................................... None
429
+ qk_head_dim ..................................... 128
430
+ qk_l2_norm ...................................... False
431
+ qk_layernorm .................................... False
432
+ qk_pos_emb_head_dim ............................. 64
433
+ query_in_block_prob ............................. 0.1
434
+ rampup_batch_size ............................... None
435
+ rank ............................................ 0
436
+ recompute_granularity ........................... None
437
+ recompute_method ................................ None
438
+ recompute_modules ............................... None
439
+ recompute_num_layers ............................ None
440
+ record_memory_history ........................... False
441
+ relative_attention_max_distance ................. 128
442
+ relative_attention_num_buckets .................. 32
443
+ replication ..................................... False
444
+ replication_factor .............................. 2
445
+ replication_jump ................................ None
446
+ rerun_mode ...................................... validate_results
447
+ reset_attention_mask ............................ False
448
+ reset_position_ids .............................. False
449
+ result_rejected_tracker_filename ................ None
450
+ retriever_report_topk_accuracies ................ []
451
+ retriever_score_scaling ......................... False
452
+ retriever_seq_length ............................ 256
453
+ retro_add_retriever ............................. False
454
+ retro_attention_gate ............................ 1
455
+ retro_cyclic_train_iters ........................ None
456
+ retro_encoder_attention_dropout ................. 0.1
457
+ retro_encoder_hidden_dropout .................... 0.1
458
+ retro_encoder_layers ............................ 2
459
+ retro_num_neighbors ............................. 2
460
+ retro_num_retrieved_chunks ...................... 2
461
+ retro_project_dir ............................... None
462
+ retro_verify_neighbor_count ..................... True
463
+ reuse_grad_buf_for_mxfp8_param_ag ............... False
464
+ rope_scaling_factor ............................. 8.0
465
+ rope_type ....................................... None
466
+ rotary_base ..................................... 1000000
467
+ rotary_interleaved .............................. False
468
+ rotary_percent .................................. 1.0
469
+ rotary_scaling_factor ........................... 1.0
470
+ rotary_seq_len_interpolation_factor ............. None
471
+ run_workload_inspector_server ................... False
472
+ sample_rate ..................................... 1.0
473
+ save ............................................ None
474
+ save_interval ................................... None
475
+ save_retain_interval ............................ None
476
+ scatter_gather_tensors_in_pipeline .............. True
477
+ seed ............................................ 1234
478
+ seq_length ...................................... 4096
479
+ sequence_parallel ............................... False
480
+ sft ............................................. False
481
+ sft_tokenizer_prompt_format ..................... nemotron-h-aligned
482
+ sgd_momentum .................................... 0.9
483
+ short_seq_prob .................................. 0.1
484
+ skip_train ...................................... False
485
+ skipped_train_samples ........................... 0
486
+ spec ............................................ None
487
+ split ........................................... 100,0,0
488
+ squared_relu .................................... False
489
+ start_weight_decay .............................. 0.1
490
+ straggler_ctrlr_port ............................ 65535
491
+ straggler_minmax_count .......................... 1
492
+ suggested_communication_unit_size ............... None
493
+ swiglu .......................................... True
494
+ swin_backbone_type .............................. tiny
495
+ symmetric_ar_type ............................... None
496
+ te_rng_tracker .................................. True
497
+ tensor_model_parallel_size ...................... 1
498
+ tensorboard_dir ................................. None
499
+ tensorboard_log_interval ........................ 1
500
+ tensorboard_queue_size .......................... 1000
501
+ test_data_path .................................. None
502
+ test_mode ....................................... False
503
+ tiktoken_num_special_tokens ..................... 1000
504
+ tiktoken_pattern ................................ None
505
+ tiktoken_special_tokens ......................... None
506
+ timing_log_level ................................ 0
507
+ timing_log_option ............................... minmax
508
+ titles_data_path ................................ None
509
+ tokenizer_model ................................. None
510
+ tokenizer_type .................................. GPT2BPETokenizer
511
+ torch_fsdp2_reshard_after_forward ............... True
512
+ tp_comm_bootstrap_backend ....................... nccl
513
+ tp_comm_bulk_dgrad .............................. True
514
+ tp_comm_bulk_wgrad .............................. True
515
+ tp_comm_overlap ................................. False
516
+ tp_comm_overlap_ag .............................. True
517
+ tp_comm_overlap_cfg ............................. None
518
+ tp_comm_overlap_rs .............................. True
519
+ tp_comm_overlap_rs_dgrad ........................ False
520
+ tp_comm_split_ag ................................ True
521
+ tp_comm_split_rs ................................ True
522
+ train_data_path ................................. None
523
+ train_iters ..................................... 12
524
+ train_samples ................................... None
525
+ train_sync_interval ............................. None
526
+ transformer_impl ................................ transformer_engine
527
+ transformer_pipeline_model_parallel_size ........ 1
528
+ untie_embeddings_and_output_weights ............. True
529
+ use_checkpoint_args ............................. False
530
+ use_checkpoint_opt_param_scheduler .............. False
531
+ use_cpu_initialization .......................... None
532
+ use_custom_fsdp ................................. False
533
+ use_dist_ckpt ................................... True
534
+ use_dist_ckpt_deprecated ........................ False
535
+ use_distributed_optimizer ....................... True
536
+ use_flash_attn .................................. False
537
+ use_legacy_models ............................... False
538
+ use_mp_args_from_checkpoint_args ................ False
539
+ use_one_sent_docs ............................... False
540
+ use_persistent_ckpt_worker ...................... False
541
+ use_precision_aware_optimizer ................... False
542
+ use_pytorch_profiler ............................ False
543
+ use_ring_exchange_p2p ........................... False
544
+ use_rope_scaling ................................ False
545
+ use_rotary_position_embeddings .................. False
546
+ use_sharp ....................................... False
547
+ use_tokenizer_model_from_checkpoint_args ........ True
548
+ use_torch_fsdp2 ................................. False
549
+ use_torch_optimizer_for_cpu_offload ............. False
550
+ use_tp_pp_dp_mapping ............................ False
551
+ v_head_dim ...................................... 128
552
+ valid_data_path ................................. None
553
+ variable_seq_lengths ............................ False
554
+ virtual_pipeline_model_parallel_size ............ None
555
+ vision_backbone_type ............................ vit
556
+ vision_pretraining .............................. False
557
+ vision_pretraining_type ......................... classify
558
+ vocab_extra_ids ................................. 0
559
+ vocab_file ...................................... /workspace/data/wikitext/gpt2-vocab.json
560
+ vocab_size ...................................... None
561
+ wandb_exp_name ..................................
562
+ wandb_project ...................................
563
+ wandb_save_dir ..................................
564
+ weight_decay .................................... 0.1
565
+ weight_decay_incr_style ......................... constant
566
+ wgrad_deferral_limit ............................ 0
567
+ world_size ...................................... 8
568
+ yaml_cfg ........................................ None
569
+ -------------------- end of arguments ---------------------
570
+ INFO:megatron.core.num_microbatches_calculator:setting number of microbatches to constant 2
571
+ > building GPT2BPETokenizer tokenizer ...
572
+ WARNING: one_logger package is required to enable e2e metrics tracking. please go to https://confluence.nvidia.com/display/MLWFO/Package+Repositories for details to install it
573
+ > padded vocab (size: 50257) with 47 dummy tokens (new size: 50304)
574
+ WARNING:megatron.core.rerun_state_machine:RerunStateMachine initialized in mode RerunMode.VALIDATE_RESULTS
575
+ > initializing torch distributed ...
576
+ [Gloo] Rank 1 is connected to 7 peer ranks. Expected number of connected peer ranks is : 7
577
+ [Gloo] Rank 0 is connected to 7 peer ranks. Expected number of connected peer ranks is : 7
578
+ [Gloo] Rank 2 is connected to 7 peer ranks. Expected number of connected peer ranks is : 7
579
+ [Gloo] Rank 6 is connected to 7 peer ranks. Expected number of connected peer ranks is : 7
580
+ [Gloo] Rank 3 is connected to 7 peer ranks. Expected number of connected peer ranks is : 7
581
+ [Gloo] Rank 4 is connected to 7 peer ranks. Expected number of connected peer ranks is : 7
582
+ [Gloo] Rank 5 is connected to 7 peer ranks. Expected number of connected peer ranks is : 7
583
+ [Gloo] Rank 7 is connected to 7 peer ranks. Expected number of connected peer ranks is : 7
584
+ [Gloo] Rank [Gloo] Rank 0[Gloo] Rank [Gloo] Rank [Gloo] Rank [Gloo] Rank 2 is connected to 135 is connected to 74 is connected to is connected to is connected to 7 peer ranks. is connected to 777 peer ranks. Expected number of connected peer ranks is : 7 peer ranks. [Gloo] Rank peer ranks. peer ranks. Expected number of connected peer ranks is : 7 peer ranks. Expected number of connected peer ranks is : Expected number of connected peer ranks is : 6Expected number of connected peer ranks is : 7
585
+ Expected number of connected peer ranks is : 77
586
+ is connected to 7
587
+ 7[Gloo] Rank
588
+ 7
589
+
590
+ 7 peer ranks. is connected to Expected number of connected peer ranks is : 77 peer ranks.
591
+ Expected number of connected peer ranks is : 7
592
+ [Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
593
+ > initialized tensor model parallel with size 1
594
+ > initialized pipeline model parallel with size 1
595
+ > setting random seeds to 1234 ...
596
+ [Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
597
+ [Gloo] Rank [Gloo] Rank 00 is connected to is connected to 0[Gloo] Rank 0 peer ranks. 0 peer ranks. Expected number of connected peer ranks is : is connected to Expected number of connected peer ranks is : 000
598
+ peer ranks.
599
+ Expected number of connected peer ranks is : 0
600
+ [Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
601
+ [Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
602
+ [Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
603
+ [rank3]:[W830 06:48:38.699027510 ProcessGroupNCCL.cpp:4789] [PG ID 0 PG GUID 0 Rank 3] using GPU 3 as device used by this process is currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect. You can specify device_id in init_process_group() to force use of a particular device.
604
+ [rank7]:[W830 06:48:38.699065652 ProcessGroupNCCL.cpp:4789] [PG ID 0 PG GUID 0 Rank 7] using GPU 7 as device used by this process is currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect. You can specify device_id in init_process_group() to force use of a particular device.
605
+ [rank5]:[W830 06:48:38.699066866 ProcessGroupNCCL.cpp:4789] [PG ID 0 PG GUID 0 Rank 5] using GPU 5 as device used by this process is currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect. You can specify device_id in init_process_group() to force use of a particular device.
606
+ [rank1]:[W830 06:48:38.699091271 ProcessGroupNCCL.cpp:4789] [PG ID 0 PG GUID 0 Rank 1] using GPU 1 as device used by this process is currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect. You can specify device_id in init_process_group() to force use of a particular device.
607
+ [rank6]:[W830 06:48:38.699130162 ProcessGroupNCCL.cpp:4789] [PG ID 0 PG GUID 0 Rank 6] using GPU 6 as device used by this process is currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect. You can specify device_id in init_process_group() to force use of a particular device.
608
+ [rank2]:[W830 06:48:38.699135530 ProcessGroupNCCL.cpp:4789] [PG ID 0 PG GUID 0 Rank 2] using GPU 2 as device used by this process is currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect. You can specify device_id in init_process_group() to force use of a particular device.
609
+ [rank4]:[W830 06:48:38.699166898 ProcessGroupNCCL.cpp:4789] [PG ID 0 PG GUID 0 Rank 4] using GPU 4 as device used by this process is currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect. You can specify device_id in init_process_group() to force use of a particular device.
610
+ > compiling dataset index builder ...
611
+ make: Entering directory '/workspace/Megatron-MoE-Echo/megatron/core/datasets'
612
+ make: Nothing to be done for 'default'.
613
+ make: Leaving directory '/workspace/Megatron-MoE-Echo/megatron/core/datasets'
614
+ >>> done with dataset index builder. Compilation time: 0.036 seconds
615
+ WARNING: constraints for invoking optimized fused softmax kernel are not met. We default back to unfused kernel invocations.
616
+ > compiling and loading fused kernels ...
617
+ [rank0]:[W830 06:48:38.859570340 ProcessGroupNCCL.cpp:4789] [PG ID 0 PG GUID 0 Rank 0] using GPU 0 as device used by this process is currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect. You can specify device_id in init_process_group() to force use of a particular device.
618
+ >>> done with compiling and loading fused kernels. Compilation time: 3.752 seconds
619
+ time to initialize megatron (seconds): 7.214
620
+ [after megatron is initialized] datetime: 2026-08-30 06:48:44
621
+ building GPT model ...
622
+ /usr/local/lib/python3.12/dist-packages/transformer_engine/pytorch/cpu_offload.py:708: DeprecationWarning: Offloading weights is deprecated. Using offload_weights=True does not have any effect.
623
+ warnings.warn(
624
+ /usr/local/lib/python3.12/dist-packages/transformer_engine/pytorch/cpu_offload.py:708: DeprecationWarning: Offloading weights is deprecated. Using offload_weights=True does not have any effect.
625
+ warnings.warn(
626
+ /usr/local/lib/python3.12/dist-packages/transformer_engine/pytorch/cpu_offload.py:708: DeprecationWarning: Offloading weights is deprecated. Using offload_weights=True does not have any effect.
627
+ warnings.warn(
628
+ /usr/local/lib/python3.12/dist-packages/transformer_engine/pytorch/cpu_offload.py:708: DeprecationWarning: Offloading weights is deprecated. Using offload_weights=True does not have any effect.
629
+ warnings.warn(
630
+ /usr/local/lib/python3.12/dist-packages/transformer_engine/pytorch/cpu_offload.py:708: DeprecationWarning: Offloading weights is deprecated. Using offload_weights=True does not have any effect.
631
+ warnings.warn(
632
+ /usr/local/lib/python3.12/dist-packages/transformer_engine/pytorch/cpu_offload.py:708: DeprecationWarning: Offloading weights is deprecated. Using offload_weights=True does not have any effect.
633
+ warnings.warn(
634
+ /usr/local/lib/python3.12/dist-packages/transformer_engine/pytorch/cpu_offload.py:708: DeprecationWarning: Offloading weights is deprecated. Using offload_weights=True does not have any effect.
635
+ warnings.warn(
636
+ /usr/local/lib/python3.12/dist-packages/transformer_engine/pytorch/cpu_offload.py:708: DeprecationWarning: Offloading weights is deprecated. Using offload_weights=True does not have any effect.
637
+ warnings.warn(
638
+ > number of parameters on (tensor, pipeline) model parallel rank (0, 0): 3903983616
639
+ INFO:megatron.core.distributed.distributed_data_parallel:Setting up DistributedDataParallel with config DistributedDataParallelConfig(grad_reduce_in_fp32=True, overlap_grad_reduce=True, overlap_param_gather=True, align_param_gather=False, use_distributed_optimizer=True, num_distributed_optimizer_instances=1, check_for_nan_in_grad=False, check_for_large_grads=False, bucket_size=40000000, pad_buckets_for_high_nccl_busbw=False, average_in_collective=False, fp8_param_gather=False, reuse_grad_buf_for_mxfp8_param_ag=False, use_custom_fsdp=False, data_parallel_sharding_strategy='no_shard', gradient_reduce_div_fusion=True, suggested_communication_unit_size=None, preserve_fp32_weights=True, keep_fp8_transpose_cache_when_using_custom_fsdp=False, nccl_ub=False, fsdp_double_buffer=False)
640
+ INFO:megatron.core.distributed.param_and_grad_buffer:Number of buckets for gradient all-reduce / reduce-scatter: 18
641
+ Params for bucket 1 (206045184 elements, 206045184 padded size):
642
+ module.output_layer.weight
643
+ Params for bucket 2 (42086400 elements, 42086400 padded size):
644
+ module.decoder.final_layernorm.weight
645
+ module.decoder.layers.15.self_attention.linear_qkv.weight
646
+ module.decoder.layers.15.pre_mlp_layernorm.weight
647
+ module.decoder.layers.15.mlp.router.weight
648
+ module.decoder.layers.15.self_attention.linear_qkv.layer_norm_weight
649
+ module.decoder.layers.15.self_attention.linear_proj.weight
650
+ Params for bucket 3 (42082304 elements, 42082304 padded size):
651
+ module.decoder.layers.14.self_attention.linear_qkv.weight
652
+ module.decoder.layers.14.pre_mlp_layernorm.weight
653
+ module.decoder.layers.14.mlp.router.weight
654
+ module.decoder.layers.14.self_attention.linear_proj.weight
655
+ module.decoder.layers.14.self_attention.linear_qkv.layer_norm_weight
656
+ Params for bucket 4 (42082304 elements, 42082304 padded size):
657
+ module.decoder.layers.13.self_attention.linear_qkv.weight
658
+ module.decoder.layers.13.pre_mlp_layernorm.weight
659
+ module.decoder.layers.13.mlp.router.weight
660
+ module.decoder.layers.13.self_attention.linear_qkv.layer_norm_weight
661
+ module.decoder.layers.13.self_attention.linear_proj.weight
662
+ Params for bucket 5 (42082304 elements, 42082304 padded size):
663
+ module.decoder.layers.12.pre_mlp_layernorm.weight
664
+ module.decoder.layers.12.self_attention.linear_qkv.weight
665
+ module.decoder.layers.12.mlp.router.weight
666
+ module.decoder.layers.12.self_attention.linear_proj.weight
667
+ module.decoder.layers.12.self_attention.linear_qkv.layer_norm_weight
668
+ Params for bucket 6 (42082304 elements, 42082304 padded size):
669
+ module.decoder.layers.11.self_attention.linear_qkv.weight
670
+ module.decoder.layers.11.pre_mlp_layernorm.weight
671
+ module.decoder.layers.11.mlp.router.weight
672
+ module.decoder.layers.11.self_attention.linear_qkv.layer_norm_weight
673
+ module.decoder.layers.11.self_attention.linear_proj.weight
674
+ Params for bucket 7 (42082304 elements, 42082304 padded size):
675
+ module.decoder.layers.10.mlp.router.weight
676
+ module.decoder.layers.10.self_attention.linear_qkv.layer_norm_weight
677
+ module.decoder.layers.10.self_attention.linear_proj.weight
678
+ module.decoder.layers.10.self_attention.linear_qkv.weight
679
+ module.decoder.layers.10.pre_mlp_layernorm.weight
680
+ Params for bucket 8 (42082304 elements, 42082304 padded size):
681
+ module.decoder.layers.9.self_attention.linear_qkv.weight
682
+ module.decoder.layers.9.pre_mlp_layernorm.weight
683
+ module.decoder.layers.9.mlp.router.weight
684
+ module.decoder.layers.9.self_attention.linear_proj.weight
685
+ module.decoder.layers.9.self_attention.linear_qkv.layer_norm_weight
686
+ Params for bucket 9 (42082304 elements, 42082304 padded size):
687
+ module.decoder.layers.8.mlp.router.weight
688
+ module.decoder.layers.8.self_attention.linear_proj.weight
689
+ module.decoder.layers.8.self_attention.linear_qkv.layer_norm_weight
690
+ module.decoder.layers.8.pre_mlp_layernorm.weight
691
+ module.decoder.layers.8.self_attention.linear_qkv.weight
692
+ Params for bucket 10 (42082304 elements, 42082304 padded size):
693
+ module.decoder.layers.7.pre_mlp_layernorm.weight
694
+ module.decoder.layers.7.self_attention.linear_qkv.weight
695
+ module.decoder.layers.7.mlp.router.weight
696
+ module.decoder.layers.7.self_attention.linear_proj.weight
697
+ module.decoder.layers.7.self_attention.linear_qkv.layer_norm_weight
698
+ Params for bucket 11 (42082304 elements, 42082304 padded size):
699
+ module.decoder.layers.6.mlp.router.weight
700
+ module.decoder.layers.6.self_attention.linear_proj.weight
701
+ module.decoder.layers.6.self_attention.linear_qkv.layer_norm_weight
702
+ module.decoder.layers.6.self_attention.linear_qkv.weight
703
+ module.decoder.layers.6.pre_mlp_layernorm.weight
704
+ Params for bucket 12 (42082304 elements, 42082304 padded size):
705
+ module.decoder.layers.5.mlp.router.weight
706
+ module.decoder.layers.5.self_attention.linear_qkv.layer_norm_weight
707
+ module.decoder.layers.5.self_attention.linear_qkv.weight
708
+ module.decoder.layers.5.self_attention.linear_proj.weight
709
+ module.decoder.layers.5.pre_mlp_layernorm.weight
710
+ Params for bucket 13 (42082304 elements, 42082304 padded size):
711
+ module.decoder.layers.4.self_attention.linear_qkv.weight
712
+ module.decoder.layers.4.pre_mlp_layernorm.weight
713
+ module.decoder.layers.4.mlp.router.weight
714
+ module.decoder.layers.4.self_attention.linear_proj.weight
715
+ module.decoder.layers.4.self_attention.linear_qkv.layer_norm_weight
716
+ Params for bucket 14 (42082304 elements, 42082304 padded size):
717
+ module.decoder.layers.3.self_attention.linear_qkv.weight
718
+ module.decoder.layers.3.pre_mlp_layernorm.weight
719
+ module.decoder.layers.3.mlp.router.weight
720
+ module.decoder.layers.3.self_attention.linear_qkv.layer_norm_weight
721
+ module.decoder.layers.3.self_attention.linear_proj.weight
722
+ Params for bucket 15 (42082304 elements, 42082304 padded size):
723
+ module.decoder.layers.2.mlp.router.weight
724
+ module.decoder.layers.2.self_attention.linear_qkv.layer_norm_weight
725
+ module.decoder.layers.2.self_attention.linear_proj.weight
726
+ module.decoder.layers.2.self_attention.linear_qkv.weight
727
+ module.decoder.layers.2.pre_mlp_layernorm.weight
728
+ Params for bucket 16 (42082304 elements, 42082304 padded size):
729
+ module.decoder.layers.1.self_attention.linear_qkv.layer_norm_weight
730
+ module.decoder.layers.1.self_attention.linear_proj.weight
731
+ module.decoder.layers.1.pre_mlp_layernorm.weight
732
+ module.decoder.layers.1.mlp.router.weight
733
+ module.decoder.layers.1.self_attention.linear_qkv.weight
734
+ Params for bucket 17 (42082304 elements, 42082304 padded size):
735
+ module.decoder.layers.0.self_attention.linear_qkv.layer_norm_weight
736
+ module.decoder.layers.0.pre_mlp_layernorm.weight
737
+ module.decoder.layers.0.self_attention.linear_proj.weight
738
+ module.decoder.layers.0.mlp.router.weight
739
+ module.decoder.layers.0.self_attention.linear_qkv.weight
740
+ Params for bucket 18 (206045184 elements, 206045184 padded size):
741
+ module.embedding.word_embeddings.weight
742
+ INFO:megatron.core.distributed.param_and_grad_buffer:Number of buckets for gradient all-reduce / reduce-scatter: 56
743
+ Params for bucket 1 (44040192 elements, 44040192 padded size):
744
+ module.decoder.layers.15.mlp.experts.linear_fc2.weight3
745
+ module.decoder.layers.15.mlp.experts.linear_fc2.weight1
746
+ module.decoder.layers.15.mlp.experts.linear_fc2.weight2
747
+ Params for bucket 2 (44040192 elements, 44040192 padded size):
748
+ module.decoder.layers.15.mlp.experts.linear_fc2.weight0
749
+ module.decoder.layers.15.mlp.experts.linear_fc1.weight3
750
+ Params for bucket 3 (58720256 elements, 58720256 padded size):
751
+ module.decoder.layers.15.mlp.experts.linear_fc1.weight2
752
+ module.decoder.layers.15.mlp.experts.linear_fc1.weight1
753
+ Params for bucket 4 (44040192 elements, 44040192 padded size):
754
+ module.decoder.layers.15.mlp.experts.linear_fc1.weight0
755
+ module.decoder.layers.14.mlp.experts.linear_fc2.weight3
756
+ Params for bucket 5 (44040192 elements, 44040192 padded size):
757
+ module.decoder.layers.14.mlp.experts.linear_fc2.weight2
758
+ module.decoder.layers.14.mlp.experts.linear_fc2.weight1
759
+ module.decoder.layers.14.mlp.experts.linear_fc2.weight0
760
+ Params for bucket 6 (58720256 elements, 58720256 padded size):
761
+ module.decoder.layers.14.mlp.experts.linear_fc1.weight3
762
+ module.decoder.layers.14.mlp.experts.linear_fc1.weight2
763
+ Params for bucket 7 (58720256 elements, 58720256 padded size):
764
+ module.decoder.layers.14.mlp.experts.linear_fc1.weight1
765
+ module.decoder.layers.14.mlp.experts.linear_fc1.weight0
766
+ Params for bucket 8 (44040192 elements, 44040192 padded size):
767
+ module.decoder.layers.13.mlp.experts.linear_fc2.weight3
768
+ module.decoder.layers.13.mlp.experts.linear_fc2.weight1
769
+ module.decoder.layers.13.mlp.experts.linear_fc2.weight2
770
+ Params for bucket 9 (44040192 elements, 44040192 padded size):
771
+ module.decoder.layers.13.mlp.experts.linear_fc2.weight0
772
+ module.decoder.layers.13.mlp.experts.linear_fc1.weight3
773
+ Params for bucket 10 (58720256 elements, 58720256 padded size):
774
+ module.decoder.layers.13.mlp.experts.linear_fc1.weight2
775
+ module.decoder.layers.13.mlp.experts.linear_fc1.weight1
776
+ Params for bucket 11 (44040192 elements, 44040192 padded size):
777
+ module.decoder.layers.13.mlp.experts.linear_fc1.weight0
778
+ module.decoder.layers.12.mlp.experts.linear_fc2.weight3
779
+ Params for bucket 12 (44040192 elements, 44040192 padded size):
780
+ module.decoder.layers.12.mlp.experts.linear_fc2.weight2
781
+ module.decoder.layers.12.mlp.experts.linear_fc2.weight0
782
+ module.decoder.layers.12.mlp.experts.linear_fc2.weight1
783
+ Params for bucket 13 (58720256 elements, 58720256 padded size):
784
+ module.decoder.layers.12.mlp.experts.linear_fc1.weight3
785
+ module.decoder.layers.12.mlp.experts.linear_fc1.weight2
786
+ Params for bucket 14 (58720256 elements, 58720256 padded size):
787
+ module.decoder.layers.12.mlp.experts.linear_fc1.weight1
788
+ module.decoder.layers.12.mlp.experts.linear_fc1.weight0
789
+ Params for bucket 15 (44040192 elements, 44040192 padded size):
790
+ module.decoder.layers.11.mlp.experts.linear_fc2.weight3
791
+ module.decoder.layers.11.mlp.experts.linear_fc2.weight2
792
+ module.decoder.layers.11.mlp.experts.linear_fc2.weight1
793
+ Params for bucket 16 (44040192 elements, 44040192 padded size):
794
+ module.decoder.layers.11.mlp.experts.linear_fc2.weight0
795
+ module.decoder.layers.11.mlp.experts.linear_fc1.weight3
796
+ Params for bucket 17 (58720256 elements, 58720256 padded size):
797
+ module.decoder.layers.11.mlp.experts.linear_fc1.weight2
798
+ module.decoder.layers.11.mlp.experts.linear_fc1.weight1
799
+ Params for bucket 18 (44040192 elements, 44040192 padded size):
800
+ module.decoder.layers.11.mlp.experts.linear_fc1.weight0
801
+ module.decoder.layers.10.mlp.experts.linear_fc2.weight3
802
+ Params for bucket 19 (44040192 elements, 44040192 padded size):
803
+ module.decoder.layers.10.mlp.experts.linear_fc2.weight2
804
+ module.decoder.layers.10.mlp.experts.linear_fc2.weight1
805
+ module.decoder.layers.10.mlp.experts.linear_fc2.weight0
806
+ Params for bucket 20 (58720256 elements, 58720256 padded size):
807
+ module.decoder.layers.10.mlp.experts.linear_fc1.weight3
808
+ module.decoder.layers.10.mlp.experts.linear_fc1.weight2
809
+ Params for bucket 21 (58720256 elements, 58720256 padded size):
810
+ module.decoder.layers.10.mlp.experts.linear_fc1.weight1
811
+ module.decoder.layers.10.mlp.experts.linear_fc1.weight0
812
+ Params for bucket 22 (44040192 elements, 44040192 padded size):
813
+ module.decoder.layers.9.mlp.experts.linear_fc2.weight3
814
+ module.decoder.layers.9.mlp.experts.linear_fc2.weight1
815
+ module.decoder.layers.9.mlp.experts.linear_fc2.weight2
816
+ Params for bucket 23 (44040192 elements, 44040192 padded size):
817
+ module.decoder.layers.9.mlp.experts.linear_fc2.weight0
818
+ module.decoder.layers.9.mlp.experts.linear_fc1.weight3
819
+ Params for bucket 24 (58720256 elements, 58720256 padded size):
820
+ module.decoder.layers.9.mlp.experts.linear_fc1.weight2
821
+ module.decoder.layers.9.mlp.experts.linear_fc1.weight1
822
+ Params for bucket 25 (44040192 elements, 44040192 padded size):
823
+ module.decoder.layers.9.mlp.experts.linear_fc1.weight0
824
+ module.decoder.layers.8.mlp.experts.linear_fc2.weight3
825
+ Params for bucket 26 (44040192 elements, 44040192 padded size):
826
+ module.decoder.layers.8.mlp.experts.linear_fc2.weight2
827
+ module.decoder.layers.8.mlp.experts.linear_fc2.weight1
828
+ module.decoder.layers.8.mlp.experts.linear_fc2.weight0
829
+ Params for bucket 27 (58720256 elements, 58720256 padded size):
830
+ module.decoder.layers.8.mlp.experts.linear_fc1.weight3
831
+ module.decoder.layers.8.mlp.experts.linear_fc1.weight2
832
+ Params for bucket 28 (58720256 elements, 58720256 padded size):
833
+ module.decoder.layers.8.mlp.experts.linear_fc1.weight1
834
+ module.decoder.layers.8.mlp.experts.linear_fc1.weight0
835
+ Params for bucket 29 (44040192 elements, 44040192 padded size):
836
+ module.decoder.layers.7.mlp.experts.linear_fc2.weight3
837
+ module.decoder.layers.7.mlp.experts.linear_fc2.weight2
838
+ module.decoder.layers.7.mlp.experts.linear_fc2.weight1
839
+ Params for bucket 30 (44040192 elements, 44040192 padded size):
840
+ module.decoder.layers.7.mlp.experts.linear_fc2.weight0
841
+ module.decoder.layers.7.mlp.experts.linear_fc1.weight3
842
+ Params for bucket 31 (58720256 elements, 58720256 padded size):
843
+ module.decoder.layers.7.mlp.experts.linear_fc1.weight2
844
+ module.decoder.layers.7.mlp.experts.linear_fc1.weight1
845
+ Params for bucket 32 (44040192 elements, 44040192 padded size):
846
+ module.decoder.layers.7.mlp.experts.linear_fc1.weight0
847
+ module.decoder.layers.6.mlp.experts.linear_fc2.weight3
848
+ Params for bucket 33 (44040192 elements, 44040192 padded size):
849
+ module.decoder.layers.6.mlp.experts.linear_fc2.weight2
850
+ module.decoder.layers.6.mlp.experts.linear_fc2.weight0
851
+ module.decoder.layers.6.mlp.experts.linear_fc2.weight1
852
+ Params for bucket 34 (58720256 elements, 58720256 padded size):
853
+ module.decoder.layers.6.mlp.experts.linear_fc1.weight3
854
+ module.decoder.layers.6.mlp.experts.linear_fc1.weight2
855
+ Params for bucket 35 (58720256 elements, 58720256 padded size):
856
+ module.decoder.layers.6.mlp.experts.linear_fc1.weight1
857
+ module.decoder.layers.6.mlp.experts.linear_fc1.weight0
858
+ Params for bucket 36 (44040192 elements, 44040192 padded size):
859
+ module.decoder.layers.5.mlp.experts.linear_fc2.weight3
860
+ module.decoder.layers.5.mlp.experts.linear_fc2.weight1
861
+ module.decoder.layers.5.mlp.experts.linear_fc2.weight2
862
+ Params for bucket 37 (44040192 elements, 44040192 padded size):
863
+ module.decoder.layers.5.mlp.experts.linear_fc2.weight0
864
+ module.decoder.layers.5.mlp.experts.linear_fc1.weight3
865
+ Params for bucket 38 (58720256 elements, 58720256 padded size):
866
+ module.decoder.layers.5.mlp.experts.linear_fc1.weight2
867
+ module.decoder.layers.5.mlp.experts.linear_fc1.weight1
868
+ Params for bucket 39 (44040192 elements, 44040192 padded size):
869
+ module.decoder.layers.5.mlp.experts.linear_fc1.weight0
870
+ module.decoder.layers.4.mlp.experts.linear_fc2.weight3
871
+ Params for bucket 40 (44040192 elements, 44040192 padded size):
872
+ module.decoder.layers.4.mlp.experts.linear_fc2.weight2
873
+ module.decoder.layers.4.mlp.experts.linear_fc2.weight1
874
+ module.decoder.layers.4.mlp.experts.linear_fc2.weight0
875
+ Params for bucket 41 (58720256 elements, 58720256 padded size):
876
+ module.decoder.layers.4.mlp.experts.linear_fc1.weight3
877
+ module.decoder.layers.4.mlp.experts.linear_fc1.weight2
878
+ Params for bucket 42 (58720256 elements, 58720256 padded size):
879
+ module.decoder.layers.4.mlp.experts.linear_fc1.weight1
880
+ module.decoder.layers.4.mlp.experts.linear_fc1.weight0
881
+ Params for bucket 43 (44040192 elements, 44040192 padded size):
882
+ module.decoder.layers.3.mlp.experts.linear_fc2.weight3
883
+ module.decoder.layers.3.mlp.experts.linear_fc2.weight2
884
+ module.decoder.layers.3.mlp.experts.linear_fc2.weight1
885
+ Params for bucket 44 (44040192 elements, 44040192 padded size):
886
+ module.decoder.layers.3.mlp.experts.linear_fc2.weight0
887
+ module.decoder.layers.3.mlp.experts.linear_fc1.weight3
888
+ Params for bucket 45 (58720256 elements, 58720256 padded size):
889
+ module.decoder.layers.3.mlp.experts.linear_fc1.weight2
890
+ module.decoder.layers.3.mlp.experts.linear_fc1.weight1
891
+ Params for bucket 46 (44040192 elements, 44040192 padded size):
892
+ module.decoder.layers.3.mlp.experts.linear_fc1.weight0
893
+ module.decoder.layers.2.mlp.experts.linear_fc2.weight3
894
+ Params for bucket 47 (44040192 elements, 44040192 padded size):
895
+ module.decoder.layers.2.mlp.experts.linear_fc2.weight2
896
+ module.decoder.layers.2.mlp.experts.linear_fc2.weight1
897
+ module.decoder.layers.2.mlp.experts.linear_fc2.weight0
898
+ Params for bucket 48 (58720256 elements, 58720256 padded size):
899
+ module.decoder.layers.2.mlp.experts.linear_fc1.weight3
900
+ module.decoder.layers.2.mlp.experts.linear_fc1.weight2
901
+ Params for bucket 49 (58720256 elements, 58720256 padded size):
902
+ module.decoder.layers.2.mlp.experts.linear_fc1.weight1
903
+ module.decoder.layers.2.mlp.experts.linear_fc1.weight0
904
+ Params for bucket 50 (44040192 elements, 44040192 padded size):
905
+ module.decoder.layers.1.mlp.experts.linear_fc2.weight3
906
+ module.decoder.layers.1.mlp.experts.linear_fc2.weight1
907
+ module.decoder.layers.1.mlp.experts.linear_fc2.weight2
908
+ Params for bucket 51 (44040192 elements, 44040192 padded size):
909
+ module.decoder.layers.1.mlp.experts.linear_fc2.weight0
910
+ module.decoder.layers.1.mlp.experts.linear_fc1.weight3
911
+ Params for bucket 52 (58720256 elements, 58720256 padded size):
912
+ module.decoder.layers.1.mlp.experts.linear_fc1.weight2
913
+ module.decoder.layers.1.mlp.experts.linear_fc1.weight1
914
+ Params for bucket 53 (44040192 elements, 44040192 padded size):
915
+ module.decoder.layers.1.mlp.experts.linear_fc1.weight0
916
+ module.decoder.layers.0.mlp.experts.linear_fc2.weight3
917
+ Params for bucket 54 (44040192 elements, 44040192 padded size):
918
+ module.decoder.layers.0.mlp.experts.linear_fc2.weight2
919
+ module.decoder.layers.0.mlp.experts.linear_fc2.weight1
920
+ module.decoder.layers.0.mlp.experts.linear_fc2.weight0
921
+ Params for bucket 55 (58720256 elements, 58720256 padded size):
922
+ module.decoder.layers.0.mlp.experts.linear_fc1.weight3
923
+ module.decoder.layers.0.mlp.experts.linear_fc1.weight2
924
+ Params for bucket 56 (58720256 elements, 58720256 padded size):
925
+ module.decoder.layers.0.mlp.experts.linear_fc1.weight1
926
+ module.decoder.layers.0.mlp.experts.linear_fc1.weight0
927
+ INFO:megatron.core.optimizer:Setting up optimizer with config OptimizerConfig(optimizer='adam', lr=0.0001, min_lr=1e-05, decoupled_lr=None, decoupled_min_lr=None, weight_decay=0.1, fp8_recipe='mxfp8', fp16=False, bf16=True, reuse_grad_buf_for_mxfp8_param_ag=False, params_dtype=torch.bfloat16, use_precision_aware_optimizer=False, store_param_remainders=True, main_grads_dtype=torch.float32, main_params_dtype=torch.float32, exp_avg_dtype=torch.float32, exp_avg_sq_dtype=torch.float32, loss_scale=None, initial_loss_scale=4294967296, min_loss_scale=1.0, loss_scale_window=1000, hysteresis=2, adam_beta1=0.9, adam_beta2=0.999, adam_eps=1e-08, sgd_momentum=0.9, use_distributed_optimizer=True, overlap_param_gather_with_optimizer_step=False, optimizer_cpu_offload=False, optimizer_offload_fraction=1.0, use_torch_optimizer_for_cpu_offload=False, overlap_cpu_optimizer_d2h_h2d=False, pin_cpu_grads=True, pin_cpu_params=True, clip_grad=1.0, log_num_zeros_in_grad=False, barrier_with_L1_time=True, timers=<megatron.core.timers.Timers object at 0x743309117800>, config_logger_dir='')
928
+ INFO:megatron.core.optimizer_param_scheduler:> learning rate decay style: cosine
929
+ [after model, optimizer, and learning rate scheduler are built] datetime: 2026-08-30 06:48:44
930
+ > building train, validation, and test datasets ...
931
+ > datasets target sizes (minimum size):
932
+ train: 192
933
+ validation: 0
934
+ test: 0
935
+ INFO:megatron.core.datasets.blended_megatron_dataset_config:Let split_matrix = [(0, 1.0), None, None]
936
+ > building train, validation, and test datasets for GPT ...
937
+ INFO:megatron.core.datasets.blended_megatron_dataset_builder:Building GPTDataset splits with sizes=(192, 0, 0) and config=GPTDatasetConfig(random_seed=1234, sequence_length=4096, blend=(['/workspace/data/wikitext/wikitext103_gpt2_text_document'], None), blend_per_split=None, split='100,0,0', split_matrix=[(0, 1.0), None, None], num_dataset_builder_threads=1, path_to_cache=None, mmap_bin_files=True, mock=False, tokenizer=<megatron.training.tokenizer.tokenizer._GPT2BPETokenizer object at 0x743308cd3440>, mid_level_dataset_surplus=0.005, reset_position_ids=False, reset_attention_mask=False, eod_mask_loss=False, create_attention_mask=True, drop_last_partial_validation_sequence=True, add_extra_token_to_sequence=True, object_storage_cache_path=None)
938
+ INFO:megatron.core.datasets.indexed_dataset:Load the _IndexReader from /workspace/data/wikitext/wikitext103_gpt2_text_document.idx
939
+ INFO:megatron.core.datasets.indexed_dataset: Extract the sequence lengths
940
+ INFO:megatron.core.datasets.indexed_dataset: Extract the sequence pointers
941
+ INFO:megatron.core.datasets.indexed_dataset: Extract the document indices
942
+ INFO:megatron.core.datasets.indexed_dataset:> total number of sequences: 1165029
943
+ INFO:megatron.core.datasets.indexed_dataset:> total number of documents: 1801350
944
+ INFO:megatron.core.datasets.gpt_dataset:Load the GPTDataset train indices
945
+ INFO:megatron.core.datasets.gpt_dataset: Load the document index from 44ed1ee5bcb9e5b030387b8d27a6c406-GPTDataset-train-document_index.npy
946
+ INFO:megatron.core.datasets.gpt_dataset: Load the sample index from 44ed1ee5bcb9e5b030387b8d27a6c406-GPTDataset-train-sample_index.npy
947
+ INFO:megatron.core.datasets.gpt_dataset: Load the shuffle index from 44ed1ee5bcb9e5b030387b8d27a6c406-GPTDataset-train-shuffle_index.npy
948
+ INFO:megatron.core.datasets.gpt_dataset:> total number of samples: 29073
949
+ > finished creating GPT datasets ...
950
+ [after dataloaders are built] datetime: 2026-08-30 06:48:44
951
+ done with setup ...
952
+ training ...
953
+ Overwriting rerun_state_machine.current_iteration from -1 to 0...
954
+ (min, max) time across ranks (ms):
955
+ model-and-optimizer-setup ......................: (393.41, 395.82)
956
+ train/valid/test-data-iterators-setup ..........: (64.49, 75.15)
957
+ [before the start of training step] datetime: 2026-08-30 06:48:44
958
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:334: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
959
+ warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
960
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:336: UserWarning: The use_host_meta is deprecated, it will be removed in the future.
961
+ warnings.warn("The use_host_meta is deprecated, it will be removed in the future.")
962
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:334: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
963
+ warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
964
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:336: UserWarning: The use_host_meta is deprecated, it will be removed in the future.
965
+ warnings.warn("The use_host_meta is deprecated, it will be removed in the future.")
966
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:334: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
967
+ warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
968
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:334: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
969
+ warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
970
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:336: UserWarning: The use_host_meta is deprecated, it will be removed in the future.
971
+ warnings.warn("The use_host_meta is deprecated, it will be removed in the future.")
972
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:336: UserWarning: The use_host_meta is deprecated, it will be removed in the future.
973
+ warnings.warn("The use_host_meta is deprecated, it will be removed in the future.")
974
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:334: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
975
+ warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
976
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:334: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
977
+ warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
978
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:336: UserWarning: The use_host_meta is deprecated, it will be removed in the future.
979
+ warnings.warn("The use_host_meta is deprecated, it will be removed in the future.")
980
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:336: UserWarning: The use_host_meta is deprecated, it will be removed in the future.
981
+ warnings.warn("The use_host_meta is deprecated, it will be removed in the future.")
982
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:334: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
983
+ warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
984
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:336: UserWarning: The use_host_meta is deprecated, it will be removed in the future.
985
+ warnings.warn("The use_host_meta is deprecated, it will be removed in the future.")
986
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:334: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
987
+ warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
988
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:336: UserWarning: The use_host_meta is deprecated, it will be removed in the future.
989
+ warnings.warn("The use_host_meta is deprecated, it will be removed in the future.")
990
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:482: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
991
+ warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
992
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:482: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
993
+ warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
994
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:482: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
995
+ warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
996
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:482: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
997
+ warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
998
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:482: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
999
+ warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
1000
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:482: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
1001
+ warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
1002
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:482: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
1003
+ warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
1004
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:482: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
1005
+ warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
1006
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:334: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
1007
+ warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
1008
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:336: UserWarning: The use_host_meta is deprecated, it will be removed in the future.
1009
+ warnings.warn("The use_host_meta is deprecated, it will be removed in the future.")
1010
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:334: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
1011
+ warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
1012
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:336: UserWarning: The use_host_meta is deprecated, it will be removed in the future.
1013
+ warnings.warn("The use_host_meta is deprecated, it will be removed in the future.")
1014
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:334: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
1015
+ warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
1016
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:336: UserWarning: The use_host_meta is deprecated, it will be removed in the future.
1017
+ warnings.warn("The use_host_meta is deprecated, it will be removed in the future.")
1018
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:334: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
1019
+ warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
1020
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:336: UserWarning: The use_host_meta is deprecated, it will be removed in the future.
1021
+ warnings.warn("The use_host_meta is deprecated, it will be removed in the future.")
1022
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:334: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
1023
+ warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
1024
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:336: UserWarning: The use_host_meta is deprecated, it will be removed in the future.
1025
+ warnings.warn("The use_host_meta is deprecated, it will be removed in the future.")
1026
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:334: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
1027
+ warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
1028
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:336: UserWarning: The use_host_meta is deprecated, it will be removed in the future.
1029
+ warnings.warn("The use_host_meta is deprecated, it will be removed in the future.")
1030
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:334: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
1031
+ warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
1032
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:336: UserWarning: The use_host_meta is deprecated, it will be removed in the future.
1033
+ warnings.warn("The use_host_meta is deprecated, it will be removed in the future.")
1034
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:334: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
1035
+ warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
1036
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:336: UserWarning: The use_host_meta is deprecated, it will be removed in the future.
1037
+ warnings.warn("The use_host_meta is deprecated, it will be removed in the future.")
1038
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:482: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
1039
+ warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
1040
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:482: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
1041
+ warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
1042
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:482: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
1043
+ warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
1044
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:482: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
1045
+ warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
1046
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:482: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
1047
+ warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
1048
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:482: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
1049
+ warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
1050
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:482: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
1051
+ warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
1052
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:482: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
1053
+ warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
1054
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:334: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
1055
+ warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
1056
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:334: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
1057
+ warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
1058
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:334: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
1059
+ warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
1060
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:334: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
1061
+ warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
1062
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:334: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
1063
+ warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
1064
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:334: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
1065
+ warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
1066
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:334: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
1067
+ warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
1068
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:334: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
1069
+ warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
1070
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:336: UserWarning: The use_host_meta is deprecated, it will be removed in the future.
1071
+ warnings.warn("The use_host_meta is deprecated, it will be removed in the future.")
1072
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:336: UserWarning: The use_host_meta is deprecated, it will be removed in the future.
1073
+ warnings.warn("The use_host_meta is deprecated, it will be removed in the future.")
1074
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:336: UserWarning: The use_host_meta is deprecated, it will be removed in the future.
1075
+ warnings.warn("The use_host_meta is deprecated, it will be removed in the future.")
1076
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:336: UserWarning: The use_host_meta is deprecated, it will be removed in the future.
1077
+ warnings.warn("The use_host_meta is deprecated, it will be removed in the future.")
1078
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:336: UserWarning: The use_host_meta is deprecated, it will be removed in the future.
1079
+ warnings.warn("The use_host_meta is deprecated, it will be removed in the future.")
1080
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:336: UserWarning: The use_host_meta is deprecated, it will be removed in the future.
1081
+ warnings.warn("The use_host_meta is deprecated, it will be removed in the future.")
1082
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:336: UserWarning: The use_host_meta is deprecated, it will be removed in the future.
1083
+ warnings.warn("The use_host_meta is deprecated, it will be removed in the future.")
1084
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:336: UserWarning: The use_host_meta is deprecated, it will be removed in the future.
1085
+ warnings.warn("The use_host_meta is deprecated, it will be removed in the future.")
1086
+ [2026-08-30 06:49:23] iteration 1/ 12 | consumed samples: 16 | elapsed time per iteration (ms): 38387.8 | throughput per GPU (TFLOP/s/GPU): 8.7 | learning rate: 2.000000E-05 | global batch size: 16 | lm loss: 1.102614E+01 | loss scale: 1.0 | grad norm: 44.313 | number of skipped iterations: 0 | number of nan iterations: 0 |
1087
+ Number of parameters in transformer block in billions: 23.22
1088
+ Number of parameters in embedding layers in billions: 0.41
1089
+ Total number of parameters in billions: 23.63
1090
+ Number of parameters in most loaded shard in billions: 23.6320
1091
+ Theoretical memory footprints: weight and optimizer=169029.43 MB
1092
+ [Rank 0] (after 1 iterations) memory (MB) | allocated: 63654.662109375 | max allocated: 85586.1083984375 | reserved: 88178.0 | max reserved: 88178.0
1093
+ [2026-08-30 06:49:23] iteration 2/ 12 | consumed samples: 32 | elapsed time per iteration (ms): 638.2 | throughput per GPU (TFLOP/s/GPU): 522.4 | learning rate: 4.000000E-05 | global batch size: 16 | lm loss: 1.101619E+01 | loss scale: 1.0 | grad norm: 45.749 | number of skipped iterations: 0 | number of nan iterations: 0 |
1094
+ [2026-08-30 06:49:24] iteration 3/ 12 | consumed samples: 48 | elapsed time per iteration (ms): 514.0 | throughput per GPU (TFLOP/s/GPU): 648.6 | learning rate: 6.000000E-05 | global batch size: 16 | lm loss: 1.032369E+01 | loss scale: 1.0 | grad norm: 204.565 | number of skipped iterations: 0 | number of nan iterations: 0 |
1095
+ Capture CUDA graph for training!!!
1096
+ CUDA graph capture done!!!
1097
+ [2026-08-30 06:49:27] iteration 4/ 12 | consumed samples: 64 | elapsed time per iteration (ms): 3146.3 | throughput per GPU (TFLOP/s/GPU): 106.0 | learning rate: 8.000000E-05 | global batch size: 16 | lm loss: 1.318609E+01 | loss scale: 1.0 | grad norm: 34.258 | number of skipped iterations: 0 | number of nan iterations: 0 |
1098
+ [2026-08-30 06:49:27] iteration 5/ 12 | consumed samples: 80 | elapsed time per iteration (ms): 415.8 | throughput per GPU (TFLOP/s/GPU): 801.7 | learning rate: 1.000000E-04 | global batch size: 16 | lm loss: 1.039552E+01 | loss scale: 1.0 | grad norm: 16.801 | number of skipped iterations: 0 | number of nan iterations: 0 |
1099
+ [2026-08-30 06:49:28] iteration 6/ 12 | consumed samples: 96 | elapsed time per iteration (ms): 415.6 | throughput per GPU (TFLOP/s/GPU): 802.2 | learning rate: 7.750000E-05 | global batch size: 16 | lm loss: 1.038954E+01 | loss scale: 1.0 | grad norm: 31.805 | number of skipped iterations: 0 | number of nan iterations: 0 |
1100
+ [2026-08-30 06:49:28] iteration 7/ 12 | consumed samples: 112 | elapsed time per iteration (ms): 416.1 | throughput per GPU (TFLOP/s/GPU): 801.3 | learning rate: 3.250000E-05 | global batch size: 16 | lm loss: 1.009632E+01 | loss scale: 1.0 | grad norm: 7.209 | number of skipped iterations: 0 | number of nan iterations: 0 |
1101
+ [2026-08-30 06:49:29] iteration 8/ 12 | consumed samples: 128 | elapsed time per iteration (ms): 415.6 | throughput per GPU (TFLOP/s/GPU): 802.2 | learning rate: 1.000000E-05 | global batch size: 16 | lm loss: 9.533159E+00 | loss scale: 1.0 | grad norm: 5.801 | number of skipped iterations: 0 | number of nan iterations: 0 |
1102
+ [2026-08-30 06:49:29] iteration 9/ 12 | consumed samples: 144 | elapsed time per iteration (ms): 417.4 | throughput per GPU (TFLOP/s/GPU): 798.7 | learning rate: 1.000000E-05 | global batch size: 16 | lm loss: 9.255018E+00 | loss scale: 1.0 | grad norm: 5.701 | number of skipped iterations: 0 | number of nan iterations: 0 |
1103
+ [2026-08-30 06:49:30] iteration 10/ 12 | consumed samples: 160 | elapsed time per iteration (ms): 413.7 | throughput per GPU (TFLOP/s/GPU): 805.9 | learning rate: 1.000000E-05 | global batch size: 16 | lm loss: 9.088298E+00 | loss scale: 1.0 | grad norm: 5.506 | number of skipped iterations: 0 | number of nan iterations: 0 |
1104
+ [2026-08-30 06:49:30] iteration 11/ 12 | consumed samples: 176 | elapsed time per iteration (ms): 415.5 | throughput per GPU (TFLOP/s/GPU): 802.3 | learning rate: 1.000000E-05 | global batch size: 16 | lm loss: 8.956705E+00 | loss scale: 1.0 | grad norm: 5.285 | number of skipped iterations: 0 | number of nan iterations: 0 |
1105
+ [2026-08-30 06:49:30] iteration 12/ 12 | consumed samples: 192 | elapsed time per iteration (ms): 415.3 | throughput per GPU (TFLOP/s/GPU): 802.8 | learning rate: 1.000000E-05 | global batch size: 16 | lm loss: 8.814399E+00 | loss scale: 1.0 | grad norm: 5.220 | number of skipped iterations: 0 | number of nan iterations: 0 |
1106
+ [after training is done] datetime: 2026-08-30 06:49:30
1107
+ [rank0]:[W830 06:49:32.380633503 ProcessGroupNCCL.cpp:1505] Warning: WARNING: destroy_process_group() was not called before program exit, which can leak resources. For more info, please see https://pytorch.org/docs/stable/distributed.html#shutdown (function operator())
tiera-rerun-fixed-moonep/echo/a2/a2_matched_gbs16/G2__echo__matched__seq4096__gbs16__cap1p08203125/rep2/mem_samples.csv ADDED
@@ -0,0 +1,184 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ 0, 0
2
+ 1, 0
3
+ 2, 0
4
+ 3, 0
5
+ 4, 0
6
+ 5, 0
7
+ 6, 0
8
+ 7, 0
9
+ 0, 0
10
+ 1, 0
11
+ 2, 0
12
+ 3, 0
13
+ 4, 0
14
+ 5, 0
15
+ 6, 0
16
+ 7, 0
17
+ 0, 1
18
+ 1, 1
19
+ 2, 1
20
+ 3, 1
21
+ 4, 1
22
+ 5, 1
23
+ 6, 1
24
+ 7, 1
25
+ 0, 4
26
+ 1, 4
27
+ 2, 4
28
+ 3, 4
29
+ 4, 1
30
+ 5, 1
31
+ 6, 1
32
+ 7, 1
33
+ 0, 4
34
+ 1, 4
35
+ 2, 4
36
+ 3, 4
37
+ 4, 4
38
+ 5, 4
39
+ 6, 4
40
+ 7, 4
41
+ 0, 821
42
+ 1, 819
43
+ 2, 819
44
+ 3, 819
45
+ 4, 821
46
+ 5, 819
47
+ 6, 821
48
+ 7, 821
49
+ 0, 1537
50
+ 1, 1537
51
+ 2, 1537
52
+ 3, 1537
53
+ 4, 1537
54
+ 5, 1537
55
+ 6, 1537
56
+ 7, 1537
57
+ 0, 2231
58
+ 1, 2231
59
+ 2, 2231
60
+ 3, 2231
61
+ 4, 2231
62
+ 5, 2231
63
+ 6, 2231
64
+ 7, 2231
65
+ 0, 39471
66
+ 1, 39477
67
+ 2, 39443
68
+ 3, 39459
69
+ 4, 39497
70
+ 5, 39463
71
+ 6, 39453
72
+ 7, 39471
73
+ 0, 39719
74
+ 1, 39711
75
+ 2, 39707
76
+ 3, 39701
77
+ 4, 39723
78
+ 5, 39693
79
+ 6, 39743
80
+ 7, 39735
81
+ 0, 42297
82
+ 1, 42289
83
+ 2, 42289
84
+ 3, 42289
85
+ 4, 42289
86
+ 5, 42289
87
+ 6, 42289
88
+ 7, 42289
89
+ 0, 42541
90
+ 1, 42533
91
+ 2, 42533
92
+ 3, 42533
93
+ 4, 42533
94
+ 5, 42533
95
+ 6, 42533
96
+ 7, 42533
97
+ 0, 42541
98
+ 1, 42533
99
+ 2, 42533
100
+ 3, 42533
101
+ 4, 42533
102
+ 5, 42533
103
+ 6, 42533
104
+ 7, 42533
105
+ 0, 43457
106
+ 1, 43449
107
+ 2, 43449
108
+ 3, 43449
109
+ 4, 43449
110
+ 5, 43449
111
+ 6, 43449
112
+ 7, 43449
113
+ 0, 43735
114
+ 1, 43727
115
+ 2, 43727
116
+ 3, 43727
117
+ 4, 43727
118
+ 5, 43727
119
+ 6, 43727
120
+ 7, 43727
121
+ 0, 45167
122
+ 1, 45159
123
+ 2, 45159
124
+ 3, 45159
125
+ 4, 45159
126
+ 5, 45159
127
+ 6, 45159
128
+ 7, 45159
129
+ 0, 45167
130
+ 1, 45159
131
+ 2, 45159
132
+ 3, 45159
133
+ 4, 45159
134
+ 5, 45159
135
+ 6, 45159
136
+ 7, 45159
137
+ 0, 75747
138
+ 1, 75739
139
+ 2, 75739
140
+ 3, 75739
141
+ 4, 75739
142
+ 5, 75739
143
+ 6, 75739
144
+ 7, 75739
145
+ 0, 76233
146
+ 1, 76225
147
+ 2, 76225
148
+ 3, 76225
149
+ 4, 76225
150
+ 5, 76225
151
+ 6, 76225
152
+ 7, 76225
153
+ 0, 76233
154
+ 1, 76225
155
+ 2, 76225
156
+ 3, 76225
157
+ 4, 76225
158
+ 5, 76225
159
+ 6, 76225
160
+ 7, 76225
161
+ 0, 76233
162
+ 1, 76225
163
+ 2, 76225
164
+ 3, 76225
165
+ 4, 76225
166
+ 5, 76225
167
+ 6, 76225
168
+ 7, 76225
169
+ 0, 94063
170
+ 1, 94037
171
+ 2, 94037
172
+ 3, 94073
173
+ 4, 94063
174
+ 5, 94053
175
+ 6, 94043
176
+ 7, 94055
177
+ 0, 94871
178
+ 1, 94859
179
+ 2, 94821
180
+ 3, 94895
181
+ 4, 94933
182
+ 5, 94957
183
+ 6, 94957
184
+ 7, 94957
tiera-rerun-fixed-moonep/echo/a2/a2_matched_gbs16/G2__echo__matched__seq4096__gbs16__cap1p08203125/rep2/run.env ADDED
@@ -0,0 +1,70 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ backend=echo
2
+ config=G2
3
+ family=tierA
4
+ repo=/workspace/Megatron-MoE-Echo
5
+ branch=bracing-regions-through-kernels
6
+ commit=9a236793
7
+ num_layers=16
8
+ hidden=4096
9
+ ffn_hidden=14336
10
+ moe_ffn_hidden=3584
11
+ num_experts=32
12
+ topk=8
13
+ experts_per_rank=4
14
+ expert_params=44040192
15
+ heads=32
16
+ kv_groups=8
17
+ kv_channels=
18
+ qk_layernorm=0
19
+ shared_ffn=0
20
+ shared_expert_overlap=0
21
+ moe_token_dispatcher=flex
22
+ mla=0
23
+ dense_first_layers=0
24
+ moe_layers=16
25
+ rotary_base=1000000
26
+ rotary_scaling=
27
+ qkv_bias=0
28
+ router_balancing=aux_loss
29
+ aux_loss_coeff=1e-2
30
+ router_pre_softmax=0
31
+ router_fusion=0
32
+ router_dtype=fp32
33
+ device_initiated_grouped_gemm=1
34
+ norm_epsilon=1e-5
35
+ seq_length=4096
36
+ micro_batch_size=1
37
+ global_batch_size=16
38
+ num_microbatches=2
39
+ assignments_per_rank=32768
40
+ mean_tokens_per_expert=8192
41
+ slot_policy=matched
42
+ slots_per_rank=4
43
+ capacity_mode=static
44
+ capacity=1.08203125
45
+ cuda_graph=1
46
+ probe_mode=0
47
+ async_grad_reduce=0
48
+ branch_role=timing
49
+ memory_dump_mode=none
50
+ memory_snapshot_path=
51
+ cuda_graph_warmup_steps=3
52
+ recompute=none
53
+ force_balance=0
54
+ train_iters=12
55
+ ep_size=8
56
+ echo_dispatch_overlap=1
57
+ echo_dispatch_recompute=0
58
+ ultraep_quota_min_tokens=
59
+ ultraep_weight_sync_join=
60
+ profile_enable=0
61
+ kernel_trace=0
62
+ fp8_mode=mxfp8
63
+ fp8_recipe=mxfp8
64
+ training_seed=1234
65
+ learning_rate=1e-4
66
+ min_learning_rate=1.0e-5
67
+ lr_warmup_iters=5
68
+ state_dump_enable=0
69
+ state_dump_dir=
70
+ moonep_assert_no_pending_reduce=0
tiera-rerun-fixed-moonep/echo/a2/a2_matched_gbs16/G2__echo__matched__seq4096__gbs16__cap1p08203125/rep2/train.log ADDED
@@ -0,0 +1,1085 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ==============================================================
2
+ G2 / echo (slots/rank=4 policy=matched)
3
+ E=32 K=8 H'=3584 m=4
4
+ L=16 seq=4096 gbs=16 (2 microbatches)
5
+ cap_mode=static cap=1.08203125 cg=1 recompute=none
6
+ memory_dump=none snapshots=/workspace/tiera-rerun-fixed-moonep/echo/a2/a2_matched_gbs16/G2__echo__matched__seq4096__gbs16__cap1p08203125/rep2/memory_snapshots/snapshot.pickle
7
+ repo=/workspace/Megatron-MoE-Echo @ 9a236793
8
+ ==============================================================
9
+ W0830 06:49:40.874000 665587 torch/distributed/run.py:766]
10
+ W0830 06:49:40.874000 665587 torch/distributed/run.py:766] *****************************************
11
+ W0830 06:49:40.874000 665587 torch/distributed/run.py:766] Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed.
12
+ W0830 06:49:40.874000 665587 torch/distributed/run.py:766] *****************************************
13
+ /workspace/Megatron-MoE-Echo/megatron/training/arguments.py:831: UserWarning: Disabling sequence parallelism because tensor model parallelism is disabled
14
+ warnings.warn("Disabling sequence parallelism because tensor model parallelism is disabled")
15
+ WARNING: one_logger package is required to enable e2e metrics tracking. please go to https://confluence.nvidia.com/display/MLWFO/Package+Repositories for details to install it
16
+ /workspace/Megatron-MoE-Echo/megatron/training/arguments.py:831: UserWarning: Disabling sequence parallelism because tensor model parallelism is disabled
17
+ warnings.warn("Disabling sequence parallelism because tensor model parallelism is disabled")
18
+ using world size: 8, data-parallel size: 8, context-parallel size: 1, hierarchical context-parallel sizes: None, tensor-model-parallel size: 1, pipeline-model-parallel size: 1
19
+ WARNING: overriding default arguments for tokenizer_type:GPT2BPETokenizer with tokenizer_type:GPT2BPETokenizer
20
+ Number of virtual stages per pipeline stage: None
21
+ accumulate and all-reduce gradients in fp32 for bfloat16 data type.
22
+ using torch.bfloat16 for parameters ...
23
+ /workspace/Megatron-MoE-Echo/megatron/training/arguments.py:831: UserWarning: Disabling sequence parallelism because tensor model parallelism is disabled
24
+ warnings.warn("Disabling sequence parallelism because tensor model parallelism is disabled")
25
+ ------------------------ arguments ------------------------
26
+ account_for_embedding_in_pipeline_split ......... False
27
+ account_for_loss_in_pipeline_split .............. False
28
+ accumulate_allreduce_grads_in_fp32 .............. True
29
+ adam_beta1 ...................................... 0.9
30
+ adam_beta2 ...................................... 0.999
31
+ adam_eps ........................................ 1e-08
32
+ add_bias_linear ................................. False
33
+ add_position_embedding .......................... False
34
+ add_qkv_bias .................................... False
35
+ adlr_autoresume ................................. False
36
+ adlr_autoresume_interval ........................ 1000
37
+ align_grad_reduce ............................... True
38
+ align_param_gather .............................. False
39
+ app_tag_run_name ................................ None
40
+ app_tag_run_version ............................. 0.0.0
41
+ apply_layernorm_1p .............................. False
42
+ apply_query_key_layer_scaling ................... False
43
+ apply_residual_connection_post_layernorm ........ False
44
+ apply_rope_fusion ............................... True
45
+ async_save ...................................... None
46
+ async_tensor_model_parallel_allreduce ........... True
47
+ attention_backend ............................... AttnBackend.auto
48
+ attention_dropout ............................... 0.0
49
+ attention_softmax_in_fp32 ....................... False
50
+ auto_detect_ckpt_format ......................... False
51
+ barrier_with_L1_time ............................ True
52
+ bert_binary_head ................................ True
53
+ bert_embedder_type .............................. megatron
54
+ bert_load ....................................... None
55
+ bf16 ............................................ True
56
+ bias_dropout_fusion ............................. True
57
+ bias_gelu_fusion ................................ False
58
+ bias_swiglu_fusion .............................. True
59
+ biencoder_projection_dim ........................ 0
60
+ biencoder_shared_query_context_model ............ False
61
+ block_data_path ................................. None
62
+ calc_ft_timeouts ................................ False
63
+ calculate_per_token_loss ........................ False
64
+ check_for_large_grads ........................... False
65
+ check_for_nan_in_loss_and_grad .................. False
66
+ check_for_spiky_loss ............................ False
67
+ check_weight_hash_across_dp_replicas_interval ... None
68
+ ckpt_assume_constant_structure .................. False
69
+ ckpt_convert_format ............................. None
70
+ ckpt_convert_save ............................... None
71
+ ckpt_convert_update_legacy_dist_opt_format ...... False
72
+ ckpt_format ..................................... torch_dist
73
+ ckpt_fully_parallel_load ........................ False
74
+ ckpt_fully_parallel_save ........................ True
75
+ ckpt_fully_parallel_save_deprecated ............. False
76
+ ckpt_step ....................................... None
77
+ classes_fraction ................................ 1.0
78
+ clip_grad ....................................... 1.0
79
+ clone_scatter_output_in_embedding ............... True
80
+ config_logger_dir ...............................
81
+ consumed_train_samples .......................... 0
82
+ consumed_valid_samples .......................... 0
83
+ context_parallel_size ........................... 1
84
+ cp_comm_type .................................... ['p2p']
85
+ create_attention_mask_in_dataloader ............. True
86
+ cross_entropy_fusion_impl ....................... native
87
+ cross_entropy_loss_fusion ....................... False
88
+ cuda_graph_scope ................................ full_iteration
89
+ cuda_graph_warmup_steps ......................... 3
90
+ data_args_path .................................. None
91
+ data_cache_path ................................. None
92
+ data_parallel_random_init ....................... False
93
+ data_parallel_sharding_strategy ................. no_shard
94
+ data_parallel_size .............................. 8
95
+ data_path ....................................... ['/workspace/data/wikitext/wikitext103_gpt2_text_document']
96
+ data_per_class_fraction ......................... 1.0
97
+ data_sharding ................................... True
98
+ dataloader_type ................................. single
99
+ ddp_average_in_collective ....................... False
100
+ ddp_bucket_size ................................. None
101
+ ddp_num_buckets ................................. None
102
+ ddp_pad_buckets_for_high_nccl_busbw ............. False
103
+ decoder_first_pipeline_num_layers ............... None
104
+ decoder_last_pipeline_num_layers ................ None
105
+ decoder_num_layers .............................. None
106
+ decoder_seq_length .............................. None
107
+ decoupled_lr .................................... None
108
+ decoupled_min_lr ................................ None
109
+ decrease_batch_size_if_needed ................... False
110
+ defer_embedding_wgrad_compute ................... False
111
+ delay_wgrad_compute ............................. False
112
+ deprecated_use_mcore_models ..................... True
113
+ deterministic_mode .............................. False
114
+ dino_bottleneck_size ............................ 256
115
+ dino_freeze_last_layer .......................... 1
116
+ dino_head_hidden_size ........................... 2048
117
+ dino_local_crops_number ......................... 10
118
+ dino_local_img_size ............................. 96
119
+ dino_norm_last_layer ............................ False
120
+ dino_teacher_temp ............................... 0.07
121
+ dino_warmup_teacher_temp ........................ 0.04
122
+ dino_warmup_teacher_temp_epochs ................. 30
123
+ disable_bf16_reduced_precision_matmul ........... False
124
+ disable_mamba_mem_eff_path ...................... False
125
+ disable_straggler_on_startup .................... False
126
+ dist_ckpt_format_deprecated ..................... None
127
+ dist_ckpt_strictness ............................ assume_ok_unexpected
128
+ distribute_saved_activations .................... False
129
+ distributed_backend ............................. nccl
130
+ distributed_timeout_minutes ..................... 10
131
+ embedding_init_method_std ....................... None
132
+ embedding_path .................................. None
133
+ empty_unused_memory_level ....................... 0
134
+ enable_cuda_graph ............................... True
135
+ enable_experimental ............................. False
136
+ enable_ft_package ............................... False
137
+ enable_gloo_process_groups ...................... True
138
+ enable_msc ...................................... True
139
+ enable_one_logger ............................... True
140
+ encoder_num_layers .............................. 16
141
+ encoder_seq_length .............................. 4096
142
+ end_weight_decay ................................ 0.1
143
+ eod_mask_loss ................................... False
144
+ error_injection_rate ............................ 0
145
+ error_injection_type ............................ transient_error
146
+ eval_interval ................................... 1000000
147
+ eval_iters ...................................... 0
148
+ evidence_data_path .............................. None
149
+ exit_duration_in_mins ........................... None
150
+ exit_interval ................................... None
151
+ exit_on_missing_checkpoint ...................... False
152
+ exit_signal_handler ............................. False
153
+ exp_avg_dtype ................................... torch.float32
154
+ exp_avg_sq_dtype ................................ torch.float32
155
+ expert_model_parallel_size ...................... 8
156
+ expert_tensor_parallel_size ..................... 1
157
+ export_force_local_attention .................... False
158
+ export_kd_cfg ................................... None
159
+ export_kd_teacher_ckpt_format ................... None
160
+ export_kd_teacher_load .......................... None
161
+ export_kv_cache_quant ........................... False
162
+ export_legacy_megatron .......................... False
163
+ export_model_type ............................... GPTModel
164
+ export_moe_apply_probs_on_input ................. False
165
+ export_qk_l2_norm ............................... False
166
+ export_quant_cfg ................................ None
167
+ export_real_quant_cfg ........................... None
168
+ export_te_mcore_model ........................... False
169
+ external_cuda_graph ............................. False
170
+ ffn_hidden_size ................................. 14336
171
+ finetune ........................................ False
172
+ finetune_data_split ............................. train
173
+ finetune_hf_dataset ............................. None
174
+ first_last_layers_bf16 .......................... False
175
+ flash_decode .................................... False
176
+ fp16 ............................................ False
177
+ fp16_lm_cross_entropy ........................... False
178
+ fp32_residual_connection ........................ False
179
+ fp8 ............................................. e4m3
180
+ fp8_amax_compute_algo ........................... most_recent
181
+ fp8_amax_history_len ............................ 1
182
+ fp8_interval .................................... 1
183
+ fp8_margin ...................................... 0
184
+ fp8_param_gather ................................ False
185
+ fp8_recipe ...................................... mxfp8
186
+ fp8_wgrad ....................................... True
187
+ fsdp_double_buffer .............................. False
188
+ global_batch_size ............................... 16
189
+ grad_reduce_in_bf16 ............................. False
190
+ gradient_accumulation_fusion .................... True
191
+ gradient_reduce_div_fusion ...................... True
192
+ group_query_attention ........................... True
193
+ head_lr_mult .................................... 1.0
194
+ heterogeneous_layers_config_encoded_json ........ None
195
+ heterogeneous_layers_config_path ................ None
196
+ hidden_dropout .................................. 0.0
197
+ hidden_size ..................................... 4096
198
+ hierarchical_context_parallel_sizes ............. None
199
+ high_priority_stream_groups ..................... []
200
+ hybrid_attention_ratio .......................... 0.0
201
+ hybrid_mlp_ratio ................................ 0.0
202
+ hybrid_override_pattern ......................... None
203
+ hysteresis ...................................... 2
204
+ ict_head_size ................................... None
205
+ ict_load ........................................ None
206
+ img_h ........................................... 224
207
+ img_w ........................................... 224
208
+ indexer_batch_size .............................. 128
209
+ indexer_log_interval ............................ 1000
210
+ inference_batch_times_seqlen_threshold .......... -1
211
+ inference_dynamic_batching ...................... False
212
+ inference_dynamic_batching_buffer_guaranteed_fraction 0.2
213
+ inference_dynamic_batching_buffer_overflow_factor None
214
+ inference_dynamic_batching_buffer_size_gb ....... 40.0
215
+ inference_dynamic_batching_chunk_size ........... 256
216
+ inference_dynamic_batching_max_requests_override None
217
+ inference_dynamic_batching_max_tokens_override .. None
218
+ inference_dynamic_batching_num_cuda_graphs ...... 16
219
+ inference_max_batch_size ........................ 8
220
+ inference_max_seq_length ........................ 2560
221
+ inference_rng_tracker ........................... False
222
+ init_method_std ................................. 0.01
223
+ init_method_xavier_uniform ...................... False
224
+ init_model_with_meta_device ..................... False
225
+ initial_loss_scale .............................. 4294967296
226
+ inprocess_active_world_size ..................... 8
227
+ inprocess_barrier_timeout ....................... 120
228
+ inprocess_completion_timeout .................... 120
229
+ inprocess_empty_cuda_cache ...................... False
230
+ inprocess_granularity ........................... node
231
+ inprocess_hard_timeout .......................... 90
232
+ inprocess_heartbeat_interval .................... 30
233
+ inprocess_heartbeat_timeout ..................... 60
234
+ inprocess_last_call_wait ........................ 1
235
+ inprocess_max_iterations ........................ None
236
+ inprocess_monitor_process_interval .............. 1.0
237
+ inprocess_monitor_thread_interval ............... 1.0
238
+ inprocess_progress_watchdog_interval ............ 1.0
239
+ inprocess_restart ............................... False
240
+ inprocess_soft_timeout .......................... 60
241
+ inprocess_termination_grace_time ................ 1
242
+ is_hybrid_model ................................. False
243
+ iter_per_epoch .................................. 1250
244
+ iterations_to_skip .............................. []
245
+ keep_fp8_transpose_cache_when_using_custom_fsdp . False
246
+ kitchen_config_file ............................. None
247
+ kitchen_recipe_number ........................... None
248
+ kv_channels ..................................... 128
249
+ kv_lora_rank .................................... 32
250
+ lazy_mpu_init ................................... None
251
+ load ............................................ None
252
+ load_main_params_from_ckpt ...................... None
253
+ load_model_opt_format ........................... False
254
+ local_rank ...................................... 0
255
+ log_energy ...................................... False
256
+ log_interval .................................... 1
257
+ log_loss_scale_to_tensorboard ................... True
258
+ log_memory_to_tensorboard ....................... False
259
+ log_num_zeros_in_grad ........................... False
260
+ log_params_norm ................................. False
261
+ log_progress .................................... False
262
+ log_straggler ................................... False
263
+ log_throughput .................................. True
264
+ log_timers_to_tensorboard ....................... False
265
+ log_validation_ppl_to_tensorboard ............... False
266
+ log_world_size_to_tensorboard ................... False
267
+ logging_level ................................... None
268
+ loss_scale ...................................... None
269
+ loss_scale_window ............................... 1000
270
+ lr .............................................. 0.0001
271
+ lr_decay_iters .................................. 8
272
+ lr_decay_samples ................................ None
273
+ lr_decay_style .................................. cosine
274
+ lr_warmup_fraction .............................. None
275
+ lr_warmup_init .................................. 0.0
276
+ lr_warmup_iters ................................. 5
277
+ lr_warmup_samples ............................... 0
278
+ lr_wsd_decay_iters .............................. None
279
+ lr_wsd_decay_samples ............................ None
280
+ lr_wsd_decay_style .............................. exponential
281
+ main_grads_dtype ................................ torch.float32
282
+ main_params_dtype ............................... torch.float32
283
+ make_vocab_size_divisible_by .................... 128
284
+ mamba_head_dim .................................. 64
285
+ mamba_num_groups ................................ 8
286
+ mamba_num_heads ................................. None
287
+ mamba_state_dim ................................. 128
288
+ manual_gc ....................................... False
289
+ manual_gc_eval .................................. True
290
+ manual_gc_interval .............................. 0
291
+ mask_factor ..................................... 1.0
292
+ mask_prob ....................................... 0.15
293
+ mask_type ....................................... random
294
+ masked_softmax_fusion ........................... False
295
+ max_position_embeddings ......................... 32768
296
+ max_tokens_to_oom ............................... 12000
297
+ memory_snapshot_path ............................ snapshot.pickle
298
+ merge_file ...................................... /workspace/data/wikitext/gpt2-merges.txt
299
+ micro_batch_size ................................ 1
300
+ microbatch_group_size_per_vp_stage .............. None
301
+ mid_level_dataset_surplus ....................... 0.005
302
+ min_loss_scale .................................. 1.0
303
+ min_lr .......................................... 1e-05
304
+ mlp_chunks_for_prefill .......................... 1
305
+ mmap_bin_files .................................. True
306
+ mock_data ....................................... False
307
+ moe_apply_probs_on_input ........................ False
308
+ moe_aux_loss_coeff .............................. 0.01
309
+ moe_deepep_num_sms .............................. 20
310
+ moe_echo_enable_random_offloading ............... False
311
+ moe_echo_expert_dispatch_overlap ................ True
312
+ moe_echo_expert_dispatcher_type ................. hybridep
313
+ moe_echo_recompute_expert_dispatch .............. False
314
+ moe_enable_deepep ............................... False
315
+ moe_enable_echo ................................. True
316
+ moe_enable_hybridep ............................. True
317
+ moe_expert_capacity_factor ...................... None
318
+ moe_extended_tp ................................. False
319
+ moe_ffn_hidden_size ............................. 3584
320
+ moe_grouped_gemm ................................ True
321
+ moe_hybridep_num_sms ............................ 16
322
+ moe_input_jitter_eps ............................ None
323
+ moe_layer_freq .................................. 1
324
+ moe_layer_recompute ............................. False
325
+ moe_num_echo_experts ............................ 32
326
+ moe_pad_expert_input_to_capacity ................ False
327
+ moe_per_layer_logging ........................... False
328
+ moe_permute_fusion .............................. False
329
+ moe_received_token_capacity ..................... 1.08203125
330
+ moe_router_bias_update_rate ..................... 0.001
331
+ moe_router_dtype ................................ fp32
332
+ moe_router_enable_expert_bias ................... False
333
+ moe_router_force_load_balancing ................. False
334
+ moe_router_group_topk ........................... None
335
+ moe_router_load_balancing_type .................. aux_loss
336
+ moe_router_num_groups ........................... None
337
+ moe_router_padding_for_fp8 ...................... False
338
+ moe_router_pre_softmax .......................... False
339
+ moe_router_score_function ....................... softmax
340
+ moe_router_topk ................................. 8
341
+ moe_router_topk_scaling_factor .................. None
342
+ moe_shared_expert_intermediate_size ............. None
343
+ moe_shared_expert_overlap ....................... False
344
+ moe_token_dispatcher_type ....................... flex
345
+ moe_token_drop_policy ........................... probs
346
+ moe_upcycling_granularity ....................... 1
347
+ moe_use_device_initiated_grouped_gemm ........... True
348
+ moe_use_legacy_grouped_gemm ..................... False
349
+ moe_use_upcycling ............................... False
350
+ moe_z_loss_coeff ................................ None
351
+ mrope_section ................................... None
352
+ mscale .......................................... 1.0
353
+ mscale_all_dim .................................. 0.0
354
+ mtp_loss_scaling_factor ......................... 0.1
355
+ mtp_num_layers .................................. None
356
+ multi_latent_attention .......................... False
357
+ nccl_all_reduce_for_prefill ..................... False
358
+ nccl_communicator_config_path ................... None
359
+ nccl_ub ......................................... False
360
+ no_load_optim ................................... None
361
+ no_load_rng ..................................... None
362
+ no_persist_layer_norm ........................... False
363
+ no_rope_freq .................................... None
364
+ no_save_optim ................................... None
365
+ no_save_rng ..................................... None
366
+ non_persistent_ckpt_type ........................ None
367
+ non_persistent_global_ckpt_dir .................. None
368
+ non_persistent_local_ckpt_algo .................. fully_parallel
369
+ non_persistent_local_ckpt_dir ................... None
370
+ non_persistent_save_interval .................... None
371
+ norm_epsilon .................................... 1e-05
372
+ normalization ................................... RMSNorm
373
+ num_attention_heads ............................. 32
374
+ num_channels .................................... 3
375
+ num_classes ..................................... 1000
376
+ num_dataset_builder_threads ..................... 1
377
+ num_distributed_optimizer_instances ............. 1
378
+ num_experts ..................................... 32
379
+ num_layers ...................................... 16
380
+ num_layers_at_end_in_bf16 ....................... 1
381
+ num_layers_at_start_in_bf16 ..................... 1
382
+ num_layers_per_virtual_pipeline_stage ........... None
383
+ num_query_groups ................................ 8
384
+ num_virtual_stages_per_pipeline_rank ............ None
385
+ num_workers ..................................... 2
386
+ object_storage_cache_path ....................... None
387
+ one_logger_async ................................ False
388
+ one_logger_project .............................. megatron-lm
389
+ one_logger_run_name ............................. None
390
+ onnx_safe ....................................... None
391
+ openai_gelu ..................................... False
392
+ optimizer ....................................... adam
393
+ optimizer_cpu_offload ........................... False
394
+ optimizer_offload_fraction ...................... 1.0
395
+ output_bert_embeddings .......................... False
396
+ overlap_cpu_optimizer_d2h_h2d ................... False
397
+ overlap_grad_reduce ............................. True
398
+ overlap_moe_expert_parallel_comm ................ False
399
+ overlap_p2p_comm ................................ False
400
+ overlap_p2p_comm_warmup_flush ................... False
401
+ overlap_param_gather ............................ True
402
+ overlap_param_gather_with_optimizer_step ........ False
403
+ override_opt_param_scheduler .................... False
404
+ params_dtype .................................... torch.bfloat16
405
+ patch_dim ....................................... 16
406
+ per_split_data_args_path ........................ None
407
+ perform_initialization .......................... True
408
+ pin_cpu_grads ................................... True
409
+ pin_cpu_params .................................. True
410
+ pipeline_model_parallel_comm_backend ............ None
411
+ pipeline_model_parallel_layout .................. None
412
+ pipeline_model_parallel_size .................... 1
413
+ position_embedding_type ......................... rope
414
+ pretrained_checkpoint ........................... None
415
+ profile ......................................... False
416
+ profile_ranks ................................... [0]
417
+ profile_step_end ................................ 12
418
+ profile_step_start .............................. 10
419
+ q_lora_rank ..................................... None
420
+ qk_head_dim ..................................... 128
421
+ qk_l2_norm ...................................... False
422
+ qk_layernorm .................................... False
423
+ qk_pos_emb_head_dim ............................. 64
424
+ query_in_block_prob ............................. 0.1
425
+ rampup_batch_size ............................... None
426
+ rank ............................................ 0
427
+ recompute_granularity ........................... None
428
+ recompute_method ................................ None
429
+ recompute_modules ............................... None
430
+ recompute_num_layers ............................ None
431
+ record_memory_history ........................... False
432
+ relative_attention_max_distance ................. 128
433
+ relative_attention_num_buckets .................. 32
434
+ replication ..................................... False
435
+ replication_factor .............................. 2
436
+ replication_jump ................................ None
437
+ rerun_mode ...................................... validate_results
438
+ reset_attention_mask ............................ False
439
+ reset_position_ids .............................. False
440
+ result_rejected_tracker_filename ................ None
441
+ retriever_report_topk_accuracies ................ []
442
+ retriever_score_scaling ......................... False
443
+ retriever_seq_length ............................ 256
444
+ retro_add_retriever ............................. False
445
+ retro_attention_gate ............................ 1
446
+ retro_cyclic_train_iters ........................ None
447
+ retro_encoder_attention_dropout ................. 0.1
448
+ retro_encoder_hidden_dropout .................... 0.1
449
+ retro_encoder_layers ............................ 2
450
+ retro_num_neighbors ............................. 2
451
+ retro_num_retrieved_chunks ...................... 2
452
+ retro_project_dir ............................... None
453
+ retro_verify_neighbor_count ..................... True
454
+ reuse_grad_buf_for_mxfp8_param_ag ............... False
455
+ rope_scaling_factor ............................. 8.0
456
+ rope_type ....................................... None
457
+ rotary_base ..................................... 1000000
458
+ rotary_interleaved .............................. False
459
+ rotary_percent .................................. 1.0
460
+ rotary_scaling_factor ........................... 1.0
461
+ rotary_seq_len_interpolation_factor ............. None
462
+ run_workload_inspector_server ................... False
463
+ sample_rate ..................................... 1.0
464
+ save ............................................ None
465
+ save_interval ................................... None
466
+ save_retain_interval ............................ None
467
+ scatter_gather_tensors_in_pipeline .............. True
468
+ seed ............................................ 1234
469
+ seq_length ...................................... 4096
470
+ sequence_parallel ............................... False
471
+ sft ............................................. False
472
+ sft_tokenizer_prompt_format ..................... nemotron-h-aligned
473
+ sgd_momentum .................................... 0.9
474
+ short_seq_prob .................................. 0.1
475
+ skip_train ...................................... False
476
+ skipped_train_samples ........................... 0
477
+ spec ............................................ None
478
+ split ........................................... 100,0,0
479
+ squared_relu .................................... False
480
+ start_weight_decay .............................. 0.1
481
+ straggler_ctrlr_port ............................ 65535
482
+ straggler_minmax_count .......................... 1
483
+ suggested_communication_unit_size ............... None
484
+ swiglu .......................................... True
485
+ swin_backbone_type .............................. tiny
486
+ symmetric_ar_type ............................... None
487
+ te_rng_tracker .................................. True
488
+ tensor_model_parallel_size ...................... 1
489
+ tensorboard_dir ................................. None
490
+ tensorboard_log_interval ........................ 1
491
+ tensorboard_queue_size .......................... 1000
492
+ test_data_path .................................. None
493
+ test_mode ....................................... False
494
+ tiktoken_num_special_tokens ..................... 1000
495
+ tiktoken_pattern ................................ None
496
+ tiktoken_special_tokens ......................... None
497
+ timing_log_level ................................ 0
498
+ timing_log_option ............................... minmax
499
+ titles_data_path ................................ None
500
+ tokenizer_model ................................. None
501
+ tokenizer_type .................................. GPT2BPETokenizer
502
+ torch_fsdp2_reshard_after_forward ............... True
503
+ tp_comm_bootstrap_backend ....................... nccl
504
+ tp_comm_bulk_dgrad .............................. True
505
+ tp_comm_bulk_wgrad .............................. True
506
+ tp_comm_overlap ................................. False
507
+ tp_comm_overlap_ag .............................. True
508
+ tp_comm_overlap_cfg ............................. None
509
+ tp_comm_overlap_rs .............................. True
510
+ tp_comm_overlap_rs_dgrad ........................ False
511
+ tp_comm_split_ag ................................ True
512
+ tp_comm_split_rs ................................ True
513
+ train_data_path ................................. None
514
+ train_iters ..................................... 12
515
+ train_samples ................................... None
516
+ train_sync_interval ............................. None
517
+ transformer_impl ................................ transformer_engine
518
+ transformer_pipeline_model_parallel_size ........ 1
519
+ untie_embeddings_and_output_weights ............. True
520
+ use_checkpoint_args ............................. False
521
+ use_checkpoint_opt_param_scheduler .............. False
522
+ use_cpu_initialization .......................... None
523
+ use_custom_fsdp ................................. False
524
+ use_dist_ckpt ................................... True
525
+ use_dist_ckpt_deprecated ........................ False
526
+ use_distributed_optimizer ....................... True
527
+ use_flash_attn .................................. False
528
+ use_legacy_models ............................... False
529
+ use_mp_args_from_checkpoint_args ................ False
530
+ use_one_sent_docs ............................... False
531
+ use_persistent_ckpt_worker ...................... False
532
+ use_precision_aware_optimizer ................... False
533
+ use_pytorch_profiler ............................ False
534
+ use_ring_exchange_p2p ........................... False
535
+ use_rope_scaling ................................ False
536
+ use_rotary_position_embeddings .................. False
537
+ use_sharp ....................................... False
538
+ use_tokenizer_model_from_checkpoint_args ........ True
539
+ use_torch_fsdp2 ................................. False
540
+ use_torch_optimizer_for_cpu_offload ............. False
541
+ use_tp_pp_dp_mapping ............................ False
542
+ v_head_dim ...................................... 128
543
+ valid_data_path ................................. None
544
+ variable_seq_lengths ............................ False
545
+ virtual_pipeline_model_parallel_size ............ None
546
+ vision_backbone_type ............................ vit
547
+ vision_pretraining .............................. False
548
+ vision_pretraining_type ......................... classify
549
+ vocab_extra_ids ................................. 0
550
+ vocab_file ...................................... /workspace/data/wikitext/gpt2-vocab.json
551
+ vocab_size ...................................... None
552
+ wandb_exp_name ..................................
553
+ wandb_project ...................................
554
+ wandb_save_dir ..................................
555
+ weight_decay .................................... 0.1
556
+ weight_decay_incr_style ......................... constant
557
+ wgrad_deferral_limit ............................ 0
558
+ world_size ...................................... 8
559
+ yaml_cfg ........................................ None
560
+ -------------------- end of arguments ---------------------
561
+ INFO:megatron.core.num_microbatches_calculator:setting number of microbatches to constant 2
562
+ > building GPT2BPETokenizer tokenizer ...
563
+ /workspace/Megatron-MoE-Echo/megatron/training/arguments.py:831: UserWarning: Disabling sequence parallelism because tensor model parallelism is disabled
564
+ warnings.warn("Disabling sequence parallelism because tensor model parallelism is disabled")
565
+ > padded vocab (size: 50257) with 47 dummy tokens (new size: 50304)
566
+ WARNING:megatron.core.rerun_state_machine:RerunStateMachine initialized in mode RerunMode.VALIDATE_RESULTS
567
+ > initializing torch distributed ...
568
+ /workspace/Megatron-MoE-Echo/megatron/training/arguments.py:831: UserWarning: Disabling sequence parallelism because tensor model parallelism is disabled
569
+ warnings.warn("Disabling sequence parallelism because tensor model parallelism is disabled")
570
+ /workspace/Megatron-MoE-Echo/megatron/training/arguments.py:831: UserWarning: Disabling sequence parallelism because tensor model parallelism is disabled
571
+ warnings.warn("Disabling sequence parallelism because tensor model parallelism is disabled")
572
+ /workspace/Megatron-MoE-Echo/megatron/training/arguments.py:831: UserWarning: Disabling sequence parallelism because tensor model parallelism is disabled
573
+ warnings.warn("Disabling sequence parallelism because tensor model parallelism is disabled")
574
+ /workspace/Megatron-MoE-Echo/megatron/training/arguments.py:831: UserWarning: Disabling sequence parallelism because tensor model parallelism is disabled
575
+ warnings.warn("Disabling sequence parallelism because tensor model parallelism is disabled")
576
+ [Gloo] Rank 1 is connected to [Gloo] Rank 7 peer ranks. Expected number of connected peer ranks is : 07 is connected to
577
+ 7 peer ranks. Expected number of connected peer ranks is : 7
578
+ [Gloo] Rank 3 is connected to 7 peer ranks. Expected number of connected peer ranks is : 7
579
+ [Gloo] Rank 5 is connected to 7 peer ranks. Expected number of connected peer ranks is : 7
580
+ [Gloo] Rank 7 is connected to 7 peer ranks. Expected number of connected peer ranks is : 7
581
+ [Gloo] Rank 2 is connected to 7 peer ranks. Expected number of connected peer ranks is : 7
582
+ [Gloo] Rank 6 is connected to 7 peer ranks. Expected number of connected peer ranks is : 7
583
+ [Gloo] Rank 4 is connected to 7 peer ranks. Expected number of connected peer ranks is : 7
584
+ [Gloo] Rank 0 is connected to 7 peer ranks. Expected number of connected peer ranks is : 7
585
+ [Gloo] Rank 2 is connected to 7 peer ranks. Expected number of connected peer ranks is : 7
586
+ [Gloo] Rank 3 is connected to 7 peer ranks. Expected number of connected peer ranks is : 7
587
+ [Gloo] Rank [Gloo] Rank [Gloo] Rank [Gloo] Rank [Gloo] Rank 1465 is connected to 7 is connected to is connected to is connected to 7 is connected to 777 peer ranks. 7 peer ranks. peer ranks. peer ranks. Expected number of connected peer ranks is : peer ranks. Expected number of connected peer ranks is : Expected number of connected peer ranks is : Expected number of connected peer ranks is : 7Expected number of connected peer ranks is :
588
+ 77
589
+
590
+ 77
591
+
592
+ [Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
593
+ > initialized tensor model parallel with size 1
594
+ > initialized pipeline model parallel with size 1
595
+ > setting random seeds to 1234 ...
596
+ [Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
597
+ [Gloo] Rank [Gloo] Rank 00 is connected to is connected to 00 peer ranks. peer ranks. Expected number of connected peer ranks is : Expected number of connected peer ranks is : 00
598
+
599
+ [Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
600
+ [Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
601
+ [Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
602
+ [Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
603
+ [rank7]:[W830 06:49:58.890026700 ProcessGroupNCCL.cpp:4789] [PG ID 0 PG GUID 0 Rank 7] using GPU 7 as device used by this process is currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect. You can specify device_id in init_process_group() to force use of a particular device.
604
+ [rank5]:[W830 06:49:58.898308384 ProcessGroupNCCL.cpp:4789] [PG ID 0 PG GUID 0 Rank 5] using GPU 5 as device used by this process is currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect. You can specify device_id in init_process_group() to force use of a particular device.
605
+ [rank2]:[W830 06:49:58.898366225 ProcessGroupNCCL.cpp:4789] [PG ID 0 PG GUID 0 Rank 2] using GPU 2 as device used by this process is currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect. You can specify device_id in init_process_group() to force use of a particular device.
606
+ [rank4]:[W830 06:49:58.898414112 ProcessGroupNCCL.cpp:4789] [PG ID 0 PG GUID 0 Rank 4] using GPU 4 as device used by this process is currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect. You can specify device_id in init_process_group() to force use of a particular device.
607
+ [rank6]:[W830 06:49:58.898462177 ProcessGroupNCCL.cpp:4789] [PG ID 0 PG GUID 0 Rank 6] using GPU 6 as device used by this process is currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect. You can specify device_id in init_process_group() to force use of a particular device.
608
+ [rank3]:[W830 06:49:58.898462684 ProcessGroupNCCL.cpp:4789] [PG ID 0 PG GUID 0 Rank 3] using GPU 3 as device used by this process is currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect. You can specify device_id in init_process_group() to force use of a particular device.
609
+ [rank1]:[W830 06:49:58.899674441 ProcessGroupNCCL.cpp:4789] [PG ID 0 PG GUID 0 Rank 1] using GPU 1 as device used by this process is currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect. You can specify device_id in init_process_group() to force use of a particular device.
610
+ > compiling dataset index builder ...
611
+ make: Entering directory '/workspace/Megatron-MoE-Echo/megatron/core/datasets'
612
+ make: Nothing to be done for 'default'.
613
+ make: Leaving directory '/workspace/Megatron-MoE-Echo/megatron/core/datasets'
614
+ >>> done with dataset index builder. Compilation time: 0.035 seconds
615
+ WARNING: constraints for invoking optimized fused softmax kernel are not met. We default back to unfused kernel invocations.
616
+ > compiling and loading fused kernels ...
617
+ [rank0]:[W830 06:49:58.045827445 ProcessGroupNCCL.cpp:4789] [PG ID 0 PG GUID 0 Rank 0] using GPU 0 as device used by this process is currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect. You can specify device_id in init_process_group() to force use of a particular device.
618
+ >>> done with compiling and loading fused kernels. Compilation time: 3.921 seconds
619
+ time to initialize megatron (seconds): 7.329
620
+ [after megatron is initialized] datetime: 2026-08-30 06:50:04
621
+ building GPT model ...
622
+ /usr/local/lib/python3.12/dist-packages/transformer_engine/pytorch/cpu_offload.py:708: DeprecationWarning: Offloading weights is deprecated. Using offload_weights=True does not have any effect.
623
+ warnings.warn(
624
+ /usr/local/lib/python3.12/dist-packages/transformer_engine/pytorch/cpu_offload.py:708: DeprecationWarning: Offloading weights is deprecated. Using offload_weights=True does not have any effect.
625
+ warnings.warn(
626
+ /usr/local/lib/python3.12/dist-packages/transformer_engine/pytorch/cpu_offload.py:708: DeprecationWarning: Offloading weights is deprecated. Using offload_weights=True does not have any effect.
627
+ warnings.warn(
628
+ /usr/local/lib/python3.12/dist-packages/transformer_engine/pytorch/cpu_offload.py:708: DeprecationWarning: Offloading weights is deprecated. Using offload_weights=True does not have any effect.
629
+ warnings.warn(
630
+ /usr/local/lib/python3.12/dist-packages/transformer_engine/pytorch/cpu_offload.py:708: DeprecationWarning: Offloading weights is deprecated. Using offload_weights=True does not have any effect.
631
+ warnings.warn(
632
+ /usr/local/lib/python3.12/dist-packages/transformer_engine/pytorch/cpu_offload.py:708: DeprecationWarning: Offloading weights is deprecated. Using offload_weights=True does not have any effect.
633
+ warnings.warn(
634
+ /usr/local/lib/python3.12/dist-packages/transformer_engine/pytorch/cpu_offload.py:708: DeprecationWarning: Offloading weights is deprecated. Using offload_weights=True does not have any effect.
635
+ warnings.warn(
636
+ /usr/local/lib/python3.12/dist-packages/transformer_engine/pytorch/cpu_offload.py:708: DeprecationWarning: Offloading weights is deprecated. Using offload_weights=True does not have any effect.
637
+ warnings.warn(
638
+ > number of parameters on (tensor, pipeline) model parallel rank (0, 0): 3903983616
639
+ INFO:megatron.core.distributed.distributed_data_parallel:Setting up DistributedDataParallel with config DistributedDataParallelConfig(grad_reduce_in_fp32=True, overlap_grad_reduce=True, overlap_param_gather=True, align_param_gather=False, use_distributed_optimizer=True, num_distributed_optimizer_instances=1, check_for_nan_in_grad=False, check_for_large_grads=False, bucket_size=40000000, pad_buckets_for_high_nccl_busbw=False, average_in_collective=False, fp8_param_gather=False, reuse_grad_buf_for_mxfp8_param_ag=False, use_custom_fsdp=False, data_parallel_sharding_strategy='no_shard', gradient_reduce_div_fusion=True, suggested_communication_unit_size=None, preserve_fp32_weights=True, keep_fp8_transpose_cache_when_using_custom_fsdp=False, nccl_ub=False, fsdp_double_buffer=False)
640
+ INFO:megatron.core.distributed.param_and_grad_buffer:Number of buckets for gradient all-reduce / reduce-scatter: 18
641
+ Params for bucket 1 (206045184 elements, 206045184 padded size):
642
+ module.output_layer.weight
643
+ Params for bucket 2 (42086400 elements, 42086400 padded size):
644
+ module.decoder.final_layernorm.weight
645
+ module.decoder.layers.15.self_attention.linear_qkv.weight
646
+ module.decoder.layers.15.pre_mlp_layernorm.weight
647
+ module.decoder.layers.15.mlp.router.weight
648
+ module.decoder.layers.15.self_attention.linear_qkv.layer_norm_weight
649
+ module.decoder.layers.15.self_attention.linear_proj.weight
650
+ Params for bucket 3 (42082304 elements, 42082304 padded size):
651
+ module.decoder.layers.14.mlp.router.weight
652
+ module.decoder.layers.14.self_attention.linear_qkv.weight
653
+ module.decoder.layers.14.pre_mlp_layernorm.weight
654
+ module.decoder.layers.14.self_attention.linear_proj.weight
655
+ module.decoder.layers.14.self_attention.linear_qkv.layer_norm_weight
656
+ Params for bucket 4 (42082304 elements, 42082304 padded size):
657
+ module.decoder.layers.13.self_attention.linear_qkv.weight
658
+ module.decoder.layers.13.pre_mlp_layernorm.weight
659
+ module.decoder.layers.13.mlp.router.weight
660
+ module.decoder.layers.13.self_attention.linear_qkv.layer_norm_weight
661
+ module.decoder.layers.13.self_attention.linear_proj.weight
662
+ Params for bucket 5 (42082304 elements, 42082304 padded size):
663
+ module.decoder.layers.12.mlp.router.weight
664
+ module.decoder.layers.12.self_attention.linear_qkv.layer_norm_weight
665
+ module.decoder.layers.12.self_attention.linear_qkv.weight
666
+ module.decoder.layers.12.self_attention.linear_proj.weight
667
+ module.decoder.layers.12.pre_mlp_layernorm.weight
668
+ Params for bucket 6 (42082304 elements, 42082304 padded size):
669
+ module.decoder.layers.11.self_attention.linear_qkv.weight
670
+ module.decoder.layers.11.pre_mlp_layernorm.weight
671
+ module.decoder.layers.11.mlp.router.weight
672
+ module.decoder.layers.11.self_attention.linear_proj.weight
673
+ module.decoder.layers.11.self_attention.linear_qkv.layer_norm_weight
674
+ Params for bucket 7 (42082304 elements, 42082304 padded size):
675
+ module.decoder.layers.10.mlp.router.weight
676
+ module.decoder.layers.10.self_attention.linear_qkv.layer_norm_weight
677
+ module.decoder.layers.10.self_attention.linear_proj.weight
678
+ module.decoder.layers.10.self_attention.linear_qkv.weight
679
+ module.decoder.layers.10.pre_mlp_layernorm.weight
680
+ Params for bucket 8 (42082304 elements, 42082304 padded size):
681
+ module.decoder.layers.9.mlp.router.weight
682
+ module.decoder.layers.9.self_attention.linear_qkv.layer_norm_weight
683
+ module.decoder.layers.9.self_attention.linear_proj.weight
684
+ module.decoder.layers.9.self_attention.linear_qkv.weight
685
+ module.decoder.layers.9.pre_mlp_layernorm.weight
686
+ Params for bucket 9 (42082304 elements, 42082304 padded size):
687
+ module.decoder.layers.8.mlp.router.weight
688
+ module.decoder.layers.8.self_attention.linear_qkv.layer_norm_weight
689
+ module.decoder.layers.8.self_attention.linear_proj.weight
690
+ module.decoder.layers.8.pre_mlp_layernorm.weight
691
+ module.decoder.layers.8.self_attention.linear_qkv.weight
692
+ Params for bucket 10 (42082304 elements, 42082304 padded size):
693
+ module.decoder.layers.7.mlp.router.weight
694
+ module.decoder.layers.7.self_attention.linear_qkv.weight
695
+ module.decoder.layers.7.pre_mlp_layernorm.weight
696
+ module.decoder.layers.7.self_attention.linear_proj.weight
697
+ module.decoder.layers.7.self_attention.linear_qkv.layer_norm_weight
698
+ Params for bucket 11 (42082304 elements, 42082304 padded size):
699
+ module.decoder.layers.6.mlp.router.weight
700
+ module.decoder.layers.6.self_attention.linear_proj.weight
701
+ module.decoder.layers.6.self_attention.linear_qkv.layer_norm_weight
702
+ module.decoder.layers.6.pre_mlp_layernorm.weight
703
+ module.decoder.layers.6.self_attention.linear_qkv.weight
704
+ Params for bucket 12 (42082304 elements, 42082304 padded size):
705
+ module.decoder.layers.5.self_attention.linear_proj.weight
706
+ module.decoder.layers.5.pre_mlp_layernorm.weight
707
+ module.decoder.layers.5.self_attention.linear_qkv.weight
708
+ module.decoder.layers.5.mlp.router.weight
709
+ module.decoder.layers.5.self_attention.linear_qkv.layer_norm_weight
710
+ Params for bucket 13 (42082304 elements, 42082304 padded size):
711
+ module.decoder.layers.4.mlp.router.weight
712
+ module.decoder.layers.4.pre_mlp_layernorm.weight
713
+ module.decoder.layers.4.self_attention.linear_qkv.weight
714
+ module.decoder.layers.4.self_attention.linear_proj.weight
715
+ module.decoder.layers.4.self_attention.linear_qkv.layer_norm_weight
716
+ Params for bucket 14 (42082304 elements, 42082304 padded size):
717
+ module.decoder.layers.3.mlp.router.weight
718
+ module.decoder.layers.3.self_attention.linear_proj.weight
719
+ module.decoder.layers.3.self_attention.linear_qkv.layer_norm_weight
720
+ module.decoder.layers.3.pre_mlp_layernorm.weight
721
+ module.decoder.layers.3.self_attention.linear_qkv.weight
722
+ Params for bucket 15 (42082304 elements, 42082304 padded size):
723
+ module.decoder.layers.2.mlp.router.weight
724
+ module.decoder.layers.2.self_attention.linear_proj.weight
725
+ module.decoder.layers.2.self_attention.linear_qkv.layer_norm_weight
726
+ module.decoder.layers.2.self_attention.linear_qkv.weight
727
+ module.decoder.layers.2.pre_mlp_layernorm.weight
728
+ Params for bucket 16 (42082304 elements, 42082304 padded size):
729
+ module.decoder.layers.1.mlp.router.weight
730
+ module.decoder.layers.1.pre_mlp_layernorm.weight
731
+ module.decoder.layers.1.self_attention.linear_qkv.layer_norm_weight
732
+ module.decoder.layers.1.self_attention.linear_qkv.weight
733
+ module.decoder.layers.1.self_attention.linear_proj.weight
734
+ Params for bucket 17 (42082304 elements, 42082304 padded size):
735
+ module.decoder.layers.0.self_attention.linear_proj.weight
736
+ module.decoder.layers.0.pre_mlp_layernorm.weight
737
+ module.decoder.layers.0.mlp.router.weight
738
+ module.decoder.layers.0.self_attention.linear_qkv.layer_norm_weight
739
+ module.decoder.layers.0.self_attention.linear_qkv.weight
740
+ Params for bucket 18 (206045184 elements, 206045184 padded size):
741
+ module.embedding.word_embeddings.weight
742
+ INFO:megatron.core.distributed.param_and_grad_buffer:Number of buckets for gradient all-reduce / reduce-scatter: 56
743
+ Params for bucket 1 (44040192 elements, 44040192 padded size):
744
+ module.decoder.layers.15.mlp.experts.linear_fc2.weight3
745
+ module.decoder.layers.15.mlp.experts.linear_fc2.weight1
746
+ module.decoder.layers.15.mlp.experts.linear_fc2.weight2
747
+ Params for bucket 2 (44040192 elements, 44040192 padded size):
748
+ module.decoder.layers.15.mlp.experts.linear_fc2.weight0
749
+ module.decoder.layers.15.mlp.experts.linear_fc1.weight3
750
+ Params for bucket 3 (58720256 elements, 58720256 padded size):
751
+ module.decoder.layers.15.mlp.experts.linear_fc1.weight2
752
+ module.decoder.layers.15.mlp.experts.linear_fc1.weight1
753
+ Params for bucket 4 (44040192 elements, 44040192 padded size):
754
+ module.decoder.layers.15.mlp.experts.linear_fc1.weight0
755
+ module.decoder.layers.14.mlp.experts.linear_fc2.weight3
756
+ Params for bucket 5 (44040192 elements, 44040192 padded size):
757
+ module.decoder.layers.14.mlp.experts.linear_fc2.weight2
758
+ module.decoder.layers.14.mlp.experts.linear_fc2.weight0
759
+ module.decoder.layers.14.mlp.experts.linear_fc2.weight1
760
+ Params for bucket 6 (58720256 elements, 58720256 padded size):
761
+ module.decoder.layers.14.mlp.experts.linear_fc1.weight3
762
+ module.decoder.layers.14.mlp.experts.linear_fc1.weight2
763
+ Params for bucket 7 (58720256 elements, 58720256 padded size):
764
+ module.decoder.layers.14.mlp.experts.linear_fc1.weight1
765
+ module.decoder.layers.14.mlp.experts.linear_fc1.weight0
766
+ Params for bucket 8 (44040192 elements, 44040192 padded size):
767
+ module.decoder.layers.13.mlp.experts.linear_fc2.weight3
768
+ module.decoder.layers.13.mlp.experts.linear_fc2.weight1
769
+ module.decoder.layers.13.mlp.experts.linear_fc2.weight2
770
+ Params for bucket 9 (44040192 elements, 44040192 padded size):
771
+ module.decoder.layers.13.mlp.experts.linear_fc2.weight0
772
+ module.decoder.layers.13.mlp.experts.linear_fc1.weight3
773
+ Params for bucket 10 (58720256 elements, 58720256 padded size):
774
+ module.decoder.layers.13.mlp.experts.linear_fc1.weight2
775
+ module.decoder.layers.13.mlp.experts.linear_fc1.weight1
776
+ Params for bucket 11 (44040192 elements, 44040192 padded size):
777
+ module.decoder.layers.13.mlp.experts.linear_fc1.weight0
778
+ module.decoder.layers.12.mlp.experts.linear_fc2.weight3
779
+ Params for bucket 12 (44040192 elements, 44040192 padded size):
780
+ module.decoder.layers.12.mlp.experts.linear_fc2.weight2
781
+ module.decoder.layers.12.mlp.experts.linear_fc2.weight0
782
+ module.decoder.layers.12.mlp.experts.linear_fc2.weight1
783
+ Params for bucket 13 (58720256 elements, 58720256 padded size):
784
+ module.decoder.layers.12.mlp.experts.linear_fc1.weight3
785
+ module.decoder.layers.12.mlp.experts.linear_fc1.weight2
786
+ Params for bucket 14 (58720256 elements, 58720256 padded size):
787
+ module.decoder.layers.12.mlp.experts.linear_fc1.weight1
788
+ module.decoder.layers.12.mlp.experts.linear_fc1.weight0
789
+ Params for bucket 15 (44040192 elements, 44040192 padded size):
790
+ module.decoder.layers.11.mlp.experts.linear_fc2.weight3
791
+ module.decoder.layers.11.mlp.experts.linear_fc2.weight1
792
+ module.decoder.layers.11.mlp.experts.linear_fc2.weight2
793
+ Params for bucket 16 (44040192 elements, 44040192 padded size):
794
+ module.decoder.layers.11.mlp.experts.linear_fc2.weight0
795
+ module.decoder.layers.11.mlp.experts.linear_fc1.weight3
796
+ Params for bucket 17 (58720256 elements, 58720256 padded size):
797
+ module.decoder.layers.11.mlp.experts.linear_fc1.weight2
798
+ module.decoder.layers.11.mlp.experts.linear_fc1.weight1
799
+ Params for bucket 18 (44040192 elements, 44040192 padded size):
800
+ module.decoder.layers.11.mlp.experts.linear_fc1.weight0
801
+ module.decoder.layers.10.mlp.experts.linear_fc2.weight3
802
+ Params for bucket 19 (44040192 elements, 44040192 padded size):
803
+ module.decoder.layers.10.mlp.experts.linear_fc2.weight2
804
+ module.decoder.layers.10.mlp.experts.linear_fc2.weight0
805
+ module.decoder.layers.10.mlp.experts.linear_fc2.weight1
806
+ Params for bucket 20 (58720256 elements, 58720256 padded size):
807
+ module.decoder.layers.10.mlp.experts.linear_fc1.weight3
808
+ module.decoder.layers.10.mlp.experts.linear_fc1.weight2
809
+ Params for bucket 21 (58720256 elements, 58720256 padded size):
810
+ module.decoder.layers.10.mlp.experts.linear_fc1.weight1
811
+ module.decoder.layers.10.mlp.experts.linear_fc1.weight0
812
+ Params for bucket 22 (44040192 elements, 44040192 padded size):
813
+ module.decoder.layers.9.mlp.experts.linear_fc2.weight3
814
+ module.decoder.layers.9.mlp.experts.linear_fc2.weight1
815
+ module.decoder.layers.9.mlp.experts.linear_fc2.weight2
816
+ Params for bucket 23 (44040192 elements, 44040192 padded size):
817
+ module.decoder.layers.9.mlp.experts.linear_fc2.weight0
818
+ module.decoder.layers.9.mlp.experts.linear_fc1.weight3
819
+ Params for bucket 24 (58720256 elements, 58720256 padded size):
820
+ module.decoder.layers.9.mlp.experts.linear_fc1.weight2
821
+ module.decoder.layers.9.mlp.experts.linear_fc1.weight1
822
+ Params for bucket 25 (44040192 elements, 44040192 padded size):
823
+ module.decoder.layers.9.mlp.experts.linear_fc1.weight0
824
+ module.decoder.layers.8.mlp.experts.linear_fc2.weight3
825
+ Params for bucket 26 (44040192 elements, 44040192 padded size):
826
+ module.decoder.layers.8.mlp.experts.linear_fc2.weight2
827
+ module.decoder.layers.8.mlp.experts.linear_fc2.weight0
828
+ module.decoder.layers.8.mlp.experts.linear_fc2.weight1
829
+ Params for bucket 27 (58720256 elements, 58720256 padded size):
830
+ module.decoder.layers.8.mlp.experts.linear_fc1.weight3
831
+ module.decoder.layers.8.mlp.experts.linear_fc1.weight2
832
+ Params for bucket 28 (58720256 elements, 58720256 padded size):
833
+ module.decoder.layers.8.mlp.experts.linear_fc1.weight1
834
+ module.decoder.layers.8.mlp.experts.linear_fc1.weight0
835
+ Params for bucket 29 (44040192 elements, 44040192 padded size):
836
+ module.decoder.layers.7.mlp.experts.linear_fc2.weight3
837
+ module.decoder.layers.7.mlp.experts.linear_fc2.weight2
838
+ module.decoder.layers.7.mlp.experts.linear_fc2.weight1
839
+ Params for bucket 30 (44040192 elements, 44040192 padded size):
840
+ module.decoder.layers.7.mlp.experts.linear_fc2.weight0
841
+ module.decoder.layers.7.mlp.experts.linear_fc1.weight3
842
+ Params for bucket 31 (58720256 elements, 58720256 padded size):
843
+ module.decoder.layers.7.mlp.experts.linear_fc1.weight2
844
+ module.decoder.layers.7.mlp.experts.linear_fc1.weight1
845
+ Params for bucket 32 (44040192 elements, 44040192 padded size):
846
+ module.decoder.layers.7.mlp.experts.linear_fc1.weight0
847
+ module.decoder.layers.6.mlp.experts.linear_fc2.weight3
848
+ Params for bucket 33 (44040192 elements, 44040192 padded size):
849
+ module.decoder.layers.6.mlp.experts.linear_fc2.weight2
850
+ module.decoder.layers.6.mlp.experts.linear_fc2.weight1
851
+ module.decoder.layers.6.mlp.experts.linear_fc2.weight0
852
+ Params for bucket 34 (58720256 elements, 58720256 padded size):
853
+ module.decoder.layers.6.mlp.experts.linear_fc1.weight3
854
+ module.decoder.layers.6.mlp.experts.linear_fc1.weight2
855
+ Params for bucket 35 (58720256 elements, 58720256 padded size):
856
+ module.decoder.layers.6.mlp.experts.linear_fc1.weight1
857
+ module.decoder.layers.6.mlp.experts.linear_fc1.weight0
858
+ Params for bucket 36 (44040192 elements, 44040192 padded size):
859
+ module.decoder.layers.5.mlp.experts.linear_fc2.weight3
860
+ module.decoder.layers.5.mlp.experts.linear_fc2.weight2
861
+ module.decoder.layers.5.mlp.experts.linear_fc2.weight1
862
+ Params for bucket 37 (44040192 elements, 44040192 padded size):
863
+ module.decoder.layers.5.mlp.experts.linear_fc2.weight0
864
+ module.decoder.layers.5.mlp.experts.linear_fc1.weight3
865
+ Params for bucket 38 (58720256 elements, 58720256 padded size):
866
+ module.decoder.layers.5.mlp.experts.linear_fc1.weight2
867
+ module.decoder.layers.5.mlp.experts.linear_fc1.weight1
868
+ Params for bucket 39 (44040192 elements, 44040192 padded size):
869
+ module.decoder.layers.5.mlp.experts.linear_fc1.weight0
870
+ module.decoder.layers.4.mlp.experts.linear_fc2.weight3
871
+ Params for bucket 40 (44040192 elements, 44040192 padded size):
872
+ module.decoder.layers.4.mlp.experts.linear_fc2.weight2
873
+ module.decoder.layers.4.mlp.experts.linear_fc2.weight0
874
+ module.decoder.layers.4.mlp.experts.linear_fc2.weight1
875
+ Params for bucket 41 (58720256 elements, 58720256 padded size):
876
+ module.decoder.layers.4.mlp.experts.linear_fc1.weight3
877
+ module.decoder.layers.4.mlp.experts.linear_fc1.weight2
878
+ Params for bucket 42 (58720256 elements, 58720256 padded size):
879
+ module.decoder.layers.4.mlp.experts.linear_fc1.weight1
880
+ module.decoder.layers.4.mlp.experts.linear_fc1.weight0
881
+ Params for bucket 43 (44040192 elements, 44040192 padded size):
882
+ module.decoder.layers.3.mlp.experts.linear_fc2.weight3
883
+ module.decoder.layers.3.mlp.experts.linear_fc2.weight2
884
+ module.decoder.layers.3.mlp.experts.linear_fc2.weight1
885
+ Params for bucket 44 (44040192 elements, 44040192 padded size):
886
+ module.decoder.layers.3.mlp.experts.linear_fc2.weight0
887
+ module.decoder.layers.3.mlp.experts.linear_fc1.weight3
888
+ Params for bucket 45 (58720256 elements, 58720256 padded size):
889
+ module.decoder.layers.3.mlp.experts.linear_fc1.weight2
890
+ module.decoder.layers.3.mlp.experts.linear_fc1.weight1
891
+ Params for bucket 46 (44040192 elements, 44040192 padded size):
892
+ module.decoder.layers.3.mlp.experts.linear_fc1.weight0
893
+ module.decoder.layers.2.mlp.experts.linear_fc2.weight3
894
+ Params for bucket 47 (44040192 elements, 44040192 padded size):
895
+ module.decoder.layers.2.mlp.experts.linear_fc2.weight2
896
+ module.decoder.layers.2.mlp.experts.linear_fc2.weight1
897
+ module.decoder.layers.2.mlp.experts.linear_fc2.weight0
898
+ Params for bucket 48 (58720256 elements, 58720256 padded size):
899
+ module.decoder.layers.2.mlp.experts.linear_fc1.weight3
900
+ module.decoder.layers.2.mlp.experts.linear_fc1.weight2
901
+ Params for bucket 49 (58720256 elements, 58720256 padded size):
902
+ module.decoder.layers.2.mlp.experts.linear_fc1.weight1
903
+ module.decoder.layers.2.mlp.experts.linear_fc1.weight0
904
+ Params for bucket 50 (44040192 elements, 44040192 padded size):
905
+ module.decoder.layers.1.mlp.experts.linear_fc2.weight3
906
+ module.decoder.layers.1.mlp.experts.linear_fc2.weight1
907
+ module.decoder.layers.1.mlp.experts.linear_fc2.weight2
908
+ Params for bucket 51 (44040192 elements, 44040192 padded size):
909
+ module.decoder.layers.1.mlp.experts.linear_fc2.weight0
910
+ module.decoder.layers.1.mlp.experts.linear_fc1.weight3
911
+ Params for bucket 52 (58720256 elements, 58720256 padded size):
912
+ module.decoder.layers.1.mlp.experts.linear_fc1.weight2
913
+ module.decoder.layers.1.mlp.experts.linear_fc1.weight1
914
+ Params for bucket 53 (44040192 elements, 44040192 padded size):
915
+ module.decoder.layers.1.mlp.experts.linear_fc1.weight0
916
+ module.decoder.layers.0.mlp.experts.linear_fc2.weight3
917
+ Params for bucket 54 (44040192 elements, 44040192 padded size):
918
+ module.decoder.layers.0.mlp.experts.linear_fc2.weight2
919
+ module.decoder.layers.0.mlp.experts.linear_fc2.weight1
920
+ module.decoder.layers.0.mlp.experts.linear_fc2.weight0
921
+ Params for bucket 55 (58720256 elements, 58720256 padded size):
922
+ module.decoder.layers.0.mlp.experts.linear_fc1.weight3
923
+ module.decoder.layers.0.mlp.experts.linear_fc1.weight2
924
+ Params for bucket 56 (58720256 elements, 58720256 padded size):
925
+ module.decoder.layers.0.mlp.experts.linear_fc1.weight1
926
+ module.decoder.layers.0.mlp.experts.linear_fc1.weight0
927
+ INFO:megatron.core.optimizer:Setting up optimizer with config OptimizerConfig(optimizer='adam', lr=0.0001, min_lr=1e-05, decoupled_lr=None, decoupled_min_lr=None, weight_decay=0.1, fp8_recipe='mxfp8', fp16=False, bf16=True, reuse_grad_buf_for_mxfp8_param_ag=False, params_dtype=torch.bfloat16, use_precision_aware_optimizer=False, store_param_remainders=True, main_grads_dtype=torch.float32, main_params_dtype=torch.float32, exp_avg_dtype=torch.float32, exp_avg_sq_dtype=torch.float32, loss_scale=None, initial_loss_scale=4294967296, min_loss_scale=1.0, loss_scale_window=1000, hysteresis=2, adam_beta1=0.9, adam_beta2=0.999, adam_eps=1e-08, sgd_momentum=0.9, use_distributed_optimizer=True, overlap_param_gather_with_optimizer_step=False, optimizer_cpu_offload=False, optimizer_offload_fraction=1.0, use_torch_optimizer_for_cpu_offload=False, overlap_cpu_optimizer_d2h_h2d=False, pin_cpu_grads=True, pin_cpu_params=True, clip_grad=1.0, log_num_zeros_in_grad=False, barrier_with_L1_time=True, timers=<megatron.core.timers.Timers object at 0x77f7a359c710>, config_logger_dir='')
928
+ INFO:megatron.core.optimizer_param_scheduler:> learning rate decay style: cosine
929
+ [after model, optimizer, and learning rate scheduler are built] datetime: 2026-08-30 06:50:05
930
+ > building train, validation, and test datasets ...
931
+ > datasets target sizes (minimum size):
932
+ train: 192
933
+ validation: 0
934
+ test: 0
935
+ INFO:megatron.core.datasets.blended_megatron_dataset_config:Let split_matrix = [(0, 1.0), None, None]
936
+ > building train, validation, and test datasets for GPT ...
937
+ INFO:megatron.core.datasets.blended_megatron_dataset_builder:Building GPTDataset splits with sizes=(192, 0, 0) and config=GPTDatasetConfig(random_seed=1234, sequence_length=4096, blend=(['/workspace/data/wikitext/wikitext103_gpt2_text_document'], None), blend_per_split=None, split='100,0,0', split_matrix=[(0, 1.0), None, None], num_dataset_builder_threads=1, path_to_cache=None, mmap_bin_files=True, mock=False, tokenizer=<megatron.training.tokenizer.tokenizer._GPT2BPETokenizer object at 0x77f7a4ef54c0>, mid_level_dataset_surplus=0.005, reset_position_ids=False, reset_attention_mask=False, eod_mask_loss=False, create_attention_mask=True, drop_last_partial_validation_sequence=True, add_extra_token_to_sequence=True, object_storage_cache_path=None)
938
+ INFO:megatron.core.datasets.indexed_dataset:Load the _IndexReader from /workspace/data/wikitext/wikitext103_gpt2_text_document.idx
939
+ INFO:megatron.core.datasets.indexed_dataset: Extract the sequence lengths
940
+ INFO:megatron.core.datasets.indexed_dataset: Extract the sequence pointers
941
+ INFO:megatron.core.datasets.indexed_dataset: Extract the document indices
942
+ INFO:megatron.core.datasets.indexed_dataset:> total number of sequences: 1165029
943
+ INFO:megatron.core.datasets.indexed_dataset:> total number of documents: 1801350
944
+ INFO:megatron.core.datasets.gpt_dataset:Load the GPTDataset train indices
945
+ INFO:megatron.core.datasets.gpt_dataset: Load the document index from 44ed1ee5bcb9e5b030387b8d27a6c406-GPTDataset-train-document_index.npy
946
+ INFO:megatron.core.datasets.gpt_dataset: Load the sample index from 44ed1ee5bcb9e5b030387b8d27a6c406-GPTDataset-train-sample_index.npy
947
+ INFO:megatron.core.datasets.gpt_dataset: Load the shuffle index from 44ed1ee5bcb9e5b030387b8d27a6c406-GPTDataset-train-shuffle_index.npy
948
+ INFO:megatron.core.datasets.gpt_dataset:> total number of samples: 29073
949
+ > finished creating GPT datasets ...
950
+ [after dataloaders are built] datetime: 2026-08-30 06:50:05
951
+ done with setup ...
952
+ training ...
953
+ Overwriting rerun_state_machine.current_iteration from -1 to 0...
954
+ (min, max) time across ranks (ms):
955
+ model-and-optimizer-setup ......................: (325.55, 328.11)
956
+ train/valid/test-data-iterators-setup ..........: (64.46, 68.67)
957
+ [before the start of training step] datetime: 2026-08-30 06:50:05
958
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:334: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
959
+ warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
960
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:334: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
961
+ warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
962
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:336: UserWarning: The use_host_meta is deprecated, it will be removed in the future.
963
+ warnings.warn("The use_host_meta is deprecated, it will be removed in the future.")
964
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:336: UserWarning: The use_host_meta is deprecated, it will be removed in the future.
965
+ warnings.warn("The use_host_meta is deprecated, it will be removed in the future.")
966
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:334: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
967
+ warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
968
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:336: UserWarning: The use_host_meta is deprecated, it will be removed in the future.
969
+ warnings.warn("The use_host_meta is deprecated, it will be removed in the future.")
970
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:334: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
971
+ warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
972
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:334: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
973
+ warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
974
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:336: UserWarning: The use_host_meta is deprecated, it will be removed in the future.
975
+ warnings.warn("The use_host_meta is deprecated, it will be removed in the future.")
976
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:336: UserWarning: The use_host_meta is deprecated, it will be removed in the future.
977
+ warnings.warn("The use_host_meta is deprecated, it will be removed in the future.")
978
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:334: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
979
+ warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
980
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:334: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
981
+ warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
982
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:336: UserWarning: The use_host_meta is deprecated, it will be removed in the future.
983
+ warnings.warn("The use_host_meta is deprecated, it will be removed in the future.")
984
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:336: UserWarning: The use_host_meta is deprecated, it will be removed in the future.
985
+ warnings.warn("The use_host_meta is deprecated, it will be removed in the future.")
986
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:334: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
987
+ warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
988
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:336: UserWarning: The use_host_meta is deprecated, it will be removed in the future.
989
+ warnings.warn("The use_host_meta is deprecated, it will be removed in the future.")
990
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:482: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
991
+ warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
992
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:482: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
993
+ warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
994
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:482: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
995
+ warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
996
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:482: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
997
+ warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
998
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:482: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
999
+ warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
1000
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:482: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
1001
+ warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
1002
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:482: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
1003
+ warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
1004
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:482: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
1005
+ warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
1006
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:334: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
1007
+ warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
1008
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:336: UserWarning: The use_host_meta is deprecated, it will be removed in the future.
1009
+ warnings.warn("The use_host_meta is deprecated, it will be removed in the future.")
1010
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:334: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
1011
+ warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
1012
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:336: UserWarning: The use_host_meta is deprecated, it will be removed in the future.
1013
+ warnings.warn("The use_host_meta is deprecated, it will be removed in the future.")
1014
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:334: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
1015
+ warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
1016
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:336: UserWarning: The use_host_meta is deprecated, it will be removed in the future.
1017
+ warnings.warn("The use_host_meta is deprecated, it will be removed in the future.")
1018
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:334: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
1019
+ warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
1020
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:336: UserWarning: The use_host_meta is deprecated, it will be removed in the future.
1021
+ warnings.warn("The use_host_meta is deprecated, it will be removed in the future.")
1022
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:334: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
1023
+ warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
1024
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:336: UserWarning: The use_host_meta is deprecated, it will be removed in the future.
1025
+ warnings.warn("The use_host_meta is deprecated, it will be removed in the future.")
1026
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:334: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
1027
+ warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
1028
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:336: UserWarning: The use_host_meta is deprecated, it will be removed in the future.
1029
+ warnings.warn("The use_host_meta is deprecated, it will be removed in the future.")
1030
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:334: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
1031
+ warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
1032
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:336: UserWarning: The use_host_meta is deprecated, it will be removed in the future.
1033
+ warnings.warn("The use_host_meta is deprecated, it will be removed in the future.")
1034
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:334: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
1035
+ warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
1036
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:336: UserWarning: The use_host_meta is deprecated, it will be removed in the future.
1037
+ warnings.warn("The use_host_meta is deprecated, it will be removed in the future.")
1038
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:482: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
1039
+ warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
1040
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:482: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
1041
+ warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
1042
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:482: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
1043
+ warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
1044
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:482: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
1045
+ warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
1046
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:482: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
1047
+ warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
1048
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:482: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
1049
+ warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
1050
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:482: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
1051
+ warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
1052
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:482: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
1053
+ warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
1054
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:334: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
1055
+ warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
1056
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:334: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
1057
+ warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
1058
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:334: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
1059
+ warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
1060
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:334: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
1061
+ warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
1062
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:334: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
1063
+ warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
1064
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:334: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
1065
+ warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
1066
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:334: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
1067
+ warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
1068
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:334: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
1069
+ warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
1070
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:336: UserWarning: The use_host_meta is deprecated, it will be removed in the future.
1071
+ warnings.warn("The use_host_meta is deprecated, it will be removed in the future.")
1072
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:336: UserWarning: The use_host_meta is deprecated, it will be removed in the future.
1073
+ warnings.warn("The use_host_meta is deprecated, it will be removed in the future.")
1074
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:336: UserWarning: The use_host_meta is deprecated, it will be removed in the future.
1075
+ warnings.warn("The use_host_meta is deprecated, it will be removed in the future.")
1076
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:336: UserWarning: The use_host_meta is deprecated, it will be removed in the future.
1077
+ warnings.warn("The use_host_meta is deprecated, it will be removed in the future.")
1078
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:336: UserWarning: The use_host_meta is deprecated, it will be removed in the future.
1079
+ warnings.warn("The use_host_meta is deprecated, it will be removed in the future.")
1080
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:336: UserWarning: The use_host_meta is deprecated, it will be removed in the future.
1081
+ warnings.warn("The use_host_meta is deprecated, it will be removed in the future.")
1082
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:336: UserWarning: The use_host_meta is deprecated, it will be removed in the future.
1083
+ warnings.warn("The use_host_meta is deprecated, it will be removed in the future.")
1084
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:336: UserWarning: The use_host_meta is deprecated, it will be removed in the future.
1085
+ warnings.warn("The use_host_meta is deprecated, it will be removed in the future.")
tiera-rerun-fixed-moonep/echo/a2/a2_matched_gbs16/results.tsv ADDED
@@ -0,0 +1,5 @@
 
 
 
 
 
 
1
+ config backend policy slots seq gbs cap recompute rep status iter_ms_mean iter_ms_min peak_mib run_dir
2
+ G1 echo matched 1 4096 16 2.40625 none 1 OK 466.86 462.10 161723 /workspace/tiera-rerun-fixed-moonep/echo/a2/a2_matched_gbs16/G1__echo__matched__seq4096__gbs16__cap2p40625/rep1
3
+ G1 echo matched 1 4096 16 2.40625 none 2 OK 466.67 462.10 161723 /workspace/tiera-rerun-fixed-moonep/echo/a2/a2_matched_gbs16/G1__echo__matched__seq4096__gbs16__cap2p40625/rep2
4
+ G1 echo matched 1 4096 16 2.40625 none 3 OK 466.91 461.80 161723 /workspace/tiera-rerun-fixed-moonep/echo/a2/a2_matched_gbs16/G1__echo__matched__seq4096__gbs16__cap2p40625/rep3
5
+ G2 echo matched 4 4096 16 1.08203125 none 1 OK 415.60 413.70 145689 /workspace/tiera-rerun-fixed-moonep/echo/a2/a2_matched_gbs16/G2__echo__matched__seq4096__gbs16__cap1p08203125/rep1
tiera-rerun-fixed-moonep/echo/a2/manifest.txt ADDED
@@ -0,0 +1,19 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ timestamp: 2026-08-30T06:44:28+00:00
2
+ host: sp-aiinfra-interactive-aghosh-7jtjl-8xb200-moe-cg-0
3
+ gpu: NVIDIA B200 x8
4
+ driver: 580.95.05
5
+ study_scripts_branch: granularity-echo
6
+ study_scripts_commit: ff2c833
7
+ campaign: echo
8
+ backend: echo
9
+ phase: a2
10
+ gbs: 16
11
+ seq: 4096
12
+ configs: G1 G2 G3 G4
13
+ capacity: 2.40625
14
+ repeats: 3
15
+ nsys: 0
16
+ megatron_repo: /workspace/Megatron-MoE-Echo
17
+ megatron_branch_required: bracing-regions-through-kernels
18
+ megatron_branch_actual: bracing-regions-through-kernels
19
+ megatron_commit: 9a236793
tiera-rerun-fixed-moonep/echo/a2/pip_freeze.txt ADDED
@@ -0,0 +1,317 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ absl-py==2.3.0
2
+ aiohappyeyeballs @ file:///rapids/aiohappyeyeballs-2.6.1-py3-none-any.whl#sha256=f349ba8f4b75cb25c99c5c2d84e997e485204d2902a9597802b0371f09331fb8
3
+ aiohttp @ file:///rapids/aiohttp-3.12.7-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl#sha256=94f98e0e5a49f89b252e115844f756c04fc8050f38252a32a3dd994ce8121f10
4
+ aiosignal @ file:///rapids/aiosignal-1.3.2-py2.py3-none-any.whl#sha256=45cde58e409a301715980c2b01d0c28bdde3770d8290b5eb2173759d9acb31a5
5
+ annotated-types==0.7.0
6
+ anyio==4.9.0
7
+ apex @ file:///opt/pytorch/apex/dist/apex-0.1-cp312-cp312-linux_x86_64.whl#sha256=a1508e4a386362e9c8f46c92ccada9110e421212b964c694235d0fd5a224b170
8
+ argon2-cffi==25.1.0
9
+ argon2-cffi-bindings==21.2.0
10
+ arrow==1.3.0
11
+ asciitree @ file:///rapids/asciitree-0.3.3-py3-none-any.whl#sha256=3e507ccfaeb417823158091fdb9d26b0205f6a59ba680b479a71db65c73d01d4
12
+ asttokens==3.0.0
13
+ astunparse==1.6.3
14
+ async-lru==2.0.5
15
+ attrs==25.3.0
16
+ audioread==3.0.1
17
+ babel==2.17.0
18
+ beautifulsoup4==4.13.4
19
+ black==25.1.0
20
+ bleach==6.2.0
21
+ blis==0.7.11
22
+ cachetools @ file:///rapids/cachetools-6.0.0-py3-none-any.whl#sha256=82e73ba88f7b30228b5507dce1a1f878498fc669d972aef2dde4f3a3c24f103e
23
+ catalogue==2.0.10
24
+ certifi==2025.4.26
25
+ cffi==1.17.1
26
+ charset-normalizer==3.4.2
27
+ click==8.5.0
28
+ cloudpathlib==0.21.1
29
+ cloudpickle @ file:///rapids/cloudpickle-3.1.1-py3-none-any.whl#sha256=c8c5a44295039331ee9dad40ba100a9c7297b6f988e50e87ccdf3765a668350e
30
+ cmake==3.31.6
31
+ comm==0.2.2
32
+ confection==0.1.5
33
+ contourpy==1.3.2
34
+ cuda-bindings @ file:///rapids/cuda_bindings-12.9.0-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl#sha256=ff0e28d1e34758654b9c961e1f55e4786e49aee6a4dbceaf3cc24c46c672df7e
35
+ cuda-python @ file:///rapids/cuda_python-12.9.0-py3-none-any.whl#sha256=926acba49b2c0a0374c61b7c98f337c085199cf51cdfe4d6423c4129c20547a7
36
+ cudf @ file:///rapids/cudf-25.4.0-cp312-cp312-linux_x86_64.whl#sha256=b779a8e8c0e8577b9b560895958496da211bfac49cdc4f11a49e8d18cb82e348
37
+ cudf-polars @ file:///rapids/cudf_polars-25.4.0-py3-none-any.whl#sha256=227b6c5147e93ea86282c8f7219118f916769fa2ca0cdd931e97a148aa66b869
38
+ cugraph @ file:///rapids/cugraph-25.4.0-cp312-cp312-linux_x86_64.whl#sha256=b837c2dc2f7b235c702e4f8df64479b8fcbb342f848516d5faf0cdc4b5f31f29
39
+ cugraph-service-client @ file:///rapids/cugraph_service_client-25.4.0-py3-none-any.whl#sha256=7e568c3e41154b9752f5173555ffcc282ac52d72343feeb79653b2e2b1232470
40
+ cugraph-service-server @ file:///rapids/cugraph_service_server-25.4.0-py3-none-any.whl#sha256=a2830dcffafcccb16e12ebf85b98f8f247869d38c01ab4dd8664a1e7439d31be
41
+ cuml @ file:///rapids/cuml-25.4.0-cp312-cp312-linux_x86_64.whl#sha256=1e90a464a18b094e24797f1676688c5d7696c4cf97c934504214a53d5aa09aa5
42
+ cupy-cuda12x @ file:///rapids/cupy_cuda12x-13.3.0-cp312-cp312-linux_x86_64.whl#sha256=6a2dfa0aa1e8189a10d1534276c08f1d2f73b747fd28bfbe7b9f4ef66d9af73f
43
+ cuvs @ file:///rapids/cuvs-25.4.0-cp312-cp312-linux_x86_64.whl#sha256=b12f6b9f80e053a3b38542c2947d2516e468c02cad54a13d8b1a4a61d5aa7914
44
+ cycler==0.12.1
45
+ cymem==2.0.11
46
+ Cython==3.1.1
47
+ dask @ file:///rapids/dask-2025.2.0-py3-none-any.whl#sha256=f0fdeef6ceb0a06569d456c9e704f220f7f54e80f3a6ea42ab98cea6bc642b6e
48
+ dask-cuda @ file:///rapids/dask_cuda-25.4.0-py3-none-any.whl#sha256=442d87da0a95d00ca2a262b64c76e785a32ad905c82000fa2227932f81390e19
49
+ dask-cudf @ file:///rapids/dask_cudf-25.4.0-py3-none-any.whl#sha256=5e7cea0b26daea17da6e0bb457435e24460f76ba740358a3a18e239d6015c95c
50
+ datasets==5.0.1
51
+ debugpy==1.8.14
52
+ decorator==5.2.1
53
+ -e git+https://github.com/deepseek-ai/DeepEP.git@34152ae28f80bcc3ee38d7a12cb2ad87cfd4ea72#egg=deep_ep
54
+ defusedxml==0.7.1
55
+ dill==0.4.1
56
+ distributed @ file:///rapids/distributed-2025.2.0-py3-none-any.whl#sha256=368462084be47092d0a835f6c64c9b852da64dc88302b76d5f93bd5131228c67
57
+ distributed-ucxx @ file:///rapids/distributed_ucxx-0.43.0-py3-none-any.whl#sha256=94f60d0ccb0c94ba8dbfa1eaf455874d18aaab17e2ae218c38590e9fdc2d721d
58
+ dm-tree==0.1.9
59
+ einops==0.8.1
60
+ execnet==2.1.1
61
+ executing==2.2.0
62
+ expecttest==0.3.0
63
+ fasteners @ file:///rapids/fasteners-0.19-py3-none-any.whl#sha256=758819cb5d94cdedf4e836988b74de396ceacb8e2794d21f82d131fd9ee77237
64
+ fastjsonschema==2.21.1
65
+ fastrlock @ file:///rapids/fastrlock-0.8.3-cp312-cp312-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_28_x86_64.whl#sha256=5f13ec08f1adb1aa916c384b05ecb7dbebb8df9ea81abd045f60941c6283a670
66
+ filelock==3.18.0
67
+ flash_attn @ file:///opt/pytorch/flash_attn-2.7.4.post1-cp312-cp312-linux_x86_64.whl#sha256=a10455d6c56b387785b7413fd0bbcb392b42b3d4f4b88fda5d86cc8a9ce9182a
68
+ fonttools==4.58.1
69
+ fqdn==1.5.1
70
+ frozenlist @ file:///rapids/frozenlist-1.6.0-cp312-cp312-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl#sha256=4e1be9111cb6756868ac242b3c2bd1f09d9aea09846e4f5c23715e7afb647103
71
+ fsspec==2025.5.1
72
+ gast==0.6.0
73
+ grpcio==1.62.1
74
+ h11==0.16.0
75
+ hf-xet==1.6.0
76
+ httpcore==1.0.9
77
+ httpx==0.28.1
78
+ huggingface_hub==1.29.0
79
+ hypothesis==6.130.8
80
+ idna==3.10
81
+ importlib_metadata==8.7.0
82
+ iniconfig==2.1.0
83
+ intel-openmp==2021.4.0
84
+ ipykernel==6.29.5
85
+ ipython==9.3.0
86
+ ipython_pygments_lexers==1.1.1
87
+ isoduration==20.11.0
88
+ isort==6.0.1
89
+ jedi==0.19.2
90
+ Jinja2==3.1.6
91
+ joblib==1.5.1
92
+ json5==0.12.0
93
+ jsonpointer==3.0.0
94
+ jsonschema==4.24.0
95
+ jsonschema-specifications==2025.4.1
96
+ jupyter-events==0.12.0
97
+ jupyter-lsp==2.2.5
98
+ jupyter_client==8.6.3
99
+ jupyter_core==5.8.1
100
+ jupyter_server==2.16.0
101
+ jupyter_server_terminals==0.5.3
102
+ jupyterlab==4.4.3
103
+ jupyterlab_code_formatter==3.0.2
104
+ jupyterlab_pygments==0.3.0
105
+ jupyterlab_server==2.27.3
106
+ jupyterlab_tensorboard_pro @ file:///builder/jupyterlab_tensorboard_pro-4.0.0-py2.py3-none-any.whl#sha256=6bfcb5ced3358eb72150f4b2ab91312faef2f0ca14d845e8379d8fa52df6902e
107
+ jupytext==1.17.2
108
+ kiwisolver==1.4.8
109
+ kvikio @ file:///rapids/kvikio-25.4.0-cp312-cp312-linux_x86_64.whl#sha256=8b40fd8ea9e4a2974c04b50b7fc26b06094ac807f9a233281d0bafbaa71b1d85
110
+ langcodes==3.5.0
111
+ language_data==1.3.0
112
+ lazy_loader==0.4
113
+ libcudf @ file:///rapids/libcudf-25.4.0-py3-none-linux_x86_64.whl#sha256=238320b4958c1737bba2a609673e2539a78216b811b5498a68e00395d9b021a4
114
+ libcugraph @ file:///rapids/libcugraph-25.4.0-py3-none-linux_x86_64.whl#sha256=29085e901b2f7569ff2653bb1355fe6fad20f2fcaa585f5649da9867d8df766d
115
+ libcuml @ file:///rapids/libcuml-25.4.0-py3-none-linux_x86_64.whl#sha256=8e2e444c72b0e995e15a48da37ff12fb410a47a2baa69ba86ccc9b1a3db91e8c
116
+ libcuvs @ file:///rapids/libcuvs-25.4.0-py3-none-linux_x86_64.whl#sha256=e2f8db61c92b769c590dd752f43439ddd62b2f6592a3c337362b6c1ef220c3b9
117
+ libkvikio @ file:///rapids/libkvikio-25.4.0-py3-none-linux_x86_64.whl#sha256=da8a5713b05fc478bf40529a3e0fb61a2d605bb5a70ca4d461f96a6005dbe8b7
118
+ libraft @ file:///rapids/libraft-25.4.0-py3-none-linux_x86_64.whl#sha256=b921dec005aa4a321dd156776a18376feb47c96a5b67eec125c8a8e19edd1e24
119
+ librmm @ file:///rapids/librmm-25.4.0-py3-none-linux_x86_64.whl#sha256=ff951811bda729d4fa1109ab906225ae1f489a6ec2eeab5d1b674cc34bd4422f
120
+ librmm-cu12 @ file:///rapids/librmm_cu12-25.4.0-py3-none-any.whl#sha256=29296ef2b0e90679a0db280e86015be12d6ab9f4e62a181aa085f62dbadd06b3
121
+ librosa==0.11.0
122
+ libucxx @ file:///rapids/libucxx-0.43.0-py3-none-linux_x86_64.whl#sha256=f15849d7ac2fb68f0fd8998838a13321ddcfb0c594f7e6cbc401a3bcc4750566
123
+ lightning-thunder @ file:///opt/pytorch/lightning-thunder
124
+ lightning-utilities==0.14.3
125
+ lintrunner==0.12.7
126
+ llvmlite==0.42.0
127
+ locket @ file:///rapids/locket-1.0.0-py2.py3-none-any.whl#sha256=b6c819a722f7b6bd955b80781788e4a66a55628b858d347536b7e81325a3a5e3
128
+ looseversion==1.3.0
129
+ marisa-trie==1.2.1
130
+ Markdown==3.8
131
+ markdown-it-py==3.0.0
132
+ MarkupSafe==3.0.2
133
+ matplotlib==3.10.3
134
+ matplotlib-inline==0.1.7
135
+ mdit-py-plugins==0.4.2
136
+ mdurl==0.1.2
137
+ -e git+ssh://git@github.com/sriram-17-17/my-Megatron-MoE-Echo.git@9a23679390d7ab1fe462bc61fa31d90d393db49c#egg=megatron_core
138
+ mistune==3.1.3
139
+ mkl==2021.1.1
140
+ mkl-devel==2021.1.1
141
+ mkl-include==2021.1.1
142
+ ml_dtypes==0.6.0
143
+ mock==5.2.0
144
+ -e git+ssh://git@github.com/sriram-17-17/MoonEP.git@d8bc6667fad7177e6e09b637d773fa67f08bae93#egg=moonep
145
+ mpmath==1.3.0
146
+ msgpack==1.1.0
147
+ multidict @ file:///rapids/multidict-6.4.4-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl#sha256=a920f9cf2abdf6e493c519492d892c362007f113c94da4c239ae88429835bad1
148
+ multiprocess==0.70.19
149
+ murmurhash==1.0.13
150
+ mypy_extensions==1.1.0
151
+ nbclient==0.10.2
152
+ nbconvert==7.16.6
153
+ nbformat==5.10.4
154
+ nest-asyncio==1.6.0
155
+ networkx==3.5
156
+ ninja==1.11.1.4
157
+ notebook==7.4.3
158
+ notebook_shim==0.2.4
159
+ numba==0.59.1
160
+ numba-cuda @ file:///rapids/numba_cuda-0.4.0-py3-none-any.whl#sha256=5b95a5f42068771e37142944d44fe844fa87f1b2f89051461a4cb0af91bcc28a
161
+ numcodecs @ file:///rapids/numcodecs-0.13.1-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl#sha256=da2230484e6102e5fa3cc1a5dd37ca1f92dfbd183d91662074d6f7574e3e8f53
162
+ numpy==1.26.4
163
+ nvdlfw_inspect @ git+https://github.com/NVIDIA/nvidia-dlfw-inspect.git@6b60eb0e675606fb2fbbfcb12667ecbec75eaf87
164
+ nvfuser==0.2.27a0+9bf5aca
165
+ nvidia-cuda-cccl-cu12==12.9.27
166
+ nvidia-cudnn-frontend @ file:///opt/pytorch/lightning-thunder/tmp/cudnn_frontend
167
+ nvidia-cutlass-dsl==4.4.2
168
+ nvidia-cutlass-dsl-libs-base==4.4.2
169
+ nvidia-dali-cuda120==1.50.0
170
+ nvidia-ml-py @ file:///rapids/nvidia_ml_py-12.575.51-py3-none-any.whl#sha256=eb8641800d98ce40a22f479873f34b482e214a7e80349c63be51c3919845446e
171
+ nvidia-modelopt==0.29.0
172
+ nvidia-modelopt-core==0.29.0
173
+ nvidia-nvcomp-cu12==4.2.0.14
174
+ nvidia-nvimgcodec-cu12==0.5.0.13
175
+ nvidia-nvjpeg-cu12==12.4.0.16
176
+ nvidia-nvjpeg2k-cu12==0.8.1.40
177
+ nvidia-nvshmem-cu12==3.7.2
178
+ nvidia-nvtiff-cu12==0.5.0.67
179
+ nvidia-resiliency-ext==0.4.0
180
+ nvtx @ file:///rapids/nvtx-0.2.11-cp312-cp312-linux_x86_64.whl#sha256=73521e37cba42b4db0b3cd68fc5eb814e55c500a61eeed6edcdf6746315ac53e
181
+ nx-cugraph @ file:///rapids/nx_cugraph-25.4.0-py3-none-any.whl#sha256=e625444486cc899a77bd48e7da76a2edd95c0006dd95fbf323745feae7f33111
182
+ onnx @ file:///opt/pytorch/pytorch/third_party/onnx
183
+ onnx-ir==1.0.0
184
+ onnxscript==0.7.1
185
+ opt_einsum==3.4.0
186
+ optree==0.16.0
187
+ overrides==7.7.0
188
+ packaging==23.2
189
+ pandas @ file:///rapids/pandas-2.2.3-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl#sha256=fffb8ae78d8af97f849404f21411c95062db1496aeb3e56f146f0355c9989319
190
+ pandocfilters==1.5.1
191
+ parso==0.8.4
192
+ partd @ file:///rapids/partd-1.4.2-py3-none-any.whl#sha256=978e4ac767ec4ba5b86c6eaa52e5a2a3bc748a2ca839e8cc798f1cc6ce6efb0f
193
+ pathspec==0.12.1
194
+ pexpect==4.9.0
195
+ pillow==11.2.1
196
+ platformdirs==4.3.8
197
+ pluggy==1.6.0
198
+ ply @ file:///rapids/ply-3.11-py2.py3-none-any.whl#sha256=096f9b8350b65ebd2fd1346b12452efe5b9607f7482813ffca50c22722a807ce
199
+ polars @ file:///rapids/polars-1.25.2-cp39-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl#sha256=f7fcbb4f476784384ccda48757fca4e8c2e2c5a0a3aef3717aaf56aee4e30e09
200
+ polygraphy==0.49.20
201
+ pooch==1.8.2
202
+ preshed==3.0.10
203
+ prometheus_client==0.22.1
204
+ prompt_toolkit==3.0.51
205
+ propcache @ file:///rapids/propcache-0.3.1-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl#sha256=563f9d8c03ad645597b8d010ef4e9eab359faeb11a0a2ac9f7b4bc8c28ebef25
206
+ protobuf==4.24.4
207
+ psutil==7.0.0
208
+ ptyprocess==0.7.0
209
+ PuLP==3.2.1
210
+ pure_eval==0.2.3
211
+ pyarrow==25.0.1
212
+ pybind11==2.13.6
213
+ pybind11_global==2.13.6
214
+ pycocotools @ git+https://github.com/nvidia/cocoapi.git@08350a157b41f0755a536441b5abec89ab934c5b#subdirectory=PythonAPI
215
+ pycparser==2.22
216
+ pydantic==2.11.5
217
+ pydantic_core==2.33.2
218
+ Pygments==2.19.1
219
+ pylibcudf @ file:///rapids/pylibcudf-25.4.0-cp312-cp312-linux_x86_64.whl#sha256=401b8bfd022d06f84e120779ca7c3d01a28b68762aaa8f7873b10d6b1975cf6a
220
+ pylibcugraph @ file:///rapids/pylibcugraph-25.4.0-cp312-cp312-linux_x86_64.whl#sha256=061c7389a88c815b54fd5d3a3584fb87bf693e5602f5be335260d0ba575ba557
221
+ pylibcugraphops @ file:///rapids/pylibcugraphops-25.4.0-cp312-cp312-linux_x86_64.whl#sha256=a25ef4b22166c80718d5c7e5bddb755e81a2fe340fa5becd0ac36ec3edfb208a
222
+ pylibraft @ file:///rapids/pylibraft-25.4.0-cp312-cp312-linux_x86_64.whl#sha256=f2ecd23fe51f9c2caa87bbbe85cb32fcaff4ecfb52ceb9a9991bc50f82782c2e
223
+ pylibwholegraph @ file:///rapids/pylibwholegraph-25.4.0-cp312-cp312-linux_x86_64.whl#sha256=b82913d1e38bda76d8c272fbd8cc9072d987209d3d47cc93a464a37c2925aca2
224
+ pynvjitlink @ file:///rapids/pynvjitlink-0.3.0-cp312-cp312-linux_x86_64.whl#sha256=066caf1314730643d3b48c9847bd65789defdd6a7c78993176eb3d8bbe3f4919
225
+ pynvml @ file:///rapids/pynvml-12.0.0-py3-none-any.whl#sha256=fdff84b62a27dbe98e08e1a647eb77342bef1aebe0878bcd15e99a83fcbecb9e
226
+ pyparsing==3.2.3
227
+ pytest==8.1.1
228
+ pytest-flakefinder==1.1.0
229
+ pytest-rerunfailures==15.1
230
+ pytest-shard==0.1.2
231
+ pytest-xdist==3.7.0
232
+ python-dateutil==2.9.0.post0
233
+ python-hostlist==2.2.1
234
+ python-json-logger==3.3.0
235
+ pytorch-triton @ file:///tmp/dist/pytorch_triton-3.3.0%2Bgit96316ce52.nvinternal-cp312-cp312-linux_x86_64.whl#sha256=6d6921772b9dfd0243c055e0d054de42a113e92cabcfdd1d3e80a5ad9c08cf7d
236
+ pytz @ file:///rapids/pytz-2023.4-py2.py3-none-any.whl#sha256=f90ef520d95e7c46951105338d918664ebfd6f1d995bd7d153127ce90efafa6a
237
+ PyYAML==6.0.2
238
+ pyzmq==26.4.0
239
+ raft-dask @ file:///rapids/raft_dask-25.4.0-cp312-cp312-linux_x86_64.whl#sha256=36cf37ba6ce35454c4729c2cb597a360f22780eefb11d514a16a07137da5eb6c
240
+ rapids-dask-dependency @ file:///rapids/rapids_dask_dependency-25.4.0a0-py3-none-any.whl#sha256=484faca626e7760112a7d8ad3426f5bdd720b609006d8dca87c2607076a348bd
241
+ rapids-logger @ file:///rapids/rapids_logger-0.1.18-py3-none-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl#sha256=3617b2d86291ac6cb957542895f20fc9737ace29773ca90337e6fb04c73bd120
242
+ referencing==0.36.2
243
+ regex==2024.11.6
244
+ requests==2.32.3
245
+ rfc3339-validator==0.1.4
246
+ rfc3986-validator==0.1.1
247
+ rich==14.0.0
248
+ rmm @ file:///rapids/rmm-25.4.0-cp312-cp312-linux_x86_64.whl#sha256=1824bcab3836296666980a997ddcb9c4bca24dfb873012cfa0a7ecea583ca712
249
+ rpds-py==0.25.1
250
+ safetensors==0.5.3
251
+ scikit-learn==1.6.1
252
+ scipy==1.15.3
253
+ Send2Trash==1.8.3
254
+ setuptools==78.1.1
255
+ shellingham==1.5.4
256
+ six==1.16.0
257
+ smart-open==7.1.0
258
+ sniffio==1.3.1
259
+ sortedcontainers==2.4.0
260
+ soundfile==0.13.1
261
+ soupsieve==2.7
262
+ soxr==0.5.0.post1
263
+ spacy==3.7.5
264
+ spacy-legacy==3.0.12
265
+ spacy-loggers==1.0.5
266
+ srsly==2.5.1
267
+ stack-data==0.6.3
268
+ sympy==1.14.0
269
+ tabulate==0.9.0
270
+ tbb==2021.13.1
271
+ tblib @ file:///rapids/tblib-3.1.0-py3-none-any.whl#sha256=670bb4582578134b3d81a84afa1b016128b429f3d48e6cbbaecc9d15675e984e
272
+ tensorboard==2.16.2
273
+ tensorboard-data-server==0.7.2
274
+ tensorrt @ file:///workspace/TensorRT-10.11.0.33/python/tensorrt-10.11.0.33-cp312-none-linux_x86_64.whl#sha256=c2f18f72758a3d54a71901495c8fda9e7094a8bcdd2a5b6b58967fbf05733b3a
275
+ terminado==0.18.1
276
+ thinc==8.2.5
277
+ threadpoolctl==3.6.0
278
+ thriftpy2 @ file:///rapids/thriftpy2-0.5.2-cp312-cp312-linux_x86_64.whl#sha256=c0c7d939bdc5ccf7739154959938445d5fdf72f1d6e12bd563947a36f67e2147
279
+ tinycss2==1.4.0
280
+ toolz @ file:///rapids/toolz-1.0.0-py3-none-any.whl#sha256=292c8f1c4e7516bf9086f8850935c799a874039c8bcf959d47b600e4c44a6236
281
+ torch @ file:///opt/transfer/torch-2.8.0a0%2B5228986c39.nv25.6-29946474-cp312-cp312-linux_x86_64.whl#sha256=7a3c9ff5288d8102a31296b3d62491f3f47c8820a5c9aaefc7dc0fe20e1e8ad5
282
+ torch_tensorrt @ file:///opt/pytorch/torch_tensorrt/dist/torch_tensorrt-2.8.0a0-cp312-cp312-linux_x86_64.whl#sha256=26f4d7a04e7e6e017a5b917b8b98b83720bb2bb0195f48d25f52d567d286bfe1
283
+ torchao @ file:///opt/pytorch/ao
284
+ torchprofile==0.0.4
285
+ torchvision @ file:///opt/transfer/torchvision-0.22.0a0%2B95f10a4e-29946474-cp312-cp312-manylinux_2_28_x86_64.whl#sha256=746a3c674cc61680a86f4a14b374b21af1b5c21f629c42765b40c9494f6b234d
286
+ tornado==6.5.1
287
+ tqdm==4.67.1
288
+ traitlets==5.14.3
289
+ transformer_engine @ file:///pvcdata/aiinfra/b-aghosh/wheels/transformer_engine-2.10.0.dev0%2B10f6128d-cp312-cp312-linux_x86_64.whl#sha256=35cf5f4b83b2baad864e12e512bb422e0929409ac05610812fa521d4942d0189
290
+ treelite @ file:///rapids/treelite-4.4.1-py3-none-linux_x86_64.whl#sha256=174741724205540a1a5b37da81483aeff55d89548f42b3461cf07acac7ae96eb
291
+ typer==0.16.0
292
+ types-dataclasses==0.6.6
293
+ types-python-dateutil==2.9.0.20250516
294
+ typing-inspection==0.4.1
295
+ typing_extensions==4.14.0
296
+ tzdata @ file:///rapids/tzdata-2025.2-py2.py3-none-any.whl#sha256=1a403fada01ff9221ca8044d701868fa132215d84beb92242d9acd2147f667a8
297
+ ucx-py @ file:///rapids/ucx_py-0.43.0-cp312-cp312-linux_x86_64.whl#sha256=4a452fd8d8c983283ada3d49086e3657dc2ee94fbad08efdd708d15ab6e0b73a
298
+ ucxx @ file:///rapids/ucxx-0.43.0-cp312-cp312-linux_x86_64.whl#sha256=f1b4e55cb45c66fe88a2d8f9c2b87751165f3cfec21fbdc9c47c4ddff75e8a5f
299
+ -e git+https://github.com/Dots-Infra/UltraEP.git@94cab099b44fffa99a82fea99e7c12d89cf65e4f#egg=ultra_ep
300
+ uri-template==1.3.0
301
+ urllib3==2.0.7
302
+ wasabi==1.1.3
303
+ wcwidth==0.2.13
304
+ weasel==0.4.1
305
+ webcolors==24.11.1
306
+ webencodings==0.5.1
307
+ websocket-client==1.8.0
308
+ Werkzeug==3.1.3
309
+ wheel==0.45.1
310
+ wrapt==1.17.2
311
+ xdoctest==1.0.2
312
+ xgboost @ file:///rapids/xgboost-2.1.4-py3-none-manylinux_2_39_x86_64.whl#sha256=25eeca27f4048fc491940b3a0b4b5782ee48c33cb3aec059cef71f8da542930b
313
+ xxhash==4.0.1
314
+ yarl @ file:///rapids/yarl-1.20.0-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl#sha256=bf099e2432131093cc611623e0b0bcc399b8cddd9a91eded8bfb50402ec35018
315
+ zarr @ file:///rapids/zarr-2.18.7-py3-none-any.whl#sha256=ac3dc4033e9ae4e9d7b5e27c97ea3eaf1003cc0a07f010bd83d5134bf8c4b223
316
+ zict @ file:///rapids/zict-3.0.0-py2.py3-none-any.whl#sha256=5796e36bd0e0cc8cf0fbc1ace6a68912611c1dbd74750a3f3026b9b9d6a327ae
317
+ zipp==3.22.0
tiera-rerun-fixed-moonep/echo/a2/provenance.txt ADDED
@@ -0,0 +1,90 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ==============================================================
2
+ PROVENANCE: echo__a2__gbs16
3
+ ==============================================================
4
+ captured_at : 2026-08-30T06:44:13+00:00
5
+ hostname : sp-aiinfra-interactive-aghosh-7jtjl-8xb200-moe-cg-0
6
+ kernel : 6.8.0-1043-nvidia
7
+ cwd : /workspace/blackwell_run_scripts
8
+ command : run_ladder.sh --phase a2 --gbs 16 --out /workspace/tiera-rerun-fixed-moonep/echo/a2
9
+
10
+ ---- GPU / driver --------------------------------------------
11
+ 0, NVIDIA B200, 10.0, 183359 MiB, 580.95.05
12
+ 1, NVIDIA B200, 10.0, 183359 MiB, 580.95.05
13
+ 2, NVIDIA B200, 10.0, 183359 MiB, 580.95.05
14
+ 3, NVIDIA B200, 10.0, 183359 MiB, 580.95.05
15
+ 4, NVIDIA B200, 10.0, 183359 MiB, 580.95.05
16
+ 5, NVIDIA B200, 10.0, 183359 MiB, 580.95.05
17
+ 6, NVIDIA B200, 10.0, 183359 MiB, 580.95.05
18
+ 7, NVIDIA B200, 10.0, 183359 MiB, 580.95.05
19
+ persistence/clocks:
20
+ 0, 165 MHz, 1965 MHz, 1000.00 W, 29
21
+ 1, 120 MHz, 1965 MHz, 1000.00 W, 34
22
+ 2, 120 MHz, 1965 MHz, 1000.00 W, 34
23
+ 3, 120 MHz, 1965 MHz, 1000.00 W, 29
24
+ 4, 120 MHz, 1965 MHz, 1000.00 W, 30
25
+ 5, 120 MHz, 1965 MHz, 1000.00 W, 34
26
+ 6, 120 MHz, 1965 MHz, 1000.00 W, 34
27
+ 7, 120 MHz, 1965 MHz, 1000.00 W, 29
28
+
29
+ NVLink topology:
30
+ GPU0 GPU1 GPU2 GPU3 GPU4 GPU5 GPU6 GPU7 NIC0 NIC1 NIC2 NIC3 NIC4 NIC5 NIC6 NIC7 NIC8 CPU Affinity NUMA Affinity GPU NUMA ID
31
+ GPU0 X NV18 NV18 NV18 NV18 NV18 NV18 NV18 PIX NODE SYS SYS SYS SYS SYS SYS NODE 0-27 0 N/A
32
+ GPU1 NV18 X NV18 NV18 NV18 NV18 NV18 NV18 NODE PIX SYS SYS SYS SYS SYS SYS NODE 0-27 0 N/A
33
+ GPU2 NV18 NV18 X NV18 NV18 NV18 NV18 NV18 SYS SYS PIX NODE SYS SYS SYS SYS SYS 28-55 1 N/A
34
+ GPU3 NV18 NV18 NV18 X NV18 NV18 NV18 NV18 SYS SYS NODE PIX SYS SYS SYS SYS SYS 28-55 1 N/A
35
+ GPU4 NV18 NV18 NV18 NV18 X NV18 NV18 NV18 SYS SYS SYS SYS PIX NODE SYS SYS SYS 56-83 2 N/A
36
+ GPU5 NV18 NV18 NV18 NV18 NV18 X NV18 NV18 SYS SYS SYS SYS NODE PIX SYS SYS SYS 56-83 2 N/A
37
+ GPU6 NV18 NV18 NV18 NV18 NV18 NV18 X NV18 SYS SYS SYS SYS SYS SYS PIX NODE SYS 84-111 3 N/A
38
+ GPU7 NV18 NV18 NV18 NV18 NV18 NV18 NV18 X SYS SYS SYS SYS SYS SYS NODE PIX SYS 84-111 3 N/A
39
+ NIC0 PIX NODE SYS SYS SYS SYS SYS SYS X NODE SYS SYS SYS SYS SYS SYS NODE
40
+ NIC1 NODE PIX SYS SYS SYS SYS SYS SYS NODE X SYS SYS SYS SYS SYS SYS NODE
41
+ NIC2 SYS SYS PIX NODE SYS SYS SYS SYS SYS SYS X NODE SYS SYS SYS SYS SYS
42
+ NIC3 SYS SYS NODE PIX SYS SYS SYS SYS SYS SYS NODE X SYS SYS SYS SYS SYS
43
+ NIC4 SYS SYS SYS SYS PIX NODE SYS SYS SYS SYS SYS SYS X NODE SYS SYS SYS
44
+ unavailable
45
+
46
+ ---- toolchain -----------------------------------------------
47
+ nvcc : release 12.9, V12.9.86
48
+ gcc : gcc (Ubuntu 13.3.0-6ubuntu2~24.04) 13.3.0
49
+ python : 3.12.3
50
+ torch : 2.8.0a0+5228986c39.nv25.06 cuda=12.9 nccl=2.27.3
51
+ torch arch : ['sm_75', 'sm_80', 'sm_86', 'sm_90', 'sm_100', 'sm_120', 'compute_120']
52
+ transformer_engine: 2.10.0.dev0+10f6128d [/usr/local/lib/python3.12/dist-packages/transformer_engine/__init__.py]
53
+ deep_ep : 1.2.1+34152ae [/workspace/DeepEP/deep_ep/__init__.py]
54
+ ultra_ep : 1.0.0 [/workspace/UltraEP/ultra_ep/__init__.py]
55
+ moonep : 0.0.1 [/workspace/MoonEP/moonep/__init__.py]
56
+ megatron : 0.14.0rc5 [None]
57
+ flash_attn : 2.7.4.post1 [/usr/local/lib/python3.12/dist-packages/flash_attn/__init__.py]
58
+ numpy : 1.26.4 [/usr/local/lib/python3.12/dist-packages/numpy/__init__.py]
59
+ huggingface_hub: 1.29.0 [/usr/local/lib/python3.12/dist-packages/huggingface_hub/__init__.py]
60
+
61
+ ---- repositories --------------------------------------------
62
+ Megatron-MoE-Echo bracing-regions-through-kernels 9a23679390d7
63
+ Megatron-UltraEP ultraep-token-drop-analysis eaf829e7ad21
64
+ UltraEP main 94cab099b44f
65
+ DeepEP HEAD 34152ae28f80 (+3 untracked: build output)
66
+ MoonEP megatron-ddp-symmetric-params d8bc6667fad7
67
+ blackwell_run_scripts granularity-echo ff2c833ea5a5
68
+
69
+ ---- environment ---------------------------------------------
70
+ CUDA_ARCH_LIST=7.5 8.0 8.6 9.0 10.0 12.0
71
+ CUDA_DRIVER_VERSION=575.57.08
72
+ CUDA_HOME=/usr/local/cuda
73
+ CUDA_MODULE_LOADING=LAZY
74
+ CUDA_VERSION=12.9.1.010
75
+ HF_REMOTE_ROOT=tiera-rerun-fixed-moonep
76
+ NCCL_VERSION=2.27.3
77
+ NVSHMEM_VERSION=3.2.5
78
+ PYTORCH_BUILD_NUMBER=0
79
+ PYTORCH_BUILD_VERSION=2.8.0a0+5228986
80
+ PYTORCH_HOME=/opt/pytorch/pytorch
81
+ PYTORCH_VERSION=2.8.0a0+5228986
82
+ TORCH_ALLOW_TF32_CUBLAS_OVERRIDE=1
83
+ TORCH_CUDA_ARCH_LIST=7.5 8.0 8.6 9.0 10.0 12.0+PTX
84
+ TORCH_NCCL_USE_COMM_NONBLOCKING=0
85
+
86
+ ---- shared library resolution -------------------------------
87
+ libnvshmem_host.so: not in ldconfig
88
+ libnccl.so: /lib/x86_64-linux-gnu/libnccl.so.2
89
+ lrwxrwxrwx 1 root root 20 Aug 29 16:41 /usr/local/lib/python3.12/dist-packages/nvidia/nvshmem/lib/libnvshmem_host.so -> libnvshmem_host.so.3
90
+ -rw-r--r-- 1 root root 179045744 Aug 29 16:37 /usr/local/lib/python3.12/dist-packages/nvidia/nvshmem/lib/libnvshmem_host.so.3
tiera-rerun-fixed-moonep/hf_watcher.log CHANGED
@@ -29,3 +29,6 @@ https://huggingface.co/MoE-CUDA-Graph-Analysis/tierb-b200-gbs8/tree/main
29
  uploaded: /workspace/tiera-rerun-fixed-moonep -> MoE-CUDA-Graph-Analysis/tierb-b200-gbs8/tiera-rerun-fixed-moonep
30
  https://huggingface.co/MoE-CUDA-Graph-Analysis/tierb-b200-gbs8/tree/main
31
  [2026-08-30T06:40:36+00:00] backup pass
 
 
 
 
29
  uploaded: /workspace/tiera-rerun-fixed-moonep -> MoE-CUDA-Graph-Analysis/tierb-b200-gbs8/tiera-rerun-fixed-moonep
30
  https://huggingface.co/MoE-CUDA-Graph-Analysis/tierb-b200-gbs8/tree/main
31
  [2026-08-30T06:40:36+00:00] backup pass
32
+ uploaded: /workspace/tiera-rerun-fixed-moonep -> MoE-CUDA-Graph-Analysis/tierb-b200-gbs8/tiera-rerun-fixed-moonep
33
+ https://huggingface.co/MoE-CUDA-Graph-Analysis/tierb-b200-gbs8/tree/main
34
+ [2026-08-30T06:50:38+00:00] backup pass
tiera-rerun-fixed-moonep/steps.tsv CHANGED
@@ -2,3 +2,4 @@ status campaign phase gbs started ended result_dir
2
  FAIL echo tierbprobe 16 2026-08-30T04:52:38+00:00 2026-08-30T05:06:38+00:00 /workspace/tiera-rerun-fixed-moonep/echo/tierbprobe
3
  FAIL ultraep tierbprobe 16 2026-08-30T05:06:44+00:00 2026-08-30T05:20:40+00:00 /workspace/tiera-rerun-fixed-moonep/ultraep/tierbprobe
4
  OK echo a1 16 2026-08-30T06:20:31+00:00 2026-08-30T06:36:28+00:00 /workspace/tiera-rerun-fixed-moonep/echo/a1
 
 
2
  FAIL echo tierbprobe 16 2026-08-30T04:52:38+00:00 2026-08-30T05:06:38+00:00 /workspace/tiera-rerun-fixed-moonep/echo/tierbprobe
3
  FAIL ultraep tierbprobe 16 2026-08-30T05:06:44+00:00 2026-08-30T05:20:40+00:00 /workspace/tiera-rerun-fixed-moonep/ultraep/tierbprobe
4
  OK echo a1 16 2026-08-30T06:20:31+00:00 2026-08-30T06:36:28+00:00 /workspace/tiera-rerun-fixed-moonep/echo/a1
5
+ OK echo ponly 16 2026-08-30T06:36:31+00:00 2026-08-30T06:44:11+00:00 /workspace/tiera-rerun-fixed-moonep/echo/ponly
tiera-rerun-fixed-moonep/study.log CHANGED
@@ -521,3 +521,51 @@ https://huggingface.co/MoE-CUDA-Graph-Analysis/tierb-b200-gbs8/tree/main
521
  [guard] tearing down process group (outcome=DONE)
522
  --------------------------------------------------------------------------
523
  [06:40:26] CELL PONLY__echo__one__seq8192__gbs16__cap2p40625 (slots=1)
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
521
  [guard] tearing down process group (outcome=DONE)
522
  --------------------------------------------------------------------------
523
  [06:40:26] CELL PONLY__echo__one__seq8192__gbs16__cap2p40625 (slots=1)
524
+ [guard] iter 12/12 (70s)
525
+ [guard] tearing down process group (outcome=DONE)
526
+ [guard] iter 12/12 (70s)
527
+ [guard] tearing down process group (outcome=DONE)
528
+ [guard] iter 12/12 (70s)
529
+ [guard] tearing down process group (outcome=DONE)
530
+ --------------------------------------------------------------------------
531
+
532
+ ==============================================================
533
+ SWEEP COMPLETE ok=6 failed=0
534
+ ==============================================================
535
+ config backend policy slots seq gbs cap recompute rep status iter_ms_mean iter_ms_min peak_mib run_dir
536
+ PONLY echo one 1 4096 16 2.40625 none 1 OK 175.19 174.10 40157 /workspace/tiera-rerun-fixed-moonep/echo/ponly/ponly_gbs16/PONLY__echo__one__seq4096__gbs16__cap2p40625/rep1
537
+ PONLY echo one 1 4096 16 2.40625 none 2 OK 174.92 173.90 40241 /workspace/tiera-rerun-fixed-moonep/echo/ponly/ponly_gbs16/PONLY__echo__one__seq4096__gbs16__cap2p40625/rep2
538
+ PONLY echo one 1 4096 16 2.40625 none 3 OK 174.81 173.80 40243 /workspace/tiera-rerun-fixed-moonep/echo/ponly/ponly_gbs16/PONLY__echo__one__seq4096__gbs16__cap2p40625/rep3
539
+ PONLY echo one 1 8192 16 2.40625 none 1 OK 316.36 315.50 53279 /workspace/tiera-rerun-fixed-moonep/echo/ponly/ponly_gbs16/PONLY__echo__one__seq8192__gbs16__cap2p40625/rep1
540
+ PONLY echo one 1 8192 16 2.40625 none 2 OK 316.47 315.50 53283 /workspace/tiera-rerun-fixed-moonep/echo/ponly/ponly_gbs16/PONLY__echo__one__seq8192__gbs16__cap2p40625/rep2
541
+ PONLY echo one 1 8192 16 2.40625 none 3 OK 316.70 315.40 53283 /workspace/tiera-rerun-fixed-moonep/echo/ponly/ponly_gbs16/PONLY__echo__one__seq8192__gbs16__cap2p40625/rep3
542
+
543
+ Results: /workspace/tiera-rerun-fixed-moonep/echo/ponly/ponly_gbs16/results.tsv
544
+ Roll up: python3 /workspace/blackwell_run_scripts/granularity/tools/collect.py /workspace/tiera-rerun-fixed-moonep/echo/ponly/ponly_gbs16
545
+ ==============================================================
546
+
547
+ Campaign results: /workspace/tiera-rerun-fixed-moonep/echo/ponly
548
+ Roll up: python3 /workspace/blackwell_run_scripts/granularity/tools/collect.py /workspace/tiera-rerun-fixed-moonep/echo/ponly --compare
549
+ Already logged in to Hugging Face as: AbhishekGhosh1998
550
+ uploaded: /workspace/tiera-rerun-fixed-moonep/echo/ponly -> MoE-CUDA-Graph-Analysis/tierb-b200-gbs8/tiera-rerun-fixed-moonep/echo/ponly
551
+ https://huggingface.co/MoE-CUDA-Graph-Analysis/tierb-b200-gbs8/tree/main
552
+
553
+ === STEP echo__a2__gbs16
554
+ ==============================================================
555
+ CAMPAIGN: echo backend=echo phase=a2
556
+ Megatron: /workspace/Megatron-MoE-Echo @ bracing-regions-through-kernels
557
+ GBS=16 seq=4096 repeats=3 nsys=0
558
+ results -> /workspace/tiera-rerun-fixed-moonep/echo/a2
559
+ ==============================================================
560
+ --------------------------------------------------------------------------
561
+ [06:44:28] CELL G1__echo__matched__seq4096__gbs16__cap2p40625 (slots=1)
562
+ [guard] iter 12/12 (70s)
563
+ [guard] tearing down process group (outcome=DONE)
564
+ [guard] iter 11/12 (70s)
565
+ [guard] iter 12/12 (80s)
566
+ [guard] iter 12/12 (70s)
567
+ [guard] tearing down process group (outcome=DONE)
568
+ --------------------------------------------------------------------------
569
+ [06:48:19] CELL G2__echo__matched__seq4096__gbs16__cap1p08203125 (slots=4)
570
+ [guard] iter 8/12 (70s)
571
+ [guard] iter 12/12 (80s)