AbhishekGhosh1998 commited on
Commit
d5f0533
·
verified ·
1 Parent(s): f6ff918

granularity results tiera-rerun-fixed-moonep/ultraep/ponly @ 2026-08-30T07:22:02Z from sp-aiinfra-interactive-aghosh-7jtjl-8xb200-moe-cg-0

Browse files
tiera-rerun-fixed-moonep/ultraep/ponly/ponly_gbs16/PONLY__ultraep__one__seq8192__gbs16__cap2p40625/rep2/mem_samples.csv CHANGED
@@ -166,3 +166,35 @@
166
  5, 39689
167
  6, 39641
168
  7, 39707
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
166
  5, 39689
167
  6, 39641
168
  7, 39707
169
+ 0, 51015
170
+ 1, 51011
171
+ 2, 51009
172
+ 3, 51009
173
+ 4, 50975
174
+ 5, 51011
175
+ 6, 51009
176
+ 7, 51009
177
+ 0, 51015
178
+ 1, 51011
179
+ 2, 51009
180
+ 3, 51009
181
+ 4, 50975
182
+ 5, 51011
183
+ 6, 51009
184
+ 7, 51009
185
+ 0, 49845
186
+ 1, 49841
187
+ 2, 49839
188
+ 3, 49839
189
+ 4, 49805
190
+ 5, 49841
191
+ 6, 49839
192
+ 7, 49839
193
+ 0, 1537
194
+ 1, 1539
195
+ 2, 1540
196
+ 3, 1540
197
+ 4, 1796
198
+ 5, 49841
199
+ 6, 2820
200
+ 7, 1540
tiera-rerun-fixed-moonep/ultraep/ponly/ponly_gbs16/PONLY__ultraep__one__seq8192__gbs16__cap2p40625/rep2/train.log CHANGED
@@ -975,3 +975,51 @@ Theoretical memory footprints: weight and optimizer=25029.43 MB
975
  [2026-08-30 07:20:47] iteration 3/ 12 | consumed samples: 48 | elapsed time per iteration (ms): 329.7 | throughput per GPU (TFLOP/s/GPU): 601.7 | learning rate: 6.000000E-05 | global batch size: 16 | lm loss: 1.053758E+01 | loss scale: 1.0 | grad norm: 249.877 | number of skipped iterations: 0 | number of nan iterations: 0 | peak recv tokens: 28416 | min safe capacity: 1.7344 |
976
  Capture CUDA graph for training!!!
977
  CUDA graph capture done!!!
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
975
  [2026-08-30 07:20:47] iteration 3/ 12 | consumed samples: 48 | elapsed time per iteration (ms): 329.7 | throughput per GPU (TFLOP/s/GPU): 601.7 | learning rate: 6.000000E-05 | global batch size: 16 | lm loss: 1.053758E+01 | loss scale: 1.0 | grad norm: 249.877 | number of skipped iterations: 0 | number of nan iterations: 0 | peak recv tokens: 28416 | min safe capacity: 1.7344 |
976
  Capture CUDA graph for training!!!
977
  CUDA graph capture done!!!
978
+ [2026-08-30 07:20:48] iteration 4/ 12 | consumed samples: 64 | elapsed time per iteration (ms): 1110.8 | throughput per GPU (TFLOP/s/GPU): 178.6 | learning rate: 8.000000E-05 | global batch size: 16 | lm loss: 1.319968E+01 | loss scale: 1.0 | grad norm: 35.719 | number of skipped iterations: 0 | number of nan iterations: 0 | peak recv tokens: 28416 | min safe capacity: 1.7344 |
979
+ [2026-08-30 07:20:48] iteration 5/ 12 | consumed samples: 80 | elapsed time per iteration (ms): 292.8 | throughput per GPU (TFLOP/s/GPU): 677.6 | learning rate: 1.000000E-04 | global batch size: 16 | lm loss: 1.052109E+01 | loss scale: 1.0 | grad norm: 17.525 | number of skipped iterations: 0 | number of nan iterations: 0 | peak recv tokens: 28416 | min safe capacity: 1.7344 |
980
+ [2026-08-30 07:20:48] iteration 6/ 12 | consumed samples: 96 | elapsed time per iteration (ms): 293.5 | throughput per GPU (TFLOP/s/GPU): 676.0 | learning rate: 7.750000E-05 | global batch size: 16 | lm loss: 1.027258E+01 | loss scale: 1.0 | grad norm: 9.627 | number of skipped iterations: 0 | number of nan iterations: 0 | peak recv tokens: 28416 | min safe capacity: 1.7344 |
981
+ [2026-08-30 07:20:49] iteration 7/ 12 | consumed samples: 112 | elapsed time per iteration (ms): 291.4 | throughput per GPU (TFLOP/s/GPU): 680.9 | learning rate: 3.250000E-05 | global batch size: 16 | lm loss: 9.465188E+00 | loss scale: 1.0 | grad norm: 7.772 | number of skipped iterations: 0 | number of nan iterations: 0 | peak recv tokens: 28416 | min safe capacity: 1.7344 |
982
+ [2026-08-30 07:20:49] iteration 8/ 12 | consumed samples: 128 | elapsed time per iteration (ms): 291.4 | throughput per GPU (TFLOP/s/GPU): 680.7 | learning rate: 1.000000E-05 | global batch size: 16 | lm loss: 8.756442E+00 | loss scale: 1.0 | grad norm: 6.065 | number of skipped iterations: 0 | number of nan iterations: 0 | peak recv tokens: 28416 | min safe capacity: 1.7344 |
983
+ [2026-08-30 07:20:49] iteration 9/ 12 | consumed samples: 144 | elapsed time per iteration (ms): 292.7 | throughput per GPU (TFLOP/s/GPU): 677.7 | learning rate: 1.000000E-05 | global batch size: 16 | lm loss: 8.545505E+00 | loss scale: 1.0 | grad norm: 7.211 | number of skipped iterations: 0 | number of nan iterations: 0 | peak recv tokens: 28416 | min safe capacity: 1.7344 |
984
+ [2026-08-30 07:20:50] iteration 10/ 12 | consumed samples: 160 | elapsed time per iteration (ms): 291.3 | throughput per GPU (TFLOP/s/GPU): 681.0 | learning rate: 1.000000E-05 | global batch size: 16 | lm loss: 8.448534E+00 | loss scale: 1.0 | grad norm: 6.020 | number of skipped iterations: 0 | number of nan iterations: 0 | peak recv tokens: 28416 | min safe capacity: 1.7344 |
985
+ [2026-08-30 07:20:50] iteration 11/ 12 | consumed samples: 176 | elapsed time per iteration (ms): 292.8 | throughput per GPU (TFLOP/s/GPU): 677.4 | learning rate: 1.000000E-05 | global batch size: 16 | lm loss: 8.375360E+00 | loss scale: 1.0 | grad norm: 4.678 | number of skipped iterations: 0 | number of nan iterations: 0 | peak recv tokens: 28416 | min safe capacity: 1.7344 |
986
+ [2026-08-30 07:20:50] iteration 12/ 12 | consumed samples: 192 | elapsed time per iteration (ms): 294.2 | throughput per GPU (TFLOP/s/GPU): 674.3 | learning rate: 1.000000E-05 | global batch size: 16 | lm loss: 8.276701E+00 | loss scale: 1.0 | grad norm: 4.158 | number of skipped iterations: 0 | number of nan iterations: 0 | peak recv tokens: 28416 | min safe capacity: 1.7344 |
987
+ [after training is done] datetime: 2026-08-30 07:20:50
988
+ [rank0]:[W830 07:20:51.917748913 ProcessGroupNCCL.cpp:1505] Warning: WARNING: destroy_process_group() was not called before program exit, which can leak resources. For more info, please see https://pytorch.org/docs/stable/distributed.html#shutdown (function operator())
989
+ W0830 07:20:52.491000 741931 torch/distributed/elastic/agent/server/api.py:719] Received 15 death signal, shutting down workers
990
+ W0830 07:20:52.491000 741931 torch/distributed/elastic/multiprocessing/api.py:900] Sending process 742021 closing signal SIGTERM
991
+ W0830 07:20:52.492000 741931 torch/distributed/elastic/multiprocessing/api.py:900] Sending process 742022 closing signal SIGTERM
992
+ W0830 07:20:52.493000 741931 torch/distributed/elastic/multiprocessing/api.py:900] Sending process 742023 closing signal SIGTERM
993
+ W0830 07:20:52.495000 741931 torch/distributed/elastic/multiprocessing/api.py:900] Sending process 742024 closing signal SIGTERM
994
+ W0830 07:20:52.497000 741931 torch/distributed/elastic/multiprocessing/api.py:900] Sending process 742025 closing signal SIGTERM
995
+ W0830 07:20:52.499000 741931 torch/distributed/elastic/multiprocessing/api.py:900] Sending process 742026 closing signal SIGTERM
996
+ W0830 07:20:52.501000 741931 torch/distributed/elastic/multiprocessing/api.py:900] Sending process 742027 closing signal SIGTERM
997
+ W0830 07:20:52.502000 741931 torch/distributed/elastic/multiprocessing/api.py:900] Sending process 742028 closing signal SIGTERM
998
+ Traceback (most recent call last):
999
+ File "/usr/local/bin/torchrun", line 8, in <module>
1000
+ sys.exit(main())
1001
+ ^^^^^^
1002
+ File "/usr/local/lib/python3.12/dist-packages/torch/distributed/elastic/multiprocessing/errors/__init__.py", line 355, in wrapper
1003
+ return f(*args, **kwargs)
1004
+ ^^^^^^^^^^^^^^^^^^
1005
+ File "/usr/local/lib/python3.12/dist-packages/torch/distributed/run.py", line 892, in main
1006
+ run(args)
1007
+ File "/usr/local/lib/python3.12/dist-packages/torch/distributed/run.py", line 883, in run
1008
+ elastic_launch(
1009
+ File "/usr/local/lib/python3.12/dist-packages/torch/distributed/launcher/api.py", line 139, in __call__
1010
+ return launch_agent(self._config, self._entrypoint, list(args))
1011
+ ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
1012
+ File "/usr/local/lib/python3.12/dist-packages/torch/distributed/launcher/api.py", line 261, in launch_agent
1013
+ result = agent.run()
1014
+ ^^^^^^^^^^^
1015
+ File "/usr/local/lib/python3.12/dist-packages/torch/distributed/elastic/metrics/api.py", line 138, in wrapper
1016
+ result = f(*args, **kwargs)
1017
+ ^^^^^^^^^^^^^^^^^^
1018
+ File "/usr/local/lib/python3.12/dist-packages/torch/distributed/elastic/agent/server/api.py", line 711, in run
1019
+ result = self._invoke_run(role)
1020
+ ^^^^^^^^^^^^^^^^^^^^^^
1021
+ File "/usr/local/lib/python3.12/dist-packages/torch/distributed/elastic/agent/server/api.py", line 870, in _invoke_run
1022
+ time.sleep(monitor_interval)
1023
+ File "/usr/local/lib/python3.12/dist-packages/torch/distributed/elastic/multiprocessing/api.py", line 84, in _terminate_process_handler
1024
+ raise SignalException(f"Process {os.getpid()} got signal: {sigval}", sigval=sigval)
1025
+ torch.distributed.elastic.multiprocessing.api.SignalException: Process 741931 got signal: 15
tiera-rerun-fixed-moonep/ultraep/ponly/ponly_gbs16/PONLY__ultraep__one__seq8192__gbs16__cap2p40625/rep3/mem_samples.csv ADDED
@@ -0,0 +1,200 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ 0, 0
2
+ 1, 0
3
+ 2, 0
4
+ 3, 0
5
+ 4, 0
6
+ 5, 0
7
+ 6, 0
8
+ 7, 0
9
+ 0, 0
10
+ 1, 0
11
+ 2, 0
12
+ 3, 0
13
+ 4, 0
14
+ 5, 0
15
+ 6, 0
16
+ 7, 0
17
+ 0, 4
18
+ 1, 1
19
+ 2, 1
20
+ 3, 1
21
+ 4, 1
22
+ 5, 1
23
+ 6, 1
24
+ 7, 1
25
+ 0, 4
26
+ 1, 4
27
+ 2, 4
28
+ 3, 4
29
+ 4, 2
30
+ 5, 1
31
+ 6, 1
32
+ 7, 1
33
+ 0, 4
34
+ 1, 4
35
+ 2, 4
36
+ 3, 4
37
+ 4, 4
38
+ 5, 4
39
+ 6, 4
40
+ 7, 4
41
+ 0, 4
42
+ 1, 27
43
+ 2, 4
44
+ 3, 4
45
+ 4, 4
46
+ 5, 4
47
+ 6, 4
48
+ 7, 4
49
+ 0, 821
50
+ 1, 821
51
+ 2, 821
52
+ 3, 821
53
+ 4, 821
54
+ 5, 821
55
+ 6, 821
56
+ 7, 821
57
+ 0, 2007
58
+ 1, 2007
59
+ 2, 2007
60
+ 3, 2007
61
+ 4, 2007
62
+ 5, 2007
63
+ 6, 2007
64
+ 7, 2007
65
+ 0, 2263
66
+ 1, 2267
67
+ 2, 2273
68
+ 3, 2269
69
+ 4, 2271
70
+ 5, 2275
71
+ 6, 2271
72
+ 7, 2271
73
+ 0, 14589
74
+ 1, 14587
75
+ 2, 14587
76
+ 3, 14587
77
+ 4, 14587
78
+ 5, 14587
79
+ 6, 14587
80
+ 7, 14587
81
+ 0, 15123
82
+ 1, 15125
83
+ 2, 15123
84
+ 3, 15121
85
+ 4, 15121
86
+ 5, 15121
87
+ 6, 15121
88
+ 7, 15125
89
+ 0, 16161
90
+ 1, 16159
91
+ 2, 16159
92
+ 3, 16159
93
+ 4, 16159
94
+ 5, 16159
95
+ 6, 16159
96
+ 7, 16159
97
+ 0, 16685
98
+ 1, 16683
99
+ 2, 16683
100
+ 3, 16683
101
+ 4, 16683
102
+ 5, 16683
103
+ 6, 16683
104
+ 7, 16683
105
+ 0, 16993
106
+ 1, 16991
107
+ 2, 16991
108
+ 3, 16991
109
+ 4, 16991
110
+ 5, 16991
111
+ 6, 16991
112
+ 7, 16991
113
+ 0, 17959
114
+ 1, 17957
115
+ 2, 17957
116
+ 3, 17957
117
+ 4, 17957
118
+ 5, 17957
119
+ 6, 17957
120
+ 7, 17957
121
+ 0, 35699
122
+ 1, 35697
123
+ 2, 35697
124
+ 3, 35697
125
+ 4, 35659
126
+ 5, 35697
127
+ 6, 35697
128
+ 7, 35697
129
+ 0, 35699
130
+ 1, 35697
131
+ 2, 35697
132
+ 3, 35697
133
+ 4, 35659
134
+ 5, 35697
135
+ 6, 35697
136
+ 7, 35697
137
+ 0, 35699
138
+ 1, 35697
139
+ 2, 35697
140
+ 3, 35697
141
+ 4, 35659
142
+ 5, 35697
143
+ 6, 35697
144
+ 7, 35697
145
+ 0, 37513
146
+ 1, 37519
147
+ 2, 37525
148
+ 3, 37511
149
+ 4, 37471
150
+ 5, 37515
151
+ 6, 37545
152
+ 7, 37519
153
+ 0, 38577
154
+ 1, 38575
155
+ 2, 38555
156
+ 3, 38571
157
+ 4, 38531
158
+ 5, 38555
159
+ 6, 38583
160
+ 7, 38571
161
+ 0, 39681
162
+ 1, 39659
163
+ 2, 39683
164
+ 3, 39663
165
+ 4, 39615
166
+ 5, 39707
167
+ 6, 39663
168
+ 7, 39747
169
+ 0, 51013
170
+ 1, 51009
171
+ 2, 51011
172
+ 3, 51009
173
+ 4, 50975
174
+ 5, 51009
175
+ 6, 51011
176
+ 7, 51009
177
+ 0, 51013
178
+ 1, 51009
179
+ 2, 51011
180
+ 3, 51009
181
+ 4, 50975
182
+ 5, 51009
183
+ 6, 51011
184
+ 7, 51009
185
+ 0, 1540
186
+ 1, 1540
187
+ 2, 1540
188
+ 3, 1540
189
+ 4, 1540
190
+ 5, 1860
191
+ 6, 50487
192
+ 7, 3140
193
+ 0, 0
194
+ 1, 0
195
+ 2, 0
196
+ 3, 0
197
+ 4, 0
198
+ 5, 0
199
+ 6, 0
200
+ 7, 0
tiera-rerun-fixed-moonep/ultraep/ponly/ponly_gbs16/PONLY__ultraep__one__seq8192__gbs16__cap2p40625/rep3/run.env ADDED
@@ -0,0 +1,70 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ backend=ultraep
2
+ config=PONLY
3
+ family=tierA
4
+ repo=/workspace/Megatron-UltraEP
5
+ branch=ultraep-bracing-regions-through-kernels
6
+ commit=03d57ef7
7
+ num_layers=16
8
+ hidden=4096
9
+ ffn_hidden=14336
10
+ moe_ffn_hidden=1536
11
+ num_experts=8
12
+ topk=2
13
+ experts_per_rank=1
14
+ expert_params=18874368
15
+ heads=32
16
+ kv_groups=8
17
+ kv_channels=
18
+ qk_layernorm=0
19
+ shared_ffn=0
20
+ shared_expert_overlap=0
21
+ moe_token_dispatcher=flex
22
+ mla=0
23
+ dense_first_layers=0
24
+ moe_layers=16
25
+ rotary_base=1000000
26
+ rotary_scaling=
27
+ qkv_bias=0
28
+ router_balancing=aux_loss
29
+ aux_loss_coeff=1e-2
30
+ router_pre_softmax=0
31
+ router_fusion=0
32
+ router_dtype=fp32
33
+ device_initiated_grouped_gemm=1
34
+ norm_epsilon=1e-5
35
+ seq_length=8192
36
+ micro_batch_size=1
37
+ global_batch_size=16
38
+ num_microbatches=2
39
+ assignments_per_rank=16384
40
+ mean_tokens_per_expert=16384
41
+ slot_policy=one
42
+ slots_per_rank=1
43
+ capacity_mode=static
44
+ capacity=2.40625
45
+ cuda_graph=1
46
+ probe_mode=0
47
+ async_grad_reduce=0
48
+ branch_role=timing
49
+ memory_dump_mode=none
50
+ memory_snapshot_path=
51
+ cuda_graph_warmup_steps=3
52
+ recompute=none
53
+ force_balance=0
54
+ train_iters=12
55
+ ep_size=8
56
+ echo_dispatch_overlap=1
57
+ echo_dispatch_recompute=0
58
+ ultraep_quota_min_tokens=2048
59
+ ultraep_weight_sync_join=
60
+ profile_enable=0
61
+ kernel_trace=0
62
+ fp8_mode=mxfp8
63
+ fp8_recipe=mxfp8
64
+ training_seed=1234
65
+ learning_rate=1e-4
66
+ min_learning_rate=1.0e-5
67
+ lr_warmup_iters=5
68
+ state_dump_enable=0
69
+ state_dump_dir=
70
+ moonep_assert_no_pending_reduce=0
tiera-rerun-fixed-moonep/ultraep/ponly/ponly_gbs16/PONLY__ultraep__one__seq8192__gbs16__cap2p40625/rep3/train.log ADDED
@@ -0,0 +1,1025 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ==============================================================
2
+ PONLY / ultraep (slots/rank=1 policy=one)
3
+ E=8 K=2 H'=1536 m=1
4
+ L=16 seq=8192 gbs=16 (2 microbatches)
5
+ cap_mode=static cap=2.40625 cg=1 recompute=none
6
+ memory_dump=none snapshots=/workspace/tiera-rerun-fixed-moonep/ultraep/ponly/ponly_gbs16/PONLY__ultraep__one__seq8192__gbs16__cap2p40625/rep3/memory_snapshots/snapshot.pickle
7
+ repo=/workspace/Megatron-UltraEP @ 03d57ef7
8
+ ==============================================================
9
+ W0830 07:20:57.989000 744857 torch/distributed/run.py:766]
10
+ W0830 07:20:57.989000 744857 torch/distributed/run.py:766] *****************************************
11
+ W0830 07:20:57.989000 744857 torch/distributed/run.py:766] Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed.
12
+ W0830 07:20:57.989000 744857 torch/distributed/run.py:766] *****************************************
13
+ /workspace/Megatron-UltraEP/megatron/training/arguments.py:831: UserWarning: Disabling sequence parallelism because tensor model parallelism is disabled
14
+ warnings.warn("Disabling sequence parallelism because tensor model parallelism is disabled")
15
+ /workspace/Megatron-UltraEP/megatron/training/arguments.py:831: UserWarning: Disabling sequence parallelism because tensor model parallelism is disabled
16
+ warnings.warn("Disabling sequence parallelism because tensor model parallelism is disabled")
17
+ /workspace/Megatron-UltraEP/megatron/training/arguments.py:831: UserWarning: Disabling sequence parallelism because tensor model parallelism is disabled
18
+ warnings.warn("Disabling sequence parallelism because tensor model parallelism is disabled")
19
+ WARNING: one_logger package is required to enable e2e metrics tracking. please go to https://confluence.nvidia.com/display/MLWFO/Package+Repositories for details to install it
20
+ /workspace/Megatron-UltraEP/megatron/training/arguments.py:831: UserWarning: Disabling sequence parallelism because tensor model parallelism is disabled
21
+ warnings.warn("Disabling sequence parallelism because tensor model parallelism is disabled")
22
+ /workspace/Megatron-UltraEP/megatron/training/arguments.py:831: UserWarning: Disabling sequence parallelism because tensor model parallelism is disabled
23
+ warnings.warn("Disabling sequence parallelism because tensor model parallelism is disabled")
24
+ /workspace/Megatron-UltraEP/megatron/training/arguments.py:831: UserWarning: Disabling sequence parallelism because tensor model parallelism is disabled
25
+ warnings.warn("Disabling sequence parallelism because tensor model parallelism is disabled")
26
+ using world size: 8, data-parallel size: 8, context-parallel size: 1, hierarchical context-parallel sizes: None, tensor-model-parallel size: 1, pipeline-model-parallel size: 1
27
+ WARNING: overriding default arguments for tokenizer_type:GPT2BPETokenizer with tokenizer_type:GPT2BPETokenizer
28
+ Number of virtual stages per pipeline stage: None
29
+ accumulate and all-reduce gradients in fp32 for bfloat16 data type.
30
+ using torch.bfloat16 for parameters ...
31
+ /workspace/Megatron-UltraEP/megatron/training/arguments.py:831: UserWarning: Disabling sequence parallelism because tensor model parallelism is disabled
32
+ warnings.warn("Disabling sequence parallelism because tensor model parallelism is disabled")
33
+ ------------------------ arguments ------------------------
34
+ account_for_embedding_in_pipeline_split ......... False
35
+ account_for_loss_in_pipeline_split .............. False
36
+ accumulate_allreduce_grads_in_fp32 .............. True
37
+ adam_beta1 ...................................... 0.9
38
+ adam_beta2 ...................................... 0.999
39
+ adam_eps ........................................ 1e-08
40
+ add_bias_linear ................................. False
41
+ add_position_embedding .......................... False
42
+ add_qkv_bias .................................... False
43
+ adlr_autoresume ................................. False
44
+ adlr_autoresume_interval ........................ 1000
45
+ align_grad_reduce ............................... True
46
+ align_param_gather .............................. False
47
+ app_tag_run_name ................................ None
48
+ app_tag_run_version ............................. 0.0.0
49
+ apply_layernorm_1p .............................. False
50
+ apply_query_key_layer_scaling ................... False
51
+ apply_residual_connection_post_layernorm ........ False
52
+ apply_rope_fusion ............................... True
53
+ async_save ...................................... None
54
+ async_tensor_model_parallel_allreduce ........... True
55
+ attention_backend ............................... AttnBackend.auto
56
+ attention_dropout ............................... 0.0
57
+ attention_softmax_in_fp32 ....................... False
58
+ auto_detect_ckpt_format ......................... False
59
+ barrier_with_L1_time ............................ True
60
+ bert_binary_head ................................ True
61
+ bert_embedder_type .............................. megatron
62
+ bert_load ....................................... None
63
+ bf16 ............................................ True
64
+ bias_dropout_fusion ............................. True
65
+ bias_gelu_fusion ................................ False
66
+ bias_swiglu_fusion .............................. True
67
+ biencoder_projection_dim ........................ 0
68
+ biencoder_shared_query_context_model ............ False
69
+ block_data_path ................................. None
70
+ calc_ft_timeouts ................................ False
71
+ calculate_per_token_loss ........................ False
72
+ check_for_large_grads ........................... False
73
+ check_for_nan_in_loss_and_grad .................. False
74
+ check_for_spiky_loss ............................ False
75
+ check_weight_hash_across_dp_replicas_interval ... None
76
+ ckpt_assume_constant_structure .................. False
77
+ ckpt_convert_format ............................. None
78
+ ckpt_convert_save ............................... None
79
+ ckpt_convert_update_legacy_dist_opt_format ...... False
80
+ ckpt_format ..................................... torch_dist
81
+ ckpt_fully_parallel_load ........................ False
82
+ ckpt_fully_parallel_save ........................ True
83
+ ckpt_fully_parallel_save_deprecated ............. False
84
+ ckpt_step ....................................... None
85
+ classes_fraction ................................ 1.0
86
+ clip_grad ....................................... 1.0
87
+ clone_scatter_output_in_embedding ............... True
88
+ config_logger_dir ...............................
89
+ consumed_train_samples .......................... 0
90
+ consumed_valid_samples .......................... 0
91
+ context_parallel_size ........................... 1
92
+ cp_comm_type .................................... ['p2p']
93
+ create_attention_mask_in_dataloader ............. True
94
+ cross_entropy_fusion_impl ....................... native
95
+ cross_entropy_loss_fusion ....................... False
96
+ cuda_graph_scope ................................ full_iteration
97
+ cuda_graph_warmup_steps ......................... 3
98
+ data_args_path .................................. None
99
+ data_cache_path ................................. None
100
+ data_parallel_random_init ....................... False
101
+ data_parallel_sharding_strategy ................. no_shard
102
+ data_parallel_size .............................. 8
103
+ data_path ....................................... ['/workspace/data/wikitext/wikitext103_gpt2_text_document']
104
+ data_per_class_fraction ......................... 1.0
105
+ data_sharding ................................... True
106
+ dataloader_type ................................. single
107
+ ddp_average_in_collective ....................... False
108
+ ddp_bucket_size ................................. None
109
+ ddp_num_buckets ................................. None
110
+ ddp_pad_buckets_for_high_nccl_busbw ............. False
111
+ decoder_first_pipeline_num_layers ............... None
112
+ decoder_last_pipeline_num_layers ................ None
113
+ decoder_num_layers .............................. None
114
+ decoder_seq_length .............................. None
115
+ decoupled_lr .................................... None
116
+ decoupled_min_lr ................................ None
117
+ decrease_batch_size_if_needed ................... False
118
+ defer_embedding_wgrad_compute ................... False
119
+ delay_wgrad_compute ............................. False
120
+ deprecated_use_mcore_models ..................... True
121
+ deterministic_mode .............................. False
122
+ dino_bottleneck_size ............................ 256
123
+ dino_freeze_last_layer .......................... 1
124
+ dino_head_hidden_size ........................... 2048
125
+ dino_local_crops_number ......................... 10
126
+ dino_local_img_size ............................. 96
127
+ dino_norm_last_layer ............................ False
128
+ dino_teacher_temp ............................... 0.07
129
+ dino_warmup_teacher_temp ........................ 0.04
130
+ dino_warmup_teacher_temp_epochs ................. 30
131
+ disable_bf16_reduced_precision_matmul ........... False
132
+ disable_mamba_mem_eff_path ...................... False
133
+ disable_straggler_on_startup .................... False
134
+ dist_ckpt_format_deprecated ..................... None
135
+ dist_ckpt_strictness ............................ assume_ok_unexpected
136
+ distribute_saved_activations .................... False
137
+ distributed_backend ............................. nccl
138
+ distributed_timeout_minutes ..................... 10
139
+ embedding_init_method_std ....................... None
140
+ embedding_path .................................. None
141
+ empty_unused_memory_level ....................... 0
142
+ enable_cuda_graph ............................... True
143
+ enable_experimental ............................. False
144
+ enable_ft_package ............................... False
145
+ enable_gloo_process_groups ...................... True
146
+ enable_msc ...................................... True
147
+ enable_one_logger ............................... True
148
+ encoder_num_layers .............................. 16
149
+ encoder_seq_length .............................. 8192
150
+ end_weight_decay ................................ 0.1
151
+ eod_mask_loss ................................... False
152
+ error_injection_rate ............................ 0
153
+ error_injection_type ............................ transient_error
154
+ eval_interval ................................... 1000000
155
+ eval_iters ...................................... 0
156
+ evidence_data_path .............................. None
157
+ exit_duration_in_mins ........................... None
158
+ exit_interval ................................... None
159
+ exit_on_missing_checkpoint ...................... False
160
+ exit_signal_handler ............................. False
161
+ exp_avg_dtype ................................... torch.float32
162
+ exp_avg_sq_dtype ................................ torch.float32
163
+ expert_model_parallel_size ...................... 8
164
+ expert_tensor_parallel_size ..................... 1
165
+ export_force_local_attention .................... False
166
+ export_kd_cfg ................................... None
167
+ export_kd_teacher_ckpt_format ................... None
168
+ export_kd_teacher_load .......................... None
169
+ export_kv_cache_quant ........................... False
170
+ export_legacy_megatron .......................... False
171
+ export_model_type ............................... GPTModel
172
+ export_moe_apply_probs_on_input ................. False
173
+ export_qk_l2_norm ............................... False
174
+ export_quant_cfg ................................ None
175
+ export_real_quant_cfg ........................... None
176
+ export_te_mcore_model ........................... False
177
+ external_cuda_graph ............................. False
178
+ ffn_hidden_size ................................. 14336
179
+ finetune ........................................ False
180
+ finetune_data_split ............................. train
181
+ finetune_hf_dataset ............................. None
182
+ first_last_layers_bf16 .......................... False
183
+ flash_decode .................................... False
184
+ fp16 ............................................ False
185
+ fp16_lm_cross_entropy ........................... False
186
+ fp32_residual_connection ........................ False
187
+ fp8 ............................................. e4m3
188
+ fp8_amax_compute_algo ........................... most_recent
189
+ fp8_amax_history_len ............................ 1
190
+ fp8_interval .................................... 1
191
+ fp8_margin ...................................... 0
192
+ fp8_param_gather ................................ False
193
+ fp8_recipe ...................................... mxfp8
194
+ fp8_wgrad ....................................... True
195
+ fsdp_double_buffer .............................. False
196
+ global_batch_size ............................... 16
197
+ grad_reduce_in_bf16 ............................. False
198
+ gradient_accumulation_fusion .................... True
199
+ gradient_reduce_div_fusion ...................... True
200
+ group_query_attention ........................... True
201
+ head_lr_mult .................................... 1.0
202
+ heterogeneous_layers_config_encoded_json ........ None
203
+ heterogeneous_layers_config_path ................ None
204
+ hidden_dropout .................................. 0.0
205
+ hidden_size ..................................... 4096
206
+ hierarchical_context_parallel_sizes ............. None
207
+ high_priority_stream_groups ..................... []
208
+ hybrid_attention_ratio .......................... 0.0
209
+ hybrid_mlp_ratio ................................ 0.0
210
+ hybrid_override_pattern ......................... None
211
+ hysteresis ...................................... 2
212
+ ict_head_size ................................... None
213
+ ict_load ........................................ None
214
+ img_h ........................................... 224
215
+ img_w ........................................... 224
216
+ indexer_batch_size .............................. 128
217
+ indexer_log_interval ............................ 1000
218
+ inference_batch_times_seqlen_threshold .......... -1
219
+ inference_dynamic_batching ...................... False
220
+ inference_dynamic_batching_buffer_guaranteed_fraction 0.2
221
+ inference_dynamic_batching_buffer_overflow_factor None
222
+ inference_dynamic_batching_buffer_size_gb ....... 40.0
223
+ inference_dynamic_batching_chunk_size ........... 256
224
+ inference_dynamic_batching_max_requests_override None
225
+ inference_dynamic_batching_max_tokens_override .. None
226
+ inference_dynamic_batching_num_cuda_graphs ...... 16
227
+ inference_max_batch_size ........................ 8
228
+ inference_max_seq_length ........................ 2560
229
+ inference_rng_tracker ........................... False
230
+ init_method_std ................................. 0.01
231
+ init_method_xavier_uniform ...................... False
232
+ init_model_with_meta_device ..................... False
233
+ initial_loss_scale .............................. 4294967296
234
+ inprocess_active_world_size ..................... 8
235
+ inprocess_barrier_timeout ....................... 120
236
+ inprocess_completion_timeout .................... 120
237
+ inprocess_empty_cuda_cache ...................... False
238
+ inprocess_granularity ........................... node
239
+ inprocess_hard_timeout .......................... 90
240
+ inprocess_heartbeat_interval .................... 30
241
+ inprocess_heartbeat_timeout ..................... 60
242
+ inprocess_last_call_wait ........................ 1
243
+ inprocess_max_iterations ........................ None
244
+ inprocess_monitor_process_interval .............. 1.0
245
+ inprocess_monitor_thread_interval ............... 1.0
246
+ inprocess_progress_watchdog_interval ............ 1.0
247
+ inprocess_restart ............................... False
248
+ inprocess_soft_timeout .......................... 60
249
+ inprocess_termination_grace_time ................ 1
250
+ is_hybrid_model ................................. False
251
+ iter_per_epoch .................................. 1250
252
+ iterations_to_skip .............................. []
253
+ keep_fp8_transpose_cache_when_using_custom_fsdp . False
254
+ kitchen_config_file ............................. None
255
+ kitchen_recipe_number ........................... None
256
+ kv_channels ..................................... 128
257
+ kv_lora_rank .................................... 32
258
+ lazy_mpu_init ................................... None
259
+ load ............................................ None
260
+ load_main_params_from_ckpt ...................... None
261
+ load_model_opt_format ........................... False
262
+ local_rank ...................................... 0
263
+ log_energy ...................................... False
264
+ log_interval .................................... 1
265
+ log_loss_scale_to_tensorboard ................... True
266
+ log_memory_to_tensorboard ....................... False
267
+ log_num_zeros_in_grad ........................... False
268
+ log_params_norm ................................. False
269
+ log_progress .................................... False
270
+ log_straggler ................................... False
271
+ log_throughput .................................. True
272
+ log_timers_to_tensorboard ....................... False
273
+ log_validation_ppl_to_tensorboard ............... False
274
+ log_world_size_to_tensorboard ................... False
275
+ logging_level ................................... None
276
+ loss_scale ...................................... None
277
+ loss_scale_window ............................... 1000
278
+ lr .............................................. 0.0001
279
+ lr_decay_iters .................................. 8
280
+ lr_decay_samples ................................ None
281
+ lr_decay_style .................................. cosine
282
+ lr_warmup_fraction .............................. None
283
+ lr_warmup_init .................................. 0.0
284
+ lr_warmup_iters ................................. 5
285
+ lr_warmup_samples ............................... 0
286
+ lr_wsd_decay_iters .............................. None
287
+ lr_wsd_decay_samples ............................ None
288
+ lr_wsd_decay_style .............................. exponential
289
+ main_grads_dtype ................................ torch.float32
290
+ main_params_dtype ............................... torch.float32
291
+ make_vocab_size_divisible_by .................... 128
292
+ mamba_head_dim .................................. 64
293
+ mamba_num_groups ................................ 8
294
+ mamba_num_heads ................................. None
295
+ mamba_state_dim ................................. 128
296
+ manual_gc ....................................... False
297
+ manual_gc_eval .................................. True
298
+ manual_gc_interval .............................. 0
299
+ mask_factor ..................................... 1.0
300
+ mask_prob ....................................... 0.15
301
+ mask_type ....................................... random
302
+ masked_softmax_fusion ........................... False
303
+ max_position_embeddings ......................... 32768
304
+ max_tokens_to_oom ............................... 12000
305
+ memory_snapshot_path ............................ snapshot.pickle
306
+ merge_file ...................................... /workspace/data/wikitext/gpt2-merges.txt
307
+ micro_batch_size ................................ 1
308
+ microbatch_group_size_per_vp_stage .............. None
309
+ mid_level_dataset_surplus ....................... 0.005
310
+ min_loss_scale .................................. 1.0
311
+ min_lr .......................................... 1e-05
312
+ mlp_chunks_for_prefill .......................... 1
313
+ mmap_bin_files .................................. True
314
+ mock_data ....................................... False
315
+ moe_apply_probs_on_input ........................ False
316
+ moe_aux_loss_coeff .............................. 0.01
317
+ moe_capacity_monitor ............................ True
318
+ moe_deepep_num_sms .............................. 20
319
+ moe_echo_enable_random_offloading ............... False
320
+ moe_echo_expert_dispatch_overlap ................ False
321
+ moe_echo_expert_dispatcher_type ................. hybridep
322
+ moe_echo_recompute_expert_dispatch .............. False
323
+ moe_enable_deepep ............................... False
324
+ moe_enable_echo ................................. False
325
+ moe_enable_hybridep ............................. True
326
+ moe_enable_ultraep .............................. True
327
+ moe_expert_capacity_factor ...................... None
328
+ moe_extended_tp ................................. False
329
+ moe_ffn_hidden_size ............................. 1536
330
+ moe_grouped_gemm ................................ True
331
+ moe_hybridep_num_sms ............................ 16
332
+ moe_input_jitter_eps ............................ None
333
+ moe_layer_freq .................................. 1
334
+ moe_layer_recompute ............................. False
335
+ moe_num_echo_experts ............................ None
336
+ moe_num_redundant_experts_per_rank .............. 1
337
+ moe_pad_expert_input_to_capacity ................ False
338
+ moe_per_layer_logging ........................... False
339
+ moe_permute_fusion .............................. False
340
+ moe_received_token_capacity ..................... 2.40625
341
+ moe_router_bias_update_rate ..................... 0.001
342
+ moe_router_dtype ................................ fp32
343
+ moe_router_enable_expert_bias ................... False
344
+ moe_router_force_load_balancing ................. False
345
+ moe_router_group_topk ........................... None
346
+ moe_router_load_balancing_type .................. aux_loss
347
+ moe_router_num_groups ........................... None
348
+ moe_router_padding_for_fp8 ...................... False
349
+ moe_router_pre_softmax .......................... False
350
+ moe_router_score_function ....................... softmax
351
+ moe_router_topk ................................. 2
352
+ moe_router_topk_scaling_factor .................. None
353
+ moe_shared_expert_intermediate_size ............. None
354
+ moe_shared_expert_overlap ....................... False
355
+ moe_token_dispatcher_type ....................... flex
356
+ moe_token_drop_policy ........................... probs
357
+ moe_ultraep_max_microbatches .................... None
358
+ moe_ultraep_overlap_weight_sync ................. True
359
+ moe_ultraep_weight_sync_join .................... before_dispatch
360
+ moe_upcycling_granularity ....................... 1
361
+ moe_use_device_initiated_grouped_gemm ........... True
362
+ moe_use_legacy_grouped_gemm ..................... False
363
+ moe_use_upcycling ............................... False
364
+ moe_z_loss_coeff ................................ None
365
+ mrope_section ................................... None
366
+ mscale .......................................... 1.0
367
+ mscale_all_dim .................................. 0.0
368
+ mtp_loss_scaling_factor ......................... 0.1
369
+ mtp_num_layers .................................. None
370
+ multi_latent_attention .......................... False
371
+ nccl_all_reduce_for_prefill ..................... False
372
+ nccl_communicator_config_path ................... None
373
+ nccl_ub ......................................... False
374
+ no_load_optim ................................... None
375
+ no_load_rng ..................................... None
376
+ no_persist_layer_norm ........................... False
377
+ no_rope_freq .................................... None
378
+ no_save_optim ................................... None
379
+ no_save_rng ..................................... None
380
+ non_persistent_ckpt_type ........................ None
381
+ non_persistent_global_ckpt_dir .................. None
382
+ non_persistent_local_ckpt_algo .................. fully_parallel
383
+ non_persistent_local_ckpt_dir ................... None
384
+ non_persistent_save_interval .................... None
385
+ norm_epsilon .................................... 1e-05
386
+ normalization ................................... RMSNorm
387
+ num_attention_heads ............................. 32
388
+ num_channels .................................... 3
389
+ num_classes ..................................... 1000
390
+ num_dataset_builder_threads ..................... 1
391
+ num_distributed_optimizer_instances ............. 1
392
+ num_experts ..................................... 8
393
+ num_layers ...................................... 16
394
+ num_layers_at_end_in_bf16 ....................... 1
395
+ num_layers_at_start_in_bf16 ..................... 1
396
+ num_layers_per_virtual_pipeline_stage ........... None
397
+ num_query_groups ................................ 8
398
+ num_virtual_stages_per_pipeline_rank ............ None
399
+ num_workers ..................................... 2
400
+ object_storage_cache_path ....................... None
401
+ one_logger_async ................................ False
402
+ one_logger_project .............................. megatron-lm
403
+ one_logger_run_name ............................. None
404
+ onnx_safe ....................................... None
405
+ openai_gelu ..................................... False
406
+ optimizer ....................................... adam
407
+ optimizer_cpu_offload ........................... False
408
+ optimizer_offload_fraction ...................... 1.0
409
+ output_bert_embeddings .......................... False
410
+ overlap_cpu_optimizer_d2h_h2d ................... False
411
+ overlap_grad_reduce ............................. True
412
+ overlap_moe_expert_parallel_comm ................ False
413
+ overlap_p2p_comm ................................ False
414
+ overlap_p2p_comm_warmup_flush ................... False
415
+ overlap_param_gather ............................ True
416
+ overlap_param_gather_with_optimizer_step ........ False
417
+ override_opt_param_scheduler .................... False
418
+ params_dtype .................................... torch.bfloat16
419
+ patch_dim ....................................... 16
420
+ per_split_data_args_path ........................ None
421
+ perform_initialization .......................... True
422
+ pin_cpu_grads ................................... True
423
+ pin_cpu_params .................................. True
424
+ pipeline_model_parallel_comm_backend ............ None
425
+ pipeline_model_parallel_layout .................. None
426
+ pipeline_model_parallel_size .................... 1
427
+ position_embedding_type ......................... rope
428
+ pretrained_checkpoint ........................... None
429
+ profile ......................................... False
430
+ profile_ranks ................................... [0]
431
+ profile_step_end ................................ 12
432
+ profile_step_start .............................. 10
433
+ q_lora_rank ..................................... None
434
+ qk_head_dim ..................................... 128
435
+ qk_l2_norm ...................................... False
436
+ qk_layernorm .................................... False
437
+ qk_pos_emb_head_dim ............................. 64
438
+ query_in_block_prob ............................. 0.1
439
+ rampup_batch_size ............................... None
440
+ rank ............................................ 0
441
+ recompute_granularity ........................... None
442
+ recompute_method ................................ None
443
+ recompute_modules ............................... None
444
+ recompute_num_layers ............................ None
445
+ record_memory_history ........................... False
446
+ relative_attention_max_distance ................. 128
447
+ relative_attention_num_buckets .................. 32
448
+ replication ..................................... False
449
+ replication_factor .............................. 2
450
+ replication_jump ................................ None
451
+ rerun_mode ...................................... validate_results
452
+ reset_attention_mask ............................ False
453
+ reset_position_ids .............................. False
454
+ result_rejected_tracker_filename ................ None
455
+ retriever_report_topk_accuracies ................ []
456
+ retriever_score_scaling ......................... False
457
+ retriever_seq_length ............................ 256
458
+ retro_add_retriever ............................. False
459
+ retro_attention_gate ............................ 1
460
+ retro_cyclic_train_iters ........................ None
461
+ retro_encoder_attention_dropout ................. 0.1
462
+ retro_encoder_hidden_dropout .................... 0.1
463
+ retro_encoder_layers ............................ 2
464
+ retro_num_neighbors ............................. 2
465
+ retro_num_retrieved_chunks ...................... 2
466
+ retro_project_dir ............................... None
467
+ retro_verify_neighbor_count ..................... True
468
+ reuse_grad_buf_for_mxfp8_param_ag ............... False
469
+ rope_scaling_factor ............................. 8.0
470
+ rope_type ....................................... None
471
+ rotary_base ..................................... 1000000
472
+ rotary_interleaved .............................. False
473
+ rotary_percent .................................. 1.0
474
+ rotary_scaling_factor ........................... 1.0
475
+ rotary_seq_len_interpolation_factor ............. None
476
+ run_workload_inspector_server ................... False
477
+ sample_rate ..................................... 1.0
478
+ save ............................................ None
479
+ save_interval ................................... None
480
+ save_retain_interval ............................ None
481
+ scatter_gather_tensors_in_pipeline .............. True
482
+ seed ............................................ 1234
483
+ seq_length ...................................... 8192
484
+ sequence_parallel ............................... False
485
+ sft ............................................. False
486
+ sft_tokenizer_prompt_format ..................... nemotron-h-aligned
487
+ sgd_momentum .................................... 0.9
488
+ short_seq_prob .................................. 0.1
489
+ skip_train ...................................... False
490
+ skipped_train_samples ........................... 0
491
+ spec ............................................ None
492
+ split ........................................... 100,0,0
493
+ squared_relu .................................... False
494
+ start_weight_decay .............................. 0.1
495
+ straggler_ctrlr_port ............................ 65535
496
+ straggler_minmax_count .......................... 1
497
+ suggested_communication_unit_size ............... None
498
+ swiglu .......................................... True
499
+ swin_backbone_type .............................. tiny
500
+ symmetric_ar_type ............................... None
501
+ te_rng_tracker .................................. True
502
+ tensor_model_parallel_size ...................... 1
503
+ tensorboard_dir ................................. None
504
+ tensorboard_log_interval ........................ 1
505
+ tensorboard_queue_size .......................... 1000
506
+ test_data_path .................................. None
507
+ test_mode ....................................... False
508
+ tiktoken_num_special_tokens ..................... 1000
509
+ tiktoken_pattern ................................ None
510
+ tiktoken_special_tokens ......................... None
511
+ timing_log_level ................................ 0
512
+ timing_log_option ............................... minmax
513
+ titles_data_path ................................ None
514
+ tokenizer_model ................................. None
515
+ tokenizer_type .................................. GPT2BPETokenizer
516
+ torch_fsdp2_reshard_after_forward ............... True
517
+ tp_comm_bootstrap_backend ....................... nccl
518
+ tp_comm_bulk_dgrad .............................. True
519
+ tp_comm_bulk_wgrad .............................. True
520
+ tp_comm_overlap ................................. False
521
+ tp_comm_overlap_ag .............................. True
522
+ tp_comm_overlap_cfg ............................. None
523
+ tp_comm_overlap_rs .............................. True
524
+ tp_comm_overlap_rs_dgrad ........................ False
525
+ tp_comm_split_ag ................................ True
526
+ tp_comm_split_rs ................................ True
527
+ train_data_path ................................. None
528
+ train_iters ..................................... 12
529
+ train_samples ................................... None
530
+ train_sync_interval ............................. None
531
+ transformer_impl ................................ transformer_engine
532
+ transformer_pipeline_model_parallel_size ........ 1
533
+ untie_embeddings_and_output_weights ............. True
534
+ use_checkpoint_args ............................. False
535
+ use_checkpoint_opt_param_scheduler .............. False
536
+ use_cpu_initialization .......................... None
537
+ use_custom_fsdp ................................. False
538
+ use_dist_ckpt ................................... True
539
+ use_dist_ckpt_deprecated ........................ False
540
+ use_distributed_optimizer ....................... True
541
+ use_flash_attn .................................. False
542
+ use_legacy_models ............................... False
543
+ use_mp_args_from_checkpoint_args ................ False
544
+ use_one_sent_docs ............................... False
545
+ use_persistent_ckpt_worker ...................... False
546
+ use_precision_aware_optimizer ................... False
547
+ use_pytorch_profiler ............................ False
548
+ use_ring_exchange_p2p ........................... False
549
+ use_rope_scaling ................................ False
550
+ use_rotary_position_embeddings .................. False
551
+ use_sharp ....................................... False
552
+ use_tokenizer_model_from_checkpoint_args ........ True
553
+ use_torch_fsdp2 ................................. False
554
+ use_torch_optimizer_for_cpu_offload ............. False
555
+ use_tp_pp_dp_mapping ............................ False
556
+ v_head_dim ...................................... 128
557
+ valid_data_path ................................. None
558
+ variable_seq_lengths ............................ False
559
+ virtual_pipeline_model_parallel_size ............ None
560
+ vision_backbone_type ............................ vit
561
+ vision_pretraining .............................. False
562
+ vision_pretraining_type ......................... classify
563
+ vocab_extra_ids ................................. 0
564
+ vocab_file ...................................... /workspace/data/wikitext/gpt2-vocab.json
565
+ vocab_size ...................................... None
566
+ wandb_exp_name ..................................
567
+ wandb_project ...................................
568
+ wandb_save_dir ..................................
569
+ weight_decay .................................... 0.1
570
+ weight_decay_incr_style ......................... constant
571
+ wgrad_deferral_limit ............................ 0
572
+ world_size ...................................... 8
573
+ yaml_cfg ........................................ None
574
+ -------------------- end of arguments ---------------------
575
+ INFO:megatron.core.num_microbatches_calculator:setting number of microbatches to constant 2
576
+ > building GPT2BPETokenizer tokenizer ...
577
+ /workspace/Megatron-UltraEP/megatron/training/arguments.py:831: UserWarning: Disabling sequence parallelism because tensor model parallelism is disabled
578
+ warnings.warn("Disabling sequence parallelism because tensor model parallelism is disabled")
579
+ > padded vocab (size: 50257) with 47 dummy tokens (new size: 50304)
580
+ WARNING:megatron.core.rerun_state_machine:RerunStateMachine initialized in mode RerunMode.VALIDATE_RESULTS
581
+ > initializing torch distributed ...
582
+ [Gloo] Rank [Gloo] Rank 2 is connected to 07 is connected to peer ranks. 7Expected number of connected peer ranks is : peer ranks. 7Expected number of connected peer ranks is : 7
583
+
584
+ [Gloo] Rank 5 is connected to 7 peer ranks. Expected number of connected peer ranks is : 7
585
+ [Gloo] Rank 4 is connected to 7 peer ranks. Expected number of connected peer ranks is : [Gloo] Rank 7
586
+ 1 is connected to 7 peer ranks. Expected number of connected peer ranks is : 7
587
+ [Gloo] Rank 3 is connected to 7 peer ranks. Expected number of connected peer ranks is : 7
588
+ [Gloo] Rank 6 is connected to 7 peer ranks. Expected number of connected peer ranks is : 7
589
+ [Gloo] Rank 7 is connected to 7 peer ranks. Expected number of connected peer ranks is : 7
590
+ [Gloo] Rank 0 is connected to 7 peer ranks. Expected number of connected peer ranks is : 7
591
+ [Gloo] Rank 1 is connected to 7 peer ranks. Expected number of connected peer ranks is : 7
592
+ [Gloo] Rank [Gloo] Rank [Gloo] Rank [Gloo] Rank [Gloo] Rank [Gloo] Rank 324576 is connected to is connected to is connected to is connected to is connected to is connected to 777777 peer ranks. peer ranks. peer ranks. peer ranks. peer ranks. peer ranks. Expected number of connected peer ranks is : Expected number of connected peer ranks is : Expected number of connected peer ranks is : Expected number of connected peer ranks is : Expected number of connected peer ranks is : Expected number of connected peer ranks is : 77777
593
+ 7
594
+
595
+
596
+
597
+
598
+ [Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
599
+ > initialized tensor model parallel with size 1
600
+ > initialized pipeline model parallel with size 1
601
+ > setting random seeds to 1234 ...
602
+ [Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
603
+ [Gloo] Rank [Gloo] Rank 00 is connected to is connected to 00 peer ranks. peer ranks. Expected number of connected peer ranks is : Expected number of connected peer ranks is : 00
604
+
605
+ [Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
606
+ [Gloo] Rank [Gloo] Rank 00 is connected to is connected to [Gloo] Rank 000 peer ranks. peer ranks. is connected to Expected number of connected peer ranks is : Expected number of connected peer ranks is : 000
607
+ peer ranks.
608
+ Expected number of connected peer ranks is : 0
609
+ [rank1]:[W830 07:21:15.733562259 ProcessGroupNCCL.cpp:4789] [PG ID 0 PG GUID 0 Rank 1] using GPU 1 as device used by this process is currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect. You can specify device_id in init_process_group() to force use of a particular device.
610
+ [rank5]:[W830 07:21:15.742400969 ProcessGroupNCCL.cpp:4789] [PG ID 0 PG GUID 0 Rank 5] using GPU 5 as device used by this process is currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect. You can specify device_id in init_process_group() to force use of a particular device.
611
+ [rank2]:[W830 07:21:15.742431416 ProcessGroupNCCL.cpp:4789] [PG ID 0 PG GUID 0 Rank 2] using GPU 2 as device used by this process is currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect. You can specify device_id in init_process_group() to force use of a particular device.
612
+ [rank4]:[W830 07:21:15.742441721 ProcessGroupNCCL.cpp:4789] [PG ID 0 PG GUID 0 Rank 4] using GPU 4 as device used by this process is currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect. You can specify device_id in init_process_group() to force use of a particular device.
613
+ [rank7]:[W830 07:21:15.742489500 ProcessGroupNCCL.cpp:4789] [PG ID 0 PG GUID 0 Rank 7] using GPU 7 as device used by this process is currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect. You can specify device_id in init_process_group() to force use of a particular device.
614
+ [rank3]:[W830 07:21:15.742508197 ProcessGroupNCCL.cpp:4789] [PG ID 0 PG GUID 0 Rank 3] using GPU 3 as device used by this process is currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect. You can specify device_id in init_process_group() to force use of a particular device.
615
+ [rank6]:[W830 07:21:15.742530488 ProcessGroupNCCL.cpp:4789] [PG ID 0 PG GUID 0 Rank 6] using GPU 6 as device used by this process is currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect. You can specify device_id in init_process_group() to force use of a particular device.
616
+ > compiling dataset index builder ...
617
+ make: Entering directory '/workspace/Megatron-UltraEP/megatron/core/datasets'
618
+ make: Nothing to be done for 'default'.
619
+ make: Leaving directory '/workspace/Megatron-UltraEP/megatron/core/datasets'
620
+ >>> done with dataset index builder. Compilation time: 0.035 seconds
621
+ WARNING: constraints for invoking optimized fused softmax kernel are not met. We default back to unfused kernel invocations.
622
+ > compiling and loading fused kernels ...
623
+ [rank0]:[W830 07:21:15.929293731 ProcessGroupNCCL.cpp:4789] [PG ID 0 PG GUID 0 Rank 0] using GPU 0 as device used by this process is currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect. You can specify device_id in init_process_group() to force use of a particular device.
624
+ >>> done with compiling and loading fused kernels. Compilation time: 3.978 seconds
625
+ time to initialize megatron (seconds): 7.314
626
+ [after megatron is initialized] datetime: 2026-08-30 07:21:21
627
+ building GPT model ...
628
+ /usr/local/lib/python3.12/dist-packages/transformer_engine/pytorch/cpu_offload.py:708: DeprecationWarning: Offloading weights is deprecated. Using offload_weights=True does not have any effect.
629
+ warnings.warn(
630
+ /usr/local/lib/python3.12/dist-packages/transformer_engine/pytorch/cpu_offload.py:708: DeprecationWarning: Offloading weights is deprecated. Using offload_weights=True does not have any effect.
631
+ warnings.warn(
632
+ /usr/local/lib/python3.12/dist-packages/transformer_engine/pytorch/cpu_offload.py:708: DeprecationWarning: Offloading weights is deprecated. Using offload_weights=True does not have any effect.
633
+ warnings.warn(
634
+ /usr/local/lib/python3.12/dist-packages/transformer_engine/pytorch/cpu_offload.py:708: DeprecationWarning: Offloading weights is deprecated. Using offload_weights=True does not have any effect.
635
+ warnings.warn(
636
+ /usr/local/lib/python3.12/dist-packages/transformer_engine/pytorch/cpu_offload.py:708: DeprecationWarning: Offloading weights is deprecated. Using offload_weights=True does not have any effect.
637
+ warnings.warn(
638
+ /usr/local/lib/python3.12/dist-packages/transformer_engine/pytorch/cpu_offload.py:708: DeprecationWarning: Offloading weights is deprecated. Using offload_weights=True does not have any effect.
639
+ warnings.warn(
640
+ /usr/local/lib/python3.12/dist-packages/transformer_engine/pytorch/cpu_offload.py:708: DeprecationWarning: Offloading weights is deprecated. Using offload_weights=True does not have any effect.
641
+ warnings.warn(
642
+ /usr/local/lib/python3.12/dist-packages/transformer_engine/pytorch/cpu_offload.py:708: DeprecationWarning: Offloading weights is deprecated. Using offload_weights=True does not have any effect.
643
+ warnings.warn(
644
+ [INFO] Use MNNVL fabric: False
645
+ [WARN] MAX_NUM_NVL_PEERS is not set. Using detected value 8.
646
+ NVSHMEM v3.7.2
647
+ /dvs/p4/build/sw/rel/gpgpu/toolkit/r12.9/main_nvshmem/src/modules/transport/ibrc/ibrc.cpp:1754: error status: 7 (NVSHMEMX_ERROR_INTERNAL) Device enumeration failed.
648
+
649
+ /dvs/p4/build/sw/rel/gpgpu/toolkit/r12.9/main_nvshmem/src/modules/transport/ibrc/ibrc.cpp:1754: error status: 7 (NVSHMEMX_ERROR_INTERNAL) Device enumeration failed.
650
+
651
+ /dvs/p4/build/sw/rel/gpgpu/toolkit/r12.9/main_nvshmem/src/modules/transport/ibrc/ibrc.cpp:1754: error status: 7 (NVSHMEMX_ERROR_INTERNAL) Device enumeration failed.
652
+
653
+ /dvs/p4/build/sw/rel/gpgpu/toolkit/r12.9/main_nvshmem/src/modules/transport/ibrc/ibrc.cpp:1754: error status: 7 (NVSHMEMX_ERROR_INTERNAL) Device enumeration failed.
654
+
655
+ /dvs/p4/build/sw/rel/gpgpu/toolkit/r12.9/main_nvshmem/src/modules/transport/ibrc/ibrc.cpp:1754: error status: 7 (NVSHMEMX_ERROR_INTERNAL) Device enumeration failed.
656
+
657
+ /dvs/p4/build/sw/rel/gpgpu/toolkit/r12.9/main_nvshmem/src/modules/transport/ibrc/ibrc.cpp:1754: error status: 7 (NVSHMEMX_ERROR_INTERNAL) Device enumeration failed.
658
+
659
+ /dvs/p4/build/sw/rel/gpgpu/toolkit/r12.9/main_nvshmem/src/modules/transport/ibrc/ibrc.cpp:1754: error status: 7 (NVSHMEMX_ERROR_INTERNAL) Device enumeration failed.
660
+
661
+ /dvs/p4/build/sw/rel/gpgpu/toolkit/r12.9/main_nvshmem/src/modules/transport/ibrc/ibrc.cpp:1754: error status: 7 (NVSHMEMX_ERROR_INTERNAL) Device enumeration failed.
662
+
663
+ UltraEP: NVL domain size <= 8; forcing weight_sync plan mode to direct (was adaptive).
664
+ > number of parameters on (tensor, pipeline) model parallel rank (0, 0): 1687818240
665
+ INFO:megatron.core.distributed.distributed_data_parallel:Setting up DistributedDataParallel with config DistributedDataParallelConfig(grad_reduce_in_fp32=True, overlap_grad_reduce=True, overlap_param_gather=True, align_param_gather=False, use_distributed_optimizer=True, num_distributed_optimizer_instances=1, check_for_nan_in_grad=False, check_for_large_grads=False, bucket_size=40000000, pad_buckets_for_high_nccl_busbw=False, average_in_collective=False, fp8_param_gather=False, reuse_grad_buf_for_mxfp8_param_ag=False, use_custom_fsdp=False, data_parallel_sharding_strategy='no_shard', gradient_reduce_div_fusion=True, suggested_communication_unit_size=None, preserve_fp32_weights=True, keep_fp8_transpose_cache_when_using_custom_fsdp=False, nccl_ub=False, fsdp_double_buffer=False)
666
+ INFO:megatron.core.distributed.param_and_grad_buffer:Number of buckets for gradient all-reduce / reduce-scatter: 18
667
+ Params for bucket 1 (206045184 elements, 206045184 padded size):
668
+ module.output_layer.weight
669
+ Params for bucket 2 (41988096 elements, 41988096 padded size):
670
+ module.decoder.layers.15.pre_mlp_layernorm.weight
671
+ module.decoder.layers.15.self_attention.linear_qkv.weight
672
+ module.decoder.final_layernorm.weight
673
+ module.decoder.layers.15.self_attention.linear_qkv.layer_norm_weight
674
+ module.decoder.layers.15.mlp.router.weight
675
+ module.decoder.layers.15.self_attention.linear_proj.weight
676
+ Params for bucket 3 (41984000 elements, 41984000 padded size):
677
+ module.decoder.layers.14.mlp.router.weight
678
+ module.decoder.layers.14.self_attention.linear_proj.weight
679
+ module.decoder.layers.14.self_attention.linear_qkv.layer_norm_weight
680
+ module.decoder.layers.14.self_attention.linear_qkv.weight
681
+ module.decoder.layers.14.pre_mlp_layernorm.weight
682
+ Params for bucket 4 (41984000 elements, 41984000 padded size):
683
+ module.decoder.layers.13.self_attention.linear_qkv.weight
684
+ module.decoder.layers.13.pre_mlp_layernorm.weight
685
+ module.decoder.layers.13.mlp.router.weight
686
+ module.decoder.layers.13.self_attention.linear_qkv.layer_norm_weight
687
+ module.decoder.layers.13.self_attention.linear_proj.weight
688
+ Params for bucket 5 (41984000 elements, 41984000 padded size):
689
+ module.decoder.layers.12.self_attention.linear_qkv.weight
690
+ module.decoder.layers.12.pre_mlp_layernorm.weight
691
+ module.decoder.layers.12.mlp.router.weight
692
+ module.decoder.layers.12.self_attention.linear_proj.weight
693
+ module.decoder.layers.12.self_attention.linear_qkv.layer_norm_weight
694
+ Params for bucket 6 (41984000 elements, 41984000 padded size):
695
+ module.decoder.layers.11.self_attention.linear_qkv.weight
696
+ module.decoder.layers.11.pre_mlp_layernorm.weight
697
+ module.decoder.layers.11.mlp.router.weight
698
+ module.decoder.layers.11.self_attention.linear_proj.weight
699
+ module.decoder.layers.11.self_attention.linear_qkv.layer_norm_weight
700
+ Params for bucket 7 (41984000 elements, 41984000 padded size):
701
+ module.decoder.layers.10.self_attention.linear_qkv.weight
702
+ module.decoder.layers.10.pre_mlp_layernorm.weight
703
+ module.decoder.layers.10.mlp.router.weight
704
+ module.decoder.layers.10.self_attention.linear_proj.weight
705
+ module.decoder.layers.10.self_attention.linear_qkv.layer_norm_weight
706
+ Params for bucket 8 (41984000 elements, 41984000 padded size):
707
+ module.decoder.layers.9.mlp.router.weight
708
+ module.decoder.layers.9.self_attention.linear_proj.weight
709
+ module.decoder.layers.9.self_attention.linear_qkv.layer_norm_weight
710
+ module.decoder.layers.9.self_attention.linear_qkv.weight
711
+ module.decoder.layers.9.pre_mlp_layernorm.weight
712
+ Params for bucket 9 (41984000 elements, 41984000 padded size):
713
+ module.decoder.layers.8.mlp.router.weight
714
+ module.decoder.layers.8.self_attention.linear_proj.weight
715
+ module.decoder.layers.8.self_attention.linear_qkv.layer_norm_weight
716
+ module.decoder.layers.8.pre_mlp_layernorm.weight
717
+ module.decoder.layers.8.self_attention.linear_qkv.weight
718
+ Params for bucket 10 (41984000 elements, 41984000 padded size):
719
+ module.decoder.layers.7.mlp.router.weight
720
+ module.decoder.layers.7.self_attention.linear_proj.weight
721
+ module.decoder.layers.7.self_attention.linear_qkv.layer_norm_weight
722
+ module.decoder.layers.7.self_attention.linear_qkv.weight
723
+ module.decoder.layers.7.pre_mlp_layernorm.weight
724
+ Params for bucket 11 (41984000 elements, 41984000 padded size):
725
+ module.decoder.layers.6.mlp.router.weight
726
+ module.decoder.layers.6.self_attention.linear_proj.weight
727
+ module.decoder.layers.6.self_attention.linear_qkv.layer_norm_weight
728
+ module.decoder.layers.6.pre_mlp_layernorm.weight
729
+ module.decoder.layers.6.self_attention.linear_qkv.weight
730
+ Params for bucket 12 (41984000 elements, 41984000 padded size):
731
+ module.decoder.layers.5.self_attention.linear_qkv.weight
732
+ module.decoder.layers.5.self_attention.linear_proj.weight
733
+ module.decoder.layers.5.pre_mlp_layernorm.weight
734
+ module.decoder.layers.5.mlp.router.weight
735
+ module.decoder.layers.5.self_attention.linear_qkv.layer_norm_weight
736
+ Params for bucket 13 (41984000 elements, 41984000 padded size):
737
+ module.decoder.layers.4.mlp.router.weight
738
+ module.decoder.layers.4.self_attention.linear_qkv.layer_norm_weight
739
+ module.decoder.layers.4.self_attention.linear_proj.weight
740
+ module.decoder.layers.4.self_attention.linear_qkv.weight
741
+ module.decoder.layers.4.pre_mlp_layernorm.weight
742
+ Params for bucket 14 (41984000 elements, 41984000 padded size):
743
+ module.decoder.layers.3.self_attention.linear_qkv.weight
744
+ module.decoder.layers.3.pre_mlp_layernorm.weight
745
+ module.decoder.layers.3.mlp.router.weight
746
+ module.decoder.layers.3.self_attention.linear_qkv.layer_norm_weight
747
+ module.decoder.layers.3.self_attention.linear_proj.weight
748
+ Params for bucket 15 (41984000 elements, 41984000 padded size):
749
+ module.decoder.layers.2.pre_mlp_layernorm.weight
750
+ module.decoder.layers.2.self_attention.linear_qkv.weight
751
+ module.decoder.layers.2.mlp.router.weight
752
+ module.decoder.layers.2.self_attention.linear_proj.weight
753
+ module.decoder.layers.2.self_attention.linear_qkv.layer_norm_weight
754
+ Params for bucket 16 (41984000 elements, 41984000 padded size):
755
+ module.decoder.layers.1.mlp.router.weight
756
+ module.decoder.layers.1.self_attention.linear_qkv.layer_norm_weight
757
+ module.decoder.layers.1.self_attention.linear_qkv.weight
758
+ module.decoder.layers.1.self_attention.linear_proj.weight
759
+ module.decoder.layers.1.pre_mlp_layernorm.weight
760
+ Params for bucket 17 (41984000 elements, 41984000 padded size):
761
+ module.decoder.layers.0.mlp.router.weight
762
+ module.decoder.layers.0.self_attention.linear_qkv.layer_norm_weight
763
+ module.decoder.layers.0.self_attention.linear_qkv.weight
764
+ module.decoder.layers.0.pre_mlp_layernorm.weight
765
+ module.decoder.layers.0.self_attention.linear_proj.weight
766
+ Params for bucket 18 (206045184 elements, 206045184 padded size):
767
+ module.embedding.word_embeddings.weight
768
+ INFO:megatron.core.distributed.param_and_grad_buffer:Number of buckets for gradient all-reduce / reduce-scatter: 7
769
+ Params for bucket 1 (44040192 elements, 44040192 padded size):
770
+ module.decoder.layers.15.mlp.experts.linear_fc2.weight0
771
+ module.decoder.layers.14.mlp.experts.linear_fc1.weight0
772
+ module.decoder.layers.13.mlp.experts.linear_fc2.weight0
773
+ module.decoder.layers.15.mlp.experts.linear_fc1.weight0
774
+ module.decoder.layers.14.mlp.experts.linear_fc2.weight0
775
+ Params for bucket 2 (50331648 elements, 50331648 padded size):
776
+ module.decoder.layers.12.mlp.experts.linear_fc2.weight0
777
+ module.decoder.layers.11.mlp.experts.linear_fc2.weight0
778
+ module.decoder.layers.13.mlp.experts.linear_fc1.weight0
779
+ module.decoder.layers.11.mlp.experts.linear_fc1.weight0
780
+ module.decoder.layers.12.mlp.experts.linear_fc1.weight0
781
+ Params for bucket 3 (44040192 elements, 44040192 padded size):
782
+ module.decoder.layers.10.mlp.experts.linear_fc2.weight0
783
+ module.decoder.layers.9.mlp.experts.linear_fc1.weight0
784
+ module.decoder.layers.9.mlp.experts.linear_fc2.weight0
785
+ module.decoder.layers.8.mlp.experts.linear_fc2.weight0
786
+ module.decoder.layers.10.mlp.experts.linear_fc1.weight0
787
+ Params for bucket 4 (50331648 elements, 50331648 padded size):
788
+ module.decoder.layers.8.mlp.experts.linear_fc1.weight0
789
+ module.decoder.layers.6.mlp.experts.linear_fc1.weight0
790
+ module.decoder.layers.7.mlp.experts.linear_fc1.weight0
791
+ module.decoder.layers.7.mlp.experts.linear_fc2.weight0
792
+ module.decoder.layers.6.mlp.experts.linear_fc2.weight0
793
+ Params for bucket 5 (44040192 elements, 44040192 padded size):
794
+ module.decoder.layers.5.mlp.experts.linear_fc1.weight0
795
+ module.decoder.layers.4.mlp.experts.linear_fc1.weight0
796
+ module.decoder.layers.3.mlp.experts.linear_fc2.weight0
797
+ module.decoder.layers.5.mlp.experts.linear_fc2.weight0
798
+ module.decoder.layers.4.mlp.experts.linear_fc2.weight0
799
+ Params for bucket 6 (50331648 elements, 50331648 padded size):
800
+ module.decoder.layers.1.mlp.experts.linear_fc1.weight0
801
+ module.decoder.layers.2.mlp.experts.linear_fc2.weight0
802
+ module.decoder.layers.1.mlp.experts.linear_fc2.weight0
803
+ module.decoder.layers.3.mlp.experts.linear_fc1.weight0
804
+ module.decoder.layers.2.mlp.experts.linear_fc1.weight0
805
+ Params for bucket 7 (18874368 elements, 18874368 padded size):
806
+ module.decoder.layers.0.mlp.experts.linear_fc2.weight0
807
+ module.decoder.layers.0.mlp.experts.linear_fc1.weight0
808
+ INFO:megatron.core.optimizer:Setting up optimizer with config OptimizerConfig(optimizer='adam', lr=0.0001, min_lr=1e-05, decoupled_lr=None, decoupled_min_lr=None, weight_decay=0.1, fp8_recipe='mxfp8', fp16=False, bf16=True, reuse_grad_buf_for_mxfp8_param_ag=False, params_dtype=torch.bfloat16, use_precision_aware_optimizer=False, store_param_remainders=True, main_grads_dtype=torch.float32, main_params_dtype=torch.float32, exp_avg_dtype=torch.float32, exp_avg_sq_dtype=torch.float32, loss_scale=None, initial_loss_scale=4294967296, min_loss_scale=1.0, loss_scale_window=1000, hysteresis=2, adam_beta1=0.9, adam_beta2=0.999, adam_eps=1e-08, sgd_momentum=0.9, use_distributed_optimizer=True, overlap_param_gather_with_optimizer_step=False, optimizer_cpu_offload=False, optimizer_offload_fraction=1.0, use_torch_optimizer_for_cpu_offload=False, overlap_cpu_optimizer_d2h_h2d=False, pin_cpu_grads=True, pin_cpu_params=True, clip_grad=1.0, log_num_zeros_in_grad=False, barrier_with_L1_time=True, timers=<megatron.core.timers.Timers object at 0x712f25374ec0>, config_logger_dir='')
809
+ INFO:megatron.core.optimizer_param_scheduler:> learning rate decay style: cosine
810
+ [after model, optimizer, and learning rate scheduler are built] datetime: 2026-08-30 07:21:25
811
+ > building train, validation, and test datasets ...
812
+ > datasets target sizes (minimum size):
813
+ train: 192
814
+ validation: 0
815
+ test: 0
816
+ INFO:megatron.core.datasets.blended_megatron_dataset_config:Let split_matrix = [(0, 1.0), None, None]
817
+ > building train, validation, and test datasets for GPT ...
818
+ INFO:megatron.core.datasets.blended_megatron_dataset_builder:Building GPTDataset splits with sizes=(192, 0, 0) and config=GPTDatasetConfig(random_seed=1234, sequence_length=8192, blend=(['/workspace/data/wikitext/wikitext103_gpt2_text_document'], None), blend_per_split=None, split='100,0,0', split_matrix=[(0, 1.0), None, None], num_dataset_builder_threads=1, path_to_cache=None, mmap_bin_files=True, mock=False, tokenizer=<megatron.training.tokenizer.tokenizer._GPT2BPETokenizer object at 0x712f254fbe90>, mid_level_dataset_surplus=0.005, reset_position_ids=False, reset_attention_mask=False, eod_mask_loss=False, create_attention_mask=True, drop_last_partial_validation_sequence=True, add_extra_token_to_sequence=True, object_storage_cache_path=None)
819
+ INFO:megatron.core.datasets.indexed_dataset:Load the _IndexReader from /workspace/data/wikitext/wikitext103_gpt2_text_document.idx
820
+ INFO:megatron.core.datasets.indexed_dataset: Extract the sequence lengths
821
+ INFO:megatron.core.datasets.indexed_dataset: Extract the sequence pointers
822
+ INFO:megatron.core.datasets.indexed_dataset: Extract the document indices
823
+ INFO:megatron.core.datasets.indexed_dataset:> total number of sequences: 1165029
824
+ INFO:megatron.core.datasets.indexed_dataset:> total number of documents: 1801350
825
+ INFO:megatron.core.datasets.gpt_dataset:Load the GPTDataset train indices
826
+ INFO:megatron.core.datasets.gpt_dataset: Load the document index from b16a908ae7f33ed67d5ec87f6f633afe-GPTDataset-train-document_index.npy
827
+ INFO:megatron.core.datasets.gpt_dataset: Load the sample index from b16a908ae7f33ed67d5ec87f6f633afe-GPTDataset-train-sample_index.npy
828
+ INFO:megatron.core.datasets.gpt_dataset: Load the shuffle index from b16a908ae7f33ed67d5ec87f6f633afe-GPTDataset-train-shuffle_index.npy
829
+ INFO:megatron.core.datasets.gpt_dataset:> total number of samples: 14536
830
+ > finished creating GPT datasets ...
831
+ [after dataloaders are built] datetime: 2026-08-30 07:21:25
832
+ done with setup ...
833
+ (min, max) time across ranks (ms):
834
+ model-and-optimizer-setup ......................: (3397.04, 3397.25)
835
+ train/valid/test-data-iterators-setup ..........: (71.49, 91.87)
836
+ training ...
837
+ Overwriting rerun_state_machine.current_iteration from -1 to 0...
838
+ [before the start of training step] datetime: 2026-08-30 07:21:25
839
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:334: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
840
+ warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
841
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:334: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
842
+ warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
843
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:334: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
844
+ warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
845
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:334: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
846
+ warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
847
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:336: UserWarning: The use_host_meta is deprecated, it will be removed in the future.
848
+ warnings.warn("The use_host_meta is deprecated, it will be removed in the future.")
849
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:336: UserWarning: The use_host_meta is deprecated, it will be removed in the future.
850
+ warnings.warn("The use_host_meta is deprecated, it will be removed in the future.")
851
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:334: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
852
+ warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
853
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:334: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
854
+ warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
855
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:336: UserWarning: The use_host_meta is deprecated, it will be removed in the future.
856
+ warnings.warn("The use_host_meta is deprecated, it will be removed in the future.")
857
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:336: UserWarning: The use_host_meta is deprecated, it will be removed in the future.
858
+ warnings.warn("The use_host_meta is deprecated, it will be removed in the future.")
859
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:334: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
860
+ warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
861
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:336: UserWarning: The use_host_meta is deprecated, it will be removed in the future.
862
+ warnings.warn("The use_host_meta is deprecated, it will be removed in the future.")
863
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:336: UserWarning: The use_host_meta is deprecated, it will be removed in the future.
864
+ warnings.warn("The use_host_meta is deprecated, it will be removed in the future.")
865
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:336: UserWarning: The use_host_meta is deprecated, it will be removed in the future.
866
+ warnings.warn("The use_host_meta is deprecated, it will be removed in the future.")
867
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:334: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
868
+ warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
869
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:336: UserWarning: The use_host_meta is deprecated, it will be removed in the future.
870
+ warnings.warn("The use_host_meta is deprecated, it will be removed in the future.")
871
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:482: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
872
+ warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
873
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:482: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
874
+ warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
875
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:482: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
876
+ warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
877
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:482: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
878
+ warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
879
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:482: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
880
+ warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
881
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:482: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
882
+ warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
883
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:482: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
884
+ warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
885
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:482: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
886
+ warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
887
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:334: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
888
+ warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
889
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:334: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
890
+ warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
891
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:334: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
892
+ warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
893
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:336: UserWarning: The use_host_meta is deprecated, it will be removed in the future.
894
+ warnings.warn("The use_host_meta is deprecated, it will be removed in the future.")
895
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:336: UserWarning: The use_host_meta is deprecated, it will be removed in the future.
896
+ warnings.warn("The use_host_meta is deprecated, it will be removed in the future.")
897
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:336: UserWarning: The use_host_meta is deprecated, it will be removed in the future.
898
+ warnings.warn("The use_host_meta is deprecated, it will be removed in the future.")
899
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:334: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
900
+ warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
901
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:336: UserWarning: The use_host_meta is deprecated, it will be removed in the future.
902
+ warnings.warn("The use_host_meta is deprecated, it will be removed in the future.")
903
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:334: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
904
+ warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
905
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:336: UserWarning: The use_host_meta is deprecated, it will be removed in the future.
906
+ warnings.warn("The use_host_meta is deprecated, it will be removed in the future.")
907
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:334: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
908
+ warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
909
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:336: UserWarning: The use_host_meta is deprecated, it will be removed in the future.
910
+ warnings.warn("The use_host_meta is deprecated, it will be removed in the future.")
911
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:334: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
912
+ warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
913
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:336: UserWarning: The use_host_meta is deprecated, it will be removed in the future.
914
+ warnings.warn("The use_host_meta is deprecated, it will be removed in the future.")
915
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:334: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
916
+ warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
917
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:336: UserWarning: The use_host_meta is deprecated, it will be removed in the future.
918
+ warnings.warn("The use_host_meta is deprecated, it will be removed in the future.")
919
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:482: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
920
+ warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
921
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:482: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
922
+ warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
923
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:482: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
924
+ warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
925
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:482: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
926
+ warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
927
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:482: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
928
+ warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
929
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:482: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
930
+ warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
931
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:482: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
932
+ warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
933
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:482: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
934
+ warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
935
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:334: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
936
+ warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
937
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:334: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
938
+ warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
939
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:334: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
940
+ warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
941
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:334: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
942
+ warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
943
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:334: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
944
+ warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
945
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:334: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
946
+ warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
947
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:334: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
948
+ warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
949
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:334: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
950
+ warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
951
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:336: UserWarning: The use_host_meta is deprecated, it will be removed in the future.
952
+ warnings.warn("The use_host_meta is deprecated, it will be removed in the future.")
953
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:336: UserWarning: The use_host_meta is deprecated, it will be removed in the future.
954
+ warnings.warn("The use_host_meta is deprecated, it will be removed in the future.")
955
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:336: UserWarning: The use_host_meta is deprecated, it will be removed in the future.
956
+ warnings.warn("The use_host_meta is deprecated, it will be removed in the future.")
957
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:336: UserWarning: The use_host_meta is deprecated, it will be removed in the future.
958
+ warnings.warn("The use_host_meta is deprecated, it will be removed in the future.")
959
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:336: UserWarning: The use_host_meta is deprecated, it will be removed in the future.
960
+ warnings.warn("The use_host_meta is deprecated, it will be removed in the future.")
961
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:336: UserWarning: The use_host_meta is deprecated, it will be removed in the future.
962
+ warnings.warn("The use_host_meta is deprecated, it will be removed in the future.")
963
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:336: UserWarning: The use_host_meta is deprecated, it will be removed in the future.
964
+ warnings.warn("The use_host_meta is deprecated, it will be removed in the future.")
965
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:336: UserWarning: The use_host_meta is deprecated, it will be removed in the future.
966
+ warnings.warn("The use_host_meta is deprecated, it will be removed in the future.")
967
+ [2026-08-30 07:21:51] iteration 1/ 12 | consumed samples: 16 | elapsed time per iteration (ms): 25963.1 | throughput per GPU (TFLOP/s/GPU): 7.6 | learning rate: 2.000000E-05 | global batch size: 16 | lm loss: 1.103051E+01 | loss scale: 1.0 | grad norm: 50.241 | number of skipped iterations: 0 | number of nan iterations: 0 | peak recv tokens: 28416 | min safe capacity: 1.7344 |
968
+ Number of parameters in transformer block in billions: 3.09
969
+ Number of parameters in embedding layers in billions: 0.41
970
+ Total number of parameters in billions: 3.50
971
+ Number of parameters in most loaded shard in billions: 3.4994
972
+ Theoretical memory footprints: weight and optimizer=25029.43 MB
973
+ [Rank 0] (after 1 iterations) memory (MB) | allocated: 14963.09814453125 | max allocated: 30454.23095703125 | reserved: 31264.0 | max reserved: 31264.0
974
+ [2026-08-30 07:21:51] iteration 2/ 12 | consumed samples: 32 | elapsed time per iteration (ms): 336.5 | throughput per GPU (TFLOP/s/GPU): 589.5 | learning rate: 4.000000E-05 | global batch size: 16 | lm loss: 1.101006E+01 | loss scale: 1.0 | grad norm: 51.122 | number of skipped iterations: 0 | number of nan iterations: 0 | peak recv tokens: 28416 | min safe capacity: 1.7344 |
975
+ [2026-08-30 07:21:51] iteration 3/ 12 | consumed samples: 48 | elapsed time per iteration (ms): 329.4 | throughput per GPU (TFLOP/s/GPU): 602.3 | learning rate: 6.000000E-05 | global batch size: 16 | lm loss: 1.053811E+01 | loss scale: 1.0 | grad norm: 249.931 | number of skipped iterations: 0 | number of nan iterations: 0 | peak recv tokens: 28416 | min safe capacity: 1.7344 |
976
+ Capture CUDA graph for training!!!
977
+ CUDA graph capture done!!!
978
+ [2026-08-30 07:21:53] iteration 4/ 12 | consumed samples: 64 | elapsed time per iteration (ms): 1232.4 | throughput per GPU (TFLOP/s/GPU): 161.0 | learning rate: 8.000000E-05 | global batch size: 16 | lm loss: 1.319993E+01 | loss scale: 1.0 | grad norm: 35.708 | number of skipped iterations: 0 | number of nan iterations: 0 | peak recv tokens: 28416 | min safe capacity: 1.7344 |
979
+ [2026-08-30 07:21:53] iteration 5/ 12 | consumed samples: 80 | elapsed time per iteration (ms): 293.2 | throughput per GPU (TFLOP/s/GPU): 676.7 | learning rate: 1.000000E-04 | global batch size: 16 | lm loss: 1.052083E+01 | loss scale: 1.0 | grad norm: 17.493 | number of skipped iterations: 0 | number of nan iterations: 0 | peak recv tokens: 28416 | min safe capacity: 1.7344 |
980
+ [2026-08-30 07:21:53] iteration 6/ 12 | consumed samples: 96 | elapsed time per iteration (ms): 292.3 | throughput per GPU (TFLOP/s/GPU): 678.6 | learning rate: 7.750000E-05 | global batch size: 16 | lm loss: 1.027140E+01 | loss scale: 1.0 | grad norm: 9.648 | number of skipped iterations: 0 | number of nan iterations: 0 | peak recv tokens: 28416 | min safe capacity: 1.7344 |
981
+ [2026-08-30 07:21:53] iteration 7/ 12 | consumed samples: 112 | elapsed time per iteration (ms): 290.2 | throughput per GPU (TFLOP/s/GPU): 683.6 | learning rate: 3.250000E-05 | global batch size: 16 | lm loss: 9.466166E+00 | loss scale: 1.0 | grad norm: 7.620 | number of skipped iterations: 0 | number of nan iterations: 0 | peak recv tokens: 28416 | min safe capacity: 1.7344 |
982
+ [2026-08-30 07:21:54] iteration 8/ 12 | consumed samples: 128 | elapsed time per iteration (ms): 290.4 | throughput per GPU (TFLOP/s/GPU): 683.1 | learning rate: 1.000000E-05 | global batch size: 16 | lm loss: 8.752878E+00 | loss scale: 1.0 | grad norm: 6.063 | number of skipped iterations: 0 | number of nan iterations: 0 | peak recv tokens: 28416 | min safe capacity: 1.7344 |
983
+ [2026-08-30 07:21:54] iteration 9/ 12 | consumed samples: 144 | elapsed time per iteration (ms): 294.4 | throughput per GPU (TFLOP/s/GPU): 673.9 | learning rate: 1.000000E-05 | global batch size: 16 | lm loss: 8.545376E+00 | loss scale: 1.0 | grad norm: 7.113 | number of skipped iterations: 0 | number of nan iterations: 0 | peak recv tokens: 28416 | min safe capacity: 1.7344 |
984
+ [2026-08-30 07:21:54] iteration 10/ 12 | consumed samples: 160 | elapsed time per iteration (ms): 292.4 | throughput per GPU (TFLOP/s/GPU): 678.5 | learning rate: 1.000000E-05 | global batch size: 16 | lm loss: 8.451232E+00 | loss scale: 1.0 | grad norm: 6.175 | number of skipped iterations: 0 | number of nan iterations: 0 | peak recv tokens: 28416 | min safe capacity: 1.7344 |
985
+ [2026-08-30 07:21:55] iteration 11/ 12 | consumed samples: 176 | elapsed time per iteration (ms): 294.5 | throughput per GPU (TFLOP/s/GPU): 673.6 | learning rate: 1.000000E-05 | global batch size: 16 | lm loss: 8.375832E+00 | loss scale: 1.0 | grad norm: 4.656 | number of skipped iterations: 0 | number of nan iterations: 0 | peak recv tokens: 28416 | min safe capacity: 1.7344 |
986
+ [2026-08-30 07:21:55] iteration 12/ 12 | consumed samples: 192 | elapsed time per iteration (ms): 294.3 | throughput per GPU (TFLOP/s/GPU): 674.0 | learning rate: 1.000000E-05 | global batch size: 16 | lm loss: 8.273790E+00 | loss scale: 1.0 | grad norm: 4.113 | number of skipped iterations: 0 | number of nan iterations: 0 | peak recv tokens: 28416 | min safe capacity: 1.7344 |
987
+ [after training is done] datetime: 2026-08-30 07:21:55
988
+ [rank0]:[W830 07:21:56.857927972 ProcessGroupNCCL.cpp:1505] Warning: WARNING: destroy_process_group() was not called before program exit, which can leak resources. For more info, please see https://pytorch.org/docs/stable/distributed.html#shutdown (function operator())
989
+ W0830 07:21:56.682000 744857 torch/distributed/elastic/agent/server/api.py:719] Received 15 death signal, shutting down workers
990
+ W0830 07:21:56.683000 744857 torch/distributed/elastic/multiprocessing/api.py:900] Sending process 744945 closing signal SIGTERM
991
+ W0830 07:21:56.683000 744857 torch/distributed/elastic/multiprocessing/api.py:900] Sending process 744946 closing signal SIGTERM
992
+ W0830 07:21:56.685000 744857 torch/distributed/elastic/multiprocessing/api.py:900] Sending process 744947 closing signal SIGTERM
993
+ W0830 07:21:56.687000 744857 torch/distributed/elastic/multiprocessing/api.py:900] Sending process 744948 closing signal SIGTERM
994
+ W0830 07:21:56.688000 744857 torch/distributed/elastic/multiprocessing/api.py:900] Sending process 744949 closing signal SIGTERM
995
+ W0830 07:21:56.691000 744857 torch/distributed/elastic/multiprocessing/api.py:900] Sending process 744950 closing signal SIGTERM
996
+ W0830 07:21:56.692000 744857 torch/distributed/elastic/multiprocessing/api.py:900] Sending process 744951 closing signal SIGTERM
997
+ W0830 07:21:56.693000 744857 torch/distributed/elastic/multiprocessing/api.py:900] Sending process 744952 closing signal SIGTERM
998
+ Traceback (most recent call last):
999
+ File "/usr/local/bin/torchrun", line 8, in <module>
1000
+ sys.exit(main())
1001
+ ^^^^^^
1002
+ File "/usr/local/lib/python3.12/dist-packages/torch/distributed/elastic/multiprocessing/errors/__init__.py", line 355, in wrapper
1003
+ return f(*args, **kwargs)
1004
+ ^^^^^^^^^^^^^^^^^^
1005
+ File "/usr/local/lib/python3.12/dist-packages/torch/distributed/run.py", line 892, in main
1006
+ run(args)
1007
+ File "/usr/local/lib/python3.12/dist-packages/torch/distributed/run.py", line 883, in run
1008
+ elastic_launch(
1009
+ File "/usr/local/lib/python3.12/dist-packages/torch/distributed/launcher/api.py", line 139, in __call__
1010
+ return launch_agent(self._config, self._entrypoint, list(args))
1011
+ ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
1012
+ File "/usr/local/lib/python3.12/dist-packages/torch/distributed/launcher/api.py", line 261, in launch_agent
1013
+ result = agent.run()
1014
+ ^^^^^^^^^^^
1015
+ File "/usr/local/lib/python3.12/dist-packages/torch/distributed/elastic/metrics/api.py", line 138, in wrapper
1016
+ result = f(*args, **kwargs)
1017
+ ^^^^^^^^^^^^^^^^^^
1018
+ File "/usr/local/lib/python3.12/dist-packages/torch/distributed/elastic/agent/server/api.py", line 711, in run
1019
+ result = self._invoke_run(role)
1020
+ ^^^^^^^^^^^^^^^^^^^^^^
1021
+ File "/usr/local/lib/python3.12/dist-packages/torch/distributed/elastic/agent/server/api.py", line 870, in _invoke_run
1022
+ time.sleep(monitor_interval)
1023
+ File "/usr/local/lib/python3.12/dist-packages/torch/distributed/elastic/multiprocessing/api.py", line 84, in _terminate_process_handler
1024
+ raise SignalException(f"Process {os.getpid()} got signal: {sigval}", sigval=sigval)
1025
+ torch.distributed.elastic.multiprocessing.api.SignalException: Process 744857 got signal: 15
tiera-rerun-fixed-moonep/ultraep/ponly/ponly_gbs16/results.tsv CHANGED
@@ -3,3 +3,5 @@ PONLY ultraep one 1 4096 16 2.40625 none 1 OK 176.80 175.80 38329 /workspace/tie
3
  PONLY ultraep one 1 4096 16 2.40625 none 2 OK 176.80 175.80 38329 /workspace/tiera-rerun-fixed-moonep/ultraep/ponly/ponly_gbs16/PONLY__ultraep__one__seq4096__gbs16__cap2p40625/rep2
4
  PONLY ultraep one 1 4096 16 2.40625 none 3 OK 176.76 175.80 38331 /workspace/tiera-rerun-fixed-moonep/ultraep/ponly/ponly_gbs16/PONLY__ultraep__one__seq4096__gbs16__cap2p40625/rep3
5
  PONLY ultraep one 1 8192 16 2.40625 none 1 OK 292.76 290.70 51017 /workspace/tiera-rerun-fixed-moonep/ultraep/ponly/ponly_gbs16/PONLY__ultraep__one__seq8192__gbs16__cap2p40625/rep1
 
 
 
3
  PONLY ultraep one 1 4096 16 2.40625 none 2 OK 176.80 175.80 38329 /workspace/tiera-rerun-fixed-moonep/ultraep/ponly/ponly_gbs16/PONLY__ultraep__one__seq4096__gbs16__cap2p40625/rep2
4
  PONLY ultraep one 1 4096 16 2.40625 none 3 OK 176.76 175.80 38331 /workspace/tiera-rerun-fixed-moonep/ultraep/ponly/ponly_gbs16/PONLY__ultraep__one__seq4096__gbs16__cap2p40625/rep3
5
  PONLY ultraep one 1 8192 16 2.40625 none 1 OK 292.76 290.70 51017 /workspace/tiera-rerun-fixed-moonep/ultraep/ponly/ponly_gbs16/PONLY__ultraep__one__seq8192__gbs16__cap2p40625/rep1
6
+ PONLY ultraep one 1 8192 16 2.40625 none 2 OK 292.61 290.70 51015 /workspace/tiera-rerun-fixed-moonep/ultraep/ponly/ponly_gbs16/PONLY__ultraep__one__seq8192__gbs16__cap2p40625/rep2
7
+ PONLY ultraep one 1 8192 16 2.40625 none 3 OK 292.62 290.20 51013 /workspace/tiera-rerun-fixed-moonep/ultraep/ponly/ponly_gbs16/PONLY__ultraep__one__seq8192__gbs16__cap2p40625/rep3