granularity results tiera-rerun-fixed-moonep @ 2026-08-30T06:50:39Z from sp-aiinfra-interactive-aghosh-7jtjl-8xb200-moe-cg-0
Browse files- tiera-rerun-fixed-moonep/.completed/echo__ponly__gbs16 +1 -0
- tiera-rerun-fixed-moonep/STATUS.txt +3 -2
- tiera-rerun-fixed-moonep/echo/a2/a2_matched_gbs16/G1__echo__matched__seq4096__gbs16__cap2p40625/rep1/mem_samples.csv +232 -0
- tiera-rerun-fixed-moonep/echo/a2/a2_matched_gbs16/G1__echo__matched__seq4096__gbs16__cap2p40625/rep1/run.env +70 -0
- tiera-rerun-fixed-moonep/echo/a2/a2_matched_gbs16/G1__echo__matched__seq4096__gbs16__cap2p40625/rep1/train.log +1023 -0
- tiera-rerun-fixed-moonep/echo/a2/a2_matched_gbs16/G1__echo__matched__seq4096__gbs16__cap2p40625/rep2/mem_samples.csv +264 -0
- tiera-rerun-fixed-moonep/echo/a2/a2_matched_gbs16/G1__echo__matched__seq4096__gbs16__cap2p40625/rep2/run.env +70 -0
- tiera-rerun-fixed-moonep/echo/a2/a2_matched_gbs16/G1__echo__matched__seq4096__gbs16__cap2p40625/rep2/train.log +987 -0
- tiera-rerun-fixed-moonep/echo/a2/a2_matched_gbs16/G1__echo__matched__seq4096__gbs16__cap2p40625/rep3/mem_samples.csv +232 -0
- tiera-rerun-fixed-moonep/echo/a2/a2_matched_gbs16/G1__echo__matched__seq4096__gbs16__cap2p40625/rep3/run.env +70 -0
- tiera-rerun-fixed-moonep/echo/a2/a2_matched_gbs16/G1__echo__matched__seq4096__gbs16__cap2p40625/rep3/train.log +1023 -0
- tiera-rerun-fixed-moonep/echo/a2/a2_matched_gbs16/G2__echo__matched__seq4096__gbs16__cap1p08203125/rep1/mem_samples.csv +256 -0
- tiera-rerun-fixed-moonep/echo/a2/a2_matched_gbs16/G2__echo__matched__seq4096__gbs16__cap1p08203125/rep1/run.env +70 -0
- tiera-rerun-fixed-moonep/echo/a2/a2_matched_gbs16/G2__echo__matched__seq4096__gbs16__cap1p08203125/rep1/train.log +1107 -0
- tiera-rerun-fixed-moonep/echo/a2/a2_matched_gbs16/G2__echo__matched__seq4096__gbs16__cap1p08203125/rep2/mem_samples.csv +184 -0
- tiera-rerun-fixed-moonep/echo/a2/a2_matched_gbs16/G2__echo__matched__seq4096__gbs16__cap1p08203125/rep2/run.env +70 -0
- tiera-rerun-fixed-moonep/echo/a2/a2_matched_gbs16/G2__echo__matched__seq4096__gbs16__cap1p08203125/rep2/train.log +1085 -0
- tiera-rerun-fixed-moonep/echo/a2/a2_matched_gbs16/results.tsv +5 -0
- tiera-rerun-fixed-moonep/echo/a2/manifest.txt +19 -0
- tiera-rerun-fixed-moonep/echo/a2/pip_freeze.txt +317 -0
- tiera-rerun-fixed-moonep/echo/a2/provenance.txt +90 -0
- tiera-rerun-fixed-moonep/hf_watcher.log +3 -0
- tiera-rerun-fixed-moonep/steps.tsv +1 -0
- tiera-rerun-fixed-moonep/study.log +48 -0
tiera-rerun-fixed-moonep/.completed/echo__ponly__gbs16
ADDED
|
@@ -0,0 +1 @@
|
|
|
|
|
|
|
| 1 |
+
2026-08-30T06:44:11+00:00
|
tiera-rerun-fixed-moonep/STATUS.txt
CHANGED
|
@@ -1,10 +1,11 @@
|
|
| 1 |
study_root: /workspace/tiera-rerun-fixed-moonep
|
| 2 |
gbs: 16
|
| 3 |
-
updated: 2026-08-30T06:
|
| 4 |
-
current:
|
| 5 |
|
| 6 |
completed steps:
|
| 7 |
echo__a1__gbs16
|
|
|
|
| 8 |
|
| 9 |
failed steps:
|
| 10 |
FAIL echo tierbprobe 16 2026-08-30T04:52:38+00:00 2026-08-30T05:06:38+00:00 /workspace/tiera-rerun-fixed-moonep/echo/tierbprobe
|
|
|
|
| 1 |
study_root: /workspace/tiera-rerun-fixed-moonep
|
| 2 |
gbs: 16
|
| 3 |
+
updated: 2026-08-30T06:44:13+00:00
|
| 4 |
+
current: echo__a2__gbs16
|
| 5 |
|
| 6 |
completed steps:
|
| 7 |
echo__a1__gbs16
|
| 8 |
+
echo__ponly__gbs16
|
| 9 |
|
| 10 |
failed steps:
|
| 11 |
FAIL echo tierbprobe 16 2026-08-30T04:52:38+00:00 2026-08-30T05:06:38+00:00 /workspace/tiera-rerun-fixed-moonep/echo/tierbprobe
|
tiera-rerun-fixed-moonep/echo/a2/a2_matched_gbs16/G1__echo__matched__seq4096__gbs16__cap2p40625/rep1/mem_samples.csv
ADDED
|
@@ -0,0 +1,232 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
0, 0
|
| 2 |
+
1, 0
|
| 3 |
+
2, 0
|
| 4 |
+
3, 0
|
| 5 |
+
4, 0
|
| 6 |
+
5, 0
|
| 7 |
+
6, 0
|
| 8 |
+
7, 0
|
| 9 |
+
0, 0
|
| 10 |
+
1, 0
|
| 11 |
+
2, 0
|
| 12 |
+
3, 0
|
| 13 |
+
4, 0
|
| 14 |
+
5, 0
|
| 15 |
+
6, 0
|
| 16 |
+
7, 0
|
| 17 |
+
0, 2
|
| 18 |
+
1, 1
|
| 19 |
+
2, 1
|
| 20 |
+
3, 1
|
| 21 |
+
4, 1
|
| 22 |
+
5, 1
|
| 23 |
+
6, 1
|
| 24 |
+
7, 1
|
| 25 |
+
0, 4
|
| 26 |
+
1, 4
|
| 27 |
+
2, 4
|
| 28 |
+
3, 4
|
| 29 |
+
4, 1
|
| 30 |
+
5, 1
|
| 31 |
+
6, 1
|
| 32 |
+
7, 1
|
| 33 |
+
0, 4
|
| 34 |
+
1, 4
|
| 35 |
+
2, 4
|
| 36 |
+
3, 4
|
| 37 |
+
4, 4
|
| 38 |
+
5, 4
|
| 39 |
+
6, 4
|
| 40 |
+
7, 4
|
| 41 |
+
0, 4
|
| 42 |
+
1, 4
|
| 43 |
+
2, 41
|
| 44 |
+
3, 4
|
| 45 |
+
4, 41
|
| 46 |
+
5, 4
|
| 47 |
+
6, 35
|
| 48 |
+
7, 4
|
| 49 |
+
0, 887
|
| 50 |
+
1, 897
|
| 51 |
+
2, 905
|
| 52 |
+
3, 895
|
| 53 |
+
4, 893
|
| 54 |
+
5, 899
|
| 55 |
+
6, 915
|
| 56 |
+
7, 911
|
| 57 |
+
0, 1895
|
| 58 |
+
1, 1895
|
| 59 |
+
2, 1895
|
| 60 |
+
3, 1895
|
| 61 |
+
4, 1895
|
| 62 |
+
5, 1895
|
| 63 |
+
6, 1895
|
| 64 |
+
7, 1895
|
| 65 |
+
0, 39467
|
| 66 |
+
1, 39493
|
| 67 |
+
2, 39463
|
| 68 |
+
3, 39483
|
| 69 |
+
4, 39529
|
| 70 |
+
5, 39505
|
| 71 |
+
6, 39503
|
| 72 |
+
7, 39527
|
| 73 |
+
0, 39907
|
| 74 |
+
1, 39907
|
| 75 |
+
2, 39883
|
| 76 |
+
3, 39907
|
| 77 |
+
4, 39907
|
| 78 |
+
5, 39907
|
| 79 |
+
6, 39907
|
| 80 |
+
7, 39907
|
| 81 |
+
0, 42357
|
| 82 |
+
1, 42355
|
| 83 |
+
2, 42355
|
| 84 |
+
3, 42355
|
| 85 |
+
4, 42355
|
| 86 |
+
5, 42355
|
| 87 |
+
6, 42355
|
| 88 |
+
7, 42355
|
| 89 |
+
0, 42583
|
| 90 |
+
1, 42581
|
| 91 |
+
2, 42581
|
| 92 |
+
3, 42581
|
| 93 |
+
4, 42581
|
| 94 |
+
5, 42581
|
| 95 |
+
6, 42581
|
| 96 |
+
7, 42581
|
| 97 |
+
0, 42769
|
| 98 |
+
1, 42759
|
| 99 |
+
2, 42745
|
| 100 |
+
3, 42779
|
| 101 |
+
4, 42763
|
| 102 |
+
5, 42759
|
| 103 |
+
6, 42755
|
| 104 |
+
7, 42769
|
| 105 |
+
0, 43493
|
| 106 |
+
1, 43491
|
| 107 |
+
2, 43491
|
| 108 |
+
3, 43491
|
| 109 |
+
4, 43491
|
| 110 |
+
5, 43491
|
| 111 |
+
6, 43491
|
| 112 |
+
7, 43491
|
| 113 |
+
0, 43493
|
| 114 |
+
1, 43491
|
| 115 |
+
2, 43491
|
| 116 |
+
3, 43491
|
| 117 |
+
4, 43491
|
| 118 |
+
5, 43491
|
| 119 |
+
6, 43491
|
| 120 |
+
7, 43491
|
| 121 |
+
0, 45265
|
| 122 |
+
1, 45263
|
| 123 |
+
2, 45263
|
| 124 |
+
3, 45263
|
| 125 |
+
4, 45263
|
| 126 |
+
5, 45263
|
| 127 |
+
6, 45263
|
| 128 |
+
7, 45263
|
| 129 |
+
0, 45265
|
| 130 |
+
1, 45263
|
| 131 |
+
2, 45263
|
| 132 |
+
3, 45263
|
| 133 |
+
4, 45263
|
| 134 |
+
5, 45263
|
| 135 |
+
6, 45263
|
| 136 |
+
7, 45263
|
| 137 |
+
0, 85827
|
| 138 |
+
1, 85825
|
| 139 |
+
2, 85825
|
| 140 |
+
3, 85825
|
| 141 |
+
4, 85825
|
| 142 |
+
5, 85825
|
| 143 |
+
6, 85825
|
| 144 |
+
7, 85825
|
| 145 |
+
0, 86905
|
| 146 |
+
1, 86903
|
| 147 |
+
2, 86903
|
| 148 |
+
3, 86903
|
| 149 |
+
4, 86903
|
| 150 |
+
5, 86903
|
| 151 |
+
6, 86903
|
| 152 |
+
7, 86903
|
| 153 |
+
0, 86905
|
| 154 |
+
1, 86903
|
| 155 |
+
2, 86903
|
| 156 |
+
3, 86903
|
| 157 |
+
4, 86903
|
| 158 |
+
5, 86903
|
| 159 |
+
6, 86903
|
| 160 |
+
7, 86903
|
| 161 |
+
0, 86905
|
| 162 |
+
1, 86903
|
| 163 |
+
2, 86903
|
| 164 |
+
3, 86903
|
| 165 |
+
4, 86903
|
| 166 |
+
5, 86903
|
| 167 |
+
6, 86903
|
| 168 |
+
7, 86903
|
| 169 |
+
0, 104407
|
| 170 |
+
1, 104349
|
| 171 |
+
2, 104349
|
| 172 |
+
3, 104349
|
| 173 |
+
4, 104405
|
| 174 |
+
5, 104349
|
| 175 |
+
6, 104349
|
| 176 |
+
7, 104349
|
| 177 |
+
0, 105875
|
| 178 |
+
1, 105817
|
| 179 |
+
2, 105817
|
| 180 |
+
3, 105817
|
| 181 |
+
4, 105873
|
| 182 |
+
5, 105817
|
| 183 |
+
6, 105817
|
| 184 |
+
7, 105817
|
| 185 |
+
0, 106523
|
| 186 |
+
1, 106465
|
| 187 |
+
2, 106465
|
| 188 |
+
3, 106465
|
| 189 |
+
4, 106521
|
| 190 |
+
5, 106465
|
| 191 |
+
6, 106465
|
| 192 |
+
7, 106465
|
| 193 |
+
0, 107183
|
| 194 |
+
1, 107131
|
| 195 |
+
2, 107075
|
| 196 |
+
3, 107119
|
| 197 |
+
4, 107157
|
| 198 |
+
5, 107169
|
| 199 |
+
6, 107147
|
| 200 |
+
7, 107183
|
| 201 |
+
0, 138613
|
| 202 |
+
1, 139203
|
| 203 |
+
2, 137959
|
| 204 |
+
3, 136499
|
| 205 |
+
4, 136849
|
| 206 |
+
5, 136653
|
| 207 |
+
6, 159453
|
| 208 |
+
7, 146361
|
| 209 |
+
0, 161641
|
| 210 |
+
1, 161525
|
| 211 |
+
2, 161525
|
| 212 |
+
3, 161525
|
| 213 |
+
4, 161723
|
| 214 |
+
5, 161525
|
| 215 |
+
6, 161525
|
| 216 |
+
7, 161525
|
| 217 |
+
0, 161641
|
| 218 |
+
1, 161525
|
| 219 |
+
2, 161525
|
| 220 |
+
3, 161525
|
| 221 |
+
4, 161723
|
| 222 |
+
5, 161525
|
| 223 |
+
6, 161525
|
| 224 |
+
7, 161525
|
| 225 |
+
0, 3695
|
| 226 |
+
1, 2098
|
| 227 |
+
2, 2098
|
| 228 |
+
3, 2098
|
| 229 |
+
4, 2098
|
| 230 |
+
5, 2098
|
| 231 |
+
6, 2418
|
| 232 |
+
7, 161525
|
tiera-rerun-fixed-moonep/echo/a2/a2_matched_gbs16/G1__echo__matched__seq4096__gbs16__cap2p40625/rep1/run.env
ADDED
|
@@ -0,0 +1,70 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
backend=echo
|
| 2 |
+
config=G1
|
| 3 |
+
family=tierA
|
| 4 |
+
repo=/workspace/Megatron-MoE-Echo
|
| 5 |
+
branch=bracing-regions-through-kernels
|
| 6 |
+
commit=9a236793
|
| 7 |
+
num_layers=16
|
| 8 |
+
hidden=4096
|
| 9 |
+
ffn_hidden=14336
|
| 10 |
+
moe_ffn_hidden=14336
|
| 11 |
+
num_experts=8
|
| 12 |
+
topk=2
|
| 13 |
+
experts_per_rank=1
|
| 14 |
+
expert_params=176160768
|
| 15 |
+
heads=32
|
| 16 |
+
kv_groups=8
|
| 17 |
+
kv_channels=
|
| 18 |
+
qk_layernorm=0
|
| 19 |
+
shared_ffn=0
|
| 20 |
+
shared_expert_overlap=0
|
| 21 |
+
moe_token_dispatcher=flex
|
| 22 |
+
mla=0
|
| 23 |
+
dense_first_layers=0
|
| 24 |
+
moe_layers=16
|
| 25 |
+
rotary_base=1000000
|
| 26 |
+
rotary_scaling=
|
| 27 |
+
qkv_bias=0
|
| 28 |
+
router_balancing=aux_loss
|
| 29 |
+
aux_loss_coeff=1e-2
|
| 30 |
+
router_pre_softmax=0
|
| 31 |
+
router_fusion=0
|
| 32 |
+
router_dtype=fp32
|
| 33 |
+
device_initiated_grouped_gemm=1
|
| 34 |
+
norm_epsilon=1e-5
|
| 35 |
+
seq_length=4096
|
| 36 |
+
micro_batch_size=1
|
| 37 |
+
global_batch_size=16
|
| 38 |
+
num_microbatches=2
|
| 39 |
+
assignments_per_rank=8192
|
| 40 |
+
mean_tokens_per_expert=8192
|
| 41 |
+
slot_policy=matched
|
| 42 |
+
slots_per_rank=1
|
| 43 |
+
capacity_mode=static
|
| 44 |
+
capacity=2.40625
|
| 45 |
+
cuda_graph=1
|
| 46 |
+
probe_mode=0
|
| 47 |
+
async_grad_reduce=0
|
| 48 |
+
branch_role=timing
|
| 49 |
+
memory_dump_mode=none
|
| 50 |
+
memory_snapshot_path=
|
| 51 |
+
cuda_graph_warmup_steps=3
|
| 52 |
+
recompute=none
|
| 53 |
+
force_balance=0
|
| 54 |
+
train_iters=12
|
| 55 |
+
ep_size=8
|
| 56 |
+
echo_dispatch_overlap=1
|
| 57 |
+
echo_dispatch_recompute=0
|
| 58 |
+
ultraep_quota_min_tokens=
|
| 59 |
+
ultraep_weight_sync_join=
|
| 60 |
+
profile_enable=0
|
| 61 |
+
kernel_trace=0
|
| 62 |
+
fp8_mode=mxfp8
|
| 63 |
+
fp8_recipe=mxfp8
|
| 64 |
+
training_seed=1234
|
| 65 |
+
learning_rate=1e-4
|
| 66 |
+
min_learning_rate=1.0e-5
|
| 67 |
+
lr_warmup_iters=5
|
| 68 |
+
state_dump_enable=0
|
| 69 |
+
state_dump_dir=
|
| 70 |
+
moonep_assert_no_pending_reduce=0
|
tiera-rerun-fixed-moonep/echo/a2/a2_matched_gbs16/G1__echo__matched__seq4096__gbs16__cap2p40625/rep1/train.log
ADDED
|
@@ -0,0 +1,1023 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
==============================================================
|
| 2 |
+
G1 / echo (slots/rank=1 policy=matched)
|
| 3 |
+
E=8 K=2 H'=14336 m=1
|
| 4 |
+
L=16 seq=4096 gbs=16 (2 microbatches)
|
| 5 |
+
cap_mode=static cap=2.40625 cg=1 recompute=none
|
| 6 |
+
memory_dump=none snapshots=/workspace/tiera-rerun-fixed-moonep/echo/a2/a2_matched_gbs16/G1__echo__matched__seq4096__gbs16__cap2p40625/rep1/memory_snapshots/snapshot.pickle
|
| 7 |
+
repo=/workspace/Megatron-MoE-Echo @ 9a236793
|
| 8 |
+
==============================================================
|
| 9 |
+
W0830 06:44:30.036000 651947 torch/distributed/run.py:766]
|
| 10 |
+
W0830 06:44:30.036000 651947 torch/distributed/run.py:766] *****************************************
|
| 11 |
+
W0830 06:44:30.036000 651947 torch/distributed/run.py:766] Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed.
|
| 12 |
+
W0830 06:44:30.036000 651947 torch/distributed/run.py:766] *****************************************
|
| 13 |
+
/workspace/Megatron-MoE-Echo/megatron/training/arguments.py:831: UserWarning: Disabling sequence parallelism because tensor model parallelism is disabled
|
| 14 |
+
warnings.warn("Disabling sequence parallelism because tensor model parallelism is disabled")
|
| 15 |
+
/workspace/Megatron-MoE-Echo/megatron/training/arguments.py:831: UserWarning: Disabling sequence parallelism because tensor model parallelism is disabled
|
| 16 |
+
warnings.warn("Disabling sequence parallelism because tensor model parallelism is disabled")
|
| 17 |
+
/workspace/Megatron-MoE-Echo/megatron/training/arguments.py:831: UserWarning: Disabling sequence parallelism because tensor model parallelism is disabled
|
| 18 |
+
warnings.warn("Disabling sequence parallelism because tensor model parallelism is disabled")
|
| 19 |
+
using world size: 8, data-parallel size: 8, context-parallel size: 1, hierarchical context-parallel sizes: None, tensor-model-parallel size: 1, pipeline-model-parallel size: 1
|
| 20 |
+
WARNING: overriding default arguments for tokenizer_type:GPT2BPETokenizer with tokenizer_type:GPT2BPETokenizer
|
| 21 |
+
Number of virtual stages per pipeline stage: None
|
| 22 |
+
accumulate and all-reduce gradients in fp32 for bfloat16 data type.
|
| 23 |
+
using torch.bfloat16 for parameters ...
|
| 24 |
+
/workspace/Megatron-MoE-Echo/megatron/training/arguments.py:831: UserWarning: Disabling sequence parallelism because tensor model parallelism is disabled
|
| 25 |
+
warnings.warn("Disabling sequence parallelism because tensor model parallelism is disabled")
|
| 26 |
+
------------------------ arguments ------------------------
|
| 27 |
+
account_for_embedding_in_pipeline_split ......... False
|
| 28 |
+
account_for_loss_in_pipeline_split .............. False
|
| 29 |
+
accumulate_allreduce_grads_in_fp32 .............. True
|
| 30 |
+
adam_beta1 ...................................... 0.9
|
| 31 |
+
adam_beta2 ...................................... 0.999
|
| 32 |
+
adam_eps ........................................ 1e-08
|
| 33 |
+
add_bias_linear ................................. False
|
| 34 |
+
add_position_embedding .......................... False
|
| 35 |
+
add_qkv_bias .................................... False
|
| 36 |
+
adlr_autoresume ................................. False
|
| 37 |
+
adlr_autoresume_interval ........................ 1000
|
| 38 |
+
align_grad_reduce ............................... True
|
| 39 |
+
align_param_gather .............................. False
|
| 40 |
+
app_tag_run_name ................................ None
|
| 41 |
+
app_tag_run_version ............................. 0.0.0
|
| 42 |
+
apply_layernorm_1p .............................. False
|
| 43 |
+
apply_query_key_layer_scaling ................... False
|
| 44 |
+
apply_residual_connection_post_layernorm ........ False
|
| 45 |
+
apply_rope_fusion ............................... True
|
| 46 |
+
async_save ...................................... None
|
| 47 |
+
async_tensor_model_parallel_allreduce ........... True
|
| 48 |
+
attention_backend ............................... AttnBackend.auto
|
| 49 |
+
attention_dropout ............................... 0.0
|
| 50 |
+
attention_softmax_in_fp32 ....................... False
|
| 51 |
+
auto_detect_ckpt_format ......................... False
|
| 52 |
+
barrier_with_L1_time ............................ True
|
| 53 |
+
bert_binary_head ................................ True
|
| 54 |
+
bert_embedder_type .............................. megatron
|
| 55 |
+
bert_load ....................................... None
|
| 56 |
+
bf16 ............................................ True
|
| 57 |
+
bias_dropout_fusion ............................. True
|
| 58 |
+
bias_gelu_fusion ................................ False
|
| 59 |
+
bias_swiglu_fusion .............................. True
|
| 60 |
+
biencoder_projection_dim ........................ 0
|
| 61 |
+
biencoder_shared_query_context_model ............ False
|
| 62 |
+
block_data_path ................................. None
|
| 63 |
+
calc_ft_timeouts ................................ False
|
| 64 |
+
calculate_per_token_loss ........................ False
|
| 65 |
+
check_for_large_grads ........................... False
|
| 66 |
+
check_for_nan_in_loss_and_grad .................. False
|
| 67 |
+
check_for_spiky_loss ............................ False
|
| 68 |
+
check_weight_hash_across_dp_replicas_interval ... None
|
| 69 |
+
ckpt_assume_constant_structure .................. False
|
| 70 |
+
ckpt_convert_format ............................. None
|
| 71 |
+
ckpt_convert_save ............................... None
|
| 72 |
+
ckpt_convert_update_legacy_dist_opt_format ...... False
|
| 73 |
+
ckpt_format ..................................... torch_dist
|
| 74 |
+
ckpt_fully_parallel_load ........................ False
|
| 75 |
+
ckpt_fully_parallel_save ........................ True
|
| 76 |
+
ckpt_fully_parallel_save_deprecated ............. False
|
| 77 |
+
ckpt_step ....................................... None
|
| 78 |
+
classes_fraction ................................ 1.0
|
| 79 |
+
clip_grad ....................................... 1.0
|
| 80 |
+
clone_scatter_output_in_embedding ............... True
|
| 81 |
+
config_logger_dir ...............................
|
| 82 |
+
consumed_train_samples .......................... 0
|
| 83 |
+
consumed_valid_samples .......................... 0
|
| 84 |
+
context_parallel_size ........................... 1
|
| 85 |
+
cp_comm_type .................................... ['p2p']
|
| 86 |
+
create_attention_mask_in_dataloader ............. True
|
| 87 |
+
cross_entropy_fusion_impl ....................... native
|
| 88 |
+
cross_entropy_loss_fusion ....................... False
|
| 89 |
+
cuda_graph_scope ................................ full_iteration
|
| 90 |
+
cuda_graph_warmup_steps ......................... 3
|
| 91 |
+
data_args_path .................................. None
|
| 92 |
+
data_cache_path ................................. None
|
| 93 |
+
data_parallel_random_init ....................... False
|
| 94 |
+
data_parallel_sharding_strategy ................. no_shard
|
| 95 |
+
data_parallel_size .............................. 8
|
| 96 |
+
data_path ....................................... ['/workspace/data/wikitext/wikitext103_gpt2_text_document']
|
| 97 |
+
data_per_class_fraction ......................... 1.0
|
| 98 |
+
data_sharding ................................... True
|
| 99 |
+
dataloader_type ................................. single
|
| 100 |
+
ddp_average_in_collective ....................... False
|
| 101 |
+
ddp_bucket_size ................................. None
|
| 102 |
+
ddp_num_buckets ................................. None
|
| 103 |
+
ddp_pad_buckets_for_high_nccl_busbw ............. False
|
| 104 |
+
decoder_first_pipeline_num_layers ............... None
|
| 105 |
+
decoder_last_pipeline_num_layers ................ None
|
| 106 |
+
decoder_num_layers .............................. None
|
| 107 |
+
decoder_seq_length .............................. None
|
| 108 |
+
decoupled_lr .................................... None
|
| 109 |
+
decoupled_min_lr ................................ None
|
| 110 |
+
decrease_batch_size_if_needed ................... False
|
| 111 |
+
defer_embedding_wgrad_compute ................... False
|
| 112 |
+
delay_wgrad_compute ............................. False
|
| 113 |
+
deprecated_use_mcore_models ..................... True
|
| 114 |
+
deterministic_mode .............................. False
|
| 115 |
+
dino_bottleneck_size ............................ 256
|
| 116 |
+
dino_freeze_last_layer .......................... 1
|
| 117 |
+
dino_head_hidden_size ........................... 2048
|
| 118 |
+
dino_local_crops_number ......................... 10
|
| 119 |
+
dino_local_img_size ............................. 96
|
| 120 |
+
dino_norm_last_layer ............................ False
|
| 121 |
+
dino_teacher_temp ............................... 0.07
|
| 122 |
+
dino_warmup_teacher_temp ........................ 0.04
|
| 123 |
+
dino_warmup_teacher_temp_epochs ................. 30
|
| 124 |
+
disable_bf16_reduced_precision_matmul ........... False
|
| 125 |
+
disable_mamba_mem_eff_path ...................... False
|
| 126 |
+
disable_straggler_on_startup .................... False
|
| 127 |
+
dist_ckpt_format_deprecated ..................... None
|
| 128 |
+
dist_ckpt_strictness ............................ assume_ok_unexpected
|
| 129 |
+
distribute_saved_activations .................... False
|
| 130 |
+
distributed_backend ............................. nccl
|
| 131 |
+
distributed_timeout_minutes ..................... 10
|
| 132 |
+
embedding_init_method_std ....................... None
|
| 133 |
+
embedding_path .................................. None
|
| 134 |
+
empty_unused_memory_level ....................... 0
|
| 135 |
+
enable_cuda_graph ............................... True
|
| 136 |
+
enable_experimental ............................. False
|
| 137 |
+
enable_ft_package ............................... False
|
| 138 |
+
enable_gloo_process_groups ...................... True
|
| 139 |
+
enable_msc ...................................... True
|
| 140 |
+
enable_one_logger ............................... True
|
| 141 |
+
encoder_num_layers .............................. 16
|
| 142 |
+
encoder_seq_length .............................. 4096
|
| 143 |
+
end_weight_decay ................................ 0.1
|
| 144 |
+
eod_mask_loss ................................... False
|
| 145 |
+
error_injection_rate ............................ 0
|
| 146 |
+
error_injection_type ............................ transient_error
|
| 147 |
+
eval_interval ................................... 1000000
|
| 148 |
+
eval_iters ...................................... 0
|
| 149 |
+
evidence_data_path .............................. None
|
| 150 |
+
exit_duration_in_mins ........................... None
|
| 151 |
+
exit_interval ................................... None
|
| 152 |
+
exit_on_missing_checkpoint ...................... False
|
| 153 |
+
exit_signal_handler ............................. False
|
| 154 |
+
exp_avg_dtype ................................... torch.float32
|
| 155 |
+
exp_avg_sq_dtype ................................ torch.float32
|
| 156 |
+
expert_model_parallel_size ...................... 8
|
| 157 |
+
expert_tensor_parallel_size ..................... 1
|
| 158 |
+
export_force_local_attention .................... False
|
| 159 |
+
export_kd_cfg ................................... None
|
| 160 |
+
export_kd_teacher_ckpt_format ................... None
|
| 161 |
+
export_kd_teacher_load .......................... None
|
| 162 |
+
export_kv_cache_quant ........................... False
|
| 163 |
+
export_legacy_megatron .......................... False
|
| 164 |
+
export_model_type ............................... GPTModel
|
| 165 |
+
export_moe_apply_probs_on_input ................. False
|
| 166 |
+
export_qk_l2_norm ............................... False
|
| 167 |
+
export_quant_cfg ................................ None
|
| 168 |
+
export_real_quant_cfg ........................... None
|
| 169 |
+
export_te_mcore_model ........................... False
|
| 170 |
+
external_cuda_graph ............................. False
|
| 171 |
+
ffn_hidden_size ................................. 14336
|
| 172 |
+
finetune ........................................ False
|
| 173 |
+
finetune_data_split ............................. train
|
| 174 |
+
finetune_hf_dataset ............................. None
|
| 175 |
+
first_last_layers_bf16 .......................... False
|
| 176 |
+
flash_decode .................................... False
|
| 177 |
+
fp16 ............................................ False
|
| 178 |
+
fp16_lm_cross_entropy ........................... False
|
| 179 |
+
fp32_residual_connection ........................ False
|
| 180 |
+
fp8 ............................................. e4m3
|
| 181 |
+
fp8_amax_compute_algo ........................... most_recent
|
| 182 |
+
fp8_amax_history_len ............................ 1
|
| 183 |
+
fp8_interval .................................... 1
|
| 184 |
+
fp8_margin ...................................... 0
|
| 185 |
+
fp8_param_gather ................................ False
|
| 186 |
+
fp8_recipe ...................................... mxfp8
|
| 187 |
+
fp8_wgrad ....................................... True
|
| 188 |
+
fsdp_double_buffer .............................. False
|
| 189 |
+
global_batch_size ............................... 16
|
| 190 |
+
grad_reduce_in_bf16 ............................. False
|
| 191 |
+
gradient_accumulation_fusion .................... True
|
| 192 |
+
gradient_reduce_div_fusion ...................... True
|
| 193 |
+
group_query_attention ........................... True
|
| 194 |
+
head_lr_mult .................................... 1.0
|
| 195 |
+
heterogeneous_layers_config_encoded_json ........ None
|
| 196 |
+
heterogeneous_layers_config_path ................ None
|
| 197 |
+
hidden_dropout .................................. 0.0
|
| 198 |
+
hidden_size ..................................... 4096
|
| 199 |
+
hierarchical_context_parallel_sizes ............. None
|
| 200 |
+
high_priority_stream_groups ..................... []
|
| 201 |
+
hybrid_attention_ratio .......................... 0.0
|
| 202 |
+
hybrid_mlp_ratio ................................ 0.0
|
| 203 |
+
hybrid_override_pattern ......................... None
|
| 204 |
+
hysteresis ...................................... 2
|
| 205 |
+
ict_head_size ................................... None
|
| 206 |
+
ict_load ........................................ None
|
| 207 |
+
img_h ........................................... 224
|
| 208 |
+
img_w ........................................... 224
|
| 209 |
+
indexer_batch_size .............................. 128
|
| 210 |
+
indexer_log_interval ............................ 1000
|
| 211 |
+
inference_batch_times_seqlen_threshold .......... -1
|
| 212 |
+
inference_dynamic_batching ...................... False
|
| 213 |
+
inference_dynamic_batching_buffer_guaranteed_fraction 0.2
|
| 214 |
+
inference_dynamic_batching_buffer_overflow_factor None
|
| 215 |
+
inference_dynamic_batching_buffer_size_gb ....... 40.0
|
| 216 |
+
inference_dynamic_batching_chunk_size ........... 256
|
| 217 |
+
inference_dynamic_batching_max_requests_override None
|
| 218 |
+
inference_dynamic_batching_max_tokens_override .. None
|
| 219 |
+
inference_dynamic_batching_num_cuda_graphs ...... 16
|
| 220 |
+
inference_max_batch_size ........................ 8
|
| 221 |
+
inference_max_seq_length ........................ 2560
|
| 222 |
+
inference_rng_tracker ........................... False
|
| 223 |
+
init_method_std ................................. 0.01
|
| 224 |
+
init_method_xavier_uniform ...................... False
|
| 225 |
+
init_model_with_meta_device ..................... False
|
| 226 |
+
initial_loss_scale .............................. 4294967296
|
| 227 |
+
inprocess_active_world_size ..................... 8
|
| 228 |
+
inprocess_barrier_timeout ....................... 120
|
| 229 |
+
inprocess_completion_timeout .................... 120
|
| 230 |
+
inprocess_empty_cuda_cache ...................... False
|
| 231 |
+
inprocess_granularity ........................... node
|
| 232 |
+
inprocess_hard_timeout .......................... 90
|
| 233 |
+
inprocess_heartbeat_interval .................... 30
|
| 234 |
+
inprocess_heartbeat_timeout ..................... 60
|
| 235 |
+
inprocess_last_call_wait ........................ 1
|
| 236 |
+
inprocess_max_iterations ........................ None
|
| 237 |
+
inprocess_monitor_process_interval .............. 1.0
|
| 238 |
+
inprocess_monitor_thread_interval ............... 1.0
|
| 239 |
+
inprocess_progress_watchdog_interval ............ 1.0
|
| 240 |
+
inprocess_restart ............................... False
|
| 241 |
+
inprocess_soft_timeout .......................... 60
|
| 242 |
+
inprocess_termination_grace_time ................ 1
|
| 243 |
+
is_hybrid_model ................................. False
|
| 244 |
+
iter_per_epoch .................................. 1250
|
| 245 |
+
iterations_to_skip .............................. []
|
| 246 |
+
keep_fp8_transpose_cache_when_using_custom_fsdp . False
|
| 247 |
+
kitchen_config_file ............................. None
|
| 248 |
+
kitchen_recipe_number ........................... None
|
| 249 |
+
kv_channels ..................................... 128
|
| 250 |
+
kv_lora_rank .................................... 32
|
| 251 |
+
lazy_mpu_init ................................... None
|
| 252 |
+
load ............................................ None
|
| 253 |
+
load_main_params_from_ckpt ...................... None
|
| 254 |
+
load_model_opt_format ........................... False
|
| 255 |
+
local_rank ...................................... 0
|
| 256 |
+
log_energy ...................................... False
|
| 257 |
+
log_interval .................................... 1
|
| 258 |
+
log_loss_scale_to_tensorboard ................... True
|
| 259 |
+
log_memory_to_tensorboard ....................... False
|
| 260 |
+
log_num_zeros_in_grad ........................... False
|
| 261 |
+
log_params_norm ................................. False
|
| 262 |
+
log_progress .................................... False
|
| 263 |
+
log_straggler ................................... False
|
| 264 |
+
log_throughput .................................. True
|
| 265 |
+
log_timers_to_tensorboard ....................... False
|
| 266 |
+
log_validation_ppl_to_tensorboard ............... False
|
| 267 |
+
log_world_size_to_tensorboard ................... False
|
| 268 |
+
logging_level ................................... None
|
| 269 |
+
loss_scale ...................................... None
|
| 270 |
+
loss_scale_window ............................... 1000
|
| 271 |
+
lr .............................................. 0.0001
|
| 272 |
+
lr_decay_iters .................................. 8
|
| 273 |
+
lr_decay_samples ................................ None
|
| 274 |
+
lr_decay_style .................................. cosine
|
| 275 |
+
lr_warmup_fraction .............................. None
|
| 276 |
+
lr_warmup_init .................................. 0.0
|
| 277 |
+
lr_warmup_iters ................................. 5
|
| 278 |
+
lr_warmup_samples ............................... 0
|
| 279 |
+
lr_wsd_decay_iters .............................. None
|
| 280 |
+
lr_wsd_decay_samples ............................ None
|
| 281 |
+
lr_wsd_decay_style .............................. exponential
|
| 282 |
+
main_grads_dtype ................................ torch.float32
|
| 283 |
+
main_params_dtype ............................... torch.float32
|
| 284 |
+
make_vocab_size_divisible_by .................... 128
|
| 285 |
+
mamba_head_dim .................................. 64
|
| 286 |
+
mamba_num_groups ................................ 8
|
| 287 |
+
mamba_num_heads ................................. None
|
| 288 |
+
mamba_state_dim ................................. 128
|
| 289 |
+
manual_gc ....................................... False
|
| 290 |
+
manual_gc_eval .................................. True
|
| 291 |
+
manual_gc_interval .............................. 0
|
| 292 |
+
mask_factor ..................................... 1.0
|
| 293 |
+
mask_prob ....................................... 0.15
|
| 294 |
+
mask_type ....................................... random
|
| 295 |
+
masked_softmax_fusion ........................... False
|
| 296 |
+
max_position_embeddings ......................... 32768
|
| 297 |
+
max_tokens_to_oom ............................... 12000
|
| 298 |
+
memory_snapshot_path ............................ snapshot.pickle
|
| 299 |
+
merge_file ...................................... /workspace/data/wikitext/gpt2-merges.txt
|
| 300 |
+
micro_batch_size ................................ 1
|
| 301 |
+
microbatch_group_size_per_vp_stage .............. None
|
| 302 |
+
mid_level_dataset_surplus ....................... 0.005
|
| 303 |
+
min_loss_scale .................................. 1.0
|
| 304 |
+
min_lr .......................................... 1e-05
|
| 305 |
+
mlp_chunks_for_prefill .......................... 1
|
| 306 |
+
mmap_bin_files .................................. True
|
| 307 |
+
mock_data ....................................... False
|
| 308 |
+
moe_apply_probs_on_input ........................ False
|
| 309 |
+
moe_aux_loss_coeff .............................. 0.01
|
| 310 |
+
moe_deepep_num_sms .............................. 20
|
| 311 |
+
moe_echo_enable_random_offloading ............... False
|
| 312 |
+
moe_echo_expert_dispatch_overlap ................ True
|
| 313 |
+
moe_echo_expert_dispatcher_type ................. hybridep
|
| 314 |
+
moe_echo_recompute_expert_dispatch .............. False
|
| 315 |
+
moe_enable_deepep ............................... False
|
| 316 |
+
moe_enable_echo ................................. True
|
| 317 |
+
moe_enable_hybridep ............................. True
|
| 318 |
+
moe_expert_capacity_factor ...................... None
|
| 319 |
+
moe_extended_tp ................................. False
|
| 320 |
+
moe_ffn_hidden_size ............................. 14336
|
| 321 |
+
moe_grouped_gemm ................................ True
|
| 322 |
+
moe_hybridep_num_sms ............................ 16
|
| 323 |
+
moe_input_jitter_eps ............................ None
|
| 324 |
+
moe_layer_freq .................................. 1
|
| 325 |
+
moe_layer_recompute ............................. False
|
| 326 |
+
moe_num_echo_experts ............................ 8
|
| 327 |
+
moe_pad_expert_input_to_capacity ................ False
|
| 328 |
+
moe_per_layer_logging ........................... False
|
| 329 |
+
moe_permute_fusion .............................. False
|
| 330 |
+
moe_received_token_capacity ..................... 2.40625
|
| 331 |
+
moe_router_bias_update_rate ..................... 0.001
|
| 332 |
+
moe_router_dtype ................................ fp32
|
| 333 |
+
moe_router_enable_expert_bias ................... False
|
| 334 |
+
moe_router_force_load_balancing ................. False
|
| 335 |
+
moe_router_group_topk ........................... None
|
| 336 |
+
moe_router_load_balancing_type .................. aux_loss
|
| 337 |
+
moe_router_num_groups ........................... None
|
| 338 |
+
moe_router_padding_for_fp8 ...................... False
|
| 339 |
+
moe_router_pre_softmax .......................... False
|
| 340 |
+
moe_router_score_function ....................... softmax
|
| 341 |
+
moe_router_topk ................................. 2
|
| 342 |
+
moe_router_topk_scaling_factor .................. None
|
| 343 |
+
moe_shared_expert_intermediate_size ............. None
|
| 344 |
+
moe_shared_expert_overlap ....................... False
|
| 345 |
+
moe_token_dispatcher_type ....................... flex
|
| 346 |
+
moe_token_drop_policy ........................... probs
|
| 347 |
+
moe_upcycling_granularity ....................... 1
|
| 348 |
+
moe_use_device_initiated_grouped_gemm ........... True
|
| 349 |
+
moe_use_legacy_grouped_gemm ..................... False
|
| 350 |
+
moe_use_upcycling ............................... False
|
| 351 |
+
moe_z_loss_coeff ................................ None
|
| 352 |
+
mrope_section ................................... None
|
| 353 |
+
mscale .......................................... 1.0
|
| 354 |
+
mscale_all_dim .................................. 0.0
|
| 355 |
+
mtp_loss_scaling_factor ......................... 0.1
|
| 356 |
+
mtp_num_layers .................................. None
|
| 357 |
+
multi_latent_attention .......................... False
|
| 358 |
+
nccl_all_reduce_for_prefill ..................... False
|
| 359 |
+
nccl_communicator_config_path ................... None
|
| 360 |
+
nccl_ub ......................................... False
|
| 361 |
+
no_load_optim ................................... None
|
| 362 |
+
no_load_rng ..................................... None
|
| 363 |
+
no_persist_layer_norm ........................... False
|
| 364 |
+
no_rope_freq .................................... None
|
| 365 |
+
no_save_optim ................................... None
|
| 366 |
+
no_save_rng ..................................... None
|
| 367 |
+
non_persistent_ckpt_type ........................ None
|
| 368 |
+
non_persistent_global_ckpt_dir .................. None
|
| 369 |
+
non_persistent_local_ckpt_algo .................. fully_parallel
|
| 370 |
+
non_persistent_local_ckpt_dir ................... None
|
| 371 |
+
non_persistent_save_interval .................... None
|
| 372 |
+
norm_epsilon .................................... 1e-05
|
| 373 |
+
normalization ................................... RMSNorm
|
| 374 |
+
num_attention_heads ............................. 32
|
| 375 |
+
num_channels .................................... 3
|
| 376 |
+
num_classes ..................................... 1000
|
| 377 |
+
num_dataset_builder_threads ..................... 1
|
| 378 |
+
num_distributed_optimizer_instances ............. 1
|
| 379 |
+
num_experts ..................................... 8
|
| 380 |
+
num_layers ...................................... 16
|
| 381 |
+
num_layers_at_end_in_bf16 ....................... 1
|
| 382 |
+
num_layers_at_start_in_bf16 ..................... 1
|
| 383 |
+
num_layers_per_virtual_pipeline_stage ........... None
|
| 384 |
+
num_query_groups ................................ 8
|
| 385 |
+
num_virtual_stages_per_pipeline_rank ............ None
|
| 386 |
+
num_workers ..................................... 2
|
| 387 |
+
object_storage_cache_path ....................... None
|
| 388 |
+
one_logger_async ................................ False
|
| 389 |
+
one_logger_project .............................. megatron-lm
|
| 390 |
+
one_logger_run_name ............................. None
|
| 391 |
+
onnx_safe ....................................... None
|
| 392 |
+
openai_gelu ..................................... False
|
| 393 |
+
optimizer ....................................... adam
|
| 394 |
+
optimizer_cpu_offload ........................... False
|
| 395 |
+
optimizer_offload_fraction ...................... 1.0
|
| 396 |
+
output_bert_embeddings .......................... False
|
| 397 |
+
overlap_cpu_optimizer_d2h_h2d ................... False
|
| 398 |
+
overlap_grad_reduce ............................. True
|
| 399 |
+
overlap_moe_expert_parallel_comm ................ False
|
| 400 |
+
overlap_p2p_comm ................................ False
|
| 401 |
+
overlap_p2p_comm_warmup_flush ................... False
|
| 402 |
+
overlap_param_gather ............................ True
|
| 403 |
+
overlap_param_gather_with_optimizer_step ........ False
|
| 404 |
+
override_opt_param_scheduler .................... False
|
| 405 |
+
params_dtype .................................... torch.bfloat16
|
| 406 |
+
patch_dim ....................................... 16
|
| 407 |
+
per_split_data_args_path ........................ None
|
| 408 |
+
perform_initialization .......................... True
|
| 409 |
+
pin_cpu_grads ................................... True
|
| 410 |
+
pin_cpu_params .................................. True
|
| 411 |
+
pipeline_model_parallel_comm_backend ............ None
|
| 412 |
+
pipeline_model_parallel_layout .................. None
|
| 413 |
+
pipeline_model_parallel_size .................... 1
|
| 414 |
+
position_embedding_type ......................... rope
|
| 415 |
+
pretrained_checkpoint ........................... None
|
| 416 |
+
profile ......................................... False
|
| 417 |
+
profile_ranks ................................... [0]
|
| 418 |
+
profile_step_end ................................ 12
|
| 419 |
+
profile_step_start .............................. 10
|
| 420 |
+
q_lora_rank ..................................... None
|
| 421 |
+
qk_head_dim ..................................... 128
|
| 422 |
+
qk_l2_norm ...................................... False
|
| 423 |
+
qk_layernorm .................................... False
|
| 424 |
+
qk_pos_emb_head_dim ............................. 64
|
| 425 |
+
query_in_block_prob ............................. 0.1
|
| 426 |
+
rampup_batch_size ............................... None
|
| 427 |
+
rank ............................................ 0
|
| 428 |
+
recompute_granularity ........................... None
|
| 429 |
+
recompute_method ................................ None
|
| 430 |
+
recompute_modules ............................... None
|
| 431 |
+
recompute_num_layers ............................ None
|
| 432 |
+
record_memory_history ........................... False
|
| 433 |
+
relative_attention_max_distance ................. 128
|
| 434 |
+
relative_attention_num_buckets .................. 32
|
| 435 |
+
replication ..................................... False
|
| 436 |
+
replication_factor .............................. 2
|
| 437 |
+
replication_jump ................................ None
|
| 438 |
+
rerun_mode ...................................... validate_results
|
| 439 |
+
reset_attention_mask ............................ False
|
| 440 |
+
reset_position_ids .............................. False
|
| 441 |
+
result_rejected_tracker_filename ................ None
|
| 442 |
+
retriever_report_topk_accuracies ................ []
|
| 443 |
+
retriever_score_scaling ......................... False
|
| 444 |
+
retriever_seq_length ............................ 256
|
| 445 |
+
retro_add_retriever ............................. False
|
| 446 |
+
retro_attention_gate ............................ 1
|
| 447 |
+
retro_cyclic_train_iters ........................ None
|
| 448 |
+
retro_encoder_attention_dropout ................. 0.1
|
| 449 |
+
retro_encoder_hidden_dropout .................... 0.1
|
| 450 |
+
retro_encoder_layers ............................ 2
|
| 451 |
+
retro_num_neighbors ............................. 2
|
| 452 |
+
retro_num_retrieved_chunks ...................... 2
|
| 453 |
+
retro_project_dir ............................... None
|
| 454 |
+
retro_verify_neighbor_count ..................... True
|
| 455 |
+
reuse_grad_buf_for_mxfp8_param_ag ............... False
|
| 456 |
+
rope_scaling_factor ............................. 8.0
|
| 457 |
+
rope_type ....................................... None
|
| 458 |
+
rotary_base ..................................... 1000000
|
| 459 |
+
rotary_interleaved .............................. False
|
| 460 |
+
rotary_percent .................................. 1.0
|
| 461 |
+
rotary_scaling_factor ........................... 1.0
|
| 462 |
+
rotary_seq_len_interpolation_factor ............. None
|
| 463 |
+
run_workload_inspector_server ................... False
|
| 464 |
+
sample_rate ..................................... 1.0
|
| 465 |
+
save ............................................ None
|
| 466 |
+
save_interval ................................... None
|
| 467 |
+
save_retain_interval ............................ None
|
| 468 |
+
scatter_gather_tensors_in_pipeline .............. True
|
| 469 |
+
seed ............................................ 1234
|
| 470 |
+
seq_length ...................................... 4096
|
| 471 |
+
sequence_parallel ............................... False
|
| 472 |
+
sft ............................................. False
|
| 473 |
+
sft_tokenizer_prompt_format ..................... nemotron-h-aligned
|
| 474 |
+
sgd_momentum .................................... 0.9
|
| 475 |
+
short_seq_prob .................................. 0.1
|
| 476 |
+
skip_train ...................................... False
|
| 477 |
+
skipped_train_samples ........................... 0
|
| 478 |
+
spec ............................................ None
|
| 479 |
+
split ........................................... 100,0,0
|
| 480 |
+
squared_relu .................................... False
|
| 481 |
+
start_weight_decay .............................. 0.1
|
| 482 |
+
straggler_ctrlr_port ............................ 65535
|
| 483 |
+
straggler_minmax_count .......................... 1
|
| 484 |
+
suggested_communication_unit_size ............... None
|
| 485 |
+
swiglu .......................................... True
|
| 486 |
+
swin_backbone_type .............................. tiny
|
| 487 |
+
symmetric_ar_type ............................... None
|
| 488 |
+
te_rng_tracker .................................. True
|
| 489 |
+
tensor_model_parallel_size ...................... 1
|
| 490 |
+
tensorboard_dir ................................. None
|
| 491 |
+
tensorboard_log_interval ........................ 1
|
| 492 |
+
tensorboard_queue_size .......................... 1000
|
| 493 |
+
test_data_path .................................. None
|
| 494 |
+
test_mode ....................................... False
|
| 495 |
+
tiktoken_num_special_tokens ..................... 1000
|
| 496 |
+
tiktoken_pattern ................................ None
|
| 497 |
+
tiktoken_special_tokens ......................... None
|
| 498 |
+
timing_log_level ................................ 0
|
| 499 |
+
timing_log_option ............................... minmax
|
| 500 |
+
titles_data_path ................................ None
|
| 501 |
+
tokenizer_model ................................. None
|
| 502 |
+
tokenizer_type .................................. GPT2BPETokenizer
|
| 503 |
+
torch_fsdp2_reshard_after_forward ............... True
|
| 504 |
+
tp_comm_bootstrap_backend ....................... nccl
|
| 505 |
+
tp_comm_bulk_dgrad .............................. True
|
| 506 |
+
tp_comm_bulk_wgrad .............................. True
|
| 507 |
+
tp_comm_overlap ................................. False
|
| 508 |
+
tp_comm_overlap_ag .............................. True
|
| 509 |
+
tp_comm_overlap_cfg ............................. None
|
| 510 |
+
tp_comm_overlap_rs .............................. True
|
| 511 |
+
tp_comm_overlap_rs_dgrad ........................ False
|
| 512 |
+
tp_comm_split_ag ................................ True
|
| 513 |
+
tp_comm_split_rs ................................ True
|
| 514 |
+
train_data_path ................................. None
|
| 515 |
+
train_iters ..................................... 12
|
| 516 |
+
train_samples ................................... None
|
| 517 |
+
train_sync_interval ............................. None
|
| 518 |
+
transformer_impl ................................ transformer_engine
|
| 519 |
+
transformer_pipeline_model_parallel_size ........ 1
|
| 520 |
+
untie_embeddings_and_output_weights ............. True
|
| 521 |
+
use_checkpoint_args ............................. False
|
| 522 |
+
use_checkpoint_opt_param_scheduler .............. False
|
| 523 |
+
use_cpu_initialization .......................... None
|
| 524 |
+
use_custom_fsdp ................................. False
|
| 525 |
+
use_dist_ckpt ................................... True
|
| 526 |
+
use_dist_ckpt_deprecated ........................ False
|
| 527 |
+
use_distributed_optimizer ....................... True
|
| 528 |
+
use_flash_attn .................................. False
|
| 529 |
+
use_legacy_models ............................... False
|
| 530 |
+
use_mp_args_from_checkpoint_args ................ False
|
| 531 |
+
use_one_sent_docs ............................... False
|
| 532 |
+
use_persistent_ckpt_worker ...................... False
|
| 533 |
+
use_precision_aware_optimizer ................... False
|
| 534 |
+
use_pytorch_profiler ............................ False
|
| 535 |
+
use_ring_exchange_p2p ........................... False
|
| 536 |
+
use_rope_scaling ................................ False
|
| 537 |
+
use_rotary_position_embeddings .................. False
|
| 538 |
+
use_sharp ....................................... False
|
| 539 |
+
use_tokenizer_model_from_checkpoint_args ........ True
|
| 540 |
+
use_torch_fsdp2 ................................. False
|
| 541 |
+
use_torch_optimizer_for_cpu_offload ............. False
|
| 542 |
+
use_tp_pp_dp_mapping ............................ False
|
| 543 |
+
v_head_dim ...................................... 128
|
| 544 |
+
valid_data_path ................................. None
|
| 545 |
+
variable_seq_lengths ............................ False
|
| 546 |
+
virtual_pipeline_model_parallel_size ............ None
|
| 547 |
+
vision_backbone_type ............................ vit
|
| 548 |
+
vision_pretraining .............................. False
|
| 549 |
+
vision_pretraining_type ......................... classify
|
| 550 |
+
vocab_extra_ids ................................. 0
|
| 551 |
+
vocab_file ...................................... /workspace/data/wikitext/gpt2-vocab.json
|
| 552 |
+
vocab_size ...................................... None
|
| 553 |
+
wandb_exp_name ..................................
|
| 554 |
+
wandb_project ...................................
|
| 555 |
+
wandb_save_dir ..................................
|
| 556 |
+
weight_decay .................................... 0.1
|
| 557 |
+
weight_decay_incr_style ......................... constant
|
| 558 |
+
wgrad_deferral_limit ............................ 0
|
| 559 |
+
world_size ...................................... 8
|
| 560 |
+
yaml_cfg ........................................ None
|
| 561 |
+
-------------------- end of arguments ---------------------
|
| 562 |
+
INFO:megatron.core.num_microbatches_calculator:setting number of microbatches to constant 2
|
| 563 |
+
> building GPT2BPETokenizer tokenizer ...
|
| 564 |
+
/workspace/Megatron-MoE-Echo/megatron/training/arguments.py:831: UserWarning: Disabling sequence parallelism because tensor model parallelism is disabled
|
| 565 |
+
warnings.warn("Disabling sequence parallelism because tensor model parallelism is disabled")
|
| 566 |
+
/workspace/Megatron-MoE-Echo/megatron/training/arguments.py:831: UserWarning: Disabling sequence parallelism because tensor model parallelism is disabled
|
| 567 |
+
warnings.warn("Disabling sequence parallelism because tensor model parallelism is disabled")
|
| 568 |
+
/workspace/Megatron-MoE-Echo/megatron/training/arguments.py:831: UserWarning: Disabling sequence parallelism because tensor model parallelism is disabled
|
| 569 |
+
warnings.warn("Disabling sequence parallelism because tensor model parallelism is disabled")
|
| 570 |
+
> padded vocab (size: 50257) with 47 dummy tokens (new size: 50304)
|
| 571 |
+
WARNING:megatron.core.rerun_state_machine:RerunStateMachine initialized in mode RerunMode.VALIDATE_RESULTS
|
| 572 |
+
> initializing torch distributed ...
|
| 573 |
+
/workspace/Megatron-MoE-Echo/megatron/training/arguments.py:831: UserWarning: Disabling sequence parallelism because tensor model parallelism is disabled
|
| 574 |
+
warnings.warn("Disabling sequence parallelism because tensor model parallelism is disabled")
|
| 575 |
+
WARNING: one_logger package is required to enable e2e metrics tracking. please go to https://confluence.nvidia.com/display/MLWFO/Package+Repositories for details to install it
|
| 576 |
+
[Gloo] Rank 0 is connected to 7 peer ranks. Expected number of connected peer ranks is : 7
|
| 577 |
+
[Gloo] Rank [Gloo] Rank 1[Gloo] Rank [Gloo] Rank is connected to 2473 is connected to is connected to peer ranks. is connected to 77Expected number of connected peer ranks is : 7 peer ranks. peer ranks. [Gloo] Rank 7 peer ranks.
|
| 578 |
+
Expected number of connected peer ranks is : Expected number of connected peer ranks is : 5Expected number of connected peer ranks is : 77 is connected to 7
|
| 579 |
+
|
| 580 |
+
|
| 581 |
+
7[Gloo] Rank [Gloo] Rank peer ranks. 7Expected number of connected peer ranks is : is connected to 776
|
| 582 |
+
peer ranks. is connected to Expected number of connected peer ranks is : 77 peer ranks.
|
| 583 |
+
Expected number of connected peer ranks is : 7
|
| 584 |
+
[Gloo] Rank 0 is connected to 7 peer ranks. Expected number of connected peer ranks is : 7
|
| 585 |
+
[Gloo] Rank 1 is connected to 7 peer ranks. Expected number of connected peer ranks is : 7
|
| 586 |
+
[Gloo] Rank 2 is connected to 7 peer ranks. Expected number of connected peer ranks is : 7
|
| 587 |
+
[Gloo] Rank 3 is connected to 7 peer ranks. Expected number of connected peer ranks is : 7
|
| 588 |
+
[Gloo] Rank 4 is connected to 7 peer ranks. Expected number of connected peer ranks is : 7
|
| 589 |
+
[Gloo] Rank 5 is connected to 7 peer ranks. Expected number of connected peer ranks is : 7
|
| 590 |
+
[Gloo] Rank [Gloo] Rank 7 is connected to 67 is connected to peer ranks. 7Expected number of connected peer ranks is : peer ranks. 7Expected number of connected peer ranks is : 7
|
| 591 |
+
|
| 592 |
+
[Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
|
| 593 |
+
> initialized tensor model parallel with size 1
|
| 594 |
+
> initialized pipeline model parallel with size 1
|
| 595 |
+
> setting random seeds to 1234 ...
|
| 596 |
+
[Gloo] Rank [Gloo] Rank 00 is connected to is connected to 00 peer ranks. peer ranks. Expected number of connected peer ranks is : Expected number of connected peer ranks is : 00
|
| 597 |
+
|
| 598 |
+
[Gloo] Rank [Gloo] Rank 00 is connected to is connected to 00 peer ranks. peer ranks. Expected number of connected peer ranks is : Expected number of connected peer ranks is : 00
|
| 599 |
+
|
| 600 |
+
[Gloo] Rank [Gloo] Rank 00 is connected to is connected to 00 peer ranks. peer ranks. Expected number of connected peer ranks is : Expected number of connected peer ranks is : 00
|
| 601 |
+
|
| 602 |
+
[Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
|
| 603 |
+
[rank3]:[W830 06:44:47.018891615 ProcessGroupNCCL.cpp:4789] [PG ID 0 PG GUID 0 Rank 3] using GPU 3 as device used by this process is currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect. You can specify device_id in init_process_group() to force use of a particular device.
|
| 604 |
+
[rank1]:[W830 06:44:47.018917720 ProcessGroupNCCL.cpp:4789] [PG ID 0 PG GUID 0 Rank 1] using GPU 1 as device used by this process is currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect. You can specify device_id in init_process_group() to force use of a particular device.
|
| 605 |
+
[rank7]:[W830 06:44:47.018941474 ProcessGroupNCCL.cpp:4789] [PG ID 0 PG GUID 0 Rank 7] using GPU 7 as device used by this process is currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect. You can specify device_id in init_process_group() to force use of a particular device.
|
| 606 |
+
[rank4]:[W830 06:44:47.018962476 ProcessGroupNCCL.cpp:4789] [PG ID 0 PG GUID 0 Rank 4] using GPU 4 as device used by this process is currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect. You can specify device_id in init_process_group() to force use of a particular device.
|
| 607 |
+
[rank5]:[W830 06:44:47.018992957 ProcessGroupNCCL.cpp:4789] [PG ID 0 PG GUID 0 Rank 5] using GPU 5 as device used by this process is currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect. You can specify device_id in init_process_group() to force use of a particular device.
|
| 608 |
+
[rank2]:[W830 06:44:47.019052212 ProcessGroupNCCL.cpp:4789] [PG ID 0 PG GUID 0 Rank 2] using GPU 2 as device used by this process is currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect. You can specify device_id in init_process_group() to force use of a particular device.
|
| 609 |
+
[rank6]:[W830 06:44:47.020946712 ProcessGroupNCCL.cpp:4789] [PG ID 0 PG GUID 0 Rank 6] using GPU 6 as device used by this process is currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect. You can specify device_id in init_process_group() to force use of a particular device.
|
| 610 |
+
> compiling dataset index builder ...
|
| 611 |
+
make: Entering directory '/workspace/Megatron-MoE-Echo/megatron/core/datasets'
|
| 612 |
+
make: Nothing to be done for 'default'.
|
| 613 |
+
make: Leaving directory '/workspace/Megatron-MoE-Echo/megatron/core/datasets'
|
| 614 |
+
>>> done with dataset index builder. Compilation time: 0.035 seconds
|
| 615 |
+
WARNING: constraints for invoking optimized fused softmax kernel are not met. We default back to unfused kernel invocations.
|
| 616 |
+
> compiling and loading fused kernels ...
|
| 617 |
+
[rank0]:[W830 06:44:47.177209335 ProcessGroupNCCL.cpp:4789] [PG ID 0 PG GUID 0 Rank 0] using GPU 0 as device used by this process is currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect. You can specify device_id in init_process_group() to force use of a particular device.
|
| 618 |
+
>>> done with compiling and loading fused kernels. Compilation time: 3.726 seconds
|
| 619 |
+
time to initialize megatron (seconds): 6.816
|
| 620 |
+
[after megatron is initialized] datetime: 2026-08-30 06:44:53
|
| 621 |
+
building GPT model ...
|
| 622 |
+
/usr/local/lib/python3.12/dist-packages/transformer_engine/pytorch/cpu_offload.py:708: DeprecationWarning: Offloading weights is deprecated. Using offload_weights=True does not have any effect.
|
| 623 |
+
warnings.warn(
|
| 624 |
+
/usr/local/lib/python3.12/dist-packages/transformer_engine/pytorch/cpu_offload.py:708: DeprecationWarning: Offloading weights is deprecated. Using offload_weights=True does not have any effect.
|
| 625 |
+
warnings.warn(
|
| 626 |
+
/usr/local/lib/python3.12/dist-packages/transformer_engine/pytorch/cpu_offload.py:708: DeprecationWarning: Offloading weights is deprecated. Using offload_weights=True does not have any effect.
|
| 627 |
+
warnings.warn(
|
| 628 |
+
/usr/local/lib/python3.12/dist-packages/transformer_engine/pytorch/cpu_offload.py:708: DeprecationWarning: Offloading weights is deprecated. Using offload_weights=True does not have any effect.
|
| 629 |
+
warnings.warn(
|
| 630 |
+
/usr/local/lib/python3.12/dist-packages/transformer_engine/pytorch/cpu_offload.py:708: DeprecationWarning: Offloading weights is deprecated. Using offload_weights=True does not have any effect.
|
| 631 |
+
warnings.warn(
|
| 632 |
+
/usr/local/lib/python3.12/dist-packages/transformer_engine/pytorch/cpu_offload.py:708: DeprecationWarning: Offloading weights is deprecated. Using offload_weights=True does not have any effect.
|
| 633 |
+
warnings.warn(
|
| 634 |
+
/usr/local/lib/python3.12/dist-packages/transformer_engine/pytorch/cpu_offload.py:708: DeprecationWarning: Offloading weights is deprecated. Using offload_weights=True does not have any effect.
|
| 635 |
+
warnings.warn(
|
| 636 |
+
/usr/local/lib/python3.12/dist-packages/transformer_engine/pytorch/cpu_offload.py:708: DeprecationWarning: Offloading weights is deprecated. Using offload_weights=True does not have any effect.
|
| 637 |
+
warnings.warn(
|
| 638 |
+
> number of parameters on (tensor, pipeline) model parallel rank (0, 0): 3902410752
|
| 639 |
+
INFO:megatron.core.distributed.distributed_data_parallel:Setting up DistributedDataParallel with config DistributedDataParallelConfig(grad_reduce_in_fp32=True, overlap_grad_reduce=True, overlap_param_gather=True, align_param_gather=False, use_distributed_optimizer=True, num_distributed_optimizer_instances=1, check_for_nan_in_grad=False, check_for_large_grads=False, bucket_size=40000000, pad_buckets_for_high_nccl_busbw=False, average_in_collective=False, fp8_param_gather=False, reuse_grad_buf_for_mxfp8_param_ag=False, use_custom_fsdp=False, data_parallel_sharding_strategy='no_shard', gradient_reduce_div_fusion=True, suggested_communication_unit_size=None, preserve_fp32_weights=True, keep_fp8_transpose_cache_when_using_custom_fsdp=False, nccl_ub=False, fsdp_double_buffer=False)
|
| 640 |
+
INFO:megatron.core.distributed.param_and_grad_buffer:Number of buckets for gradient all-reduce / reduce-scatter: 18
|
| 641 |
+
Params for bucket 1 (206045184 elements, 206045184 padded size):
|
| 642 |
+
module.output_layer.weight
|
| 643 |
+
Params for bucket 2 (41988096 elements, 41988096 padded size):
|
| 644 |
+
module.decoder.final_layernorm.weight
|
| 645 |
+
module.decoder.layers.15.self_attention.linear_qkv.weight
|
| 646 |
+
module.decoder.layers.15.pre_mlp_layernorm.weight
|
| 647 |
+
module.decoder.layers.15.mlp.router.weight
|
| 648 |
+
module.decoder.layers.15.self_attention.linear_qkv.layer_norm_weight
|
| 649 |
+
module.decoder.layers.15.self_attention.linear_proj.weight
|
| 650 |
+
Params for bucket 3 (41984000 elements, 41984000 padded size):
|
| 651 |
+
module.decoder.layers.14.mlp.router.weight
|
| 652 |
+
module.decoder.layers.14.self_attention.linear_proj.weight
|
| 653 |
+
module.decoder.layers.14.self_attention.linear_qkv.layer_norm_weight
|
| 654 |
+
module.decoder.layers.14.pre_mlp_layernorm.weight
|
| 655 |
+
module.decoder.layers.14.self_attention.linear_qkv.weight
|
| 656 |
+
Params for bucket 4 (41984000 elements, 41984000 padded size):
|
| 657 |
+
module.decoder.layers.13.self_attention.linear_qkv.weight
|
| 658 |
+
module.decoder.layers.13.pre_mlp_layernorm.weight
|
| 659 |
+
module.decoder.layers.13.mlp.router.weight
|
| 660 |
+
module.decoder.layers.13.self_attention.linear_proj.weight
|
| 661 |
+
module.decoder.layers.13.self_attention.linear_qkv.layer_norm_weight
|
| 662 |
+
Params for bucket 5 (41984000 elements, 41984000 padded size):
|
| 663 |
+
module.decoder.layers.12.mlp.router.weight
|
| 664 |
+
module.decoder.layers.12.self_attention.linear_proj.weight
|
| 665 |
+
module.decoder.layers.12.self_attention.linear_qkv.layer_norm_weight
|
| 666 |
+
module.decoder.layers.12.pre_mlp_layernorm.weight
|
| 667 |
+
module.decoder.layers.12.self_attention.linear_qkv.weight
|
| 668 |
+
Params for bucket 6 (41984000 elements, 41984000 padded size):
|
| 669 |
+
module.decoder.layers.11.self_attention.linear_qkv.weight
|
| 670 |
+
module.decoder.layers.11.pre_mlp_layernorm.weight
|
| 671 |
+
module.decoder.layers.11.mlp.router.weight
|
| 672 |
+
module.decoder.layers.11.self_attention.linear_proj.weight
|
| 673 |
+
module.decoder.layers.11.self_attention.linear_qkv.layer_norm_weight
|
| 674 |
+
Params for bucket 7 (41984000 elements, 41984000 padded size):
|
| 675 |
+
module.decoder.layers.10.mlp.router.weight
|
| 676 |
+
module.decoder.layers.10.self_attention.linear_qkv.weight
|
| 677 |
+
module.decoder.layers.10.pre_mlp_layernorm.weight
|
| 678 |
+
module.decoder.layers.10.self_attention.linear_proj.weight
|
| 679 |
+
module.decoder.layers.10.self_attention.linear_qkv.layer_norm_weight
|
| 680 |
+
Params for bucket 8 (41984000 elements, 41984000 padded size):
|
| 681 |
+
module.decoder.layers.9.mlp.router.weight
|
| 682 |
+
module.decoder.layers.9.self_attention.linear_proj.weight
|
| 683 |
+
module.decoder.layers.9.self_attention.linear_qkv.layer_norm_weight
|
| 684 |
+
module.decoder.layers.9.self_attention.linear_qkv.weight
|
| 685 |
+
module.decoder.layers.9.pre_mlp_layernorm.weight
|
| 686 |
+
Params for bucket 9 (41984000 elements, 41984000 padded size):
|
| 687 |
+
module.decoder.layers.8.self_attention.linear_qkv.layer_norm_weight
|
| 688 |
+
module.decoder.layers.8.self_attention.linear_proj.weight
|
| 689 |
+
module.decoder.layers.8.mlp.router.weight
|
| 690 |
+
module.decoder.layers.8.self_attention.linear_qkv.weight
|
| 691 |
+
module.decoder.layers.8.pre_mlp_layernorm.weight
|
| 692 |
+
Params for bucket 10 (41984000 elements, 41984000 padded size):
|
| 693 |
+
module.decoder.layers.7.mlp.router.weight
|
| 694 |
+
module.decoder.layers.7.pre_mlp_layernorm.weight
|
| 695 |
+
module.decoder.layers.7.self_attention.linear_qkv.weight
|
| 696 |
+
module.decoder.layers.7.self_attention.linear_proj.weight
|
| 697 |
+
module.decoder.layers.7.self_attention.linear_qkv.layer_norm_weight
|
| 698 |
+
Params for bucket 11 (41984000 elements, 41984000 padded size):
|
| 699 |
+
module.decoder.layers.6.mlp.router.weight
|
| 700 |
+
module.decoder.layers.6.self_attention.linear_qkv.layer_norm_weight
|
| 701 |
+
module.decoder.layers.6.self_attention.linear_proj.weight
|
| 702 |
+
module.decoder.layers.6.pre_mlp_layernorm.weight
|
| 703 |
+
module.decoder.layers.6.self_attention.linear_qkv.weight
|
| 704 |
+
Params for bucket 12 (41984000 elements, 41984000 padded size):
|
| 705 |
+
module.decoder.layers.5.mlp.router.weight
|
| 706 |
+
module.decoder.layers.5.self_attention.linear_qkv.layer_norm_weight
|
| 707 |
+
module.decoder.layers.5.self_attention.linear_qkv.weight
|
| 708 |
+
module.decoder.layers.5.pre_mlp_layernorm.weight
|
| 709 |
+
module.decoder.layers.5.self_attention.linear_proj.weight
|
| 710 |
+
Params for bucket 13 (41984000 elements, 41984000 padded size):
|
| 711 |
+
module.decoder.layers.4.mlp.router.weight
|
| 712 |
+
module.decoder.layers.4.self_attention.linear_qkv.layer_norm_weight
|
| 713 |
+
module.decoder.layers.4.self_attention.linear_proj.weight
|
| 714 |
+
module.decoder.layers.4.self_attention.linear_qkv.weight
|
| 715 |
+
module.decoder.layers.4.pre_mlp_layernorm.weight
|
| 716 |
+
Params for bucket 14 (41984000 elements, 41984000 padded size):
|
| 717 |
+
module.decoder.layers.3.self_attention.linear_qkv.weight
|
| 718 |
+
module.decoder.layers.3.pre_mlp_layernorm.weight
|
| 719 |
+
module.decoder.layers.3.mlp.router.weight
|
| 720 |
+
module.decoder.layers.3.self_attention.linear_proj.weight
|
| 721 |
+
module.decoder.layers.3.self_attention.linear_qkv.layer_norm_weight
|
| 722 |
+
Params for bucket 15 (41984000 elements, 41984000 padded size):
|
| 723 |
+
module.decoder.layers.2.mlp.router.weight
|
| 724 |
+
module.decoder.layers.2.self_attention.linear_qkv.layer_norm_weight
|
| 725 |
+
module.decoder.layers.2.self_attention.linear_proj.weight
|
| 726 |
+
module.decoder.layers.2.self_attention.linear_qkv.weight
|
| 727 |
+
module.decoder.layers.2.pre_mlp_layernorm.weight
|
| 728 |
+
Params for bucket 16 (41984000 elements, 41984000 padded size):
|
| 729 |
+
module.decoder.layers.1.self_attention.linear_qkv.layer_norm_weight
|
| 730 |
+
module.decoder.layers.1.self_attention.linear_proj.weight
|
| 731 |
+
module.decoder.layers.1.self_attention.linear_qkv.weight
|
| 732 |
+
module.decoder.layers.1.mlp.router.weight
|
| 733 |
+
module.decoder.layers.1.pre_mlp_layernorm.weight
|
| 734 |
+
Params for bucket 17 (41984000 elements, 41984000 padded size):
|
| 735 |
+
module.decoder.layers.0.mlp.router.weight
|
| 736 |
+
module.decoder.layers.0.pre_mlp_layernorm.weight
|
| 737 |
+
module.decoder.layers.0.self_attention.linear_qkv.layer_norm_weight
|
| 738 |
+
module.decoder.layers.0.self_attention.linear_qkv.weight
|
| 739 |
+
module.decoder.layers.0.self_attention.linear_proj.weight
|
| 740 |
+
Params for bucket 18 (206045184 elements, 206045184 padded size):
|
| 741 |
+
module.embedding.word_embeddings.weight
|
| 742 |
+
INFO:megatron.core.distributed.param_and_grad_buffer:Number of buckets for gradient all-reduce / reduce-scatter: 32
|
| 743 |
+
Params for bucket 1 (58720256 elements, 58720256 padded size):
|
| 744 |
+
module.decoder.layers.15.mlp.experts.linear_fc2.weight0
|
| 745 |
+
Params for bucket 2 (117440512 elements, 117440512 padded size):
|
| 746 |
+
module.decoder.layers.15.mlp.experts.linear_fc1.weight0
|
| 747 |
+
Params for bucket 3 (58720256 elements, 58720256 padded size):
|
| 748 |
+
module.decoder.layers.14.mlp.experts.linear_fc2.weight0
|
| 749 |
+
Params for bucket 4 (117440512 elements, 117440512 padded size):
|
| 750 |
+
module.decoder.layers.14.mlp.experts.linear_fc1.weight0
|
| 751 |
+
Params for bucket 5 (58720256 elements, 58720256 padded size):
|
| 752 |
+
module.decoder.layers.13.mlp.experts.linear_fc2.weight0
|
| 753 |
+
Params for bucket 6 (117440512 elements, 117440512 padded size):
|
| 754 |
+
module.decoder.layers.13.mlp.experts.linear_fc1.weight0
|
| 755 |
+
Params for bucket 7 (58720256 elements, 58720256 padded size):
|
| 756 |
+
module.decoder.layers.12.mlp.experts.linear_fc2.weight0
|
| 757 |
+
Params for bucket 8 (117440512 elements, 117440512 padded size):
|
| 758 |
+
module.decoder.layers.12.mlp.experts.linear_fc1.weight0
|
| 759 |
+
Params for bucket 9 (58720256 elements, 58720256 padded size):
|
| 760 |
+
module.decoder.layers.11.mlp.experts.linear_fc2.weight0
|
| 761 |
+
Params for bucket 10 (117440512 elements, 117440512 padded size):
|
| 762 |
+
module.decoder.layers.11.mlp.experts.linear_fc1.weight0
|
| 763 |
+
Params for bucket 11 (58720256 elements, 58720256 padded size):
|
| 764 |
+
module.decoder.layers.10.mlp.experts.linear_fc2.weight0
|
| 765 |
+
Params for bucket 12 (117440512 elements, 117440512 padded size):
|
| 766 |
+
module.decoder.layers.10.mlp.experts.linear_fc1.weight0
|
| 767 |
+
Params for bucket 13 (58720256 elements, 58720256 padded size):
|
| 768 |
+
module.decoder.layers.9.mlp.experts.linear_fc2.weight0
|
| 769 |
+
Params for bucket 14 (117440512 elements, 117440512 padded size):
|
| 770 |
+
module.decoder.layers.9.mlp.experts.linear_fc1.weight0
|
| 771 |
+
Params for bucket 15 (58720256 elements, 58720256 padded size):
|
| 772 |
+
module.decoder.layers.8.mlp.experts.linear_fc2.weight0
|
| 773 |
+
Params for bucket 16 (117440512 elements, 117440512 padded size):
|
| 774 |
+
module.decoder.layers.8.mlp.experts.linear_fc1.weight0
|
| 775 |
+
Params for bucket 17 (58720256 elements, 58720256 padded size):
|
| 776 |
+
module.decoder.layers.7.mlp.experts.linear_fc2.weight0
|
| 777 |
+
Params for bucket 18 (117440512 elements, 117440512 padded size):
|
| 778 |
+
module.decoder.layers.7.mlp.experts.linear_fc1.weight0
|
| 779 |
+
Params for bucket 19 (58720256 elements, 58720256 padded size):
|
| 780 |
+
module.decoder.layers.6.mlp.experts.linear_fc2.weight0
|
| 781 |
+
Params for bucket 20 (117440512 elements, 117440512 padded size):
|
| 782 |
+
module.decoder.layers.6.mlp.experts.linear_fc1.weight0
|
| 783 |
+
Params for bucket 21 (58720256 elements, 58720256 padded size):
|
| 784 |
+
module.decoder.layers.5.mlp.experts.linear_fc2.weight0
|
| 785 |
+
Params for bucket 22 (117440512 elements, 117440512 padded size):
|
| 786 |
+
module.decoder.layers.5.mlp.experts.linear_fc1.weight0
|
| 787 |
+
Params for bucket 23 (58720256 elements, 58720256 padded size):
|
| 788 |
+
module.decoder.layers.4.mlp.experts.linear_fc2.weight0
|
| 789 |
+
Params for bucket 24 (117440512 elements, 117440512 padded size):
|
| 790 |
+
module.decoder.layers.4.mlp.experts.linear_fc1.weight0
|
| 791 |
+
Params for bucket 25 (58720256 elements, 58720256 padded size):
|
| 792 |
+
module.decoder.layers.3.mlp.experts.linear_fc2.weight0
|
| 793 |
+
Params for bucket 26 (117440512 elements, 117440512 padded size):
|
| 794 |
+
module.decoder.layers.3.mlp.experts.linear_fc1.weight0
|
| 795 |
+
Params for bucket 27 (58720256 elements, 58720256 padded size):
|
| 796 |
+
module.decoder.layers.2.mlp.experts.linear_fc2.weight0
|
| 797 |
+
Params for bucket 28 (117440512 elements, 117440512 padded size):
|
| 798 |
+
module.decoder.layers.2.mlp.experts.linear_fc1.weight0
|
| 799 |
+
Params for bucket 29 (58720256 elements, 58720256 padded size):
|
| 800 |
+
module.decoder.layers.1.mlp.experts.linear_fc2.weight0
|
| 801 |
+
Params for bucket 30 (117440512 elements, 117440512 padded size):
|
| 802 |
+
module.decoder.layers.1.mlp.experts.linear_fc1.weight0
|
| 803 |
+
Params for bucket 31 (58720256 elements, 58720256 padded size):
|
| 804 |
+
module.decoder.layers.0.mlp.experts.linear_fc2.weight0
|
| 805 |
+
Params for bucket 32 (117440512 elements, 117440512 padded size):
|
| 806 |
+
module.decoder.layers.0.mlp.experts.linear_fc1.weight0
|
| 807 |
+
INFO:megatron.core.optimizer:Setting up optimizer with config OptimizerConfig(optimizer='adam', lr=0.0001, min_lr=1e-05, decoupled_lr=None, decoupled_min_lr=None, weight_decay=0.1, fp8_recipe='mxfp8', fp16=False, bf16=True, reuse_grad_buf_for_mxfp8_param_ag=False, params_dtype=torch.bfloat16, use_precision_aware_optimizer=False, store_param_remainders=True, main_grads_dtype=torch.float32, main_params_dtype=torch.float32, exp_avg_dtype=torch.float32, exp_avg_sq_dtype=torch.float32, loss_scale=None, initial_loss_scale=4294967296, min_loss_scale=1.0, loss_scale_window=1000, hysteresis=2, adam_beta1=0.9, adam_beta2=0.999, adam_eps=1e-08, sgd_momentum=0.9, use_distributed_optimizer=True, overlap_param_gather_with_optimizer_step=False, optimizer_cpu_offload=False, optimizer_offload_fraction=1.0, use_torch_optimizer_for_cpu_offload=False, overlap_cpu_optimizer_d2h_h2d=False, pin_cpu_grads=True, pin_cpu_params=True, clip_grad=1.0, log_num_zeros_in_grad=False, barrier_with_L1_time=True, timers=<megatron.core.timers.Timers object at 0x72cef00f86e0>, config_logger_dir='')
|
| 808 |
+
INFO:megatron.core.optimizer_param_scheduler:> learning rate decay style: cosine
|
| 809 |
+
[after model, optimizer, and learning rate scheduler are built] datetime: 2026-08-30 06:44:53
|
| 810 |
+
> building train, validation, and test datasets ...
|
| 811 |
+
> datasets target sizes (minimum size):
|
| 812 |
+
train: 192
|
| 813 |
+
validation: 0
|
| 814 |
+
test: 0
|
| 815 |
+
INFO:megatron.core.datasets.blended_megatron_dataset_config:Let split_matrix = [(0, 1.0), None, None]
|
| 816 |
+
> building train, validation, and test datasets for GPT ...
|
| 817 |
+
INFO:megatron.core.datasets.blended_megatron_dataset_builder:Building GPTDataset splits with sizes=(192, 0, 0) and config=GPTDatasetConfig(random_seed=1234, sequence_length=4096, blend=(['/workspace/data/wikitext/wikitext103_gpt2_text_document'], None), blend_per_split=None, split='100,0,0', split_matrix=[(0, 1.0), None, None], num_dataset_builder_threads=1, path_to_cache=None, mmap_bin_files=True, mock=False, tokenizer=<megatron.training.tokenizer.tokenizer._GPT2BPETokenizer object at 0x72cef03b4ef0>, mid_level_dataset_surplus=0.005, reset_position_ids=False, reset_attention_mask=False, eod_mask_loss=False, create_attention_mask=True, drop_last_partial_validation_sequence=True, add_extra_token_to_sequence=True, object_storage_cache_path=None)
|
| 818 |
+
INFO:megatron.core.datasets.indexed_dataset:Load the _IndexReader from /workspace/data/wikitext/wikitext103_gpt2_text_document.idx
|
| 819 |
+
INFO:megatron.core.datasets.indexed_dataset: Extract the sequence lengths
|
| 820 |
+
INFO:megatron.core.datasets.indexed_dataset: Extract the sequence pointers
|
| 821 |
+
INFO:megatron.core.datasets.indexed_dataset: Extract the document indices
|
| 822 |
+
INFO:megatron.core.datasets.indexed_dataset:> total number of sequences: 1165029
|
| 823 |
+
INFO:megatron.core.datasets.indexed_dataset:> total number of documents: 1801350
|
| 824 |
+
INFO:megatron.core.datasets.gpt_dataset:Load the GPTDataset train indices
|
| 825 |
+
INFO:megatron.core.datasets.gpt_dataset: Load the document index from 44ed1ee5bcb9e5b030387b8d27a6c406-GPTDataset-train-document_index.npy
|
| 826 |
+
INFO:megatron.core.datasets.gpt_dataset: Load the sample index from 44ed1ee5bcb9e5b030387b8d27a6c406-GPTDataset-train-sample_index.npy
|
| 827 |
+
INFO:megatron.core.datasets.gpt_dataset: Load the shuffle index from 44ed1ee5bcb9e5b030387b8d27a6c406-GPTDataset-train-shuffle_index.npy
|
| 828 |
+
INFO:megatron.core.datasets.gpt_dataset:> total number of samples: 29073
|
| 829 |
+
> finished creating GPT datasets ...
|
| 830 |
+
[after dataloaders are built] datetime: 2026-08-30 06:44:53
|
| 831 |
+
done with setup ...
|
| 832 |
+
training ...
|
| 833 |
+
Overwriting rerun_state_machine.current_iteration from -1 to 0...
|
| 834 |
+
(min, max) time across ranks (ms):
|
| 835 |
+
model-and-optimizer-setup ......................: (207.42, 209.76)
|
| 836 |
+
train/valid/test-data-iterators-setup ..........: (63.21, 71.94)
|
| 837 |
+
[before the start of training step] datetime: 2026-08-30 06:44:53
|
| 838 |
+
/workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:334: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
|
| 839 |
+
warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
|
| 840 |
+
/workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:334: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
|
| 841 |
+
warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
|
| 842 |
+
/workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:334: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
|
| 843 |
+
warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
|
| 844 |
+
/workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:334: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
|
| 845 |
+
warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
|
| 846 |
+
/workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:334: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
|
| 847 |
+
warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
|
| 848 |
+
/workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:336: UserWarning: The use_host_meta is deprecated, it will be removed in the future.
|
| 849 |
+
warnings.warn("The use_host_meta is deprecated, it will be removed in the future.")
|
| 850 |
+
/workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:336: UserWarning: The use_host_meta is deprecated, it will be removed in the future.
|
| 851 |
+
warnings.warn("The use_host_meta is deprecated, it will be removed in the future.")
|
| 852 |
+
/workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:334: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
|
| 853 |
+
warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
|
| 854 |
+
/workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:336: UserWarning: The use_host_meta is deprecated, it will be removed in the future.
|
| 855 |
+
warnings.warn("The use_host_meta is deprecated, it will be removed in the future.")
|
| 856 |
+
/workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:336: UserWarning: The use_host_meta is deprecated, it will be removed in the future.
|
| 857 |
+
warnings.warn("The use_host_meta is deprecated, it will be removed in the future.")
|
| 858 |
+
/workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:336: UserWarning: The use_host_meta is deprecated, it will be removed in the future.
|
| 859 |
+
warnings.warn("The use_host_meta is deprecated, it will be removed in the future.")
|
| 860 |
+
/workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:334: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
|
| 861 |
+
warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
|
| 862 |
+
/workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:334: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
|
| 863 |
+
warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
|
| 864 |
+
/workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:336: UserWarning: The use_host_meta is deprecated, it will be removed in the future.
|
| 865 |
+
warnings.warn("The use_host_meta is deprecated, it will be removed in the future.")
|
| 866 |
+
/workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:336: UserWarning: The use_host_meta is deprecated, it will be removed in the future.
|
| 867 |
+
warnings.warn("The use_host_meta is deprecated, it will be removed in the future.")
|
| 868 |
+
/workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:336: UserWarning: The use_host_meta is deprecated, it will be removed in the future.
|
| 869 |
+
warnings.warn("The use_host_meta is deprecated, it will be removed in the future.")
|
| 870 |
+
/workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:482: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
|
| 871 |
+
warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
|
| 872 |
+
/workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:482: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
|
| 873 |
+
warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
|
| 874 |
+
/workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:482: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
|
| 875 |
+
warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
|
| 876 |
+
/workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:482: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
|
| 877 |
+
warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
|
| 878 |
+
/workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:482: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
|
| 879 |
+
warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
|
| 880 |
+
/workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:482: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
|
| 881 |
+
warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
|
| 882 |
+
/workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:482: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
|
| 883 |
+
warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
|
| 884 |
+
/workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:482: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
|
| 885 |
+
warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
|
| 886 |
+
/workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:334: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
|
| 887 |
+
warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
|
| 888 |
+
/workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:334: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
|
| 889 |
+
warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
|
| 890 |
+
/workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:336: UserWarning: The use_host_meta is deprecated, it will be removed in the future.
|
| 891 |
+
warnings.warn("The use_host_meta is deprecated, it will be removed in the future.")
|
| 892 |
+
/workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:336: UserWarning: The use_host_meta is deprecated, it will be removed in the future.
|
| 893 |
+
warnings.warn("The use_host_meta is deprecated, it will be removed in the future.")
|
| 894 |
+
/workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:334: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
|
| 895 |
+
warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
|
| 896 |
+
/workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:336: UserWarning: The use_host_meta is deprecated, it will be removed in the future.
|
| 897 |
+
warnings.warn("The use_host_meta is deprecated, it will be removed in the future.")
|
| 898 |
+
/workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:334: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
|
| 899 |
+
warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
|
| 900 |
+
/workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:336: UserWarning: The use_host_meta is deprecated, it will be removed in the future.
|
| 901 |
+
warnings.warn("The use_host_meta is deprecated, it will be removed in the future.")
|
| 902 |
+
/workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:334: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
|
| 903 |
+
warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
|
| 904 |
+
/workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:336: UserWarning: The use_host_meta is deprecated, it will be removed in the future.
|
| 905 |
+
warnings.warn("The use_host_meta is deprecated, it will be removed in the future.")
|
| 906 |
+
/workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:334: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
|
| 907 |
+
warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
|
| 908 |
+
/workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:336: UserWarning: The use_host_meta is deprecated, it will be removed in the future.
|
| 909 |
+
warnings.warn("The use_host_meta is deprecated, it will be removed in the future.")
|
| 910 |
+
/workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:334: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
|
| 911 |
+
warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
|
| 912 |
+
/workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:336: UserWarning: The use_host_meta is deprecated, it will be removed in the future.
|
| 913 |
+
warnings.warn("The use_host_meta is deprecated, it will be removed in the future.")
|
| 914 |
+
/workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:334: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
|
| 915 |
+
warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
|
| 916 |
+
/workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:336: UserWarning: The use_host_meta is deprecated, it will be removed in the future.
|
| 917 |
+
warnings.warn("The use_host_meta is deprecated, it will be removed in the future.")
|
| 918 |
+
/workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:482: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
|
| 919 |
+
warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
|
| 920 |
+
/workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:482: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
|
| 921 |
+
warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
|
| 922 |
+
/workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:482: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
|
| 923 |
+
warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
|
| 924 |
+
/workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:482: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
|
| 925 |
+
warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
|
| 926 |
+
/workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:482: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
|
| 927 |
+
warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
|
| 928 |
+
/workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:482: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
|
| 929 |
+
warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
|
| 930 |
+
/workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:482: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
|
| 931 |
+
warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
|
| 932 |
+
/workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:482: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
|
| 933 |
+
warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
|
| 934 |
+
/workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:334: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
|
| 935 |
+
warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
|
| 936 |
+
/workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:334: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
|
| 937 |
+
warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
|
| 938 |
+
/workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:334: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
|
| 939 |
+
warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
|
| 940 |
+
/workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:334: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
|
| 941 |
+
warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
|
| 942 |
+
/workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:334: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
|
| 943 |
+
warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
|
| 944 |
+
/workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:334: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
|
| 945 |
+
warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
|
| 946 |
+
/workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:334: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
|
| 947 |
+
warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
|
| 948 |
+
/workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:334: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
|
| 949 |
+
warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
|
| 950 |
+
/workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:336: UserWarning: The use_host_meta is deprecated, it will be removed in the future.
|
| 951 |
+
warnings.warn("The use_host_meta is deprecated, it will be removed in the future.")
|
| 952 |
+
/workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:336: UserWarning: The use_host_meta is deprecated, it will be removed in the future.
|
| 953 |
+
warnings.warn("The use_host_meta is deprecated, it will be removed in the future.")
|
| 954 |
+
/workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:336: UserWarning: The use_host_meta is deprecated, it will be removed in the future.
|
| 955 |
+
warnings.warn("The use_host_meta is deprecated, it will be removed in the future.")
|
| 956 |
+
/workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:336: UserWarning: The use_host_meta is deprecated, it will be removed in the future.
|
| 957 |
+
warnings.warn("The use_host_meta is deprecated, it will be removed in the future.")
|
| 958 |
+
/workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:336: UserWarning: The use_host_meta is deprecated, it will be removed in the future.
|
| 959 |
+
warnings.warn("The use_host_meta is deprecated, it will be removed in the future.")
|
| 960 |
+
/workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:336: UserWarning: The use_host_meta is deprecated, it will be removed in the future.
|
| 961 |
+
warnings.warn("The use_host_meta is deprecated, it will be removed in the future.")
|
| 962 |
+
/workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:336: UserWarning: The use_host_meta is deprecated, it will be removed in the future.
|
| 963 |
+
warnings.warn("The use_host_meta is deprecated, it will be removed in the future.")
|
| 964 |
+
/workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:336: UserWarning: The use_host_meta is deprecated, it will be removed in the future.
|
| 965 |
+
warnings.warn("The use_host_meta is deprecated, it will be removed in the future.")
|
| 966 |
+
[2026-08-30 06:45:31] iteration 1/ 12 | consumed samples: 16 | elapsed time per iteration (ms): 37891.4 | throughput per GPU (TFLOP/s/GPU): 8.8 | learning rate: 2.000000E-05 | global batch size: 16 | lm loss: 1.104376E+01 | loss scale: 1.0 | grad norm: 34.298 | number of skipped iterations: 0 | number of nan iterations: 0 |Number of parameters in transformer block in billions: 23.22
|
| 967 |
+
|
| 968 |
+
Number of parameters in embedding layers in billions: 0.41
|
| 969 |
+
Total number of parameters in billions: 23.63
|
| 970 |
+
Number of parameters in most loaded shard in billions: 23.6320
|
| 971 |
+
Theoretical memory footprints: weight and optimizer=169029.43 MB
|
| 972 |
+
[Rank 0] (after 1 iterations) memory (MB) | allocated: 63795.404296875 | max allocated: 96519.67578125 | reserved: 99122.0 | max reserved: 99122.0
|
| 973 |
+
[2026-08-30 06:45:32] iteration 2/ 12 | consumed samples: 32 | elapsed time per iteration (ms): 579.4 | throughput per GPU (TFLOP/s/GPU): 575.4 | learning rate: 4.000000E-05 | global batch size: 16 | lm loss: 1.104935E+01 | loss scale: 1.0 | grad norm: 35.989 | number of skipped iterations: 0 | number of nan iterations: 0 |
|
| 974 |
+
[2026-08-30 06:45:33] iteration 3/ 12 | consumed samples: 48 | elapsed time per iteration (ms): 559.7 | throughput per GPU (TFLOP/s/GPU): 595.7 | learning rate: 6.000000E-05 | global batch size: 16 | lm loss: 9.657161E+00 | loss scale: 1.0 | grad norm: 40.337 | number of skipped iterations: 0 | number of nan iterations: 0 |
|
| 975 |
+
Capture CUDA graph for training!!!
|
| 976 |
+
CUDA graph capture done!!!
|
| 977 |
+
[2026-08-30 06:45:34] iteration 4/ 12 | consumed samples: 64 | elapsed time per iteration (ms): 1645.4 | throughput per GPU (TFLOP/s/GPU): 202.6 | learning rate: 8.000000E-05 | global batch size: 16 | lm loss: 1.080619E+01 | loss scale: 1.0 | grad norm: 93.469 | number of skipped iterations: 0 | number of nan iterations: 0 |
|
| 978 |
+
[2026-08-30 06:45:35] iteration 5/ 12 | consumed samples: 80 | elapsed time per iteration (ms): 470.5 | throughput per GPU (TFLOP/s/GPU): 708.5 | learning rate: 1.000000E-04 | global batch size: 16 | lm loss: 1.068921E+01 | loss scale: 1.0 | grad norm: 17.369 | number of skipped iterations: 0 | number of nan iterations: 0 |
|
| 979 |
+
[2026-08-30 06:45:35] iteration 6/ 12 | consumed samples: 96 | elapsed time per iteration (ms): 467.0 | throughput per GPU (TFLOP/s/GPU): 713.9 | learning rate: 7.750000E-05 | global batch size: 16 | lm loss: 1.001069E+01 | loss scale: 1.0 | grad norm: 45.636 | number of skipped iterations: 0 | number of nan iterations: 0 |
|
| 980 |
+
[2026-08-30 06:45:36] iteration 7/ 12 | consumed samples: 112 | elapsed time per iteration (ms): 469.0 | throughput per GPU (TFLOP/s/GPU): 710.8 | learning rate: 3.250000E-05 | global batch size: 16 | lm loss: 9.327785E+00 | loss scale: 1.0 | grad norm: 7.423 | number of skipped iterations: 0 | number of nan iterations: 0 |
|
| 981 |
+
[2026-08-30 06:45:36] iteration 8/ 12 | consumed samples: 128 | elapsed time per iteration (ms): 465.5 | throughput per GPU (TFLOP/s/GPU): 716.2 | learning rate: 1.000000E-05 | global batch size: 16 | lm loss: 9.224079E+00 | loss scale: 1.0 | grad norm: 29.331 | number of skipped iterations: 0 | number of nan iterations: 0 |
|
| 982 |
+
[2026-08-30 06:45:36] iteration 9/ 12 | consumed samples: 144 | elapsed time per iteration (ms): 468.3 | throughput per GPU (TFLOP/s/GPU): 711.9 | learning rate: 1.000000E-05 | global batch size: 16 | lm loss: 8.703784E+00 | loss scale: 1.0 | grad norm: 5.921 | number of skipped iterations: 0 | number of nan iterations: 0 |
|
| 983 |
+
[2026-08-30 06:45:37] iteration 10/ 12 | consumed samples: 160 | elapsed time per iteration (ms): 462.1 | throughput per GPU (TFLOP/s/GPU): 721.5 | learning rate: 1.000000E-05 | global batch size: 16 | lm loss: 8.598076E+00 | loss scale: 1.0 | grad norm: 5.870 | number of skipped iterations: 0 | number of nan iterations: 0 |
|
| 984 |
+
[2026-08-30 06:45:37] iteration 11/ 12 | consumed samples: 176 | elapsed time per iteration (ms): 468.0 | throughput per GPU (TFLOP/s/GPU): 712.3 | learning rate: 1.000000E-05 | global batch size: 16 | lm loss: 8.479810E+00 | loss scale: 1.0 | grad norm: 4.778 | number of skipped iterations: 0 | number of nan iterations: 0 |
|
| 985 |
+
[2026-08-30 06:45:38] iteration 12/ 12 | consumed samples: 192 | elapsed time per iteration (ms): 468.1 | throughput per GPU (TFLOP/s/GPU): 712.3 | learning rate: 1.000000E-05 | global batch size: 16 | lm loss: 8.394531E+00 | loss scale: 1.0 | grad norm: 4.727 | number of skipped iterations: 0 | number of nan iterations: 0 |
|
| 986 |
+
[after training is done] datetime: 2026-08-30 06:45:38
|
| 987 |
+
W0830 06:45:38.726000 651947 torch/distributed/elastic/agent/server/api.py:719] Received 15 death signal, shutting down workers
|
| 988 |
+
W0830 06:45:38.727000 651947 torch/distributed/elastic/multiprocessing/api.py:900] Sending process 652036 closing signal SIGTERM
|
| 989 |
+
W0830 06:45:38.728000 651947 torch/distributed/elastic/multiprocessing/api.py:900] Sending process 652037 closing signal SIGTERM
|
| 990 |
+
W0830 06:45:38.730000 651947 torch/distributed/elastic/multiprocessing/api.py:900] Sending process 652038 closing signal SIGTERM
|
| 991 |
+
W0830 06:45:38.732000 651947 torch/distributed/elastic/multiprocessing/api.py:900] Sending process 652039 closing signal SIGTERM
|
| 992 |
+
W0830 06:45:38.732000 651947 torch/distributed/elastic/multiprocessing/api.py:900] Sending process 652040 closing signal SIGTERM
|
| 993 |
+
W0830 06:45:38.735000 651947 torch/distributed/elastic/multiprocessing/api.py:900] Sending process 652041 closing signal SIGTERM
|
| 994 |
+
W0830 06:45:38.736000 651947 torch/distributed/elastic/multiprocessing/api.py:900] Sending process 652042 closing signal SIGTERM
|
| 995 |
+
W0830 06:45:38.738000 651947 torch/distributed/elastic/multiprocessing/api.py:900] Sending process 652043 closing signal SIGTERM
|
| 996 |
+
Traceback (most recent call last):
|
| 997 |
+
File "/usr/local/bin/torchrun", line 8, in <module>
|
| 998 |
+
sys.exit(main())
|
| 999 |
+
^^^^^^
|
| 1000 |
+
File "/usr/local/lib/python3.12/dist-packages/torch/distributed/elastic/multiprocessing/errors/__init__.py", line 355, in wrapper
|
| 1001 |
+
return f(*args, **kwargs)
|
| 1002 |
+
^^^^^^^^^^^^^^^^^^
|
| 1003 |
+
File "/usr/local/lib/python3.12/dist-packages/torch/distributed/run.py", line 892, in main
|
| 1004 |
+
run(args)
|
| 1005 |
+
File "/usr/local/lib/python3.12/dist-packages/torch/distributed/run.py", line 883, in run
|
| 1006 |
+
elastic_launch(
|
| 1007 |
+
File "/usr/local/lib/python3.12/dist-packages/torch/distributed/launcher/api.py", line 139, in __call__
|
| 1008 |
+
return launch_agent(self._config, self._entrypoint, list(args))
|
| 1009 |
+
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
| 1010 |
+
File "/usr/local/lib/python3.12/dist-packages/torch/distributed/launcher/api.py", line 261, in launch_agent
|
| 1011 |
+
result = agent.run()
|
| 1012 |
+
^^^^^^^^^^^
|
| 1013 |
+
File "/usr/local/lib/python3.12/dist-packages/torch/distributed/elastic/metrics/api.py", line 138, in wrapper
|
| 1014 |
+
result = f(*args, **kwargs)
|
| 1015 |
+
^^^^^^^^^^^^^^^^^^
|
| 1016 |
+
File "/usr/local/lib/python3.12/dist-packages/torch/distributed/elastic/agent/server/api.py", line 711, in run
|
| 1017 |
+
result = self._invoke_run(role)
|
| 1018 |
+
^^^^^^^^^^^^^^^^^^^^^^
|
| 1019 |
+
File "/usr/local/lib/python3.12/dist-packages/torch/distributed/elastic/agent/server/api.py", line 870, in _invoke_run
|
| 1020 |
+
time.sleep(monitor_interval)
|
| 1021 |
+
File "/usr/local/lib/python3.12/dist-packages/torch/distributed/elastic/multiprocessing/api.py", line 84, in _terminate_process_handler
|
| 1022 |
+
raise SignalException(f"Process {os.getpid()} got signal: {sigval}", sigval=sigval)
|
| 1023 |
+
torch.distributed.elastic.multiprocessing.api.SignalException: Process 651947 got signal: 15
|
tiera-rerun-fixed-moonep/echo/a2/a2_matched_gbs16/G1__echo__matched__seq4096__gbs16__cap2p40625/rep2/mem_samples.csv
ADDED
|
@@ -0,0 +1,264 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
0, 0
|
| 2 |
+
1, 0
|
| 3 |
+
2, 0
|
| 4 |
+
3, 0
|
| 5 |
+
4, 0
|
| 6 |
+
5, 0
|
| 7 |
+
6, 0
|
| 8 |
+
7, 0
|
| 9 |
+
0, 0
|
| 10 |
+
1, 0
|
| 11 |
+
2, 0
|
| 12 |
+
3, 0
|
| 13 |
+
4, 0
|
| 14 |
+
5, 0
|
| 15 |
+
6, 0
|
| 16 |
+
7, 0
|
| 17 |
+
0, 4
|
| 18 |
+
1, 1
|
| 19 |
+
2, 1
|
| 20 |
+
3, 1
|
| 21 |
+
4, 1
|
| 22 |
+
5, 1
|
| 23 |
+
6, 1
|
| 24 |
+
7, 1
|
| 25 |
+
0, 4
|
| 26 |
+
1, 4
|
| 27 |
+
2, 4
|
| 28 |
+
3, 4
|
| 29 |
+
4, 4
|
| 30 |
+
5, 1
|
| 31 |
+
6, 1
|
| 32 |
+
7, 1
|
| 33 |
+
0, 4
|
| 34 |
+
1, 4
|
| 35 |
+
2, 4
|
| 36 |
+
3, 4
|
| 37 |
+
4, 4
|
| 38 |
+
5, 4
|
| 39 |
+
6, 4
|
| 40 |
+
7, 4
|
| 41 |
+
0, 4
|
| 42 |
+
1, 4
|
| 43 |
+
2, 4
|
| 44 |
+
3, 4
|
| 45 |
+
4, 4
|
| 46 |
+
5, 4
|
| 47 |
+
6, 4
|
| 48 |
+
7, 4
|
| 49 |
+
0, 821
|
| 50 |
+
1, 821
|
| 51 |
+
2, 821
|
| 52 |
+
3, 821
|
| 53 |
+
4, 821
|
| 54 |
+
5, 821
|
| 55 |
+
6, 821
|
| 56 |
+
7, 821
|
| 57 |
+
0, 1537
|
| 58 |
+
1, 1537
|
| 59 |
+
2, 1537
|
| 60 |
+
3, 1537
|
| 61 |
+
4, 1537
|
| 62 |
+
5, 1537
|
| 63 |
+
6, 1537
|
| 64 |
+
7, 1537
|
| 65 |
+
0, 2231
|
| 66 |
+
1, 2231
|
| 67 |
+
2, 2231
|
| 68 |
+
3, 2231
|
| 69 |
+
4, 2231
|
| 70 |
+
5, 2231
|
| 71 |
+
6, 2231
|
| 72 |
+
7, 2231
|
| 73 |
+
0, 39545
|
| 74 |
+
1, 39537
|
| 75 |
+
2, 39569
|
| 76 |
+
3, 39577
|
| 77 |
+
4, 39587
|
| 78 |
+
5, 39599
|
| 79 |
+
6, 39571
|
| 80 |
+
7, 39565
|
| 81 |
+
0, 39823
|
| 82 |
+
1, 39847
|
| 83 |
+
2, 39845
|
| 84 |
+
3, 39847
|
| 85 |
+
4, 39829
|
| 86 |
+
5, 39849
|
| 87 |
+
6, 39867
|
| 88 |
+
7, 39843
|
| 89 |
+
0, 42357
|
| 90 |
+
1, 42355
|
| 91 |
+
2, 42355
|
| 92 |
+
3, 42355
|
| 93 |
+
4, 42355
|
| 94 |
+
5, 42355
|
| 95 |
+
6, 42355
|
| 96 |
+
7, 42355
|
| 97 |
+
0, 42583
|
| 98 |
+
1, 42581
|
| 99 |
+
2, 42581
|
| 100 |
+
3, 42581
|
| 101 |
+
4, 42581
|
| 102 |
+
5, 42581
|
| 103 |
+
6, 42581
|
| 104 |
+
7, 42581
|
| 105 |
+
0, 42635
|
| 106 |
+
1, 42631
|
| 107 |
+
2, 42631
|
| 108 |
+
3, 42623
|
| 109 |
+
4, 42635
|
| 110 |
+
5, 42641
|
| 111 |
+
6, 42643
|
| 112 |
+
7, 42633
|
| 113 |
+
0, 43493
|
| 114 |
+
1, 43491
|
| 115 |
+
2, 43491
|
| 116 |
+
3, 43491
|
| 117 |
+
4, 43491
|
| 118 |
+
5, 43491
|
| 119 |
+
6, 43491
|
| 120 |
+
7, 43491
|
| 121 |
+
0, 43493
|
| 122 |
+
1, 43491
|
| 123 |
+
2, 43491
|
| 124 |
+
3, 43491
|
| 125 |
+
4, 43491
|
| 126 |
+
5, 43491
|
| 127 |
+
6, 43491
|
| 128 |
+
7, 43491
|
| 129 |
+
0, 44571
|
| 130 |
+
1, 44569
|
| 131 |
+
2, 44569
|
| 132 |
+
3, 44569
|
| 133 |
+
4, 44569
|
| 134 |
+
5, 44569
|
| 135 |
+
6, 44569
|
| 136 |
+
7, 44569
|
| 137 |
+
0, 45265
|
| 138 |
+
1, 45263
|
| 139 |
+
2, 45263
|
| 140 |
+
3, 45263
|
| 141 |
+
4, 45263
|
| 142 |
+
5, 45263
|
| 143 |
+
6, 45263
|
| 144 |
+
7, 45263
|
| 145 |
+
0, 85827
|
| 146 |
+
1, 85825
|
| 147 |
+
2, 85825
|
| 148 |
+
3, 85825
|
| 149 |
+
4, 85825
|
| 150 |
+
5, 85825
|
| 151 |
+
6, 85825
|
| 152 |
+
7, 85825
|
| 153 |
+
0, 86905
|
| 154 |
+
1, 86903
|
| 155 |
+
2, 86903
|
| 156 |
+
3, 86903
|
| 157 |
+
4, 86903
|
| 158 |
+
5, 86903
|
| 159 |
+
6, 86903
|
| 160 |
+
7, 86903
|
| 161 |
+
0, 86905
|
| 162 |
+
1, 86903
|
| 163 |
+
2, 86903
|
| 164 |
+
3, 86903
|
| 165 |
+
4, 86903
|
| 166 |
+
5, 86903
|
| 167 |
+
6, 86903
|
| 168 |
+
7, 86903
|
| 169 |
+
0, 86905
|
| 170 |
+
1, 86903
|
| 171 |
+
2, 86903
|
| 172 |
+
3, 86903
|
| 173 |
+
4, 86903
|
| 174 |
+
5, 86903
|
| 175 |
+
6, 86903
|
| 176 |
+
7, 86903
|
| 177 |
+
0, 103929
|
| 178 |
+
1, 103889
|
| 179 |
+
2, 103877
|
| 180 |
+
3, 103893
|
| 181 |
+
4, 103947
|
| 182 |
+
5, 103931
|
| 183 |
+
6, 103877
|
| 184 |
+
7, 103883
|
| 185 |
+
0, 105801
|
| 186 |
+
1, 105771
|
| 187 |
+
2, 105791
|
| 188 |
+
3, 105781
|
| 189 |
+
4, 105871
|
| 190 |
+
5, 105759
|
| 191 |
+
6, 105815
|
| 192 |
+
7, 105815
|
| 193 |
+
0, 106523
|
| 194 |
+
1, 106465
|
| 195 |
+
2, 106465
|
| 196 |
+
3, 106465
|
| 197 |
+
4, 106521
|
| 198 |
+
5, 106465
|
| 199 |
+
6, 106465
|
| 200 |
+
7, 106465
|
| 201 |
+
0, 107149
|
| 202 |
+
1, 107111
|
| 203 |
+
2, 107041
|
| 204 |
+
3, 107139
|
| 205 |
+
4, 107171
|
| 206 |
+
5, 107171
|
| 207 |
+
6, 107185
|
| 208 |
+
7, 107139
|
| 209 |
+
0, 147989
|
| 210 |
+
1, 135369
|
| 211 |
+
2, 141587
|
| 212 |
+
3, 146035
|
| 213 |
+
4, 140705
|
| 214 |
+
5, 137961
|
| 215 |
+
6, 139429
|
| 216 |
+
7, 141585
|
| 217 |
+
0, 161639
|
| 218 |
+
1, 161525
|
| 219 |
+
2, 161527
|
| 220 |
+
3, 161525
|
| 221 |
+
4, 161723
|
| 222 |
+
5, 161529
|
| 223 |
+
6, 161525
|
| 224 |
+
7, 161525
|
| 225 |
+
0, 161639
|
| 226 |
+
1, 161525
|
| 227 |
+
2, 161527
|
| 228 |
+
3, 161525
|
| 229 |
+
4, 161723
|
| 230 |
+
5, 161529
|
| 231 |
+
6, 161525
|
| 232 |
+
7, 161525
|
| 233 |
+
0, 160731
|
| 234 |
+
1, 160617
|
| 235 |
+
2, 160619
|
| 236 |
+
3, 160617
|
| 237 |
+
4, 160815
|
| 238 |
+
5, 160621
|
| 239 |
+
6, 160617
|
| 240 |
+
7, 160617
|
| 241 |
+
0, 2090
|
| 242 |
+
1, 160617
|
| 243 |
+
2, 3114
|
| 244 |
+
3, 1834
|
| 245 |
+
4, 1834
|
| 246 |
+
5, 1834
|
| 247 |
+
6, 1834
|
| 248 |
+
7, 1834
|
| 249 |
+
0, 0
|
| 250 |
+
1, 0
|
| 251 |
+
2, 0
|
| 252 |
+
3, 0
|
| 253 |
+
4, 0
|
| 254 |
+
5, 0
|
| 255 |
+
6, 0
|
| 256 |
+
7, 0
|
| 257 |
+
0, 0
|
| 258 |
+
1, 0
|
| 259 |
+
2, 0
|
| 260 |
+
3, 0
|
| 261 |
+
4, 0
|
| 262 |
+
5, 0
|
| 263 |
+
6, 0
|
| 264 |
+
7, 0
|
tiera-rerun-fixed-moonep/echo/a2/a2_matched_gbs16/G1__echo__matched__seq4096__gbs16__cap2p40625/rep2/run.env
ADDED
|
@@ -0,0 +1,70 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
backend=echo
|
| 2 |
+
config=G1
|
| 3 |
+
family=tierA
|
| 4 |
+
repo=/workspace/Megatron-MoE-Echo
|
| 5 |
+
branch=bracing-regions-through-kernels
|
| 6 |
+
commit=9a236793
|
| 7 |
+
num_layers=16
|
| 8 |
+
hidden=4096
|
| 9 |
+
ffn_hidden=14336
|
| 10 |
+
moe_ffn_hidden=14336
|
| 11 |
+
num_experts=8
|
| 12 |
+
topk=2
|
| 13 |
+
experts_per_rank=1
|
| 14 |
+
expert_params=176160768
|
| 15 |
+
heads=32
|
| 16 |
+
kv_groups=8
|
| 17 |
+
kv_channels=
|
| 18 |
+
qk_layernorm=0
|
| 19 |
+
shared_ffn=0
|
| 20 |
+
shared_expert_overlap=0
|
| 21 |
+
moe_token_dispatcher=flex
|
| 22 |
+
mla=0
|
| 23 |
+
dense_first_layers=0
|
| 24 |
+
moe_layers=16
|
| 25 |
+
rotary_base=1000000
|
| 26 |
+
rotary_scaling=
|
| 27 |
+
qkv_bias=0
|
| 28 |
+
router_balancing=aux_loss
|
| 29 |
+
aux_loss_coeff=1e-2
|
| 30 |
+
router_pre_softmax=0
|
| 31 |
+
router_fusion=0
|
| 32 |
+
router_dtype=fp32
|
| 33 |
+
device_initiated_grouped_gemm=1
|
| 34 |
+
norm_epsilon=1e-5
|
| 35 |
+
seq_length=4096
|
| 36 |
+
micro_batch_size=1
|
| 37 |
+
global_batch_size=16
|
| 38 |
+
num_microbatches=2
|
| 39 |
+
assignments_per_rank=8192
|
| 40 |
+
mean_tokens_per_expert=8192
|
| 41 |
+
slot_policy=matched
|
| 42 |
+
slots_per_rank=1
|
| 43 |
+
capacity_mode=static
|
| 44 |
+
capacity=2.40625
|
| 45 |
+
cuda_graph=1
|
| 46 |
+
probe_mode=0
|
| 47 |
+
async_grad_reduce=0
|
| 48 |
+
branch_role=timing
|
| 49 |
+
memory_dump_mode=none
|
| 50 |
+
memory_snapshot_path=
|
| 51 |
+
cuda_graph_warmup_steps=3
|
| 52 |
+
recompute=none
|
| 53 |
+
force_balance=0
|
| 54 |
+
train_iters=12
|
| 55 |
+
ep_size=8
|
| 56 |
+
echo_dispatch_overlap=1
|
| 57 |
+
echo_dispatch_recompute=0
|
| 58 |
+
ultraep_quota_min_tokens=
|
| 59 |
+
ultraep_weight_sync_join=
|
| 60 |
+
profile_enable=0
|
| 61 |
+
kernel_trace=0
|
| 62 |
+
fp8_mode=mxfp8
|
| 63 |
+
fp8_recipe=mxfp8
|
| 64 |
+
training_seed=1234
|
| 65 |
+
learning_rate=1e-4
|
| 66 |
+
min_learning_rate=1.0e-5
|
| 67 |
+
lr_warmup_iters=5
|
| 68 |
+
state_dump_enable=0
|
| 69 |
+
state_dump_dir=
|
| 70 |
+
moonep_assert_no_pending_reduce=0
|
tiera-rerun-fixed-moonep/echo/a2/a2_matched_gbs16/G1__echo__matched__seq4096__gbs16__cap2p40625/rep2/train.log
ADDED
|
@@ -0,0 +1,987 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
==============================================================
|
| 2 |
+
G1 / echo (slots/rank=1 policy=matched)
|
| 3 |
+
E=8 K=2 H'=14336 m=1
|
| 4 |
+
L=16 seq=4096 gbs=16 (2 microbatches)
|
| 5 |
+
cap_mode=static cap=2.40625 cg=1 recompute=none
|
| 6 |
+
memory_dump=none snapshots=/workspace/tiera-rerun-fixed-moonep/echo/a2/a2_matched_gbs16/G1__echo__matched__seq4096__gbs16__cap2p40625/rep2/memory_snapshots/snapshot.pickle
|
| 7 |
+
repo=/workspace/Megatron-MoE-Echo @ 9a236793
|
| 8 |
+
==============================================================
|
| 9 |
+
W0830 06:45:45.250000 655335 torch/distributed/run.py:766]
|
| 10 |
+
W0830 06:45:45.250000 655335 torch/distributed/run.py:766] *****************************************
|
| 11 |
+
W0830 06:45:45.250000 655335 torch/distributed/run.py:766] Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed.
|
| 12 |
+
W0830 06:45:45.250000 655335 torch/distributed/run.py:766] *****************************************
|
| 13 |
+
using world size: 8, data-parallel size: 8, context-parallel size: 1, hierarchical context-parallel sizes: None, tensor-model-parallel size: 1, pipeline-model-parallel size: 1
|
| 14 |
+
WARNING: overriding default arguments for tokenizer_type:GPT2BPETokenizer with tokenizer_type:GPT2BPETokenizer
|
| 15 |
+
Number of virtual stages per pipeline stage: None
|
| 16 |
+
accumulate and all-reduce gradients in fp32 for bfloat16 data type.
|
| 17 |
+
using torch.bfloat16 for parameters ...
|
| 18 |
+
/workspace/Megatron-MoE-Echo/megatron/training/arguments.py:831: UserWarning: Disabling sequence parallelism because tensor model parallelism is disabled
|
| 19 |
+
warnings.warn("Disabling sequence parallelism because tensor model parallelism is disabled")
|
| 20 |
+
------------------------ arguments ------------------------
|
| 21 |
+
account_for_embedding_in_pipeline_split ......... False
|
| 22 |
+
account_for_loss_in_pipeline_split .............. False
|
| 23 |
+
accumulate_allreduce_grads_in_fp32 .............. True
|
| 24 |
+
adam_beta1 ...................................... 0.9
|
| 25 |
+
adam_beta2 ...................................... 0.999
|
| 26 |
+
adam_eps ........................................ 1e-08
|
| 27 |
+
add_bias_linear ................................. False
|
| 28 |
+
add_position_embedding .......................... False
|
| 29 |
+
add_qkv_bias .................................... False
|
| 30 |
+
adlr_autoresume ................................. False
|
| 31 |
+
adlr_autoresume_interval ........................ 1000
|
| 32 |
+
align_grad_reduce ............................... True
|
| 33 |
+
align_param_gather .............................. False
|
| 34 |
+
app_tag_run_name ................................ None
|
| 35 |
+
app_tag_run_version ............................. 0.0.0
|
| 36 |
+
apply_layernorm_1p .............................. False
|
| 37 |
+
apply_query_key_layer_scaling ................... False
|
| 38 |
+
apply_residual_connection_post_layernorm ........ False
|
| 39 |
+
apply_rope_fusion ............................... True
|
| 40 |
+
async_save ...................................... None
|
| 41 |
+
async_tensor_model_parallel_allreduce ........... True
|
| 42 |
+
attention_backend ............................... AttnBackend.auto
|
| 43 |
+
attention_dropout ............................... 0.0
|
| 44 |
+
attention_softmax_in_fp32 ....................... False
|
| 45 |
+
auto_detect_ckpt_format ......................... False
|
| 46 |
+
barrier_with_L1_time ............................ True
|
| 47 |
+
bert_binary_head ................................ True
|
| 48 |
+
bert_embedder_type .............................. megatron
|
| 49 |
+
bert_load ....................................... None
|
| 50 |
+
bf16 ............................................ True
|
| 51 |
+
bias_dropout_fusion ............................. True
|
| 52 |
+
bias_gelu_fusion ................................ False
|
| 53 |
+
bias_swiglu_fusion .............................. True
|
| 54 |
+
biencoder_projection_dim ........................ 0
|
| 55 |
+
biencoder_shared_query_context_model ............ False
|
| 56 |
+
block_data_path ................................. None
|
| 57 |
+
calc_ft_timeouts ................................ False
|
| 58 |
+
calculate_per_token_loss ........................ False
|
| 59 |
+
check_for_large_grads ........................... False
|
| 60 |
+
check_for_nan_in_loss_and_grad .................. False
|
| 61 |
+
check_for_spiky_loss ............................ False
|
| 62 |
+
check_weight_hash_across_dp_replicas_interval ... None
|
| 63 |
+
ckpt_assume_constant_structure .................. False
|
| 64 |
+
ckpt_convert_format ............................. None
|
| 65 |
+
ckpt_convert_save ............................... None
|
| 66 |
+
ckpt_convert_update_legacy_dist_opt_format ...... False
|
| 67 |
+
ckpt_format ..................................... torch_dist
|
| 68 |
+
ckpt_fully_parallel_load ........................ False
|
| 69 |
+
ckpt_fully_parallel_save ........................ True
|
| 70 |
+
ckpt_fully_parallel_save_deprecated ............. False
|
| 71 |
+
ckpt_step ....................................... None
|
| 72 |
+
classes_fraction ................................ 1.0
|
| 73 |
+
clip_grad ....................................... 1.0
|
| 74 |
+
clone_scatter_output_in_embedding ............... True
|
| 75 |
+
config_logger_dir ...............................
|
| 76 |
+
consumed_train_samples .......................... 0
|
| 77 |
+
consumed_valid_samples .......................... 0
|
| 78 |
+
context_parallel_size ........................... 1
|
| 79 |
+
cp_comm_type .................................... ['p2p']
|
| 80 |
+
create_attention_mask_in_dataloader ............. True
|
| 81 |
+
cross_entropy_fusion_impl ....................... native
|
| 82 |
+
cross_entropy_loss_fusion ....................... False
|
| 83 |
+
cuda_graph_scope ................................ full_iteration
|
| 84 |
+
cuda_graph_warmup_steps ......................... 3
|
| 85 |
+
data_args_path .................................. None
|
| 86 |
+
data_cache_path ................................. None
|
| 87 |
+
data_parallel_random_init ....................... False
|
| 88 |
+
data_parallel_sharding_strategy ................. no_shard
|
| 89 |
+
data_parallel_size .............................. 8
|
| 90 |
+
data_path ....................................... ['/workspace/data/wikitext/wikitext103_gpt2_text_document']
|
| 91 |
+
data_per_class_fraction ......................... 1.0
|
| 92 |
+
data_sharding ................................... True
|
| 93 |
+
dataloader_type ................................. single
|
| 94 |
+
ddp_average_in_collective ....................... False
|
| 95 |
+
ddp_bucket_size ................................. None
|
| 96 |
+
ddp_num_buckets ................................. None
|
| 97 |
+
ddp_pad_buckets_for_high_nccl_busbw ............. False
|
| 98 |
+
decoder_first_pipeline_num_layers ............... None
|
| 99 |
+
decoder_last_pipeline_num_layers ................ None
|
| 100 |
+
decoder_num_layers .............................. None
|
| 101 |
+
decoder_seq_length .............................. None
|
| 102 |
+
decoupled_lr .................................... None
|
| 103 |
+
decoupled_min_lr ................................ None
|
| 104 |
+
decrease_batch_size_if_needed ................... False
|
| 105 |
+
defer_embedding_wgrad_compute ................... False
|
| 106 |
+
delay_wgrad_compute ............................. False
|
| 107 |
+
deprecated_use_mcore_models ..................... True
|
| 108 |
+
deterministic_mode .............................. False
|
| 109 |
+
dino_bottleneck_size ............................ 256
|
| 110 |
+
dino_freeze_last_layer .......................... 1
|
| 111 |
+
dino_head_hidden_size ........................... 2048
|
| 112 |
+
dino_local_crops_number ......................... 10
|
| 113 |
+
dino_local_img_size ............................. 96
|
| 114 |
+
dino_norm_last_layer ............................ False
|
| 115 |
+
dino_teacher_temp ............................... 0.07
|
| 116 |
+
dino_warmup_teacher_temp ........................ 0.04
|
| 117 |
+
dino_warmup_teacher_temp_epochs ................. 30
|
| 118 |
+
disable_bf16_reduced_precision_matmul ........... False
|
| 119 |
+
disable_mamba_mem_eff_path ...................... False
|
| 120 |
+
disable_straggler_on_startup .................... False
|
| 121 |
+
dist_ckpt_format_deprecated ..................... None
|
| 122 |
+
dist_ckpt_strictness ............................ assume_ok_unexpected
|
| 123 |
+
distribute_saved_activations .................... False
|
| 124 |
+
distributed_backend ............................. nccl
|
| 125 |
+
distributed_timeout_minutes ..................... 10
|
| 126 |
+
embedding_init_method_std ....................... None
|
| 127 |
+
embedding_path .................................. None
|
| 128 |
+
empty_unused_memory_level ....................... 0
|
| 129 |
+
enable_cuda_graph ............................... True
|
| 130 |
+
enable_experimental ............................. False
|
| 131 |
+
enable_ft_package ............................... False
|
| 132 |
+
enable_gloo_process_groups ...................... True
|
| 133 |
+
enable_msc ...................................... True
|
| 134 |
+
enable_one_logger ............................... True
|
| 135 |
+
encoder_num_layers .............................. 16
|
| 136 |
+
encoder_seq_length .............................. 4096
|
| 137 |
+
end_weight_decay ................................ 0.1
|
| 138 |
+
eod_mask_loss ................................... False
|
| 139 |
+
error_injection_rate ............................ 0
|
| 140 |
+
error_injection_type ............................ transient_error
|
| 141 |
+
eval_interval ................................... 1000000
|
| 142 |
+
eval_iters ...................................... 0
|
| 143 |
+
evidence_data_path .............................. None
|
| 144 |
+
exit_duration_in_mins ........................... None
|
| 145 |
+
exit_interval ................................... None
|
| 146 |
+
exit_on_missing_checkpoint ...................... False
|
| 147 |
+
exit_signal_handler ............................. False
|
| 148 |
+
exp_avg_dtype ................................... torch.float32
|
| 149 |
+
exp_avg_sq_dtype ................................ torch.float32
|
| 150 |
+
expert_model_parallel_size ...................... 8
|
| 151 |
+
expert_tensor_parallel_size ..................... 1
|
| 152 |
+
export_force_local_attention .................... False
|
| 153 |
+
export_kd_cfg ................................... None
|
| 154 |
+
export_kd_teacher_ckpt_format ................... None
|
| 155 |
+
export_kd_teacher_load .......................... None
|
| 156 |
+
export_kv_cache_quant ........................... False
|
| 157 |
+
export_legacy_megatron .......................... False
|
| 158 |
+
export_model_type ............................... GPTModel
|
| 159 |
+
export_moe_apply_probs_on_input ................. False
|
| 160 |
+
export_qk_l2_norm ............................... False
|
| 161 |
+
export_quant_cfg ................................ None
|
| 162 |
+
export_real_quant_cfg ........................... None
|
| 163 |
+
export_te_mcore_model ........................... False
|
| 164 |
+
external_cuda_graph ............................. False
|
| 165 |
+
ffn_hidden_size ................................. 14336
|
| 166 |
+
finetune ........................................ False
|
| 167 |
+
finetune_data_split ............................. train
|
| 168 |
+
finetune_hf_dataset ............................. None
|
| 169 |
+
first_last_layers_bf16 .......................... False
|
| 170 |
+
flash_decode .................................... False
|
| 171 |
+
fp16 ............................................ False
|
| 172 |
+
fp16_lm_cross_entropy ........................... False
|
| 173 |
+
fp32_residual_connection ........................ False
|
| 174 |
+
fp8 ............................................. e4m3
|
| 175 |
+
fp8_amax_compute_algo ........................... most_recent
|
| 176 |
+
fp8_amax_history_len ............................ 1
|
| 177 |
+
fp8_interval .................................... 1
|
| 178 |
+
fp8_margin ...................................... 0
|
| 179 |
+
fp8_param_gather ................................ False
|
| 180 |
+
fp8_recipe ...................................... mxfp8
|
| 181 |
+
fp8_wgrad ....................................... True
|
| 182 |
+
fsdp_double_buffer .............................. False
|
| 183 |
+
global_batch_size ............................... 16
|
| 184 |
+
grad_reduce_in_bf16 ............................. False
|
| 185 |
+
gradient_accumulation_fusion .................... True
|
| 186 |
+
gradient_reduce_div_fusion ...................... True
|
| 187 |
+
group_query_attention ........................... True
|
| 188 |
+
head_lr_mult .................................... 1.0
|
| 189 |
+
heterogeneous_layers_config_encoded_json ........ None
|
| 190 |
+
heterogeneous_layers_config_path ................ None
|
| 191 |
+
hidden_dropout .................................. 0.0
|
| 192 |
+
hidden_size ..................................... 4096
|
| 193 |
+
hierarchical_context_parallel_sizes ............. None
|
| 194 |
+
high_priority_stream_groups ..................... []
|
| 195 |
+
hybrid_attention_ratio .......................... 0.0
|
| 196 |
+
hybrid_mlp_ratio ................................ 0.0
|
| 197 |
+
hybrid_override_pattern ......................... None
|
| 198 |
+
hysteresis ...................................... 2
|
| 199 |
+
ict_head_size ................................... None
|
| 200 |
+
ict_load ........................................ None
|
| 201 |
+
img_h ........................................... 224
|
| 202 |
+
img_w ........................................... 224
|
| 203 |
+
indexer_batch_size .............................. 128
|
| 204 |
+
indexer_log_interval ............................ 1000
|
| 205 |
+
inference_batch_times_seqlen_threshold .......... -1
|
| 206 |
+
inference_dynamic_batching ...................... False
|
| 207 |
+
inference_dynamic_batching_buffer_guaranteed_fraction 0.2
|
| 208 |
+
inference_dynamic_batching_buffer_overflow_factor None
|
| 209 |
+
inference_dynamic_batching_buffer_size_gb ....... 40.0
|
| 210 |
+
inference_dynamic_batching_chunk_size ........... 256
|
| 211 |
+
inference_dynamic_batching_max_requests_override None
|
| 212 |
+
inference_dynamic_batching_max_tokens_override .. None
|
| 213 |
+
inference_dynamic_batching_num_cuda_graphs ...... 16
|
| 214 |
+
inference_max_batch_size ........................ 8
|
| 215 |
+
inference_max_seq_length ........................ 2560
|
| 216 |
+
inference_rng_tracker ........................... False
|
| 217 |
+
init_method_std ................................. 0.01
|
| 218 |
+
init_method_xavier_uniform ...................... False
|
| 219 |
+
init_model_with_meta_device ..................... False
|
| 220 |
+
initial_loss_scale .............................. 4294967296
|
| 221 |
+
inprocess_active_world_size ..................... 8
|
| 222 |
+
inprocess_barrier_timeout ....................... 120
|
| 223 |
+
inprocess_completion_timeout .................... 120
|
| 224 |
+
inprocess_empty_cuda_cache ...................... False
|
| 225 |
+
inprocess_granularity ........................... node
|
| 226 |
+
inprocess_hard_timeout .......................... 90
|
| 227 |
+
inprocess_heartbeat_interval .................... 30
|
| 228 |
+
inprocess_heartbeat_timeout ..................... 60
|
| 229 |
+
inprocess_last_call_wait ........................ 1
|
| 230 |
+
inprocess_max_iterations ........................ None
|
| 231 |
+
inprocess_monitor_process_interval .............. 1.0
|
| 232 |
+
inprocess_monitor_thread_interval ............... 1.0
|
| 233 |
+
inprocess_progress_watchdog_interval ............ 1.0
|
| 234 |
+
inprocess_restart ............................... False
|
| 235 |
+
inprocess_soft_timeout .......................... 60
|
| 236 |
+
inprocess_termination_grace_time ................ 1
|
| 237 |
+
is_hybrid_model ................................. False
|
| 238 |
+
iter_per_epoch .................................. 1250
|
| 239 |
+
iterations_to_skip .............................. []
|
| 240 |
+
keep_fp8_transpose_cache_when_using_custom_fsdp . False
|
| 241 |
+
kitchen_config_file ............................. None
|
| 242 |
+
kitchen_recipe_number ........................... None
|
| 243 |
+
kv_channels ..................................... 128
|
| 244 |
+
kv_lora_rank .................................... 32
|
| 245 |
+
lazy_mpu_init ................................... None
|
| 246 |
+
load ............................................ None
|
| 247 |
+
load_main_params_from_ckpt ...................... None
|
| 248 |
+
load_model_opt_format ........................... False
|
| 249 |
+
local_rank ...................................... 0
|
| 250 |
+
log_energy ...................................... False
|
| 251 |
+
log_interval .................................... 1
|
| 252 |
+
log_loss_scale_to_tensorboard ................... True
|
| 253 |
+
log_memory_to_tensorboard ....................... False
|
| 254 |
+
log_num_zeros_in_grad ........................... False
|
| 255 |
+
log_params_norm ................................. False
|
| 256 |
+
log_progress .................................... False
|
| 257 |
+
log_straggler ................................... False
|
| 258 |
+
log_throughput .................................. True
|
| 259 |
+
log_timers_to_tensorboard ....................... False
|
| 260 |
+
log_validation_ppl_to_tensorboard ............... False
|
| 261 |
+
log_world_size_to_tensorboard ................... False
|
| 262 |
+
logging_level ................................... None
|
| 263 |
+
loss_scale ...................................... None
|
| 264 |
+
loss_scale_window ............................... 1000
|
| 265 |
+
lr .............................................. 0.0001
|
| 266 |
+
lr_decay_iters .................................. 8
|
| 267 |
+
lr_decay_samples ................................ None
|
| 268 |
+
lr_decay_style .................................. cosine
|
| 269 |
+
lr_warmup_fraction .............................. None
|
| 270 |
+
lr_warmup_init .................................. 0.0
|
| 271 |
+
lr_warmup_iters ................................. 5
|
| 272 |
+
lr_warmup_samples ............................... 0
|
| 273 |
+
lr_wsd_decay_iters .............................. None
|
| 274 |
+
lr_wsd_decay_samples ............................ None
|
| 275 |
+
lr_wsd_decay_style .............................. exponential
|
| 276 |
+
main_grads_dtype ................................ torch.float32
|
| 277 |
+
main_params_dtype ............................... torch.float32
|
| 278 |
+
make_vocab_size_divisible_by .................... 128
|
| 279 |
+
mamba_head_dim .................................. 64
|
| 280 |
+
mamba_num_groups ................................ 8
|
| 281 |
+
mamba_num_heads ................................. None
|
| 282 |
+
mamba_state_dim ................................. 128
|
| 283 |
+
manual_gc ....................................... False
|
| 284 |
+
manual_gc_eval .................................. True
|
| 285 |
+
manual_gc_interval .............................. 0
|
| 286 |
+
mask_factor ..................................... 1.0
|
| 287 |
+
mask_prob ....................................... 0.15
|
| 288 |
+
mask_type ....................................... random
|
| 289 |
+
masked_softmax_fusion ........................... False
|
| 290 |
+
max_position_embeddings ......................... 32768
|
| 291 |
+
max_tokens_to_oom ............................... 12000
|
| 292 |
+
memory_snapshot_path ............................ snapshot.pickle
|
| 293 |
+
merge_file ...................................... /workspace/data/wikitext/gpt2-merges.txt
|
| 294 |
+
micro_batch_size ................................ 1
|
| 295 |
+
microbatch_group_size_per_vp_stage .............. None
|
| 296 |
+
mid_level_dataset_surplus ....................... 0.005
|
| 297 |
+
min_loss_scale .................................. 1.0
|
| 298 |
+
min_lr .......................................... 1e-05
|
| 299 |
+
mlp_chunks_for_prefill .......................... 1
|
| 300 |
+
mmap_bin_files .................................. True
|
| 301 |
+
mock_data ....................................... False
|
| 302 |
+
moe_apply_probs_on_input ........................ False
|
| 303 |
+
moe_aux_loss_coeff .............................. 0.01
|
| 304 |
+
moe_deepep_num_sms .............................. 20
|
| 305 |
+
moe_echo_enable_random_offloading ............... False
|
| 306 |
+
moe_echo_expert_dispatch_overlap ................ True
|
| 307 |
+
moe_echo_expert_dispatcher_type ................. hybridep
|
| 308 |
+
moe_echo_recompute_expert_dispatch .............. False
|
| 309 |
+
moe_enable_deepep ............................... False
|
| 310 |
+
moe_enable_echo ................................. True
|
| 311 |
+
moe_enable_hybridep ............................. True
|
| 312 |
+
moe_expert_capacity_factor ...................... None
|
| 313 |
+
moe_extended_tp ................................. False
|
| 314 |
+
moe_ffn_hidden_size ............................. 14336
|
| 315 |
+
moe_grouped_gemm ................................ True
|
| 316 |
+
moe_hybridep_num_sms ............................ 16
|
| 317 |
+
moe_input_jitter_eps ............................ None
|
| 318 |
+
moe_layer_freq .................................. 1
|
| 319 |
+
moe_layer_recompute ............................. False
|
| 320 |
+
moe_num_echo_experts ............................ 8
|
| 321 |
+
moe_pad_expert_input_to_capacity ................ False
|
| 322 |
+
moe_per_layer_logging ........................... False
|
| 323 |
+
moe_permute_fusion .............................. False
|
| 324 |
+
moe_received_token_capacity ..................... 2.40625
|
| 325 |
+
moe_router_bias_update_rate ..................... 0.001
|
| 326 |
+
moe_router_dtype ................................ fp32
|
| 327 |
+
moe_router_enable_expert_bias ................... False
|
| 328 |
+
moe_router_force_load_balancing ................. False
|
| 329 |
+
moe_router_group_topk ........................... None
|
| 330 |
+
moe_router_load_balancing_type .................. aux_loss
|
| 331 |
+
moe_router_num_groups ........................... None
|
| 332 |
+
moe_router_padding_for_fp8 ...................... False
|
| 333 |
+
moe_router_pre_softmax .......................... False
|
| 334 |
+
moe_router_score_function ....................... softmax
|
| 335 |
+
moe_router_topk ................................. 2
|
| 336 |
+
moe_router_topk_scaling_factor .................. None
|
| 337 |
+
moe_shared_expert_intermediate_size ............. None
|
| 338 |
+
moe_shared_expert_overlap ....................... False
|
| 339 |
+
moe_token_dispatcher_type ....................... flex
|
| 340 |
+
moe_token_drop_policy ........................... probs
|
| 341 |
+
moe_upcycling_granularity ....................... 1
|
| 342 |
+
moe_use_device_initiated_grouped_gemm ........... True
|
| 343 |
+
moe_use_legacy_grouped_gemm ..................... False
|
| 344 |
+
moe_use_upcycling ............................... False
|
| 345 |
+
moe_z_loss_coeff ................................ None
|
| 346 |
+
mrope_section ................................... None
|
| 347 |
+
mscale .......................................... 1.0
|
| 348 |
+
mscale_all_dim .................................. 0.0
|
| 349 |
+
mtp_loss_scaling_factor ......................... 0.1
|
| 350 |
+
mtp_num_layers .................................. None
|
| 351 |
+
multi_latent_attention .......................... False
|
| 352 |
+
nccl_all_reduce_for_prefill ..................... False
|
| 353 |
+
nccl_communicator_config_path ................... None
|
| 354 |
+
nccl_ub ......................................... False
|
| 355 |
+
no_load_optim ................................... None
|
| 356 |
+
no_load_rng ..................................... None
|
| 357 |
+
no_persist_layer_norm ........................... False
|
| 358 |
+
no_rope_freq .................................... None
|
| 359 |
+
no_save_optim ................................... None
|
| 360 |
+
no_save_rng ..................................... None
|
| 361 |
+
non_persistent_ckpt_type ........................ None
|
| 362 |
+
non_persistent_global_ckpt_dir .................. None
|
| 363 |
+
non_persistent_local_ckpt_algo .................. fully_parallel
|
| 364 |
+
non_persistent_local_ckpt_dir ................... None
|
| 365 |
+
non_persistent_save_interval .................... None
|
| 366 |
+
norm_epsilon .................................... 1e-05
|
| 367 |
+
normalization ................................... RMSNorm
|
| 368 |
+
num_attention_heads ............................. 32
|
| 369 |
+
num_channels .................................... 3
|
| 370 |
+
num_classes ..................................... 1000
|
| 371 |
+
num_dataset_builder_threads ..................... 1
|
| 372 |
+
num_distributed_optimizer_instances ............. 1
|
| 373 |
+
num_experts ..................................... 8
|
| 374 |
+
num_layers ...................................... 16
|
| 375 |
+
num_layers_at_end_in_bf16 ....................... 1
|
| 376 |
+
num_layers_at_start_in_bf16 ..................... 1
|
| 377 |
+
num_layers_per_virtual_pipeline_stage ........... None
|
| 378 |
+
num_query_groups ................................ 8
|
| 379 |
+
num_virtual_stages_per_pipeline_rank ............ None
|
| 380 |
+
num_workers ..................................... 2
|
| 381 |
+
object_storage_cache_path ....................... None
|
| 382 |
+
one_logger_async ................................ False
|
| 383 |
+
one_logger_project .............................. megatron-lm
|
| 384 |
+
one_logger_run_name ............................. None
|
| 385 |
+
onnx_safe ....................................... None
|
| 386 |
+
openai_gelu ..................................... False
|
| 387 |
+
optimizer ....................................... adam
|
| 388 |
+
optimizer_cpu_offload ........................... False
|
| 389 |
+
optimizer_offload_fraction ...................... 1.0
|
| 390 |
+
output_bert_embeddings .......................... False
|
| 391 |
+
overlap_cpu_optimizer_d2h_h2d ................... False
|
| 392 |
+
overlap_grad_reduce ............................. True
|
| 393 |
+
overlap_moe_expert_parallel_comm ................ False
|
| 394 |
+
overlap_p2p_comm ................................ False
|
| 395 |
+
overlap_p2p_comm_warmup_flush ................... False
|
| 396 |
+
overlap_param_gather ............................ True
|
| 397 |
+
overlap_param_gather_with_optimizer_step ........ False
|
| 398 |
+
override_opt_param_scheduler .................... False
|
| 399 |
+
params_dtype .................................... torch.bfloat16
|
| 400 |
+
patch_dim ....................................... 16
|
| 401 |
+
per_split_data_args_path ........................ None
|
| 402 |
+
perform_initialization .......................... True
|
| 403 |
+
pin_cpu_grads ................................... True
|
| 404 |
+
pin_cpu_params .................................. True
|
| 405 |
+
pipeline_model_parallel_comm_backend ............ None
|
| 406 |
+
pipeline_model_parallel_layout .................. None
|
| 407 |
+
pipeline_model_parallel_size .................... 1
|
| 408 |
+
position_embedding_type ......................... rope
|
| 409 |
+
pretrained_checkpoint ........................... None
|
| 410 |
+
profile ......................................... False
|
| 411 |
+
profile_ranks ................................... [0]
|
| 412 |
+
profile_step_end ................................ 12
|
| 413 |
+
profile_step_start .............................. 10
|
| 414 |
+
q_lora_rank ..................................... None
|
| 415 |
+
qk_head_dim ..................................... 128
|
| 416 |
+
qk_l2_norm ...................................... False
|
| 417 |
+
qk_layernorm .................................... False
|
| 418 |
+
qk_pos_emb_head_dim ............................. 64
|
| 419 |
+
query_in_block_prob ............................. 0.1
|
| 420 |
+
rampup_batch_size ............................... None
|
| 421 |
+
rank ............................................ 0
|
| 422 |
+
recompute_granularity ........................... None
|
| 423 |
+
recompute_method ................................ None
|
| 424 |
+
recompute_modules ............................... None
|
| 425 |
+
recompute_num_layers ............................ None
|
| 426 |
+
record_memory_history ........................... False
|
| 427 |
+
relative_attention_max_distance ................. 128
|
| 428 |
+
relative_attention_num_buckets .................. 32
|
| 429 |
+
replication ..................................... False
|
| 430 |
+
replication_factor .............................. 2
|
| 431 |
+
replication_jump ................................ None
|
| 432 |
+
rerun_mode ...................................... validate_results
|
| 433 |
+
reset_attention_mask ............................ False
|
| 434 |
+
reset_position_ids .............................. False
|
| 435 |
+
result_rejected_tracker_filename ................ None
|
| 436 |
+
retriever_report_topk_accuracies ................ []
|
| 437 |
+
retriever_score_scaling ......................... False
|
| 438 |
+
retriever_seq_length ............................ 256
|
| 439 |
+
retro_add_retriever ............................. False
|
| 440 |
+
retro_attention_gate ............................ 1
|
| 441 |
+
retro_cyclic_train_iters ........................ None
|
| 442 |
+
retro_encoder_attention_dropout ................. 0.1
|
| 443 |
+
retro_encoder_hidden_dropout .................... 0.1
|
| 444 |
+
retro_encoder_layers ............................ 2
|
| 445 |
+
retro_num_neighbors ............................. 2
|
| 446 |
+
retro_num_retrieved_chunks ...................... 2
|
| 447 |
+
retro_project_dir ............................... None
|
| 448 |
+
retro_verify_neighbor_count ..................... True
|
| 449 |
+
reuse_grad_buf_for_mxfp8_param_ag ............... False
|
| 450 |
+
rope_scaling_factor ............................. 8.0
|
| 451 |
+
rope_type ....................................... None
|
| 452 |
+
rotary_base ..................................... 1000000
|
| 453 |
+
rotary_interleaved .............................. False
|
| 454 |
+
rotary_percent .................................. 1.0
|
| 455 |
+
rotary_scaling_factor ........................... 1.0
|
| 456 |
+
rotary_seq_len_interpolation_factor ............. None
|
| 457 |
+
run_workload_inspector_server ................... False
|
| 458 |
+
sample_rate ..................................... 1.0
|
| 459 |
+
save ............................................ None
|
| 460 |
+
save_interval ................................... None
|
| 461 |
+
save_retain_interval ............................ None
|
| 462 |
+
scatter_gather_tensors_in_pipeline .............. True
|
| 463 |
+
seed ............................................ 1234
|
| 464 |
+
seq_length ...................................... 4096
|
| 465 |
+
sequence_parallel ............................... False
|
| 466 |
+
sft ............................................. False
|
| 467 |
+
sft_tokenizer_prompt_format ..................... nemotron-h-aligned
|
| 468 |
+
sgd_momentum .................................... 0.9
|
| 469 |
+
short_seq_prob .................................. 0.1
|
| 470 |
+
skip_train ...................................... False
|
| 471 |
+
skipped_train_samples ........................... 0
|
| 472 |
+
spec ............................................ None
|
| 473 |
+
split ........................................... 100,0,0
|
| 474 |
+
squared_relu .................................... False
|
| 475 |
+
start_weight_decay .............................. 0.1
|
| 476 |
+
straggler_ctrlr_port ............................ 65535
|
| 477 |
+
straggler_minmax_count .......................... 1
|
| 478 |
+
suggested_communication_unit_size ............... None
|
| 479 |
+
swiglu .......................................... True
|
| 480 |
+
swin_backbone_type .............................. tiny
|
| 481 |
+
symmetric_ar_type ............................... None
|
| 482 |
+
te_rng_tracker .................................. True
|
| 483 |
+
tensor_model_parallel_size ...................... 1
|
| 484 |
+
tensorboard_dir ................................. None
|
| 485 |
+
tensorboard_log_interval ........................ 1
|
| 486 |
+
tensorboard_queue_size .......................... 1000
|
| 487 |
+
test_data_path .................................. None
|
| 488 |
+
test_mode ....................................... False
|
| 489 |
+
tiktoken_num_special_tokens ..................... 1000
|
| 490 |
+
tiktoken_pattern ................................ None
|
| 491 |
+
tiktoken_special_tokens ......................... None
|
| 492 |
+
timing_log_level ................................ 0
|
| 493 |
+
timing_log_option ............................... minmax
|
| 494 |
+
titles_data_path ................................ None
|
| 495 |
+
tokenizer_model ................................. None
|
| 496 |
+
tokenizer_type .................................. GPT2BPETokenizer
|
| 497 |
+
torch_fsdp2_reshard_after_forward ............... True
|
| 498 |
+
tp_comm_bootstrap_backend ....................... nccl
|
| 499 |
+
tp_comm_bulk_dgrad .............................. True
|
| 500 |
+
tp_comm_bulk_wgrad .............................. True
|
| 501 |
+
tp_comm_overlap ................................. False
|
| 502 |
+
tp_comm_overlap_ag .............................. True
|
| 503 |
+
tp_comm_overlap_cfg ............................. None
|
| 504 |
+
tp_comm_overlap_rs .............................. True
|
| 505 |
+
tp_comm_overlap_rs_dgrad ........................ False
|
| 506 |
+
tp_comm_split_ag ................................ True
|
| 507 |
+
tp_comm_split_rs ................................ True
|
| 508 |
+
train_data_path ................................. None
|
| 509 |
+
train_iters ..................................... 12
|
| 510 |
+
train_samples ................................... None
|
| 511 |
+
train_sync_interval ............................. None
|
| 512 |
+
transformer_impl ................................ transformer_engine
|
| 513 |
+
transformer_pipeline_model_parallel_size ........ 1
|
| 514 |
+
untie_embeddings_and_output_weights ............. True
|
| 515 |
+
use_checkpoint_args ............................. False
|
| 516 |
+
use_checkpoint_opt_param_scheduler .............. False
|
| 517 |
+
use_cpu_initialization .......................... None
|
| 518 |
+
use_custom_fsdp ................................. False
|
| 519 |
+
use_dist_ckpt ................................... True
|
| 520 |
+
use_dist_ckpt_deprecated ........................ False
|
| 521 |
+
use_distributed_optimizer ....................... True
|
| 522 |
+
use_flash_attn .................................. False
|
| 523 |
+
use_legacy_models ............................... False
|
| 524 |
+
use_mp_args_from_checkpoint_args ................ False
|
| 525 |
+
use_one_sent_docs ............................... False
|
| 526 |
+
use_persistent_ckpt_worker ...................... False
|
| 527 |
+
use_precision_aware_optimizer ................... False
|
| 528 |
+
use_pytorch_profiler ............................ False
|
| 529 |
+
use_ring_exchange_p2p ........................... False
|
| 530 |
+
use_rope_scaling ................................ False
|
| 531 |
+
use_rotary_position_embeddings .................. False
|
| 532 |
+
use_sharp ....................................... False
|
| 533 |
+
use_tokenizer_model_from_checkpoint_args ........ True
|
| 534 |
+
use_torch_fsdp2 ................................. False
|
| 535 |
+
use_torch_optimizer_for_cpu_offload ............. False
|
| 536 |
+
use_tp_pp_dp_mapping ............................ False
|
| 537 |
+
v_head_dim ...................................... 128
|
| 538 |
+
valid_data_path ................................. None
|
| 539 |
+
variable_seq_lengths ............................ False
|
| 540 |
+
virtual_pipeline_model_parallel_size ............ None
|
| 541 |
+
vision_backbone_type ............................ vit
|
| 542 |
+
vision_pretraining .............................. False
|
| 543 |
+
vision_pretraining_type ......................... classify
|
| 544 |
+
vocab_extra_ids ................................. 0
|
| 545 |
+
vocab_file ...................................... /workspace/data/wikitext/gpt2-vocab.json
|
| 546 |
+
vocab_size ...................................... None
|
| 547 |
+
wandb_exp_name ..................................
|
| 548 |
+
wandb_project ...................................
|
| 549 |
+
wandb_save_dir ..................................
|
| 550 |
+
weight_decay .................................... 0.1
|
| 551 |
+
weight_decay_incr_style ......................... constant
|
| 552 |
+
wgrad_deferral_limit ............................ 0
|
| 553 |
+
world_size ...................................... 8
|
| 554 |
+
yaml_cfg ........................................ None
|
| 555 |
+
-------------------- end of arguments ---------------------
|
| 556 |
+
INFO:megatron.core.num_microbatches_calculator:setting number of microbatches to constant 2
|
| 557 |
+
> building GPT2BPETokenizer tokenizer ...
|
| 558 |
+
> padded vocab (size: 50257) with 47 dummy tokens (new size: 50304)
|
| 559 |
+
WARNING:megatron.core.rerun_state_machine:RerunStateMachine initialized in mode RerunMode.VALIDATE_RESULTS
|
| 560 |
+
> initializing torch distributed ...
|
| 561 |
+
/workspace/Megatron-MoE-Echo/megatron/training/arguments.py:831: UserWarning: Disabling sequence parallelism because tensor model parallelism is disabled
|
| 562 |
+
warnings.warn("Disabling sequence parallelism because tensor model parallelism is disabled")
|
| 563 |
+
/workspace/Megatron-MoE-Echo/megatron/training/arguments.py:831: UserWarning: Disabling sequence parallelism because tensor model parallelism is disabled
|
| 564 |
+
warnings.warn("Disabling sequence parallelism because tensor model parallelism is disabled")
|
| 565 |
+
/workspace/Megatron-MoE-Echo/megatron/training/arguments.py:831: UserWarning: Disabling sequence parallelism because tensor model parallelism is disabled
|
| 566 |
+
warnings.warn("Disabling sequence parallelism because tensor model parallelism is disabled")
|
| 567 |
+
/workspace/Megatron-MoE-Echo/megatron/training/arguments.py:831: UserWarning: Disabling sequence parallelism because tensor model parallelism is disabled
|
| 568 |
+
warnings.warn("Disabling sequence parallelism because tensor model parallelism is disabled")
|
| 569 |
+
WARNING: one_logger package is required to enable e2e metrics tracking. please go to https://confluence.nvidia.com/display/MLWFO/Package+Repositories for details to install it
|
| 570 |
+
/workspace/Megatron-MoE-Echo/megatron/training/arguments.py:831: UserWarning: Disabling sequence parallelism because tensor model parallelism is disabled
|
| 571 |
+
warnings.warn("Disabling sequence parallelism because tensor model parallelism is disabled")
|
| 572 |
+
/workspace/Megatron-MoE-Echo/megatron/training/arguments.py:831: UserWarning: Disabling sequence parallelism because tensor model parallelism is disabled
|
| 573 |
+
warnings.warn("Disabling sequence parallelism because tensor model parallelism is disabled")
|
| 574 |
+
/workspace/Megatron-MoE-Echo/megatron/training/arguments.py:831: UserWarning: Disabling sequence parallelism because tensor model parallelism is disabled
|
| 575 |
+
warnings.warn("Disabling sequence parallelism because tensor model parallelism is disabled")
|
| 576 |
+
[Gloo] Rank [Gloo] Rank 2[Gloo] Rank [Gloo] Rank [Gloo] Rank is connected to 01475 is connected to is connected to is connected to peer ranks. is connected to 777Expected number of connected peer ranks is : 7 peer ranks. peer ranks. peer ranks. 7 peer ranks. Expected number of connected peer ranks is : Expected number of connected peer ranks is : Expected number of connected peer ranks is :
|
| 577 |
+
Expected number of connected peer ranks is : 7777
|
| 578 |
+
|
| 579 |
+
|
| 580 |
+
|
| 581 |
+
[Gloo] Rank 3 is connected to 7 peer ranks. Expected number of connected peer ranks is : 7
|
| 582 |
+
[Gloo] Rank [Gloo] Rank 7 is connected to 67 is connected to peer ranks. 7Expected number of connected peer ranks is : peer ranks. 7Expected number of connected peer ranks is :
|
| 583 |
+
7
|
| 584 |
+
[Gloo] Rank 0 is connected to 7 peer ranks. Expected number of connected peer ranks is : 7
|
| 585 |
+
[Gloo] Rank 1 is connected to 7 peer ranks. Expected number of connected peer ranks is : 7
|
| 586 |
+
[Gloo] Rank 2 is connected to 7 peer ranks. Expected number of connected peer ranks is : 7
|
| 587 |
+
[Gloo] Rank 3 is connected to 7 peer ranks. Expected number of connected peer ranks is : 7
|
| 588 |
+
[Gloo] Rank 4 is connected to 7 peer ranks. Expected number of connected peer ranks is : 7
|
| 589 |
+
[Gloo] Rank 5 is connected to 7 peer ranks. Expected number of connected peer ranks is : 7
|
| 590 |
+
[Gloo] Rank 7 is connected to 7 peer ranks. Expected number of connected peer ranks is : 7
|
| 591 |
+
[Gloo] Rank 6 is connected to 7 peer ranks. Expected number of connected peer ranks is : 7
|
| 592 |
+
[Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
|
| 593 |
+
> initialized tensor model parallel with size 1
|
| 594 |
+
> initialized pipeline model parallel with size 1
|
| 595 |
+
> setting random seeds to 1234 ...
|
| 596 |
+
[Gloo] Rank [Gloo] Rank 0 is connected to 00 is connected to peer ranks. 0Expected number of connected peer ranks is : peer ranks. 0Expected number of connected peer ranks is :
|
| 597 |
+
0
|
| 598 |
+
[Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
|
| 599 |
+
[Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
|
| 600 |
+
[Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
|
| 601 |
+
[Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
|
| 602 |
+
[Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
|
| 603 |
+
[rank7]:[W830 06:46:03.578790501 ProcessGroupNCCL.cpp:4789] [PG ID 0 PG GUID 0 Rank 7] using GPU 7 as device used by this process is currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect. You can specify device_id in init_process_group() to force use of a particular device.
|
| 604 |
+
[rank2]:[W830 06:46:03.578807856 ProcessGroupNCCL.cpp:4789] [PG ID 0 PG GUID 0 Rank 2] using GPU 2 as device used by this process is currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect. You can specify device_id in init_process_group() to force use of a particular device.
|
| 605 |
+
[rank4]:[W830 06:46:03.578834248 ProcessGroupNCCL.cpp:4789] [PG ID 0 PG GUID 0 Rank 4] using GPU 4 as device used by this process is currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect. You can specify device_id in init_process_group() to force use of a particular device.
|
| 606 |
+
[rank1]:[W830 06:46:03.578850380 ProcessGroupNCCL.cpp:4789] [PG ID 0 PG GUID 0 Rank 1] using GPU 1 as device used by this process is currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect. You can specify device_id in init_process_group() to force use of a particular device.
|
| 607 |
+
[rank5]:[W830 06:46:03.578865697 ProcessGroupNCCL.cpp:4789] [PG ID 0 PG GUID 0 Rank 5] using GPU 5 as device used by this process is currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect. You can specify device_id in init_process_group() to force use of a particular device.
|
| 608 |
+
[rank3]:[W830 06:46:03.578912375 ProcessGroupNCCL.cpp:4789] [PG ID 0 PG GUID 0 Rank 3] using GPU 3 as device used by this process is currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect. You can specify device_id in init_process_group() to force use of a particular device.
|
| 609 |
+
[rank6]:[W830 06:46:03.578911167 ProcessGroupNCCL.cpp:4789] [PG ID 0 PG GUID 0 Rank 6] using GPU 6 as device used by this process is currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect. You can specify device_id in init_process_group() to force use of a particular device.
|
| 610 |
+
> compiling dataset index builder ...
|
| 611 |
+
make: Entering directory '/workspace/Megatron-MoE-Echo/megatron/core/datasets'
|
| 612 |
+
make: Nothing to be done for 'default'.
|
| 613 |
+
make: Leaving directory '/workspace/Megatron-MoE-Echo/megatron/core/datasets'
|
| 614 |
+
>>> done with dataset index builder. Compilation time: 0.034 seconds
|
| 615 |
+
WARNING: constraints for invoking optimized fused softmax kernel are not met. We default back to unfused kernel invocations.
|
| 616 |
+
> compiling and loading fused kernels ...
|
| 617 |
+
[rank0]:[W830 06:46:03.730419083 ProcessGroupNCCL.cpp:4789] [PG ID 0 PG GUID 0 Rank 0] using GPU 0 as device used by this process is currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect. You can specify device_id in init_process_group() to force use of a particular device.
|
| 618 |
+
>>> done with compiling and loading fused kernels. Compilation time: 3.842 seconds
|
| 619 |
+
time to initialize megatron (seconds): 7.199
|
| 620 |
+
[after megatron is initialized] datetime: 2026-08-30 06:46:09
|
| 621 |
+
building GPT model ...
|
| 622 |
+
/usr/local/lib/python3.12/dist-packages/transformer_engine/pytorch/cpu_offload.py:708: DeprecationWarning: Offloading weights is deprecated. Using offload_weights=True does not have any effect.
|
| 623 |
+
warnings.warn(
|
| 624 |
+
/usr/local/lib/python3.12/dist-packages/transformer_engine/pytorch/cpu_offload.py:708: DeprecationWarning: Offloading weights is deprecated. Using offload_weights=True does not have any effect.
|
| 625 |
+
warnings.warn(
|
| 626 |
+
/usr/local/lib/python3.12/dist-packages/transformer_engine/pytorch/cpu_offload.py:708: DeprecationWarning: Offloading weights is deprecated. Using offload_weights=True does not have any effect.
|
| 627 |
+
warnings.warn(
|
| 628 |
+
/usr/local/lib/python3.12/dist-packages/transformer_engine/pytorch/cpu_offload.py:708: DeprecationWarning: Offloading weights is deprecated. Using offload_weights=True does not have any effect.
|
| 629 |
+
warnings.warn(
|
| 630 |
+
/usr/local/lib/python3.12/dist-packages/transformer_engine/pytorch/cpu_offload.py:708: DeprecationWarning: Offloading weights is deprecated. Using offload_weights=True does not have any effect.
|
| 631 |
+
warnings.warn(
|
| 632 |
+
/usr/local/lib/python3.12/dist-packages/transformer_engine/pytorch/cpu_offload.py:708: DeprecationWarning: Offloading weights is deprecated. Using offload_weights=True does not have any effect.
|
| 633 |
+
warnings.warn(
|
| 634 |
+
/usr/local/lib/python3.12/dist-packages/transformer_engine/pytorch/cpu_offload.py:708: DeprecationWarning: Offloading weights is deprecated. Using offload_weights=True does not have any effect.
|
| 635 |
+
warnings.warn(
|
| 636 |
+
/usr/local/lib/python3.12/dist-packages/transformer_engine/pytorch/cpu_offload.py:708: DeprecationWarning: Offloading weights is deprecated. Using offload_weights=True does not have any effect.
|
| 637 |
+
warnings.warn(
|
| 638 |
+
> number of parameters on (tensor, pipeline) model parallel rank (0, 0): 3902410752
|
| 639 |
+
INFO:megatron.core.distributed.distributed_data_parallel:Setting up DistributedDataParallel with config DistributedDataParallelConfig(grad_reduce_in_fp32=True, overlap_grad_reduce=True, overlap_param_gather=True, align_param_gather=False, use_distributed_optimizer=True, num_distributed_optimizer_instances=1, check_for_nan_in_grad=False, check_for_large_grads=False, bucket_size=40000000, pad_buckets_for_high_nccl_busbw=False, average_in_collective=False, fp8_param_gather=False, reuse_grad_buf_for_mxfp8_param_ag=False, use_custom_fsdp=False, data_parallel_sharding_strategy='no_shard', gradient_reduce_div_fusion=True, suggested_communication_unit_size=None, preserve_fp32_weights=True, keep_fp8_transpose_cache_when_using_custom_fsdp=False, nccl_ub=False, fsdp_double_buffer=False)
|
| 640 |
+
INFO:megatron.core.distributed.param_and_grad_buffer:Number of buckets for gradient all-reduce / reduce-scatter: 18
|
| 641 |
+
Params for bucket 1 (206045184 elements, 206045184 padded size):
|
| 642 |
+
module.output_layer.weight
|
| 643 |
+
Params for bucket 2 (41988096 elements, 41988096 padded size):
|
| 644 |
+
module.decoder.final_layernorm.weight
|
| 645 |
+
module.decoder.layers.15.mlp.router.weight
|
| 646 |
+
module.decoder.layers.15.self_attention.linear_qkv.layer_norm_weight
|
| 647 |
+
module.decoder.layers.15.self_attention.linear_proj.weight
|
| 648 |
+
module.decoder.layers.15.pre_mlp_layernorm.weight
|
| 649 |
+
module.decoder.layers.15.self_attention.linear_qkv.weight
|
| 650 |
+
Params for bucket 3 (41984000 elements, 41984000 padded size):
|
| 651 |
+
module.decoder.layers.14.pre_mlp_layernorm.weight
|
| 652 |
+
module.decoder.layers.14.self_attention.linear_qkv.weight
|
| 653 |
+
module.decoder.layers.14.mlp.router.weight
|
| 654 |
+
module.decoder.layers.14.self_attention.linear_proj.weight
|
| 655 |
+
module.decoder.layers.14.self_attention.linear_qkv.layer_norm_weight
|
| 656 |
+
Params for bucket 4 (41984000 elements, 41984000 padded size):
|
| 657 |
+
module.decoder.layers.13.self_attention.linear_qkv.weight
|
| 658 |
+
module.decoder.layers.13.pre_mlp_layernorm.weight
|
| 659 |
+
module.decoder.layers.13.mlp.router.weight
|
| 660 |
+
module.decoder.layers.13.self_attention.linear_proj.weight
|
| 661 |
+
module.decoder.layers.13.self_attention.linear_qkv.layer_norm_weight
|
| 662 |
+
Params for bucket 5 (41984000 elements, 41984000 padded size):
|
| 663 |
+
module.decoder.layers.12.mlp.router.weight
|
| 664 |
+
module.decoder.layers.12.self_attention.linear_proj.weight
|
| 665 |
+
module.decoder.layers.12.self_attention.linear_qkv.layer_norm_weight
|
| 666 |
+
module.decoder.layers.12.self_attention.linear_qkv.weight
|
| 667 |
+
module.decoder.layers.12.pre_mlp_layernorm.weight
|
| 668 |
+
Params for bucket 6 (41984000 elements, 41984000 padded size):
|
| 669 |
+
module.decoder.layers.11.self_attention.linear_qkv.weight
|
| 670 |
+
module.decoder.layers.11.pre_mlp_layernorm.weight
|
| 671 |
+
module.decoder.layers.11.mlp.router.weight
|
| 672 |
+
module.decoder.layers.11.self_attention.linear_qkv.layer_norm_weight
|
| 673 |
+
module.decoder.layers.11.self_attention.linear_proj.weight
|
| 674 |
+
Params for bucket 7 (41984000 elements, 41984000 padded size):
|
| 675 |
+
module.decoder.layers.10.mlp.router.weight
|
| 676 |
+
module.decoder.layers.10.self_attention.linear_proj.weight
|
| 677 |
+
module.decoder.layers.10.self_attention.linear_qkv.layer_norm_weight
|
| 678 |
+
module.decoder.layers.10.self_attention.linear_qkv.weight
|
| 679 |
+
module.decoder.layers.10.pre_mlp_layernorm.weight
|
| 680 |
+
Params for bucket 8 (41984000 elements, 41984000 padded size):
|
| 681 |
+
module.decoder.layers.9.self_attention.linear_qkv.weight
|
| 682 |
+
module.decoder.layers.9.pre_mlp_layernorm.weight
|
| 683 |
+
module.decoder.layers.9.mlp.router.weight
|
| 684 |
+
module.decoder.layers.9.self_attention.linear_proj.weight
|
| 685 |
+
module.decoder.layers.9.self_attention.linear_qkv.layer_norm_weight
|
| 686 |
+
Params for bucket 9 (41984000 elements, 41984000 padded size):
|
| 687 |
+
module.decoder.layers.8.self_attention.linear_qkv.weight
|
| 688 |
+
module.decoder.layers.8.pre_mlp_layernorm.weight
|
| 689 |
+
module.decoder.layers.8.mlp.router.weight
|
| 690 |
+
module.decoder.layers.8.self_attention.linear_qkv.layer_norm_weight
|
| 691 |
+
module.decoder.layers.8.self_attention.linear_proj.weight
|
| 692 |
+
Params for bucket 10 (41984000 elements, 41984000 padded size):
|
| 693 |
+
module.decoder.layers.7.self_attention.linear_qkv.weight
|
| 694 |
+
module.decoder.layers.7.pre_mlp_layernorm.weight
|
| 695 |
+
module.decoder.layers.7.mlp.router.weight
|
| 696 |
+
module.decoder.layers.7.self_attention.linear_proj.weight
|
| 697 |
+
module.decoder.layers.7.self_attention.linear_qkv.layer_norm_weight
|
| 698 |
+
Params for bucket 11 (41984000 elements, 41984000 padded size):
|
| 699 |
+
module.decoder.layers.6.mlp.router.weight
|
| 700 |
+
module.decoder.layers.6.self_attention.linear_qkv.layer_norm_weight
|
| 701 |
+
module.decoder.layers.6.self_attention.linear_proj.weight
|
| 702 |
+
module.decoder.layers.6.self_attention.linear_qkv.weight
|
| 703 |
+
module.decoder.layers.6.pre_mlp_layernorm.weight
|
| 704 |
+
Params for bucket 12 (41984000 elements, 41984000 padded size):
|
| 705 |
+
module.decoder.layers.5.pre_mlp_layernorm.weight
|
| 706 |
+
module.decoder.layers.5.self_attention.linear_qkv.layer_norm_weight
|
| 707 |
+
module.decoder.layers.5.mlp.router.weight
|
| 708 |
+
module.decoder.layers.5.self_attention.linear_qkv.weight
|
| 709 |
+
module.decoder.layers.5.self_attention.linear_proj.weight
|
| 710 |
+
Params for bucket 13 (41984000 elements, 41984000 padded size):
|
| 711 |
+
module.decoder.layers.4.pre_mlp_layernorm.weight
|
| 712 |
+
module.decoder.layers.4.self_attention.linear_qkv.weight
|
| 713 |
+
module.decoder.layers.4.mlp.router.weight
|
| 714 |
+
module.decoder.layers.4.self_attention.linear_proj.weight
|
| 715 |
+
module.decoder.layers.4.self_attention.linear_qkv.layer_norm_weight
|
| 716 |
+
Params for bucket 14 (41984000 elements, 41984000 padded size):
|
| 717 |
+
module.decoder.layers.3.self_attention.linear_proj.weight
|
| 718 |
+
module.decoder.layers.3.self_attention.linear_qkv.layer_norm_weight
|
| 719 |
+
module.decoder.layers.3.mlp.router.weight
|
| 720 |
+
module.decoder.layers.3.pre_mlp_layernorm.weight
|
| 721 |
+
module.decoder.layers.3.self_attention.linear_qkv.weight
|
| 722 |
+
Params for bucket 15 (41984000 elements, 41984000 padded size):
|
| 723 |
+
module.decoder.layers.2.mlp.router.weight
|
| 724 |
+
module.decoder.layers.2.self_attention.linear_qkv.layer_norm_weight
|
| 725 |
+
module.decoder.layers.2.self_attention.linear_qkv.weight
|
| 726 |
+
module.decoder.layers.2.pre_mlp_layernorm.weight
|
| 727 |
+
module.decoder.layers.2.self_attention.linear_proj.weight
|
| 728 |
+
Params for bucket 16 (41984000 elements, 41984000 padded size):
|
| 729 |
+
module.decoder.layers.1.mlp.router.weight
|
| 730 |
+
module.decoder.layers.1.self_attention.linear_qkv.layer_norm_weight
|
| 731 |
+
module.decoder.layers.1.self_attention.linear_proj.weight
|
| 732 |
+
module.decoder.layers.1.pre_mlp_layernorm.weight
|
| 733 |
+
module.decoder.layers.1.self_attention.linear_qkv.weight
|
| 734 |
+
Params for bucket 17 (41984000 elements, 41984000 padded size):
|
| 735 |
+
module.decoder.layers.0.mlp.router.weight
|
| 736 |
+
module.decoder.layers.0.pre_mlp_layernorm.weight
|
| 737 |
+
module.decoder.layers.0.self_attention.linear_proj.weight
|
| 738 |
+
module.decoder.layers.0.self_attention.linear_qkv.weight
|
| 739 |
+
module.decoder.layers.0.self_attention.linear_qkv.layer_norm_weight
|
| 740 |
+
Params for bucket 18 (206045184 elements, 206045184 padded size):
|
| 741 |
+
module.embedding.word_embeddings.weight
|
| 742 |
+
INFO:megatron.core.distributed.param_and_grad_buffer:Number of buckets for gradient all-reduce / reduce-scatter: 32
|
| 743 |
+
Params for bucket 1 (58720256 elements, 58720256 padded size):
|
| 744 |
+
module.decoder.layers.15.mlp.experts.linear_fc2.weight0
|
| 745 |
+
Params for bucket 2 (117440512 elements, 117440512 padded size):
|
| 746 |
+
module.decoder.layers.15.mlp.experts.linear_fc1.weight0
|
| 747 |
+
Params for bucket 3 (58720256 elements, 58720256 padded size):
|
| 748 |
+
module.decoder.layers.14.mlp.experts.linear_fc2.weight0
|
| 749 |
+
Params for bucket 4 (117440512 elements, 117440512 padded size):
|
| 750 |
+
module.decoder.layers.14.mlp.experts.linear_fc1.weight0
|
| 751 |
+
Params for bucket 5 (58720256 elements, 58720256 padded size):
|
| 752 |
+
module.decoder.layers.13.mlp.experts.linear_fc2.weight0
|
| 753 |
+
Params for bucket 6 (117440512 elements, 117440512 padded size):
|
| 754 |
+
module.decoder.layers.13.mlp.experts.linear_fc1.weight0
|
| 755 |
+
Params for bucket 7 (58720256 elements, 58720256 padded size):
|
| 756 |
+
module.decoder.layers.12.mlp.experts.linear_fc2.weight0
|
| 757 |
+
Params for bucket 8 (117440512 elements, 117440512 padded size):
|
| 758 |
+
module.decoder.layers.12.mlp.experts.linear_fc1.weight0
|
| 759 |
+
Params for bucket 9 (58720256 elements, 58720256 padded size):
|
| 760 |
+
module.decoder.layers.11.mlp.experts.linear_fc2.weight0
|
| 761 |
+
Params for bucket 10 (117440512 elements, 117440512 padded size):
|
| 762 |
+
module.decoder.layers.11.mlp.experts.linear_fc1.weight0
|
| 763 |
+
Params for bucket 11 (58720256 elements, 58720256 padded size):
|
| 764 |
+
module.decoder.layers.10.mlp.experts.linear_fc2.weight0
|
| 765 |
+
Params for bucket 12 (117440512 elements, 117440512 padded size):
|
| 766 |
+
module.decoder.layers.10.mlp.experts.linear_fc1.weight0
|
| 767 |
+
Params for bucket 13 (58720256 elements, 58720256 padded size):
|
| 768 |
+
module.decoder.layers.9.mlp.experts.linear_fc2.weight0
|
| 769 |
+
Params for bucket 14 (117440512 elements, 117440512 padded size):
|
| 770 |
+
module.decoder.layers.9.mlp.experts.linear_fc1.weight0
|
| 771 |
+
Params for bucket 15 (58720256 elements, 58720256 padded size):
|
| 772 |
+
module.decoder.layers.8.mlp.experts.linear_fc2.weight0
|
| 773 |
+
Params for bucket 16 (117440512 elements, 117440512 padded size):
|
| 774 |
+
module.decoder.layers.8.mlp.experts.linear_fc1.weight0
|
| 775 |
+
Params for bucket 17 (58720256 elements, 58720256 padded size):
|
| 776 |
+
module.decoder.layers.7.mlp.experts.linear_fc2.weight0
|
| 777 |
+
Params for bucket 18 (117440512 elements, 117440512 padded size):
|
| 778 |
+
module.decoder.layers.7.mlp.experts.linear_fc1.weight0
|
| 779 |
+
Params for bucket 19 (58720256 elements, 58720256 padded size):
|
| 780 |
+
module.decoder.layers.6.mlp.experts.linear_fc2.weight0
|
| 781 |
+
Params for bucket 20 (117440512 elements, 117440512 padded size):
|
| 782 |
+
module.decoder.layers.6.mlp.experts.linear_fc1.weight0
|
| 783 |
+
Params for bucket 21 (58720256 elements, 58720256 padded size):
|
| 784 |
+
module.decoder.layers.5.mlp.experts.linear_fc2.weight0
|
| 785 |
+
Params for bucket 22 (117440512 elements, 117440512 padded size):
|
| 786 |
+
module.decoder.layers.5.mlp.experts.linear_fc1.weight0
|
| 787 |
+
Params for bucket 23 (58720256 elements, 58720256 padded size):
|
| 788 |
+
module.decoder.layers.4.mlp.experts.linear_fc2.weight0
|
| 789 |
+
Params for bucket 24 (117440512 elements, 117440512 padded size):
|
| 790 |
+
module.decoder.layers.4.mlp.experts.linear_fc1.weight0
|
| 791 |
+
Params for bucket 25 (58720256 elements, 58720256 padded size):
|
| 792 |
+
module.decoder.layers.3.mlp.experts.linear_fc2.weight0
|
| 793 |
+
Params for bucket 26 (117440512 elements, 117440512 padded size):
|
| 794 |
+
module.decoder.layers.3.mlp.experts.linear_fc1.weight0
|
| 795 |
+
Params for bucket 27 (58720256 elements, 58720256 padded size):
|
| 796 |
+
module.decoder.layers.2.mlp.experts.linear_fc2.weight0
|
| 797 |
+
Params for bucket 28 (117440512 elements, 117440512 padded size):
|
| 798 |
+
module.decoder.layers.2.mlp.experts.linear_fc1.weight0
|
| 799 |
+
Params for bucket 29 (58720256 elements, 58720256 padded size):
|
| 800 |
+
module.decoder.layers.1.mlp.experts.linear_fc2.weight0
|
| 801 |
+
Params for bucket 30 (117440512 elements, 117440512 padded size):
|
| 802 |
+
module.decoder.layers.1.mlp.experts.linear_fc1.weight0
|
| 803 |
+
Params for bucket 31 (58720256 elements, 58720256 padded size):
|
| 804 |
+
module.decoder.layers.0.mlp.experts.linear_fc2.weight0
|
| 805 |
+
Params for bucket 32 (117440512 elements, 117440512 padded size):
|
| 806 |
+
module.decoder.layers.0.mlp.experts.linear_fc1.weight0
|
| 807 |
+
INFO:megatron.core.optimizer:Setting up optimizer with config OptimizerConfig(optimizer='adam', lr=0.0001, min_lr=1e-05, decoupled_lr=None, decoupled_min_lr=None, weight_decay=0.1, fp8_recipe='mxfp8', fp16=False, bf16=True, reuse_grad_buf_for_mxfp8_param_ag=False, params_dtype=torch.bfloat16, use_precision_aware_optimizer=False, store_param_remainders=True, main_grads_dtype=torch.float32, main_params_dtype=torch.float32, exp_avg_dtype=torch.float32, exp_avg_sq_dtype=torch.float32, loss_scale=None, initial_loss_scale=4294967296, min_loss_scale=1.0, loss_scale_window=1000, hysteresis=2, adam_beta1=0.9, adam_beta2=0.999, adam_eps=1e-08, sgd_momentum=0.9, use_distributed_optimizer=True, overlap_param_gather_with_optimizer_step=False, optimizer_cpu_offload=False, optimizer_offload_fraction=1.0, use_torch_optimizer_for_cpu_offload=False, overlap_cpu_optimizer_d2h_h2d=False, pin_cpu_grads=True, pin_cpu_params=True, clip_grad=1.0, log_num_zeros_in_grad=False, barrier_with_L1_time=True, timers=<megatron.core.timers.Timers object at 0x73a55704c740>, config_logger_dir='')
|
| 808 |
+
INFO:megatron.core.optimizer_param_scheduler:> learning rate decay style: cosine
|
| 809 |
+
[after model, optimizer, and learning rate scheduler are built] datetime: 2026-08-30 06:46:09
|
| 810 |
+
> building train, validation, and test datasets ...
|
| 811 |
+
> datasets target sizes (minimum size):
|
| 812 |
+
train: 192
|
| 813 |
+
validation: 0
|
| 814 |
+
test: 0
|
| 815 |
+
INFO:megatron.core.datasets.blended_megatron_dataset_config:Let split_matrix = [(0, 1.0), None, None]
|
| 816 |
+
> building train, validation, and test datasets for GPT ...
|
| 817 |
+
INFO:megatron.core.datasets.blended_megatron_dataset_builder:Building GPTDataset splits with sizes=(192, 0, 0) and config=GPTDatasetConfig(random_seed=1234, sequence_length=4096, blend=(['/workspace/data/wikitext/wikitext103_gpt2_text_document'], None), blend_per_split=None, split='100,0,0', split_matrix=[(0, 1.0), None, None], num_dataset_builder_threads=1, path_to_cache=None, mmap_bin_files=True, mock=False, tokenizer=<megatron.training.tokenizer.tokenizer._GPT2BPETokenizer object at 0x73a55892f440>, mid_level_dataset_surplus=0.005, reset_position_ids=False, reset_attention_mask=False, eod_mask_loss=False, create_attention_mask=True, drop_last_partial_validation_sequence=True, add_extra_token_to_sequence=True, object_storage_cache_path=None)
|
| 818 |
+
INFO:megatron.core.datasets.indexed_dataset:Load the _IndexReader from /workspace/data/wikitext/wikitext103_gpt2_text_document.idx
|
| 819 |
+
INFO:megatron.core.datasets.indexed_dataset: Extract the sequence lengths
|
| 820 |
+
INFO:megatron.core.datasets.indexed_dataset: Extract the sequence pointers
|
| 821 |
+
INFO:megatron.core.datasets.indexed_dataset: Extract the document indices
|
| 822 |
+
INFO:megatron.core.datasets.indexed_dataset:> total number of sequences: 1165029
|
| 823 |
+
INFO:megatron.core.datasets.indexed_dataset:> total number of documents: 1801350
|
| 824 |
+
INFO:megatron.core.datasets.gpt_dataset:Load the GPTDataset train indices
|
| 825 |
+
INFO:megatron.core.datasets.gpt_dataset: Load the document index from 44ed1ee5bcb9e5b030387b8d27a6c406-GPTDataset-train-document_index.npy
|
| 826 |
+
INFO:megatron.core.datasets.gpt_dataset: Load the sample index from 44ed1ee5bcb9e5b030387b8d27a6c406-GPTDataset-train-sample_index.npy
|
| 827 |
+
INFO:megatron.core.datasets.gpt_dataset: Load the shuffle index from 44ed1ee5bcb9e5b030387b8d27a6c406-GPTDataset-train-shuffle_index.npy
|
| 828 |
+
INFO:megatron.core.datasets.gpt_dataset:> total number of samples: 29073
|
| 829 |
+
> finished creating GPT datasets ...
|
| 830 |
+
[after dataloaders are built] datetime: 2026-08-30 06:46:09
|
| 831 |
+
done with setup ...
|
| 832 |
+
(min, max) time across ranks (ms):
|
| 833 |
+
model-and-optimizer-setup ......................: (204.86, 207.92)
|
| 834 |
+
train/valid/test-data-iterators-setup ..........: (65.32, 74.72)
|
| 835 |
+
training ...
|
| 836 |
+
Overwriting rerun_state_machine.current_iteration from -1 to 0...
|
| 837 |
+
[before the start of training step] datetime: 2026-08-30 06:46:09
|
| 838 |
+
/workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:334: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
|
| 839 |
+
warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
|
| 840 |
+
/workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:334: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
|
| 841 |
+
warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
|
| 842 |
+
/workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:334: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
|
| 843 |
+
warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
|
| 844 |
+
/workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:334: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
|
| 845 |
+
warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
|
| 846 |
+
/workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:334: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
|
| 847 |
+
warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
|
| 848 |
+
/workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:336: UserWarning: The use_host_meta is deprecated, it will be removed in the future.
|
| 849 |
+
warnings.warn("The use_host_meta is deprecated, it will be removed in the future.")
|
| 850 |
+
/workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:334: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
|
| 851 |
+
warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
|
| 852 |
+
/workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:336: UserWarning: The use_host_meta is deprecated, it will be removed in the future.
|
| 853 |
+
warnings.warn("The use_host_meta is deprecated, it will be removed in the future.")
|
| 854 |
+
/workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:336: UserWarning: The use_host_meta is deprecated, it will be removed in the future.
|
| 855 |
+
warnings.warn("The use_host_meta is deprecated, it will be removed in the future.")
|
| 856 |
+
/workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:334: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
|
| 857 |
+
warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
|
| 858 |
+
/workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:334: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
|
| 859 |
+
warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
|
| 860 |
+
/workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:336: UserWarning: The use_host_meta is deprecated, it will be removed in the future.
|
| 861 |
+
warnings.warn("The use_host_meta is deprecated, it will be removed in the future.")
|
| 862 |
+
/workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:336: UserWarning: The use_host_meta is deprecated, it will be removed in the future.
|
| 863 |
+
warnings.warn("The use_host_meta is deprecated, it will be removed in the future.")
|
| 864 |
+
/workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:336: UserWarning: The use_host_meta is deprecated, it will be removed in the future.
|
| 865 |
+
warnings.warn("The use_host_meta is deprecated, it will be removed in the future.")
|
| 866 |
+
/workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:336: UserWarning: The use_host_meta is deprecated, it will be removed in the future.
|
| 867 |
+
warnings.warn("The use_host_meta is deprecated, it will be removed in the future.")
|
| 868 |
+
/workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:336: UserWarning: The use_host_meta is deprecated, it will be removed in the future.
|
| 869 |
+
warnings.warn("The use_host_meta is deprecated, it will be removed in the future.")
|
| 870 |
+
/workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:482: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
|
| 871 |
+
warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
|
| 872 |
+
/workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:482: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
|
| 873 |
+
warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
|
| 874 |
+
/workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:482: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
|
| 875 |
+
warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
|
| 876 |
+
/workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:482: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
|
| 877 |
+
warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
|
| 878 |
+
/workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:482: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
|
| 879 |
+
warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
|
| 880 |
+
/workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:482: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
|
| 881 |
+
warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
|
| 882 |
+
/workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:482: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
|
| 883 |
+
warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
|
| 884 |
+
/workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:482: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
|
| 885 |
+
warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
|
| 886 |
+
/workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:334: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
|
| 887 |
+
warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
|
| 888 |
+
/workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:336: UserWarning: The use_host_meta is deprecated, it will be removed in the future.
|
| 889 |
+
warnings.warn("The use_host_meta is deprecated, it will be removed in the future.")
|
| 890 |
+
/workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:334: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
|
| 891 |
+
warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
|
| 892 |
+
/workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:336: UserWarning: The use_host_meta is deprecated, it will be removed in the future.
|
| 893 |
+
warnings.warn("The use_host_meta is deprecated, it will be removed in the future.")
|
| 894 |
+
/workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:334: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
|
| 895 |
+
warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
|
| 896 |
+
/workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:336: UserWarning: The use_host_meta is deprecated, it will be removed in the future.
|
| 897 |
+
warnings.warn("The use_host_meta is deprecated, it will be removed in the future.")
|
| 898 |
+
/workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:334: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
|
| 899 |
+
warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
|
| 900 |
+
/workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:336: UserWarning: The use_host_meta is deprecated, it will be removed in the future.
|
| 901 |
+
warnings.warn("The use_host_meta is deprecated, it will be removed in the future.")
|
| 902 |
+
/workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:334: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
|
| 903 |
+
warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
|
| 904 |
+
/workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:336: UserWarning: The use_host_meta is deprecated, it will be removed in the future.
|
| 905 |
+
warnings.warn("The use_host_meta is deprecated, it will be removed in the future.")
|
| 906 |
+
/workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:334: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
|
| 907 |
+
warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
|
| 908 |
+
/workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:336: UserWarning: The use_host_meta is deprecated, it will be removed in the future.
|
| 909 |
+
warnings.warn("The use_host_meta is deprecated, it will be removed in the future.")
|
| 910 |
+
/workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:334: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
|
| 911 |
+
warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
|
| 912 |
+
/workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:336: UserWarning: The use_host_meta is deprecated, it will be removed in the future.
|
| 913 |
+
warnings.warn("The use_host_meta is deprecated, it will be removed in the future.")
|
| 914 |
+
/workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:334: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
|
| 915 |
+
warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
|
| 916 |
+
/workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:336: UserWarning: The use_host_meta is deprecated, it will be removed in the future.
|
| 917 |
+
warnings.warn("The use_host_meta is deprecated, it will be removed in the future.")
|
| 918 |
+
/workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:482: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
|
| 919 |
+
warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
|
| 920 |
+
/workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:482: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
|
| 921 |
+
warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
|
| 922 |
+
/workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:482: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
|
| 923 |
+
warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
|
| 924 |
+
/workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:482: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
|
| 925 |
+
warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
|
| 926 |
+
/workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:482: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
|
| 927 |
+
warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
|
| 928 |
+
/workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:482: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
|
| 929 |
+
warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
|
| 930 |
+
/workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:482: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
|
| 931 |
+
warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
|
| 932 |
+
/workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:482: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
|
| 933 |
+
warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
|
| 934 |
+
/workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:334: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
|
| 935 |
+
warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
|
| 936 |
+
/workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:334: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
|
| 937 |
+
warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
|
| 938 |
+
/workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:334: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
|
| 939 |
+
warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
|
| 940 |
+
/workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:334: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
|
| 941 |
+
warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
|
| 942 |
+
/workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:334: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
|
| 943 |
+
warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
|
| 944 |
+
/workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:334: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
|
| 945 |
+
warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
|
| 946 |
+
/workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:334: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
|
| 947 |
+
warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
|
| 948 |
+
/workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:334: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
|
| 949 |
+
warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
|
| 950 |
+
/workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:336: UserWarning: The use_host_meta is deprecated, it will be removed in the future.
|
| 951 |
+
warnings.warn("The use_host_meta is deprecated, it will be removed in the future.")
|
| 952 |
+
/workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:336: UserWarning: The use_host_meta is deprecated, it will be removed in the future.
|
| 953 |
+
warnings.warn("The use_host_meta is deprecated, it will be removed in the future.")
|
| 954 |
+
/workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:336: UserWarning: The use_host_meta is deprecated, it will be removed in the future.
|
| 955 |
+
warnings.warn("The use_host_meta is deprecated, it will be removed in the future.")
|
| 956 |
+
/workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:336: UserWarning: The use_host_meta is deprecated, it will be removed in the future.
|
| 957 |
+
warnings.warn("The use_host_meta is deprecated, it will be removed in the future.")
|
| 958 |
+
/workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:336: UserWarning: The use_host_meta is deprecated, it will be removed in the future.
|
| 959 |
+
warnings.warn("The use_host_meta is deprecated, it will be removed in the future.")
|
| 960 |
+
/workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:336: UserWarning: The use_host_meta is deprecated, it will be removed in the future.
|
| 961 |
+
warnings.warn("The use_host_meta is deprecated, it will be removed in the future.")
|
| 962 |
+
/workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:336: UserWarning: The use_host_meta is deprecated, it will be removed in the future.
|
| 963 |
+
warnings.warn("The use_host_meta is deprecated, it will be removed in the future.")
|
| 964 |
+
/workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:336: UserWarning: The use_host_meta is deprecated, it will be removed in the future.
|
| 965 |
+
warnings.warn("The use_host_meta is deprecated, it will be removed in the future.")
|
| 966 |
+
[2026-08-30 06:46:47] iteration 1/ 12 | consumed samples: 16 | elapsed time per iteration (ms): 38131.4 | throughput per GPU (TFLOP/s/GPU): 8.7 | learning rate: 2.000000E-05 | global batch size: 16 | lm loss: 1.104376E+01 | loss scale: 1.0 | grad norm: 34.299 | number of skipped iterations: 0 | number of nan iterations: 0 |Number of parameters in transformer block in billions: 23.22
|
| 967 |
+
|
| 968 |
+
Number of parameters in embedding layers in billions: 0.41
|
| 969 |
+
Total number of parameters in billions: 23.63
|
| 970 |
+
Number of parameters in most loaded shard in billions: 23.6320
|
| 971 |
+
Theoretical memory footprints: weight and optimizer=169029.43 MB
|
| 972 |
+
[Rank 0] (after 1 iterations) memory (MB) | allocated: 63795.404296875 | max allocated: 96519.67578125 | reserved: 99122.0 | max reserved: 99122.0
|
| 973 |
+
[2026-08-30 06:46:48] iteration 2/ 12 | consumed samples: 32 | elapsed time per iteration (ms): 573.2 | throughput per GPU (TFLOP/s/GPU): 581.6 | learning rate: 4.000000E-05 | global batch size: 16 | lm loss: 1.104935E+01 | loss scale: 1.0 | grad norm: 35.988 | number of skipped iterations: 0 | number of nan iterations: 0 |
|
| 974 |
+
[2026-08-30 06:46:48] iteration 3/ 12 | consumed samples: 48 | elapsed time per iteration (ms): 559.6 | throughput per GPU (TFLOP/s/GPU): 595.8 | learning rate: 6.000000E-05 | global batch size: 16 | lm loss: 9.657887E+00 | loss scale: 1.0 | grad norm: 40.365 | number of skipped iterations: 0 | number of nan iterations: 0 |
|
| 975 |
+
Capture CUDA graph for training!!!
|
| 976 |
+
CUDA graph capture done!!!
|
| 977 |
+
[2026-08-30 06:46:50] iteration 4/ 12 | consumed samples: 64 | elapsed time per iteration (ms): 1666.0 | throughput per GPU (TFLOP/s/GPU): 200.1 | learning rate: 8.000000E-05 | global batch size: 16 | lm loss: 1.081011E+01 | loss scale: 1.0 | grad norm: 92.418 | number of skipped iterations: 0 | number of nan iterations: 0 |
|
| 978 |
+
[2026-08-30 06:46:51] iteration 5/ 12 | consumed samples: 80 | elapsed time per iteration (ms): 469.9 | throughput per GPU (TFLOP/s/GPU): 709.4 | learning rate: 1.000000E-04 | global batch size: 16 | lm loss: 1.069354E+01 | loss scale: 1.0 | grad norm: 17.104 | number of skipped iterations: 0 | number of nan iterations: 0 |
|
| 979 |
+
[2026-08-30 06:46:51] iteration 6/ 12 | consumed samples: 96 | elapsed time per iteration (ms): 469.0 | throughput per GPU (TFLOP/s/GPU): 710.9 | learning rate: 7.750000E-05 | global batch size: 16 | lm loss: 9.820256E+00 | loss scale: 1.0 | grad norm: 38.716 | number of skipped iterations: 0 | number of nan iterations: 0 |
|
| 980 |
+
[2026-08-30 06:46:52] iteration 7/ 12 | consumed samples: 112 | elapsed time per iteration (ms): 466.1 | throughput per GPU (TFLOP/s/GPU): 715.2 | learning rate: 3.250000E-05 | global batch size: 16 | lm loss: 9.326885E+00 | loss scale: 1.0 | grad norm: 6.996 | number of skipped iterations: 0 | number of nan iterations: 0 |
|
| 981 |
+
[2026-08-30 06:46:52] iteration 8/ 12 | consumed samples: 128 | elapsed time per iteration (ms): 466.8 | throughput per GPU (TFLOP/s/GPU): 714.2 | learning rate: 1.000000E-05 | global batch size: 16 | lm loss: 9.158098E+00 | loss scale: 1.0 | grad norm: 28.651 | number of skipped iterations: 0 | number of nan iterations: 0 |
|
| 982 |
+
[2026-08-30 06:46:52] iteration 9/ 12 | consumed samples: 144 | elapsed time per iteration (ms): 463.5 | throughput per GPU (TFLOP/s/GPU): 719.3 | learning rate: 1.000000E-05 | global batch size: 16 | lm loss: 8.723837E+00 | loss scale: 1.0 | grad norm: 5.718 | number of skipped iterations: 0 | number of nan iterations: 0 |
|
| 983 |
+
[2026-08-30 06:46:53] iteration 10/ 12 | consumed samples: 160 | elapsed time per iteration (ms): 462.6 | throughput per GPU (TFLOP/s/GPU): 720.6 | learning rate: 1.000000E-05 | global batch size: 16 | lm loss: 8.580932E+00 | loss scale: 1.0 | grad norm: 5.040 | number of skipped iterations: 0 | number of nan iterations: 0 |
|
| 984 |
+
[2026-08-30 06:46:53] iteration 11/ 12 | consumed samples: 176 | elapsed time per iteration (ms): 470.7 | throughput per GPU (TFLOP/s/GPU): 708.2 | learning rate: 1.000000E-05 | global batch size: 16 | lm loss: 8.497838E+00 | loss scale: 1.0 | grad norm: 4.420 | number of skipped iterations: 0 | number of nan iterations: 0 |
|
| 985 |
+
[2026-08-30 06:46:54] iteration 12/ 12 | consumed samples: 192 | elapsed time per iteration (ms): 466.7 | throughput per GPU (TFLOP/s/GPU): 714.3 | learning rate: 1.000000E-05 | global batch size: 16 | lm loss: 8.392807E+00 | loss scale: 1.0 | grad norm: 4.866 | number of skipped iterations: 0 | number of nan iterations: 0 |
|
| 986 |
+
[after training is done] datetime: 2026-08-30 06:46:54
|
| 987 |
+
[rank0]:[W830 06:46:55.659787361 ProcessGroupNCCL.cpp:1505] Warning: WARNING: destroy_process_group() was not called before program exit, which can leak resources. For more info, please see https://pytorch.org/docs/stable/distributed.html#shutdown (function operator())
|
tiera-rerun-fixed-moonep/echo/a2/a2_matched_gbs16/G1__echo__matched__seq4096__gbs16__cap2p40625/rep3/mem_samples.csv
ADDED
|
@@ -0,0 +1,232 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
0, 0
|
| 2 |
+
1, 0
|
| 3 |
+
2, 0
|
| 4 |
+
3, 0
|
| 5 |
+
4, 0
|
| 6 |
+
5, 0
|
| 7 |
+
6, 0
|
| 8 |
+
7, 0
|
| 9 |
+
0, 0
|
| 10 |
+
1, 0
|
| 11 |
+
2, 0
|
| 12 |
+
3, 0
|
| 13 |
+
4, 0
|
| 14 |
+
5, 0
|
| 15 |
+
6, 0
|
| 16 |
+
7, 0
|
| 17 |
+
0, 4
|
| 18 |
+
1, 1
|
| 19 |
+
2, 1
|
| 20 |
+
3, 1
|
| 21 |
+
4, 1
|
| 22 |
+
5, 1
|
| 23 |
+
6, 1
|
| 24 |
+
7, 1
|
| 25 |
+
0, 4
|
| 26 |
+
1, 4
|
| 27 |
+
2, 4
|
| 28 |
+
3, 4
|
| 29 |
+
4, 2
|
| 30 |
+
5, 1
|
| 31 |
+
6, 1
|
| 32 |
+
7, 1
|
| 33 |
+
0, 4
|
| 34 |
+
1, 4
|
| 35 |
+
2, 4
|
| 36 |
+
3, 4
|
| 37 |
+
4, 4
|
| 38 |
+
5, 4
|
| 39 |
+
6, 4
|
| 40 |
+
7, 4
|
| 41 |
+
0, 821
|
| 42 |
+
1, 819
|
| 43 |
+
2, 821
|
| 44 |
+
3, 819
|
| 45 |
+
4, 819
|
| 46 |
+
5, 819
|
| 47 |
+
6, 821
|
| 48 |
+
7, 821
|
| 49 |
+
0, 1537
|
| 50 |
+
1, 1537
|
| 51 |
+
2, 1537
|
| 52 |
+
3, 1537
|
| 53 |
+
4, 1537
|
| 54 |
+
5, 1537
|
| 55 |
+
6, 1537
|
| 56 |
+
7, 1537
|
| 57 |
+
0, 2231
|
| 58 |
+
1, 2231
|
| 59 |
+
2, 2231
|
| 60 |
+
3, 2231
|
| 61 |
+
4, 2231
|
| 62 |
+
5, 2231
|
| 63 |
+
6, 2231
|
| 64 |
+
7, 2231
|
| 65 |
+
0, 39533
|
| 66 |
+
1, 39545
|
| 67 |
+
2, 39523
|
| 68 |
+
3, 39531
|
| 69 |
+
4, 39541
|
| 70 |
+
5, 39549
|
| 71 |
+
6, 39557
|
| 72 |
+
7, 39595
|
| 73 |
+
0, 39821
|
| 74 |
+
1, 39803
|
| 75 |
+
2, 39823
|
| 76 |
+
3, 39823
|
| 77 |
+
4, 39827
|
| 78 |
+
5, 39845
|
| 79 |
+
6, 39833
|
| 80 |
+
7, 39831
|
| 81 |
+
0, 42357
|
| 82 |
+
1, 42355
|
| 83 |
+
2, 42355
|
| 84 |
+
3, 42355
|
| 85 |
+
4, 42355
|
| 86 |
+
5, 42355
|
| 87 |
+
6, 42355
|
| 88 |
+
7, 42355
|
| 89 |
+
0, 42583
|
| 90 |
+
1, 42581
|
| 91 |
+
2, 42581
|
| 92 |
+
3, 42581
|
| 93 |
+
4, 42581
|
| 94 |
+
5, 42581
|
| 95 |
+
6, 42581
|
| 96 |
+
7, 42581
|
| 97 |
+
0, 42713
|
| 98 |
+
1, 42739
|
| 99 |
+
2, 42723
|
| 100 |
+
3, 42697
|
| 101 |
+
4, 42723
|
| 102 |
+
5, 42759
|
| 103 |
+
6, 42735
|
| 104 |
+
7, 42721
|
| 105 |
+
0, 43493
|
| 106 |
+
1, 43491
|
| 107 |
+
2, 43491
|
| 108 |
+
3, 43491
|
| 109 |
+
4, 43491
|
| 110 |
+
5, 43491
|
| 111 |
+
6, 43491
|
| 112 |
+
7, 43491
|
| 113 |
+
0, 43493
|
| 114 |
+
1, 43491
|
| 115 |
+
2, 43491
|
| 116 |
+
3, 43491
|
| 117 |
+
4, 43491
|
| 118 |
+
5, 43491
|
| 119 |
+
6, 43491
|
| 120 |
+
7, 43491
|
| 121 |
+
0, 45265
|
| 122 |
+
1, 45263
|
| 123 |
+
2, 45263
|
| 124 |
+
3, 45263
|
| 125 |
+
4, 45263
|
| 126 |
+
5, 45263
|
| 127 |
+
6, 45263
|
| 128 |
+
7, 45263
|
| 129 |
+
0, 45265
|
| 130 |
+
1, 45263
|
| 131 |
+
2, 45263
|
| 132 |
+
3, 45263
|
| 133 |
+
4, 45263
|
| 134 |
+
5, 45263
|
| 135 |
+
6, 45263
|
| 136 |
+
7, 45263
|
| 137 |
+
0, 85827
|
| 138 |
+
1, 85825
|
| 139 |
+
2, 85825
|
| 140 |
+
3, 85825
|
| 141 |
+
4, 85825
|
| 142 |
+
5, 85825
|
| 143 |
+
6, 85825
|
| 144 |
+
7, 85825
|
| 145 |
+
0, 86905
|
| 146 |
+
1, 86903
|
| 147 |
+
2, 86903
|
| 148 |
+
3, 86903
|
| 149 |
+
4, 86903
|
| 150 |
+
5, 86903
|
| 151 |
+
6, 86903
|
| 152 |
+
7, 86903
|
| 153 |
+
0, 86905
|
| 154 |
+
1, 86903
|
| 155 |
+
2, 86903
|
| 156 |
+
3, 86903
|
| 157 |
+
4, 86903
|
| 158 |
+
5, 86903
|
| 159 |
+
6, 86903
|
| 160 |
+
7, 86903
|
| 161 |
+
0, 86905
|
| 162 |
+
1, 86903
|
| 163 |
+
2, 86903
|
| 164 |
+
3, 86903
|
| 165 |
+
4, 86903
|
| 166 |
+
5, 86903
|
| 167 |
+
6, 86903
|
| 168 |
+
7, 86903
|
| 169 |
+
0, 104113
|
| 170 |
+
1, 104035
|
| 171 |
+
2, 104045
|
| 172 |
+
3, 104045
|
| 173 |
+
4, 104121
|
| 174 |
+
5, 104075
|
| 175 |
+
6, 104065
|
| 176 |
+
7, 104095
|
| 177 |
+
0, 105837
|
| 178 |
+
1, 105815
|
| 179 |
+
2, 105815
|
| 180 |
+
3, 105801
|
| 181 |
+
4, 105871
|
| 182 |
+
5, 105815
|
| 183 |
+
6, 105815
|
| 184 |
+
7, 105815
|
| 185 |
+
0, 106523
|
| 186 |
+
1, 106465
|
| 187 |
+
2, 106465
|
| 188 |
+
3, 106465
|
| 189 |
+
4, 106521
|
| 190 |
+
5, 106465
|
| 191 |
+
6, 106465
|
| 192 |
+
7, 106465
|
| 193 |
+
0, 107185
|
| 194 |
+
1, 107049
|
| 195 |
+
2, 107065
|
| 196 |
+
3, 107077
|
| 197 |
+
4, 107101
|
| 198 |
+
5, 107067
|
| 199 |
+
6, 107061
|
| 200 |
+
7, 107085
|
| 201 |
+
0, 136161
|
| 202 |
+
1, 138979
|
| 203 |
+
2, 156317
|
| 204 |
+
3, 136653
|
| 205 |
+
4, 135565
|
| 206 |
+
5, 146571
|
| 207 |
+
6, 135821
|
| 208 |
+
7, 135821
|
| 209 |
+
0, 161639
|
| 210 |
+
1, 161527
|
| 211 |
+
2, 161525
|
| 212 |
+
3, 161525
|
| 213 |
+
4, 161723
|
| 214 |
+
5, 161525
|
| 215 |
+
6, 161525
|
| 216 |
+
7, 161525
|
| 217 |
+
0, 161639
|
| 218 |
+
1, 161527
|
| 219 |
+
2, 161525
|
| 220 |
+
3, 161525
|
| 221 |
+
4, 161723
|
| 222 |
+
5, 161525
|
| 223 |
+
6, 161525
|
| 224 |
+
7, 161525
|
| 225 |
+
0, 2098
|
| 226 |
+
1, 2098
|
| 227 |
+
2, 2418
|
| 228 |
+
3, 161525
|
| 229 |
+
4, 3698
|
| 230 |
+
5, 2098
|
| 231 |
+
6, 2098
|
| 232 |
+
7, 2098
|
tiera-rerun-fixed-moonep/echo/a2/a2_matched_gbs16/G1__echo__matched__seq4096__gbs16__cap2p40625/rep3/run.env
ADDED
|
@@ -0,0 +1,70 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
backend=echo
|
| 2 |
+
config=G1
|
| 3 |
+
family=tierA
|
| 4 |
+
repo=/workspace/Megatron-MoE-Echo
|
| 5 |
+
branch=bracing-regions-through-kernels
|
| 6 |
+
commit=9a236793
|
| 7 |
+
num_layers=16
|
| 8 |
+
hidden=4096
|
| 9 |
+
ffn_hidden=14336
|
| 10 |
+
moe_ffn_hidden=14336
|
| 11 |
+
num_experts=8
|
| 12 |
+
topk=2
|
| 13 |
+
experts_per_rank=1
|
| 14 |
+
expert_params=176160768
|
| 15 |
+
heads=32
|
| 16 |
+
kv_groups=8
|
| 17 |
+
kv_channels=
|
| 18 |
+
qk_layernorm=0
|
| 19 |
+
shared_ffn=0
|
| 20 |
+
shared_expert_overlap=0
|
| 21 |
+
moe_token_dispatcher=flex
|
| 22 |
+
mla=0
|
| 23 |
+
dense_first_layers=0
|
| 24 |
+
moe_layers=16
|
| 25 |
+
rotary_base=1000000
|
| 26 |
+
rotary_scaling=
|
| 27 |
+
qkv_bias=0
|
| 28 |
+
router_balancing=aux_loss
|
| 29 |
+
aux_loss_coeff=1e-2
|
| 30 |
+
router_pre_softmax=0
|
| 31 |
+
router_fusion=0
|
| 32 |
+
router_dtype=fp32
|
| 33 |
+
device_initiated_grouped_gemm=1
|
| 34 |
+
norm_epsilon=1e-5
|
| 35 |
+
seq_length=4096
|
| 36 |
+
micro_batch_size=1
|
| 37 |
+
global_batch_size=16
|
| 38 |
+
num_microbatches=2
|
| 39 |
+
assignments_per_rank=8192
|
| 40 |
+
mean_tokens_per_expert=8192
|
| 41 |
+
slot_policy=matched
|
| 42 |
+
slots_per_rank=1
|
| 43 |
+
capacity_mode=static
|
| 44 |
+
capacity=2.40625
|
| 45 |
+
cuda_graph=1
|
| 46 |
+
probe_mode=0
|
| 47 |
+
async_grad_reduce=0
|
| 48 |
+
branch_role=timing
|
| 49 |
+
memory_dump_mode=none
|
| 50 |
+
memory_snapshot_path=
|
| 51 |
+
cuda_graph_warmup_steps=3
|
| 52 |
+
recompute=none
|
| 53 |
+
force_balance=0
|
| 54 |
+
train_iters=12
|
| 55 |
+
ep_size=8
|
| 56 |
+
echo_dispatch_overlap=1
|
| 57 |
+
echo_dispatch_recompute=0
|
| 58 |
+
ultraep_quota_min_tokens=
|
| 59 |
+
ultraep_weight_sync_join=
|
| 60 |
+
profile_enable=0
|
| 61 |
+
kernel_trace=0
|
| 62 |
+
fp8_mode=mxfp8
|
| 63 |
+
fp8_recipe=mxfp8
|
| 64 |
+
training_seed=1234
|
| 65 |
+
learning_rate=1e-4
|
| 66 |
+
min_learning_rate=1.0e-5
|
| 67 |
+
lr_warmup_iters=5
|
| 68 |
+
state_dump_enable=0
|
| 69 |
+
state_dump_dir=
|
| 70 |
+
moonep_assert_no_pending_reduce=0
|
tiera-rerun-fixed-moonep/echo/a2/a2_matched_gbs16/G1__echo__matched__seq4096__gbs16__cap2p40625/rep3/train.log
ADDED
|
@@ -0,0 +1,1023 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
==============================================================
|
| 2 |
+
G1 / echo (slots/rank=1 policy=matched)
|
| 3 |
+
E=8 K=2 H'=14336 m=1
|
| 4 |
+
L=16 seq=4096 gbs=16 (2 microbatches)
|
| 5 |
+
cap_mode=static cap=2.40625 cg=1 recompute=none
|
| 6 |
+
memory_dump=none snapshots=/workspace/tiera-rerun-fixed-moonep/echo/a2/a2_matched_gbs16/G1__echo__matched__seq4096__gbs16__cap2p40625/rep3/memory_snapshots/snapshot.pickle
|
| 7 |
+
repo=/workspace/Megatron-MoE-Echo @ 9a236793
|
| 8 |
+
==============================================================
|
| 9 |
+
W0830 06:47:05.447000 658758 torch/distributed/run.py:766]
|
| 10 |
+
W0830 06:47:05.447000 658758 torch/distributed/run.py:766] *****************************************
|
| 11 |
+
W0830 06:47:05.447000 658758 torch/distributed/run.py:766] Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed.
|
| 12 |
+
W0830 06:47:05.447000 658758 torch/distributed/run.py:766] *****************************************
|
| 13 |
+
/workspace/Megatron-MoE-Echo/megatron/training/arguments.py:831: UserWarning: Disabling sequence parallelism because tensor model parallelism is disabled
|
| 14 |
+
warnings.warn("Disabling sequence parallelism because tensor model parallelism is disabled")
|
| 15 |
+
using world size: 8, data-parallel size: 8, context-parallel size: 1, hierarchical context-parallel sizes: None, tensor-model-parallel size: 1, pipeline-model-parallel size: 1
|
| 16 |
+
WARNING: overriding default arguments for tokenizer_type:GPT2BPETokenizer with tokenizer_type:GPT2BPETokenizer
|
| 17 |
+
Number of virtual stages per pipeline stage: None
|
| 18 |
+
accumulate and all-reduce gradients in fp32 for bfloat16 data type.
|
| 19 |
+
using torch.bfloat16 for parameters ...
|
| 20 |
+
/workspace/Megatron-MoE-Echo/megatron/training/arguments.py:831: UserWarning: Disabling sequence parallelism because tensor model parallelism is disabled
|
| 21 |
+
warnings.warn("Disabling sequence parallelism because tensor model parallelism is disabled")
|
| 22 |
+
------------------------ arguments ------------------------
|
| 23 |
+
account_for_embedding_in_pipeline_split ......... False
|
| 24 |
+
account_for_loss_in_pipeline_split .............. False
|
| 25 |
+
accumulate_allreduce_grads_in_fp32 .............. True
|
| 26 |
+
adam_beta1 ...................................... 0.9
|
| 27 |
+
adam_beta2 ...................................... 0.999
|
| 28 |
+
adam_eps ........................................ 1e-08
|
| 29 |
+
add_bias_linear ................................. False
|
| 30 |
+
add_position_embedding .......................... False
|
| 31 |
+
add_qkv_bias .................................... False
|
| 32 |
+
adlr_autoresume ................................. False
|
| 33 |
+
adlr_autoresume_interval ........................ 1000
|
| 34 |
+
align_grad_reduce ............................... True
|
| 35 |
+
align_param_gather .............................. False
|
| 36 |
+
app_tag_run_name ................................ None
|
| 37 |
+
app_tag_run_version ............................. 0.0.0
|
| 38 |
+
apply_layernorm_1p .............................. False
|
| 39 |
+
apply_query_key_layer_scaling ................... False
|
| 40 |
+
apply_residual_connection_post_layernorm ........ False
|
| 41 |
+
apply_rope_fusion ............................... True
|
| 42 |
+
async_save ...................................... None
|
| 43 |
+
async_tensor_model_parallel_allreduce ........... True
|
| 44 |
+
attention_backend ............................... AttnBackend.auto
|
| 45 |
+
attention_dropout ............................... 0.0
|
| 46 |
+
attention_softmax_in_fp32 ....................... False
|
| 47 |
+
auto_detect_ckpt_format ......................... False
|
| 48 |
+
barrier_with_L1_time ............................ True
|
| 49 |
+
bert_binary_head ................................ True
|
| 50 |
+
bert_embedder_type .............................. megatron
|
| 51 |
+
bert_load ....................................... None
|
| 52 |
+
bf16 ............................................ True
|
| 53 |
+
bias_dropout_fusion ............................. True
|
| 54 |
+
bias_gelu_fusion ................................ False
|
| 55 |
+
bias_swiglu_fusion .............................. True
|
| 56 |
+
biencoder_projection_dim ........................ 0
|
| 57 |
+
biencoder_shared_query_context_model ............ False
|
| 58 |
+
block_data_path ................................. None
|
| 59 |
+
calc_ft_timeouts ................................ False
|
| 60 |
+
calculate_per_token_loss ........................ False
|
| 61 |
+
check_for_large_grads ........................... False
|
| 62 |
+
check_for_nan_in_loss_and_grad .................. False
|
| 63 |
+
check_for_spiky_loss ............................ False
|
| 64 |
+
check_weight_hash_across_dp_replicas_interval ... None
|
| 65 |
+
ckpt_assume_constant_structure .................. False
|
| 66 |
+
ckpt_convert_format ............................. None
|
| 67 |
+
ckpt_convert_save ............................... None
|
| 68 |
+
ckpt_convert_update_legacy_dist_opt_format ...... False
|
| 69 |
+
ckpt_format ..................................... torch_dist
|
| 70 |
+
ckpt_fully_parallel_load ........................ False
|
| 71 |
+
ckpt_fully_parallel_save ........................ True
|
| 72 |
+
ckpt_fully_parallel_save_deprecated ............. False
|
| 73 |
+
ckpt_step ....................................... None
|
| 74 |
+
classes_fraction ................................ 1.0
|
| 75 |
+
clip_grad ....................................... 1.0
|
| 76 |
+
clone_scatter_output_in_embedding ............... True
|
| 77 |
+
config_logger_dir ...............................
|
| 78 |
+
consumed_train_samples .......................... 0
|
| 79 |
+
consumed_valid_samples .......................... 0
|
| 80 |
+
context_parallel_size ........................... 1
|
| 81 |
+
cp_comm_type .................................... ['p2p']
|
| 82 |
+
create_attention_mask_in_dataloader ............. True
|
| 83 |
+
cross_entropy_fusion_impl ....................... native
|
| 84 |
+
cross_entropy_loss_fusion ....................... False
|
| 85 |
+
cuda_graph_scope ................................ full_iteration
|
| 86 |
+
cuda_graph_warmup_steps ......................... 3
|
| 87 |
+
data_args_path .................................. None
|
| 88 |
+
data_cache_path ................................. None
|
| 89 |
+
data_parallel_random_init ....................... False
|
| 90 |
+
data_parallel_sharding_strategy ................. no_shard
|
| 91 |
+
data_parallel_size .............................. 8
|
| 92 |
+
data_path ....................................... ['/workspace/data/wikitext/wikitext103_gpt2_text_document']
|
| 93 |
+
data_per_class_fraction ......................... 1.0
|
| 94 |
+
data_sharding ................................... True
|
| 95 |
+
dataloader_type ................................. single
|
| 96 |
+
ddp_average_in_collective ....................... False
|
| 97 |
+
ddp_bucket_size ................................. None
|
| 98 |
+
ddp_num_buckets ................................. None
|
| 99 |
+
ddp_pad_buckets_for_high_nccl_busbw ............. False
|
| 100 |
+
decoder_first_pipeline_num_layers ............... None
|
| 101 |
+
decoder_last_pipeline_num_layers ................ None
|
| 102 |
+
decoder_num_layers .............................. None
|
| 103 |
+
decoder_seq_length .............................. None
|
| 104 |
+
decoupled_lr .................................... None
|
| 105 |
+
decoupled_min_lr ................................ None
|
| 106 |
+
decrease_batch_size_if_needed ................... False
|
| 107 |
+
defer_embedding_wgrad_compute ................... False
|
| 108 |
+
delay_wgrad_compute ............................. False
|
| 109 |
+
deprecated_use_mcore_models ..................... True
|
| 110 |
+
deterministic_mode .............................. False
|
| 111 |
+
dino_bottleneck_size ............................ 256
|
| 112 |
+
dino_freeze_last_layer .......................... 1
|
| 113 |
+
dino_head_hidden_size ........................... 2048
|
| 114 |
+
dino_local_crops_number ......................... 10
|
| 115 |
+
dino_local_img_size ............................. 96
|
| 116 |
+
dino_norm_last_layer ............................ False
|
| 117 |
+
dino_teacher_temp ............................... 0.07
|
| 118 |
+
dino_warmup_teacher_temp ........................ 0.04
|
| 119 |
+
dino_warmup_teacher_temp_epochs ................. 30
|
| 120 |
+
disable_bf16_reduced_precision_matmul ........... False
|
| 121 |
+
disable_mamba_mem_eff_path ...................... False
|
| 122 |
+
disable_straggler_on_startup .................... False
|
| 123 |
+
dist_ckpt_format_deprecated ..................... None
|
| 124 |
+
dist_ckpt_strictness ............................ assume_ok_unexpected
|
| 125 |
+
distribute_saved_activations .................... False
|
| 126 |
+
distributed_backend ............................. nccl
|
| 127 |
+
distributed_timeout_minutes ..................... 10
|
| 128 |
+
embedding_init_method_std ....................... None
|
| 129 |
+
embedding_path .................................. None
|
| 130 |
+
empty_unused_memory_level ....................... 0
|
| 131 |
+
enable_cuda_graph ............................... True
|
| 132 |
+
enable_experimental ............................. False
|
| 133 |
+
enable_ft_package ............................... False
|
| 134 |
+
enable_gloo_process_groups ...................... True
|
| 135 |
+
enable_msc ...................................... True
|
| 136 |
+
enable_one_logger ............................... True
|
| 137 |
+
encoder_num_layers .............................. 16
|
| 138 |
+
encoder_seq_length .............................. 4096
|
| 139 |
+
end_weight_decay ................................ 0.1
|
| 140 |
+
eod_mask_loss ................................... False
|
| 141 |
+
error_injection_rate ............................ 0
|
| 142 |
+
error_injection_type ............................ transient_error
|
| 143 |
+
eval_interval ................................... 1000000
|
| 144 |
+
eval_iters ...................................... 0
|
| 145 |
+
evidence_data_path .............................. None
|
| 146 |
+
exit_duration_in_mins ........................... None
|
| 147 |
+
exit_interval ................................... None
|
| 148 |
+
exit_on_missing_checkpoint ...................... False
|
| 149 |
+
exit_signal_handler ............................. False
|
| 150 |
+
exp_avg_dtype ................................... torch.float32
|
| 151 |
+
exp_avg_sq_dtype ................................ torch.float32
|
| 152 |
+
expert_model_parallel_size ...................... 8
|
| 153 |
+
expert_tensor_parallel_size ..................... 1
|
| 154 |
+
export_force_local_attention .................... False
|
| 155 |
+
export_kd_cfg ................................... None
|
| 156 |
+
export_kd_teacher_ckpt_format ................... None
|
| 157 |
+
export_kd_teacher_load .......................... None
|
| 158 |
+
export_kv_cache_quant ........................... False
|
| 159 |
+
export_legacy_megatron .......................... False
|
| 160 |
+
export_model_type ............................... GPTModel
|
| 161 |
+
export_moe_apply_probs_on_input ................. False
|
| 162 |
+
export_qk_l2_norm ............................... False
|
| 163 |
+
export_quant_cfg ................................ None
|
| 164 |
+
export_real_quant_cfg ........................... None
|
| 165 |
+
export_te_mcore_model ........................... False
|
| 166 |
+
external_cuda_graph ............................. False
|
| 167 |
+
ffn_hidden_size ................................. 14336
|
| 168 |
+
finetune ........................................ False
|
| 169 |
+
finetune_data_split ............................. train
|
| 170 |
+
finetune_hf_dataset ............................. None
|
| 171 |
+
first_last_layers_bf16 .......................... False
|
| 172 |
+
flash_decode .................................... False
|
| 173 |
+
fp16 ............................................ False
|
| 174 |
+
fp16_lm_cross_entropy ........................... False
|
| 175 |
+
fp32_residual_connection ........................ False
|
| 176 |
+
fp8 ............................................. e4m3
|
| 177 |
+
fp8_amax_compute_algo ........................... most_recent
|
| 178 |
+
fp8_amax_history_len ............................ 1
|
| 179 |
+
fp8_interval .................................... 1
|
| 180 |
+
fp8_margin ...................................... 0
|
| 181 |
+
fp8_param_gather ................................ False
|
| 182 |
+
fp8_recipe ...................................... mxfp8
|
| 183 |
+
fp8_wgrad ....................................... True
|
| 184 |
+
fsdp_double_buffer .............................. False
|
| 185 |
+
global_batch_size ............................... 16
|
| 186 |
+
grad_reduce_in_bf16 ............................. False
|
| 187 |
+
gradient_accumulation_fusion .................... True
|
| 188 |
+
gradient_reduce_div_fusion ...................... True
|
| 189 |
+
group_query_attention ........................... True
|
| 190 |
+
head_lr_mult .................................... 1.0
|
| 191 |
+
heterogeneous_layers_config_encoded_json ........ None
|
| 192 |
+
heterogeneous_layers_config_path ................ None
|
| 193 |
+
hidden_dropout .................................. 0.0
|
| 194 |
+
hidden_size ..................................... 4096
|
| 195 |
+
hierarchical_context_parallel_sizes ............. None
|
| 196 |
+
high_priority_stream_groups ..................... []
|
| 197 |
+
hybrid_attention_ratio .......................... 0.0
|
| 198 |
+
hybrid_mlp_ratio ................................ 0.0
|
| 199 |
+
hybrid_override_pattern ......................... None
|
| 200 |
+
hysteresis ...................................... 2
|
| 201 |
+
ict_head_size ................................... None
|
| 202 |
+
ict_load ........................................ None
|
| 203 |
+
img_h ........................................... 224
|
| 204 |
+
img_w ........................................... 224
|
| 205 |
+
indexer_batch_size .............................. 128
|
| 206 |
+
indexer_log_interval ............................ 1000
|
| 207 |
+
inference_batch_times_seqlen_threshold .......... -1
|
| 208 |
+
inference_dynamic_batching ...................... False
|
| 209 |
+
inference_dynamic_batching_buffer_guaranteed_fraction 0.2
|
| 210 |
+
inference_dynamic_batching_buffer_overflow_factor None
|
| 211 |
+
inference_dynamic_batching_buffer_size_gb ....... 40.0
|
| 212 |
+
inference_dynamic_batching_chunk_size ........... 256
|
| 213 |
+
inference_dynamic_batching_max_requests_override None
|
| 214 |
+
inference_dynamic_batching_max_tokens_override .. None
|
| 215 |
+
inference_dynamic_batching_num_cuda_graphs ...... 16
|
| 216 |
+
inference_max_batch_size ........................ 8
|
| 217 |
+
inference_max_seq_length ........................ 2560
|
| 218 |
+
inference_rng_tracker ........................... False
|
| 219 |
+
init_method_std ................................. 0.01
|
| 220 |
+
init_method_xavier_uniform ...................... False
|
| 221 |
+
init_model_with_meta_device ..................... False
|
| 222 |
+
initial_loss_scale .............................. 4294967296
|
| 223 |
+
inprocess_active_world_size ..................... 8
|
| 224 |
+
inprocess_barrier_timeout ....................... 120
|
| 225 |
+
inprocess_completion_timeout .................... 120
|
| 226 |
+
inprocess_empty_cuda_cache ...................... False
|
| 227 |
+
inprocess_granularity ........................... node
|
| 228 |
+
inprocess_hard_timeout .......................... 90
|
| 229 |
+
inprocess_heartbeat_interval .................... 30
|
| 230 |
+
inprocess_heartbeat_timeout ..................... 60
|
| 231 |
+
inprocess_last_call_wait ........................ 1
|
| 232 |
+
inprocess_max_iterations ........................ None
|
| 233 |
+
inprocess_monitor_process_interval .............. 1.0
|
| 234 |
+
inprocess_monitor_thread_interval ............... 1.0
|
| 235 |
+
inprocess_progress_watchdog_interval ............ 1.0
|
| 236 |
+
inprocess_restart ............................... False
|
| 237 |
+
inprocess_soft_timeout .......................... 60
|
| 238 |
+
inprocess_termination_grace_time ................ 1
|
| 239 |
+
is_hybrid_model ................................. False
|
| 240 |
+
iter_per_epoch .................................. 1250
|
| 241 |
+
iterations_to_skip .............................. []
|
| 242 |
+
keep_fp8_transpose_cache_when_using_custom_fsdp . False
|
| 243 |
+
kitchen_config_file ............................. None
|
| 244 |
+
kitchen_recipe_number ........................... None
|
| 245 |
+
kv_channels ..................................... 128
|
| 246 |
+
kv_lora_rank .................................... 32
|
| 247 |
+
lazy_mpu_init ................................... None
|
| 248 |
+
load ............................................ None
|
| 249 |
+
load_main_params_from_ckpt ...................... None
|
| 250 |
+
load_model_opt_format ........................... False
|
| 251 |
+
local_rank ...................................... 0
|
| 252 |
+
log_energy ...................................... False
|
| 253 |
+
log_interval .................................... 1
|
| 254 |
+
log_loss_scale_to_tensorboard ................... True
|
| 255 |
+
log_memory_to_tensorboard ....................... False
|
| 256 |
+
log_num_zeros_in_grad ........................... False
|
| 257 |
+
log_params_norm ................................. False
|
| 258 |
+
log_progress .................................... False
|
| 259 |
+
log_straggler ................................... False
|
| 260 |
+
log_throughput .................................. True
|
| 261 |
+
log_timers_to_tensorboard ....................... False
|
| 262 |
+
log_validation_ppl_to_tensorboard ............... False
|
| 263 |
+
log_world_size_to_tensorboard ................... False
|
| 264 |
+
logging_level ................................... None
|
| 265 |
+
loss_scale ...................................... None
|
| 266 |
+
loss_scale_window ............................... 1000
|
| 267 |
+
lr .............................................. 0.0001
|
| 268 |
+
lr_decay_iters .................................. 8
|
| 269 |
+
lr_decay_samples ................................ None
|
| 270 |
+
lr_decay_style .................................. cosine
|
| 271 |
+
lr_warmup_fraction .............................. None
|
| 272 |
+
lr_warmup_init .................................. 0.0
|
| 273 |
+
lr_warmup_iters ................................. 5
|
| 274 |
+
lr_warmup_samples ............................... 0
|
| 275 |
+
lr_wsd_decay_iters .............................. None
|
| 276 |
+
lr_wsd_decay_samples ............................ None
|
| 277 |
+
lr_wsd_decay_style .............................. exponential
|
| 278 |
+
main_grads_dtype ................................ torch.float32
|
| 279 |
+
main_params_dtype ............................... torch.float32
|
| 280 |
+
make_vocab_size_divisible_by .................... 128
|
| 281 |
+
mamba_head_dim .................................. 64
|
| 282 |
+
mamba_num_groups ................................ 8
|
| 283 |
+
mamba_num_heads ................................. None
|
| 284 |
+
mamba_state_dim ................................. 128
|
| 285 |
+
manual_gc ....................................... False
|
| 286 |
+
manual_gc_eval .................................. True
|
| 287 |
+
manual_gc_interval .............................. 0
|
| 288 |
+
mask_factor ..................................... 1.0
|
| 289 |
+
mask_prob ....................................... 0.15
|
| 290 |
+
mask_type ....................................... random
|
| 291 |
+
masked_softmax_fusion ........................... False
|
| 292 |
+
max_position_embeddings ......................... 32768
|
| 293 |
+
max_tokens_to_oom ............................... 12000
|
| 294 |
+
memory_snapshot_path ............................ snapshot.pickle
|
| 295 |
+
merge_file ...................................... /workspace/data/wikitext/gpt2-merges.txt
|
| 296 |
+
micro_batch_size ................................ 1
|
| 297 |
+
microbatch_group_size_per_vp_stage .............. None
|
| 298 |
+
mid_level_dataset_surplus ....................... 0.005
|
| 299 |
+
min_loss_scale .................................. 1.0
|
| 300 |
+
min_lr .......................................... 1e-05
|
| 301 |
+
mlp_chunks_for_prefill .......................... 1
|
| 302 |
+
mmap_bin_files .................................. True
|
| 303 |
+
mock_data ....................................... False
|
| 304 |
+
moe_apply_probs_on_input ........................ False
|
| 305 |
+
moe_aux_loss_coeff .............................. 0.01
|
| 306 |
+
moe_deepep_num_sms .............................. 20
|
| 307 |
+
moe_echo_enable_random_offloading ............... False
|
| 308 |
+
moe_echo_expert_dispatch_overlap ................ True
|
| 309 |
+
moe_echo_expert_dispatcher_type ................. hybridep
|
| 310 |
+
moe_echo_recompute_expert_dispatch .............. False
|
| 311 |
+
moe_enable_deepep ............................... False
|
| 312 |
+
moe_enable_echo ................................. True
|
| 313 |
+
moe_enable_hybridep ............................. True
|
| 314 |
+
moe_expert_capacity_factor ...................... None
|
| 315 |
+
moe_extended_tp ................................. False
|
| 316 |
+
moe_ffn_hidden_size ............................. 14336
|
| 317 |
+
moe_grouped_gemm ................................ True
|
| 318 |
+
moe_hybridep_num_sms ............................ 16
|
| 319 |
+
moe_input_jitter_eps ............................ None
|
| 320 |
+
moe_layer_freq .................................. 1
|
| 321 |
+
moe_layer_recompute ............................. False
|
| 322 |
+
moe_num_echo_experts ............................ 8
|
| 323 |
+
moe_pad_expert_input_to_capacity ................ False
|
| 324 |
+
moe_per_layer_logging ........................... False
|
| 325 |
+
moe_permute_fusion .............................. False
|
| 326 |
+
moe_received_token_capacity ..................... 2.40625
|
| 327 |
+
moe_router_bias_update_rate ..................... 0.001
|
| 328 |
+
moe_router_dtype ................................ fp32
|
| 329 |
+
moe_router_enable_expert_bias ................... False
|
| 330 |
+
moe_router_force_load_balancing ................. False
|
| 331 |
+
moe_router_group_topk ........................... None
|
| 332 |
+
moe_router_load_balancing_type .................. aux_loss
|
| 333 |
+
moe_router_num_groups ........................... None
|
| 334 |
+
moe_router_padding_for_fp8 ...................... False
|
| 335 |
+
moe_router_pre_softmax .......................... False
|
| 336 |
+
moe_router_score_function ....................... softmax
|
| 337 |
+
moe_router_topk ................................. 2
|
| 338 |
+
moe_router_topk_scaling_factor .................. None
|
| 339 |
+
moe_shared_expert_intermediate_size ............. None
|
| 340 |
+
moe_shared_expert_overlap ....................... False
|
| 341 |
+
moe_token_dispatcher_type ....................... flex
|
| 342 |
+
moe_token_drop_policy ........................... probs
|
| 343 |
+
moe_upcycling_granularity ....................... 1
|
| 344 |
+
moe_use_device_initiated_grouped_gemm ........... True
|
| 345 |
+
moe_use_legacy_grouped_gemm ..................... False
|
| 346 |
+
moe_use_upcycling ............................... False
|
| 347 |
+
moe_z_loss_coeff ................................ None
|
| 348 |
+
mrope_section ................................... None
|
| 349 |
+
mscale .......................................... 1.0
|
| 350 |
+
mscale_all_dim .................................. 0.0
|
| 351 |
+
mtp_loss_scaling_factor ......................... 0.1
|
| 352 |
+
mtp_num_layers .................................. None
|
| 353 |
+
multi_latent_attention .......................... False
|
| 354 |
+
nccl_all_reduce_for_prefill ..................... False
|
| 355 |
+
nccl_communicator_config_path ................... None
|
| 356 |
+
nccl_ub ......................................... False
|
| 357 |
+
no_load_optim ................................... None
|
| 358 |
+
no_load_rng ..................................... None
|
| 359 |
+
no_persist_layer_norm ........................... False
|
| 360 |
+
no_rope_freq .................................... None
|
| 361 |
+
no_save_optim ................................... None
|
| 362 |
+
no_save_rng ..................................... None
|
| 363 |
+
non_persistent_ckpt_type ........................ None
|
| 364 |
+
non_persistent_global_ckpt_dir .................. None
|
| 365 |
+
non_persistent_local_ckpt_algo .................. fully_parallel
|
| 366 |
+
non_persistent_local_ckpt_dir ................... None
|
| 367 |
+
non_persistent_save_interval .................... None
|
| 368 |
+
norm_epsilon .................................... 1e-05
|
| 369 |
+
normalization ................................... RMSNorm
|
| 370 |
+
num_attention_heads ............................. 32
|
| 371 |
+
num_channels .................................... 3
|
| 372 |
+
num_classes ..................................... 1000
|
| 373 |
+
num_dataset_builder_threads ..................... 1
|
| 374 |
+
num_distributed_optimizer_instances ............. 1
|
| 375 |
+
num_experts ..................................... 8
|
| 376 |
+
num_layers ...................................... 16
|
| 377 |
+
num_layers_at_end_in_bf16 ....................... 1
|
| 378 |
+
num_layers_at_start_in_bf16 ..................... 1
|
| 379 |
+
num_layers_per_virtual_pipeline_stage ........... None
|
| 380 |
+
num_query_groups ................................ 8
|
| 381 |
+
num_virtual_stages_per_pipeline_rank ............ None
|
| 382 |
+
num_workers ..................................... 2
|
| 383 |
+
object_storage_cache_path ....................... None
|
| 384 |
+
one_logger_async ................................ False
|
| 385 |
+
one_logger_project .............................. megatron-lm
|
| 386 |
+
one_logger_run_name ............................. None
|
| 387 |
+
onnx_safe ....................................... None
|
| 388 |
+
openai_gelu ..................................... False
|
| 389 |
+
optimizer ....................................... adam
|
| 390 |
+
optimizer_cpu_offload ........................... False
|
| 391 |
+
optimizer_offload_fraction ...................... 1.0
|
| 392 |
+
output_bert_embeddings .......................... False
|
| 393 |
+
overlap_cpu_optimizer_d2h_h2d ................... False
|
| 394 |
+
overlap_grad_reduce ............................. True
|
| 395 |
+
overlap_moe_expert_parallel_comm ................ False
|
| 396 |
+
overlap_p2p_comm ................................ False
|
| 397 |
+
overlap_p2p_comm_warmup_flush ................... False
|
| 398 |
+
overlap_param_gather ............................ True
|
| 399 |
+
overlap_param_gather_with_optimizer_step ........ False
|
| 400 |
+
override_opt_param_scheduler .................... False
|
| 401 |
+
params_dtype .................................... torch.bfloat16
|
| 402 |
+
patch_dim ....................................... 16
|
| 403 |
+
per_split_data_args_path ........................ None
|
| 404 |
+
perform_initialization .......................... True
|
| 405 |
+
pin_cpu_grads ................................... True
|
| 406 |
+
pin_cpu_params .................................. True
|
| 407 |
+
pipeline_model_parallel_comm_backend ............ None
|
| 408 |
+
pipeline_model_parallel_layout .................. None
|
| 409 |
+
pipeline_model_parallel_size .................... 1
|
| 410 |
+
position_embedding_type ......................... rope
|
| 411 |
+
pretrained_checkpoint ........................... None
|
| 412 |
+
profile ......................................... False
|
| 413 |
+
profile_ranks ................................... [0]
|
| 414 |
+
profile_step_end ................................ 12
|
| 415 |
+
profile_step_start .............................. 10
|
| 416 |
+
q_lora_rank ..................................... None
|
| 417 |
+
qk_head_dim ..................................... 128
|
| 418 |
+
qk_l2_norm ...................................... False
|
| 419 |
+
qk_layernorm .................................... False
|
| 420 |
+
qk_pos_emb_head_dim ............................. 64
|
| 421 |
+
query_in_block_prob ............................. 0.1
|
| 422 |
+
rampup_batch_size ............................... None
|
| 423 |
+
rank ............................................ 0
|
| 424 |
+
recompute_granularity ........................... None
|
| 425 |
+
recompute_method ................................ None
|
| 426 |
+
recompute_modules ............................... None
|
| 427 |
+
recompute_num_layers ............................ None
|
| 428 |
+
record_memory_history ........................... False
|
| 429 |
+
relative_attention_max_distance ................. 128
|
| 430 |
+
relative_attention_num_buckets .................. 32
|
| 431 |
+
replication ..................................... False
|
| 432 |
+
replication_factor .............................. 2
|
| 433 |
+
replication_jump ................................ None
|
| 434 |
+
rerun_mode ...................................... validate_results
|
| 435 |
+
reset_attention_mask ............................ False
|
| 436 |
+
reset_position_ids .............................. False
|
| 437 |
+
result_rejected_tracker_filename ................ None
|
| 438 |
+
retriever_report_topk_accuracies ................ []
|
| 439 |
+
retriever_score_scaling ......................... False
|
| 440 |
+
retriever_seq_length ............................ 256
|
| 441 |
+
retro_add_retriever ............................. False
|
| 442 |
+
retro_attention_gate ............................ 1
|
| 443 |
+
retro_cyclic_train_iters ........................ None
|
| 444 |
+
retro_encoder_attention_dropout ................. 0.1
|
| 445 |
+
retro_encoder_hidden_dropout .................... 0.1
|
| 446 |
+
retro_encoder_layers ............................ 2
|
| 447 |
+
retro_num_neighbors ............................. 2
|
| 448 |
+
retro_num_retrieved_chunks ...................... 2
|
| 449 |
+
retro_project_dir ............................... None
|
| 450 |
+
retro_verify_neighbor_count ..................... True
|
| 451 |
+
reuse_grad_buf_for_mxfp8_param_ag ............... False
|
| 452 |
+
rope_scaling_factor ............................. 8.0
|
| 453 |
+
rope_type ....................................... None
|
| 454 |
+
rotary_base ..................................... 1000000
|
| 455 |
+
rotary_interleaved .............................. False
|
| 456 |
+
rotary_percent .................................. 1.0
|
| 457 |
+
rotary_scaling_factor ........................... 1.0
|
| 458 |
+
rotary_seq_len_interpolation_factor ............. None
|
| 459 |
+
run_workload_inspector_server ................... False
|
| 460 |
+
sample_rate ..................................... 1.0
|
| 461 |
+
save ............................................ None
|
| 462 |
+
save_interval ................................... None
|
| 463 |
+
save_retain_interval ............................ None
|
| 464 |
+
scatter_gather_tensors_in_pipeline .............. True
|
| 465 |
+
seed ............................................ 1234
|
| 466 |
+
seq_length ...................................... 4096
|
| 467 |
+
sequence_parallel ............................... False
|
| 468 |
+
sft ............................................. False
|
| 469 |
+
sft_tokenizer_prompt_format ..................... nemotron-h-aligned
|
| 470 |
+
sgd_momentum .................................... 0.9
|
| 471 |
+
short_seq_prob .................................. 0.1
|
| 472 |
+
skip_train ...................................... False
|
| 473 |
+
skipped_train_samples ........................... 0
|
| 474 |
+
spec ............................................ None
|
| 475 |
+
split ........................................... 100,0,0
|
| 476 |
+
squared_relu .................................... False
|
| 477 |
+
start_weight_decay .............................. 0.1
|
| 478 |
+
straggler_ctrlr_port ............................ 65535
|
| 479 |
+
straggler_minmax_count .......................... 1
|
| 480 |
+
suggested_communication_unit_size ............... None
|
| 481 |
+
swiglu .......................................... True
|
| 482 |
+
swin_backbone_type .............................. tiny
|
| 483 |
+
symmetric_ar_type ............................... None
|
| 484 |
+
te_rng_tracker .................................. True
|
| 485 |
+
tensor_model_parallel_size ...................... 1
|
| 486 |
+
tensorboard_dir ................................. None
|
| 487 |
+
tensorboard_log_interval ........................ 1
|
| 488 |
+
tensorboard_queue_size .......................... 1000
|
| 489 |
+
test_data_path .................................. None
|
| 490 |
+
test_mode ....................................... False
|
| 491 |
+
tiktoken_num_special_tokens ..................... 1000
|
| 492 |
+
tiktoken_pattern ................................ None
|
| 493 |
+
tiktoken_special_tokens ......................... None
|
| 494 |
+
timing_log_level ................................ 0
|
| 495 |
+
timing_log_option ............................... minmax
|
| 496 |
+
titles_data_path ................................ None
|
| 497 |
+
tokenizer_model ................................. None
|
| 498 |
+
tokenizer_type .................................. GPT2BPETokenizer
|
| 499 |
+
torch_fsdp2_reshard_after_forward ............... True
|
| 500 |
+
tp_comm_bootstrap_backend ....................... nccl
|
| 501 |
+
tp_comm_bulk_dgrad .............................. True
|
| 502 |
+
tp_comm_bulk_wgrad .............................. True
|
| 503 |
+
tp_comm_overlap ................................. False
|
| 504 |
+
tp_comm_overlap_ag .............................. True
|
| 505 |
+
tp_comm_overlap_cfg ............................. None
|
| 506 |
+
tp_comm_overlap_rs .............................. True
|
| 507 |
+
tp_comm_overlap_rs_dgrad ........................ False
|
| 508 |
+
tp_comm_split_ag ................................ True
|
| 509 |
+
tp_comm_split_rs ................................ True
|
| 510 |
+
train_data_path ................................. None
|
| 511 |
+
train_iters ..................................... 12
|
| 512 |
+
train_samples ................................... None
|
| 513 |
+
train_sync_interval ............................. None
|
| 514 |
+
transformer_impl ................................ transformer_engine
|
| 515 |
+
transformer_pipeline_model_parallel_size ........ 1
|
| 516 |
+
untie_embeddings_and_output_weights ............. True
|
| 517 |
+
use_checkpoint_args ............................. False
|
| 518 |
+
use_checkpoint_opt_param_scheduler .............. False
|
| 519 |
+
use_cpu_initialization .......................... None
|
| 520 |
+
use_custom_fsdp ................................. False
|
| 521 |
+
use_dist_ckpt ................................... True
|
| 522 |
+
use_dist_ckpt_deprecated ........................ False
|
| 523 |
+
use_distributed_optimizer ....................... True
|
| 524 |
+
use_flash_attn .................................. False
|
| 525 |
+
use_legacy_models ............................... False
|
| 526 |
+
use_mp_args_from_checkpoint_args ................ False
|
| 527 |
+
use_one_sent_docs ............................... False
|
| 528 |
+
use_persistent_ckpt_worker ...................... False
|
| 529 |
+
use_precision_aware_optimizer ................... False
|
| 530 |
+
use_pytorch_profiler ............................ False
|
| 531 |
+
use_ring_exchange_p2p ........................... False
|
| 532 |
+
use_rope_scaling ................................ False
|
| 533 |
+
use_rotary_position_embeddings .................. False
|
| 534 |
+
use_sharp ....................................... False
|
| 535 |
+
use_tokenizer_model_from_checkpoint_args ........ True
|
| 536 |
+
use_torch_fsdp2 ................................. False
|
| 537 |
+
use_torch_optimizer_for_cpu_offload ............. False
|
| 538 |
+
use_tp_pp_dp_mapping ............................ False
|
| 539 |
+
v_head_dim ...................................... 128
|
| 540 |
+
valid_data_path ................................. None
|
| 541 |
+
variable_seq_lengths ............................ False
|
| 542 |
+
virtual_pipeline_model_parallel_size ............ None
|
| 543 |
+
vision_backbone_type ............................ vit
|
| 544 |
+
vision_pretraining .............................. False
|
| 545 |
+
vision_pretraining_type ......................... classify
|
| 546 |
+
vocab_extra_ids ................................. 0
|
| 547 |
+
vocab_file ...................................... /workspace/data/wikitext/gpt2-vocab.json
|
| 548 |
+
vocab_size ...................................... None
|
| 549 |
+
wandb_exp_name ..................................
|
| 550 |
+
wandb_project ...................................
|
| 551 |
+
wandb_save_dir ..................................
|
| 552 |
+
weight_decay .................................... 0.1
|
| 553 |
+
weight_decay_incr_style ......................... constant
|
| 554 |
+
wgrad_deferral_limit ............................ 0
|
| 555 |
+
world_size ...................................... 8
|
| 556 |
+
yaml_cfg ........................................ None
|
| 557 |
+
-------------------- end of arguments ---------------------
|
| 558 |
+
INFO:megatron.core.num_microbatches_calculator:setting number of microbatches to constant 2
|
| 559 |
+
> building GPT2BPETokenizer tokenizer ...
|
| 560 |
+
/workspace/Megatron-MoE-Echo/megatron/training/arguments.py:831: UserWarning: Disabling sequence parallelism because tensor model parallelism is disabled
|
| 561 |
+
warnings.warn("Disabling sequence parallelism because tensor model parallelism is disabled")
|
| 562 |
+
> padded vocab (size: 50257) with 47 dummy tokens (new size: 50304)
|
| 563 |
+
WARNING:megatron.core.rerun_state_machine:RerunStateMachine initialized in mode RerunMode.VALIDATE_RESULTS
|
| 564 |
+
> initializing torch distributed ...
|
| 565 |
+
/workspace/Megatron-MoE-Echo/megatron/training/arguments.py:831: UserWarning: Disabling sequence parallelism because tensor model parallelism is disabled
|
| 566 |
+
warnings.warn("Disabling sequence parallelism because tensor model parallelism is disabled")
|
| 567 |
+
/workspace/Megatron-MoE-Echo/megatron/training/arguments.py:831: UserWarning: Disabling sequence parallelism because tensor model parallelism is disabled
|
| 568 |
+
warnings.warn("Disabling sequence parallelism because tensor model parallelism is disabled")
|
| 569 |
+
/workspace/Megatron-MoE-Echo/megatron/training/arguments.py:831: UserWarning: Disabling sequence parallelism because tensor model parallelism is disabled
|
| 570 |
+
warnings.warn("Disabling sequence parallelism because tensor model parallelism is disabled")
|
| 571 |
+
/workspace/Megatron-MoE-Echo/megatron/training/arguments.py:831: UserWarning: Disabling sequence parallelism because tensor model parallelism is disabled
|
| 572 |
+
warnings.warn("Disabling sequence parallelism because tensor model parallelism is disabled")
|
| 573 |
+
/workspace/Megatron-MoE-Echo/megatron/training/arguments.py:831: UserWarning: Disabling sequence parallelism because tensor model parallelism is disabled
|
| 574 |
+
warnings.warn("Disabling sequence parallelism because tensor model parallelism is disabled")
|
| 575 |
+
WARNING: one_logger package is required to enable e2e metrics tracking. please go to https://confluence.nvidia.com/display/MLWFO/Package+Repositories for details to install it
|
| 576 |
+
[Gloo] Rank 4 is connected to 7 peer ranks. Expected number of connected peer ranks is : [Gloo] Rank 70
|
| 577 |
+
is connected to 7 peer ranks. Expected number of connected peer ranks is : 7
|
| 578 |
+
[Gloo] Rank 7 is connected to 7 peer ranks. Expected number of connected peer ranks is : 7
|
| 579 |
+
[Gloo] Rank [Gloo] Rank [Gloo] Rank [Gloo] Rank [Gloo] Rank 16235 is connected to is connected to is connected to is connected to is connected to 77777 peer ranks. peer ranks. peer ranks. peer ranks. peer ranks. Expected number of connected peer ranks is : Expected number of connected peer ranks is : Expected number of connected peer ranks is : Expected number of connected peer ranks is : Expected number of connected peer ranks is : 77777
|
| 580 |
+
|
| 581 |
+
|
| 582 |
+
|
| 583 |
+
|
| 584 |
+
[Gloo] Rank 2 is connected to 7 peer ranks. Expected number of connected peer ranks is : 7
|
| 585 |
+
[Gloo] Rank 3 is connected to 7 peer ranks. Expected number of connected peer ranks is : 7
|
| 586 |
+
[Gloo] Rank 0 is connected to 7 peer ranks. Expected number of connected peer ranks is : 7
|
| 587 |
+
[Gloo] Rank 1 is connected to 7 peer ranks. Expected number of connected peer ranks is : 7
|
| 588 |
+
[Gloo] Rank 4 is connected to 7 peer ranks. Expected number of connected peer ranks is : 7
|
| 589 |
+
[Gloo] Rank 6 is connected to 7 peer ranks. Expected number of connected peer ranks is : 7
|
| 590 |
+
[Gloo] Rank 5 is connected to 7 peer ranks. Expected number of connected peer ranks is : 7
|
| 591 |
+
[Gloo] Rank 7 is connected to 7 peer ranks. Expected number of connected peer ranks is : 7
|
| 592 |
+
[Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
|
| 593 |
+
> initialized tensor model parallel with size 1
|
| 594 |
+
> initialized pipeline model parallel with size 1
|
| 595 |
+
> setting random seeds to 1234 ...
|
| 596 |
+
[Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
|
| 597 |
+
[Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
|
| 598 |
+
[Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
|
| 599 |
+
[Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
|
| 600 |
+
[Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
|
| 601 |
+
[Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
|
| 602 |
+
[Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
|
| 603 |
+
[rank5]:[W830 06:47:23.364184986 ProcessGroupNCCL.cpp:4789] [PG ID 0 PG GUID 0 Rank 5] using GPU 5 as device used by this process is currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect. You can specify device_id in init_process_group() to force use of a particular device.
|
| 604 |
+
[rank3]:[W830 06:47:23.364342895 ProcessGroupNCCL.cpp:4789] [PG ID 0 PG GUID 0 Rank 3] using GPU 3 as device used by this process is currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect. You can specify device_id in init_process_group() to force use of a particular device.
|
| 605 |
+
[rank7]:[W830 06:47:23.374833410 ProcessGroupNCCL.cpp:4789] [PG ID 0 PG GUID 0 Rank 7] using GPU 7 as device used by this process is currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect. You can specify device_id in init_process_group() to force use of a particular device.
|
| 606 |
+
[rank6]:[W830 06:47:23.374883654 ProcessGroupNCCL.cpp:4789] [PG ID 0 PG GUID 0 Rank 6] using GPU 6 as device used by this process is currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect. You can specify device_id in init_process_group() to force use of a particular device.
|
| 607 |
+
[rank1]:[W830 06:47:23.374911183 ProcessGroupNCCL.cpp:4789] [PG ID 0 PG GUID 0 Rank 1] using GPU 1 as device used by this process is currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect. You can specify device_id in init_process_group() to force use of a particular device.
|
| 608 |
+
[rank4]:[W830 06:47:23.374918935 ProcessGroupNCCL.cpp:4789] [PG ID 0 PG GUID 0 Rank 4] using GPU 4 as device used by this process is currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect. You can specify device_id in init_process_group() to force use of a particular device.
|
| 609 |
+
[rank2]:[W830 06:47:23.374927356 ProcessGroupNCCL.cpp:4789] [PG ID 0 PG GUID 0 Rank 2] using GPU 2 as device used by this process is currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect. You can specify device_id in init_process_group() to force use of a particular device.
|
| 610 |
+
> compiling dataset index builder ...
|
| 611 |
+
make: Entering directory '/workspace/Megatron-MoE-Echo/megatron/core/datasets'
|
| 612 |
+
make: Nothing to be done for 'default'.
|
| 613 |
+
make: Leaving directory '/workspace/Megatron-MoE-Echo/megatron/core/datasets'
|
| 614 |
+
>>> done with dataset index builder. Compilation time: 0.036 seconds
|
| 615 |
+
WARNING: constraints for invoking optimized fused softmax kernel are not met. We default back to unfused kernel invocations.
|
| 616 |
+
> compiling and loading fused kernels ...
|
| 617 |
+
[rank0]:[W830 06:47:23.534087244 ProcessGroupNCCL.cpp:4789] [PG ID 0 PG GUID 0 Rank 0] using GPU 0 as device used by this process is currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect. You can specify device_id in init_process_group() to force use of a particular device.
|
| 618 |
+
>>> done with compiling and loading fused kernels. Compilation time: 3.783 seconds
|
| 619 |
+
time to initialize megatron (seconds): 7.176
|
| 620 |
+
[after megatron is initialized] datetime: 2026-08-30 06:47:29
|
| 621 |
+
building GPT model ...
|
| 622 |
+
/usr/local/lib/python3.12/dist-packages/transformer_engine/pytorch/cpu_offload.py:708: DeprecationWarning: Offloading weights is deprecated. Using offload_weights=True does not have any effect.
|
| 623 |
+
warnings.warn(
|
| 624 |
+
/usr/local/lib/python3.12/dist-packages/transformer_engine/pytorch/cpu_offload.py:708: DeprecationWarning: Offloading weights is deprecated. Using offload_weights=True does not have any effect.
|
| 625 |
+
warnings.warn(
|
| 626 |
+
/usr/local/lib/python3.12/dist-packages/transformer_engine/pytorch/cpu_offload.py:708: DeprecationWarning: Offloading weights is deprecated. Using offload_weights=True does not have any effect.
|
| 627 |
+
warnings.warn(
|
| 628 |
+
/usr/local/lib/python3.12/dist-packages/transformer_engine/pytorch/cpu_offload.py:708: DeprecationWarning: Offloading weights is deprecated. Using offload_weights=True does not have any effect.
|
| 629 |
+
warnings.warn(
|
| 630 |
+
/usr/local/lib/python3.12/dist-packages/transformer_engine/pytorch/cpu_offload.py:708: DeprecationWarning: Offloading weights is deprecated. Using offload_weights=True does not have any effect.
|
| 631 |
+
warnings.warn(
|
| 632 |
+
/usr/local/lib/python3.12/dist-packages/transformer_engine/pytorch/cpu_offload.py:708: DeprecationWarning: Offloading weights is deprecated. Using offload_weights=True does not have any effect.
|
| 633 |
+
warnings.warn(
|
| 634 |
+
/usr/local/lib/python3.12/dist-packages/transformer_engine/pytorch/cpu_offload.py:708: DeprecationWarning: Offloading weights is deprecated. Using offload_weights=True does not have any effect.
|
| 635 |
+
warnings.warn(
|
| 636 |
+
/usr/local/lib/python3.12/dist-packages/transformer_engine/pytorch/cpu_offload.py:708: DeprecationWarning: Offloading weights is deprecated. Using offload_weights=True does not have any effect.
|
| 637 |
+
warnings.warn(
|
| 638 |
+
> number of parameters on (tensor, pipeline) model parallel rank (0, 0): 3902410752
|
| 639 |
+
INFO:megatron.core.distributed.distributed_data_parallel:Setting up DistributedDataParallel with config DistributedDataParallelConfig(grad_reduce_in_fp32=True, overlap_grad_reduce=True, overlap_param_gather=True, align_param_gather=False, use_distributed_optimizer=True, num_distributed_optimizer_instances=1, check_for_nan_in_grad=False, check_for_large_grads=False, bucket_size=40000000, pad_buckets_for_high_nccl_busbw=False, average_in_collective=False, fp8_param_gather=False, reuse_grad_buf_for_mxfp8_param_ag=False, use_custom_fsdp=False, data_parallel_sharding_strategy='no_shard', gradient_reduce_div_fusion=True, suggested_communication_unit_size=None, preserve_fp32_weights=True, keep_fp8_transpose_cache_when_using_custom_fsdp=False, nccl_ub=False, fsdp_double_buffer=False)
|
| 640 |
+
INFO:megatron.core.distributed.param_and_grad_buffer:Number of buckets for gradient all-reduce / reduce-scatter: 18
|
| 641 |
+
Params for bucket 1 (206045184 elements, 206045184 padded size):
|
| 642 |
+
module.output_layer.weight
|
| 643 |
+
Params for bucket 2 (41988096 elements, 41988096 padded size):
|
| 644 |
+
module.decoder.final_layernorm.weight
|
| 645 |
+
module.decoder.layers.15.pre_mlp_layernorm.weight
|
| 646 |
+
module.decoder.layers.15.self_attention.linear_qkv.weight
|
| 647 |
+
module.decoder.layers.15.mlp.router.weight
|
| 648 |
+
module.decoder.layers.15.self_attention.linear_qkv.layer_norm_weight
|
| 649 |
+
module.decoder.layers.15.self_attention.linear_proj.weight
|
| 650 |
+
Params for bucket 3 (41984000 elements, 41984000 padded size):
|
| 651 |
+
module.decoder.layers.14.mlp.router.weight
|
| 652 |
+
module.decoder.layers.14.self_attention.linear_qkv.layer_norm_weight
|
| 653 |
+
module.decoder.layers.14.self_attention.linear_proj.weight
|
| 654 |
+
module.decoder.layers.14.self_attention.linear_qkv.weight
|
| 655 |
+
module.decoder.layers.14.pre_mlp_layernorm.weight
|
| 656 |
+
Params for bucket 4 (41984000 elements, 41984000 padded size):
|
| 657 |
+
module.decoder.layers.13.self_attention.linear_qkv.weight
|
| 658 |
+
module.decoder.layers.13.pre_mlp_layernorm.weight
|
| 659 |
+
module.decoder.layers.13.mlp.router.weight
|
| 660 |
+
module.decoder.layers.13.self_attention.linear_proj.weight
|
| 661 |
+
module.decoder.layers.13.self_attention.linear_qkv.layer_norm_weight
|
| 662 |
+
Params for bucket 5 (41984000 elements, 41984000 padded size):
|
| 663 |
+
module.decoder.layers.12.mlp.router.weight
|
| 664 |
+
module.decoder.layers.12.self_attention.linear_qkv.weight
|
| 665 |
+
module.decoder.layers.12.self_attention.linear_proj.weight
|
| 666 |
+
module.decoder.layers.12.self_attention.linear_qkv.layer_norm_weight
|
| 667 |
+
module.decoder.layers.12.pre_mlp_layernorm.weight
|
| 668 |
+
Params for bucket 6 (41984000 elements, 41984000 padded size):
|
| 669 |
+
module.decoder.layers.11.mlp.router.weight
|
| 670 |
+
module.decoder.layers.11.self_attention.linear_proj.weight
|
| 671 |
+
module.decoder.layers.11.self_attention.linear_qkv.layer_norm_weight
|
| 672 |
+
module.decoder.layers.11.self_attention.linear_qkv.weight
|
| 673 |
+
module.decoder.layers.11.pre_mlp_layernorm.weight
|
| 674 |
+
Params for bucket 7 (41984000 elements, 41984000 padded size):
|
| 675 |
+
module.decoder.layers.10.mlp.router.weight
|
| 676 |
+
module.decoder.layers.10.self_attention.linear_proj.weight
|
| 677 |
+
module.decoder.layers.10.self_attention.linear_qkv.layer_norm_weight
|
| 678 |
+
module.decoder.layers.10.self_attention.linear_qkv.weight
|
| 679 |
+
module.decoder.layers.10.pre_mlp_layernorm.weight
|
| 680 |
+
Params for bucket 8 (41984000 elements, 41984000 padded size):
|
| 681 |
+
module.decoder.layers.9.mlp.router.weight
|
| 682 |
+
module.decoder.layers.9.self_attention.linear_proj.weight
|
| 683 |
+
module.decoder.layers.9.self_attention.linear_qkv.layer_norm_weight
|
| 684 |
+
module.decoder.layers.9.pre_mlp_layernorm.weight
|
| 685 |
+
module.decoder.layers.9.self_attention.linear_qkv.weight
|
| 686 |
+
Params for bucket 9 (41984000 elements, 41984000 padded size):
|
| 687 |
+
module.decoder.layers.8.mlp.router.weight
|
| 688 |
+
module.decoder.layers.8.self_attention.linear_proj.weight
|
| 689 |
+
module.decoder.layers.8.self_attention.linear_qkv.layer_norm_weight
|
| 690 |
+
module.decoder.layers.8.pre_mlp_layernorm.weight
|
| 691 |
+
module.decoder.layers.8.self_attention.linear_qkv.weight
|
| 692 |
+
Params for bucket 10 (41984000 elements, 41984000 padded size):
|
| 693 |
+
module.decoder.layers.7.mlp.router.weight
|
| 694 |
+
module.decoder.layers.7.self_attention.linear_proj.weight
|
| 695 |
+
module.decoder.layers.7.self_attention.linear_qkv.layer_norm_weight
|
| 696 |
+
module.decoder.layers.7.pre_mlp_layernorm.weight
|
| 697 |
+
module.decoder.layers.7.self_attention.linear_qkv.weight
|
| 698 |
+
Params for bucket 11 (41984000 elements, 41984000 padded size):
|
| 699 |
+
module.decoder.layers.6.self_attention.linear_qkv.layer_norm_weight
|
| 700 |
+
module.decoder.layers.6.self_attention.linear_proj.weight
|
| 701 |
+
module.decoder.layers.6.mlp.router.weight
|
| 702 |
+
module.decoder.layers.6.self_attention.linear_qkv.weight
|
| 703 |
+
module.decoder.layers.6.pre_mlp_layernorm.weight
|
| 704 |
+
Params for bucket 12 (41984000 elements, 41984000 padded size):
|
| 705 |
+
module.decoder.layers.5.self_attention.linear_proj.weight
|
| 706 |
+
module.decoder.layers.5.pre_mlp_layernorm.weight
|
| 707 |
+
module.decoder.layers.5.self_attention.linear_qkv.weight
|
| 708 |
+
module.decoder.layers.5.mlp.router.weight
|
| 709 |
+
module.decoder.layers.5.self_attention.linear_qkv.layer_norm_weight
|
| 710 |
+
Params for bucket 13 (41984000 elements, 41984000 padded size):
|
| 711 |
+
module.decoder.layers.4.self_attention.linear_proj.weight
|
| 712 |
+
module.decoder.layers.4.self_attention.linear_qkv.layer_norm_weight
|
| 713 |
+
module.decoder.layers.4.mlp.router.weight
|
| 714 |
+
module.decoder.layers.4.self_attention.linear_qkv.weight
|
| 715 |
+
module.decoder.layers.4.pre_mlp_layernorm.weight
|
| 716 |
+
Params for bucket 14 (41984000 elements, 41984000 padded size):
|
| 717 |
+
module.decoder.layers.3.pre_mlp_layernorm.weight
|
| 718 |
+
module.decoder.layers.3.self_attention.linear_qkv.weight
|
| 719 |
+
module.decoder.layers.3.mlp.router.weight
|
| 720 |
+
module.decoder.layers.3.self_attention.linear_proj.weight
|
| 721 |
+
module.decoder.layers.3.self_attention.linear_qkv.layer_norm_weight
|
| 722 |
+
Params for bucket 15 (41984000 elements, 41984000 padded size):
|
| 723 |
+
module.decoder.layers.2.self_attention.linear_qkv.weight
|
| 724 |
+
module.decoder.layers.2.mlp.router.weight
|
| 725 |
+
module.decoder.layers.2.pre_mlp_layernorm.weight
|
| 726 |
+
module.decoder.layers.2.self_attention.linear_qkv.layer_norm_weight
|
| 727 |
+
module.decoder.layers.2.self_attention.linear_proj.weight
|
| 728 |
+
Params for bucket 16 (41984000 elements, 41984000 padded size):
|
| 729 |
+
module.decoder.layers.1.self_attention.linear_qkv.weight
|
| 730 |
+
module.decoder.layers.1.pre_mlp_layernorm.weight
|
| 731 |
+
module.decoder.layers.1.mlp.router.weight
|
| 732 |
+
module.decoder.layers.1.self_attention.linear_proj.weight
|
| 733 |
+
module.decoder.layers.1.self_attention.linear_qkv.layer_norm_weight
|
| 734 |
+
Params for bucket 17 (41984000 elements, 41984000 padded size):
|
| 735 |
+
module.decoder.layers.0.self_attention.linear_qkv.layer_norm_weight
|
| 736 |
+
module.decoder.layers.0.pre_mlp_layernorm.weight
|
| 737 |
+
module.decoder.layers.0.mlp.router.weight
|
| 738 |
+
module.decoder.layers.0.self_attention.linear_qkv.weight
|
| 739 |
+
module.decoder.layers.0.self_attention.linear_proj.weight
|
| 740 |
+
Params for bucket 18 (206045184 elements, 206045184 padded size):
|
| 741 |
+
module.embedding.word_embeddings.weight
|
| 742 |
+
INFO:megatron.core.distributed.param_and_grad_buffer:Number of buckets for gradient all-reduce / reduce-scatter: 32
|
| 743 |
+
Params for bucket 1 (58720256 elements, 58720256 padded size):
|
| 744 |
+
module.decoder.layers.15.mlp.experts.linear_fc2.weight0
|
| 745 |
+
Params for bucket 2 (117440512 elements, 117440512 padded size):
|
| 746 |
+
module.decoder.layers.15.mlp.experts.linear_fc1.weight0
|
| 747 |
+
Params for bucket 3 (58720256 elements, 58720256 padded size):
|
| 748 |
+
module.decoder.layers.14.mlp.experts.linear_fc2.weight0
|
| 749 |
+
Params for bucket 4 (117440512 elements, 117440512 padded size):
|
| 750 |
+
module.decoder.layers.14.mlp.experts.linear_fc1.weight0
|
| 751 |
+
Params for bucket 5 (58720256 elements, 58720256 padded size):
|
| 752 |
+
module.decoder.layers.13.mlp.experts.linear_fc2.weight0
|
| 753 |
+
Params for bucket 6 (117440512 elements, 117440512 padded size):
|
| 754 |
+
module.decoder.layers.13.mlp.experts.linear_fc1.weight0
|
| 755 |
+
Params for bucket 7 (58720256 elements, 58720256 padded size):
|
| 756 |
+
module.decoder.layers.12.mlp.experts.linear_fc2.weight0
|
| 757 |
+
Params for bucket 8 (117440512 elements, 117440512 padded size):
|
| 758 |
+
module.decoder.layers.12.mlp.experts.linear_fc1.weight0
|
| 759 |
+
Params for bucket 9 (58720256 elements, 58720256 padded size):
|
| 760 |
+
module.decoder.layers.11.mlp.experts.linear_fc2.weight0
|
| 761 |
+
Params for bucket 10 (117440512 elements, 117440512 padded size):
|
| 762 |
+
module.decoder.layers.11.mlp.experts.linear_fc1.weight0
|
| 763 |
+
Params for bucket 11 (58720256 elements, 58720256 padded size):
|
| 764 |
+
module.decoder.layers.10.mlp.experts.linear_fc2.weight0
|
| 765 |
+
Params for bucket 12 (117440512 elements, 117440512 padded size):
|
| 766 |
+
module.decoder.layers.10.mlp.experts.linear_fc1.weight0
|
| 767 |
+
Params for bucket 13 (58720256 elements, 58720256 padded size):
|
| 768 |
+
module.decoder.layers.9.mlp.experts.linear_fc2.weight0
|
| 769 |
+
Params for bucket 14 (117440512 elements, 117440512 padded size):
|
| 770 |
+
module.decoder.layers.9.mlp.experts.linear_fc1.weight0
|
| 771 |
+
Params for bucket 15 (58720256 elements, 58720256 padded size):
|
| 772 |
+
module.decoder.layers.8.mlp.experts.linear_fc2.weight0
|
| 773 |
+
Params for bucket 16 (117440512 elements, 117440512 padded size):
|
| 774 |
+
module.decoder.layers.8.mlp.experts.linear_fc1.weight0
|
| 775 |
+
Params for bucket 17 (58720256 elements, 58720256 padded size):
|
| 776 |
+
module.decoder.layers.7.mlp.experts.linear_fc2.weight0
|
| 777 |
+
Params for bucket 18 (117440512 elements, 117440512 padded size):
|
| 778 |
+
module.decoder.layers.7.mlp.experts.linear_fc1.weight0
|
| 779 |
+
Params for bucket 19 (58720256 elements, 58720256 padded size):
|
| 780 |
+
module.decoder.layers.6.mlp.experts.linear_fc2.weight0
|
| 781 |
+
Params for bucket 20 (117440512 elements, 117440512 padded size):
|
| 782 |
+
module.decoder.layers.6.mlp.experts.linear_fc1.weight0
|
| 783 |
+
Params for bucket 21 (58720256 elements, 58720256 padded size):
|
| 784 |
+
module.decoder.layers.5.mlp.experts.linear_fc2.weight0
|
| 785 |
+
Params for bucket 22 (117440512 elements, 117440512 padded size):
|
| 786 |
+
module.decoder.layers.5.mlp.experts.linear_fc1.weight0
|
| 787 |
+
Params for bucket 23 (58720256 elements, 58720256 padded size):
|
| 788 |
+
module.decoder.layers.4.mlp.experts.linear_fc2.weight0
|
| 789 |
+
Params for bucket 24 (117440512 elements, 117440512 padded size):
|
| 790 |
+
module.decoder.layers.4.mlp.experts.linear_fc1.weight0
|
| 791 |
+
Params for bucket 25 (58720256 elements, 58720256 padded size):
|
| 792 |
+
module.decoder.layers.3.mlp.experts.linear_fc2.weight0
|
| 793 |
+
Params for bucket 26 (117440512 elements, 117440512 padded size):
|
| 794 |
+
module.decoder.layers.3.mlp.experts.linear_fc1.weight0
|
| 795 |
+
Params for bucket 27 (58720256 elements, 58720256 padded size):
|
| 796 |
+
module.decoder.layers.2.mlp.experts.linear_fc2.weight0
|
| 797 |
+
Params for bucket 28 (117440512 elements, 117440512 padded size):
|
| 798 |
+
module.decoder.layers.2.mlp.experts.linear_fc1.weight0
|
| 799 |
+
Params for bucket 29 (58720256 elements, 58720256 padded size):
|
| 800 |
+
module.decoder.layers.1.mlp.experts.linear_fc2.weight0
|
| 801 |
+
Params for bucket 30 (117440512 elements, 117440512 padded size):
|
| 802 |
+
module.decoder.layers.1.mlp.experts.linear_fc1.weight0
|
| 803 |
+
Params for bucket 31 (58720256 elements, 58720256 padded size):
|
| 804 |
+
module.decoder.layers.0.mlp.experts.linear_fc2.weight0
|
| 805 |
+
Params for bucket 32 (117440512 elements, 117440512 padded size):
|
| 806 |
+
module.decoder.layers.0.mlp.experts.linear_fc1.weight0
|
| 807 |
+
INFO:megatron.core.optimizer:Setting up optimizer with config OptimizerConfig(optimizer='adam', lr=0.0001, min_lr=1e-05, decoupled_lr=None, decoupled_min_lr=None, weight_decay=0.1, fp8_recipe='mxfp8', fp16=False, bf16=True, reuse_grad_buf_for_mxfp8_param_ag=False, params_dtype=torch.bfloat16, use_precision_aware_optimizer=False, store_param_remainders=True, main_grads_dtype=torch.float32, main_params_dtype=torch.float32, exp_avg_dtype=torch.float32, exp_avg_sq_dtype=torch.float32, loss_scale=None, initial_loss_scale=4294967296, min_loss_scale=1.0, loss_scale_window=1000, hysteresis=2, adam_beta1=0.9, adam_beta2=0.999, adam_eps=1e-08, sgd_momentum=0.9, use_distributed_optimizer=True, overlap_param_gather_with_optimizer_step=False, optimizer_cpu_offload=False, optimizer_offload_fraction=1.0, use_torch_optimizer_for_cpu_offload=False, overlap_cpu_optimizer_d2h_h2d=False, pin_cpu_grads=True, pin_cpu_params=True, clip_grad=1.0, log_num_zeros_in_grad=False, barrier_with_L1_time=True, timers=<megatron.core.timers.Timers object at 0x7d521f580590>, config_logger_dir='')
|
| 808 |
+
INFO:megatron.core.optimizer_param_scheduler:> learning rate decay style: cosine
|
| 809 |
+
[after model, optimizer, and learning rate scheduler are built] datetime: 2026-08-30 06:47:29
|
| 810 |
+
> building train, validation, and test datasets ...
|
| 811 |
+
> datasets target sizes (minimum size):
|
| 812 |
+
train: 192
|
| 813 |
+
validation: 0
|
| 814 |
+
test: 0
|
| 815 |
+
INFO:megatron.core.datasets.blended_megatron_dataset_config:Let split_matrix = [(0, 1.0), None, None]
|
| 816 |
+
> building train, validation, and test datasets for GPT ...
|
| 817 |
+
INFO:megatron.core.datasets.blended_megatron_dataset_builder:Building GPTDataset splits with sizes=(192, 0, 0) and config=GPTDatasetConfig(random_seed=1234, sequence_length=4096, blend=(['/workspace/data/wikitext/wikitext103_gpt2_text_document'], None), blend_per_split=None, split='100,0,0', split_matrix=[(0, 1.0), None, None], num_dataset_builder_threads=1, path_to_cache=None, mmap_bin_files=True, mock=False, tokenizer=<megatron.training.tokenizer.tokenizer._GPT2BPETokenizer object at 0x7d5222159eb0>, mid_level_dataset_surplus=0.005, reset_position_ids=False, reset_attention_mask=False, eod_mask_loss=False, create_attention_mask=True, drop_last_partial_validation_sequence=True, add_extra_token_to_sequence=True, object_storage_cache_path=None)
|
| 818 |
+
INFO:megatron.core.datasets.indexed_dataset:Load the _IndexReader from /workspace/data/wikitext/wikitext103_gpt2_text_document.idx
|
| 819 |
+
INFO:megatron.core.datasets.indexed_dataset: Extract the sequence lengths
|
| 820 |
+
INFO:megatron.core.datasets.indexed_dataset: Extract the sequence pointers
|
| 821 |
+
INFO:megatron.core.datasets.indexed_dataset: Extract the document indices
|
| 822 |
+
INFO:megatron.core.datasets.indexed_dataset:> total number of sequences: 1165029
|
| 823 |
+
INFO:megatron.core.datasets.indexed_dataset:> total number of documents: 1801350
|
| 824 |
+
INFO:megatron.core.datasets.gpt_dataset:Load the GPTDataset train indices
|
| 825 |
+
INFO:megatron.core.datasets.gpt_dataset: Load the document index from 44ed1ee5bcb9e5b030387b8d27a6c406-GPTDataset-train-document_index.npy
|
| 826 |
+
INFO:megatron.core.datasets.gpt_dataset: Load the sample index from 44ed1ee5bcb9e5b030387b8d27a6c406-GPTDataset-train-sample_index.npy
|
| 827 |
+
INFO:megatron.core.datasets.gpt_dataset: Load the shuffle index from 44ed1ee5bcb9e5b030387b8d27a6c406-GPTDataset-train-shuffle_index.npy
|
| 828 |
+
INFO:megatron.core.datasets.gpt_dataset:> total number of samples: 29073
|
| 829 |
+
> finished creating GPT datasets ...
|
| 830 |
+
[after dataloaders are built] datetime: 2026-08-30 06:47:29
|
| 831 |
+
done with setup ...
|
| 832 |
+
(min, max) time across ranks (ms):
|
| 833 |
+
model-and-optimizer-setup ......................: (203.77, 206.93)
|
| 834 |
+
train/valid/test-data-iterators-setup ..........: (59.74, 71.63)
|
| 835 |
+
training ...
|
| 836 |
+
Overwriting rerun_state_machine.current_iteration from -1 to 0...
|
| 837 |
+
[before the start of training step] datetime: 2026-08-30 06:47:29
|
| 838 |
+
/workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:334: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
|
| 839 |
+
warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
|
| 840 |
+
/workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:334: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
|
| 841 |
+
warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
|
| 842 |
+
/workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:334: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
|
| 843 |
+
warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
|
| 844 |
+
/workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:334: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
|
| 845 |
+
warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
|
| 846 |
+
/workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:336: UserWarning: The use_host_meta is deprecated, it will be removed in the future.
|
| 847 |
+
warnings.warn("The use_host_meta is deprecated, it will be removed in the future.")
|
| 848 |
+
/workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:336: UserWarning: The use_host_meta is deprecated, it will be removed in the future.
|
| 849 |
+
warnings.warn("The use_host_meta is deprecated, it will be removed in the future.")
|
| 850 |
+
/workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:334: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
|
| 851 |
+
warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
|
| 852 |
+
/workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:334: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
|
| 853 |
+
warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
|
| 854 |
+
/workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:336: UserWarning: The use_host_meta is deprecated, it will be removed in the future.
|
| 855 |
+
warnings.warn("The use_host_meta is deprecated, it will be removed in the future.")
|
| 856 |
+
/workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:334: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
|
| 857 |
+
warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
|
| 858 |
+
/workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:336: UserWarning: The use_host_meta is deprecated, it will be removed in the future.
|
| 859 |
+
warnings.warn("The use_host_meta is deprecated, it will be removed in the future.")
|
| 860 |
+
/workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:334: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
|
| 861 |
+
warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
|
| 862 |
+
/workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:336: UserWarning: The use_host_meta is deprecated, it will be removed in the future.
|
| 863 |
+
warnings.warn("The use_host_meta is deprecated, it will be removed in the future.")
|
| 864 |
+
/workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:336: UserWarning: The use_host_meta is deprecated, it will be removed in the future.
|
| 865 |
+
warnings.warn("The use_host_meta is deprecated, it will be removed in the future.")
|
| 866 |
+
/workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:336: UserWarning: The use_host_meta is deprecated, it will be removed in the future.
|
| 867 |
+
warnings.warn("The use_host_meta is deprecated, it will be removed in the future.")
|
| 868 |
+
/workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:336: UserWarning: The use_host_meta is deprecated, it will be removed in the future.
|
| 869 |
+
warnings.warn("The use_host_meta is deprecated, it will be removed in the future.")
|
| 870 |
+
/workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:482: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
|
| 871 |
+
warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
|
| 872 |
+
/workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:482: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
|
| 873 |
+
warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
|
| 874 |
+
/workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:482: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
|
| 875 |
+
warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
|
| 876 |
+
/workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:482: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
|
| 877 |
+
warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
|
| 878 |
+
/workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:482: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
|
| 879 |
+
warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
|
| 880 |
+
/workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:482: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
|
| 881 |
+
warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
|
| 882 |
+
/workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:482: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
|
| 883 |
+
warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
|
| 884 |
+
/workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:482: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
|
| 885 |
+
warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
|
| 886 |
+
/workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:334: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
|
| 887 |
+
warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
|
| 888 |
+
/workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:336: UserWarning: The use_host_meta is deprecated, it will be removed in the future.
|
| 889 |
+
warnings.warn("The use_host_meta is deprecated, it will be removed in the future.")
|
| 890 |
+
/workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:334: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
|
| 891 |
+
warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
|
| 892 |
+
/workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:336: UserWarning: The use_host_meta is deprecated, it will be removed in the future.
|
| 893 |
+
warnings.warn("The use_host_meta is deprecated, it will be removed in the future.")
|
| 894 |
+
/workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:334: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
|
| 895 |
+
warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
|
| 896 |
+
/workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:336: UserWarning: The use_host_meta is deprecated, it will be removed in the future.
|
| 897 |
+
warnings.warn("The use_host_meta is deprecated, it will be removed in the future.")
|
| 898 |
+
/workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:334: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
|
| 899 |
+
warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
|
| 900 |
+
/workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:336: UserWarning: The use_host_meta is deprecated, it will be removed in the future.
|
| 901 |
+
warnings.warn("The use_host_meta is deprecated, it will be removed in the future.")
|
| 902 |
+
/workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:334: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
|
| 903 |
+
warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
|
| 904 |
+
/workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:336: UserWarning: The use_host_meta is deprecated, it will be removed in the future.
|
| 905 |
+
warnings.warn("The use_host_meta is deprecated, it will be removed in the future.")
|
| 906 |
+
/workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:334: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
|
| 907 |
+
warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
|
| 908 |
+
/workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:336: UserWarning: The use_host_meta is deprecated, it will be removed in the future.
|
| 909 |
+
warnings.warn("The use_host_meta is deprecated, it will be removed in the future.")
|
| 910 |
+
/workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:334: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
|
| 911 |
+
warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
|
| 912 |
+
/workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:336: UserWarning: The use_host_meta is deprecated, it will be removed in the future.
|
| 913 |
+
warnings.warn("The use_host_meta is deprecated, it will be removed in the future.")
|
| 914 |
+
/workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:334: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
|
| 915 |
+
warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
|
| 916 |
+
/workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:336: UserWarning: The use_host_meta is deprecated, it will be removed in the future.
|
| 917 |
+
warnings.warn("The use_host_meta is deprecated, it will be removed in the future.")
|
| 918 |
+
/workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:482: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
|
| 919 |
+
warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
|
| 920 |
+
/workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:482: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
|
| 921 |
+
warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
|
| 922 |
+
/workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:482: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
|
| 923 |
+
warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
|
| 924 |
+
/workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:482: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
|
| 925 |
+
warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
|
| 926 |
+
/workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:482: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
|
| 927 |
+
warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
|
| 928 |
+
/workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:482: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
|
| 929 |
+
warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
|
| 930 |
+
/workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:482: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
|
| 931 |
+
warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
|
| 932 |
+
/workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:482: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
|
| 933 |
+
warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
|
| 934 |
+
/workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:334: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
|
| 935 |
+
warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
|
| 936 |
+
/workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:334: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
|
| 937 |
+
warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
|
| 938 |
+
/workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:334: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
|
| 939 |
+
warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
|
| 940 |
+
/workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:334: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
|
| 941 |
+
warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
|
| 942 |
+
/workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:334: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
|
| 943 |
+
warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
|
| 944 |
+
/workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:334: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
|
| 945 |
+
warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
|
| 946 |
+
/workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:334: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
|
| 947 |
+
warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
|
| 948 |
+
/workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:334: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
|
| 949 |
+
warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
|
| 950 |
+
/workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:336: UserWarning: The use_host_meta is deprecated, it will be removed in the future.
|
| 951 |
+
warnings.warn("The use_host_meta is deprecated, it will be removed in the future.")
|
| 952 |
+
/workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:336: UserWarning: The use_host_meta is deprecated, it will be removed in the future.
|
| 953 |
+
warnings.warn("The use_host_meta is deprecated, it will be removed in the future.")
|
| 954 |
+
/workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:336: UserWarning: The use_host_meta is deprecated, it will be removed in the future.
|
| 955 |
+
warnings.warn("The use_host_meta is deprecated, it will be removed in the future.")
|
| 956 |
+
/workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:336: UserWarning: The use_host_meta is deprecated, it will be removed in the future.
|
| 957 |
+
warnings.warn("The use_host_meta is deprecated, it will be removed in the future.")
|
| 958 |
+
/workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:336: UserWarning: The use_host_meta is deprecated, it will be removed in the future.
|
| 959 |
+
warnings.warn("The use_host_meta is deprecated, it will be removed in the future.")
|
| 960 |
+
/workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:336: UserWarning: The use_host_meta is deprecated, it will be removed in the future.
|
| 961 |
+
warnings.warn("The use_host_meta is deprecated, it will be removed in the future.")
|
| 962 |
+
/workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:336: UserWarning: The use_host_meta is deprecated, it will be removed in the future.
|
| 963 |
+
warnings.warn("The use_host_meta is deprecated, it will be removed in the future.")
|
| 964 |
+
/workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:336: UserWarning: The use_host_meta is deprecated, it will be removed in the future.
|
| 965 |
+
warnings.warn("The use_host_meta is deprecated, it will be removed in the future.")
|
| 966 |
+
[2026-08-30 06:48:07] iteration 1/ 12 | consumed samples: 16 | elapsed time per iteration (ms): 37805.9 | throughput per GPU (TFLOP/s/GPU): 8.8 | learning rate: 2.000000E-05 | global batch size: 16 | lm loss: 1.104376E+01 | loss scale: 1.0 | grad norm: 34.296 | number of skipped iterations: 0 | number of nan iterations: 0 |
|
| 967 |
+
Number of parameters in transformer block in billions: 23.22
|
| 968 |
+
Number of parameters in embedding layers in billions: 0.41
|
| 969 |
+
Total number of parameters in billions: 23.63
|
| 970 |
+
Number of parameters in most loaded shard in billions: 23.6320
|
| 971 |
+
Theoretical memory footprints: weight and optimizer=169029.43 MB
|
| 972 |
+
[Rank 0] (after 1 iterations) memory (MB) | allocated: 63795.404296875 | max allocated: 96519.67578125 | reserved: 99122.0 | max reserved: 99122.0
|
| 973 |
+
[2026-08-30 06:48:07] iteration 2/ 12 | consumed samples: 32 | elapsed time per iteration (ms): 570.6 | throughput per GPU (TFLOP/s/GPU): 584.3 | learning rate: 4.000000E-05 | global batch size: 16 | lm loss: 1.104935E+01 | loss scale: 1.0 | grad norm: 35.986 | number of skipped iterations: 0 | number of nan iterations: 0 |
|
| 974 |
+
[2026-08-30 06:48:08] iteration 3/ 12 | consumed samples: 48 | elapsed time per iteration (ms): 557.9 | throughput per GPU (TFLOP/s/GPU): 597.6 | learning rate: 6.000000E-05 | global batch size: 16 | lm loss: 9.657968E+00 | loss scale: 1.0 | grad norm: 40.332 | number of skipped iterations: 0 | number of nan iterations: 0 |
|
| 975 |
+
Capture CUDA graph for training!!!
|
| 976 |
+
CUDA graph capture done!!!
|
| 977 |
+
[2026-08-30 06:48:09] iteration 4/ 12 | consumed samples: 64 | elapsed time per iteration (ms): 1653.3 | throughput per GPU (TFLOP/s/GPU): 201.6 | learning rate: 8.000000E-05 | global batch size: 16 | lm loss: 1.081336E+01 | loss scale: 1.0 | grad norm: 93.363 | number of skipped iterations: 0 | number of nan iterations: 0 |
|
| 978 |
+
[2026-08-30 06:48:10] iteration 5/ 12 | consumed samples: 80 | elapsed time per iteration (ms): 469.0 | throughput per GPU (TFLOP/s/GPU): 710.9 | learning rate: 1.000000E-04 | global batch size: 16 | lm loss: 1.068684E+01 | loss scale: 1.0 | grad norm: 17.191 | number of skipped iterations: 0 | number of nan iterations: 0 |
|
| 979 |
+
[2026-08-30 06:48:10] iteration 6/ 12 | consumed samples: 96 | elapsed time per iteration (ms): 467.1 | throughput per GPU (TFLOP/s/GPU): 713.7 | learning rate: 7.750000E-05 | global batch size: 16 | lm loss: 9.917974E+00 | loss scale: 1.0 | grad norm: 42.077 | number of skipped iterations: 0 | number of nan iterations: 0 |
|
| 980 |
+
[2026-08-30 06:48:11] iteration 7/ 12 | consumed samples: 112 | elapsed time per iteration (ms): 472.2 | throughput per GPU (TFLOP/s/GPU): 706.0 | learning rate: 3.250000E-05 | global batch size: 16 | lm loss: 9.323465E+00 | loss scale: 1.0 | grad norm: 7.381 | number of skipped iterations: 0 | number of nan iterations: 0 |
|
| 981 |
+
[2026-08-30 06:48:11] iteration 8/ 12 | consumed samples: 128 | elapsed time per iteration (ms): 466.8 | throughput per GPU (TFLOP/s/GPU): 714.2 | learning rate: 1.000000E-05 | global batch size: 16 | lm loss: 9.001214E+00 | loss scale: 1.0 | grad norm: 19.429 | number of skipped iterations: 0 | number of nan iterations: 0 |
|
| 982 |
+
[2026-08-30 06:48:12] iteration 9/ 12 | consumed samples: 144 | elapsed time per iteration (ms): 465.3 | throughput per GPU (TFLOP/s/GPU): 716.4 | learning rate: 1.000000E-05 | global batch size: 16 | lm loss: 8.649192E+00 | loss scale: 1.0 | grad norm: 5.915 | number of skipped iterations: 0 | number of nan iterations: 0 |
|
| 983 |
+
[2026-08-30 06:48:12] iteration 10/ 12 | consumed samples: 160 | elapsed time per iteration (ms): 461.8 | throughput per GPU (TFLOP/s/GPU): 721.9 | learning rate: 1.000000E-05 | global batch size: 16 | lm loss: 8.543521E+00 | loss scale: 1.0 | grad norm: 5.254 | number of skipped iterations: 0 | number of nan iterations: 0 |
|
| 984 |
+
[2026-08-30 06:48:13] iteration 11/ 12 | consumed samples: 176 | elapsed time per iteration (ms): 467.3 | throughput per GPU (TFLOP/s/GPU): 713.4 | learning rate: 1.000000E-05 | global batch size: 16 | lm loss: 8.433981E+00 | loss scale: 1.0 | grad norm: 4.838 | number of skipped iterations: 0 | number of nan iterations: 0 |
|
| 985 |
+
[2026-08-30 06:48:13] iteration 12/ 12 | consumed samples: 192 | elapsed time per iteration (ms): 471.2 | throughput per GPU (TFLOP/s/GPU): 707.4 | learning rate: 1.000000E-05 | global batch size: 16 | lm loss: 8.369435E+00 | loss scale: 1.0 | grad norm: 4.479 | number of skipped iterations: 0 | number of nan iterations: 0 |
|
| 986 |
+
[after training is done] datetime: 2026-08-30 06:48:13
|
| 987 |
+
W0830 06:48:14.145000 658758 torch/distributed/elastic/agent/server/api.py:719] Received 15 death signal, shutting down workers
|
| 988 |
+
W0830 06:48:14.146000 658758 torch/distributed/elastic/multiprocessing/api.py:900] Sending process 658847 closing signal SIGTERM
|
| 989 |
+
W0830 06:48:14.147000 658758 torch/distributed/elastic/multiprocessing/api.py:900] Sending process 658848 closing signal SIGTERM
|
| 990 |
+
W0830 06:48:14.149000 658758 torch/distributed/elastic/multiprocessing/api.py:900] Sending process 658849 closing signal SIGTERM
|
| 991 |
+
W0830 06:48:14.151000 658758 torch/distributed/elastic/multiprocessing/api.py:900] Sending process 658850 closing signal SIGTERM
|
| 992 |
+
W0830 06:48:14.153000 658758 torch/distributed/elastic/multiprocessing/api.py:900] Sending process 658851 closing signal SIGTERM
|
| 993 |
+
W0830 06:48:14.155000 658758 torch/distributed/elastic/multiprocessing/api.py:900] Sending process 658852 closing signal SIGTERM
|
| 994 |
+
W0830 06:48:14.157000 658758 torch/distributed/elastic/multiprocessing/api.py:900] Sending process 658853 closing signal SIGTERM
|
| 995 |
+
W0830 06:48:14.159000 658758 torch/distributed/elastic/multiprocessing/api.py:900] Sending process 658854 closing signal SIGTERM
|
| 996 |
+
Traceback (most recent call last):
|
| 997 |
+
File "/usr/local/bin/torchrun", line 8, in <module>
|
| 998 |
+
sys.exit(main())
|
| 999 |
+
^^^^^^
|
| 1000 |
+
File "/usr/local/lib/python3.12/dist-packages/torch/distributed/elastic/multiprocessing/errors/__init__.py", line 355, in wrapper
|
| 1001 |
+
return f(*args, **kwargs)
|
| 1002 |
+
^^^^^^^^^^^^^^^^^^
|
| 1003 |
+
File "/usr/local/lib/python3.12/dist-packages/torch/distributed/run.py", line 892, in main
|
| 1004 |
+
run(args)
|
| 1005 |
+
File "/usr/local/lib/python3.12/dist-packages/torch/distributed/run.py", line 883, in run
|
| 1006 |
+
elastic_launch(
|
| 1007 |
+
File "/usr/local/lib/python3.12/dist-packages/torch/distributed/launcher/api.py", line 139, in __call__
|
| 1008 |
+
return launch_agent(self._config, self._entrypoint, list(args))
|
| 1009 |
+
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
| 1010 |
+
File "/usr/local/lib/python3.12/dist-packages/torch/distributed/launcher/api.py", line 261, in launch_agent
|
| 1011 |
+
result = agent.run()
|
| 1012 |
+
^^^^^^^^^^^
|
| 1013 |
+
File "/usr/local/lib/python3.12/dist-packages/torch/distributed/elastic/metrics/api.py", line 138, in wrapper
|
| 1014 |
+
result = f(*args, **kwargs)
|
| 1015 |
+
^^^^^^^^^^^^^^^^^^
|
| 1016 |
+
File "/usr/local/lib/python3.12/dist-packages/torch/distributed/elastic/agent/server/api.py", line 711, in run
|
| 1017 |
+
result = self._invoke_run(role)
|
| 1018 |
+
^^^^^^^^^^^^^^^^^^^^^^
|
| 1019 |
+
File "/usr/local/lib/python3.12/dist-packages/torch/distributed/elastic/agent/server/api.py", line 870, in _invoke_run
|
| 1020 |
+
time.sleep(monitor_interval)
|
| 1021 |
+
File "/usr/local/lib/python3.12/dist-packages/torch/distributed/elastic/multiprocessing/api.py", line 84, in _terminate_process_handler
|
| 1022 |
+
raise SignalException(f"Process {os.getpid()} got signal: {sigval}", sigval=sigval)
|
| 1023 |
+
torch.distributed.elastic.multiprocessing.api.SignalException: Process 658758 got signal: 15
|
tiera-rerun-fixed-moonep/echo/a2/a2_matched_gbs16/G2__echo__matched__seq4096__gbs16__cap1p08203125/rep1/mem_samples.csv
ADDED
|
@@ -0,0 +1,256 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
0, 0
|
| 2 |
+
1, 0
|
| 3 |
+
2, 0
|
| 4 |
+
3, 0
|
| 5 |
+
4, 0
|
| 6 |
+
5, 0
|
| 7 |
+
6, 0
|
| 8 |
+
7, 0
|
| 9 |
+
0, 0
|
| 10 |
+
1, 0
|
| 11 |
+
2, 0
|
| 12 |
+
3, 0
|
| 13 |
+
4, 0
|
| 14 |
+
5, 0
|
| 15 |
+
6, 0
|
| 16 |
+
7, 0
|
| 17 |
+
0, 1
|
| 18 |
+
1, 1
|
| 19 |
+
2, 1
|
| 20 |
+
3, 1
|
| 21 |
+
4, 1
|
| 22 |
+
5, 1
|
| 23 |
+
6, 1
|
| 24 |
+
7, 1
|
| 25 |
+
0, 4
|
| 26 |
+
1, 4
|
| 27 |
+
2, 4
|
| 28 |
+
3, 4
|
| 29 |
+
4, 1
|
| 30 |
+
5, 1
|
| 31 |
+
6, 1
|
| 32 |
+
7, 1
|
| 33 |
+
0, 4
|
| 34 |
+
1, 4
|
| 35 |
+
2, 4
|
| 36 |
+
3, 4
|
| 37 |
+
4, 4
|
| 38 |
+
5, 4
|
| 39 |
+
6, 4
|
| 40 |
+
7, 4
|
| 41 |
+
0, 4
|
| 42 |
+
1, 629
|
| 43 |
+
2, 629
|
| 44 |
+
3, 629
|
| 45 |
+
4, 629
|
| 46 |
+
5, 629
|
| 47 |
+
6, 629
|
| 48 |
+
7, 629
|
| 49 |
+
0, 891
|
| 50 |
+
1, 899
|
| 51 |
+
2, 887
|
| 52 |
+
3, 887
|
| 53 |
+
4, 885
|
| 54 |
+
5, 885
|
| 55 |
+
6, 899
|
| 56 |
+
7, 891
|
| 57 |
+
0, 1895
|
| 58 |
+
1, 1895
|
| 59 |
+
2, 1895
|
| 60 |
+
3, 1895
|
| 61 |
+
4, 1895
|
| 62 |
+
5, 1895
|
| 63 |
+
6, 1895
|
| 64 |
+
7, 1895
|
| 65 |
+
0, 39401
|
| 66 |
+
1, 39365
|
| 67 |
+
2, 39379
|
| 68 |
+
3, 39417
|
| 69 |
+
4, 39345
|
| 70 |
+
5, 39397
|
| 71 |
+
6, 39439
|
| 72 |
+
7, 39409
|
| 73 |
+
0, 39785
|
| 74 |
+
1, 39741
|
| 75 |
+
2, 39767
|
| 76 |
+
3, 39773
|
| 77 |
+
4, 39757
|
| 78 |
+
5, 39803
|
| 79 |
+
6, 39789
|
| 80 |
+
7, 39779
|
| 81 |
+
0, 42297
|
| 82 |
+
1, 42289
|
| 83 |
+
2, 42289
|
| 84 |
+
3, 42289
|
| 85 |
+
4, 42289
|
| 86 |
+
5, 42289
|
| 87 |
+
6, 42289
|
| 88 |
+
7, 42289
|
| 89 |
+
0, 42541
|
| 90 |
+
1, 42533
|
| 91 |
+
2, 42533
|
| 92 |
+
3, 42533
|
| 93 |
+
4, 42533
|
| 94 |
+
5, 42533
|
| 95 |
+
6, 42533
|
| 96 |
+
7, 42533
|
| 97 |
+
0, 42709
|
| 98 |
+
1, 42695
|
| 99 |
+
2, 42699
|
| 100 |
+
3, 42703
|
| 101 |
+
4, 42735
|
| 102 |
+
5, 42715
|
| 103 |
+
6, 42725
|
| 104 |
+
7, 42713
|
| 105 |
+
0, 43457
|
| 106 |
+
1, 43449
|
| 107 |
+
2, 43449
|
| 108 |
+
3, 43449
|
| 109 |
+
4, 43449
|
| 110 |
+
5, 43449
|
| 111 |
+
6, 43449
|
| 112 |
+
7, 43449
|
| 113 |
+
0, 43735
|
| 114 |
+
1, 43727
|
| 115 |
+
2, 43727
|
| 116 |
+
3, 43727
|
| 117 |
+
4, 43727
|
| 118 |
+
5, 43727
|
| 119 |
+
6, 43727
|
| 120 |
+
7, 43727
|
| 121 |
+
0, 45167
|
| 122 |
+
1, 45159
|
| 123 |
+
2, 45159
|
| 124 |
+
3, 45159
|
| 125 |
+
4, 45159
|
| 126 |
+
5, 45159
|
| 127 |
+
6, 45159
|
| 128 |
+
7, 45159
|
| 129 |
+
0, 45167
|
| 130 |
+
1, 45159
|
| 131 |
+
2, 45159
|
| 132 |
+
3, 45159
|
| 133 |
+
4, 45159
|
| 134 |
+
5, 45159
|
| 135 |
+
6, 45159
|
| 136 |
+
7, 45159
|
| 137 |
+
0, 75747
|
| 138 |
+
1, 75739
|
| 139 |
+
2, 75739
|
| 140 |
+
3, 75739
|
| 141 |
+
4, 75739
|
| 142 |
+
5, 75739
|
| 143 |
+
6, 75739
|
| 144 |
+
7, 75739
|
| 145 |
+
0, 76233
|
| 146 |
+
1, 76225
|
| 147 |
+
2, 76225
|
| 148 |
+
3, 76225
|
| 149 |
+
4, 76225
|
| 150 |
+
5, 76225
|
| 151 |
+
6, 76225
|
| 152 |
+
7, 76225
|
| 153 |
+
0, 76233
|
| 154 |
+
1, 76225
|
| 155 |
+
2, 76225
|
| 156 |
+
3, 76225
|
| 157 |
+
4, 76225
|
| 158 |
+
5, 76225
|
| 159 |
+
6, 76225
|
| 160 |
+
7, 76225
|
| 161 |
+
0, 76233
|
| 162 |
+
1, 76225
|
| 163 |
+
2, 76225
|
| 164 |
+
3, 76225
|
| 165 |
+
4, 76225
|
| 166 |
+
5, 76225
|
| 167 |
+
6, 76225
|
| 168 |
+
7, 76225
|
| 169 |
+
0, 94137
|
| 170 |
+
1, 94155
|
| 171 |
+
2, 94127
|
| 172 |
+
3, 94147
|
| 173 |
+
4, 94155
|
| 174 |
+
5, 94159
|
| 175 |
+
6, 94175
|
| 176 |
+
7, 94165
|
| 177 |
+
0, 94853
|
| 178 |
+
1, 94833
|
| 179 |
+
2, 94887
|
| 180 |
+
3, 94899
|
| 181 |
+
4, 94925
|
| 182 |
+
5, 94879
|
| 183 |
+
6, 94923
|
| 184 |
+
7, 94955
|
| 185 |
+
0, 95613
|
| 186 |
+
1, 95605
|
| 187 |
+
2, 95605
|
| 188 |
+
3, 95605
|
| 189 |
+
4, 95605
|
| 190 |
+
5, 95605
|
| 191 |
+
6, 95605
|
| 192 |
+
7, 95605
|
| 193 |
+
0, 96223
|
| 194 |
+
1, 96235
|
| 195 |
+
2, 96225
|
| 196 |
+
3, 96233
|
| 197 |
+
4, 96237
|
| 198 |
+
5, 96223
|
| 199 |
+
6, 96227
|
| 200 |
+
7, 96249
|
| 201 |
+
0, 77247
|
| 202 |
+
1, 106731
|
| 203 |
+
2, 77251
|
| 204 |
+
3, 98447
|
| 205 |
+
4, 76989
|
| 206 |
+
5, 77251
|
| 207 |
+
6, 77235
|
| 208 |
+
7, 77035
|
| 209 |
+
0, 145689
|
| 210 |
+
1, 145681
|
| 211 |
+
2, 145683
|
| 212 |
+
3, 145459
|
| 213 |
+
4, 145461
|
| 214 |
+
5, 145683
|
| 215 |
+
6, 145681
|
| 216 |
+
7, 145455
|
| 217 |
+
0, 145689
|
| 218 |
+
1, 145681
|
| 219 |
+
2, 145683
|
| 220 |
+
3, 145459
|
| 221 |
+
4, 145461
|
| 222 |
+
5, 145683
|
| 223 |
+
6, 145681
|
| 224 |
+
7, 145455
|
| 225 |
+
0, 145689
|
| 226 |
+
1, 145681
|
| 227 |
+
2, 145683
|
| 228 |
+
3, 145459
|
| 229 |
+
4, 145461
|
| 230 |
+
5, 145683
|
| 231 |
+
6, 145681
|
| 232 |
+
7, 145455
|
| 233 |
+
0, 144773
|
| 234 |
+
1, 144765
|
| 235 |
+
2, 144767
|
| 236 |
+
3, 144543
|
| 237 |
+
4, 144545
|
| 238 |
+
5, 144767
|
| 239 |
+
6, 3398
|
| 240 |
+
7, 144539
|
| 241 |
+
0, 2115
|
| 242 |
+
1, 144762
|
| 243 |
+
2, 3139
|
| 244 |
+
3, 1862
|
| 245 |
+
4, 1862
|
| 246 |
+
5, 1862
|
| 247 |
+
6, 1862
|
| 248 |
+
7, 1862
|
| 249 |
+
0, 0
|
| 250 |
+
1, 0
|
| 251 |
+
2, 0
|
| 252 |
+
3, 0
|
| 253 |
+
4, 0
|
| 254 |
+
5, 0
|
| 255 |
+
6, 0
|
| 256 |
+
7, 0
|
tiera-rerun-fixed-moonep/echo/a2/a2_matched_gbs16/G2__echo__matched__seq4096__gbs16__cap1p08203125/rep1/run.env
ADDED
|
@@ -0,0 +1,70 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
backend=echo
|
| 2 |
+
config=G2
|
| 3 |
+
family=tierA
|
| 4 |
+
repo=/workspace/Megatron-MoE-Echo
|
| 5 |
+
branch=bracing-regions-through-kernels
|
| 6 |
+
commit=9a236793
|
| 7 |
+
num_layers=16
|
| 8 |
+
hidden=4096
|
| 9 |
+
ffn_hidden=14336
|
| 10 |
+
moe_ffn_hidden=3584
|
| 11 |
+
num_experts=32
|
| 12 |
+
topk=8
|
| 13 |
+
experts_per_rank=4
|
| 14 |
+
expert_params=44040192
|
| 15 |
+
heads=32
|
| 16 |
+
kv_groups=8
|
| 17 |
+
kv_channels=
|
| 18 |
+
qk_layernorm=0
|
| 19 |
+
shared_ffn=0
|
| 20 |
+
shared_expert_overlap=0
|
| 21 |
+
moe_token_dispatcher=flex
|
| 22 |
+
mla=0
|
| 23 |
+
dense_first_layers=0
|
| 24 |
+
moe_layers=16
|
| 25 |
+
rotary_base=1000000
|
| 26 |
+
rotary_scaling=
|
| 27 |
+
qkv_bias=0
|
| 28 |
+
router_balancing=aux_loss
|
| 29 |
+
aux_loss_coeff=1e-2
|
| 30 |
+
router_pre_softmax=0
|
| 31 |
+
router_fusion=0
|
| 32 |
+
router_dtype=fp32
|
| 33 |
+
device_initiated_grouped_gemm=1
|
| 34 |
+
norm_epsilon=1e-5
|
| 35 |
+
seq_length=4096
|
| 36 |
+
micro_batch_size=1
|
| 37 |
+
global_batch_size=16
|
| 38 |
+
num_microbatches=2
|
| 39 |
+
assignments_per_rank=32768
|
| 40 |
+
mean_tokens_per_expert=8192
|
| 41 |
+
slot_policy=matched
|
| 42 |
+
slots_per_rank=4
|
| 43 |
+
capacity_mode=static
|
| 44 |
+
capacity=1.08203125
|
| 45 |
+
cuda_graph=1
|
| 46 |
+
probe_mode=0
|
| 47 |
+
async_grad_reduce=0
|
| 48 |
+
branch_role=timing
|
| 49 |
+
memory_dump_mode=none
|
| 50 |
+
memory_snapshot_path=
|
| 51 |
+
cuda_graph_warmup_steps=3
|
| 52 |
+
recompute=none
|
| 53 |
+
force_balance=0
|
| 54 |
+
train_iters=12
|
| 55 |
+
ep_size=8
|
| 56 |
+
echo_dispatch_overlap=1
|
| 57 |
+
echo_dispatch_recompute=0
|
| 58 |
+
ultraep_quota_min_tokens=
|
| 59 |
+
ultraep_weight_sync_join=
|
| 60 |
+
profile_enable=0
|
| 61 |
+
kernel_trace=0
|
| 62 |
+
fp8_mode=mxfp8
|
| 63 |
+
fp8_recipe=mxfp8
|
| 64 |
+
training_seed=1234
|
| 65 |
+
learning_rate=1e-4
|
| 66 |
+
min_learning_rate=1.0e-5
|
| 67 |
+
lr_warmup_iters=5
|
| 68 |
+
state_dump_enable=0
|
| 69 |
+
state_dump_dir=
|
| 70 |
+
moonep_assert_no_pending_reduce=0
|
tiera-rerun-fixed-moonep/echo/a2/a2_matched_gbs16/G2__echo__matched__seq4096__gbs16__cap1p08203125/rep1/train.log
ADDED
|
@@ -0,0 +1,1107 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
==============================================================
|
| 2 |
+
G2 / echo (slots/rank=4 policy=matched)
|
| 3 |
+
E=32 K=8 H'=3584 m=4
|
| 4 |
+
L=16 seq=4096 gbs=16 (2 microbatches)
|
| 5 |
+
cap_mode=static cap=1.08203125 cg=1 recompute=none
|
| 6 |
+
memory_dump=none snapshots=/workspace/tiera-rerun-fixed-moonep/echo/a2/a2_matched_gbs16/G2__echo__matched__seq4096__gbs16__cap1p08203125/rep1/memory_snapshots/snapshot.pickle
|
| 7 |
+
repo=/workspace/Megatron-MoE-Echo @ 9a236793
|
| 8 |
+
==============================================================
|
| 9 |
+
W0830 06:48:20.670000 662151 torch/distributed/run.py:766]
|
| 10 |
+
W0830 06:48:20.670000 662151 torch/distributed/run.py:766] *****************************************
|
| 11 |
+
W0830 06:48:20.670000 662151 torch/distributed/run.py:766] Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed.
|
| 12 |
+
W0830 06:48:20.670000 662151 torch/distributed/run.py:766] *****************************************
|
| 13 |
+
/workspace/Megatron-MoE-Echo/megatron/training/arguments.py:831: UserWarning: Disabling sequence parallelism because tensor model parallelism is disabled
|
| 14 |
+
warnings.warn("Disabling sequence parallelism because tensor model parallelism is disabled")
|
| 15 |
+
/workspace/Megatron-MoE-Echo/megatron/training/arguments.py:831: UserWarning: Disabling sequence parallelism because tensor model parallelism is disabled
|
| 16 |
+
warnings.warn("Disabling sequence parallelism because tensor model parallelism is disabled")
|
| 17 |
+
/workspace/Megatron-MoE-Echo/megatron/training/arguments.py:831: UserWarning: Disabling sequence parallelism because tensor model parallelism is disabled
|
| 18 |
+
warnings.warn("Disabling sequence parallelism because tensor model parallelism is disabled")
|
| 19 |
+
/workspace/Megatron-MoE-Echo/megatron/training/arguments.py:831: UserWarning: Disabling sequence parallelism because tensor model parallelism is disabled
|
| 20 |
+
warnings.warn("Disabling sequence parallelism because tensor model parallelism is disabled")
|
| 21 |
+
/workspace/Megatron-MoE-Echo/megatron/training/arguments.py:831: UserWarning: Disabling sequence parallelism because tensor model parallelism is disabled
|
| 22 |
+
warnings.warn("Disabling sequence parallelism because tensor model parallelism is disabled")
|
| 23 |
+
/workspace/Megatron-MoE-Echo/megatron/training/arguments.py:831: UserWarning: Disabling sequence parallelism because tensor model parallelism is disabled
|
| 24 |
+
warnings.warn("Disabling sequence parallelism because tensor model parallelism is disabled")
|
| 25 |
+
/workspace/Megatron-MoE-Echo/megatron/training/arguments.py:831: UserWarning: Disabling sequence parallelism because tensor model parallelism is disabled
|
| 26 |
+
warnings.warn("Disabling sequence parallelism because tensor model parallelism is disabled")
|
| 27 |
+
using world size: 8, data-parallel size: 8, context-parallel size: 1, hierarchical context-parallel sizes: None, tensor-model-parallel size: 1, pipeline-model-parallel size: 1
|
| 28 |
+
WARNING: overriding default arguments for tokenizer_type:GPT2BPETokenizer with tokenizer_type:GPT2BPETokenizer
|
| 29 |
+
Number of virtual stages per pipeline stage: None
|
| 30 |
+
accumulate and all-reduce gradients in fp32 for bfloat16 data type.
|
| 31 |
+
using torch.bfloat16 for parameters ...
|
| 32 |
+
/workspace/Megatron-MoE-Echo/megatron/training/arguments.py:831: UserWarning: Disabling sequence parallelism because tensor model parallelism is disabled
|
| 33 |
+
warnings.warn("Disabling sequence parallelism because tensor model parallelism is disabled")
|
| 34 |
+
------------------------ arguments ------------------------
|
| 35 |
+
account_for_embedding_in_pipeline_split ......... False
|
| 36 |
+
account_for_loss_in_pipeline_split .............. False
|
| 37 |
+
accumulate_allreduce_grads_in_fp32 .............. True
|
| 38 |
+
adam_beta1 ...................................... 0.9
|
| 39 |
+
adam_beta2 ...................................... 0.999
|
| 40 |
+
adam_eps ........................................ 1e-08
|
| 41 |
+
add_bias_linear ................................. False
|
| 42 |
+
add_position_embedding .......................... False
|
| 43 |
+
add_qkv_bias .................................... False
|
| 44 |
+
adlr_autoresume ................................. False
|
| 45 |
+
adlr_autoresume_interval ........................ 1000
|
| 46 |
+
align_grad_reduce ............................... True
|
| 47 |
+
align_param_gather .............................. False
|
| 48 |
+
app_tag_run_name ................................ None
|
| 49 |
+
app_tag_run_version ............................. 0.0.0
|
| 50 |
+
apply_layernorm_1p .............................. False
|
| 51 |
+
apply_query_key_layer_scaling ................... False
|
| 52 |
+
apply_residual_connection_post_layernorm ........ False
|
| 53 |
+
apply_rope_fusion ............................... True
|
| 54 |
+
async_save ...................................... None
|
| 55 |
+
async_tensor_model_parallel_allreduce ........... True
|
| 56 |
+
attention_backend ............................... AttnBackend.auto
|
| 57 |
+
attention_dropout ............................... 0.0
|
| 58 |
+
attention_softmax_in_fp32 ....................... False
|
| 59 |
+
auto_detect_ckpt_format ......................... False
|
| 60 |
+
barrier_with_L1_time ............................ True
|
| 61 |
+
bert_binary_head ................................ True
|
| 62 |
+
bert_embedder_type .............................. megatron
|
| 63 |
+
bert_load ....................................... None
|
| 64 |
+
bf16 ............................................ True
|
| 65 |
+
bias_dropout_fusion ............................. True
|
| 66 |
+
bias_gelu_fusion ................................ False
|
| 67 |
+
bias_swiglu_fusion .............................. True
|
| 68 |
+
biencoder_projection_dim ........................ 0
|
| 69 |
+
biencoder_shared_query_context_model ............ False
|
| 70 |
+
block_data_path ................................. None
|
| 71 |
+
calc_ft_timeouts ................................ False
|
| 72 |
+
calculate_per_token_loss ........................ False
|
| 73 |
+
check_for_large_grads ........................... False
|
| 74 |
+
check_for_nan_in_loss_and_grad .................. False
|
| 75 |
+
check_for_spiky_loss ............................ False
|
| 76 |
+
check_weight_hash_across_dp_replicas_interval ... None
|
| 77 |
+
ckpt_assume_constant_structure .................. False
|
| 78 |
+
ckpt_convert_format ............................. None
|
| 79 |
+
ckpt_convert_save ............................... None
|
| 80 |
+
ckpt_convert_update_legacy_dist_opt_format ...... False
|
| 81 |
+
ckpt_format ..................................... torch_dist
|
| 82 |
+
ckpt_fully_parallel_load ........................ False
|
| 83 |
+
ckpt_fully_parallel_save ........................ True
|
| 84 |
+
ckpt_fully_parallel_save_deprecated ............. False
|
| 85 |
+
ckpt_step ....................................... None
|
| 86 |
+
classes_fraction ................................ 1.0
|
| 87 |
+
clip_grad ....................................... 1.0
|
| 88 |
+
clone_scatter_output_in_embedding ............... True
|
| 89 |
+
config_logger_dir ...............................
|
| 90 |
+
consumed_train_samples .......................... 0
|
| 91 |
+
consumed_valid_samples .......................... 0
|
| 92 |
+
context_parallel_size ........................... 1
|
| 93 |
+
cp_comm_type .................................... ['p2p']
|
| 94 |
+
create_attention_mask_in_dataloader ............. True
|
| 95 |
+
cross_entropy_fusion_impl ....................... native
|
| 96 |
+
cross_entropy_loss_fusion ....................... False
|
| 97 |
+
cuda_graph_scope ................................ full_iteration
|
| 98 |
+
cuda_graph_warmup_steps ......................... 3
|
| 99 |
+
data_args_path .................................. None
|
| 100 |
+
data_cache_path ................................. None
|
| 101 |
+
data_parallel_random_init ....................... False
|
| 102 |
+
data_parallel_sharding_strategy ................. no_shard
|
| 103 |
+
data_parallel_size .............................. 8
|
| 104 |
+
data_path ....................................... ['/workspace/data/wikitext/wikitext103_gpt2_text_document']
|
| 105 |
+
data_per_class_fraction ......................... 1.0
|
| 106 |
+
data_sharding ................................... True
|
| 107 |
+
dataloader_type ................................. single
|
| 108 |
+
ddp_average_in_collective ....................... False
|
| 109 |
+
ddp_bucket_size ................................. None
|
| 110 |
+
ddp_num_buckets ................................. None
|
| 111 |
+
ddp_pad_buckets_for_high_nccl_busbw ............. False
|
| 112 |
+
decoder_first_pipeline_num_layers ............... None
|
| 113 |
+
decoder_last_pipeline_num_layers ................ None
|
| 114 |
+
decoder_num_layers .............................. None
|
| 115 |
+
decoder_seq_length .............................. None
|
| 116 |
+
decoupled_lr .................................... None
|
| 117 |
+
decoupled_min_lr ................................ None
|
| 118 |
+
decrease_batch_size_if_needed ................... False
|
| 119 |
+
defer_embedding_wgrad_compute ................... False
|
| 120 |
+
delay_wgrad_compute ............................. False
|
| 121 |
+
deprecated_use_mcore_models ..................... True
|
| 122 |
+
deterministic_mode .............................. False
|
| 123 |
+
dino_bottleneck_size ............................ 256
|
| 124 |
+
dino_freeze_last_layer .......................... 1
|
| 125 |
+
dino_head_hidden_size ........................... 2048
|
| 126 |
+
dino_local_crops_number ......................... 10
|
| 127 |
+
dino_local_img_size ............................. 96
|
| 128 |
+
dino_norm_last_layer ............................ False
|
| 129 |
+
dino_teacher_temp ............................... 0.07
|
| 130 |
+
dino_warmup_teacher_temp ........................ 0.04
|
| 131 |
+
dino_warmup_teacher_temp_epochs ................. 30
|
| 132 |
+
disable_bf16_reduced_precision_matmul ........... False
|
| 133 |
+
disable_mamba_mem_eff_path ...................... False
|
| 134 |
+
disable_straggler_on_startup .................... False
|
| 135 |
+
dist_ckpt_format_deprecated ..................... None
|
| 136 |
+
dist_ckpt_strictness ............................ assume_ok_unexpected
|
| 137 |
+
distribute_saved_activations .................... False
|
| 138 |
+
distributed_backend ............................. nccl
|
| 139 |
+
distributed_timeout_minutes ..................... 10
|
| 140 |
+
embedding_init_method_std ....................... None
|
| 141 |
+
embedding_path .................................. None
|
| 142 |
+
empty_unused_memory_level ....................... 0
|
| 143 |
+
enable_cuda_graph ............................... True
|
| 144 |
+
enable_experimental ............................. False
|
| 145 |
+
enable_ft_package ............................... False
|
| 146 |
+
enable_gloo_process_groups ...................... True
|
| 147 |
+
enable_msc ...................................... True
|
| 148 |
+
enable_one_logger ............................... True
|
| 149 |
+
encoder_num_layers .............................. 16
|
| 150 |
+
encoder_seq_length .............................. 4096
|
| 151 |
+
end_weight_decay ................................ 0.1
|
| 152 |
+
eod_mask_loss ................................... False
|
| 153 |
+
error_injection_rate ............................ 0
|
| 154 |
+
error_injection_type ............................ transient_error
|
| 155 |
+
eval_interval ................................... 1000000
|
| 156 |
+
eval_iters ...................................... 0
|
| 157 |
+
evidence_data_path .............................. None
|
| 158 |
+
exit_duration_in_mins ........................... None
|
| 159 |
+
exit_interval ................................... None
|
| 160 |
+
exit_on_missing_checkpoint ...................... False
|
| 161 |
+
exit_signal_handler ............................. False
|
| 162 |
+
exp_avg_dtype ................................... torch.float32
|
| 163 |
+
exp_avg_sq_dtype ................................ torch.float32
|
| 164 |
+
expert_model_parallel_size ...................... 8
|
| 165 |
+
expert_tensor_parallel_size ..................... 1
|
| 166 |
+
export_force_local_attention .................... False
|
| 167 |
+
export_kd_cfg ................................... None
|
| 168 |
+
export_kd_teacher_ckpt_format ................... None
|
| 169 |
+
export_kd_teacher_load .......................... None
|
| 170 |
+
export_kv_cache_quant ........................... False
|
| 171 |
+
export_legacy_megatron .......................... False
|
| 172 |
+
export_model_type ............................... GPTModel
|
| 173 |
+
export_moe_apply_probs_on_input ................. False
|
| 174 |
+
export_qk_l2_norm ............................... False
|
| 175 |
+
export_quant_cfg ................................ None
|
| 176 |
+
export_real_quant_cfg ........................... None
|
| 177 |
+
export_te_mcore_model ........................... False
|
| 178 |
+
external_cuda_graph ............................. False
|
| 179 |
+
ffn_hidden_size ................................. 14336
|
| 180 |
+
finetune ........................................ False
|
| 181 |
+
finetune_data_split ............................. train
|
| 182 |
+
finetune_hf_dataset ............................. None
|
| 183 |
+
first_last_layers_bf16 .......................... False
|
| 184 |
+
flash_decode .................................... False
|
| 185 |
+
fp16 ............................................ False
|
| 186 |
+
fp16_lm_cross_entropy ........................... False
|
| 187 |
+
fp32_residual_connection ........................ False
|
| 188 |
+
fp8 ............................................. e4m3
|
| 189 |
+
fp8_amax_compute_algo ........................... most_recent
|
| 190 |
+
fp8_amax_history_len ............................ 1
|
| 191 |
+
fp8_interval .................................... 1
|
| 192 |
+
fp8_margin ...................................... 0
|
| 193 |
+
fp8_param_gather ................................ False
|
| 194 |
+
fp8_recipe ...................................... mxfp8
|
| 195 |
+
fp8_wgrad ....................................... True
|
| 196 |
+
fsdp_double_buffer .............................. False
|
| 197 |
+
global_batch_size ............................... 16
|
| 198 |
+
grad_reduce_in_bf16 ............................. False
|
| 199 |
+
gradient_accumulation_fusion .................... True
|
| 200 |
+
gradient_reduce_div_fusion ...................... True
|
| 201 |
+
group_query_attention ........................... True
|
| 202 |
+
head_lr_mult .................................... 1.0
|
| 203 |
+
heterogeneous_layers_config_encoded_json ........ None
|
| 204 |
+
heterogeneous_layers_config_path ................ None
|
| 205 |
+
hidden_dropout .................................. 0.0
|
| 206 |
+
hidden_size ..................................... 4096
|
| 207 |
+
hierarchical_context_parallel_sizes ............. None
|
| 208 |
+
high_priority_stream_groups ..................... []
|
| 209 |
+
hybrid_attention_ratio .......................... 0.0
|
| 210 |
+
hybrid_mlp_ratio ................................ 0.0
|
| 211 |
+
hybrid_override_pattern ......................... None
|
| 212 |
+
hysteresis ...................................... 2
|
| 213 |
+
ict_head_size ................................... None
|
| 214 |
+
ict_load ........................................ None
|
| 215 |
+
img_h ........................................... 224
|
| 216 |
+
img_w ........................................... 224
|
| 217 |
+
indexer_batch_size .............................. 128
|
| 218 |
+
indexer_log_interval ............................ 1000
|
| 219 |
+
inference_batch_times_seqlen_threshold .......... -1
|
| 220 |
+
inference_dynamic_batching ...................... False
|
| 221 |
+
inference_dynamic_batching_buffer_guaranteed_fraction 0.2
|
| 222 |
+
inference_dynamic_batching_buffer_overflow_factor None
|
| 223 |
+
inference_dynamic_batching_buffer_size_gb ....... 40.0
|
| 224 |
+
inference_dynamic_batching_chunk_size ........... 256
|
| 225 |
+
inference_dynamic_batching_max_requests_override None
|
| 226 |
+
inference_dynamic_batching_max_tokens_override .. None
|
| 227 |
+
inference_dynamic_batching_num_cuda_graphs ...... 16
|
| 228 |
+
inference_max_batch_size ........................ 8
|
| 229 |
+
inference_max_seq_length ........................ 2560
|
| 230 |
+
inference_rng_tracker ........................... False
|
| 231 |
+
init_method_std ................................. 0.01
|
| 232 |
+
init_method_xavier_uniform ...................... False
|
| 233 |
+
init_model_with_meta_device ..................... False
|
| 234 |
+
initial_loss_scale .............................. 4294967296
|
| 235 |
+
inprocess_active_world_size ..................... 8
|
| 236 |
+
inprocess_barrier_timeout ....................... 120
|
| 237 |
+
inprocess_completion_timeout .................... 120
|
| 238 |
+
inprocess_empty_cuda_cache ...................... False
|
| 239 |
+
inprocess_granularity ........................... node
|
| 240 |
+
inprocess_hard_timeout .......................... 90
|
| 241 |
+
inprocess_heartbeat_interval .................... 30
|
| 242 |
+
inprocess_heartbeat_timeout ..................... 60
|
| 243 |
+
inprocess_last_call_wait ........................ 1
|
| 244 |
+
inprocess_max_iterations ........................ None
|
| 245 |
+
inprocess_monitor_process_interval .............. 1.0
|
| 246 |
+
inprocess_monitor_thread_interval ............... 1.0
|
| 247 |
+
inprocess_progress_watchdog_interval ............ 1.0
|
| 248 |
+
inprocess_restart ............................... False
|
| 249 |
+
inprocess_soft_timeout .......................... 60
|
| 250 |
+
inprocess_termination_grace_time ................ 1
|
| 251 |
+
is_hybrid_model ................................. False
|
| 252 |
+
iter_per_epoch .................................. 1250
|
| 253 |
+
iterations_to_skip .............................. []
|
| 254 |
+
keep_fp8_transpose_cache_when_using_custom_fsdp . False
|
| 255 |
+
kitchen_config_file ............................. None
|
| 256 |
+
kitchen_recipe_number ........................... None
|
| 257 |
+
kv_channels ..................................... 128
|
| 258 |
+
kv_lora_rank .................................... 32
|
| 259 |
+
lazy_mpu_init ................................... None
|
| 260 |
+
load ............................................ None
|
| 261 |
+
load_main_params_from_ckpt ...................... None
|
| 262 |
+
load_model_opt_format ........................... False
|
| 263 |
+
local_rank ...................................... 0
|
| 264 |
+
log_energy ...................................... False
|
| 265 |
+
log_interval .................................... 1
|
| 266 |
+
log_loss_scale_to_tensorboard ................... True
|
| 267 |
+
log_memory_to_tensorboard ....................... False
|
| 268 |
+
log_num_zeros_in_grad ........................... False
|
| 269 |
+
log_params_norm ................................. False
|
| 270 |
+
log_progress .................................... False
|
| 271 |
+
log_straggler ................................... False
|
| 272 |
+
log_throughput .................................. True
|
| 273 |
+
log_timers_to_tensorboard ....................... False
|
| 274 |
+
log_validation_ppl_to_tensorboard ............... False
|
| 275 |
+
log_world_size_to_tensorboard ................... False
|
| 276 |
+
logging_level ................................... None
|
| 277 |
+
loss_scale ...................................... None
|
| 278 |
+
loss_scale_window ............................... 1000
|
| 279 |
+
lr .............................................. 0.0001
|
| 280 |
+
lr_decay_iters .................................. 8
|
| 281 |
+
lr_decay_samples ................................ None
|
| 282 |
+
lr_decay_style .................................. cosine
|
| 283 |
+
lr_warmup_fraction .............................. None
|
| 284 |
+
lr_warmup_init .................................. 0.0
|
| 285 |
+
lr_warmup_iters ................................. 5
|
| 286 |
+
lr_warmup_samples ............................... 0
|
| 287 |
+
lr_wsd_decay_iters .............................. None
|
| 288 |
+
lr_wsd_decay_samples ............................ None
|
| 289 |
+
lr_wsd_decay_style .............................. exponential
|
| 290 |
+
main_grads_dtype ................................ torch.float32
|
| 291 |
+
main_params_dtype ............................... torch.float32
|
| 292 |
+
make_vocab_size_divisible_by .................... 128
|
| 293 |
+
mamba_head_dim .................................. 64
|
| 294 |
+
mamba_num_groups ................................ 8
|
| 295 |
+
mamba_num_heads ................................. None
|
| 296 |
+
mamba_state_dim ................................. 128
|
| 297 |
+
manual_gc ....................................... False
|
| 298 |
+
manual_gc_eval .................................. True
|
| 299 |
+
manual_gc_interval .............................. 0
|
| 300 |
+
mask_factor ..................................... 1.0
|
| 301 |
+
mask_prob ....................................... 0.15
|
| 302 |
+
mask_type ....................................... random
|
| 303 |
+
masked_softmax_fusion ........................... False
|
| 304 |
+
max_position_embeddings ......................... 32768
|
| 305 |
+
max_tokens_to_oom ............................... 12000
|
| 306 |
+
memory_snapshot_path ............................ snapshot.pickle
|
| 307 |
+
merge_file ...................................... /workspace/data/wikitext/gpt2-merges.txt
|
| 308 |
+
micro_batch_size ................................ 1
|
| 309 |
+
microbatch_group_size_per_vp_stage .............. None
|
| 310 |
+
mid_level_dataset_surplus ....................... 0.005
|
| 311 |
+
min_loss_scale .................................. 1.0
|
| 312 |
+
min_lr .......................................... 1e-05
|
| 313 |
+
mlp_chunks_for_prefill .......................... 1
|
| 314 |
+
mmap_bin_files .................................. True
|
| 315 |
+
mock_data ....................................... False
|
| 316 |
+
moe_apply_probs_on_input ........................ False
|
| 317 |
+
moe_aux_loss_coeff .............................. 0.01
|
| 318 |
+
moe_deepep_num_sms .............................. 20
|
| 319 |
+
moe_echo_enable_random_offloading ............... False
|
| 320 |
+
moe_echo_expert_dispatch_overlap ................ True
|
| 321 |
+
moe_echo_expert_dispatcher_type ................. hybridep
|
| 322 |
+
moe_echo_recompute_expert_dispatch .............. False
|
| 323 |
+
moe_enable_deepep ............................... False
|
| 324 |
+
moe_enable_echo ................................. True
|
| 325 |
+
moe_enable_hybridep ............................. True
|
| 326 |
+
moe_expert_capacity_factor ...................... None
|
| 327 |
+
moe_extended_tp ................................. False
|
| 328 |
+
moe_ffn_hidden_size ............................. 3584
|
| 329 |
+
moe_grouped_gemm ................................ True
|
| 330 |
+
moe_hybridep_num_sms ............................ 16
|
| 331 |
+
moe_input_jitter_eps ............................ None
|
| 332 |
+
moe_layer_freq .................................. 1
|
| 333 |
+
moe_layer_recompute ............................. False
|
| 334 |
+
moe_num_echo_experts ............................ 32
|
| 335 |
+
moe_pad_expert_input_to_capacity ................ False
|
| 336 |
+
moe_per_layer_logging ........................... False
|
| 337 |
+
moe_permute_fusion .............................. False
|
| 338 |
+
moe_received_token_capacity ..................... 1.08203125
|
| 339 |
+
moe_router_bias_update_rate ..................... 0.001
|
| 340 |
+
moe_router_dtype ................................ fp32
|
| 341 |
+
moe_router_enable_expert_bias ................... False
|
| 342 |
+
moe_router_force_load_balancing ................. False
|
| 343 |
+
moe_router_group_topk ........................... None
|
| 344 |
+
moe_router_load_balancing_type .................. aux_loss
|
| 345 |
+
moe_router_num_groups ........................... None
|
| 346 |
+
moe_router_padding_for_fp8 ...................... False
|
| 347 |
+
moe_router_pre_softmax .......................... False
|
| 348 |
+
moe_router_score_function ....................... softmax
|
| 349 |
+
moe_router_topk ................................. 8
|
| 350 |
+
moe_router_topk_scaling_factor .................. None
|
| 351 |
+
moe_shared_expert_intermediate_size ............. None
|
| 352 |
+
moe_shared_expert_overlap ....................... False
|
| 353 |
+
moe_token_dispatcher_type ....................... flex
|
| 354 |
+
moe_token_drop_policy ........................... probs
|
| 355 |
+
moe_upcycling_granularity ....................... 1
|
| 356 |
+
moe_use_device_initiated_grouped_gemm ........... True
|
| 357 |
+
moe_use_legacy_grouped_gemm ..................... False
|
| 358 |
+
moe_use_upcycling ............................... False
|
| 359 |
+
moe_z_loss_coeff ................................ None
|
| 360 |
+
mrope_section ................................... None
|
| 361 |
+
mscale .......................................... 1.0
|
| 362 |
+
mscale_all_dim .................................. 0.0
|
| 363 |
+
mtp_loss_scaling_factor ......................... 0.1
|
| 364 |
+
mtp_num_layers .................................. None
|
| 365 |
+
multi_latent_attention .......................... False
|
| 366 |
+
nccl_all_reduce_for_prefill ..................... False
|
| 367 |
+
nccl_communicator_config_path ................... None
|
| 368 |
+
nccl_ub ......................................... False
|
| 369 |
+
no_load_optim ................................... None
|
| 370 |
+
no_load_rng ..................................... None
|
| 371 |
+
no_persist_layer_norm ........................... False
|
| 372 |
+
no_rope_freq .................................... None
|
| 373 |
+
no_save_optim ................................... None
|
| 374 |
+
no_save_rng ..................................... None
|
| 375 |
+
non_persistent_ckpt_type ........................ None
|
| 376 |
+
non_persistent_global_ckpt_dir .................. None
|
| 377 |
+
non_persistent_local_ckpt_algo .................. fully_parallel
|
| 378 |
+
non_persistent_local_ckpt_dir ................... None
|
| 379 |
+
non_persistent_save_interval .................... None
|
| 380 |
+
norm_epsilon .................................... 1e-05
|
| 381 |
+
normalization ................................... RMSNorm
|
| 382 |
+
num_attention_heads ............................. 32
|
| 383 |
+
num_channels .................................... 3
|
| 384 |
+
num_classes ..................................... 1000
|
| 385 |
+
num_dataset_builder_threads ..................... 1
|
| 386 |
+
num_distributed_optimizer_instances ............. 1
|
| 387 |
+
num_experts ..................................... 32
|
| 388 |
+
num_layers ...................................... 16
|
| 389 |
+
num_layers_at_end_in_bf16 ....................... 1
|
| 390 |
+
num_layers_at_start_in_bf16 ..................... 1
|
| 391 |
+
num_layers_per_virtual_pipeline_stage ........... None
|
| 392 |
+
num_query_groups ................................ 8
|
| 393 |
+
num_virtual_stages_per_pipeline_rank ............ None
|
| 394 |
+
num_workers ..................................... 2
|
| 395 |
+
object_storage_cache_path ....................... None
|
| 396 |
+
one_logger_async ................................ False
|
| 397 |
+
one_logger_project .............................. megatron-lm
|
| 398 |
+
one_logger_run_name ............................. None
|
| 399 |
+
onnx_safe ....................................... None
|
| 400 |
+
openai_gelu ..................................... False
|
| 401 |
+
optimizer ....................................... adam
|
| 402 |
+
optimizer_cpu_offload ........................... False
|
| 403 |
+
optimizer_offload_fraction ...................... 1.0
|
| 404 |
+
output_bert_embeddings .......................... False
|
| 405 |
+
overlap_cpu_optimizer_d2h_h2d ................... False
|
| 406 |
+
overlap_grad_reduce ............................. True
|
| 407 |
+
overlap_moe_expert_parallel_comm ................ False
|
| 408 |
+
overlap_p2p_comm ................................ False
|
| 409 |
+
overlap_p2p_comm_warmup_flush ................... False
|
| 410 |
+
overlap_param_gather ............................ True
|
| 411 |
+
overlap_param_gather_with_optimizer_step ........ False
|
| 412 |
+
override_opt_param_scheduler .................... False
|
| 413 |
+
params_dtype .................................... torch.bfloat16
|
| 414 |
+
patch_dim ....................................... 16
|
| 415 |
+
per_split_data_args_path ........................ None
|
| 416 |
+
perform_initialization .......................... True
|
| 417 |
+
pin_cpu_grads ................................... True
|
| 418 |
+
pin_cpu_params .................................. True
|
| 419 |
+
pipeline_model_parallel_comm_backend ............ None
|
| 420 |
+
pipeline_model_parallel_layout .................. None
|
| 421 |
+
pipeline_model_parallel_size .................... 1
|
| 422 |
+
position_embedding_type ......................... rope
|
| 423 |
+
pretrained_checkpoint ........................... None
|
| 424 |
+
profile ......................................... False
|
| 425 |
+
profile_ranks ................................... [0]
|
| 426 |
+
profile_step_end ................................ 12
|
| 427 |
+
profile_step_start .............................. 10
|
| 428 |
+
q_lora_rank ..................................... None
|
| 429 |
+
qk_head_dim ..................................... 128
|
| 430 |
+
qk_l2_norm ...................................... False
|
| 431 |
+
qk_layernorm .................................... False
|
| 432 |
+
qk_pos_emb_head_dim ............................. 64
|
| 433 |
+
query_in_block_prob ............................. 0.1
|
| 434 |
+
rampup_batch_size ............................... None
|
| 435 |
+
rank ............................................ 0
|
| 436 |
+
recompute_granularity ........................... None
|
| 437 |
+
recompute_method ................................ None
|
| 438 |
+
recompute_modules ............................... None
|
| 439 |
+
recompute_num_layers ............................ None
|
| 440 |
+
record_memory_history ........................... False
|
| 441 |
+
relative_attention_max_distance ................. 128
|
| 442 |
+
relative_attention_num_buckets .................. 32
|
| 443 |
+
replication ..................................... False
|
| 444 |
+
replication_factor .............................. 2
|
| 445 |
+
replication_jump ................................ None
|
| 446 |
+
rerun_mode ...................................... validate_results
|
| 447 |
+
reset_attention_mask ............................ False
|
| 448 |
+
reset_position_ids .............................. False
|
| 449 |
+
result_rejected_tracker_filename ................ None
|
| 450 |
+
retriever_report_topk_accuracies ................ []
|
| 451 |
+
retriever_score_scaling ......................... False
|
| 452 |
+
retriever_seq_length ............................ 256
|
| 453 |
+
retro_add_retriever ............................. False
|
| 454 |
+
retro_attention_gate ............................ 1
|
| 455 |
+
retro_cyclic_train_iters ........................ None
|
| 456 |
+
retro_encoder_attention_dropout ................. 0.1
|
| 457 |
+
retro_encoder_hidden_dropout .................... 0.1
|
| 458 |
+
retro_encoder_layers ............................ 2
|
| 459 |
+
retro_num_neighbors ............................. 2
|
| 460 |
+
retro_num_retrieved_chunks ...................... 2
|
| 461 |
+
retro_project_dir ............................... None
|
| 462 |
+
retro_verify_neighbor_count ..................... True
|
| 463 |
+
reuse_grad_buf_for_mxfp8_param_ag ............... False
|
| 464 |
+
rope_scaling_factor ............................. 8.0
|
| 465 |
+
rope_type ....................................... None
|
| 466 |
+
rotary_base ..................................... 1000000
|
| 467 |
+
rotary_interleaved .............................. False
|
| 468 |
+
rotary_percent .................................. 1.0
|
| 469 |
+
rotary_scaling_factor ........................... 1.0
|
| 470 |
+
rotary_seq_len_interpolation_factor ............. None
|
| 471 |
+
run_workload_inspector_server ................... False
|
| 472 |
+
sample_rate ..................................... 1.0
|
| 473 |
+
save ............................................ None
|
| 474 |
+
save_interval ................................... None
|
| 475 |
+
save_retain_interval ............................ None
|
| 476 |
+
scatter_gather_tensors_in_pipeline .............. True
|
| 477 |
+
seed ............................................ 1234
|
| 478 |
+
seq_length ...................................... 4096
|
| 479 |
+
sequence_parallel ............................... False
|
| 480 |
+
sft ............................................. False
|
| 481 |
+
sft_tokenizer_prompt_format ..................... nemotron-h-aligned
|
| 482 |
+
sgd_momentum .................................... 0.9
|
| 483 |
+
short_seq_prob .................................. 0.1
|
| 484 |
+
skip_train ...................................... False
|
| 485 |
+
skipped_train_samples ........................... 0
|
| 486 |
+
spec ............................................ None
|
| 487 |
+
split ........................................... 100,0,0
|
| 488 |
+
squared_relu .................................... False
|
| 489 |
+
start_weight_decay .............................. 0.1
|
| 490 |
+
straggler_ctrlr_port ............................ 65535
|
| 491 |
+
straggler_minmax_count .......................... 1
|
| 492 |
+
suggested_communication_unit_size ............... None
|
| 493 |
+
swiglu .......................................... True
|
| 494 |
+
swin_backbone_type .............................. tiny
|
| 495 |
+
symmetric_ar_type ............................... None
|
| 496 |
+
te_rng_tracker .................................. True
|
| 497 |
+
tensor_model_parallel_size ...................... 1
|
| 498 |
+
tensorboard_dir ................................. None
|
| 499 |
+
tensorboard_log_interval ........................ 1
|
| 500 |
+
tensorboard_queue_size .......................... 1000
|
| 501 |
+
test_data_path .................................. None
|
| 502 |
+
test_mode ....................................... False
|
| 503 |
+
tiktoken_num_special_tokens ..................... 1000
|
| 504 |
+
tiktoken_pattern ................................ None
|
| 505 |
+
tiktoken_special_tokens ......................... None
|
| 506 |
+
timing_log_level ................................ 0
|
| 507 |
+
timing_log_option ............................... minmax
|
| 508 |
+
titles_data_path ................................ None
|
| 509 |
+
tokenizer_model ................................. None
|
| 510 |
+
tokenizer_type .................................. GPT2BPETokenizer
|
| 511 |
+
torch_fsdp2_reshard_after_forward ............... True
|
| 512 |
+
tp_comm_bootstrap_backend ....................... nccl
|
| 513 |
+
tp_comm_bulk_dgrad .............................. True
|
| 514 |
+
tp_comm_bulk_wgrad .............................. True
|
| 515 |
+
tp_comm_overlap ................................. False
|
| 516 |
+
tp_comm_overlap_ag .............................. True
|
| 517 |
+
tp_comm_overlap_cfg ............................. None
|
| 518 |
+
tp_comm_overlap_rs .............................. True
|
| 519 |
+
tp_comm_overlap_rs_dgrad ........................ False
|
| 520 |
+
tp_comm_split_ag ................................ True
|
| 521 |
+
tp_comm_split_rs ................................ True
|
| 522 |
+
train_data_path ................................. None
|
| 523 |
+
train_iters ..................................... 12
|
| 524 |
+
train_samples ................................... None
|
| 525 |
+
train_sync_interval ............................. None
|
| 526 |
+
transformer_impl ................................ transformer_engine
|
| 527 |
+
transformer_pipeline_model_parallel_size ........ 1
|
| 528 |
+
untie_embeddings_and_output_weights ............. True
|
| 529 |
+
use_checkpoint_args ............................. False
|
| 530 |
+
use_checkpoint_opt_param_scheduler .............. False
|
| 531 |
+
use_cpu_initialization .......................... None
|
| 532 |
+
use_custom_fsdp ................................. False
|
| 533 |
+
use_dist_ckpt ................................... True
|
| 534 |
+
use_dist_ckpt_deprecated ........................ False
|
| 535 |
+
use_distributed_optimizer ....................... True
|
| 536 |
+
use_flash_attn .................................. False
|
| 537 |
+
use_legacy_models ............................... False
|
| 538 |
+
use_mp_args_from_checkpoint_args ................ False
|
| 539 |
+
use_one_sent_docs ............................... False
|
| 540 |
+
use_persistent_ckpt_worker ...................... False
|
| 541 |
+
use_precision_aware_optimizer ................... False
|
| 542 |
+
use_pytorch_profiler ............................ False
|
| 543 |
+
use_ring_exchange_p2p ........................... False
|
| 544 |
+
use_rope_scaling ................................ False
|
| 545 |
+
use_rotary_position_embeddings .................. False
|
| 546 |
+
use_sharp ....................................... False
|
| 547 |
+
use_tokenizer_model_from_checkpoint_args ........ True
|
| 548 |
+
use_torch_fsdp2 ................................. False
|
| 549 |
+
use_torch_optimizer_for_cpu_offload ............. False
|
| 550 |
+
use_tp_pp_dp_mapping ............................ False
|
| 551 |
+
v_head_dim ...................................... 128
|
| 552 |
+
valid_data_path ................................. None
|
| 553 |
+
variable_seq_lengths ............................ False
|
| 554 |
+
virtual_pipeline_model_parallel_size ............ None
|
| 555 |
+
vision_backbone_type ............................ vit
|
| 556 |
+
vision_pretraining .............................. False
|
| 557 |
+
vision_pretraining_type ......................... classify
|
| 558 |
+
vocab_extra_ids ................................. 0
|
| 559 |
+
vocab_file ...................................... /workspace/data/wikitext/gpt2-vocab.json
|
| 560 |
+
vocab_size ...................................... None
|
| 561 |
+
wandb_exp_name ..................................
|
| 562 |
+
wandb_project ...................................
|
| 563 |
+
wandb_save_dir ..................................
|
| 564 |
+
weight_decay .................................... 0.1
|
| 565 |
+
weight_decay_incr_style ......................... constant
|
| 566 |
+
wgrad_deferral_limit ............................ 0
|
| 567 |
+
world_size ...................................... 8
|
| 568 |
+
yaml_cfg ........................................ None
|
| 569 |
+
-------------------- end of arguments ---------------------
|
| 570 |
+
INFO:megatron.core.num_microbatches_calculator:setting number of microbatches to constant 2
|
| 571 |
+
> building GPT2BPETokenizer tokenizer ...
|
| 572 |
+
WARNING: one_logger package is required to enable e2e metrics tracking. please go to https://confluence.nvidia.com/display/MLWFO/Package+Repositories for details to install it
|
| 573 |
+
> padded vocab (size: 50257) with 47 dummy tokens (new size: 50304)
|
| 574 |
+
WARNING:megatron.core.rerun_state_machine:RerunStateMachine initialized in mode RerunMode.VALIDATE_RESULTS
|
| 575 |
+
> initializing torch distributed ...
|
| 576 |
+
[Gloo] Rank 1 is connected to 7 peer ranks. Expected number of connected peer ranks is : 7
|
| 577 |
+
[Gloo] Rank 0 is connected to 7 peer ranks. Expected number of connected peer ranks is : 7
|
| 578 |
+
[Gloo] Rank 2 is connected to 7 peer ranks. Expected number of connected peer ranks is : 7
|
| 579 |
+
[Gloo] Rank 6 is connected to 7 peer ranks. Expected number of connected peer ranks is : 7
|
| 580 |
+
[Gloo] Rank 3 is connected to 7 peer ranks. Expected number of connected peer ranks is : 7
|
| 581 |
+
[Gloo] Rank 4 is connected to 7 peer ranks. Expected number of connected peer ranks is : 7
|
| 582 |
+
[Gloo] Rank 5 is connected to 7 peer ranks. Expected number of connected peer ranks is : 7
|
| 583 |
+
[Gloo] Rank 7 is connected to 7 peer ranks. Expected number of connected peer ranks is : 7
|
| 584 |
+
[Gloo] Rank [Gloo] Rank 0[Gloo] Rank [Gloo] Rank [Gloo] Rank [Gloo] Rank 2 is connected to 135 is connected to 74 is connected to is connected to is connected to 7 peer ranks. is connected to 777 peer ranks. Expected number of connected peer ranks is : 7 peer ranks. [Gloo] Rank peer ranks. peer ranks. Expected number of connected peer ranks is : 7 peer ranks. Expected number of connected peer ranks is : Expected number of connected peer ranks is : 6Expected number of connected peer ranks is : 7
|
| 585 |
+
Expected number of connected peer ranks is : 77
|
| 586 |
+
is connected to 7
|
| 587 |
+
7[Gloo] Rank
|
| 588 |
+
7
|
| 589 |
+
|
| 590 |
+
7 peer ranks. is connected to Expected number of connected peer ranks is : 77 peer ranks.
|
| 591 |
+
Expected number of connected peer ranks is : 7
|
| 592 |
+
[Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
|
| 593 |
+
> initialized tensor model parallel with size 1
|
| 594 |
+
> initialized pipeline model parallel with size 1
|
| 595 |
+
> setting random seeds to 1234 ...
|
| 596 |
+
[Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
|
| 597 |
+
[Gloo] Rank [Gloo] Rank 00 is connected to is connected to 0[Gloo] Rank 0 peer ranks. 0 peer ranks. Expected number of connected peer ranks is : is connected to Expected number of connected peer ranks is : 000
|
| 598 |
+
peer ranks.
|
| 599 |
+
Expected number of connected peer ranks is : 0
|
| 600 |
+
[Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
|
| 601 |
+
[Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
|
| 602 |
+
[Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
|
| 603 |
+
[rank3]:[W830 06:48:38.699027510 ProcessGroupNCCL.cpp:4789] [PG ID 0 PG GUID 0 Rank 3] using GPU 3 as device used by this process is currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect. You can specify device_id in init_process_group() to force use of a particular device.
|
| 604 |
+
[rank7]:[W830 06:48:38.699065652 ProcessGroupNCCL.cpp:4789] [PG ID 0 PG GUID 0 Rank 7] using GPU 7 as device used by this process is currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect. You can specify device_id in init_process_group() to force use of a particular device.
|
| 605 |
+
[rank5]:[W830 06:48:38.699066866 ProcessGroupNCCL.cpp:4789] [PG ID 0 PG GUID 0 Rank 5] using GPU 5 as device used by this process is currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect. You can specify device_id in init_process_group() to force use of a particular device.
|
| 606 |
+
[rank1]:[W830 06:48:38.699091271 ProcessGroupNCCL.cpp:4789] [PG ID 0 PG GUID 0 Rank 1] using GPU 1 as device used by this process is currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect. You can specify device_id in init_process_group() to force use of a particular device.
|
| 607 |
+
[rank6]:[W830 06:48:38.699130162 ProcessGroupNCCL.cpp:4789] [PG ID 0 PG GUID 0 Rank 6] using GPU 6 as device used by this process is currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect. You can specify device_id in init_process_group() to force use of a particular device.
|
| 608 |
+
[rank2]:[W830 06:48:38.699135530 ProcessGroupNCCL.cpp:4789] [PG ID 0 PG GUID 0 Rank 2] using GPU 2 as device used by this process is currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect. You can specify device_id in init_process_group() to force use of a particular device.
|
| 609 |
+
[rank4]:[W830 06:48:38.699166898 ProcessGroupNCCL.cpp:4789] [PG ID 0 PG GUID 0 Rank 4] using GPU 4 as device used by this process is currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect. You can specify device_id in init_process_group() to force use of a particular device.
|
| 610 |
+
> compiling dataset index builder ...
|
| 611 |
+
make: Entering directory '/workspace/Megatron-MoE-Echo/megatron/core/datasets'
|
| 612 |
+
make: Nothing to be done for 'default'.
|
| 613 |
+
make: Leaving directory '/workspace/Megatron-MoE-Echo/megatron/core/datasets'
|
| 614 |
+
>>> done with dataset index builder. Compilation time: 0.036 seconds
|
| 615 |
+
WARNING: constraints for invoking optimized fused softmax kernel are not met. We default back to unfused kernel invocations.
|
| 616 |
+
> compiling and loading fused kernels ...
|
| 617 |
+
[rank0]:[W830 06:48:38.859570340 ProcessGroupNCCL.cpp:4789] [PG ID 0 PG GUID 0 Rank 0] using GPU 0 as device used by this process is currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect. You can specify device_id in init_process_group() to force use of a particular device.
|
| 618 |
+
>>> done with compiling and loading fused kernels. Compilation time: 3.752 seconds
|
| 619 |
+
time to initialize megatron (seconds): 7.214
|
| 620 |
+
[after megatron is initialized] datetime: 2026-08-30 06:48:44
|
| 621 |
+
building GPT model ...
|
| 622 |
+
/usr/local/lib/python3.12/dist-packages/transformer_engine/pytorch/cpu_offload.py:708: DeprecationWarning: Offloading weights is deprecated. Using offload_weights=True does not have any effect.
|
| 623 |
+
warnings.warn(
|
| 624 |
+
/usr/local/lib/python3.12/dist-packages/transformer_engine/pytorch/cpu_offload.py:708: DeprecationWarning: Offloading weights is deprecated. Using offload_weights=True does not have any effect.
|
| 625 |
+
warnings.warn(
|
| 626 |
+
/usr/local/lib/python3.12/dist-packages/transformer_engine/pytorch/cpu_offload.py:708: DeprecationWarning: Offloading weights is deprecated. Using offload_weights=True does not have any effect.
|
| 627 |
+
warnings.warn(
|
| 628 |
+
/usr/local/lib/python3.12/dist-packages/transformer_engine/pytorch/cpu_offload.py:708: DeprecationWarning: Offloading weights is deprecated. Using offload_weights=True does not have any effect.
|
| 629 |
+
warnings.warn(
|
| 630 |
+
/usr/local/lib/python3.12/dist-packages/transformer_engine/pytorch/cpu_offload.py:708: DeprecationWarning: Offloading weights is deprecated. Using offload_weights=True does not have any effect.
|
| 631 |
+
warnings.warn(
|
| 632 |
+
/usr/local/lib/python3.12/dist-packages/transformer_engine/pytorch/cpu_offload.py:708: DeprecationWarning: Offloading weights is deprecated. Using offload_weights=True does not have any effect.
|
| 633 |
+
warnings.warn(
|
| 634 |
+
/usr/local/lib/python3.12/dist-packages/transformer_engine/pytorch/cpu_offload.py:708: DeprecationWarning: Offloading weights is deprecated. Using offload_weights=True does not have any effect.
|
| 635 |
+
warnings.warn(
|
| 636 |
+
/usr/local/lib/python3.12/dist-packages/transformer_engine/pytorch/cpu_offload.py:708: DeprecationWarning: Offloading weights is deprecated. Using offload_weights=True does not have any effect.
|
| 637 |
+
warnings.warn(
|
| 638 |
+
> number of parameters on (tensor, pipeline) model parallel rank (0, 0): 3903983616
|
| 639 |
+
INFO:megatron.core.distributed.distributed_data_parallel:Setting up DistributedDataParallel with config DistributedDataParallelConfig(grad_reduce_in_fp32=True, overlap_grad_reduce=True, overlap_param_gather=True, align_param_gather=False, use_distributed_optimizer=True, num_distributed_optimizer_instances=1, check_for_nan_in_grad=False, check_for_large_grads=False, bucket_size=40000000, pad_buckets_for_high_nccl_busbw=False, average_in_collective=False, fp8_param_gather=False, reuse_grad_buf_for_mxfp8_param_ag=False, use_custom_fsdp=False, data_parallel_sharding_strategy='no_shard', gradient_reduce_div_fusion=True, suggested_communication_unit_size=None, preserve_fp32_weights=True, keep_fp8_transpose_cache_when_using_custom_fsdp=False, nccl_ub=False, fsdp_double_buffer=False)
|
| 640 |
+
INFO:megatron.core.distributed.param_and_grad_buffer:Number of buckets for gradient all-reduce / reduce-scatter: 18
|
| 641 |
+
Params for bucket 1 (206045184 elements, 206045184 padded size):
|
| 642 |
+
module.output_layer.weight
|
| 643 |
+
Params for bucket 2 (42086400 elements, 42086400 padded size):
|
| 644 |
+
module.decoder.final_layernorm.weight
|
| 645 |
+
module.decoder.layers.15.self_attention.linear_qkv.weight
|
| 646 |
+
module.decoder.layers.15.pre_mlp_layernorm.weight
|
| 647 |
+
module.decoder.layers.15.mlp.router.weight
|
| 648 |
+
module.decoder.layers.15.self_attention.linear_qkv.layer_norm_weight
|
| 649 |
+
module.decoder.layers.15.self_attention.linear_proj.weight
|
| 650 |
+
Params for bucket 3 (42082304 elements, 42082304 padded size):
|
| 651 |
+
module.decoder.layers.14.self_attention.linear_qkv.weight
|
| 652 |
+
module.decoder.layers.14.pre_mlp_layernorm.weight
|
| 653 |
+
module.decoder.layers.14.mlp.router.weight
|
| 654 |
+
module.decoder.layers.14.self_attention.linear_proj.weight
|
| 655 |
+
module.decoder.layers.14.self_attention.linear_qkv.layer_norm_weight
|
| 656 |
+
Params for bucket 4 (42082304 elements, 42082304 padded size):
|
| 657 |
+
module.decoder.layers.13.self_attention.linear_qkv.weight
|
| 658 |
+
module.decoder.layers.13.pre_mlp_layernorm.weight
|
| 659 |
+
module.decoder.layers.13.mlp.router.weight
|
| 660 |
+
module.decoder.layers.13.self_attention.linear_qkv.layer_norm_weight
|
| 661 |
+
module.decoder.layers.13.self_attention.linear_proj.weight
|
| 662 |
+
Params for bucket 5 (42082304 elements, 42082304 padded size):
|
| 663 |
+
module.decoder.layers.12.pre_mlp_layernorm.weight
|
| 664 |
+
module.decoder.layers.12.self_attention.linear_qkv.weight
|
| 665 |
+
module.decoder.layers.12.mlp.router.weight
|
| 666 |
+
module.decoder.layers.12.self_attention.linear_proj.weight
|
| 667 |
+
module.decoder.layers.12.self_attention.linear_qkv.layer_norm_weight
|
| 668 |
+
Params for bucket 6 (42082304 elements, 42082304 padded size):
|
| 669 |
+
module.decoder.layers.11.self_attention.linear_qkv.weight
|
| 670 |
+
module.decoder.layers.11.pre_mlp_layernorm.weight
|
| 671 |
+
module.decoder.layers.11.mlp.router.weight
|
| 672 |
+
module.decoder.layers.11.self_attention.linear_qkv.layer_norm_weight
|
| 673 |
+
module.decoder.layers.11.self_attention.linear_proj.weight
|
| 674 |
+
Params for bucket 7 (42082304 elements, 42082304 padded size):
|
| 675 |
+
module.decoder.layers.10.mlp.router.weight
|
| 676 |
+
module.decoder.layers.10.self_attention.linear_qkv.layer_norm_weight
|
| 677 |
+
module.decoder.layers.10.self_attention.linear_proj.weight
|
| 678 |
+
module.decoder.layers.10.self_attention.linear_qkv.weight
|
| 679 |
+
module.decoder.layers.10.pre_mlp_layernorm.weight
|
| 680 |
+
Params for bucket 8 (42082304 elements, 42082304 padded size):
|
| 681 |
+
module.decoder.layers.9.self_attention.linear_qkv.weight
|
| 682 |
+
module.decoder.layers.9.pre_mlp_layernorm.weight
|
| 683 |
+
module.decoder.layers.9.mlp.router.weight
|
| 684 |
+
module.decoder.layers.9.self_attention.linear_proj.weight
|
| 685 |
+
module.decoder.layers.9.self_attention.linear_qkv.layer_norm_weight
|
| 686 |
+
Params for bucket 9 (42082304 elements, 42082304 padded size):
|
| 687 |
+
module.decoder.layers.8.mlp.router.weight
|
| 688 |
+
module.decoder.layers.8.self_attention.linear_proj.weight
|
| 689 |
+
module.decoder.layers.8.self_attention.linear_qkv.layer_norm_weight
|
| 690 |
+
module.decoder.layers.8.pre_mlp_layernorm.weight
|
| 691 |
+
module.decoder.layers.8.self_attention.linear_qkv.weight
|
| 692 |
+
Params for bucket 10 (42082304 elements, 42082304 padded size):
|
| 693 |
+
module.decoder.layers.7.pre_mlp_layernorm.weight
|
| 694 |
+
module.decoder.layers.7.self_attention.linear_qkv.weight
|
| 695 |
+
module.decoder.layers.7.mlp.router.weight
|
| 696 |
+
module.decoder.layers.7.self_attention.linear_proj.weight
|
| 697 |
+
module.decoder.layers.7.self_attention.linear_qkv.layer_norm_weight
|
| 698 |
+
Params for bucket 11 (42082304 elements, 42082304 padded size):
|
| 699 |
+
module.decoder.layers.6.mlp.router.weight
|
| 700 |
+
module.decoder.layers.6.self_attention.linear_proj.weight
|
| 701 |
+
module.decoder.layers.6.self_attention.linear_qkv.layer_norm_weight
|
| 702 |
+
module.decoder.layers.6.self_attention.linear_qkv.weight
|
| 703 |
+
module.decoder.layers.6.pre_mlp_layernorm.weight
|
| 704 |
+
Params for bucket 12 (42082304 elements, 42082304 padded size):
|
| 705 |
+
module.decoder.layers.5.mlp.router.weight
|
| 706 |
+
module.decoder.layers.5.self_attention.linear_qkv.layer_norm_weight
|
| 707 |
+
module.decoder.layers.5.self_attention.linear_qkv.weight
|
| 708 |
+
module.decoder.layers.5.self_attention.linear_proj.weight
|
| 709 |
+
module.decoder.layers.5.pre_mlp_layernorm.weight
|
| 710 |
+
Params for bucket 13 (42082304 elements, 42082304 padded size):
|
| 711 |
+
module.decoder.layers.4.self_attention.linear_qkv.weight
|
| 712 |
+
module.decoder.layers.4.pre_mlp_layernorm.weight
|
| 713 |
+
module.decoder.layers.4.mlp.router.weight
|
| 714 |
+
module.decoder.layers.4.self_attention.linear_proj.weight
|
| 715 |
+
module.decoder.layers.4.self_attention.linear_qkv.layer_norm_weight
|
| 716 |
+
Params for bucket 14 (42082304 elements, 42082304 padded size):
|
| 717 |
+
module.decoder.layers.3.self_attention.linear_qkv.weight
|
| 718 |
+
module.decoder.layers.3.pre_mlp_layernorm.weight
|
| 719 |
+
module.decoder.layers.3.mlp.router.weight
|
| 720 |
+
module.decoder.layers.3.self_attention.linear_qkv.layer_norm_weight
|
| 721 |
+
module.decoder.layers.3.self_attention.linear_proj.weight
|
| 722 |
+
Params for bucket 15 (42082304 elements, 42082304 padded size):
|
| 723 |
+
module.decoder.layers.2.mlp.router.weight
|
| 724 |
+
module.decoder.layers.2.self_attention.linear_qkv.layer_norm_weight
|
| 725 |
+
module.decoder.layers.2.self_attention.linear_proj.weight
|
| 726 |
+
module.decoder.layers.2.self_attention.linear_qkv.weight
|
| 727 |
+
module.decoder.layers.2.pre_mlp_layernorm.weight
|
| 728 |
+
Params for bucket 16 (42082304 elements, 42082304 padded size):
|
| 729 |
+
module.decoder.layers.1.self_attention.linear_qkv.layer_norm_weight
|
| 730 |
+
module.decoder.layers.1.self_attention.linear_proj.weight
|
| 731 |
+
module.decoder.layers.1.pre_mlp_layernorm.weight
|
| 732 |
+
module.decoder.layers.1.mlp.router.weight
|
| 733 |
+
module.decoder.layers.1.self_attention.linear_qkv.weight
|
| 734 |
+
Params for bucket 17 (42082304 elements, 42082304 padded size):
|
| 735 |
+
module.decoder.layers.0.self_attention.linear_qkv.layer_norm_weight
|
| 736 |
+
module.decoder.layers.0.pre_mlp_layernorm.weight
|
| 737 |
+
module.decoder.layers.0.self_attention.linear_proj.weight
|
| 738 |
+
module.decoder.layers.0.mlp.router.weight
|
| 739 |
+
module.decoder.layers.0.self_attention.linear_qkv.weight
|
| 740 |
+
Params for bucket 18 (206045184 elements, 206045184 padded size):
|
| 741 |
+
module.embedding.word_embeddings.weight
|
| 742 |
+
INFO:megatron.core.distributed.param_and_grad_buffer:Number of buckets for gradient all-reduce / reduce-scatter: 56
|
| 743 |
+
Params for bucket 1 (44040192 elements, 44040192 padded size):
|
| 744 |
+
module.decoder.layers.15.mlp.experts.linear_fc2.weight3
|
| 745 |
+
module.decoder.layers.15.mlp.experts.linear_fc2.weight1
|
| 746 |
+
module.decoder.layers.15.mlp.experts.linear_fc2.weight2
|
| 747 |
+
Params for bucket 2 (44040192 elements, 44040192 padded size):
|
| 748 |
+
module.decoder.layers.15.mlp.experts.linear_fc2.weight0
|
| 749 |
+
module.decoder.layers.15.mlp.experts.linear_fc1.weight3
|
| 750 |
+
Params for bucket 3 (58720256 elements, 58720256 padded size):
|
| 751 |
+
module.decoder.layers.15.mlp.experts.linear_fc1.weight2
|
| 752 |
+
module.decoder.layers.15.mlp.experts.linear_fc1.weight1
|
| 753 |
+
Params for bucket 4 (44040192 elements, 44040192 padded size):
|
| 754 |
+
module.decoder.layers.15.mlp.experts.linear_fc1.weight0
|
| 755 |
+
module.decoder.layers.14.mlp.experts.linear_fc2.weight3
|
| 756 |
+
Params for bucket 5 (44040192 elements, 44040192 padded size):
|
| 757 |
+
module.decoder.layers.14.mlp.experts.linear_fc2.weight2
|
| 758 |
+
module.decoder.layers.14.mlp.experts.linear_fc2.weight1
|
| 759 |
+
module.decoder.layers.14.mlp.experts.linear_fc2.weight0
|
| 760 |
+
Params for bucket 6 (58720256 elements, 58720256 padded size):
|
| 761 |
+
module.decoder.layers.14.mlp.experts.linear_fc1.weight3
|
| 762 |
+
module.decoder.layers.14.mlp.experts.linear_fc1.weight2
|
| 763 |
+
Params for bucket 7 (58720256 elements, 58720256 padded size):
|
| 764 |
+
module.decoder.layers.14.mlp.experts.linear_fc1.weight1
|
| 765 |
+
module.decoder.layers.14.mlp.experts.linear_fc1.weight0
|
| 766 |
+
Params for bucket 8 (44040192 elements, 44040192 padded size):
|
| 767 |
+
module.decoder.layers.13.mlp.experts.linear_fc2.weight3
|
| 768 |
+
module.decoder.layers.13.mlp.experts.linear_fc2.weight1
|
| 769 |
+
module.decoder.layers.13.mlp.experts.linear_fc2.weight2
|
| 770 |
+
Params for bucket 9 (44040192 elements, 44040192 padded size):
|
| 771 |
+
module.decoder.layers.13.mlp.experts.linear_fc2.weight0
|
| 772 |
+
module.decoder.layers.13.mlp.experts.linear_fc1.weight3
|
| 773 |
+
Params for bucket 10 (58720256 elements, 58720256 padded size):
|
| 774 |
+
module.decoder.layers.13.mlp.experts.linear_fc1.weight2
|
| 775 |
+
module.decoder.layers.13.mlp.experts.linear_fc1.weight1
|
| 776 |
+
Params for bucket 11 (44040192 elements, 44040192 padded size):
|
| 777 |
+
module.decoder.layers.13.mlp.experts.linear_fc1.weight0
|
| 778 |
+
module.decoder.layers.12.mlp.experts.linear_fc2.weight3
|
| 779 |
+
Params for bucket 12 (44040192 elements, 44040192 padded size):
|
| 780 |
+
module.decoder.layers.12.mlp.experts.linear_fc2.weight2
|
| 781 |
+
module.decoder.layers.12.mlp.experts.linear_fc2.weight0
|
| 782 |
+
module.decoder.layers.12.mlp.experts.linear_fc2.weight1
|
| 783 |
+
Params for bucket 13 (58720256 elements, 58720256 padded size):
|
| 784 |
+
module.decoder.layers.12.mlp.experts.linear_fc1.weight3
|
| 785 |
+
module.decoder.layers.12.mlp.experts.linear_fc1.weight2
|
| 786 |
+
Params for bucket 14 (58720256 elements, 58720256 padded size):
|
| 787 |
+
module.decoder.layers.12.mlp.experts.linear_fc1.weight1
|
| 788 |
+
module.decoder.layers.12.mlp.experts.linear_fc1.weight0
|
| 789 |
+
Params for bucket 15 (44040192 elements, 44040192 padded size):
|
| 790 |
+
module.decoder.layers.11.mlp.experts.linear_fc2.weight3
|
| 791 |
+
module.decoder.layers.11.mlp.experts.linear_fc2.weight2
|
| 792 |
+
module.decoder.layers.11.mlp.experts.linear_fc2.weight1
|
| 793 |
+
Params for bucket 16 (44040192 elements, 44040192 padded size):
|
| 794 |
+
module.decoder.layers.11.mlp.experts.linear_fc2.weight0
|
| 795 |
+
module.decoder.layers.11.mlp.experts.linear_fc1.weight3
|
| 796 |
+
Params for bucket 17 (58720256 elements, 58720256 padded size):
|
| 797 |
+
module.decoder.layers.11.mlp.experts.linear_fc1.weight2
|
| 798 |
+
module.decoder.layers.11.mlp.experts.linear_fc1.weight1
|
| 799 |
+
Params for bucket 18 (44040192 elements, 44040192 padded size):
|
| 800 |
+
module.decoder.layers.11.mlp.experts.linear_fc1.weight0
|
| 801 |
+
module.decoder.layers.10.mlp.experts.linear_fc2.weight3
|
| 802 |
+
Params for bucket 19 (44040192 elements, 44040192 padded size):
|
| 803 |
+
module.decoder.layers.10.mlp.experts.linear_fc2.weight2
|
| 804 |
+
module.decoder.layers.10.mlp.experts.linear_fc2.weight1
|
| 805 |
+
module.decoder.layers.10.mlp.experts.linear_fc2.weight0
|
| 806 |
+
Params for bucket 20 (58720256 elements, 58720256 padded size):
|
| 807 |
+
module.decoder.layers.10.mlp.experts.linear_fc1.weight3
|
| 808 |
+
module.decoder.layers.10.mlp.experts.linear_fc1.weight2
|
| 809 |
+
Params for bucket 21 (58720256 elements, 58720256 padded size):
|
| 810 |
+
module.decoder.layers.10.mlp.experts.linear_fc1.weight1
|
| 811 |
+
module.decoder.layers.10.mlp.experts.linear_fc1.weight0
|
| 812 |
+
Params for bucket 22 (44040192 elements, 44040192 padded size):
|
| 813 |
+
module.decoder.layers.9.mlp.experts.linear_fc2.weight3
|
| 814 |
+
module.decoder.layers.9.mlp.experts.linear_fc2.weight1
|
| 815 |
+
module.decoder.layers.9.mlp.experts.linear_fc2.weight2
|
| 816 |
+
Params for bucket 23 (44040192 elements, 44040192 padded size):
|
| 817 |
+
module.decoder.layers.9.mlp.experts.linear_fc2.weight0
|
| 818 |
+
module.decoder.layers.9.mlp.experts.linear_fc1.weight3
|
| 819 |
+
Params for bucket 24 (58720256 elements, 58720256 padded size):
|
| 820 |
+
module.decoder.layers.9.mlp.experts.linear_fc1.weight2
|
| 821 |
+
module.decoder.layers.9.mlp.experts.linear_fc1.weight1
|
| 822 |
+
Params for bucket 25 (44040192 elements, 44040192 padded size):
|
| 823 |
+
module.decoder.layers.9.mlp.experts.linear_fc1.weight0
|
| 824 |
+
module.decoder.layers.8.mlp.experts.linear_fc2.weight3
|
| 825 |
+
Params for bucket 26 (44040192 elements, 44040192 padded size):
|
| 826 |
+
module.decoder.layers.8.mlp.experts.linear_fc2.weight2
|
| 827 |
+
module.decoder.layers.8.mlp.experts.linear_fc2.weight1
|
| 828 |
+
module.decoder.layers.8.mlp.experts.linear_fc2.weight0
|
| 829 |
+
Params for bucket 27 (58720256 elements, 58720256 padded size):
|
| 830 |
+
module.decoder.layers.8.mlp.experts.linear_fc1.weight3
|
| 831 |
+
module.decoder.layers.8.mlp.experts.linear_fc1.weight2
|
| 832 |
+
Params for bucket 28 (58720256 elements, 58720256 padded size):
|
| 833 |
+
module.decoder.layers.8.mlp.experts.linear_fc1.weight1
|
| 834 |
+
module.decoder.layers.8.mlp.experts.linear_fc1.weight0
|
| 835 |
+
Params for bucket 29 (44040192 elements, 44040192 padded size):
|
| 836 |
+
module.decoder.layers.7.mlp.experts.linear_fc2.weight3
|
| 837 |
+
module.decoder.layers.7.mlp.experts.linear_fc2.weight2
|
| 838 |
+
module.decoder.layers.7.mlp.experts.linear_fc2.weight1
|
| 839 |
+
Params for bucket 30 (44040192 elements, 44040192 padded size):
|
| 840 |
+
module.decoder.layers.7.mlp.experts.linear_fc2.weight0
|
| 841 |
+
module.decoder.layers.7.mlp.experts.linear_fc1.weight3
|
| 842 |
+
Params for bucket 31 (58720256 elements, 58720256 padded size):
|
| 843 |
+
module.decoder.layers.7.mlp.experts.linear_fc1.weight2
|
| 844 |
+
module.decoder.layers.7.mlp.experts.linear_fc1.weight1
|
| 845 |
+
Params for bucket 32 (44040192 elements, 44040192 padded size):
|
| 846 |
+
module.decoder.layers.7.mlp.experts.linear_fc1.weight0
|
| 847 |
+
module.decoder.layers.6.mlp.experts.linear_fc2.weight3
|
| 848 |
+
Params for bucket 33 (44040192 elements, 44040192 padded size):
|
| 849 |
+
module.decoder.layers.6.mlp.experts.linear_fc2.weight2
|
| 850 |
+
module.decoder.layers.6.mlp.experts.linear_fc2.weight0
|
| 851 |
+
module.decoder.layers.6.mlp.experts.linear_fc2.weight1
|
| 852 |
+
Params for bucket 34 (58720256 elements, 58720256 padded size):
|
| 853 |
+
module.decoder.layers.6.mlp.experts.linear_fc1.weight3
|
| 854 |
+
module.decoder.layers.6.mlp.experts.linear_fc1.weight2
|
| 855 |
+
Params for bucket 35 (58720256 elements, 58720256 padded size):
|
| 856 |
+
module.decoder.layers.6.mlp.experts.linear_fc1.weight1
|
| 857 |
+
module.decoder.layers.6.mlp.experts.linear_fc1.weight0
|
| 858 |
+
Params for bucket 36 (44040192 elements, 44040192 padded size):
|
| 859 |
+
module.decoder.layers.5.mlp.experts.linear_fc2.weight3
|
| 860 |
+
module.decoder.layers.5.mlp.experts.linear_fc2.weight1
|
| 861 |
+
module.decoder.layers.5.mlp.experts.linear_fc2.weight2
|
| 862 |
+
Params for bucket 37 (44040192 elements, 44040192 padded size):
|
| 863 |
+
module.decoder.layers.5.mlp.experts.linear_fc2.weight0
|
| 864 |
+
module.decoder.layers.5.mlp.experts.linear_fc1.weight3
|
| 865 |
+
Params for bucket 38 (58720256 elements, 58720256 padded size):
|
| 866 |
+
module.decoder.layers.5.mlp.experts.linear_fc1.weight2
|
| 867 |
+
module.decoder.layers.5.mlp.experts.linear_fc1.weight1
|
| 868 |
+
Params for bucket 39 (44040192 elements, 44040192 padded size):
|
| 869 |
+
module.decoder.layers.5.mlp.experts.linear_fc1.weight0
|
| 870 |
+
module.decoder.layers.4.mlp.experts.linear_fc2.weight3
|
| 871 |
+
Params for bucket 40 (44040192 elements, 44040192 padded size):
|
| 872 |
+
module.decoder.layers.4.mlp.experts.linear_fc2.weight2
|
| 873 |
+
module.decoder.layers.4.mlp.experts.linear_fc2.weight1
|
| 874 |
+
module.decoder.layers.4.mlp.experts.linear_fc2.weight0
|
| 875 |
+
Params for bucket 41 (58720256 elements, 58720256 padded size):
|
| 876 |
+
module.decoder.layers.4.mlp.experts.linear_fc1.weight3
|
| 877 |
+
module.decoder.layers.4.mlp.experts.linear_fc1.weight2
|
| 878 |
+
Params for bucket 42 (58720256 elements, 58720256 padded size):
|
| 879 |
+
module.decoder.layers.4.mlp.experts.linear_fc1.weight1
|
| 880 |
+
module.decoder.layers.4.mlp.experts.linear_fc1.weight0
|
| 881 |
+
Params for bucket 43 (44040192 elements, 44040192 padded size):
|
| 882 |
+
module.decoder.layers.3.mlp.experts.linear_fc2.weight3
|
| 883 |
+
module.decoder.layers.3.mlp.experts.linear_fc2.weight2
|
| 884 |
+
module.decoder.layers.3.mlp.experts.linear_fc2.weight1
|
| 885 |
+
Params for bucket 44 (44040192 elements, 44040192 padded size):
|
| 886 |
+
module.decoder.layers.3.mlp.experts.linear_fc2.weight0
|
| 887 |
+
module.decoder.layers.3.mlp.experts.linear_fc1.weight3
|
| 888 |
+
Params for bucket 45 (58720256 elements, 58720256 padded size):
|
| 889 |
+
module.decoder.layers.3.mlp.experts.linear_fc1.weight2
|
| 890 |
+
module.decoder.layers.3.mlp.experts.linear_fc1.weight1
|
| 891 |
+
Params for bucket 46 (44040192 elements, 44040192 padded size):
|
| 892 |
+
module.decoder.layers.3.mlp.experts.linear_fc1.weight0
|
| 893 |
+
module.decoder.layers.2.mlp.experts.linear_fc2.weight3
|
| 894 |
+
Params for bucket 47 (44040192 elements, 44040192 padded size):
|
| 895 |
+
module.decoder.layers.2.mlp.experts.linear_fc2.weight2
|
| 896 |
+
module.decoder.layers.2.mlp.experts.linear_fc2.weight1
|
| 897 |
+
module.decoder.layers.2.mlp.experts.linear_fc2.weight0
|
| 898 |
+
Params for bucket 48 (58720256 elements, 58720256 padded size):
|
| 899 |
+
module.decoder.layers.2.mlp.experts.linear_fc1.weight3
|
| 900 |
+
module.decoder.layers.2.mlp.experts.linear_fc1.weight2
|
| 901 |
+
Params for bucket 49 (58720256 elements, 58720256 padded size):
|
| 902 |
+
module.decoder.layers.2.mlp.experts.linear_fc1.weight1
|
| 903 |
+
module.decoder.layers.2.mlp.experts.linear_fc1.weight0
|
| 904 |
+
Params for bucket 50 (44040192 elements, 44040192 padded size):
|
| 905 |
+
module.decoder.layers.1.mlp.experts.linear_fc2.weight3
|
| 906 |
+
module.decoder.layers.1.mlp.experts.linear_fc2.weight1
|
| 907 |
+
module.decoder.layers.1.mlp.experts.linear_fc2.weight2
|
| 908 |
+
Params for bucket 51 (44040192 elements, 44040192 padded size):
|
| 909 |
+
module.decoder.layers.1.mlp.experts.linear_fc2.weight0
|
| 910 |
+
module.decoder.layers.1.mlp.experts.linear_fc1.weight3
|
| 911 |
+
Params for bucket 52 (58720256 elements, 58720256 padded size):
|
| 912 |
+
module.decoder.layers.1.mlp.experts.linear_fc1.weight2
|
| 913 |
+
module.decoder.layers.1.mlp.experts.linear_fc1.weight1
|
| 914 |
+
Params for bucket 53 (44040192 elements, 44040192 padded size):
|
| 915 |
+
module.decoder.layers.1.mlp.experts.linear_fc1.weight0
|
| 916 |
+
module.decoder.layers.0.mlp.experts.linear_fc2.weight3
|
| 917 |
+
Params for bucket 54 (44040192 elements, 44040192 padded size):
|
| 918 |
+
module.decoder.layers.0.mlp.experts.linear_fc2.weight2
|
| 919 |
+
module.decoder.layers.0.mlp.experts.linear_fc2.weight1
|
| 920 |
+
module.decoder.layers.0.mlp.experts.linear_fc2.weight0
|
| 921 |
+
Params for bucket 55 (58720256 elements, 58720256 padded size):
|
| 922 |
+
module.decoder.layers.0.mlp.experts.linear_fc1.weight3
|
| 923 |
+
module.decoder.layers.0.mlp.experts.linear_fc1.weight2
|
| 924 |
+
Params for bucket 56 (58720256 elements, 58720256 padded size):
|
| 925 |
+
module.decoder.layers.0.mlp.experts.linear_fc1.weight1
|
| 926 |
+
module.decoder.layers.0.mlp.experts.linear_fc1.weight0
|
| 927 |
+
INFO:megatron.core.optimizer:Setting up optimizer with config OptimizerConfig(optimizer='adam', lr=0.0001, min_lr=1e-05, decoupled_lr=None, decoupled_min_lr=None, weight_decay=0.1, fp8_recipe='mxfp8', fp16=False, bf16=True, reuse_grad_buf_for_mxfp8_param_ag=False, params_dtype=torch.bfloat16, use_precision_aware_optimizer=False, store_param_remainders=True, main_grads_dtype=torch.float32, main_params_dtype=torch.float32, exp_avg_dtype=torch.float32, exp_avg_sq_dtype=torch.float32, loss_scale=None, initial_loss_scale=4294967296, min_loss_scale=1.0, loss_scale_window=1000, hysteresis=2, adam_beta1=0.9, adam_beta2=0.999, adam_eps=1e-08, sgd_momentum=0.9, use_distributed_optimizer=True, overlap_param_gather_with_optimizer_step=False, optimizer_cpu_offload=False, optimizer_offload_fraction=1.0, use_torch_optimizer_for_cpu_offload=False, overlap_cpu_optimizer_d2h_h2d=False, pin_cpu_grads=True, pin_cpu_params=True, clip_grad=1.0, log_num_zeros_in_grad=False, barrier_with_L1_time=True, timers=<megatron.core.timers.Timers object at 0x743309117800>, config_logger_dir='')
|
| 928 |
+
INFO:megatron.core.optimizer_param_scheduler:> learning rate decay style: cosine
|
| 929 |
+
[after model, optimizer, and learning rate scheduler are built] datetime: 2026-08-30 06:48:44
|
| 930 |
+
> building train, validation, and test datasets ...
|
| 931 |
+
> datasets target sizes (minimum size):
|
| 932 |
+
train: 192
|
| 933 |
+
validation: 0
|
| 934 |
+
test: 0
|
| 935 |
+
INFO:megatron.core.datasets.blended_megatron_dataset_config:Let split_matrix = [(0, 1.0), None, None]
|
| 936 |
+
> building train, validation, and test datasets for GPT ...
|
| 937 |
+
INFO:megatron.core.datasets.blended_megatron_dataset_builder:Building GPTDataset splits with sizes=(192, 0, 0) and config=GPTDatasetConfig(random_seed=1234, sequence_length=4096, blend=(['/workspace/data/wikitext/wikitext103_gpt2_text_document'], None), blend_per_split=None, split='100,0,0', split_matrix=[(0, 1.0), None, None], num_dataset_builder_threads=1, path_to_cache=None, mmap_bin_files=True, mock=False, tokenizer=<megatron.training.tokenizer.tokenizer._GPT2BPETokenizer object at 0x743308cd3440>, mid_level_dataset_surplus=0.005, reset_position_ids=False, reset_attention_mask=False, eod_mask_loss=False, create_attention_mask=True, drop_last_partial_validation_sequence=True, add_extra_token_to_sequence=True, object_storage_cache_path=None)
|
| 938 |
+
INFO:megatron.core.datasets.indexed_dataset:Load the _IndexReader from /workspace/data/wikitext/wikitext103_gpt2_text_document.idx
|
| 939 |
+
INFO:megatron.core.datasets.indexed_dataset: Extract the sequence lengths
|
| 940 |
+
INFO:megatron.core.datasets.indexed_dataset: Extract the sequence pointers
|
| 941 |
+
INFO:megatron.core.datasets.indexed_dataset: Extract the document indices
|
| 942 |
+
INFO:megatron.core.datasets.indexed_dataset:> total number of sequences: 1165029
|
| 943 |
+
INFO:megatron.core.datasets.indexed_dataset:> total number of documents: 1801350
|
| 944 |
+
INFO:megatron.core.datasets.gpt_dataset:Load the GPTDataset train indices
|
| 945 |
+
INFO:megatron.core.datasets.gpt_dataset: Load the document index from 44ed1ee5bcb9e5b030387b8d27a6c406-GPTDataset-train-document_index.npy
|
| 946 |
+
INFO:megatron.core.datasets.gpt_dataset: Load the sample index from 44ed1ee5bcb9e5b030387b8d27a6c406-GPTDataset-train-sample_index.npy
|
| 947 |
+
INFO:megatron.core.datasets.gpt_dataset: Load the shuffle index from 44ed1ee5bcb9e5b030387b8d27a6c406-GPTDataset-train-shuffle_index.npy
|
| 948 |
+
INFO:megatron.core.datasets.gpt_dataset:> total number of samples: 29073
|
| 949 |
+
> finished creating GPT datasets ...
|
| 950 |
+
[after dataloaders are built] datetime: 2026-08-30 06:48:44
|
| 951 |
+
done with setup ...
|
| 952 |
+
training ...
|
| 953 |
+
Overwriting rerun_state_machine.current_iteration from -1 to 0...
|
| 954 |
+
(min, max) time across ranks (ms):
|
| 955 |
+
model-and-optimizer-setup ......................: (393.41, 395.82)
|
| 956 |
+
train/valid/test-data-iterators-setup ..........: (64.49, 75.15)
|
| 957 |
+
[before the start of training step] datetime: 2026-08-30 06:48:44
|
| 958 |
+
/workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:334: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
|
| 959 |
+
warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
|
| 960 |
+
/workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:336: UserWarning: The use_host_meta is deprecated, it will be removed in the future.
|
| 961 |
+
warnings.warn("The use_host_meta is deprecated, it will be removed in the future.")
|
| 962 |
+
/workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:334: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
|
| 963 |
+
warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
|
| 964 |
+
/workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:336: UserWarning: The use_host_meta is deprecated, it will be removed in the future.
|
| 965 |
+
warnings.warn("The use_host_meta is deprecated, it will be removed in the future.")
|
| 966 |
+
/workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:334: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
|
| 967 |
+
warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
|
| 968 |
+
/workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:334: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
|
| 969 |
+
warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
|
| 970 |
+
/workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:336: UserWarning: The use_host_meta is deprecated, it will be removed in the future.
|
| 971 |
+
warnings.warn("The use_host_meta is deprecated, it will be removed in the future.")
|
| 972 |
+
/workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:336: UserWarning: The use_host_meta is deprecated, it will be removed in the future.
|
| 973 |
+
warnings.warn("The use_host_meta is deprecated, it will be removed in the future.")
|
| 974 |
+
/workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:334: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
|
| 975 |
+
warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
|
| 976 |
+
/workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:334: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
|
| 977 |
+
warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
|
| 978 |
+
/workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:336: UserWarning: The use_host_meta is deprecated, it will be removed in the future.
|
| 979 |
+
warnings.warn("The use_host_meta is deprecated, it will be removed in the future.")
|
| 980 |
+
/workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:336: UserWarning: The use_host_meta is deprecated, it will be removed in the future.
|
| 981 |
+
warnings.warn("The use_host_meta is deprecated, it will be removed in the future.")
|
| 982 |
+
/workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:334: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
|
| 983 |
+
warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
|
| 984 |
+
/workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:336: UserWarning: The use_host_meta is deprecated, it will be removed in the future.
|
| 985 |
+
warnings.warn("The use_host_meta is deprecated, it will be removed in the future.")
|
| 986 |
+
/workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:334: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
|
| 987 |
+
warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
|
| 988 |
+
/workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:336: UserWarning: The use_host_meta is deprecated, it will be removed in the future.
|
| 989 |
+
warnings.warn("The use_host_meta is deprecated, it will be removed in the future.")
|
| 990 |
+
/workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:482: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
|
| 991 |
+
warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
|
| 992 |
+
/workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:482: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
|
| 993 |
+
warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
|
| 994 |
+
/workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:482: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
|
| 995 |
+
warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
|
| 996 |
+
/workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:482: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
|
| 997 |
+
warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
|
| 998 |
+
/workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:482: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
|
| 999 |
+
warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
|
| 1000 |
+
/workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:482: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
|
| 1001 |
+
warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
|
| 1002 |
+
/workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:482: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
|
| 1003 |
+
warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
|
| 1004 |
+
/workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:482: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
|
| 1005 |
+
warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
|
| 1006 |
+
/workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:334: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
|
| 1007 |
+
warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
|
| 1008 |
+
/workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:336: UserWarning: The use_host_meta is deprecated, it will be removed in the future.
|
| 1009 |
+
warnings.warn("The use_host_meta is deprecated, it will be removed in the future.")
|
| 1010 |
+
/workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:334: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
|
| 1011 |
+
warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
|
| 1012 |
+
/workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:336: UserWarning: The use_host_meta is deprecated, it will be removed in the future.
|
| 1013 |
+
warnings.warn("The use_host_meta is deprecated, it will be removed in the future.")
|
| 1014 |
+
/workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:334: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
|
| 1015 |
+
warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
|
| 1016 |
+
/workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:336: UserWarning: The use_host_meta is deprecated, it will be removed in the future.
|
| 1017 |
+
warnings.warn("The use_host_meta is deprecated, it will be removed in the future.")
|
| 1018 |
+
/workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:334: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
|
| 1019 |
+
warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
|
| 1020 |
+
/workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:336: UserWarning: The use_host_meta is deprecated, it will be removed in the future.
|
| 1021 |
+
warnings.warn("The use_host_meta is deprecated, it will be removed in the future.")
|
| 1022 |
+
/workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:334: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
|
| 1023 |
+
warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
|
| 1024 |
+
/workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:336: UserWarning: The use_host_meta is deprecated, it will be removed in the future.
|
| 1025 |
+
warnings.warn("The use_host_meta is deprecated, it will be removed in the future.")
|
| 1026 |
+
/workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:334: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
|
| 1027 |
+
warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
|
| 1028 |
+
/workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:336: UserWarning: The use_host_meta is deprecated, it will be removed in the future.
|
| 1029 |
+
warnings.warn("The use_host_meta is deprecated, it will be removed in the future.")
|
| 1030 |
+
/workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:334: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
|
| 1031 |
+
warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
|
| 1032 |
+
/workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:336: UserWarning: The use_host_meta is deprecated, it will be removed in the future.
|
| 1033 |
+
warnings.warn("The use_host_meta is deprecated, it will be removed in the future.")
|
| 1034 |
+
/workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:334: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
|
| 1035 |
+
warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
|
| 1036 |
+
/workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:336: UserWarning: The use_host_meta is deprecated, it will be removed in the future.
|
| 1037 |
+
warnings.warn("The use_host_meta is deprecated, it will be removed in the future.")
|
| 1038 |
+
/workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:482: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
|
| 1039 |
+
warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
|
| 1040 |
+
/workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:482: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
|
| 1041 |
+
warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
|
| 1042 |
+
/workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:482: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
|
| 1043 |
+
warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
|
| 1044 |
+
/workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:482: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
|
| 1045 |
+
warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
|
| 1046 |
+
/workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:482: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
|
| 1047 |
+
warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
|
| 1048 |
+
/workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:482: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
|
| 1049 |
+
warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
|
| 1050 |
+
/workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:482: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
|
| 1051 |
+
warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
|
| 1052 |
+
/workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:482: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
|
| 1053 |
+
warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
|
| 1054 |
+
/workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:334: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
|
| 1055 |
+
warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
|
| 1056 |
+
/workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:334: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
|
| 1057 |
+
warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
|
| 1058 |
+
/workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:334: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
|
| 1059 |
+
warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
|
| 1060 |
+
/workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:334: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
|
| 1061 |
+
warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
|
| 1062 |
+
/workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:334: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
|
| 1063 |
+
warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
|
| 1064 |
+
/workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:334: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
|
| 1065 |
+
warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
|
| 1066 |
+
/workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:334: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
|
| 1067 |
+
warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
|
| 1068 |
+
/workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:334: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
|
| 1069 |
+
warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
|
| 1070 |
+
/workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:336: UserWarning: The use_host_meta is deprecated, it will be removed in the future.
|
| 1071 |
+
warnings.warn("The use_host_meta is deprecated, it will be removed in the future.")
|
| 1072 |
+
/workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:336: UserWarning: The use_host_meta is deprecated, it will be removed in the future.
|
| 1073 |
+
warnings.warn("The use_host_meta is deprecated, it will be removed in the future.")
|
| 1074 |
+
/workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:336: UserWarning: The use_host_meta is deprecated, it will be removed in the future.
|
| 1075 |
+
warnings.warn("The use_host_meta is deprecated, it will be removed in the future.")
|
| 1076 |
+
/workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:336: UserWarning: The use_host_meta is deprecated, it will be removed in the future.
|
| 1077 |
+
warnings.warn("The use_host_meta is deprecated, it will be removed in the future.")
|
| 1078 |
+
/workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:336: UserWarning: The use_host_meta is deprecated, it will be removed in the future.
|
| 1079 |
+
warnings.warn("The use_host_meta is deprecated, it will be removed in the future.")
|
| 1080 |
+
/workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:336: UserWarning: The use_host_meta is deprecated, it will be removed in the future.
|
| 1081 |
+
warnings.warn("The use_host_meta is deprecated, it will be removed in the future.")
|
| 1082 |
+
/workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:336: UserWarning: The use_host_meta is deprecated, it will be removed in the future.
|
| 1083 |
+
warnings.warn("The use_host_meta is deprecated, it will be removed in the future.")
|
| 1084 |
+
/workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:336: UserWarning: The use_host_meta is deprecated, it will be removed in the future.
|
| 1085 |
+
warnings.warn("The use_host_meta is deprecated, it will be removed in the future.")
|
| 1086 |
+
[2026-08-30 06:49:23] iteration 1/ 12 | consumed samples: 16 | elapsed time per iteration (ms): 38387.8 | throughput per GPU (TFLOP/s/GPU): 8.7 | learning rate: 2.000000E-05 | global batch size: 16 | lm loss: 1.102614E+01 | loss scale: 1.0 | grad norm: 44.313 | number of skipped iterations: 0 | number of nan iterations: 0 |
|
| 1087 |
+
Number of parameters in transformer block in billions: 23.22
|
| 1088 |
+
Number of parameters in embedding layers in billions: 0.41
|
| 1089 |
+
Total number of parameters in billions: 23.63
|
| 1090 |
+
Number of parameters in most loaded shard in billions: 23.6320
|
| 1091 |
+
Theoretical memory footprints: weight and optimizer=169029.43 MB
|
| 1092 |
+
[Rank 0] (after 1 iterations) memory (MB) | allocated: 63654.662109375 | max allocated: 85586.1083984375 | reserved: 88178.0 | max reserved: 88178.0
|
| 1093 |
+
[2026-08-30 06:49:23] iteration 2/ 12 | consumed samples: 32 | elapsed time per iteration (ms): 638.2 | throughput per GPU (TFLOP/s/GPU): 522.4 | learning rate: 4.000000E-05 | global batch size: 16 | lm loss: 1.101619E+01 | loss scale: 1.0 | grad norm: 45.749 | number of skipped iterations: 0 | number of nan iterations: 0 |
|
| 1094 |
+
[2026-08-30 06:49:24] iteration 3/ 12 | consumed samples: 48 | elapsed time per iteration (ms): 514.0 | throughput per GPU (TFLOP/s/GPU): 648.6 | learning rate: 6.000000E-05 | global batch size: 16 | lm loss: 1.032369E+01 | loss scale: 1.0 | grad norm: 204.565 | number of skipped iterations: 0 | number of nan iterations: 0 |
|
| 1095 |
+
Capture CUDA graph for training!!!
|
| 1096 |
+
CUDA graph capture done!!!
|
| 1097 |
+
[2026-08-30 06:49:27] iteration 4/ 12 | consumed samples: 64 | elapsed time per iteration (ms): 3146.3 | throughput per GPU (TFLOP/s/GPU): 106.0 | learning rate: 8.000000E-05 | global batch size: 16 | lm loss: 1.318609E+01 | loss scale: 1.0 | grad norm: 34.258 | number of skipped iterations: 0 | number of nan iterations: 0 |
|
| 1098 |
+
[2026-08-30 06:49:27] iteration 5/ 12 | consumed samples: 80 | elapsed time per iteration (ms): 415.8 | throughput per GPU (TFLOP/s/GPU): 801.7 | learning rate: 1.000000E-04 | global batch size: 16 | lm loss: 1.039552E+01 | loss scale: 1.0 | grad norm: 16.801 | number of skipped iterations: 0 | number of nan iterations: 0 |
|
| 1099 |
+
[2026-08-30 06:49:28] iteration 6/ 12 | consumed samples: 96 | elapsed time per iteration (ms): 415.6 | throughput per GPU (TFLOP/s/GPU): 802.2 | learning rate: 7.750000E-05 | global batch size: 16 | lm loss: 1.038954E+01 | loss scale: 1.0 | grad norm: 31.805 | number of skipped iterations: 0 | number of nan iterations: 0 |
|
| 1100 |
+
[2026-08-30 06:49:28] iteration 7/ 12 | consumed samples: 112 | elapsed time per iteration (ms): 416.1 | throughput per GPU (TFLOP/s/GPU): 801.3 | learning rate: 3.250000E-05 | global batch size: 16 | lm loss: 1.009632E+01 | loss scale: 1.0 | grad norm: 7.209 | number of skipped iterations: 0 | number of nan iterations: 0 |
|
| 1101 |
+
[2026-08-30 06:49:29] iteration 8/ 12 | consumed samples: 128 | elapsed time per iteration (ms): 415.6 | throughput per GPU (TFLOP/s/GPU): 802.2 | learning rate: 1.000000E-05 | global batch size: 16 | lm loss: 9.533159E+00 | loss scale: 1.0 | grad norm: 5.801 | number of skipped iterations: 0 | number of nan iterations: 0 |
|
| 1102 |
+
[2026-08-30 06:49:29] iteration 9/ 12 | consumed samples: 144 | elapsed time per iteration (ms): 417.4 | throughput per GPU (TFLOP/s/GPU): 798.7 | learning rate: 1.000000E-05 | global batch size: 16 | lm loss: 9.255018E+00 | loss scale: 1.0 | grad norm: 5.701 | number of skipped iterations: 0 | number of nan iterations: 0 |
|
| 1103 |
+
[2026-08-30 06:49:30] iteration 10/ 12 | consumed samples: 160 | elapsed time per iteration (ms): 413.7 | throughput per GPU (TFLOP/s/GPU): 805.9 | learning rate: 1.000000E-05 | global batch size: 16 | lm loss: 9.088298E+00 | loss scale: 1.0 | grad norm: 5.506 | number of skipped iterations: 0 | number of nan iterations: 0 |
|
| 1104 |
+
[2026-08-30 06:49:30] iteration 11/ 12 | consumed samples: 176 | elapsed time per iteration (ms): 415.5 | throughput per GPU (TFLOP/s/GPU): 802.3 | learning rate: 1.000000E-05 | global batch size: 16 | lm loss: 8.956705E+00 | loss scale: 1.0 | grad norm: 5.285 | number of skipped iterations: 0 | number of nan iterations: 0 |
|
| 1105 |
+
[2026-08-30 06:49:30] iteration 12/ 12 | consumed samples: 192 | elapsed time per iteration (ms): 415.3 | throughput per GPU (TFLOP/s/GPU): 802.8 | learning rate: 1.000000E-05 | global batch size: 16 | lm loss: 8.814399E+00 | loss scale: 1.0 | grad norm: 5.220 | number of skipped iterations: 0 | number of nan iterations: 0 |
|
| 1106 |
+
[after training is done] datetime: 2026-08-30 06:49:30
|
| 1107 |
+
[rank0]:[W830 06:49:32.380633503 ProcessGroupNCCL.cpp:1505] Warning: WARNING: destroy_process_group() was not called before program exit, which can leak resources. For more info, please see https://pytorch.org/docs/stable/distributed.html#shutdown (function operator())
|
tiera-rerun-fixed-moonep/echo/a2/a2_matched_gbs16/G2__echo__matched__seq4096__gbs16__cap1p08203125/rep2/mem_samples.csv
ADDED
|
@@ -0,0 +1,184 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
0, 0
|
| 2 |
+
1, 0
|
| 3 |
+
2, 0
|
| 4 |
+
3, 0
|
| 5 |
+
4, 0
|
| 6 |
+
5, 0
|
| 7 |
+
6, 0
|
| 8 |
+
7, 0
|
| 9 |
+
0, 0
|
| 10 |
+
1, 0
|
| 11 |
+
2, 0
|
| 12 |
+
3, 0
|
| 13 |
+
4, 0
|
| 14 |
+
5, 0
|
| 15 |
+
6, 0
|
| 16 |
+
7, 0
|
| 17 |
+
0, 1
|
| 18 |
+
1, 1
|
| 19 |
+
2, 1
|
| 20 |
+
3, 1
|
| 21 |
+
4, 1
|
| 22 |
+
5, 1
|
| 23 |
+
6, 1
|
| 24 |
+
7, 1
|
| 25 |
+
0, 4
|
| 26 |
+
1, 4
|
| 27 |
+
2, 4
|
| 28 |
+
3, 4
|
| 29 |
+
4, 1
|
| 30 |
+
5, 1
|
| 31 |
+
6, 1
|
| 32 |
+
7, 1
|
| 33 |
+
0, 4
|
| 34 |
+
1, 4
|
| 35 |
+
2, 4
|
| 36 |
+
3, 4
|
| 37 |
+
4, 4
|
| 38 |
+
5, 4
|
| 39 |
+
6, 4
|
| 40 |
+
7, 4
|
| 41 |
+
0, 821
|
| 42 |
+
1, 819
|
| 43 |
+
2, 819
|
| 44 |
+
3, 819
|
| 45 |
+
4, 821
|
| 46 |
+
5, 819
|
| 47 |
+
6, 821
|
| 48 |
+
7, 821
|
| 49 |
+
0, 1537
|
| 50 |
+
1, 1537
|
| 51 |
+
2, 1537
|
| 52 |
+
3, 1537
|
| 53 |
+
4, 1537
|
| 54 |
+
5, 1537
|
| 55 |
+
6, 1537
|
| 56 |
+
7, 1537
|
| 57 |
+
0, 2231
|
| 58 |
+
1, 2231
|
| 59 |
+
2, 2231
|
| 60 |
+
3, 2231
|
| 61 |
+
4, 2231
|
| 62 |
+
5, 2231
|
| 63 |
+
6, 2231
|
| 64 |
+
7, 2231
|
| 65 |
+
0, 39471
|
| 66 |
+
1, 39477
|
| 67 |
+
2, 39443
|
| 68 |
+
3, 39459
|
| 69 |
+
4, 39497
|
| 70 |
+
5, 39463
|
| 71 |
+
6, 39453
|
| 72 |
+
7, 39471
|
| 73 |
+
0, 39719
|
| 74 |
+
1, 39711
|
| 75 |
+
2, 39707
|
| 76 |
+
3, 39701
|
| 77 |
+
4, 39723
|
| 78 |
+
5, 39693
|
| 79 |
+
6, 39743
|
| 80 |
+
7, 39735
|
| 81 |
+
0, 42297
|
| 82 |
+
1, 42289
|
| 83 |
+
2, 42289
|
| 84 |
+
3, 42289
|
| 85 |
+
4, 42289
|
| 86 |
+
5, 42289
|
| 87 |
+
6, 42289
|
| 88 |
+
7, 42289
|
| 89 |
+
0, 42541
|
| 90 |
+
1, 42533
|
| 91 |
+
2, 42533
|
| 92 |
+
3, 42533
|
| 93 |
+
4, 42533
|
| 94 |
+
5, 42533
|
| 95 |
+
6, 42533
|
| 96 |
+
7, 42533
|
| 97 |
+
0, 42541
|
| 98 |
+
1, 42533
|
| 99 |
+
2, 42533
|
| 100 |
+
3, 42533
|
| 101 |
+
4, 42533
|
| 102 |
+
5, 42533
|
| 103 |
+
6, 42533
|
| 104 |
+
7, 42533
|
| 105 |
+
0, 43457
|
| 106 |
+
1, 43449
|
| 107 |
+
2, 43449
|
| 108 |
+
3, 43449
|
| 109 |
+
4, 43449
|
| 110 |
+
5, 43449
|
| 111 |
+
6, 43449
|
| 112 |
+
7, 43449
|
| 113 |
+
0, 43735
|
| 114 |
+
1, 43727
|
| 115 |
+
2, 43727
|
| 116 |
+
3, 43727
|
| 117 |
+
4, 43727
|
| 118 |
+
5, 43727
|
| 119 |
+
6, 43727
|
| 120 |
+
7, 43727
|
| 121 |
+
0, 45167
|
| 122 |
+
1, 45159
|
| 123 |
+
2, 45159
|
| 124 |
+
3, 45159
|
| 125 |
+
4, 45159
|
| 126 |
+
5, 45159
|
| 127 |
+
6, 45159
|
| 128 |
+
7, 45159
|
| 129 |
+
0, 45167
|
| 130 |
+
1, 45159
|
| 131 |
+
2, 45159
|
| 132 |
+
3, 45159
|
| 133 |
+
4, 45159
|
| 134 |
+
5, 45159
|
| 135 |
+
6, 45159
|
| 136 |
+
7, 45159
|
| 137 |
+
0, 75747
|
| 138 |
+
1, 75739
|
| 139 |
+
2, 75739
|
| 140 |
+
3, 75739
|
| 141 |
+
4, 75739
|
| 142 |
+
5, 75739
|
| 143 |
+
6, 75739
|
| 144 |
+
7, 75739
|
| 145 |
+
0, 76233
|
| 146 |
+
1, 76225
|
| 147 |
+
2, 76225
|
| 148 |
+
3, 76225
|
| 149 |
+
4, 76225
|
| 150 |
+
5, 76225
|
| 151 |
+
6, 76225
|
| 152 |
+
7, 76225
|
| 153 |
+
0, 76233
|
| 154 |
+
1, 76225
|
| 155 |
+
2, 76225
|
| 156 |
+
3, 76225
|
| 157 |
+
4, 76225
|
| 158 |
+
5, 76225
|
| 159 |
+
6, 76225
|
| 160 |
+
7, 76225
|
| 161 |
+
0, 76233
|
| 162 |
+
1, 76225
|
| 163 |
+
2, 76225
|
| 164 |
+
3, 76225
|
| 165 |
+
4, 76225
|
| 166 |
+
5, 76225
|
| 167 |
+
6, 76225
|
| 168 |
+
7, 76225
|
| 169 |
+
0, 94063
|
| 170 |
+
1, 94037
|
| 171 |
+
2, 94037
|
| 172 |
+
3, 94073
|
| 173 |
+
4, 94063
|
| 174 |
+
5, 94053
|
| 175 |
+
6, 94043
|
| 176 |
+
7, 94055
|
| 177 |
+
0, 94871
|
| 178 |
+
1, 94859
|
| 179 |
+
2, 94821
|
| 180 |
+
3, 94895
|
| 181 |
+
4, 94933
|
| 182 |
+
5, 94957
|
| 183 |
+
6, 94957
|
| 184 |
+
7, 94957
|
tiera-rerun-fixed-moonep/echo/a2/a2_matched_gbs16/G2__echo__matched__seq4096__gbs16__cap1p08203125/rep2/run.env
ADDED
|
@@ -0,0 +1,70 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
backend=echo
|
| 2 |
+
config=G2
|
| 3 |
+
family=tierA
|
| 4 |
+
repo=/workspace/Megatron-MoE-Echo
|
| 5 |
+
branch=bracing-regions-through-kernels
|
| 6 |
+
commit=9a236793
|
| 7 |
+
num_layers=16
|
| 8 |
+
hidden=4096
|
| 9 |
+
ffn_hidden=14336
|
| 10 |
+
moe_ffn_hidden=3584
|
| 11 |
+
num_experts=32
|
| 12 |
+
topk=8
|
| 13 |
+
experts_per_rank=4
|
| 14 |
+
expert_params=44040192
|
| 15 |
+
heads=32
|
| 16 |
+
kv_groups=8
|
| 17 |
+
kv_channels=
|
| 18 |
+
qk_layernorm=0
|
| 19 |
+
shared_ffn=0
|
| 20 |
+
shared_expert_overlap=0
|
| 21 |
+
moe_token_dispatcher=flex
|
| 22 |
+
mla=0
|
| 23 |
+
dense_first_layers=0
|
| 24 |
+
moe_layers=16
|
| 25 |
+
rotary_base=1000000
|
| 26 |
+
rotary_scaling=
|
| 27 |
+
qkv_bias=0
|
| 28 |
+
router_balancing=aux_loss
|
| 29 |
+
aux_loss_coeff=1e-2
|
| 30 |
+
router_pre_softmax=0
|
| 31 |
+
router_fusion=0
|
| 32 |
+
router_dtype=fp32
|
| 33 |
+
device_initiated_grouped_gemm=1
|
| 34 |
+
norm_epsilon=1e-5
|
| 35 |
+
seq_length=4096
|
| 36 |
+
micro_batch_size=1
|
| 37 |
+
global_batch_size=16
|
| 38 |
+
num_microbatches=2
|
| 39 |
+
assignments_per_rank=32768
|
| 40 |
+
mean_tokens_per_expert=8192
|
| 41 |
+
slot_policy=matched
|
| 42 |
+
slots_per_rank=4
|
| 43 |
+
capacity_mode=static
|
| 44 |
+
capacity=1.08203125
|
| 45 |
+
cuda_graph=1
|
| 46 |
+
probe_mode=0
|
| 47 |
+
async_grad_reduce=0
|
| 48 |
+
branch_role=timing
|
| 49 |
+
memory_dump_mode=none
|
| 50 |
+
memory_snapshot_path=
|
| 51 |
+
cuda_graph_warmup_steps=3
|
| 52 |
+
recompute=none
|
| 53 |
+
force_balance=0
|
| 54 |
+
train_iters=12
|
| 55 |
+
ep_size=8
|
| 56 |
+
echo_dispatch_overlap=1
|
| 57 |
+
echo_dispatch_recompute=0
|
| 58 |
+
ultraep_quota_min_tokens=
|
| 59 |
+
ultraep_weight_sync_join=
|
| 60 |
+
profile_enable=0
|
| 61 |
+
kernel_trace=0
|
| 62 |
+
fp8_mode=mxfp8
|
| 63 |
+
fp8_recipe=mxfp8
|
| 64 |
+
training_seed=1234
|
| 65 |
+
learning_rate=1e-4
|
| 66 |
+
min_learning_rate=1.0e-5
|
| 67 |
+
lr_warmup_iters=5
|
| 68 |
+
state_dump_enable=0
|
| 69 |
+
state_dump_dir=
|
| 70 |
+
moonep_assert_no_pending_reduce=0
|
tiera-rerun-fixed-moonep/echo/a2/a2_matched_gbs16/G2__echo__matched__seq4096__gbs16__cap1p08203125/rep2/train.log
ADDED
|
@@ -0,0 +1,1085 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
==============================================================
|
| 2 |
+
G2 / echo (slots/rank=4 policy=matched)
|
| 3 |
+
E=32 K=8 H'=3584 m=4
|
| 4 |
+
L=16 seq=4096 gbs=16 (2 microbatches)
|
| 5 |
+
cap_mode=static cap=1.08203125 cg=1 recompute=none
|
| 6 |
+
memory_dump=none snapshots=/workspace/tiera-rerun-fixed-moonep/echo/a2/a2_matched_gbs16/G2__echo__matched__seq4096__gbs16__cap1p08203125/rep2/memory_snapshots/snapshot.pickle
|
| 7 |
+
repo=/workspace/Megatron-MoE-Echo @ 9a236793
|
| 8 |
+
==============================================================
|
| 9 |
+
W0830 06:49:40.874000 665587 torch/distributed/run.py:766]
|
| 10 |
+
W0830 06:49:40.874000 665587 torch/distributed/run.py:766] *****************************************
|
| 11 |
+
W0830 06:49:40.874000 665587 torch/distributed/run.py:766] Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed.
|
| 12 |
+
W0830 06:49:40.874000 665587 torch/distributed/run.py:766] *****************************************
|
| 13 |
+
/workspace/Megatron-MoE-Echo/megatron/training/arguments.py:831: UserWarning: Disabling sequence parallelism because tensor model parallelism is disabled
|
| 14 |
+
warnings.warn("Disabling sequence parallelism because tensor model parallelism is disabled")
|
| 15 |
+
WARNING: one_logger package is required to enable e2e metrics tracking. please go to https://confluence.nvidia.com/display/MLWFO/Package+Repositories for details to install it
|
| 16 |
+
/workspace/Megatron-MoE-Echo/megatron/training/arguments.py:831: UserWarning: Disabling sequence parallelism because tensor model parallelism is disabled
|
| 17 |
+
warnings.warn("Disabling sequence parallelism because tensor model parallelism is disabled")
|
| 18 |
+
using world size: 8, data-parallel size: 8, context-parallel size: 1, hierarchical context-parallel sizes: None, tensor-model-parallel size: 1, pipeline-model-parallel size: 1
|
| 19 |
+
WARNING: overriding default arguments for tokenizer_type:GPT2BPETokenizer with tokenizer_type:GPT2BPETokenizer
|
| 20 |
+
Number of virtual stages per pipeline stage: None
|
| 21 |
+
accumulate and all-reduce gradients in fp32 for bfloat16 data type.
|
| 22 |
+
using torch.bfloat16 for parameters ...
|
| 23 |
+
/workspace/Megatron-MoE-Echo/megatron/training/arguments.py:831: UserWarning: Disabling sequence parallelism because tensor model parallelism is disabled
|
| 24 |
+
warnings.warn("Disabling sequence parallelism because tensor model parallelism is disabled")
|
| 25 |
+
------------------------ arguments ------------------------
|
| 26 |
+
account_for_embedding_in_pipeline_split ......... False
|
| 27 |
+
account_for_loss_in_pipeline_split .............. False
|
| 28 |
+
accumulate_allreduce_grads_in_fp32 .............. True
|
| 29 |
+
adam_beta1 ...................................... 0.9
|
| 30 |
+
adam_beta2 ...................................... 0.999
|
| 31 |
+
adam_eps ........................................ 1e-08
|
| 32 |
+
add_bias_linear ................................. False
|
| 33 |
+
add_position_embedding .......................... False
|
| 34 |
+
add_qkv_bias .................................... False
|
| 35 |
+
adlr_autoresume ................................. False
|
| 36 |
+
adlr_autoresume_interval ........................ 1000
|
| 37 |
+
align_grad_reduce ............................... True
|
| 38 |
+
align_param_gather .............................. False
|
| 39 |
+
app_tag_run_name ................................ None
|
| 40 |
+
app_tag_run_version ............................. 0.0.0
|
| 41 |
+
apply_layernorm_1p .............................. False
|
| 42 |
+
apply_query_key_layer_scaling ................... False
|
| 43 |
+
apply_residual_connection_post_layernorm ........ False
|
| 44 |
+
apply_rope_fusion ............................... True
|
| 45 |
+
async_save ...................................... None
|
| 46 |
+
async_tensor_model_parallel_allreduce ........... True
|
| 47 |
+
attention_backend ............................... AttnBackend.auto
|
| 48 |
+
attention_dropout ............................... 0.0
|
| 49 |
+
attention_softmax_in_fp32 ....................... False
|
| 50 |
+
auto_detect_ckpt_format ......................... False
|
| 51 |
+
barrier_with_L1_time ............................ True
|
| 52 |
+
bert_binary_head ................................ True
|
| 53 |
+
bert_embedder_type .............................. megatron
|
| 54 |
+
bert_load ....................................... None
|
| 55 |
+
bf16 ............................................ True
|
| 56 |
+
bias_dropout_fusion ............................. True
|
| 57 |
+
bias_gelu_fusion ................................ False
|
| 58 |
+
bias_swiglu_fusion .............................. True
|
| 59 |
+
biencoder_projection_dim ........................ 0
|
| 60 |
+
biencoder_shared_query_context_model ............ False
|
| 61 |
+
block_data_path ................................. None
|
| 62 |
+
calc_ft_timeouts ................................ False
|
| 63 |
+
calculate_per_token_loss ........................ False
|
| 64 |
+
check_for_large_grads ........................... False
|
| 65 |
+
check_for_nan_in_loss_and_grad .................. False
|
| 66 |
+
check_for_spiky_loss ............................ False
|
| 67 |
+
check_weight_hash_across_dp_replicas_interval ... None
|
| 68 |
+
ckpt_assume_constant_structure .................. False
|
| 69 |
+
ckpt_convert_format ............................. None
|
| 70 |
+
ckpt_convert_save ............................... None
|
| 71 |
+
ckpt_convert_update_legacy_dist_opt_format ...... False
|
| 72 |
+
ckpt_format ..................................... torch_dist
|
| 73 |
+
ckpt_fully_parallel_load ........................ False
|
| 74 |
+
ckpt_fully_parallel_save ........................ True
|
| 75 |
+
ckpt_fully_parallel_save_deprecated ............. False
|
| 76 |
+
ckpt_step ....................................... None
|
| 77 |
+
classes_fraction ................................ 1.0
|
| 78 |
+
clip_grad ....................................... 1.0
|
| 79 |
+
clone_scatter_output_in_embedding ............... True
|
| 80 |
+
config_logger_dir ...............................
|
| 81 |
+
consumed_train_samples .......................... 0
|
| 82 |
+
consumed_valid_samples .......................... 0
|
| 83 |
+
context_parallel_size ........................... 1
|
| 84 |
+
cp_comm_type .................................... ['p2p']
|
| 85 |
+
create_attention_mask_in_dataloader ............. True
|
| 86 |
+
cross_entropy_fusion_impl ....................... native
|
| 87 |
+
cross_entropy_loss_fusion ....................... False
|
| 88 |
+
cuda_graph_scope ................................ full_iteration
|
| 89 |
+
cuda_graph_warmup_steps ......................... 3
|
| 90 |
+
data_args_path .................................. None
|
| 91 |
+
data_cache_path ................................. None
|
| 92 |
+
data_parallel_random_init ....................... False
|
| 93 |
+
data_parallel_sharding_strategy ................. no_shard
|
| 94 |
+
data_parallel_size .............................. 8
|
| 95 |
+
data_path ....................................... ['/workspace/data/wikitext/wikitext103_gpt2_text_document']
|
| 96 |
+
data_per_class_fraction ......................... 1.0
|
| 97 |
+
data_sharding ................................... True
|
| 98 |
+
dataloader_type ................................. single
|
| 99 |
+
ddp_average_in_collective ....................... False
|
| 100 |
+
ddp_bucket_size ................................. None
|
| 101 |
+
ddp_num_buckets ................................. None
|
| 102 |
+
ddp_pad_buckets_for_high_nccl_busbw ............. False
|
| 103 |
+
decoder_first_pipeline_num_layers ............... None
|
| 104 |
+
decoder_last_pipeline_num_layers ................ None
|
| 105 |
+
decoder_num_layers .............................. None
|
| 106 |
+
decoder_seq_length .............................. None
|
| 107 |
+
decoupled_lr .................................... None
|
| 108 |
+
decoupled_min_lr ................................ None
|
| 109 |
+
decrease_batch_size_if_needed ................... False
|
| 110 |
+
defer_embedding_wgrad_compute ................... False
|
| 111 |
+
delay_wgrad_compute ............................. False
|
| 112 |
+
deprecated_use_mcore_models ..................... True
|
| 113 |
+
deterministic_mode .............................. False
|
| 114 |
+
dino_bottleneck_size ............................ 256
|
| 115 |
+
dino_freeze_last_layer .......................... 1
|
| 116 |
+
dino_head_hidden_size ........................... 2048
|
| 117 |
+
dino_local_crops_number ......................... 10
|
| 118 |
+
dino_local_img_size ............................. 96
|
| 119 |
+
dino_norm_last_layer ............................ False
|
| 120 |
+
dino_teacher_temp ............................... 0.07
|
| 121 |
+
dino_warmup_teacher_temp ........................ 0.04
|
| 122 |
+
dino_warmup_teacher_temp_epochs ................. 30
|
| 123 |
+
disable_bf16_reduced_precision_matmul ........... False
|
| 124 |
+
disable_mamba_mem_eff_path ...................... False
|
| 125 |
+
disable_straggler_on_startup .................... False
|
| 126 |
+
dist_ckpt_format_deprecated ..................... None
|
| 127 |
+
dist_ckpt_strictness ............................ assume_ok_unexpected
|
| 128 |
+
distribute_saved_activations .................... False
|
| 129 |
+
distributed_backend ............................. nccl
|
| 130 |
+
distributed_timeout_minutes ..................... 10
|
| 131 |
+
embedding_init_method_std ....................... None
|
| 132 |
+
embedding_path .................................. None
|
| 133 |
+
empty_unused_memory_level ....................... 0
|
| 134 |
+
enable_cuda_graph ............................... True
|
| 135 |
+
enable_experimental ............................. False
|
| 136 |
+
enable_ft_package ............................... False
|
| 137 |
+
enable_gloo_process_groups ...................... True
|
| 138 |
+
enable_msc ...................................... True
|
| 139 |
+
enable_one_logger ............................... True
|
| 140 |
+
encoder_num_layers .............................. 16
|
| 141 |
+
encoder_seq_length .............................. 4096
|
| 142 |
+
end_weight_decay ................................ 0.1
|
| 143 |
+
eod_mask_loss ................................... False
|
| 144 |
+
error_injection_rate ............................ 0
|
| 145 |
+
error_injection_type ............................ transient_error
|
| 146 |
+
eval_interval ................................... 1000000
|
| 147 |
+
eval_iters ...................................... 0
|
| 148 |
+
evidence_data_path .............................. None
|
| 149 |
+
exit_duration_in_mins ........................... None
|
| 150 |
+
exit_interval ................................... None
|
| 151 |
+
exit_on_missing_checkpoint ...................... False
|
| 152 |
+
exit_signal_handler ............................. False
|
| 153 |
+
exp_avg_dtype ................................... torch.float32
|
| 154 |
+
exp_avg_sq_dtype ................................ torch.float32
|
| 155 |
+
expert_model_parallel_size ...................... 8
|
| 156 |
+
expert_tensor_parallel_size ..................... 1
|
| 157 |
+
export_force_local_attention .................... False
|
| 158 |
+
export_kd_cfg ................................... None
|
| 159 |
+
export_kd_teacher_ckpt_format ................... None
|
| 160 |
+
export_kd_teacher_load .......................... None
|
| 161 |
+
export_kv_cache_quant ........................... False
|
| 162 |
+
export_legacy_megatron .......................... False
|
| 163 |
+
export_model_type ............................... GPTModel
|
| 164 |
+
export_moe_apply_probs_on_input ................. False
|
| 165 |
+
export_qk_l2_norm ............................... False
|
| 166 |
+
export_quant_cfg ................................ None
|
| 167 |
+
export_real_quant_cfg ........................... None
|
| 168 |
+
export_te_mcore_model ........................... False
|
| 169 |
+
external_cuda_graph ............................. False
|
| 170 |
+
ffn_hidden_size ................................. 14336
|
| 171 |
+
finetune ........................................ False
|
| 172 |
+
finetune_data_split ............................. train
|
| 173 |
+
finetune_hf_dataset ............................. None
|
| 174 |
+
first_last_layers_bf16 .......................... False
|
| 175 |
+
flash_decode .................................... False
|
| 176 |
+
fp16 ............................................ False
|
| 177 |
+
fp16_lm_cross_entropy ........................... False
|
| 178 |
+
fp32_residual_connection ........................ False
|
| 179 |
+
fp8 ............................................. e4m3
|
| 180 |
+
fp8_amax_compute_algo ........................... most_recent
|
| 181 |
+
fp8_amax_history_len ............................ 1
|
| 182 |
+
fp8_interval .................................... 1
|
| 183 |
+
fp8_margin ...................................... 0
|
| 184 |
+
fp8_param_gather ................................ False
|
| 185 |
+
fp8_recipe ...................................... mxfp8
|
| 186 |
+
fp8_wgrad ....................................... True
|
| 187 |
+
fsdp_double_buffer .............................. False
|
| 188 |
+
global_batch_size ............................... 16
|
| 189 |
+
grad_reduce_in_bf16 ............................. False
|
| 190 |
+
gradient_accumulation_fusion .................... True
|
| 191 |
+
gradient_reduce_div_fusion ...................... True
|
| 192 |
+
group_query_attention ........................... True
|
| 193 |
+
head_lr_mult .................................... 1.0
|
| 194 |
+
heterogeneous_layers_config_encoded_json ........ None
|
| 195 |
+
heterogeneous_layers_config_path ................ None
|
| 196 |
+
hidden_dropout .................................. 0.0
|
| 197 |
+
hidden_size ..................................... 4096
|
| 198 |
+
hierarchical_context_parallel_sizes ............. None
|
| 199 |
+
high_priority_stream_groups ..................... []
|
| 200 |
+
hybrid_attention_ratio .......................... 0.0
|
| 201 |
+
hybrid_mlp_ratio ................................ 0.0
|
| 202 |
+
hybrid_override_pattern ......................... None
|
| 203 |
+
hysteresis ...................................... 2
|
| 204 |
+
ict_head_size ................................... None
|
| 205 |
+
ict_load ........................................ None
|
| 206 |
+
img_h ........................................... 224
|
| 207 |
+
img_w ........................................... 224
|
| 208 |
+
indexer_batch_size .............................. 128
|
| 209 |
+
indexer_log_interval ............................ 1000
|
| 210 |
+
inference_batch_times_seqlen_threshold .......... -1
|
| 211 |
+
inference_dynamic_batching ...................... False
|
| 212 |
+
inference_dynamic_batching_buffer_guaranteed_fraction 0.2
|
| 213 |
+
inference_dynamic_batching_buffer_overflow_factor None
|
| 214 |
+
inference_dynamic_batching_buffer_size_gb ....... 40.0
|
| 215 |
+
inference_dynamic_batching_chunk_size ........... 256
|
| 216 |
+
inference_dynamic_batching_max_requests_override None
|
| 217 |
+
inference_dynamic_batching_max_tokens_override .. None
|
| 218 |
+
inference_dynamic_batching_num_cuda_graphs ...... 16
|
| 219 |
+
inference_max_batch_size ........................ 8
|
| 220 |
+
inference_max_seq_length ........................ 2560
|
| 221 |
+
inference_rng_tracker ........................... False
|
| 222 |
+
init_method_std ................................. 0.01
|
| 223 |
+
init_method_xavier_uniform ...................... False
|
| 224 |
+
init_model_with_meta_device ..................... False
|
| 225 |
+
initial_loss_scale .............................. 4294967296
|
| 226 |
+
inprocess_active_world_size ..................... 8
|
| 227 |
+
inprocess_barrier_timeout ....................... 120
|
| 228 |
+
inprocess_completion_timeout .................... 120
|
| 229 |
+
inprocess_empty_cuda_cache ...................... False
|
| 230 |
+
inprocess_granularity ........................... node
|
| 231 |
+
inprocess_hard_timeout .......................... 90
|
| 232 |
+
inprocess_heartbeat_interval .................... 30
|
| 233 |
+
inprocess_heartbeat_timeout ..................... 60
|
| 234 |
+
inprocess_last_call_wait ........................ 1
|
| 235 |
+
inprocess_max_iterations ........................ None
|
| 236 |
+
inprocess_monitor_process_interval .............. 1.0
|
| 237 |
+
inprocess_monitor_thread_interval ............... 1.0
|
| 238 |
+
inprocess_progress_watchdog_interval ............ 1.0
|
| 239 |
+
inprocess_restart ............................... False
|
| 240 |
+
inprocess_soft_timeout .......................... 60
|
| 241 |
+
inprocess_termination_grace_time ................ 1
|
| 242 |
+
is_hybrid_model ................................. False
|
| 243 |
+
iter_per_epoch .................................. 1250
|
| 244 |
+
iterations_to_skip .............................. []
|
| 245 |
+
keep_fp8_transpose_cache_when_using_custom_fsdp . False
|
| 246 |
+
kitchen_config_file ............................. None
|
| 247 |
+
kitchen_recipe_number ........................... None
|
| 248 |
+
kv_channels ..................................... 128
|
| 249 |
+
kv_lora_rank .................................... 32
|
| 250 |
+
lazy_mpu_init ................................... None
|
| 251 |
+
load ............................................ None
|
| 252 |
+
load_main_params_from_ckpt ...................... None
|
| 253 |
+
load_model_opt_format ........................... False
|
| 254 |
+
local_rank ...................................... 0
|
| 255 |
+
log_energy ...................................... False
|
| 256 |
+
log_interval .................................... 1
|
| 257 |
+
log_loss_scale_to_tensorboard ................... True
|
| 258 |
+
log_memory_to_tensorboard ....................... False
|
| 259 |
+
log_num_zeros_in_grad ........................... False
|
| 260 |
+
log_params_norm ................................. False
|
| 261 |
+
log_progress .................................... False
|
| 262 |
+
log_straggler ................................... False
|
| 263 |
+
log_throughput .................................. True
|
| 264 |
+
log_timers_to_tensorboard ....................... False
|
| 265 |
+
log_validation_ppl_to_tensorboard ............... False
|
| 266 |
+
log_world_size_to_tensorboard ................... False
|
| 267 |
+
logging_level ................................... None
|
| 268 |
+
loss_scale ...................................... None
|
| 269 |
+
loss_scale_window ............................... 1000
|
| 270 |
+
lr .............................................. 0.0001
|
| 271 |
+
lr_decay_iters .................................. 8
|
| 272 |
+
lr_decay_samples ................................ None
|
| 273 |
+
lr_decay_style .................................. cosine
|
| 274 |
+
lr_warmup_fraction .............................. None
|
| 275 |
+
lr_warmup_init .................................. 0.0
|
| 276 |
+
lr_warmup_iters ................................. 5
|
| 277 |
+
lr_warmup_samples ............................... 0
|
| 278 |
+
lr_wsd_decay_iters .............................. None
|
| 279 |
+
lr_wsd_decay_samples ............................ None
|
| 280 |
+
lr_wsd_decay_style .............................. exponential
|
| 281 |
+
main_grads_dtype ................................ torch.float32
|
| 282 |
+
main_params_dtype ............................... torch.float32
|
| 283 |
+
make_vocab_size_divisible_by .................... 128
|
| 284 |
+
mamba_head_dim .................................. 64
|
| 285 |
+
mamba_num_groups ................................ 8
|
| 286 |
+
mamba_num_heads ................................. None
|
| 287 |
+
mamba_state_dim ................................. 128
|
| 288 |
+
manual_gc ....................................... False
|
| 289 |
+
manual_gc_eval .................................. True
|
| 290 |
+
manual_gc_interval .............................. 0
|
| 291 |
+
mask_factor ..................................... 1.0
|
| 292 |
+
mask_prob ....................................... 0.15
|
| 293 |
+
mask_type ....................................... random
|
| 294 |
+
masked_softmax_fusion ........................... False
|
| 295 |
+
max_position_embeddings ......................... 32768
|
| 296 |
+
max_tokens_to_oom ............................... 12000
|
| 297 |
+
memory_snapshot_path ............................ snapshot.pickle
|
| 298 |
+
merge_file ...................................... /workspace/data/wikitext/gpt2-merges.txt
|
| 299 |
+
micro_batch_size ................................ 1
|
| 300 |
+
microbatch_group_size_per_vp_stage .............. None
|
| 301 |
+
mid_level_dataset_surplus ....................... 0.005
|
| 302 |
+
min_loss_scale .................................. 1.0
|
| 303 |
+
min_lr .......................................... 1e-05
|
| 304 |
+
mlp_chunks_for_prefill .......................... 1
|
| 305 |
+
mmap_bin_files .................................. True
|
| 306 |
+
mock_data ....................................... False
|
| 307 |
+
moe_apply_probs_on_input ........................ False
|
| 308 |
+
moe_aux_loss_coeff .............................. 0.01
|
| 309 |
+
moe_deepep_num_sms .............................. 20
|
| 310 |
+
moe_echo_enable_random_offloading ............... False
|
| 311 |
+
moe_echo_expert_dispatch_overlap ................ True
|
| 312 |
+
moe_echo_expert_dispatcher_type ................. hybridep
|
| 313 |
+
moe_echo_recompute_expert_dispatch .............. False
|
| 314 |
+
moe_enable_deepep ............................... False
|
| 315 |
+
moe_enable_echo ................................. True
|
| 316 |
+
moe_enable_hybridep ............................. True
|
| 317 |
+
moe_expert_capacity_factor ...................... None
|
| 318 |
+
moe_extended_tp ................................. False
|
| 319 |
+
moe_ffn_hidden_size ............................. 3584
|
| 320 |
+
moe_grouped_gemm ................................ True
|
| 321 |
+
moe_hybridep_num_sms ............................ 16
|
| 322 |
+
moe_input_jitter_eps ............................ None
|
| 323 |
+
moe_layer_freq .................................. 1
|
| 324 |
+
moe_layer_recompute ............................. False
|
| 325 |
+
moe_num_echo_experts ............................ 32
|
| 326 |
+
moe_pad_expert_input_to_capacity ................ False
|
| 327 |
+
moe_per_layer_logging ........................... False
|
| 328 |
+
moe_permute_fusion .............................. False
|
| 329 |
+
moe_received_token_capacity ..................... 1.08203125
|
| 330 |
+
moe_router_bias_update_rate ..................... 0.001
|
| 331 |
+
moe_router_dtype ................................ fp32
|
| 332 |
+
moe_router_enable_expert_bias ................... False
|
| 333 |
+
moe_router_force_load_balancing ................. False
|
| 334 |
+
moe_router_group_topk ........................... None
|
| 335 |
+
moe_router_load_balancing_type .................. aux_loss
|
| 336 |
+
moe_router_num_groups ........................... None
|
| 337 |
+
moe_router_padding_for_fp8 ...................... False
|
| 338 |
+
moe_router_pre_softmax .......................... False
|
| 339 |
+
moe_router_score_function ....................... softmax
|
| 340 |
+
moe_router_topk ................................. 8
|
| 341 |
+
moe_router_topk_scaling_factor .................. None
|
| 342 |
+
moe_shared_expert_intermediate_size ............. None
|
| 343 |
+
moe_shared_expert_overlap ....................... False
|
| 344 |
+
moe_token_dispatcher_type ....................... flex
|
| 345 |
+
moe_token_drop_policy ........................... probs
|
| 346 |
+
moe_upcycling_granularity ....................... 1
|
| 347 |
+
moe_use_device_initiated_grouped_gemm ........... True
|
| 348 |
+
moe_use_legacy_grouped_gemm ..................... False
|
| 349 |
+
moe_use_upcycling ............................... False
|
| 350 |
+
moe_z_loss_coeff ................................ None
|
| 351 |
+
mrope_section ................................... None
|
| 352 |
+
mscale .......................................... 1.0
|
| 353 |
+
mscale_all_dim .................................. 0.0
|
| 354 |
+
mtp_loss_scaling_factor ......................... 0.1
|
| 355 |
+
mtp_num_layers .................................. None
|
| 356 |
+
multi_latent_attention .......................... False
|
| 357 |
+
nccl_all_reduce_for_prefill ..................... False
|
| 358 |
+
nccl_communicator_config_path ................... None
|
| 359 |
+
nccl_ub ......................................... False
|
| 360 |
+
no_load_optim ................................... None
|
| 361 |
+
no_load_rng ..................................... None
|
| 362 |
+
no_persist_layer_norm ........................... False
|
| 363 |
+
no_rope_freq .................................... None
|
| 364 |
+
no_save_optim ................................... None
|
| 365 |
+
no_save_rng ..................................... None
|
| 366 |
+
non_persistent_ckpt_type ........................ None
|
| 367 |
+
non_persistent_global_ckpt_dir .................. None
|
| 368 |
+
non_persistent_local_ckpt_algo .................. fully_parallel
|
| 369 |
+
non_persistent_local_ckpt_dir ................... None
|
| 370 |
+
non_persistent_save_interval .................... None
|
| 371 |
+
norm_epsilon .................................... 1e-05
|
| 372 |
+
normalization ................................... RMSNorm
|
| 373 |
+
num_attention_heads ............................. 32
|
| 374 |
+
num_channels .................................... 3
|
| 375 |
+
num_classes ..................................... 1000
|
| 376 |
+
num_dataset_builder_threads ..................... 1
|
| 377 |
+
num_distributed_optimizer_instances ............. 1
|
| 378 |
+
num_experts ..................................... 32
|
| 379 |
+
num_layers ...................................... 16
|
| 380 |
+
num_layers_at_end_in_bf16 ....................... 1
|
| 381 |
+
num_layers_at_start_in_bf16 ..................... 1
|
| 382 |
+
num_layers_per_virtual_pipeline_stage ........... None
|
| 383 |
+
num_query_groups ................................ 8
|
| 384 |
+
num_virtual_stages_per_pipeline_rank ............ None
|
| 385 |
+
num_workers ..................................... 2
|
| 386 |
+
object_storage_cache_path ....................... None
|
| 387 |
+
one_logger_async ................................ False
|
| 388 |
+
one_logger_project .............................. megatron-lm
|
| 389 |
+
one_logger_run_name ............................. None
|
| 390 |
+
onnx_safe ....................................... None
|
| 391 |
+
openai_gelu ..................................... False
|
| 392 |
+
optimizer ....................................... adam
|
| 393 |
+
optimizer_cpu_offload ........................... False
|
| 394 |
+
optimizer_offload_fraction ...................... 1.0
|
| 395 |
+
output_bert_embeddings .......................... False
|
| 396 |
+
overlap_cpu_optimizer_d2h_h2d ................... False
|
| 397 |
+
overlap_grad_reduce ............................. True
|
| 398 |
+
overlap_moe_expert_parallel_comm ................ False
|
| 399 |
+
overlap_p2p_comm ................................ False
|
| 400 |
+
overlap_p2p_comm_warmup_flush ................... False
|
| 401 |
+
overlap_param_gather ............................ True
|
| 402 |
+
overlap_param_gather_with_optimizer_step ........ False
|
| 403 |
+
override_opt_param_scheduler .................... False
|
| 404 |
+
params_dtype .................................... torch.bfloat16
|
| 405 |
+
patch_dim ....................................... 16
|
| 406 |
+
per_split_data_args_path ........................ None
|
| 407 |
+
perform_initialization .......................... True
|
| 408 |
+
pin_cpu_grads ................................... True
|
| 409 |
+
pin_cpu_params .................................. True
|
| 410 |
+
pipeline_model_parallel_comm_backend ............ None
|
| 411 |
+
pipeline_model_parallel_layout .................. None
|
| 412 |
+
pipeline_model_parallel_size .................... 1
|
| 413 |
+
position_embedding_type ......................... rope
|
| 414 |
+
pretrained_checkpoint ........................... None
|
| 415 |
+
profile ......................................... False
|
| 416 |
+
profile_ranks ................................... [0]
|
| 417 |
+
profile_step_end ................................ 12
|
| 418 |
+
profile_step_start .............................. 10
|
| 419 |
+
q_lora_rank ..................................... None
|
| 420 |
+
qk_head_dim ..................................... 128
|
| 421 |
+
qk_l2_norm ...................................... False
|
| 422 |
+
qk_layernorm .................................... False
|
| 423 |
+
qk_pos_emb_head_dim ............................. 64
|
| 424 |
+
query_in_block_prob ............................. 0.1
|
| 425 |
+
rampup_batch_size ............................... None
|
| 426 |
+
rank ............................................ 0
|
| 427 |
+
recompute_granularity ........................... None
|
| 428 |
+
recompute_method ................................ None
|
| 429 |
+
recompute_modules ............................... None
|
| 430 |
+
recompute_num_layers ............................ None
|
| 431 |
+
record_memory_history ........................... False
|
| 432 |
+
relative_attention_max_distance ................. 128
|
| 433 |
+
relative_attention_num_buckets .................. 32
|
| 434 |
+
replication ..................................... False
|
| 435 |
+
replication_factor .............................. 2
|
| 436 |
+
replication_jump ................................ None
|
| 437 |
+
rerun_mode ...................................... validate_results
|
| 438 |
+
reset_attention_mask ............................ False
|
| 439 |
+
reset_position_ids .............................. False
|
| 440 |
+
result_rejected_tracker_filename ................ None
|
| 441 |
+
retriever_report_topk_accuracies ................ []
|
| 442 |
+
retriever_score_scaling ......................... False
|
| 443 |
+
retriever_seq_length ............................ 256
|
| 444 |
+
retro_add_retriever ............................. False
|
| 445 |
+
retro_attention_gate ............................ 1
|
| 446 |
+
retro_cyclic_train_iters ........................ None
|
| 447 |
+
retro_encoder_attention_dropout ................. 0.1
|
| 448 |
+
retro_encoder_hidden_dropout .................... 0.1
|
| 449 |
+
retro_encoder_layers ............................ 2
|
| 450 |
+
retro_num_neighbors ............................. 2
|
| 451 |
+
retro_num_retrieved_chunks ...................... 2
|
| 452 |
+
retro_project_dir ............................... None
|
| 453 |
+
retro_verify_neighbor_count ..................... True
|
| 454 |
+
reuse_grad_buf_for_mxfp8_param_ag ............... False
|
| 455 |
+
rope_scaling_factor ............................. 8.0
|
| 456 |
+
rope_type ....................................... None
|
| 457 |
+
rotary_base ..................................... 1000000
|
| 458 |
+
rotary_interleaved .............................. False
|
| 459 |
+
rotary_percent .................................. 1.0
|
| 460 |
+
rotary_scaling_factor ........................... 1.0
|
| 461 |
+
rotary_seq_len_interpolation_factor ............. None
|
| 462 |
+
run_workload_inspector_server ................... False
|
| 463 |
+
sample_rate ..................................... 1.0
|
| 464 |
+
save ............................................ None
|
| 465 |
+
save_interval ................................... None
|
| 466 |
+
save_retain_interval ............................ None
|
| 467 |
+
scatter_gather_tensors_in_pipeline .............. True
|
| 468 |
+
seed ............................................ 1234
|
| 469 |
+
seq_length ...................................... 4096
|
| 470 |
+
sequence_parallel ............................... False
|
| 471 |
+
sft ............................................. False
|
| 472 |
+
sft_tokenizer_prompt_format ..................... nemotron-h-aligned
|
| 473 |
+
sgd_momentum .................................... 0.9
|
| 474 |
+
short_seq_prob .................................. 0.1
|
| 475 |
+
skip_train ...................................... False
|
| 476 |
+
skipped_train_samples ........................... 0
|
| 477 |
+
spec ............................................ None
|
| 478 |
+
split ........................................... 100,0,0
|
| 479 |
+
squared_relu .................................... False
|
| 480 |
+
start_weight_decay .............................. 0.1
|
| 481 |
+
straggler_ctrlr_port ............................ 65535
|
| 482 |
+
straggler_minmax_count .......................... 1
|
| 483 |
+
suggested_communication_unit_size ............... None
|
| 484 |
+
swiglu .......................................... True
|
| 485 |
+
swin_backbone_type .............................. tiny
|
| 486 |
+
symmetric_ar_type ............................... None
|
| 487 |
+
te_rng_tracker .................................. True
|
| 488 |
+
tensor_model_parallel_size ...................... 1
|
| 489 |
+
tensorboard_dir ................................. None
|
| 490 |
+
tensorboard_log_interval ........................ 1
|
| 491 |
+
tensorboard_queue_size .......................... 1000
|
| 492 |
+
test_data_path .................................. None
|
| 493 |
+
test_mode ....................................... False
|
| 494 |
+
tiktoken_num_special_tokens ..................... 1000
|
| 495 |
+
tiktoken_pattern ................................ None
|
| 496 |
+
tiktoken_special_tokens ......................... None
|
| 497 |
+
timing_log_level ................................ 0
|
| 498 |
+
timing_log_option ............................... minmax
|
| 499 |
+
titles_data_path ................................ None
|
| 500 |
+
tokenizer_model ................................. None
|
| 501 |
+
tokenizer_type .................................. GPT2BPETokenizer
|
| 502 |
+
torch_fsdp2_reshard_after_forward ............... True
|
| 503 |
+
tp_comm_bootstrap_backend ....................... nccl
|
| 504 |
+
tp_comm_bulk_dgrad .............................. True
|
| 505 |
+
tp_comm_bulk_wgrad .............................. True
|
| 506 |
+
tp_comm_overlap ................................. False
|
| 507 |
+
tp_comm_overlap_ag .............................. True
|
| 508 |
+
tp_comm_overlap_cfg ............................. None
|
| 509 |
+
tp_comm_overlap_rs .............................. True
|
| 510 |
+
tp_comm_overlap_rs_dgrad ........................ False
|
| 511 |
+
tp_comm_split_ag ................................ True
|
| 512 |
+
tp_comm_split_rs ................................ True
|
| 513 |
+
train_data_path ................................. None
|
| 514 |
+
train_iters ..................................... 12
|
| 515 |
+
train_samples ................................... None
|
| 516 |
+
train_sync_interval ............................. None
|
| 517 |
+
transformer_impl ................................ transformer_engine
|
| 518 |
+
transformer_pipeline_model_parallel_size ........ 1
|
| 519 |
+
untie_embeddings_and_output_weights ............. True
|
| 520 |
+
use_checkpoint_args ............................. False
|
| 521 |
+
use_checkpoint_opt_param_scheduler .............. False
|
| 522 |
+
use_cpu_initialization .......................... None
|
| 523 |
+
use_custom_fsdp ................................. False
|
| 524 |
+
use_dist_ckpt ................................... True
|
| 525 |
+
use_dist_ckpt_deprecated ........................ False
|
| 526 |
+
use_distributed_optimizer ....................... True
|
| 527 |
+
use_flash_attn .................................. False
|
| 528 |
+
use_legacy_models ............................... False
|
| 529 |
+
use_mp_args_from_checkpoint_args ................ False
|
| 530 |
+
use_one_sent_docs ............................... False
|
| 531 |
+
use_persistent_ckpt_worker ...................... False
|
| 532 |
+
use_precision_aware_optimizer ................... False
|
| 533 |
+
use_pytorch_profiler ............................ False
|
| 534 |
+
use_ring_exchange_p2p ........................... False
|
| 535 |
+
use_rope_scaling ................................ False
|
| 536 |
+
use_rotary_position_embeddings .................. False
|
| 537 |
+
use_sharp ....................................... False
|
| 538 |
+
use_tokenizer_model_from_checkpoint_args ........ True
|
| 539 |
+
use_torch_fsdp2 ................................. False
|
| 540 |
+
use_torch_optimizer_for_cpu_offload ............. False
|
| 541 |
+
use_tp_pp_dp_mapping ............................ False
|
| 542 |
+
v_head_dim ...................................... 128
|
| 543 |
+
valid_data_path ................................. None
|
| 544 |
+
variable_seq_lengths ............................ False
|
| 545 |
+
virtual_pipeline_model_parallel_size ............ None
|
| 546 |
+
vision_backbone_type ............................ vit
|
| 547 |
+
vision_pretraining .............................. False
|
| 548 |
+
vision_pretraining_type ......................... classify
|
| 549 |
+
vocab_extra_ids ................................. 0
|
| 550 |
+
vocab_file ...................................... /workspace/data/wikitext/gpt2-vocab.json
|
| 551 |
+
vocab_size ...................................... None
|
| 552 |
+
wandb_exp_name ..................................
|
| 553 |
+
wandb_project ...................................
|
| 554 |
+
wandb_save_dir ..................................
|
| 555 |
+
weight_decay .................................... 0.1
|
| 556 |
+
weight_decay_incr_style ......................... constant
|
| 557 |
+
wgrad_deferral_limit ............................ 0
|
| 558 |
+
world_size ...................................... 8
|
| 559 |
+
yaml_cfg ........................................ None
|
| 560 |
+
-------------------- end of arguments ---------------------
|
| 561 |
+
INFO:megatron.core.num_microbatches_calculator:setting number of microbatches to constant 2
|
| 562 |
+
> building GPT2BPETokenizer tokenizer ...
|
| 563 |
+
/workspace/Megatron-MoE-Echo/megatron/training/arguments.py:831: UserWarning: Disabling sequence parallelism because tensor model parallelism is disabled
|
| 564 |
+
warnings.warn("Disabling sequence parallelism because tensor model parallelism is disabled")
|
| 565 |
+
> padded vocab (size: 50257) with 47 dummy tokens (new size: 50304)
|
| 566 |
+
WARNING:megatron.core.rerun_state_machine:RerunStateMachine initialized in mode RerunMode.VALIDATE_RESULTS
|
| 567 |
+
> initializing torch distributed ...
|
| 568 |
+
/workspace/Megatron-MoE-Echo/megatron/training/arguments.py:831: UserWarning: Disabling sequence parallelism because tensor model parallelism is disabled
|
| 569 |
+
warnings.warn("Disabling sequence parallelism because tensor model parallelism is disabled")
|
| 570 |
+
/workspace/Megatron-MoE-Echo/megatron/training/arguments.py:831: UserWarning: Disabling sequence parallelism because tensor model parallelism is disabled
|
| 571 |
+
warnings.warn("Disabling sequence parallelism because tensor model parallelism is disabled")
|
| 572 |
+
/workspace/Megatron-MoE-Echo/megatron/training/arguments.py:831: UserWarning: Disabling sequence parallelism because tensor model parallelism is disabled
|
| 573 |
+
warnings.warn("Disabling sequence parallelism because tensor model parallelism is disabled")
|
| 574 |
+
/workspace/Megatron-MoE-Echo/megatron/training/arguments.py:831: UserWarning: Disabling sequence parallelism because tensor model parallelism is disabled
|
| 575 |
+
warnings.warn("Disabling sequence parallelism because tensor model parallelism is disabled")
|
| 576 |
+
[Gloo] Rank 1 is connected to [Gloo] Rank 7 peer ranks. Expected number of connected peer ranks is : 07 is connected to
|
| 577 |
+
7 peer ranks. Expected number of connected peer ranks is : 7
|
| 578 |
+
[Gloo] Rank 3 is connected to 7 peer ranks. Expected number of connected peer ranks is : 7
|
| 579 |
+
[Gloo] Rank 5 is connected to 7 peer ranks. Expected number of connected peer ranks is : 7
|
| 580 |
+
[Gloo] Rank 7 is connected to 7 peer ranks. Expected number of connected peer ranks is : 7
|
| 581 |
+
[Gloo] Rank 2 is connected to 7 peer ranks. Expected number of connected peer ranks is : 7
|
| 582 |
+
[Gloo] Rank 6 is connected to 7 peer ranks. Expected number of connected peer ranks is : 7
|
| 583 |
+
[Gloo] Rank 4 is connected to 7 peer ranks. Expected number of connected peer ranks is : 7
|
| 584 |
+
[Gloo] Rank 0 is connected to 7 peer ranks. Expected number of connected peer ranks is : 7
|
| 585 |
+
[Gloo] Rank 2 is connected to 7 peer ranks. Expected number of connected peer ranks is : 7
|
| 586 |
+
[Gloo] Rank 3 is connected to 7 peer ranks. Expected number of connected peer ranks is : 7
|
| 587 |
+
[Gloo] Rank [Gloo] Rank [Gloo] Rank [Gloo] Rank [Gloo] Rank 1465 is connected to 7 is connected to is connected to is connected to 7 is connected to 777 peer ranks. 7 peer ranks. peer ranks. peer ranks. Expected number of connected peer ranks is : peer ranks. Expected number of connected peer ranks is : Expected number of connected peer ranks is : Expected number of connected peer ranks is : 7Expected number of connected peer ranks is :
|
| 588 |
+
77
|
| 589 |
+
|
| 590 |
+
77
|
| 591 |
+
|
| 592 |
+
[Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
|
| 593 |
+
> initialized tensor model parallel with size 1
|
| 594 |
+
> initialized pipeline model parallel with size 1
|
| 595 |
+
> setting random seeds to 1234 ...
|
| 596 |
+
[Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
|
| 597 |
+
[Gloo] Rank [Gloo] Rank 00 is connected to is connected to 00 peer ranks. peer ranks. Expected number of connected peer ranks is : Expected number of connected peer ranks is : 00
|
| 598 |
+
|
| 599 |
+
[Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
|
| 600 |
+
[Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
|
| 601 |
+
[Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
|
| 602 |
+
[Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
|
| 603 |
+
[rank7]:[W830 06:49:58.890026700 ProcessGroupNCCL.cpp:4789] [PG ID 0 PG GUID 0 Rank 7] using GPU 7 as device used by this process is currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect. You can specify device_id in init_process_group() to force use of a particular device.
|
| 604 |
+
[rank5]:[W830 06:49:58.898308384 ProcessGroupNCCL.cpp:4789] [PG ID 0 PG GUID 0 Rank 5] using GPU 5 as device used by this process is currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect. You can specify device_id in init_process_group() to force use of a particular device.
|
| 605 |
+
[rank2]:[W830 06:49:58.898366225 ProcessGroupNCCL.cpp:4789] [PG ID 0 PG GUID 0 Rank 2] using GPU 2 as device used by this process is currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect. You can specify device_id in init_process_group() to force use of a particular device.
|
| 606 |
+
[rank4]:[W830 06:49:58.898414112 ProcessGroupNCCL.cpp:4789] [PG ID 0 PG GUID 0 Rank 4] using GPU 4 as device used by this process is currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect. You can specify device_id in init_process_group() to force use of a particular device.
|
| 607 |
+
[rank6]:[W830 06:49:58.898462177 ProcessGroupNCCL.cpp:4789] [PG ID 0 PG GUID 0 Rank 6] using GPU 6 as device used by this process is currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect. You can specify device_id in init_process_group() to force use of a particular device.
|
| 608 |
+
[rank3]:[W830 06:49:58.898462684 ProcessGroupNCCL.cpp:4789] [PG ID 0 PG GUID 0 Rank 3] using GPU 3 as device used by this process is currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect. You can specify device_id in init_process_group() to force use of a particular device.
|
| 609 |
+
[rank1]:[W830 06:49:58.899674441 ProcessGroupNCCL.cpp:4789] [PG ID 0 PG GUID 0 Rank 1] using GPU 1 as device used by this process is currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect. You can specify device_id in init_process_group() to force use of a particular device.
|
| 610 |
+
> compiling dataset index builder ...
|
| 611 |
+
make: Entering directory '/workspace/Megatron-MoE-Echo/megatron/core/datasets'
|
| 612 |
+
make: Nothing to be done for 'default'.
|
| 613 |
+
make: Leaving directory '/workspace/Megatron-MoE-Echo/megatron/core/datasets'
|
| 614 |
+
>>> done with dataset index builder. Compilation time: 0.035 seconds
|
| 615 |
+
WARNING: constraints for invoking optimized fused softmax kernel are not met. We default back to unfused kernel invocations.
|
| 616 |
+
> compiling and loading fused kernels ...
|
| 617 |
+
[rank0]:[W830 06:49:58.045827445 ProcessGroupNCCL.cpp:4789] [PG ID 0 PG GUID 0 Rank 0] using GPU 0 as device used by this process is currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect. You can specify device_id in init_process_group() to force use of a particular device.
|
| 618 |
+
>>> done with compiling and loading fused kernels. Compilation time: 3.921 seconds
|
| 619 |
+
time to initialize megatron (seconds): 7.329
|
| 620 |
+
[after megatron is initialized] datetime: 2026-08-30 06:50:04
|
| 621 |
+
building GPT model ...
|
| 622 |
+
/usr/local/lib/python3.12/dist-packages/transformer_engine/pytorch/cpu_offload.py:708: DeprecationWarning: Offloading weights is deprecated. Using offload_weights=True does not have any effect.
|
| 623 |
+
warnings.warn(
|
| 624 |
+
/usr/local/lib/python3.12/dist-packages/transformer_engine/pytorch/cpu_offload.py:708: DeprecationWarning: Offloading weights is deprecated. Using offload_weights=True does not have any effect.
|
| 625 |
+
warnings.warn(
|
| 626 |
+
/usr/local/lib/python3.12/dist-packages/transformer_engine/pytorch/cpu_offload.py:708: DeprecationWarning: Offloading weights is deprecated. Using offload_weights=True does not have any effect.
|
| 627 |
+
warnings.warn(
|
| 628 |
+
/usr/local/lib/python3.12/dist-packages/transformer_engine/pytorch/cpu_offload.py:708: DeprecationWarning: Offloading weights is deprecated. Using offload_weights=True does not have any effect.
|
| 629 |
+
warnings.warn(
|
| 630 |
+
/usr/local/lib/python3.12/dist-packages/transformer_engine/pytorch/cpu_offload.py:708: DeprecationWarning: Offloading weights is deprecated. Using offload_weights=True does not have any effect.
|
| 631 |
+
warnings.warn(
|
| 632 |
+
/usr/local/lib/python3.12/dist-packages/transformer_engine/pytorch/cpu_offload.py:708: DeprecationWarning: Offloading weights is deprecated. Using offload_weights=True does not have any effect.
|
| 633 |
+
warnings.warn(
|
| 634 |
+
/usr/local/lib/python3.12/dist-packages/transformer_engine/pytorch/cpu_offload.py:708: DeprecationWarning: Offloading weights is deprecated. Using offload_weights=True does not have any effect.
|
| 635 |
+
warnings.warn(
|
| 636 |
+
/usr/local/lib/python3.12/dist-packages/transformer_engine/pytorch/cpu_offload.py:708: DeprecationWarning: Offloading weights is deprecated. Using offload_weights=True does not have any effect.
|
| 637 |
+
warnings.warn(
|
| 638 |
+
> number of parameters on (tensor, pipeline) model parallel rank (0, 0): 3903983616
|
| 639 |
+
INFO:megatron.core.distributed.distributed_data_parallel:Setting up DistributedDataParallel with config DistributedDataParallelConfig(grad_reduce_in_fp32=True, overlap_grad_reduce=True, overlap_param_gather=True, align_param_gather=False, use_distributed_optimizer=True, num_distributed_optimizer_instances=1, check_for_nan_in_grad=False, check_for_large_grads=False, bucket_size=40000000, pad_buckets_for_high_nccl_busbw=False, average_in_collective=False, fp8_param_gather=False, reuse_grad_buf_for_mxfp8_param_ag=False, use_custom_fsdp=False, data_parallel_sharding_strategy='no_shard', gradient_reduce_div_fusion=True, suggested_communication_unit_size=None, preserve_fp32_weights=True, keep_fp8_transpose_cache_when_using_custom_fsdp=False, nccl_ub=False, fsdp_double_buffer=False)
|
| 640 |
+
INFO:megatron.core.distributed.param_and_grad_buffer:Number of buckets for gradient all-reduce / reduce-scatter: 18
|
| 641 |
+
Params for bucket 1 (206045184 elements, 206045184 padded size):
|
| 642 |
+
module.output_layer.weight
|
| 643 |
+
Params for bucket 2 (42086400 elements, 42086400 padded size):
|
| 644 |
+
module.decoder.final_layernorm.weight
|
| 645 |
+
module.decoder.layers.15.self_attention.linear_qkv.weight
|
| 646 |
+
module.decoder.layers.15.pre_mlp_layernorm.weight
|
| 647 |
+
module.decoder.layers.15.mlp.router.weight
|
| 648 |
+
module.decoder.layers.15.self_attention.linear_qkv.layer_norm_weight
|
| 649 |
+
module.decoder.layers.15.self_attention.linear_proj.weight
|
| 650 |
+
Params for bucket 3 (42082304 elements, 42082304 padded size):
|
| 651 |
+
module.decoder.layers.14.mlp.router.weight
|
| 652 |
+
module.decoder.layers.14.self_attention.linear_qkv.weight
|
| 653 |
+
module.decoder.layers.14.pre_mlp_layernorm.weight
|
| 654 |
+
module.decoder.layers.14.self_attention.linear_proj.weight
|
| 655 |
+
module.decoder.layers.14.self_attention.linear_qkv.layer_norm_weight
|
| 656 |
+
Params for bucket 4 (42082304 elements, 42082304 padded size):
|
| 657 |
+
module.decoder.layers.13.self_attention.linear_qkv.weight
|
| 658 |
+
module.decoder.layers.13.pre_mlp_layernorm.weight
|
| 659 |
+
module.decoder.layers.13.mlp.router.weight
|
| 660 |
+
module.decoder.layers.13.self_attention.linear_qkv.layer_norm_weight
|
| 661 |
+
module.decoder.layers.13.self_attention.linear_proj.weight
|
| 662 |
+
Params for bucket 5 (42082304 elements, 42082304 padded size):
|
| 663 |
+
module.decoder.layers.12.mlp.router.weight
|
| 664 |
+
module.decoder.layers.12.self_attention.linear_qkv.layer_norm_weight
|
| 665 |
+
module.decoder.layers.12.self_attention.linear_qkv.weight
|
| 666 |
+
module.decoder.layers.12.self_attention.linear_proj.weight
|
| 667 |
+
module.decoder.layers.12.pre_mlp_layernorm.weight
|
| 668 |
+
Params for bucket 6 (42082304 elements, 42082304 padded size):
|
| 669 |
+
module.decoder.layers.11.self_attention.linear_qkv.weight
|
| 670 |
+
module.decoder.layers.11.pre_mlp_layernorm.weight
|
| 671 |
+
module.decoder.layers.11.mlp.router.weight
|
| 672 |
+
module.decoder.layers.11.self_attention.linear_proj.weight
|
| 673 |
+
module.decoder.layers.11.self_attention.linear_qkv.layer_norm_weight
|
| 674 |
+
Params for bucket 7 (42082304 elements, 42082304 padded size):
|
| 675 |
+
module.decoder.layers.10.mlp.router.weight
|
| 676 |
+
module.decoder.layers.10.self_attention.linear_qkv.layer_norm_weight
|
| 677 |
+
module.decoder.layers.10.self_attention.linear_proj.weight
|
| 678 |
+
module.decoder.layers.10.self_attention.linear_qkv.weight
|
| 679 |
+
module.decoder.layers.10.pre_mlp_layernorm.weight
|
| 680 |
+
Params for bucket 8 (42082304 elements, 42082304 padded size):
|
| 681 |
+
module.decoder.layers.9.mlp.router.weight
|
| 682 |
+
module.decoder.layers.9.self_attention.linear_qkv.layer_norm_weight
|
| 683 |
+
module.decoder.layers.9.self_attention.linear_proj.weight
|
| 684 |
+
module.decoder.layers.9.self_attention.linear_qkv.weight
|
| 685 |
+
module.decoder.layers.9.pre_mlp_layernorm.weight
|
| 686 |
+
Params for bucket 9 (42082304 elements, 42082304 padded size):
|
| 687 |
+
module.decoder.layers.8.mlp.router.weight
|
| 688 |
+
module.decoder.layers.8.self_attention.linear_qkv.layer_norm_weight
|
| 689 |
+
module.decoder.layers.8.self_attention.linear_proj.weight
|
| 690 |
+
module.decoder.layers.8.pre_mlp_layernorm.weight
|
| 691 |
+
module.decoder.layers.8.self_attention.linear_qkv.weight
|
| 692 |
+
Params for bucket 10 (42082304 elements, 42082304 padded size):
|
| 693 |
+
module.decoder.layers.7.mlp.router.weight
|
| 694 |
+
module.decoder.layers.7.self_attention.linear_qkv.weight
|
| 695 |
+
module.decoder.layers.7.pre_mlp_layernorm.weight
|
| 696 |
+
module.decoder.layers.7.self_attention.linear_proj.weight
|
| 697 |
+
module.decoder.layers.7.self_attention.linear_qkv.layer_norm_weight
|
| 698 |
+
Params for bucket 11 (42082304 elements, 42082304 padded size):
|
| 699 |
+
module.decoder.layers.6.mlp.router.weight
|
| 700 |
+
module.decoder.layers.6.self_attention.linear_proj.weight
|
| 701 |
+
module.decoder.layers.6.self_attention.linear_qkv.layer_norm_weight
|
| 702 |
+
module.decoder.layers.6.pre_mlp_layernorm.weight
|
| 703 |
+
module.decoder.layers.6.self_attention.linear_qkv.weight
|
| 704 |
+
Params for bucket 12 (42082304 elements, 42082304 padded size):
|
| 705 |
+
module.decoder.layers.5.self_attention.linear_proj.weight
|
| 706 |
+
module.decoder.layers.5.pre_mlp_layernorm.weight
|
| 707 |
+
module.decoder.layers.5.self_attention.linear_qkv.weight
|
| 708 |
+
module.decoder.layers.5.mlp.router.weight
|
| 709 |
+
module.decoder.layers.5.self_attention.linear_qkv.layer_norm_weight
|
| 710 |
+
Params for bucket 13 (42082304 elements, 42082304 padded size):
|
| 711 |
+
module.decoder.layers.4.mlp.router.weight
|
| 712 |
+
module.decoder.layers.4.pre_mlp_layernorm.weight
|
| 713 |
+
module.decoder.layers.4.self_attention.linear_qkv.weight
|
| 714 |
+
module.decoder.layers.4.self_attention.linear_proj.weight
|
| 715 |
+
module.decoder.layers.4.self_attention.linear_qkv.layer_norm_weight
|
| 716 |
+
Params for bucket 14 (42082304 elements, 42082304 padded size):
|
| 717 |
+
module.decoder.layers.3.mlp.router.weight
|
| 718 |
+
module.decoder.layers.3.self_attention.linear_proj.weight
|
| 719 |
+
module.decoder.layers.3.self_attention.linear_qkv.layer_norm_weight
|
| 720 |
+
module.decoder.layers.3.pre_mlp_layernorm.weight
|
| 721 |
+
module.decoder.layers.3.self_attention.linear_qkv.weight
|
| 722 |
+
Params for bucket 15 (42082304 elements, 42082304 padded size):
|
| 723 |
+
module.decoder.layers.2.mlp.router.weight
|
| 724 |
+
module.decoder.layers.2.self_attention.linear_proj.weight
|
| 725 |
+
module.decoder.layers.2.self_attention.linear_qkv.layer_norm_weight
|
| 726 |
+
module.decoder.layers.2.self_attention.linear_qkv.weight
|
| 727 |
+
module.decoder.layers.2.pre_mlp_layernorm.weight
|
| 728 |
+
Params for bucket 16 (42082304 elements, 42082304 padded size):
|
| 729 |
+
module.decoder.layers.1.mlp.router.weight
|
| 730 |
+
module.decoder.layers.1.pre_mlp_layernorm.weight
|
| 731 |
+
module.decoder.layers.1.self_attention.linear_qkv.layer_norm_weight
|
| 732 |
+
module.decoder.layers.1.self_attention.linear_qkv.weight
|
| 733 |
+
module.decoder.layers.1.self_attention.linear_proj.weight
|
| 734 |
+
Params for bucket 17 (42082304 elements, 42082304 padded size):
|
| 735 |
+
module.decoder.layers.0.self_attention.linear_proj.weight
|
| 736 |
+
module.decoder.layers.0.pre_mlp_layernorm.weight
|
| 737 |
+
module.decoder.layers.0.mlp.router.weight
|
| 738 |
+
module.decoder.layers.0.self_attention.linear_qkv.layer_norm_weight
|
| 739 |
+
module.decoder.layers.0.self_attention.linear_qkv.weight
|
| 740 |
+
Params for bucket 18 (206045184 elements, 206045184 padded size):
|
| 741 |
+
module.embedding.word_embeddings.weight
|
| 742 |
+
INFO:megatron.core.distributed.param_and_grad_buffer:Number of buckets for gradient all-reduce / reduce-scatter: 56
|
| 743 |
+
Params for bucket 1 (44040192 elements, 44040192 padded size):
|
| 744 |
+
module.decoder.layers.15.mlp.experts.linear_fc2.weight3
|
| 745 |
+
module.decoder.layers.15.mlp.experts.linear_fc2.weight1
|
| 746 |
+
module.decoder.layers.15.mlp.experts.linear_fc2.weight2
|
| 747 |
+
Params for bucket 2 (44040192 elements, 44040192 padded size):
|
| 748 |
+
module.decoder.layers.15.mlp.experts.linear_fc2.weight0
|
| 749 |
+
module.decoder.layers.15.mlp.experts.linear_fc1.weight3
|
| 750 |
+
Params for bucket 3 (58720256 elements, 58720256 padded size):
|
| 751 |
+
module.decoder.layers.15.mlp.experts.linear_fc1.weight2
|
| 752 |
+
module.decoder.layers.15.mlp.experts.linear_fc1.weight1
|
| 753 |
+
Params for bucket 4 (44040192 elements, 44040192 padded size):
|
| 754 |
+
module.decoder.layers.15.mlp.experts.linear_fc1.weight0
|
| 755 |
+
module.decoder.layers.14.mlp.experts.linear_fc2.weight3
|
| 756 |
+
Params for bucket 5 (44040192 elements, 44040192 padded size):
|
| 757 |
+
module.decoder.layers.14.mlp.experts.linear_fc2.weight2
|
| 758 |
+
module.decoder.layers.14.mlp.experts.linear_fc2.weight0
|
| 759 |
+
module.decoder.layers.14.mlp.experts.linear_fc2.weight1
|
| 760 |
+
Params for bucket 6 (58720256 elements, 58720256 padded size):
|
| 761 |
+
module.decoder.layers.14.mlp.experts.linear_fc1.weight3
|
| 762 |
+
module.decoder.layers.14.mlp.experts.linear_fc1.weight2
|
| 763 |
+
Params for bucket 7 (58720256 elements, 58720256 padded size):
|
| 764 |
+
module.decoder.layers.14.mlp.experts.linear_fc1.weight1
|
| 765 |
+
module.decoder.layers.14.mlp.experts.linear_fc1.weight0
|
| 766 |
+
Params for bucket 8 (44040192 elements, 44040192 padded size):
|
| 767 |
+
module.decoder.layers.13.mlp.experts.linear_fc2.weight3
|
| 768 |
+
module.decoder.layers.13.mlp.experts.linear_fc2.weight1
|
| 769 |
+
module.decoder.layers.13.mlp.experts.linear_fc2.weight2
|
| 770 |
+
Params for bucket 9 (44040192 elements, 44040192 padded size):
|
| 771 |
+
module.decoder.layers.13.mlp.experts.linear_fc2.weight0
|
| 772 |
+
module.decoder.layers.13.mlp.experts.linear_fc1.weight3
|
| 773 |
+
Params for bucket 10 (58720256 elements, 58720256 padded size):
|
| 774 |
+
module.decoder.layers.13.mlp.experts.linear_fc1.weight2
|
| 775 |
+
module.decoder.layers.13.mlp.experts.linear_fc1.weight1
|
| 776 |
+
Params for bucket 11 (44040192 elements, 44040192 padded size):
|
| 777 |
+
module.decoder.layers.13.mlp.experts.linear_fc1.weight0
|
| 778 |
+
module.decoder.layers.12.mlp.experts.linear_fc2.weight3
|
| 779 |
+
Params for bucket 12 (44040192 elements, 44040192 padded size):
|
| 780 |
+
module.decoder.layers.12.mlp.experts.linear_fc2.weight2
|
| 781 |
+
module.decoder.layers.12.mlp.experts.linear_fc2.weight0
|
| 782 |
+
module.decoder.layers.12.mlp.experts.linear_fc2.weight1
|
| 783 |
+
Params for bucket 13 (58720256 elements, 58720256 padded size):
|
| 784 |
+
module.decoder.layers.12.mlp.experts.linear_fc1.weight3
|
| 785 |
+
module.decoder.layers.12.mlp.experts.linear_fc1.weight2
|
| 786 |
+
Params for bucket 14 (58720256 elements, 58720256 padded size):
|
| 787 |
+
module.decoder.layers.12.mlp.experts.linear_fc1.weight1
|
| 788 |
+
module.decoder.layers.12.mlp.experts.linear_fc1.weight0
|
| 789 |
+
Params for bucket 15 (44040192 elements, 44040192 padded size):
|
| 790 |
+
module.decoder.layers.11.mlp.experts.linear_fc2.weight3
|
| 791 |
+
module.decoder.layers.11.mlp.experts.linear_fc2.weight1
|
| 792 |
+
module.decoder.layers.11.mlp.experts.linear_fc2.weight2
|
| 793 |
+
Params for bucket 16 (44040192 elements, 44040192 padded size):
|
| 794 |
+
module.decoder.layers.11.mlp.experts.linear_fc2.weight0
|
| 795 |
+
module.decoder.layers.11.mlp.experts.linear_fc1.weight3
|
| 796 |
+
Params for bucket 17 (58720256 elements, 58720256 padded size):
|
| 797 |
+
module.decoder.layers.11.mlp.experts.linear_fc1.weight2
|
| 798 |
+
module.decoder.layers.11.mlp.experts.linear_fc1.weight1
|
| 799 |
+
Params for bucket 18 (44040192 elements, 44040192 padded size):
|
| 800 |
+
module.decoder.layers.11.mlp.experts.linear_fc1.weight0
|
| 801 |
+
module.decoder.layers.10.mlp.experts.linear_fc2.weight3
|
| 802 |
+
Params for bucket 19 (44040192 elements, 44040192 padded size):
|
| 803 |
+
module.decoder.layers.10.mlp.experts.linear_fc2.weight2
|
| 804 |
+
module.decoder.layers.10.mlp.experts.linear_fc2.weight0
|
| 805 |
+
module.decoder.layers.10.mlp.experts.linear_fc2.weight1
|
| 806 |
+
Params for bucket 20 (58720256 elements, 58720256 padded size):
|
| 807 |
+
module.decoder.layers.10.mlp.experts.linear_fc1.weight3
|
| 808 |
+
module.decoder.layers.10.mlp.experts.linear_fc1.weight2
|
| 809 |
+
Params for bucket 21 (58720256 elements, 58720256 padded size):
|
| 810 |
+
module.decoder.layers.10.mlp.experts.linear_fc1.weight1
|
| 811 |
+
module.decoder.layers.10.mlp.experts.linear_fc1.weight0
|
| 812 |
+
Params for bucket 22 (44040192 elements, 44040192 padded size):
|
| 813 |
+
module.decoder.layers.9.mlp.experts.linear_fc2.weight3
|
| 814 |
+
module.decoder.layers.9.mlp.experts.linear_fc2.weight1
|
| 815 |
+
module.decoder.layers.9.mlp.experts.linear_fc2.weight2
|
| 816 |
+
Params for bucket 23 (44040192 elements, 44040192 padded size):
|
| 817 |
+
module.decoder.layers.9.mlp.experts.linear_fc2.weight0
|
| 818 |
+
module.decoder.layers.9.mlp.experts.linear_fc1.weight3
|
| 819 |
+
Params for bucket 24 (58720256 elements, 58720256 padded size):
|
| 820 |
+
module.decoder.layers.9.mlp.experts.linear_fc1.weight2
|
| 821 |
+
module.decoder.layers.9.mlp.experts.linear_fc1.weight1
|
| 822 |
+
Params for bucket 25 (44040192 elements, 44040192 padded size):
|
| 823 |
+
module.decoder.layers.9.mlp.experts.linear_fc1.weight0
|
| 824 |
+
module.decoder.layers.8.mlp.experts.linear_fc2.weight3
|
| 825 |
+
Params for bucket 26 (44040192 elements, 44040192 padded size):
|
| 826 |
+
module.decoder.layers.8.mlp.experts.linear_fc2.weight2
|
| 827 |
+
module.decoder.layers.8.mlp.experts.linear_fc2.weight0
|
| 828 |
+
module.decoder.layers.8.mlp.experts.linear_fc2.weight1
|
| 829 |
+
Params for bucket 27 (58720256 elements, 58720256 padded size):
|
| 830 |
+
module.decoder.layers.8.mlp.experts.linear_fc1.weight3
|
| 831 |
+
module.decoder.layers.8.mlp.experts.linear_fc1.weight2
|
| 832 |
+
Params for bucket 28 (58720256 elements, 58720256 padded size):
|
| 833 |
+
module.decoder.layers.8.mlp.experts.linear_fc1.weight1
|
| 834 |
+
module.decoder.layers.8.mlp.experts.linear_fc1.weight0
|
| 835 |
+
Params for bucket 29 (44040192 elements, 44040192 padded size):
|
| 836 |
+
module.decoder.layers.7.mlp.experts.linear_fc2.weight3
|
| 837 |
+
module.decoder.layers.7.mlp.experts.linear_fc2.weight2
|
| 838 |
+
module.decoder.layers.7.mlp.experts.linear_fc2.weight1
|
| 839 |
+
Params for bucket 30 (44040192 elements, 44040192 padded size):
|
| 840 |
+
module.decoder.layers.7.mlp.experts.linear_fc2.weight0
|
| 841 |
+
module.decoder.layers.7.mlp.experts.linear_fc1.weight3
|
| 842 |
+
Params for bucket 31 (58720256 elements, 58720256 padded size):
|
| 843 |
+
module.decoder.layers.7.mlp.experts.linear_fc1.weight2
|
| 844 |
+
module.decoder.layers.7.mlp.experts.linear_fc1.weight1
|
| 845 |
+
Params for bucket 32 (44040192 elements, 44040192 padded size):
|
| 846 |
+
module.decoder.layers.7.mlp.experts.linear_fc1.weight0
|
| 847 |
+
module.decoder.layers.6.mlp.experts.linear_fc2.weight3
|
| 848 |
+
Params for bucket 33 (44040192 elements, 44040192 padded size):
|
| 849 |
+
module.decoder.layers.6.mlp.experts.linear_fc2.weight2
|
| 850 |
+
module.decoder.layers.6.mlp.experts.linear_fc2.weight1
|
| 851 |
+
module.decoder.layers.6.mlp.experts.linear_fc2.weight0
|
| 852 |
+
Params for bucket 34 (58720256 elements, 58720256 padded size):
|
| 853 |
+
module.decoder.layers.6.mlp.experts.linear_fc1.weight3
|
| 854 |
+
module.decoder.layers.6.mlp.experts.linear_fc1.weight2
|
| 855 |
+
Params for bucket 35 (58720256 elements, 58720256 padded size):
|
| 856 |
+
module.decoder.layers.6.mlp.experts.linear_fc1.weight1
|
| 857 |
+
module.decoder.layers.6.mlp.experts.linear_fc1.weight0
|
| 858 |
+
Params for bucket 36 (44040192 elements, 44040192 padded size):
|
| 859 |
+
module.decoder.layers.5.mlp.experts.linear_fc2.weight3
|
| 860 |
+
module.decoder.layers.5.mlp.experts.linear_fc2.weight2
|
| 861 |
+
module.decoder.layers.5.mlp.experts.linear_fc2.weight1
|
| 862 |
+
Params for bucket 37 (44040192 elements, 44040192 padded size):
|
| 863 |
+
module.decoder.layers.5.mlp.experts.linear_fc2.weight0
|
| 864 |
+
module.decoder.layers.5.mlp.experts.linear_fc1.weight3
|
| 865 |
+
Params for bucket 38 (58720256 elements, 58720256 padded size):
|
| 866 |
+
module.decoder.layers.5.mlp.experts.linear_fc1.weight2
|
| 867 |
+
module.decoder.layers.5.mlp.experts.linear_fc1.weight1
|
| 868 |
+
Params for bucket 39 (44040192 elements, 44040192 padded size):
|
| 869 |
+
module.decoder.layers.5.mlp.experts.linear_fc1.weight0
|
| 870 |
+
module.decoder.layers.4.mlp.experts.linear_fc2.weight3
|
| 871 |
+
Params for bucket 40 (44040192 elements, 44040192 padded size):
|
| 872 |
+
module.decoder.layers.4.mlp.experts.linear_fc2.weight2
|
| 873 |
+
module.decoder.layers.4.mlp.experts.linear_fc2.weight0
|
| 874 |
+
module.decoder.layers.4.mlp.experts.linear_fc2.weight1
|
| 875 |
+
Params for bucket 41 (58720256 elements, 58720256 padded size):
|
| 876 |
+
module.decoder.layers.4.mlp.experts.linear_fc1.weight3
|
| 877 |
+
module.decoder.layers.4.mlp.experts.linear_fc1.weight2
|
| 878 |
+
Params for bucket 42 (58720256 elements, 58720256 padded size):
|
| 879 |
+
module.decoder.layers.4.mlp.experts.linear_fc1.weight1
|
| 880 |
+
module.decoder.layers.4.mlp.experts.linear_fc1.weight0
|
| 881 |
+
Params for bucket 43 (44040192 elements, 44040192 padded size):
|
| 882 |
+
module.decoder.layers.3.mlp.experts.linear_fc2.weight3
|
| 883 |
+
module.decoder.layers.3.mlp.experts.linear_fc2.weight2
|
| 884 |
+
module.decoder.layers.3.mlp.experts.linear_fc2.weight1
|
| 885 |
+
Params for bucket 44 (44040192 elements, 44040192 padded size):
|
| 886 |
+
module.decoder.layers.3.mlp.experts.linear_fc2.weight0
|
| 887 |
+
module.decoder.layers.3.mlp.experts.linear_fc1.weight3
|
| 888 |
+
Params for bucket 45 (58720256 elements, 58720256 padded size):
|
| 889 |
+
module.decoder.layers.3.mlp.experts.linear_fc1.weight2
|
| 890 |
+
module.decoder.layers.3.mlp.experts.linear_fc1.weight1
|
| 891 |
+
Params for bucket 46 (44040192 elements, 44040192 padded size):
|
| 892 |
+
module.decoder.layers.3.mlp.experts.linear_fc1.weight0
|
| 893 |
+
module.decoder.layers.2.mlp.experts.linear_fc2.weight3
|
| 894 |
+
Params for bucket 47 (44040192 elements, 44040192 padded size):
|
| 895 |
+
module.decoder.layers.2.mlp.experts.linear_fc2.weight2
|
| 896 |
+
module.decoder.layers.2.mlp.experts.linear_fc2.weight1
|
| 897 |
+
module.decoder.layers.2.mlp.experts.linear_fc2.weight0
|
| 898 |
+
Params for bucket 48 (58720256 elements, 58720256 padded size):
|
| 899 |
+
module.decoder.layers.2.mlp.experts.linear_fc1.weight3
|
| 900 |
+
module.decoder.layers.2.mlp.experts.linear_fc1.weight2
|
| 901 |
+
Params for bucket 49 (58720256 elements, 58720256 padded size):
|
| 902 |
+
module.decoder.layers.2.mlp.experts.linear_fc1.weight1
|
| 903 |
+
module.decoder.layers.2.mlp.experts.linear_fc1.weight0
|
| 904 |
+
Params for bucket 50 (44040192 elements, 44040192 padded size):
|
| 905 |
+
module.decoder.layers.1.mlp.experts.linear_fc2.weight3
|
| 906 |
+
module.decoder.layers.1.mlp.experts.linear_fc2.weight1
|
| 907 |
+
module.decoder.layers.1.mlp.experts.linear_fc2.weight2
|
| 908 |
+
Params for bucket 51 (44040192 elements, 44040192 padded size):
|
| 909 |
+
module.decoder.layers.1.mlp.experts.linear_fc2.weight0
|
| 910 |
+
module.decoder.layers.1.mlp.experts.linear_fc1.weight3
|
| 911 |
+
Params for bucket 52 (58720256 elements, 58720256 padded size):
|
| 912 |
+
module.decoder.layers.1.mlp.experts.linear_fc1.weight2
|
| 913 |
+
module.decoder.layers.1.mlp.experts.linear_fc1.weight1
|
| 914 |
+
Params for bucket 53 (44040192 elements, 44040192 padded size):
|
| 915 |
+
module.decoder.layers.1.mlp.experts.linear_fc1.weight0
|
| 916 |
+
module.decoder.layers.0.mlp.experts.linear_fc2.weight3
|
| 917 |
+
Params for bucket 54 (44040192 elements, 44040192 padded size):
|
| 918 |
+
module.decoder.layers.0.mlp.experts.linear_fc2.weight2
|
| 919 |
+
module.decoder.layers.0.mlp.experts.linear_fc2.weight1
|
| 920 |
+
module.decoder.layers.0.mlp.experts.linear_fc2.weight0
|
| 921 |
+
Params for bucket 55 (58720256 elements, 58720256 padded size):
|
| 922 |
+
module.decoder.layers.0.mlp.experts.linear_fc1.weight3
|
| 923 |
+
module.decoder.layers.0.mlp.experts.linear_fc1.weight2
|
| 924 |
+
Params for bucket 56 (58720256 elements, 58720256 padded size):
|
| 925 |
+
module.decoder.layers.0.mlp.experts.linear_fc1.weight1
|
| 926 |
+
module.decoder.layers.0.mlp.experts.linear_fc1.weight0
|
| 927 |
+
INFO:megatron.core.optimizer:Setting up optimizer with config OptimizerConfig(optimizer='adam', lr=0.0001, min_lr=1e-05, decoupled_lr=None, decoupled_min_lr=None, weight_decay=0.1, fp8_recipe='mxfp8', fp16=False, bf16=True, reuse_grad_buf_for_mxfp8_param_ag=False, params_dtype=torch.bfloat16, use_precision_aware_optimizer=False, store_param_remainders=True, main_grads_dtype=torch.float32, main_params_dtype=torch.float32, exp_avg_dtype=torch.float32, exp_avg_sq_dtype=torch.float32, loss_scale=None, initial_loss_scale=4294967296, min_loss_scale=1.0, loss_scale_window=1000, hysteresis=2, adam_beta1=0.9, adam_beta2=0.999, adam_eps=1e-08, sgd_momentum=0.9, use_distributed_optimizer=True, overlap_param_gather_with_optimizer_step=False, optimizer_cpu_offload=False, optimizer_offload_fraction=1.0, use_torch_optimizer_for_cpu_offload=False, overlap_cpu_optimizer_d2h_h2d=False, pin_cpu_grads=True, pin_cpu_params=True, clip_grad=1.0, log_num_zeros_in_grad=False, barrier_with_L1_time=True, timers=<megatron.core.timers.Timers object at 0x77f7a359c710>, config_logger_dir='')
|
| 928 |
+
INFO:megatron.core.optimizer_param_scheduler:> learning rate decay style: cosine
|
| 929 |
+
[after model, optimizer, and learning rate scheduler are built] datetime: 2026-08-30 06:50:05
|
| 930 |
+
> building train, validation, and test datasets ...
|
| 931 |
+
> datasets target sizes (minimum size):
|
| 932 |
+
train: 192
|
| 933 |
+
validation: 0
|
| 934 |
+
test: 0
|
| 935 |
+
INFO:megatron.core.datasets.blended_megatron_dataset_config:Let split_matrix = [(0, 1.0), None, None]
|
| 936 |
+
> building train, validation, and test datasets for GPT ...
|
| 937 |
+
INFO:megatron.core.datasets.blended_megatron_dataset_builder:Building GPTDataset splits with sizes=(192, 0, 0) and config=GPTDatasetConfig(random_seed=1234, sequence_length=4096, blend=(['/workspace/data/wikitext/wikitext103_gpt2_text_document'], None), blend_per_split=None, split='100,0,0', split_matrix=[(0, 1.0), None, None], num_dataset_builder_threads=1, path_to_cache=None, mmap_bin_files=True, mock=False, tokenizer=<megatron.training.tokenizer.tokenizer._GPT2BPETokenizer object at 0x77f7a4ef54c0>, mid_level_dataset_surplus=0.005, reset_position_ids=False, reset_attention_mask=False, eod_mask_loss=False, create_attention_mask=True, drop_last_partial_validation_sequence=True, add_extra_token_to_sequence=True, object_storage_cache_path=None)
|
| 938 |
+
INFO:megatron.core.datasets.indexed_dataset:Load the _IndexReader from /workspace/data/wikitext/wikitext103_gpt2_text_document.idx
|
| 939 |
+
INFO:megatron.core.datasets.indexed_dataset: Extract the sequence lengths
|
| 940 |
+
INFO:megatron.core.datasets.indexed_dataset: Extract the sequence pointers
|
| 941 |
+
INFO:megatron.core.datasets.indexed_dataset: Extract the document indices
|
| 942 |
+
INFO:megatron.core.datasets.indexed_dataset:> total number of sequences: 1165029
|
| 943 |
+
INFO:megatron.core.datasets.indexed_dataset:> total number of documents: 1801350
|
| 944 |
+
INFO:megatron.core.datasets.gpt_dataset:Load the GPTDataset train indices
|
| 945 |
+
INFO:megatron.core.datasets.gpt_dataset: Load the document index from 44ed1ee5bcb9e5b030387b8d27a6c406-GPTDataset-train-document_index.npy
|
| 946 |
+
INFO:megatron.core.datasets.gpt_dataset: Load the sample index from 44ed1ee5bcb9e5b030387b8d27a6c406-GPTDataset-train-sample_index.npy
|
| 947 |
+
INFO:megatron.core.datasets.gpt_dataset: Load the shuffle index from 44ed1ee5bcb9e5b030387b8d27a6c406-GPTDataset-train-shuffle_index.npy
|
| 948 |
+
INFO:megatron.core.datasets.gpt_dataset:> total number of samples: 29073
|
| 949 |
+
> finished creating GPT datasets ...
|
| 950 |
+
[after dataloaders are built] datetime: 2026-08-30 06:50:05
|
| 951 |
+
done with setup ...
|
| 952 |
+
training ...
|
| 953 |
+
Overwriting rerun_state_machine.current_iteration from -1 to 0...
|
| 954 |
+
(min, max) time across ranks (ms):
|
| 955 |
+
model-and-optimizer-setup ......................: (325.55, 328.11)
|
| 956 |
+
train/valid/test-data-iterators-setup ..........: (64.46, 68.67)
|
| 957 |
+
[before the start of training step] datetime: 2026-08-30 06:50:05
|
| 958 |
+
/workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:334: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
|
| 959 |
+
warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
|
| 960 |
+
/workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:334: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
|
| 961 |
+
warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
|
| 962 |
+
/workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:336: UserWarning: The use_host_meta is deprecated, it will be removed in the future.
|
| 963 |
+
warnings.warn("The use_host_meta is deprecated, it will be removed in the future.")
|
| 964 |
+
/workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:336: UserWarning: The use_host_meta is deprecated, it will be removed in the future.
|
| 965 |
+
warnings.warn("The use_host_meta is deprecated, it will be removed in the future.")
|
| 966 |
+
/workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:334: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
|
| 967 |
+
warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
|
| 968 |
+
/workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:336: UserWarning: The use_host_meta is deprecated, it will be removed in the future.
|
| 969 |
+
warnings.warn("The use_host_meta is deprecated, it will be removed in the future.")
|
| 970 |
+
/workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:334: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
|
| 971 |
+
warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
|
| 972 |
+
/workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:334: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
|
| 973 |
+
warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
|
| 974 |
+
/workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:336: UserWarning: The use_host_meta is deprecated, it will be removed in the future.
|
| 975 |
+
warnings.warn("The use_host_meta is deprecated, it will be removed in the future.")
|
| 976 |
+
/workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:336: UserWarning: The use_host_meta is deprecated, it will be removed in the future.
|
| 977 |
+
warnings.warn("The use_host_meta is deprecated, it will be removed in the future.")
|
| 978 |
+
/workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:334: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
|
| 979 |
+
warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
|
| 980 |
+
/workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:334: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
|
| 981 |
+
warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
|
| 982 |
+
/workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:336: UserWarning: The use_host_meta is deprecated, it will be removed in the future.
|
| 983 |
+
warnings.warn("The use_host_meta is deprecated, it will be removed in the future.")
|
| 984 |
+
/workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:336: UserWarning: The use_host_meta is deprecated, it will be removed in the future.
|
| 985 |
+
warnings.warn("The use_host_meta is deprecated, it will be removed in the future.")
|
| 986 |
+
/workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:334: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
|
| 987 |
+
warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
|
| 988 |
+
/workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:336: UserWarning: The use_host_meta is deprecated, it will be removed in the future.
|
| 989 |
+
warnings.warn("The use_host_meta is deprecated, it will be removed in the future.")
|
| 990 |
+
/workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:482: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
|
| 991 |
+
warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
|
| 992 |
+
/workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:482: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
|
| 993 |
+
warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
|
| 994 |
+
/workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:482: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
|
| 995 |
+
warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
|
| 996 |
+
/workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:482: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
|
| 997 |
+
warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
|
| 998 |
+
/workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:482: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
|
| 999 |
+
warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
|
| 1000 |
+
/workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:482: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
|
| 1001 |
+
warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
|
| 1002 |
+
/workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:482: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
|
| 1003 |
+
warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
|
| 1004 |
+
/workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:482: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
|
| 1005 |
+
warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
|
| 1006 |
+
/workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:334: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
|
| 1007 |
+
warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
|
| 1008 |
+
/workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:336: UserWarning: The use_host_meta is deprecated, it will be removed in the future.
|
| 1009 |
+
warnings.warn("The use_host_meta is deprecated, it will be removed in the future.")
|
| 1010 |
+
/workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:334: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
|
| 1011 |
+
warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
|
| 1012 |
+
/workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:336: UserWarning: The use_host_meta is deprecated, it will be removed in the future.
|
| 1013 |
+
warnings.warn("The use_host_meta is deprecated, it will be removed in the future.")
|
| 1014 |
+
/workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:334: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
|
| 1015 |
+
warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
|
| 1016 |
+
/workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:336: UserWarning: The use_host_meta is deprecated, it will be removed in the future.
|
| 1017 |
+
warnings.warn("The use_host_meta is deprecated, it will be removed in the future.")
|
| 1018 |
+
/workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:334: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
|
| 1019 |
+
warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
|
| 1020 |
+
/workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:336: UserWarning: The use_host_meta is deprecated, it will be removed in the future.
|
| 1021 |
+
warnings.warn("The use_host_meta is deprecated, it will be removed in the future.")
|
| 1022 |
+
/workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:334: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
|
| 1023 |
+
warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
|
| 1024 |
+
/workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:336: UserWarning: The use_host_meta is deprecated, it will be removed in the future.
|
| 1025 |
+
warnings.warn("The use_host_meta is deprecated, it will be removed in the future.")
|
| 1026 |
+
/workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:334: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
|
| 1027 |
+
warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
|
| 1028 |
+
/workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:336: UserWarning: The use_host_meta is deprecated, it will be removed in the future.
|
| 1029 |
+
warnings.warn("The use_host_meta is deprecated, it will be removed in the future.")
|
| 1030 |
+
/workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:334: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
|
| 1031 |
+
warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
|
| 1032 |
+
/workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:336: UserWarning: The use_host_meta is deprecated, it will be removed in the future.
|
| 1033 |
+
warnings.warn("The use_host_meta is deprecated, it will be removed in the future.")
|
| 1034 |
+
/workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:334: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
|
| 1035 |
+
warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
|
| 1036 |
+
/workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:336: UserWarning: The use_host_meta is deprecated, it will be removed in the future.
|
| 1037 |
+
warnings.warn("The use_host_meta is deprecated, it will be removed in the future.")
|
| 1038 |
+
/workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:482: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
|
| 1039 |
+
warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
|
| 1040 |
+
/workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:482: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
|
| 1041 |
+
warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
|
| 1042 |
+
/workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:482: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
|
| 1043 |
+
warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
|
| 1044 |
+
/workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:482: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
|
| 1045 |
+
warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
|
| 1046 |
+
/workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:482: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
|
| 1047 |
+
warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
|
| 1048 |
+
/workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:482: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
|
| 1049 |
+
warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
|
| 1050 |
+
/workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:482: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
|
| 1051 |
+
warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
|
| 1052 |
+
/workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:482: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
|
| 1053 |
+
warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
|
| 1054 |
+
/workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:334: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
|
| 1055 |
+
warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
|
| 1056 |
+
/workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:334: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
|
| 1057 |
+
warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
|
| 1058 |
+
/workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:334: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
|
| 1059 |
+
warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
|
| 1060 |
+
/workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:334: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
|
| 1061 |
+
warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
|
| 1062 |
+
/workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:334: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
|
| 1063 |
+
warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
|
| 1064 |
+
/workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:334: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
|
| 1065 |
+
warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
|
| 1066 |
+
/workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:334: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
|
| 1067 |
+
warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
|
| 1068 |
+
/workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:334: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
|
| 1069 |
+
warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
|
| 1070 |
+
/workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:336: UserWarning: The use_host_meta is deprecated, it will be removed in the future.
|
| 1071 |
+
warnings.warn("The use_host_meta is deprecated, it will be removed in the future.")
|
| 1072 |
+
/workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:336: UserWarning: The use_host_meta is deprecated, it will be removed in the future.
|
| 1073 |
+
warnings.warn("The use_host_meta is deprecated, it will be removed in the future.")
|
| 1074 |
+
/workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:336: UserWarning: The use_host_meta is deprecated, it will be removed in the future.
|
| 1075 |
+
warnings.warn("The use_host_meta is deprecated, it will be removed in the future.")
|
| 1076 |
+
/workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:336: UserWarning: The use_host_meta is deprecated, it will be removed in the future.
|
| 1077 |
+
warnings.warn("The use_host_meta is deprecated, it will be removed in the future.")
|
| 1078 |
+
/workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:336: UserWarning: The use_host_meta is deprecated, it will be removed in the future.
|
| 1079 |
+
warnings.warn("The use_host_meta is deprecated, it will be removed in the future.")
|
| 1080 |
+
/workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:336: UserWarning: The use_host_meta is deprecated, it will be removed in the future.
|
| 1081 |
+
warnings.warn("The use_host_meta is deprecated, it will be removed in the future.")
|
| 1082 |
+
/workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:336: UserWarning: The use_host_meta is deprecated, it will be removed in the future.
|
| 1083 |
+
warnings.warn("The use_host_meta is deprecated, it will be removed in the future.")
|
| 1084 |
+
/workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:336: UserWarning: The use_host_meta is deprecated, it will be removed in the future.
|
| 1085 |
+
warnings.warn("The use_host_meta is deprecated, it will be removed in the future.")
|
tiera-rerun-fixed-moonep/echo/a2/a2_matched_gbs16/results.tsv
ADDED
|
@@ -0,0 +1,5 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
config backend policy slots seq gbs cap recompute rep status iter_ms_mean iter_ms_min peak_mib run_dir
|
| 2 |
+
G1 echo matched 1 4096 16 2.40625 none 1 OK 466.86 462.10 161723 /workspace/tiera-rerun-fixed-moonep/echo/a2/a2_matched_gbs16/G1__echo__matched__seq4096__gbs16__cap2p40625/rep1
|
| 3 |
+
G1 echo matched 1 4096 16 2.40625 none 2 OK 466.67 462.10 161723 /workspace/tiera-rerun-fixed-moonep/echo/a2/a2_matched_gbs16/G1__echo__matched__seq4096__gbs16__cap2p40625/rep2
|
| 4 |
+
G1 echo matched 1 4096 16 2.40625 none 3 OK 466.91 461.80 161723 /workspace/tiera-rerun-fixed-moonep/echo/a2/a2_matched_gbs16/G1__echo__matched__seq4096__gbs16__cap2p40625/rep3
|
| 5 |
+
G2 echo matched 4 4096 16 1.08203125 none 1 OK 415.60 413.70 145689 /workspace/tiera-rerun-fixed-moonep/echo/a2/a2_matched_gbs16/G2__echo__matched__seq4096__gbs16__cap1p08203125/rep1
|
tiera-rerun-fixed-moonep/echo/a2/manifest.txt
ADDED
|
@@ -0,0 +1,19 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
timestamp: 2026-08-30T06:44:28+00:00
|
| 2 |
+
host: sp-aiinfra-interactive-aghosh-7jtjl-8xb200-moe-cg-0
|
| 3 |
+
gpu: NVIDIA B200 x8
|
| 4 |
+
driver: 580.95.05
|
| 5 |
+
study_scripts_branch: granularity-echo
|
| 6 |
+
study_scripts_commit: ff2c833
|
| 7 |
+
campaign: echo
|
| 8 |
+
backend: echo
|
| 9 |
+
phase: a2
|
| 10 |
+
gbs: 16
|
| 11 |
+
seq: 4096
|
| 12 |
+
configs: G1 G2 G3 G4
|
| 13 |
+
capacity: 2.40625
|
| 14 |
+
repeats: 3
|
| 15 |
+
nsys: 0
|
| 16 |
+
megatron_repo: /workspace/Megatron-MoE-Echo
|
| 17 |
+
megatron_branch_required: bracing-regions-through-kernels
|
| 18 |
+
megatron_branch_actual: bracing-regions-through-kernels
|
| 19 |
+
megatron_commit: 9a236793
|
tiera-rerun-fixed-moonep/echo/a2/pip_freeze.txt
ADDED
|
@@ -0,0 +1,317 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
absl-py==2.3.0
|
| 2 |
+
aiohappyeyeballs @ file:///rapids/aiohappyeyeballs-2.6.1-py3-none-any.whl#sha256=f349ba8f4b75cb25c99c5c2d84e997e485204d2902a9597802b0371f09331fb8
|
| 3 |
+
aiohttp @ file:///rapids/aiohttp-3.12.7-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl#sha256=94f98e0e5a49f89b252e115844f756c04fc8050f38252a32a3dd994ce8121f10
|
| 4 |
+
aiosignal @ file:///rapids/aiosignal-1.3.2-py2.py3-none-any.whl#sha256=45cde58e409a301715980c2b01d0c28bdde3770d8290b5eb2173759d9acb31a5
|
| 5 |
+
annotated-types==0.7.0
|
| 6 |
+
anyio==4.9.0
|
| 7 |
+
apex @ file:///opt/pytorch/apex/dist/apex-0.1-cp312-cp312-linux_x86_64.whl#sha256=a1508e4a386362e9c8f46c92ccada9110e421212b964c694235d0fd5a224b170
|
| 8 |
+
argon2-cffi==25.1.0
|
| 9 |
+
argon2-cffi-bindings==21.2.0
|
| 10 |
+
arrow==1.3.0
|
| 11 |
+
asciitree @ file:///rapids/asciitree-0.3.3-py3-none-any.whl#sha256=3e507ccfaeb417823158091fdb9d26b0205f6a59ba680b479a71db65c73d01d4
|
| 12 |
+
asttokens==3.0.0
|
| 13 |
+
astunparse==1.6.3
|
| 14 |
+
async-lru==2.0.5
|
| 15 |
+
attrs==25.3.0
|
| 16 |
+
audioread==3.0.1
|
| 17 |
+
babel==2.17.0
|
| 18 |
+
beautifulsoup4==4.13.4
|
| 19 |
+
black==25.1.0
|
| 20 |
+
bleach==6.2.0
|
| 21 |
+
blis==0.7.11
|
| 22 |
+
cachetools @ file:///rapids/cachetools-6.0.0-py3-none-any.whl#sha256=82e73ba88f7b30228b5507dce1a1f878498fc669d972aef2dde4f3a3c24f103e
|
| 23 |
+
catalogue==2.0.10
|
| 24 |
+
certifi==2025.4.26
|
| 25 |
+
cffi==1.17.1
|
| 26 |
+
charset-normalizer==3.4.2
|
| 27 |
+
click==8.5.0
|
| 28 |
+
cloudpathlib==0.21.1
|
| 29 |
+
cloudpickle @ file:///rapids/cloudpickle-3.1.1-py3-none-any.whl#sha256=c8c5a44295039331ee9dad40ba100a9c7297b6f988e50e87ccdf3765a668350e
|
| 30 |
+
cmake==3.31.6
|
| 31 |
+
comm==0.2.2
|
| 32 |
+
confection==0.1.5
|
| 33 |
+
contourpy==1.3.2
|
| 34 |
+
cuda-bindings @ file:///rapids/cuda_bindings-12.9.0-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl#sha256=ff0e28d1e34758654b9c961e1f55e4786e49aee6a4dbceaf3cc24c46c672df7e
|
| 35 |
+
cuda-python @ file:///rapids/cuda_python-12.9.0-py3-none-any.whl#sha256=926acba49b2c0a0374c61b7c98f337c085199cf51cdfe4d6423c4129c20547a7
|
| 36 |
+
cudf @ file:///rapids/cudf-25.4.0-cp312-cp312-linux_x86_64.whl#sha256=b779a8e8c0e8577b9b560895958496da211bfac49cdc4f11a49e8d18cb82e348
|
| 37 |
+
cudf-polars @ file:///rapids/cudf_polars-25.4.0-py3-none-any.whl#sha256=227b6c5147e93ea86282c8f7219118f916769fa2ca0cdd931e97a148aa66b869
|
| 38 |
+
cugraph @ file:///rapids/cugraph-25.4.0-cp312-cp312-linux_x86_64.whl#sha256=b837c2dc2f7b235c702e4f8df64479b8fcbb342f848516d5faf0cdc4b5f31f29
|
| 39 |
+
cugraph-service-client @ file:///rapids/cugraph_service_client-25.4.0-py3-none-any.whl#sha256=7e568c3e41154b9752f5173555ffcc282ac52d72343feeb79653b2e2b1232470
|
| 40 |
+
cugraph-service-server @ file:///rapids/cugraph_service_server-25.4.0-py3-none-any.whl#sha256=a2830dcffafcccb16e12ebf85b98f8f247869d38c01ab4dd8664a1e7439d31be
|
| 41 |
+
cuml @ file:///rapids/cuml-25.4.0-cp312-cp312-linux_x86_64.whl#sha256=1e90a464a18b094e24797f1676688c5d7696c4cf97c934504214a53d5aa09aa5
|
| 42 |
+
cupy-cuda12x @ file:///rapids/cupy_cuda12x-13.3.0-cp312-cp312-linux_x86_64.whl#sha256=6a2dfa0aa1e8189a10d1534276c08f1d2f73b747fd28bfbe7b9f4ef66d9af73f
|
| 43 |
+
cuvs @ file:///rapids/cuvs-25.4.0-cp312-cp312-linux_x86_64.whl#sha256=b12f6b9f80e053a3b38542c2947d2516e468c02cad54a13d8b1a4a61d5aa7914
|
| 44 |
+
cycler==0.12.1
|
| 45 |
+
cymem==2.0.11
|
| 46 |
+
Cython==3.1.1
|
| 47 |
+
dask @ file:///rapids/dask-2025.2.0-py3-none-any.whl#sha256=f0fdeef6ceb0a06569d456c9e704f220f7f54e80f3a6ea42ab98cea6bc642b6e
|
| 48 |
+
dask-cuda @ file:///rapids/dask_cuda-25.4.0-py3-none-any.whl#sha256=442d87da0a95d00ca2a262b64c76e785a32ad905c82000fa2227932f81390e19
|
| 49 |
+
dask-cudf @ file:///rapids/dask_cudf-25.4.0-py3-none-any.whl#sha256=5e7cea0b26daea17da6e0bb457435e24460f76ba740358a3a18e239d6015c95c
|
| 50 |
+
datasets==5.0.1
|
| 51 |
+
debugpy==1.8.14
|
| 52 |
+
decorator==5.2.1
|
| 53 |
+
-e git+https://github.com/deepseek-ai/DeepEP.git@34152ae28f80bcc3ee38d7a12cb2ad87cfd4ea72#egg=deep_ep
|
| 54 |
+
defusedxml==0.7.1
|
| 55 |
+
dill==0.4.1
|
| 56 |
+
distributed @ file:///rapids/distributed-2025.2.0-py3-none-any.whl#sha256=368462084be47092d0a835f6c64c9b852da64dc88302b76d5f93bd5131228c67
|
| 57 |
+
distributed-ucxx @ file:///rapids/distributed_ucxx-0.43.0-py3-none-any.whl#sha256=94f60d0ccb0c94ba8dbfa1eaf455874d18aaab17e2ae218c38590e9fdc2d721d
|
| 58 |
+
dm-tree==0.1.9
|
| 59 |
+
einops==0.8.1
|
| 60 |
+
execnet==2.1.1
|
| 61 |
+
executing==2.2.0
|
| 62 |
+
expecttest==0.3.0
|
| 63 |
+
fasteners @ file:///rapids/fasteners-0.19-py3-none-any.whl#sha256=758819cb5d94cdedf4e836988b74de396ceacb8e2794d21f82d131fd9ee77237
|
| 64 |
+
fastjsonschema==2.21.1
|
| 65 |
+
fastrlock @ file:///rapids/fastrlock-0.8.3-cp312-cp312-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_28_x86_64.whl#sha256=5f13ec08f1adb1aa916c384b05ecb7dbebb8df9ea81abd045f60941c6283a670
|
| 66 |
+
filelock==3.18.0
|
| 67 |
+
flash_attn @ file:///opt/pytorch/flash_attn-2.7.4.post1-cp312-cp312-linux_x86_64.whl#sha256=a10455d6c56b387785b7413fd0bbcb392b42b3d4f4b88fda5d86cc8a9ce9182a
|
| 68 |
+
fonttools==4.58.1
|
| 69 |
+
fqdn==1.5.1
|
| 70 |
+
frozenlist @ file:///rapids/frozenlist-1.6.0-cp312-cp312-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl#sha256=4e1be9111cb6756868ac242b3c2bd1f09d9aea09846e4f5c23715e7afb647103
|
| 71 |
+
fsspec==2025.5.1
|
| 72 |
+
gast==0.6.0
|
| 73 |
+
grpcio==1.62.1
|
| 74 |
+
h11==0.16.0
|
| 75 |
+
hf-xet==1.6.0
|
| 76 |
+
httpcore==1.0.9
|
| 77 |
+
httpx==0.28.1
|
| 78 |
+
huggingface_hub==1.29.0
|
| 79 |
+
hypothesis==6.130.8
|
| 80 |
+
idna==3.10
|
| 81 |
+
importlib_metadata==8.7.0
|
| 82 |
+
iniconfig==2.1.0
|
| 83 |
+
intel-openmp==2021.4.0
|
| 84 |
+
ipykernel==6.29.5
|
| 85 |
+
ipython==9.3.0
|
| 86 |
+
ipython_pygments_lexers==1.1.1
|
| 87 |
+
isoduration==20.11.0
|
| 88 |
+
isort==6.0.1
|
| 89 |
+
jedi==0.19.2
|
| 90 |
+
Jinja2==3.1.6
|
| 91 |
+
joblib==1.5.1
|
| 92 |
+
json5==0.12.0
|
| 93 |
+
jsonpointer==3.0.0
|
| 94 |
+
jsonschema==4.24.0
|
| 95 |
+
jsonschema-specifications==2025.4.1
|
| 96 |
+
jupyter-events==0.12.0
|
| 97 |
+
jupyter-lsp==2.2.5
|
| 98 |
+
jupyter_client==8.6.3
|
| 99 |
+
jupyter_core==5.8.1
|
| 100 |
+
jupyter_server==2.16.0
|
| 101 |
+
jupyter_server_terminals==0.5.3
|
| 102 |
+
jupyterlab==4.4.3
|
| 103 |
+
jupyterlab_code_formatter==3.0.2
|
| 104 |
+
jupyterlab_pygments==0.3.0
|
| 105 |
+
jupyterlab_server==2.27.3
|
| 106 |
+
jupyterlab_tensorboard_pro @ file:///builder/jupyterlab_tensorboard_pro-4.0.0-py2.py3-none-any.whl#sha256=6bfcb5ced3358eb72150f4b2ab91312faef2f0ca14d845e8379d8fa52df6902e
|
| 107 |
+
jupytext==1.17.2
|
| 108 |
+
kiwisolver==1.4.8
|
| 109 |
+
kvikio @ file:///rapids/kvikio-25.4.0-cp312-cp312-linux_x86_64.whl#sha256=8b40fd8ea9e4a2974c04b50b7fc26b06094ac807f9a233281d0bafbaa71b1d85
|
| 110 |
+
langcodes==3.5.0
|
| 111 |
+
language_data==1.3.0
|
| 112 |
+
lazy_loader==0.4
|
| 113 |
+
libcudf @ file:///rapids/libcudf-25.4.0-py3-none-linux_x86_64.whl#sha256=238320b4958c1737bba2a609673e2539a78216b811b5498a68e00395d9b021a4
|
| 114 |
+
libcugraph @ file:///rapids/libcugraph-25.4.0-py3-none-linux_x86_64.whl#sha256=29085e901b2f7569ff2653bb1355fe6fad20f2fcaa585f5649da9867d8df766d
|
| 115 |
+
libcuml @ file:///rapids/libcuml-25.4.0-py3-none-linux_x86_64.whl#sha256=8e2e444c72b0e995e15a48da37ff12fb410a47a2baa69ba86ccc9b1a3db91e8c
|
| 116 |
+
libcuvs @ file:///rapids/libcuvs-25.4.0-py3-none-linux_x86_64.whl#sha256=e2f8db61c92b769c590dd752f43439ddd62b2f6592a3c337362b6c1ef220c3b9
|
| 117 |
+
libkvikio @ file:///rapids/libkvikio-25.4.0-py3-none-linux_x86_64.whl#sha256=da8a5713b05fc478bf40529a3e0fb61a2d605bb5a70ca4d461f96a6005dbe8b7
|
| 118 |
+
libraft @ file:///rapids/libraft-25.4.0-py3-none-linux_x86_64.whl#sha256=b921dec005aa4a321dd156776a18376feb47c96a5b67eec125c8a8e19edd1e24
|
| 119 |
+
librmm @ file:///rapids/librmm-25.4.0-py3-none-linux_x86_64.whl#sha256=ff951811bda729d4fa1109ab906225ae1f489a6ec2eeab5d1b674cc34bd4422f
|
| 120 |
+
librmm-cu12 @ file:///rapids/librmm_cu12-25.4.0-py3-none-any.whl#sha256=29296ef2b0e90679a0db280e86015be12d6ab9f4e62a181aa085f62dbadd06b3
|
| 121 |
+
librosa==0.11.0
|
| 122 |
+
libucxx @ file:///rapids/libucxx-0.43.0-py3-none-linux_x86_64.whl#sha256=f15849d7ac2fb68f0fd8998838a13321ddcfb0c594f7e6cbc401a3bcc4750566
|
| 123 |
+
lightning-thunder @ file:///opt/pytorch/lightning-thunder
|
| 124 |
+
lightning-utilities==0.14.3
|
| 125 |
+
lintrunner==0.12.7
|
| 126 |
+
llvmlite==0.42.0
|
| 127 |
+
locket @ file:///rapids/locket-1.0.0-py2.py3-none-any.whl#sha256=b6c819a722f7b6bd955b80781788e4a66a55628b858d347536b7e81325a3a5e3
|
| 128 |
+
looseversion==1.3.0
|
| 129 |
+
marisa-trie==1.2.1
|
| 130 |
+
Markdown==3.8
|
| 131 |
+
markdown-it-py==3.0.0
|
| 132 |
+
MarkupSafe==3.0.2
|
| 133 |
+
matplotlib==3.10.3
|
| 134 |
+
matplotlib-inline==0.1.7
|
| 135 |
+
mdit-py-plugins==0.4.2
|
| 136 |
+
mdurl==0.1.2
|
| 137 |
+
-e git+ssh://git@github.com/sriram-17-17/my-Megatron-MoE-Echo.git@9a23679390d7ab1fe462bc61fa31d90d393db49c#egg=megatron_core
|
| 138 |
+
mistune==3.1.3
|
| 139 |
+
mkl==2021.1.1
|
| 140 |
+
mkl-devel==2021.1.1
|
| 141 |
+
mkl-include==2021.1.1
|
| 142 |
+
ml_dtypes==0.6.0
|
| 143 |
+
mock==5.2.0
|
| 144 |
+
-e git+ssh://git@github.com/sriram-17-17/MoonEP.git@d8bc6667fad7177e6e09b637d773fa67f08bae93#egg=moonep
|
| 145 |
+
mpmath==1.3.0
|
| 146 |
+
msgpack==1.1.0
|
| 147 |
+
multidict @ file:///rapids/multidict-6.4.4-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl#sha256=a920f9cf2abdf6e493c519492d892c362007f113c94da4c239ae88429835bad1
|
| 148 |
+
multiprocess==0.70.19
|
| 149 |
+
murmurhash==1.0.13
|
| 150 |
+
mypy_extensions==1.1.0
|
| 151 |
+
nbclient==0.10.2
|
| 152 |
+
nbconvert==7.16.6
|
| 153 |
+
nbformat==5.10.4
|
| 154 |
+
nest-asyncio==1.6.0
|
| 155 |
+
networkx==3.5
|
| 156 |
+
ninja==1.11.1.4
|
| 157 |
+
notebook==7.4.3
|
| 158 |
+
notebook_shim==0.2.4
|
| 159 |
+
numba==0.59.1
|
| 160 |
+
numba-cuda @ file:///rapids/numba_cuda-0.4.0-py3-none-any.whl#sha256=5b95a5f42068771e37142944d44fe844fa87f1b2f89051461a4cb0af91bcc28a
|
| 161 |
+
numcodecs @ file:///rapids/numcodecs-0.13.1-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl#sha256=da2230484e6102e5fa3cc1a5dd37ca1f92dfbd183d91662074d6f7574e3e8f53
|
| 162 |
+
numpy==1.26.4
|
| 163 |
+
nvdlfw_inspect @ git+https://github.com/NVIDIA/nvidia-dlfw-inspect.git@6b60eb0e675606fb2fbbfcb12667ecbec75eaf87
|
| 164 |
+
nvfuser==0.2.27a0+9bf5aca
|
| 165 |
+
nvidia-cuda-cccl-cu12==12.9.27
|
| 166 |
+
nvidia-cudnn-frontend @ file:///opt/pytorch/lightning-thunder/tmp/cudnn_frontend
|
| 167 |
+
nvidia-cutlass-dsl==4.4.2
|
| 168 |
+
nvidia-cutlass-dsl-libs-base==4.4.2
|
| 169 |
+
nvidia-dali-cuda120==1.50.0
|
| 170 |
+
nvidia-ml-py @ file:///rapids/nvidia_ml_py-12.575.51-py3-none-any.whl#sha256=eb8641800d98ce40a22f479873f34b482e214a7e80349c63be51c3919845446e
|
| 171 |
+
nvidia-modelopt==0.29.0
|
| 172 |
+
nvidia-modelopt-core==0.29.0
|
| 173 |
+
nvidia-nvcomp-cu12==4.2.0.14
|
| 174 |
+
nvidia-nvimgcodec-cu12==0.5.0.13
|
| 175 |
+
nvidia-nvjpeg-cu12==12.4.0.16
|
| 176 |
+
nvidia-nvjpeg2k-cu12==0.8.1.40
|
| 177 |
+
nvidia-nvshmem-cu12==3.7.2
|
| 178 |
+
nvidia-nvtiff-cu12==0.5.0.67
|
| 179 |
+
nvidia-resiliency-ext==0.4.0
|
| 180 |
+
nvtx @ file:///rapids/nvtx-0.2.11-cp312-cp312-linux_x86_64.whl#sha256=73521e37cba42b4db0b3cd68fc5eb814e55c500a61eeed6edcdf6746315ac53e
|
| 181 |
+
nx-cugraph @ file:///rapids/nx_cugraph-25.4.0-py3-none-any.whl#sha256=e625444486cc899a77bd48e7da76a2edd95c0006dd95fbf323745feae7f33111
|
| 182 |
+
onnx @ file:///opt/pytorch/pytorch/third_party/onnx
|
| 183 |
+
onnx-ir==1.0.0
|
| 184 |
+
onnxscript==0.7.1
|
| 185 |
+
opt_einsum==3.4.0
|
| 186 |
+
optree==0.16.0
|
| 187 |
+
overrides==7.7.0
|
| 188 |
+
packaging==23.2
|
| 189 |
+
pandas @ file:///rapids/pandas-2.2.3-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl#sha256=fffb8ae78d8af97f849404f21411c95062db1496aeb3e56f146f0355c9989319
|
| 190 |
+
pandocfilters==1.5.1
|
| 191 |
+
parso==0.8.4
|
| 192 |
+
partd @ file:///rapids/partd-1.4.2-py3-none-any.whl#sha256=978e4ac767ec4ba5b86c6eaa52e5a2a3bc748a2ca839e8cc798f1cc6ce6efb0f
|
| 193 |
+
pathspec==0.12.1
|
| 194 |
+
pexpect==4.9.0
|
| 195 |
+
pillow==11.2.1
|
| 196 |
+
platformdirs==4.3.8
|
| 197 |
+
pluggy==1.6.0
|
| 198 |
+
ply @ file:///rapids/ply-3.11-py2.py3-none-any.whl#sha256=096f9b8350b65ebd2fd1346b12452efe5b9607f7482813ffca50c22722a807ce
|
| 199 |
+
polars @ file:///rapids/polars-1.25.2-cp39-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl#sha256=f7fcbb4f476784384ccda48757fca4e8c2e2c5a0a3aef3717aaf56aee4e30e09
|
| 200 |
+
polygraphy==0.49.20
|
| 201 |
+
pooch==1.8.2
|
| 202 |
+
preshed==3.0.10
|
| 203 |
+
prometheus_client==0.22.1
|
| 204 |
+
prompt_toolkit==3.0.51
|
| 205 |
+
propcache @ file:///rapids/propcache-0.3.1-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl#sha256=563f9d8c03ad645597b8d010ef4e9eab359faeb11a0a2ac9f7b4bc8c28ebef25
|
| 206 |
+
protobuf==4.24.4
|
| 207 |
+
psutil==7.0.0
|
| 208 |
+
ptyprocess==0.7.0
|
| 209 |
+
PuLP==3.2.1
|
| 210 |
+
pure_eval==0.2.3
|
| 211 |
+
pyarrow==25.0.1
|
| 212 |
+
pybind11==2.13.6
|
| 213 |
+
pybind11_global==2.13.6
|
| 214 |
+
pycocotools @ git+https://github.com/nvidia/cocoapi.git@08350a157b41f0755a536441b5abec89ab934c5b#subdirectory=PythonAPI
|
| 215 |
+
pycparser==2.22
|
| 216 |
+
pydantic==2.11.5
|
| 217 |
+
pydantic_core==2.33.2
|
| 218 |
+
Pygments==2.19.1
|
| 219 |
+
pylibcudf @ file:///rapids/pylibcudf-25.4.0-cp312-cp312-linux_x86_64.whl#sha256=401b8bfd022d06f84e120779ca7c3d01a28b68762aaa8f7873b10d6b1975cf6a
|
| 220 |
+
pylibcugraph @ file:///rapids/pylibcugraph-25.4.0-cp312-cp312-linux_x86_64.whl#sha256=061c7389a88c815b54fd5d3a3584fb87bf693e5602f5be335260d0ba575ba557
|
| 221 |
+
pylibcugraphops @ file:///rapids/pylibcugraphops-25.4.0-cp312-cp312-linux_x86_64.whl#sha256=a25ef4b22166c80718d5c7e5bddb755e81a2fe340fa5becd0ac36ec3edfb208a
|
| 222 |
+
pylibraft @ file:///rapids/pylibraft-25.4.0-cp312-cp312-linux_x86_64.whl#sha256=f2ecd23fe51f9c2caa87bbbe85cb32fcaff4ecfb52ceb9a9991bc50f82782c2e
|
| 223 |
+
pylibwholegraph @ file:///rapids/pylibwholegraph-25.4.0-cp312-cp312-linux_x86_64.whl#sha256=b82913d1e38bda76d8c272fbd8cc9072d987209d3d47cc93a464a37c2925aca2
|
| 224 |
+
pynvjitlink @ file:///rapids/pynvjitlink-0.3.0-cp312-cp312-linux_x86_64.whl#sha256=066caf1314730643d3b48c9847bd65789defdd6a7c78993176eb3d8bbe3f4919
|
| 225 |
+
pynvml @ file:///rapids/pynvml-12.0.0-py3-none-any.whl#sha256=fdff84b62a27dbe98e08e1a647eb77342bef1aebe0878bcd15e99a83fcbecb9e
|
| 226 |
+
pyparsing==3.2.3
|
| 227 |
+
pytest==8.1.1
|
| 228 |
+
pytest-flakefinder==1.1.0
|
| 229 |
+
pytest-rerunfailures==15.1
|
| 230 |
+
pytest-shard==0.1.2
|
| 231 |
+
pytest-xdist==3.7.0
|
| 232 |
+
python-dateutil==2.9.0.post0
|
| 233 |
+
python-hostlist==2.2.1
|
| 234 |
+
python-json-logger==3.3.0
|
| 235 |
+
pytorch-triton @ file:///tmp/dist/pytorch_triton-3.3.0%2Bgit96316ce52.nvinternal-cp312-cp312-linux_x86_64.whl#sha256=6d6921772b9dfd0243c055e0d054de42a113e92cabcfdd1d3e80a5ad9c08cf7d
|
| 236 |
+
pytz @ file:///rapids/pytz-2023.4-py2.py3-none-any.whl#sha256=f90ef520d95e7c46951105338d918664ebfd6f1d995bd7d153127ce90efafa6a
|
| 237 |
+
PyYAML==6.0.2
|
| 238 |
+
pyzmq==26.4.0
|
| 239 |
+
raft-dask @ file:///rapids/raft_dask-25.4.0-cp312-cp312-linux_x86_64.whl#sha256=36cf37ba6ce35454c4729c2cb597a360f22780eefb11d514a16a07137da5eb6c
|
| 240 |
+
rapids-dask-dependency @ file:///rapids/rapids_dask_dependency-25.4.0a0-py3-none-any.whl#sha256=484faca626e7760112a7d8ad3426f5bdd720b609006d8dca87c2607076a348bd
|
| 241 |
+
rapids-logger @ file:///rapids/rapids_logger-0.1.18-py3-none-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl#sha256=3617b2d86291ac6cb957542895f20fc9737ace29773ca90337e6fb04c73bd120
|
| 242 |
+
referencing==0.36.2
|
| 243 |
+
regex==2024.11.6
|
| 244 |
+
requests==2.32.3
|
| 245 |
+
rfc3339-validator==0.1.4
|
| 246 |
+
rfc3986-validator==0.1.1
|
| 247 |
+
rich==14.0.0
|
| 248 |
+
rmm @ file:///rapids/rmm-25.4.0-cp312-cp312-linux_x86_64.whl#sha256=1824bcab3836296666980a997ddcb9c4bca24dfb873012cfa0a7ecea583ca712
|
| 249 |
+
rpds-py==0.25.1
|
| 250 |
+
safetensors==0.5.3
|
| 251 |
+
scikit-learn==1.6.1
|
| 252 |
+
scipy==1.15.3
|
| 253 |
+
Send2Trash==1.8.3
|
| 254 |
+
setuptools==78.1.1
|
| 255 |
+
shellingham==1.5.4
|
| 256 |
+
six==1.16.0
|
| 257 |
+
smart-open==7.1.0
|
| 258 |
+
sniffio==1.3.1
|
| 259 |
+
sortedcontainers==2.4.0
|
| 260 |
+
soundfile==0.13.1
|
| 261 |
+
soupsieve==2.7
|
| 262 |
+
soxr==0.5.0.post1
|
| 263 |
+
spacy==3.7.5
|
| 264 |
+
spacy-legacy==3.0.12
|
| 265 |
+
spacy-loggers==1.0.5
|
| 266 |
+
srsly==2.5.1
|
| 267 |
+
stack-data==0.6.3
|
| 268 |
+
sympy==1.14.0
|
| 269 |
+
tabulate==0.9.0
|
| 270 |
+
tbb==2021.13.1
|
| 271 |
+
tblib @ file:///rapids/tblib-3.1.0-py3-none-any.whl#sha256=670bb4582578134b3d81a84afa1b016128b429f3d48e6cbbaecc9d15675e984e
|
| 272 |
+
tensorboard==2.16.2
|
| 273 |
+
tensorboard-data-server==0.7.2
|
| 274 |
+
tensorrt @ file:///workspace/TensorRT-10.11.0.33/python/tensorrt-10.11.0.33-cp312-none-linux_x86_64.whl#sha256=c2f18f72758a3d54a71901495c8fda9e7094a8bcdd2a5b6b58967fbf05733b3a
|
| 275 |
+
terminado==0.18.1
|
| 276 |
+
thinc==8.2.5
|
| 277 |
+
threadpoolctl==3.6.0
|
| 278 |
+
thriftpy2 @ file:///rapids/thriftpy2-0.5.2-cp312-cp312-linux_x86_64.whl#sha256=c0c7d939bdc5ccf7739154959938445d5fdf72f1d6e12bd563947a36f67e2147
|
| 279 |
+
tinycss2==1.4.0
|
| 280 |
+
toolz @ file:///rapids/toolz-1.0.0-py3-none-any.whl#sha256=292c8f1c4e7516bf9086f8850935c799a874039c8bcf959d47b600e4c44a6236
|
| 281 |
+
torch @ file:///opt/transfer/torch-2.8.0a0%2B5228986c39.nv25.6-29946474-cp312-cp312-linux_x86_64.whl#sha256=7a3c9ff5288d8102a31296b3d62491f3f47c8820a5c9aaefc7dc0fe20e1e8ad5
|
| 282 |
+
torch_tensorrt @ file:///opt/pytorch/torch_tensorrt/dist/torch_tensorrt-2.8.0a0-cp312-cp312-linux_x86_64.whl#sha256=26f4d7a04e7e6e017a5b917b8b98b83720bb2bb0195f48d25f52d567d286bfe1
|
| 283 |
+
torchao @ file:///opt/pytorch/ao
|
| 284 |
+
torchprofile==0.0.4
|
| 285 |
+
torchvision @ file:///opt/transfer/torchvision-0.22.0a0%2B95f10a4e-29946474-cp312-cp312-manylinux_2_28_x86_64.whl#sha256=746a3c674cc61680a86f4a14b374b21af1b5c21f629c42765b40c9494f6b234d
|
| 286 |
+
tornado==6.5.1
|
| 287 |
+
tqdm==4.67.1
|
| 288 |
+
traitlets==5.14.3
|
| 289 |
+
transformer_engine @ file:///pvcdata/aiinfra/b-aghosh/wheels/transformer_engine-2.10.0.dev0%2B10f6128d-cp312-cp312-linux_x86_64.whl#sha256=35cf5f4b83b2baad864e12e512bb422e0929409ac05610812fa521d4942d0189
|
| 290 |
+
treelite @ file:///rapids/treelite-4.4.1-py3-none-linux_x86_64.whl#sha256=174741724205540a1a5b37da81483aeff55d89548f42b3461cf07acac7ae96eb
|
| 291 |
+
typer==0.16.0
|
| 292 |
+
types-dataclasses==0.6.6
|
| 293 |
+
types-python-dateutil==2.9.0.20250516
|
| 294 |
+
typing-inspection==0.4.1
|
| 295 |
+
typing_extensions==4.14.0
|
| 296 |
+
tzdata @ file:///rapids/tzdata-2025.2-py2.py3-none-any.whl#sha256=1a403fada01ff9221ca8044d701868fa132215d84beb92242d9acd2147f667a8
|
| 297 |
+
ucx-py @ file:///rapids/ucx_py-0.43.0-cp312-cp312-linux_x86_64.whl#sha256=4a452fd8d8c983283ada3d49086e3657dc2ee94fbad08efdd708d15ab6e0b73a
|
| 298 |
+
ucxx @ file:///rapids/ucxx-0.43.0-cp312-cp312-linux_x86_64.whl#sha256=f1b4e55cb45c66fe88a2d8f9c2b87751165f3cfec21fbdc9c47c4ddff75e8a5f
|
| 299 |
+
-e git+https://github.com/Dots-Infra/UltraEP.git@94cab099b44fffa99a82fea99e7c12d89cf65e4f#egg=ultra_ep
|
| 300 |
+
uri-template==1.3.0
|
| 301 |
+
urllib3==2.0.7
|
| 302 |
+
wasabi==1.1.3
|
| 303 |
+
wcwidth==0.2.13
|
| 304 |
+
weasel==0.4.1
|
| 305 |
+
webcolors==24.11.1
|
| 306 |
+
webencodings==0.5.1
|
| 307 |
+
websocket-client==1.8.0
|
| 308 |
+
Werkzeug==3.1.3
|
| 309 |
+
wheel==0.45.1
|
| 310 |
+
wrapt==1.17.2
|
| 311 |
+
xdoctest==1.0.2
|
| 312 |
+
xgboost @ file:///rapids/xgboost-2.1.4-py3-none-manylinux_2_39_x86_64.whl#sha256=25eeca27f4048fc491940b3a0b4b5782ee48c33cb3aec059cef71f8da542930b
|
| 313 |
+
xxhash==4.0.1
|
| 314 |
+
yarl @ file:///rapids/yarl-1.20.0-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl#sha256=bf099e2432131093cc611623e0b0bcc399b8cddd9a91eded8bfb50402ec35018
|
| 315 |
+
zarr @ file:///rapids/zarr-2.18.7-py3-none-any.whl#sha256=ac3dc4033e9ae4e9d7b5e27c97ea3eaf1003cc0a07f010bd83d5134bf8c4b223
|
| 316 |
+
zict @ file:///rapids/zict-3.0.0-py2.py3-none-any.whl#sha256=5796e36bd0e0cc8cf0fbc1ace6a68912611c1dbd74750a3f3026b9b9d6a327ae
|
| 317 |
+
zipp==3.22.0
|
tiera-rerun-fixed-moonep/echo/a2/provenance.txt
ADDED
|
@@ -0,0 +1,90 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
==============================================================
|
| 2 |
+
PROVENANCE: echo__a2__gbs16
|
| 3 |
+
==============================================================
|
| 4 |
+
captured_at : 2026-08-30T06:44:13+00:00
|
| 5 |
+
hostname : sp-aiinfra-interactive-aghosh-7jtjl-8xb200-moe-cg-0
|
| 6 |
+
kernel : 6.8.0-1043-nvidia
|
| 7 |
+
cwd : /workspace/blackwell_run_scripts
|
| 8 |
+
command : run_ladder.sh --phase a2 --gbs 16 --out /workspace/tiera-rerun-fixed-moonep/echo/a2
|
| 9 |
+
|
| 10 |
+
---- GPU / driver --------------------------------------------
|
| 11 |
+
0, NVIDIA B200, 10.0, 183359 MiB, 580.95.05
|
| 12 |
+
1, NVIDIA B200, 10.0, 183359 MiB, 580.95.05
|
| 13 |
+
2, NVIDIA B200, 10.0, 183359 MiB, 580.95.05
|
| 14 |
+
3, NVIDIA B200, 10.0, 183359 MiB, 580.95.05
|
| 15 |
+
4, NVIDIA B200, 10.0, 183359 MiB, 580.95.05
|
| 16 |
+
5, NVIDIA B200, 10.0, 183359 MiB, 580.95.05
|
| 17 |
+
6, NVIDIA B200, 10.0, 183359 MiB, 580.95.05
|
| 18 |
+
7, NVIDIA B200, 10.0, 183359 MiB, 580.95.05
|
| 19 |
+
persistence/clocks:
|
| 20 |
+
0, 165 MHz, 1965 MHz, 1000.00 W, 29
|
| 21 |
+
1, 120 MHz, 1965 MHz, 1000.00 W, 34
|
| 22 |
+
2, 120 MHz, 1965 MHz, 1000.00 W, 34
|
| 23 |
+
3, 120 MHz, 1965 MHz, 1000.00 W, 29
|
| 24 |
+
4, 120 MHz, 1965 MHz, 1000.00 W, 30
|
| 25 |
+
5, 120 MHz, 1965 MHz, 1000.00 W, 34
|
| 26 |
+
6, 120 MHz, 1965 MHz, 1000.00 W, 34
|
| 27 |
+
7, 120 MHz, 1965 MHz, 1000.00 W, 29
|
| 28 |
+
|
| 29 |
+
NVLink topology:
|
| 30 |
+
[4mGPU0 GPU1 GPU2 GPU3 GPU4 GPU5 GPU6 GPU7 NIC0 NIC1 NIC2 NIC3 NIC4 NIC5 NIC6 NIC7 NIC8 CPU Affinity NUMA Affinity GPU NUMA ID[0m
|
| 31 |
+
GPU0 X NV18 NV18 NV18 NV18 NV18 NV18 NV18 PIX NODE SYS SYS SYS SYS SYS SYS NODE 0-27 0 N/A
|
| 32 |
+
GPU1 NV18 X NV18 NV18 NV18 NV18 NV18 NV18 NODE PIX SYS SYS SYS SYS SYS SYS NODE 0-27 0 N/A
|
| 33 |
+
GPU2 NV18 NV18 X NV18 NV18 NV18 NV18 NV18 SYS SYS PIX NODE SYS SYS SYS SYS SYS 28-55 1 N/A
|
| 34 |
+
GPU3 NV18 NV18 NV18 X NV18 NV18 NV18 NV18 SYS SYS NODE PIX SYS SYS SYS SYS SYS 28-55 1 N/A
|
| 35 |
+
GPU4 NV18 NV18 NV18 NV18 X NV18 NV18 NV18 SYS SYS SYS SYS PIX NODE SYS SYS SYS 56-83 2 N/A
|
| 36 |
+
GPU5 NV18 NV18 NV18 NV18 NV18 X NV18 NV18 SYS SYS SYS SYS NODE PIX SYS SYS SYS 56-83 2 N/A
|
| 37 |
+
GPU6 NV18 NV18 NV18 NV18 NV18 NV18 X NV18 SYS SYS SYS SYS SYS SYS PIX NODE SYS 84-111 3 N/A
|
| 38 |
+
GPU7 NV18 NV18 NV18 NV18 NV18 NV18 NV18 X SYS SYS SYS SYS SYS SYS NODE PIX SYS 84-111 3 N/A
|
| 39 |
+
NIC0 PIX NODE SYS SYS SYS SYS SYS SYS X NODE SYS SYS SYS SYS SYS SYS NODE
|
| 40 |
+
NIC1 NODE PIX SYS SYS SYS SYS SYS SYS NODE X SYS SYS SYS SYS SYS SYS NODE
|
| 41 |
+
NIC2 SYS SYS PIX NODE SYS SYS SYS SYS SYS SYS X NODE SYS SYS SYS SYS SYS
|
| 42 |
+
NIC3 SYS SYS NODE PIX SYS SYS SYS SYS SYS SYS NODE X SYS SYS SYS SYS SYS
|
| 43 |
+
NIC4 SYS SYS SYS SYS PIX NODE SYS SYS SYS SYS SYS SYS X NODE SYS SYS SYS
|
| 44 |
+
unavailable
|
| 45 |
+
|
| 46 |
+
---- toolchain -----------------------------------------------
|
| 47 |
+
nvcc : release 12.9, V12.9.86
|
| 48 |
+
gcc : gcc (Ubuntu 13.3.0-6ubuntu2~24.04) 13.3.0
|
| 49 |
+
python : 3.12.3
|
| 50 |
+
torch : 2.8.0a0+5228986c39.nv25.06 cuda=12.9 nccl=2.27.3
|
| 51 |
+
torch arch : ['sm_75', 'sm_80', 'sm_86', 'sm_90', 'sm_100', 'sm_120', 'compute_120']
|
| 52 |
+
transformer_engine: 2.10.0.dev0+10f6128d [/usr/local/lib/python3.12/dist-packages/transformer_engine/__init__.py]
|
| 53 |
+
deep_ep : 1.2.1+34152ae [/workspace/DeepEP/deep_ep/__init__.py]
|
| 54 |
+
ultra_ep : 1.0.0 [/workspace/UltraEP/ultra_ep/__init__.py]
|
| 55 |
+
moonep : 0.0.1 [/workspace/MoonEP/moonep/__init__.py]
|
| 56 |
+
megatron : 0.14.0rc5 [None]
|
| 57 |
+
flash_attn : 2.7.4.post1 [/usr/local/lib/python3.12/dist-packages/flash_attn/__init__.py]
|
| 58 |
+
numpy : 1.26.4 [/usr/local/lib/python3.12/dist-packages/numpy/__init__.py]
|
| 59 |
+
huggingface_hub: 1.29.0 [/usr/local/lib/python3.12/dist-packages/huggingface_hub/__init__.py]
|
| 60 |
+
|
| 61 |
+
---- repositories --------------------------------------------
|
| 62 |
+
Megatron-MoE-Echo bracing-regions-through-kernels 9a23679390d7
|
| 63 |
+
Megatron-UltraEP ultraep-token-drop-analysis eaf829e7ad21
|
| 64 |
+
UltraEP main 94cab099b44f
|
| 65 |
+
DeepEP HEAD 34152ae28f80 (+3 untracked: build output)
|
| 66 |
+
MoonEP megatron-ddp-symmetric-params d8bc6667fad7
|
| 67 |
+
blackwell_run_scripts granularity-echo ff2c833ea5a5
|
| 68 |
+
|
| 69 |
+
---- environment ---------------------------------------------
|
| 70 |
+
CUDA_ARCH_LIST=7.5 8.0 8.6 9.0 10.0 12.0
|
| 71 |
+
CUDA_DRIVER_VERSION=575.57.08
|
| 72 |
+
CUDA_HOME=/usr/local/cuda
|
| 73 |
+
CUDA_MODULE_LOADING=LAZY
|
| 74 |
+
CUDA_VERSION=12.9.1.010
|
| 75 |
+
HF_REMOTE_ROOT=tiera-rerun-fixed-moonep
|
| 76 |
+
NCCL_VERSION=2.27.3
|
| 77 |
+
NVSHMEM_VERSION=3.2.5
|
| 78 |
+
PYTORCH_BUILD_NUMBER=0
|
| 79 |
+
PYTORCH_BUILD_VERSION=2.8.0a0+5228986
|
| 80 |
+
PYTORCH_HOME=/opt/pytorch/pytorch
|
| 81 |
+
PYTORCH_VERSION=2.8.0a0+5228986
|
| 82 |
+
TORCH_ALLOW_TF32_CUBLAS_OVERRIDE=1
|
| 83 |
+
TORCH_CUDA_ARCH_LIST=7.5 8.0 8.6 9.0 10.0 12.0+PTX
|
| 84 |
+
TORCH_NCCL_USE_COMM_NONBLOCKING=0
|
| 85 |
+
|
| 86 |
+
---- shared library resolution -------------------------------
|
| 87 |
+
libnvshmem_host.so: not in ldconfig
|
| 88 |
+
libnccl.so: /lib/x86_64-linux-gnu/libnccl.so.2
|
| 89 |
+
lrwxrwxrwx 1 root root 20 Aug 29 16:41 /usr/local/lib/python3.12/dist-packages/nvidia/nvshmem/lib/libnvshmem_host.so -> libnvshmem_host.so.3
|
| 90 |
+
-rw-r--r-- 1 root root 179045744 Aug 29 16:37 /usr/local/lib/python3.12/dist-packages/nvidia/nvshmem/lib/libnvshmem_host.so.3
|
tiera-rerun-fixed-moonep/hf_watcher.log
CHANGED
|
@@ -29,3 +29,6 @@ https://huggingface.co/MoE-CUDA-Graph-Analysis/tierb-b200-gbs8/tree/main
|
|
| 29 |
uploaded: /workspace/tiera-rerun-fixed-moonep -> MoE-CUDA-Graph-Analysis/tierb-b200-gbs8/tiera-rerun-fixed-moonep
|
| 30 |
https://huggingface.co/MoE-CUDA-Graph-Analysis/tierb-b200-gbs8/tree/main
|
| 31 |
[2026-08-30T06:40:36+00:00] backup pass
|
|
|
|
|
|
|
|
|
|
|
|
| 29 |
uploaded: /workspace/tiera-rerun-fixed-moonep -> MoE-CUDA-Graph-Analysis/tierb-b200-gbs8/tiera-rerun-fixed-moonep
|
| 30 |
https://huggingface.co/MoE-CUDA-Graph-Analysis/tierb-b200-gbs8/tree/main
|
| 31 |
[2026-08-30T06:40:36+00:00] backup pass
|
| 32 |
+
uploaded: /workspace/tiera-rerun-fixed-moonep -> MoE-CUDA-Graph-Analysis/tierb-b200-gbs8/tiera-rerun-fixed-moonep
|
| 33 |
+
https://huggingface.co/MoE-CUDA-Graph-Analysis/tierb-b200-gbs8/tree/main
|
| 34 |
+
[2026-08-30T06:50:38+00:00] backup pass
|
tiera-rerun-fixed-moonep/steps.tsv
CHANGED
|
@@ -2,3 +2,4 @@ status campaign phase gbs started ended result_dir
|
|
| 2 |
FAIL echo tierbprobe 16 2026-08-30T04:52:38+00:00 2026-08-30T05:06:38+00:00 /workspace/tiera-rerun-fixed-moonep/echo/tierbprobe
|
| 3 |
FAIL ultraep tierbprobe 16 2026-08-30T05:06:44+00:00 2026-08-30T05:20:40+00:00 /workspace/tiera-rerun-fixed-moonep/ultraep/tierbprobe
|
| 4 |
OK echo a1 16 2026-08-30T06:20:31+00:00 2026-08-30T06:36:28+00:00 /workspace/tiera-rerun-fixed-moonep/echo/a1
|
|
|
|
|
|
| 2 |
FAIL echo tierbprobe 16 2026-08-30T04:52:38+00:00 2026-08-30T05:06:38+00:00 /workspace/tiera-rerun-fixed-moonep/echo/tierbprobe
|
| 3 |
FAIL ultraep tierbprobe 16 2026-08-30T05:06:44+00:00 2026-08-30T05:20:40+00:00 /workspace/tiera-rerun-fixed-moonep/ultraep/tierbprobe
|
| 4 |
OK echo a1 16 2026-08-30T06:20:31+00:00 2026-08-30T06:36:28+00:00 /workspace/tiera-rerun-fixed-moonep/echo/a1
|
| 5 |
+
OK echo ponly 16 2026-08-30T06:36:31+00:00 2026-08-30T06:44:11+00:00 /workspace/tiera-rerun-fixed-moonep/echo/ponly
|
tiera-rerun-fixed-moonep/study.log
CHANGED
|
@@ -521,3 +521,51 @@ https://huggingface.co/MoE-CUDA-Graph-Analysis/tierb-b200-gbs8/tree/main
|
|
| 521 |
[guard] tearing down process group (outcome=DONE)
|
| 522 |
--------------------------------------------------------------------------
|
| 523 |
[06:40:26] CELL PONLY__echo__one__seq8192__gbs16__cap2p40625 (slots=1)
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 521 |
[guard] tearing down process group (outcome=DONE)
|
| 522 |
--------------------------------------------------------------------------
|
| 523 |
[06:40:26] CELL PONLY__echo__one__seq8192__gbs16__cap2p40625 (slots=1)
|
| 524 |
+
[guard] iter 12/12 (70s)
|
| 525 |
+
[guard] tearing down process group (outcome=DONE)
|
| 526 |
+
[guard] iter 12/12 (70s)
|
| 527 |
+
[guard] tearing down process group (outcome=DONE)
|
| 528 |
+
[guard] iter 12/12 (70s)
|
| 529 |
+
[guard] tearing down process group (outcome=DONE)
|
| 530 |
+
--------------------------------------------------------------------------
|
| 531 |
+
|
| 532 |
+
==============================================================
|
| 533 |
+
SWEEP COMPLETE ok=6 failed=0
|
| 534 |
+
==============================================================
|
| 535 |
+
config backend policy slots seq gbs cap recompute rep status iter_ms_mean iter_ms_min peak_mib run_dir
|
| 536 |
+
PONLY echo one 1 4096 16 2.40625 none 1 OK 175.19 174.10 40157 /workspace/tiera-rerun-fixed-moonep/echo/ponly/ponly_gbs16/PONLY__echo__one__seq4096__gbs16__cap2p40625/rep1
|
| 537 |
+
PONLY echo one 1 4096 16 2.40625 none 2 OK 174.92 173.90 40241 /workspace/tiera-rerun-fixed-moonep/echo/ponly/ponly_gbs16/PONLY__echo__one__seq4096__gbs16__cap2p40625/rep2
|
| 538 |
+
PONLY echo one 1 4096 16 2.40625 none 3 OK 174.81 173.80 40243 /workspace/tiera-rerun-fixed-moonep/echo/ponly/ponly_gbs16/PONLY__echo__one__seq4096__gbs16__cap2p40625/rep3
|
| 539 |
+
PONLY echo one 1 8192 16 2.40625 none 1 OK 316.36 315.50 53279 /workspace/tiera-rerun-fixed-moonep/echo/ponly/ponly_gbs16/PONLY__echo__one__seq8192__gbs16__cap2p40625/rep1
|
| 540 |
+
PONLY echo one 1 8192 16 2.40625 none 2 OK 316.47 315.50 53283 /workspace/tiera-rerun-fixed-moonep/echo/ponly/ponly_gbs16/PONLY__echo__one__seq8192__gbs16__cap2p40625/rep2
|
| 541 |
+
PONLY echo one 1 8192 16 2.40625 none 3 OK 316.70 315.40 53283 /workspace/tiera-rerun-fixed-moonep/echo/ponly/ponly_gbs16/PONLY__echo__one__seq8192__gbs16__cap2p40625/rep3
|
| 542 |
+
|
| 543 |
+
Results: /workspace/tiera-rerun-fixed-moonep/echo/ponly/ponly_gbs16/results.tsv
|
| 544 |
+
Roll up: python3 /workspace/blackwell_run_scripts/granularity/tools/collect.py /workspace/tiera-rerun-fixed-moonep/echo/ponly/ponly_gbs16
|
| 545 |
+
==============================================================
|
| 546 |
+
|
| 547 |
+
Campaign results: /workspace/tiera-rerun-fixed-moonep/echo/ponly
|
| 548 |
+
Roll up: python3 /workspace/blackwell_run_scripts/granularity/tools/collect.py /workspace/tiera-rerun-fixed-moonep/echo/ponly --compare
|
| 549 |
+
Already logged in to Hugging Face as: AbhishekGhosh1998
|
| 550 |
+
uploaded: /workspace/tiera-rerun-fixed-moonep/echo/ponly -> MoE-CUDA-Graph-Analysis/tierb-b200-gbs8/tiera-rerun-fixed-moonep/echo/ponly
|
| 551 |
+
https://huggingface.co/MoE-CUDA-Graph-Analysis/tierb-b200-gbs8/tree/main
|
| 552 |
+
|
| 553 |
+
=== STEP echo__a2__gbs16
|
| 554 |
+
==============================================================
|
| 555 |
+
CAMPAIGN: echo backend=echo phase=a2
|
| 556 |
+
Megatron: /workspace/Megatron-MoE-Echo @ bracing-regions-through-kernels
|
| 557 |
+
GBS=16 seq=4096 repeats=3 nsys=0
|
| 558 |
+
results -> /workspace/tiera-rerun-fixed-moonep/echo/a2
|
| 559 |
+
==============================================================
|
| 560 |
+
--------------------------------------------------------------------------
|
| 561 |
+
[06:44:28] CELL G1__echo__matched__seq4096__gbs16__cap2p40625 (slots=1)
|
| 562 |
+
[guard] iter 12/12 (70s)
|
| 563 |
+
[guard] tearing down process group (outcome=DONE)
|
| 564 |
+
[guard] iter 11/12 (70s)
|
| 565 |
+
[guard] iter 12/12 (80s)
|
| 566 |
+
[guard] iter 12/12 (70s)
|
| 567 |
+
[guard] tearing down process group (outcome=DONE)
|
| 568 |
+
--------------------------------------------------------------------------
|
| 569 |
+
[06:48:19] CELL G2__echo__matched__seq4096__gbs16__cap1p08203125 (slots=4)
|
| 570 |
+
[guard] iter 8/12 (70s)
|
| 571 |
+
[guard] iter 12/12 (80s)
|