granularity results tierb-fixed-moonep-full/moonep/tierb @ 2026-08-30T01:38:56Z from sp-aiinfra-interactive-aghosh-7jtjl-8xb200-moe-cg-0
Browse files- tierb-fixed-moonep-full/moonep/tierb/manifest.txt +2 -2
- tierb-fixed-moonep-full/moonep/tierb/pip_freeze.txt +1 -1
- tierb-fixed-moonep-full/moonep/tierb/provenance.txt +63 -0
- tierb-fixed-moonep-full/moonep/tierb/tierb_gbs8/DSV2_LITE_L24__moonep__matched__seq8192__gbs8__capnone__async/rep1/mem_samples.csv +3 -179
- tierb-fixed-moonep-full/moonep/tierb/tierb_gbs8/DSV2_LITE_L24__moonep__matched__seq8192__gbs8__capnone__async/rep1/run.env +1 -1
- tierb-fixed-moonep-full/moonep/tierb/tierb_gbs8/DSV2_LITE_L24__moonep__matched__seq8192__gbs8__capnone__async/rep1/train.log +0 -0
- tierb-fixed-moonep-full/moonep/tierb/tierb_gbs8/QWEN3_235B__moonep__matched__seq8192__gbs8__capnone__async/rep1/mem_samples.csv +137 -137
- tierb-fixed-moonep-full/moonep/tierb/tierb_gbs8/QWEN3_235B__moonep__matched__seq8192__gbs8__capnone__async/rep1/run.env +1 -1
- tierb-fixed-moonep-full/moonep/tierb/tierb_gbs8/QWEN3_235B__moonep__matched__seq8192__gbs8__capnone__async/rep1/train.log +198 -199
- tierb-fixed-moonep-full/moonep/tierb/tierb_gbs8/QWEN3_235B__moonep__matched__seq8192__gbs8__capnone__async/rep2/mem_samples.csv +119 -119
- tierb-fixed-moonep-full/moonep/tierb/tierb_gbs8/QWEN3_235B__moonep__matched__seq8192__gbs8__capnone__async/rep2/run.env +1 -1
- tierb-fixed-moonep-full/moonep/tierb/tierb_gbs8/QWEN3_235B__moonep__matched__seq8192__gbs8__capnone__async/rep2/train.log +213 -215
- tierb-fixed-moonep-full/moonep/tierb/tierb_gbs8/QWEN3_235B__moonep__matched__seq8192__gbs8__capnone__async/rep3/mem_samples.csv +128 -136
- tierb-fixed-moonep-full/moonep/tierb/tierb_gbs8/QWEN3_235B__moonep__matched__seq8192__gbs8__capnone__async/rep3/run.env +1 -1
- tierb-fixed-moonep-full/moonep/tierb/tierb_gbs8/QWEN3_235B__moonep__matched__seq8192__gbs8__capnone__async/rep3/train.log +219 -221
- tierb-fixed-moonep-full/moonep/tierb/tierb_gbs8/QWEN3_30B__moonep__matched__seq4096__gbs8__capnone__async/rep1/mem_samples.csv +129 -137
- tierb-fixed-moonep-full/moonep/tierb/tierb_gbs8/QWEN3_30B__moonep__matched__seq4096__gbs8__capnone__async/rep1/run.env +1 -1
- tierb-fixed-moonep-full/moonep/tierb/tierb_gbs8/QWEN3_30B__moonep__matched__seq4096__gbs8__capnone__async/rep1/train.log +0 -0
- tierb-fixed-moonep-full/moonep/tierb/tierb_gbs8/QWEN3_30B__moonep__matched__seq4096__gbs8__capnone__async/rep2/mem_samples.csv +133 -133
- tierb-fixed-moonep-full/moonep/tierb/tierb_gbs8/QWEN3_30B__moonep__matched__seq4096__gbs8__capnone__async/rep2/run.env +1 -1
- tierb-fixed-moonep-full/moonep/tierb/tierb_gbs8/QWEN3_30B__moonep__matched__seq4096__gbs8__capnone__async/rep2/train.log +0 -0
- tierb-fixed-moonep-full/moonep/tierb/tierb_gbs8/QWEN3_30B__moonep__matched__seq4096__gbs8__capnone__async/rep3/mem_samples.csv +118 -118
- tierb-fixed-moonep-full/moonep/tierb/tierb_gbs8/QWEN3_30B__moonep__matched__seq4096__gbs8__capnone__async/rep3/run.env +1 -1
- tierb-fixed-moonep-full/moonep/tierb/tierb_gbs8/QWEN3_30B__moonep__matched__seq4096__gbs8__capnone__async/rep3/train.log +0 -0
- tierb-fixed-moonep-full/moonep/tierb/tierb_gbs8/results.tsv +6 -12
tierb-fixed-moonep-full/moonep/tierb/manifest.txt
CHANGED
|
@@ -1,4 +1,4 @@
|
|
| 1 |
-
timestamp: 2026-08-
|
| 2 |
host: sp-aiinfra-interactive-aghosh-7jtjl-8xb200-moe-cg-0
|
| 3 |
gpu: NVIDIA B200 x8
|
| 4 |
driver: 580.95.05
|
|
@@ -16,4 +16,4 @@ nsys: 1
|
|
| 16 |
megatron_repo: /workspace/Megatron-MoE-Echo
|
| 17 |
megatron_branch_required: moonep_backend_async_grad_reduce
|
| 18 |
megatron_branch_actual: moonep_backend_async_grad_reduce
|
| 19 |
-
megatron_commit:
|
|
|
|
| 1 |
+
timestamp: 2026-08-30T01:31:08+00:00
|
| 2 |
host: sp-aiinfra-interactive-aghosh-7jtjl-8xb200-moe-cg-0
|
| 3 |
gpu: NVIDIA B200 x8
|
| 4 |
driver: 580.95.05
|
|
|
|
| 16 |
megatron_repo: /workspace/Megatron-MoE-Echo
|
| 17 |
megatron_branch_required: moonep_backend_async_grad_reduce
|
| 18 |
megatron_branch_actual: moonep_backend_async_grad_reduce
|
| 19 |
+
megatron_commit: 2e68c8ad
|
tierb-fixed-moonep-full/moonep/tierb/pip_freeze.txt
CHANGED
|
@@ -134,7 +134,7 @@ matplotlib==3.10.3
|
|
| 134 |
matplotlib-inline==0.1.7
|
| 135 |
mdit-py-plugins==0.4.2
|
| 136 |
mdurl==0.1.2
|
| 137 |
-
-e git+ssh://git@github.com/sriram-17-17/my-Megatron-MoE-Echo.git@
|
| 138 |
mistune==3.1.3
|
| 139 |
mkl==2021.1.1
|
| 140 |
mkl-devel==2021.1.1
|
|
|
|
| 134 |
matplotlib-inline==0.1.7
|
| 135 |
mdit-py-plugins==0.4.2
|
| 136 |
mdurl==0.1.2
|
| 137 |
+
-e git+ssh://git@github.com/sriram-17-17/my-Megatron-MoE-Echo.git@2e68c8ad6bfb5bac5d6eb962fa7d4fbf0a5fb095#egg=megatron_core
|
| 138 |
mistune==3.1.3
|
| 139 |
mkl==2021.1.1
|
| 140 |
mkl-devel==2021.1.1
|
tierb-fixed-moonep-full/moonep/tierb/provenance.txt
CHANGED
|
@@ -27,3 +27,66 @@ command : run_ladder.sh --phase tierb --gbs 8 --out /workspace/tierb-fixed
|
|
| 27 |
7, 120 MHz, 1965 MHz, 1000.00 W, 28
|
| 28 |
|
| 29 |
NVLink topology:
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 27 |
7, 120 MHz, 1965 MHz, 1000.00 W, 28
|
| 28 |
|
| 29 |
NVLink topology:
|
| 30 |
+
[4mGPU0 GPU1 GPU2 GPU3 GPU4 GPU5 GPU6 GPU7 NIC0 NIC1 NIC2 NIC3 NIC4 NIC5 NIC6 NIC7 NIC8 CPU Affinity NUMA Affinity GPU NUMA ID[0m
|
| 31 |
+
GPU0 X NV18 NV18 NV18 NV18 NV18 NV18 NV18 PIX NODE SYS SYS SYS SYS SYS SYS NODE 0-27 0 N/A
|
| 32 |
+
GPU1 NV18 X NV18 NV18 NV18 NV18 NV18 NV18 NODE PIX SYS SYS SYS SYS SYS SYS NODE 0-27 0 N/A
|
| 33 |
+
GPU2 NV18 NV18 X NV18 NV18 NV18 NV18 NV18 SYS SYS PIX NODE SYS SYS SYS SYS SYS 28-55 1 N/A
|
| 34 |
+
GPU3 NV18 NV18 NV18 X NV18 NV18 NV18 NV18 SYS SYS NODE PIX SYS SYS SYS SYS SYS 28-55 1 N/A
|
| 35 |
+
GPU4 NV18 NV18 NV18 NV18 X NV18 NV18 NV18 SYS SYS SYS SYS PIX NODE SYS SYS SYS 56-83 2 N/A
|
| 36 |
+
GPU5 NV18 NV18 NV18 NV18 NV18 X NV18 NV18 SYS SYS SYS SYS NODE PIX SYS SYS SYS 56-83 2 N/A
|
| 37 |
+
GPU6 NV18 NV18 NV18 NV18 NV18 NV18 X NV18 SYS SYS SYS SYS SYS SYS PIX NODE SYS 84-111 3 N/A
|
| 38 |
+
GPU7 NV18 NV18 NV18 NV18 NV18 NV18 NV18 X SYS SYS SYS SYS SYS SYS NODE PIX SYS 84-111 3 N/A
|
| 39 |
+
NIC0 PIX NODE SYS SYS SYS SYS SYS SYS X NODE SYS SYS SYS SYS SYS SYS NODE
|
| 40 |
+
NIC1 NODE PIX SYS SYS SYS SYS SYS SYS NODE X SYS SYS SYS SYS SYS SYS NODE
|
| 41 |
+
NIC2 SYS SYS PIX NODE SYS SYS SYS SYS SYS SYS X NODE SYS SYS SYS SYS SYS
|
| 42 |
+
NIC3 SYS SYS NODE PIX SYS SYS SYS SYS SYS SYS NODE X SYS SYS SYS SYS SYS
|
| 43 |
+
NIC4 SYS SYS SYS SYS PIX NODE SYS SYS SYS SYS SYS SYS X NODE SYS SYS SYS
|
| 44 |
+
unavailable
|
| 45 |
+
|
| 46 |
+
---- toolchain -----------------------------------------------
|
| 47 |
+
nvcc : release 12.9, V12.9.86
|
| 48 |
+
gcc : gcc (Ubuntu 13.3.0-6ubuntu2~24.04) 13.3.0
|
| 49 |
+
python : 3.12.3
|
| 50 |
+
torch : 2.8.0a0+5228986c39.nv25.06 cuda=12.9 nccl=2.27.3
|
| 51 |
+
torch arch : ['sm_75', 'sm_80', 'sm_86', 'sm_90', 'sm_100', 'sm_120', 'compute_120']
|
| 52 |
+
transformer_engine: 2.10.0.dev0+10f6128d [/usr/local/lib/python3.12/dist-packages/transformer_engine/__init__.py]
|
| 53 |
+
deep_ep : 1.2.1+34152ae [/workspace/DeepEP/deep_ep/__init__.py]
|
| 54 |
+
ultra_ep : 1.0.0 [/workspace/UltraEP/ultra_ep/__init__.py]
|
| 55 |
+
moonep : 0.0.1 [/workspace/MoonEP/moonep/__init__.py]
|
| 56 |
+
megatron : 0.14.0rc5 [None]
|
| 57 |
+
flash_attn : 2.7.4.post1 [/usr/local/lib/python3.12/dist-packages/flash_attn/__init__.py]
|
| 58 |
+
numpy : 1.26.4 [/usr/local/lib/python3.12/dist-packages/numpy/__init__.py]
|
| 59 |
+
huggingface_hub: 1.29.0 [/usr/local/lib/python3.12/dist-packages/huggingface_hub/__init__.py]
|
| 60 |
+
|
| 61 |
+
---- repositories --------------------------------------------
|
| 62 |
+
Megatron-MoE-Echo moonep_backend_async_grad_reduce 2e68c8ad6bfb
|
| 63 |
+
Megatron-UltraEP ultraep-bracing-regions-through-kernels 03d57ef7d476
|
| 64 |
+
UltraEP main 94cab099b44f
|
| 65 |
+
DeepEP HEAD 34152ae28f80 (+3 untracked: build output)
|
| 66 |
+
MoonEP megatron-ddp-symmetric-params d8bc6667fad7
|
| 67 |
+
blackwell_run_scripts granularity-moonep eec9aee807fa
|
| 68 |
+
|
| 69 |
+
---- environment ---------------------------------------------
|
| 70 |
+
CUDA_ARCH_LIST=7.5 8.0 8.6 9.0 10.0 12.0
|
| 71 |
+
CUDA_DRIVER_VERSION=575.57.08
|
| 72 |
+
CUDA_HOME=/usr/local/cuda
|
| 73 |
+
CUDA_MODULE_LOADING=LAZY
|
| 74 |
+
CUDA_VERSION=12.9.1.010
|
| 75 |
+
HF_REMOTE_ROOT=tierb-fixed-moonep-full
|
| 76 |
+
MOONEP_ASSERT_NO_PENDING_REDUCE=1
|
| 77 |
+
MOONEP_ASYNC_GRAD_REDUCE=1
|
| 78 |
+
NCCL_VERSION=2.27.3
|
| 79 |
+
NVSHMEM_VERSION=3.2.5
|
| 80 |
+
PYTORCH_BUILD_NUMBER=0
|
| 81 |
+
PYTORCH_BUILD_VERSION=2.8.0a0+5228986
|
| 82 |
+
PYTORCH_HOME=/opt/pytorch/pytorch
|
| 83 |
+
PYTORCH_VERSION=2.8.0a0+5228986
|
| 84 |
+
TORCH_ALLOW_TF32_CUBLAS_OVERRIDE=1
|
| 85 |
+
TORCH_CUDA_ARCH_LIST=7.5 8.0 8.6 9.0 10.0 12.0+PTX
|
| 86 |
+
TORCH_NCCL_USE_COMM_NONBLOCKING=0
|
| 87 |
+
|
| 88 |
+
---- shared library resolution -------------------------------
|
| 89 |
+
libnvshmem_host.so: not in ldconfig
|
| 90 |
+
libnccl.so: /lib/x86_64-linux-gnu/libnccl.so.2
|
| 91 |
+
lrwxrwxrwx 1 root root 20 Aug 29 16:41 /usr/local/lib/python3.12/dist-packages/nvidia/nvshmem/lib/libnvshmem_host.so -> libnvshmem_host.so.3
|
| 92 |
+
-rw-r--r-- 1 root root 179045744 Aug 29 16:37 /usr/local/lib/python3.12/dist-packages/nvidia/nvshmem/lib/libnvshmem_host.so.3
|
tierb-fixed-moonep-full/moonep/tierb/tierb_gbs8/DSV2_LITE_L24__moonep__matched__seq8192__gbs8__capnone__async/rep1/mem_samples.csv
CHANGED
|
@@ -7,7 +7,7 @@
|
|
| 7 |
6, 0
|
| 8 |
7, 0
|
| 9 |
0, 4
|
| 10 |
-
1,
|
| 11 |
2, 1
|
| 12 |
3, 1
|
| 13 |
4, 1
|
|
@@ -20,13 +20,13 @@
|
|
| 20 |
3, 4
|
| 21 |
4, 4
|
| 22 |
5, 4
|
| 23 |
-
6,
|
| 24 |
7, 1
|
| 25 |
0, 1
|
| 26 |
1, 1
|
| 27 |
2, 1
|
| 28 |
3, 1
|
| 29 |
-
4,
|
| 30 |
5, 4
|
| 31 |
6, 4
|
| 32 |
7, 4
|
|
@@ -54,179 +54,3 @@
|
|
| 54 |
5, 1
|
| 55 |
6, 1
|
| 56 |
7, 1
|
| 57 |
-
0, 4
|
| 58 |
-
1, 4
|
| 59 |
-
2, 4
|
| 60 |
-
3, 4
|
| 61 |
-
4, 4
|
| 62 |
-
5, 1
|
| 63 |
-
6, 1
|
| 64 |
-
7, 1
|
| 65 |
-
0, 4
|
| 66 |
-
1, 4
|
| 67 |
-
2, 4
|
| 68 |
-
3, 4
|
| 69 |
-
4, 4
|
| 70 |
-
5, 4
|
| 71 |
-
6, 4
|
| 72 |
-
7, 4
|
| 73 |
-
0, 4
|
| 74 |
-
1, 4
|
| 75 |
-
2, 4
|
| 76 |
-
3, 4
|
| 77 |
-
4, 4
|
| 78 |
-
5, 4
|
| 79 |
-
6, 35
|
| 80 |
-
7, 4
|
| 81 |
-
0, 821
|
| 82 |
-
1, 821
|
| 83 |
-
2, 819
|
| 84 |
-
3, 821
|
| 85 |
-
4, 819
|
| 86 |
-
5, 819
|
| 87 |
-
6, 821
|
| 88 |
-
7, 821
|
| 89 |
-
0, 1537
|
| 90 |
-
1, 1537
|
| 91 |
-
2, 1537
|
| 92 |
-
3, 1537
|
| 93 |
-
4, 1537
|
| 94 |
-
5, 1537
|
| 95 |
-
6, 1537
|
| 96 |
-
7, 1537
|
| 97 |
-
0, 2471
|
| 98 |
-
1, 2471
|
| 99 |
-
2, 2471
|
| 100 |
-
3, 2471
|
| 101 |
-
4, 2471
|
| 102 |
-
5, 2471
|
| 103 |
-
6, 2471
|
| 104 |
-
7, 2471
|
| 105 |
-
0, 26195
|
| 106 |
-
1, 26193
|
| 107 |
-
2, 26199
|
| 108 |
-
3, 26187
|
| 109 |
-
4, 26235
|
| 110 |
-
5, 26211
|
| 111 |
-
6, 26207
|
| 112 |
-
7, 26203
|
| 113 |
-
0, 30319
|
| 114 |
-
1, 30407
|
| 115 |
-
2, 30253
|
| 116 |
-
3, 30253
|
| 117 |
-
4, 30253
|
| 118 |
-
5, 30253
|
| 119 |
-
6, 30385
|
| 120 |
-
7, 30407
|
| 121 |
-
0, 36177
|
| 122 |
-
1, 36263
|
| 123 |
-
2, 36113
|
| 124 |
-
3, 36111
|
| 125 |
-
4, 36111
|
| 126 |
-
5, 36109
|
| 127 |
-
6, 36243
|
| 128 |
-
7, 36263
|
| 129 |
-
0, 40669
|
| 130 |
-
1, 40739
|
| 131 |
-
2, 40611
|
| 132 |
-
3, 40613
|
| 133 |
-
4, 40591
|
| 134 |
-
5, 40641
|
| 135 |
-
6, 40727
|
| 136 |
-
7, 40805
|
| 137 |
-
0, 41133
|
| 138 |
-
1, 41217
|
| 139 |
-
2, 41067
|
| 140 |
-
3, 41065
|
| 141 |
-
4, 41065
|
| 142 |
-
5, 41063
|
| 143 |
-
6, 41199
|
| 144 |
-
7, 41217
|
| 145 |
-
0, 41137
|
| 146 |
-
1, 41221
|
| 147 |
-
2, 41071
|
| 148 |
-
3, 41071
|
| 149 |
-
4, 41071
|
| 150 |
-
5, 41067
|
| 151 |
-
6, 41203
|
| 152 |
-
7, 41221
|
| 153 |
-
0, 108757
|
| 154 |
-
1, 108839
|
| 155 |
-
2, 108697
|
| 156 |
-
3, 108689
|
| 157 |
-
4, 108699
|
| 158 |
-
5, 108695
|
| 159 |
-
6, 108829
|
| 160 |
-
7, 108843
|
| 161 |
-
0, 109391
|
| 162 |
-
1, 109475
|
| 163 |
-
2, 109325
|
| 164 |
-
3, 109325
|
| 165 |
-
4, 109325
|
| 166 |
-
5, 109321
|
| 167 |
-
6, 109457
|
| 168 |
-
7, 109475
|
| 169 |
-
0, 109665
|
| 170 |
-
1, 109763
|
| 171 |
-
2, 109639
|
| 172 |
-
3, 109613
|
| 173 |
-
4, 109633
|
| 174 |
-
5, 109651
|
| 175 |
-
6, 109833
|
| 176 |
-
7, 109763
|
| 177 |
-
0, 110429
|
| 178 |
-
1, 110513
|
| 179 |
-
2, 110363
|
| 180 |
-
3, 110363
|
| 181 |
-
4, 110363
|
| 182 |
-
5, 110359
|
| 183 |
-
6, 110495
|
| 184 |
-
7, 110513
|
| 185 |
-
0, 111079
|
| 186 |
-
1, 111169
|
| 187 |
-
2, 111003
|
| 188 |
-
3, 110985
|
| 189 |
-
4, 111047
|
| 190 |
-
5, 111029
|
| 191 |
-
6, 111149
|
| 192 |
-
7, 111201
|
| 193 |
-
0, 119291
|
| 194 |
-
1, 119363
|
| 195 |
-
2, 127427
|
| 196 |
-
3, 121263
|
| 197 |
-
4, 127429
|
| 198 |
-
5, 117155
|
| 199 |
-
6, 119357
|
| 200 |
-
7, 119365
|
| 201 |
-
0, 133237
|
| 202 |
-
1, 133311
|
| 203 |
-
2, 133157
|
| 204 |
-
3, 133155
|
| 205 |
-
4, 133155
|
| 206 |
-
5, 133157
|
| 207 |
-
6, 133303
|
| 208 |
-
7, 133311
|
| 209 |
-
0, 133237
|
| 210 |
-
1, 133311
|
| 211 |
-
2, 133157
|
| 212 |
-
3, 133155
|
| 213 |
-
4, 133155
|
| 214 |
-
5, 133157
|
| 215 |
-
6, 133303
|
| 216 |
-
7, 133311
|
| 217 |
-
0, 133237
|
| 218 |
-
1, 133311
|
| 219 |
-
2, 133157
|
| 220 |
-
3, 133155
|
| 221 |
-
4, 133155
|
| 222 |
-
5, 133157
|
| 223 |
-
6, 133303
|
| 224 |
-
7, 133311
|
| 225 |
-
0, 9575
|
| 226 |
-
1, 9578
|
| 227 |
-
2, 9578
|
| 228 |
-
3, 9578
|
| 229 |
-
4, 9898
|
| 230 |
-
5, 133155
|
| 231 |
-
6, 11178
|
| 232 |
-
7, 9578
|
|
|
|
| 7 |
6, 0
|
| 8 |
7, 0
|
| 9 |
0, 4
|
| 10 |
+
1, 4
|
| 11 |
2, 1
|
| 12 |
3, 1
|
| 13 |
4, 1
|
|
|
|
| 20 |
3, 4
|
| 21 |
4, 4
|
| 22 |
5, 4
|
| 23 |
+
6, 4
|
| 24 |
7, 1
|
| 25 |
0, 1
|
| 26 |
1, 1
|
| 27 |
2, 1
|
| 28 |
3, 1
|
| 29 |
+
4, 4
|
| 30 |
5, 4
|
| 31 |
6, 4
|
| 32 |
7, 4
|
|
|
|
| 54 |
5, 1
|
| 55 |
6, 1
|
| 56 |
7, 1
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
tierb-fixed-moonep-full/moonep/tierb/tierb_gbs8/DSV2_LITE_L24__moonep__matched__seq8192__gbs8__capnone__async/rep1/run.env
CHANGED
|
@@ -3,7 +3,7 @@ config=DSV2_LITE_L24
|
|
| 3 |
family=tierB
|
| 4 |
repo=/workspace/Megatron-MoE-Echo
|
| 5 |
branch=moonep_backend_async_grad_reduce
|
| 6 |
-
commit=
|
| 7 |
moonep_source_dir=/workspace/MoonEP
|
| 8 |
moonep_source_branch=megatron-ddp-symmetric-params
|
| 9 |
moonep_source_commit=d8bc666
|
|
|
|
| 3 |
family=tierB
|
| 4 |
repo=/workspace/Megatron-MoE-Echo
|
| 5 |
branch=moonep_backend_async_grad_reduce
|
| 6 |
+
commit=2e68c8ad
|
| 7 |
moonep_source_dir=/workspace/MoonEP
|
| 8 |
moonep_source_branch=megatron-ddp-symmetric-params
|
| 9 |
moonep_source_commit=d8bc666
|
tierb-fixed-moonep-full/moonep/tierb/tierb_gbs8/DSV2_LITE_L24__moonep__matched__seq8192__gbs8__capnone__async/rep1/train.log
CHANGED
|
The diff for this file is too large to render.
See raw diff
|
|
|
tierb-fixed-moonep-full/moonep/tierb/tierb_gbs8/QWEN3_235B__moonep__matched__seq8192__gbs8__capnone__async/rep1/mem_samples.csv
CHANGED
|
@@ -26,7 +26,7 @@
|
|
| 26 |
1, 1
|
| 27 |
2, 1
|
| 28 |
3, 1
|
| 29 |
-
4,
|
| 30 |
5, 4
|
| 31 |
6, 4
|
| 32 |
7, 4
|
|
@@ -62,86 +62,86 @@
|
|
| 62 |
5, 4
|
| 63 |
6, 4
|
| 64 |
7, 4
|
| 65 |
-
0,
|
| 66 |
-
1,
|
| 67 |
-
2,
|
| 68 |
-
3,
|
| 69 |
-
4,
|
| 70 |
-
5,
|
| 71 |
-
6,
|
| 72 |
-
7,
|
| 73 |
-
0,
|
| 74 |
-
1,
|
| 75 |
-
2,
|
| 76 |
-
3,
|
| 77 |
-
4,
|
| 78 |
-
5,
|
| 79 |
-
6,
|
| 80 |
-
7,
|
| 81 |
-
0,
|
| 82 |
-
1,
|
| 83 |
-
2,
|
| 84 |
-
3,
|
| 85 |
-
4,
|
| 86 |
-
5,
|
| 87 |
-
6,
|
| 88 |
-
7,
|
| 89 |
-
0,
|
| 90 |
-
1,
|
| 91 |
-
2,
|
| 92 |
-
3,
|
| 93 |
-
4,
|
| 94 |
-
5,
|
| 95 |
-
6,
|
| 96 |
-
7,
|
| 97 |
-
0,
|
| 98 |
-
1,
|
| 99 |
-
2,
|
| 100 |
-
3,
|
| 101 |
-
4,
|
| 102 |
-
5,
|
| 103 |
-
6,
|
| 104 |
-
7,
|
| 105 |
-
0,
|
| 106 |
-
1,
|
| 107 |
-
2,
|
| 108 |
-
3,
|
| 109 |
-
4,
|
| 110 |
-
5,
|
| 111 |
-
6,
|
| 112 |
-
7,
|
| 113 |
-
0,
|
| 114 |
-
1,
|
| 115 |
-
2,
|
| 116 |
-
3,
|
| 117 |
-
4,
|
| 118 |
-
5,
|
| 119 |
-
6,
|
| 120 |
-
7,
|
| 121 |
-
0,
|
| 122 |
1, 59731
|
| 123 |
-
2,
|
| 124 |
-
3,
|
| 125 |
4, 59443
|
| 126 |
5, 59731
|
| 127 |
6, 59733
|
| 128 |
-
7,
|
| 129 |
0, 59733
|
| 130 |
-
1,
|
| 131 |
2, 59733
|
| 132 |
3, 59733
|
| 133 |
4, 59445
|
| 134 |
5, 59733
|
| 135 |
6, 59733
|
| 136 |
7, 59733
|
| 137 |
-
0,
|
| 138 |
-
1,
|
| 139 |
-
2,
|
| 140 |
-
3,
|
| 141 |
-
4,
|
| 142 |
-
5,
|
| 143 |
-
6,
|
| 144 |
-
7,
|
| 145 |
0, 87633
|
| 146 |
1, 87673
|
| 147 |
2, 87673
|
|
@@ -150,75 +150,75 @@
|
|
| 150 |
5, 87673
|
| 151 |
6, 87673
|
| 152 |
7, 87673
|
| 153 |
-
0,
|
| 154 |
-
1,
|
| 155 |
-
2,
|
| 156 |
-
3,
|
| 157 |
-
4,
|
| 158 |
-
5,
|
| 159 |
-
6,
|
| 160 |
-
7,
|
| 161 |
-
0,
|
| 162 |
-
1,
|
| 163 |
-
2,
|
| 164 |
-
3,
|
| 165 |
-
4,
|
| 166 |
-
5,
|
| 167 |
-
6,
|
| 168 |
-
7,
|
| 169 |
-
0,
|
| 170 |
-
1,
|
| 171 |
-
2,
|
| 172 |
-
3,
|
| 173 |
-
4,
|
| 174 |
-
5,
|
| 175 |
-
6,
|
| 176 |
-
7,
|
| 177 |
-
0,
|
| 178 |
-
1,
|
| 179 |
-
2,
|
| 180 |
-
3,
|
| 181 |
-
4,
|
| 182 |
-
5,
|
| 183 |
-
6,
|
| 184 |
-
7,
|
| 185 |
-
0,
|
| 186 |
-
1,
|
| 187 |
-
2,
|
| 188 |
-
3,
|
| 189 |
-
4,
|
| 190 |
-
5,
|
| 191 |
-
6,
|
| 192 |
-
7,
|
| 193 |
-
0,
|
| 194 |
-
1,
|
| 195 |
-
2,
|
| 196 |
-
3,
|
| 197 |
-
4,
|
| 198 |
-
5,
|
| 199 |
-
6,
|
| 200 |
-
7,
|
| 201 |
-
0,
|
| 202 |
-
1,
|
| 203 |
-
2,
|
| 204 |
-
3,
|
| 205 |
-
4,
|
| 206 |
-
5,
|
| 207 |
-
6,
|
| 208 |
-
7,
|
| 209 |
-
0,
|
| 210 |
-
1,
|
| 211 |
-
2,
|
| 212 |
-
3,
|
| 213 |
-
4,
|
| 214 |
-
5,
|
| 215 |
-
6,
|
| 216 |
-
7,
|
| 217 |
0, 1
|
| 218 |
-
1,
|
| 219 |
2, 4
|
| 220 |
-
3,
|
| 221 |
-
4,
|
| 222 |
5, 4
|
| 223 |
6, 4
|
| 224 |
7, 4
|
|
|
|
| 26 |
1, 1
|
| 27 |
2, 1
|
| 28 |
3, 1
|
| 29 |
+
4, 1
|
| 30 |
5, 4
|
| 31 |
6, 4
|
| 32 |
7, 4
|
|
|
|
| 62 |
5, 4
|
| 63 |
6, 4
|
| 64 |
7, 4
|
| 65 |
+
0, 777
|
| 66 |
+
1, 751
|
| 67 |
+
2, 725
|
| 68 |
+
3, 767
|
| 69 |
+
4, 725
|
| 70 |
+
5, 731
|
| 71 |
+
6, 753
|
| 72 |
+
7, 725
|
| 73 |
+
0, 945
|
| 74 |
+
1, 943
|
| 75 |
+
2, 945
|
| 76 |
+
3, 939
|
| 77 |
+
4, 949
|
| 78 |
+
5, 963
|
| 79 |
+
6, 961
|
| 80 |
+
7, 957
|
| 81 |
+
0, 1975
|
| 82 |
+
1, 1975
|
| 83 |
+
2, 1975
|
| 84 |
+
3, 1975
|
| 85 |
+
4, 1975
|
| 86 |
+
5, 1975
|
| 87 |
+
6, 1975
|
| 88 |
+
7, 1975
|
| 89 |
+
0, 32245
|
| 90 |
+
1, 31861
|
| 91 |
+
2, 35701
|
| 92 |
+
3, 43765
|
| 93 |
+
4, 35701
|
| 94 |
+
5, 41461
|
| 95 |
+
6, 30517
|
| 96 |
+
7, 30517
|
| 97 |
+
0, 48447
|
| 98 |
+
1, 48449
|
| 99 |
+
2, 48449
|
| 100 |
+
3, 48449
|
| 101 |
+
4, 48449
|
| 102 |
+
5, 48449
|
| 103 |
+
6, 48449
|
| 104 |
+
7, 48449
|
| 105 |
+
0, 56867
|
| 106 |
+
1, 56821
|
| 107 |
+
2, 56921
|
| 108 |
+
3, 56933
|
| 109 |
+
4, 56617
|
| 110 |
+
5, 56883
|
| 111 |
+
6, 56835
|
| 112 |
+
7, 56885
|
| 113 |
+
0, 58733
|
| 114 |
+
1, 58755
|
| 115 |
+
2, 58741
|
| 116 |
+
3, 58729
|
| 117 |
+
4, 58447
|
| 118 |
+
5, 58749
|
| 119 |
+
6, 58733
|
| 120 |
+
7, 58759
|
| 121 |
+
0, 59731
|
| 122 |
1, 59731
|
| 123 |
+
2, 59733
|
| 124 |
+
3, 59731
|
| 125 |
4, 59443
|
| 126 |
5, 59731
|
| 127 |
6, 59733
|
| 128 |
+
7, 59733
|
| 129 |
0, 59733
|
| 130 |
+
1, 59733
|
| 131 |
2, 59733
|
| 132 |
3, 59733
|
| 133 |
4, 59445
|
| 134 |
5, 59733
|
| 135 |
6, 59733
|
| 136 |
7, 59733
|
| 137 |
+
0, 85347
|
| 138 |
+
1, 85387
|
| 139 |
+
2, 85387
|
| 140 |
+
3, 85387
|
| 141 |
+
4, 85347
|
| 142 |
+
5, 85387
|
| 143 |
+
6, 85387
|
| 144 |
+
7, 85387
|
| 145 |
0, 87633
|
| 146 |
1, 87673
|
| 147 |
2, 87673
|
|
|
|
| 150 |
5, 87673
|
| 151 |
6, 87673
|
| 152 |
7, 87673
|
| 153 |
+
0, 87633
|
| 154 |
+
1, 87673
|
| 155 |
+
2, 87673
|
| 156 |
+
3, 87673
|
| 157 |
+
4, 87633
|
| 158 |
+
5, 87673
|
| 159 |
+
6, 87673
|
| 160 |
+
7, 87673
|
| 161 |
+
0, 88183
|
| 162 |
+
1, 88229
|
| 163 |
+
2, 88215
|
| 164 |
+
3, 88229
|
| 165 |
+
4, 88177
|
| 166 |
+
5, 88267
|
| 167 |
+
6, 88267
|
| 168 |
+
7, 88259
|
| 169 |
+
0, 97425
|
| 170 |
+
1, 97499
|
| 171 |
+
2, 97511
|
| 172 |
+
3, 97499
|
| 173 |
+
4, 97395
|
| 174 |
+
5, 97541
|
| 175 |
+
6, 97511
|
| 176 |
+
7, 97625
|
| 177 |
+
0, 120273
|
| 178 |
+
1, 120309
|
| 179 |
+
2, 120309
|
| 180 |
+
3, 120309
|
| 181 |
+
4, 120309
|
| 182 |
+
5, 120309
|
| 183 |
+
6, 120309
|
| 184 |
+
7, 120309
|
| 185 |
+
0, 120273
|
| 186 |
+
1, 120309
|
| 187 |
+
2, 120309
|
| 188 |
+
3, 120309
|
| 189 |
+
4, 120309
|
| 190 |
+
5, 120309
|
| 191 |
+
6, 120309
|
| 192 |
+
7, 120309
|
| 193 |
+
0, 119365
|
| 194 |
+
1, 119663
|
| 195 |
+
2, 119663
|
| 196 |
+
3, 119663
|
| 197 |
+
4, 119663
|
| 198 |
+
5, 119401
|
| 199 |
+
6, 119663
|
| 200 |
+
7, 119663
|
| 201 |
+
0, 117333
|
| 202 |
+
1, 117437
|
| 203 |
+
2, 117049
|
| 204 |
+
3, 117501
|
| 205 |
+
4, 117843
|
| 206 |
+
5, 117369
|
| 207 |
+
6, 117501
|
| 208 |
+
7, 118083
|
| 209 |
+
0, 4
|
| 210 |
+
1, 115541
|
| 211 |
+
2, 92905
|
| 212 |
+
3, 109993
|
| 213 |
+
4, 115801
|
| 214 |
+
5, 92905
|
| 215 |
+
6, 103465
|
| 216 |
+
7, 115757
|
| 217 |
0, 1
|
| 218 |
+
1, 3
|
| 219 |
2, 4
|
| 220 |
+
3, 4
|
| 221 |
+
4, 114027
|
| 222 |
5, 4
|
| 223 |
6, 4
|
| 224 |
7, 4
|
tierb-fixed-moonep-full/moonep/tierb/tierb_gbs8/QWEN3_235B__moonep__matched__seq8192__gbs8__capnone__async/rep1/run.env
CHANGED
|
@@ -3,7 +3,7 @@ config=QWEN3_235B
|
|
| 3 |
family=tierB
|
| 4 |
repo=/workspace/Megatron-MoE-Echo
|
| 5 |
branch=moonep_backend_async_grad_reduce
|
| 6 |
-
commit=
|
| 7 |
moonep_source_dir=/workspace/MoonEP
|
| 8 |
moonep_source_branch=megatron-ddp-symmetric-params
|
| 9 |
moonep_source_commit=d8bc666
|
|
|
|
| 3 |
family=tierB
|
| 4 |
repo=/workspace/Megatron-MoE-Echo
|
| 5 |
branch=moonep_backend_async_grad_reduce
|
| 6 |
+
commit=2e68c8ad
|
| 7 |
moonep_source_dir=/workspace/MoonEP
|
| 8 |
moonep_source_branch=megatron-ddp-symmetric-params
|
| 9 |
moonep_source_commit=d8bc666
|
tierb-fixed-moonep-full/moonep/tierb/tierb_gbs8/QWEN3_235B__moonep__matched__seq8192__gbs8__capnone__async/rep1/train.log
CHANGED
|
@@ -1,23 +1,16 @@
|
|
| 1 |
-
[
|
| 2 |
==============================================================
|
| 3 |
QWEN3_235B / moonep (slots/rank=16 policy=matched)
|
| 4 |
E=128 K=8 H'=1536 m=16
|
| 5 |
L=12 seq=8192 gbs=8 (1 microbatches)
|
| 6 |
cap_mode=none cap=none cg=1 recompute=none
|
| 7 |
memory_dump=none snapshots=/workspace/tierb-fixed-moonep-full/moonep/tierb/tierb_gbs8/QWEN3_235B__moonep__matched__seq8192__gbs8__capnone__async/rep1/memory_snapshots/snapshot.pickle
|
| 8 |
-
repo=/workspace/Megatron-MoE-Echo @
|
| 9 |
==============================================================
|
| 10 |
-
|
| 11 |
-
|
| 12 |
-
|
| 13 |
-
|
| 14 |
-
/workspace/Megatron-MoE-Echo/megatron/training/arguments.py:831: UserWarning: Disabling sequence parallelism because tensor model parallelism is disabled
|
| 15 |
-
warnings.warn("Disabling sequence parallelism because tensor model parallelism is disabled")
|
| 16 |
-
WARNING: one_logger package is required to enable e2e metrics tracking. please go to https://confluence.nvidia.com/display/MLWFO/Package+Repositories for details to install it
|
| 17 |
-
/workspace/Megatron-MoE-Echo/megatron/training/arguments.py:831: UserWarning: Disabling sequence parallelism because tensor model parallelism is disabled
|
| 18 |
-
warnings.warn("Disabling sequence parallelism because tensor model parallelism is disabled")
|
| 19 |
-
/workspace/Megatron-MoE-Echo/megatron/training/arguments.py:831: UserWarning: Disabling sequence parallelism because tensor model parallelism is disabled
|
| 20 |
-
warnings.warn("Disabling sequence parallelism because tensor model parallelism is disabled")
|
| 21 |
/workspace/Megatron-MoE-Echo/megatron/training/arguments.py:831: UserWarning: Disabling sequence parallelism because tensor model parallelism is disabled
|
| 22 |
warnings.warn("Disabling sequence parallelism because tensor model parallelism is disabled")
|
| 23 |
using world size: 8, data-parallel size: 8, context-parallel size: 1, hierarchical context-parallel sizes: None, tensor-model-parallel size: 1, pipeline-model-parallel size: 1
|
|
@@ -575,53 +568,60 @@ WARNING:megatron.core.rerun_state_machine:RerunStateMachine initialized in mode
|
|
| 575 |
warnings.warn("Disabling sequence parallelism because tensor model parallelism is disabled")
|
| 576 |
/workspace/Megatron-MoE-Echo/megatron/training/arguments.py:831: UserWarning: Disabling sequence parallelism because tensor model parallelism is disabled
|
| 577 |
warnings.warn("Disabling sequence parallelism because tensor model parallelism is disabled")
|
|
|
|
| 578 |
/workspace/Megatron-MoE-Echo/megatron/training/arguments.py:831: UserWarning: Disabling sequence parallelism because tensor model parallelism is disabled
|
| 579 |
warnings.warn("Disabling sequence parallelism because tensor model parallelism is disabled")
|
| 580 |
-
|
| 581 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 582 |
[Gloo] Rank 4 is connected to 7 peer ranks. Expected number of connected peer ranks is : 7
|
| 583 |
[Gloo] Rank 1 is connected to 7 peer ranks. Expected number of connected peer ranks is : 7
|
| 584 |
-
[Gloo] Rank
|
| 585 |
-
|
| 586 |
-
7
|
| 587 |
[Gloo] Rank 6 is connected to 7 peer ranks. Expected number of connected peer ranks is : 7
|
| 588 |
-
[Gloo] Rank
|
| 589 |
-
peer ranks. Expected number of connected peer ranks is : 7
|
| 590 |
-
[Gloo] Rank 5 is connected to 7 peer ranks. Expected number of connected peer ranks is : 7
|
| 591 |
[Gloo] Rank 4 is connected to 7 peer ranks. Expected number of connected peer ranks is : 7
|
| 592 |
[Gloo] Rank 0 is connected to 7 peer ranks. Expected number of connected peer ranks is : 7
|
|
|
|
| 593 |
[Gloo] Rank 1 is connected to 7 peer ranks. Expected number of connected peer ranks is : 7
|
| 594 |
-
[Gloo] Rank
|
| 595 |
-
[Gloo] Rank
|
|
|
|
|
|
|
| 596 |
[Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
|
| 597 |
> initialized tensor model parallel with size 1
|
| 598 |
> initialized pipeline model parallel with size 1
|
| 599 |
> setting random seeds to 1234 ...
|
| 600 |
-
[Gloo] Rank [Gloo] Rank 00 is connected to is connected to
|
| 601 |
|
| 602 |
-
[Gloo] Rank [Gloo] Rank 00 is connected to is connected to 00 peer ranks. peer ranks. Expected number of connected peer ranks is : Expected number of connected peer ranks is : 00
|
| 603 |
|
|
|
|
|
|
|
|
|
|
| 604 |
[Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
|
| 605 |
-
[
|
| 606 |
-
[
|
| 607 |
-
[
|
| 608 |
-
[
|
| 609 |
-
[
|
| 610 |
-
[
|
| 611 |
-
[
|
| 612 |
-
[rank3]:[W829 23:45:23.529332491 ProcessGroupNCCL.cpp:4789] [PG ID 0 PG GUID 0 Rank 3] using GPU 3 as device used by this process is currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect. You can specify device_id in init_process_group() to force use of a particular device.
|
| 613 |
-
[rank7]:[W829 23:45:23.529419740 ProcessGroupNCCL.cpp:4789] [PG ID 0 PG GUID 0 Rank 7] using GPU 7 as device used by this process is currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect. You can specify device_id in init_process_group() to force use of a particular device.
|
| 614 |
> compiling dataset index builder ...
|
| 615 |
make: Entering directory '/workspace/Megatron-MoE-Echo/megatron/core/datasets'
|
| 616 |
make: Nothing to be done for 'default'.
|
| 617 |
make: Leaving directory '/workspace/Megatron-MoE-Echo/megatron/core/datasets'
|
| 618 |
-
>>> done with dataset index builder. Compilation time: 0.
|
| 619 |
WARNING: constraints for invoking optimized fused softmax kernel are not met. We default back to unfused kernel invocations.
|
| 620 |
> compiling and loading fused kernels ...
|
| 621 |
-
[rank0]:[
|
| 622 |
-
>>> done with compiling and loading fused kernels. Compilation time: 3.
|
| 623 |
-
time to initialize megatron (seconds): 7.
|
| 624 |
-
[after megatron is initialized] datetime: 2026-08-
|
| 625 |
building GPT model ...
|
| 626 |
/usr/local/lib/python3.12/dist-packages/transformer_engine/pytorch/cpu_offload.py:708: DeprecationWarning: Offloading weights is deprecated. Using offload_weights=True does not have any effect.
|
| 627 |
warnings.warn(
|
|
@@ -646,100 +646,100 @@ Params for bucket 1 (206045184 elements, 206045184 padded size):
|
|
| 646 |
module.output_layer.weight
|
| 647 |
Params for bucket 2 (71840000 elements, 71840000 padded size):
|
| 648 |
module.decoder.layers.11.pre_mlp_layernorm.weight
|
|
|
|
| 649 |
module.decoder.layers.11.mlp.router.weight
|
| 650 |
module.decoder.layers.11.self_attention.q_layernorm.weight
|
| 651 |
module.decoder.layers.11.self_attention.linear_qkv.weight
|
| 652 |
module.decoder.layers.11.self_attention.linear_qkv.layer_norm_weight
|
| 653 |
module.decoder.final_layernorm.weight
|
| 654 |
-
module.decoder.layers.11.self_attention.k_layernorm.weight
|
| 655 |
module.decoder.layers.11.self_attention.linear_proj.weight
|
| 656 |
Params for bucket 3 (71835904 elements, 71835904 padded size):
|
| 657 |
module.decoder.layers.10.mlp.router.weight
|
| 658 |
-
module.decoder.layers.10.pre_mlp_layernorm.weight
|
| 659 |
-
module.decoder.layers.10.self_attention.k_layernorm.weight
|
| 660 |
-
module.decoder.layers.10.self_attention.q_layernorm.weight
|
| 661 |
module.decoder.layers.10.self_attention.linear_qkv.weight
|
|
|
|
|
|
|
| 662 |
module.decoder.layers.10.self_attention.linear_qkv.layer_norm_weight
|
|
|
|
| 663 |
module.decoder.layers.10.self_attention.linear_proj.weight
|
| 664 |
Params for bucket 4 (71835904 elements, 71835904 padded size):
|
| 665 |
module.decoder.layers.9.mlp.router.weight
|
| 666 |
-
module.decoder.layers.9.pre_mlp_layernorm.weight
|
| 667 |
-
module.decoder.layers.9.self_attention.linear_proj.weight
|
| 668 |
-
module.decoder.layers.9.self_attention.q_layernorm.weight
|
| 669 |
module.decoder.layers.9.self_attention.k_layernorm.weight
|
|
|
|
|
|
|
| 670 |
module.decoder.layers.9.self_attention.linear_qkv.weight
|
| 671 |
module.decoder.layers.9.self_attention.linear_qkv.layer_norm_weight
|
|
|
|
| 672 |
Params for bucket 5 (71835904 elements, 71835904 padded size):
|
| 673 |
module.decoder.layers.8.mlp.router.weight
|
| 674 |
-
module.decoder.layers.8.pre_mlp_layernorm.weight
|
| 675 |
module.decoder.layers.8.self_attention.k_layernorm.weight
|
|
|
|
| 676 |
module.decoder.layers.8.self_attention.linear_qkv.weight
|
| 677 |
module.decoder.layers.8.self_attention.q_layernorm.weight
|
| 678 |
module.decoder.layers.8.self_attention.linear_qkv.layer_norm_weight
|
| 679 |
module.decoder.layers.8.self_attention.linear_proj.weight
|
| 680 |
Params for bucket 6 (71835904 elements, 71835904 padded size):
|
| 681 |
module.decoder.layers.7.mlp.router.weight
|
| 682 |
-
module.decoder.layers.7.pre_mlp_layernorm.weight
|
| 683 |
module.decoder.layers.7.self_attention.q_layernorm.weight
|
| 684 |
module.decoder.layers.7.self_attention.k_layernorm.weight
|
| 685 |
module.decoder.layers.7.self_attention.linear_qkv.layer_norm_weight
|
| 686 |
module.decoder.layers.7.self_attention.linear_qkv.weight
|
|
|
|
| 687 |
module.decoder.layers.7.self_attention.linear_proj.weight
|
| 688 |
Params for bucket 7 (71835904 elements, 71835904 padded size):
|
| 689 |
module.decoder.layers.6.mlp.router.weight
|
| 690 |
module.decoder.layers.6.pre_mlp_layernorm.weight
|
| 691 |
module.decoder.layers.6.self_attention.k_layernorm.weight
|
| 692 |
-
module.decoder.layers.6.self_attention.linear_qkv.weight
|
| 693 |
module.decoder.layers.6.self_attention.q_layernorm.weight
|
|
|
|
| 694 |
module.decoder.layers.6.self_attention.linear_qkv.layer_norm_weight
|
| 695 |
module.decoder.layers.6.self_attention.linear_proj.weight
|
| 696 |
Params for bucket 8 (71835904 elements, 71835904 padded size):
|
| 697 |
-
module.decoder.layers.5.mlp.router.weight
|
| 698 |
module.decoder.layers.5.pre_mlp_layernorm.weight
|
| 699 |
-
module.decoder.layers.5.self_attention.linear_qkv.weight
|
| 700 |
module.decoder.layers.5.self_attention.q_layernorm.weight
|
| 701 |
module.decoder.layers.5.self_attention.linear_qkv.layer_norm_weight
|
|
|
|
|
|
|
| 702 |
module.decoder.layers.5.self_attention.k_layernorm.weight
|
| 703 |
module.decoder.layers.5.self_attention.linear_proj.weight
|
| 704 |
Params for bucket 9 (71835904 elements, 71835904 padded size):
|
| 705 |
-
module.decoder.layers.4.mlp.router.weight
|
| 706 |
module.decoder.layers.4.pre_mlp_layernorm.weight
|
| 707 |
-
module.decoder.layers.4.self_attention.k_layernorm.weight
|
| 708 |
module.decoder.layers.4.self_attention.q_layernorm.weight
|
| 709 |
module.decoder.layers.4.self_attention.linear_qkv.weight
|
| 710 |
module.decoder.layers.4.self_attention.linear_qkv.layer_norm_weight
|
|
|
|
|
|
|
| 711 |
module.decoder.layers.4.self_attention.linear_proj.weight
|
| 712 |
Params for bucket 10 (71835904 elements, 71835904 padded size):
|
| 713 |
module.decoder.layers.3.mlp.router.weight
|
| 714 |
-
module.decoder.layers.3.pre_mlp_layernorm.weight
|
| 715 |
module.decoder.layers.3.self_attention.linear_qkv.weight
|
| 716 |
module.decoder.layers.3.self_attention.q_layernorm.weight
|
|
|
|
| 717 |
module.decoder.layers.3.self_attention.k_layernorm.weight
|
| 718 |
module.decoder.layers.3.self_attention.linear_qkv.layer_norm_weight
|
| 719 |
module.decoder.layers.3.self_attention.linear_proj.weight
|
| 720 |
Params for bucket 11 (71835904 elements, 71835904 padded size):
|
| 721 |
-
module.decoder.layers.2.mlp.router.weight
|
| 722 |
-
module.decoder.layers.2.self_attention.k_layernorm.weight
|
| 723 |
module.decoder.layers.2.self_attention.q_layernorm.weight
|
|
|
|
|
|
|
| 724 |
module.decoder.layers.2.self_attention.linear_qkv.weight
|
|
|
|
| 725 |
module.decoder.layers.2.self_attention.linear_qkv.layer_norm_weight
|
| 726 |
module.decoder.layers.2.self_attention.linear_proj.weight
|
| 727 |
-
module.decoder.layers.2.pre_mlp_layernorm.weight
|
| 728 |
Params for bucket 12 (71835904 elements, 71835904 padded size):
|
| 729 |
module.decoder.layers.1.mlp.router.weight
|
|
|
|
| 730 |
module.decoder.layers.1.self_attention.linear_qkv.weight
|
| 731 |
-
module.decoder.layers.1.self_attention.q_layernorm.weight
|
| 732 |
module.decoder.layers.1.pre_mlp_layernorm.weight
|
| 733 |
-
module.decoder.layers.1.self_attention.
|
| 734 |
module.decoder.layers.1.self_attention.linear_qkv.layer_norm_weight
|
| 735 |
module.decoder.layers.1.self_attention.linear_proj.weight
|
| 736 |
Params for bucket 13 (71835904 elements, 71835904 padded size):
|
| 737 |
-
module.decoder.layers.0.pre_mlp_layernorm.weight
|
| 738 |
-
module.decoder.layers.0.mlp.router.weight
|
| 739 |
-
module.decoder.layers.0.self_attention.q_layernorm.weight
|
| 740 |
module.decoder.layers.0.self_attention.linear_qkv.weight
|
| 741 |
-
module.decoder.layers.0.self_attention.k_layernorm.weight
|
| 742 |
module.decoder.layers.0.self_attention.linear_qkv.layer_norm_weight
|
|
|
|
|
|
|
|
|
|
|
|
|
| 743 |
module.decoder.layers.0.self_attention.linear_proj.weight
|
| 744 |
Params for bucket 14 (206045184 elements, 206045184 padded size):
|
| 745 |
module.embedding.word_embeddings.weight
|
|
@@ -749,8 +749,8 @@ Params for bucket 1 (44040192 elements, 44040192 padded size):
|
|
| 749 |
module.decoder.layers.11.mlp.experts.linear_fc2.weight2
|
| 750 |
module.decoder.layers.11.mlp.experts.linear_fc2.weight4
|
| 751 |
module.decoder.layers.11.mlp.experts.linear_fc2.weight6
|
| 752 |
-
module.decoder.layers.11.mlp.experts.linear_fc2.weight3
|
| 753 |
module.decoder.layers.11.mlp.experts.linear_fc2.weight1
|
|
|
|
| 754 |
module.decoder.layers.11.mlp.experts.linear_fc2.weight5
|
| 755 |
Params for bucket 2 (44040192 elements, 44040192 padded size):
|
| 756 |
module.decoder.layers.11.mlp.experts.linear_fc2.weight10
|
|
@@ -764,12 +764,12 @@ Params for bucket 3 (50331648 elements, 50331648 padded size):
|
|
| 764 |
module.decoder.layers.11.mlp.experts.linear_fc2.weight14
|
| 765 |
module.decoder.layers.11.mlp.experts.linear_fc1.weight1
|
| 766 |
module.decoder.layers.11.mlp.experts.linear_fc1.weight0
|
| 767 |
-
module.decoder.layers.11.mlp.experts.linear_fc2.weight15
|
| 768 |
module.decoder.layers.11.mlp.experts.linear_fc1.weight2
|
|
|
|
| 769 |
Params for bucket 4 (50331648 elements, 50331648 padded size):
|
| 770 |
module.decoder.layers.11.mlp.experts.linear_fc1.weight3
|
| 771 |
-
module.decoder.layers.11.mlp.experts.linear_fc1.weight6
|
| 772 |
module.decoder.layers.11.mlp.experts.linear_fc1.weight4
|
|
|
|
| 773 |
module.decoder.layers.11.mlp.experts.linear_fc1.weight5
|
| 774 |
Params for bucket 5 (50331648 elements, 50331648 padded size):
|
| 775 |
module.decoder.layers.11.mlp.experts.linear_fc1.weight7
|
|
@@ -783,63 +783,63 @@ Params for bucket 6 (50331648 elements, 50331648 padded size):
|
|
| 783 |
module.decoder.layers.11.mlp.experts.linear_fc1.weight12
|
| 784 |
Params for bucket 7 (44040192 elements, 44040192 padded size):
|
| 785 |
module.decoder.layers.11.mlp.experts.linear_fc1.weight15
|
| 786 |
-
module.decoder.layers.10.mlp.experts.linear_fc2.weight3
|
| 787 |
-
module.decoder.layers.10.mlp.experts.linear_fc2.weight1
|
| 788 |
-
module.decoder.layers.10.mlp.experts.linear_fc2.weight0
|
| 789 |
module.decoder.layers.10.mlp.experts.linear_fc2.weight2
|
|
|
|
| 790 |
module.decoder.layers.10.mlp.experts.linear_fc2.weight4
|
|
|
|
|
|
|
| 791 |
Params for bucket 8 (44040192 elements, 44040192 padded size):
|
|
|
|
|
|
|
|
|
|
| 792 |
module.decoder.layers.10.mlp.experts.linear_fc2.weight5
|
| 793 |
-
module.decoder.layers.10.mlp.experts.linear_fc2.weight7
|
| 794 |
module.decoder.layers.10.mlp.experts.linear_fc2.weight9
|
|
|
|
| 795 |
module.decoder.layers.10.mlp.experts.linear_fc2.weight11
|
| 796 |
-
module.decoder.layers.10.mlp.experts.linear_fc2.weight6
|
| 797 |
-
module.decoder.layers.10.mlp.experts.linear_fc2.weight8
|
| 798 |
-
module.decoder.layers.10.mlp.experts.linear_fc2.weight10
|
| 799 |
Params for bucket 9 (50331648 elements, 50331648 padded size):
|
| 800 |
-
module.decoder.layers.10.mlp.experts.linear_fc2.weight13
|
| 801 |
-
module.decoder.layers.10.mlp.experts.linear_fc2.weight15
|
| 802 |
-
module.decoder.layers.10.mlp.experts.linear_fc1.weight0
|
| 803 |
module.decoder.layers.10.mlp.experts.linear_fc2.weight12
|
| 804 |
module.decoder.layers.10.mlp.experts.linear_fc2.weight14
|
| 805 |
module.decoder.layers.10.mlp.experts.linear_fc1.weight1
|
|
|
|
|
|
|
|
|
|
| 806 |
Params for bucket 10 (50331648 elements, 50331648 padded size):
|
| 807 |
module.decoder.layers.10.mlp.experts.linear_fc1.weight2
|
|
|
|
| 808 |
module.decoder.layers.10.mlp.experts.linear_fc1.weight3
|
| 809 |
module.decoder.layers.10.mlp.experts.linear_fc1.weight4
|
| 810 |
-
module.decoder.layers.10.mlp.experts.linear_fc1.weight5
|
| 811 |
Params for bucket 11 (50331648 elements, 50331648 padded size):
|
| 812 |
module.decoder.layers.10.mlp.experts.linear_fc1.weight6
|
|
|
|
| 813 |
module.decoder.layers.10.mlp.experts.linear_fc1.weight9
|
| 814 |
module.decoder.layers.10.mlp.experts.linear_fc1.weight7
|
| 815 |
-
module.decoder.layers.10.mlp.experts.linear_fc1.weight8
|
| 816 |
Params for bucket 12 (50331648 elements, 50331648 padded size):
|
| 817 |
module.decoder.layers.10.mlp.experts.linear_fc1.weight10
|
|
|
|
| 818 |
module.decoder.layers.10.mlp.experts.linear_fc1.weight12
|
| 819 |
module.decoder.layers.10.mlp.experts.linear_fc1.weight13
|
| 820 |
-
module.decoder.layers.10.mlp.experts.linear_fc1.weight11
|
| 821 |
Params for bucket 13 (44040192 elements, 44040192 padded size):
|
|
|
|
|
|
|
| 822 |
module.decoder.layers.10.mlp.experts.linear_fc1.weight14
|
| 823 |
module.decoder.layers.9.mlp.experts.linear_fc2.weight2
|
| 824 |
module.decoder.layers.9.mlp.experts.linear_fc2.weight0
|
| 825 |
-
module.decoder.layers.9.mlp.experts.linear_fc2.weight1
|
| 826 |
-
module.decoder.layers.10.mlp.experts.linear_fc1.weight15
|
| 827 |
Params for bucket 14 (44040192 elements, 44040192 padded size):
|
| 828 |
-
module.decoder.layers.9.mlp.experts.linear_fc2.weight6
|
| 829 |
-
module.decoder.layers.9.mlp.experts.linear_fc2.weight4
|
| 830 |
-
module.decoder.layers.9.mlp.experts.linear_fc2.weight8
|
| 831 |
module.decoder.layers.9.mlp.experts.linear_fc2.weight3
|
| 832 |
-
module.decoder.layers.9.mlp.experts.linear_fc2.weight7
|
| 833 |
module.decoder.layers.9.mlp.experts.linear_fc2.weight5
|
|
|
|
| 834 |
module.decoder.layers.9.mlp.experts.linear_fc2.weight9
|
|
|
|
|
|
|
|
|
|
| 835 |
Params for bucket 15 (50331648 elements, 50331648 padded size):
|
| 836 |
-
module.decoder.layers.9.mlp.experts.linear_fc2.weight10
|
| 837 |
-
module.decoder.layers.9.mlp.experts.linear_fc2.weight12
|
| 838 |
-
module.decoder.layers.9.mlp.experts.linear_fc2.weight14
|
| 839 |
-
module.decoder.layers.9.mlp.experts.linear_fc1.weight0
|
| 840 |
-
module.decoder.layers.9.mlp.experts.linear_fc2.weight13
|
| 841 |
module.decoder.layers.9.mlp.experts.linear_fc2.weight11
|
|
|
|
| 842 |
module.decoder.layers.9.mlp.experts.linear_fc2.weight15
|
|
|
|
|
|
|
|
|
|
|
|
|
| 843 |
Params for bucket 16 (50331648 elements, 50331648 padded size):
|
| 844 |
module.decoder.layers.9.mlp.experts.linear_fc1.weight1
|
| 845 |
module.decoder.layers.9.mlp.experts.linear_fc1.weight4
|
|
@@ -870,122 +870,122 @@ Params for bucket 20 (44040192 elements, 44040192 padded size):
|
|
| 870 |
module.decoder.layers.8.mlp.experts.linear_fc2.weight7
|
| 871 |
Params for bucket 21 (44040192 elements, 44040192 padded size):
|
| 872 |
module.decoder.layers.8.mlp.experts.linear_fc2.weight8
|
| 873 |
-
module.decoder.layers.8.mlp.experts.linear_fc2.weight10
|
| 874 |
module.decoder.layers.8.mlp.experts.linear_fc2.weight12
|
|
|
|
| 875 |
module.decoder.layers.8.mlp.experts.linear_fc2.weight14
|
| 876 |
module.decoder.layers.8.mlp.experts.linear_fc2.weight9
|
| 877 |
module.decoder.layers.8.mlp.experts.linear_fc2.weight11
|
| 878 |
module.decoder.layers.8.mlp.experts.linear_fc2.weight13
|
| 879 |
Params for bucket 22 (44040192 elements, 44040192 padded size):
|
| 880 |
module.decoder.layers.8.mlp.experts.linear_fc2.weight15
|
| 881 |
-
module.decoder.layers.8.mlp.experts.linear_fc1.weight1
|
| 882 |
module.decoder.layers.8.mlp.experts.linear_fc1.weight0
|
| 883 |
module.decoder.layers.8.mlp.experts.linear_fc1.weight2
|
|
|
|
| 884 |
Params for bucket 23 (50331648 elements, 50331648 padded size):
|
| 885 |
module.decoder.layers.8.mlp.experts.linear_fc1.weight3
|
|
|
|
| 886 |
module.decoder.layers.8.mlp.experts.linear_fc1.weight4
|
| 887 |
module.decoder.layers.8.mlp.experts.linear_fc1.weight5
|
| 888 |
-
module.decoder.layers.8.mlp.experts.linear_fc1.weight6
|
| 889 |
Params for bucket 24 (50331648 elements, 50331648 padded size):
|
| 890 |
module.decoder.layers.8.mlp.experts.linear_fc1.weight7
|
|
|
|
| 891 |
module.decoder.layers.8.mlp.experts.linear_fc1.weight10
|
| 892 |
module.decoder.layers.8.mlp.experts.linear_fc1.weight8
|
| 893 |
-
module.decoder.layers.8.mlp.experts.linear_fc1.weight9
|
| 894 |
Params for bucket 25 (50331648 elements, 50331648 padded size):
|
| 895 |
module.decoder.layers.8.mlp.experts.linear_fc1.weight11
|
|
|
|
| 896 |
module.decoder.layers.8.mlp.experts.linear_fc1.weight13
|
| 897 |
module.decoder.layers.8.mlp.experts.linear_fc1.weight14
|
| 898 |
-
module.decoder.layers.8.mlp.experts.linear_fc1.weight12
|
| 899 |
Params for bucket 26 (44040192 elements, 44040192 padded size):
|
| 900 |
-
module.decoder.layers.8.mlp.experts.linear_fc1.weight15
|
| 901 |
-
module.decoder.layers.7.mlp.experts.linear_fc2.weight3
|
| 902 |
-
module.decoder.layers.7.mlp.experts.linear_fc2.weight1
|
| 903 |
-
module.decoder.layers.7.mlp.experts.linear_fc2.weight0
|
| 904 |
module.decoder.layers.7.mlp.experts.linear_fc2.weight2
|
|
|
|
| 905 |
module.decoder.layers.7.mlp.experts.linear_fc2.weight4
|
|
|
|
|
|
|
|
|
|
| 906 |
Params for bucket 27 (44040192 elements, 44040192 padded size):
|
|
|
|
|
|
|
|
|
|
| 907 |
module.decoder.layers.7.mlp.experts.linear_fc2.weight5
|
| 908 |
module.decoder.layers.7.mlp.experts.linear_fc2.weight9
|
| 909 |
module.decoder.layers.7.mlp.experts.linear_fc2.weight7
|
| 910 |
module.decoder.layers.7.mlp.experts.linear_fc2.weight11
|
| 911 |
-
module.decoder.layers.7.mlp.experts.linear_fc2.weight6
|
| 912 |
-
module.decoder.layers.7.mlp.experts.linear_fc2.weight8
|
| 913 |
-
module.decoder.layers.7.mlp.experts.linear_fc2.weight10
|
| 914 |
Params for bucket 28 (50331648 elements, 50331648 padded size):
|
| 915 |
-
module.decoder.layers.7.mlp.experts.linear_fc2.weight15
|
| 916 |
-
module.decoder.layers.7.mlp.experts.linear_fc1.weight0
|
| 917 |
-
module.decoder.layers.7.mlp.experts.linear_fc2.weight13
|
| 918 |
module.decoder.layers.7.mlp.experts.linear_fc2.weight12
|
| 919 |
module.decoder.layers.7.mlp.experts.linear_fc2.weight14
|
| 920 |
module.decoder.layers.7.mlp.experts.linear_fc1.weight1
|
|
|
|
|
|
|
|
|
|
| 921 |
Params for bucket 29 (50331648 elements, 50331648 padded size):
|
| 922 |
module.decoder.layers.7.mlp.experts.linear_fc1.weight2
|
|
|
|
| 923 |
module.decoder.layers.7.mlp.experts.linear_fc1.weight3
|
| 924 |
module.decoder.layers.7.mlp.experts.linear_fc1.weight4
|
| 925 |
-
module.decoder.layers.7.mlp.experts.linear_fc1.weight5
|
| 926 |
Params for bucket 30 (50331648 elements, 50331648 padded size):
|
| 927 |
module.decoder.layers.7.mlp.experts.linear_fc1.weight6
|
|
|
|
| 928 |
module.decoder.layers.7.mlp.experts.linear_fc1.weight7
|
| 929 |
module.decoder.layers.7.mlp.experts.linear_fc1.weight9
|
| 930 |
-
module.decoder.layers.7.mlp.experts.linear_fc1.weight8
|
| 931 |
Params for bucket 31 (50331648 elements, 50331648 padded size):
|
| 932 |
module.decoder.layers.7.mlp.experts.linear_fc1.weight10
|
| 933 |
-
module.decoder.layers.7.mlp.experts.linear_fc1.weight13
|
| 934 |
module.decoder.layers.7.mlp.experts.linear_fc1.weight12
|
| 935 |
module.decoder.layers.7.mlp.experts.linear_fc1.weight11
|
|
|
|
| 936 |
Params for bucket 32 (44040192 elements, 44040192 padded size):
|
| 937 |
-
module.decoder.layers.7.mlp.experts.linear_fc1.
|
| 938 |
-
module.decoder.layers.6.mlp.experts.linear_fc2.weight2
|
| 939 |
module.decoder.layers.6.mlp.experts.linear_fc2.weight0
|
|
|
|
|
|
|
| 940 |
module.decoder.layers.6.mlp.experts.linear_fc2.weight1
|
| 941 |
-
module.decoder.layers.7.mlp.experts.linear_fc1.weight15
|
| 942 |
Params for bucket 33 (44040192 elements, 44040192 padded size):
|
| 943 |
module.decoder.layers.6.mlp.experts.linear_fc2.weight6
|
| 944 |
module.decoder.layers.6.mlp.experts.linear_fc2.weight4
|
| 945 |
module.decoder.layers.6.mlp.experts.linear_fc2.weight8
|
| 946 |
module.decoder.layers.6.mlp.experts.linear_fc2.weight3
|
| 947 |
-
module.decoder.layers.6.mlp.experts.linear_fc2.weight5
|
| 948 |
module.decoder.layers.6.mlp.experts.linear_fc2.weight7
|
|
|
|
| 949 |
module.decoder.layers.6.mlp.experts.linear_fc2.weight9
|
| 950 |
Params for bucket 34 (50331648 elements, 50331648 padded size):
|
| 951 |
module.decoder.layers.6.mlp.experts.linear_fc2.weight10
|
| 952 |
module.decoder.layers.6.mlp.experts.linear_fc2.weight14
|
| 953 |
module.decoder.layers.6.mlp.experts.linear_fc2.weight12
|
| 954 |
-
module.decoder.layers.6.mlp.experts.linear_fc2.weight11
|
| 955 |
module.decoder.layers.6.mlp.experts.linear_fc2.weight13
|
|
|
|
| 956 |
module.decoder.layers.6.mlp.experts.linear_fc2.weight15
|
| 957 |
module.decoder.layers.6.mlp.experts.linear_fc1.weight0
|
| 958 |
Params for bucket 35 (50331648 elements, 50331648 padded size):
|
| 959 |
module.decoder.layers.6.mlp.experts.linear_fc1.weight1
|
| 960 |
-
module.decoder.layers.6.mlp.experts.linear_fc1.weight2
|
| 961 |
module.decoder.layers.6.mlp.experts.linear_fc1.weight3
|
| 962 |
module.decoder.layers.6.mlp.experts.linear_fc1.weight4
|
|
|
|
| 963 |
Params for bucket 36 (50331648 elements, 50331648 padded size):
|
| 964 |
module.decoder.layers.6.mlp.experts.linear_fc1.weight5
|
| 965 |
-
module.decoder.layers.6.mlp.experts.linear_fc1.weight8
|
| 966 |
module.decoder.layers.6.mlp.experts.linear_fc1.weight6
|
| 967 |
module.decoder.layers.6.mlp.experts.linear_fc1.weight7
|
|
|
|
| 968 |
Params for bucket 37 (50331648 elements, 50331648 padded size):
|
| 969 |
module.decoder.layers.6.mlp.experts.linear_fc1.weight9
|
| 970 |
-
module.decoder.layers.6.mlp.experts.linear_fc1.weight11
|
| 971 |
module.decoder.layers.6.mlp.experts.linear_fc1.weight12
|
| 972 |
module.decoder.layers.6.mlp.experts.linear_fc1.weight10
|
|
|
|
| 973 |
Params for bucket 38 (44040192 elements, 44040192 padded size):
|
| 974 |
module.decoder.layers.6.mlp.experts.linear_fc1.weight13
|
|
|
|
| 975 |
module.decoder.layers.6.mlp.experts.linear_fc1.weight15
|
| 976 |
module.decoder.layers.6.mlp.experts.linear_fc1.weight14
|
| 977 |
-
module.decoder.layers.5.mlp.experts.linear_fc2.weight0
|
| 978 |
Params for bucket 39 (44040192 elements, 44040192 padded size):
|
| 979 |
module.decoder.layers.5.mlp.experts.linear_fc2.weight1
|
| 980 |
-
module.decoder.layers.5.mlp.experts.linear_fc2.weight5
|
| 981 |
module.decoder.layers.5.mlp.experts.linear_fc2.weight3
|
|
|
|
| 982 |
module.decoder.layers.5.mlp.experts.linear_fc2.weight7
|
| 983 |
module.decoder.layers.5.mlp.experts.linear_fc2.weight2
|
| 984 |
module.decoder.layers.5.mlp.experts.linear_fc2.weight4
|
| 985 |
module.decoder.layers.5.mlp.experts.linear_fc2.weight6
|
| 986 |
Params for bucket 40 (44040192 elements, 44040192 padded size):
|
| 987 |
-
module.decoder.layers.5.mlp.experts.linear_fc2.weight11
|
| 988 |
module.decoder.layers.5.mlp.experts.linear_fc2.weight9
|
|
|
|
| 989 |
module.decoder.layers.5.mlp.experts.linear_fc2.weight13
|
| 990 |
module.decoder.layers.5.mlp.experts.linear_fc2.weight8
|
| 991 |
module.decoder.layers.5.mlp.experts.linear_fc2.weight12
|
|
@@ -993,45 +993,45 @@ Params for bucket 40 (44040192 elements, 44040192 padded size):
|
|
| 993 |
module.decoder.layers.5.mlp.experts.linear_fc2.weight14
|
| 994 |
Params for bucket 41 (44040192 elements, 44040192 padded size):
|
| 995 |
module.decoder.layers.5.mlp.experts.linear_fc2.weight15
|
| 996 |
-
module.decoder.layers.5.mlp.experts.linear_fc1.weight0
|
| 997 |
module.decoder.layers.5.mlp.experts.linear_fc1.weight1
|
| 998 |
module.decoder.layers.5.mlp.experts.linear_fc1.weight2
|
|
|
|
| 999 |
Params for bucket 42 (50331648 elements, 50331648 padded size):
|
| 1000 |
module.decoder.layers.5.mlp.experts.linear_fc1.weight3
|
|
|
|
| 1001 |
module.decoder.layers.5.mlp.experts.linear_fc1.weight6
|
| 1002 |
module.decoder.layers.5.mlp.experts.linear_fc1.weight4
|
| 1003 |
-
module.decoder.layers.5.mlp.experts.linear_fc1.weight5
|
| 1004 |
Params for bucket 43 (50331648 elements, 50331648 padded size):
|
| 1005 |
module.decoder.layers.5.mlp.experts.linear_fc1.weight7
|
| 1006 |
-
module.decoder.layers.5.mlp.experts.linear_fc1.weight10
|
| 1007 |
module.decoder.layers.5.mlp.experts.linear_fc1.weight8
|
| 1008 |
module.decoder.layers.5.mlp.experts.linear_fc1.weight9
|
|
|
|
| 1009 |
Params for bucket 44 (50331648 elements, 50331648 padded size):
|
| 1010 |
module.decoder.layers.5.mlp.experts.linear_fc1.weight11
|
| 1011 |
-
module.decoder.layers.5.mlp.experts.linear_fc1.weight13
|
| 1012 |
module.decoder.layers.5.mlp.experts.linear_fc1.weight14
|
| 1013 |
module.decoder.layers.5.mlp.experts.linear_fc1.weight12
|
|
|
|
| 1014 |
Params for bucket 45 (44040192 elements, 44040192 padded size):
|
| 1015 |
module.decoder.layers.5.mlp.experts.linear_fc1.weight15
|
|
|
|
|
|
|
| 1016 |
module.decoder.layers.4.mlp.experts.linear_fc2.weight0
|
| 1017 |
module.decoder.layers.4.mlp.experts.linear_fc2.weight2
|
| 1018 |
module.decoder.layers.4.mlp.experts.linear_fc2.weight4
|
| 1019 |
-
module.decoder.layers.4.mlp.experts.linear_fc2.weight3
|
| 1020 |
-
module.decoder.layers.4.mlp.experts.linear_fc2.weight1
|
| 1021 |
Params for bucket 46 (44040192 elements, 44040192 padded size):
|
| 1022 |
-
module.decoder.layers.4.mlp.experts.linear_fc2.weight8
|
| 1023 |
-
module.decoder.layers.4.mlp.experts.linear_fc2.weight6
|
| 1024 |
-
module.decoder.layers.4.mlp.experts.linear_fc2.weight10
|
| 1025 |
module.decoder.layers.4.mlp.experts.linear_fc2.weight5
|
| 1026 |
-
module.decoder.layers.4.mlp.experts.linear_fc2.weight7
|
| 1027 |
module.decoder.layers.4.mlp.experts.linear_fc2.weight9
|
|
|
|
| 1028 |
module.decoder.layers.4.mlp.experts.linear_fc2.weight11
|
|
|
|
|
|
|
|
|
|
| 1029 |
Params for bucket 47 (50331648 elements, 50331648 padded size):
|
| 1030 |
-
module.decoder.layers.4.mlp.experts.linear_fc2.
|
| 1031 |
-
module.decoder.layers.4.mlp.experts.linear_fc2.weight14
|
| 1032 |
module.decoder.layers.4.mlp.experts.linear_fc1.weight0
|
| 1033 |
module.decoder.layers.4.mlp.experts.linear_fc2.weight15
|
| 1034 |
-
module.decoder.layers.4.mlp.experts.linear_fc2.
|
|
|
|
| 1035 |
module.decoder.layers.4.mlp.experts.linear_fc1.weight1
|
| 1036 |
Params for bucket 48 (50331648 elements, 50331648 padded size):
|
| 1037 |
module.decoder.layers.4.mlp.experts.linear_fc1.weight2
|
|
@@ -1091,21 +1091,21 @@ Params for bucket 57 (44040192 elements, 44040192 padded size):
|
|
| 1091 |
module.decoder.layers.2.mlp.experts.linear_fc2.weight0
|
| 1092 |
module.decoder.layers.3.mlp.experts.linear_fc1.weight14
|
| 1093 |
Params for bucket 58 (44040192 elements, 44040192 padded size):
|
| 1094 |
-
module.decoder.layers.2.mlp.experts.linear_fc2.weight4
|
| 1095 |
-
module.decoder.layers.2.mlp.experts.linear_fc2.weight2
|
| 1096 |
-
module.decoder.layers.2.mlp.experts.linear_fc2.weight6
|
| 1097 |
module.decoder.layers.2.mlp.experts.linear_fc2.weight1
|
| 1098 |
-
module.decoder.layers.2.mlp.experts.linear_fc2.weight3
|
| 1099 |
module.decoder.layers.2.mlp.experts.linear_fc2.weight5
|
|
|
|
| 1100 |
module.decoder.layers.2.mlp.experts.linear_fc2.weight7
|
|
|
|
|
|
|
|
|
|
| 1101 |
Params for bucket 59 (44040192 elements, 44040192 padded size):
|
|
|
|
|
|
|
|
|
|
| 1102 |
module.decoder.layers.2.mlp.experts.linear_fc2.weight8
|
| 1103 |
module.decoder.layers.2.mlp.experts.linear_fc2.weight10
|
| 1104 |
module.decoder.layers.2.mlp.experts.linear_fc2.weight12
|
| 1105 |
module.decoder.layers.2.mlp.experts.linear_fc2.weight14
|
| 1106 |
-
module.decoder.layers.2.mlp.experts.linear_fc2.weight9
|
| 1107 |
-
module.decoder.layers.2.mlp.experts.linear_fc2.weight11
|
| 1108 |
-
module.decoder.layers.2.mlp.experts.linear_fc2.weight13
|
| 1109 |
Params for bucket 60 (44040192 elements, 44040192 padded size):
|
| 1110 |
module.decoder.layers.2.mlp.experts.linear_fc2.weight15
|
| 1111 |
module.decoder.layers.2.mlp.experts.linear_fc1.weight1
|
|
@@ -1128,14 +1128,14 @@ Params for bucket 63 (50331648 elements, 50331648 padded size):
|
|
| 1128 |
module.decoder.layers.2.mlp.experts.linear_fc1.weight12
|
| 1129 |
Params for bucket 64 (44040192 elements, 44040192 padded size):
|
| 1130 |
module.decoder.layers.2.mlp.experts.linear_fc1.weight15
|
| 1131 |
-
module.decoder.layers.1.mlp.experts.linear_fc2.weight2
|
| 1132 |
module.decoder.layers.1.mlp.experts.linear_fc2.weight0
|
|
|
|
| 1133 |
module.decoder.layers.1.mlp.experts.linear_fc2.weight4
|
| 1134 |
module.decoder.layers.1.mlp.experts.linear_fc2.weight3
|
| 1135 |
module.decoder.layers.1.mlp.experts.linear_fc2.weight1
|
| 1136 |
Params for bucket 65 (44040192 elements, 44040192 padded size):
|
| 1137 |
-
module.decoder.layers.1.mlp.experts.linear_fc2.weight8
|
| 1138 |
module.decoder.layers.1.mlp.experts.linear_fc2.weight6
|
|
|
|
| 1139 |
module.decoder.layers.1.mlp.experts.linear_fc2.weight10
|
| 1140 |
module.decoder.layers.1.mlp.experts.linear_fc2.weight5
|
| 1141 |
module.decoder.layers.1.mlp.experts.linear_fc2.weight9
|
|
@@ -1145,14 +1145,14 @@ Params for bucket 66 (50331648 elements, 50331648 padded size):
|
|
| 1145 |
module.decoder.layers.1.mlp.experts.linear_fc2.weight12
|
| 1146 |
module.decoder.layers.1.mlp.experts.linear_fc2.weight14
|
| 1147 |
module.decoder.layers.1.mlp.experts.linear_fc1.weight1
|
| 1148 |
-
module.decoder.layers.1.mlp.experts.linear_fc2.weight15
|
| 1149 |
-
module.decoder.layers.1.mlp.experts.linear_fc2.weight13
|
| 1150 |
module.decoder.layers.1.mlp.experts.linear_fc1.weight0
|
|
|
|
|
|
|
| 1151 |
Params for bucket 67 (50331648 elements, 50331648 padded size):
|
| 1152 |
module.decoder.layers.1.mlp.experts.linear_fc1.weight2
|
|
|
|
| 1153 |
module.decoder.layers.1.mlp.experts.linear_fc1.weight4
|
| 1154 |
module.decoder.layers.1.mlp.experts.linear_fc1.weight5
|
| 1155 |
-
module.decoder.layers.1.mlp.experts.linear_fc1.weight3
|
| 1156 |
Params for bucket 68 (50331648 elements, 50331648 padded size):
|
| 1157 |
module.decoder.layers.1.mlp.experts.linear_fc1.weight6
|
| 1158 |
module.decoder.layers.1.mlp.experts.linear_fc1.weight7
|
|
@@ -1160,53 +1160,53 @@ Params for bucket 68 (50331648 elements, 50331648 padded size):
|
|
| 1160 |
module.decoder.layers.1.mlp.experts.linear_fc1.weight9
|
| 1161 |
Params for bucket 69 (50331648 elements, 50331648 padded size):
|
| 1162 |
module.decoder.layers.1.mlp.experts.linear_fc1.weight10
|
|
|
|
| 1163 |
module.decoder.layers.1.mlp.experts.linear_fc1.weight11
|
| 1164 |
module.decoder.layers.1.mlp.experts.linear_fc1.weight12
|
| 1165 |
-
module.decoder.layers.1.mlp.experts.linear_fc1.weight13
|
| 1166 |
Params for bucket 70 (44040192 elements, 44040192 padded size):
|
|
|
|
| 1167 |
module.decoder.layers.1.mlp.experts.linear_fc1.weight15
|
| 1168 |
-
module.decoder.layers.0.mlp.experts.linear_fc2.weight2
|
| 1169 |
-
module.decoder.layers.0.mlp.experts.linear_fc2.weight0
|
| 1170 |
module.decoder.layers.1.mlp.experts.linear_fc1.weight14
|
| 1171 |
-
module.decoder.layers.0.mlp.experts.linear_fc2.
|
|
|
|
| 1172 |
Params for bucket 71 (44040192 elements, 44040192 padded size):
|
| 1173 |
-
module.decoder.layers.0.mlp.experts.linear_fc2.weight4
|
| 1174 |
-
module.decoder.layers.0.mlp.experts.linear_fc2.weight6
|
| 1175 |
-
module.decoder.layers.0.mlp.experts.linear_fc2.weight8
|
| 1176 |
module.decoder.layers.0.mlp.experts.linear_fc2.weight3
|
| 1177 |
-
module.decoder.layers.0.mlp.experts.linear_fc2.weight5
|
| 1178 |
module.decoder.layers.0.mlp.experts.linear_fc2.weight7
|
|
|
|
| 1179 |
module.decoder.layers.0.mlp.experts.linear_fc2.weight9
|
|
|
|
|
|
|
|
|
|
| 1180 |
Params for bucket 72 (50331648 elements, 50331648 padded size):
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1181 |
module.decoder.layers.0.mlp.experts.linear_fc2.weight10
|
| 1182 |
module.decoder.layers.0.mlp.experts.linear_fc2.weight14
|
| 1183 |
module.decoder.layers.0.mlp.experts.linear_fc2.weight12
|
| 1184 |
-
module.decoder.layers.0.mlp.experts.linear_fc1.weight0
|
| 1185 |
-
module.decoder.layers.0.mlp.experts.linear_fc2.weight11
|
| 1186 |
-
module.decoder.layers.0.mlp.experts.linear_fc2.weight13
|
| 1187 |
-
module.decoder.layers.0.mlp.experts.linear_fc2.weight15
|
| 1188 |
Params for bucket 73 (50331648 elements, 50331648 padded size):
|
| 1189 |
module.decoder.layers.0.mlp.experts.linear_fc1.weight1
|
| 1190 |
-
module.decoder.layers.0.mlp.experts.linear_fc1.weight4
|
| 1191 |
module.decoder.layers.0.mlp.experts.linear_fc1.weight3
|
| 1192 |
module.decoder.layers.0.mlp.experts.linear_fc1.weight2
|
|
|
|
| 1193 |
Params for bucket 74 (50331648 elements, 50331648 padded size):
|
| 1194 |
module.decoder.layers.0.mlp.experts.linear_fc1.weight5
|
| 1195 |
module.decoder.layers.0.mlp.experts.linear_fc1.weight8
|
| 1196 |
-
module.decoder.layers.0.mlp.experts.linear_fc1.weight7
|
| 1197 |
module.decoder.layers.0.mlp.experts.linear_fc1.weight6
|
|
|
|
| 1198 |
Params for bucket 75 (50331648 elements, 50331648 padded size):
|
| 1199 |
module.decoder.layers.0.mlp.experts.linear_fc1.weight9
|
| 1200 |
-
module.decoder.layers.0.mlp.experts.linear_fc1.weight11
|
| 1201 |
-
module.decoder.layers.0.mlp.experts.linear_fc1.weight10
|
| 1202 |
module.decoder.layers.0.mlp.experts.linear_fc1.weight12
|
|
|
|
|
|
|
| 1203 |
Params for bucket 76 (37748736 elements, 37748736 padded size):
|
| 1204 |
module.decoder.layers.0.mlp.experts.linear_fc1.weight13
|
| 1205 |
module.decoder.layers.0.mlp.experts.linear_fc1.weight14
|
| 1206 |
module.decoder.layers.0.mlp.experts.linear_fc1.weight15
|
| 1207 |
-
INFO:megatron.core.optimizer:Setting up optimizer with config OptimizerConfig(optimizer='adam', lr=0.0001, min_lr=1e-05, decoupled_lr=None, decoupled_min_lr=None, weight_decay=0.1, fp8_recipe='mxfp8', fp16=False, bf16=True, reuse_grad_buf_for_mxfp8_param_ag=False, params_dtype=torch.bfloat16, use_precision_aware_optimizer=False, store_param_remainders=True, main_grads_dtype=torch.float32, main_params_dtype=torch.float32, exp_avg_dtype=torch.float32, exp_avg_sq_dtype=torch.float32, loss_scale=None, initial_loss_scale=4294967296, min_loss_scale=1.0, loss_scale_window=1000, hysteresis=2, adam_beta1=0.9, adam_beta2=0.999, adam_eps=1e-08, sgd_momentum=0.9, use_distributed_optimizer=True, overlap_param_gather_with_optimizer_step=False, optimizer_cpu_offload=False, optimizer_offload_fraction=1.0, use_torch_optimizer_for_cpu_offload=False, overlap_cpu_optimizer_d2h_h2d=False, pin_cpu_grads=True, pin_cpu_params=True, clip_grad=1.0, log_num_zeros_in_grad=False, barrier_with_L1_time=True, timers=<megatron.core.timers.Timers object at
|
| 1208 |
INFO:megatron.core.optimizer_param_scheduler:> learning rate decay style: cosine
|
| 1209 |
-
[after model, optimizer, and learning rate scheduler are built] datetime: 2026-08-
|
| 1210 |
> building train, validation, and test datasets ...
|
| 1211 |
> datasets target sizes (minimum size):
|
| 1212 |
train: 96
|
|
@@ -1214,7 +1214,7 @@ INFO:megatron.core.optimizer_param_scheduler:> learning rate decay style: cosine
|
|
| 1214 |
test: 0
|
| 1215 |
INFO:megatron.core.datasets.blended_megatron_dataset_config:Let split_matrix = [(0, 1.0), None, None]
|
| 1216 |
> building train, validation, and test datasets for GPT ...
|
| 1217 |
-
INFO:megatron.core.datasets.blended_megatron_dataset_builder:Building GPTDataset splits with sizes=(96, 0, 0) and config=GPTDatasetConfig(random_seed=1234, sequence_length=8192, blend=(['/workspace/data/wikitext/wikitext103_gpt2_text_document'], None), blend_per_split=None, split='100,0,0', split_matrix=[(0, 1.0), None, None], num_dataset_builder_threads=1, path_to_cache=None, mmap_bin_files=True, mock=False, tokenizer=<megatron.training.tokenizer.tokenizer._GPT2BPETokenizer object at
|
| 1218 |
INFO:megatron.core.datasets.indexed_dataset:Load the _IndexReader from /workspace/data/wikitext/wikitext103_gpt2_text_document.idx
|
| 1219 |
INFO:megatron.core.datasets.indexed_dataset: Extract the sequence lengths
|
| 1220 |
INFO:megatron.core.datasets.indexed_dataset: Extract the sequence pointers
|
|
@@ -1227,14 +1227,14 @@ INFO:megatron.core.datasets.gpt_dataset: Load the sample index from eadc3756447e
|
|
| 1227 |
INFO:megatron.core.datasets.gpt_dataset: Load the shuffle index from eadc3756447e39d609e8aa728a91f451-GPTDataset-train-shuffle_index.npy
|
| 1228 |
INFO:megatron.core.datasets.gpt_dataset:> total number of samples: 14536
|
| 1229 |
> finished creating GPT datasets ...
|
| 1230 |
-
[after dataloaders are built] datetime: 2026-08-
|
| 1231 |
done with setup ...
|
| 1232 |
-
(min, max) time across ranks (ms):
|
| 1233 |
-
model-and-optimizer-setup ......................: (3261.11, 3293.81)
|
| 1234 |
-
train/valid/test-data-iterators-setup ..........: (63.05, 68.76)
|
| 1235 |
training ...
|
| 1236 |
Overwriting rerun_state_machine.current_iteration from -1 to 0...
|
| 1237 |
-
|
|
|
|
|
|
|
|
|
|
| 1238 |
INFO:moonep.api:MoonEP comm buffer sizes:
|
| 1239 |
config : R=8, S=8192, H=4096, E=128, K=8, NvS=69600, NvS_padded=69632
|
| 1240 |
sm_count : num_sms=32, num_sms_dedup=148
|
|
@@ -1246,44 +1246,43 @@ INFO:moonep.api:MoonEP comm buffer sizes:
|
|
| 1246 |
[MoonEP selfcheck 2/4] dropped groups empty OK | covered rows [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 128, 129, 130, 131, 132, 133, 134, 135, 136, 137, 138, 139, 140, 141, 142, 143] sizes [9344, 2688, 128, 24192, 128, 0, 128, 256, 128, 768, 10496, 128, 128, 2432, 128, 1536, 14208, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0] | duplicated experts [[69, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1], [88, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1], [55, 52, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1], [-1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1], [63, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1], [124, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1], [10, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1], [58, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1]]
|
| 1247 |
[MoonEP selfcheck 3/4] dropped groups empty OK | covered rows [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 128, 129, 130, 131, 132, 133, 134, 135, 136, 137, 138, 139, 140, 141, 142, 143] sizes [128, 128, 128, 0, 128, 44416, 128, 128, 128, 6912, 128, 256, 128, 128, 128, 128, 13824, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0] | duplicated experts [[78, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1], [111, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1], [62, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1], [-1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1], [57, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1], [106, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1], [121, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1], [57, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1]]
|
| 1248 |
[MoonEP selfcheck 4/4] dropped groups empty OK | covered rows [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 128, 129, 130, 131, 132, 133, 134, 135, 136, 137, 138, 139, 140, 141, 142, 143] sizes [2304, 128, 128, 512, 128, 12416, 128, 1152, 1280, 16768, 640, 29056, 128, 128, 128, 1664, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0] | duplicated experts [[-1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1], [118, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1], [11, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1], [118, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1], [39, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1], [11, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1], [73, 79, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1], [86, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1]]
|
|
|
|
| 1249 |
Number of parameters in transformer block in billions: 29.85
|
| 1250 |
Number of parameters in embedding layers in billions: 0.41
|
| 1251 |
Total number of parameters in billions: 30.26
|
| 1252 |
Number of parameters in most loaded shard in billions: 30.2590
|
| 1253 |
Theoretical memory footprints: weight and optimizer=216428.96 MB
|
| 1254 |
-
[2026-08-29 23:45:53] iteration 1/ 12 | consumed samples: 8 | elapsed time per iteration (ms): 21086.4 | throughput per GPU (TFLOP/s/GPU): 8.6 | learning rate: 2.000000E-05 | global batch size: 8 | lm loss: 1.115993E+01 | loss scale: 1.0 | grad norm: 44.807 | number of skipped iterations: 0 | number of nan iterations: 0 |
|
| 1255 |
[Rank 0] (after 1 iterations) memory (MB) | allocated: 67239.548828125 | max allocated: 67239.54931640625 | reserved: 68548.0 | max reserved: 68548.0
|
| 1256 |
-
[2026-08-
|
| 1257 |
-
[2026-08-
|
| 1258 |
Capture CUDA graph for training!!!
|
| 1259 |
CUDA graph capture done!!!
|
| 1260 |
-
[2026-08-
|
| 1261 |
-
[2026-08-
|
| 1262 |
-
[2026-08-
|
| 1263 |
-
[2026-08-
|
| 1264 |
-
[2026-08-
|
| 1265 |
-
[2026-08-
|
| 1266 |
-
[2026-08-
|
| 1267 |
-
[2026-08-
|
| 1268 |
-
[2026-08-
|
| 1269 |
-
[after training is done] datetime: 2026-08-
|
| 1270 |
-
[rank0]:[
|
| 1271 |
-
[
|
| 1272 |
-
[rank6]:[
|
| 1273 |
-
[
|
| 1274 |
-
[
|
| 1275 |
-
[
|
| 1276 |
-
[
|
| 1277 |
-
[rank1]:[
|
| 1278 |
-
|
| 1279 |
-
|
| 1280 |
-
|
| 1281 |
-
|
| 1282 |
-
|
| 1283 |
-
|
| 1284 |
-
|
| 1285 |
-
|
| 1286 |
-
W0829 23:46:04.692000 237461 torch/distributed/elastic/multiprocessing/api.py:900] Sending process 237658 closing signal SIGTERM
|
| 1287 |
Traceback (most recent call last):
|
| 1288 |
File "/usr/local/bin/torchrun", line 8, in <module>
|
| 1289 |
sys.exit(main())
|
|
@@ -1311,4 +1310,4 @@ Traceback (most recent call last):
|
|
| 1311 |
time.sleep(monitor_interval)
|
| 1312 |
File "/usr/local/lib/python3.12/dist-packages/torch/distributed/elastic/multiprocessing/api.py", line 84, in _terminate_process_handler
|
| 1313 |
raise SignalException(f"Process {os.getpid()} got signal: {sigval}", sigval=sigval)
|
| 1314 |
-
torch.distributed.elastic.multiprocessing.api.SignalException: Process
|
|
|
|
| 1 |
+
[01:31:18] WARN: moonep_backend_async_grad_reduce lacks --moe-router-fusion; QWEN3_235B runs unfused (timing only, routing identical)
|
| 2 |
==============================================================
|
| 3 |
QWEN3_235B / moonep (slots/rank=16 policy=matched)
|
| 4 |
E=128 K=8 H'=1536 m=16
|
| 5 |
L=12 seq=8192 gbs=8 (1 microbatches)
|
| 6 |
cap_mode=none cap=none cg=1 recompute=none
|
| 7 |
memory_dump=none snapshots=/workspace/tierb-fixed-moonep-full/moonep/tierb/tierb_gbs8/QWEN3_235B__moonep__matched__seq8192__gbs8__capnone__async/rep1/memory_snapshots/snapshot.pickle
|
| 8 |
+
repo=/workspace/Megatron-MoE-Echo @ 2e68c8ad
|
| 9 |
==============================================================
|
| 10 |
+
W0830 01:31:19.434000 380220 torch/distributed/run.py:766]
|
| 11 |
+
W0830 01:31:19.434000 380220 torch/distributed/run.py:766] *****************************************
|
| 12 |
+
W0830 01:31:19.434000 380220 torch/distributed/run.py:766] Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed.
|
| 13 |
+
W0830 01:31:19.434000 380220 torch/distributed/run.py:766] *****************************************
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 14 |
/workspace/Megatron-MoE-Echo/megatron/training/arguments.py:831: UserWarning: Disabling sequence parallelism because tensor model parallelism is disabled
|
| 15 |
warnings.warn("Disabling sequence parallelism because tensor model parallelism is disabled")
|
| 16 |
using world size: 8, data-parallel size: 8, context-parallel size: 1, hierarchical context-parallel sizes: None, tensor-model-parallel size: 1, pipeline-model-parallel size: 1
|
|
|
|
| 568 |
warnings.warn("Disabling sequence parallelism because tensor model parallelism is disabled")
|
| 569 |
/workspace/Megatron-MoE-Echo/megatron/training/arguments.py:831: UserWarning: Disabling sequence parallelism because tensor model parallelism is disabled
|
| 570 |
warnings.warn("Disabling sequence parallelism because tensor model parallelism is disabled")
|
| 571 |
+
WARNING: one_logger package is required to enable e2e metrics tracking. please go to https://confluence.nvidia.com/display/MLWFO/Package+Repositories for details to install it
|
| 572 |
/workspace/Megatron-MoE-Echo/megatron/training/arguments.py:831: UserWarning: Disabling sequence parallelism because tensor model parallelism is disabled
|
| 573 |
warnings.warn("Disabling sequence parallelism because tensor model parallelism is disabled")
|
| 574 |
+
/workspace/Megatron-MoE-Echo/megatron/training/arguments.py:831: UserWarning: Disabling sequence parallelism because tensor model parallelism is disabled
|
| 575 |
+
warnings.warn("Disabling sequence parallelism because tensor model parallelism is disabled")
|
| 576 |
+
/workspace/Megatron-MoE-Echo/megatron/training/arguments.py:831: UserWarning: Disabling sequence parallelism because tensor model parallelism is disabled
|
| 577 |
+
warnings.warn("Disabling sequence parallelism because tensor model parallelism is disabled")
|
| 578 |
+
/workspace/Megatron-MoE-Echo/megatron/training/arguments.py:831: UserWarning: Disabling sequence parallelism because tensor model parallelism is disabled
|
| 579 |
+
warnings.warn("Disabling sequence parallelism because tensor model parallelism is disabled")
|
| 580 |
+
[Gloo] Rank [Gloo] Rank 0 is connected to 72 peer ranks. is connected to Expected number of connected peer ranks is : 77 peer ranks. Expected number of connected peer ranks is :
|
| 581 |
+
7
|
| 582 |
[Gloo] Rank 4 is connected to 7 peer ranks. Expected number of connected peer ranks is : 7
|
| 583 |
[Gloo] Rank 1 is connected to 7 peer ranks. Expected number of connected peer ranks is : 7
|
| 584 |
+
[Gloo] Rank 5 is connected to 7 peer ranks. Expected number of connected peer ranks is : 7[Gloo] Rank
|
| 585 |
+
3 is connected to 7 peer ranks. Expected number of connected peer ranks is : 7
|
|
|
|
| 586 |
[Gloo] Rank 6 is connected to 7 peer ranks. Expected number of connected peer ranks is : 7
|
| 587 |
+
[Gloo] Rank 7 is connected to 7 peer ranks. Expected number of connected peer ranks is : 7
|
|
|
|
|
|
|
| 588 |
[Gloo] Rank 4 is connected to 7 peer ranks. Expected number of connected peer ranks is : 7
|
| 589 |
[Gloo] Rank 0 is connected to 7 peer ranks. Expected number of connected peer ranks is : 7
|
| 590 |
+
[Gloo] Rank 2 is connected to 7 peer ranks. Expected number of connected peer ranks is : 7
|
| 591 |
[Gloo] Rank 1 is connected to 7 peer ranks. Expected number of connected peer ranks is : 7
|
| 592 |
+
[Gloo] Rank 3 is connected to 7 peer ranks. Expected number of connected peer ranks is : 7
|
| 593 |
+
[Gloo] Rank 5 is connected to 7 peer ranks. Expected number of connected peer ranks is : 7
|
| 594 |
+
[Gloo] Rank [Gloo] Rank 6 is connected to 77 is connected to peer ranks. 7Expected number of connected peer ranks is : peer ranks. 7Expected number of connected peer ranks is :
|
| 595 |
+
7
|
| 596 |
[Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
|
| 597 |
> initialized tensor model parallel with size 1
|
| 598 |
> initialized pipeline model parallel with size 1
|
| 599 |
> setting random seeds to 1234 ...
|
| 600 |
+
[Gloo] Rank [Gloo] Rank 0[Gloo] Rank 00 is connected to is connected to is connected to 000 peer ranks. peer ranks. peer ranks. Expected number of connected peer ranks is : Expected number of connected peer ranks is : Expected number of connected peer ranks is : 000
|
| 601 |
|
|
|
|
| 602 |
|
| 603 |
+
[Gloo] Rank [Gloo] Rank 00 is connected to is connected to 00 peer ranks. [Gloo] Rank peer ranks. Expected number of connected peer ranks is : Expected number of connected peer ranks is : 000 is connected to
|
| 604 |
+
|
| 605 |
+
0 peer ranks. Expected number of connected peer ranks is : 0
|
| 606 |
[Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
|
| 607 |
+
[rank7]:[W830 01:31:37.479166448 ProcessGroupNCCL.cpp:4789] [PG ID 0 PG GUID 0 Rank 7] using GPU 7 as device used by this process is currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect. You can specify device_id in init_process_group() to force use of a particular device.
|
| 608 |
+
[rank6]:[W830 01:31:37.479214134 ProcessGroupNCCL.cpp:4789] [PG ID 0 PG GUID 0 Rank 6] using GPU 6 as device used by this process is currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect. You can specify device_id in init_process_group() to force use of a particular device.
|
| 609 |
+
[rank1]:[W830 01:31:37.479251837 ProcessGroupNCCL.cpp:4789] [PG ID 0 PG GUID 0 Rank 1] using GPU 1 as device used by this process is currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect. You can specify device_id in init_process_group() to force use of a particular device.
|
| 610 |
+
[rank4]:[W830 01:31:37.481705539 ProcessGroupNCCL.cpp:4789] [PG ID 0 PG GUID 0 Rank 4] using GPU 4 as device used by this process is currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect. You can specify device_id in init_process_group() to force use of a particular device.
|
| 611 |
+
[rank3]:[W830 01:31:37.481736938 ProcessGroupNCCL.cpp:4789] [PG ID 0 PG GUID 0 Rank 3] using GPU 3 as device used by this process is currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect. You can specify device_id in init_process_group() to force use of a particular device.
|
| 612 |
+
[rank2]:[W830 01:31:37.481915707 ProcessGroupNCCL.cpp:4789] [PG ID 0 PG GUID 0 Rank 2] using GPU 2 as device used by this process is currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect. You can specify device_id in init_process_group() to force use of a particular device.
|
| 613 |
+
[rank5]:[W830 01:31:37.481926780 ProcessGroupNCCL.cpp:4789] [PG ID 0 PG GUID 0 Rank 5] using GPU 5 as device used by this process is currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect. You can specify device_id in init_process_group() to force use of a particular device.
|
|
|
|
|
|
|
| 614 |
> compiling dataset index builder ...
|
| 615 |
make: Entering directory '/workspace/Megatron-MoE-Echo/megatron/core/datasets'
|
| 616 |
make: Nothing to be done for 'default'.
|
| 617 |
make: Leaving directory '/workspace/Megatron-MoE-Echo/megatron/core/datasets'
|
| 618 |
+
>>> done with dataset index builder. Compilation time: 0.036 seconds
|
| 619 |
WARNING: constraints for invoking optimized fused softmax kernel are not met. We default back to unfused kernel invocations.
|
| 620 |
> compiling and loading fused kernels ...
|
| 621 |
+
[rank0]:[W830 01:31:37.645502642 ProcessGroupNCCL.cpp:4789] [PG ID 0 PG GUID 0 Rank 0] using GPU 0 as device used by this process is currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect. You can specify device_id in init_process_group() to force use of a particular device.
|
| 622 |
+
>>> done with compiling and loading fused kernels. Compilation time: 3.763 seconds
|
| 623 |
+
time to initialize megatron (seconds): 7.462
|
| 624 |
+
[after megatron is initialized] datetime: 2026-08-30 01:31:43
|
| 625 |
building GPT model ...
|
| 626 |
/usr/local/lib/python3.12/dist-packages/transformer_engine/pytorch/cpu_offload.py:708: DeprecationWarning: Offloading weights is deprecated. Using offload_weights=True does not have any effect.
|
| 627 |
warnings.warn(
|
|
|
|
| 646 |
module.output_layer.weight
|
| 647 |
Params for bucket 2 (71840000 elements, 71840000 padded size):
|
| 648 |
module.decoder.layers.11.pre_mlp_layernorm.weight
|
| 649 |
+
module.decoder.layers.11.self_attention.k_layernorm.weight
|
| 650 |
module.decoder.layers.11.mlp.router.weight
|
| 651 |
module.decoder.layers.11.self_attention.q_layernorm.weight
|
| 652 |
module.decoder.layers.11.self_attention.linear_qkv.weight
|
| 653 |
module.decoder.layers.11.self_attention.linear_qkv.layer_norm_weight
|
| 654 |
module.decoder.final_layernorm.weight
|
|
|
|
| 655 |
module.decoder.layers.11.self_attention.linear_proj.weight
|
| 656 |
Params for bucket 3 (71835904 elements, 71835904 padded size):
|
| 657 |
module.decoder.layers.10.mlp.router.weight
|
|
|
|
|
|
|
|
|
|
| 658 |
module.decoder.layers.10.self_attention.linear_qkv.weight
|
| 659 |
+
module.decoder.layers.10.self_attention.q_layernorm.weight
|
| 660 |
+
module.decoder.layers.10.pre_mlp_layernorm.weight
|
| 661 |
module.decoder.layers.10.self_attention.linear_qkv.layer_norm_weight
|
| 662 |
+
module.decoder.layers.10.self_attention.k_layernorm.weight
|
| 663 |
module.decoder.layers.10.self_attention.linear_proj.weight
|
| 664 |
Params for bucket 4 (71835904 elements, 71835904 padded size):
|
| 665 |
module.decoder.layers.9.mlp.router.weight
|
|
|
|
|
|
|
|
|
|
| 666 |
module.decoder.layers.9.self_attention.k_layernorm.weight
|
| 667 |
+
module.decoder.layers.9.self_attention.q_layernorm.weight
|
| 668 |
+
module.decoder.layers.9.pre_mlp_layernorm.weight
|
| 669 |
module.decoder.layers.9.self_attention.linear_qkv.weight
|
| 670 |
module.decoder.layers.9.self_attention.linear_qkv.layer_norm_weight
|
| 671 |
+
module.decoder.layers.9.self_attention.linear_proj.weight
|
| 672 |
Params for bucket 5 (71835904 elements, 71835904 padded size):
|
| 673 |
module.decoder.layers.8.mlp.router.weight
|
|
|
|
| 674 |
module.decoder.layers.8.self_attention.k_layernorm.weight
|
| 675 |
+
module.decoder.layers.8.pre_mlp_layernorm.weight
|
| 676 |
module.decoder.layers.8.self_attention.linear_qkv.weight
|
| 677 |
module.decoder.layers.8.self_attention.q_layernorm.weight
|
| 678 |
module.decoder.layers.8.self_attention.linear_qkv.layer_norm_weight
|
| 679 |
module.decoder.layers.8.self_attention.linear_proj.weight
|
| 680 |
Params for bucket 6 (71835904 elements, 71835904 padded size):
|
| 681 |
module.decoder.layers.7.mlp.router.weight
|
|
|
|
| 682 |
module.decoder.layers.7.self_attention.q_layernorm.weight
|
| 683 |
module.decoder.layers.7.self_attention.k_layernorm.weight
|
| 684 |
module.decoder.layers.7.self_attention.linear_qkv.layer_norm_weight
|
| 685 |
module.decoder.layers.7.self_attention.linear_qkv.weight
|
| 686 |
+
module.decoder.layers.7.pre_mlp_layernorm.weight
|
| 687 |
module.decoder.layers.7.self_attention.linear_proj.weight
|
| 688 |
Params for bucket 7 (71835904 elements, 71835904 padded size):
|
| 689 |
module.decoder.layers.6.mlp.router.weight
|
| 690 |
module.decoder.layers.6.pre_mlp_layernorm.weight
|
| 691 |
module.decoder.layers.6.self_attention.k_layernorm.weight
|
|
|
|
| 692 |
module.decoder.layers.6.self_attention.q_layernorm.weight
|
| 693 |
+
module.decoder.layers.6.self_attention.linear_qkv.weight
|
| 694 |
module.decoder.layers.6.self_attention.linear_qkv.layer_norm_weight
|
| 695 |
module.decoder.layers.6.self_attention.linear_proj.weight
|
| 696 |
Params for bucket 8 (71835904 elements, 71835904 padded size):
|
|
|
|
| 697 |
module.decoder.layers.5.pre_mlp_layernorm.weight
|
|
|
|
| 698 |
module.decoder.layers.5.self_attention.q_layernorm.weight
|
| 699 |
module.decoder.layers.5.self_attention.linear_qkv.layer_norm_weight
|
| 700 |
+
module.decoder.layers.5.mlp.router.weight
|
| 701 |
+
module.decoder.layers.5.self_attention.linear_qkv.weight
|
| 702 |
module.decoder.layers.5.self_attention.k_layernorm.weight
|
| 703 |
module.decoder.layers.5.self_attention.linear_proj.weight
|
| 704 |
Params for bucket 9 (71835904 elements, 71835904 padded size):
|
|
|
|
| 705 |
module.decoder.layers.4.pre_mlp_layernorm.weight
|
|
|
|
| 706 |
module.decoder.layers.4.self_attention.q_layernorm.weight
|
| 707 |
module.decoder.layers.4.self_attention.linear_qkv.weight
|
| 708 |
module.decoder.layers.4.self_attention.linear_qkv.layer_norm_weight
|
| 709 |
+
module.decoder.layers.4.mlp.router.weight
|
| 710 |
+
module.decoder.layers.4.self_attention.k_layernorm.weight
|
| 711 |
module.decoder.layers.4.self_attention.linear_proj.weight
|
| 712 |
Params for bucket 10 (71835904 elements, 71835904 padded size):
|
| 713 |
module.decoder.layers.3.mlp.router.weight
|
|
|
|
| 714 |
module.decoder.layers.3.self_attention.linear_qkv.weight
|
| 715 |
module.decoder.layers.3.self_attention.q_layernorm.weight
|
| 716 |
+
module.decoder.layers.3.pre_mlp_layernorm.weight
|
| 717 |
module.decoder.layers.3.self_attention.k_layernorm.weight
|
| 718 |
module.decoder.layers.3.self_attention.linear_qkv.layer_norm_weight
|
| 719 |
module.decoder.layers.3.self_attention.linear_proj.weight
|
| 720 |
Params for bucket 11 (71835904 elements, 71835904 padded size):
|
|
|
|
|
|
|
| 721 |
module.decoder.layers.2.self_attention.q_layernorm.weight
|
| 722 |
+
module.decoder.layers.2.mlp.router.weight
|
| 723 |
+
module.decoder.layers.2.pre_mlp_layernorm.weight
|
| 724 |
module.decoder.layers.2.self_attention.linear_qkv.weight
|
| 725 |
+
module.decoder.layers.2.self_attention.k_layernorm.weight
|
| 726 |
module.decoder.layers.2.self_attention.linear_qkv.layer_norm_weight
|
| 727 |
module.decoder.layers.2.self_attention.linear_proj.weight
|
|
|
|
| 728 |
Params for bucket 12 (71835904 elements, 71835904 padded size):
|
| 729 |
module.decoder.layers.1.mlp.router.weight
|
| 730 |
+
module.decoder.layers.1.self_attention.k_layernorm.weight
|
| 731 |
module.decoder.layers.1.self_attention.linear_qkv.weight
|
|
|
|
| 732 |
module.decoder.layers.1.pre_mlp_layernorm.weight
|
| 733 |
+
module.decoder.layers.1.self_attention.q_layernorm.weight
|
| 734 |
module.decoder.layers.1.self_attention.linear_qkv.layer_norm_weight
|
| 735 |
module.decoder.layers.1.self_attention.linear_proj.weight
|
| 736 |
Params for bucket 13 (71835904 elements, 71835904 padded size):
|
|
|
|
|
|
|
|
|
|
| 737 |
module.decoder.layers.0.self_attention.linear_qkv.weight
|
|
|
|
| 738 |
module.decoder.layers.0.self_attention.linear_qkv.layer_norm_weight
|
| 739 |
+
module.decoder.layers.0.mlp.router.weight
|
| 740 |
+
module.decoder.layers.0.pre_mlp_layernorm.weight
|
| 741 |
+
module.decoder.layers.0.self_attention.k_layernorm.weight
|
| 742 |
+
module.decoder.layers.0.self_attention.q_layernorm.weight
|
| 743 |
module.decoder.layers.0.self_attention.linear_proj.weight
|
| 744 |
Params for bucket 14 (206045184 elements, 206045184 padded size):
|
| 745 |
module.embedding.word_embeddings.weight
|
|
|
|
| 749 |
module.decoder.layers.11.mlp.experts.linear_fc2.weight2
|
| 750 |
module.decoder.layers.11.mlp.experts.linear_fc2.weight4
|
| 751 |
module.decoder.layers.11.mlp.experts.linear_fc2.weight6
|
|
|
|
| 752 |
module.decoder.layers.11.mlp.experts.linear_fc2.weight1
|
| 753 |
+
module.decoder.layers.11.mlp.experts.linear_fc2.weight3
|
| 754 |
module.decoder.layers.11.mlp.experts.linear_fc2.weight5
|
| 755 |
Params for bucket 2 (44040192 elements, 44040192 padded size):
|
| 756 |
module.decoder.layers.11.mlp.experts.linear_fc2.weight10
|
|
|
|
| 764 |
module.decoder.layers.11.mlp.experts.linear_fc2.weight14
|
| 765 |
module.decoder.layers.11.mlp.experts.linear_fc1.weight1
|
| 766 |
module.decoder.layers.11.mlp.experts.linear_fc1.weight0
|
|
|
|
| 767 |
module.decoder.layers.11.mlp.experts.linear_fc1.weight2
|
| 768 |
+
module.decoder.layers.11.mlp.experts.linear_fc2.weight15
|
| 769 |
Params for bucket 4 (50331648 elements, 50331648 padded size):
|
| 770 |
module.decoder.layers.11.mlp.experts.linear_fc1.weight3
|
|
|
|
| 771 |
module.decoder.layers.11.mlp.experts.linear_fc1.weight4
|
| 772 |
+
module.decoder.layers.11.mlp.experts.linear_fc1.weight6
|
| 773 |
module.decoder.layers.11.mlp.experts.linear_fc1.weight5
|
| 774 |
Params for bucket 5 (50331648 elements, 50331648 padded size):
|
| 775 |
module.decoder.layers.11.mlp.experts.linear_fc1.weight7
|
|
|
|
| 783 |
module.decoder.layers.11.mlp.experts.linear_fc1.weight12
|
| 784 |
Params for bucket 7 (44040192 elements, 44040192 padded size):
|
| 785 |
module.decoder.layers.11.mlp.experts.linear_fc1.weight15
|
|
|
|
|
|
|
|
|
|
| 786 |
module.decoder.layers.10.mlp.experts.linear_fc2.weight2
|
| 787 |
+
module.decoder.layers.10.mlp.experts.linear_fc2.weight0
|
| 788 |
module.decoder.layers.10.mlp.experts.linear_fc2.weight4
|
| 789 |
+
module.decoder.layers.10.mlp.experts.linear_fc2.weight3
|
| 790 |
+
module.decoder.layers.10.mlp.experts.linear_fc2.weight1
|
| 791 |
Params for bucket 8 (44040192 elements, 44040192 padded size):
|
| 792 |
+
module.decoder.layers.10.mlp.experts.linear_fc2.weight8
|
| 793 |
+
module.decoder.layers.10.mlp.experts.linear_fc2.weight6
|
| 794 |
+
module.decoder.layers.10.mlp.experts.linear_fc2.weight10
|
| 795 |
module.decoder.layers.10.mlp.experts.linear_fc2.weight5
|
|
|
|
| 796 |
module.decoder.layers.10.mlp.experts.linear_fc2.weight9
|
| 797 |
+
module.decoder.layers.10.mlp.experts.linear_fc2.weight7
|
| 798 |
module.decoder.layers.10.mlp.experts.linear_fc2.weight11
|
|
|
|
|
|
|
|
|
|
| 799 |
Params for bucket 9 (50331648 elements, 50331648 padded size):
|
|
|
|
|
|
|
|
|
|
| 800 |
module.decoder.layers.10.mlp.experts.linear_fc2.weight12
|
| 801 |
module.decoder.layers.10.mlp.experts.linear_fc2.weight14
|
| 802 |
module.decoder.layers.10.mlp.experts.linear_fc1.weight1
|
| 803 |
+
module.decoder.layers.10.mlp.experts.linear_fc2.weight15
|
| 804 |
+
module.decoder.layers.10.mlp.experts.linear_fc2.weight13
|
| 805 |
+
module.decoder.layers.10.mlp.experts.linear_fc1.weight0
|
| 806 |
Params for bucket 10 (50331648 elements, 50331648 padded size):
|
| 807 |
module.decoder.layers.10.mlp.experts.linear_fc1.weight2
|
| 808 |
+
module.decoder.layers.10.mlp.experts.linear_fc1.weight5
|
| 809 |
module.decoder.layers.10.mlp.experts.linear_fc1.weight3
|
| 810 |
module.decoder.layers.10.mlp.experts.linear_fc1.weight4
|
|
|
|
| 811 |
Params for bucket 11 (50331648 elements, 50331648 padded size):
|
| 812 |
module.decoder.layers.10.mlp.experts.linear_fc1.weight6
|
| 813 |
+
module.decoder.layers.10.mlp.experts.linear_fc1.weight8
|
| 814 |
module.decoder.layers.10.mlp.experts.linear_fc1.weight9
|
| 815 |
module.decoder.layers.10.mlp.experts.linear_fc1.weight7
|
|
|
|
| 816 |
Params for bucket 12 (50331648 elements, 50331648 padded size):
|
| 817 |
module.decoder.layers.10.mlp.experts.linear_fc1.weight10
|
| 818 |
+
module.decoder.layers.10.mlp.experts.linear_fc1.weight11
|
| 819 |
module.decoder.layers.10.mlp.experts.linear_fc1.weight12
|
| 820 |
module.decoder.layers.10.mlp.experts.linear_fc1.weight13
|
|
|
|
| 821 |
Params for bucket 13 (44040192 elements, 44040192 padded size):
|
| 822 |
+
module.decoder.layers.9.mlp.experts.linear_fc2.weight1
|
| 823 |
+
module.decoder.layers.10.mlp.experts.linear_fc1.weight15
|
| 824 |
module.decoder.layers.10.mlp.experts.linear_fc1.weight14
|
| 825 |
module.decoder.layers.9.mlp.experts.linear_fc2.weight2
|
| 826 |
module.decoder.layers.9.mlp.experts.linear_fc2.weight0
|
|
|
|
|
|
|
| 827 |
Params for bucket 14 (44040192 elements, 44040192 padded size):
|
|
|
|
|
|
|
|
|
|
| 828 |
module.decoder.layers.9.mlp.experts.linear_fc2.weight3
|
|
|
|
| 829 |
module.decoder.layers.9.mlp.experts.linear_fc2.weight5
|
| 830 |
+
module.decoder.layers.9.mlp.experts.linear_fc2.weight7
|
| 831 |
module.decoder.layers.9.mlp.experts.linear_fc2.weight9
|
| 832 |
+
module.decoder.layers.9.mlp.experts.linear_fc2.weight6
|
| 833 |
+
module.decoder.layers.9.mlp.experts.linear_fc2.weight4
|
| 834 |
+
module.decoder.layers.9.mlp.experts.linear_fc2.weight8
|
| 835 |
Params for bucket 15 (50331648 elements, 50331648 padded size):
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 836 |
module.decoder.layers.9.mlp.experts.linear_fc2.weight11
|
| 837 |
+
module.decoder.layers.9.mlp.experts.linear_fc2.weight13
|
| 838 |
module.decoder.layers.9.mlp.experts.linear_fc2.weight15
|
| 839 |
+
module.decoder.layers.9.mlp.experts.linear_fc1.weight0
|
| 840 |
+
module.decoder.layers.9.mlp.experts.linear_fc2.weight10
|
| 841 |
+
module.decoder.layers.9.mlp.experts.linear_fc2.weight14
|
| 842 |
+
module.decoder.layers.9.mlp.experts.linear_fc2.weight12
|
| 843 |
Params for bucket 16 (50331648 elements, 50331648 padded size):
|
| 844 |
module.decoder.layers.9.mlp.experts.linear_fc1.weight1
|
| 845 |
module.decoder.layers.9.mlp.experts.linear_fc1.weight4
|
|
|
|
| 870 |
module.decoder.layers.8.mlp.experts.linear_fc2.weight7
|
| 871 |
Params for bucket 21 (44040192 elements, 44040192 padded size):
|
| 872 |
module.decoder.layers.8.mlp.experts.linear_fc2.weight8
|
|
|
|
| 873 |
module.decoder.layers.8.mlp.experts.linear_fc2.weight12
|
| 874 |
+
module.decoder.layers.8.mlp.experts.linear_fc2.weight10
|
| 875 |
module.decoder.layers.8.mlp.experts.linear_fc2.weight14
|
| 876 |
module.decoder.layers.8.mlp.experts.linear_fc2.weight9
|
| 877 |
module.decoder.layers.8.mlp.experts.linear_fc2.weight11
|
| 878 |
module.decoder.layers.8.mlp.experts.linear_fc2.weight13
|
| 879 |
Params for bucket 22 (44040192 elements, 44040192 padded size):
|
| 880 |
module.decoder.layers.8.mlp.experts.linear_fc2.weight15
|
|
|
|
| 881 |
module.decoder.layers.8.mlp.experts.linear_fc1.weight0
|
| 882 |
module.decoder.layers.8.mlp.experts.linear_fc1.weight2
|
| 883 |
+
module.decoder.layers.8.mlp.experts.linear_fc1.weight1
|
| 884 |
Params for bucket 23 (50331648 elements, 50331648 padded size):
|
| 885 |
module.decoder.layers.8.mlp.experts.linear_fc1.weight3
|
| 886 |
+
module.decoder.layers.8.mlp.experts.linear_fc1.weight6
|
| 887 |
module.decoder.layers.8.mlp.experts.linear_fc1.weight4
|
| 888 |
module.decoder.layers.8.mlp.experts.linear_fc1.weight5
|
|
|
|
| 889 |
Params for bucket 24 (50331648 elements, 50331648 padded size):
|
| 890 |
module.decoder.layers.8.mlp.experts.linear_fc1.weight7
|
| 891 |
+
module.decoder.layers.8.mlp.experts.linear_fc1.weight9
|
| 892 |
module.decoder.layers.8.mlp.experts.linear_fc1.weight10
|
| 893 |
module.decoder.layers.8.mlp.experts.linear_fc1.weight8
|
|
|
|
| 894 |
Params for bucket 25 (50331648 elements, 50331648 padded size):
|
| 895 |
module.decoder.layers.8.mlp.experts.linear_fc1.weight11
|
| 896 |
+
module.decoder.layers.8.mlp.experts.linear_fc1.weight12
|
| 897 |
module.decoder.layers.8.mlp.experts.linear_fc1.weight13
|
| 898 |
module.decoder.layers.8.mlp.experts.linear_fc1.weight14
|
|
|
|
| 899 |
Params for bucket 26 (44040192 elements, 44040192 padded size):
|
|
|
|
|
|
|
|
|
|
|
|
|
| 900 |
module.decoder.layers.7.mlp.experts.linear_fc2.weight2
|
| 901 |
+
module.decoder.layers.7.mlp.experts.linear_fc2.weight0
|
| 902 |
module.decoder.layers.7.mlp.experts.linear_fc2.weight4
|
| 903 |
+
module.decoder.layers.8.mlp.experts.linear_fc1.weight15
|
| 904 |
+
module.decoder.layers.7.mlp.experts.linear_fc2.weight1
|
| 905 |
+
module.decoder.layers.7.mlp.experts.linear_fc2.weight3
|
| 906 |
Params for bucket 27 (44040192 elements, 44040192 padded size):
|
| 907 |
+
module.decoder.layers.7.mlp.experts.linear_fc2.weight8
|
| 908 |
+
module.decoder.layers.7.mlp.experts.linear_fc2.weight6
|
| 909 |
+
module.decoder.layers.7.mlp.experts.linear_fc2.weight10
|
| 910 |
module.decoder.layers.7.mlp.experts.linear_fc2.weight5
|
| 911 |
module.decoder.layers.7.mlp.experts.linear_fc2.weight9
|
| 912 |
module.decoder.layers.7.mlp.experts.linear_fc2.weight7
|
| 913 |
module.decoder.layers.7.mlp.experts.linear_fc2.weight11
|
|
|
|
|
|
|
|
|
|
| 914 |
Params for bucket 28 (50331648 elements, 50331648 padded size):
|
|
|
|
|
|
|
|
|
|
| 915 |
module.decoder.layers.7.mlp.experts.linear_fc2.weight12
|
| 916 |
module.decoder.layers.7.mlp.experts.linear_fc2.weight14
|
| 917 |
module.decoder.layers.7.mlp.experts.linear_fc1.weight1
|
| 918 |
+
module.decoder.layers.7.mlp.experts.linear_fc2.weight15
|
| 919 |
+
module.decoder.layers.7.mlp.experts.linear_fc1.weight0
|
| 920 |
+
module.decoder.layers.7.mlp.experts.linear_fc2.weight13
|
| 921 |
Params for bucket 29 (50331648 elements, 50331648 padded size):
|
| 922 |
module.decoder.layers.7.mlp.experts.linear_fc1.weight2
|
| 923 |
+
module.decoder.layers.7.mlp.experts.linear_fc1.weight5
|
| 924 |
module.decoder.layers.7.mlp.experts.linear_fc1.weight3
|
| 925 |
module.decoder.layers.7.mlp.experts.linear_fc1.weight4
|
|
|
|
| 926 |
Params for bucket 30 (50331648 elements, 50331648 padded size):
|
| 927 |
module.decoder.layers.7.mlp.experts.linear_fc1.weight6
|
| 928 |
+
module.decoder.layers.7.mlp.experts.linear_fc1.weight8
|
| 929 |
module.decoder.layers.7.mlp.experts.linear_fc1.weight7
|
| 930 |
module.decoder.layers.7.mlp.experts.linear_fc1.weight9
|
|
|
|
| 931 |
Params for bucket 31 (50331648 elements, 50331648 padded size):
|
| 932 |
module.decoder.layers.7.mlp.experts.linear_fc1.weight10
|
|
|
|
| 933 |
module.decoder.layers.7.mlp.experts.linear_fc1.weight12
|
| 934 |
module.decoder.layers.7.mlp.experts.linear_fc1.weight11
|
| 935 |
+
module.decoder.layers.7.mlp.experts.linear_fc1.weight13
|
| 936 |
Params for bucket 32 (44040192 elements, 44040192 padded size):
|
| 937 |
+
module.decoder.layers.7.mlp.experts.linear_fc1.weight15
|
|
|
|
| 938 |
module.decoder.layers.6.mlp.experts.linear_fc2.weight0
|
| 939 |
+
module.decoder.layers.6.mlp.experts.linear_fc2.weight2
|
| 940 |
+
module.decoder.layers.7.mlp.experts.linear_fc1.weight14
|
| 941 |
module.decoder.layers.6.mlp.experts.linear_fc2.weight1
|
|
|
|
| 942 |
Params for bucket 33 (44040192 elements, 44040192 padded size):
|
| 943 |
module.decoder.layers.6.mlp.experts.linear_fc2.weight6
|
| 944 |
module.decoder.layers.6.mlp.experts.linear_fc2.weight4
|
| 945 |
module.decoder.layers.6.mlp.experts.linear_fc2.weight8
|
| 946 |
module.decoder.layers.6.mlp.experts.linear_fc2.weight3
|
|
|
|
| 947 |
module.decoder.layers.6.mlp.experts.linear_fc2.weight7
|
| 948 |
+
module.decoder.layers.6.mlp.experts.linear_fc2.weight5
|
| 949 |
module.decoder.layers.6.mlp.experts.linear_fc2.weight9
|
| 950 |
Params for bucket 34 (50331648 elements, 50331648 padded size):
|
| 951 |
module.decoder.layers.6.mlp.experts.linear_fc2.weight10
|
| 952 |
module.decoder.layers.6.mlp.experts.linear_fc2.weight14
|
| 953 |
module.decoder.layers.6.mlp.experts.linear_fc2.weight12
|
|
|
|
| 954 |
module.decoder.layers.6.mlp.experts.linear_fc2.weight13
|
| 955 |
+
module.decoder.layers.6.mlp.experts.linear_fc2.weight11
|
| 956 |
module.decoder.layers.6.mlp.experts.linear_fc2.weight15
|
| 957 |
module.decoder.layers.6.mlp.experts.linear_fc1.weight0
|
| 958 |
Params for bucket 35 (50331648 elements, 50331648 padded size):
|
| 959 |
module.decoder.layers.6.mlp.experts.linear_fc1.weight1
|
|
|
|
| 960 |
module.decoder.layers.6.mlp.experts.linear_fc1.weight3
|
| 961 |
module.decoder.layers.6.mlp.experts.linear_fc1.weight4
|
| 962 |
+
module.decoder.layers.6.mlp.experts.linear_fc1.weight2
|
| 963 |
Params for bucket 36 (50331648 elements, 50331648 padded size):
|
| 964 |
module.decoder.layers.6.mlp.experts.linear_fc1.weight5
|
|
|
|
| 965 |
module.decoder.layers.6.mlp.experts.linear_fc1.weight6
|
| 966 |
module.decoder.layers.6.mlp.experts.linear_fc1.weight7
|
| 967 |
+
module.decoder.layers.6.mlp.experts.linear_fc1.weight8
|
| 968 |
Params for bucket 37 (50331648 elements, 50331648 padded size):
|
| 969 |
module.decoder.layers.6.mlp.experts.linear_fc1.weight9
|
|
|
|
| 970 |
module.decoder.layers.6.mlp.experts.linear_fc1.weight12
|
| 971 |
module.decoder.layers.6.mlp.experts.linear_fc1.weight10
|
| 972 |
+
module.decoder.layers.6.mlp.experts.linear_fc1.weight11
|
| 973 |
Params for bucket 38 (44040192 elements, 44040192 padded size):
|
| 974 |
module.decoder.layers.6.mlp.experts.linear_fc1.weight13
|
| 975 |
+
module.decoder.layers.5.mlp.experts.linear_fc2.weight0
|
| 976 |
module.decoder.layers.6.mlp.experts.linear_fc1.weight15
|
| 977 |
module.decoder.layers.6.mlp.experts.linear_fc1.weight14
|
|
|
|
| 978 |
Params for bucket 39 (44040192 elements, 44040192 padded size):
|
| 979 |
module.decoder.layers.5.mlp.experts.linear_fc2.weight1
|
|
|
|
| 980 |
module.decoder.layers.5.mlp.experts.linear_fc2.weight3
|
| 981 |
+
module.decoder.layers.5.mlp.experts.linear_fc2.weight5
|
| 982 |
module.decoder.layers.5.mlp.experts.linear_fc2.weight7
|
| 983 |
module.decoder.layers.5.mlp.experts.linear_fc2.weight2
|
| 984 |
module.decoder.layers.5.mlp.experts.linear_fc2.weight4
|
| 985 |
module.decoder.layers.5.mlp.experts.linear_fc2.weight6
|
| 986 |
Params for bucket 40 (44040192 elements, 44040192 padded size):
|
|
|
|
| 987 |
module.decoder.layers.5.mlp.experts.linear_fc2.weight9
|
| 988 |
+
module.decoder.layers.5.mlp.experts.linear_fc2.weight11
|
| 989 |
module.decoder.layers.5.mlp.experts.linear_fc2.weight13
|
| 990 |
module.decoder.layers.5.mlp.experts.linear_fc2.weight8
|
| 991 |
module.decoder.layers.5.mlp.experts.linear_fc2.weight12
|
|
|
|
| 993 |
module.decoder.layers.5.mlp.experts.linear_fc2.weight14
|
| 994 |
Params for bucket 41 (44040192 elements, 44040192 padded size):
|
| 995 |
module.decoder.layers.5.mlp.experts.linear_fc2.weight15
|
|
|
|
| 996 |
module.decoder.layers.5.mlp.experts.linear_fc1.weight1
|
| 997 |
module.decoder.layers.5.mlp.experts.linear_fc1.weight2
|
| 998 |
+
module.decoder.layers.5.mlp.experts.linear_fc1.weight0
|
| 999 |
Params for bucket 42 (50331648 elements, 50331648 padded size):
|
| 1000 |
module.decoder.layers.5.mlp.experts.linear_fc1.weight3
|
| 1001 |
+
module.decoder.layers.5.mlp.experts.linear_fc1.weight5
|
| 1002 |
module.decoder.layers.5.mlp.experts.linear_fc1.weight6
|
| 1003 |
module.decoder.layers.5.mlp.experts.linear_fc1.weight4
|
|
|
|
| 1004 |
Params for bucket 43 (50331648 elements, 50331648 padded size):
|
| 1005 |
module.decoder.layers.5.mlp.experts.linear_fc1.weight7
|
|
|
|
| 1006 |
module.decoder.layers.5.mlp.experts.linear_fc1.weight8
|
| 1007 |
module.decoder.layers.5.mlp.experts.linear_fc1.weight9
|
| 1008 |
+
module.decoder.layers.5.mlp.experts.linear_fc1.weight10
|
| 1009 |
Params for bucket 44 (50331648 elements, 50331648 padded size):
|
| 1010 |
module.decoder.layers.5.mlp.experts.linear_fc1.weight11
|
|
|
|
| 1011 |
module.decoder.layers.5.mlp.experts.linear_fc1.weight14
|
| 1012 |
module.decoder.layers.5.mlp.experts.linear_fc1.weight12
|
| 1013 |
+
module.decoder.layers.5.mlp.experts.linear_fc1.weight13
|
| 1014 |
Params for bucket 45 (44040192 elements, 44040192 padded size):
|
| 1015 |
module.decoder.layers.5.mlp.experts.linear_fc1.weight15
|
| 1016 |
+
module.decoder.layers.4.mlp.experts.linear_fc2.weight3
|
| 1017 |
+
module.decoder.layers.4.mlp.experts.linear_fc2.weight1
|
| 1018 |
module.decoder.layers.4.mlp.experts.linear_fc2.weight0
|
| 1019 |
module.decoder.layers.4.mlp.experts.linear_fc2.weight2
|
| 1020 |
module.decoder.layers.4.mlp.experts.linear_fc2.weight4
|
|
|
|
|
|
|
| 1021 |
Params for bucket 46 (44040192 elements, 44040192 padded size):
|
|
|
|
|
|
|
|
|
|
| 1022 |
module.decoder.layers.4.mlp.experts.linear_fc2.weight5
|
|
|
|
| 1023 |
module.decoder.layers.4.mlp.experts.linear_fc2.weight9
|
| 1024 |
+
module.decoder.layers.4.mlp.experts.linear_fc2.weight7
|
| 1025 |
module.decoder.layers.4.mlp.experts.linear_fc2.weight11
|
| 1026 |
+
module.decoder.layers.4.mlp.experts.linear_fc2.weight6
|
| 1027 |
+
module.decoder.layers.4.mlp.experts.linear_fc2.weight8
|
| 1028 |
+
module.decoder.layers.4.mlp.experts.linear_fc2.weight10
|
| 1029 |
Params for bucket 47 (50331648 elements, 50331648 padded size):
|
| 1030 |
+
module.decoder.layers.4.mlp.experts.linear_fc2.weight13
|
|
|
|
| 1031 |
module.decoder.layers.4.mlp.experts.linear_fc1.weight0
|
| 1032 |
module.decoder.layers.4.mlp.experts.linear_fc2.weight15
|
| 1033 |
+
module.decoder.layers.4.mlp.experts.linear_fc2.weight12
|
| 1034 |
+
module.decoder.layers.4.mlp.experts.linear_fc2.weight14
|
| 1035 |
module.decoder.layers.4.mlp.experts.linear_fc1.weight1
|
| 1036 |
Params for bucket 48 (50331648 elements, 50331648 padded size):
|
| 1037 |
module.decoder.layers.4.mlp.experts.linear_fc1.weight2
|
|
|
|
| 1091 |
module.decoder.layers.2.mlp.experts.linear_fc2.weight0
|
| 1092 |
module.decoder.layers.3.mlp.experts.linear_fc1.weight14
|
| 1093 |
Params for bucket 58 (44040192 elements, 44040192 padded size):
|
|
|
|
|
|
|
|
|
|
| 1094 |
module.decoder.layers.2.mlp.experts.linear_fc2.weight1
|
|
|
|
| 1095 |
module.decoder.layers.2.mlp.experts.linear_fc2.weight5
|
| 1096 |
+
module.decoder.layers.2.mlp.experts.linear_fc2.weight3
|
| 1097 |
module.decoder.layers.2.mlp.experts.linear_fc2.weight7
|
| 1098 |
+
module.decoder.layers.2.mlp.experts.linear_fc2.weight4
|
| 1099 |
+
module.decoder.layers.2.mlp.experts.linear_fc2.weight2
|
| 1100 |
+
module.decoder.layers.2.mlp.experts.linear_fc2.weight6
|
| 1101 |
Params for bucket 59 (44040192 elements, 44040192 padded size):
|
| 1102 |
+
module.decoder.layers.2.mlp.experts.linear_fc2.weight11
|
| 1103 |
+
module.decoder.layers.2.mlp.experts.linear_fc2.weight9
|
| 1104 |
+
module.decoder.layers.2.mlp.experts.linear_fc2.weight13
|
| 1105 |
module.decoder.layers.2.mlp.experts.linear_fc2.weight8
|
| 1106 |
module.decoder.layers.2.mlp.experts.linear_fc2.weight10
|
| 1107 |
module.decoder.layers.2.mlp.experts.linear_fc2.weight12
|
| 1108 |
module.decoder.layers.2.mlp.experts.linear_fc2.weight14
|
|
|
|
|
|
|
|
|
|
| 1109 |
Params for bucket 60 (44040192 elements, 44040192 padded size):
|
| 1110 |
module.decoder.layers.2.mlp.experts.linear_fc2.weight15
|
| 1111 |
module.decoder.layers.2.mlp.experts.linear_fc1.weight1
|
|
|
|
| 1128 |
module.decoder.layers.2.mlp.experts.linear_fc1.weight12
|
| 1129 |
Params for bucket 64 (44040192 elements, 44040192 padded size):
|
| 1130 |
module.decoder.layers.2.mlp.experts.linear_fc1.weight15
|
|
|
|
| 1131 |
module.decoder.layers.1.mlp.experts.linear_fc2.weight0
|
| 1132 |
+
module.decoder.layers.1.mlp.experts.linear_fc2.weight2
|
| 1133 |
module.decoder.layers.1.mlp.experts.linear_fc2.weight4
|
| 1134 |
module.decoder.layers.1.mlp.experts.linear_fc2.weight3
|
| 1135 |
module.decoder.layers.1.mlp.experts.linear_fc2.weight1
|
| 1136 |
Params for bucket 65 (44040192 elements, 44040192 padded size):
|
|
|
|
| 1137 |
module.decoder.layers.1.mlp.experts.linear_fc2.weight6
|
| 1138 |
+
module.decoder.layers.1.mlp.experts.linear_fc2.weight8
|
| 1139 |
module.decoder.layers.1.mlp.experts.linear_fc2.weight10
|
| 1140 |
module.decoder.layers.1.mlp.experts.linear_fc2.weight5
|
| 1141 |
module.decoder.layers.1.mlp.experts.linear_fc2.weight9
|
|
|
|
| 1145 |
module.decoder.layers.1.mlp.experts.linear_fc2.weight12
|
| 1146 |
module.decoder.layers.1.mlp.experts.linear_fc2.weight14
|
| 1147 |
module.decoder.layers.1.mlp.experts.linear_fc1.weight1
|
|
|
|
|
|
|
| 1148 |
module.decoder.layers.1.mlp.experts.linear_fc1.weight0
|
| 1149 |
+
module.decoder.layers.1.mlp.experts.linear_fc2.weight13
|
| 1150 |
+
module.decoder.layers.1.mlp.experts.linear_fc2.weight15
|
| 1151 |
Params for bucket 67 (50331648 elements, 50331648 padded size):
|
| 1152 |
module.decoder.layers.1.mlp.experts.linear_fc1.weight2
|
| 1153 |
+
module.decoder.layers.1.mlp.experts.linear_fc1.weight3
|
| 1154 |
module.decoder.layers.1.mlp.experts.linear_fc1.weight4
|
| 1155 |
module.decoder.layers.1.mlp.experts.linear_fc1.weight5
|
|
|
|
| 1156 |
Params for bucket 68 (50331648 elements, 50331648 padded size):
|
| 1157 |
module.decoder.layers.1.mlp.experts.linear_fc1.weight6
|
| 1158 |
module.decoder.layers.1.mlp.experts.linear_fc1.weight7
|
|
|
|
| 1160 |
module.decoder.layers.1.mlp.experts.linear_fc1.weight9
|
| 1161 |
Params for bucket 69 (50331648 elements, 50331648 padded size):
|
| 1162 |
module.decoder.layers.1.mlp.experts.linear_fc1.weight10
|
| 1163 |
+
module.decoder.layers.1.mlp.experts.linear_fc1.weight13
|
| 1164 |
module.decoder.layers.1.mlp.experts.linear_fc1.weight11
|
| 1165 |
module.decoder.layers.1.mlp.experts.linear_fc1.weight12
|
|
|
|
| 1166 |
Params for bucket 70 (44040192 elements, 44040192 padded size):
|
| 1167 |
+
module.decoder.layers.0.mlp.experts.linear_fc2.weight1
|
| 1168 |
module.decoder.layers.1.mlp.experts.linear_fc1.weight15
|
|
|
|
|
|
|
| 1169 |
module.decoder.layers.1.mlp.experts.linear_fc1.weight14
|
| 1170 |
+
module.decoder.layers.0.mlp.experts.linear_fc2.weight0
|
| 1171 |
+
module.decoder.layers.0.mlp.experts.linear_fc2.weight2
|
| 1172 |
Params for bucket 71 (44040192 elements, 44040192 padded size):
|
|
|
|
|
|
|
|
|
|
| 1173 |
module.decoder.layers.0.mlp.experts.linear_fc2.weight3
|
|
|
|
| 1174 |
module.decoder.layers.0.mlp.experts.linear_fc2.weight7
|
| 1175 |
+
module.decoder.layers.0.mlp.experts.linear_fc2.weight5
|
| 1176 |
module.decoder.layers.0.mlp.experts.linear_fc2.weight9
|
| 1177 |
+
module.decoder.layers.0.mlp.experts.linear_fc2.weight6
|
| 1178 |
+
module.decoder.layers.0.mlp.experts.linear_fc2.weight4
|
| 1179 |
+
module.decoder.layers.0.mlp.experts.linear_fc2.weight8
|
| 1180 |
Params for bucket 72 (50331648 elements, 50331648 padded size):
|
| 1181 |
+
module.decoder.layers.0.mlp.experts.linear_fc2.weight13
|
| 1182 |
+
module.decoder.layers.0.mlp.experts.linear_fc2.weight11
|
| 1183 |
+
module.decoder.layers.0.mlp.experts.linear_fc2.weight15
|
| 1184 |
+
module.decoder.layers.0.mlp.experts.linear_fc1.weight0
|
| 1185 |
module.decoder.layers.0.mlp.experts.linear_fc2.weight10
|
| 1186 |
module.decoder.layers.0.mlp.experts.linear_fc2.weight14
|
| 1187 |
module.decoder.layers.0.mlp.experts.linear_fc2.weight12
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1188 |
Params for bucket 73 (50331648 elements, 50331648 padded size):
|
| 1189 |
module.decoder.layers.0.mlp.experts.linear_fc1.weight1
|
|
|
|
| 1190 |
module.decoder.layers.0.mlp.experts.linear_fc1.weight3
|
| 1191 |
module.decoder.layers.0.mlp.experts.linear_fc1.weight2
|
| 1192 |
+
module.decoder.layers.0.mlp.experts.linear_fc1.weight4
|
| 1193 |
Params for bucket 74 (50331648 elements, 50331648 padded size):
|
| 1194 |
module.decoder.layers.0.mlp.experts.linear_fc1.weight5
|
| 1195 |
module.decoder.layers.0.mlp.experts.linear_fc1.weight8
|
|
|
|
| 1196 |
module.decoder.layers.0.mlp.experts.linear_fc1.weight6
|
| 1197 |
+
module.decoder.layers.0.mlp.experts.linear_fc1.weight7
|
| 1198 |
Params for bucket 75 (50331648 elements, 50331648 padded size):
|
| 1199 |
module.decoder.layers.0.mlp.experts.linear_fc1.weight9
|
|
|
|
|
|
|
| 1200 |
module.decoder.layers.0.mlp.experts.linear_fc1.weight12
|
| 1201 |
+
module.decoder.layers.0.mlp.experts.linear_fc1.weight10
|
| 1202 |
+
module.decoder.layers.0.mlp.experts.linear_fc1.weight11
|
| 1203 |
Params for bucket 76 (37748736 elements, 37748736 padded size):
|
| 1204 |
module.decoder.layers.0.mlp.experts.linear_fc1.weight13
|
| 1205 |
module.decoder.layers.0.mlp.experts.linear_fc1.weight14
|
| 1206 |
module.decoder.layers.0.mlp.experts.linear_fc1.weight15
|
| 1207 |
+
INFO:megatron.core.optimizer:Setting up optimizer with config OptimizerConfig(optimizer='adam', lr=0.0001, min_lr=1e-05, decoupled_lr=None, decoupled_min_lr=None, weight_decay=0.1, fp8_recipe='mxfp8', fp16=False, bf16=True, reuse_grad_buf_for_mxfp8_param_ag=False, params_dtype=torch.bfloat16, use_precision_aware_optimizer=False, store_param_remainders=True, main_grads_dtype=torch.float32, main_params_dtype=torch.float32, exp_avg_dtype=torch.float32, exp_avg_sq_dtype=torch.float32, loss_scale=None, initial_loss_scale=4294967296, min_loss_scale=1.0, loss_scale_window=1000, hysteresis=2, adam_beta1=0.9, adam_beta2=0.999, adam_eps=1e-08, sgd_momentum=0.9, use_distributed_optimizer=True, overlap_param_gather_with_optimizer_step=False, optimizer_cpu_offload=False, optimizer_offload_fraction=1.0, use_torch_optimizer_for_cpu_offload=False, overlap_cpu_optimizer_d2h_h2d=False, pin_cpu_grads=True, pin_cpu_params=True, clip_grad=1.0, log_num_zeros_in_grad=False, barrier_with_L1_time=True, timers=<megatron.core.timers.Timers object at 0x790345189280>, config_logger_dir='')
|
| 1208 |
INFO:megatron.core.optimizer_param_scheduler:> learning rate decay style: cosine
|
| 1209 |
+
[after model, optimizer, and learning rate scheduler are built] datetime: 2026-08-30 01:31:46
|
| 1210 |
> building train, validation, and test datasets ...
|
| 1211 |
> datasets target sizes (minimum size):
|
| 1212 |
train: 96
|
|
|
|
| 1214 |
test: 0
|
| 1215 |
INFO:megatron.core.datasets.blended_megatron_dataset_config:Let split_matrix = [(0, 1.0), None, None]
|
| 1216 |
> building train, validation, and test datasets for GPT ...
|
| 1217 |
+
INFO:megatron.core.datasets.blended_megatron_dataset_builder:Building GPTDataset splits with sizes=(96, 0, 0) and config=GPTDatasetConfig(random_seed=1234, sequence_length=8192, blend=(['/workspace/data/wikitext/wikitext103_gpt2_text_document'], None), blend_per_split=None, split='100,0,0', split_matrix=[(0, 1.0), None, None], num_dataset_builder_threads=1, path_to_cache=None, mmap_bin_files=True, mock=False, tokenizer=<megatron.training.tokenizer.tokenizer._GPT2BPETokenizer object at 0x79009ea156a0>, mid_level_dataset_surplus=0.005, reset_position_ids=False, reset_attention_mask=False, eod_mask_loss=False, create_attention_mask=True, drop_last_partial_validation_sequence=True, add_extra_token_to_sequence=True, object_storage_cache_path=None)
|
| 1218 |
INFO:megatron.core.datasets.indexed_dataset:Load the _IndexReader from /workspace/data/wikitext/wikitext103_gpt2_text_document.idx
|
| 1219 |
INFO:megatron.core.datasets.indexed_dataset: Extract the sequence lengths
|
| 1220 |
INFO:megatron.core.datasets.indexed_dataset: Extract the sequence pointers
|
|
|
|
| 1227 |
INFO:megatron.core.datasets.gpt_dataset: Load the shuffle index from eadc3756447e39d609e8aa728a91f451-GPTDataset-train-shuffle_index.npy
|
| 1228 |
INFO:megatron.core.datasets.gpt_dataset:> total number of samples: 14536
|
| 1229 |
> finished creating GPT datasets ...
|
| 1230 |
+
[after dataloaders are built] datetime: 2026-08-30 01:31:46
|
| 1231 |
done with setup ...
|
|
|
|
|
|
|
|
|
|
| 1232 |
training ...
|
| 1233 |
Overwriting rerun_state_machine.current_iteration from -1 to 0...
|
| 1234 |
+
(min, max) time across ranks (ms):
|
| 1235 |
+
model-and-optimizer-setup ......................: (3305.74, 3311.41)
|
| 1236 |
+
train/valid/test-data-iterators-setup ..........: (66.88, 70.10)
|
| 1237 |
+
[before the start of training step] datetime: 2026-08-30 01:31:46
|
| 1238 |
INFO:moonep.api:MoonEP comm buffer sizes:
|
| 1239 |
config : R=8, S=8192, H=4096, E=128, K=8, NvS=69600, NvS_padded=69632
|
| 1240 |
sm_count : num_sms=32, num_sms_dedup=148
|
|
|
|
| 1246 |
[MoonEP selfcheck 2/4] dropped groups empty OK | covered rows [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 128, 129, 130, 131, 132, 133, 134, 135, 136, 137, 138, 139, 140, 141, 142, 143] sizes [9344, 2688, 128, 24192, 128, 0, 128, 256, 128, 768, 10496, 128, 128, 2432, 128, 1536, 14208, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0] | duplicated experts [[69, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1], [88, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1], [55, 52, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1], [-1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1], [63, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1], [124, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1], [10, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1], [58, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1]]
|
| 1247 |
[MoonEP selfcheck 3/4] dropped groups empty OK | covered rows [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 128, 129, 130, 131, 132, 133, 134, 135, 136, 137, 138, 139, 140, 141, 142, 143] sizes [128, 128, 128, 0, 128, 44416, 128, 128, 128, 6912, 128, 256, 128, 128, 128, 128, 13824, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0] | duplicated experts [[78, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1], [111, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1], [62, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1], [-1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1], [57, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1], [106, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1], [121, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1], [57, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1]]
|
| 1248 |
[MoonEP selfcheck 4/4] dropped groups empty OK | covered rows [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 128, 129, 130, 131, 132, 133, 134, 135, 136, 137, 138, 139, 140, 141, 142, 143] sizes [2304, 128, 128, 512, 128, 12416, 128, 1152, 1280, 16768, 640, 29056, 128, 128, 128, 1664, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0] | duplicated experts [[-1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1], [118, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1], [11, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1], [118, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1], [39, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1], [11, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1], [73, 79, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1], [86, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1]]
|
| 1249 |
+
[2026-08-30 01:32:08] iteration 1/ 12 | consumed samples: 8 | elapsed time per iteration (ms): 21390.2 | throughput per GPU (TFLOP/s/GPU): 8.5 | learning rate: 2.000000E-05 | global batch size: 8 | lm loss: 1.115993E+01 | loss scale: 1.0 | grad norm: 44.807 | number of skipped iterations: 0 | number of nan iterations: 0 |
|
| 1250 |
Number of parameters in transformer block in billions: 29.85
|
| 1251 |
Number of parameters in embedding layers in billions: 0.41
|
| 1252 |
Total number of parameters in billions: 30.26
|
| 1253 |
Number of parameters in most loaded shard in billions: 30.2590
|
| 1254 |
Theoretical memory footprints: weight and optimizer=216428.96 MB
|
|
|
|
| 1255 |
[Rank 0] (after 1 iterations) memory (MB) | allocated: 67239.548828125 | max allocated: 67239.54931640625 | reserved: 68548.0 | max reserved: 68548.0
|
| 1256 |
+
[2026-08-30 01:32:08] iteration 2/ 12 | consumed samples: 16 | elapsed time per iteration (ms): 361.6 | throughput per GPU (TFLOP/s/GPU): 500.1 | learning rate: 4.000000E-05 | global batch size: 8 | lm loss: 1.117600E+01 | loss scale: 1.0 | grad norm: 44.337 | number of skipped iterations: 0 | number of nan iterations: 0 |
|
| 1257 |
+
[2026-08-30 01:32:08] iteration 3/ 12 | consumed samples: 24 | elapsed time per iteration (ms): 332.0 | throughput per GPU (TFLOP/s/GPU): 544.7 | learning rate: 6.000000E-05 | global batch size: 8 | lm loss: 9.685843E+00 | loss scale: 1.0 | grad norm: 62.891 | number of skipped iterations: 0 | number of nan iterations: 0 |
|
| 1258 |
Capture CUDA graph for training!!!
|
| 1259 |
CUDA graph capture done!!!
|
| 1260 |
+
[2026-08-30 01:32:09] iteration 4/ 12 | consumed samples: 32 | elapsed time per iteration (ms): 945.4 | throughput per GPU (TFLOP/s/GPU): 191.3 | learning rate: 8.000000E-05 | global batch size: 8 | lm loss: 1.338080E+01 | loss scale: 1.0 | grad norm: 20.297 | number of skipped iterations: 0 | number of nan iterations: 0 |
|
| 1261 |
+
[2026-08-30 01:32:10] iteration 5/ 12 | consumed samples: 40 | elapsed time per iteration (ms): 319.2 | throughput per GPU (TFLOP/s/GPU): 566.6 | learning rate: 1.000000E-04 | global batch size: 8 | lm loss: 1.049032E+01 | loss scale: 1.0 | grad norm: 95.133 | number of skipped iterations: 0 | number of nan iterations: 0 |
|
| 1262 |
+
[2026-08-30 01:32:10] iteration 6/ 12 | consumed samples: 48 | elapsed time per iteration (ms): 317.9 | throughput per GPU (TFLOP/s/GPU): 568.7 | learning rate: 7.750000E-05 | global batch size: 8 | lm loss: 1.049771E+01 | loss scale: 1.0 | grad norm: 55.144 | number of skipped iterations: 0 | number of nan iterations: 0 |
|
| 1263 |
+
[2026-08-30 01:32:10] iteration 7/ 12 | consumed samples: 56 | elapsed time per iteration (ms): 316.7 | throughput per GPU (TFLOP/s/GPU): 571.0 | learning rate: 3.250000E-05 | global batch size: 8 | lm loss: 1.027093E+01 | loss scale: 1.0 | grad norm: 6.340 | number of skipped iterations: 0 | number of nan iterations: 0 |
|
| 1264 |
+
[2026-08-30 01:32:11] iteration 8/ 12 | consumed samples: 64 | elapsed time per iteration (ms): 317.5 | throughput per GPU (TFLOP/s/GPU): 569.6 | learning rate: 1.000000E-05 | global batch size: 8 | lm loss: 9.810117E+00 | loss scale: 1.0 | grad norm: 5.519 | number of skipped iterations: 0 | number of nan iterations: 0 |
|
| 1265 |
+
[2026-08-30 01:32:11] iteration 9/ 12 | consumed samples: 72 | elapsed time per iteration (ms): 316.4 | throughput per GPU (TFLOP/s/GPU): 571.5 | learning rate: 1.000000E-05 | global batch size: 8 | lm loss: 9.549583E+00 | loss scale: 1.0 | grad norm: 5.211 | number of skipped iterations: 0 | number of nan iterations: 0 |
|
| 1266 |
+
[2026-08-30 01:32:11] iteration 10/ 12 | consumed samples: 80 | elapsed time per iteration (ms): 318.1 | throughput per GPU (TFLOP/s/GPU): 568.4 | learning rate: 1.000000E-05 | global batch size: 8 | lm loss: 9.450918E+00 | loss scale: 1.0 | grad norm: 4.911 | number of skipped iterations: 0 | number of nan iterations: 0 |
|
| 1267 |
+
[2026-08-30 01:32:12] iteration 11/ 12 | consumed samples: 88 | elapsed time per iteration (ms): 317.1 | throughput per GPU (TFLOP/s/GPU): 570.2 | learning rate: 1.000000E-05 | global batch size: 8 | lm loss: 9.332552E+00 | loss scale: 1.0 | grad norm: 4.618 | number of skipped iterations: 0 | number of nan iterations: 0 |
|
| 1268 |
+
[2026-08-30 01:32:12] iteration 12/ 12 | consumed samples: 96 | elapsed time per iteration (ms): 316.9 | throughput per GPU (TFLOP/s/GPU): 570.6 | learning rate: 1.000000E-05 | global batch size: 8 | lm loss: 9.219189E+00 | loss scale: 1.0 | grad norm: 4.492 | number of skipped iterations: 0 | number of nan iterations: 0 |
|
| 1269 |
+
[after training is done] datetime: 2026-08-30 01:32:12
|
| 1270 |
+
[rank0]:[W830 01:32:13.801912097 ProcessGroupNCCL.cpp:1505] Warning: WARNING: destroy_process_group() was not called before program exit, which can leak resources. For more info, please see https://pytorch.org/docs/stable/distributed.html#shutdown (function operator())
|
| 1271 |
+
[rank3]:[W830 01:32:14.885010634 ProcessGroupNCCL.cpp:1505] Warning: WARNING: destroy_process_group() was not called before program exit, which can leak resources. For more info, please see https://pytorch.org/docs/stable/distributed.html#shutdown (function operator())
|
| 1272 |
+
[rank6]:[W830 01:32:14.949664138 ProcessGroupNCCL.cpp:1505] Warning: WARNING: destroy_process_group() was not called before program exit, which can leak resources. For more info, please see https://pytorch.org/docs/stable/distributed.html#shutdown (function operator())
|
| 1273 |
+
[rank5]:[W830 01:32:14.003043426 ProcessGroupNCCL.cpp:1505] Warning: WARNING: destroy_process_group() was not called before program exit, which can leak resources. For more info, please see https://pytorch.org/docs/stable/distributed.html#shutdown (function operator())
|
| 1274 |
+
[rank2]:[W830 01:32:14.056089290 ProcessGroupNCCL.cpp:1505] Warning: WARNING: destroy_process_group() was not called before program exit, which can leak resources. For more info, please see https://pytorch.org/docs/stable/distributed.html#shutdown (function operator())
|
| 1275 |
+
[rank7]:[W830 01:32:15.487751536 ProcessGroupNCCL.cpp:1505] Warning: WARNING: destroy_process_group() was not called before program exit, which can leak resources. For more info, please see https://pytorch.org/docs/stable/distributed.html#shutdown (function operator())
|
| 1276 |
+
[rank4]:[W830 01:32:15.505877593 ProcessGroupNCCL.cpp:1505] Warning: WARNING: destroy_process_group() was not called before program exit, which can leak resources. For more info, please see https://pytorch.org/docs/stable/distributed.html#shutdown (function operator())
|
| 1277 |
+
[rank1]:[W830 01:32:15.558402976 ProcessGroupNCCL.cpp:1505] Warning: WARNING: destroy_process_group() was not called before program exit, which can leak resources. For more info, please see https://pytorch.org/docs/stable/distributed.html#shutdown (function operator())
|
| 1278 |
+
W0830 01:32:19.433000 380220 torch/distributed/elastic/agent/server/api.py:719] Received 15 death signal, shutting down workers
|
| 1279 |
+
W0830 01:32:19.433000 380220 torch/distributed/elastic/multiprocessing/api.py:900] Sending process 380405 closing signal SIGTERM
|
| 1280 |
+
W0830 01:32:19.434000 380220 torch/distributed/elastic/multiprocessing/api.py:900] Sending process 380406 closing signal SIGTERM
|
| 1281 |
+
W0830 01:32:19.434000 380220 torch/distributed/elastic/multiprocessing/api.py:900] Sending process 380407 closing signal SIGTERM
|
| 1282 |
+
W0830 01:32:19.434000 380220 torch/distributed/elastic/multiprocessing/api.py:900] Sending process 380408 closing signal SIGTERM
|
| 1283 |
+
W0830 01:32:19.434000 380220 torch/distributed/elastic/multiprocessing/api.py:900] Sending process 380409 closing signal SIGTERM
|
| 1284 |
+
W0830 01:32:19.435000 380220 torch/distributed/elastic/multiprocessing/api.py:900] Sending process 380410 closing signal SIGTERM
|
| 1285 |
+
W0830 01:32:19.435000 380220 torch/distributed/elastic/multiprocessing/api.py:900] Sending process 380411 closing signal SIGTERM
|
|
|
|
| 1286 |
Traceback (most recent call last):
|
| 1287 |
File "/usr/local/bin/torchrun", line 8, in <module>
|
| 1288 |
sys.exit(main())
|
|
|
|
| 1310 |
time.sleep(monitor_interval)
|
| 1311 |
File "/usr/local/lib/python3.12/dist-packages/torch/distributed/elastic/multiprocessing/api.py", line 84, in _terminate_process_handler
|
| 1312 |
raise SignalException(f"Process {os.getpid()} got signal: {sigval}", sigval=sigval)
|
| 1313 |
+
torch.distributed.elastic.multiprocessing.api.SignalException: Process 380220 got signal: 15
|
tierb-fixed-moonep-full/moonep/tierb/tierb_gbs8/QWEN3_235B__moonep__matched__seq8192__gbs8__capnone__async/rep2/mem_samples.csv
CHANGED
|
@@ -27,7 +27,7 @@
|
|
| 27 |
2, 1
|
| 28 |
3, 1
|
| 29 |
4, 1
|
| 30 |
-
5,
|
| 31 |
6, 4
|
| 32 |
7, 4
|
| 33 |
0, 0
|
|
@@ -78,52 +78,52 @@
|
|
| 78 |
5, 821
|
| 79 |
6, 821
|
| 80 |
7, 821
|
| 81 |
-
0,
|
| 82 |
-
1,
|
| 83 |
-
2,
|
| 84 |
-
3,
|
| 85 |
-
4,
|
| 86 |
-
5,
|
| 87 |
-
6,
|
| 88 |
-
7,
|
| 89 |
-
0,
|
| 90 |
-
1,
|
| 91 |
-
2,
|
| 92 |
-
3,
|
| 93 |
-
4,
|
| 94 |
-
5,
|
| 95 |
-
6,
|
| 96 |
-
7,
|
| 97 |
-
0,
|
| 98 |
-
1,
|
| 99 |
-
2,
|
| 100 |
-
3,
|
| 101 |
-
4,
|
| 102 |
-
5,
|
| 103 |
-
6,
|
| 104 |
-
7,
|
| 105 |
-
0,
|
| 106 |
-
1,
|
| 107 |
-
2,
|
| 108 |
-
3,
|
| 109 |
-
4,
|
| 110 |
-
5,
|
| 111 |
-
6,
|
| 112 |
-
7,
|
| 113 |
-
0,
|
| 114 |
-
1,
|
| 115 |
-
2,
|
| 116 |
-
3,
|
| 117 |
-
4,
|
| 118 |
-
5,
|
| 119 |
-
6,
|
| 120 |
-
7,
|
| 121 |
0, 59733
|
| 122 |
-
1,
|
| 123 |
-
2,
|
| 124 |
3, 59731
|
| 125 |
4, 59445
|
| 126 |
-
5,
|
| 127 |
6, 59731
|
| 128 |
7, 59733
|
| 129 |
0, 59733
|
|
@@ -132,16 +132,24 @@
|
|
| 132 |
3, 59733
|
| 133 |
4, 59445
|
| 134 |
5, 59733
|
| 135 |
-
6,
|
| 136 |
7, 59733
|
| 137 |
-
0,
|
| 138 |
-
1,
|
| 139 |
-
2,
|
| 140 |
-
3,
|
| 141 |
-
4,
|
| 142 |
-
5,
|
| 143 |
6, 86961
|
| 144 |
-
7,
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 145 |
0, 87633
|
| 146 |
1, 87673
|
| 147 |
2, 87673
|
|
@@ -150,74 +158,66 @@
|
|
| 150 |
5, 87673
|
| 151 |
6, 87673
|
| 152 |
7, 87673
|
| 153 |
-
0,
|
| 154 |
-
1,
|
| 155 |
-
2,
|
| 156 |
-
3,
|
| 157 |
-
4,
|
| 158 |
-
5,
|
| 159 |
-
6,
|
| 160 |
-
7,
|
| 161 |
-
0,
|
| 162 |
-
1,
|
| 163 |
-
2,
|
| 164 |
-
3,
|
| 165 |
-
4,
|
| 166 |
-
5,
|
| 167 |
-
6,
|
| 168 |
-
7,
|
| 169 |
-
0,
|
| 170 |
-
1,
|
| 171 |
-
2,
|
| 172 |
-
3,
|
| 173 |
-
4,
|
| 174 |
-
5,
|
| 175 |
-
6,
|
| 176 |
-
7,
|
| 177 |
-
0,
|
| 178 |
-
1,
|
| 179 |
-
2,
|
| 180 |
-
3,
|
| 181 |
-
4,
|
| 182 |
-
5,
|
| 183 |
-
6,
|
| 184 |
-
7,
|
| 185 |
-
0,
|
| 186 |
-
1,
|
| 187 |
-
2,
|
| 188 |
-
3,
|
| 189 |
-
4,
|
| 190 |
-
5,
|
| 191 |
-
6,
|
| 192 |
-
7,
|
| 193 |
-
0,
|
| 194 |
-
1,
|
| 195 |
-
2,
|
| 196 |
-
3,
|
| 197 |
-
4,
|
| 198 |
-
5,
|
| 199 |
-
6,
|
| 200 |
-
7,
|
| 201 |
-
0,
|
| 202 |
-
1,
|
| 203 |
-
2,
|
| 204 |
-
3,
|
| 205 |
-
4,
|
| 206 |
-
5,
|
| 207 |
-
6,
|
| 208 |
-
7,
|
| 209 |
-
0, 114307
|
| 210 |
-
1, 114087
|
| 211 |
-
2, 114669
|
| 212 |
-
3, 114343
|
| 213 |
-
4, 114087
|
| 214 |
-
5, 114669
|
| 215 |
-
6, 114407
|
| 216 |
-
7, 114669
|
| 217 |
0, 1
|
| 218 |
-
1,
|
| 219 |
-
2,
|
| 220 |
-
3,
|
| 221 |
4, 4
|
| 222 |
5, 4
|
| 223 |
6, 4
|
|
|
|
| 27 |
2, 1
|
| 28 |
3, 1
|
| 29 |
4, 1
|
| 30 |
+
5, 1
|
| 31 |
6, 4
|
| 32 |
7, 4
|
| 33 |
0, 0
|
|
|
|
| 78 |
5, 821
|
| 79 |
6, 821
|
| 80 |
7, 821
|
| 81 |
+
0, 1537
|
| 82 |
+
1, 1537
|
| 83 |
+
2, 1537
|
| 84 |
+
3, 1537
|
| 85 |
+
4, 1537
|
| 86 |
+
5, 1537
|
| 87 |
+
6, 1537
|
| 88 |
+
7, 1537
|
| 89 |
+
0, 2551
|
| 90 |
+
1, 2551
|
| 91 |
+
2, 2551
|
| 92 |
+
3, 1783
|
| 93 |
+
4, 2551
|
| 94 |
+
5, 1783
|
| 95 |
+
6, 2551
|
| 96 |
+
7, 2551
|
| 97 |
+
0, 46595
|
| 98 |
+
1, 46609
|
| 99 |
+
2, 46605
|
| 100 |
+
3, 46633
|
| 101 |
+
4, 46595
|
| 102 |
+
5, 46603
|
| 103 |
+
6, 46605
|
| 104 |
+
7, 46571
|
| 105 |
+
0, 56769
|
| 106 |
+
1, 56737
|
| 107 |
+
2, 56805
|
| 108 |
+
3, 56785
|
| 109 |
+
4, 56561
|
| 110 |
+
5, 56801
|
| 111 |
+
6, 56851
|
| 112 |
+
7, 56831
|
| 113 |
+
0, 58731
|
| 114 |
+
1, 58731
|
| 115 |
+
2, 58727
|
| 116 |
+
3, 58747
|
| 117 |
+
4, 58469
|
| 118 |
+
5, 58751
|
| 119 |
+
6, 58751
|
| 120 |
+
7, 58747
|
| 121 |
0, 59733
|
| 122 |
+
1, 59733
|
| 123 |
+
2, 59731
|
| 124 |
3, 59731
|
| 125 |
4, 59445
|
| 126 |
+
5, 59733
|
| 127 |
6, 59731
|
| 128 |
7, 59733
|
| 129 |
0, 59733
|
|
|
|
| 132 |
3, 59733
|
| 133 |
4, 59445
|
| 134 |
5, 59733
|
| 135 |
+
6, 59733
|
| 136 |
7, 59733
|
| 137 |
+
0, 85347
|
| 138 |
+
1, 85387
|
| 139 |
+
2, 85387
|
| 140 |
+
3, 85387
|
| 141 |
+
4, 85347
|
| 142 |
+
5, 85387
|
| 143 |
6, 86961
|
| 144 |
+
7, 86961
|
| 145 |
+
0, 87633
|
| 146 |
+
1, 87673
|
| 147 |
+
2, 87673
|
| 148 |
+
3, 87673
|
| 149 |
+
4, 87633
|
| 150 |
+
5, 87673
|
| 151 |
+
6, 87673
|
| 152 |
+
7, 87673
|
| 153 |
0, 87633
|
| 154 |
1, 87673
|
| 155 |
2, 87673
|
|
|
|
| 158 |
5, 87673
|
| 159 |
6, 87673
|
| 160 |
7, 87673
|
| 161 |
+
0, 88091
|
| 162 |
+
1, 88131
|
| 163 |
+
2, 88129
|
| 164 |
+
3, 88143
|
| 165 |
+
4, 88091
|
| 166 |
+
5, 88153
|
| 167 |
+
6, 88149
|
| 168 |
+
7, 88145
|
| 169 |
+
0, 97523
|
| 170 |
+
1, 97533
|
| 171 |
+
2, 97555
|
| 172 |
+
3, 97541
|
| 173 |
+
4, 97557
|
| 174 |
+
5, 97583
|
| 175 |
+
6, 97567
|
| 176 |
+
7, 97643
|
| 177 |
+
0, 120273
|
| 178 |
+
1, 120309
|
| 179 |
+
2, 120309
|
| 180 |
+
3, 120309
|
| 181 |
+
4, 120309
|
| 182 |
+
5, 120309
|
| 183 |
+
6, 120309
|
| 184 |
+
7, 120309
|
| 185 |
+
0, 120273
|
| 186 |
+
1, 120309
|
| 187 |
+
2, 120309
|
| 188 |
+
3, 120309
|
| 189 |
+
4, 120309
|
| 190 |
+
5, 120309
|
| 191 |
+
6, 120309
|
| 192 |
+
7, 120309
|
| 193 |
+
0, 119627
|
| 194 |
+
1, 119663
|
| 195 |
+
2, 119663
|
| 196 |
+
3, 119663
|
| 197 |
+
4, 119663
|
| 198 |
+
5, 119663
|
| 199 |
+
6, 119663
|
| 200 |
+
7, 119663
|
| 201 |
+
0, 117401
|
| 202 |
+
1, 117825
|
| 203 |
+
2, 117437
|
| 204 |
+
3, 117437
|
| 205 |
+
4, 117437
|
| 206 |
+
5, 117437
|
| 207 |
+
6, 117437
|
| 208 |
+
7, 117825
|
| 209 |
+
0, 92869
|
| 210 |
+
1, 115757
|
| 211 |
+
2, 115757
|
| 212 |
+
3, 92905
|
| 213 |
+
4, 115757
|
| 214 |
+
5, 115757
|
| 215 |
+
6, 94633
|
| 216 |
+
7, 115757
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 217 |
0, 1
|
| 218 |
+
1, 113926
|
| 219 |
+
2, 4
|
| 220 |
+
3, 4
|
| 221 |
4, 4
|
| 222 |
5, 4
|
| 223 |
6, 4
|
tierb-fixed-moonep-full/moonep/tierb/tierb_gbs8/QWEN3_235B__moonep__matched__seq8192__gbs8__capnone__async/rep2/run.env
CHANGED
|
@@ -3,7 +3,7 @@ config=QWEN3_235B
|
|
| 3 |
family=tierB
|
| 4 |
repo=/workspace/Megatron-MoE-Echo
|
| 5 |
branch=moonep_backend_async_grad_reduce
|
| 6 |
-
commit=
|
| 7 |
moonep_source_dir=/workspace/MoonEP
|
| 8 |
moonep_source_branch=megatron-ddp-symmetric-params
|
| 9 |
moonep_source_commit=d8bc666
|
|
|
|
| 3 |
family=tierB
|
| 4 |
repo=/workspace/Megatron-MoE-Echo
|
| 5 |
branch=moonep_backend_async_grad_reduce
|
| 6 |
+
commit=2e68c8ad
|
| 7 |
moonep_source_dir=/workspace/MoonEP
|
| 8 |
moonep_source_branch=megatron-ddp-symmetric-params
|
| 9 |
moonep_source_commit=d8bc666
|
tierb-fixed-moonep-full/moonep/tierb/tierb_gbs8/QWEN3_235B__moonep__matched__seq8192__gbs8__capnone__async/rep2/train.log
CHANGED
|
@@ -1,25 +1,23 @@
|
|
| 1 |
-
[
|
| 2 |
==============================================================
|
| 3 |
QWEN3_235B / moonep (slots/rank=16 policy=matched)
|
| 4 |
E=128 K=8 H'=1536 m=16
|
| 5 |
L=12 seq=8192 gbs=8 (1 microbatches)
|
| 6 |
cap_mode=none cap=none cg=1 recompute=none
|
| 7 |
memory_dump=none snapshots=/workspace/tierb-fixed-moonep-full/moonep/tierb/tierb_gbs8/QWEN3_235B__moonep__matched__seq8192__gbs8__capnone__async/rep2/memory_snapshots/snapshot.pickle
|
| 8 |
-
repo=/workspace/Megatron-MoE-Echo @
|
| 9 |
==============================================================
|
| 10 |
-
|
| 11 |
-
|
| 12 |
-
|
| 13 |
-
|
| 14 |
/workspace/Megatron-MoE-Echo/megatron/training/arguments.py:831: UserWarning: Disabling sequence parallelism because tensor model parallelism is disabled
|
| 15 |
warnings.warn("Disabling sequence parallelism because tensor model parallelism is disabled")
|
| 16 |
-
/workspace/Megatron-MoE-Echo/megatron/training/arguments.py:831: UserWarning: Disabling sequence parallelism because tensor model parallelism is disabled
|
| 17 |
-
warnings.warn("Disabling sequence parallelism because tensor model parallelism is disabled")
|
| 18 |
-
WARNING: one_logger package is required to enable e2e metrics tracking. please go to https://confluence.nvidia.com/display/MLWFO/Package+Repositories for details to install it
|
| 19 |
/workspace/Megatron-MoE-Echo/megatron/training/arguments.py:831: UserWarning: Disabling sequence parallelism because tensor model parallelism is disabled
|
| 20 |
warnings.warn("Disabling sequence parallelism because tensor model parallelism is disabled")
|
| 21 |
/workspace/Megatron-MoE-Echo/megatron/training/arguments.py:831: UserWarning: Disabling sequence parallelism because tensor model parallelism is disabled
|
| 22 |
warnings.warn("Disabling sequence parallelism because tensor model parallelism is disabled")
|
|
|
|
| 23 |
/workspace/Megatron-MoE-Echo/megatron/training/arguments.py:831: UserWarning: Disabling sequence parallelism because tensor model parallelism is disabled
|
| 24 |
warnings.warn("Disabling sequence parallelism because tensor model parallelism is disabled")
|
| 25 |
using world size: 8, data-parallel size: 8, context-parallel size: 1, hierarchical context-parallel sizes: None, tensor-model-parallel size: 1, pipeline-model-parallel size: 1
|
|
@@ -577,51 +575,53 @@ WARNING:megatron.core.rerun_state_machine:RerunStateMachine initialized in mode
|
|
| 577 |
warnings.warn("Disabling sequence parallelism because tensor model parallelism is disabled")
|
| 578 |
/workspace/Megatron-MoE-Echo/megatron/training/arguments.py:831: UserWarning: Disabling sequence parallelism because tensor model parallelism is disabled
|
| 579 |
warnings.warn("Disabling sequence parallelism because tensor model parallelism is disabled")
|
| 580 |
-
|
| 581 |
-
|
| 582 |
-
[Gloo] Rank 1 is connected to 7 peer ranks. Expected number of connected peer ranks is : 7
|
| 583 |
[Gloo] Rank 4 is connected to 7 peer ranks. Expected number of connected peer ranks is : 7
|
| 584 |
-
[Gloo] Rank [Gloo] Rank [Gloo] Rank [Gloo] Rank 3657 is connected to is connected to is connected to is connected to 7777 peer ranks. peer ranks. peer ranks. peer ranks. Expected number of connected peer ranks is : Expected number of connected peer ranks is : Expected number of connected peer ranks is : Expected number of connected peer ranks is : 7777
|
| 585 |
-
|
| 586 |
-
|
| 587 |
-
|
| 588 |
[Gloo] Rank 0 is connected to 7 peer ranks. Expected number of connected peer ranks is : 7
|
|
|
|
| 589 |
[Gloo] Rank 2 is connected to 7 peer ranks. Expected number of connected peer ranks is : 7
|
|
|
|
|
|
|
| 590 |
[Gloo] Rank 5 is connected to 7 peer ranks. Expected number of connected peer ranks is : 7
|
|
|
|
| 591 |
[Gloo] Rank 3 is connected to 7 peer ranks. Expected number of connected peer ranks is : 7
|
|
|
|
| 592 |
[Gloo] Rank 1 is connected to 7 peer ranks. Expected number of connected peer ranks is : 7
|
| 593 |
-
[Gloo] Rank
|
| 594 |
-
[Gloo] Rank [Gloo] Rank
|
| 595 |
-
|
|
|
|
|
|
|
| 596 |
[Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
|
| 597 |
> initialized tensor model parallel with size 1
|
| 598 |
> initialized pipeline model parallel with size 1
|
| 599 |
> setting random seeds to 1234 ...
|
|
|
|
|
|
|
|
|
|
|
|
|
| 600 |
[Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
|
| 601 |
[Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
|
| 602 |
[Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
|
| 603 |
-
[
|
| 604 |
-
[
|
| 605 |
-
|
| 606 |
-
[
|
| 607 |
-
[
|
| 608 |
-
[rank5]:[
|
| 609 |
-
[
|
| 610 |
-
[rank1]:[W829 23:46:37.521143928 ProcessGroupNCCL.cpp:4789] [PG ID 0 PG GUID 0 Rank 1] using GPU 1 as device used by this process is currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect. You can specify device_id in init_process_group() to force use of a particular device.
|
| 611 |
-
[rank6]:[W829 23:46:37.522800573 ProcessGroupNCCL.cpp:4789] [PG ID 0 PG GUID 0 Rank 6] using GPU 6 as device used by this process is currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect. You can specify device_id in init_process_group() to force use of a particular device.
|
| 612 |
-
[rank7]:[W829 23:46:37.522847520 ProcessGroupNCCL.cpp:4789] [PG ID 0 PG GUID 0 Rank 7] using GPU 7 as device used by this process is currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect. You can specify device_id in init_process_group() to force use of a particular device.
|
| 613 |
-
[rank3]:[W829 23:46:37.522872548 ProcessGroupNCCL.cpp:4789] [PG ID 0 PG GUID 0 Rank 3] using GPU 3 as device used by this process is currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect. You can specify device_id in init_process_group() to force use of a particular device.
|
| 614 |
> compiling dataset index builder ...
|
| 615 |
make: Entering directory '/workspace/Megatron-MoE-Echo/megatron/core/datasets'
|
| 616 |
make: Nothing to be done for 'default'.
|
| 617 |
make: Leaving directory '/workspace/Megatron-MoE-Echo/megatron/core/datasets'
|
| 618 |
-
>>> done with dataset index builder. Compilation time: 0.
|
| 619 |
WARNING: constraints for invoking optimized fused softmax kernel are not met. We default back to unfused kernel invocations.
|
| 620 |
> compiling and loading fused kernels ...
|
| 621 |
-
[rank0]:[
|
| 622 |
-
>>> done with compiling and loading fused kernels. Compilation time:
|
| 623 |
-
time to initialize megatron (seconds): 7.
|
| 624 |
-
[after megatron is initialized] datetime: 2026-08-
|
| 625 |
building GPT model ...
|
| 626 |
/usr/local/lib/python3.12/dist-packages/transformer_engine/pytorch/cpu_offload.py:708: DeprecationWarning: Offloading weights is deprecated. Using offload_weights=True does not have any effect.
|
| 627 |
warnings.warn(
|
|
@@ -645,43 +645,43 @@ INFO:megatron.core.distributed.param_and_grad_buffer:Number of buckets for gradi
|
|
| 645 |
Params for bucket 1 (206045184 elements, 206045184 padded size):
|
| 646 |
module.output_layer.weight
|
| 647 |
Params for bucket 2 (71840000 elements, 71840000 padded size):
|
|
|
|
|
|
|
| 648 |
module.decoder.layers.11.pre_mlp_layernorm.weight
|
| 649 |
module.decoder.layers.11.self_attention.q_layernorm.weight
|
| 650 |
-
module.decoder.layers.11.
|
| 651 |
module.decoder.layers.11.self_attention.linear_qkv.weight
|
| 652 |
module.decoder.layers.11.self_attention.linear_qkv.layer_norm_weight
|
| 653 |
-
module.decoder.final_layernorm.weight
|
| 654 |
-
module.decoder.layers.11.self_attention.k_layernorm.weight
|
| 655 |
module.decoder.layers.11.self_attention.linear_proj.weight
|
| 656 |
Params for bucket 3 (71835904 elements, 71835904 padded size):
|
| 657 |
-
module.decoder.layers.10.mlp.router.weight
|
| 658 |
module.decoder.layers.10.pre_mlp_layernorm.weight
|
| 659 |
module.decoder.layers.10.self_attention.q_layernorm.weight
|
| 660 |
-
module.decoder.layers.10.self_attention.linear_qkv.weight
|
| 661 |
module.decoder.layers.10.self_attention.k_layernorm.weight
|
|
|
|
| 662 |
module.decoder.layers.10.self_attention.linear_qkv.layer_norm_weight
|
|
|
|
| 663 |
module.decoder.layers.10.self_attention.linear_proj.weight
|
| 664 |
Params for bucket 4 (71835904 elements, 71835904 padded size):
|
| 665 |
-
module.decoder.layers.9.mlp.router.weight
|
| 666 |
-
module.decoder.layers.9.pre_mlp_layernorm.weight
|
| 667 |
-
module.decoder.layers.9.self_attention.q_layernorm.weight
|
| 668 |
-
module.decoder.layers.9.self_attention.linear_qkv.layer_norm_weight
|
| 669 |
module.decoder.layers.9.self_attention.linear_qkv.weight
|
| 670 |
module.decoder.layers.9.self_attention.k_layernorm.weight
|
|
|
|
| 671 |
module.decoder.layers.9.self_attention.linear_proj.weight
|
|
|
|
|
|
|
|
|
|
| 672 |
Params for bucket 5 (71835904 elements, 71835904 padded size):
|
| 673 |
module.decoder.layers.8.mlp.router.weight
|
|
|
|
| 674 |
module.decoder.layers.8.self_attention.k_layernorm.weight
|
| 675 |
module.decoder.layers.8.self_attention.q_layernorm.weight
|
| 676 |
-
module.decoder.layers.8.pre_mlp_layernorm.weight
|
| 677 |
module.decoder.layers.8.self_attention.linear_qkv.weight
|
| 678 |
module.decoder.layers.8.self_attention.linear_qkv.layer_norm_weight
|
| 679 |
module.decoder.layers.8.self_attention.linear_proj.weight
|
| 680 |
Params for bucket 6 (71835904 elements, 71835904 padded size):
|
| 681 |
module.decoder.layers.7.mlp.router.weight
|
|
|
|
| 682 |
module.decoder.layers.7.pre_mlp_layernorm.weight
|
| 683 |
module.decoder.layers.7.self_attention.k_layernorm.weight
|
| 684 |
-
module.decoder.layers.7.self_attention.q_layernorm.weight
|
| 685 |
module.decoder.layers.7.self_attention.linear_qkv.layer_norm_weight
|
| 686 |
module.decoder.layers.7.self_attention.linear_qkv.weight
|
| 687 |
module.decoder.layers.7.self_attention.linear_proj.weight
|
|
@@ -689,167 +689,167 @@ Params for bucket 7 (71835904 elements, 71835904 padded size):
|
|
| 689 |
module.decoder.layers.6.mlp.router.weight
|
| 690 |
module.decoder.layers.6.self_attention.linear_qkv.weight
|
| 691 |
module.decoder.layers.6.self_attention.k_layernorm.weight
|
| 692 |
-
module.decoder.layers.6.self_attention.
|
| 693 |
module.decoder.layers.6.pre_mlp_layernorm.weight
|
| 694 |
module.decoder.layers.6.self_attention.q_layernorm.weight
|
| 695 |
-
module.decoder.layers.6.self_attention.
|
| 696 |
Params for bucket 8 (71835904 elements, 71835904 padded size):
|
| 697 |
module.decoder.layers.5.mlp.router.weight
|
|
|
|
| 698 |
module.decoder.layers.5.self_attention.linear_qkv.weight
|
| 699 |
-
module.decoder.layers.5.self_attention.q_layernorm.weight
|
| 700 |
module.decoder.layers.5.pre_mlp_layernorm.weight
|
|
|
|
| 701 |
module.decoder.layers.5.self_attention.linear_qkv.layer_norm_weight
|
| 702 |
-
module.decoder.layers.5.self_attention.k_layernorm.weight
|
| 703 |
module.decoder.layers.5.self_attention.linear_proj.weight
|
| 704 |
Params for bucket 9 (71835904 elements, 71835904 padded size):
|
| 705 |
-
module.decoder.layers.4.
|
| 706 |
-
module.decoder.layers.4.self_attention.linear_qkv.weight
|
| 707 |
module.decoder.layers.4.self_attention.q_layernorm.weight
|
|
|
|
| 708 |
module.decoder.layers.4.self_attention.linear_qkv.layer_norm_weight
|
| 709 |
-
module.decoder.layers.4.mlp.router.weight
|
| 710 |
module.decoder.layers.4.self_attention.k_layernorm.weight
|
|
|
|
| 711 |
module.decoder.layers.4.self_attention.linear_proj.weight
|
| 712 |
Params for bucket 10 (71835904 elements, 71835904 padded size):
|
| 713 |
module.decoder.layers.3.mlp.router.weight
|
| 714 |
-
module.decoder.layers.3.pre_mlp_layernorm.weight
|
| 715 |
module.decoder.layers.3.self_attention.q_layernorm.weight
|
| 716 |
module.decoder.layers.3.self_attention.linear_qkv.weight
|
|
|
|
| 717 |
module.decoder.layers.3.self_attention.k_layernorm.weight
|
| 718 |
module.decoder.layers.3.self_attention.linear_qkv.layer_norm_weight
|
| 719 |
module.decoder.layers.3.self_attention.linear_proj.weight
|
| 720 |
Params for bucket 11 (71835904 elements, 71835904 padded size):
|
| 721 |
module.decoder.layers.2.mlp.router.weight
|
|
|
|
|
|
|
| 722 |
module.decoder.layers.2.pre_mlp_layernorm.weight
|
| 723 |
module.decoder.layers.2.self_attention.linear_qkv.weight
|
| 724 |
-
module.decoder.layers.2.self_attention.q_layernorm.weight
|
| 725 |
-
module.decoder.layers.2.self_attention.k_layernorm.weight
|
| 726 |
module.decoder.layers.2.self_attention.linear_qkv.layer_norm_weight
|
| 727 |
module.decoder.layers.2.self_attention.linear_proj.weight
|
| 728 |
Params for bucket 12 (71835904 elements, 71835904 padded size):
|
| 729 |
module.decoder.layers.1.mlp.router.weight
|
| 730 |
module.decoder.layers.1.pre_mlp_layernorm.weight
|
| 731 |
-
module.decoder.layers.1.self_attention.q_layernorm.weight
|
| 732 |
-
module.decoder.layers.1.self_attention.linear_qkv.weight
|
| 733 |
module.decoder.layers.1.self_attention.k_layernorm.weight
|
|
|
|
|
|
|
| 734 |
module.decoder.layers.1.self_attention.linear_qkv.layer_norm_weight
|
| 735 |
module.decoder.layers.1.self_attention.linear_proj.weight
|
| 736 |
Params for bucket 13 (71835904 elements, 71835904 padded size):
|
|
|
|
|
|
|
| 737 |
module.decoder.layers.0.self_attention.q_layernorm.weight
|
| 738 |
module.decoder.layers.0.pre_mlp_layernorm.weight
|
| 739 |
-
module.decoder.layers.0.self_attention.linear_qkv.layer_norm_weight
|
| 740 |
-
module.decoder.layers.0.mlp.router.weight
|
| 741 |
-
module.decoder.layers.0.self_attention.linear_qkv.weight
|
| 742 |
-
module.decoder.layers.0.self_attention.k_layernorm.weight
|
| 743 |
module.decoder.layers.0.self_attention.linear_proj.weight
|
|
|
|
|
|
|
| 744 |
Params for bucket 14 (206045184 elements, 206045184 padded size):
|
| 745 |
module.embedding.word_embeddings.weight
|
| 746 |
INFO:megatron.core.distributed.param_and_grad_buffer:Number of buckets for gradient all-reduce / reduce-scatter: 76
|
| 747 |
Params for bucket 1 (44040192 elements, 44040192 padded size):
|
| 748 |
-
module.decoder.layers.11.mlp.experts.linear_fc2.weight0
|
| 749 |
-
module.decoder.layers.11.mlp.experts.linear_fc2.weight2
|
| 750 |
-
module.decoder.layers.11.mlp.experts.linear_fc2.weight4
|
| 751 |
-
module.decoder.layers.11.mlp.experts.linear_fc2.weight6
|
| 752 |
module.decoder.layers.11.mlp.experts.linear_fc2.weight1
|
| 753 |
module.decoder.layers.11.mlp.experts.linear_fc2.weight3
|
| 754 |
module.decoder.layers.11.mlp.experts.linear_fc2.weight5
|
|
|
|
|
|
|
|
|
|
|
|
|
| 755 |
Params for bucket 2 (44040192 elements, 44040192 padded size):
|
| 756 |
-
module.decoder.layers.11.mlp.experts.linear_fc2.weight8
|
| 757 |
-
module.decoder.layers.11.mlp.experts.linear_fc2.weight10
|
| 758 |
-
module.decoder.layers.11.mlp.experts.linear_fc2.weight12
|
| 759 |
module.decoder.layers.11.mlp.experts.linear_fc2.weight7
|
| 760 |
module.decoder.layers.11.mlp.experts.linear_fc2.weight11
|
| 761 |
module.decoder.layers.11.mlp.experts.linear_fc2.weight9
|
| 762 |
module.decoder.layers.11.mlp.experts.linear_fc2.weight13
|
|
|
|
|
|
|
|
|
|
| 763 |
Params for bucket 3 (50331648 elements, 50331648 padded size):
|
| 764 |
-
module.decoder.layers.11.mlp.experts.
|
| 765 |
-
module.decoder.layers.11.mlp.experts.linear_fc1.weight1
|
| 766 |
module.decoder.layers.11.mlp.experts.linear_fc1.weight2
|
| 767 |
module.decoder.layers.11.mlp.experts.linear_fc2.weight15
|
| 768 |
-
module.decoder.layers.11.mlp.experts.
|
|
|
|
| 769 |
Params for bucket 4 (50331648 elements, 50331648 padded size):
|
| 770 |
module.decoder.layers.11.mlp.experts.linear_fc1.weight3
|
| 771 |
module.decoder.layers.11.mlp.experts.linear_fc1.weight4
|
| 772 |
-
module.decoder.layers.11.mlp.experts.linear_fc1.weight5
|
| 773 |
module.decoder.layers.11.mlp.experts.linear_fc1.weight6
|
|
|
|
| 774 |
Params for bucket 5 (50331648 elements, 50331648 padded size):
|
| 775 |
module.decoder.layers.11.mlp.experts.linear_fc1.weight7
|
| 776 |
-
module.decoder.layers.11.mlp.experts.linear_fc1.weight8
|
| 777 |
module.decoder.layers.11.mlp.experts.linear_fc1.weight10
|
|
|
|
| 778 |
module.decoder.layers.11.mlp.experts.linear_fc1.weight9
|
| 779 |
Params for bucket 6 (50331648 elements, 50331648 padded size):
|
| 780 |
module.decoder.layers.11.mlp.experts.linear_fc1.weight11
|
| 781 |
-
module.decoder.layers.11.mlp.experts.linear_fc1.weight14
|
| 782 |
module.decoder.layers.11.mlp.experts.linear_fc1.weight13
|
|
|
|
| 783 |
module.decoder.layers.11.mlp.experts.linear_fc1.weight12
|
| 784 |
Params for bucket 7 (44040192 elements, 44040192 padded size):
|
| 785 |
-
module.decoder.layers.11.mlp.experts.linear_fc1.weight15
|
| 786 |
module.decoder.layers.10.mlp.experts.linear_fc2.weight3
|
| 787 |
module.decoder.layers.10.mlp.experts.linear_fc2.weight1
|
| 788 |
-
module.decoder.layers.
|
| 789 |
module.decoder.layers.10.mlp.experts.linear_fc2.weight2
|
|
|
|
| 790 |
module.decoder.layers.10.mlp.experts.linear_fc2.weight4
|
| 791 |
Params for bucket 8 (44040192 elements, 44040192 padded size):
|
| 792 |
module.decoder.layers.10.mlp.experts.linear_fc2.weight5
|
| 793 |
-
module.decoder.layers.10.mlp.experts.linear_fc2.weight7
|
| 794 |
module.decoder.layers.10.mlp.experts.linear_fc2.weight9
|
|
|
|
| 795 |
module.decoder.layers.10.mlp.experts.linear_fc2.weight11
|
| 796 |
-
module.decoder.layers.10.mlp.experts.linear_fc2.weight8
|
| 797 |
module.decoder.layers.10.mlp.experts.linear_fc2.weight6
|
|
|
|
| 798 |
module.decoder.layers.10.mlp.experts.linear_fc2.weight10
|
| 799 |
Params for bucket 9 (50331648 elements, 50331648 padded size):
|
| 800 |
-
module.decoder.layers.10.mlp.experts.linear_fc2.weight13
|
| 801 |
module.decoder.layers.10.mlp.experts.linear_fc1.weight0
|
| 802 |
module.decoder.layers.10.mlp.experts.linear_fc2.weight15
|
|
|
|
| 803 |
module.decoder.layers.10.mlp.experts.linear_fc2.weight12
|
| 804 |
module.decoder.layers.10.mlp.experts.linear_fc2.weight14
|
| 805 |
module.decoder.layers.10.mlp.experts.linear_fc1.weight1
|
| 806 |
Params for bucket 10 (50331648 elements, 50331648 padded size):
|
| 807 |
module.decoder.layers.10.mlp.experts.linear_fc1.weight2
|
| 808 |
-
module.decoder.layers.10.mlp.experts.linear_fc1.weight4
|
| 809 |
module.decoder.layers.10.mlp.experts.linear_fc1.weight5
|
| 810 |
module.decoder.layers.10.mlp.experts.linear_fc1.weight3
|
|
|
|
| 811 |
Params for bucket 11 (50331648 elements, 50331648 padded size):
|
| 812 |
module.decoder.layers.10.mlp.experts.linear_fc1.weight6
|
| 813 |
-
module.decoder.layers.10.mlp.experts.linear_fc1.weight7
|
| 814 |
module.decoder.layers.10.mlp.experts.linear_fc1.weight8
|
| 815 |
module.decoder.layers.10.mlp.experts.linear_fc1.weight9
|
|
|
|
| 816 |
Params for bucket 12 (50331648 elements, 50331648 padded size):
|
| 817 |
module.decoder.layers.10.mlp.experts.linear_fc1.weight10
|
|
|
|
| 818 |
module.decoder.layers.10.mlp.experts.linear_fc1.weight13
|
| 819 |
module.decoder.layers.10.mlp.experts.linear_fc1.weight11
|
| 820 |
-
module.decoder.layers.10.mlp.experts.linear_fc1.weight12
|
| 821 |
Params for bucket 13 (44040192 elements, 44040192 padded size):
|
| 822 |
module.decoder.layers.10.mlp.experts.linear_fc1.weight14
|
| 823 |
-
module.decoder.layers.9.mlp.experts.linear_fc2.weight1
|
| 824 |
module.decoder.layers.9.mlp.experts.linear_fc2.weight2
|
| 825 |
module.decoder.layers.9.mlp.experts.linear_fc2.weight0
|
|
|
|
| 826 |
module.decoder.layers.10.mlp.experts.linear_fc1.weight15
|
| 827 |
Params for bucket 14 (44040192 elements, 44040192 padded size):
|
|
|
|
|
|
|
|
|
|
| 828 |
module.decoder.layers.9.mlp.experts.linear_fc2.weight3
|
| 829 |
module.decoder.layers.9.mlp.experts.linear_fc2.weight5
|
| 830 |
module.decoder.layers.9.mlp.experts.linear_fc2.weight7
|
| 831 |
module.decoder.layers.9.mlp.experts.linear_fc2.weight9
|
| 832 |
-
module.decoder.layers.9.mlp.experts.linear_fc2.weight6
|
| 833 |
-
module.decoder.layers.9.mlp.experts.linear_fc2.weight4
|
| 834 |
-
module.decoder.layers.9.mlp.experts.linear_fc2.weight8
|
| 835 |
Params for bucket 15 (50331648 elements, 50331648 padded size):
|
| 836 |
-
module.decoder.layers.9.mlp.experts.linear_fc2.
|
|
|
|
|
|
|
| 837 |
module.decoder.layers.9.mlp.experts.linear_fc2.weight11
|
|
|
|
| 838 |
module.decoder.layers.9.mlp.experts.linear_fc2.weight15
|
| 839 |
module.decoder.layers.9.mlp.experts.linear_fc1.weight0
|
| 840 |
-
module.decoder.layers.9.mlp.experts.linear_fc2.weight10
|
| 841 |
-
module.decoder.layers.9.mlp.experts.linear_fc2.weight14
|
| 842 |
-
module.decoder.layers.9.mlp.experts.linear_fc2.weight12
|
| 843 |
Params for bucket 16 (50331648 elements, 50331648 padded size):
|
| 844 |
module.decoder.layers.9.mlp.experts.linear_fc1.weight1
|
|
|
|
| 845 |
module.decoder.layers.9.mlp.experts.linear_fc1.weight2
|
| 846 |
module.decoder.layers.9.mlp.experts.linear_fc1.weight3
|
| 847 |
-
module.decoder.layers.9.mlp.experts.linear_fc1.weight4
|
| 848 |
Params for bucket 17 (50331648 elements, 50331648 padded size):
|
| 849 |
module.decoder.layers.9.mlp.experts.linear_fc1.weight5
|
|
|
|
| 850 |
module.decoder.layers.9.mlp.experts.linear_fc1.weight8
|
| 851 |
module.decoder.layers.9.mlp.experts.linear_fc1.weight6
|
| 852 |
-
module.decoder.layers.9.mlp.experts.linear_fc1.weight7
|
| 853 |
Params for bucket 18 (50331648 elements, 50331648 padded size):
|
| 854 |
module.decoder.layers.9.mlp.experts.linear_fc1.weight9
|
| 855 |
module.decoder.layers.9.mlp.experts.linear_fc1.weight11
|
|
@@ -857,81 +857,81 @@ Params for bucket 18 (50331648 elements, 50331648 padded size):
|
|
| 857 |
module.decoder.layers.9.mlp.experts.linear_fc1.weight10
|
| 858 |
Params for bucket 19 (44040192 elements, 44040192 padded size):
|
| 859 |
module.decoder.layers.9.mlp.experts.linear_fc1.weight13
|
| 860 |
-
module.decoder.layers.8.mlp.experts.linear_fc2.weight0
|
| 861 |
module.decoder.layers.9.mlp.experts.linear_fc1.weight14
|
| 862 |
module.decoder.layers.9.mlp.experts.linear_fc1.weight15
|
|
|
|
| 863 |
Params for bucket 20 (44040192 elements, 44040192 padded size):
|
| 864 |
-
module.decoder.layers.8.mlp.experts.linear_fc2.weight4
|
| 865 |
-
module.decoder.layers.8.mlp.experts.linear_fc2.weight2
|
| 866 |
-
module.decoder.layers.8.mlp.experts.linear_fc2.weight6
|
| 867 |
module.decoder.layers.8.mlp.experts.linear_fc2.weight1
|
| 868 |
-
module.decoder.layers.8.mlp.experts.linear_fc2.weight5
|
| 869 |
module.decoder.layers.8.mlp.experts.linear_fc2.weight3
|
|
|
|
| 870 |
module.decoder.layers.8.mlp.experts.linear_fc2.weight7
|
|
|
|
|
|
|
|
|
|
| 871 |
Params for bucket 21 (44040192 elements, 44040192 padded size):
|
|
|
|
|
|
|
|
|
|
| 872 |
module.decoder.layers.8.mlp.experts.linear_fc2.weight8
|
| 873 |
module.decoder.layers.8.mlp.experts.linear_fc2.weight10
|
| 874 |
module.decoder.layers.8.mlp.experts.linear_fc2.weight12
|
| 875 |
module.decoder.layers.8.mlp.experts.linear_fc2.weight14
|
| 876 |
-
module.decoder.layers.8.mlp.experts.linear_fc2.weight11
|
| 877 |
-
module.decoder.layers.8.mlp.experts.linear_fc2.weight9
|
| 878 |
-
module.decoder.layers.8.mlp.experts.linear_fc2.weight13
|
| 879 |
Params for bucket 22 (44040192 elements, 44040192 padded size):
|
| 880 |
module.decoder.layers.8.mlp.experts.linear_fc2.weight15
|
| 881 |
module.decoder.layers.8.mlp.experts.linear_fc1.weight1
|
| 882 |
-
module.decoder.layers.8.mlp.experts.linear_fc1.weight2
|
| 883 |
module.decoder.layers.8.mlp.experts.linear_fc1.weight0
|
|
|
|
| 884 |
Params for bucket 23 (50331648 elements, 50331648 padded size):
|
| 885 |
module.decoder.layers.8.mlp.experts.linear_fc1.weight3
|
| 886 |
-
module.decoder.layers.8.mlp.experts.linear_fc1.weight5
|
| 887 |
module.decoder.layers.8.mlp.experts.linear_fc1.weight4
|
|
|
|
| 888 |
module.decoder.layers.8.mlp.experts.linear_fc1.weight6
|
| 889 |
Params for bucket 24 (50331648 elements, 50331648 padded size):
|
| 890 |
module.decoder.layers.8.mlp.experts.linear_fc1.weight7
|
|
|
|
| 891 |
module.decoder.layers.8.mlp.experts.linear_fc1.weight8
|
| 892 |
module.decoder.layers.8.mlp.experts.linear_fc1.weight9
|
| 893 |
-
module.decoder.layers.8.mlp.experts.linear_fc1.weight10
|
| 894 |
Params for bucket 25 (50331648 elements, 50331648 padded size):
|
| 895 |
module.decoder.layers.8.mlp.experts.linear_fc1.weight11
|
| 896 |
module.decoder.layers.8.mlp.experts.linear_fc1.weight14
|
| 897 |
-
module.decoder.layers.8.mlp.experts.linear_fc1.weight12
|
| 898 |
module.decoder.layers.8.mlp.experts.linear_fc1.weight13
|
|
|
|
| 899 |
Params for bucket 26 (44040192 elements, 44040192 padded size):
|
|
|
|
|
|
|
| 900 |
module.decoder.layers.8.mlp.experts.linear_fc1.weight15
|
| 901 |
module.decoder.layers.7.mlp.experts.linear_fc2.weight2
|
| 902 |
module.decoder.layers.7.mlp.experts.linear_fc2.weight0
|
| 903 |
module.decoder.layers.7.mlp.experts.linear_fc2.weight4
|
| 904 |
-
module.decoder.layers.7.mlp.experts.linear_fc2.weight3
|
| 905 |
-
module.decoder.layers.7.mlp.experts.linear_fc2.weight1
|
| 906 |
Params for bucket 27 (44040192 elements, 44040192 padded size):
|
| 907 |
-
module.decoder.layers.7.mlp.experts.linear_fc2.weight8
|
| 908 |
-
module.decoder.layers.7.mlp.experts.linear_fc2.weight6
|
| 909 |
-
module.decoder.layers.7.mlp.experts.linear_fc2.weight10
|
| 910 |
module.decoder.layers.7.mlp.experts.linear_fc2.weight5
|
| 911 |
-
module.decoder.layers.7.mlp.experts.linear_fc2.weight7
|
| 912 |
module.decoder.layers.7.mlp.experts.linear_fc2.weight9
|
|
|
|
| 913 |
module.decoder.layers.7.mlp.experts.linear_fc2.weight11
|
|
|
|
|
|
|
|
|
|
| 914 |
Params for bucket 28 (50331648 elements, 50331648 padded size):
|
| 915 |
-
module.decoder.layers.7.mlp.experts.linear_fc2.weight12
|
| 916 |
module.decoder.layers.7.mlp.experts.linear_fc1.weight0
|
| 917 |
-
module.decoder.layers.7.mlp.experts.linear_fc2.weight14
|
| 918 |
module.decoder.layers.7.mlp.experts.linear_fc2.weight15
|
| 919 |
module.decoder.layers.7.mlp.experts.linear_fc2.weight13
|
|
|
|
|
|
|
| 920 |
module.decoder.layers.7.mlp.experts.linear_fc1.weight1
|
| 921 |
Params for bucket 29 (50331648 elements, 50331648 padded size):
|
| 922 |
module.decoder.layers.7.mlp.experts.linear_fc1.weight2
|
| 923 |
-
module.decoder.layers.7.mlp.experts.linear_fc1.weight5
|
| 924 |
module.decoder.layers.7.mlp.experts.linear_fc1.weight3
|
|
|
|
| 925 |
module.decoder.layers.7.mlp.experts.linear_fc1.weight4
|
| 926 |
Params for bucket 30 (50331648 elements, 50331648 padded size):
|
| 927 |
module.decoder.layers.7.mlp.experts.linear_fc1.weight6
|
| 928 |
-
module.decoder.layers.7.mlp.experts.linear_fc1.weight8
|
| 929 |
module.decoder.layers.7.mlp.experts.linear_fc1.weight9
|
|
|
|
| 930 |
module.decoder.layers.7.mlp.experts.linear_fc1.weight7
|
| 931 |
Params for bucket 31 (50331648 elements, 50331648 padded size):
|
| 932 |
module.decoder.layers.7.mlp.experts.linear_fc1.weight10
|
| 933 |
-
module.decoder.layers.7.mlp.experts.linear_fc1.weight13
|
| 934 |
module.decoder.layers.7.mlp.experts.linear_fc1.weight12
|
|
|
|
| 935 |
module.decoder.layers.7.mlp.experts.linear_fc1.weight11
|
| 936 |
Params for bucket 32 (44040192 elements, 44040192 padded size):
|
| 937 |
module.decoder.layers.7.mlp.experts.linear_fc1.weight14
|
|
@@ -944,37 +944,37 @@ Params for bucket 33 (44040192 elements, 44040192 padded size):
|
|
| 944 |
module.decoder.layers.6.mlp.experts.linear_fc2.weight6
|
| 945 |
module.decoder.layers.6.mlp.experts.linear_fc2.weight8
|
| 946 |
module.decoder.layers.6.mlp.experts.linear_fc2.weight3
|
| 947 |
-
module.decoder.layers.6.mlp.experts.linear_fc2.weight7
|
| 948 |
module.decoder.layers.6.mlp.experts.linear_fc2.weight5
|
|
|
|
| 949 |
module.decoder.layers.6.mlp.experts.linear_fc2.weight9
|
| 950 |
Params for bucket 34 (50331648 elements, 50331648 padded size):
|
| 951 |
module.decoder.layers.6.mlp.experts.linear_fc2.weight10
|
| 952 |
-
module.decoder.layers.6.mlp.experts.linear_fc2.weight12
|
| 953 |
module.decoder.layers.6.mlp.experts.linear_fc2.weight14
|
|
|
|
| 954 |
module.decoder.layers.6.mlp.experts.linear_fc2.weight11
|
| 955 |
module.decoder.layers.6.mlp.experts.linear_fc2.weight13
|
| 956 |
module.decoder.layers.6.mlp.experts.linear_fc2.weight15
|
| 957 |
module.decoder.layers.6.mlp.experts.linear_fc1.weight0
|
| 958 |
Params for bucket 35 (50331648 elements, 50331648 padded size):
|
| 959 |
module.decoder.layers.6.mlp.experts.linear_fc1.weight1
|
|
|
|
| 960 |
module.decoder.layers.6.mlp.experts.linear_fc1.weight3
|
| 961 |
module.decoder.layers.6.mlp.experts.linear_fc1.weight4
|
| 962 |
-
module.decoder.layers.6.mlp.experts.linear_fc1.weight2
|
| 963 |
Params for bucket 36 (50331648 elements, 50331648 padded size):
|
| 964 |
module.decoder.layers.6.mlp.experts.linear_fc1.weight5
|
| 965 |
-
module.decoder.layers.6.mlp.experts.linear_fc1.weight7
|
| 966 |
module.decoder.layers.6.mlp.experts.linear_fc1.weight8
|
| 967 |
module.decoder.layers.6.mlp.experts.linear_fc1.weight6
|
|
|
|
| 968 |
Params for bucket 37 (50331648 elements, 50331648 padded size):
|
| 969 |
module.decoder.layers.6.mlp.experts.linear_fc1.weight9
|
| 970 |
-
module.decoder.layers.6.mlp.experts.linear_fc1.weight10
|
| 971 |
-
module.decoder.layers.6.mlp.experts.linear_fc1.weight11
|
| 972 |
module.decoder.layers.6.mlp.experts.linear_fc1.weight12
|
|
|
|
|
|
|
| 973 |
Params for bucket 38 (44040192 elements, 44040192 padded size):
|
| 974 |
module.decoder.layers.6.mlp.experts.linear_fc1.weight13
|
|
|
|
| 975 |
module.decoder.layers.5.mlp.experts.linear_fc2.weight0
|
| 976 |
module.decoder.layers.6.mlp.experts.linear_fc1.weight14
|
| 977 |
-
module.decoder.layers.6.mlp.experts.linear_fc1.weight15
|
| 978 |
Params for bucket 39 (44040192 elements, 44040192 padded size):
|
| 979 |
module.decoder.layers.5.mlp.experts.linear_fc2.weight1
|
| 980 |
module.decoder.layers.5.mlp.experts.linear_fc2.weight3
|
|
@@ -1003,78 +1003,78 @@ Params for bucket 42 (50331648 elements, 50331648 padded size):
|
|
| 1003 |
module.decoder.layers.5.mlp.experts.linear_fc1.weight5
|
| 1004 |
Params for bucket 43 (50331648 elements, 50331648 padded size):
|
| 1005 |
module.decoder.layers.5.mlp.experts.linear_fc1.weight7
|
|
|
|
| 1006 |
module.decoder.layers.5.mlp.experts.linear_fc1.weight10
|
| 1007 |
module.decoder.layers.5.mlp.experts.linear_fc1.weight8
|
| 1008 |
-
module.decoder.layers.5.mlp.experts.linear_fc1.weight9
|
| 1009 |
Params for bucket 44 (50331648 elements, 50331648 padded size):
|
| 1010 |
module.decoder.layers.5.mlp.experts.linear_fc1.weight11
|
|
|
|
| 1011 |
module.decoder.layers.5.mlp.experts.linear_fc1.weight13
|
| 1012 |
module.decoder.layers.5.mlp.experts.linear_fc1.weight14
|
| 1013 |
-
module.decoder.layers.5.mlp.experts.linear_fc1.weight12
|
| 1014 |
Params for bucket 45 (44040192 elements, 44040192 padded size):
|
| 1015 |
-
module.decoder.layers.4.mlp.experts.linear_fc2.weight2
|
| 1016 |
-
module.decoder.layers.4.mlp.experts.linear_fc2.weight0
|
| 1017 |
-
module.decoder.layers.4.mlp.experts.linear_fc2.weight4
|
| 1018 |
-
module.decoder.layers.5.mlp.experts.linear_fc1.weight15
|
| 1019 |
module.decoder.layers.4.mlp.experts.linear_fc2.weight3
|
| 1020 |
module.decoder.layers.4.mlp.experts.linear_fc2.weight1
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1021 |
Params for bucket 46 (44040192 elements, 44040192 padded size):
|
| 1022 |
-
module.decoder.layers.4.mlp.experts.linear_fc2.weight8
|
| 1023 |
-
module.decoder.layers.4.mlp.experts.linear_fc2.weight6
|
| 1024 |
-
module.decoder.layers.4.mlp.experts.linear_fc2.weight10
|
| 1025 |
module.decoder.layers.4.mlp.experts.linear_fc2.weight5
|
| 1026 |
-
module.decoder.layers.4.mlp.experts.linear_fc2.weight9
|
| 1027 |
module.decoder.layers.4.mlp.experts.linear_fc2.weight7
|
|
|
|
| 1028 |
module.decoder.layers.4.mlp.experts.linear_fc2.weight11
|
|
|
|
|
|
|
|
|
|
| 1029 |
Params for bucket 47 (50331648 elements, 50331648 padded size):
|
|
|
|
|
|
|
|
|
|
| 1030 |
module.decoder.layers.4.mlp.experts.linear_fc2.weight12
|
| 1031 |
module.decoder.layers.4.mlp.experts.linear_fc2.weight14
|
| 1032 |
module.decoder.layers.4.mlp.experts.linear_fc1.weight1
|
| 1033 |
-
module.decoder.layers.4.mlp.experts.linear_fc1.weight0
|
| 1034 |
-
module.decoder.layers.4.mlp.experts.linear_fc2.weight15
|
| 1035 |
-
module.decoder.layers.4.mlp.experts.linear_fc2.weight13
|
| 1036 |
Params for bucket 48 (50331648 elements, 50331648 padded size):
|
| 1037 |
module.decoder.layers.4.mlp.experts.linear_fc1.weight2
|
|
|
|
| 1038 |
module.decoder.layers.4.mlp.experts.linear_fc1.weight4
|
| 1039 |
module.decoder.layers.4.mlp.experts.linear_fc1.weight5
|
| 1040 |
-
module.decoder.layers.4.mlp.experts.linear_fc1.weight3
|
| 1041 |
Params for bucket 49 (50331648 elements, 50331648 padded size):
|
| 1042 |
module.decoder.layers.4.mlp.experts.linear_fc1.weight6
|
|
|
|
| 1043 |
module.decoder.layers.4.mlp.experts.linear_fc1.weight7
|
| 1044 |
module.decoder.layers.4.mlp.experts.linear_fc1.weight8
|
| 1045 |
-
module.decoder.layers.4.mlp.experts.linear_fc1.weight9
|
| 1046 |
Params for bucket 50 (50331648 elements, 50331648 padded size):
|
| 1047 |
module.decoder.layers.4.mlp.experts.linear_fc1.weight10
|
| 1048 |
-
module.decoder.layers.4.mlp.experts.linear_fc1.weight11
|
| 1049 |
module.decoder.layers.4.mlp.experts.linear_fc1.weight12
|
| 1050 |
module.decoder.layers.4.mlp.experts.linear_fc1.weight13
|
|
|
|
| 1051 |
Params for bucket 51 (44040192 elements, 44040192 padded size):
|
| 1052 |
-
module.decoder.layers.3.mlp.experts.linear_fc2.weight1
|
| 1053 |
-
module.decoder.layers.4.mlp.experts.linear_fc1.weight15
|
| 1054 |
module.decoder.layers.4.mlp.experts.linear_fc1.weight14
|
| 1055 |
-
module.decoder.layers.3.mlp.experts.linear_fc2.weight2
|
| 1056 |
module.decoder.layers.3.mlp.experts.linear_fc2.weight0
|
|
|
|
|
|
|
|
|
|
| 1057 |
Params for bucket 52 (44040192 elements, 44040192 padded size):
|
|
|
|
|
|
|
|
|
|
| 1058 |
module.decoder.layers.3.mlp.experts.linear_fc2.weight3
|
| 1059 |
module.decoder.layers.3.mlp.experts.linear_fc2.weight7
|
| 1060 |
module.decoder.layers.3.mlp.experts.linear_fc2.weight5
|
| 1061 |
module.decoder.layers.3.mlp.experts.linear_fc2.weight9
|
| 1062 |
-
module.decoder.layers.3.mlp.experts.linear_fc2.weight6
|
| 1063 |
-
module.decoder.layers.3.mlp.experts.linear_fc2.weight4
|
| 1064 |
-
module.decoder.layers.3.mlp.experts.linear_fc2.weight8
|
| 1065 |
Params for bucket 53 (50331648 elements, 50331648 padded size):
|
|
|
|
|
|
|
|
|
|
| 1066 |
module.decoder.layers.3.mlp.experts.linear_fc2.weight13
|
| 1067 |
module.decoder.layers.3.mlp.experts.linear_fc2.weight11
|
| 1068 |
module.decoder.layers.3.mlp.experts.linear_fc2.weight15
|
| 1069 |
module.decoder.layers.3.mlp.experts.linear_fc1.weight0
|
| 1070 |
-
module.decoder.layers.3.mlp.experts.linear_fc2.weight10
|
| 1071 |
-
module.decoder.layers.3.mlp.experts.linear_fc2.weight14
|
| 1072 |
-
module.decoder.layers.3.mlp.experts.linear_fc2.weight12
|
| 1073 |
Params for bucket 54 (50331648 elements, 50331648 padded size):
|
| 1074 |
module.decoder.layers.3.mlp.experts.linear_fc1.weight1
|
| 1075 |
module.decoder.layers.3.mlp.experts.linear_fc1.weight4
|
| 1076 |
-
module.decoder.layers.3.mlp.experts.linear_fc1.weight2
|
| 1077 |
module.decoder.layers.3.mlp.experts.linear_fc1.weight3
|
|
|
|
| 1078 |
Params for bucket 55 (50331648 elements, 50331648 padded size):
|
| 1079 |
module.decoder.layers.3.mlp.experts.linear_fc1.weight5
|
| 1080 |
module.decoder.layers.3.mlp.experts.linear_fc1.weight7
|
|
@@ -1087,126 +1087,126 @@ Params for bucket 56 (50331648 elements, 50331648 padded size):
|
|
| 1087 |
module.decoder.layers.3.mlp.experts.linear_fc1.weight12
|
| 1088 |
Params for bucket 57 (44040192 elements, 44040192 padded size):
|
| 1089 |
module.decoder.layers.3.mlp.experts.linear_fc1.weight13
|
| 1090 |
-
module.decoder.layers.3.mlp.experts.linear_fc1.weight14
|
| 1091 |
module.decoder.layers.2.mlp.experts.linear_fc2.weight0
|
|
|
|
| 1092 |
module.decoder.layers.3.mlp.experts.linear_fc1.weight15
|
| 1093 |
Params for bucket 58 (44040192 elements, 44040192 padded size):
|
| 1094 |
-
module.decoder.layers.2.mlp.experts.linear_fc2.weight1
|
| 1095 |
-
module.decoder.layers.2.mlp.experts.linear_fc2.weight3
|
| 1096 |
-
module.decoder.layers.2.mlp.experts.linear_fc2.weight5
|
| 1097 |
-
module.decoder.layers.2.mlp.experts.linear_fc2.weight7
|
| 1098 |
module.decoder.layers.2.mlp.experts.linear_fc2.weight2
|
| 1099 |
module.decoder.layers.2.mlp.experts.linear_fc2.weight4
|
| 1100 |
module.decoder.layers.2.mlp.experts.linear_fc2.weight6
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1101 |
Params for bucket 59 (44040192 elements, 44040192 padded size):
|
| 1102 |
-
module.decoder.layers.2.mlp.experts.linear_fc2.weight11
|
| 1103 |
-
module.decoder.layers.2.mlp.experts.linear_fc2.weight9
|
| 1104 |
-
module.decoder.layers.2.mlp.experts.linear_fc2.weight13
|
| 1105 |
module.decoder.layers.2.mlp.experts.linear_fc2.weight8
|
| 1106 |
module.decoder.layers.2.mlp.experts.linear_fc2.weight12
|
| 1107 |
module.decoder.layers.2.mlp.experts.linear_fc2.weight10
|
| 1108 |
module.decoder.layers.2.mlp.experts.linear_fc2.weight14
|
|
|
|
|
|
|
|
|
|
| 1109 |
Params for bucket 60 (44040192 elements, 44040192 padded size):
|
| 1110 |
module.decoder.layers.2.mlp.experts.linear_fc2.weight15
|
|
|
|
| 1111 |
module.decoder.layers.2.mlp.experts.linear_fc1.weight2
|
| 1112 |
module.decoder.layers.2.mlp.experts.linear_fc1.weight0
|
| 1113 |
-
module.decoder.layers.2.mlp.experts.linear_fc1.weight1
|
| 1114 |
Params for bucket 61 (50331648 elements, 50331648 padded size):
|
| 1115 |
module.decoder.layers.2.mlp.experts.linear_fc1.weight3
|
|
|
|
| 1116 |
module.decoder.layers.2.mlp.experts.linear_fc1.weight5
|
| 1117 |
module.decoder.layers.2.mlp.experts.linear_fc1.weight6
|
| 1118 |
-
module.decoder.layers.2.mlp.experts.linear_fc1.weight4
|
| 1119 |
Params for bucket 62 (50331648 elements, 50331648 padded size):
|
| 1120 |
module.decoder.layers.2.mlp.experts.linear_fc1.weight7
|
| 1121 |
-
module.decoder.layers.2.mlp.experts.linear_fc1.weight9
|
| 1122 |
module.decoder.layers.2.mlp.experts.linear_fc1.weight10
|
| 1123 |
module.decoder.layers.2.mlp.experts.linear_fc1.weight8
|
|
|
|
| 1124 |
Params for bucket 63 (50331648 elements, 50331648 padded size):
|
| 1125 |
module.decoder.layers.2.mlp.experts.linear_fc1.weight11
|
|
|
|
| 1126 |
module.decoder.layers.2.mlp.experts.linear_fc1.weight12
|
| 1127 |
module.decoder.layers.2.mlp.experts.linear_fc1.weight13
|
| 1128 |
-
module.decoder.layers.2.mlp.experts.linear_fc1.weight14
|
| 1129 |
Params for bucket 64 (44040192 elements, 44040192 padded size):
|
| 1130 |
-
module.decoder.layers.1.mlp.experts.linear_fc2.weight2
|
| 1131 |
module.decoder.layers.1.mlp.experts.linear_fc2.weight0
|
|
|
|
| 1132 |
module.decoder.layers.1.mlp.experts.linear_fc2.weight4
|
| 1133 |
module.decoder.layers.2.mlp.experts.linear_fc1.weight15
|
| 1134 |
-
module.decoder.layers.1.mlp.experts.linear_fc2.weight1
|
| 1135 |
module.decoder.layers.1.mlp.experts.linear_fc2.weight3
|
|
|
|
| 1136 |
Params for bucket 65 (44040192 elements, 44040192 padded size):
|
| 1137 |
-
module.decoder.layers.1.mlp.experts.linear_fc2.weight8
|
| 1138 |
module.decoder.layers.1.mlp.experts.linear_fc2.weight6
|
|
|
|
| 1139 |
module.decoder.layers.1.mlp.experts.linear_fc2.weight10
|
| 1140 |
module.decoder.layers.1.mlp.experts.linear_fc2.weight5
|
| 1141 |
-
module.decoder.layers.1.mlp.experts.linear_fc2.weight9
|
| 1142 |
module.decoder.layers.1.mlp.experts.linear_fc2.weight7
|
|
|
|
| 1143 |
module.decoder.layers.1.mlp.experts.linear_fc2.weight11
|
| 1144 |
Params for bucket 66 (50331648 elements, 50331648 padded size):
|
| 1145 |
module.decoder.layers.1.mlp.experts.linear_fc2.weight12
|
| 1146 |
-
module.decoder.layers.1.mlp.experts.linear_fc2.weight14
|
| 1147 |
module.decoder.layers.1.mlp.experts.linear_fc1.weight0
|
| 1148 |
-
module.decoder.layers.1.mlp.experts.linear_fc2.
|
| 1149 |
module.decoder.layers.1.mlp.experts.linear_fc2.weight13
|
|
|
|
| 1150 |
module.decoder.layers.1.mlp.experts.linear_fc1.weight1
|
| 1151 |
Params for bucket 67 (50331648 elements, 50331648 padded size):
|
| 1152 |
module.decoder.layers.1.mlp.experts.linear_fc1.weight2
|
|
|
|
| 1153 |
module.decoder.layers.1.mlp.experts.linear_fc1.weight3
|
| 1154 |
module.decoder.layers.1.mlp.experts.linear_fc1.weight4
|
| 1155 |
-
module.decoder.layers.1.mlp.experts.linear_fc1.weight5
|
| 1156 |
Params for bucket 68 (50331648 elements, 50331648 padded size):
|
| 1157 |
module.decoder.layers.1.mlp.experts.linear_fc1.weight6
|
|
|
|
| 1158 |
module.decoder.layers.1.mlp.experts.linear_fc1.weight9
|
| 1159 |
module.decoder.layers.1.mlp.experts.linear_fc1.weight7
|
| 1160 |
-
module.decoder.layers.1.mlp.experts.linear_fc1.weight8
|
| 1161 |
Params for bucket 69 (50331648 elements, 50331648 padded size):
|
| 1162 |
module.decoder.layers.1.mlp.experts.linear_fc1.weight10
|
| 1163 |
module.decoder.layers.1.mlp.experts.linear_fc1.weight12
|
| 1164 |
-
module.decoder.layers.1.mlp.experts.linear_fc1.weight13
|
| 1165 |
module.decoder.layers.1.mlp.experts.linear_fc1.weight11
|
|
|
|
| 1166 |
Params for bucket 70 (44040192 elements, 44040192 padded size):
|
| 1167 |
module.decoder.layers.1.mlp.experts.linear_fc1.weight14
|
| 1168 |
-
module.decoder.layers.0.mlp.experts.linear_fc2.weight0
|
| 1169 |
module.decoder.layers.0.mlp.experts.linear_fc2.weight2
|
| 1170 |
-
module.decoder.layers.0.mlp.experts.linear_fc2.
|
| 1171 |
module.decoder.layers.1.mlp.experts.linear_fc1.weight15
|
|
|
|
| 1172 |
Params for bucket 71 (44040192 elements, 44040192 padded size):
|
| 1173 |
module.decoder.layers.0.mlp.experts.linear_fc2.weight4
|
| 1174 |
module.decoder.layers.0.mlp.experts.linear_fc2.weight6
|
| 1175 |
module.decoder.layers.0.mlp.experts.linear_fc2.weight8
|
| 1176 |
module.decoder.layers.0.mlp.experts.linear_fc2.weight3
|
| 1177 |
-
module.decoder.layers.0.mlp.experts.linear_fc2.weight7
|
| 1178 |
module.decoder.layers.0.mlp.experts.linear_fc2.weight5
|
|
|
|
| 1179 |
module.decoder.layers.0.mlp.experts.linear_fc2.weight9
|
| 1180 |
Params for bucket 72 (50331648 elements, 50331648 padded size):
|
| 1181 |
module.decoder.layers.0.mlp.experts.linear_fc2.weight10
|
| 1182 |
module.decoder.layers.0.mlp.experts.linear_fc2.weight14
|
| 1183 |
module.decoder.layers.0.mlp.experts.linear_fc2.weight12
|
| 1184 |
-
module.decoder.layers.0.mlp.experts.linear_fc1.weight0
|
| 1185 |
module.decoder.layers.0.mlp.experts.linear_fc2.weight11
|
| 1186 |
module.decoder.layers.0.mlp.experts.linear_fc2.weight13
|
| 1187 |
module.decoder.layers.0.mlp.experts.linear_fc2.weight15
|
|
|
|
| 1188 |
Params for bucket 73 (50331648 elements, 50331648 padded size):
|
| 1189 |
module.decoder.layers.0.mlp.experts.linear_fc1.weight1
|
|
|
|
| 1190 |
module.decoder.layers.0.mlp.experts.linear_fc1.weight3
|
| 1191 |
module.decoder.layers.0.mlp.experts.linear_fc1.weight4
|
| 1192 |
-
module.decoder.layers.0.mlp.experts.linear_fc1.weight2
|
| 1193 |
Params for bucket 74 (50331648 elements, 50331648 padded size):
|
| 1194 |
module.decoder.layers.0.mlp.experts.linear_fc1.weight5
|
| 1195 |
module.decoder.layers.0.mlp.experts.linear_fc1.weight6
|
| 1196 |
-
module.decoder.layers.0.mlp.experts.linear_fc1.weight7
|
| 1197 |
module.decoder.layers.0.mlp.experts.linear_fc1.weight8
|
|
|
|
| 1198 |
Params for bucket 75 (50331648 elements, 50331648 padded size):
|
| 1199 |
module.decoder.layers.0.mlp.experts.linear_fc1.weight9
|
| 1200 |
-
module.decoder.layers.0.mlp.experts.linear_fc1.weight12
|
| 1201 |
module.decoder.layers.0.mlp.experts.linear_fc1.weight10
|
| 1202 |
module.decoder.layers.0.mlp.experts.linear_fc1.weight11
|
|
|
|
| 1203 |
Params for bucket 76 (37748736 elements, 37748736 padded size):
|
| 1204 |
module.decoder.layers.0.mlp.experts.linear_fc1.weight13
|
| 1205 |
module.decoder.layers.0.mlp.experts.linear_fc1.weight14
|
| 1206 |
module.decoder.layers.0.mlp.experts.linear_fc1.weight15
|
| 1207 |
-
INFO:megatron.core.optimizer:Setting up optimizer with config OptimizerConfig(optimizer='adam', lr=0.0001, min_lr=1e-05, decoupled_lr=None, decoupled_min_lr=None, weight_decay=0.1, fp8_recipe='mxfp8', fp16=False, bf16=True, reuse_grad_buf_for_mxfp8_param_ag=False, params_dtype=torch.bfloat16, use_precision_aware_optimizer=False, store_param_remainders=True, main_grads_dtype=torch.float32, main_params_dtype=torch.float32, exp_avg_dtype=torch.float32, exp_avg_sq_dtype=torch.float32, loss_scale=None, initial_loss_scale=4294967296, min_loss_scale=1.0, loss_scale_window=1000, hysteresis=2, adam_beta1=0.9, adam_beta2=0.999, adam_eps=1e-08, sgd_momentum=0.9, use_distributed_optimizer=True, overlap_param_gather_with_optimizer_step=False, optimizer_cpu_offload=False, optimizer_offload_fraction=1.0, use_torch_optimizer_for_cpu_offload=False, overlap_cpu_optimizer_d2h_h2d=False, pin_cpu_grads=True, pin_cpu_params=True, clip_grad=1.0, log_num_zeros_in_grad=False, barrier_with_L1_time=True, timers=<megatron.core.timers.Timers object at
|
| 1208 |
INFO:megatron.core.optimizer_param_scheduler:> learning rate decay style: cosine
|
| 1209 |
-
[after model, optimizer, and learning rate scheduler are built] datetime: 2026-08-
|
| 1210 |
> building train, validation, and test datasets ...
|
| 1211 |
> datasets target sizes (minimum size):
|
| 1212 |
train: 96
|
|
@@ -1214,7 +1214,7 @@ INFO:megatron.core.optimizer_param_scheduler:> learning rate decay style: cosine
|
|
| 1214 |
test: 0
|
| 1215 |
INFO:megatron.core.datasets.blended_megatron_dataset_config:Let split_matrix = [(0, 1.0), None, None]
|
| 1216 |
> building train, validation, and test datasets for GPT ...
|
| 1217 |
-
INFO:megatron.core.datasets.blended_megatron_dataset_builder:Building GPTDataset splits with sizes=(96, 0, 0) and config=GPTDatasetConfig(random_seed=1234, sequence_length=8192, blend=(['/workspace/data/wikitext/wikitext103_gpt2_text_document'], None), blend_per_split=None, split='100,0,0', split_matrix=[(0, 1.0), None, None], num_dataset_builder_threads=1, path_to_cache=None, mmap_bin_files=True, mock=False, tokenizer=<megatron.training.tokenizer.tokenizer._GPT2BPETokenizer object at
|
| 1218 |
INFO:megatron.core.datasets.indexed_dataset:Load the _IndexReader from /workspace/data/wikitext/wikitext103_gpt2_text_document.idx
|
| 1219 |
INFO:megatron.core.datasets.indexed_dataset: Extract the sequence lengths
|
| 1220 |
INFO:megatron.core.datasets.indexed_dataset: Extract the sequence pointers
|
|
@@ -1227,14 +1227,14 @@ INFO:megatron.core.datasets.gpt_dataset: Load the sample index from eadc3756447e
|
|
| 1227 |
INFO:megatron.core.datasets.gpt_dataset: Load the shuffle index from eadc3756447e39d609e8aa728a91f451-GPTDataset-train-shuffle_index.npy
|
| 1228 |
INFO:megatron.core.datasets.gpt_dataset:> total number of samples: 14536
|
| 1229 |
> finished creating GPT datasets ...
|
| 1230 |
-
[after dataloaders are built] datetime: 2026-08-
|
| 1231 |
done with setup ...
|
| 1232 |
(min, max) time across ranks (ms):
|
| 1233 |
-
model-and-optimizer-setup ......................: (
|
| 1234 |
-
train/valid/test-data-iterators-setup ..........: (
|
| 1235 |
training ...
|
| 1236 |
Overwriting rerun_state_machine.current_iteration from -1 to 0...
|
| 1237 |
-
[before the start of training step] datetime: 2026-08-
|
| 1238 |
INFO:moonep.api:MoonEP comm buffer sizes:
|
| 1239 |
config : R=8, S=8192, H=4096, E=128, K=8, NvS=69600, NvS_padded=69632
|
| 1240 |
sm_count : num_sms=32, num_sms_dedup=148
|
|
@@ -1246,44 +1246,42 @@ INFO:moonep.api:MoonEP comm buffer sizes:
|
|
| 1246 |
[MoonEP selfcheck 2/4] dropped groups empty OK | covered rows [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 128, 129, 130, 131, 132, 133, 134, 135, 136, 137, 138, 139, 140, 141, 142, 143] sizes [9344, 2688, 128, 24192, 128, 0, 128, 256, 128, 768, 10496, 128, 128, 2432, 128, 1536, 14208, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0] | duplicated experts [[69, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1], [88, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1], [55, 52, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1], [-1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1], [63, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1], [124, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1], [10, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1], [58, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1]]
|
| 1247 |
[MoonEP selfcheck 3/4] dropped groups empty OK | covered rows [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 128, 129, 130, 131, 132, 133, 134, 135, 136, 137, 138, 139, 140, 141, 142, 143] sizes [128, 128, 128, 0, 128, 44416, 128, 128, 128, 6912, 128, 256, 128, 128, 128, 128, 13824, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0] | duplicated experts [[78, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1], [111, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1], [62, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1], [-1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1], [57, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1], [106, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1], [121, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1], [57, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1]]
|
| 1248 |
[MoonEP selfcheck 4/4] dropped groups empty OK | covered rows [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 128, 129, 130, 131, 132, 133, 134, 135, 136, 137, 138, 139, 140, 141, 142, 143] sizes [2304, 128, 128, 512, 128, 12416, 128, 1152, 1280, 16768, 640, 29056, 128, 128, 128, 1664, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0] | duplicated experts [[-1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1], [118, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1], [11, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1], [118, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1], [39, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1], [11, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1], [73, 79, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1], [86, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1]]
|
| 1249 |
-
[2026-08-
|
| 1250 |
Number of parameters in transformer block in billions: 29.85
|
| 1251 |
Number of parameters in embedding layers in billions: 0.41
|
| 1252 |
Total number of parameters in billions: 30.26
|
| 1253 |
Number of parameters in most loaded shard in billions: 30.2590
|
| 1254 |
Theoretical memory footprints: weight and optimizer=216428.96 MB
|
| 1255 |
[Rank 0] (after 1 iterations) memory (MB) | allocated: 67239.548828125 | max allocated: 67239.54931640625 | reserved: 68548.0 | max reserved: 68548.0
|
| 1256 |
-
[2026-08-
|
| 1257 |
-
[2026-08-
|
| 1258 |
Capture CUDA graph for training!!!
|
| 1259 |
CUDA graph capture done!!!
|
| 1260 |
-
[2026-08-
|
| 1261 |
-
[2026-08-
|
| 1262 |
-
[2026-08-
|
| 1263 |
-
[2026-08-
|
| 1264 |
-
[2026-08-
|
| 1265 |
-
[2026-08-
|
| 1266 |
-
[2026-08-
|
| 1267 |
-
[2026-08-
|
| 1268 |
-
[2026-08-
|
| 1269 |
-
[after training is done] datetime: 2026-08-
|
| 1270 |
-
[rank0]:[
|
| 1271 |
-
[
|
| 1272 |
-
[
|
| 1273 |
-
[
|
| 1274 |
-
[
|
| 1275 |
-
[
|
| 1276 |
-
[
|
| 1277 |
-
[
|
| 1278 |
-
|
| 1279 |
-
|
| 1280 |
-
|
| 1281 |
-
|
| 1282 |
-
|
| 1283 |
-
|
| 1284 |
-
|
| 1285 |
-
W0829 23:47:18.888000 240032 torch/distributed/elastic/multiprocessing/api.py:900] Sending process 240222 closing signal SIGTERM
|
| 1286 |
-
W0829 23:47:18.888000 240032 torch/distributed/elastic/multiprocessing/api.py:900] Sending process 240223 closing signal SIGTERM
|
| 1287 |
Traceback (most recent call last):
|
| 1288 |
File "/usr/local/bin/torchrun", line 8, in <module>
|
| 1289 |
sys.exit(main())
|
|
@@ -1311,4 +1309,4 @@ Traceback (most recent call last):
|
|
| 1311 |
time.sleep(monitor_interval)
|
| 1312 |
File "/usr/local/lib/python3.12/dist-packages/torch/distributed/elastic/multiprocessing/api.py", line 84, in _terminate_process_handler
|
| 1313 |
raise SignalException(f"Process {os.getpid()} got signal: {sigval}", sigval=sigval)
|
| 1314 |
-
torch.distributed.elastic.multiprocessing.api.SignalException: Process
|
|
|
|
| 1 |
+
[01:32:32] WARN: moonep_backend_async_grad_reduce lacks --moe-router-fusion; QWEN3_235B runs unfused (timing only, routing identical)
|
| 2 |
==============================================================
|
| 3 |
QWEN3_235B / moonep (slots/rank=16 policy=matched)
|
| 4 |
E=128 K=8 H'=1536 m=16
|
| 5 |
L=12 seq=8192 gbs=8 (1 microbatches)
|
| 6 |
cap_mode=none cap=none cg=1 recompute=none
|
| 7 |
memory_dump=none snapshots=/workspace/tierb-fixed-moonep-full/moonep/tierb/tierb_gbs8/QWEN3_235B__moonep__matched__seq8192__gbs8__capnone__async/rep2/memory_snapshots/snapshot.pickle
|
| 8 |
+
repo=/workspace/Megatron-MoE-Echo @ 2e68c8ad
|
| 9 |
==============================================================
|
| 10 |
+
W0830 01:32:33.428000 382773 torch/distributed/run.py:766]
|
| 11 |
+
W0830 01:32:33.428000 382773 torch/distributed/run.py:766] *****************************************
|
| 12 |
+
W0830 01:32:33.428000 382773 torch/distributed/run.py:766] Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed.
|
| 13 |
+
W0830 01:32:33.428000 382773 torch/distributed/run.py:766] *****************************************
|
| 14 |
/workspace/Megatron-MoE-Echo/megatron/training/arguments.py:831: UserWarning: Disabling sequence parallelism because tensor model parallelism is disabled
|
| 15 |
warnings.warn("Disabling sequence parallelism because tensor model parallelism is disabled")
|
|
|
|
|
|
|
|
|
|
| 16 |
/workspace/Megatron-MoE-Echo/megatron/training/arguments.py:831: UserWarning: Disabling sequence parallelism because tensor model parallelism is disabled
|
| 17 |
warnings.warn("Disabling sequence parallelism because tensor model parallelism is disabled")
|
| 18 |
/workspace/Megatron-MoE-Echo/megatron/training/arguments.py:831: UserWarning: Disabling sequence parallelism because tensor model parallelism is disabled
|
| 19 |
warnings.warn("Disabling sequence parallelism because tensor model parallelism is disabled")
|
| 20 |
+
WARNING: one_logger package is required to enable e2e metrics tracking. please go to https://confluence.nvidia.com/display/MLWFO/Package+Repositories for details to install it
|
| 21 |
/workspace/Megatron-MoE-Echo/megatron/training/arguments.py:831: UserWarning: Disabling sequence parallelism because tensor model parallelism is disabled
|
| 22 |
warnings.warn("Disabling sequence parallelism because tensor model parallelism is disabled")
|
| 23 |
using world size: 8, data-parallel size: 8, context-parallel size: 1, hierarchical context-parallel sizes: None, tensor-model-parallel size: 1, pipeline-model-parallel size: 1
|
|
|
|
| 575 |
warnings.warn("Disabling sequence parallelism because tensor model parallelism is disabled")
|
| 576 |
/workspace/Megatron-MoE-Echo/megatron/training/arguments.py:831: UserWarning: Disabling sequence parallelism because tensor model parallelism is disabled
|
| 577 |
warnings.warn("Disabling sequence parallelism because tensor model parallelism is disabled")
|
| 578 |
+
/workspace/Megatron-MoE-Echo/megatron/training/arguments.py:831: UserWarning: Disabling sequence parallelism because tensor model parallelism is disabled
|
| 579 |
+
warnings.warn("Disabling sequence parallelism because tensor model parallelism is disabled")
|
|
|
|
| 580 |
[Gloo] Rank 4 is connected to 7 peer ranks. Expected number of connected peer ranks is : 7
|
|
|
|
|
|
|
|
|
|
|
|
|
| 581 |
[Gloo] Rank 0 is connected to 7 peer ranks. Expected number of connected peer ranks is : 7
|
| 582 |
+
[Gloo] Rank 1 is connected to 7 peer ranks. Expected number of connected peer ranks is : 7
|
| 583 |
[Gloo] Rank 2 is connected to 7 peer ranks. Expected number of connected peer ranks is : 7
|
| 584 |
+
[Gloo] Rank 3 is connected to 7 peer ranks. Expected number of connected peer ranks is : 7
|
| 585 |
+
[Gloo] Rank 6 is connected to 7 peer ranks. Expected number of connected peer ranks is : 7
|
| 586 |
[Gloo] Rank 5 is connected to 7 peer ranks. Expected number of connected peer ranks is : 7
|
| 587 |
+
[Gloo] Rank 7 is connected to 7 peer ranks. Expected number of connected peer ranks is : 7
|
| 588 |
[Gloo] Rank 3 is connected to 7 peer ranks. Expected number of connected peer ranks is : 7
|
| 589 |
+
[Gloo] Rank 0 is connected to 7 peer ranks. Expected number of connected peer ranks is : 7
|
| 590 |
[Gloo] Rank 1 is connected to 7 peer ranks. Expected number of connected peer ranks is : 7
|
| 591 |
+
[Gloo] Rank 2 is connected to 7 peer ranks. Expected number of connected peer ranks is : 7
|
| 592 |
+
[Gloo] Rank [Gloo] Rank [Gloo] Rank [Gloo] Rank 4567 is connected to is connected to is connected to is connected to 7777 peer ranks. peer ranks. peer ranks. peer ranks. Expected number of connected peer ranks is : Expected number of connected peer ranks is : Expected number of connected peer ranks is : Expected number of connected peer ranks is : 7777
|
| 593 |
+
|
| 594 |
+
|
| 595 |
+
|
| 596 |
[Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
|
| 597 |
> initialized tensor model parallel with size 1
|
| 598 |
> initialized pipeline model parallel with size 1
|
| 599 |
> setting random seeds to 1234 ...
|
| 600 |
+
[Gloo] Rank [Gloo] Rank 0[Gloo] Rank [Gloo] Rank 0 is connected to 00 is connected to 0 is connected to is connected to 0 peer ranks. 00 peer ranks. Expected number of connected peer ranks is : peer ranks. peer ranks. Expected number of connected peer ranks is : 0Expected number of connected peer ranks is :
|
| 601 |
+
Expected number of connected peer ranks is : 000
|
| 602 |
+
|
| 603 |
+
|
| 604 |
[Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
|
| 605 |
[Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
|
| 606 |
[Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
|
| 607 |
+
[rank7]:[W830 01:32:51.378489561 ProcessGroupNCCL.cpp:4789] [PG ID 0 PG GUID 0 Rank 7] using GPU 7 as device used by this process is currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect. You can specify device_id in init_process_group() to force use of a particular device.
|
| 608 |
+
[rank1]:[W830 01:32:51.378526948 ProcessGroupNCCL.cpp:4789] [PG ID 0 PG GUID 0 Rank 1] using GPU 1 as device used by this process is currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect. You can specify device_id in init_process_group() to force use of a particular device.
|
| 609 |
+
[rank4]:[W830 01:32:51.378555820 ProcessGroupNCCL.cpp:4789] [PG ID 0 PG GUID 0 Rank 4] using GPU 4 as device used by this process is currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect. You can specify device_id in init_process_group() to force use of a particular device.
|
| 610 |
+
[rank3]:[W830 01:32:51.378565188 ProcessGroupNCCL.cpp:4789] [PG ID 0 PG GUID 0 Rank 3] using GPU 3 as device used by this process is currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect. You can specify device_id in init_process_group() to force use of a particular device.
|
| 611 |
+
[rank2]:[W830 01:32:51.380630702 ProcessGroupNCCL.cpp:4789] [PG ID 0 PG GUID 0 Rank 2] using GPU 2 as device used by this process is currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect. You can specify device_id in init_process_group() to force use of a particular device.
|
| 612 |
+
[rank5]:[W830 01:32:51.381181551 ProcessGroupNCCL.cpp:4789] [PG ID 0 PG GUID 0 Rank 5] using GPU 5 as device used by this process is currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect. You can specify device_id in init_process_group() to force use of a particular device.
|
| 613 |
+
[rank6]:[W830 01:32:51.381335334 ProcessGroupNCCL.cpp:4789] [PG ID 0 PG GUID 0 Rank 6] using GPU 6 as device used by this process is currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect. You can specify device_id in init_process_group() to force use of a particular device.
|
|
|
|
|
|
|
|
|
|
|
|
|
| 614 |
> compiling dataset index builder ...
|
| 615 |
make: Entering directory '/workspace/Megatron-MoE-Echo/megatron/core/datasets'
|
| 616 |
make: Nothing to be done for 'default'.
|
| 617 |
make: Leaving directory '/workspace/Megatron-MoE-Echo/megatron/core/datasets'
|
| 618 |
+
>>> done with dataset index builder. Compilation time: 0.033 seconds
|
| 619 |
WARNING: constraints for invoking optimized fused softmax kernel are not met. We default back to unfused kernel invocations.
|
| 620 |
> compiling and loading fused kernels ...
|
| 621 |
+
[rank0]:[W830 01:32:51.542543761 ProcessGroupNCCL.cpp:4789] [PG ID 0 PG GUID 0 Rank 0] using GPU 0 as device used by this process is currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect. You can specify device_id in init_process_group() to force use of a particular device.
|
| 622 |
+
>>> done with compiling and loading fused kernels. Compilation time: 4.053 seconds
|
| 623 |
+
time to initialize megatron (seconds): 7.444
|
| 624 |
+
[after megatron is initialized] datetime: 2026-08-30 01:32:57
|
| 625 |
building GPT model ...
|
| 626 |
/usr/local/lib/python3.12/dist-packages/transformer_engine/pytorch/cpu_offload.py:708: DeprecationWarning: Offloading weights is deprecated. Using offload_weights=True does not have any effect.
|
| 627 |
warnings.warn(
|
|
|
|
| 645 |
Params for bucket 1 (206045184 elements, 206045184 padded size):
|
| 646 |
module.output_layer.weight
|
| 647 |
Params for bucket 2 (71840000 elements, 71840000 padded size):
|
| 648 |
+
module.decoder.final_layernorm.weight
|
| 649 |
+
module.decoder.layers.11.mlp.router.weight
|
| 650 |
module.decoder.layers.11.pre_mlp_layernorm.weight
|
| 651 |
module.decoder.layers.11.self_attention.q_layernorm.weight
|
| 652 |
+
module.decoder.layers.11.self_attention.k_layernorm.weight
|
| 653 |
module.decoder.layers.11.self_attention.linear_qkv.weight
|
| 654 |
module.decoder.layers.11.self_attention.linear_qkv.layer_norm_weight
|
|
|
|
|
|
|
| 655 |
module.decoder.layers.11.self_attention.linear_proj.weight
|
| 656 |
Params for bucket 3 (71835904 elements, 71835904 padded size):
|
|
|
|
| 657 |
module.decoder.layers.10.pre_mlp_layernorm.weight
|
| 658 |
module.decoder.layers.10.self_attention.q_layernorm.weight
|
|
|
|
| 659 |
module.decoder.layers.10.self_attention.k_layernorm.weight
|
| 660 |
+
module.decoder.layers.10.self_attention.linear_qkv.weight
|
| 661 |
module.decoder.layers.10.self_attention.linear_qkv.layer_norm_weight
|
| 662 |
+
module.decoder.layers.10.mlp.router.weight
|
| 663 |
module.decoder.layers.10.self_attention.linear_proj.weight
|
| 664 |
Params for bucket 4 (71835904 elements, 71835904 padded size):
|
|
|
|
|
|
|
|
|
|
|
|
|
| 665 |
module.decoder.layers.9.self_attention.linear_qkv.weight
|
| 666 |
module.decoder.layers.9.self_attention.k_layernorm.weight
|
| 667 |
+
module.decoder.layers.9.self_attention.linear_qkv.layer_norm_weight
|
| 668 |
module.decoder.layers.9.self_attention.linear_proj.weight
|
| 669 |
+
module.decoder.layers.9.mlp.router.weight
|
| 670 |
+
module.decoder.layers.9.self_attention.q_layernorm.weight
|
| 671 |
+
module.decoder.layers.9.pre_mlp_layernorm.weight
|
| 672 |
Params for bucket 5 (71835904 elements, 71835904 padded size):
|
| 673 |
module.decoder.layers.8.mlp.router.weight
|
| 674 |
+
module.decoder.layers.8.pre_mlp_layernorm.weight
|
| 675 |
module.decoder.layers.8.self_attention.k_layernorm.weight
|
| 676 |
module.decoder.layers.8.self_attention.q_layernorm.weight
|
|
|
|
| 677 |
module.decoder.layers.8.self_attention.linear_qkv.weight
|
| 678 |
module.decoder.layers.8.self_attention.linear_qkv.layer_norm_weight
|
| 679 |
module.decoder.layers.8.self_attention.linear_proj.weight
|
| 680 |
Params for bucket 6 (71835904 elements, 71835904 padded size):
|
| 681 |
module.decoder.layers.7.mlp.router.weight
|
| 682 |
+
module.decoder.layers.7.self_attention.q_layernorm.weight
|
| 683 |
module.decoder.layers.7.pre_mlp_layernorm.weight
|
| 684 |
module.decoder.layers.7.self_attention.k_layernorm.weight
|
|
|
|
| 685 |
module.decoder.layers.7.self_attention.linear_qkv.layer_norm_weight
|
| 686 |
module.decoder.layers.7.self_attention.linear_qkv.weight
|
| 687 |
module.decoder.layers.7.self_attention.linear_proj.weight
|
|
|
|
| 689 |
module.decoder.layers.6.mlp.router.weight
|
| 690 |
module.decoder.layers.6.self_attention.linear_qkv.weight
|
| 691 |
module.decoder.layers.6.self_attention.k_layernorm.weight
|
| 692 |
+
module.decoder.layers.6.self_attention.linear_proj.weight
|
| 693 |
module.decoder.layers.6.pre_mlp_layernorm.weight
|
| 694 |
module.decoder.layers.6.self_attention.q_layernorm.weight
|
| 695 |
+
module.decoder.layers.6.self_attention.linear_qkv.layer_norm_weight
|
| 696 |
Params for bucket 8 (71835904 elements, 71835904 padded size):
|
| 697 |
module.decoder.layers.5.mlp.router.weight
|
| 698 |
+
module.decoder.layers.5.self_attention.k_layernorm.weight
|
| 699 |
module.decoder.layers.5.self_attention.linear_qkv.weight
|
|
|
|
| 700 |
module.decoder.layers.5.pre_mlp_layernorm.weight
|
| 701 |
+
module.decoder.layers.5.self_attention.q_layernorm.weight
|
| 702 |
module.decoder.layers.5.self_attention.linear_qkv.layer_norm_weight
|
|
|
|
| 703 |
module.decoder.layers.5.self_attention.linear_proj.weight
|
| 704 |
Params for bucket 9 (71835904 elements, 71835904 padded size):
|
| 705 |
+
module.decoder.layers.4.mlp.router.weight
|
|
|
|
| 706 |
module.decoder.layers.4.self_attention.q_layernorm.weight
|
| 707 |
+
module.decoder.layers.4.pre_mlp_layernorm.weight
|
| 708 |
module.decoder.layers.4.self_attention.linear_qkv.layer_norm_weight
|
|
|
|
| 709 |
module.decoder.layers.4.self_attention.k_layernorm.weight
|
| 710 |
+
module.decoder.layers.4.self_attention.linear_qkv.weight
|
| 711 |
module.decoder.layers.4.self_attention.linear_proj.weight
|
| 712 |
Params for bucket 10 (71835904 elements, 71835904 padded size):
|
| 713 |
module.decoder.layers.3.mlp.router.weight
|
|
|
|
| 714 |
module.decoder.layers.3.self_attention.q_layernorm.weight
|
| 715 |
module.decoder.layers.3.self_attention.linear_qkv.weight
|
| 716 |
+
module.decoder.layers.3.pre_mlp_layernorm.weight
|
| 717 |
module.decoder.layers.3.self_attention.k_layernorm.weight
|
| 718 |
module.decoder.layers.3.self_attention.linear_qkv.layer_norm_weight
|
| 719 |
module.decoder.layers.3.self_attention.linear_proj.weight
|
| 720 |
Params for bucket 11 (71835904 elements, 71835904 padded size):
|
| 721 |
module.decoder.layers.2.mlp.router.weight
|
| 722 |
+
module.decoder.layers.2.self_attention.k_layernorm.weight
|
| 723 |
+
module.decoder.layers.2.self_attention.q_layernorm.weight
|
| 724 |
module.decoder.layers.2.pre_mlp_layernorm.weight
|
| 725 |
module.decoder.layers.2.self_attention.linear_qkv.weight
|
|
|
|
|
|
|
| 726 |
module.decoder.layers.2.self_attention.linear_qkv.layer_norm_weight
|
| 727 |
module.decoder.layers.2.self_attention.linear_proj.weight
|
| 728 |
Params for bucket 12 (71835904 elements, 71835904 padded size):
|
| 729 |
module.decoder.layers.1.mlp.router.weight
|
| 730 |
module.decoder.layers.1.pre_mlp_layernorm.weight
|
|
|
|
|
|
|
| 731 |
module.decoder.layers.1.self_attention.k_layernorm.weight
|
| 732 |
+
module.decoder.layers.1.self_attention.linear_qkv.weight
|
| 733 |
+
module.decoder.layers.1.self_attention.q_layernorm.weight
|
| 734 |
module.decoder.layers.1.self_attention.linear_qkv.layer_norm_weight
|
| 735 |
module.decoder.layers.1.self_attention.linear_proj.weight
|
| 736 |
Params for bucket 13 (71835904 elements, 71835904 padded size):
|
| 737 |
+
module.decoder.layers.0.self_attention.k_layernorm.weight
|
| 738 |
+
module.decoder.layers.0.self_attention.linear_qkv.weight
|
| 739 |
module.decoder.layers.0.self_attention.q_layernorm.weight
|
| 740 |
module.decoder.layers.0.pre_mlp_layernorm.weight
|
|
|
|
|
|
|
|
|
|
|
|
|
| 741 |
module.decoder.layers.0.self_attention.linear_proj.weight
|
| 742 |
+
module.decoder.layers.0.mlp.router.weight
|
| 743 |
+
module.decoder.layers.0.self_attention.linear_qkv.layer_norm_weight
|
| 744 |
Params for bucket 14 (206045184 elements, 206045184 padded size):
|
| 745 |
module.embedding.word_embeddings.weight
|
| 746 |
INFO:megatron.core.distributed.param_and_grad_buffer:Number of buckets for gradient all-reduce / reduce-scatter: 76
|
| 747 |
Params for bucket 1 (44040192 elements, 44040192 padded size):
|
|
|
|
|
|
|
|
|
|
|
|
|
| 748 |
module.decoder.layers.11.mlp.experts.linear_fc2.weight1
|
| 749 |
module.decoder.layers.11.mlp.experts.linear_fc2.weight3
|
| 750 |
module.decoder.layers.11.mlp.experts.linear_fc2.weight5
|
| 751 |
+
module.decoder.layers.11.mlp.experts.linear_fc2.weight0
|
| 752 |
+
module.decoder.layers.11.mlp.experts.linear_fc2.weight4
|
| 753 |
+
module.decoder.layers.11.mlp.experts.linear_fc2.weight2
|
| 754 |
+
module.decoder.layers.11.mlp.experts.linear_fc2.weight6
|
| 755 |
Params for bucket 2 (44040192 elements, 44040192 padded size):
|
|
|
|
|
|
|
|
|
|
| 756 |
module.decoder.layers.11.mlp.experts.linear_fc2.weight7
|
| 757 |
module.decoder.layers.11.mlp.experts.linear_fc2.weight11
|
| 758 |
module.decoder.layers.11.mlp.experts.linear_fc2.weight9
|
| 759 |
module.decoder.layers.11.mlp.experts.linear_fc2.weight13
|
| 760 |
+
module.decoder.layers.11.mlp.experts.linear_fc2.weight10
|
| 761 |
+
module.decoder.layers.11.mlp.experts.linear_fc2.weight8
|
| 762 |
+
module.decoder.layers.11.mlp.experts.linear_fc2.weight12
|
| 763 |
Params for bucket 3 (50331648 elements, 50331648 padded size):
|
| 764 |
+
module.decoder.layers.11.mlp.experts.linear_fc1.weight0
|
|
|
|
| 765 |
module.decoder.layers.11.mlp.experts.linear_fc1.weight2
|
| 766 |
module.decoder.layers.11.mlp.experts.linear_fc2.weight15
|
| 767 |
+
module.decoder.layers.11.mlp.experts.linear_fc2.weight14
|
| 768 |
+
module.decoder.layers.11.mlp.experts.linear_fc1.weight1
|
| 769 |
Params for bucket 4 (50331648 elements, 50331648 padded size):
|
| 770 |
module.decoder.layers.11.mlp.experts.linear_fc1.weight3
|
| 771 |
module.decoder.layers.11.mlp.experts.linear_fc1.weight4
|
|
|
|
| 772 |
module.decoder.layers.11.mlp.experts.linear_fc1.weight6
|
| 773 |
+
module.decoder.layers.11.mlp.experts.linear_fc1.weight5
|
| 774 |
Params for bucket 5 (50331648 elements, 50331648 padded size):
|
| 775 |
module.decoder.layers.11.mlp.experts.linear_fc1.weight7
|
|
|
|
| 776 |
module.decoder.layers.11.mlp.experts.linear_fc1.weight10
|
| 777 |
+
module.decoder.layers.11.mlp.experts.linear_fc1.weight8
|
| 778 |
module.decoder.layers.11.mlp.experts.linear_fc1.weight9
|
| 779 |
Params for bucket 6 (50331648 elements, 50331648 padded size):
|
| 780 |
module.decoder.layers.11.mlp.experts.linear_fc1.weight11
|
|
|
|
| 781 |
module.decoder.layers.11.mlp.experts.linear_fc1.weight13
|
| 782 |
+
module.decoder.layers.11.mlp.experts.linear_fc1.weight14
|
| 783 |
module.decoder.layers.11.mlp.experts.linear_fc1.weight12
|
| 784 |
Params for bucket 7 (44040192 elements, 44040192 padded size):
|
|
|
|
| 785 |
module.decoder.layers.10.mlp.experts.linear_fc2.weight3
|
| 786 |
module.decoder.layers.10.mlp.experts.linear_fc2.weight1
|
| 787 |
+
module.decoder.layers.11.mlp.experts.linear_fc1.weight15
|
| 788 |
module.decoder.layers.10.mlp.experts.linear_fc2.weight2
|
| 789 |
+
module.decoder.layers.10.mlp.experts.linear_fc2.weight0
|
| 790 |
module.decoder.layers.10.mlp.experts.linear_fc2.weight4
|
| 791 |
Params for bucket 8 (44040192 elements, 44040192 padded size):
|
| 792 |
module.decoder.layers.10.mlp.experts.linear_fc2.weight5
|
|
|
|
| 793 |
module.decoder.layers.10.mlp.experts.linear_fc2.weight9
|
| 794 |
+
module.decoder.layers.10.mlp.experts.linear_fc2.weight7
|
| 795 |
module.decoder.layers.10.mlp.experts.linear_fc2.weight11
|
|
|
|
| 796 |
module.decoder.layers.10.mlp.experts.linear_fc2.weight6
|
| 797 |
+
module.decoder.layers.10.mlp.experts.linear_fc2.weight8
|
| 798 |
module.decoder.layers.10.mlp.experts.linear_fc2.weight10
|
| 799 |
Params for bucket 9 (50331648 elements, 50331648 padded size):
|
|
|
|
| 800 |
module.decoder.layers.10.mlp.experts.linear_fc1.weight0
|
| 801 |
module.decoder.layers.10.mlp.experts.linear_fc2.weight15
|
| 802 |
+
module.decoder.layers.10.mlp.experts.linear_fc2.weight13
|
| 803 |
module.decoder.layers.10.mlp.experts.linear_fc2.weight12
|
| 804 |
module.decoder.layers.10.mlp.experts.linear_fc2.weight14
|
| 805 |
module.decoder.layers.10.mlp.experts.linear_fc1.weight1
|
| 806 |
Params for bucket 10 (50331648 elements, 50331648 padded size):
|
| 807 |
module.decoder.layers.10.mlp.experts.linear_fc1.weight2
|
|
|
|
| 808 |
module.decoder.layers.10.mlp.experts.linear_fc1.weight5
|
| 809 |
module.decoder.layers.10.mlp.experts.linear_fc1.weight3
|
| 810 |
+
module.decoder.layers.10.mlp.experts.linear_fc1.weight4
|
| 811 |
Params for bucket 11 (50331648 elements, 50331648 padded size):
|
| 812 |
module.decoder.layers.10.mlp.experts.linear_fc1.weight6
|
|
|
|
| 813 |
module.decoder.layers.10.mlp.experts.linear_fc1.weight8
|
| 814 |
module.decoder.layers.10.mlp.experts.linear_fc1.weight9
|
| 815 |
+
module.decoder.layers.10.mlp.experts.linear_fc1.weight7
|
| 816 |
Params for bucket 12 (50331648 elements, 50331648 padded size):
|
| 817 |
module.decoder.layers.10.mlp.experts.linear_fc1.weight10
|
| 818 |
+
module.decoder.layers.10.mlp.experts.linear_fc1.weight12
|
| 819 |
module.decoder.layers.10.mlp.experts.linear_fc1.weight13
|
| 820 |
module.decoder.layers.10.mlp.experts.linear_fc1.weight11
|
|
|
|
| 821 |
Params for bucket 13 (44040192 elements, 44040192 padded size):
|
| 822 |
module.decoder.layers.10.mlp.experts.linear_fc1.weight14
|
|
|
|
| 823 |
module.decoder.layers.9.mlp.experts.linear_fc2.weight2
|
| 824 |
module.decoder.layers.9.mlp.experts.linear_fc2.weight0
|
| 825 |
+
module.decoder.layers.9.mlp.experts.linear_fc2.weight1
|
| 826 |
module.decoder.layers.10.mlp.experts.linear_fc1.weight15
|
| 827 |
Params for bucket 14 (44040192 elements, 44040192 padded size):
|
| 828 |
+
module.decoder.layers.9.mlp.experts.linear_fc2.weight6
|
| 829 |
+
module.decoder.layers.9.mlp.experts.linear_fc2.weight4
|
| 830 |
+
module.decoder.layers.9.mlp.experts.linear_fc2.weight8
|
| 831 |
module.decoder.layers.9.mlp.experts.linear_fc2.weight3
|
| 832 |
module.decoder.layers.9.mlp.experts.linear_fc2.weight5
|
| 833 |
module.decoder.layers.9.mlp.experts.linear_fc2.weight7
|
| 834 |
module.decoder.layers.9.mlp.experts.linear_fc2.weight9
|
|
|
|
|
|
|
|
|
|
| 835 |
Params for bucket 15 (50331648 elements, 50331648 padded size):
|
| 836 |
+
module.decoder.layers.9.mlp.experts.linear_fc2.weight10
|
| 837 |
+
module.decoder.layers.9.mlp.experts.linear_fc2.weight12
|
| 838 |
+
module.decoder.layers.9.mlp.experts.linear_fc2.weight14
|
| 839 |
module.decoder.layers.9.mlp.experts.linear_fc2.weight11
|
| 840 |
+
module.decoder.layers.9.mlp.experts.linear_fc2.weight13
|
| 841 |
module.decoder.layers.9.mlp.experts.linear_fc2.weight15
|
| 842 |
module.decoder.layers.9.mlp.experts.linear_fc1.weight0
|
|
|
|
|
|
|
|
|
|
| 843 |
Params for bucket 16 (50331648 elements, 50331648 padded size):
|
| 844 |
module.decoder.layers.9.mlp.experts.linear_fc1.weight1
|
| 845 |
+
module.decoder.layers.9.mlp.experts.linear_fc1.weight4
|
| 846 |
module.decoder.layers.9.mlp.experts.linear_fc1.weight2
|
| 847 |
module.decoder.layers.9.mlp.experts.linear_fc1.weight3
|
|
|
|
| 848 |
Params for bucket 17 (50331648 elements, 50331648 padded size):
|
| 849 |
module.decoder.layers.9.mlp.experts.linear_fc1.weight5
|
| 850 |
+
module.decoder.layers.9.mlp.experts.linear_fc1.weight7
|
| 851 |
module.decoder.layers.9.mlp.experts.linear_fc1.weight8
|
| 852 |
module.decoder.layers.9.mlp.experts.linear_fc1.weight6
|
|
|
|
| 853 |
Params for bucket 18 (50331648 elements, 50331648 padded size):
|
| 854 |
module.decoder.layers.9.mlp.experts.linear_fc1.weight9
|
| 855 |
module.decoder.layers.9.mlp.experts.linear_fc1.weight11
|
|
|
|
| 857 |
module.decoder.layers.9.mlp.experts.linear_fc1.weight10
|
| 858 |
Params for bucket 19 (44040192 elements, 44040192 padded size):
|
| 859 |
module.decoder.layers.9.mlp.experts.linear_fc1.weight13
|
|
|
|
| 860 |
module.decoder.layers.9.mlp.experts.linear_fc1.weight14
|
| 861 |
module.decoder.layers.9.mlp.experts.linear_fc1.weight15
|
| 862 |
+
module.decoder.layers.8.mlp.experts.linear_fc2.weight0
|
| 863 |
Params for bucket 20 (44040192 elements, 44040192 padded size):
|
|
|
|
|
|
|
|
|
|
| 864 |
module.decoder.layers.8.mlp.experts.linear_fc2.weight1
|
|
|
|
| 865 |
module.decoder.layers.8.mlp.experts.linear_fc2.weight3
|
| 866 |
+
module.decoder.layers.8.mlp.experts.linear_fc2.weight5
|
| 867 |
module.decoder.layers.8.mlp.experts.linear_fc2.weight7
|
| 868 |
+
module.decoder.layers.8.mlp.experts.linear_fc2.weight4
|
| 869 |
+
module.decoder.layers.8.mlp.experts.linear_fc2.weight2
|
| 870 |
+
module.decoder.layers.8.mlp.experts.linear_fc2.weight6
|
| 871 |
Params for bucket 21 (44040192 elements, 44040192 padded size):
|
| 872 |
+
module.decoder.layers.8.mlp.experts.linear_fc2.weight9
|
| 873 |
+
module.decoder.layers.8.mlp.experts.linear_fc2.weight11
|
| 874 |
+
module.decoder.layers.8.mlp.experts.linear_fc2.weight13
|
| 875 |
module.decoder.layers.8.mlp.experts.linear_fc2.weight8
|
| 876 |
module.decoder.layers.8.mlp.experts.linear_fc2.weight10
|
| 877 |
module.decoder.layers.8.mlp.experts.linear_fc2.weight12
|
| 878 |
module.decoder.layers.8.mlp.experts.linear_fc2.weight14
|
|
|
|
|
|
|
|
|
|
| 879 |
Params for bucket 22 (44040192 elements, 44040192 padded size):
|
| 880 |
module.decoder.layers.8.mlp.experts.linear_fc2.weight15
|
| 881 |
module.decoder.layers.8.mlp.experts.linear_fc1.weight1
|
|
|
|
| 882 |
module.decoder.layers.8.mlp.experts.linear_fc1.weight0
|
| 883 |
+
module.decoder.layers.8.mlp.experts.linear_fc1.weight2
|
| 884 |
Params for bucket 23 (50331648 elements, 50331648 padded size):
|
| 885 |
module.decoder.layers.8.mlp.experts.linear_fc1.weight3
|
|
|
|
| 886 |
module.decoder.layers.8.mlp.experts.linear_fc1.weight4
|
| 887 |
+
module.decoder.layers.8.mlp.experts.linear_fc1.weight5
|
| 888 |
module.decoder.layers.8.mlp.experts.linear_fc1.weight6
|
| 889 |
Params for bucket 24 (50331648 elements, 50331648 padded size):
|
| 890 |
module.decoder.layers.8.mlp.experts.linear_fc1.weight7
|
| 891 |
+
module.decoder.layers.8.mlp.experts.linear_fc1.weight10
|
| 892 |
module.decoder.layers.8.mlp.experts.linear_fc1.weight8
|
| 893 |
module.decoder.layers.8.mlp.experts.linear_fc1.weight9
|
|
|
|
| 894 |
Params for bucket 25 (50331648 elements, 50331648 padded size):
|
| 895 |
module.decoder.layers.8.mlp.experts.linear_fc1.weight11
|
| 896 |
module.decoder.layers.8.mlp.experts.linear_fc1.weight14
|
|
|
|
| 897 |
module.decoder.layers.8.mlp.experts.linear_fc1.weight13
|
| 898 |
+
module.decoder.layers.8.mlp.experts.linear_fc1.weight12
|
| 899 |
Params for bucket 26 (44040192 elements, 44040192 padded size):
|
| 900 |
+
module.decoder.layers.7.mlp.experts.linear_fc2.weight3
|
| 901 |
+
module.decoder.layers.7.mlp.experts.linear_fc2.weight1
|
| 902 |
module.decoder.layers.8.mlp.experts.linear_fc1.weight15
|
| 903 |
module.decoder.layers.7.mlp.experts.linear_fc2.weight2
|
| 904 |
module.decoder.layers.7.mlp.experts.linear_fc2.weight0
|
| 905 |
module.decoder.layers.7.mlp.experts.linear_fc2.weight4
|
|
|
|
|
|
|
| 906 |
Params for bucket 27 (44040192 elements, 44040192 padded size):
|
|
|
|
|
|
|
|
|
|
| 907 |
module.decoder.layers.7.mlp.experts.linear_fc2.weight5
|
|
|
|
| 908 |
module.decoder.layers.7.mlp.experts.linear_fc2.weight9
|
| 909 |
+
module.decoder.layers.7.mlp.experts.linear_fc2.weight7
|
| 910 |
module.decoder.layers.7.mlp.experts.linear_fc2.weight11
|
| 911 |
+
module.decoder.layers.7.mlp.experts.linear_fc2.weight8
|
| 912 |
+
module.decoder.layers.7.mlp.experts.linear_fc2.weight6
|
| 913 |
+
module.decoder.layers.7.mlp.experts.linear_fc2.weight10
|
| 914 |
Params for bucket 28 (50331648 elements, 50331648 padded size):
|
|
|
|
| 915 |
module.decoder.layers.7.mlp.experts.linear_fc1.weight0
|
|
|
|
| 916 |
module.decoder.layers.7.mlp.experts.linear_fc2.weight15
|
| 917 |
module.decoder.layers.7.mlp.experts.linear_fc2.weight13
|
| 918 |
+
module.decoder.layers.7.mlp.experts.linear_fc2.weight12
|
| 919 |
+
module.decoder.layers.7.mlp.experts.linear_fc2.weight14
|
| 920 |
module.decoder.layers.7.mlp.experts.linear_fc1.weight1
|
| 921 |
Params for bucket 29 (50331648 elements, 50331648 padded size):
|
| 922 |
module.decoder.layers.7.mlp.experts.linear_fc1.weight2
|
|
|
|
| 923 |
module.decoder.layers.7.mlp.experts.linear_fc1.weight3
|
| 924 |
+
module.decoder.layers.7.mlp.experts.linear_fc1.weight5
|
| 925 |
module.decoder.layers.7.mlp.experts.linear_fc1.weight4
|
| 926 |
Params for bucket 30 (50331648 elements, 50331648 padded size):
|
| 927 |
module.decoder.layers.7.mlp.experts.linear_fc1.weight6
|
|
|
|
| 928 |
module.decoder.layers.7.mlp.experts.linear_fc1.weight9
|
| 929 |
+
module.decoder.layers.7.mlp.experts.linear_fc1.weight8
|
| 930 |
module.decoder.layers.7.mlp.experts.linear_fc1.weight7
|
| 931 |
Params for bucket 31 (50331648 elements, 50331648 padded size):
|
| 932 |
module.decoder.layers.7.mlp.experts.linear_fc1.weight10
|
|
|
|
| 933 |
module.decoder.layers.7.mlp.experts.linear_fc1.weight12
|
| 934 |
+
module.decoder.layers.7.mlp.experts.linear_fc1.weight13
|
| 935 |
module.decoder.layers.7.mlp.experts.linear_fc1.weight11
|
| 936 |
Params for bucket 32 (44040192 elements, 44040192 padded size):
|
| 937 |
module.decoder.layers.7.mlp.experts.linear_fc1.weight14
|
|
|
|
| 944 |
module.decoder.layers.6.mlp.experts.linear_fc2.weight6
|
| 945 |
module.decoder.layers.6.mlp.experts.linear_fc2.weight8
|
| 946 |
module.decoder.layers.6.mlp.experts.linear_fc2.weight3
|
|
|
|
| 947 |
module.decoder.layers.6.mlp.experts.linear_fc2.weight5
|
| 948 |
+
module.decoder.layers.6.mlp.experts.linear_fc2.weight7
|
| 949 |
module.decoder.layers.6.mlp.experts.linear_fc2.weight9
|
| 950 |
Params for bucket 34 (50331648 elements, 50331648 padded size):
|
| 951 |
module.decoder.layers.6.mlp.experts.linear_fc2.weight10
|
|
|
|
| 952 |
module.decoder.layers.6.mlp.experts.linear_fc2.weight14
|
| 953 |
+
module.decoder.layers.6.mlp.experts.linear_fc2.weight12
|
| 954 |
module.decoder.layers.6.mlp.experts.linear_fc2.weight11
|
| 955 |
module.decoder.layers.6.mlp.experts.linear_fc2.weight13
|
| 956 |
module.decoder.layers.6.mlp.experts.linear_fc2.weight15
|
| 957 |
module.decoder.layers.6.mlp.experts.linear_fc1.weight0
|
| 958 |
Params for bucket 35 (50331648 elements, 50331648 padded size):
|
| 959 |
module.decoder.layers.6.mlp.experts.linear_fc1.weight1
|
| 960 |
+
module.decoder.layers.6.mlp.experts.linear_fc1.weight2
|
| 961 |
module.decoder.layers.6.mlp.experts.linear_fc1.weight3
|
| 962 |
module.decoder.layers.6.mlp.experts.linear_fc1.weight4
|
|
|
|
| 963 |
Params for bucket 36 (50331648 elements, 50331648 padded size):
|
| 964 |
module.decoder.layers.6.mlp.experts.linear_fc1.weight5
|
|
|
|
| 965 |
module.decoder.layers.6.mlp.experts.linear_fc1.weight8
|
| 966 |
module.decoder.layers.6.mlp.experts.linear_fc1.weight6
|
| 967 |
+
module.decoder.layers.6.mlp.experts.linear_fc1.weight7
|
| 968 |
Params for bucket 37 (50331648 elements, 50331648 padded size):
|
| 969 |
module.decoder.layers.6.mlp.experts.linear_fc1.weight9
|
|
|
|
|
|
|
| 970 |
module.decoder.layers.6.mlp.experts.linear_fc1.weight12
|
| 971 |
+
module.decoder.layers.6.mlp.experts.linear_fc1.weight11
|
| 972 |
+
module.decoder.layers.6.mlp.experts.linear_fc1.weight10
|
| 973 |
Params for bucket 38 (44040192 elements, 44040192 padded size):
|
| 974 |
module.decoder.layers.6.mlp.experts.linear_fc1.weight13
|
| 975 |
+
module.decoder.layers.6.mlp.experts.linear_fc1.weight15
|
| 976 |
module.decoder.layers.5.mlp.experts.linear_fc2.weight0
|
| 977 |
module.decoder.layers.6.mlp.experts.linear_fc1.weight14
|
|
|
|
| 978 |
Params for bucket 39 (44040192 elements, 44040192 padded size):
|
| 979 |
module.decoder.layers.5.mlp.experts.linear_fc2.weight1
|
| 980 |
module.decoder.layers.5.mlp.experts.linear_fc2.weight3
|
|
|
|
| 1003 |
module.decoder.layers.5.mlp.experts.linear_fc1.weight5
|
| 1004 |
Params for bucket 43 (50331648 elements, 50331648 padded size):
|
| 1005 |
module.decoder.layers.5.mlp.experts.linear_fc1.weight7
|
| 1006 |
+
module.decoder.layers.5.mlp.experts.linear_fc1.weight9
|
| 1007 |
module.decoder.layers.5.mlp.experts.linear_fc1.weight10
|
| 1008 |
module.decoder.layers.5.mlp.experts.linear_fc1.weight8
|
|
|
|
| 1009 |
Params for bucket 44 (50331648 elements, 50331648 padded size):
|
| 1010 |
module.decoder.layers.5.mlp.experts.linear_fc1.weight11
|
| 1011 |
+
module.decoder.layers.5.mlp.experts.linear_fc1.weight12
|
| 1012 |
module.decoder.layers.5.mlp.experts.linear_fc1.weight13
|
| 1013 |
module.decoder.layers.5.mlp.experts.linear_fc1.weight14
|
|
|
|
| 1014 |
Params for bucket 45 (44040192 elements, 44040192 padded size):
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1015 |
module.decoder.layers.4.mlp.experts.linear_fc2.weight3
|
| 1016 |
module.decoder.layers.4.mlp.experts.linear_fc2.weight1
|
| 1017 |
+
module.decoder.layers.5.mlp.experts.linear_fc1.weight15
|
| 1018 |
+
module.decoder.layers.4.mlp.experts.linear_fc2.weight0
|
| 1019 |
+
module.decoder.layers.4.mlp.experts.linear_fc2.weight2
|
| 1020 |
+
module.decoder.layers.4.mlp.experts.linear_fc2.weight4
|
| 1021 |
Params for bucket 46 (44040192 elements, 44040192 padded size):
|
|
|
|
|
|
|
|
|
|
| 1022 |
module.decoder.layers.4.mlp.experts.linear_fc2.weight5
|
|
|
|
| 1023 |
module.decoder.layers.4.mlp.experts.linear_fc2.weight7
|
| 1024 |
+
module.decoder.layers.4.mlp.experts.linear_fc2.weight9
|
| 1025 |
module.decoder.layers.4.mlp.experts.linear_fc2.weight11
|
| 1026 |
+
module.decoder.layers.4.mlp.experts.linear_fc2.weight8
|
| 1027 |
+
module.decoder.layers.4.mlp.experts.linear_fc2.weight6
|
| 1028 |
+
module.decoder.layers.4.mlp.experts.linear_fc2.weight10
|
| 1029 |
Params for bucket 47 (50331648 elements, 50331648 padded size):
|
| 1030 |
+
module.decoder.layers.4.mlp.experts.linear_fc2.weight13
|
| 1031 |
+
module.decoder.layers.4.mlp.experts.linear_fc1.weight0
|
| 1032 |
+
module.decoder.layers.4.mlp.experts.linear_fc2.weight15
|
| 1033 |
module.decoder.layers.4.mlp.experts.linear_fc2.weight12
|
| 1034 |
module.decoder.layers.4.mlp.experts.linear_fc2.weight14
|
| 1035 |
module.decoder.layers.4.mlp.experts.linear_fc1.weight1
|
|
|
|
|
|
|
|
|
|
| 1036 |
Params for bucket 48 (50331648 elements, 50331648 padded size):
|
| 1037 |
module.decoder.layers.4.mlp.experts.linear_fc1.weight2
|
| 1038 |
+
module.decoder.layers.4.mlp.experts.linear_fc1.weight3
|
| 1039 |
module.decoder.layers.4.mlp.experts.linear_fc1.weight4
|
| 1040 |
module.decoder.layers.4.mlp.experts.linear_fc1.weight5
|
|
|
|
| 1041 |
Params for bucket 49 (50331648 elements, 50331648 padded size):
|
| 1042 |
module.decoder.layers.4.mlp.experts.linear_fc1.weight6
|
| 1043 |
+
module.decoder.layers.4.mlp.experts.linear_fc1.weight9
|
| 1044 |
module.decoder.layers.4.mlp.experts.linear_fc1.weight7
|
| 1045 |
module.decoder.layers.4.mlp.experts.linear_fc1.weight8
|
|
|
|
| 1046 |
Params for bucket 50 (50331648 elements, 50331648 padded size):
|
| 1047 |
module.decoder.layers.4.mlp.experts.linear_fc1.weight10
|
|
|
|
| 1048 |
module.decoder.layers.4.mlp.experts.linear_fc1.weight12
|
| 1049 |
module.decoder.layers.4.mlp.experts.linear_fc1.weight13
|
| 1050 |
+
module.decoder.layers.4.mlp.experts.linear_fc1.weight11
|
| 1051 |
Params for bucket 51 (44040192 elements, 44040192 padded size):
|
|
|
|
|
|
|
| 1052 |
module.decoder.layers.4.mlp.experts.linear_fc1.weight14
|
|
|
|
| 1053 |
module.decoder.layers.3.mlp.experts.linear_fc2.weight0
|
| 1054 |
+
module.decoder.layers.3.mlp.experts.linear_fc2.weight2
|
| 1055 |
+
module.decoder.layers.3.mlp.experts.linear_fc2.weight1
|
| 1056 |
+
module.decoder.layers.4.mlp.experts.linear_fc1.weight15
|
| 1057 |
Params for bucket 52 (44040192 elements, 44040192 padded size):
|
| 1058 |
+
module.decoder.layers.3.mlp.experts.linear_fc2.weight6
|
| 1059 |
+
module.decoder.layers.3.mlp.experts.linear_fc2.weight4
|
| 1060 |
+
module.decoder.layers.3.mlp.experts.linear_fc2.weight8
|
| 1061 |
module.decoder.layers.3.mlp.experts.linear_fc2.weight3
|
| 1062 |
module.decoder.layers.3.mlp.experts.linear_fc2.weight7
|
| 1063 |
module.decoder.layers.3.mlp.experts.linear_fc2.weight5
|
| 1064 |
module.decoder.layers.3.mlp.experts.linear_fc2.weight9
|
|
|
|
|
|
|
|
|
|
| 1065 |
Params for bucket 53 (50331648 elements, 50331648 padded size):
|
| 1066 |
+
module.decoder.layers.3.mlp.experts.linear_fc2.weight10
|
| 1067 |
+
module.decoder.layers.3.mlp.experts.linear_fc2.weight12
|
| 1068 |
+
module.decoder.layers.3.mlp.experts.linear_fc2.weight14
|
| 1069 |
module.decoder.layers.3.mlp.experts.linear_fc2.weight13
|
| 1070 |
module.decoder.layers.3.mlp.experts.linear_fc2.weight11
|
| 1071 |
module.decoder.layers.3.mlp.experts.linear_fc2.weight15
|
| 1072 |
module.decoder.layers.3.mlp.experts.linear_fc1.weight0
|
|
|
|
|
|
|
|
|
|
| 1073 |
Params for bucket 54 (50331648 elements, 50331648 padded size):
|
| 1074 |
module.decoder.layers.3.mlp.experts.linear_fc1.weight1
|
| 1075 |
module.decoder.layers.3.mlp.experts.linear_fc1.weight4
|
|
|
|
| 1076 |
module.decoder.layers.3.mlp.experts.linear_fc1.weight3
|
| 1077 |
+
module.decoder.layers.3.mlp.experts.linear_fc1.weight2
|
| 1078 |
Params for bucket 55 (50331648 elements, 50331648 padded size):
|
| 1079 |
module.decoder.layers.3.mlp.experts.linear_fc1.weight5
|
| 1080 |
module.decoder.layers.3.mlp.experts.linear_fc1.weight7
|
|
|
|
| 1087 |
module.decoder.layers.3.mlp.experts.linear_fc1.weight12
|
| 1088 |
Params for bucket 57 (44040192 elements, 44040192 padded size):
|
| 1089 |
module.decoder.layers.3.mlp.experts.linear_fc1.weight13
|
|
|
|
| 1090 |
module.decoder.layers.2.mlp.experts.linear_fc2.weight0
|
| 1091 |
+
module.decoder.layers.3.mlp.experts.linear_fc1.weight14
|
| 1092 |
module.decoder.layers.3.mlp.experts.linear_fc1.weight15
|
| 1093 |
Params for bucket 58 (44040192 elements, 44040192 padded size):
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1094 |
module.decoder.layers.2.mlp.experts.linear_fc2.weight2
|
| 1095 |
module.decoder.layers.2.mlp.experts.linear_fc2.weight4
|
| 1096 |
module.decoder.layers.2.mlp.experts.linear_fc2.weight6
|
| 1097 |
+
module.decoder.layers.2.mlp.experts.linear_fc2.weight1
|
| 1098 |
+
module.decoder.layers.2.mlp.experts.linear_fc2.weight5
|
| 1099 |
+
module.decoder.layers.2.mlp.experts.linear_fc2.weight3
|
| 1100 |
+
module.decoder.layers.2.mlp.experts.linear_fc2.weight7
|
| 1101 |
Params for bucket 59 (44040192 elements, 44040192 padded size):
|
|
|
|
|
|
|
|
|
|
| 1102 |
module.decoder.layers.2.mlp.experts.linear_fc2.weight8
|
| 1103 |
module.decoder.layers.2.mlp.experts.linear_fc2.weight12
|
| 1104 |
module.decoder.layers.2.mlp.experts.linear_fc2.weight10
|
| 1105 |
module.decoder.layers.2.mlp.experts.linear_fc2.weight14
|
| 1106 |
+
module.decoder.layers.2.mlp.experts.linear_fc2.weight11
|
| 1107 |
+
module.decoder.layers.2.mlp.experts.linear_fc2.weight9
|
| 1108 |
+
module.decoder.layers.2.mlp.experts.linear_fc2.weight13
|
| 1109 |
Params for bucket 60 (44040192 elements, 44040192 padded size):
|
| 1110 |
module.decoder.layers.2.mlp.experts.linear_fc2.weight15
|
| 1111 |
+
module.decoder.layers.2.mlp.experts.linear_fc1.weight1
|
| 1112 |
module.decoder.layers.2.mlp.experts.linear_fc1.weight2
|
| 1113 |
module.decoder.layers.2.mlp.experts.linear_fc1.weight0
|
|
|
|
| 1114 |
Params for bucket 61 (50331648 elements, 50331648 padded size):
|
| 1115 |
module.decoder.layers.2.mlp.experts.linear_fc1.weight3
|
| 1116 |
+
module.decoder.layers.2.mlp.experts.linear_fc1.weight4
|
| 1117 |
module.decoder.layers.2.mlp.experts.linear_fc1.weight5
|
| 1118 |
module.decoder.layers.2.mlp.experts.linear_fc1.weight6
|
|
|
|
| 1119 |
Params for bucket 62 (50331648 elements, 50331648 padded size):
|
| 1120 |
module.decoder.layers.2.mlp.experts.linear_fc1.weight7
|
|
|
|
| 1121 |
module.decoder.layers.2.mlp.experts.linear_fc1.weight10
|
| 1122 |
module.decoder.layers.2.mlp.experts.linear_fc1.weight8
|
| 1123 |
+
module.decoder.layers.2.mlp.experts.linear_fc1.weight9
|
| 1124 |
Params for bucket 63 (50331648 elements, 50331648 padded size):
|
| 1125 |
module.decoder.layers.2.mlp.experts.linear_fc1.weight11
|
| 1126 |
+
module.decoder.layers.2.mlp.experts.linear_fc1.weight14
|
| 1127 |
module.decoder.layers.2.mlp.experts.linear_fc1.weight12
|
| 1128 |
module.decoder.layers.2.mlp.experts.linear_fc1.weight13
|
|
|
|
| 1129 |
Params for bucket 64 (44040192 elements, 44040192 padded size):
|
|
|
|
| 1130 |
module.decoder.layers.1.mlp.experts.linear_fc2.weight0
|
| 1131 |
+
module.decoder.layers.1.mlp.experts.linear_fc2.weight2
|
| 1132 |
module.decoder.layers.1.mlp.experts.linear_fc2.weight4
|
| 1133 |
module.decoder.layers.2.mlp.experts.linear_fc1.weight15
|
|
|
|
| 1134 |
module.decoder.layers.1.mlp.experts.linear_fc2.weight3
|
| 1135 |
+
module.decoder.layers.1.mlp.experts.linear_fc2.weight1
|
| 1136 |
Params for bucket 65 (44040192 elements, 44040192 padded size):
|
|
|
|
| 1137 |
module.decoder.layers.1.mlp.experts.linear_fc2.weight6
|
| 1138 |
+
module.decoder.layers.1.mlp.experts.linear_fc2.weight8
|
| 1139 |
module.decoder.layers.1.mlp.experts.linear_fc2.weight10
|
| 1140 |
module.decoder.layers.1.mlp.experts.linear_fc2.weight5
|
|
|
|
| 1141 |
module.decoder.layers.1.mlp.experts.linear_fc2.weight7
|
| 1142 |
+
module.decoder.layers.1.mlp.experts.linear_fc2.weight9
|
| 1143 |
module.decoder.layers.1.mlp.experts.linear_fc2.weight11
|
| 1144 |
Params for bucket 66 (50331648 elements, 50331648 padded size):
|
| 1145 |
module.decoder.layers.1.mlp.experts.linear_fc2.weight12
|
|
|
|
| 1146 |
module.decoder.layers.1.mlp.experts.linear_fc1.weight0
|
| 1147 |
+
module.decoder.layers.1.mlp.experts.linear_fc2.weight14
|
| 1148 |
module.decoder.layers.1.mlp.experts.linear_fc2.weight13
|
| 1149 |
+
module.decoder.layers.1.mlp.experts.linear_fc2.weight15
|
| 1150 |
module.decoder.layers.1.mlp.experts.linear_fc1.weight1
|
| 1151 |
Params for bucket 67 (50331648 elements, 50331648 padded size):
|
| 1152 |
module.decoder.layers.1.mlp.experts.linear_fc1.weight2
|
| 1153 |
+
module.decoder.layers.1.mlp.experts.linear_fc1.weight5
|
| 1154 |
module.decoder.layers.1.mlp.experts.linear_fc1.weight3
|
| 1155 |
module.decoder.layers.1.mlp.experts.linear_fc1.weight4
|
|
|
|
| 1156 |
Params for bucket 68 (50331648 elements, 50331648 padded size):
|
| 1157 |
module.decoder.layers.1.mlp.experts.linear_fc1.weight6
|
| 1158 |
+
module.decoder.layers.1.mlp.experts.linear_fc1.weight8
|
| 1159 |
module.decoder.layers.1.mlp.experts.linear_fc1.weight9
|
| 1160 |
module.decoder.layers.1.mlp.experts.linear_fc1.weight7
|
|
|
|
| 1161 |
Params for bucket 69 (50331648 elements, 50331648 padded size):
|
| 1162 |
module.decoder.layers.1.mlp.experts.linear_fc1.weight10
|
| 1163 |
module.decoder.layers.1.mlp.experts.linear_fc1.weight12
|
|
|
|
| 1164 |
module.decoder.layers.1.mlp.experts.linear_fc1.weight11
|
| 1165 |
+
module.decoder.layers.1.mlp.experts.linear_fc1.weight13
|
| 1166 |
Params for bucket 70 (44040192 elements, 44040192 padded size):
|
| 1167 |
module.decoder.layers.1.mlp.experts.linear_fc1.weight14
|
|
|
|
| 1168 |
module.decoder.layers.0.mlp.experts.linear_fc2.weight2
|
| 1169 |
+
module.decoder.layers.0.mlp.experts.linear_fc2.weight0
|
| 1170 |
module.decoder.layers.1.mlp.experts.linear_fc1.weight15
|
| 1171 |
+
module.decoder.layers.0.mlp.experts.linear_fc2.weight1
|
| 1172 |
Params for bucket 71 (44040192 elements, 44040192 padded size):
|
| 1173 |
module.decoder.layers.0.mlp.experts.linear_fc2.weight4
|
| 1174 |
module.decoder.layers.0.mlp.experts.linear_fc2.weight6
|
| 1175 |
module.decoder.layers.0.mlp.experts.linear_fc2.weight8
|
| 1176 |
module.decoder.layers.0.mlp.experts.linear_fc2.weight3
|
|
|
|
| 1177 |
module.decoder.layers.0.mlp.experts.linear_fc2.weight5
|
| 1178 |
+
module.decoder.layers.0.mlp.experts.linear_fc2.weight7
|
| 1179 |
module.decoder.layers.0.mlp.experts.linear_fc2.weight9
|
| 1180 |
Params for bucket 72 (50331648 elements, 50331648 padded size):
|
| 1181 |
module.decoder.layers.0.mlp.experts.linear_fc2.weight10
|
| 1182 |
module.decoder.layers.0.mlp.experts.linear_fc2.weight14
|
| 1183 |
module.decoder.layers.0.mlp.experts.linear_fc2.weight12
|
|
|
|
| 1184 |
module.decoder.layers.0.mlp.experts.linear_fc2.weight11
|
| 1185 |
module.decoder.layers.0.mlp.experts.linear_fc2.weight13
|
| 1186 |
module.decoder.layers.0.mlp.experts.linear_fc2.weight15
|
| 1187 |
+
module.decoder.layers.0.mlp.experts.linear_fc1.weight0
|
| 1188 |
Params for bucket 73 (50331648 elements, 50331648 padded size):
|
| 1189 |
module.decoder.layers.0.mlp.experts.linear_fc1.weight1
|
| 1190 |
+
module.decoder.layers.0.mlp.experts.linear_fc1.weight2
|
| 1191 |
module.decoder.layers.0.mlp.experts.linear_fc1.weight3
|
| 1192 |
module.decoder.layers.0.mlp.experts.linear_fc1.weight4
|
|
|
|
| 1193 |
Params for bucket 74 (50331648 elements, 50331648 padded size):
|
| 1194 |
module.decoder.layers.0.mlp.experts.linear_fc1.weight5
|
| 1195 |
module.decoder.layers.0.mlp.experts.linear_fc1.weight6
|
|
|
|
| 1196 |
module.decoder.layers.0.mlp.experts.linear_fc1.weight8
|
| 1197 |
+
module.decoder.layers.0.mlp.experts.linear_fc1.weight7
|
| 1198 |
Params for bucket 75 (50331648 elements, 50331648 padded size):
|
| 1199 |
module.decoder.layers.0.mlp.experts.linear_fc1.weight9
|
|
|
|
| 1200 |
module.decoder.layers.0.mlp.experts.linear_fc1.weight10
|
| 1201 |
module.decoder.layers.0.mlp.experts.linear_fc1.weight11
|
| 1202 |
+
module.decoder.layers.0.mlp.experts.linear_fc1.weight12
|
| 1203 |
Params for bucket 76 (37748736 elements, 37748736 padded size):
|
| 1204 |
module.decoder.layers.0.mlp.experts.linear_fc1.weight13
|
| 1205 |
module.decoder.layers.0.mlp.experts.linear_fc1.weight14
|
| 1206 |
module.decoder.layers.0.mlp.experts.linear_fc1.weight15
|
| 1207 |
+
INFO:megatron.core.optimizer:Setting up optimizer with config OptimizerConfig(optimizer='adam', lr=0.0001, min_lr=1e-05, decoupled_lr=None, decoupled_min_lr=None, weight_decay=0.1, fp8_recipe='mxfp8', fp16=False, bf16=True, reuse_grad_buf_for_mxfp8_param_ag=False, params_dtype=torch.bfloat16, use_precision_aware_optimizer=False, store_param_remainders=True, main_grads_dtype=torch.float32, main_params_dtype=torch.float32, exp_avg_dtype=torch.float32, exp_avg_sq_dtype=torch.float32, loss_scale=None, initial_loss_scale=4294967296, min_loss_scale=1.0, loss_scale_window=1000, hysteresis=2, adam_beta1=0.9, adam_beta2=0.999, adam_eps=1e-08, sgd_momentum=0.9, use_distributed_optimizer=True, overlap_param_gather_with_optimizer_step=False, optimizer_cpu_offload=False, optimizer_offload_fraction=1.0, use_torch_optimizer_for_cpu_offload=False, overlap_cpu_optimizer_d2h_h2d=False, pin_cpu_grads=True, pin_cpu_params=True, clip_grad=1.0, log_num_zeros_in_grad=False, barrier_with_L1_time=True, timers=<megatron.core.timers.Timers object at 0x719122c89400>, config_logger_dir='')
|
| 1208 |
INFO:megatron.core.optimizer_param_scheduler:> learning rate decay style: cosine
|
| 1209 |
+
[after model, optimizer, and learning rate scheduler are built] datetime: 2026-08-30 01:33:00
|
| 1210 |
> building train, validation, and test datasets ...
|
| 1211 |
> datasets target sizes (minimum size):
|
| 1212 |
train: 96
|
|
|
|
| 1214 |
test: 0
|
| 1215 |
INFO:megatron.core.datasets.blended_megatron_dataset_config:Let split_matrix = [(0, 1.0), None, None]
|
| 1216 |
> building train, validation, and test datasets for GPT ...
|
| 1217 |
+
INFO:megatron.core.datasets.blended_megatron_dataset_builder:Building GPTDataset splits with sizes=(96, 0, 0) and config=GPTDatasetConfig(random_seed=1234, sequence_length=8192, blend=(['/workspace/data/wikitext/wikitext103_gpt2_text_document'], None), blend_per_split=None, split='100,0,0', split_matrix=[(0, 1.0), None, None], num_dataset_builder_threads=1, path_to_cache=None, mmap_bin_files=True, mock=False, tokenizer=<megatron.training.tokenizer.tokenizer._GPT2BPETokenizer object at 0x718e7c70d6a0>, mid_level_dataset_surplus=0.005, reset_position_ids=False, reset_attention_mask=False, eod_mask_loss=False, create_attention_mask=True, drop_last_partial_validation_sequence=True, add_extra_token_to_sequence=True, object_storage_cache_path=None)
|
| 1218 |
INFO:megatron.core.datasets.indexed_dataset:Load the _IndexReader from /workspace/data/wikitext/wikitext103_gpt2_text_document.idx
|
| 1219 |
INFO:megatron.core.datasets.indexed_dataset: Extract the sequence lengths
|
| 1220 |
INFO:megatron.core.datasets.indexed_dataset: Extract the sequence pointers
|
|
|
|
| 1227 |
INFO:megatron.core.datasets.gpt_dataset: Load the shuffle index from eadc3756447e39d609e8aa728a91f451-GPTDataset-train-shuffle_index.npy
|
| 1228 |
INFO:megatron.core.datasets.gpt_dataset:> total number of samples: 14536
|
| 1229 |
> finished creating GPT datasets ...
|
| 1230 |
+
[after dataloaders are built] datetime: 2026-08-30 01:33:01
|
| 1231 |
done with setup ...
|
| 1232 |
(min, max) time across ranks (ms):
|
| 1233 |
+
model-and-optimizer-setup ......................: (3407.31, 3431.39)
|
| 1234 |
+
train/valid/test-data-iterators-setup ..........: (66.04, 78.01)
|
| 1235 |
training ...
|
| 1236 |
Overwriting rerun_state_machine.current_iteration from -1 to 0...
|
| 1237 |
+
[before the start of training step] datetime: 2026-08-30 01:33:01
|
| 1238 |
INFO:moonep.api:MoonEP comm buffer sizes:
|
| 1239 |
config : R=8, S=8192, H=4096, E=128, K=8, NvS=69600, NvS_padded=69632
|
| 1240 |
sm_count : num_sms=32, num_sms_dedup=148
|
|
|
|
| 1246 |
[MoonEP selfcheck 2/4] dropped groups empty OK | covered rows [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 128, 129, 130, 131, 132, 133, 134, 135, 136, 137, 138, 139, 140, 141, 142, 143] sizes [9344, 2688, 128, 24192, 128, 0, 128, 256, 128, 768, 10496, 128, 128, 2432, 128, 1536, 14208, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0] | duplicated experts [[69, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1], [88, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1], [55, 52, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1], [-1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1], [63, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1], [124, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1], [10, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1], [58, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1]]
|
| 1247 |
[MoonEP selfcheck 3/4] dropped groups empty OK | covered rows [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 128, 129, 130, 131, 132, 133, 134, 135, 136, 137, 138, 139, 140, 141, 142, 143] sizes [128, 128, 128, 0, 128, 44416, 128, 128, 128, 6912, 128, 256, 128, 128, 128, 128, 13824, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0] | duplicated experts [[78, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1], [111, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1], [62, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1], [-1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1], [57, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1], [106, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1], [121, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1], [57, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1]]
|
| 1248 |
[MoonEP selfcheck 4/4] dropped groups empty OK | covered rows [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 128, 129, 130, 131, 132, 133, 134, 135, 136, 137, 138, 139, 140, 141, 142, 143] sizes [2304, 128, 128, 512, 128, 12416, 128, 1152, 1280, 16768, 640, 29056, 128, 128, 128, 1664, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0] | duplicated experts [[-1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1], [118, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1], [11, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1], [118, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1], [39, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1], [11, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1], [73, 79, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1], [86, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1]]
|
| 1249 |
+
[2026-08-30 01:33:22] iteration 1/ 12 | consumed samples: 8 | elapsed time per iteration (ms): 20983.8 | throughput per GPU (TFLOP/s/GPU): 8.6 | learning rate: 2.000000E-05 | global batch size: 8 | lm loss: 1.115993E+01 | loss scale: 1.0 | grad norm: 44.807 | number of skipped iterations: 0 | number of nan iterations: 0 |
|
| 1250 |
Number of parameters in transformer block in billions: 29.85
|
| 1251 |
Number of parameters in embedding layers in billions: 0.41
|
| 1252 |
Total number of parameters in billions: 30.26
|
| 1253 |
Number of parameters in most loaded shard in billions: 30.2590
|
| 1254 |
Theoretical memory footprints: weight and optimizer=216428.96 MB
|
| 1255 |
[Rank 0] (after 1 iterations) memory (MB) | allocated: 67239.548828125 | max allocated: 67239.54931640625 | reserved: 68548.0 | max reserved: 68548.0
|
| 1256 |
+
[2026-08-30 01:33:22] iteration 2/ 12 | consumed samples: 16 | elapsed time per iteration (ms): 359.1 | throughput per GPU (TFLOP/s/GPU): 503.6 | learning rate: 4.000000E-05 | global batch size: 8 | lm loss: 1.117600E+01 | loss scale: 1.0 | grad norm: 44.337 | number of skipped iterations: 0 | number of nan iterations: 0 |
|
| 1257 |
+
[2026-08-30 01:33:22] iteration 3/ 12 | consumed samples: 24 | elapsed time per iteration (ms): 330.3 | throughput per GPU (TFLOP/s/GPU): 547.5 | learning rate: 6.000000E-05 | global batch size: 8 | lm loss: 9.685940E+00 | loss scale: 1.0 | grad norm: 62.876 | number of skipped iterations: 0 | number of nan iterations: 0 |
|
| 1258 |
Capture CUDA graph for training!!!
|
| 1259 |
CUDA graph capture done!!!
|
| 1260 |
+
[2026-08-30 01:33:23] iteration 4/ 12 | consumed samples: 32 | elapsed time per iteration (ms): 958.6 | throughput per GPU (TFLOP/s/GPU): 188.6 | learning rate: 8.000000E-05 | global batch size: 8 | lm loss: 1.338060E+01 | loss scale: 1.0 | grad norm: 20.309 | number of skipped iterations: 0 | number of nan iterations: 0 |
|
| 1261 |
+
[2026-08-30 01:33:24] iteration 5/ 12 | consumed samples: 40 | elapsed time per iteration (ms): 318.8 | throughput per GPU (TFLOP/s/GPU): 567.1 | learning rate: 1.000000E-04 | global batch size: 8 | lm loss: 1.050054E+01 | loss scale: 1.0 | grad norm: 95.222 | number of skipped iterations: 0 | number of nan iterations: 0 |
|
| 1262 |
+
[2026-08-30 01:33:24] iteration 6/ 12 | consumed samples: 48 | elapsed time per iteration (ms): 318.0 | throughput per GPU (TFLOP/s/GPU): 568.6 | learning rate: 7.750000E-05 | global batch size: 8 | lm loss: 1.042314E+01 | loss scale: 1.0 | grad norm: 50.787 | number of skipped iterations: 0 | number of nan iterations: 0 |
|
| 1263 |
+
[2026-08-30 01:33:24] iteration 7/ 12 | consumed samples: 56 | elapsed time per iteration (ms): 316.6 | throughput per GPU (TFLOP/s/GPU): 571.1 | learning rate: 3.250000E-05 | global batch size: 8 | lm loss: 1.026939E+01 | loss scale: 1.0 | grad norm: 6.314 | number of skipped iterations: 0 | number of nan iterations: 0 |
|
| 1264 |
+
[2026-08-30 01:33:25] iteration 8/ 12 | consumed samples: 64 | elapsed time per iteration (ms): 317.1 | throughput per GPU (TFLOP/s/GPU): 570.2 | learning rate: 1.000000E-05 | global batch size: 8 | lm loss: 9.807120E+00 | loss scale: 1.0 | grad norm: 5.549 | number of skipped iterations: 0 | number of nan iterations: 0 |
|
| 1265 |
+
[2026-08-30 01:33:25] iteration 9/ 12 | consumed samples: 72 | elapsed time per iteration (ms): 316.5 | throughput per GPU (TFLOP/s/GPU): 571.4 | learning rate: 1.000000E-05 | global batch size: 8 | lm loss: 9.543880E+00 | loss scale: 1.0 | grad norm: 5.309 | number of skipped iterations: 0 | number of nan iterations: 0 |
|
| 1266 |
+
[2026-08-30 01:33:25] iteration 10/ 12 | consumed samples: 80 | elapsed time per iteration (ms): 319.1 | throughput per GPU (TFLOP/s/GPU): 566.7 | learning rate: 1.000000E-05 | global batch size: 8 | lm loss: 9.444735E+00 | loss scale: 1.0 | grad norm: 5.055 | number of skipped iterations: 0 | number of nan iterations: 0 |
|
| 1267 |
+
[2026-08-30 01:33:25] iteration 11/ 12 | consumed samples: 88 | elapsed time per iteration (ms): 319.8 | throughput per GPU (TFLOP/s/GPU): 565.5 | learning rate: 1.000000E-05 | global batch size: 8 | lm loss: 9.322056E+00 | loss scale: 1.0 | grad norm: 4.732 | number of skipped iterations: 0 | number of nan iterations: 0 |
|
| 1268 |
+
[2026-08-30 01:33:26] iteration 12/ 12 | consumed samples: 96 | elapsed time per iteration (ms): 317.7 | throughput per GPU (TFLOP/s/GPU): 569.1 | learning rate: 1.000000E-05 | global batch size: 8 | lm loss: 9.206485E+00 | loss scale: 1.0 | grad norm: 4.466 | number of skipped iterations: 0 | number of nan iterations: 0 |
|
| 1269 |
+
[after training is done] datetime: 2026-08-30 01:33:26
|
| 1270 |
+
[rank0]:[W830 01:33:27.749790371 ProcessGroupNCCL.cpp:1505] Warning: WARNING: destroy_process_group() was not called before program exit, which can leak resources. For more info, please see https://pytorch.org/docs/stable/distributed.html#shutdown (function operator())
|
| 1271 |
+
[rank6]:[W830 01:33:28.189206623 ProcessGroupNCCL.cpp:1505] Warning: WARNING: destroy_process_group() was not called before program exit, which can leak resources. For more info, please see https://pytorch.org/docs/stable/distributed.html#shutdown (function operator())
|
| 1272 |
+
[rank3]:[W830 01:33:28.191561772 ProcessGroupNCCL.cpp:1505] Warning: WARNING: destroy_process_group() was not called before program exit, which can leak resources. For more info, please see https://pytorch.org/docs/stable/distributed.html#shutdown (function operator())
|
| 1273 |
+
[rank2]:[W830 01:33:28.192020147 ProcessGroupNCCL.cpp:1505] Warning: WARNING: destroy_process_group() was not called before program exit, which can leak resources. For more info, please see https://pytorch.org/docs/stable/distributed.html#shutdown (function operator())
|
| 1274 |
+
[rank5]:[W830 01:33:28.201417052 ProcessGroupNCCL.cpp:1505] Warning: WARNING: destroy_process_group() was not called before program exit, which can leak resources. For more info, please see https://pytorch.org/docs/stable/distributed.html#shutdown (function operator())
|
| 1275 |
+
[rank4]:[W830 01:33:29.333803578 ProcessGroupNCCL.cpp:1505] Warning: WARNING: destroy_process_group() was not called before program exit, which can leak resources. For more info, please see https://pytorch.org/docs/stable/distributed.html#shutdown (function operator())
|
| 1276 |
+
[rank7]:[W830 01:33:29.413464158 ProcessGroupNCCL.cpp:1505] Warning: WARNING: destroy_process_group() was not called before program exit, which can leak resources. For more info, please see https://pytorch.org/docs/stable/distributed.html#shutdown (function operator())
|
| 1277 |
+
[rank1]:[W830 01:33:29.627573333 ProcessGroupNCCL.cpp:1505] Warning: WARNING: destroy_process_group() was not called before program exit, which can leak resources. For more info, please see https://pytorch.org/docs/stable/distributed.html#shutdown (function operator())
|
| 1278 |
+
W0830 01:33:33.630000 382773 torch/distributed/elastic/agent/server/api.py:719] Received 15 death signal, shutting down workers
|
| 1279 |
+
W0830 01:33:33.631000 382773 torch/distributed/elastic/multiprocessing/api.py:900] Sending process 382951 closing signal SIGTERM
|
| 1280 |
+
W0830 01:33:33.631000 382773 torch/distributed/elastic/multiprocessing/api.py:900] Sending process 382952 closing signal SIGTERM
|
| 1281 |
+
W0830 01:33:33.631000 382773 torch/distributed/elastic/multiprocessing/api.py:900] Sending process 382953 closing signal SIGTERM
|
| 1282 |
+
W0830 01:33:33.632000 382773 torch/distributed/elastic/multiprocessing/api.py:900] Sending process 382955 closing signal SIGTERM
|
| 1283 |
+
W0830 01:33:33.632000 382773 torch/distributed/elastic/multiprocessing/api.py:900] Sending process 382956 closing signal SIGTERM
|
| 1284 |
+
W0830 01:33:33.634000 382773 torch/distributed/elastic/multiprocessing/api.py:900] Sending process 382958 closing signal SIGTERM
|
|
|
|
|
|
|
| 1285 |
Traceback (most recent call last):
|
| 1286 |
File "/usr/local/bin/torchrun", line 8, in <module>
|
| 1287 |
sys.exit(main())
|
|
|
|
| 1309 |
time.sleep(monitor_interval)
|
| 1310 |
File "/usr/local/lib/python3.12/dist-packages/torch/distributed/elastic/multiprocessing/api.py", line 84, in _terminate_process_handler
|
| 1311 |
raise SignalException(f"Process {os.getpid()} got signal: {sigval}", sigval=sigval)
|
| 1312 |
+
torch.distributed.elastic.multiprocessing.api.SignalException: Process 382773 got signal: 15
|
tierb-fixed-moonep-full/moonep/tierb/tierb_gbs8/QWEN3_235B__moonep__matched__seq8192__gbs8__capnone__async/rep3/mem_samples.csv
CHANGED
|
@@ -50,7 +50,7 @@
|
|
| 50 |
1, 4
|
| 51 |
2, 4
|
| 52 |
3, 4
|
| 53 |
-
4,
|
| 54 |
5, 1
|
| 55 |
6, 1
|
| 56 |
7, 1
|
|
@@ -62,86 +62,78 @@
|
|
| 62 |
5, 4
|
| 63 |
6, 4
|
| 64 |
7, 4
|
| 65 |
-
0,
|
| 66 |
-
1,
|
| 67 |
-
2,
|
| 68 |
-
3,
|
| 69 |
-
4,
|
| 70 |
-
5,
|
| 71 |
-
6,
|
| 72 |
-
7,
|
| 73 |
-
0,
|
| 74 |
-
1,
|
| 75 |
-
2,
|
| 76 |
-
3,
|
| 77 |
-
4,
|
| 78 |
-
5,
|
| 79 |
-
6,
|
| 80 |
-
7,
|
| 81 |
-
0,
|
| 82 |
-
1,
|
| 83 |
-
2,
|
| 84 |
-
3,
|
| 85 |
-
4,
|
| 86 |
-
5,
|
| 87 |
-
6,
|
| 88 |
-
7,
|
| 89 |
-
0,
|
| 90 |
-
1,
|
| 91 |
-
2,
|
| 92 |
-
3,
|
| 93 |
-
4,
|
| 94 |
-
5,
|
| 95 |
-
6,
|
| 96 |
-
7,
|
| 97 |
-
0,
|
| 98 |
-
1,
|
| 99 |
-
2,
|
| 100 |
-
3,
|
| 101 |
-
4,
|
| 102 |
-
5,
|
| 103 |
-
6,
|
| 104 |
-
7,
|
| 105 |
-
0,
|
| 106 |
-
1,
|
| 107 |
-
2,
|
| 108 |
-
3,
|
| 109 |
-
4,
|
| 110 |
-
5,
|
| 111 |
-
6,
|
| 112 |
-
7,
|
| 113 |
-
0, 59185
|
| 114 |
-
1, 59185
|
| 115 |
-
2, 59185
|
| 116 |
-
3, 59185
|
| 117 |
-
4, 58897
|
| 118 |
-
5, 59185
|
| 119 |
-
6, 59185
|
| 120 |
-
7, 59187
|
| 121 |
0, 59731
|
| 122 |
1, 59731
|
| 123 |
-
2,
|
| 124 |
3, 59731
|
| 125 |
4, 59443
|
| 126 |
5, 59731
|
| 127 |
6, 59731
|
| 128 |
7, 59733
|
| 129 |
0, 59733
|
| 130 |
-
1,
|
| 131 |
2, 59733
|
| 132 |
3, 59733
|
| 133 |
4, 59445
|
| 134 |
5, 59733
|
| 135 |
-
6,
|
| 136 |
7, 59733
|
| 137 |
-
0,
|
| 138 |
-
1,
|
| 139 |
-
2,
|
| 140 |
-
3,
|
| 141 |
-
4,
|
| 142 |
-
5,
|
| 143 |
-
6,
|
| 144 |
-
7,
|
| 145 |
0, 87633
|
| 146 |
1, 87673
|
| 147 |
2, 87673
|
|
@@ -150,75 +142,75 @@
|
|
| 150 |
5, 87673
|
| 151 |
6, 87673
|
| 152 |
7, 87673
|
| 153 |
-
0,
|
| 154 |
-
1,
|
| 155 |
-
2,
|
| 156 |
-
3,
|
| 157 |
-
4,
|
| 158 |
-
5,
|
| 159 |
-
6,
|
| 160 |
-
7,
|
| 161 |
-
0,
|
| 162 |
-
1,
|
| 163 |
-
2,
|
| 164 |
-
3,
|
| 165 |
-
4,
|
| 166 |
-
5,
|
| 167 |
-
6,
|
| 168 |
-
7,
|
| 169 |
-
0,
|
| 170 |
-
1,
|
| 171 |
-
2,
|
| 172 |
-
3,
|
| 173 |
-
4,
|
| 174 |
-
5,
|
| 175 |
-
6,
|
| 176 |
-
7,
|
| 177 |
-
0,
|
| 178 |
-
1,
|
| 179 |
-
2,
|
| 180 |
-
3,
|
| 181 |
-
4,
|
| 182 |
-
5,
|
| 183 |
-
6,
|
| 184 |
-
7,
|
| 185 |
-
0,
|
| 186 |
-
1,
|
| 187 |
-
2,
|
| 188 |
-
3,
|
| 189 |
-
4,
|
| 190 |
-
5,
|
| 191 |
-
6,
|
| 192 |
-
7,
|
| 193 |
-
0,
|
| 194 |
-
1,
|
| 195 |
-
2,
|
| 196 |
-
3,
|
| 197 |
-
4,
|
| 198 |
-
5,
|
| 199 |
-
6,
|
| 200 |
-
7,
|
| 201 |
-
0,
|
| 202 |
-
1,
|
| 203 |
-
2,
|
| 204 |
-
3,
|
| 205 |
-
4,
|
| 206 |
-
5,
|
| 207 |
-
6,
|
| 208 |
-
7,
|
| 209 |
-
0,
|
| 210 |
-
1,
|
| 211 |
-
2,
|
| 212 |
-
3,
|
| 213 |
-
4,
|
| 214 |
-
5,
|
| 215 |
-
6,
|
| 216 |
-
7,
|
| 217 |
0, 1
|
| 218 |
-
1,
|
| 219 |
-
2,
|
| 220 |
-
3,
|
| 221 |
-
4,
|
| 222 |
5, 4
|
| 223 |
6, 4
|
| 224 |
7, 4
|
|
|
|
| 50 |
1, 4
|
| 51 |
2, 4
|
| 52 |
3, 4
|
| 53 |
+
4, 2
|
| 54 |
5, 1
|
| 55 |
6, 1
|
| 56 |
7, 1
|
|
|
|
| 62 |
5, 4
|
| 63 |
6, 4
|
| 64 |
7, 4
|
| 65 |
+
0, 819
|
| 66 |
+
1, 819
|
| 67 |
+
2, 819
|
| 68 |
+
3, 819
|
| 69 |
+
4, 819
|
| 70 |
+
5, 819
|
| 71 |
+
6, 819
|
| 72 |
+
7, 819
|
| 73 |
+
0, 1537
|
| 74 |
+
1, 1537
|
| 75 |
+
2, 1537
|
| 76 |
+
3, 1537
|
| 77 |
+
4, 1537
|
| 78 |
+
5, 1537
|
| 79 |
+
6, 1537
|
| 80 |
+
7, 1537
|
| 81 |
+
0, 2551
|
| 82 |
+
1, 2551
|
| 83 |
+
2, 2551
|
| 84 |
+
3, 2551
|
| 85 |
+
4, 2551
|
| 86 |
+
5, 2551
|
| 87 |
+
6, 2551
|
| 88 |
+
7, 2551
|
| 89 |
+
0, 46711
|
| 90 |
+
1, 46701
|
| 91 |
+
2, 46709
|
| 92 |
+
3, 46689
|
| 93 |
+
4, 46711
|
| 94 |
+
5, 46711
|
| 95 |
+
6, 46711
|
| 96 |
+
7, 46711
|
| 97 |
+
0, 56781
|
| 98 |
+
1, 56813
|
| 99 |
+
2, 56805
|
| 100 |
+
3, 56843
|
| 101 |
+
4, 56537
|
| 102 |
+
5, 56875
|
| 103 |
+
6, 56909
|
| 104 |
+
7, 56871
|
| 105 |
+
0, 58753
|
| 106 |
+
1, 58717
|
| 107 |
+
2, 58755
|
| 108 |
+
3, 58787
|
| 109 |
+
4, 58473
|
| 110 |
+
5, 58753
|
| 111 |
+
6, 58781
|
| 112 |
+
7, 58741
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 113 |
0, 59731
|
| 114 |
1, 59731
|
| 115 |
+
2, 59733
|
| 116 |
3, 59731
|
| 117 |
4, 59443
|
| 118 |
5, 59731
|
| 119 |
6, 59731
|
| 120 |
7, 59733
|
| 121 |
0, 59733
|
| 122 |
+
1, 59733
|
| 123 |
2, 59733
|
| 124 |
3, 59733
|
| 125 |
4, 59445
|
| 126 |
5, 59733
|
| 127 |
+
6, 59733
|
| 128 |
7, 59733
|
| 129 |
+
0, 85347
|
| 130 |
+
1, 85387
|
| 131 |
+
2, 85387
|
| 132 |
+
3, 85387
|
| 133 |
+
4, 85347
|
| 134 |
+
5, 85387
|
| 135 |
+
6, 85387
|
| 136 |
+
7, 85387
|
| 137 |
0, 87633
|
| 138 |
1, 87673
|
| 139 |
2, 87673
|
|
|
|
| 142 |
5, 87673
|
| 143 |
6, 87673
|
| 144 |
7, 87673
|
| 145 |
+
0, 87633
|
| 146 |
+
1, 87673
|
| 147 |
+
2, 87673
|
| 148 |
+
3, 87673
|
| 149 |
+
4, 87633
|
| 150 |
+
5, 87673
|
| 151 |
+
6, 87673
|
| 152 |
+
7, 87673
|
| 153 |
+
0, 88039
|
| 154 |
+
1, 88079
|
| 155 |
+
2, 88079
|
| 156 |
+
3, 88079
|
| 157 |
+
4, 88039
|
| 158 |
+
5, 88079
|
| 159 |
+
6, 88079
|
| 160 |
+
7, 88079
|
| 161 |
+
0, 97463
|
| 162 |
+
1, 97489
|
| 163 |
+
2, 97505
|
| 164 |
+
3, 97503
|
| 165 |
+
4, 97421
|
| 166 |
+
5, 97551
|
| 167 |
+
6, 97573
|
| 168 |
+
7, 97533
|
| 169 |
+
0, 120273
|
| 170 |
+
1, 120309
|
| 171 |
+
2, 120309
|
| 172 |
+
3, 120309
|
| 173 |
+
4, 120309
|
| 174 |
+
5, 120309
|
| 175 |
+
6, 120309
|
| 176 |
+
7, 120309
|
| 177 |
+
0, 120273
|
| 178 |
+
1, 120309
|
| 179 |
+
2, 120309
|
| 180 |
+
3, 120309
|
| 181 |
+
4, 120309
|
| 182 |
+
5, 120309
|
| 183 |
+
6, 120309
|
| 184 |
+
7, 120309
|
| 185 |
+
0, 119045
|
| 186 |
+
1, 119663
|
| 187 |
+
2, 119663
|
| 188 |
+
3, 119663
|
| 189 |
+
4, 119663
|
| 190 |
+
5, 119663
|
| 191 |
+
6, 119337
|
| 192 |
+
7, 119017
|
| 193 |
+
0, 117077
|
| 194 |
+
1, 118083
|
| 195 |
+
2, 117433
|
| 196 |
+
3, 118083
|
| 197 |
+
4, 117757
|
| 198 |
+
5, 117437
|
| 199 |
+
6, 117437
|
| 200 |
+
7, 116787
|
| 201 |
+
0, 92867
|
| 202 |
+
1, 115757
|
| 203 |
+
2, 94633
|
| 204 |
+
3, 115757
|
| 205 |
+
4, 115757
|
| 206 |
+
5, 115757
|
| 207 |
+
6, 110761
|
| 208 |
+
7, 4
|
| 209 |
0, 1
|
| 210 |
+
1, 4
|
| 211 |
+
2, 4
|
| 212 |
+
3, 114027
|
| 213 |
+
4, 4
|
| 214 |
5, 4
|
| 215 |
6, 4
|
| 216 |
7, 4
|
tierb-fixed-moonep-full/moonep/tierb/tierb_gbs8/QWEN3_235B__moonep__matched__seq8192__gbs8__capnone__async/rep3/run.env
CHANGED
|
@@ -3,7 +3,7 @@ config=QWEN3_235B
|
|
| 3 |
family=tierB
|
| 4 |
repo=/workspace/Megatron-MoE-Echo
|
| 5 |
branch=moonep_backend_async_grad_reduce
|
| 6 |
-
commit=
|
| 7 |
moonep_source_dir=/workspace/MoonEP
|
| 8 |
moonep_source_branch=megatron-ddp-symmetric-params
|
| 9 |
moonep_source_commit=d8bc666
|
|
|
|
| 3 |
family=tierB
|
| 4 |
repo=/workspace/Megatron-MoE-Echo
|
| 5 |
branch=moonep_backend_async_grad_reduce
|
| 6 |
+
commit=2e68c8ad
|
| 7 |
moonep_source_dir=/workspace/MoonEP
|
| 8 |
moonep_source_branch=megatron-ddp-symmetric-params
|
| 9 |
moonep_source_commit=d8bc666
|
tierb-fixed-moonep-full/moonep/tierb/tierb_gbs8/QWEN3_235B__moonep__matched__seq8192__gbs8__capnone__async/rep3/train.log
CHANGED
|
@@ -1,25 +1,23 @@
|
|
| 1 |
-
[
|
| 2 |
==============================================================
|
| 3 |
QWEN3_235B / moonep (slots/rank=16 policy=matched)
|
| 4 |
E=128 K=8 H'=1536 m=16
|
| 5 |
L=12 seq=8192 gbs=8 (1 microbatches)
|
| 6 |
cap_mode=none cap=none cg=1 recompute=none
|
| 7 |
memory_dump=none snapshots=/workspace/tierb-fixed-moonep-full/moonep/tierb/tierb_gbs8/QWEN3_235B__moonep__matched__seq8192__gbs8__capnone__async/rep3/memory_snapshots/snapshot.pickle
|
| 8 |
-
repo=/workspace/Megatron-MoE-Echo @
|
| 9 |
==============================================================
|
| 10 |
-
|
| 11 |
-
|
| 12 |
-
|
| 13 |
-
|
| 14 |
/workspace/Megatron-MoE-Echo/megatron/training/arguments.py:831: UserWarning: Disabling sequence parallelism because tensor model parallelism is disabled
|
| 15 |
warnings.warn("Disabling sequence parallelism because tensor model parallelism is disabled")
|
|
|
|
| 16 |
/workspace/Megatron-MoE-Echo/megatron/training/arguments.py:831: UserWarning: Disabling sequence parallelism because tensor model parallelism is disabled
|
| 17 |
warnings.warn("Disabling sequence parallelism because tensor model parallelism is disabled")
|
| 18 |
/workspace/Megatron-MoE-Echo/megatron/training/arguments.py:831: UserWarning: Disabling sequence parallelism because tensor model parallelism is disabled
|
| 19 |
warnings.warn("Disabling sequence parallelism because tensor model parallelism is disabled")
|
| 20 |
-
/workspace/Megatron-MoE-Echo/megatron/training/arguments.py:831: UserWarning: Disabling sequence parallelism because tensor model parallelism is disabled
|
| 21 |
-
warnings.warn("Disabling sequence parallelism because tensor model parallelism is disabled")
|
| 22 |
-
WARNING: one_logger package is required to enable e2e metrics tracking. please go to https://confluence.nvidia.com/display/MLWFO/Package+Repositories for details to install it
|
| 23 |
/workspace/Megatron-MoE-Echo/megatron/training/arguments.py:831: UserWarning: Disabling sequence parallelism because tensor model parallelism is disabled
|
| 24 |
warnings.warn("Disabling sequence parallelism because tensor model parallelism is disabled")
|
| 25 |
/workspace/Megatron-MoE-Echo/megatron/training/arguments.py:831: UserWarning: Disabling sequence parallelism because tensor model parallelism is disabled
|
|
@@ -574,54 +572,56 @@ using torch.bfloat16 for parameters ...
|
|
| 574 |
-------------------- end of arguments ---------------------
|
| 575 |
INFO:megatron.core.num_microbatches_calculator:setting number of microbatches to constant 1
|
| 576 |
> building GPT2BPETokenizer tokenizer ...
|
|
|
|
|
|
|
| 577 |
> padded vocab (size: 50257) with 47 dummy tokens (new size: 50304)
|
| 578 |
WARNING:megatron.core.rerun_state_machine:RerunStateMachine initialized in mode RerunMode.VALIDATE_RESULTS
|
| 579 |
> initializing torch distributed ...
|
| 580 |
[Gloo] Rank 0 is connected to 7 peer ranks. Expected number of connected peer ranks is : 7
|
|
|
|
| 581 |
[Gloo] Rank 1 is connected to 7 peer ranks. Expected number of connected peer ranks is : 7
|
| 582 |
[Gloo] Rank 2 is connected to 7 peer ranks. Expected number of connected peer ranks is : 7
|
| 583 |
-
[Gloo] Rank 3 is connected to 7 peer ranks. Expected number of connected peer ranks is : 7
|
| 584 |
[Gloo] Rank 4 is connected to 7 peer ranks. Expected number of connected peer ranks is : 7
|
| 585 |
-
[Gloo] Rank 5 is connected to 7 peer ranks. Expected number of connected peer ranks is : 7
|
| 586 |
-
[Gloo] Rank
|
| 587 |
7
|
| 588 |
-
[Gloo] Rank
|
| 589 |
-
|
| 590 |
-
7 peer ranks. Expected number of connected peer ranks is : 7
|
| 591 |
-
[Gloo] Rank
|
| 592 |
-
|
| 593 |
[Gloo] Rank 3 is connected to 7 peer ranks. Expected number of connected peer ranks is : 7
|
| 594 |
-
[Gloo] Rank [Gloo] Rank 7 is connected to 67 is connected to
|
| 595 |
7
|
| 596 |
[Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
|
| 597 |
> initialized tensor model parallel with size 1
|
| 598 |
> initialized pipeline model parallel with size 1
|
| 599 |
> setting random seeds to 1234 ...
|
| 600 |
-
[Gloo] Rank [Gloo] Rank
|
| 601 |
-
0
|
| 602 |
|
|
|
|
|
|
|
|
|
|
| 603 |
[Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
|
| 604 |
[Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
|
| 605 |
-
[
|
| 606 |
-
[
|
| 607 |
-
[
|
| 608 |
-
[rank4]:[
|
| 609 |
-
[
|
| 610 |
-
[
|
| 611 |
-
[
|
| 612 |
-
[rank1]:[W829 23:47:52.066710093 ProcessGroupNCCL.cpp:4789] [PG ID 0 PG GUID 0 Rank 1] using GPU 1 as device used by this process is currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect. You can specify device_id in init_process_group() to force use of a particular device.
|
| 613 |
-
[rank7]:[W829 23:47:52.068582353 ProcessGroupNCCL.cpp:4789] [PG ID 0 PG GUID 0 Rank 7] using GPU 7 as device used by this process is currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect. You can specify device_id in init_process_group() to force use of a particular device.
|
| 614 |
> compiling dataset index builder ...
|
| 615 |
make: Entering directory '/workspace/Megatron-MoE-Echo/megatron/core/datasets'
|
| 616 |
make: Nothing to be done for 'default'.
|
| 617 |
make: Leaving directory '/workspace/Megatron-MoE-Echo/megatron/core/datasets'
|
| 618 |
-
>>> done with dataset index builder. Compilation time: 0.
|
| 619 |
WARNING: constraints for invoking optimized fused softmax kernel are not met. We default back to unfused kernel invocations.
|
| 620 |
> compiling and loading fused kernels ...
|
| 621 |
-
[rank0]:[
|
| 622 |
-
>>> done with compiling and loading fused kernels. Compilation time: 3.
|
| 623 |
-
time to initialize megatron (seconds): 7.
|
| 624 |
-
[after megatron is initialized] datetime: 2026-08-
|
| 625 |
building GPT model ...
|
| 626 |
/usr/local/lib/python3.12/dist-packages/transformer_engine/pytorch/cpu_offload.py:708: DeprecationWarning: Offloading weights is deprecated. Using offload_weights=True does not have any effect.
|
| 627 |
warnings.warn(
|
|
@@ -645,36 +645,36 @@ INFO:megatron.core.distributed.param_and_grad_buffer:Number of buckets for gradi
|
|
| 645 |
Params for bucket 1 (206045184 elements, 206045184 padded size):
|
| 646 |
module.output_layer.weight
|
| 647 |
Params for bucket 2 (71840000 elements, 71840000 padded size):
|
| 648 |
-
module.decoder.layers.11.pre_mlp_layernorm.weight
|
| 649 |
module.decoder.layers.11.self_attention.q_layernorm.weight
|
|
|
|
| 650 |
module.decoder.layers.11.mlp.router.weight
|
| 651 |
module.decoder.layers.11.self_attention.k_layernorm.weight
|
| 652 |
module.decoder.layers.11.self_attention.linear_qkv.weight
|
| 653 |
module.decoder.layers.11.self_attention.linear_qkv.layer_norm_weight
|
| 654 |
-
module.decoder.final_layernorm.weight
|
| 655 |
module.decoder.layers.11.self_attention.linear_proj.weight
|
|
|
|
| 656 |
Params for bucket 3 (71835904 elements, 71835904 padded size):
|
|
|
|
| 657 |
module.decoder.layers.10.self_attention.linear_qkv.weight
|
|
|
|
| 658 |
module.decoder.layers.10.self_attention.linear_qkv.layer_norm_weight
|
| 659 |
module.decoder.layers.10.mlp.router.weight
|
| 660 |
-
module.decoder.layers.10.pre_mlp_layernorm.weight
|
| 661 |
module.decoder.layers.10.self_attention.k_layernorm.weight
|
| 662 |
-
module.decoder.layers.10.self_attention.q_layernorm.weight
|
| 663 |
module.decoder.layers.10.self_attention.linear_proj.weight
|
| 664 |
Params for bucket 4 (71835904 elements, 71835904 padded size):
|
| 665 |
module.decoder.layers.9.self_attention.k_layernorm.weight
|
|
|
|
| 666 |
module.decoder.layers.9.self_attention.linear_proj.weight
|
| 667 |
module.decoder.layers.9.mlp.router.weight
|
| 668 |
module.decoder.layers.9.pre_mlp_layernorm.weight
|
| 669 |
-
module.decoder.layers.9.self_attention.linear_qkv.weight
|
| 670 |
module.decoder.layers.9.self_attention.q_layernorm.weight
|
| 671 |
module.decoder.layers.9.self_attention.linear_qkv.layer_norm_weight
|
| 672 |
Params for bucket 5 (71835904 elements, 71835904 padded size):
|
| 673 |
module.decoder.layers.8.mlp.router.weight
|
| 674 |
module.decoder.layers.8.pre_mlp_layernorm.weight
|
|
|
|
| 675 |
module.decoder.layers.8.self_attention.linear_qkv.weight
|
| 676 |
module.decoder.layers.8.self_attention.q_layernorm.weight
|
| 677 |
-
module.decoder.layers.8.self_attention.k_layernorm.weight
|
| 678 |
module.decoder.layers.8.self_attention.linear_qkv.layer_norm_weight
|
| 679 |
module.decoder.layers.8.self_attention.linear_proj.weight
|
| 680 |
Params for bucket 6 (71835904 elements, 71835904 padded size):
|
|
@@ -683,62 +683,62 @@ Params for bucket 6 (71835904 elements, 71835904 padded size):
|
|
| 683 |
module.decoder.layers.7.self_attention.k_layernorm.weight
|
| 684 |
module.decoder.layers.7.self_attention.linear_qkv.weight
|
| 685 |
module.decoder.layers.7.self_attention.q_layernorm.weight
|
| 686 |
-
module.decoder.layers.7.self_attention.linear_proj.weight
|
| 687 |
module.decoder.layers.7.self_attention.linear_qkv.layer_norm_weight
|
|
|
|
| 688 |
Params for bucket 7 (71835904 elements, 71835904 padded size):
|
| 689 |
module.decoder.layers.6.mlp.router.weight
|
| 690 |
-
module.decoder.layers.6.self_attention.q_layernorm.weight
|
| 691 |
-
module.decoder.layers.6.self_attention.linear_qkv.weight
|
| 692 |
module.decoder.layers.6.pre_mlp_layernorm.weight
|
|
|
|
| 693 |
module.decoder.layers.6.self_attention.k_layernorm.weight
|
| 694 |
module.decoder.layers.6.self_attention.linear_qkv.layer_norm_weight
|
|
|
|
| 695 |
module.decoder.layers.6.self_attention.linear_proj.weight
|
| 696 |
Params for bucket 8 (71835904 elements, 71835904 padded size):
|
| 697 |
-
module.decoder.layers.5.
|
| 698 |
module.decoder.layers.5.pre_mlp_layernorm.weight
|
|
|
|
| 699 |
module.decoder.layers.5.self_attention.linear_qkv.layer_norm_weight
|
| 700 |
-
module.decoder.layers.5.mlp.router.weight
|
| 701 |
module.decoder.layers.5.self_attention.linear_qkv.weight
|
| 702 |
module.decoder.layers.5.self_attention.k_layernorm.weight
|
| 703 |
module.decoder.layers.5.self_attention.linear_proj.weight
|
| 704 |
Params for bucket 9 (71835904 elements, 71835904 padded size):
|
| 705 |
-
module.decoder.layers.4.mlp.router.weight
|
| 706 |
module.decoder.layers.4.pre_mlp_layernorm.weight
|
| 707 |
module.decoder.layers.4.self_attention.q_layernorm.weight
|
|
|
|
|
|
|
| 708 |
module.decoder.layers.4.self_attention.k_layernorm.weight
|
| 709 |
module.decoder.layers.4.self_attention.linear_qkv.weight
|
| 710 |
-
module.decoder.layers.4.self_attention.linear_qkv.layer_norm_weight
|
| 711 |
module.decoder.layers.4.self_attention.linear_proj.weight
|
| 712 |
Params for bucket 10 (71835904 elements, 71835904 padded size):
|
| 713 |
module.decoder.layers.3.mlp.router.weight
|
| 714 |
-
module.decoder.layers.3.pre_mlp_layernorm.weight
|
| 715 |
module.decoder.layers.3.self_attention.q_layernorm.weight
|
| 716 |
module.decoder.layers.3.self_attention.linear_qkv.weight
|
|
|
|
| 717 |
module.decoder.layers.3.self_attention.k_layernorm.weight
|
| 718 |
module.decoder.layers.3.self_attention.linear_qkv.layer_norm_weight
|
| 719 |
module.decoder.layers.3.self_attention.linear_proj.weight
|
| 720 |
Params for bucket 11 (71835904 elements, 71835904 padded size):
|
| 721 |
module.decoder.layers.2.mlp.router.weight
|
|
|
|
| 722 |
module.decoder.layers.2.pre_mlp_layernorm.weight
|
| 723 |
-
module.decoder.layers.2.self_attention.k_layernorm.weight
|
| 724 |
module.decoder.layers.2.self_attention.linear_qkv.weight
|
| 725 |
-
module.decoder.layers.2.self_attention.
|
| 726 |
module.decoder.layers.2.self_attention.linear_qkv.layer_norm_weight
|
| 727 |
module.decoder.layers.2.self_attention.linear_proj.weight
|
| 728 |
Params for bucket 12 (71835904 elements, 71835904 padded size):
|
| 729 |
module.decoder.layers.1.mlp.router.weight
|
| 730 |
module.decoder.layers.1.pre_mlp_layernorm.weight
|
| 731 |
module.decoder.layers.1.self_attention.k_layernorm.weight
|
| 732 |
-
module.decoder.layers.1.self_attention.q_layernorm.weight
|
| 733 |
module.decoder.layers.1.self_attention.linear_qkv.weight
|
|
|
|
| 734 |
module.decoder.layers.1.self_attention.linear_qkv.layer_norm_weight
|
| 735 |
module.decoder.layers.1.self_attention.linear_proj.weight
|
| 736 |
Params for bucket 13 (71835904 elements, 71835904 padded size):
|
| 737 |
-
module.decoder.layers.0.
|
| 738 |
module.decoder.layers.0.self_attention.q_layernorm.weight
|
| 739 |
-
module.decoder.layers.0.pre_mlp_layernorm.weight
|
| 740 |
module.decoder.layers.0.self_attention.linear_proj.weight
|
| 741 |
-
module.decoder.layers.0.
|
|
|
|
| 742 |
module.decoder.layers.0.self_attention.linear_qkv.weight
|
| 743 |
module.decoder.layers.0.self_attention.linear_qkv.layer_norm_weight
|
| 744 |
Params for bucket 14 (206045184 elements, 206045184 padded size):
|
|
@@ -746,8 +746,8 @@ Params for bucket 14 (206045184 elements, 206045184 padded size):
|
|
| 746 |
INFO:megatron.core.distributed.param_and_grad_buffer:Number of buckets for gradient all-reduce / reduce-scatter: 76
|
| 747 |
Params for bucket 1 (44040192 elements, 44040192 padded size):
|
| 748 |
module.decoder.layers.11.mlp.experts.linear_fc2.weight0
|
| 749 |
-
module.decoder.layers.11.mlp.experts.linear_fc2.weight2
|
| 750 |
module.decoder.layers.11.mlp.experts.linear_fc2.weight4
|
|
|
|
| 751 |
module.decoder.layers.11.mlp.experts.linear_fc2.weight6
|
| 752 |
module.decoder.layers.11.mlp.experts.linear_fc2.weight3
|
| 753 |
module.decoder.layers.11.mlp.experts.linear_fc2.weight1
|
|
@@ -757,40 +757,40 @@ Params for bucket 2 (44040192 elements, 44040192 padded size):
|
|
| 757 |
module.decoder.layers.11.mlp.experts.linear_fc2.weight10
|
| 758 |
module.decoder.layers.11.mlp.experts.linear_fc2.weight12
|
| 759 |
module.decoder.layers.11.mlp.experts.linear_fc2.weight7
|
| 760 |
-
module.decoder.layers.11.mlp.experts.linear_fc2.weight9
|
| 761 |
module.decoder.layers.11.mlp.experts.linear_fc2.weight11
|
|
|
|
| 762 |
module.decoder.layers.11.mlp.experts.linear_fc2.weight13
|
| 763 |
Params for bucket 3 (50331648 elements, 50331648 padded size):
|
| 764 |
module.decoder.layers.11.mlp.experts.linear_fc2.weight14
|
| 765 |
module.decoder.layers.11.mlp.experts.linear_fc1.weight1
|
| 766 |
-
module.decoder.layers.11.mlp.experts.linear_fc2.weight15
|
| 767 |
module.decoder.layers.11.mlp.experts.linear_fc1.weight2
|
|
|
|
| 768 |
module.decoder.layers.11.mlp.experts.linear_fc1.weight0
|
| 769 |
Params for bucket 4 (50331648 elements, 50331648 padded size):
|
| 770 |
module.decoder.layers.11.mlp.experts.linear_fc1.weight3
|
| 771 |
module.decoder.layers.11.mlp.experts.linear_fc1.weight5
|
| 772 |
-
module.decoder.layers.11.mlp.experts.linear_fc1.weight6
|
| 773 |
module.decoder.layers.11.mlp.experts.linear_fc1.weight4
|
|
|
|
| 774 |
Params for bucket 5 (50331648 elements, 50331648 padded size):
|
| 775 |
module.decoder.layers.11.mlp.experts.linear_fc1.weight7
|
| 776 |
-
module.decoder.layers.11.mlp.experts.linear_fc1.weight9
|
| 777 |
module.decoder.layers.11.mlp.experts.linear_fc1.weight8
|
|
|
|
| 778 |
module.decoder.layers.11.mlp.experts.linear_fc1.weight10
|
| 779 |
Params for bucket 6 (50331648 elements, 50331648 padded size):
|
| 780 |
module.decoder.layers.11.mlp.experts.linear_fc1.weight11
|
|
|
|
| 781 |
module.decoder.layers.11.mlp.experts.linear_fc1.weight12
|
| 782 |
module.decoder.layers.11.mlp.experts.linear_fc1.weight13
|
| 783 |
-
module.decoder.layers.11.mlp.experts.linear_fc1.weight14
|
| 784 |
Params for bucket 7 (44040192 elements, 44040192 padded size):
|
| 785 |
module.decoder.layers.11.mlp.experts.linear_fc1.weight15
|
| 786 |
module.decoder.layers.10.mlp.experts.linear_fc2.weight0
|
| 787 |
module.decoder.layers.10.mlp.experts.linear_fc2.weight2
|
| 788 |
module.decoder.layers.10.mlp.experts.linear_fc2.weight4
|
| 789 |
-
module.decoder.layers.10.mlp.experts.linear_fc2.weight1
|
| 790 |
module.decoder.layers.10.mlp.experts.linear_fc2.weight3
|
|
|
|
| 791 |
Params for bucket 8 (44040192 elements, 44040192 padded size):
|
| 792 |
-
module.decoder.layers.10.mlp.experts.linear_fc2.weight8
|
| 793 |
module.decoder.layers.10.mlp.experts.linear_fc2.weight6
|
|
|
|
| 794 |
module.decoder.layers.10.mlp.experts.linear_fc2.weight10
|
| 795 |
module.decoder.layers.10.mlp.experts.linear_fc2.weight5
|
| 796 |
module.decoder.layers.10.mlp.experts.linear_fc2.weight7
|
|
@@ -798,68 +798,68 @@ Params for bucket 8 (44040192 elements, 44040192 padded size):
|
|
| 798 |
module.decoder.layers.10.mlp.experts.linear_fc2.weight11
|
| 799 |
Params for bucket 9 (50331648 elements, 50331648 padded size):
|
| 800 |
module.decoder.layers.10.mlp.experts.linear_fc2.weight12
|
|
|
|
| 801 |
module.decoder.layers.10.mlp.experts.linear_fc2.weight14
|
| 802 |
-
module.decoder.layers.10.mlp.experts.linear_fc1.weight1
|
| 803 |
module.decoder.layers.10.mlp.experts.linear_fc2.weight13
|
| 804 |
module.decoder.layers.10.mlp.experts.linear_fc2.weight15
|
| 805 |
-
module.decoder.layers.10.mlp.experts.linear_fc1.
|
| 806 |
Params for bucket 10 (50331648 elements, 50331648 padded size):
|
| 807 |
module.decoder.layers.10.mlp.experts.linear_fc1.weight2
|
| 808 |
-
module.decoder.layers.10.mlp.experts.linear_fc1.weight4
|
| 809 |
module.decoder.layers.10.mlp.experts.linear_fc1.weight5
|
| 810 |
module.decoder.layers.10.mlp.experts.linear_fc1.weight3
|
|
|
|
| 811 |
Params for bucket 11 (50331648 elements, 50331648 padded size):
|
| 812 |
module.decoder.layers.10.mlp.experts.linear_fc1.weight6
|
| 813 |
-
module.decoder.layers.10.mlp.experts.linear_fc1.weight7
|
| 814 |
module.decoder.layers.10.mlp.experts.linear_fc1.weight8
|
| 815 |
module.decoder.layers.10.mlp.experts.linear_fc1.weight9
|
|
|
|
| 816 |
Params for bucket 12 (50331648 elements, 50331648 padded size):
|
| 817 |
module.decoder.layers.10.mlp.experts.linear_fc1.weight10
|
|
|
|
| 818 |
module.decoder.layers.10.mlp.experts.linear_fc1.weight13
|
| 819 |
module.decoder.layers.10.mlp.experts.linear_fc1.weight11
|
| 820 |
-
module.decoder.layers.10.mlp.experts.linear_fc1.weight12
|
| 821 |
Params for bucket 13 (44040192 elements, 44040192 padded size):
|
| 822 |
-
module.decoder.layers.9.mlp.experts.linear_fc2.weight1
|
| 823 |
-
module.decoder.layers.10.mlp.experts.linear_fc1.weight15
|
| 824 |
module.decoder.layers.10.mlp.experts.linear_fc1.weight14
|
| 825 |
-
module.decoder.layers.9.mlp.experts.linear_fc2.weight0
|
| 826 |
module.decoder.layers.9.mlp.experts.linear_fc2.weight2
|
|
|
|
|
|
|
|
|
|
| 827 |
Params for bucket 14 (44040192 elements, 44040192 padded size):
|
| 828 |
-
module.decoder.layers.9.mlp.experts.linear_fc2.weight3
|
| 829 |
-
module.decoder.layers.9.mlp.experts.linear_fc2.weight7
|
| 830 |
-
module.decoder.layers.9.mlp.experts.linear_fc2.weight5
|
| 831 |
-
module.decoder.layers.9.mlp.experts.linear_fc2.weight9
|
| 832 |
module.decoder.layers.9.mlp.experts.linear_fc2.weight6
|
| 833 |
module.decoder.layers.9.mlp.experts.linear_fc2.weight4
|
| 834 |
module.decoder.layers.9.mlp.experts.linear_fc2.weight8
|
|
|
|
|
|
|
|
|
|
|
|
|
| 835 |
Params for bucket 15 (50331648 elements, 50331648 padded size):
|
| 836 |
-
module.decoder.layers.9.mlp.experts.linear_fc2.
|
|
|
|
|
|
|
| 837 |
module.decoder.layers.9.mlp.experts.linear_fc2.weight11
|
|
|
|
| 838 |
module.decoder.layers.9.mlp.experts.linear_fc2.weight15
|
| 839 |
module.decoder.layers.9.mlp.experts.linear_fc1.weight0
|
| 840 |
-
module.decoder.layers.9.mlp.experts.linear_fc2.weight10
|
| 841 |
-
module.decoder.layers.9.mlp.experts.linear_fc2.weight14
|
| 842 |
-
module.decoder.layers.9.mlp.experts.linear_fc2.weight12
|
| 843 |
Params for bucket 16 (50331648 elements, 50331648 padded size):
|
| 844 |
module.decoder.layers.9.mlp.experts.linear_fc1.weight1
|
| 845 |
-
module.decoder.layers.9.mlp.experts.linear_fc1.weight3
|
| 846 |
module.decoder.layers.9.mlp.experts.linear_fc1.weight4
|
| 847 |
module.decoder.layers.9.mlp.experts.linear_fc1.weight2
|
|
|
|
| 848 |
Params for bucket 17 (50331648 elements, 50331648 padded size):
|
| 849 |
module.decoder.layers.9.mlp.experts.linear_fc1.weight5
|
| 850 |
-
module.decoder.layers.9.mlp.experts.linear_fc1.weight6
|
| 851 |
module.decoder.layers.9.mlp.experts.linear_fc1.weight7
|
| 852 |
module.decoder.layers.9.mlp.experts.linear_fc1.weight8
|
|
|
|
| 853 |
Params for bucket 18 (50331648 elements, 50331648 padded size):
|
| 854 |
module.decoder.layers.9.mlp.experts.linear_fc1.weight9
|
|
|
|
| 855 |
module.decoder.layers.9.mlp.experts.linear_fc1.weight12
|
| 856 |
module.decoder.layers.9.mlp.experts.linear_fc1.weight10
|
| 857 |
-
module.decoder.layers.9.mlp.experts.linear_fc1.weight11
|
| 858 |
Params for bucket 19 (44040192 elements, 44040192 padded size):
|
| 859 |
module.decoder.layers.9.mlp.experts.linear_fc1.weight13
|
| 860 |
-
module.decoder.layers.8.mlp.experts.linear_fc2.weight0
|
| 861 |
module.decoder.layers.9.mlp.experts.linear_fc1.weight14
|
| 862 |
module.decoder.layers.9.mlp.experts.linear_fc1.weight15
|
|
|
|
| 863 |
Params for bucket 20 (44040192 elements, 44040192 padded size):
|
| 864 |
module.decoder.layers.8.mlp.experts.linear_fc2.weight1
|
| 865 |
module.decoder.layers.8.mlp.experts.linear_fc2.weight5
|
|
@@ -873,286 +873,286 @@ Params for bucket 21 (44040192 elements, 44040192 padded size):
|
|
| 873 |
module.decoder.layers.8.mlp.experts.linear_fc2.weight9
|
| 874 |
module.decoder.layers.8.mlp.experts.linear_fc2.weight13
|
| 875 |
module.decoder.layers.8.mlp.experts.linear_fc2.weight8
|
| 876 |
-
module.decoder.layers.8.mlp.experts.linear_fc2.weight10
|
| 877 |
module.decoder.layers.8.mlp.experts.linear_fc2.weight12
|
|
|
|
| 878 |
module.decoder.layers.8.mlp.experts.linear_fc2.weight14
|
| 879 |
Params for bucket 22 (44040192 elements, 44040192 padded size):
|
| 880 |
module.decoder.layers.8.mlp.experts.linear_fc2.weight15
|
| 881 |
-
module.decoder.layers.8.mlp.experts.linear_fc1.weight0
|
| 882 |
module.decoder.layers.8.mlp.experts.linear_fc1.weight2
|
|
|
|
| 883 |
module.decoder.layers.8.mlp.experts.linear_fc1.weight1
|
| 884 |
Params for bucket 23 (50331648 elements, 50331648 padded size):
|
| 885 |
module.decoder.layers.8.mlp.experts.linear_fc1.weight3
|
|
|
|
| 886 |
module.decoder.layers.8.mlp.experts.linear_fc1.weight6
|
| 887 |
module.decoder.layers.8.mlp.experts.linear_fc1.weight4
|
| 888 |
-
module.decoder.layers.8.mlp.experts.linear_fc1.weight5
|
| 889 |
Params for bucket 24 (50331648 elements, 50331648 padded size):
|
| 890 |
module.decoder.layers.8.mlp.experts.linear_fc1.weight7
|
|
|
|
| 891 |
module.decoder.layers.8.mlp.experts.linear_fc1.weight9
|
| 892 |
module.decoder.layers.8.mlp.experts.linear_fc1.weight10
|
| 893 |
-
module.decoder.layers.8.mlp.experts.linear_fc1.weight8
|
| 894 |
Params for bucket 25 (50331648 elements, 50331648 padded size):
|
| 895 |
module.decoder.layers.8.mlp.experts.linear_fc1.weight11
|
| 896 |
module.decoder.layers.8.mlp.experts.linear_fc1.weight12
|
| 897 |
module.decoder.layers.8.mlp.experts.linear_fc1.weight13
|
| 898 |
module.decoder.layers.8.mlp.experts.linear_fc1.weight14
|
| 899 |
Params for bucket 26 (44040192 elements, 44040192 padded size):
|
| 900 |
-
module.decoder.layers.7.mlp.experts.linear_fc2.weight3
|
| 901 |
-
module.decoder.layers.7.mlp.experts.linear_fc2.weight1
|
| 902 |
-
module.decoder.layers.8.mlp.experts.linear_fc1.weight15
|
| 903 |
module.decoder.layers.7.mlp.experts.linear_fc2.weight0
|
| 904 |
module.decoder.layers.7.mlp.experts.linear_fc2.weight2
|
| 905 |
module.decoder.layers.7.mlp.experts.linear_fc2.weight4
|
|
|
|
|
|
|
|
|
|
| 906 |
Params for bucket 27 (44040192 elements, 44040192 padded size):
|
|
|
|
|
|
|
|
|
|
| 907 |
module.decoder.layers.7.mlp.experts.linear_fc2.weight5
|
| 908 |
module.decoder.layers.7.mlp.experts.linear_fc2.weight7
|
| 909 |
module.decoder.layers.7.mlp.experts.linear_fc2.weight9
|
| 910 |
module.decoder.layers.7.mlp.experts.linear_fc2.weight11
|
| 911 |
-
module.decoder.layers.7.mlp.experts.linear_fc2.weight8
|
| 912 |
-
module.decoder.layers.7.mlp.experts.linear_fc2.weight6
|
| 913 |
-
module.decoder.layers.7.mlp.experts.linear_fc2.weight10
|
| 914 |
Params for bucket 28 (50331648 elements, 50331648 padded size):
|
| 915 |
-
module.decoder.layers.7.mlp.experts.linear_fc2.weight13
|
| 916 |
-
module.decoder.layers.7.mlp.experts.linear_fc2.weight15
|
| 917 |
-
module.decoder.layers.7.mlp.experts.linear_fc1.weight0
|
| 918 |
module.decoder.layers.7.mlp.experts.linear_fc2.weight12
|
| 919 |
module.decoder.layers.7.mlp.experts.linear_fc2.weight14
|
| 920 |
module.decoder.layers.7.mlp.experts.linear_fc1.weight1
|
|
|
|
|
|
|
|
|
|
| 921 |
Params for bucket 29 (50331648 elements, 50331648 padded size):
|
| 922 |
module.decoder.layers.7.mlp.experts.linear_fc1.weight2
|
| 923 |
-
module.decoder.layers.7.mlp.experts.linear_fc1.weight4
|
| 924 |
module.decoder.layers.7.mlp.experts.linear_fc1.weight5
|
|
|
|
| 925 |
module.decoder.layers.7.mlp.experts.linear_fc1.weight3
|
| 926 |
Params for bucket 30 (50331648 elements, 50331648 padded size):
|
| 927 |
module.decoder.layers.7.mlp.experts.linear_fc1.weight6
|
| 928 |
-
module.decoder.layers.7.mlp.experts.linear_fc1.weight7
|
| 929 |
module.decoder.layers.7.mlp.experts.linear_fc1.weight9
|
| 930 |
module.decoder.layers.7.mlp.experts.linear_fc1.weight8
|
|
|
|
| 931 |
Params for bucket 31 (50331648 elements, 50331648 padded size):
|
| 932 |
module.decoder.layers.7.mlp.experts.linear_fc1.weight10
|
| 933 |
module.decoder.layers.7.mlp.experts.linear_fc1.weight11
|
| 934 |
module.decoder.layers.7.mlp.experts.linear_fc1.weight12
|
| 935 |
module.decoder.layers.7.mlp.experts.linear_fc1.weight13
|
| 936 |
Params for bucket 32 (44040192 elements, 44040192 padded size):
|
|
|
|
|
|
|
| 937 |
module.decoder.layers.7.mlp.experts.linear_fc1.weight14
|
| 938 |
module.decoder.layers.6.mlp.experts.linear_fc2.weight0
|
| 939 |
module.decoder.layers.6.mlp.experts.linear_fc2.weight2
|
| 940 |
-
module.decoder.layers.7.mlp.experts.linear_fc1.weight15
|
| 941 |
-
module.decoder.layers.6.mlp.experts.linear_fc2.weight1
|
| 942 |
Params for bucket 33 (44040192 elements, 44040192 padded size):
|
| 943 |
-
module.decoder.layers.6.mlp.experts.linear_fc2.weight6
|
| 944 |
-
module.decoder.layers.6.mlp.experts.linear_fc2.weight4
|
| 945 |
-
module.decoder.layers.6.mlp.experts.linear_fc2.weight8
|
| 946 |
module.decoder.layers.6.mlp.experts.linear_fc2.weight3
|
| 947 |
-
module.decoder.layers.6.mlp.experts.linear_fc2.weight7
|
| 948 |
module.decoder.layers.6.mlp.experts.linear_fc2.weight5
|
|
|
|
| 949 |
module.decoder.layers.6.mlp.experts.linear_fc2.weight9
|
|
|
|
|
|
|
|
|
|
| 950 |
Params for bucket 34 (50331648 elements, 50331648 padded size):
|
| 951 |
-
module.decoder.layers.6.mlp.experts.linear_fc2.weight10
|
| 952 |
-
module.decoder.layers.6.mlp.experts.linear_fc2.weight12
|
| 953 |
-
module.decoder.layers.6.mlp.experts.linear_fc2.weight14
|
| 954 |
module.decoder.layers.6.mlp.experts.linear_fc2.weight13
|
| 955 |
module.decoder.layers.6.mlp.experts.linear_fc2.weight11
|
| 956 |
module.decoder.layers.6.mlp.experts.linear_fc2.weight15
|
|
|
|
|
|
|
|
|
|
| 957 |
module.decoder.layers.6.mlp.experts.linear_fc1.weight0
|
| 958 |
Params for bucket 35 (50331648 elements, 50331648 padded size):
|
| 959 |
module.decoder.layers.6.mlp.experts.linear_fc1.weight1
|
|
|
|
| 960 |
module.decoder.layers.6.mlp.experts.linear_fc1.weight3
|
| 961 |
module.decoder.layers.6.mlp.experts.linear_fc1.weight4
|
| 962 |
-
module.decoder.layers.6.mlp.experts.linear_fc1.weight2
|
| 963 |
Params for bucket 36 (50331648 elements, 50331648 padded size):
|
| 964 |
module.decoder.layers.6.mlp.experts.linear_fc1.weight5
|
| 965 |
module.decoder.layers.6.mlp.experts.linear_fc1.weight6
|
| 966 |
-
module.decoder.layers.6.mlp.experts.linear_fc1.weight7
|
| 967 |
module.decoder.layers.6.mlp.experts.linear_fc1.weight8
|
|
|
|
| 968 |
Params for bucket 37 (50331648 elements, 50331648 padded size):
|
| 969 |
module.decoder.layers.6.mlp.experts.linear_fc1.weight9
|
| 970 |
-
module.decoder.layers.6.mlp.experts.linear_fc1.weight10
|
| 971 |
-
module.decoder.layers.6.mlp.experts.linear_fc1.weight11
|
| 972 |
module.decoder.layers.6.mlp.experts.linear_fc1.weight12
|
|
|
|
|
|
|
| 973 |
Params for bucket 38 (44040192 elements, 44040192 padded size):
|
| 974 |
module.decoder.layers.6.mlp.experts.linear_fc1.weight13
|
| 975 |
module.decoder.layers.6.mlp.experts.linear_fc1.weight15
|
| 976 |
-
module.decoder.layers.6.mlp.experts.linear_fc1.weight14
|
| 977 |
module.decoder.layers.5.mlp.experts.linear_fc2.weight0
|
|
|
|
| 978 |
Params for bucket 39 (44040192 elements, 44040192 padded size):
|
| 979 |
-
module.decoder.layers.5.mlp.experts.linear_fc2.weight4
|
| 980 |
-
module.decoder.layers.5.mlp.experts.linear_fc2.weight2
|
| 981 |
-
module.decoder.layers.5.mlp.experts.linear_fc2.weight6
|
| 982 |
module.decoder.layers.5.mlp.experts.linear_fc2.weight1
|
| 983 |
-
module.decoder.layers.5.mlp.experts.linear_fc2.weight5
|
| 984 |
module.decoder.layers.5.mlp.experts.linear_fc2.weight3
|
|
|
|
| 985 |
module.decoder.layers.5.mlp.experts.linear_fc2.weight7
|
|
|
|
|
|
|
|
|
|
| 986 |
Params for bucket 40 (44040192 elements, 44040192 padded size):
|
|
|
|
|
|
|
|
|
|
| 987 |
module.decoder.layers.5.mlp.experts.linear_fc2.weight8
|
| 988 |
-
module.decoder.layers.5.mlp.experts.linear_fc2.weight12
|
| 989 |
module.decoder.layers.5.mlp.experts.linear_fc2.weight10
|
|
|
|
| 990 |
module.decoder.layers.5.mlp.experts.linear_fc2.weight14
|
| 991 |
-
module.decoder.layers.5.mlp.experts.linear_fc2.weight11
|
| 992 |
-
module.decoder.layers.5.mlp.experts.linear_fc2.weight9
|
| 993 |
-
module.decoder.layers.5.mlp.experts.linear_fc2.weight13
|
| 994 |
Params for bucket 41 (44040192 elements, 44040192 padded size):
|
| 995 |
module.decoder.layers.5.mlp.experts.linear_fc2.weight15
|
|
|
|
| 996 |
module.decoder.layers.5.mlp.experts.linear_fc1.weight1
|
| 997 |
module.decoder.layers.5.mlp.experts.linear_fc1.weight2
|
| 998 |
-
module.decoder.layers.5.mlp.experts.linear_fc1.weight0
|
| 999 |
Params for bucket 42 (50331648 elements, 50331648 padded size):
|
| 1000 |
module.decoder.layers.5.mlp.experts.linear_fc1.weight3
|
| 1001 |
-
module.decoder.layers.5.mlp.experts.linear_fc1.weight5
|
| 1002 |
module.decoder.layers.5.mlp.experts.linear_fc1.weight6
|
| 1003 |
module.decoder.layers.5.mlp.experts.linear_fc1.weight4
|
|
|
|
| 1004 |
Params for bucket 43 (50331648 elements, 50331648 padded size):
|
| 1005 |
module.decoder.layers.5.mlp.experts.linear_fc1.weight7
|
| 1006 |
-
module.decoder.layers.5.mlp.experts.linear_fc1.weight8
|
| 1007 |
module.decoder.layers.5.mlp.experts.linear_fc1.weight9
|
| 1008 |
module.decoder.layers.5.mlp.experts.linear_fc1.weight10
|
|
|
|
| 1009 |
Params for bucket 44 (50331648 elements, 50331648 padded size):
|
| 1010 |
module.decoder.layers.5.mlp.experts.linear_fc1.weight11
|
| 1011 |
-
module.decoder.layers.5.mlp.experts.linear_fc1.weight14
|
| 1012 |
module.decoder.layers.5.mlp.experts.linear_fc1.weight12
|
| 1013 |
module.decoder.layers.5.mlp.experts.linear_fc1.weight13
|
|
|
|
| 1014 |
Params for bucket 45 (44040192 elements, 44040192 padded size):
|
| 1015 |
-
module.decoder.layers.5.mlp.experts.linear_fc1.weight15
|
| 1016 |
-
module.decoder.layers.4.mlp.experts.linear_fc2.weight3
|
| 1017 |
-
module.decoder.layers.4.mlp.experts.linear_fc2.weight1
|
| 1018 |
-
module.decoder.layers.4.mlp.experts.linear_fc2.weight0
|
| 1019 |
module.decoder.layers.4.mlp.experts.linear_fc2.weight2
|
|
|
|
| 1020 |
module.decoder.layers.4.mlp.experts.linear_fc2.weight4
|
|
|
|
|
|
|
|
|
|
| 1021 |
Params for bucket 46 (44040192 elements, 44040192 padded size):
|
|
|
|
|
|
|
|
|
|
| 1022 |
module.decoder.layers.4.mlp.experts.linear_fc2.weight5
|
| 1023 |
-
module.decoder.layers.4.mlp.experts.linear_fc2.weight9
|
| 1024 |
module.decoder.layers.4.mlp.experts.linear_fc2.weight7
|
|
|
|
| 1025 |
module.decoder.layers.4.mlp.experts.linear_fc2.weight11
|
| 1026 |
-
module.decoder.layers.4.mlp.experts.linear_fc2.weight6
|
| 1027 |
-
module.decoder.layers.4.mlp.experts.linear_fc2.weight8
|
| 1028 |
-
module.decoder.layers.4.mlp.experts.linear_fc2.weight10
|
| 1029 |
Params for bucket 47 (50331648 elements, 50331648 padded size):
|
| 1030 |
-
module.decoder.layers.4.mlp.experts.linear_fc2.weight13
|
| 1031 |
-
module.decoder.layers.4.mlp.experts.linear_fc1.weight0
|
| 1032 |
-
module.decoder.layers.4.mlp.experts.linear_fc2.weight15
|
| 1033 |
module.decoder.layers.4.mlp.experts.linear_fc2.weight12
|
| 1034 |
module.decoder.layers.4.mlp.experts.linear_fc2.weight14
|
| 1035 |
module.decoder.layers.4.mlp.experts.linear_fc1.weight1
|
|
|
|
|
|
|
|
|
|
| 1036 |
Params for bucket 48 (50331648 elements, 50331648 padded size):
|
| 1037 |
module.decoder.layers.4.mlp.experts.linear_fc1.weight2
|
| 1038 |
-
module.decoder.layers.4.mlp.experts.linear_fc1.weight3
|
| 1039 |
module.decoder.layers.4.mlp.experts.linear_fc1.weight4
|
| 1040 |
module.decoder.layers.4.mlp.experts.linear_fc1.weight5
|
|
|
|
| 1041 |
Params for bucket 49 (50331648 elements, 50331648 padded size):
|
| 1042 |
module.decoder.layers.4.mlp.experts.linear_fc1.weight6
|
| 1043 |
-
module.decoder.layers.4.mlp.experts.linear_fc1.weight9
|
| 1044 |
module.decoder.layers.4.mlp.experts.linear_fc1.weight7
|
| 1045 |
module.decoder.layers.4.mlp.experts.linear_fc1.weight8
|
|
|
|
| 1046 |
Params for bucket 50 (50331648 elements, 50331648 padded size):
|
| 1047 |
module.decoder.layers.4.mlp.experts.linear_fc1.weight10
|
| 1048 |
-
module.decoder.layers.4.mlp.experts.linear_fc1.weight12
|
| 1049 |
module.decoder.layers.4.mlp.experts.linear_fc1.weight13
|
| 1050 |
module.decoder.layers.4.mlp.experts.linear_fc1.weight11
|
|
|
|
| 1051 |
Params for bucket 51 (44040192 elements, 44040192 padded size):
|
|
|
|
|
|
|
| 1052 |
module.decoder.layers.4.mlp.experts.linear_fc1.weight14
|
| 1053 |
module.decoder.layers.3.mlp.experts.linear_fc2.weight2
|
| 1054 |
module.decoder.layers.3.mlp.experts.linear_fc2.weight0
|
| 1055 |
-
module.decoder.layers.4.mlp.experts.linear_fc1.weight15
|
| 1056 |
-
module.decoder.layers.3.mlp.experts.linear_fc2.weight1
|
| 1057 |
Params for bucket 52 (44040192 elements, 44040192 padded size):
|
| 1058 |
-
module.decoder.layers.3.mlp.experts.linear_fc2.weight4
|
| 1059 |
-
module.decoder.layers.3.mlp.experts.linear_fc2.weight6
|
| 1060 |
-
module.decoder.layers.3.mlp.experts.linear_fc2.weight8
|
| 1061 |
module.decoder.layers.3.mlp.experts.linear_fc2.weight3
|
| 1062 |
module.decoder.layers.3.mlp.experts.linear_fc2.weight5
|
| 1063 |
module.decoder.layers.3.mlp.experts.linear_fc2.weight7
|
| 1064 |
module.decoder.layers.3.mlp.experts.linear_fc2.weight9
|
|
|
|
|
|
|
|
|
|
| 1065 |
Params for bucket 53 (50331648 elements, 50331648 padded size):
|
| 1066 |
-
module.decoder.layers.3.mlp.experts.linear_fc2.weight10
|
| 1067 |
-
module.decoder.layers.3.mlp.experts.linear_fc2.weight12
|
| 1068 |
-
module.decoder.layers.3.mlp.experts.linear_fc2.weight14
|
| 1069 |
-
module.decoder.layers.3.mlp.experts.linear_fc1.weight0
|
| 1070 |
-
module.decoder.layers.3.mlp.experts.linear_fc2.weight11
|
| 1071 |
module.decoder.layers.3.mlp.experts.linear_fc2.weight13
|
|
|
|
| 1072 |
module.decoder.layers.3.mlp.experts.linear_fc2.weight15
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1073 |
Params for bucket 54 (50331648 elements, 50331648 padded size):
|
| 1074 |
module.decoder.layers.3.mlp.experts.linear_fc1.weight1
|
| 1075 |
-
module.decoder.layers.3.mlp.experts.linear_fc1.weight4
|
| 1076 |
module.decoder.layers.3.mlp.experts.linear_fc1.weight2
|
|
|
|
| 1077 |
module.decoder.layers.3.mlp.experts.linear_fc1.weight3
|
| 1078 |
Params for bucket 55 (50331648 elements, 50331648 padded size):
|
| 1079 |
module.decoder.layers.3.mlp.experts.linear_fc1.weight5
|
| 1080 |
module.decoder.layers.3.mlp.experts.linear_fc1.weight8
|
| 1081 |
-
module.decoder.layers.3.mlp.experts.linear_fc1.weight7
|
| 1082 |
module.decoder.layers.3.mlp.experts.linear_fc1.weight6
|
|
|
|
| 1083 |
Params for bucket 56 (50331648 elements, 50331648 padded size):
|
| 1084 |
module.decoder.layers.3.mlp.experts.linear_fc1.weight9
|
| 1085 |
module.decoder.layers.3.mlp.experts.linear_fc1.weight11
|
| 1086 |
-
module.decoder.layers.3.mlp.experts.linear_fc1.weight10
|
| 1087 |
module.decoder.layers.3.mlp.experts.linear_fc1.weight12
|
|
|
|
| 1088 |
Params for bucket 57 (44040192 elements, 44040192 padded size):
|
| 1089 |
module.decoder.layers.3.mlp.experts.linear_fc1.weight13
|
| 1090 |
module.decoder.layers.3.mlp.experts.linear_fc1.weight14
|
| 1091 |
module.decoder.layers.3.mlp.experts.linear_fc1.weight15
|
| 1092 |
module.decoder.layers.2.mlp.experts.linear_fc2.weight0
|
| 1093 |
Params for bucket 58 (44040192 elements, 44040192 padded size):
|
| 1094 |
-
module.decoder.layers.2.mlp.experts.linear_fc2.weight4
|
| 1095 |
-
module.decoder.layers.2.mlp.experts.linear_fc2.weight2
|
| 1096 |
-
module.decoder.layers.2.mlp.experts.linear_fc2.weight6
|
| 1097 |
module.decoder.layers.2.mlp.experts.linear_fc2.weight1
|
| 1098 |
module.decoder.layers.2.mlp.experts.linear_fc2.weight3
|
| 1099 |
module.decoder.layers.2.mlp.experts.linear_fc2.weight5
|
| 1100 |
module.decoder.layers.2.mlp.experts.linear_fc2.weight7
|
|
|
|
|
|
|
|
|
|
| 1101 |
Params for bucket 59 (44040192 elements, 44040192 padded size):
|
|
|
|
|
|
|
|
|
|
| 1102 |
module.decoder.layers.2.mlp.experts.linear_fc2.weight8
|
| 1103 |
module.decoder.layers.2.mlp.experts.linear_fc2.weight12
|
| 1104 |
module.decoder.layers.2.mlp.experts.linear_fc2.weight10
|
| 1105 |
module.decoder.layers.2.mlp.experts.linear_fc2.weight14
|
| 1106 |
-
module.decoder.layers.2.mlp.experts.linear_fc2.weight9
|
| 1107 |
-
module.decoder.layers.2.mlp.experts.linear_fc2.weight11
|
| 1108 |
-
module.decoder.layers.2.mlp.experts.linear_fc2.weight13
|
| 1109 |
Params for bucket 60 (44040192 elements, 44040192 padded size):
|
| 1110 |
module.decoder.layers.2.mlp.experts.linear_fc2.weight15
|
| 1111 |
-
module.decoder.layers.2.mlp.experts.linear_fc1.weight1
|
| 1112 |
module.decoder.layers.2.mlp.experts.linear_fc1.weight2
|
| 1113 |
module.decoder.layers.2.mlp.experts.linear_fc1.weight0
|
|
|
|
| 1114 |
Params for bucket 61 (50331648 elements, 50331648 padded size):
|
| 1115 |
module.decoder.layers.2.mlp.experts.linear_fc1.weight3
|
| 1116 |
-
module.decoder.layers.2.mlp.experts.linear_fc1.weight4
|
| 1117 |
module.decoder.layers.2.mlp.experts.linear_fc1.weight5
|
| 1118 |
module.decoder.layers.2.mlp.experts.linear_fc1.weight6
|
|
|
|
| 1119 |
Params for bucket 62 (50331648 elements, 50331648 padded size):
|
| 1120 |
module.decoder.layers.2.mlp.experts.linear_fc1.weight7
|
| 1121 |
-
module.decoder.layers.2.mlp.experts.linear_fc1.weight10
|
| 1122 |
-
module.decoder.layers.2.mlp.experts.linear_fc1.weight8
|
| 1123 |
module.decoder.layers.2.mlp.experts.linear_fc1.weight9
|
|
|
|
|
|
|
| 1124 |
Params for bucket 63 (50331648 elements, 50331648 padded size):
|
| 1125 |
module.decoder.layers.2.mlp.experts.linear_fc1.weight11
|
|
|
|
| 1126 |
module.decoder.layers.2.mlp.experts.linear_fc1.weight13
|
| 1127 |
module.decoder.layers.2.mlp.experts.linear_fc1.weight14
|
| 1128 |
-
module.decoder.layers.2.mlp.experts.linear_fc1.weight12
|
| 1129 |
Params for bucket 64 (44040192 elements, 44040192 padded size):
|
| 1130 |
module.decoder.layers.2.mlp.experts.linear_fc1.weight15
|
|
|
|
|
|
|
| 1131 |
module.decoder.layers.1.mlp.experts.linear_fc2.weight2
|
| 1132 |
module.decoder.layers.1.mlp.experts.linear_fc2.weight0
|
| 1133 |
module.decoder.layers.1.mlp.experts.linear_fc2.weight4
|
| 1134 |
-
module.decoder.layers.1.mlp.experts.linear_fc2.weight3
|
| 1135 |
-
module.decoder.layers.1.mlp.experts.linear_fc2.weight1
|
| 1136 |
Params for bucket 65 (44040192 elements, 44040192 padded size):
|
| 1137 |
-
module.decoder.layers.1.mlp.experts.linear_fc2.weight8
|
| 1138 |
-
module.decoder.layers.1.mlp.experts.linear_fc2.weight6
|
| 1139 |
-
module.decoder.layers.1.mlp.experts.linear_fc2.weight10
|
| 1140 |
module.decoder.layers.1.mlp.experts.linear_fc2.weight5
|
| 1141 |
-
module.decoder.layers.1.mlp.experts.linear_fc2.weight9
|
| 1142 |
module.decoder.layers.1.mlp.experts.linear_fc2.weight7
|
|
|
|
| 1143 |
module.decoder.layers.1.mlp.experts.linear_fc2.weight11
|
|
|
|
|
|
|
|
|
|
| 1144 |
Params for bucket 66 (50331648 elements, 50331648 padded size):
|
|
|
|
|
|
|
|
|
|
| 1145 |
module.decoder.layers.1.mlp.experts.linear_fc2.weight12
|
| 1146 |
module.decoder.layers.1.mlp.experts.linear_fc2.weight14
|
| 1147 |
-
module.decoder.layers.1.mlp.experts.linear_fc1.weight0
|
| 1148 |
-
module.decoder.layers.1.mlp.experts.linear_fc2.weight15
|
| 1149 |
-
module.decoder.layers.1.mlp.experts.linear_fc2.weight13
|
| 1150 |
module.decoder.layers.1.mlp.experts.linear_fc1.weight1
|
| 1151 |
Params for bucket 67 (50331648 elements, 50331648 padded size):
|
| 1152 |
module.decoder.layers.1.mlp.experts.linear_fc1.weight2
|
| 1153 |
-
module.decoder.layers.1.mlp.experts.linear_fc1.weight3
|
| 1154 |
module.decoder.layers.1.mlp.experts.linear_fc1.weight4
|
| 1155 |
module.decoder.layers.1.mlp.experts.linear_fc1.weight5
|
|
|
|
| 1156 |
Params for bucket 68 (50331648 elements, 50331648 padded size):
|
| 1157 |
module.decoder.layers.1.mlp.experts.linear_fc1.weight6
|
| 1158 |
module.decoder.layers.1.mlp.experts.linear_fc1.weight7
|
|
@@ -1160,53 +1160,53 @@ Params for bucket 68 (50331648 elements, 50331648 padded size):
|
|
| 1160 |
module.decoder.layers.1.mlp.experts.linear_fc1.weight9
|
| 1161 |
Params for bucket 69 (50331648 elements, 50331648 padded size):
|
| 1162 |
module.decoder.layers.1.mlp.experts.linear_fc1.weight10
|
| 1163 |
-
module.decoder.layers.1.mlp.experts.linear_fc1.weight13
|
| 1164 |
module.decoder.layers.1.mlp.experts.linear_fc1.weight11
|
| 1165 |
module.decoder.layers.1.mlp.experts.linear_fc1.weight12
|
|
|
|
| 1166 |
Params for bucket 70 (44040192 elements, 44040192 padded size):
|
| 1167 |
module.decoder.layers.1.mlp.experts.linear_fc1.weight14
|
| 1168 |
module.decoder.layers.0.mlp.experts.linear_fc2.weight1
|
| 1169 |
-
module.decoder.layers.0.mlp.experts.linear_fc2.weight2
|
| 1170 |
module.decoder.layers.0.mlp.experts.linear_fc2.weight0
|
| 1171 |
module.decoder.layers.1.mlp.experts.linear_fc1.weight15
|
|
|
|
| 1172 |
Params for bucket 71 (44040192 elements, 44040192 padded size):
|
| 1173 |
module.decoder.layers.0.mlp.experts.linear_fc2.weight3
|
| 1174 |
module.decoder.layers.0.mlp.experts.linear_fc2.weight5
|
| 1175 |
module.decoder.layers.0.mlp.experts.linear_fc2.weight7
|
| 1176 |
-
module.decoder.layers.0.mlp.experts.linear_fc2.weight9
|
| 1177 |
-
module.decoder.layers.0.mlp.experts.linear_fc2.weight4
|
| 1178 |
-
module.decoder.layers.0.mlp.experts.linear_fc2.weight6
|
| 1179 |
module.decoder.layers.0.mlp.experts.linear_fc2.weight8
|
|
|
|
|
|
|
|
|
|
| 1180 |
Params for bucket 72 (50331648 elements, 50331648 padded size):
|
| 1181 |
-
module.decoder.layers.0.mlp.experts.linear_fc2.weight11
|
| 1182 |
-
module.decoder.layers.0.mlp.experts.linear_fc2.weight13
|
| 1183 |
-
module.decoder.layers.0.mlp.experts.linear_fc2.weight15
|
| 1184 |
module.decoder.layers.0.mlp.experts.linear_fc2.weight10
|
| 1185 |
module.decoder.layers.0.mlp.experts.linear_fc2.weight14
|
| 1186 |
module.decoder.layers.0.mlp.experts.linear_fc2.weight12
|
| 1187 |
module.decoder.layers.0.mlp.experts.linear_fc1.weight0
|
|
|
|
|
|
|
|
|
|
| 1188 |
Params for bucket 73 (50331648 elements, 50331648 padded size):
|
| 1189 |
module.decoder.layers.0.mlp.experts.linear_fc1.weight1
|
| 1190 |
-
module.decoder.layers.0.mlp.experts.linear_fc1.weight4
|
| 1191 |
-
module.decoder.layers.0.mlp.experts.linear_fc1.weight3
|
| 1192 |
module.decoder.layers.0.mlp.experts.linear_fc1.weight2
|
|
|
|
|
|
|
| 1193 |
Params for bucket 74 (50331648 elements, 50331648 padded size):
|
| 1194 |
module.decoder.layers.0.mlp.experts.linear_fc1.weight5
|
| 1195 |
-
module.decoder.layers.0.mlp.experts.linear_fc1.weight7
|
| 1196 |
module.decoder.layers.0.mlp.experts.linear_fc1.weight8
|
| 1197 |
module.decoder.layers.0.mlp.experts.linear_fc1.weight6
|
|
|
|
| 1198 |
Params for bucket 75 (50331648 elements, 50331648 padded size):
|
| 1199 |
module.decoder.layers.0.mlp.experts.linear_fc1.weight9
|
| 1200 |
-
module.decoder.layers.0.mlp.experts.linear_fc1.weight11
|
| 1201 |
module.decoder.layers.0.mlp.experts.linear_fc1.weight12
|
| 1202 |
module.decoder.layers.0.mlp.experts.linear_fc1.weight10
|
|
|
|
| 1203 |
Params for bucket 76 (37748736 elements, 37748736 padded size):
|
| 1204 |
module.decoder.layers.0.mlp.experts.linear_fc1.weight13
|
| 1205 |
module.decoder.layers.0.mlp.experts.linear_fc1.weight15
|
| 1206 |
module.decoder.layers.0.mlp.experts.linear_fc1.weight14
|
| 1207 |
-
INFO:megatron.core.optimizer:Setting up optimizer with config OptimizerConfig(optimizer='adam', lr=0.0001, min_lr=1e-05, decoupled_lr=None, decoupled_min_lr=None, weight_decay=0.1, fp8_recipe='mxfp8', fp16=False, bf16=True, reuse_grad_buf_for_mxfp8_param_ag=False, params_dtype=torch.bfloat16, use_precision_aware_optimizer=False, store_param_remainders=True, main_grads_dtype=torch.float32, main_params_dtype=torch.float32, exp_avg_dtype=torch.float32, exp_avg_sq_dtype=torch.float32, loss_scale=None, initial_loss_scale=4294967296, min_loss_scale=1.0, loss_scale_window=1000, hysteresis=2, adam_beta1=0.9, adam_beta2=0.999, adam_eps=1e-08, sgd_momentum=0.9, use_distributed_optimizer=True, overlap_param_gather_with_optimizer_step=False, optimizer_cpu_offload=False, optimizer_offload_fraction=1.0, use_torch_optimizer_for_cpu_offload=False, overlap_cpu_optimizer_d2h_h2d=False, pin_cpu_grads=True, pin_cpu_params=True, clip_grad=1.0, log_num_zeros_in_grad=False, barrier_with_L1_time=True, timers=<megatron.core.timers.Timers object at
|
| 1208 |
INFO:megatron.core.optimizer_param_scheduler:> learning rate decay style: cosine
|
| 1209 |
-
[after model, optimizer, and learning rate scheduler are built] datetime: 2026-08-
|
| 1210 |
> building train, validation, and test datasets ...
|
| 1211 |
> datasets target sizes (minimum size):
|
| 1212 |
train: 96
|
|
@@ -1214,7 +1214,7 @@ INFO:megatron.core.optimizer_param_scheduler:> learning rate decay style: cosine
|
|
| 1214 |
test: 0
|
| 1215 |
INFO:megatron.core.datasets.blended_megatron_dataset_config:Let split_matrix = [(0, 1.0), None, None]
|
| 1216 |
> building train, validation, and test datasets for GPT ...
|
| 1217 |
-
INFO:megatron.core.datasets.blended_megatron_dataset_builder:Building GPTDataset splits with sizes=(96, 0, 0) and config=GPTDatasetConfig(random_seed=1234, sequence_length=8192, blend=(['/workspace/data/wikitext/wikitext103_gpt2_text_document'], None), blend_per_split=None, split='100,0,0', split_matrix=[(0, 1.0), None, None], num_dataset_builder_threads=1, path_to_cache=None, mmap_bin_files=True, mock=False, tokenizer=<megatron.training.tokenizer.tokenizer._GPT2BPETokenizer object at
|
| 1218 |
INFO:megatron.core.datasets.indexed_dataset:Load the _IndexReader from /workspace/data/wikitext/wikitext103_gpt2_text_document.idx
|
| 1219 |
INFO:megatron.core.datasets.indexed_dataset: Extract the sequence lengths
|
| 1220 |
INFO:megatron.core.datasets.indexed_dataset: Extract the sequence pointers
|
|
@@ -1227,14 +1227,14 @@ INFO:megatron.core.datasets.gpt_dataset: Load the sample index from eadc3756447e
|
|
| 1227 |
INFO:megatron.core.datasets.gpt_dataset: Load the shuffle index from eadc3756447e39d609e8aa728a91f451-GPTDataset-train-shuffle_index.npy
|
| 1228 |
INFO:megatron.core.datasets.gpt_dataset:> total number of samples: 14536
|
| 1229 |
> finished creating GPT datasets ...
|
| 1230 |
-
[after dataloaders are built] datetime: 2026-08-
|
| 1231 |
done with setup ...
|
| 1232 |
training ...
|
| 1233 |
Overwriting rerun_state_machine.current_iteration from -1 to 0...
|
| 1234 |
(min, max) time across ranks (ms):
|
| 1235 |
-
model-and-optimizer-setup ......................: (
|
| 1236 |
-
train/valid/test-data-iterators-setup ..........: (
|
| 1237 |
-
[before the start of training step] datetime: 2026-08-
|
| 1238 |
INFO:moonep.api:MoonEP comm buffer sizes:
|
| 1239 |
config : R=8, S=8192, H=4096, E=128, K=8, NvS=69600, NvS_padded=69632
|
| 1240 |
sm_count : num_sms=32, num_sms_dedup=148
|
|
@@ -1246,44 +1246,42 @@ INFO:moonep.api:MoonEP comm buffer sizes:
|
|
| 1246 |
[MoonEP selfcheck 2/4] dropped groups empty OK | covered rows [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 128, 129, 130, 131, 132, 133, 134, 135, 136, 137, 138, 139, 140, 141, 142, 143] sizes [9344, 2688, 128, 24192, 128, 0, 128, 256, 128, 768, 10496, 128, 128, 2432, 128, 1536, 14208, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0] | duplicated experts [[69, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1], [88, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1], [55, 52, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1], [-1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1], [63, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1], [124, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1], [10, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1], [58, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1]]
|
| 1247 |
[MoonEP selfcheck 3/4] dropped groups empty OK | covered rows [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 128, 129, 130, 131, 132, 133, 134, 135, 136, 137, 138, 139, 140, 141, 142, 143] sizes [128, 128, 128, 0, 128, 44416, 128, 128, 128, 6912, 128, 256, 128, 128, 128, 128, 13824, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0] | duplicated experts [[78, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1], [111, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1], [62, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1], [-1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1], [57, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1], [106, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1], [121, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1], [57, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1]]
|
| 1248 |
[MoonEP selfcheck 4/4] dropped groups empty OK | covered rows [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 128, 129, 130, 131, 132, 133, 134, 135, 136, 137, 138, 139, 140, 141, 142, 143] sizes [2304, 128, 128, 512, 128, 12416, 128, 1152, 1280, 16768, 640, 29056, 128, 128, 128, 1664, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0] | duplicated experts [[-1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1], [118, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1], [11, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1], [118, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1], [39, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1], [11, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1], [73, 79, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1], [86, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1]]
|
| 1249 |
-
[2026-08-
|
| 1250 |
Number of parameters in transformer block in billions: 29.85
|
| 1251 |
Number of parameters in embedding layers in billions: 0.41
|
| 1252 |
Total number of parameters in billions: 30.26
|
| 1253 |
Number of parameters in most loaded shard in billions: 30.2590
|
| 1254 |
Theoretical memory footprints: weight and optimizer=216428.96 MB
|
| 1255 |
[Rank 0] (after 1 iterations) memory (MB) | allocated: 67239.548828125 | max allocated: 67239.54931640625 | reserved: 68548.0 | max reserved: 68548.0
|
| 1256 |
-
[2026-08-
|
| 1257 |
-
[2026-08-
|
| 1258 |
Capture CUDA graph for training!!!
|
| 1259 |
CUDA graph capture done!!!
|
| 1260 |
-
[2026-08-
|
| 1261 |
-
[2026-08-
|
| 1262 |
-
[2026-08-
|
| 1263 |
-
[2026-08-
|
| 1264 |
-
[2026-08-
|
| 1265 |
-
[2026-08-
|
| 1266 |
-
[2026-08-
|
| 1267 |
-
[2026-08-
|
| 1268 |
-
[2026-08-
|
| 1269 |
-
[after training is done] datetime: 2026-08-
|
| 1270 |
-
[rank0]:[
|
| 1271 |
-
[
|
| 1272 |
-
[rank4]:[
|
| 1273 |
-
[
|
| 1274 |
-
[rank7]:[
|
| 1275 |
-
[
|
| 1276 |
-
[
|
| 1277 |
-
[
|
| 1278 |
-
|
| 1279 |
-
|
| 1280 |
-
|
| 1281 |
-
|
| 1282 |
-
|
| 1283 |
-
|
| 1284 |
-
|
| 1285 |
-
W0829 23:48:34.085000 242741 torch/distributed/elastic/multiprocessing/api.py:900] Sending process 242937 closing signal SIGTERM
|
| 1286 |
-
W0829 23:48:34.086000 242741 torch/distributed/elastic/multiprocessing/api.py:900] Sending process 242938 closing signal SIGTERM
|
| 1287 |
Traceback (most recent call last):
|
| 1288 |
File "/usr/local/bin/torchrun", line 8, in <module>
|
| 1289 |
sys.exit(main())
|
|
@@ -1311,4 +1309,4 @@ Traceback (most recent call last):
|
|
| 1311 |
time.sleep(monitor_interval)
|
| 1312 |
File "/usr/local/lib/python3.12/dist-packages/torch/distributed/elastic/multiprocessing/api.py", line 84, in _terminate_process_handler
|
| 1313 |
raise SignalException(f"Process {os.getpid()} got signal: {sigval}", sigval=sigval)
|
| 1314 |
-
torch.distributed.elastic.multiprocessing.api.SignalException: Process
|
|
|
|
| 1 |
+
[01:33:45] WARN: moonep_backend_async_grad_reduce lacks --moe-router-fusion; QWEN3_235B runs unfused (timing only, routing identical)
|
| 2 |
==============================================================
|
| 3 |
QWEN3_235B / moonep (slots/rank=16 policy=matched)
|
| 4 |
E=128 K=8 H'=1536 m=16
|
| 5 |
L=12 seq=8192 gbs=8 (1 microbatches)
|
| 6 |
cap_mode=none cap=none cg=1 recompute=none
|
| 7 |
memory_dump=none snapshots=/workspace/tierb-fixed-moonep-full/moonep/tierb/tierb_gbs8/QWEN3_235B__moonep__matched__seq8192__gbs8__capnone__async/rep3/memory_snapshots/snapshot.pickle
|
| 8 |
+
repo=/workspace/Megatron-MoE-Echo @ 2e68c8ad
|
| 9 |
==============================================================
|
| 10 |
+
W0830 01:33:47.017000 385325 torch/distributed/run.py:766]
|
| 11 |
+
W0830 01:33:47.017000 385325 torch/distributed/run.py:766] *****************************************
|
| 12 |
+
W0830 01:33:47.017000 385325 torch/distributed/run.py:766] Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed.
|
| 13 |
+
W0830 01:33:47.017000 385325 torch/distributed/run.py:766] *****************************************
|
| 14 |
/workspace/Megatron-MoE-Echo/megatron/training/arguments.py:831: UserWarning: Disabling sequence parallelism because tensor model parallelism is disabled
|
| 15 |
warnings.warn("Disabling sequence parallelism because tensor model parallelism is disabled")
|
| 16 |
+
WARNING: one_logger package is required to enable e2e metrics tracking. please go to https://confluence.nvidia.com/display/MLWFO/Package+Repositories for details to install it
|
| 17 |
/workspace/Megatron-MoE-Echo/megatron/training/arguments.py:831: UserWarning: Disabling sequence parallelism because tensor model parallelism is disabled
|
| 18 |
warnings.warn("Disabling sequence parallelism because tensor model parallelism is disabled")
|
| 19 |
/workspace/Megatron-MoE-Echo/megatron/training/arguments.py:831: UserWarning: Disabling sequence parallelism because tensor model parallelism is disabled
|
| 20 |
warnings.warn("Disabling sequence parallelism because tensor model parallelism is disabled")
|
|
|
|
|
|
|
|
|
|
| 21 |
/workspace/Megatron-MoE-Echo/megatron/training/arguments.py:831: UserWarning: Disabling sequence parallelism because tensor model parallelism is disabled
|
| 22 |
warnings.warn("Disabling sequence parallelism because tensor model parallelism is disabled")
|
| 23 |
/workspace/Megatron-MoE-Echo/megatron/training/arguments.py:831: UserWarning: Disabling sequence parallelism because tensor model parallelism is disabled
|
|
|
|
| 572 |
-------------------- end of arguments ---------------------
|
| 573 |
INFO:megatron.core.num_microbatches_calculator:setting number of microbatches to constant 1
|
| 574 |
> building GPT2BPETokenizer tokenizer ...
|
| 575 |
+
/workspace/Megatron-MoE-Echo/megatron/training/arguments.py:831: UserWarning: Disabling sequence parallelism because tensor model parallelism is disabled
|
| 576 |
+
warnings.warn("Disabling sequence parallelism because tensor model parallelism is disabled")
|
| 577 |
> padded vocab (size: 50257) with 47 dummy tokens (new size: 50304)
|
| 578 |
WARNING:megatron.core.rerun_state_machine:RerunStateMachine initialized in mode RerunMode.VALIDATE_RESULTS
|
| 579 |
> initializing torch distributed ...
|
| 580 |
[Gloo] Rank 0 is connected to 7 peer ranks. Expected number of connected peer ranks is : 7
|
| 581 |
+
[Gloo] Rank 3 is connected to 7 peer ranks. Expected number of connected peer ranks is : 7
|
| 582 |
[Gloo] Rank 1 is connected to 7 peer ranks. Expected number of connected peer ranks is : 7
|
| 583 |
[Gloo] Rank 2 is connected to 7 peer ranks. Expected number of connected peer ranks is : 7
|
|
|
|
| 584 |
[Gloo] Rank 4 is connected to 7 peer ranks. Expected number of connected peer ranks is : 7
|
| 585 |
+
[Gloo] Rank 5 is connected to 7 peer ranks. Expected number of connected peer ranks is : 7[Gloo] Rank
|
| 586 |
+
[Gloo] Rank 6 is connected to 77 is connected to peer ranks. 7Expected number of connected peer ranks is : peer ranks. 7Expected number of connected peer ranks is :
|
| 587 |
7
|
| 588 |
+
[Gloo] Rank 0 is connected to 7 peer ranks. Expected number of connected peer ranks is : 7
|
| 589 |
+
[Gloo] Rank 4 is connected to 7 peer ranks. Expected number of connected peer ranks is : 7
|
| 590 |
+
[Gloo] Rank 5 is connected to 7 peer ranks. Expected number of connected peer ranks is : 7
|
| 591 |
+
[Gloo] Rank 1 is connected to 7 peer ranks. Expected number of connected peer ranks is : 7
|
| 592 |
+
[Gloo] Rank 2 is connected to 7 peer ranks. Expected number of connected peer ranks is : 7
|
| 593 |
[Gloo] Rank 3 is connected to 7 peer ranks. Expected number of connected peer ranks is : 7
|
| 594 |
+
[Gloo] Rank [Gloo] Rank 7 is connected to 67 is connected to peer ranks. 7Expected number of connected peer ranks is : peer ranks. 7Expected number of connected peer ranks is :
|
| 595 |
7
|
| 596 |
[Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
|
| 597 |
> initialized tensor model parallel with size 1
|
| 598 |
> initialized pipeline model parallel with size 1
|
| 599 |
> setting random seeds to 1234 ...
|
| 600 |
+
[Gloo] Rank [Gloo] Rank 00 is connected to is connected to 00 peer ranks. peer ranks. Expected number of connected peer ranks is : Expected number of connected peer ranks is : 00
|
|
|
|
| 601 |
|
| 602 |
+
[Gloo] Rank 0[Gloo] Rank is connected to 00[Gloo] Rank is connected to peer ranks. 00Expected number of connected peer ranks is : is connected to peer ranks. 00Expected number of connected peer ranks is :
|
| 603 |
+
peer ranks. 0Expected number of connected peer ranks is :
|
| 604 |
+
0
|
| 605 |
[Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
|
| 606 |
[Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
|
| 607 |
+
[rank5]:[W830 01:34:04.105198961 ProcessGroupNCCL.cpp:4789] [PG ID 0 PG GUID 0 Rank 5] using GPU 5 as device used by this process is currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect. You can specify device_id in init_process_group() to force use of a particular device.
|
| 608 |
+
[rank1]:[W830 01:34:04.105212291 ProcessGroupNCCL.cpp:4789] [PG ID 0 PG GUID 0 Rank 1] using GPU 1 as device used by this process is currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect. You can specify device_id in init_process_group() to force use of a particular device.
|
| 609 |
+
[rank7]:[W830 01:34:04.105213047 ProcessGroupNCCL.cpp:4789] [PG ID 0 PG GUID 0 Rank 7] using GPU 7 as device used by this process is currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect. You can specify device_id in init_process_group() to force use of a particular device.
|
| 610 |
+
[rank4]:[W830 01:34:04.105279090 ProcessGroupNCCL.cpp:4789] [PG ID 0 PG GUID 0 Rank 4] using GPU 4 as device used by this process is currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect. You can specify device_id in init_process_group() to force use of a particular device.
|
| 611 |
+
[rank3]:[W830 01:34:04.105289555 ProcessGroupNCCL.cpp:4789] [PG ID 0 PG GUID 0 Rank 3] using GPU 3 as device used by this process is currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect. You can specify device_id in init_process_group() to force use of a particular device.
|
| 612 |
+
[rank6]:[W830 01:34:04.105317293 ProcessGroupNCCL.cpp:4789] [PG ID 0 PG GUID 0 Rank 6] using GPU 6 as device used by this process is currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect. You can specify device_id in init_process_group() to force use of a particular device.
|
| 613 |
+
[rank2]:[W830 01:34:04.105351538 ProcessGroupNCCL.cpp:4789] [PG ID 0 PG GUID 0 Rank 2] using GPU 2 as device used by this process is currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect. You can specify device_id in init_process_group() to force use of a particular device.
|
|
|
|
|
|
|
| 614 |
> compiling dataset index builder ...
|
| 615 |
make: Entering directory '/workspace/Megatron-MoE-Echo/megatron/core/datasets'
|
| 616 |
make: Nothing to be done for 'default'.
|
| 617 |
make: Leaving directory '/workspace/Megatron-MoE-Echo/megatron/core/datasets'
|
| 618 |
+
>>> done with dataset index builder. Compilation time: 0.033 seconds
|
| 619 |
WARNING: constraints for invoking optimized fused softmax kernel are not met. We default back to unfused kernel invocations.
|
| 620 |
> compiling and loading fused kernels ...
|
| 621 |
+
[rank0]:[W830 01:34:05.270035787 ProcessGroupNCCL.cpp:4789] [PG ID 0 PG GUID 0 Rank 0] using GPU 0 as device used by this process is currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect. You can specify device_id in init_process_group() to force use of a particular device.
|
| 622 |
+
>>> done with compiling and loading fused kernels. Compilation time: 3.693 seconds
|
| 623 |
+
time to initialize megatron (seconds): 7.525
|
| 624 |
+
[after megatron is initialized] datetime: 2026-08-30 01:34:10
|
| 625 |
building GPT model ...
|
| 626 |
/usr/local/lib/python3.12/dist-packages/transformer_engine/pytorch/cpu_offload.py:708: DeprecationWarning: Offloading weights is deprecated. Using offload_weights=True does not have any effect.
|
| 627 |
warnings.warn(
|
|
|
|
| 645 |
Params for bucket 1 (206045184 elements, 206045184 padded size):
|
| 646 |
module.output_layer.weight
|
| 647 |
Params for bucket 2 (71840000 elements, 71840000 padded size):
|
|
|
|
| 648 |
module.decoder.layers.11.self_attention.q_layernorm.weight
|
| 649 |
+
module.decoder.layers.11.pre_mlp_layernorm.weight
|
| 650 |
module.decoder.layers.11.mlp.router.weight
|
| 651 |
module.decoder.layers.11.self_attention.k_layernorm.weight
|
| 652 |
module.decoder.layers.11.self_attention.linear_qkv.weight
|
| 653 |
module.decoder.layers.11.self_attention.linear_qkv.layer_norm_weight
|
|
|
|
| 654 |
module.decoder.layers.11.self_attention.linear_proj.weight
|
| 655 |
+
module.decoder.final_layernorm.weight
|
| 656 |
Params for bucket 3 (71835904 elements, 71835904 padded size):
|
| 657 |
+
module.decoder.layers.10.pre_mlp_layernorm.weight
|
| 658 |
module.decoder.layers.10.self_attention.linear_qkv.weight
|
| 659 |
+
module.decoder.layers.10.self_attention.q_layernorm.weight
|
| 660 |
module.decoder.layers.10.self_attention.linear_qkv.layer_norm_weight
|
| 661 |
module.decoder.layers.10.mlp.router.weight
|
|
|
|
| 662 |
module.decoder.layers.10.self_attention.k_layernorm.weight
|
|
|
|
| 663 |
module.decoder.layers.10.self_attention.linear_proj.weight
|
| 664 |
Params for bucket 4 (71835904 elements, 71835904 padded size):
|
| 665 |
module.decoder.layers.9.self_attention.k_layernorm.weight
|
| 666 |
+
module.decoder.layers.9.self_attention.linear_qkv.weight
|
| 667 |
module.decoder.layers.9.self_attention.linear_proj.weight
|
| 668 |
module.decoder.layers.9.mlp.router.weight
|
| 669 |
module.decoder.layers.9.pre_mlp_layernorm.weight
|
|
|
|
| 670 |
module.decoder.layers.9.self_attention.q_layernorm.weight
|
| 671 |
module.decoder.layers.9.self_attention.linear_qkv.layer_norm_weight
|
| 672 |
Params for bucket 5 (71835904 elements, 71835904 padded size):
|
| 673 |
module.decoder.layers.8.mlp.router.weight
|
| 674 |
module.decoder.layers.8.pre_mlp_layernorm.weight
|
| 675 |
+
module.decoder.layers.8.self_attention.k_layernorm.weight
|
| 676 |
module.decoder.layers.8.self_attention.linear_qkv.weight
|
| 677 |
module.decoder.layers.8.self_attention.q_layernorm.weight
|
|
|
|
| 678 |
module.decoder.layers.8.self_attention.linear_qkv.layer_norm_weight
|
| 679 |
module.decoder.layers.8.self_attention.linear_proj.weight
|
| 680 |
Params for bucket 6 (71835904 elements, 71835904 padded size):
|
|
|
|
| 683 |
module.decoder.layers.7.self_attention.k_layernorm.weight
|
| 684 |
module.decoder.layers.7.self_attention.linear_qkv.weight
|
| 685 |
module.decoder.layers.7.self_attention.q_layernorm.weight
|
|
|
|
| 686 |
module.decoder.layers.7.self_attention.linear_qkv.layer_norm_weight
|
| 687 |
+
module.decoder.layers.7.self_attention.linear_proj.weight
|
| 688 |
Params for bucket 7 (71835904 elements, 71835904 padded size):
|
| 689 |
module.decoder.layers.6.mlp.router.weight
|
|
|
|
|
|
|
| 690 |
module.decoder.layers.6.pre_mlp_layernorm.weight
|
| 691 |
+
module.decoder.layers.6.self_attention.linear_qkv.weight
|
| 692 |
module.decoder.layers.6.self_attention.k_layernorm.weight
|
| 693 |
module.decoder.layers.6.self_attention.linear_qkv.layer_norm_weight
|
| 694 |
+
module.decoder.layers.6.self_attention.q_layernorm.weight
|
| 695 |
module.decoder.layers.6.self_attention.linear_proj.weight
|
| 696 |
Params for bucket 8 (71835904 elements, 71835904 padded size):
|
| 697 |
+
module.decoder.layers.5.mlp.router.weight
|
| 698 |
module.decoder.layers.5.pre_mlp_layernorm.weight
|
| 699 |
+
module.decoder.layers.5.self_attention.q_layernorm.weight
|
| 700 |
module.decoder.layers.5.self_attention.linear_qkv.layer_norm_weight
|
|
|
|
| 701 |
module.decoder.layers.5.self_attention.linear_qkv.weight
|
| 702 |
module.decoder.layers.5.self_attention.k_layernorm.weight
|
| 703 |
module.decoder.layers.5.self_attention.linear_proj.weight
|
| 704 |
Params for bucket 9 (71835904 elements, 71835904 padded size):
|
|
|
|
| 705 |
module.decoder.layers.4.pre_mlp_layernorm.weight
|
| 706 |
module.decoder.layers.4.self_attention.q_layernorm.weight
|
| 707 |
+
module.decoder.layers.4.self_attention.linear_qkv.layer_norm_weight
|
| 708 |
+
module.decoder.layers.4.mlp.router.weight
|
| 709 |
module.decoder.layers.4.self_attention.k_layernorm.weight
|
| 710 |
module.decoder.layers.4.self_attention.linear_qkv.weight
|
|
|
|
| 711 |
module.decoder.layers.4.self_attention.linear_proj.weight
|
| 712 |
Params for bucket 10 (71835904 elements, 71835904 padded size):
|
| 713 |
module.decoder.layers.3.mlp.router.weight
|
|
|
|
| 714 |
module.decoder.layers.3.self_attention.q_layernorm.weight
|
| 715 |
module.decoder.layers.3.self_attention.linear_qkv.weight
|
| 716 |
+
module.decoder.layers.3.pre_mlp_layernorm.weight
|
| 717 |
module.decoder.layers.3.self_attention.k_layernorm.weight
|
| 718 |
module.decoder.layers.3.self_attention.linear_qkv.layer_norm_weight
|
| 719 |
module.decoder.layers.3.self_attention.linear_proj.weight
|
| 720 |
Params for bucket 11 (71835904 elements, 71835904 padded size):
|
| 721 |
module.decoder.layers.2.mlp.router.weight
|
| 722 |
+
module.decoder.layers.2.self_attention.q_layernorm.weight
|
| 723 |
module.decoder.layers.2.pre_mlp_layernorm.weight
|
|
|
|
| 724 |
module.decoder.layers.2.self_attention.linear_qkv.weight
|
| 725 |
+
module.decoder.layers.2.self_attention.k_layernorm.weight
|
| 726 |
module.decoder.layers.2.self_attention.linear_qkv.layer_norm_weight
|
| 727 |
module.decoder.layers.2.self_attention.linear_proj.weight
|
| 728 |
Params for bucket 12 (71835904 elements, 71835904 padded size):
|
| 729 |
module.decoder.layers.1.mlp.router.weight
|
| 730 |
module.decoder.layers.1.pre_mlp_layernorm.weight
|
| 731 |
module.decoder.layers.1.self_attention.k_layernorm.weight
|
|
|
|
| 732 |
module.decoder.layers.1.self_attention.linear_qkv.weight
|
| 733 |
+
module.decoder.layers.1.self_attention.q_layernorm.weight
|
| 734 |
module.decoder.layers.1.self_attention.linear_qkv.layer_norm_weight
|
| 735 |
module.decoder.layers.1.self_attention.linear_proj.weight
|
| 736 |
Params for bucket 13 (71835904 elements, 71835904 padded size):
|
| 737 |
+
module.decoder.layers.0.mlp.router.weight
|
| 738 |
module.decoder.layers.0.self_attention.q_layernorm.weight
|
|
|
|
| 739 |
module.decoder.layers.0.self_attention.linear_proj.weight
|
| 740 |
+
module.decoder.layers.0.pre_mlp_layernorm.weight
|
| 741 |
+
module.decoder.layers.0.self_attention.k_layernorm.weight
|
| 742 |
module.decoder.layers.0.self_attention.linear_qkv.weight
|
| 743 |
module.decoder.layers.0.self_attention.linear_qkv.layer_norm_weight
|
| 744 |
Params for bucket 14 (206045184 elements, 206045184 padded size):
|
|
|
|
| 746 |
INFO:megatron.core.distributed.param_and_grad_buffer:Number of buckets for gradient all-reduce / reduce-scatter: 76
|
| 747 |
Params for bucket 1 (44040192 elements, 44040192 padded size):
|
| 748 |
module.decoder.layers.11.mlp.experts.linear_fc2.weight0
|
|
|
|
| 749 |
module.decoder.layers.11.mlp.experts.linear_fc2.weight4
|
| 750 |
+
module.decoder.layers.11.mlp.experts.linear_fc2.weight2
|
| 751 |
module.decoder.layers.11.mlp.experts.linear_fc2.weight6
|
| 752 |
module.decoder.layers.11.mlp.experts.linear_fc2.weight3
|
| 753 |
module.decoder.layers.11.mlp.experts.linear_fc2.weight1
|
|
|
|
| 757 |
module.decoder.layers.11.mlp.experts.linear_fc2.weight10
|
| 758 |
module.decoder.layers.11.mlp.experts.linear_fc2.weight12
|
| 759 |
module.decoder.layers.11.mlp.experts.linear_fc2.weight7
|
|
|
|
| 760 |
module.decoder.layers.11.mlp.experts.linear_fc2.weight11
|
| 761 |
+
module.decoder.layers.11.mlp.experts.linear_fc2.weight9
|
| 762 |
module.decoder.layers.11.mlp.experts.linear_fc2.weight13
|
| 763 |
Params for bucket 3 (50331648 elements, 50331648 padded size):
|
| 764 |
module.decoder.layers.11.mlp.experts.linear_fc2.weight14
|
| 765 |
module.decoder.layers.11.mlp.experts.linear_fc1.weight1
|
|
|
|
| 766 |
module.decoder.layers.11.mlp.experts.linear_fc1.weight2
|
| 767 |
+
module.decoder.layers.11.mlp.experts.linear_fc2.weight15
|
| 768 |
module.decoder.layers.11.mlp.experts.linear_fc1.weight0
|
| 769 |
Params for bucket 4 (50331648 elements, 50331648 padded size):
|
| 770 |
module.decoder.layers.11.mlp.experts.linear_fc1.weight3
|
| 771 |
module.decoder.layers.11.mlp.experts.linear_fc1.weight5
|
|
|
|
| 772 |
module.decoder.layers.11.mlp.experts.linear_fc1.weight4
|
| 773 |
+
module.decoder.layers.11.mlp.experts.linear_fc1.weight6
|
| 774 |
Params for bucket 5 (50331648 elements, 50331648 padded size):
|
| 775 |
module.decoder.layers.11.mlp.experts.linear_fc1.weight7
|
|
|
|
| 776 |
module.decoder.layers.11.mlp.experts.linear_fc1.weight8
|
| 777 |
+
module.decoder.layers.11.mlp.experts.linear_fc1.weight9
|
| 778 |
module.decoder.layers.11.mlp.experts.linear_fc1.weight10
|
| 779 |
Params for bucket 6 (50331648 elements, 50331648 padded size):
|
| 780 |
module.decoder.layers.11.mlp.experts.linear_fc1.weight11
|
| 781 |
+
module.decoder.layers.11.mlp.experts.linear_fc1.weight14
|
| 782 |
module.decoder.layers.11.mlp.experts.linear_fc1.weight12
|
| 783 |
module.decoder.layers.11.mlp.experts.linear_fc1.weight13
|
|
|
|
| 784 |
Params for bucket 7 (44040192 elements, 44040192 padded size):
|
| 785 |
module.decoder.layers.11.mlp.experts.linear_fc1.weight15
|
| 786 |
module.decoder.layers.10.mlp.experts.linear_fc2.weight0
|
| 787 |
module.decoder.layers.10.mlp.experts.linear_fc2.weight2
|
| 788 |
module.decoder.layers.10.mlp.experts.linear_fc2.weight4
|
|
|
|
| 789 |
module.decoder.layers.10.mlp.experts.linear_fc2.weight3
|
| 790 |
+
module.decoder.layers.10.mlp.experts.linear_fc2.weight1
|
| 791 |
Params for bucket 8 (44040192 elements, 44040192 padded size):
|
|
|
|
| 792 |
module.decoder.layers.10.mlp.experts.linear_fc2.weight6
|
| 793 |
+
module.decoder.layers.10.mlp.experts.linear_fc2.weight8
|
| 794 |
module.decoder.layers.10.mlp.experts.linear_fc2.weight10
|
| 795 |
module.decoder.layers.10.mlp.experts.linear_fc2.weight5
|
| 796 |
module.decoder.layers.10.mlp.experts.linear_fc2.weight7
|
|
|
|
| 798 |
module.decoder.layers.10.mlp.experts.linear_fc2.weight11
|
| 799 |
Params for bucket 9 (50331648 elements, 50331648 padded size):
|
| 800 |
module.decoder.layers.10.mlp.experts.linear_fc2.weight12
|
| 801 |
+
module.decoder.layers.10.mlp.experts.linear_fc1.weight0
|
| 802 |
module.decoder.layers.10.mlp.experts.linear_fc2.weight14
|
|
|
|
| 803 |
module.decoder.layers.10.mlp.experts.linear_fc2.weight13
|
| 804 |
module.decoder.layers.10.mlp.experts.linear_fc2.weight15
|
| 805 |
+
module.decoder.layers.10.mlp.experts.linear_fc1.weight1
|
| 806 |
Params for bucket 10 (50331648 elements, 50331648 padded size):
|
| 807 |
module.decoder.layers.10.mlp.experts.linear_fc1.weight2
|
|
|
|
| 808 |
module.decoder.layers.10.mlp.experts.linear_fc1.weight5
|
| 809 |
module.decoder.layers.10.mlp.experts.linear_fc1.weight3
|
| 810 |
+
module.decoder.layers.10.mlp.experts.linear_fc1.weight4
|
| 811 |
Params for bucket 11 (50331648 elements, 50331648 padded size):
|
| 812 |
module.decoder.layers.10.mlp.experts.linear_fc1.weight6
|
|
|
|
| 813 |
module.decoder.layers.10.mlp.experts.linear_fc1.weight8
|
| 814 |
module.decoder.layers.10.mlp.experts.linear_fc1.weight9
|
| 815 |
+
module.decoder.layers.10.mlp.experts.linear_fc1.weight7
|
| 816 |
Params for bucket 12 (50331648 elements, 50331648 padded size):
|
| 817 |
module.decoder.layers.10.mlp.experts.linear_fc1.weight10
|
| 818 |
+
module.decoder.layers.10.mlp.experts.linear_fc1.weight12
|
| 819 |
module.decoder.layers.10.mlp.experts.linear_fc1.weight13
|
| 820 |
module.decoder.layers.10.mlp.experts.linear_fc1.weight11
|
|
|
|
| 821 |
Params for bucket 13 (44040192 elements, 44040192 padded size):
|
|
|
|
|
|
|
| 822 |
module.decoder.layers.10.mlp.experts.linear_fc1.weight14
|
|
|
|
| 823 |
module.decoder.layers.9.mlp.experts.linear_fc2.weight2
|
| 824 |
+
module.decoder.layers.9.mlp.experts.linear_fc2.weight0
|
| 825 |
+
module.decoder.layers.10.mlp.experts.linear_fc1.weight15
|
| 826 |
+
module.decoder.layers.9.mlp.experts.linear_fc2.weight1
|
| 827 |
Params for bucket 14 (44040192 elements, 44040192 padded size):
|
|
|
|
|
|
|
|
|
|
|
|
|
| 828 |
module.decoder.layers.9.mlp.experts.linear_fc2.weight6
|
| 829 |
module.decoder.layers.9.mlp.experts.linear_fc2.weight4
|
| 830 |
module.decoder.layers.9.mlp.experts.linear_fc2.weight8
|
| 831 |
+
module.decoder.layers.9.mlp.experts.linear_fc2.weight3
|
| 832 |
+
module.decoder.layers.9.mlp.experts.linear_fc2.weight5
|
| 833 |
+
module.decoder.layers.9.mlp.experts.linear_fc2.weight7
|
| 834 |
+
module.decoder.layers.9.mlp.experts.linear_fc2.weight9
|
| 835 |
Params for bucket 15 (50331648 elements, 50331648 padded size):
|
| 836 |
+
module.decoder.layers.9.mlp.experts.linear_fc2.weight10
|
| 837 |
+
module.decoder.layers.9.mlp.experts.linear_fc2.weight12
|
| 838 |
+
module.decoder.layers.9.mlp.experts.linear_fc2.weight14
|
| 839 |
module.decoder.layers.9.mlp.experts.linear_fc2.weight11
|
| 840 |
+
module.decoder.layers.9.mlp.experts.linear_fc2.weight13
|
| 841 |
module.decoder.layers.9.mlp.experts.linear_fc2.weight15
|
| 842 |
module.decoder.layers.9.mlp.experts.linear_fc1.weight0
|
|
|
|
|
|
|
|
|
|
| 843 |
Params for bucket 16 (50331648 elements, 50331648 padded size):
|
| 844 |
module.decoder.layers.9.mlp.experts.linear_fc1.weight1
|
|
|
|
| 845 |
module.decoder.layers.9.mlp.experts.linear_fc1.weight4
|
| 846 |
module.decoder.layers.9.mlp.experts.linear_fc1.weight2
|
| 847 |
+
module.decoder.layers.9.mlp.experts.linear_fc1.weight3
|
| 848 |
Params for bucket 17 (50331648 elements, 50331648 padded size):
|
| 849 |
module.decoder.layers.9.mlp.experts.linear_fc1.weight5
|
|
|
|
| 850 |
module.decoder.layers.9.mlp.experts.linear_fc1.weight7
|
| 851 |
module.decoder.layers.9.mlp.experts.linear_fc1.weight8
|
| 852 |
+
module.decoder.layers.9.mlp.experts.linear_fc1.weight6
|
| 853 |
Params for bucket 18 (50331648 elements, 50331648 padded size):
|
| 854 |
module.decoder.layers.9.mlp.experts.linear_fc1.weight9
|
| 855 |
+
module.decoder.layers.9.mlp.experts.linear_fc1.weight11
|
| 856 |
module.decoder.layers.9.mlp.experts.linear_fc1.weight12
|
| 857 |
module.decoder.layers.9.mlp.experts.linear_fc1.weight10
|
|
|
|
| 858 |
Params for bucket 19 (44040192 elements, 44040192 padded size):
|
| 859 |
module.decoder.layers.9.mlp.experts.linear_fc1.weight13
|
|
|
|
| 860 |
module.decoder.layers.9.mlp.experts.linear_fc1.weight14
|
| 861 |
module.decoder.layers.9.mlp.experts.linear_fc1.weight15
|
| 862 |
+
module.decoder.layers.8.mlp.experts.linear_fc2.weight0
|
| 863 |
Params for bucket 20 (44040192 elements, 44040192 padded size):
|
| 864 |
module.decoder.layers.8.mlp.experts.linear_fc2.weight1
|
| 865 |
module.decoder.layers.8.mlp.experts.linear_fc2.weight5
|
|
|
|
| 873 |
module.decoder.layers.8.mlp.experts.linear_fc2.weight9
|
| 874 |
module.decoder.layers.8.mlp.experts.linear_fc2.weight13
|
| 875 |
module.decoder.layers.8.mlp.experts.linear_fc2.weight8
|
|
|
|
| 876 |
module.decoder.layers.8.mlp.experts.linear_fc2.weight12
|
| 877 |
+
module.decoder.layers.8.mlp.experts.linear_fc2.weight10
|
| 878 |
module.decoder.layers.8.mlp.experts.linear_fc2.weight14
|
| 879 |
Params for bucket 22 (44040192 elements, 44040192 padded size):
|
| 880 |
module.decoder.layers.8.mlp.experts.linear_fc2.weight15
|
|
|
|
| 881 |
module.decoder.layers.8.mlp.experts.linear_fc1.weight2
|
| 882 |
+
module.decoder.layers.8.mlp.experts.linear_fc1.weight0
|
| 883 |
module.decoder.layers.8.mlp.experts.linear_fc1.weight1
|
| 884 |
Params for bucket 23 (50331648 elements, 50331648 padded size):
|
| 885 |
module.decoder.layers.8.mlp.experts.linear_fc1.weight3
|
| 886 |
+
module.decoder.layers.8.mlp.experts.linear_fc1.weight5
|
| 887 |
module.decoder.layers.8.mlp.experts.linear_fc1.weight6
|
| 888 |
module.decoder.layers.8.mlp.experts.linear_fc1.weight4
|
|
|
|
| 889 |
Params for bucket 24 (50331648 elements, 50331648 padded size):
|
| 890 |
module.decoder.layers.8.mlp.experts.linear_fc1.weight7
|
| 891 |
+
module.decoder.layers.8.mlp.experts.linear_fc1.weight8
|
| 892 |
module.decoder.layers.8.mlp.experts.linear_fc1.weight9
|
| 893 |
module.decoder.layers.8.mlp.experts.linear_fc1.weight10
|
|
|
|
| 894 |
Params for bucket 25 (50331648 elements, 50331648 padded size):
|
| 895 |
module.decoder.layers.8.mlp.experts.linear_fc1.weight11
|
| 896 |
module.decoder.layers.8.mlp.experts.linear_fc1.weight12
|
| 897 |
module.decoder.layers.8.mlp.experts.linear_fc1.weight13
|
| 898 |
module.decoder.layers.8.mlp.experts.linear_fc1.weight14
|
| 899 |
Params for bucket 26 (44040192 elements, 44040192 padded size):
|
|
|
|
|
|
|
|
|
|
| 900 |
module.decoder.layers.7.mlp.experts.linear_fc2.weight0
|
| 901 |
module.decoder.layers.7.mlp.experts.linear_fc2.weight2
|
| 902 |
module.decoder.layers.7.mlp.experts.linear_fc2.weight4
|
| 903 |
+
module.decoder.layers.8.mlp.experts.linear_fc1.weight15
|
| 904 |
+
module.decoder.layers.7.mlp.experts.linear_fc2.weight1
|
| 905 |
+
module.decoder.layers.7.mlp.experts.linear_fc2.weight3
|
| 906 |
Params for bucket 27 (44040192 elements, 44040192 padded size):
|
| 907 |
+
module.decoder.layers.7.mlp.experts.linear_fc2.weight8
|
| 908 |
+
module.decoder.layers.7.mlp.experts.linear_fc2.weight6
|
| 909 |
+
module.decoder.layers.7.mlp.experts.linear_fc2.weight10
|
| 910 |
module.decoder.layers.7.mlp.experts.linear_fc2.weight5
|
| 911 |
module.decoder.layers.7.mlp.experts.linear_fc2.weight7
|
| 912 |
module.decoder.layers.7.mlp.experts.linear_fc2.weight9
|
| 913 |
module.decoder.layers.7.mlp.experts.linear_fc2.weight11
|
|
|
|
|
|
|
|
|
|
| 914 |
Params for bucket 28 (50331648 elements, 50331648 padded size):
|
|
|
|
|
|
|
|
|
|
| 915 |
module.decoder.layers.7.mlp.experts.linear_fc2.weight12
|
| 916 |
module.decoder.layers.7.mlp.experts.linear_fc2.weight14
|
| 917 |
module.decoder.layers.7.mlp.experts.linear_fc1.weight1
|
| 918 |
+
module.decoder.layers.7.mlp.experts.linear_fc1.weight0
|
| 919 |
+
module.decoder.layers.7.mlp.experts.linear_fc2.weight15
|
| 920 |
+
module.decoder.layers.7.mlp.experts.linear_fc2.weight13
|
| 921 |
Params for bucket 29 (50331648 elements, 50331648 padded size):
|
| 922 |
module.decoder.layers.7.mlp.experts.linear_fc1.weight2
|
|
|
|
| 923 |
module.decoder.layers.7.mlp.experts.linear_fc1.weight5
|
| 924 |
+
module.decoder.layers.7.mlp.experts.linear_fc1.weight4
|
| 925 |
module.decoder.layers.7.mlp.experts.linear_fc1.weight3
|
| 926 |
Params for bucket 30 (50331648 elements, 50331648 padded size):
|
| 927 |
module.decoder.layers.7.mlp.experts.linear_fc1.weight6
|
|
|
|
| 928 |
module.decoder.layers.7.mlp.experts.linear_fc1.weight9
|
| 929 |
module.decoder.layers.7.mlp.experts.linear_fc1.weight8
|
| 930 |
+
module.decoder.layers.7.mlp.experts.linear_fc1.weight7
|
| 931 |
Params for bucket 31 (50331648 elements, 50331648 padded size):
|
| 932 |
module.decoder.layers.7.mlp.experts.linear_fc1.weight10
|
| 933 |
module.decoder.layers.7.mlp.experts.linear_fc1.weight11
|
| 934 |
module.decoder.layers.7.mlp.experts.linear_fc1.weight12
|
| 935 |
module.decoder.layers.7.mlp.experts.linear_fc1.weight13
|
| 936 |
Params for bucket 32 (44040192 elements, 44040192 padded size):
|
| 937 |
+
module.decoder.layers.6.mlp.experts.linear_fc2.weight1
|
| 938 |
+
module.decoder.layers.7.mlp.experts.linear_fc1.weight15
|
| 939 |
module.decoder.layers.7.mlp.experts.linear_fc1.weight14
|
| 940 |
module.decoder.layers.6.mlp.experts.linear_fc2.weight0
|
| 941 |
module.decoder.layers.6.mlp.experts.linear_fc2.weight2
|
|
|
|
|
|
|
| 942 |
Params for bucket 33 (44040192 elements, 44040192 padded size):
|
|
|
|
|
|
|
|
|
|
| 943 |
module.decoder.layers.6.mlp.experts.linear_fc2.weight3
|
|
|
|
| 944 |
module.decoder.layers.6.mlp.experts.linear_fc2.weight5
|
| 945 |
+
module.decoder.layers.6.mlp.experts.linear_fc2.weight7
|
| 946 |
module.decoder.layers.6.mlp.experts.linear_fc2.weight9
|
| 947 |
+
module.decoder.layers.6.mlp.experts.linear_fc2.weight4
|
| 948 |
+
module.decoder.layers.6.mlp.experts.linear_fc2.weight6
|
| 949 |
+
module.decoder.layers.6.mlp.experts.linear_fc2.weight8
|
| 950 |
Params for bucket 34 (50331648 elements, 50331648 padded size):
|
|
|
|
|
|
|
|
|
|
| 951 |
module.decoder.layers.6.mlp.experts.linear_fc2.weight13
|
| 952 |
module.decoder.layers.6.mlp.experts.linear_fc2.weight11
|
| 953 |
module.decoder.layers.6.mlp.experts.linear_fc2.weight15
|
| 954 |
+
module.decoder.layers.6.mlp.experts.linear_fc2.weight10
|
| 955 |
+
module.decoder.layers.6.mlp.experts.linear_fc2.weight14
|
| 956 |
+
module.decoder.layers.6.mlp.experts.linear_fc2.weight12
|
| 957 |
module.decoder.layers.6.mlp.experts.linear_fc1.weight0
|
| 958 |
Params for bucket 35 (50331648 elements, 50331648 padded size):
|
| 959 |
module.decoder.layers.6.mlp.experts.linear_fc1.weight1
|
| 960 |
+
module.decoder.layers.6.mlp.experts.linear_fc1.weight2
|
| 961 |
module.decoder.layers.6.mlp.experts.linear_fc1.weight3
|
| 962 |
module.decoder.layers.6.mlp.experts.linear_fc1.weight4
|
|
|
|
| 963 |
Params for bucket 36 (50331648 elements, 50331648 padded size):
|
| 964 |
module.decoder.layers.6.mlp.experts.linear_fc1.weight5
|
| 965 |
module.decoder.layers.6.mlp.experts.linear_fc1.weight6
|
|
|
|
| 966 |
module.decoder.layers.6.mlp.experts.linear_fc1.weight8
|
| 967 |
+
module.decoder.layers.6.mlp.experts.linear_fc1.weight7
|
| 968 |
Params for bucket 37 (50331648 elements, 50331648 padded size):
|
| 969 |
module.decoder.layers.6.mlp.experts.linear_fc1.weight9
|
|
|
|
|
|
|
| 970 |
module.decoder.layers.6.mlp.experts.linear_fc1.weight12
|
| 971 |
+
module.decoder.layers.6.mlp.experts.linear_fc1.weight11
|
| 972 |
+
module.decoder.layers.6.mlp.experts.linear_fc1.weight10
|
| 973 |
Params for bucket 38 (44040192 elements, 44040192 padded size):
|
| 974 |
module.decoder.layers.6.mlp.experts.linear_fc1.weight13
|
| 975 |
module.decoder.layers.6.mlp.experts.linear_fc1.weight15
|
|
|
|
| 976 |
module.decoder.layers.5.mlp.experts.linear_fc2.weight0
|
| 977 |
+
module.decoder.layers.6.mlp.experts.linear_fc1.weight14
|
| 978 |
Params for bucket 39 (44040192 elements, 44040192 padded size):
|
|
|
|
|
|
|
|
|
|
| 979 |
module.decoder.layers.5.mlp.experts.linear_fc2.weight1
|
|
|
|
| 980 |
module.decoder.layers.5.mlp.experts.linear_fc2.weight3
|
| 981 |
+
module.decoder.layers.5.mlp.experts.linear_fc2.weight5
|
| 982 |
module.decoder.layers.5.mlp.experts.linear_fc2.weight7
|
| 983 |
+
module.decoder.layers.5.mlp.experts.linear_fc2.weight4
|
| 984 |
+
module.decoder.layers.5.mlp.experts.linear_fc2.weight2
|
| 985 |
+
module.decoder.layers.5.mlp.experts.linear_fc2.weight6
|
| 986 |
Params for bucket 40 (44040192 elements, 44040192 padded size):
|
| 987 |
+
module.decoder.layers.5.mlp.experts.linear_fc2.weight9
|
| 988 |
+
module.decoder.layers.5.mlp.experts.linear_fc2.weight11
|
| 989 |
+
module.decoder.layers.5.mlp.experts.linear_fc2.weight13
|
| 990 |
module.decoder.layers.5.mlp.experts.linear_fc2.weight8
|
|
|
|
| 991 |
module.decoder.layers.5.mlp.experts.linear_fc2.weight10
|
| 992 |
+
module.decoder.layers.5.mlp.experts.linear_fc2.weight12
|
| 993 |
module.decoder.layers.5.mlp.experts.linear_fc2.weight14
|
|
|
|
|
|
|
|
|
|
| 994 |
Params for bucket 41 (44040192 elements, 44040192 padded size):
|
| 995 |
module.decoder.layers.5.mlp.experts.linear_fc2.weight15
|
| 996 |
+
module.decoder.layers.5.mlp.experts.linear_fc1.weight0
|
| 997 |
module.decoder.layers.5.mlp.experts.linear_fc1.weight1
|
| 998 |
module.decoder.layers.5.mlp.experts.linear_fc1.weight2
|
|
|
|
| 999 |
Params for bucket 42 (50331648 elements, 50331648 padded size):
|
| 1000 |
module.decoder.layers.5.mlp.experts.linear_fc1.weight3
|
|
|
|
| 1001 |
module.decoder.layers.5.mlp.experts.linear_fc1.weight6
|
| 1002 |
module.decoder.layers.5.mlp.experts.linear_fc1.weight4
|
| 1003 |
+
module.decoder.layers.5.mlp.experts.linear_fc1.weight5
|
| 1004 |
Params for bucket 43 (50331648 elements, 50331648 padded size):
|
| 1005 |
module.decoder.layers.5.mlp.experts.linear_fc1.weight7
|
|
|
|
| 1006 |
module.decoder.layers.5.mlp.experts.linear_fc1.weight9
|
| 1007 |
module.decoder.layers.5.mlp.experts.linear_fc1.weight10
|
| 1008 |
+
module.decoder.layers.5.mlp.experts.linear_fc1.weight8
|
| 1009 |
Params for bucket 44 (50331648 elements, 50331648 padded size):
|
| 1010 |
module.decoder.layers.5.mlp.experts.linear_fc1.weight11
|
|
|
|
| 1011 |
module.decoder.layers.5.mlp.experts.linear_fc1.weight12
|
| 1012 |
module.decoder.layers.5.mlp.experts.linear_fc1.weight13
|
| 1013 |
+
module.decoder.layers.5.mlp.experts.linear_fc1.weight14
|
| 1014 |
Params for bucket 45 (44040192 elements, 44040192 padded size):
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1015 |
module.decoder.layers.4.mlp.experts.linear_fc2.weight2
|
| 1016 |
+
module.decoder.layers.4.mlp.experts.linear_fc2.weight0
|
| 1017 |
module.decoder.layers.4.mlp.experts.linear_fc2.weight4
|
| 1018 |
+
module.decoder.layers.5.mlp.experts.linear_fc1.weight15
|
| 1019 |
+
module.decoder.layers.4.mlp.experts.linear_fc2.weight1
|
| 1020 |
+
module.decoder.layers.4.mlp.experts.linear_fc2.weight3
|
| 1021 |
Params for bucket 46 (44040192 elements, 44040192 padded size):
|
| 1022 |
+
module.decoder.layers.4.mlp.experts.linear_fc2.weight8
|
| 1023 |
+
module.decoder.layers.4.mlp.experts.linear_fc2.weight6
|
| 1024 |
+
module.decoder.layers.4.mlp.experts.linear_fc2.weight10
|
| 1025 |
module.decoder.layers.4.mlp.experts.linear_fc2.weight5
|
|
|
|
| 1026 |
module.decoder.layers.4.mlp.experts.linear_fc2.weight7
|
| 1027 |
+
module.decoder.layers.4.mlp.experts.linear_fc2.weight9
|
| 1028 |
module.decoder.layers.4.mlp.experts.linear_fc2.weight11
|
|
|
|
|
|
|
|
|
|
| 1029 |
Params for bucket 47 (50331648 elements, 50331648 padded size):
|
|
|
|
|
|
|
|
|
|
| 1030 |
module.decoder.layers.4.mlp.experts.linear_fc2.weight12
|
| 1031 |
module.decoder.layers.4.mlp.experts.linear_fc2.weight14
|
| 1032 |
module.decoder.layers.4.mlp.experts.linear_fc1.weight1
|
| 1033 |
+
module.decoder.layers.4.mlp.experts.linear_fc1.weight0
|
| 1034 |
+
module.decoder.layers.4.mlp.experts.linear_fc2.weight15
|
| 1035 |
+
module.decoder.layers.4.mlp.experts.linear_fc2.weight13
|
| 1036 |
Params for bucket 48 (50331648 elements, 50331648 padded size):
|
| 1037 |
module.decoder.layers.4.mlp.experts.linear_fc1.weight2
|
|
|
|
| 1038 |
module.decoder.layers.4.mlp.experts.linear_fc1.weight4
|
| 1039 |
module.decoder.layers.4.mlp.experts.linear_fc1.weight5
|
| 1040 |
+
module.decoder.layers.4.mlp.experts.linear_fc1.weight3
|
| 1041 |
Params for bucket 49 (50331648 elements, 50331648 padded size):
|
| 1042 |
module.decoder.layers.4.mlp.experts.linear_fc1.weight6
|
|
|
|
| 1043 |
module.decoder.layers.4.mlp.experts.linear_fc1.weight7
|
| 1044 |
module.decoder.layers.4.mlp.experts.linear_fc1.weight8
|
| 1045 |
+
module.decoder.layers.4.mlp.experts.linear_fc1.weight9
|
| 1046 |
Params for bucket 50 (50331648 elements, 50331648 padded size):
|
| 1047 |
module.decoder.layers.4.mlp.experts.linear_fc1.weight10
|
|
|
|
| 1048 |
module.decoder.layers.4.mlp.experts.linear_fc1.weight13
|
| 1049 |
module.decoder.layers.4.mlp.experts.linear_fc1.weight11
|
| 1050 |
+
module.decoder.layers.4.mlp.experts.linear_fc1.weight12
|
| 1051 |
Params for bucket 51 (44040192 elements, 44040192 padded size):
|
| 1052 |
+
module.decoder.layers.3.mlp.experts.linear_fc2.weight1
|
| 1053 |
+
module.decoder.layers.4.mlp.experts.linear_fc1.weight15
|
| 1054 |
module.decoder.layers.4.mlp.experts.linear_fc1.weight14
|
| 1055 |
module.decoder.layers.3.mlp.experts.linear_fc2.weight2
|
| 1056 |
module.decoder.layers.3.mlp.experts.linear_fc2.weight0
|
|
|
|
|
|
|
| 1057 |
Params for bucket 52 (44040192 elements, 44040192 padded size):
|
|
|
|
|
|
|
|
|
|
| 1058 |
module.decoder.layers.3.mlp.experts.linear_fc2.weight3
|
| 1059 |
module.decoder.layers.3.mlp.experts.linear_fc2.weight5
|
| 1060 |
module.decoder.layers.3.mlp.experts.linear_fc2.weight7
|
| 1061 |
module.decoder.layers.3.mlp.experts.linear_fc2.weight9
|
| 1062 |
+
module.decoder.layers.3.mlp.experts.linear_fc2.weight4
|
| 1063 |
+
module.decoder.layers.3.mlp.experts.linear_fc2.weight6
|
| 1064 |
+
module.decoder.layers.3.mlp.experts.linear_fc2.weight8
|
| 1065 |
Params for bucket 53 (50331648 elements, 50331648 padded size):
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1066 |
module.decoder.layers.3.mlp.experts.linear_fc2.weight13
|
| 1067 |
+
module.decoder.layers.3.mlp.experts.linear_fc2.weight11
|
| 1068 |
module.decoder.layers.3.mlp.experts.linear_fc2.weight15
|
| 1069 |
+
module.decoder.layers.3.mlp.experts.linear_fc1.weight0
|
| 1070 |
+
module.decoder.layers.3.mlp.experts.linear_fc2.weight10
|
| 1071 |
+
module.decoder.layers.3.mlp.experts.linear_fc2.weight14
|
| 1072 |
+
module.decoder.layers.3.mlp.experts.linear_fc2.weight12
|
| 1073 |
Params for bucket 54 (50331648 elements, 50331648 padded size):
|
| 1074 |
module.decoder.layers.3.mlp.experts.linear_fc1.weight1
|
|
|
|
| 1075 |
module.decoder.layers.3.mlp.experts.linear_fc1.weight2
|
| 1076 |
+
module.decoder.layers.3.mlp.experts.linear_fc1.weight4
|
| 1077 |
module.decoder.layers.3.mlp.experts.linear_fc1.weight3
|
| 1078 |
Params for bucket 55 (50331648 elements, 50331648 padded size):
|
| 1079 |
module.decoder.layers.3.mlp.experts.linear_fc1.weight5
|
| 1080 |
module.decoder.layers.3.mlp.experts.linear_fc1.weight8
|
|
|
|
| 1081 |
module.decoder.layers.3.mlp.experts.linear_fc1.weight6
|
| 1082 |
+
module.decoder.layers.3.mlp.experts.linear_fc1.weight7
|
| 1083 |
Params for bucket 56 (50331648 elements, 50331648 padded size):
|
| 1084 |
module.decoder.layers.3.mlp.experts.linear_fc1.weight9
|
| 1085 |
module.decoder.layers.3.mlp.experts.linear_fc1.weight11
|
|
|
|
| 1086 |
module.decoder.layers.3.mlp.experts.linear_fc1.weight12
|
| 1087 |
+
module.decoder.layers.3.mlp.experts.linear_fc1.weight10
|
| 1088 |
Params for bucket 57 (44040192 elements, 44040192 padded size):
|
| 1089 |
module.decoder.layers.3.mlp.experts.linear_fc1.weight13
|
| 1090 |
module.decoder.layers.3.mlp.experts.linear_fc1.weight14
|
| 1091 |
module.decoder.layers.3.mlp.experts.linear_fc1.weight15
|
| 1092 |
module.decoder.layers.2.mlp.experts.linear_fc2.weight0
|
| 1093 |
Params for bucket 58 (44040192 elements, 44040192 padded size):
|
|
|
|
|
|
|
|
|
|
| 1094 |
module.decoder.layers.2.mlp.experts.linear_fc2.weight1
|
| 1095 |
module.decoder.layers.2.mlp.experts.linear_fc2.weight3
|
| 1096 |
module.decoder.layers.2.mlp.experts.linear_fc2.weight5
|
| 1097 |
module.decoder.layers.2.mlp.experts.linear_fc2.weight7
|
| 1098 |
+
module.decoder.layers.2.mlp.experts.linear_fc2.weight2
|
| 1099 |
+
module.decoder.layers.2.mlp.experts.linear_fc2.weight4
|
| 1100 |
+
module.decoder.layers.2.mlp.experts.linear_fc2.weight6
|
| 1101 |
Params for bucket 59 (44040192 elements, 44040192 padded size):
|
| 1102 |
+
module.decoder.layers.2.mlp.experts.linear_fc2.weight9
|
| 1103 |
+
module.decoder.layers.2.mlp.experts.linear_fc2.weight11
|
| 1104 |
+
module.decoder.layers.2.mlp.experts.linear_fc2.weight13
|
| 1105 |
module.decoder.layers.2.mlp.experts.linear_fc2.weight8
|
| 1106 |
module.decoder.layers.2.mlp.experts.linear_fc2.weight12
|
| 1107 |
module.decoder.layers.2.mlp.experts.linear_fc2.weight10
|
| 1108 |
module.decoder.layers.2.mlp.experts.linear_fc2.weight14
|
|
|
|
|
|
|
|
|
|
| 1109 |
Params for bucket 60 (44040192 elements, 44040192 padded size):
|
| 1110 |
module.decoder.layers.2.mlp.experts.linear_fc2.weight15
|
|
|
|
| 1111 |
module.decoder.layers.2.mlp.experts.linear_fc1.weight2
|
| 1112 |
module.decoder.layers.2.mlp.experts.linear_fc1.weight0
|
| 1113 |
+
module.decoder.layers.2.mlp.experts.linear_fc1.weight1
|
| 1114 |
Params for bucket 61 (50331648 elements, 50331648 padded size):
|
| 1115 |
module.decoder.layers.2.mlp.experts.linear_fc1.weight3
|
|
|
|
| 1116 |
module.decoder.layers.2.mlp.experts.linear_fc1.weight5
|
| 1117 |
module.decoder.layers.2.mlp.experts.linear_fc1.weight6
|
| 1118 |
+
module.decoder.layers.2.mlp.experts.linear_fc1.weight4
|
| 1119 |
Params for bucket 62 (50331648 elements, 50331648 padded size):
|
| 1120 |
module.decoder.layers.2.mlp.experts.linear_fc1.weight7
|
|
|
|
|
|
|
| 1121 |
module.decoder.layers.2.mlp.experts.linear_fc1.weight9
|
| 1122 |
+
module.decoder.layers.2.mlp.experts.linear_fc1.weight8
|
| 1123 |
+
module.decoder.layers.2.mlp.experts.linear_fc1.weight10
|
| 1124 |
Params for bucket 63 (50331648 elements, 50331648 padded size):
|
| 1125 |
module.decoder.layers.2.mlp.experts.linear_fc1.weight11
|
| 1126 |
+
module.decoder.layers.2.mlp.experts.linear_fc1.weight12
|
| 1127 |
module.decoder.layers.2.mlp.experts.linear_fc1.weight13
|
| 1128 |
module.decoder.layers.2.mlp.experts.linear_fc1.weight14
|
|
|
|
| 1129 |
Params for bucket 64 (44040192 elements, 44040192 padded size):
|
| 1130 |
module.decoder.layers.2.mlp.experts.linear_fc1.weight15
|
| 1131 |
+
module.decoder.layers.1.mlp.experts.linear_fc2.weight1
|
| 1132 |
+
module.decoder.layers.1.mlp.experts.linear_fc2.weight3
|
| 1133 |
module.decoder.layers.1.mlp.experts.linear_fc2.weight2
|
| 1134 |
module.decoder.layers.1.mlp.experts.linear_fc2.weight0
|
| 1135 |
module.decoder.layers.1.mlp.experts.linear_fc2.weight4
|
|
|
|
|
|
|
| 1136 |
Params for bucket 65 (44040192 elements, 44040192 padded size):
|
|
|
|
|
|
|
|
|
|
| 1137 |
module.decoder.layers.1.mlp.experts.linear_fc2.weight5
|
|
|
|
| 1138 |
module.decoder.layers.1.mlp.experts.linear_fc2.weight7
|
| 1139 |
+
module.decoder.layers.1.mlp.experts.linear_fc2.weight9
|
| 1140 |
module.decoder.layers.1.mlp.experts.linear_fc2.weight11
|
| 1141 |
+
module.decoder.layers.1.mlp.experts.linear_fc2.weight8
|
| 1142 |
+
module.decoder.layers.1.mlp.experts.linear_fc2.weight6
|
| 1143 |
+
module.decoder.layers.1.mlp.experts.linear_fc2.weight10
|
| 1144 |
Params for bucket 66 (50331648 elements, 50331648 padded size):
|
| 1145 |
+
module.decoder.layers.1.mlp.experts.linear_fc2.weight13
|
| 1146 |
+
module.decoder.layers.1.mlp.experts.linear_fc2.weight15
|
| 1147 |
+
module.decoder.layers.1.mlp.experts.linear_fc1.weight0
|
| 1148 |
module.decoder.layers.1.mlp.experts.linear_fc2.weight12
|
| 1149 |
module.decoder.layers.1.mlp.experts.linear_fc2.weight14
|
|
|
|
|
|
|
|
|
|
| 1150 |
module.decoder.layers.1.mlp.experts.linear_fc1.weight1
|
| 1151 |
Params for bucket 67 (50331648 elements, 50331648 padded size):
|
| 1152 |
module.decoder.layers.1.mlp.experts.linear_fc1.weight2
|
|
|
|
| 1153 |
module.decoder.layers.1.mlp.experts.linear_fc1.weight4
|
| 1154 |
module.decoder.layers.1.mlp.experts.linear_fc1.weight5
|
| 1155 |
+
module.decoder.layers.1.mlp.experts.linear_fc1.weight3
|
| 1156 |
Params for bucket 68 (50331648 elements, 50331648 padded size):
|
| 1157 |
module.decoder.layers.1.mlp.experts.linear_fc1.weight6
|
| 1158 |
module.decoder.layers.1.mlp.experts.linear_fc1.weight7
|
|
|
|
| 1160 |
module.decoder.layers.1.mlp.experts.linear_fc1.weight9
|
| 1161 |
Params for bucket 69 (50331648 elements, 50331648 padded size):
|
| 1162 |
module.decoder.layers.1.mlp.experts.linear_fc1.weight10
|
|
|
|
| 1163 |
module.decoder.layers.1.mlp.experts.linear_fc1.weight11
|
| 1164 |
module.decoder.layers.1.mlp.experts.linear_fc1.weight12
|
| 1165 |
+
module.decoder.layers.1.mlp.experts.linear_fc1.weight13
|
| 1166 |
Params for bucket 70 (44040192 elements, 44040192 padded size):
|
| 1167 |
module.decoder.layers.1.mlp.experts.linear_fc1.weight14
|
| 1168 |
module.decoder.layers.0.mlp.experts.linear_fc2.weight1
|
|
|
|
| 1169 |
module.decoder.layers.0.mlp.experts.linear_fc2.weight0
|
| 1170 |
module.decoder.layers.1.mlp.experts.linear_fc1.weight15
|
| 1171 |
+
module.decoder.layers.0.mlp.experts.linear_fc2.weight2
|
| 1172 |
Params for bucket 71 (44040192 elements, 44040192 padded size):
|
| 1173 |
module.decoder.layers.0.mlp.experts.linear_fc2.weight3
|
| 1174 |
module.decoder.layers.0.mlp.experts.linear_fc2.weight5
|
| 1175 |
module.decoder.layers.0.mlp.experts.linear_fc2.weight7
|
|
|
|
|
|
|
|
|
|
| 1176 |
module.decoder.layers.0.mlp.experts.linear_fc2.weight8
|
| 1177 |
+
module.decoder.layers.0.mlp.experts.linear_fc2.weight6
|
| 1178 |
+
module.decoder.layers.0.mlp.experts.linear_fc2.weight4
|
| 1179 |
+
module.decoder.layers.0.mlp.experts.linear_fc2.weight9
|
| 1180 |
Params for bucket 72 (50331648 elements, 50331648 padded size):
|
|
|
|
|
|
|
|
|
|
| 1181 |
module.decoder.layers.0.mlp.experts.linear_fc2.weight10
|
| 1182 |
module.decoder.layers.0.mlp.experts.linear_fc2.weight14
|
| 1183 |
module.decoder.layers.0.mlp.experts.linear_fc2.weight12
|
| 1184 |
module.decoder.layers.0.mlp.experts.linear_fc1.weight0
|
| 1185 |
+
module.decoder.layers.0.mlp.experts.linear_fc2.weight13
|
| 1186 |
+
module.decoder.layers.0.mlp.experts.linear_fc2.weight11
|
| 1187 |
+
module.decoder.layers.0.mlp.experts.linear_fc2.weight15
|
| 1188 |
Params for bucket 73 (50331648 elements, 50331648 padded size):
|
| 1189 |
module.decoder.layers.0.mlp.experts.linear_fc1.weight1
|
|
|
|
|
|
|
| 1190 |
module.decoder.layers.0.mlp.experts.linear_fc1.weight2
|
| 1191 |
+
module.decoder.layers.0.mlp.experts.linear_fc1.weight3
|
| 1192 |
+
module.decoder.layers.0.mlp.experts.linear_fc1.weight4
|
| 1193 |
Params for bucket 74 (50331648 elements, 50331648 padded size):
|
| 1194 |
module.decoder.layers.0.mlp.experts.linear_fc1.weight5
|
|
|
|
| 1195 |
module.decoder.layers.0.mlp.experts.linear_fc1.weight8
|
| 1196 |
module.decoder.layers.0.mlp.experts.linear_fc1.weight6
|
| 1197 |
+
module.decoder.layers.0.mlp.experts.linear_fc1.weight7
|
| 1198 |
Params for bucket 75 (50331648 elements, 50331648 padded size):
|
| 1199 |
module.decoder.layers.0.mlp.experts.linear_fc1.weight9
|
|
|
|
| 1200 |
module.decoder.layers.0.mlp.experts.linear_fc1.weight12
|
| 1201 |
module.decoder.layers.0.mlp.experts.linear_fc1.weight10
|
| 1202 |
+
module.decoder.layers.0.mlp.experts.linear_fc1.weight11
|
| 1203 |
Params for bucket 76 (37748736 elements, 37748736 padded size):
|
| 1204 |
module.decoder.layers.0.mlp.experts.linear_fc1.weight13
|
| 1205 |
module.decoder.layers.0.mlp.experts.linear_fc1.weight15
|
| 1206 |
module.decoder.layers.0.mlp.experts.linear_fc1.weight14
|
| 1207 |
+
INFO:megatron.core.optimizer:Setting up optimizer with config OptimizerConfig(optimizer='adam', lr=0.0001, min_lr=1e-05, decoupled_lr=None, decoupled_min_lr=None, weight_decay=0.1, fp8_recipe='mxfp8', fp16=False, bf16=True, reuse_grad_buf_for_mxfp8_param_ag=False, params_dtype=torch.bfloat16, use_precision_aware_optimizer=False, store_param_remainders=True, main_grads_dtype=torch.float32, main_params_dtype=torch.float32, exp_avg_dtype=torch.float32, exp_avg_sq_dtype=torch.float32, loss_scale=None, initial_loss_scale=4294967296, min_loss_scale=1.0, loss_scale_window=1000, hysteresis=2, adam_beta1=0.9, adam_beta2=0.999, adam_eps=1e-08, sgd_momentum=0.9, use_distributed_optimizer=True, overlap_param_gather_with_optimizer_step=False, optimizer_cpu_offload=False, optimizer_offload_fraction=1.0, use_torch_optimizer_for_cpu_offload=False, overlap_cpu_optimizer_d2h_h2d=False, pin_cpu_grads=True, pin_cpu_params=True, clip_grad=1.0, log_num_zeros_in_grad=False, barrier_with_L1_time=True, timers=<megatron.core.timers.Timers object at 0x7741db81d250>, config_logger_dir='')
|
| 1208 |
INFO:megatron.core.optimizer_param_scheduler:> learning rate decay style: cosine
|
| 1209 |
+
[after model, optimizer, and learning rate scheduler are built] datetime: 2026-08-30 01:34:14
|
| 1210 |
> building train, validation, and test datasets ...
|
| 1211 |
> datasets target sizes (minimum size):
|
| 1212 |
train: 96
|
|
|
|
| 1214 |
test: 0
|
| 1215 |
INFO:megatron.core.datasets.blended_megatron_dataset_config:Let split_matrix = [(0, 1.0), None, None]
|
| 1216 |
> building train, validation, and test datasets for GPT ...
|
| 1217 |
+
INFO:megatron.core.datasets.blended_megatron_dataset_builder:Building GPTDataset splits with sizes=(96, 0, 0) and config=GPTDatasetConfig(random_seed=1234, sequence_length=8192, blend=(['/workspace/data/wikitext/wikitext103_gpt2_text_document'], None), blend_per_split=None, split='100,0,0', split_matrix=[(0, 1.0), None, None], num_dataset_builder_threads=1, path_to_cache=None, mmap_bin_files=True, mock=False, tokenizer=<megatron.training.tokenizer.tokenizer._GPT2BPETokenizer object at 0x773f352a9970>, mid_level_dataset_surplus=0.005, reset_position_ids=False, reset_attention_mask=False, eod_mask_loss=False, create_attention_mask=True, drop_last_partial_validation_sequence=True, add_extra_token_to_sequence=True, object_storage_cache_path=None)
|
| 1218 |
INFO:megatron.core.datasets.indexed_dataset:Load the _IndexReader from /workspace/data/wikitext/wikitext103_gpt2_text_document.idx
|
| 1219 |
INFO:megatron.core.datasets.indexed_dataset: Extract the sequence lengths
|
| 1220 |
INFO:megatron.core.datasets.indexed_dataset: Extract the sequence pointers
|
|
|
|
| 1227 |
INFO:megatron.core.datasets.gpt_dataset: Load the shuffle index from eadc3756447e39d609e8aa728a91f451-GPTDataset-train-shuffle_index.npy
|
| 1228 |
INFO:megatron.core.datasets.gpt_dataset:> total number of samples: 14536
|
| 1229 |
> finished creating GPT datasets ...
|
| 1230 |
+
[after dataloaders are built] datetime: 2026-08-30 01:34:14
|
| 1231 |
done with setup ...
|
| 1232 |
training ...
|
| 1233 |
Overwriting rerun_state_machine.current_iteration from -1 to 0...
|
| 1234 |
(min, max) time across ranks (ms):
|
| 1235 |
+
model-and-optimizer-setup ......................: (3338.00, 3360.91)
|
| 1236 |
+
train/valid/test-data-iterators-setup ..........: (63.69, 75.16)
|
| 1237 |
+
[before the start of training step] datetime: 2026-08-30 01:34:14
|
| 1238 |
INFO:moonep.api:MoonEP comm buffer sizes:
|
| 1239 |
config : R=8, S=8192, H=4096, E=128, K=8, NvS=69600, NvS_padded=69632
|
| 1240 |
sm_count : num_sms=32, num_sms_dedup=148
|
|
|
|
| 1246 |
[MoonEP selfcheck 2/4] dropped groups empty OK | covered rows [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 128, 129, 130, 131, 132, 133, 134, 135, 136, 137, 138, 139, 140, 141, 142, 143] sizes [9344, 2688, 128, 24192, 128, 0, 128, 256, 128, 768, 10496, 128, 128, 2432, 128, 1536, 14208, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0] | duplicated experts [[69, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1], [88, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1], [55, 52, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1], [-1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1], [63, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1], [124, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1], [10, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1], [58, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1]]
|
| 1247 |
[MoonEP selfcheck 3/4] dropped groups empty OK | covered rows [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 128, 129, 130, 131, 132, 133, 134, 135, 136, 137, 138, 139, 140, 141, 142, 143] sizes [128, 128, 128, 0, 128, 44416, 128, 128, 128, 6912, 128, 256, 128, 128, 128, 128, 13824, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0] | duplicated experts [[78, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1], [111, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1], [62, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1], [-1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1], [57, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1], [106, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1], [121, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1], [57, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1]]
|
| 1248 |
[MoonEP selfcheck 4/4] dropped groups empty OK | covered rows [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 128, 129, 130, 131, 132, 133, 134, 135, 136, 137, 138, 139, 140, 141, 142, 143] sizes [2304, 128, 128, 512, 128, 12416, 128, 1152, 1280, 16768, 640, 29056, 128, 128, 128, 1664, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0] | duplicated experts [[-1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1], [118, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1], [11, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1], [118, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1], [39, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1], [11, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1], [73, 79, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1], [86, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1]]
|
| 1249 |
+
[2026-08-30 01:34:35] iteration 1/ 12 | consumed samples: 8 | elapsed time per iteration (ms): 21275.7 | throughput per GPU (TFLOP/s/GPU): 8.5 | learning rate: 2.000000E-05 | global batch size: 8 | lm loss: 1.115993E+01 | loss scale: 1.0 | grad norm: 44.806 | number of skipped iterations: 0 | number of nan iterations: 0 |
|
| 1250 |
Number of parameters in transformer block in billions: 29.85
|
| 1251 |
Number of parameters in embedding layers in billions: 0.41
|
| 1252 |
Total number of parameters in billions: 30.26
|
| 1253 |
Number of parameters in most loaded shard in billions: 30.2590
|
| 1254 |
Theoretical memory footprints: weight and optimizer=216428.96 MB
|
| 1255 |
[Rank 0] (after 1 iterations) memory (MB) | allocated: 67239.548828125 | max allocated: 67239.54931640625 | reserved: 68548.0 | max reserved: 68548.0
|
| 1256 |
+
[2026-08-30 01:34:36] iteration 2/ 12 | consumed samples: 16 | elapsed time per iteration (ms): 364.0 | throughput per GPU (TFLOP/s/GPU): 496.8 | learning rate: 4.000000E-05 | global batch size: 8 | lm loss: 1.117600E+01 | loss scale: 1.0 | grad norm: 44.336 | number of skipped iterations: 0 | number of nan iterations: 0 |
|
| 1257 |
+
[2026-08-30 01:34:36] iteration 3/ 12 | consumed samples: 24 | elapsed time per iteration (ms): 331.4 | throughput per GPU (TFLOP/s/GPU): 545.7 | learning rate: 6.000000E-05 | global batch size: 8 | lm loss: 9.686085E+00 | loss scale: 1.0 | grad norm: 62.885 | number of skipped iterations: 0 | number of nan iterations: 0 |
|
| 1258 |
Capture CUDA graph for training!!!
|
| 1259 |
CUDA graph capture done!!!
|
| 1260 |
+
[2026-08-30 01:34:37] iteration 4/ 12 | consumed samples: 32 | elapsed time per iteration (ms): 979.0 | throughput per GPU (TFLOP/s/GPU): 184.7 | learning rate: 8.000000E-05 | global batch size: 8 | lm loss: 1.338064E+01 | loss scale: 1.0 | grad norm: 20.299 | number of skipped iterations: 0 | number of nan iterations: 0 |
|
| 1261 |
+
[2026-08-30 01:34:37] iteration 5/ 12 | consumed samples: 40 | elapsed time per iteration (ms): 319.7 | throughput per GPU (TFLOP/s/GPU): 565.7 | learning rate: 1.000000E-04 | global batch size: 8 | lm loss: 1.049333E+01 | loss scale: 1.0 | grad norm: 95.200 | number of skipped iterations: 0 | number of nan iterations: 0 |
|
| 1262 |
+
[2026-08-30 01:34:38] iteration 6/ 12 | consumed samples: 48 | elapsed time per iteration (ms): 318.9 | throughput per GPU (TFLOP/s/GPU): 567.0 | learning rate: 7.750000E-05 | global batch size: 8 | lm loss: 1.048383E+01 | loss scale: 1.0 | grad norm: 54.327 | number of skipped iterations: 0 | number of nan iterations: 0 |
|
| 1263 |
+
[2026-08-30 01:34:38] iteration 7/ 12 | consumed samples: 56 | elapsed time per iteration (ms): 318.4 | throughput per GPU (TFLOP/s/GPU): 568.0 | learning rate: 3.250000E-05 | global batch size: 8 | lm loss: 1.027052E+01 | loss scale: 1.0 | grad norm: 6.338 | number of skipped iterations: 0 | number of nan iterations: 0 |
|
| 1264 |
+
[2026-08-30 01:34:38] iteration 8/ 12 | consumed samples: 64 | elapsed time per iteration (ms): 318.4 | throughput per GPU (TFLOP/s/GPU): 567.8 | learning rate: 1.000000E-05 | global batch size: 8 | lm loss: 9.809936E+00 | loss scale: 1.0 | grad norm: 5.526 | number of skipped iterations: 0 | number of nan iterations: 0 |
|
| 1265 |
+
[2026-08-30 01:34:38] iteration 9/ 12 | consumed samples: 72 | elapsed time per iteration (ms): 317.7 | throughput per GPU (TFLOP/s/GPU): 569.2 | learning rate: 1.000000E-05 | global batch size: 8 | lm loss: 9.548426E+00 | loss scale: 1.0 | grad norm: 5.232 | number of skipped iterations: 0 | number of nan iterations: 0 |
|
| 1266 |
+
[2026-08-30 01:34:39] iteration 10/ 12 | consumed samples: 80 | elapsed time per iteration (ms): 318.2 | throughput per GPU (TFLOP/s/GPU): 568.2 | learning rate: 1.000000E-05 | global batch size: 8 | lm loss: 9.449459E+00 | loss scale: 1.0 | grad norm: 4.938 | number of skipped iterations: 0 | number of nan iterations: 0 |
|
| 1267 |
+
[2026-08-30 01:34:39] iteration 11/ 12 | consumed samples: 88 | elapsed time per iteration (ms): 318.6 | throughput per GPU (TFLOP/s/GPU): 567.6 | learning rate: 1.000000E-05 | global batch size: 8 | lm loss: 9.331503E+00 | loss scale: 1.0 | grad norm: 4.636 | number of skipped iterations: 0 | number of nan iterations: 0 |
|
| 1268 |
+
[2026-08-30 01:34:39] iteration 12/ 12 | consumed samples: 96 | elapsed time per iteration (ms): 318.2 | throughput per GPU (TFLOP/s/GPU): 568.4 | learning rate: 1.000000E-05 | global batch size: 8 | lm loss: 9.217281E+00 | loss scale: 1.0 | grad norm: 4.482 | number of skipped iterations: 0 | number of nan iterations: 0 |
|
| 1269 |
+
[after training is done] datetime: 2026-08-30 01:34:39
|
| 1270 |
+
[rank0]:[W830 01:34:41.434365646 ProcessGroupNCCL.cpp:1505] Warning: WARNING: destroy_process_group() was not called before program exit, which can leak resources. For more info, please see https://pytorch.org/docs/stable/distributed.html#shutdown (function operator())
|
| 1271 |
+
[rank6]:[W830 01:34:42.271159288 ProcessGroupNCCL.cpp:1505] Warning: WARNING: destroy_process_group() was not called before program exit, which can leak resources. For more info, please see https://pytorch.org/docs/stable/distributed.html#shutdown (function operator())
|
| 1272 |
+
[rank4]:[W830 01:34:42.512519329 ProcessGroupNCCL.cpp:1505] Warning: WARNING: destroy_process_group() was not called before program exit, which can leak resources. For more info, please see https://pytorch.org/docs/stable/distributed.html#shutdown (function operator())
|
| 1273 |
+
[rank2]:[W830 01:34:42.516537505 ProcessGroupNCCL.cpp:1505] Warning: WARNING: destroy_process_group() was not called before program exit, which can leak resources. For more info, please see https://pytorch.org/docs/stable/distributed.html#shutdown (function operator())
|
| 1274 |
+
[rank7]:[W830 01:34:42.636432904 ProcessGroupNCCL.cpp:1505] Warning: WARNING: destroy_process_group() was not called before program exit, which can leak resources. For more info, please see https://pytorch.org/docs/stable/distributed.html#shutdown (function operator())
|
| 1275 |
+
[rank3]:[W830 01:34:42.082170425 ProcessGroupNCCL.cpp:1505] Warning: WARNING: destroy_process_group() was not called before program exit, which can leak resources. For more info, please see https://pytorch.org/docs/stable/distributed.html#shutdown (function operator())
|
| 1276 |
+
[rank5]:[W830 01:34:42.086366756 ProcessGroupNCCL.cpp:1505] Warning: WARNING: destroy_process_group() was not called before program exit, which can leak resources. For more info, please see https://pytorch.org/docs/stable/distributed.html#shutdown (function operator())
|
| 1277 |
+
[rank1]:[W830 01:34:42.088352473 ProcessGroupNCCL.cpp:1505] Warning: WARNING: destroy_process_group() was not called before program exit, which can leak resources. For more info, please see https://pytorch.org/docs/stable/distributed.html#shutdown (function operator())
|
| 1278 |
+
W0830 01:34:46.823000 385325 torch/distributed/elastic/agent/server/api.py:719] Received 15 death signal, shutting down workers
|
| 1279 |
+
W0830 01:34:46.823000 385325 torch/distributed/elastic/multiprocessing/api.py:900] Sending process 385510 closing signal SIGTERM
|
| 1280 |
+
W0830 01:34:46.824000 385325 torch/distributed/elastic/multiprocessing/api.py:900] Sending process 385511 closing signal SIGTERM
|
| 1281 |
+
W0830 01:34:46.824000 385325 torch/distributed/elastic/multiprocessing/api.py:900] Sending process 385512 closing signal SIGTERM
|
| 1282 |
+
W0830 01:34:46.825000 385325 torch/distributed/elastic/multiprocessing/api.py:900] Sending process 385513 closing signal SIGTERM
|
| 1283 |
+
W0830 01:34:46.825000 385325 torch/distributed/elastic/multiprocessing/api.py:900] Sending process 385514 closing signal SIGTERM
|
| 1284 |
+
W0830 01:34:46.826000 385325 torch/distributed/elastic/multiprocessing/api.py:900] Sending process 385515 closing signal SIGTERM
|
|
|
|
|
|
|
| 1285 |
Traceback (most recent call last):
|
| 1286 |
File "/usr/local/bin/torchrun", line 8, in <module>
|
| 1287 |
sys.exit(main())
|
|
|
|
| 1309 |
time.sleep(monitor_interval)
|
| 1310 |
File "/usr/local/lib/python3.12/dist-packages/torch/distributed/elastic/multiprocessing/api.py", line 84, in _terminate_process_handler
|
| 1311 |
raise SignalException(f"Process {os.getpid()} got signal: {sigval}", sigval=sigval)
|
| 1312 |
+
torch.distributed.elastic.multiprocessing.api.SignalException: Process 385325 got signal: 15
|
tierb-fixed-moonep-full/moonep/tierb/tierb_gbs8/QWEN3_30B__moonep__matched__seq4096__gbs8__capnone__async/rep1/mem_samples.csv
CHANGED
|
@@ -20,13 +20,13 @@
|
|
| 20 |
3, 4
|
| 21 |
4, 4
|
| 22 |
5, 4
|
| 23 |
-
6,
|
| 24 |
7, 1
|
| 25 |
0, 1
|
| 26 |
1, 1
|
| 27 |
2, 1
|
| 28 |
3, 1
|
| 29 |
-
4,
|
| 30 |
5, 4
|
| 31 |
6, 4
|
| 32 |
7, 4
|
|
@@ -45,20 +45,12 @@
|
|
| 45 |
4, 1
|
| 46 |
5, 1
|
| 47 |
6, 1
|
| 48 |
-
7, 0
|
| 49 |
-
0, 4
|
| 50 |
-
1, 1
|
| 51 |
-
2, 1
|
| 52 |
-
3, 1
|
| 53 |
-
4, 1
|
| 54 |
-
5, 1
|
| 55 |
-
6, 1
|
| 56 |
7, 1
|
| 57 |
0, 4
|
| 58 |
1, 4
|
| 59 |
2, 4
|
| 60 |
3, 4
|
| 61 |
-
4,
|
| 62 |
5, 1
|
| 63 |
6, 1
|
| 64 |
7, 1
|
|
@@ -70,22 +62,22 @@
|
|
| 70 |
5, 4
|
| 71 |
6, 4
|
| 72 |
7, 4
|
| 73 |
-
0,
|
| 74 |
-
1,
|
| 75 |
-
2,
|
| 76 |
-
3,
|
| 77 |
-
4,
|
| 78 |
-
5,
|
| 79 |
-
6,
|
| 80 |
-
7,
|
| 81 |
-
0,
|
| 82 |
-
1,
|
| 83 |
-
2,
|
| 84 |
-
3,
|
| 85 |
-
4,
|
| 86 |
-
5,
|
| 87 |
-
6,
|
| 88 |
-
7,
|
| 89 |
0, 1831
|
| 90 |
1, 1831
|
| 91 |
2, 1831
|
|
@@ -94,14 +86,14 @@
|
|
| 94 |
5, 1831
|
| 95 |
6, 1831
|
| 96 |
7, 1831
|
| 97 |
-
0,
|
| 98 |
-
1,
|
| 99 |
-
2,
|
| 100 |
-
3,
|
| 101 |
-
4,
|
| 102 |
-
5,
|
| 103 |
-
6,
|
| 104 |
-
7,
|
| 105 |
0, 47207
|
| 106 |
1, 47207
|
| 107 |
2, 47207
|
|
@@ -110,7 +102,7 @@
|
|
| 110 |
5, 47207
|
| 111 |
6, 47207
|
| 112 |
7, 47207
|
| 113 |
-
0,
|
| 114 |
1, 47551
|
| 115 |
2, 47551
|
| 116 |
3, 47551
|
|
@@ -118,54 +110,54 @@
|
|
| 118 |
5, 47551
|
| 119 |
6, 47551
|
| 120 |
7, 47551
|
| 121 |
-
0,
|
| 122 |
-
1,
|
| 123 |
-
2,
|
| 124 |
-
3,
|
| 125 |
-
4,
|
| 126 |
-
5,
|
| 127 |
-
6,
|
| 128 |
-
7,
|
| 129 |
-
0,
|
| 130 |
-
1,
|
| 131 |
-
2,
|
| 132 |
-
3,
|
| 133 |
-
4,
|
| 134 |
-
5,
|
| 135 |
-
6,
|
| 136 |
-
7,
|
| 137 |
-
0,
|
| 138 |
1, 48905
|
| 139 |
-
2,
|
| 140 |
3, 48905
|
| 141 |
4, 49041
|
| 142 |
-
5,
|
| 143 |
-
6,
|
| 144 |
-
7,
|
| 145 |
0, 49251
|
| 146 |
-
1,
|
| 147 |
2, 49279
|
| 148 |
-
3,
|
| 149 |
4, 49043
|
| 150 |
-
5,
|
| 151 |
6, 48907
|
| 152 |
7, 49279
|
| 153 |
-
0,
|
| 154 |
-
1,
|
| 155 |
-
2,
|
| 156 |
-
3,
|
| 157 |
-
4,
|
| 158 |
-
5,
|
| 159 |
-
6,
|
| 160 |
-
7,
|
| 161 |
-
0,
|
| 162 |
-
1,
|
| 163 |
-
2,
|
| 164 |
-
3,
|
| 165 |
-
4,
|
| 166 |
-
5,
|
| 167 |
-
6,
|
| 168 |
-
7,
|
| 169 |
0, 83947
|
| 170 |
1, 83951
|
| 171 |
2, 83941
|
|
@@ -174,67 +166,67 @@
|
|
| 174 |
5, 83941
|
| 175 |
6, 83931
|
| 176 |
7, 83941
|
| 177 |
-
0,
|
| 178 |
-
1,
|
| 179 |
-
2,
|
| 180 |
-
3,
|
| 181 |
-
4,
|
| 182 |
-
5,
|
| 183 |
-
6,
|
| 184 |
-
7,
|
| 185 |
-
0,
|
| 186 |
-
1,
|
| 187 |
-
2,
|
| 188 |
-
3,
|
| 189 |
-
4,
|
| 190 |
-
5,
|
| 191 |
-
6,
|
| 192 |
-
7,
|
| 193 |
-
0,
|
| 194 |
-
1,
|
| 195 |
-
2,
|
| 196 |
-
3,
|
| 197 |
-
4,
|
| 198 |
-
5,
|
| 199 |
-
6,
|
| 200 |
-
7,
|
| 201 |
-
0,
|
| 202 |
-
1,
|
| 203 |
-
2,
|
| 204 |
-
3,
|
| 205 |
-
4,
|
| 206 |
-
5,
|
| 207 |
-
6,
|
| 208 |
-
7,
|
| 209 |
-
0,
|
| 210 |
-
1,
|
| 211 |
-
2,
|
| 212 |
-
3,
|
| 213 |
-
4,
|
| 214 |
-
5,
|
| 215 |
-
6,
|
| 216 |
-
7,
|
| 217 |
-
0,
|
| 218 |
-
1,
|
| 219 |
-
2,
|
| 220 |
-
3,
|
| 221 |
-
4,
|
| 222 |
-
5,
|
| 223 |
-
6,
|
| 224 |
-
7,
|
| 225 |
-
0,
|
| 226 |
-
1,
|
| 227 |
-
2,
|
| 228 |
-
3,
|
| 229 |
-
4,
|
| 230 |
-
5,
|
| 231 |
-
6,
|
| 232 |
-
7,
|
| 233 |
-
0,
|
| 234 |
-
1,
|
| 235 |
-
2,
|
| 236 |
-
3,
|
| 237 |
-
4,
|
| 238 |
5, 21174
|
| 239 |
6, 21174
|
| 240 |
7, 21174
|
|
|
|
| 20 |
3, 4
|
| 21 |
4, 4
|
| 22 |
5, 4
|
| 23 |
+
6, 4
|
| 24 |
7, 1
|
| 25 |
0, 1
|
| 26 |
1, 1
|
| 27 |
2, 1
|
| 28 |
3, 1
|
| 29 |
+
4, 1
|
| 30 |
5, 4
|
| 31 |
6, 4
|
| 32 |
7, 4
|
|
|
|
| 45 |
4, 1
|
| 46 |
5, 1
|
| 47 |
6, 1
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 48 |
7, 1
|
| 49 |
0, 4
|
| 50 |
1, 4
|
| 51 |
2, 4
|
| 52 |
3, 4
|
| 53 |
+
4, 1
|
| 54 |
5, 1
|
| 55 |
6, 1
|
| 56 |
7, 1
|
|
|
|
| 62 |
5, 4
|
| 63 |
6, 4
|
| 64 |
7, 4
|
| 65 |
+
0, 725
|
| 66 |
+
1, 725
|
| 67 |
+
2, 725
|
| 68 |
+
3, 725
|
| 69 |
+
4, 725
|
| 70 |
+
5, 725
|
| 71 |
+
6, 725
|
| 72 |
+
7, 725
|
| 73 |
+
0, 1079
|
| 74 |
+
1, 1077
|
| 75 |
+
2, 1079
|
| 76 |
+
3, 1079
|
| 77 |
+
4, 1079
|
| 78 |
+
5, 1079
|
| 79 |
+
6, 1079
|
| 80 |
+
7, 1079
|
| 81 |
0, 1831
|
| 82 |
1, 1831
|
| 83 |
2, 1831
|
|
|
|
| 86 |
5, 1831
|
| 87 |
6, 1831
|
| 88 |
7, 1831
|
| 89 |
+
0, 19271
|
| 90 |
+
1, 19269
|
| 91 |
+
2, 19277
|
| 92 |
+
3, 19283
|
| 93 |
+
4, 19291
|
| 94 |
+
5, 19287
|
| 95 |
+
6, 19291
|
| 96 |
+
7, 19277
|
| 97 |
0, 47207
|
| 98 |
1, 47207
|
| 99 |
2, 47207
|
|
|
|
| 102 |
5, 47207
|
| 103 |
6, 47207
|
| 104 |
7, 47207
|
| 105 |
+
0, 47549
|
| 106 |
1, 47551
|
| 107 |
2, 47551
|
| 108 |
3, 47551
|
|
|
|
| 110 |
5, 47551
|
| 111 |
6, 47551
|
| 112 |
7, 47551
|
| 113 |
+
0, 48313
|
| 114 |
+
1, 47969
|
| 115 |
+
2, 48341
|
| 116 |
+
3, 47969
|
| 117 |
+
4, 48105
|
| 118 |
+
5, 48341
|
| 119 |
+
6, 47969
|
| 120 |
+
7, 48341
|
| 121 |
+
0, 49103
|
| 122 |
+
1, 48759
|
| 123 |
+
2, 49131
|
| 124 |
+
3, 48759
|
| 125 |
+
4, 48895
|
| 126 |
+
5, 49131
|
| 127 |
+
6, 48761
|
| 128 |
+
7, 49131
|
| 129 |
+
0, 49249
|
| 130 |
1, 48905
|
| 131 |
+
2, 49277
|
| 132 |
3, 48905
|
| 133 |
4, 49041
|
| 134 |
+
5, 49277
|
| 135 |
+
6, 48907
|
| 136 |
+
7, 49277
|
| 137 |
0, 49251
|
| 138 |
+
1, 48907
|
| 139 |
2, 49279
|
| 140 |
+
3, 48905
|
| 141 |
4, 49043
|
| 142 |
+
5, 49277
|
| 143 |
6, 48907
|
| 144 |
7, 49279
|
| 145 |
+
0, 62969
|
| 146 |
+
1, 62867
|
| 147 |
+
2, 62963
|
| 148 |
+
3, 62953
|
| 149 |
+
4, 62981
|
| 150 |
+
5, 62963
|
| 151 |
+
6, 62953
|
| 152 |
+
7, 62963
|
| 153 |
+
0, 83461
|
| 154 |
+
1, 83465
|
| 155 |
+
2, 83463
|
| 156 |
+
3, 83461
|
| 157 |
+
4, 83517
|
| 158 |
+
5, 83475
|
| 159 |
+
6, 83467
|
| 160 |
+
7, 83469
|
| 161 |
0, 83947
|
| 162 |
1, 83951
|
| 163 |
2, 83941
|
|
|
|
| 166 |
5, 83941
|
| 167 |
6, 83931
|
| 168 |
7, 83941
|
| 169 |
+
0, 84269
|
| 170 |
+
1, 84223
|
| 171 |
+
2, 84199
|
| 172 |
+
3, 84223
|
| 173 |
+
4, 84339
|
| 174 |
+
5, 84293
|
| 175 |
+
6, 84327
|
| 176 |
+
7, 84265
|
| 177 |
+
0, 89737
|
| 178 |
+
1, 89989
|
| 179 |
+
2, 89763
|
| 180 |
+
3, 89657
|
| 181 |
+
4, 89661
|
| 182 |
+
5, 89907
|
| 183 |
+
6, 89813
|
| 184 |
+
7, 90003
|
| 185 |
+
0, 95559
|
| 186 |
+
1, 95447
|
| 187 |
+
2, 95535
|
| 188 |
+
3, 95295
|
| 189 |
+
4, 95449
|
| 190 |
+
5, 95557
|
| 191 |
+
6, 95403
|
| 192 |
+
7, 95569
|
| 193 |
+
0, 105447
|
| 194 |
+
1, 106009
|
| 195 |
+
2, 105845
|
| 196 |
+
3, 107203
|
| 197 |
+
4, 106127
|
| 198 |
+
5, 106645
|
| 199 |
+
6, 105807
|
| 200 |
+
7, 111847
|
| 201 |
+
0, 116707
|
| 202 |
+
1, 116615
|
| 203 |
+
2, 116705
|
| 204 |
+
3, 116463
|
| 205 |
+
4, 116587
|
| 206 |
+
5, 116705
|
| 207 |
+
6, 116559
|
| 208 |
+
7, 116705
|
| 209 |
+
0, 116707
|
| 210 |
+
1, 116615
|
| 211 |
+
2, 116705
|
| 212 |
+
3, 116463
|
| 213 |
+
4, 116587
|
| 214 |
+
5, 116705
|
| 215 |
+
6, 116559
|
| 216 |
+
7, 116705
|
| 217 |
+
0, 115739
|
| 218 |
+
1, 115967
|
| 219 |
+
2, 116057
|
| 220 |
+
3, 115815
|
| 221 |
+
4, 115939
|
| 222 |
+
5, 116057
|
| 223 |
+
6, 115911
|
| 224 |
+
7, 115795
|
| 225 |
+
0, 21173
|
| 226 |
+
1, 21430
|
| 227 |
+
2, 116057
|
| 228 |
+
3, 22454
|
| 229 |
+
4, 21174
|
| 230 |
5, 21174
|
| 231 |
6, 21174
|
| 232 |
7, 21174
|
tierb-fixed-moonep-full/moonep/tierb/tierb_gbs8/QWEN3_30B__moonep__matched__seq4096__gbs8__capnone__async/rep1/run.env
CHANGED
|
@@ -3,7 +3,7 @@ config=QWEN3_30B
|
|
| 3 |
family=tierB
|
| 4 |
repo=/workspace/Megatron-MoE-Echo
|
| 5 |
branch=moonep_backend_async_grad_reduce
|
| 6 |
-
commit=
|
| 7 |
moonep_source_dir=/workspace/MoonEP
|
| 8 |
moonep_source_branch=megatron-ddp-symmetric-params
|
| 9 |
moonep_source_commit=d8bc666
|
|
|
|
| 3 |
family=tierB
|
| 4 |
repo=/workspace/Megatron-MoE-Echo
|
| 5 |
branch=moonep_backend_async_grad_reduce
|
| 6 |
+
commit=2e68c8ad
|
| 7 |
moonep_source_dir=/workspace/MoonEP
|
| 8 |
moonep_source_branch=megatron-ddp-symmetric-params
|
| 9 |
moonep_source_commit=d8bc666
|
tierb-fixed-moonep-full/moonep/tierb/tierb_gbs8/QWEN3_30B__moonep__matched__seq4096__gbs8__capnone__async/rep1/train.log
CHANGED
|
The diff for this file is too large to render.
See raw diff
|
|
|
tierb-fixed-moonep-full/moonep/tierb/tierb_gbs8/QWEN3_30B__moonep__matched__seq4096__gbs8__capnone__async/rep2/mem_samples.csv
CHANGED
|
@@ -26,8 +26,8 @@
|
|
| 26 |
1, 1
|
| 27 |
2, 1
|
| 28 |
3, 1
|
| 29 |
-
4,
|
| 30 |
-
5,
|
| 31 |
6, 4
|
| 32 |
7, 4
|
| 33 |
0, 0
|
|
@@ -38,7 +38,7 @@
|
|
| 38 |
5, 0
|
| 39 |
6, 0
|
| 40 |
7, 0
|
| 41 |
-
0,
|
| 42 |
1, 1
|
| 43 |
2, 1
|
| 44 |
3, 1
|
|
@@ -71,37 +71,37 @@
|
|
| 71 |
6, 4
|
| 72 |
7, 4
|
| 73 |
0, 821
|
| 74 |
-
1,
|
| 75 |
-
2,
|
| 76 |
3, 821
|
| 77 |
-
4,
|
| 78 |
-
5,
|
| 79 |
-
6,
|
| 80 |
-
7,
|
| 81 |
-
0,
|
| 82 |
-
1,
|
| 83 |
-
2,
|
| 84 |
-
3,
|
| 85 |
-
4,
|
| 86 |
-
5,
|
| 87 |
-
6,
|
| 88 |
-
7,
|
| 89 |
-
0,
|
| 90 |
-
1,
|
| 91 |
-
2,
|
| 92 |
-
3,
|
| 93 |
-
4,
|
| 94 |
-
5,
|
| 95 |
-
6,
|
| 96 |
-
7,
|
| 97 |
-
0,
|
| 98 |
-
1,
|
| 99 |
-
2,
|
| 100 |
-
3,
|
| 101 |
-
4,
|
| 102 |
-
5,
|
| 103 |
-
6,
|
| 104 |
-
7,
|
| 105 |
0, 47547
|
| 106 |
1, 47549
|
| 107 |
2, 47549
|
|
@@ -110,54 +110,54 @@
|
|
| 110 |
5, 47549
|
| 111 |
6, 47549
|
| 112 |
7, 47549
|
| 113 |
-
0,
|
| 114 |
-
1,
|
| 115 |
-
2,
|
| 116 |
-
3,
|
| 117 |
-
4,
|
| 118 |
-
5,
|
| 119 |
-
6,
|
| 120 |
-
7,
|
| 121 |
-
0,
|
| 122 |
-
1,
|
| 123 |
-
2,
|
| 124 |
-
3,
|
| 125 |
-
4,
|
| 126 |
-
5,
|
| 127 |
-
6,
|
| 128 |
-
7,
|
| 129 |
-
0,
|
| 130 |
1, 48905
|
| 131 |
2, 49279
|
| 132 |
-
3,
|
| 133 |
4, 49041
|
| 134 |
5, 49277
|
| 135 |
6, 48905
|
| 136 |
-
7,
|
| 137 |
0, 49251
|
| 138 |
-
1,
|
| 139 |
2, 49279
|
| 140 |
3, 48907
|
| 141 |
4, 49043
|
| 142 |
5, 49279
|
| 143 |
6, 48907
|
| 144 |
7, 49279
|
| 145 |
-
0,
|
| 146 |
-
1,
|
| 147 |
-
2,
|
| 148 |
-
3,
|
| 149 |
-
4,
|
| 150 |
-
5,
|
| 151 |
-
6,
|
| 152 |
-
7,
|
| 153 |
-
0,
|
| 154 |
-
1,
|
| 155 |
-
2,
|
| 156 |
-
3,
|
| 157 |
-
4,
|
| 158 |
-
5,
|
| 159 |
-
6,
|
| 160 |
-
7,
|
| 161 |
0, 83947
|
| 162 |
1, 83951
|
| 163 |
2, 83941
|
|
@@ -166,67 +166,67 @@
|
|
| 166 |
5, 83941
|
| 167 |
6, 83931
|
| 168 |
7, 83941
|
| 169 |
-
0,
|
| 170 |
-
1,
|
| 171 |
-
2,
|
| 172 |
-
3,
|
| 173 |
-
4,
|
| 174 |
-
5,
|
| 175 |
-
6,
|
| 176 |
-
7,
|
| 177 |
-
0,
|
| 178 |
-
1,
|
| 179 |
-
2,
|
| 180 |
-
3,
|
| 181 |
-
4,
|
| 182 |
-
5,
|
| 183 |
-
6,
|
| 184 |
-
7,
|
| 185 |
-
0,
|
| 186 |
-
1,
|
| 187 |
-
2,
|
| 188 |
-
3,
|
| 189 |
-
4,
|
| 190 |
-
5,
|
| 191 |
-
6,
|
| 192 |
-
7,
|
| 193 |
-
0,
|
| 194 |
-
1,
|
| 195 |
-
2,
|
| 196 |
-
3,
|
| 197 |
-
4,
|
| 198 |
-
5,
|
| 199 |
-
6,
|
| 200 |
-
7,
|
| 201 |
-
0,
|
| 202 |
-
1,
|
| 203 |
-
2,
|
| 204 |
-
3,
|
| 205 |
-
4,
|
| 206 |
-
5,
|
| 207 |
-
6,
|
| 208 |
-
7,
|
| 209 |
-
0,
|
| 210 |
-
1,
|
| 211 |
-
2,
|
| 212 |
-
3,
|
| 213 |
-
4,
|
| 214 |
-
5,
|
| 215 |
-
6,
|
| 216 |
-
7,
|
| 217 |
-
0,
|
| 218 |
-
1,
|
| 219 |
-
2,
|
| 220 |
-
3,
|
| 221 |
-
4,
|
| 222 |
-
5,
|
| 223 |
-
6,
|
| 224 |
-
7,
|
| 225 |
-
0,
|
| 226 |
-
1,
|
| 227 |
-
2,
|
| 228 |
-
3,
|
| 229 |
-
4,
|
| 230 |
-
5,
|
| 231 |
-
6,
|
| 232 |
-
7,
|
|
|
|
| 26 |
1, 1
|
| 27 |
2, 1
|
| 28 |
3, 1
|
| 29 |
+
4, 1
|
| 30 |
+
5, 3
|
| 31 |
6, 4
|
| 32 |
7, 4
|
| 33 |
0, 0
|
|
|
|
| 38 |
5, 0
|
| 39 |
6, 0
|
| 40 |
7, 0
|
| 41 |
+
0, 4
|
| 42 |
1, 1
|
| 43 |
2, 1
|
| 44 |
3, 1
|
|
|
|
| 71 |
6, 4
|
| 72 |
7, 4
|
| 73 |
0, 821
|
| 74 |
+
1, 819
|
| 75 |
+
2, 819
|
| 76 |
3, 821
|
| 77 |
+
4, 819
|
| 78 |
+
5, 819
|
| 79 |
+
6, 819
|
| 80 |
+
7, 819
|
| 81 |
+
0, 1537
|
| 82 |
+
1, 1537
|
| 83 |
+
2, 1537
|
| 84 |
+
3, 1537
|
| 85 |
+
4, 1537
|
| 86 |
+
5, 1537
|
| 87 |
+
6, 1537
|
| 88 |
+
7, 1537
|
| 89 |
+
0, 1975
|
| 90 |
+
1, 1975
|
| 91 |
+
2, 1975
|
| 92 |
+
3, 1975
|
| 93 |
+
4, 1975
|
| 94 |
+
5, 1975
|
| 95 |
+
6, 1975
|
| 96 |
+
7, 1975
|
| 97 |
+
0, 46653
|
| 98 |
+
1, 46653
|
| 99 |
+
2, 46645
|
| 100 |
+
3, 46629
|
| 101 |
+
4, 46653
|
| 102 |
+
5, 46657
|
| 103 |
+
6, 46665
|
| 104 |
+
7, 46651
|
| 105 |
0, 47547
|
| 106 |
1, 47549
|
| 107 |
2, 47549
|
|
|
|
| 110 |
5, 47549
|
| 111 |
6, 47549
|
| 112 |
7, 47549
|
| 113 |
+
0, 48127
|
| 114 |
+
1, 47943
|
| 115 |
+
2, 48147
|
| 116 |
+
3, 47929
|
| 117 |
+
4, 47959
|
| 118 |
+
5, 48155
|
| 119 |
+
6, 47929
|
| 120 |
+
7, 48161
|
| 121 |
+
0, 48611
|
| 122 |
+
1, 48293
|
| 123 |
+
2, 48643
|
| 124 |
+
3, 48267
|
| 125 |
+
4, 48369
|
| 126 |
+
5, 48631
|
| 127 |
+
6, 48267
|
| 128 |
+
7, 48655
|
| 129 |
+
0, 49251
|
| 130 |
1, 48905
|
| 131 |
2, 49279
|
| 132 |
+
3, 48905
|
| 133 |
4, 49041
|
| 134 |
5, 49277
|
| 135 |
6, 48905
|
| 136 |
+
7, 49277
|
| 137 |
0, 49251
|
| 138 |
+
1, 48907
|
| 139 |
2, 49279
|
| 140 |
3, 48907
|
| 141 |
4, 49043
|
| 142 |
5, 49279
|
| 143 |
6, 48907
|
| 144 |
7, 49279
|
| 145 |
+
0, 61515
|
| 146 |
+
1, 61473
|
| 147 |
+
2, 61549
|
| 148 |
+
3, 61647
|
| 149 |
+
4, 61675
|
| 150 |
+
5, 61549
|
| 151 |
+
6, 61519
|
| 152 |
+
7, 61657
|
| 153 |
+
0, 83447
|
| 154 |
+
1, 83453
|
| 155 |
+
2, 83441
|
| 156 |
+
3, 83439
|
| 157 |
+
4, 83495
|
| 158 |
+
5, 83441
|
| 159 |
+
6, 83447
|
| 160 |
+
7, 83459
|
| 161 |
0, 83947
|
| 162 |
1, 83951
|
| 163 |
2, 83941
|
|
|
|
| 166 |
5, 83941
|
| 167 |
6, 83931
|
| 168 |
7, 83941
|
| 169 |
+
0, 84243
|
| 170 |
+
1, 84235
|
| 171 |
+
2, 84193
|
| 172 |
+
3, 84217
|
| 173 |
+
4, 84255
|
| 174 |
+
5, 84271
|
| 175 |
+
6, 84265
|
| 176 |
+
7, 84283
|
| 177 |
+
0, 90373
|
| 178 |
+
1, 90401
|
| 179 |
+
2, 91611
|
| 180 |
+
3, 90369
|
| 181 |
+
4, 91097
|
| 182 |
+
5, 90391
|
| 183 |
+
6, 90001
|
| 184 |
+
7, 90243
|
| 185 |
+
0, 95551
|
| 186 |
+
1, 95415
|
| 187 |
+
2, 95557
|
| 188 |
+
3, 95319
|
| 189 |
+
4, 95455
|
| 190 |
+
5, 95591
|
| 191 |
+
6, 95427
|
| 192 |
+
7, 95569
|
| 193 |
+
0, 102643
|
| 194 |
+
1, 102153
|
| 195 |
+
2, 103697
|
| 196 |
+
3, 102399
|
| 197 |
+
4, 102125
|
| 198 |
+
5, 102751
|
| 199 |
+
6, 105409
|
| 200 |
+
7, 104899
|
| 201 |
+
0, 116707
|
| 202 |
+
1, 116615
|
| 203 |
+
2, 116705
|
| 204 |
+
3, 116463
|
| 205 |
+
4, 116587
|
| 206 |
+
5, 116705
|
| 207 |
+
6, 116559
|
| 208 |
+
7, 116705
|
| 209 |
+
0, 116707
|
| 210 |
+
1, 116615
|
| 211 |
+
2, 116705
|
| 212 |
+
3, 116463
|
| 213 |
+
4, 116587
|
| 214 |
+
5, 116705
|
| 215 |
+
6, 116559
|
| 216 |
+
7, 116705
|
| 217 |
+
0, 115653
|
| 218 |
+
1, 115969
|
| 219 |
+
2, 115651
|
| 220 |
+
3, 115817
|
| 221 |
+
4, 115533
|
| 222 |
+
5, 116059
|
| 223 |
+
6, 115505
|
| 224 |
+
7, 116059
|
| 225 |
+
0, 16945
|
| 226 |
+
1, 16948
|
| 227 |
+
2, 16948
|
| 228 |
+
3, 16948
|
| 229 |
+
4, 17140
|
| 230 |
+
5, 17332
|
| 231 |
+
6, 115119
|
| 232 |
+
7, 18100
|
tierb-fixed-moonep-full/moonep/tierb/tierb_gbs8/QWEN3_30B__moonep__matched__seq4096__gbs8__capnone__async/rep2/run.env
CHANGED
|
@@ -3,7 +3,7 @@ config=QWEN3_30B
|
|
| 3 |
family=tierB
|
| 4 |
repo=/workspace/Megatron-MoE-Echo
|
| 5 |
branch=moonep_backend_async_grad_reduce
|
| 6 |
-
commit=
|
| 7 |
moonep_source_dir=/workspace/MoonEP
|
| 8 |
moonep_source_branch=megatron-ddp-symmetric-params
|
| 9 |
moonep_source_commit=d8bc666
|
|
|
|
| 3 |
family=tierB
|
| 4 |
repo=/workspace/Megatron-MoE-Echo
|
| 5 |
branch=moonep_backend_async_grad_reduce
|
| 6 |
+
commit=2e68c8ad
|
| 7 |
moonep_source_dir=/workspace/MoonEP
|
| 8 |
moonep_source_branch=megatron-ddp-symmetric-params
|
| 9 |
moonep_source_commit=d8bc666
|
tierb-fixed-moonep-full/moonep/tierb/tierb_gbs8/QWEN3_30B__moonep__matched__seq4096__gbs8__capnone__async/rep2/train.log
CHANGED
|
The diff for this file is too large to render.
See raw diff
|
|
|
tierb-fixed-moonep-full/moonep/tierb/tierb_gbs8/QWEN3_30B__moonep__matched__seq4096__gbs8__capnone__async/rep3/mem_samples.csv
CHANGED
|
@@ -26,7 +26,7 @@
|
|
| 26 |
1, 1
|
| 27 |
2, 1
|
| 28 |
3, 1
|
| 29 |
-
4,
|
| 30 |
5, 4
|
| 31 |
6, 4
|
| 32 |
7, 4
|
|
@@ -64,20 +64,20 @@
|
|
| 64 |
7, 4
|
| 65 |
0, 4
|
| 66 |
1, 629
|
| 67 |
-
2,
|
| 68 |
-
3,
|
| 69 |
4, 629
|
| 70 |
5, 629
|
| 71 |
6, 629
|
| 72 |
7, 629
|
| 73 |
-
0,
|
| 74 |
-
1,
|
| 75 |
-
2,
|
| 76 |
-
3,
|
| 77 |
-
4,
|
| 78 |
-
5,
|
| 79 |
-
6,
|
| 80 |
-
7,
|
| 81 |
0, 1831
|
| 82 |
1, 1831
|
| 83 |
2, 1831
|
|
@@ -86,14 +86,14 @@
|
|
| 86 |
5, 1831
|
| 87 |
6, 1831
|
| 88 |
7, 1831
|
| 89 |
-
0,
|
| 90 |
-
1,
|
| 91 |
-
2,
|
| 92 |
-
3,
|
| 93 |
-
4,
|
| 94 |
-
5,
|
| 95 |
-
6,
|
| 96 |
-
7,
|
| 97 |
0, 47207
|
| 98 |
1, 47207
|
| 99 |
2, 47207
|
|
@@ -110,54 +110,54 @@
|
|
| 110 |
5, 47551
|
| 111 |
6, 47551
|
| 112 |
7, 47551
|
| 113 |
-
0,
|
| 114 |
-
1,
|
| 115 |
-
2,
|
| 116 |
3, 47959
|
| 117 |
-
4,
|
| 118 |
-
5,
|
| 119 |
6, 47959
|
| 120 |
-
7,
|
| 121 |
0, 49103
|
| 122 |
-
1,
|
| 123 |
-
2,
|
| 124 |
3, 48759
|
| 125 |
-
4,
|
| 126 |
-
5,
|
| 127 |
-
6,
|
| 128 |
-
7,
|
| 129 |
0, 49249
|
| 130 |
1, 48907
|
| 131 |
-
2,
|
| 132 |
3, 48905
|
| 133 |
-
4,
|
| 134 |
-
5,
|
| 135 |
-
6,
|
| 136 |
-
7,
|
| 137 |
0, 49251
|
| 138 |
1, 48907
|
| 139 |
-
2,
|
| 140 |
3, 48907
|
| 141 |
-
4,
|
| 142 |
5, 49279
|
| 143 |
6, 48907
|
| 144 |
7, 49279
|
| 145 |
-
0,
|
| 146 |
-
1,
|
| 147 |
-
2,
|
| 148 |
-
3,
|
| 149 |
-
4,
|
| 150 |
-
5,
|
| 151 |
-
6,
|
| 152 |
-
7,
|
| 153 |
-
0,
|
| 154 |
-
1,
|
| 155 |
-
2,
|
| 156 |
-
3,
|
| 157 |
-
4,
|
| 158 |
-
5,
|
| 159 |
-
6,
|
| 160 |
-
7,
|
| 161 |
0, 83947
|
| 162 |
1, 83951
|
| 163 |
2, 83941
|
|
@@ -166,67 +166,67 @@
|
|
| 166 |
5, 83941
|
| 167 |
6, 83931
|
| 168 |
7, 83941
|
| 169 |
-
0,
|
| 170 |
-
1,
|
| 171 |
-
2,
|
| 172 |
-
3,
|
| 173 |
-
4,
|
| 174 |
-
5,
|
| 175 |
-
6,
|
| 176 |
-
7,
|
| 177 |
-
0,
|
| 178 |
-
1,
|
| 179 |
-
2,
|
| 180 |
-
3,
|
| 181 |
-
4,
|
| 182 |
-
5,
|
| 183 |
-
6,
|
| 184 |
-
7,
|
| 185 |
-
0,
|
| 186 |
-
1,
|
| 187 |
-
2,
|
| 188 |
-
3,
|
| 189 |
-
4,
|
| 190 |
-
5,
|
| 191 |
-
6,
|
| 192 |
-
7,
|
| 193 |
-
0,
|
| 194 |
-
1,
|
| 195 |
-
2,
|
| 196 |
-
3,
|
| 197 |
-
4,
|
| 198 |
-
5,
|
| 199 |
-
6,
|
| 200 |
-
7,
|
| 201 |
-
0,
|
| 202 |
-
1,
|
| 203 |
-
2,
|
| 204 |
-
3,
|
| 205 |
-
4,
|
| 206 |
-
5,
|
| 207 |
-
6,
|
| 208 |
-
7,
|
| 209 |
-
0,
|
| 210 |
-
1,
|
| 211 |
-
2,
|
| 212 |
-
3,
|
| 213 |
-
4,
|
| 214 |
-
5,
|
| 215 |
-
6,
|
| 216 |
-
7,
|
| 217 |
-
0,
|
| 218 |
-
1,
|
| 219 |
-
2,
|
| 220 |
-
3,
|
| 221 |
-
4,
|
| 222 |
-
5,
|
| 223 |
-
6,
|
| 224 |
-
7,
|
| 225 |
-
0,
|
| 226 |
-
1,
|
| 227 |
-
2,
|
| 228 |
-
3,
|
| 229 |
-
4,
|
| 230 |
-
5,
|
| 231 |
-
6,
|
| 232 |
-
7,
|
|
|
|
| 26 |
1, 1
|
| 27 |
2, 1
|
| 28 |
3, 1
|
| 29 |
+
4, 1
|
| 30 |
5, 4
|
| 31 |
6, 4
|
| 32 |
7, 4
|
|
|
|
| 64 |
7, 4
|
| 65 |
0, 4
|
| 66 |
1, 629
|
| 67 |
+
2, 45
|
| 68 |
+
3, 35
|
| 69 |
4, 629
|
| 70 |
5, 629
|
| 71 |
6, 629
|
| 72 |
7, 629
|
| 73 |
+
0, 875
|
| 74 |
+
1, 875
|
| 75 |
+
2, 877
|
| 76 |
+
3, 893
|
| 77 |
+
4, 903
|
| 78 |
+
5, 905
|
| 79 |
+
6, 899
|
| 80 |
+
7, 909
|
| 81 |
0, 1831
|
| 82 |
1, 1831
|
| 83 |
2, 1831
|
|
|
|
| 86 |
5, 1831
|
| 87 |
6, 1831
|
| 88 |
7, 1831
|
| 89 |
+
0, 16477
|
| 90 |
+
1, 16833
|
| 91 |
+
2, 17345
|
| 92 |
+
3, 16517
|
| 93 |
+
4, 17209
|
| 94 |
+
5, 16913
|
| 95 |
+
6, 16773
|
| 96 |
+
7, 16577
|
| 97 |
0, 47207
|
| 98 |
1, 47207
|
| 99 |
2, 47207
|
|
|
|
| 110 |
5, 47551
|
| 111 |
6, 47551
|
| 112 |
7, 47551
|
| 113 |
+
0, 48207
|
| 114 |
+
1, 47959
|
| 115 |
+
2, 48235
|
| 116 |
3, 47959
|
| 117 |
+
4, 47999
|
| 118 |
+
5, 48235
|
| 119 |
6, 47959
|
| 120 |
+
7, 48331
|
| 121 |
0, 49103
|
| 122 |
+
1, 48759
|
| 123 |
+
2, 49133
|
| 124 |
3, 48759
|
| 125 |
+
4, 48895
|
| 126 |
+
5, 49131
|
| 127 |
+
6, 48759
|
| 128 |
+
7, 49133
|
| 129 |
0, 49249
|
| 130 |
1, 48907
|
| 131 |
+
2, 49279
|
| 132 |
3, 48905
|
| 133 |
+
4, 49041
|
| 134 |
+
5, 49277
|
| 135 |
+
6, 48905
|
| 136 |
+
7, 49279
|
| 137 |
0, 49251
|
| 138 |
1, 48907
|
| 139 |
+
2, 49279
|
| 140 |
3, 48907
|
| 141 |
+
4, 49041
|
| 142 |
5, 49279
|
| 143 |
6, 48907
|
| 144 |
7, 49279
|
| 145 |
+
0, 66095
|
| 146 |
+
1, 66077
|
| 147 |
+
2, 65899
|
| 148 |
+
3, 65887
|
| 149 |
+
4, 65957
|
| 150 |
+
5, 65919
|
| 151 |
+
6, 65909
|
| 152 |
+
7, 65899
|
| 153 |
+
0, 83463
|
| 154 |
+
1, 83509
|
| 155 |
+
2, 83483
|
| 156 |
+
3, 83467
|
| 157 |
+
4, 83535
|
| 158 |
+
5, 83455
|
| 159 |
+
6, 83487
|
| 160 |
+
7, 83515
|
| 161 |
0, 83947
|
| 162 |
1, 83951
|
| 163 |
2, 83941
|
|
|
|
| 166 |
5, 83941
|
| 167 |
6, 83931
|
| 168 |
7, 83941
|
| 169 |
+
0, 84249
|
| 170 |
+
1, 84237
|
| 171 |
+
2, 84221
|
| 172 |
+
3, 84217
|
| 173 |
+
4, 84273
|
| 174 |
+
5, 84265
|
| 175 |
+
6, 84201
|
| 176 |
+
7, 84257
|
| 177 |
+
0, 89637
|
| 178 |
+
1, 89977
|
| 179 |
+
2, 90223
|
| 180 |
+
3, 90153
|
| 181 |
+
4, 89697
|
| 182 |
+
5, 90063
|
| 183 |
+
6, 89849
|
| 184 |
+
7, 89919
|
| 185 |
+
0, 95507
|
| 186 |
+
1, 95411
|
| 187 |
+
2, 95461
|
| 188 |
+
3, 95249
|
| 189 |
+
4, 95409
|
| 190 |
+
5, 95503
|
| 191 |
+
6, 95353
|
| 192 |
+
7, 95529
|
| 193 |
+
0, 107557
|
| 194 |
+
1, 108155
|
| 195 |
+
2, 111849
|
| 196 |
+
3, 106913
|
| 197 |
+
4, 107327
|
| 198 |
+
5, 107955
|
| 199 |
+
6, 114103
|
| 200 |
+
7, 107845
|
| 201 |
+
0, 116707
|
| 202 |
+
1, 116615
|
| 203 |
+
2, 116705
|
| 204 |
+
3, 116463
|
| 205 |
+
4, 116587
|
| 206 |
+
5, 116705
|
| 207 |
+
6, 116559
|
| 208 |
+
7, 116705
|
| 209 |
+
0, 116707
|
| 210 |
+
1, 116615
|
| 211 |
+
2, 116705
|
| 212 |
+
3, 116463
|
| 213 |
+
4, 116587
|
| 214 |
+
5, 116705
|
| 215 |
+
6, 116559
|
| 216 |
+
7, 116705
|
| 217 |
+
0, 116061
|
| 218 |
+
1, 115969
|
| 219 |
+
2, 116059
|
| 220 |
+
3, 115817
|
| 221 |
+
4, 115941
|
| 222 |
+
5, 116059
|
| 223 |
+
6, 115913
|
| 224 |
+
7, 116059
|
| 225 |
+
0, 20737
|
| 226 |
+
1, 20740
|
| 227 |
+
2, 20740
|
| 228 |
+
3, 20740
|
| 229 |
+
4, 20996
|
| 230 |
+
5, 116057
|
| 231 |
+
6, 22020
|
| 232 |
+
7, 20740
|
tierb-fixed-moonep-full/moonep/tierb/tierb_gbs8/QWEN3_30B__moonep__matched__seq4096__gbs8__capnone__async/rep3/run.env
CHANGED
|
@@ -3,7 +3,7 @@ config=QWEN3_30B
|
|
| 3 |
family=tierB
|
| 4 |
repo=/workspace/Megatron-MoE-Echo
|
| 5 |
branch=moonep_backend_async_grad_reduce
|
| 6 |
-
commit=
|
| 7 |
moonep_source_dir=/workspace/MoonEP
|
| 8 |
moonep_source_branch=megatron-ddp-symmetric-params
|
| 9 |
moonep_source_commit=d8bc666
|
|
|
|
| 3 |
family=tierB
|
| 4 |
repo=/workspace/Megatron-MoE-Echo
|
| 5 |
branch=moonep_backend_async_grad_reduce
|
| 6 |
+
commit=2e68c8ad
|
| 7 |
moonep_source_dir=/workspace/MoonEP
|
| 8 |
moonep_source_branch=megatron-ddp-symmetric-params
|
| 9 |
moonep_source_commit=d8bc666
|
tierb-fixed-moonep-full/moonep/tierb/tierb_gbs8/QWEN3_30B__moonep__matched__seq4096__gbs8__capnone__async/rep3/train.log
CHANGED
|
The diff for this file is too large to render.
See raw diff
|
|
|
tierb-fixed-moonep-full/moonep/tierb/tierb_gbs8/results.tsv
CHANGED
|
@@ -1,13 +1,7 @@
|
|
| 1 |
config backend policy slots seq gbs cap recompute rep status iter_ms_mean iter_ms_min peak_mib run_dir
|
| 2 |
-
QWEN3_235B moonep matched 16 8192 8 none none 1 OK
|
| 3 |
-
QWEN3_235B moonep matched 16 8192 8 none none 2 OK
|
| 4 |
-
QWEN3_235B moonep matched 16 8192 8 none none 3 OK
|
| 5 |
-
QWEN3_30B moonep matched 16 4096 8 none none 1 OK
|
| 6 |
-
QWEN3_30B moonep matched 16 4096 8 none none 2 OK
|
| 7 |
-
QWEN3_30B moonep matched 16 4096 8 none none 3 OK
|
| 8 |
-
DSV2_LITE_L24 moonep matched 8 8192 8 none none 1 OK 596.03 595.70 133311 /workspace/tierb-fixed-moonep-full/moonep/tierb/tierb_gbs8/DSV2_LITE_L24__moonep__matched__seq8192__gbs8__capnone__async/rep1
|
| 9 |
-
DSV2_LITE_L24 moonep matched 8 8192 8 none none 2 OK 596.18 594.10 133311 /workspace/tierb-fixed-moonep-full/moonep/tierb/tierb_gbs8/DSV2_LITE_L24__moonep__matched__seq8192__gbs8__capnone__async/rep2
|
| 10 |
-
DSV2_LITE_L24 moonep matched 8 8192 8 none none 3 OK 595.97 594.00 133311 /workspace/tierb-fixed-moonep-full/moonep/tierb/tierb_gbs8/DSV2_LITE_L24__moonep__matched__seq8192__gbs8__capnone__async/rep3
|
| 11 |
-
QWEN2_57B moonep matched 8 4096 8 none none 1 OK 183.70 182.60 96737 /workspace/tierb-fixed-moonep-full/moonep/tierb/tierb_gbs8/QWEN2_57B__moonep__matched__seq4096__gbs8__capnone__async/rep1
|
| 12 |
-
QWEN2_57B moonep matched 8 4096 8 none none 2 OK 184.22 182.60 96737 /workspace/tierb-fixed-moonep-full/moonep/tierb/tierb_gbs8/QWEN2_57B__moonep__matched__seq4096__gbs8__capnone__async/rep2
|
| 13 |
-
QWEN2_57B moonep matched 8 4096 8 none none 3 OK 184.20 182.60 96737 /workspace/tierb-fixed-moonep-full/moonep/tierb/tierb_gbs8/QWEN2_57B__moonep__matched__seq4096__gbs8__capnone__async/rep3
|
|
|
|
| 1 |
config backend policy slots seq gbs cap recompute rep status iter_ms_mean iter_ms_min peak_mib run_dir
|
| 2 |
+
QWEN3_235B moonep matched 16 8192 8 none none 1 OK 317.23 316.40 120309 /workspace/tierb-fixed-moonep-full/moonep/tierb/tierb_gbs8/QWEN3_235B__moonep__matched__seq8192__gbs8__capnone__async/rep1
|
| 3 |
+
QWEN3_235B moonep matched 16 8192 8 none none 2 OK 317.53 316.40 120309 /workspace/tierb-fixed-moonep-full/moonep/tierb/tierb_gbs8/QWEN3_235B__moonep__matched__seq8192__gbs8__capnone__async/rep2
|
| 4 |
+
QWEN3_235B moonep matched 16 8192 8 none none 3 OK 317.80 316.40 120309 /workspace/tierb-fixed-moonep-full/moonep/tierb/tierb_gbs8/QWEN3_235B__moonep__matched__seq8192__gbs8__capnone__async/rep3
|
| 5 |
+
QWEN3_30B moonep matched 16 4096 8 none none 1 OK 354.41 352.30 116707 /workspace/tierb-fixed-moonep-full/moonep/tierb/tierb_gbs8/QWEN3_30B__moonep__matched__seq4096__gbs8__capnone__async/rep1
|
| 6 |
+
QWEN3_30B moonep matched 16 4096 8 none none 2 OK 354.46 352.20 116707 /workspace/tierb-fixed-moonep-full/moonep/tierb/tierb_gbs8/QWEN3_30B__moonep__matched__seq4096__gbs8__capnone__async/rep2
|
| 7 |
+
QWEN3_30B moonep matched 16 4096 8 none none 3 OK 355.16 352.20 116707 /workspace/tierb-fixed-moonep-full/moonep/tierb/tierb_gbs8/QWEN3_30B__moonep__matched__seq4096__gbs8__capnone__async/rep3
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|