granularity results tiera-rerun-fixed-moonep/ultraep/ponly @ 2026-08-30T07:22:02Z from sp-aiinfra-interactive-aghosh-7jtjl-8xb200-moe-cg-0
Browse files- tiera-rerun-fixed-moonep/ultraep/ponly/ponly_gbs16/PONLY__ultraep__one__seq8192__gbs16__cap2p40625/rep2/mem_samples.csv +32 -0
- tiera-rerun-fixed-moonep/ultraep/ponly/ponly_gbs16/PONLY__ultraep__one__seq8192__gbs16__cap2p40625/rep2/train.log +48 -0
- tiera-rerun-fixed-moonep/ultraep/ponly/ponly_gbs16/PONLY__ultraep__one__seq8192__gbs16__cap2p40625/rep3/mem_samples.csv +200 -0
- tiera-rerun-fixed-moonep/ultraep/ponly/ponly_gbs16/PONLY__ultraep__one__seq8192__gbs16__cap2p40625/rep3/run.env +70 -0
- tiera-rerun-fixed-moonep/ultraep/ponly/ponly_gbs16/PONLY__ultraep__one__seq8192__gbs16__cap2p40625/rep3/train.log +1025 -0
- tiera-rerun-fixed-moonep/ultraep/ponly/ponly_gbs16/results.tsv +2 -0
tiera-rerun-fixed-moonep/ultraep/ponly/ponly_gbs16/PONLY__ultraep__one__seq8192__gbs16__cap2p40625/rep2/mem_samples.csv
CHANGED
|
@@ -166,3 +166,35 @@
|
|
| 166 |
5, 39689
|
| 167 |
6, 39641
|
| 168 |
7, 39707
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 166 |
5, 39689
|
| 167 |
6, 39641
|
| 168 |
7, 39707
|
| 169 |
+
0, 51015
|
| 170 |
+
1, 51011
|
| 171 |
+
2, 51009
|
| 172 |
+
3, 51009
|
| 173 |
+
4, 50975
|
| 174 |
+
5, 51011
|
| 175 |
+
6, 51009
|
| 176 |
+
7, 51009
|
| 177 |
+
0, 51015
|
| 178 |
+
1, 51011
|
| 179 |
+
2, 51009
|
| 180 |
+
3, 51009
|
| 181 |
+
4, 50975
|
| 182 |
+
5, 51011
|
| 183 |
+
6, 51009
|
| 184 |
+
7, 51009
|
| 185 |
+
0, 49845
|
| 186 |
+
1, 49841
|
| 187 |
+
2, 49839
|
| 188 |
+
3, 49839
|
| 189 |
+
4, 49805
|
| 190 |
+
5, 49841
|
| 191 |
+
6, 49839
|
| 192 |
+
7, 49839
|
| 193 |
+
0, 1537
|
| 194 |
+
1, 1539
|
| 195 |
+
2, 1540
|
| 196 |
+
3, 1540
|
| 197 |
+
4, 1796
|
| 198 |
+
5, 49841
|
| 199 |
+
6, 2820
|
| 200 |
+
7, 1540
|
tiera-rerun-fixed-moonep/ultraep/ponly/ponly_gbs16/PONLY__ultraep__one__seq8192__gbs16__cap2p40625/rep2/train.log
CHANGED
|
@@ -975,3 +975,51 @@ Theoretical memory footprints: weight and optimizer=25029.43 MB
|
|
| 975 |
[2026-08-30 07:20:47] iteration 3/ 12 | consumed samples: 48 | elapsed time per iteration (ms): 329.7 | throughput per GPU (TFLOP/s/GPU): 601.7 | learning rate: 6.000000E-05 | global batch size: 16 | lm loss: 1.053758E+01 | loss scale: 1.0 | grad norm: 249.877 | number of skipped iterations: 0 | number of nan iterations: 0 | peak recv tokens: 28416 | min safe capacity: 1.7344 |
|
| 976 |
Capture CUDA graph for training!!!
|
| 977 |
CUDA graph capture done!!!
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 975 |
[2026-08-30 07:20:47] iteration 3/ 12 | consumed samples: 48 | elapsed time per iteration (ms): 329.7 | throughput per GPU (TFLOP/s/GPU): 601.7 | learning rate: 6.000000E-05 | global batch size: 16 | lm loss: 1.053758E+01 | loss scale: 1.0 | grad norm: 249.877 | number of skipped iterations: 0 | number of nan iterations: 0 | peak recv tokens: 28416 | min safe capacity: 1.7344 |
|
| 976 |
Capture CUDA graph for training!!!
|
| 977 |
CUDA graph capture done!!!
|
| 978 |
+
[2026-08-30 07:20:48] iteration 4/ 12 | consumed samples: 64 | elapsed time per iteration (ms): 1110.8 | throughput per GPU (TFLOP/s/GPU): 178.6 | learning rate: 8.000000E-05 | global batch size: 16 | lm loss: 1.319968E+01 | loss scale: 1.0 | grad norm: 35.719 | number of skipped iterations: 0 | number of nan iterations: 0 | peak recv tokens: 28416 | min safe capacity: 1.7344 |
|
| 979 |
+
[2026-08-30 07:20:48] iteration 5/ 12 | consumed samples: 80 | elapsed time per iteration (ms): 292.8 | throughput per GPU (TFLOP/s/GPU): 677.6 | learning rate: 1.000000E-04 | global batch size: 16 | lm loss: 1.052109E+01 | loss scale: 1.0 | grad norm: 17.525 | number of skipped iterations: 0 | number of nan iterations: 0 | peak recv tokens: 28416 | min safe capacity: 1.7344 |
|
| 980 |
+
[2026-08-30 07:20:48] iteration 6/ 12 | consumed samples: 96 | elapsed time per iteration (ms): 293.5 | throughput per GPU (TFLOP/s/GPU): 676.0 | learning rate: 7.750000E-05 | global batch size: 16 | lm loss: 1.027258E+01 | loss scale: 1.0 | grad norm: 9.627 | number of skipped iterations: 0 | number of nan iterations: 0 | peak recv tokens: 28416 | min safe capacity: 1.7344 |
|
| 981 |
+
[2026-08-30 07:20:49] iteration 7/ 12 | consumed samples: 112 | elapsed time per iteration (ms): 291.4 | throughput per GPU (TFLOP/s/GPU): 680.9 | learning rate: 3.250000E-05 | global batch size: 16 | lm loss: 9.465188E+00 | loss scale: 1.0 | grad norm: 7.772 | number of skipped iterations: 0 | number of nan iterations: 0 | peak recv tokens: 28416 | min safe capacity: 1.7344 |
|
| 982 |
+
[2026-08-30 07:20:49] iteration 8/ 12 | consumed samples: 128 | elapsed time per iteration (ms): 291.4 | throughput per GPU (TFLOP/s/GPU): 680.7 | learning rate: 1.000000E-05 | global batch size: 16 | lm loss: 8.756442E+00 | loss scale: 1.0 | grad norm: 6.065 | number of skipped iterations: 0 | number of nan iterations: 0 | peak recv tokens: 28416 | min safe capacity: 1.7344 |
|
| 983 |
+
[2026-08-30 07:20:49] iteration 9/ 12 | consumed samples: 144 | elapsed time per iteration (ms): 292.7 | throughput per GPU (TFLOP/s/GPU): 677.7 | learning rate: 1.000000E-05 | global batch size: 16 | lm loss: 8.545505E+00 | loss scale: 1.0 | grad norm: 7.211 | number of skipped iterations: 0 | number of nan iterations: 0 | peak recv tokens: 28416 | min safe capacity: 1.7344 |
|
| 984 |
+
[2026-08-30 07:20:50] iteration 10/ 12 | consumed samples: 160 | elapsed time per iteration (ms): 291.3 | throughput per GPU (TFLOP/s/GPU): 681.0 | learning rate: 1.000000E-05 | global batch size: 16 | lm loss: 8.448534E+00 | loss scale: 1.0 | grad norm: 6.020 | number of skipped iterations: 0 | number of nan iterations: 0 | peak recv tokens: 28416 | min safe capacity: 1.7344 |
|
| 985 |
+
[2026-08-30 07:20:50] iteration 11/ 12 | consumed samples: 176 | elapsed time per iteration (ms): 292.8 | throughput per GPU (TFLOP/s/GPU): 677.4 | learning rate: 1.000000E-05 | global batch size: 16 | lm loss: 8.375360E+00 | loss scale: 1.0 | grad norm: 4.678 | number of skipped iterations: 0 | number of nan iterations: 0 | peak recv tokens: 28416 | min safe capacity: 1.7344 |
|
| 986 |
+
[2026-08-30 07:20:50] iteration 12/ 12 | consumed samples: 192 | elapsed time per iteration (ms): 294.2 | throughput per GPU (TFLOP/s/GPU): 674.3 | learning rate: 1.000000E-05 | global batch size: 16 | lm loss: 8.276701E+00 | loss scale: 1.0 | grad norm: 4.158 | number of skipped iterations: 0 | number of nan iterations: 0 | peak recv tokens: 28416 | min safe capacity: 1.7344 |
|
| 987 |
+
[after training is done] datetime: 2026-08-30 07:20:50
|
| 988 |
+
[rank0]:[W830 07:20:51.917748913 ProcessGroupNCCL.cpp:1505] Warning: WARNING: destroy_process_group() was not called before program exit, which can leak resources. For more info, please see https://pytorch.org/docs/stable/distributed.html#shutdown (function operator())
|
| 989 |
+
W0830 07:20:52.491000 741931 torch/distributed/elastic/agent/server/api.py:719] Received 15 death signal, shutting down workers
|
| 990 |
+
W0830 07:20:52.491000 741931 torch/distributed/elastic/multiprocessing/api.py:900] Sending process 742021 closing signal SIGTERM
|
| 991 |
+
W0830 07:20:52.492000 741931 torch/distributed/elastic/multiprocessing/api.py:900] Sending process 742022 closing signal SIGTERM
|
| 992 |
+
W0830 07:20:52.493000 741931 torch/distributed/elastic/multiprocessing/api.py:900] Sending process 742023 closing signal SIGTERM
|
| 993 |
+
W0830 07:20:52.495000 741931 torch/distributed/elastic/multiprocessing/api.py:900] Sending process 742024 closing signal SIGTERM
|
| 994 |
+
W0830 07:20:52.497000 741931 torch/distributed/elastic/multiprocessing/api.py:900] Sending process 742025 closing signal SIGTERM
|
| 995 |
+
W0830 07:20:52.499000 741931 torch/distributed/elastic/multiprocessing/api.py:900] Sending process 742026 closing signal SIGTERM
|
| 996 |
+
W0830 07:20:52.501000 741931 torch/distributed/elastic/multiprocessing/api.py:900] Sending process 742027 closing signal SIGTERM
|
| 997 |
+
W0830 07:20:52.502000 741931 torch/distributed/elastic/multiprocessing/api.py:900] Sending process 742028 closing signal SIGTERM
|
| 998 |
+
Traceback (most recent call last):
|
| 999 |
+
File "/usr/local/bin/torchrun", line 8, in <module>
|
| 1000 |
+
sys.exit(main())
|
| 1001 |
+
^^^^^^
|
| 1002 |
+
File "/usr/local/lib/python3.12/dist-packages/torch/distributed/elastic/multiprocessing/errors/__init__.py", line 355, in wrapper
|
| 1003 |
+
return f(*args, **kwargs)
|
| 1004 |
+
^^^^^^^^^^^^^^^^^^
|
| 1005 |
+
File "/usr/local/lib/python3.12/dist-packages/torch/distributed/run.py", line 892, in main
|
| 1006 |
+
run(args)
|
| 1007 |
+
File "/usr/local/lib/python3.12/dist-packages/torch/distributed/run.py", line 883, in run
|
| 1008 |
+
elastic_launch(
|
| 1009 |
+
File "/usr/local/lib/python3.12/dist-packages/torch/distributed/launcher/api.py", line 139, in __call__
|
| 1010 |
+
return launch_agent(self._config, self._entrypoint, list(args))
|
| 1011 |
+
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
| 1012 |
+
File "/usr/local/lib/python3.12/dist-packages/torch/distributed/launcher/api.py", line 261, in launch_agent
|
| 1013 |
+
result = agent.run()
|
| 1014 |
+
^^^^^^^^^^^
|
| 1015 |
+
File "/usr/local/lib/python3.12/dist-packages/torch/distributed/elastic/metrics/api.py", line 138, in wrapper
|
| 1016 |
+
result = f(*args, **kwargs)
|
| 1017 |
+
^^^^^^^^^^^^^^^^^^
|
| 1018 |
+
File "/usr/local/lib/python3.12/dist-packages/torch/distributed/elastic/agent/server/api.py", line 711, in run
|
| 1019 |
+
result = self._invoke_run(role)
|
| 1020 |
+
^^^^^^^^^^^^^^^^^^^^^^
|
| 1021 |
+
File "/usr/local/lib/python3.12/dist-packages/torch/distributed/elastic/agent/server/api.py", line 870, in _invoke_run
|
| 1022 |
+
time.sleep(monitor_interval)
|
| 1023 |
+
File "/usr/local/lib/python3.12/dist-packages/torch/distributed/elastic/multiprocessing/api.py", line 84, in _terminate_process_handler
|
| 1024 |
+
raise SignalException(f"Process {os.getpid()} got signal: {sigval}", sigval=sigval)
|
| 1025 |
+
torch.distributed.elastic.multiprocessing.api.SignalException: Process 741931 got signal: 15
|
tiera-rerun-fixed-moonep/ultraep/ponly/ponly_gbs16/PONLY__ultraep__one__seq8192__gbs16__cap2p40625/rep3/mem_samples.csv
ADDED
|
@@ -0,0 +1,200 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
0, 0
|
| 2 |
+
1, 0
|
| 3 |
+
2, 0
|
| 4 |
+
3, 0
|
| 5 |
+
4, 0
|
| 6 |
+
5, 0
|
| 7 |
+
6, 0
|
| 8 |
+
7, 0
|
| 9 |
+
0, 0
|
| 10 |
+
1, 0
|
| 11 |
+
2, 0
|
| 12 |
+
3, 0
|
| 13 |
+
4, 0
|
| 14 |
+
5, 0
|
| 15 |
+
6, 0
|
| 16 |
+
7, 0
|
| 17 |
+
0, 4
|
| 18 |
+
1, 1
|
| 19 |
+
2, 1
|
| 20 |
+
3, 1
|
| 21 |
+
4, 1
|
| 22 |
+
5, 1
|
| 23 |
+
6, 1
|
| 24 |
+
7, 1
|
| 25 |
+
0, 4
|
| 26 |
+
1, 4
|
| 27 |
+
2, 4
|
| 28 |
+
3, 4
|
| 29 |
+
4, 2
|
| 30 |
+
5, 1
|
| 31 |
+
6, 1
|
| 32 |
+
7, 1
|
| 33 |
+
0, 4
|
| 34 |
+
1, 4
|
| 35 |
+
2, 4
|
| 36 |
+
3, 4
|
| 37 |
+
4, 4
|
| 38 |
+
5, 4
|
| 39 |
+
6, 4
|
| 40 |
+
7, 4
|
| 41 |
+
0, 4
|
| 42 |
+
1, 27
|
| 43 |
+
2, 4
|
| 44 |
+
3, 4
|
| 45 |
+
4, 4
|
| 46 |
+
5, 4
|
| 47 |
+
6, 4
|
| 48 |
+
7, 4
|
| 49 |
+
0, 821
|
| 50 |
+
1, 821
|
| 51 |
+
2, 821
|
| 52 |
+
3, 821
|
| 53 |
+
4, 821
|
| 54 |
+
5, 821
|
| 55 |
+
6, 821
|
| 56 |
+
7, 821
|
| 57 |
+
0, 2007
|
| 58 |
+
1, 2007
|
| 59 |
+
2, 2007
|
| 60 |
+
3, 2007
|
| 61 |
+
4, 2007
|
| 62 |
+
5, 2007
|
| 63 |
+
6, 2007
|
| 64 |
+
7, 2007
|
| 65 |
+
0, 2263
|
| 66 |
+
1, 2267
|
| 67 |
+
2, 2273
|
| 68 |
+
3, 2269
|
| 69 |
+
4, 2271
|
| 70 |
+
5, 2275
|
| 71 |
+
6, 2271
|
| 72 |
+
7, 2271
|
| 73 |
+
0, 14589
|
| 74 |
+
1, 14587
|
| 75 |
+
2, 14587
|
| 76 |
+
3, 14587
|
| 77 |
+
4, 14587
|
| 78 |
+
5, 14587
|
| 79 |
+
6, 14587
|
| 80 |
+
7, 14587
|
| 81 |
+
0, 15123
|
| 82 |
+
1, 15125
|
| 83 |
+
2, 15123
|
| 84 |
+
3, 15121
|
| 85 |
+
4, 15121
|
| 86 |
+
5, 15121
|
| 87 |
+
6, 15121
|
| 88 |
+
7, 15125
|
| 89 |
+
0, 16161
|
| 90 |
+
1, 16159
|
| 91 |
+
2, 16159
|
| 92 |
+
3, 16159
|
| 93 |
+
4, 16159
|
| 94 |
+
5, 16159
|
| 95 |
+
6, 16159
|
| 96 |
+
7, 16159
|
| 97 |
+
0, 16685
|
| 98 |
+
1, 16683
|
| 99 |
+
2, 16683
|
| 100 |
+
3, 16683
|
| 101 |
+
4, 16683
|
| 102 |
+
5, 16683
|
| 103 |
+
6, 16683
|
| 104 |
+
7, 16683
|
| 105 |
+
0, 16993
|
| 106 |
+
1, 16991
|
| 107 |
+
2, 16991
|
| 108 |
+
3, 16991
|
| 109 |
+
4, 16991
|
| 110 |
+
5, 16991
|
| 111 |
+
6, 16991
|
| 112 |
+
7, 16991
|
| 113 |
+
0, 17959
|
| 114 |
+
1, 17957
|
| 115 |
+
2, 17957
|
| 116 |
+
3, 17957
|
| 117 |
+
4, 17957
|
| 118 |
+
5, 17957
|
| 119 |
+
6, 17957
|
| 120 |
+
7, 17957
|
| 121 |
+
0, 35699
|
| 122 |
+
1, 35697
|
| 123 |
+
2, 35697
|
| 124 |
+
3, 35697
|
| 125 |
+
4, 35659
|
| 126 |
+
5, 35697
|
| 127 |
+
6, 35697
|
| 128 |
+
7, 35697
|
| 129 |
+
0, 35699
|
| 130 |
+
1, 35697
|
| 131 |
+
2, 35697
|
| 132 |
+
3, 35697
|
| 133 |
+
4, 35659
|
| 134 |
+
5, 35697
|
| 135 |
+
6, 35697
|
| 136 |
+
7, 35697
|
| 137 |
+
0, 35699
|
| 138 |
+
1, 35697
|
| 139 |
+
2, 35697
|
| 140 |
+
3, 35697
|
| 141 |
+
4, 35659
|
| 142 |
+
5, 35697
|
| 143 |
+
6, 35697
|
| 144 |
+
7, 35697
|
| 145 |
+
0, 37513
|
| 146 |
+
1, 37519
|
| 147 |
+
2, 37525
|
| 148 |
+
3, 37511
|
| 149 |
+
4, 37471
|
| 150 |
+
5, 37515
|
| 151 |
+
6, 37545
|
| 152 |
+
7, 37519
|
| 153 |
+
0, 38577
|
| 154 |
+
1, 38575
|
| 155 |
+
2, 38555
|
| 156 |
+
3, 38571
|
| 157 |
+
4, 38531
|
| 158 |
+
5, 38555
|
| 159 |
+
6, 38583
|
| 160 |
+
7, 38571
|
| 161 |
+
0, 39681
|
| 162 |
+
1, 39659
|
| 163 |
+
2, 39683
|
| 164 |
+
3, 39663
|
| 165 |
+
4, 39615
|
| 166 |
+
5, 39707
|
| 167 |
+
6, 39663
|
| 168 |
+
7, 39747
|
| 169 |
+
0, 51013
|
| 170 |
+
1, 51009
|
| 171 |
+
2, 51011
|
| 172 |
+
3, 51009
|
| 173 |
+
4, 50975
|
| 174 |
+
5, 51009
|
| 175 |
+
6, 51011
|
| 176 |
+
7, 51009
|
| 177 |
+
0, 51013
|
| 178 |
+
1, 51009
|
| 179 |
+
2, 51011
|
| 180 |
+
3, 51009
|
| 181 |
+
4, 50975
|
| 182 |
+
5, 51009
|
| 183 |
+
6, 51011
|
| 184 |
+
7, 51009
|
| 185 |
+
0, 1540
|
| 186 |
+
1, 1540
|
| 187 |
+
2, 1540
|
| 188 |
+
3, 1540
|
| 189 |
+
4, 1540
|
| 190 |
+
5, 1860
|
| 191 |
+
6, 50487
|
| 192 |
+
7, 3140
|
| 193 |
+
0, 0
|
| 194 |
+
1, 0
|
| 195 |
+
2, 0
|
| 196 |
+
3, 0
|
| 197 |
+
4, 0
|
| 198 |
+
5, 0
|
| 199 |
+
6, 0
|
| 200 |
+
7, 0
|
tiera-rerun-fixed-moonep/ultraep/ponly/ponly_gbs16/PONLY__ultraep__one__seq8192__gbs16__cap2p40625/rep3/run.env
ADDED
|
@@ -0,0 +1,70 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
backend=ultraep
|
| 2 |
+
config=PONLY
|
| 3 |
+
family=tierA
|
| 4 |
+
repo=/workspace/Megatron-UltraEP
|
| 5 |
+
branch=ultraep-bracing-regions-through-kernels
|
| 6 |
+
commit=03d57ef7
|
| 7 |
+
num_layers=16
|
| 8 |
+
hidden=4096
|
| 9 |
+
ffn_hidden=14336
|
| 10 |
+
moe_ffn_hidden=1536
|
| 11 |
+
num_experts=8
|
| 12 |
+
topk=2
|
| 13 |
+
experts_per_rank=1
|
| 14 |
+
expert_params=18874368
|
| 15 |
+
heads=32
|
| 16 |
+
kv_groups=8
|
| 17 |
+
kv_channels=
|
| 18 |
+
qk_layernorm=0
|
| 19 |
+
shared_ffn=0
|
| 20 |
+
shared_expert_overlap=0
|
| 21 |
+
moe_token_dispatcher=flex
|
| 22 |
+
mla=0
|
| 23 |
+
dense_first_layers=0
|
| 24 |
+
moe_layers=16
|
| 25 |
+
rotary_base=1000000
|
| 26 |
+
rotary_scaling=
|
| 27 |
+
qkv_bias=0
|
| 28 |
+
router_balancing=aux_loss
|
| 29 |
+
aux_loss_coeff=1e-2
|
| 30 |
+
router_pre_softmax=0
|
| 31 |
+
router_fusion=0
|
| 32 |
+
router_dtype=fp32
|
| 33 |
+
device_initiated_grouped_gemm=1
|
| 34 |
+
norm_epsilon=1e-5
|
| 35 |
+
seq_length=8192
|
| 36 |
+
micro_batch_size=1
|
| 37 |
+
global_batch_size=16
|
| 38 |
+
num_microbatches=2
|
| 39 |
+
assignments_per_rank=16384
|
| 40 |
+
mean_tokens_per_expert=16384
|
| 41 |
+
slot_policy=one
|
| 42 |
+
slots_per_rank=1
|
| 43 |
+
capacity_mode=static
|
| 44 |
+
capacity=2.40625
|
| 45 |
+
cuda_graph=1
|
| 46 |
+
probe_mode=0
|
| 47 |
+
async_grad_reduce=0
|
| 48 |
+
branch_role=timing
|
| 49 |
+
memory_dump_mode=none
|
| 50 |
+
memory_snapshot_path=
|
| 51 |
+
cuda_graph_warmup_steps=3
|
| 52 |
+
recompute=none
|
| 53 |
+
force_balance=0
|
| 54 |
+
train_iters=12
|
| 55 |
+
ep_size=8
|
| 56 |
+
echo_dispatch_overlap=1
|
| 57 |
+
echo_dispatch_recompute=0
|
| 58 |
+
ultraep_quota_min_tokens=2048
|
| 59 |
+
ultraep_weight_sync_join=
|
| 60 |
+
profile_enable=0
|
| 61 |
+
kernel_trace=0
|
| 62 |
+
fp8_mode=mxfp8
|
| 63 |
+
fp8_recipe=mxfp8
|
| 64 |
+
training_seed=1234
|
| 65 |
+
learning_rate=1e-4
|
| 66 |
+
min_learning_rate=1.0e-5
|
| 67 |
+
lr_warmup_iters=5
|
| 68 |
+
state_dump_enable=0
|
| 69 |
+
state_dump_dir=
|
| 70 |
+
moonep_assert_no_pending_reduce=0
|
tiera-rerun-fixed-moonep/ultraep/ponly/ponly_gbs16/PONLY__ultraep__one__seq8192__gbs16__cap2p40625/rep3/train.log
ADDED
|
@@ -0,0 +1,1025 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
==============================================================
|
| 2 |
+
PONLY / ultraep (slots/rank=1 policy=one)
|
| 3 |
+
E=8 K=2 H'=1536 m=1
|
| 4 |
+
L=16 seq=8192 gbs=16 (2 microbatches)
|
| 5 |
+
cap_mode=static cap=2.40625 cg=1 recompute=none
|
| 6 |
+
memory_dump=none snapshots=/workspace/tiera-rerun-fixed-moonep/ultraep/ponly/ponly_gbs16/PONLY__ultraep__one__seq8192__gbs16__cap2p40625/rep3/memory_snapshots/snapshot.pickle
|
| 7 |
+
repo=/workspace/Megatron-UltraEP @ 03d57ef7
|
| 8 |
+
==============================================================
|
| 9 |
+
W0830 07:20:57.989000 744857 torch/distributed/run.py:766]
|
| 10 |
+
W0830 07:20:57.989000 744857 torch/distributed/run.py:766] *****************************************
|
| 11 |
+
W0830 07:20:57.989000 744857 torch/distributed/run.py:766] Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed.
|
| 12 |
+
W0830 07:20:57.989000 744857 torch/distributed/run.py:766] *****************************************
|
| 13 |
+
/workspace/Megatron-UltraEP/megatron/training/arguments.py:831: UserWarning: Disabling sequence parallelism because tensor model parallelism is disabled
|
| 14 |
+
warnings.warn("Disabling sequence parallelism because tensor model parallelism is disabled")
|
| 15 |
+
/workspace/Megatron-UltraEP/megatron/training/arguments.py:831: UserWarning: Disabling sequence parallelism because tensor model parallelism is disabled
|
| 16 |
+
warnings.warn("Disabling sequence parallelism because tensor model parallelism is disabled")
|
| 17 |
+
/workspace/Megatron-UltraEP/megatron/training/arguments.py:831: UserWarning: Disabling sequence parallelism because tensor model parallelism is disabled
|
| 18 |
+
warnings.warn("Disabling sequence parallelism because tensor model parallelism is disabled")
|
| 19 |
+
WARNING: one_logger package is required to enable e2e metrics tracking. please go to https://confluence.nvidia.com/display/MLWFO/Package+Repositories for details to install it
|
| 20 |
+
/workspace/Megatron-UltraEP/megatron/training/arguments.py:831: UserWarning: Disabling sequence parallelism because tensor model parallelism is disabled
|
| 21 |
+
warnings.warn("Disabling sequence parallelism because tensor model parallelism is disabled")
|
| 22 |
+
/workspace/Megatron-UltraEP/megatron/training/arguments.py:831: UserWarning: Disabling sequence parallelism because tensor model parallelism is disabled
|
| 23 |
+
warnings.warn("Disabling sequence parallelism because tensor model parallelism is disabled")
|
| 24 |
+
/workspace/Megatron-UltraEP/megatron/training/arguments.py:831: UserWarning: Disabling sequence parallelism because tensor model parallelism is disabled
|
| 25 |
+
warnings.warn("Disabling sequence parallelism because tensor model parallelism is disabled")
|
| 26 |
+
using world size: 8, data-parallel size: 8, context-parallel size: 1, hierarchical context-parallel sizes: None, tensor-model-parallel size: 1, pipeline-model-parallel size: 1
|
| 27 |
+
WARNING: overriding default arguments for tokenizer_type:GPT2BPETokenizer with tokenizer_type:GPT2BPETokenizer
|
| 28 |
+
Number of virtual stages per pipeline stage: None
|
| 29 |
+
accumulate and all-reduce gradients in fp32 for bfloat16 data type.
|
| 30 |
+
using torch.bfloat16 for parameters ...
|
| 31 |
+
/workspace/Megatron-UltraEP/megatron/training/arguments.py:831: UserWarning: Disabling sequence parallelism because tensor model parallelism is disabled
|
| 32 |
+
warnings.warn("Disabling sequence parallelism because tensor model parallelism is disabled")
|
| 33 |
+
------------------------ arguments ------------------------
|
| 34 |
+
account_for_embedding_in_pipeline_split ......... False
|
| 35 |
+
account_for_loss_in_pipeline_split .............. False
|
| 36 |
+
accumulate_allreduce_grads_in_fp32 .............. True
|
| 37 |
+
adam_beta1 ...................................... 0.9
|
| 38 |
+
adam_beta2 ...................................... 0.999
|
| 39 |
+
adam_eps ........................................ 1e-08
|
| 40 |
+
add_bias_linear ................................. False
|
| 41 |
+
add_position_embedding .......................... False
|
| 42 |
+
add_qkv_bias .................................... False
|
| 43 |
+
adlr_autoresume ................................. False
|
| 44 |
+
adlr_autoresume_interval ........................ 1000
|
| 45 |
+
align_grad_reduce ............................... True
|
| 46 |
+
align_param_gather .............................. False
|
| 47 |
+
app_tag_run_name ................................ None
|
| 48 |
+
app_tag_run_version ............................. 0.0.0
|
| 49 |
+
apply_layernorm_1p .............................. False
|
| 50 |
+
apply_query_key_layer_scaling ................... False
|
| 51 |
+
apply_residual_connection_post_layernorm ........ False
|
| 52 |
+
apply_rope_fusion ............................... True
|
| 53 |
+
async_save ...................................... None
|
| 54 |
+
async_tensor_model_parallel_allreduce ........... True
|
| 55 |
+
attention_backend ............................... AttnBackend.auto
|
| 56 |
+
attention_dropout ............................... 0.0
|
| 57 |
+
attention_softmax_in_fp32 ....................... False
|
| 58 |
+
auto_detect_ckpt_format ......................... False
|
| 59 |
+
barrier_with_L1_time ............................ True
|
| 60 |
+
bert_binary_head ................................ True
|
| 61 |
+
bert_embedder_type .............................. megatron
|
| 62 |
+
bert_load ....................................... None
|
| 63 |
+
bf16 ............................................ True
|
| 64 |
+
bias_dropout_fusion ............................. True
|
| 65 |
+
bias_gelu_fusion ................................ False
|
| 66 |
+
bias_swiglu_fusion .............................. True
|
| 67 |
+
biencoder_projection_dim ........................ 0
|
| 68 |
+
biencoder_shared_query_context_model ............ False
|
| 69 |
+
block_data_path ................................. None
|
| 70 |
+
calc_ft_timeouts ................................ False
|
| 71 |
+
calculate_per_token_loss ........................ False
|
| 72 |
+
check_for_large_grads ........................... False
|
| 73 |
+
check_for_nan_in_loss_and_grad .................. False
|
| 74 |
+
check_for_spiky_loss ............................ False
|
| 75 |
+
check_weight_hash_across_dp_replicas_interval ... None
|
| 76 |
+
ckpt_assume_constant_structure .................. False
|
| 77 |
+
ckpt_convert_format ............................. None
|
| 78 |
+
ckpt_convert_save ............................... None
|
| 79 |
+
ckpt_convert_update_legacy_dist_opt_format ...... False
|
| 80 |
+
ckpt_format ..................................... torch_dist
|
| 81 |
+
ckpt_fully_parallel_load ........................ False
|
| 82 |
+
ckpt_fully_parallel_save ........................ True
|
| 83 |
+
ckpt_fully_parallel_save_deprecated ............. False
|
| 84 |
+
ckpt_step ....................................... None
|
| 85 |
+
classes_fraction ................................ 1.0
|
| 86 |
+
clip_grad ....................................... 1.0
|
| 87 |
+
clone_scatter_output_in_embedding ............... True
|
| 88 |
+
config_logger_dir ...............................
|
| 89 |
+
consumed_train_samples .......................... 0
|
| 90 |
+
consumed_valid_samples .......................... 0
|
| 91 |
+
context_parallel_size ........................... 1
|
| 92 |
+
cp_comm_type .................................... ['p2p']
|
| 93 |
+
create_attention_mask_in_dataloader ............. True
|
| 94 |
+
cross_entropy_fusion_impl ....................... native
|
| 95 |
+
cross_entropy_loss_fusion ....................... False
|
| 96 |
+
cuda_graph_scope ................................ full_iteration
|
| 97 |
+
cuda_graph_warmup_steps ......................... 3
|
| 98 |
+
data_args_path .................................. None
|
| 99 |
+
data_cache_path ................................. None
|
| 100 |
+
data_parallel_random_init ....................... False
|
| 101 |
+
data_parallel_sharding_strategy ................. no_shard
|
| 102 |
+
data_parallel_size .............................. 8
|
| 103 |
+
data_path ....................................... ['/workspace/data/wikitext/wikitext103_gpt2_text_document']
|
| 104 |
+
data_per_class_fraction ......................... 1.0
|
| 105 |
+
data_sharding ................................... True
|
| 106 |
+
dataloader_type ................................. single
|
| 107 |
+
ddp_average_in_collective ....................... False
|
| 108 |
+
ddp_bucket_size ................................. None
|
| 109 |
+
ddp_num_buckets ................................. None
|
| 110 |
+
ddp_pad_buckets_for_high_nccl_busbw ............. False
|
| 111 |
+
decoder_first_pipeline_num_layers ............... None
|
| 112 |
+
decoder_last_pipeline_num_layers ................ None
|
| 113 |
+
decoder_num_layers .............................. None
|
| 114 |
+
decoder_seq_length .............................. None
|
| 115 |
+
decoupled_lr .................................... None
|
| 116 |
+
decoupled_min_lr ................................ None
|
| 117 |
+
decrease_batch_size_if_needed ................... False
|
| 118 |
+
defer_embedding_wgrad_compute ................... False
|
| 119 |
+
delay_wgrad_compute ............................. False
|
| 120 |
+
deprecated_use_mcore_models ..................... True
|
| 121 |
+
deterministic_mode .............................. False
|
| 122 |
+
dino_bottleneck_size ............................ 256
|
| 123 |
+
dino_freeze_last_layer .......................... 1
|
| 124 |
+
dino_head_hidden_size ........................... 2048
|
| 125 |
+
dino_local_crops_number ......................... 10
|
| 126 |
+
dino_local_img_size ............................. 96
|
| 127 |
+
dino_norm_last_layer ............................ False
|
| 128 |
+
dino_teacher_temp ............................... 0.07
|
| 129 |
+
dino_warmup_teacher_temp ........................ 0.04
|
| 130 |
+
dino_warmup_teacher_temp_epochs ................. 30
|
| 131 |
+
disable_bf16_reduced_precision_matmul ........... False
|
| 132 |
+
disable_mamba_mem_eff_path ...................... False
|
| 133 |
+
disable_straggler_on_startup .................... False
|
| 134 |
+
dist_ckpt_format_deprecated ..................... None
|
| 135 |
+
dist_ckpt_strictness ............................ assume_ok_unexpected
|
| 136 |
+
distribute_saved_activations .................... False
|
| 137 |
+
distributed_backend ............................. nccl
|
| 138 |
+
distributed_timeout_minutes ..................... 10
|
| 139 |
+
embedding_init_method_std ....................... None
|
| 140 |
+
embedding_path .................................. None
|
| 141 |
+
empty_unused_memory_level ....................... 0
|
| 142 |
+
enable_cuda_graph ............................... True
|
| 143 |
+
enable_experimental ............................. False
|
| 144 |
+
enable_ft_package ............................... False
|
| 145 |
+
enable_gloo_process_groups ...................... True
|
| 146 |
+
enable_msc ...................................... True
|
| 147 |
+
enable_one_logger ............................... True
|
| 148 |
+
encoder_num_layers .............................. 16
|
| 149 |
+
encoder_seq_length .............................. 8192
|
| 150 |
+
end_weight_decay ................................ 0.1
|
| 151 |
+
eod_mask_loss ................................... False
|
| 152 |
+
error_injection_rate ............................ 0
|
| 153 |
+
error_injection_type ............................ transient_error
|
| 154 |
+
eval_interval ................................... 1000000
|
| 155 |
+
eval_iters ...................................... 0
|
| 156 |
+
evidence_data_path .............................. None
|
| 157 |
+
exit_duration_in_mins ........................... None
|
| 158 |
+
exit_interval ................................... None
|
| 159 |
+
exit_on_missing_checkpoint ...................... False
|
| 160 |
+
exit_signal_handler ............................. False
|
| 161 |
+
exp_avg_dtype ................................... torch.float32
|
| 162 |
+
exp_avg_sq_dtype ................................ torch.float32
|
| 163 |
+
expert_model_parallel_size ...................... 8
|
| 164 |
+
expert_tensor_parallel_size ..................... 1
|
| 165 |
+
export_force_local_attention .................... False
|
| 166 |
+
export_kd_cfg ................................... None
|
| 167 |
+
export_kd_teacher_ckpt_format ................... None
|
| 168 |
+
export_kd_teacher_load .......................... None
|
| 169 |
+
export_kv_cache_quant ........................... False
|
| 170 |
+
export_legacy_megatron .......................... False
|
| 171 |
+
export_model_type ............................... GPTModel
|
| 172 |
+
export_moe_apply_probs_on_input ................. False
|
| 173 |
+
export_qk_l2_norm ............................... False
|
| 174 |
+
export_quant_cfg ................................ None
|
| 175 |
+
export_real_quant_cfg ........................... None
|
| 176 |
+
export_te_mcore_model ........................... False
|
| 177 |
+
external_cuda_graph ............................. False
|
| 178 |
+
ffn_hidden_size ................................. 14336
|
| 179 |
+
finetune ........................................ False
|
| 180 |
+
finetune_data_split ............................. train
|
| 181 |
+
finetune_hf_dataset ............................. None
|
| 182 |
+
first_last_layers_bf16 .......................... False
|
| 183 |
+
flash_decode .................................... False
|
| 184 |
+
fp16 ............................................ False
|
| 185 |
+
fp16_lm_cross_entropy ........................... False
|
| 186 |
+
fp32_residual_connection ........................ False
|
| 187 |
+
fp8 ............................................. e4m3
|
| 188 |
+
fp8_amax_compute_algo ........................... most_recent
|
| 189 |
+
fp8_amax_history_len ............................ 1
|
| 190 |
+
fp8_interval .................................... 1
|
| 191 |
+
fp8_margin ...................................... 0
|
| 192 |
+
fp8_param_gather ................................ False
|
| 193 |
+
fp8_recipe ...................................... mxfp8
|
| 194 |
+
fp8_wgrad ....................................... True
|
| 195 |
+
fsdp_double_buffer .............................. False
|
| 196 |
+
global_batch_size ............................... 16
|
| 197 |
+
grad_reduce_in_bf16 ............................. False
|
| 198 |
+
gradient_accumulation_fusion .................... True
|
| 199 |
+
gradient_reduce_div_fusion ...................... True
|
| 200 |
+
group_query_attention ........................... True
|
| 201 |
+
head_lr_mult .................................... 1.0
|
| 202 |
+
heterogeneous_layers_config_encoded_json ........ None
|
| 203 |
+
heterogeneous_layers_config_path ................ None
|
| 204 |
+
hidden_dropout .................................. 0.0
|
| 205 |
+
hidden_size ..................................... 4096
|
| 206 |
+
hierarchical_context_parallel_sizes ............. None
|
| 207 |
+
high_priority_stream_groups ..................... []
|
| 208 |
+
hybrid_attention_ratio .......................... 0.0
|
| 209 |
+
hybrid_mlp_ratio ................................ 0.0
|
| 210 |
+
hybrid_override_pattern ......................... None
|
| 211 |
+
hysteresis ...................................... 2
|
| 212 |
+
ict_head_size ................................... None
|
| 213 |
+
ict_load ........................................ None
|
| 214 |
+
img_h ........................................... 224
|
| 215 |
+
img_w ........................................... 224
|
| 216 |
+
indexer_batch_size .............................. 128
|
| 217 |
+
indexer_log_interval ............................ 1000
|
| 218 |
+
inference_batch_times_seqlen_threshold .......... -1
|
| 219 |
+
inference_dynamic_batching ...................... False
|
| 220 |
+
inference_dynamic_batching_buffer_guaranteed_fraction 0.2
|
| 221 |
+
inference_dynamic_batching_buffer_overflow_factor None
|
| 222 |
+
inference_dynamic_batching_buffer_size_gb ....... 40.0
|
| 223 |
+
inference_dynamic_batching_chunk_size ........... 256
|
| 224 |
+
inference_dynamic_batching_max_requests_override None
|
| 225 |
+
inference_dynamic_batching_max_tokens_override .. None
|
| 226 |
+
inference_dynamic_batching_num_cuda_graphs ...... 16
|
| 227 |
+
inference_max_batch_size ........................ 8
|
| 228 |
+
inference_max_seq_length ........................ 2560
|
| 229 |
+
inference_rng_tracker ........................... False
|
| 230 |
+
init_method_std ................................. 0.01
|
| 231 |
+
init_method_xavier_uniform ...................... False
|
| 232 |
+
init_model_with_meta_device ..................... False
|
| 233 |
+
initial_loss_scale .............................. 4294967296
|
| 234 |
+
inprocess_active_world_size ..................... 8
|
| 235 |
+
inprocess_barrier_timeout ....................... 120
|
| 236 |
+
inprocess_completion_timeout .................... 120
|
| 237 |
+
inprocess_empty_cuda_cache ...................... False
|
| 238 |
+
inprocess_granularity ........................... node
|
| 239 |
+
inprocess_hard_timeout .......................... 90
|
| 240 |
+
inprocess_heartbeat_interval .................... 30
|
| 241 |
+
inprocess_heartbeat_timeout ..................... 60
|
| 242 |
+
inprocess_last_call_wait ........................ 1
|
| 243 |
+
inprocess_max_iterations ........................ None
|
| 244 |
+
inprocess_monitor_process_interval .............. 1.0
|
| 245 |
+
inprocess_monitor_thread_interval ............... 1.0
|
| 246 |
+
inprocess_progress_watchdog_interval ............ 1.0
|
| 247 |
+
inprocess_restart ............................... False
|
| 248 |
+
inprocess_soft_timeout .......................... 60
|
| 249 |
+
inprocess_termination_grace_time ................ 1
|
| 250 |
+
is_hybrid_model ................................. False
|
| 251 |
+
iter_per_epoch .................................. 1250
|
| 252 |
+
iterations_to_skip .............................. []
|
| 253 |
+
keep_fp8_transpose_cache_when_using_custom_fsdp . False
|
| 254 |
+
kitchen_config_file ............................. None
|
| 255 |
+
kitchen_recipe_number ........................... None
|
| 256 |
+
kv_channels ..................................... 128
|
| 257 |
+
kv_lora_rank .................................... 32
|
| 258 |
+
lazy_mpu_init ................................... None
|
| 259 |
+
load ............................................ None
|
| 260 |
+
load_main_params_from_ckpt ...................... None
|
| 261 |
+
load_model_opt_format ........................... False
|
| 262 |
+
local_rank ...................................... 0
|
| 263 |
+
log_energy ...................................... False
|
| 264 |
+
log_interval .................................... 1
|
| 265 |
+
log_loss_scale_to_tensorboard ................... True
|
| 266 |
+
log_memory_to_tensorboard ....................... False
|
| 267 |
+
log_num_zeros_in_grad ........................... False
|
| 268 |
+
log_params_norm ................................. False
|
| 269 |
+
log_progress .................................... False
|
| 270 |
+
log_straggler ................................... False
|
| 271 |
+
log_throughput .................................. True
|
| 272 |
+
log_timers_to_tensorboard ....................... False
|
| 273 |
+
log_validation_ppl_to_tensorboard ............... False
|
| 274 |
+
log_world_size_to_tensorboard ................... False
|
| 275 |
+
logging_level ................................... None
|
| 276 |
+
loss_scale ...................................... None
|
| 277 |
+
loss_scale_window ............................... 1000
|
| 278 |
+
lr .............................................. 0.0001
|
| 279 |
+
lr_decay_iters .................................. 8
|
| 280 |
+
lr_decay_samples ................................ None
|
| 281 |
+
lr_decay_style .................................. cosine
|
| 282 |
+
lr_warmup_fraction .............................. None
|
| 283 |
+
lr_warmup_init .................................. 0.0
|
| 284 |
+
lr_warmup_iters ................................. 5
|
| 285 |
+
lr_warmup_samples ............................... 0
|
| 286 |
+
lr_wsd_decay_iters .............................. None
|
| 287 |
+
lr_wsd_decay_samples ............................ None
|
| 288 |
+
lr_wsd_decay_style .............................. exponential
|
| 289 |
+
main_grads_dtype ................................ torch.float32
|
| 290 |
+
main_params_dtype ............................... torch.float32
|
| 291 |
+
make_vocab_size_divisible_by .................... 128
|
| 292 |
+
mamba_head_dim .................................. 64
|
| 293 |
+
mamba_num_groups ................................ 8
|
| 294 |
+
mamba_num_heads ................................. None
|
| 295 |
+
mamba_state_dim ................................. 128
|
| 296 |
+
manual_gc ....................................... False
|
| 297 |
+
manual_gc_eval .................................. True
|
| 298 |
+
manual_gc_interval .............................. 0
|
| 299 |
+
mask_factor ..................................... 1.0
|
| 300 |
+
mask_prob ....................................... 0.15
|
| 301 |
+
mask_type ....................................... random
|
| 302 |
+
masked_softmax_fusion ........................... False
|
| 303 |
+
max_position_embeddings ......................... 32768
|
| 304 |
+
max_tokens_to_oom ............................... 12000
|
| 305 |
+
memory_snapshot_path ............................ snapshot.pickle
|
| 306 |
+
merge_file ...................................... /workspace/data/wikitext/gpt2-merges.txt
|
| 307 |
+
micro_batch_size ................................ 1
|
| 308 |
+
microbatch_group_size_per_vp_stage .............. None
|
| 309 |
+
mid_level_dataset_surplus ....................... 0.005
|
| 310 |
+
min_loss_scale .................................. 1.0
|
| 311 |
+
min_lr .......................................... 1e-05
|
| 312 |
+
mlp_chunks_for_prefill .......................... 1
|
| 313 |
+
mmap_bin_files .................................. True
|
| 314 |
+
mock_data ....................................... False
|
| 315 |
+
moe_apply_probs_on_input ........................ False
|
| 316 |
+
moe_aux_loss_coeff .............................. 0.01
|
| 317 |
+
moe_capacity_monitor ............................ True
|
| 318 |
+
moe_deepep_num_sms .............................. 20
|
| 319 |
+
moe_echo_enable_random_offloading ............... False
|
| 320 |
+
moe_echo_expert_dispatch_overlap ................ False
|
| 321 |
+
moe_echo_expert_dispatcher_type ................. hybridep
|
| 322 |
+
moe_echo_recompute_expert_dispatch .............. False
|
| 323 |
+
moe_enable_deepep ............................... False
|
| 324 |
+
moe_enable_echo ................................. False
|
| 325 |
+
moe_enable_hybridep ............................. True
|
| 326 |
+
moe_enable_ultraep .............................. True
|
| 327 |
+
moe_expert_capacity_factor ...................... None
|
| 328 |
+
moe_extended_tp ................................. False
|
| 329 |
+
moe_ffn_hidden_size ............................. 1536
|
| 330 |
+
moe_grouped_gemm ................................ True
|
| 331 |
+
moe_hybridep_num_sms ............................ 16
|
| 332 |
+
moe_input_jitter_eps ............................ None
|
| 333 |
+
moe_layer_freq .................................. 1
|
| 334 |
+
moe_layer_recompute ............................. False
|
| 335 |
+
moe_num_echo_experts ............................ None
|
| 336 |
+
moe_num_redundant_experts_per_rank .............. 1
|
| 337 |
+
moe_pad_expert_input_to_capacity ................ False
|
| 338 |
+
moe_per_layer_logging ........................... False
|
| 339 |
+
moe_permute_fusion .............................. False
|
| 340 |
+
moe_received_token_capacity ..................... 2.40625
|
| 341 |
+
moe_router_bias_update_rate ..................... 0.001
|
| 342 |
+
moe_router_dtype ................................ fp32
|
| 343 |
+
moe_router_enable_expert_bias ................... False
|
| 344 |
+
moe_router_force_load_balancing ................. False
|
| 345 |
+
moe_router_group_topk ........................... None
|
| 346 |
+
moe_router_load_balancing_type .................. aux_loss
|
| 347 |
+
moe_router_num_groups ........................... None
|
| 348 |
+
moe_router_padding_for_fp8 ...................... False
|
| 349 |
+
moe_router_pre_softmax .......................... False
|
| 350 |
+
moe_router_score_function ....................... softmax
|
| 351 |
+
moe_router_topk ................................. 2
|
| 352 |
+
moe_router_topk_scaling_factor .................. None
|
| 353 |
+
moe_shared_expert_intermediate_size ............. None
|
| 354 |
+
moe_shared_expert_overlap ....................... False
|
| 355 |
+
moe_token_dispatcher_type ....................... flex
|
| 356 |
+
moe_token_drop_policy ........................... probs
|
| 357 |
+
moe_ultraep_max_microbatches .................... None
|
| 358 |
+
moe_ultraep_overlap_weight_sync ................. True
|
| 359 |
+
moe_ultraep_weight_sync_join .................... before_dispatch
|
| 360 |
+
moe_upcycling_granularity ....................... 1
|
| 361 |
+
moe_use_device_initiated_grouped_gemm ........... True
|
| 362 |
+
moe_use_legacy_grouped_gemm ..................... False
|
| 363 |
+
moe_use_upcycling ............................... False
|
| 364 |
+
moe_z_loss_coeff ................................ None
|
| 365 |
+
mrope_section ................................... None
|
| 366 |
+
mscale .......................................... 1.0
|
| 367 |
+
mscale_all_dim .................................. 0.0
|
| 368 |
+
mtp_loss_scaling_factor ......................... 0.1
|
| 369 |
+
mtp_num_layers .................................. None
|
| 370 |
+
multi_latent_attention .......................... False
|
| 371 |
+
nccl_all_reduce_for_prefill ..................... False
|
| 372 |
+
nccl_communicator_config_path ................... None
|
| 373 |
+
nccl_ub ......................................... False
|
| 374 |
+
no_load_optim ................................... None
|
| 375 |
+
no_load_rng ..................................... None
|
| 376 |
+
no_persist_layer_norm ........................... False
|
| 377 |
+
no_rope_freq .................................... None
|
| 378 |
+
no_save_optim ................................... None
|
| 379 |
+
no_save_rng ..................................... None
|
| 380 |
+
non_persistent_ckpt_type ........................ None
|
| 381 |
+
non_persistent_global_ckpt_dir .................. None
|
| 382 |
+
non_persistent_local_ckpt_algo .................. fully_parallel
|
| 383 |
+
non_persistent_local_ckpt_dir ................... None
|
| 384 |
+
non_persistent_save_interval .................... None
|
| 385 |
+
norm_epsilon .................................... 1e-05
|
| 386 |
+
normalization ................................... RMSNorm
|
| 387 |
+
num_attention_heads ............................. 32
|
| 388 |
+
num_channels .................................... 3
|
| 389 |
+
num_classes ..................................... 1000
|
| 390 |
+
num_dataset_builder_threads ..................... 1
|
| 391 |
+
num_distributed_optimizer_instances ............. 1
|
| 392 |
+
num_experts ..................................... 8
|
| 393 |
+
num_layers ...................................... 16
|
| 394 |
+
num_layers_at_end_in_bf16 ....................... 1
|
| 395 |
+
num_layers_at_start_in_bf16 ..................... 1
|
| 396 |
+
num_layers_per_virtual_pipeline_stage ........... None
|
| 397 |
+
num_query_groups ................................ 8
|
| 398 |
+
num_virtual_stages_per_pipeline_rank ............ None
|
| 399 |
+
num_workers ..................................... 2
|
| 400 |
+
object_storage_cache_path ....................... None
|
| 401 |
+
one_logger_async ................................ False
|
| 402 |
+
one_logger_project .............................. megatron-lm
|
| 403 |
+
one_logger_run_name ............................. None
|
| 404 |
+
onnx_safe ....................................... None
|
| 405 |
+
openai_gelu ..................................... False
|
| 406 |
+
optimizer ....................................... adam
|
| 407 |
+
optimizer_cpu_offload ........................... False
|
| 408 |
+
optimizer_offload_fraction ...................... 1.0
|
| 409 |
+
output_bert_embeddings .......................... False
|
| 410 |
+
overlap_cpu_optimizer_d2h_h2d ................... False
|
| 411 |
+
overlap_grad_reduce ............................. True
|
| 412 |
+
overlap_moe_expert_parallel_comm ................ False
|
| 413 |
+
overlap_p2p_comm ................................ False
|
| 414 |
+
overlap_p2p_comm_warmup_flush ................... False
|
| 415 |
+
overlap_param_gather ............................ True
|
| 416 |
+
overlap_param_gather_with_optimizer_step ........ False
|
| 417 |
+
override_opt_param_scheduler .................... False
|
| 418 |
+
params_dtype .................................... torch.bfloat16
|
| 419 |
+
patch_dim ....................................... 16
|
| 420 |
+
per_split_data_args_path ........................ None
|
| 421 |
+
perform_initialization .......................... True
|
| 422 |
+
pin_cpu_grads ................................... True
|
| 423 |
+
pin_cpu_params .................................. True
|
| 424 |
+
pipeline_model_parallel_comm_backend ............ None
|
| 425 |
+
pipeline_model_parallel_layout .................. None
|
| 426 |
+
pipeline_model_parallel_size .................... 1
|
| 427 |
+
position_embedding_type ......................... rope
|
| 428 |
+
pretrained_checkpoint ........................... None
|
| 429 |
+
profile ......................................... False
|
| 430 |
+
profile_ranks ................................... [0]
|
| 431 |
+
profile_step_end ................................ 12
|
| 432 |
+
profile_step_start .............................. 10
|
| 433 |
+
q_lora_rank ..................................... None
|
| 434 |
+
qk_head_dim ..................................... 128
|
| 435 |
+
qk_l2_norm ...................................... False
|
| 436 |
+
qk_layernorm .................................... False
|
| 437 |
+
qk_pos_emb_head_dim ............................. 64
|
| 438 |
+
query_in_block_prob ............................. 0.1
|
| 439 |
+
rampup_batch_size ............................... None
|
| 440 |
+
rank ............................................ 0
|
| 441 |
+
recompute_granularity ........................... None
|
| 442 |
+
recompute_method ................................ None
|
| 443 |
+
recompute_modules ............................... None
|
| 444 |
+
recompute_num_layers ............................ None
|
| 445 |
+
record_memory_history ........................... False
|
| 446 |
+
relative_attention_max_distance ................. 128
|
| 447 |
+
relative_attention_num_buckets .................. 32
|
| 448 |
+
replication ..................................... False
|
| 449 |
+
replication_factor .............................. 2
|
| 450 |
+
replication_jump ................................ None
|
| 451 |
+
rerun_mode ...................................... validate_results
|
| 452 |
+
reset_attention_mask ............................ False
|
| 453 |
+
reset_position_ids .............................. False
|
| 454 |
+
result_rejected_tracker_filename ................ None
|
| 455 |
+
retriever_report_topk_accuracies ................ []
|
| 456 |
+
retriever_score_scaling ......................... False
|
| 457 |
+
retriever_seq_length ............................ 256
|
| 458 |
+
retro_add_retriever ............................. False
|
| 459 |
+
retro_attention_gate ............................ 1
|
| 460 |
+
retro_cyclic_train_iters ........................ None
|
| 461 |
+
retro_encoder_attention_dropout ................. 0.1
|
| 462 |
+
retro_encoder_hidden_dropout .................... 0.1
|
| 463 |
+
retro_encoder_layers ............................ 2
|
| 464 |
+
retro_num_neighbors ............................. 2
|
| 465 |
+
retro_num_retrieved_chunks ...................... 2
|
| 466 |
+
retro_project_dir ............................... None
|
| 467 |
+
retro_verify_neighbor_count ..................... True
|
| 468 |
+
reuse_grad_buf_for_mxfp8_param_ag ............... False
|
| 469 |
+
rope_scaling_factor ............................. 8.0
|
| 470 |
+
rope_type ....................................... None
|
| 471 |
+
rotary_base ..................................... 1000000
|
| 472 |
+
rotary_interleaved .............................. False
|
| 473 |
+
rotary_percent .................................. 1.0
|
| 474 |
+
rotary_scaling_factor ........................... 1.0
|
| 475 |
+
rotary_seq_len_interpolation_factor ............. None
|
| 476 |
+
run_workload_inspector_server ................... False
|
| 477 |
+
sample_rate ..................................... 1.0
|
| 478 |
+
save ............................................ None
|
| 479 |
+
save_interval ................................... None
|
| 480 |
+
save_retain_interval ............................ None
|
| 481 |
+
scatter_gather_tensors_in_pipeline .............. True
|
| 482 |
+
seed ............................................ 1234
|
| 483 |
+
seq_length ...................................... 8192
|
| 484 |
+
sequence_parallel ............................... False
|
| 485 |
+
sft ............................................. False
|
| 486 |
+
sft_tokenizer_prompt_format ..................... nemotron-h-aligned
|
| 487 |
+
sgd_momentum .................................... 0.9
|
| 488 |
+
short_seq_prob .................................. 0.1
|
| 489 |
+
skip_train ...................................... False
|
| 490 |
+
skipped_train_samples ........................... 0
|
| 491 |
+
spec ............................................ None
|
| 492 |
+
split ........................................... 100,0,0
|
| 493 |
+
squared_relu .................................... False
|
| 494 |
+
start_weight_decay .............................. 0.1
|
| 495 |
+
straggler_ctrlr_port ............................ 65535
|
| 496 |
+
straggler_minmax_count .......................... 1
|
| 497 |
+
suggested_communication_unit_size ............... None
|
| 498 |
+
swiglu .......................................... True
|
| 499 |
+
swin_backbone_type .............................. tiny
|
| 500 |
+
symmetric_ar_type ............................... None
|
| 501 |
+
te_rng_tracker .................................. True
|
| 502 |
+
tensor_model_parallel_size ...................... 1
|
| 503 |
+
tensorboard_dir ................................. None
|
| 504 |
+
tensorboard_log_interval ........................ 1
|
| 505 |
+
tensorboard_queue_size .......................... 1000
|
| 506 |
+
test_data_path .................................. None
|
| 507 |
+
test_mode ....................................... False
|
| 508 |
+
tiktoken_num_special_tokens ..................... 1000
|
| 509 |
+
tiktoken_pattern ................................ None
|
| 510 |
+
tiktoken_special_tokens ......................... None
|
| 511 |
+
timing_log_level ................................ 0
|
| 512 |
+
timing_log_option ............................... minmax
|
| 513 |
+
titles_data_path ................................ None
|
| 514 |
+
tokenizer_model ................................. None
|
| 515 |
+
tokenizer_type .................................. GPT2BPETokenizer
|
| 516 |
+
torch_fsdp2_reshard_after_forward ............... True
|
| 517 |
+
tp_comm_bootstrap_backend ....................... nccl
|
| 518 |
+
tp_comm_bulk_dgrad .............................. True
|
| 519 |
+
tp_comm_bulk_wgrad .............................. True
|
| 520 |
+
tp_comm_overlap ................................. False
|
| 521 |
+
tp_comm_overlap_ag .............................. True
|
| 522 |
+
tp_comm_overlap_cfg ............................. None
|
| 523 |
+
tp_comm_overlap_rs .............................. True
|
| 524 |
+
tp_comm_overlap_rs_dgrad ........................ False
|
| 525 |
+
tp_comm_split_ag ................................ True
|
| 526 |
+
tp_comm_split_rs ................................ True
|
| 527 |
+
train_data_path ................................. None
|
| 528 |
+
train_iters ..................................... 12
|
| 529 |
+
train_samples ................................... None
|
| 530 |
+
train_sync_interval ............................. None
|
| 531 |
+
transformer_impl ................................ transformer_engine
|
| 532 |
+
transformer_pipeline_model_parallel_size ........ 1
|
| 533 |
+
untie_embeddings_and_output_weights ............. True
|
| 534 |
+
use_checkpoint_args ............................. False
|
| 535 |
+
use_checkpoint_opt_param_scheduler .............. False
|
| 536 |
+
use_cpu_initialization .......................... None
|
| 537 |
+
use_custom_fsdp ................................. False
|
| 538 |
+
use_dist_ckpt ................................... True
|
| 539 |
+
use_dist_ckpt_deprecated ........................ False
|
| 540 |
+
use_distributed_optimizer ....................... True
|
| 541 |
+
use_flash_attn .................................. False
|
| 542 |
+
use_legacy_models ............................... False
|
| 543 |
+
use_mp_args_from_checkpoint_args ................ False
|
| 544 |
+
use_one_sent_docs ............................... False
|
| 545 |
+
use_persistent_ckpt_worker ...................... False
|
| 546 |
+
use_precision_aware_optimizer ................... False
|
| 547 |
+
use_pytorch_profiler ............................ False
|
| 548 |
+
use_ring_exchange_p2p ........................... False
|
| 549 |
+
use_rope_scaling ................................ False
|
| 550 |
+
use_rotary_position_embeddings .................. False
|
| 551 |
+
use_sharp ....................................... False
|
| 552 |
+
use_tokenizer_model_from_checkpoint_args ........ True
|
| 553 |
+
use_torch_fsdp2 ................................. False
|
| 554 |
+
use_torch_optimizer_for_cpu_offload ............. False
|
| 555 |
+
use_tp_pp_dp_mapping ............................ False
|
| 556 |
+
v_head_dim ...................................... 128
|
| 557 |
+
valid_data_path ................................. None
|
| 558 |
+
variable_seq_lengths ............................ False
|
| 559 |
+
virtual_pipeline_model_parallel_size ............ None
|
| 560 |
+
vision_backbone_type ............................ vit
|
| 561 |
+
vision_pretraining .............................. False
|
| 562 |
+
vision_pretraining_type ......................... classify
|
| 563 |
+
vocab_extra_ids ................................. 0
|
| 564 |
+
vocab_file ...................................... /workspace/data/wikitext/gpt2-vocab.json
|
| 565 |
+
vocab_size ...................................... None
|
| 566 |
+
wandb_exp_name ..................................
|
| 567 |
+
wandb_project ...................................
|
| 568 |
+
wandb_save_dir ..................................
|
| 569 |
+
weight_decay .................................... 0.1
|
| 570 |
+
weight_decay_incr_style ......................... constant
|
| 571 |
+
wgrad_deferral_limit ............................ 0
|
| 572 |
+
world_size ...................................... 8
|
| 573 |
+
yaml_cfg ........................................ None
|
| 574 |
+
-------------------- end of arguments ---------------------
|
| 575 |
+
INFO:megatron.core.num_microbatches_calculator:setting number of microbatches to constant 2
|
| 576 |
+
> building GPT2BPETokenizer tokenizer ...
|
| 577 |
+
/workspace/Megatron-UltraEP/megatron/training/arguments.py:831: UserWarning: Disabling sequence parallelism because tensor model parallelism is disabled
|
| 578 |
+
warnings.warn("Disabling sequence parallelism because tensor model parallelism is disabled")
|
| 579 |
+
> padded vocab (size: 50257) with 47 dummy tokens (new size: 50304)
|
| 580 |
+
WARNING:megatron.core.rerun_state_machine:RerunStateMachine initialized in mode RerunMode.VALIDATE_RESULTS
|
| 581 |
+
> initializing torch distributed ...
|
| 582 |
+
[Gloo] Rank [Gloo] Rank 2 is connected to 07 is connected to peer ranks. 7Expected number of connected peer ranks is : peer ranks. 7Expected number of connected peer ranks is : 7
|
| 583 |
+
|
| 584 |
+
[Gloo] Rank 5 is connected to 7 peer ranks. Expected number of connected peer ranks is : 7
|
| 585 |
+
[Gloo] Rank 4 is connected to 7 peer ranks. Expected number of connected peer ranks is : [Gloo] Rank 7
|
| 586 |
+
1 is connected to 7 peer ranks. Expected number of connected peer ranks is : 7
|
| 587 |
+
[Gloo] Rank 3 is connected to 7 peer ranks. Expected number of connected peer ranks is : 7
|
| 588 |
+
[Gloo] Rank 6 is connected to 7 peer ranks. Expected number of connected peer ranks is : 7
|
| 589 |
+
[Gloo] Rank 7 is connected to 7 peer ranks. Expected number of connected peer ranks is : 7
|
| 590 |
+
[Gloo] Rank 0 is connected to 7 peer ranks. Expected number of connected peer ranks is : 7
|
| 591 |
+
[Gloo] Rank 1 is connected to 7 peer ranks. Expected number of connected peer ranks is : 7
|
| 592 |
+
[Gloo] Rank [Gloo] Rank [Gloo] Rank [Gloo] Rank [Gloo] Rank [Gloo] Rank 324576 is connected to is connected to is connected to is connected to is connected to is connected to 777777 peer ranks. peer ranks. peer ranks. peer ranks. peer ranks. peer ranks. Expected number of connected peer ranks is : Expected number of connected peer ranks is : Expected number of connected peer ranks is : Expected number of connected peer ranks is : Expected number of connected peer ranks is : Expected number of connected peer ranks is : 77777
|
| 593 |
+
7
|
| 594 |
+
|
| 595 |
+
|
| 596 |
+
|
| 597 |
+
|
| 598 |
+
[Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
|
| 599 |
+
> initialized tensor model parallel with size 1
|
| 600 |
+
> initialized pipeline model parallel with size 1
|
| 601 |
+
> setting random seeds to 1234 ...
|
| 602 |
+
[Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
|
| 603 |
+
[Gloo] Rank [Gloo] Rank 00 is connected to is connected to 00 peer ranks. peer ranks. Expected number of connected peer ranks is : Expected number of connected peer ranks is : 00
|
| 604 |
+
|
| 605 |
+
[Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
|
| 606 |
+
[Gloo] Rank [Gloo] Rank 00 is connected to is connected to [Gloo] Rank 000 peer ranks. peer ranks. is connected to Expected number of connected peer ranks is : Expected number of connected peer ranks is : 000
|
| 607 |
+
peer ranks.
|
| 608 |
+
Expected number of connected peer ranks is : 0
|
| 609 |
+
[rank1]:[W830 07:21:15.733562259 ProcessGroupNCCL.cpp:4789] [PG ID 0 PG GUID 0 Rank 1] using GPU 1 as device used by this process is currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect. You can specify device_id in init_process_group() to force use of a particular device.
|
| 610 |
+
[rank5]:[W830 07:21:15.742400969 ProcessGroupNCCL.cpp:4789] [PG ID 0 PG GUID 0 Rank 5] using GPU 5 as device used by this process is currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect. You can specify device_id in init_process_group() to force use of a particular device.
|
| 611 |
+
[rank2]:[W830 07:21:15.742431416 ProcessGroupNCCL.cpp:4789] [PG ID 0 PG GUID 0 Rank 2] using GPU 2 as device used by this process is currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect. You can specify device_id in init_process_group() to force use of a particular device.
|
| 612 |
+
[rank4]:[W830 07:21:15.742441721 ProcessGroupNCCL.cpp:4789] [PG ID 0 PG GUID 0 Rank 4] using GPU 4 as device used by this process is currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect. You can specify device_id in init_process_group() to force use of a particular device.
|
| 613 |
+
[rank7]:[W830 07:21:15.742489500 ProcessGroupNCCL.cpp:4789] [PG ID 0 PG GUID 0 Rank 7] using GPU 7 as device used by this process is currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect. You can specify device_id in init_process_group() to force use of a particular device.
|
| 614 |
+
[rank3]:[W830 07:21:15.742508197 ProcessGroupNCCL.cpp:4789] [PG ID 0 PG GUID 0 Rank 3] using GPU 3 as device used by this process is currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect. You can specify device_id in init_process_group() to force use of a particular device.
|
| 615 |
+
[rank6]:[W830 07:21:15.742530488 ProcessGroupNCCL.cpp:4789] [PG ID 0 PG GUID 0 Rank 6] using GPU 6 as device used by this process is currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect. You can specify device_id in init_process_group() to force use of a particular device.
|
| 616 |
+
> compiling dataset index builder ...
|
| 617 |
+
make: Entering directory '/workspace/Megatron-UltraEP/megatron/core/datasets'
|
| 618 |
+
make: Nothing to be done for 'default'.
|
| 619 |
+
make: Leaving directory '/workspace/Megatron-UltraEP/megatron/core/datasets'
|
| 620 |
+
>>> done with dataset index builder. Compilation time: 0.035 seconds
|
| 621 |
+
WARNING: constraints for invoking optimized fused softmax kernel are not met. We default back to unfused kernel invocations.
|
| 622 |
+
> compiling and loading fused kernels ...
|
| 623 |
+
[rank0]:[W830 07:21:15.929293731 ProcessGroupNCCL.cpp:4789] [PG ID 0 PG GUID 0 Rank 0] using GPU 0 as device used by this process is currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect. You can specify device_id in init_process_group() to force use of a particular device.
|
| 624 |
+
>>> done with compiling and loading fused kernels. Compilation time: 3.978 seconds
|
| 625 |
+
time to initialize megatron (seconds): 7.314
|
| 626 |
+
[after megatron is initialized] datetime: 2026-08-30 07:21:21
|
| 627 |
+
building GPT model ...
|
| 628 |
+
/usr/local/lib/python3.12/dist-packages/transformer_engine/pytorch/cpu_offload.py:708: DeprecationWarning: Offloading weights is deprecated. Using offload_weights=True does not have any effect.
|
| 629 |
+
warnings.warn(
|
| 630 |
+
/usr/local/lib/python3.12/dist-packages/transformer_engine/pytorch/cpu_offload.py:708: DeprecationWarning: Offloading weights is deprecated. Using offload_weights=True does not have any effect.
|
| 631 |
+
warnings.warn(
|
| 632 |
+
/usr/local/lib/python3.12/dist-packages/transformer_engine/pytorch/cpu_offload.py:708: DeprecationWarning: Offloading weights is deprecated. Using offload_weights=True does not have any effect.
|
| 633 |
+
warnings.warn(
|
| 634 |
+
/usr/local/lib/python3.12/dist-packages/transformer_engine/pytorch/cpu_offload.py:708: DeprecationWarning: Offloading weights is deprecated. Using offload_weights=True does not have any effect.
|
| 635 |
+
warnings.warn(
|
| 636 |
+
/usr/local/lib/python3.12/dist-packages/transformer_engine/pytorch/cpu_offload.py:708: DeprecationWarning: Offloading weights is deprecated. Using offload_weights=True does not have any effect.
|
| 637 |
+
warnings.warn(
|
| 638 |
+
/usr/local/lib/python3.12/dist-packages/transformer_engine/pytorch/cpu_offload.py:708: DeprecationWarning: Offloading weights is deprecated. Using offload_weights=True does not have any effect.
|
| 639 |
+
warnings.warn(
|
| 640 |
+
/usr/local/lib/python3.12/dist-packages/transformer_engine/pytorch/cpu_offload.py:708: DeprecationWarning: Offloading weights is deprecated. Using offload_weights=True does not have any effect.
|
| 641 |
+
warnings.warn(
|
| 642 |
+
/usr/local/lib/python3.12/dist-packages/transformer_engine/pytorch/cpu_offload.py:708: DeprecationWarning: Offloading weights is deprecated. Using offload_weights=True does not have any effect.
|
| 643 |
+
warnings.warn(
|
| 644 |
+
[INFO] Use MNNVL fabric: False
|
| 645 |
+
[WARN] MAX_NUM_NVL_PEERS is not set. Using detected value 8.
|
| 646 |
+
NVSHMEM v3.7.2
|
| 647 |
+
/dvs/p4/build/sw/rel/gpgpu/toolkit/r12.9/main_nvshmem/src/modules/transport/ibrc/ibrc.cpp:1754: error status: 7 (NVSHMEMX_ERROR_INTERNAL) Device enumeration failed.
|
| 648 |
+
|
| 649 |
+
/dvs/p4/build/sw/rel/gpgpu/toolkit/r12.9/main_nvshmem/src/modules/transport/ibrc/ibrc.cpp:1754: error status: 7 (NVSHMEMX_ERROR_INTERNAL) Device enumeration failed.
|
| 650 |
+
|
| 651 |
+
/dvs/p4/build/sw/rel/gpgpu/toolkit/r12.9/main_nvshmem/src/modules/transport/ibrc/ibrc.cpp:1754: error status: 7 (NVSHMEMX_ERROR_INTERNAL) Device enumeration failed.
|
| 652 |
+
|
| 653 |
+
/dvs/p4/build/sw/rel/gpgpu/toolkit/r12.9/main_nvshmem/src/modules/transport/ibrc/ibrc.cpp:1754: error status: 7 (NVSHMEMX_ERROR_INTERNAL) Device enumeration failed.
|
| 654 |
+
|
| 655 |
+
/dvs/p4/build/sw/rel/gpgpu/toolkit/r12.9/main_nvshmem/src/modules/transport/ibrc/ibrc.cpp:1754: error status: 7 (NVSHMEMX_ERROR_INTERNAL) Device enumeration failed.
|
| 656 |
+
|
| 657 |
+
/dvs/p4/build/sw/rel/gpgpu/toolkit/r12.9/main_nvshmem/src/modules/transport/ibrc/ibrc.cpp:1754: error status: 7 (NVSHMEMX_ERROR_INTERNAL) Device enumeration failed.
|
| 658 |
+
|
| 659 |
+
/dvs/p4/build/sw/rel/gpgpu/toolkit/r12.9/main_nvshmem/src/modules/transport/ibrc/ibrc.cpp:1754: error status: 7 (NVSHMEMX_ERROR_INTERNAL) Device enumeration failed.
|
| 660 |
+
|
| 661 |
+
/dvs/p4/build/sw/rel/gpgpu/toolkit/r12.9/main_nvshmem/src/modules/transport/ibrc/ibrc.cpp:1754: error status: 7 (NVSHMEMX_ERROR_INTERNAL) Device enumeration failed.
|
| 662 |
+
|
| 663 |
+
UltraEP: NVL domain size <= 8; forcing weight_sync plan mode to direct (was adaptive).
|
| 664 |
+
> number of parameters on (tensor, pipeline) model parallel rank (0, 0): 1687818240
|
| 665 |
+
INFO:megatron.core.distributed.distributed_data_parallel:Setting up DistributedDataParallel with config DistributedDataParallelConfig(grad_reduce_in_fp32=True, overlap_grad_reduce=True, overlap_param_gather=True, align_param_gather=False, use_distributed_optimizer=True, num_distributed_optimizer_instances=1, check_for_nan_in_grad=False, check_for_large_grads=False, bucket_size=40000000, pad_buckets_for_high_nccl_busbw=False, average_in_collective=False, fp8_param_gather=False, reuse_grad_buf_for_mxfp8_param_ag=False, use_custom_fsdp=False, data_parallel_sharding_strategy='no_shard', gradient_reduce_div_fusion=True, suggested_communication_unit_size=None, preserve_fp32_weights=True, keep_fp8_transpose_cache_when_using_custom_fsdp=False, nccl_ub=False, fsdp_double_buffer=False)
|
| 666 |
+
INFO:megatron.core.distributed.param_and_grad_buffer:Number of buckets for gradient all-reduce / reduce-scatter: 18
|
| 667 |
+
Params for bucket 1 (206045184 elements, 206045184 padded size):
|
| 668 |
+
module.output_layer.weight
|
| 669 |
+
Params for bucket 2 (41988096 elements, 41988096 padded size):
|
| 670 |
+
module.decoder.layers.15.pre_mlp_layernorm.weight
|
| 671 |
+
module.decoder.layers.15.self_attention.linear_qkv.weight
|
| 672 |
+
module.decoder.final_layernorm.weight
|
| 673 |
+
module.decoder.layers.15.self_attention.linear_qkv.layer_norm_weight
|
| 674 |
+
module.decoder.layers.15.mlp.router.weight
|
| 675 |
+
module.decoder.layers.15.self_attention.linear_proj.weight
|
| 676 |
+
Params for bucket 3 (41984000 elements, 41984000 padded size):
|
| 677 |
+
module.decoder.layers.14.mlp.router.weight
|
| 678 |
+
module.decoder.layers.14.self_attention.linear_proj.weight
|
| 679 |
+
module.decoder.layers.14.self_attention.linear_qkv.layer_norm_weight
|
| 680 |
+
module.decoder.layers.14.self_attention.linear_qkv.weight
|
| 681 |
+
module.decoder.layers.14.pre_mlp_layernorm.weight
|
| 682 |
+
Params for bucket 4 (41984000 elements, 41984000 padded size):
|
| 683 |
+
module.decoder.layers.13.self_attention.linear_qkv.weight
|
| 684 |
+
module.decoder.layers.13.pre_mlp_layernorm.weight
|
| 685 |
+
module.decoder.layers.13.mlp.router.weight
|
| 686 |
+
module.decoder.layers.13.self_attention.linear_qkv.layer_norm_weight
|
| 687 |
+
module.decoder.layers.13.self_attention.linear_proj.weight
|
| 688 |
+
Params for bucket 5 (41984000 elements, 41984000 padded size):
|
| 689 |
+
module.decoder.layers.12.self_attention.linear_qkv.weight
|
| 690 |
+
module.decoder.layers.12.pre_mlp_layernorm.weight
|
| 691 |
+
module.decoder.layers.12.mlp.router.weight
|
| 692 |
+
module.decoder.layers.12.self_attention.linear_proj.weight
|
| 693 |
+
module.decoder.layers.12.self_attention.linear_qkv.layer_norm_weight
|
| 694 |
+
Params for bucket 6 (41984000 elements, 41984000 padded size):
|
| 695 |
+
module.decoder.layers.11.self_attention.linear_qkv.weight
|
| 696 |
+
module.decoder.layers.11.pre_mlp_layernorm.weight
|
| 697 |
+
module.decoder.layers.11.mlp.router.weight
|
| 698 |
+
module.decoder.layers.11.self_attention.linear_proj.weight
|
| 699 |
+
module.decoder.layers.11.self_attention.linear_qkv.layer_norm_weight
|
| 700 |
+
Params for bucket 7 (41984000 elements, 41984000 padded size):
|
| 701 |
+
module.decoder.layers.10.self_attention.linear_qkv.weight
|
| 702 |
+
module.decoder.layers.10.pre_mlp_layernorm.weight
|
| 703 |
+
module.decoder.layers.10.mlp.router.weight
|
| 704 |
+
module.decoder.layers.10.self_attention.linear_proj.weight
|
| 705 |
+
module.decoder.layers.10.self_attention.linear_qkv.layer_norm_weight
|
| 706 |
+
Params for bucket 8 (41984000 elements, 41984000 padded size):
|
| 707 |
+
module.decoder.layers.9.mlp.router.weight
|
| 708 |
+
module.decoder.layers.9.self_attention.linear_proj.weight
|
| 709 |
+
module.decoder.layers.9.self_attention.linear_qkv.layer_norm_weight
|
| 710 |
+
module.decoder.layers.9.self_attention.linear_qkv.weight
|
| 711 |
+
module.decoder.layers.9.pre_mlp_layernorm.weight
|
| 712 |
+
Params for bucket 9 (41984000 elements, 41984000 padded size):
|
| 713 |
+
module.decoder.layers.8.mlp.router.weight
|
| 714 |
+
module.decoder.layers.8.self_attention.linear_proj.weight
|
| 715 |
+
module.decoder.layers.8.self_attention.linear_qkv.layer_norm_weight
|
| 716 |
+
module.decoder.layers.8.pre_mlp_layernorm.weight
|
| 717 |
+
module.decoder.layers.8.self_attention.linear_qkv.weight
|
| 718 |
+
Params for bucket 10 (41984000 elements, 41984000 padded size):
|
| 719 |
+
module.decoder.layers.7.mlp.router.weight
|
| 720 |
+
module.decoder.layers.7.self_attention.linear_proj.weight
|
| 721 |
+
module.decoder.layers.7.self_attention.linear_qkv.layer_norm_weight
|
| 722 |
+
module.decoder.layers.7.self_attention.linear_qkv.weight
|
| 723 |
+
module.decoder.layers.7.pre_mlp_layernorm.weight
|
| 724 |
+
Params for bucket 11 (41984000 elements, 41984000 padded size):
|
| 725 |
+
module.decoder.layers.6.mlp.router.weight
|
| 726 |
+
module.decoder.layers.6.self_attention.linear_proj.weight
|
| 727 |
+
module.decoder.layers.6.self_attention.linear_qkv.layer_norm_weight
|
| 728 |
+
module.decoder.layers.6.pre_mlp_layernorm.weight
|
| 729 |
+
module.decoder.layers.6.self_attention.linear_qkv.weight
|
| 730 |
+
Params for bucket 12 (41984000 elements, 41984000 padded size):
|
| 731 |
+
module.decoder.layers.5.self_attention.linear_qkv.weight
|
| 732 |
+
module.decoder.layers.5.self_attention.linear_proj.weight
|
| 733 |
+
module.decoder.layers.5.pre_mlp_layernorm.weight
|
| 734 |
+
module.decoder.layers.5.mlp.router.weight
|
| 735 |
+
module.decoder.layers.5.self_attention.linear_qkv.layer_norm_weight
|
| 736 |
+
Params for bucket 13 (41984000 elements, 41984000 padded size):
|
| 737 |
+
module.decoder.layers.4.mlp.router.weight
|
| 738 |
+
module.decoder.layers.4.self_attention.linear_qkv.layer_norm_weight
|
| 739 |
+
module.decoder.layers.4.self_attention.linear_proj.weight
|
| 740 |
+
module.decoder.layers.4.self_attention.linear_qkv.weight
|
| 741 |
+
module.decoder.layers.4.pre_mlp_layernorm.weight
|
| 742 |
+
Params for bucket 14 (41984000 elements, 41984000 padded size):
|
| 743 |
+
module.decoder.layers.3.self_attention.linear_qkv.weight
|
| 744 |
+
module.decoder.layers.3.pre_mlp_layernorm.weight
|
| 745 |
+
module.decoder.layers.3.mlp.router.weight
|
| 746 |
+
module.decoder.layers.3.self_attention.linear_qkv.layer_norm_weight
|
| 747 |
+
module.decoder.layers.3.self_attention.linear_proj.weight
|
| 748 |
+
Params for bucket 15 (41984000 elements, 41984000 padded size):
|
| 749 |
+
module.decoder.layers.2.pre_mlp_layernorm.weight
|
| 750 |
+
module.decoder.layers.2.self_attention.linear_qkv.weight
|
| 751 |
+
module.decoder.layers.2.mlp.router.weight
|
| 752 |
+
module.decoder.layers.2.self_attention.linear_proj.weight
|
| 753 |
+
module.decoder.layers.2.self_attention.linear_qkv.layer_norm_weight
|
| 754 |
+
Params for bucket 16 (41984000 elements, 41984000 padded size):
|
| 755 |
+
module.decoder.layers.1.mlp.router.weight
|
| 756 |
+
module.decoder.layers.1.self_attention.linear_qkv.layer_norm_weight
|
| 757 |
+
module.decoder.layers.1.self_attention.linear_qkv.weight
|
| 758 |
+
module.decoder.layers.1.self_attention.linear_proj.weight
|
| 759 |
+
module.decoder.layers.1.pre_mlp_layernorm.weight
|
| 760 |
+
Params for bucket 17 (41984000 elements, 41984000 padded size):
|
| 761 |
+
module.decoder.layers.0.mlp.router.weight
|
| 762 |
+
module.decoder.layers.0.self_attention.linear_qkv.layer_norm_weight
|
| 763 |
+
module.decoder.layers.0.self_attention.linear_qkv.weight
|
| 764 |
+
module.decoder.layers.0.pre_mlp_layernorm.weight
|
| 765 |
+
module.decoder.layers.0.self_attention.linear_proj.weight
|
| 766 |
+
Params for bucket 18 (206045184 elements, 206045184 padded size):
|
| 767 |
+
module.embedding.word_embeddings.weight
|
| 768 |
+
INFO:megatron.core.distributed.param_and_grad_buffer:Number of buckets for gradient all-reduce / reduce-scatter: 7
|
| 769 |
+
Params for bucket 1 (44040192 elements, 44040192 padded size):
|
| 770 |
+
module.decoder.layers.15.mlp.experts.linear_fc2.weight0
|
| 771 |
+
module.decoder.layers.14.mlp.experts.linear_fc1.weight0
|
| 772 |
+
module.decoder.layers.13.mlp.experts.linear_fc2.weight0
|
| 773 |
+
module.decoder.layers.15.mlp.experts.linear_fc1.weight0
|
| 774 |
+
module.decoder.layers.14.mlp.experts.linear_fc2.weight0
|
| 775 |
+
Params for bucket 2 (50331648 elements, 50331648 padded size):
|
| 776 |
+
module.decoder.layers.12.mlp.experts.linear_fc2.weight0
|
| 777 |
+
module.decoder.layers.11.mlp.experts.linear_fc2.weight0
|
| 778 |
+
module.decoder.layers.13.mlp.experts.linear_fc1.weight0
|
| 779 |
+
module.decoder.layers.11.mlp.experts.linear_fc1.weight0
|
| 780 |
+
module.decoder.layers.12.mlp.experts.linear_fc1.weight0
|
| 781 |
+
Params for bucket 3 (44040192 elements, 44040192 padded size):
|
| 782 |
+
module.decoder.layers.10.mlp.experts.linear_fc2.weight0
|
| 783 |
+
module.decoder.layers.9.mlp.experts.linear_fc1.weight0
|
| 784 |
+
module.decoder.layers.9.mlp.experts.linear_fc2.weight0
|
| 785 |
+
module.decoder.layers.8.mlp.experts.linear_fc2.weight0
|
| 786 |
+
module.decoder.layers.10.mlp.experts.linear_fc1.weight0
|
| 787 |
+
Params for bucket 4 (50331648 elements, 50331648 padded size):
|
| 788 |
+
module.decoder.layers.8.mlp.experts.linear_fc1.weight0
|
| 789 |
+
module.decoder.layers.6.mlp.experts.linear_fc1.weight0
|
| 790 |
+
module.decoder.layers.7.mlp.experts.linear_fc1.weight0
|
| 791 |
+
module.decoder.layers.7.mlp.experts.linear_fc2.weight0
|
| 792 |
+
module.decoder.layers.6.mlp.experts.linear_fc2.weight0
|
| 793 |
+
Params for bucket 5 (44040192 elements, 44040192 padded size):
|
| 794 |
+
module.decoder.layers.5.mlp.experts.linear_fc1.weight0
|
| 795 |
+
module.decoder.layers.4.mlp.experts.linear_fc1.weight0
|
| 796 |
+
module.decoder.layers.3.mlp.experts.linear_fc2.weight0
|
| 797 |
+
module.decoder.layers.5.mlp.experts.linear_fc2.weight0
|
| 798 |
+
module.decoder.layers.4.mlp.experts.linear_fc2.weight0
|
| 799 |
+
Params for bucket 6 (50331648 elements, 50331648 padded size):
|
| 800 |
+
module.decoder.layers.1.mlp.experts.linear_fc1.weight0
|
| 801 |
+
module.decoder.layers.2.mlp.experts.linear_fc2.weight0
|
| 802 |
+
module.decoder.layers.1.mlp.experts.linear_fc2.weight0
|
| 803 |
+
module.decoder.layers.3.mlp.experts.linear_fc1.weight0
|
| 804 |
+
module.decoder.layers.2.mlp.experts.linear_fc1.weight0
|
| 805 |
+
Params for bucket 7 (18874368 elements, 18874368 padded size):
|
| 806 |
+
module.decoder.layers.0.mlp.experts.linear_fc2.weight0
|
| 807 |
+
module.decoder.layers.0.mlp.experts.linear_fc1.weight0
|
| 808 |
+
INFO:megatron.core.optimizer:Setting up optimizer with config OptimizerConfig(optimizer='adam', lr=0.0001, min_lr=1e-05, decoupled_lr=None, decoupled_min_lr=None, weight_decay=0.1, fp8_recipe='mxfp8', fp16=False, bf16=True, reuse_grad_buf_for_mxfp8_param_ag=False, params_dtype=torch.bfloat16, use_precision_aware_optimizer=False, store_param_remainders=True, main_grads_dtype=torch.float32, main_params_dtype=torch.float32, exp_avg_dtype=torch.float32, exp_avg_sq_dtype=torch.float32, loss_scale=None, initial_loss_scale=4294967296, min_loss_scale=1.0, loss_scale_window=1000, hysteresis=2, adam_beta1=0.9, adam_beta2=0.999, adam_eps=1e-08, sgd_momentum=0.9, use_distributed_optimizer=True, overlap_param_gather_with_optimizer_step=False, optimizer_cpu_offload=False, optimizer_offload_fraction=1.0, use_torch_optimizer_for_cpu_offload=False, overlap_cpu_optimizer_d2h_h2d=False, pin_cpu_grads=True, pin_cpu_params=True, clip_grad=1.0, log_num_zeros_in_grad=False, barrier_with_L1_time=True, timers=<megatron.core.timers.Timers object at 0x712f25374ec0>, config_logger_dir='')
|
| 809 |
+
INFO:megatron.core.optimizer_param_scheduler:> learning rate decay style: cosine
|
| 810 |
+
[after model, optimizer, and learning rate scheduler are built] datetime: 2026-08-30 07:21:25
|
| 811 |
+
> building train, validation, and test datasets ...
|
| 812 |
+
> datasets target sizes (minimum size):
|
| 813 |
+
train: 192
|
| 814 |
+
validation: 0
|
| 815 |
+
test: 0
|
| 816 |
+
INFO:megatron.core.datasets.blended_megatron_dataset_config:Let split_matrix = [(0, 1.0), None, None]
|
| 817 |
+
> building train, validation, and test datasets for GPT ...
|
| 818 |
+
INFO:megatron.core.datasets.blended_megatron_dataset_builder:Building GPTDataset splits with sizes=(192, 0, 0) and config=GPTDatasetConfig(random_seed=1234, sequence_length=8192, blend=(['/workspace/data/wikitext/wikitext103_gpt2_text_document'], None), blend_per_split=None, split='100,0,0', split_matrix=[(0, 1.0), None, None], num_dataset_builder_threads=1, path_to_cache=None, mmap_bin_files=True, mock=False, tokenizer=<megatron.training.tokenizer.tokenizer._GPT2BPETokenizer object at 0x712f254fbe90>, mid_level_dataset_surplus=0.005, reset_position_ids=False, reset_attention_mask=False, eod_mask_loss=False, create_attention_mask=True, drop_last_partial_validation_sequence=True, add_extra_token_to_sequence=True, object_storage_cache_path=None)
|
| 819 |
+
INFO:megatron.core.datasets.indexed_dataset:Load the _IndexReader from /workspace/data/wikitext/wikitext103_gpt2_text_document.idx
|
| 820 |
+
INFO:megatron.core.datasets.indexed_dataset: Extract the sequence lengths
|
| 821 |
+
INFO:megatron.core.datasets.indexed_dataset: Extract the sequence pointers
|
| 822 |
+
INFO:megatron.core.datasets.indexed_dataset: Extract the document indices
|
| 823 |
+
INFO:megatron.core.datasets.indexed_dataset:> total number of sequences: 1165029
|
| 824 |
+
INFO:megatron.core.datasets.indexed_dataset:> total number of documents: 1801350
|
| 825 |
+
INFO:megatron.core.datasets.gpt_dataset:Load the GPTDataset train indices
|
| 826 |
+
INFO:megatron.core.datasets.gpt_dataset: Load the document index from b16a908ae7f33ed67d5ec87f6f633afe-GPTDataset-train-document_index.npy
|
| 827 |
+
INFO:megatron.core.datasets.gpt_dataset: Load the sample index from b16a908ae7f33ed67d5ec87f6f633afe-GPTDataset-train-sample_index.npy
|
| 828 |
+
INFO:megatron.core.datasets.gpt_dataset: Load the shuffle index from b16a908ae7f33ed67d5ec87f6f633afe-GPTDataset-train-shuffle_index.npy
|
| 829 |
+
INFO:megatron.core.datasets.gpt_dataset:> total number of samples: 14536
|
| 830 |
+
> finished creating GPT datasets ...
|
| 831 |
+
[after dataloaders are built] datetime: 2026-08-30 07:21:25
|
| 832 |
+
done with setup ...
|
| 833 |
+
(min, max) time across ranks (ms):
|
| 834 |
+
model-and-optimizer-setup ......................: (3397.04, 3397.25)
|
| 835 |
+
train/valid/test-data-iterators-setup ..........: (71.49, 91.87)
|
| 836 |
+
training ...
|
| 837 |
+
Overwriting rerun_state_machine.current_iteration from -1 to 0...
|
| 838 |
+
[before the start of training step] datetime: 2026-08-30 07:21:25
|
| 839 |
+
/workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:334: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
|
| 840 |
+
warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
|
| 841 |
+
/workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:334: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
|
| 842 |
+
warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
|
| 843 |
+
/workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:334: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
|
| 844 |
+
warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
|
| 845 |
+
/workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:334: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
|
| 846 |
+
warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
|
| 847 |
+
/workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:336: UserWarning: The use_host_meta is deprecated, it will be removed in the future.
|
| 848 |
+
warnings.warn("The use_host_meta is deprecated, it will be removed in the future.")
|
| 849 |
+
/workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:336: UserWarning: The use_host_meta is deprecated, it will be removed in the future.
|
| 850 |
+
warnings.warn("The use_host_meta is deprecated, it will be removed in the future.")
|
| 851 |
+
/workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:334: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
|
| 852 |
+
warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
|
| 853 |
+
/workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:334: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
|
| 854 |
+
warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
|
| 855 |
+
/workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:336: UserWarning: The use_host_meta is deprecated, it will be removed in the future.
|
| 856 |
+
warnings.warn("The use_host_meta is deprecated, it will be removed in the future.")
|
| 857 |
+
/workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:336: UserWarning: The use_host_meta is deprecated, it will be removed in the future.
|
| 858 |
+
warnings.warn("The use_host_meta is deprecated, it will be removed in the future.")
|
| 859 |
+
/workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:334: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
|
| 860 |
+
warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
|
| 861 |
+
/workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:336: UserWarning: The use_host_meta is deprecated, it will be removed in the future.
|
| 862 |
+
warnings.warn("The use_host_meta is deprecated, it will be removed in the future.")
|
| 863 |
+
/workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:336: UserWarning: The use_host_meta is deprecated, it will be removed in the future.
|
| 864 |
+
warnings.warn("The use_host_meta is deprecated, it will be removed in the future.")
|
| 865 |
+
/workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:336: UserWarning: The use_host_meta is deprecated, it will be removed in the future.
|
| 866 |
+
warnings.warn("The use_host_meta is deprecated, it will be removed in the future.")
|
| 867 |
+
/workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:334: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
|
| 868 |
+
warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
|
| 869 |
+
/workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:336: UserWarning: The use_host_meta is deprecated, it will be removed in the future.
|
| 870 |
+
warnings.warn("The use_host_meta is deprecated, it will be removed in the future.")
|
| 871 |
+
/workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:482: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
|
| 872 |
+
warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
|
| 873 |
+
/workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:482: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
|
| 874 |
+
warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
|
| 875 |
+
/workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:482: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
|
| 876 |
+
warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
|
| 877 |
+
/workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:482: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
|
| 878 |
+
warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
|
| 879 |
+
/workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:482: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
|
| 880 |
+
warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
|
| 881 |
+
/workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:482: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
|
| 882 |
+
warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
|
| 883 |
+
/workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:482: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
|
| 884 |
+
warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
|
| 885 |
+
/workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:482: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
|
| 886 |
+
warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
|
| 887 |
+
/workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:334: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
|
| 888 |
+
warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
|
| 889 |
+
/workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:334: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
|
| 890 |
+
warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
|
| 891 |
+
/workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:334: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
|
| 892 |
+
warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
|
| 893 |
+
/workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:336: UserWarning: The use_host_meta is deprecated, it will be removed in the future.
|
| 894 |
+
warnings.warn("The use_host_meta is deprecated, it will be removed in the future.")
|
| 895 |
+
/workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:336: UserWarning: The use_host_meta is deprecated, it will be removed in the future.
|
| 896 |
+
warnings.warn("The use_host_meta is deprecated, it will be removed in the future.")
|
| 897 |
+
/workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:336: UserWarning: The use_host_meta is deprecated, it will be removed in the future.
|
| 898 |
+
warnings.warn("The use_host_meta is deprecated, it will be removed in the future.")
|
| 899 |
+
/workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:334: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
|
| 900 |
+
warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
|
| 901 |
+
/workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:336: UserWarning: The use_host_meta is deprecated, it will be removed in the future.
|
| 902 |
+
warnings.warn("The use_host_meta is deprecated, it will be removed in the future.")
|
| 903 |
+
/workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:334: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
|
| 904 |
+
warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
|
| 905 |
+
/workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:336: UserWarning: The use_host_meta is deprecated, it will be removed in the future.
|
| 906 |
+
warnings.warn("The use_host_meta is deprecated, it will be removed in the future.")
|
| 907 |
+
/workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:334: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
|
| 908 |
+
warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
|
| 909 |
+
/workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:336: UserWarning: The use_host_meta is deprecated, it will be removed in the future.
|
| 910 |
+
warnings.warn("The use_host_meta is deprecated, it will be removed in the future.")
|
| 911 |
+
/workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:334: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
|
| 912 |
+
warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
|
| 913 |
+
/workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:336: UserWarning: The use_host_meta is deprecated, it will be removed in the future.
|
| 914 |
+
warnings.warn("The use_host_meta is deprecated, it will be removed in the future.")
|
| 915 |
+
/workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:334: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
|
| 916 |
+
warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
|
| 917 |
+
/workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:336: UserWarning: The use_host_meta is deprecated, it will be removed in the future.
|
| 918 |
+
warnings.warn("The use_host_meta is deprecated, it will be removed in the future.")
|
| 919 |
+
/workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:482: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
|
| 920 |
+
warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
|
| 921 |
+
/workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:482: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
|
| 922 |
+
warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
|
| 923 |
+
/workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:482: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
|
| 924 |
+
warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
|
| 925 |
+
/workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:482: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
|
| 926 |
+
warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
|
| 927 |
+
/workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:482: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
|
| 928 |
+
warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
|
| 929 |
+
/workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:482: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
|
| 930 |
+
warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
|
| 931 |
+
/workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:482: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
|
| 932 |
+
warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
|
| 933 |
+
/workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:482: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
|
| 934 |
+
warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
|
| 935 |
+
/workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:334: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
|
| 936 |
+
warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
|
| 937 |
+
/workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:334: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
|
| 938 |
+
warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
|
| 939 |
+
/workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:334: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
|
| 940 |
+
warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
|
| 941 |
+
/workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:334: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
|
| 942 |
+
warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
|
| 943 |
+
/workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:334: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
|
| 944 |
+
warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
|
| 945 |
+
/workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:334: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
|
| 946 |
+
warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
|
| 947 |
+
/workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:334: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
|
| 948 |
+
warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
|
| 949 |
+
/workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:334: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
|
| 950 |
+
warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
|
| 951 |
+
/workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:336: UserWarning: The use_host_meta is deprecated, it will be removed in the future.
|
| 952 |
+
warnings.warn("The use_host_meta is deprecated, it will be removed in the future.")
|
| 953 |
+
/workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:336: UserWarning: The use_host_meta is deprecated, it will be removed in the future.
|
| 954 |
+
warnings.warn("The use_host_meta is deprecated, it will be removed in the future.")
|
| 955 |
+
/workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:336: UserWarning: The use_host_meta is deprecated, it will be removed in the future.
|
| 956 |
+
warnings.warn("The use_host_meta is deprecated, it will be removed in the future.")
|
| 957 |
+
/workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:336: UserWarning: The use_host_meta is deprecated, it will be removed in the future.
|
| 958 |
+
warnings.warn("The use_host_meta is deprecated, it will be removed in the future.")
|
| 959 |
+
/workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:336: UserWarning: The use_host_meta is deprecated, it will be removed in the future.
|
| 960 |
+
warnings.warn("The use_host_meta is deprecated, it will be removed in the future.")
|
| 961 |
+
/workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:336: UserWarning: The use_host_meta is deprecated, it will be removed in the future.
|
| 962 |
+
warnings.warn("The use_host_meta is deprecated, it will be removed in the future.")
|
| 963 |
+
/workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:336: UserWarning: The use_host_meta is deprecated, it will be removed in the future.
|
| 964 |
+
warnings.warn("The use_host_meta is deprecated, it will be removed in the future.")
|
| 965 |
+
/workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:336: UserWarning: The use_host_meta is deprecated, it will be removed in the future.
|
| 966 |
+
warnings.warn("The use_host_meta is deprecated, it will be removed in the future.")
|
| 967 |
+
[2026-08-30 07:21:51] iteration 1/ 12 | consumed samples: 16 | elapsed time per iteration (ms): 25963.1 | throughput per GPU (TFLOP/s/GPU): 7.6 | learning rate: 2.000000E-05 | global batch size: 16 | lm loss: 1.103051E+01 | loss scale: 1.0 | grad norm: 50.241 | number of skipped iterations: 0 | number of nan iterations: 0 | peak recv tokens: 28416 | min safe capacity: 1.7344 |
|
| 968 |
+
Number of parameters in transformer block in billions: 3.09
|
| 969 |
+
Number of parameters in embedding layers in billions: 0.41
|
| 970 |
+
Total number of parameters in billions: 3.50
|
| 971 |
+
Number of parameters in most loaded shard in billions: 3.4994
|
| 972 |
+
Theoretical memory footprints: weight and optimizer=25029.43 MB
|
| 973 |
+
[Rank 0] (after 1 iterations) memory (MB) | allocated: 14963.09814453125 | max allocated: 30454.23095703125 | reserved: 31264.0 | max reserved: 31264.0
|
| 974 |
+
[2026-08-30 07:21:51] iteration 2/ 12 | consumed samples: 32 | elapsed time per iteration (ms): 336.5 | throughput per GPU (TFLOP/s/GPU): 589.5 | learning rate: 4.000000E-05 | global batch size: 16 | lm loss: 1.101006E+01 | loss scale: 1.0 | grad norm: 51.122 | number of skipped iterations: 0 | number of nan iterations: 0 | peak recv tokens: 28416 | min safe capacity: 1.7344 |
|
| 975 |
+
[2026-08-30 07:21:51] iteration 3/ 12 | consumed samples: 48 | elapsed time per iteration (ms): 329.4 | throughput per GPU (TFLOP/s/GPU): 602.3 | learning rate: 6.000000E-05 | global batch size: 16 | lm loss: 1.053811E+01 | loss scale: 1.0 | grad norm: 249.931 | number of skipped iterations: 0 | number of nan iterations: 0 | peak recv tokens: 28416 | min safe capacity: 1.7344 |
|
| 976 |
+
Capture CUDA graph for training!!!
|
| 977 |
+
CUDA graph capture done!!!
|
| 978 |
+
[2026-08-30 07:21:53] iteration 4/ 12 | consumed samples: 64 | elapsed time per iteration (ms): 1232.4 | throughput per GPU (TFLOP/s/GPU): 161.0 | learning rate: 8.000000E-05 | global batch size: 16 | lm loss: 1.319993E+01 | loss scale: 1.0 | grad norm: 35.708 | number of skipped iterations: 0 | number of nan iterations: 0 | peak recv tokens: 28416 | min safe capacity: 1.7344 |
|
| 979 |
+
[2026-08-30 07:21:53] iteration 5/ 12 | consumed samples: 80 | elapsed time per iteration (ms): 293.2 | throughput per GPU (TFLOP/s/GPU): 676.7 | learning rate: 1.000000E-04 | global batch size: 16 | lm loss: 1.052083E+01 | loss scale: 1.0 | grad norm: 17.493 | number of skipped iterations: 0 | number of nan iterations: 0 | peak recv tokens: 28416 | min safe capacity: 1.7344 |
|
| 980 |
+
[2026-08-30 07:21:53] iteration 6/ 12 | consumed samples: 96 | elapsed time per iteration (ms): 292.3 | throughput per GPU (TFLOP/s/GPU): 678.6 | learning rate: 7.750000E-05 | global batch size: 16 | lm loss: 1.027140E+01 | loss scale: 1.0 | grad norm: 9.648 | number of skipped iterations: 0 | number of nan iterations: 0 | peak recv tokens: 28416 | min safe capacity: 1.7344 |
|
| 981 |
+
[2026-08-30 07:21:53] iteration 7/ 12 | consumed samples: 112 | elapsed time per iteration (ms): 290.2 | throughput per GPU (TFLOP/s/GPU): 683.6 | learning rate: 3.250000E-05 | global batch size: 16 | lm loss: 9.466166E+00 | loss scale: 1.0 | grad norm: 7.620 | number of skipped iterations: 0 | number of nan iterations: 0 | peak recv tokens: 28416 | min safe capacity: 1.7344 |
|
| 982 |
+
[2026-08-30 07:21:54] iteration 8/ 12 | consumed samples: 128 | elapsed time per iteration (ms): 290.4 | throughput per GPU (TFLOP/s/GPU): 683.1 | learning rate: 1.000000E-05 | global batch size: 16 | lm loss: 8.752878E+00 | loss scale: 1.0 | grad norm: 6.063 | number of skipped iterations: 0 | number of nan iterations: 0 | peak recv tokens: 28416 | min safe capacity: 1.7344 |
|
| 983 |
+
[2026-08-30 07:21:54] iteration 9/ 12 | consumed samples: 144 | elapsed time per iteration (ms): 294.4 | throughput per GPU (TFLOP/s/GPU): 673.9 | learning rate: 1.000000E-05 | global batch size: 16 | lm loss: 8.545376E+00 | loss scale: 1.0 | grad norm: 7.113 | number of skipped iterations: 0 | number of nan iterations: 0 | peak recv tokens: 28416 | min safe capacity: 1.7344 |
|
| 984 |
+
[2026-08-30 07:21:54] iteration 10/ 12 | consumed samples: 160 | elapsed time per iteration (ms): 292.4 | throughput per GPU (TFLOP/s/GPU): 678.5 | learning rate: 1.000000E-05 | global batch size: 16 | lm loss: 8.451232E+00 | loss scale: 1.0 | grad norm: 6.175 | number of skipped iterations: 0 | number of nan iterations: 0 | peak recv tokens: 28416 | min safe capacity: 1.7344 |
|
| 985 |
+
[2026-08-30 07:21:55] iteration 11/ 12 | consumed samples: 176 | elapsed time per iteration (ms): 294.5 | throughput per GPU (TFLOP/s/GPU): 673.6 | learning rate: 1.000000E-05 | global batch size: 16 | lm loss: 8.375832E+00 | loss scale: 1.0 | grad norm: 4.656 | number of skipped iterations: 0 | number of nan iterations: 0 | peak recv tokens: 28416 | min safe capacity: 1.7344 |
|
| 986 |
+
[2026-08-30 07:21:55] iteration 12/ 12 | consumed samples: 192 | elapsed time per iteration (ms): 294.3 | throughput per GPU (TFLOP/s/GPU): 674.0 | learning rate: 1.000000E-05 | global batch size: 16 | lm loss: 8.273790E+00 | loss scale: 1.0 | grad norm: 4.113 | number of skipped iterations: 0 | number of nan iterations: 0 | peak recv tokens: 28416 | min safe capacity: 1.7344 |
|
| 987 |
+
[after training is done] datetime: 2026-08-30 07:21:55
|
| 988 |
+
[rank0]:[W830 07:21:56.857927972 ProcessGroupNCCL.cpp:1505] Warning: WARNING: destroy_process_group() was not called before program exit, which can leak resources. For more info, please see https://pytorch.org/docs/stable/distributed.html#shutdown (function operator())
|
| 989 |
+
W0830 07:21:56.682000 744857 torch/distributed/elastic/agent/server/api.py:719] Received 15 death signal, shutting down workers
|
| 990 |
+
W0830 07:21:56.683000 744857 torch/distributed/elastic/multiprocessing/api.py:900] Sending process 744945 closing signal SIGTERM
|
| 991 |
+
W0830 07:21:56.683000 744857 torch/distributed/elastic/multiprocessing/api.py:900] Sending process 744946 closing signal SIGTERM
|
| 992 |
+
W0830 07:21:56.685000 744857 torch/distributed/elastic/multiprocessing/api.py:900] Sending process 744947 closing signal SIGTERM
|
| 993 |
+
W0830 07:21:56.687000 744857 torch/distributed/elastic/multiprocessing/api.py:900] Sending process 744948 closing signal SIGTERM
|
| 994 |
+
W0830 07:21:56.688000 744857 torch/distributed/elastic/multiprocessing/api.py:900] Sending process 744949 closing signal SIGTERM
|
| 995 |
+
W0830 07:21:56.691000 744857 torch/distributed/elastic/multiprocessing/api.py:900] Sending process 744950 closing signal SIGTERM
|
| 996 |
+
W0830 07:21:56.692000 744857 torch/distributed/elastic/multiprocessing/api.py:900] Sending process 744951 closing signal SIGTERM
|
| 997 |
+
W0830 07:21:56.693000 744857 torch/distributed/elastic/multiprocessing/api.py:900] Sending process 744952 closing signal SIGTERM
|
| 998 |
+
Traceback (most recent call last):
|
| 999 |
+
File "/usr/local/bin/torchrun", line 8, in <module>
|
| 1000 |
+
sys.exit(main())
|
| 1001 |
+
^^^^^^
|
| 1002 |
+
File "/usr/local/lib/python3.12/dist-packages/torch/distributed/elastic/multiprocessing/errors/__init__.py", line 355, in wrapper
|
| 1003 |
+
return f(*args, **kwargs)
|
| 1004 |
+
^^^^^^^^^^^^^^^^^^
|
| 1005 |
+
File "/usr/local/lib/python3.12/dist-packages/torch/distributed/run.py", line 892, in main
|
| 1006 |
+
run(args)
|
| 1007 |
+
File "/usr/local/lib/python3.12/dist-packages/torch/distributed/run.py", line 883, in run
|
| 1008 |
+
elastic_launch(
|
| 1009 |
+
File "/usr/local/lib/python3.12/dist-packages/torch/distributed/launcher/api.py", line 139, in __call__
|
| 1010 |
+
return launch_agent(self._config, self._entrypoint, list(args))
|
| 1011 |
+
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
| 1012 |
+
File "/usr/local/lib/python3.12/dist-packages/torch/distributed/launcher/api.py", line 261, in launch_agent
|
| 1013 |
+
result = agent.run()
|
| 1014 |
+
^^^^^^^^^^^
|
| 1015 |
+
File "/usr/local/lib/python3.12/dist-packages/torch/distributed/elastic/metrics/api.py", line 138, in wrapper
|
| 1016 |
+
result = f(*args, **kwargs)
|
| 1017 |
+
^^^^^^^^^^^^^^^^^^
|
| 1018 |
+
File "/usr/local/lib/python3.12/dist-packages/torch/distributed/elastic/agent/server/api.py", line 711, in run
|
| 1019 |
+
result = self._invoke_run(role)
|
| 1020 |
+
^^^^^^^^^^^^^^^^^^^^^^
|
| 1021 |
+
File "/usr/local/lib/python3.12/dist-packages/torch/distributed/elastic/agent/server/api.py", line 870, in _invoke_run
|
| 1022 |
+
time.sleep(monitor_interval)
|
| 1023 |
+
File "/usr/local/lib/python3.12/dist-packages/torch/distributed/elastic/multiprocessing/api.py", line 84, in _terminate_process_handler
|
| 1024 |
+
raise SignalException(f"Process {os.getpid()} got signal: {sigval}", sigval=sigval)
|
| 1025 |
+
torch.distributed.elastic.multiprocessing.api.SignalException: Process 744857 got signal: 15
|
tiera-rerun-fixed-moonep/ultraep/ponly/ponly_gbs16/results.tsv
CHANGED
|
@@ -3,3 +3,5 @@ PONLY ultraep one 1 4096 16 2.40625 none 1 OK 176.80 175.80 38329 /workspace/tie
|
|
| 3 |
PONLY ultraep one 1 4096 16 2.40625 none 2 OK 176.80 175.80 38329 /workspace/tiera-rerun-fixed-moonep/ultraep/ponly/ponly_gbs16/PONLY__ultraep__one__seq4096__gbs16__cap2p40625/rep2
|
| 4 |
PONLY ultraep one 1 4096 16 2.40625 none 3 OK 176.76 175.80 38331 /workspace/tiera-rerun-fixed-moonep/ultraep/ponly/ponly_gbs16/PONLY__ultraep__one__seq4096__gbs16__cap2p40625/rep3
|
| 5 |
PONLY ultraep one 1 8192 16 2.40625 none 1 OK 292.76 290.70 51017 /workspace/tiera-rerun-fixed-moonep/ultraep/ponly/ponly_gbs16/PONLY__ultraep__one__seq8192__gbs16__cap2p40625/rep1
|
|
|
|
|
|
|
|
|
| 3 |
PONLY ultraep one 1 4096 16 2.40625 none 2 OK 176.80 175.80 38329 /workspace/tiera-rerun-fixed-moonep/ultraep/ponly/ponly_gbs16/PONLY__ultraep__one__seq4096__gbs16__cap2p40625/rep2
|
| 4 |
PONLY ultraep one 1 4096 16 2.40625 none 3 OK 176.76 175.80 38331 /workspace/tiera-rerun-fixed-moonep/ultraep/ponly/ponly_gbs16/PONLY__ultraep__one__seq4096__gbs16__cap2p40625/rep3
|
| 5 |
PONLY ultraep one 1 8192 16 2.40625 none 1 OK 292.76 290.70 51017 /workspace/tiera-rerun-fixed-moonep/ultraep/ponly/ponly_gbs16/PONLY__ultraep__one__seq8192__gbs16__cap2p40625/rep1
|
| 6 |
+
PONLY ultraep one 1 8192 16 2.40625 none 2 OK 292.61 290.70 51015 /workspace/tiera-rerun-fixed-moonep/ultraep/ponly/ponly_gbs16/PONLY__ultraep__one__seq8192__gbs16__cap2p40625/rep2
|
| 7 |
+
PONLY ultraep one 1 8192 16 2.40625 none 3 OK 292.62 290.20 51013 /workspace/tiera-rerun-fixed-moonep/ultraep/ponly/ponly_gbs16/PONLY__ultraep__one__seq8192__gbs16__cap2p40625/rep3
|