granularity results tiera-rerun-fixed-moonep/moonep/a1 @ 2026-08-30T07:41:06Z from sp-aiinfra-interactive-aghosh-7jtjl-8xb200-moe-cg-0
Browse files- tiera-rerun-fixed-moonep/moonep/a1/a1_oneslot_gbs16/G1__moonep__one__seq4096__gbs16__capnone__async/rep3/mem_samples.csv +40 -0
- tiera-rerun-fixed-moonep/moonep/a1/a1_oneslot_gbs16/G1__moonep__one__seq4096__gbs16__capnone__async/rep3/train.log +47 -0
- tiera-rerun-fixed-moonep/moonep/a1/a1_oneslot_gbs16/results.tsv +1 -0
tiera-rerun-fixed-moonep/moonep/a1/a1_oneslot_gbs16/G1__moonep__one__seq4096__gbs16__capnone__async/rep3/mem_samples.csv
CHANGED
|
@@ -174,3 +174,43 @@
|
|
| 174 |
5, 96843
|
| 175 |
6, 96843
|
| 176 |
7, 96843
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 174 |
5, 96843
|
| 175 |
6, 96843
|
| 176 |
7, 96843
|
| 177 |
+
0, 96849
|
| 178 |
+
1, 96843
|
| 179 |
+
2, 96843
|
| 180 |
+
3, 96843
|
| 181 |
+
4, 96843
|
| 182 |
+
5, 96843
|
| 183 |
+
6, 96843
|
| 184 |
+
7, 96843
|
| 185 |
+
0, 95553
|
| 186 |
+
1, 96197
|
| 187 |
+
2, 95867
|
| 188 |
+
3, 95877
|
| 189 |
+
4, 96197
|
| 190 |
+
5, 95867
|
| 191 |
+
6, 96197
|
| 192 |
+
7, 95803
|
| 193 |
+
0, 94067
|
| 194 |
+
1, 94449
|
| 195 |
+
2, 94061
|
| 196 |
+
3, 94449
|
| 197 |
+
4, 94449
|
| 198 |
+
5, 94449
|
| 199 |
+
6, 94449
|
| 200 |
+
7, 94061
|
| 201 |
+
0, 4
|
| 202 |
+
1, 91851
|
| 203 |
+
2, 4
|
| 204 |
+
3, 4
|
| 205 |
+
4, 91565
|
| 206 |
+
5, 75245
|
| 207 |
+
6, 91755
|
| 208 |
+
7, 78605
|
| 209 |
+
0, 0
|
| 210 |
+
1, 0
|
| 211 |
+
2, 0
|
| 212 |
+
3, 0
|
| 213 |
+
4, 0
|
| 214 |
+
5, 0
|
| 215 |
+
6, 0
|
| 216 |
+
7, 0
|
tiera-rerun-fixed-moonep/moonep/a1/a1_oneslot_gbs16/G1__moonep__one__seq4096__gbs16__capnone__async/rep3/train.log
CHANGED
|
@@ -865,3 +865,50 @@ CUDA graph capture done!!!
|
|
| 865 |
[2026-08-30 07:40:52] iteration 6/ 12 | consumed samples: 96 | elapsed time per iteration (ms): 498.2 | throughput per GPU (TFLOP/s/GPU): 669.2 | learning rate: 7.750000E-05 | global batch size: 16 | lm loss: 1.014399E+01 | loss scale: 1.0 | grad norm: 43.370 | number of skipped iterations: 0 | number of nan iterations: 0 |
|
| 866 |
[2026-08-30 07:40:52] iteration 7/ 12 | consumed samples: 112 | elapsed time per iteration (ms): 485.1 | throughput per GPU (TFLOP/s/GPU): 687.2 | learning rate: 3.250000E-05 | global batch size: 16 | lm loss: 9.904866E+00 | loss scale: 1.0 | grad norm: 8.280 | number of skipped iterations: 0 | number of nan iterations: 0 |
|
| 867 |
[2026-08-30 07:40:53] iteration 8/ 12 | consumed samples: 128 | elapsed time per iteration (ms): 487.6 | throughput per GPU (TFLOP/s/GPU): 683.7 | learning rate: 1.000000E-05 | global batch size: 16 | lm loss: 8.981642E+00 | loss scale: 1.0 | grad norm: 7.258 | number of skipped iterations: 0 | number of nan iterations: 0 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 865 |
[2026-08-30 07:40:52] iteration 6/ 12 | consumed samples: 96 | elapsed time per iteration (ms): 498.2 | throughput per GPU (TFLOP/s/GPU): 669.2 | learning rate: 7.750000E-05 | global batch size: 16 | lm loss: 1.014399E+01 | loss scale: 1.0 | grad norm: 43.370 | number of skipped iterations: 0 | number of nan iterations: 0 |
|
| 866 |
[2026-08-30 07:40:52] iteration 7/ 12 | consumed samples: 112 | elapsed time per iteration (ms): 485.1 | throughput per GPU (TFLOP/s/GPU): 687.2 | learning rate: 3.250000E-05 | global batch size: 16 | lm loss: 9.904866E+00 | loss scale: 1.0 | grad norm: 8.280 | number of skipped iterations: 0 | number of nan iterations: 0 |
|
| 867 |
[2026-08-30 07:40:53] iteration 8/ 12 | consumed samples: 128 | elapsed time per iteration (ms): 487.6 | throughput per GPU (TFLOP/s/GPU): 683.7 | learning rate: 1.000000E-05 | global batch size: 16 | lm loss: 8.981642E+00 | loss scale: 1.0 | grad norm: 7.258 | number of skipped iterations: 0 | number of nan iterations: 0 |
|
| 868 |
+
[2026-08-30 07:40:53] iteration 9/ 12 | consumed samples: 144 | elapsed time per iteration (ms): 469.0 | throughput per GPU (TFLOP/s/GPU): 710.9 | learning rate: 1.000000E-05 | global batch size: 16 | lm loss: 8.626698E+00 | loss scale: 1.0 | grad norm: 6.470 | number of skipped iterations: 0 | number of nan iterations: 0 |
|
| 869 |
+
[2026-08-30 07:40:54] iteration 10/ 12 | consumed samples: 160 | elapsed time per iteration (ms): 476.7 | throughput per GPU (TFLOP/s/GPU): 699.3 | learning rate: 1.000000E-05 | global batch size: 16 | lm loss: 8.429865E+00 | loss scale: 1.0 | grad norm: 5.282 | number of skipped iterations: 0 | number of nan iterations: 0 |
|
| 870 |
+
[2026-08-30 07:40:54] iteration 11/ 12 | consumed samples: 176 | elapsed time per iteration (ms): 462.2 | throughput per GPU (TFLOP/s/GPU): 721.3 | learning rate: 1.000000E-05 | global batch size: 16 | lm loss: 8.372684E+00 | loss scale: 1.0 | grad norm: 4.866 | number of skipped iterations: 0 | number of nan iterations: 0 |
|
| 871 |
+
[2026-08-30 07:40:55] iteration 12/ 12 | consumed samples: 192 | elapsed time per iteration (ms): 494.0 | throughput per GPU (TFLOP/s/GPU): 674.9 | learning rate: 1.000000E-05 | global batch size: 16 | lm loss: 8.250052E+00 | loss scale: 1.0 | grad norm: 4.802 | number of skipped iterations: 0 | number of nan iterations: 0 |
|
| 872 |
+
[after training is done] datetime: 2026-08-30 07:40:55
|
| 873 |
+
[rank0]:[W830 07:40:56.598775869 ProcessGroupNCCL.cpp:1505] Warning: WARNING: destroy_process_group() was not called before program exit, which can leak resources. For more info, please see https://pytorch.org/docs/stable/distributed.html#shutdown (function operator())
|
| 874 |
+
[rank2]:[W830 07:40:57.647323593 ProcessGroupNCCL.cpp:1505] Warning: WARNING: destroy_process_group() was not called before program exit, which can leak resources. For more info, please see https://pytorch.org/docs/stable/distributed.html#shutdown (function operator())
|
| 875 |
+
[rank7]:[W830 07:40:57.859849150 ProcessGroupNCCL.cpp:1505] Warning: WARNING: destroy_process_group() was not called before program exit, which can leak resources. For more info, please see https://pytorch.org/docs/stable/distributed.html#shutdown (function operator())
|
| 876 |
+
[rank5]:[W830 07:40:57.860248885 ProcessGroupNCCL.cpp:1505] Warning: WARNING: destroy_process_group() was not called before program exit, which can leak resources. For more info, please see https://pytorch.org/docs/stable/distributed.html#shutdown (function operator())
|
| 877 |
+
[rank3]:[W830 07:40:57.094639357 ProcessGroupNCCL.cpp:1505] Warning: WARNING: destroy_process_group() was not called before program exit, which can leak resources. For more info, please see https://pytorch.org/docs/stable/distributed.html#shutdown (function operator())
|
| 878 |
+
[rank1]:[W830 07:40:57.214688954 ProcessGroupNCCL.cpp:1505] Warning: WARNING: destroy_process_group() was not called before program exit, which can leak resources. For more info, please see https://pytorch.org/docs/stable/distributed.html#shutdown (function operator())
|
| 879 |
+
[rank6]:[W830 07:40:58.321615314 ProcessGroupNCCL.cpp:1505] Warning: WARNING: destroy_process_group() was not called before program exit, which can leak resources. For more info, please see https://pytorch.org/docs/stable/distributed.html#shutdown (function operator())
|
| 880 |
+
[rank4]:[W830 07:40:58.342260354 ProcessGroupNCCL.cpp:1505] Warning: WARNING: destroy_process_group() was not called before program exit, which can leak resources. For more info, please see https://pytorch.org/docs/stable/distributed.html#shutdown (function operator())
|
| 881 |
+
W0830 07:41:02.058000 790344 torch/distributed/elastic/agent/server/api.py:719] Received 15 death signal, shutting down workers
|
| 882 |
+
W0830 07:41:02.058000 790344 torch/distributed/elastic/multiprocessing/api.py:900] Sending process 790526 closing signal SIGTERM
|
| 883 |
+
W0830 07:41:02.059000 790344 torch/distributed/elastic/multiprocessing/api.py:900] Sending process 790529 closing signal SIGTERM
|
| 884 |
+
W0830 07:41:02.060000 790344 torch/distributed/elastic/multiprocessing/api.py:900] Sending process 790530 closing signal SIGTERM
|
| 885 |
+
W0830 07:41:02.061000 790344 torch/distributed/elastic/multiprocessing/api.py:900] Sending process 790531 closing signal SIGTERM
|
| 886 |
+
W0830 07:41:02.061000 790344 torch/distributed/elastic/multiprocessing/api.py:900] Sending process 790532 closing signal SIGTERM
|
| 887 |
+
Traceback (most recent call last):
|
| 888 |
+
File "/usr/local/bin/torchrun", line 8, in <module>
|
| 889 |
+
sys.exit(main())
|
| 890 |
+
^^^^^^
|
| 891 |
+
File "/usr/local/lib/python3.12/dist-packages/torch/distributed/elastic/multiprocessing/errors/__init__.py", line 355, in wrapper
|
| 892 |
+
return f(*args, **kwargs)
|
| 893 |
+
^^^^^^^^^^^^^^^^^^
|
| 894 |
+
File "/usr/local/lib/python3.12/dist-packages/torch/distributed/run.py", line 892, in main
|
| 895 |
+
run(args)
|
| 896 |
+
File "/usr/local/lib/python3.12/dist-packages/torch/distributed/run.py", line 883, in run
|
| 897 |
+
elastic_launch(
|
| 898 |
+
File "/usr/local/lib/python3.12/dist-packages/torch/distributed/launcher/api.py", line 139, in __call__
|
| 899 |
+
return launch_agent(self._config, self._entrypoint, list(args))
|
| 900 |
+
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
| 901 |
+
File "/usr/local/lib/python3.12/dist-packages/torch/distributed/launcher/api.py", line 261, in launch_agent
|
| 902 |
+
result = agent.run()
|
| 903 |
+
^^^^^^^^^^^
|
| 904 |
+
File "/usr/local/lib/python3.12/dist-packages/torch/distributed/elastic/metrics/api.py", line 138, in wrapper
|
| 905 |
+
result = f(*args, **kwargs)
|
| 906 |
+
^^^^^^^^^^^^^^^^^^
|
| 907 |
+
File "/usr/local/lib/python3.12/dist-packages/torch/distributed/elastic/agent/server/api.py", line 711, in run
|
| 908 |
+
result = self._invoke_run(role)
|
| 909 |
+
^^^^^^^^^^^^^^^^^^^^^^
|
| 910 |
+
File "/usr/local/lib/python3.12/dist-packages/torch/distributed/elastic/agent/server/api.py", line 870, in _invoke_run
|
| 911 |
+
time.sleep(monitor_interval)
|
| 912 |
+
File "/usr/local/lib/python3.12/dist-packages/torch/distributed/elastic/multiprocessing/api.py", line 84, in _terminate_process_handler
|
| 913 |
+
raise SignalException(f"Process {os.getpid()} got signal: {sigval}", sigval=sigval)
|
| 914 |
+
torch.distributed.elastic.multiprocessing.api.SignalException: Process 790344 got signal: 15
|
tiera-rerun-fixed-moonep/moonep/a1/a1_oneslot_gbs16/results.tsv
CHANGED
|
@@ -1,3 +1,4 @@
|
|
| 1 |
config backend policy slots seq gbs cap recompute rep status iter_ms_mean iter_ms_min peak_mib run_dir
|
| 2 |
G1 moonep one 1 4096 16 none none 1 OK 477.56 460.30 96849 /workspace/tiera-rerun-fixed-moonep/moonep/a1/a1_oneslot_gbs16/G1__moonep__one__seq4096__gbs16__capnone__async/rep1
|
| 3 |
G1 moonep one 1 4096 16 none none 2 OK 479.54 460.30 96849 /workspace/tiera-rerun-fixed-moonep/moonep/a1/a1_oneslot_gbs16/G1__moonep__one__seq4096__gbs16__capnone__async/rep2
|
|
|
|
|
|
| 1 |
config backend policy slots seq gbs cap recompute rep status iter_ms_mean iter_ms_min peak_mib run_dir
|
| 2 |
G1 moonep one 1 4096 16 none none 1 OK 477.56 460.30 96849 /workspace/tiera-rerun-fixed-moonep/moonep/a1/a1_oneslot_gbs16/G1__moonep__one__seq4096__gbs16__capnone__async/rep1
|
| 3 |
G1 moonep one 1 4096 16 none none 2 OK 479.54 460.30 96849 /workspace/tiera-rerun-fixed-moonep/moonep/a1/a1_oneslot_gbs16/G1__moonep__one__seq4096__gbs16__capnone__async/rep2
|
| 4 |
+
G1 moonep one 1 4096 16 none none 3 OK 480.30 460.30 96849 /workspace/tiera-rerun-fixed-moonep/moonep/a1/a1_oneslot_gbs16/G1__moonep__one__seq4096__gbs16__capnone__async/rep3
|