AbhishekGhosh1998 commited on
Commit
0856b1a
·
verified ·
1 Parent(s): cfc1548

granularity results tiera-rerun-fixed-moonep/moonep/a1 @ 2026-08-30T07:41:06Z from sp-aiinfra-interactive-aghosh-7jtjl-8xb200-moe-cg-0

Browse files
tiera-rerun-fixed-moonep/moonep/a1/a1_oneslot_gbs16/G1__moonep__one__seq4096__gbs16__capnone__async/rep3/mem_samples.csv CHANGED
@@ -174,3 +174,43 @@
174
  5, 96843
175
  6, 96843
176
  7, 96843
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
174
  5, 96843
175
  6, 96843
176
  7, 96843
177
+ 0, 96849
178
+ 1, 96843
179
+ 2, 96843
180
+ 3, 96843
181
+ 4, 96843
182
+ 5, 96843
183
+ 6, 96843
184
+ 7, 96843
185
+ 0, 95553
186
+ 1, 96197
187
+ 2, 95867
188
+ 3, 95877
189
+ 4, 96197
190
+ 5, 95867
191
+ 6, 96197
192
+ 7, 95803
193
+ 0, 94067
194
+ 1, 94449
195
+ 2, 94061
196
+ 3, 94449
197
+ 4, 94449
198
+ 5, 94449
199
+ 6, 94449
200
+ 7, 94061
201
+ 0, 4
202
+ 1, 91851
203
+ 2, 4
204
+ 3, 4
205
+ 4, 91565
206
+ 5, 75245
207
+ 6, 91755
208
+ 7, 78605
209
+ 0, 0
210
+ 1, 0
211
+ 2, 0
212
+ 3, 0
213
+ 4, 0
214
+ 5, 0
215
+ 6, 0
216
+ 7, 0
tiera-rerun-fixed-moonep/moonep/a1/a1_oneslot_gbs16/G1__moonep__one__seq4096__gbs16__capnone__async/rep3/train.log CHANGED
@@ -865,3 +865,50 @@ CUDA graph capture done!!!
865
  [2026-08-30 07:40:52] iteration 6/ 12 | consumed samples: 96 | elapsed time per iteration (ms): 498.2 | throughput per GPU (TFLOP/s/GPU): 669.2 | learning rate: 7.750000E-05 | global batch size: 16 | lm loss: 1.014399E+01 | loss scale: 1.0 | grad norm: 43.370 | number of skipped iterations: 0 | number of nan iterations: 0 |
866
  [2026-08-30 07:40:52] iteration 7/ 12 | consumed samples: 112 | elapsed time per iteration (ms): 485.1 | throughput per GPU (TFLOP/s/GPU): 687.2 | learning rate: 3.250000E-05 | global batch size: 16 | lm loss: 9.904866E+00 | loss scale: 1.0 | grad norm: 8.280 | number of skipped iterations: 0 | number of nan iterations: 0 |
867
  [2026-08-30 07:40:53] iteration 8/ 12 | consumed samples: 128 | elapsed time per iteration (ms): 487.6 | throughput per GPU (TFLOP/s/GPU): 683.7 | learning rate: 1.000000E-05 | global batch size: 16 | lm loss: 8.981642E+00 | loss scale: 1.0 | grad norm: 7.258 | number of skipped iterations: 0 | number of nan iterations: 0 |
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
865
  [2026-08-30 07:40:52] iteration 6/ 12 | consumed samples: 96 | elapsed time per iteration (ms): 498.2 | throughput per GPU (TFLOP/s/GPU): 669.2 | learning rate: 7.750000E-05 | global batch size: 16 | lm loss: 1.014399E+01 | loss scale: 1.0 | grad norm: 43.370 | number of skipped iterations: 0 | number of nan iterations: 0 |
866
  [2026-08-30 07:40:52] iteration 7/ 12 | consumed samples: 112 | elapsed time per iteration (ms): 485.1 | throughput per GPU (TFLOP/s/GPU): 687.2 | learning rate: 3.250000E-05 | global batch size: 16 | lm loss: 9.904866E+00 | loss scale: 1.0 | grad norm: 8.280 | number of skipped iterations: 0 | number of nan iterations: 0 |
867
  [2026-08-30 07:40:53] iteration 8/ 12 | consumed samples: 128 | elapsed time per iteration (ms): 487.6 | throughput per GPU (TFLOP/s/GPU): 683.7 | learning rate: 1.000000E-05 | global batch size: 16 | lm loss: 8.981642E+00 | loss scale: 1.0 | grad norm: 7.258 | number of skipped iterations: 0 | number of nan iterations: 0 |
868
+ [2026-08-30 07:40:53] iteration 9/ 12 | consumed samples: 144 | elapsed time per iteration (ms): 469.0 | throughput per GPU (TFLOP/s/GPU): 710.9 | learning rate: 1.000000E-05 | global batch size: 16 | lm loss: 8.626698E+00 | loss scale: 1.0 | grad norm: 6.470 | number of skipped iterations: 0 | number of nan iterations: 0 |
869
+ [2026-08-30 07:40:54] iteration 10/ 12 | consumed samples: 160 | elapsed time per iteration (ms): 476.7 | throughput per GPU (TFLOP/s/GPU): 699.3 | learning rate: 1.000000E-05 | global batch size: 16 | lm loss: 8.429865E+00 | loss scale: 1.0 | grad norm: 5.282 | number of skipped iterations: 0 | number of nan iterations: 0 |
870
+ [2026-08-30 07:40:54] iteration 11/ 12 | consumed samples: 176 | elapsed time per iteration (ms): 462.2 | throughput per GPU (TFLOP/s/GPU): 721.3 | learning rate: 1.000000E-05 | global batch size: 16 | lm loss: 8.372684E+00 | loss scale: 1.0 | grad norm: 4.866 | number of skipped iterations: 0 | number of nan iterations: 0 |
871
+ [2026-08-30 07:40:55] iteration 12/ 12 | consumed samples: 192 | elapsed time per iteration (ms): 494.0 | throughput per GPU (TFLOP/s/GPU): 674.9 | learning rate: 1.000000E-05 | global batch size: 16 | lm loss: 8.250052E+00 | loss scale: 1.0 | grad norm: 4.802 | number of skipped iterations: 0 | number of nan iterations: 0 |
872
+ [after training is done] datetime: 2026-08-30 07:40:55
873
+ [rank0]:[W830 07:40:56.598775869 ProcessGroupNCCL.cpp:1505] Warning: WARNING: destroy_process_group() was not called before program exit, which can leak resources. For more info, please see https://pytorch.org/docs/stable/distributed.html#shutdown (function operator())
874
+ [rank2]:[W830 07:40:57.647323593 ProcessGroupNCCL.cpp:1505] Warning: WARNING: destroy_process_group() was not called before program exit, which can leak resources. For more info, please see https://pytorch.org/docs/stable/distributed.html#shutdown (function operator())
875
+ [rank7]:[W830 07:40:57.859849150 ProcessGroupNCCL.cpp:1505] Warning: WARNING: destroy_process_group() was not called before program exit, which can leak resources. For more info, please see https://pytorch.org/docs/stable/distributed.html#shutdown (function operator())
876
+ [rank5]:[W830 07:40:57.860248885 ProcessGroupNCCL.cpp:1505] Warning: WARNING: destroy_process_group() was not called before program exit, which can leak resources. For more info, please see https://pytorch.org/docs/stable/distributed.html#shutdown (function operator())
877
+ [rank3]:[W830 07:40:57.094639357 ProcessGroupNCCL.cpp:1505] Warning: WARNING: destroy_process_group() was not called before program exit, which can leak resources. For more info, please see https://pytorch.org/docs/stable/distributed.html#shutdown (function operator())
878
+ [rank1]:[W830 07:40:57.214688954 ProcessGroupNCCL.cpp:1505] Warning: WARNING: destroy_process_group() was not called before program exit, which can leak resources. For more info, please see https://pytorch.org/docs/stable/distributed.html#shutdown (function operator())
879
+ [rank6]:[W830 07:40:58.321615314 ProcessGroupNCCL.cpp:1505] Warning: WARNING: destroy_process_group() was not called before program exit, which can leak resources. For more info, please see https://pytorch.org/docs/stable/distributed.html#shutdown (function operator())
880
+ [rank4]:[W830 07:40:58.342260354 ProcessGroupNCCL.cpp:1505] Warning: WARNING: destroy_process_group() was not called before program exit, which can leak resources. For more info, please see https://pytorch.org/docs/stable/distributed.html#shutdown (function operator())
881
+ W0830 07:41:02.058000 790344 torch/distributed/elastic/agent/server/api.py:719] Received 15 death signal, shutting down workers
882
+ W0830 07:41:02.058000 790344 torch/distributed/elastic/multiprocessing/api.py:900] Sending process 790526 closing signal SIGTERM
883
+ W0830 07:41:02.059000 790344 torch/distributed/elastic/multiprocessing/api.py:900] Sending process 790529 closing signal SIGTERM
884
+ W0830 07:41:02.060000 790344 torch/distributed/elastic/multiprocessing/api.py:900] Sending process 790530 closing signal SIGTERM
885
+ W0830 07:41:02.061000 790344 torch/distributed/elastic/multiprocessing/api.py:900] Sending process 790531 closing signal SIGTERM
886
+ W0830 07:41:02.061000 790344 torch/distributed/elastic/multiprocessing/api.py:900] Sending process 790532 closing signal SIGTERM
887
+ Traceback (most recent call last):
888
+ File "/usr/local/bin/torchrun", line 8, in <module>
889
+ sys.exit(main())
890
+ ^^^^^^
891
+ File "/usr/local/lib/python3.12/dist-packages/torch/distributed/elastic/multiprocessing/errors/__init__.py", line 355, in wrapper
892
+ return f(*args, **kwargs)
893
+ ^^^^^^^^^^^^^^^^^^
894
+ File "/usr/local/lib/python3.12/dist-packages/torch/distributed/run.py", line 892, in main
895
+ run(args)
896
+ File "/usr/local/lib/python3.12/dist-packages/torch/distributed/run.py", line 883, in run
897
+ elastic_launch(
898
+ File "/usr/local/lib/python3.12/dist-packages/torch/distributed/launcher/api.py", line 139, in __call__
899
+ return launch_agent(self._config, self._entrypoint, list(args))
900
+ ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
901
+ File "/usr/local/lib/python3.12/dist-packages/torch/distributed/launcher/api.py", line 261, in launch_agent
902
+ result = agent.run()
903
+ ^^^^^^^^^^^
904
+ File "/usr/local/lib/python3.12/dist-packages/torch/distributed/elastic/metrics/api.py", line 138, in wrapper
905
+ result = f(*args, **kwargs)
906
+ ^^^^^^^^^^^^^^^^^^
907
+ File "/usr/local/lib/python3.12/dist-packages/torch/distributed/elastic/agent/server/api.py", line 711, in run
908
+ result = self._invoke_run(role)
909
+ ^^^^^^^^^^^^^^^^^^^^^^
910
+ File "/usr/local/lib/python3.12/dist-packages/torch/distributed/elastic/agent/server/api.py", line 870, in _invoke_run
911
+ time.sleep(monitor_interval)
912
+ File "/usr/local/lib/python3.12/dist-packages/torch/distributed/elastic/multiprocessing/api.py", line 84, in _terminate_process_handler
913
+ raise SignalException(f"Process {os.getpid()} got signal: {sigval}", sigval=sigval)
914
+ torch.distributed.elastic.multiprocessing.api.SignalException: Process 790344 got signal: 15
tiera-rerun-fixed-moonep/moonep/a1/a1_oneslot_gbs16/results.tsv CHANGED
@@ -1,3 +1,4 @@
1
  config backend policy slots seq gbs cap recompute rep status iter_ms_mean iter_ms_min peak_mib run_dir
2
  G1 moonep one 1 4096 16 none none 1 OK 477.56 460.30 96849 /workspace/tiera-rerun-fixed-moonep/moonep/a1/a1_oneslot_gbs16/G1__moonep__one__seq4096__gbs16__capnone__async/rep1
3
  G1 moonep one 1 4096 16 none none 2 OK 479.54 460.30 96849 /workspace/tiera-rerun-fixed-moonep/moonep/a1/a1_oneslot_gbs16/G1__moonep__one__seq4096__gbs16__capnone__async/rep2
 
 
1
  config backend policy slots seq gbs cap recompute rep status iter_ms_mean iter_ms_min peak_mib run_dir
2
  G1 moonep one 1 4096 16 none none 1 OK 477.56 460.30 96849 /workspace/tiera-rerun-fixed-moonep/moonep/a1/a1_oneslot_gbs16/G1__moonep__one__seq4096__gbs16__capnone__async/rep1
3
  G1 moonep one 1 4096 16 none none 2 OK 479.54 460.30 96849 /workspace/tiera-rerun-fixed-moonep/moonep/a1/a1_oneslot_gbs16/G1__moonep__one__seq4096__gbs16__capnone__async/rep2
4
+ G1 moonep one 1 4096 16 none none 3 OK 480.30 460.30 96849 /workspace/tiera-rerun-fixed-moonep/moonep/a1/a1_oneslot_gbs16/G1__moonep__one__seq4096__gbs16__capnone__async/rep3