AbhishekGhosh1998 commited on
Commit
e6035c0
·
verified ·
1 Parent(s): bac91ca

granularity results tierb-fixed-moonep-full/echo/tierbtrace @ 2026-08-30T00:38:30Z from sp-aiinfra-interactive-aghosh-7jtjl-8xb200-moe-cg-0

Browse files
.gitattributes CHANGED
@@ -58,3 +58,4 @@ moonep/tierbtrace/tierbtrace_gbs8/QWEN2_57B__moonep__matched__seq4096__gbs8__cap
58
  tierb-fixed-moonep-full/echo/tierbtrace/tierbtrace_gbs8/QWEN3_235B__echo__matched__seq8192__gbs8__cap1p083984375/nsys/QWEN3_235B__echo__matched__seq8192__gbs8__cap1p083984375.nsys-rep filter=lfs diff=lfs merge=lfs -text
59
  tierb-fixed-moonep-full/echo/tierbtrace/tierbtrace_gbs8/QWEN3_30B__echo__matched__seq4096__gbs8__cap1p11328125/nsys/QWEN3_30B__echo__matched__seq4096__gbs8__cap1p11328125.nsys-rep filter=lfs diff=lfs merge=lfs -text
60
  tierb-fixed-moonep-full/echo/tierbtrace/tierbtrace_gbs8/DSV2_LITE_L24__echo__matched__seq8192__gbs8__cap1p0755208333333333/nsys/DSV2_LITE_L24__echo__matched__seq8192__gbs8__cap1p0755208333333333.nsys-rep filter=lfs diff=lfs merge=lfs -text
 
 
58
  tierb-fixed-moonep-full/echo/tierbtrace/tierbtrace_gbs8/QWEN3_235B__echo__matched__seq8192__gbs8__cap1p083984375/nsys/QWEN3_235B__echo__matched__seq8192__gbs8__cap1p083984375.nsys-rep filter=lfs diff=lfs merge=lfs -text
59
  tierb-fixed-moonep-full/echo/tierbtrace/tierbtrace_gbs8/QWEN3_30B__echo__matched__seq4096__gbs8__cap1p11328125/nsys/QWEN3_30B__echo__matched__seq4096__gbs8__cap1p11328125.nsys-rep filter=lfs diff=lfs merge=lfs -text
60
  tierb-fixed-moonep-full/echo/tierbtrace/tierbtrace_gbs8/DSV2_LITE_L24__echo__matched__seq8192__gbs8__cap1p0755208333333333/nsys/DSV2_LITE_L24__echo__matched__seq8192__gbs8__cap1p0755208333333333.nsys-rep filter=lfs diff=lfs merge=lfs -text
61
+ tierb-fixed-moonep-full/echo/tierbtrace/tierbtrace_gbs8/QWEN2_57B__echo__matched__seq4096__gbs8__cap1p08984375/nsys/QWEN2_57B__echo__matched__seq4096__gbs8__cap1p08984375.nsys-rep filter=lfs diff=lfs merge=lfs -text
tierb-fixed-moonep-full/echo/tierbtrace/tierbtrace_gbs8/QWEN2_57B__echo__matched__seq4096__gbs8__cap1p08984375/nsys/QWEN2_57B__echo__matched__seq4096__gbs8__cap1p08984375.nsys-rep ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:449b399393ec9daa424670d976c1a7646c8426a82661c01e731b42ce4ccb47fd
3
+ size 7711080
tierb-fixed-moonep-full/echo/tierbtrace/tierbtrace_gbs8/QWEN2_57B__echo__matched__seq4096__gbs8__cap1p08984375/nsys/train.log CHANGED
@@ -1155,3 +1155,79 @@ CUDA graph capture done!!!
1155
  [2026-08-30 00:37:13] iteration 6/ 10 | consumed samples: 48 | elapsed time per iteration (ms): 184.7 | throughput per GPU (TFLOP/s/GPU): 668.6 | learning rate: 7.750000E-05 | global batch size: 8 | lm loss: 1.049675E+01 | loss scale: 1.0 | grad norm: 8.719 | number of skipped iterations: 0 | number of nan iterations: 0 |
1156
  [2026-08-30 00:37:16] iteration 7/ 10 | consumed samples: 56 | elapsed time per iteration (ms): 2960.4 | throughput per GPU (TFLOP/s/GPU): 41.7 | learning rate: 3.250000E-05 | global batch size: 8 | lm loss: 9.439878E+00 | loss scale: 1.0 | grad norm: 5.358 | number of skipped iterations: 0 | number of nan iterations: 0 |
1157
  [2026-08-30 00:37:16] iteration 8/ 10 | consumed samples: 64 | elapsed time per iteration (ms): 189.0 | throughput per GPU (TFLOP/s/GPU): 653.3 | learning rate: 1.000000E-05 | global batch size: 8 | lm loss: 8.960054E+00 | loss scale: 1.0 | grad norm: 4.541 | number of skipped iterations: 0 | number of nan iterations: 0 |
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1155
  [2026-08-30 00:37:13] iteration 6/ 10 | consumed samples: 48 | elapsed time per iteration (ms): 184.7 | throughput per GPU (TFLOP/s/GPU): 668.6 | learning rate: 7.750000E-05 | global batch size: 8 | lm loss: 1.049675E+01 | loss scale: 1.0 | grad norm: 8.719 | number of skipped iterations: 0 | number of nan iterations: 0 |
1156
  [2026-08-30 00:37:16] iteration 7/ 10 | consumed samples: 56 | elapsed time per iteration (ms): 2960.4 | throughput per GPU (TFLOP/s/GPU): 41.7 | learning rate: 3.250000E-05 | global batch size: 8 | lm loss: 9.439878E+00 | loss scale: 1.0 | grad norm: 5.358 | number of skipped iterations: 0 | number of nan iterations: 0 |
1157
  [2026-08-30 00:37:16] iteration 8/ 10 | consumed samples: 64 | elapsed time per iteration (ms): 189.0 | throughput per GPU (TFLOP/s/GPU): 653.3 | learning rate: 1.000000E-05 | global batch size: 8 | lm loss: 8.960054E+00 | loss scale: 1.0 | grad norm: 4.541 | number of skipped iterations: 0 | number of nan iterations: 0 |
1158
+ Capture range started in the application.
1159
+ Capture range ended in the application.
1160
+ Generating '/tmp/nsys-report-4c4a.qdstrm'
1161
+
1162
+
1163
+ [2026-08-30 00:37:57] iteration 9/ 10 | consumed samples: 72 | elapsed time per iteration (ms): 40812.7 | throughput per GPU (TFLOP/s/GPU): 3.0 | learning rate: 1.000000E-05 | global batch size: 8 | lm loss: 8.744094E+00 | loss scale: 1.0 | grad norm: 4.286 | number of skipped iterations: 0 | number of nan iterations: 0 |
1164
+ [2026-08-30 00:37:57] iteration 10/ 10 | consumed samples: 80 | elapsed time per iteration (ms): 188.4 | throughput per GPU (TFLOP/s/GPU): 655.4 | learning rate: 1.000000E-05 | global batch size: 8 | lm loss: 8.683288E+00 | loss scale: 1.0 | grad norm: 4.413 | number of skipped iterations: 0 | number of nan iterations: 0 |
1165
+ [after training is done] datetime: 2026-08-30 00:37:57
1166
+ Collection stop is not allowed in this state.
1167
+ [rank0]:[W830 00:37:59.602654275 ProcessGroupNCCL.cpp:1505] Warning: WARNING: destroy_process_group() was not called before program exit, which can leak resources. For more info, please see https://pytorch.org/docs/stable/distributed.html#shutdown (function operator())
1168
+ [rank3]:[W830 00:37:59.855384171 observer.cpp:116] Warning: Leaked callback handle: 1 (function operator())
1169
+ pure virtual method called
1170
+ terminate called without an active exception
1171
+ [rank5]:[W830 00:37:59.906998686 observer.cpp:116] Warning: Leaked callback handle: 1 (function operator())
1172
+ pure virtual method called
1173
+ terminate called without an active exception
1174
+ [rank2]:[W830 00:37:59.938439357 observer.cpp:116] Warning: Leaked callback handle: 1 (function operator())
1175
+ pure virtual method called
1176
+ terminate called without an active exception
1177
+ [rank1]:[W830 00:38:00.380517284 observer.cpp:116] Warning: Leaked callback handle: 1 (function operator())
1178
+ [rank0]:[W830 00:38:00.395553292 observer.cpp:116] Warning: Leaked callback handle: 1 (function operator())
1179
+ pure virtual method called
1180
+ terminate called without an active exception
1181
+ pure virtual method called
1182
+ terminate called without an active exception
1183
+ [rank6]:[W830 00:38:00.420723333 observer.cpp:116] Warning: Leaked callback handle: 1 (function operator())
1184
+ [rank7]:[W830 00:38:00.424333105 observer.cpp:116] Warning: Leaked callback handle: 1 (function operator())
1185
+ pure virtual method called
1186
+ terminate called without an active exception
1187
+ pure virtual method called
1188
+ terminate called without an active exception
1189
+ [rank4]:[W830 00:38:00.476276242 observer.cpp:116] Warning: Leaked callback handle: 1 (function operator())
1190
+ pure virtual method called
1191
+ terminate called without an active exception
1192
+ W0830 00:38:00.428000 322485 torch/distributed/elastic/multiprocessing/api.py:900] Sending process 322597 closing signal SIGTERM
1193
+ W0830 00:38:00.428000 322485 torch/distributed/elastic/multiprocessing/api.py:900] Sending process 322598 closing signal SIGTERM
1194
+ W0830 00:38:00.429000 322485 torch/distributed/elastic/multiprocessing/api.py:900] Sending process 322599 closing signal SIGTERM
1195
+ W0830 00:38:00.429000 322485 torch/distributed/elastic/multiprocessing/api.py:900] Sending process 322601 closing signal SIGTERM
1196
+ W0830 00:38:00.429000 322485 torch/distributed/elastic/multiprocessing/api.py:900] Sending process 322602 closing signal SIGTERM
1197
+ W0830 00:38:00.429000 322485 torch/distributed/elastic/multiprocessing/api.py:900] Sending process 322603 closing signal SIGTERM
1198
+ W0830 00:38:00.430000 322485 torch/distributed/elastic/multiprocessing/api.py:900] Sending process 322604 closing signal SIGTERM
1199
+ E0830 00:38:02.712000 322485 torch/distributed/elastic/multiprocessing/api.py:874] failed (exitcode: -6) local_rank: 3 (pid: 322600) of binary: /usr/bin/python
1200
+ Traceback (most recent call last):
1201
+ File "/usr/local/bin/torchrun", line 8, in <module>
1202
+ sys.exit(main())
1203
+ ^^^^^^
1204
+ File "/usr/local/lib/python3.12/dist-packages/torch/distributed/elastic/multiprocessing/errors/__init__.py", line 355, in wrapper
1205
+ return f(*args, **kwargs)
1206
+ ^^^^^^^^^^^^^^^^^^
1207
+ File "/usr/local/lib/python3.12/dist-packages/torch/distributed/run.py", line 892, in main
1208
+ run(args)
1209
+ File "/usr/local/lib/python3.12/dist-packages/torch/distributed/run.py", line 883, in run
1210
+ elastic_launch(
1211
+ File "/usr/local/lib/python3.12/dist-packages/torch/distributed/launcher/api.py", line 139, in __call__
1212
+ return launch_agent(self._config, self._entrypoint, list(args))
1213
+ ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
1214
+ File "/usr/local/lib/python3.12/dist-packages/torch/distributed/launcher/api.py", line 270, in launch_agent
1215
+ raise ChildFailedError(
1216
+ torch.distributed.elastic.multiprocessing.errors.ChildFailedError:
1217
+ ============================================================
1218
+ pretrain_gpt.py FAILED
1219
+ ------------------------------------------------------------
1220
+ Failures:
1221
+ <NO_OTHER_FAILURES>
1222
+ ------------------------------------------------------------
1223
+ Root Cause (first observed failure):
1224
+ [0]:
1225
+ time : 2026-08-30_00:38:00
1226
+ host : sp-aiinfra-interactive-aghosh-7jtjl-8xb200-moe-cg-0
1227
+ rank : 3 (local_rank: 3)
1228
+ exitcode : -6 (pid: 322600)
1229
+ error_file: <N/A>
1230
+ traceback : Signal 6 (SIGABRT) received by PID 322600
1231
+ ============================================================
1232
+ Generated:
1233
+ /workspace/tierb-fixed-moonep-full/echo/tierbtrace/tierbtrace_gbs8/QWEN2_57B__echo__matched__seq4096__gbs8__cap1p08984375/nsys/QWEN2_57B__echo__matched__seq4096__gbs8__cap1p08984375.nsys-rep
tierb-fixed-moonep-full/echo/tierbtrace/tierbtrace_gbs8/QWEN2_57B__echo__matched__seq4096__gbs8__cap1p08984375/nsys/verify_trace_ranks.log ADDED
@@ -0,0 +1,6 @@
 
 
 
 
 
 
 
1
+ report : /workspace/tierb-fixed-moonep-full/echo/tierbtrace/tierbtrace_gbs8/QWEN2_57B__echo__matched__seq4096__gbs8__cap1p08984375/nsys/QWEN2_57B__echo__matched__seq4096__gbs8__cap1p08984375.nsys-rep
2
+ kernels : 90563
3
+ devices : 8 of 8 -> [0, 1, 2, 3, 4, 5, 6, 7]
4
+ processes: 1
5
+ markers : 8000 echo_trace__ kernels on devices [0, 1, 2, 3, 4, 5, 6, 7]
6
+ PASS: all ranks present and backend markers emitted
tierb-fixed-moonep-full/echo/tierbtrace/tierbtrace_gbs8/trace_results.tsv CHANGED
@@ -2,3 +2,4 @@ model backend capacity status report async_overlap_json
2
  QWEN3_235B echo 1.083984375 OK /workspace/tierb-fixed-moonep-full/echo/tierbtrace/tierbtrace_gbs8/QWEN3_235B__echo__matched__seq8192__gbs8__cap1p083984375/nsys/QWEN3_235B__echo__matched__seq8192__gbs8__cap1p083984375.nsys-rep
3
  QWEN3_30B echo 1.11328125 OK /workspace/tierb-fixed-moonep-full/echo/tierbtrace/tierbtrace_gbs8/QWEN3_30B__echo__matched__seq4096__gbs8__cap1p11328125/nsys/QWEN3_30B__echo__matched__seq4096__gbs8__cap1p11328125.nsys-rep
4
  DSV2_LITE_L24 echo 1.0755208333333333 OK /workspace/tierb-fixed-moonep-full/echo/tierbtrace/tierbtrace_gbs8/DSV2_LITE_L24__echo__matched__seq8192__gbs8__cap1p0755208333333333/nsys/DSV2_LITE_L24__echo__matched__seq8192__gbs8__cap1p0755208333333333.nsys-rep
 
 
2
  QWEN3_235B echo 1.083984375 OK /workspace/tierb-fixed-moonep-full/echo/tierbtrace/tierbtrace_gbs8/QWEN3_235B__echo__matched__seq8192__gbs8__cap1p083984375/nsys/QWEN3_235B__echo__matched__seq8192__gbs8__cap1p083984375.nsys-rep
3
  QWEN3_30B echo 1.11328125 OK /workspace/tierb-fixed-moonep-full/echo/tierbtrace/tierbtrace_gbs8/QWEN3_30B__echo__matched__seq4096__gbs8__cap1p11328125/nsys/QWEN3_30B__echo__matched__seq4096__gbs8__cap1p11328125.nsys-rep
4
  DSV2_LITE_L24 echo 1.0755208333333333 OK /workspace/tierb-fixed-moonep-full/echo/tierbtrace/tierbtrace_gbs8/DSV2_LITE_L24__echo__matched__seq8192__gbs8__cap1p0755208333333333/nsys/DSV2_LITE_L24__echo__matched__seq8192__gbs8__cap1p0755208333333333.nsys-rep
5
+ QWEN2_57B echo 1.08984375 OK /workspace/tierb-fixed-moonep-full/echo/tierbtrace/tierbtrace_gbs8/QWEN2_57B__echo__matched__seq4096__gbs8__cap1p08984375/nsys/QWEN2_57B__echo__matched__seq4096__gbs8__cap1p08984375.nsys-rep