granularity results tierb-fixed-moonep-full/echo/tierbtrace @ 2026-08-30T00:38:30Z from sp-aiinfra-interactive-aghosh-7jtjl-8xb200-moe-cg-0
Browse files- .gitattributes +1 -0
- tierb-fixed-moonep-full/echo/tierbtrace/tierbtrace_gbs8/QWEN2_57B__echo__matched__seq4096__gbs8__cap1p08984375/nsys/QWEN2_57B__echo__matched__seq4096__gbs8__cap1p08984375.nsys-rep +3 -0
- tierb-fixed-moonep-full/echo/tierbtrace/tierbtrace_gbs8/QWEN2_57B__echo__matched__seq4096__gbs8__cap1p08984375/nsys/train.log +76 -0
- tierb-fixed-moonep-full/echo/tierbtrace/tierbtrace_gbs8/QWEN2_57B__echo__matched__seq4096__gbs8__cap1p08984375/nsys/verify_trace_ranks.log +6 -0
- tierb-fixed-moonep-full/echo/tierbtrace/tierbtrace_gbs8/trace_results.tsv +1 -0
.gitattributes
CHANGED
|
@@ -58,3 +58,4 @@ moonep/tierbtrace/tierbtrace_gbs8/QWEN2_57B__moonep__matched__seq4096__gbs8__cap
|
|
| 58 |
tierb-fixed-moonep-full/echo/tierbtrace/tierbtrace_gbs8/QWEN3_235B__echo__matched__seq8192__gbs8__cap1p083984375/nsys/QWEN3_235B__echo__matched__seq8192__gbs8__cap1p083984375.nsys-rep filter=lfs diff=lfs merge=lfs -text
|
| 59 |
tierb-fixed-moonep-full/echo/tierbtrace/tierbtrace_gbs8/QWEN3_30B__echo__matched__seq4096__gbs8__cap1p11328125/nsys/QWEN3_30B__echo__matched__seq4096__gbs8__cap1p11328125.nsys-rep filter=lfs diff=lfs merge=lfs -text
|
| 60 |
tierb-fixed-moonep-full/echo/tierbtrace/tierbtrace_gbs8/DSV2_LITE_L24__echo__matched__seq8192__gbs8__cap1p0755208333333333/nsys/DSV2_LITE_L24__echo__matched__seq8192__gbs8__cap1p0755208333333333.nsys-rep filter=lfs diff=lfs merge=lfs -text
|
|
|
|
|
|
| 58 |
tierb-fixed-moonep-full/echo/tierbtrace/tierbtrace_gbs8/QWEN3_235B__echo__matched__seq8192__gbs8__cap1p083984375/nsys/QWEN3_235B__echo__matched__seq8192__gbs8__cap1p083984375.nsys-rep filter=lfs diff=lfs merge=lfs -text
|
| 59 |
tierb-fixed-moonep-full/echo/tierbtrace/tierbtrace_gbs8/QWEN3_30B__echo__matched__seq4096__gbs8__cap1p11328125/nsys/QWEN3_30B__echo__matched__seq4096__gbs8__cap1p11328125.nsys-rep filter=lfs diff=lfs merge=lfs -text
|
| 60 |
tierb-fixed-moonep-full/echo/tierbtrace/tierbtrace_gbs8/DSV2_LITE_L24__echo__matched__seq8192__gbs8__cap1p0755208333333333/nsys/DSV2_LITE_L24__echo__matched__seq8192__gbs8__cap1p0755208333333333.nsys-rep filter=lfs diff=lfs merge=lfs -text
|
| 61 |
+
tierb-fixed-moonep-full/echo/tierbtrace/tierbtrace_gbs8/QWEN2_57B__echo__matched__seq4096__gbs8__cap1p08984375/nsys/QWEN2_57B__echo__matched__seq4096__gbs8__cap1p08984375.nsys-rep filter=lfs diff=lfs merge=lfs -text
|
tierb-fixed-moonep-full/echo/tierbtrace/tierbtrace_gbs8/QWEN2_57B__echo__matched__seq4096__gbs8__cap1p08984375/nsys/QWEN2_57B__echo__matched__seq4096__gbs8__cap1p08984375.nsys-rep
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:449b399393ec9daa424670d976c1a7646c8426a82661c01e731b42ce4ccb47fd
|
| 3 |
+
size 7711080
|
tierb-fixed-moonep-full/echo/tierbtrace/tierbtrace_gbs8/QWEN2_57B__echo__matched__seq4096__gbs8__cap1p08984375/nsys/train.log
CHANGED
|
@@ -1155,3 +1155,79 @@ CUDA graph capture done!!!
|
|
| 1155 |
[2026-08-30 00:37:13] iteration 6/ 10 | consumed samples: 48 | elapsed time per iteration (ms): 184.7 | throughput per GPU (TFLOP/s/GPU): 668.6 | learning rate: 7.750000E-05 | global batch size: 8 | lm loss: 1.049675E+01 | loss scale: 1.0 | grad norm: 8.719 | number of skipped iterations: 0 | number of nan iterations: 0 |
|
| 1156 |
[2026-08-30 00:37:16] iteration 7/ 10 | consumed samples: 56 | elapsed time per iteration (ms): 2960.4 | throughput per GPU (TFLOP/s/GPU): 41.7 | learning rate: 3.250000E-05 | global batch size: 8 | lm loss: 9.439878E+00 | loss scale: 1.0 | grad norm: 5.358 | number of skipped iterations: 0 | number of nan iterations: 0 |
|
| 1157 |
[2026-08-30 00:37:16] iteration 8/ 10 | consumed samples: 64 | elapsed time per iteration (ms): 189.0 | throughput per GPU (TFLOP/s/GPU): 653.3 | learning rate: 1.000000E-05 | global batch size: 8 | lm loss: 8.960054E+00 | loss scale: 1.0 | grad norm: 4.541 | number of skipped iterations: 0 | number of nan iterations: 0 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1155 |
[2026-08-30 00:37:13] iteration 6/ 10 | consumed samples: 48 | elapsed time per iteration (ms): 184.7 | throughput per GPU (TFLOP/s/GPU): 668.6 | learning rate: 7.750000E-05 | global batch size: 8 | lm loss: 1.049675E+01 | loss scale: 1.0 | grad norm: 8.719 | number of skipped iterations: 0 | number of nan iterations: 0 |
|
| 1156 |
[2026-08-30 00:37:16] iteration 7/ 10 | consumed samples: 56 | elapsed time per iteration (ms): 2960.4 | throughput per GPU (TFLOP/s/GPU): 41.7 | learning rate: 3.250000E-05 | global batch size: 8 | lm loss: 9.439878E+00 | loss scale: 1.0 | grad norm: 5.358 | number of skipped iterations: 0 | number of nan iterations: 0 |
|
| 1157 |
[2026-08-30 00:37:16] iteration 8/ 10 | consumed samples: 64 | elapsed time per iteration (ms): 189.0 | throughput per GPU (TFLOP/s/GPU): 653.3 | learning rate: 1.000000E-05 | global batch size: 8 | lm loss: 8.960054E+00 | loss scale: 1.0 | grad norm: 4.541 | number of skipped iterations: 0 | number of nan iterations: 0 |
|
| 1158 |
+
Capture range started in the application.
|
| 1159 |
+
Capture range ended in the application.
|
| 1160 |
+
Generating '/tmp/nsys-report-4c4a.qdstrm'
|
| 1161 |
+
|
| 1162 |
+
|
| 1163 |
+
[2026-08-30 00:37:57] iteration 9/ 10 | consumed samples: 72 | elapsed time per iteration (ms): 40812.7 | throughput per GPU (TFLOP/s/GPU): 3.0 | learning rate: 1.000000E-05 | global batch size: 8 | lm loss: 8.744094E+00 | loss scale: 1.0 | grad norm: 4.286 | number of skipped iterations: 0 | number of nan iterations: 0 |
|
| 1164 |
+
[2026-08-30 00:37:57] iteration 10/ 10 | consumed samples: 80 | elapsed time per iteration (ms): 188.4 | throughput per GPU (TFLOP/s/GPU): 655.4 | learning rate: 1.000000E-05 | global batch size: 8 | lm loss: 8.683288E+00 | loss scale: 1.0 | grad norm: 4.413 | number of skipped iterations: 0 | number of nan iterations: 0 |
|
| 1165 |
+
[after training is done] datetime: 2026-08-30 00:37:57
|
| 1166 |
+
Collection stop is not allowed in this state.
|
| 1167 |
+
[rank0]:[W830 00:37:59.602654275 ProcessGroupNCCL.cpp:1505] Warning: WARNING: destroy_process_group() was not called before program exit, which can leak resources. For more info, please see https://pytorch.org/docs/stable/distributed.html#shutdown (function operator())
|
| 1168 |
+
[rank3]:[W830 00:37:59.855384171 observer.cpp:116] Warning: Leaked callback handle: 1 (function operator())
|
| 1169 |
+
pure virtual method called
|
| 1170 |
+
terminate called without an active exception
|
| 1171 |
+
[rank5]:[W830 00:37:59.906998686 observer.cpp:116] Warning: Leaked callback handle: 1 (function operator())
|
| 1172 |
+
pure virtual method called
|
| 1173 |
+
terminate called without an active exception
|
| 1174 |
+
[rank2]:[W830 00:37:59.938439357 observer.cpp:116] Warning: Leaked callback handle: 1 (function operator())
|
| 1175 |
+
pure virtual method called
|
| 1176 |
+
terminate called without an active exception
|
| 1177 |
+
[rank1]:[W830 00:38:00.380517284 observer.cpp:116] Warning: Leaked callback handle: 1 (function operator())
|
| 1178 |
+
[rank0]:[W830 00:38:00.395553292 observer.cpp:116] Warning: Leaked callback handle: 1 (function operator())
|
| 1179 |
+
pure virtual method called
|
| 1180 |
+
terminate called without an active exception
|
| 1181 |
+
pure virtual method called
|
| 1182 |
+
terminate called without an active exception
|
| 1183 |
+
[rank6]:[W830 00:38:00.420723333 observer.cpp:116] Warning: Leaked callback handle: 1 (function operator())
|
| 1184 |
+
[rank7]:[W830 00:38:00.424333105 observer.cpp:116] Warning: Leaked callback handle: 1 (function operator())
|
| 1185 |
+
pure virtual method called
|
| 1186 |
+
terminate called without an active exception
|
| 1187 |
+
pure virtual method called
|
| 1188 |
+
terminate called without an active exception
|
| 1189 |
+
[rank4]:[W830 00:38:00.476276242 observer.cpp:116] Warning: Leaked callback handle: 1 (function operator())
|
| 1190 |
+
pure virtual method called
|
| 1191 |
+
terminate called without an active exception
|
| 1192 |
+
W0830 00:38:00.428000 322485 torch/distributed/elastic/multiprocessing/api.py:900] Sending process 322597 closing signal SIGTERM
|
| 1193 |
+
W0830 00:38:00.428000 322485 torch/distributed/elastic/multiprocessing/api.py:900] Sending process 322598 closing signal SIGTERM
|
| 1194 |
+
W0830 00:38:00.429000 322485 torch/distributed/elastic/multiprocessing/api.py:900] Sending process 322599 closing signal SIGTERM
|
| 1195 |
+
W0830 00:38:00.429000 322485 torch/distributed/elastic/multiprocessing/api.py:900] Sending process 322601 closing signal SIGTERM
|
| 1196 |
+
W0830 00:38:00.429000 322485 torch/distributed/elastic/multiprocessing/api.py:900] Sending process 322602 closing signal SIGTERM
|
| 1197 |
+
W0830 00:38:00.429000 322485 torch/distributed/elastic/multiprocessing/api.py:900] Sending process 322603 closing signal SIGTERM
|
| 1198 |
+
W0830 00:38:00.430000 322485 torch/distributed/elastic/multiprocessing/api.py:900] Sending process 322604 closing signal SIGTERM
|
| 1199 |
+
E0830 00:38:02.712000 322485 torch/distributed/elastic/multiprocessing/api.py:874] failed (exitcode: -6) local_rank: 3 (pid: 322600) of binary: /usr/bin/python
|
| 1200 |
+
Traceback (most recent call last):
|
| 1201 |
+
File "/usr/local/bin/torchrun", line 8, in <module>
|
| 1202 |
+
sys.exit(main())
|
| 1203 |
+
^^^^^^
|
| 1204 |
+
File "/usr/local/lib/python3.12/dist-packages/torch/distributed/elastic/multiprocessing/errors/__init__.py", line 355, in wrapper
|
| 1205 |
+
return f(*args, **kwargs)
|
| 1206 |
+
^^^^^^^^^^^^^^^^^^
|
| 1207 |
+
File "/usr/local/lib/python3.12/dist-packages/torch/distributed/run.py", line 892, in main
|
| 1208 |
+
run(args)
|
| 1209 |
+
File "/usr/local/lib/python3.12/dist-packages/torch/distributed/run.py", line 883, in run
|
| 1210 |
+
elastic_launch(
|
| 1211 |
+
File "/usr/local/lib/python3.12/dist-packages/torch/distributed/launcher/api.py", line 139, in __call__
|
| 1212 |
+
return launch_agent(self._config, self._entrypoint, list(args))
|
| 1213 |
+
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
| 1214 |
+
File "/usr/local/lib/python3.12/dist-packages/torch/distributed/launcher/api.py", line 270, in launch_agent
|
| 1215 |
+
raise ChildFailedError(
|
| 1216 |
+
torch.distributed.elastic.multiprocessing.errors.ChildFailedError:
|
| 1217 |
+
============================================================
|
| 1218 |
+
pretrain_gpt.py FAILED
|
| 1219 |
+
------------------------------------------------------------
|
| 1220 |
+
Failures:
|
| 1221 |
+
<NO_OTHER_FAILURES>
|
| 1222 |
+
------------------------------------------------------------
|
| 1223 |
+
Root Cause (first observed failure):
|
| 1224 |
+
[0]:
|
| 1225 |
+
time : 2026-08-30_00:38:00
|
| 1226 |
+
host : sp-aiinfra-interactive-aghosh-7jtjl-8xb200-moe-cg-0
|
| 1227 |
+
rank : 3 (local_rank: 3)
|
| 1228 |
+
exitcode : -6 (pid: 322600)
|
| 1229 |
+
error_file: <N/A>
|
| 1230 |
+
traceback : Signal 6 (SIGABRT) received by PID 322600
|
| 1231 |
+
============================================================
|
| 1232 |
+
Generated:
|
| 1233 |
+
/workspace/tierb-fixed-moonep-full/echo/tierbtrace/tierbtrace_gbs8/QWEN2_57B__echo__matched__seq4096__gbs8__cap1p08984375/nsys/QWEN2_57B__echo__matched__seq4096__gbs8__cap1p08984375.nsys-rep
|
tierb-fixed-moonep-full/echo/tierbtrace/tierbtrace_gbs8/QWEN2_57B__echo__matched__seq4096__gbs8__cap1p08984375/nsys/verify_trace_ranks.log
ADDED
|
@@ -0,0 +1,6 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
report : /workspace/tierb-fixed-moonep-full/echo/tierbtrace/tierbtrace_gbs8/QWEN2_57B__echo__matched__seq4096__gbs8__cap1p08984375/nsys/QWEN2_57B__echo__matched__seq4096__gbs8__cap1p08984375.nsys-rep
|
| 2 |
+
kernels : 90563
|
| 3 |
+
devices : 8 of 8 -> [0, 1, 2, 3, 4, 5, 6, 7]
|
| 4 |
+
processes: 1
|
| 5 |
+
markers : 8000 echo_trace__ kernels on devices [0, 1, 2, 3, 4, 5, 6, 7]
|
| 6 |
+
PASS: all ranks present and backend markers emitted
|
tierb-fixed-moonep-full/echo/tierbtrace/tierbtrace_gbs8/trace_results.tsv
CHANGED
|
@@ -2,3 +2,4 @@ model backend capacity status report async_overlap_json
|
|
| 2 |
QWEN3_235B echo 1.083984375 OK /workspace/tierb-fixed-moonep-full/echo/tierbtrace/tierbtrace_gbs8/QWEN3_235B__echo__matched__seq8192__gbs8__cap1p083984375/nsys/QWEN3_235B__echo__matched__seq8192__gbs8__cap1p083984375.nsys-rep
|
| 3 |
QWEN3_30B echo 1.11328125 OK /workspace/tierb-fixed-moonep-full/echo/tierbtrace/tierbtrace_gbs8/QWEN3_30B__echo__matched__seq4096__gbs8__cap1p11328125/nsys/QWEN3_30B__echo__matched__seq4096__gbs8__cap1p11328125.nsys-rep
|
| 4 |
DSV2_LITE_L24 echo 1.0755208333333333 OK /workspace/tierb-fixed-moonep-full/echo/tierbtrace/tierbtrace_gbs8/DSV2_LITE_L24__echo__matched__seq8192__gbs8__cap1p0755208333333333/nsys/DSV2_LITE_L24__echo__matched__seq8192__gbs8__cap1p0755208333333333.nsys-rep
|
|
|
|
|
|
| 2 |
QWEN3_235B echo 1.083984375 OK /workspace/tierb-fixed-moonep-full/echo/tierbtrace/tierbtrace_gbs8/QWEN3_235B__echo__matched__seq8192__gbs8__cap1p083984375/nsys/QWEN3_235B__echo__matched__seq8192__gbs8__cap1p083984375.nsys-rep
|
| 3 |
QWEN3_30B echo 1.11328125 OK /workspace/tierb-fixed-moonep-full/echo/tierbtrace/tierbtrace_gbs8/QWEN3_30B__echo__matched__seq4096__gbs8__cap1p11328125/nsys/QWEN3_30B__echo__matched__seq4096__gbs8__cap1p11328125.nsys-rep
|
| 4 |
DSV2_LITE_L24 echo 1.0755208333333333 OK /workspace/tierb-fixed-moonep-full/echo/tierbtrace/tierbtrace_gbs8/DSV2_LITE_L24__echo__matched__seq8192__gbs8__cap1p0755208333333333/nsys/DSV2_LITE_L24__echo__matched__seq8192__gbs8__cap1p0755208333333333.nsys-rep
|
| 5 |
+
QWEN2_57B echo 1.08984375 OK /workspace/tierb-fixed-moonep-full/echo/tierbtrace/tierbtrace_gbs8/QWEN2_57B__echo__matched__seq4096__gbs8__cap1p08984375/nsys/QWEN2_57B__echo__matched__seq4096__gbs8__cap1p08984375.nsys-rep
|