granularity results tiera-rerun-fixed-moonep @ 2026-08-30T09:13:36Z from sp-aiinfra-interactive-aghosh-7jtjl-8xb200-moe-cg-0 (part 2)
Browse files
tiera-rerun-fixed-moonep/study.log
CHANGED
|
@@ -2011,3 +2011,10 @@ https://huggingface.co/MoE-CUDA-Graph-Analysis/tierb-b200-gbs8/tree/main
|
|
| 2011 |
[09:01:03] TRACE-ONLY G2__ultraep__matched__seq4096__gbs16__cap1p08203125 (ranks: 0 1 2 3 4 5 6 7)
|
| 2012 |
[guard] iter 6/10 (70s)
|
| 2013 |
[guard] iter 8/10 (80s)
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 2011 |
[09:01:03] TRACE-ONLY G2__ultraep__matched__seq4096__gbs16__cap1p08203125 (ranks: 0 1 2 3 4 5 6 7)
|
| 2012 |
[guard] iter 6/10 (70s)
|
| 2013 |
[guard] iter 8/10 (80s)
|
| 2014 |
+
[guard] no progress for 420s at iter 8/10; treating as done
|
| 2015 |
+
[guard] nsys stop (flushing report)
|
| 2016 |
+
[guard] tearing down process group (outcome=STALLED)
|
| 2017 |
+
[09:10:10] trace verified -> G2__ultraep__matched__seq4096__gbs16__cap1p08203125.nsys-rep (11M, all 8 GPUs + ultraep markers)
|
| 2018 |
+
[09:10:10] TRACE-ONLY G3__ultraep__matched__seq4096__gbs16__cap1p072265625 (ranks: 0 1 2 3 4 5 6 7)
|
| 2019 |
+
[guard] iter 6/10 (70s)
|
| 2020 |
+
[guard] iter 8/10 (80s)
|