granularity results tiera-rerun-fixed-moonep @ 2026-08-30T08:53:20Z from sp-aiinfra-interactive-aghosh-7jtjl-8xb200-moe-cg-0 (part 2)
Browse files
tiera-rerun-fixed-moonep/study.log
CHANGED
|
@@ -1965,3 +1965,12 @@ driver_repo: /workspace/blackwell_run_scripts @ f69d9ab
|
|
| 1965 |
[08:35:29] TRACE-ONLY G3__echo__matched__seq4096__gbs16__cap1p072265625 (ranks: 0 1 2 3 4 5 6 7)
|
| 1966 |
[guard] iter 6/10 (80s)
|
| 1967 |
[guard] iter 8/10 (90s)
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1965 |
[08:35:29] TRACE-ONLY G3__echo__matched__seq4096__gbs16__cap1p072265625 (ranks: 0 1 2 3 4 5 6 7)
|
| 1966 |
[guard] iter 6/10 (80s)
|
| 1967 |
[guard] iter 8/10 (90s)
|
| 1968 |
+
[guard] no progress for 420s at iter 8/10; treating as done
|
| 1969 |
+
[guard] nsys stop (flushing report)
|
| 1970 |
+
[guard] tearing down process group (outcome=STALLED)
|
| 1971 |
+
[08:44:48] trace verified -> G3__echo__matched__seq4096__gbs16__cap1p072265625.nsys-rep (17M, all 8 GPUs + echo markers)
|
| 1972 |
+
[08:44:48] TRACE-ONLY G4__echo__matched__seq4096__gbs16__cap1p095703125 (ranks: 0 1 2 3 4 5 6 7)
|
| 1973 |
+
[guard] iter 6/10 (80s)
|
| 1974 |
+
[guard] iter 8/10 (90s)
|
| 1975 |
+
[guard] no progress for 420s at iter 8/10; treating as done
|
| 1976 |
+
[guard] nsys stop (flushing report)
|