granularity results tiera-rerun-fixed-moonep @ 2026-08-30T08:03:48Z from sp-aiinfra-interactive-aghosh-7jtjl-8xb200-moe-cg-0
Browse files
tiera-rerun-fixed-moonep/CAPACITY_DECISION.txt
ADDED
|
@@ -0,0 +1,9 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
capacity_mode=perrung
|
| 2 |
+
derived_global=2.40625
|
| 3 |
+
per_rung_G1=2.40625
|
| 4 |
+
per_rung_G2=1.08203125
|
| 5 |
+
per_rung_G3=1.072265625
|
| 6 |
+
per_rung_G4=1.095703125
|
| 7 |
+
reason=the single global capacity OOMed at least one cell, so each rung
|
| 8 |
+
reason=uses its own capacity. Backends remain comparable WITHIN a rung;
|
| 9 |
+
reason=the cross-rung trend must be read with the capacity column.
|
tiera-rerun-fixed-moonep/STATUS.txt
CHANGED
|
@@ -1,7 +1,7 @@
|
|
| 1 |
study_root: /workspace/tiera-rerun-fixed-moonep
|
| 2 |
gbs: 16
|
| 3 |
-
updated: 2026-08-30T08:03:
|
| 4 |
-
current:
|
| 5 |
|
| 6 |
completed steps:
|
| 7 |
echo__a1__gbs16
|
|
|
|
| 1 |
study_root: /workspace/tiera-rerun-fixed-moonep
|
| 2 |
gbs: 16
|
| 3 |
+
updated: 2026-08-30T08:03:47+00:00
|
| 4 |
+
current: echo__tierbmem__gbs16
|
| 5 |
|
| 6 |
completed steps:
|
| 7 |
echo__a1__gbs16
|
tiera-rerun-fixed-moonep/echo/tierbmem/provenance.txt
ADDED
|
@@ -0,0 +1,29 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
==============================================================
|
| 2 |
+
PROVENANCE: echo__tierbmem__gbs16
|
| 3 |
+
==============================================================
|
| 4 |
+
captured_at : 2026-08-30T08:03:47+00:00
|
| 5 |
+
hostname : sp-aiinfra-interactive-aghosh-7jtjl-8xb200-moe-cg-0
|
| 6 |
+
kernel : 6.8.0-1043-nvidia
|
| 7 |
+
cwd : /workspace/blackwell_run_scripts
|
| 8 |
+
command : run_ladder.sh --phase tierbmem --gbs 16 --out /workspace/tiera-rerun-fixed-moonep/echo/tierbmem
|
| 9 |
+
|
| 10 |
+
---- GPU / driver --------------------------------------------
|
| 11 |
+
0, NVIDIA B200, 10.0, 183359 MiB, 580.95.05
|
| 12 |
+
1, NVIDIA B200, 10.0, 183359 MiB, 580.95.05
|
| 13 |
+
2, NVIDIA B200, 10.0, 183359 MiB, 580.95.05
|
| 14 |
+
3, NVIDIA B200, 10.0, 183359 MiB, 580.95.05
|
| 15 |
+
4, NVIDIA B200, 10.0, 183359 MiB, 580.95.05
|
| 16 |
+
5, NVIDIA B200, 10.0, 183359 MiB, 580.95.05
|
| 17 |
+
6, NVIDIA B200, 10.0, 183359 MiB, 580.95.05
|
| 18 |
+
7, NVIDIA B200, 10.0, 183359 MiB, 580.95.05
|
| 19 |
+
persistence/clocks:
|
| 20 |
+
0, 120 MHz, 1965 MHz, 1000.00 W, 29
|
| 21 |
+
1, 120 MHz, 1965 MHz, 1000.00 W, 34
|
| 22 |
+
2, 120 MHz, 1965 MHz, 1000.00 W, 34
|
| 23 |
+
3, 120 MHz, 1965 MHz, 1000.00 W, 29
|
| 24 |
+
4, 120 MHz, 1965 MHz, 1000.00 W, 30
|
| 25 |
+
5, 120 MHz, 1965 MHz, 1000.00 W, 34
|
| 26 |
+
6, 120 MHz, 1965 MHz, 1000.00 W, 34
|
| 27 |
+
7, 120 MHz, 1965 MHz, 1000.00 W, 30
|
| 28 |
+
|
| 29 |
+
NVLink topology:
|
tiera-rerun-fixed-moonep/hf_watcher.log
CHANGED
|
@@ -55,3 +55,6 @@ https://huggingface.co/MoE-CUDA-Graph-Analysis/tierb-b200-gbs8/tree/main
|
|
| 55 |
[2026-08-30T08:00:57+00:00] backup pass
|
| 56 |
uploaded: /workspace/tiera-rerun-fixed-moonep -> MoE-CUDA-Graph-Analysis/tierb-b200-gbs8/tiera-rerun-fixed-moonep
|
| 57 |
https://huggingface.co/MoE-CUDA-Graph-Analysis/tierb-b200-gbs8/tree/main
|
|
|
|
|
|
|
|
|
|
|
|
| 55 |
[2026-08-30T08:00:57+00:00] backup pass
|
| 56 |
uploaded: /workspace/tiera-rerun-fixed-moonep -> MoE-CUDA-Graph-Analysis/tierb-b200-gbs8/tiera-rerun-fixed-moonep
|
| 57 |
https://huggingface.co/MoE-CUDA-Graph-Analysis/tierb-b200-gbs8/tree/main
|
| 58 |
+
Already logged in to Hugging Face as: AbhishekGhosh1998
|
| 59 |
+
Watching /workspace/tiera-rerun-fixed-moonep -> HF every 600s (kill to stop)
|
| 60 |
+
[2026-08-30T08:03:48+00:00] backup pass
|
tiera-rerun-fixed-moonep/hf_watcher.pid
CHANGED
|
@@ -1 +1 @@
|
|
| 1 |
-
|
|
|
|
| 1 |
+
841364
|
tiera-rerun-fixed-moonep/study.log
CHANGED
|
@@ -1101,3 +1101,36 @@ ECHO note: echo_logged_pre_rank_metric_valid=0 and echo_logged_load_reduction_va
|
|
| 1101 |
The branch's built-in fields mix source-rank and destination-rank axes; this tool reconstructs both globally.
|
| 1102 |
no memory-dump runs found under /workspace/tiera-rerun-fixed-moonep
|
| 1103 |
Already logged in to Hugging Face as: AbhishekGhosh1998
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1101 |
The branch's built-in fields mix source-rank and destination-rank axes; this tool reconstructs both globally.
|
| 1102 |
no memory-dump runs found under /workspace/tiera-rerun-fixed-moonep
|
| 1103 |
Already logged in to Hugging Face as: AbhishekGhosh1998
|
| 1104 |
+
uploaded: /workspace/tiera-rerun-fixed-moonep -> MoE-CUDA-Graph-Analysis/tierb-b200-gbs8/tiera-rerun-fixed-moonep
|
| 1105 |
+
https://huggingface.co/MoE-CUDA-Graph-Analysis/tierb-b200-gbs8/tree/main
|
| 1106 |
+
|
| 1107 |
+
=== STUDY COMPLETE
|
| 1108 |
+
results : /workspace/tiera-rerun-fixed-moonep
|
| 1109 |
+
status : /workspace/tiera-rerun-fixed-moonep/STATUS.txt
|
| 1110 |
+
steps : /workspace/tiera-rerun-fixed-moonep/steps.tsv
|
| 1111 |
+
hub : https://huggingface.co/datasets/MoE-CUDA-Graph-Analysis/tierb-b200-gbs8
|
| 1112 |
+
|
| 1113 |
+
status campaign phase gbs started ended result_dir
|
| 1114 |
+
FAIL echo tierbprobe 16 2026-08-30T04:52:38+00:00 2026-08-30T05:06:38+00:00 /workspace/tiera-rerun-fixed-moonep/echo/tierbprobe
|
| 1115 |
+
FAIL ultraep tierbprobe 16 2026-08-30T05:06:44+00:00 2026-08-30T05:20:40+00:00 /workspace/tiera-rerun-fixed-moonep/ultraep/tierbprobe
|
| 1116 |
+
OK echo a1 16 2026-08-30T06:20:31+00:00 2026-08-30T06:36:28+00:00 /workspace/tiera-rerun-fixed-moonep/echo/a1
|
| 1117 |
+
OK echo ponly 16 2026-08-30T06:36:31+00:00 2026-08-30T06:44:11+00:00 /workspace/tiera-rerun-fixed-moonep/echo/ponly
|
| 1118 |
+
OK echo a2 16 2026-08-30T06:44:13+00:00 2026-08-30T07:00:25+00:00 /workspace/tiera-rerun-fixed-moonep/echo/a2
|
| 1119 |
+
OK ultraep a1 16 2026-08-30T07:00:27+00:00 2026-08-30T07:15:17+00:00 /workspace/tiera-rerun-fixed-moonep/ultraep/a1
|
| 1120 |
+
OK ultraep ponly 16 2026-08-30T07:15:19+00:00 2026-08-30T07:22:01+00:00 /workspace/tiera-rerun-fixed-moonep/ultraep/ponly
|
| 1121 |
+
OK ultraep a2 16 2026-08-30T07:22:03+00:00 2026-08-30T07:37:07+00:00 /workspace/tiera-rerun-fixed-moonep/ultraep/a2
|
| 1122 |
+
OK moonep a1 16 2026-08-30T07:37:09+00:00 2026-08-30T07:41:05+00:00 /workspace/tiera-rerun-fixed-moonep/moonep/a1
|
| 1123 |
+
OK moonep ponly 16 2026-08-30T07:41:07+00:00 2026-08-30T07:48:20+00:00 /workspace/tiera-rerun-fixed-moonep/moonep/ponly
|
| 1124 |
+
OK moonep a2 16 2026-08-30T07:48:22+00:00 2026-08-30T08:03:34+00:00 /workspace/tiera-rerun-fixed-moonep/moonep/a2
|
| 1125 |
+
study_root: /workspace/tiera-rerun-fixed-moonep
|
| 1126 |
+
gbs: 16
|
| 1127 |
+
stages: 8
|
| 1128 |
+
nsys: 0
|
| 1129 |
+
hf_repo: MoE-CUDA-Graph-Analysis/tierb-b200-gbs8
|
| 1130 |
+
hf_remote_root: tiera-rerun-fixed-moonep
|
| 1131 |
+
started: 2026-08-30T08:03:47+00:00
|
| 1132 |
+
driver_repo: /workspace/blackwell_run_scripts @ f69d9ab
|
| 1133 |
+
|
| 1134 |
+
=== STAGE 8 -- Tier B allocator dumps (static CG capacities + dynamic no-CG)
|
| 1135 |
+
|
| 1136 |
+
=== STEP echo__tierbmem__gbs16
|