granularity results tierb-fixed-moonep-full @ 2026-08-30T02:50:07Z from sp-aiinfra-interactive-aghosh-7jtjl-8xb200-moe-cg-0 (part 2)
Browse files
tierb-fixed-moonep-full/steps.tsv
CHANGED
|
@@ -6,3 +6,4 @@ OK echo tierbtrace 8 2026-08-30T00:10:16+00:00 2026-08-30T00:38:30+00:00 /worksp
|
|
| 6 |
OK ultraep tierbtrace 8 2026-08-30T00:38:33+00:00 2026-08-30T00:59:28+00:00 /workspace/tierb-fixed-moonep-full/ultraep/tierbtrace
|
| 7 |
SUPERSEDED moonep tierbtrace 8 2026-08-30T00:59:40+00:00 2026-08-30T01:06:12+00:00 /workspace/tierb-fixed-moonep-full/moonep/tierbtrace
|
| 8 |
OK moonep tierb 8 2026-08-30T01:30:54+00:00 2026-08-30T01:46:01+00:00 /workspace/tierb-fixed-moonep-full/moonep/tierb
|
|
|
|
|
|
| 6 |
OK ultraep tierbtrace 8 2026-08-30T00:38:33+00:00 2026-08-30T00:59:28+00:00 /workspace/tierb-fixed-moonep-full/ultraep/tierbtrace
|
| 7 |
SUPERSEDED moonep tierbtrace 8 2026-08-30T00:59:40+00:00 2026-08-30T01:06:12+00:00 /workspace/tierb-fixed-moonep-full/moonep/tierbtrace
|
| 8 |
OK moonep tierb 8 2026-08-30T01:30:54+00:00 2026-08-30T01:46:01+00:00 /workspace/tierb-fixed-moonep-full/moonep/tierb
|
| 9 |
+
OK moonep tierbtrace 8 2026-08-30T02:38:13+00:00 2026-08-30T02:49:59+00:00 /workspace/tierb-fixed-moonep-full/moonep/tierbtrace
|
tierb-fixed-moonep-full/study.log
CHANGED
|
@@ -1114,3 +1114,86 @@ driver_repo: /workspace/blackwell_run_scripts @ e8bd2bd
|
|
| 1114 |
[guard] nsys stop (flushing report)
|
| 1115 |
[02:47:36] trace verified -> DSV2_LITE_L24__moonep__matched__seq8192__gbs8__capnone__async.nsys-rep (13M, all 8 GPUs + moonep markers)
|
| 1116 |
[02:47:37] TRACE-ONLY QWEN2_57B__moonep__matched__seq4096__gbs8__capnone__async (ranks: 0 1 2 3 4 5 6 7)
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1114 |
[guard] nsys stop (flushing report)
|
| 1115 |
[02:47:36] trace verified -> DSV2_LITE_L24__moonep__matched__seq8192__gbs8__capnone__async.nsys-rep (13M, all 8 GPUs + moonep markers)
|
| 1116 |
[02:47:37] TRACE-ONLY QWEN2_57B__moonep__matched__seq4096__gbs8__capnone__async (ranks: 0 1 2 3 4 5 6 7)
|
| 1117 |
+
[guard] iter 8/10 (60s)
|
| 1118 |
+
[guard] iter 10/10 (110s)
|
| 1119 |
+
[guard] nsys stop (flushing report)
|
| 1120 |
+
[02:49:59] trace verified -> QWEN2_57B__moonep__matched__seq4096__gbs8__capnone__async.nsys-rep (5.6M, all 8 GPUs + moonep markers)
|
| 1121 |
+
--------------------------------------------------------------------------
|
| 1122 |
+
|
| 1123 |
+
==============================================================
|
| 1124 |
+
SWEEP COMPLETE ok=4 failed=0
|
| 1125 |
+
==============================================================
|
| 1126 |
+
config backend policy slots seq gbs cap recompute rep status iter_ms_mean iter_ms_min peak_mib run_dir
|
| 1127 |
+
|
| 1128 |
+
Results: /workspace/tierb-fixed-moonep-full/moonep/tierbtrace/tierbtrace_gbs8/results.tsv
|
| 1129 |
+
Roll up: python3 /workspace/blackwell_run_scripts/granularity/tools/collect.py /workspace/tierb-fixed-moonep-full/moonep/tierbtrace/tierbtrace_gbs8
|
| 1130 |
+
==============================================================
|
| 1131 |
+
|
| 1132 |
+
Campaign results: /workspace/tierb-fixed-moonep-full/moonep/tierbtrace
|
| 1133 |
+
Roll up: python3 /workspace/blackwell_run_scripts/granularity/tools/collect.py /workspace/tierb-fixed-moonep-full/moonep/tierbtrace --compare
|
| 1134 |
+
Already logged in to Hugging Face as: AbhishekGhosh1998
|
| 1135 |
+
uploaded: /workspace/tierb-fixed-moonep-full/moonep/tierbtrace -> MoE-CUDA-Graph-Analysis/tierb-b200-gbs8/tierb-fixed-moonep-full/moonep/tierbtrace
|
| 1136 |
+
https://huggingface.co/MoE-CUDA-Graph-Analysis/tierb-b200-gbs8/tree/main
|
| 1137 |
+
|
| 1138 |
+
=== ROLL-UP
|
| 1139 |
+
40 runs ok=36 oom=0 -> /workspace/tierb-fixed-moonep-full/collected_all.tsv
|
| 1140 |
+
|
| 1141 |
+
(excluded 12 nsys-traced run(s) from the timing comparison)
|
| 1142 |
+
|
| 1143 |
+
phase config seq gbs policy rc cg asy backend iter_ms vs_best
|
| 1144 |
+
-------------------------------------------------------------------------------------
|
| 1145 |
+
tierb DSV2_LITE_L24 8192 8 matched none 1 1 moonep 593.43 1.00x
|
| 1146 |
+
|
| 1147 |
+
tierb QWEN2_57B 4096 8 matched none 1 1 moonep 180.12 1.00x
|
| 1148 |
+
|
| 1149 |
+
tierb QWEN3_235B 8192 8 matched none 1 1 moonep 317.80 1.00x
|
| 1150 |
+
|
| 1151 |
+
tierb QWEN3_30B 4096 8 matched none 1 1 moonep 355.16 1.00x
|
| 1152 |
+
|
| 1153 |
+
tierbmem DSV2_LITE_L24 8192 8 matched none 1 1 moonep 622.80 1.00x
|
| 1154 |
+
|
| 1155 |
+
tierbmem QWEN2_57B 4096 8 matched none 1 1 moonep 196.60 1.00x
|
| 1156 |
+
|
| 1157 |
+
tierbmem QWEN3_235B 8192 8 matched none 1 1 moonep 341.70 1.00x
|
| 1158 |
+
|
| 1159 |
+
tierbmem QWEN3_30B 4096 8 matched none 1 1 moonep 435.30 1.00x
|
| 1160 |
+
|
| 1161 |
+
tierbprobeDSV2_LITE_L24 8192 8 matched none 0 1 moonep 673.94 1.00x
|
| 1162 |
+
|
| 1163 |
+
tierbprobeQWEN2_57B 4096 8 matched none 0 1 moonep 238.91 1.00x
|
| 1164 |
+
|
| 1165 |
+
tierbprobeQWEN3_235B 8192 8 matched none 0 1 moonep 384.76 1.00x
|
| 1166 |
+
|
| 1167 |
+
tierbprobeQWEN3_30B 4096 8 matched none 0 1 moonep 824.23 1.00x
|
| 1168 |
+
|
| 1169 |
+
8 probe runs -> /workspace/tierb-fixed-moonep-full/probe_summary_all.tsv
|
| 1170 |
+
run arm cap rawExpI preRankI postRankI drop% reqCap
|
| 1171 |
+
--------------------------------------------------------------------------------------------------------------------
|
| 1172 |
+
moonep/tierbprobe/tierbprobe_gbs8/DSV2_LITE_L24__moo natural none 7.5443* 2.1316 1.1058
|
| 1173 |
+
moonep/tierbprobe/tierbprobe_gbs8/DSV2_LITE_L24__moo forced none 1.0286 1.0061 1.0026
|
| 1174 |
+
moonep/tierbprobe/tierbprobe_gbs8/QWEN2_57B__moonep_ natural none 7.4132* 2.3019 1.0979
|
| 1175 |
+
moonep/tierbprobe/tierbprobe_gbs8/QWEN2_57B__moonep_ forced none 1.0348 1.0073 1.0033
|
| 1176 |
+
moonep/tierbprobe/tierbprobe_gbs8/QWEN3_235B__moonep natural none 15.9227* 2.5851 1.0712
|
| 1177 |
+
moonep/tierbprobe/tierbprobe_gbs8/QWEN3_235B__moonep forced none 1.0386 1.0053 1.0025
|
| 1178 |
+
moonep/tierbprobe/tierbprobe_gbs8/QWEN3_30B__moonep_ natural none 15.7078* 2.4672 1.0635
|
| 1179 |
+
moonep/tierbprobe/tierbprobe_gbs8/QWEN3_30B__moonep_ forced none 1.0565 1.0077 1.0036
|
| 1180 |
+
|
| 1181 |
+
* At least one globally aggregated snapshot reached the E/K expert-Imax ceiling.
|
| 1182 |
+
Inspect raw_expert_imax_mean and hottest_expert_token_fraction_mean before generalising.
|
| 1183 |
+
|
| 1184 |
+
ECHO note: echo_logged_pre_rank_metric_valid=0 and echo_logged_load_reduction_valid=0 are intentional.
|
| 1185 |
+
The branch's built-in fields mix source-rank and destination-rank axes; this tool reconstructs both globally.
|
| 1186 |
+
8 memory-dump runs -> /workspace/tierb-fixed-moonep-full/memory_summary_all.tsv
|
| 1187 |
+
model backend cg allocation capacity status peak GiB dumps
|
| 1188 |
+
----------------------------------------------------------------------------------
|
| 1189 |
+
DSV2_LITE_L24 moonep 1 none none OK 130.2 48/48
|
| 1190 |
+
DSV2_LITE_L24 moonep 0 none none OK 122.2 40/40
|
| 1191 |
+
QWEN2_57B moonep 1 none none OK 94.5 48/48
|
| 1192 |
+
QWEN2_57B moonep 0 none none OK 93.6 40/40
|
| 1193 |
+
QWEN3_235B moonep 1 none none OK 117.5 48/48
|
| 1194 |
+
QWEN3_235B moonep 0 none none OK 117.3 40/40
|
| 1195 |
+
QWEN3_30B moonep 1 none none OK 114.0 48/48
|
| 1196 |
+
QWEN3_30B moonep 0 none none OK 113.7 40/40
|
| 1197 |
+
|
| 1198 |
+
ECHO/UltraEP note: graph-on uses static capacity + device-initiated GEMM while graph-off uses dynamic allocation + host-derived sizes. Their difference is not labelled as pure graph cost.
|
| 1199 |
+
Already logged in to Hugging Face as: AbhishekGhosh1998
|