granularity results tiera-rerun-fixed-moonep @ 2026-08-30T08:12:09Z from sp-aiinfra-interactive-aghosh-7jtjl-8xb200-moe-cg-0 (part 3)
Browse files
tiera-rerun-fixed-moonep/steps.tsv
CHANGED
|
@@ -10,3 +10,6 @@ OK ultraep a2 16 2026-08-30T07:22:03+00:00 2026-08-30T07:37:07+00:00 /workspace/
|
|
| 10 |
OK moonep a1 16 2026-08-30T07:37:09+00:00 2026-08-30T07:41:05+00:00 /workspace/tiera-rerun-fixed-moonep/moonep/a1
|
| 11 |
OK moonep ponly 16 2026-08-30T07:41:07+00:00 2026-08-30T07:48:20+00:00 /workspace/tiera-rerun-fixed-moonep/moonep/ponly
|
| 12 |
OK moonep a2 16 2026-08-30T07:48:22+00:00 2026-08-30T08:03:34+00:00 /workspace/tiera-rerun-fixed-moonep/moonep/a2
|
|
|
|
|
|
|
|
|
|
|
|
| 10 |
OK moonep a1 16 2026-08-30T07:37:09+00:00 2026-08-30T07:41:05+00:00 /workspace/tiera-rerun-fixed-moonep/moonep/a1
|
| 11 |
OK moonep ponly 16 2026-08-30T07:41:07+00:00 2026-08-30T07:48:20+00:00 /workspace/tiera-rerun-fixed-moonep/moonep/ponly
|
| 12 |
OK moonep a2 16 2026-08-30T07:48:22+00:00 2026-08-30T08:03:34+00:00 /workspace/tiera-rerun-fixed-moonep/moonep/a2
|
| 13 |
+
OK echo tierbmem 16 2026-08-30T08:03:47+00:00 2026-08-30T08:06:37+00:00 /workspace/tiera-rerun-fixed-moonep/echo/tierbmem
|
| 14 |
+
OK ultraep tierbmem 16 2026-08-30T08:06:42+00:00 2026-08-30T08:09:23+00:00 /workspace/tiera-rerun-fixed-moonep/ultraep/tierbmem
|
| 15 |
+
OK moonep tierbmem 16 2026-08-30T08:09:29+00:00 2026-08-30T08:11:55+00:00 /workspace/tiera-rerun-fixed-moonep/moonep/tierbmem
|
tiera-rerun-fixed-moonep/study.log
CHANGED
|
@@ -1134,3 +1134,319 @@ driver_repo: /workspace/blackwell_run_scripts @ f69d9ab
|
|
| 1134 |
=== STAGE 8 -- Tier B allocator dumps (static CG capacities + dynamic no-CG)
|
| 1135 |
|
| 1136 |
=== STEP echo__tierbmem__gbs16
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1134 |
=== STAGE 8 -- Tier B allocator dumps (static CG capacities + dynamic no-CG)
|
| 1135 |
|
| 1136 |
=== STEP echo__tierbmem__gbs16
|
| 1137 |
+
==============================================================
|
| 1138 |
+
CAMPAIGN: echo backend=echo phase=tierbmem
|
| 1139 |
+
Megatron: /workspace/Megatron-MoE-Echo @ better-memory-dump-support
|
| 1140 |
+
GBS=16 seq=4096 repeats=3 nsys=0
|
| 1141 |
+
results -> /workspace/tiera-rerun-fixed-moonep/echo/tierbmem
|
| 1142 |
+
==============================================================
|
| 1143 |
+
--------------------------------------------------------------------------
|
| 1144 |
+
[08:04:02] CELL G1__echo__matched__seq4096__gbs16__cap2p40625__memiteration__it6 (slots=1)
|
| 1145 |
+
[guard] iter 6/6 (70s)
|
| 1146 |
+
[guard] tearing down process group (outcome=DONE)
|
| 1147 |
+
--------------------------------------------------------------------------
|
| 1148 |
+
[08:05:17] CELL G1__echo__matched__seq4096__gbs16__capdynamic__nocg__memiteration__it5 (slots=1)
|
| 1149 |
+
[guard] iter 2/5 (70s)
|
| 1150 |
+
[guard] iter 5/5 (80s)
|
| 1151 |
+
--------------------------------------------------------------------------
|
| 1152 |
+
|
| 1153 |
+
==============================================================
|
| 1154 |
+
SWEEP COMPLETE ok=2 failed=0
|
| 1155 |
+
==============================================================
|
| 1156 |
+
config backend policy slots seq gbs cap recompute rep status iter_ms_mean iter_ms_min peak_mib run_dir
|
| 1157 |
+
G1 echo matched 1 4096 16 2.40625 none 1 OK 474.40 474.40 161723 /workspace/tiera-rerun-fixed-moonep/echo/tierbmem/tierbmem_gbs16/G1__echo__matched__seq4096__gbs16__cap2p40625__memiteration__it6/rep1
|
| 1158 |
+
G1 echo matched 1 4096 16 dynamic none 1 OK NA NA 115283 /workspace/tiera-rerun-fixed-moonep/echo/tierbmem/tierbmem_gbs16/G1__echo__matched__seq4096__gbs16__capdynamic__nocg__memiteration__it5/rep1
|
| 1159 |
+
|
| 1160 |
+
Results: /workspace/tiera-rerun-fixed-moonep/echo/tierbmem/tierbmem_gbs16/results.tsv
|
| 1161 |
+
Roll up: python3 /workspace/blackwell_run_scripts/granularity/tools/collect.py /workspace/tiera-rerun-fixed-moonep/echo/tierbmem/tierbmem_gbs16
|
| 1162 |
+
==============================================================
|
| 1163 |
+
2 memory-dump runs -> /workspace/tiera-rerun-fixed-moonep/echo/tierbmem/tierbmem_gbs16/memory_summary.tsv
|
| 1164 |
+
model backend cg allocation capacity status peak GiB dumps
|
| 1165 |
+
----------------------------------------------------------------------------------
|
| 1166 |
+
G1 echo 1 static 2.40625 OK 157.9 48/48
|
| 1167 |
+
G1 echo 0 dynamic dynamic OK 112.6 40/40
|
| 1168 |
+
|
| 1169 |
+
ECHO/UltraEP note: graph-on uses static capacity + device-initiated GEMM while graph-off uses dynamic allocation + host-derived sizes. Their difference is not labelled as pure graph cost.
|
| 1170 |
+
|
| 1171 |
+
TIER B MEMORY DUMPS -- how to read this
|
| 1172 |
+
---------------------------------------
|
| 1173 |
+
ECHO / UltraEP:
|
| 1174 |
+
cap1p00__memiteration__it6 graph on, static C=1.00
|
| 1175 |
+
cap1p125__memiteration__it6 graph on, static C=1.125
|
| 1176 |
+
cap1p25__memiteration__it6 graph on, static C=1.25
|
| 1177 |
+
capdynamic__nocg__memiteration__it5
|
| 1178 |
+
graph off, no capacity flag, dynamic allocation
|
| 1179 |
+
|
| 1180 |
+
MoonEP:
|
| 1181 |
+
capnone__memiteration__it6 graph on, no capacity factor
|
| 1182 |
+
capnone__nocg__memiteration__it5 graph off, no capacity factor
|
| 1183 |
+
|
| 1184 |
+
Every successful cell contains:
|
| 1185 |
+
train.log Megatron and OOM diagnostics
|
| 1186 |
+
mem_samples.csv driver-level nvidia-smi samples every 2 seconds
|
| 1187 |
+
run.env exact branch/config/capacity provenance
|
| 1188 |
+
memory_snapshots/*.pickle 48 captured / 40 eager allocator histories
|
| 1189 |
+
memory_snapshots/*.json matching summaries and before/after deltas
|
| 1190 |
+
|
| 1191 |
+
The graph-on/static vs graph-off/dynamic difference is intentionally NOT called
|
| 1192 |
+
"graph cost": capacity allocation policy and device-initiated grouped GEMM also
|
| 1193 |
+
change. Compare the three graph-on rows to measure static-capacity sensitivity,
|
| 1194 |
+
and use the dynamic row as the natural eager-allocation reference.
|
| 1195 |
+
|
| 1196 |
+
Campaign results: /workspace/tiera-rerun-fixed-moonep/echo/tierbmem
|
| 1197 |
+
Roll up: python3 /workspace/blackwell_run_scripts/granularity/tools/collect.py /workspace/tiera-rerun-fixed-moonep/echo/tierbmem --compare
|
| 1198 |
+
Already logged in to Hugging Face as: AbhishekGhosh1998
|
| 1199 |
+
uploaded: /workspace/tiera-rerun-fixed-moonep/echo/tierbmem -> MoE-CUDA-Graph-Analysis/tierb-b200-gbs8/tiera-rerun-fixed-moonep/echo/tierbmem
|
| 1200 |
+
https://huggingface.co/MoE-CUDA-Graph-Analysis/tierb-b200-gbs8/tree/main
|
| 1201 |
+
|
| 1202 |
+
=== STEP ultraep__tierbmem__gbs16
|
| 1203 |
+
==============================================================
|
| 1204 |
+
CAMPAIGN: ultraep backend=ultraep phase=tierbmem
|
| 1205 |
+
Megatron: /workspace/Megatron-UltraEP @ ultraep-better-memory-dump-support
|
| 1206 |
+
GBS=16 seq=4096 repeats=3 nsys=0
|
| 1207 |
+
results -> /workspace/tiera-rerun-fixed-moonep/ultraep/tierbmem
|
| 1208 |
+
==============================================================
|
| 1209 |
+
--------------------------------------------------------------------------
|
| 1210 |
+
[08:06:58] CELL G1__ultraep__matched__seq4096__gbs16__cap2p40625__memiteration__it6 (slots=1)
|
| 1211 |
+
[guard] iter 6/6 (60s)
|
| 1212 |
+
[guard] tearing down process group (outcome=DONE)
|
| 1213 |
+
--------------------------------------------------------------------------
|
| 1214 |
+
[08:08:00] CELL G1__ultraep__matched__seq4096__gbs16__capdynamic__nocg__memiteration__it5 (slots=1)
|
| 1215 |
+
[08:08:02] waiting for stale ranks to exit ...
|
| 1216 |
+
[guard] iter 4/5 (60s)
|
| 1217 |
+
[guard] iter 5/5 (70s)
|
| 1218 |
+
[guard] tearing down process group (outcome=DONE)
|
| 1219 |
+
--------------------------------------------------------------------------
|
| 1220 |
+
|
| 1221 |
+
==============================================================
|
| 1222 |
+
SWEEP COMPLETE ok=2 failed=0
|
| 1223 |
+
==============================================================
|
| 1224 |
+
config backend policy slots seq gbs cap recompute rep status iter_ms_mean iter_ms_min peak_mib run_dir
|
| 1225 |
+
G1 ultraep matched 1 4096 16 2.40625 none 1 OK 909.90 909.90 129473 /workspace/tiera-rerun-fixed-moonep/ultraep/tierbmem/tierbmem_gbs16/G1__ultraep__matched__seq4096__gbs16__cap2p40625__memiteration__it6/rep1
|
| 1226 |
+
G1 ultraep matched 1 4096 16 dynamic none 1 OK NA NA 109261 /workspace/tiera-rerun-fixed-moonep/ultraep/tierbmem/tierbmem_gbs16/G1__ultraep__matched__seq4096__gbs16__capdynamic__nocg__memiteration__it5/rep1
|
| 1227 |
+
|
| 1228 |
+
Results: /workspace/tiera-rerun-fixed-moonep/ultraep/tierbmem/tierbmem_gbs16/results.tsv
|
| 1229 |
+
Roll up: python3 /workspace/blackwell_run_scripts/granularity/tools/collect.py /workspace/tiera-rerun-fixed-moonep/ultraep/tierbmem/tierbmem_gbs16
|
| 1230 |
+
==============================================================
|
| 1231 |
+
2 memory-dump runs -> /workspace/tiera-rerun-fixed-moonep/ultraep/tierbmem/tierbmem_gbs16/memory_summary.tsv
|
| 1232 |
+
model backend cg allocation capacity status peak GiB dumps
|
| 1233 |
+
----------------------------------------------------------------------------------
|
| 1234 |
+
G1 ultraep 1 static 2.40625 OK 126.4 48/48
|
| 1235 |
+
G1 ultraep 0 dynamic dynamic OK 106.7 40/40
|
| 1236 |
+
|
| 1237 |
+
ECHO/UltraEP note: graph-on uses static capacity + device-initiated GEMM while graph-off uses dynamic allocation + host-derived sizes. Their difference is not labelled as pure graph cost.
|
| 1238 |
+
|
| 1239 |
+
TIER B MEMORY DUMPS -- how to read this
|
| 1240 |
+
---------------------------------------
|
| 1241 |
+
ECHO / UltraEP:
|
| 1242 |
+
cap1p00__memiteration__it6 graph on, static C=1.00
|
| 1243 |
+
cap1p125__memiteration__it6 graph on, static C=1.125
|
| 1244 |
+
cap1p25__memiteration__it6 graph on, static C=1.25
|
| 1245 |
+
capdynamic__nocg__memiteration__it5
|
| 1246 |
+
graph off, no capacity flag, dynamic allocation
|
| 1247 |
+
|
| 1248 |
+
MoonEP:
|
| 1249 |
+
capnone__memiteration__it6 graph on, no capacity factor
|
| 1250 |
+
capnone__nocg__memiteration__it5 graph off, no capacity factor
|
| 1251 |
+
|
| 1252 |
+
Every successful cell contains:
|
| 1253 |
+
train.log Megatron and OOM diagnostics
|
| 1254 |
+
mem_samples.csv driver-level nvidia-smi samples every 2 seconds
|
| 1255 |
+
run.env exact branch/config/capacity provenance
|
| 1256 |
+
memory_snapshots/*.pickle 48 captured / 40 eager allocator histories
|
| 1257 |
+
memory_snapshots/*.json matching summaries and before/after deltas
|
| 1258 |
+
|
| 1259 |
+
The graph-on/static vs graph-off/dynamic difference is intentionally NOT called
|
| 1260 |
+
"graph cost": capacity allocation policy and device-initiated grouped GEMM also
|
| 1261 |
+
change. Compare the three graph-on rows to measure static-capacity sensitivity,
|
| 1262 |
+
and use the dynamic row as the natural eager-allocation reference.
|
| 1263 |
+
|
| 1264 |
+
Campaign results: /workspace/tiera-rerun-fixed-moonep/ultraep/tierbmem
|
| 1265 |
+
Roll up: python3 /workspace/blackwell_run_scripts/granularity/tools/collect.py /workspace/tiera-rerun-fixed-moonep/ultraep/tierbmem --compare
|
| 1266 |
+
Already logged in to Hugging Face as: AbhishekGhosh1998
|
| 1267 |
+
uploaded: /workspace/tiera-rerun-fixed-moonep/ultraep/tierbmem -> MoE-CUDA-Graph-Analysis/tierb-b200-gbs8/tiera-rerun-fixed-moonep/ultraep/tierbmem
|
| 1268 |
+
https://huggingface.co/MoE-CUDA-Graph-Analysis/tierb-b200-gbs8/tree/main
|
| 1269 |
+
|
| 1270 |
+
=== STEP moonep__tierbmem__gbs16
|
| 1271 |
+
==============================================================
|
| 1272 |
+
CAMPAIGN: moonep backend=moonep phase=tierbmem
|
| 1273 |
+
Megatron: /workspace/Megatron-MoE-Echo @ moonep_backend_async_grad_reduce_better_memory_dump_support
|
| 1274 |
+
GBS=16 seq=4096 repeats=3 nsys=0
|
| 1275 |
+
results -> /workspace/tiera-rerun-fixed-moonep/moonep/tierbmem
|
| 1276 |
+
==============================================================
|
| 1277 |
+
--------------------------------------------------------------------------
|
| 1278 |
+
[08:09:44] CELL G1__moonep__matched__seq4096__gbs16__capnone__async__memiteration__it6 (slots=1)
|
| 1279 |
+
[guard] iter 6/6 (60s)
|
| 1280 |
+
[guard] tearing down process group (outcome=DONE)
|
| 1281 |
+
--------------------------------------------------------------------------
|
| 1282 |
+
[08:10:49] CELL G1__moonep__matched__seq4096__gbs16__capnone__nocg__async__memiteration__it5 (slots=1)
|
| 1283 |
+
[guard] iter 5/5 (60s)
|
| 1284 |
+
[guard] tearing down process group (outcome=DONE)
|
| 1285 |
+
--------------------------------------------------------------------------
|
| 1286 |
+
|
| 1287 |
+
==============================================================
|
| 1288 |
+
SWEEP COMPLETE ok=2 failed=0
|
| 1289 |
+
==============================================================
|
| 1290 |
+
config backend policy slots seq gbs cap recompute rep status iter_ms_mean iter_ms_min peak_mib run_dir
|
| 1291 |
+
G1 moonep matched 1 4096 16 none none 1 OK 497.40 497.40 96849 /workspace/tiera-rerun-fixed-moonep/moonep/tierbmem/tierbmem_gbs16/G1__moonep__matched__seq4096__gbs16__capnone__async__memiteration__it6/rep1
|
| 1292 |
+
G1 moonep matched 1 4096 16 none none 1 OK NA NA 92579 /workspace/tiera-rerun-fixed-moonep/moonep/tierbmem/tierbmem_gbs16/G1__moonep__matched__seq4096__gbs16__capnone__nocg__async__memiteration__it5/rep1
|
| 1293 |
+
|
| 1294 |
+
Results: /workspace/tiera-rerun-fixed-moonep/moonep/tierbmem/tierbmem_gbs16/results.tsv
|
| 1295 |
+
Roll up: python3 /workspace/blackwell_run_scripts/granularity/tools/collect.py /workspace/tiera-rerun-fixed-moonep/moonep/tierbmem/tierbmem_gbs16
|
| 1296 |
+
==============================================================
|
| 1297 |
+
2 memory-dump runs -> /workspace/tiera-rerun-fixed-moonep/moonep/tierbmem/tierbmem_gbs16/memory_summary.tsv
|
| 1298 |
+
model backend cg allocation capacity status peak GiB dumps
|
| 1299 |
+
----------------------------------------------------------------------------------
|
| 1300 |
+
G1 moonep 1 none none OK 94.6 48/48
|
| 1301 |
+
G1 moonep 0 none none OK 90.4 40/40
|
| 1302 |
+
|
| 1303 |
+
ECHO/UltraEP note: graph-on uses static capacity + device-initiated GEMM while graph-off uses dynamic allocation + host-derived sizes. Their difference is not labelled as pure graph cost.
|
| 1304 |
+
|
| 1305 |
+
TIER B MEMORY DUMPS -- how to read this
|
| 1306 |
+
---------------------------------------
|
| 1307 |
+
ECHO / UltraEP:
|
| 1308 |
+
cap1p00__memiteration__it6 graph on, static C=1.00
|
| 1309 |
+
cap1p125__memiteration__it6 graph on, static C=1.125
|
| 1310 |
+
cap1p25__memiteration__it6 graph on, static C=1.25
|
| 1311 |
+
capdynamic__nocg__memiteration__it5
|
| 1312 |
+
graph off, no capacity flag, dynamic allocation
|
| 1313 |
+
|
| 1314 |
+
MoonEP:
|
| 1315 |
+
capnone__memiteration__it6 graph on, no capacity factor
|
| 1316 |
+
capnone__nocg__memiteration__it5 graph off, no capacity factor
|
| 1317 |
+
|
| 1318 |
+
Every successful cell contains:
|
| 1319 |
+
train.log Megatron and OOM diagnostics
|
| 1320 |
+
mem_samples.csv driver-level nvidia-smi samples every 2 seconds
|
| 1321 |
+
run.env exact branch/config/capacity provenance
|
| 1322 |
+
memory_snapshots/*.pickle 48 captured / 40 eager allocator histories
|
| 1323 |
+
memory_snapshots/*.json matching summaries and before/after deltas
|
| 1324 |
+
|
| 1325 |
+
The graph-on/static vs graph-off/dynamic difference is intentionally NOT called
|
| 1326 |
+
"graph cost": capacity allocation policy and device-initiated grouped GEMM also
|
| 1327 |
+
change. Compare the three graph-on rows to measure static-capacity sensitivity,
|
| 1328 |
+
and use the dynamic row as the natural eager-allocation reference.
|
| 1329 |
+
|
| 1330 |
+
Campaign results: /workspace/tiera-rerun-fixed-moonep/moonep/tierbmem
|
| 1331 |
+
Roll up: python3 /workspace/blackwell_run_scripts/granularity/tools/collect.py /workspace/tiera-rerun-fixed-moonep/moonep/tierbmem --compare
|
| 1332 |
+
Already logged in to Hugging Face as: AbhishekGhosh1998
|
| 1333 |
+
uploaded: /workspace/tiera-rerun-fixed-moonep/moonep/tierbmem -> MoE-CUDA-Graph-Analysis/tierb-b200-gbs8/tiera-rerun-fixed-moonep/moonep/tierbmem
|
| 1334 |
+
https://huggingface.co/MoE-CUDA-Graph-Analysis/tierb-b200-gbs8/tree/main
|
| 1335 |
+
|
| 1336 |
+
=== ROLL-UP
|
| 1337 |
+
110 runs ok=107 oom=0 -> /workspace/tiera-rerun-fixed-moonep/collected_all.tsv
|
| 1338 |
+
|
| 1339 |
+
WARNING: 11 run(s) mention capacity saturation. A run that drops assignments does less expert work and must not be quoted as a win:
|
| 1340 |
+
ultraep/tierbprobe/tierbprobe_gbs16/G1__ultraep__matched__seq4096__gbs16__cap1p25/rep1: WARNING:megatron.training.drop_debug_utils:[MOE_DROP_DEBUG][iteration_summary] global_rank=0 iteration=0 records=32 total_expected_recv=283999 total_dropped_tok
|
| 1341 |
+
ultraep/tierbprobe/tierbprobe_gbs16/G1__ultraep__matched__seq4096__gbs16__cap1p8125/rep1: WARNING:megatron.training.drop_debug_utils:[MOE_DROP_DEBUG][iteration_summary] global_rank=0 iteration=0 records=32 total_expected_recv=284540 total_dropped_tok
|
| 1342 |
+
ultraep/tierbprobe/tierbprobe_gbs16/G2__ultraep__matched__seq4096__gbs16__cap1p0625/rep1: WARNING:megatron.training.drop_debug_utils:[MOE_DROP_DEBUG][iteration_summary] global_rank=0 iteration=0 records=32 total_expected_recv=1048800 total_dropped_to
|
| 1343 |
+
ultraep/tierbprobe/tierbprobe_gbs16/G2__ultraep__matched__seq4096__gbs16__cap1p25/rep1: WARNING:megatron.training.drop_debug_utils:[MOE_DROP_DEBUG][iteration_summary] global_rank=0 iteration=0 records=32 total_expected_recv=1048800 total_dropped_to
|
| 1344 |
+
ultraep/tierbprobe/tierbprobe_gbs16/G2__ultraep__matched__seq4096__gbs16__cap1p25__forcebal/rep1: WARNING:megatron.training.drop_debug_utils:[MOE_DROP_DEBUG][iteration_summary] global_rank=1 iteration=0 records=32 total_expected_recv=1047984 total_dropped_to
|
| 1345 |
+
|
| 1346 |
+
(excluded 2 nsys-traced run(s) from the timing comparison)
|
| 1347 |
+
|
| 1348 |
+
phase config seq gbs policy rc cg asy backend iter_ms vs_best
|
| 1349 |
+
-------------------------------------------------------------------------------------
|
| 1350 |
+
a1 G1 4096 16 one none 1 0 echo 466.68 1.00x
|
| 1351 |
+
ultraep 891.00 1.91x
|
| 1352 |
+
|
| 1353 |
+
a1 G1 4096 16 one none 1 1 moonep 480.30 1.00x
|
| 1354 |
+
|
| 1355 |
+
a1 G2 4096 16 one none 1 0 echo 394.28 1.00x
|
| 1356 |
+
ultraep 466.65 1.18x
|
| 1357 |
+
|
| 1358 |
+
a1 G3 4096 16 one none 1 0 echo 412.83 1.00x
|
| 1359 |
+
ultraep 447.84 1.08x
|
| 1360 |
+
|
| 1361 |
+
a1 G4 4096 16 one none 1 0 echo 461.03 1.00x
|
| 1362 |
+
ultraep 467.57 1.01x
|
| 1363 |
+
|
| 1364 |
+
a2 G1 4096 16 matched none 1 0 echo 466.91 1.00x
|
| 1365 |
+
ultraep 891.92 1.91x
|
| 1366 |
+
|
| 1367 |
+
a2 G1 4096 16 matched none 1 1 moonep 481.37 1.00x
|
| 1368 |
+
|
| 1369 |
+
a2 G2 4096 16 matched none 1 0 echo 415.21 1.00x
|
| 1370 |
+
ultraep 492.54 1.19x
|
| 1371 |
+
|
| 1372 |
+
a2 G2 4096 16 matched none 1 1 moonep 432.02 1.00x
|
| 1373 |
+
|
| 1374 |
+
a2 G3 4096 16 matched none 1 0 echo 437.35 1.00x
|
| 1375 |
+
ultraep 459.60 1.05x
|
| 1376 |
+
|
| 1377 |
+
a2 G3 4096 16 matched none 1 1 moonep 559.88 1.00x
|
| 1378 |
+
|
| 1379 |
+
a2 G4 4096 16 matched none 1 0 ultraep 482.64 1.00x
|
| 1380 |
+
echo 486.54 1.01x
|
| 1381 |
+
|
| 1382 |
+
a2 G4 4096 16 matched none 1 1 moonep 783.34 1.00x
|
| 1383 |
+
|
| 1384 |
+
ponly PONLY 4096 16 one none 1 0 echo 174.81 1.00x
|
| 1385 |
+
ultraep 176.76 1.01x
|
| 1386 |
+
|
| 1387 |
+
ponly PONLY 4096 16 one none 1 1 moonep 165.22 1.00x
|
| 1388 |
+
|
| 1389 |
+
ponly PONLY 8192 16 one none 1 0 ultraep 292.62 1.00x
|
| 1390 |
+
echo 316.70 1.08x
|
| 1391 |
+
|
| 1392 |
+
ponly PONLY 8192 16 one none 1 1 moonep 300.61 1.00x
|
| 1393 |
+
|
| 1394 |
+
tierbmem G1 4096 16 matched none 1 0 echo 474.40 1.00x
|
| 1395 |
+
ultraep 909.90 1.92x
|
| 1396 |
+
|
| 1397 |
+
tierbmem G1 4096 16 matched none 1 1 moonep 497.40 1.00x
|
| 1398 |
+
|
| 1399 |
+
tierbprobeG1 4096 16 matched none 0 0 echo 671.53 1.00x
|
| 1400 |
+
ultraep 904.16 1.35x
|
| 1401 |
+
|
| 1402 |
+
tierbprobeG2 4096 16 matched none 0 0 ultraep 512.37 1.00x
|
| 1403 |
+
echo 654.76 1.28x
|
| 1404 |
+
|
| 1405 |
+
tierbprobeG3 4096 16 matched none 0 0 ultraep 512.31 1.00x
|
| 1406 |
+
echo 689.13 1.35x
|
| 1407 |
+
|
| 1408 |
+
tierbprobeG4 4096 16 matched none 0 0 ultraep 600.19 1.00x
|
| 1409 |
+
echo 790.97 1.32x
|
| 1410 |
+
|
| 1411 |
+
21 probe runs -> /workspace/tiera-rerun-fixed-moonep/probe_summary_all.tsv
|
| 1412 |
+
run arm cap rawExpI preRankI postRankI drop% reqCap
|
| 1413 |
+
--------------------------------------------------------------------------------------------------------------------
|
| 1414 |
+
echo/tierbprobe/tierbprobe_gbs16/G1__echo__matched__ natural 2.00 3.6198* 3.6198 1.0827 1.9062
|
| 1415 |
+
echo/tierbprobe/tierbprobe_gbs16/G2__echo__matched__ natural 1.08203125 3.8720* 2.1703 1.0000 1.0273
|
| 1416 |
+
echo/tierbprobe/tierbprobe_gbs16/G2__echo__matched__ forced 1.08203125 1.0201 1.0069 1.0000 1.0234
|
| 1417 |
+
echo/tierbprobe/tierbprobe_gbs16/G2__echo__matched__ natural 2.00 3.8724* 2.1875 1.0000 1.0273
|
| 1418 |
+
echo/tierbprobe/tierbprobe_gbs16/G3__echo__matched__ natural 1.072265625 3.9096* 1.7838 1.0000 1.0195
|
| 1419 |
+
echo/tierbprobe/tierbprobe_gbs16/G3__echo__matched__ forced 1.072265625 1.0231 1.0048 1.0000 1.0176
|
| 1420 |
+
echo/tierbprobe/tierbprobe_gbs16/G3__echo__matched__ natural 2.00 3.9096* 1.7916 1.0000 1.0195
|
| 1421 |
+
echo/tierbprobe/tierbprobe_gbs16/G4__echo__matched__ natural 1.068359375 3.9303* 1.5507 1.0000 1.0166
|
| 1422 |
+
echo/tierbprobe/tierbprobe_gbs16/G4__echo__matched__ forced 1.068359375 1.0249 1.0034 1.0000 1.0137
|
| 1423 |
+
echo/tierbprobe/tierbprobe_gbs16/G4__echo__matched__ natural 2.00 3.9301* 1.5543 1.0000 1.0166
|
| 1424 |
+
ultraep/tierbprobe/tierbprobe_gbs16/G1__ultraep__mat natural 1.25 3.6085 1.0640 2.8918 1.7246
|
| 1425 |
+
ultraep/tierbprobe/tierbprobe_gbs16/G1__ultraep__mat natural 1.8125 3.6152 1.0747 1.5460 2.3091
|
| 1426 |
+
ultraep/tierbprobe/tierbprobe_gbs16/G2__ultraep__mat natural 1.0625 2.2032 1.0100 0.0000 1.0101
|
| 1427 |
+
ultraep/tierbprobe/tierbprobe_gbs16/G2__ultraep__mat natural 1.25 2.2136 1.0100 0.0000 1.0101
|
| 1428 |
+
ultraep/tierbprobe/tierbprobe_gbs16/G2__ultraep__mat forced 1.25 1.0069 1.0067 0.0000 1.0100
|
| 1429 |
+
ultraep/tierbprobe/tierbprobe_gbs16/G3__ultraep__mat natural 1.0625 1.7842 1.0100 0.0000 1.0101
|
| 1430 |
+
ultraep/tierbprobe/tierbprobe_gbs16/G3__ultraep__mat natural 1.25 1.7752 1.0100 0.0000 1.0101
|
| 1431 |
+
ultraep/tierbprobe/tierbprobe_gbs16/G3__ultraep__mat forced 1.25 1.0047 1.0047 0.0000 1.0100
|
| 1432 |
+
ultraep/tierbprobe/tierbprobe_gbs16/G4__ultraep__mat natural 1.095703125 1.5612 1.0100 0.0000 1.0429
|
| 1433 |
+
ultraep/tierbprobe/tierbprobe_gbs16/G4__ultraep__mat natural 1.25 1.5504 1.0100 0.0000 1.0429
|
| 1434 |
+
ultraep/tierbprobe/tierbprobe_gbs16/G4__ultraep__mat forced 1.25 1.0033 1.0033 0.0000 1.0071
|
| 1435 |
+
|
| 1436 |
+
* At least one globally aggregated snapshot reached the E/K expert-Imax ceiling.
|
| 1437 |
+
Inspect raw_expert_imax_mean and hottest_expert_token_fraction_mean before generalising.
|
| 1438 |
+
|
| 1439 |
+
ECHO note: echo_logged_pre_rank_metric_valid=0 and echo_logged_load_reduction_valid=0 are intentional.
|
| 1440 |
+
The branch's built-in fields mix source-rank and destination-rank axes; this tool reconstructs both globally.
|
| 1441 |
+
6 memory-dump runs -> /workspace/tiera-rerun-fixed-moonep/memory_summary_all.tsv
|
| 1442 |
+
model backend cg allocation capacity status peak GiB dumps
|
| 1443 |
+
----------------------------------------------------------------------------------
|
| 1444 |
+
G1 echo 1 static 2.40625 OK 157.9 48/48
|
| 1445 |
+
G1 echo 0 dynamic dynamic OK 112.6 40/40
|
| 1446 |
+
G1 moonep 1 none none OK 94.6 48/48
|
| 1447 |
+
G1 moonep 0 none none OK 90.4 40/40
|
| 1448 |
+
G1 ultraep 1 static 2.40625 OK 126.4 48/48
|
| 1449 |
+
G1 ultraep 0 dynamic dynamic OK 106.7 40/40
|
| 1450 |
+
|
| 1451 |
+
ECHO/UltraEP note: graph-on uses static capacity + device-initiated GEMM while graph-off uses dynamic allocation + host-derived sizes. Their difference is not labelled as pure graph cost.
|
| 1452 |
+
Already logged in to Hugging Face as: AbhishekGhosh1998
|