AbhishekGhosh1998 commited on
Commit
61baa01
·
verified ·
1 Parent(s): 1f61cf9

granularity results tiera-rerun-fixed-moonep @ 2026-08-30T08:12:09Z from sp-aiinfra-interactive-aghosh-7jtjl-8xb200-moe-cg-0 (part 3)

Browse files
tiera-rerun-fixed-moonep/steps.tsv CHANGED
@@ -10,3 +10,6 @@ OK ultraep a2 16 2026-08-30T07:22:03+00:00 2026-08-30T07:37:07+00:00 /workspace/
10
  OK moonep a1 16 2026-08-30T07:37:09+00:00 2026-08-30T07:41:05+00:00 /workspace/tiera-rerun-fixed-moonep/moonep/a1
11
  OK moonep ponly 16 2026-08-30T07:41:07+00:00 2026-08-30T07:48:20+00:00 /workspace/tiera-rerun-fixed-moonep/moonep/ponly
12
  OK moonep a2 16 2026-08-30T07:48:22+00:00 2026-08-30T08:03:34+00:00 /workspace/tiera-rerun-fixed-moonep/moonep/a2
 
 
 
 
10
  OK moonep a1 16 2026-08-30T07:37:09+00:00 2026-08-30T07:41:05+00:00 /workspace/tiera-rerun-fixed-moonep/moonep/a1
11
  OK moonep ponly 16 2026-08-30T07:41:07+00:00 2026-08-30T07:48:20+00:00 /workspace/tiera-rerun-fixed-moonep/moonep/ponly
12
  OK moonep a2 16 2026-08-30T07:48:22+00:00 2026-08-30T08:03:34+00:00 /workspace/tiera-rerun-fixed-moonep/moonep/a2
13
+ OK echo tierbmem 16 2026-08-30T08:03:47+00:00 2026-08-30T08:06:37+00:00 /workspace/tiera-rerun-fixed-moonep/echo/tierbmem
14
+ OK ultraep tierbmem 16 2026-08-30T08:06:42+00:00 2026-08-30T08:09:23+00:00 /workspace/tiera-rerun-fixed-moonep/ultraep/tierbmem
15
+ OK moonep tierbmem 16 2026-08-30T08:09:29+00:00 2026-08-30T08:11:55+00:00 /workspace/tiera-rerun-fixed-moonep/moonep/tierbmem
tiera-rerun-fixed-moonep/study.log CHANGED
@@ -1134,3 +1134,319 @@ driver_repo: /workspace/blackwell_run_scripts @ f69d9ab
1134
  === STAGE 8 -- Tier B allocator dumps (static CG capacities + dynamic no-CG)
1135
 
1136
  === STEP echo__tierbmem__gbs16
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1134
  === STAGE 8 -- Tier B allocator dumps (static CG capacities + dynamic no-CG)
1135
 
1136
  === STEP echo__tierbmem__gbs16
1137
+ ==============================================================
1138
+ CAMPAIGN: echo backend=echo phase=tierbmem
1139
+ Megatron: /workspace/Megatron-MoE-Echo @ better-memory-dump-support
1140
+ GBS=16 seq=4096 repeats=3 nsys=0
1141
+ results -> /workspace/tiera-rerun-fixed-moonep/echo/tierbmem
1142
+ ==============================================================
1143
+ --------------------------------------------------------------------------
1144
+ [08:04:02] CELL G1__echo__matched__seq4096__gbs16__cap2p40625__memiteration__it6 (slots=1)
1145
+ [guard] iter 6/6 (70s)
1146
+ [guard] tearing down process group (outcome=DONE)
1147
+ --------------------------------------------------------------------------
1148
+ [08:05:17] CELL G1__echo__matched__seq4096__gbs16__capdynamic__nocg__memiteration__it5 (slots=1)
1149
+ [guard] iter 2/5 (70s)
1150
+ [guard] iter 5/5 (80s)
1151
+ --------------------------------------------------------------------------
1152
+
1153
+ ==============================================================
1154
+ SWEEP COMPLETE ok=2 failed=0
1155
+ ==============================================================
1156
+ config backend policy slots seq gbs cap recompute rep status iter_ms_mean iter_ms_min peak_mib run_dir
1157
+ G1 echo matched 1 4096 16 2.40625 none 1 OK 474.40 474.40 161723 /workspace/tiera-rerun-fixed-moonep/echo/tierbmem/tierbmem_gbs16/G1__echo__matched__seq4096__gbs16__cap2p40625__memiteration__it6/rep1
1158
+ G1 echo matched 1 4096 16 dynamic none 1 OK NA NA 115283 /workspace/tiera-rerun-fixed-moonep/echo/tierbmem/tierbmem_gbs16/G1__echo__matched__seq4096__gbs16__capdynamic__nocg__memiteration__it5/rep1
1159
+
1160
+ Results: /workspace/tiera-rerun-fixed-moonep/echo/tierbmem/tierbmem_gbs16/results.tsv
1161
+ Roll up: python3 /workspace/blackwell_run_scripts/granularity/tools/collect.py /workspace/tiera-rerun-fixed-moonep/echo/tierbmem/tierbmem_gbs16
1162
+ ==============================================================
1163
+ 2 memory-dump runs -> /workspace/tiera-rerun-fixed-moonep/echo/tierbmem/tierbmem_gbs16/memory_summary.tsv
1164
+ model backend cg allocation capacity status peak GiB dumps
1165
+ ----------------------------------------------------------------------------------
1166
+ G1 echo 1 static 2.40625 OK 157.9 48/48
1167
+ G1 echo 0 dynamic dynamic OK 112.6 40/40
1168
+
1169
+ ECHO/UltraEP note: graph-on uses static capacity + device-initiated GEMM while graph-off uses dynamic allocation + host-derived sizes. Their difference is not labelled as pure graph cost.
1170
+
1171
+ TIER B MEMORY DUMPS -- how to read this
1172
+ ---------------------------------------
1173
+ ECHO / UltraEP:
1174
+ cap1p00__memiteration__it6 graph on, static C=1.00
1175
+ cap1p125__memiteration__it6 graph on, static C=1.125
1176
+ cap1p25__memiteration__it6 graph on, static C=1.25
1177
+ capdynamic__nocg__memiteration__it5
1178
+ graph off, no capacity flag, dynamic allocation
1179
+
1180
+ MoonEP:
1181
+ capnone__memiteration__it6 graph on, no capacity factor
1182
+ capnone__nocg__memiteration__it5 graph off, no capacity factor
1183
+
1184
+ Every successful cell contains:
1185
+ train.log Megatron and OOM diagnostics
1186
+ mem_samples.csv driver-level nvidia-smi samples every 2 seconds
1187
+ run.env exact branch/config/capacity provenance
1188
+ memory_snapshots/*.pickle 48 captured / 40 eager allocator histories
1189
+ memory_snapshots/*.json matching summaries and before/after deltas
1190
+
1191
+ The graph-on/static vs graph-off/dynamic difference is intentionally NOT called
1192
+ "graph cost": capacity allocation policy and device-initiated grouped GEMM also
1193
+ change. Compare the three graph-on rows to measure static-capacity sensitivity,
1194
+ and use the dynamic row as the natural eager-allocation reference.
1195
+
1196
+ Campaign results: /workspace/tiera-rerun-fixed-moonep/echo/tierbmem
1197
+ Roll up: python3 /workspace/blackwell_run_scripts/granularity/tools/collect.py /workspace/tiera-rerun-fixed-moonep/echo/tierbmem --compare
1198
+ Already logged in to Hugging Face as: AbhishekGhosh1998
1199
+ uploaded: /workspace/tiera-rerun-fixed-moonep/echo/tierbmem -> MoE-CUDA-Graph-Analysis/tierb-b200-gbs8/tiera-rerun-fixed-moonep/echo/tierbmem
1200
+ https://huggingface.co/MoE-CUDA-Graph-Analysis/tierb-b200-gbs8/tree/main
1201
+
1202
+ === STEP ultraep__tierbmem__gbs16
1203
+ ==============================================================
1204
+ CAMPAIGN: ultraep backend=ultraep phase=tierbmem
1205
+ Megatron: /workspace/Megatron-UltraEP @ ultraep-better-memory-dump-support
1206
+ GBS=16 seq=4096 repeats=3 nsys=0
1207
+ results -> /workspace/tiera-rerun-fixed-moonep/ultraep/tierbmem
1208
+ ==============================================================
1209
+ --------------------------------------------------------------------------
1210
+ [08:06:58] CELL G1__ultraep__matched__seq4096__gbs16__cap2p40625__memiteration__it6 (slots=1)
1211
+ [guard] iter 6/6 (60s)
1212
+ [guard] tearing down process group (outcome=DONE)
1213
+ --------------------------------------------------------------------------
1214
+ [08:08:00] CELL G1__ultraep__matched__seq4096__gbs16__capdynamic__nocg__memiteration__it5 (slots=1)
1215
+ [08:08:02] waiting for stale ranks to exit ...
1216
+ [guard] iter 4/5 (60s)
1217
+ [guard] iter 5/5 (70s)
1218
+ [guard] tearing down process group (outcome=DONE)
1219
+ --------------------------------------------------------------------------
1220
+
1221
+ ==============================================================
1222
+ SWEEP COMPLETE ok=2 failed=0
1223
+ ==============================================================
1224
+ config backend policy slots seq gbs cap recompute rep status iter_ms_mean iter_ms_min peak_mib run_dir
1225
+ G1 ultraep matched 1 4096 16 2.40625 none 1 OK 909.90 909.90 129473 /workspace/tiera-rerun-fixed-moonep/ultraep/tierbmem/tierbmem_gbs16/G1__ultraep__matched__seq4096__gbs16__cap2p40625__memiteration__it6/rep1
1226
+ G1 ultraep matched 1 4096 16 dynamic none 1 OK NA NA 109261 /workspace/tiera-rerun-fixed-moonep/ultraep/tierbmem/tierbmem_gbs16/G1__ultraep__matched__seq4096__gbs16__capdynamic__nocg__memiteration__it5/rep1
1227
+
1228
+ Results: /workspace/tiera-rerun-fixed-moonep/ultraep/tierbmem/tierbmem_gbs16/results.tsv
1229
+ Roll up: python3 /workspace/blackwell_run_scripts/granularity/tools/collect.py /workspace/tiera-rerun-fixed-moonep/ultraep/tierbmem/tierbmem_gbs16
1230
+ ==============================================================
1231
+ 2 memory-dump runs -> /workspace/tiera-rerun-fixed-moonep/ultraep/tierbmem/tierbmem_gbs16/memory_summary.tsv
1232
+ model backend cg allocation capacity status peak GiB dumps
1233
+ ----------------------------------------------------------------------------------
1234
+ G1 ultraep 1 static 2.40625 OK 126.4 48/48
1235
+ G1 ultraep 0 dynamic dynamic OK 106.7 40/40
1236
+
1237
+ ECHO/UltraEP note: graph-on uses static capacity + device-initiated GEMM while graph-off uses dynamic allocation + host-derived sizes. Their difference is not labelled as pure graph cost.
1238
+
1239
+ TIER B MEMORY DUMPS -- how to read this
1240
+ ---------------------------------------
1241
+ ECHO / UltraEP:
1242
+ cap1p00__memiteration__it6 graph on, static C=1.00
1243
+ cap1p125__memiteration__it6 graph on, static C=1.125
1244
+ cap1p25__memiteration__it6 graph on, static C=1.25
1245
+ capdynamic__nocg__memiteration__it5
1246
+ graph off, no capacity flag, dynamic allocation
1247
+
1248
+ MoonEP:
1249
+ capnone__memiteration__it6 graph on, no capacity factor
1250
+ capnone__nocg__memiteration__it5 graph off, no capacity factor
1251
+
1252
+ Every successful cell contains:
1253
+ train.log Megatron and OOM diagnostics
1254
+ mem_samples.csv driver-level nvidia-smi samples every 2 seconds
1255
+ run.env exact branch/config/capacity provenance
1256
+ memory_snapshots/*.pickle 48 captured / 40 eager allocator histories
1257
+ memory_snapshots/*.json matching summaries and before/after deltas
1258
+
1259
+ The graph-on/static vs graph-off/dynamic difference is intentionally NOT called
1260
+ "graph cost": capacity allocation policy and device-initiated grouped GEMM also
1261
+ change. Compare the three graph-on rows to measure static-capacity sensitivity,
1262
+ and use the dynamic row as the natural eager-allocation reference.
1263
+
1264
+ Campaign results: /workspace/tiera-rerun-fixed-moonep/ultraep/tierbmem
1265
+ Roll up: python3 /workspace/blackwell_run_scripts/granularity/tools/collect.py /workspace/tiera-rerun-fixed-moonep/ultraep/tierbmem --compare
1266
+ Already logged in to Hugging Face as: AbhishekGhosh1998
1267
+ uploaded: /workspace/tiera-rerun-fixed-moonep/ultraep/tierbmem -> MoE-CUDA-Graph-Analysis/tierb-b200-gbs8/tiera-rerun-fixed-moonep/ultraep/tierbmem
1268
+ https://huggingface.co/MoE-CUDA-Graph-Analysis/tierb-b200-gbs8/tree/main
1269
+
1270
+ === STEP moonep__tierbmem__gbs16
1271
+ ==============================================================
1272
+ CAMPAIGN: moonep backend=moonep phase=tierbmem
1273
+ Megatron: /workspace/Megatron-MoE-Echo @ moonep_backend_async_grad_reduce_better_memory_dump_support
1274
+ GBS=16 seq=4096 repeats=3 nsys=0
1275
+ results -> /workspace/tiera-rerun-fixed-moonep/moonep/tierbmem
1276
+ ==============================================================
1277
+ --------------------------------------------------------------------------
1278
+ [08:09:44] CELL G1__moonep__matched__seq4096__gbs16__capnone__async__memiteration__it6 (slots=1)
1279
+ [guard] iter 6/6 (60s)
1280
+ [guard] tearing down process group (outcome=DONE)
1281
+ --------------------------------------------------------------------------
1282
+ [08:10:49] CELL G1__moonep__matched__seq4096__gbs16__capnone__nocg__async__memiteration__it5 (slots=1)
1283
+ [guard] iter 5/5 (60s)
1284
+ [guard] tearing down process group (outcome=DONE)
1285
+ --------------------------------------------------------------------------
1286
+
1287
+ ==============================================================
1288
+ SWEEP COMPLETE ok=2 failed=0
1289
+ ==============================================================
1290
+ config backend policy slots seq gbs cap recompute rep status iter_ms_mean iter_ms_min peak_mib run_dir
1291
+ G1 moonep matched 1 4096 16 none none 1 OK 497.40 497.40 96849 /workspace/tiera-rerun-fixed-moonep/moonep/tierbmem/tierbmem_gbs16/G1__moonep__matched__seq4096__gbs16__capnone__async__memiteration__it6/rep1
1292
+ G1 moonep matched 1 4096 16 none none 1 OK NA NA 92579 /workspace/tiera-rerun-fixed-moonep/moonep/tierbmem/tierbmem_gbs16/G1__moonep__matched__seq4096__gbs16__capnone__nocg__async__memiteration__it5/rep1
1293
+
1294
+ Results: /workspace/tiera-rerun-fixed-moonep/moonep/tierbmem/tierbmem_gbs16/results.tsv
1295
+ Roll up: python3 /workspace/blackwell_run_scripts/granularity/tools/collect.py /workspace/tiera-rerun-fixed-moonep/moonep/tierbmem/tierbmem_gbs16
1296
+ ==============================================================
1297
+ 2 memory-dump runs -> /workspace/tiera-rerun-fixed-moonep/moonep/tierbmem/tierbmem_gbs16/memory_summary.tsv
1298
+ model backend cg allocation capacity status peak GiB dumps
1299
+ ----------------------------------------------------------------------------------
1300
+ G1 moonep 1 none none OK 94.6 48/48
1301
+ G1 moonep 0 none none OK 90.4 40/40
1302
+
1303
+ ECHO/UltraEP note: graph-on uses static capacity + device-initiated GEMM while graph-off uses dynamic allocation + host-derived sizes. Their difference is not labelled as pure graph cost.
1304
+
1305
+ TIER B MEMORY DUMPS -- how to read this
1306
+ ---------------------------------------
1307
+ ECHO / UltraEP:
1308
+ cap1p00__memiteration__it6 graph on, static C=1.00
1309
+ cap1p125__memiteration__it6 graph on, static C=1.125
1310
+ cap1p25__memiteration__it6 graph on, static C=1.25
1311
+ capdynamic__nocg__memiteration__it5
1312
+ graph off, no capacity flag, dynamic allocation
1313
+
1314
+ MoonEP:
1315
+ capnone__memiteration__it6 graph on, no capacity factor
1316
+ capnone__nocg__memiteration__it5 graph off, no capacity factor
1317
+
1318
+ Every successful cell contains:
1319
+ train.log Megatron and OOM diagnostics
1320
+ mem_samples.csv driver-level nvidia-smi samples every 2 seconds
1321
+ run.env exact branch/config/capacity provenance
1322
+ memory_snapshots/*.pickle 48 captured / 40 eager allocator histories
1323
+ memory_snapshots/*.json matching summaries and before/after deltas
1324
+
1325
+ The graph-on/static vs graph-off/dynamic difference is intentionally NOT called
1326
+ "graph cost": capacity allocation policy and device-initiated grouped GEMM also
1327
+ change. Compare the three graph-on rows to measure static-capacity sensitivity,
1328
+ and use the dynamic row as the natural eager-allocation reference.
1329
+
1330
+ Campaign results: /workspace/tiera-rerun-fixed-moonep/moonep/tierbmem
1331
+ Roll up: python3 /workspace/blackwell_run_scripts/granularity/tools/collect.py /workspace/tiera-rerun-fixed-moonep/moonep/tierbmem --compare
1332
+ Already logged in to Hugging Face as: AbhishekGhosh1998
1333
+ uploaded: /workspace/tiera-rerun-fixed-moonep/moonep/tierbmem -> MoE-CUDA-Graph-Analysis/tierb-b200-gbs8/tiera-rerun-fixed-moonep/moonep/tierbmem
1334
+ https://huggingface.co/MoE-CUDA-Graph-Analysis/tierb-b200-gbs8/tree/main
1335
+
1336
+ === ROLL-UP
1337
+ 110 runs ok=107 oom=0 -> /workspace/tiera-rerun-fixed-moonep/collected_all.tsv
1338
+
1339
+ WARNING: 11 run(s) mention capacity saturation. A run that drops assignments does less expert work and must not be quoted as a win:
1340
+ ultraep/tierbprobe/tierbprobe_gbs16/G1__ultraep__matched__seq4096__gbs16__cap1p25/rep1: WARNING:megatron.training.drop_debug_utils:[MOE_DROP_DEBUG][iteration_summary] global_rank=0 iteration=0 records=32 total_expected_recv=283999 total_dropped_tok
1341
+ ultraep/tierbprobe/tierbprobe_gbs16/G1__ultraep__matched__seq4096__gbs16__cap1p8125/rep1: WARNING:megatron.training.drop_debug_utils:[MOE_DROP_DEBUG][iteration_summary] global_rank=0 iteration=0 records=32 total_expected_recv=284540 total_dropped_tok
1342
+ ultraep/tierbprobe/tierbprobe_gbs16/G2__ultraep__matched__seq4096__gbs16__cap1p0625/rep1: WARNING:megatron.training.drop_debug_utils:[MOE_DROP_DEBUG][iteration_summary] global_rank=0 iteration=0 records=32 total_expected_recv=1048800 total_dropped_to
1343
+ ultraep/tierbprobe/tierbprobe_gbs16/G2__ultraep__matched__seq4096__gbs16__cap1p25/rep1: WARNING:megatron.training.drop_debug_utils:[MOE_DROP_DEBUG][iteration_summary] global_rank=0 iteration=0 records=32 total_expected_recv=1048800 total_dropped_to
1344
+ ultraep/tierbprobe/tierbprobe_gbs16/G2__ultraep__matched__seq4096__gbs16__cap1p25__forcebal/rep1: WARNING:megatron.training.drop_debug_utils:[MOE_DROP_DEBUG][iteration_summary] global_rank=1 iteration=0 records=32 total_expected_recv=1047984 total_dropped_to
1345
+
1346
+ (excluded 2 nsys-traced run(s) from the timing comparison)
1347
+
1348
+ phase config seq gbs policy rc cg asy backend iter_ms vs_best
1349
+ -------------------------------------------------------------------------------------
1350
+ a1 G1 4096 16 one none 1 0 echo 466.68 1.00x
1351
+ ultraep 891.00 1.91x
1352
+
1353
+ a1 G1 4096 16 one none 1 1 moonep 480.30 1.00x
1354
+
1355
+ a1 G2 4096 16 one none 1 0 echo 394.28 1.00x
1356
+ ultraep 466.65 1.18x
1357
+
1358
+ a1 G3 4096 16 one none 1 0 echo 412.83 1.00x
1359
+ ultraep 447.84 1.08x
1360
+
1361
+ a1 G4 4096 16 one none 1 0 echo 461.03 1.00x
1362
+ ultraep 467.57 1.01x
1363
+
1364
+ a2 G1 4096 16 matched none 1 0 echo 466.91 1.00x
1365
+ ultraep 891.92 1.91x
1366
+
1367
+ a2 G1 4096 16 matched none 1 1 moonep 481.37 1.00x
1368
+
1369
+ a2 G2 4096 16 matched none 1 0 echo 415.21 1.00x
1370
+ ultraep 492.54 1.19x
1371
+
1372
+ a2 G2 4096 16 matched none 1 1 moonep 432.02 1.00x
1373
+
1374
+ a2 G3 4096 16 matched none 1 0 echo 437.35 1.00x
1375
+ ultraep 459.60 1.05x
1376
+
1377
+ a2 G3 4096 16 matched none 1 1 moonep 559.88 1.00x
1378
+
1379
+ a2 G4 4096 16 matched none 1 0 ultraep 482.64 1.00x
1380
+ echo 486.54 1.01x
1381
+
1382
+ a2 G4 4096 16 matched none 1 1 moonep 783.34 1.00x
1383
+
1384
+ ponly PONLY 4096 16 one none 1 0 echo 174.81 1.00x
1385
+ ultraep 176.76 1.01x
1386
+
1387
+ ponly PONLY 4096 16 one none 1 1 moonep 165.22 1.00x
1388
+
1389
+ ponly PONLY 8192 16 one none 1 0 ultraep 292.62 1.00x
1390
+ echo 316.70 1.08x
1391
+
1392
+ ponly PONLY 8192 16 one none 1 1 moonep 300.61 1.00x
1393
+
1394
+ tierbmem G1 4096 16 matched none 1 0 echo 474.40 1.00x
1395
+ ultraep 909.90 1.92x
1396
+
1397
+ tierbmem G1 4096 16 matched none 1 1 moonep 497.40 1.00x
1398
+
1399
+ tierbprobeG1 4096 16 matched none 0 0 echo 671.53 1.00x
1400
+ ultraep 904.16 1.35x
1401
+
1402
+ tierbprobeG2 4096 16 matched none 0 0 ultraep 512.37 1.00x
1403
+ echo 654.76 1.28x
1404
+
1405
+ tierbprobeG3 4096 16 matched none 0 0 ultraep 512.31 1.00x
1406
+ echo 689.13 1.35x
1407
+
1408
+ tierbprobeG4 4096 16 matched none 0 0 ultraep 600.19 1.00x
1409
+ echo 790.97 1.32x
1410
+
1411
+ 21 probe runs -> /workspace/tiera-rerun-fixed-moonep/probe_summary_all.tsv
1412
+ run arm cap rawExpI preRankI postRankI drop% reqCap
1413
+ --------------------------------------------------------------------------------------------------------------------
1414
+ echo/tierbprobe/tierbprobe_gbs16/G1__echo__matched__ natural 2.00 3.6198* 3.6198 1.0827 1.9062
1415
+ echo/tierbprobe/tierbprobe_gbs16/G2__echo__matched__ natural 1.08203125 3.8720* 2.1703 1.0000 1.0273
1416
+ echo/tierbprobe/tierbprobe_gbs16/G2__echo__matched__ forced 1.08203125 1.0201 1.0069 1.0000 1.0234
1417
+ echo/tierbprobe/tierbprobe_gbs16/G2__echo__matched__ natural 2.00 3.8724* 2.1875 1.0000 1.0273
1418
+ echo/tierbprobe/tierbprobe_gbs16/G3__echo__matched__ natural 1.072265625 3.9096* 1.7838 1.0000 1.0195
1419
+ echo/tierbprobe/tierbprobe_gbs16/G3__echo__matched__ forced 1.072265625 1.0231 1.0048 1.0000 1.0176
1420
+ echo/tierbprobe/tierbprobe_gbs16/G3__echo__matched__ natural 2.00 3.9096* 1.7916 1.0000 1.0195
1421
+ echo/tierbprobe/tierbprobe_gbs16/G4__echo__matched__ natural 1.068359375 3.9303* 1.5507 1.0000 1.0166
1422
+ echo/tierbprobe/tierbprobe_gbs16/G4__echo__matched__ forced 1.068359375 1.0249 1.0034 1.0000 1.0137
1423
+ echo/tierbprobe/tierbprobe_gbs16/G4__echo__matched__ natural 2.00 3.9301* 1.5543 1.0000 1.0166
1424
+ ultraep/tierbprobe/tierbprobe_gbs16/G1__ultraep__mat natural 1.25 3.6085 1.0640 2.8918 1.7246
1425
+ ultraep/tierbprobe/tierbprobe_gbs16/G1__ultraep__mat natural 1.8125 3.6152 1.0747 1.5460 2.3091
1426
+ ultraep/tierbprobe/tierbprobe_gbs16/G2__ultraep__mat natural 1.0625 2.2032 1.0100 0.0000 1.0101
1427
+ ultraep/tierbprobe/tierbprobe_gbs16/G2__ultraep__mat natural 1.25 2.2136 1.0100 0.0000 1.0101
1428
+ ultraep/tierbprobe/tierbprobe_gbs16/G2__ultraep__mat forced 1.25 1.0069 1.0067 0.0000 1.0100
1429
+ ultraep/tierbprobe/tierbprobe_gbs16/G3__ultraep__mat natural 1.0625 1.7842 1.0100 0.0000 1.0101
1430
+ ultraep/tierbprobe/tierbprobe_gbs16/G3__ultraep__mat natural 1.25 1.7752 1.0100 0.0000 1.0101
1431
+ ultraep/tierbprobe/tierbprobe_gbs16/G3__ultraep__mat forced 1.25 1.0047 1.0047 0.0000 1.0100
1432
+ ultraep/tierbprobe/tierbprobe_gbs16/G4__ultraep__mat natural 1.095703125 1.5612 1.0100 0.0000 1.0429
1433
+ ultraep/tierbprobe/tierbprobe_gbs16/G4__ultraep__mat natural 1.25 1.5504 1.0100 0.0000 1.0429
1434
+ ultraep/tierbprobe/tierbprobe_gbs16/G4__ultraep__mat forced 1.25 1.0033 1.0033 0.0000 1.0071
1435
+
1436
+ * At least one globally aggregated snapshot reached the E/K expert-Imax ceiling.
1437
+ Inspect raw_expert_imax_mean and hottest_expert_token_fraction_mean before generalising.
1438
+
1439
+ ECHO note: echo_logged_pre_rank_metric_valid=0 and echo_logged_load_reduction_valid=0 are intentional.
1440
+ The branch's built-in fields mix source-rank and destination-rank axes; this tool reconstructs both globally.
1441
+ 6 memory-dump runs -> /workspace/tiera-rerun-fixed-moonep/memory_summary_all.tsv
1442
+ model backend cg allocation capacity status peak GiB dumps
1443
+ ----------------------------------------------------------------------------------
1444
+ G1 echo 1 static 2.40625 OK 157.9 48/48
1445
+ G1 echo 0 dynamic dynamic OK 112.6 40/40
1446
+ G1 moonep 1 none none OK 94.6 48/48
1447
+ G1 moonep 0 none none OK 90.4 40/40
1448
+ G1 ultraep 1 static 2.40625 OK 126.4 48/48
1449
+ G1 ultraep 0 dynamic dynamic OK 106.7 40/40
1450
+
1451
+ ECHO/UltraEP note: graph-on uses static capacity + device-initiated GEMM while graph-off uses dynamic allocation + host-derived sizes. Their difference is not labelled as pure graph cost.
1452
+ Already logged in to Hugging Face as: AbhishekGhosh1998