granularity results tiera-rerun-fixed-moonep @ 2026-08-30T08:12:16Z from sp-aiinfra-interactive-aghosh-7jtjl-8xb200-moe-cg-0 (part 3)
Browse files
tiera-rerun-fixed-moonep/study.log
CHANGED
|
@@ -1450,3 +1450,113 @@ G1 ultraep 0 dynamic dynamic OK 106.7 40/
|
|
| 1450 |
|
| 1451 |
ECHO/UltraEP note: graph-on uses static capacity + device-initiated GEMM while graph-off uses dynamic allocation + host-derived sizes. Their difference is not labelled as pure graph cost.
|
| 1452 |
Already logged in to Hugging Face as: AbhishekGhosh1998
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1450 |
|
| 1451 |
ECHO/UltraEP note: graph-on uses static capacity + device-initiated GEMM while graph-off uses dynamic allocation + host-derived sizes. Their difference is not labelled as pure graph cost.
|
| 1452 |
Already logged in to Hugging Face as: AbhishekGhosh1998
|
| 1453 |
+
uploaded: /workspace/tiera-rerun-fixed-moonep -> MoE-CUDA-Graph-Analysis/tierb-b200-gbs8/tiera-rerun-fixed-moonep
|
| 1454 |
+
https://huggingface.co/MoE-CUDA-Graph-Analysis/tierb-b200-gbs8/tree/main
|
| 1455 |
+
|
| 1456 |
+
=== STUDY COMPLETE
|
| 1457 |
+
results : /workspace/tiera-rerun-fixed-moonep
|
| 1458 |
+
status : /workspace/tiera-rerun-fixed-moonep/STATUS.txt
|
| 1459 |
+
steps : /workspace/tiera-rerun-fixed-moonep/steps.tsv
|
| 1460 |
+
hub : https://huggingface.co/datasets/MoE-CUDA-Graph-Analysis/tierb-b200-gbs8
|
| 1461 |
+
|
| 1462 |
+
status campaign phase gbs started ended result_dir
|
| 1463 |
+
FAIL echo tierbprobe 16 2026-08-30T04:52:38+00:00 2026-08-30T05:06:38+00:00 /workspace/tiera-rerun-fixed-moonep/echo/tierbprobe
|
| 1464 |
+
FAIL ultraep tierbprobe 16 2026-08-30T05:06:44+00:00 2026-08-30T05:20:40+00:00 /workspace/tiera-rerun-fixed-moonep/ultraep/tierbprobe
|
| 1465 |
+
OK echo a1 16 2026-08-30T06:20:31+00:00 2026-08-30T06:36:28+00:00 /workspace/tiera-rerun-fixed-moonep/echo/a1
|
| 1466 |
+
OK echo ponly 16 2026-08-30T06:36:31+00:00 2026-08-30T06:44:11+00:00 /workspace/tiera-rerun-fixed-moonep/echo/ponly
|
| 1467 |
+
OK echo a2 16 2026-08-30T06:44:13+00:00 2026-08-30T07:00:25+00:00 /workspace/tiera-rerun-fixed-moonep/echo/a2
|
| 1468 |
+
OK ultraep a1 16 2026-08-30T07:00:27+00:00 2026-08-30T07:15:17+00:00 /workspace/tiera-rerun-fixed-moonep/ultraep/a1
|
| 1469 |
+
OK ultraep ponly 16 2026-08-30T07:15:19+00:00 2026-08-30T07:22:01+00:00 /workspace/tiera-rerun-fixed-moonep/ultraep/ponly
|
| 1470 |
+
OK ultraep a2 16 2026-08-30T07:22:03+00:00 2026-08-30T07:37:07+00:00 /workspace/tiera-rerun-fixed-moonep/ultraep/a2
|
| 1471 |
+
OK moonep a1 16 2026-08-30T07:37:09+00:00 2026-08-30T07:41:05+00:00 /workspace/tiera-rerun-fixed-moonep/moonep/a1
|
| 1472 |
+
OK moonep ponly 16 2026-08-30T07:41:07+00:00 2026-08-30T07:48:20+00:00 /workspace/tiera-rerun-fixed-moonep/moonep/ponly
|
| 1473 |
+
OK moonep a2 16 2026-08-30T07:48:22+00:00 2026-08-30T08:03:34+00:00 /workspace/tiera-rerun-fixed-moonep/moonep/a2
|
| 1474 |
+
OK echo tierbmem 16 2026-08-30T08:03:47+00:00 2026-08-30T08:06:37+00:00 /workspace/tiera-rerun-fixed-moonep/echo/tierbmem
|
| 1475 |
+
OK ultraep tierbmem 16 2026-08-30T08:06:42+00:00 2026-08-30T08:09:23+00:00 /workspace/tiera-rerun-fixed-moonep/ultraep/tierbmem
|
| 1476 |
+
OK moonep tierbmem 16 2026-08-30T08:09:29+00:00 2026-08-30T08:11:55+00:00 /workspace/tiera-rerun-fixed-moonep/moonep/tierbmem
|
| 1477 |
+
study_root: /workspace/tiera-rerun-fixed-moonep
|
| 1478 |
+
gbs: 16
|
| 1479 |
+
stages: 8
|
| 1480 |
+
nsys: 0
|
| 1481 |
+
hf_repo: MoE-CUDA-Graph-Analysis/tierb-b200-gbs8
|
| 1482 |
+
hf_remote_root: tiera-rerun-fixed-moonep
|
| 1483 |
+
started: 2026-08-30T08:12:15+00:00
|
| 1484 |
+
driver_repo: /workspace/blackwell_run_scripts @ f69d9ab
|
| 1485 |
+
|
| 1486 |
+
=== STAGE 8 -- Tier B allocator dumps (static CG capacities + dynamic no-CG)
|
| 1487 |
+
|
| 1488 |
+
=== ROLL-UP
|
| 1489 |
+
110 runs ok=107 oom=0 -> /workspace/tiera-rerun-fixed-moonep/collected_all.tsv
|
| 1490 |
+
|
| 1491 |
+
WARNING: 11 run(s) mention capacity saturation. A run that drops assignments does less expert work and must not be quoted as a win:
|
| 1492 |
+
ultraep/tierbprobe/tierbprobe_gbs16/G1__ultraep__matched__seq4096__gbs16__cap1p25/rep1: WARNING:megatron.training.drop_debug_utils:[MOE_DROP_DEBUG][iteration_summary] global_rank=0 iteration=0 records=32 total_expected_recv=283999 total_dropped_tok
|
| 1493 |
+
ultraep/tierbprobe/tierbprobe_gbs16/G1__ultraep__matched__seq4096__gbs16__cap1p8125/rep1: WARNING:megatron.training.drop_debug_utils:[MOE_DROP_DEBUG][iteration_summary] global_rank=0 iteration=0 records=32 total_expected_recv=284540 total_dropped_tok
|
| 1494 |
+
ultraep/tierbprobe/tierbprobe_gbs16/G2__ultraep__matched__seq4096__gbs16__cap1p0625/rep1: WARNING:megatron.training.drop_debug_utils:[MOE_DROP_DEBUG][iteration_summary] global_rank=0 iteration=0 records=32 total_expected_recv=1048800 total_dropped_to
|
| 1495 |
+
ultraep/tierbprobe/tierbprobe_gbs16/G2__ultraep__matched__seq4096__gbs16__cap1p25/rep1: WARNING:megatron.training.drop_debug_utils:[MOE_DROP_DEBUG][iteration_summary] global_rank=0 iteration=0 records=32 total_expected_recv=1048800 total_dropped_to
|
| 1496 |
+
ultraep/tierbprobe/tierbprobe_gbs16/G2__ultraep__matched__seq4096__gbs16__cap1p25__forcebal/rep1: WARNING:megatron.training.drop_debug_utils:[MOE_DROP_DEBUG][iteration_summary] global_rank=1 iteration=0 records=32 total_expected_recv=1047984 total_dropped_to
|
| 1497 |
+
|
| 1498 |
+
(excluded 2 nsys-traced run(s) from the timing comparison)
|
| 1499 |
+
|
| 1500 |
+
phase config seq gbs policy rc cg asy backend iter_ms vs_best
|
| 1501 |
+
-------------------------------------------------------------------------------------
|
| 1502 |
+
a1 G1 4096 16 one none 1 0 echo 466.68 1.00x
|
| 1503 |
+
ultraep 891.00 1.91x
|
| 1504 |
+
|
| 1505 |
+
a1 G1 4096 16 one none 1 1 moonep 480.30 1.00x
|
| 1506 |
+
|
| 1507 |
+
a1 G2 4096 16 one none 1 0 echo 394.28 1.00x
|
| 1508 |
+
ultraep 466.65 1.18x
|
| 1509 |
+
|
| 1510 |
+
a1 G3 4096 16 one none 1 0 echo 412.83 1.00x
|
| 1511 |
+
ultraep 447.84 1.08x
|
| 1512 |
+
|
| 1513 |
+
a1 G4 4096 16 one none 1 0 echo 461.03 1.00x
|
| 1514 |
+
ultraep 467.57 1.01x
|
| 1515 |
+
|
| 1516 |
+
a2 G1 4096 16 matched none 1 0 echo 466.91 1.00x
|
| 1517 |
+
ultraep 891.92 1.91x
|
| 1518 |
+
|
| 1519 |
+
a2 G1 4096 16 matched none 1 1 moonep 481.37 1.00x
|
| 1520 |
+
|
| 1521 |
+
a2 G2 4096 16 matched none 1 0 echo 415.21 1.00x
|
| 1522 |
+
ultraep 492.54 1.19x
|
| 1523 |
+
|
| 1524 |
+
a2 G2 4096 16 matched none 1 1 moonep 432.02 1.00x
|
| 1525 |
+
|
| 1526 |
+
a2 G3 4096 16 matched none 1 0 echo 437.35 1.00x
|
| 1527 |
+
ultraep 459.60 1.05x
|
| 1528 |
+
|
| 1529 |
+
a2 G3 4096 16 matched none 1 1 moonep 559.88 1.00x
|
| 1530 |
+
|
| 1531 |
+
a2 G4 4096 16 matched none 1 0 ultraep 482.64 1.00x
|
| 1532 |
+
echo 486.54 1.01x
|
| 1533 |
+
|
| 1534 |
+
a2 G4 4096 16 matched none 1 1 moonep 783.34 1.00x
|
| 1535 |
+
|
| 1536 |
+
ponly PONLY 4096 16 one none 1 0 echo 174.81 1.00x
|
| 1537 |
+
ultraep 176.76 1.01x
|
| 1538 |
+
|
| 1539 |
+
ponly PONLY 4096 16 one none 1 1 moonep 165.22 1.00x
|
| 1540 |
+
|
| 1541 |
+
ponly PONLY 8192 16 one none 1 0 ultraep 292.62 1.00x
|
| 1542 |
+
echo 316.70 1.08x
|
| 1543 |
+
|
| 1544 |
+
ponly PONLY 8192 16 one none 1 1 moonep 300.61 1.00x
|
| 1545 |
+
|
| 1546 |
+
tierbmem G1 4096 16 matched none 1 0 echo 474.40 1.00x
|
| 1547 |
+
ultraep 909.90 1.92x
|
| 1548 |
+
|
| 1549 |
+
tierbmem G1 4096 16 matched none 1 1 moonep 497.40 1.00x
|
| 1550 |
+
|
| 1551 |
+
tierbprobeG1 4096 16 matched none 0 0 echo 671.53 1.00x
|
| 1552 |
+
ultraep 904.16 1.35x
|
| 1553 |
+
|
| 1554 |
+
tierbprobeG2 4096 16 matched none 0 0 ultraep 512.37 1.00x
|
| 1555 |
+
echo 654.76 1.28x
|
| 1556 |
+
|
| 1557 |
+
tierbprobeG3 4096 16 matched none 0 0 ultraep 512.31 1.00x
|
| 1558 |
+
echo 689.13 1.35x
|
| 1559 |
+
|
| 1560 |
+
tierbprobeG4 4096 16 matched none 0 0 ultraep 600.19 1.00x
|
| 1561 |
+
echo 790.97 1.32x
|
| 1562 |
+
|