AbhishekGhosh1998 commited on
Commit
dbf207f
·
verified ·
1 Parent(s): cc93347

granularity results tiera-rerun-fixed-moonep @ 2026-08-30T08:12:29Z from sp-aiinfra-interactive-aghosh-7jtjl-8xb200-moe-cg-0 (part 3)

Browse files
Files changed (1) hide show
  1. tiera-rerun-fixed-moonep/study.log +110 -0
tiera-rerun-fixed-moonep/study.log CHANGED
@@ -1602,3 +1602,113 @@ G1 ultraep 0 dynamic dynamic OK 106.7 40/
1602
 
1603
  ECHO/UltraEP note: graph-on uses static capacity + device-initiated GEMM while graph-off uses dynamic allocation + host-derived sizes. Their difference is not labelled as pure graph cost.
1604
  Already logged in to Hugging Face as: AbhishekGhosh1998
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1602
 
1603
  ECHO/UltraEP note: graph-on uses static capacity + device-initiated GEMM while graph-off uses dynamic allocation + host-derived sizes. Their difference is not labelled as pure graph cost.
1604
  Already logged in to Hugging Face as: AbhishekGhosh1998
1605
+ uploaded: /workspace/tiera-rerun-fixed-moonep -> MoE-CUDA-Graph-Analysis/tierb-b200-gbs8/tiera-rerun-fixed-moonep
1606
+ https://huggingface.co/MoE-CUDA-Graph-Analysis/tierb-b200-gbs8/tree/main
1607
+
1608
+ === STUDY COMPLETE
1609
+ results : /workspace/tiera-rerun-fixed-moonep
1610
+ status : /workspace/tiera-rerun-fixed-moonep/STATUS.txt
1611
+ steps : /workspace/tiera-rerun-fixed-moonep/steps.tsv
1612
+ hub : https://huggingface.co/datasets/MoE-CUDA-Graph-Analysis/tierb-b200-gbs8
1613
+
1614
+ status campaign phase gbs started ended result_dir
1615
+ FAIL echo tierbprobe 16 2026-08-30T04:52:38+00:00 2026-08-30T05:06:38+00:00 /workspace/tiera-rerun-fixed-moonep/echo/tierbprobe
1616
+ FAIL ultraep tierbprobe 16 2026-08-30T05:06:44+00:00 2026-08-30T05:20:40+00:00 /workspace/tiera-rerun-fixed-moonep/ultraep/tierbprobe
1617
+ OK echo a1 16 2026-08-30T06:20:31+00:00 2026-08-30T06:36:28+00:00 /workspace/tiera-rerun-fixed-moonep/echo/a1
1618
+ OK echo ponly 16 2026-08-30T06:36:31+00:00 2026-08-30T06:44:11+00:00 /workspace/tiera-rerun-fixed-moonep/echo/ponly
1619
+ OK echo a2 16 2026-08-30T06:44:13+00:00 2026-08-30T07:00:25+00:00 /workspace/tiera-rerun-fixed-moonep/echo/a2
1620
+ OK ultraep a1 16 2026-08-30T07:00:27+00:00 2026-08-30T07:15:17+00:00 /workspace/tiera-rerun-fixed-moonep/ultraep/a1
1621
+ OK ultraep ponly 16 2026-08-30T07:15:19+00:00 2026-08-30T07:22:01+00:00 /workspace/tiera-rerun-fixed-moonep/ultraep/ponly
1622
+ OK ultraep a2 16 2026-08-30T07:22:03+00:00 2026-08-30T07:37:07+00:00 /workspace/tiera-rerun-fixed-moonep/ultraep/a2
1623
+ OK moonep a1 16 2026-08-30T07:37:09+00:00 2026-08-30T07:41:05+00:00 /workspace/tiera-rerun-fixed-moonep/moonep/a1
1624
+ OK moonep ponly 16 2026-08-30T07:41:07+00:00 2026-08-30T07:48:20+00:00 /workspace/tiera-rerun-fixed-moonep/moonep/ponly
1625
+ OK moonep a2 16 2026-08-30T07:48:22+00:00 2026-08-30T08:03:34+00:00 /workspace/tiera-rerun-fixed-moonep/moonep/a2
1626
+ OK echo tierbmem 16 2026-08-30T08:03:47+00:00 2026-08-30T08:06:37+00:00 /workspace/tiera-rerun-fixed-moonep/echo/tierbmem
1627
+ OK ultraep tierbmem 16 2026-08-30T08:06:42+00:00 2026-08-30T08:09:23+00:00 /workspace/tiera-rerun-fixed-moonep/ultraep/tierbmem
1628
+ OK moonep tierbmem 16 2026-08-30T08:09:29+00:00 2026-08-30T08:11:55+00:00 /workspace/tiera-rerun-fixed-moonep/moonep/tierbmem
1629
+ study_root: /workspace/tiera-rerun-fixed-moonep
1630
+ gbs: 16
1631
+ stages: 8
1632
+ nsys: 0
1633
+ hf_repo: MoE-CUDA-Graph-Analysis/tierb-b200-gbs8
1634
+ hf_remote_root: tiera-rerun-fixed-moonep
1635
+ started: 2026-08-30T08:12:28+00:00
1636
+ driver_repo: /workspace/blackwell_run_scripts @ f69d9ab
1637
+
1638
+ === STAGE 8 -- Tier B allocator dumps (static CG capacities + dynamic no-CG)
1639
+
1640
+ === ROLL-UP
1641
+ 110 runs ok=107 oom=0 -> /workspace/tiera-rerun-fixed-moonep/collected_all.tsv
1642
+
1643
+ WARNING: 11 run(s) mention capacity saturation. A run that drops assignments does less expert work and must not be quoted as a win:
1644
+ ultraep/tierbprobe/tierbprobe_gbs16/G1__ultraep__matched__seq4096__gbs16__cap1p25/rep1: WARNING:megatron.training.drop_debug_utils:[MOE_DROP_DEBUG][iteration_summary] global_rank=0 iteration=0 records=32 total_expected_recv=283999 total_dropped_tok
1645
+ ultraep/tierbprobe/tierbprobe_gbs16/G1__ultraep__matched__seq4096__gbs16__cap1p8125/rep1: WARNING:megatron.training.drop_debug_utils:[MOE_DROP_DEBUG][iteration_summary] global_rank=0 iteration=0 records=32 total_expected_recv=284540 total_dropped_tok
1646
+ ultraep/tierbprobe/tierbprobe_gbs16/G2__ultraep__matched__seq4096__gbs16__cap1p0625/rep1: WARNING:megatron.training.drop_debug_utils:[MOE_DROP_DEBUG][iteration_summary] global_rank=0 iteration=0 records=32 total_expected_recv=1048800 total_dropped_to
1647
+ ultraep/tierbprobe/tierbprobe_gbs16/G2__ultraep__matched__seq4096__gbs16__cap1p25/rep1: WARNING:megatron.training.drop_debug_utils:[MOE_DROP_DEBUG][iteration_summary] global_rank=0 iteration=0 records=32 total_expected_recv=1048800 total_dropped_to
1648
+ ultraep/tierbprobe/tierbprobe_gbs16/G2__ultraep__matched__seq4096__gbs16__cap1p25__forcebal/rep1: WARNING:megatron.training.drop_debug_utils:[MOE_DROP_DEBUG][iteration_summary] global_rank=1 iteration=0 records=32 total_expected_recv=1047984 total_dropped_to
1649
+
1650
+ (excluded 2 nsys-traced run(s) from the timing comparison)
1651
+
1652
+ phase config seq gbs policy rc cg asy backend iter_ms vs_best
1653
+ -------------------------------------------------------------------------------------
1654
+ a1 G1 4096 16 one none 1 0 echo 466.68 1.00x
1655
+ ultraep 891.00 1.91x
1656
+
1657
+ a1 G1 4096 16 one none 1 1 moonep 480.30 1.00x
1658
+
1659
+ a1 G2 4096 16 one none 1 0 echo 394.28 1.00x
1660
+ ultraep 466.65 1.18x
1661
+
1662
+ a1 G3 4096 16 one none 1 0 echo 412.83 1.00x
1663
+ ultraep 447.84 1.08x
1664
+
1665
+ a1 G4 4096 16 one none 1 0 echo 461.03 1.00x
1666
+ ultraep 467.57 1.01x
1667
+
1668
+ a2 G1 4096 16 matched none 1 0 echo 466.91 1.00x
1669
+ ultraep 891.92 1.91x
1670
+
1671
+ a2 G1 4096 16 matched none 1 1 moonep 481.37 1.00x
1672
+
1673
+ a2 G2 4096 16 matched none 1 0 echo 415.21 1.00x
1674
+ ultraep 492.54 1.19x
1675
+
1676
+ a2 G2 4096 16 matched none 1 1 moonep 432.02 1.00x
1677
+
1678
+ a2 G3 4096 16 matched none 1 0 echo 437.35 1.00x
1679
+ ultraep 459.60 1.05x
1680
+
1681
+ a2 G3 4096 16 matched none 1 1 moonep 559.88 1.00x
1682
+
1683
+ a2 G4 4096 16 matched none 1 0 ultraep 482.64 1.00x
1684
+ echo 486.54 1.01x
1685
+
1686
+ a2 G4 4096 16 matched none 1 1 moonep 783.34 1.00x
1687
+
1688
+ ponly PONLY 4096 16 one none 1 0 echo 174.81 1.00x
1689
+ ultraep 176.76 1.01x
1690
+
1691
+ ponly PONLY 4096 16 one none 1 1 moonep 165.22 1.00x
1692
+
1693
+ ponly PONLY 8192 16 one none 1 0 ultraep 292.62 1.00x
1694
+ echo 316.70 1.08x
1695
+
1696
+ ponly PONLY 8192 16 one none 1 1 moonep 300.61 1.00x
1697
+
1698
+ tierbmem G1 4096 16 matched none 1 0 echo 474.40 1.00x
1699
+ ultraep 909.90 1.92x
1700
+
1701
+ tierbmem G1 4096 16 matched none 1 1 moonep 497.40 1.00x
1702
+
1703
+ tierbprobeG1 4096 16 matched none 0 0 echo 671.53 1.00x
1704
+ ultraep 904.16 1.35x
1705
+
1706
+ tierbprobeG2 4096 16 matched none 0 0 ultraep 512.37 1.00x
1707
+ echo 654.76 1.28x
1708
+
1709
+ tierbprobeG3 4096 16 matched none 0 0 ultraep 512.31 1.00x
1710
+ echo 689.13 1.35x
1711
+
1712
+ tierbprobeG4 4096 16 matched none 0 0 ultraep 600.19 1.00x
1713
+ echo 790.97 1.32x
1714
+