granularity results tiera-rerun-fixed-moonep @ 2026-08-30T08:12:29Z from sp-aiinfra-interactive-aghosh-7jtjl-8xb200-moe-cg-0 (part 3)
Browse files
tiera-rerun-fixed-moonep/study.log
CHANGED
|
@@ -1602,3 +1602,113 @@ G1 ultraep 0 dynamic dynamic OK 106.7 40/
|
|
| 1602 |
|
| 1603 |
ECHO/UltraEP note: graph-on uses static capacity + device-initiated GEMM while graph-off uses dynamic allocation + host-derived sizes. Their difference is not labelled as pure graph cost.
|
| 1604 |
Already logged in to Hugging Face as: AbhishekGhosh1998
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1602 |
|
| 1603 |
ECHO/UltraEP note: graph-on uses static capacity + device-initiated GEMM while graph-off uses dynamic allocation + host-derived sizes. Their difference is not labelled as pure graph cost.
|
| 1604 |
Already logged in to Hugging Face as: AbhishekGhosh1998
|
| 1605 |
+
uploaded: /workspace/tiera-rerun-fixed-moonep -> MoE-CUDA-Graph-Analysis/tierb-b200-gbs8/tiera-rerun-fixed-moonep
|
| 1606 |
+
https://huggingface.co/MoE-CUDA-Graph-Analysis/tierb-b200-gbs8/tree/main
|
| 1607 |
+
|
| 1608 |
+
=== STUDY COMPLETE
|
| 1609 |
+
results : /workspace/tiera-rerun-fixed-moonep
|
| 1610 |
+
status : /workspace/tiera-rerun-fixed-moonep/STATUS.txt
|
| 1611 |
+
steps : /workspace/tiera-rerun-fixed-moonep/steps.tsv
|
| 1612 |
+
hub : https://huggingface.co/datasets/MoE-CUDA-Graph-Analysis/tierb-b200-gbs8
|
| 1613 |
+
|
| 1614 |
+
status campaign phase gbs started ended result_dir
|
| 1615 |
+
FAIL echo tierbprobe 16 2026-08-30T04:52:38+00:00 2026-08-30T05:06:38+00:00 /workspace/tiera-rerun-fixed-moonep/echo/tierbprobe
|
| 1616 |
+
FAIL ultraep tierbprobe 16 2026-08-30T05:06:44+00:00 2026-08-30T05:20:40+00:00 /workspace/tiera-rerun-fixed-moonep/ultraep/tierbprobe
|
| 1617 |
+
OK echo a1 16 2026-08-30T06:20:31+00:00 2026-08-30T06:36:28+00:00 /workspace/tiera-rerun-fixed-moonep/echo/a1
|
| 1618 |
+
OK echo ponly 16 2026-08-30T06:36:31+00:00 2026-08-30T06:44:11+00:00 /workspace/tiera-rerun-fixed-moonep/echo/ponly
|
| 1619 |
+
OK echo a2 16 2026-08-30T06:44:13+00:00 2026-08-30T07:00:25+00:00 /workspace/tiera-rerun-fixed-moonep/echo/a2
|
| 1620 |
+
OK ultraep a1 16 2026-08-30T07:00:27+00:00 2026-08-30T07:15:17+00:00 /workspace/tiera-rerun-fixed-moonep/ultraep/a1
|
| 1621 |
+
OK ultraep ponly 16 2026-08-30T07:15:19+00:00 2026-08-30T07:22:01+00:00 /workspace/tiera-rerun-fixed-moonep/ultraep/ponly
|
| 1622 |
+
OK ultraep a2 16 2026-08-30T07:22:03+00:00 2026-08-30T07:37:07+00:00 /workspace/tiera-rerun-fixed-moonep/ultraep/a2
|
| 1623 |
+
OK moonep a1 16 2026-08-30T07:37:09+00:00 2026-08-30T07:41:05+00:00 /workspace/tiera-rerun-fixed-moonep/moonep/a1
|
| 1624 |
+
OK moonep ponly 16 2026-08-30T07:41:07+00:00 2026-08-30T07:48:20+00:00 /workspace/tiera-rerun-fixed-moonep/moonep/ponly
|
| 1625 |
+
OK moonep a2 16 2026-08-30T07:48:22+00:00 2026-08-30T08:03:34+00:00 /workspace/tiera-rerun-fixed-moonep/moonep/a2
|
| 1626 |
+
OK echo tierbmem 16 2026-08-30T08:03:47+00:00 2026-08-30T08:06:37+00:00 /workspace/tiera-rerun-fixed-moonep/echo/tierbmem
|
| 1627 |
+
OK ultraep tierbmem 16 2026-08-30T08:06:42+00:00 2026-08-30T08:09:23+00:00 /workspace/tiera-rerun-fixed-moonep/ultraep/tierbmem
|
| 1628 |
+
OK moonep tierbmem 16 2026-08-30T08:09:29+00:00 2026-08-30T08:11:55+00:00 /workspace/tiera-rerun-fixed-moonep/moonep/tierbmem
|
| 1629 |
+
study_root: /workspace/tiera-rerun-fixed-moonep
|
| 1630 |
+
gbs: 16
|
| 1631 |
+
stages: 8
|
| 1632 |
+
nsys: 0
|
| 1633 |
+
hf_repo: MoE-CUDA-Graph-Analysis/tierb-b200-gbs8
|
| 1634 |
+
hf_remote_root: tiera-rerun-fixed-moonep
|
| 1635 |
+
started: 2026-08-30T08:12:28+00:00
|
| 1636 |
+
driver_repo: /workspace/blackwell_run_scripts @ f69d9ab
|
| 1637 |
+
|
| 1638 |
+
=== STAGE 8 -- Tier B allocator dumps (static CG capacities + dynamic no-CG)
|
| 1639 |
+
|
| 1640 |
+
=== ROLL-UP
|
| 1641 |
+
110 runs ok=107 oom=0 -> /workspace/tiera-rerun-fixed-moonep/collected_all.tsv
|
| 1642 |
+
|
| 1643 |
+
WARNING: 11 run(s) mention capacity saturation. A run that drops assignments does less expert work and must not be quoted as a win:
|
| 1644 |
+
ultraep/tierbprobe/tierbprobe_gbs16/G1__ultraep__matched__seq4096__gbs16__cap1p25/rep1: WARNING:megatron.training.drop_debug_utils:[MOE_DROP_DEBUG][iteration_summary] global_rank=0 iteration=0 records=32 total_expected_recv=283999 total_dropped_tok
|
| 1645 |
+
ultraep/tierbprobe/tierbprobe_gbs16/G1__ultraep__matched__seq4096__gbs16__cap1p8125/rep1: WARNING:megatron.training.drop_debug_utils:[MOE_DROP_DEBUG][iteration_summary] global_rank=0 iteration=0 records=32 total_expected_recv=284540 total_dropped_tok
|
| 1646 |
+
ultraep/tierbprobe/tierbprobe_gbs16/G2__ultraep__matched__seq4096__gbs16__cap1p0625/rep1: WARNING:megatron.training.drop_debug_utils:[MOE_DROP_DEBUG][iteration_summary] global_rank=0 iteration=0 records=32 total_expected_recv=1048800 total_dropped_to
|
| 1647 |
+
ultraep/tierbprobe/tierbprobe_gbs16/G2__ultraep__matched__seq4096__gbs16__cap1p25/rep1: WARNING:megatron.training.drop_debug_utils:[MOE_DROP_DEBUG][iteration_summary] global_rank=0 iteration=0 records=32 total_expected_recv=1048800 total_dropped_to
|
| 1648 |
+
ultraep/tierbprobe/tierbprobe_gbs16/G2__ultraep__matched__seq4096__gbs16__cap1p25__forcebal/rep1: WARNING:megatron.training.drop_debug_utils:[MOE_DROP_DEBUG][iteration_summary] global_rank=1 iteration=0 records=32 total_expected_recv=1047984 total_dropped_to
|
| 1649 |
+
|
| 1650 |
+
(excluded 2 nsys-traced run(s) from the timing comparison)
|
| 1651 |
+
|
| 1652 |
+
phase config seq gbs policy rc cg asy backend iter_ms vs_best
|
| 1653 |
+
-------------------------------------------------------------------------------------
|
| 1654 |
+
a1 G1 4096 16 one none 1 0 echo 466.68 1.00x
|
| 1655 |
+
ultraep 891.00 1.91x
|
| 1656 |
+
|
| 1657 |
+
a1 G1 4096 16 one none 1 1 moonep 480.30 1.00x
|
| 1658 |
+
|
| 1659 |
+
a1 G2 4096 16 one none 1 0 echo 394.28 1.00x
|
| 1660 |
+
ultraep 466.65 1.18x
|
| 1661 |
+
|
| 1662 |
+
a1 G3 4096 16 one none 1 0 echo 412.83 1.00x
|
| 1663 |
+
ultraep 447.84 1.08x
|
| 1664 |
+
|
| 1665 |
+
a1 G4 4096 16 one none 1 0 echo 461.03 1.00x
|
| 1666 |
+
ultraep 467.57 1.01x
|
| 1667 |
+
|
| 1668 |
+
a2 G1 4096 16 matched none 1 0 echo 466.91 1.00x
|
| 1669 |
+
ultraep 891.92 1.91x
|
| 1670 |
+
|
| 1671 |
+
a2 G1 4096 16 matched none 1 1 moonep 481.37 1.00x
|
| 1672 |
+
|
| 1673 |
+
a2 G2 4096 16 matched none 1 0 echo 415.21 1.00x
|
| 1674 |
+
ultraep 492.54 1.19x
|
| 1675 |
+
|
| 1676 |
+
a2 G2 4096 16 matched none 1 1 moonep 432.02 1.00x
|
| 1677 |
+
|
| 1678 |
+
a2 G3 4096 16 matched none 1 0 echo 437.35 1.00x
|
| 1679 |
+
ultraep 459.60 1.05x
|
| 1680 |
+
|
| 1681 |
+
a2 G3 4096 16 matched none 1 1 moonep 559.88 1.00x
|
| 1682 |
+
|
| 1683 |
+
a2 G4 4096 16 matched none 1 0 ultraep 482.64 1.00x
|
| 1684 |
+
echo 486.54 1.01x
|
| 1685 |
+
|
| 1686 |
+
a2 G4 4096 16 matched none 1 1 moonep 783.34 1.00x
|
| 1687 |
+
|
| 1688 |
+
ponly PONLY 4096 16 one none 1 0 echo 174.81 1.00x
|
| 1689 |
+
ultraep 176.76 1.01x
|
| 1690 |
+
|
| 1691 |
+
ponly PONLY 4096 16 one none 1 1 moonep 165.22 1.00x
|
| 1692 |
+
|
| 1693 |
+
ponly PONLY 8192 16 one none 1 0 ultraep 292.62 1.00x
|
| 1694 |
+
echo 316.70 1.08x
|
| 1695 |
+
|
| 1696 |
+
ponly PONLY 8192 16 one none 1 1 moonep 300.61 1.00x
|
| 1697 |
+
|
| 1698 |
+
tierbmem G1 4096 16 matched none 1 0 echo 474.40 1.00x
|
| 1699 |
+
ultraep 909.90 1.92x
|
| 1700 |
+
|
| 1701 |
+
tierbmem G1 4096 16 matched none 1 1 moonep 497.40 1.00x
|
| 1702 |
+
|
| 1703 |
+
tierbprobeG1 4096 16 matched none 0 0 echo 671.53 1.00x
|
| 1704 |
+
ultraep 904.16 1.35x
|
| 1705 |
+
|
| 1706 |
+
tierbprobeG2 4096 16 matched none 0 0 ultraep 512.37 1.00x
|
| 1707 |
+
echo 654.76 1.28x
|
| 1708 |
+
|
| 1709 |
+
tierbprobeG3 4096 16 matched none 0 0 ultraep 512.31 1.00x
|
| 1710 |
+
echo 689.13 1.35x
|
| 1711 |
+
|
| 1712 |
+
tierbprobeG4 4096 16 matched none 0 0 ultraep 600.19 1.00x
|
| 1713 |
+
echo 790.97 1.32x
|
| 1714 |
+
|