AbhishekGhosh1998 commited on
Commit
f6d5d4d
·
verified ·
1 Parent(s): a4f251b

granularity results tiera-rerun-fixed-moonep @ 2026-08-30T08:12:16Z from sp-aiinfra-interactive-aghosh-7jtjl-8xb200-moe-cg-0 (part 3)

Browse files
Files changed (1) hide show
  1. tiera-rerun-fixed-moonep/study.log +110 -0
tiera-rerun-fixed-moonep/study.log CHANGED
@@ -1450,3 +1450,113 @@ G1 ultraep 0 dynamic dynamic OK 106.7 40/
1450
 
1451
  ECHO/UltraEP note: graph-on uses static capacity + device-initiated GEMM while graph-off uses dynamic allocation + host-derived sizes. Their difference is not labelled as pure graph cost.
1452
  Already logged in to Hugging Face as: AbhishekGhosh1998
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1450
 
1451
  ECHO/UltraEP note: graph-on uses static capacity + device-initiated GEMM while graph-off uses dynamic allocation + host-derived sizes. Their difference is not labelled as pure graph cost.
1452
  Already logged in to Hugging Face as: AbhishekGhosh1998
1453
+ uploaded: /workspace/tiera-rerun-fixed-moonep -> MoE-CUDA-Graph-Analysis/tierb-b200-gbs8/tiera-rerun-fixed-moonep
1454
+ https://huggingface.co/MoE-CUDA-Graph-Analysis/tierb-b200-gbs8/tree/main
1455
+
1456
+ === STUDY COMPLETE
1457
+ results : /workspace/tiera-rerun-fixed-moonep
1458
+ status : /workspace/tiera-rerun-fixed-moonep/STATUS.txt
1459
+ steps : /workspace/tiera-rerun-fixed-moonep/steps.tsv
1460
+ hub : https://huggingface.co/datasets/MoE-CUDA-Graph-Analysis/tierb-b200-gbs8
1461
+
1462
+ status campaign phase gbs started ended result_dir
1463
+ FAIL echo tierbprobe 16 2026-08-30T04:52:38+00:00 2026-08-30T05:06:38+00:00 /workspace/tiera-rerun-fixed-moonep/echo/tierbprobe
1464
+ FAIL ultraep tierbprobe 16 2026-08-30T05:06:44+00:00 2026-08-30T05:20:40+00:00 /workspace/tiera-rerun-fixed-moonep/ultraep/tierbprobe
1465
+ OK echo a1 16 2026-08-30T06:20:31+00:00 2026-08-30T06:36:28+00:00 /workspace/tiera-rerun-fixed-moonep/echo/a1
1466
+ OK echo ponly 16 2026-08-30T06:36:31+00:00 2026-08-30T06:44:11+00:00 /workspace/tiera-rerun-fixed-moonep/echo/ponly
1467
+ OK echo a2 16 2026-08-30T06:44:13+00:00 2026-08-30T07:00:25+00:00 /workspace/tiera-rerun-fixed-moonep/echo/a2
1468
+ OK ultraep a1 16 2026-08-30T07:00:27+00:00 2026-08-30T07:15:17+00:00 /workspace/tiera-rerun-fixed-moonep/ultraep/a1
1469
+ OK ultraep ponly 16 2026-08-30T07:15:19+00:00 2026-08-30T07:22:01+00:00 /workspace/tiera-rerun-fixed-moonep/ultraep/ponly
1470
+ OK ultraep a2 16 2026-08-30T07:22:03+00:00 2026-08-30T07:37:07+00:00 /workspace/tiera-rerun-fixed-moonep/ultraep/a2
1471
+ OK moonep a1 16 2026-08-30T07:37:09+00:00 2026-08-30T07:41:05+00:00 /workspace/tiera-rerun-fixed-moonep/moonep/a1
1472
+ OK moonep ponly 16 2026-08-30T07:41:07+00:00 2026-08-30T07:48:20+00:00 /workspace/tiera-rerun-fixed-moonep/moonep/ponly
1473
+ OK moonep a2 16 2026-08-30T07:48:22+00:00 2026-08-30T08:03:34+00:00 /workspace/tiera-rerun-fixed-moonep/moonep/a2
1474
+ OK echo tierbmem 16 2026-08-30T08:03:47+00:00 2026-08-30T08:06:37+00:00 /workspace/tiera-rerun-fixed-moonep/echo/tierbmem
1475
+ OK ultraep tierbmem 16 2026-08-30T08:06:42+00:00 2026-08-30T08:09:23+00:00 /workspace/tiera-rerun-fixed-moonep/ultraep/tierbmem
1476
+ OK moonep tierbmem 16 2026-08-30T08:09:29+00:00 2026-08-30T08:11:55+00:00 /workspace/tiera-rerun-fixed-moonep/moonep/tierbmem
1477
+ study_root: /workspace/tiera-rerun-fixed-moonep
1478
+ gbs: 16
1479
+ stages: 8
1480
+ nsys: 0
1481
+ hf_repo: MoE-CUDA-Graph-Analysis/tierb-b200-gbs8
1482
+ hf_remote_root: tiera-rerun-fixed-moonep
1483
+ started: 2026-08-30T08:12:15+00:00
1484
+ driver_repo: /workspace/blackwell_run_scripts @ f69d9ab
1485
+
1486
+ === STAGE 8 -- Tier B allocator dumps (static CG capacities + dynamic no-CG)
1487
+
1488
+ === ROLL-UP
1489
+ 110 runs ok=107 oom=0 -> /workspace/tiera-rerun-fixed-moonep/collected_all.tsv
1490
+
1491
+ WARNING: 11 run(s) mention capacity saturation. A run that drops assignments does less expert work and must not be quoted as a win:
1492
+ ultraep/tierbprobe/tierbprobe_gbs16/G1__ultraep__matched__seq4096__gbs16__cap1p25/rep1: WARNING:megatron.training.drop_debug_utils:[MOE_DROP_DEBUG][iteration_summary] global_rank=0 iteration=0 records=32 total_expected_recv=283999 total_dropped_tok
1493
+ ultraep/tierbprobe/tierbprobe_gbs16/G1__ultraep__matched__seq4096__gbs16__cap1p8125/rep1: WARNING:megatron.training.drop_debug_utils:[MOE_DROP_DEBUG][iteration_summary] global_rank=0 iteration=0 records=32 total_expected_recv=284540 total_dropped_tok
1494
+ ultraep/tierbprobe/tierbprobe_gbs16/G2__ultraep__matched__seq4096__gbs16__cap1p0625/rep1: WARNING:megatron.training.drop_debug_utils:[MOE_DROP_DEBUG][iteration_summary] global_rank=0 iteration=0 records=32 total_expected_recv=1048800 total_dropped_to
1495
+ ultraep/tierbprobe/tierbprobe_gbs16/G2__ultraep__matched__seq4096__gbs16__cap1p25/rep1: WARNING:megatron.training.drop_debug_utils:[MOE_DROP_DEBUG][iteration_summary] global_rank=0 iteration=0 records=32 total_expected_recv=1048800 total_dropped_to
1496
+ ultraep/tierbprobe/tierbprobe_gbs16/G2__ultraep__matched__seq4096__gbs16__cap1p25__forcebal/rep1: WARNING:megatron.training.drop_debug_utils:[MOE_DROP_DEBUG][iteration_summary] global_rank=1 iteration=0 records=32 total_expected_recv=1047984 total_dropped_to
1497
+
1498
+ (excluded 2 nsys-traced run(s) from the timing comparison)
1499
+
1500
+ phase config seq gbs policy rc cg asy backend iter_ms vs_best
1501
+ -------------------------------------------------------------------------------------
1502
+ a1 G1 4096 16 one none 1 0 echo 466.68 1.00x
1503
+ ultraep 891.00 1.91x
1504
+
1505
+ a1 G1 4096 16 one none 1 1 moonep 480.30 1.00x
1506
+
1507
+ a1 G2 4096 16 one none 1 0 echo 394.28 1.00x
1508
+ ultraep 466.65 1.18x
1509
+
1510
+ a1 G3 4096 16 one none 1 0 echo 412.83 1.00x
1511
+ ultraep 447.84 1.08x
1512
+
1513
+ a1 G4 4096 16 one none 1 0 echo 461.03 1.00x
1514
+ ultraep 467.57 1.01x
1515
+
1516
+ a2 G1 4096 16 matched none 1 0 echo 466.91 1.00x
1517
+ ultraep 891.92 1.91x
1518
+
1519
+ a2 G1 4096 16 matched none 1 1 moonep 481.37 1.00x
1520
+
1521
+ a2 G2 4096 16 matched none 1 0 echo 415.21 1.00x
1522
+ ultraep 492.54 1.19x
1523
+
1524
+ a2 G2 4096 16 matched none 1 1 moonep 432.02 1.00x
1525
+
1526
+ a2 G3 4096 16 matched none 1 0 echo 437.35 1.00x
1527
+ ultraep 459.60 1.05x
1528
+
1529
+ a2 G3 4096 16 matched none 1 1 moonep 559.88 1.00x
1530
+
1531
+ a2 G4 4096 16 matched none 1 0 ultraep 482.64 1.00x
1532
+ echo 486.54 1.01x
1533
+
1534
+ a2 G4 4096 16 matched none 1 1 moonep 783.34 1.00x
1535
+
1536
+ ponly PONLY 4096 16 one none 1 0 echo 174.81 1.00x
1537
+ ultraep 176.76 1.01x
1538
+
1539
+ ponly PONLY 4096 16 one none 1 1 moonep 165.22 1.00x
1540
+
1541
+ ponly PONLY 8192 16 one none 1 0 ultraep 292.62 1.00x
1542
+ echo 316.70 1.08x
1543
+
1544
+ ponly PONLY 8192 16 one none 1 1 moonep 300.61 1.00x
1545
+
1546
+ tierbmem G1 4096 16 matched none 1 0 echo 474.40 1.00x
1547
+ ultraep 909.90 1.92x
1548
+
1549
+ tierbmem G1 4096 16 matched none 1 1 moonep 497.40 1.00x
1550
+
1551
+ tierbprobeG1 4096 16 matched none 0 0 echo 671.53 1.00x
1552
+ ultraep 904.16 1.35x
1553
+
1554
+ tierbprobeG2 4096 16 matched none 0 0 ultraep 512.37 1.00x
1555
+ echo 654.76 1.28x
1556
+
1557
+ tierbprobeG3 4096 16 matched none 0 0 ultraep 512.31 1.00x
1558
+ echo 689.13 1.35x
1559
+
1560
+ tierbprobeG4 4096 16 matched none 0 0 ultraep 600.19 1.00x
1561
+ echo 790.97 1.32x
1562
+