Upload folder using huggingface_hub
Browse files- driver-logs-20260830/README.txt +14 -0
- driver-logs-20260830/capgate.log +25 -0
- driver-logs-20260830/capgate_g1.log +24 -0
- driver-logs-20260830/tiera1.log +397 -0
- driver-logs-20260830/tiera3.log +0 -0
- driver-logs-20260830/tiera_g1.log +241 -0
- driver-logs-20260830/tiera_mem.log +0 -0
- driver-logs-20260830/tierbfix4.log +242 -0
driver-logs-20260830/README.txt
ADDED
|
@@ -0,0 +1,14 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
Driver-level logs for the 2026-08-29/30 session.
|
| 2 |
+
|
| 3 |
+
These are the stdout of the campaign drivers, which is a superset of the
|
| 4 |
+
study.log stored inside each study root: it also captures the driver's own
|
| 5 |
+
decisions (capacity derivation, per-rung fallback, marker checks) and the two
|
| 6 |
+
capacity gates that were run outside any study root.
|
| 7 |
+
|
| 8 |
+
tiera1.log Tier A stage 7 capacity probe (halted: G1 unresolved)
|
| 9 |
+
tiera_g1.log G1 re-probe at a raised ceiling
|
| 10 |
+
capgate.log G4/echo at the global capacity 2.40625 -> OOM
|
| 11 |
+
capgate_g1.log G1/echo at the global capacity 2.40625 -> OK
|
| 12 |
+
tiera3.log Tier A timing + memory + traces
|
| 13 |
+
tiera_mem.log Tier A memory re-run after the marker fix
|
| 14 |
+
tierbfix4.log Tier B stage 9 MoonEP traces
|
driver-logs-20260830/capgate.log
ADDED
|
@@ -0,0 +1,25 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
==============================================================
|
| 2 |
+
CAMPAIGN: echo backend=echo phase=a2
|
| 3 |
+
Megatron: /workspace/Megatron-MoE-Echo @ bracing-regions-through-kernels
|
| 4 |
+
GBS=16 seq=4096 repeats=1 nsys=0
|
| 5 |
+
results -> /workspace/granularity_echo_20260830_054430
|
| 6 |
+
==============================================================
|
| 7 |
+
--------------------------------------------------------------------------
|
| 8 |
+
[05:44:30] CELL G4__echo__matched__seq4096__gbs16__cap2p40625 (slots=16)
|
| 9 |
+
[guard] tearing down process group (outcome=OOM)
|
| 10 |
+
[05:45:42] WARN: G4__echo__matched__seq4096__gbs16__cap2p40625 rep1 -> OOM
|
| 11 |
+
[05:45:42] waiting for stale ranks to exit ...
|
| 12 |
+
--------------------------------------------------------------------------
|
| 13 |
+
|
| 14 |
+
==============================================================
|
| 15 |
+
SWEEP COMPLETE ok=0 failed=1
|
| 16 |
+
==============================================================
|
| 17 |
+
config backend policy slots seq gbs cap recompute rep status iter_ms_mean iter_ms_min peak_mib run_dir
|
| 18 |
+
G4 echo matched 16 4096 16 2.40625 none 1 OOM NA NA 182627 /workspace/granularity_echo_20260830_054430/a2_matched_gbs16/G4__echo__matched__seq4096__gbs16__cap2p40625/rep1
|
| 19 |
+
|
| 20 |
+
Results: /workspace/granularity_echo_20260830_054430/a2_matched_gbs16/results.tsv
|
| 21 |
+
Roll up: python3 /workspace/blackwell_run_scripts/granularity/tools/collect.py /workspace/granularity_echo_20260830_054430/a2_matched_gbs16
|
| 22 |
+
==============================================================
|
| 23 |
+
|
| 24 |
+
Campaign results: /workspace/granularity_echo_20260830_054430
|
| 25 |
+
Roll up: python3 /workspace/blackwell_run_scripts/granularity/tools/collect.py /workspace/granularity_echo_20260830_054430 --compare
|
driver-logs-20260830/capgate_g1.log
ADDED
|
@@ -0,0 +1,24 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
==============================================================
|
| 2 |
+
CAMPAIGN: echo backend=echo phase=a2
|
| 3 |
+
Megatron: /workspace/Megatron-MoE-Echo @ bracing-regions-through-kernels
|
| 4 |
+
GBS=16 seq=4096 repeats=1 nsys=0
|
| 5 |
+
results -> /workspace/granularity_echo_20260830_054608
|
| 6 |
+
==============================================================
|
| 7 |
+
--------------------------------------------------------------------------
|
| 8 |
+
[05:46:09] CELL G1__echo__matched__seq4096__gbs16__cap2p40625 (slots=1)
|
| 9 |
+
[guard] iter 12/12 (70s)
|
| 10 |
+
[guard] tearing down process group (outcome=DONE)
|
| 11 |
+
--------------------------------------------------------------------------
|
| 12 |
+
|
| 13 |
+
==============================================================
|
| 14 |
+
SWEEP COMPLETE ok=1 failed=0
|
| 15 |
+
==============================================================
|
| 16 |
+
config backend policy slots seq gbs cap recompute rep status iter_ms_mean iter_ms_min peak_mib run_dir
|
| 17 |
+
G1 echo matched 1 4096 16 2.40625 none 1 OK 468.54 463.00 161863 /workspace/granularity_echo_20260830_054608/a2_matched_gbs16/G1__echo__matched__seq4096__gbs16__cap2p40625/rep1
|
| 18 |
+
|
| 19 |
+
Results: /workspace/granularity_echo_20260830_054608/a2_matched_gbs16/results.tsv
|
| 20 |
+
Roll up: python3 /workspace/blackwell_run_scripts/granularity/tools/collect.py /workspace/granularity_echo_20260830_054608/a2_matched_gbs16
|
| 21 |
+
==============================================================
|
| 22 |
+
|
| 23 |
+
Campaign results: /workspace/granularity_echo_20260830_054608
|
| 24 |
+
Roll up: python3 /workspace/blackwell_run_scripts/granularity/tools/collect.py /workspace/granularity_echo_20260830_054608 --compare
|
driver-logs-20260830/tiera1.log
ADDED
|
@@ -0,0 +1,397 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
[04:52:38] STAGE 7 -- Tier A capacity probe (echo, ultraep)
|
| 2 |
+
study_root: /workspace/tiera-rerun-fixed-moonep
|
| 3 |
+
gbs: 16
|
| 4 |
+
stages: 7
|
| 5 |
+
nsys: 1
|
| 6 |
+
hf_repo: MoE-CUDA-Graph-Analysis/tierb-b200-gbs8
|
| 7 |
+
hf_remote_root: tiera-rerun-fixed-moonep
|
| 8 |
+
started: 2026-08-30T04:52:38+00:00
|
| 9 |
+
driver_repo: /workspace/blackwell_run_scripts @ 774e9e2
|
| 10 |
+
[04:52:38] starting HF watcher: every 600s -> MoE-CUDA-Graph-Analysis/tierb-b200-gbs8
|
| 11 |
+
[04:52:38] watcher pid 439568 (log: /workspace/tiera-rerun-fixed-moonep/hf_watcher.log)
|
| 12 |
+
|
| 13 |
+
=== STAGE 7 -- Tier B probe and zero-drop capacity selection
|
| 14 |
+
|
| 15 |
+
=== STEP echo__tierbprobe__gbs16
|
| 16 |
+
[04:52:38] megatron: /workspace/Megatron-MoE-Echo moonep_backend_async_grad_reduce -> token-drop-analysis
|
| 17 |
+
[04:52:38] megatron: now on token-drop-analysis (b099fd84)
|
| 18 |
+
==============================================================
|
| 19 |
+
CAMPAIGN: echo backend=echo phase=tierbprobe
|
| 20 |
+
Megatron: /workspace/Megatron-MoE-Echo @ token-drop-analysis
|
| 21 |
+
GBS=16 seq=4096 repeats=3 nsys=1
|
| 22 |
+
results -> /workspace/tiera-rerun-fixed-moonep/echo/tierbprobe
|
| 23 |
+
==============================================================
|
| 24 |
+
--------------------------------------------------------------------------
|
| 25 |
+
[04:52:54] CELL G1__echo__matched__seq4096__gbs16__cap2p00 (slots=1)
|
| 26 |
+
[guard] iter 6/12 (70s)
|
| 27 |
+
[guard] iter 12/12 (80s)
|
| 28 |
+
--------------------------------------------------------------------------
|
| 29 |
+
[04:54:14] CELL G2__echo__matched__seq4096__gbs16__cap2p00 (slots=4)
|
| 30 |
+
[guard] iter 6/12 (70s)
|
| 31 |
+
[guard] iter 12/12 (80s)
|
| 32 |
+
[guard] tearing down process group (outcome=DONE)
|
| 33 |
+
[04:55:36] G2 echo: measured required=1.02734375 at C=2.00 -> validating C=1.08203125 (safety=+5%)
|
| 34 |
+
--------------------------------------------------------------------------
|
| 35 |
+
[04:55:36] CELL G2__echo__matched__seq4096__gbs16__cap1p08203125 (slots=4)
|
| 36 |
+
[guard] iter 5/12 (70s)
|
| 37 |
+
[guard] iter 12/12 (80s)
|
| 38 |
+
[guard] tearing down process group (outcome=DONE)
|
| 39 |
+
--------------------------------------------------------------------------
|
| 40 |
+
[04:56:58] CELL G2__echo__matched__seq4096__gbs16__cap1p08203125__forcebal (slots=4)
|
| 41 |
+
[guard] iter 9/12 (70s)
|
| 42 |
+
[guard] iter 12/12 (80s)
|
| 43 |
+
--------------------------------------------------------------------------
|
| 44 |
+
[04:58:18] CELL G3__echo__matched__seq4096__gbs16__cap2p00 (slots=8)
|
| 45 |
+
[guard] iter 5/12 (70s)
|
| 46 |
+
[guard] iter 12/12 (80s)
|
| 47 |
+
[guard] tearing down process group (outcome=DONE)
|
| 48 |
+
[04:59:41] G3 echo: measured required=1.01953125 at C=2.00 -> validating C=1.072265625 (safety=+5%)
|
| 49 |
+
--------------------------------------------------------------------------
|
| 50 |
+
[04:59:41] CELL G3__echo__matched__seq4096__gbs16__cap1p072265625 (slots=8)
|
| 51 |
+
[guard] iter 6/12 (70s)
|
| 52 |
+
[guard] iter 12/12 (80s)
|
| 53 |
+
[guard] tearing down process group (outcome=DONE)
|
| 54 |
+
--------------------------------------------------------------------------
|
| 55 |
+
[05:01:03] CELL G3__echo__matched__seq4096__gbs16__cap1p072265625__forcebal (slots=8)
|
| 56 |
+
[guard] iter 8/12 (70s)
|
| 57 |
+
[guard] iter 12/12 (80s)
|
| 58 |
+
--------------------------------------------------------------------------
|
| 59 |
+
[05:02:24] CELL G4__echo__matched__seq4096__gbs16__cap2p00 (slots=16)
|
| 60 |
+
[guard] iter 4/12 (70s)
|
| 61 |
+
[guard] iter 12/12 (80s)
|
| 62 |
+
[guard] tearing down process group (outcome=DONE)
|
| 63 |
+
[05:03:48] G4 echo: measured required=1.0166015625 at C=2.00 -> validating C=1.068359375 (safety=+5%)
|
| 64 |
+
--------------------------------------------------------------------------
|
| 65 |
+
[05:03:48] CELL G4__echo__matched__seq4096__gbs16__cap1p068359375 (slots=16)
|
| 66 |
+
[guard] iter 4/12 (70s)
|
| 67 |
+
[guard] iter 12/12 (80s)
|
| 68 |
+
[guard] tearing down process group (outcome=DONE)
|
| 69 |
+
--------------------------------------------------------------------------
|
| 70 |
+
[05:05:12] CELL G4__echo__matched__seq4096__gbs16__cap1p068359375__forcebal (slots=16)
|
| 71 |
+
[guard] iter 7/12 (70s)
|
| 72 |
+
[guard] iter 12/12 (80s)
|
| 73 |
+
[guard] tearing down process group (outcome=DONE)
|
| 74 |
+
--------------------------------------------------------------------------
|
| 75 |
+
|
| 76 |
+
==============================================================
|
| 77 |
+
SWEEP COMPLETE ok=10 failed=0
|
| 78 |
+
==============================================================
|
| 79 |
+
config backend policy slots seq gbs cap recompute rep status iter_ms_mean iter_ms_min peak_mib run_dir
|
| 80 |
+
G1 echo matched 1 4096 16 2.00 none 1 OK 671.53 666.90 125477 /workspace/tiera-rerun-fixed-moonep/echo/tierbprobe/tierbprobe_gbs16/G1__echo__matched__seq4096__gbs16__cap2p00/rep1
|
| 81 |
+
G2 echo matched 4 4096 16 2.00 none 1 OK 675.77 669.20 130289 /workspace/tiera-rerun-fixed-moonep/echo/tierbprobe/tierbprobe_gbs16/G2__echo__matched__seq4096__gbs16__cap2p00/rep1
|
| 82 |
+
G2 echo matched 4 4096 16 1.08203125 none 1 OK 660.43 642.00 119267 /workspace/tiera-rerun-fixed-moonep/echo/tierbprobe/tierbprobe_gbs16/G2__echo__matched__seq4096__gbs16__cap1p08203125/rep1
|
| 83 |
+
G2 echo matched 4 4096 16 1.08203125 none+forcebal 1 OK 628.07 616.10 119267 /workspace/tiera-rerun-fixed-moonep/echo/tierbprobe/tierbprobe_gbs16/G2__echo__matched__seq4096__gbs16__cap1p08203125__forcebal/rep1
|
| 84 |
+
G3 echo matched 8 4096 16 2.00 none 1 OK 721.76 713.60 136111 /workspace/tiera-rerun-fixed-moonep/echo/tierbprobe/tierbprobe_gbs16/G3__echo__matched__seq4096__gbs16__cap2p00/rep1
|
| 85 |
+
G3 echo matched 8 4096 16 1.072265625 none 1 OK 681.37 670.20 122853 /workspace/tiera-rerun-fixed-moonep/echo/tierbprobe/tierbprobe_gbs16/G3__echo__matched__seq4096__gbs16__cap1p072265625/rep1
|
| 86 |
+
G3 echo matched 8 4096 16 1.072265625 none+forcebal 1 OK 664.27 659.10 122899 /workspace/tiera-rerun-fixed-moonep/echo/tierbprobe/tierbprobe_gbs16/G3__echo__matched__seq4096__gbs16__cap1p072265625__forcebal/rep1
|
| 87 |
+
G4 echo matched 16 4096 16 2.00 none 1 OK 829.14 790.80 146037 /workspace/tiera-rerun-fixed-moonep/echo/tierbprobe/tierbprobe_gbs16/G4__echo__matched__seq4096__gbs16__cap2p00/rep1
|
| 88 |
+
G4 echo matched 16 4096 16 1.068359375 none 1 OK 775.73 738.90 129621 /workspace/tiera-rerun-fixed-moonep/echo/tierbprobe/tierbprobe_gbs16/G4__echo__matched__seq4096__gbs16__cap1p068359375/rep1
|
| 89 |
+
G4 echo matched 16 4096 16 1.068359375 none+forcebal 1 OK 768.04 733.40 129579 /workspace/tiera-rerun-fixed-moonep/echo/tierbprobe/tierbprobe_gbs16/G4__echo__matched__seq4096__gbs16__cap1p068359375__forcebal/rep1
|
| 90 |
+
|
| 91 |
+
Results: /workspace/tiera-rerun-fixed-moonep/echo/tierbprobe/tierbprobe_gbs16/results.tsv
|
| 92 |
+
Roll up: python3 /workspace/blackwell_run_scripts/granularity/tools/collect.py /workspace/tiera-rerun-fixed-moonep/echo/tierbprobe/tierbprobe_gbs16
|
| 93 |
+
==============================================================
|
| 94 |
+
10 probe runs -> /workspace/tiera-rerun-fixed-moonep/echo/tierbprobe/tierbprobe_gbs16/probe_summary.tsv
|
| 95 |
+
run arm cap rawExpI preRankI postRankI drop% reqCap
|
| 96 |
+
--------------------------------------------------------------------------------------------------------------------
|
| 97 |
+
G1__echo__matched__seq4096__gbs16__cap2p00/rep1 natural 2.00 3.6198* 3.6198 1.0827 1.9062
|
| 98 |
+
G2__echo__matched__seq4096__gbs16__cap1p08203125/rep natural 1.08203125 3.8720* 2.1703 1.0000 1.0273
|
| 99 |
+
G2__echo__matched__seq4096__gbs16__cap1p08203125__fo forced 1.08203125 1.0201 1.0069 1.0000 1.0234
|
| 100 |
+
G2__echo__matched__seq4096__gbs16__cap2p00/rep1 natural 2.00 3.8724* 2.1875 1.0000 1.0273
|
| 101 |
+
G3__echo__matched__seq4096__gbs16__cap1p072265625/re natural 1.072265625 3.9096* 1.7838 1.0000 1.0195
|
| 102 |
+
G3__echo__matched__seq4096__gbs16__cap1p072265625__f forced 1.072265625 1.0231 1.0048 1.0000 1.0176
|
| 103 |
+
G3__echo__matched__seq4096__gbs16__cap2p00/rep1 natural 2.00 3.9096* 1.7916 1.0000 1.0195
|
| 104 |
+
G4__echo__matched__seq4096__gbs16__cap1p068359375/re natural 1.068359375 3.9303* 1.5507 1.0000 1.0166
|
| 105 |
+
G4__echo__matched__seq4096__gbs16__cap1p068359375__f forced 1.068359375 1.0249 1.0034 1.0000 1.0137
|
| 106 |
+
G4__echo__matched__seq4096__gbs16__cap2p00/rep1 natural 2.00 3.9301* 1.5543 1.0000 1.0166
|
| 107 |
+
|
| 108 |
+
* At least one globally aggregated snapshot reached the E/K expert-Imax ceiling.
|
| 109 |
+
Inspect raw_expert_imax_mean and hottest_expert_token_fraction_mean before generalising.
|
| 110 |
+
|
| 111 |
+
ECHO note: echo_logged_pre_rank_metric_valid=0 and echo_logged_load_reduction_valid=0 are intentional.
|
| 112 |
+
The branch's built-in fields mix source-rank and destination-rank axes; this tool reconstructs both globally.
|
| 113 |
+
|
| 114 |
+
TIER B PROBE -- what these logs contain, per backend
|
| 115 |
+
----------------------------------------------------
|
| 116 |
+
Capacity matrix:
|
| 117 |
+
ECHO: start C=2.00; on OOM descend 1.75, 1.50, 1.25. Stop at
|
| 118 |
+
the highest fitting zero-drop capacity. If a fitting capacity drops,
|
| 119 |
+
smaller values cannot help and the model is blocked.
|
| 120 |
+
UltraEP: measure required_capacity at C=1.25, round it UP to the next
|
| 121 |
+
0.01, then validate that capacity once.
|
| 122 |
+
MoonEP: natural + forced, no capacity factor.
|
| 123 |
+
|
| 124 |
+
The forced arm runs once at the selected/high-headroom capacity. It validates
|
| 125 |
+
instrumentation and is not part of capacity selection.
|
| 126 |
+
|
| 127 |
+
Each backend instruments a different thing:
|
| 128 |
+
|
| 129 |
+
echo local raw-router histograms and local post-reroute placement.
|
| 130 |
+
reduce_probe.py aligns and sums all eight source ranks before
|
| 131 |
+
calculating raw_expert_*, pre_home_rank_*,
|
| 132 |
+
post_physical_rank_* and valid_max_rank_reduction_*.
|
| 133 |
+
|
| 134 |
+
ultraep global pre/post rank vectors plus explicit drop accounting:
|
| 135 |
+
drop_rate_pct and required_capacity -- the capacity factor the run
|
| 136 |
+
WOULD have needed. The vectors are duplicated identically on all
|
| 137 |
+
logging ranks and are deduplicated, not summed.
|
| 138 |
+
|
| 139 |
+
moonep local raw-router histograms, actual payload rows per physical
|
| 140 |
+
destination, and the bytes-copy side. The reducer globally sums
|
| 141 |
+
each (iteration, layer) snapshot first. ECHO reports no comparable
|
| 142 |
+
byte counters, so MoonEP bytes are a self-comparison across models,
|
| 143 |
+
never a cross-backend ranking.
|
| 144 |
+
|
| 145 |
+
Do NOT quote ECHO's logged pre_echo imbalance_ratio or
|
| 146 |
+
max_load_reduction_pct. The branch mixes source-rank and destination-rank
|
| 147 |
+
axes; the reducer marks both invalid and reconstructs the comparison globally.
|
| 148 |
+
|
| 149 |
+
Two boundary cases:
|
| 150 |
+
raw_expert_imax_max == E/K
|
| 151 |
+
-> at least one global snapshot had one expert in every
|
| 152 |
+
token's top-K set. Inspect the MEAN before generalising.
|
| 153 |
+
required_capacity or min-safe-capacity at the configured capacity
|
| 154 |
+
-> clipped by the buffer; it means "at or over", not
|
| 155 |
+
"fits comfortably".
|
| 156 |
+
|
| 157 |
+
tools/reduce_probe.py has already written probe_summary.tsv and .json here.
|
| 158 |
+
|
| 159 |
+
Campaign results: /workspace/tiera-rerun-fixed-moonep/echo/tierbprobe
|
| 160 |
+
Roll up: python3 /workspace/blackwell_run_scripts/granularity/tools/collect.py /workspace/tiera-rerun-fixed-moonep/echo/tierbprobe --compare
|
| 161 |
+
[05:06:38] WARN: step echo__tierbprobe__gbs16 failed (rc=1)
|
| 162 |
+
[05:06:38] step push: tiera-rerun-fixed-moonep/echo/tierbprobe -> MoE-CUDA-Graph-Analysis/tierb-b200-gbs8
|
| 163 |
+
Already logged in to Hugging Face as: AbhishekGhosh1998
|
| 164 |
+
uploaded: /workspace/tiera-rerun-fixed-moonep/echo/tierbprobe -> MoE-CUDA-Graph-Analysis/tierb-b200-gbs8/tiera-rerun-fixed-moonep/echo/tierbprobe
|
| 165 |
+
https://huggingface.co/MoE-CUDA-Graph-Analysis/tierb-b200-gbs8/tree/main
|
| 166 |
+
|
| 167 |
+
=== STEP ultraep__tierbprobe__gbs16
|
| 168 |
+
[05:06:43] campaign: granularity-echo -> granularity-ultraep
|
| 169 |
+
[05:06:43] megatron: /workspace/Megatron-UltraEP ultraep-bracing-regions-through-kernels -> ultraep-token-drop-analysis
|
| 170 |
+
[05:06:44] megatron: now on ultraep-token-drop-analysis (eaf829e7)
|
| 171 |
+
==============================================================
|
| 172 |
+
CAMPAIGN: ultraep backend=ultraep phase=tierbprobe
|
| 173 |
+
Megatron: /workspace/Megatron-UltraEP @ ultraep-token-drop-analysis
|
| 174 |
+
GBS=16 seq=4096 repeats=3 nsys=1
|
| 175 |
+
results -> /workspace/tiera-rerun-fixed-moonep/ultraep/tierbprobe
|
| 176 |
+
==============================================================
|
| 177 |
+
--------------------------------------------------------------------------
|
| 178 |
+
[05:06:59] CELL G1__ultraep__matched__seq4096__gbs16__cap1p25 (slots=1)
|
| 179 |
+
[guard] iter 4/12 (60s)
|
| 180 |
+
[guard] iter 12/12 (70s)
|
| 181 |
+
[guard] tearing down process group (outcome=DONE)
|
| 182 |
+
[05:08:13] G1 ultraep: validating C=1.8125 (safety=+5%)
|
| 183 |
+
--------------------------------------------------------------------------
|
| 184 |
+
[05:08:13] CELL G1__ultraep__matched__seq4096__gbs16__cap1p8125 (slots=1)
|
| 185 |
+
[guard] iter 4/12 (60s)
|
| 186 |
+
[guard] iter 12/12 (70s)
|
| 187 |
+
[guard] tearing down process group (outcome=DONE)
|
| 188 |
+
[05:09:28] G1 ultraep: C=1.8125 dropped; increasing headroom
|
| 189 |
+
--------------------------------------------------------------------------
|
| 190 |
+
[05:09:28] CELL G2__ultraep__matched__seq4096__gbs16__cap1p25 (slots=4)
|
| 191 |
+
[guard] iter 4/12 (60s)
|
| 192 |
+
[guard] iter 12/12 (70s)
|
| 193 |
+
[guard] tearing down process group (outcome=DONE)
|
| 194 |
+
[05:10:42] G2 ultraep: validating C=1.0625 (safety=+5%)
|
| 195 |
+
--------------------------------------------------------------------------
|
| 196 |
+
[05:10:42] CELL G2__ultraep__matched__seq4096__gbs16__cap1p0625 (slots=4)
|
| 197 |
+
[guard] iter 5/12 (60s)
|
| 198 |
+
[guard] iter 12/12 (70s)
|
| 199 |
+
[guard] tearing down process group (outcome=DONE)
|
| 200 |
+
--------------------------------------------------------------------------
|
| 201 |
+
[05:11:57] CELL G2__ultraep__matched__seq4096__gbs16__cap1p25__forcebal (slots=4)
|
| 202 |
+
[guard] iter 10/12 (60s)
|
| 203 |
+
[guard] iter 12/12 (70s)
|
| 204 |
+
[guard] tearing down process group (outcome=DONE)
|
| 205 |
+
--------------------------------------------------------------------------
|
| 206 |
+
[05:13:09] CELL G3__ultraep__matched__seq4096__gbs16__cap1p25 (slots=8)
|
| 207 |
+
[guard] iter 3/12 (60s)
|
| 208 |
+
[guard] iter 12/12 (70s)
|
| 209 |
+
[guard] tearing down process group (outcome=DONE)
|
| 210 |
+
[05:14:24] G3 ultraep: validating C=1.0625 (safety=+5%)
|
| 211 |
+
--------------------------------------------------------------------------
|
| 212 |
+
[05:14:24] CELL G3__ultraep__matched__seq4096__gbs16__cap1p0625 (slots=8)
|
| 213 |
+
[guard] iter 3/12 (60s)
|
| 214 |
+
[guard] iter 12/12 (70s)
|
| 215 |
+
[guard] tearing down process group (outcome=DONE)
|
| 216 |
+
--------------------------------------------------------------------------
|
| 217 |
+
[05:15:39] CELL G3__ultraep__matched__seq4096__gbs16__cap1p25__forcebal (slots=8)
|
| 218 |
+
[guard] iter 7/12 (60s)
|
| 219 |
+
[guard] iter 12/12 (70s)
|
| 220 |
+
[guard] tearing down process group (outcome=DONE)
|
| 221 |
+
--------------------------------------------------------------------------
|
| 222 |
+
[05:16:53] CELL G4__ultraep__matched__seq4096__gbs16__cap1p25 (slots=16)
|
| 223 |
+
[guard] iter 4/12 (60s)
|
| 224 |
+
[guard] iter 12/12 (70s)
|
| 225 |
+
[guard] tearing down process group (outcome=DONE)
|
| 226 |
+
[05:18:08] G4 ultraep: validating C=1.095703125 (safety=+5%)
|
| 227 |
+
--------------------------------------------------------------------------
|
| 228 |
+
[05:18:08] CELL G4__ultraep__matched__seq4096__gbs16__cap1p095703125 (slots=16)
|
| 229 |
+
[guard] iter 3/12 (60s)
|
| 230 |
+
[guard] iter 12/12 (70s)
|
| 231 |
+
[guard] tearing down process group (outcome=DONE)
|
| 232 |
+
--------------------------------------------------------------------------
|
| 233 |
+
[05:19:24] CELL G4__ultraep__matched__seq4096__gbs16__cap1p25__forcebal (slots=16)
|
| 234 |
+
[guard] iter 6/12 (60s)
|
| 235 |
+
[guard] iter 12/12 (70s)
|
| 236 |
+
[guard] tearing down process group (outcome=DONE)
|
| 237 |
+
--------------------------------------------------------------------------
|
| 238 |
+
|
| 239 |
+
==============================================================
|
| 240 |
+
SWEEP COMPLETE ok=11 failed=0
|
| 241 |
+
==============================================================
|
| 242 |
+
config backend policy slots seq gbs cap recompute rep status iter_ms_mean iter_ms_min peak_mib run_dir
|
| 243 |
+
G1 ultraep matched 1 4096 16 1.25 none 1 OK 893.49 886.90 108731 /workspace/tiera-rerun-fixed-moonep/ultraep/tierbprobe/tierbprobe_gbs16/G1__ultraep__matched__seq4096__gbs16__cap1p25/rep1
|
| 244 |
+
G1 ultraep matched 1 4096 16 1.8125 none 1 OK 914.84 903.60 113767 /workspace/tiera-rerun-fixed-moonep/ultraep/tierbprobe/tierbprobe_gbs16/G1__ultraep__matched__seq4096__gbs16__cap1p8125/rep1
|
| 245 |
+
G2 ultraep matched 4 4096 16 1.25 none 1 OK 558.91 532.70 110343 /workspace/tiera-rerun-fixed-moonep/ultraep/tierbprobe/tierbprobe_gbs16/G2__ultraep__matched__seq4096__gbs16__cap1p25/rep1
|
| 246 |
+
G2 ultraep matched 4 4096 16 1.0625 none 1 OK 551.94 527.00 107987 /workspace/tiera-rerun-fixed-moonep/ultraep/tierbprobe/tierbprobe_gbs16/G2__ultraep__matched__seq4096__gbs16__cap1p0625/rep1
|
| 247 |
+
G2 ultraep matched 4 4096 16 1.25 none+forcebal 1 OK 426.26 422.10 110343 /workspace/tiera-rerun-fixed-moonep/ultraep/tierbprobe/tierbprobe_gbs16/G2__ultraep__matched__seq4096__gbs16__cap1p25__forcebal/rep1
|
| 248 |
+
G3 ultraep matched 8 4096 16 1.25 none 1 OK 536.43 529.00 114319 /workspace/tiera-rerun-fixed-moonep/ultraep/tierbprobe/tierbprobe_gbs16/G3__ultraep__matched__seq4096__gbs16__cap1p25/rep1
|
| 249 |
+
G3 ultraep matched 8 4096 16 1.0625 none 1 OK 528.17 522.80 111567 /workspace/tiera-rerun-fixed-moonep/ultraep/tierbprobe/tierbprobe_gbs16/G3__ultraep__matched__seq4096__gbs16__cap1p0625/rep1
|
| 250 |
+
G3 ultraep matched 8 4096 16 1.25 none+forcebal 1 OK 472.34 470.60 114267 /workspace/tiera-rerun-fixed-moonep/ultraep/tierbprobe/tierbprobe_gbs16/G3__ultraep__matched__seq4096__gbs16__cap1p25__forcebal/rep1
|
| 251 |
+
G4 ultraep matched 16 4096 16 1.25 none 1 OK 610.06 573.90 120487 /workspace/tiera-rerun-fixed-moonep/ultraep/tierbprobe/tierbprobe_gbs16/G4__ultraep__matched__seq4096__gbs16__cap1p25/rep1
|
| 252 |
+
G4 ultraep matched 16 4096 16 1.095703125 none 1 OK 594.23 563.90 118667 /workspace/tiera-rerun-fixed-moonep/ultraep/tierbprobe/tierbprobe_gbs16/G4__ultraep__matched__seq4096__gbs16__cap1p095703125/rep1
|
| 253 |
+
G4 ultraep matched 16 4096 16 1.25 none+forcebal 1 OK 596.29 545.60 120487 /workspace/tiera-rerun-fixed-moonep/ultraep/tierbprobe/tierbprobe_gbs16/G4__ultraep__matched__seq4096__gbs16__cap1p25__forcebal/rep1
|
| 254 |
+
|
| 255 |
+
Results: /workspace/tiera-rerun-fixed-moonep/ultraep/tierbprobe/tierbprobe_gbs16/results.tsv
|
| 256 |
+
Roll up: python3 /workspace/blackwell_run_scripts/granularity/tools/collect.py /workspace/tiera-rerun-fixed-moonep/ultraep/tierbprobe/tierbprobe_gbs16
|
| 257 |
+
==============================================================
|
| 258 |
+
11 probe runs -> /workspace/tiera-rerun-fixed-moonep/ultraep/tierbprobe/tierbprobe_gbs16/probe_summary.tsv
|
| 259 |
+
run arm cap rawExpI preRankI postRankI drop% reqCap
|
| 260 |
+
--------------------------------------------------------------------------------------------------------------------
|
| 261 |
+
G1__ultraep__matched__seq4096__gbs16__cap1p25/rep1 natural 1.25 3.6085 1.0640 2.8918 1.7246
|
| 262 |
+
G1__ultraep__matched__seq4096__gbs16__cap1p8125/rep1 natural 1.8125 3.6152 1.0747 1.5460 2.3091
|
| 263 |
+
G2__ultraep__matched__seq4096__gbs16__cap1p0625/rep1 natural 1.0625 2.2032 1.0100 0.0000 1.0101
|
| 264 |
+
G2__ultraep__matched__seq4096__gbs16__cap1p25/rep1 natural 1.25 2.2136 1.0100 0.0000 1.0101
|
| 265 |
+
G2__ultraep__matched__seq4096__gbs16__cap1p25__force forced 1.25 1.0069 1.0067 0.0000 1.0100
|
| 266 |
+
G3__ultraep__matched__seq4096__gbs16__cap1p0625/rep1 natural 1.0625 1.7842 1.0100 0.0000 1.0101
|
| 267 |
+
G3__ultraep__matched__seq4096__gbs16__cap1p25/rep1 natural 1.25 1.7752 1.0100 0.0000 1.0101
|
| 268 |
+
G3__ultraep__matched__seq4096__gbs16__cap1p25__force forced 1.25 1.0047 1.0047 0.0000 1.0100
|
| 269 |
+
G4__ultraep__matched__seq4096__gbs16__cap1p095703125 natural 1.095703125 1.5612 1.0100 0.0000 1.0429
|
| 270 |
+
G4__ultraep__matched__seq4096__gbs16__cap1p25/rep1 natural 1.25 1.5504 1.0100 0.0000 1.0429
|
| 271 |
+
G4__ultraep__matched__seq4096__gbs16__cap1p25__force forced 1.25 1.0033 1.0033 0.0000 1.0071
|
| 272 |
+
|
| 273 |
+
ECHO note: echo_logged_pre_rank_metric_valid=0 and echo_logged_load_reduction_valid=0 are intentional.
|
| 274 |
+
The branch's built-in fields mix source-rank and destination-rank axes; this tool reconstructs both globally.
|
| 275 |
+
|
| 276 |
+
TIER B PROBE -- what these logs contain, per backend
|
| 277 |
+
----------------------------------------------------
|
| 278 |
+
Capacity matrix:
|
| 279 |
+
ECHO: start C=2.00; on OOM descend 1.75, 1.50, 1.25. Stop at
|
| 280 |
+
the highest fitting zero-drop capacity. If a fitting capacity drops,
|
| 281 |
+
smaller values cannot help and the model is blocked.
|
| 282 |
+
UltraEP: measure required_capacity at C=1.25, round it UP to the next
|
| 283 |
+
0.01, then validate that capacity once.
|
| 284 |
+
MoonEP: natural + forced, no capacity factor.
|
| 285 |
+
|
| 286 |
+
The forced arm runs once at the selected/high-headroom capacity. It validates
|
| 287 |
+
instrumentation and is not part of capacity selection.
|
| 288 |
+
|
| 289 |
+
Each backend instruments a different thing:
|
| 290 |
+
|
| 291 |
+
echo local raw-router histograms and local post-reroute placement.
|
| 292 |
+
reduce_probe.py aligns and sums all eight source ranks before
|
| 293 |
+
calculating raw_expert_*, pre_home_rank_*,
|
| 294 |
+
post_physical_rank_* and valid_max_rank_reduction_*.
|
| 295 |
+
|
| 296 |
+
ultraep global pre/post rank vectors plus explicit drop accounting:
|
| 297 |
+
drop_rate_pct and required_capacity -- the capacity factor the run
|
| 298 |
+
WOULD have needed. The vectors are duplicated identically on all
|
| 299 |
+
logging ranks and are deduplicated, not summed.
|
| 300 |
+
|
| 301 |
+
moonep local raw-router histograms, actual payload rows per physical
|
| 302 |
+
destination, and the bytes-copy side. The reducer globally sums
|
| 303 |
+
each (iteration, layer) snapshot first. ECHO reports no comparable
|
| 304 |
+
byte counters, so MoonEP bytes are a self-comparison across models,
|
| 305 |
+
never a cross-backend ranking.
|
| 306 |
+
|
| 307 |
+
Do NOT quote ECHO's logged pre_echo imbalance_ratio or
|
| 308 |
+
max_load_reduction_pct. The branch mixes source-rank and destination-rank
|
| 309 |
+
axes; the reducer marks both invalid and reconstructs the comparison globally.
|
| 310 |
+
|
| 311 |
+
Two boundary cases:
|
| 312 |
+
raw_expert_imax_max == E/K
|
| 313 |
+
-> at least one global snapshot had one expert in every
|
| 314 |
+
token's top-K set. Inspect the MEAN before generalising.
|
| 315 |
+
required_capacity or min-safe-capacity at the configured capacity
|
| 316 |
+
-> clipped by the buffer; it means "at or over", not
|
| 317 |
+
"fits comfortably".
|
| 318 |
+
|
| 319 |
+
tools/reduce_probe.py has already written probe_summary.tsv and .json here.
|
| 320 |
+
|
| 321 |
+
Campaign results: /workspace/tiera-rerun-fixed-moonep/ultraep/tierbprobe
|
| 322 |
+
Roll up: python3 /workspace/blackwell_run_scripts/granularity/tools/collect.py /workspace/tiera-rerun-fixed-moonep/ultraep/tierbprobe --compare
|
| 323 |
+
[05:20:40] WARN: step ultraep__tierbprobe__gbs16 failed (rc=1)
|
| 324 |
+
[05:20:40] step push: tiera-rerun-fixed-moonep/ultraep/tierbprobe -> MoE-CUDA-Graph-Analysis/tierb-b200-gbs8
|
| 325 |
+
Already logged in to Hugging Face as: AbhishekGhosh1998
|
| 326 |
+
uploaded: /workspace/tiera-rerun-fixed-moonep/ultraep/tierbprobe -> MoE-CUDA-Graph-Analysis/tierb-b200-gbs8/tiera-rerun-fixed-moonep/ultraep/tierbprobe
|
| 327 |
+
https://huggingface.co/MoE-CUDA-Graph-Analysis/tierb-b200-gbs8/tree/main
|
| 328 |
+
|
| 329 |
+
=== ROLL-UP
|
| 330 |
+
21 runs ok=21 oom=0 -> /workspace/tiera-rerun-fixed-moonep/collected_all.tsv
|
| 331 |
+
|
| 332 |
+
WARNING: 11 run(s) mention capacity saturation. A run that drops assignments does less expert work and must not be quoted as a win:
|
| 333 |
+
ultraep/tierbprobe/tierbprobe_gbs16/G1__ultraep__matched__seq4096__gbs16__cap1p25/rep1: WARNING:megatron.training.drop_debug_utils:[MOE_DROP_DEBUG][iteration_summary] global_rank=0 iteration=0 records=32 total_expected_recv=283999 total_dropped_tok
|
| 334 |
+
ultraep/tierbprobe/tierbprobe_gbs16/G1__ultraep__matched__seq4096__gbs16__cap1p8125/rep1: WARNING:megatron.training.drop_debug_utils:[MOE_DROP_DEBUG][iteration_summary] global_rank=0 iteration=0 records=32 total_expected_recv=284540 total_dropped_tok
|
| 335 |
+
ultraep/tierbprobe/tierbprobe_gbs16/G2__ultraep__matched__seq4096__gbs16__cap1p0625/rep1: WARNING:megatron.training.drop_debug_utils:[MOE_DROP_DEBUG][iteration_summary] global_rank=0 iteration=0 records=32 total_expected_recv=1048800 total_dropped_to
|
| 336 |
+
ultraep/tierbprobe/tierbprobe_gbs16/G2__ultraep__matched__seq4096__gbs16__cap1p25/rep1: WARNING:megatron.training.drop_debug_utils:[MOE_DROP_DEBUG][iteration_summary] global_rank=0 iteration=0 records=32 total_expected_recv=1048800 total_dropped_to
|
| 337 |
+
ultraep/tierbprobe/tierbprobe_gbs16/G2__ultraep__matched__seq4096__gbs16__cap1p25__forcebal/rep1: WARNING:megatron.training.drop_debug_utils:[MOE_DROP_DEBUG][iteration_summary] global_rank=1 iteration=0 records=32 total_expected_recv=1047984 total_dropped_to
|
| 338 |
+
|
| 339 |
+
phase config seq gbs policy rc cg asy backend iter_ms vs_best
|
| 340 |
+
-------------------------------------------------------------------------------------
|
| 341 |
+
tierbprobeG1 4096 16 matched none 0 0 echo 671.53 1.00x
|
| 342 |
+
ultraep 904.16 1.35x
|
| 343 |
+
|
| 344 |
+
tierbprobeG2 4096 16 matched none 0 0 ultraep 512.37 1.00x
|
| 345 |
+
echo 654.76 1.28x
|
| 346 |
+
|
| 347 |
+
tierbprobeG3 4096 16 matched none 0 0 ultraep 512.31 1.00x
|
| 348 |
+
echo 689.13 1.35x
|
| 349 |
+
|
| 350 |
+
tierbprobeG4 4096 16 matched none 0 0 ultraep 600.19 1.00x
|
| 351 |
+
echo 790.97 1.32x
|
| 352 |
+
|
| 353 |
+
21 probe runs -> /workspace/tiera-rerun-fixed-moonep/probe_summary_all.tsv
|
| 354 |
+
run arm cap rawExpI preRankI postRankI drop% reqCap
|
| 355 |
+
--------------------------------------------------------------------------------------------------------------------
|
| 356 |
+
echo/tierbprobe/tierbprobe_gbs16/G1__echo__matched__ natural 2.00 3.6198* 3.6198 1.0827 1.9062
|
| 357 |
+
echo/tierbprobe/tierbprobe_gbs16/G2__echo__matched__ natural 1.08203125 3.8720* 2.1703 1.0000 1.0273
|
| 358 |
+
echo/tierbprobe/tierbprobe_gbs16/G2__echo__matched__ forced 1.08203125 1.0201 1.0069 1.0000 1.0234
|
| 359 |
+
echo/tierbprobe/tierbprobe_gbs16/G2__echo__matched__ natural 2.00 3.8724* 2.1875 1.0000 1.0273
|
| 360 |
+
echo/tierbprobe/tierbprobe_gbs16/G3__echo__matched__ natural 1.072265625 3.9096* 1.7838 1.0000 1.0195
|
| 361 |
+
echo/tierbprobe/tierbprobe_gbs16/G3__echo__matched__ forced 1.072265625 1.0231 1.0048 1.0000 1.0176
|
| 362 |
+
echo/tierbprobe/tierbprobe_gbs16/G3__echo__matched__ natural 2.00 3.9096* 1.7916 1.0000 1.0195
|
| 363 |
+
echo/tierbprobe/tierbprobe_gbs16/G4__echo__matched__ natural 1.068359375 3.9303* 1.5507 1.0000 1.0166
|
| 364 |
+
echo/tierbprobe/tierbprobe_gbs16/G4__echo__matched__ forced 1.068359375 1.0249 1.0034 1.0000 1.0137
|
| 365 |
+
echo/tierbprobe/tierbprobe_gbs16/G4__echo__matched__ natural 2.00 3.9301* 1.5543 1.0000 1.0166
|
| 366 |
+
ultraep/tierbprobe/tierbprobe_gbs16/G1__ultraep__mat natural 1.25 3.6085 1.0640 2.8918 1.7246
|
| 367 |
+
ultraep/tierbprobe/tierbprobe_gbs16/G1__ultraep__mat natural 1.8125 3.6152 1.0747 1.5460 2.3091
|
| 368 |
+
ultraep/tierbprobe/tierbprobe_gbs16/G2__ultraep__mat natural 1.0625 2.2032 1.0100 0.0000 1.0101
|
| 369 |
+
ultraep/tierbprobe/tierbprobe_gbs16/G2__ultraep__mat natural 1.25 2.2136 1.0100 0.0000 1.0101
|
| 370 |
+
ultraep/tierbprobe/tierbprobe_gbs16/G2__ultraep__mat forced 1.25 1.0069 1.0067 0.0000 1.0100
|
| 371 |
+
ultraep/tierbprobe/tierbprobe_gbs16/G3__ultraep__mat natural 1.0625 1.7842 1.0100 0.0000 1.0101
|
| 372 |
+
ultraep/tierbprobe/tierbprobe_gbs16/G3__ultraep__mat natural 1.25 1.7752 1.0100 0.0000 1.0101
|
| 373 |
+
ultraep/tierbprobe/tierbprobe_gbs16/G3__ultraep__mat forced 1.25 1.0047 1.0047 0.0000 1.0100
|
| 374 |
+
ultraep/tierbprobe/tierbprobe_gbs16/G4__ultraep__mat natural 1.095703125 1.5612 1.0100 0.0000 1.0429
|
| 375 |
+
ultraep/tierbprobe/tierbprobe_gbs16/G4__ultraep__mat natural 1.25 1.5504 1.0100 0.0000 1.0429
|
| 376 |
+
ultraep/tierbprobe/tierbprobe_gbs16/G4__ultraep__mat forced 1.25 1.0033 1.0033 0.0000 1.0071
|
| 377 |
+
|
| 378 |
+
* At least one globally aggregated snapshot reached the E/K expert-Imax ceiling.
|
| 379 |
+
Inspect raw_expert_imax_mean and hottest_expert_token_fraction_mean before generalising.
|
| 380 |
+
|
| 381 |
+
ECHO note: echo_logged_pre_rank_metric_valid=0 and echo_logged_load_reduction_valid=0 are intentional.
|
| 382 |
+
The branch's built-in fields mix source-rank and destination-rank axes; this tool reconstructs both globally.
|
| 383 |
+
no memory-dump runs found under /workspace/tiera-rerun-fixed-moonep
|
| 384 |
+
[05:20:51] step push: tiera-rerun-fixed-moonep -> MoE-CUDA-Graph-Analysis/tierb-b200-gbs8
|
| 385 |
+
Already logged in to Hugging Face as: AbhishekGhosh1998
|
| 386 |
+
uploaded: /workspace/tiera-rerun-fixed-moonep -> MoE-CUDA-Graph-Analysis/tierb-b200-gbs8/tiera-rerun-fixed-moonep
|
| 387 |
+
https://huggingface.co/MoE-CUDA-Graph-Analysis/tierb-b200-gbs8/tree/main
|
| 388 |
+
|
| 389 |
+
=== STUDY COMPLETE
|
| 390 |
+
results : /workspace/tiera-rerun-fixed-moonep
|
| 391 |
+
status : /workspace/tiera-rerun-fixed-moonep/STATUS.txt
|
| 392 |
+
steps : /workspace/tiera-rerun-fixed-moonep/steps.tsv
|
| 393 |
+
hub : https://huggingface.co/datasets/MoE-CUDA-Graph-Analysis/tierb-b200-gbs8
|
| 394 |
+
|
| 395 |
+
status campaign phase gbs started ended result_dir
|
| 396 |
+
FAIL echo tierbprobe 16 2026-08-30T04:52:38+00:00 2026-08-30T05:06:38+00:00 /workspace/tiera-rerun-fixed-moonep/echo/tierbprobe
|
| 397 |
+
FAIL ultraep tierbprobe 16 2026-08-30T05:06:44+00:00 2026-08-30T05:20:40+00:00 /workspace/tiera-rerun-fixed-moonep/ultraep/tierbprobe
|
driver-logs-20260830/tiera3.log
ADDED
|
The diff for this file is too large to render.
See raw diff
|
|
|
driver-logs-20260830/tiera_g1.log
ADDED
|
@@ -0,0 +1,241 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
study_root: /workspace/tiera-g1-reprobe
|
| 2 |
+
gbs: 16
|
| 3 |
+
stages: 7
|
| 4 |
+
nsys: 1
|
| 5 |
+
hf_repo: none
|
| 6 |
+
hf_remote_root: tiera-g1-reprobe
|
| 7 |
+
started: 2026-08-30T05:29:23+00:00
|
| 8 |
+
driver_repo: /workspace/blackwell_run_scripts @ 818ab62
|
| 9 |
+
|
| 10 |
+
=== STAGE 7 -- Tier B probe and zero-drop capacity selection
|
| 11 |
+
|
| 12 |
+
=== STEP echo__tierbprobe__gbs16
|
| 13 |
+
[05:29:23] campaign: granularity-ultraep -> granularity-echo
|
| 14 |
+
[05:29:23] megatron: /workspace/Megatron-MoE-Echo already on token-drop-analysis (b099fd84)
|
| 15 |
+
==============================================================
|
| 16 |
+
CAMPAIGN: echo backend=echo phase=tierbprobe
|
| 17 |
+
Megatron: /workspace/Megatron-MoE-Echo @ token-drop-analysis
|
| 18 |
+
GBS=16 seq=4096 repeats=3 nsys=1
|
| 19 |
+
results -> /workspace/tiera-g1-reprobe/echo/tierbprobe
|
| 20 |
+
==============================================================
|
| 21 |
+
--------------------------------------------------------------------------
|
| 22 |
+
[05:29:38] CELL G1__echo__matched__seq4096__gbs16__cap3p00 (slots=1)
|
| 23 |
+
[guard] iter 8/12 (70s)
|
| 24 |
+
[guard] iter 12/12 (80s)
|
| 25 |
+
[05:30:59] G1 echo: measured required=1.765625 at C=3.00 -> validating C=1.859375 (safety=+5%)
|
| 26 |
+
--------------------------------------------------------------------------
|
| 27 |
+
[05:30:59] CELL G1__echo__matched__seq4096__gbs16__cap1p859375 (slots=1)
|
| 28 |
+
[guard] iter 8/12 (70s)
|
| 29 |
+
[guard] iter 12/12 (80s)
|
| 30 |
+
--------------------------------------------------------------------------
|
| 31 |
+
[05:32:20] CELL G1__echo__matched__seq4096__gbs16__cap1p859375__forcebal (slots=1)
|
| 32 |
+
[guard] iter 10/12 (70s)
|
| 33 |
+
[guard] iter 12/12 (80s)
|
| 34 |
+
--------------------------------------------------------------------------
|
| 35 |
+
|
| 36 |
+
==============================================================
|
| 37 |
+
SWEEP COMPLETE ok=3 failed=0
|
| 38 |
+
==============================================================
|
| 39 |
+
config backend policy slots seq gbs cap recompute rep status iter_ms_mean iter_ms_min peak_mib run_dir
|
| 40 |
+
G1 echo matched 1 4096 16 3.00 none 1 OK 696.00 690.60 135941 /workspace/tiera-g1-reprobe/echo/tierbprobe/tierbprobe_gbs16/G1__echo__matched__seq4096__gbs16__cap3p00/rep1
|
| 41 |
+
G1 echo matched 1 4096 16 1.859375 none 1 OK 678.00 673.80 124457 /workspace/tiera-g1-reprobe/echo/tierbprobe/tierbprobe_gbs16/G1__echo__matched__seq4096__gbs16__cap1p859375/rep1
|
| 42 |
+
G1 echo matched 1 4096 16 1.859375 none+forcebal 1 OK 680.44 624.70 124455 /workspace/tiera-g1-reprobe/echo/tierbprobe/tierbprobe_gbs16/G1__echo__matched__seq4096__gbs16__cap1p859375__forcebal/rep1
|
| 43 |
+
|
| 44 |
+
Results: /workspace/tiera-g1-reprobe/echo/tierbprobe/tierbprobe_gbs16/results.tsv
|
| 45 |
+
Roll up: python3 /workspace/blackwell_run_scripts/granularity/tools/collect.py /workspace/tiera-g1-reprobe/echo/tierbprobe/tierbprobe_gbs16
|
| 46 |
+
==============================================================
|
| 47 |
+
3 probe runs -> /workspace/tiera-g1-reprobe/echo/tierbprobe/tierbprobe_gbs16/probe_summary.tsv
|
| 48 |
+
run arm cap rawExpI preRankI postRankI drop% reqCap
|
| 49 |
+
--------------------------------------------------------------------------------------------------------------------
|
| 50 |
+
G1__echo__matched__seq4096__gbs16__cap1p859375/rep1 natural 1.859375 3.6222* 3.6222 1.0790 1.6406
|
| 51 |
+
G1__echo__matched__seq4096__gbs16__cap1p859375__forc forced 1.859375 1.0145 1.0145 1.0041 1.0156
|
| 52 |
+
G1__echo__matched__seq4096__gbs16__cap3p00/rep1 natural 3.00 3.6267* 3.6267 1.0815 1.7656
|
| 53 |
+
|
| 54 |
+
* At least one globally aggregated snapshot reached the E/K expert-Imax ceiling.
|
| 55 |
+
Inspect raw_expert_imax_mean and hottest_expert_token_fraction_mean before generalising.
|
| 56 |
+
|
| 57 |
+
ECHO note: echo_logged_pre_rank_metric_valid=0 and echo_logged_load_reduction_valid=0 are intentional.
|
| 58 |
+
The branch's built-in fields mix source-rank and destination-rank axes; this tool reconstructs both globally.
|
| 59 |
+
|
| 60 |
+
TIER B PROBE -- what these logs contain, per backend
|
| 61 |
+
----------------------------------------------------
|
| 62 |
+
Capacity matrix:
|
| 63 |
+
ECHO: start C=2.00; on OOM descend 1.75, 1.50, 1.25. Stop at
|
| 64 |
+
the highest fitting zero-drop capacity. If a fitting capacity drops,
|
| 65 |
+
smaller values cannot help and the model is blocked.
|
| 66 |
+
UltraEP: measure required_capacity at C=1.25, round it UP to the next
|
| 67 |
+
0.01, then validate that capacity once.
|
| 68 |
+
MoonEP: natural + forced, no capacity factor.
|
| 69 |
+
|
| 70 |
+
The forced arm runs once at the selected/high-headroom capacity. It validates
|
| 71 |
+
instrumentation and is not part of capacity selection.
|
| 72 |
+
|
| 73 |
+
Each backend instruments a different thing:
|
| 74 |
+
|
| 75 |
+
echo local raw-router histograms and local post-reroute placement.
|
| 76 |
+
reduce_probe.py aligns and sums all eight source ranks before
|
| 77 |
+
calculating raw_expert_*, pre_home_rank_*,
|
| 78 |
+
post_physical_rank_* and valid_max_rank_reduction_*.
|
| 79 |
+
|
| 80 |
+
ultraep global pre/post rank vectors plus explicit drop accounting:
|
| 81 |
+
drop_rate_pct and required_capacity -- the capacity factor the run
|
| 82 |
+
WOULD have needed. The vectors are duplicated identically on all
|
| 83 |
+
logging ranks and are deduplicated, not summed.
|
| 84 |
+
|
| 85 |
+
moonep local raw-router histograms, actual payload rows per physical
|
| 86 |
+
destination, and the bytes-copy side. The reducer globally sums
|
| 87 |
+
each (iteration, layer) snapshot first. ECHO reports no comparable
|
| 88 |
+
byte counters, so MoonEP bytes are a self-comparison across models,
|
| 89 |
+
never a cross-backend ranking.
|
| 90 |
+
|
| 91 |
+
Do NOT quote ECHO's logged pre_echo imbalance_ratio or
|
| 92 |
+
max_load_reduction_pct. The branch mixes source-rank and destination-rank
|
| 93 |
+
axes; the reducer marks both invalid and reconstructs the comparison globally.
|
| 94 |
+
|
| 95 |
+
Two boundary cases:
|
| 96 |
+
raw_expert_imax_max == E/K
|
| 97 |
+
-> at least one global snapshot had one expert in every
|
| 98 |
+
token's top-K set. Inspect the MEAN before generalising.
|
| 99 |
+
required_capacity or min-safe-capacity at the configured capacity
|
| 100 |
+
-> clipped by the buffer; it means "at or over", not
|
| 101 |
+
"fits comfortably".
|
| 102 |
+
|
| 103 |
+
tools/reduce_probe.py has already written probe_summary.tsv and .json here.
|
| 104 |
+
|
| 105 |
+
Campaign results: /workspace/tiera-g1-reprobe/echo/tierbprobe
|
| 106 |
+
Roll up: python3 /workspace/blackwell_run_scripts/granularity/tools/collect.py /workspace/tiera-g1-reprobe/echo/tierbprobe --compare
|
| 107 |
+
|
| 108 |
+
=== STEP ultraep__tierbprobe__gbs16
|
| 109 |
+
[05:33:41] campaign: granularity-echo -> granularity-ultraep
|
| 110 |
+
[05:33:41] megatron: /workspace/Megatron-UltraEP already on ultraep-token-drop-analysis (eaf829e7)
|
| 111 |
+
==============================================================
|
| 112 |
+
CAMPAIGN: ultraep backend=ultraep phase=tierbprobe
|
| 113 |
+
Megatron: /workspace/Megatron-UltraEP @ ultraep-token-drop-analysis
|
| 114 |
+
GBS=16 seq=4096 repeats=3 nsys=1
|
| 115 |
+
results -> /workspace/tiera-g1-reprobe/ultraep/tierbprobe
|
| 116 |
+
==============================================================
|
| 117 |
+
--------------------------------------------------------------------------
|
| 118 |
+
[05:33:57] CELL G1__ultraep__matched__seq4096__gbs16__cap3p00 (slots=1)
|
| 119 |
+
[guard] iter 7/12 (60s)
|
| 120 |
+
[guard] iter 12/12 (70s)
|
| 121 |
+
[guard] tearing down process group (outcome=DONE)
|
| 122 |
+
[05:35:11] G1 ultraep: validating C=2.40625 (safety=+5%)
|
| 123 |
+
--------------------------------------------------------------------------
|
| 124 |
+
[05:35:11] CELL G1__ultraep__matched__seq4096__gbs16__cap2p40625 (slots=1)
|
| 125 |
+
[guard] iter 4/12 (60s)
|
| 126 |
+
[guard] iter 12/12 (70s)
|
| 127 |
+
[guard] tearing down process group (outcome=DONE)
|
| 128 |
+
--------------------------------------------------------------------------
|
| 129 |
+
[05:36:26] CELL G1__ultraep__matched__seq4096__gbs16__cap3p00__forcebal (slots=1)
|
| 130 |
+
[guard] iter 9/12 (60s)
|
| 131 |
+
[guard] iter 12/12 (70s)
|
| 132 |
+
[guard] tearing down process group (outcome=DONE)
|
| 133 |
+
--------------------------------------------------------------------------
|
| 134 |
+
|
| 135 |
+
==============================================================
|
| 136 |
+
SWEEP COMPLETE ok=3 failed=0
|
| 137 |
+
==============================================================
|
| 138 |
+
config backend policy slots seq gbs cap recompute rep status iter_ms_mean iter_ms_min peak_mib run_dir
|
| 139 |
+
G1 ultraep matched 1 4096 16 3.00 none 1 OK 980.37 914.80 125603 /workspace/tiera-g1-reprobe/ultraep/tierbprobe/tierbprobe_gbs16/G1__ultraep__matched__seq4096__gbs16__cap3p00/rep1
|
| 140 |
+
G1 ultraep matched 1 4096 16 2.40625 none 1 OK 925.04 908.00 119929 /workspace/tiera-g1-reprobe/ultraep/tierbprobe/tierbprobe_gbs16/G1__ultraep__matched__seq4096__gbs16__cap2p40625/rep1
|
| 141 |
+
G1 ultraep matched 1 4096 16 3.00 none+forcebal 1 OK 742.86 661.90 125603 /workspace/tiera-g1-reprobe/ultraep/tierbprobe/tierbprobe_gbs16/G1__ultraep__matched__seq4096__gbs16__cap3p00__forcebal/rep1
|
| 142 |
+
|
| 143 |
+
Results: /workspace/tiera-g1-reprobe/ultraep/tierbprobe/tierbprobe_gbs16/results.tsv
|
| 144 |
+
Roll up: python3 /workspace/blackwell_run_scripts/granularity/tools/collect.py /workspace/tiera-g1-reprobe/ultraep/tierbprobe/tierbprobe_gbs16
|
| 145 |
+
==============================================================
|
| 146 |
+
3 probe runs -> /workspace/tiera-g1-reprobe/ultraep/tierbprobe/tierbprobe_gbs16/probe_summary.tsv
|
| 147 |
+
run arm cap rawExpI preRankI postRankI drop% reqCap
|
| 148 |
+
--------------------------------------------------------------------------------------------------------------------
|
| 149 |
+
G1__ultraep__matched__seq4096__gbs16__cap2p40625/rep natural 2.40625 3.6155 1.0675 0.0000 2.2798
|
| 150 |
+
G1__ultraep__matched__seq4096__gbs16__cap3p00/rep1 natural 3.00 3.6038 1.0757 0.0000 2.2882
|
| 151 |
+
G1__ultraep__matched__seq4096__gbs16__cap3p00__force forced 3.00 1.0143 1.0096 0.0000 1.0100
|
| 152 |
+
|
| 153 |
+
ECHO note: echo_logged_pre_rank_metric_valid=0 and echo_logged_load_reduction_valid=0 are intentional.
|
| 154 |
+
The branch's built-in fields mix source-rank and destination-rank axes; this tool reconstructs both globally.
|
| 155 |
+
|
| 156 |
+
TIER B PROBE -- what these logs contain, per backend
|
| 157 |
+
----------------------------------------------------
|
| 158 |
+
Capacity matrix:
|
| 159 |
+
ECHO: start C=2.00; on OOM descend 1.75, 1.50, 1.25. Stop at
|
| 160 |
+
the highest fitting zero-drop capacity. If a fitting capacity drops,
|
| 161 |
+
smaller values cannot help and the model is blocked.
|
| 162 |
+
UltraEP: measure required_capacity at C=1.25, round it UP to the next
|
| 163 |
+
0.01, then validate that capacity once.
|
| 164 |
+
MoonEP: natural + forced, no capacity factor.
|
| 165 |
+
|
| 166 |
+
The forced arm runs once at the selected/high-headroom capacity. It validates
|
| 167 |
+
instrumentation and is not part of capacity selection.
|
| 168 |
+
|
| 169 |
+
Each backend instruments a different thing:
|
| 170 |
+
|
| 171 |
+
echo local raw-router histograms and local post-reroute placement.
|
| 172 |
+
reduce_probe.py aligns and sums all eight source ranks before
|
| 173 |
+
calculating raw_expert_*, pre_home_rank_*,
|
| 174 |
+
post_physical_rank_* and valid_max_rank_reduction_*.
|
| 175 |
+
|
| 176 |
+
ultraep global pre/post rank vectors plus explicit drop accounting:
|
| 177 |
+
drop_rate_pct and required_capacity -- the capacity factor the run
|
| 178 |
+
WOULD have needed. The vectors are duplicated identically on all
|
| 179 |
+
logging ranks and are deduplicated, not summed.
|
| 180 |
+
|
| 181 |
+
moonep local raw-router histograms, actual payload rows per physical
|
| 182 |
+
destination, and the bytes-copy side. The reducer globally sums
|
| 183 |
+
each (iteration, layer) snapshot first. ECHO reports no comparable
|
| 184 |
+
byte counters, so MoonEP bytes are a self-comparison across models,
|
| 185 |
+
never a cross-backend ranking.
|
| 186 |
+
|
| 187 |
+
Do NOT quote ECHO's logged pre_echo imbalance_ratio or
|
| 188 |
+
max_load_reduction_pct. The branch mixes source-rank and destination-rank
|
| 189 |
+
axes; the reducer marks both invalid and reconstructs the comparison globally.
|
| 190 |
+
|
| 191 |
+
Two boundary cases:
|
| 192 |
+
raw_expert_imax_max == E/K
|
| 193 |
+
-> at least one global snapshot had one expert in every
|
| 194 |
+
token's top-K set. Inspect the MEAN before generalising.
|
| 195 |
+
required_capacity or min-safe-capacity at the configured capacity
|
| 196 |
+
-> clipped by the buffer; it means "at or over", not
|
| 197 |
+
"fits comfortably".
|
| 198 |
+
|
| 199 |
+
tools/reduce_probe.py has already written probe_summary.tsv and .json here.
|
| 200 |
+
|
| 201 |
+
Campaign results: /workspace/tiera-g1-reprobe/ultraep/tierbprobe
|
| 202 |
+
Roll up: python3 /workspace/blackwell_run_scripts/granularity/tools/collect.py /workspace/tiera-g1-reprobe/ultraep/tierbprobe --compare
|
| 203 |
+
|
| 204 |
+
=== ROLL-UP
|
| 205 |
+
6 runs ok=6 oom=0 -> /workspace/tiera-g1-reprobe/collected_all.tsv
|
| 206 |
+
|
| 207 |
+
WARNING: 3 run(s) mention capacity saturation. A run that drops assignments does less expert work and must not be quoted as a win:
|
| 208 |
+
ultraep/tierbprobe/tierbprobe_gbs16/G1__ultraep__matched__seq4096__gbs16__cap2p40625/rep1: WARNING:megatron.training.drop_debug_utils:[MOE_DROP_DEBUG][iteration_summary] global_rank=2 iteration=0 records=32 total_expected_recv=242239 total_dropped_tok
|
| 209 |
+
ultraep/tierbprobe/tierbprobe_gbs16/G1__ultraep__matched__seq4096__gbs16__cap3p00/rep1: WARNING:megatron.training.drop_debug_utils:[MOE_DROP_DEBUG][iteration_summary] global_rank=6 iteration=0 records=32 total_expected_recv=262011 total_dropped_tok
|
| 210 |
+
ultraep/tierbprobe/tierbprobe_gbs16/G1__ultraep__matched__seq4096__gbs16__cap3p00__forcebal/rep1: WARNING:megatron.training.drop_debug_utils:[MOE_DROP_DEBUG][iteration_summary] global_rank=7 iteration=0 records=32 total_expected_recv=261946 total_dropped_tok
|
| 211 |
+
|
| 212 |
+
phase config seq gbs policy rc cg asy backend iter_ms vs_best
|
| 213 |
+
-------------------------------------------------------------------------------------
|
| 214 |
+
tierbprobeG1 4096 16 matched none 0 0 echo 684.81 1.00x
|
| 215 |
+
ultraep 882.76 1.29x
|
| 216 |
+
|
| 217 |
+
6 probe runs -> /workspace/tiera-g1-reprobe/probe_summary_all.tsv
|
| 218 |
+
run arm cap rawExpI preRankI postRankI drop% reqCap
|
| 219 |
+
--------------------------------------------------------------------------------------------------------------------
|
| 220 |
+
echo/tierbprobe/tierbprobe_gbs16/G1__echo__matched__ natural 1.859375 3.6222* 3.6222 1.0790 1.6406
|
| 221 |
+
echo/tierbprobe/tierbprobe_gbs16/G1__echo__matched__ forced 1.859375 1.0145 1.0145 1.0041 1.0156
|
| 222 |
+
echo/tierbprobe/tierbprobe_gbs16/G1__echo__matched__ natural 3.00 3.6267* 3.6267 1.0815 1.7656
|
| 223 |
+
ultraep/tierbprobe/tierbprobe_gbs16/G1__ultraep__mat natural 2.40625 3.6155 1.0675 0.0000 2.2798
|
| 224 |
+
ultraep/tierbprobe/tierbprobe_gbs16/G1__ultraep__mat natural 3.00 3.6038 1.0757 0.0000 2.2882
|
| 225 |
+
ultraep/tierbprobe/tierbprobe_gbs16/G1__ultraep__mat forced 3.00 1.0143 1.0096 0.0000 1.0100
|
| 226 |
+
|
| 227 |
+
* At least one globally aggregated snapshot reached the E/K expert-Imax ceiling.
|
| 228 |
+
Inspect raw_expert_imax_mean and hottest_expert_token_fraction_mean before generalising.
|
| 229 |
+
|
| 230 |
+
ECHO note: echo_logged_pre_rank_metric_valid=0 and echo_logged_load_reduction_valid=0 are intentional.
|
| 231 |
+
The branch's built-in fields mix source-rank and destination-rank axes; this tool reconstructs both globally.
|
| 232 |
+
no memory-dump runs found under /workspace/tiera-g1-reprobe
|
| 233 |
+
|
| 234 |
+
=== STUDY COMPLETE
|
| 235 |
+
results : /workspace/tiera-g1-reprobe
|
| 236 |
+
status : /workspace/tiera-g1-reprobe/STATUS.txt
|
| 237 |
+
steps : /workspace/tiera-g1-reprobe/steps.tsv
|
| 238 |
+
|
| 239 |
+
status campaign phase gbs started ended result_dir
|
| 240 |
+
OK echo tierbprobe 16 2026-08-30T05:29:23+00:00 2026-08-30T05:33:41+00:00 /workspace/tiera-g1-reprobe/echo/tierbprobe
|
| 241 |
+
OK ultraep tierbprobe 16 2026-08-30T05:33:41+00:00 2026-08-30T05:37:39+00:00 /workspace/tiera-g1-reprobe/ultraep/tierbprobe
|
driver-logs-20260830/tiera_mem.log
ADDED
|
The diff for this file is too large to render.
See raw diff
|
|
|
driver-logs-20260830/tierbfix4.log
ADDED
|
@@ -0,0 +1,242 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
stage 9: ECHO/UltraEP capacities taken from /workspace/prev_campaign/tierb_study_20260826_144644/tierb_timing_capacities.env
|
| 2 |
+
study_root: /workspace/tierb-fixed-moonep-full
|
| 3 |
+
gbs: 8
|
| 4 |
+
stages: 9
|
| 5 |
+
nsys: 1
|
| 6 |
+
hf_repo: MoE-CUDA-Graph-Analysis/tierb-b200-gbs8
|
| 7 |
+
hf_remote_root: tierb-fixed-moonep-full
|
| 8 |
+
started: 2026-08-30T02:38:05+00:00
|
| 9 |
+
driver_repo: /workspace/blackwell_run_scripts @ e8bd2bd
|
| 10 |
+
[02:38:05] starting HF watcher: every 600s -> MoE-CUDA-Graph-Analysis/tierb-b200-gbs8
|
| 11 |
+
[02:38:05] watcher pid 415916 (log: /workspace/tierb-fixed-moonep-full/hf_watcher.log)
|
| 12 |
+
|
| 13 |
+
=== STAGE 9 -- Tier B nsys traces (final, trace-only)
|
| 14 |
+
[02:38:05] SKIP echo__tierbtrace__gbs8 (already completed: 2026-08-30T00:38:30+00:00)
|
| 15 |
+
[02:38:05] SKIP ultraep__tierbtrace__gbs8 (already completed: 2026-08-30T00:59:28+00:00)
|
| 16 |
+
|
| 17 |
+
=== ROLL-UP
|
| 18 |
+
36 runs ok=32 oom=0 -> /workspace/tierb-fixed-moonep-full/collected_all.tsv
|
| 19 |
+
|
| 20 |
+
(excluded 8 nsys-traced run(s) from the timing comparison)
|
| 21 |
+
|
| 22 |
+
phase config seq gbs policy rc cg asy backend iter_ms vs_best
|
| 23 |
+
-------------------------------------------------------------------------------------
|
| 24 |
+
tierb DSV2_LITE_L24 8192 8 matched none 1 1 moonep 593.43 1.00x
|
| 25 |
+
|
| 26 |
+
tierb QWEN2_57B 4096 8 matched none 1 1 moonep 180.12 1.00x
|
| 27 |
+
|
| 28 |
+
tierb QWEN3_235B 8192 8 matched none 1 1 moonep 317.80 1.00x
|
| 29 |
+
|
| 30 |
+
tierb QWEN3_30B 4096 8 matched none 1 1 moonep 355.16 1.00x
|
| 31 |
+
|
| 32 |
+
tierbmem DSV2_LITE_L24 8192 8 matched none 1 1 moonep 622.80 1.00x
|
| 33 |
+
|
| 34 |
+
tierbmem QWEN2_57B 4096 8 matched none 1 1 moonep 196.60 1.00x
|
| 35 |
+
|
| 36 |
+
tierbmem QWEN3_235B 8192 8 matched none 1 1 moonep 341.70 1.00x
|
| 37 |
+
|
| 38 |
+
tierbmem QWEN3_30B 4096 8 matched none 1 1 moonep 435.30 1.00x
|
| 39 |
+
|
| 40 |
+
tierbprobeDSV2_LITE_L24 8192 8 matched none 0 1 moonep 673.94 1.00x
|
| 41 |
+
|
| 42 |
+
tierbprobeQWEN2_57B 4096 8 matched none 0 1 moonep 238.91 1.00x
|
| 43 |
+
|
| 44 |
+
tierbprobeQWEN3_235B 8192 8 matched none 0 1 moonep 384.76 1.00x
|
| 45 |
+
|
| 46 |
+
tierbprobeQWEN3_30B 4096 8 matched none 0 1 moonep 824.23 1.00x
|
| 47 |
+
|
| 48 |
+
8 probe runs -> /workspace/tierb-fixed-moonep-full/probe_summary_all.tsv
|
| 49 |
+
run arm cap rawExpI preRankI postRankI drop% reqCap
|
| 50 |
+
--------------------------------------------------------------------------------------------------------------------
|
| 51 |
+
moonep/tierbprobe/tierbprobe_gbs8/DSV2_LITE_L24__moo natural none 7.5443* 2.1316 1.1058
|
| 52 |
+
moonep/tierbprobe/tierbprobe_gbs8/DSV2_LITE_L24__moo forced none 1.0286 1.0061 1.0026
|
| 53 |
+
moonep/tierbprobe/tierbprobe_gbs8/QWEN2_57B__moonep_ natural none 7.4132* 2.3019 1.0979
|
| 54 |
+
moonep/tierbprobe/tierbprobe_gbs8/QWEN2_57B__moonep_ forced none 1.0348 1.0073 1.0033
|
| 55 |
+
moonep/tierbprobe/tierbprobe_gbs8/QWEN3_235B__moonep natural none 15.9227* 2.5851 1.0712
|
| 56 |
+
moonep/tierbprobe/tierbprobe_gbs8/QWEN3_235B__moonep forced none 1.0386 1.0053 1.0025
|
| 57 |
+
moonep/tierbprobe/tierbprobe_gbs8/QWEN3_30B__moonep_ natural none 15.7078* 2.4672 1.0635
|
| 58 |
+
moonep/tierbprobe/tierbprobe_gbs8/QWEN3_30B__moonep_ forced none 1.0565 1.0077 1.0036
|
| 59 |
+
|
| 60 |
+
* At least one globally aggregated snapshot reached the E/K expert-Imax ceiling.
|
| 61 |
+
Inspect raw_expert_imax_mean and hottest_expert_token_fraction_mean before generalising.
|
| 62 |
+
|
| 63 |
+
ECHO note: echo_logged_pre_rank_metric_valid=0 and echo_logged_load_reduction_valid=0 are intentional.
|
| 64 |
+
The branch's built-in fields mix source-rank and destination-rank axes; this tool reconstructs both globally.
|
| 65 |
+
8 memory-dump runs -> /workspace/tierb-fixed-moonep-full/memory_summary_all.tsv
|
| 66 |
+
model backend cg allocation capacity status peak GiB dumps
|
| 67 |
+
----------------------------------------------------------------------------------
|
| 68 |
+
DSV2_LITE_L24 moonep 1 none none OK 130.2 48/48
|
| 69 |
+
DSV2_LITE_L24 moonep 0 none none OK 122.2 40/40
|
| 70 |
+
QWEN2_57B moonep 1 none none OK 94.5 48/48
|
| 71 |
+
QWEN2_57B moonep 0 none none OK 93.6 40/40
|
| 72 |
+
QWEN3_235B moonep 1 none none OK 117.5 48/48
|
| 73 |
+
QWEN3_235B moonep 0 none none OK 117.3 40/40
|
| 74 |
+
QWEN3_30B moonep 1 none none OK 114.0 48/48
|
| 75 |
+
QWEN3_30B moonep 0 none none OK 113.7 40/40
|
| 76 |
+
|
| 77 |
+
ECHO/UltraEP note: graph-on uses static capacity + device-initiated GEMM while graph-off uses dynamic allocation + host-derived sizes. Their difference is not labelled as pure graph cost.
|
| 78 |
+
[02:38:09] step push: tierb-fixed-moonep-full -> MoE-CUDA-Graph-Analysis/tierb-b200-gbs8
|
| 79 |
+
Already logged in to Hugging Face as: AbhishekGhosh1998
|
| 80 |
+
uploaded: /workspace/tierb-fixed-moonep-full -> MoE-CUDA-Graph-Analysis/tierb-b200-gbs8/tierb-fixed-moonep-full
|
| 81 |
+
https://huggingface.co/MoE-CUDA-Graph-Analysis/tierb-b200-gbs8/tree/main
|
| 82 |
+
|
| 83 |
+
=== STUDY COMPLETE
|
| 84 |
+
results : /workspace/tierb-fixed-moonep-full
|
| 85 |
+
status : /workspace/tierb-fixed-moonep-full/STATUS.txt
|
| 86 |
+
steps : /workspace/tierb-fixed-moonep-full/steps.tsv
|
| 87 |
+
hub : https://huggingface.co/datasets/MoE-CUDA-Graph-Analysis/tierb-b200-gbs8
|
| 88 |
+
|
| 89 |
+
status campaign phase gbs started ended result_dir
|
| 90 |
+
OK moonep tierbprobe 8 2026-08-29T23:33:50+00:00 2026-08-29T23:44:30+00:00 /workspace/tierb-fixed-moonep-full/moonep/tierbprobe
|
| 91 |
+
OK moonep tierb 8 2026-08-29T23:44:39+00:00 2026-08-29T23:59:52+00:00 /workspace/tierb-fixed-moonep-full/moonep/tierb
|
| 92 |
+
OK moonep tierbmem 8 2026-08-29T23:59:59+00:00 2026-08-30T00:09:58+00:00 /workspace/tierb-fixed-moonep-full/moonep/tierbmem
|
| 93 |
+
OK echo tierbtrace 8 2026-08-30T00:10:16+00:00 2026-08-30T00:38:30+00:00 /workspace/tierb-fixed-moonep-full/echo/tierbtrace
|
| 94 |
+
OK ultraep tierbtrace 8 2026-08-30T00:38:33+00:00 2026-08-30T00:59:28+00:00 /workspace/tierb-fixed-moonep-full/ultraep/tierbtrace
|
| 95 |
+
SUPERSEDED moonep tierbtrace 8 2026-08-30T00:59:40+00:00 2026-08-30T01:06:12+00:00 /workspace/tierb-fixed-moonep-full/moonep/tierbtrace
|
| 96 |
+
OK moonep tierb 8 2026-08-30T01:30:54+00:00 2026-08-30T01:46:01+00:00 /workspace/tierb-fixed-moonep-full/moonep/tierb
|
| 97 |
+
study_root: /workspace/tierb-fixed-moonep-full
|
| 98 |
+
gbs: 8
|
| 99 |
+
stages: 9
|
| 100 |
+
nsys: 1
|
| 101 |
+
hf_repo: MoE-CUDA-Graph-Analysis/tierb-b200-gbs8
|
| 102 |
+
hf_remote_root: tierb-fixed-moonep-full
|
| 103 |
+
started: 2026-08-30T02:38:13+00:00
|
| 104 |
+
driver_repo: /workspace/blackwell_run_scripts @ e8bd2bd
|
| 105 |
+
[02:38:13] starting HF watcher: every 600s -> MoE-CUDA-Graph-Analysis/tierb-b200-gbs8
|
| 106 |
+
[02:38:13] watcher pid 416688 (log: /workspace/tierb-fixed-moonep-full/hf_watcher.log)
|
| 107 |
+
|
| 108 |
+
=== STAGE 9 -- Tier B nsys traces (final, trace-only)
|
| 109 |
+
|
| 110 |
+
=== STEP moonep__tierbtrace__gbs8
|
| 111 |
+
[02:38:13] campaign: granularity-echo -> granularity-moonep
|
| 112 |
+
[02:38:13] megatron: /workspace/Megatron-MoE-Echo already on moonep_backend_async_grad_reduce (2e68c8ad)
|
| 113 |
+
==============================================================
|
| 114 |
+
CAMPAIGN: moonep backend=moonep phase=tierbtrace
|
| 115 |
+
Megatron: /workspace/Megatron-MoE-Echo @ moonep_backend_async_grad_reduce
|
| 116 |
+
GBS=8 seq=4096 repeats=3 nsys=1
|
| 117 |
+
results -> /workspace/tierb-fixed-moonep-full/moonep/tierbtrace
|
| 118 |
+
==============================================================
|
| 119 |
+
[02:38:28] building moonep marker extension (log: /workspace/tierb-fixed-moonep-full/moonep/tierbtrace/tierbtrace_gbs8/trace_extension_build.log)
|
| 120 |
+
[02:38:46] marker extension ready: _moonep_trace_C.cpython-312-x86_64-linux-gnu.so
|
| 121 |
+
[02:38:46] TRACE-ONLY QWEN3_235B__moonep__matched__seq8192__gbs8__capnone__async (ranks: 0 1 2 3 4 5 6 7)
|
| 122 |
+
[guard] iter 6/10 (60s)
|
| 123 |
+
[guard] iter 8/10 (70s)
|
| 124 |
+
[guard] iter 10/10 (110s)
|
| 125 |
+
[guard] nsys stop (flushing report)
|
| 126 |
+
[02:41:08] trace verified -> QWEN3_235B__moonep__matched__seq8192__gbs8__capnone__async.nsys-rep (7.3M, all 8 GPUs + moonep markers)
|
| 127 |
+
[02:41:09] TRACE-ONLY QWEN3_30B__moonep__matched__seq4096__gbs8__capnone__async (ranks: 0 1 2 3 4 5 6 7)
|
| 128 |
+
[guard] iter 1/10 (60s)
|
| 129 |
+
[guard] iter 8/10 (70s)
|
| 130 |
+
[02:45:10] WARN: nsys run failed for QWEN3_30B__moonep__matched__seq4096__gbs8__capnone__async (checking report anyway)
|
| 131 |
+
[02:45:12] trace verified -> QWEN3_30B__moonep__matched__seq4096__gbs8__capnone__async.nsys-rep (24M, all 8 GPUs + moonep markers)
|
| 132 |
+
[02:45:14] TRACE-ONLY DSV2_LITE_L24__moonep__matched__seq8192__gbs8__capnone__async (ranks: 0 1 2 3 4 5 6 7)
|
| 133 |
+
[guard] iter 3/10 (60s)
|
| 134 |
+
[guard] iter 8/10 (70s)
|
| 135 |
+
[guard] iter 10/10 (110s)
|
| 136 |
+
[guard] nsys stop (flushing report)
|
| 137 |
+
[02:47:36] trace verified -> DSV2_LITE_L24__moonep__matched__seq8192__gbs8__capnone__async.nsys-rep (13M, all 8 GPUs + moonep markers)
|
| 138 |
+
[02:47:37] TRACE-ONLY QWEN2_57B__moonep__matched__seq4096__gbs8__capnone__async (ranks: 0 1 2 3 4 5 6 7)
|
| 139 |
+
[guard] iter 8/10 (60s)
|
| 140 |
+
[guard] iter 10/10 (110s)
|
| 141 |
+
[guard] nsys stop (flushing report)
|
| 142 |
+
[02:49:59] trace verified -> QWEN2_57B__moonep__matched__seq4096__gbs8__capnone__async.nsys-rep (5.6M, all 8 GPUs + moonep markers)
|
| 143 |
+
--------------------------------------------------------------------------
|
| 144 |
+
|
| 145 |
+
==============================================================
|
| 146 |
+
SWEEP COMPLETE ok=4 failed=0
|
| 147 |
+
==============================================================
|
| 148 |
+
config backend policy slots seq gbs cap recompute rep status iter_ms_mean iter_ms_min peak_mib run_dir
|
| 149 |
+
|
| 150 |
+
Results: /workspace/tierb-fixed-moonep-full/moonep/tierbtrace/tierbtrace_gbs8/results.tsv
|
| 151 |
+
Roll up: python3 /workspace/blackwell_run_scripts/granularity/tools/collect.py /workspace/tierb-fixed-moonep-full/moonep/tierbtrace/tierbtrace_gbs8
|
| 152 |
+
==============================================================
|
| 153 |
+
|
| 154 |
+
Campaign results: /workspace/tierb-fixed-moonep-full/moonep/tierbtrace
|
| 155 |
+
Roll up: python3 /workspace/blackwell_run_scripts/granularity/tools/collect.py /workspace/tierb-fixed-moonep-full/moonep/tierbtrace --compare
|
| 156 |
+
[02:49:59] step push: tierb-fixed-moonep-full/moonep/tierbtrace -> MoE-CUDA-Graph-Analysis/tierb-b200-gbs8
|
| 157 |
+
Already logged in to Hugging Face as: AbhishekGhosh1998
|
| 158 |
+
uploaded: /workspace/tierb-fixed-moonep-full/moonep/tierbtrace -> MoE-CUDA-Graph-Analysis/tierb-b200-gbs8/tierb-fixed-moonep-full/moonep/tierbtrace
|
| 159 |
+
https://huggingface.co/MoE-CUDA-Graph-Analysis/tierb-b200-gbs8/tree/main
|
| 160 |
+
|
| 161 |
+
=== ROLL-UP
|
| 162 |
+
40 runs ok=36 oom=0 -> /workspace/tierb-fixed-moonep-full/collected_all.tsv
|
| 163 |
+
|
| 164 |
+
(excluded 12 nsys-traced run(s) from the timing comparison)
|
| 165 |
+
|
| 166 |
+
phase config seq gbs policy rc cg asy backend iter_ms vs_best
|
| 167 |
+
-------------------------------------------------------------------------------------
|
| 168 |
+
tierb DSV2_LITE_L24 8192 8 matched none 1 1 moonep 593.43 1.00x
|
| 169 |
+
|
| 170 |
+
tierb QWEN2_57B 4096 8 matched none 1 1 moonep 180.12 1.00x
|
| 171 |
+
|
| 172 |
+
tierb QWEN3_235B 8192 8 matched none 1 1 moonep 317.80 1.00x
|
| 173 |
+
|
| 174 |
+
tierb QWEN3_30B 4096 8 matched none 1 1 moonep 355.16 1.00x
|
| 175 |
+
|
| 176 |
+
tierbmem DSV2_LITE_L24 8192 8 matched none 1 1 moonep 622.80 1.00x
|
| 177 |
+
|
| 178 |
+
tierbmem QWEN2_57B 4096 8 matched none 1 1 moonep 196.60 1.00x
|
| 179 |
+
|
| 180 |
+
tierbmem QWEN3_235B 8192 8 matched none 1 1 moonep 341.70 1.00x
|
| 181 |
+
|
| 182 |
+
tierbmem QWEN3_30B 4096 8 matched none 1 1 moonep 435.30 1.00x
|
| 183 |
+
|
| 184 |
+
tierbprobeDSV2_LITE_L24 8192 8 matched none 0 1 moonep 673.94 1.00x
|
| 185 |
+
|
| 186 |
+
tierbprobeQWEN2_57B 4096 8 matched none 0 1 moonep 238.91 1.00x
|
| 187 |
+
|
| 188 |
+
tierbprobeQWEN3_235B 8192 8 matched none 0 1 moonep 384.76 1.00x
|
| 189 |
+
|
| 190 |
+
tierbprobeQWEN3_30B 4096 8 matched none 0 1 moonep 824.23 1.00x
|
| 191 |
+
|
| 192 |
+
8 probe runs -> /workspace/tierb-fixed-moonep-full/probe_summary_all.tsv
|
| 193 |
+
run arm cap rawExpI preRankI postRankI drop% reqCap
|
| 194 |
+
--------------------------------------------------------------------------------------------------------------------
|
| 195 |
+
moonep/tierbprobe/tierbprobe_gbs8/DSV2_LITE_L24__moo natural none 7.5443* 2.1316 1.1058
|
| 196 |
+
moonep/tierbprobe/tierbprobe_gbs8/DSV2_LITE_L24__moo forced none 1.0286 1.0061 1.0026
|
| 197 |
+
moonep/tierbprobe/tierbprobe_gbs8/QWEN2_57B__moonep_ natural none 7.4132* 2.3019 1.0979
|
| 198 |
+
moonep/tierbprobe/tierbprobe_gbs8/QWEN2_57B__moonep_ forced none 1.0348 1.0073 1.0033
|
| 199 |
+
moonep/tierbprobe/tierbprobe_gbs8/QWEN3_235B__moonep natural none 15.9227* 2.5851 1.0712
|
| 200 |
+
moonep/tierbprobe/tierbprobe_gbs8/QWEN3_235B__moonep forced none 1.0386 1.0053 1.0025
|
| 201 |
+
moonep/tierbprobe/tierbprobe_gbs8/QWEN3_30B__moonep_ natural none 15.7078* 2.4672 1.0635
|
| 202 |
+
moonep/tierbprobe/tierbprobe_gbs8/QWEN3_30B__moonep_ forced none 1.0565 1.0077 1.0036
|
| 203 |
+
|
| 204 |
+
* At least one globally aggregated snapshot reached the E/K expert-Imax ceiling.
|
| 205 |
+
Inspect raw_expert_imax_mean and hottest_expert_token_fraction_mean before generalising.
|
| 206 |
+
|
| 207 |
+
ECHO note: echo_logged_pre_rank_metric_valid=0 and echo_logged_load_reduction_valid=0 are intentional.
|
| 208 |
+
The branch's built-in fields mix source-rank and destination-rank axes; this tool reconstructs both globally.
|
| 209 |
+
8 memory-dump runs -> /workspace/tierb-fixed-moonep-full/memory_summary_all.tsv
|
| 210 |
+
model backend cg allocation capacity status peak GiB dumps
|
| 211 |
+
----------------------------------------------------------------------------------
|
| 212 |
+
DSV2_LITE_L24 moonep 1 none none OK 130.2 48/48
|
| 213 |
+
DSV2_LITE_L24 moonep 0 none none OK 122.2 40/40
|
| 214 |
+
QWEN2_57B moonep 1 none none OK 94.5 48/48
|
| 215 |
+
QWEN2_57B moonep 0 none none OK 93.6 40/40
|
| 216 |
+
QWEN3_235B moonep 1 none none OK 117.5 48/48
|
| 217 |
+
QWEN3_235B moonep 0 none none OK 117.3 40/40
|
| 218 |
+
QWEN3_30B moonep 1 none none OK 114.0 48/48
|
| 219 |
+
QWEN3_30B moonep 0 none none OK 113.7 40/40
|
| 220 |
+
|
| 221 |
+
ECHO/UltraEP note: graph-on uses static capacity + device-initiated GEMM while graph-off uses dynamic allocation + host-derived sizes. Their difference is not labelled as pure graph cost.
|
| 222 |
+
[02:50:06] step push: tierb-fixed-moonep-full -> MoE-CUDA-Graph-Analysis/tierb-b200-gbs8
|
| 223 |
+
Already logged in to Hugging Face as: AbhishekGhosh1998
|
| 224 |
+
uploaded: /workspace/tierb-fixed-moonep-full -> MoE-CUDA-Graph-Analysis/tierb-b200-gbs8/tierb-fixed-moonep-full
|
| 225 |
+
https://huggingface.co/MoE-CUDA-Graph-Analysis/tierb-b200-gbs8/tree/main
|
| 226 |
+
|
| 227 |
+
=== STUDY COMPLETE
|
| 228 |
+
results : /workspace/tierb-fixed-moonep-full
|
| 229 |
+
status : /workspace/tierb-fixed-moonep-full/STATUS.txt
|
| 230 |
+
steps : /workspace/tierb-fixed-moonep-full/steps.tsv
|
| 231 |
+
hub : https://huggingface.co/datasets/MoE-CUDA-Graph-Analysis/tierb-b200-gbs8
|
| 232 |
+
|
| 233 |
+
status campaign phase gbs started ended result_dir
|
| 234 |
+
OK moonep tierbprobe 8 2026-08-29T23:33:50+00:00 2026-08-29T23:44:30+00:00 /workspace/tierb-fixed-moonep-full/moonep/tierbprobe
|
| 235 |
+
OK moonep tierb 8 2026-08-29T23:44:39+00:00 2026-08-29T23:59:52+00:00 /workspace/tierb-fixed-moonep-full/moonep/tierb
|
| 236 |
+
OK moonep tierbmem 8 2026-08-29T23:59:59+00:00 2026-08-30T00:09:58+00:00 /workspace/tierb-fixed-moonep-full/moonep/tierbmem
|
| 237 |
+
OK echo tierbtrace 8 2026-08-30T00:10:16+00:00 2026-08-30T00:38:30+00:00 /workspace/tierb-fixed-moonep-full/echo/tierbtrace
|
| 238 |
+
OK ultraep tierbtrace 8 2026-08-30T00:38:33+00:00 2026-08-30T00:59:28+00:00 /workspace/tierb-fixed-moonep-full/ultraep/tierbtrace
|
| 239 |
+
SUPERSEDED moonep tierbtrace 8 2026-08-30T00:59:40+00:00 2026-08-30T01:06:12+00:00 /workspace/tierb-fixed-moonep-full/moonep/tierbtrace
|
| 240 |
+
OK moonep tierb 8 2026-08-30T01:30:54+00:00 2026-08-30T01:46:01+00:00 /workspace/tierb-fixed-moonep-full/moonep/tierb
|
| 241 |
+
OK moonep tierbtrace 8 2026-08-30T02:38:13+00:00 2026-08-30T02:49:59+00:00 /workspace/tierb-fixed-moonep-full/moonep/tierbtrace
|
| 242 |
+
PATCH RERUN COMPLETE: /workspace/tierb-fixed-moonep-full
|