AbhishekGhosh1998 commited on
Commit
a64c7f6
·
verified ·
1 Parent(s): 37654a5

Upload folder using huggingface_hub

Browse files
driver-logs-20260830/README.txt ADDED
@@ -0,0 +1,14 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ Driver-level logs for the 2026-08-29/30 session.
2
+
3
+ These are the stdout of the campaign drivers, which is a superset of the
4
+ study.log stored inside each study root: it also captures the driver's own
5
+ decisions (capacity derivation, per-rung fallback, marker checks) and the two
6
+ capacity gates that were run outside any study root.
7
+
8
+ tiera1.log Tier A stage 7 capacity probe (halted: G1 unresolved)
9
+ tiera_g1.log G1 re-probe at a raised ceiling
10
+ capgate.log G4/echo at the global capacity 2.40625 -> OOM
11
+ capgate_g1.log G1/echo at the global capacity 2.40625 -> OK
12
+ tiera3.log Tier A timing + memory + traces
13
+ tiera_mem.log Tier A memory re-run after the marker fix
14
+ tierbfix4.log Tier B stage 9 MoonEP traces
driver-logs-20260830/capgate.log ADDED
@@ -0,0 +1,25 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ==============================================================
2
+ CAMPAIGN: echo backend=echo phase=a2
3
+ Megatron: /workspace/Megatron-MoE-Echo @ bracing-regions-through-kernels
4
+ GBS=16 seq=4096 repeats=1 nsys=0
5
+ results -> /workspace/granularity_echo_20260830_054430
6
+ ==============================================================
7
+ --------------------------------------------------------------------------
8
+ [05:44:30] CELL G4__echo__matched__seq4096__gbs16__cap2p40625 (slots=16)
9
+ [guard] tearing down process group (outcome=OOM)
10
+ [05:45:42] WARN: G4__echo__matched__seq4096__gbs16__cap2p40625 rep1 -> OOM
11
+ [05:45:42] waiting for stale ranks to exit ...
12
+ --------------------------------------------------------------------------
13
+
14
+ ==============================================================
15
+ SWEEP COMPLETE ok=0 failed=1
16
+ ==============================================================
17
+ config backend policy slots seq gbs cap recompute rep status iter_ms_mean iter_ms_min peak_mib run_dir
18
+ G4 echo matched 16 4096 16 2.40625 none 1 OOM NA NA 182627 /workspace/granularity_echo_20260830_054430/a2_matched_gbs16/G4__echo__matched__seq4096__gbs16__cap2p40625/rep1
19
+
20
+ Results: /workspace/granularity_echo_20260830_054430/a2_matched_gbs16/results.tsv
21
+ Roll up: python3 /workspace/blackwell_run_scripts/granularity/tools/collect.py /workspace/granularity_echo_20260830_054430/a2_matched_gbs16
22
+ ==============================================================
23
+
24
+ Campaign results: /workspace/granularity_echo_20260830_054430
25
+ Roll up: python3 /workspace/blackwell_run_scripts/granularity/tools/collect.py /workspace/granularity_echo_20260830_054430 --compare
driver-logs-20260830/capgate_g1.log ADDED
@@ -0,0 +1,24 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ==============================================================
2
+ CAMPAIGN: echo backend=echo phase=a2
3
+ Megatron: /workspace/Megatron-MoE-Echo @ bracing-regions-through-kernels
4
+ GBS=16 seq=4096 repeats=1 nsys=0
5
+ results -> /workspace/granularity_echo_20260830_054608
6
+ ==============================================================
7
+ --------------------------------------------------------------------------
8
+ [05:46:09] CELL G1__echo__matched__seq4096__gbs16__cap2p40625 (slots=1)
9
+ [guard] iter 12/12 (70s)
10
+ [guard] tearing down process group (outcome=DONE)
11
+ --------------------------------------------------------------------------
12
+
13
+ ==============================================================
14
+ SWEEP COMPLETE ok=1 failed=0
15
+ ==============================================================
16
+ config backend policy slots seq gbs cap recompute rep status iter_ms_mean iter_ms_min peak_mib run_dir
17
+ G1 echo matched 1 4096 16 2.40625 none 1 OK 468.54 463.00 161863 /workspace/granularity_echo_20260830_054608/a2_matched_gbs16/G1__echo__matched__seq4096__gbs16__cap2p40625/rep1
18
+
19
+ Results: /workspace/granularity_echo_20260830_054608/a2_matched_gbs16/results.tsv
20
+ Roll up: python3 /workspace/blackwell_run_scripts/granularity/tools/collect.py /workspace/granularity_echo_20260830_054608/a2_matched_gbs16
21
+ ==============================================================
22
+
23
+ Campaign results: /workspace/granularity_echo_20260830_054608
24
+ Roll up: python3 /workspace/blackwell_run_scripts/granularity/tools/collect.py /workspace/granularity_echo_20260830_054608 --compare
driver-logs-20260830/tiera1.log ADDED
@@ -0,0 +1,397 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ [04:52:38] STAGE 7 -- Tier A capacity probe (echo, ultraep)
2
+ study_root: /workspace/tiera-rerun-fixed-moonep
3
+ gbs: 16
4
+ stages: 7
5
+ nsys: 1
6
+ hf_repo: MoE-CUDA-Graph-Analysis/tierb-b200-gbs8
7
+ hf_remote_root: tiera-rerun-fixed-moonep
8
+ started: 2026-08-30T04:52:38+00:00
9
+ driver_repo: /workspace/blackwell_run_scripts @ 774e9e2
10
+ [04:52:38] starting HF watcher: every 600s -> MoE-CUDA-Graph-Analysis/tierb-b200-gbs8
11
+ [04:52:38] watcher pid 439568 (log: /workspace/tiera-rerun-fixed-moonep/hf_watcher.log)
12
+
13
+ === STAGE 7 -- Tier B probe and zero-drop capacity selection
14
+
15
+ === STEP echo__tierbprobe__gbs16
16
+ [04:52:38] megatron: /workspace/Megatron-MoE-Echo moonep_backend_async_grad_reduce -> token-drop-analysis
17
+ [04:52:38] megatron: now on token-drop-analysis (b099fd84)
18
+ ==============================================================
19
+ CAMPAIGN: echo backend=echo phase=tierbprobe
20
+ Megatron: /workspace/Megatron-MoE-Echo @ token-drop-analysis
21
+ GBS=16 seq=4096 repeats=3 nsys=1
22
+ results -> /workspace/tiera-rerun-fixed-moonep/echo/tierbprobe
23
+ ==============================================================
24
+ --------------------------------------------------------------------------
25
+ [04:52:54] CELL G1__echo__matched__seq4096__gbs16__cap2p00 (slots=1)
26
+ [guard] iter 6/12 (70s)
27
+ [guard] iter 12/12 (80s)
28
+ --------------------------------------------------------------------------
29
+ [04:54:14] CELL G2__echo__matched__seq4096__gbs16__cap2p00 (slots=4)
30
+ [guard] iter 6/12 (70s)
31
+ [guard] iter 12/12 (80s)
32
+ [guard] tearing down process group (outcome=DONE)
33
+ [04:55:36] G2 echo: measured required=1.02734375 at C=2.00 -> validating C=1.08203125 (safety=+5%)
34
+ --------------------------------------------------------------------------
35
+ [04:55:36] CELL G2__echo__matched__seq4096__gbs16__cap1p08203125 (slots=4)
36
+ [guard] iter 5/12 (70s)
37
+ [guard] iter 12/12 (80s)
38
+ [guard] tearing down process group (outcome=DONE)
39
+ --------------------------------------------------------------------------
40
+ [04:56:58] CELL G2__echo__matched__seq4096__gbs16__cap1p08203125__forcebal (slots=4)
41
+ [guard] iter 9/12 (70s)
42
+ [guard] iter 12/12 (80s)
43
+ --------------------------------------------------------------------------
44
+ [04:58:18] CELL G3__echo__matched__seq4096__gbs16__cap2p00 (slots=8)
45
+ [guard] iter 5/12 (70s)
46
+ [guard] iter 12/12 (80s)
47
+ [guard] tearing down process group (outcome=DONE)
48
+ [04:59:41] G3 echo: measured required=1.01953125 at C=2.00 -> validating C=1.072265625 (safety=+5%)
49
+ --------------------------------------------------------------------------
50
+ [04:59:41] CELL G3__echo__matched__seq4096__gbs16__cap1p072265625 (slots=8)
51
+ [guard] iter 6/12 (70s)
52
+ [guard] iter 12/12 (80s)
53
+ [guard] tearing down process group (outcome=DONE)
54
+ --------------------------------------------------------------------------
55
+ [05:01:03] CELL G3__echo__matched__seq4096__gbs16__cap1p072265625__forcebal (slots=8)
56
+ [guard] iter 8/12 (70s)
57
+ [guard] iter 12/12 (80s)
58
+ --------------------------------------------------------------------------
59
+ [05:02:24] CELL G4__echo__matched__seq4096__gbs16__cap2p00 (slots=16)
60
+ [guard] iter 4/12 (70s)
61
+ [guard] iter 12/12 (80s)
62
+ [guard] tearing down process group (outcome=DONE)
63
+ [05:03:48] G4 echo: measured required=1.0166015625 at C=2.00 -> validating C=1.068359375 (safety=+5%)
64
+ --------------------------------------------------------------------------
65
+ [05:03:48] CELL G4__echo__matched__seq4096__gbs16__cap1p068359375 (slots=16)
66
+ [guard] iter 4/12 (70s)
67
+ [guard] iter 12/12 (80s)
68
+ [guard] tearing down process group (outcome=DONE)
69
+ --------------------------------------------------------------------------
70
+ [05:05:12] CELL G4__echo__matched__seq4096__gbs16__cap1p068359375__forcebal (slots=16)
71
+ [guard] iter 7/12 (70s)
72
+ [guard] iter 12/12 (80s)
73
+ [guard] tearing down process group (outcome=DONE)
74
+ --------------------------------------------------------------------------
75
+
76
+ ==============================================================
77
+ SWEEP COMPLETE ok=10 failed=0
78
+ ==============================================================
79
+ config backend policy slots seq gbs cap recompute rep status iter_ms_mean iter_ms_min peak_mib run_dir
80
+ G1 echo matched 1 4096 16 2.00 none 1 OK 671.53 666.90 125477 /workspace/tiera-rerun-fixed-moonep/echo/tierbprobe/tierbprobe_gbs16/G1__echo__matched__seq4096__gbs16__cap2p00/rep1
81
+ G2 echo matched 4 4096 16 2.00 none 1 OK 675.77 669.20 130289 /workspace/tiera-rerun-fixed-moonep/echo/tierbprobe/tierbprobe_gbs16/G2__echo__matched__seq4096__gbs16__cap2p00/rep1
82
+ G2 echo matched 4 4096 16 1.08203125 none 1 OK 660.43 642.00 119267 /workspace/tiera-rerun-fixed-moonep/echo/tierbprobe/tierbprobe_gbs16/G2__echo__matched__seq4096__gbs16__cap1p08203125/rep1
83
+ G2 echo matched 4 4096 16 1.08203125 none+forcebal 1 OK 628.07 616.10 119267 /workspace/tiera-rerun-fixed-moonep/echo/tierbprobe/tierbprobe_gbs16/G2__echo__matched__seq4096__gbs16__cap1p08203125__forcebal/rep1
84
+ G3 echo matched 8 4096 16 2.00 none 1 OK 721.76 713.60 136111 /workspace/tiera-rerun-fixed-moonep/echo/tierbprobe/tierbprobe_gbs16/G3__echo__matched__seq4096__gbs16__cap2p00/rep1
85
+ G3 echo matched 8 4096 16 1.072265625 none 1 OK 681.37 670.20 122853 /workspace/tiera-rerun-fixed-moonep/echo/tierbprobe/tierbprobe_gbs16/G3__echo__matched__seq4096__gbs16__cap1p072265625/rep1
86
+ G3 echo matched 8 4096 16 1.072265625 none+forcebal 1 OK 664.27 659.10 122899 /workspace/tiera-rerun-fixed-moonep/echo/tierbprobe/tierbprobe_gbs16/G3__echo__matched__seq4096__gbs16__cap1p072265625__forcebal/rep1
87
+ G4 echo matched 16 4096 16 2.00 none 1 OK 829.14 790.80 146037 /workspace/tiera-rerun-fixed-moonep/echo/tierbprobe/tierbprobe_gbs16/G4__echo__matched__seq4096__gbs16__cap2p00/rep1
88
+ G4 echo matched 16 4096 16 1.068359375 none 1 OK 775.73 738.90 129621 /workspace/tiera-rerun-fixed-moonep/echo/tierbprobe/tierbprobe_gbs16/G4__echo__matched__seq4096__gbs16__cap1p068359375/rep1
89
+ G4 echo matched 16 4096 16 1.068359375 none+forcebal 1 OK 768.04 733.40 129579 /workspace/tiera-rerun-fixed-moonep/echo/tierbprobe/tierbprobe_gbs16/G4__echo__matched__seq4096__gbs16__cap1p068359375__forcebal/rep1
90
+
91
+ Results: /workspace/tiera-rerun-fixed-moonep/echo/tierbprobe/tierbprobe_gbs16/results.tsv
92
+ Roll up: python3 /workspace/blackwell_run_scripts/granularity/tools/collect.py /workspace/tiera-rerun-fixed-moonep/echo/tierbprobe/tierbprobe_gbs16
93
+ ==============================================================
94
+ 10 probe runs -> /workspace/tiera-rerun-fixed-moonep/echo/tierbprobe/tierbprobe_gbs16/probe_summary.tsv
95
+ run arm cap rawExpI preRankI postRankI drop% reqCap
96
+ --------------------------------------------------------------------------------------------------------------------
97
+ G1__echo__matched__seq4096__gbs16__cap2p00/rep1 natural 2.00 3.6198* 3.6198 1.0827 1.9062
98
+ G2__echo__matched__seq4096__gbs16__cap1p08203125/rep natural 1.08203125 3.8720* 2.1703 1.0000 1.0273
99
+ G2__echo__matched__seq4096__gbs16__cap1p08203125__fo forced 1.08203125 1.0201 1.0069 1.0000 1.0234
100
+ G2__echo__matched__seq4096__gbs16__cap2p00/rep1 natural 2.00 3.8724* 2.1875 1.0000 1.0273
101
+ G3__echo__matched__seq4096__gbs16__cap1p072265625/re natural 1.072265625 3.9096* 1.7838 1.0000 1.0195
102
+ G3__echo__matched__seq4096__gbs16__cap1p072265625__f forced 1.072265625 1.0231 1.0048 1.0000 1.0176
103
+ G3__echo__matched__seq4096__gbs16__cap2p00/rep1 natural 2.00 3.9096* 1.7916 1.0000 1.0195
104
+ G4__echo__matched__seq4096__gbs16__cap1p068359375/re natural 1.068359375 3.9303* 1.5507 1.0000 1.0166
105
+ G4__echo__matched__seq4096__gbs16__cap1p068359375__f forced 1.068359375 1.0249 1.0034 1.0000 1.0137
106
+ G4__echo__matched__seq4096__gbs16__cap2p00/rep1 natural 2.00 3.9301* 1.5543 1.0000 1.0166
107
+
108
+ * At least one globally aggregated snapshot reached the E/K expert-Imax ceiling.
109
+ Inspect raw_expert_imax_mean and hottest_expert_token_fraction_mean before generalising.
110
+
111
+ ECHO note: echo_logged_pre_rank_metric_valid=0 and echo_logged_load_reduction_valid=0 are intentional.
112
+ The branch's built-in fields mix source-rank and destination-rank axes; this tool reconstructs both globally.
113
+
114
+ TIER B PROBE -- what these logs contain, per backend
115
+ ----------------------------------------------------
116
+ Capacity matrix:
117
+ ECHO: start C=2.00; on OOM descend 1.75, 1.50, 1.25. Stop at
118
+ the highest fitting zero-drop capacity. If a fitting capacity drops,
119
+ smaller values cannot help and the model is blocked.
120
+ UltraEP: measure required_capacity at C=1.25, round it UP to the next
121
+ 0.01, then validate that capacity once.
122
+ MoonEP: natural + forced, no capacity factor.
123
+
124
+ The forced arm runs once at the selected/high-headroom capacity. It validates
125
+ instrumentation and is not part of capacity selection.
126
+
127
+ Each backend instruments a different thing:
128
+
129
+ echo local raw-router histograms and local post-reroute placement.
130
+ reduce_probe.py aligns and sums all eight source ranks before
131
+ calculating raw_expert_*, pre_home_rank_*,
132
+ post_physical_rank_* and valid_max_rank_reduction_*.
133
+
134
+ ultraep global pre/post rank vectors plus explicit drop accounting:
135
+ drop_rate_pct and required_capacity -- the capacity factor the run
136
+ WOULD have needed. The vectors are duplicated identically on all
137
+ logging ranks and are deduplicated, not summed.
138
+
139
+ moonep local raw-router histograms, actual payload rows per physical
140
+ destination, and the bytes-copy side. The reducer globally sums
141
+ each (iteration, layer) snapshot first. ECHO reports no comparable
142
+ byte counters, so MoonEP bytes are a self-comparison across models,
143
+ never a cross-backend ranking.
144
+
145
+ Do NOT quote ECHO's logged pre_echo imbalance_ratio or
146
+ max_load_reduction_pct. The branch mixes source-rank and destination-rank
147
+ axes; the reducer marks both invalid and reconstructs the comparison globally.
148
+
149
+ Two boundary cases:
150
+ raw_expert_imax_max == E/K
151
+ -> at least one global snapshot had one expert in every
152
+ token's top-K set. Inspect the MEAN before generalising.
153
+ required_capacity or min-safe-capacity at the configured capacity
154
+ -> clipped by the buffer; it means "at or over", not
155
+ "fits comfortably".
156
+
157
+ tools/reduce_probe.py has already written probe_summary.tsv and .json here.
158
+
159
+ Campaign results: /workspace/tiera-rerun-fixed-moonep/echo/tierbprobe
160
+ Roll up: python3 /workspace/blackwell_run_scripts/granularity/tools/collect.py /workspace/tiera-rerun-fixed-moonep/echo/tierbprobe --compare
161
+ [05:06:38] WARN: step echo__tierbprobe__gbs16 failed (rc=1)
162
+ [05:06:38] step push: tiera-rerun-fixed-moonep/echo/tierbprobe -> MoE-CUDA-Graph-Analysis/tierb-b200-gbs8
163
+ Already logged in to Hugging Face as: AbhishekGhosh1998
164
+ uploaded: /workspace/tiera-rerun-fixed-moonep/echo/tierbprobe -> MoE-CUDA-Graph-Analysis/tierb-b200-gbs8/tiera-rerun-fixed-moonep/echo/tierbprobe
165
+ https://huggingface.co/MoE-CUDA-Graph-Analysis/tierb-b200-gbs8/tree/main
166
+
167
+ === STEP ultraep__tierbprobe__gbs16
168
+ [05:06:43] campaign: granularity-echo -> granularity-ultraep
169
+ [05:06:43] megatron: /workspace/Megatron-UltraEP ultraep-bracing-regions-through-kernels -> ultraep-token-drop-analysis
170
+ [05:06:44] megatron: now on ultraep-token-drop-analysis (eaf829e7)
171
+ ==============================================================
172
+ CAMPAIGN: ultraep backend=ultraep phase=tierbprobe
173
+ Megatron: /workspace/Megatron-UltraEP @ ultraep-token-drop-analysis
174
+ GBS=16 seq=4096 repeats=3 nsys=1
175
+ results -> /workspace/tiera-rerun-fixed-moonep/ultraep/tierbprobe
176
+ ==============================================================
177
+ --------------------------------------------------------------------------
178
+ [05:06:59] CELL G1__ultraep__matched__seq4096__gbs16__cap1p25 (slots=1)
179
+ [guard] iter 4/12 (60s)
180
+ [guard] iter 12/12 (70s)
181
+ [guard] tearing down process group (outcome=DONE)
182
+ [05:08:13] G1 ultraep: validating C=1.8125 (safety=+5%)
183
+ --------------------------------------------------------------------------
184
+ [05:08:13] CELL G1__ultraep__matched__seq4096__gbs16__cap1p8125 (slots=1)
185
+ [guard] iter 4/12 (60s)
186
+ [guard] iter 12/12 (70s)
187
+ [guard] tearing down process group (outcome=DONE)
188
+ [05:09:28] G1 ultraep: C=1.8125 dropped; increasing headroom
189
+ --------------------------------------------------------------------------
190
+ [05:09:28] CELL G2__ultraep__matched__seq4096__gbs16__cap1p25 (slots=4)
191
+ [guard] iter 4/12 (60s)
192
+ [guard] iter 12/12 (70s)
193
+ [guard] tearing down process group (outcome=DONE)
194
+ [05:10:42] G2 ultraep: validating C=1.0625 (safety=+5%)
195
+ --------------------------------------------------------------------------
196
+ [05:10:42] CELL G2__ultraep__matched__seq4096__gbs16__cap1p0625 (slots=4)
197
+ [guard] iter 5/12 (60s)
198
+ [guard] iter 12/12 (70s)
199
+ [guard] tearing down process group (outcome=DONE)
200
+ --------------------------------------------------------------------------
201
+ [05:11:57] CELL G2__ultraep__matched__seq4096__gbs16__cap1p25__forcebal (slots=4)
202
+ [guard] iter 10/12 (60s)
203
+ [guard] iter 12/12 (70s)
204
+ [guard] tearing down process group (outcome=DONE)
205
+ --------------------------------------------------------------------------
206
+ [05:13:09] CELL G3__ultraep__matched__seq4096__gbs16__cap1p25 (slots=8)
207
+ [guard] iter 3/12 (60s)
208
+ [guard] iter 12/12 (70s)
209
+ [guard] tearing down process group (outcome=DONE)
210
+ [05:14:24] G3 ultraep: validating C=1.0625 (safety=+5%)
211
+ --------------------------------------------------------------------------
212
+ [05:14:24] CELL G3__ultraep__matched__seq4096__gbs16__cap1p0625 (slots=8)
213
+ [guard] iter 3/12 (60s)
214
+ [guard] iter 12/12 (70s)
215
+ [guard] tearing down process group (outcome=DONE)
216
+ --------------------------------------------------------------------------
217
+ [05:15:39] CELL G3__ultraep__matched__seq4096__gbs16__cap1p25__forcebal (slots=8)
218
+ [guard] iter 7/12 (60s)
219
+ [guard] iter 12/12 (70s)
220
+ [guard] tearing down process group (outcome=DONE)
221
+ --------------------------------------------------------------------------
222
+ [05:16:53] CELL G4__ultraep__matched__seq4096__gbs16__cap1p25 (slots=16)
223
+ [guard] iter 4/12 (60s)
224
+ [guard] iter 12/12 (70s)
225
+ [guard] tearing down process group (outcome=DONE)
226
+ [05:18:08] G4 ultraep: validating C=1.095703125 (safety=+5%)
227
+ --------------------------------------------------------------------------
228
+ [05:18:08] CELL G4__ultraep__matched__seq4096__gbs16__cap1p095703125 (slots=16)
229
+ [guard] iter 3/12 (60s)
230
+ [guard] iter 12/12 (70s)
231
+ [guard] tearing down process group (outcome=DONE)
232
+ --------------------------------------------------------------------------
233
+ [05:19:24] CELL G4__ultraep__matched__seq4096__gbs16__cap1p25__forcebal (slots=16)
234
+ [guard] iter 6/12 (60s)
235
+ [guard] iter 12/12 (70s)
236
+ [guard] tearing down process group (outcome=DONE)
237
+ --------------------------------------------------------------------------
238
+
239
+ ==============================================================
240
+ SWEEP COMPLETE ok=11 failed=0
241
+ ==============================================================
242
+ config backend policy slots seq gbs cap recompute rep status iter_ms_mean iter_ms_min peak_mib run_dir
243
+ G1 ultraep matched 1 4096 16 1.25 none 1 OK 893.49 886.90 108731 /workspace/tiera-rerun-fixed-moonep/ultraep/tierbprobe/tierbprobe_gbs16/G1__ultraep__matched__seq4096__gbs16__cap1p25/rep1
244
+ G1 ultraep matched 1 4096 16 1.8125 none 1 OK 914.84 903.60 113767 /workspace/tiera-rerun-fixed-moonep/ultraep/tierbprobe/tierbprobe_gbs16/G1__ultraep__matched__seq4096__gbs16__cap1p8125/rep1
245
+ G2 ultraep matched 4 4096 16 1.25 none 1 OK 558.91 532.70 110343 /workspace/tiera-rerun-fixed-moonep/ultraep/tierbprobe/tierbprobe_gbs16/G2__ultraep__matched__seq4096__gbs16__cap1p25/rep1
246
+ G2 ultraep matched 4 4096 16 1.0625 none 1 OK 551.94 527.00 107987 /workspace/tiera-rerun-fixed-moonep/ultraep/tierbprobe/tierbprobe_gbs16/G2__ultraep__matched__seq4096__gbs16__cap1p0625/rep1
247
+ G2 ultraep matched 4 4096 16 1.25 none+forcebal 1 OK 426.26 422.10 110343 /workspace/tiera-rerun-fixed-moonep/ultraep/tierbprobe/tierbprobe_gbs16/G2__ultraep__matched__seq4096__gbs16__cap1p25__forcebal/rep1
248
+ G3 ultraep matched 8 4096 16 1.25 none 1 OK 536.43 529.00 114319 /workspace/tiera-rerun-fixed-moonep/ultraep/tierbprobe/tierbprobe_gbs16/G3__ultraep__matched__seq4096__gbs16__cap1p25/rep1
249
+ G3 ultraep matched 8 4096 16 1.0625 none 1 OK 528.17 522.80 111567 /workspace/tiera-rerun-fixed-moonep/ultraep/tierbprobe/tierbprobe_gbs16/G3__ultraep__matched__seq4096__gbs16__cap1p0625/rep1
250
+ G3 ultraep matched 8 4096 16 1.25 none+forcebal 1 OK 472.34 470.60 114267 /workspace/tiera-rerun-fixed-moonep/ultraep/tierbprobe/tierbprobe_gbs16/G3__ultraep__matched__seq4096__gbs16__cap1p25__forcebal/rep1
251
+ G4 ultraep matched 16 4096 16 1.25 none 1 OK 610.06 573.90 120487 /workspace/tiera-rerun-fixed-moonep/ultraep/tierbprobe/tierbprobe_gbs16/G4__ultraep__matched__seq4096__gbs16__cap1p25/rep1
252
+ G4 ultraep matched 16 4096 16 1.095703125 none 1 OK 594.23 563.90 118667 /workspace/tiera-rerun-fixed-moonep/ultraep/tierbprobe/tierbprobe_gbs16/G4__ultraep__matched__seq4096__gbs16__cap1p095703125/rep1
253
+ G4 ultraep matched 16 4096 16 1.25 none+forcebal 1 OK 596.29 545.60 120487 /workspace/tiera-rerun-fixed-moonep/ultraep/tierbprobe/tierbprobe_gbs16/G4__ultraep__matched__seq4096__gbs16__cap1p25__forcebal/rep1
254
+
255
+ Results: /workspace/tiera-rerun-fixed-moonep/ultraep/tierbprobe/tierbprobe_gbs16/results.tsv
256
+ Roll up: python3 /workspace/blackwell_run_scripts/granularity/tools/collect.py /workspace/tiera-rerun-fixed-moonep/ultraep/tierbprobe/tierbprobe_gbs16
257
+ ==============================================================
258
+ 11 probe runs -> /workspace/tiera-rerun-fixed-moonep/ultraep/tierbprobe/tierbprobe_gbs16/probe_summary.tsv
259
+ run arm cap rawExpI preRankI postRankI drop% reqCap
260
+ --------------------------------------------------------------------------------------------------------------------
261
+ G1__ultraep__matched__seq4096__gbs16__cap1p25/rep1 natural 1.25 3.6085 1.0640 2.8918 1.7246
262
+ G1__ultraep__matched__seq4096__gbs16__cap1p8125/rep1 natural 1.8125 3.6152 1.0747 1.5460 2.3091
263
+ G2__ultraep__matched__seq4096__gbs16__cap1p0625/rep1 natural 1.0625 2.2032 1.0100 0.0000 1.0101
264
+ G2__ultraep__matched__seq4096__gbs16__cap1p25/rep1 natural 1.25 2.2136 1.0100 0.0000 1.0101
265
+ G2__ultraep__matched__seq4096__gbs16__cap1p25__force forced 1.25 1.0069 1.0067 0.0000 1.0100
266
+ G3__ultraep__matched__seq4096__gbs16__cap1p0625/rep1 natural 1.0625 1.7842 1.0100 0.0000 1.0101
267
+ G3__ultraep__matched__seq4096__gbs16__cap1p25/rep1 natural 1.25 1.7752 1.0100 0.0000 1.0101
268
+ G3__ultraep__matched__seq4096__gbs16__cap1p25__force forced 1.25 1.0047 1.0047 0.0000 1.0100
269
+ G4__ultraep__matched__seq4096__gbs16__cap1p095703125 natural 1.095703125 1.5612 1.0100 0.0000 1.0429
270
+ G4__ultraep__matched__seq4096__gbs16__cap1p25/rep1 natural 1.25 1.5504 1.0100 0.0000 1.0429
271
+ G4__ultraep__matched__seq4096__gbs16__cap1p25__force forced 1.25 1.0033 1.0033 0.0000 1.0071
272
+
273
+ ECHO note: echo_logged_pre_rank_metric_valid=0 and echo_logged_load_reduction_valid=0 are intentional.
274
+ The branch's built-in fields mix source-rank and destination-rank axes; this tool reconstructs both globally.
275
+
276
+ TIER B PROBE -- what these logs contain, per backend
277
+ ----------------------------------------------------
278
+ Capacity matrix:
279
+ ECHO: start C=2.00; on OOM descend 1.75, 1.50, 1.25. Stop at
280
+ the highest fitting zero-drop capacity. If a fitting capacity drops,
281
+ smaller values cannot help and the model is blocked.
282
+ UltraEP: measure required_capacity at C=1.25, round it UP to the next
283
+ 0.01, then validate that capacity once.
284
+ MoonEP: natural + forced, no capacity factor.
285
+
286
+ The forced arm runs once at the selected/high-headroom capacity. It validates
287
+ instrumentation and is not part of capacity selection.
288
+
289
+ Each backend instruments a different thing:
290
+
291
+ echo local raw-router histograms and local post-reroute placement.
292
+ reduce_probe.py aligns and sums all eight source ranks before
293
+ calculating raw_expert_*, pre_home_rank_*,
294
+ post_physical_rank_* and valid_max_rank_reduction_*.
295
+
296
+ ultraep global pre/post rank vectors plus explicit drop accounting:
297
+ drop_rate_pct and required_capacity -- the capacity factor the run
298
+ WOULD have needed. The vectors are duplicated identically on all
299
+ logging ranks and are deduplicated, not summed.
300
+
301
+ moonep local raw-router histograms, actual payload rows per physical
302
+ destination, and the bytes-copy side. The reducer globally sums
303
+ each (iteration, layer) snapshot first. ECHO reports no comparable
304
+ byte counters, so MoonEP bytes are a self-comparison across models,
305
+ never a cross-backend ranking.
306
+
307
+ Do NOT quote ECHO's logged pre_echo imbalance_ratio or
308
+ max_load_reduction_pct. The branch mixes source-rank and destination-rank
309
+ axes; the reducer marks both invalid and reconstructs the comparison globally.
310
+
311
+ Two boundary cases:
312
+ raw_expert_imax_max == E/K
313
+ -> at least one global snapshot had one expert in every
314
+ token's top-K set. Inspect the MEAN before generalising.
315
+ required_capacity or min-safe-capacity at the configured capacity
316
+ -> clipped by the buffer; it means "at or over", not
317
+ "fits comfortably".
318
+
319
+ tools/reduce_probe.py has already written probe_summary.tsv and .json here.
320
+
321
+ Campaign results: /workspace/tiera-rerun-fixed-moonep/ultraep/tierbprobe
322
+ Roll up: python3 /workspace/blackwell_run_scripts/granularity/tools/collect.py /workspace/tiera-rerun-fixed-moonep/ultraep/tierbprobe --compare
323
+ [05:20:40] WARN: step ultraep__tierbprobe__gbs16 failed (rc=1)
324
+ [05:20:40] step push: tiera-rerun-fixed-moonep/ultraep/tierbprobe -> MoE-CUDA-Graph-Analysis/tierb-b200-gbs8
325
+ Already logged in to Hugging Face as: AbhishekGhosh1998
326
+ uploaded: /workspace/tiera-rerun-fixed-moonep/ultraep/tierbprobe -> MoE-CUDA-Graph-Analysis/tierb-b200-gbs8/tiera-rerun-fixed-moonep/ultraep/tierbprobe
327
+ https://huggingface.co/MoE-CUDA-Graph-Analysis/tierb-b200-gbs8/tree/main
328
+
329
+ === ROLL-UP
330
+ 21 runs ok=21 oom=0 -> /workspace/tiera-rerun-fixed-moonep/collected_all.tsv
331
+
332
+ WARNING: 11 run(s) mention capacity saturation. A run that drops assignments does less expert work and must not be quoted as a win:
333
+ ultraep/tierbprobe/tierbprobe_gbs16/G1__ultraep__matched__seq4096__gbs16__cap1p25/rep1: WARNING:megatron.training.drop_debug_utils:[MOE_DROP_DEBUG][iteration_summary] global_rank=0 iteration=0 records=32 total_expected_recv=283999 total_dropped_tok
334
+ ultraep/tierbprobe/tierbprobe_gbs16/G1__ultraep__matched__seq4096__gbs16__cap1p8125/rep1: WARNING:megatron.training.drop_debug_utils:[MOE_DROP_DEBUG][iteration_summary] global_rank=0 iteration=0 records=32 total_expected_recv=284540 total_dropped_tok
335
+ ultraep/tierbprobe/tierbprobe_gbs16/G2__ultraep__matched__seq4096__gbs16__cap1p0625/rep1: WARNING:megatron.training.drop_debug_utils:[MOE_DROP_DEBUG][iteration_summary] global_rank=0 iteration=0 records=32 total_expected_recv=1048800 total_dropped_to
336
+ ultraep/tierbprobe/tierbprobe_gbs16/G2__ultraep__matched__seq4096__gbs16__cap1p25/rep1: WARNING:megatron.training.drop_debug_utils:[MOE_DROP_DEBUG][iteration_summary] global_rank=0 iteration=0 records=32 total_expected_recv=1048800 total_dropped_to
337
+ ultraep/tierbprobe/tierbprobe_gbs16/G2__ultraep__matched__seq4096__gbs16__cap1p25__forcebal/rep1: WARNING:megatron.training.drop_debug_utils:[MOE_DROP_DEBUG][iteration_summary] global_rank=1 iteration=0 records=32 total_expected_recv=1047984 total_dropped_to
338
+
339
+ phase config seq gbs policy rc cg asy backend iter_ms vs_best
340
+ -------------------------------------------------------------------------------------
341
+ tierbprobeG1 4096 16 matched none 0 0 echo 671.53 1.00x
342
+ ultraep 904.16 1.35x
343
+
344
+ tierbprobeG2 4096 16 matched none 0 0 ultraep 512.37 1.00x
345
+ echo 654.76 1.28x
346
+
347
+ tierbprobeG3 4096 16 matched none 0 0 ultraep 512.31 1.00x
348
+ echo 689.13 1.35x
349
+
350
+ tierbprobeG4 4096 16 matched none 0 0 ultraep 600.19 1.00x
351
+ echo 790.97 1.32x
352
+
353
+ 21 probe runs -> /workspace/tiera-rerun-fixed-moonep/probe_summary_all.tsv
354
+ run arm cap rawExpI preRankI postRankI drop% reqCap
355
+ --------------------------------------------------------------------------------------------------------------------
356
+ echo/tierbprobe/tierbprobe_gbs16/G1__echo__matched__ natural 2.00 3.6198* 3.6198 1.0827 1.9062
357
+ echo/tierbprobe/tierbprobe_gbs16/G2__echo__matched__ natural 1.08203125 3.8720* 2.1703 1.0000 1.0273
358
+ echo/tierbprobe/tierbprobe_gbs16/G2__echo__matched__ forced 1.08203125 1.0201 1.0069 1.0000 1.0234
359
+ echo/tierbprobe/tierbprobe_gbs16/G2__echo__matched__ natural 2.00 3.8724* 2.1875 1.0000 1.0273
360
+ echo/tierbprobe/tierbprobe_gbs16/G3__echo__matched__ natural 1.072265625 3.9096* 1.7838 1.0000 1.0195
361
+ echo/tierbprobe/tierbprobe_gbs16/G3__echo__matched__ forced 1.072265625 1.0231 1.0048 1.0000 1.0176
362
+ echo/tierbprobe/tierbprobe_gbs16/G3__echo__matched__ natural 2.00 3.9096* 1.7916 1.0000 1.0195
363
+ echo/tierbprobe/tierbprobe_gbs16/G4__echo__matched__ natural 1.068359375 3.9303* 1.5507 1.0000 1.0166
364
+ echo/tierbprobe/tierbprobe_gbs16/G4__echo__matched__ forced 1.068359375 1.0249 1.0034 1.0000 1.0137
365
+ echo/tierbprobe/tierbprobe_gbs16/G4__echo__matched__ natural 2.00 3.9301* 1.5543 1.0000 1.0166
366
+ ultraep/tierbprobe/tierbprobe_gbs16/G1__ultraep__mat natural 1.25 3.6085 1.0640 2.8918 1.7246
367
+ ultraep/tierbprobe/tierbprobe_gbs16/G1__ultraep__mat natural 1.8125 3.6152 1.0747 1.5460 2.3091
368
+ ultraep/tierbprobe/tierbprobe_gbs16/G2__ultraep__mat natural 1.0625 2.2032 1.0100 0.0000 1.0101
369
+ ultraep/tierbprobe/tierbprobe_gbs16/G2__ultraep__mat natural 1.25 2.2136 1.0100 0.0000 1.0101
370
+ ultraep/tierbprobe/tierbprobe_gbs16/G2__ultraep__mat forced 1.25 1.0069 1.0067 0.0000 1.0100
371
+ ultraep/tierbprobe/tierbprobe_gbs16/G3__ultraep__mat natural 1.0625 1.7842 1.0100 0.0000 1.0101
372
+ ultraep/tierbprobe/tierbprobe_gbs16/G3__ultraep__mat natural 1.25 1.7752 1.0100 0.0000 1.0101
373
+ ultraep/tierbprobe/tierbprobe_gbs16/G3__ultraep__mat forced 1.25 1.0047 1.0047 0.0000 1.0100
374
+ ultraep/tierbprobe/tierbprobe_gbs16/G4__ultraep__mat natural 1.095703125 1.5612 1.0100 0.0000 1.0429
375
+ ultraep/tierbprobe/tierbprobe_gbs16/G4__ultraep__mat natural 1.25 1.5504 1.0100 0.0000 1.0429
376
+ ultraep/tierbprobe/tierbprobe_gbs16/G4__ultraep__mat forced 1.25 1.0033 1.0033 0.0000 1.0071
377
+
378
+ * At least one globally aggregated snapshot reached the E/K expert-Imax ceiling.
379
+ Inspect raw_expert_imax_mean and hottest_expert_token_fraction_mean before generalising.
380
+
381
+ ECHO note: echo_logged_pre_rank_metric_valid=0 and echo_logged_load_reduction_valid=0 are intentional.
382
+ The branch's built-in fields mix source-rank and destination-rank axes; this tool reconstructs both globally.
383
+ no memory-dump runs found under /workspace/tiera-rerun-fixed-moonep
384
+ [05:20:51] step push: tiera-rerun-fixed-moonep -> MoE-CUDA-Graph-Analysis/tierb-b200-gbs8
385
+ Already logged in to Hugging Face as: AbhishekGhosh1998
386
+ uploaded: /workspace/tiera-rerun-fixed-moonep -> MoE-CUDA-Graph-Analysis/tierb-b200-gbs8/tiera-rerun-fixed-moonep
387
+ https://huggingface.co/MoE-CUDA-Graph-Analysis/tierb-b200-gbs8/tree/main
388
+
389
+ === STUDY COMPLETE
390
+ results : /workspace/tiera-rerun-fixed-moonep
391
+ status : /workspace/tiera-rerun-fixed-moonep/STATUS.txt
392
+ steps : /workspace/tiera-rerun-fixed-moonep/steps.tsv
393
+ hub : https://huggingface.co/datasets/MoE-CUDA-Graph-Analysis/tierb-b200-gbs8
394
+
395
+ status campaign phase gbs started ended result_dir
396
+ FAIL echo tierbprobe 16 2026-08-30T04:52:38+00:00 2026-08-30T05:06:38+00:00 /workspace/tiera-rerun-fixed-moonep/echo/tierbprobe
397
+ FAIL ultraep tierbprobe 16 2026-08-30T05:06:44+00:00 2026-08-30T05:20:40+00:00 /workspace/tiera-rerun-fixed-moonep/ultraep/tierbprobe
driver-logs-20260830/tiera3.log ADDED
The diff for this file is too large to render. See raw diff
 
driver-logs-20260830/tiera_g1.log ADDED
@@ -0,0 +1,241 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ study_root: /workspace/tiera-g1-reprobe
2
+ gbs: 16
3
+ stages: 7
4
+ nsys: 1
5
+ hf_repo: none
6
+ hf_remote_root: tiera-g1-reprobe
7
+ started: 2026-08-30T05:29:23+00:00
8
+ driver_repo: /workspace/blackwell_run_scripts @ 818ab62
9
+
10
+ === STAGE 7 -- Tier B probe and zero-drop capacity selection
11
+
12
+ === STEP echo__tierbprobe__gbs16
13
+ [05:29:23] campaign: granularity-ultraep -> granularity-echo
14
+ [05:29:23] megatron: /workspace/Megatron-MoE-Echo already on token-drop-analysis (b099fd84)
15
+ ==============================================================
16
+ CAMPAIGN: echo backend=echo phase=tierbprobe
17
+ Megatron: /workspace/Megatron-MoE-Echo @ token-drop-analysis
18
+ GBS=16 seq=4096 repeats=3 nsys=1
19
+ results -> /workspace/tiera-g1-reprobe/echo/tierbprobe
20
+ ==============================================================
21
+ --------------------------------------------------------------------------
22
+ [05:29:38] CELL G1__echo__matched__seq4096__gbs16__cap3p00 (slots=1)
23
+ [guard] iter 8/12 (70s)
24
+ [guard] iter 12/12 (80s)
25
+ [05:30:59] G1 echo: measured required=1.765625 at C=3.00 -> validating C=1.859375 (safety=+5%)
26
+ --------------------------------------------------------------------------
27
+ [05:30:59] CELL G1__echo__matched__seq4096__gbs16__cap1p859375 (slots=1)
28
+ [guard] iter 8/12 (70s)
29
+ [guard] iter 12/12 (80s)
30
+ --------------------------------------------------------------------------
31
+ [05:32:20] CELL G1__echo__matched__seq4096__gbs16__cap1p859375__forcebal (slots=1)
32
+ [guard] iter 10/12 (70s)
33
+ [guard] iter 12/12 (80s)
34
+ --------------------------------------------------------------------------
35
+
36
+ ==============================================================
37
+ SWEEP COMPLETE ok=3 failed=0
38
+ ==============================================================
39
+ config backend policy slots seq gbs cap recompute rep status iter_ms_mean iter_ms_min peak_mib run_dir
40
+ G1 echo matched 1 4096 16 3.00 none 1 OK 696.00 690.60 135941 /workspace/tiera-g1-reprobe/echo/tierbprobe/tierbprobe_gbs16/G1__echo__matched__seq4096__gbs16__cap3p00/rep1
41
+ G1 echo matched 1 4096 16 1.859375 none 1 OK 678.00 673.80 124457 /workspace/tiera-g1-reprobe/echo/tierbprobe/tierbprobe_gbs16/G1__echo__matched__seq4096__gbs16__cap1p859375/rep1
42
+ G1 echo matched 1 4096 16 1.859375 none+forcebal 1 OK 680.44 624.70 124455 /workspace/tiera-g1-reprobe/echo/tierbprobe/tierbprobe_gbs16/G1__echo__matched__seq4096__gbs16__cap1p859375__forcebal/rep1
43
+
44
+ Results: /workspace/tiera-g1-reprobe/echo/tierbprobe/tierbprobe_gbs16/results.tsv
45
+ Roll up: python3 /workspace/blackwell_run_scripts/granularity/tools/collect.py /workspace/tiera-g1-reprobe/echo/tierbprobe/tierbprobe_gbs16
46
+ ==============================================================
47
+ 3 probe runs -> /workspace/tiera-g1-reprobe/echo/tierbprobe/tierbprobe_gbs16/probe_summary.tsv
48
+ run arm cap rawExpI preRankI postRankI drop% reqCap
49
+ --------------------------------------------------------------------------------------------------------------------
50
+ G1__echo__matched__seq4096__gbs16__cap1p859375/rep1 natural 1.859375 3.6222* 3.6222 1.0790 1.6406
51
+ G1__echo__matched__seq4096__gbs16__cap1p859375__forc forced 1.859375 1.0145 1.0145 1.0041 1.0156
52
+ G1__echo__matched__seq4096__gbs16__cap3p00/rep1 natural 3.00 3.6267* 3.6267 1.0815 1.7656
53
+
54
+ * At least one globally aggregated snapshot reached the E/K expert-Imax ceiling.
55
+ Inspect raw_expert_imax_mean and hottest_expert_token_fraction_mean before generalising.
56
+
57
+ ECHO note: echo_logged_pre_rank_metric_valid=0 and echo_logged_load_reduction_valid=0 are intentional.
58
+ The branch's built-in fields mix source-rank and destination-rank axes; this tool reconstructs both globally.
59
+
60
+ TIER B PROBE -- what these logs contain, per backend
61
+ ----------------------------------------------------
62
+ Capacity matrix:
63
+ ECHO: start C=2.00; on OOM descend 1.75, 1.50, 1.25. Stop at
64
+ the highest fitting zero-drop capacity. If a fitting capacity drops,
65
+ smaller values cannot help and the model is blocked.
66
+ UltraEP: measure required_capacity at C=1.25, round it UP to the next
67
+ 0.01, then validate that capacity once.
68
+ MoonEP: natural + forced, no capacity factor.
69
+
70
+ The forced arm runs once at the selected/high-headroom capacity. It validates
71
+ instrumentation and is not part of capacity selection.
72
+
73
+ Each backend instruments a different thing:
74
+
75
+ echo local raw-router histograms and local post-reroute placement.
76
+ reduce_probe.py aligns and sums all eight source ranks before
77
+ calculating raw_expert_*, pre_home_rank_*,
78
+ post_physical_rank_* and valid_max_rank_reduction_*.
79
+
80
+ ultraep global pre/post rank vectors plus explicit drop accounting:
81
+ drop_rate_pct and required_capacity -- the capacity factor the run
82
+ WOULD have needed. The vectors are duplicated identically on all
83
+ logging ranks and are deduplicated, not summed.
84
+
85
+ moonep local raw-router histograms, actual payload rows per physical
86
+ destination, and the bytes-copy side. The reducer globally sums
87
+ each (iteration, layer) snapshot first. ECHO reports no comparable
88
+ byte counters, so MoonEP bytes are a self-comparison across models,
89
+ never a cross-backend ranking.
90
+
91
+ Do NOT quote ECHO's logged pre_echo imbalance_ratio or
92
+ max_load_reduction_pct. The branch mixes source-rank and destination-rank
93
+ axes; the reducer marks both invalid and reconstructs the comparison globally.
94
+
95
+ Two boundary cases:
96
+ raw_expert_imax_max == E/K
97
+ -> at least one global snapshot had one expert in every
98
+ token's top-K set. Inspect the MEAN before generalising.
99
+ required_capacity or min-safe-capacity at the configured capacity
100
+ -> clipped by the buffer; it means "at or over", not
101
+ "fits comfortably".
102
+
103
+ tools/reduce_probe.py has already written probe_summary.tsv and .json here.
104
+
105
+ Campaign results: /workspace/tiera-g1-reprobe/echo/tierbprobe
106
+ Roll up: python3 /workspace/blackwell_run_scripts/granularity/tools/collect.py /workspace/tiera-g1-reprobe/echo/tierbprobe --compare
107
+
108
+ === STEP ultraep__tierbprobe__gbs16
109
+ [05:33:41] campaign: granularity-echo -> granularity-ultraep
110
+ [05:33:41] megatron: /workspace/Megatron-UltraEP already on ultraep-token-drop-analysis (eaf829e7)
111
+ ==============================================================
112
+ CAMPAIGN: ultraep backend=ultraep phase=tierbprobe
113
+ Megatron: /workspace/Megatron-UltraEP @ ultraep-token-drop-analysis
114
+ GBS=16 seq=4096 repeats=3 nsys=1
115
+ results -> /workspace/tiera-g1-reprobe/ultraep/tierbprobe
116
+ ==============================================================
117
+ --------------------------------------------------------------------------
118
+ [05:33:57] CELL G1__ultraep__matched__seq4096__gbs16__cap3p00 (slots=1)
119
+ [guard] iter 7/12 (60s)
120
+ [guard] iter 12/12 (70s)
121
+ [guard] tearing down process group (outcome=DONE)
122
+ [05:35:11] G1 ultraep: validating C=2.40625 (safety=+5%)
123
+ --------------------------------------------------------------------------
124
+ [05:35:11] CELL G1__ultraep__matched__seq4096__gbs16__cap2p40625 (slots=1)
125
+ [guard] iter 4/12 (60s)
126
+ [guard] iter 12/12 (70s)
127
+ [guard] tearing down process group (outcome=DONE)
128
+ --------------------------------------------------------------------------
129
+ [05:36:26] CELL G1__ultraep__matched__seq4096__gbs16__cap3p00__forcebal (slots=1)
130
+ [guard] iter 9/12 (60s)
131
+ [guard] iter 12/12 (70s)
132
+ [guard] tearing down process group (outcome=DONE)
133
+ --------------------------------------------------------------------------
134
+
135
+ ==============================================================
136
+ SWEEP COMPLETE ok=3 failed=0
137
+ ==============================================================
138
+ config backend policy slots seq gbs cap recompute rep status iter_ms_mean iter_ms_min peak_mib run_dir
139
+ G1 ultraep matched 1 4096 16 3.00 none 1 OK 980.37 914.80 125603 /workspace/tiera-g1-reprobe/ultraep/tierbprobe/tierbprobe_gbs16/G1__ultraep__matched__seq4096__gbs16__cap3p00/rep1
140
+ G1 ultraep matched 1 4096 16 2.40625 none 1 OK 925.04 908.00 119929 /workspace/tiera-g1-reprobe/ultraep/tierbprobe/tierbprobe_gbs16/G1__ultraep__matched__seq4096__gbs16__cap2p40625/rep1
141
+ G1 ultraep matched 1 4096 16 3.00 none+forcebal 1 OK 742.86 661.90 125603 /workspace/tiera-g1-reprobe/ultraep/tierbprobe/tierbprobe_gbs16/G1__ultraep__matched__seq4096__gbs16__cap3p00__forcebal/rep1
142
+
143
+ Results: /workspace/tiera-g1-reprobe/ultraep/tierbprobe/tierbprobe_gbs16/results.tsv
144
+ Roll up: python3 /workspace/blackwell_run_scripts/granularity/tools/collect.py /workspace/tiera-g1-reprobe/ultraep/tierbprobe/tierbprobe_gbs16
145
+ ==============================================================
146
+ 3 probe runs -> /workspace/tiera-g1-reprobe/ultraep/tierbprobe/tierbprobe_gbs16/probe_summary.tsv
147
+ run arm cap rawExpI preRankI postRankI drop% reqCap
148
+ --------------------------------------------------------------------------------------------------------------------
149
+ G1__ultraep__matched__seq4096__gbs16__cap2p40625/rep natural 2.40625 3.6155 1.0675 0.0000 2.2798
150
+ G1__ultraep__matched__seq4096__gbs16__cap3p00/rep1 natural 3.00 3.6038 1.0757 0.0000 2.2882
151
+ G1__ultraep__matched__seq4096__gbs16__cap3p00__force forced 3.00 1.0143 1.0096 0.0000 1.0100
152
+
153
+ ECHO note: echo_logged_pre_rank_metric_valid=0 and echo_logged_load_reduction_valid=0 are intentional.
154
+ The branch's built-in fields mix source-rank and destination-rank axes; this tool reconstructs both globally.
155
+
156
+ TIER B PROBE -- what these logs contain, per backend
157
+ ----------------------------------------------------
158
+ Capacity matrix:
159
+ ECHO: start C=2.00; on OOM descend 1.75, 1.50, 1.25. Stop at
160
+ the highest fitting zero-drop capacity. If a fitting capacity drops,
161
+ smaller values cannot help and the model is blocked.
162
+ UltraEP: measure required_capacity at C=1.25, round it UP to the next
163
+ 0.01, then validate that capacity once.
164
+ MoonEP: natural + forced, no capacity factor.
165
+
166
+ The forced arm runs once at the selected/high-headroom capacity. It validates
167
+ instrumentation and is not part of capacity selection.
168
+
169
+ Each backend instruments a different thing:
170
+
171
+ echo local raw-router histograms and local post-reroute placement.
172
+ reduce_probe.py aligns and sums all eight source ranks before
173
+ calculating raw_expert_*, pre_home_rank_*,
174
+ post_physical_rank_* and valid_max_rank_reduction_*.
175
+
176
+ ultraep global pre/post rank vectors plus explicit drop accounting:
177
+ drop_rate_pct and required_capacity -- the capacity factor the run
178
+ WOULD have needed. The vectors are duplicated identically on all
179
+ logging ranks and are deduplicated, not summed.
180
+
181
+ moonep local raw-router histograms, actual payload rows per physical
182
+ destination, and the bytes-copy side. The reducer globally sums
183
+ each (iteration, layer) snapshot first. ECHO reports no comparable
184
+ byte counters, so MoonEP bytes are a self-comparison across models,
185
+ never a cross-backend ranking.
186
+
187
+ Do NOT quote ECHO's logged pre_echo imbalance_ratio or
188
+ max_load_reduction_pct. The branch mixes source-rank and destination-rank
189
+ axes; the reducer marks both invalid and reconstructs the comparison globally.
190
+
191
+ Two boundary cases:
192
+ raw_expert_imax_max == E/K
193
+ -> at least one global snapshot had one expert in every
194
+ token's top-K set. Inspect the MEAN before generalising.
195
+ required_capacity or min-safe-capacity at the configured capacity
196
+ -> clipped by the buffer; it means "at or over", not
197
+ "fits comfortably".
198
+
199
+ tools/reduce_probe.py has already written probe_summary.tsv and .json here.
200
+
201
+ Campaign results: /workspace/tiera-g1-reprobe/ultraep/tierbprobe
202
+ Roll up: python3 /workspace/blackwell_run_scripts/granularity/tools/collect.py /workspace/tiera-g1-reprobe/ultraep/tierbprobe --compare
203
+
204
+ === ROLL-UP
205
+ 6 runs ok=6 oom=0 -> /workspace/tiera-g1-reprobe/collected_all.tsv
206
+
207
+ WARNING: 3 run(s) mention capacity saturation. A run that drops assignments does less expert work and must not be quoted as a win:
208
+ ultraep/tierbprobe/tierbprobe_gbs16/G1__ultraep__matched__seq4096__gbs16__cap2p40625/rep1: WARNING:megatron.training.drop_debug_utils:[MOE_DROP_DEBUG][iteration_summary] global_rank=2 iteration=0 records=32 total_expected_recv=242239 total_dropped_tok
209
+ ultraep/tierbprobe/tierbprobe_gbs16/G1__ultraep__matched__seq4096__gbs16__cap3p00/rep1: WARNING:megatron.training.drop_debug_utils:[MOE_DROP_DEBUG][iteration_summary] global_rank=6 iteration=0 records=32 total_expected_recv=262011 total_dropped_tok
210
+ ultraep/tierbprobe/tierbprobe_gbs16/G1__ultraep__matched__seq4096__gbs16__cap3p00__forcebal/rep1: WARNING:megatron.training.drop_debug_utils:[MOE_DROP_DEBUG][iteration_summary] global_rank=7 iteration=0 records=32 total_expected_recv=261946 total_dropped_tok
211
+
212
+ phase config seq gbs policy rc cg asy backend iter_ms vs_best
213
+ -------------------------------------------------------------------------------------
214
+ tierbprobeG1 4096 16 matched none 0 0 echo 684.81 1.00x
215
+ ultraep 882.76 1.29x
216
+
217
+ 6 probe runs -> /workspace/tiera-g1-reprobe/probe_summary_all.tsv
218
+ run arm cap rawExpI preRankI postRankI drop% reqCap
219
+ --------------------------------------------------------------------------------------------------------------------
220
+ echo/tierbprobe/tierbprobe_gbs16/G1__echo__matched__ natural 1.859375 3.6222* 3.6222 1.0790 1.6406
221
+ echo/tierbprobe/tierbprobe_gbs16/G1__echo__matched__ forced 1.859375 1.0145 1.0145 1.0041 1.0156
222
+ echo/tierbprobe/tierbprobe_gbs16/G1__echo__matched__ natural 3.00 3.6267* 3.6267 1.0815 1.7656
223
+ ultraep/tierbprobe/tierbprobe_gbs16/G1__ultraep__mat natural 2.40625 3.6155 1.0675 0.0000 2.2798
224
+ ultraep/tierbprobe/tierbprobe_gbs16/G1__ultraep__mat natural 3.00 3.6038 1.0757 0.0000 2.2882
225
+ ultraep/tierbprobe/tierbprobe_gbs16/G1__ultraep__mat forced 3.00 1.0143 1.0096 0.0000 1.0100
226
+
227
+ * At least one globally aggregated snapshot reached the E/K expert-Imax ceiling.
228
+ Inspect raw_expert_imax_mean and hottest_expert_token_fraction_mean before generalising.
229
+
230
+ ECHO note: echo_logged_pre_rank_metric_valid=0 and echo_logged_load_reduction_valid=0 are intentional.
231
+ The branch's built-in fields mix source-rank and destination-rank axes; this tool reconstructs both globally.
232
+ no memory-dump runs found under /workspace/tiera-g1-reprobe
233
+
234
+ === STUDY COMPLETE
235
+ results : /workspace/tiera-g1-reprobe
236
+ status : /workspace/tiera-g1-reprobe/STATUS.txt
237
+ steps : /workspace/tiera-g1-reprobe/steps.tsv
238
+
239
+ status campaign phase gbs started ended result_dir
240
+ OK echo tierbprobe 16 2026-08-30T05:29:23+00:00 2026-08-30T05:33:41+00:00 /workspace/tiera-g1-reprobe/echo/tierbprobe
241
+ OK ultraep tierbprobe 16 2026-08-30T05:33:41+00:00 2026-08-30T05:37:39+00:00 /workspace/tiera-g1-reprobe/ultraep/tierbprobe
driver-logs-20260830/tiera_mem.log ADDED
The diff for this file is too large to render. See raw diff
 
driver-logs-20260830/tierbfix4.log ADDED
@@ -0,0 +1,242 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ stage 9: ECHO/UltraEP capacities taken from /workspace/prev_campaign/tierb_study_20260826_144644/tierb_timing_capacities.env
2
+ study_root: /workspace/tierb-fixed-moonep-full
3
+ gbs: 8
4
+ stages: 9
5
+ nsys: 1
6
+ hf_repo: MoE-CUDA-Graph-Analysis/tierb-b200-gbs8
7
+ hf_remote_root: tierb-fixed-moonep-full
8
+ started: 2026-08-30T02:38:05+00:00
9
+ driver_repo: /workspace/blackwell_run_scripts @ e8bd2bd
10
+ [02:38:05] starting HF watcher: every 600s -> MoE-CUDA-Graph-Analysis/tierb-b200-gbs8
11
+ [02:38:05] watcher pid 415916 (log: /workspace/tierb-fixed-moonep-full/hf_watcher.log)
12
+
13
+ === STAGE 9 -- Tier B nsys traces (final, trace-only)
14
+ [02:38:05] SKIP echo__tierbtrace__gbs8 (already completed: 2026-08-30T00:38:30+00:00)
15
+ [02:38:05] SKIP ultraep__tierbtrace__gbs8 (already completed: 2026-08-30T00:59:28+00:00)
16
+
17
+ === ROLL-UP
18
+ 36 runs ok=32 oom=0 -> /workspace/tierb-fixed-moonep-full/collected_all.tsv
19
+
20
+ (excluded 8 nsys-traced run(s) from the timing comparison)
21
+
22
+ phase config seq gbs policy rc cg asy backend iter_ms vs_best
23
+ -------------------------------------------------------------------------------------
24
+ tierb DSV2_LITE_L24 8192 8 matched none 1 1 moonep 593.43 1.00x
25
+
26
+ tierb QWEN2_57B 4096 8 matched none 1 1 moonep 180.12 1.00x
27
+
28
+ tierb QWEN3_235B 8192 8 matched none 1 1 moonep 317.80 1.00x
29
+
30
+ tierb QWEN3_30B 4096 8 matched none 1 1 moonep 355.16 1.00x
31
+
32
+ tierbmem DSV2_LITE_L24 8192 8 matched none 1 1 moonep 622.80 1.00x
33
+
34
+ tierbmem QWEN2_57B 4096 8 matched none 1 1 moonep 196.60 1.00x
35
+
36
+ tierbmem QWEN3_235B 8192 8 matched none 1 1 moonep 341.70 1.00x
37
+
38
+ tierbmem QWEN3_30B 4096 8 matched none 1 1 moonep 435.30 1.00x
39
+
40
+ tierbprobeDSV2_LITE_L24 8192 8 matched none 0 1 moonep 673.94 1.00x
41
+
42
+ tierbprobeQWEN2_57B 4096 8 matched none 0 1 moonep 238.91 1.00x
43
+
44
+ tierbprobeQWEN3_235B 8192 8 matched none 0 1 moonep 384.76 1.00x
45
+
46
+ tierbprobeQWEN3_30B 4096 8 matched none 0 1 moonep 824.23 1.00x
47
+
48
+ 8 probe runs -> /workspace/tierb-fixed-moonep-full/probe_summary_all.tsv
49
+ run arm cap rawExpI preRankI postRankI drop% reqCap
50
+ --------------------------------------------------------------------------------------------------------------------
51
+ moonep/tierbprobe/tierbprobe_gbs8/DSV2_LITE_L24__moo natural none 7.5443* 2.1316 1.1058
52
+ moonep/tierbprobe/tierbprobe_gbs8/DSV2_LITE_L24__moo forced none 1.0286 1.0061 1.0026
53
+ moonep/tierbprobe/tierbprobe_gbs8/QWEN2_57B__moonep_ natural none 7.4132* 2.3019 1.0979
54
+ moonep/tierbprobe/tierbprobe_gbs8/QWEN2_57B__moonep_ forced none 1.0348 1.0073 1.0033
55
+ moonep/tierbprobe/tierbprobe_gbs8/QWEN3_235B__moonep natural none 15.9227* 2.5851 1.0712
56
+ moonep/tierbprobe/tierbprobe_gbs8/QWEN3_235B__moonep forced none 1.0386 1.0053 1.0025
57
+ moonep/tierbprobe/tierbprobe_gbs8/QWEN3_30B__moonep_ natural none 15.7078* 2.4672 1.0635
58
+ moonep/tierbprobe/tierbprobe_gbs8/QWEN3_30B__moonep_ forced none 1.0565 1.0077 1.0036
59
+
60
+ * At least one globally aggregated snapshot reached the E/K expert-Imax ceiling.
61
+ Inspect raw_expert_imax_mean and hottest_expert_token_fraction_mean before generalising.
62
+
63
+ ECHO note: echo_logged_pre_rank_metric_valid=0 and echo_logged_load_reduction_valid=0 are intentional.
64
+ The branch's built-in fields mix source-rank and destination-rank axes; this tool reconstructs both globally.
65
+ 8 memory-dump runs -> /workspace/tierb-fixed-moonep-full/memory_summary_all.tsv
66
+ model backend cg allocation capacity status peak GiB dumps
67
+ ----------------------------------------------------------------------------------
68
+ DSV2_LITE_L24 moonep 1 none none OK 130.2 48/48
69
+ DSV2_LITE_L24 moonep 0 none none OK 122.2 40/40
70
+ QWEN2_57B moonep 1 none none OK 94.5 48/48
71
+ QWEN2_57B moonep 0 none none OK 93.6 40/40
72
+ QWEN3_235B moonep 1 none none OK 117.5 48/48
73
+ QWEN3_235B moonep 0 none none OK 117.3 40/40
74
+ QWEN3_30B moonep 1 none none OK 114.0 48/48
75
+ QWEN3_30B moonep 0 none none OK 113.7 40/40
76
+
77
+ ECHO/UltraEP note: graph-on uses static capacity + device-initiated GEMM while graph-off uses dynamic allocation + host-derived sizes. Their difference is not labelled as pure graph cost.
78
+ [02:38:09] step push: tierb-fixed-moonep-full -> MoE-CUDA-Graph-Analysis/tierb-b200-gbs8
79
+ Already logged in to Hugging Face as: AbhishekGhosh1998
80
+ uploaded: /workspace/tierb-fixed-moonep-full -> MoE-CUDA-Graph-Analysis/tierb-b200-gbs8/tierb-fixed-moonep-full
81
+ https://huggingface.co/MoE-CUDA-Graph-Analysis/tierb-b200-gbs8/tree/main
82
+
83
+ === STUDY COMPLETE
84
+ results : /workspace/tierb-fixed-moonep-full
85
+ status : /workspace/tierb-fixed-moonep-full/STATUS.txt
86
+ steps : /workspace/tierb-fixed-moonep-full/steps.tsv
87
+ hub : https://huggingface.co/datasets/MoE-CUDA-Graph-Analysis/tierb-b200-gbs8
88
+
89
+ status campaign phase gbs started ended result_dir
90
+ OK moonep tierbprobe 8 2026-08-29T23:33:50+00:00 2026-08-29T23:44:30+00:00 /workspace/tierb-fixed-moonep-full/moonep/tierbprobe
91
+ OK moonep tierb 8 2026-08-29T23:44:39+00:00 2026-08-29T23:59:52+00:00 /workspace/tierb-fixed-moonep-full/moonep/tierb
92
+ OK moonep tierbmem 8 2026-08-29T23:59:59+00:00 2026-08-30T00:09:58+00:00 /workspace/tierb-fixed-moonep-full/moonep/tierbmem
93
+ OK echo tierbtrace 8 2026-08-30T00:10:16+00:00 2026-08-30T00:38:30+00:00 /workspace/tierb-fixed-moonep-full/echo/tierbtrace
94
+ OK ultraep tierbtrace 8 2026-08-30T00:38:33+00:00 2026-08-30T00:59:28+00:00 /workspace/tierb-fixed-moonep-full/ultraep/tierbtrace
95
+ SUPERSEDED moonep tierbtrace 8 2026-08-30T00:59:40+00:00 2026-08-30T01:06:12+00:00 /workspace/tierb-fixed-moonep-full/moonep/tierbtrace
96
+ OK moonep tierb 8 2026-08-30T01:30:54+00:00 2026-08-30T01:46:01+00:00 /workspace/tierb-fixed-moonep-full/moonep/tierb
97
+ study_root: /workspace/tierb-fixed-moonep-full
98
+ gbs: 8
99
+ stages: 9
100
+ nsys: 1
101
+ hf_repo: MoE-CUDA-Graph-Analysis/tierb-b200-gbs8
102
+ hf_remote_root: tierb-fixed-moonep-full
103
+ started: 2026-08-30T02:38:13+00:00
104
+ driver_repo: /workspace/blackwell_run_scripts @ e8bd2bd
105
+ [02:38:13] starting HF watcher: every 600s -> MoE-CUDA-Graph-Analysis/tierb-b200-gbs8
106
+ [02:38:13] watcher pid 416688 (log: /workspace/tierb-fixed-moonep-full/hf_watcher.log)
107
+
108
+ === STAGE 9 -- Tier B nsys traces (final, trace-only)
109
+
110
+ === STEP moonep__tierbtrace__gbs8
111
+ [02:38:13] campaign: granularity-echo -> granularity-moonep
112
+ [02:38:13] megatron: /workspace/Megatron-MoE-Echo already on moonep_backend_async_grad_reduce (2e68c8ad)
113
+ ==============================================================
114
+ CAMPAIGN: moonep backend=moonep phase=tierbtrace
115
+ Megatron: /workspace/Megatron-MoE-Echo @ moonep_backend_async_grad_reduce
116
+ GBS=8 seq=4096 repeats=3 nsys=1
117
+ results -> /workspace/tierb-fixed-moonep-full/moonep/tierbtrace
118
+ ==============================================================
119
+ [02:38:28] building moonep marker extension (log: /workspace/tierb-fixed-moonep-full/moonep/tierbtrace/tierbtrace_gbs8/trace_extension_build.log)
120
+ [02:38:46] marker extension ready: _moonep_trace_C.cpython-312-x86_64-linux-gnu.so
121
+ [02:38:46] TRACE-ONLY QWEN3_235B__moonep__matched__seq8192__gbs8__capnone__async (ranks: 0 1 2 3 4 5 6 7)
122
+ [guard] iter 6/10 (60s)
123
+ [guard] iter 8/10 (70s)
124
+ [guard] iter 10/10 (110s)
125
+ [guard] nsys stop (flushing report)
126
+ [02:41:08] trace verified -> QWEN3_235B__moonep__matched__seq8192__gbs8__capnone__async.nsys-rep (7.3M, all 8 GPUs + moonep markers)
127
+ [02:41:09] TRACE-ONLY QWEN3_30B__moonep__matched__seq4096__gbs8__capnone__async (ranks: 0 1 2 3 4 5 6 7)
128
+ [guard] iter 1/10 (60s)
129
+ [guard] iter 8/10 (70s)
130
+ [02:45:10] WARN: nsys run failed for QWEN3_30B__moonep__matched__seq4096__gbs8__capnone__async (checking report anyway)
131
+ [02:45:12] trace verified -> QWEN3_30B__moonep__matched__seq4096__gbs8__capnone__async.nsys-rep (24M, all 8 GPUs + moonep markers)
132
+ [02:45:14] TRACE-ONLY DSV2_LITE_L24__moonep__matched__seq8192__gbs8__capnone__async (ranks: 0 1 2 3 4 5 6 7)
133
+ [guard] iter 3/10 (60s)
134
+ [guard] iter 8/10 (70s)
135
+ [guard] iter 10/10 (110s)
136
+ [guard] nsys stop (flushing report)
137
+ [02:47:36] trace verified -> DSV2_LITE_L24__moonep__matched__seq8192__gbs8__capnone__async.nsys-rep (13M, all 8 GPUs + moonep markers)
138
+ [02:47:37] TRACE-ONLY QWEN2_57B__moonep__matched__seq4096__gbs8__capnone__async (ranks: 0 1 2 3 4 5 6 7)
139
+ [guard] iter 8/10 (60s)
140
+ [guard] iter 10/10 (110s)
141
+ [guard] nsys stop (flushing report)
142
+ [02:49:59] trace verified -> QWEN2_57B__moonep__matched__seq4096__gbs8__capnone__async.nsys-rep (5.6M, all 8 GPUs + moonep markers)
143
+ --------------------------------------------------------------------------
144
+
145
+ ==============================================================
146
+ SWEEP COMPLETE ok=4 failed=0
147
+ ==============================================================
148
+ config backend policy slots seq gbs cap recompute rep status iter_ms_mean iter_ms_min peak_mib run_dir
149
+
150
+ Results: /workspace/tierb-fixed-moonep-full/moonep/tierbtrace/tierbtrace_gbs8/results.tsv
151
+ Roll up: python3 /workspace/blackwell_run_scripts/granularity/tools/collect.py /workspace/tierb-fixed-moonep-full/moonep/tierbtrace/tierbtrace_gbs8
152
+ ==============================================================
153
+
154
+ Campaign results: /workspace/tierb-fixed-moonep-full/moonep/tierbtrace
155
+ Roll up: python3 /workspace/blackwell_run_scripts/granularity/tools/collect.py /workspace/tierb-fixed-moonep-full/moonep/tierbtrace --compare
156
+ [02:49:59] step push: tierb-fixed-moonep-full/moonep/tierbtrace -> MoE-CUDA-Graph-Analysis/tierb-b200-gbs8
157
+ Already logged in to Hugging Face as: AbhishekGhosh1998
158
+ uploaded: /workspace/tierb-fixed-moonep-full/moonep/tierbtrace -> MoE-CUDA-Graph-Analysis/tierb-b200-gbs8/tierb-fixed-moonep-full/moonep/tierbtrace
159
+ https://huggingface.co/MoE-CUDA-Graph-Analysis/tierb-b200-gbs8/tree/main
160
+
161
+ === ROLL-UP
162
+ 40 runs ok=36 oom=0 -> /workspace/tierb-fixed-moonep-full/collected_all.tsv
163
+
164
+ (excluded 12 nsys-traced run(s) from the timing comparison)
165
+
166
+ phase config seq gbs policy rc cg asy backend iter_ms vs_best
167
+ -------------------------------------------------------------------------------------
168
+ tierb DSV2_LITE_L24 8192 8 matched none 1 1 moonep 593.43 1.00x
169
+
170
+ tierb QWEN2_57B 4096 8 matched none 1 1 moonep 180.12 1.00x
171
+
172
+ tierb QWEN3_235B 8192 8 matched none 1 1 moonep 317.80 1.00x
173
+
174
+ tierb QWEN3_30B 4096 8 matched none 1 1 moonep 355.16 1.00x
175
+
176
+ tierbmem DSV2_LITE_L24 8192 8 matched none 1 1 moonep 622.80 1.00x
177
+
178
+ tierbmem QWEN2_57B 4096 8 matched none 1 1 moonep 196.60 1.00x
179
+
180
+ tierbmem QWEN3_235B 8192 8 matched none 1 1 moonep 341.70 1.00x
181
+
182
+ tierbmem QWEN3_30B 4096 8 matched none 1 1 moonep 435.30 1.00x
183
+
184
+ tierbprobeDSV2_LITE_L24 8192 8 matched none 0 1 moonep 673.94 1.00x
185
+
186
+ tierbprobeQWEN2_57B 4096 8 matched none 0 1 moonep 238.91 1.00x
187
+
188
+ tierbprobeQWEN3_235B 8192 8 matched none 0 1 moonep 384.76 1.00x
189
+
190
+ tierbprobeQWEN3_30B 4096 8 matched none 0 1 moonep 824.23 1.00x
191
+
192
+ 8 probe runs -> /workspace/tierb-fixed-moonep-full/probe_summary_all.tsv
193
+ run arm cap rawExpI preRankI postRankI drop% reqCap
194
+ --------------------------------------------------------------------------------------------------------------------
195
+ moonep/tierbprobe/tierbprobe_gbs8/DSV2_LITE_L24__moo natural none 7.5443* 2.1316 1.1058
196
+ moonep/tierbprobe/tierbprobe_gbs8/DSV2_LITE_L24__moo forced none 1.0286 1.0061 1.0026
197
+ moonep/tierbprobe/tierbprobe_gbs8/QWEN2_57B__moonep_ natural none 7.4132* 2.3019 1.0979
198
+ moonep/tierbprobe/tierbprobe_gbs8/QWEN2_57B__moonep_ forced none 1.0348 1.0073 1.0033
199
+ moonep/tierbprobe/tierbprobe_gbs8/QWEN3_235B__moonep natural none 15.9227* 2.5851 1.0712
200
+ moonep/tierbprobe/tierbprobe_gbs8/QWEN3_235B__moonep forced none 1.0386 1.0053 1.0025
201
+ moonep/tierbprobe/tierbprobe_gbs8/QWEN3_30B__moonep_ natural none 15.7078* 2.4672 1.0635
202
+ moonep/tierbprobe/tierbprobe_gbs8/QWEN3_30B__moonep_ forced none 1.0565 1.0077 1.0036
203
+
204
+ * At least one globally aggregated snapshot reached the E/K expert-Imax ceiling.
205
+ Inspect raw_expert_imax_mean and hottest_expert_token_fraction_mean before generalising.
206
+
207
+ ECHO note: echo_logged_pre_rank_metric_valid=0 and echo_logged_load_reduction_valid=0 are intentional.
208
+ The branch's built-in fields mix source-rank and destination-rank axes; this tool reconstructs both globally.
209
+ 8 memory-dump runs -> /workspace/tierb-fixed-moonep-full/memory_summary_all.tsv
210
+ model backend cg allocation capacity status peak GiB dumps
211
+ ----------------------------------------------------------------------------------
212
+ DSV2_LITE_L24 moonep 1 none none OK 130.2 48/48
213
+ DSV2_LITE_L24 moonep 0 none none OK 122.2 40/40
214
+ QWEN2_57B moonep 1 none none OK 94.5 48/48
215
+ QWEN2_57B moonep 0 none none OK 93.6 40/40
216
+ QWEN3_235B moonep 1 none none OK 117.5 48/48
217
+ QWEN3_235B moonep 0 none none OK 117.3 40/40
218
+ QWEN3_30B moonep 1 none none OK 114.0 48/48
219
+ QWEN3_30B moonep 0 none none OK 113.7 40/40
220
+
221
+ ECHO/UltraEP note: graph-on uses static capacity + device-initiated GEMM while graph-off uses dynamic allocation + host-derived sizes. Their difference is not labelled as pure graph cost.
222
+ [02:50:06] step push: tierb-fixed-moonep-full -> MoE-CUDA-Graph-Analysis/tierb-b200-gbs8
223
+ Already logged in to Hugging Face as: AbhishekGhosh1998
224
+ uploaded: /workspace/tierb-fixed-moonep-full -> MoE-CUDA-Graph-Analysis/tierb-b200-gbs8/tierb-fixed-moonep-full
225
+ https://huggingface.co/MoE-CUDA-Graph-Analysis/tierb-b200-gbs8/tree/main
226
+
227
+ === STUDY COMPLETE
228
+ results : /workspace/tierb-fixed-moonep-full
229
+ status : /workspace/tierb-fixed-moonep-full/STATUS.txt
230
+ steps : /workspace/tierb-fixed-moonep-full/steps.tsv
231
+ hub : https://huggingface.co/datasets/MoE-CUDA-Graph-Analysis/tierb-b200-gbs8
232
+
233
+ status campaign phase gbs started ended result_dir
234
+ OK moonep tierbprobe 8 2026-08-29T23:33:50+00:00 2026-08-29T23:44:30+00:00 /workspace/tierb-fixed-moonep-full/moonep/tierbprobe
235
+ OK moonep tierb 8 2026-08-29T23:44:39+00:00 2026-08-29T23:59:52+00:00 /workspace/tierb-fixed-moonep-full/moonep/tierb
236
+ OK moonep tierbmem 8 2026-08-29T23:59:59+00:00 2026-08-30T00:09:58+00:00 /workspace/tierb-fixed-moonep-full/moonep/tierbmem
237
+ OK echo tierbtrace 8 2026-08-30T00:10:16+00:00 2026-08-30T00:38:30+00:00 /workspace/tierb-fixed-moonep-full/echo/tierbtrace
238
+ OK ultraep tierbtrace 8 2026-08-30T00:38:33+00:00 2026-08-30T00:59:28+00:00 /workspace/tierb-fixed-moonep-full/ultraep/tierbtrace
239
+ SUPERSEDED moonep tierbtrace 8 2026-08-30T00:59:40+00:00 2026-08-30T01:06:12+00:00 /workspace/tierb-fixed-moonep-full/moonep/tierbtrace
240
+ OK moonep tierb 8 2026-08-30T01:30:54+00:00 2026-08-30T01:46:01+00:00 /workspace/tierb-fixed-moonep-full/moonep/tierb
241
+ OK moonep tierbtrace 8 2026-08-30T02:38:13+00:00 2026-08-30T02:49:59+00:00 /workspace/tierb-fixed-moonep-full/moonep/tierbtrace
242
+ PATCH RERUN COMPLETE: /workspace/tierb-fixed-moonep-full