AbhishekGhosh1998 commited on
Commit
bf9442c
Β·
verified Β·
1 Parent(s): 0662f0e

granularity results tiera-rerun-fixed-moonep @ 2026-08-30T10:33:04Z from sp-aiinfra-interactive-aghosh-7jtjl-8xb200-moe-cg-0

Browse files
tiera-rerun-fixed-moonep/STATUS.txt CHANGED
@@ -1,7 +1,7 @@
1
  study_root: /workspace/tiera-rerun-fixed-moonep
2
  gbs: 16
3
- updated: 2026-08-30T10:26:00+00:00
4
- current: idle
5
 
6
  completed steps:
7
  echo__a1__gbs16
 
1
  study_root: /workspace/tiera-rerun-fixed-moonep
2
  gbs: 16
3
+ updated: 2026-08-30T10:26:19+00:00
4
+ current: done
5
 
6
  completed steps:
7
  echo__a1__gbs16
tiera-rerun-fixed-moonep/TIERA_RERUN.md ADDED
@@ -0,0 +1,339 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # Tier A rerun β€” fixed MoonEP, measured capacity
2
+
3
+ Driver: `./granularity/run_tiera_rerun.sh`
4
+ Study root: `/workspace/tiera-rerun-fixed-moonep`
5
+ HF: `MoE-CUDA-Graph-Analysis/tierb-b200-gbs8` under `tiera-rerun-fixed-moonep/`
6
+ Hardware: 8Γ—B200, **GBS 16**, seq 4096, EP 8.
7
+
8
+ This campaign re-measures the whole Tier A ladder. It supersedes the Tier A
9
+ tables in `RESULTS.md`.
10
+
11
+ ## 1. Why the published Tier A ladder had to be re-run
12
+
13
+ Two independent defects, either of which alone invalidates the MoonEP column.
14
+
15
+ **The MoonEP numbers measured the wrong computation.** The published ladder used
16
+ the pre-fix MoonEP integration, where DDP applied its `1/dp_size` expert-gradient
17
+ scaling during backward, *before* MoonEP's `reduce_grad` delivered the duplicated
18
+ experts' contribution β€” so that contribution ended up 8Γ— too large. The stack
19
+ still trained, still reported a plausible loss, and still produced timings. It
20
+ separately held two copies of every expert weight. Both its throughput and its
21
+ memory are therefore unusable. Root cause and fix:
22
+ `MOONEP_GRAD_BUG_AND_FIX.md`.
23
+
24
+ **Capacity was hardcoded, never measured.** Every Tier A cell ran at
25
+ `CAPACITY:-1.25`, a value nothing verified. `RESULTS.md` Β§3 of the caveats
26
+ records that *"8 runs mention capacity saturation (UltraEP probe cells dropping
27
+ tokens)"*. A run that drops assignments does **less expert work**, so it looks
28
+ faster for the wrong reason. Per the repository rule, a run that dropped tokens
29
+ is not a timing result.
30
+
31
+ ## 2. Capacity policy
32
+
33
+ The first choice was **one capacity shared by every cell**, so that nothing along
34
+ the ladder differs except granularity. **That turned out to be impossible**, and
35
+ the measurement that settles it is recorded below.
36
+
37
+ MoonEP takes no capacity factor and is excluded from the choice; it is tagged
38
+ `capnone` throughout.
39
+
40
+ ### What the probe measured
41
+
42
+ Validated zero-drop capacity per rung and backend, GBS 16, seq 4096:
43
+
44
+ | rung | ECHO | UltraEP | rung max |
45
+ | --- | ---: | ---: | ---: |
46
+ | **G1** | **1.859375** | **2.40625** | **2.40625** |
47
+ | G2 | 1.08203125 | 1.0625 | 1.08203125 |
48
+ | G3 | 1.072265625 | 1.0625 | 1.072265625 |
49
+ | G4 | 1.068359375 | 1.095703125 | 1.095703125 |
50
+
51
+ **G1 is a factor-of-two outlier and every other rung sits barely above 1.0.**
52
+ G1 has only 8 experts at top-2 across 8 ranks, so there is almost nothing to
53
+ average imbalance over; G4's 128 experts average it away almost completely.
54
+
55
+ That the cause is imbalance rather than shape is directly confirmed by the
56
+ probe's force-balanced control: with `--moe-router-force-load-balancing`, G1
57
+ needs only **1.01** on both backends. The capacity is being spent entirely on
58
+ routing skew.
59
+
60
+ ### Why a single global capacity is impossible
61
+
62
+ The global max is 2.40625, driven entirely by G1. A direct memory gate on the
63
+ two extreme rungs, CUDA graphs on, settles whether it can be used everywhere:
64
+
65
+ | gate cell | capacity | peak | verdict |
66
+ | --- | ---: | ---: | --- |
67
+ | G4/echo | 2.40625 | 182,627 MiB | **OOM** (ceiling 182,630) |
68
+ | G1/echo | 2.40625 | 161,863 MiB | OK |
69
+
70
+ So no single capacity satisfies both "no drops anywhere" and "fits everywhere".
71
+ The campaign uses **one capacity per rung**, recorded in `tiera_capacity.env`
72
+ with `TIERA_CAPACITY_GLOBAL_FEASIBLE=0` so the driver does not spend a full
73
+ timing stage rediscovering the OOM.
74
+
75
+ Within a rung every backend still shares one capacity, so the **backend
76
+ comparison stays valid**. Only the cross-rung trend must be read alongside the
77
+ capacity column. This is a deliberate, narrow relaxation of the "never mix
78
+ capacity within one comparison" rule, justified because each rung is its own
79
+ backend comparison.
80
+
81
+ ### The consequence for the published ladder
82
+
83
+ The published Tier A ran **every** rung at a hardcoded 1.25. Against the
84
+ measured requirements:
85
+
86
+ | rung | required | ran at | verdict |
87
+ | --- | ---: | ---: | --- |
88
+ | **G1** | 1.86 / 2.41 | 1.25 | **both backends dropped tokens** |
89
+ | G2 | 1.08 / 1.06 | 1.25 | fine |
90
+ | G3 | 1.07 / 1.06 | 1.25 | fine |
91
+ | G4 | 1.07 / 1.10 | 1.25 | fine |
92
+
93
+ The defect is **isolated to G1** β€” and G1 is precisely the rung the published
94
+ headline rests on ("ECHO wins only at the coarsest rung"). A run that drops
95
+ assignments does less expert work and therefore looks faster, so the published
96
+ G1 row cannot support that claim. G2–G4 were never capacity-limited and their
97
+ relative ordering is unaffected.
98
+
99
+ ### Why the probe needed a second pass at G1
100
+
101
+ The probe's own ladder could not resolve G1. ECHO's ladder tops out at 2.00 and
102
+ G1 needed 1.906 + 5% margin, just above it. UltraEP starts at 1.25 and was still
103
+ dropping at 1.8125 β€” and a dropping run **under-reports** demand, because
104
+ `actual_valid_recv` is clipped at the buffer, which is why its estimate climbed
105
+ from 1.72 to 2.31 as the capacity rose. G1 was therefore re-probed with
106
+ `TIERB_ECHO_PROBE_CAPACITIES_DESC="3.00 2.75 2.50 2.25"` and
107
+ `TIERB_ULTRAEP_INITIAL_CAP=3.00`, where both backends measured cleanly with zero
108
+ drops.
109
+
110
+ ### Why the grid is coarsest-wins
111
+
112
+ A capacity is executable only when `int(C Β· seq Β· mbs Β· topk)` is a multiple of
113
+ 128. `topk` varies 16Γ— along the ladder, so the rungs do **not** share a grid:
114
+
115
+ | rung | topk | step |
116
+ | --- | ---: | --- |
117
+ | G1 | 2 | 1/128 ← coarsest, therefore binding on any shared capacity |
118
+ | G2 | 8 | 1/512 |
119
+ | G3 | 16 | 1/1024 |
120
+ | G4 | 32 | 1/2048 |
121
+
122
+ `tools/tiera_global_capacity.py` enforces this and re-verifies alignment on every
123
+ rung before emitting a value.
124
+
125
+ ### Why GBS stays 16
126
+
127
+ ECHO G1 OOMs at GBS 32. The published ladder is GBS 16 and this rerun matches it
128
+ so the two remain comparable.
129
+
130
+ ## 3. Branch matrix
131
+
132
+ Each phase demands a different Megatron checkout. The orchestrator switches
133
+ automatically; `run_ladder.sh` asserts the result.
134
+
135
+ | phase | ECHO | UltraEP | MoonEP |
136
+ | --- | --- | --- | --- |
137
+ | probe (stage 7) | `token-drop-analysis` | `ultraep-token-drop-analysis` | *(excluded β€” no capacity factor)* |
138
+ | timing (stage 3) | `bracing-regions-through-kernels` | `ultraep-bracing-regions-through-kernels` | `moonep_backend_async_grad_reduce` |
139
+ | memory (stage 8) | `better-memory-dump-support` | `ultraep-better-memory-dump-support` | `moonep_backend_async_grad_reduce_better_memory_dump_support` |
140
+ | trace (stage 9) | `bracing-regions-through-kernels` | `ultraep-bracing-regions-through-kernels` | `moonep_backend_async_grad_reduce` |
141
+
142
+ **There is no separate MoonEP "async + bracing" branch, and none is needed.**
143
+ `moonep_backend_bracing_regions_through_kernels` is an *ancestor* of
144
+ `moonep_backend_async_grad_reduce`, so the async branch already carries every
145
+ bracing region β€” plus two the bracing branch lacks:
146
+
147
+ ```
148
+ _(moonep_weight_prefetch, 1100)
149
+ _(moonep_grad_reduce, 1110)
150
+ ```
151
+
152
+ Those brace the weight-movement collectives that live in `fused_moonep.py`
153
+ rather than in the layer or dispatcher. `moonep_grad_reduce` is the marker the
154
+ overlap analyzer keys on; without it the async deferral cannot be measured.
155
+
156
+ **The four pre-async MoonEP branches do not carry the gradient fix**
157
+ (`moonep_backend`, `..._bracing_regions_through_kernels`,
158
+ `..._token_routing_analysis`, `..._better_memory_dump_support`). They must never
159
+ be selected. `run_ladder.sh` now refuses to run MoonEP unless
160
+ `external_contributors` is present in `param_and_grad_buffer.py`;
161
+ `STRICT_MOONEP_GRAD_FIX=0` overrides it, deliberately explicitly.
162
+
163
+ ## 4. MoonEP now appears in a1 and ponly
164
+
165
+ `RESULTS.md` states that MoonEP *"does not appear in `a1`/`ponly` by
166
+ construction, not by omission: it forces B = E/R and cannot run the one-slot
167
+ policy."*
168
+
169
+ **That is true for G2–G4 only.** MoonEP forces `B = E/R`, which conflicts with
170
+ the one-slot policy only when `E/R > 1`. G1 and PONLY are both `E=8`, so at
171
+ `R=8` the forced value already **is** one slot:
172
+
173
+ | config | E | E/R at R=8 | one-slot policy |
174
+ | --- | ---: | ---: | --- |
175
+ | G1 | 8 | 1 | βœ… runnable |
176
+ | PONLY | 8 | 1 | βœ… runnable |
177
+ | G2 | 32 | 4 | ❌ |
178
+ | G3 | 64 | 8 | ❌ |
179
+ | G4 | 128 | 16 | ❌ |
180
+
181
+ `phase_a1`/`phase_ponly` carried a blanket backend-level skip that threw those
182
+ cells away. `sweep_cell` already makes the correct per-config decision and warns
183
+ when it skips, so the phases now defer to it. This campaign therefore produces
184
+ **MoonEP PONLY numbers at both sequence lengths, and MoonEP `a1` at G1** β€” data
185
+ the published ladder never collected.
186
+
187
+ Note that at G1 the `one` and `matched` policies are the same configuration
188
+ (B=1) for *every* backend, so `a1@G1` and `a2@G1` are the same measurement taken
189
+ twice. That is a consistency check, not two independent results.
190
+
191
+ ## 5. A tagging correction
192
+
193
+ `train_moe.sh` already downgrades `CAPACITY_MODE=static` to `none` for MoonEP,
194
+ so passing a capacity through never changed the run. But the Tier A phases still
195
+ put that capacity into the run-directory tag and the results table, which made
196
+ MoonEP rows look as though they had been measured at a capacity they ignore β€”
197
+ and would have re-tagged byte-identical MoonEP cells as new measurements when
198
+ the global capacity changed. Tier B already tags them `capnone`; Tier A now
199
+ agrees.
200
+
201
+ ## 6. What this campaign collects
202
+
203
+ | stage | phase | content |
204
+ | --- | --- | --- |
205
+ | 7 | `tierbprobe` | zero-drop capacity per rung Γ— backend, CG off, instrumented branches |
206
+ | 3 | `a1`, `ponly`, `a2` | throughput, CG on, 3 repeats, all three backends |
207
+ | 8 | `tierbmem` | allocator snapshots at the chosen capacity, CG on and off |
208
+ | 9 | `tierbtrace` | nsys traces with bracing markers, all 8 GPUs verified |
209
+
210
+ Tier A previously recorded `peak_mib` only and never captured allocator
211
+ snapshots, so stage 8 is new for this tier.
212
+
213
+ ## 7. Reading the results
214
+
215
+ **Post-fix MoonEP is a new implementation baseline, not an optimization delta.**
216
+ The pre-fix stack computed something different, so a before/after speedup across
217
+ the fix is meaningless. Do not subtract the old Tier A MoonEP column from the
218
+ new one.
219
+
220
+ **Do not compare across capacity modes.** If `CAPACITY_DECISION.txt` records
221
+ `capacity_mode=perrung`, the cross-rung trend must be read together with the
222
+ capacity column.
223
+
224
+ **Verify zero drops before quoting any timing.** That is the whole reason this
225
+ campaign exists.
226
+
227
+ ## 8. Results
228
+
229
+ All numbers: GBS 16, seq 4096 (except PONLY 8192), mean of 3 unprofiled repeats,
230
+ zero token drops, CUDA graphs on. 81 timing rows, none failed.
231
+
232
+ ### a2 β€” capacity-matched, B = E/R
233
+
234
+ | rung | capacity | ECHO | UltraEP | MoonEP |
235
+ | --- | ---: | ---: | ---: | ---: |
236
+ | G1 | 2.40625 | **466.8** | 892.7 | 480.6 |
237
+ | G2 | 1.08203125 | **415.4** | 492.3 | 432.2 |
238
+ | G3 | 1.072265625 | **437.2** | 459.7 | 559.9 |
239
+ | G4 | 1.095703125 | 486.7 | **482.9** | 782.8 |
240
+
241
+ ### Peak memory (MiB, a2)
242
+
243
+ | rung | ECHO | UltraEP | MoonEP |
244
+ | --- | ---: | ---: | ---: |
245
+ | G1 | 161,723 | 129,475 | **96,849** |
246
+ | G2 | 145,691 | 112,315 | **97,665** |
247
+ | G3 | 147,933 | 113,951 | **99,409** |
248
+ | G4 | 168,485 | 132,437 | **106,031** |
249
+
250
+ ### a1 β€” one-slot policy
251
+
252
+ | rung | ECHO | UltraEP | MoonEP |
253
+ | --- | ---: | ---: | ---: |
254
+ | G1 | **467.3** | 890.4 | 479.1 |
255
+ | G2 | **394.2** | 466.6 | β€” |
256
+ | G3 | **412.7** | 448.0 | β€” |
257
+ | G4 | **460.7** | 467.6 | β€” |
258
+
259
+ MoonEP's G1 entry is new. Its dashes at G2–G4 are structural (`E/R > 1`), not
260
+ missing data. As a consistency check, MoonEP G1 in `a1` (479.1) and in `a2`
261
+ (480.6) are the same configuration measured twice and agree to 0.3%.
262
+
263
+ ### ponly β€” payload-only diagnostic
264
+
265
+ | seq | ECHO | UltraEP | MoonEP |
266
+ | --- | ---: | ---: | ---: |
267
+ | 4096 | 175.0 | 176.8 | **165.2** |
268
+ | 8192 | 316.5 | **292.7** | 300.6 |
269
+
270
+ Also new: the published ladder has no MoonEP `ponly` row at all.
271
+
272
+ PONLY was not probed separately; it inherits G1's capacity because it has the
273
+ same E=8 and K=2, and capacity demand here is driven by routing imbalance over E
274
+ and K. Inheriting a *higher* capacity cannot cause drops, so this is safe, but it
275
+ is an inheritance rather than a measurement.
276
+
277
+ ### MoonEP async grad-reduce overlap
278
+
279
+ From the stage-9 traces, 8 devices:
280
+
281
+ | rung | overlap | reduce, ms/iter |
282
+ | --- | ---: | ---: |
283
+ | G1 | 53.5% | 58.44 |
284
+ | G2 | 97.2% | 27.94 |
285
+ | G3 | 95.7% | 16.46 |
286
+ | G4 | 94.1% | 10.79 |
287
+
288
+ Overlap effectiveness falls sharply as experts get larger. At G1 each expert is
289
+ `H'=14336` wide and the gradient reduce is 58 ms per iteration β€” too much to
290
+ hide behind one layer's backward, so nearly half of it stays exposed. By G4 the
291
+ expert is 16Γ— narrower, the reduce costs 10.8 ms, and 94% of it disappears.
292
+ This is a plausible part of why MoonEP's G1 disadvantage is smallest and its G4
293
+ disadvantage largest, though the two move in opposite directions and the
294
+ exposed-reduce term alone does not explain the G4 result.
295
+
296
+ ## 9. What changed versus the published ladder
297
+
298
+ **Two published conclusions do not survive.**
299
+
300
+ *"ECHO wins only at the coarsest rung; UltraEP wins everywhere finer."* Not
301
+ reproduced. ECHO now wins G1, G2 and G3; UltraEP wins only G4, by 0.8%
302
+ (482.9 vs 486.7). The published claim rested on a G1 row where both backends
303
+ were dropping tokens.
304
+
305
+ *"UltraEP is 30–35 GB lighter than ECHO at every rung."* Still true of UltraEP
306
+ versus ECHO, but it is no longer the headline: **MoonEP is now the lightest
307
+ backend at every rung**, by 30–60 GB against UltraEP and 50–65 GB against ECHO.
308
+ Its published memory (154–166 GB) came from a stack that held two copies of
309
+ every expert weight; the DDP-owned symmetric-parameter implementation brings it
310
+ to 97–106 GB.
311
+
312
+ **MoonEP's throughput ranking changed too**, from slowest everywhere to second
313
+ at G1/G2. It still degrades fastest along the ladder.
314
+
315
+ | rung | MoonEP published | MoonEP now |
316
+ | --- | ---: | ---: |
317
+ | G1 | 696.2 | 480.6 |
318
+ | G2 | 656.1 | 432.2 |
319
+ | G3 | 769.7 | 559.9 |
320
+ | G4 | 984.5 | 782.8 |
321
+
322
+ These are **not** a speedup measurement. The published stack computed something
323
+ different, so the difference across the fix is not an optimization delta. Treat
324
+ the new column as a first correct baseline.
325
+
326
+ ## 10. Audit
327
+
328
+ | check | result |
329
+ | --- | --- |
330
+ | probe selection for all 8 (rung, backend) pairs | yes, after G1 re-probe |
331
+ | capacities executable on every rung | verified by the tool |
332
+ | `CAPACITY_DECISION.txt` records mode and reason | `capacity_mode=perrung` |
333
+ | timing rows, 3 repeats, zero failures | 81 OK, 0 failed |
334
+ | zero token drops | yes, capacities chosen for it |
335
+ | `a1` skipped exactly G2/G3/G4 for MoonEP | yes, with reasons logged |
336
+ | MoonEP PONLY at seq 4096 and 8192 | yes |
337
+ | memory dumps, all rungs Γ— backends | 12/12 markers |
338
+ | nsys reports | 12/12, markers on all 8 GPUs |
339
+ | MoonEP async overlap nonzero | 53–97% |