AbhishekGhosh1998 commited on
Commit
42ce09c
Β·
verified Β·
1 Parent(s): b7ea177

Upload granularity_next_b200/NEXT_CAMPAIGN_RESULTS.md with huggingface_hub

Browse files
granularity_next_b200/NEXT_CAMPAIGN_RESULTS.md ADDED
@@ -0,0 +1,227 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # Next campaign results: Tier A top-k separation and new Tier B models
2
+
3
+ Study root: `/workspace/granularity_next_b200`
4
+ HF: `MoE-CUDA-Graph-Analysis/tierb-b200-gbs8` under `granularity_next_b200/`
5
+ Hardware: 8Γ—B200. Tier A GBS 16, Tier B GBS 8, seq 4096, EP 8, MXFP8.
6
+
7
+ Executed per `NEXT_TOPK_AND_TIERB_PLAN.md`. **0 failed cells** across the whole
8
+ campaign; 36 nsys reports, every one carrying markers on all 8 GPUs.
9
+
10
+ ## 1. What ran
11
+
12
+ | suite | scope | status |
13
+ | --- | --- | --- |
14
+ | Tier A MoonEP routing refresh | G1–G4 on the fixed probe branch | complete |
15
+ | Tier A top-k matrix | 9 cells Γ— 3 backends Γ— probe/timing/memory/trace | complete |
16
+ | Tier B new models | 3 of 4 Γ— 3 backends Γ— all four stages | complete |
17
+ | `QWEN3_NEXT_L16` | β€” | **deferred, see Β§5** |
18
+
19
+ ## 2. Tier A: top-k separated from expert shape
20
+
21
+ Step time (ms) and peak memory (GiB), mean of 3 unprofiled repeats, CUDA graphs
22
+ on, zero token drops.
23
+
24
+ | cell | ECHO | UltraEP | MoonEP |
25
+ | --- | --- | --- | --- |
26
+ | G2_K2 | 270.9 / 132 | 335.3 / 99 | **254.4** / **86** |
27
+ | G3_K2 | 247.0 / 133 | 248.5 / 100 | **226.3** / **85** |
28
+ | G3_K8 | **352.6** / 138 | 361.3 / 104 | 361.5 / **90** |
29
+ | G4_K2 | 251.9 / 136 | **215.0** / 102 | 222.4 / **84** |
30
+ | G4_K4 | 279.6 / 138 | **247.5** / 103 | 259.6 / **85** |
31
+ | G4_K6 | 304.9 / 139 | **290.8** / 105 | 299.9 / **86** |
32
+ | G4_K8 | 335.2 / 139 | **315.9** / 104 | 335.9 / **88** |
33
+ | G4_K10 | 345.7 / 141 | **336.8** / 106 | 369.3 / **89** |
34
+ | G4_K16 | 385.8 / 146 | **378.8** / 111 | 457.4 / **91** |
35
+
36
+ ### G4's cost is fanout, not small experts
37
+
38
+ This is what the campaign was built to settle, and the answer is unambiguous.
39
+ Holding G4's expert shape fixed (E=128, Hβ€²=896) and varying only top-k:
40
+
41
+ | K | ECHO ms | vs K=2 |
42
+ | ---: | ---: | ---: |
43
+ | 2 | 251.9 | β€” |
44
+ | 4 | 279.6 | +11% |
45
+ | 8 | 335.2 | +33% |
46
+ | 16 | 385.8 | +53% |
47
+ | *32 (original diagonal)* | *486.7* | *+93%* |
48
+
49
+ The decisive comparison is **G4_K16 (385.8 ms) against the original G3 diagonal
50
+ (437.2 ms)**: G4_K16 has twice the experts at half the width, yet runs 12%
51
+ faster. Small experts are not what made G4 expensive. Its published 486.7 ms was
52
+ a consequence of K=32.
53
+
54
+ The curve is smooth and monotonic with no knee, so the optional compute-matched
55
+ fanout microbenchmark the plan holds in reserve is **not needed**.
56
+
57
+ ### The K=2 cells show granularity alone is cheap, and non-monotonic for UltraEP
58
+
59
+ At fixed low fanout, going from 32 to 128 experts barely moves ECHO
60
+ (270.9 β†’ 247.0 β†’ 251.9) but moves UltraEP a great deal
61
+ (335.3 β†’ 248.5 β†’ **215.0**). UltraEP is the *worst* backend at G2_K2, by 24%,
62
+ and the *best* at G4_K2, by 15%. That crossover at constant fanout is invisible
63
+ on the original diagonal, where E and K move together.
64
+
65
+ ### MoonEP crosses over at Kβ‰ˆ8
66
+
67
+ MoonEP is fastest at G2_K2 and G3_K2 and within 3% of UltraEP up to K=6, then
68
+ degrades: +21% versus UltraEP by K=16. Its weakness is therefore **fanout**, not
69
+ granularity β€” the same effect that produced its published G4 deficit at K=32.
70
+
71
+ Its async grad-reduce overlap stays healthy across the ladder (92.9–97.0% on 8
72
+ devices), so the degradation is not exposed reduce time.
73
+
74
+ ### Memory: MoonEP is lightest everywhere
75
+
76
+ 84–91 GiB against UltraEP's 99–111 and ECHO's 132–146 β€” roughly 50 GiB below
77
+ ECHO at every cell. This reproduces the Tier A/B ordering of the fixed
78
+ DDP-owned stack on an entirely new set of shapes.
79
+
80
+ ### Capacity demand falls as fanout rises
81
+
82
+ Validated zero-drop capacity, ECHO: 1.203 at K=2 down to 1.031 at K=16,
83
+ monotonically. Higher top-k spreads each token over more experts, so per-expert
84
+ imbalance averages out. UltraEP stays flat at ~1.010 across nearly the whole
85
+ ladder β€” its balancer absorbs exactly the low-fanout imbalance ECHO leaves
86
+ exposed. Its one elevated point is G3_K2 (1.090).
87
+
88
+ MoonEP takes no capacity factor and is tagged `capnone` throughout.
89
+
90
+ ## 3. Tier B: new real-model slices
91
+
92
+ | model | ECHO | UltraEP | MoonEP |
93
+ | --- | --- | --- | --- |
94
+ | `DEEPSEEK_V3_L4` | 198.5 ms / 128.0 GiB | 195.8 / 99.6 | **192.7** / **95.6** |
95
+ | `KIMI_K2_L2` | 119.4 / 129.1 | 110.1 / 88.3 | **108.6** / **84.5** |
96
+ | `OLMOE_1B_7B_L16` | 110.1 / 42.0 | **103.4** / 38.2 | 105.5 / **35.6** |
97
+
98
+ **MoonEP is lightest on all three and fastest on two.** That is a reversal of
99
+ the pre-fix stack, where it was slowest everywhere.
100
+
101
+ OLMoE is the exception on throughput, and the exception is consistent with the
102
+ Tier A finding: OLMoE has 64 experts over 8 ranks, so 8 slots per rank, against
103
+ Kimi's 48. With so few duplicated experts there is little for MoonEP's machinery
104
+ to exploit, and UltraEP edges it by 2%.
105
+
106
+ ### The memory fallback fired exactly once, and correctly
107
+
108
+ | requested | selected | max timing peak | reason |
109
+ | --- | --- | ---: | --- |
110
+ | `DEEPSEEK_V3_L5` | **`DEEPSEEK_V3_L4`** | 128.0 GiB | L5 exceeded the ceiling on ECHO |
111
+ | `KIMI_K2_L2` | `KIMI_K2_L2` | 129.1 GiB | fits as requested |
112
+ | `OLMOE_1B_7B_L16` | `OLMOE_1B_7B_L16` | 42.0 GiB | fits as requested |
113
+
114
+ `DEEPSEEK_V3_L5` ran successfully on all three backends, but ECHO peaked at
115
+ **175.1 GiB**, above the 165 GiB promotion ceiling (UltraEP 138.9, MoonEP
116
+ 137.1). This is a promotion-policy decision, not an OOM.
117
+
118
+ **The gate has to be on timing, not on the probe.** ECHO's graph-on timing peak
119
+ came in **17 GiB above** its graph-off probe peak (158.3 GiB), so probing alone
120
+ would have promoted a slice that does not fit. Dropping one layer cut ECHO's
121
+ peak by 47 GiB, which shows how completely the MoE layer dominates at
122
+ hidden=7168.
123
+
124
+ `OLMOE_1B_7B_L16` at 42.0 GiB sits **below** the desired 90–155 GiB matching
125
+ band. Per the plan it is retained with this note rather than treated as a
126
+ failure: the full 16-layer model simply does not fill an 8Γ—B200 node, and
127
+ reducing it further was never the intent.
128
+
129
+ ## 4. Interpretation constraints
130
+
131
+ **Across the K=2 cells, active expert FLOPs are not constant.** Routed expert
132
+ width is part of granularity, so a G2β†’G4 comparison at fixed K changes the work
133
+ as well as the shape. Report those rows alongside active FLOPs or per-routed-row
134
+ time, never as a pure compute-matched experiment.
135
+
136
+ **Tier B slices are reduced-depth prefixes**, and every model uses the same
137
+ GPT2BPE vocabulary and WikiText binary, TP=1, PP=1, EP=8, MXFP8, and a short
138
+ fixed horizon. The architecture is preserved; the data path is not the models'
139
+ native pretraining recipe and must not be described as such.
140
+
141
+ **MoonEP numbers are a post-fix baseline, not a delta.** The pre-fix stack
142
+ computed something different, so differences across the fix are not speedups.
143
+
144
+ ## 5. `QWEN3_NEXT_L16` is deferred, not failed
145
+
146
+ `preflight_next_models.sh` fails it on **all nine branch-role combinations**
147
+ (timing, probe, memory Γ— three backends). Ten model-defining flags are absent:
148
+
149
+ ```
150
+ --attention-output-gate --no-weight-decay-cond-type
151
+ --moe-shared-expert-gate --linear-attention-type
152
+ --linear-attention-freq --linear-conv-kernel-dim
153
+ --linear-key-head-dim --linear-value-head-dim
154
+ --linear-num-key-heads --linear-num-value-heads
155
+ ```
156
+
157
+ The integration branches predate Megatron's Gated DeltaNet and
158
+ attention-output-gate support. Per the plan these must be **ported or rebased,
159
+ never stripped to make the preflight pass** β€” removing them would measure a
160
+ different model while reporting the Qwen3-Next name. The other three models were
161
+ run meanwhile, which is what the plan prescribes.
162
+
163
+ ## 6. Acceptance checklist
164
+
165
+ | check | result |
166
+ | --- | --- |
167
+ | intended timing/probe/memory branch per backend | yes, orchestrator-enforced |
168
+ | MoonEP on the fixed duplicated-gradient lineage | `moonep_backend_async_grad_reduce` |
169
+ | ECHO/UltraEP zero drops at the common capacity | yes |
170
+ | unprofiled timing on all three backends | yes, 3 repeats every cell |
171
+ | Tier B process HBM ≀ 165 GiB on every backend | yes, after the L5β†’L4 fallback |
172
+ | allocator dumps complete | 57/60 cells; see below |
173
+ | nsys: all 8 GPUs + backend markers | 36/36 |
174
+ | MoonEP async overlap nonzero | Tier A 92.9–97.0%, Tier B 58.2–89.2% |
175
+ | no `DEFERRED_UNSUPPORTED`/`BADFLAG` reclassified as OOM | none |
176
+
177
+ ### The three `INCOMPLETE_DUMP` rows are a checker artifact
178
+
179
+ `G2_K2/echo`, `G4_K16/echo` and `G4_K8/moonep` are flagged, and the flag is
180
+ **not** a loss of measurement data:
181
+
182
+ * all 8 ranks reported in every one of the 60 memory cells; only the iteration
183
+ count differs (4 or 6 captured against an expected 5), because `TIMING_ITERS`
184
+ differs between the graph-on and no-graph arms while the check is calibrated
185
+ to a single value;
186
+ * `pickle_count` is **0 in all 60 cells**, including every passing one β€” this
187
+ branch writes JSON snapshots only, so half the equality test can never be
188
+ satisfied and contributes nothing;
189
+ * all three flagged cells still produced valid peaks, which come from
190
+ `mem_samples.csv` rather than from the snapshots;
191
+ * all three are the `dynamic`/no-graph *reference* arms. Every primary
192
+ static-capacity graph-on cell, which carries the headline memory numbers,
193
+ passed `OK`.
194
+
195
+ Worth fixing in the checker: derive `expected_dumps` from the arm's own
196
+ `TIMING_ITERS`, and drop the pickle half of the test on JSON-only branches.
197
+
198
+ ## 7. Campaign resilience
199
+
200
+ The pod was preempted mid-campaign, during the Tier A trace stage. Recovery cost
201
+ no measurement work: 29 completion markers across 10 sub-roots were restored
202
+ from the PVC mirror, the driver skipped every finished step, and it resumed
203
+ exactly at `ultraep__tierbtrace`. All results to that point were already on
204
+ HuggingFace.
205
+
206
+ Two harness defects were found and fixed during the campaign; both are recorded
207
+ in the driver repository.
208
+
209
+ **Traced cells burned 420 s each doing nothing.** The trace phase profiles steps
210
+ 6–8 but requests 10 iterations, so every traced cell reached iteration 8, wrote
211
+ its complete report, then deadlocked in a collective. The guard waited its full
212
+ stall timeout for two iterations that could never arrive and would not have been
213
+ in the report. Callers now pass `GUARD_PROFILE_END_ITER` and a stall past that
214
+ point uses 60 s. Measured effect: 9.2 β†’ 3.2 min per trace.
215
+
216
+ **`cp -a` cannot restore this PVC.** NFS rejects the chmod/chown it implies, so
217
+ cp aborts partway. Restoring the preempted campaign with `-a` copied 20 of 2433
218
+ files and 1 of 29 markers before failing β€” which would have silently re-run most
219
+ of the campaign despite the state being intact. Changed to `cp -r`.
220
+
221
+ ## 8. Open items
222
+
223
+ * `QWEN3_NEXT_L16` needs its architecture support ported onto the nine
224
+ branch-roles before it can be measured.
225
+ * The `INCOMPLETE_DUMP` check should be calibrated per arm (Β§6).
226
+ * The compute-matched fanout microbenchmark remains unnecessary; the core matrix
227
+ explains the G4 curve.