Upload granularity_next_b200/NEXT_CAMPAIGN_RESULTS.md with huggingface_hub
Browse files
granularity_next_b200/NEXT_CAMPAIGN_RESULTS.md
ADDED
|
@@ -0,0 +1,227 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
# Next campaign results: Tier A top-k separation and new Tier B models
|
| 2 |
+
|
| 3 |
+
Study root: `/workspace/granularity_next_b200`
|
| 4 |
+
HF: `MoE-CUDA-Graph-Analysis/tierb-b200-gbs8` under `granularity_next_b200/`
|
| 5 |
+
Hardware: 8ΓB200. Tier A GBS 16, Tier B GBS 8, seq 4096, EP 8, MXFP8.
|
| 6 |
+
|
| 7 |
+
Executed per `NEXT_TOPK_AND_TIERB_PLAN.md`. **0 failed cells** across the whole
|
| 8 |
+
campaign; 36 nsys reports, every one carrying markers on all 8 GPUs.
|
| 9 |
+
|
| 10 |
+
## 1. What ran
|
| 11 |
+
|
| 12 |
+
| suite | scope | status |
|
| 13 |
+
| --- | --- | --- |
|
| 14 |
+
| Tier A MoonEP routing refresh | G1βG4 on the fixed probe branch | complete |
|
| 15 |
+
| Tier A top-k matrix | 9 cells Γ 3 backends Γ probe/timing/memory/trace | complete |
|
| 16 |
+
| Tier B new models | 3 of 4 Γ 3 backends Γ all four stages | complete |
|
| 17 |
+
| `QWEN3_NEXT_L16` | β | **deferred, see Β§5** |
|
| 18 |
+
|
| 19 |
+
## 2. Tier A: top-k separated from expert shape
|
| 20 |
+
|
| 21 |
+
Step time (ms) and peak memory (GiB), mean of 3 unprofiled repeats, CUDA graphs
|
| 22 |
+
on, zero token drops.
|
| 23 |
+
|
| 24 |
+
| cell | ECHO | UltraEP | MoonEP |
|
| 25 |
+
| --- | --- | --- | --- |
|
| 26 |
+
| G2_K2 | 270.9 / 132 | 335.3 / 99 | **254.4** / **86** |
|
| 27 |
+
| G3_K2 | 247.0 / 133 | 248.5 / 100 | **226.3** / **85** |
|
| 28 |
+
| G3_K8 | **352.6** / 138 | 361.3 / 104 | 361.5 / **90** |
|
| 29 |
+
| G4_K2 | 251.9 / 136 | **215.0** / 102 | 222.4 / **84** |
|
| 30 |
+
| G4_K4 | 279.6 / 138 | **247.5** / 103 | 259.6 / **85** |
|
| 31 |
+
| G4_K6 | 304.9 / 139 | **290.8** / 105 | 299.9 / **86** |
|
| 32 |
+
| G4_K8 | 335.2 / 139 | **315.9** / 104 | 335.9 / **88** |
|
| 33 |
+
| G4_K10 | 345.7 / 141 | **336.8** / 106 | 369.3 / **89** |
|
| 34 |
+
| G4_K16 | 385.8 / 146 | **378.8** / 111 | 457.4 / **91** |
|
| 35 |
+
|
| 36 |
+
### G4's cost is fanout, not small experts
|
| 37 |
+
|
| 38 |
+
This is what the campaign was built to settle, and the answer is unambiguous.
|
| 39 |
+
Holding G4's expert shape fixed (E=128, Hβ²=896) and varying only top-k:
|
| 40 |
+
|
| 41 |
+
| K | ECHO ms | vs K=2 |
|
| 42 |
+
| ---: | ---: | ---: |
|
| 43 |
+
| 2 | 251.9 | β |
|
| 44 |
+
| 4 | 279.6 | +11% |
|
| 45 |
+
| 8 | 335.2 | +33% |
|
| 46 |
+
| 16 | 385.8 | +53% |
|
| 47 |
+
| *32 (original diagonal)* | *486.7* | *+93%* |
|
| 48 |
+
|
| 49 |
+
The decisive comparison is **G4_K16 (385.8 ms) against the original G3 diagonal
|
| 50 |
+
(437.2 ms)**: G4_K16 has twice the experts at half the width, yet runs 12%
|
| 51 |
+
faster. Small experts are not what made G4 expensive. Its published 486.7 ms was
|
| 52 |
+
a consequence of K=32.
|
| 53 |
+
|
| 54 |
+
The curve is smooth and monotonic with no knee, so the optional compute-matched
|
| 55 |
+
fanout microbenchmark the plan holds in reserve is **not needed**.
|
| 56 |
+
|
| 57 |
+
### The K=2 cells show granularity alone is cheap, and non-monotonic for UltraEP
|
| 58 |
+
|
| 59 |
+
At fixed low fanout, going from 32 to 128 experts barely moves ECHO
|
| 60 |
+
(270.9 β 247.0 β 251.9) but moves UltraEP a great deal
|
| 61 |
+
(335.3 β 248.5 β **215.0**). UltraEP is the *worst* backend at G2_K2, by 24%,
|
| 62 |
+
and the *best* at G4_K2, by 15%. That crossover at constant fanout is invisible
|
| 63 |
+
on the original diagonal, where E and K move together.
|
| 64 |
+
|
| 65 |
+
### MoonEP crosses over at Kβ8
|
| 66 |
+
|
| 67 |
+
MoonEP is fastest at G2_K2 and G3_K2 and within 3% of UltraEP up to K=6, then
|
| 68 |
+
degrades: +21% versus UltraEP by K=16. Its weakness is therefore **fanout**, not
|
| 69 |
+
granularity β the same effect that produced its published G4 deficit at K=32.
|
| 70 |
+
|
| 71 |
+
Its async grad-reduce overlap stays healthy across the ladder (92.9β97.0% on 8
|
| 72 |
+
devices), so the degradation is not exposed reduce time.
|
| 73 |
+
|
| 74 |
+
### Memory: MoonEP is lightest everywhere
|
| 75 |
+
|
| 76 |
+
84β91 GiB against UltraEP's 99β111 and ECHO's 132β146 β roughly 50 GiB below
|
| 77 |
+
ECHO at every cell. This reproduces the Tier A/B ordering of the fixed
|
| 78 |
+
DDP-owned stack on an entirely new set of shapes.
|
| 79 |
+
|
| 80 |
+
### Capacity demand falls as fanout rises
|
| 81 |
+
|
| 82 |
+
Validated zero-drop capacity, ECHO: 1.203 at K=2 down to 1.031 at K=16,
|
| 83 |
+
monotonically. Higher top-k spreads each token over more experts, so per-expert
|
| 84 |
+
imbalance averages out. UltraEP stays flat at ~1.010 across nearly the whole
|
| 85 |
+
ladder β its balancer absorbs exactly the low-fanout imbalance ECHO leaves
|
| 86 |
+
exposed. Its one elevated point is G3_K2 (1.090).
|
| 87 |
+
|
| 88 |
+
MoonEP takes no capacity factor and is tagged `capnone` throughout.
|
| 89 |
+
|
| 90 |
+
## 3. Tier B: new real-model slices
|
| 91 |
+
|
| 92 |
+
| model | ECHO | UltraEP | MoonEP |
|
| 93 |
+
| --- | --- | --- | --- |
|
| 94 |
+
| `DEEPSEEK_V3_L4` | 198.5 ms / 128.0 GiB | 195.8 / 99.6 | **192.7** / **95.6** |
|
| 95 |
+
| `KIMI_K2_L2` | 119.4 / 129.1 | 110.1 / 88.3 | **108.6** / **84.5** |
|
| 96 |
+
| `OLMOE_1B_7B_L16` | 110.1 / 42.0 | **103.4** / 38.2 | 105.5 / **35.6** |
|
| 97 |
+
|
| 98 |
+
**MoonEP is lightest on all three and fastest on two.** That is a reversal of
|
| 99 |
+
the pre-fix stack, where it was slowest everywhere.
|
| 100 |
+
|
| 101 |
+
OLMoE is the exception on throughput, and the exception is consistent with the
|
| 102 |
+
Tier A finding: OLMoE has 64 experts over 8 ranks, so 8 slots per rank, against
|
| 103 |
+
Kimi's 48. With so few duplicated experts there is little for MoonEP's machinery
|
| 104 |
+
to exploit, and UltraEP edges it by 2%.
|
| 105 |
+
|
| 106 |
+
### The memory fallback fired exactly once, and correctly
|
| 107 |
+
|
| 108 |
+
| requested | selected | max timing peak | reason |
|
| 109 |
+
| --- | --- | ---: | --- |
|
| 110 |
+
| `DEEPSEEK_V3_L5` | **`DEEPSEEK_V3_L4`** | 128.0 GiB | L5 exceeded the ceiling on ECHO |
|
| 111 |
+
| `KIMI_K2_L2` | `KIMI_K2_L2` | 129.1 GiB | fits as requested |
|
| 112 |
+
| `OLMOE_1B_7B_L16` | `OLMOE_1B_7B_L16` | 42.0 GiB | fits as requested |
|
| 113 |
+
|
| 114 |
+
`DEEPSEEK_V3_L5` ran successfully on all three backends, but ECHO peaked at
|
| 115 |
+
**175.1 GiB**, above the 165 GiB promotion ceiling (UltraEP 138.9, MoonEP
|
| 116 |
+
137.1). This is a promotion-policy decision, not an OOM.
|
| 117 |
+
|
| 118 |
+
**The gate has to be on timing, not on the probe.** ECHO's graph-on timing peak
|
| 119 |
+
came in **17 GiB above** its graph-off probe peak (158.3 GiB), so probing alone
|
| 120 |
+
would have promoted a slice that does not fit. Dropping one layer cut ECHO's
|
| 121 |
+
peak by 47 GiB, which shows how completely the MoE layer dominates at
|
| 122 |
+
hidden=7168.
|
| 123 |
+
|
| 124 |
+
`OLMOE_1B_7B_L16` at 42.0 GiB sits **below** the desired 90β155 GiB matching
|
| 125 |
+
band. Per the plan it is retained with this note rather than treated as a
|
| 126 |
+
failure: the full 16-layer model simply does not fill an 8ΓB200 node, and
|
| 127 |
+
reducing it further was never the intent.
|
| 128 |
+
|
| 129 |
+
## 4. Interpretation constraints
|
| 130 |
+
|
| 131 |
+
**Across the K=2 cells, active expert FLOPs are not constant.** Routed expert
|
| 132 |
+
width is part of granularity, so a G2βG4 comparison at fixed K changes the work
|
| 133 |
+
as well as the shape. Report those rows alongside active FLOPs or per-routed-row
|
| 134 |
+
time, never as a pure compute-matched experiment.
|
| 135 |
+
|
| 136 |
+
**Tier B slices are reduced-depth prefixes**, and every model uses the same
|
| 137 |
+
GPT2BPE vocabulary and WikiText binary, TP=1, PP=1, EP=8, MXFP8, and a short
|
| 138 |
+
fixed horizon. The architecture is preserved; the data path is not the models'
|
| 139 |
+
native pretraining recipe and must not be described as such.
|
| 140 |
+
|
| 141 |
+
**MoonEP numbers are a post-fix baseline, not a delta.** The pre-fix stack
|
| 142 |
+
computed something different, so differences across the fix are not speedups.
|
| 143 |
+
|
| 144 |
+
## 5. `QWEN3_NEXT_L16` is deferred, not failed
|
| 145 |
+
|
| 146 |
+
`preflight_next_models.sh` fails it on **all nine branch-role combinations**
|
| 147 |
+
(timing, probe, memory Γ three backends). Ten model-defining flags are absent:
|
| 148 |
+
|
| 149 |
+
```
|
| 150 |
+
--attention-output-gate --no-weight-decay-cond-type
|
| 151 |
+
--moe-shared-expert-gate --linear-attention-type
|
| 152 |
+
--linear-attention-freq --linear-conv-kernel-dim
|
| 153 |
+
--linear-key-head-dim --linear-value-head-dim
|
| 154 |
+
--linear-num-key-heads --linear-num-value-heads
|
| 155 |
+
```
|
| 156 |
+
|
| 157 |
+
The integration branches predate Megatron's Gated DeltaNet and
|
| 158 |
+
attention-output-gate support. Per the plan these must be **ported or rebased,
|
| 159 |
+
never stripped to make the preflight pass** β removing them would measure a
|
| 160 |
+
different model while reporting the Qwen3-Next name. The other three models were
|
| 161 |
+
run meanwhile, which is what the plan prescribes.
|
| 162 |
+
|
| 163 |
+
## 6. Acceptance checklist
|
| 164 |
+
|
| 165 |
+
| check | result |
|
| 166 |
+
| --- | --- |
|
| 167 |
+
| intended timing/probe/memory branch per backend | yes, orchestrator-enforced |
|
| 168 |
+
| MoonEP on the fixed duplicated-gradient lineage | `moonep_backend_async_grad_reduce` |
|
| 169 |
+
| ECHO/UltraEP zero drops at the common capacity | yes |
|
| 170 |
+
| unprofiled timing on all three backends | yes, 3 repeats every cell |
|
| 171 |
+
| Tier B process HBM β€ 165 GiB on every backend | yes, after the L5βL4 fallback |
|
| 172 |
+
| allocator dumps complete | 57/60 cells; see below |
|
| 173 |
+
| nsys: all 8 GPUs + backend markers | 36/36 |
|
| 174 |
+
| MoonEP async overlap nonzero | Tier A 92.9β97.0%, Tier B 58.2β89.2% |
|
| 175 |
+
| no `DEFERRED_UNSUPPORTED`/`BADFLAG` reclassified as OOM | none |
|
| 176 |
+
|
| 177 |
+
### The three `INCOMPLETE_DUMP` rows are a checker artifact
|
| 178 |
+
|
| 179 |
+
`G2_K2/echo`, `G4_K16/echo` and `G4_K8/moonep` are flagged, and the flag is
|
| 180 |
+
**not** a loss of measurement data:
|
| 181 |
+
|
| 182 |
+
* all 8 ranks reported in every one of the 60 memory cells; only the iteration
|
| 183 |
+
count differs (4 or 6 captured against an expected 5), because `TIMING_ITERS`
|
| 184 |
+
differs between the graph-on and no-graph arms while the check is calibrated
|
| 185 |
+
to a single value;
|
| 186 |
+
* `pickle_count` is **0 in all 60 cells**, including every passing one β this
|
| 187 |
+
branch writes JSON snapshots only, so half the equality test can never be
|
| 188 |
+
satisfied and contributes nothing;
|
| 189 |
+
* all three flagged cells still produced valid peaks, which come from
|
| 190 |
+
`mem_samples.csv` rather than from the snapshots;
|
| 191 |
+
* all three are the `dynamic`/no-graph *reference* arms. Every primary
|
| 192 |
+
static-capacity graph-on cell, which carries the headline memory numbers,
|
| 193 |
+
passed `OK`.
|
| 194 |
+
|
| 195 |
+
Worth fixing in the checker: derive `expected_dumps` from the arm's own
|
| 196 |
+
`TIMING_ITERS`, and drop the pickle half of the test on JSON-only branches.
|
| 197 |
+
|
| 198 |
+
## 7. Campaign resilience
|
| 199 |
+
|
| 200 |
+
The pod was preempted mid-campaign, during the Tier A trace stage. Recovery cost
|
| 201 |
+
no measurement work: 29 completion markers across 10 sub-roots were restored
|
| 202 |
+
from the PVC mirror, the driver skipped every finished step, and it resumed
|
| 203 |
+
exactly at `ultraep__tierbtrace`. All results to that point were already on
|
| 204 |
+
HuggingFace.
|
| 205 |
+
|
| 206 |
+
Two harness defects were found and fixed during the campaign; both are recorded
|
| 207 |
+
in the driver repository.
|
| 208 |
+
|
| 209 |
+
**Traced cells burned 420 s each doing nothing.** The trace phase profiles steps
|
| 210 |
+
6β8 but requests 10 iterations, so every traced cell reached iteration 8, wrote
|
| 211 |
+
its complete report, then deadlocked in a collective. The guard waited its full
|
| 212 |
+
stall timeout for two iterations that could never arrive and would not have been
|
| 213 |
+
in the report. Callers now pass `GUARD_PROFILE_END_ITER` and a stall past that
|
| 214 |
+
point uses 60 s. Measured effect: 9.2 β 3.2 min per trace.
|
| 215 |
+
|
| 216 |
+
**`cp -a` cannot restore this PVC.** NFS rejects the chmod/chown it implies, so
|
| 217 |
+
cp aborts partway. Restoring the preempted campaign with `-a` copied 20 of 2433
|
| 218 |
+
files and 1 of 29 markers before failing β which would have silently re-run most
|
| 219 |
+
of the campaign despite the state being intact. Changed to `cp -r`.
|
| 220 |
+
|
| 221 |
+
## 8. Open items
|
| 222 |
+
|
| 223 |
+
* `QWEN3_NEXT_L16` needs its architecture support ported onto the nine
|
| 224 |
+
branch-roles before it can be measured.
|
| 225 |
+
* The `INCOMPLETE_DUMP` check should be calibrated per arm (Β§6).
|
| 226 |
+
* The compute-matched fanout microbenchmark remains unnecessary; the core matrix
|
| 227 |
+
explains the G4 curve.
|