|
Download NEXT_CAMPAIGN_RESULTS.md from MoE-CUDA-Graph-Analysis/tierb-b200-gbs8: direct link, hf CLI and curl.
- Browser
- Download file 10.7 kB
-
https://huggingface.co/MoE-CUDA-Graph-Analysis/tierb-b200-gbs8/resolve/main/NEXT_CAMPAIGN_RESULTS.md
- Command line
-
hf download hf://MoE-CUDA-Graph-Analysis/tierb-b200-gbs8/NEXT_CAMPAIGN_RESULTS.md
-
curl -L -o NEXT_CAMPAIGN_RESULTS.md https://huggingface.co/MoE-CUDA-Graph-Analysis/tierb-b200-gbs8/resolve/main/NEXT_CAMPAIGN_RESULTS.md
10.7 kB
| # Next campaign results: Tier A top-k separation and new Tier B models | |
| Study root: `/workspace/granularity_next_b200` | |
| HF: `MoE-CUDA-Graph-Analysis/tierb-b200-gbs8` under `granularity_next_b200/` | |
| Hardware: 8ΓB200. Tier A GBS 16, Tier B GBS 8, seq 4096, EP 8, MXFP8. | |
| Executed per `NEXT_TOPK_AND_TIERB_PLAN.md`. **0 failed cells** across the whole | |
| campaign; 36 nsys reports, every one carrying markers on all 8 GPUs. | |
| ## 1. What ran | |
| | suite | scope | status | | |
| | --- | --- | --- | | |
| | Tier A MoonEP routing refresh | G1βG4 on the fixed probe branch | complete | | |
| | Tier A top-k matrix | 9 cells Γ 3 backends Γ probe/timing/memory/trace | complete | | |
| | Tier B new models | 3 of 4 Γ 3 backends Γ all four stages | complete | | |
| | `QWEN3_NEXT_L16` | β | **deferred, see Β§5** | | |
| ## 2. Tier A: top-k separated from expert shape | |
| Step time (ms) and peak memory (GiB), mean of 3 unprofiled repeats, CUDA graphs | |
| on, zero token drops. | |
| | cell | ECHO | UltraEP | MoonEP | | |
| | --- | --- | --- | --- | | |
| | G2_K2 | 270.9 / 132 | 335.3 / 99 | **254.4** / **86** | | |
| | G3_K2 | 247.0 / 133 | 248.5 / 100 | **226.3** / **85** | | |
| | G3_K8 | **352.6** / 138 | 361.3 / 104 | 361.5 / **90** | | |
| | G4_K2 | 251.9 / 136 | **215.0** / 102 | 222.4 / **84** | | |
| | G4_K4 | 279.6 / 138 | **247.5** / 103 | 259.6 / **85** | | |
| | G4_K6 | 304.9 / 139 | **290.8** / 105 | 299.9 / **86** | | |
| | G4_K8 | 335.2 / 139 | **315.9** / 104 | 335.9 / **88** | | |
| | G4_K10 | 345.7 / 141 | **336.8** / 106 | 369.3 / **89** | | |
| | G4_K16 | 385.8 / 146 | **378.8** / 111 | 457.4 / **91** | | |
| ### G4's cost is fanout, not small experts | |
| This is what the campaign was built to settle, and the answer is unambiguous. | |
| Holding G4's expert shape fixed (E=128, Hβ²=896) and varying only top-k: | |
| | K | ECHO ms | vs K=2 | | |
| | ---: | ---: | ---: | | |
| | 2 | 251.9 | β | | |
| | 4 | 279.6 | +11% | | |
| | 8 | 335.2 | +33% | | |
| | 16 | 385.8 | +53% | | |
| | *32 (original diagonal)* | *486.7* | *+93%* | | |
| The decisive comparison is **G4_K16 (385.8 ms) against the original G3 diagonal | |
| (437.2 ms)**: G4_K16 has twice the experts at half the width, yet runs 12% | |
| faster. Small experts are not what made G4 expensive. Its published 486.7 ms was | |
| a consequence of K=32. | |
| The curve is smooth and monotonic with no knee, so the optional compute-matched | |
| fanout microbenchmark the plan holds in reserve is **not needed**. | |
| ### The K=2 cells show granularity alone is cheap, and non-monotonic for UltraEP | |
| At fixed low fanout, going from 32 to 128 experts barely moves ECHO | |
| (270.9 β 247.0 β 251.9) but moves UltraEP a great deal | |
| (335.3 β 248.5 β **215.0**). UltraEP is the *worst* backend at G2_K2, by 24%, | |
| and the *best* at G4_K2, by 15%. That crossover at constant fanout is invisible | |
| on the original diagonal, where E and K move together. | |
| ### MoonEP crosses over at Kβ8 | |
| MoonEP is fastest at G2_K2 and G3_K2 and within 3% of UltraEP up to K=6, then | |
| degrades: +21% versus UltraEP by K=16. Its weakness is therefore **fanout**, not | |
| granularity β the same effect that produced its published G4 deficit at K=32. | |
| Its async grad-reduce overlap stays healthy across the ladder (92.9β97.0% on 8 | |
| devices), so the degradation is not exposed reduce time. | |
| ### Memory: MoonEP is lightest everywhere | |
| 84β91 GiB against UltraEP's 99β111 and ECHO's 132β146 β roughly 50 GiB below | |
| ECHO at every cell. This reproduces the Tier A/B ordering of the fixed | |
| DDP-owned stack on an entirely new set of shapes. | |
| ### Capacity demand falls as fanout rises | |
| Validated zero-drop capacity, ECHO: 1.203 at K=2 down to 1.031 at K=16, | |
| monotonically. Higher top-k spreads each token over more experts, so per-expert | |
| imbalance averages out. UltraEP stays flat at ~1.010 across nearly the whole | |
| ladder β its balancer absorbs exactly the low-fanout imbalance ECHO leaves | |
| exposed. Its one elevated point is G3_K2 (1.090). | |
| MoonEP takes no capacity factor and is tagged `capnone` throughout. | |
| ## 3. Tier B: new real-model slices | |
| | model | ECHO | UltraEP | MoonEP | | |
| | --- | --- | --- | --- | | |
| | `DEEPSEEK_V3_L4` | 198.5 ms / 128.0 GiB | 195.8 / 99.6 | **192.7** / **95.6** | | |
| | `KIMI_K2_L2` | 119.4 / 129.1 | 110.1 / 88.3 | **108.6** / **84.5** | | |
| | `OLMOE_1B_7B_L16` | 110.1 / 42.0 | **103.4** / 38.2 | 105.5 / **35.6** | | |
| **MoonEP is lightest on all three and fastest on two.** That is a reversal of | |
| the pre-fix stack, where it was slowest everywhere. | |
| OLMoE is the exception on throughput, and the exception is consistent with the | |
| Tier A finding: OLMoE has 64 experts over 8 ranks, so 8 slots per rank, against | |
| Kimi's 48. With so few duplicated experts there is little for MoonEP's machinery | |
| to exploit, and UltraEP edges it by 2%. | |
| ### The memory fallback fired exactly once, and correctly | |
| | requested | selected | max timing peak | reason | | |
| | --- | --- | ---: | --- | | |
| | `DEEPSEEK_V3_L5` | **`DEEPSEEK_V3_L4`** | 128.0 GiB | L5 exceeded the ceiling on ECHO | | |
| | `KIMI_K2_L2` | `KIMI_K2_L2` | 129.1 GiB | fits as requested | | |
| | `OLMOE_1B_7B_L16` | `OLMOE_1B_7B_L16` | 42.0 GiB | fits as requested | | |
| `DEEPSEEK_V3_L5` ran successfully on all three backends, but ECHO peaked at | |
| **175.1 GiB**, above the 165 GiB promotion ceiling (UltraEP 138.9, MoonEP | |
| 137.1). This is a promotion-policy decision, not an OOM. | |
| **The gate has to be on timing, not on the probe.** ECHO's graph-on timing peak | |
| came in **17 GiB above** its graph-off probe peak (158.3 GiB), so probing alone | |
| would have promoted a slice that does not fit. Dropping one layer cut ECHO's | |
| peak by 47 GiB, which shows how completely the MoE layer dominates at | |
| hidden=7168. | |
| `OLMOE_1B_7B_L16` at 42.0 GiB sits **below** the desired 90β155 GiB matching | |
| band. Per the plan it is retained with this note rather than treated as a | |
| failure: the full 16-layer model simply does not fill an 8ΓB200 node, and | |
| reducing it further was never the intent. | |
| ## 4. Interpretation constraints | |
| **Across the K=2 cells, active expert FLOPs are not constant.** Routed expert | |
| width is part of granularity, so a G2βG4 comparison at fixed K changes the work | |
| as well as the shape. Report those rows alongside active FLOPs or per-routed-row | |
| time, never as a pure compute-matched experiment. | |
| **Tier B slices are reduced-depth prefixes**, and every model uses the same | |
| GPT2BPE vocabulary and WikiText binary, TP=1, PP=1, EP=8, MXFP8, and a short | |
| fixed horizon. The architecture is preserved; the data path is not the models' | |
| native pretraining recipe and must not be described as such. | |
| **MoonEP numbers are a post-fix baseline, not a delta.** The pre-fix stack | |
| computed something different, so differences across the fix are not speedups. | |
| ## 5. `QWEN3_NEXT_L16` is deferred, not failed | |
| `preflight_next_models.sh` fails it on **all nine branch-role combinations** | |
| (timing, probe, memory Γ three backends). Ten model-defining flags are absent: | |
| ``` | |
| --attention-output-gate --no-weight-decay-cond-type | |
| --moe-shared-expert-gate --linear-attention-type | |
| --linear-attention-freq --linear-conv-kernel-dim | |
| --linear-key-head-dim --linear-value-head-dim | |
| --linear-num-key-heads --linear-num-value-heads | |
| ``` | |
| The integration branches predate Megatron's Gated DeltaNet and | |
| attention-output-gate support. Per the plan these must be **ported or rebased, | |
| never stripped to make the preflight pass** β removing them would measure a | |
| different model while reporting the Qwen3-Next name. The other three models were | |
| run meanwhile, which is what the plan prescribes. | |
| ## 6. Acceptance checklist | |
| | check | result | | |
| | --- | --- | | |
| | intended timing/probe/memory branch per backend | yes, orchestrator-enforced | | |
| | MoonEP on the fixed duplicated-gradient lineage | `moonep_backend_async_grad_reduce` | | |
| | ECHO/UltraEP zero drops at the common capacity | yes | | |
| | unprofiled timing on all three backends | yes, 3 repeats every cell | | |
| | Tier B process HBM β€ 165 GiB on every backend | yes, after the L5βL4 fallback | | |
| | allocator dumps complete | 57/60 cells; see below | | |
| | nsys: all 8 GPUs + backend markers | 36/36 | | |
| | MoonEP async overlap nonzero | Tier A 92.9β97.0%, Tier B 58.2β89.2% | | |
| | no `DEFERRED_UNSUPPORTED`/`BADFLAG` reclassified as OOM | none | | |
| ### The three `INCOMPLETE_DUMP` rows are a checker artifact | |
| `G2_K2/echo`, `G4_K16/echo` and `G4_K8/moonep` are flagged, and the flag is | |
| **not** a loss of measurement data: | |
| * all 8 ranks reported in every one of the 60 memory cells; only the iteration | |
| count differs (4 or 6 captured against an expected 5), because `TIMING_ITERS` | |
| differs between the graph-on and no-graph arms while the check is calibrated | |
| to a single value; | |
| * `pickle_count` is **0 in all 60 cells**, including every passing one β this | |
| branch writes JSON snapshots only, so half the equality test can never be | |
| satisfied and contributes nothing; | |
| * all three flagged cells still produced valid peaks, which come from | |
| `mem_samples.csv` rather than from the snapshots; | |
| * all three are the `dynamic`/no-graph *reference* arms. Every primary | |
| static-capacity graph-on cell, which carries the headline memory numbers, | |
| passed `OK`. | |
| Worth fixing in the checker: derive `expected_dumps` from the arm's own | |
| `TIMING_ITERS`, and drop the pickle half of the test on JSON-only branches. | |
| ## 7. Campaign resilience | |
| The pod was preempted mid-campaign, during the Tier A trace stage. Recovery cost | |
| no measurement work: 29 completion markers across 10 sub-roots were restored | |
| from the PVC mirror, the driver skipped every finished step, and it resumed | |
| exactly at `ultraep__tierbtrace`. All results to that point were already on | |
| HuggingFace. | |
| Two harness defects were found and fixed during the campaign; both are recorded | |
| in the driver repository. | |
| **Traced cells burned 420 s each doing nothing.** The trace phase profiles steps | |
| 6β8 but requests 10 iterations, so every traced cell reached iteration 8, wrote | |
| its complete report, then deadlocked in a collective. The guard waited its full | |
| stall timeout for two iterations that could never arrive and would not have been | |
| in the report. Callers now pass `GUARD_PROFILE_END_ITER` and a stall past that | |
| point uses 60 s. Measured effect: 9.2 β 3.2 min per trace. | |
| **`cp -a` cannot restore this PVC.** NFS rejects the chmod/chown it implies, so | |
| cp aborts partway. Restoring the preempted campaign with `-a` copied 20 of 2433 | |
| files and 1 of 29 markers before failing β which would have silently re-run most | |
| of the campaign despite the state being intact. Changed to `cp -r`. | |
| ## 8. Open items | |
| * `QWEN3_NEXT_L16` needs its architecture support ported onto the nine | |
| branch-roles before it can be measured. | |
| * The `INCOMPLETE_DUMP` check should be calibrated per arm (Β§6). | |
| * The compute-matched fanout microbenchmark remains unnecessary; the core matrix | |
| explains the G4 curve. | |