tierb-b200-gbs8 / NEXT_CAMPAIGN_RESULTS.md
AbhishekGhosh1998's picture
Upload NEXT_CAMPAIGN_RESULTS.md with huggingface_hub
c6d077b verified
|
Raw History Blame Contribute Delete
10.7 kB

Next campaign results: Tier A top-k separation and new Tier B models

Study root: /workspace/granularity_next_b200 HF: MoE-CUDA-Graph-Analysis/tierb-b200-gbs8 under granularity_next_b200/ Hardware: 8Γ—B200. Tier A GBS 16, Tier B GBS 8, seq 4096, EP 8, MXFP8.

Executed per NEXT_TOPK_AND_TIERB_PLAN.md. 0 failed cells across the whole campaign; 36 nsys reports, every one carrying markers on all 8 GPUs.

1. What ran

suite scope status
Tier A MoonEP routing refresh G1–G4 on the fixed probe branch complete
Tier A top-k matrix 9 cells Γ— 3 backends Γ— probe/timing/memory/trace complete
Tier B new models 3 of 4 Γ— 3 backends Γ— all four stages complete
QWEN3_NEXT_L16 β€” deferred, see Β§5

2. Tier A: top-k separated from expert shape

Step time (ms) and peak memory (GiB), mean of 3 unprofiled repeats, CUDA graphs on, zero token drops.

cell ECHO UltraEP MoonEP
G2_K2 270.9 / 132 335.3 / 99 254.4 / 86
G3_K2 247.0 / 133 248.5 / 100 226.3 / 85
G3_K8 352.6 / 138 361.3 / 104 361.5 / 90
G4_K2 251.9 / 136 215.0 / 102 222.4 / 84
G4_K4 279.6 / 138 247.5 / 103 259.6 / 85
G4_K6 304.9 / 139 290.8 / 105 299.9 / 86
G4_K8 335.2 / 139 315.9 / 104 335.9 / 88
G4_K10 345.7 / 141 336.8 / 106 369.3 / 89
G4_K16 385.8 / 146 378.8 / 111 457.4 / 91

G4's cost is fanout, not small experts

This is what the campaign was built to settle, and the answer is unambiguous. Holding G4's expert shape fixed (E=128, Hβ€²=896) and varying only top-k:

K ECHO ms vs K=2
2 251.9 β€”
4 279.6 +11%
8 335.2 +33%
16 385.8 +53%
32 (original diagonal) 486.7 +93%

The decisive comparison is G4_K16 (385.8 ms) against the original G3 diagonal (437.2 ms): G4_K16 has twice the experts at half the width, yet runs 12% faster. Small experts are not what made G4 expensive. Its published 486.7 ms was a consequence of K=32.

The curve is smooth and monotonic with no knee, so the optional compute-matched fanout microbenchmark the plan holds in reserve is not needed.

The K=2 cells show granularity alone is cheap, and non-monotonic for UltraEP

At fixed low fanout, going from 32 to 128 experts barely moves ECHO (270.9 β†’ 247.0 β†’ 251.9) but moves UltraEP a great deal (335.3 β†’ 248.5 β†’ 215.0). UltraEP is the worst backend at G2_K2, by 24%, and the best at G4_K2, by 15%. That crossover at constant fanout is invisible on the original diagonal, where E and K move together.

MoonEP crosses over at Kβ‰ˆ8

MoonEP is fastest at G2_K2 and G3_K2 and within 3% of UltraEP up to K=6, then degrades: +21% versus UltraEP by K=16. Its weakness is therefore fanout, not granularity β€” the same effect that produced its published G4 deficit at K=32.

Its async grad-reduce overlap stays healthy across the ladder (92.9–97.0% on 8 devices), so the degradation is not exposed reduce time.

Memory: MoonEP is lightest everywhere

84–91 GiB against UltraEP's 99–111 and ECHO's 132–146 β€” roughly 50 GiB below ECHO at every cell. This reproduces the Tier A/B ordering of the fixed DDP-owned stack on an entirely new set of shapes.

Capacity demand falls as fanout rises

Validated zero-drop capacity, ECHO: 1.203 at K=2 down to 1.031 at K=16, monotonically. Higher top-k spreads each token over more experts, so per-expert imbalance averages out. UltraEP stays flat at ~1.010 across nearly the whole ladder β€” its balancer absorbs exactly the low-fanout imbalance ECHO leaves exposed. Its one elevated point is G3_K2 (1.090).

MoonEP takes no capacity factor and is tagged capnone throughout.

3. Tier B: new real-model slices

model ECHO UltraEP MoonEP
DEEPSEEK_V3_L4 198.5 ms / 128.0 GiB 195.8 / 99.6 192.7 / 95.6
KIMI_K2_L2 119.4 / 129.1 110.1 / 88.3 108.6 / 84.5
OLMOE_1B_7B_L16 110.1 / 42.0 103.4 / 38.2 105.5 / 35.6

MoonEP is lightest on all three and fastest on two. That is a reversal of the pre-fix stack, where it was slowest everywhere.

OLMoE is the exception on throughput, and the exception is consistent with the Tier A finding: OLMoE has 64 experts over 8 ranks, so 8 slots per rank, against Kimi's 48. With so few duplicated experts there is little for MoonEP's machinery to exploit, and UltraEP edges it by 2%.

The memory fallback fired exactly once, and correctly

requested selected max timing peak reason
DEEPSEEK_V3_L5 DEEPSEEK_V3_L4 128.0 GiB L5 exceeded the ceiling on ECHO
KIMI_K2_L2 KIMI_K2_L2 129.1 GiB fits as requested
OLMOE_1B_7B_L16 OLMOE_1B_7B_L16 42.0 GiB fits as requested

DEEPSEEK_V3_L5 ran successfully on all three backends, but ECHO peaked at 175.1 GiB, above the 165 GiB promotion ceiling (UltraEP 138.9, MoonEP 137.1). This is a promotion-policy decision, not an OOM.

The gate has to be on timing, not on the probe. ECHO's graph-on timing peak came in 17 GiB above its graph-off probe peak (158.3 GiB), so probing alone would have promoted a slice that does not fit. Dropping one layer cut ECHO's peak by 47 GiB, which shows how completely the MoE layer dominates at hidden=7168.

OLMOE_1B_7B_L16 at 42.0 GiB sits below the desired 90–155 GiB matching band. Per the plan it is retained with this note rather than treated as a failure: the full 16-layer model simply does not fill an 8Γ—B200 node, and reducing it further was never the intent.

4. Interpretation constraints

Across the K=2 cells, active expert FLOPs are not constant. Routed expert width is part of granularity, so a G2β†’G4 comparison at fixed K changes the work as well as the shape. Report those rows alongside active FLOPs or per-routed-row time, never as a pure compute-matched experiment.

Tier B slices are reduced-depth prefixes, and every model uses the same GPT2BPE vocabulary and WikiText binary, TP=1, PP=1, EP=8, MXFP8, and a short fixed horizon. The architecture is preserved; the data path is not the models' native pretraining recipe and must not be described as such.

MoonEP numbers are a post-fix baseline, not a delta. The pre-fix stack computed something different, so differences across the fix are not speedups.

5. QWEN3_NEXT_L16 is deferred, not failed

preflight_next_models.sh fails it on all nine branch-role combinations (timing, probe, memory Γ— three backends). Ten model-defining flags are absent:

--attention-output-gate      --no-weight-decay-cond-type
--moe-shared-expert-gate     --linear-attention-type
--linear-attention-freq      --linear-conv-kernel-dim
--linear-key-head-dim        --linear-value-head-dim
--linear-num-key-heads       --linear-num-value-heads

The integration branches predate Megatron's Gated DeltaNet and attention-output-gate support. Per the plan these must be ported or rebased, never stripped to make the preflight pass β€” removing them would measure a different model while reporting the Qwen3-Next name. The other three models were run meanwhile, which is what the plan prescribes.

6. Acceptance checklist

check result
intended timing/probe/memory branch per backend yes, orchestrator-enforced
MoonEP on the fixed duplicated-gradient lineage moonep_backend_async_grad_reduce
ECHO/UltraEP zero drops at the common capacity yes
unprofiled timing on all three backends yes, 3 repeats every cell
Tier B process HBM ≀ 165 GiB on every backend yes, after the L5β†’L4 fallback
allocator dumps complete 57/60 cells; see below
nsys: all 8 GPUs + backend markers 36/36
MoonEP async overlap nonzero Tier A 92.9–97.0%, Tier B 58.2–89.2%
no DEFERRED_UNSUPPORTED/BADFLAG reclassified as OOM none

The three INCOMPLETE_DUMP rows are a checker artifact

G2_K2/echo, G4_K16/echo and G4_K8/moonep are flagged, and the flag is not a loss of measurement data:

  • all 8 ranks reported in every one of the 60 memory cells; only the iteration count differs (4 or 6 captured against an expected 5), because TIMING_ITERS differs between the graph-on and no-graph arms while the check is calibrated to a single value;
  • pickle_count is 0 in all 60 cells, including every passing one β€” this branch writes JSON snapshots only, so half the equality test can never be satisfied and contributes nothing;
  • all three flagged cells still produced valid peaks, which come from mem_samples.csv rather than from the snapshots;
  • all three are the dynamic/no-graph reference arms. Every primary static-capacity graph-on cell, which carries the headline memory numbers, passed OK.

Worth fixing in the checker: derive expected_dumps from the arm's own TIMING_ITERS, and drop the pickle half of the test on JSON-only branches.

7. Campaign resilience

The pod was preempted mid-campaign, during the Tier A trace stage. Recovery cost no measurement work: 29 completion markers across 10 sub-roots were restored from the PVC mirror, the driver skipped every finished step, and it resumed exactly at ultraep__tierbtrace. All results to that point were already on HuggingFace.

Two harness defects were found and fixed during the campaign; both are recorded in the driver repository.

Traced cells burned 420 s each doing nothing. The trace phase profiles steps 6–8 but requests 10 iterations, so every traced cell reached iteration 8, wrote its complete report, then deadlocked in a collective. The guard waited its full stall timeout for two iterations that could never arrive and would not have been in the report. Callers now pass GUARD_PROFILE_END_ITER and a stall past that point uses 60 s. Measured effect: 9.2 β†’ 3.2 min per trace.

cp -a cannot restore this PVC. NFS rejects the chmod/chown it implies, so cp aborts partway. Restoring the preempted campaign with -a copied 20 of 2433 files and 1 of 29 markers before failing β€” which would have silently re-run most of the campaign despite the state being intact. Changed to cp -r.

8. Open items

  • QWEN3_NEXT_L16 needs its architecture support ported onto the nine branch-roles before it can be measured.
  • The INCOMPLETE_DUMP check should be calibrated per arm (Β§6).
  • The compute-matched fanout microbenchmark remains unnecessary; the core matrix explains the G4 curve.