# Tier B run history and errata — `tierb-b200-gbs8` The HuggingFace repository accumulates every upload, and this campaign was restarted several times on 2026-08-26 while real bugs in the capacity-selection logic were found and fixed. Result files from those earlier attempts are still present alongside the good ones. **Only results whose ECHO `capacity_attempts.tsv` contains `measure` and `validate` rows are valid.** Everything below explains how to tell them apart and why the earlier ones are wrong. Local result directories were overwritten in place by each rerun, so the local tree holds only the newest data. HuggingFace does not overwrite, so it holds all of it, which is why this file exists. --- ## How to tell a valid run from an invalid one The capacity tag in a directory name (`__cap2p00`, `__cap1p25`) is **not** sufficient. `cap2p00` means two different things depending on when it was written: in the old algorithm it was the *selected timing capacity*, in the current one it is only the *measurement point* used to observe peak demand. Use these instead, in `capacity_attempts.tsv` / `capacity_selection.tsv`: | signal | meaning | |---|---| | ECHO rows with `purpose=measure` **and** `purpose=validate` | current, valid | | ECHO rows with `purpose=natural` | old selection, invalid | | `required_capacity` exactly `1.0` for every model | fabricated, invalid | | `required_capacity` like `1.0293`, varying per model | measured, valid | | any cell tagged `__cap1p02` | crashed before iteration 0, invalid | | any cell tagged `__cap1p75` | abandoned descent search, ignore | --- ## What was wrong, in the order it was found ### 1. UltraEP crashed on every model — unexecutable capacities (`__cap1p02`) The selection rounded the measured requirement up to the next 0.01. UltraEP's dispatch buffer is `int(capacity * seq * mbs * topk)` and the kernel asserts it is a multiple of 128, so 1.02 produced 66846 tokens and every rank died before iteration 0: ``` num_permuted_tokens (66846) must be a multiple of pad_multiple (128) ``` On this model matrix `seq*topk` is a large power-of-two multiple of 128, so one 0.01 capacity step is `128/25` of a boundary: only capacities that are multiples of 0.25 are executable at all. The 0.01 rule could essentially never work here. `__cap1p02` cells contain no measurement. They are crash evidence only. ### 2. QWEN2_57B lost the fp32 router it had always been measured with Until commit `f39dedd`, `--moe-router-dtype fp32` was passed unconditionally. That commit made it per-model and set QWEN2_57B to the ModelZoo-faithful empty value, which UltraEP cannot run — it asserts on the probs dtype at `ultra_ep.cpp:909`. Restored to fp32, which is also the configuration every previously published QWEN2_57B number was taken under. ### 3. A MoonEP OOM halted the whole campaign MoonEP does not fit QWEN3_235B or QWEN3_30B at GBS8. The Stage 7 gate treated that identically to a crash and aborted the study for all three backends, which contradicts the runbook's own rule that an OOM is data and only BADFLAG/ASSERT are bugs. An OOM is now recorded as `DOES_NOT_FIT` for that one (model, backend) pair and the campaign continues. **This is a result, not an error.** Those two `DOES_NOT_FIT` rows are real findings and should be reported as such. ### 4. ECHO's `required_capacity` was not a measurement `summarise_echo` derived the requirement from `[MOE_DROP_DEBUG][setup_metadata]`, which reports what the dispatcher *expects before routing*: its `exact_recv_per_rank` is the perfectly uniform `[32768]*8`. The requirement came out as exactly `1.0` for every model at every capacity — the uniform lower bound, structurally unable to see the imbalance the probe exists to measure. The observed data is in the `[dispatch]` rows (`actual_valid_recv`), the same fields the older `summarize_token_drop_log.py` keyed on. Reading those gives per-model requirements of **1.023–1.055** instead of a flat 1.0, and makes ECHO's number directly comparable with UltraEP's `1.0101`: both are now peak actual demand over the uniform share. **Any result showing ECHO `required_capacity = 1.0` predates this fix.** ### 5. ECHO selected the highest capacity that fit, not the lowest that worked The old loop stopped at the first rung that did not OOM, so 2.00 was chosen for all four models on the first attempt and nothing lower was tried. Because UltraEP is required to time at ECHO's capacity, that inflated the receive buffer for **both** backends. ECHO now follows UltraEP's procedure — measure, round up to an executable value, validate zero drops there — and the common timing capacity is `max(ECHO, UltraEP)` per model. Effect: the timing capacity is **1.25**, not 2.00. ### 6. Drop detection reported "zero drops" exactly when tokens were dropped `observed_loss` is `expected_recv - actual_valid_recv`, not a drop count. When the buffer genuinely saturates, `actual_valid_recv` is clipped at `budget`, so `observed_loss` goes strongly negative and `max(0, observed_loss)` evaluates to zero. Every earlier zero-drop verdict was therefore produced by a test that could not detect the failure it was checking for. It was masked because every capacity probed until then was heavily oversized. It became load-bearing as soon as the selection started probing downward toward the saturation point. Drops are now detected by buffer saturation (`actual_valid_recv >= budget`), which survives clipping. --- ## What this means when reading the results - **Selected capacity does not discriminate the backends.** ECHO needs ~1.03 and UltraEP ~1.01, but the 128-alignment forces both onto the same 0.25 grid, so both select 1.25. The meaningful comparison is the measured `required_capacity`, not the selected one. - **Both backends are over-provisioned by roughly 21%** at C=1.25 against a real requirement near 1.03. That is a property of the kernel's alignment, not of the routing, and memory results should say so rather than describing the buffer as right-sized. - **Probe peaks taken at different capacities are not comparable.** Early ECHO numbers were measured at C=2.00 and UltraEP at C=1.25; the valid comparison is the Stage 4 timing runs, which use one common capacity per model. --- ## Provenance Every step writes `provenance.txt` next to its results with the driver, the CUDA/NCCL/NVSHMEM versions and the commit of each repository. The fixes above correspond to these commits on `granularity-base`: ``` e3f7c0b Tier B probe: pick capacities UltraEP can execute, and keep QWEN2_57B on fp32 4e4081f Tier B: an OOM excludes one model on one backend, not the whole campaign Measure ECHO's real capacity requirement, and search for the lowest zero-drop rung Select ECHO's capacity the way UltraEP does, and time both at max(ECHO, UltraEP) Detect ECHO token drops by buffer saturation, not by observed_loss ``` A result predates a fix if its `provenance.txt` records a `blackwell_run_scripts` commit older than the corresponding one above.