File size: 7,078 Bytes
3230332 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 | # Tier B run history and errata — `tierb-b200-gbs8`
The HuggingFace repository accumulates every upload, and this campaign was
restarted several times on 2026-08-26 while real bugs in the capacity-selection
logic were found and fixed. Result files from those earlier attempts are still
present alongside the good ones.
**Only results whose ECHO `capacity_attempts.tsv` contains `measure` and
`validate` rows are valid.** Everything below explains how to tell them apart
and why the earlier ones are wrong.
Local result directories were overwritten in place by each rerun, so the local
tree holds only the newest data. HuggingFace does not overwrite, so it holds all
of it, which is why this file exists.
---
## How to tell a valid run from an invalid one
The capacity tag in a directory name (`__cap2p00`, `__cap1p25`) is **not**
sufficient. `cap2p00` means two different things depending on when it was
written: in the old algorithm it was the *selected timing capacity*, in the
current one it is only the *measurement point* used to observe peak demand.
Use these instead, in `capacity_attempts.tsv` / `capacity_selection.tsv`:
| signal | meaning |
|---|---|
| ECHO rows with `purpose=measure` **and** `purpose=validate` | current, valid |
| ECHO rows with `purpose=natural` | old selection, invalid |
| `required_capacity` exactly `1.0` for every model | fabricated, invalid |
| `required_capacity` like `1.0293`, varying per model | measured, valid |
| any cell tagged `__cap1p02` | crashed before iteration 0, invalid |
| any cell tagged `__cap1p75` | abandoned descent search, ignore |
---
## What was wrong, in the order it was found
### 1. UltraEP crashed on every model — unexecutable capacities (`__cap1p02`)
The selection rounded the measured requirement up to the next 0.01. UltraEP's
dispatch buffer is `int(capacity * seq * mbs * topk)` and the kernel asserts it
is a multiple of 128, so 1.02 produced 66846 tokens and every rank died before
iteration 0:
```
num_permuted_tokens (66846) must be a multiple of pad_multiple (128)
```
On this model matrix `seq*topk` is a large power-of-two multiple of 128, so one
0.01 capacity step is `128/25` of a boundary: only capacities that are multiples
of 0.25 are executable at all. The 0.01 rule could essentially never work here.
`__cap1p02` cells contain no measurement. They are crash evidence only.
### 2. QWEN2_57B lost the fp32 router it had always been measured with
Until commit `f39dedd`, `--moe-router-dtype fp32` was passed unconditionally.
That commit made it per-model and set QWEN2_57B to the ModelZoo-faithful empty
value, which UltraEP cannot run — it asserts on the probs dtype at
`ultra_ep.cpp:909`. Restored to fp32, which is also the configuration every
previously published QWEN2_57B number was taken under.
### 3. A MoonEP OOM halted the whole campaign
MoonEP does not fit QWEN3_235B or QWEN3_30B at GBS8. The Stage 7 gate treated
that identically to a crash and aborted the study for all three backends, which
contradicts the runbook's own rule that an OOM is data and only BADFLAG/ASSERT
are bugs. An OOM is now recorded as `DOES_NOT_FIT` for that one
(model, backend) pair and the campaign continues.
**This is a result, not an error.** Those two `DOES_NOT_FIT` rows are real
findings and should be reported as such.
### 4. ECHO's `required_capacity` was not a measurement
`summarise_echo` derived the requirement from `[MOE_DROP_DEBUG][setup_metadata]`,
which reports what the dispatcher *expects before routing*: its
`exact_recv_per_rank` is the perfectly uniform `[32768]*8`. The requirement came
out as exactly `1.0` for every model at every capacity — the uniform lower
bound, structurally unable to see the imbalance the probe exists to measure.
The observed data is in the `[dispatch]` rows (`actual_valid_recv`), the same
fields the older `summarize_token_drop_log.py` keyed on. Reading those gives
per-model requirements of **1.023–1.055** instead of a flat 1.0, and makes
ECHO's number directly comparable with UltraEP's `1.0101`: both are now
peak actual demand over the uniform share.
**Any result showing ECHO `required_capacity = 1.0` predates this fix.**
### 5. ECHO selected the highest capacity that fit, not the lowest that worked
The old loop stopped at the first rung that did not OOM, so 2.00 was chosen for
all four models on the first attempt and nothing lower was tried. Because
UltraEP is required to time at ECHO's capacity, that inflated the receive buffer
for **both** backends. ECHO now follows UltraEP's procedure — measure, round up
to an executable value, validate zero drops there — and the common timing
capacity is `max(ECHO, UltraEP)` per model.
Effect: the timing capacity is **1.25**, not 2.00.
### 6. Drop detection reported "zero drops" exactly when tokens were dropped
`observed_loss` is `expected_recv - actual_valid_recv`, not a drop count. When
the buffer genuinely saturates, `actual_valid_recv` is clipped at `budget`, so
`observed_loss` goes strongly negative and `max(0, observed_loss)` evaluates to
zero. Every earlier zero-drop verdict was therefore produced by a test that
could not detect the failure it was checking for.
It was masked because every capacity probed until then was heavily oversized.
It became load-bearing as soon as the selection started probing downward toward
the saturation point. Drops are now detected by buffer saturation
(`actual_valid_recv >= budget`), which survives clipping.
---
## What this means when reading the results
- **Selected capacity does not discriminate the backends.** ECHO needs ~1.03 and
UltraEP ~1.01, but the 128-alignment forces both onto the same 0.25 grid, so
both select 1.25. The meaningful comparison is the measured
`required_capacity`, not the selected one.
- **Both backends are over-provisioned by roughly 21%** at C=1.25 against a real
requirement near 1.03. That is a property of the kernel's alignment, not of
the routing, and memory results should say so rather than describing the
buffer as right-sized.
- **Probe peaks taken at different capacities are not comparable.** Early ECHO
numbers were measured at C=2.00 and UltraEP at C=1.25; the valid comparison is
the Stage 4 timing runs, which use one common capacity per model.
---
## Provenance
Every step writes `provenance.txt` next to its results with the driver, the
CUDA/NCCL/NVSHMEM versions and the commit of each repository. The fixes above
correspond to these commits on `granularity-base`:
```
e3f7c0b Tier B probe: pick capacities UltraEP can execute, and keep QWEN2_57B on fp32
4e4081f Tier B: an OOM excludes one model on one backend, not the whole campaign
Measure ECHO's real capacity requirement, and search for the lowest zero-drop rung
Select ECHO's capacity the way UltraEP does, and time both at max(ECHO, UltraEP)
Detect ECHO token drops by buffer saturation, not by observed_loss
```
A result predates a fix if its `provenance.txt` records a
`blackwell_run_scripts` commit older than the corresponding one above.
|