Errata: which Tier B runs are valid and why
Browse files- TIERB_RUN_HISTORY.md +149 -0
TIERB_RUN_HISTORY.md
ADDED
|
@@ -0,0 +1,149 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
# Tier B run history and errata — `tierb-b200-gbs8`
|
| 2 |
+
|
| 3 |
+
The HuggingFace repository accumulates every upload, and this campaign was
|
| 4 |
+
restarted several times on 2026-08-26 while real bugs in the capacity-selection
|
| 5 |
+
logic were found and fixed. Result files from those earlier attempts are still
|
| 6 |
+
present alongside the good ones.
|
| 7 |
+
|
| 8 |
+
**Only results whose ECHO `capacity_attempts.tsv` contains `measure` and
|
| 9 |
+
`validate` rows are valid.** Everything below explains how to tell them apart
|
| 10 |
+
and why the earlier ones are wrong.
|
| 11 |
+
|
| 12 |
+
Local result directories were overwritten in place by each rerun, so the local
|
| 13 |
+
tree holds only the newest data. HuggingFace does not overwrite, so it holds all
|
| 14 |
+
of it, which is why this file exists.
|
| 15 |
+
|
| 16 |
+
---
|
| 17 |
+
|
| 18 |
+
## How to tell a valid run from an invalid one
|
| 19 |
+
|
| 20 |
+
The capacity tag in a directory name (`__cap2p00`, `__cap1p25`) is **not**
|
| 21 |
+
sufficient. `cap2p00` means two different things depending on when it was
|
| 22 |
+
written: in the old algorithm it was the *selected timing capacity*, in the
|
| 23 |
+
current one it is only the *measurement point* used to observe peak demand.
|
| 24 |
+
|
| 25 |
+
Use these instead, in `capacity_attempts.tsv` / `capacity_selection.tsv`:
|
| 26 |
+
|
| 27 |
+
| signal | meaning |
|
| 28 |
+
|---|---|
|
| 29 |
+
| ECHO rows with `purpose=measure` **and** `purpose=validate` | current, valid |
|
| 30 |
+
| ECHO rows with `purpose=natural` | old selection, invalid |
|
| 31 |
+
| `required_capacity` exactly `1.0` for every model | fabricated, invalid |
|
| 32 |
+
| `required_capacity` like `1.0293`, varying per model | measured, valid |
|
| 33 |
+
| any cell tagged `__cap1p02` | crashed before iteration 0, invalid |
|
| 34 |
+
| any cell tagged `__cap1p75` | abandoned descent search, ignore |
|
| 35 |
+
|
| 36 |
+
---
|
| 37 |
+
|
| 38 |
+
## What was wrong, in the order it was found
|
| 39 |
+
|
| 40 |
+
### 1. UltraEP crashed on every model — unexecutable capacities (`__cap1p02`)
|
| 41 |
+
|
| 42 |
+
The selection rounded the measured requirement up to the next 0.01. UltraEP's
|
| 43 |
+
dispatch buffer is `int(capacity * seq * mbs * topk)` and the kernel asserts it
|
| 44 |
+
is a multiple of 128, so 1.02 produced 66846 tokens and every rank died before
|
| 45 |
+
iteration 0:
|
| 46 |
+
|
| 47 |
+
```
|
| 48 |
+
num_permuted_tokens (66846) must be a multiple of pad_multiple (128)
|
| 49 |
+
```
|
| 50 |
+
|
| 51 |
+
On this model matrix `seq*topk` is a large power-of-two multiple of 128, so one
|
| 52 |
+
0.01 capacity step is `128/25` of a boundary: only capacities that are multiples
|
| 53 |
+
of 0.25 are executable at all. The 0.01 rule could essentially never work here.
|
| 54 |
+
|
| 55 |
+
`__cap1p02` cells contain no measurement. They are crash evidence only.
|
| 56 |
+
|
| 57 |
+
### 2. QWEN2_57B lost the fp32 router it had always been measured with
|
| 58 |
+
|
| 59 |
+
Until commit `f39dedd`, `--moe-router-dtype fp32` was passed unconditionally.
|
| 60 |
+
That commit made it per-model and set QWEN2_57B to the ModelZoo-faithful empty
|
| 61 |
+
value, which UltraEP cannot run — it asserts on the probs dtype at
|
| 62 |
+
`ultra_ep.cpp:909`. Restored to fp32, which is also the configuration every
|
| 63 |
+
previously published QWEN2_57B number was taken under.
|
| 64 |
+
|
| 65 |
+
### 3. A MoonEP OOM halted the whole campaign
|
| 66 |
+
|
| 67 |
+
MoonEP does not fit QWEN3_235B or QWEN3_30B at GBS8. The Stage 7 gate treated
|
| 68 |
+
that identically to a crash and aborted the study for all three backends, which
|
| 69 |
+
contradicts the runbook's own rule that an OOM is data and only BADFLAG/ASSERT
|
| 70 |
+
are bugs. An OOM is now recorded as `DOES_NOT_FIT` for that one
|
| 71 |
+
(model, backend) pair and the campaign continues.
|
| 72 |
+
|
| 73 |
+
**This is a result, not an error.** Those two `DOES_NOT_FIT` rows are real
|
| 74 |
+
findings and should be reported as such.
|
| 75 |
+
|
| 76 |
+
### 4. ECHO's `required_capacity` was not a measurement
|
| 77 |
+
|
| 78 |
+
`summarise_echo` derived the requirement from `[MOE_DROP_DEBUG][setup_metadata]`,
|
| 79 |
+
which reports what the dispatcher *expects before routing*: its
|
| 80 |
+
`exact_recv_per_rank` is the perfectly uniform `[32768]*8`. The requirement came
|
| 81 |
+
out as exactly `1.0` for every model at every capacity — the uniform lower
|
| 82 |
+
bound, structurally unable to see the imbalance the probe exists to measure.
|
| 83 |
+
|
| 84 |
+
The observed data is in the `[dispatch]` rows (`actual_valid_recv`), the same
|
| 85 |
+
fields the older `summarize_token_drop_log.py` keyed on. Reading those gives
|
| 86 |
+
per-model requirements of **1.023–1.055** instead of a flat 1.0, and makes
|
| 87 |
+
ECHO's number directly comparable with UltraEP's `1.0101`: both are now
|
| 88 |
+
peak actual demand over the uniform share.
|
| 89 |
+
|
| 90 |
+
**Any result showing ECHO `required_capacity = 1.0` predates this fix.**
|
| 91 |
+
|
| 92 |
+
### 5. ECHO selected the highest capacity that fit, not the lowest that worked
|
| 93 |
+
|
| 94 |
+
The old loop stopped at the first rung that did not OOM, so 2.00 was chosen for
|
| 95 |
+
all four models on the first attempt and nothing lower was tried. Because
|
| 96 |
+
UltraEP is required to time at ECHO's capacity, that inflated the receive buffer
|
| 97 |
+
for **both** backends. ECHO now follows UltraEP's procedure — measure, round up
|
| 98 |
+
to an executable value, validate zero drops there — and the common timing
|
| 99 |
+
capacity is `max(ECHO, UltraEP)` per model.
|
| 100 |
+
|
| 101 |
+
Effect: the timing capacity is **1.25**, not 2.00.
|
| 102 |
+
|
| 103 |
+
### 6. Drop detection reported "zero drops" exactly when tokens were dropped
|
| 104 |
+
|
| 105 |
+
`observed_loss` is `expected_recv - actual_valid_recv`, not a drop count. When
|
| 106 |
+
the buffer genuinely saturates, `actual_valid_recv` is clipped at `budget`, so
|
| 107 |
+
`observed_loss` goes strongly negative and `max(0, observed_loss)` evaluates to
|
| 108 |
+
zero. Every earlier zero-drop verdict was therefore produced by a test that
|
| 109 |
+
could not detect the failure it was checking for.
|
| 110 |
+
|
| 111 |
+
It was masked because every capacity probed until then was heavily oversized.
|
| 112 |
+
It became load-bearing as soon as the selection started probing downward toward
|
| 113 |
+
the saturation point. Drops are now detected by buffer saturation
|
| 114 |
+
(`actual_valid_recv >= budget`), which survives clipping.
|
| 115 |
+
|
| 116 |
+
---
|
| 117 |
+
|
| 118 |
+
## What this means when reading the results
|
| 119 |
+
|
| 120 |
+
- **Selected capacity does not discriminate the backends.** ECHO needs ~1.03 and
|
| 121 |
+
UltraEP ~1.01, but the 128-alignment forces both onto the same 0.25 grid, so
|
| 122 |
+
both select 1.25. The meaningful comparison is the measured
|
| 123 |
+
`required_capacity`, not the selected one.
|
| 124 |
+
- **Both backends are over-provisioned by roughly 21%** at C=1.25 against a real
|
| 125 |
+
requirement near 1.03. That is a property of the kernel's alignment, not of
|
| 126 |
+
the routing, and memory results should say so rather than describing the
|
| 127 |
+
buffer as right-sized.
|
| 128 |
+
- **Probe peaks taken at different capacities are not comparable.** Early ECHO
|
| 129 |
+
numbers were measured at C=2.00 and UltraEP at C=1.25; the valid comparison is
|
| 130 |
+
the Stage 4 timing runs, which use one common capacity per model.
|
| 131 |
+
|
| 132 |
+
---
|
| 133 |
+
|
| 134 |
+
## Provenance
|
| 135 |
+
|
| 136 |
+
Every step writes `provenance.txt` next to its results with the driver, the
|
| 137 |
+
CUDA/NCCL/NVSHMEM versions and the commit of each repository. The fixes above
|
| 138 |
+
correspond to these commits on `granularity-base`:
|
| 139 |
+
|
| 140 |
+
```
|
| 141 |
+
e3f7c0b Tier B probe: pick capacities UltraEP can execute, and keep QWEN2_57B on fp32
|
| 142 |
+
4e4081f Tier B: an OOM excludes one model on one backend, not the whole campaign
|
| 143 |
+
Measure ECHO's real capacity requirement, and search for the lowest zero-drop rung
|
| 144 |
+
Select ECHO's capacity the way UltraEP does, and time both at max(ECHO, UltraEP)
|
| 145 |
+
Detect ECHO token drops by buffer saturation, not by observed_loss
|
| 146 |
+
```
|
| 147 |
+
|
| 148 |
+
A result predates a fix if its `provenance.txt` records a
|
| 149 |
+
`blackwell_run_scripts` commit older than the corresponding one above.
|