File size: 7,078 Bytes
3230332
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
# Tier B run history and errata — `tierb-b200-gbs8`

The HuggingFace repository accumulates every upload, and this campaign was
restarted several times on 2026-08-26 while real bugs in the capacity-selection
logic were found and fixed. Result files from those earlier attempts are still
present alongside the good ones.

**Only results whose ECHO `capacity_attempts.tsv` contains `measure` and
`validate` rows are valid.** Everything below explains how to tell them apart
and why the earlier ones are wrong.

Local result directories were overwritten in place by each rerun, so the local
tree holds only the newest data. HuggingFace does not overwrite, so it holds all
of it, which is why this file exists.

---

## How to tell a valid run from an invalid one

The capacity tag in a directory name (`__cap2p00`, `__cap1p25`) is **not**
sufficient. `cap2p00` means two different things depending on when it was
written: in the old algorithm it was the *selected timing capacity*, in the
current one it is only the *measurement point* used to observe peak demand.

Use these instead, in `capacity_attempts.tsv` / `capacity_selection.tsv`:

| signal | meaning |
|---|---|
| ECHO rows with `purpose=measure` **and** `purpose=validate` | current, valid |
| ECHO rows with `purpose=natural` | old selection, invalid |
| `required_capacity` exactly `1.0` for every model | fabricated, invalid |
| `required_capacity` like `1.0293`, varying per model | measured, valid |
| any cell tagged `__cap1p02` | crashed before iteration 0, invalid |
| any cell tagged `__cap1p75` | abandoned descent search, ignore |

---

## What was wrong, in the order it was found

### 1. UltraEP crashed on every model — unexecutable capacities (`__cap1p02`)

The selection rounded the measured requirement up to the next 0.01. UltraEP's
dispatch buffer is `int(capacity * seq * mbs * topk)` and the kernel asserts it
is a multiple of 128, so 1.02 produced 66846 tokens and every rank died before
iteration 0:

```
num_permuted_tokens (66846) must be a multiple of pad_multiple (128)
```

On this model matrix `seq*topk` is a large power-of-two multiple of 128, so one
0.01 capacity step is `128/25` of a boundary: only capacities that are multiples
of 0.25 are executable at all. The 0.01 rule could essentially never work here.

`__cap1p02` cells contain no measurement. They are crash evidence only.

### 2. QWEN2_57B lost the fp32 router it had always been measured with

Until commit `f39dedd`, `--moe-router-dtype fp32` was passed unconditionally.
That commit made it per-model and set QWEN2_57B to the ModelZoo-faithful empty
value, which UltraEP cannot run — it asserts on the probs dtype at
`ultra_ep.cpp:909`. Restored to fp32, which is also the configuration every
previously published QWEN2_57B number was taken under.

### 3. A MoonEP OOM halted the whole campaign

MoonEP does not fit QWEN3_235B or QWEN3_30B at GBS8. The Stage 7 gate treated
that identically to a crash and aborted the study for all three backends, which
contradicts the runbook's own rule that an OOM is data and only BADFLAG/ASSERT
are bugs. An OOM is now recorded as `DOES_NOT_FIT` for that one
(model, backend) pair and the campaign continues.

**This is a result, not an error.** Those two `DOES_NOT_FIT` rows are real
findings and should be reported as such.

### 4. ECHO's `required_capacity` was not a measurement

`summarise_echo` derived the requirement from `[MOE_DROP_DEBUG][setup_metadata]`,
which reports what the dispatcher *expects before routing*: its
`exact_recv_per_rank` is the perfectly uniform `[32768]*8`. The requirement came
out as exactly `1.0` for every model at every capacity — the uniform lower
bound, structurally unable to see the imbalance the probe exists to measure.

The observed data is in the `[dispatch]` rows (`actual_valid_recv`), the same
fields the older `summarize_token_drop_log.py` keyed on. Reading those gives
per-model requirements of **1.023–1.055** instead of a flat 1.0, and makes
ECHO's number directly comparable with UltraEP's `1.0101`: both are now
peak actual demand over the uniform share.

**Any result showing ECHO `required_capacity = 1.0` predates this fix.**

### 5. ECHO selected the highest capacity that fit, not the lowest that worked

The old loop stopped at the first rung that did not OOM, so 2.00 was chosen for
all four models on the first attempt and nothing lower was tried. Because
UltraEP is required to time at ECHO's capacity, that inflated the receive buffer
for **both** backends. ECHO now follows UltraEP's procedure — measure, round up
to an executable value, validate zero drops there — and the common timing
capacity is `max(ECHO, UltraEP)` per model.

Effect: the timing capacity is **1.25**, not 2.00.

### 6. Drop detection reported "zero drops" exactly when tokens were dropped

`observed_loss` is `expected_recv - actual_valid_recv`, not a drop count. When
the buffer genuinely saturates, `actual_valid_recv` is clipped at `budget`, so
`observed_loss` goes strongly negative and `max(0, observed_loss)` evaluates to
zero. Every earlier zero-drop verdict was therefore produced by a test that
could not detect the failure it was checking for.

It was masked because every capacity probed until then was heavily oversized.
It became load-bearing as soon as the selection started probing downward toward
the saturation point. Drops are now detected by buffer saturation
(`actual_valid_recv >= budget`), which survives clipping.

---

## What this means when reading the results

- **Selected capacity does not discriminate the backends.** ECHO needs ~1.03 and
  UltraEP ~1.01, but the 128-alignment forces both onto the same 0.25 grid, so
  both select 1.25. The meaningful comparison is the measured
  `required_capacity`, not the selected one.
- **Both backends are over-provisioned by roughly 21%** at C=1.25 against a real
  requirement near 1.03. That is a property of the kernel's alignment, not of
  the routing, and memory results should say so rather than describing the
  buffer as right-sized.
- **Probe peaks taken at different capacities are not comparable.** Early ECHO
  numbers were measured at C=2.00 and UltraEP at C=1.25; the valid comparison is
  the Stage 4 timing runs, which use one common capacity per model.

---

## Provenance

Every step writes `provenance.txt` next to its results with the driver, the
CUDA/NCCL/NVSHMEM versions and the commit of each repository. The fixes above
correspond to these commits on `granularity-base`:

```
e3f7c0b  Tier B probe: pick capacities UltraEP can execute, and keep QWEN2_57B on fp32
4e4081f  Tier B: an OOM excludes one model on one backend, not the whole campaign
         Measure ECHO's real capacity requirement, and search for the lowest zero-drop rung
         Select ECHO's capacity the way UltraEP does, and time both at max(ECHO, UltraEP)
         Detect ECHO token drops by buffer saturation, not by observed_loss
```

A result predates a fix if its `provenance.txt` records a
`blackwell_run_scripts` commit older than the corresponding one above.