AbhishekGhosh1998 commited on
Commit
56a942f
·
verified ·
1 Parent(s): 3230332

Errata: which Tier B runs are valid and why

Browse files
Files changed (1) hide show
  1. TIERB_RUN_HISTORY.md +149 -0
TIERB_RUN_HISTORY.md ADDED
@@ -0,0 +1,149 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # Tier B run history and errata — `tierb-b200-gbs8`
2
+
3
+ The HuggingFace repository accumulates every upload, and this campaign was
4
+ restarted several times on 2026-08-26 while real bugs in the capacity-selection
5
+ logic were found and fixed. Result files from those earlier attempts are still
6
+ present alongside the good ones.
7
+
8
+ **Only results whose ECHO `capacity_attempts.tsv` contains `measure` and
9
+ `validate` rows are valid.** Everything below explains how to tell them apart
10
+ and why the earlier ones are wrong.
11
+
12
+ Local result directories were overwritten in place by each rerun, so the local
13
+ tree holds only the newest data. HuggingFace does not overwrite, so it holds all
14
+ of it, which is why this file exists.
15
+
16
+ ---
17
+
18
+ ## How to tell a valid run from an invalid one
19
+
20
+ The capacity tag in a directory name (`__cap2p00`, `__cap1p25`) is **not**
21
+ sufficient. `cap2p00` means two different things depending on when it was
22
+ written: in the old algorithm it was the *selected timing capacity*, in the
23
+ current one it is only the *measurement point* used to observe peak demand.
24
+
25
+ Use these instead, in `capacity_attempts.tsv` / `capacity_selection.tsv`:
26
+
27
+ | signal | meaning |
28
+ |---|---|
29
+ | ECHO rows with `purpose=measure` **and** `purpose=validate` | current, valid |
30
+ | ECHO rows with `purpose=natural` | old selection, invalid |
31
+ | `required_capacity` exactly `1.0` for every model | fabricated, invalid |
32
+ | `required_capacity` like `1.0293`, varying per model | measured, valid |
33
+ | any cell tagged `__cap1p02` | crashed before iteration 0, invalid |
34
+ | any cell tagged `__cap1p75` | abandoned descent search, ignore |
35
+
36
+ ---
37
+
38
+ ## What was wrong, in the order it was found
39
+
40
+ ### 1. UltraEP crashed on every model — unexecutable capacities (`__cap1p02`)
41
+
42
+ The selection rounded the measured requirement up to the next 0.01. UltraEP's
43
+ dispatch buffer is `int(capacity * seq * mbs * topk)` and the kernel asserts it
44
+ is a multiple of 128, so 1.02 produced 66846 tokens and every rank died before
45
+ iteration 0:
46
+
47
+ ```
48
+ num_permuted_tokens (66846) must be a multiple of pad_multiple (128)
49
+ ```
50
+
51
+ On this model matrix `seq*topk` is a large power-of-two multiple of 128, so one
52
+ 0.01 capacity step is `128/25` of a boundary: only capacities that are multiples
53
+ of 0.25 are executable at all. The 0.01 rule could essentially never work here.
54
+
55
+ `__cap1p02` cells contain no measurement. They are crash evidence only.
56
+
57
+ ### 2. QWEN2_57B lost the fp32 router it had always been measured with
58
+
59
+ Until commit `f39dedd`, `--moe-router-dtype fp32` was passed unconditionally.
60
+ That commit made it per-model and set QWEN2_57B to the ModelZoo-faithful empty
61
+ value, which UltraEP cannot run — it asserts on the probs dtype at
62
+ `ultra_ep.cpp:909`. Restored to fp32, which is also the configuration every
63
+ previously published QWEN2_57B number was taken under.
64
+
65
+ ### 3. A MoonEP OOM halted the whole campaign
66
+
67
+ MoonEP does not fit QWEN3_235B or QWEN3_30B at GBS8. The Stage 7 gate treated
68
+ that identically to a crash and aborted the study for all three backends, which
69
+ contradicts the runbook's own rule that an OOM is data and only BADFLAG/ASSERT
70
+ are bugs. An OOM is now recorded as `DOES_NOT_FIT` for that one
71
+ (model, backend) pair and the campaign continues.
72
+
73
+ **This is a result, not an error.** Those two `DOES_NOT_FIT` rows are real
74
+ findings and should be reported as such.
75
+
76
+ ### 4. ECHO's `required_capacity` was not a measurement
77
+
78
+ `summarise_echo` derived the requirement from `[MOE_DROP_DEBUG][setup_metadata]`,
79
+ which reports what the dispatcher *expects before routing*: its
80
+ `exact_recv_per_rank` is the perfectly uniform `[32768]*8`. The requirement came
81
+ out as exactly `1.0` for every model at every capacity — the uniform lower
82
+ bound, structurally unable to see the imbalance the probe exists to measure.
83
+
84
+ The observed data is in the `[dispatch]` rows (`actual_valid_recv`), the same
85
+ fields the older `summarize_token_drop_log.py` keyed on. Reading those gives
86
+ per-model requirements of **1.023–1.055** instead of a flat 1.0, and makes
87
+ ECHO's number directly comparable with UltraEP's `1.0101`: both are now
88
+ peak actual demand over the uniform share.
89
+
90
+ **Any result showing ECHO `required_capacity = 1.0` predates this fix.**
91
+
92
+ ### 5. ECHO selected the highest capacity that fit, not the lowest that worked
93
+
94
+ The old loop stopped at the first rung that did not OOM, so 2.00 was chosen for
95
+ all four models on the first attempt and nothing lower was tried. Because
96
+ UltraEP is required to time at ECHO's capacity, that inflated the receive buffer
97
+ for **both** backends. ECHO now follows UltraEP's procedure — measure, round up
98
+ to an executable value, validate zero drops there — and the common timing
99
+ capacity is `max(ECHO, UltraEP)` per model.
100
+
101
+ Effect: the timing capacity is **1.25**, not 2.00.
102
+
103
+ ### 6. Drop detection reported "zero drops" exactly when tokens were dropped
104
+
105
+ `observed_loss` is `expected_recv - actual_valid_recv`, not a drop count. When
106
+ the buffer genuinely saturates, `actual_valid_recv` is clipped at `budget`, so
107
+ `observed_loss` goes strongly negative and `max(0, observed_loss)` evaluates to
108
+ zero. Every earlier zero-drop verdict was therefore produced by a test that
109
+ could not detect the failure it was checking for.
110
+
111
+ It was masked because every capacity probed until then was heavily oversized.
112
+ It became load-bearing as soon as the selection started probing downward toward
113
+ the saturation point. Drops are now detected by buffer saturation
114
+ (`actual_valid_recv >= budget`), which survives clipping.
115
+
116
+ ---
117
+
118
+ ## What this means when reading the results
119
+
120
+ - **Selected capacity does not discriminate the backends.** ECHO needs ~1.03 and
121
+ UltraEP ~1.01, but the 128-alignment forces both onto the same 0.25 grid, so
122
+ both select 1.25. The meaningful comparison is the measured
123
+ `required_capacity`, not the selected one.
124
+ - **Both backends are over-provisioned by roughly 21%** at C=1.25 against a real
125
+ requirement near 1.03. That is a property of the kernel's alignment, not of
126
+ the routing, and memory results should say so rather than describing the
127
+ buffer as right-sized.
128
+ - **Probe peaks taken at different capacities are not comparable.** Early ECHO
129
+ numbers were measured at C=2.00 and UltraEP at C=1.25; the valid comparison is
130
+ the Stage 4 timing runs, which use one common capacity per model.
131
+
132
+ ---
133
+
134
+ ## Provenance
135
+
136
+ Every step writes `provenance.txt` next to its results with the driver, the
137
+ CUDA/NCCL/NVSHMEM versions and the commit of each repository. The fixes above
138
+ correspond to these commits on `granularity-base`:
139
+
140
+ ```
141
+ e3f7c0b Tier B probe: pick capacities UltraEP can execute, and keep QWEN2_57B on fp32
142
+ 4e4081f Tier B: an OOM excludes one model on one backend, not the whole campaign
143
+ Measure ECHO's real capacity requirement, and search for the lowest zero-drop rung
144
+ Select ECHO's capacity the way UltraEP does, and time both at max(ECHO, UltraEP)
145
+ Detect ECHO token drops by buffer saturation, not by observed_loss
146
+ ```
147
+
148
+ A result predates a fix if its `provenance.txt` records a
149
+ `blackwell_run_scripts` commit older than the corresponding one above.