|
Download README.md from MoE-CUDA-Graph-Analysis/tierb-b200-gbs8: direct link, hf CLI and curl.
- Browser
- Download file 7.08 kB
-
https://huggingface.co/MoE-CUDA-Graph-Analysis/tierb-b200-gbs8/resolve/3ea3f15544251ec00aaf2d8ec1bf2111c4523762/README.md
- Command line
-
hf download hf://MoE-CUDA-Graph-Analysis/tierb-b200-gbs8@3ea3f15544251ec00aaf2d8ec1bf2111c4523762/README.md
-
curl -L -o README.md https://huggingface.co/MoE-CUDA-Graph-Analysis/tierb-b200-gbs8/resolve/3ea3f15544251ec00aaf2d8ec1bf2111c4523762/README.md
7.08 kB
| # Tier B run history and errata — `tierb-b200-gbs8` | |
| The HuggingFace repository accumulates every upload, and this campaign was | |
| restarted several times on 2026-08-26 while real bugs in the capacity-selection | |
| logic were found and fixed. Result files from those earlier attempts are still | |
| present alongside the good ones. | |
| **Only results whose ECHO `capacity_attempts.tsv` contains `measure` and | |
| `validate` rows are valid.** Everything below explains how to tell them apart | |
| and why the earlier ones are wrong. | |
| Local result directories were overwritten in place by each rerun, so the local | |
| tree holds only the newest data. HuggingFace does not overwrite, so it holds all | |
| of it, which is why this file exists. | |
| --- | |
| ## How to tell a valid run from an invalid one | |
| The capacity tag in a directory name (`__cap2p00`, `__cap1p25`) is **not** | |
| sufficient. `cap2p00` means two different things depending on when it was | |
| written: in the old algorithm it was the *selected timing capacity*, in the | |
| current one it is only the *measurement point* used to observe peak demand. | |
| Use these instead, in `capacity_attempts.tsv` / `capacity_selection.tsv`: | |
| | signal | meaning | | |
| |---|---| | |
| | ECHO rows with `purpose=measure` **and** `purpose=validate` | current, valid | | |
| | ECHO rows with `purpose=natural` | old selection, invalid | | |
| | `required_capacity` exactly `1.0` for every model | fabricated, invalid | | |
| | `required_capacity` like `1.0293`, varying per model | measured, valid | | |
| | any cell tagged `__cap1p02` | crashed before iteration 0, invalid | | |
| | any cell tagged `__cap1p75` | abandoned descent search, ignore | | |
| --- | |
| ## What was wrong, in the order it was found | |
| ### 1. UltraEP crashed on every model — unexecutable capacities (`__cap1p02`) | |
| The selection rounded the measured requirement up to the next 0.01. UltraEP's | |
| dispatch buffer is `int(capacity * seq * mbs * topk)` and the kernel asserts it | |
| is a multiple of 128, so 1.02 produced 66846 tokens and every rank died before | |
| iteration 0: | |
| ``` | |
| num_permuted_tokens (66846) must be a multiple of pad_multiple (128) | |
| ``` | |
| On this model matrix `seq*topk` is a large power-of-two multiple of 128, so one | |
| 0.01 capacity step is `128/25` of a boundary: only capacities that are multiples | |
| of 0.25 are executable at all. The 0.01 rule could essentially never work here. | |
| `__cap1p02` cells contain no measurement. They are crash evidence only. | |
| ### 2. QWEN2_57B lost the fp32 router it had always been measured with | |
| Until commit `f39dedd`, `--moe-router-dtype fp32` was passed unconditionally. | |
| That commit made it per-model and set QWEN2_57B to the ModelZoo-faithful empty | |
| value, which UltraEP cannot run — it asserts on the probs dtype at | |
| `ultra_ep.cpp:909`. Restored to fp32, which is also the configuration every | |
| previously published QWEN2_57B number was taken under. | |
| ### 3. A MoonEP OOM halted the whole campaign | |
| MoonEP does not fit QWEN3_235B or QWEN3_30B at GBS8. The Stage 7 gate treated | |
| that identically to a crash and aborted the study for all three backends, which | |
| contradicts the runbook's own rule that an OOM is data and only BADFLAG/ASSERT | |
| are bugs. An OOM is now recorded as `DOES_NOT_FIT` for that one | |
| (model, backend) pair and the campaign continues. | |
| **This is a result, not an error.** Those two `DOES_NOT_FIT` rows are real | |
| findings and should be reported as such. | |
| ### 4. ECHO's `required_capacity` was not a measurement | |
| `summarise_echo` derived the requirement from `[MOE_DROP_DEBUG][setup_metadata]`, | |
| which reports what the dispatcher *expects before routing*: its | |
| `exact_recv_per_rank` is the perfectly uniform `[32768]*8`. The requirement came | |
| out as exactly `1.0` for every model at every capacity — the uniform lower | |
| bound, structurally unable to see the imbalance the probe exists to measure. | |
| The observed data is in the `[dispatch]` rows (`actual_valid_recv`), the same | |
| fields the older `summarize_token_drop_log.py` keyed on. Reading those gives | |
| per-model requirements of **1.023–1.055** instead of a flat 1.0, and makes | |
| ECHO's number directly comparable with UltraEP's `1.0101`: both are now | |
| peak actual demand over the uniform share. | |
| **Any result showing ECHO `required_capacity = 1.0` predates this fix.** | |
| ### 5. ECHO selected the highest capacity that fit, not the lowest that worked | |
| The old loop stopped at the first rung that did not OOM, so 2.00 was chosen for | |
| all four models on the first attempt and nothing lower was tried. Because | |
| UltraEP is required to time at ECHO's capacity, that inflated the receive buffer | |
| for **both** backends. ECHO now follows UltraEP's procedure — measure, round up | |
| to an executable value, validate zero drops there — and the common timing | |
| capacity is `max(ECHO, UltraEP)` per model. | |
| Effect: the timing capacity is **1.25**, not 2.00. | |
| ### 6. Drop detection reported "zero drops" exactly when tokens were dropped | |
| `observed_loss` is `expected_recv - actual_valid_recv`, not a drop count. When | |
| the buffer genuinely saturates, `actual_valid_recv` is clipped at `budget`, so | |
| `observed_loss` goes strongly negative and `max(0, observed_loss)` evaluates to | |
| zero. Every earlier zero-drop verdict was therefore produced by a test that | |
| could not detect the failure it was checking for. | |
| It was masked because every capacity probed until then was heavily oversized. | |
| It became load-bearing as soon as the selection started probing downward toward | |
| the saturation point. Drops are now detected by buffer saturation | |
| (`actual_valid_recv >= budget`), which survives clipping. | |
| --- | |
| ## What this means when reading the results | |
| - **Selected capacity does not discriminate the backends.** ECHO needs ~1.03 and | |
| UltraEP ~1.01, but the 128-alignment forces both onto the same 0.25 grid, so | |
| both select 1.25. The meaningful comparison is the measured | |
| `required_capacity`, not the selected one. | |
| - **Both backends are over-provisioned by roughly 21%** at C=1.25 against a real | |
| requirement near 1.03. That is a property of the kernel's alignment, not of | |
| the routing, and memory results should say so rather than describing the | |
| buffer as right-sized. | |
| - **Probe peaks taken at different capacities are not comparable.** Early ECHO | |
| numbers were measured at C=2.00 and UltraEP at C=1.25; the valid comparison is | |
| the Stage 4 timing runs, which use one common capacity per model. | |
| --- | |
| ## Provenance | |
| Every step writes `provenance.txt` next to its results with the driver, the | |
| CUDA/NCCL/NVSHMEM versions and the commit of each repository. The fixes above | |
| correspond to these commits on `granularity-base`: | |
| ``` | |
| e3f7c0b Tier B probe: pick capacities UltraEP can execute, and keep QWEN2_57B on fp32 | |
| 4e4081f Tier B: an OOM excludes one model on one backend, not the whole campaign | |
| Measure ECHO's real capacity requirement, and search for the lowest zero-drop rung | |
| Select ECHO's capacity the way UltraEP does, and time both at max(ECHO, UltraEP) | |
| Detect ECHO token drops by buffer saturation, not by observed_loss | |
| ``` | |
| A result predates a fix if its `provenance.txt` records a | |
| `blackwell_run_scripts` commit older than the corresponding one above. | |