Upload tierb-fixed-moonep-full/TIERB_HF_ARTIFACT_INVENTORY.md with huggingface_hub
Browse files
tierb-fixed-moonep-full/TIERB_HF_ARTIFACT_INVENTORY.md
ADDED
|
@@ -0,0 +1,106 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
# HuggingFace artifact inventory — `tierb-b200-gbs8`
|
| 2 |
+
|
| 3 |
+
Repository: `MoE-CUDA-Graph-Analysis/tierb-b200-gbs8`
|
| 4 |
+
**Type: `model`, not `dataset`.** Querying it as a dataset returns a 404 that
|
| 5 |
+
looks like an authentication failure. `hf_backup.sh` defaults `HF_REPO_TYPE` to
|
| 6 |
+
`model`, which is what every upload in this study has used.
|
| 7 |
+
|
| 8 |
+
```python
|
| 9 |
+
api.list_repo_files("MoE-CUDA-Graph-Analysis/tierb-b200-gbs8", repo_type="model")
|
| 10 |
+
```
|
| 11 |
+
|
| 12 |
+
Inventory taken 2026-08-30, immediately after the fixed-MoonEP Tier B campaign
|
| 13 |
+
completed. Total: **5,053 files**.
|
| 14 |
+
|
| 15 |
+
## 1. Repository top level
|
| 16 |
+
|
| 17 |
+
| directory | files | what it holds |
|
| 18 |
+
| --- | ---: | --- |
|
| 19 |
+
| `echo/` | 1,688 | ECHO campaign, all phases, earlier campaigns |
|
| 20 |
+
| `ultraep/` | 1,640 | UltraEP campaign, all phases, earlier campaigns |
|
| 21 |
+
| `tierb-fixed-moonep-full/` | 911 | **the fixed-MoonEP campaign — see §2** |
|
| 22 |
+
| `moonep/` | 431 | MoonEP campaign, pre-fix |
|
| 23 |
+
| `tierb_study_20260826_144644/` | 242 | the earlier Tier B study whose capacities seeded stage 9 |
|
| 24 |
+
| `tierb-patch-rerun-moonep-ddp-qwen2-overlap/` | 138 | patch-rerun campaign — see §3 |
|
| 25 |
+
| `README.md`, `TIERB_RUN_HISTORY.md`, `.gitattributes` | 3 | repo-level docs |
|
| 26 |
+
|
| 27 |
+
`echo/` and `ultraep/` are large because they accumulated across every campaign,
|
| 28 |
+
not because this run wrote to them.
|
| 29 |
+
|
| 30 |
+
## 2. `tierb-fixed-moonep-full/` — the fixed-MoonEP campaign
|
| 31 |
+
|
| 32 |
+
911 files, 1.7 GB local. This is the first Tier B measurement taken with the
|
| 33 |
+
MoonEP duplicated-expert gradient bug fixed.
|
| 34 |
+
|
| 35 |
+
| path | files | size | contents |
|
| 36 |
+
| --- | ---: | ---: | --- |
|
| 37 |
+
| `moonep/tierbmem` | 734 | 954M | stage 8 allocator dumps: 8 cells (4 models × CUDA graph on/off), 712 snapshots |
|
| 38 |
+
| `moonep/tierbtrace` | 34 | 209M | 4 nsys traces + `moonep_async_overlap.{txt,json}` per model |
|
| 39 |
+
| `moonep/tierbprobe` | 32 | 34M | stage 7 capacity probe; all 4 models SELECTED |
|
| 40 |
+
| `moonep/tierb` | 40 | 1.5M | stage 4 timing: 4 models × 3 repeats, plus `results.tsv` |
|
| 41 |
+
| `echo/tierbtrace` | 22 | 277M | 4 nsys traces |
|
| 42 |
+
| `ultraep/tierbtrace` | 22 | 220M | 4 nsys traces |
|
| 43 |
+
| `echo/tierbprobe` | 1 | — | capacities seeded from the previous campaign via `--skip-echo-ultra` |
|
| 44 |
+
| roll-ups | 8 | — | `collected_all.tsv`, `probe_summary_all.{tsv,json}`, `memory_summary_all.{tsv,json}`, `tierb_timing_capacities.env`, `STATUS.txt`, `PATCH_RERUN_MANIFEST.txt` |
|
| 45 |
+
| provenance | 17 | — | `steps.tsv`, `steps_attempt1_preserved.tsv`, `study.log`, `hf_watcher.log`, `hf_watcher.pid`, `.completed/` and `completed_markers/` (6 each) |
|
| 46 |
+
| doc | 1 | — | `TIERB_FIXED_MOONEP_RESULTS.md` — the results write-up |
|
| 47 |
+
|
| 48 |
+
Groups sum to 885 + 8 + 17 + 1 = 911.
|
| 49 |
+
|
| 50 |
+
File types:
|
| 51 |
+
|
| 52 |
+
| type | count | type | count |
|
| 53 |
+
| --- | ---: | --- | ---: |
|
| 54 |
+
| `.json` | 360 | `.csv` | 28 |
|
| 55 |
+
| `.pickle` | 352 | `.tsv` | 18 |
|
| 56 |
+
| `.log` | 57 | *(no extension)* | 12 |
|
| 57 |
+
| `.env` | 41 | **`.nsys-rep`** | **12** |
|
| 58 |
+
| `.txt` | 29 | `.md` | 1 |
|
| 59 |
+
| | | `.pid` | 1 |
|
| 60 |
+
|
| 61 |
+
The 12 `.nsys-rep` are the complete stage-9 trace set: 4 models × 3 backends,
|
| 62 |
+
every one verified to carry markers on all 8 GPUs.
|
| 63 |
+
|
| 64 |
+
## 3. `tierb-patch-rerun-moonep-ddp-qwen2-overlap/` — bug investigation
|
| 65 |
+
|
| 66 |
+
| path | files | contents |
|
| 67 |
+
| --- | ---: | --- |
|
| 68 |
+
| `moonep-grad-bug-analysis/` | 16 | rank dumps and analysis that isolated the 0.800259× gradient scale and the corrupted duplicated-expert gradients |
|
| 69 |
+
| `moonep-grad-bug-fix/` | 13 | fix validation: grad norms, 32/32 exact expert gradients, CUDA-graph-on A/B, root-cause and fix documents |
|
| 70 |
+
| `correctness/` | 3 | ECHO-vs-MoonEP state and gradient gate output |
|
| 71 |
+
| `echo/`, `ultraep/`, `moonep/` | 91 | that campaign's phase results |
|
| 72 |
+
| roll-ups and provenance | 15 | manifests, `steps.tsv`, `study.log`, markers |
|
| 73 |
+
|
| 74 |
+
## 4. Caveats when reading these artifacts
|
| 75 |
+
|
| 76 |
+
Three things are easy to get wrong.
|
| 77 |
+
|
| 78 |
+
**ECHO and UltraEP probe/timing/memory in `tierb-fixed-moonep-full/` are carried
|
| 79 |
+
over, not re-measured.** Only their *traces* are new. The fix is confined to
|
| 80 |
+
Megatron's DDP bucket bookkeeping on the MoonEP path, so their configuration and
|
| 81 |
+
code are byte-identical to the previous campaign; re-timing them would have cost
|
| 82 |
+
hours and produced the same numbers. Their stage-9 capacities were seeded from
|
| 83 |
+
`tierb_study_20260826_144644/tierb_timing_capacities.env`.
|
| 84 |
+
|
| 85 |
+
**MoonEP results from `moonep/` and from campaigns predating the fix describe a
|
| 86 |
+
stack that was mistraining.** Its `DOES_NOT_FIT` memory verdicts and its
|
| 87 |
+
iteration times are not comparable with the fixed stack, and a "before/after"
|
| 88 |
+
speedup computed across the fix is meaningless because the two do not perform
|
| 89 |
+
the same computation. See `TIERB_FIXED_MOONEP_RESULTS.md` §1–2 and
|
| 90 |
+
`TIERB_MOONEP_MEMORY.md`.
|
| 91 |
+
|
| 92 |
+
**nsys `.sqlite` exports are deliberately not uploaded.** 12 exist locally; they
|
| 93 |
+
are derived from the uploaded `.nsys-rep` and are regenerable, so only the
|
| 94 |
+
`.nsys-rep` are treated as the artifact of record.
|
| 95 |
+
|
| 96 |
+
## 5. Related documents in this repository
|
| 97 |
+
|
| 98 |
+
| document | subject |
|
| 99 |
+
| --- | --- |
|
| 100 |
+
| `TIERB_FIXED_MOONEP_RESULTS.md` | results of the fixed-MoonEP campaign, plus the two driver bugs found during it |
|
| 101 |
+
| `TIERB_MOONEP_GRAD_ROOT_CAUSE.md` | root cause of the duplicated-expert gradient bug |
|
| 102 |
+
| `TIERB_MOONEP_CORRECTNESS_ANALYSIS.md` | why the fixed stack still failed the ECHO-vs-MoonEP gate |
|
| 103 |
+
| `TIERB_MOONEP_MEMORY.md` | why the pre-fix stack did not fit two models; obsolete for the fixed stack |
|
| 104 |
+
| `TIERB_SHARED_EXPERT_OVERLAP.md` | why `--moe-shared-expert-overlap` cannot be enabled for any backend here |
|
| 105 |
+
| `TIERB_PATCH_RERUN.md` | the campaign runbook |
|
| 106 |
+
| `MOONEP_GRAD_BUG_AND_FIX.md` | the bug and fix in full (Megatron repo, branch `moonep_grad_correctness`) |
|