AbhishekGhosh1998 commited on
Commit
75be0f0
·
verified ·
1 Parent(s): 74cca39

Upload TIERB_HF_ARTIFACT_INVENTORY.md with huggingface_hub

Browse files
Files changed (1) hide show
  1. TIERB_HF_ARTIFACT_INVENTORY.md +106 -0
TIERB_HF_ARTIFACT_INVENTORY.md ADDED
@@ -0,0 +1,106 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # HuggingFace artifact inventory — `tierb-b200-gbs8`
2
+
3
+ Repository: `MoE-CUDA-Graph-Analysis/tierb-b200-gbs8`
4
+ **Type: `model`, not `dataset`.** Querying it as a dataset returns a 404 that
5
+ looks like an authentication failure. `hf_backup.sh` defaults `HF_REPO_TYPE` to
6
+ `model`, which is what every upload in this study has used.
7
+
8
+ ```python
9
+ api.list_repo_files("MoE-CUDA-Graph-Analysis/tierb-b200-gbs8", repo_type="model")
10
+ ```
11
+
12
+ Inventory taken 2026-08-30, immediately after the fixed-MoonEP Tier B campaign
13
+ completed. Total: **5,053 files**.
14
+
15
+ ## 1. Repository top level
16
+
17
+ | directory | files | what it holds |
18
+ | --- | ---: | --- |
19
+ | `echo/` | 1,688 | ECHO campaign, all phases, earlier campaigns |
20
+ | `ultraep/` | 1,640 | UltraEP campaign, all phases, earlier campaigns |
21
+ | `tierb-fixed-moonep-full/` | 911 | **the fixed-MoonEP campaign — see §2** |
22
+ | `moonep/` | 431 | MoonEP campaign, pre-fix |
23
+ | `tierb_study_20260826_144644/` | 242 | the earlier Tier B study whose capacities seeded stage 9 |
24
+ | `tierb-patch-rerun-moonep-ddp-qwen2-overlap/` | 138 | patch-rerun campaign — see §3 |
25
+ | `README.md`, `TIERB_RUN_HISTORY.md`, `.gitattributes` | 3 | repo-level docs |
26
+
27
+ `echo/` and `ultraep/` are large because they accumulated across every campaign,
28
+ not because this run wrote to them.
29
+
30
+ ## 2. `tierb-fixed-moonep-full/` — the fixed-MoonEP campaign
31
+
32
+ 911 files, 1.7 GB local. This is the first Tier B measurement taken with the
33
+ MoonEP duplicated-expert gradient bug fixed.
34
+
35
+ | path | files | size | contents |
36
+ | --- | ---: | ---: | --- |
37
+ | `moonep/tierbmem` | 734 | 954M | stage 8 allocator dumps: 8 cells (4 models × CUDA graph on/off), 712 snapshots |
38
+ | `moonep/tierbtrace` | 34 | 209M | 4 nsys traces + `moonep_async_overlap.{txt,json}` per model |
39
+ | `moonep/tierbprobe` | 32 | 34M | stage 7 capacity probe; all 4 models SELECTED |
40
+ | `moonep/tierb` | 40 | 1.5M | stage 4 timing: 4 models × 3 repeats, plus `results.tsv` |
41
+ | `echo/tierbtrace` | 22 | 277M | 4 nsys traces |
42
+ | `ultraep/tierbtrace` | 22 | 220M | 4 nsys traces |
43
+ | `echo/tierbprobe` | 1 | — | capacities seeded from the previous campaign via `--skip-echo-ultra` |
44
+ | roll-ups | 8 | — | `collected_all.tsv`, `probe_summary_all.{tsv,json}`, `memory_summary_all.{tsv,json}`, `tierb_timing_capacities.env`, `STATUS.txt`, `PATCH_RERUN_MANIFEST.txt` |
45
+ | provenance | 17 | — | `steps.tsv`, `steps_attempt1_preserved.tsv`, `study.log`, `hf_watcher.log`, `hf_watcher.pid`, `.completed/` and `completed_markers/` (6 each) |
46
+ | doc | 1 | — | `TIERB_FIXED_MOONEP_RESULTS.md` — the results write-up |
47
+
48
+ Groups sum to 885 + 8 + 17 + 1 = 911.
49
+
50
+ File types:
51
+
52
+ | type | count | type | count |
53
+ | --- | ---: | --- | ---: |
54
+ | `.json` | 360 | `.csv` | 28 |
55
+ | `.pickle` | 352 | `.tsv` | 18 |
56
+ | `.log` | 57 | *(no extension)* | 12 |
57
+ | `.env` | 41 | **`.nsys-rep`** | **12** |
58
+ | `.txt` | 29 | `.md` | 1 |
59
+ | | | `.pid` | 1 |
60
+
61
+ The 12 `.nsys-rep` are the complete stage-9 trace set: 4 models × 3 backends,
62
+ every one verified to carry markers on all 8 GPUs.
63
+
64
+ ## 3. `tierb-patch-rerun-moonep-ddp-qwen2-overlap/` — bug investigation
65
+
66
+ | path | files | contents |
67
+ | --- | ---: | --- |
68
+ | `moonep-grad-bug-analysis/` | 16 | rank dumps and analysis that isolated the 0.800259× gradient scale and the corrupted duplicated-expert gradients |
69
+ | `moonep-grad-bug-fix/` | 13 | fix validation: grad norms, 32/32 exact expert gradients, CUDA-graph-on A/B, root-cause and fix documents |
70
+ | `correctness/` | 3 | ECHO-vs-MoonEP state and gradient gate output |
71
+ | `echo/`, `ultraep/`, `moonep/` | 91 | that campaign's phase results |
72
+ | roll-ups and provenance | 15 | manifests, `steps.tsv`, `study.log`, markers |
73
+
74
+ ## 4. Caveats when reading these artifacts
75
+
76
+ Three things are easy to get wrong.
77
+
78
+ **ECHO and UltraEP probe/timing/memory in `tierb-fixed-moonep-full/` are carried
79
+ over, not re-measured.** Only their *traces* are new. The fix is confined to
80
+ Megatron's DDP bucket bookkeeping on the MoonEP path, so their configuration and
81
+ code are byte-identical to the previous campaign; re-timing them would have cost
82
+ hours and produced the same numbers. Their stage-9 capacities were seeded from
83
+ `tierb_study_20260826_144644/tierb_timing_capacities.env`.
84
+
85
+ **MoonEP results from `moonep/` and from campaigns predating the fix describe a
86
+ stack that was mistraining.** Its `DOES_NOT_FIT` memory verdicts and its
87
+ iteration times are not comparable with the fixed stack, and a "before/after"
88
+ speedup computed across the fix is meaningless because the two do not perform
89
+ the same computation. See `TIERB_FIXED_MOONEP_RESULTS.md` §1–2 and
90
+ `TIERB_MOONEP_MEMORY.md`.
91
+
92
+ **nsys `.sqlite` exports are deliberately not uploaded.** 12 exist locally; they
93
+ are derived from the uploaded `.nsys-rep` and are regenerable, so only the
94
+ `.nsys-rep` are treated as the artifact of record.
95
+
96
+ ## 5. Related documents in this repository
97
+
98
+ | document | subject |
99
+ | --- | --- |
100
+ | `TIERB_FIXED_MOONEP_RESULTS.md` | results of the fixed-MoonEP campaign, plus the two driver bugs found during it |
101
+ | `TIERB_MOONEP_GRAD_ROOT_CAUSE.md` | root cause of the duplicated-expert gradient bug |
102
+ | `TIERB_MOONEP_CORRECTNESS_ANALYSIS.md` | why the fixed stack still failed the ECHO-vs-MoonEP gate |
103
+ | `TIERB_MOONEP_MEMORY.md` | why the pre-fix stack did not fit two models; obsolete for the fixed stack |
104
+ | `TIERB_SHARED_EXPERT_OVERLAP.md` | why `--moe-shared-expert-overlap` cannot be enabled for any backend here |
105
+ | `TIERB_PATCH_RERUN.md` | the campaign runbook |
106
+ | `MOONEP_GRAD_BUG_AND_FIX.md` | the bug and fix in full (Megatron repo, branch `moonep_grad_correctness`) |