AbhishekGhosh1998 commited on
Commit
01b8bc9
·
verified ·
1 Parent(s): 92e38b9

granularity results granularity_next_b200/tierb_models/memory/OLMOE_1B_7B_L16 @ 2026-09-21T08:51:37Z from sp-aiinfra-interactive-aghosh-8-mb99k-8xb200-moe-cg-0 (part 2)

Browse files
granularity_next_b200/tierb_models/memory/OLMOE_1B_7B_L16/steps.tsv CHANGED
@@ -1 +1,4 @@
1
  status campaign phase gbs started ended result_dir
 
 
 
 
1
  status campaign phase gbs started ended result_dir
2
+ OK echo tierbmem 8 2026-09-21T08:43:55+00:00 2026-09-21T08:46:34+00:00 /workspace/granularity_next_b200/tierb_models/memory/OLMOE_1B_7B_L16/echo/tierbmem
3
+ OK ultraep tierbmem 8 2026-09-21T08:46:39+00:00 2026-09-21T08:49:02+00:00 /workspace/granularity_next_b200/tierb_models/memory/OLMOE_1B_7B_L16/ultraep/tierbmem
4
+ OK moonep tierbmem 8 2026-09-21T08:49:07+00:00 2026-09-21T08:51:30+00:00 /workspace/granularity_next_b200/tierb_models/memory/OLMOE_1B_7B_L16/moonep/tierbmem
granularity_next_b200/tierb_models/memory/OLMOE_1B_7B_L16/study.log CHANGED
@@ -10,3 +10,223 @@ driver_repo: /workspace/blackwell_run_scripts @ d29f657
10
  === STAGE 8 -- Tier B allocator dumps (static CG capacities + dynamic no-CG)
11
 
12
  === STEP echo__tierbmem__gbs8
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
10
  === STAGE 8 -- Tier B allocator dumps (static CG capacities + dynamic no-CG)
11
 
12
  === STEP echo__tierbmem__gbs8
13
+ ==============================================================
14
+ CAMPAIGN: echo backend=echo phase=tierbmem
15
+ Megatron: /workspace/Megatron-MoE-Echo @ better-memory-dump-support
16
+ GBS=8 seq=4096 repeats=3 nsys=1
17
+ results -> /workspace/granularity_next_b200/tierb_models/memory/OLMOE_1B_7B_L16/echo/tierbmem
18
+ ==============================================================
19
+ --------------------------------------------------------------------------
20
+ [08:44:09] CELL OLMOE_1B_7B_L16__echo__matched__seq4096__gbs8__cap1p08984375__memiteration__it6 (slots=8)
21
+ [guard] iter 6/6 (70s)
22
+ --------------------------------------------------------------------------
23
+ [08:45:19] CELL OLMOE_1B_7B_L16__echo__matched__seq4096__gbs8__capdynamic__nocg__memiteration__it5 (slots=8)
24
+ [guard] iter 5/5 (70s)
25
+ [guard] tearing down process group (outcome=DONE)
26
+ --------------------------------------------------------------------------
27
+
28
+ ==============================================================
29
+ SWEEP COMPLETE ok=2 failed=0
30
+ ==============================================================
31
+ config backend policy slots seq gbs cap recompute rep status iter_ms_mean iter_ms_min peak_mib run_dir
32
+ OLMOE_1B_7B_L16 echo matched 8 4096 8 1.08984375 none 1 OK 118.60 118.60 43043 /workspace/granularity_next_b200/tierb_models/memory/OLMOE_1B_7B_L16/echo/tierbmem/tierbmem_gbs8/OLMOE_1B_7B_L16__echo__matched__seq4096__gbs8__cap1p08984375__memiteration__it6/rep1
33
+ OLMOE_1B_7B_L16 echo matched 8 4096 8 dynamic none 1 OK NA NA 38025 /workspace/granularity_next_b200/tierb_models/memory/OLMOE_1B_7B_L16/echo/tierbmem/tierbmem_gbs8/OLMOE_1B_7B_L16__echo__matched__seq4096__gbs8__capdynamic__nocg__memiteration__it5/rep1
34
+
35
+ Results: /workspace/granularity_next_b200/tierb_models/memory/OLMOE_1B_7B_L16/echo/tierbmem/tierbmem_gbs8/results.tsv
36
+ Roll up: python3 /workspace/blackwell_run_scripts/granularity/tools/collect.py /workspace/granularity_next_b200/tierb_models/memory/OLMOE_1B_7B_L16/echo/tierbmem/tierbmem_gbs8
37
+ ==============================================================
38
+ 2 memory-dump runs -> /workspace/granularity_next_b200/tierb_models/memory/OLMOE_1B_7B_L16/echo/tierbmem/tierbmem_gbs8/memory_summary.tsv
39
+ model backend cg allocation capacity status peak GiB dumps
40
+ ----------------------------------------------------------------------------------
41
+ OLMOE_1B_7B_L16 echo 1 static1.08984375 OK 42.0 48/48
42
+ OLMOE_1B_7B_L16 echo 0 dynamic dynamic OK 37.1 40/40
43
+
44
+ ECHO/UltraEP note: graph-on uses static capacity + device-initiated GEMM while graph-off uses dynamic allocation + host-derived sizes. Their difference is not labelled as pure graph cost.
45
+
46
+ TIER B MEMORY DUMPS -- how to read this
47
+ ---------------------------------------
48
+ ECHO / UltraEP:
49
+ cap1p00__memiteration__it6 graph on, static C=1.00
50
+ cap1p125__memiteration__it6 graph on, static C=1.125
51
+ cap1p25__memiteration__it6 graph on, static C=1.25
52
+ capdynamic__nocg__memiteration__it5
53
+ graph off, no capacity flag, dynamic allocation
54
+
55
+ MoonEP:
56
+ capnone__memiteration__it6 graph on, no capacity factor
57
+ capnone__nocg__memiteration__it5 graph off, no capacity factor
58
+
59
+ Every successful cell contains:
60
+ train.log Megatron and OOM diagnostics
61
+ mem_samples.csv driver-level nvidia-smi samples every 2 seconds
62
+ run.env exact branch/config/capacity provenance
63
+ memory_snapshots/*.pickle 48 captured / 40 eager allocator histories
64
+ memory_snapshots/*.json matching summaries and before/after deltas
65
+
66
+ The graph-on/static vs graph-off/dynamic difference is intentionally NOT called
67
+ "graph cost": capacity allocation policy and device-initiated grouped GEMM also
68
+ change. Compare the three graph-on rows to measure static-capacity sensitivity,
69
+ and use the dynamic row as the natural eager-allocation reference.
70
+
71
+ Campaign results: /workspace/granularity_next_b200/tierb_models/memory/OLMOE_1B_7B_L16/echo/tierbmem
72
+ Roll up: python3 /workspace/blackwell_run_scripts/granularity/tools/collect.py /workspace/granularity_next_b200/tierb_models/memory/OLMOE_1B_7B_L16/echo/tierbmem --compare
73
+ Already logged in to Hugging Face as: AbhishekGhosh1998
74
+ uploaded: /workspace/granularity_next_b200/tierb_models/memory/OLMOE_1B_7B_L16/echo/tierbmem -> MoE-CUDA-Graph-Analysis/tierb-b200-gbs8/granularity_next_b200/tierb_models/memory/OLMOE_1B_7B_L16/echo/tierbmem
75
+ https://huggingface.co/MoE-CUDA-Graph-Analysis/tierb-b200-gbs8/tree/main
76
+
77
+ === STEP ultraep__tierbmem__gbs8
78
+ ==============================================================
79
+ CAMPAIGN: ultraep backend=ultraep phase=tierbmem
80
+ Megatron: /workspace/Megatron-UltraEP @ ultraep-better-memory-dump-support
81
+ GBS=8 seq=4096 repeats=3 nsys=1
82
+ results -> /workspace/granularity_next_b200/tierb_models/memory/OLMOE_1B_7B_L16/ultraep/tierbmem
83
+ ==============================================================
84
+ --------------------------------------------------------------------------
85
+ [08:46:53] CELL OLMOE_1B_7B_L16__ultraep__matched__seq4096__gbs8__cap1p08984375__memiteration__it6 (slots=8)
86
+ [guard] iter 6/6 (60s)
87
+ [guard] tearing down process group (outcome=DONE)
88
+ --------------------------------------------------------------------------
89
+ [08:47:57] CELL OLMOE_1B_7B_L16__ultraep__matched__seq4096__gbs8__capdynamic__nocg__memiteration__it5 (slots=8)
90
+ [guard] iter 5/5 (60s)
91
+ [guard] tearing down process group (outcome=DONE)
92
+ --------------------------------------------------------------------------
93
+
94
+ ==============================================================
95
+ SWEEP COMPLETE ok=2 failed=0
96
+ ==============================================================
97
+ config backend policy slots seq gbs cap recompute rep status iter_ms_mean iter_ms_min peak_mib run_dir
98
+ OLMOE_1B_7B_L16 ultraep matched 8 4096 8 1.08984375 none 1 OK 111.20 111.20 39097 /workspace/granularity_next_b200/tierb_models/memory/OLMOE_1B_7B_L16/ultraep/tierbmem/tierbmem_gbs8/OLMOE_1B_7B_L16__ultraep__matched__seq4096__gbs8__cap1p08984375__memiteration__it6/rep1
99
+ OLMOE_1B_7B_L16 ultraep matched 8 4096 8 dynamic none 1 OK NA NA 37091 /workspace/granularity_next_b200/tierb_models/memory/OLMOE_1B_7B_L16/ultraep/tierbmem/tierbmem_gbs8/OLMOE_1B_7B_L16__ultraep__matched__seq4096__gbs8__capdynamic__nocg__memiteration__it5/rep1
100
+
101
+ Results: /workspace/granularity_next_b200/tierb_models/memory/OLMOE_1B_7B_L16/ultraep/tierbmem/tierbmem_gbs8/results.tsv
102
+ Roll up: python3 /workspace/blackwell_run_scripts/granularity/tools/collect.py /workspace/granularity_next_b200/tierb_models/memory/OLMOE_1B_7B_L16/ultraep/tierbmem/tierbmem_gbs8
103
+ ==============================================================
104
+ 2 memory-dump runs -> /workspace/granularity_next_b200/tierb_models/memory/OLMOE_1B_7B_L16/ultraep/tierbmem/tierbmem_gbs8/memory_summary.tsv
105
+ model backend cg allocation capacity status peak GiB dumps
106
+ ----------------------------------------------------------------------------------
107
+ OLMOE_1B_7B_L16 ultraep 1 static1.08984375 OK 38.2 48/48
108
+ OLMOE_1B_7B_L16 ultraep 0 dynamic dynamic OK 36.2 40/40
109
+
110
+ ECHO/UltraEP note: graph-on uses static capacity + device-initiated GEMM while graph-off uses dynamic allocation + host-derived sizes. Their difference is not labelled as pure graph cost.
111
+
112
+ TIER B MEMORY DUMPS -- how to read this
113
+ ---------------------------------------
114
+ ECHO / UltraEP:
115
+ cap1p00__memiteration__it6 graph on, static C=1.00
116
+ cap1p125__memiteration__it6 graph on, static C=1.125
117
+ cap1p25__memiteration__it6 graph on, static C=1.25
118
+ capdynamic__nocg__memiteration__it5
119
+ graph off, no capacity flag, dynamic allocation
120
+
121
+ MoonEP:
122
+ capnone__memiteration__it6 graph on, no capacity factor
123
+ capnone__nocg__memiteration__it5 graph off, no capacity factor
124
+
125
+ Every successful cell contains:
126
+ train.log Megatron and OOM diagnostics
127
+ mem_samples.csv driver-level nvidia-smi samples every 2 seconds
128
+ run.env exact branch/config/capacity provenance
129
+ memory_snapshots/*.pickle 48 captured / 40 eager allocator histories
130
+ memory_snapshots/*.json matching summaries and before/after deltas
131
+
132
+ The graph-on/static vs graph-off/dynamic difference is intentionally NOT called
133
+ "graph cost": capacity allocation policy and device-initiated grouped GEMM also
134
+ change. Compare the three graph-on rows to measure static-capacity sensitivity,
135
+ and use the dynamic row as the natural eager-allocation reference.
136
+
137
+ Campaign results: /workspace/granularity_next_b200/tierb_models/memory/OLMOE_1B_7B_L16/ultraep/tierbmem
138
+ Roll up: python3 /workspace/blackwell_run_scripts/granularity/tools/collect.py /workspace/granularity_next_b200/tierb_models/memory/OLMOE_1B_7B_L16/ultraep/tierbmem --compare
139
+ Already logged in to Hugging Face as: AbhishekGhosh1998
140
+ uploaded: /workspace/granularity_next_b200/tierb_models/memory/OLMOE_1B_7B_L16/ultraep/tierbmem -> MoE-CUDA-Graph-Analysis/tierb-b200-gbs8/granularity_next_b200/tierb_models/memory/OLMOE_1B_7B_L16/ultraep/tierbmem
141
+ https://huggingface.co/MoE-CUDA-Graph-Analysis/tierb-b200-gbs8/tree/main
142
+
143
+ === STEP moonep__tierbmem__gbs8
144
+ ==============================================================
145
+ CAMPAIGN: moonep backend=moonep phase=tierbmem
146
+ Megatron: /workspace/Megatron-MoE-Echo @ moonep_backend_async_grad_reduce_better_memory_dump_support
147
+ GBS=8 seq=4096 repeats=3 nsys=1
148
+ results -> /workspace/granularity_next_b200/tierb_models/memory/OLMOE_1B_7B_L16/moonep/tierbmem
149
+ ==============================================================
150
+ --------------------------------------------------------------------------
151
+ [08:49:21] CELL OLMOE_1B_7B_L16__moonep__matched__seq4096__gbs8__capnone__async__memiteration__it6 (slots=8)
152
+ [guard] iter 6/6 (60s)
153
+ [guard] tearing down process group (outcome=DONE)
154
+ --------------------------------------------------------------------------
155
+ [08:50:25] CELL OLMOE_1B_7B_L16__moonep__matched__seq4096__gbs8__capnone__nocg__async__memiteration__it5 (slots=8)
156
+ [guard] iter 5/5 (60s)
157
+ [guard] tearing down process group (outcome=DONE)
158
+ --------------------------------------------------------------------------
159
+
160
+ ==============================================================
161
+ SWEEP COMPLETE ok=2 failed=0
162
+ ==============================================================
163
+ config backend policy slots seq gbs cap recompute rep status iter_ms_mean iter_ms_min peak_mib run_dir
164
+ OLMOE_1B_7B_L16 moonep matched 8 4096 8 none none 1 OK 118.90 118.90 36421 /workspace/granularity_next_b200/tierb_models/memory/OLMOE_1B_7B_L16/moonep/tierbmem/tierbmem_gbs8/OLMOE_1B_7B_L16__moonep__matched__seq4096__gbs8__capnone__async__memiteration__it6/rep1
165
+ OLMOE_1B_7B_L16 moonep matched 8 4096 8 none none 1 OK NA NA 35751 /workspace/granularity_next_b200/tierb_models/memory/OLMOE_1B_7B_L16/moonep/tierbmem/tierbmem_gbs8/OLMOE_1B_7B_L16__moonep__matched__seq4096__gbs8__capnone__nocg__async__memiteration__it5/rep1
166
+
167
+ Results: /workspace/granularity_next_b200/tierb_models/memory/OLMOE_1B_7B_L16/moonep/tierbmem/tierbmem_gbs8/results.tsv
168
+ Roll up: python3 /workspace/blackwell_run_scripts/granularity/tools/collect.py /workspace/granularity_next_b200/tierb_models/memory/OLMOE_1B_7B_L16/moonep/tierbmem/tierbmem_gbs8
169
+ ==============================================================
170
+ 2 memory-dump runs -> /workspace/granularity_next_b200/tierb_models/memory/OLMOE_1B_7B_L16/moonep/tierbmem/tierbmem_gbs8/memory_summary.tsv
171
+ model backend cg allocation capacity status peak GiB dumps
172
+ ----------------------------------------------------------------------------------
173
+ OLMOE_1B_7B_L16 moonep 1 none none OK 35.6 48/48
174
+ OLMOE_1B_7B_L16 moonep 0 none none OK 34.9 40/40
175
+
176
+ ECHO/UltraEP note: graph-on uses static capacity + device-initiated GEMM while graph-off uses dynamic allocation + host-derived sizes. Their difference is not labelled as pure graph cost.
177
+
178
+ TIER B MEMORY DUMPS -- how to read this
179
+ ---------------------------------------
180
+ ECHO / UltraEP:
181
+ cap1p00__memiteration__it6 graph on, static C=1.00
182
+ cap1p125__memiteration__it6 graph on, static C=1.125
183
+ cap1p25__memiteration__it6 graph on, static C=1.25
184
+ capdynamic__nocg__memiteration__it5
185
+ graph off, no capacity flag, dynamic allocation
186
+
187
+ MoonEP:
188
+ capnone__memiteration__it6 graph on, no capacity factor
189
+ capnone__nocg__memiteration__it5 graph off, no capacity factor
190
+
191
+ Every successful cell contains:
192
+ train.log Megatron and OOM diagnostics
193
+ mem_samples.csv driver-level nvidia-smi samples every 2 seconds
194
+ run.env exact branch/config/capacity provenance
195
+ memory_snapshots/*.pickle 48 captured / 40 eager allocator histories
196
+ memory_snapshots/*.json matching summaries and before/after deltas
197
+
198
+ The graph-on/static vs graph-off/dynamic difference is intentionally NOT called
199
+ "graph cost": capacity allocation policy and device-initiated grouped GEMM also
200
+ change. Compare the three graph-on rows to measure static-capacity sensitivity,
201
+ and use the dynamic row as the natural eager-allocation reference.
202
+
203
+ Campaign results: /workspace/granularity_next_b200/tierb_models/memory/OLMOE_1B_7B_L16/moonep/tierbmem
204
+ Roll up: python3 /workspace/blackwell_run_scripts/granularity/tools/collect.py /workspace/granularity_next_b200/tierb_models/memory/OLMOE_1B_7B_L16/moonep/tierbmem --compare
205
+ Already logged in to Hugging Face as: AbhishekGhosh1998
206
+ uploaded: /workspace/granularity_next_b200/tierb_models/memory/OLMOE_1B_7B_L16/moonep/tierbmem -> MoE-CUDA-Graph-Analysis/tierb-b200-gbs8/granularity_next_b200/tierb_models/memory/OLMOE_1B_7B_L16/moonep/tierbmem
207
+ https://huggingface.co/MoE-CUDA-Graph-Analysis/tierb-b200-gbs8/tree/main
208
+
209
+ === ROLL-UP
210
+ 6 runs ok=3 oom=0 -> /workspace/granularity_next_b200/tierb_models/memory/OLMOE_1B_7B_L16/collected_all.tsv
211
+
212
+ phase config seq gbs policy rc cg asy backend iter_ms vs_best
213
+ -------------------------------------------------------------------------------------
214
+ tierbmem OLMOE_1B_7B_L16 4096 8 matched none 1 0 ultraep 111.20 1.00x
215
+ echo 118.60 1.07x
216
+
217
+ tierbmem OLMOE_1B_7B_L16 4096 8 matched none 1 1 moonep 118.90 1.00x
218
+
219
+ no probe telemetry found under /workspace/granularity_next_b200/tierb_models/memory/OLMOE_1B_7B_L16
220
+ expected instrumented branches: token-drop-analysis / ultraep-token-drop-analysis / moonep_backend_token_routing_analysis
221
+ 6 memory-dump runs -> /workspace/granularity_next_b200/tierb_models/memory/OLMOE_1B_7B_L16/memory_summary_all.tsv
222
+ model backend cg allocation capacity status peak GiB dumps
223
+ ----------------------------------------------------------------------------------
224
+ OLMOE_1B_7B_L16 echo 1 static1.08984375 OK 42.0 48/48
225
+ OLMOE_1B_7B_L16 echo 0 dynamic dynamic OK 37.1 40/40
226
+ OLMOE_1B_7B_L16 moonep 1 none none OK 35.6 48/48
227
+ OLMOE_1B_7B_L16 moonep 0 none none OK 34.9 40/40
228
+ OLMOE_1B_7B_L16 ultraep 1 static1.08984375 OK 38.2 48/48
229
+ OLMOE_1B_7B_L16 ultraep 0 dynamic dynamic OK 36.2 40/40
230
+
231
+ ECHO/UltraEP note: graph-on uses static capacity + device-initiated GEMM while graph-off uses dynamic allocation + host-derived sizes. Their difference is not labelled as pure graph cost.
232
+ Already logged in to Hugging Face as: AbhishekGhosh1998