granularity results tiera-rerun-fixed-moonep @ 2026-08-30T10:26:09Z from sp-aiinfra-interactive-aghosh-7jtjl-8xb200-moe-cg-0 (part 3)
Browse files
tiera-rerun-fixed-moonep/steps.tsv
CHANGED
|
@@ -22,3 +22,6 @@ OK moonep tierbmem 16 2026-08-30T10:05:11+00:00 2026-08-30T10:07:42+00:00 /works
|
|
| 22 |
OK echo tierbmem 16 2026-08-30T10:08:07+00:00 2026-08-30T10:11:03+00:00 /workspace/tiera-rerun-fixed-moonep/echo/tierbmem
|
| 23 |
OK ultraep tierbmem 16 2026-08-30T10:11:14+00:00 2026-08-30T10:13:56+00:00 /workspace/tiera-rerun-fixed-moonep/ultraep/tierbmem
|
| 24 |
OK moonep tierbmem 16 2026-08-30T10:14:05+00:00 2026-08-30T10:16:46+00:00 /workspace/tiera-rerun-fixed-moonep/moonep/tierbmem
|
|
|
|
|
|
|
|
|
|
|
|
| 22 |
OK echo tierbmem 16 2026-08-30T10:08:07+00:00 2026-08-30T10:11:03+00:00 /workspace/tiera-rerun-fixed-moonep/echo/tierbmem
|
| 23 |
OK ultraep tierbmem 16 2026-08-30T10:11:14+00:00 2026-08-30T10:13:56+00:00 /workspace/tiera-rerun-fixed-moonep/ultraep/tierbmem
|
| 24 |
OK moonep tierbmem 16 2026-08-30T10:14:05+00:00 2026-08-30T10:16:46+00:00 /workspace/tiera-rerun-fixed-moonep/moonep/tierbmem
|
| 25 |
+
OK echo tierbmem 16 2026-08-30T10:17:11+00:00 2026-08-30T10:20:07+00:00 /workspace/tiera-rerun-fixed-moonep/echo/tierbmem
|
| 26 |
+
OK ultraep tierbmem 16 2026-08-30T10:20:18+00:00 2026-08-30T10:23:00+00:00 /workspace/tiera-rerun-fixed-moonep/ultraep/tierbmem
|
| 27 |
+
OK moonep tierbmem 16 2026-08-30T10:23:07+00:00 2026-08-30T10:25:51+00:00 /workspace/tiera-rerun-fixed-moonep/moonep/tierbmem
|
tiera-rerun-fixed-moonep/study.log
CHANGED
|
@@ -3178,3 +3178,372 @@ driver_repo: /workspace/blackwell_run_scripts @ 48db41e
|
|
| 3178 |
=== STAGE 8 -- Tier B allocator dumps (static CG capacities + dynamic no-CG)
|
| 3179 |
|
| 3180 |
=== STEP echo__tierbmem__gbs16__G4
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 3178 |
=== STAGE 8 -- Tier B allocator dumps (static CG capacities + dynamic no-CG)
|
| 3179 |
|
| 3180 |
=== STEP echo__tierbmem__gbs16__G4
|
| 3181 |
+
==============================================================
|
| 3182 |
+
CAMPAIGN: echo backend=echo phase=tierbmem
|
| 3183 |
+
Megatron: /workspace/Megatron-MoE-Echo @ better-memory-dump-support
|
| 3184 |
+
GBS=16 seq=4096 repeats=3 nsys=0
|
| 3185 |
+
results -> /workspace/tiera-rerun-fixed-moonep/echo/tierbmem
|
| 3186 |
+
==============================================================
|
| 3187 |
+
--------------------------------------------------------------------------
|
| 3188 |
+
[10:17:26] CELL G4__echo__matched__seq4096__gbs16__cap1p095703125__memiteration__it6 (slots=16)
|
| 3189 |
+
[guard] iter 4/6 (70s)
|
| 3190 |
+
[guard] iter 6/6 (80s)
|
| 3191 |
+
--------------------------------------------------------------------------
|
| 3192 |
+
[10:18:46] CELL G4__echo__matched__seq4096__gbs16__capdynamic__nocg__memiteration__it5 (slots=16)
|
| 3193 |
+
[guard] iter 1/5 (70s)
|
| 3194 |
+
[guard] iter 5/5 (80s)
|
| 3195 |
+
[guard] tearing down process group (outcome=DONE)
|
| 3196 |
+
--------------------------------------------------------------------------
|
| 3197 |
+
|
| 3198 |
+
==============================================================
|
| 3199 |
+
SWEEP COMPLETE ok=2 failed=0
|
| 3200 |
+
==============================================================
|
| 3201 |
+
config backend policy slots seq gbs cap recompute rep status iter_ms_mean iter_ms_min peak_mib run_dir
|
| 3202 |
+
G4 echo matched 16 4096 16 1.095703125 none 1 OK 501.20 501.20 168261 /workspace/tiera-rerun-fixed-moonep/echo/tierbmem/tierbmem_gbs16/G4__echo__matched__seq4096__gbs16__cap1p095703125__memiteration__it6/rep1
|
| 3203 |
+
G4 echo matched 16 4096 16 dynamic none 1 OK NA NA 125233 /workspace/tiera-rerun-fixed-moonep/echo/tierbmem/tierbmem_gbs16/G4__echo__matched__seq4096__gbs16__capdynamic__nocg__memiteration__it5/rep1
|
| 3204 |
+
|
| 3205 |
+
Results: /workspace/tiera-rerun-fixed-moonep/echo/tierbmem/tierbmem_gbs16/results.tsv
|
| 3206 |
+
Roll up: python3 /workspace/blackwell_run_scripts/granularity/tools/collect.py /workspace/tiera-rerun-fixed-moonep/echo/tierbmem/tierbmem_gbs16
|
| 3207 |
+
==============================================================
|
| 3208 |
+
8 memory-dump runs -> /workspace/tiera-rerun-fixed-moonep/echo/tierbmem/tierbmem_gbs16/memory_summary.tsv
|
| 3209 |
+
model backend cg allocation capacity status peak GiB dumps
|
| 3210 |
+
----------------------------------------------------------------------------------
|
| 3211 |
+
G1 echo 1 static 2.40625 OK 157.9 48/48
|
| 3212 |
+
G1 echo 0 dynamic dynamic OK 112.6 40/40
|
| 3213 |
+
G2 echo 1 static1.08203125 OK 142.5 48/48
|
| 3214 |
+
G2 echo 0 dynamic dynamic OK 115.1 40/40
|
| 3215 |
+
G3 echo 1 static1.072265625 OK 144.5 48/48
|
| 3216 |
+
G3 echo 0 dynamic dynamic OK 116.1 40/40
|
| 3217 |
+
G4 echo 1 static1.095703125 OK 164.3 48/48
|
| 3218 |
+
G4 echo 0 dynamic dynamic OK 122.3 40/40
|
| 3219 |
+
|
| 3220 |
+
ECHO/UltraEP note: graph-on uses static capacity + device-initiated GEMM while graph-off uses dynamic allocation + host-derived sizes. Their difference is not labelled as pure graph cost.
|
| 3221 |
+
|
| 3222 |
+
TIER B MEMORY DUMPS -- how to read this
|
| 3223 |
+
---------------------------------------
|
| 3224 |
+
ECHO / UltraEP:
|
| 3225 |
+
cap1p00__memiteration__it6 graph on, static C=1.00
|
| 3226 |
+
cap1p125__memiteration__it6 graph on, static C=1.125
|
| 3227 |
+
cap1p25__memiteration__it6 graph on, static C=1.25
|
| 3228 |
+
capdynamic__nocg__memiteration__it5
|
| 3229 |
+
graph off, no capacity flag, dynamic allocation
|
| 3230 |
+
|
| 3231 |
+
MoonEP:
|
| 3232 |
+
capnone__memiteration__it6 graph on, no capacity factor
|
| 3233 |
+
capnone__nocg__memiteration__it5 graph off, no capacity factor
|
| 3234 |
+
|
| 3235 |
+
Every successful cell contains:
|
| 3236 |
+
train.log Megatron and OOM diagnostics
|
| 3237 |
+
mem_samples.csv driver-level nvidia-smi samples every 2 seconds
|
| 3238 |
+
run.env exact branch/config/capacity provenance
|
| 3239 |
+
memory_snapshots/*.pickle 48 captured / 40 eager allocator histories
|
| 3240 |
+
memory_snapshots/*.json matching summaries and before/after deltas
|
| 3241 |
+
|
| 3242 |
+
The graph-on/static vs graph-off/dynamic difference is intentionally NOT called
|
| 3243 |
+
"graph cost": capacity allocation policy and device-initiated grouped GEMM also
|
| 3244 |
+
change. Compare the three graph-on rows to measure static-capacity sensitivity,
|
| 3245 |
+
and use the dynamic row as the natural eager-allocation reference.
|
| 3246 |
+
|
| 3247 |
+
Campaign results: /workspace/tiera-rerun-fixed-moonep/echo/tierbmem
|
| 3248 |
+
Roll up: python3 /workspace/blackwell_run_scripts/granularity/tools/collect.py /workspace/tiera-rerun-fixed-moonep/echo/tierbmem --compare
|
| 3249 |
+
Already logged in to Hugging Face as: AbhishekGhosh1998
|
| 3250 |
+
uploaded: /workspace/tiera-rerun-fixed-moonep/echo/tierbmem -> MoE-CUDA-Graph-Analysis/tierb-b200-gbs8/tiera-rerun-fixed-moonep/echo/tierbmem
|
| 3251 |
+
https://huggingface.co/MoE-CUDA-Graph-Analysis/tierb-b200-gbs8/tree/main
|
| 3252 |
+
|
| 3253 |
+
=== STEP ultraep__tierbmem__gbs16__G4
|
| 3254 |
+
==============================================================
|
| 3255 |
+
CAMPAIGN: ultraep backend=ultraep phase=tierbmem
|
| 3256 |
+
Megatron: /workspace/Megatron-UltraEP @ ultraep-better-memory-dump-support
|
| 3257 |
+
GBS=16 seq=4096 repeats=3 nsys=0
|
| 3258 |
+
results -> /workspace/tiera-rerun-fixed-moonep/ultraep/tierbmem
|
| 3259 |
+
==============================================================
|
| 3260 |
+
--------------------------------------------------------------------------
|
| 3261 |
+
[10:20:33] CELL G4__ultraep__matched__seq4096__gbs16__cap1p095703125__memiteration__it6 (slots=16)
|
| 3262 |
+
[guard] iter 3/6 (60s)
|
| 3263 |
+
[guard] iter 6/6 (70s)
|
| 3264 |
+
[guard] tearing down process group (outcome=DONE)
|
| 3265 |
+
--------------------------------------------------------------------------
|
| 3266 |
+
[10:21:46] CELL G4__ultraep__matched__seq4096__gbs16__capdynamic__nocg__memiteration__it5 (slots=16)
|
| 3267 |
+
[guard] iter 1/5 (60s)
|
| 3268 |
+
[guard] iter 5/5 (70s)
|
| 3269 |
+
[guard] tearing down process group (outcome=DONE)
|
| 3270 |
+
--------------------------------------------------------------------------
|
| 3271 |
+
|
| 3272 |
+
==============================================================
|
| 3273 |
+
SWEEP COMPLETE ok=2 failed=0
|
| 3274 |
+
==============================================================
|
| 3275 |
+
config backend policy slots seq gbs cap recompute rep status iter_ms_mean iter_ms_min peak_mib run_dir
|
| 3276 |
+
G4 ultraep matched 16 4096 16 1.095703125 none 1 OK 498.70 498.70 132437 /workspace/tiera-rerun-fixed-moonep/ultraep/tierbmem/tierbmem_gbs16/G4__ultraep__matched__seq4096__gbs16__cap1p095703125__memiteration__it6/rep1
|
| 3277 |
+
G4 ultraep matched 16 4096 16 dynamic none 1 OK NA NA 120957 /workspace/tiera-rerun-fixed-moonep/ultraep/tierbmem/tierbmem_gbs16/G4__ultraep__matched__seq4096__gbs16__capdynamic__nocg__memiteration__it5/rep1
|
| 3278 |
+
|
| 3279 |
+
Results: /workspace/tiera-rerun-fixed-moonep/ultraep/tierbmem/tierbmem_gbs16/results.tsv
|
| 3280 |
+
Roll up: python3 /workspace/blackwell_run_scripts/granularity/tools/collect.py /workspace/tiera-rerun-fixed-moonep/ultraep/tierbmem/tierbmem_gbs16
|
| 3281 |
+
==============================================================
|
| 3282 |
+
8 memory-dump runs -> /workspace/tiera-rerun-fixed-moonep/ultraep/tierbmem/tierbmem_gbs16/memory_summary.tsv
|
| 3283 |
+
model backend cg allocation capacity status peak GiB dumps
|
| 3284 |
+
----------------------------------------------------------------------------------
|
| 3285 |
+
G1 ultraep 1 static 2.40625 OK 126.4 48/48
|
| 3286 |
+
G1 ultraep 0 dynamic dynamic OK 106.7 40/40
|
| 3287 |
+
G2 ultraep 1 static1.08203125 OK 109.7 48/48
|
| 3288 |
+
G2 ultraep 0 dynamic dynamic OK 105.7 40/40
|
| 3289 |
+
G3 ultraep 1 static1.072265625 OK 111.3 48/48
|
| 3290 |
+
G3 ultraep 0 dynamic dynamic OK 109.1 40/40
|
| 3291 |
+
G4 ultraep 1 static1.095703125 OK 129.3 48/48
|
| 3292 |
+
G4 ultraep 0 dynamic dynamic OK 118.1 40/40
|
| 3293 |
+
|
| 3294 |
+
ECHO/UltraEP note: graph-on uses static capacity + device-initiated GEMM while graph-off uses dynamic allocation + host-derived sizes. Their difference is not labelled as pure graph cost.
|
| 3295 |
+
|
| 3296 |
+
TIER B MEMORY DUMPS -- how to read this
|
| 3297 |
+
---------------------------------------
|
| 3298 |
+
ECHO / UltraEP:
|
| 3299 |
+
cap1p00__memiteration__it6 graph on, static C=1.00
|
| 3300 |
+
cap1p125__memiteration__it6 graph on, static C=1.125
|
| 3301 |
+
cap1p25__memiteration__it6 graph on, static C=1.25
|
| 3302 |
+
capdynamic__nocg__memiteration__it5
|
| 3303 |
+
graph off, no capacity flag, dynamic allocation
|
| 3304 |
+
|
| 3305 |
+
MoonEP:
|
| 3306 |
+
capnone__memiteration__it6 graph on, no capacity factor
|
| 3307 |
+
capnone__nocg__memiteration__it5 graph off, no capacity factor
|
| 3308 |
+
|
| 3309 |
+
Every successful cell contains:
|
| 3310 |
+
train.log Megatron and OOM diagnostics
|
| 3311 |
+
mem_samples.csv driver-level nvidia-smi samples every 2 seconds
|
| 3312 |
+
run.env exact branch/config/capacity provenance
|
| 3313 |
+
memory_snapshots/*.pickle 48 captured / 40 eager allocator histories
|
| 3314 |
+
memory_snapshots/*.json matching summaries and before/after deltas
|
| 3315 |
+
|
| 3316 |
+
The graph-on/static vs graph-off/dynamic difference is intentionally NOT called
|
| 3317 |
+
"graph cost": capacity allocation policy and device-initiated grouped GEMM also
|
| 3318 |
+
change. Compare the three graph-on rows to measure static-capacity sensitivity,
|
| 3319 |
+
and use the dynamic row as the natural eager-allocation reference.
|
| 3320 |
+
|
| 3321 |
+
Campaign results: /workspace/tiera-rerun-fixed-moonep/ultraep/tierbmem
|
| 3322 |
+
Roll up: python3 /workspace/blackwell_run_scripts/granularity/tools/collect.py /workspace/tiera-rerun-fixed-moonep/ultraep/tierbmem --compare
|
| 3323 |
+
Already logged in to Hugging Face as: AbhishekGhosh1998
|
| 3324 |
+
uploaded: /workspace/tiera-rerun-fixed-moonep/ultraep/tierbmem -> MoE-CUDA-Graph-Analysis/tierb-b200-gbs8/tiera-rerun-fixed-moonep/ultraep/tierbmem
|
| 3325 |
+
https://huggingface.co/MoE-CUDA-Graph-Analysis/tierb-b200-gbs8/tree/main
|
| 3326 |
+
|
| 3327 |
+
=== STEP moonep__tierbmem__gbs16__G4
|
| 3328 |
+
==============================================================
|
| 3329 |
+
CAMPAIGN: moonep backend=moonep phase=tierbmem
|
| 3330 |
+
Megatron: /workspace/Megatron-MoE-Echo @ moonep_backend_async_grad_reduce_better_memory_dump_support
|
| 3331 |
+
GBS=16 seq=4096 repeats=3 nsys=0
|
| 3332 |
+
results -> /workspace/tiera-rerun-fixed-moonep/moonep/tierbmem
|
| 3333 |
+
==============================================================
|
| 3334 |
+
--------------------------------------------------------------------------
|
| 3335 |
+
[10:23:22] CELL G4__moonep__matched__seq4096__gbs16__capnone__async__memiteration__it6 (slots=16)
|
| 3336 |
+
[guard] iter 2/6 (60s)
|
| 3337 |
+
[guard] iter 6/6 (70s)
|
| 3338 |
+
[guard] tearing down process group (outcome=DONE)
|
| 3339 |
+
--------------------------------------------------------------------------
|
| 3340 |
+
[10:24:36] CELL G4__moonep__matched__seq4096__gbs16__capnone__nocg__async__memiteration__it5 (slots=16)
|
| 3341 |
+
[guard] iter 5/5 (70s)
|
| 3342 |
+
[guard] tearing down process group (outcome=DONE)
|
| 3343 |
+
--------------------------------------------------------------------------
|
| 3344 |
+
|
| 3345 |
+
==============================================================
|
| 3346 |
+
SWEEP COMPLETE ok=2 failed=0
|
| 3347 |
+
==============================================================
|
| 3348 |
+
config backend policy slots seq gbs cap recompute rep status iter_ms_mean iter_ms_min peak_mib run_dir
|
| 3349 |
+
G4 moonep matched 16 4096 16 none none 1 OK 810.50 810.50 106029 /workspace/tiera-rerun-fixed-moonep/moonep/tierbmem/tierbmem_gbs16/G4__moonep__matched__seq4096__gbs16__capnone__async__memiteration__it6/rep1
|
| 3350 |
+
G4 moonep matched 16 4096 16 none none 1 OK NA NA 105101 /workspace/tiera-rerun-fixed-moonep/moonep/tierbmem/tierbmem_gbs16/G4__moonep__matched__seq4096__gbs16__capnone__nocg__async__memiteration__it5/rep1
|
| 3351 |
+
|
| 3352 |
+
Results: /workspace/tiera-rerun-fixed-moonep/moonep/tierbmem/tierbmem_gbs16/results.tsv
|
| 3353 |
+
Roll up: python3 /workspace/blackwell_run_scripts/granularity/tools/collect.py /workspace/tiera-rerun-fixed-moonep/moonep/tierbmem/tierbmem_gbs16
|
| 3354 |
+
==============================================================
|
| 3355 |
+
8 memory-dump runs -> /workspace/tiera-rerun-fixed-moonep/moonep/tierbmem/tierbmem_gbs16/memory_summary.tsv
|
| 3356 |
+
model backend cg allocation capacity status peak GiB dumps
|
| 3357 |
+
----------------------------------------------------------------------------------
|
| 3358 |
+
G1 moonep 1 none none OK 94.6 48/48
|
| 3359 |
+
G1 moonep 0 none none OK 90.4 40/40
|
| 3360 |
+
G2 moonep 1 none none OK 95.4 48/48
|
| 3361 |
+
G2 moonep 0 none none OK 93.2 40/40
|
| 3362 |
+
G3 moonep 1 none none OK 97.1 48/48
|
| 3363 |
+
G3 moonep 0 none none OK 96.3 40/40
|
| 3364 |
+
G4 moonep 1 none none OK 103.5 48/48
|
| 3365 |
+
G4 moonep 0 none none OK 102.6 40/40
|
| 3366 |
+
|
| 3367 |
+
ECHO/UltraEP note: graph-on uses static capacity + device-initiated GEMM while graph-off uses dynamic allocation + host-derived sizes. Their difference is not labelled as pure graph cost.
|
| 3368 |
+
|
| 3369 |
+
TIER B MEMORY DUMPS -- how to read this
|
| 3370 |
+
---------------------------------------
|
| 3371 |
+
ECHO / UltraEP:
|
| 3372 |
+
cap1p00__memiteration__it6 graph on, static C=1.00
|
| 3373 |
+
cap1p125__memiteration__it6 graph on, static C=1.125
|
| 3374 |
+
cap1p25__memiteration__it6 graph on, static C=1.25
|
| 3375 |
+
capdynamic__nocg__memiteration__it5
|
| 3376 |
+
graph off, no capacity flag, dynamic allocation
|
| 3377 |
+
|
| 3378 |
+
MoonEP:
|
| 3379 |
+
capnone__memiteration__it6 graph on, no capacity factor
|
| 3380 |
+
capnone__nocg__memiteration__it5 graph off, no capacity factor
|
| 3381 |
+
|
| 3382 |
+
Every successful cell contains:
|
| 3383 |
+
train.log Megatron and OOM diagnostics
|
| 3384 |
+
mem_samples.csv driver-level nvidia-smi samples every 2 seconds
|
| 3385 |
+
run.env exact branch/config/capacity provenance
|
| 3386 |
+
memory_snapshots/*.pickle 48 captured / 40 eager allocator histories
|
| 3387 |
+
memory_snapshots/*.json matching summaries and before/after deltas
|
| 3388 |
+
|
| 3389 |
+
The graph-on/static vs graph-off/dynamic difference is intentionally NOT called
|
| 3390 |
+
"graph cost": capacity allocation policy and device-initiated grouped GEMM also
|
| 3391 |
+
change. Compare the three graph-on rows to measure static-capacity sensitivity,
|
| 3392 |
+
and use the dynamic row as the natural eager-allocation reference.
|
| 3393 |
+
|
| 3394 |
+
Campaign results: /workspace/tiera-rerun-fixed-moonep/moonep/tierbmem
|
| 3395 |
+
Roll up: python3 /workspace/blackwell_run_scripts/granularity/tools/collect.py /workspace/tiera-rerun-fixed-moonep/moonep/tierbmem --compare
|
| 3396 |
+
Already logged in to Hugging Face as: AbhishekGhosh1998
|
| 3397 |
+
uploaded: /workspace/tiera-rerun-fixed-moonep/moonep/tierbmem -> MoE-CUDA-Graph-Analysis/tierb-b200-gbs8/tiera-rerun-fixed-moonep/moonep/tierbmem
|
| 3398 |
+
https://huggingface.co/MoE-CUDA-Graph-Analysis/tierb-b200-gbs8/tree/main
|
| 3399 |
+
|
| 3400 |
+
=== ROLL-UP
|
| 3401 |
+
140 runs ok=128 oom=0 -> /workspace/tiera-rerun-fixed-moonep/collected_all.tsv
|
| 3402 |
+
|
| 3403 |
+
WARNING: 11 run(s) mention capacity saturation. A run that drops assignments does less expert work and must not be quoted as a win:
|
| 3404 |
+
ultraep/tierbprobe/tierbprobe_gbs16/G1__ultraep__matched__seq4096__gbs16__cap1p25/rep1: WARNING:megatron.training.drop_debug_utils:[MOE_DROP_DEBUG][iteration_summary] global_rank=0 iteration=0 records=32 total_expected_recv=283999 total_dropped_tok
|
| 3405 |
+
ultraep/tierbprobe/tierbprobe_gbs16/G1__ultraep__matched__seq4096__gbs16__cap1p8125/rep1: WARNING:megatron.training.drop_debug_utils:[MOE_DROP_DEBUG][iteration_summary] global_rank=0 iteration=0 records=32 total_expected_recv=284540 total_dropped_tok
|
| 3406 |
+
ultraep/tierbprobe/tierbprobe_gbs16/G2__ultraep__matched__seq4096__gbs16__cap1p0625/rep1: WARNING:megatron.training.drop_debug_utils:[MOE_DROP_DEBUG][iteration_summary] global_rank=0 iteration=0 records=32 total_expected_recv=1048800 total_dropped_to
|
| 3407 |
+
ultraep/tierbprobe/tierbprobe_gbs16/G2__ultraep__matched__seq4096__gbs16__cap1p25/rep1: WARNING:megatron.training.drop_debug_utils:[MOE_DROP_DEBUG][iteration_summary] global_rank=0 iteration=0 records=32 total_expected_recv=1048800 total_dropped_to
|
| 3408 |
+
ultraep/tierbprobe/tierbprobe_gbs16/G2__ultraep__matched__seq4096__gbs16__cap1p25__forcebal/rep1: WARNING:megatron.training.drop_debug_utils:[MOE_DROP_DEBUG][iteration_summary] global_rank=1 iteration=0 records=32 total_expected_recv=1047984 total_dropped_to
|
| 3409 |
+
|
| 3410 |
+
(excluded 14 nsys-traced run(s) from the timing comparison)
|
| 3411 |
+
|
| 3412 |
+
phase config seq gbs policy rc cg asy backend iter_ms vs_best
|
| 3413 |
+
-------------------------------------------------------------------------------------
|
| 3414 |
+
a1 G1 4096 16 one none 1 0 echo 466.68 1.00x
|
| 3415 |
+
ultraep 891.00 1.91x
|
| 3416 |
+
|
| 3417 |
+
a1 G1 4096 16 one none 1 1 moonep 480.30 1.00x
|
| 3418 |
+
|
| 3419 |
+
a1 G2 4096 16 one none 1 0 echo 394.28 1.00x
|
| 3420 |
+
ultraep 466.65 1.18x
|
| 3421 |
+
|
| 3422 |
+
a1 G3 4096 16 one none 1 0 echo 412.83 1.00x
|
| 3423 |
+
ultraep 447.84 1.08x
|
| 3424 |
+
|
| 3425 |
+
a1 G4 4096 16 one none 1 0 echo 461.03 1.00x
|
| 3426 |
+
ultraep 467.57 1.01x
|
| 3427 |
+
|
| 3428 |
+
a2 G1 4096 16 matched none 1 0 echo 466.91 1.00x
|
| 3429 |
+
ultraep 891.92 1.91x
|
| 3430 |
+
|
| 3431 |
+
a2 G1 4096 16 matched none 1 1 moonep 481.37 1.00x
|
| 3432 |
+
|
| 3433 |
+
a2 G2 4096 16 matched none 1 0 echo 415.21 1.00x
|
| 3434 |
+
ultraep 492.54 1.19x
|
| 3435 |
+
|
| 3436 |
+
a2 G2 4096 16 matched none 1 1 moonep 432.02 1.00x
|
| 3437 |
+
|
| 3438 |
+
a2 G3 4096 16 matched none 1 0 echo 437.35 1.00x
|
| 3439 |
+
ultraep 459.60 1.05x
|
| 3440 |
+
|
| 3441 |
+
a2 G3 4096 16 matched none 1 1 moonep 559.88 1.00x
|
| 3442 |
+
|
| 3443 |
+
a2 G4 4096 16 matched none 1 0 ultraep 482.64 1.00x
|
| 3444 |
+
echo 486.54 1.01x
|
| 3445 |
+
|
| 3446 |
+
a2 G4 4096 16 matched none 1 1 moonep 783.34 1.00x
|
| 3447 |
+
|
| 3448 |
+
ponly PONLY 4096 16 one none 1 0 echo 174.81 1.00x
|
| 3449 |
+
ultraep 176.76 1.01x
|
| 3450 |
+
|
| 3451 |
+
ponly PONLY 4096 16 one none 1 1 moonep 165.22 1.00x
|
| 3452 |
+
|
| 3453 |
+
ponly PONLY 8192 16 one none 1 0 ultraep 292.62 1.00x
|
| 3454 |
+
echo 316.70 1.08x
|
| 3455 |
+
|
| 3456 |
+
ponly PONLY 8192 16 one none 1 1 moonep 300.61 1.00x
|
| 3457 |
+
|
| 3458 |
+
tierbmem G1 4096 16 matched none 1 0 echo 474.40 1.00x
|
| 3459 |
+
ultraep 909.90 1.92x
|
| 3460 |
+
|
| 3461 |
+
tierbmem G1 4096 16 matched none 1 1 moonep 497.40 1.00x
|
| 3462 |
+
|
| 3463 |
+
tierbmem G2 4096 16 matched none 1 0 echo 430.10 1.00x
|
| 3464 |
+
ultraep 475.40 1.11x
|
| 3465 |
+
|
| 3466 |
+
tierbmem G2 4096 16 matched none 1 1 moonep 439.70 1.00x
|
| 3467 |
+
|
| 3468 |
+
tierbmem G3 4096 16 matched none 1 0 echo 452.40 1.00x
|
| 3469 |
+
ultraep 477.80 1.06x
|
| 3470 |
+
|
| 3471 |
+
tierbmem G3 4096 16 matched none 1 1 moonep 578.30 1.00x
|
| 3472 |
+
|
| 3473 |
+
tierbmem G4 4096 16 matched none 1 0 ultraep 498.70 1.00x
|
| 3474 |
+
echo 501.20 1.01x
|
| 3475 |
+
|
| 3476 |
+
tierbmem G4 4096 16 matched none 1 1 moonep 810.50 1.00x
|
| 3477 |
+
|
| 3478 |
+
tierbprobeG1 4096 16 matched none 0 0 echo 671.53 1.00x
|
| 3479 |
+
ultraep 904.16 1.35x
|
| 3480 |
+
|
| 3481 |
+
tierbprobeG2 4096 16 matched none 0 0 ultraep 512.37 1.00x
|
| 3482 |
+
echo 654.76 1.28x
|
| 3483 |
+
|
| 3484 |
+
tierbprobeG3 4096 16 matched none 0 0 ultraep 512.31 1.00x
|
| 3485 |
+
echo 689.13 1.35x
|
| 3486 |
+
|
| 3487 |
+
tierbprobeG4 4096 16 matched none 0 0 ultraep 600.19 1.00x
|
| 3488 |
+
echo 790.97 1.32x
|
| 3489 |
+
|
| 3490 |
+
21 probe runs -> /workspace/tiera-rerun-fixed-moonep/probe_summary_all.tsv
|
| 3491 |
+
run arm cap rawExpI preRankI postRankI drop% reqCap
|
| 3492 |
+
--------------------------------------------------------------------------------------------------------------------
|
| 3493 |
+
echo/tierbprobe/tierbprobe_gbs16/G1__echo__matched__ natural 2.00 3.6198* 3.6198 1.0827 1.9062
|
| 3494 |
+
echo/tierbprobe/tierbprobe_gbs16/G2__echo__matched__ natural 1.08203125 3.8720* 2.1703 1.0000 1.0273
|
| 3495 |
+
echo/tierbprobe/tierbprobe_gbs16/G2__echo__matched__ forced 1.08203125 1.0201 1.0069 1.0000 1.0234
|
| 3496 |
+
echo/tierbprobe/tierbprobe_gbs16/G2__echo__matched__ natural 2.00 3.8724* 2.1875 1.0000 1.0273
|
| 3497 |
+
echo/tierbprobe/tierbprobe_gbs16/G3__echo__matched__ natural 1.072265625 3.9096* 1.7838 1.0000 1.0195
|
| 3498 |
+
echo/tierbprobe/tierbprobe_gbs16/G3__echo__matched__ forced 1.072265625 1.0231 1.0048 1.0000 1.0176
|
| 3499 |
+
echo/tierbprobe/tierbprobe_gbs16/G3__echo__matched__ natural 2.00 3.9096* 1.7916 1.0000 1.0195
|
| 3500 |
+
echo/tierbprobe/tierbprobe_gbs16/G4__echo__matched__ natural 1.068359375 3.9303* 1.5507 1.0000 1.0166
|
| 3501 |
+
echo/tierbprobe/tierbprobe_gbs16/G4__echo__matched__ forced 1.068359375 1.0249 1.0034 1.0000 1.0137
|
| 3502 |
+
echo/tierbprobe/tierbprobe_gbs16/G4__echo__matched__ natural 2.00 3.9301* 1.5543 1.0000 1.0166
|
| 3503 |
+
ultraep/tierbprobe/tierbprobe_gbs16/G1__ultraep__mat natural 1.25 3.6085 1.0640 2.8918 1.7246
|
| 3504 |
+
ultraep/tierbprobe/tierbprobe_gbs16/G1__ultraep__mat natural 1.8125 3.6152 1.0747 1.5460 2.3091
|
| 3505 |
+
ultraep/tierbprobe/tierbprobe_gbs16/G2__ultraep__mat natural 1.0625 2.2032 1.0100 0.0000 1.0101
|
| 3506 |
+
ultraep/tierbprobe/tierbprobe_gbs16/G2__ultraep__mat natural 1.25 2.2136 1.0100 0.0000 1.0101
|
| 3507 |
+
ultraep/tierbprobe/tierbprobe_gbs16/G2__ultraep__mat forced 1.25 1.0069 1.0067 0.0000 1.0100
|
| 3508 |
+
ultraep/tierbprobe/tierbprobe_gbs16/G3__ultraep__mat natural 1.0625 1.7842 1.0100 0.0000 1.0101
|
| 3509 |
+
ultraep/tierbprobe/tierbprobe_gbs16/G3__ultraep__mat natural 1.25 1.7752 1.0100 0.0000 1.0101
|
| 3510 |
+
ultraep/tierbprobe/tierbprobe_gbs16/G3__ultraep__mat forced 1.25 1.0047 1.0047 0.0000 1.0100
|
| 3511 |
+
ultraep/tierbprobe/tierbprobe_gbs16/G4__ultraep__mat natural 1.095703125 1.5612 1.0100 0.0000 1.0429
|
| 3512 |
+
ultraep/tierbprobe/tierbprobe_gbs16/G4__ultraep__mat natural 1.25 1.5504 1.0100 0.0000 1.0429
|
| 3513 |
+
ultraep/tierbprobe/tierbprobe_gbs16/G4__ultraep__mat forced 1.25 1.0033 1.0033 0.0000 1.0071
|
| 3514 |
+
|
| 3515 |
+
* At least one globally aggregated snapshot reached the E/K expert-Imax ceiling.
|
| 3516 |
+
Inspect raw_expert_imax_mean and hottest_expert_token_fraction_mean before generalising.
|
| 3517 |
+
|
| 3518 |
+
ECHO note: echo_logged_pre_rank_metric_valid=0 and echo_logged_load_reduction_valid=0 are intentional.
|
| 3519 |
+
The branch's built-in fields mix source-rank and destination-rank axes; this tool reconstructs both globally.
|
| 3520 |
+
24 memory-dump runs -> /workspace/tiera-rerun-fixed-moonep/memory_summary_all.tsv
|
| 3521 |
+
model backend cg allocation capacity status peak GiB dumps
|
| 3522 |
+
----------------------------------------------------------------------------------
|
| 3523 |
+
G1 echo 1 static 2.40625 OK 157.9 48/48
|
| 3524 |
+
G1 echo 0 dynamic dynamic OK 112.6 40/40
|
| 3525 |
+
G1 moonep 1 none none OK 94.6 48/48
|
| 3526 |
+
G1 moonep 0 none none OK 90.4 40/40
|
| 3527 |
+
G1 ultraep 1 static 2.40625 OK 126.4 48/48
|
| 3528 |
+
G1 ultraep 0 dynamic dynamic OK 106.7 40/40
|
| 3529 |
+
G2 echo 1 static1.08203125 OK 142.5 48/48
|
| 3530 |
+
G2 echo 0 dynamic dynamic OK 115.1 40/40
|
| 3531 |
+
G2 moonep 1 none none OK 95.4 48/48
|
| 3532 |
+
G2 moonep 0 none none OK 93.2 40/40
|
| 3533 |
+
G2 ultraep 1 static1.08203125 OK 109.7 48/48
|
| 3534 |
+
G2 ultraep 0 dynamic dynamic OK 105.7 40/40
|
| 3535 |
+
G3 echo 1 static1.072265625 OK 144.5 48/48
|
| 3536 |
+
G3 echo 0 dynamic dynamic OK 116.1 40/40
|
| 3537 |
+
G3 moonep 1 none none OK 97.1 48/48
|
| 3538 |
+
G3 moonep 0 none none OK 96.3 40/40
|
| 3539 |
+
G3 ultraep 1 static1.072265625 OK 111.3 48/48
|
| 3540 |
+
G3 ultraep 0 dynamic dynamic OK 109.1 40/40
|
| 3541 |
+
G4 echo 1 static1.095703125 OK 164.3 48/48
|
| 3542 |
+
G4 echo 0 dynamic dynamic OK 122.3 40/40
|
| 3543 |
+
G4 moonep 1 none none OK 103.5 48/48
|
| 3544 |
+
G4 moonep 0 none none OK 102.6 40/40
|
| 3545 |
+
G4 ultraep 1 static1.095703125 OK 129.3 48/48
|
| 3546 |
+
G4 ultraep 0 dynamic dynamic OK 118.1 40/40
|
| 3547 |
+
|
| 3548 |
+
ECHO/UltraEP note: graph-on uses static capacity + device-initiated GEMM while graph-off uses dynamic allocation + host-derived sizes. Their difference is not labelled as pure graph cost.
|
| 3549 |
+
Already logged in to Hugging Face as: AbhishekGhosh1998
|