granularity results tierb-fixed-moonep-full @ 2026-08-30T00:10:02Z from sp-aiinfra-interactive-aghosh-7jtjl-8xb200-moe-cg-0 (part 3)
Browse files
tierb-fixed-moonep-full/steps.tsv
CHANGED
|
@@ -1,3 +1,4 @@
|
|
| 1 |
status campaign phase gbs started ended result_dir
|
| 2 |
OK moonep tierbprobe 8 2026-08-29T23:33:50+00:00 2026-08-29T23:44:30+00:00 /workspace/tierb-fixed-moonep-full/moonep/tierbprobe
|
| 3 |
OK moonep tierb 8 2026-08-29T23:44:39+00:00 2026-08-29T23:59:52+00:00 /workspace/tierb-fixed-moonep-full/moonep/tierb
|
|
|
|
|
|
| 1 |
status campaign phase gbs started ended result_dir
|
| 2 |
OK moonep tierbprobe 8 2026-08-29T23:33:50+00:00 2026-08-29T23:44:30+00:00 /workspace/tierb-fixed-moonep-full/moonep/tierbprobe
|
| 3 |
OK moonep tierb 8 2026-08-29T23:44:39+00:00 2026-08-29T23:59:52+00:00 /workspace/tierb-fixed-moonep-full/moonep/tierb
|
| 4 |
+
OK moonep tierbmem 8 2026-08-29T23:59:59+00:00 2026-08-30T00:09:58+00:00 /workspace/tierb-fixed-moonep-full/moonep/tierbmem
|
tierb-fixed-moonep-full/study.log
CHANGED
|
@@ -362,3 +362,78 @@ driver_repo: /workspace/blackwell_run_scripts @ eec9aee
|
|
| 362 |
[guard] tearing down process group (outcome=DONE)
|
| 363 |
--------------------------------------------------------------------------
|
| 364 |
[00:06:27] CELL DSV2_LITE_L24__moonep__matched__seq8192__gbs8__capnone__nocg__async__memiteration__it5 (slots=8)
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 362 |
[guard] tearing down process group (outcome=DONE)
|
| 363 |
--------------------------------------------------------------------------
|
| 364 |
[00:06:27] CELL DSV2_LITE_L24__moonep__matched__seq8192__gbs8__capnone__nocg__async__memiteration__it5 (slots=8)
|
| 365 |
+
[guard] iter 5/5 (70s)
|
| 366 |
+
[guard] tearing down process group (outcome=DONE)
|
| 367 |
+
--------------------------------------------------------------------------
|
| 368 |
+
[00:07:42] CELL QWEN2_57B__moonep__matched__seq4096__gbs8__capnone__async__memiteration__it6 (slots=8)
|
| 369 |
+
[guard] iter 5/6 (60s)
|
| 370 |
+
[guard] iter 6/6 (70s)
|
| 371 |
+
[guard] tearing down process group (outcome=DONE)
|
| 372 |
+
--------------------------------------------------------------------------
|
| 373 |
+
[00:08:53] CELL QWEN2_57B__moonep__matched__seq4096__gbs8__capnone__nocg__async__memiteration__it5 (slots=8)
|
| 374 |
+
[guard] iter 5/5 (60s)
|
| 375 |
+
[guard] tearing down process group (outcome=DONE)
|
| 376 |
+
--------------------------------------------------------------------------
|
| 377 |
+
|
| 378 |
+
==============================================================
|
| 379 |
+
SWEEP COMPLETE ok=8 failed=0
|
| 380 |
+
==============================================================
|
| 381 |
+
config backend policy slots seq gbs cap recompute rep status iter_ms_mean iter_ms_min peak_mib run_dir
|
| 382 |
+
QWEN3_235B moonep matched 16 8192 8 none none 1 OK 341.70 341.70 120303 /workspace/tierb-fixed-moonep-full/moonep/tierbmem/tierbmem_gbs8/QWEN3_235B__moonep__matched__seq8192__gbs8__capnone__async__memiteration__it6/rep1
|
| 383 |
+
QWEN3_235B moonep matched 16 8192 8 none none 1 OK NA NA 120131 /workspace/tierb-fixed-moonep-full/moonep/tierbmem/tierbmem_gbs8/QWEN3_235B__moonep__matched__seq8192__gbs8__capnone__nocg__async__memiteration__it5/rep1
|
| 384 |
+
QWEN3_30B moonep matched 16 4096 8 none none 1 OK 435.30 435.30 116703 /workspace/tierb-fixed-moonep-full/moonep/tierbmem/tierbmem_gbs8/QWEN3_30B__moonep__matched__seq4096__gbs8__capnone__async__memiteration__it6/rep1
|
| 385 |
+
QWEN3_30B moonep matched 16 4096 8 none none 1 OK NA NA 116445 /workspace/tierb-fixed-moonep-full/moonep/tierbmem/tierbmem_gbs8/QWEN3_30B__moonep__matched__seq4096__gbs8__capnone__nocg__async__memiteration__it5/rep1
|
| 386 |
+
DSV2_LITE_L24 moonep matched 8 8192 8 none none 1 OK 622.80 622.80 133311 /workspace/tierb-fixed-moonep-full/moonep/tierbmem/tierbmem_gbs8/DSV2_LITE_L24__moonep__matched__seq8192__gbs8__capnone__async__memiteration__it6/rep1
|
| 387 |
+
DSV2_LITE_L24 moonep matched 8 8192 8 none none 1 OK NA NA 125157 /workspace/tierb-fixed-moonep-full/moonep/tierbmem/tierbmem_gbs8/DSV2_LITE_L24__moonep__matched__seq8192__gbs8__capnone__nocg__async__memiteration__it5/rep1
|
| 388 |
+
QWEN2_57B moonep matched 8 4096 8 none none 1 OK 196.60 196.60 96737 /workspace/tierb-fixed-moonep-full/moonep/tierbmem/tierbmem_gbs8/QWEN2_57B__moonep__matched__seq4096__gbs8__capnone__async__memiteration__it6/rep1
|
| 389 |
+
QWEN2_57B moonep matched 8 4096 8 none none 1 OK NA NA 95855 /workspace/tierb-fixed-moonep-full/moonep/tierbmem/tierbmem_gbs8/QWEN2_57B__moonep__matched__seq4096__gbs8__capnone__nocg__async__memiteration__it5/rep1
|
| 390 |
+
|
| 391 |
+
Results: /workspace/tierb-fixed-moonep-full/moonep/tierbmem/tierbmem_gbs8/results.tsv
|
| 392 |
+
Roll up: python3 /workspace/blackwell_run_scripts/granularity/tools/collect.py /workspace/tierb-fixed-moonep-full/moonep/tierbmem/tierbmem_gbs8
|
| 393 |
+
==============================================================
|
| 394 |
+
8 memory-dump runs -> /workspace/tierb-fixed-moonep-full/moonep/tierbmem/tierbmem_gbs8/memory_summary.tsv
|
| 395 |
+
model backend cg allocation capacity status peak GiB dumps
|
| 396 |
+
----------------------------------------------------------------------------------
|
| 397 |
+
DSV2_LITE_L24 moonep 1 none none OK 130.2 48/48
|
| 398 |
+
DSV2_LITE_L24 moonep 0 none none OK 122.2 40/40
|
| 399 |
+
QWEN2_57B moonep 1 none none OK 94.5 48/48
|
| 400 |
+
QWEN2_57B moonep 0 none none OK 93.6 40/40
|
| 401 |
+
QWEN3_235B moonep 1 none none OK 117.5 48/48
|
| 402 |
+
QWEN3_235B moonep 0 none none OK 117.3 40/40
|
| 403 |
+
QWEN3_30B moonep 1 none none OK 114.0 48/48
|
| 404 |
+
QWEN3_30B moonep 0 none none OK 113.7 40/40
|
| 405 |
+
|
| 406 |
+
ECHO/UltraEP note: graph-on uses static capacity + device-initiated GEMM while graph-off uses dynamic allocation + host-derived sizes. Their difference is not labelled as pure graph cost.
|
| 407 |
+
|
| 408 |
+
TIER B MEMORY DUMPS -- how to read this
|
| 409 |
+
---------------------------------------
|
| 410 |
+
ECHO / UltraEP:
|
| 411 |
+
cap1p00__memiteration__it6 graph on, static C=1.00
|
| 412 |
+
cap1p125__memiteration__it6 graph on, static C=1.125
|
| 413 |
+
cap1p25__memiteration__it6 graph on, static C=1.25
|
| 414 |
+
capdynamic__nocg__memiteration__it5
|
| 415 |
+
graph off, no capacity flag, dynamic allocation
|
| 416 |
+
|
| 417 |
+
MoonEP:
|
| 418 |
+
capnone__memiteration__it6 graph on, no capacity factor
|
| 419 |
+
capnone__nocg__memiteration__it5 graph off, no capacity factor
|
| 420 |
+
|
| 421 |
+
Every successful cell contains:
|
| 422 |
+
train.log Megatron and OOM diagnostics
|
| 423 |
+
mem_samples.csv driver-level nvidia-smi samples every 2 seconds
|
| 424 |
+
run.env exact branch/config/capacity provenance
|
| 425 |
+
memory_snapshots/*.pickle 48 captured / 40 eager allocator histories
|
| 426 |
+
memory_snapshots/*.json matching summaries and before/after deltas
|
| 427 |
+
|
| 428 |
+
The graph-on/static vs graph-off/dynamic difference is intentionally NOT called
|
| 429 |
+
"graph cost": capacity allocation policy and device-initiated grouped GEMM also
|
| 430 |
+
change. Compare the three graph-on rows to measure static-capacity sensitivity,
|
| 431 |
+
and use the dynamic row as the natural eager-allocation reference.
|
| 432 |
+
|
| 433 |
+
Campaign results: /workspace/tierb-fixed-moonep-full/moonep/tierbmem
|
| 434 |
+
Roll up: python3 /workspace/blackwell_run_scripts/granularity/tools/collect.py /workspace/tierb-fixed-moonep-full/moonep/tierbmem --compare
|
| 435 |
+
Already logged in to Hugging Face as: AbhishekGhosh1998
|
| 436 |
+
uploaded: /workspace/tierb-fixed-moonep-full/moonep/tierbmem -> MoE-CUDA-Graph-Analysis/tierb-b200-gbs8/tierb-fixed-moonep-full/moonep/tierbmem
|
| 437 |
+
https://huggingface.co/MoE-CUDA-Graph-Analysis/tierb-b200-gbs8/tree/main
|
| 438 |
+
|
| 439 |
+
=== ROLL-UP
|