AbhishekGhosh1998 commited on
Commit
f07f871
·
verified ·
1 Parent(s): 2d8909d

granularity results tierb-fixed-moonep-full @ 2026-08-30T00:10:02Z from sp-aiinfra-interactive-aghosh-7jtjl-8xb200-moe-cg-0 (part 3)

Browse files
tierb-fixed-moonep-full/steps.tsv CHANGED
@@ -1,3 +1,4 @@
1
  status campaign phase gbs started ended result_dir
2
  OK moonep tierbprobe 8 2026-08-29T23:33:50+00:00 2026-08-29T23:44:30+00:00 /workspace/tierb-fixed-moonep-full/moonep/tierbprobe
3
  OK moonep tierb 8 2026-08-29T23:44:39+00:00 2026-08-29T23:59:52+00:00 /workspace/tierb-fixed-moonep-full/moonep/tierb
 
 
1
  status campaign phase gbs started ended result_dir
2
  OK moonep tierbprobe 8 2026-08-29T23:33:50+00:00 2026-08-29T23:44:30+00:00 /workspace/tierb-fixed-moonep-full/moonep/tierbprobe
3
  OK moonep tierb 8 2026-08-29T23:44:39+00:00 2026-08-29T23:59:52+00:00 /workspace/tierb-fixed-moonep-full/moonep/tierb
4
+ OK moonep tierbmem 8 2026-08-29T23:59:59+00:00 2026-08-30T00:09:58+00:00 /workspace/tierb-fixed-moonep-full/moonep/tierbmem
tierb-fixed-moonep-full/study.log CHANGED
@@ -362,3 +362,78 @@ driver_repo: /workspace/blackwell_run_scripts @ eec9aee
362
  [guard] tearing down process group (outcome=DONE)
363
  --------------------------------------------------------------------------
364
  [00:06:27] CELL DSV2_LITE_L24__moonep__matched__seq8192__gbs8__capnone__nocg__async__memiteration__it5 (slots=8)
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
362
  [guard] tearing down process group (outcome=DONE)
363
  --------------------------------------------------------------------------
364
  [00:06:27] CELL DSV2_LITE_L24__moonep__matched__seq8192__gbs8__capnone__nocg__async__memiteration__it5 (slots=8)
365
+ [guard] iter 5/5 (70s)
366
+ [guard] tearing down process group (outcome=DONE)
367
+ --------------------------------------------------------------------------
368
+ [00:07:42] CELL QWEN2_57B__moonep__matched__seq4096__gbs8__capnone__async__memiteration__it6 (slots=8)
369
+ [guard] iter 5/6 (60s)
370
+ [guard] iter 6/6 (70s)
371
+ [guard] tearing down process group (outcome=DONE)
372
+ --------------------------------------------------------------------------
373
+ [00:08:53] CELL QWEN2_57B__moonep__matched__seq4096__gbs8__capnone__nocg__async__memiteration__it5 (slots=8)
374
+ [guard] iter 5/5 (60s)
375
+ [guard] tearing down process group (outcome=DONE)
376
+ --------------------------------------------------------------------------
377
+
378
+ ==============================================================
379
+ SWEEP COMPLETE ok=8 failed=0
380
+ ==============================================================
381
+ config backend policy slots seq gbs cap recompute rep status iter_ms_mean iter_ms_min peak_mib run_dir
382
+ QWEN3_235B moonep matched 16 8192 8 none none 1 OK 341.70 341.70 120303 /workspace/tierb-fixed-moonep-full/moonep/tierbmem/tierbmem_gbs8/QWEN3_235B__moonep__matched__seq8192__gbs8__capnone__async__memiteration__it6/rep1
383
+ QWEN3_235B moonep matched 16 8192 8 none none 1 OK NA NA 120131 /workspace/tierb-fixed-moonep-full/moonep/tierbmem/tierbmem_gbs8/QWEN3_235B__moonep__matched__seq8192__gbs8__capnone__nocg__async__memiteration__it5/rep1
384
+ QWEN3_30B moonep matched 16 4096 8 none none 1 OK 435.30 435.30 116703 /workspace/tierb-fixed-moonep-full/moonep/tierbmem/tierbmem_gbs8/QWEN3_30B__moonep__matched__seq4096__gbs8__capnone__async__memiteration__it6/rep1
385
+ QWEN3_30B moonep matched 16 4096 8 none none 1 OK NA NA 116445 /workspace/tierb-fixed-moonep-full/moonep/tierbmem/tierbmem_gbs8/QWEN3_30B__moonep__matched__seq4096__gbs8__capnone__nocg__async__memiteration__it5/rep1
386
+ DSV2_LITE_L24 moonep matched 8 8192 8 none none 1 OK 622.80 622.80 133311 /workspace/tierb-fixed-moonep-full/moonep/tierbmem/tierbmem_gbs8/DSV2_LITE_L24__moonep__matched__seq8192__gbs8__capnone__async__memiteration__it6/rep1
387
+ DSV2_LITE_L24 moonep matched 8 8192 8 none none 1 OK NA NA 125157 /workspace/tierb-fixed-moonep-full/moonep/tierbmem/tierbmem_gbs8/DSV2_LITE_L24__moonep__matched__seq8192__gbs8__capnone__nocg__async__memiteration__it5/rep1
388
+ QWEN2_57B moonep matched 8 4096 8 none none 1 OK 196.60 196.60 96737 /workspace/tierb-fixed-moonep-full/moonep/tierbmem/tierbmem_gbs8/QWEN2_57B__moonep__matched__seq4096__gbs8__capnone__async__memiteration__it6/rep1
389
+ QWEN2_57B moonep matched 8 4096 8 none none 1 OK NA NA 95855 /workspace/tierb-fixed-moonep-full/moonep/tierbmem/tierbmem_gbs8/QWEN2_57B__moonep__matched__seq4096__gbs8__capnone__nocg__async__memiteration__it5/rep1
390
+
391
+ Results: /workspace/tierb-fixed-moonep-full/moonep/tierbmem/tierbmem_gbs8/results.tsv
392
+ Roll up: python3 /workspace/blackwell_run_scripts/granularity/tools/collect.py /workspace/tierb-fixed-moonep-full/moonep/tierbmem/tierbmem_gbs8
393
+ ==============================================================
394
+ 8 memory-dump runs -> /workspace/tierb-fixed-moonep-full/moonep/tierbmem/tierbmem_gbs8/memory_summary.tsv
395
+ model backend cg allocation capacity status peak GiB dumps
396
+ ----------------------------------------------------------------------------------
397
+ DSV2_LITE_L24 moonep 1 none none OK 130.2 48/48
398
+ DSV2_LITE_L24 moonep 0 none none OK 122.2 40/40
399
+ QWEN2_57B moonep 1 none none OK 94.5 48/48
400
+ QWEN2_57B moonep 0 none none OK 93.6 40/40
401
+ QWEN3_235B moonep 1 none none OK 117.5 48/48
402
+ QWEN3_235B moonep 0 none none OK 117.3 40/40
403
+ QWEN3_30B moonep 1 none none OK 114.0 48/48
404
+ QWEN3_30B moonep 0 none none OK 113.7 40/40
405
+
406
+ ECHO/UltraEP note: graph-on uses static capacity + device-initiated GEMM while graph-off uses dynamic allocation + host-derived sizes. Their difference is not labelled as pure graph cost.
407
+
408
+ TIER B MEMORY DUMPS -- how to read this
409
+ ---------------------------------------
410
+ ECHO / UltraEP:
411
+ cap1p00__memiteration__it6 graph on, static C=1.00
412
+ cap1p125__memiteration__it6 graph on, static C=1.125
413
+ cap1p25__memiteration__it6 graph on, static C=1.25
414
+ capdynamic__nocg__memiteration__it5
415
+ graph off, no capacity flag, dynamic allocation
416
+
417
+ MoonEP:
418
+ capnone__memiteration__it6 graph on, no capacity factor
419
+ capnone__nocg__memiteration__it5 graph off, no capacity factor
420
+
421
+ Every successful cell contains:
422
+ train.log Megatron and OOM diagnostics
423
+ mem_samples.csv driver-level nvidia-smi samples every 2 seconds
424
+ run.env exact branch/config/capacity provenance
425
+ memory_snapshots/*.pickle 48 captured / 40 eager allocator histories
426
+ memory_snapshots/*.json matching summaries and before/after deltas
427
+
428
+ The graph-on/static vs graph-off/dynamic difference is intentionally NOT called
429
+ "graph cost": capacity allocation policy and device-initiated grouped GEMM also
430
+ change. Compare the three graph-on rows to measure static-capacity sensitivity,
431
+ and use the dynamic row as the natural eager-allocation reference.
432
+
433
+ Campaign results: /workspace/tierb-fixed-moonep-full/moonep/tierbmem
434
+ Roll up: python3 /workspace/blackwell_run_scripts/granularity/tools/collect.py /workspace/tierb-fixed-moonep-full/moonep/tierbmem --compare
435
+ Already logged in to Hugging Face as: AbhishekGhosh1998
436
+ uploaded: /workspace/tierb-fixed-moonep-full/moonep/tierbmem -> MoE-CUDA-Graph-Analysis/tierb-b200-gbs8/tierb-fixed-moonep-full/moonep/tierbmem
437
+ https://huggingface.co/MoE-CUDA-Graph-Analysis/tierb-b200-gbs8/tree/main
438
+
439
+ === ROLL-UP