granularity results tiera-rerun-fixed-moonep @ 2026-08-30T10:17:03Z from sp-aiinfra-interactive-aghosh-7jtjl-8xb200-moe-cg-0 (part 3)
Browse files
tiera-rerun-fixed-moonep/steps.tsv
CHANGED
|
@@ -19,3 +19,6 @@ OK moonep tierbtrace 16 2026-08-30T09:28:30+00:00 2026-08-30T09:45:26+00:00 /wor
|
|
| 19 |
OK echo tierbmem 16 2026-08-30T09:59:28+00:00 2026-08-30T10:02:20+00:00 /workspace/tiera-rerun-fixed-moonep/echo/tierbmem
|
| 20 |
OK ultraep tierbmem 16 2026-08-30T10:02:26+00:00 2026-08-30T10:05:04+00:00 /workspace/tiera-rerun-fixed-moonep/ultraep/tierbmem
|
| 21 |
OK moonep tierbmem 16 2026-08-30T10:05:11+00:00 2026-08-30T10:07:42+00:00 /workspace/tiera-rerun-fixed-moonep/moonep/tierbmem
|
|
|
|
|
|
|
|
|
|
|
|
| 19 |
OK echo tierbmem 16 2026-08-30T09:59:28+00:00 2026-08-30T10:02:20+00:00 /workspace/tiera-rerun-fixed-moonep/echo/tierbmem
|
| 20 |
OK ultraep tierbmem 16 2026-08-30T10:02:26+00:00 2026-08-30T10:05:04+00:00 /workspace/tiera-rerun-fixed-moonep/ultraep/tierbmem
|
| 21 |
OK moonep tierbmem 16 2026-08-30T10:05:11+00:00 2026-08-30T10:07:42+00:00 /workspace/tiera-rerun-fixed-moonep/moonep/tierbmem
|
| 22 |
+
OK echo tierbmem 16 2026-08-30T10:08:07+00:00 2026-08-30T10:11:03+00:00 /workspace/tiera-rerun-fixed-moonep/echo/tierbmem
|
| 23 |
+
OK ultraep tierbmem 16 2026-08-30T10:11:14+00:00 2026-08-30T10:13:56+00:00 /workspace/tiera-rerun-fixed-moonep/ultraep/tierbmem
|
| 24 |
+
OK moonep tierbmem 16 2026-08-30T10:14:05+00:00 2026-08-30T10:16:46+00:00 /workspace/tiera-rerun-fixed-moonep/moonep/tierbmem
|
tiera-rerun-fixed-moonep/study.log
CHANGED
|
@@ -2781,3 +2781,355 @@ driver_repo: /workspace/blackwell_run_scripts @ 48db41e
|
|
| 2781 |
=== STAGE 8 -- Tier B allocator dumps (static CG capacities + dynamic no-CG)
|
| 2782 |
|
| 2783 |
=== STEP echo__tierbmem__gbs16__G3
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 2781 |
=== STAGE 8 -- Tier B allocator dumps (static CG capacities + dynamic no-CG)
|
| 2782 |
|
| 2783 |
=== STEP echo__tierbmem__gbs16__G3
|
| 2784 |
+
==============================================================
|
| 2785 |
+
CAMPAIGN: echo backend=echo phase=tierbmem
|
| 2786 |
+
Megatron: /workspace/Megatron-MoE-Echo @ better-memory-dump-support
|
| 2787 |
+
GBS=16 seq=4096 repeats=3 nsys=0
|
| 2788 |
+
results -> /workspace/tiera-rerun-fixed-moonep/echo/tierbmem
|
| 2789 |
+
==============================================================
|
| 2790 |
+
--------------------------------------------------------------------------
|
| 2791 |
+
[10:08:21] CELL G3__echo__matched__seq4096__gbs16__cap1p072265625__memiteration__it6 (slots=8)
|
| 2792 |
+
[guard] iter 3/6 (70s)
|
| 2793 |
+
[guard] iter 6/6 (80s)
|
| 2794 |
+
--------------------------------------------------------------------------
|
| 2795 |
+
[10:09:42] CELL G3__echo__matched__seq4096__gbs16__capdynamic__nocg__memiteration__it5 (slots=8)
|
| 2796 |
+
[guard] iter 5/5 (80s)
|
| 2797 |
+
[guard] tearing down process group (outcome=DONE)
|
| 2798 |
+
--------------------------------------------------------------------------
|
| 2799 |
+
|
| 2800 |
+
==============================================================
|
| 2801 |
+
SWEEP COMPLETE ok=2 failed=0
|
| 2802 |
+
==============================================================
|
| 2803 |
+
config backend policy slots seq gbs cap recompute rep status iter_ms_mean iter_ms_min peak_mib run_dir
|
| 2804 |
+
G3 echo matched 8 4096 16 1.072265625 none 1 OK 452.40 452.40 148009 /workspace/tiera-rerun-fixed-moonep/echo/tierbmem/tierbmem_gbs16/G3__echo__matched__seq4096__gbs16__cap1p072265625__memiteration__it6/rep1
|
| 2805 |
+
G3 echo matched 8 4096 16 dynamic none 1 OK NA NA 118901 /workspace/tiera-rerun-fixed-moonep/echo/tierbmem/tierbmem_gbs16/G3__echo__matched__seq4096__gbs16__capdynamic__nocg__memiteration__it5/rep1
|
| 2806 |
+
|
| 2807 |
+
Results: /workspace/tiera-rerun-fixed-moonep/echo/tierbmem/tierbmem_gbs16/results.tsv
|
| 2808 |
+
Roll up: python3 /workspace/blackwell_run_scripts/granularity/tools/collect.py /workspace/tiera-rerun-fixed-moonep/echo/tierbmem/tierbmem_gbs16
|
| 2809 |
+
==============================================================
|
| 2810 |
+
6 memory-dump runs -> /workspace/tiera-rerun-fixed-moonep/echo/tierbmem/tierbmem_gbs16/memory_summary.tsv
|
| 2811 |
+
model backend cg allocation capacity status peak GiB dumps
|
| 2812 |
+
----------------------------------------------------------------------------------
|
| 2813 |
+
G1 echo 1 static 2.40625 OK 157.9 48/48
|
| 2814 |
+
G1 echo 0 dynamic dynamic OK 112.6 40/40
|
| 2815 |
+
G2 echo 1 static1.08203125 OK 142.5 48/48
|
| 2816 |
+
G2 echo 0 dynamic dynamic OK 115.1 40/40
|
| 2817 |
+
G3 echo 1 static1.072265625 OK 144.5 48/48
|
| 2818 |
+
G3 echo 0 dynamic dynamic OK 116.1 40/40
|
| 2819 |
+
|
| 2820 |
+
ECHO/UltraEP note: graph-on uses static capacity + device-initiated GEMM while graph-off uses dynamic allocation + host-derived sizes. Their difference is not labelled as pure graph cost.
|
| 2821 |
+
|
| 2822 |
+
TIER B MEMORY DUMPS -- how to read this
|
| 2823 |
+
---------------------------------------
|
| 2824 |
+
ECHO / UltraEP:
|
| 2825 |
+
cap1p00__memiteration__it6 graph on, static C=1.00
|
| 2826 |
+
cap1p125__memiteration__it6 graph on, static C=1.125
|
| 2827 |
+
cap1p25__memiteration__it6 graph on, static C=1.25
|
| 2828 |
+
capdynamic__nocg__memiteration__it5
|
| 2829 |
+
graph off, no capacity flag, dynamic allocation
|
| 2830 |
+
|
| 2831 |
+
MoonEP:
|
| 2832 |
+
capnone__memiteration__it6 graph on, no capacity factor
|
| 2833 |
+
capnone__nocg__memiteration__it5 graph off, no capacity factor
|
| 2834 |
+
|
| 2835 |
+
Every successful cell contains:
|
| 2836 |
+
train.log Megatron and OOM diagnostics
|
| 2837 |
+
mem_samples.csv driver-level nvidia-smi samples every 2 seconds
|
| 2838 |
+
run.env exact branch/config/capacity provenance
|
| 2839 |
+
memory_snapshots/*.pickle 48 captured / 40 eager allocator histories
|
| 2840 |
+
memory_snapshots/*.json matching summaries and before/after deltas
|
| 2841 |
+
|
| 2842 |
+
The graph-on/static vs graph-off/dynamic difference is intentionally NOT called
|
| 2843 |
+
"graph cost": capacity allocation policy and device-initiated grouped GEMM also
|
| 2844 |
+
change. Compare the three graph-on rows to measure static-capacity sensitivity,
|
| 2845 |
+
and use the dynamic row as the natural eager-allocation reference.
|
| 2846 |
+
|
| 2847 |
+
Campaign results: /workspace/tiera-rerun-fixed-moonep/echo/tierbmem
|
| 2848 |
+
Roll up: python3 /workspace/blackwell_run_scripts/granularity/tools/collect.py /workspace/tiera-rerun-fixed-moonep/echo/tierbmem --compare
|
| 2849 |
+
Already logged in to Hugging Face as: AbhishekGhosh1998
|
| 2850 |
+
uploaded: /workspace/tiera-rerun-fixed-moonep/echo/tierbmem -> MoE-CUDA-Graph-Analysis/tierb-b200-gbs8/tiera-rerun-fixed-moonep/echo/tierbmem
|
| 2851 |
+
https://huggingface.co/MoE-CUDA-Graph-Analysis/tierb-b200-gbs8/tree/main
|
| 2852 |
+
|
| 2853 |
+
=== STEP ultraep__tierbmem__gbs16__G3
|
| 2854 |
+
==============================================================
|
| 2855 |
+
CAMPAIGN: ultraep backend=ultraep phase=tierbmem
|
| 2856 |
+
Megatron: /workspace/Megatron-UltraEP @ ultraep-better-memory-dump-support
|
| 2857 |
+
GBS=16 seq=4096 repeats=3 nsys=0
|
| 2858 |
+
results -> /workspace/tiera-rerun-fixed-moonep/ultraep/tierbmem
|
| 2859 |
+
==============================================================
|
| 2860 |
+
--------------------------------------------------------------------------
|
| 2861 |
+
[10:11:29] CELL G3__ultraep__matched__seq4096__gbs16__cap1p072265625__memiteration__it6 (slots=8)
|
| 2862 |
+
[guard] iter 3/6 (60s)
|
| 2863 |
+
[guard] iter 6/6 (70s)
|
| 2864 |
+
[guard] tearing down process group (outcome=DONE)
|
| 2865 |
+
--------------------------------------------------------------------------
|
| 2866 |
+
[10:12:42] CELL G3__ultraep__matched__seq4096__gbs16__capdynamic__nocg__memiteration__it5 (slots=8)
|
| 2867 |
+
[guard] iter 2/5 (60s)
|
| 2868 |
+
[guard] iter 5/5 (70s)
|
| 2869 |
+
[guard] tearing down process group (outcome=DONE)
|
| 2870 |
+
--------------------------------------------------------------------------
|
| 2871 |
+
|
| 2872 |
+
==============================================================
|
| 2873 |
+
SWEEP COMPLETE ok=2 failed=0
|
| 2874 |
+
==============================================================
|
| 2875 |
+
config backend policy slots seq gbs cap recompute rep status iter_ms_mean iter_ms_min peak_mib run_dir
|
| 2876 |
+
G3 ultraep matched 8 4096 16 1.072265625 none 1 OK 477.80 477.80 113951 /workspace/tiera-rerun-fixed-moonep/ultraep/tierbmem/tierbmem_gbs16/G3__ultraep__matched__seq4096__gbs16__cap1p072265625__memiteration__it6/rep1
|
| 2877 |
+
G3 ultraep matched 8 4096 16 dynamic none 1 OK NA NA 111707 /workspace/tiera-rerun-fixed-moonep/ultraep/tierbmem/tierbmem_gbs16/G3__ultraep__matched__seq4096__gbs16__capdynamic__nocg__memiteration__it5/rep1
|
| 2878 |
+
|
| 2879 |
+
Results: /workspace/tiera-rerun-fixed-moonep/ultraep/tierbmem/tierbmem_gbs16/results.tsv
|
| 2880 |
+
Roll up: python3 /workspace/blackwell_run_scripts/granularity/tools/collect.py /workspace/tiera-rerun-fixed-moonep/ultraep/tierbmem/tierbmem_gbs16
|
| 2881 |
+
==============================================================
|
| 2882 |
+
6 memory-dump runs -> /workspace/tiera-rerun-fixed-moonep/ultraep/tierbmem/tierbmem_gbs16/memory_summary.tsv
|
| 2883 |
+
model backend cg allocation capacity status peak GiB dumps
|
| 2884 |
+
----------------------------------------------------------------------------------
|
| 2885 |
+
G1 ultraep 1 static 2.40625 OK 126.4 48/48
|
| 2886 |
+
G1 ultraep 0 dynamic dynamic OK 106.7 40/40
|
| 2887 |
+
G2 ultraep 1 static1.08203125 OK 109.7 48/48
|
| 2888 |
+
G2 ultraep 0 dynamic dynamic OK 105.7 40/40
|
| 2889 |
+
G3 ultraep 1 static1.072265625 OK 111.3 48/48
|
| 2890 |
+
G3 ultraep 0 dynamic dynamic OK 109.1 40/40
|
| 2891 |
+
|
| 2892 |
+
ECHO/UltraEP note: graph-on uses static capacity + device-initiated GEMM while graph-off uses dynamic allocation + host-derived sizes. Their difference is not labelled as pure graph cost.
|
| 2893 |
+
|
| 2894 |
+
TIER B MEMORY DUMPS -- how to read this
|
| 2895 |
+
---------------------------------------
|
| 2896 |
+
ECHO / UltraEP:
|
| 2897 |
+
cap1p00__memiteration__it6 graph on, static C=1.00
|
| 2898 |
+
cap1p125__memiteration__it6 graph on, static C=1.125
|
| 2899 |
+
cap1p25__memiteration__it6 graph on, static C=1.25
|
| 2900 |
+
capdynamic__nocg__memiteration__it5
|
| 2901 |
+
graph off, no capacity flag, dynamic allocation
|
| 2902 |
+
|
| 2903 |
+
MoonEP:
|
| 2904 |
+
capnone__memiteration__it6 graph on, no capacity factor
|
| 2905 |
+
capnone__nocg__memiteration__it5 graph off, no capacity factor
|
| 2906 |
+
|
| 2907 |
+
Every successful cell contains:
|
| 2908 |
+
train.log Megatron and OOM diagnostics
|
| 2909 |
+
mem_samples.csv driver-level nvidia-smi samples every 2 seconds
|
| 2910 |
+
run.env exact branch/config/capacity provenance
|
| 2911 |
+
memory_snapshots/*.pickle 48 captured / 40 eager allocator histories
|
| 2912 |
+
memory_snapshots/*.json matching summaries and before/after deltas
|
| 2913 |
+
|
| 2914 |
+
The graph-on/static vs graph-off/dynamic difference is intentionally NOT called
|
| 2915 |
+
"graph cost": capacity allocation policy and device-initiated grouped GEMM also
|
| 2916 |
+
change. Compare the three graph-on rows to measure static-capacity sensitivity,
|
| 2917 |
+
and use the dynamic row as the natural eager-allocation reference.
|
| 2918 |
+
|
| 2919 |
+
Campaign results: /workspace/tiera-rerun-fixed-moonep/ultraep/tierbmem
|
| 2920 |
+
Roll up: python3 /workspace/blackwell_run_scripts/granularity/tools/collect.py /workspace/tiera-rerun-fixed-moonep/ultraep/tierbmem --compare
|
| 2921 |
+
Already logged in to Hugging Face as: AbhishekGhosh1998
|
| 2922 |
+
uploaded: /workspace/tiera-rerun-fixed-moonep/ultraep/tierbmem -> MoE-CUDA-Graph-Analysis/tierb-b200-gbs8/tiera-rerun-fixed-moonep/ultraep/tierbmem
|
| 2923 |
+
https://huggingface.co/MoE-CUDA-Graph-Analysis/tierb-b200-gbs8/tree/main
|
| 2924 |
+
|
| 2925 |
+
=== STEP moonep__tierbmem__gbs16__G3
|
| 2926 |
+
==============================================================
|
| 2927 |
+
CAMPAIGN: moonep backend=moonep phase=tierbmem
|
| 2928 |
+
Megatron: /workspace/Megatron-MoE-Echo @ moonep_backend_async_grad_reduce_better_memory_dump_support
|
| 2929 |
+
GBS=16 seq=4096 repeats=3 nsys=0
|
| 2930 |
+
results -> /workspace/tiera-rerun-fixed-moonep/moonep/tierbmem
|
| 2931 |
+
==============================================================
|
| 2932 |
+
--------------------------------------------------------------------------
|
| 2933 |
+
[10:14:20] CELL G3__moonep__matched__seq4096__gbs16__capnone__async__memiteration__it6 (slots=8)
|
| 2934 |
+
[guard] iter 3/6 (60s)
|
| 2935 |
+
[guard] iter 6/6 (70s)
|
| 2936 |
+
[guard] tearing down process group (outcome=DONE)
|
| 2937 |
+
--------------------------------------------------------------------------
|
| 2938 |
+
[10:15:34] CELL G3__moonep__matched__seq4096__gbs16__capnone__nocg__async__memiteration__it5 (slots=8)
|
| 2939 |
+
[guard] iter 3/5 (60s)
|
| 2940 |
+
[guard] iter 5/5 (70s)
|
| 2941 |
+
[guard] tearing down process group (outcome=DONE)
|
| 2942 |
+
--------------------------------------------------------------------------
|
| 2943 |
+
|
| 2944 |
+
==============================================================
|
| 2945 |
+
SWEEP COMPLETE ok=2 failed=0
|
| 2946 |
+
==============================================================
|
| 2947 |
+
config backend policy slots seq gbs cap recompute rep status iter_ms_mean iter_ms_min peak_mib run_dir
|
| 2948 |
+
G3 moonep matched 8 4096 16 none none 1 OK 578.30 578.30 99411 /workspace/tiera-rerun-fixed-moonep/moonep/tierbmem/tierbmem_gbs16/G3__moonep__matched__seq4096__gbs16__capnone__async__memiteration__it6/rep1
|
| 2949 |
+
G3 moonep matched 8 4096 16 none none 1 OK NA NA 98611 /workspace/tiera-rerun-fixed-moonep/moonep/tierbmem/tierbmem_gbs16/G3__moonep__matched__seq4096__gbs16__capnone__nocg__async__memiteration__it5/rep1
|
| 2950 |
+
|
| 2951 |
+
Results: /workspace/tiera-rerun-fixed-moonep/moonep/tierbmem/tierbmem_gbs16/results.tsv
|
| 2952 |
+
Roll up: python3 /workspace/blackwell_run_scripts/granularity/tools/collect.py /workspace/tiera-rerun-fixed-moonep/moonep/tierbmem/tierbmem_gbs16
|
| 2953 |
+
==============================================================
|
| 2954 |
+
6 memory-dump runs -> /workspace/tiera-rerun-fixed-moonep/moonep/tierbmem/tierbmem_gbs16/memory_summary.tsv
|
| 2955 |
+
model backend cg allocation capacity status peak GiB dumps
|
| 2956 |
+
----------------------------------------------------------------------------------
|
| 2957 |
+
G1 moonep 1 none none OK 94.6 48/48
|
| 2958 |
+
G1 moonep 0 none none OK 90.4 40/40
|
| 2959 |
+
G2 moonep 1 none none OK 95.4 48/48
|
| 2960 |
+
G2 moonep 0 none none OK 93.2 40/40
|
| 2961 |
+
G3 moonep 1 none none OK 97.1 48/48
|
| 2962 |
+
G3 moonep 0 none none OK 96.3 40/40
|
| 2963 |
+
|
| 2964 |
+
ECHO/UltraEP note: graph-on uses static capacity + device-initiated GEMM while graph-off uses dynamic allocation + host-derived sizes. Their difference is not labelled as pure graph cost.
|
| 2965 |
+
|
| 2966 |
+
TIER B MEMORY DUMPS -- how to read this
|
| 2967 |
+
---------------------------------------
|
| 2968 |
+
ECHO / UltraEP:
|
| 2969 |
+
cap1p00__memiteration__it6 graph on, static C=1.00
|
| 2970 |
+
cap1p125__memiteration__it6 graph on, static C=1.125
|
| 2971 |
+
cap1p25__memiteration__it6 graph on, static C=1.25
|
| 2972 |
+
capdynamic__nocg__memiteration__it5
|
| 2973 |
+
graph off, no capacity flag, dynamic allocation
|
| 2974 |
+
|
| 2975 |
+
MoonEP:
|
| 2976 |
+
capnone__memiteration__it6 graph on, no capacity factor
|
| 2977 |
+
capnone__nocg__memiteration__it5 graph off, no capacity factor
|
| 2978 |
+
|
| 2979 |
+
Every successful cell contains:
|
| 2980 |
+
train.log Megatron and OOM diagnostics
|
| 2981 |
+
mem_samples.csv driver-level nvidia-smi samples every 2 seconds
|
| 2982 |
+
run.env exact branch/config/capacity provenance
|
| 2983 |
+
memory_snapshots/*.pickle 48 captured / 40 eager allocator histories
|
| 2984 |
+
memory_snapshots/*.json matching summaries and before/after deltas
|
| 2985 |
+
|
| 2986 |
+
The graph-on/static vs graph-off/dynamic difference is intentionally NOT called
|
| 2987 |
+
"graph cost": capacity allocation policy and device-initiated grouped GEMM also
|
| 2988 |
+
change. Compare the three graph-on rows to measure static-capacity sensitivity,
|
| 2989 |
+
and use the dynamic row as the natural eager-allocation reference.
|
| 2990 |
+
|
| 2991 |
+
Campaign results: /workspace/tiera-rerun-fixed-moonep/moonep/tierbmem
|
| 2992 |
+
Roll up: python3 /workspace/blackwell_run_scripts/granularity/tools/collect.py /workspace/tiera-rerun-fixed-moonep/moonep/tierbmem --compare
|
| 2993 |
+
Already logged in to Hugging Face as: AbhishekGhosh1998
|
| 2994 |
+
uploaded: /workspace/tiera-rerun-fixed-moonep/moonep/tierbmem -> MoE-CUDA-Graph-Analysis/tierb-b200-gbs8/tiera-rerun-fixed-moonep/moonep/tierbmem
|
| 2995 |
+
https://huggingface.co/MoE-CUDA-Graph-Analysis/tierb-b200-gbs8/tree/main
|
| 2996 |
+
|
| 2997 |
+
=== ROLL-UP
|
| 2998 |
+
134 runs ok=125 oom=0 -> /workspace/tiera-rerun-fixed-moonep/collected_all.tsv
|
| 2999 |
+
|
| 3000 |
+
WARNING: 11 run(s) mention capacity saturation. A run that drops assignments does less expert work and must not be quoted as a win:
|
| 3001 |
+
ultraep/tierbprobe/tierbprobe_gbs16/G1__ultraep__matched__seq4096__gbs16__cap1p25/rep1: WARNING:megatron.training.drop_debug_utils:[MOE_DROP_DEBUG][iteration_summary] global_rank=0 iteration=0 records=32 total_expected_recv=283999 total_dropped_tok
|
| 3002 |
+
ultraep/tierbprobe/tierbprobe_gbs16/G1__ultraep__matched__seq4096__gbs16__cap1p8125/rep1: WARNING:megatron.training.drop_debug_utils:[MOE_DROP_DEBUG][iteration_summary] global_rank=0 iteration=0 records=32 total_expected_recv=284540 total_dropped_tok
|
| 3003 |
+
ultraep/tierbprobe/tierbprobe_gbs16/G2__ultraep__matched__seq4096__gbs16__cap1p0625/rep1: WARNING:megatron.training.drop_debug_utils:[MOE_DROP_DEBUG][iteration_summary] global_rank=0 iteration=0 records=32 total_expected_recv=1048800 total_dropped_to
|
| 3004 |
+
ultraep/tierbprobe/tierbprobe_gbs16/G2__ultraep__matched__seq4096__gbs16__cap1p25/rep1: WARNING:megatron.training.drop_debug_utils:[MOE_DROP_DEBUG][iteration_summary] global_rank=0 iteration=0 records=32 total_expected_recv=1048800 total_dropped_to
|
| 3005 |
+
ultraep/tierbprobe/tierbprobe_gbs16/G2__ultraep__matched__seq4096__gbs16__cap1p25__forcebal/rep1: WARNING:megatron.training.drop_debug_utils:[MOE_DROP_DEBUG][iteration_summary] global_rank=1 iteration=0 records=32 total_expected_recv=1047984 total_dropped_to
|
| 3006 |
+
|
| 3007 |
+
(excluded 14 nsys-traced run(s) from the timing comparison)
|
| 3008 |
+
|
| 3009 |
+
phase config seq gbs policy rc cg asy backend iter_ms vs_best
|
| 3010 |
+
-------------------------------------------------------------------------------------
|
| 3011 |
+
a1 G1 4096 16 one none 1 0 echo 466.68 1.00x
|
| 3012 |
+
ultraep 891.00 1.91x
|
| 3013 |
+
|
| 3014 |
+
a1 G1 4096 16 one none 1 1 moonep 480.30 1.00x
|
| 3015 |
+
|
| 3016 |
+
a1 G2 4096 16 one none 1 0 echo 394.28 1.00x
|
| 3017 |
+
ultraep 466.65 1.18x
|
| 3018 |
+
|
| 3019 |
+
a1 G3 4096 16 one none 1 0 echo 412.83 1.00x
|
| 3020 |
+
ultraep 447.84 1.08x
|
| 3021 |
+
|
| 3022 |
+
a1 G4 4096 16 one none 1 0 echo 461.03 1.00x
|
| 3023 |
+
ultraep 467.57 1.01x
|
| 3024 |
+
|
| 3025 |
+
a2 G1 4096 16 matched none 1 0 echo 466.91 1.00x
|
| 3026 |
+
ultraep 891.92 1.91x
|
| 3027 |
+
|
| 3028 |
+
a2 G1 4096 16 matched none 1 1 moonep 481.37 1.00x
|
| 3029 |
+
|
| 3030 |
+
a2 G2 4096 16 matched none 1 0 echo 415.21 1.00x
|
| 3031 |
+
ultraep 492.54 1.19x
|
| 3032 |
+
|
| 3033 |
+
a2 G2 4096 16 matched none 1 1 moonep 432.02 1.00x
|
| 3034 |
+
|
| 3035 |
+
a2 G3 4096 16 matched none 1 0 echo 437.35 1.00x
|
| 3036 |
+
ultraep 459.60 1.05x
|
| 3037 |
+
|
| 3038 |
+
a2 G3 4096 16 matched none 1 1 moonep 559.88 1.00x
|
| 3039 |
+
|
| 3040 |
+
a2 G4 4096 16 matched none 1 0 ultraep 482.64 1.00x
|
| 3041 |
+
echo 486.54 1.01x
|
| 3042 |
+
|
| 3043 |
+
a2 G4 4096 16 matched none 1 1 moonep 783.34 1.00x
|
| 3044 |
+
|
| 3045 |
+
ponly PONLY 4096 16 one none 1 0 echo 174.81 1.00x
|
| 3046 |
+
ultraep 176.76 1.01x
|
| 3047 |
+
|
| 3048 |
+
ponly PONLY 4096 16 one none 1 1 moonep 165.22 1.00x
|
| 3049 |
+
|
| 3050 |
+
ponly PONLY 8192 16 one none 1 0 ultraep 292.62 1.00x
|
| 3051 |
+
echo 316.70 1.08x
|
| 3052 |
+
|
| 3053 |
+
ponly PONLY 8192 16 one none 1 1 moonep 300.61 1.00x
|
| 3054 |
+
|
| 3055 |
+
tierbmem G1 4096 16 matched none 1 0 echo 474.40 1.00x
|
| 3056 |
+
ultraep 909.90 1.92x
|
| 3057 |
+
|
| 3058 |
+
tierbmem G1 4096 16 matched none 1 1 moonep 497.40 1.00x
|
| 3059 |
+
|
| 3060 |
+
tierbmem G2 4096 16 matched none 1 0 echo 430.10 1.00x
|
| 3061 |
+
ultraep 475.40 1.11x
|
| 3062 |
+
|
| 3063 |
+
tierbmem G2 4096 16 matched none 1 1 moonep 439.70 1.00x
|
| 3064 |
+
|
| 3065 |
+
tierbmem G3 4096 16 matched none 1 0 echo 452.40 1.00x
|
| 3066 |
+
ultraep 477.80 1.06x
|
| 3067 |
+
|
| 3068 |
+
tierbmem G3 4096 16 matched none 1 1 moonep 578.30 1.00x
|
| 3069 |
+
|
| 3070 |
+
tierbprobeG1 4096 16 matched none 0 0 echo 671.53 1.00x
|
| 3071 |
+
ultraep 904.16 1.35x
|
| 3072 |
+
|
| 3073 |
+
tierbprobeG2 4096 16 matched none 0 0 ultraep 512.37 1.00x
|
| 3074 |
+
echo 654.76 1.28x
|
| 3075 |
+
|
| 3076 |
+
tierbprobeG3 4096 16 matched none 0 0 ultraep 512.31 1.00x
|
| 3077 |
+
echo 689.13 1.35x
|
| 3078 |
+
|
| 3079 |
+
tierbprobeG4 4096 16 matched none 0 0 ultraep 600.19 1.00x
|
| 3080 |
+
echo 790.97 1.32x
|
| 3081 |
+
|
| 3082 |
+
21 probe runs -> /workspace/tiera-rerun-fixed-moonep/probe_summary_all.tsv
|
| 3083 |
+
run arm cap rawExpI preRankI postRankI drop% reqCap
|
| 3084 |
+
--------------------------------------------------------------------------------------------------------------------
|
| 3085 |
+
echo/tierbprobe/tierbprobe_gbs16/G1__echo__matched__ natural 2.00 3.6198* 3.6198 1.0827 1.9062
|
| 3086 |
+
echo/tierbprobe/tierbprobe_gbs16/G2__echo__matched__ natural 1.08203125 3.8720* 2.1703 1.0000 1.0273
|
| 3087 |
+
echo/tierbprobe/tierbprobe_gbs16/G2__echo__matched__ forced 1.08203125 1.0201 1.0069 1.0000 1.0234
|
| 3088 |
+
echo/tierbprobe/tierbprobe_gbs16/G2__echo__matched__ natural 2.00 3.8724* 2.1875 1.0000 1.0273
|
| 3089 |
+
echo/tierbprobe/tierbprobe_gbs16/G3__echo__matched__ natural 1.072265625 3.9096* 1.7838 1.0000 1.0195
|
| 3090 |
+
echo/tierbprobe/tierbprobe_gbs16/G3__echo__matched__ forced 1.072265625 1.0231 1.0048 1.0000 1.0176
|
| 3091 |
+
echo/tierbprobe/tierbprobe_gbs16/G3__echo__matched__ natural 2.00 3.9096* 1.7916 1.0000 1.0195
|
| 3092 |
+
echo/tierbprobe/tierbprobe_gbs16/G4__echo__matched__ natural 1.068359375 3.9303* 1.5507 1.0000 1.0166
|
| 3093 |
+
echo/tierbprobe/tierbprobe_gbs16/G4__echo__matched__ forced 1.068359375 1.0249 1.0034 1.0000 1.0137
|
| 3094 |
+
echo/tierbprobe/tierbprobe_gbs16/G4__echo__matched__ natural 2.00 3.9301* 1.5543 1.0000 1.0166
|
| 3095 |
+
ultraep/tierbprobe/tierbprobe_gbs16/G1__ultraep__mat natural 1.25 3.6085 1.0640 2.8918 1.7246
|
| 3096 |
+
ultraep/tierbprobe/tierbprobe_gbs16/G1__ultraep__mat natural 1.8125 3.6152 1.0747 1.5460 2.3091
|
| 3097 |
+
ultraep/tierbprobe/tierbprobe_gbs16/G2__ultraep__mat natural 1.0625 2.2032 1.0100 0.0000 1.0101
|
| 3098 |
+
ultraep/tierbprobe/tierbprobe_gbs16/G2__ultraep__mat natural 1.25 2.2136 1.0100 0.0000 1.0101
|
| 3099 |
+
ultraep/tierbprobe/tierbprobe_gbs16/G2__ultraep__mat forced 1.25 1.0069 1.0067 0.0000 1.0100
|
| 3100 |
+
ultraep/tierbprobe/tierbprobe_gbs16/G3__ultraep__mat natural 1.0625 1.7842 1.0100 0.0000 1.0101
|
| 3101 |
+
ultraep/tierbprobe/tierbprobe_gbs16/G3__ultraep__mat natural 1.25 1.7752 1.0100 0.0000 1.0101
|
| 3102 |
+
ultraep/tierbprobe/tierbprobe_gbs16/G3__ultraep__mat forced 1.25 1.0047 1.0047 0.0000 1.0100
|
| 3103 |
+
ultraep/tierbprobe/tierbprobe_gbs16/G4__ultraep__mat natural 1.095703125 1.5612 1.0100 0.0000 1.0429
|
| 3104 |
+
ultraep/tierbprobe/tierbprobe_gbs16/G4__ultraep__mat natural 1.25 1.5504 1.0100 0.0000 1.0429
|
| 3105 |
+
ultraep/tierbprobe/tierbprobe_gbs16/G4__ultraep__mat forced 1.25 1.0033 1.0033 0.0000 1.0071
|
| 3106 |
+
|
| 3107 |
+
* At least one globally aggregated snapshot reached the E/K expert-Imax ceiling.
|
| 3108 |
+
Inspect raw_expert_imax_mean and hottest_expert_token_fraction_mean before generalising.
|
| 3109 |
+
|
| 3110 |
+
ECHO note: echo_logged_pre_rank_metric_valid=0 and echo_logged_load_reduction_valid=0 are intentional.
|
| 3111 |
+
The branch's built-in fields mix source-rank and destination-rank axes; this tool reconstructs both globally.
|
| 3112 |
+
18 memory-dump runs -> /workspace/tiera-rerun-fixed-moonep/memory_summary_all.tsv
|
| 3113 |
+
model backend cg allocation capacity status peak GiB dumps
|
| 3114 |
+
----------------------------------------------------------------------------------
|
| 3115 |
+
G1 echo 1 static 2.40625 OK 157.9 48/48
|
| 3116 |
+
G1 echo 0 dynamic dynamic OK 112.6 40/40
|
| 3117 |
+
G1 moonep 1 none none OK 94.6 48/48
|
| 3118 |
+
G1 moonep 0 none none OK 90.4 40/40
|
| 3119 |
+
G1 ultraep 1 static 2.40625 OK 126.4 48/48
|
| 3120 |
+
G1 ultraep 0 dynamic dynamic OK 106.7 40/40
|
| 3121 |
+
G2 echo 1 static1.08203125 OK 142.5 48/48
|
| 3122 |
+
G2 echo 0 dynamic dynamic OK 115.1 40/40
|
| 3123 |
+
G2 moonep 1 none none OK 95.4 48/48
|
| 3124 |
+
G2 moonep 0 none none OK 93.2 40/40
|
| 3125 |
+
G2 ultraep 1 static1.08203125 OK 109.7 48/48
|
| 3126 |
+
G2 ultraep 0 dynamic dynamic OK 105.7 40/40
|
| 3127 |
+
G3 echo 1 static1.072265625 OK 144.5 48/48
|
| 3128 |
+
G3 echo 0 dynamic dynamic OK 116.1 40/40
|
| 3129 |
+
G3 moonep 1 none none OK 97.1 48/48
|
| 3130 |
+
G3 moonep 0 none none OK 96.3 40/40
|
| 3131 |
+
G3 ultraep 1 static1.072265625 OK 111.3 48/48
|
| 3132 |
+
G3 ultraep 0 dynamic dynamic OK 109.1 40/40
|
| 3133 |
+
|
| 3134 |
+
ECHO/UltraEP note: graph-on uses static capacity + device-initiated GEMM while graph-off uses dynamic allocation + host-derived sizes. Their difference is not labelled as pure graph cost.
|
| 3135 |
+
Already logged in to Hugging Face as: AbhishekGhosh1998
|