AbhishekGhosh1998 commited on
Commit
67812db
·
verified ·
1 Parent(s): 1894dc2

granularity results tiera-rerun-fixed-moonep @ 2026-08-30T10:17:03Z from sp-aiinfra-interactive-aghosh-7jtjl-8xb200-moe-cg-0 (part 3)

Browse files
tiera-rerun-fixed-moonep/steps.tsv CHANGED
@@ -19,3 +19,6 @@ OK moonep tierbtrace 16 2026-08-30T09:28:30+00:00 2026-08-30T09:45:26+00:00 /wor
19
  OK echo tierbmem 16 2026-08-30T09:59:28+00:00 2026-08-30T10:02:20+00:00 /workspace/tiera-rerun-fixed-moonep/echo/tierbmem
20
  OK ultraep tierbmem 16 2026-08-30T10:02:26+00:00 2026-08-30T10:05:04+00:00 /workspace/tiera-rerun-fixed-moonep/ultraep/tierbmem
21
  OK moonep tierbmem 16 2026-08-30T10:05:11+00:00 2026-08-30T10:07:42+00:00 /workspace/tiera-rerun-fixed-moonep/moonep/tierbmem
 
 
 
 
19
  OK echo tierbmem 16 2026-08-30T09:59:28+00:00 2026-08-30T10:02:20+00:00 /workspace/tiera-rerun-fixed-moonep/echo/tierbmem
20
  OK ultraep tierbmem 16 2026-08-30T10:02:26+00:00 2026-08-30T10:05:04+00:00 /workspace/tiera-rerun-fixed-moonep/ultraep/tierbmem
21
  OK moonep tierbmem 16 2026-08-30T10:05:11+00:00 2026-08-30T10:07:42+00:00 /workspace/tiera-rerun-fixed-moonep/moonep/tierbmem
22
+ OK echo tierbmem 16 2026-08-30T10:08:07+00:00 2026-08-30T10:11:03+00:00 /workspace/tiera-rerun-fixed-moonep/echo/tierbmem
23
+ OK ultraep tierbmem 16 2026-08-30T10:11:14+00:00 2026-08-30T10:13:56+00:00 /workspace/tiera-rerun-fixed-moonep/ultraep/tierbmem
24
+ OK moonep tierbmem 16 2026-08-30T10:14:05+00:00 2026-08-30T10:16:46+00:00 /workspace/tiera-rerun-fixed-moonep/moonep/tierbmem
tiera-rerun-fixed-moonep/study.log CHANGED
@@ -2781,3 +2781,355 @@ driver_repo: /workspace/blackwell_run_scripts @ 48db41e
2781
  === STAGE 8 -- Tier B allocator dumps (static CG capacities + dynamic no-CG)
2782
 
2783
  === STEP echo__tierbmem__gbs16__G3
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
2781
  === STAGE 8 -- Tier B allocator dumps (static CG capacities + dynamic no-CG)
2782
 
2783
  === STEP echo__tierbmem__gbs16__G3
2784
+ ==============================================================
2785
+ CAMPAIGN: echo backend=echo phase=tierbmem
2786
+ Megatron: /workspace/Megatron-MoE-Echo @ better-memory-dump-support
2787
+ GBS=16 seq=4096 repeats=3 nsys=0
2788
+ results -> /workspace/tiera-rerun-fixed-moonep/echo/tierbmem
2789
+ ==============================================================
2790
+ --------------------------------------------------------------------------
2791
+ [10:08:21] CELL G3__echo__matched__seq4096__gbs16__cap1p072265625__memiteration__it6 (slots=8)
2792
+ [guard] iter 3/6 (70s)
2793
+ [guard] iter 6/6 (80s)
2794
+ --------------------------------------------------------------------------
2795
+ [10:09:42] CELL G3__echo__matched__seq4096__gbs16__capdynamic__nocg__memiteration__it5 (slots=8)
2796
+ [guard] iter 5/5 (80s)
2797
+ [guard] tearing down process group (outcome=DONE)
2798
+ --------------------------------------------------------------------------
2799
+
2800
+ ==============================================================
2801
+ SWEEP COMPLETE ok=2 failed=0
2802
+ ==============================================================
2803
+ config backend policy slots seq gbs cap recompute rep status iter_ms_mean iter_ms_min peak_mib run_dir
2804
+ G3 echo matched 8 4096 16 1.072265625 none 1 OK 452.40 452.40 148009 /workspace/tiera-rerun-fixed-moonep/echo/tierbmem/tierbmem_gbs16/G3__echo__matched__seq4096__gbs16__cap1p072265625__memiteration__it6/rep1
2805
+ G3 echo matched 8 4096 16 dynamic none 1 OK NA NA 118901 /workspace/tiera-rerun-fixed-moonep/echo/tierbmem/tierbmem_gbs16/G3__echo__matched__seq4096__gbs16__capdynamic__nocg__memiteration__it5/rep1
2806
+
2807
+ Results: /workspace/tiera-rerun-fixed-moonep/echo/tierbmem/tierbmem_gbs16/results.tsv
2808
+ Roll up: python3 /workspace/blackwell_run_scripts/granularity/tools/collect.py /workspace/tiera-rerun-fixed-moonep/echo/tierbmem/tierbmem_gbs16
2809
+ ==============================================================
2810
+ 6 memory-dump runs -> /workspace/tiera-rerun-fixed-moonep/echo/tierbmem/tierbmem_gbs16/memory_summary.tsv
2811
+ model backend cg allocation capacity status peak GiB dumps
2812
+ ----------------------------------------------------------------------------------
2813
+ G1 echo 1 static 2.40625 OK 157.9 48/48
2814
+ G1 echo 0 dynamic dynamic OK 112.6 40/40
2815
+ G2 echo 1 static1.08203125 OK 142.5 48/48
2816
+ G2 echo 0 dynamic dynamic OK 115.1 40/40
2817
+ G3 echo 1 static1.072265625 OK 144.5 48/48
2818
+ G3 echo 0 dynamic dynamic OK 116.1 40/40
2819
+
2820
+ ECHO/UltraEP note: graph-on uses static capacity + device-initiated GEMM while graph-off uses dynamic allocation + host-derived sizes. Their difference is not labelled as pure graph cost.
2821
+
2822
+ TIER B MEMORY DUMPS -- how to read this
2823
+ ---------------------------------------
2824
+ ECHO / UltraEP:
2825
+ cap1p00__memiteration__it6 graph on, static C=1.00
2826
+ cap1p125__memiteration__it6 graph on, static C=1.125
2827
+ cap1p25__memiteration__it6 graph on, static C=1.25
2828
+ capdynamic__nocg__memiteration__it5
2829
+ graph off, no capacity flag, dynamic allocation
2830
+
2831
+ MoonEP:
2832
+ capnone__memiteration__it6 graph on, no capacity factor
2833
+ capnone__nocg__memiteration__it5 graph off, no capacity factor
2834
+
2835
+ Every successful cell contains:
2836
+ train.log Megatron and OOM diagnostics
2837
+ mem_samples.csv driver-level nvidia-smi samples every 2 seconds
2838
+ run.env exact branch/config/capacity provenance
2839
+ memory_snapshots/*.pickle 48 captured / 40 eager allocator histories
2840
+ memory_snapshots/*.json matching summaries and before/after deltas
2841
+
2842
+ The graph-on/static vs graph-off/dynamic difference is intentionally NOT called
2843
+ "graph cost": capacity allocation policy and device-initiated grouped GEMM also
2844
+ change. Compare the three graph-on rows to measure static-capacity sensitivity,
2845
+ and use the dynamic row as the natural eager-allocation reference.
2846
+
2847
+ Campaign results: /workspace/tiera-rerun-fixed-moonep/echo/tierbmem
2848
+ Roll up: python3 /workspace/blackwell_run_scripts/granularity/tools/collect.py /workspace/tiera-rerun-fixed-moonep/echo/tierbmem --compare
2849
+ Already logged in to Hugging Face as: AbhishekGhosh1998
2850
+ uploaded: /workspace/tiera-rerun-fixed-moonep/echo/tierbmem -> MoE-CUDA-Graph-Analysis/tierb-b200-gbs8/tiera-rerun-fixed-moonep/echo/tierbmem
2851
+ https://huggingface.co/MoE-CUDA-Graph-Analysis/tierb-b200-gbs8/tree/main
2852
+
2853
+ === STEP ultraep__tierbmem__gbs16__G3
2854
+ ==============================================================
2855
+ CAMPAIGN: ultraep backend=ultraep phase=tierbmem
2856
+ Megatron: /workspace/Megatron-UltraEP @ ultraep-better-memory-dump-support
2857
+ GBS=16 seq=4096 repeats=3 nsys=0
2858
+ results -> /workspace/tiera-rerun-fixed-moonep/ultraep/tierbmem
2859
+ ==============================================================
2860
+ --------------------------------------------------------------------------
2861
+ [10:11:29] CELL G3__ultraep__matched__seq4096__gbs16__cap1p072265625__memiteration__it6 (slots=8)
2862
+ [guard] iter 3/6 (60s)
2863
+ [guard] iter 6/6 (70s)
2864
+ [guard] tearing down process group (outcome=DONE)
2865
+ --------------------------------------------------------------------------
2866
+ [10:12:42] CELL G3__ultraep__matched__seq4096__gbs16__capdynamic__nocg__memiteration__it5 (slots=8)
2867
+ [guard] iter 2/5 (60s)
2868
+ [guard] iter 5/5 (70s)
2869
+ [guard] tearing down process group (outcome=DONE)
2870
+ --------------------------------------------------------------------------
2871
+
2872
+ ==============================================================
2873
+ SWEEP COMPLETE ok=2 failed=0
2874
+ ==============================================================
2875
+ config backend policy slots seq gbs cap recompute rep status iter_ms_mean iter_ms_min peak_mib run_dir
2876
+ G3 ultraep matched 8 4096 16 1.072265625 none 1 OK 477.80 477.80 113951 /workspace/tiera-rerun-fixed-moonep/ultraep/tierbmem/tierbmem_gbs16/G3__ultraep__matched__seq4096__gbs16__cap1p072265625__memiteration__it6/rep1
2877
+ G3 ultraep matched 8 4096 16 dynamic none 1 OK NA NA 111707 /workspace/tiera-rerun-fixed-moonep/ultraep/tierbmem/tierbmem_gbs16/G3__ultraep__matched__seq4096__gbs16__capdynamic__nocg__memiteration__it5/rep1
2878
+
2879
+ Results: /workspace/tiera-rerun-fixed-moonep/ultraep/tierbmem/tierbmem_gbs16/results.tsv
2880
+ Roll up: python3 /workspace/blackwell_run_scripts/granularity/tools/collect.py /workspace/tiera-rerun-fixed-moonep/ultraep/tierbmem/tierbmem_gbs16
2881
+ ==============================================================
2882
+ 6 memory-dump runs -> /workspace/tiera-rerun-fixed-moonep/ultraep/tierbmem/tierbmem_gbs16/memory_summary.tsv
2883
+ model backend cg allocation capacity status peak GiB dumps
2884
+ ----------------------------------------------------------------------------------
2885
+ G1 ultraep 1 static 2.40625 OK 126.4 48/48
2886
+ G1 ultraep 0 dynamic dynamic OK 106.7 40/40
2887
+ G2 ultraep 1 static1.08203125 OK 109.7 48/48
2888
+ G2 ultraep 0 dynamic dynamic OK 105.7 40/40
2889
+ G3 ultraep 1 static1.072265625 OK 111.3 48/48
2890
+ G3 ultraep 0 dynamic dynamic OK 109.1 40/40
2891
+
2892
+ ECHO/UltraEP note: graph-on uses static capacity + device-initiated GEMM while graph-off uses dynamic allocation + host-derived sizes. Their difference is not labelled as pure graph cost.
2893
+
2894
+ TIER B MEMORY DUMPS -- how to read this
2895
+ ---------------------------------------
2896
+ ECHO / UltraEP:
2897
+ cap1p00__memiteration__it6 graph on, static C=1.00
2898
+ cap1p125__memiteration__it6 graph on, static C=1.125
2899
+ cap1p25__memiteration__it6 graph on, static C=1.25
2900
+ capdynamic__nocg__memiteration__it5
2901
+ graph off, no capacity flag, dynamic allocation
2902
+
2903
+ MoonEP:
2904
+ capnone__memiteration__it6 graph on, no capacity factor
2905
+ capnone__nocg__memiteration__it5 graph off, no capacity factor
2906
+
2907
+ Every successful cell contains:
2908
+ train.log Megatron and OOM diagnostics
2909
+ mem_samples.csv driver-level nvidia-smi samples every 2 seconds
2910
+ run.env exact branch/config/capacity provenance
2911
+ memory_snapshots/*.pickle 48 captured / 40 eager allocator histories
2912
+ memory_snapshots/*.json matching summaries and before/after deltas
2913
+
2914
+ The graph-on/static vs graph-off/dynamic difference is intentionally NOT called
2915
+ "graph cost": capacity allocation policy and device-initiated grouped GEMM also
2916
+ change. Compare the three graph-on rows to measure static-capacity sensitivity,
2917
+ and use the dynamic row as the natural eager-allocation reference.
2918
+
2919
+ Campaign results: /workspace/tiera-rerun-fixed-moonep/ultraep/tierbmem
2920
+ Roll up: python3 /workspace/blackwell_run_scripts/granularity/tools/collect.py /workspace/tiera-rerun-fixed-moonep/ultraep/tierbmem --compare
2921
+ Already logged in to Hugging Face as: AbhishekGhosh1998
2922
+ uploaded: /workspace/tiera-rerun-fixed-moonep/ultraep/tierbmem -> MoE-CUDA-Graph-Analysis/tierb-b200-gbs8/tiera-rerun-fixed-moonep/ultraep/tierbmem
2923
+ https://huggingface.co/MoE-CUDA-Graph-Analysis/tierb-b200-gbs8/tree/main
2924
+
2925
+ === STEP moonep__tierbmem__gbs16__G3
2926
+ ==============================================================
2927
+ CAMPAIGN: moonep backend=moonep phase=tierbmem
2928
+ Megatron: /workspace/Megatron-MoE-Echo @ moonep_backend_async_grad_reduce_better_memory_dump_support
2929
+ GBS=16 seq=4096 repeats=3 nsys=0
2930
+ results -> /workspace/tiera-rerun-fixed-moonep/moonep/tierbmem
2931
+ ==============================================================
2932
+ --------------------------------------------------------------------------
2933
+ [10:14:20] CELL G3__moonep__matched__seq4096__gbs16__capnone__async__memiteration__it6 (slots=8)
2934
+ [guard] iter 3/6 (60s)
2935
+ [guard] iter 6/6 (70s)
2936
+ [guard] tearing down process group (outcome=DONE)
2937
+ --------------------------------------------------------------------------
2938
+ [10:15:34] CELL G3__moonep__matched__seq4096__gbs16__capnone__nocg__async__memiteration__it5 (slots=8)
2939
+ [guard] iter 3/5 (60s)
2940
+ [guard] iter 5/5 (70s)
2941
+ [guard] tearing down process group (outcome=DONE)
2942
+ --------------------------------------------------------------------------
2943
+
2944
+ ==============================================================
2945
+ SWEEP COMPLETE ok=2 failed=0
2946
+ ==============================================================
2947
+ config backend policy slots seq gbs cap recompute rep status iter_ms_mean iter_ms_min peak_mib run_dir
2948
+ G3 moonep matched 8 4096 16 none none 1 OK 578.30 578.30 99411 /workspace/tiera-rerun-fixed-moonep/moonep/tierbmem/tierbmem_gbs16/G3__moonep__matched__seq4096__gbs16__capnone__async__memiteration__it6/rep1
2949
+ G3 moonep matched 8 4096 16 none none 1 OK NA NA 98611 /workspace/tiera-rerun-fixed-moonep/moonep/tierbmem/tierbmem_gbs16/G3__moonep__matched__seq4096__gbs16__capnone__nocg__async__memiteration__it5/rep1
2950
+
2951
+ Results: /workspace/tiera-rerun-fixed-moonep/moonep/tierbmem/tierbmem_gbs16/results.tsv
2952
+ Roll up: python3 /workspace/blackwell_run_scripts/granularity/tools/collect.py /workspace/tiera-rerun-fixed-moonep/moonep/tierbmem/tierbmem_gbs16
2953
+ ==============================================================
2954
+ 6 memory-dump runs -> /workspace/tiera-rerun-fixed-moonep/moonep/tierbmem/tierbmem_gbs16/memory_summary.tsv
2955
+ model backend cg allocation capacity status peak GiB dumps
2956
+ ----------------------------------------------------------------------------------
2957
+ G1 moonep 1 none none OK 94.6 48/48
2958
+ G1 moonep 0 none none OK 90.4 40/40
2959
+ G2 moonep 1 none none OK 95.4 48/48
2960
+ G2 moonep 0 none none OK 93.2 40/40
2961
+ G3 moonep 1 none none OK 97.1 48/48
2962
+ G3 moonep 0 none none OK 96.3 40/40
2963
+
2964
+ ECHO/UltraEP note: graph-on uses static capacity + device-initiated GEMM while graph-off uses dynamic allocation + host-derived sizes. Their difference is not labelled as pure graph cost.
2965
+
2966
+ TIER B MEMORY DUMPS -- how to read this
2967
+ ---------------------------------------
2968
+ ECHO / UltraEP:
2969
+ cap1p00__memiteration__it6 graph on, static C=1.00
2970
+ cap1p125__memiteration__it6 graph on, static C=1.125
2971
+ cap1p25__memiteration__it6 graph on, static C=1.25
2972
+ capdynamic__nocg__memiteration__it5
2973
+ graph off, no capacity flag, dynamic allocation
2974
+
2975
+ MoonEP:
2976
+ capnone__memiteration__it6 graph on, no capacity factor
2977
+ capnone__nocg__memiteration__it5 graph off, no capacity factor
2978
+
2979
+ Every successful cell contains:
2980
+ train.log Megatron and OOM diagnostics
2981
+ mem_samples.csv driver-level nvidia-smi samples every 2 seconds
2982
+ run.env exact branch/config/capacity provenance
2983
+ memory_snapshots/*.pickle 48 captured / 40 eager allocator histories
2984
+ memory_snapshots/*.json matching summaries and before/after deltas
2985
+
2986
+ The graph-on/static vs graph-off/dynamic difference is intentionally NOT called
2987
+ "graph cost": capacity allocation policy and device-initiated grouped GEMM also
2988
+ change. Compare the three graph-on rows to measure static-capacity sensitivity,
2989
+ and use the dynamic row as the natural eager-allocation reference.
2990
+
2991
+ Campaign results: /workspace/tiera-rerun-fixed-moonep/moonep/tierbmem
2992
+ Roll up: python3 /workspace/blackwell_run_scripts/granularity/tools/collect.py /workspace/tiera-rerun-fixed-moonep/moonep/tierbmem --compare
2993
+ Already logged in to Hugging Face as: AbhishekGhosh1998
2994
+ uploaded: /workspace/tiera-rerun-fixed-moonep/moonep/tierbmem -> MoE-CUDA-Graph-Analysis/tierb-b200-gbs8/tiera-rerun-fixed-moonep/moonep/tierbmem
2995
+ https://huggingface.co/MoE-CUDA-Graph-Analysis/tierb-b200-gbs8/tree/main
2996
+
2997
+ === ROLL-UP
2998
+ 134 runs ok=125 oom=0 -> /workspace/tiera-rerun-fixed-moonep/collected_all.tsv
2999
+
3000
+ WARNING: 11 run(s) mention capacity saturation. A run that drops assignments does less expert work and must not be quoted as a win:
3001
+ ultraep/tierbprobe/tierbprobe_gbs16/G1__ultraep__matched__seq4096__gbs16__cap1p25/rep1: WARNING:megatron.training.drop_debug_utils:[MOE_DROP_DEBUG][iteration_summary] global_rank=0 iteration=0 records=32 total_expected_recv=283999 total_dropped_tok
3002
+ ultraep/tierbprobe/tierbprobe_gbs16/G1__ultraep__matched__seq4096__gbs16__cap1p8125/rep1: WARNING:megatron.training.drop_debug_utils:[MOE_DROP_DEBUG][iteration_summary] global_rank=0 iteration=0 records=32 total_expected_recv=284540 total_dropped_tok
3003
+ ultraep/tierbprobe/tierbprobe_gbs16/G2__ultraep__matched__seq4096__gbs16__cap1p0625/rep1: WARNING:megatron.training.drop_debug_utils:[MOE_DROP_DEBUG][iteration_summary] global_rank=0 iteration=0 records=32 total_expected_recv=1048800 total_dropped_to
3004
+ ultraep/tierbprobe/tierbprobe_gbs16/G2__ultraep__matched__seq4096__gbs16__cap1p25/rep1: WARNING:megatron.training.drop_debug_utils:[MOE_DROP_DEBUG][iteration_summary] global_rank=0 iteration=0 records=32 total_expected_recv=1048800 total_dropped_to
3005
+ ultraep/tierbprobe/tierbprobe_gbs16/G2__ultraep__matched__seq4096__gbs16__cap1p25__forcebal/rep1: WARNING:megatron.training.drop_debug_utils:[MOE_DROP_DEBUG][iteration_summary] global_rank=1 iteration=0 records=32 total_expected_recv=1047984 total_dropped_to
3006
+
3007
+ (excluded 14 nsys-traced run(s) from the timing comparison)
3008
+
3009
+ phase config seq gbs policy rc cg asy backend iter_ms vs_best
3010
+ -------------------------------------------------------------------------------------
3011
+ a1 G1 4096 16 one none 1 0 echo 466.68 1.00x
3012
+ ultraep 891.00 1.91x
3013
+
3014
+ a1 G1 4096 16 one none 1 1 moonep 480.30 1.00x
3015
+
3016
+ a1 G2 4096 16 one none 1 0 echo 394.28 1.00x
3017
+ ultraep 466.65 1.18x
3018
+
3019
+ a1 G3 4096 16 one none 1 0 echo 412.83 1.00x
3020
+ ultraep 447.84 1.08x
3021
+
3022
+ a1 G4 4096 16 one none 1 0 echo 461.03 1.00x
3023
+ ultraep 467.57 1.01x
3024
+
3025
+ a2 G1 4096 16 matched none 1 0 echo 466.91 1.00x
3026
+ ultraep 891.92 1.91x
3027
+
3028
+ a2 G1 4096 16 matched none 1 1 moonep 481.37 1.00x
3029
+
3030
+ a2 G2 4096 16 matched none 1 0 echo 415.21 1.00x
3031
+ ultraep 492.54 1.19x
3032
+
3033
+ a2 G2 4096 16 matched none 1 1 moonep 432.02 1.00x
3034
+
3035
+ a2 G3 4096 16 matched none 1 0 echo 437.35 1.00x
3036
+ ultraep 459.60 1.05x
3037
+
3038
+ a2 G3 4096 16 matched none 1 1 moonep 559.88 1.00x
3039
+
3040
+ a2 G4 4096 16 matched none 1 0 ultraep 482.64 1.00x
3041
+ echo 486.54 1.01x
3042
+
3043
+ a2 G4 4096 16 matched none 1 1 moonep 783.34 1.00x
3044
+
3045
+ ponly PONLY 4096 16 one none 1 0 echo 174.81 1.00x
3046
+ ultraep 176.76 1.01x
3047
+
3048
+ ponly PONLY 4096 16 one none 1 1 moonep 165.22 1.00x
3049
+
3050
+ ponly PONLY 8192 16 one none 1 0 ultraep 292.62 1.00x
3051
+ echo 316.70 1.08x
3052
+
3053
+ ponly PONLY 8192 16 one none 1 1 moonep 300.61 1.00x
3054
+
3055
+ tierbmem G1 4096 16 matched none 1 0 echo 474.40 1.00x
3056
+ ultraep 909.90 1.92x
3057
+
3058
+ tierbmem G1 4096 16 matched none 1 1 moonep 497.40 1.00x
3059
+
3060
+ tierbmem G2 4096 16 matched none 1 0 echo 430.10 1.00x
3061
+ ultraep 475.40 1.11x
3062
+
3063
+ tierbmem G2 4096 16 matched none 1 1 moonep 439.70 1.00x
3064
+
3065
+ tierbmem G3 4096 16 matched none 1 0 echo 452.40 1.00x
3066
+ ultraep 477.80 1.06x
3067
+
3068
+ tierbmem G3 4096 16 matched none 1 1 moonep 578.30 1.00x
3069
+
3070
+ tierbprobeG1 4096 16 matched none 0 0 echo 671.53 1.00x
3071
+ ultraep 904.16 1.35x
3072
+
3073
+ tierbprobeG2 4096 16 matched none 0 0 ultraep 512.37 1.00x
3074
+ echo 654.76 1.28x
3075
+
3076
+ tierbprobeG3 4096 16 matched none 0 0 ultraep 512.31 1.00x
3077
+ echo 689.13 1.35x
3078
+
3079
+ tierbprobeG4 4096 16 matched none 0 0 ultraep 600.19 1.00x
3080
+ echo 790.97 1.32x
3081
+
3082
+ 21 probe runs -> /workspace/tiera-rerun-fixed-moonep/probe_summary_all.tsv
3083
+ run arm cap rawExpI preRankI postRankI drop% reqCap
3084
+ --------------------------------------------------------------------------------------------------------------------
3085
+ echo/tierbprobe/tierbprobe_gbs16/G1__echo__matched__ natural 2.00 3.6198* 3.6198 1.0827 1.9062
3086
+ echo/tierbprobe/tierbprobe_gbs16/G2__echo__matched__ natural 1.08203125 3.8720* 2.1703 1.0000 1.0273
3087
+ echo/tierbprobe/tierbprobe_gbs16/G2__echo__matched__ forced 1.08203125 1.0201 1.0069 1.0000 1.0234
3088
+ echo/tierbprobe/tierbprobe_gbs16/G2__echo__matched__ natural 2.00 3.8724* 2.1875 1.0000 1.0273
3089
+ echo/tierbprobe/tierbprobe_gbs16/G3__echo__matched__ natural 1.072265625 3.9096* 1.7838 1.0000 1.0195
3090
+ echo/tierbprobe/tierbprobe_gbs16/G3__echo__matched__ forced 1.072265625 1.0231 1.0048 1.0000 1.0176
3091
+ echo/tierbprobe/tierbprobe_gbs16/G3__echo__matched__ natural 2.00 3.9096* 1.7916 1.0000 1.0195
3092
+ echo/tierbprobe/tierbprobe_gbs16/G4__echo__matched__ natural 1.068359375 3.9303* 1.5507 1.0000 1.0166
3093
+ echo/tierbprobe/tierbprobe_gbs16/G4__echo__matched__ forced 1.068359375 1.0249 1.0034 1.0000 1.0137
3094
+ echo/tierbprobe/tierbprobe_gbs16/G4__echo__matched__ natural 2.00 3.9301* 1.5543 1.0000 1.0166
3095
+ ultraep/tierbprobe/tierbprobe_gbs16/G1__ultraep__mat natural 1.25 3.6085 1.0640 2.8918 1.7246
3096
+ ultraep/tierbprobe/tierbprobe_gbs16/G1__ultraep__mat natural 1.8125 3.6152 1.0747 1.5460 2.3091
3097
+ ultraep/tierbprobe/tierbprobe_gbs16/G2__ultraep__mat natural 1.0625 2.2032 1.0100 0.0000 1.0101
3098
+ ultraep/tierbprobe/tierbprobe_gbs16/G2__ultraep__mat natural 1.25 2.2136 1.0100 0.0000 1.0101
3099
+ ultraep/tierbprobe/tierbprobe_gbs16/G2__ultraep__mat forced 1.25 1.0069 1.0067 0.0000 1.0100
3100
+ ultraep/tierbprobe/tierbprobe_gbs16/G3__ultraep__mat natural 1.0625 1.7842 1.0100 0.0000 1.0101
3101
+ ultraep/tierbprobe/tierbprobe_gbs16/G3__ultraep__mat natural 1.25 1.7752 1.0100 0.0000 1.0101
3102
+ ultraep/tierbprobe/tierbprobe_gbs16/G3__ultraep__mat forced 1.25 1.0047 1.0047 0.0000 1.0100
3103
+ ultraep/tierbprobe/tierbprobe_gbs16/G4__ultraep__mat natural 1.095703125 1.5612 1.0100 0.0000 1.0429
3104
+ ultraep/tierbprobe/tierbprobe_gbs16/G4__ultraep__mat natural 1.25 1.5504 1.0100 0.0000 1.0429
3105
+ ultraep/tierbprobe/tierbprobe_gbs16/G4__ultraep__mat forced 1.25 1.0033 1.0033 0.0000 1.0071
3106
+
3107
+ * At least one globally aggregated snapshot reached the E/K expert-Imax ceiling.
3108
+ Inspect raw_expert_imax_mean and hottest_expert_token_fraction_mean before generalising.
3109
+
3110
+ ECHO note: echo_logged_pre_rank_metric_valid=0 and echo_logged_load_reduction_valid=0 are intentional.
3111
+ The branch's built-in fields mix source-rank and destination-rank axes; this tool reconstructs both globally.
3112
+ 18 memory-dump runs -> /workspace/tiera-rerun-fixed-moonep/memory_summary_all.tsv
3113
+ model backend cg allocation capacity status peak GiB dumps
3114
+ ----------------------------------------------------------------------------------
3115
+ G1 echo 1 static 2.40625 OK 157.9 48/48
3116
+ G1 echo 0 dynamic dynamic OK 112.6 40/40
3117
+ G1 moonep 1 none none OK 94.6 48/48
3118
+ G1 moonep 0 none none OK 90.4 40/40
3119
+ G1 ultraep 1 static 2.40625 OK 126.4 48/48
3120
+ G1 ultraep 0 dynamic dynamic OK 106.7 40/40
3121
+ G2 echo 1 static1.08203125 OK 142.5 48/48
3122
+ G2 echo 0 dynamic dynamic OK 115.1 40/40
3123
+ G2 moonep 1 none none OK 95.4 48/48
3124
+ G2 moonep 0 none none OK 93.2 40/40
3125
+ G2 ultraep 1 static1.08203125 OK 109.7 48/48
3126
+ G2 ultraep 0 dynamic dynamic OK 105.7 40/40
3127
+ G3 echo 1 static1.072265625 OK 144.5 48/48
3128
+ G3 echo 0 dynamic dynamic OK 116.1 40/40
3129
+ G3 moonep 1 none none OK 97.1 48/48
3130
+ G3 moonep 0 none none OK 96.3 40/40
3131
+ G3 ultraep 1 static1.072265625 OK 111.3 48/48
3132
+ G3 ultraep 0 dynamic dynamic OK 109.1 40/40
3133
+
3134
+ ECHO/UltraEP note: graph-on uses static capacity + device-initiated GEMM while graph-off uses dynamic allocation + host-derived sizes. Their difference is not labelled as pure graph cost.
3135
+ Already logged in to Hugging Face as: AbhishekGhosh1998