AbhishekGhosh1998 commited on
Commit
9a166fd
·
verified ·
1 Parent(s): c8abfc9

granularity results tierb-fixed-moonep-full @ 2026-08-30T01:46:08Z from sp-aiinfra-interactive-aghosh-7jtjl-8xb200-moe-cg-0 (part 2)

Browse files
tierb-fixed-moonep-full/steps.tsv CHANGED
@@ -5,3 +5,4 @@ OK moonep tierbmem 8 2026-08-29T23:59:59+00:00 2026-08-30T00:09:58+00:00 /worksp
5
  OK echo tierbtrace 8 2026-08-30T00:10:16+00:00 2026-08-30T00:38:30+00:00 /workspace/tierb-fixed-moonep-full/echo/tierbtrace
6
  OK ultraep tierbtrace 8 2026-08-30T00:38:33+00:00 2026-08-30T00:59:28+00:00 /workspace/tierb-fixed-moonep-full/ultraep/tierbtrace
7
  FAIL moonep tierbtrace 8 2026-08-30T00:59:40+00:00 2026-08-30T01:06:12+00:00 /workspace/tierb-fixed-moonep-full/moonep/tierbtrace
 
 
5
  OK echo tierbtrace 8 2026-08-30T00:10:16+00:00 2026-08-30T00:38:30+00:00 /workspace/tierb-fixed-moonep-full/echo/tierbtrace
6
  OK ultraep tierbtrace 8 2026-08-30T00:38:33+00:00 2026-08-30T00:59:28+00:00 /workspace/tierb-fixed-moonep-full/ultraep/tierbtrace
7
  FAIL moonep tierbtrace 8 2026-08-30T00:59:40+00:00 2026-08-30T01:06:12+00:00 /workspace/tierb-fixed-moonep-full/moonep/tierbtrace
8
+ OK moonep tierb 8 2026-08-30T01:30:54+00:00 2026-08-30T01:46:01+00:00 /workspace/tierb-fixed-moonep-full/moonep/tierb
tierb-fixed-moonep-full/study.log CHANGED
@@ -771,3 +771,110 @@ driver_repo: /workspace/blackwell_run_scripts @ eec9aee
771
  [guard] iter 12/12 (70s)
772
  [guard] tearing down process group (outcome=DONE)
773
  [guard] iter 1/12 (60s)
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
771
  [guard] iter 12/12 (70s)
772
  [guard] tearing down process group (outcome=DONE)
773
  [guard] iter 1/12 (60s)
774
+ [guard] iter 12/12 (70s)
775
+ [guard] tearing down process group (outcome=DONE)
776
+ [guard] iter 1/12 (60s)
777
+ [guard] iter 12/12 (70s)
778
+ [guard] tearing down process group (outcome=DONE)
779
+ --------------------------------------------------------------------------
780
+ [01:42:26] CELL QWEN2_57B__moonep__matched__seq4096__gbs8__capnone__async (slots=8)
781
+ [guard] iter 4/12 (60s)
782
+ [guard] iter 12/12 (70s)
783
+ [guard] tearing down process group (outcome=DONE)
784
+ [guard] iter 8/12 (60s)
785
+ [guard] iter 12/12 (70s)
786
+ [guard] tearing down process group (outcome=DONE)
787
+ [guard] iter 5/12 (60s)
788
+ [guard] iter 12/12 (70s)
789
+ [guard] tearing down process group (outcome=DONE)
790
+ --------------------------------------------------------------------------
791
+
792
+ ==============================================================
793
+ SWEEP COMPLETE ok=12 failed=0
794
+ ==============================================================
795
+ config backend policy slots seq gbs cap recompute rep status iter_ms_mean iter_ms_min peak_mib run_dir
796
+ QWEN3_235B moonep matched 16 8192 8 none none 1 OK 317.23 316.40 120309 /workspace/tierb-fixed-moonep-full/moonep/tierb/tierb_gbs8/QWEN3_235B__moonep__matched__seq8192__gbs8__capnone__async/rep1
797
+ QWEN3_235B moonep matched 16 8192 8 none none 2 OK 317.53 316.40 120309 /workspace/tierb-fixed-moonep-full/moonep/tierb/tierb_gbs8/QWEN3_235B__moonep__matched__seq8192__gbs8__capnone__async/rep2
798
+ QWEN3_235B moonep matched 16 8192 8 none none 3 OK 317.80 316.40 120309 /workspace/tierb-fixed-moonep-full/moonep/tierb/tierb_gbs8/QWEN3_235B__moonep__matched__seq8192__gbs8__capnone__async/rep3
799
+ QWEN3_30B moonep matched 16 4096 8 none none 1 OK 354.41 352.30 116707 /workspace/tierb-fixed-moonep-full/moonep/tierb/tierb_gbs8/QWEN3_30B__moonep__matched__seq4096__gbs8__capnone__async/rep1
800
+ QWEN3_30B moonep matched 16 4096 8 none none 2 OK 354.46 352.20 116707 /workspace/tierb-fixed-moonep-full/moonep/tierb/tierb_gbs8/QWEN3_30B__moonep__matched__seq4096__gbs8__capnone__async/rep2
801
+ QWEN3_30B moonep matched 16 4096 8 none none 3 OK 355.16 352.20 116707 /workspace/tierb-fixed-moonep-full/moonep/tierb/tierb_gbs8/QWEN3_30B__moonep__matched__seq4096__gbs8__capnone__async/rep3
802
+ DSV2_LITE_L24 moonep matched 8 8192 8 none none 1 OK 591.91 590.10 133311 /workspace/tierb-fixed-moonep-full/moonep/tierb/tierb_gbs8/DSV2_LITE_L24__moonep__matched__seq8192__gbs8__capnone__async/rep1
803
+ DSV2_LITE_L24 moonep matched 8 8192 8 none none 2 OK 593.74 590.10 133311 /workspace/tierb-fixed-moonep-full/moonep/tierb/tierb_gbs8/DSV2_LITE_L24__moonep__matched__seq8192__gbs8__capnone__async/rep2
804
+ DSV2_LITE_L24 moonep matched 8 8192 8 none none 3 OK 593.43 590.10 133311 /workspace/tierb-fixed-moonep-full/moonep/tierb/tierb_gbs8/DSV2_LITE_L24__moonep__matched__seq8192__gbs8__capnone__async/rep3
805
+ QWEN2_57B moonep matched 8 4096 8 none none 1 OK 180.76 179.90 96739 /workspace/tierb-fixed-moonep-full/moonep/tierb/tierb_gbs8/QWEN2_57B__moonep__matched__seq4096__gbs8__capnone__async/rep1
806
+ QWEN2_57B moonep matched 8 4096 8 none none 2 OK 180.24 178.80 96739 /workspace/tierb-fixed-moonep-full/moonep/tierb/tierb_gbs8/QWEN2_57B__moonep__matched__seq4096__gbs8__capnone__async/rep2
807
+ QWEN2_57B moonep matched 8 4096 8 none none 3 OK 180.11 178.80 96739 /workspace/tierb-fixed-moonep-full/moonep/tierb/tierb_gbs8/QWEN2_57B__moonep__matched__seq4096__gbs8__capnone__async/rep3
808
+
809
+ Results: /workspace/tierb-fixed-moonep-full/moonep/tierb/tierb_gbs8/results.tsv
810
+ Roll up: python3 /workspace/blackwell_run_scripts/granularity/tools/collect.py /workspace/tierb-fixed-moonep-full/moonep/tierb/tierb_gbs8
811
+ ==============================================================
812
+
813
+ Campaign results: /workspace/tierb-fixed-moonep-full/moonep/tierb
814
+ Roll up: python3 /workspace/blackwell_run_scripts/granularity/tools/collect.py /workspace/tierb-fixed-moonep-full/moonep/tierb --compare
815
+ Already logged in to Hugging Face as: AbhishekGhosh1998
816
+ uploaded: /workspace/tierb-fixed-moonep-full/moonep/tierb -> MoE-CUDA-Graph-Analysis/tierb-b200-gbs8/tierb-fixed-moonep-full/moonep/tierb
817
+ https://huggingface.co/MoE-CUDA-Graph-Analysis/tierb-b200-gbs8/tree/main
818
+
819
+ === ROLL-UP
820
+ 36 runs ok=32 oom=0 -> /workspace/tierb-fixed-moonep-full/collected_all.tsv
821
+
822
+ (excluded 8 nsys-traced run(s) from the timing comparison)
823
+
824
+ phase config seq gbs policy rc cg asy backend iter_ms vs_best
825
+ -------------------------------------------------------------------------------------
826
+ tierb DSV2_LITE_L24 8192 8 matched none 1 1 moonep 593.43 1.00x
827
+
828
+ tierb QWEN2_57B 4096 8 matched none 1 1 moonep 180.12 1.00x
829
+
830
+ tierb QWEN3_235B 8192 8 matched none 1 1 moonep 317.80 1.00x
831
+
832
+ tierb QWEN3_30B 4096 8 matched none 1 1 moonep 355.16 1.00x
833
+
834
+ tierbmem DSV2_LITE_L24 8192 8 matched none 1 1 moonep 622.80 1.00x
835
+
836
+ tierbmem QWEN2_57B 4096 8 matched none 1 1 moonep 196.60 1.00x
837
+
838
+ tierbmem QWEN3_235B 8192 8 matched none 1 1 moonep 341.70 1.00x
839
+
840
+ tierbmem QWEN3_30B 4096 8 matched none 1 1 moonep 435.30 1.00x
841
+
842
+ tierbprobeDSV2_LITE_L24 8192 8 matched none 0 1 moonep 673.94 1.00x
843
+
844
+ tierbprobeQWEN2_57B 4096 8 matched none 0 1 moonep 238.91 1.00x
845
+
846
+ tierbprobeQWEN3_235B 8192 8 matched none 0 1 moonep 384.76 1.00x
847
+
848
+ tierbprobeQWEN3_30B 4096 8 matched none 0 1 moonep 824.23 1.00x
849
+
850
+ 8 probe runs -> /workspace/tierb-fixed-moonep-full/probe_summary_all.tsv
851
+ run arm cap rawExpI preRankI postRankI drop% reqCap
852
+ --------------------------------------------------------------------------------------------------------------------
853
+ moonep/tierbprobe/tierbprobe_gbs8/DSV2_LITE_L24__moo natural none 7.5443* 2.1316 1.1058
854
+ moonep/tierbprobe/tierbprobe_gbs8/DSV2_LITE_L24__moo forced none 1.0286 1.0061 1.0026
855
+ moonep/tierbprobe/tierbprobe_gbs8/QWEN2_57B__moonep_ natural none 7.4132* 2.3019 1.0979
856
+ moonep/tierbprobe/tierbprobe_gbs8/QWEN2_57B__moonep_ forced none 1.0348 1.0073 1.0033
857
+ moonep/tierbprobe/tierbprobe_gbs8/QWEN3_235B__moonep natural none 15.9227* 2.5851 1.0712
858
+ moonep/tierbprobe/tierbprobe_gbs8/QWEN3_235B__moonep forced none 1.0386 1.0053 1.0025
859
+ moonep/tierbprobe/tierbprobe_gbs8/QWEN3_30B__moonep_ natural none 15.7078* 2.4672 1.0635
860
+ moonep/tierbprobe/tierbprobe_gbs8/QWEN3_30B__moonep_ forced none 1.0565 1.0077 1.0036
861
+
862
+ * At least one globally aggregated snapshot reached the E/K expert-Imax ceiling.
863
+ Inspect raw_expert_imax_mean and hottest_expert_token_fraction_mean before generalising.
864
+
865
+ ECHO note: echo_logged_pre_rank_metric_valid=0 and echo_logged_load_reduction_valid=0 are intentional.
866
+ The branch's built-in fields mix source-rank and destination-rank axes; this tool reconstructs both globally.
867
+ 8 memory-dump runs -> /workspace/tierb-fixed-moonep-full/memory_summary_all.tsv
868
+ model backend cg allocation capacity status peak GiB dumps
869
+ ----------------------------------------------------------------------------------
870
+ DSV2_LITE_L24 moonep 1 none none OK 130.2 48/48
871
+ DSV2_LITE_L24 moonep 0 none none OK 122.2 40/40
872
+ QWEN2_57B moonep 1 none none OK 94.5 48/48
873
+ QWEN2_57B moonep 0 none none OK 93.6 40/40
874
+ QWEN3_235B moonep 1 none none OK 117.5 48/48
875
+ QWEN3_235B moonep 0 none none OK 117.3 40/40
876
+ QWEN3_30B moonep 1 none none OK 114.0 48/48
877
+ QWEN3_30B moonep 0 none none OK 113.7 40/40
878
+
879
+ ECHO/UltraEP note: graph-on uses static capacity + device-initiated GEMM while graph-off uses dynamic allocation + host-derived sizes. Their difference is not labelled as pure graph cost.
880
+ Already logged in to Hugging Face as: AbhishekGhosh1998