granularity results tierb-fixed-moonep-full @ 2026-08-30T01:46:08Z from sp-aiinfra-interactive-aghosh-7jtjl-8xb200-moe-cg-0 (part 2)
Browse files
tierb-fixed-moonep-full/steps.tsv
CHANGED
|
@@ -5,3 +5,4 @@ OK moonep tierbmem 8 2026-08-29T23:59:59+00:00 2026-08-30T00:09:58+00:00 /worksp
|
|
| 5 |
OK echo tierbtrace 8 2026-08-30T00:10:16+00:00 2026-08-30T00:38:30+00:00 /workspace/tierb-fixed-moonep-full/echo/tierbtrace
|
| 6 |
OK ultraep tierbtrace 8 2026-08-30T00:38:33+00:00 2026-08-30T00:59:28+00:00 /workspace/tierb-fixed-moonep-full/ultraep/tierbtrace
|
| 7 |
FAIL moonep tierbtrace 8 2026-08-30T00:59:40+00:00 2026-08-30T01:06:12+00:00 /workspace/tierb-fixed-moonep-full/moonep/tierbtrace
|
|
|
|
|
|
| 5 |
OK echo tierbtrace 8 2026-08-30T00:10:16+00:00 2026-08-30T00:38:30+00:00 /workspace/tierb-fixed-moonep-full/echo/tierbtrace
|
| 6 |
OK ultraep tierbtrace 8 2026-08-30T00:38:33+00:00 2026-08-30T00:59:28+00:00 /workspace/tierb-fixed-moonep-full/ultraep/tierbtrace
|
| 7 |
FAIL moonep tierbtrace 8 2026-08-30T00:59:40+00:00 2026-08-30T01:06:12+00:00 /workspace/tierb-fixed-moonep-full/moonep/tierbtrace
|
| 8 |
+
OK moonep tierb 8 2026-08-30T01:30:54+00:00 2026-08-30T01:46:01+00:00 /workspace/tierb-fixed-moonep-full/moonep/tierb
|
tierb-fixed-moonep-full/study.log
CHANGED
|
@@ -771,3 +771,110 @@ driver_repo: /workspace/blackwell_run_scripts @ eec9aee
|
|
| 771 |
[guard] iter 12/12 (70s)
|
| 772 |
[guard] tearing down process group (outcome=DONE)
|
| 773 |
[guard] iter 1/12 (60s)
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 771 |
[guard] iter 12/12 (70s)
|
| 772 |
[guard] tearing down process group (outcome=DONE)
|
| 773 |
[guard] iter 1/12 (60s)
|
| 774 |
+
[guard] iter 12/12 (70s)
|
| 775 |
+
[guard] tearing down process group (outcome=DONE)
|
| 776 |
+
[guard] iter 1/12 (60s)
|
| 777 |
+
[guard] iter 12/12 (70s)
|
| 778 |
+
[guard] tearing down process group (outcome=DONE)
|
| 779 |
+
--------------------------------------------------------------------------
|
| 780 |
+
[01:42:26] CELL QWEN2_57B__moonep__matched__seq4096__gbs8__capnone__async (slots=8)
|
| 781 |
+
[guard] iter 4/12 (60s)
|
| 782 |
+
[guard] iter 12/12 (70s)
|
| 783 |
+
[guard] tearing down process group (outcome=DONE)
|
| 784 |
+
[guard] iter 8/12 (60s)
|
| 785 |
+
[guard] iter 12/12 (70s)
|
| 786 |
+
[guard] tearing down process group (outcome=DONE)
|
| 787 |
+
[guard] iter 5/12 (60s)
|
| 788 |
+
[guard] iter 12/12 (70s)
|
| 789 |
+
[guard] tearing down process group (outcome=DONE)
|
| 790 |
+
--------------------------------------------------------------------------
|
| 791 |
+
|
| 792 |
+
==============================================================
|
| 793 |
+
SWEEP COMPLETE ok=12 failed=0
|
| 794 |
+
==============================================================
|
| 795 |
+
config backend policy slots seq gbs cap recompute rep status iter_ms_mean iter_ms_min peak_mib run_dir
|
| 796 |
+
QWEN3_235B moonep matched 16 8192 8 none none 1 OK 317.23 316.40 120309 /workspace/tierb-fixed-moonep-full/moonep/tierb/tierb_gbs8/QWEN3_235B__moonep__matched__seq8192__gbs8__capnone__async/rep1
|
| 797 |
+
QWEN3_235B moonep matched 16 8192 8 none none 2 OK 317.53 316.40 120309 /workspace/tierb-fixed-moonep-full/moonep/tierb/tierb_gbs8/QWEN3_235B__moonep__matched__seq8192__gbs8__capnone__async/rep2
|
| 798 |
+
QWEN3_235B moonep matched 16 8192 8 none none 3 OK 317.80 316.40 120309 /workspace/tierb-fixed-moonep-full/moonep/tierb/tierb_gbs8/QWEN3_235B__moonep__matched__seq8192__gbs8__capnone__async/rep3
|
| 799 |
+
QWEN3_30B moonep matched 16 4096 8 none none 1 OK 354.41 352.30 116707 /workspace/tierb-fixed-moonep-full/moonep/tierb/tierb_gbs8/QWEN3_30B__moonep__matched__seq4096__gbs8__capnone__async/rep1
|
| 800 |
+
QWEN3_30B moonep matched 16 4096 8 none none 2 OK 354.46 352.20 116707 /workspace/tierb-fixed-moonep-full/moonep/tierb/tierb_gbs8/QWEN3_30B__moonep__matched__seq4096__gbs8__capnone__async/rep2
|
| 801 |
+
QWEN3_30B moonep matched 16 4096 8 none none 3 OK 355.16 352.20 116707 /workspace/tierb-fixed-moonep-full/moonep/tierb/tierb_gbs8/QWEN3_30B__moonep__matched__seq4096__gbs8__capnone__async/rep3
|
| 802 |
+
DSV2_LITE_L24 moonep matched 8 8192 8 none none 1 OK 591.91 590.10 133311 /workspace/tierb-fixed-moonep-full/moonep/tierb/tierb_gbs8/DSV2_LITE_L24__moonep__matched__seq8192__gbs8__capnone__async/rep1
|
| 803 |
+
DSV2_LITE_L24 moonep matched 8 8192 8 none none 2 OK 593.74 590.10 133311 /workspace/tierb-fixed-moonep-full/moonep/tierb/tierb_gbs8/DSV2_LITE_L24__moonep__matched__seq8192__gbs8__capnone__async/rep2
|
| 804 |
+
DSV2_LITE_L24 moonep matched 8 8192 8 none none 3 OK 593.43 590.10 133311 /workspace/tierb-fixed-moonep-full/moonep/tierb/tierb_gbs8/DSV2_LITE_L24__moonep__matched__seq8192__gbs8__capnone__async/rep3
|
| 805 |
+
QWEN2_57B moonep matched 8 4096 8 none none 1 OK 180.76 179.90 96739 /workspace/tierb-fixed-moonep-full/moonep/tierb/tierb_gbs8/QWEN2_57B__moonep__matched__seq4096__gbs8__capnone__async/rep1
|
| 806 |
+
QWEN2_57B moonep matched 8 4096 8 none none 2 OK 180.24 178.80 96739 /workspace/tierb-fixed-moonep-full/moonep/tierb/tierb_gbs8/QWEN2_57B__moonep__matched__seq4096__gbs8__capnone__async/rep2
|
| 807 |
+
QWEN2_57B moonep matched 8 4096 8 none none 3 OK 180.11 178.80 96739 /workspace/tierb-fixed-moonep-full/moonep/tierb/tierb_gbs8/QWEN2_57B__moonep__matched__seq4096__gbs8__capnone__async/rep3
|
| 808 |
+
|
| 809 |
+
Results: /workspace/tierb-fixed-moonep-full/moonep/tierb/tierb_gbs8/results.tsv
|
| 810 |
+
Roll up: python3 /workspace/blackwell_run_scripts/granularity/tools/collect.py /workspace/tierb-fixed-moonep-full/moonep/tierb/tierb_gbs8
|
| 811 |
+
==============================================================
|
| 812 |
+
|
| 813 |
+
Campaign results: /workspace/tierb-fixed-moonep-full/moonep/tierb
|
| 814 |
+
Roll up: python3 /workspace/blackwell_run_scripts/granularity/tools/collect.py /workspace/tierb-fixed-moonep-full/moonep/tierb --compare
|
| 815 |
+
Already logged in to Hugging Face as: AbhishekGhosh1998
|
| 816 |
+
uploaded: /workspace/tierb-fixed-moonep-full/moonep/tierb -> MoE-CUDA-Graph-Analysis/tierb-b200-gbs8/tierb-fixed-moonep-full/moonep/tierb
|
| 817 |
+
https://huggingface.co/MoE-CUDA-Graph-Analysis/tierb-b200-gbs8/tree/main
|
| 818 |
+
|
| 819 |
+
=== ROLL-UP
|
| 820 |
+
36 runs ok=32 oom=0 -> /workspace/tierb-fixed-moonep-full/collected_all.tsv
|
| 821 |
+
|
| 822 |
+
(excluded 8 nsys-traced run(s) from the timing comparison)
|
| 823 |
+
|
| 824 |
+
phase config seq gbs policy rc cg asy backend iter_ms vs_best
|
| 825 |
+
-------------------------------------------------------------------------------------
|
| 826 |
+
tierb DSV2_LITE_L24 8192 8 matched none 1 1 moonep 593.43 1.00x
|
| 827 |
+
|
| 828 |
+
tierb QWEN2_57B 4096 8 matched none 1 1 moonep 180.12 1.00x
|
| 829 |
+
|
| 830 |
+
tierb QWEN3_235B 8192 8 matched none 1 1 moonep 317.80 1.00x
|
| 831 |
+
|
| 832 |
+
tierb QWEN3_30B 4096 8 matched none 1 1 moonep 355.16 1.00x
|
| 833 |
+
|
| 834 |
+
tierbmem DSV2_LITE_L24 8192 8 matched none 1 1 moonep 622.80 1.00x
|
| 835 |
+
|
| 836 |
+
tierbmem QWEN2_57B 4096 8 matched none 1 1 moonep 196.60 1.00x
|
| 837 |
+
|
| 838 |
+
tierbmem QWEN3_235B 8192 8 matched none 1 1 moonep 341.70 1.00x
|
| 839 |
+
|
| 840 |
+
tierbmem QWEN3_30B 4096 8 matched none 1 1 moonep 435.30 1.00x
|
| 841 |
+
|
| 842 |
+
tierbprobeDSV2_LITE_L24 8192 8 matched none 0 1 moonep 673.94 1.00x
|
| 843 |
+
|
| 844 |
+
tierbprobeQWEN2_57B 4096 8 matched none 0 1 moonep 238.91 1.00x
|
| 845 |
+
|
| 846 |
+
tierbprobeQWEN3_235B 8192 8 matched none 0 1 moonep 384.76 1.00x
|
| 847 |
+
|
| 848 |
+
tierbprobeQWEN3_30B 4096 8 matched none 0 1 moonep 824.23 1.00x
|
| 849 |
+
|
| 850 |
+
8 probe runs -> /workspace/tierb-fixed-moonep-full/probe_summary_all.tsv
|
| 851 |
+
run arm cap rawExpI preRankI postRankI drop% reqCap
|
| 852 |
+
--------------------------------------------------------------------------------------------------------------------
|
| 853 |
+
moonep/tierbprobe/tierbprobe_gbs8/DSV2_LITE_L24__moo natural none 7.5443* 2.1316 1.1058
|
| 854 |
+
moonep/tierbprobe/tierbprobe_gbs8/DSV2_LITE_L24__moo forced none 1.0286 1.0061 1.0026
|
| 855 |
+
moonep/tierbprobe/tierbprobe_gbs8/QWEN2_57B__moonep_ natural none 7.4132* 2.3019 1.0979
|
| 856 |
+
moonep/tierbprobe/tierbprobe_gbs8/QWEN2_57B__moonep_ forced none 1.0348 1.0073 1.0033
|
| 857 |
+
moonep/tierbprobe/tierbprobe_gbs8/QWEN3_235B__moonep natural none 15.9227* 2.5851 1.0712
|
| 858 |
+
moonep/tierbprobe/tierbprobe_gbs8/QWEN3_235B__moonep forced none 1.0386 1.0053 1.0025
|
| 859 |
+
moonep/tierbprobe/tierbprobe_gbs8/QWEN3_30B__moonep_ natural none 15.7078* 2.4672 1.0635
|
| 860 |
+
moonep/tierbprobe/tierbprobe_gbs8/QWEN3_30B__moonep_ forced none 1.0565 1.0077 1.0036
|
| 861 |
+
|
| 862 |
+
* At least one globally aggregated snapshot reached the E/K expert-Imax ceiling.
|
| 863 |
+
Inspect raw_expert_imax_mean and hottest_expert_token_fraction_mean before generalising.
|
| 864 |
+
|
| 865 |
+
ECHO note: echo_logged_pre_rank_metric_valid=0 and echo_logged_load_reduction_valid=0 are intentional.
|
| 866 |
+
The branch's built-in fields mix source-rank and destination-rank axes; this tool reconstructs both globally.
|
| 867 |
+
8 memory-dump runs -> /workspace/tierb-fixed-moonep-full/memory_summary_all.tsv
|
| 868 |
+
model backend cg allocation capacity status peak GiB dumps
|
| 869 |
+
----------------------------------------------------------------------------------
|
| 870 |
+
DSV2_LITE_L24 moonep 1 none none OK 130.2 48/48
|
| 871 |
+
DSV2_LITE_L24 moonep 0 none none OK 122.2 40/40
|
| 872 |
+
QWEN2_57B moonep 1 none none OK 94.5 48/48
|
| 873 |
+
QWEN2_57B moonep 0 none none OK 93.6 40/40
|
| 874 |
+
QWEN3_235B moonep 1 none none OK 117.5 48/48
|
| 875 |
+
QWEN3_235B moonep 0 none none OK 117.3 40/40
|
| 876 |
+
QWEN3_30B moonep 1 none none OK 114.0 48/48
|
| 877 |
+
QWEN3_30B moonep 0 none none OK 113.7 40/40
|
| 878 |
+
|
| 879 |
+
ECHO/UltraEP note: graph-on uses static capacity + device-initiated GEMM while graph-off uses dynamic allocation + host-derived sizes. Their difference is not labelled as pure graph cost.
|
| 880 |
+
Already logged in to Hugging Face as: AbhishekGhosh1998
|