AbhishekGhosh1998's picture
granularity results tierb-fixed-moonep-full @ 2026-08-30T02:50:16Z from sp-aiinfra-interactive-aghosh-7jtjl-8xb200-moe-cg-0
d5ef96d verified
Raw History Blame Contribute Delete
81.9 kB
study_root: /workspace/tierb-fixed-moonep-full gbs: 8 stages: 7 nsys: 1 hf_repo: MoE-CUDA-Graph-Analysis/tierb-b200-gbs8 hf_remote_root: tierb-fixed-moonep-full started: 2026-08-29T23:33:50+00:00 driver_repo: /workspace/blackwell_run_scripts @ 5054c89 === STAGE 7 -- Tier B probe and zero-drop capacity selection === STEP moonep__tierbprobe__gbs8 ============================================================== CAMPAIGN: moonep backend=moonep phase=tierbprobe Megatron: /workspace/Megatron-MoE-Echo @ moonep_backend_async_grad_reduce_token_routing_analysis GBS=8 seq=4096 repeats=3 nsys=1 results -> /workspace/tierb-fixed-moonep-full/moonep/tierbprobe ============================================================== -------------------------------------------------------------------------- [23:34:05] CELL QWEN3_235B__moonep__matched__seq8192__gbs8__capnone__async (slots=16) [guard] iter 12/12 (70s) [guard] tearing down process group (outcome=DONE) -------------------------------------------------------------------------- [23:35:21] CELL QWEN3_235B__moonep__matched__seq8192__gbs8__capnone__forcebal__async (slots=16) [guard] iter 12/12 (70s) [guard] tearing down process group (outcome=DONE) -------------------------------------------------------------------------- [23:36:36] CELL QWEN3_30B__moonep__matched__seq4096__gbs8__capnone__async (slots=16) [guard] iter 8/12 (70s) [guard] iter 12/12 (80s) [guard] tearing down process group (outcome=DONE) -------------------------------------------------------------------------- [23:38:01] CELL QWEN3_30B__moonep__matched__seq4096__gbs8__capnone__forcebal__async (slots=16) [guard] iter 9/12 (70s) [guard] iter 12/12 (80s) [guard] tearing down process group (outcome=DONE) -------------------------------------------------------------------------- [23:39:25] CELL DSV2_LITE_L24__moonep__matched__seq8192__gbs8__capnone__async (slots=8) [guard] iter 7/12 (70s) [guard] iter 12/12 (80s) [guard] tearing down process group (outcome=DONE) -------------------------------------------------------------------------- [23:40:49] CELL DSV2_LITE_L24__moonep__matched__seq8192__gbs8__capnone__forcebal__async (slots=8) [guard] iter 12/12 (70s) [guard] tearing down process group (outcome=DONE) -------------------------------------------------------------------------- [23:42:04] CELL QWEN2_57B__moonep__matched__seq4096__gbs8__capnone__async (slots=8) [guard] iter 6/12 (60s) [guard] iter 12/12 (70s) [guard] tearing down process group (outcome=DONE) -------------------------------------------------------------------------- [23:43:17] CELL QWEN2_57B__moonep__matched__seq4096__gbs8__capnone__forcebal__async (slots=8) [guard] iter 8/12 (60s) [guard] iter 12/12 (70s) [guard] tearing down process group (outcome=DONE) -------------------------------------------------------------------------- ============================================================== SWEEP COMPLETE ok=8 failed=0 ============================================================== config backend policy slots seq gbs cap recompute rep status iter_ms_mean iter_ms_min peak_mib run_dir QWEN3_235B moonep matched 16 8192 8 none none 1 OK 370.33 368.00 120129 /workspace/tierb-fixed-moonep-full/moonep/tierbprobe/tierbprobe_gbs8/QWEN3_235B__moonep__matched__seq8192__gbs8__capnone__async/rep1 QWEN3_235B moonep matched 16 8192 8 none none+forcebal 1 OK 399.19 354.00 120081 /workspace/tierb-fixed-moonep-full/moonep/tierbprobe/tierbprobe_gbs8/QWEN3_235B__moonep__matched__seq8192__gbs8__capnone__forcebal__async/rep1 QWEN3_30B moonep matched 16 4096 8 none none 1 OK 829.20 747.60 116419 /workspace/tierb-fixed-moonep-full/moonep/tierbprobe/tierbprobe_gbs8/QWEN3_30B__moonep__matched__seq4096__gbs8__capnone__async/rep1 QWEN3_30B moonep matched 16 4096 8 none none+forcebal 1 OK 819.26 774.40 116515 /workspace/tierb-fixed-moonep-full/moonep/tierbprobe/tierbprobe_gbs8/QWEN3_30B__moonep__matched__seq4096__gbs8__capnone__forcebal__async/rep1 DSV2_LITE_L24 moonep matched 8 8192 8 none none 1 OK 673.29 669.50 125157 /workspace/tierb-fixed-moonep-full/moonep/tierbprobe/tierbprobe_gbs8/DSV2_LITE_L24__moonep__matched__seq8192__gbs8__capnone__async/rep1 DSV2_LITE_L24 moonep matched 8 8192 8 none none+forcebal 1 OK 674.59 672.10 125633 /workspace/tierb-fixed-moonep-full/moonep/tierbprobe/tierbprobe_gbs8/DSV2_LITE_L24__moonep__matched__seq8192__gbs8__capnone__forcebal__async/rep1 QWEN2_57B moonep matched 8 4096 8 none none 1 OK 238.47 236.30 95855 /workspace/tierb-fixed-moonep-full/moonep/tierbprobe/tierbprobe_gbs8/QWEN2_57B__moonep__matched__seq4096__gbs8__capnone__async/rep1 QWEN2_57B moonep matched 8 4096 8 none none+forcebal 1 OK 239.34 236.70 95855 /workspace/tierb-fixed-moonep-full/moonep/tierbprobe/tierbprobe_gbs8/QWEN2_57B__moonep__matched__seq4096__gbs8__capnone__forcebal__async/rep1 Results: /workspace/tierb-fixed-moonep-full/moonep/tierbprobe/tierbprobe_gbs8/results.tsv Roll up: python3 /workspace/blackwell_run_scripts/granularity/tools/collect.py /workspace/tierb-fixed-moonep-full/moonep/tierbprobe/tierbprobe_gbs8 ============================================================== 8 probe runs -> /workspace/tierb-fixed-moonep-full/moonep/tierbprobe/tierbprobe_gbs8/probe_summary.tsv run arm cap rawExpI preRankI postRankI drop% reqCap -------------------------------------------------------------------------------------------------------------------- DSV2_LITE_L24__moonep__matched__seq8192__gbs8__capno natural none 7.5443* 2.1316 1.1058 DSV2_LITE_L24__moonep__matched__seq8192__gbs8__capno forced none 1.0286 1.0061 1.0026 QWEN2_57B__moonep__matched__seq4096__gbs8__capnone__ natural none 7.4132* 2.3019 1.0979 QWEN2_57B__moonep__matched__seq4096__gbs8__capnone__ forced none 1.0348 1.0073 1.0033 QWEN3_235B__moonep__matched__seq8192__gbs8__capnone_ natural none 15.9227* 2.5851 1.0712 QWEN3_235B__moonep__matched__seq8192__gbs8__capnone_ forced none 1.0386 1.0053 1.0025 QWEN3_30B__moonep__matched__seq4096__gbs8__capnone__ natural none 15.7078* 2.4672 1.0635 QWEN3_30B__moonep__matched__seq4096__gbs8__capnone__ forced none 1.0565 1.0077 1.0036 * At least one globally aggregated snapshot reached the E/K expert-Imax ceiling. Inspect raw_expert_imax_mean and hottest_expert_token_fraction_mean before generalising. ECHO note: echo_logged_pre_rank_metric_valid=0 and echo_logged_load_reduction_valid=0 are intentional. The branch's built-in fields mix source-rank and destination-rank axes; this tool reconstructs both globally. TIER B PROBE -- what these logs contain, per backend ---------------------------------------------------- Capacity matrix: ECHO: start C=2.00; on OOM descend 1.75, 1.50, 1.25. Stop at the highest fitting zero-drop capacity. If a fitting capacity drops, smaller values cannot help and the model is blocked. UltraEP: measure required_capacity at C=1.25, round it UP to the next 0.01, then validate that capacity once. MoonEP: natural + forced, no capacity factor. The forced arm runs once at the selected/high-headroom capacity. It validates instrumentation and is not part of capacity selection. Each backend instruments a different thing: echo local raw-router histograms and local post-reroute placement. reduce_probe.py aligns and sums all eight source ranks before calculating raw_expert_*, pre_home_rank_*, post_physical_rank_* and valid_max_rank_reduction_*. ultraep global pre/post rank vectors plus explicit drop accounting: drop_rate_pct and required_capacity -- the capacity factor the run WOULD have needed. The vectors are duplicated identically on all logging ranks and are deduplicated, not summed. moonep local raw-router histograms, actual payload rows per physical destination, and the bytes-copy side. The reducer globally sums each (iteration, layer) snapshot first. ECHO reports no comparable byte counters, so MoonEP bytes are a self-comparison across models, never a cross-backend ranking. Do NOT quote ECHO's logged pre_echo imbalance_ratio or max_load_reduction_pct. The branch mixes source-rank and destination-rank axes; the reducer marks both invalid and reconstructs the comparison globally. Two boundary cases: raw_expert_imax_max == E/K -> at least one global snapshot had one expert in every token's top-K set. Inspect the MEAN before generalising. required_capacity or min-safe-capacity at the configured capacity -> clipped by the buffer; it means "at or over", not "fits comfortably". tools/reduce_probe.py has already written probe_summary.tsv and .json here. Campaign results: /workspace/tierb-fixed-moonep-full/moonep/tierbprobe Roll up: python3 /workspace/blackwell_run_scripts/granularity/tools/collect.py /workspace/tierb-fixed-moonep-full/moonep/tierbprobe --compare Already logged in to Hugging Face as: AbhishekGhosh1998 uploaded: /workspace/tierb-fixed-moonep-full/moonep/tierbprobe -> MoE-CUDA-Graph-Analysis/tierb-b200-gbs8/tierb-fixed-moonep-full/moonep/tierbprobe https://huggingface.co/MoE-CUDA-Graph-Analysis/tierb-b200-gbs8/tree/main === ROLL-UP 8 runs ok=8 oom=0 -> /workspace/tierb-fixed-moonep-full/collected_all.tsv phase config seq gbs policy rc cg asy backend iter_ms vs_best ------------------------------------------------------------------------------------- tierbprobeDSV2_LITE_L24 8192 8 matched none 0 1 moonep 673.94 1.00x tierbprobeQWEN2_57B 4096 8 matched none 0 1 moonep 238.91 1.00x tierbprobeQWEN3_235B 8192 8 matched none 0 1 moonep 384.76 1.00x tierbprobeQWEN3_30B 4096 8 matched none 0 1 moonep 824.23 1.00x 8 probe runs -> /workspace/tierb-fixed-moonep-full/probe_summary_all.tsv run arm cap rawExpI preRankI postRankI drop% reqCap -------------------------------------------------------------------------------------------------------------------- moonep/tierbprobe/tierbprobe_gbs8/DSV2_LITE_L24__moo natural none 7.5443* 2.1316 1.1058 moonep/tierbprobe/tierbprobe_gbs8/DSV2_LITE_L24__moo forced none 1.0286 1.0061 1.0026 moonep/tierbprobe/tierbprobe_gbs8/QWEN2_57B__moonep_ natural none 7.4132* 2.3019 1.0979 moonep/tierbprobe/tierbprobe_gbs8/QWEN2_57B__moonep_ forced none 1.0348 1.0073 1.0033 moonep/tierbprobe/tierbprobe_gbs8/QWEN3_235B__moonep natural none 15.9227* 2.5851 1.0712 moonep/tierbprobe/tierbprobe_gbs8/QWEN3_235B__moonep forced none 1.0386 1.0053 1.0025 moonep/tierbprobe/tierbprobe_gbs8/QWEN3_30B__moonep_ natural none 15.7078* 2.4672 1.0635 moonep/tierbprobe/tierbprobe_gbs8/QWEN3_30B__moonep_ forced none 1.0565 1.0077 1.0036 * At least one globally aggregated snapshot reached the E/K expert-Imax ceiling. Inspect raw_expert_imax_mean and hottest_expert_token_fraction_mean before generalising. ECHO note: echo_logged_pre_rank_metric_valid=0 and echo_logged_load_reduction_valid=0 are intentional. The branch's built-in fields mix source-rank and destination-rank axes; this tool reconstructs both globally. no memory-dump runs found under /workspace/tierb-fixed-moonep-full Already logged in to Hugging Face as: AbhishekGhosh1998 uploaded: /workspace/tierb-fixed-moonep-full -> MoE-CUDA-Graph-Analysis/tierb-b200-gbs8/tierb-fixed-moonep-full https://huggingface.co/MoE-CUDA-Graph-Analysis/tierb-b200-gbs8/tree/main === STUDY COMPLETE results : /workspace/tierb-fixed-moonep-full status : /workspace/tierb-fixed-moonep-full/STATUS.txt steps : /workspace/tierb-fixed-moonep-full/steps.tsv hub : https://huggingface.co/datasets/MoE-CUDA-Graph-Analysis/tierb-b200-gbs8 status campaign phase gbs started ended result_dir OK moonep tierbprobe 8 2026-08-29T23:33:50+00:00 2026-08-29T23:44:30+00:00 /workspace/tierb-fixed-moonep-full/moonep/tierbprobe study_root: /workspace/tierb-fixed-moonep-full gbs: 8 stages: 4 nsys: 1 hf_repo: MoE-CUDA-Graph-Analysis/tierb-b200-gbs8 hf_remote_root: tierb-fixed-moonep-full started: 2026-08-29T23:44:38+00:00 driver_repo: /workspace/blackwell_run_scripts @ eec9aee === STAGE 4 -- Tier B timing (probe-selected zero-drop capacities) === STEP moonep__tierb__gbs8 ============================================================== CAMPAIGN: moonep backend=moonep phase=tierb Megatron: /workspace/Megatron-MoE-Echo @ moonep_backend_async_grad_reduce GBS=8 seq=4096 repeats=3 nsys=1 results -> /workspace/tierb-fixed-moonep-full/moonep/tierb ============================================================== -------------------------------------------------------------------------- [23:44:54] CELL QWEN3_235B__moonep__matched__seq8192__gbs8__capnone__async (slots=16) [guard] iter 2/12 (60s) [guard] iter 12/12 (70s) [guard] tearing down process group (outcome=DONE) [guard] iter 1/12 (60s) [guard] iter 12/12 (70s) [guard] tearing down process group (outcome=DONE) [guard] iter 1/12 (60s) [guard] iter 12/12 (70s) [guard] tearing down process group (outcome=DONE) -------------------------------------------------------------------------- [23:48:39] CELL QWEN3_30B__moonep__matched__seq4096__gbs8__capnone__async (slots=16) [guard] iter 12/12 (70s) [guard] tearing down process group (outcome=DONE) [guard] iter 12/12 (70s) [guard] tearing down process group (outcome=DONE) [guard] iter 12/12 (70s) [guard] tearing down process group (outcome=DONE) -------------------------------------------------------------------------- [23:52:30] CELL DSV2_LITE_L24__moonep__matched__seq8192__gbs8__capnone__async (slots=8) [guard] iter 12/12 (70s) [guard] tearing down process group (outcome=DONE) [guard] iter 12/12 (70s) [guard] tearing down process group (outcome=DONE) [guard] iter 12/12 (70s) [guard] tearing down process group (outcome=DONE) -------------------------------------------------------------------------- [23:56:18] CELL QWEN2_57B__moonep__matched__seq4096__gbs8__capnone__async (slots=8) [guard] iter 6/12 (60s) [guard] iter 12/12 (70s) [guard] tearing down process group (outcome=DONE) [guard] iter 7/12 (60s) [guard] iter 12/12 (70s) [guard] tearing down process group (outcome=DONE) [guard] iter 10/12 (60s) [guard] iter 12/12 (70s) [guard] tearing down process group (outcome=DONE) -------------------------------------------------------------------------- ============================================================== SWEEP COMPLETE ok=12 failed=0 ============================================================== config backend policy slots seq gbs cap recompute rep status iter_ms_mean iter_ms_min peak_mib run_dir QWEN3_235B moonep matched 16 8192 8 none none 1 OK 321.60 320.40 120303 /workspace/tierb-fixed-moonep-full/moonep/tierb/tierb_gbs8/QWEN3_235B__moonep__matched__seq8192__gbs8__capnone__async/rep1 QWEN3_235B moonep matched 16 8192 8 none none 2 OK 322.05 320.40 120303 /workspace/tierb-fixed-moonep-full/moonep/tierb/tierb_gbs8/QWEN3_235B__moonep__matched__seq8192__gbs8__capnone__async/rep2 QWEN3_235B moonep matched 16 8192 8 none none 3 OK 322.10 320.40 120303 /workspace/tierb-fixed-moonep-full/moonep/tierb/tierb_gbs8/QWEN3_235B__moonep__matched__seq8192__gbs8__capnone__async/rep3 QWEN3_30B moonep matched 16 4096 8 none none 1 OK 361.53 358.70 116703 /workspace/tierb-fixed-moonep-full/moonep/tierb/tierb_gbs8/QWEN3_30B__moonep__matched__seq4096__gbs8__capnone__async/rep1 QWEN3_30B moonep matched 16 4096 8 none none 2 OK 361.17 358.70 116703 /workspace/tierb-fixed-moonep-full/moonep/tierb/tierb_gbs8/QWEN3_30B__moonep__matched__seq4096__gbs8__capnone__async/rep2 QWEN3_30B moonep matched 16 4096 8 none none 3 OK 360.76 357.60 116703 /workspace/tierb-fixed-moonep-full/moonep/tierb/tierb_gbs8/QWEN3_30B__moonep__matched__seq4096__gbs8__capnone__async/rep3 DSV2_LITE_L24 moonep matched 8 8192 8 none none 1 OK 596.03 595.70 133311 /workspace/tierb-fixed-moonep-full/moonep/tierb/tierb_gbs8/DSV2_LITE_L24__moonep__matched__seq8192__gbs8__capnone__async/rep1 DSV2_LITE_L24 moonep matched 8 8192 8 none none 2 OK 596.18 594.10 133311 /workspace/tierb-fixed-moonep-full/moonep/tierb/tierb_gbs8/DSV2_LITE_L24__moonep__matched__seq8192__gbs8__capnone__async/rep2 DSV2_LITE_L24 moonep matched 8 8192 8 none none 3 OK 595.97 594.00 133311 /workspace/tierb-fixed-moonep-full/moonep/tierb/tierb_gbs8/DSV2_LITE_L24__moonep__matched__seq8192__gbs8__capnone__async/rep3 QWEN2_57B moonep matched 8 4096 8 none none 1 OK 183.70 182.60 96737 /workspace/tierb-fixed-moonep-full/moonep/tierb/tierb_gbs8/QWEN2_57B__moonep__matched__seq4096__gbs8__capnone__async/rep1 QWEN2_57B moonep matched 8 4096 8 none none 2 OK 184.22 182.60 96737 /workspace/tierb-fixed-moonep-full/moonep/tierb/tierb_gbs8/QWEN2_57B__moonep__matched__seq4096__gbs8__capnone__async/rep2 QWEN2_57B moonep matched 8 4096 8 none none 3 OK 184.20 182.60 96737 /workspace/tierb-fixed-moonep-full/moonep/tierb/tierb_gbs8/QWEN2_57B__moonep__matched__seq4096__gbs8__capnone__async/rep3 Results: /workspace/tierb-fixed-moonep-full/moonep/tierb/tierb_gbs8/results.tsv Roll up: python3 /workspace/blackwell_run_scripts/granularity/tools/collect.py /workspace/tierb-fixed-moonep-full/moonep/tierb/tierb_gbs8 ============================================================== Campaign results: /workspace/tierb-fixed-moonep-full/moonep/tierb Roll up: python3 /workspace/blackwell_run_scripts/granularity/tools/collect.py /workspace/tierb-fixed-moonep-full/moonep/tierb --compare Already logged in to Hugging Face as: AbhishekGhosh1998 uploaded: /workspace/tierb-fixed-moonep-full/moonep/tierb -> MoE-CUDA-Graph-Analysis/tierb-b200-gbs8/tierb-fixed-moonep-full/moonep/tierb https://huggingface.co/MoE-CUDA-Graph-Analysis/tierb-b200-gbs8/tree/main === ROLL-UP 20 runs ok=20 oom=0 -> /workspace/tierb-fixed-moonep-full/collected_all.tsv phase config seq gbs policy rc cg asy backend iter_ms vs_best ------------------------------------------------------------------------------------- tierb DSV2_LITE_L24 8192 8 matched none 1 1 moonep 595.97 1.00x tierb QWEN2_57B 4096 8 matched none 1 1 moonep 184.20 1.00x tierb QWEN3_235B 8192 8 matched none 1 1 moonep 322.10 1.00x tierb QWEN3_30B 4096 8 matched none 1 1 moonep 360.76 1.00x tierbprobeDSV2_LITE_L24 8192 8 matched none 0 1 moonep 673.94 1.00x tierbprobeQWEN2_57B 4096 8 matched none 0 1 moonep 238.91 1.00x tierbprobeQWEN3_235B 8192 8 matched none 0 1 moonep 384.76 1.00x tierbprobeQWEN3_30B 4096 8 matched none 0 1 moonep 824.23 1.00x 8 probe runs -> /workspace/tierb-fixed-moonep-full/probe_summary_all.tsv run arm cap rawExpI preRankI postRankI drop% reqCap -------------------------------------------------------------------------------------------------------------------- moonep/tierbprobe/tierbprobe_gbs8/DSV2_LITE_L24__moo natural none 7.5443* 2.1316 1.1058 moonep/tierbprobe/tierbprobe_gbs8/DSV2_LITE_L24__moo forced none 1.0286 1.0061 1.0026 moonep/tierbprobe/tierbprobe_gbs8/QWEN2_57B__moonep_ natural none 7.4132* 2.3019 1.0979 moonep/tierbprobe/tierbprobe_gbs8/QWEN2_57B__moonep_ forced none 1.0348 1.0073 1.0033 moonep/tierbprobe/tierbprobe_gbs8/QWEN3_235B__moonep natural none 15.9227* 2.5851 1.0712 moonep/tierbprobe/tierbprobe_gbs8/QWEN3_235B__moonep forced none 1.0386 1.0053 1.0025 moonep/tierbprobe/tierbprobe_gbs8/QWEN3_30B__moonep_ natural none 15.7078* 2.4672 1.0635 moonep/tierbprobe/tierbprobe_gbs8/QWEN3_30B__moonep_ forced none 1.0565 1.0077 1.0036 * At least one globally aggregated snapshot reached the E/K expert-Imax ceiling. Inspect raw_expert_imax_mean and hottest_expert_token_fraction_mean before generalising. ECHO note: echo_logged_pre_rank_metric_valid=0 and echo_logged_load_reduction_valid=0 are intentional. The branch's built-in fields mix source-rank and destination-rank axes; this tool reconstructs both globally. no memory-dump runs found under /workspace/tierb-fixed-moonep-full Already logged in to Hugging Face as: AbhishekGhosh1998 uploaded: /workspace/tierb-fixed-moonep-full -> MoE-CUDA-Graph-Analysis/tierb-b200-gbs8/tierb-fixed-moonep-full https://huggingface.co/MoE-CUDA-Graph-Analysis/tierb-b200-gbs8/tree/main === STUDY COMPLETE results : /workspace/tierb-fixed-moonep-full status : /workspace/tierb-fixed-moonep-full/STATUS.txt steps : /workspace/tierb-fixed-moonep-full/steps.tsv hub : https://huggingface.co/datasets/MoE-CUDA-Graph-Analysis/tierb-b200-gbs8 status campaign phase gbs started ended result_dir OK moonep tierbprobe 8 2026-08-29T23:33:50+00:00 2026-08-29T23:44:30+00:00 /workspace/tierb-fixed-moonep-full/moonep/tierbprobe OK moonep tierb 8 2026-08-29T23:44:39+00:00 2026-08-29T23:59:52+00:00 /workspace/tierb-fixed-moonep-full/moonep/tierb study_root: /workspace/tierb-fixed-moonep-full gbs: 8 stages: 8 nsys: 1 hf_repo: MoE-CUDA-Graph-Analysis/tierb-b200-gbs8 hf_remote_root: tierb-fixed-moonep-full started: 2026-08-29T23:59:59+00:00 driver_repo: /workspace/blackwell_run_scripts @ eec9aee === STAGE 8 -- Tier B allocator dumps (static CG capacities + dynamic no-CG) === STEP moonep__tierbmem__gbs8 ============================================================== CAMPAIGN: moonep backend=moonep phase=tierbmem Megatron: /workspace/Megatron-MoE-Echo @ moonep_backend_async_grad_reduce_better_memory_dump_support GBS=8 seq=4096 repeats=3 nsys=1 results -> /workspace/tierb-fixed-moonep-full/moonep/tierbmem ============================================================== -------------------------------------------------------------------------- [00:00:14] CELL QWEN3_235B__moonep__matched__seq8192__gbs8__capnone__async__memiteration__it6 (slots=16) [guard] iter 1/6 (60s) [guard] iter 6/6 (70s) [guard] tearing down process group (outcome=DONE) -------------------------------------------------------------------------- [00:01:27] CELL QWEN3_235B__moonep__matched__seq8192__gbs8__capnone__nocg__async__memiteration__it5 (slots=16) [guard] iter 1/5 (60s) [guard] iter 5/5 (70s) [guard] tearing down process group (outcome=DONE) -------------------------------------------------------------------------- [00:02:41] CELL QWEN3_30B__moonep__matched__seq4096__gbs8__capnone__async__memiteration__it6 (slots=16) [guard] iter 6/6 (70s) [guard] tearing down process group (outcome=DONE) -------------------------------------------------------------------------- [00:03:57] CELL QWEN3_30B__moonep__matched__seq4096__gbs8__capnone__nocg__async__memiteration__it5 (slots=16) [guard] iter 5/5 (70s) [guard] tearing down process group (outcome=DONE) -------------------------------------------------------------------------- [00:05:12] CELL DSV2_LITE_L24__moonep__matched__seq8192__gbs8__capnone__async__memiteration__it6 (slots=8) [guard] iter 6/6 (70s) [guard] tearing down process group (outcome=DONE) -------------------------------------------------------------------------- [00:06:27] CELL DSV2_LITE_L24__moonep__matched__seq8192__gbs8__capnone__nocg__async__memiteration__it5 (slots=8) [guard] iter 5/5 (70s) [guard] tearing down process group (outcome=DONE) -------------------------------------------------------------------------- [00:07:42] CELL QWEN2_57B__moonep__matched__seq4096__gbs8__capnone__async__memiteration__it6 (slots=8) [guard] iter 5/6 (60s) [guard] iter 6/6 (70s) [guard] tearing down process group (outcome=DONE) -------------------------------------------------------------------------- [00:08:53] CELL QWEN2_57B__moonep__matched__seq4096__gbs8__capnone__nocg__async__memiteration__it5 (slots=8) [guard] iter 5/5 (60s) [guard] tearing down process group (outcome=DONE) -------------------------------------------------------------------------- ============================================================== SWEEP COMPLETE ok=8 failed=0 ============================================================== config backend policy slots seq gbs cap recompute rep status iter_ms_mean iter_ms_min peak_mib run_dir QWEN3_235B moonep matched 16 8192 8 none none 1 OK 341.70 341.70 120303 /workspace/tierb-fixed-moonep-full/moonep/tierbmem/tierbmem_gbs8/QWEN3_235B__moonep__matched__seq8192__gbs8__capnone__async__memiteration__it6/rep1 QWEN3_235B moonep matched 16 8192 8 none none 1 OK NA NA 120131 /workspace/tierb-fixed-moonep-full/moonep/tierbmem/tierbmem_gbs8/QWEN3_235B__moonep__matched__seq8192__gbs8__capnone__nocg__async__memiteration__it5/rep1 QWEN3_30B moonep matched 16 4096 8 none none 1 OK 435.30 435.30 116703 /workspace/tierb-fixed-moonep-full/moonep/tierbmem/tierbmem_gbs8/QWEN3_30B__moonep__matched__seq4096__gbs8__capnone__async__memiteration__it6/rep1 QWEN3_30B moonep matched 16 4096 8 none none 1 OK NA NA 116445 /workspace/tierb-fixed-moonep-full/moonep/tierbmem/tierbmem_gbs8/QWEN3_30B__moonep__matched__seq4096__gbs8__capnone__nocg__async__memiteration__it5/rep1 DSV2_LITE_L24 moonep matched 8 8192 8 none none 1 OK 622.80 622.80 133311 /workspace/tierb-fixed-moonep-full/moonep/tierbmem/tierbmem_gbs8/DSV2_LITE_L24__moonep__matched__seq8192__gbs8__capnone__async__memiteration__it6/rep1 DSV2_LITE_L24 moonep matched 8 8192 8 none none 1 OK NA NA 125157 /workspace/tierb-fixed-moonep-full/moonep/tierbmem/tierbmem_gbs8/DSV2_LITE_L24__moonep__matched__seq8192__gbs8__capnone__nocg__async__memiteration__it5/rep1 QWEN2_57B moonep matched 8 4096 8 none none 1 OK 196.60 196.60 96737 /workspace/tierb-fixed-moonep-full/moonep/tierbmem/tierbmem_gbs8/QWEN2_57B__moonep__matched__seq4096__gbs8__capnone__async__memiteration__it6/rep1 QWEN2_57B moonep matched 8 4096 8 none none 1 OK NA NA 95855 /workspace/tierb-fixed-moonep-full/moonep/tierbmem/tierbmem_gbs8/QWEN2_57B__moonep__matched__seq4096__gbs8__capnone__nocg__async__memiteration__it5/rep1 Results: /workspace/tierb-fixed-moonep-full/moonep/tierbmem/tierbmem_gbs8/results.tsv Roll up: python3 /workspace/blackwell_run_scripts/granularity/tools/collect.py /workspace/tierb-fixed-moonep-full/moonep/tierbmem/tierbmem_gbs8 ============================================================== 8 memory-dump runs -> /workspace/tierb-fixed-moonep-full/moonep/tierbmem/tierbmem_gbs8/memory_summary.tsv model backend cg allocation capacity status peak GiB dumps ---------------------------------------------------------------------------------- DSV2_LITE_L24 moonep 1 none none OK 130.2 48/48 DSV2_LITE_L24 moonep 0 none none OK 122.2 40/40 QWEN2_57B moonep 1 none none OK 94.5 48/48 QWEN2_57B moonep 0 none none OK 93.6 40/40 QWEN3_235B moonep 1 none none OK 117.5 48/48 QWEN3_235B moonep 0 none none OK 117.3 40/40 QWEN3_30B moonep 1 none none OK 114.0 48/48 QWEN3_30B moonep 0 none none OK 113.7 40/40 ECHO/UltraEP note: graph-on uses static capacity + device-initiated GEMM while graph-off uses dynamic allocation + host-derived sizes. Their difference is not labelled as pure graph cost. TIER B MEMORY DUMPS -- how to read this --------------------------------------- ECHO / UltraEP: cap1p00__memiteration__it6 graph on, static C=1.00 cap1p125__memiteration__it6 graph on, static C=1.125 cap1p25__memiteration__it6 graph on, static C=1.25 capdynamic__nocg__memiteration__it5 graph off, no capacity flag, dynamic allocation MoonEP: capnone__memiteration__it6 graph on, no capacity factor capnone__nocg__memiteration__it5 graph off, no capacity factor Every successful cell contains: train.log Megatron and OOM diagnostics mem_samples.csv driver-level nvidia-smi samples every 2 seconds run.env exact branch/config/capacity provenance memory_snapshots/*.pickle 48 captured / 40 eager allocator histories memory_snapshots/*.json matching summaries and before/after deltas The graph-on/static vs graph-off/dynamic difference is intentionally NOT called "graph cost": capacity allocation policy and device-initiated grouped GEMM also change. Compare the three graph-on rows to measure static-capacity sensitivity, and use the dynamic row as the natural eager-allocation reference. Campaign results: /workspace/tierb-fixed-moonep-full/moonep/tierbmem Roll up: python3 /workspace/blackwell_run_scripts/granularity/tools/collect.py /workspace/tierb-fixed-moonep-full/moonep/tierbmem --compare Already logged in to Hugging Face as: AbhishekGhosh1998 uploaded: /workspace/tierb-fixed-moonep-full/moonep/tierbmem -> MoE-CUDA-Graph-Analysis/tierb-b200-gbs8/tierb-fixed-moonep-full/moonep/tierbmem https://huggingface.co/MoE-CUDA-Graph-Analysis/tierb-b200-gbs8/tree/main === ROLL-UP 28 runs ok=24 oom=0 -> /workspace/tierb-fixed-moonep-full/collected_all.tsv phase config seq gbs policy rc cg asy backend iter_ms vs_best ------------------------------------------------------------------------------------- tierb DSV2_LITE_L24 8192 8 matched none 1 1 moonep 595.97 1.00x tierb QWEN2_57B 4096 8 matched none 1 1 moonep 184.20 1.00x tierb QWEN3_235B 8192 8 matched none 1 1 moonep 322.10 1.00x tierb QWEN3_30B 4096 8 matched none 1 1 moonep 360.76 1.00x tierbmem DSV2_LITE_L24 8192 8 matched none 1 1 moonep 622.80 1.00x tierbmem QWEN2_57B 4096 8 matched none 1 1 moonep 196.60 1.00x tierbmem QWEN3_235B 8192 8 matched none 1 1 moonep 341.70 1.00x tierbmem QWEN3_30B 4096 8 matched none 1 1 moonep 435.30 1.00x tierbprobeDSV2_LITE_L24 8192 8 matched none 0 1 moonep 673.94 1.00x tierbprobeQWEN2_57B 4096 8 matched none 0 1 moonep 238.91 1.00x tierbprobeQWEN3_235B 8192 8 matched none 0 1 moonep 384.76 1.00x tierbprobeQWEN3_30B 4096 8 matched none 0 1 moonep 824.23 1.00x 8 probe runs -> /workspace/tierb-fixed-moonep-full/probe_summary_all.tsv run arm cap rawExpI preRankI postRankI drop% reqCap -------------------------------------------------------------------------------------------------------------------- moonep/tierbprobe/tierbprobe_gbs8/DSV2_LITE_L24__moo natural none 7.5443* 2.1316 1.1058 moonep/tierbprobe/tierbprobe_gbs8/DSV2_LITE_L24__moo forced none 1.0286 1.0061 1.0026 moonep/tierbprobe/tierbprobe_gbs8/QWEN2_57B__moonep_ natural none 7.4132* 2.3019 1.0979 moonep/tierbprobe/tierbprobe_gbs8/QWEN2_57B__moonep_ forced none 1.0348 1.0073 1.0033 moonep/tierbprobe/tierbprobe_gbs8/QWEN3_235B__moonep natural none 15.9227* 2.5851 1.0712 moonep/tierbprobe/tierbprobe_gbs8/QWEN3_235B__moonep forced none 1.0386 1.0053 1.0025 moonep/tierbprobe/tierbprobe_gbs8/QWEN3_30B__moonep_ natural none 15.7078* 2.4672 1.0635 moonep/tierbprobe/tierbprobe_gbs8/QWEN3_30B__moonep_ forced none 1.0565 1.0077 1.0036 * At least one globally aggregated snapshot reached the E/K expert-Imax ceiling. Inspect raw_expert_imax_mean and hottest_expert_token_fraction_mean before generalising. ECHO note: echo_logged_pre_rank_metric_valid=0 and echo_logged_load_reduction_valid=0 are intentional. The branch's built-in fields mix source-rank and destination-rank axes; this tool reconstructs both globally. 8 memory-dump runs -> /workspace/tierb-fixed-moonep-full/memory_summary_all.tsv model backend cg allocation capacity status peak GiB dumps ---------------------------------------------------------------------------------- DSV2_LITE_L24 moonep 1 none none OK 130.2 48/48 DSV2_LITE_L24 moonep 0 none none OK 122.2 40/40 QWEN2_57B moonep 1 none none OK 94.5 48/48 QWEN2_57B moonep 0 none none OK 93.6 40/40 QWEN3_235B moonep 1 none none OK 117.5 48/48 QWEN3_235B moonep 0 none none OK 117.3 40/40 QWEN3_30B moonep 1 none none OK 114.0 48/48 QWEN3_30B moonep 0 none none OK 113.7 40/40 ECHO/UltraEP note: graph-on uses static capacity + device-initiated GEMM while graph-off uses dynamic allocation + host-derived sizes. Their difference is not labelled as pure graph cost. Already logged in to Hugging Face as: AbhishekGhosh1998 uploaded: /workspace/tierb-fixed-moonep-full -> MoE-CUDA-Graph-Analysis/tierb-b200-gbs8/tierb-fixed-moonep-full https://huggingface.co/MoE-CUDA-Graph-Analysis/tierb-b200-gbs8/tree/main === STUDY COMPLETE results : /workspace/tierb-fixed-moonep-full status : /workspace/tierb-fixed-moonep-full/STATUS.txt steps : /workspace/tierb-fixed-moonep-full/steps.tsv hub : https://huggingface.co/datasets/MoE-CUDA-Graph-Analysis/tierb-b200-gbs8 status campaign phase gbs started ended result_dir OK moonep tierbprobe 8 2026-08-29T23:33:50+00:00 2026-08-29T23:44:30+00:00 /workspace/tierb-fixed-moonep-full/moonep/tierbprobe OK moonep tierb 8 2026-08-29T23:44:39+00:00 2026-08-29T23:59:52+00:00 /workspace/tierb-fixed-moonep-full/moonep/tierb OK moonep tierbmem 8 2026-08-29T23:59:59+00:00 2026-08-30T00:09:58+00:00 /workspace/tierb-fixed-moonep-full/moonep/tierbmem study_root: /workspace/tierb-fixed-moonep-full gbs: 8 stages: 9 nsys: 1 hf_repo: MoE-CUDA-Graph-Analysis/tierb-b200-gbs8 hf_remote_root: tierb-fixed-moonep-full started: 2026-08-30T00:10:16+00:00 driver_repo: /workspace/blackwell_run_scripts @ eec9aee === STAGE 9 -- Tier B nsys traces (final, trace-only) === STEP echo__tierbtrace__gbs8 ============================================================== CAMPAIGN: echo backend=echo phase=tierbtrace Megatron: /workspace/Megatron-MoE-Echo @ bracing-regions-through-kernels GBS=8 seq=4096 repeats=3 nsys=1 results -> /workspace/tierb-fixed-moonep-full/echo/tierbtrace ============================================================== [00:10:31] building echo marker extension (log: /workspace/tierb-fixed-moonep-full/echo/tierbtrace/tierbtrace_gbs8/trace_extension_build.log) [00:14:37] marker extension ready: _echo_trace_C.cpython-312-x86_64-linux-gnu.so [00:14:37] TRACE-ONLY QWEN3_235B__echo__matched__seq8192__gbs8__cap1p083984375 (ranks: 0 1 2 3 4 5 6 7) [guard] iter 6/10 (80s) [guard] iter 8/10 (90s) [guard] iter 10/10 (130s) [guard] nsys stop (flushing report) [00:17:19] trace verified -> QWEN3_235B__echo__matched__seq8192__gbs8__cap1p083984375.nsys-rep (9.7M, all 8 GPUs + echo markers) [00:17:19] TRACE-ONLY QWEN3_30B__echo__matched__seq4096__gbs8__cap1p11328125 (ranks: 0 1 2 3 4 5 6 7) [guard] iter 3/10 (80s) [guard] iter 8/10 (90s) [guard] no progress for 420s at iter 8/10; treating as done [guard] nsys stop (flushing report) [guard] tearing down process group (outcome=STALLED) [00:26:39] trace verified -> QWEN3_30B__echo__matched__seq4096__gbs8__cap1p11328125.nsys-rep (33M, all 8 GPUs + echo markers) [00:26:39] TRACE-ONLY DSV2_LITE_L24__echo__matched__seq8192__gbs8__cap1p0755208333333333 (ranks: 0 1 2 3 4 5 6 7) [guard] iter 5/10 (80s) [guard] iter 8/10 (90s) [guard] no progress for 420s at iter 8/10; treating as done [guard] nsys stop (flushing report) [guard] tearing down process group (outcome=STALLED) [00:35:58] trace verified -> DSV2_LITE_L24__echo__matched__seq8192__gbs8__cap1p0755208333333333.nsys-rep (18M, all 8 GPUs + echo markers) [00:35:58] TRACE-ONLY QWEN2_57B__echo__matched__seq4096__gbs8__cap1p08984375 (ranks: 0 1 2 3 4 5 6 7) [guard] iter 8/10 (80s) [guard] iter 10/10 (120s) [guard] nsys stop (flushing report) [00:38:29] trace verified -> QWEN2_57B__echo__matched__seq4096__gbs8__cap1p08984375.nsys-rep (7.4M, all 8 GPUs + echo markers) -------------------------------------------------------------------------- ============================================================== SWEEP COMPLETE ok=4 failed=0 ============================================================== config backend policy slots seq gbs cap recompute rep status iter_ms_mean iter_ms_min peak_mib run_dir Results: /workspace/tierb-fixed-moonep-full/echo/tierbtrace/tierbtrace_gbs8/results.tsv Roll up: python3 /workspace/blackwell_run_scripts/granularity/tools/collect.py /workspace/tierb-fixed-moonep-full/echo/tierbtrace/tierbtrace_gbs8 ============================================================== Campaign results: /workspace/tierb-fixed-moonep-full/echo/tierbtrace Roll up: python3 /workspace/blackwell_run_scripts/granularity/tools/collect.py /workspace/tierb-fixed-moonep-full/echo/tierbtrace --compare Already logged in to Hugging Face as: AbhishekGhosh1998 uploaded: /workspace/tierb-fixed-moonep-full/echo/tierbtrace -> MoE-CUDA-Graph-Analysis/tierb-b200-gbs8/tierb-fixed-moonep-full/echo/tierbtrace https://huggingface.co/MoE-CUDA-Graph-Analysis/tierb-b200-gbs8/tree/main === STEP ultraep__tierbtrace__gbs8 ============================================================== CAMPAIGN: ultraep backend=ultraep phase=tierbtrace Megatron: /workspace/Megatron-UltraEP @ ultraep-bracing-regions-through-kernels GBS=8 seq=4096 repeats=3 nsys=1 results -> /workspace/tierb-fixed-moonep-full/ultraep/tierbtrace ============================================================== [00:38:48] building ultraep marker extension (log: /workspace/tierb-fixed-moonep-full/ultraep/tierbtrace/tierbtrace_gbs8/trace_extension_build.log) [00:42:50] marker extension ready: _ultraep_trace_C.cpython-312-x86_64-linux-gnu.so [00:42:50] TRACE-ONLY QWEN3_235B__ultraep__matched__seq8192__gbs8__cap1p083984375 (ranks: 0 1 2 3 4 5 6 7) [guard] iter 6/10 (70s) [guard] iter 8/10 (80s) [guard] iter 10/10 (120s) [guard] nsys stop (flushing report) [00:45:24] trace verified -> QWEN3_235B__ultraep__matched__seq8192__gbs8__cap1p083984375.nsys-rep (7.6M, all 8 GPUs + ultraep markers) [00:45:24] TRACE-ONLY QWEN3_30B__ultraep__matched__seq4096__gbs8__cap1p11328125 (ranks: 0 1 2 3 4 5 6 7) [guard] iter 6/10 (70s) [guard] iter 8/10 (80s) [guard] no progress for 420s at iter 8/10; treating as done [guard] nsys stop (flushing report) [guard] tearing down process group (outcome=STALLED) [00:54:34] trace verified -> QWEN3_30B__ultraep__matched__seq4096__gbs8__cap1p11328125.nsys-rep (25M, all 8 GPUs + ultraep markers) [00:54:34] TRACE-ONLY DSV2_LITE_L24__ultraep__matched__seq8192__gbs8__cap1p0755208333333333 (ranks: 0 1 2 3 4 5 6 7) [guard] iter 6/10 (70s) [guard] iter 8/10 (80s) [guard] iter 10/10 (120s) [guard] nsys stop (flushing report) [00:57:06] trace verified -> DSV2_LITE_L24__ultraep__matched__seq8192__gbs8__cap1p0755208333333333.nsys-rep (14M, all 8 GPUs + ultraep markers) [00:57:06] TRACE-ONLY QWEN2_57B__ultraep__matched__seq4096__gbs8__cap1p08984375 (ranks: 0 1 2 3 4 5 6 7) [guard] iter 8/10 (70s) [guard] iter 10/10 (110s) [guard] nsys stop (flushing report) [00:59:28] trace verified -> QWEN2_57B__ultraep__matched__seq4096__gbs8__cap1p08984375.nsys-rep (5.8M, all 8 GPUs + ultraep markers) -------------------------------------------------------------------------- ============================================================== SWEEP COMPLETE ok=4 failed=0 ============================================================== config backend policy slots seq gbs cap recompute rep status iter_ms_mean iter_ms_min peak_mib run_dir Results: /workspace/tierb-fixed-moonep-full/ultraep/tierbtrace/tierbtrace_gbs8/results.tsv Roll up: python3 /workspace/blackwell_run_scripts/granularity/tools/collect.py /workspace/tierb-fixed-moonep-full/ultraep/tierbtrace/tierbtrace_gbs8 ============================================================== Campaign results: /workspace/tierb-fixed-moonep-full/ultraep/tierbtrace Roll up: python3 /workspace/blackwell_run_scripts/granularity/tools/collect.py /workspace/tierb-fixed-moonep-full/ultraep/tierbtrace --compare Already logged in to Hugging Face as: AbhishekGhosh1998 uploaded: /workspace/tierb-fixed-moonep-full/ultraep/tierbtrace -> MoE-CUDA-Graph-Analysis/tierb-b200-gbs8/tierb-fixed-moonep-full/ultraep/tierbtrace https://huggingface.co/MoE-CUDA-Graph-Analysis/tierb-b200-gbs8/tree/main === ROLL-UP 36 runs ok=32 oom=0 -> /workspace/tierb-fixed-moonep-full/collected_all.tsv (excluded 8 nsys-traced run(s) from the timing comparison) phase config seq gbs policy rc cg asy backend iter_ms vs_best ------------------------------------------------------------------------------------- tierb DSV2_LITE_L24 8192 8 matched none 1 1 moonep 595.97 1.00x tierb QWEN2_57B 4096 8 matched none 1 1 moonep 184.20 1.00x tierb QWEN3_235B 8192 8 matched none 1 1 moonep 322.10 1.00x tierb QWEN3_30B 4096 8 matched none 1 1 moonep 360.76 1.00x tierbmem DSV2_LITE_L24 8192 8 matched none 1 1 moonep 622.80 1.00x tierbmem QWEN2_57B 4096 8 matched none 1 1 moonep 196.60 1.00x tierbmem QWEN3_235B 8192 8 matched none 1 1 moonep 341.70 1.00x tierbmem QWEN3_30B 4096 8 matched none 1 1 moonep 435.30 1.00x tierbprobeDSV2_LITE_L24 8192 8 matched none 0 1 moonep 673.94 1.00x tierbprobeQWEN2_57B 4096 8 matched none 0 1 moonep 238.91 1.00x tierbprobeQWEN3_235B 8192 8 matched none 0 1 moonep 384.76 1.00x tierbprobeQWEN3_30B 4096 8 matched none 0 1 moonep 824.23 1.00x 8 probe runs -> /workspace/tierb-fixed-moonep-full/probe_summary_all.tsv run arm cap rawExpI preRankI postRankI drop% reqCap -------------------------------------------------------------------------------------------------------------------- moonep/tierbprobe/tierbprobe_gbs8/DSV2_LITE_L24__moo natural none 7.5443* 2.1316 1.1058 moonep/tierbprobe/tierbprobe_gbs8/DSV2_LITE_L24__moo forced none 1.0286 1.0061 1.0026 moonep/tierbprobe/tierbprobe_gbs8/QWEN2_57B__moonep_ natural none 7.4132* 2.3019 1.0979 moonep/tierbprobe/tierbprobe_gbs8/QWEN2_57B__moonep_ forced none 1.0348 1.0073 1.0033 moonep/tierbprobe/tierbprobe_gbs8/QWEN3_235B__moonep natural none 15.9227* 2.5851 1.0712 moonep/tierbprobe/tierbprobe_gbs8/QWEN3_235B__moonep forced none 1.0386 1.0053 1.0025 moonep/tierbprobe/tierbprobe_gbs8/QWEN3_30B__moonep_ natural none 15.7078* 2.4672 1.0635 moonep/tierbprobe/tierbprobe_gbs8/QWEN3_30B__moonep_ forced none 1.0565 1.0077 1.0036 * At least one globally aggregated snapshot reached the E/K expert-Imax ceiling. Inspect raw_expert_imax_mean and hottest_expert_token_fraction_mean before generalising. ECHO note: echo_logged_pre_rank_metric_valid=0 and echo_logged_load_reduction_valid=0 are intentional. The branch's built-in fields mix source-rank and destination-rank axes; this tool reconstructs both globally. 8 memory-dump runs -> /workspace/tierb-fixed-moonep-full/memory_summary_all.tsv model backend cg allocation capacity status peak GiB dumps ---------------------------------------------------------------------------------- DSV2_LITE_L24 moonep 1 none none OK 130.2 48/48 DSV2_LITE_L24 moonep 0 none none OK 122.2 40/40 QWEN2_57B moonep 1 none none OK 94.5 48/48 QWEN2_57B moonep 0 none none OK 93.6 40/40 QWEN3_235B moonep 1 none none OK 117.5 48/48 QWEN3_235B moonep 0 none none OK 117.3 40/40 QWEN3_30B moonep 1 none none OK 114.0 48/48 QWEN3_30B moonep 0 none none OK 113.7 40/40 ECHO/UltraEP note: graph-on uses static capacity + device-initiated GEMM while graph-off uses dynamic allocation + host-derived sizes. Their difference is not labelled as pure graph cost. Already logged in to Hugging Face as: AbhishekGhosh1998 uploaded: /workspace/tierb-fixed-moonep-full -> MoE-CUDA-Graph-Analysis/tierb-b200-gbs8/tierb-fixed-moonep-full https://huggingface.co/MoE-CUDA-Graph-Analysis/tierb-b200-gbs8/tree/main === STUDY COMPLETE results : /workspace/tierb-fixed-moonep-full status : /workspace/tierb-fixed-moonep-full/STATUS.txt steps : /workspace/tierb-fixed-moonep-full/steps.tsv hub : https://huggingface.co/datasets/MoE-CUDA-Graph-Analysis/tierb-b200-gbs8 status campaign phase gbs started ended result_dir OK moonep tierbprobe 8 2026-08-29T23:33:50+00:00 2026-08-29T23:44:30+00:00 /workspace/tierb-fixed-moonep-full/moonep/tierbprobe OK moonep tierb 8 2026-08-29T23:44:39+00:00 2026-08-29T23:59:52+00:00 /workspace/tierb-fixed-moonep-full/moonep/tierb OK moonep tierbmem 8 2026-08-29T23:59:59+00:00 2026-08-30T00:09:58+00:00 /workspace/tierb-fixed-moonep-full/moonep/tierbmem OK echo tierbtrace 8 2026-08-30T00:10:16+00:00 2026-08-30T00:38:30+00:00 /workspace/tierb-fixed-moonep-full/echo/tierbtrace OK ultraep tierbtrace 8 2026-08-30T00:38:33+00:00 2026-08-30T00:59:28+00:00 /workspace/tierb-fixed-moonep-full/ultraep/tierbtrace study_root: /workspace/tierb-fixed-moonep-full gbs: 8 stages: 9 nsys: 1 hf_repo: MoE-CUDA-Graph-Analysis/tierb-b200-gbs8 hf_remote_root: tierb-fixed-moonep-full started: 2026-08-30T00:59:40+00:00 driver_repo: /workspace/blackwell_run_scripts @ 110b5d8 === STAGE 9 -- Tier B nsys traces (final, trace-only) === STEP moonep__tierbtrace__gbs8 ============================================================== CAMPAIGN: moonep backend=moonep phase=tierbtrace Megatron: /workspace/Megatron-MoE-Echo @ moonep_backend_async_grad_reduce GBS=8 seq=4096 repeats=3 nsys=1 results -> /workspace/tierb-fixed-moonep-full/moonep/tierbtrace ============================================================== [00:59:55] building moonep marker extension (log: /workspace/tierb-fixed-moonep-full/moonep/tierbtrace/tierbtrace_gbs8/trace_extension_build.log) [01:03:50] marker extension ready: _moonep_trace_C.cpython-312-x86_64-linux-gnu.so [01:03:50] TRACE-ONLY QWEN3_235B__moonep__matched__seq8192__gbs8__capnone__async (ranks: 0 1 2 3 4 5 6 7) [guard] iter 6/10 (60s) [guard] iter 8/10 (70s) [guard] iter 10/10 (110s) [guard] nsys stop (flushing report) [01:06:12] trace verified -> QWEN3_235B__moonep__matched__seq8192__gbs8__capnone__async.nsys-rep (7.3M, all 8 GPUs + moonep markers) MoonEP async overlap 0.000% is below 0.100% [01:06:12] ERROR: QWEN3_235B MoonEP trace did not demonstrate async grad-reduce overlap Already logged in to Hugging Face as: AbhishekGhosh1998 uploaded: /workspace/tierb-fixed-moonep-full/moonep/tierbtrace -> MoE-CUDA-Graph-Analysis/tierb-b200-gbs8/tierb-fixed-moonep-full/moonep/tierbtrace https://huggingface.co/MoE-CUDA-Graph-Analysis/tierb-b200-gbs8/tree/main study_root: /workspace/tierb-fixed-moonep-full gbs: 8 stages: 4 nsys: 1 hf_repo: MoE-CUDA-Graph-Analysis/tierb-b200-gbs8 hf_remote_root: tierb-fixed-moonep-full started: 2026-08-30T01:30:54+00:00 driver_repo: /workspace/blackwell_run_scripts @ eec9aee === STAGE 4 -- Tier B timing (probe-selected zero-drop capacities) === STEP moonep__tierb__gbs8 ============================================================== CAMPAIGN: moonep backend=moonep phase=tierb Megatron: /workspace/Megatron-MoE-Echo @ moonep_backend_async_grad_reduce GBS=8 seq=4096 repeats=3 nsys=1 results -> /workspace/tierb-fixed-moonep-full/moonep/tierb ============================================================== -------------------------------------------------------------------------- [01:31:09] CELL QWEN3_235B__moonep__matched__seq8192__gbs8__capnone__async (slots=16) [guard] iter 3/12 (60s) [guard] iter 12/12 (70s) [guard] tearing down process group (outcome=DONE) [guard] iter 3/12 (60s) [guard] iter 12/12 (70s) [guard] tearing down process group (outcome=DONE) [guard] iter 3/12 (60s) [guard] iter 12/12 (70s) [guard] tearing down process group (outcome=DONE) -------------------------------------------------------------------------- [01:34:50] CELL QWEN3_30B__moonep__matched__seq4096__gbs8__capnone__async (slots=16) [guard] iter 12/12 (70s) [guard] tearing down process group (outcome=DONE) [guard] iter 12/12 (70s) [guard] tearing down process group (outcome=DONE) [guard] iter 12/12 (70s) [guard] tearing down process group (outcome=DONE) -------------------------------------------------------------------------- [01:38:39] CELL DSV2_LITE_L24__moonep__matched__seq8192__gbs8__capnone__async (slots=8) [guard] iter 12/12 (70s) [guard] tearing down process group (outcome=DONE) [guard] iter 1/12 (60s) [guard] iter 12/12 (70s) [guard] tearing down process group (outcome=DONE) [guard] iter 1/12 (60s) [guard] iter 12/12 (70s) [guard] tearing down process group (outcome=DONE) -------------------------------------------------------------------------- [01:42:26] CELL QWEN2_57B__moonep__matched__seq4096__gbs8__capnone__async (slots=8) [guard] iter 4/12 (60s) [guard] iter 12/12 (70s) [guard] tearing down process group (outcome=DONE) [guard] iter 8/12 (60s) [guard] iter 12/12 (70s) [guard] tearing down process group (outcome=DONE) [guard] iter 5/12 (60s) [guard] iter 12/12 (70s) [guard] tearing down process group (outcome=DONE) -------------------------------------------------------------------------- ============================================================== SWEEP COMPLETE ok=12 failed=0 ============================================================== config backend policy slots seq gbs cap recompute rep status iter_ms_mean iter_ms_min peak_mib run_dir QWEN3_235B moonep matched 16 8192 8 none none 1 OK 317.23 316.40 120309 /workspace/tierb-fixed-moonep-full/moonep/tierb/tierb_gbs8/QWEN3_235B__moonep__matched__seq8192__gbs8__capnone__async/rep1 QWEN3_235B moonep matched 16 8192 8 none none 2 OK 317.53 316.40 120309 /workspace/tierb-fixed-moonep-full/moonep/tierb/tierb_gbs8/QWEN3_235B__moonep__matched__seq8192__gbs8__capnone__async/rep2 QWEN3_235B moonep matched 16 8192 8 none none 3 OK 317.80 316.40 120309 /workspace/tierb-fixed-moonep-full/moonep/tierb/tierb_gbs8/QWEN3_235B__moonep__matched__seq8192__gbs8__capnone__async/rep3 QWEN3_30B moonep matched 16 4096 8 none none 1 OK 354.41 352.30 116707 /workspace/tierb-fixed-moonep-full/moonep/tierb/tierb_gbs8/QWEN3_30B__moonep__matched__seq4096__gbs8__capnone__async/rep1 QWEN3_30B moonep matched 16 4096 8 none none 2 OK 354.46 352.20 116707 /workspace/tierb-fixed-moonep-full/moonep/tierb/tierb_gbs8/QWEN3_30B__moonep__matched__seq4096__gbs8__capnone__async/rep2 QWEN3_30B moonep matched 16 4096 8 none none 3 OK 355.16 352.20 116707 /workspace/tierb-fixed-moonep-full/moonep/tierb/tierb_gbs8/QWEN3_30B__moonep__matched__seq4096__gbs8__capnone__async/rep3 DSV2_LITE_L24 moonep matched 8 8192 8 none none 1 OK 591.91 590.10 133311 /workspace/tierb-fixed-moonep-full/moonep/tierb/tierb_gbs8/DSV2_LITE_L24__moonep__matched__seq8192__gbs8__capnone__async/rep1 DSV2_LITE_L24 moonep matched 8 8192 8 none none 2 OK 593.74 590.10 133311 /workspace/tierb-fixed-moonep-full/moonep/tierb/tierb_gbs8/DSV2_LITE_L24__moonep__matched__seq8192__gbs8__capnone__async/rep2 DSV2_LITE_L24 moonep matched 8 8192 8 none none 3 OK 593.43 590.10 133311 /workspace/tierb-fixed-moonep-full/moonep/tierb/tierb_gbs8/DSV2_LITE_L24__moonep__matched__seq8192__gbs8__capnone__async/rep3 QWEN2_57B moonep matched 8 4096 8 none none 1 OK 180.76 179.90 96739 /workspace/tierb-fixed-moonep-full/moonep/tierb/tierb_gbs8/QWEN2_57B__moonep__matched__seq4096__gbs8__capnone__async/rep1 QWEN2_57B moonep matched 8 4096 8 none none 2 OK 180.24 178.80 96739 /workspace/tierb-fixed-moonep-full/moonep/tierb/tierb_gbs8/QWEN2_57B__moonep__matched__seq4096__gbs8__capnone__async/rep2 QWEN2_57B moonep matched 8 4096 8 none none 3 OK 180.11 178.80 96739 /workspace/tierb-fixed-moonep-full/moonep/tierb/tierb_gbs8/QWEN2_57B__moonep__matched__seq4096__gbs8__capnone__async/rep3 Results: /workspace/tierb-fixed-moonep-full/moonep/tierb/tierb_gbs8/results.tsv Roll up: python3 /workspace/blackwell_run_scripts/granularity/tools/collect.py /workspace/tierb-fixed-moonep-full/moonep/tierb/tierb_gbs8 ============================================================== Campaign results: /workspace/tierb-fixed-moonep-full/moonep/tierb Roll up: python3 /workspace/blackwell_run_scripts/granularity/tools/collect.py /workspace/tierb-fixed-moonep-full/moonep/tierb --compare Already logged in to Hugging Face as: AbhishekGhosh1998 uploaded: /workspace/tierb-fixed-moonep-full/moonep/tierb -> MoE-CUDA-Graph-Analysis/tierb-b200-gbs8/tierb-fixed-moonep-full/moonep/tierb https://huggingface.co/MoE-CUDA-Graph-Analysis/tierb-b200-gbs8/tree/main === ROLL-UP 36 runs ok=32 oom=0 -> /workspace/tierb-fixed-moonep-full/collected_all.tsv (excluded 8 nsys-traced run(s) from the timing comparison) phase config seq gbs policy rc cg asy backend iter_ms vs_best ------------------------------------------------------------------------------------- tierb DSV2_LITE_L24 8192 8 matched none 1 1 moonep 593.43 1.00x tierb QWEN2_57B 4096 8 matched none 1 1 moonep 180.12 1.00x tierb QWEN3_235B 8192 8 matched none 1 1 moonep 317.80 1.00x tierb QWEN3_30B 4096 8 matched none 1 1 moonep 355.16 1.00x tierbmem DSV2_LITE_L24 8192 8 matched none 1 1 moonep 622.80 1.00x tierbmem QWEN2_57B 4096 8 matched none 1 1 moonep 196.60 1.00x tierbmem QWEN3_235B 8192 8 matched none 1 1 moonep 341.70 1.00x tierbmem QWEN3_30B 4096 8 matched none 1 1 moonep 435.30 1.00x tierbprobeDSV2_LITE_L24 8192 8 matched none 0 1 moonep 673.94 1.00x tierbprobeQWEN2_57B 4096 8 matched none 0 1 moonep 238.91 1.00x tierbprobeQWEN3_235B 8192 8 matched none 0 1 moonep 384.76 1.00x tierbprobeQWEN3_30B 4096 8 matched none 0 1 moonep 824.23 1.00x 8 probe runs -> /workspace/tierb-fixed-moonep-full/probe_summary_all.tsv run arm cap rawExpI preRankI postRankI drop% reqCap -------------------------------------------------------------------------------------------------------------------- moonep/tierbprobe/tierbprobe_gbs8/DSV2_LITE_L24__moo natural none 7.5443* 2.1316 1.1058 moonep/tierbprobe/tierbprobe_gbs8/DSV2_LITE_L24__moo forced none 1.0286 1.0061 1.0026 moonep/tierbprobe/tierbprobe_gbs8/QWEN2_57B__moonep_ natural none 7.4132* 2.3019 1.0979 moonep/tierbprobe/tierbprobe_gbs8/QWEN2_57B__moonep_ forced none 1.0348 1.0073 1.0033 moonep/tierbprobe/tierbprobe_gbs8/QWEN3_235B__moonep natural none 15.9227* 2.5851 1.0712 moonep/tierbprobe/tierbprobe_gbs8/QWEN3_235B__moonep forced none 1.0386 1.0053 1.0025 moonep/tierbprobe/tierbprobe_gbs8/QWEN3_30B__moonep_ natural none 15.7078* 2.4672 1.0635 moonep/tierbprobe/tierbprobe_gbs8/QWEN3_30B__moonep_ forced none 1.0565 1.0077 1.0036 * At least one globally aggregated snapshot reached the E/K expert-Imax ceiling. Inspect raw_expert_imax_mean and hottest_expert_token_fraction_mean before generalising. ECHO note: echo_logged_pre_rank_metric_valid=0 and echo_logged_load_reduction_valid=0 are intentional. The branch's built-in fields mix source-rank and destination-rank axes; this tool reconstructs both globally. 8 memory-dump runs -> /workspace/tierb-fixed-moonep-full/memory_summary_all.tsv model backend cg allocation capacity status peak GiB dumps ---------------------------------------------------------------------------------- DSV2_LITE_L24 moonep 1 none none OK 130.2 48/48 DSV2_LITE_L24 moonep 0 none none OK 122.2 40/40 QWEN2_57B moonep 1 none none OK 94.5 48/48 QWEN2_57B moonep 0 none none OK 93.6 40/40 QWEN3_235B moonep 1 none none OK 117.5 48/48 QWEN3_235B moonep 0 none none OK 117.3 40/40 QWEN3_30B moonep 1 none none OK 114.0 48/48 QWEN3_30B moonep 0 none none OK 113.7 40/40 ECHO/UltraEP note: graph-on uses static capacity + device-initiated GEMM while graph-off uses dynamic allocation + host-derived sizes. Their difference is not labelled as pure graph cost. Already logged in to Hugging Face as: AbhishekGhosh1998 uploaded: /workspace/tierb-fixed-moonep-full -> MoE-CUDA-Graph-Analysis/tierb-b200-gbs8/tierb-fixed-moonep-full https://huggingface.co/MoE-CUDA-Graph-Analysis/tierb-b200-gbs8/tree/main === STUDY COMPLETE results : /workspace/tierb-fixed-moonep-full status : /workspace/tierb-fixed-moonep-full/STATUS.txt steps : /workspace/tierb-fixed-moonep-full/steps.tsv hub : https://huggingface.co/datasets/MoE-CUDA-Graph-Analysis/tierb-b200-gbs8 status campaign phase gbs started ended result_dir OK moonep tierbprobe 8 2026-08-29T23:33:50+00:00 2026-08-29T23:44:30+00:00 /workspace/tierb-fixed-moonep-full/moonep/tierbprobe OK moonep tierb 8 2026-08-29T23:44:39+00:00 2026-08-29T23:59:52+00:00 /workspace/tierb-fixed-moonep-full/moonep/tierb OK moonep tierbmem 8 2026-08-29T23:59:59+00:00 2026-08-30T00:09:58+00:00 /workspace/tierb-fixed-moonep-full/moonep/tierbmem OK echo tierbtrace 8 2026-08-30T00:10:16+00:00 2026-08-30T00:38:30+00:00 /workspace/tierb-fixed-moonep-full/echo/tierbtrace OK ultraep tierbtrace 8 2026-08-30T00:38:33+00:00 2026-08-30T00:59:28+00:00 /workspace/tierb-fixed-moonep-full/ultraep/tierbtrace FAIL moonep tierbtrace 8 2026-08-30T00:59:40+00:00 2026-08-30T01:06:12+00:00 /workspace/tierb-fixed-moonep-full/moonep/tierbtrace OK moonep tierb 8 2026-08-30T01:30:54+00:00 2026-08-30T01:46:01+00:00 /workspace/tierb-fixed-moonep-full/moonep/tierb study_root: /workspace/tierb-fixed-moonep-full gbs: 8 stages: 9 nsys: 1 hf_repo: MoE-CUDA-Graph-Analysis/tierb-b200-gbs8 hf_remote_root: tierb-fixed-moonep-full started: 2026-08-30T02:33:16+00:00 driver_repo: /workspace/blackwell_run_scripts @ e8bd2bd === STAGE 9 -- Tier B nsys traces (final, trace-only) === ROLL-UP 36 runs ok=32 oom=0 -> /workspace/tierb-fixed-moonep-full/collected_all.tsv (excluded 8 nsys-traced run(s) from the timing comparison) phase config seq gbs policy rc cg asy backend iter_ms vs_best ------------------------------------------------------------------------------------- tierb DSV2_LITE_L24 8192 8 matched none 1 1 moonep 593.43 1.00x tierb QWEN2_57B 4096 8 matched none 1 1 moonep 180.12 1.00x tierb QWEN3_235B 8192 8 matched none 1 1 moonep 317.80 1.00x tierb QWEN3_30B 4096 8 matched none 1 1 moonep 355.16 1.00x tierbmem DSV2_LITE_L24 8192 8 matched none 1 1 moonep 622.80 1.00x tierbmem QWEN2_57B 4096 8 matched none 1 1 moonep 196.60 1.00x tierbmem QWEN3_235B 8192 8 matched none 1 1 moonep 341.70 1.00x tierbmem QWEN3_30B 4096 8 matched none 1 1 moonep 435.30 1.00x tierbprobeDSV2_LITE_L24 8192 8 matched none 0 1 moonep 673.94 1.00x tierbprobeQWEN2_57B 4096 8 matched none 0 1 moonep 238.91 1.00x tierbprobeQWEN3_235B 8192 8 matched none 0 1 moonep 384.76 1.00x tierbprobeQWEN3_30B 4096 8 matched none 0 1 moonep 824.23 1.00x 8 probe runs -> /workspace/tierb-fixed-moonep-full/probe_summary_all.tsv run arm cap rawExpI preRankI postRankI drop% reqCap -------------------------------------------------------------------------------------------------------------------- moonep/tierbprobe/tierbprobe_gbs8/DSV2_LITE_L24__moo natural none 7.5443* 2.1316 1.1058 moonep/tierbprobe/tierbprobe_gbs8/DSV2_LITE_L24__moo forced none 1.0286 1.0061 1.0026 moonep/tierbprobe/tierbprobe_gbs8/QWEN2_57B__moonep_ natural none 7.4132* 2.3019 1.0979 moonep/tierbprobe/tierbprobe_gbs8/QWEN2_57B__moonep_ forced none 1.0348 1.0073 1.0033 moonep/tierbprobe/tierbprobe_gbs8/QWEN3_235B__moonep natural none 15.9227* 2.5851 1.0712 moonep/tierbprobe/tierbprobe_gbs8/QWEN3_235B__moonep forced none 1.0386 1.0053 1.0025 moonep/tierbprobe/tierbprobe_gbs8/QWEN3_30B__moonep_ natural none 15.7078* 2.4672 1.0635 moonep/tierbprobe/tierbprobe_gbs8/QWEN3_30B__moonep_ forced none 1.0565 1.0077 1.0036 * At least one globally aggregated snapshot reached the E/K expert-Imax ceiling. Inspect raw_expert_imax_mean and hottest_expert_token_fraction_mean before generalising. ECHO note: echo_logged_pre_rank_metric_valid=0 and echo_logged_load_reduction_valid=0 are intentional. The branch's built-in fields mix source-rank and destination-rank axes; this tool reconstructs both globally. 8 memory-dump runs -> /workspace/tierb-fixed-moonep-full/memory_summary_all.tsv model backend cg allocation capacity status peak GiB dumps ---------------------------------------------------------------------------------- DSV2_LITE_L24 moonep 1 none none OK 130.2 48/48 DSV2_LITE_L24 moonep 0 none none OK 122.2 40/40 QWEN2_57B moonep 1 none none OK 94.5 48/48 QWEN2_57B moonep 0 none none OK 93.6 40/40 QWEN3_235B moonep 1 none none OK 117.5 48/48 QWEN3_235B moonep 0 none none OK 117.3 40/40 QWEN3_30B moonep 1 none none OK 114.0 48/48 QWEN3_30B moonep 0 none none OK 113.7 40/40 ECHO/UltraEP note: graph-on uses static capacity + device-initiated GEMM while graph-off uses dynamic allocation + host-derived sizes. Their difference is not labelled as pure graph cost. Already logged in to Hugging Face as: AbhishekGhosh1998 No files have been modified since last commit. Skipping to prevent empty commit. uploaded: /workspace/tierb-fixed-moonep-full -> MoE-CUDA-Graph-Analysis/tierb-b200-gbs8/tierb-fixed-moonep-full https://huggingface.co/MoE-CUDA-Graph-Analysis/tierb-b200-gbs8/tree/main === STUDY COMPLETE results : /workspace/tierb-fixed-moonep-full status : /workspace/tierb-fixed-moonep-full/STATUS.txt steps : /workspace/tierb-fixed-moonep-full/steps.tsv hub : https://huggingface.co/datasets/MoE-CUDA-Graph-Analysis/tierb-b200-gbs8 status campaign phase gbs started ended result_dir OK moonep tierbprobe 8 2026-08-29T23:33:50+00:00 2026-08-29T23:44:30+00:00 /workspace/tierb-fixed-moonep-full/moonep/tierbprobe OK moonep tierb 8 2026-08-29T23:44:39+00:00 2026-08-29T23:59:52+00:00 /workspace/tierb-fixed-moonep-full/moonep/tierb OK moonep tierbmem 8 2026-08-29T23:59:59+00:00 2026-08-30T00:09:58+00:00 /workspace/tierb-fixed-moonep-full/moonep/tierbmem OK echo tierbtrace 8 2026-08-30T00:10:16+00:00 2026-08-30T00:38:30+00:00 /workspace/tierb-fixed-moonep-full/echo/tierbtrace OK ultraep tierbtrace 8 2026-08-30T00:38:33+00:00 2026-08-30T00:59:28+00:00 /workspace/tierb-fixed-moonep-full/ultraep/tierbtrace FAIL moonep tierbtrace 8 2026-08-30T00:59:40+00:00 2026-08-30T01:06:12+00:00 /workspace/tierb-fixed-moonep-full/moonep/tierbtrace OK moonep tierb 8 2026-08-30T01:30:54+00:00 2026-08-30T01:46:01+00:00 /workspace/tierb-fixed-moonep-full/moonep/tierb study_root: /workspace/tierb-fixed-moonep-full gbs: 8 stages: 9 nsys: 1 hf_repo: MoE-CUDA-Graph-Analysis/tierb-b200-gbs8 hf_remote_root: tierb-fixed-moonep-full started: 2026-08-30T02:38:05+00:00 driver_repo: /workspace/blackwell_run_scripts @ e8bd2bd === STAGE 9 -- Tier B nsys traces (final, trace-only) === ROLL-UP 36 runs ok=32 oom=0 -> /workspace/tierb-fixed-moonep-full/collected_all.tsv (excluded 8 nsys-traced run(s) from the timing comparison) phase config seq gbs policy rc cg asy backend iter_ms vs_best ------------------------------------------------------------------------------------- tierb DSV2_LITE_L24 8192 8 matched none 1 1 moonep 593.43 1.00x tierb QWEN2_57B 4096 8 matched none 1 1 moonep 180.12 1.00x tierb QWEN3_235B 8192 8 matched none 1 1 moonep 317.80 1.00x tierb QWEN3_30B 4096 8 matched none 1 1 moonep 355.16 1.00x tierbmem DSV2_LITE_L24 8192 8 matched none 1 1 moonep 622.80 1.00x tierbmem QWEN2_57B 4096 8 matched none 1 1 moonep 196.60 1.00x tierbmem QWEN3_235B 8192 8 matched none 1 1 moonep 341.70 1.00x tierbmem QWEN3_30B 4096 8 matched none 1 1 moonep 435.30 1.00x tierbprobeDSV2_LITE_L24 8192 8 matched none 0 1 moonep 673.94 1.00x tierbprobeQWEN2_57B 4096 8 matched none 0 1 moonep 238.91 1.00x tierbprobeQWEN3_235B 8192 8 matched none 0 1 moonep 384.76 1.00x tierbprobeQWEN3_30B 4096 8 matched none 0 1 moonep 824.23 1.00x 8 probe runs -> /workspace/tierb-fixed-moonep-full/probe_summary_all.tsv run arm cap rawExpI preRankI postRankI drop% reqCap -------------------------------------------------------------------------------------------------------------------- moonep/tierbprobe/tierbprobe_gbs8/DSV2_LITE_L24__moo natural none 7.5443* 2.1316 1.1058 moonep/tierbprobe/tierbprobe_gbs8/DSV2_LITE_L24__moo forced none 1.0286 1.0061 1.0026 moonep/tierbprobe/tierbprobe_gbs8/QWEN2_57B__moonep_ natural none 7.4132* 2.3019 1.0979 moonep/tierbprobe/tierbprobe_gbs8/QWEN2_57B__moonep_ forced none 1.0348 1.0073 1.0033 moonep/tierbprobe/tierbprobe_gbs8/QWEN3_235B__moonep natural none 15.9227* 2.5851 1.0712 moonep/tierbprobe/tierbprobe_gbs8/QWEN3_235B__moonep forced none 1.0386 1.0053 1.0025 moonep/tierbprobe/tierbprobe_gbs8/QWEN3_30B__moonep_ natural none 15.7078* 2.4672 1.0635 moonep/tierbprobe/tierbprobe_gbs8/QWEN3_30B__moonep_ forced none 1.0565 1.0077 1.0036 * At least one globally aggregated snapshot reached the E/K expert-Imax ceiling. Inspect raw_expert_imax_mean and hottest_expert_token_fraction_mean before generalising. ECHO note: echo_logged_pre_rank_metric_valid=0 and echo_logged_load_reduction_valid=0 are intentional. The branch's built-in fields mix source-rank and destination-rank axes; this tool reconstructs both globally. 8 memory-dump runs -> /workspace/tierb-fixed-moonep-full/memory_summary_all.tsv model backend cg allocation capacity status peak GiB dumps ---------------------------------------------------------------------------------- DSV2_LITE_L24 moonep 1 none none OK 130.2 48/48 DSV2_LITE_L24 moonep 0 none none OK 122.2 40/40 QWEN2_57B moonep 1 none none OK 94.5 48/48 QWEN2_57B moonep 0 none none OK 93.6 40/40 QWEN3_235B moonep 1 none none OK 117.5 48/48 QWEN3_235B moonep 0 none none OK 117.3 40/40 QWEN3_30B moonep 1 none none OK 114.0 48/48 QWEN3_30B moonep 0 none none OK 113.7 40/40 ECHO/UltraEP note: graph-on uses static capacity + device-initiated GEMM while graph-off uses dynamic allocation + host-derived sizes. Their difference is not labelled as pure graph cost. Already logged in to Hugging Face as: AbhishekGhosh1998 uploaded: /workspace/tierb-fixed-moonep-full -> MoE-CUDA-Graph-Analysis/tierb-b200-gbs8/tierb-fixed-moonep-full https://huggingface.co/MoE-CUDA-Graph-Analysis/tierb-b200-gbs8/tree/main === STUDY COMPLETE results : /workspace/tierb-fixed-moonep-full status : /workspace/tierb-fixed-moonep-full/STATUS.txt steps : /workspace/tierb-fixed-moonep-full/steps.tsv hub : https://huggingface.co/datasets/MoE-CUDA-Graph-Analysis/tierb-b200-gbs8 status campaign phase gbs started ended result_dir OK moonep tierbprobe 8 2026-08-29T23:33:50+00:00 2026-08-29T23:44:30+00:00 /workspace/tierb-fixed-moonep-full/moonep/tierbprobe OK moonep tierb 8 2026-08-29T23:44:39+00:00 2026-08-29T23:59:52+00:00 /workspace/tierb-fixed-moonep-full/moonep/tierb OK moonep tierbmem 8 2026-08-29T23:59:59+00:00 2026-08-30T00:09:58+00:00 /workspace/tierb-fixed-moonep-full/moonep/tierbmem OK echo tierbtrace 8 2026-08-30T00:10:16+00:00 2026-08-30T00:38:30+00:00 /workspace/tierb-fixed-moonep-full/echo/tierbtrace OK ultraep tierbtrace 8 2026-08-30T00:38:33+00:00 2026-08-30T00:59:28+00:00 /workspace/tierb-fixed-moonep-full/ultraep/tierbtrace SUPERSEDED moonep tierbtrace 8 2026-08-30T00:59:40+00:00 2026-08-30T01:06:12+00:00 /workspace/tierb-fixed-moonep-full/moonep/tierbtrace OK moonep tierb 8 2026-08-30T01:30:54+00:00 2026-08-30T01:46:01+00:00 /workspace/tierb-fixed-moonep-full/moonep/tierb study_root: /workspace/tierb-fixed-moonep-full gbs: 8 stages: 9 nsys: 1 hf_repo: MoE-CUDA-Graph-Analysis/tierb-b200-gbs8 hf_remote_root: tierb-fixed-moonep-full started: 2026-08-30T02:38:13+00:00 driver_repo: /workspace/blackwell_run_scripts @ e8bd2bd === STAGE 9 -- Tier B nsys traces (final, trace-only) === STEP moonep__tierbtrace__gbs8 ============================================================== CAMPAIGN: moonep backend=moonep phase=tierbtrace Megatron: /workspace/Megatron-MoE-Echo @ moonep_backend_async_grad_reduce GBS=8 seq=4096 repeats=3 nsys=1 results -> /workspace/tierb-fixed-moonep-full/moonep/tierbtrace ============================================================== [02:38:28] building moonep marker extension (log: /workspace/tierb-fixed-moonep-full/moonep/tierbtrace/tierbtrace_gbs8/trace_extension_build.log) [02:38:46] marker extension ready: _moonep_trace_C.cpython-312-x86_64-linux-gnu.so [02:38:46] TRACE-ONLY QWEN3_235B__moonep__matched__seq8192__gbs8__capnone__async (ranks: 0 1 2 3 4 5 6 7) [guard] iter 6/10 (60s) [guard] iter 8/10 (70s) [guard] iter 10/10 (110s) [guard] nsys stop (flushing report) [02:41:08] trace verified -> QWEN3_235B__moonep__matched__seq8192__gbs8__capnone__async.nsys-rep (7.3M, all 8 GPUs + moonep markers) [02:41:09] TRACE-ONLY QWEN3_30B__moonep__matched__seq4096__gbs8__capnone__async (ranks: 0 1 2 3 4 5 6 7) [guard] iter 1/10 (60s) [guard] iter 8/10 (70s) [02:45:10] WARN: nsys run failed for QWEN3_30B__moonep__matched__seq4096__gbs8__capnone__async (checking report anyway) [02:45:12] trace verified -> QWEN3_30B__moonep__matched__seq4096__gbs8__capnone__async.nsys-rep (24M, all 8 GPUs + moonep markers) [02:45:14] TRACE-ONLY DSV2_LITE_L24__moonep__matched__seq8192__gbs8__capnone__async (ranks: 0 1 2 3 4 5 6 7) [guard] iter 3/10 (60s) [guard] iter 8/10 (70s) [guard] iter 10/10 (110s) [guard] nsys stop (flushing report) [02:47:36] trace verified -> DSV2_LITE_L24__moonep__matched__seq8192__gbs8__capnone__async.nsys-rep (13M, all 8 GPUs + moonep markers) [02:47:37] TRACE-ONLY QWEN2_57B__moonep__matched__seq4096__gbs8__capnone__async (ranks: 0 1 2 3 4 5 6 7) [guard] iter 8/10 (60s) [guard] iter 10/10 (110s) [guard] nsys stop (flushing report) [02:49:59] trace verified -> QWEN2_57B__moonep__matched__seq4096__gbs8__capnone__async.nsys-rep (5.6M, all 8 GPUs + moonep markers) -------------------------------------------------------------------------- ============================================================== SWEEP COMPLETE ok=4 failed=0 ============================================================== config backend policy slots seq gbs cap recompute rep status iter_ms_mean iter_ms_min peak_mib run_dir Results: /workspace/tierb-fixed-moonep-full/moonep/tierbtrace/tierbtrace_gbs8/results.tsv Roll up: python3 /workspace/blackwell_run_scripts/granularity/tools/collect.py /workspace/tierb-fixed-moonep-full/moonep/tierbtrace/tierbtrace_gbs8 ============================================================== Campaign results: /workspace/tierb-fixed-moonep-full/moonep/tierbtrace Roll up: python3 /workspace/blackwell_run_scripts/granularity/tools/collect.py /workspace/tierb-fixed-moonep-full/moonep/tierbtrace --compare Already logged in to Hugging Face as: AbhishekGhosh1998 uploaded: /workspace/tierb-fixed-moonep-full/moonep/tierbtrace -> MoE-CUDA-Graph-Analysis/tierb-b200-gbs8/tierb-fixed-moonep-full/moonep/tierbtrace https://huggingface.co/MoE-CUDA-Graph-Analysis/tierb-b200-gbs8/tree/main === ROLL-UP 40 runs ok=36 oom=0 -> /workspace/tierb-fixed-moonep-full/collected_all.tsv (excluded 12 nsys-traced run(s) from the timing comparison) phase config seq gbs policy rc cg asy backend iter_ms vs_best ------------------------------------------------------------------------------------- tierb DSV2_LITE_L24 8192 8 matched none 1 1 moonep 593.43 1.00x tierb QWEN2_57B 4096 8 matched none 1 1 moonep 180.12 1.00x tierb QWEN3_235B 8192 8 matched none 1 1 moonep 317.80 1.00x tierb QWEN3_30B 4096 8 matched none 1 1 moonep 355.16 1.00x tierbmem DSV2_LITE_L24 8192 8 matched none 1 1 moonep 622.80 1.00x tierbmem QWEN2_57B 4096 8 matched none 1 1 moonep 196.60 1.00x tierbmem QWEN3_235B 8192 8 matched none 1 1 moonep 341.70 1.00x tierbmem QWEN3_30B 4096 8 matched none 1 1 moonep 435.30 1.00x tierbprobeDSV2_LITE_L24 8192 8 matched none 0 1 moonep 673.94 1.00x tierbprobeQWEN2_57B 4096 8 matched none 0 1 moonep 238.91 1.00x tierbprobeQWEN3_235B 8192 8 matched none 0 1 moonep 384.76 1.00x tierbprobeQWEN3_30B 4096 8 matched none 0 1 moonep 824.23 1.00x 8 probe runs -> /workspace/tierb-fixed-moonep-full/probe_summary_all.tsv run arm cap rawExpI preRankI postRankI drop% reqCap -------------------------------------------------------------------------------------------------------------------- moonep/tierbprobe/tierbprobe_gbs8/DSV2_LITE_L24__moo natural none 7.5443* 2.1316 1.1058 moonep/tierbprobe/tierbprobe_gbs8/DSV2_LITE_L24__moo forced none 1.0286 1.0061 1.0026 moonep/tierbprobe/tierbprobe_gbs8/QWEN2_57B__moonep_ natural none 7.4132* 2.3019 1.0979 moonep/tierbprobe/tierbprobe_gbs8/QWEN2_57B__moonep_ forced none 1.0348 1.0073 1.0033 moonep/tierbprobe/tierbprobe_gbs8/QWEN3_235B__moonep natural none 15.9227* 2.5851 1.0712 moonep/tierbprobe/tierbprobe_gbs8/QWEN3_235B__moonep forced none 1.0386 1.0053 1.0025 moonep/tierbprobe/tierbprobe_gbs8/QWEN3_30B__moonep_ natural none 15.7078* 2.4672 1.0635 moonep/tierbprobe/tierbprobe_gbs8/QWEN3_30B__moonep_ forced none 1.0565 1.0077 1.0036 * At least one globally aggregated snapshot reached the E/K expert-Imax ceiling. Inspect raw_expert_imax_mean and hottest_expert_token_fraction_mean before generalising. ECHO note: echo_logged_pre_rank_metric_valid=0 and echo_logged_load_reduction_valid=0 are intentional. The branch's built-in fields mix source-rank and destination-rank axes; this tool reconstructs both globally. 8 memory-dump runs -> /workspace/tierb-fixed-moonep-full/memory_summary_all.tsv model backend cg allocation capacity status peak GiB dumps ---------------------------------------------------------------------------------- DSV2_LITE_L24 moonep 1 none none OK 130.2 48/48 DSV2_LITE_L24 moonep 0 none none OK 122.2 40/40 QWEN2_57B moonep 1 none none OK 94.5 48/48 QWEN2_57B moonep 0 none none OK 93.6 40/40 QWEN3_235B moonep 1 none none OK 117.5 48/48 QWEN3_235B moonep 0 none none OK 117.3 40/40 QWEN3_30B moonep 1 none none OK 114.0 48/48 QWEN3_30B moonep 0 none none OK 113.7 40/40 ECHO/UltraEP note: graph-on uses static capacity + device-initiated GEMM while graph-off uses dynamic allocation + host-derived sizes. Their difference is not labelled as pure graph cost. Already logged in to Hugging Face as: AbhishekGhosh1998 uploaded: /workspace/tierb-fixed-moonep-full -> MoE-CUDA-Graph-Analysis/tierb-b200-gbs8/tierb-fixed-moonep-full https://huggingface.co/MoE-CUDA-Graph-Analysis/tierb-b200-gbs8/tree/main === STUDY COMPLETE results : /workspace/tierb-fixed-moonep-full status : /workspace/tierb-fixed-moonep-full/STATUS.txt steps : /workspace/tierb-fixed-moonep-full/steps.tsv hub : https://huggingface.co/datasets/MoE-CUDA-Graph-Analysis/tierb-b200-gbs8 status campaign phase gbs started ended result_dir OK moonep tierbprobe 8 2026-08-29T23:33:50+00:00 2026-08-29T23:44:30+00:00 /workspace/tierb-fixed-moonep-full/moonep/tierbprobe OK moonep tierb 8 2026-08-29T23:44:39+00:00 2026-08-29T23:59:52+00:00 /workspace/tierb-fixed-moonep-full/moonep/tierb OK moonep tierbmem 8 2026-08-29T23:59:59+00:00 2026-08-30T00:09:58+00:00 /workspace/tierb-fixed-moonep-full/moonep/tierbmem OK echo tierbtrace 8 2026-08-30T00:10:16+00:00 2026-08-30T00:38:30+00:00 /workspace/tierb-fixed-moonep-full/echo/tierbtrace OK ultraep tierbtrace 8 2026-08-30T00:38:33+00:00 2026-08-30T00:59:28+00:00 /workspace/tierb-fixed-moonep-full/ultraep/tierbtrace SUPERSEDED moonep tierbtrace 8 2026-08-30T00:59:40+00:00 2026-08-30T01:06:12+00:00 /workspace/tierb-fixed-moonep-full/moonep/tierbtrace OK moonep tierb 8 2026-08-30T01:30:54+00:00 2026-08-30T01:46:01+00:00 /workspace/tierb-fixed-moonep-full/moonep/tierb OK moonep tierbtrace 8 2026-08-30T02:38:13+00:00 2026-08-30T02:49:59+00:00 /workspace/tierb-fixed-moonep-full/moonep/tierbtrace
study_root: /workspace/tierb-fixed-moonep-full
gbs: 8
stages: 7
nsys: 1
hf_repo: MoE-CUDA-Graph-Analysis/tierb-b200-gbs8
hf_remote_root: tierb-fixed-moonep-full
started: 2026-08-29T23:33:50+00:00
driver_repo: /workspace/blackwell_run_scripts @ 5054c89
=== STAGE 7 -- Tier B probe and zero-drop capacity selection
=== STEP moonep__tierbprobe__gbs8
==============================================================
CAMPAIGN: moonep backend=moonep phase=tierbprobe
Megatron: /workspace/Megatron-MoE-Echo @ moonep_backend_async_grad_reduce_token_routing_analysis
GBS=8 seq=4096 repeats=3 nsys=1
results -> /workspace/tierb-fixed-moonep-full/moonep/tierbprobe
==============================================================
--------------------------------------------------------------------------
[23:34:05] CELL QWEN3_235B__moonep__matched__seq8192__gbs8__capnone__async (slots=16)
[guard] iter 12/12 (70s)
[guard] tearing down process group (outcome=DONE)
--------------------------------------------------------------------------
[23:35:21] CELL QWEN3_235B__moonep__matched__seq8192__gbs8__capnone__forcebal__async (slots=16)
[guard] iter 12/12 (70s)
[guard] tearing down process group (outcome=DONE)
--------------------------------------------------------------------------
[23:36:36] CELL QWEN3_30B__moonep__matched__seq4096__gbs8__capnone__async (slots=16)
[guard] iter 8/12 (70s)
[guard] iter 12/12 (80s)
[guard] tearing down process group (outcome=DONE)
--------------------------------------------------------------------------
[23:38:01] CELL QWEN3_30B__moonep__matched__seq4096__gbs8__capnone__forcebal__async (slots=16)
[guard] iter 9/12 (70s)
[guard] iter 12/12 (80s)
[guard] tearing down process group (outcome=DONE)
--------------------------------------------------------------------------
[23:39:25] CELL DSV2_LITE_L24__moonep__matched__seq8192__gbs8__capnone__async (slots=8)
[guard] iter 7/12 (70s)
[guard] iter 12/12 (80s)
[guard] tearing down process group (outcome=DONE)
--------------------------------------------------------------------------
[23:40:49] CELL DSV2_LITE_L24__moonep__matched__seq8192__gbs8__capnone__forcebal__async (slots=8)
[guard] iter 12/12 (70s)
[guard] tearing down process group (outcome=DONE)
--------------------------------------------------------------------------
[23:42:04] CELL QWEN2_57B__moonep__matched__seq4096__gbs8__capnone__async (slots=8)
[guard] iter 6/12 (60s)
[guard] iter 12/12 (70s)
[guard] tearing down process group (outcome=DONE)
--------------------------------------------------------------------------
[23:43:17] CELL QWEN2_57B__moonep__matched__seq4096__gbs8__capnone__forcebal__async (slots=8)
[guard] iter 8/12 (60s)
[guard] iter 12/12 (70s)
[guard] tearing down process group (outcome=DONE)
--------------------------------------------------------------------------
==============================================================
SWEEP COMPLETE ok=8 failed=0
==============================================================
config backend policy slots seq gbs cap recompute rep status iter_ms_mean iter_ms_min peak_mib run_dir
QWEN3_235B moonep matched 16 8192 8 none none 1 OK 370.33 368.00 120129 /workspace/tierb-fixed-moonep-full/moonep/tierbprobe/tierbprobe_gbs8/QWEN3_235B__moonep__matched__seq8192__gbs8__capnone__async/rep1
QWEN3_235B moonep matched 16 8192 8 none none+forcebal 1 OK 399.19 354.00 120081 /workspace/tierb-fixed-moonep-full/moonep/tierbprobe/tierbprobe_gbs8/QWEN3_235B__moonep__matched__seq8192__gbs8__capnone__forcebal__async/rep1
QWEN3_30B moonep matched 16 4096 8 none none 1 OK 829.20 747.60 116419 /workspace/tierb-fixed-moonep-full/moonep/tierbprobe/tierbprobe_gbs8/QWEN3_30B__moonep__matched__seq4096__gbs8__capnone__async/rep1
QWEN3_30B moonep matched 16 4096 8 none none+forcebal 1 OK 819.26 774.40 116515 /workspace/tierb-fixed-moonep-full/moonep/tierbprobe/tierbprobe_gbs8/QWEN3_30B__moonep__matched__seq4096__gbs8__capnone__forcebal__async/rep1
DSV2_LITE_L24 moonep matched 8 8192 8 none none 1 OK 673.29 669.50 125157 /workspace/tierb-fixed-moonep-full/moonep/tierbprobe/tierbprobe_gbs8/DSV2_LITE_L24__moonep__matched__seq8192__gbs8__capnone__async/rep1
DSV2_LITE_L24 moonep matched 8 8192 8 none none+forcebal 1 OK 674.59 672.10 125633 /workspace/tierb-fixed-moonep-full/moonep/tierbprobe/tierbprobe_gbs8/DSV2_LITE_L24__moonep__matched__seq8192__gbs8__capnone__forcebal__async/rep1
QWEN2_57B moonep matched 8 4096 8 none none 1 OK 238.47 236.30 95855 /workspace/tierb-fixed-moonep-full/moonep/tierbprobe/tierbprobe_gbs8/QWEN2_57B__moonep__matched__seq4096__gbs8__capnone__async/rep1
QWEN2_57B moonep matched 8 4096 8 none none+forcebal 1 OK 239.34 236.70 95855 /workspace/tierb-fixed-moonep-full/moonep/tierbprobe/tierbprobe_gbs8/QWEN2_57B__moonep__matched__seq4096__gbs8__capnone__forcebal__async/rep1
Results: /workspace/tierb-fixed-moonep-full/moonep/tierbprobe/tierbprobe_gbs8/results.tsv
Roll up: python3 /workspace/blackwell_run_scripts/granularity/tools/collect.py /workspace/tierb-fixed-moonep-full/moonep/tierbprobe/tierbprobe_gbs8
==============================================================
8 probe runs -> /workspace/tierb-fixed-moonep-full/moonep/tierbprobe/tierbprobe_gbs8/probe_summary.tsv
run arm cap rawExpI preRankI postRankI drop% reqCap
--------------------------------------------------------------------------------------------------------------------
DSV2_LITE_L24__moonep__matched__seq8192__gbs8__capno natural none 7.5443* 2.1316 1.1058
DSV2_LITE_L24__moonep__matched__seq8192__gbs8__capno forced none 1.0286 1.0061 1.0026
QWEN2_57B__moonep__matched__seq4096__gbs8__capnone__ natural none 7.4132* 2.3019 1.0979
QWEN2_57B__moonep__matched__seq4096__gbs8__capnone__ forced none 1.0348 1.0073 1.0033
QWEN3_235B__moonep__matched__seq8192__gbs8__capnone_ natural none 15.9227* 2.5851 1.0712
QWEN3_235B__moonep__matched__seq8192__gbs8__capnone_ forced none 1.0386 1.0053 1.0025
QWEN3_30B__moonep__matched__seq4096__gbs8__capnone__ natural none 15.7078* 2.4672 1.0635
QWEN3_30B__moonep__matched__seq4096__gbs8__capnone__ forced none 1.0565 1.0077 1.0036
* At least one globally aggregated snapshot reached the E/K expert-Imax ceiling.
Inspect raw_expert_imax_mean and hottest_expert_token_fraction_mean before generalising.
ECHO note: echo_logged_pre_rank_metric_valid=0 and echo_logged_load_reduction_valid=0 are intentional.
The branch's built-in fields mix source-rank and destination-rank axes; this tool reconstructs both globally.
TIER B PROBE -- what these logs contain, per backend
----------------------------------------------------
Capacity matrix:
ECHO: start C=2.00; on OOM descend 1.75, 1.50, 1.25. Stop at
the highest fitting zero-drop capacity. If a fitting capacity drops,
smaller values cannot help and the model is blocked.
UltraEP: measure required_capacity at C=1.25, round it UP to the next
0.01, then validate that capacity once.
MoonEP: natural + forced, no capacity factor.