AbhishekGhosh1998 commited on
Commit
3aaf01e
·
verified ·
1 Parent(s): 2a772b7

granularity results tierb_study_20260826_144644 @ 2026-08-26T15:22:50Z from sp-aiinfra-interactive-aghosh-b5xzj-8xb200-moe-cg-0

Browse files
tierb_study_20260826_144644/STATUS.txt CHANGED
@@ -1,10 +1,10 @@
1
  study_root: /workspace/tierb_study_20260826_144644
2
  gbs: 8
3
- updated: 2026-08-26T15:05:24+00:00
4
- current: moonep__tierbprobe__gbs8
5
 
6
  completed steps:
7
- echo__tierbprobe__gbs8
8
 
9
  failed steps:
10
  FAIL ultraep tierbprobe 8 2026-08-26T14:57:57+00:00 2026-08-26T15:05:21+00:00 /workspace/tierb_study_20260826_144644/ultraep/tierbprobe
 
 
1
  study_root: /workspace/tierb_study_20260826_144644
2
  gbs: 8
3
+ updated: 2026-08-26T15:22:49+00:00
4
+ current: echo__tierbprobe__gbs8
5
 
6
  completed steps:
 
7
 
8
  failed steps:
9
  FAIL ultraep tierbprobe 8 2026-08-26T14:57:57+00:00 2026-08-26T15:05:21+00:00 /workspace/tierb_study_20260826_144644/ultraep/tierbprobe
10
+ FAIL moonep tierbprobe 8 2026-08-26T15:05:24+00:00 2026-08-26T15:12:52+00:00 /workspace/tierb_study_20260826_144644/moonep/tierbprobe
tierb_study_20260826_144644/echo/tierbprobe/provenance.txt CHANGED
@@ -1,7 +1,7 @@
1
  ==============================================================
2
  PROVENANCE: echo__tierbprobe__gbs8
3
  ==============================================================
4
- captured_at : 2026-08-26T14:46:44+00:00
5
  hostname : sp-aiinfra-interactive-aghosh-b5xzj-8xb200-moe-cg-0
6
  kernel : 6.8.0-1043-nvidia
7
  cwd : /workspace/blackwell_run_scripts
@@ -17,7 +17,7 @@ command : run_ladder.sh --phase tierbprobe --gbs 8 --out /workspace/tierb_
17
  6, NVIDIA B200, 10.0, 183359 MiB, 580.95.05
18
  7, NVIDIA B200, 10.0, 183359 MiB, 580.95.05
19
  persistence/clocks:
20
- 0, 120 MHz, 1965 MHz, 1000.00 W, 27
21
  1, 120 MHz, 1965 MHz, 1000.00 W, 32
22
  2, 120 MHz, 1965 MHz, 1000.00 W, 32
23
  3, 120 MHz, 1965 MHz, 1000.00 W, 29
@@ -27,64 +27,3 @@ command : run_ladder.sh --phase tierbprobe --gbs 8 --out /workspace/tierb_
27
  7, 120 MHz, 1965 MHz, 1000.00 W, 28
28
 
29
  NVLink topology:
30
- GPU0 GPU1 GPU2 GPU3 GPU4 GPU5 GPU6 GPU7 NIC0 NIC1 NIC2 NIC3 NIC4 NIC5 NIC6 NIC7 NIC8 CPU Affinity NUMA Affinity GPU NUMA ID
31
- GPU0 X NV18 NV18 NV18 NV18 NV18 NV18 NV18 PIX NODE SYS SYS SYS SYS SYS SYS NODE 0-27 0 N/A
32
- GPU1 NV18 X NV18 NV18 NV18 NV18 NV18 NV18 NODE PIX SYS SYS SYS SYS SYS SYS NODE 0-27 0 N/A
33
- GPU2 NV18 NV18 X NV18 NV18 NV18 NV18 NV18 SYS SYS PIX NODE SYS SYS SYS SYS SYS 28-55 1 N/A
34
- GPU3 NV18 NV18 NV18 X NV18 NV18 NV18 NV18 SYS SYS NODE PIX SYS SYS SYS SYS SYS 28-55 1 N/A
35
- GPU4 NV18 NV18 NV18 NV18 X NV18 NV18 NV18 SYS SYS SYS SYS PIX NODE SYS SYS SYS 56-83 2 N/A
36
- GPU5 NV18 NV18 NV18 NV18 NV18 X NV18 NV18 SYS SYS SYS SYS NODE PIX SYS SYS SYS 56-83 2 N/A
37
- GPU6 NV18 NV18 NV18 NV18 NV18 NV18 X NV18 SYS SYS SYS SYS SYS SYS PIX NODE SYS 84-111 3 N/A
38
- GPU7 NV18 NV18 NV18 NV18 NV18 NV18 NV18 X SYS SYS SYS SYS SYS SYS NODE PIX SYS 84-111 3 N/A
39
- NIC0 PIX NODE SYS SYS SYS SYS SYS SYS X NODE SYS SYS SYS SYS SYS SYS NODE
40
- NIC1 NODE PIX SYS SYS SYS SYS SYS SYS NODE X SYS SYS SYS SYS SYS SYS NODE
41
- NIC2 SYS SYS PIX NODE SYS SYS SYS SYS SYS SYS X NODE SYS SYS SYS SYS SYS
42
- NIC3 SYS SYS NODE PIX SYS SYS SYS SYS SYS SYS NODE X SYS SYS SYS SYS SYS
43
- NIC4 SYS SYS SYS SYS PIX NODE SYS SYS SYS SYS SYS SYS X NODE SYS SYS SYS
44
- unavailable
45
-
46
- ---- toolchain -----------------------------------------------
47
- nvcc : release 12.9, V12.9.86
48
- gcc : gcc (Ubuntu 13.3.0-6ubuntu2~24.04) 13.3.0
49
- python : 3.12.3
50
- torch : 2.8.0a0+5228986c39.nv25.06 cuda=12.9 nccl=2.27.3
51
- torch arch : ['sm_75', 'sm_80', 'sm_86', 'sm_90', 'sm_100', 'sm_120', 'compute_120']
52
- transformer_engine: 2.10.0.dev0+10f6128d [/usr/local/lib/python3.12/dist-packages/transformer_engine/__init__.py]
53
- deep_ep : 1.2.1+34152ae [/workspace/DeepEP/deep_ep/__init__.py]
54
- ultra_ep : 1.0.0 [/workspace/UltraEP/ultra_ep/__init__.py]
55
- moonep : 0.0.1 [/workspace/MoonEP/moonep/__init__.py]
56
- megatron : 0.14.0rc5 [None]
57
- flash_attn : 2.7.4.post1 [/usr/local/lib/python3.12/dist-packages/flash_attn/__init__.py]
58
- numpy : 1.26.4 [/usr/local/lib/python3.12/dist-packages/numpy/__init__.py]
59
- huggingface_hub: 1.28.0 [/usr/local/lib/python3.12/dist-packages/huggingface_hub/__init__.py]
60
-
61
- ---- repositories --------------------------------------------
62
- Megatron-MoE-Echo token-drop-analysis b099fd8494b0
63
- Megatron-UltraEP ultraep-bracing-regions-through-kernels 03d57ef7d476
64
- UltraEP main 94cab099b44f
65
- DeepEP HEAD 34152ae28f80 (+3 untracked: build output)
66
- MoonEP megatron-integration c27d35da15f2
67
- blackwell_run_scripts granularity-echo 6b6ef3daf514 (+1 untracked: build output)
68
-
69
- ---- environment ---------------------------------------------
70
- CUDA_ARCH_LIST=7.5 8.0 8.6 9.0 10.0 12.0
71
- CUDA_DRIVER_VERSION=575.57.08
72
- CUDA_HOME=/usr/local/cuda
73
- CUDA_MODULE_LOADING=LAZY
74
- CUDA_VERSION=12.9.1.010
75
- HF_REMOTE_ROOT=tierb_study_20260826_144644
76
- NCCL_VERSION=2.27.3
77
- NVSHMEM_VERSION=3.2.5
78
- PYTORCH_BUILD_NUMBER=0
79
- PYTORCH_BUILD_VERSION=2.8.0a0+5228986
80
- PYTORCH_HOME=/opt/pytorch/pytorch
81
- PYTORCH_VERSION=2.8.0a0+5228986
82
- TORCH_ALLOW_TF32_CUBLAS_OVERRIDE=1
83
- TORCH_CUDA_ARCH_LIST=7.5 8.0 8.6 9.0 10.0 12.0+PTX
84
- TORCH_NCCL_USE_COMM_NONBLOCKING=0
85
-
86
- ---- shared library resolution -------------------------------
87
- libnvshmem_host.so: not in ldconfig
88
- libnccl.so: /lib/x86_64-linux-gnu/libnccl.so.2
89
- lrwxrwxrwx 1 root root 20 Aug 26 14:37 /usr/local/lib/python3.12/dist-packages/nvidia/nvshmem/lib/libnvshmem_host.so -> libnvshmem_host.so.3
90
- -rw-r--r-- 1 root root 179045744 Aug 26 14:09 /usr/local/lib/python3.12/dist-packages/nvidia/nvshmem/lib/libnvshmem_host.so.3
 
1
  ==============================================================
2
  PROVENANCE: echo__tierbprobe__gbs8
3
  ==============================================================
4
+ captured_at : 2026-08-26T15:22:49+00:00
5
  hostname : sp-aiinfra-interactive-aghosh-b5xzj-8xb200-moe-cg-0
6
  kernel : 6.8.0-1043-nvidia
7
  cwd : /workspace/blackwell_run_scripts
 
17
  6, NVIDIA B200, 10.0, 183359 MiB, 580.95.05
18
  7, NVIDIA B200, 10.0, 183359 MiB, 580.95.05
19
  persistence/clocks:
20
+ 0, 120 MHz, 1965 MHz, 1000.00 W, 28
21
  1, 120 MHz, 1965 MHz, 1000.00 W, 32
22
  2, 120 MHz, 1965 MHz, 1000.00 W, 32
23
  3, 120 MHz, 1965 MHz, 1000.00 W, 29
 
27
  7, 120 MHz, 1965 MHz, 1000.00 W, 28
28
 
29
  NVLink topology:
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
tierb_study_20260826_144644/hf_watcher.log CHANGED
@@ -7,3 +7,8 @@ https://huggingface.co/MoE-CUDA-Graph-Analysis/tierb-b200-gbs8/tree/main
7
  uploaded: /workspace/tierb_study_20260826_144644 -> MoE-CUDA-Graph-Analysis/tierb-b200-gbs8/tierb_study_20260826_144644
8
  https://huggingface.co/MoE-CUDA-Graph-Analysis/tierb-b200-gbs8/tree/main
9
  [2026-08-26T15:06:49+00:00] backup pass
 
 
 
 
 
 
7
  uploaded: /workspace/tierb_study_20260826_144644 -> MoE-CUDA-Graph-Analysis/tierb-b200-gbs8/tierb_study_20260826_144644
8
  https://huggingface.co/MoE-CUDA-Graph-Analysis/tierb-b200-gbs8/tree/main
9
  [2026-08-26T15:06:49+00:00] backup pass
10
+ uploaded: /workspace/tierb_study_20260826_144644 -> MoE-CUDA-Graph-Analysis/tierb-b200-gbs8/tierb_study_20260826_144644
11
+ https://huggingface.co/MoE-CUDA-Graph-Analysis/tierb-b200-gbs8/tree/main
12
+ Already logged in to Hugging Face as: AbhishekGhosh1998
13
+ Watching /workspace/tierb_study_20260826_144644 -> HF every 600s (kill to stop)
14
+ [2026-08-26T15:22:49+00:00] backup pass
tierb_study_20260826_144644/hf_watcher.pid CHANGED
@@ -1 +1 @@
1
- 11378
 
1
+ 79542
tierb_study_20260826_144644/steps.tsv CHANGED
@@ -1,3 +1,4 @@
1
  status campaign phase gbs started ended result_dir
2
  OK echo tierbprobe 8 2026-08-26T14:46:44+00:00 2026-08-26T14:57:54+00:00 /workspace/tierb_study_20260826_144644/echo/tierbprobe
3
  FAIL ultraep tierbprobe 8 2026-08-26T14:57:57+00:00 2026-08-26T15:05:21+00:00 /workspace/tierb_study_20260826_144644/ultraep/tierbprobe
 
 
1
  status campaign phase gbs started ended result_dir
2
  OK echo tierbprobe 8 2026-08-26T14:46:44+00:00 2026-08-26T14:57:54+00:00 /workspace/tierb_study_20260826_144644/echo/tierbprobe
3
  FAIL ultraep tierbprobe 8 2026-08-26T14:57:57+00:00 2026-08-26T15:05:21+00:00 /workspace/tierb_study_20260826_144644/ultraep/tierbprobe
4
+ FAIL moonep tierbprobe 8 2026-08-26T15:05:24+00:00 2026-08-26T15:12:52+00:00 /workspace/tierb_study_20260826_144644/moonep/tierbprobe
tierb_study_20260826_144644/study.log CHANGED
@@ -313,3 +313,116 @@ https://huggingface.co/MoE-CUDA-Graph-Analysis/tierb-b200-gbs8/tree/main
313
  --------------------------------------------------------------------------
314
  [15:06:40] CELL QWEN3_30B__moonep__matched__seq4096__gbs8__capnone (slots=16)
315
  [15:06:41] waiting for stale ranks to exit ...
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
313
  --------------------------------------------------------------------------
314
  [15:06:40] CELL QWEN3_30B__moonep__matched__seq4096__gbs8__capnone (slots=16)
315
  [15:06:41] waiting for stale ranks to exit ...
316
+ [guard] tearing down process group (outcome=OOM)
317
+ [15:08:02] WARN: QWEN3_30B__moonep__matched__seq4096__gbs8__capnone rep1 -> OOM
318
+ --------------------------------------------------------------------------
319
+ [15:08:02] CELL DSV2_LITE_L24__moonep__matched__seq8192__gbs8__capnone (slots=8)
320
+ [15:08:03] waiting for stale ranks to exit ...
321
+ [guard] iter 9/12 (60s)
322
+ [guard] iter 12/12 (70s)
323
+ [guard] tearing down process group (outcome=DONE)
324
+ --------------------------------------------------------------------------
325
+ [15:09:26] CELL DSV2_LITE_L24__moonep__matched__seq8192__gbs8__capnone__forcebal (slots=8)
326
+ [guard] iter 10/12 (60s)
327
+ [guard] iter 12/12 (70s)
328
+ [guard] tearing down process group (outcome=DONE)
329
+ --------------------------------------------------------------------------
330
+ [15:10:38] CELL QWEN2_57B__moonep__matched__seq4096__gbs8__capnone (slots=8)
331
+ [guard] iter 12/12 (60s)
332
+ [guard] tearing down process group (outcome=DONE)
333
+ --------------------------------------------------------------------------
334
+ [15:11:45] CELL QWEN2_57B__moonep__matched__seq4096__gbs8__capnone__forcebal (slots=8)
335
+ [guard] iter 12/12 (60s)
336
+ [guard] tearing down process group (outcome=DONE)
337
+ --------------------------------------------------------------------------
338
+
339
+ ==============================================================
340
+ SWEEP COMPLETE ok=4 failed=2
341
+ ==============================================================
342
+ config backend policy slots seq gbs cap recompute rep status iter_ms_mean iter_ms_min peak_mib run_dir
343
+ QWEN3_235B moonep matched 16 8192 8 none none 1 OOM NA NA 182587 /workspace/tierb_study_20260826_144644/moonep/tierbprobe/tierbprobe_gbs8/QWEN3_235B__moonep__matched__seq8192__gbs8__capnone/rep1
344
+ QWEN3_30B moonep matched 16 4096 8 none none 1 OOM NA NA 182625 /workspace/tierb_study_20260826_144644/moonep/tierbprobe/tierbprobe_gbs8/QWEN3_30B__moonep__matched__seq4096__gbs8__capnone/rep1
345
+ DSV2_LITE_L24 moonep matched 8 8192 8 none none 1 OK 848.31 791.80 157151 /workspace/tierb_study_20260826_144644/moonep/tierbprobe/tierbprobe_gbs8/DSV2_LITE_L24__moonep__matched__seq8192__gbs8__capnone/rep1
346
+ DSV2_LITE_L24 moonep matched 8 8192 8 none none+forcebal 1 OK 791.39 788.20 157151 /workspace/tierb_study_20260826_144644/moonep/tierbprobe/tierbprobe_gbs8/DSV2_LITE_L24__moonep__matched__seq8192__gbs8__capnone__forcebal/rep1
347
+ QWEN2_57B moonep matched 8 4096 8 none none 1 OK 366.93 362.10 141553 /workspace/tierb_study_20260826_144644/moonep/tierbprobe/tierbprobe_gbs8/QWEN2_57B__moonep__matched__seq4096__gbs8__capnone/rep1
348
+ QWEN2_57B moonep matched 8 4096 8 none none+forcebal 1 OK 407.76 358.70 141551 /workspace/tierb_study_20260826_144644/moonep/tierbprobe/tierbprobe_gbs8/QWEN2_57B__moonep__matched__seq4096__gbs8__capnone__forcebal/rep1
349
+
350
+ Results: /workspace/tierb_study_20260826_144644/moonep/tierbprobe/tierbprobe_gbs8/results.tsv
351
+ Roll up: python3 /workspace/blackwell_run_scripts/granularity/tools/collect.py /workspace/tierb_study_20260826_144644/moonep/tierbprobe/tierbprobe_gbs8
352
+ ==============================================================
353
+ 4 probe runs -> /workspace/tierb_study_20260826_144644/moonep/tierbprobe/tierbprobe_gbs8/probe_summary.tsv
354
+ run arm cap rawExpI preRankI postRankI drop% reqCap
355
+ --------------------------------------------------------------------------------------------------------------------
356
+ DSV2_LITE_L24__moonep__matched__seq8192__gbs8__capno natural none 8.3174* 2.2370 1.1122
357
+ DSV2_LITE_L24__moonep__matched__seq8192__gbs8__capno forced none 1.0286 1.0061 1.0026
358
+ QWEN2_57B__moonep__matched__seq4096__gbs8__capnone/r natural none 7.4145* 2.3224 1.1036
359
+ QWEN2_57B__moonep__matched__seq4096__gbs8__capnone__ forced none 1.0357 1.0090 1.0033
360
+
361
+ * At least one globally aggregated snapshot reached the E/K expert-Imax ceiling.
362
+ Inspect raw_expert_imax_mean and hottest_expert_token_fraction_mean before generalising.
363
+
364
+ ECHO note: echo_logged_pre_rank_metric_valid=0 and echo_logged_load_reduction_valid=0 are intentional.
365
+ The branch's built-in fields mix source-rank and destination-rank axes; this tool reconstructs both globally.
366
+
367
+ TIER B PROBE -- what these logs contain, per backend
368
+ ----------------------------------------------------
369
+ Capacity matrix:
370
+ ECHO: start C=2.00; on OOM descend 1.75, 1.50, 1.25. Stop at
371
+ the highest fitting zero-drop capacity. If a fitting capacity drops,
372
+ smaller values cannot help and the model is blocked.
373
+ UltraEP: measure required_capacity at C=1.25, round it UP to the next
374
+ 0.01, then validate that capacity once.
375
+ MoonEP: natural + forced, no capacity factor.
376
+
377
+ The forced arm runs once at the selected/high-headroom capacity. It validates
378
+ instrumentation and is not part of capacity selection.
379
+
380
+ Each backend instruments a different thing:
381
+
382
+ echo local raw-router histograms and local post-reroute placement.
383
+ reduce_probe.py aligns and sums all eight source ranks before
384
+ calculating raw_expert_*, pre_home_rank_*,
385
+ post_physical_rank_* and valid_max_rank_reduction_*.
386
+
387
+ ultraep global pre/post rank vectors plus explicit drop accounting:
388
+ drop_rate_pct and required_capacity -- the capacity factor the run
389
+ WOULD have needed. The vectors are duplicated identically on all
390
+ logging ranks and are deduplicated, not summed.
391
+
392
+ moonep local raw-router histograms, actual payload rows per physical
393
+ destination, and the bytes-copy side. The reducer globally sums
394
+ each (iteration, layer) snapshot first. ECHO reports no comparable
395
+ byte counters, so MoonEP bytes are a self-comparison across models,
396
+ never a cross-backend ranking.
397
+
398
+ Do NOT quote ECHO's logged pre_echo imbalance_ratio or
399
+ max_load_reduction_pct. The branch mixes source-rank and destination-rank
400
+ axes; the reducer marks both invalid and reconstructs the comparison globally.
401
+
402
+ Two boundary cases:
403
+ raw_expert_imax_max == E/K
404
+ -> at least one global snapshot had one expert in every
405
+ token's top-K set. Inspect the MEAN before generalising.
406
+ required_capacity or min-safe-capacity at the configured capacity
407
+ -> clipped by the buffer; it means "at or over", not
408
+ "fits comfortably".
409
+
410
+ tools/reduce_probe.py has already written probe_summary.tsv and .json here.
411
+
412
+ Campaign results: /workspace/tierb_study_20260826_144644/moonep/tierbprobe
413
+ Roll up: python3 /workspace/blackwell_run_scripts/granularity/tools/collect.py /workspace/tierb_study_20260826_144644/moonep/tierbprobe --compare
414
+ Already logged in to Hugging Face as: AbhishekGhosh1998
415
+ uploaded: /workspace/tierb_study_20260826_144644/moonep/tierbprobe -> MoE-CUDA-Graph-Analysis/tierb-b200-gbs8/tierb_study_20260826_144644/moonep/tierbprobe
416
+ https://huggingface.co/MoE-CUDA-Graph-Analysis/tierb-b200-gbs8/tree/main
417
+ study_root: /workspace/tierb_study_20260826_144644
418
+ gbs: 8
419
+ stages: 7,4,8,9
420
+ nsys: 1
421
+ hf_repo: MoE-CUDA-Graph-Analysis/tierb-b200-gbs8
422
+ hf_remote_root: tierb_study_20260826_144644
423
+ started: 2026-08-26T15:22:49+00:00
424
+ driver_repo: /workspace/blackwell_run_scripts @ 7ec1185
425
+
426
+ === STAGE 7 -- Tier B probe and zero-drop capacity selection
427
+
428
+ === STEP echo__tierbprobe__gbs8