AbhishekGhosh1998 commited on
Commit
3d2a32e
·
verified ·
1 Parent(s): f23b68c

granularity results tierb_study_20260826_144644 @ 2026-08-26T17:21:49Z from sp-aiinfra-interactive-aghosh-b5xzj-8xb200-moe-cg-0

Browse files
tierb_study_20260826_144644/STATUS.txt CHANGED
@@ -1,6 +1,6 @@
1
  study_root: /workspace/tierb_study_20260826_144644
2
  gbs: 8
3
- updated: 2026-08-26T17:09:11+00:00
4
  current: echo__tierbprobe__gbs8
5
 
6
  completed steps:
 
1
  study_root: /workspace/tierb_study_20260826_144644
2
  gbs: 8
3
+ updated: 2026-08-26T17:21:48+00:00
4
  current: echo__tierbprobe__gbs8
5
 
6
  completed steps:
tierb_study_20260826_144644/echo/tierbprobe/provenance.txt CHANGED
@@ -1,7 +1,7 @@
1
  ==============================================================
2
  PROVENANCE: echo__tierbprobe__gbs8
3
  ==============================================================
4
- captured_at : 2026-08-26T17:09:11+00:00
5
  hostname : sp-aiinfra-interactive-aghosh-b5xzj-8xb200-moe-cg-0
6
  kernel : 6.8.0-1043-nvidia
7
  cwd : /workspace/blackwell_run_scripts
@@ -22,69 +22,8 @@ command : run_ladder.sh --phase tierbprobe --gbs 8 --out /workspace/tierb_
22
  2, 120 MHz, 1965 MHz, 1000.00 W, 32
23
  3, 120 MHz, 1965 MHz, 1000.00 W, 29
24
  4, 120 MHz, 1965 MHz, 1000.00 W, 28
25
- 5, 120 MHz, 1965 MHz, 1000.00 W, 33
26
  6, 120 MHz, 1965 MHz, 1000.00 W, 32
27
  7, 120 MHz, 1965 MHz, 1000.00 W, 28
28
 
29
  NVLink topology:
30
- GPU0 GPU1 GPU2 GPU3 GPU4 GPU5 GPU6 GPU7 NIC0 NIC1 NIC2 NIC3 NIC4 NIC5 NIC6 NIC7 NIC8 CPU Affinity NUMA Affinity GPU NUMA ID
31
- GPU0 X NV18 NV18 NV18 NV18 NV18 NV18 NV18 PIX NODE SYS SYS SYS SYS SYS SYS NODE 0-27 0 N/A
32
- GPU1 NV18 X NV18 NV18 NV18 NV18 NV18 NV18 NODE PIX SYS SYS SYS SYS SYS SYS NODE 0-27 0 N/A
33
- GPU2 NV18 NV18 X NV18 NV18 NV18 NV18 NV18 SYS SYS PIX NODE SYS SYS SYS SYS SYS 28-55 1 N/A
34
- GPU3 NV18 NV18 NV18 X NV18 NV18 NV18 NV18 SYS SYS NODE PIX SYS SYS SYS SYS SYS 28-55 1 N/A
35
- GPU4 NV18 NV18 NV18 NV18 X NV18 NV18 NV18 SYS SYS SYS SYS PIX NODE SYS SYS SYS 56-83 2 N/A
36
- GPU5 NV18 NV18 NV18 NV18 NV18 X NV18 NV18 SYS SYS SYS SYS NODE PIX SYS SYS SYS 56-83 2 N/A
37
- GPU6 NV18 NV18 NV18 NV18 NV18 NV18 X NV18 SYS SYS SYS SYS SYS SYS PIX NODE SYS 84-111 3 N/A
38
- GPU7 NV18 NV18 NV18 NV18 NV18 NV18 NV18 X SYS SYS SYS SYS SYS SYS NODE PIX SYS 84-111 3 N/A
39
- NIC0 PIX NODE SYS SYS SYS SYS SYS SYS X NODE SYS SYS SYS SYS SYS SYS NODE
40
- NIC1 NODE PIX SYS SYS SYS SYS SYS SYS NODE X SYS SYS SYS SYS SYS SYS NODE
41
- NIC2 SYS SYS PIX NODE SYS SYS SYS SYS SYS SYS X NODE SYS SYS SYS SYS SYS
42
- NIC3 SYS SYS NODE PIX SYS SYS SYS SYS SYS SYS NODE X SYS SYS SYS SYS SYS
43
- NIC4 SYS SYS SYS SYS PIX NODE SYS SYS SYS SYS SYS SYS X NODE SYS SYS SYS
44
- unavailable
45
-
46
- ---- toolchain -----------------------------------------------
47
- nvcc : release 12.9, V12.9.86
48
- gcc : gcc (Ubuntu 13.3.0-6ubuntu2~24.04) 13.3.0
49
- python : 3.12.3
50
- torch : 2.8.0a0+5228986c39.nv25.06 cuda=12.9 nccl=2.27.3
51
- torch arch : ['sm_75', 'sm_80', 'sm_86', 'sm_90', 'sm_100', 'sm_120', 'compute_120']
52
- transformer_engine: 2.10.0.dev0+10f6128d [/usr/local/lib/python3.12/dist-packages/transformer_engine/__init__.py]
53
- deep_ep : 1.2.1+34152ae [/workspace/DeepEP/deep_ep/__init__.py]
54
- ultra_ep : 1.0.0 [/workspace/UltraEP/ultra_ep/__init__.py]
55
- moonep : 0.0.1 [/workspace/MoonEP/moonep/__init__.py]
56
- megatron : 0.14.0rc5 [None]
57
- flash_attn : 2.7.4.post1 [/usr/local/lib/python3.12/dist-packages/flash_attn/__init__.py]
58
- numpy : 1.26.4 [/usr/local/lib/python3.12/dist-packages/numpy/__init__.py]
59
- huggingface_hub: 1.28.0 [/usr/local/lib/python3.12/dist-packages/huggingface_hub/__init__.py]
60
-
61
- ---- repositories --------------------------------------------
62
- Megatron-MoE-Echo token-drop-analysis b099fd8494b0
63
- Megatron-UltraEP ultraep-token-drop-analysis eaf829e7ad21
64
- UltraEP main 94cab099b44f
65
- DeepEP HEAD 34152ae28f80 (+3 untracked: build output)
66
- MoonEP megatron-integration c27d35da15f2
67
- blackwell_run_scripts granularity-echo 655301106287
68
-
69
- ---- environment ---------------------------------------------
70
- CUDA_ARCH_LIST=7.5 8.0 8.6 9.0 10.0 12.0
71
- CUDA_DRIVER_VERSION=575.57.08
72
- CUDA_HOME=/usr/local/cuda
73
- CUDA_MODULE_LOADING=LAZY
74
- CUDA_VERSION=12.9.1.010
75
- HF_REMOTE_ROOT=tierb_study_20260826_144644
76
- NCCL_VERSION=2.27.3
77
- NVSHMEM_VERSION=3.2.5
78
- PYTORCH_BUILD_NUMBER=0
79
- PYTORCH_BUILD_VERSION=2.8.0a0+5228986
80
- PYTORCH_HOME=/opt/pytorch/pytorch
81
- PYTORCH_VERSION=2.8.0a0+5228986
82
- TORCH_ALLOW_TF32_CUBLAS_OVERRIDE=1
83
- TORCH_CUDA_ARCH_LIST=7.5 8.0 8.6 9.0 10.0 12.0+PTX
84
- TORCH_NCCL_USE_COMM_NONBLOCKING=0
85
-
86
- ---- shared library resolution -------------------------------
87
- libnvshmem_host.so: not in ldconfig
88
- libnccl.so: /lib/x86_64-linux-gnu/libnccl.so.2
89
- lrwxrwxrwx 1 root root 20 Aug 26 14:37 /usr/local/lib/python3.12/dist-packages/nvidia/nvshmem/lib/libnvshmem_host.so -> libnvshmem_host.so.3
90
- -rw-r--r-- 1 root root 179045744 Aug 26 14:09 /usr/local/lib/python3.12/dist-packages/nvidia/nvshmem/lib/libnvshmem_host.so.3
 
1
  ==============================================================
2
  PROVENANCE: echo__tierbprobe__gbs8
3
  ==============================================================
4
+ captured_at : 2026-08-26T17:21:48+00:00
5
  hostname : sp-aiinfra-interactive-aghosh-b5xzj-8xb200-moe-cg-0
6
  kernel : 6.8.0-1043-nvidia
7
  cwd : /workspace/blackwell_run_scripts
 
22
  2, 120 MHz, 1965 MHz, 1000.00 W, 32
23
  3, 120 MHz, 1965 MHz, 1000.00 W, 29
24
  4, 120 MHz, 1965 MHz, 1000.00 W, 28
25
+ 5, 120 MHz, 1965 MHz, 1000.00 W, 32
26
  6, 120 MHz, 1965 MHz, 1000.00 W, 32
27
  7, 120 MHz, 1965 MHz, 1000.00 W, 28
28
 
29
  NVLink topology:
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
tierb_study_20260826_144644/echo/tierbprobe/tierbprobe_gbs8/DSV2_LITE_L24__echo__matched__seq8192__gbs8__cap1p0234375/rep1/mem_samples.csv CHANGED
@@ -38,3 +38,107 @@
38
  5, 4
39
  6, 4
40
  7, 1
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
38
  5, 4
39
  6, 4
40
  7, 1
41
+ 0, 4
42
+ 1, 4
43
+ 2, 4
44
+ 3, 4
45
+ 4, 4
46
+ 5, 4
47
+ 6, 27
48
+ 7, 4
49
+ 0, 821
50
+ 1, 821
51
+ 2, 821
52
+ 3, 821
53
+ 4, 821
54
+ 5, 821
55
+ 6, 821
56
+ 7, 821
57
+ 0, 1955
58
+ 1, 1955
59
+ 2, 1955
60
+ 3, 1955
61
+ 4, 1955
62
+ 5, 1955
63
+ 6, 1955
64
+ 7, 1955
65
+ 0, 3535
66
+ 1, 3789
67
+ 2, 3583
68
+ 3, 3547
69
+ 4, 3789
70
+ 5, 3767
71
+ 6, 3667
72
+ 7, 3727
73
+ 0, 26591
74
+ 1, 26663
75
+ 2, 26543
76
+ 3, 26529
77
+ 4, 26537
78
+ 5, 26543
79
+ 6, 26637
80
+ 7, 26701
81
+ 0, 36387
82
+ 1, 36481
83
+ 2, 36341
84
+ 3, 36329
85
+ 4, 36311
86
+ 5, 36299
87
+ 6, 36451
88
+ 7, 36455
89
+ 0, 37783
90
+ 1, 37867
91
+ 2, 37717
92
+ 3, 37715
93
+ 4, 37715
94
+ 5, 37713
95
+ 6, 37849
96
+ 7, 37867
97
+ 0, 37871
98
+ 1, 37955
99
+ 2, 37805
100
+ 3, 37803
101
+ 4, 37803
102
+ 5, 37801
103
+ 6, 37937
104
+ 7, 37955
105
+ 0, 37873
106
+ 1, 37957
107
+ 2, 37807
108
+ 3, 37805
109
+ 4, 37805
110
+ 5, 37803
111
+ 6, 37939
112
+ 7, 37957
113
+ 0, 38819
114
+ 1, 38903
115
+ 2, 38753
116
+ 3, 38753
117
+ 4, 38753
118
+ 5, 38749
119
+ 6, 38885
120
+ 7, 38903
121
+ 0, 38819
122
+ 1, 38903
123
+ 2, 38753
124
+ 3, 38753
125
+ 4, 38753
126
+ 5, 38749
127
+ 6, 38885
128
+ 7, 38903
129
+ 0, 38825
130
+ 1, 38909
131
+ 2, 38759
132
+ 3, 38759
133
+ 4, 38759
134
+ 5, 38755
135
+ 6, 38891
136
+ 7, 38909
137
+ 0, 38827
138
+ 1, 38913
139
+ 2, 38761
140
+ 3, 38761
141
+ 4, 38761
142
+ 5, 38759
143
+ 6, 38893
144
+ 7, 38913
tierb_study_20260826_144644/echo/tierbprobe/tierbprobe_gbs8/DSV2_LITE_L24__echo__matched__seq8192__gbs8__cap1p0234375/rep1/train.log CHANGED
The diff for this file is too large to render. See raw diff
 
tierb_study_20260826_144644/hf_watcher.log CHANGED
@@ -58,3 +58,8 @@ Watching /workspace/tierb_study_20260826_144644 -> HF every 600s (kill to stop)
58
  uploaded: /workspace/tierb_study_20260826_144644 -> MoE-CUDA-Graph-Analysis/tierb-b200-gbs8/tierb_study_20260826_144644
59
  https://huggingface.co/MoE-CUDA-Graph-Analysis/tierb-b200-gbs8/tree/main
60
  [2026-08-26T17:19:17+00:00] backup pass
 
 
 
 
 
 
58
  uploaded: /workspace/tierb_study_20260826_144644 -> MoE-CUDA-Graph-Analysis/tierb-b200-gbs8/tierb_study_20260826_144644
59
  https://huggingface.co/MoE-CUDA-Graph-Analysis/tierb-b200-gbs8/tree/main
60
  [2026-08-26T17:19:17+00:00] backup pass
61
+ uploaded: /workspace/tierb_study_20260826_144644 -> MoE-CUDA-Graph-Analysis/tierb-b200-gbs8/tierb_study_20260826_144644
62
+ https://huggingface.co/MoE-CUDA-Graph-Analysis/tierb-b200-gbs8/tree/main
63
+ Already logged in to Hugging Face as: AbhishekGhosh1998
64
+ Watching /workspace/tierb_study_20260826_144644 -> HF every 600s (kill to stop)
65
+ [2026-08-26T17:21:49+00:00] backup pass
tierb_study_20260826_144644/hf_watcher.pid CHANGED
@@ -1 +1 @@
1
- 325115
 
1
+ 358610
tierb_study_20260826_144644/study.log CHANGED
@@ -1422,3 +1422,15 @@ driver_repo: /workspace/blackwell_run_scripts @ 6553011
1422
  [17:19:04] DSV2_LITE_L24 echo: measured required=1.0234375 at C=2.00 -> validating C=1.0234375
1423
  --------------------------------------------------------------------------
1424
  [17:19:04] CELL DSV2_LITE_L24__echo__matched__seq8192__gbs8__cap1p0234375 (slots=8)
 
 
 
 
 
 
 
 
 
 
 
 
 
1422
  [17:19:04] DSV2_LITE_L24 echo: measured required=1.0234375 at C=2.00 -> validating C=1.0234375
1423
  --------------------------------------------------------------------------
1424
  [17:19:04] CELL DSV2_LITE_L24__echo__matched__seq8192__gbs8__cap1p0234375 (slots=8)
1425
+ study_root: /workspace/tierb_study_20260826_144644
1426
+ gbs: 8
1427
+ stages: 7,4,8,9
1428
+ nsys: 1
1429
+ hf_repo: MoE-CUDA-Graph-Analysis/tierb-b200-gbs8
1430
+ hf_remote_root: tierb_study_20260826_144644
1431
+ started: 2026-08-26T17:21:48+00:00
1432
+ driver_repo: /workspace/blackwell_run_scripts @ db00a1b
1433
+
1434
+ === STAGE 7 -- Tier B probe and zero-drop capacity selection
1435
+
1436
+ === STEP echo__tierbprobe__gbs8