AbhishekGhosh1998 commited on
Commit
5c30fc6
·
verified ·
1 Parent(s): 67812db

granularity results tiera-rerun-fixed-moonep @ 2026-08-30T10:17:11Z from sp-aiinfra-interactive-aghosh-7jtjl-8xb200-moe-cg-0

Browse files
tiera-rerun-fixed-moonep/STATUS.txt CHANGED
@@ -1,7 +1,7 @@
1
  study_root: /workspace/tiera-rerun-fixed-moonep
2
  gbs: 16
3
- updated: 2026-08-30T10:16:54+00:00
4
- current: idle
5
 
6
  completed steps:
7
  echo__a1__gbs16
 
1
  study_root: /workspace/tiera-rerun-fixed-moonep
2
  gbs: 16
3
+ updated: 2026-08-30T10:17:11+00:00
4
+ current: echo__tierbmem__gbs16__G4
5
 
6
  completed steps:
7
  echo__a1__gbs16
tiera-rerun-fixed-moonep/echo/tierbmem/provenance.txt CHANGED
@@ -1,7 +1,7 @@
1
  ==============================================================
2
- PROVENANCE: echo__tierbmem__gbs16__G3
3
  ==============================================================
4
- captured_at : 2026-08-30T10:08:07+00:00
5
  hostname : sp-aiinfra-interactive-aghosh-7jtjl-8xb200-moe-cg-0
6
  kernel : 6.8.0-1043-nvidia
7
  cwd : /workspace/blackwell_run_scripts
@@ -24,67 +24,6 @@ command : run_ladder.sh --phase tierbmem --gbs 16 --out /workspace/tiera-r
24
  4, 120 MHz, 1965 MHz, 1000.00 W, 29
25
  5, 120 MHz, 1965 MHz, 1000.00 W, 32
26
  6, 120 MHz, 1965 MHz, 1000.00 W, 33
27
- 7, 120 MHz, 1965 MHz, 1000.00 W, 29
28
 
29
  NVLink topology:
30
- GPU0 GPU1 GPU2 GPU3 GPU4 GPU5 GPU6 GPU7 NIC0 NIC1 NIC2 NIC3 NIC4 NIC5 NIC6 NIC7 NIC8 CPU Affinity NUMA Affinity GPU NUMA ID
31
- GPU0 X NV18 NV18 NV18 NV18 NV18 NV18 NV18 PIX NODE SYS SYS SYS SYS SYS SYS NODE 0-27 0 N/A
32
- GPU1 NV18 X NV18 NV18 NV18 NV18 NV18 NV18 NODE PIX SYS SYS SYS SYS SYS SYS NODE 0-27 0 N/A
33
- GPU2 NV18 NV18 X NV18 NV18 NV18 NV18 NV18 SYS SYS PIX NODE SYS SYS SYS SYS SYS 28-55 1 N/A
34
- GPU3 NV18 NV18 NV18 X NV18 NV18 NV18 NV18 SYS SYS NODE PIX SYS SYS SYS SYS SYS 28-55 1 N/A
35
- GPU4 NV18 NV18 NV18 NV18 X NV18 NV18 NV18 SYS SYS SYS SYS PIX NODE SYS SYS SYS 56-83 2 N/A
36
- GPU5 NV18 NV18 NV18 NV18 NV18 X NV18 NV18 SYS SYS SYS SYS NODE PIX SYS SYS SYS 56-83 2 N/A
37
- GPU6 NV18 NV18 NV18 NV18 NV18 NV18 X NV18 SYS SYS SYS SYS SYS SYS PIX NODE SYS 84-111 3 N/A
38
- GPU7 NV18 NV18 NV18 NV18 NV18 NV18 NV18 X SYS SYS SYS SYS SYS SYS NODE PIX SYS 84-111 3 N/A
39
- NIC0 PIX NODE SYS SYS SYS SYS SYS SYS X NODE SYS SYS SYS SYS SYS SYS NODE
40
- NIC1 NODE PIX SYS SYS SYS SYS SYS SYS NODE X SYS SYS SYS SYS SYS SYS NODE
41
- NIC2 SYS SYS PIX NODE SYS SYS SYS SYS SYS SYS X NODE SYS SYS SYS SYS SYS
42
- NIC3 SYS SYS NODE PIX SYS SYS SYS SYS SYS SYS NODE X SYS SYS SYS SYS SYS
43
- NIC4 SYS SYS SYS SYS PIX NODE SYS SYS SYS SYS SYS SYS X NODE SYS SYS SYS
44
- unavailable
45
-
46
- ---- toolchain -----------------------------------------------
47
- nvcc : release 12.9, V12.9.86
48
- gcc : gcc (Ubuntu 13.3.0-6ubuntu2~24.04) 13.3.0
49
- python : 3.12.3
50
- torch : 2.8.0a0+5228986c39.nv25.06 cuda=12.9 nccl=2.27.3
51
- torch arch : ['sm_75', 'sm_80', 'sm_86', 'sm_90', 'sm_100', 'sm_120', 'compute_120']
52
- transformer_engine: 2.10.0.dev0+10f6128d [/usr/local/lib/python3.12/dist-packages/transformer_engine/__init__.py]
53
- deep_ep : 1.2.1+34152ae [/workspace/DeepEP/deep_ep/__init__.py]
54
- ultra_ep : 1.0.0 [/workspace/UltraEP/ultra_ep/__init__.py]
55
- moonep : 0.0.1 [/workspace/MoonEP/moonep/__init__.py]
56
- megatron : 0.14.0rc5 [None]
57
- flash_attn : 2.7.4.post1 [/usr/local/lib/python3.12/dist-packages/flash_attn/__init__.py]
58
- numpy : 1.26.4 [/usr/local/lib/python3.12/dist-packages/numpy/__init__.py]
59
- huggingface_hub: 1.29.0 [/usr/local/lib/python3.12/dist-packages/huggingface_hub/__init__.py]
60
-
61
- ---- repositories --------------------------------------------
62
- Megatron-MoE-Echo better-memory-dump-support b7c52442d011
63
- Megatron-UltraEP ultraep-better-memory-dump-support c3a8a8a7b69d
64
- UltraEP main 94cab099b44f
65
- DeepEP HEAD 34152ae28f80 (+3 untracked: build output)
66
- MoonEP megatron-ddp-symmetric-params d8bc6667fad7
67
- blackwell_run_scripts granularity-echo 55767276f872
68
-
69
- ---- environment ---------------------------------------------
70
- CUDA_ARCH_LIST=7.5 8.0 8.6 9.0 10.0 12.0
71
- CUDA_DRIVER_VERSION=575.57.08
72
- CUDA_HOME=/usr/local/cuda
73
- CUDA_MODULE_LOADING=LAZY
74
- CUDA_VERSION=12.9.1.010
75
- HF_REMOTE_ROOT=tiera-rerun-fixed-moonep
76
- NCCL_VERSION=2.27.3
77
- NVSHMEM_VERSION=3.2.5
78
- PYTORCH_BUILD_NUMBER=0
79
- PYTORCH_BUILD_VERSION=2.8.0a0+5228986
80
- PYTORCH_HOME=/opt/pytorch/pytorch
81
- PYTORCH_VERSION=2.8.0a0+5228986
82
- TORCH_ALLOW_TF32_CUBLAS_OVERRIDE=1
83
- TORCH_CUDA_ARCH_LIST=7.5 8.0 8.6 9.0 10.0 12.0+PTX
84
- TORCH_NCCL_USE_COMM_NONBLOCKING=0
85
-
86
- ---- shared library resolution -------------------------------
87
- libnvshmem_host.so: not in ldconfig
88
- libnccl.so: /lib/x86_64-linux-gnu/libnccl.so.2
89
- lrwxrwxrwx 1 root root 20 Aug 29 16:41 /usr/local/lib/python3.12/dist-packages/nvidia/nvshmem/lib/libnvshmem_host.so -> libnvshmem_host.so.3
90
- -rw-r--r-- 1 root root 179045744 Aug 29 16:37 /usr/local/lib/python3.12/dist-packages/nvidia/nvshmem/lib/libnvshmem_host.so.3
 
1
  ==============================================================
2
+ PROVENANCE: echo__tierbmem__gbs16__G4
3
  ==============================================================
4
+ captured_at : 2026-08-30T10:17:11+00:00
5
  hostname : sp-aiinfra-interactive-aghosh-7jtjl-8xb200-moe-cg-0
6
  kernel : 6.8.0-1043-nvidia
7
  cwd : /workspace/blackwell_run_scripts
 
24
  4, 120 MHz, 1965 MHz, 1000.00 W, 29
25
  5, 120 MHz, 1965 MHz, 1000.00 W, 32
26
  6, 120 MHz, 1965 MHz, 1000.00 W, 33
27
+ 7, 120 MHz, 1965 MHz, 1000.00 W, 28
28
 
29
  NVLink topology: