AbhishekGhosh1998 commited on
Commit
69260c3
·
verified ·
1 Parent(s): 91b144d

granularity results tiera-rerun-fixed-moonep @ 2026-08-30T06:20:31Z from sp-aiinfra-interactive-aghosh-7jtjl-8xb200-moe-cg-0

Browse files
tiera-rerun-fixed-moonep/STATUS.txt CHANGED
@@ -1,6 +1,6 @@
1
  study_root: /workspace/tiera-rerun-fixed-moonep
2
  gbs: 16
3
- updated: 2026-08-30T05:49:16+00:00
4
  current: echo__a1__gbs16
5
 
6
  completed steps:
 
1
  study_root: /workspace/tiera-rerun-fixed-moonep
2
  gbs: 16
3
+ updated: 2026-08-30T06:20:31+00:00
4
  current: echo__a1__gbs16
5
 
6
  completed steps:
tiera-rerun-fixed-moonep/TIERA_RERUN_MANIFEST.txt CHANGED
@@ -1,5 +1,5 @@
1
  tier_a_rerun=1
2
- generated=2026-08-30T05:49:16+00:00
3
  study_root=/workspace/tiera-rerun-fixed-moonep
4
  gbs=16
5
  micro_batch=2
 
1
  tier_a_rerun=1
2
+ generated=2026-08-30T06:20:31+00:00
3
  study_root=/workspace/tiera-rerun-fixed-moonep
4
  gbs=16
5
  micro_batch=2
tiera-rerun-fixed-moonep/echo/a1/a1_oneslot_gbs16/G2__echo__one__seq4096__gbs16__cap1p08203125/nsys/train.log CHANGED
@@ -1104,3 +1104,11 @@ Capture range ended in the application.
1104
  Generating '/tmp/nsys-report-7467.qdstrm'
1105
 
1106
 
 
 
 
 
 
 
 
 
 
1104
  Generating '/tmp/nsys-report-7467.qdstrm'
1105
 
1106
 
1107
+ Connection to Agent lost. This is most likely a bug. Internal reason: 'End of file'.
1108
+ Please refer to the troubleshooting section of the docs:
1109
+ https://docs.nvidia.com/nsight-systems/UserGuide/index.html#troubleshooting
1110
+ Generated:
1111
+ /workspace/tiera-rerun-fixed-moonep/echo/a1/a1_oneslot_gbs16/G2__echo__one__seq4096__gbs16__cap1p08203125/nsys/G2__echo__one__seq4096__gbs16__cap1p08203125.nsys-rep
1112
+ The target application terminated. One or more process it created re-parented.
1113
+ Waiting for termination of re-parented processes.
1114
+ Use the `--wait` option to modify this behavior.
tiera-rerun-fixed-moonep/echo/a1/provenance.txt CHANGED
@@ -1,11 +1,11 @@
1
  ==============================================================
2
  PROVENANCE: echo__a1__gbs16
3
  ==============================================================
4
- captured_at : 2026-08-30T05:49:16+00:00
5
  hostname : sp-aiinfra-interactive-aghosh-7jtjl-8xb200-moe-cg-0
6
  kernel : 6.8.0-1043-nvidia
7
  cwd : /workspace/blackwell_run_scripts
8
- command : run_ladder.sh --phase a1 --gbs 16 --out /workspace/tiera-rerun-fixed-moonep/echo/a1 --nsys
9
 
10
  ---- GPU / driver --------------------------------------------
11
  0, NVIDIA B200, 10.0, 183359 MiB, 580.95.05
@@ -17,74 +17,13 @@ command : run_ladder.sh --phase a1 --gbs 16 --out /workspace/tiera-rerun-f
17
  6, NVIDIA B200, 10.0, 183359 MiB, 580.95.05
18
  7, NVIDIA B200, 10.0, 183359 MiB, 580.95.05
19
  persistence/clocks:
20
- 0, 120 MHz, 1965 MHz, 1000.00 W, 27
21
  1, 120 MHz, 1965 MHz, 1000.00 W, 32
22
  2, 120 MHz, 1965 MHz, 1000.00 W, 32
23
- 3, 120 MHz, 1965 MHz, 1000.00 W, 27
24
  4, 120 MHz, 1965 MHz, 1000.00 W, 29
25
  5, 120 MHz, 1965 MHz, 1000.00 W, 31
26
  6, 120 MHz, 1965 MHz, 1000.00 W, 32
27
  7, 120 MHz, 1965 MHz, 1000.00 W, 28
28
 
29
  NVLink topology:
30
- GPU0 GPU1 GPU2 GPU3 GPU4 GPU5 GPU6 GPU7 NIC0 NIC1 NIC2 NIC3 NIC4 NIC5 NIC6 NIC7 NIC8 CPU Affinity NUMA Affinity GPU NUMA ID
31
- GPU0 X NV18 NV18 NV18 NV18 NV18 NV18 NV18 PIX NODE SYS SYS SYS SYS SYS SYS NODE 0-27 0 N/A
32
- GPU1 NV18 X NV18 NV18 NV18 NV18 NV18 NV18 NODE PIX SYS SYS SYS SYS SYS SYS NODE 0-27 0 N/A
33
- GPU2 NV18 NV18 X NV18 NV18 NV18 NV18 NV18 SYS SYS PIX NODE SYS SYS SYS SYS SYS 28-55 1 N/A
34
- GPU3 NV18 NV18 NV18 X NV18 NV18 NV18 NV18 SYS SYS NODE PIX SYS SYS SYS SYS SYS 28-55 1 N/A
35
- GPU4 NV18 NV18 NV18 NV18 X NV18 NV18 NV18 SYS SYS SYS SYS PIX NODE SYS SYS SYS 56-83 2 N/A
36
- GPU5 NV18 NV18 NV18 NV18 NV18 X NV18 NV18 SYS SYS SYS SYS NODE PIX SYS SYS SYS 56-83 2 N/A
37
- GPU6 NV18 NV18 NV18 NV18 NV18 NV18 X NV18 SYS SYS SYS SYS SYS SYS PIX NODE SYS 84-111 3 N/A
38
- GPU7 NV18 NV18 NV18 NV18 NV18 NV18 NV18 X SYS SYS SYS SYS SYS SYS NODE PIX SYS 84-111 3 N/A
39
- NIC0 PIX NODE SYS SYS SYS SYS SYS SYS X NODE SYS SYS SYS SYS SYS SYS NODE
40
- NIC1 NODE PIX SYS SYS SYS SYS SYS SYS NODE X SYS SYS SYS SYS SYS SYS NODE
41
- NIC2 SYS SYS PIX NODE SYS SYS SYS SYS SYS SYS X NODE SYS SYS SYS SYS SYS
42
- NIC3 SYS SYS NODE PIX SYS SYS SYS SYS SYS SYS NODE X SYS SYS SYS SYS SYS
43
- NIC4 SYS SYS SYS SYS PIX NODE SYS SYS SYS SYS SYS SYS X NODE SYS SYS SYS
44
- unavailable
45
-
46
- ---- toolchain -----------------------------------------------
47
- nvcc : release 12.9, V12.9.86
48
- gcc : gcc (Ubuntu 13.3.0-6ubuntu2~24.04) 13.3.0
49
- python : 3.12.3
50
- torch : 2.8.0a0+5228986c39.nv25.06 cuda=12.9 nccl=2.27.3
51
- torch arch : ['sm_75', 'sm_80', 'sm_86', 'sm_90', 'sm_100', 'sm_120', 'compute_120']
52
- transformer_engine: 2.10.0.dev0+10f6128d [/usr/local/lib/python3.12/dist-packages/transformer_engine/__init__.py]
53
- deep_ep : 1.2.1+34152ae [/workspace/DeepEP/deep_ep/__init__.py]
54
- ultra_ep : 1.0.0 [/workspace/UltraEP/ultra_ep/__init__.py]
55
- moonep : 0.0.1 [/workspace/MoonEP/moonep/__init__.py]
56
- megatron : 0.14.0rc5 [None]
57
- flash_attn : 2.7.4.post1 [/usr/local/lib/python3.12/dist-packages/flash_attn/__init__.py]
58
- numpy : 1.26.4 [/usr/local/lib/python3.12/dist-packages/numpy/__init__.py]
59
- huggingface_hub: 1.29.0 [/usr/local/lib/python3.12/dist-packages/huggingface_hub/__init__.py]
60
-
61
- ---- repositories --------------------------------------------
62
- Megatron-MoE-Echo bracing-regions-through-kernels 9a23679390d7
63
- Megatron-UltraEP ultraep-token-drop-analysis eaf829e7ad21
64
- UltraEP main 94cab099b44f
65
- DeepEP HEAD 34152ae28f80 (+3 untracked: build output)
66
- MoonEP megatron-ddp-symmetric-params d8bc6667fad7
67
- blackwell_run_scripts granularity-echo 9925dccb2715
68
-
69
- ---- environment ---------------------------------------------
70
- CUDA_ARCH_LIST=7.5 8.0 8.6 9.0 10.0 12.0
71
- CUDA_DRIVER_VERSION=575.57.08
72
- CUDA_HOME=/usr/local/cuda
73
- CUDA_MODULE_LOADING=LAZY
74
- CUDA_VERSION=12.9.1.010
75
- HF_REMOTE_ROOT=tiera-rerun-fixed-moonep
76
- NCCL_VERSION=2.27.3
77
- NVSHMEM_VERSION=3.2.5
78
- PYTORCH_BUILD_NUMBER=0
79
- PYTORCH_BUILD_VERSION=2.8.0a0+5228986
80
- PYTORCH_HOME=/opt/pytorch/pytorch
81
- PYTORCH_VERSION=2.8.0a0+5228986
82
- TORCH_ALLOW_TF32_CUBLAS_OVERRIDE=1
83
- TORCH_CUDA_ARCH_LIST=7.5 8.0 8.6 9.0 10.0 12.0+PTX
84
- TORCH_NCCL_USE_COMM_NONBLOCKING=0
85
-
86
- ---- shared library resolution -------------------------------
87
- libnvshmem_host.so: not in ldconfig
88
- libnccl.so: /lib/x86_64-linux-gnu/libnccl.so.2
89
- lrwxrwxrwx 1 root root 20 Aug 29 16:41 /usr/local/lib/python3.12/dist-packages/nvidia/nvshmem/lib/libnvshmem_host.so -> libnvshmem_host.so.3
90
- -rw-r--r-- 1 root root 179045744 Aug 29 16:37 /usr/local/lib/python3.12/dist-packages/nvidia/nvshmem/lib/libnvshmem_host.so.3
 
1
  ==============================================================
2
  PROVENANCE: echo__a1__gbs16
3
  ==============================================================
4
+ captured_at : 2026-08-30T06:20:31+00:00
5
  hostname : sp-aiinfra-interactive-aghosh-7jtjl-8xb200-moe-cg-0
6
  kernel : 6.8.0-1043-nvidia
7
  cwd : /workspace/blackwell_run_scripts
8
+ command : run_ladder.sh --phase a1 --gbs 16 --out /workspace/tiera-rerun-fixed-moonep/echo/a1
9
 
10
  ---- GPU / driver --------------------------------------------
11
  0, NVIDIA B200, 10.0, 183359 MiB, 580.95.05
 
17
  6, NVIDIA B200, 10.0, 183359 MiB, 580.95.05
18
  7, NVIDIA B200, 10.0, 183359 MiB, 580.95.05
19
  persistence/clocks:
20
+ 0, 120 MHz, 1965 MHz, 1000.00 W, 28
21
  1, 120 MHz, 1965 MHz, 1000.00 W, 32
22
  2, 120 MHz, 1965 MHz, 1000.00 W, 32
23
+ 3, 120 MHz, 1965 MHz, 1000.00 W, 28
24
  4, 120 MHz, 1965 MHz, 1000.00 W, 29
25
  5, 120 MHz, 1965 MHz, 1000.00 W, 31
26
  6, 120 MHz, 1965 MHz, 1000.00 W, 32
27
  7, 120 MHz, 1965 MHz, 1000.00 W, 28
28
 
29
  NVLink topology:
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
tiera-rerun-fixed-moonep/hf_watcher.log CHANGED
@@ -18,3 +18,8 @@ https://huggingface.co/MoE-CUDA-Graph-Analysis/tierb-b200-gbs8/tree/main
18
  uploaded: /workspace/tiera-rerun-fixed-moonep -> MoE-CUDA-Graph-Analysis/tierb-b200-gbs8/tiera-rerun-fixed-moonep
19
  https://huggingface.co/MoE-CUDA-Graph-Analysis/tierb-b200-gbs8/tree/main
20
  [2026-08-30T06:09:22+00:00] backup pass
 
 
 
 
 
 
18
  uploaded: /workspace/tiera-rerun-fixed-moonep -> MoE-CUDA-Graph-Analysis/tierb-b200-gbs8/tiera-rerun-fixed-moonep
19
  https://huggingface.co/MoE-CUDA-Graph-Analysis/tierb-b200-gbs8/tree/main
20
  [2026-08-30T06:09:22+00:00] backup pass
21
+ uploaded: /workspace/tiera-rerun-fixed-moonep -> MoE-CUDA-Graph-Analysis/tierb-b200-gbs8/tiera-rerun-fixed-moonep
22
+ https://huggingface.co/MoE-CUDA-Graph-Analysis/tierb-b200-gbs8/tree/main
23
+ Already logged in to Hugging Face as: AbhishekGhosh1998
24
+ Watching /workspace/tiera-rerun-fixed-moonep -> HF every 600s (kill to stop)
25
+ [2026-08-30T06:20:31+00:00] backup pass
tiera-rerun-fixed-moonep/hf_watcher.pid CHANGED
@@ -1 +1 @@
1
- 545120
 
1
+ 588250
tiera-rerun-fixed-moonep/study.log CHANGED
@@ -425,3 +425,15 @@ driver_repo: /workspace/blackwell_run_scripts @ 9925dcc
425
  [guard] iter 12/12 (80s)
426
  [06:06:26] tracing G2__echo__one__seq4096__gbs16__cap1p08203125 (ranks: 0 1 2 3 4 5 6 7)
427
  [guard] iter 8/10 (80s)
 
 
 
 
 
 
 
 
 
 
 
 
 
425
  [guard] iter 12/12 (80s)
426
  [06:06:26] tracing G2__echo__one__seq4096__gbs16__cap1p08203125 (ranks: 0 1 2 3 4 5 6 7)
427
  [guard] iter 8/10 (80s)
428
+ study_root: /workspace/tiera-rerun-fixed-moonep
429
+ gbs: 16
430
+ stages: 3
431
+ nsys: 0
432
+ hf_repo: MoE-CUDA-Graph-Analysis/tierb-b200-gbs8
433
+ hf_remote_root: tiera-rerun-fixed-moonep
434
+ started: 2026-08-30T06:20:31+00:00
435
+ driver_repo: /workspace/blackwell_run_scripts @ ff2c833
436
+
437
+ === STAGE 3 -- Tier A (the causal experiment)
438
+
439
+ === STEP echo__a1__gbs16