AbhishekGhosh1998 commited on
Commit
c6519f8
·
verified ·
1 Parent(s): 6979dd0

granularity results granularity_next_b200/tiera_topk_trace @ 2026-09-21T05:32:24Z from sp-aiinfra-interactive-aghosh-8-mb99k-8xb200-moe-cg-0

Browse files
granularity_next_b200/tiera_topk_trace/hf_watcher.log CHANGED
@@ -36,3 +36,6 @@ https://huggingface.co/MoE-CUDA-Graph-Analysis/tierb-b200-gbs8/tree/main
36
  Already logged in to Hugging Face as: AbhishekGhosh1998
37
  Watching /workspace/granularity_next_b200/tiera_topk_trace -> HF every 600s (kill to stop)
38
  [2026-09-21T05:22:22+00:00] backup pass
 
 
 
 
36
  Already logged in to Hugging Face as: AbhishekGhosh1998
37
  Watching /workspace/granularity_next_b200/tiera_topk_trace -> HF every 600s (kill to stop)
38
  [2026-09-21T05:22:22+00:00] backup pass
39
+ uploaded: /workspace/granularity_next_b200/tiera_topk_trace -> MoE-CUDA-Graph-Analysis/tierb-b200-gbs8/granularity_next_b200/tiera_topk_trace
40
+ https://huggingface.co/MoE-CUDA-Graph-Analysis/tierb-b200-gbs8/tree/main
41
+ [2026-09-21T05:32:24+00:00] backup pass
granularity_next_b200/tiera_topk_trace/study.log CHANGED
@@ -143,3 +143,25 @@ driver_repo: /workspace/blackwell_run_scripts @ b8bd100
143
  === STAGE 9 -- Tier B nsys traces (final, trace-only)
144
 
145
  === STEP ultraep__tierbtrace__gbs16
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
143
  === STAGE 9 -- Tier B nsys traces (final, trace-only)
144
 
145
  === STEP ultraep__tierbtrace__gbs16
146
+ ==============================================================
147
+ CAMPAIGN: ultraep backend=ultraep phase=tierbtrace
148
+ Megatron: /workspace/Megatron-UltraEP @ ultraep-bracing-regions-through-kernels
149
+ GBS=16 seq=4096 repeats=3 nsys=1
150
+ results -> /workspace/granularity_next_b200/tiera_topk_trace/ultraep/tierbtrace
151
+ ==============================================================
152
+ [05:22:36] building ultraep marker extension (log: /workspace/granularity_next_b200/tiera_topk_trace/ultraep/tierbtrace/tierbtrace_gbs16/trace_extension_build.log)
153
+ [05:26:43] marker extension ready: _ultraep_trace_C.cpython-312-x86_64-linux-gnu.so
154
+ [05:26:43] TRACE-ONLY G2_K2__ultraep__matched__seq4096__gbs16__cap1p15625 (ranks: 0 1 2 3 4 5 6 7)
155
+ [guard] iter 6/10 (70s)
156
+ [guard] iter 8/10 (80s)
157
+ [guard] iter 10/10 (120s)
158
+ [guard] nsys stop (flushing report)
159
+ [05:29:15] trace verified -> G2_K2__ultraep__matched__seq4096__gbs16__cap1p15625.nsys-rep (11M, all 8 GPUs + ultraep markers)
160
+ [05:29:15] TRACE-ONLY G3_K2__ultraep__matched__seq4096__gbs16__cap1p1875 (ranks: 0 1 2 3 4 5 6 7)
161
+ [guard] iter 6/10 (70s)
162
+ [guard] iter 8/10 (80s)
163
+ [guard] no progress for 60s at iter 8/10 (past profile window); treating as done
164
+ [guard] nsys stop (flushing report)
165
+ [guard] tearing down process group (outcome=STALLED)
166
+ [05:32:23] trace verified -> G3_K2__ultraep__matched__seq4096__gbs16__cap1p1875.nsys-rep (12M, all 8 GPUs + ultraep markers)
167
+ [05:32:23] TRACE-ONLY G3_K8__ultraep__matched__seq4096__gbs16__cap1p09375 (ranks: 0 1 2 3 4 5 6 7)
granularity_next_b200/tiera_topk_trace/ultraep/tierbtrace/manifest.txt CHANGED
@@ -1,9 +1,9 @@
1
- timestamp: 2026-09-17T03:22:41+00:00
2
- host: hp-aiinfra-interactive-aghosh-8-xzthl-8xb200-moe-cg-0
3
  gpu: NVIDIA B200 x8
4
  driver: 580.95.05
5
  study_scripts_branch: granularity-ultraep
6
- study_scripts_commit: 78aa53a
7
  campaign: ultraep
8
  backend: ultraep
9
  phase: tierbtrace
 
1
+ timestamp: 2026-09-21T05:22:36+00:00
2
+ host: sp-aiinfra-interactive-aghosh-8-mb99k-8xb200-moe-cg-0
3
  gpu: NVIDIA B200 x8
4
  driver: 580.95.05
5
  study_scripts_branch: granularity-ultraep
6
+ study_scripts_commit: 05dd2cb
7
  campaign: ultraep
8
  backend: ultraep
9
  phase: tierbtrace
granularity_next_b200/tiera_topk_trace/ultraep/tierbtrace/pip_freeze.txt CHANGED
@@ -75,7 +75,7 @@ h11==0.16.0
75
  hf-xet==1.6.0
76
  httpcore==1.0.9
77
  httpx==0.28.1
78
- huggingface_hub==1.31.0
79
  hypothesis==6.130.8
80
  idna==3.10
81
  importlib_metadata==8.7.0
 
75
  hf-xet==1.6.0
76
  httpcore==1.0.9
77
  httpx==0.28.1
78
+ huggingface_hub==1.32.0
79
  hypothesis==6.130.8
80
  idna==3.10
81
  importlib_metadata==8.7.0
granularity_next_b200/tiera_topk_trace/ultraep/tierbtrace/provenance.txt CHANGED
@@ -27,3 +27,64 @@ command : run_ladder.sh --phase tierbtrace --gbs 16 --out /workspace/granu
27
  7, 120 MHz, 1965 MHz, 1000.00 W, 28
28
 
29
  NVLink topology:
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
27
  7, 120 MHz, 1965 MHz, 1000.00 W, 28
28
 
29
  NVLink topology:
30
+ GPU0 GPU1 GPU2 GPU3 GPU4 GPU5 GPU6 GPU7 NIC0 NIC1 NIC2 NIC3 NIC4 NIC5 NIC6 NIC7 NIC8 CPU Affinity NUMA Affinity GPU NUMA ID
31
+ GPU0 X NV18 NV18 NV18 NV18 NV18 NV18 NV18 PIX NODE SYS SYS SYS SYS SYS SYS NODE 0-27 0 N/A
32
+ GPU1 NV18 X NV18 NV18 NV18 NV18 NV18 NV18 NODE PIX SYS SYS SYS SYS SYS SYS NODE 0-27 0 N/A
33
+ GPU2 NV18 NV18 X NV18 NV18 NV18 NV18 NV18 SYS SYS PIX NODE SYS SYS SYS SYS SYS 28-55 1 N/A
34
+ GPU3 NV18 NV18 NV18 X NV18 NV18 NV18 NV18 SYS SYS NODE PIX SYS SYS SYS SYS SYS 28-55 1 N/A
35
+ GPU4 NV18 NV18 NV18 NV18 X NV18 NV18 NV18 SYS SYS SYS SYS PIX NODE SYS SYS SYS 56-83 2 N/A
36
+ GPU5 NV18 NV18 NV18 NV18 NV18 X NV18 NV18 SYS SYS SYS SYS NODE PIX SYS SYS SYS 56-83 2 N/A
37
+ GPU6 NV18 NV18 NV18 NV18 NV18 NV18 X NV18 SYS SYS SYS SYS SYS SYS PIX NODE SYS 84-111 3 N/A
38
+ GPU7 NV18 NV18 NV18 NV18 NV18 NV18 NV18 X SYS SYS SYS SYS SYS SYS NODE PIX SYS 84-111 3 N/A
39
+ NIC0 PIX NODE SYS SYS SYS SYS SYS SYS X NODE SYS SYS SYS SYS SYS SYS NODE
40
+ NIC1 NODE PIX SYS SYS SYS SYS SYS SYS NODE X SYS SYS SYS SYS SYS SYS NODE
41
+ NIC2 SYS SYS PIX NODE SYS SYS SYS SYS SYS SYS X NODE SYS SYS SYS SYS SYS
42
+ NIC3 SYS SYS NODE PIX SYS SYS SYS SYS SYS SYS NODE X SYS SYS SYS SYS SYS
43
+ NIC4 SYS SYS SYS SYS PIX NODE SYS SYS SYS SYS SYS SYS X NODE SYS SYS SYS
44
+ unavailable
45
+
46
+ ---- toolchain -----------------------------------------------
47
+ nvcc : release 12.9, V12.9.86
48
+ gcc : gcc (Ubuntu 13.3.0-6ubuntu2~24.04) 13.3.0
49
+ python : 3.12.3
50
+ torch : 2.8.0a0+5228986c39.nv25.06 cuda=12.9 nccl=2.27.3
51
+ torch arch : ['sm_75', 'sm_80', 'sm_86', 'sm_90', 'sm_100', 'sm_120', 'compute_120']
52
+ transformer_engine: 2.10.0.dev0+10f6128d [/usr/local/lib/python3.12/dist-packages/transformer_engine/__init__.py]
53
+ deep_ep : 1.2.1+34152ae [/workspace/DeepEP/deep_ep/__init__.py]
54
+ ultra_ep : 1.0.0 [/workspace/UltraEP/ultra_ep/__init__.py]
55
+ moonep : 0.0.1 [/workspace/MoonEP/moonep/__init__.py]
56
+ megatron : 0.14.0rc5 [None]
57
+ flash_attn : 2.7.4.post1 [/usr/local/lib/python3.12/dist-packages/flash_attn/__init__.py]
58
+ numpy : 1.26.4 [/usr/local/lib/python3.12/dist-packages/numpy/__init__.py]
59
+ huggingface_hub: 1.32.0 [/usr/local/lib/python3.12/dist-packages/huggingface_hub/__init__.py]
60
+
61
+ ---- repositories --------------------------------------------
62
+ Megatron-MoE-Echo bracing-regions-through-kernels 9a23679390d7
63
+ Megatron-UltraEP ultraep-bracing-regions-through-kernels 03d57ef7d476
64
+ UltraEP main 94cab099b44f
65
+ DeepEP HEAD 34152ae28f80 (+3 untracked: build output)
66
+ MoonEP megatron-ddp-symmetric-params d8bc6667fad7
67
+ blackwell_run_scripts granularity-ultraep 05dd2cbfb1eb
68
+
69
+ ---- environment ---------------------------------------------
70
+ CUDA_ARCH_LIST=7.5 8.0 8.6 9.0 10.0 12.0
71
+ CUDA_DRIVER_VERSION=575.57.08
72
+ CUDA_HOME=/usr/local/cuda
73
+ CUDA_MODULE_LOADING=LAZY
74
+ CUDA_VERSION=12.9.1.010
75
+ HF_REMOTE_ROOT=granularity_next_b200/tiera_topk_trace
76
+ NCCL_VERSION=2.27.3
77
+ NVSHMEM_VERSION=3.2.5
78
+ PYTORCH_BUILD_NUMBER=0
79
+ PYTORCH_BUILD_VERSION=2.8.0a0+5228986
80
+ PYTORCH_HOME=/opt/pytorch/pytorch
81
+ PYTORCH_VERSION=2.8.0a0+5228986
82
+ TORCH_ALLOW_TF32_CUBLAS_OVERRIDE=1
83
+ TORCH_CUDA_ARCH_LIST=7.5 8.0 8.6 9.0 10.0 12.0+PTX
84
+ TORCH_NCCL_USE_COMM_NONBLOCKING=0
85
+
86
+ ---- shared library resolution -------------------------------
87
+ libnvshmem_host.so: not in ldconfig
88
+ libnccl.so: /lib/x86_64-linux-gnu/libnccl.so.2
89
+ lrwxrwxrwx 1 root root 20 Sep 21 04:57 /usr/local/lib/python3.12/dist-packages/nvidia/nvshmem/lib/libnvshmem_host.so -> libnvshmem_host.so.3
90
+ -rw-r--r-- 1 root root 179045744 Sep 21 04:54 /usr/local/lib/python3.12/dist-packages/nvidia/nvshmem/lib/libnvshmem_host.so.3
granularity_next_b200/tiera_topk_trace/ultraep/tierbtrace/tierbtrace_gbs16/G2_K2__ultraep__matched__seq4096__gbs16__cap1p15625/nsys/G2_K2__ultraep__matched__seq4096__gbs16__cap1p15625.nsys-rep CHANGED
@@ -1,3 +1,3 @@
1
  version https://git-lfs.github.com/spec/v1
2
- oid sha256:16d8b184e95b4cc82e363b6b774e38586cdc69ef72bb35e09082e92a2190d7f6
3
- size 10833175
 
1
  version https://git-lfs.github.com/spec/v1
2
+ oid sha256:95b9bff1008473c2a14f43f44b047326c6ad5bb7512c6ef71354c84edee07304
3
+ size 10756575
granularity_next_b200/tiera_topk_trace/ultraep/tierbtrace/tierbtrace_gbs16/G2_K2__ultraep__matched__seq4096__gbs16__cap1p15625/nsys/train.log CHANGED
@@ -6,12 +6,10 @@
6
  memory_dump=none snapshots=/workspace/granularity_next_b200/tiera_topk_trace/ultraep/tierbtrace/tierbtrace_gbs16/G2_K2__ultraep__matched__seq4096__gbs16__cap1p15625/nsys/memory_snapshots/snapshot.pickle
7
  repo=/workspace/Megatron-UltraEP @ 03d57ef7
8
  ==============================================================
9
- W0917 03:26:59.698000 948326 torch/distributed/run.py:766]
10
- W0917 03:26:59.698000 948326 torch/distributed/run.py:766] *****************************************
11
- W0917 03:26:59.698000 948326 torch/distributed/run.py:766] Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed.
12
- W0917 03:26:59.698000 948326 torch/distributed/run.py:766] *****************************************
13
- /workspace/Megatron-UltraEP/megatron/training/arguments.py:831: UserWarning: Disabling sequence parallelism because tensor model parallelism is disabled
14
- warnings.warn("Disabling sequence parallelism because tensor model parallelism is disabled")
15
  using world size: 8, data-parallel size: 8, context-parallel size: 1, hierarchical context-parallel sizes: None, tensor-model-parallel size: 1, pipeline-model-parallel size: 1
16
  WARNING: overriding default arguments for tokenizer_type:GPT2BPETokenizer with tokenizer_type:GPT2BPETokenizer
17
  Number of virtual stages per pipeline stage: None
@@ -572,58 +570,60 @@ WARNING:megatron.core.rerun_state_machine:RerunStateMachine initialized in mode
572
  warnings.warn("Disabling sequence parallelism because tensor model parallelism is disabled")
573
  /workspace/Megatron-UltraEP/megatron/training/arguments.py:831: UserWarning: Disabling sequence parallelism because tensor model parallelism is disabled
574
  warnings.warn("Disabling sequence parallelism because tensor model parallelism is disabled")
 
575
  /workspace/Megatron-UltraEP/megatron/training/arguments.py:831: UserWarning: Disabling sequence parallelism because tensor model parallelism is disabled
576
  warnings.warn("Disabling sequence parallelism because tensor model parallelism is disabled")
577
  /workspace/Megatron-UltraEP/megatron/training/arguments.py:831: UserWarning: Disabling sequence parallelism because tensor model parallelism is disabled
578
  warnings.warn("Disabling sequence parallelism because tensor model parallelism is disabled")
579
- WARNING: one_logger package is required to enable e2e metrics tracking. please go to https://confluence.nvidia.com/display/MLWFO/Package+Repositories for details to install it
580
  /workspace/Megatron-UltraEP/megatron/training/arguments.py:831: UserWarning: Disabling sequence parallelism because tensor model parallelism is disabled
581
  warnings.warn("Disabling sequence parallelism because tensor model parallelism is disabled")
582
- [Gloo] Rank 1 is connected to 7 peer ranks. Expected number of connected peer ranks is : 7
583
- [Gloo] Rank 2 is connected to 7 peer ranks. Expected number of connected peer ranks is : 7
 
 
584
  [Gloo] Rank 3 is connected to 7 peer ranks. Expected number of connected peer ranks is : 7
 
585
  [Gloo] Rank 4 is connected to 7 peer ranks. Expected number of connected peer ranks is : 7
586
- [Gloo] Rank 7[Gloo] Rank is connected to 7 peer ranks. Expected number of connected peer ranks is : 7
587
- 6 is connected to 7 peer ranks. Expected number of connected peer ranks is : 7
588
- [Gloo] Rank 0 is connected to 7 peer ranks. Expected number of connected peer ranks is : 7
589
  [Gloo] Rank 5 is connected to 7 peer ranks. Expected number of connected peer ranks is : 7
 
 
590
  [Gloo] Rank 0 is connected to 7 peer ranks. Expected number of connected peer ranks is : 7
591
  [Gloo] Rank 1 is connected to 7 peer ranks. Expected number of connected peer ranks is : 7
592
- [Gloo] Rank [Gloo] Rank 45 is connected to is connected to 77 peer ranks. peer ranks. Expected number of connected peer ranks is : 7Expected number of connected peer ranks is : 7
593
-
594
  [Gloo] Rank 2 is connected to 7 peer ranks. Expected number of connected peer ranks is : 7
595
  [Gloo] Rank 3 is connected to 7 peer ranks. Expected number of connected peer ranks is : 7
596
- [Gloo] Rank [Gloo] Rank 76 is connected to is connected to 77 peer ranks. peer ranks. Expected number of connected peer ranks is : Expected number of connected peer ranks is : 77
597
-
598
- [Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
599
- [Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
600
- [Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
601
- [Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
602
- [Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
603
  [Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
604
  > initialized tensor model parallel with size 1
605
  > initialized pipeline model parallel with size 1
606
  > setting random seeds to 1234 ...
607
  [Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
608
  [Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
609
- [rank3]:[W917 03:27:12.759074705 ProcessGroupNCCL.cpp:4789] [PG ID 0 PG GUID 0 Rank 3] using GPU 3 as device used by this process is currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect. You can specify device_id in init_process_group() to force use of a particular device.
610
- [rank1]:[W917 03:27:12.759291933 ProcessGroupNCCL.cpp:4789] [PG ID 0 PG GUID 0 Rank 1] using GPU 1 as device used by this process is currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect. You can specify device_id in init_process_group() to force use of a particular device.
611
- [rank5]:[W917 03:27:12.759731800 ProcessGroupNCCL.cpp:4789] [PG ID 0 PG GUID 0 Rank 5] using GPU 5 as device used by this process is currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect. You can specify device_id in init_process_group() to force use of a particular device.
612
- [rank6]:[W917 03:27:12.760213509 ProcessGroupNCCL.cpp:4789] [PG ID 0 PG GUID 0 Rank 6] using GPU 6 as device used by this process is currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect. You can specify device_id in init_process_group() to force use of a particular device.
613
- [rank2]:[W917 03:27:12.763662209 ProcessGroupNCCL.cpp:4789] [PG ID 0 PG GUID 0 Rank 2] using GPU 2 as device used by this process is currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect. You can specify device_id in init_process_group() to force use of a particular device.
614
- [rank4]:[W917 03:27:12.763770125 ProcessGroupNCCL.cpp:4789] [PG ID 0 PG GUID 0 Rank 4] using GPU 4 as device used by this process is currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect. You can specify device_id in init_process_group() to force use of a particular device.
615
- [rank7]:[W917 03:27:12.763810340 ProcessGroupNCCL.cpp:4789] [PG ID 0 PG GUID 0 Rank 7] using GPU 7 as device used by this process is currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect. You can specify device_id in init_process_group() to force use of a particular device.
 
 
 
 
 
616
  > compiling dataset index builder ...
617
  make: Entering directory '/workspace/Megatron-UltraEP/megatron/core/datasets'
618
  make: Nothing to be done for 'default'.
619
  make: Leaving directory '/workspace/Megatron-UltraEP/megatron/core/datasets'
620
- >>> done with dataset index builder. Compilation time: 0.437 seconds
621
  WARNING: constraints for invoking optimized fused softmax kernel are not met. We default back to unfused kernel invocations.
622
  > compiling and loading fused kernels ...
623
- [rank0]:[W917 03:27:13.386048103 ProcessGroupNCCL.cpp:4789] [PG ID 0 PG GUID 0 Rank 0] using GPU 0 as device used by this process is currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect. You can specify device_id in init_process_group() to force use of a particular device.
624
- >>> done with compiling and loading fused kernels. Compilation time: 4.013 seconds
625
- time to initialize megatron (seconds): 9.439
626
- [after megatron is initialized] datetime: 2026-09-17 03:27:19
627
  building GPT model ...
628
  /usr/local/lib/python3.12/dist-packages/transformer_engine/pytorch/cpu_offload.py:708: DeprecationWarning: Offloading weights is deprecated. Using offload_weights=True does not have any effect.
629
  warnings.warn(
@@ -667,11 +667,11 @@ INFO:megatron.core.distributed.param_and_grad_buffer:Number of buckets for gradi
667
  Params for bucket 1 (206045184 elements, 206045184 padded size):
668
  module.output_layer.weight
669
  Params for bucket 2 (42086400 elements, 42086400 padded size):
 
670
  module.decoder.layers.15.self_attention.linear_qkv.weight
671
  module.decoder.layers.15.pre_mlp_layernorm.weight
672
- module.decoder.final_layernorm.weight
673
- module.decoder.layers.15.self_attention.linear_qkv.layer_norm_weight
674
  module.decoder.layers.15.mlp.router.weight
 
675
  module.decoder.layers.15.self_attention.linear_proj.weight
676
  Params for bucket 3 (42082304 elements, 42082304 padded size):
677
  module.decoder.layers.14.self_attention.linear_qkv.weight
@@ -680,29 +680,29 @@ Params for bucket 3 (42082304 elements, 42082304 padded size):
680
  module.decoder.layers.14.self_attention.linear_proj.weight
681
  module.decoder.layers.14.self_attention.linear_qkv.layer_norm_weight
682
  Params for bucket 4 (42082304 elements, 42082304 padded size):
683
- module.decoder.layers.13.pre_mlp_layernorm.weight
684
  module.decoder.layers.13.self_attention.linear_qkv.weight
 
685
  module.decoder.layers.13.mlp.router.weight
686
- module.decoder.layers.13.self_attention.linear_proj.weight
687
  module.decoder.layers.13.self_attention.linear_qkv.layer_norm_weight
 
688
  Params for bucket 5 (42082304 elements, 42082304 padded size):
689
- module.decoder.layers.12.self_attention.linear_qkv.weight
690
- module.decoder.layers.12.pre_mlp_layernorm.weight
691
  module.decoder.layers.12.mlp.router.weight
692
  module.decoder.layers.12.self_attention.linear_proj.weight
693
  module.decoder.layers.12.self_attention.linear_qkv.layer_norm_weight
 
 
694
  Params for bucket 6 (42082304 elements, 42082304 padded size):
695
- module.decoder.layers.11.self_attention.linear_qkv.weight
696
  module.decoder.layers.11.pre_mlp_layernorm.weight
 
697
  module.decoder.layers.11.mlp.router.weight
698
- module.decoder.layers.11.self_attention.linear_proj.weight
699
  module.decoder.layers.11.self_attention.linear_qkv.layer_norm_weight
 
700
  Params for bucket 7 (42082304 elements, 42082304 padded size):
701
  module.decoder.layers.10.self_attention.linear_qkv.weight
702
  module.decoder.layers.10.pre_mlp_layernorm.weight
703
  module.decoder.layers.10.mlp.router.weight
704
- module.decoder.layers.10.self_attention.linear_proj.weight
705
  module.decoder.layers.10.self_attention.linear_qkv.layer_norm_weight
 
706
  Params for bucket 8 (42082304 elements, 42082304 padded size):
707
  module.decoder.layers.9.self_attention.linear_qkv.weight
708
  module.decoder.layers.9.pre_mlp_layernorm.weight
@@ -716,60 +716,60 @@ Params for bucket 9 (42082304 elements, 42082304 padded size):
716
  module.decoder.layers.8.self_attention.linear_qkv.layer_norm_weight
717
  module.decoder.layers.8.self_attention.linear_proj.weight
718
  Params for bucket 10 (42082304 elements, 42082304 padded size):
 
 
719
  module.decoder.layers.7.mlp.router.weight
720
  module.decoder.layers.7.self_attention.linear_proj.weight
721
  module.decoder.layers.7.self_attention.linear_qkv.layer_norm_weight
722
- module.decoder.layers.7.self_attention.linear_qkv.weight
723
- module.decoder.layers.7.pre_mlp_layernorm.weight
724
  Params for bucket 11 (42082304 elements, 42082304 padded size):
725
  module.decoder.layers.6.mlp.router.weight
726
- module.decoder.layers.6.self_attention.linear_proj.weight
727
  module.decoder.layers.6.self_attention.linear_qkv.layer_norm_weight
 
728
  module.decoder.layers.6.self_attention.linear_qkv.weight
729
  module.decoder.layers.6.pre_mlp_layernorm.weight
730
  Params for bucket 12 (42082304 elements, 42082304 padded size):
 
 
731
  module.decoder.layers.5.self_attention.linear_qkv.weight
732
  module.decoder.layers.5.self_attention.linear_proj.weight
733
  module.decoder.layers.5.pre_mlp_layernorm.weight
734
- module.decoder.layers.5.mlp.router.weight
735
- module.decoder.layers.5.self_attention.linear_qkv.layer_norm_weight
736
  Params for bucket 13 (42082304 elements, 42082304 padded size):
 
 
737
  module.decoder.layers.4.mlp.router.weight
738
  module.decoder.layers.4.self_attention.linear_proj.weight
739
  module.decoder.layers.4.self_attention.linear_qkv.layer_norm_weight
740
- module.decoder.layers.4.self_attention.linear_qkv.weight
741
- module.decoder.layers.4.pre_mlp_layernorm.weight
742
  Params for bucket 14 (42082304 elements, 42082304 padded size):
 
 
743
  module.decoder.layers.3.mlp.router.weight
744
  module.decoder.layers.3.self_attention.linear_proj.weight
745
  module.decoder.layers.3.self_attention.linear_qkv.layer_norm_weight
746
- module.decoder.layers.3.self_attention.linear_qkv.weight
747
- module.decoder.layers.3.pre_mlp_layernorm.weight
748
  Params for bucket 15 (42082304 elements, 42082304 padded size):
 
 
749
  module.decoder.layers.2.mlp.router.weight
750
  module.decoder.layers.2.self_attention.linear_proj.weight
751
  module.decoder.layers.2.self_attention.linear_qkv.layer_norm_weight
752
- module.decoder.layers.2.pre_mlp_layernorm.weight
753
- module.decoder.layers.2.self_attention.linear_qkv.weight
754
  Params for bucket 16 (42082304 elements, 42082304 padded size):
755
- module.decoder.layers.1.self_attention.linear_qkv.layer_norm_weight
756
- module.decoder.layers.1.self_attention.linear_proj.weight
757
- module.decoder.layers.1.mlp.router.weight
758
  module.decoder.layers.1.self_attention.linear_qkv.weight
759
  module.decoder.layers.1.pre_mlp_layernorm.weight
 
 
 
760
  Params for bucket 17 (42082304 elements, 42082304 padded size):
761
- module.decoder.layers.0.pre_mlp_layernorm.weight
762
  module.decoder.layers.0.self_attention.linear_proj.weight
 
763
  module.decoder.layers.0.mlp.router.weight
764
- module.decoder.layers.0.self_attention.linear_qkv.weight
765
  module.decoder.layers.0.self_attention.linear_qkv.layer_norm_weight
766
  Params for bucket 18 (206045184 elements, 206045184 padded size):
767
  module.embedding.word_embeddings.weight
768
  INFO:megatron.core.distributed.param_and_grad_buffer:Number of buckets for gradient all-reduce / reduce-scatter: 56
769
  Params for bucket 1 (44040192 elements, 44040192 padded size):
770
  module.decoder.layers.15.mlp.experts.linear_fc2.weight3
771
- module.decoder.layers.15.mlp.experts.linear_fc2.weight1
772
  module.decoder.layers.15.mlp.experts.linear_fc2.weight2
 
773
  Params for bucket 2 (44040192 elements, 44040192 padded size):
774
  module.decoder.layers.15.mlp.experts.linear_fc2.weight0
775
  module.decoder.layers.15.mlp.experts.linear_fc1.weight3
@@ -781,8 +781,8 @@ Params for bucket 4 (44040192 elements, 44040192 padded size):
781
  module.decoder.layers.14.mlp.experts.linear_fc2.weight3
782
  Params for bucket 5 (44040192 elements, 44040192 padded size):
783
  module.decoder.layers.14.mlp.experts.linear_fc2.weight2
784
- module.decoder.layers.14.mlp.experts.linear_fc2.weight0
785
  module.decoder.layers.14.mlp.experts.linear_fc2.weight1
 
786
  Params for bucket 6 (58720256 elements, 58720256 padded size):
787
  module.decoder.layers.14.mlp.experts.linear_fc1.weight3
788
  module.decoder.layers.14.mlp.experts.linear_fc1.weight2
@@ -827,8 +827,8 @@ Params for bucket 18 (44040192 elements, 44040192 padded size):
827
  module.decoder.layers.10.mlp.experts.linear_fc2.weight3
828
  Params for bucket 19 (44040192 elements, 44040192 padded size):
829
  module.decoder.layers.10.mlp.experts.linear_fc2.weight2
830
- module.decoder.layers.10.mlp.experts.linear_fc2.weight1
831
  module.decoder.layers.10.mlp.experts.linear_fc2.weight0
 
832
  Params for bucket 20 (58720256 elements, 58720256 padded size):
833
  module.decoder.layers.10.mlp.experts.linear_fc1.weight3
834
  module.decoder.layers.10.mlp.experts.linear_fc1.weight2
@@ -850,8 +850,8 @@ Params for bucket 25 (44040192 elements, 44040192 padded size):
850
  module.decoder.layers.8.mlp.experts.linear_fc2.weight3
851
  Params for bucket 26 (44040192 elements, 44040192 padded size):
852
  module.decoder.layers.8.mlp.experts.linear_fc2.weight2
853
- module.decoder.layers.8.mlp.experts.linear_fc2.weight1
854
  module.decoder.layers.8.mlp.experts.linear_fc2.weight0
 
855
  Params for bucket 27 (58720256 elements, 58720256 padded size):
856
  module.decoder.layers.8.mlp.experts.linear_fc1.weight3
857
  module.decoder.layers.8.mlp.experts.linear_fc1.weight2
@@ -883,8 +883,8 @@ Params for bucket 35 (58720256 elements, 58720256 padded size):
883
  module.decoder.layers.6.mlp.experts.linear_fc1.weight0
884
  Params for bucket 36 (44040192 elements, 44040192 padded size):
885
  module.decoder.layers.5.mlp.experts.linear_fc2.weight3
886
- module.decoder.layers.5.mlp.experts.linear_fc2.weight1
887
  module.decoder.layers.5.mlp.experts.linear_fc2.weight2
 
888
  Params for bucket 37 (44040192 elements, 44040192 padded size):
889
  module.decoder.layers.5.mlp.experts.linear_fc2.weight0
890
  module.decoder.layers.5.mlp.experts.linear_fc1.weight3
@@ -906,8 +906,8 @@ Params for bucket 42 (58720256 elements, 58720256 padded size):
906
  module.decoder.layers.4.mlp.experts.linear_fc1.weight0
907
  Params for bucket 43 (44040192 elements, 44040192 padded size):
908
  module.decoder.layers.3.mlp.experts.linear_fc2.weight3
909
- module.decoder.layers.3.mlp.experts.linear_fc2.weight1
910
  module.decoder.layers.3.mlp.experts.linear_fc2.weight2
 
911
  Params for bucket 44 (44040192 elements, 44040192 padded size):
912
  module.decoder.layers.3.mlp.experts.linear_fc2.weight0
913
  module.decoder.layers.3.mlp.experts.linear_fc1.weight3
@@ -942,17 +942,17 @@ Params for bucket 53 (44040192 elements, 44040192 padded size):
942
  module.decoder.layers.0.mlp.experts.linear_fc2.weight3
943
  Params for bucket 54 (44040192 elements, 44040192 padded size):
944
  module.decoder.layers.0.mlp.experts.linear_fc2.weight2
945
- module.decoder.layers.0.mlp.experts.linear_fc2.weight0
946
  module.decoder.layers.0.mlp.experts.linear_fc2.weight1
 
947
  Params for bucket 55 (58720256 elements, 58720256 padded size):
948
  module.decoder.layers.0.mlp.experts.linear_fc1.weight3
949
  module.decoder.layers.0.mlp.experts.linear_fc1.weight2
950
  Params for bucket 56 (58720256 elements, 58720256 padded size):
951
  module.decoder.layers.0.mlp.experts.linear_fc1.weight1
952
  module.decoder.layers.0.mlp.experts.linear_fc1.weight0
953
- INFO:megatron.core.optimizer:Setting up optimizer with config OptimizerConfig(optimizer='adam', lr=0.0001, min_lr=1e-05, decoupled_lr=None, decoupled_min_lr=None, weight_decay=0.1, fp8_recipe='mxfp8', fp16=False, bf16=True, reuse_grad_buf_for_mxfp8_param_ag=False, params_dtype=torch.bfloat16, use_precision_aware_optimizer=False, store_param_remainders=True, main_grads_dtype=torch.float32, main_params_dtype=torch.float32, exp_avg_dtype=torch.float32, exp_avg_sq_dtype=torch.float32, loss_scale=None, initial_loss_scale=4294967296, min_loss_scale=1.0, loss_scale_window=1000, hysteresis=2, adam_beta1=0.9, adam_beta2=0.999, adam_eps=1e-08, sgd_momentum=0.9, use_distributed_optimizer=True, overlap_param_gather_with_optimizer_step=False, optimizer_cpu_offload=False, optimizer_offload_fraction=1.0, use_torch_optimizer_for_cpu_offload=False, overlap_cpu_optimizer_d2h_h2d=False, pin_cpu_grads=True, pin_cpu_params=True, clip_grad=1.0, log_num_zeros_in_grad=False, barrier_with_L1_time=True, timers=<megatron.core.timers.Timers object at 0x7de4a8c9d520>, config_logger_dir='')
954
  INFO:megatron.core.optimizer_param_scheduler:> learning rate decay style: cosine
955
- [after model, optimizer, and learning rate scheduler are built] datetime: 2026-09-17 03:27:22
956
  > building train, validation, and test datasets ...
957
  > datasets target sizes (minimum size):
958
  train: 160
@@ -960,27 +960,25 @@ INFO:megatron.core.optimizer_param_scheduler:> learning rate decay style: cosine
960
  test: 0
961
  INFO:megatron.core.datasets.blended_megatron_dataset_config:Let split_matrix = [(0, 1.0), None, None]
962
  > building train, validation, and test datasets for GPT ...
963
- INFO:megatron.core.datasets.blended_megatron_dataset_builder:Building GPTDataset splits with sizes=(160, 0, 0) and config=GPTDatasetConfig(random_seed=1234, sequence_length=4096, blend=(['/workspace/data/wikitext/wikitext103_gpt2_text_document'], None), blend_per_split=None, split='100,0,0', split_matrix=[(0, 1.0), None, None], num_dataset_builder_threads=1, path_to_cache=None, mmap_bin_files=True, mock=False, tokenizer=<megatron.training.tokenizer.tokenizer._GPT2BPETokenizer object at 0x7de4abd3b290>, mid_level_dataset_surplus=0.005, reset_position_ids=False, reset_attention_mask=False, eod_mask_loss=False, create_attention_mask=True, drop_last_partial_validation_sequence=True, add_extra_token_to_sequence=True, object_storage_cache_path=None)
964
  INFO:megatron.core.datasets.indexed_dataset:Load the _IndexReader from /workspace/data/wikitext/wikitext103_gpt2_text_document.idx
965
  INFO:megatron.core.datasets.indexed_dataset: Extract the sequence lengths
966
  INFO:megatron.core.datasets.indexed_dataset: Extract the sequence pointers
967
  INFO:megatron.core.datasets.indexed_dataset: Extract the document indices
968
  INFO:megatron.core.datasets.indexed_dataset:> total number of sequences: 1165029
969
  INFO:megatron.core.datasets.indexed_dataset:> total number of documents: 1801350
970
- INFO:megatron.core.datasets.gpt_dataset:Load the GPTDataset train indices
971
- INFO:megatron.core.datasets.gpt_dataset: Load the document index from 68684a515e118253a8572c4517cf2ffe-GPTDataset-train-document_index.npy
972
- INFO:megatron.core.datasets.gpt_dataset: Load the sample index from 68684a515e118253a8572c4517cf2ffe-GPTDataset-train-sample_index.npy
973
- INFO:megatron.core.datasets.gpt_dataset: Load the shuffle index from 68684a515e118253a8572c4517cf2ffe-GPTDataset-train-shuffle_index.npy
974
  INFO:megatron.core.datasets.gpt_dataset:> total number of samples: 29073
 
975
  > finished creating GPT datasets ...
976
- [after dataloaders are built] datetime: 2026-09-17 03:27:22
977
  done with setup ...
978
  (min, max) time across ranks (ms):
979
- model-and-optimizer-setup ......................: (3347.95, 3351.05)
980
- train/valid/test-data-iterators-setup ..........: (113.53, 139.33)
981
  training ...
982
  Overwriting rerun_state_machine.current_iteration from -1 to 0...
983
- [before the start of training step] datetime: 2026-09-17 03:27:23
984
  /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:334: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
985
  warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
986
  /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:336: UserWarning: The use_host_meta is deprecated, it will be removed in the future.
@@ -1035,16 +1033,16 @@ Overwriting rerun_state_machine.current_iteration from -1 to 0...
1035
  warnings.warn("The use_host_meta is deprecated, it will be removed in the future.")
1036
  /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:334: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
1037
  warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
1038
- /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:336: UserWarning: The use_host_meta is deprecated, it will be removed in the future.
1039
- warnings.warn("The use_host_meta is deprecated, it will be removed in the future.")
1040
  /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:334: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
1041
  warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
1042
- /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:336: UserWarning: The use_host_meta is deprecated, it will be removed in the future.
1043
- warnings.warn("The use_host_meta is deprecated, it will be removed in the future.")
1044
  /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:334: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
1045
  warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
1046
  /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:336: UserWarning: The use_host_meta is deprecated, it will be removed in the future.
1047
  warnings.warn("The use_host_meta is deprecated, it will be removed in the future.")
 
 
 
 
1048
  /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:334: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
1049
  warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
1050
  /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:336: UserWarning: The use_host_meta is deprecated, it will be removed in the future.
@@ -1109,39 +1107,50 @@ Overwriting rerun_state_machine.current_iteration from -1 to 0...
1109
  warnings.warn("The use_host_meta is deprecated, it will be removed in the future.")
1110
  /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:336: UserWarning: The use_host_meta is deprecated, it will be removed in the future.
1111
  warnings.warn("The use_host_meta is deprecated, it will be removed in the future.")
1112
- [2026-09-17 03:27:52] iteration 1/ 10 | consumed samples: 16 | elapsed time per iteration (ms): 29813.9 | throughput per GPU (TFLOP/s/GPU): 4.2 | learning rate: 2.000000E-05 | global batch size: 16 | lm loss: 1.102902E+01 | loss scale: 1.0 | grad norm: 44.186 | number of skipped iterations: 0 | number of nan iterations: 0 | peak recv tokens: 8832 | min safe capacity: 1.0781 |
1113
  Number of parameters in transformer block in billions: 23.22
1114
  Number of parameters in embedding layers in billions: 0.41
1115
  Total number of parameters in billions: 23.63
1116
  Number of parameters in most loaded shard in billions: 23.6320
1117
  Theoretical memory footprints: weight and optimizer=169029.43 MB
1118
- [Rank 0] (after 1 iterations) memory (MB) | allocated: 58124.0458984375 | max allocated: 65718.16796875 | reserved: 79458.0 | max reserved: 79458.0
1119
- [2026-09-17 03:27:53] iteration 2/ 10 | consumed samples: 32 | elapsed time per iteration (ms): 586.2 | throughput per GPU (TFLOP/s/GPU): 214.2 | learning rate: 4.000000E-05 | global batch size: 16 | lm loss: 1.101823E+01 | loss scale: 1.0 | grad norm: 45.729 | number of skipped iterations: 0 | number of nan iterations: 0 | peak recv tokens: 8832 | min safe capacity: 1.0781 |
1120
- [2026-09-17 03:27:53] iteration 3/ 10 | consumed samples: 48 | elapsed time per iteration (ms): 434.7 | throughput per GPU (TFLOP/s/GPU): 288.9 | learning rate: 6.000000E-05 | global batch size: 16 | lm loss: 9.972256E+00 | loss scale: 1.0 | grad norm: 157.296 | number of skipped iterations: 0 | number of nan iterations: 0 | peak recv tokens: 8832 | min safe capacity: 1.0781 |
1121
  Capture CUDA graph for training!!!
1122
  CUDA graph capture done!!!
1123
- [2026-09-17 03:27:57] iteration 4/ 10 | consumed samples: 64 | elapsed time per iteration (ms): 3115.4 | throughput per GPU (TFLOP/s/GPU): 40.3 | learning rate: 8.000000E-05 | global batch size: 16 | lm loss: 1.311298E+01 | loss scale: 1.0 | grad norm: 41.251 | number of skipped iterations: 0 | number of nan iterations: 0 | peak recv tokens: 8832 | min safe capacity: 1.0781 |
1124
- [2026-09-17 03:27:57] iteration 5/ 10 | consumed samples: 80 | elapsed time per iteration (ms): 323.9 | throughput per GPU (TFLOP/s/GPU): 387.7 | learning rate: 1.000000E-04 | global batch size: 16 | lm loss: 1.040733E+01 | loss scale: 1.0 | grad norm: 25.625 | number of skipped iterations: 0 | number of nan iterations: 0 | peak recv tokens: 8832 | min safe capacity: 1.0781 |
1125
- [2026-09-17 03:27:57] iteration 6/ 10 | consumed samples: 96 | elapsed time per iteration (ms): 333.2 | throughput per GPU (TFLOP/s/GPU): 376.9 | learning rate: 7.750000E-05 | global batch size: 16 | lm loss: 1.025381E+01 | loss scale: 1.0 | grad norm: 19.172 | number of skipped iterations: 0 | number of nan iterations: 0 | peak recv tokens: 8832 | min safe capacity: 1.0781 |
1126
- [2026-09-17 03:28:00] iteration 7/ 10 | consumed samples: 112 | elapsed time per iteration (ms): 3161.7 | throughput per GPU (TFLOP/s/GPU): 39.7 | learning rate: 3.250000E-05 | global batch size: 16 | lm loss: 1.002502E+01 | loss scale: 1.0 | grad norm: 8.664 | number of skipped iterations: 0 | number of nan iterations: 0 | peak recv tokens: 8832 | min safe capacity: 1.0781 |
1127
- [2026-09-17 03:28:01] iteration 8/ 10 | consumed samples: 128 | elapsed time per iteration (ms): 342.7 | throughput per GPU (TFLOP/s/GPU): 366.4 | learning rate: 1.000000E-05 | global batch size: 16 | lm loss: 9.464363E+00 | loss scale: 1.0 | grad norm: 8.108 | number of skipped iterations: 0 | number of nan iterations: 0 | peak recv tokens: 8832 | min safe capacity: 1.0781 |
1128
  Capture range started in the application.
1129
  Capture range ended in the application.
1130
- Generating '/tmp/nsys-report-2929.qdstrm'
1131
 
1132
-
 
 
 
 
 
 
 
 
 
 
 
1133
  Collection stop is not allowed in this state.
1134
- W0917 03:29:32.760000 948326 torch/distributed/elastic/agent/server/api.py:719] Received 15 death signal, shutting down workers
1135
- W0917 03:29:32.761000 948326 torch/distributed/elastic/multiprocessing/api.py:900] Sending process 948649 closing signal SIGTERM
1136
- Generated:
1137
- /workspace/granularity_next_b200/tiera_topk_trace/ultraep/tierbtrace/tierbtrace_gbs16/G2_K2__ultraep__matched__seq4096__gbs16__cap1p15625/nsys/G2_K2__ultraep__matched__seq4096__gbs16__cap1p15625.nsys-rep
1138
- W0917 03:29:32.765000 948326 torch/distributed/elastic/multiprocessing/api.py:900] Sending process 948650 closing signal SIGTERM
1139
- W0917 03:29:32.772000 948326 torch/distributed/elastic/multiprocessing/api.py:900] Sending process 948651 closing signal SIGTERM
1140
- W0917 03:29:32.773000 948326 torch/distributed/elastic/multiprocessing/api.py:900] Sending process 948652 closing signal SIGTERM
1141
- W0917 03:29:32.777000 948326 torch/distributed/elastic/multiprocessing/api.py:900] Sending process 948653 closing signal SIGTERM
1142
- W0917 03:29:32.783000 948326 torch/distributed/elastic/multiprocessing/api.py:900] Sending process 948654 closing signal SIGTERM
1143
- W0917 03:29:32.794000 948326 torch/distributed/elastic/multiprocessing/api.py:900] Sending process 948655 closing signal SIGTERM
1144
- W0917 03:29:32.804000 948326 torch/distributed/elastic/multiprocessing/api.py:900] Sending process 948656 closing signal SIGTERM
1145
  Traceback (most recent call last):
1146
  File "/usr/local/bin/torchrun", line 8, in <module>
1147
  sys.exit(main())
@@ -1156,21 +1165,23 @@ Traceback (most recent call last):
1156
  File "/usr/local/lib/python3.12/dist-packages/torch/distributed/launcher/api.py", line 139, in __call__
1157
  return launch_agent(self._config, self._entrypoint, list(args))
1158
  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
1159
- File "/usr/local/lib/python3.12/dist-packages/torch/distributed/launcher/api.py", line 261, in launch_agent
1160
- result = agent.run()
1161
- ^^^^^^^^^^^
1162
- File "/usr/local/lib/python3.12/dist-packages/torch/distributed/elastic/metrics/api.py", line 138, in wrapper
1163
- result = f(*args, **kwargs)
1164
- ^^^^^^^^^^^^^^^^^^
1165
- File "/usr/local/lib/python3.12/dist-packages/torch/distributed/elastic/agent/server/api.py", line 711, in run
1166
- result = self._invoke_run(role)
1167
- ^^^^^^^^^^^^^^^^^^^^^^
1168
- File "/usr/local/lib/python3.12/dist-packages/torch/distributed/elastic/agent/server/api.py", line 870, in _invoke_run
1169
- time.sleep(monitor_interval)
1170
- File "/usr/local/lib/python3.12/dist-packages/torch/distributed/elastic/multiprocessing/api.py", line 84, in _terminate_process_handler
1171
- raise SignalException(f"Process {os.getpid()} got signal: {sigval}", sigval=sigval)
1172
- torch.distributed.elastic.multiprocessing.api.SignalException: Process 948326 got signal: 15
1173
- The target application terminated. One or more process it created re-parented.
1174
- Waiting for termination of re-parented processes.
1175
- Use the `--wait` option to modify this behavior.
1176
- There are no active sessions.
 
 
 
6
  memory_dump=none snapshots=/workspace/granularity_next_b200/tiera_topk_trace/ultraep/tierbtrace/tierbtrace_gbs16/G2_K2__ultraep__matched__seq4096__gbs16__cap1p15625/nsys/memory_snapshots/snapshot.pickle
7
  repo=/workspace/Megatron-UltraEP @ 03d57ef7
8
  ==============================================================
9
+ W0921 05:26:51.825000 24330 torch/distributed/run.py:766]
10
+ W0921 05:26:51.825000 24330 torch/distributed/run.py:766] *****************************************
11
+ W0921 05:26:51.825000 24330 torch/distributed/run.py:766] Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed.
12
+ W0921 05:26:51.825000 24330 torch/distributed/run.py:766] *****************************************
 
 
13
  using world size: 8, data-parallel size: 8, context-parallel size: 1, hierarchical context-parallel sizes: None, tensor-model-parallel size: 1, pipeline-model-parallel size: 1
14
  WARNING: overriding default arguments for tokenizer_type:GPT2BPETokenizer with tokenizer_type:GPT2BPETokenizer
15
  Number of virtual stages per pipeline stage: None
 
570
  warnings.warn("Disabling sequence parallelism because tensor model parallelism is disabled")
571
  /workspace/Megatron-UltraEP/megatron/training/arguments.py:831: UserWarning: Disabling sequence parallelism because tensor model parallelism is disabled
572
  warnings.warn("Disabling sequence parallelism because tensor model parallelism is disabled")
573
+ WARNING: one_logger package is required to enable e2e metrics tracking. please go to https://confluence.nvidia.com/display/MLWFO/Package+Repositories for details to install it
574
  /workspace/Megatron-UltraEP/megatron/training/arguments.py:831: UserWarning: Disabling sequence parallelism because tensor model parallelism is disabled
575
  warnings.warn("Disabling sequence parallelism because tensor model parallelism is disabled")
576
  /workspace/Megatron-UltraEP/megatron/training/arguments.py:831: UserWarning: Disabling sequence parallelism because tensor model parallelism is disabled
577
  warnings.warn("Disabling sequence parallelism because tensor model parallelism is disabled")
 
578
  /workspace/Megatron-UltraEP/megatron/training/arguments.py:831: UserWarning: Disabling sequence parallelism because tensor model parallelism is disabled
579
  warnings.warn("Disabling sequence parallelism because tensor model parallelism is disabled")
580
+ /workspace/Megatron-UltraEP/megatron/training/arguments.py:831: UserWarning: Disabling sequence parallelism because tensor model parallelism is disabled
581
+ warnings.warn("Disabling sequence parallelism because tensor model parallelism is disabled")
582
+ [Gloo] Rank [Gloo] Rank 2 is connected to 70 peer ranks. is connected to Expected number of connected peer ranks is : 77 peer ranks. Expected number of connected peer ranks is : 7
583
+
584
  [Gloo] Rank 3 is connected to 7 peer ranks. Expected number of connected peer ranks is : 7
585
+ [Gloo] Rank 1 is connected to 7 peer ranks. Expected number of connected peer ranks is : 7
586
  [Gloo] Rank 4 is connected to 7 peer ranks. Expected number of connected peer ranks is : 7
 
 
 
587
  [Gloo] Rank 5 is connected to 7 peer ranks. Expected number of connected peer ranks is : 7
588
+ [Gloo] Rank 6 is connected to 7 peer ranks. Expected number of connected peer ranks is : 7
589
+ [Gloo] Rank 7 is connected to 7 peer ranks. Expected number of connected peer ranks is : 7
590
  [Gloo] Rank 0 is connected to 7 peer ranks. Expected number of connected peer ranks is : 7
591
  [Gloo] Rank 1 is connected to 7 peer ranks. Expected number of connected peer ranks is : 7
 
 
592
  [Gloo] Rank 2 is connected to 7 peer ranks. Expected number of connected peer ranks is : 7
593
  [Gloo] Rank 3 is connected to 7 peer ranks. Expected number of connected peer ranks is : 7
594
+ [Gloo] Rank 4 is connected to 7 peer ranks. Expected number of connected peer ranks is : 7
595
+ [Gloo] Rank 5 is connected to 7 peer ranks. Expected number of connected peer ranks is : 7
596
+ [Gloo] Rank 7 is connected to 7 peer ranks. Expected number of connected peer ranks is : 7[Gloo] Rank
597
+ 6 is connected to 7 peer ranks. Expected number of connected peer ranks is : 7
 
 
 
598
  [Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
599
  > initialized tensor model parallel with size 1
600
  > initialized pipeline model parallel with size 1
601
  > setting random seeds to 1234 ...
602
  [Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
603
  [Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
604
+ [Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0[Gloo] Rank
605
+ 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
606
+ [Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
607
+ [Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
608
+ [Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
609
+ [rank3]:[W921 05:27:04.261394981 ProcessGroupNCCL.cpp:4789] [PG ID 0 PG GUID 0 Rank 3] using GPU 3 as device used by this process is currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect. You can specify device_id in init_process_group() to force use of a particular device.
610
+ [rank6]:[W921 05:27:04.262518453 ProcessGroupNCCL.cpp:4789] [PG ID 0 PG GUID 0 Rank 6] using GPU 6 as device used by this process is currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect. You can specify device_id in init_process_group() to force use of a particular device.
611
+ [rank2]:[W921 05:27:04.262639934 ProcessGroupNCCL.cpp:4789] [PG ID 0 PG GUID 0 Rank 2] using GPU 2 as device used by this process is currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect. You can specify device_id in init_process_group() to force use of a particular device.
612
+ [rank5]:[W921 05:27:04.262843243 ProcessGroupNCCL.cpp:4789] [PG ID 0 PG GUID 0 Rank 5] using GPU 5 as device used by this process is currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect. You can specify device_id in init_process_group() to force use of a particular device.
613
+ [rank1]:[W921 05:27:04.262845359 ProcessGroupNCCL.cpp:4789] [PG ID 0 PG GUID 0 Rank 1] using GPU 1 as device used by this process is currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect. You can specify device_id in init_process_group() to force use of a particular device.
614
+ [rank7]:[W921 05:27:04.262903971 ProcessGroupNCCL.cpp:4789] [PG ID 0 PG GUID 0 Rank 7] using GPU 7 as device used by this process is currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect. You can specify device_id in init_process_group() to force use of a particular device.
615
+ [rank4]:[W921 05:27:04.262939575 ProcessGroupNCCL.cpp:4789] [PG ID 0 PG GUID 0 Rank 4] using GPU 4 as device used by this process is currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect. You can specify device_id in init_process_group() to force use of a particular device.
616
  > compiling dataset index builder ...
617
  make: Entering directory '/workspace/Megatron-UltraEP/megatron/core/datasets'
618
  make: Nothing to be done for 'default'.
619
  make: Leaving directory '/workspace/Megatron-UltraEP/megatron/core/datasets'
620
+ >>> done with dataset index builder. Compilation time: 0.425 seconds
621
  WARNING: constraints for invoking optimized fused softmax kernel are not met. We default back to unfused kernel invocations.
622
  > compiling and loading fused kernels ...
623
+ [rank0]:[W921 05:27:04.862236191 ProcessGroupNCCL.cpp:4789] [PG ID 0 PG GUID 0 Rank 0] using GPU 0 as device used by this process is currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect. You can specify device_id in init_process_group() to force use of a particular device.
624
+ >>> done with compiling and loading fused kernels. Compilation time: 3.498 seconds
625
+ time to initialize megatron (seconds): 10.193
626
+ [after megatron is initialized] datetime: 2026-09-21 05:27:12
627
  building GPT model ...
628
  /usr/local/lib/python3.12/dist-packages/transformer_engine/pytorch/cpu_offload.py:708: DeprecationWarning: Offloading weights is deprecated. Using offload_weights=True does not have any effect.
629
  warnings.warn(
 
667
  Params for bucket 1 (206045184 elements, 206045184 padded size):
668
  module.output_layer.weight
669
  Params for bucket 2 (42086400 elements, 42086400 padded size):
670
+ module.decoder.final_layernorm.weight
671
  module.decoder.layers.15.self_attention.linear_qkv.weight
672
  module.decoder.layers.15.pre_mlp_layernorm.weight
 
 
673
  module.decoder.layers.15.mlp.router.weight
674
+ module.decoder.layers.15.self_attention.linear_qkv.layer_norm_weight
675
  module.decoder.layers.15.self_attention.linear_proj.weight
676
  Params for bucket 3 (42082304 elements, 42082304 padded size):
677
  module.decoder.layers.14.self_attention.linear_qkv.weight
 
680
  module.decoder.layers.14.self_attention.linear_proj.weight
681
  module.decoder.layers.14.self_attention.linear_qkv.layer_norm_weight
682
  Params for bucket 4 (42082304 elements, 42082304 padded size):
 
683
  module.decoder.layers.13.self_attention.linear_qkv.weight
684
+ module.decoder.layers.13.pre_mlp_layernorm.weight
685
  module.decoder.layers.13.mlp.router.weight
 
686
  module.decoder.layers.13.self_attention.linear_qkv.layer_norm_weight
687
+ module.decoder.layers.13.self_attention.linear_proj.weight
688
  Params for bucket 5 (42082304 elements, 42082304 padded size):
 
 
689
  module.decoder.layers.12.mlp.router.weight
690
  module.decoder.layers.12.self_attention.linear_proj.weight
691
  module.decoder.layers.12.self_attention.linear_qkv.layer_norm_weight
692
+ module.decoder.layers.12.self_attention.linear_qkv.weight
693
+ module.decoder.layers.12.pre_mlp_layernorm.weight
694
  Params for bucket 6 (42082304 elements, 42082304 padded size):
 
695
  module.decoder.layers.11.pre_mlp_layernorm.weight
696
+ module.decoder.layers.11.self_attention.linear_qkv.weight
697
  module.decoder.layers.11.mlp.router.weight
 
698
  module.decoder.layers.11.self_attention.linear_qkv.layer_norm_weight
699
+ module.decoder.layers.11.self_attention.linear_proj.weight
700
  Params for bucket 7 (42082304 elements, 42082304 padded size):
701
  module.decoder.layers.10.self_attention.linear_qkv.weight
702
  module.decoder.layers.10.pre_mlp_layernorm.weight
703
  module.decoder.layers.10.mlp.router.weight
 
704
  module.decoder.layers.10.self_attention.linear_qkv.layer_norm_weight
705
+ module.decoder.layers.10.self_attention.linear_proj.weight
706
  Params for bucket 8 (42082304 elements, 42082304 padded size):
707
  module.decoder.layers.9.self_attention.linear_qkv.weight
708
  module.decoder.layers.9.pre_mlp_layernorm.weight
 
716
  module.decoder.layers.8.self_attention.linear_qkv.layer_norm_weight
717
  module.decoder.layers.8.self_attention.linear_proj.weight
718
  Params for bucket 10 (42082304 elements, 42082304 padded size):
719
+ module.decoder.layers.7.pre_mlp_layernorm.weight
720
+ module.decoder.layers.7.self_attention.linear_qkv.weight
721
  module.decoder.layers.7.mlp.router.weight
722
  module.decoder.layers.7.self_attention.linear_proj.weight
723
  module.decoder.layers.7.self_attention.linear_qkv.layer_norm_weight
 
 
724
  Params for bucket 11 (42082304 elements, 42082304 padded size):
725
  module.decoder.layers.6.mlp.router.weight
 
726
  module.decoder.layers.6.self_attention.linear_qkv.layer_norm_weight
727
+ module.decoder.layers.6.self_attention.linear_proj.weight
728
  module.decoder.layers.6.self_attention.linear_qkv.weight
729
  module.decoder.layers.6.pre_mlp_layernorm.weight
730
  Params for bucket 12 (42082304 elements, 42082304 padded size):
731
+ module.decoder.layers.5.mlp.router.weight
732
+ module.decoder.layers.5.self_attention.linear_qkv.layer_norm_weight
733
  module.decoder.layers.5.self_attention.linear_qkv.weight
734
  module.decoder.layers.5.self_attention.linear_proj.weight
735
  module.decoder.layers.5.pre_mlp_layernorm.weight
 
 
736
  Params for bucket 13 (42082304 elements, 42082304 padded size):
737
+ module.decoder.layers.4.self_attention.linear_qkv.weight
738
+ module.decoder.layers.4.pre_mlp_layernorm.weight
739
  module.decoder.layers.4.mlp.router.weight
740
  module.decoder.layers.4.self_attention.linear_proj.weight
741
  module.decoder.layers.4.self_attention.linear_qkv.layer_norm_weight
 
 
742
  Params for bucket 14 (42082304 elements, 42082304 padded size):
743
+ module.decoder.layers.3.self_attention.linear_qkv.weight
744
+ module.decoder.layers.3.pre_mlp_layernorm.weight
745
  module.decoder.layers.3.mlp.router.weight
746
  module.decoder.layers.3.self_attention.linear_proj.weight
747
  module.decoder.layers.3.self_attention.linear_qkv.layer_norm_weight
 
 
748
  Params for bucket 15 (42082304 elements, 42082304 padded size):
749
+ module.decoder.layers.2.self_attention.linear_qkv.weight
750
+ module.decoder.layers.2.pre_mlp_layernorm.weight
751
  module.decoder.layers.2.mlp.router.weight
752
  module.decoder.layers.2.self_attention.linear_proj.weight
753
  module.decoder.layers.2.self_attention.linear_qkv.layer_norm_weight
 
 
754
  Params for bucket 16 (42082304 elements, 42082304 padded size):
 
 
 
755
  module.decoder.layers.1.self_attention.linear_qkv.weight
756
  module.decoder.layers.1.pre_mlp_layernorm.weight
757
+ module.decoder.layers.1.mlp.router.weight
758
+ module.decoder.layers.1.self_attention.linear_qkv.layer_norm_weight
759
+ module.decoder.layers.1.self_attention.linear_proj.weight
760
  Params for bucket 17 (42082304 elements, 42082304 padded size):
761
+ module.decoder.layers.0.self_attention.linear_qkv.weight
762
  module.decoder.layers.0.self_attention.linear_proj.weight
763
+ module.decoder.layers.0.pre_mlp_layernorm.weight
764
  module.decoder.layers.0.mlp.router.weight
 
765
  module.decoder.layers.0.self_attention.linear_qkv.layer_norm_weight
766
  Params for bucket 18 (206045184 elements, 206045184 padded size):
767
  module.embedding.word_embeddings.weight
768
  INFO:megatron.core.distributed.param_and_grad_buffer:Number of buckets for gradient all-reduce / reduce-scatter: 56
769
  Params for bucket 1 (44040192 elements, 44040192 padded size):
770
  module.decoder.layers.15.mlp.experts.linear_fc2.weight3
 
771
  module.decoder.layers.15.mlp.experts.linear_fc2.weight2
772
+ module.decoder.layers.15.mlp.experts.linear_fc2.weight1
773
  Params for bucket 2 (44040192 elements, 44040192 padded size):
774
  module.decoder.layers.15.mlp.experts.linear_fc2.weight0
775
  module.decoder.layers.15.mlp.experts.linear_fc1.weight3
 
781
  module.decoder.layers.14.mlp.experts.linear_fc2.weight3
782
  Params for bucket 5 (44040192 elements, 44040192 padded size):
783
  module.decoder.layers.14.mlp.experts.linear_fc2.weight2
 
784
  module.decoder.layers.14.mlp.experts.linear_fc2.weight1
785
+ module.decoder.layers.14.mlp.experts.linear_fc2.weight0
786
  Params for bucket 6 (58720256 elements, 58720256 padded size):
787
  module.decoder.layers.14.mlp.experts.linear_fc1.weight3
788
  module.decoder.layers.14.mlp.experts.linear_fc1.weight2
 
827
  module.decoder.layers.10.mlp.experts.linear_fc2.weight3
828
  Params for bucket 19 (44040192 elements, 44040192 padded size):
829
  module.decoder.layers.10.mlp.experts.linear_fc2.weight2
 
830
  module.decoder.layers.10.mlp.experts.linear_fc2.weight0
831
+ module.decoder.layers.10.mlp.experts.linear_fc2.weight1
832
  Params for bucket 20 (58720256 elements, 58720256 padded size):
833
  module.decoder.layers.10.mlp.experts.linear_fc1.weight3
834
  module.decoder.layers.10.mlp.experts.linear_fc1.weight2
 
850
  module.decoder.layers.8.mlp.experts.linear_fc2.weight3
851
  Params for bucket 26 (44040192 elements, 44040192 padded size):
852
  module.decoder.layers.8.mlp.experts.linear_fc2.weight2
 
853
  module.decoder.layers.8.mlp.experts.linear_fc2.weight0
854
+ module.decoder.layers.8.mlp.experts.linear_fc2.weight1
855
  Params for bucket 27 (58720256 elements, 58720256 padded size):
856
  module.decoder.layers.8.mlp.experts.linear_fc1.weight3
857
  module.decoder.layers.8.mlp.experts.linear_fc1.weight2
 
883
  module.decoder.layers.6.mlp.experts.linear_fc1.weight0
884
  Params for bucket 36 (44040192 elements, 44040192 padded size):
885
  module.decoder.layers.5.mlp.experts.linear_fc2.weight3
 
886
  module.decoder.layers.5.mlp.experts.linear_fc2.weight2
887
+ module.decoder.layers.5.mlp.experts.linear_fc2.weight1
888
  Params for bucket 37 (44040192 elements, 44040192 padded size):
889
  module.decoder.layers.5.mlp.experts.linear_fc2.weight0
890
  module.decoder.layers.5.mlp.experts.linear_fc1.weight3
 
906
  module.decoder.layers.4.mlp.experts.linear_fc1.weight0
907
  Params for bucket 43 (44040192 elements, 44040192 padded size):
908
  module.decoder.layers.3.mlp.experts.linear_fc2.weight3
 
909
  module.decoder.layers.3.mlp.experts.linear_fc2.weight2
910
+ module.decoder.layers.3.mlp.experts.linear_fc2.weight1
911
  Params for bucket 44 (44040192 elements, 44040192 padded size):
912
  module.decoder.layers.3.mlp.experts.linear_fc2.weight0
913
  module.decoder.layers.3.mlp.experts.linear_fc1.weight3
 
942
  module.decoder.layers.0.mlp.experts.linear_fc2.weight3
943
  Params for bucket 54 (44040192 elements, 44040192 padded size):
944
  module.decoder.layers.0.mlp.experts.linear_fc2.weight2
 
945
  module.decoder.layers.0.mlp.experts.linear_fc2.weight1
946
+ module.decoder.layers.0.mlp.experts.linear_fc2.weight0
947
  Params for bucket 55 (58720256 elements, 58720256 padded size):
948
  module.decoder.layers.0.mlp.experts.linear_fc1.weight3
949
  module.decoder.layers.0.mlp.experts.linear_fc1.weight2
950
  Params for bucket 56 (58720256 elements, 58720256 padded size):
951
  module.decoder.layers.0.mlp.experts.linear_fc1.weight1
952
  module.decoder.layers.0.mlp.experts.linear_fc1.weight0
953
+ INFO:megatron.core.optimizer:Setting up optimizer with config OptimizerConfig(optimizer='adam', lr=0.0001, min_lr=1e-05, decoupled_lr=None, decoupled_min_lr=None, weight_decay=0.1, fp8_recipe='mxfp8', fp16=False, bf16=True, reuse_grad_buf_for_mxfp8_param_ag=False, params_dtype=torch.bfloat16, use_precision_aware_optimizer=False, store_param_remainders=True, main_grads_dtype=torch.float32, main_params_dtype=torch.float32, exp_avg_dtype=torch.float32, exp_avg_sq_dtype=torch.float32, loss_scale=None, initial_loss_scale=4294967296, min_loss_scale=1.0, loss_scale_window=1000, hysteresis=2, adam_beta1=0.9, adam_beta2=0.999, adam_eps=1e-08, sgd_momentum=0.9, use_distributed_optimizer=True, overlap_param_gather_with_optimizer_step=False, optimizer_cpu_offload=False, optimizer_offload_fraction=1.0, use_torch_optimizer_for_cpu_offload=False, overlap_cpu_optimizer_d2h_h2d=False, pin_cpu_grads=True, pin_cpu_params=True, clip_grad=1.0, log_num_zeros_in_grad=False, barrier_with_L1_time=True, timers=<megatron.core.timers.Timers object at 0x70226fbc4e60>, config_logger_dir='')
954
  INFO:megatron.core.optimizer_param_scheduler:> learning rate decay style: cosine
955
+ [after model, optimizer, and learning rate scheduler are built] datetime: 2026-09-21 05:27:16
956
  > building train, validation, and test datasets ...
957
  > datasets target sizes (minimum size):
958
  train: 160
 
960
  test: 0
961
  INFO:megatron.core.datasets.blended_megatron_dataset_config:Let split_matrix = [(0, 1.0), None, None]
962
  > building train, validation, and test datasets for GPT ...
963
+ INFO:megatron.core.datasets.blended_megatron_dataset_builder:Building GPTDataset splits with sizes=(160, 0, 0) and config=GPTDatasetConfig(random_seed=1234, sequence_length=4096, blend=(['/workspace/data/wikitext/wikitext103_gpt2_text_document'], None), blend_per_split=None, split='100,0,0', split_matrix=[(0, 1.0), None, None], num_dataset_builder_threads=1, path_to_cache=None, mmap_bin_files=True, mock=False, tokenizer=<megatron.training.tokenizer.tokenizer._GPT2BPETokenizer object at 0x702272ce3a40>, mid_level_dataset_surplus=0.005, reset_position_ids=False, reset_attention_mask=False, eod_mask_loss=False, create_attention_mask=True, drop_last_partial_validation_sequence=True, add_extra_token_to_sequence=True, object_storage_cache_path=None)
964
  INFO:megatron.core.datasets.indexed_dataset:Load the _IndexReader from /workspace/data/wikitext/wikitext103_gpt2_text_document.idx
965
  INFO:megatron.core.datasets.indexed_dataset: Extract the sequence lengths
966
  INFO:megatron.core.datasets.indexed_dataset: Extract the sequence pointers
967
  INFO:megatron.core.datasets.indexed_dataset: Extract the document indices
968
  INFO:megatron.core.datasets.indexed_dataset:> total number of sequences: 1165029
969
  INFO:megatron.core.datasets.indexed_dataset:> total number of documents: 1801350
970
+ INFO:megatron.core.datasets.gpt_dataset:Build and save the GPTDataset train indices
 
 
 
971
  INFO:megatron.core.datasets.gpt_dataset:> total number of samples: 29073
972
+ INFO:megatron.core.datasets.gpt_dataset:> total number of epochs: 1
973
  > finished creating GPT datasets ...
974
+ [after dataloaders are built] datetime: 2026-09-21 05:27:16
975
  done with setup ...
976
  (min, max) time across ranks (ms):
977
+ model-and-optimizer-setup ......................: (3579.39, 3581.84)
978
+ train/valid/test-data-iterators-setup ..........: (168.94, 187.22)
979
  training ...
980
  Overwriting rerun_state_machine.current_iteration from -1 to 0...
981
+ [before the start of training step] datetime: 2026-09-21 05:27:16
982
  /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:334: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
983
  warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
984
  /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:336: UserWarning: The use_host_meta is deprecated, it will be removed in the future.
 
1033
  warnings.warn("The use_host_meta is deprecated, it will be removed in the future.")
1034
  /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:334: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
1035
  warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
 
 
1036
  /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:334: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
1037
  warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
 
 
1038
  /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:334: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
1039
  warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
1040
  /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:336: UserWarning: The use_host_meta is deprecated, it will be removed in the future.
1041
  warnings.warn("The use_host_meta is deprecated, it will be removed in the future.")
1042
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:336: UserWarning: The use_host_meta is deprecated, it will be removed in the future.
1043
+ warnings.warn("The use_host_meta is deprecated, it will be removed in the future.")
1044
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:336: UserWarning: The use_host_meta is deprecated, it will be removed in the future.
1045
+ warnings.warn("The use_host_meta is deprecated, it will be removed in the future.")
1046
  /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:334: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
1047
  warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
1048
  /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:336: UserWarning: The use_host_meta is deprecated, it will be removed in the future.
 
1107
  warnings.warn("The use_host_meta is deprecated, it will be removed in the future.")
1108
  /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:336: UserWarning: The use_host_meta is deprecated, it will be removed in the future.
1109
  warnings.warn("The use_host_meta is deprecated, it will be removed in the future.")
1110
+ [2026-09-21 05:27:49] iteration 1/ 10 | consumed samples: 16 | elapsed time per iteration (ms): 33028.6 | throughput per GPU (TFLOP/s/GPU): 3.8 | learning rate: 2.000000E-05 | global batch size: 16 | lm loss: 1.102902E+01 | loss scale: 1.0 | grad norm: 44.184 | number of skipped iterations: 0 | number of nan iterations: 0 | peak recv tokens: 8832 | min safe capacity: 1.0781 |
1111
  Number of parameters in transformer block in billions: 23.22
1112
  Number of parameters in embedding layers in billions: 0.41
1113
  Total number of parameters in billions: 23.63
1114
  Number of parameters in most loaded shard in billions: 23.6320
1115
  Theoretical memory footprints: weight and optimizer=169029.43 MB
1116
+ [Rank 0] (after 1 iterations) memory (MB) | allocated: 58124.0458984375 | max allocated: 65718.0 | reserved: 79284.0 | max reserved: 79284.0
1117
+ [2026-09-21 05:27:50] iteration 2/ 10 | consumed samples: 32 | elapsed time per iteration (ms): 564.2 | throughput per GPU (TFLOP/s/GPU): 222.6 | learning rate: 4.000000E-05 | global batch size: 16 | lm loss: 1.101823E+01 | loss scale: 1.0 | grad norm: 45.733 | number of skipped iterations: 0 | number of nan iterations: 0 | peak recv tokens: 8832 | min safe capacity: 1.0781 |
1118
+ [2026-09-21 05:27:50] iteration 3/ 10 | consumed samples: 48 | elapsed time per iteration (ms): 437.4 | throughput per GPU (TFLOP/s/GPU): 287.1 | learning rate: 6.000000E-05 | global batch size: 16 | lm loss: 9.971716E+00 | loss scale: 1.0 | grad norm: 157.169 | number of skipped iterations: 0 | number of nan iterations: 0 | peak recv tokens: 8832 | min safe capacity: 1.0781 |
1119
  Capture CUDA graph for training!!!
1120
  CUDA graph capture done!!!
1121
+ [2026-09-21 05:27:53] iteration 4/ 10 | consumed samples: 64 | elapsed time per iteration (ms): 3034.2 | throughput per GPU (TFLOP/s/GPU): 41.4 | learning rate: 8.000000E-05 | global batch size: 16 | lm loss: 1.311224E+01 | loss scale: 1.0 | grad norm: 41.298 | number of skipped iterations: 0 | number of nan iterations: 0 | peak recv tokens: 8832 | min safe capacity: 1.0781 |
1122
+ [2026-09-21 05:27:53] iteration 5/ 10 | consumed samples: 80 | elapsed time per iteration (ms): 325.6 | throughput per GPU (TFLOP/s/GPU): 385.6 | learning rate: 1.000000E-04 | global batch size: 16 | lm loss: 1.040747E+01 | loss scale: 1.0 | grad norm: 25.670 | number of skipped iterations: 0 | number of nan iterations: 0 | peak recv tokens: 8832 | min safe capacity: 1.0781 |
1123
+ [2026-09-21 05:27:54] iteration 6/ 10 | consumed samples: 96 | elapsed time per iteration (ms): 335.0 | throughput per GPU (TFLOP/s/GPU): 374.9 | learning rate: 7.750000E-05 | global batch size: 16 | lm loss: 1.025583E+01 | loss scale: 1.0 | grad norm: 19.185 | number of skipped iterations: 0 | number of nan iterations: 0 | peak recv tokens: 8832 | min safe capacity: 1.0781 |
1124
+ [2026-09-21 05:27:57] iteration 7/ 10 | consumed samples: 112 | elapsed time per iteration (ms): 3243.3 | throughput per GPU (TFLOP/s/GPU): 38.7 | learning rate: 3.250000E-05 | global batch size: 16 | lm loss: 1.002962E+01 | loss scale: 1.0 | grad norm: 8.693 | number of skipped iterations: 0 | number of nan iterations: 0 | peak recv tokens: 8832 | min safe capacity: 1.0781 |
1125
+ [2026-09-21 05:27:57] iteration 8/ 10 | consumed samples: 128 | elapsed time per iteration (ms): 343.8 | throughput per GPU (TFLOP/s/GPU): 365.2 | learning rate: 1.000000E-05 | global batch size: 16 | lm loss: 9.465693E+00 | loss scale: 1.0 | grad norm: 7.851 | number of skipped iterations: 0 | number of nan iterations: 0 | peak recv tokens: 8832 | min safe capacity: 1.0781 |
1126
  Capture range started in the application.
1127
  Capture range ended in the application.
1128
+ Generating '/tmp/nsys-report-2397.qdstrm'
1129
 
1130
+
1131
+ [2026-09-21 05:28:40] iteration 9/ 10 | consumed samples: 144 | elapsed time per iteration (ms): 42919.9 | throughput per GPU (TFLOP/s/GPU): 2.9 | learning rate: 1.000000E-05 | global batch size: 16 | lm loss: 9.198565E+00 | loss scale: 1.0 | grad norm: 7.259 | number of skipped iterations: 0 | number of nan iterations: 0 | peak recv tokens: 8832 | min safe capacity: 1.0781 |
1132
+ [2026-09-21 05:28:41] iteration 10/ 10 | consumed samples: 160 | elapsed time per iteration (ms): 363.9 | throughput per GPU (TFLOP/s/GPU): 345.1 | learning rate: 1.000000E-05 | global batch size: 16 | lm loss: 9.043821E+00 | loss scale: 1.0 | grad norm: 6.923 | number of skipped iterations: 0 | number of nan iterations: 0 | peak recv tokens: 8832 | min safe capacity: 1.0781 |
1133
+ [after training is done] datetime: 2026-09-21 05:28:41
1134
+ [rank0]:[W921 05:28:42.332826642 ProcessGroupNCCL.cpp:1505] Warning: WARNING: destroy_process_group() was not called before program exit, which can leak resources. For more info, please see https://pytorch.org/docs/stable/distributed.html#shutdown (function operator())
1135
+ [rank1]:[W921 05:28:43.605743009 ProcessGroupNCCL.cpp:1505] Warning: WARNING: destroy_process_group() was not called before program exit, which can leak resources. For more info, please see https://pytorch.org/docs/stable/distributed.html#shutdown (function operator())
1136
+ [rank4]:[W921 05:28:43.608715326 ProcessGroupNCCL.cpp:1505] Warning: WARNING: destroy_process_group() was not called before program exit, which can leak resources. For more info, please see https://pytorch.org/docs/stable/distributed.html#shutdown (function operator())
1137
+ [rank5]:[W921 05:28:43.624970738 ProcessGroupNCCL.cpp:1505] Warning: WARNING: destroy_process_group() was not called before program exit, which can leak resources. For more info, please see https://pytorch.org/docs/stable/distributed.html#shutdown (function operator())
1138
+ [rank6]:[W921 05:28:44.043986247 ProcessGroupNCCL.cpp:1505] Warning: WARNING: destroy_process_group() was not called before program exit, which can leak resources. For more info, please see https://pytorch.org/docs/stable/distributed.html#shutdown (function operator())
1139
+ [rank2]:[W921 05:28:44.074015347 ProcessGroupNCCL.cpp:1505] Warning: WARNING: destroy_process_group() was not called before program exit, which can leak resources. For more info, please see https://pytorch.org/docs/stable/distributed.html#shutdown (function operator())
1140
+ [rank3]:[W921 05:28:44.075061371 ProcessGroupNCCL.cpp:1505] Warning: WARNING: destroy_process_group() was not called before program exit, which can leak resources. For more info, please see https://pytorch.org/docs/stable/distributed.html#shutdown (function operator())
1141
+ [rank7]:[W921 05:28:44.524534589 ProcessGroupNCCL.cpp:1505] Warning: WARNING: destroy_process_group() was not called before program exit, which can leak resources. For more info, please see https://pytorch.org/docs/stable/distributed.html#shutdown (function operator())
1142
  Collection stop is not allowed in this state.
1143
+ [rank0]:[W921 05:28:46.626373583 observer.cpp:116] Warning: Leaked callback handle: 1 (function operator())
1144
+ pure virtual method called
1145
+ terminate called without an active exception
1146
+ W0921 05:28:47.337000 24330 torch/distributed/elastic/multiprocessing/api.py:900] Sending process 24442 closing signal SIGTERM
1147
+ W0921 05:28:47.338000 24330 torch/distributed/elastic/multiprocessing/api.py:900] Sending process 24443 closing signal SIGTERM
1148
+ W0921 05:28:47.338000 24330 torch/distributed/elastic/multiprocessing/api.py:900] Sending process 24444 closing signal SIGTERM
1149
+ W0921 05:28:47.338000 24330 torch/distributed/elastic/multiprocessing/api.py:900] Sending process 24445 closing signal SIGTERM
1150
+ W0921 05:28:47.339000 24330 torch/distributed/elastic/multiprocessing/api.py:900] Sending process 24446 closing signal SIGTERM
1151
+ W0921 05:28:47.339000 24330 torch/distributed/elastic/multiprocessing/api.py:900] Sending process 24447 closing signal SIGTERM
1152
+ W0921 05:28:47.340000 24330 torch/distributed/elastic/multiprocessing/api.py:900] Sending process 24448 closing signal SIGTERM
1153
+ E0921 05:28:49.885000 24330 torch/distributed/elastic/multiprocessing/api.py:874] failed (exitcode: -6) local_rank: 0 (pid: 24441) of binary: /usr/bin/python
1154
  Traceback (most recent call last):
1155
  File "/usr/local/bin/torchrun", line 8, in <module>
1156
  sys.exit(main())
 
1165
  File "/usr/local/lib/python3.12/dist-packages/torch/distributed/launcher/api.py", line 139, in __call__
1166
  return launch_agent(self._config, self._entrypoint, list(args))
1167
  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
1168
+ File "/usr/local/lib/python3.12/dist-packages/torch/distributed/launcher/api.py", line 270, in launch_agent
1169
+ raise ChildFailedError(
1170
+ torch.distributed.elastic.multiprocessing.errors.ChildFailedError:
1171
+ ============================================================
1172
+ pretrain_gpt.py FAILED
1173
+ ------------------------------------------------------------
1174
+ Failures:
1175
+ <NO_OTHER_FAILURES>
1176
+ ------------------------------------------------------------
1177
+ Root Cause (first observed failure):
1178
+ [0]:
1179
+ time : 2026-09-21_05:28:47
1180
+ host : sp-aiinfra-interactive-aghosh-8-mb99k-8xb200-moe-cg-0
1181
+ rank : 0 (local_rank: 0)
1182
+ exitcode : -6 (pid: 24441)
1183
+ error_file: <N/A>
1184
+ traceback : Signal 6 (SIGABRT) received by PID 24441
1185
+ ============================================================
1186
+ Generated:
1187
+ /workspace/granularity_next_b200/tiera_topk_trace/ultraep/tierbtrace/tierbtrace_gbs16/G2_K2__ultraep__matched__seq4096__gbs16__cap1p15625/nsys/G2_K2__ultraep__matched__seq4096__gbs16__cap1p15625.nsys-rep
granularity_next_b200/tiera_topk_trace/ultraep/tierbtrace/tierbtrace_gbs16/G3_K2__ultraep__matched__seq4096__gbs16__cap1p1875/nsys/G3_K2__ultraep__matched__seq4096__gbs16__cap1p1875.nsys-rep CHANGED
@@ -1,3 +1,3 @@
1
  version https://git-lfs.github.com/spec/v1
2
- oid sha256:b72936061abb85f8abaf8716a77beb00af3bf28399bb4c5824e9a1ef123b6169
3
- size 12226799
 
1
  version https://git-lfs.github.com/spec/v1
2
+ oid sha256:aa5ad8c4f434530d5f7e35a6d1c4637caaf159082ca5fa7b61f6f8487d392cb2
3
+ size 11939180
granularity_next_b200/tiera_topk_trace/ultraep/tierbtrace/tierbtrace_gbs16/G3_K2__ultraep__matched__seq4096__gbs16__cap1p1875/nsys/train.log CHANGED
@@ -6,12 +6,10 @@
6
  memory_dump=none snapshots=/workspace/granularity_next_b200/tiera_topk_trace/ultraep/tierbtrace/tierbtrace_gbs16/G3_K2__ultraep__matched__seq4096__gbs16__cap1p1875/nsys/memory_snapshots/snapshot.pickle
7
  repo=/workspace/Megatron-UltraEP @ 03d57ef7
8
  ==============================================================
9
- W0917 03:29:57.172000 960711 torch/distributed/run.py:766]
10
- W0917 03:29:57.172000 960711 torch/distributed/run.py:766] *****************************************
11
- W0917 03:29:57.172000 960711 torch/distributed/run.py:766] Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed.
12
- W0917 03:29:57.172000 960711 torch/distributed/run.py:766] *****************************************
13
- /workspace/Megatron-UltraEP/megatron/training/arguments.py:831: UserWarning: Disabling sequence parallelism because tensor model parallelism is disabled
14
- warnings.warn("Disabling sequence parallelism because tensor model parallelism is disabled")
15
  /workspace/Megatron-UltraEP/megatron/training/arguments.py:831: UserWarning: Disabling sequence parallelism because tensor model parallelism is disabled
16
  warnings.warn("Disabling sequence parallelism because tensor model parallelism is disabled")
17
  /workspace/Megatron-UltraEP/megatron/training/arguments.py:831: UserWarning: Disabling sequence parallelism because tensor model parallelism is disabled
@@ -23,8 +21,6 @@ WARNING: overriding default arguments for tokenizer_type:GPT2BPETokenizer
23
  Number of virtual stages per pipeline stage: None
24
  accumulate and all-reduce gradients in fp32 for bfloat16 data type.
25
  using torch.bfloat16 for parameters ...
26
- /workspace/Megatron-UltraEP/megatron/training/arguments.py:831: UserWarning: Disabling sequence parallelism because tensor model parallelism is disabled
27
- warnings.warn("Disabling sequence parallelism because tensor model parallelism is disabled")
28
  /workspace/Megatron-UltraEP/megatron/training/arguments.py:831: UserWarning: Disabling sequence parallelism because tensor model parallelism is disabled
29
  warnings.warn("Disabling sequence parallelism because tensor model parallelism is disabled")
30
  ------------------------ arguments ------------------------
@@ -573,28 +569,35 @@ INFO:megatron.core.num_microbatches_calculator:setting number of microbatches to
573
  > building GPT2BPETokenizer tokenizer ...
574
  /workspace/Megatron-UltraEP/megatron/training/arguments.py:831: UserWarning: Disabling sequence parallelism because tensor model parallelism is disabled
575
  warnings.warn("Disabling sequence parallelism because tensor model parallelism is disabled")
 
 
 
576
  /workspace/Megatron-UltraEP/megatron/training/arguments.py:831: UserWarning: Disabling sequence parallelism because tensor model parallelism is disabled
577
  warnings.warn("Disabling sequence parallelism because tensor model parallelism is disabled")
578
  > padded vocab (size: 50257) with 47 dummy tokens (new size: 50304)
579
  WARNING:megatron.core.rerun_state_machine:RerunStateMachine initialized in mode RerunMode.VALIDATE_RESULTS
580
  > initializing torch distributed ...
581
- WARNING: one_logger package is required to enable e2e metrics tracking. please go to https://confluence.nvidia.com/display/MLWFO/Package+Repositories for details to install it
 
 
582
  [Gloo] Rank 1 is connected to 7 peer ranks. Expected number of connected peer ranks is : 7
583
- [Gloo] Rank [Gloo] Rank 0 is connected to 37 is connected to peer ranks. Expected number of connected peer ranks is : 77 peer ranks. Expected number of connected peer ranks is : 7
584
-
585
  [Gloo] Rank 2 is connected to 7 peer ranks. Expected number of connected peer ranks is : 7
 
 
586
  [Gloo] Rank 4 is connected to 7 peer ranks. Expected number of connected peer ranks is : 7
587
  [Gloo] Rank 5 is connected to 7 peer ranks. Expected number of connected peer ranks is : 7
588
- [Gloo] Rank [Gloo] Rank 7 is connected to 76 peer ranks. Expected number of connected peer ranks is : is connected to 77 peer ranks.
589
- Expected number of connected peer ranks is : 7
590
  [Gloo] Rank 0 is connected to 7 peer ranks. Expected number of connected peer ranks is : 7
591
  [Gloo] Rank 1 is connected to 7 peer ranks. Expected number of connected peer ranks is : 7
592
  [Gloo] Rank 2 is connected to 7 peer ranks. Expected number of connected peer ranks is : 7
593
- [Gloo] Rank [Gloo] Rank 5 is connected to 37 is connected to peer ranks. 7Expected number of connected peer ranks is : peer ranks. 7Expected number of connected peer ranks is : 7
594
-
595
  [Gloo] Rank 4 is connected to 7 peer ranks. Expected number of connected peer ranks is : 7
596
- [Gloo] Rank 6 is connected to 7 peer ranks. Expected number of connected peer ranks is : 7
597
  [Gloo] Rank 7 is connected to 7 peer ranks. Expected number of connected peer ranks is : 7
 
 
 
 
598
  [Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
599
  [Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
600
  [Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
@@ -603,27 +606,24 @@ Expected number of connected peer ranks is : 7
603
  > setting random seeds to 1234 ...
604
  [Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
605
  [Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
606
- [Gloo] Rank 0 is connected to 0[Gloo] Rank peer ranks. Expected number of connected peer ranks is : 0
607
- 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
608
- [Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
609
- [rank3]:[W917 03:30:09.927160051 ProcessGroupNCCL.cpp:4789] [PG ID 0 PG GUID 0 Rank 3] using GPU 3 as device used by this process is currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect. You can specify device_id in init_process_group() to force use of a particular device.
610
- [rank1]:[W917 03:30:09.927160874 ProcessGroupNCCL.cpp:4789] [PG ID 0 PG GUID 0 Rank 1] using GPU 1 as device used by this process is currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect. You can specify device_id in init_process_group() to force use of a particular device.
611
- [rank5]:[W917 03:30:09.927340509 ProcessGroupNCCL.cpp:4789] [PG ID 0 PG GUID 0 Rank 5] using GPU 5 as device used by this process is currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect. You can specify device_id in init_process_group() to force use of a particular device.
612
- [rank6]:[W917 03:30:09.927344996 ProcessGroupNCCL.cpp:4789] [PG ID 0 PG GUID 0 Rank 6] using GPU 6 as device used by this process is currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect. You can specify device_id in init_process_group() to force use of a particular device.
613
- [rank4]:[W917 03:30:09.927925286 ProcessGroupNCCL.cpp:4789] [PG ID 0 PG GUID 0 Rank 4] using GPU 4 as device used by this process is currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect. You can specify device_id in init_process_group() to force use of a particular device.
614
- [rank7]:[W917 03:30:09.927971518 ProcessGroupNCCL.cpp:4789] [PG ID 0 PG GUID 0 Rank 7] using GPU 7 as device used by this process is currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect. You can specify device_id in init_process_group() to force use of a particular device.
615
- [rank2]:[W917 03:30:09.928149080 ProcessGroupNCCL.cpp:4789] [PG ID 0 PG GUID 0 Rank 2] using GPU 2 as device used by this process is currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect. You can specify device_id in init_process_group() to force use of a particular device.
616
  > compiling dataset index builder ...
617
  make: Entering directory '/workspace/Megatron-UltraEP/megatron/core/datasets'
618
  make: Nothing to be done for 'default'.
619
  make: Leaving directory '/workspace/Megatron-UltraEP/megatron/core/datasets'
620
- >>> done with dataset index builder. Compilation time: 0.437 seconds
621
  WARNING: constraints for invoking optimized fused softmax kernel are not met. We default back to unfused kernel invocations.
622
  > compiling and loading fused kernels ...
623
- [rank0]:[W917 03:30:10.553623237 ProcessGroupNCCL.cpp:4789] [PG ID 0 PG GUID 0 Rank 0] using GPU 0 as device used by this process is currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect. You can specify device_id in init_process_group() to force use of a particular device.
624
- >>> done with compiling and loading fused kernels. Compilation time: 3.679 seconds
625
- time to initialize megatron (seconds): 9.143
626
- [after megatron is initialized] datetime: 2026-09-17 03:30:16
627
  building GPT model ...
628
  /usr/local/lib/python3.12/dist-packages/transformer_engine/pytorch/cpu_offload.py:708: DeprecationWarning: Offloading weights is deprecated. Using offload_weights=True does not have any effect.
629
  warnings.warn(
@@ -667,12 +667,12 @@ INFO:megatron.core.distributed.param_and_grad_buffer:Number of buckets for gradi
667
  Params for bucket 1 (206045184 elements, 206045184 padded size):
668
  module.output_layer.weight
669
  Params for bucket 2 (42217472 elements, 42217472 padded size):
670
- module.decoder.layers.15.self_attention.linear_qkv.layer_norm_weight
671
  module.decoder.layers.15.mlp.router.weight
 
672
  module.decoder.layers.15.self_attention.linear_proj.weight
673
- module.decoder.final_layernorm.weight
674
- module.decoder.layers.15.self_attention.linear_qkv.weight
675
  module.decoder.layers.15.pre_mlp_layernorm.weight
 
676
  Params for bucket 3 (42213376 elements, 42213376 padded size):
677
  module.decoder.layers.14.mlp.router.weight
678
  module.decoder.layers.14.self_attention.linear_qkv.layer_norm_weight
@@ -681,16 +681,16 @@ Params for bucket 3 (42213376 elements, 42213376 padded size):
681
  module.decoder.layers.14.pre_mlp_layernorm.weight
682
  Params for bucket 4 (42213376 elements, 42213376 padded size):
683
  module.decoder.layers.13.mlp.router.weight
684
- module.decoder.layers.13.self_attention.linear_qkv.layer_norm_weight
685
  module.decoder.layers.13.self_attention.linear_proj.weight
 
686
  module.decoder.layers.13.self_attention.linear_qkv.weight
687
  module.decoder.layers.13.pre_mlp_layernorm.weight
688
  Params for bucket 5 (42213376 elements, 42213376 padded size):
689
  module.decoder.layers.12.mlp.router.weight
690
  module.decoder.layers.12.self_attention.linear_proj.weight
691
  module.decoder.layers.12.self_attention.linear_qkv.layer_norm_weight
692
- module.decoder.layers.12.pre_mlp_layernorm.weight
693
  module.decoder.layers.12.self_attention.linear_qkv.weight
 
694
  Params for bucket 6 (42213376 elements, 42213376 padded size):
695
  module.decoder.layers.11.mlp.router.weight
696
  module.decoder.layers.11.self_attention.linear_proj.weight
@@ -698,17 +698,17 @@ Params for bucket 6 (42213376 elements, 42213376 padded size):
698
  module.decoder.layers.11.self_attention.linear_qkv.weight
699
  module.decoder.layers.11.pre_mlp_layernorm.weight
700
  Params for bucket 7 (42213376 elements, 42213376 padded size):
 
 
701
  module.decoder.layers.10.mlp.router.weight
702
  module.decoder.layers.10.self_attention.linear_proj.weight
703
  module.decoder.layers.10.self_attention.linear_qkv.layer_norm_weight
704
- module.decoder.layers.10.pre_mlp_layernorm.weight
705
- module.decoder.layers.10.self_attention.linear_qkv.weight
706
  Params for bucket 8 (42213376 elements, 42213376 padded size):
707
- module.decoder.layers.9.mlp.router.weight
708
- module.decoder.layers.9.self_attention.linear_qkv.layer_norm_weight
709
- module.decoder.layers.9.self_attention.linear_proj.weight
710
  module.decoder.layers.9.self_attention.linear_qkv.weight
711
  module.decoder.layers.9.pre_mlp_layernorm.weight
 
 
 
712
  Params for bucket 9 (42213376 elements, 42213376 padded size):
713
  module.decoder.layers.8.self_attention.linear_qkv.weight
714
  module.decoder.layers.8.pre_mlp_layernorm.weight
@@ -716,41 +716,41 @@ Params for bucket 9 (42213376 elements, 42213376 padded size):
716
  module.decoder.layers.8.self_attention.linear_qkv.layer_norm_weight
717
  module.decoder.layers.8.self_attention.linear_proj.weight
718
  Params for bucket 10 (42213376 elements, 42213376 padded size):
719
- module.decoder.layers.7.pre_mlp_layernorm.weight
720
  module.decoder.layers.7.self_attention.linear_qkv.weight
 
721
  module.decoder.layers.7.mlp.router.weight
722
- module.decoder.layers.7.self_attention.linear_proj.weight
723
  module.decoder.layers.7.self_attention.linear_qkv.layer_norm_weight
 
724
  Params for bucket 11 (42213376 elements, 42213376 padded size):
725
- module.decoder.layers.6.self_attention.linear_qkv.weight
726
- module.decoder.layers.6.pre_mlp_layernorm.weight
727
  module.decoder.layers.6.mlp.router.weight
728
- module.decoder.layers.6.self_attention.linear_qkv.layer_norm_weight
729
  module.decoder.layers.6.self_attention.linear_proj.weight
 
 
 
730
  Params for bucket 12 (42213376 elements, 42213376 padded size):
731
- module.decoder.layers.5.self_attention.linear_proj.weight
732
- module.decoder.layers.5.pre_mlp_layernorm.weight
733
- module.decoder.layers.5.self_attention.linear_qkv.weight
734
  module.decoder.layers.5.mlp.router.weight
735
  module.decoder.layers.5.self_attention.linear_qkv.layer_norm_weight
 
 
 
736
  Params for bucket 13 (42213376 elements, 42213376 padded size):
737
- module.decoder.layers.4.mlp.router.weight
738
- module.decoder.layers.4.self_attention.linear_qkv.layer_norm_weight
739
- module.decoder.layers.4.self_attention.linear_proj.weight
740
  module.decoder.layers.4.self_attention.linear_qkv.weight
741
  module.decoder.layers.4.pre_mlp_layernorm.weight
 
 
 
742
  Params for bucket 14 (42213376 elements, 42213376 padded size):
743
  module.decoder.layers.3.mlp.router.weight
744
- module.decoder.layers.3.self_attention.linear_proj.weight
745
  module.decoder.layers.3.self_attention.linear_qkv.layer_norm_weight
746
- module.decoder.layers.3.pre_mlp_layernorm.weight
747
  module.decoder.layers.3.self_attention.linear_qkv.weight
 
748
  Params for bucket 15 (42213376 elements, 42213376 padded size):
749
  module.decoder.layers.2.mlp.router.weight
750
- module.decoder.layers.2.self_attention.linear_qkv.layer_norm_weight
751
  module.decoder.layers.2.self_attention.linear_proj.weight
752
- module.decoder.layers.2.self_attention.linear_qkv.weight
753
  module.decoder.layers.2.pre_mlp_layernorm.weight
 
754
  Params for bucket 16 (42213376 elements, 42213376 padded size):
755
  module.decoder.layers.1.mlp.router.weight
756
  module.decoder.layers.1.self_attention.linear_proj.weight
@@ -758,21 +758,21 @@ Params for bucket 16 (42213376 elements, 42213376 padded size):
758
  module.decoder.layers.1.self_attention.linear_qkv.weight
759
  module.decoder.layers.1.pre_mlp_layernorm.weight
760
  Params for bucket 17 (42213376 elements, 42213376 padded size):
761
- module.decoder.layers.0.pre_mlp_layernorm.weight
762
- module.decoder.layers.0.self_attention.linear_qkv.layer_norm_weight
763
- module.decoder.layers.0.self_attention.linear_qkv.weight
764
  module.decoder.layers.0.mlp.router.weight
 
765
  module.decoder.layers.0.self_attention.linear_proj.weight
 
 
766
  Params for bucket 18 (206045184 elements, 206045184 padded size):
767
  module.embedding.word_embeddings.weight
768
  INFO:megatron.core.distributed.param_and_grad_buffer:Number of buckets for gradient all-reduce / reduce-scatter: 64
769
  Params for bucket 1 (44040192 elements, 44040192 padded size):
770
- module.decoder.layers.15.mlp.experts.linear_fc2.weight4
771
- module.decoder.layers.15.mlp.experts.linear_fc2.weight6
772
- module.decoder.layers.15.mlp.experts.linear_fc2.weight2
773
  module.decoder.layers.15.mlp.experts.linear_fc2.weight7
774
  module.decoder.layers.15.mlp.experts.linear_fc2.weight3
775
  module.decoder.layers.15.mlp.experts.linear_fc2.weight5
 
 
 
776
  Params for bucket 2 (44040192 elements, 44040192 padded size):
777
  module.decoder.layers.15.mlp.experts.linear_fc2.weight1
778
  module.decoder.layers.15.mlp.experts.linear_fc2.weight0
@@ -796,8 +796,8 @@ Params for bucket 5 (44040192 elements, 44040192 padded size):
796
  Params for bucket 6 (44040192 elements, 44040192 padded size):
797
  module.decoder.layers.14.mlp.experts.linear_fc2.weight1
798
  module.decoder.layers.14.mlp.experts.linear_fc2.weight0
799
- module.decoder.layers.14.mlp.experts.linear_fc1.weight7
800
  module.decoder.layers.14.mlp.experts.linear_fc1.weight6
 
801
  Params for bucket 7 (44040192 elements, 44040192 padded size):
802
  module.decoder.layers.14.mlp.experts.linear_fc1.weight5
803
  module.decoder.layers.14.mlp.experts.linear_fc1.weight4
@@ -828,20 +828,20 @@ Params for bucket 12 (44040192 elements, 44040192 padded size):
828
  module.decoder.layers.13.mlp.experts.linear_fc1.weight1
829
  Params for bucket 13 (44040192 elements, 44040192 padded size):
830
  module.decoder.layers.12.mlp.experts.linear_fc2.weight7
831
- module.decoder.layers.12.mlp.experts.linear_fc2.weight5
832
  module.decoder.layers.12.mlp.experts.linear_fc2.weight3
833
- module.decoder.layers.12.mlp.experts.linear_fc2.weight6
834
  module.decoder.layers.12.mlp.experts.linear_fc2.weight4
 
835
  module.decoder.layers.12.mlp.experts.linear_fc2.weight2
836
  Params for bucket 14 (44040192 elements, 44040192 padded size):
837
  module.decoder.layers.12.mlp.experts.linear_fc2.weight1
838
- module.decoder.layers.12.mlp.experts.linear_fc1.weight6
839
- module.decoder.layers.12.mlp.experts.linear_fc2.weight0
840
  module.decoder.layers.12.mlp.experts.linear_fc1.weight7
 
 
841
  Params for bucket 15 (44040192 elements, 44040192 padded size):
842
  module.decoder.layers.12.mlp.experts.linear_fc1.weight5
843
- module.decoder.layers.12.mlp.experts.linear_fc1.weight3
844
  module.decoder.layers.12.mlp.experts.linear_fc1.weight4
 
845
  Params for bucket 16 (44040192 elements, 44040192 padded size):
846
  module.decoder.layers.12.mlp.experts.linear_fc1.weight2
847
  module.decoder.layers.12.mlp.experts.linear_fc1.weight0
@@ -850,13 +850,13 @@ Params for bucket 17 (44040192 elements, 44040192 padded size):
850
  module.decoder.layers.11.mlp.experts.linear_fc2.weight7
851
  module.decoder.layers.11.mlp.experts.linear_fc2.weight5
852
  module.decoder.layers.11.mlp.experts.linear_fc2.weight3
853
- module.decoder.layers.11.mlp.experts.linear_fc2.weight4
854
  module.decoder.layers.11.mlp.experts.linear_fc2.weight6
 
855
  module.decoder.layers.11.mlp.experts.linear_fc2.weight2
856
  Params for bucket 18 (44040192 elements, 44040192 padded size):
857
  module.decoder.layers.11.mlp.experts.linear_fc2.weight1
858
- module.decoder.layers.11.mlp.experts.linear_fc1.weight7
859
  module.decoder.layers.11.mlp.experts.linear_fc1.weight6
 
860
  module.decoder.layers.11.mlp.experts.linear_fc2.weight0
861
  Params for bucket 19 (44040192 elements, 44040192 padded size):
862
  module.decoder.layers.11.mlp.experts.linear_fc1.weight5
@@ -867,37 +867,37 @@ Params for bucket 20 (44040192 elements, 44040192 padded size):
867
  module.decoder.layers.11.mlp.experts.linear_fc1.weight1
868
  module.decoder.layers.11.mlp.experts.linear_fc1.weight0
869
  Params for bucket 21 (44040192 elements, 44040192 padded size):
870
- module.decoder.layers.10.mlp.experts.linear_fc2.weight7
871
- module.decoder.layers.10.mlp.experts.linear_fc2.weight3
872
- module.decoder.layers.10.mlp.experts.linear_fc2.weight5
873
  module.decoder.layers.10.mlp.experts.linear_fc2.weight6
874
  module.decoder.layers.10.mlp.experts.linear_fc2.weight4
875
  module.decoder.layers.10.mlp.experts.linear_fc2.weight2
 
 
 
876
  Params for bucket 22 (44040192 elements, 44040192 padded size):
877
  module.decoder.layers.10.mlp.experts.linear_fc2.weight1
 
878
  module.decoder.layers.10.mlp.experts.linear_fc2.weight0
879
  module.decoder.layers.10.mlp.experts.linear_fc1.weight7
880
- module.decoder.layers.10.mlp.experts.linear_fc1.weight6
881
  Params for bucket 23 (44040192 elements, 44040192 padded size):
882
  module.decoder.layers.10.mlp.experts.linear_fc1.weight5
883
  module.decoder.layers.10.mlp.experts.linear_fc1.weight3
884
  module.decoder.layers.10.mlp.experts.linear_fc1.weight4
885
  Params for bucket 24 (44040192 elements, 44040192 padded size):
886
  module.decoder.layers.10.mlp.experts.linear_fc1.weight2
887
- module.decoder.layers.10.mlp.experts.linear_fc1.weight0
888
  module.decoder.layers.10.mlp.experts.linear_fc1.weight1
 
889
  Params for bucket 25 (44040192 elements, 44040192 padded size):
 
 
 
890
  module.decoder.layers.9.mlp.experts.linear_fc2.weight7
891
  module.decoder.layers.9.mlp.experts.linear_fc2.weight3
892
  module.decoder.layers.9.mlp.experts.linear_fc2.weight5
893
- module.decoder.layers.9.mlp.experts.linear_fc2.weight4
894
- module.decoder.layers.9.mlp.experts.linear_fc2.weight6
895
- module.decoder.layers.9.mlp.experts.linear_fc2.weight2
896
  Params for bucket 26 (44040192 elements, 44040192 padded size):
897
  module.decoder.layers.9.mlp.experts.linear_fc2.weight1
 
898
  module.decoder.layers.9.mlp.experts.linear_fc2.weight0
899
  module.decoder.layers.9.mlp.experts.linear_fc1.weight7
900
- module.decoder.layers.9.mlp.experts.linear_fc1.weight6
901
  Params for bucket 27 (44040192 elements, 44040192 padded size):
902
  module.decoder.layers.9.mlp.experts.linear_fc1.weight5
903
  module.decoder.layers.9.mlp.experts.linear_fc1.weight3
@@ -907,92 +907,92 @@ Params for bucket 28 (44040192 elements, 44040192 padded size):
907
  module.decoder.layers.9.mlp.experts.linear_fc1.weight0
908
  module.decoder.layers.9.mlp.experts.linear_fc1.weight1
909
  Params for bucket 29 (44040192 elements, 44040192 padded size):
910
- module.decoder.layers.8.mlp.experts.linear_fc2.weight7
911
- module.decoder.layers.8.mlp.experts.linear_fc2.weight5
912
- module.decoder.layers.8.mlp.experts.linear_fc2.weight3
913
- module.decoder.layers.8.mlp.experts.linear_fc2.weight6
914
  module.decoder.layers.8.mlp.experts.linear_fc2.weight4
 
915
  module.decoder.layers.8.mlp.experts.linear_fc2.weight2
 
 
 
916
  Params for bucket 30 (44040192 elements, 44040192 padded size):
917
  module.decoder.layers.8.mlp.experts.linear_fc2.weight1
918
  module.decoder.layers.8.mlp.experts.linear_fc1.weight7
919
- module.decoder.layers.8.mlp.experts.linear_fc2.weight0
920
  module.decoder.layers.8.mlp.experts.linear_fc1.weight6
 
921
  Params for bucket 31 (44040192 elements, 44040192 padded size):
922
  module.decoder.layers.8.mlp.experts.linear_fc1.weight5
923
  module.decoder.layers.8.mlp.experts.linear_fc1.weight4
924
  module.decoder.layers.8.mlp.experts.linear_fc1.weight3
925
  Params for bucket 32 (44040192 elements, 44040192 padded size):
926
  module.decoder.layers.8.mlp.experts.linear_fc1.weight2
927
- module.decoder.layers.8.mlp.experts.linear_fc1.weight0
928
  module.decoder.layers.8.mlp.experts.linear_fc1.weight1
 
929
  Params for bucket 33 (44040192 elements, 44040192 padded size):
930
- module.decoder.layers.7.mlp.experts.linear_fc2.weight7
931
- module.decoder.layers.7.mlp.experts.linear_fc2.weight3
932
- module.decoder.layers.7.mlp.experts.linear_fc2.weight5
933
- module.decoder.layers.7.mlp.experts.linear_fc2.weight4
934
  module.decoder.layers.7.mlp.experts.linear_fc2.weight6
 
935
  module.decoder.layers.7.mlp.experts.linear_fc2.weight2
 
 
 
936
  Params for bucket 34 (44040192 elements, 44040192 padded size):
937
  module.decoder.layers.7.mlp.experts.linear_fc2.weight1
938
- module.decoder.layers.7.mlp.experts.linear_fc1.weight6
939
- module.decoder.layers.7.mlp.experts.linear_fc2.weight0
940
  module.decoder.layers.7.mlp.experts.linear_fc1.weight7
 
 
941
  Params for bucket 35 (44040192 elements, 44040192 padded size):
942
  module.decoder.layers.7.mlp.experts.linear_fc1.weight5
943
- module.decoder.layers.7.mlp.experts.linear_fc1.weight3
944
  module.decoder.layers.7.mlp.experts.linear_fc1.weight4
 
945
  Params for bucket 36 (44040192 elements, 44040192 padded size):
946
  module.decoder.layers.7.mlp.experts.linear_fc1.weight2
947
- module.decoder.layers.7.mlp.experts.linear_fc1.weight0
948
  module.decoder.layers.7.mlp.experts.linear_fc1.weight1
 
949
  Params for bucket 37 (44040192 elements, 44040192 padded size):
950
- module.decoder.layers.6.mlp.experts.linear_fc2.weight4
951
- module.decoder.layers.6.mlp.experts.linear_fc2.weight6
952
- module.decoder.layers.6.mlp.experts.linear_fc2.weight2
953
  module.decoder.layers.6.mlp.experts.linear_fc2.weight7
954
  module.decoder.layers.6.mlp.experts.linear_fc2.weight5
955
  module.decoder.layers.6.mlp.experts.linear_fc2.weight3
 
 
 
956
  Params for bucket 38 (44040192 elements, 44040192 padded size):
957
  module.decoder.layers.6.mlp.experts.linear_fc2.weight1
 
958
  module.decoder.layers.6.mlp.experts.linear_fc1.weight7
959
  module.decoder.layers.6.mlp.experts.linear_fc1.weight6
960
- module.decoder.layers.6.mlp.experts.linear_fc2.weight0
961
  Params for bucket 39 (44040192 elements, 44040192 padded size):
962
  module.decoder.layers.6.mlp.experts.linear_fc1.weight5
963
  module.decoder.layers.6.mlp.experts.linear_fc1.weight4
964
  module.decoder.layers.6.mlp.experts.linear_fc1.weight3
965
  Params for bucket 40 (44040192 elements, 44040192 padded size):
966
  module.decoder.layers.6.mlp.experts.linear_fc1.weight2
967
- module.decoder.layers.6.mlp.experts.linear_fc1.weight1
968
  module.decoder.layers.6.mlp.experts.linear_fc1.weight0
 
969
  Params for bucket 41 (44040192 elements, 44040192 padded size):
970
- module.decoder.layers.5.mlp.experts.linear_fc2.weight6
971
- module.decoder.layers.5.mlp.experts.linear_fc2.weight4
972
- module.decoder.layers.5.mlp.experts.linear_fc2.weight2
973
  module.decoder.layers.5.mlp.experts.linear_fc2.weight7
974
  module.decoder.layers.5.mlp.experts.linear_fc2.weight5
975
  module.decoder.layers.5.mlp.experts.linear_fc2.weight3
 
 
 
976
  Params for bucket 42 (44040192 elements, 44040192 padded size):
977
  module.decoder.layers.5.mlp.experts.linear_fc2.weight1
978
- module.decoder.layers.5.mlp.experts.linear_fc1.weight6
979
- module.decoder.layers.5.mlp.experts.linear_fc2.weight0
980
  module.decoder.layers.5.mlp.experts.linear_fc1.weight7
 
 
981
  Params for bucket 43 (44040192 elements, 44040192 padded size):
982
  module.decoder.layers.5.mlp.experts.linear_fc1.weight5
983
- module.decoder.layers.5.mlp.experts.linear_fc1.weight3
984
  module.decoder.layers.5.mlp.experts.linear_fc1.weight4
 
985
  Params for bucket 44 (44040192 elements, 44040192 padded size):
986
  module.decoder.layers.5.mlp.experts.linear_fc1.weight2
987
- module.decoder.layers.5.mlp.experts.linear_fc1.weight0
988
  module.decoder.layers.5.mlp.experts.linear_fc1.weight1
 
989
  Params for bucket 45 (44040192 elements, 44040192 padded size):
990
- module.decoder.layers.4.mlp.experts.linear_fc2.weight7
991
- module.decoder.layers.4.mlp.experts.linear_fc2.weight5
992
- module.decoder.layers.4.mlp.experts.linear_fc2.weight3
993
  module.decoder.layers.4.mlp.experts.linear_fc2.weight4
994
  module.decoder.layers.4.mlp.experts.linear_fc2.weight6
995
  module.decoder.layers.4.mlp.experts.linear_fc2.weight2
 
 
 
996
  Params for bucket 46 (44040192 elements, 44040192 padded size):
997
  module.decoder.layers.4.mlp.experts.linear_fc2.weight1
998
  module.decoder.layers.4.mlp.experts.linear_fc1.weight7
@@ -1007,57 +1007,57 @@ Params for bucket 48 (44040192 elements, 44040192 padded size):
1007
  module.decoder.layers.4.mlp.experts.linear_fc1.weight1
1008
  module.decoder.layers.4.mlp.experts.linear_fc1.weight0
1009
  Params for bucket 49 (44040192 elements, 44040192 padded size):
1010
- module.decoder.layers.3.mlp.experts.linear_fc2.weight7
1011
- module.decoder.layers.3.mlp.experts.linear_fc2.weight3
1012
- module.decoder.layers.3.mlp.experts.linear_fc2.weight5
1013
  module.decoder.layers.3.mlp.experts.linear_fc2.weight6
1014
  module.decoder.layers.3.mlp.experts.linear_fc2.weight4
1015
  module.decoder.layers.3.mlp.experts.linear_fc2.weight2
 
 
 
1016
  Params for bucket 50 (44040192 elements, 44040192 padded size):
1017
  module.decoder.layers.3.mlp.experts.linear_fc2.weight1
1018
- module.decoder.layers.3.mlp.experts.linear_fc2.weight0
1019
  module.decoder.layers.3.mlp.experts.linear_fc1.weight7
 
1020
  module.decoder.layers.3.mlp.experts.linear_fc1.weight6
1021
  Params for bucket 51 (44040192 elements, 44040192 padded size):
1022
  module.decoder.layers.3.mlp.experts.linear_fc1.weight5
1023
- module.decoder.layers.3.mlp.experts.linear_fc1.weight3
1024
  module.decoder.layers.3.mlp.experts.linear_fc1.weight4
 
1025
  Params for bucket 52 (44040192 elements, 44040192 padded size):
1026
  module.decoder.layers.3.mlp.experts.linear_fc1.weight2
1027
- module.decoder.layers.3.mlp.experts.linear_fc1.weight0
1028
  module.decoder.layers.3.mlp.experts.linear_fc1.weight1
 
1029
  Params for bucket 53 (44040192 elements, 44040192 padded size):
1030
  module.decoder.layers.2.mlp.experts.linear_fc2.weight7
1031
  module.decoder.layers.2.mlp.experts.linear_fc2.weight5
1032
  module.decoder.layers.2.mlp.experts.linear_fc2.weight3
1033
- module.decoder.layers.2.mlp.experts.linear_fc2.weight4
1034
  module.decoder.layers.2.mlp.experts.linear_fc2.weight6
 
1035
  module.decoder.layers.2.mlp.experts.linear_fc2.weight2
1036
  Params for bucket 54 (44040192 elements, 44040192 padded size):
1037
  module.decoder.layers.2.mlp.experts.linear_fc2.weight1
 
1038
  module.decoder.layers.2.mlp.experts.linear_fc1.weight7
1039
  module.decoder.layers.2.mlp.experts.linear_fc1.weight6
1040
- module.decoder.layers.2.mlp.experts.linear_fc2.weight0
1041
  Params for bucket 55 (44040192 elements, 44040192 padded size):
1042
  module.decoder.layers.2.mlp.experts.linear_fc1.weight5
1043
- module.decoder.layers.2.mlp.experts.linear_fc1.weight4
1044
  module.decoder.layers.2.mlp.experts.linear_fc1.weight3
 
1045
  Params for bucket 56 (44040192 elements, 44040192 padded size):
1046
  module.decoder.layers.2.mlp.experts.linear_fc1.weight2
1047
  module.decoder.layers.2.mlp.experts.linear_fc1.weight0
1048
  module.decoder.layers.2.mlp.experts.linear_fc1.weight1
1049
  Params for bucket 57 (44040192 elements, 44040192 padded size):
1050
  module.decoder.layers.1.mlp.experts.linear_fc2.weight7
1051
- module.decoder.layers.1.mlp.experts.linear_fc2.weight5
1052
  module.decoder.layers.1.mlp.experts.linear_fc2.weight3
1053
- module.decoder.layers.1.mlp.experts.linear_fc2.weight6
1054
  module.decoder.layers.1.mlp.experts.linear_fc2.weight4
 
1055
  module.decoder.layers.1.mlp.experts.linear_fc2.weight2
1056
  Params for bucket 58 (44040192 elements, 44040192 padded size):
1057
  module.decoder.layers.1.mlp.experts.linear_fc2.weight1
 
1058
  module.decoder.layers.1.mlp.experts.linear_fc1.weight6
1059
  module.decoder.layers.1.mlp.experts.linear_fc2.weight0
1060
- module.decoder.layers.1.mlp.experts.linear_fc1.weight7
1061
  Params for bucket 59 (44040192 elements, 44040192 padded size):
1062
  module.decoder.layers.1.mlp.experts.linear_fc1.weight5
1063
  module.decoder.layers.1.mlp.experts.linear_fc1.weight3
@@ -1067,16 +1067,16 @@ Params for bucket 60 (44040192 elements, 44040192 padded size):
1067
  module.decoder.layers.1.mlp.experts.linear_fc1.weight0
1068
  module.decoder.layers.1.mlp.experts.linear_fc1.weight1
1069
  Params for bucket 61 (44040192 elements, 44040192 padded size):
1070
- module.decoder.layers.0.mlp.experts.linear_fc2.weight6
1071
  module.decoder.layers.0.mlp.experts.linear_fc2.weight4
 
1072
  module.decoder.layers.0.mlp.experts.linear_fc2.weight2
1073
  module.decoder.layers.0.mlp.experts.linear_fc2.weight7
1074
  module.decoder.layers.0.mlp.experts.linear_fc2.weight5
1075
  module.decoder.layers.0.mlp.experts.linear_fc2.weight3
1076
  Params for bucket 62 (44040192 elements, 44040192 padded size):
1077
  module.decoder.layers.0.mlp.experts.linear_fc2.weight1
1078
- module.decoder.layers.0.mlp.experts.linear_fc1.weight6
1079
  module.decoder.layers.0.mlp.experts.linear_fc2.weight0
 
1080
  module.decoder.layers.0.mlp.experts.linear_fc1.weight7
1081
  Params for bucket 63 (44040192 elements, 44040192 padded size):
1082
  module.decoder.layers.0.mlp.experts.linear_fc1.weight5
@@ -1086,9 +1086,9 @@ Params for bucket 64 (44040192 elements, 44040192 padded size):
1086
  module.decoder.layers.0.mlp.experts.linear_fc1.weight2
1087
  module.decoder.layers.0.mlp.experts.linear_fc1.weight1
1088
  module.decoder.layers.0.mlp.experts.linear_fc1.weight0
1089
- INFO:megatron.core.optimizer:Setting up optimizer with config OptimizerConfig(optimizer='adam', lr=0.0001, min_lr=1e-05, decoupled_lr=None, decoupled_min_lr=None, weight_decay=0.1, fp8_recipe='mxfp8', fp16=False, bf16=True, reuse_grad_buf_for_mxfp8_param_ag=False, params_dtype=torch.bfloat16, use_precision_aware_optimizer=False, store_param_remainders=True, main_grads_dtype=torch.float32, main_params_dtype=torch.float32, exp_avg_dtype=torch.float32, exp_avg_sq_dtype=torch.float32, loss_scale=None, initial_loss_scale=4294967296, min_loss_scale=1.0, loss_scale_window=1000, hysteresis=2, adam_beta1=0.9, adam_beta2=0.999, adam_eps=1e-08, sgd_momentum=0.9, use_distributed_optimizer=True, overlap_param_gather_with_optimizer_step=False, optimizer_cpu_offload=False, optimizer_offload_fraction=1.0, use_torch_optimizer_for_cpu_offload=False, overlap_cpu_optimizer_d2h_h2d=False, pin_cpu_grads=True, pin_cpu_params=True, clip_grad=1.0, log_num_zeros_in_grad=False, barrier_with_L1_time=True, timers=<megatron.core.timers.Timers object at 0x7a0dfef5e690>, config_logger_dir='')
1090
  INFO:megatron.core.optimizer_param_scheduler:> learning rate decay style: cosine
1091
- [after model, optimizer, and learning rate scheduler are built] datetime: 2026-09-17 03:30:19
1092
  > building train, validation, and test datasets ...
1093
  > datasets target sizes (minimum size):
1094
  train: 160
@@ -1096,7 +1096,7 @@ INFO:megatron.core.optimizer_param_scheduler:> learning rate decay style: cosine
1096
  test: 0
1097
  INFO:megatron.core.datasets.blended_megatron_dataset_config:Let split_matrix = [(0, 1.0), None, None]
1098
  > building train, validation, and test datasets for GPT ...
1099
- INFO:megatron.core.datasets.blended_megatron_dataset_builder:Building GPTDataset splits with sizes=(160, 0, 0) and config=GPTDatasetConfig(random_seed=1234, sequence_length=4096, blend=(['/workspace/data/wikitext/wikitext103_gpt2_text_document'], None), blend_per_split=None, split='100,0,0', split_matrix=[(0, 1.0), None, None], num_dataset_builder_threads=1, path_to_cache=None, mmap_bin_files=True, mock=False, tokenizer=<megatron.training.tokenizer.tokenizer._GPT2BPETokenizer object at 0x7a0e01c02f90>, mid_level_dataset_surplus=0.005, reset_position_ids=False, reset_attention_mask=False, eod_mask_loss=False, create_attention_mask=True, drop_last_partial_validation_sequence=True, add_extra_token_to_sequence=True, object_storage_cache_path=None)
1100
  INFO:megatron.core.datasets.indexed_dataset:Load the _IndexReader from /workspace/data/wikitext/wikitext103_gpt2_text_document.idx
1101
  INFO:megatron.core.datasets.indexed_dataset: Extract the sequence lengths
1102
  INFO:megatron.core.datasets.indexed_dataset: Extract the sequence pointers
@@ -1109,14 +1109,14 @@ INFO:megatron.core.datasets.gpt_dataset: Load the sample index from 68684a515e11
1109
  INFO:megatron.core.datasets.gpt_dataset: Load the shuffle index from 68684a515e118253a8572c4517cf2ffe-GPTDataset-train-shuffle_index.npy
1110
  INFO:megatron.core.datasets.gpt_dataset:> total number of samples: 29073
1111
  > finished creating GPT datasets ...
1112
- [after dataloaders are built] datetime: 2026-09-17 03:30:19
1113
  done with setup ...
1114
  (min, max) time across ranks (ms):
1115
- model-and-optimizer-setup ......................: (3225.24, 3230.28)
1116
- train/valid/test-data-iterators-setup ..........: (108.05, 137.65)
1117
  training ...
1118
  Overwriting rerun_state_machine.current_iteration from -1 to 0...
1119
- [before the start of training step] datetime: 2026-09-17 03:30:19
1120
  /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:334: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
1121
  warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
1122
  /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:336: UserWarning: The use_host_meta is deprecated, it will be removed in the future.
@@ -1139,12 +1139,12 @@ Overwriting rerun_state_machine.current_iteration from -1 to 0...
1139
  warnings.warn("The use_host_meta is deprecated, it will be removed in the future.")
1140
  /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:334: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
1141
  warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
1142
- /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:336: UserWarning: The use_host_meta is deprecated, it will be removed in the future.
1143
- warnings.warn("The use_host_meta is deprecated, it will be removed in the future.")
1144
  /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:334: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
1145
  warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
1146
  /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:336: UserWarning: The use_host_meta is deprecated, it will be removed in the future.
1147
  warnings.warn("The use_host_meta is deprecated, it will be removed in the future.")
 
 
1148
  /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:334: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
1149
  warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
1150
  /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:336: UserWarning: The use_host_meta is deprecated, it will be removed in the future.
@@ -1167,18 +1167,14 @@ Overwriting rerun_state_machine.current_iteration from -1 to 0...
1167
  warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
1168
  /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:334: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
1169
  warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
1170
- /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:336: UserWarning: The use_host_meta is deprecated, it will be removed in the future.
1171
- warnings.warn("The use_host_meta is deprecated, it will be removed in the future.")
1172
  /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:334: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
1173
  warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
1174
  /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:336: UserWarning: The use_host_meta is deprecated, it will be removed in the future.
1175
  warnings.warn("The use_host_meta is deprecated, it will be removed in the future.")
1176
- /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:334: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
1177
- warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
1178
- /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:334: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
1179
- warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
1180
  /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:336: UserWarning: The use_host_meta is deprecated, it will be removed in the future.
1181
  warnings.warn("The use_host_meta is deprecated, it will be removed in the future.")
 
 
1182
  /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:336: UserWarning: The use_host_meta is deprecated, it will be removed in the future.
1183
  warnings.warn("The use_host_meta is deprecated, it will be removed in the future.")
1184
  /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:334: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
@@ -1187,6 +1183,10 @@ Overwriting rerun_state_machine.current_iteration from -1 to 0...
1187
  warnings.warn("The use_host_meta is deprecated, it will be removed in the future.")
1188
  /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:334: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
1189
  warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
 
 
 
 
1190
  /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:336: UserWarning: The use_host_meta is deprecated, it will be removed in the future.
1191
  warnings.warn("The use_host_meta is deprecated, it will be removed in the future.")
1192
  /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:334: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
@@ -1245,39 +1245,39 @@ Overwriting rerun_state_machine.current_iteration from -1 to 0...
1245
  warnings.warn("The use_host_meta is deprecated, it will be removed in the future.")
1246
  /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:336: UserWarning: The use_host_meta is deprecated, it will be removed in the future.
1247
  warnings.warn("The use_host_meta is deprecated, it will be removed in the future.")
1248
- [2026-09-17 03:30:50] iteration 1/ 10 | consumed samples: 16 | elapsed time per iteration (ms): 31014.6 | throughput per GPU (TFLOP/s/GPU): 2.9 | learning rate: 2.000000E-05 | global batch size: 16 | lm loss: 1.102587E+01 | loss scale: 1.0 | grad norm: 46.862 | number of skipped iterations: 0 | number of nan iterations: 0 | peak recv tokens: 9216 | min safe capacity: 1.1250 |
1249
  Number of parameters in transformer block in billions: 23.22
1250
  Number of parameters in embedding layers in billions: 0.41
1251
  Total number of parameters in billions: 23.63
1252
  Number of parameters in most loaded shard in billions: 23.6320
1253
  Theoretical memory footprints: weight and optimizer=169029.43 MB
1254
- [Rank 0] (after 1 iterations) memory (MB) | allocated: 58124.50390625 | max allocated: 64453.58203125 | reserved: 80694.0 | max reserved: 80694.0
1255
- [2026-09-17 03:30:51] iteration 2/ 10 | consumed samples: 32 | elapsed time per iteration (ms): 948.0 | throughput per GPU (TFLOP/s/GPU): 95.9 | learning rate: 4.000000E-05 | global batch size: 16 | lm loss: 1.101135E+01 | loss scale: 1.0 | grad norm: 47.659 | number of skipped iterations: 0 | number of nan iterations: 0 | peak recv tokens: 9216 | min safe capacity: 1.1250 |
1256
- [2026-09-17 03:30:52] iteration 3/ 10 | consumed samples: 48 | elapsed time per iteration (ms): 389.4 | throughput per GPU (TFLOP/s/GPU): 233.5 | learning rate: 6.000000E-05 | global batch size: 16 | lm loss: 1.024894E+01 | loss scale: 1.0 | grad norm: 187.969 | number of skipped iterations: 0 | number of nan iterations: 0 | peak recv tokens: 9216 | min safe capacity: 1.1250 |
1257
  Capture CUDA graph for training!!!
1258
  CUDA graph capture done!!!
1259
- [2026-09-17 03:30:56] iteration 4/ 10 | consumed samples: 64 | elapsed time per iteration (ms): 3909.6 | throughput per GPU (TFLOP/s/GPU): 23.3 | learning rate: 8.000000E-05 | global batch size: 16 | lm loss: 1.308593E+01 | loss scale: 1.0 | grad norm: 44.378 | number of skipped iterations: 0 | number of nan iterations: 0 | peak recv tokens: 9216 | min safe capacity: 1.1250 |
1260
- [2026-09-17 03:30:56] iteration 5/ 10 | consumed samples: 80 | elapsed time per iteration (ms): 250.6 | throughput per GPU (TFLOP/s/GPU): 362.9 | learning rate: 1.000000E-04 | global batch size: 16 | lm loss: 1.040534E+01 | loss scale: 1.0 | grad norm: 20.456 | number of skipped iterations: 0 | number of nan iterations: 0 | peak recv tokens: 9216 | min safe capacity: 1.1250 |
1261
- [2026-09-17 03:30:56] iteration 6/ 10 | consumed samples: 96 | elapsed time per iteration (ms): 251.5 | throughput per GPU (TFLOP/s/GPU): 361.6 | learning rate: 7.750000E-05 | global batch size: 16 | lm loss: 1.022187E+01 | loss scale: 1.0 | grad norm: 31.356 | number of skipped iterations: 0 | number of nan iterations: 0 | peak recv tokens: 9216 | min safe capacity: 1.1250 |
1262
- [2026-09-17 03:30:59] iteration 7/ 10 | consumed samples: 112 | elapsed time per iteration (ms): 3070.8 | throughput per GPU (TFLOP/s/GPU): 29.6 | learning rate: 3.250000E-05 | global batch size: 16 | lm loss: 9.913305E+00 | loss scale: 1.0 | grad norm: 9.749 | number of skipped iterations: 0 | number of nan iterations: 0 | peak recv tokens: 9216 | min safe capacity: 1.1250 |
1263
- [2026-09-17 03:30:59] iteration 8/ 10 | consumed samples: 128 | elapsed time per iteration (ms): 270.2 | throughput per GPU (TFLOP/s/GPU): 336.6 | learning rate: 1.000000E-05 | global batch size: 16 | lm loss: 9.461720E+00 | loss scale: 1.0 | grad norm: 8.259 | number of skipped iterations: 0 | number of nan iterations: 0 | peak recv tokens: 9216 | min safe capacity: 1.1250 |
1264
  Capture range started in the application.
1265
  Capture range ended in the application.
1266
- Generating '/tmp/nsys-report-5702.qdstrm'
1267
 
1268
-
1269
  Collection stop is not allowed in this state.
1270
- W0917 03:32:40.178000 960711 torch/distributed/elastic/agent/server/api.py:719] Received 15 death signal, shutting down workers
1271
- W0917 03:32:40.179000 960711 torch/distributed/elastic/multiprocessing/api.py:900] Sending process 960827 closing signal SIGTERM
1272
  Generated:
1273
  /workspace/granularity_next_b200/tiera_topk_trace/ultraep/tierbtrace/tierbtrace_gbs16/G3_K2__ultraep__matched__seq4096__gbs16__cap1p1875/nsys/G3_K2__ultraep__matched__seq4096__gbs16__cap1p1875.nsys-rep
1274
- W0917 03:32:40.180000 960711 torch/distributed/elastic/multiprocessing/api.py:900] Sending process 960828 closing signal SIGTERM
1275
- W0917 03:32:40.187000 960711 torch/distributed/elastic/multiprocessing/api.py:900] Sending process 960829 closing signal SIGTERM
1276
- W0917 03:32:40.189000 960711 torch/distributed/elastic/multiprocessing/api.py:900] Sending process 960830 closing signal SIGTERM
1277
- W0917 03:32:40.196000 960711 torch/distributed/elastic/multiprocessing/api.py:900] Sending process 960831 closing signal SIGTERM
1278
- W0917 03:32:40.201000 960711 torch/distributed/elastic/multiprocessing/api.py:900] Sending process 960832 closing signal SIGTERM
1279
- W0917 03:32:40.214000 960711 torch/distributed/elastic/multiprocessing/api.py:900] Sending process 960833 closing signal SIGTERM
1280
- W0917 03:32:40.216000 960711 torch/distributed/elastic/multiprocessing/api.py:900] Sending process 960834 closing signal SIGTERM
1281
  Traceback (most recent call last):
1282
  File "/usr/local/bin/torchrun", line 8, in <module>
1283
  sys.exit(main())
@@ -1305,7 +1305,7 @@ Traceback (most recent call last):
1305
  time.sleep(monitor_interval)
1306
  File "/usr/local/lib/python3.12/dist-packages/torch/distributed/elastic/multiprocessing/api.py", line 84, in _terminate_process_handler
1307
  raise SignalException(f"Process {os.getpid()} got signal: {sigval}", sigval=sigval)
1308
- torch.distributed.elastic.multiprocessing.api.SignalException: Process 960711 got signal: 15
1309
  The target application terminated. One or more process it created re-parented.
1310
  Waiting for termination of re-parented processes.
1311
  Use the `--wait` option to modify this behavior.
 
6
  memory_dump=none snapshots=/workspace/granularity_next_b200/tiera_topk_trace/ultraep/tierbtrace/tierbtrace_gbs16/G3_K2__ultraep__matched__seq4096__gbs16__cap1p1875/nsys/memory_snapshots/snapshot.pickle
7
  repo=/workspace/Megatron-UltraEP @ 03d57ef7
8
  ==============================================================
9
+ W0921 05:29:24.331000 40554 torch/distributed/run.py:766]
10
+ W0921 05:29:24.331000 40554 torch/distributed/run.py:766] *****************************************
11
+ W0921 05:29:24.331000 40554 torch/distributed/run.py:766] Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed.
12
+ W0921 05:29:24.331000 40554 torch/distributed/run.py:766] *****************************************
 
 
13
  /workspace/Megatron-UltraEP/megatron/training/arguments.py:831: UserWarning: Disabling sequence parallelism because tensor model parallelism is disabled
14
  warnings.warn("Disabling sequence parallelism because tensor model parallelism is disabled")
15
  /workspace/Megatron-UltraEP/megatron/training/arguments.py:831: UserWarning: Disabling sequence parallelism because tensor model parallelism is disabled
 
21
  Number of virtual stages per pipeline stage: None
22
  accumulate and all-reduce gradients in fp32 for bfloat16 data type.
23
  using torch.bfloat16 for parameters ...
 
 
24
  /workspace/Megatron-UltraEP/megatron/training/arguments.py:831: UserWarning: Disabling sequence parallelism because tensor model parallelism is disabled
25
  warnings.warn("Disabling sequence parallelism because tensor model parallelism is disabled")
26
  ------------------------ arguments ------------------------
 
569
  > building GPT2BPETokenizer tokenizer ...
570
  /workspace/Megatron-UltraEP/megatron/training/arguments.py:831: UserWarning: Disabling sequence parallelism because tensor model parallelism is disabled
571
  warnings.warn("Disabling sequence parallelism because tensor model parallelism is disabled")
572
+ /workspace/Megatron-UltraEP/megatron/training/arguments.py:831: UserWarning: Disabling sequence parallelism because tensor model parallelism is disabled
573
+ warnings.warn("Disabling sequence parallelism because tensor model parallelism is disabled")
574
+ WARNING: one_logger package is required to enable e2e metrics tracking. please go to https://confluence.nvidia.com/display/MLWFO/Package+Repositories for details to install it
575
  /workspace/Megatron-UltraEP/megatron/training/arguments.py:831: UserWarning: Disabling sequence parallelism because tensor model parallelism is disabled
576
  warnings.warn("Disabling sequence parallelism because tensor model parallelism is disabled")
577
  > padded vocab (size: 50257) with 47 dummy tokens (new size: 50304)
578
  WARNING:megatron.core.rerun_state_machine:RerunStateMachine initialized in mode RerunMode.VALIDATE_RESULTS
579
  > initializing torch distributed ...
580
+ /workspace/Megatron-UltraEP/megatron/training/arguments.py:831: UserWarning: Disabling sequence parallelism because tensor model parallelism is disabled
581
+ warnings.warn("Disabling sequence parallelism because tensor model parallelism is disabled")
582
+ [Gloo] Rank 0 is connected to 7 peer ranks. Expected number of connected peer ranks is : 7
583
  [Gloo] Rank 1 is connected to 7 peer ranks. Expected number of connected peer ranks is : 7
 
 
584
  [Gloo] Rank 2 is connected to 7 peer ranks. Expected number of connected peer ranks is : 7
585
+ [Gloo] Rank [Gloo] Rank 6 is connected to 7 peer ranks. Expected number of connected peer ranks is : 73 is connected to
586
+ 7 peer ranks. Expected number of connected peer ranks is : 7
587
  [Gloo] Rank 4 is connected to 7 peer ranks. Expected number of connected peer ranks is : 7
588
  [Gloo] Rank 5 is connected to 7 peer ranks. Expected number of connected peer ranks is : 7
589
+ [Gloo] Rank 7 is connected to 7 peer ranks. Expected number of connected peer ranks is : 7
 
590
  [Gloo] Rank 0 is connected to 7 peer ranks. Expected number of connected peer ranks is : 7
591
  [Gloo] Rank 1 is connected to 7 peer ranks. Expected number of connected peer ranks is : 7
592
  [Gloo] Rank 2 is connected to 7 peer ranks. Expected number of connected peer ranks is : 7
593
+ [Gloo] Rank 3 is connected to 7 peer ranks. Expected number of connected peer ranks is : 7
 
594
  [Gloo] Rank 4 is connected to 7 peer ranks. Expected number of connected peer ranks is : 7
595
+ [Gloo] Rank 5 is connected to 7 peer ranks. Expected number of connected peer ranks is : 7
596
  [Gloo] Rank 7 is connected to 7 peer ranks. Expected number of connected peer ranks is : 7
597
+ [Gloo] Rank 6 is connected to 7 peer ranks. Expected number of connected peer ranks is : 7
598
+ [Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
599
+ [Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
600
+ [Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
601
  [Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
602
  [Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
603
  [Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
 
606
  > setting random seeds to 1234 ...
607
  [Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
608
  [Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
609
+ [rank4]:[W921 05:29:37.103162947 ProcessGroupNCCL.cpp:4789] [PG ID 0 PG GUID 0 Rank 4] using GPU 4 as device used by this process is currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect. You can specify device_id in init_process_group() to force use of a particular device.
610
+ [rank2]:[W921 05:29:37.103195924 ProcessGroupNCCL.cpp:4789] [PG ID 0 PG GUID 0 Rank 2] using GPU 2 as device used by this process is currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect. You can specify device_id in init_process_group() to force use of a particular device.
611
+ [rank1]:[W921 05:29:37.108298812 ProcessGroupNCCL.cpp:4789] [PG ID 0 PG GUID 0 Rank 1] using GPU 1 as device used by this process is currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect. You can specify device_id in init_process_group() to force use of a particular device.
612
+ [rank7]:[W921 05:29:37.108324544 ProcessGroupNCCL.cpp:4789] [PG ID 0 PG GUID 0 Rank 7] using GPU 7 as device used by this process is currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect. You can specify device_id in init_process_group() to force use of a particular device.
613
+ [rank6]:[W921 05:29:37.108372298 ProcessGroupNCCL.cpp:4789] [PG ID 0 PG GUID 0 Rank 6] using GPU 6 as device used by this process is currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect. You can specify device_id in init_process_group() to force use of a particular device.
614
+ [rank5]:[W921 05:29:37.108439648 ProcessGroupNCCL.cpp:4789] [PG ID 0 PG GUID 0 Rank 5] using GPU 5 as device used by this process is currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect. You can specify device_id in init_process_group() to force use of a particular device.
615
+ [rank3]:[W921 05:29:37.108449582 ProcessGroupNCCL.cpp:4789] [PG ID 0 PG GUID 0 Rank 3] using GPU 3 as device used by this process is currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect. You can specify device_id in init_process_group() to force use of a particular device.
 
 
 
616
  > compiling dataset index builder ...
617
  make: Entering directory '/workspace/Megatron-UltraEP/megatron/core/datasets'
618
  make: Nothing to be done for 'default'.
619
  make: Leaving directory '/workspace/Megatron-UltraEP/megatron/core/datasets'
620
+ >>> done with dataset index builder. Compilation time: 0.425 seconds
621
  WARNING: constraints for invoking optimized fused softmax kernel are not met. We default back to unfused kernel invocations.
622
  > compiling and loading fused kernels ...
623
+ [rank0]:[W921 05:29:37.720116691 ProcessGroupNCCL.cpp:4789] [PG ID 0 PG GUID 0 Rank 0] using GPU 0 as device used by this process is currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect. You can specify device_id in init_process_group() to force use of a particular device.
624
+ >>> done with compiling and loading fused kernels. Compilation time: 4.130 seconds
625
+ time to initialize megatron (seconds): 9.722
626
+ [after megatron is initialized] datetime: 2026-09-21 05:29:44
627
  building GPT model ...
628
  /usr/local/lib/python3.12/dist-packages/transformer_engine/pytorch/cpu_offload.py:708: DeprecationWarning: Offloading weights is deprecated. Using offload_weights=True does not have any effect.
629
  warnings.warn(
 
667
  Params for bucket 1 (206045184 elements, 206045184 padded size):
668
  module.output_layer.weight
669
  Params for bucket 2 (42217472 elements, 42217472 padded size):
670
+ module.decoder.final_layernorm.weight
671
  module.decoder.layers.15.mlp.router.weight
672
+ module.decoder.layers.15.self_attention.linear_qkv.layer_norm_weight
673
  module.decoder.layers.15.self_attention.linear_proj.weight
 
 
674
  module.decoder.layers.15.pre_mlp_layernorm.weight
675
+ module.decoder.layers.15.self_attention.linear_qkv.weight
676
  Params for bucket 3 (42213376 elements, 42213376 padded size):
677
  module.decoder.layers.14.mlp.router.weight
678
  module.decoder.layers.14.self_attention.linear_qkv.layer_norm_weight
 
681
  module.decoder.layers.14.pre_mlp_layernorm.weight
682
  Params for bucket 4 (42213376 elements, 42213376 padded size):
683
  module.decoder.layers.13.mlp.router.weight
 
684
  module.decoder.layers.13.self_attention.linear_proj.weight
685
+ module.decoder.layers.13.self_attention.linear_qkv.layer_norm_weight
686
  module.decoder.layers.13.self_attention.linear_qkv.weight
687
  module.decoder.layers.13.pre_mlp_layernorm.weight
688
  Params for bucket 5 (42213376 elements, 42213376 padded size):
689
  module.decoder.layers.12.mlp.router.weight
690
  module.decoder.layers.12.self_attention.linear_proj.weight
691
  module.decoder.layers.12.self_attention.linear_qkv.layer_norm_weight
 
692
  module.decoder.layers.12.self_attention.linear_qkv.weight
693
+ module.decoder.layers.12.pre_mlp_layernorm.weight
694
  Params for bucket 6 (42213376 elements, 42213376 padded size):
695
  module.decoder.layers.11.mlp.router.weight
696
  module.decoder.layers.11.self_attention.linear_proj.weight
 
698
  module.decoder.layers.11.self_attention.linear_qkv.weight
699
  module.decoder.layers.11.pre_mlp_layernorm.weight
700
  Params for bucket 7 (42213376 elements, 42213376 padded size):
701
+ module.decoder.layers.10.self_attention.linear_qkv.weight
702
+ module.decoder.layers.10.pre_mlp_layernorm.weight
703
  module.decoder.layers.10.mlp.router.weight
704
  module.decoder.layers.10.self_attention.linear_proj.weight
705
  module.decoder.layers.10.self_attention.linear_qkv.layer_norm_weight
 
 
706
  Params for bucket 8 (42213376 elements, 42213376 padded size):
 
 
 
707
  module.decoder.layers.9.self_attention.linear_qkv.weight
708
  module.decoder.layers.9.pre_mlp_layernorm.weight
709
+ module.decoder.layers.9.mlp.router.weight
710
+ module.decoder.layers.9.self_attention.linear_proj.weight
711
+ module.decoder.layers.9.self_attention.linear_qkv.layer_norm_weight
712
  Params for bucket 9 (42213376 elements, 42213376 padded size):
713
  module.decoder.layers.8.self_attention.linear_qkv.weight
714
  module.decoder.layers.8.pre_mlp_layernorm.weight
 
716
  module.decoder.layers.8.self_attention.linear_qkv.layer_norm_weight
717
  module.decoder.layers.8.self_attention.linear_proj.weight
718
  Params for bucket 10 (42213376 elements, 42213376 padded size):
 
719
  module.decoder.layers.7.self_attention.linear_qkv.weight
720
+ module.decoder.layers.7.pre_mlp_layernorm.weight
721
  module.decoder.layers.7.mlp.router.weight
 
722
  module.decoder.layers.7.self_attention.linear_qkv.layer_norm_weight
723
+ module.decoder.layers.7.self_attention.linear_proj.weight
724
  Params for bucket 11 (42213376 elements, 42213376 padded size):
 
 
725
  module.decoder.layers.6.mlp.router.weight
 
726
  module.decoder.layers.6.self_attention.linear_proj.weight
727
+ module.decoder.layers.6.self_attention.linear_qkv.layer_norm_weight
728
+ module.decoder.layers.6.self_attention.linear_qkv.weight
729
+ module.decoder.layers.6.pre_mlp_layernorm.weight
730
  Params for bucket 12 (42213376 elements, 42213376 padded size):
 
 
 
731
  module.decoder.layers.5.mlp.router.weight
732
  module.decoder.layers.5.self_attention.linear_qkv.layer_norm_weight
733
+ module.decoder.layers.5.self_attention.linear_qkv.weight
734
+ module.decoder.layers.5.self_attention.linear_proj.weight
735
+ module.decoder.layers.5.pre_mlp_layernorm.weight
736
  Params for bucket 13 (42213376 elements, 42213376 padded size):
 
 
 
737
  module.decoder.layers.4.self_attention.linear_qkv.weight
738
  module.decoder.layers.4.pre_mlp_layernorm.weight
739
+ module.decoder.layers.4.mlp.router.weight
740
+ module.decoder.layers.4.self_attention.linear_proj.weight
741
+ module.decoder.layers.4.self_attention.linear_qkv.layer_norm_weight
742
  Params for bucket 14 (42213376 elements, 42213376 padded size):
743
  module.decoder.layers.3.mlp.router.weight
 
744
  module.decoder.layers.3.self_attention.linear_qkv.layer_norm_weight
745
+ module.decoder.layers.3.self_attention.linear_proj.weight
746
  module.decoder.layers.3.self_attention.linear_qkv.weight
747
+ module.decoder.layers.3.pre_mlp_layernorm.weight
748
  Params for bucket 15 (42213376 elements, 42213376 padded size):
749
  module.decoder.layers.2.mlp.router.weight
 
750
  module.decoder.layers.2.self_attention.linear_proj.weight
751
+ module.decoder.layers.2.self_attention.linear_qkv.layer_norm_weight
752
  module.decoder.layers.2.pre_mlp_layernorm.weight
753
+ module.decoder.layers.2.self_attention.linear_qkv.weight
754
  Params for bucket 16 (42213376 elements, 42213376 padded size):
755
  module.decoder.layers.1.mlp.router.weight
756
  module.decoder.layers.1.self_attention.linear_proj.weight
 
758
  module.decoder.layers.1.self_attention.linear_qkv.weight
759
  module.decoder.layers.1.pre_mlp_layernorm.weight
760
  Params for bucket 17 (42213376 elements, 42213376 padded size):
 
 
 
761
  module.decoder.layers.0.mlp.router.weight
762
+ module.decoder.layers.0.self_attention.linear_qkv.layer_norm_weight
763
  module.decoder.layers.0.self_attention.linear_proj.weight
764
+ module.decoder.layers.0.self_attention.linear_qkv.weight
765
+ module.decoder.layers.0.pre_mlp_layernorm.weight
766
  Params for bucket 18 (206045184 elements, 206045184 padded size):
767
  module.embedding.word_embeddings.weight
768
  INFO:megatron.core.distributed.param_and_grad_buffer:Number of buckets for gradient all-reduce / reduce-scatter: 64
769
  Params for bucket 1 (44040192 elements, 44040192 padded size):
 
 
 
770
  module.decoder.layers.15.mlp.experts.linear_fc2.weight7
771
  module.decoder.layers.15.mlp.experts.linear_fc2.weight3
772
  module.decoder.layers.15.mlp.experts.linear_fc2.weight5
773
+ module.decoder.layers.15.mlp.experts.linear_fc2.weight6
774
+ module.decoder.layers.15.mlp.experts.linear_fc2.weight4
775
+ module.decoder.layers.15.mlp.experts.linear_fc2.weight2
776
  Params for bucket 2 (44040192 elements, 44040192 padded size):
777
  module.decoder.layers.15.mlp.experts.linear_fc2.weight1
778
  module.decoder.layers.15.mlp.experts.linear_fc2.weight0
 
796
  Params for bucket 6 (44040192 elements, 44040192 padded size):
797
  module.decoder.layers.14.mlp.experts.linear_fc2.weight1
798
  module.decoder.layers.14.mlp.experts.linear_fc2.weight0
 
799
  module.decoder.layers.14.mlp.experts.linear_fc1.weight6
800
+ module.decoder.layers.14.mlp.experts.linear_fc1.weight7
801
  Params for bucket 7 (44040192 elements, 44040192 padded size):
802
  module.decoder.layers.14.mlp.experts.linear_fc1.weight5
803
  module.decoder.layers.14.mlp.experts.linear_fc1.weight4
 
828
  module.decoder.layers.13.mlp.experts.linear_fc1.weight1
829
  Params for bucket 13 (44040192 elements, 44040192 padded size):
830
  module.decoder.layers.12.mlp.experts.linear_fc2.weight7
 
831
  module.decoder.layers.12.mlp.experts.linear_fc2.weight3
832
+ module.decoder.layers.12.mlp.experts.linear_fc2.weight5
833
  module.decoder.layers.12.mlp.experts.linear_fc2.weight4
834
+ module.decoder.layers.12.mlp.experts.linear_fc2.weight6
835
  module.decoder.layers.12.mlp.experts.linear_fc2.weight2
836
  Params for bucket 14 (44040192 elements, 44040192 padded size):
837
  module.decoder.layers.12.mlp.experts.linear_fc2.weight1
 
 
838
  module.decoder.layers.12.mlp.experts.linear_fc1.weight7
839
+ module.decoder.layers.12.mlp.experts.linear_fc2.weight0
840
+ module.decoder.layers.12.mlp.experts.linear_fc1.weight6
841
  Params for bucket 15 (44040192 elements, 44040192 padded size):
842
  module.decoder.layers.12.mlp.experts.linear_fc1.weight5
 
843
  module.decoder.layers.12.mlp.experts.linear_fc1.weight4
844
+ module.decoder.layers.12.mlp.experts.linear_fc1.weight3
845
  Params for bucket 16 (44040192 elements, 44040192 padded size):
846
  module.decoder.layers.12.mlp.experts.linear_fc1.weight2
847
  module.decoder.layers.12.mlp.experts.linear_fc1.weight0
 
850
  module.decoder.layers.11.mlp.experts.linear_fc2.weight7
851
  module.decoder.layers.11.mlp.experts.linear_fc2.weight5
852
  module.decoder.layers.11.mlp.experts.linear_fc2.weight3
 
853
  module.decoder.layers.11.mlp.experts.linear_fc2.weight6
854
+ module.decoder.layers.11.mlp.experts.linear_fc2.weight4
855
  module.decoder.layers.11.mlp.experts.linear_fc2.weight2
856
  Params for bucket 18 (44040192 elements, 44040192 padded size):
857
  module.decoder.layers.11.mlp.experts.linear_fc2.weight1
 
858
  module.decoder.layers.11.mlp.experts.linear_fc1.weight6
859
+ module.decoder.layers.11.mlp.experts.linear_fc1.weight7
860
  module.decoder.layers.11.mlp.experts.linear_fc2.weight0
861
  Params for bucket 19 (44040192 elements, 44040192 padded size):
862
  module.decoder.layers.11.mlp.experts.linear_fc1.weight5
 
867
  module.decoder.layers.11.mlp.experts.linear_fc1.weight1
868
  module.decoder.layers.11.mlp.experts.linear_fc1.weight0
869
  Params for bucket 21 (44040192 elements, 44040192 padded size):
 
 
 
870
  module.decoder.layers.10.mlp.experts.linear_fc2.weight6
871
  module.decoder.layers.10.mlp.experts.linear_fc2.weight4
872
  module.decoder.layers.10.mlp.experts.linear_fc2.weight2
873
+ module.decoder.layers.10.mlp.experts.linear_fc2.weight7
874
+ module.decoder.layers.10.mlp.experts.linear_fc2.weight5
875
+ module.decoder.layers.10.mlp.experts.linear_fc2.weight3
876
  Params for bucket 22 (44040192 elements, 44040192 padded size):
877
  module.decoder.layers.10.mlp.experts.linear_fc2.weight1
878
+ module.decoder.layers.10.mlp.experts.linear_fc1.weight6
879
  module.decoder.layers.10.mlp.experts.linear_fc2.weight0
880
  module.decoder.layers.10.mlp.experts.linear_fc1.weight7
 
881
  Params for bucket 23 (44040192 elements, 44040192 padded size):
882
  module.decoder.layers.10.mlp.experts.linear_fc1.weight5
883
  module.decoder.layers.10.mlp.experts.linear_fc1.weight3
884
  module.decoder.layers.10.mlp.experts.linear_fc1.weight4
885
  Params for bucket 24 (44040192 elements, 44040192 padded size):
886
  module.decoder.layers.10.mlp.experts.linear_fc1.weight2
 
887
  module.decoder.layers.10.mlp.experts.linear_fc1.weight1
888
+ module.decoder.layers.10.mlp.experts.linear_fc1.weight0
889
  Params for bucket 25 (44040192 elements, 44040192 padded size):
890
+ module.decoder.layers.9.mlp.experts.linear_fc2.weight6
891
+ module.decoder.layers.9.mlp.experts.linear_fc2.weight4
892
+ module.decoder.layers.9.mlp.experts.linear_fc2.weight2
893
  module.decoder.layers.9.mlp.experts.linear_fc2.weight7
894
  module.decoder.layers.9.mlp.experts.linear_fc2.weight3
895
  module.decoder.layers.9.mlp.experts.linear_fc2.weight5
 
 
 
896
  Params for bucket 26 (44040192 elements, 44040192 padded size):
897
  module.decoder.layers.9.mlp.experts.linear_fc2.weight1
898
+ module.decoder.layers.9.mlp.experts.linear_fc1.weight6
899
  module.decoder.layers.9.mlp.experts.linear_fc2.weight0
900
  module.decoder.layers.9.mlp.experts.linear_fc1.weight7
 
901
  Params for bucket 27 (44040192 elements, 44040192 padded size):
902
  module.decoder.layers.9.mlp.experts.linear_fc1.weight5
903
  module.decoder.layers.9.mlp.experts.linear_fc1.weight3
 
907
  module.decoder.layers.9.mlp.experts.linear_fc1.weight0
908
  module.decoder.layers.9.mlp.experts.linear_fc1.weight1
909
  Params for bucket 29 (44040192 elements, 44040192 padded size):
 
 
 
 
910
  module.decoder.layers.8.mlp.experts.linear_fc2.weight4
911
+ module.decoder.layers.8.mlp.experts.linear_fc2.weight6
912
  module.decoder.layers.8.mlp.experts.linear_fc2.weight2
913
+ module.decoder.layers.8.mlp.experts.linear_fc2.weight7
914
+ module.decoder.layers.8.mlp.experts.linear_fc2.weight3
915
+ module.decoder.layers.8.mlp.experts.linear_fc2.weight5
916
  Params for bucket 30 (44040192 elements, 44040192 padded size):
917
  module.decoder.layers.8.mlp.experts.linear_fc2.weight1
918
  module.decoder.layers.8.mlp.experts.linear_fc1.weight7
 
919
  module.decoder.layers.8.mlp.experts.linear_fc1.weight6
920
+ module.decoder.layers.8.mlp.experts.linear_fc2.weight0
921
  Params for bucket 31 (44040192 elements, 44040192 padded size):
922
  module.decoder.layers.8.mlp.experts.linear_fc1.weight5
923
  module.decoder.layers.8.mlp.experts.linear_fc1.weight4
924
  module.decoder.layers.8.mlp.experts.linear_fc1.weight3
925
  Params for bucket 32 (44040192 elements, 44040192 padded size):
926
  module.decoder.layers.8.mlp.experts.linear_fc1.weight2
 
927
  module.decoder.layers.8.mlp.experts.linear_fc1.weight1
928
+ module.decoder.layers.8.mlp.experts.linear_fc1.weight0
929
  Params for bucket 33 (44040192 elements, 44040192 padded size):
 
 
 
 
930
  module.decoder.layers.7.mlp.experts.linear_fc2.weight6
931
+ module.decoder.layers.7.mlp.experts.linear_fc2.weight4
932
  module.decoder.layers.7.mlp.experts.linear_fc2.weight2
933
+ module.decoder.layers.7.mlp.experts.linear_fc2.weight7
934
+ module.decoder.layers.7.mlp.experts.linear_fc2.weight5
935
+ module.decoder.layers.7.mlp.experts.linear_fc2.weight3
936
  Params for bucket 34 (44040192 elements, 44040192 padded size):
937
  module.decoder.layers.7.mlp.experts.linear_fc2.weight1
 
 
938
  module.decoder.layers.7.mlp.experts.linear_fc1.weight7
939
+ module.decoder.layers.7.mlp.experts.linear_fc2.weight0
940
+ module.decoder.layers.7.mlp.experts.linear_fc1.weight6
941
  Params for bucket 35 (44040192 elements, 44040192 padded size):
942
  module.decoder.layers.7.mlp.experts.linear_fc1.weight5
 
943
  module.decoder.layers.7.mlp.experts.linear_fc1.weight4
944
+ module.decoder.layers.7.mlp.experts.linear_fc1.weight3
945
  Params for bucket 36 (44040192 elements, 44040192 padded size):
946
  module.decoder.layers.7.mlp.experts.linear_fc1.weight2
 
947
  module.decoder.layers.7.mlp.experts.linear_fc1.weight1
948
+ module.decoder.layers.7.mlp.experts.linear_fc1.weight0
949
  Params for bucket 37 (44040192 elements, 44040192 padded size):
 
 
 
950
  module.decoder.layers.6.mlp.experts.linear_fc2.weight7
951
  module.decoder.layers.6.mlp.experts.linear_fc2.weight5
952
  module.decoder.layers.6.mlp.experts.linear_fc2.weight3
953
+ module.decoder.layers.6.mlp.experts.linear_fc2.weight4
954
+ module.decoder.layers.6.mlp.experts.linear_fc2.weight6
955
+ module.decoder.layers.6.mlp.experts.linear_fc2.weight2
956
  Params for bucket 38 (44040192 elements, 44040192 padded size):
957
  module.decoder.layers.6.mlp.experts.linear_fc2.weight1
958
+ module.decoder.layers.6.mlp.experts.linear_fc2.weight0
959
  module.decoder.layers.6.mlp.experts.linear_fc1.weight7
960
  module.decoder.layers.6.mlp.experts.linear_fc1.weight6
 
961
  Params for bucket 39 (44040192 elements, 44040192 padded size):
962
  module.decoder.layers.6.mlp.experts.linear_fc1.weight5
963
  module.decoder.layers.6.mlp.experts.linear_fc1.weight4
964
  module.decoder.layers.6.mlp.experts.linear_fc1.weight3
965
  Params for bucket 40 (44040192 elements, 44040192 padded size):
966
  module.decoder.layers.6.mlp.experts.linear_fc1.weight2
 
967
  module.decoder.layers.6.mlp.experts.linear_fc1.weight0
968
+ module.decoder.layers.6.mlp.experts.linear_fc1.weight1
969
  Params for bucket 41 (44040192 elements, 44040192 padded size):
 
 
 
970
  module.decoder.layers.5.mlp.experts.linear_fc2.weight7
971
  module.decoder.layers.5.mlp.experts.linear_fc2.weight5
972
  module.decoder.layers.5.mlp.experts.linear_fc2.weight3
973
+ module.decoder.layers.5.mlp.experts.linear_fc2.weight4
974
+ module.decoder.layers.5.mlp.experts.linear_fc2.weight6
975
+ module.decoder.layers.5.mlp.experts.linear_fc2.weight2
976
  Params for bucket 42 (44040192 elements, 44040192 padded size):
977
  module.decoder.layers.5.mlp.experts.linear_fc2.weight1
 
 
978
  module.decoder.layers.5.mlp.experts.linear_fc1.weight7
979
+ module.decoder.layers.5.mlp.experts.linear_fc2.weight0
980
+ module.decoder.layers.5.mlp.experts.linear_fc1.weight6
981
  Params for bucket 43 (44040192 elements, 44040192 padded size):
982
  module.decoder.layers.5.mlp.experts.linear_fc1.weight5
 
983
  module.decoder.layers.5.mlp.experts.linear_fc1.weight4
984
+ module.decoder.layers.5.mlp.experts.linear_fc1.weight3
985
  Params for bucket 44 (44040192 elements, 44040192 padded size):
986
  module.decoder.layers.5.mlp.experts.linear_fc1.weight2
 
987
  module.decoder.layers.5.mlp.experts.linear_fc1.weight1
988
+ module.decoder.layers.5.mlp.experts.linear_fc1.weight0
989
  Params for bucket 45 (44040192 elements, 44040192 padded size):
 
 
 
990
  module.decoder.layers.4.mlp.experts.linear_fc2.weight4
991
  module.decoder.layers.4.mlp.experts.linear_fc2.weight6
992
  module.decoder.layers.4.mlp.experts.linear_fc2.weight2
993
+ module.decoder.layers.4.mlp.experts.linear_fc2.weight7
994
+ module.decoder.layers.4.mlp.experts.linear_fc2.weight3
995
+ module.decoder.layers.4.mlp.experts.linear_fc2.weight5
996
  Params for bucket 46 (44040192 elements, 44040192 padded size):
997
  module.decoder.layers.4.mlp.experts.linear_fc2.weight1
998
  module.decoder.layers.4.mlp.experts.linear_fc1.weight7
 
1007
  module.decoder.layers.4.mlp.experts.linear_fc1.weight1
1008
  module.decoder.layers.4.mlp.experts.linear_fc1.weight0
1009
  Params for bucket 49 (44040192 elements, 44040192 padded size):
 
 
 
1010
  module.decoder.layers.3.mlp.experts.linear_fc2.weight6
1011
  module.decoder.layers.3.mlp.experts.linear_fc2.weight4
1012
  module.decoder.layers.3.mlp.experts.linear_fc2.weight2
1013
+ module.decoder.layers.3.mlp.experts.linear_fc2.weight7
1014
+ module.decoder.layers.3.mlp.experts.linear_fc2.weight3
1015
+ module.decoder.layers.3.mlp.experts.linear_fc2.weight5
1016
  Params for bucket 50 (44040192 elements, 44040192 padded size):
1017
  module.decoder.layers.3.mlp.experts.linear_fc2.weight1
 
1018
  module.decoder.layers.3.mlp.experts.linear_fc1.weight7
1019
+ module.decoder.layers.3.mlp.experts.linear_fc2.weight0
1020
  module.decoder.layers.3.mlp.experts.linear_fc1.weight6
1021
  Params for bucket 51 (44040192 elements, 44040192 padded size):
1022
  module.decoder.layers.3.mlp.experts.linear_fc1.weight5
 
1023
  module.decoder.layers.3.mlp.experts.linear_fc1.weight4
1024
+ module.decoder.layers.3.mlp.experts.linear_fc1.weight3
1025
  Params for bucket 52 (44040192 elements, 44040192 padded size):
1026
  module.decoder.layers.3.mlp.experts.linear_fc1.weight2
 
1027
  module.decoder.layers.3.mlp.experts.linear_fc1.weight1
1028
+ module.decoder.layers.3.mlp.experts.linear_fc1.weight0
1029
  Params for bucket 53 (44040192 elements, 44040192 padded size):
1030
  module.decoder.layers.2.mlp.experts.linear_fc2.weight7
1031
  module.decoder.layers.2.mlp.experts.linear_fc2.weight5
1032
  module.decoder.layers.2.mlp.experts.linear_fc2.weight3
 
1033
  module.decoder.layers.2.mlp.experts.linear_fc2.weight6
1034
+ module.decoder.layers.2.mlp.experts.linear_fc2.weight4
1035
  module.decoder.layers.2.mlp.experts.linear_fc2.weight2
1036
  Params for bucket 54 (44040192 elements, 44040192 padded size):
1037
  module.decoder.layers.2.mlp.experts.linear_fc2.weight1
1038
+ module.decoder.layers.2.mlp.experts.linear_fc2.weight0
1039
  module.decoder.layers.2.mlp.experts.linear_fc1.weight7
1040
  module.decoder.layers.2.mlp.experts.linear_fc1.weight6
 
1041
  Params for bucket 55 (44040192 elements, 44040192 padded size):
1042
  module.decoder.layers.2.mlp.experts.linear_fc1.weight5
 
1043
  module.decoder.layers.2.mlp.experts.linear_fc1.weight3
1044
+ module.decoder.layers.2.mlp.experts.linear_fc1.weight4
1045
  Params for bucket 56 (44040192 elements, 44040192 padded size):
1046
  module.decoder.layers.2.mlp.experts.linear_fc1.weight2
1047
  module.decoder.layers.2.mlp.experts.linear_fc1.weight0
1048
  module.decoder.layers.2.mlp.experts.linear_fc1.weight1
1049
  Params for bucket 57 (44040192 elements, 44040192 padded size):
1050
  module.decoder.layers.1.mlp.experts.linear_fc2.weight7
 
1051
  module.decoder.layers.1.mlp.experts.linear_fc2.weight3
1052
+ module.decoder.layers.1.mlp.experts.linear_fc2.weight5
1053
  module.decoder.layers.1.mlp.experts.linear_fc2.weight4
1054
+ module.decoder.layers.1.mlp.experts.linear_fc2.weight6
1055
  module.decoder.layers.1.mlp.experts.linear_fc2.weight2
1056
  Params for bucket 58 (44040192 elements, 44040192 padded size):
1057
  module.decoder.layers.1.mlp.experts.linear_fc2.weight1
1058
+ module.decoder.layers.1.mlp.experts.linear_fc1.weight7
1059
  module.decoder.layers.1.mlp.experts.linear_fc1.weight6
1060
  module.decoder.layers.1.mlp.experts.linear_fc2.weight0
 
1061
  Params for bucket 59 (44040192 elements, 44040192 padded size):
1062
  module.decoder.layers.1.mlp.experts.linear_fc1.weight5
1063
  module.decoder.layers.1.mlp.experts.linear_fc1.weight3
 
1067
  module.decoder.layers.1.mlp.experts.linear_fc1.weight0
1068
  module.decoder.layers.1.mlp.experts.linear_fc1.weight1
1069
  Params for bucket 61 (44040192 elements, 44040192 padded size):
 
1070
  module.decoder.layers.0.mlp.experts.linear_fc2.weight4
1071
+ module.decoder.layers.0.mlp.experts.linear_fc2.weight6
1072
  module.decoder.layers.0.mlp.experts.linear_fc2.weight2
1073
  module.decoder.layers.0.mlp.experts.linear_fc2.weight7
1074
  module.decoder.layers.0.mlp.experts.linear_fc2.weight5
1075
  module.decoder.layers.0.mlp.experts.linear_fc2.weight3
1076
  Params for bucket 62 (44040192 elements, 44040192 padded size):
1077
  module.decoder.layers.0.mlp.experts.linear_fc2.weight1
 
1078
  module.decoder.layers.0.mlp.experts.linear_fc2.weight0
1079
+ module.decoder.layers.0.mlp.experts.linear_fc1.weight6
1080
  module.decoder.layers.0.mlp.experts.linear_fc1.weight7
1081
  Params for bucket 63 (44040192 elements, 44040192 padded size):
1082
  module.decoder.layers.0.mlp.experts.linear_fc1.weight5
 
1086
  module.decoder.layers.0.mlp.experts.linear_fc1.weight2
1087
  module.decoder.layers.0.mlp.experts.linear_fc1.weight1
1088
  module.decoder.layers.0.mlp.experts.linear_fc1.weight0
1089
+ INFO:megatron.core.optimizer:Setting up optimizer with config OptimizerConfig(optimizer='adam', lr=0.0001, min_lr=1e-05, decoupled_lr=None, decoupled_min_lr=None, weight_decay=0.1, fp8_recipe='mxfp8', fp16=False, bf16=True, reuse_grad_buf_for_mxfp8_param_ag=False, params_dtype=torch.bfloat16, use_precision_aware_optimizer=False, store_param_remainders=True, main_grads_dtype=torch.float32, main_params_dtype=torch.float32, exp_avg_dtype=torch.float32, exp_avg_sq_dtype=torch.float32, loss_scale=None, initial_loss_scale=4294967296, min_loss_scale=1.0, loss_scale_window=1000, hysteresis=2, adam_beta1=0.9, adam_beta2=0.999, adam_eps=1e-08, sgd_momentum=0.9, use_distributed_optimizer=True, overlap_param_gather_with_optimizer_step=False, optimizer_cpu_offload=False, optimizer_offload_fraction=1.0, use_torch_optimizer_for_cpu_offload=False, overlap_cpu_optimizer_d2h_h2d=False, pin_cpu_grads=True, pin_cpu_params=True, clip_grad=1.0, log_num_zeros_in_grad=False, barrier_with_L1_time=True, timers=<megatron.core.timers.Timers object at 0x7aeb16eacd10>, config_logger_dir='')
1090
  INFO:megatron.core.optimizer_param_scheduler:> learning rate decay style: cosine
1091
+ [after model, optimizer, and learning rate scheduler are built] datetime: 2026-09-21 05:29:47
1092
  > building train, validation, and test datasets ...
1093
  > datasets target sizes (minimum size):
1094
  train: 160
 
1096
  test: 0
1097
  INFO:megatron.core.datasets.blended_megatron_dataset_config:Let split_matrix = [(0, 1.0), None, None]
1098
  > building train, validation, and test datasets for GPT ...
1099
+ INFO:megatron.core.datasets.blended_megatron_dataset_builder:Building GPTDataset splits with sizes=(160, 0, 0) and config=GPTDatasetConfig(random_seed=1234, sequence_length=4096, blend=(['/workspace/data/wikitext/wikitext103_gpt2_text_document'], None), blend_per_split=None, split='100,0,0', split_matrix=[(0, 1.0), None, None], num_dataset_builder_threads=1, path_to_cache=None, mmap_bin_files=True, mock=False, tokenizer=<megatron.training.tokenizer.tokenizer._GPT2BPETokenizer object at 0x7aeb18c02fc0>, mid_level_dataset_surplus=0.005, reset_position_ids=False, reset_attention_mask=False, eod_mask_loss=False, create_attention_mask=True, drop_last_partial_validation_sequence=True, add_extra_token_to_sequence=True, object_storage_cache_path=None)
1100
  INFO:megatron.core.datasets.indexed_dataset:Load the _IndexReader from /workspace/data/wikitext/wikitext103_gpt2_text_document.idx
1101
  INFO:megatron.core.datasets.indexed_dataset: Extract the sequence lengths
1102
  INFO:megatron.core.datasets.indexed_dataset: Extract the sequence pointers
 
1109
  INFO:megatron.core.datasets.gpt_dataset: Load the shuffle index from 68684a515e118253a8572c4517cf2ffe-GPTDataset-train-shuffle_index.npy
1110
  INFO:megatron.core.datasets.gpt_dataset:> total number of samples: 29073
1111
  > finished creating GPT datasets ...
1112
+ [after dataloaders are built] datetime: 2026-09-21 05:29:47
1113
  done with setup ...
1114
  (min, max) time across ranks (ms):
1115
+ model-and-optimizer-setup ......................: (3612.13, 3618.21)
1116
+ train/valid/test-data-iterators-setup ..........: (108.00, 131.25)
1117
  training ...
1118
  Overwriting rerun_state_machine.current_iteration from -1 to 0...
1119
+ [before the start of training step] datetime: 2026-09-21 05:29:47
1120
  /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:334: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
1121
  warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
1122
  /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:336: UserWarning: The use_host_meta is deprecated, it will be removed in the future.
 
1139
  warnings.warn("The use_host_meta is deprecated, it will be removed in the future.")
1140
  /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:334: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
1141
  warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
 
 
1142
  /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:334: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
1143
  warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
1144
  /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:336: UserWarning: The use_host_meta is deprecated, it will be removed in the future.
1145
  warnings.warn("The use_host_meta is deprecated, it will be removed in the future.")
1146
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:336: UserWarning: The use_host_meta is deprecated, it will be removed in the future.
1147
+ warnings.warn("The use_host_meta is deprecated, it will be removed in the future.")
1148
  /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:334: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
1149
  warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
1150
  /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:336: UserWarning: The use_host_meta is deprecated, it will be removed in the future.
 
1167
  warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
1168
  /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:334: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
1169
  warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
 
 
1170
  /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:334: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
1171
  warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
1172
  /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:336: UserWarning: The use_host_meta is deprecated, it will be removed in the future.
1173
  warnings.warn("The use_host_meta is deprecated, it will be removed in the future.")
 
 
 
 
1174
  /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:336: UserWarning: The use_host_meta is deprecated, it will be removed in the future.
1175
  warnings.warn("The use_host_meta is deprecated, it will be removed in the future.")
1176
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:334: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
1177
+ warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
1178
  /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:336: UserWarning: The use_host_meta is deprecated, it will be removed in the future.
1179
  warnings.warn("The use_host_meta is deprecated, it will be removed in the future.")
1180
  /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:334: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
 
1183
  warnings.warn("The use_host_meta is deprecated, it will be removed in the future.")
1184
  /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:334: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
1185
  warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
1186
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:334: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
1187
+ warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
1188
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:336: UserWarning: The use_host_meta is deprecated, it will be removed in the future.
1189
+ warnings.warn("The use_host_meta is deprecated, it will be removed in the future.")
1190
  /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:336: UserWarning: The use_host_meta is deprecated, it will be removed in the future.
1191
  warnings.warn("The use_host_meta is deprecated, it will be removed in the future.")
1192
  /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:334: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
 
1245
  warnings.warn("The use_host_meta is deprecated, it will be removed in the future.")
1246
  /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:336: UserWarning: The use_host_meta is deprecated, it will be removed in the future.
1247
  warnings.warn("The use_host_meta is deprecated, it will be removed in the future.")
1248
+ [2026-09-21 05:30:20] iteration 1/ 10 | consumed samples: 16 | elapsed time per iteration (ms): 32858.0 | throughput per GPU (TFLOP/s/GPU): 2.8 | learning rate: 2.000000E-05 | global batch size: 16 | lm loss: 1.102587E+01 | loss scale: 1.0 | grad norm: 46.865 | number of skipped iterations: 0 | number of nan iterations: 0 | peak recv tokens: 9216 | min safe capacity: 1.1250 |
1249
  Number of parameters in transformer block in billions: 23.22
1250
  Number of parameters in embedding layers in billions: 0.41
1251
  Total number of parameters in billions: 23.63
1252
  Number of parameters in most loaded shard in billions: 23.6320
1253
  Theoretical memory footprints: weight and optimizer=169029.43 MB
1254
+ [Rank 0] (after 1 iterations) memory (MB) | allocated: 58124.75390625 | max allocated: 64480.37890625 | reserved: 80670.0 | max reserved: 80670.0
1255
+ [2026-09-21 05:30:21] iteration 2/ 10 | consumed samples: 32 | elapsed time per iteration (ms): 565.1 | throughput per GPU (TFLOP/s/GPU): 160.9 | learning rate: 4.000000E-05 | global batch size: 16 | lm loss: 1.101135E+01 | loss scale: 1.0 | grad norm: 47.659 | number of skipped iterations: 0 | number of nan iterations: 0 | peak recv tokens: 9216 | min safe capacity: 1.1250 |
1256
+ [2026-09-21 05:30:21] iteration 3/ 10 | consumed samples: 48 | elapsed time per iteration (ms): 390.3 | throughput per GPU (TFLOP/s/GPU): 233.0 | learning rate: 6.000000E-05 | global batch size: 16 | lm loss: 1.024875E+01 | loss scale: 1.0 | grad norm: 187.935 | number of skipped iterations: 0 | number of nan iterations: 0 | peak recv tokens: 9216 | min safe capacity: 1.1250 |
1257
  Capture CUDA graph for training!!!
1258
  CUDA graph capture done!!!
1259
+ [2026-09-21 05:30:25] iteration 4/ 10 | consumed samples: 64 | elapsed time per iteration (ms): 3753.4 | throughput per GPU (TFLOP/s/GPU): 24.2 | learning rate: 8.000000E-05 | global batch size: 16 | lm loss: 1.308636E+01 | loss scale: 1.0 | grad norm: 44.344 | number of skipped iterations: 0 | number of nan iterations: 0 | peak recv tokens: 9216 | min safe capacity: 1.1250 |
1260
+ [2026-09-21 05:30:25] iteration 5/ 10 | consumed samples: 80 | elapsed time per iteration (ms): 250.6 | throughput per GPU (TFLOP/s/GPU): 362.9 | learning rate: 1.000000E-04 | global batch size: 16 | lm loss: 1.040598E+01 | loss scale: 1.0 | grad norm: 20.458 | number of skipped iterations: 0 | number of nan iterations: 0 | peak recv tokens: 9216 | min safe capacity: 1.1250 |
1261
+ [2026-09-21 05:30:26] iteration 6/ 10 | consumed samples: 96 | elapsed time per iteration (ms): 251.7 | throughput per GPU (TFLOP/s/GPU): 361.4 | learning rate: 7.750000E-05 | global batch size: 16 | lm loss: 1.022778E+01 | loss scale: 1.0 | grad norm: 31.694 | number of skipped iterations: 0 | number of nan iterations: 0 | peak recv tokens: 9216 | min safe capacity: 1.1250 |
1262
+ [2026-09-21 05:30:29] iteration 7/ 10 | consumed samples: 112 | elapsed time per iteration (ms): 3026.7 | throughput per GPU (TFLOP/s/GPU): 30.0 | learning rate: 3.250000E-05 | global batch size: 16 | lm loss: 9.919262E+00 | loss scale: 1.0 | grad norm: 9.716 | number of skipped iterations: 0 | number of nan iterations: 0 | peak recv tokens: 9216 | min safe capacity: 1.1250 |
1263
+ [2026-09-21 05:30:29] iteration 8/ 10 | consumed samples: 128 | elapsed time per iteration (ms): 263.7 | throughput per GPU (TFLOP/s/GPU): 344.9 | learning rate: 1.000000E-05 | global batch size: 16 | lm loss: 9.460276E+00 | loss scale: 1.0 | grad norm: 8.308 | number of skipped iterations: 0 | number of nan iterations: 0 | peak recv tokens: 9216 | min safe capacity: 1.1250 |
1264
  Capture range started in the application.
1265
  Capture range ended in the application.
1266
+ Generating '/tmp/nsys-report-c7be.qdstrm'
1267
 
1268
+
1269
  Collection stop is not allowed in this state.
1270
+ W0921 05:32:07.016000 40554 torch/distributed/elastic/agent/server/api.py:719] Received 15 death signal, shutting down workers
1271
+ W0921 05:32:07.016000 40554 torch/distributed/elastic/multiprocessing/api.py:900] Sending process 40665 closing signal SIGTERM
1272
  Generated:
1273
  /workspace/granularity_next_b200/tiera_topk_trace/ultraep/tierbtrace/tierbtrace_gbs16/G3_K2__ultraep__matched__seq4096__gbs16__cap1p1875/nsys/G3_K2__ultraep__matched__seq4096__gbs16__cap1p1875.nsys-rep
1274
+ W0921 05:32:07.018000 40554 torch/distributed/elastic/multiprocessing/api.py:900] Sending process 40666 closing signal SIGTERM
1275
+ W0921 05:32:07.021000 40554 torch/distributed/elastic/multiprocessing/api.py:900] Sending process 40667 closing signal SIGTERM
1276
+ W0921 05:32:07.022000 40554 torch/distributed/elastic/multiprocessing/api.py:900] Sending process 40668 closing signal SIGTERM
1277
+ W0921 05:32:07.027000 40554 torch/distributed/elastic/multiprocessing/api.py:900] Sending process 40669 closing signal SIGTERM
1278
+ W0921 05:32:07.028000 40554 torch/distributed/elastic/multiprocessing/api.py:900] Sending process 40670 closing signal SIGTERM
1279
+ W0921 05:32:07.029000 40554 torch/distributed/elastic/multiprocessing/api.py:900] Sending process 40671 closing signal SIGTERM
1280
+ W0921 05:32:07.035000 40554 torch/distributed/elastic/multiprocessing/api.py:900] Sending process 40672 closing signal SIGTERM
1281
  Traceback (most recent call last):
1282
  File "/usr/local/bin/torchrun", line 8, in <module>
1283
  sys.exit(main())
 
1305
  time.sleep(monitor_interval)
1306
  File "/usr/local/lib/python3.12/dist-packages/torch/distributed/elastic/multiprocessing/api.py", line 84, in _terminate_process_handler
1307
  raise SignalException(f"Process {os.getpid()} got signal: {sigval}", sigval=sigval)
1308
+ torch.distributed.elastic.multiprocessing.api.SignalException: Process 40554 got signal: 15
1309
  The target application terminated. One or more process it created re-parented.
1310
  Waiting for termination of re-parented processes.
1311
  Use the `--wait` option to modify this behavior.
granularity_next_b200/tiera_topk_trace/ultraep/tierbtrace/tierbtrace_gbs16/G3_K8__ultraep__matched__seq4096__gbs16__cap1p09375/nsys/train.log CHANGED
The diff for this file is too large to render. See raw diff
 
granularity_next_b200/tiera_topk_trace/ultraep/tierbtrace/tierbtrace_gbs16/trace_results.tsv CHANGED
@@ -1,5 +1,3 @@
1
  model backend capacity status report async_overlap_json
2
  G2_K2 ultraep 1.15625 OK /workspace/granularity_next_b200/tiera_topk_trace/ultraep/tierbtrace/tierbtrace_gbs16/G2_K2__ultraep__matched__seq4096__gbs16__cap1p15625/nsys/G2_K2__ultraep__matched__seq4096__gbs16__cap1p15625.nsys-rep
3
  G3_K2 ultraep 1.1875 OK /workspace/granularity_next_b200/tiera_topk_trace/ultraep/tierbtrace/tierbtrace_gbs16/G3_K2__ultraep__matched__seq4096__gbs16__cap1p1875/nsys/G3_K2__ultraep__matched__seq4096__gbs16__cap1p1875.nsys-rep
4
- G3_K8 ultraep 1.09375 OK /workspace/granularity_next_b200/tiera_topk_trace/ultraep/tierbtrace/tierbtrace_gbs16/G3_K8__ultraep__matched__seq4096__gbs16__cap1p09375/nsys/G3_K8__ultraep__matched__seq4096__gbs16__cap1p09375.nsys-rep
5
- G4_K2 ultraep 1.265625 OK /workspace/granularity_next_b200/tiera_topk_trace/ultraep/tierbtrace/tierbtrace_gbs16/G4_K2__ultraep__matched__seq4096__gbs16__cap1p265625/nsys/G4_K2__ultraep__matched__seq4096__gbs16__cap1p265625.nsys-rep
 
1
  model backend capacity status report async_overlap_json
2
  G2_K2 ultraep 1.15625 OK /workspace/granularity_next_b200/tiera_topk_trace/ultraep/tierbtrace/tierbtrace_gbs16/G2_K2__ultraep__matched__seq4096__gbs16__cap1p15625/nsys/G2_K2__ultraep__matched__seq4096__gbs16__cap1p15625.nsys-rep
3
  G3_K2 ultraep 1.1875 OK /workspace/granularity_next_b200/tiera_topk_trace/ultraep/tierbtrace/tierbtrace_gbs16/G3_K2__ultraep__matched__seq4096__gbs16__cap1p1875/nsys/G3_K2__ultraep__matched__seq4096__gbs16__cap1p1875.nsys-rep