AbhishekGhosh1998 commited on
Commit
792e78b
·
verified ·
1 Parent(s): b666176

granularity results granularity_next_b200/tiera_topk_trace @ 2026-09-17T03:04:56Z from hp-aiinfra-interactive-aghosh-8-xzthl-8xb200-moe-cg-0

Browse files
.gitattributes CHANGED
@@ -88,3 +88,4 @@ granularity_next_b200/tiera_topk_trace/echo/tierbtrace/tierbtrace_gbs16/G3_K8__e
88
  granularity_next_b200/tiera_topk_trace/echo/tierbtrace/tierbtrace_gbs16/G4_K2__echo__matched__seq4096__gbs16__cap1p265625/nsys/G4_K2__echo__matched__seq4096__gbs16__cap1p265625.nsys-rep filter=lfs diff=lfs merge=lfs -text
89
  granularity_next_b200/tiera_topk_trace/echo/tierbtrace/tierbtrace_gbs16/G4_K4__echo__matched__seq4096__gbs16__cap1p171875/nsys/G4_K4__echo__matched__seq4096__gbs16__cap1p171875.nsys-rep filter=lfs diff=lfs merge=lfs -text
90
  granularity_next_b200/tiera_topk_trace/echo/tierbtrace/tierbtrace_gbs16/G4_K6__echo__matched__seq4096__gbs16__cap1p1354166666666667/nsys/G4_K6__echo__matched__seq4096__gbs16__cap1p1354166666666667.nsys-rep filter=lfs diff=lfs merge=lfs -text
 
 
88
  granularity_next_b200/tiera_topk_trace/echo/tierbtrace/tierbtrace_gbs16/G4_K2__echo__matched__seq4096__gbs16__cap1p265625/nsys/G4_K2__echo__matched__seq4096__gbs16__cap1p265625.nsys-rep filter=lfs diff=lfs merge=lfs -text
89
  granularity_next_b200/tiera_topk_trace/echo/tierbtrace/tierbtrace_gbs16/G4_K4__echo__matched__seq4096__gbs16__cap1p171875/nsys/G4_K4__echo__matched__seq4096__gbs16__cap1p171875.nsys-rep filter=lfs diff=lfs merge=lfs -text
90
  granularity_next_b200/tiera_topk_trace/echo/tierbtrace/tierbtrace_gbs16/G4_K6__echo__matched__seq4096__gbs16__cap1p1354166666666667/nsys/G4_K6__echo__matched__seq4096__gbs16__cap1p1354166666666667.nsys-rep filter=lfs diff=lfs merge=lfs -text
91
+ granularity_next_b200/tiera_topk_trace/echo/tierbtrace/tierbtrace_gbs16/G4_K8__echo__matched__seq4096__gbs16__cap1p109375/nsys/G4_K8__echo__matched__seq4096__gbs16__cap1p109375.nsys-rep filter=lfs diff=lfs merge=lfs -text
granularity_next_b200/tiera_topk_trace/echo/tierbtrace/tierbtrace_gbs16/G4_K10__echo__matched__seq4096__gbs16__cap1p1/nsys/run.env ADDED
@@ -0,0 +1,102 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ backend=echo
2
+ config=G4_K10
3
+ family=tierA
4
+ source_model=
5
+ standard_num_layers=
6
+ base_rung=G4
7
+ repo=/workspace/Megatron-MoE-Echo
8
+ branch=bracing-regions-through-kernels
9
+ commit=9a236793
10
+ num_layers=16
11
+ hidden=4096
12
+ ffn_hidden=14336
13
+ moe_ffn_hidden=896
14
+ num_experts=128
15
+ topk=10
16
+ experts_per_rank=16
17
+ expert_params=11010048
18
+ heads=32
19
+ kv_groups=8
20
+ kv_channels=
21
+ qk_layernorm=0
22
+ shared_ffn=0
23
+ shared_expert_overlap=0
24
+ moe_token_dispatcher=flex
25
+ mla=0
26
+ q_lora_rank=
27
+ kv_lora_rank=512
28
+ qk_head_dim=128
29
+ qk_pos_emb_head_dim=64
30
+ v_head_dim=128
31
+ dense_first_layers=0
32
+ moe_layers=16
33
+ rotary_base=1000000
34
+ rotary_percent=1.0
35
+ rotary_scaling=
36
+ mscale=
37
+ mscale_all_dim=
38
+ qkv_bias=0
39
+ router_balancing=aux_loss
40
+ aux_loss_coeff=1e-2
41
+ router_pre_softmax=0
42
+ router_fusion=0
43
+ router_dtype=fp32
44
+ router_groups=
45
+ router_group_topk=
46
+ router_score_function=softmax
47
+ router_expert_bias=0
48
+ router_bias_update_rate=
49
+ router_topk_scaling_factor=
50
+ shared_expert_gate=0
51
+ apply_layernorm_1p=0
52
+ attention_output_gate=0
53
+ linear_attention_type=
54
+ linear_attention_freq=
55
+ linear_conv_kernel_dim=
56
+ linear_key_head_dim=
57
+ linear_value_head_dim=
58
+ linear_num_key_heads=
59
+ linear_num_value_heads=
60
+ mtp_num_layers=0
61
+ mtp_loss_scaling_factor=0.1
62
+ no_weight_decay_cond_type=
63
+ max_position_embeddings=32768
64
+ init_method_std=0.01
65
+ device_initiated_grouped_gemm=1
66
+ norm_epsilon=1e-5
67
+ seq_length=4096
68
+ micro_batch_size=1
69
+ global_batch_size=16
70
+ num_microbatches=2
71
+ assignments_per_rank=40960
72
+ mean_tokens_per_expert=2560
73
+ slot_policy=matched
74
+ slots_per_rank=16
75
+ capacity_mode=static
76
+ capacity=1.1
77
+ cuda_graph=1
78
+ probe_mode=0
79
+ async_grad_reduce=0
80
+ branch_role=timing
81
+ memory_dump_mode=none
82
+ memory_snapshot_path=
83
+ cuda_graph_warmup_steps=3
84
+ recompute=none
85
+ force_balance=0
86
+ train_iters=10
87
+ ep_size=8
88
+ echo_dispatch_overlap=1
89
+ echo_dispatch_recompute=0
90
+ ultraep_quota_min_tokens=
91
+ ultraep_weight_sync_join=
92
+ profile_enable=1
93
+ kernel_trace=1
94
+ fp8_mode=mxfp8
95
+ fp8_recipe=mxfp8
96
+ training_seed=1234
97
+ learning_rate=1e-4
98
+ min_learning_rate=1.0e-5
99
+ lr_warmup_iters=5
100
+ state_dump_enable=0
101
+ state_dump_dir=
102
+ moonep_assert_no_pending_reduce=0
granularity_next_b200/tiera_topk_trace/echo/tierbtrace/tierbtrace_gbs16/G4_K10__echo__matched__seq4096__gbs16__cap1p1/nsys/train.log ADDED
The diff for this file is too large to render. See raw diff
 
granularity_next_b200/tiera_topk_trace/echo/tierbtrace/tierbtrace_gbs16/G4_K8__echo__matched__seq4096__gbs16__cap1p109375/nsys/G4_K8__echo__matched__seq4096__gbs16__cap1p109375.nsys-rep ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:29cbcc6522af47be42598f13f2a180ba3814ec548870bbccc6be9e1a5dc54bc6
3
+ size 19955945
granularity_next_b200/tiera_topk_trace/echo/tierbtrace/tierbtrace_gbs16/G4_K8__echo__matched__seq4096__gbs16__cap1p109375/nsys/train.log CHANGED
@@ -560,3 +560,986 @@ WARNING:megatron.core.rerun_state_machine:RerunStateMachine initialized in mode
560
  > initializing torch distributed ...
561
  /workspace/Megatron-MoE-Echo/megatron/training/arguments.py:831: UserWarning: Disabling sequence parallelism because tensor model parallelism is disabled
562
  warnings.warn("Disabling sequence parallelism because tensor model parallelism is disabled")
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
560
  > initializing torch distributed ...
561
  /workspace/Megatron-MoE-Echo/megatron/training/arguments.py:831: UserWarning: Disabling sequence parallelism because tensor model parallelism is disabled
562
  warnings.warn("Disabling sequence parallelism because tensor model parallelism is disabled")
563
+ /workspace/Megatron-MoE-Echo/megatron/training/arguments.py:831: UserWarning: Disabling sequence parallelism because tensor model parallelism is disabled
564
+ warnings.warn("Disabling sequence parallelism because tensor model parallelism is disabled")
565
+ /workspace/Megatron-MoE-Echo/megatron/training/arguments.py:831: UserWarning: Disabling sequence parallelism because tensor model parallelism is disabled
566
+ warnings.warn("Disabling sequence parallelism because tensor model parallelism is disabled")
567
+ /workspace/Megatron-MoE-Echo/megatron/training/arguments.py:831: UserWarning: Disabling sequence parallelism because tensor model parallelism is disabled
568
+ warnings.warn("Disabling sequence parallelism because tensor model parallelism is disabled")
569
+ /workspace/Megatron-MoE-Echo/megatron/training/arguments.py:831: UserWarning: Disabling sequence parallelism because tensor model parallelism is disabled
570
+ warnings.warn("Disabling sequence parallelism because tensor model parallelism is disabled")
571
+ /workspace/Megatron-MoE-Echo/megatron/training/arguments.py:831: UserWarning: Disabling sequence parallelism because tensor model parallelism is disabled
572
+ warnings.warn("Disabling sequence parallelism because tensor model parallelism is disabled")
573
+ /workspace/Megatron-MoE-Echo/megatron/training/arguments.py:831: UserWarning: Disabling sequence parallelism because tensor model parallelism is disabled
574
+ warnings.warn("Disabling sequence parallelism because tensor model parallelism is disabled")
575
+ WARNING: one_logger package is required to enable e2e metrics tracking. please go to https://confluence.nvidia.com/display/MLWFO/Package+Repositories for details to install it
576
+ [Gloo] Rank 1 is connected to 7 peer ranks. Expected number of connected peer ranks is : 7
577
+ [Gloo] Rank 0 is connected to 7 peer ranks. Expected number of connected peer ranks is : 7
578
+ [Gloo] Rank 2 is connected to 7 peer ranks. Expected number of connected peer ranks is : 7
579
+ [Gloo] Rank 3 is connected to 7 peer ranks. [Gloo] Rank Expected number of connected peer ranks is : 7
580
+ 5 is connected to 7 peer ranks. Expected number of connected peer ranks is : 7
581
+ [Gloo] Rank 4 is connected to 7 peer ranks. Expected number of connected peer ranks is : 7
582
+ [Gloo] Rank [Gloo] Rank 7 is connected to 76 peer ranks. is connected to Expected number of connected peer ranks is : 77 peer ranks. Expected number of connected peer ranks is : 7
583
+
584
+ [Gloo] Rank [Gloo] Rank 7 is connected to 70 peer ranks. is connected to Expected number of connected peer ranks is : 77 peer ranks. Expected number of connected peer ranks is : 7
585
+
586
+ [Gloo] Rank 1 is connected to 7 peer ranks. Expected number of connected peer ranks is : 7
587
+ [Gloo] Rank 2 is connected to 7 peer ranks. Expected number of connected peer ranks is : 7
588
+ [Gloo] Rank 4 is connected to 7 peer ranks. Expected number of connected peer ranks is : 7
589
+ [Gloo] Rank 3 is connected to [Gloo] Rank 57 is connected to peer ranks. 7Expected number of connected peer ranks is : peer ranks. 7Expected number of connected peer ranks is :
590
+ 7
591
+ [Gloo] Rank 6 is connected to 7 peer ranks. Expected number of connected peer ranks is : 7
592
+ [Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
593
+ [Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
594
+ [Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
595
+ [Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
596
+ [Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
597
+ [Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
598
+ [Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
599
+ [Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
600
+ > initialized tensor model parallel with size 1
601
+ > initialized pipeline model parallel with size 1
602
+ > setting random seeds to 1234 ...
603
+ [rank1]:[W917 02:54:57.228066964 ProcessGroupNCCL.cpp:4789] [PG ID 0 PG GUID 0 Rank 1] using GPU 1 as device used by this process is currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect. You can specify device_id in init_process_group() to force use of a particular device.
604
+ [rank7]:[W917 02:54:57.228203329 ProcessGroupNCCL.cpp:4789] [PG ID 0 PG GUID 0 Rank 7] using GPU 7 as device used by this process is currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect. You can specify device_id in init_process_group() to force use of a particular device.
605
+ [rank3]:[W917 02:54:57.228215283 ProcessGroupNCCL.cpp:4789] [PG ID 0 PG GUID 0 Rank 3] using GPU 3 as device used by this process is currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect. You can specify device_id in init_process_group() to force use of a particular device.
606
+ [rank2]:[W917 02:54:57.228230940 ProcessGroupNCCL.cpp:4789] [PG ID 0 PG GUID 0 Rank 2] using GPU 2 as device used by this process is currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect. You can specify device_id in init_process_group() to force use of a particular device.
607
+ [rank4]:[W917 02:54:57.228248236 ProcessGroupNCCL.cpp:4789] [PG ID 0 PG GUID 0 Rank 4] using GPU 4 as device used by this process is currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect. You can specify device_id in init_process_group() to force use of a particular device.
608
+ [rank6]:[W917 02:54:57.228282259 ProcessGroupNCCL.cpp:4789] [PG ID 0 PG GUID 0 Rank 6] using GPU 6 as device used by this process is currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect. You can specify device_id in init_process_group() to force use of a particular device.
609
+ [rank5]:[W917 02:54:57.228608053 ProcessGroupNCCL.cpp:4789] [PG ID 0 PG GUID 0 Rank 5] using GPU 5 as device used by this process is currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect. You can specify device_id in init_process_group() to force use of a particular device.
610
+ > compiling dataset index builder ...
611
+ make: Entering directory '/workspace/Megatron-MoE-Echo/megatron/core/datasets'
612
+ make: Nothing to be done for 'default'.
613
+ make: Leaving directory '/workspace/Megatron-MoE-Echo/megatron/core/datasets'
614
+ >>> done with dataset index builder. Compilation time: 0.429 seconds
615
+ WARNING: constraints for invoking optimized fused softmax kernel are not met. We default back to unfused kernel invocations.
616
+ > compiling and loading fused kernels ...
617
+ [rank0]:[W917 02:54:57.830667453 ProcessGroupNCCL.cpp:4789] [PG ID 0 PG GUID 0 Rank 0] using GPU 0 as device used by this process is currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect. You can specify device_id in init_process_group() to force use of a particular device.
618
+ >>> done with compiling and loading fused kernels. Compilation time: 3.957 seconds
619
+ time to initialize megatron (seconds): 9.916
620
+ [after megatron is initialized] datetime: 2026-09-17 02:55:03
621
+ building GPT model ...
622
+ /usr/local/lib/python3.12/dist-packages/transformer_engine/pytorch/cpu_offload.py:708: DeprecationWarning: Offloading weights is deprecated. Using offload_weights=True does not have any effect.
623
+ warnings.warn(
624
+ /usr/local/lib/python3.12/dist-packages/transformer_engine/pytorch/cpu_offload.py:708: DeprecationWarning: Offloading weights is deprecated. Using offload_weights=True does not have any effect.
625
+ warnings.warn(
626
+ /usr/local/lib/python3.12/dist-packages/transformer_engine/pytorch/cpu_offload.py:708: DeprecationWarning: Offloading weights is deprecated. Using offload_weights=True does not have any effect.
627
+ warnings.warn(
628
+ /usr/local/lib/python3.12/dist-packages/transformer_engine/pytorch/cpu_offload.py:708: DeprecationWarning: Offloading weights is deprecated. Using offload_weights=True does not have any effect.
629
+ warnings.warn(
630
+ /usr/local/lib/python3.12/dist-packages/transformer_engine/pytorch/cpu_offload.py:708: DeprecationWarning: Offloading weights is deprecated. Using offload_weights=True does not have any effect.
631
+ warnings.warn(
632
+ /usr/local/lib/python3.12/dist-packages/transformer_engine/pytorch/cpu_offload.py:708: DeprecationWarning: Offloading weights is deprecated. Using offload_weights=True does not have any effect.
633
+ warnings.warn(
634
+ /usr/local/lib/python3.12/dist-packages/transformer_engine/pytorch/cpu_offload.py:708: DeprecationWarning: Offloading weights is deprecated. Using offload_weights=True does not have any effect.
635
+ warnings.warn(
636
+ /usr/local/lib/python3.12/dist-packages/transformer_engine/pytorch/cpu_offload.py:708: DeprecationWarning: Offloading weights is deprecated. Using offload_weights=True does not have any effect.
637
+ warnings.warn(
638
+ > number of parameters on (tensor, pipeline) model parallel rank (0, 0): 3910275072
639
+ INFO:megatron.core.distributed.distributed_data_parallel:Setting up DistributedDataParallel with config DistributedDataParallelConfig(grad_reduce_in_fp32=True, overlap_grad_reduce=True, overlap_param_gather=True, align_param_gather=False, use_distributed_optimizer=True, num_distributed_optimizer_instances=1, check_for_nan_in_grad=False, check_for_large_grads=False, bucket_size=40000000, pad_buckets_for_high_nccl_busbw=False, average_in_collective=False, fp8_param_gather=False, reuse_grad_buf_for_mxfp8_param_ag=False, use_custom_fsdp=False, data_parallel_sharding_strategy='no_shard', gradient_reduce_div_fusion=True, suggested_communication_unit_size=None, preserve_fp32_weights=True, keep_fp8_transpose_cache_when_using_custom_fsdp=False, nccl_ub=False, fsdp_double_buffer=False)
640
+ INFO:megatron.core.distributed.param_and_grad_buffer:Number of buckets for gradient all-reduce / reduce-scatter: 18
641
+ Params for bucket 1 (206045184 elements, 206045184 padded size):
642
+ module.output_layer.weight
643
+ Params for bucket 2 (42479616 elements, 42479616 padded size):
644
+ module.decoder.final_layernorm.weight
645
+ module.decoder.layers.15.mlp.router.weight
646
+ module.decoder.layers.15.self_attention.linear_qkv.layer_norm_weight
647
+ module.decoder.layers.15.pre_mlp_layernorm.weight
648
+ module.decoder.layers.15.self_attention.linear_qkv.weight
649
+ module.decoder.layers.15.self_attention.linear_proj.weight
650
+ Params for bucket 3 (42475520 elements, 42475520 padded size):
651
+ module.decoder.layers.14.self_attention.linear_qkv.weight
652
+ module.decoder.layers.14.pre_mlp_layernorm.weight
653
+ module.decoder.layers.14.mlp.router.weight
654
+ module.decoder.layers.14.self_attention.linear_proj.weight
655
+ module.decoder.layers.14.self_attention.linear_qkv.layer_norm_weight
656
+ Params for bucket 4 (42475520 elements, 42475520 padded size):
657
+ module.decoder.layers.13.mlp.router.weight
658
+ module.decoder.layers.13.self_attention.linear_qkv.layer_norm_weight
659
+ module.decoder.layers.13.self_attention.linear_proj.weight
660
+ module.decoder.layers.13.pre_mlp_layernorm.weight
661
+ module.decoder.layers.13.self_attention.linear_qkv.weight
662
+ Params for bucket 5 (42475520 elements, 42475520 padded size):
663
+ module.decoder.layers.12.pre_mlp_layernorm.weight
664
+ module.decoder.layers.12.mlp.router.weight
665
+ module.decoder.layers.12.self_attention.linear_qkv.weight
666
+ module.decoder.layers.12.self_attention.linear_qkv.layer_norm_weight
667
+ module.decoder.layers.12.self_attention.linear_proj.weight
668
+ Params for bucket 6 (42475520 elements, 42475520 padded size):
669
+ module.decoder.layers.11.mlp.router.weight
670
+ module.decoder.layers.11.self_attention.linear_qkv.weight
671
+ module.decoder.layers.11.pre_mlp_layernorm.weight
672
+ module.decoder.layers.11.self_attention.linear_qkv.layer_norm_weight
673
+ module.decoder.layers.11.self_attention.linear_proj.weight
674
+ Params for bucket 7 (42475520 elements, 42475520 padded size):
675
+ module.decoder.layers.10.self_attention.linear_qkv.weight
676
+ module.decoder.layers.10.pre_mlp_layernorm.weight
677
+ module.decoder.layers.10.self_attention.linear_proj.weight
678
+ module.decoder.layers.10.mlp.router.weight
679
+ module.decoder.layers.10.self_attention.linear_qkv.layer_norm_weight
680
+ Params for bucket 8 (42475520 elements, 42475520 padded size):
681
+ module.decoder.layers.9.mlp.router.weight
682
+ module.decoder.layers.9.self_attention.linear_proj.weight
683
+ module.decoder.layers.9.self_attention.linear_qkv.weight
684
+ module.decoder.layers.9.self_attention.linear_qkv.layer_norm_weight
685
+ module.decoder.layers.9.pre_mlp_layernorm.weight
686
+ Params for bucket 9 (42475520 elements, 42475520 padded size):
687
+ module.decoder.layers.8.mlp.router.weight
688
+ module.decoder.layers.8.self_attention.linear_qkv.weight
689
+ module.decoder.layers.8.pre_mlp_layernorm.weight
690
+ module.decoder.layers.8.self_attention.linear_proj.weight
691
+ module.decoder.layers.8.self_attention.linear_qkv.layer_norm_weight
692
+ Params for bucket 10 (42475520 elements, 42475520 padded size):
693
+ module.decoder.layers.7.self_attention.linear_qkv.weight
694
+ module.decoder.layers.7.pre_mlp_layernorm.weight
695
+ module.decoder.layers.7.self_attention.linear_proj.weight
696
+ module.decoder.layers.7.mlp.router.weight
697
+ module.decoder.layers.7.self_attention.linear_qkv.layer_norm_weight
698
+ Params for bucket 11 (42475520 elements, 42475520 padded size):
699
+ module.decoder.layers.6.mlp.router.weight
700
+ module.decoder.layers.6.pre_mlp_layernorm.weight
701
+ module.decoder.layers.6.self_attention.linear_qkv.weight
702
+ module.decoder.layers.6.self_attention.linear_qkv.layer_norm_weight
703
+ module.decoder.layers.6.self_attention.linear_proj.weight
704
+ Params for bucket 12 (42475520 elements, 42475520 padded size):
705
+ module.decoder.layers.5.self_attention.linear_qkv.weight
706
+ module.decoder.layers.5.pre_mlp_layernorm.weight
707
+ module.decoder.layers.5.mlp.router.weight
708
+ module.decoder.layers.5.self_attention.linear_proj.weight
709
+ module.decoder.layers.5.self_attention.linear_qkv.layer_norm_weight
710
+ Params for bucket 13 (42475520 elements, 42475520 padded size):
711
+ module.decoder.layers.4.pre_mlp_layernorm.weight
712
+ module.decoder.layers.4.self_attention.linear_qkv.weight
713
+ module.decoder.layers.4.mlp.router.weight
714
+ module.decoder.layers.4.self_attention.linear_proj.weight
715
+ module.decoder.layers.4.self_attention.linear_qkv.layer_norm_weight
716
+ Params for bucket 14 (42475520 elements, 42475520 padded size):
717
+ module.decoder.layers.3.self_attention.linear_qkv.weight
718
+ module.decoder.layers.3.self_attention.linear_proj.weight
719
+ module.decoder.layers.3.pre_mlp_layernorm.weight
720
+ module.decoder.layers.3.mlp.router.weight
721
+ module.decoder.layers.3.self_attention.linear_qkv.layer_norm_weight
722
+ Params for bucket 15 (42475520 elements, 42475520 padded size):
723
+ module.decoder.layers.2.pre_mlp_layernorm.weight
724
+ module.decoder.layers.2.self_attention.linear_proj.weight
725
+ module.decoder.layers.2.self_attention.linear_qkv.weight
726
+ module.decoder.layers.2.mlp.router.weight
727
+ module.decoder.layers.2.self_attention.linear_qkv.layer_norm_weight
728
+ Params for bucket 16 (42475520 elements, 42475520 padded size):
729
+ module.decoder.layers.1.mlp.router.weight
730
+ module.decoder.layers.1.self_attention.linear_proj.weight
731
+ module.decoder.layers.1.self_attention.linear_qkv.layer_norm_weight
732
+ module.decoder.layers.1.pre_mlp_layernorm.weight
733
+ module.decoder.layers.1.self_attention.linear_qkv.weight
734
+ Params for bucket 17 (42475520 elements, 42475520 padded size):
735
+ module.decoder.layers.0.mlp.router.weight
736
+ module.decoder.layers.0.self_attention.linear_qkv.weight
737
+ module.decoder.layers.0.self_attention.linear_proj.weight
738
+ module.decoder.layers.0.pre_mlp_layernorm.weight
739
+ module.decoder.layers.0.self_attention.linear_qkv.layer_norm_weight
740
+ Params for bucket 18 (206045184 elements, 206045184 padded size):
741
+ module.embedding.word_embeddings.weight
742
+ INFO:megatron.core.distributed.param_and_grad_buffer:Number of buckets for gradient all-reduce / reduce-scatter: 67
743
+ Params for bucket 1 (40370176 elements, 40370176 padded size):
744
+ module.decoder.layers.15.mlp.experts.linear_fc2.weight12
745
+ module.decoder.layers.15.mlp.experts.linear_fc2.weight14
746
+ module.decoder.layers.15.mlp.experts.linear_fc2.weight10
747
+ module.decoder.layers.15.mlp.experts.linear_fc2.weight8
748
+ module.decoder.layers.15.mlp.experts.linear_fc2.weight6
749
+ module.decoder.layers.15.mlp.experts.linear_fc2.weight15
750
+ module.decoder.layers.15.mlp.experts.linear_fc2.weight11
751
+ module.decoder.layers.15.mlp.experts.linear_fc2.weight13
752
+ module.decoder.layers.15.mlp.experts.linear_fc2.weight9
753
+ module.decoder.layers.15.mlp.experts.linear_fc2.weight7
754
+ module.decoder.layers.15.mlp.experts.linear_fc2.weight5
755
+ Params for bucket 2 (40370176 elements, 40370176 padded size):
756
+ module.decoder.layers.15.mlp.experts.linear_fc2.weight4
757
+ module.decoder.layers.15.mlp.experts.linear_fc2.weight2
758
+ module.decoder.layers.15.mlp.experts.linear_fc2.weight0
759
+ module.decoder.layers.15.mlp.experts.linear_fc1.weight15
760
+ module.decoder.layers.15.mlp.experts.linear_fc1.weight13
761
+ module.decoder.layers.15.mlp.experts.linear_fc2.weight3
762
+ module.decoder.layers.15.mlp.experts.linear_fc2.weight1
763
+ module.decoder.layers.15.mlp.experts.linear_fc1.weight14
764
+ Params for bucket 3 (44040192 elements, 44040192 padded size):
765
+ module.decoder.layers.15.mlp.experts.linear_fc1.weight9
766
+ module.decoder.layers.15.mlp.experts.linear_fc1.weight11
767
+ module.decoder.layers.15.mlp.experts.linear_fc1.weight7
768
+ module.decoder.layers.15.mlp.experts.linear_fc1.weight12
769
+ module.decoder.layers.15.mlp.experts.linear_fc1.weight8
770
+ module.decoder.layers.15.mlp.experts.linear_fc1.weight10
771
+ Params for bucket 4 (44040192 elements, 44040192 padded size):
772
+ module.decoder.layers.15.mlp.experts.linear_fc1.weight3
773
+ module.decoder.layers.15.mlp.experts.linear_fc1.weight5
774
+ module.decoder.layers.15.mlp.experts.linear_fc1.weight1
775
+ module.decoder.layers.15.mlp.experts.linear_fc1.weight6
776
+ module.decoder.layers.15.mlp.experts.linear_fc1.weight2
777
+ module.decoder.layers.15.mlp.experts.linear_fc1.weight4
778
+ Params for bucket 5 (40370176 elements, 40370176 padded size):
779
+ module.decoder.layers.14.mlp.experts.linear_fc2.weight14
780
+ module.decoder.layers.14.mlp.experts.linear_fc2.weight12
781
+ module.decoder.layers.14.mlp.experts.linear_fc2.weight10
782
+ module.decoder.layers.14.mlp.experts.linear_fc2.weight8
783
+ module.decoder.layers.15.mlp.experts.linear_fc1.weight0
784
+ module.decoder.layers.14.mlp.experts.linear_fc2.weight13
785
+ module.decoder.layers.14.mlp.experts.linear_fc2.weight15
786
+ module.decoder.layers.14.mlp.experts.linear_fc2.weight11
787
+ module.decoder.layers.14.mlp.experts.linear_fc2.weight9
788
+ module.decoder.layers.14.mlp.experts.linear_fc2.weight7
789
+ Params for bucket 6 (40370176 elements, 40370176 padded size):
790
+ module.decoder.layers.14.mlp.experts.linear_fc2.weight6
791
+ module.decoder.layers.14.mlp.experts.linear_fc2.weight4
792
+ module.decoder.layers.14.mlp.experts.linear_fc2.weight1
793
+ module.decoder.layers.14.mlp.experts.linear_fc2.weight0
794
+ module.decoder.layers.14.mlp.experts.linear_fc1.weight14
795
+ module.decoder.layers.14.mlp.experts.linear_fc2.weight3
796
+ module.decoder.layers.14.mlp.experts.linear_fc2.weight2
797
+ module.decoder.layers.14.mlp.experts.linear_fc2.weight5
798
+ module.decoder.layers.14.mlp.experts.linear_fc1.weight15
799
+ Params for bucket 7 (44040192 elements, 44040192 padded size):
800
+ module.decoder.layers.14.mlp.experts.linear_fc1.weight10
801
+ module.decoder.layers.14.mlp.experts.linear_fc1.weight12
802
+ module.decoder.layers.14.mlp.experts.linear_fc1.weight8
803
+ module.decoder.layers.14.mlp.experts.linear_fc1.weight13
804
+ module.decoder.layers.14.mlp.experts.linear_fc1.weight9
805
+ module.decoder.layers.14.mlp.experts.linear_fc1.weight11
806
+ Params for bucket 8 (44040192 elements, 44040192 padded size):
807
+ module.decoder.layers.14.mlp.experts.linear_fc1.weight6
808
+ module.decoder.layers.14.mlp.experts.linear_fc1.weight4
809
+ module.decoder.layers.14.mlp.experts.linear_fc1.weight2
810
+ module.decoder.layers.14.mlp.experts.linear_fc1.weight7
811
+ module.decoder.layers.14.mlp.experts.linear_fc1.weight3
812
+ module.decoder.layers.14.mlp.experts.linear_fc1.weight5
813
+ Params for bucket 9 (40370176 elements, 40370176 padded size):
814
+ module.decoder.layers.14.mlp.experts.linear_fc1.weight0
815
+ module.decoder.layers.13.mlp.experts.linear_fc2.weight15
816
+ module.decoder.layers.13.mlp.experts.linear_fc2.weight13
817
+ module.decoder.layers.13.mlp.experts.linear_fc2.weight11
818
+ module.decoder.layers.13.mlp.experts.linear_fc2.weight9
819
+ module.decoder.layers.14.mlp.experts.linear_fc1.weight1
820
+ module.decoder.layers.13.mlp.experts.linear_fc2.weight14
821
+ module.decoder.layers.13.mlp.experts.linear_fc2.weight12
822
+ module.decoder.layers.13.mlp.experts.linear_fc2.weight10
823
+ Params for bucket 10 (40370176 elements, 40370176 padded size):
824
+ module.decoder.layers.13.mlp.experts.linear_fc2.weight7
825
+ module.decoder.layers.13.mlp.experts.linear_fc2.weight5
826
+ module.decoder.layers.13.mlp.experts.linear_fc2.weight3
827
+ module.decoder.layers.13.mlp.experts.linear_fc2.weight1
828
+ module.decoder.layers.13.mlp.experts.linear_fc2.weight8
829
+ module.decoder.layers.13.mlp.experts.linear_fc2.weight4
830
+ module.decoder.layers.13.mlp.experts.linear_fc2.weight6
831
+ module.decoder.layers.13.mlp.experts.linear_fc2.weight2
832
+ module.decoder.layers.13.mlp.experts.linear_fc2.weight0
833
+ module.decoder.layers.13.mlp.experts.linear_fc1.weight15
834
+ Params for bucket 11 (44040192 elements, 44040192 padded size):
835
+ module.decoder.layers.13.mlp.experts.linear_fc1.weight14
836
+ module.decoder.layers.13.mlp.experts.linear_fc1.weight10
837
+ module.decoder.layers.13.mlp.experts.linear_fc1.weight12
838
+ module.decoder.layers.13.mlp.experts.linear_fc1.weight13
839
+ module.decoder.layers.13.mlp.experts.linear_fc1.weight11
840
+ module.decoder.layers.13.mlp.experts.linear_fc1.weight9
841
+ Params for bucket 12 (44040192 elements, 44040192 padded size):
842
+ module.decoder.layers.13.mlp.experts.linear_fc1.weight8
843
+ module.decoder.layers.13.mlp.experts.linear_fc1.weight4
844
+ module.decoder.layers.13.mlp.experts.linear_fc1.weight6
845
+ module.decoder.layers.13.mlp.experts.linear_fc1.weight7
846
+ module.decoder.layers.13.mlp.experts.linear_fc1.weight5
847
+ module.decoder.layers.13.mlp.experts.linear_fc1.weight3
848
+ Params for bucket 13 (40370176 elements, 40370176 padded size):
849
+ module.decoder.layers.13.mlp.experts.linear_fc1.weight2
850
+ module.decoder.layers.13.mlp.experts.linear_fc1.weight0
851
+ module.decoder.layers.12.mlp.experts.linear_fc2.weight14
852
+ module.decoder.layers.12.mlp.experts.linear_fc2.weight12
853
+ module.decoder.layers.13.mlp.experts.linear_fc1.weight1
854
+ module.decoder.layers.12.mlp.experts.linear_fc2.weight15
855
+ module.decoder.layers.12.mlp.experts.linear_fc2.weight13
856
+ module.decoder.layers.12.mlp.experts.linear_fc2.weight11
857
+ Params for bucket 14 (40370176 elements, 40370176 padded size):
858
+ module.decoder.layers.12.mlp.experts.linear_fc2.weight10
859
+ module.decoder.layers.12.mlp.experts.linear_fc2.weight6
860
+ module.decoder.layers.12.mlp.experts.linear_fc2.weight8
861
+ module.decoder.layers.12.mlp.experts.linear_fc2.weight4
862
+ module.decoder.layers.12.mlp.experts.linear_fc2.weight2
863
+ module.decoder.layers.12.mlp.experts.linear_fc2.weight0
864
+ module.decoder.layers.12.mlp.experts.linear_fc2.weight7
865
+ module.decoder.layers.12.mlp.experts.linear_fc2.weight9
866
+ module.decoder.layers.12.mlp.experts.linear_fc2.weight5
867
+ module.decoder.layers.12.mlp.experts.linear_fc2.weight3
868
+ module.decoder.layers.12.mlp.experts.linear_fc2.weight1
869
+ Params for bucket 15 (44040192 elements, 44040192 padded size):
870
+ module.decoder.layers.12.mlp.experts.linear_fc1.weight15
871
+ module.decoder.layers.12.mlp.experts.linear_fc1.weight13
872
+ module.decoder.layers.12.mlp.experts.linear_fc1.weight11
873
+ module.decoder.layers.12.mlp.experts.linear_fc1.weight14
874
+ module.decoder.layers.12.mlp.experts.linear_fc1.weight12
875
+ module.decoder.layers.12.mlp.experts.linear_fc1.weight10
876
+ Params for bucket 16 (44040192 elements, 44040192 padded size):
877
+ module.decoder.layers.12.mlp.experts.linear_fc1.weight9
878
+ module.decoder.layers.12.mlp.experts.linear_fc1.weight7
879
+ module.decoder.layers.12.mlp.experts.linear_fc1.weight5
880
+ module.decoder.layers.12.mlp.experts.linear_fc1.weight8
881
+ module.decoder.layers.12.mlp.experts.linear_fc1.weight6
882
+ module.decoder.layers.12.mlp.experts.linear_fc1.weight4
883
+ Params for bucket 17 (40370176 elements, 40370176 padded size):
884
+ module.decoder.layers.12.mlp.experts.linear_fc1.weight3
885
+ module.decoder.layers.12.mlp.experts.linear_fc1.weight1
886
+ module.decoder.layers.11.mlp.experts.linear_fc2.weight14
887
+ module.decoder.layers.12.mlp.experts.linear_fc1.weight0
888
+ module.decoder.layers.12.mlp.experts.linear_fc1.weight2
889
+ module.decoder.layers.11.mlp.experts.linear_fc2.weight15
890
+ module.decoder.layers.11.mlp.experts.linear_fc2.weight13
891
+ Params for bucket 18 (40370176 elements, 40370176 padded size):
892
+ module.decoder.layers.11.mlp.experts.linear_fc2.weight12
893
+ module.decoder.layers.11.mlp.experts.linear_fc2.weight8
894
+ module.decoder.layers.11.mlp.experts.linear_fc2.weight10
895
+ module.decoder.layers.11.mlp.experts.linear_fc2.weight6
896
+ module.decoder.layers.11.mlp.experts.linear_fc2.weight4
897
+ module.decoder.layers.11.mlp.experts.linear_fc2.weight2
898
+ module.decoder.layers.11.mlp.experts.linear_fc2.weight11
899
+ module.decoder.layers.11.mlp.experts.linear_fc2.weight9
900
+ module.decoder.layers.11.mlp.experts.linear_fc2.weight7
901
+ module.decoder.layers.11.mlp.experts.linear_fc2.weight5
902
+ module.decoder.layers.11.mlp.experts.linear_fc2.weight3
903
+ Params for bucket 19 (44040192 elements, 44040192 padded size):
904
+ module.decoder.layers.11.mlp.experts.linear_fc1.weight15
905
+ module.decoder.layers.11.mlp.experts.linear_fc1.weight13
906
+ module.decoder.layers.11.mlp.experts.linear_fc2.weight0
907
+ module.decoder.layers.11.mlp.experts.linear_fc1.weight11
908
+ module.decoder.layers.11.mlp.experts.linear_fc2.weight1
909
+ module.decoder.layers.11.mlp.experts.linear_fc1.weight14
910
+ module.decoder.layers.11.mlp.experts.linear_fc1.weight12
911
+ Params for bucket 20 (44040192 elements, 44040192 padded size):
912
+ module.decoder.layers.11.mlp.experts.linear_fc1.weight9
913
+ module.decoder.layers.11.mlp.experts.linear_fc1.weight7
914
+ module.decoder.layers.11.mlp.experts.linear_fc1.weight5
915
+ module.decoder.layers.11.mlp.experts.linear_fc1.weight10
916
+ module.decoder.layers.11.mlp.experts.linear_fc1.weight8
917
+ module.decoder.layers.11.mlp.experts.linear_fc1.weight6
918
+ Params for bucket 21 (40370176 elements, 40370176 padded size):
919
+ module.decoder.layers.11.mlp.experts.linear_fc1.weight1
920
+ module.decoder.layers.11.mlp.experts.linear_fc1.weight3
921
+ module.decoder.layers.11.mlp.experts.linear_fc1.weight4
922
+ module.decoder.layers.11.mlp.experts.linear_fc1.weight2
923
+ module.decoder.layers.11.mlp.experts.linear_fc1.weight0
924
+ module.decoder.layers.10.mlp.experts.linear_fc2.weight15
925
+ Params for bucket 22 (40370176 elements, 40370176 padded size):
926
+ module.decoder.layers.10.mlp.experts.linear_fc2.weight14
927
+ module.decoder.layers.10.mlp.experts.linear_fc2.weight12
928
+ module.decoder.layers.10.mlp.experts.linear_fc2.weight10
929
+ module.decoder.layers.10.mlp.experts.linear_fc2.weight8
930
+ module.decoder.layers.10.mlp.experts.linear_fc2.weight6
931
+ module.decoder.layers.10.mlp.experts.linear_fc2.weight4
932
+ module.decoder.layers.10.mlp.experts.linear_fc2.weight13
933
+ module.decoder.layers.10.mlp.experts.linear_fc2.weight11
934
+ module.decoder.layers.10.mlp.experts.linear_fc2.weight9
935
+ module.decoder.layers.10.mlp.experts.linear_fc2.weight7
936
+ module.decoder.layers.10.mlp.experts.linear_fc2.weight5
937
+ Params for bucket 23 (44040192 elements, 44040192 padded size):
938
+ module.decoder.layers.10.mlp.experts.linear_fc1.weight15
939
+ module.decoder.layers.10.mlp.experts.linear_fc2.weight0
940
+ module.decoder.layers.10.mlp.experts.linear_fc2.weight2
941
+ module.decoder.layers.10.mlp.experts.linear_fc1.weight13
942
+ module.decoder.layers.10.mlp.experts.linear_fc2.weight3
943
+ module.decoder.layers.10.mlp.experts.linear_fc2.weight1
944
+ module.decoder.layers.10.mlp.experts.linear_fc1.weight14
945
+ module.decoder.layers.10.mlp.experts.linear_fc1.weight12
946
+ Params for bucket 24 (44040192 elements, 44040192 padded size):
947
+ module.decoder.layers.10.mlp.experts.linear_fc1.weight11
948
+ module.decoder.layers.10.mlp.experts.linear_fc1.weight9
949
+ module.decoder.layers.10.mlp.experts.linear_fc1.weight7
950
+ module.decoder.layers.10.mlp.experts.linear_fc1.weight10
951
+ module.decoder.layers.10.mlp.experts.linear_fc1.weight8
952
+ module.decoder.layers.10.mlp.experts.linear_fc1.weight6
953
+ Params for bucket 25 (44040192 elements, 44040192 padded size):
954
+ module.decoder.layers.10.mlp.experts.linear_fc1.weight5
955
+ module.decoder.layers.10.mlp.experts.linear_fc1.weight3
956
+ module.decoder.layers.10.mlp.experts.linear_fc1.weight1
957
+ module.decoder.layers.10.mlp.experts.linear_fc1.weight2
958
+ module.decoder.layers.10.mlp.experts.linear_fc1.weight4
959
+ module.decoder.layers.10.mlp.experts.linear_fc1.weight0
960
+ Params for bucket 26 (40370176 elements, 40370176 padded size):
961
+ module.decoder.layers.9.mlp.experts.linear_fc2.weight15
962
+ module.decoder.layers.9.mlp.experts.linear_fc2.weight11
963
+ module.decoder.layers.9.mlp.experts.linear_fc2.weight13
964
+ module.decoder.layers.9.mlp.experts.linear_fc2.weight9
965
+ module.decoder.layers.9.mlp.experts.linear_fc2.weight7
966
+ module.decoder.layers.9.mlp.experts.linear_fc2.weight5
967
+ module.decoder.layers.9.mlp.experts.linear_fc2.weight12
968
+ module.decoder.layers.9.mlp.experts.linear_fc2.weight14
969
+ module.decoder.layers.9.mlp.experts.linear_fc2.weight10
970
+ module.decoder.layers.9.mlp.experts.linear_fc2.weight8
971
+ module.decoder.layers.9.mlp.experts.linear_fc2.weight6
972
+ Params for bucket 27 (40370176 elements, 40370176 padded size):
973
+ module.decoder.layers.9.mlp.experts.linear_fc2.weight3
974
+ module.decoder.layers.9.mlp.experts.linear_fc2.weight1
975
+ module.decoder.layers.9.mlp.experts.linear_fc1.weight14
976
+ module.decoder.layers.9.mlp.experts.linear_fc2.weight4
977
+ module.decoder.layers.9.mlp.experts.linear_fc2.weight2
978
+ module.decoder.layers.9.mlp.experts.linear_fc2.weight0
979
+ module.decoder.layers.9.mlp.experts.linear_fc1.weight15
980
+ module.decoder.layers.9.mlp.experts.linear_fc1.weight13
981
+ Params for bucket 28 (44040192 elements, 44040192 padded size):
982
+ module.decoder.layers.9.mlp.experts.linear_fc1.weight12
983
+ module.decoder.layers.9.mlp.experts.linear_fc1.weight8
984
+ module.decoder.layers.9.mlp.experts.linear_fc1.weight10
985
+ module.decoder.layers.9.mlp.experts.linear_fc1.weight11
986
+ module.decoder.layers.9.mlp.experts.linear_fc1.weight9
987
+ module.decoder.layers.9.mlp.experts.linear_fc1.weight7
988
+ Params for bucket 29 (44040192 elements, 44040192 padded size):
989
+ module.decoder.layers.9.mlp.experts.linear_fc1.weight6
990
+ module.decoder.layers.9.mlp.experts.linear_fc1.weight2
991
+ module.decoder.layers.9.mlp.experts.linear_fc1.weight4
992
+ module.decoder.layers.9.mlp.experts.linear_fc1.weight5
993
+ module.decoder.layers.9.mlp.experts.linear_fc1.weight3
994
+ module.decoder.layers.9.mlp.experts.linear_fc1.weight1
995
+ Params for bucket 30 (40370176 elements, 40370176 padded size):
996
+ module.decoder.layers.9.mlp.experts.linear_fc1.weight0
997
+ module.decoder.layers.8.mlp.experts.linear_fc2.weight14
998
+ module.decoder.layers.8.mlp.experts.linear_fc2.weight12
999
+ module.decoder.layers.8.mlp.experts.linear_fc2.weight10
1000
+ module.decoder.layers.8.mlp.experts.linear_fc2.weight8
1001
+ module.decoder.layers.8.mlp.experts.linear_fc2.weight15
1002
+ module.decoder.layers.8.mlp.experts.linear_fc2.weight13
1003
+ module.decoder.layers.8.mlp.experts.linear_fc2.weight11
1004
+ module.decoder.layers.8.mlp.experts.linear_fc2.weight9
1005
+ module.decoder.layers.8.mlp.experts.linear_fc2.weight7
1006
+ Params for bucket 31 (40370176 elements, 40370176 padded size):
1007
+ module.decoder.layers.8.mlp.experts.linear_fc2.weight6
1008
+ module.decoder.layers.8.mlp.experts.linear_fc2.weight4
1009
+ module.decoder.layers.8.mlp.experts.linear_fc2.weight2
1010
+ module.decoder.layers.8.mlp.experts.linear_fc2.weight0
1011
+ module.decoder.layers.8.mlp.experts.linear_fc1.weight15
1012
+ module.decoder.layers.8.mlp.experts.linear_fc2.weight5
1013
+ module.decoder.layers.8.mlp.experts.linear_fc2.weight3
1014
+ module.decoder.layers.8.mlp.experts.linear_fc2.weight1
1015
+ module.decoder.layers.8.mlp.experts.linear_fc1.weight14
1016
+ Params for bucket 32 (44040192 elements, 44040192 padded size):
1017
+ module.decoder.layers.8.mlp.experts.linear_fc1.weight13
1018
+ module.decoder.layers.8.mlp.experts.linear_fc1.weight11
1019
+ module.decoder.layers.8.mlp.experts.linear_fc1.weight9
1020
+ module.decoder.layers.8.mlp.experts.linear_fc1.weight12
1021
+ module.decoder.layers.8.mlp.experts.linear_fc1.weight10
1022
+ module.decoder.layers.8.mlp.experts.linear_fc1.weight8
1023
+ Params for bucket 33 (44040192 elements, 44040192 padded size):
1024
+ module.decoder.layers.8.mlp.experts.linear_fc1.weight7
1025
+ module.decoder.layers.8.mlp.experts.linear_fc1.weight5
1026
+ module.decoder.layers.8.mlp.experts.linear_fc1.weight3
1027
+ module.decoder.layers.8.mlp.experts.linear_fc1.weight6
1028
+ module.decoder.layers.8.mlp.experts.linear_fc1.weight4
1029
+ module.decoder.layers.8.mlp.experts.linear_fc1.weight2
1030
+ Params for bucket 34 (40370176 elements, 40370176 padded size):
1031
+ module.decoder.layers.8.mlp.experts.linear_fc1.weight1
1032
+ module.decoder.layers.7.mlp.experts.linear_fc2.weight14
1033
+ module.decoder.layers.7.mlp.experts.linear_fc2.weight12
1034
+ module.decoder.layers.7.mlp.experts.linear_fc2.weight10
1035
+ module.decoder.layers.7.mlp.experts.linear_fc2.weight15
1036
+ module.decoder.layers.7.mlp.experts.linear_fc2.weight13
1037
+ module.decoder.layers.8.mlp.experts.linear_fc1.weight0
1038
+ module.decoder.layers.7.mlp.experts.linear_fc2.weight11
1039
+ module.decoder.layers.7.mlp.experts.linear_fc2.weight9
1040
+ Params for bucket 35 (40370176 elements, 40370176 padded size):
1041
+ module.decoder.layers.7.mlp.experts.linear_fc2.weight8
1042
+ module.decoder.layers.7.mlp.experts.linear_fc2.weight6
1043
+ module.decoder.layers.7.mlp.experts.linear_fc2.weight4
1044
+ module.decoder.layers.7.mlp.experts.linear_fc2.weight2
1045
+ module.decoder.layers.7.mlp.experts.linear_fc2.weight0
1046
+ module.decoder.layers.7.mlp.experts.linear_fc1.weight15
1047
+ module.decoder.layers.7.mlp.experts.linear_fc2.weight7
1048
+ module.decoder.layers.7.mlp.experts.linear_fc2.weight5
1049
+ module.decoder.layers.7.mlp.experts.linear_fc2.weight3
1050
+ module.decoder.layers.7.mlp.experts.linear_fc2.weight1
1051
+ Params for bucket 36 (44040192 elements, 44040192 padded size):
1052
+ module.decoder.layers.7.mlp.experts.linear_fc1.weight11
1053
+ module.decoder.layers.7.mlp.experts.linear_fc1.weight13
1054
+ module.decoder.layers.7.mlp.experts.linear_fc1.weight9
1055
+ module.decoder.layers.7.mlp.experts.linear_fc1.weight14
1056
+ module.decoder.layers.7.mlp.experts.linear_fc1.weight12
1057
+ module.decoder.layers.7.mlp.experts.linear_fc1.weight10
1058
+ Params for bucket 37 (44040192 elements, 44040192 padded size):
1059
+ module.decoder.layers.7.mlp.experts.linear_fc1.weight5
1060
+ module.decoder.layers.7.mlp.experts.linear_fc1.weight7
1061
+ module.decoder.layers.7.mlp.experts.linear_fc1.weight3
1062
+ module.decoder.layers.7.mlp.experts.linear_fc1.weight8
1063
+ module.decoder.layers.7.mlp.experts.linear_fc1.weight6
1064
+ module.decoder.layers.7.mlp.experts.linear_fc1.weight4
1065
+ Params for bucket 38 (40370176 elements, 40370176 padded size):
1066
+ module.decoder.layers.6.mlp.experts.linear_fc2.weight15
1067
+ module.decoder.layers.7.mlp.experts.linear_fc1.weight1
1068
+ module.decoder.layers.6.mlp.experts.linear_fc2.weight13
1069
+ module.decoder.layers.6.mlp.experts.linear_fc2.weight11
1070
+ module.decoder.layers.7.mlp.experts.linear_fc1.weight2
1071
+ module.decoder.layers.7.mlp.experts.linear_fc1.weight0
1072
+ module.decoder.layers.6.mlp.experts.linear_fc2.weight14
1073
+ module.decoder.layers.6.mlp.experts.linear_fc2.weight12
1074
+ Params for bucket 39 (40370176 elements, 40370176 padded size):
1075
+ module.decoder.layers.6.mlp.experts.linear_fc2.weight9
1076
+ module.decoder.layers.6.mlp.experts.linear_fc2.weight7
1077
+ module.decoder.layers.6.mlp.experts.linear_fc2.weight5
1078
+ module.decoder.layers.6.mlp.experts.linear_fc2.weight3
1079
+ module.decoder.layers.6.mlp.experts.linear_fc2.weight1
1080
+ module.decoder.layers.6.mlp.experts.linear_fc2.weight10
1081
+ module.decoder.layers.6.mlp.experts.linear_fc2.weight8
1082
+ module.decoder.layers.6.mlp.experts.linear_fc2.weight6
1083
+ module.decoder.layers.6.mlp.experts.linear_fc2.weight4
1084
+ module.decoder.layers.6.mlp.experts.linear_fc2.weight2
1085
+ module.decoder.layers.6.mlp.experts.linear_fc2.weight0
1086
+ Params for bucket 40 (44040192 elements, 44040192 padded size):
1087
+ module.decoder.layers.6.mlp.experts.linear_fc1.weight14
1088
+ module.decoder.layers.6.mlp.experts.linear_fc1.weight12
1089
+ module.decoder.layers.6.mlp.experts.linear_fc1.weight10
1090
+ module.decoder.layers.6.mlp.experts.linear_fc1.weight15
1091
+ module.decoder.layers.6.mlp.experts.linear_fc1.weight13
1092
+ module.decoder.layers.6.mlp.experts.linear_fc1.weight11
1093
+ Params for bucket 41 (44040192 elements, 44040192 padded size):
1094
+ module.decoder.layers.6.mlp.experts.linear_fc1.weight8
1095
+ module.decoder.layers.6.mlp.experts.linear_fc1.weight6
1096
+ module.decoder.layers.6.mlp.experts.linear_fc1.weight4
1097
+ module.decoder.layers.6.mlp.experts.linear_fc1.weight9
1098
+ module.decoder.layers.6.mlp.experts.linear_fc1.weight5
1099
+ module.decoder.layers.6.mlp.experts.linear_fc1.weight7
1100
+ Params for bucket 42 (40370176 elements, 40370176 padded size):
1101
+ module.decoder.layers.6.mlp.experts.linear_fc1.weight2
1102
+ module.decoder.layers.6.mlp.experts.linear_fc1.weight0
1103
+ module.decoder.layers.5.mlp.experts.linear_fc2.weight14
1104
+ module.decoder.layers.6.mlp.experts.linear_fc1.weight3
1105
+ module.decoder.layers.6.mlp.experts.linear_fc1.weight1
1106
+ module.decoder.layers.5.mlp.experts.linear_fc2.weight15
1107
+ module.decoder.layers.5.mlp.experts.linear_fc2.weight13
1108
+ Params for bucket 43 (40370176 elements, 40370176 padded size):
1109
+ module.decoder.layers.5.mlp.experts.linear_fc2.weight12
1110
+ module.decoder.layers.5.mlp.experts.linear_fc2.weight8
1111
+ module.decoder.layers.5.mlp.experts.linear_fc2.weight10
1112
+ module.decoder.layers.5.mlp.experts.linear_fc2.weight6
1113
+ module.decoder.layers.5.mlp.experts.linear_fc2.weight4
1114
+ module.decoder.layers.5.mlp.experts.linear_fc2.weight2
1115
+ module.decoder.layers.5.mlp.experts.linear_fc2.weight11
1116
+ module.decoder.layers.5.mlp.experts.linear_fc2.weight9
1117
+ module.decoder.layers.5.mlp.experts.linear_fc2.weight7
1118
+ module.decoder.layers.5.mlp.experts.linear_fc2.weight5
1119
+ module.decoder.layers.5.mlp.experts.linear_fc2.weight3
1120
+ Params for bucket 44 (44040192 elements, 44040192 padded size):
1121
+ module.decoder.layers.5.mlp.experts.linear_fc1.weight15
1122
+ module.decoder.layers.5.mlp.experts.linear_fc1.weight13
1123
+ module.decoder.layers.5.mlp.experts.linear_fc2.weight0
1124
+ module.decoder.layers.5.mlp.experts.linear_fc1.weight11
1125
+ module.decoder.layers.5.mlp.experts.linear_fc2.weight1
1126
+ module.decoder.layers.5.mlp.experts.linear_fc1.weight14
1127
+ module.decoder.layers.5.mlp.experts.linear_fc1.weight12
1128
+ Params for bucket 45 (44040192 elements, 44040192 padded size):
1129
+ module.decoder.layers.5.mlp.experts.linear_fc1.weight9
1130
+ module.decoder.layers.5.mlp.experts.linear_fc1.weight7
1131
+ module.decoder.layers.5.mlp.experts.linear_fc1.weight5
1132
+ module.decoder.layers.5.mlp.experts.linear_fc1.weight10
1133
+ module.decoder.layers.5.mlp.experts.linear_fc1.weight6
1134
+ module.decoder.layers.5.mlp.experts.linear_fc1.weight8
1135
+ Params for bucket 46 (40370176 elements, 40370176 padded size):
1136
+ module.decoder.layers.5.mlp.experts.linear_fc1.weight3
1137
+ module.decoder.layers.5.mlp.experts.linear_fc1.weight1
1138
+ module.decoder.layers.4.mlp.experts.linear_fc2.weight15
1139
+ module.decoder.layers.5.mlp.experts.linear_fc1.weight4
1140
+ module.decoder.layers.5.mlp.experts.linear_fc1.weight2
1141
+ module.decoder.layers.5.mlp.experts.linear_fc1.weight0
1142
+ Params for bucket 47 (40370176 elements, 40370176 padded size):
1143
+ module.decoder.layers.4.mlp.experts.linear_fc2.weight13
1144
+ module.decoder.layers.4.mlp.experts.linear_fc2.weight11
1145
+ module.decoder.layers.4.mlp.experts.linear_fc2.weight9
1146
+ module.decoder.layers.4.mlp.experts.linear_fc2.weight7
1147
+ module.decoder.layers.4.mlp.experts.linear_fc2.weight5
1148
+ module.decoder.layers.4.mlp.experts.linear_fc2.weight14
1149
+ module.decoder.layers.4.mlp.experts.linear_fc2.weight12
1150
+ module.decoder.layers.4.mlp.experts.linear_fc2.weight10
1151
+ module.decoder.layers.4.mlp.experts.linear_fc2.weight8
1152
+ module.decoder.layers.4.mlp.experts.linear_fc2.weight6
1153
+ module.decoder.layers.4.mlp.experts.linear_fc2.weight4
1154
+ Params for bucket 48 (44040192 elements, 44040192 padded size):
1155
+ module.decoder.layers.4.mlp.experts.linear_fc2.weight3
1156
+ module.decoder.layers.4.mlp.experts.linear_fc2.weight1
1157
+ module.decoder.layers.4.mlp.experts.linear_fc1.weight14
1158
+ module.decoder.layers.4.mlp.experts.linear_fc1.weight12
1159
+ module.decoder.layers.4.mlp.experts.linear_fc2.weight0
1160
+ module.decoder.layers.4.mlp.experts.linear_fc1.weight15
1161
+ module.decoder.layers.4.mlp.experts.linear_fc2.weight2
1162
+ module.decoder.layers.4.mlp.experts.linear_fc1.weight13
1163
+ Params for bucket 49 (44040192 elements, 44040192 padded size):
1164
+ module.decoder.layers.4.mlp.experts.linear_fc1.weight10
1165
+ module.decoder.layers.4.mlp.experts.linear_fc1.weight8
1166
+ module.decoder.layers.4.mlp.experts.linear_fc1.weight6
1167
+ module.decoder.layers.4.mlp.experts.linear_fc1.weight11
1168
+ module.decoder.layers.4.mlp.experts.linear_fc1.weight9
1169
+ module.decoder.layers.4.mlp.experts.linear_fc1.weight7
1170
+ Params for bucket 50 (44040192 elements, 44040192 padded size):
1171
+ module.decoder.layers.4.mlp.experts.linear_fc1.weight2
1172
+ module.decoder.layers.4.mlp.experts.linear_fc1.weight4
1173
+ module.decoder.layers.4.mlp.experts.linear_fc1.weight0
1174
+ module.decoder.layers.4.mlp.experts.linear_fc1.weight5
1175
+ module.decoder.layers.4.mlp.experts.linear_fc1.weight3
1176
+ module.decoder.layers.4.mlp.experts.linear_fc1.weight1
1177
+ Params for bucket 51 (40370176 elements, 40370176 padded size):
1178
+ module.decoder.layers.3.mlp.experts.linear_fc2.weight14
1179
+ module.decoder.layers.3.mlp.experts.linear_fc2.weight12
1180
+ module.decoder.layers.3.mlp.experts.linear_fc2.weight10
1181
+ module.decoder.layers.3.mlp.experts.linear_fc2.weight8
1182
+ module.decoder.layers.3.mlp.experts.linear_fc2.weight6
1183
+ module.decoder.layers.3.mlp.experts.linear_fc2.weight15
1184
+ module.decoder.layers.3.mlp.experts.linear_fc2.weight13
1185
+ module.decoder.layers.3.mlp.experts.linear_fc2.weight11
1186
+ module.decoder.layers.3.mlp.experts.linear_fc2.weight9
1187
+ module.decoder.layers.3.mlp.experts.linear_fc2.weight7
1188
+ module.decoder.layers.3.mlp.experts.linear_fc2.weight5
1189
+ Params for bucket 52 (40370176 elements, 40370176 padded size):
1190
+ module.decoder.layers.3.mlp.experts.linear_fc2.weight4
1191
+ module.decoder.layers.3.mlp.experts.linear_fc2.weight0
1192
+ module.decoder.layers.3.mlp.experts.linear_fc2.weight2
1193
+ module.decoder.layers.3.mlp.experts.linear_fc1.weight14
1194
+ module.decoder.layers.3.mlp.experts.linear_fc2.weight1
1195
+ module.decoder.layers.3.mlp.experts.linear_fc2.weight3
1196
+ module.decoder.layers.3.mlp.experts.linear_fc1.weight15
1197
+ module.decoder.layers.3.mlp.experts.linear_fc1.weight13
1198
+ Params for bucket 53 (44040192 elements, 44040192 padded size):
1199
+ module.decoder.layers.3.mlp.experts.linear_fc1.weight12
1200
+ module.decoder.layers.3.mlp.experts.linear_fc1.weight8
1201
+ module.decoder.layers.3.mlp.experts.linear_fc1.weight10
1202
+ module.decoder.layers.3.mlp.experts.linear_fc1.weight9
1203
+ module.decoder.layers.3.mlp.experts.linear_fc1.weight11
1204
+ module.decoder.layers.3.mlp.experts.linear_fc1.weight7
1205
+ Params for bucket 54 (44040192 elements, 44040192 padded size):
1206
+ module.decoder.layers.3.mlp.experts.linear_fc1.weight6
1207
+ module.decoder.layers.3.mlp.experts.linear_fc1.weight4
1208
+ module.decoder.layers.3.mlp.experts.linear_fc1.weight2
1209
+ module.decoder.layers.3.mlp.experts.linear_fc1.weight5
1210
+ module.decoder.layers.3.mlp.experts.linear_fc1.weight3
1211
+ module.decoder.layers.3.mlp.experts.linear_fc1.weight1
1212
+ Params for bucket 55 (40370176 elements, 40370176 padded size):
1213
+ module.decoder.layers.3.mlp.experts.linear_fc1.weight0
1214
+ module.decoder.layers.2.mlp.experts.linear_fc2.weight12
1215
+ module.decoder.layers.2.mlp.experts.linear_fc2.weight14
1216
+ module.decoder.layers.2.mlp.experts.linear_fc2.weight10
1217
+ module.decoder.layers.2.mlp.experts.linear_fc2.weight8
1218
+ module.decoder.layers.2.mlp.experts.linear_fc2.weight15
1219
+ module.decoder.layers.2.mlp.experts.linear_fc2.weight13
1220
+ module.decoder.layers.2.mlp.experts.linear_fc2.weight11
1221
+ module.decoder.layers.2.mlp.experts.linear_fc2.weight9
1222
+ module.decoder.layers.2.mlp.experts.linear_fc2.weight7
1223
+ Params for bucket 56 (40370176 elements, 40370176 padded size):
1224
+ module.decoder.layers.2.mlp.experts.linear_fc2.weight6
1225
+ module.decoder.layers.2.mlp.experts.linear_fc2.weight2
1226
+ module.decoder.layers.2.mlp.experts.linear_fc2.weight4
1227
+ module.decoder.layers.2.mlp.experts.linear_fc2.weight0
1228
+ module.decoder.layers.2.mlp.experts.linear_fc1.weight14
1229
+ module.decoder.layers.2.mlp.experts.linear_fc2.weight3
1230
+ module.decoder.layers.2.mlp.experts.linear_fc2.weight5
1231
+ module.decoder.layers.2.mlp.experts.linear_fc2.weight1
1232
+ module.decoder.layers.2.mlp.experts.linear_fc1.weight15
1233
+ Params for bucket 57 (44040192 elements, 44040192 padded size):
1234
+ module.decoder.layers.2.mlp.experts.linear_fc1.weight12
1235
+ module.decoder.layers.2.mlp.experts.linear_fc1.weight10
1236
+ module.decoder.layers.2.mlp.experts.linear_fc1.weight8
1237
+ module.decoder.layers.2.mlp.experts.linear_fc1.weight13
1238
+ module.decoder.layers.2.mlp.experts.linear_fc1.weight11
1239
+ module.decoder.layers.2.mlp.experts.linear_fc1.weight9
1240
+ Params for bucket 58 (44040192 elements, 44040192 padded size):
1241
+ module.decoder.layers.2.mlp.experts.linear_fc1.weight6
1242
+ module.decoder.layers.2.mlp.experts.linear_fc1.weight4
1243
+ module.decoder.layers.2.mlp.experts.linear_fc1.weight2
1244
+ module.decoder.layers.2.mlp.experts.linear_fc1.weight7
1245
+ module.decoder.layers.2.mlp.experts.linear_fc1.weight5
1246
+ module.decoder.layers.2.mlp.experts.linear_fc1.weight3
1247
+ Params for bucket 59 (40370176 elements, 40370176 padded size):
1248
+ module.decoder.layers.1.mlp.experts.linear_fc2.weight13
1249
+ module.decoder.layers.1.mlp.experts.linear_fc2.weight15
1250
+ module.decoder.layers.2.mlp.experts.linear_fc1.weight0
1251
+ module.decoder.layers.1.mlp.experts.linear_fc2.weight11
1252
+ module.decoder.layers.1.mlp.experts.linear_fc2.weight9
1253
+ module.decoder.layers.2.mlp.experts.linear_fc1.weight1
1254
+ module.decoder.layers.1.mlp.experts.linear_fc2.weight14
1255
+ module.decoder.layers.1.mlp.experts.linear_fc2.weight12
1256
+ module.decoder.layers.1.mlp.experts.linear_fc2.weight10
1257
+ Params for bucket 60 (40370176 elements, 40370176 padded size):
1258
+ module.decoder.layers.1.mlp.experts.linear_fc2.weight7
1259
+ module.decoder.layers.1.mlp.experts.linear_fc2.weight5
1260
+ module.decoder.layers.1.mlp.experts.linear_fc2.weight3
1261
+ module.decoder.layers.1.mlp.experts.linear_fc2.weight1
1262
+ module.decoder.layers.1.mlp.experts.linear_fc2.weight8
1263
+ module.decoder.layers.1.mlp.experts.linear_fc2.weight6
1264
+ module.decoder.layers.1.mlp.experts.linear_fc2.weight4
1265
+ module.decoder.layers.1.mlp.experts.linear_fc2.weight2
1266
+ module.decoder.layers.1.mlp.experts.linear_fc2.weight0
1267
+ module.decoder.layers.1.mlp.experts.linear_fc1.weight15
1268
+ Params for bucket 61 (44040192 elements, 44040192 padded size):
1269
+ module.decoder.layers.1.mlp.experts.linear_fc1.weight14
1270
+ module.decoder.layers.1.mlp.experts.linear_fc1.weight12
1271
+ module.decoder.layers.1.mlp.experts.linear_fc1.weight10
1272
+ module.decoder.layers.1.mlp.experts.linear_fc1.weight13
1273
+ module.decoder.layers.1.mlp.experts.linear_fc1.weight11
1274
+ module.decoder.layers.1.mlp.experts.linear_fc1.weight9
1275
+ Params for bucket 62 (44040192 elements, 44040192 padded size):
1276
+ module.decoder.layers.1.mlp.experts.linear_fc1.weight8
1277
+ module.decoder.layers.1.mlp.experts.linear_fc1.weight6
1278
+ module.decoder.layers.1.mlp.experts.linear_fc1.weight4
1279
+ module.decoder.layers.1.mlp.experts.linear_fc1.weight7
1280
+ module.decoder.layers.1.mlp.experts.linear_fc1.weight5
1281
+ module.decoder.layers.1.mlp.experts.linear_fc1.weight3
1282
+ Params for bucket 63 (40370176 elements, 40370176 padded size):
1283
+ module.decoder.layers.1.mlp.experts.linear_fc1.weight2
1284
+ module.decoder.layers.1.mlp.experts.linear_fc1.weight0
1285
+ module.decoder.layers.0.mlp.experts.linear_fc2.weight14
1286
+ module.decoder.layers.0.mlp.experts.linear_fc2.weight12
1287
+ module.decoder.layers.0.mlp.experts.linear_fc2.weight15
1288
+ module.decoder.layers.1.mlp.experts.linear_fc1.weight1
1289
+ module.decoder.layers.0.mlp.experts.linear_fc2.weight13
1290
+ module.decoder.layers.0.mlp.experts.linear_fc2.weight11
1291
+ Params for bucket 64 (40370176 elements, 40370176 padded size):
1292
+ module.decoder.layers.0.mlp.experts.linear_fc2.weight10
1293
+ module.decoder.layers.0.mlp.experts.linear_fc2.weight8
1294
+ module.decoder.layers.0.mlp.experts.linear_fc2.weight6
1295
+ module.decoder.layers.0.mlp.experts.linear_fc2.weight4
1296
+ module.decoder.layers.0.mlp.experts.linear_fc2.weight2
1297
+ module.decoder.layers.0.mlp.experts.linear_fc2.weight0
1298
+ module.decoder.layers.0.mlp.experts.linear_fc2.weight9
1299
+ module.decoder.layers.0.mlp.experts.linear_fc2.weight7
1300
+ module.decoder.layers.0.mlp.experts.linear_fc2.weight5
1301
+ module.decoder.layers.0.mlp.experts.linear_fc2.weight3
1302
+ module.decoder.layers.0.mlp.experts.linear_fc2.weight1
1303
+ Params for bucket 65 (44040192 elements, 44040192 padded size):
1304
+ module.decoder.layers.0.mlp.experts.linear_fc1.weight15
1305
+ module.decoder.layers.0.mlp.experts.linear_fc1.weight11
1306
+ module.decoder.layers.0.mlp.experts.linear_fc1.weight13
1307
+ module.decoder.layers.0.mlp.experts.linear_fc1.weight10
1308
+ module.decoder.layers.0.mlp.experts.linear_fc1.weight14
1309
+ module.decoder.layers.0.mlp.experts.linear_fc1.weight12
1310
+ Params for bucket 66 (44040192 elements, 44040192 padded size):
1311
+ module.decoder.layers.0.mlp.experts.linear_fc1.weight7
1312
+ module.decoder.layers.0.mlp.experts.linear_fc1.weight8
1313
+ module.decoder.layers.0.mlp.experts.linear_fc1.weight4
1314
+ module.decoder.layers.0.mlp.experts.linear_fc1.weight9
1315
+ module.decoder.layers.0.mlp.experts.linear_fc1.weight5
1316
+ module.decoder.layers.0.mlp.experts.linear_fc1.weight6
1317
+ Params for bucket 67 (29360128 elements, 29360128 padded size):
1318
+ module.decoder.layers.0.mlp.experts.linear_fc1.weight3
1319
+ module.decoder.layers.0.mlp.experts.linear_fc1.weight0
1320
+ module.decoder.layers.0.mlp.experts.linear_fc1.weight2
1321
+ module.decoder.layers.0.mlp.experts.linear_fc1.weight1
1322
+ INFO:megatron.core.optimizer:Setting up optimizer with config OptimizerConfig(optimizer='adam', lr=0.0001, min_lr=1e-05, decoupled_lr=None, decoupled_min_lr=None, weight_decay=0.1, fp8_recipe='mxfp8', fp16=False, bf16=True, reuse_grad_buf_for_mxfp8_param_ag=False, params_dtype=torch.bfloat16, use_precision_aware_optimizer=False, store_param_remainders=True, main_grads_dtype=torch.float32, main_params_dtype=torch.float32, exp_avg_dtype=torch.float32, exp_avg_sq_dtype=torch.float32, loss_scale=None, initial_loss_scale=4294967296, min_loss_scale=1.0, loss_scale_window=1000, hysteresis=2, adam_beta1=0.9, adam_beta2=0.999, adam_eps=1e-08, sgd_momentum=0.9, use_distributed_optimizer=True, overlap_param_gather_with_optimizer_step=False, optimizer_cpu_offload=False, optimizer_offload_fraction=1.0, use_torch_optimizer_for_cpu_offload=False, overlap_cpu_optimizer_d2h_h2d=False, pin_cpu_grads=True, pin_cpu_params=True, clip_grad=1.0, log_num_zeros_in_grad=False, barrier_with_L1_time=True, timers=<megatron.core.timers.Timers object at 0x78d63de35b50>, config_logger_dir='')
1323
+ INFO:megatron.core.optimizer_param_scheduler:> learning rate decay style: cosine
1324
+ [after model, optimizer, and learning rate scheduler are built] datetime: 2026-09-17 02:55:04
1325
+ > building train, validation, and test datasets ...
1326
+ > datasets target sizes (minimum size):
1327
+ train: 160
1328
+ validation: 0
1329
+ test: 0
1330
+ INFO:megatron.core.datasets.blended_megatron_dataset_config:Let split_matrix = [(0, 1.0), None, None]
1331
+ > building train, validation, and test datasets for GPT ...
1332
+ INFO:megatron.core.datasets.blended_megatron_dataset_builder:Building GPTDataset splits with sizes=(160, 0, 0) and config=GPTDatasetConfig(random_seed=1234, sequence_length=4096, blend=(['/workspace/data/wikitext/wikitext103_gpt2_text_document'], None), blend_per_split=None, split='100,0,0', split_matrix=[(0, 1.0), None, None], num_dataset_builder_threads=1, path_to_cache=None, mmap_bin_files=True, mock=False, tokenizer=<megatron.training.tokenizer.tokenizer._GPT2BPETokenizer object at 0x78d63eb9afc0>, mid_level_dataset_surplus=0.005, reset_position_ids=False, reset_attention_mask=False, eod_mask_loss=False, create_attention_mask=True, drop_last_partial_validation_sequence=True, add_extra_token_to_sequence=True, object_storage_cache_path=None)
1333
+ INFO:megatron.core.datasets.indexed_dataset:Load the _IndexReader from /workspace/data/wikitext/wikitext103_gpt2_text_document.idx
1334
+ INFO:megatron.core.datasets.indexed_dataset: Extract the sequence lengths
1335
+ INFO:megatron.core.datasets.indexed_dataset: Extract the sequence pointers
1336
+ INFO:megatron.core.datasets.indexed_dataset: Extract the document indices
1337
+ INFO:megatron.core.datasets.indexed_dataset:> total number of sequences: 1165029
1338
+ INFO:megatron.core.datasets.indexed_dataset:> total number of documents: 1801350
1339
+ INFO:megatron.core.datasets.gpt_dataset:Load the GPTDataset train indices
1340
+ INFO:megatron.core.datasets.gpt_dataset: Load the document index from 68684a515e118253a8572c4517cf2ffe-GPTDataset-train-document_index.npy
1341
+ INFO:megatron.core.datasets.gpt_dataset: Load the sample index from 68684a515e118253a8572c4517cf2ffe-GPTDataset-train-sample_index.npy
1342
+ INFO:megatron.core.datasets.gpt_dataset: Load the shuffle index from 68684a515e118253a8572c4517cf2ffe-GPTDataset-train-shuffle_index.npy
1343
+ INFO:megatron.core.datasets.gpt_dataset:> total number of samples: 29073
1344
+ > finished creating GPT datasets ...
1345
+ [after dataloaders are built] datetime: 2026-09-17 02:55:04
1346
+ done with setup ...
1347
+ (min, max) time across ranks (ms):
1348
+ model-and-optimizer-setup ......................: (828.03, 861.50)
1349
+ train/valid/test-data-iterators-setup ..........: (96.75, 107.45)
1350
+ training ...
1351
+ Overwriting rerun_state_machine.current_iteration from -1 to 0...
1352
+ [before the start of training step] datetime: 2026-09-17 02:55:04
1353
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:334: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
1354
+ warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
1355
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:336: UserWarning: The use_host_meta is deprecated, it will be removed in the future.
1356
+ warnings.warn("The use_host_meta is deprecated, it will be removed in the future.")
1357
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:334: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
1358
+ warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
1359
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:336: UserWarning: The use_host_meta is deprecated, it will be removed in the future.
1360
+ warnings.warn("The use_host_meta is deprecated, it will be removed in the future.")
1361
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:334: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
1362
+ warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
1363
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:336: UserWarning: The use_host_meta is deprecated, it will be removed in the future.
1364
+ warnings.warn("The use_host_meta is deprecated, it will be removed in the future.")
1365
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:334: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
1366
+ warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
1367
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:336: UserWarning: The use_host_meta is deprecated, it will be removed in the future.
1368
+ warnings.warn("The use_host_meta is deprecated, it will be removed in the future.")
1369
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:334: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
1370
+ warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
1371
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:334: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
1372
+ warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
1373
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:336: UserWarning: The use_host_meta is deprecated, it will be removed in the future.
1374
+ warnings.warn("The use_host_meta is deprecated, it will be removed in the future.")
1375
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:336: UserWarning: The use_host_meta is deprecated, it will be removed in the future.
1376
+ warnings.warn("The use_host_meta is deprecated, it will be removed in the future.")
1377
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:334: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
1378
+ warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
1379
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:336: UserWarning: The use_host_meta is deprecated, it will be removed in the future.
1380
+ warnings.warn("The use_host_meta is deprecated, it will be removed in the future.")
1381
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:334: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
1382
+ warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
1383
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:336: UserWarning: The use_host_meta is deprecated, it will be removed in the future.
1384
+ warnings.warn("The use_host_meta is deprecated, it will be removed in the future.")
1385
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:482: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
1386
+ warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
1387
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:482: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
1388
+ warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
1389
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:482: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
1390
+ warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
1391
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:482: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
1392
+ warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
1393
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:482: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
1394
+ warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
1395
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:482: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
1396
+ warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
1397
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:482: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
1398
+ warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
1399
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:482: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
1400
+ warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
1401
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:334: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
1402
+ warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
1403
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:336: UserWarning: The use_host_meta is deprecated, it will be removed in the future.
1404
+ warnings.warn("The use_host_meta is deprecated, it will be removed in the future.")
1405
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:334: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
1406
+ warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
1407
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:336: UserWarning: The use_host_meta is deprecated, it will be removed in the future.
1408
+ warnings.warn("The use_host_meta is deprecated, it will be removed in the future.")
1409
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:334: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
1410
+ warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
1411
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:336: UserWarning: The use_host_meta is deprecated, it will be removed in the future.
1412
+ warnings.warn("The use_host_meta is deprecated, it will be removed in the future.")
1413
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:334: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
1414
+ warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
1415
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:336: UserWarning: The use_host_meta is deprecated, it will be removed in the future.
1416
+ warnings.warn("The use_host_meta is deprecated, it will be removed in the future.")
1417
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:334: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
1418
+ warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
1419
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:336: UserWarning: The use_host_meta is deprecated, it will be removed in the future.
1420
+ warnings.warn("The use_host_meta is deprecated, it will be removed in the future.")
1421
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:334: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
1422
+ warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
1423
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:336: UserWarning: The use_host_meta is deprecated, it will be removed in the future.
1424
+ warnings.warn("The use_host_meta is deprecated, it will be removed in the future.")
1425
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:334: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
1426
+ warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
1427
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:336: UserWarning: The use_host_meta is deprecated, it will be removed in the future.
1428
+ warnings.warn("The use_host_meta is deprecated, it will be removed in the future.")
1429
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:334: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
1430
+ warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
1431
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:336: UserWarning: The use_host_meta is deprecated, it will be removed in the future.
1432
+ warnings.warn("The use_host_meta is deprecated, it will be removed in the future.")
1433
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:482: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
1434
+ warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
1435
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:482: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
1436
+ warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
1437
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:482: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
1438
+ warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
1439
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:482: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
1440
+ warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
1441
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:482: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
1442
+ warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
1443
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:482: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
1444
+ warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
1445
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:482: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
1446
+ warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
1447
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:482: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
1448
+ warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
1449
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:334: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
1450
+ warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
1451
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:334: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
1452
+ warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
1453
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:334: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
1454
+ warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
1455
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:334: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
1456
+ warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
1457
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:334: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
1458
+ warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
1459
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:334: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
1460
+ warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
1461
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:334: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
1462
+ warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
1463
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:334: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
1464
+ warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
1465
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:336: UserWarning: The use_host_meta is deprecated, it will be removed in the future.
1466
+ warnings.warn("The use_host_meta is deprecated, it will be removed in the future.")
1467
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:336: UserWarning: The use_host_meta is deprecated, it will be removed in the future.
1468
+ warnings.warn("The use_host_meta is deprecated, it will be removed in the future.")
1469
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:336: UserWarning: The use_host_meta is deprecated, it will be removed in the future.
1470
+ warnings.warn("The use_host_meta is deprecated, it will be removed in the future.")
1471
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:336: UserWarning: The use_host_meta is deprecated, it will be removed in the future.
1472
+ warnings.warn("The use_host_meta is deprecated, it will be removed in the future.")
1473
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:336: UserWarning: The use_host_meta is deprecated, it will be removed in the future.
1474
+ warnings.warn("The use_host_meta is deprecated, it will be removed in the future.")
1475
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:336: UserWarning: The use_host_meta is deprecated, it will be removed in the future.
1476
+ warnings.warn("The use_host_meta is deprecated, it will be removed in the future.")
1477
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:336: UserWarning: The use_host_meta is deprecated, it will be removed in the future.
1478
+ warnings.warn("The use_host_meta is deprecated, it will be removed in the future.")
1479
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:336: UserWarning: The use_host_meta is deprecated, it will be removed in the future.
1480
+ warnings.warn("The use_host_meta is deprecated, it will be removed in the future.")
1481
+ [2026-09-17 02:55:48] iteration 1/ 10 | consumed samples: 16 | elapsed time per iteration (ms): 43511.8 | throughput per GPU (TFLOP/s/GPU): 2.9 | learning rate: 2.000000E-05 | global batch size: 16 | lm loss: 1.102484E+01 | loss scale: 1.0 | grad norm: 44.692 | number of skipped iterations: 0 | number of nan iterations: 0 |
1482
+ Number of parameters in transformer block in billions: 23.22
1483
+ Number of parameters in embedding layers in billions: 0.41
1484
+ Total number of parameters in billions: 23.63
1485
+ Number of parameters in most loaded shard in billions: 23.6320
1486
+ Theoretical memory footprints: weight and optimizer=169029.43 MB
1487
+ [Rank 0] (after 1 iterations) memory (MB) | allocated: 64034.8828125 | max allocated: 79445.076171875 | reserved: 82888.0 | max reserved: 82888.0
1488
+ [2026-09-17 02:55:49] iteration 2/ 10 | consumed samples: 32 | elapsed time per iteration (ms): 946.1 | throughput per GPU (TFLOP/s/GPU): 132.7 | learning rate: 4.000000E-05 | global batch size: 16 | lm loss: 1.101710E+01 | loss scale: 1.0 | grad norm: 46.419 | number of skipped iterations: 0 | number of nan iterations: 0 |
1489
+ [2026-09-17 02:55:49] iteration 3/ 10 | consumed samples: 48 | elapsed time per iteration (ms): 592.0 | throughput per GPU (TFLOP/s/GPU): 212.1 | learning rate: 6.000000E-05 | global batch size: 16 | lm loss: 1.055648E+01 | loss scale: 1.0 | grad norm: 213.531 | number of skipped iterations: 0 | number of nan iterations: 0 |
1490
+ Capture CUDA graph for training!!!
1491
+ CUDA graph capture done!!!
1492
+ [2026-09-17 02:55:53] iteration 4/ 10 | consumed samples: 64 | elapsed time per iteration (ms): 3377.2 | throughput per GPU (TFLOP/s/GPU): 37.2 | learning rate: 8.000000E-05 | global batch size: 16 | lm loss: 1.297811E+01 | loss scale: 1.0 | grad norm: 49.016 | number of skipped iterations: 0 | number of nan iterations: 0 |
1493
+ [2026-09-17 02:55:53] iteration 5/ 10 | consumed samples: 80 | elapsed time per iteration (ms): 341.6 | throughput per GPU (TFLOP/s/GPU): 367.7 | learning rate: 1.000000E-04 | global batch size: 16 | lm loss: 1.032672E+01 | loss scale: 1.0 | grad norm: 17.807 | number of skipped iterations: 0 | number of nan iterations: 0 |
1494
+ [2026-09-17 02:55:54] iteration 6/ 10 | consumed samples: 96 | elapsed time per iteration (ms): 341.5 | throughput per GPU (TFLOP/s/GPU): 367.7 | learning rate: 7.750000E-05 | global batch size: 16 | lm loss: 1.009215E+01 | loss scale: 1.0 | grad norm: 13.671 | number of skipped iterations: 0 | number of nan iterations: 0 |
1495
+ [2026-09-17 02:55:57] iteration 7/ 10 | consumed samples: 112 | elapsed time per iteration (ms): 3269.7 | throughput per GPU (TFLOP/s/GPU): 38.4 | learning rate: 3.250000E-05 | global batch size: 16 | lm loss: 9.764772E+00 | loss scale: 1.0 | grad norm: 6.485 | number of skipped iterations: 0 | number of nan iterations: 0 |
1496
+ [2026-09-17 02:55:57] iteration 8/ 10 | consumed samples: 128 | elapsed time per iteration (ms): 355.8 | throughput per GPU (TFLOP/s/GPU): 352.9 | learning rate: 1.000000E-05 | global batch size: 16 | lm loss: 9.185248E+00 | loss scale: 1.0 | grad norm: 5.739 | number of skipped iterations: 0 | number of nan iterations: 0 |
1497
+ Capture range started in the application.
1498
+ Capture range ended in the application.
1499
+ Generating '/tmp/nsys-report-418c.qdstrm'
1500
+
1501
+
1502
+ Collection stop is not allowed in this state.
1503
+ W0917 03:03:37.518000 880794 torch/distributed/elastic/agent/server/api.py:719] Received 15 death signal, shutting down workers
1504
+ W0917 03:03:37.519000 880794 torch/distributed/elastic/multiprocessing/api.py:900] Sending process 880908 closing signal SIGTERM
1505
+ Generated:
1506
+ /workspace/granularity_next_b200/tiera_topk_trace/echo/tierbtrace/tierbtrace_gbs16/G4_K8__echo__matched__seq4096__gbs16__cap1p109375/nsys/G4_K8__echo__matched__seq4096__gbs16__cap1p109375.nsys-rep
1507
+ W0917 03:03:37.524000 880794 torch/distributed/elastic/multiprocessing/api.py:900] Sending process 880909 closing signal SIGTERM
1508
+ W0917 03:03:37.527000 880794 torch/distributed/elastic/multiprocessing/api.py:900] Sending process 880910 closing signal SIGTERM
1509
+ W0917 03:03:37.532000 880794 torch/distributed/elastic/multiprocessing/api.py:900] Sending process 880911 closing signal SIGTERM
1510
+ W0917 03:03:37.536000 880794 torch/distributed/elastic/multiprocessing/api.py:900] Sending process 880912 closing signal SIGTERM
1511
+ W0917 03:03:37.538000 880794 torch/distributed/elastic/multiprocessing/api.py:900] Sending process 880913 closing signal SIGTERM
1512
+ W0917 03:03:37.551000 880794 torch/distributed/elastic/multiprocessing/api.py:900] Sending process 880914 closing signal SIGTERM
1513
+ W0917 03:03:37.558000 880794 torch/distributed/elastic/multiprocessing/api.py:900] Sending process 880915 closing signal SIGTERM
1514
+ Traceback (most recent call last):
1515
+ File "/usr/local/bin/torchrun", line 8, in <module>
1516
+ sys.exit(main())
1517
+ ^^^^^^
1518
+ File "/usr/local/lib/python3.12/dist-packages/torch/distributed/elastic/multiprocessing/errors/__init__.py", line 355, in wrapper
1519
+ return f(*args, **kwargs)
1520
+ ^^^^^^^^^^^^^^^^^^
1521
+ File "/usr/local/lib/python3.12/dist-packages/torch/distributed/run.py", line 892, in main
1522
+ run(args)
1523
+ File "/usr/local/lib/python3.12/dist-packages/torch/distributed/run.py", line 883, in run
1524
+ elastic_launch(
1525
+ File "/usr/local/lib/python3.12/dist-packages/torch/distributed/launcher/api.py", line 139, in __call__
1526
+ return launch_agent(self._config, self._entrypoint, list(args))
1527
+ ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
1528
+ File "/usr/local/lib/python3.12/dist-packages/torch/distributed/launcher/api.py", line 261, in launch_agent
1529
+ result = agent.run()
1530
+ ^^^^^^^^^^^
1531
+ File "/usr/local/lib/python3.12/dist-packages/torch/distributed/elastic/metrics/api.py", line 138, in wrapper
1532
+ result = f(*args, **kwargs)
1533
+ ^^^^^^^^^^^^^^^^^^
1534
+ File "/usr/local/lib/python3.12/dist-packages/torch/distributed/elastic/agent/server/api.py", line 711, in run
1535
+ result = self._invoke_run(role)
1536
+ ^^^^^^^^^^^^^^^^^^^^^^
1537
+ File "/usr/local/lib/python3.12/dist-packages/torch/distributed/elastic/agent/server/api.py", line 870, in _invoke_run
1538
+ time.sleep(monitor_interval)
1539
+ File "/usr/local/lib/python3.12/dist-packages/torch/distributed/elastic/multiprocessing/api.py", line 84, in _terminate_process_handler
1540
+ raise SignalException(f"Process {os.getpid()} got signal: {sigval}", sigval=sigval)
1541
+ torch.distributed.elastic.multiprocessing.api.SignalException: Process 880794 got signal: 15
1542
+ The target application terminated. One or more process it created re-parented.
1543
+ Waiting for termination of re-parented processes.
1544
+ Use the `--wait` option to modify this behavior.
1545
+ There are no active sessions.
granularity_next_b200/tiera_topk_trace/echo/tierbtrace/tierbtrace_gbs16/G4_K8__echo__matched__seq4096__gbs16__cap1p109375/nsys/verify_trace_ranks.log ADDED
@@ -0,0 +1,6 @@
 
 
 
 
 
 
 
1
+ report : /workspace/granularity_next_b200/tiera_topk_trace/echo/tierbtrace/tierbtrace_gbs16/G4_K8__echo__matched__seq4096__gbs16__cap1p109375/nsys/G4_K8__echo__matched__seq4096__gbs16__cap1p109375.nsys-rep
2
+ kernels : 281286
3
+ devices : 8 of 8 -> [0, 1, 2, 3, 4, 5, 6, 7]
4
+ processes: 1
5
+ markers : 23552 echo_trace__ kernels on devices [0, 1, 2, 3, 4, 5, 6, 7]
6
+ PASS: all ranks present and backend markers emitted
granularity_next_b200/tiera_topk_trace/echo/tierbtrace/tierbtrace_gbs16/trace_results.tsv CHANGED
@@ -5,3 +5,4 @@ G3_K8 echo 1.09375 OK /workspace/granularity_next_b200/tiera_topk_trace/echo/tie
5
  G4_K2 echo 1.265625 OK /workspace/granularity_next_b200/tiera_topk_trace/echo/tierbtrace/tierbtrace_gbs16/G4_K2__echo__matched__seq4096__gbs16__cap1p265625/nsys/G4_K2__echo__matched__seq4096__gbs16__cap1p265625.nsys-rep
6
  G4_K4 echo 1.171875 OK /workspace/granularity_next_b200/tiera_topk_trace/echo/tierbtrace/tierbtrace_gbs16/G4_K4__echo__matched__seq4096__gbs16__cap1p171875/nsys/G4_K4__echo__matched__seq4096__gbs16__cap1p171875.nsys-rep
7
  G4_K6 echo 1.1354166666666667 OK /workspace/granularity_next_b200/tiera_topk_trace/echo/tierbtrace/tierbtrace_gbs16/G4_K6__echo__matched__seq4096__gbs16__cap1p1354166666666667/nsys/G4_K6__echo__matched__seq4096__gbs16__cap1p1354166666666667.nsys-rep
 
 
5
  G4_K2 echo 1.265625 OK /workspace/granularity_next_b200/tiera_topk_trace/echo/tierbtrace/tierbtrace_gbs16/G4_K2__echo__matched__seq4096__gbs16__cap1p265625/nsys/G4_K2__echo__matched__seq4096__gbs16__cap1p265625.nsys-rep
6
  G4_K4 echo 1.171875 OK /workspace/granularity_next_b200/tiera_topk_trace/echo/tierbtrace/tierbtrace_gbs16/G4_K4__echo__matched__seq4096__gbs16__cap1p171875/nsys/G4_K4__echo__matched__seq4096__gbs16__cap1p171875.nsys-rep
7
  G4_K6 echo 1.1354166666666667 OK /workspace/granularity_next_b200/tiera_topk_trace/echo/tierbtrace/tierbtrace_gbs16/G4_K6__echo__matched__seq4096__gbs16__cap1p1354166666666667/nsys/G4_K6__echo__matched__seq4096__gbs16__cap1p1354166666666667.nsys-rep
8
+ G4_K8 echo 1.109375 OK /workspace/granularity_next_b200/tiera_topk_trace/echo/tierbtrace/tierbtrace_gbs16/G4_K8__echo__matched__seq4096__gbs16__cap1p109375/nsys/G4_K8__echo__matched__seq4096__gbs16__cap1p109375.nsys-rep
granularity_next_b200/tiera_topk_trace/hf_watcher.log CHANGED
@@ -19,3 +19,6 @@ https://huggingface.co/MoE-CUDA-Graph-Analysis/tierb-b200-gbs8/tree/main
19
  uploaded: /workspace/granularity_next_b200/tiera_topk_trace -> MoE-CUDA-Graph-Analysis/tierb-b200-gbs8/granularity_next_b200/tiera_topk_trace
20
  https://huggingface.co/MoE-CUDA-Graph-Analysis/tierb-b200-gbs8/tree/main
21
  [2026-09-17T02:54:52+00:00] backup pass
 
 
 
 
19
  uploaded: /workspace/granularity_next_b200/tiera_topk_trace -> MoE-CUDA-Graph-Analysis/tierb-b200-gbs8/granularity_next_b200/tiera_topk_trace
20
  https://huggingface.co/MoE-CUDA-Graph-Analysis/tierb-b200-gbs8/tree/main
21
  [2026-09-17T02:54:52+00:00] backup pass
22
+ uploaded: /workspace/granularity_next_b200/tiera_topk_trace -> MoE-CUDA-Graph-Analysis/tierb-b200-gbs8/granularity_next_b200/tiera_topk_trace
23
+ https://huggingface.co/MoE-CUDA-Graph-Analysis/tierb-b200-gbs8/tree/main
24
+ [2026-09-17T03:04:55+00:00] backup pass
granularity_next_b200/tiera_topk_trace/study.log CHANGED
@@ -60,3 +60,10 @@ driver_repo: /workspace/blackwell_run_scripts @ 10756d4
60
  [guard] tearing down process group (outcome=STALLED)
61
  [02:54:36] trace verified -> G4_K6__echo__matched__seq4096__gbs16__cap1p1354166666666667.nsys-rep (19M, all 8 GPUs + echo markers)
62
  [02:54:36] TRACE-ONLY G4_K8__echo__matched__seq4096__gbs16__cap1p109375 (ranks: 0 1 2 3 4 5 6 7)
 
 
 
 
 
 
 
 
60
  [guard] tearing down process group (outcome=STALLED)
61
  [02:54:36] trace verified -> G4_K6__echo__matched__seq4096__gbs16__cap1p1354166666666667.nsys-rep (19M, all 8 GPUs + echo markers)
62
  [02:54:36] TRACE-ONLY G4_K8__echo__matched__seq4096__gbs16__cap1p109375 (ranks: 0 1 2 3 4 5 6 7)
63
+ [guard] iter 6/10 (80s)
64
+ [guard] iter 8/10 (90s)
65
+ [guard] no progress for 420s at iter 8/10; treating as done
66
+ [guard] nsys stop (flushing report)
67
+ [guard] tearing down process group (outcome=STALLED)
68
+ [03:03:55] trace verified -> G4_K8__echo__matched__seq4096__gbs16__cap1p109375.nsys-rep (20M, all 8 GPUs + echo markers)
69
+ [03:03:55] TRACE-ONLY G4_K10__echo__matched__seq4096__gbs16__cap1p1 (ranks: 0 1 2 3 4 5 6 7)