AbhishekGhosh1998 commited on
Commit
c085e2c
·
verified ·
1 Parent(s): 45cc5cc

granularity results granularity_next_b200/tiera_moonep_probe_refresh @ 2026-09-21T05:22:01Z from sp-aiinfra-interactive-aghosh-8-mb99k-8xb200-moe-cg-0

Browse files
granularity_next_b200/tiera_moonep_probe_refresh/STATUS.txt CHANGED
@@ -1,7 +1,7 @@
1
  study_root: /workspace/granularity_next_b200/tiera_moonep_probe_refresh
2
  gbs: 16
3
- updated: 2026-09-16T21:43:32+00:00
4
- current: idle
5
 
6
  completed steps:
7
  moonep__tierbprobe__gbs16
 
1
  study_root: /workspace/granularity_next_b200/tiera_moonep_probe_refresh
2
  gbs: 16
3
+ updated: 2026-09-21T05:22:00+00:00
4
+ current: starting
5
 
6
  completed steps:
7
  moonep__tierbprobe__gbs16
granularity_next_b200/tiera_moonep_probe_refresh/collected_all.tsv CHANGED
@@ -1,9 +1,9 @@
1
  run_dir rep phase is_trace async config backend slot_policy slots_per_rank num_experts topk moe_ffn_hidden experts_per_rank expert_params num_layers seq_length global_batch_size num_microbatches assignments_per_rank mean_tokens_per_expert capacity capacity_mode recompute force_balance cuda_graph memory_dump_mode memory_snapshot_path ultraep_quota_min_tokens shared_expert_overlap branch commit dense_first_layers moe_layers router_balancing aux_loss_coeff router_pre_softmax router_fusion rotary_base qkv_bias source_model standard_num_layers base_rung q_lora_rank kv_lora_rank qk_head_dim qk_pos_emb_head_dim v_head_dim rotary_percent rotary_scaling mscale mscale_all_dim router_groups router_group_topk router_score_function router_expert_bias router_bias_update_rate router_topk_scaling_factor shared_expert_gate apply_layernorm_1p attention_output_gate linear_attention_type linear_attention_freq linear_conv_kernel_dim linear_key_head_dim linear_value_head_dim linear_num_key_heads linear_num_value_heads mtp_num_layers mtp_loss_scaling_factor no_weight_decay_cond_type max_position_embeddings init_method_std status iters_seen iter_ms_mean iter_ms_min iter_ms_stdev tflops peak_mib_sampled megatron_max_reserved_mib drop_warning
2
- moonep/tierbprobe/tierbprobe_gbs16/G1__moonep__matched__seq4096__gbs16__capnone__async/rep1 rep1 tierbprobe 0 1 G1 moonep matched 1 8 2 14336 1 176160768 16 4096 16 2 8192 8192 none none none 0 0 none 0 moonep_backend_async_grad_reduce_token_routing_analysis a6a1a221 0 16 aux_loss 1e-2 0 0 1000000 0 512 128 64 128 1.0 softmax 0 0 0 0 0 0.1 32768 0.01 OK 12 597.53 584.50 18.18 558.4 92581 59020
3
- moonep/tierbprobe/tierbprobe_gbs16/G1__moonep__matched__seq4096__gbs16__capnone__forcebal__async/rep1 rep1 tierbprobe 0 1 G1 moonep matched 1 8 2 14336 1 176160768 16 4096 16 2 8192 8192 none none none 1 0 none 0 moonep_backend_async_grad_reduce_token_routing_analysis a6a1a221 0 16 aux_loss 1e-2 0 0 1000000 0 512 128 64 128 1.0 softmax 0 0 0 0 0 0.1 32768 0.01 OK 12 573.83 569.10 4.07 581.0 92581 59020
4
- moonep/tierbprobe/tierbprobe_gbs16/G2__moonep__matched__seq4096__gbs16__capnone__async/rep1 rep1 tierbprobe 0 1 G2 moonep matched 4 32 8 3584 4 44040192 16 4096 16 2 32768 8192 none none none 0 0 none 0 moonep_backend_async_grad_reduce_token_routing_analysis a6a1a221 0 16 aux_loss 1e-2 0 0 1000000 0 512 128 64 128 1.0 softmax 0 0 0 0 0 0.1 32768 0.01 OK 12 650.03 566.50 137.54 529.7 95485 56290
5
- moonep/tierbprobe/tierbprobe_gbs16/G2__moonep__matched__seq4096__gbs16__capnone__forcebal__async/rep1 rep1 tierbprobe 0 1 G2 moonep matched 4 32 8 3584 4 44040192 16 4096 16 2 32768 8192 none none none 1 0 none 0 moonep_backend_async_grad_reduce_token_routing_analysis a6a1a221 0 16 aux_loss 1e-2 0 0 1000000 0 512 128 64 128 1.0 softmax 0 0 0 0 0 0.1 32768 0.01 OK 12 551.20 545.40 4.35 604.8 95485 56290
6
- moonep/tierbprobe/tierbprobe_gbs16/G3__moonep__matched__seq4096__gbs16__capnone__async/rep1 rep1 tierbprobe 0 1 G3 moonep matched 8 64 16 1792 8 22020096 16 4096 16 2 65536 8192 none none none 0 0 none 0 moonep_backend_async_grad_reduce_token_routing_analysis a6a1a221 0 16 aux_loss 1e-2 0 0 1000000 0 512 128 64 128 1.0 softmax 0 0 0 0 0 0.1 32768 0.01 OK 12 611.76 607.20 5.12 545.0 98677 55690
7
- moonep/tierbprobe/tierbprobe_gbs16/G3__moonep__matched__seq4096__gbs16__capnone__forcebal__async/rep1 rep1 tierbprobe 0 1 G3 moonep matched 8 64 16 1792 8 22020096 16 4096 16 2 65536 8192 none none none 1 0 none 0 moonep_backend_async_grad_reduce_token_routing_analysis a6a1a221 0 16 aux_loss 1e-2 0 0 1000000 0 512 128 64 128 1.0 softmax 0 0 0 0 0 0.1 32768 0.01 OK 12 708.73 601.50 184.17 492.1 98613 55624
8
- moonep/tierbprobe/tierbprobe_gbs16/G4__moonep__matched__seq4096__gbs16__capnone__async/rep1 rep1 tierbprobe 0 1 G4 moonep matched 16 128 32 896 16 11010048 16 4096 16 2 131072 8192 none none none 0 0 none 0 moonep_backend_async_grad_reduce_token_routing_analysis a6a1a221 0 16 aux_loss 1e-2 0 0 1000000 0 512 128 64 128 1.0 softmax 0 0 0 0 0 0.1 32768 0.01 OK 12 791.24 788.30 3.38 421.3 105107 57126
9
- moonep/tierbprobe/tierbprobe_gbs16/G4__moonep__matched__seq4096__gbs16__capnone__forcebal__async/rep1 rep1 tierbprobe 0 1 G4 moonep matched 16 128 32 896 16 11010048 16 4096 16 2 131072 8192 none none none 1 0 none 0 moonep_backend_async_grad_reduce_token_routing_analysis a6a1a221 0 16 aux_loss 1e-2 0 0 1000000 0 512 128 64 128 1.0 softmax 0 0 0 0 0 0.1 32768 0.01 OK 12 787.61 783.30 3.15 423.3 105107 57126
 
1
  run_dir rep phase is_trace async config backend slot_policy slots_per_rank num_experts topk moe_ffn_hidden experts_per_rank expert_params num_layers seq_length global_batch_size num_microbatches assignments_per_rank mean_tokens_per_expert capacity capacity_mode recompute force_balance cuda_graph memory_dump_mode memory_snapshot_path ultraep_quota_min_tokens shared_expert_overlap branch commit dense_first_layers moe_layers router_balancing aux_loss_coeff router_pre_softmax router_fusion rotary_base qkv_bias source_model standard_num_layers base_rung q_lora_rank kv_lora_rank qk_head_dim qk_pos_emb_head_dim v_head_dim rotary_percent rotary_scaling mscale mscale_all_dim router_groups router_group_topk router_score_function router_expert_bias router_bias_update_rate router_topk_scaling_factor shared_expert_gate apply_layernorm_1p attention_output_gate linear_attention_type linear_attention_freq linear_conv_kernel_dim linear_key_head_dim linear_value_head_dim linear_num_key_heads linear_num_value_heads mtp_num_layers mtp_loss_scaling_factor no_weight_decay_cond_type max_position_embeddings init_method_std status iters_seen iter_ms_mean iter_ms_min iter_ms_stdev tflops peak_mib_sampled megatron_max_reserved_mib drop_warning
2
+ moonep/tierbprobe/tierbprobe_gbs16/G1__moonep__matched__seq4096__gbs16__capnone__async/rep1 rep1 tierbprobe 0 1 G1 moonep matched 1 8 2 14336 1 176160768 16 4096 16 2 8192 8192 none none none 0 0 none 0 moonep_backend_async_grad_reduce_token_routing_analysis a6a1a221 0 16 aux_loss 1e-2 0 0 1000000 0 512 128 64 128 1.0 softmax 0 0 0 0 0 0.1 32768 0.01 INCOMPLETE 0
3
+ moonep/tierbprobe/tierbprobe_gbs16/G1__moonep__matched__seq4096__gbs16__capnone__forcebal__async/rep1 rep1 tierbprobe 0 1 G1 moonep matched 1 8 2 14336 1 176160768 16 4096 16 2 8192 8192 none none none 1 0 none 0 moonep_backend_async_grad_reduce_token_routing_analysis a6a1a221 0 16 aux_loss 1e-2 0 0 1000000 0 512 128 64 128 1.0 softmax 0 0 0 0 0 0.1 32768 0.01 INCOMPLETE 0
4
+ moonep/tierbprobe/tierbprobe_gbs16/G2__moonep__matched__seq4096__gbs16__capnone__async/rep1 rep1 tierbprobe 0 1 G2 moonep matched 4 32 8 3584 4 44040192 16 4096 16 2 32768 8192 none none none 0 0 none 0 moonep_backend_async_grad_reduce_token_routing_analysis a6a1a221 0 16 aux_loss 1e-2 0 0 1000000 0 512 128 64 128 1.0 softmax 0 0 0 0 0 0.1 32768 0.01 INCOMPLETE 0
5
+ moonep/tierbprobe/tierbprobe_gbs16/G2__moonep__matched__seq4096__gbs16__capnone__forcebal__async/rep1 rep1 tierbprobe 0 1 G2 moonep matched 4 32 8 3584 4 44040192 16 4096 16 2 32768 8192 none none none 1 0 none 0 moonep_backend_async_grad_reduce_token_routing_analysis a6a1a221 0 16 aux_loss 1e-2 0 0 1000000 0 512 128 64 128 1.0 softmax 0 0 0 0 0 0.1 32768 0.01 NOLOG 0
6
+ moonep/tierbprobe/tierbprobe_gbs16/G3__moonep__matched__seq4096__gbs16__capnone__async/rep1 rep1 tierbprobe 0 1 G3 moonep matched 8 64 16 1792 8 22020096 16 4096 16 2 65536 8192 none none none 0 0 none 0 moonep_backend_async_grad_reduce_token_routing_analysis a6a1a221 0 16 aux_loss 1e-2 0 0 1000000 0 512 128 64 128 1.0 softmax 0 0 0 0 0 0.1 32768 0.01 INCOMPLETE 0
7
+ moonep/tierbprobe/tierbprobe_gbs16/G3__moonep__matched__seq4096__gbs16__capnone__forcebal__async/rep1 rep1 tierbprobe 0 1 G3 moonep matched 8 64 16 1792 8 22020096 16 4096 16 2 65536 8192 none none none 1 0 none 0 moonep_backend_async_grad_reduce_token_routing_analysis a6a1a221 0 16 aux_loss 1e-2 0 0 1000000 0 512 128 64 128 1.0 softmax 0 0 0 0 0 0.1 32768 0.01 NOLOG 0
8
+ moonep/tierbprobe/tierbprobe_gbs16/G4__moonep__matched__seq4096__gbs16__capnone__async/rep1 rep1 tierbprobe 0 1 G4 moonep matched 16 128 32 896 16 11010048 16 4096 16 2 131072 8192 none none none 0 0 none 0 moonep_backend_async_grad_reduce_token_routing_analysis a6a1a221 0 16 aux_loss 1e-2 0 0 1000000 0 512 128 64 128 1.0 softmax 0 0 0 0 0 0.1 32768 0.01 NOLOG 0
9
+ moonep/tierbprobe/tierbprobe_gbs16/G4__moonep__matched__seq4096__gbs16__capnone__forcebal__async/rep1 rep1 tierbprobe 0 1 G4 moonep matched 16 128 32 896 16 11010048 16 4096 16 2 131072 8192 none none none 1 0 none 0 moonep_backend_async_grad_reduce_token_routing_analysis a6a1a221 0 16 aux_loss 1e-2 0 0 1000000 0 512 128 64 128 1.0 softmax 0 0 0 0 0 0.1 32768 0.01 NOLOG 0
granularity_next_b200/tiera_moonep_probe_refresh/hf_watcher.log CHANGED
@@ -6,3 +6,4 @@ https://huggingface.co/MoE-CUDA-Graph-Analysis/tierb-b200-gbs8/tree/main
6
  [2026-09-16T21:43:29+00:00] backup pass
7
  uploaded: /workspace/granularity_next_b200/tiera_moonep_probe_refresh -> MoE-CUDA-Graph-Analysis/tierb-b200-gbs8/granularity_next_b200/tiera_moonep_probe_refresh
8
  https://huggingface.co/MoE-CUDA-Graph-Analysis/tierb-b200-gbs8/tree/main
 
 
6
  [2026-09-16T21:43:29+00:00] backup pass
7
  uploaded: /workspace/granularity_next_b200/tiera_moonep_probe_refresh -> MoE-CUDA-Graph-Analysis/tierb-b200-gbs8/granularity_next_b200/tiera_moonep_probe_refresh
8
  https://huggingface.co/MoE-CUDA-Graph-Analysis/tierb-b200-gbs8/tree/main
9
+ Already logged in to Hugging Face as: AbhishekGhosh1998
granularity_next_b200/tiera_moonep_probe_refresh/moonep/tierbprobe/tierbprobe_gbs16/G1__moonep__matched__seq4096__gbs16__capnone__async/rep1/train.log CHANGED
The diff for this file is too large to render. See raw diff
 
granularity_next_b200/tiera_moonep_probe_refresh/moonep/tierbprobe/tierbprobe_gbs16/G1__moonep__matched__seq4096__gbs16__capnone__forcebal__async/rep1/train.log CHANGED
The diff for this file is too large to render. See raw diff
 
granularity_next_b200/tiera_moonep_probe_refresh/moonep/tierbprobe/tierbprobe_gbs16/G2__moonep__matched__seq4096__gbs16__capnone__async/rep1/train.log CHANGED
The diff for this file is too large to render. See raw diff
 
granularity_next_b200/tiera_moonep_probe_refresh/moonep/tierbprobe/tierbprobe_gbs16/G3__moonep__matched__seq4096__gbs16__capnone__async/rep1/train.log CHANGED
The diff for this file is too large to render. See raw diff
 
granularity_next_b200/tiera_moonep_probe_refresh/study.log CHANGED
@@ -174,4 +174,34 @@ moonep/tierbprobe/tierbprobe_gbs16/G4__moonep__match forced none 1.0177
174
  ECHO note: echo_logged_pre_rank_metric_valid=0 and echo_logged_load_reduction_valid=0 are intentional.
175
  The branch's built-in fields mix source-rank and destination-rank axes; this tool reconstructs both globally.
176
  no memory-dump runs found under /workspace/granularity_next_b200/tiera_moonep_probe_refresh
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
177
  Already logged in to Hugging Face as: AbhishekGhosh1998
 
174
  ECHO note: echo_logged_pre_rank_metric_valid=0 and echo_logged_load_reduction_valid=0 are intentional.
175
  The branch's built-in fields mix source-rank and destination-rank axes; this tool reconstructs both globally.
176
  no memory-dump runs found under /workspace/granularity_next_b200/tiera_moonep_probe_refresh
177
+ Already logged in to Hugging Face as: AbhishekGhosh1998
178
+ uploaded: /workspace/granularity_next_b200/tiera_moonep_probe_refresh -> MoE-CUDA-Graph-Analysis/tierb-b200-gbs8/granularity_next_b200/tiera_moonep_probe_refresh
179
+ https://huggingface.co/MoE-CUDA-Graph-Analysis/tierb-b200-gbs8/tree/main
180
+
181
+ === STUDY COMPLETE
182
+ results : /workspace/granularity_next_b200/tiera_moonep_probe_refresh
183
+ status : /workspace/granularity_next_b200/tiera_moonep_probe_refresh/STATUS.txt
184
+ steps : /workspace/granularity_next_b200/tiera_moonep_probe_refresh/steps.tsv
185
+ hub : https://huggingface.co/datasets/MoE-CUDA-Graph-Analysis/tierb-b200-gbs8
186
+
187
+ status campaign phase gbs started ended result_dir
188
+ OK moonep tierbprobe 16 2026-09-16T21:33:28+00:00 2026-09-16T21:43:29+00:00 /workspace/granularity_next_b200/tiera_moonep_probe_refresh/moonep/tierbprobe
189
+ study_root: /workspace/granularity_next_b200/tiera_moonep_probe_refresh
190
+ gbs: 16
191
+ stages: 7
192
+ nsys: 1
193
+ hf_repo: MoE-CUDA-Graph-Analysis/tierb-b200-gbs8
194
+ hf_remote_root: granularity_next_b200/tiera_moonep_probe_refresh
195
+ started: 2026-09-21T05:22:00+00:00
196
+ driver_repo: /workspace/blackwell_run_scripts @ b8bd100
197
+
198
+ === STAGE 7 -- Tier B probe and zero-drop capacity selection
199
+
200
+ === ROLL-UP
201
+ no successful runs to compare
202
+ 8 runs ok=0 oom=0 -> /workspace/granularity_next_b200/tiera_moonep_probe_refresh/collected_all.tsv
203
+
204
+ no probe telemetry found under /workspace/granularity_next_b200/tiera_moonep_probe_refresh
205
+ expected instrumented branches: token-drop-analysis / ultraep-token-drop-analysis / moonep_backend_token_routing_analysis
206
+ no memory-dump runs found under /workspace/granularity_next_b200/tiera_moonep_probe_refresh
207
  Already logged in to Hugging Face as: AbhishekGhosh1998