granularity results granularity_next_b200/tiera_moonep_probe_refresh @ 2026-09-21T05:22:01Z from sp-aiinfra-interactive-aghosh-8-mb99k-8xb200-moe-cg-0
Browse files- granularity_next_b200/tiera_moonep_probe_refresh/STATUS.txt +2 -2
- granularity_next_b200/tiera_moonep_probe_refresh/collected_all.tsv +8 -8
- granularity_next_b200/tiera_moonep_probe_refresh/hf_watcher.log +1 -0
- granularity_next_b200/tiera_moonep_probe_refresh/moonep/tierbprobe/tierbprobe_gbs16/G1__moonep__matched__seq4096__gbs16__capnone__async/rep1/train.log +0 -0
- granularity_next_b200/tiera_moonep_probe_refresh/moonep/tierbprobe/tierbprobe_gbs16/G1__moonep__matched__seq4096__gbs16__capnone__forcebal__async/rep1/train.log +0 -0
- granularity_next_b200/tiera_moonep_probe_refresh/moonep/tierbprobe/tierbprobe_gbs16/G2__moonep__matched__seq4096__gbs16__capnone__async/rep1/train.log +0 -0
- granularity_next_b200/tiera_moonep_probe_refresh/moonep/tierbprobe/tierbprobe_gbs16/G3__moonep__matched__seq4096__gbs16__capnone__async/rep1/train.log +0 -0
- granularity_next_b200/tiera_moonep_probe_refresh/study.log +30 -0
granularity_next_b200/tiera_moonep_probe_refresh/STATUS.txt
CHANGED
|
@@ -1,7 +1,7 @@
|
|
| 1 |
study_root: /workspace/granularity_next_b200/tiera_moonep_probe_refresh
|
| 2 |
gbs: 16
|
| 3 |
-
updated: 2026-09-
|
| 4 |
-
current:
|
| 5 |
|
| 6 |
completed steps:
|
| 7 |
moonep__tierbprobe__gbs16
|
|
|
|
| 1 |
study_root: /workspace/granularity_next_b200/tiera_moonep_probe_refresh
|
| 2 |
gbs: 16
|
| 3 |
+
updated: 2026-09-21T05:22:00+00:00
|
| 4 |
+
current: starting
|
| 5 |
|
| 6 |
completed steps:
|
| 7 |
moonep__tierbprobe__gbs16
|
granularity_next_b200/tiera_moonep_probe_refresh/collected_all.tsv
CHANGED
|
@@ -1,9 +1,9 @@
|
|
| 1 |
run_dir rep phase is_trace async config backend slot_policy slots_per_rank num_experts topk moe_ffn_hidden experts_per_rank expert_params num_layers seq_length global_batch_size num_microbatches assignments_per_rank mean_tokens_per_expert capacity capacity_mode recompute force_balance cuda_graph memory_dump_mode memory_snapshot_path ultraep_quota_min_tokens shared_expert_overlap branch commit dense_first_layers moe_layers router_balancing aux_loss_coeff router_pre_softmax router_fusion rotary_base qkv_bias source_model standard_num_layers base_rung q_lora_rank kv_lora_rank qk_head_dim qk_pos_emb_head_dim v_head_dim rotary_percent rotary_scaling mscale mscale_all_dim router_groups router_group_topk router_score_function router_expert_bias router_bias_update_rate router_topk_scaling_factor shared_expert_gate apply_layernorm_1p attention_output_gate linear_attention_type linear_attention_freq linear_conv_kernel_dim linear_key_head_dim linear_value_head_dim linear_num_key_heads linear_num_value_heads mtp_num_layers mtp_loss_scaling_factor no_weight_decay_cond_type max_position_embeddings init_method_std status iters_seen iter_ms_mean iter_ms_min iter_ms_stdev tflops peak_mib_sampled megatron_max_reserved_mib drop_warning
|
| 2 |
-
moonep/tierbprobe/tierbprobe_gbs16/G1__moonep__matched__seq4096__gbs16__capnone__async/rep1 rep1 tierbprobe 0 1 G1 moonep matched 1 8 2 14336 1 176160768 16 4096 16 2 8192 8192 none none none 0 0 none 0 moonep_backend_async_grad_reduce_token_routing_analysis a6a1a221 0 16 aux_loss 1e-2 0 0 1000000 0 512 128 64 128 1.0 softmax 0 0 0 0 0 0.1 32768 0.01
|
| 3 |
-
moonep/tierbprobe/tierbprobe_gbs16/G1__moonep__matched__seq4096__gbs16__capnone__forcebal__async/rep1 rep1 tierbprobe 0 1 G1 moonep matched 1 8 2 14336 1 176160768 16 4096 16 2 8192 8192 none none none 1 0 none 0 moonep_backend_async_grad_reduce_token_routing_analysis a6a1a221 0 16 aux_loss 1e-2 0 0 1000000 0 512 128 64 128 1.0 softmax 0 0 0 0 0 0.1 32768 0.01
|
| 4 |
-
moonep/tierbprobe/tierbprobe_gbs16/G2__moonep__matched__seq4096__gbs16__capnone__async/rep1 rep1 tierbprobe 0 1 G2 moonep matched 4 32 8 3584 4 44040192 16 4096 16 2 32768 8192 none none none 0 0 none 0 moonep_backend_async_grad_reduce_token_routing_analysis a6a1a221 0 16 aux_loss 1e-2 0 0 1000000 0 512 128 64 128 1.0 softmax 0 0 0 0 0 0.1 32768 0.01
|
| 5 |
-
moonep/tierbprobe/tierbprobe_gbs16/G2__moonep__matched__seq4096__gbs16__capnone__forcebal__async/rep1 rep1 tierbprobe 0 1 G2 moonep matched 4 32 8 3584 4 44040192 16 4096 16 2 32768 8192 none none none 1 0 none 0 moonep_backend_async_grad_reduce_token_routing_analysis a6a1a221 0 16 aux_loss 1e-2 0 0 1000000 0 512 128 64 128 1.0 softmax 0 0 0 0 0 0.1 32768 0.01
|
| 6 |
-
moonep/tierbprobe/tierbprobe_gbs16/G3__moonep__matched__seq4096__gbs16__capnone__async/rep1 rep1 tierbprobe 0 1 G3 moonep matched 8 64 16 1792 8 22020096 16 4096 16 2 65536 8192 none none none 0 0 none 0 moonep_backend_async_grad_reduce_token_routing_analysis a6a1a221 0 16 aux_loss 1e-2 0 0 1000000 0 512 128 64 128 1.0 softmax 0 0 0 0 0 0.1 32768 0.01
|
| 7 |
-
moonep/tierbprobe/tierbprobe_gbs16/G3__moonep__matched__seq4096__gbs16__capnone__forcebal__async/rep1 rep1 tierbprobe 0 1 G3 moonep matched 8 64 16 1792 8 22020096 16 4096 16 2 65536 8192 none none none 1 0 none 0 moonep_backend_async_grad_reduce_token_routing_analysis a6a1a221 0 16 aux_loss 1e-2 0 0 1000000 0 512 128 64 128 1.0 softmax 0 0 0 0 0 0.1 32768 0.01
|
| 8 |
-
moonep/tierbprobe/tierbprobe_gbs16/G4__moonep__matched__seq4096__gbs16__capnone__async/rep1 rep1 tierbprobe 0 1 G4 moonep matched 16 128 32 896 16 11010048 16 4096 16 2 131072 8192 none none none 0 0 none 0 moonep_backend_async_grad_reduce_token_routing_analysis a6a1a221 0 16 aux_loss 1e-2 0 0 1000000 0 512 128 64 128 1.0 softmax 0 0 0 0 0 0.1 32768 0.01
|
| 9 |
-
moonep/tierbprobe/tierbprobe_gbs16/G4__moonep__matched__seq4096__gbs16__capnone__forcebal__async/rep1 rep1 tierbprobe 0 1 G4 moonep matched 16 128 32 896 16 11010048 16 4096 16 2 131072 8192 none none none 1 0 none 0 moonep_backend_async_grad_reduce_token_routing_analysis a6a1a221 0 16 aux_loss 1e-2 0 0 1000000 0 512 128 64 128 1.0 softmax 0 0 0 0 0 0.1 32768 0.01
|
|
|
|
| 1 |
run_dir rep phase is_trace async config backend slot_policy slots_per_rank num_experts topk moe_ffn_hidden experts_per_rank expert_params num_layers seq_length global_batch_size num_microbatches assignments_per_rank mean_tokens_per_expert capacity capacity_mode recompute force_balance cuda_graph memory_dump_mode memory_snapshot_path ultraep_quota_min_tokens shared_expert_overlap branch commit dense_first_layers moe_layers router_balancing aux_loss_coeff router_pre_softmax router_fusion rotary_base qkv_bias source_model standard_num_layers base_rung q_lora_rank kv_lora_rank qk_head_dim qk_pos_emb_head_dim v_head_dim rotary_percent rotary_scaling mscale mscale_all_dim router_groups router_group_topk router_score_function router_expert_bias router_bias_update_rate router_topk_scaling_factor shared_expert_gate apply_layernorm_1p attention_output_gate linear_attention_type linear_attention_freq linear_conv_kernel_dim linear_key_head_dim linear_value_head_dim linear_num_key_heads linear_num_value_heads mtp_num_layers mtp_loss_scaling_factor no_weight_decay_cond_type max_position_embeddings init_method_std status iters_seen iter_ms_mean iter_ms_min iter_ms_stdev tflops peak_mib_sampled megatron_max_reserved_mib drop_warning
|
| 2 |
+
moonep/tierbprobe/tierbprobe_gbs16/G1__moonep__matched__seq4096__gbs16__capnone__async/rep1 rep1 tierbprobe 0 1 G1 moonep matched 1 8 2 14336 1 176160768 16 4096 16 2 8192 8192 none none none 0 0 none 0 moonep_backend_async_grad_reduce_token_routing_analysis a6a1a221 0 16 aux_loss 1e-2 0 0 1000000 0 512 128 64 128 1.0 softmax 0 0 0 0 0 0.1 32768 0.01 INCOMPLETE 0
|
| 3 |
+
moonep/tierbprobe/tierbprobe_gbs16/G1__moonep__matched__seq4096__gbs16__capnone__forcebal__async/rep1 rep1 tierbprobe 0 1 G1 moonep matched 1 8 2 14336 1 176160768 16 4096 16 2 8192 8192 none none none 1 0 none 0 moonep_backend_async_grad_reduce_token_routing_analysis a6a1a221 0 16 aux_loss 1e-2 0 0 1000000 0 512 128 64 128 1.0 softmax 0 0 0 0 0 0.1 32768 0.01 INCOMPLETE 0
|
| 4 |
+
moonep/tierbprobe/tierbprobe_gbs16/G2__moonep__matched__seq4096__gbs16__capnone__async/rep1 rep1 tierbprobe 0 1 G2 moonep matched 4 32 8 3584 4 44040192 16 4096 16 2 32768 8192 none none none 0 0 none 0 moonep_backend_async_grad_reduce_token_routing_analysis a6a1a221 0 16 aux_loss 1e-2 0 0 1000000 0 512 128 64 128 1.0 softmax 0 0 0 0 0 0.1 32768 0.01 INCOMPLETE 0
|
| 5 |
+
moonep/tierbprobe/tierbprobe_gbs16/G2__moonep__matched__seq4096__gbs16__capnone__forcebal__async/rep1 rep1 tierbprobe 0 1 G2 moonep matched 4 32 8 3584 4 44040192 16 4096 16 2 32768 8192 none none none 1 0 none 0 moonep_backend_async_grad_reduce_token_routing_analysis a6a1a221 0 16 aux_loss 1e-2 0 0 1000000 0 512 128 64 128 1.0 softmax 0 0 0 0 0 0.1 32768 0.01 NOLOG 0
|
| 6 |
+
moonep/tierbprobe/tierbprobe_gbs16/G3__moonep__matched__seq4096__gbs16__capnone__async/rep1 rep1 tierbprobe 0 1 G3 moonep matched 8 64 16 1792 8 22020096 16 4096 16 2 65536 8192 none none none 0 0 none 0 moonep_backend_async_grad_reduce_token_routing_analysis a6a1a221 0 16 aux_loss 1e-2 0 0 1000000 0 512 128 64 128 1.0 softmax 0 0 0 0 0 0.1 32768 0.01 INCOMPLETE 0
|
| 7 |
+
moonep/tierbprobe/tierbprobe_gbs16/G3__moonep__matched__seq4096__gbs16__capnone__forcebal__async/rep1 rep1 tierbprobe 0 1 G3 moonep matched 8 64 16 1792 8 22020096 16 4096 16 2 65536 8192 none none none 1 0 none 0 moonep_backend_async_grad_reduce_token_routing_analysis a6a1a221 0 16 aux_loss 1e-2 0 0 1000000 0 512 128 64 128 1.0 softmax 0 0 0 0 0 0.1 32768 0.01 NOLOG 0
|
| 8 |
+
moonep/tierbprobe/tierbprobe_gbs16/G4__moonep__matched__seq4096__gbs16__capnone__async/rep1 rep1 tierbprobe 0 1 G4 moonep matched 16 128 32 896 16 11010048 16 4096 16 2 131072 8192 none none none 0 0 none 0 moonep_backend_async_grad_reduce_token_routing_analysis a6a1a221 0 16 aux_loss 1e-2 0 0 1000000 0 512 128 64 128 1.0 softmax 0 0 0 0 0 0.1 32768 0.01 NOLOG 0
|
| 9 |
+
moonep/tierbprobe/tierbprobe_gbs16/G4__moonep__matched__seq4096__gbs16__capnone__forcebal__async/rep1 rep1 tierbprobe 0 1 G4 moonep matched 16 128 32 896 16 11010048 16 4096 16 2 131072 8192 none none none 1 0 none 0 moonep_backend_async_grad_reduce_token_routing_analysis a6a1a221 0 16 aux_loss 1e-2 0 0 1000000 0 512 128 64 128 1.0 softmax 0 0 0 0 0 0.1 32768 0.01 NOLOG 0
|
granularity_next_b200/tiera_moonep_probe_refresh/hf_watcher.log
CHANGED
|
@@ -6,3 +6,4 @@ https://huggingface.co/MoE-CUDA-Graph-Analysis/tierb-b200-gbs8/tree/main
|
|
| 6 |
[2026-09-16T21:43:29+00:00] backup pass
|
| 7 |
uploaded: /workspace/granularity_next_b200/tiera_moonep_probe_refresh -> MoE-CUDA-Graph-Analysis/tierb-b200-gbs8/granularity_next_b200/tiera_moonep_probe_refresh
|
| 8 |
https://huggingface.co/MoE-CUDA-Graph-Analysis/tierb-b200-gbs8/tree/main
|
|
|
|
|
|
| 6 |
[2026-09-16T21:43:29+00:00] backup pass
|
| 7 |
uploaded: /workspace/granularity_next_b200/tiera_moonep_probe_refresh -> MoE-CUDA-Graph-Analysis/tierb-b200-gbs8/granularity_next_b200/tiera_moonep_probe_refresh
|
| 8 |
https://huggingface.co/MoE-CUDA-Graph-Analysis/tierb-b200-gbs8/tree/main
|
| 9 |
+
Already logged in to Hugging Face as: AbhishekGhosh1998
|
granularity_next_b200/tiera_moonep_probe_refresh/moonep/tierbprobe/tierbprobe_gbs16/G1__moonep__matched__seq4096__gbs16__capnone__async/rep1/train.log
CHANGED
|
The diff for this file is too large to render.
See raw diff
|
|
|
granularity_next_b200/tiera_moonep_probe_refresh/moonep/tierbprobe/tierbprobe_gbs16/G1__moonep__matched__seq4096__gbs16__capnone__forcebal__async/rep1/train.log
CHANGED
|
The diff for this file is too large to render.
See raw diff
|
|
|
granularity_next_b200/tiera_moonep_probe_refresh/moonep/tierbprobe/tierbprobe_gbs16/G2__moonep__matched__seq4096__gbs16__capnone__async/rep1/train.log
CHANGED
|
The diff for this file is too large to render.
See raw diff
|
|
|
granularity_next_b200/tiera_moonep_probe_refresh/moonep/tierbprobe/tierbprobe_gbs16/G3__moonep__matched__seq4096__gbs16__capnone__async/rep1/train.log
CHANGED
|
The diff for this file is too large to render.
See raw diff
|
|
|
granularity_next_b200/tiera_moonep_probe_refresh/study.log
CHANGED
|
@@ -174,4 +174,34 @@ moonep/tierbprobe/tierbprobe_gbs16/G4__moonep__match forced none 1.0177
|
|
| 174 |
ECHO note: echo_logged_pre_rank_metric_valid=0 and echo_logged_load_reduction_valid=0 are intentional.
|
| 175 |
The branch's built-in fields mix source-rank and destination-rank axes; this tool reconstructs both globally.
|
| 176 |
no memory-dump runs found under /workspace/granularity_next_b200/tiera_moonep_probe_refresh
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 177 |
Already logged in to Hugging Face as: AbhishekGhosh1998
|
|
|
|
| 174 |
ECHO note: echo_logged_pre_rank_metric_valid=0 and echo_logged_load_reduction_valid=0 are intentional.
|
| 175 |
The branch's built-in fields mix source-rank and destination-rank axes; this tool reconstructs both globally.
|
| 176 |
no memory-dump runs found under /workspace/granularity_next_b200/tiera_moonep_probe_refresh
|
| 177 |
+
Already logged in to Hugging Face as: AbhishekGhosh1998
|
| 178 |
+
uploaded: /workspace/granularity_next_b200/tiera_moonep_probe_refresh -> MoE-CUDA-Graph-Analysis/tierb-b200-gbs8/granularity_next_b200/tiera_moonep_probe_refresh
|
| 179 |
+
https://huggingface.co/MoE-CUDA-Graph-Analysis/tierb-b200-gbs8/tree/main
|
| 180 |
+
|
| 181 |
+
=== STUDY COMPLETE
|
| 182 |
+
results : /workspace/granularity_next_b200/tiera_moonep_probe_refresh
|
| 183 |
+
status : /workspace/granularity_next_b200/tiera_moonep_probe_refresh/STATUS.txt
|
| 184 |
+
steps : /workspace/granularity_next_b200/tiera_moonep_probe_refresh/steps.tsv
|
| 185 |
+
hub : https://huggingface.co/datasets/MoE-CUDA-Graph-Analysis/tierb-b200-gbs8
|
| 186 |
+
|
| 187 |
+
status campaign phase gbs started ended result_dir
|
| 188 |
+
OK moonep tierbprobe 16 2026-09-16T21:33:28+00:00 2026-09-16T21:43:29+00:00 /workspace/granularity_next_b200/tiera_moonep_probe_refresh/moonep/tierbprobe
|
| 189 |
+
study_root: /workspace/granularity_next_b200/tiera_moonep_probe_refresh
|
| 190 |
+
gbs: 16
|
| 191 |
+
stages: 7
|
| 192 |
+
nsys: 1
|
| 193 |
+
hf_repo: MoE-CUDA-Graph-Analysis/tierb-b200-gbs8
|
| 194 |
+
hf_remote_root: granularity_next_b200/tiera_moonep_probe_refresh
|
| 195 |
+
started: 2026-09-21T05:22:00+00:00
|
| 196 |
+
driver_repo: /workspace/blackwell_run_scripts @ b8bd100
|
| 197 |
+
|
| 198 |
+
=== STAGE 7 -- Tier B probe and zero-drop capacity selection
|
| 199 |
+
|
| 200 |
+
=== ROLL-UP
|
| 201 |
+
no successful runs to compare
|
| 202 |
+
8 runs ok=0 oom=0 -> /workspace/granularity_next_b200/tiera_moonep_probe_refresh/collected_all.tsv
|
| 203 |
+
|
| 204 |
+
no probe telemetry found under /workspace/granularity_next_b200/tiera_moonep_probe_refresh
|
| 205 |
+
expected instrumented branches: token-drop-analysis / ultraep-token-drop-analysis / moonep_backend_token_routing_analysis
|
| 206 |
+
no memory-dump runs found under /workspace/granularity_next_b200/tiera_moonep_probe_refresh
|
| 207 |
Already logged in to Hugging Face as: AbhishekGhosh1998
|