granularity results granularity_next_b200/tiera_topk_trace @ 2026-09-21T06:19:40Z from sp-aiinfra-interactive-aghosh-8-mb99k-8xb200-moe-cg-0
Browse files- granularity_next_b200/tiera_topk_trace/.completed/moonep__tierbtrace__gbs16 +1 -0
- granularity_next_b200/tiera_topk_trace/STATUS.txt +3 -2
- granularity_next_b200/tiera_topk_trace/collected_all.tsv +28 -0
- granularity_next_b200/tiera_topk_trace/hf_watcher.log +2 -0
- granularity_next_b200/tiera_topk_trace/steps.tsv +1 -0
- granularity_next_b200/tiera_topk_trace/study.log +44 -0
granularity_next_b200/tiera_topk_trace/.completed/moonep__tierbtrace__gbs16
ADDED
|
@@ -0,0 +1 @@
|
|
|
|
|
|
|
| 1 |
+
2026-09-21T06:19:33+00:00
|
granularity_next_b200/tiera_topk_trace/STATUS.txt
CHANGED
|
@@ -1,10 +1,11 @@
|
|
| 1 |
study_root: /workspace/granularity_next_b200/tiera_topk_trace
|
| 2 |
gbs: 16
|
| 3 |
-
updated: 2026-09-
|
| 4 |
-
current:
|
| 5 |
|
| 6 |
completed steps:
|
| 7 |
echo__tierbtrace__gbs16
|
|
|
|
| 8 |
ultraep__tierbtrace__gbs16
|
| 9 |
|
| 10 |
failed steps:
|
|
|
|
| 1 |
study_root: /workspace/granularity_next_b200/tiera_topk_trace
|
| 2 |
gbs: 16
|
| 3 |
+
updated: 2026-09-21T06:19:38+00:00
|
| 4 |
+
current: idle
|
| 5 |
|
| 6 |
completed steps:
|
| 7 |
echo__tierbtrace__gbs16
|
| 8 |
+
moonep__tierbtrace__gbs16
|
| 9 |
ultraep__tierbtrace__gbs16
|
| 10 |
|
| 11 |
failed steps:
|
granularity_next_b200/tiera_topk_trace/collected_all.tsv
ADDED
|
@@ -0,0 +1,28 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
run_dir rep phase is_trace async config backend slot_policy slots_per_rank num_experts topk moe_ffn_hidden experts_per_rank expert_params num_layers seq_length global_batch_size num_microbatches assignments_per_rank mean_tokens_per_expert capacity capacity_mode recompute force_balance cuda_graph memory_dump_mode memory_snapshot_path ultraep_quota_min_tokens shared_expert_overlap branch commit dense_first_layers moe_layers router_balancing aux_loss_coeff router_pre_softmax router_fusion rotary_base qkv_bias source_model standard_num_layers base_rung q_lora_rank kv_lora_rank qk_head_dim qk_pos_emb_head_dim v_head_dim rotary_percent rotary_scaling mscale mscale_all_dim router_groups router_group_topk router_score_function router_expert_bias router_bias_update_rate router_topk_scaling_factor shared_expert_gate apply_layernorm_1p attention_output_gate linear_attention_type linear_attention_freq linear_conv_kernel_dim linear_key_head_dim linear_value_head_dim linear_num_key_heads linear_num_value_heads mtp_num_layers mtp_loss_scaling_factor no_weight_decay_cond_type max_position_embeddings init_method_std status iters_seen iter_ms_mean iter_ms_min iter_ms_stdev tflops peak_mib_sampled megatron_max_reserved_mib drop_warning
|
| 2 |
+
echo/tierbtrace/tierbtrace_gbs16/G2_K2__echo__matched__seq4096__gbs16__cap1p15625/nsys tierbtrace 1 0 G2_K2 echo matched 4 32 2 3584 4 44040192 16 4096 16 2 8192 2048 1.15625 static none 0 1 none 0 bracing-regions-through-kernels 9a236793 0 16 aux_loss 1e-2 0 0 1000000 0 G2 512 128 64 128 1.0 softmax 0 0 0 0 0 0.1 32768 0.01 OK 8 1189.47 275.90 1574.99 312.8 91132
|
| 3 |
+
echo/tierbtrace/tierbtrace_gbs16/G3_K2__echo__matched__seq4096__gbs16__cap1p1875/nsys tierbtrace 1 0 G3_K2 echo matched 8 64 2 1792 8 22020096 16 4096 16 2 8192 1024 1.1875 static none 0 1 none 0 bracing-regions-through-kernels 9a236793 0 16 aux_loss 1e-2 0 0 1000000 0 G3 512 128 64 128 1.0 softmax 0 0 0 0 0 0.1 32768 0.01 OK 8 1188.93 252.00 1613.99 245.8 92666
|
| 4 |
+
echo/tierbtrace/tierbtrace_gbs16/G3_K8__echo__matched__seq4096__gbs16__cap1p09375/nsys tierbtrace 1 0 G3_K8 echo matched 8 64 8 1792 8 22020096 16 4096 16 2 32768 4096 1.09375 static none 0 1 none 0 bracing-regions-through-kernels 9a236793 0 16 aux_loss 1e-2 0 0 1000000 0 G3 512 128 64 128 1.0 softmax 0 0 0 0 0 0.1 32768 0.01 OK 8 1316.07 360.30 1643.50 374.1 88092
|
| 5 |
+
echo/tierbtrace/tierbtrace_gbs16/G4_K10__echo__matched__seq4096__gbs16__cap1p1/nsys tierbtrace 1 0 G4_K10 echo matched 16 128 10 896 16 11010048 16 4096 16 2 40960 2560 1.1 static none 0 1 none 0 bracing-regions-through-kernels 9a236793 0 16 aux_loss 1e-2 0 0 1000000 0 G4 512 128 64 128 1.0 softmax 0 0 0 0 0 0.1 32768 0.01 OK 8 1317.13 353.10 1654.36 278.1 84052
|
| 6 |
+
echo/tierbtrace/tierbtrace_gbs16/G4_K16__echo__matched__seq4096__gbs16__cap1p08203125/nsys tierbtrace 1 0 G4_K16 echo matched 16 128 16 896 16 11010048 16 4096 16 2 65536 4096 1.08203125 static none 0 1 none 0 bracing-regions-through-kernels 9a236793 0 16 aux_loss 1e-2 0 0 1000000 0 G4 512 128 64 128 1.0 softmax 0 0 0 0 0 0.1 32768 0.01 OK 8 1427.67 392.70 1780.94 343.9 87402
|
| 7 |
+
echo/tierbtrace/tierbtrace_gbs16/G4_K2__echo__matched__seq4096__gbs16__cap1p265625/nsys tierbtrace 1 0 G4_K2 echo matched 16 128 2 896 16 11010048 16 4096 16 2 8192 512 1.265625 static none 0 1 none 0 bracing-regions-through-kernels 9a236793 0 16 aux_loss 1e-2 0 0 1000000 0 G4 512 128 64 128 1.0 softmax 0 0 0 0 0 0.1 32768 0.01 OK 8 1196.23 254.40 1617.12 195.1 85790
|
| 8 |
+
echo/tierbtrace/tierbtrace_gbs16/G4_K4__echo__matched__seq4096__gbs16__cap1p171875/nsys tierbtrace 1 0 G4_K4 echo matched 16 128 4 896 16 11010048 16 4096 16 2 16384 1024 1.171875 static none 0 1 none 0 bracing-regions-through-kernels 9a236793 0 16 aux_loss 1e-2 0 0 1000000 0 G4 512 128 64 128 1.0 softmax 0 0 0 0 0 0.1 32768 0.01 OK 8 1211.43 283.00 1592.44 217.7 84426
|
| 9 |
+
echo/tierbtrace/tierbtrace_gbs16/G4_K6__echo__matched__seq4096__gbs16__cap1p1354166666666667/nsys tierbtrace 1 0 G4_K6 echo matched 16 128 6 896 16 11010048 16 4096 16 2 24576 1536 1.1354166666666667 static none 0 1 none 0 bracing-regions-through-kernels 9a236793 0 16 aux_loss 1e-2 0 0 1000000 0 G4 512 128 64 128 1.0 softmax 0 0 0 0 0 0.1 32768 0.01 OK 8 1222.50 310.70 1565.62 238.6 82684
|
| 10 |
+
echo/tierbtrace/tierbtrace_gbs16/G4_K8__echo__matched__seq4096__gbs16__cap1p109375/nsys tierbtrace 1 0 G4_K8 echo matched 16 128 8 896 16 11010048 16 4096 16 2 32768 2048 1.109375 static none 0 1 none 0 bracing-regions-through-kernels 9a236793 0 16 aux_loss 1e-2 0 0 1000000 0 G4 512 128 64 128 1.0 softmax 0 0 0 0 0 0.1 32768 0.01 OK 8 1322.33 341.50 1686.48 253.0 82888
|
| 11 |
+
moonep/tierbtrace/tierbtrace_gbs16/G2_K2__moonep__matched__seq4096__gbs16__capnone__async/nsys tierbtrace 1 1 G2_K2 moonep matched 4 32 2 3584 4 44040192 16 4096 16 2 8192 2048 none none none 0 1 none 0 moonep_backend_async_grad_reduce 2e68c8ad 0 16 aux_loss 1e-2 0 0 1000000 0 G2 512 128 64 128 1.0 softmax 0 0 0 0 0 0.1 32768 0.01 OK 10 9382.30 257.00 18811.09 290.2 57302
|
| 12 |
+
moonep/tierbtrace/tierbtrace_gbs16/G3_K2__moonep__matched__seq4096__gbs16__capnone__async/nsys tierbtrace 1 1 G3_K2 moonep matched 8 64 2 1792 8 22020096 16 4096 16 2 8192 1024 none none none 0 1 none 0 moonep_backend_async_grad_reduce 2e68c8ad 0 16 aux_loss 1e-2 0 0 1000000 0 G3 512 128 64 128 1.0 softmax 0 0 0 0 0 0.1 32768 0.01 OK 10 9418.78 229.30 18933.32 233.2 56294
|
| 13 |
+
moonep/tierbtrace/tierbtrace_gbs16/G3_K8__moonep__matched__seq4096__gbs16__capnone__async/nsys tierbtrace 1 1 G3_K8 moonep matched 8 64 8 1792 8 22020096 16 4096 16 2 32768 4096 none none none 0 1 none 0 moonep_backend_async_grad_reduce 2e68c8ad 0 16 aux_loss 1e-2 0 0 1000000 0 G3 512 128 64 128 1.0 softmax 0 0 0 0 0 0.1 32768 0.01 OK 10 9577.18 365.30 19002.74 323.9 56022
|
| 14 |
+
moonep/tierbtrace/tierbtrace_gbs16/G4_K10__moonep__matched__seq4096__gbs16__capnone__async/nsys tierbtrace 1 1 G4_K10 moonep matched 16 128 10 896 16 11010048 16 4096 16 2 40960 2560 none none none 0 1 none 0 moonep_backend_async_grad_reduce 2e68c8ad 0 16 aux_loss 1e-2 0 0 1000000 0 G4 512 128 64 128 1.0 softmax 0 0 0 0 0 0.1 32768 0.01 OK 10 9675.82 373.20 19244.90 233.4 55002
|
| 15 |
+
moonep/tierbtrace/tierbtrace_gbs16/G4_K16__moonep__matched__seq4096__gbs16__capnone__async/nsys tierbtrace 1 1 G4_K16 moonep matched 16 128 16 896 16 11010048 16 4096 16 2 65536 4096 none none none 0 1 none 0 moonep_backend_async_grad_reduce 2e68c8ad 0 16 aux_loss 1e-2 0 0 1000000 0 G4 512 128 64 128 1.0 softmax 0 0 0 0 0 0.1 32768 0.01 OK 10 9755.22 461.10 19235.63 261.0 54818
|
| 16 |
+
moonep/tierbtrace/tierbtrace_gbs16/G4_K2__moonep__matched__seq4096__gbs16__capnone__async/nsys tierbtrace 1 1 G4_K2 moonep matched 16 128 2 896 16 11010048 16 4096 16 2 8192 512 none none none 0 1 none 0 moonep_backend_async_grad_reduce 2e68c8ad 0 16 aux_loss 1e-2 0 0 1000000 0 G4 512 128 64 128 1.0 softmax 0 0 0 0 0 0.1 32768 0.01 OK 8 1238.70 225.10 1740.39 217.7 55428
|
| 17 |
+
moonep/tierbtrace/tierbtrace_gbs16/G4_K4__moonep__matched__seq4096__gbs16__capnone__async/nsys tierbtrace 1 1 G4_K4 moonep matched 16 128 4 896 16 11010048 16 4096 16 2 16384 1024 none none none 0 1 none 0 moonep_backend_async_grad_reduce 2e68c8ad 0 16 aux_loss 1e-2 0 0 1000000 0 G4 512 128 64 128 1.0 softmax 0 0 0 0 0 0.1 32768 0.01 OK 10 9602.56 260.40 19184.66 206.1 55068
|
| 18 |
+
moonep/tierbtrace/tierbtrace_gbs16/G4_K6__moonep__matched__seq4096__gbs16__capnone__async/nsys tierbtrace 1 1 G4_K6 moonep matched 16 128 6 896 16 11010048 16 4096 16 2 24576 1536 none none none 0 1 none 0 moonep_backend_async_grad_reduce 2e68c8ad 0 16 aux_loss 1e-2 0 0 1000000 0 G4 512 128 64 128 1.0 softmax 0 0 0 0 0 0.1 32768 0.01 OK 10 9614.28 302.60 19178.87 212.9 54700
|
| 19 |
+
moonep/tierbtrace/tierbtrace_gbs16/G4_K8__moonep__matched__seq4096__gbs16__capnone__async/nsys tierbtrace 1 1 G4_K8 moonep matched 16 128 8 896 16 11010048 16 4096 16 2 32768 2048 none none none 0 1 none 0 moonep_backend_async_grad_reduce 2e68c8ad 0 16 aux_loss 1e-2 0 0 1000000 0 G4 512 128 64 128 1.0 softmax 0 0 0 0 0 0.1 32768 0.01 OK 10 9677.08 338.60 19270.50 224.0 55046
|
| 20 |
+
ultraep/tierbtrace/tierbtrace_gbs16/G2_K2__ultraep__matched__seq4096__gbs16__cap1p15625/nsys tierbtrace 1 0 G2_K2 ultraep matched 4 32 2 3584 4 44040192 16 4096 16 2 8192 2048 1.15625 static none 0 1 none 256 0 ultraep-bracing-regions-through-kernels 03d57ef7 0 16 aux_loss 1e-2 0 0 1000000 0 G2 512 128 64 128 1.0 softmax 0 0 0 0 0 0.1 32768 0.01 OK 10 9441.18 335.00 18757.13 225.4 79284
|
| 21 |
+
ultraep/tierbtrace/tierbtrace_gbs16/G3_K2__ultraep__matched__seq4096__gbs16__cap1p1875/nsys tierbtrace 1 0 G3_K2 ultraep matched 8 64 2 1792 8 22020096 16 4096 16 2 8192 1024 1.1875 static none 0 1 none 128 0 ultraep-bracing-regions-through-kernels 03d57ef7 0 16 aux_loss 1e-2 0 0 1000000 0 G3 512 128 64 128 1.0 softmax 0 0 0 0 0 0.1 32768 0.01 OK 8 1180.70 251.70 1598.69 245.4 80670
|
| 22 |
+
ultraep/tierbtrace/tierbtrace_gbs16/G3_K8__ultraep__matched__seq4096__gbs16__cap1p09375/nsys tierbtrace 1 0 G3_K8 ultraep matched 8 64 8 1792 8 22020096 16 4096 16 2 32768 4096 1.09375 static none 0 1 none 512 0 ultraep-bracing-regions-through-kernels 03d57ef7 0 16 aux_loss 1e-2 0 0 1000000 0 G3 512 128 64 128 1.0 softmax 0 0 0 0 0 0.1 32768 0.01 OK 10 9481.50 369.80 18722.67 320.7 76122
|
| 23 |
+
ultraep/tierbtrace/tierbtrace_gbs16/G4_K10__ultraep__matched__seq4096__gbs16__cap1p1/nsys tierbtrace 1 0 G4_K10 ultraep matched 16 128 10 896 16 11010048 16 4096 16 2 40960 2560 1.1 static none 0 1 none 320 0 ultraep-bracing-regions-through-kernels 03d57ef7 0 16 aux_loss 1e-2 0 0 1000000 0 G4 512 128 64 128 1.0 softmax 0 0 0 0 0 0.1 32768 0.01 OK 8 1269.97 347.70 1591.01 286.5 71676
|
| 24 |
+
ultraep/tierbtrace/tierbtrace_gbs16/G4_K16__ultraep__matched__seq4096__gbs16__cap1p08203125/nsys tierbtrace 1 0 G4_K16 ultraep matched 16 128 16 896 16 11010048 16 4096 16 2 65536 4096 1.08203125 static none 0 1 none 512 0 ultraep-bracing-regions-through-kernels 03d57ef7 0 16 aux_loss 1e-2 0 0 1000000 0 G4 512 128 64 128 1.0 softmax 0 0 0 0 0 0.1 32768 0.01 OK 8 1342.37 392.20 1642.01 349.5 75054
|
| 25 |
+
ultraep/tierbtrace/tierbtrace_gbs16/G4_K2__ultraep__matched__seq4096__gbs16__cap1p265625/nsys tierbtrace 1 0 G4_K2 ultraep matched 16 128 2 896 16 11010048 16 4096 16 2 8192 512 1.265625 static none 0 1 none 64 0 ultraep-bracing-regions-through-kernels 03d57ef7 0 16 aux_loss 1e-2 0 0 1000000 0 G4 512 128 64 128 1.0 softmax 0 0 0 0 0 0.1 32768 0.01 OK 10 9446.86 218.50 19048.80 194.7 73358
|
| 26 |
+
ultraep/tierbtrace/tierbtrace_gbs16/G4_K4__ultraep__matched__seq4096__gbs16__cap1p171875/nsys tierbtrace 1 0 G4_K4 ultraep matched 16 128 4 896 16 11010048 16 4096 16 2 16384 1024 1.171875 static none 0 1 none 128 0 ultraep-bracing-regions-through-kernels 03d57ef7 0 16 aux_loss 1e-2 0 0 1000000 0 G4 512 128 64 128 1.0 softmax 0 0 0 0 0 0.1 32768 0.01 OK 8 1296.37 248.70 1801.64 245.8 71948
|
| 27 |
+
ultraep/tierbtrace/tierbtrace_gbs16/G4_K6__ultraep__matched__seq4096__gbs16__cap1p1354166666666667/nsys tierbtrace 1 0 G4_K6 ultraep matched 16 128 6 896 16 11010048 16 4096 16 2 24576 1536 1.1354166666666667 static none 0 1 none 192 0 ultraep-bracing-regions-through-kernels 03d57ef7 0 16 aux_loss 1e-2 0 0 1000000 0 G4 512 128 64 128 1.0 softmax 0 0 0 0 0 0.1 32768 0.01 OK 8 1279.93 297.50 1692.71 249.7 70362
|
| 28 |
+
ultraep/tierbtrace/tierbtrace_gbs16/G4_K8__ultraep__matched__seq4096__gbs16__cap1p109375/nsys tierbtrace 1 0 G4_K8 ultraep matched 16 128 8 896 16 11010048 16 4096 16 2 32768 2048 1.109375 static none 0 1 none 256 0 ultraep-bracing-regions-through-kernels 03d57ef7 0 16 aux_loss 1e-2 0 0 1000000 0 G4 512 128 64 128 1.0 softmax 0 0 0 0 0 0.1 32768 0.01 OK 8 1324.80 322.70 1727.55 268.4 70522
|
granularity_next_b200/tiera_topk_trace/hf_watcher.log
CHANGED
|
@@ -51,3 +51,5 @@ https://huggingface.co/MoE-CUDA-Graph-Analysis/tierb-b200-gbs8/tree/main
|
|
| 51 |
uploaded: /workspace/granularity_next_b200/tiera_topk_trace -> MoE-CUDA-Graph-Analysis/tierb-b200-gbs8/granularity_next_b200/tiera_topk_trace
|
| 52 |
https://huggingface.co/MoE-CUDA-Graph-Analysis/tierb-b200-gbs8/tree/main
|
| 53 |
[2026-09-21T06:12:40+00:00] backup pass
|
|
|
|
|
|
|
|
|
| 51 |
uploaded: /workspace/granularity_next_b200/tiera_topk_trace -> MoE-CUDA-Graph-Analysis/tierb-b200-gbs8/granularity_next_b200/tiera_topk_trace
|
| 52 |
https://huggingface.co/MoE-CUDA-Graph-Analysis/tierb-b200-gbs8/tree/main
|
| 53 |
[2026-09-21T06:12:40+00:00] backup pass
|
| 54 |
+
uploaded: /workspace/granularity_next_b200/tiera_topk_trace -> MoE-CUDA-Graph-Analysis/tierb-b200-gbs8/granularity_next_b200/tiera_topk_trace
|
| 55 |
+
https://huggingface.co/MoE-CUDA-Graph-Analysis/tierb-b200-gbs8/tree/main
|
granularity_next_b200/tiera_topk_trace/steps.tsv
CHANGED
|
@@ -1,3 +1,4 @@
|
|
| 1 |
status campaign phase gbs started ended result_dir
|
| 2 |
OK echo tierbtrace 16 2026-09-17T01:54:35+00:00 2026-09-17T03:22:23+00:00 /workspace/granularity_next_b200/tiera_topk_trace/echo/tierbtrace
|
| 3 |
OK ultraep tierbtrace 16 2026-09-21T05:22:22+00:00 2026-09-21T05:53:08+00:00 /workspace/granularity_next_b200/tiera_topk_trace/ultraep/tierbtrace
|
|
|
|
|
|
| 1 |
status campaign phase gbs started ended result_dir
|
| 2 |
OK echo tierbtrace 16 2026-09-17T01:54:35+00:00 2026-09-17T03:22:23+00:00 /workspace/granularity_next_b200/tiera_topk_trace/echo/tierbtrace
|
| 3 |
OK ultraep tierbtrace 16 2026-09-21T05:22:22+00:00 2026-09-21T05:53:08+00:00 /workspace/granularity_next_b200/tiera_topk_trace/ultraep/tierbtrace
|
| 4 |
+
OK moonep tierbtrace 16 2026-09-21T05:53:11+00:00 2026-09-21T06:19:33+00:00 /workspace/granularity_next_b200/tiera_topk_trace/moonep/tierbtrace
|
granularity_next_b200/tiera_topk_trace/study.log
CHANGED
|
@@ -275,3 +275,47 @@ https://huggingface.co/MoE-CUDA-Graph-Analysis/tierb-b200-gbs8/tree/main
|
|
| 275 |
[guard] nsys stop (flushing report)
|
| 276 |
[06:12:23] trace verified -> G4_K6__moonep__matched__seq4096__gbs16__capnone__async.nsys-rep (13M, all 8 GPUs + moonep markers)
|
| 277 |
[06:12:23] TRACE-ONLY G4_K8__moonep__matched__seq4096__gbs16__capnone__async (ranks: 0 1 2 3 4 5 6 7)
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 275 |
[guard] nsys stop (flushing report)
|
| 276 |
[06:12:23] trace verified -> G4_K6__moonep__matched__seq4096__gbs16__capnone__async.nsys-rep (13M, all 8 GPUs + moonep markers)
|
| 277 |
[06:12:23] TRACE-ONLY G4_K8__moonep__matched__seq4096__gbs16__capnone__async (ranks: 0 1 2 3 4 5 6 7)
|
| 278 |
+
[guard] iter 6/10 (60s)
|
| 279 |
+
[guard] iter 8/10 (70s)
|
| 280 |
+
[guard] iter 10/10 (110s)
|
| 281 |
+
[guard] nsys stop (flushing report)
|
| 282 |
+
[06:14:46] trace verified -> G4_K8__moonep__matched__seq4096__gbs16__capnone__async.nsys-rep (13M, all 8 GPUs + moonep markers)
|
| 283 |
+
[06:14:47] TRACE-ONLY G4_K10__moonep__matched__seq4096__gbs16__capnone__async (ranks: 0 1 2 3 4 5 6 7)
|
| 284 |
+
[guard] iter 6/10 (60s)
|
| 285 |
+
[guard] iter 8/10 (70s)
|
| 286 |
+
[guard] iter 10/10 (110s)
|
| 287 |
+
[guard] nsys stop (flushing report)
|
| 288 |
+
[06:17:09] trace verified -> G4_K10__moonep__matched__seq4096__gbs16__capnone__async.nsys-rep (13M, all 8 GPUs + moonep markers)
|
| 289 |
+
[06:17:10] TRACE-ONLY G4_K16__moonep__matched__seq4096__gbs16__capnone__async (ranks: 0 1 2 3 4 5 6 7)
|
| 290 |
+
[guard] iter 4/10 (60s)
|
| 291 |
+
[guard] iter 8/10 (70s)
|
| 292 |
+
[guard] iter 10/10 (110s)
|
| 293 |
+
[guard] nsys stop (flushing report)
|
| 294 |
+
[06:19:32] trace verified -> G4_K16__moonep__matched__seq4096__gbs16__capnone__async.nsys-rep (13M, all 8 GPUs + moonep markers)
|
| 295 |
+
--------------------------------------------------------------------------
|
| 296 |
+
|
| 297 |
+
==============================================================
|
| 298 |
+
SWEEP COMPLETE ok=9 failed=0
|
| 299 |
+
==============================================================
|
| 300 |
+
config backend policy slots seq gbs cap recompute rep status iter_ms_mean iter_ms_min peak_mib run_dir
|
| 301 |
+
|
| 302 |
+
Results: /workspace/granularity_next_b200/tiera_topk_trace/moonep/tierbtrace/tierbtrace_gbs16/results.tsv
|
| 303 |
+
Roll up: python3 /workspace/blackwell_run_scripts/granularity/tools/collect.py /workspace/granularity_next_b200/tiera_topk_trace/moonep/tierbtrace/tierbtrace_gbs16
|
| 304 |
+
==============================================================
|
| 305 |
+
|
| 306 |
+
Campaign results: /workspace/granularity_next_b200/tiera_topk_trace/moonep/tierbtrace
|
| 307 |
+
Roll up: python3 /workspace/blackwell_run_scripts/granularity/tools/collect.py /workspace/granularity_next_b200/tiera_topk_trace/moonep/tierbtrace --compare
|
| 308 |
+
Already logged in to Hugging Face as: AbhishekGhosh1998
|
| 309 |
+
uploaded: /workspace/granularity_next_b200/tiera_topk_trace/moonep/tierbtrace -> MoE-CUDA-Graph-Analysis/tierb-b200-gbs8/granularity_next_b200/tiera_topk_trace/moonep/tierbtrace
|
| 310 |
+
https://huggingface.co/MoE-CUDA-Graph-Analysis/tierb-b200-gbs8/tree/main
|
| 311 |
+
|
| 312 |
+
=== ROLL-UP
|
| 313 |
+
no successful runs to compare
|
| 314 |
+
27 runs ok=27 oom=0 -> /workspace/granularity_next_b200/tiera_topk_trace/collected_all.tsv
|
| 315 |
+
|
| 316 |
+
(excluded 27 nsys-traced run(s) from the timing comparison)
|
| 317 |
+
|
| 318 |
+
no probe telemetry found under /workspace/granularity_next_b200/tiera_topk_trace
|
| 319 |
+
expected instrumented branches: token-drop-analysis / ultraep-token-drop-analysis / moonep_backend_token_routing_analysis
|
| 320 |
+
no memory-dump runs found under /workspace/granularity_next_b200/tiera_topk_trace
|
| 321 |
+
Already logged in to Hugging Face as: AbhishekGhosh1998
|