granularity results tiera-rerun-fixed-moonep @ 2026-08-30T05:49:17Z from sp-aiinfra-interactive-aghosh-7jtjl-8xb200-moe-cg-0
Browse files- tiera-rerun-fixed-moonep/STATUS.txt +2 -2
- tiera-rerun-fixed-moonep/TIERA_RERUN_MANIFEST.txt +2 -2
- tiera-rerun-fixed-moonep/echo/a1/provenance.txt +29 -0
- tiera-rerun-fixed-moonep/hf_watcher.log +3 -0
- tiera-rerun-fixed-moonep/hf_watcher.pid +1 -1
- tiera-rerun-fixed-moonep/study.log +24 -0
- tiera-rerun-fixed-moonep/tiera_capacity.env +41 -0
tiera-rerun-fixed-moonep/STATUS.txt
CHANGED
|
@@ -1,7 +1,7 @@
|
|
| 1 |
study_root: /workspace/tiera-rerun-fixed-moonep
|
| 2 |
gbs: 16
|
| 3 |
-
updated: 2026-08-30T05:
|
| 4 |
-
current:
|
| 5 |
|
| 6 |
completed steps:
|
| 7 |
|
|
|
|
| 1 |
study_root: /workspace/tiera-rerun-fixed-moonep
|
| 2 |
gbs: 16
|
| 3 |
+
updated: 2026-08-30T05:49:16+00:00
|
| 4 |
+
current: echo__a1__gbs16
|
| 5 |
|
| 6 |
completed steps:
|
| 7 |
|
tiera-rerun-fixed-moonep/TIERA_RERUN_MANIFEST.txt
CHANGED
|
@@ -1,10 +1,10 @@
|
|
| 1 |
tier_a_rerun=1
|
| 2 |
-
generated=2026-08-
|
| 3 |
study_root=/workspace/tiera-rerun-fixed-moonep
|
| 4 |
gbs=16
|
| 5 |
micro_batch=2
|
| 6 |
ladder=G1 G2 G3 G4
|
| 7 |
-
parts=
|
| 8 |
moonep_megatron_branch=moonep_backend_async_grad_reduce
|
| 9 |
moonep_source_branch=megatron-ddp-symmetric-params
|
| 10 |
moonep_grad_fix_required=1
|
|
|
|
| 1 |
tier_a_rerun=1
|
| 2 |
+
generated=2026-08-30T05:49:16+00:00
|
| 3 |
study_root=/workspace/tiera-rerun-fixed-moonep
|
| 4 |
gbs=16
|
| 5 |
micro_batch=2
|
| 6 |
ladder=G1 G2 G3 G4
|
| 7 |
+
parts=timing memory trace
|
| 8 |
moonep_megatron_branch=moonep_backend_async_grad_reduce
|
| 9 |
moonep_source_branch=megatron-ddp-symmetric-params
|
| 10 |
moonep_grad_fix_required=1
|
tiera-rerun-fixed-moonep/echo/a1/provenance.txt
ADDED
|
@@ -0,0 +1,29 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
==============================================================
|
| 2 |
+
PROVENANCE: echo__a1__gbs16
|
| 3 |
+
==============================================================
|
| 4 |
+
captured_at : 2026-08-30T05:49:16+00:00
|
| 5 |
+
hostname : sp-aiinfra-interactive-aghosh-7jtjl-8xb200-moe-cg-0
|
| 6 |
+
kernel : 6.8.0-1043-nvidia
|
| 7 |
+
cwd : /workspace/blackwell_run_scripts
|
| 8 |
+
command : run_ladder.sh --phase a1 --gbs 16 --out /workspace/tiera-rerun-fixed-moonep/echo/a1 --nsys
|
| 9 |
+
|
| 10 |
+
---- GPU / driver --------------------------------------------
|
| 11 |
+
0, NVIDIA B200, 10.0, 183359 MiB, 580.95.05
|
| 12 |
+
1, NVIDIA B200, 10.0, 183359 MiB, 580.95.05
|
| 13 |
+
2, NVIDIA B200, 10.0, 183359 MiB, 580.95.05
|
| 14 |
+
3, NVIDIA B200, 10.0, 183359 MiB, 580.95.05
|
| 15 |
+
4, NVIDIA B200, 10.0, 183359 MiB, 580.95.05
|
| 16 |
+
5, NVIDIA B200, 10.0, 183359 MiB, 580.95.05
|
| 17 |
+
6, NVIDIA B200, 10.0, 183359 MiB, 580.95.05
|
| 18 |
+
7, NVIDIA B200, 10.0, 183359 MiB, 580.95.05
|
| 19 |
+
persistence/clocks:
|
| 20 |
+
0, 120 MHz, 1965 MHz, 1000.00 W, 27
|
| 21 |
+
1, 120 MHz, 1965 MHz, 1000.00 W, 32
|
| 22 |
+
2, 120 MHz, 1965 MHz, 1000.00 W, 32
|
| 23 |
+
3, 120 MHz, 1965 MHz, 1000.00 W, 27
|
| 24 |
+
4, 120 MHz, 1965 MHz, 1000.00 W, 29
|
| 25 |
+
5, 120 MHz, 1965 MHz, 1000.00 W, 31
|
| 26 |
+
6, 120 MHz, 1965 MHz, 1000.00 W, 32
|
| 27 |
+
7, 120 MHz, 1965 MHz, 1000.00 W, 28
|
| 28 |
+
|
| 29 |
+
NVLink topology:
|
tiera-rerun-fixed-moonep/hf_watcher.log
CHANGED
|
@@ -9,3 +9,6 @@ https://huggingface.co/MoE-CUDA-Graph-Analysis/tierb-b200-gbs8/tree/main
|
|
| 9 |
[2026-08-30T05:12:45+00:00] backup pass
|
| 10 |
uploaded: /workspace/tiera-rerun-fixed-moonep -> MoE-CUDA-Graph-Analysis/tierb-b200-gbs8/tiera-rerun-fixed-moonep
|
| 11 |
https://huggingface.co/MoE-CUDA-Graph-Analysis/tierb-b200-gbs8/tree/main
|
|
|
|
|
|
|
|
|
|
|
|
| 9 |
[2026-08-30T05:12:45+00:00] backup pass
|
| 10 |
uploaded: /workspace/tiera-rerun-fixed-moonep -> MoE-CUDA-Graph-Analysis/tierb-b200-gbs8/tiera-rerun-fixed-moonep
|
| 11 |
https://huggingface.co/MoE-CUDA-Graph-Analysis/tierb-b200-gbs8/tree/main
|
| 12 |
+
Already logged in to Hugging Face as: AbhishekGhosh1998
|
| 13 |
+
Watching /workspace/tiera-rerun-fixed-moonep -> HF every 600s (kill to stop)
|
| 14 |
+
[2026-08-30T05:49:17+00:00] backup pass
|
tiera-rerun-fixed-moonep/hf_watcher.pid
CHANGED
|
@@ -1 +1 @@
|
|
| 1 |
-
|
|
|
|
| 1 |
+
545120
|
tiera-rerun-fixed-moonep/study.log
CHANGED
|
@@ -370,3 +370,27 @@ ECHO note: echo_logged_pre_rank_metric_valid=0 and echo_logged_load_reduction_va
|
|
| 370 |
The branch's built-in fields mix source-rank and destination-rank axes; this tool reconstructs both globally.
|
| 371 |
no memory-dump runs found under /workspace/tiera-rerun-fixed-moonep
|
| 372 |
Already logged in to Hugging Face as: AbhishekGhosh1998
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 370 |
The branch's built-in fields mix source-rank and destination-rank axes; this tool reconstructs both globally.
|
| 371 |
no memory-dump runs found under /workspace/tiera-rerun-fixed-moonep
|
| 372 |
Already logged in to Hugging Face as: AbhishekGhosh1998
|
| 373 |
+
uploaded: /workspace/tiera-rerun-fixed-moonep -> MoE-CUDA-Graph-Analysis/tierb-b200-gbs8/tiera-rerun-fixed-moonep
|
| 374 |
+
https://huggingface.co/MoE-CUDA-Graph-Analysis/tierb-b200-gbs8/tree/main
|
| 375 |
+
|
| 376 |
+
=== STUDY COMPLETE
|
| 377 |
+
results : /workspace/tiera-rerun-fixed-moonep
|
| 378 |
+
status : /workspace/tiera-rerun-fixed-moonep/STATUS.txt
|
| 379 |
+
steps : /workspace/tiera-rerun-fixed-moonep/steps.tsv
|
| 380 |
+
hub : https://huggingface.co/datasets/MoE-CUDA-Graph-Analysis/tierb-b200-gbs8
|
| 381 |
+
|
| 382 |
+
status campaign phase gbs started ended result_dir
|
| 383 |
+
FAIL echo tierbprobe 16 2026-08-30T04:52:38+00:00 2026-08-30T05:06:38+00:00 /workspace/tiera-rerun-fixed-moonep/echo/tierbprobe
|
| 384 |
+
FAIL ultraep tierbprobe 16 2026-08-30T05:06:44+00:00 2026-08-30T05:20:40+00:00 /workspace/tiera-rerun-fixed-moonep/ultraep/tierbprobe
|
| 385 |
+
study_root: /workspace/tiera-rerun-fixed-moonep
|
| 386 |
+
gbs: 16
|
| 387 |
+
stages: 3
|
| 388 |
+
nsys: 1
|
| 389 |
+
hf_repo: MoE-CUDA-Graph-Analysis/tierb-b200-gbs8
|
| 390 |
+
hf_remote_root: tiera-rerun-fixed-moonep
|
| 391 |
+
started: 2026-08-30T05:49:16+00:00
|
| 392 |
+
driver_repo: /workspace/blackwell_run_scripts @ 9925dcc
|
| 393 |
+
|
| 394 |
+
=== STAGE 3 -- Tier A (the causal experiment)
|
| 395 |
+
|
| 396 |
+
=== STEP echo__a1__gbs16
|
tiera-rerun-fixed-moonep/tiera_capacity.env
ADDED
|
@@ -0,0 +1,41 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
# Tier A capacities -- MEASURED, not assumed.
|
| 2 |
+
#
|
| 3 |
+
# Derived from the stage-7 capacity probe (zero-drop validated) plus a direct
|
| 4 |
+
# memory gate on the tightest cell. See TIERA_RERUN.md section 2.
|
| 5 |
+
#
|
| 6 |
+
# Per-rung requirement, from the probe's validated zero-drop selection:
|
| 7 |
+
#
|
| 8 |
+
# rung echo ultraep max (used below)
|
| 9 |
+
# G1 1.859375 2.40625 2.40625
|
| 10 |
+
# G2 1.08203125 1.0625 1.08203125
|
| 11 |
+
# G3 1.072265625 1.0625 1.072265625
|
| 12 |
+
# G4 1.068359375 1.095703125 1.095703125
|
| 13 |
+
#
|
| 14 |
+
# A SINGLE global capacity is impossible here. The max is 2.40625, driven
|
| 15 |
+
# entirely by G1, and a direct gate showed:
|
| 16 |
+
#
|
| 17 |
+
# G4/echo @ 2.40625, CG on -> OOM (peak 182627 MiB vs 182630 ceiling)
|
| 18 |
+
# G1/echo @ 2.40625, CG on -> OK (peak 161863 MiB)
|
| 19 |
+
#
|
| 20 |
+
# so the campaign uses one capacity per rung. Within a rung every backend still
|
| 21 |
+
# shares one capacity, so the backend comparison stays valid; the cross-rung
|
| 22 |
+
# trend must be read alongside the capacity column.
|
| 23 |
+
#
|
| 24 |
+
# G1's requirement is imbalance, not shape: with
|
| 25 |
+
# --moe-router-force-load-balancing the probe measured G1 needing only 1.01 for
|
| 26 |
+
# both backends. The published Tier A ladder ran every rung at a hardcoded 1.25,
|
| 27 |
+
# so G1 dropped tokens on BOTH backends while G2-G4 did not.
|
| 28 |
+
TIERA_CAPACITY=2.40625
|
| 29 |
+
TIERA_CAPACITY_GLOBAL_FEASIBLE=0
|
| 30 |
+
TIERA_CAP_G1=2.40625
|
| 31 |
+
TIERA_CAP_G2=1.08203125
|
| 32 |
+
TIERA_CAP_G3=1.072265625
|
| 33 |
+
TIERA_CAP_G4=1.095703125
|
| 34 |
+
TIERA_PROBE_G1_ECHO=1.859375
|
| 35 |
+
TIERA_PROBE_G1_ULTRAEP=2.40625
|
| 36 |
+
TIERA_PROBE_G2_ECHO=1.08203125
|
| 37 |
+
TIERA_PROBE_G2_ULTRAEP=1.0625
|
| 38 |
+
TIERA_PROBE_G3_ECHO=1.072265625
|
| 39 |
+
TIERA_PROBE_G3_ULTRAEP=1.0625
|
| 40 |
+
TIERA_PROBE_G4_ECHO=1.068359375
|
| 41 |
+
TIERA_PROBE_G4_ULTRAEP=1.095703125
|