granularity results tiera-rerun-fixed-moonep @ 2026-08-30T04:52:39Z from sp-aiinfra-interactive-aghosh-7jtjl-8xb200-moe-cg-0
Browse files- tiera-rerun-fixed-moonep/STATUS.txt +9 -0
- tiera-rerun-fixed-moonep/TIERA_RERUN_MANIFEST.txt +11 -0
- tiera-rerun-fixed-moonep/echo/tierbprobe/provenance.txt +29 -0
- tiera-rerun-fixed-moonep/hf_watcher.log +3 -0
- tiera-rerun-fixed-moonep/hf_watcher.pid +1 -0
- tiera-rerun-fixed-moonep/steps.tsv +1 -0
- tiera-rerun-fixed-moonep/study.log +12 -0
tiera-rerun-fixed-moonep/STATUS.txt
ADDED
|
@@ -0,0 +1,9 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
study_root: /workspace/tiera-rerun-fixed-moonep
|
| 2 |
+
gbs: 16
|
| 3 |
+
updated: 2026-08-30T04:52:38+00:00
|
| 4 |
+
current: echo__tierbprobe__gbs16
|
| 5 |
+
|
| 6 |
+
completed steps:
|
| 7 |
+
|
| 8 |
+
failed steps:
|
| 9 |
+
(none)
|
tiera-rerun-fixed-moonep/TIERA_RERUN_MANIFEST.txt
ADDED
|
@@ -0,0 +1,11 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
tier_a_rerun=1
|
| 2 |
+
generated=2026-08-30T04:52:38+00:00
|
| 3 |
+
study_root=/workspace/tiera-rerun-fixed-moonep
|
| 4 |
+
gbs=16
|
| 5 |
+
micro_batch=2
|
| 6 |
+
ladder=G1 G2 G3 G4
|
| 7 |
+
parts=probe timing memory trace
|
| 8 |
+
moonep_megatron_branch=moonep_backend_async_grad_reduce
|
| 9 |
+
moonep_source_branch=megatron-ddp-symmetric-params
|
| 10 |
+
moonep_grad_fix_required=1
|
| 11 |
+
capacity_policy=global-then-per-rung
|
tiera-rerun-fixed-moonep/echo/tierbprobe/provenance.txt
ADDED
|
@@ -0,0 +1,29 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
==============================================================
|
| 2 |
+
PROVENANCE: echo__tierbprobe__gbs16
|
| 3 |
+
==============================================================
|
| 4 |
+
captured_at : 2026-08-30T04:52:38+00:00
|
| 5 |
+
hostname : sp-aiinfra-interactive-aghosh-7jtjl-8xb200-moe-cg-0
|
| 6 |
+
kernel : 6.8.0-1043-nvidia
|
| 7 |
+
cwd : /workspace/blackwell_run_scripts
|
| 8 |
+
command : run_ladder.sh --phase tierbprobe --gbs 16 --out /workspace/tiera-rerun-fixed-moonep/echo/tierbprobe --nsys
|
| 9 |
+
|
| 10 |
+
---- GPU / driver --------------------------------------------
|
| 11 |
+
0, NVIDIA B200, 10.0, 183359 MiB, 580.95.05
|
| 12 |
+
1, NVIDIA B200, 10.0, 183359 MiB, 580.95.05
|
| 13 |
+
2, NVIDIA B200, 10.0, 183359 MiB, 580.95.05
|
| 14 |
+
3, NVIDIA B200, 10.0, 183359 MiB, 580.95.05
|
| 15 |
+
4, NVIDIA B200, 10.0, 183359 MiB, 580.95.05
|
| 16 |
+
5, NVIDIA B200, 10.0, 183359 MiB, 580.95.05
|
| 17 |
+
6, NVIDIA B200, 10.0, 183359 MiB, 580.95.05
|
| 18 |
+
7, NVIDIA B200, 10.0, 183359 MiB, 580.95.05
|
| 19 |
+
persistence/clocks:
|
| 20 |
+
0, 120 MHz, 1965 MHz, 1000.00 W, 27
|
| 21 |
+
1, 120 MHz, 1965 MHz, 1000.00 W, 32
|
| 22 |
+
2, 120 MHz, 1965 MHz, 1000.00 W, 32
|
| 23 |
+
3, 120 MHz, 1965 MHz, 1000.00 W, 27
|
| 24 |
+
4, 120 MHz, 1965 MHz, 1000.00 W, 29
|
| 25 |
+
5, 120 MHz, 1965 MHz, 1000.00 W, 31
|
| 26 |
+
6, 120 MHz, 1965 MHz, 1000.00 W, 32
|
| 27 |
+
7, 120 MHz, 1965 MHz, 1000.00 W, 28
|
| 28 |
+
|
| 29 |
+
NVLink topology:
|
tiera-rerun-fixed-moonep/hf_watcher.log
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
Already logged in to Hugging Face as: AbhishekGhosh1998
|
| 2 |
+
Watching /workspace/tiera-rerun-fixed-moonep -> HF every 600s (kill to stop)
|
| 3 |
+
[2026-08-30T04:52:39+00:00] backup pass
|
tiera-rerun-fixed-moonep/hf_watcher.pid
ADDED
|
@@ -0,0 +1 @@
|
|
|
|
|
|
|
| 1 |
+
439568
|
tiera-rerun-fixed-moonep/steps.tsv
ADDED
|
@@ -0,0 +1 @@
|
|
|
|
|
|
|
| 1 |
+
status campaign phase gbs started ended result_dir
|
tiera-rerun-fixed-moonep/study.log
ADDED
|
@@ -0,0 +1,12 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
study_root: /workspace/tiera-rerun-fixed-moonep
|
| 2 |
+
gbs: 16
|
| 3 |
+
stages: 7
|
| 4 |
+
nsys: 1
|
| 5 |
+
hf_repo: MoE-CUDA-Graph-Analysis/tierb-b200-gbs8
|
| 6 |
+
hf_remote_root: tiera-rerun-fixed-moonep
|
| 7 |
+
started: 2026-08-30T04:52:38+00:00
|
| 8 |
+
driver_repo: /workspace/blackwell_run_scripts @ 774e9e2
|
| 9 |
+
|
| 10 |
+
=== STAGE 7 -- Tier B probe and zero-drop capacity selection
|
| 11 |
+
|
| 12 |
+
=== STEP echo__tierbprobe__gbs16
|