AbhishekGhosh1998 commited on
Commit
e4f9ea5
·
verified ·
1 Parent(s): 277ce5b

granularity results tiera-rerun-fixed-moonep @ 2026-08-30T05:49:17Z from sp-aiinfra-interactive-aghosh-7jtjl-8xb200-moe-cg-0

Browse files
tiera-rerun-fixed-moonep/STATUS.txt CHANGED
@@ -1,7 +1,7 @@
1
  study_root: /workspace/tiera-rerun-fixed-moonep
2
  gbs: 16
3
- updated: 2026-08-30T05:20:43+00:00
4
- current: idle
5
 
6
  completed steps:
7
 
 
1
  study_root: /workspace/tiera-rerun-fixed-moonep
2
  gbs: 16
3
+ updated: 2026-08-30T05:49:16+00:00
4
+ current: echo__a1__gbs16
5
 
6
  completed steps:
7
 
tiera-rerun-fixed-moonep/TIERA_RERUN_MANIFEST.txt CHANGED
@@ -1,10 +1,10 @@
1
  tier_a_rerun=1
2
- generated=2026-08-30T04:52:38+00:00
3
  study_root=/workspace/tiera-rerun-fixed-moonep
4
  gbs=16
5
  micro_batch=2
6
  ladder=G1 G2 G3 G4
7
- parts=probe timing memory trace
8
  moonep_megatron_branch=moonep_backend_async_grad_reduce
9
  moonep_source_branch=megatron-ddp-symmetric-params
10
  moonep_grad_fix_required=1
 
1
  tier_a_rerun=1
2
+ generated=2026-08-30T05:49:16+00:00
3
  study_root=/workspace/tiera-rerun-fixed-moonep
4
  gbs=16
5
  micro_batch=2
6
  ladder=G1 G2 G3 G4
7
+ parts=timing memory trace
8
  moonep_megatron_branch=moonep_backend_async_grad_reduce
9
  moonep_source_branch=megatron-ddp-symmetric-params
10
  moonep_grad_fix_required=1
tiera-rerun-fixed-moonep/echo/a1/provenance.txt ADDED
@@ -0,0 +1,29 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ==============================================================
2
+ PROVENANCE: echo__a1__gbs16
3
+ ==============================================================
4
+ captured_at : 2026-08-30T05:49:16+00:00
5
+ hostname : sp-aiinfra-interactive-aghosh-7jtjl-8xb200-moe-cg-0
6
+ kernel : 6.8.0-1043-nvidia
7
+ cwd : /workspace/blackwell_run_scripts
8
+ command : run_ladder.sh --phase a1 --gbs 16 --out /workspace/tiera-rerun-fixed-moonep/echo/a1 --nsys
9
+
10
+ ---- GPU / driver --------------------------------------------
11
+ 0, NVIDIA B200, 10.0, 183359 MiB, 580.95.05
12
+ 1, NVIDIA B200, 10.0, 183359 MiB, 580.95.05
13
+ 2, NVIDIA B200, 10.0, 183359 MiB, 580.95.05
14
+ 3, NVIDIA B200, 10.0, 183359 MiB, 580.95.05
15
+ 4, NVIDIA B200, 10.0, 183359 MiB, 580.95.05
16
+ 5, NVIDIA B200, 10.0, 183359 MiB, 580.95.05
17
+ 6, NVIDIA B200, 10.0, 183359 MiB, 580.95.05
18
+ 7, NVIDIA B200, 10.0, 183359 MiB, 580.95.05
19
+ persistence/clocks:
20
+ 0, 120 MHz, 1965 MHz, 1000.00 W, 27
21
+ 1, 120 MHz, 1965 MHz, 1000.00 W, 32
22
+ 2, 120 MHz, 1965 MHz, 1000.00 W, 32
23
+ 3, 120 MHz, 1965 MHz, 1000.00 W, 27
24
+ 4, 120 MHz, 1965 MHz, 1000.00 W, 29
25
+ 5, 120 MHz, 1965 MHz, 1000.00 W, 31
26
+ 6, 120 MHz, 1965 MHz, 1000.00 W, 32
27
+ 7, 120 MHz, 1965 MHz, 1000.00 W, 28
28
+
29
+ NVLink topology:
tiera-rerun-fixed-moonep/hf_watcher.log CHANGED
@@ -9,3 +9,6 @@ https://huggingface.co/MoE-CUDA-Graph-Analysis/tierb-b200-gbs8/tree/main
9
  [2026-08-30T05:12:45+00:00] backup pass
10
  uploaded: /workspace/tiera-rerun-fixed-moonep -> MoE-CUDA-Graph-Analysis/tierb-b200-gbs8/tiera-rerun-fixed-moonep
11
  https://huggingface.co/MoE-CUDA-Graph-Analysis/tierb-b200-gbs8/tree/main
 
 
 
 
9
  [2026-08-30T05:12:45+00:00] backup pass
10
  uploaded: /workspace/tiera-rerun-fixed-moonep -> MoE-CUDA-Graph-Analysis/tierb-b200-gbs8/tiera-rerun-fixed-moonep
11
  https://huggingface.co/MoE-CUDA-Graph-Analysis/tierb-b200-gbs8/tree/main
12
+ Already logged in to Hugging Face as: AbhishekGhosh1998
13
+ Watching /workspace/tiera-rerun-fixed-moonep -> HF every 600s (kill to stop)
14
+ [2026-08-30T05:49:17+00:00] backup pass
tiera-rerun-fixed-moonep/hf_watcher.pid CHANGED
@@ -1 +1 @@
1
- 439568
 
1
+ 545120
tiera-rerun-fixed-moonep/study.log CHANGED
@@ -370,3 +370,27 @@ ECHO note: echo_logged_pre_rank_metric_valid=0 and echo_logged_load_reduction_va
370
  The branch's built-in fields mix source-rank and destination-rank axes; this tool reconstructs both globally.
371
  no memory-dump runs found under /workspace/tiera-rerun-fixed-moonep
372
  Already logged in to Hugging Face as: AbhishekGhosh1998
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
370
  The branch's built-in fields mix source-rank and destination-rank axes; this tool reconstructs both globally.
371
  no memory-dump runs found under /workspace/tiera-rerun-fixed-moonep
372
  Already logged in to Hugging Face as: AbhishekGhosh1998
373
+ uploaded: /workspace/tiera-rerun-fixed-moonep -> MoE-CUDA-Graph-Analysis/tierb-b200-gbs8/tiera-rerun-fixed-moonep
374
+ https://huggingface.co/MoE-CUDA-Graph-Analysis/tierb-b200-gbs8/tree/main
375
+
376
+ === STUDY COMPLETE
377
+ results : /workspace/tiera-rerun-fixed-moonep
378
+ status : /workspace/tiera-rerun-fixed-moonep/STATUS.txt
379
+ steps : /workspace/tiera-rerun-fixed-moonep/steps.tsv
380
+ hub : https://huggingface.co/datasets/MoE-CUDA-Graph-Analysis/tierb-b200-gbs8
381
+
382
+ status campaign phase gbs started ended result_dir
383
+ FAIL echo tierbprobe 16 2026-08-30T04:52:38+00:00 2026-08-30T05:06:38+00:00 /workspace/tiera-rerun-fixed-moonep/echo/tierbprobe
384
+ FAIL ultraep tierbprobe 16 2026-08-30T05:06:44+00:00 2026-08-30T05:20:40+00:00 /workspace/tiera-rerun-fixed-moonep/ultraep/tierbprobe
385
+ study_root: /workspace/tiera-rerun-fixed-moonep
386
+ gbs: 16
387
+ stages: 3
388
+ nsys: 1
389
+ hf_repo: MoE-CUDA-Graph-Analysis/tierb-b200-gbs8
390
+ hf_remote_root: tiera-rerun-fixed-moonep
391
+ started: 2026-08-30T05:49:16+00:00
392
+ driver_repo: /workspace/blackwell_run_scripts @ 9925dcc
393
+
394
+ === STAGE 3 -- Tier A (the causal experiment)
395
+
396
+ === STEP echo__a1__gbs16
tiera-rerun-fixed-moonep/tiera_capacity.env ADDED
@@ -0,0 +1,41 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # Tier A capacities -- MEASURED, not assumed.
2
+ #
3
+ # Derived from the stage-7 capacity probe (zero-drop validated) plus a direct
4
+ # memory gate on the tightest cell. See TIERA_RERUN.md section 2.
5
+ #
6
+ # Per-rung requirement, from the probe's validated zero-drop selection:
7
+ #
8
+ # rung echo ultraep max (used below)
9
+ # G1 1.859375 2.40625 2.40625
10
+ # G2 1.08203125 1.0625 1.08203125
11
+ # G3 1.072265625 1.0625 1.072265625
12
+ # G4 1.068359375 1.095703125 1.095703125
13
+ #
14
+ # A SINGLE global capacity is impossible here. The max is 2.40625, driven
15
+ # entirely by G1, and a direct gate showed:
16
+ #
17
+ # G4/echo @ 2.40625, CG on -> OOM (peak 182627 MiB vs 182630 ceiling)
18
+ # G1/echo @ 2.40625, CG on -> OK (peak 161863 MiB)
19
+ #
20
+ # so the campaign uses one capacity per rung. Within a rung every backend still
21
+ # shares one capacity, so the backend comparison stays valid; the cross-rung
22
+ # trend must be read alongside the capacity column.
23
+ #
24
+ # G1's requirement is imbalance, not shape: with
25
+ # --moe-router-force-load-balancing the probe measured G1 needing only 1.01 for
26
+ # both backends. The published Tier A ladder ran every rung at a hardcoded 1.25,
27
+ # so G1 dropped tokens on BOTH backends while G2-G4 did not.
28
+ TIERA_CAPACITY=2.40625
29
+ TIERA_CAPACITY_GLOBAL_FEASIBLE=0
30
+ TIERA_CAP_G1=2.40625
31
+ TIERA_CAP_G2=1.08203125
32
+ TIERA_CAP_G3=1.072265625
33
+ TIERA_CAP_G4=1.095703125
34
+ TIERA_PROBE_G1_ECHO=1.859375
35
+ TIERA_PROBE_G1_ULTRAEP=2.40625
36
+ TIERA_PROBE_G2_ECHO=1.08203125
37
+ TIERA_PROBE_G2_ULTRAEP=1.0625
38
+ TIERA_PROBE_G3_ECHO=1.072265625
39
+ TIERA_PROBE_G3_ULTRAEP=1.0625
40
+ TIERA_PROBE_G4_ECHO=1.068359375
41
+ TIERA_PROBE_G4_ULTRAEP=1.095703125