AbhishekGhosh1998 commited on
Commit
2562d81
·
verified ·
1 Parent(s): f07f871

granularity results tierb-fixed-moonep-full @ 2026-08-30T00:10:12Z from sp-aiinfra-interactive-aghosh-7jtjl-8xb200-moe-cg-0

Browse files
tierb-fixed-moonep-full/STATUS.txt CHANGED
@@ -1,10 +1,11 @@
1
  study_root: /workspace/tierb-fixed-moonep-full
2
  gbs: 8
3
- updated: 2026-08-29T23:59:59+00:00
4
- current: moonep__tierbmem__gbs8
5
 
6
  completed steps:
7
  moonep__tierb__gbs8
 
8
  moonep__tierbprobe__gbs8
9
 
10
  failed steps:
 
1
  study_root: /workspace/tierb-fixed-moonep-full
2
  gbs: 8
3
+ updated: 2026-08-30T00:10:08+00:00
4
+ current: idle
5
 
6
  completed steps:
7
  moonep__tierb__gbs8
8
+ moonep__tierbmem__gbs8
9
  moonep__tierbprobe__gbs8
10
 
11
  failed steps:
tierb-fixed-moonep-full/collected_all.tsv CHANGED
@@ -11,6 +11,14 @@ moonep/tierb/tierb_gbs8/QWEN3_235B__moonep__matched__seq8192__gbs8__capnone__asy
11
  moonep/tierb/tierb_gbs8/QWEN3_30B__moonep__matched__seq4096__gbs8__capnone__async/rep1 rep1 tierb 0 1 QWEN3_30B moonep matched 16 128 8 768 16 4718592 48 4096 8 1 32768 2048 none none none 0 1 none 0 moonep_backend_async_grad_reduce 7bc235bf 0 48 aux_loss 1e-3 0 0 1000000 0 OK 12 361.53 358.70 4.45 246.5 116703 68288
12
  moonep/tierb/tierb_gbs8/QWEN3_30B__moonep__matched__seq4096__gbs8__capnone__async/rep2 rep2 tierb 0 1 QWEN3_30B moonep matched 16 128 8 768 16 4718592 48 4096 8 1 32768 2048 none none none 0 1 none 0 moonep_backend_async_grad_reduce 7bc235bf 0 48 aux_loss 1e-3 0 0 1000000 0 OK 12 360.81 359.20 1.23 247.0 116703 68288
13
  moonep/tierb/tierb_gbs8/QWEN3_30B__moonep__matched__seq4096__gbs8__capnone__async/rep3 rep3 tierb 0 1 QWEN3_30B moonep matched 16 128 8 768 16 4718592 48 4096 8 1 32768 2048 none none none 0 1 none 0 moonep_backend_async_grad_reduce 7bc235bf 0 48 aux_loss 1e-3 0 0 1000000 0 OK 12 359.94 357.60 1.93 247.6 116703 68288
 
 
 
 
 
 
 
 
14
  moonep/tierbprobe/tierbprobe_gbs8/DSV2_LITE_L24__moonep__matched__seq8192__gbs8__capnone__async/rep1 rep1 tierbprobe 0 1 DSV2_LITE_L24 moonep matched 8 64 6 1408 8 8650752 24 8192 8 1 49152 6144 none none none 0 0 none 0 moonep_backend_async_grad_reduce_token_routing_analysis 22fd6865 1 23 seq_aux_loss 1e-3 1 0 10000 0 OK 12 673.29 669.50 3.07 189.5 125157 95432
15
  moonep/tierbprobe/tierbprobe_gbs8/DSV2_LITE_L24__moonep__matched__seq8192__gbs8__capnone__forcebal__async/rep1 rep1 tierbprobe 0 1 DSV2_LITE_L24 moonep matched 8 64 6 1408 8 8650752 24 8192 8 1 49152 6144 none none none 1 0 none 0 moonep_backend_async_grad_reduce_token_routing_analysis 22fd6865 1 23 seq_aux_loss 1e-3 1 0 10000 0 OK 12 674.59 672.10 3.68 189.1 125633 95432
16
  moonep/tierbprobe/tierbprobe_gbs8/QWEN2_57B__moonep__matched__seq4096__gbs8__capnone__async/rep1 rep1 tierbprobe 0 1 QWEN2_57B moonep matched 8 64 8 2560 8 27525120 10 4096 8 1 32768 4096 none none none 0 0 none 0 moonep_backend_async_grad_reduce_token_routing_analysis 22fd6865 0 10 aux_loss 1e-2 1 0 1000000 1 OK 12 238.47 236.30 2.03 517.8 95855 62924
 
11
  moonep/tierb/tierb_gbs8/QWEN3_30B__moonep__matched__seq4096__gbs8__capnone__async/rep1 rep1 tierb 0 1 QWEN3_30B moonep matched 16 128 8 768 16 4718592 48 4096 8 1 32768 2048 none none none 0 1 none 0 moonep_backend_async_grad_reduce 7bc235bf 0 48 aux_loss 1e-3 0 0 1000000 0 OK 12 361.53 358.70 4.45 246.5 116703 68288
12
  moonep/tierb/tierb_gbs8/QWEN3_30B__moonep__matched__seq4096__gbs8__capnone__async/rep2 rep2 tierb 0 1 QWEN3_30B moonep matched 16 128 8 768 16 4718592 48 4096 8 1 32768 2048 none none none 0 1 none 0 moonep_backend_async_grad_reduce 7bc235bf 0 48 aux_loss 1e-3 0 0 1000000 0 OK 12 360.81 359.20 1.23 247.0 116703 68288
13
  moonep/tierb/tierb_gbs8/QWEN3_30B__moonep__matched__seq4096__gbs8__capnone__async/rep3 rep3 tierb 0 1 QWEN3_30B moonep matched 16 128 8 768 16 4718592 48 4096 8 1 32768 2048 none none none 0 1 none 0 moonep_backend_async_grad_reduce 7bc235bf 0 48 aux_loss 1e-3 0 0 1000000 0 OK 12 359.94 357.60 1.93 247.6 116703 68288
14
+ moonep/tierbmem/tierbmem_gbs8/DSV2_LITE_L24__moonep__matched__seq8192__gbs8__capnone__async__memiteration__it6/rep1 rep1 tierbmem 0 1 DSV2_LITE_L24 moonep matched 8 64 6 1408 8 8650752 24 8192 8 1 49152 6144 none none none 0 1 iteration /workspace/tierb-fixed-moonep-full/moonep/tierbmem/tierbmem_gbs8/DSV2_LITE_L24__moonep__matched__seq8192__gbs8__capnone__async__memiteration__it6/rep1/memory_snapshots/snapshot.pickle 0 moonep_backend_async_grad_reduce_better_memory_dump_support 51c387c7 1 23 seq_aux_loss 1e-3 1 0 10000 0 OK 6 622.80 622.80 204.9 133311 95432
15
+ moonep/tierbmem/tierbmem_gbs8/DSV2_LITE_L24__moonep__matched__seq8192__gbs8__capnone__nocg__async__memiteration__it5/rep1 rep1 tierbmem 0 1 DSV2_LITE_L24 moonep matched 8 64 6 1408 8 8650752 24 8192 8 1 49152 6144 none none none 0 0 iteration /workspace/tierb-fixed-moonep-full/moonep/tierbmem/tierbmem_gbs8/DSV2_LITE_L24__moonep__matched__seq8192__gbs8__capnone__nocg__async__memiteration__it5/rep1/memory_snapshots/snapshot.pickle 0 moonep_backend_async_grad_reduce_better_memory_dump_support 51c387c7 1 23 seq_aux_loss 1e-3 1 0 10000 0 FAIL 5 125157 95432
16
+ moonep/tierbmem/tierbmem_gbs8/QWEN2_57B__moonep__matched__seq4096__gbs8__capnone__async__memiteration__it6/rep1 rep1 tierbmem 0 1 QWEN2_57B moonep matched 8 64 8 2560 8 27525120 10 4096 8 1 32768 4096 none none none 0 1 iteration /workspace/tierb-fixed-moonep-full/moonep/tierbmem/tierbmem_gbs8/QWEN2_57B__moonep__matched__seq4096__gbs8__capnone__async__memiteration__it6/rep1/memory_snapshots/snapshot.pickle 0 moonep_backend_async_grad_reduce_better_memory_dump_support 51c387c7 0 10 aux_loss 1e-2 1 0 1000000 1 OK 6 196.60 196.60 628.0 96737 62900
17
+ moonep/tierbmem/tierbmem_gbs8/QWEN2_57B__moonep__matched__seq4096__gbs8__capnone__nocg__async__memiteration__it5/rep1 rep1 tierbmem 0 1 QWEN2_57B moonep matched 8 64 8 2560 8 27525120 10 4096 8 1 32768 4096 none none none 0 0 iteration /workspace/tierb-fixed-moonep-full/moonep/tierbmem/tierbmem_gbs8/QWEN2_57B__moonep__matched__seq4096__gbs8__capnone__nocg__async__memiteration__it5/rep1/memory_snapshots/snapshot.pickle 0 moonep_backend_async_grad_reduce_better_memory_dump_support 51c387c7 0 10 aux_loss 1e-2 1 0 1000000 1 FAIL 5 95855 62924
18
+ moonep/tierbmem/tierbmem_gbs8/QWEN3_235B__moonep__matched__seq8192__gbs8__capnone__async__memiteration__it6/rep1 rep1 tierbmem 0 1 QWEN3_235B moonep matched 16 128 8 1536 16 18874368 12 8192 8 1 65536 4096 none none none 0 1 iteration /workspace/tierb-fixed-moonep-full/moonep/tierbmem/tierbmem_gbs8/QWEN3_235B__moonep__matched__seq8192__gbs8__capnone__async__memiteration__it6/rep1/memory_snapshots/snapshot.pickle 0 moonep_backend_async_grad_reduce_better_memory_dump_support 51c387c7 0 12 aux_loss 1e-3 0 0 1000000 0 OK 6 341.70 341.70 529.3 120303 68548
19
+ moonep/tierbmem/tierbmem_gbs8/QWEN3_235B__moonep__matched__seq8192__gbs8__capnone__nocg__async__memiteration__it5/rep1 rep1 tierbmem 0 1 QWEN3_235B moonep matched 16 128 8 1536 16 18874368 12 8192 8 1 65536 4096 none none none 0 0 iteration /workspace/tierb-fixed-moonep-full/moonep/tierbmem/tierbmem_gbs8/QWEN3_235B__moonep__matched__seq8192__gbs8__capnone__nocg__async__memiteration__it5/rep1/memory_snapshots/snapshot.pickle 0 moonep_backend_async_grad_reduce_better_memory_dump_support 51c387c7 0 12 aux_loss 1e-3 0 0 1000000 0 FAIL 5 120131 68476
20
+ moonep/tierbmem/tierbmem_gbs8/QWEN3_30B__moonep__matched__seq4096__gbs8__capnone__async__memiteration__it6/rep1 rep1 tierbmem 0 1 QWEN3_30B moonep matched 16 128 8 768 16 4718592 48 4096 8 1 32768 2048 none none none 0 1 iteration /workspace/tierb-fixed-moonep-full/moonep/tierbmem/tierbmem_gbs8/QWEN3_30B__moonep__matched__seq4096__gbs8__capnone__async__memiteration__it6/rep1/memory_snapshots/snapshot.pickle 0 moonep_backend_async_grad_reduce_better_memory_dump_support 51c387c7 0 48 aux_loss 1e-3 0 0 1000000 0 OK 6 435.30 435.30 204.7 116703 68288
21
+ moonep/tierbmem/tierbmem_gbs8/QWEN3_30B__moonep__matched__seq4096__gbs8__capnone__nocg__async__memiteration__it5/rep1 rep1 tierbmem 0 1 QWEN3_30B moonep matched 16 128 8 768 16 4718592 48 4096 8 1 32768 2048 none none none 0 0 iteration /workspace/tierb-fixed-moonep-full/moonep/tierbmem/tierbmem_gbs8/QWEN3_30B__moonep__matched__seq4096__gbs8__capnone__nocg__async__memiteration__it5/rep1/memory_snapshots/snapshot.pickle 0 moonep_backend_async_grad_reduce_better_memory_dump_support 51c387c7 0 48 aux_loss 1e-3 0 0 1000000 0 FAIL 5 116445 68280
22
  moonep/tierbprobe/tierbprobe_gbs8/DSV2_LITE_L24__moonep__matched__seq8192__gbs8__capnone__async/rep1 rep1 tierbprobe 0 1 DSV2_LITE_L24 moonep matched 8 64 6 1408 8 8650752 24 8192 8 1 49152 6144 none none none 0 0 none 0 moonep_backend_async_grad_reduce_token_routing_analysis 22fd6865 1 23 seq_aux_loss 1e-3 1 0 10000 0 OK 12 673.29 669.50 3.07 189.5 125157 95432
23
  moonep/tierbprobe/tierbprobe_gbs8/DSV2_LITE_L24__moonep__matched__seq8192__gbs8__capnone__forcebal__async/rep1 rep1 tierbprobe 0 1 DSV2_LITE_L24 moonep matched 8 64 6 1408 8 8650752 24 8192 8 1 49152 6144 none none none 1 0 none 0 moonep_backend_async_grad_reduce_token_routing_analysis 22fd6865 1 23 seq_aux_loss 1e-3 1 0 10000 0 OK 12 674.59 672.10 3.68 189.1 125633 95432
24
  moonep/tierbprobe/tierbprobe_gbs8/QWEN2_57B__moonep__matched__seq4096__gbs8__capnone__async/rep1 rep1 tierbprobe 0 1 QWEN2_57B moonep matched 8 64 8 2560 8 27525120 10 4096 8 1 32768 4096 none none none 0 0 none 0 moonep_backend_async_grad_reduce_token_routing_analysis 22fd6865 0 10 aux_loss 1e-2 1 0 1000000 1 OK 12 238.47 236.30 2.03 517.8 95855 62924
tierb-fixed-moonep-full/hf_watcher.log CHANGED
@@ -20,3 +20,5 @@ Watching /workspace/tierb-fixed-moonep-full -> HF every 600s (kill to stop)
20
  uploaded: /workspace/tierb-fixed-moonep-full -> MoE-CUDA-Graph-Analysis/tierb-b200-gbs8/tierb-fixed-moonep-full
21
  https://huggingface.co/MoE-CUDA-Graph-Analysis/tierb-b200-gbs8/tree/main
22
  [2026-08-30T00:10:01+00:00] backup pass
 
 
 
20
  uploaded: /workspace/tierb-fixed-moonep-full -> MoE-CUDA-Graph-Analysis/tierb-b200-gbs8/tierb-fixed-moonep-full
21
  https://huggingface.co/MoE-CUDA-Graph-Analysis/tierb-b200-gbs8/tree/main
22
  [2026-08-30T00:10:01+00:00] backup pass
23
+ uploaded: /workspace/tierb-fixed-moonep-full -> MoE-CUDA-Graph-Analysis/tierb-b200-gbs8/tierb-fixed-moonep-full
24
+ https://huggingface.co/MoE-CUDA-Graph-Analysis/tierb-b200-gbs8/tree/main
tierb-fixed-moonep-full/memory_summary_all.json ADDED
@@ -0,0 +1,250 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "runs": [
3
+ {
4
+ "run_dir": "moonep/tierbmem/tierbmem_gbs8/DSV2_LITE_L24__moonep__matched__seq8192__gbs8__capnone__async__memiteration__it6/rep1",
5
+ "config": "DSV2_LITE_L24",
6
+ "backend": "moonep",
7
+ "num_layers": "24",
8
+ "moe_layers": "23",
9
+ "seq_length": "8192",
10
+ "global_batch_size": "8",
11
+ "cuda_graph": "1",
12
+ "capacity_mode": "none",
13
+ "capacity": "none",
14
+ "memory_dump_mode": "iteration",
15
+ "branch": "moonep_backend_async_grad_reduce_better_memory_dump_support",
16
+ "commit": "51c387c7",
17
+ "status": "OK",
18
+ "max_allocated_gib": 91.17775774002075,
19
+ "max_reserved_gib": 93.1953125,
20
+ "fragmentation_gib": 2.017554759979248,
21
+ "theoretical_weight_optim_gib": 95.94234375,
22
+ "peak_sampled_gib": 130.1865234375,
23
+ "snapshot_pickle_count": 48,
24
+ "snapshot_json_count": 48,
25
+ "snapshot_expected_each": 48,
26
+ "snapshot_complete": 1,
27
+ "effective_peak_gib": 130.1865234375
28
+ },
29
+ {
30
+ "run_dir": "moonep/tierbmem/tierbmem_gbs8/DSV2_LITE_L24__moonep__matched__seq8192__gbs8__capnone__nocg__async__memiteration__it5/rep1",
31
+ "config": "DSV2_LITE_L24",
32
+ "backend": "moonep",
33
+ "num_layers": "24",
34
+ "moe_layers": "23",
35
+ "seq_length": "8192",
36
+ "global_batch_size": "8",
37
+ "cuda_graph": "0",
38
+ "capacity_mode": "none",
39
+ "capacity": "none",
40
+ "memory_dump_mode": "iteration",
41
+ "branch": "moonep_backend_async_grad_reduce_better_memory_dump_support",
42
+ "commit": "51c387c7",
43
+ "status": "OK",
44
+ "max_allocated_gib": 91.17775774002075,
45
+ "max_reserved_gib": 93.1953125,
46
+ "fragmentation_gib": 2.017554759979248,
47
+ "theoretical_weight_optim_gib": 95.94234375,
48
+ "peak_sampled_gib": 122.2236328125,
49
+ "snapshot_pickle_count": 40,
50
+ "snapshot_json_count": 40,
51
+ "snapshot_expected_each": 40,
52
+ "snapshot_complete": 1,
53
+ "effective_peak_gib": 122.2236328125
54
+ },
55
+ {
56
+ "run_dir": "moonep/tierbmem/tierbmem_gbs8/QWEN2_57B__moonep__matched__seq4096__gbs8__capnone__async__memiteration__it6/rep1",
57
+ "config": "QWEN2_57B",
58
+ "backend": "moonep",
59
+ "num_layers": "10",
60
+ "moe_layers": "10",
61
+ "seq_length": "4096",
62
+ "global_batch_size": "8",
63
+ "cuda_graph": "1",
64
+ "capacity_mode": "none",
65
+ "capacity": "none",
66
+ "memory_dump_mode": "iteration",
67
+ "branch": "moonep_backend_async_grad_reduce_better_memory_dump_support",
68
+ "commit": "51c387c7",
69
+ "status": "OK",
70
+ "max_allocated_gib": 59.11568737030029,
71
+ "max_reserved_gib": 61.42578125,
72
+ "fragmentation_gib": 2.310093879699707,
73
+ "theoretical_weight_optim_gib": 142.99818359375,
74
+ "peak_sampled_gib": 94.4697265625,
75
+ "snapshot_pickle_count": 48,
76
+ "snapshot_json_count": 48,
77
+ "snapshot_expected_each": 48,
78
+ "snapshot_complete": 1,
79
+ "effective_peak_gib": 94.4697265625
80
+ },
81
+ {
82
+ "run_dir": "moonep/tierbmem/tierbmem_gbs8/QWEN2_57B__moonep__matched__seq4096__gbs8__capnone__nocg__async__memiteration__it5/rep1",
83
+ "config": "QWEN2_57B",
84
+ "backend": "moonep",
85
+ "num_layers": "10",
86
+ "moe_layers": "10",
87
+ "seq_length": "4096",
88
+ "global_batch_size": "8",
89
+ "cuda_graph": "0",
90
+ "capacity_mode": "none",
91
+ "capacity": "none",
92
+ "memory_dump_mode": "iteration",
93
+ "branch": "moonep_backend_async_grad_reduce_better_memory_dump_support",
94
+ "commit": "51c387c7",
95
+ "status": "OK",
96
+ "max_allocated_gib": 59.09463548660278,
97
+ "max_reserved_gib": 61.44921875,
98
+ "fragmentation_gib": 2.354583263397217,
99
+ "theoretical_weight_optim_gib": 142.99818359375,
100
+ "peak_sampled_gib": 93.6083984375,
101
+ "snapshot_pickle_count": 40,
102
+ "snapshot_json_count": 40,
103
+ "snapshot_expected_each": 40,
104
+ "snapshot_complete": 1,
105
+ "effective_peak_gib": 93.6083984375
106
+ },
107
+ {
108
+ "run_dir": "moonep/tierbmem/tierbmem_gbs8/QWEN3_235B__moonep__matched__seq8192__gbs8__capnone__async__memiteration__it6/rep1",
109
+ "config": "QWEN3_235B",
110
+ "backend": "moonep",
111
+ "num_layers": "12",
112
+ "moe_layers": "12",
113
+ "seq_length": "8192",
114
+ "global_batch_size": "8",
115
+ "cuda_graph": "1",
116
+ "capacity_mode": "none",
117
+ "capacity": "none",
118
+ "memory_dump_mode": "iteration",
119
+ "branch": "moonep_backend_async_grad_reduce_better_memory_dump_support",
120
+ "commit": "51c387c7",
121
+ "status": "OK",
122
+ "max_allocated_gib": 65.66362237930298,
123
+ "max_reserved_gib": 66.94140625,
124
+ "fragmentation_gib": 1.2777838706970215,
125
+ "theoretical_weight_optim_gib": 211.35640625,
126
+ "peak_sampled_gib": 117.4833984375,
127
+ "snapshot_pickle_count": 48,
128
+ "snapshot_json_count": 48,
129
+ "snapshot_expected_each": 48,
130
+ "snapshot_complete": 1,
131
+ "effective_peak_gib": 117.4833984375
132
+ },
133
+ {
134
+ "run_dir": "moonep/tierbmem/tierbmem_gbs8/QWEN3_235B__moonep__matched__seq8192__gbs8__capnone__nocg__async__memiteration__it5/rep1",
135
+ "config": "QWEN3_235B",
136
+ "backend": "moonep",
137
+ "num_layers": "12",
138
+ "moe_layers": "12",
139
+ "seq_length": "8192",
140
+ "global_batch_size": "8",
141
+ "cuda_graph": "0",
142
+ "capacity_mode": "none",
143
+ "capacity": "none",
144
+ "memory_dump_mode": "iteration",
145
+ "branch": "moonep_backend_async_grad_reduce_better_memory_dump_support",
146
+ "commit": "51c387c7",
147
+ "status": "OK",
148
+ "max_allocated_gib": 65.60090827941895,
149
+ "max_reserved_gib": 66.87109375,
150
+ "fragmentation_gib": 1.2701854705810547,
151
+ "theoretical_weight_optim_gib": 211.35640625,
152
+ "peak_sampled_gib": 117.3154296875,
153
+ "snapshot_pickle_count": 40,
154
+ "snapshot_json_count": 40,
155
+ "snapshot_expected_each": 40,
156
+ "snapshot_complete": 1,
157
+ "effective_peak_gib": 117.3154296875
158
+ },
159
+ {
160
+ "run_dir": "moonep/tierbmem/tierbmem_gbs8/QWEN3_30B__moonep__matched__seq4096__gbs8__capnone__async__memiteration__it6/rep1",
161
+ "config": "QWEN3_30B",
162
+ "backend": "moonep",
163
+ "num_layers": "48",
164
+ "moe_layers": "48",
165
+ "seq_length": "4096",
166
+ "global_batch_size": "8",
167
+ "cuda_graph": "1",
168
+ "capacity_mode": "none",
169
+ "capacity": "none",
170
+ "memory_dump_mode": "iteration",
171
+ "branch": "moonep_backend_async_grad_reduce_better_memory_dump_support",
172
+ "commit": "51c387c7",
173
+ "status": "OK",
174
+ "max_allocated_gib": 65.04211616516113,
175
+ "max_reserved_gib": 66.6875,
176
+ "fragmentation_gib": 1.6453838348388672,
177
+ "theoretical_weight_optim_gib": 210.270107421875,
178
+ "peak_sampled_gib": 113.9677734375,
179
+ "snapshot_pickle_count": 48,
180
+ "snapshot_json_count": 48,
181
+ "snapshot_expected_each": 48,
182
+ "snapshot_complete": 1,
183
+ "effective_peak_gib": 113.9677734375
184
+ },
185
+ {
186
+ "run_dir": "moonep/tierbmem/tierbmem_gbs8/QWEN3_30B__moonep__matched__seq4096__gbs8__capnone__nocg__async__memiteration__it5/rep1",
187
+ "config": "QWEN3_30B",
188
+ "backend": "moonep",
189
+ "num_layers": "48",
190
+ "moe_layers": "48",
191
+ "seq_length": "4096",
192
+ "global_batch_size": "8",
193
+ "cuda_graph": "0",
194
+ "capacity_mode": "none",
195
+ "capacity": "none",
196
+ "memory_dump_mode": "iteration",
197
+ "branch": "moonep_backend_async_grad_reduce_better_memory_dump_support",
198
+ "commit": "51c387c7",
199
+ "status": "OK",
200
+ "max_allocated_gib": 65.03077840805054,
201
+ "max_reserved_gib": 66.6796875,
202
+ "fragmentation_gib": 1.648909091949463,
203
+ "theoretical_weight_optim_gib": 210.270107421875,
204
+ "peak_sampled_gib": 113.7158203125,
205
+ "snapshot_pickle_count": 40,
206
+ "snapshot_json_count": 40,
207
+ "snapshot_expected_each": 40,
208
+ "snapshot_complete": 1,
209
+ "effective_peak_gib": 113.7158203125
210
+ }
211
+ ],
212
+ "comparisons": [
213
+ {
214
+ "config": "DSV2_LITE_L24",
215
+ "backend": "moonep",
216
+ "moon_graph_on_status": "OK",
217
+ "moon_graph_on_peak_gib": 130.1865234375,
218
+ "moon_graph_off_status": "OK",
219
+ "moon_graph_off_peak_gib": 122.2236328125,
220
+ "moon_graph_delta_gib": 7.962890625
221
+ },
222
+ {
223
+ "config": "QWEN2_57B",
224
+ "backend": "moonep",
225
+ "moon_graph_on_status": "OK",
226
+ "moon_graph_on_peak_gib": 94.4697265625,
227
+ "moon_graph_off_status": "OK",
228
+ "moon_graph_off_peak_gib": 93.6083984375,
229
+ "moon_graph_delta_gib": 0.861328125
230
+ },
231
+ {
232
+ "config": "QWEN3_235B",
233
+ "backend": "moonep",
234
+ "moon_graph_on_status": "OK",
235
+ "moon_graph_on_peak_gib": 117.4833984375,
236
+ "moon_graph_off_status": "OK",
237
+ "moon_graph_off_peak_gib": 117.3154296875,
238
+ "moon_graph_delta_gib": 0.16796875
239
+ },
240
+ {
241
+ "config": "QWEN3_30B",
242
+ "backend": "moonep",
243
+ "moon_graph_on_status": "OK",
244
+ "moon_graph_on_peak_gib": 113.9677734375,
245
+ "moon_graph_off_status": "OK",
246
+ "moon_graph_off_peak_gib": 113.7158203125,
247
+ "moon_graph_delta_gib": 0.251953125
248
+ }
249
+ ]
250
+ }
tierb-fixed-moonep-full/memory_summary_all.tsv ADDED
@@ -0,0 +1,9 @@
 
 
 
 
 
 
 
 
 
 
1
+ run_dir config backend cuda_graph capacity_mode capacity status effective_peak_gib max_allocated_gib max_reserved_gib fragmentation_gib peak_sampled_gib snapshot_pickle_count snapshot_json_count snapshot_expected_each snapshot_complete branch commit
2
+ moonep/tierbmem/tierbmem_gbs8/DSV2_LITE_L24__moonep__matched__seq8192__gbs8__capnone__async__memiteration__it6/rep1 DSV2_LITE_L24 moonep 1 none none OK 130.1865234375 91.17775774002075 93.1953125 2.017554759979248 130.1865234375 48 48 48 1 moonep_backend_async_grad_reduce_better_memory_dump_support 51c387c7
3
+ moonep/tierbmem/tierbmem_gbs8/DSV2_LITE_L24__moonep__matched__seq8192__gbs8__capnone__nocg__async__memiteration__it5/rep1 DSV2_LITE_L24 moonep 0 none none OK 122.2236328125 91.17775774002075 93.1953125 2.017554759979248 122.2236328125 40 40 40 1 moonep_backend_async_grad_reduce_better_memory_dump_support 51c387c7
4
+ moonep/tierbmem/tierbmem_gbs8/QWEN2_57B__moonep__matched__seq4096__gbs8__capnone__async__memiteration__it6/rep1 QWEN2_57B moonep 1 none none OK 94.4697265625 59.11568737030029 61.42578125 2.310093879699707 94.4697265625 48 48 48 1 moonep_backend_async_grad_reduce_better_memory_dump_support 51c387c7
5
+ moonep/tierbmem/tierbmem_gbs8/QWEN2_57B__moonep__matched__seq4096__gbs8__capnone__nocg__async__memiteration__it5/rep1 QWEN2_57B moonep 0 none none OK 93.6083984375 59.09463548660278 61.44921875 2.354583263397217 93.6083984375 40 40 40 1 moonep_backend_async_grad_reduce_better_memory_dump_support 51c387c7
6
+ moonep/tierbmem/tierbmem_gbs8/QWEN3_235B__moonep__matched__seq8192__gbs8__capnone__async__memiteration__it6/rep1 QWEN3_235B moonep 1 none none OK 117.4833984375 65.66362237930298 66.94140625 1.2777838706970215 117.4833984375 48 48 48 1 moonep_backend_async_grad_reduce_better_memory_dump_support 51c387c7
7
+ moonep/tierbmem/tierbmem_gbs8/QWEN3_235B__moonep__matched__seq8192__gbs8__capnone__nocg__async__memiteration__it5/rep1 QWEN3_235B moonep 0 none none OK 117.3154296875 65.60090827941895 66.87109375 1.2701854705810547 117.3154296875 40 40 40 1 moonep_backend_async_grad_reduce_better_memory_dump_support 51c387c7
8
+ moonep/tierbmem/tierbmem_gbs8/QWEN3_30B__moonep__matched__seq4096__gbs8__capnone__async__memiteration__it6/rep1 QWEN3_30B moonep 1 none none OK 113.9677734375 65.04211616516113 66.6875 1.6453838348388672 113.9677734375 48 48 48 1 moonep_backend_async_grad_reduce_better_memory_dump_support 51c387c7
9
+ moonep/tierbmem/tierbmem_gbs8/QWEN3_30B__moonep__matched__seq4096__gbs8__capnone__nocg__async__memiteration__it5/rep1 QWEN3_30B moonep 0 none none OK 113.7158203125 65.03077840805054 66.6796875 1.648909091949463 113.7158203125 40 40 40 1 moonep_backend_async_grad_reduce_better_memory_dump_support 51c387c7