Hugging Face's logo

deeprcurs
/
MBG-1.0

Text Generation
PyTorch
Safetensors
mbg
model-bahasa-garuda
trinity-mirror
omni-sparse-hybrid
mixture-of-experts
Mixture of Experts
ssm
mamba
mla
ternary
gum
galore
muon
low-rank-optimization
language-model
Model card Files
xet
 Community
deeprcurs-staff commited on
Commit
c8a9aa1
·
verified ·
1 Parent(s): 1bec462

Upload reports/ablate_optimizers_20260901-161212.json with huggingface_hub

Browse files
Files changed (1) hide show
  1. reports/ablate_optimizers_20260901-161212.json +164 -0
reports/ablate_optimizers_20260901-161212.json ADDED
@@ -0,0 +1,164 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "embd64 L2 MoE4": {
3
+ "adamw": {
4
+ "final_val": 2.30124405225118,
5
+ "wall_s": 12.318387031555176,
6
+ "hist": [
7
+ {
8
+ "step": 1,
9
+ "train": 3.4174320697784424,
10
+ "val": 3.4070684631665547
11
+ },
12
+ {
13
+ "step": 50,
14
+ "train": 2.7972233295440674,
15
+ "val": 2.761105451981227
16
+ },
17
+ {
18
+ "step": 100,
19
+ "train": 2.4643044471740723,
20
+ "val": 2.4710233728090922
21
+ },
22
+ {
23
+ "step": 150,
24
+ "train": 2.330726146697998,
25
+ "val": 2.30124405225118
26
+ }
27
+ ]
28
+ },
29
+ "gum": {
30
+ "final_val": 2.9501822789510093,
31
+ "wall_s": 13.34013032913208,
32
+ "hist": [
33
+ {
34
+ "step": 1,
35
+ "train": 3.410735845565796,
36
+ "val": 3.4028487265110017
37
+ },
38
+ {
39
+ "step": 50,
40
+ "train": 3.2526257038116455,
41
+ "val": 3.242358034849167
42
+ },
43
+ {
44
+ "step": 100,
45
+ "train": 3.062805414199829,
46
+ "val": 3.0734894990921022
47
+ },
48
+ {
49
+ "step": 150,
50
+ "train": 2.962660551071167,
51
+ "val": 2.9501822789510093
52
+ }
53
+ ]
54
+ }
55
+ },
56
+ "embd128 L4 MoE4": {
57
+ "adamw": {
58
+ "final_val": 2.838918560743332,
59
+ "wall_s": 23.61824631690979,
60
+ "hist": [
61
+ {
62
+ "step": 1,
63
+ "train": 3.4587783813476562,
64
+ "val": 3.3166430473327635
65
+ },
66
+ {
67
+ "step": 50,
68
+ "train": 2.8018648624420166,
69
+ "val": 2.8400912046432496
70
+ },
71
+ {
72
+ "step": 100,
73
+ "train": 2.8464834690093994,
74
+ "val": 2.8382199803988137
75
+ },
76
+ {
77
+ "step": 150,
78
+ "train": 2.84476637840271,
79
+ "val": 2.838918560743332
80
+ }
81
+ ]
82
+ },
83
+ "gum": {
84
+ "final_val": 2.916172901789347,
85
+ "wall_s": 26.45097541809082,
86
+ "hist": [
87
+ {
88
+ "step": 1,
89
+ "train": 3.4262964725494385,
90
+ "val": 3.4049311498800914
91
+ },
92
+ {
93
+ "step": 50,
94
+ "train": 3.2132856845855713,
95
+ "val": 3.20825012922287
96
+ },
97
+ {
98
+ "step": 100,
99
+ "train": 3.0302419662475586,
100
+ "val": 3.035431573788325
101
+ },
102
+ {
103
+ "step": 150,
104
+ "train": 2.915634870529175,
105
+ "val": 2.916172901789347
106
+ }
107
+ ]
108
+ }
109
+ },
110
+ "embd192 L6 MoE8": {
111
+ "adamw": {
112
+ "final_val": 2.8389736115932465,
113
+ "wall_s": 43.53582835197449,
114
+ "hist": [
115
+ {
116
+ "step": 1,
117
+ "train": 3.436568021774292,
118
+ "val": 3.2055450399716694
119
+ },
120
+ {
121
+ "step": 50,
122
+ "train": 4.309718608856201,
123
+ "val": 2.8477032919724783
124
+ },
125
+ {
126
+ "step": 100,
127
+ "train": 2.8408186435699463,
128
+ "val": 2.8455129086971285
129
+ },
130
+ {
131
+ "step": 150,
132
+ "train": 2.8663265705108643,
133
+ "val": 2.8389736115932465
134
+ }
135
+ ]
136
+ },
137
+ "gum": {
138
+ "final_val": 2.9717845698197682,
139
+ "wall_s": 54.89743709564209,
140
+ "hist": [
141
+ {
142
+ "step": 1,
143
+ "train": 3.4295766353607178,
144
+ "val": 3.407314678033193
145
+ },
146
+ {
147
+ "step": 50,
148
+ "train": 3.2045648097991943,
149
+ "val": 3.160946196317673
150
+ },
151
+ {
152
+ "step": 100,
153
+ "train": 2.967801332473755,
154
+ "val": 2.9622103214263915
155
+ },
156
+ {
157
+ "step": 150,
158
+ "train": 2.975261926651001,
159
+ "val": 2.9717845698197682
160
+ }
161
+ ]
162
+ }
163
+ }
164
+ }