ProCreations commited on
Commit
aff0f3f
·
verified ·
1 Parent(s): f7787aa

Replace weights and README with strict W1.58A8 ternary artifact

Browse files
Files changed (5) hide show
  1. README.md +24 -46
  2. minima_config.json +191 -187
  3. model.safetensors +2 -2
  4. probe_report.json +0 -33
  5. training_report.json +0 -0
README.md CHANGED
@@ -5,24 +5,25 @@ license_link: https://huggingface.co/LiquidAI/LFM2.5-Encoder-350M/blob/main/LICE
5
  library_name: minima-lfm
6
  base_model: LiquidAI/LFM2.5-Encoder-350M
7
  tags:
8
- - ternary
9
- - 1.58-bit
10
- - encoder
11
- - cpu
 
12
  ---
13
 
14
  # Minima
15
 
16
- A W1.58A8 adaptation of
17
- [LiquidAI/LFM2.5-Encoder-350M](https://huggingface.co/LiquidAI/LFM2.5-Encoder-350M),
18
  built with [SSHDotCodes/minima](https://github.com/SSHDotCodes/minima).
19
 
20
  - Logical matrix values: `{-1, 0, +1}` (1.585 bits)
21
  - Physical artifact format: I2_S, four trits per byte
22
  - Dynamic int8 activations
23
- - Group size 32 with rank-128 FP16 recovery adapters
24
  - Full 8,192-token encoder context
25
- - 223.9 MB weight file, 84.2% smaller than the 1,418.0 MB source weight file
 
26
 
27
  ## Use
28
 
@@ -38,47 +39,24 @@ outputs = model(input_ids=input_ids, attention_mask=attention_mask)
38
  ```
39
 
40
  CPU inference defaults to a one-time FBGEMM dynamic-int8 packing of each
41
- effective ternary-plus-recovery matrix, after which source projection tensors are
42
- released. Set `MINIMA_CPU_BACKEND=i2s` for the direct packed 2-bit AVX2/ARM NEON
43
- kernel. The direct I2_S path has the smaller projection representation; the
44
- FBGEMM path is the measured throughput default.
45
-
46
- ## Measured CPU results
47
-
48
- Hugging Face `cpu-performance`, Linux x86-64, FBGEMM, 16 threads, one warmup and
49
- five measured runs:
50
-
51
- | Sequence | FP32 median | Minima median | Speedup | Peak RSS reduction |
52
- |---:|---:|---:|---:|---:|
53
- | 128 | 181.62 ms | 80.82 ms | 2.25x | 23.81% |
54
- | 512 | 479.12 ms | 247.94 ms | 1.93x | 26.77% |
55
- | 2,048 | 1,402.74 ms | 1,280.92 ms | 1.10x | 26.94% |
56
- | 8,192 | 7,878.43 ms | 7,312.03 ms | 1.08x | 24.42% |
57
-
58
- Peak RSS includes framework and activation memory, so it does not shrink by the
59
- same 84.2% as the weight file. Raw reports are in the
60
- [results dataset](https://huggingface.co/datasets/ProCreations/minima-results).
61
 
62
  ## Encoder quality
63
 
64
- The six-task downstream gate retained **96.66%** of the matched FP32 baseline
65
- after per-task ratios were capped at 100%. Five non-CoLA tasks averaged 98.05%;
66
- CoLA retained 89.70%. This misses the declared 97% threshold by 0.34 percentage
67
- points, so this artifact is a **release candidate**, not a quality-gated release.
68
- The validation-selected CoLA schedule and all eight candidates are published in
69
- the [raw report](https://huggingface.co/datasets/ProCreations/minima-results/blob/main/quality_gate.json).
70
-
71
- ## CUDA status
72
-
73
- The fused Triton path avoids materializing a full dequantized weight tensor and
74
- passes correctness checks, but the current kernel is slower than upstream BF16
75
- on an H200. It is included for optimization work, not advertised as a GPU speedup.
76
-
77
- ## Diagnostics
78
-
79
- The release distillation probe measured hidden-state cosine 0.8491, relative L2
80
- 0.5877, and 99.67% masked-token top-1 agreement across 601 positions. These are
81
- diagnostics; the downstream task gate above is the release-quality measure.
82
 
83
  ## License
84
 
 
5
  library_name: minima-lfm
6
  base_model: LiquidAI/LFM2.5-Encoder-350M
7
  tags:
8
+ - lfm2
9
+ - encoder
10
+ - ternary
11
+ - bitnet
12
+ - quantization
13
  ---
14
 
15
  # Minima
16
 
17
+ A strict W1.58A8 adaptation of [LiquidAI/LFM2.5-Encoder-350M](https://huggingface.co/LiquidAI/LFM2.5-Encoder-350M),
 
18
  built with [SSHDotCodes/minima](https://github.com/SSHDotCodes/minima).
19
 
20
  - Logical matrix values: `{-1, 0, +1}` (1.585 bits)
21
  - Physical artifact format: I2_S, four trits per byte
22
  - Dynamic int8 activations
23
+ - Group size 128, **no recovery adapters**
24
  - Full 8,192-token encoder context
25
+ - Packed weight file: **89.9 MB** (94,298,568 bytes)
26
+ - Release status: **release candidate** (six-task gated retention 85.45% vs FP32)
27
 
28
  ## Use
29
 
 
39
  ```
40
 
41
  CPU inference defaults to a one-time FBGEMM dynamic-int8 packing of each
42
+ ternary matrix. Set `MINIMA_CPU_BACKEND=i2s` for the direct packed 2-bit
43
+ AVX2/ARM NEON kernel.
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
44
 
45
  ## Encoder quality
46
 
47
+ The six-task downstream gate compares matched 800-step fine-tunes. Packed
48
+ ternary matrices stay frozen; only the task head and non-matrix parameters
49
+ adapt. Required relative mean: **>= 96%**.
50
+
51
+ | Task | FP32 | Minima | Capped retention |
52
+ | --- | ---: | ---: | ---: |
53
+ | SST2 | 0.79817 | 0.75459 | 94.54% |
54
+ | QNLI | 0.60608 | 0.59491 | 98.16% |
55
+ | MNLI | 0.40621 | 0.38074 | 93.73% |
56
+ | MRPC | 0.81694 | 0.81873 | 100.00% |
57
+ | STSB | 0.54812 | 0.40660 | 74.18% |
58
+ | COLA | 0.11382 | 0.05927 | 52.07% |
59
+ | **Mean** | | | **85.45%** |
 
 
 
 
 
60
 
61
  ## License
62
 
minima_config.json CHANGED
@@ -5,753 +5,757 @@
5
  "physical_weight_bits": 2,
6
  "activation_bits": 8,
7
  "base_model": "LiquidAI/LFM2.5-Encoder-350M",
 
8
  "model_kind": "encoder",
9
  "max_context": 8192,
 
 
 
10
  "modules": {
11
  "embed_tokens": {
12
  "kind": "embedding",
13
  "num_embeddings": 65536,
14
  "embedding_dim": 1024,
15
  "padding_idx": 0,
16
- "group_size": 32,
17
- "recovery_rank": 128
18
  },
19
  "layers.0.conv.in_proj": {
20
  "kind": "linear",
21
  "in_features": 1024,
22
  "out_features": 3072,
23
- "group_size": 32,
24
  "bias": false,
25
- "recovery_rank": 128
26
  },
27
  "layers.0.conv.out_proj": {
28
  "kind": "linear",
29
  "in_features": 1024,
30
  "out_features": 1024,
31
- "group_size": 32,
32
  "bias": false,
33
- "recovery_rank": 128
34
  },
35
  "layers.0.feed_forward.w1": {
36
  "kind": "linear",
37
  "in_features": 1024,
38
  "out_features": 4608,
39
- "group_size": 32,
40
  "bias": false,
41
- "recovery_rank": 128
42
  },
43
  "layers.0.feed_forward.w3": {
44
  "kind": "linear",
45
  "in_features": 1024,
46
  "out_features": 4608,
47
- "group_size": 32,
48
  "bias": false,
49
- "recovery_rank": 128
50
  },
51
  "layers.0.feed_forward.w2": {
52
  "kind": "linear",
53
  "in_features": 4608,
54
  "out_features": 1024,
55
- "group_size": 32,
56
  "bias": false,
57
- "recovery_rank": 128
58
  },
59
  "layers.1.conv.in_proj": {
60
  "kind": "linear",
61
  "in_features": 1024,
62
  "out_features": 3072,
63
- "group_size": 32,
64
  "bias": false,
65
- "recovery_rank": 128
66
  },
67
  "layers.1.conv.out_proj": {
68
  "kind": "linear",
69
  "in_features": 1024,
70
  "out_features": 1024,
71
- "group_size": 32,
72
  "bias": false,
73
- "recovery_rank": 128
74
  },
75
  "layers.1.feed_forward.w1": {
76
  "kind": "linear",
77
  "in_features": 1024,
78
  "out_features": 4608,
79
- "group_size": 32,
80
  "bias": false,
81
- "recovery_rank": 128
82
  },
83
  "layers.1.feed_forward.w3": {
84
  "kind": "linear",
85
  "in_features": 1024,
86
  "out_features": 4608,
87
- "group_size": 32,
88
  "bias": false,
89
- "recovery_rank": 128
90
  },
91
  "layers.1.feed_forward.w2": {
92
  "kind": "linear",
93
  "in_features": 4608,
94
  "out_features": 1024,
95
- "group_size": 32,
96
  "bias": false,
97
- "recovery_rank": 128
98
  },
99
  "layers.2.self_attn.q_proj": {
100
  "kind": "linear",
101
  "in_features": 1024,
102
  "out_features": 1024,
103
- "group_size": 32,
104
  "bias": false,
105
- "recovery_rank": 128
106
  },
107
  "layers.2.self_attn.k_proj": {
108
  "kind": "linear",
109
  "in_features": 1024,
110
  "out_features": 512,
111
- "group_size": 32,
112
  "bias": false,
113
- "recovery_rank": 128
114
  },
115
  "layers.2.self_attn.v_proj": {
116
  "kind": "linear",
117
  "in_features": 1024,
118
  "out_features": 512,
119
- "group_size": 32,
120
  "bias": false,
121
- "recovery_rank": 128
122
  },
123
  "layers.2.self_attn.out_proj": {
124
  "kind": "linear",
125
  "in_features": 1024,
126
  "out_features": 1024,
127
- "group_size": 32,
128
  "bias": false,
129
- "recovery_rank": 128
130
  },
131
  "layers.2.feed_forward.w1": {
132
  "kind": "linear",
133
  "in_features": 1024,
134
  "out_features": 4608,
135
- "group_size": 32,
136
  "bias": false,
137
- "recovery_rank": 128
138
  },
139
  "layers.2.feed_forward.w3": {
140
  "kind": "linear",
141
  "in_features": 1024,
142
  "out_features": 4608,
143
- "group_size": 32,
144
  "bias": false,
145
- "recovery_rank": 128
146
  },
147
  "layers.2.feed_forward.w2": {
148
  "kind": "linear",
149
  "in_features": 4608,
150
  "out_features": 1024,
151
- "group_size": 32,
152
  "bias": false,
153
- "recovery_rank": 128
154
  },
155
  "layers.3.conv.in_proj": {
156
  "kind": "linear",
157
  "in_features": 1024,
158
  "out_features": 3072,
159
- "group_size": 32,
160
  "bias": false,
161
- "recovery_rank": 128
162
  },
163
  "layers.3.conv.out_proj": {
164
  "kind": "linear",
165
  "in_features": 1024,
166
  "out_features": 1024,
167
- "group_size": 32,
168
  "bias": false,
169
- "recovery_rank": 128
170
  },
171
  "layers.3.feed_forward.w1": {
172
  "kind": "linear",
173
  "in_features": 1024,
174
  "out_features": 4608,
175
- "group_size": 32,
176
  "bias": false,
177
- "recovery_rank": 128
178
  },
179
  "layers.3.feed_forward.w3": {
180
  "kind": "linear",
181
  "in_features": 1024,
182
  "out_features": 4608,
183
- "group_size": 32,
184
  "bias": false,
185
- "recovery_rank": 128
186
  },
187
  "layers.3.feed_forward.w2": {
188
  "kind": "linear",
189
  "in_features": 4608,
190
  "out_features": 1024,
191
- "group_size": 32,
192
  "bias": false,
193
- "recovery_rank": 128
194
  },
195
  "layers.4.conv.in_proj": {
196
  "kind": "linear",
197
  "in_features": 1024,
198
  "out_features": 3072,
199
- "group_size": 32,
200
  "bias": false,
201
- "recovery_rank": 128
202
  },
203
  "layers.4.conv.out_proj": {
204
  "kind": "linear",
205
  "in_features": 1024,
206
  "out_features": 1024,
207
- "group_size": 32,
208
  "bias": false,
209
- "recovery_rank": 128
210
  },
211
  "layers.4.feed_forward.w1": {
212
  "kind": "linear",
213
  "in_features": 1024,
214
  "out_features": 4608,
215
- "group_size": 32,
216
  "bias": false,
217
- "recovery_rank": 128
218
  },
219
  "layers.4.feed_forward.w3": {
220
  "kind": "linear",
221
  "in_features": 1024,
222
  "out_features": 4608,
223
- "group_size": 32,
224
  "bias": false,
225
- "recovery_rank": 128
226
  },
227
  "layers.4.feed_forward.w2": {
228
  "kind": "linear",
229
  "in_features": 4608,
230
  "out_features": 1024,
231
- "group_size": 32,
232
  "bias": false,
233
- "recovery_rank": 128
234
  },
235
  "layers.5.self_attn.q_proj": {
236
  "kind": "linear",
237
  "in_features": 1024,
238
  "out_features": 1024,
239
- "group_size": 32,
240
  "bias": false,
241
- "recovery_rank": 128
242
  },
243
  "layers.5.self_attn.k_proj": {
244
  "kind": "linear",
245
  "in_features": 1024,
246
  "out_features": 512,
247
- "group_size": 32,
248
  "bias": false,
249
- "recovery_rank": 128
250
  },
251
  "layers.5.self_attn.v_proj": {
252
  "kind": "linear",
253
  "in_features": 1024,
254
  "out_features": 512,
255
- "group_size": 32,
256
  "bias": false,
257
- "recovery_rank": 128
258
  },
259
  "layers.5.self_attn.out_proj": {
260
  "kind": "linear",
261
  "in_features": 1024,
262
  "out_features": 1024,
263
- "group_size": 32,
264
  "bias": false,
265
- "recovery_rank": 128
266
  },
267
  "layers.5.feed_forward.w1": {
268
  "kind": "linear",
269
  "in_features": 1024,
270
  "out_features": 4608,
271
- "group_size": 32,
272
  "bias": false,
273
- "recovery_rank": 128
274
  },
275
  "layers.5.feed_forward.w3": {
276
  "kind": "linear",
277
  "in_features": 1024,
278
  "out_features": 4608,
279
- "group_size": 32,
280
  "bias": false,
281
- "recovery_rank": 128
282
  },
283
  "layers.5.feed_forward.w2": {
284
  "kind": "linear",
285
  "in_features": 4608,
286
  "out_features": 1024,
287
- "group_size": 32,
288
  "bias": false,
289
- "recovery_rank": 128
290
  },
291
  "layers.6.conv.in_proj": {
292
  "kind": "linear",
293
  "in_features": 1024,
294
  "out_features": 3072,
295
- "group_size": 32,
296
  "bias": false,
297
- "recovery_rank": 128
298
  },
299
  "layers.6.conv.out_proj": {
300
  "kind": "linear",
301
  "in_features": 1024,
302
  "out_features": 1024,
303
- "group_size": 32,
304
  "bias": false,
305
- "recovery_rank": 128
306
  },
307
  "layers.6.feed_forward.w1": {
308
  "kind": "linear",
309
  "in_features": 1024,
310
  "out_features": 4608,
311
- "group_size": 32,
312
  "bias": false,
313
- "recovery_rank": 128
314
  },
315
  "layers.6.feed_forward.w3": {
316
  "kind": "linear",
317
  "in_features": 1024,
318
  "out_features": 4608,
319
- "group_size": 32,
320
  "bias": false,
321
- "recovery_rank": 128
322
  },
323
  "layers.6.feed_forward.w2": {
324
  "kind": "linear",
325
  "in_features": 4608,
326
  "out_features": 1024,
327
- "group_size": 32,
328
  "bias": false,
329
- "recovery_rank": 128
330
  },
331
  "layers.7.conv.in_proj": {
332
  "kind": "linear",
333
  "in_features": 1024,
334
  "out_features": 3072,
335
- "group_size": 32,
336
  "bias": false,
337
- "recovery_rank": 128
338
  },
339
  "layers.7.conv.out_proj": {
340
  "kind": "linear",
341
  "in_features": 1024,
342
  "out_features": 1024,
343
- "group_size": 32,
344
  "bias": false,
345
- "recovery_rank": 128
346
  },
347
  "layers.7.feed_forward.w1": {
348
  "kind": "linear",
349
  "in_features": 1024,
350
  "out_features": 4608,
351
- "group_size": 32,
352
  "bias": false,
353
- "recovery_rank": 128
354
  },
355
  "layers.7.feed_forward.w3": {
356
  "kind": "linear",
357
  "in_features": 1024,
358
  "out_features": 4608,
359
- "group_size": 32,
360
  "bias": false,
361
- "recovery_rank": 128
362
  },
363
  "layers.7.feed_forward.w2": {
364
  "kind": "linear",
365
  "in_features": 4608,
366
  "out_features": 1024,
367
- "group_size": 32,
368
  "bias": false,
369
- "recovery_rank": 128
370
  },
371
  "layers.8.self_attn.q_proj": {
372
  "kind": "linear",
373
  "in_features": 1024,
374
  "out_features": 1024,
375
- "group_size": 32,
376
  "bias": false,
377
- "recovery_rank": 128
378
  },
379
  "layers.8.self_attn.k_proj": {
380
  "kind": "linear",
381
  "in_features": 1024,
382
  "out_features": 512,
383
- "group_size": 32,
384
  "bias": false,
385
- "recovery_rank": 128
386
  },
387
  "layers.8.self_attn.v_proj": {
388
  "kind": "linear",
389
  "in_features": 1024,
390
  "out_features": 512,
391
- "group_size": 32,
392
  "bias": false,
393
- "recovery_rank": 128
394
  },
395
  "layers.8.self_attn.out_proj": {
396
  "kind": "linear",
397
  "in_features": 1024,
398
  "out_features": 1024,
399
- "group_size": 32,
400
  "bias": false,
401
- "recovery_rank": 128
402
  },
403
  "layers.8.feed_forward.w1": {
404
  "kind": "linear",
405
  "in_features": 1024,
406
  "out_features": 4608,
407
- "group_size": 32,
408
  "bias": false,
409
- "recovery_rank": 128
410
  },
411
  "layers.8.feed_forward.w3": {
412
  "kind": "linear",
413
  "in_features": 1024,
414
  "out_features": 4608,
415
- "group_size": 32,
416
  "bias": false,
417
- "recovery_rank": 128
418
  },
419
  "layers.8.feed_forward.w2": {
420
  "kind": "linear",
421
  "in_features": 4608,
422
  "out_features": 1024,
423
- "group_size": 32,
424
  "bias": false,
425
- "recovery_rank": 128
426
  },
427
  "layers.9.conv.in_proj": {
428
  "kind": "linear",
429
  "in_features": 1024,
430
  "out_features": 3072,
431
- "group_size": 32,
432
  "bias": false,
433
- "recovery_rank": 128
434
  },
435
  "layers.9.conv.out_proj": {
436
  "kind": "linear",
437
  "in_features": 1024,
438
  "out_features": 1024,
439
- "group_size": 32,
440
  "bias": false,
441
- "recovery_rank": 128
442
  },
443
  "layers.9.feed_forward.w1": {
444
  "kind": "linear",
445
  "in_features": 1024,
446
  "out_features": 4608,
447
- "group_size": 32,
448
  "bias": false,
449
- "recovery_rank": 128
450
  },
451
  "layers.9.feed_forward.w3": {
452
  "kind": "linear",
453
  "in_features": 1024,
454
  "out_features": 4608,
455
- "group_size": 32,
456
  "bias": false,
457
- "recovery_rank": 128
458
  },
459
  "layers.9.feed_forward.w2": {
460
  "kind": "linear",
461
  "in_features": 4608,
462
  "out_features": 1024,
463
- "group_size": 32,
464
  "bias": false,
465
- "recovery_rank": 128
466
  },
467
  "layers.10.self_attn.q_proj": {
468
  "kind": "linear",
469
  "in_features": 1024,
470
  "out_features": 1024,
471
- "group_size": 32,
472
  "bias": false,
473
- "recovery_rank": 128
474
  },
475
  "layers.10.self_attn.k_proj": {
476
  "kind": "linear",
477
  "in_features": 1024,
478
  "out_features": 512,
479
- "group_size": 32,
480
  "bias": false,
481
- "recovery_rank": 128
482
  },
483
  "layers.10.self_attn.v_proj": {
484
  "kind": "linear",
485
  "in_features": 1024,
486
  "out_features": 512,
487
- "group_size": 32,
488
  "bias": false,
489
- "recovery_rank": 128
490
  },
491
  "layers.10.self_attn.out_proj": {
492
  "kind": "linear",
493
  "in_features": 1024,
494
  "out_features": 1024,
495
- "group_size": 32,
496
  "bias": false,
497
- "recovery_rank": 128
498
  },
499
  "layers.10.feed_forward.w1": {
500
  "kind": "linear",
501
  "in_features": 1024,
502
  "out_features": 4608,
503
- "group_size": 32,
504
  "bias": false,
505
- "recovery_rank": 128
506
  },
507
  "layers.10.feed_forward.w3": {
508
  "kind": "linear",
509
  "in_features": 1024,
510
  "out_features": 4608,
511
- "group_size": 32,
512
  "bias": false,
513
- "recovery_rank": 128
514
  },
515
  "layers.10.feed_forward.w2": {
516
  "kind": "linear",
517
  "in_features": 4608,
518
  "out_features": 1024,
519
- "group_size": 32,
520
  "bias": false,
521
- "recovery_rank": 128
522
  },
523
  "layers.11.conv.in_proj": {
524
  "kind": "linear",
525
  "in_features": 1024,
526
  "out_features": 3072,
527
- "group_size": 32,
528
  "bias": false,
529
- "recovery_rank": 128
530
  },
531
  "layers.11.conv.out_proj": {
532
  "kind": "linear",
533
  "in_features": 1024,
534
  "out_features": 1024,
535
- "group_size": 32,
536
  "bias": false,
537
- "recovery_rank": 128
538
  },
539
  "layers.11.feed_forward.w1": {
540
  "kind": "linear",
541
  "in_features": 1024,
542
  "out_features": 4608,
543
- "group_size": 32,
544
  "bias": false,
545
- "recovery_rank": 128
546
  },
547
  "layers.11.feed_forward.w3": {
548
  "kind": "linear",
549
  "in_features": 1024,
550
  "out_features": 4608,
551
- "group_size": 32,
552
  "bias": false,
553
- "recovery_rank": 128
554
  },
555
  "layers.11.feed_forward.w2": {
556
  "kind": "linear",
557
  "in_features": 4608,
558
  "out_features": 1024,
559
- "group_size": 32,
560
  "bias": false,
561
- "recovery_rank": 128
562
  },
563
  "layers.12.self_attn.q_proj": {
564
  "kind": "linear",
565
  "in_features": 1024,
566
  "out_features": 1024,
567
- "group_size": 32,
568
  "bias": false,
569
- "recovery_rank": 128
570
  },
571
  "layers.12.self_attn.k_proj": {
572
  "kind": "linear",
573
  "in_features": 1024,
574
  "out_features": 512,
575
- "group_size": 32,
576
  "bias": false,
577
- "recovery_rank": 128
578
  },
579
  "layers.12.self_attn.v_proj": {
580
  "kind": "linear",
581
  "in_features": 1024,
582
  "out_features": 512,
583
- "group_size": 32,
584
  "bias": false,
585
- "recovery_rank": 128
586
  },
587
  "layers.12.self_attn.out_proj": {
588
  "kind": "linear",
589
  "in_features": 1024,
590
  "out_features": 1024,
591
- "group_size": 32,
592
  "bias": false,
593
- "recovery_rank": 128
594
  },
595
  "layers.12.feed_forward.w1": {
596
  "kind": "linear",
597
  "in_features": 1024,
598
  "out_features": 4608,
599
- "group_size": 32,
600
  "bias": false,
601
- "recovery_rank": 128
602
  },
603
  "layers.12.feed_forward.w3": {
604
  "kind": "linear",
605
  "in_features": 1024,
606
  "out_features": 4608,
607
- "group_size": 32,
608
  "bias": false,
609
- "recovery_rank": 128
610
  },
611
  "layers.12.feed_forward.w2": {
612
  "kind": "linear",
613
  "in_features": 4608,
614
  "out_features": 1024,
615
- "group_size": 32,
616
  "bias": false,
617
- "recovery_rank": 128
618
  },
619
  "layers.13.conv.in_proj": {
620
  "kind": "linear",
621
  "in_features": 1024,
622
  "out_features": 3072,
623
- "group_size": 32,
624
  "bias": false,
625
- "recovery_rank": 128
626
  },
627
  "layers.13.conv.out_proj": {
628
  "kind": "linear",
629
  "in_features": 1024,
630
  "out_features": 1024,
631
- "group_size": 32,
632
  "bias": false,
633
- "recovery_rank": 128
634
  },
635
  "layers.13.feed_forward.w1": {
636
  "kind": "linear",
637
  "in_features": 1024,
638
  "out_features": 4608,
639
- "group_size": 32,
640
  "bias": false,
641
- "recovery_rank": 128
642
  },
643
  "layers.13.feed_forward.w3": {
644
  "kind": "linear",
645
  "in_features": 1024,
646
  "out_features": 4608,
647
- "group_size": 32,
648
  "bias": false,
649
- "recovery_rank": 128
650
  },
651
  "layers.13.feed_forward.w2": {
652
  "kind": "linear",
653
  "in_features": 4608,
654
  "out_features": 1024,
655
- "group_size": 32,
656
  "bias": false,
657
- "recovery_rank": 128
658
  },
659
  "layers.14.self_attn.q_proj": {
660
  "kind": "linear",
661
  "in_features": 1024,
662
  "out_features": 1024,
663
- "group_size": 32,
664
  "bias": false,
665
- "recovery_rank": 128
666
  },
667
  "layers.14.self_attn.k_proj": {
668
  "kind": "linear",
669
  "in_features": 1024,
670
  "out_features": 512,
671
- "group_size": 32,
672
  "bias": false,
673
- "recovery_rank": 128
674
  },
675
  "layers.14.self_attn.v_proj": {
676
  "kind": "linear",
677
  "in_features": 1024,
678
  "out_features": 512,
679
- "group_size": 32,
680
  "bias": false,
681
- "recovery_rank": 128
682
  },
683
  "layers.14.self_attn.out_proj": {
684
  "kind": "linear",
685
  "in_features": 1024,
686
  "out_features": 1024,
687
- "group_size": 32,
688
  "bias": false,
689
- "recovery_rank": 128
690
  },
691
  "layers.14.feed_forward.w1": {
692
  "kind": "linear",
693
  "in_features": 1024,
694
  "out_features": 4608,
695
- "group_size": 32,
696
  "bias": false,
697
- "recovery_rank": 128
698
  },
699
  "layers.14.feed_forward.w3": {
700
  "kind": "linear",
701
  "in_features": 1024,
702
  "out_features": 4608,
703
- "group_size": 32,
704
  "bias": false,
705
- "recovery_rank": 128
706
  },
707
  "layers.14.feed_forward.w2": {
708
  "kind": "linear",
709
  "in_features": 4608,
710
  "out_features": 1024,
711
- "group_size": 32,
712
  "bias": false,
713
- "recovery_rank": 128
714
  },
715
  "layers.15.conv.in_proj": {
716
  "kind": "linear",
717
  "in_features": 1024,
718
  "out_features": 3072,
719
- "group_size": 32,
720
  "bias": false,
721
- "recovery_rank": 128
722
  },
723
  "layers.15.conv.out_proj": {
724
  "kind": "linear",
725
  "in_features": 1024,
726
  "out_features": 1024,
727
- "group_size": 32,
728
  "bias": false,
729
- "recovery_rank": 128
730
  },
731
  "layers.15.feed_forward.w1": {
732
  "kind": "linear",
733
  "in_features": 1024,
734
  "out_features": 4608,
735
- "group_size": 32,
736
  "bias": false,
737
- "recovery_rank": 128
738
  },
739
  "layers.15.feed_forward.w3": {
740
  "kind": "linear",
741
  "in_features": 1024,
742
  "out_features": 4608,
743
- "group_size": 32,
744
  "bias": false,
745
- "recovery_rank": 128
746
  },
747
  "layers.15.feed_forward.w2": {
748
  "kind": "linear",
749
  "in_features": 4608,
750
  "out_features": 1024,
751
- "group_size": 32,
752
  "bias": false,
753
- "recovery_rank": 128
754
  }
755
  },
756
- "base_revision": "b886781f7c6f10ca9b7096e21b83e30a073c2f39"
757
  }
 
5
  "physical_weight_bits": 2,
6
  "activation_bits": 8,
7
  "base_model": "LiquidAI/LFM2.5-Encoder-350M",
8
+ "base_revision": null,
9
  "model_kind": "encoder",
10
  "max_context": 8192,
11
+ "group_size": 128,
12
+ "recovery_rank": 0,
13
+ "profile": "strict",
14
  "modules": {
15
  "embed_tokens": {
16
  "kind": "embedding",
17
  "num_embeddings": 65536,
18
  "embedding_dim": 1024,
19
  "padding_idx": 0,
20
+ "group_size": 128,
21
+ "recovery_rank": 0
22
  },
23
  "layers.0.conv.in_proj": {
24
  "kind": "linear",
25
  "in_features": 1024,
26
  "out_features": 3072,
27
+ "group_size": 128,
28
  "bias": false,
29
+ "recovery_rank": 0
30
  },
31
  "layers.0.conv.out_proj": {
32
  "kind": "linear",
33
  "in_features": 1024,
34
  "out_features": 1024,
35
+ "group_size": 128,
36
  "bias": false,
37
+ "recovery_rank": 0
38
  },
39
  "layers.0.feed_forward.w1": {
40
  "kind": "linear",
41
  "in_features": 1024,
42
  "out_features": 4608,
43
+ "group_size": 128,
44
  "bias": false,
45
+ "recovery_rank": 0
46
  },
47
  "layers.0.feed_forward.w3": {
48
  "kind": "linear",
49
  "in_features": 1024,
50
  "out_features": 4608,
51
+ "group_size": 128,
52
  "bias": false,
53
+ "recovery_rank": 0
54
  },
55
  "layers.0.feed_forward.w2": {
56
  "kind": "linear",
57
  "in_features": 4608,
58
  "out_features": 1024,
59
+ "group_size": 128,
60
  "bias": false,
61
+ "recovery_rank": 0
62
  },
63
  "layers.1.conv.in_proj": {
64
  "kind": "linear",
65
  "in_features": 1024,
66
  "out_features": 3072,
67
+ "group_size": 128,
68
  "bias": false,
69
+ "recovery_rank": 0
70
  },
71
  "layers.1.conv.out_proj": {
72
  "kind": "linear",
73
  "in_features": 1024,
74
  "out_features": 1024,
75
+ "group_size": 128,
76
  "bias": false,
77
+ "recovery_rank": 0
78
  },
79
  "layers.1.feed_forward.w1": {
80
  "kind": "linear",
81
  "in_features": 1024,
82
  "out_features": 4608,
83
+ "group_size": 128,
84
  "bias": false,
85
+ "recovery_rank": 0
86
  },
87
  "layers.1.feed_forward.w3": {
88
  "kind": "linear",
89
  "in_features": 1024,
90
  "out_features": 4608,
91
+ "group_size": 128,
92
  "bias": false,
93
+ "recovery_rank": 0
94
  },
95
  "layers.1.feed_forward.w2": {
96
  "kind": "linear",
97
  "in_features": 4608,
98
  "out_features": 1024,
99
+ "group_size": 128,
100
  "bias": false,
101
+ "recovery_rank": 0
102
  },
103
  "layers.2.self_attn.q_proj": {
104
  "kind": "linear",
105
  "in_features": 1024,
106
  "out_features": 1024,
107
+ "group_size": 128,
108
  "bias": false,
109
+ "recovery_rank": 0
110
  },
111
  "layers.2.self_attn.k_proj": {
112
  "kind": "linear",
113
  "in_features": 1024,
114
  "out_features": 512,
115
+ "group_size": 128,
116
  "bias": false,
117
+ "recovery_rank": 0
118
  },
119
  "layers.2.self_attn.v_proj": {
120
  "kind": "linear",
121
  "in_features": 1024,
122
  "out_features": 512,
123
+ "group_size": 128,
124
  "bias": false,
125
+ "recovery_rank": 0
126
  },
127
  "layers.2.self_attn.out_proj": {
128
  "kind": "linear",
129
  "in_features": 1024,
130
  "out_features": 1024,
131
+ "group_size": 128,
132
  "bias": false,
133
+ "recovery_rank": 0
134
  },
135
  "layers.2.feed_forward.w1": {
136
  "kind": "linear",
137
  "in_features": 1024,
138
  "out_features": 4608,
139
+ "group_size": 128,
140
  "bias": false,
141
+ "recovery_rank": 0
142
  },
143
  "layers.2.feed_forward.w3": {
144
  "kind": "linear",
145
  "in_features": 1024,
146
  "out_features": 4608,
147
+ "group_size": 128,
148
  "bias": false,
149
+ "recovery_rank": 0
150
  },
151
  "layers.2.feed_forward.w2": {
152
  "kind": "linear",
153
  "in_features": 4608,
154
  "out_features": 1024,
155
+ "group_size": 128,
156
  "bias": false,
157
+ "recovery_rank": 0
158
  },
159
  "layers.3.conv.in_proj": {
160
  "kind": "linear",
161
  "in_features": 1024,
162
  "out_features": 3072,
163
+ "group_size": 128,
164
  "bias": false,
165
+ "recovery_rank": 0
166
  },
167
  "layers.3.conv.out_proj": {
168
  "kind": "linear",
169
  "in_features": 1024,
170
  "out_features": 1024,
171
+ "group_size": 128,
172
  "bias": false,
173
+ "recovery_rank": 0
174
  },
175
  "layers.3.feed_forward.w1": {
176
  "kind": "linear",
177
  "in_features": 1024,
178
  "out_features": 4608,
179
+ "group_size": 128,
180
  "bias": false,
181
+ "recovery_rank": 0
182
  },
183
  "layers.3.feed_forward.w3": {
184
  "kind": "linear",
185
  "in_features": 1024,
186
  "out_features": 4608,
187
+ "group_size": 128,
188
  "bias": false,
189
+ "recovery_rank": 0
190
  },
191
  "layers.3.feed_forward.w2": {
192
  "kind": "linear",
193
  "in_features": 4608,
194
  "out_features": 1024,
195
+ "group_size": 128,
196
  "bias": false,
197
+ "recovery_rank": 0
198
  },
199
  "layers.4.conv.in_proj": {
200
  "kind": "linear",
201
  "in_features": 1024,
202
  "out_features": 3072,
203
+ "group_size": 128,
204
  "bias": false,
205
+ "recovery_rank": 0
206
  },
207
  "layers.4.conv.out_proj": {
208
  "kind": "linear",
209
  "in_features": 1024,
210
  "out_features": 1024,
211
+ "group_size": 128,
212
  "bias": false,
213
+ "recovery_rank": 0
214
  },
215
  "layers.4.feed_forward.w1": {
216
  "kind": "linear",
217
  "in_features": 1024,
218
  "out_features": 4608,
219
+ "group_size": 128,
220
  "bias": false,
221
+ "recovery_rank": 0
222
  },
223
  "layers.4.feed_forward.w3": {
224
  "kind": "linear",
225
  "in_features": 1024,
226
  "out_features": 4608,
227
+ "group_size": 128,
228
  "bias": false,
229
+ "recovery_rank": 0
230
  },
231
  "layers.4.feed_forward.w2": {
232
  "kind": "linear",
233
  "in_features": 4608,
234
  "out_features": 1024,
235
+ "group_size": 128,
236
  "bias": false,
237
+ "recovery_rank": 0
238
  },
239
  "layers.5.self_attn.q_proj": {
240
  "kind": "linear",
241
  "in_features": 1024,
242
  "out_features": 1024,
243
+ "group_size": 128,
244
  "bias": false,
245
+ "recovery_rank": 0
246
  },
247
  "layers.5.self_attn.k_proj": {
248
  "kind": "linear",
249
  "in_features": 1024,
250
  "out_features": 512,
251
+ "group_size": 128,
252
  "bias": false,
253
+ "recovery_rank": 0
254
  },
255
  "layers.5.self_attn.v_proj": {
256
  "kind": "linear",
257
  "in_features": 1024,
258
  "out_features": 512,
259
+ "group_size": 128,
260
  "bias": false,
261
+ "recovery_rank": 0
262
  },
263
  "layers.5.self_attn.out_proj": {
264
  "kind": "linear",
265
  "in_features": 1024,
266
  "out_features": 1024,
267
+ "group_size": 128,
268
  "bias": false,
269
+ "recovery_rank": 0
270
  },
271
  "layers.5.feed_forward.w1": {
272
  "kind": "linear",
273
  "in_features": 1024,
274
  "out_features": 4608,
275
+ "group_size": 128,
276
  "bias": false,
277
+ "recovery_rank": 0
278
  },
279
  "layers.5.feed_forward.w3": {
280
  "kind": "linear",
281
  "in_features": 1024,
282
  "out_features": 4608,
283
+ "group_size": 128,
284
  "bias": false,
285
+ "recovery_rank": 0
286
  },
287
  "layers.5.feed_forward.w2": {
288
  "kind": "linear",
289
  "in_features": 4608,
290
  "out_features": 1024,
291
+ "group_size": 128,
292
  "bias": false,
293
+ "recovery_rank": 0
294
  },
295
  "layers.6.conv.in_proj": {
296
  "kind": "linear",
297
  "in_features": 1024,
298
  "out_features": 3072,
299
+ "group_size": 128,
300
  "bias": false,
301
+ "recovery_rank": 0
302
  },
303
  "layers.6.conv.out_proj": {
304
  "kind": "linear",
305
  "in_features": 1024,
306
  "out_features": 1024,
307
+ "group_size": 128,
308
  "bias": false,
309
+ "recovery_rank": 0
310
  },
311
  "layers.6.feed_forward.w1": {
312
  "kind": "linear",
313
  "in_features": 1024,
314
  "out_features": 4608,
315
+ "group_size": 128,
316
  "bias": false,
317
+ "recovery_rank": 0
318
  },
319
  "layers.6.feed_forward.w3": {
320
  "kind": "linear",
321
  "in_features": 1024,
322
  "out_features": 4608,
323
+ "group_size": 128,
324
  "bias": false,
325
+ "recovery_rank": 0
326
  },
327
  "layers.6.feed_forward.w2": {
328
  "kind": "linear",
329
  "in_features": 4608,
330
  "out_features": 1024,
331
+ "group_size": 128,
332
  "bias": false,
333
+ "recovery_rank": 0
334
  },
335
  "layers.7.conv.in_proj": {
336
  "kind": "linear",
337
  "in_features": 1024,
338
  "out_features": 3072,
339
+ "group_size": 128,
340
  "bias": false,
341
+ "recovery_rank": 0
342
  },
343
  "layers.7.conv.out_proj": {
344
  "kind": "linear",
345
  "in_features": 1024,
346
  "out_features": 1024,
347
+ "group_size": 128,
348
  "bias": false,
349
+ "recovery_rank": 0
350
  },
351
  "layers.7.feed_forward.w1": {
352
  "kind": "linear",
353
  "in_features": 1024,
354
  "out_features": 4608,
355
+ "group_size": 128,
356
  "bias": false,
357
+ "recovery_rank": 0
358
  },
359
  "layers.7.feed_forward.w3": {
360
  "kind": "linear",
361
  "in_features": 1024,
362
  "out_features": 4608,
363
+ "group_size": 128,
364
  "bias": false,
365
+ "recovery_rank": 0
366
  },
367
  "layers.7.feed_forward.w2": {
368
  "kind": "linear",
369
  "in_features": 4608,
370
  "out_features": 1024,
371
+ "group_size": 128,
372
  "bias": false,
373
+ "recovery_rank": 0
374
  },
375
  "layers.8.self_attn.q_proj": {
376
  "kind": "linear",
377
  "in_features": 1024,
378
  "out_features": 1024,
379
+ "group_size": 128,
380
  "bias": false,
381
+ "recovery_rank": 0
382
  },
383
  "layers.8.self_attn.k_proj": {
384
  "kind": "linear",
385
  "in_features": 1024,
386
  "out_features": 512,
387
+ "group_size": 128,
388
  "bias": false,
389
+ "recovery_rank": 0
390
  },
391
  "layers.8.self_attn.v_proj": {
392
  "kind": "linear",
393
  "in_features": 1024,
394
  "out_features": 512,
395
+ "group_size": 128,
396
  "bias": false,
397
+ "recovery_rank": 0
398
  },
399
  "layers.8.self_attn.out_proj": {
400
  "kind": "linear",
401
  "in_features": 1024,
402
  "out_features": 1024,
403
+ "group_size": 128,
404
  "bias": false,
405
+ "recovery_rank": 0
406
  },
407
  "layers.8.feed_forward.w1": {
408
  "kind": "linear",
409
  "in_features": 1024,
410
  "out_features": 4608,
411
+ "group_size": 128,
412
  "bias": false,
413
+ "recovery_rank": 0
414
  },
415
  "layers.8.feed_forward.w3": {
416
  "kind": "linear",
417
  "in_features": 1024,
418
  "out_features": 4608,
419
+ "group_size": 128,
420
  "bias": false,
421
+ "recovery_rank": 0
422
  },
423
  "layers.8.feed_forward.w2": {
424
  "kind": "linear",
425
  "in_features": 4608,
426
  "out_features": 1024,
427
+ "group_size": 128,
428
  "bias": false,
429
+ "recovery_rank": 0
430
  },
431
  "layers.9.conv.in_proj": {
432
  "kind": "linear",
433
  "in_features": 1024,
434
  "out_features": 3072,
435
+ "group_size": 128,
436
  "bias": false,
437
+ "recovery_rank": 0
438
  },
439
  "layers.9.conv.out_proj": {
440
  "kind": "linear",
441
  "in_features": 1024,
442
  "out_features": 1024,
443
+ "group_size": 128,
444
  "bias": false,
445
+ "recovery_rank": 0
446
  },
447
  "layers.9.feed_forward.w1": {
448
  "kind": "linear",
449
  "in_features": 1024,
450
  "out_features": 4608,
451
+ "group_size": 128,
452
  "bias": false,
453
+ "recovery_rank": 0
454
  },
455
  "layers.9.feed_forward.w3": {
456
  "kind": "linear",
457
  "in_features": 1024,
458
  "out_features": 4608,
459
+ "group_size": 128,
460
  "bias": false,
461
+ "recovery_rank": 0
462
  },
463
  "layers.9.feed_forward.w2": {
464
  "kind": "linear",
465
  "in_features": 4608,
466
  "out_features": 1024,
467
+ "group_size": 128,
468
  "bias": false,
469
+ "recovery_rank": 0
470
  },
471
  "layers.10.self_attn.q_proj": {
472
  "kind": "linear",
473
  "in_features": 1024,
474
  "out_features": 1024,
475
+ "group_size": 128,
476
  "bias": false,
477
+ "recovery_rank": 0
478
  },
479
  "layers.10.self_attn.k_proj": {
480
  "kind": "linear",
481
  "in_features": 1024,
482
  "out_features": 512,
483
+ "group_size": 128,
484
  "bias": false,
485
+ "recovery_rank": 0
486
  },
487
  "layers.10.self_attn.v_proj": {
488
  "kind": "linear",
489
  "in_features": 1024,
490
  "out_features": 512,
491
+ "group_size": 128,
492
  "bias": false,
493
+ "recovery_rank": 0
494
  },
495
  "layers.10.self_attn.out_proj": {
496
  "kind": "linear",
497
  "in_features": 1024,
498
  "out_features": 1024,
499
+ "group_size": 128,
500
  "bias": false,
501
+ "recovery_rank": 0
502
  },
503
  "layers.10.feed_forward.w1": {
504
  "kind": "linear",
505
  "in_features": 1024,
506
  "out_features": 4608,
507
+ "group_size": 128,
508
  "bias": false,
509
+ "recovery_rank": 0
510
  },
511
  "layers.10.feed_forward.w3": {
512
  "kind": "linear",
513
  "in_features": 1024,
514
  "out_features": 4608,
515
+ "group_size": 128,
516
  "bias": false,
517
+ "recovery_rank": 0
518
  },
519
  "layers.10.feed_forward.w2": {
520
  "kind": "linear",
521
  "in_features": 4608,
522
  "out_features": 1024,
523
+ "group_size": 128,
524
  "bias": false,
525
+ "recovery_rank": 0
526
  },
527
  "layers.11.conv.in_proj": {
528
  "kind": "linear",
529
  "in_features": 1024,
530
  "out_features": 3072,
531
+ "group_size": 128,
532
  "bias": false,
533
+ "recovery_rank": 0
534
  },
535
  "layers.11.conv.out_proj": {
536
  "kind": "linear",
537
  "in_features": 1024,
538
  "out_features": 1024,
539
+ "group_size": 128,
540
  "bias": false,
541
+ "recovery_rank": 0
542
  },
543
  "layers.11.feed_forward.w1": {
544
  "kind": "linear",
545
  "in_features": 1024,
546
  "out_features": 4608,
547
+ "group_size": 128,
548
  "bias": false,
549
+ "recovery_rank": 0
550
  },
551
  "layers.11.feed_forward.w3": {
552
  "kind": "linear",
553
  "in_features": 1024,
554
  "out_features": 4608,
555
+ "group_size": 128,
556
  "bias": false,
557
+ "recovery_rank": 0
558
  },
559
  "layers.11.feed_forward.w2": {
560
  "kind": "linear",
561
  "in_features": 4608,
562
  "out_features": 1024,
563
+ "group_size": 128,
564
  "bias": false,
565
+ "recovery_rank": 0
566
  },
567
  "layers.12.self_attn.q_proj": {
568
  "kind": "linear",
569
  "in_features": 1024,
570
  "out_features": 1024,
571
+ "group_size": 128,
572
  "bias": false,
573
+ "recovery_rank": 0
574
  },
575
  "layers.12.self_attn.k_proj": {
576
  "kind": "linear",
577
  "in_features": 1024,
578
  "out_features": 512,
579
+ "group_size": 128,
580
  "bias": false,
581
+ "recovery_rank": 0
582
  },
583
  "layers.12.self_attn.v_proj": {
584
  "kind": "linear",
585
  "in_features": 1024,
586
  "out_features": 512,
587
+ "group_size": 128,
588
  "bias": false,
589
+ "recovery_rank": 0
590
  },
591
  "layers.12.self_attn.out_proj": {
592
  "kind": "linear",
593
  "in_features": 1024,
594
  "out_features": 1024,
595
+ "group_size": 128,
596
  "bias": false,
597
+ "recovery_rank": 0
598
  },
599
  "layers.12.feed_forward.w1": {
600
  "kind": "linear",
601
  "in_features": 1024,
602
  "out_features": 4608,
603
+ "group_size": 128,
604
  "bias": false,
605
+ "recovery_rank": 0
606
  },
607
  "layers.12.feed_forward.w3": {
608
  "kind": "linear",
609
  "in_features": 1024,
610
  "out_features": 4608,
611
+ "group_size": 128,
612
  "bias": false,
613
+ "recovery_rank": 0
614
  },
615
  "layers.12.feed_forward.w2": {
616
  "kind": "linear",
617
  "in_features": 4608,
618
  "out_features": 1024,
619
+ "group_size": 128,
620
  "bias": false,
621
+ "recovery_rank": 0
622
  },
623
  "layers.13.conv.in_proj": {
624
  "kind": "linear",
625
  "in_features": 1024,
626
  "out_features": 3072,
627
+ "group_size": 128,
628
  "bias": false,
629
+ "recovery_rank": 0
630
  },
631
  "layers.13.conv.out_proj": {
632
  "kind": "linear",
633
  "in_features": 1024,
634
  "out_features": 1024,
635
+ "group_size": 128,
636
  "bias": false,
637
+ "recovery_rank": 0
638
  },
639
  "layers.13.feed_forward.w1": {
640
  "kind": "linear",
641
  "in_features": 1024,
642
  "out_features": 4608,
643
+ "group_size": 128,
644
  "bias": false,
645
+ "recovery_rank": 0
646
  },
647
  "layers.13.feed_forward.w3": {
648
  "kind": "linear",
649
  "in_features": 1024,
650
  "out_features": 4608,
651
+ "group_size": 128,
652
  "bias": false,
653
+ "recovery_rank": 0
654
  },
655
  "layers.13.feed_forward.w2": {
656
  "kind": "linear",
657
  "in_features": 4608,
658
  "out_features": 1024,
659
+ "group_size": 128,
660
  "bias": false,
661
+ "recovery_rank": 0
662
  },
663
  "layers.14.self_attn.q_proj": {
664
  "kind": "linear",
665
  "in_features": 1024,
666
  "out_features": 1024,
667
+ "group_size": 128,
668
  "bias": false,
669
+ "recovery_rank": 0
670
  },
671
  "layers.14.self_attn.k_proj": {
672
  "kind": "linear",
673
  "in_features": 1024,
674
  "out_features": 512,
675
+ "group_size": 128,
676
  "bias": false,
677
+ "recovery_rank": 0
678
  },
679
  "layers.14.self_attn.v_proj": {
680
  "kind": "linear",
681
  "in_features": 1024,
682
  "out_features": 512,
683
+ "group_size": 128,
684
  "bias": false,
685
+ "recovery_rank": 0
686
  },
687
  "layers.14.self_attn.out_proj": {
688
  "kind": "linear",
689
  "in_features": 1024,
690
  "out_features": 1024,
691
+ "group_size": 128,
692
  "bias": false,
693
+ "recovery_rank": 0
694
  },
695
  "layers.14.feed_forward.w1": {
696
  "kind": "linear",
697
  "in_features": 1024,
698
  "out_features": 4608,
699
+ "group_size": 128,
700
  "bias": false,
701
+ "recovery_rank": 0
702
  },
703
  "layers.14.feed_forward.w3": {
704
  "kind": "linear",
705
  "in_features": 1024,
706
  "out_features": 4608,
707
+ "group_size": 128,
708
  "bias": false,
709
+ "recovery_rank": 0
710
  },
711
  "layers.14.feed_forward.w2": {
712
  "kind": "linear",
713
  "in_features": 4608,
714
  "out_features": 1024,
715
+ "group_size": 128,
716
  "bias": false,
717
+ "recovery_rank": 0
718
  },
719
  "layers.15.conv.in_proj": {
720
  "kind": "linear",
721
  "in_features": 1024,
722
  "out_features": 3072,
723
+ "group_size": 128,
724
  "bias": false,
725
+ "recovery_rank": 0
726
  },
727
  "layers.15.conv.out_proj": {
728
  "kind": "linear",
729
  "in_features": 1024,
730
  "out_features": 1024,
731
+ "group_size": 128,
732
  "bias": false,
733
+ "recovery_rank": 0
734
  },
735
  "layers.15.feed_forward.w1": {
736
  "kind": "linear",
737
  "in_features": 1024,
738
  "out_features": 4608,
739
+ "group_size": 128,
740
  "bias": false,
741
+ "recovery_rank": 0
742
  },
743
  "layers.15.feed_forward.w3": {
744
  "kind": "linear",
745
  "in_features": 1024,
746
  "out_features": 4608,
747
+ "group_size": 128,
748
  "bias": false,
749
+ "recovery_rank": 0
750
  },
751
  "layers.15.feed_forward.w2": {
752
  "kind": "linear",
753
  "in_features": 4608,
754
  "out_features": 1024,
755
+ "group_size": 128,
756
  "bias": false,
757
+ "recovery_rank": 0
758
  }
759
  },
760
+ "weight_bytes": 94298568
761
  }
model.safetensors CHANGED
@@ -1,3 +1,3 @@
1
  version https://git-lfs.github.com/spec/v1
2
- oid sha256:97e0d48c47b275d16ebb90b3174cf1d8d0f90d680f5ceaac17890ffd4df5f39f
3
- size 223916888
 
1
  version https://git-lfs.github.com/spec/v1
2
+ oid sha256:cc8bc0fc09aebab0adab006325f30e19703cf9babda0829285223bbe0b0b44ea
3
+ size 94298568
probe_report.json DELETED
@@ -1,33 +0,0 @@
1
- {
2
- "hidden_cosine": 0.8490977564318156,
3
- "hidden_relative_l2": 0.5876564026615595,
4
- "mlm_top1_agreement": 0.9966722129783694,
5
- "mlm_positions": 601.0,
6
- "measurements": [
7
- {
8
- "length": 128,
9
- "base_ms": 8.843307005008683,
10
- "minima_ms": 31.13792001386173,
11
- "speedup": 0.28400442293743094
12
- },
13
- {
14
- "length": 512,
15
- "base_ms": 8.5014560027048,
16
- "minima_ms": 39.459095016354695,
17
- "speedup": 0.21544984747321708
18
- },
19
- {
20
- "length": 2048,
21
- "base_ms": 8.611652010586113,
22
- "minima_ms": 137.49787202687003,
23
- "speedup": 0.06263116573108282
24
- },
25
- {
26
- "length": 8192,
27
- "base_ms": 36.73721800441854,
28
- "minima_ms": 569.83526199474,
29
- "speedup": 0.06446989236120254
30
- }
31
- ],
32
- "gpu": "NVIDIA H200"
33
- }
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
training_report.json CHANGED
The diff for this file is too large to render. See raw diff