File size: 5,274 Bytes
6f9e1a0 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 | # BananaMind-KV1-v2-1.2B-Liquid-Experimental-Instruct
2-bit KV-cache quantization-aware training (KV1 v2 recipe: Hadamard rotation, 4 fp16 sink tokens, logits + hidden-state distillation) of [LiquidAI/LFM2.5-1.2B-Instruct](https://huggingface.co/LiquidAI/LFM2.5-1.2B-Instruct)
by self-distillation: the frozen original model with fp KV is the teacher, the loss is KL(teacher || student) on every
token. Input text mix: UltraData-SFT-2605 (no_think, rendered with the Instruct chat template) 60%, UltraData-SFT-Agent-2609 (tool-use trajectories) 16%, Ultra-FineWeb en 7%, UltraData-Code L2 12%, UltraData-Math L2-preview 5%.
* Checkpoint: step 2,862, 0.750B / 0.75B tokens
* KV quantization: post-RoPE K and V, per token x per KV head over head_dim=64, 4 levels (2-bit), mode=`affine`,
Hadamard rotation=True, STE. 2.500 bits/value incl. fp16 metadata (6.40x vs fp16 KV).
* Only the 6 GQA layers have a KV cache; LFM2 conv layers are untouched.
* Weights are standard LFM2 (bf16). Loading normally gives fp KV; to run with the ternary KV path:
```python
import kv_ternary as kvt # file in this repo
kvt.configure(levels=4, mode="affine", hadamard=True, sink=4)
model = AutoModelForCausalLM.from_pretrained("BananaMind/BananaMind-KV1-v2-1.2B-Liquid-Experimental-Instruct", attn_implementation=kvt.register())
```
## Held-out KL to the fp-KV teacher, per domain (this checkpoint)
```json
{
"groups": {
"chat": {
"kld_ternary_bits": 0.2494346898506812,
"kld_fp_bits": 0.1419860143176317,
"ce_teacher": 2.2847921764478087,
"ce_student_ternary": 2.125999036245048,
"eval_tokens": 262144,
"top1": 83.85467529296875,
"top1_conf": 99.1996169106581
},
"agent": {
"kld_ternary_bits": 0.33562924788600773,
"kld_fp_bits": 0.2308978811055062,
"ce_teacher": 2.1201335513032973,
"ce_student_ternary": 1.8623797902837396,
"eval_tokens": 131072,
"top1": 83.07723999023438,
"top1_conf": 98.4333874450346
},
"web": {
"kld_ternary_bits": 0.30666880462890567,
"kld_fp_bits": 0.17421943807342427,
"ce_teacher": 2.7278448045253754,
"ce_student_ternary": 2.7597829401493073,
"eval_tokens": 131072,
"top1": 77.75039672851562,
"top1_conf": 99.12357581069237
},
"code": {
"kld_ternary_bits": 0.32928879322411847,
"kld_fp_bits": 0.19199722187913681,
"ce_teacher": 1.5456873746588826,
"ce_student_ternary": 1.5028161080554128,
"eval_tokens": 131072,
"top1": 83.27713012695312,
"top1_conf": 98.53736973932403
},
"math": {
"kld_ternary_bits": 0.29759900914698034,
"kld_fp_bits": 0.17996008644524872,
"ce_teacher": 2.1323286555707455,
"ce_student_ternary": 2.083978658542037,
"eval_tokens": 131072,
"top1": 80.85784912109375,
"top1_conf": 98.96853511148156
}
},
"kld_ternary_bits": 0.2946758724312291,
"kld_fp_bits": 0.17684110935642988,
"ce_teacher": 2.1825964564923197,
"ce_student_ternary": 2.0768259282534323,
"eval_tokens": 786432,
"top1": 82.11199442545573,
"top1_conf": 98.86640786462193
}
```
## Baseline (original Instruct weights, before training)
```json
{
"groups": {
"chat": {
"kld_ternary_bits": 1.0920360561536933,
"kld_fp_bits": 0.002457815578676199,
"ce_teacher": 2.2847921764478087,
"ce_student_ternary": 2.555535623803735,
"eval_tokens": 262144,
"top1": 69.39353942871094,
"top1_conf": 91.60110822273909
},
"agent": {
"kld_ternary_bits": 1.5093662310168983,
"kld_fp_bits": 0.0024159978223440575,
"ce_teacher": 2.1201335513032973,
"ce_student_ternary": 2.3469435470178723,
"eval_tokens": 131072,
"top1": 65.64254760742188,
"top1_conf": 87.50492521552744
},
"web": {
"kld_ternary_bits": 0.6309144172280389,
"kld_fp_bits": 0.0013030000546159451,
"ce_teacher": 2.7278448045253754,
"ce_student_ternary": 2.98703445866704,
"eval_tokens": 131072,
"top1": 68.902587890625,
"top1_conf": 96.38475021910605
},
"code": {
"kld_ternary_bits": 1.006716521157041,
"kld_fp_bits": 0.0012283830409732235,
"ce_teacher": 1.5456873746588826,
"ce_student_ternary": 2.0545026026666164,
"eval_tokens": 131072,
"top1": 70.14541625976562,
"top1_conf": 89.01945019944958
},
"math": {
"kld_ternary_bits": 0.8196465303417501,
"kld_fp_bits": 0.0012562345679355744,
"ce_teacher": 2.1323286555707455,
"ce_student_ternary": 2.5224284287542105,
"eval_tokens": 131072,
"top1": 68.45626831054688,
"top1_conf": 91.1075224005001
}
},
"kld_ternary_bits": 1.0251193020085192,
"kld_fp_bits": 0.0018532077738701996,
"ce_teacher": 2.1825964564923197,
"ce_student_ternary": 2.5036633807855346,
"eval_tokens": 786432,
"top1": 68.65564982096355,
"top1_conf": 90.50192142560965
}
```
Training: KD temperature 1.0, seq_len 32768, 262,144 tokens/step, AdamW lr 2e-05 cosine->2e-06,
warmup 100, wd 0.1, quant ramp 100 steps, 1x RTX Pro 6000.
License follows the base model (LFM Open License).
|