| # BananaMind-KV1-v2-1.2B-Liquid-Experimental-Instruct |
|
|
| 2-bit KV-cache quantization-aware training (KV1 v2 recipe: Hadamard rotation, 4 fp16 sink tokens, logits + hidden-state distillation) of [LiquidAI/LFM2.5-1.2B-Instruct](https://huggingface.co/LiquidAI/LFM2.5-1.2B-Instruct) |
| by self-distillation: the frozen original model with fp KV is the teacher, the loss is KL(teacher || student) on every |
| token. Input text mix: UltraData-SFT-2605 (no_think, rendered with the Instruct chat template) 60%, UltraData-SFT-Agent-2609 (tool-use trajectories) 16%, Ultra-FineWeb en 7%, UltraData-Code L2 12%, UltraData-Math L2-preview 5%. |
| |
| * Checkpoint: step 2,862, 0.750B / 0.75B tokens |
| * KV quantization: post-RoPE K and V, per token x per KV head over head_dim=64, 4 levels (2-bit), mode=`affine`, |
| Hadamard rotation=True, STE. 2.500 bits/value incl. fp16 metadata (6.40x vs fp16 KV). |
| * Only the 6 GQA layers have a KV cache; LFM2 conv layers are untouched. |
| * Weights are standard LFM2 (bf16). Loading normally gives fp KV; to run with the ternary KV path: |
| |
| ```python |
| import kv_ternary as kvt # file in this repo |
| kvt.configure(levels=4, mode="affine", hadamard=True, sink=4) |
| model = AutoModelForCausalLM.from_pretrained("BananaMind/BananaMind-KV1-v2-1.2B-Liquid-Experimental-Instruct", attn_implementation=kvt.register()) |
| ``` |
| |
| ## Held-out KL to the fp-KV teacher, per domain (this checkpoint) |
| ```json |
| { |
| "groups": { |
| "chat": { |
| "kld_ternary_bits": 0.2494346898506812, |
| "kld_fp_bits": 0.1419860143176317, |
| "ce_teacher": 2.2847921764478087, |
| "ce_student_ternary": 2.125999036245048, |
| "eval_tokens": 262144, |
| "top1": 83.85467529296875, |
| "top1_conf": 99.1996169106581 |
| }, |
| "agent": { |
| "kld_ternary_bits": 0.33562924788600773, |
| "kld_fp_bits": 0.2308978811055062, |
| "ce_teacher": 2.1201335513032973, |
| "ce_student_ternary": 1.8623797902837396, |
| "eval_tokens": 131072, |
| "top1": 83.07723999023438, |
| "top1_conf": 98.4333874450346 |
| }, |
| "web": { |
| "kld_ternary_bits": 0.30666880462890567, |
| "kld_fp_bits": 0.17421943807342427, |
| "ce_teacher": 2.7278448045253754, |
| "ce_student_ternary": 2.7597829401493073, |
| "eval_tokens": 131072, |
| "top1": 77.75039672851562, |
| "top1_conf": 99.12357581069237 |
| }, |
| "code": { |
| "kld_ternary_bits": 0.32928879322411847, |
| "kld_fp_bits": 0.19199722187913681, |
| "ce_teacher": 1.5456873746588826, |
| "ce_student_ternary": 1.5028161080554128, |
| "eval_tokens": 131072, |
| "top1": 83.27713012695312, |
| "top1_conf": 98.53736973932403 |
| }, |
| "math": { |
| "kld_ternary_bits": 0.29759900914698034, |
| "kld_fp_bits": 0.17996008644524872, |
| "ce_teacher": 2.1323286555707455, |
| "ce_student_ternary": 2.083978658542037, |
| "eval_tokens": 131072, |
| "top1": 80.85784912109375, |
| "top1_conf": 98.96853511148156 |
| } |
| }, |
| "kld_ternary_bits": 0.2946758724312291, |
| "kld_fp_bits": 0.17684110935642988, |
| "ce_teacher": 2.1825964564923197, |
| "ce_student_ternary": 2.0768259282534323, |
| "eval_tokens": 786432, |
| "top1": 82.11199442545573, |
| "top1_conf": 98.86640786462193 |
| } |
| ``` |
| |
| ## Baseline (original Instruct weights, before training) |
| ```json |
| { |
| "groups": { |
| "chat": { |
| "kld_ternary_bits": 1.0920360561536933, |
| "kld_fp_bits": 0.002457815578676199, |
| "ce_teacher": 2.2847921764478087, |
| "ce_student_ternary": 2.555535623803735, |
| "eval_tokens": 262144, |
| "top1": 69.39353942871094, |
| "top1_conf": 91.60110822273909 |
| }, |
| "agent": { |
| "kld_ternary_bits": 1.5093662310168983, |
| "kld_fp_bits": 0.0024159978223440575, |
| "ce_teacher": 2.1201335513032973, |
| "ce_student_ternary": 2.3469435470178723, |
| "eval_tokens": 131072, |
| "top1": 65.64254760742188, |
| "top1_conf": 87.50492521552744 |
| }, |
| "web": { |
| "kld_ternary_bits": 0.6309144172280389, |
| "kld_fp_bits": 0.0013030000546159451, |
| "ce_teacher": 2.7278448045253754, |
| "ce_student_ternary": 2.98703445866704, |
| "eval_tokens": 131072, |
| "top1": 68.902587890625, |
| "top1_conf": 96.38475021910605 |
| }, |
| "code": { |
| "kld_ternary_bits": 1.006716521157041, |
| "kld_fp_bits": 0.0012283830409732235, |
| "ce_teacher": 1.5456873746588826, |
| "ce_student_ternary": 2.0545026026666164, |
| "eval_tokens": 131072, |
| "top1": 70.14541625976562, |
| "top1_conf": 89.01945019944958 |
| }, |
| "math": { |
| "kld_ternary_bits": 0.8196465303417501, |
| "kld_fp_bits": 0.0012562345679355744, |
| "ce_teacher": 2.1323286555707455, |
| "ce_student_ternary": 2.5224284287542105, |
| "eval_tokens": 131072, |
| "top1": 68.45626831054688, |
| "top1_conf": 91.1075224005001 |
| } |
| }, |
| "kld_ternary_bits": 1.0251193020085192, |
| "kld_fp_bits": 0.0018532077738701996, |
| "ce_teacher": 2.1825964564923197, |
| "ce_student_ternary": 2.5036633807855346, |
| "eval_tokens": 786432, |
| "top1": 68.65564982096355, |
| "top1_conf": 90.50192142560965 |
| } |
| ``` |
| |
| Training: KD temperature 1.0, seq_len 32768, 262,144 tokens/step, AdamW lr 2e-05 cosine->2e-06, |
| warmup 100, wd 0.1, quant ramp 100 steps, 1x RTX Pro 6000. |
| License follows the base model (LFM Open License). |
| |
