# BananaMind-KV1-v2-1.2B-Liquid-Experimental-Instruct

    2-bit KV-cache quantization-aware training (KV1 v2 recipe: Hadamard rotation, 4 fp16 sink tokens, logits + hidden-state distillation) of [LiquidAI/LFM2.5-1.2B-Instruct](https://huggingface.co/LiquidAI/LFM2.5-1.2B-Instruct)
    by self-distillation: the frozen original model with fp KV is the teacher, the loss is KL(teacher || student) on every
    token. Input text mix: UltraData-SFT-2605 (no_think, rendered with the Instruct chat template) 60%, UltraData-SFT-Agent-2609 (tool-use trajectories) 16%, Ultra-FineWeb en 7%, UltraData-Code L2 12%, UltraData-Math L2-preview 5%.

    * Checkpoint: step 2,862, 0.750B / 0.75B tokens
    * KV quantization: post-RoPE K and V, per token x per KV head over head_dim=64, 4 levels (2-bit), mode=`affine`,
      Hadamard rotation=True, STE. 2.500 bits/value incl. fp16 metadata (6.40x vs fp16 KV).
    * Only the 6 GQA layers have a KV cache; LFM2 conv layers are untouched.
    * Weights are standard LFM2 (bf16). Loading normally gives fp KV; to run with the ternary KV path:

    ```python
    import kv_ternary as kvt   # file in this repo
    kvt.configure(levels=4, mode="affine", hadamard=True, sink=4)
    model = AutoModelForCausalLM.from_pretrained("BananaMind/BananaMind-KV1-v2-1.2B-Liquid-Experimental-Instruct", attn_implementation=kvt.register())
    ```

    ## Held-out KL to the fp-KV teacher, per domain (this checkpoint)
    ```json
    {
  "groups": {
    "chat": {
      "kld_ternary_bits": 0.2494346898506812,
      "kld_fp_bits": 0.1419860143176317,
      "ce_teacher": 2.2847921764478087,
      "ce_student_ternary": 2.125999036245048,
      "eval_tokens": 262144,
      "top1": 83.85467529296875,
      "top1_conf": 99.1996169106581
    },
    "agent": {
      "kld_ternary_bits": 0.33562924788600773,
      "kld_fp_bits": 0.2308978811055062,
      "ce_teacher": 2.1201335513032973,
      "ce_student_ternary": 1.8623797902837396,
      "eval_tokens": 131072,
      "top1": 83.07723999023438,
      "top1_conf": 98.4333874450346
    },
    "web": {
      "kld_ternary_bits": 0.30666880462890567,
      "kld_fp_bits": 0.17421943807342427,
      "ce_teacher": 2.7278448045253754,
      "ce_student_ternary": 2.7597829401493073,
      "eval_tokens": 131072,
      "top1": 77.75039672851562,
      "top1_conf": 99.12357581069237
    },
    "code": {
      "kld_ternary_bits": 0.32928879322411847,
      "kld_fp_bits": 0.19199722187913681,
      "ce_teacher": 1.5456873746588826,
      "ce_student_ternary": 1.5028161080554128,
      "eval_tokens": 131072,
      "top1": 83.27713012695312,
      "top1_conf": 98.53736973932403
    },
    "math": {
      "kld_ternary_bits": 0.29759900914698034,
      "kld_fp_bits": 0.17996008644524872,
      "ce_teacher": 2.1323286555707455,
      "ce_student_ternary": 2.083978658542037,
      "eval_tokens": 131072,
      "top1": 80.85784912109375,
      "top1_conf": 98.96853511148156
    }
  },
  "kld_ternary_bits": 0.2946758724312291,
  "kld_fp_bits": 0.17684110935642988,
  "ce_teacher": 2.1825964564923197,
  "ce_student_ternary": 2.0768259282534323,
  "eval_tokens": 786432,
  "top1": 82.11199442545573,
  "top1_conf": 98.86640786462193
}
    ```

    ## Baseline (original Instruct weights, before training)
    ```json
    {
  "groups": {
    "chat": {
      "kld_ternary_bits": 1.0920360561536933,
      "kld_fp_bits": 0.002457815578676199,
      "ce_teacher": 2.2847921764478087,
      "ce_student_ternary": 2.555535623803735,
      "eval_tokens": 262144,
      "top1": 69.39353942871094,
      "top1_conf": 91.60110822273909
    },
    "agent": {
      "kld_ternary_bits": 1.5093662310168983,
      "kld_fp_bits": 0.0024159978223440575,
      "ce_teacher": 2.1201335513032973,
      "ce_student_ternary": 2.3469435470178723,
      "eval_tokens": 131072,
      "top1": 65.64254760742188,
      "top1_conf": 87.50492521552744
    },
    "web": {
      "kld_ternary_bits": 0.6309144172280389,
      "kld_fp_bits": 0.0013030000546159451,
      "ce_teacher": 2.7278448045253754,
      "ce_student_ternary": 2.98703445866704,
      "eval_tokens": 131072,
      "top1": 68.902587890625,
      "top1_conf": 96.38475021910605
    },
    "code": {
      "kld_ternary_bits": 1.006716521157041,
      "kld_fp_bits": 0.0012283830409732235,
      "ce_teacher": 1.5456873746588826,
      "ce_student_ternary": 2.0545026026666164,
      "eval_tokens": 131072,
      "top1": 70.14541625976562,
      "top1_conf": 89.01945019944958
    },
    "math": {
      "kld_ternary_bits": 0.8196465303417501,
      "kld_fp_bits": 0.0012562345679355744,
      "ce_teacher": 2.1323286555707455,
      "ce_student_ternary": 2.5224284287542105,
      "eval_tokens": 131072,
      "top1": 68.45626831054688,
      "top1_conf": 91.1075224005001
    }
  },
  "kld_ternary_bits": 1.0251193020085192,
  "kld_fp_bits": 0.0018532077738701996,
  "ce_teacher": 2.1825964564923197,
  "ce_student_ternary": 2.5036633807855346,
  "eval_tokens": 786432,
  "top1": 68.65564982096355,
  "top1_conf": 90.50192142560965
}
    ```

    Training: KD temperature 1.0, seq_len 32768, 262,144 tokens/step, AdamW lr 2e-05 cosine->2e-06,
    warmup 100, wd 0.1, quant ramp 100 steps, 1x RTX Pro 6000.
    License follows the base model (LFM Open License).