| --- |
| license: apache-2.0 |
| base_model: IFM/K2-Horizon-MoVA-36B-A4B |
| base_model_relation: finetune |
| library_name: transformers |
| pipeline_tag: text-generation |
| tags: |
| - moe |
| - continued-pretraining |
| - research |
| --- |
| |
| # K2H c4-ctrl |
|
|
| The recipe control arm of a continued-pretraining study on IFM/K2-Horizon-MoVA-36B-A4B: clean domain data plus script-verified math replay. |
|
|
| Research checkpoint, published 2026-11-03. It is not a release candidate. Compare it with the release, not in isolation. |
|
|
| ## Model |
|
|
| - Base: [IFM/K2-Horizon-MoVA-36B-A4B](https://huggingface.co/IFM/K2-Horizon-MoVA-36B-A4B) (36B total, about 4B active, MoE). Same architecture, tokenizer and chat template. |
| - Weights: bf16 safetensors. The custom modeling code ships with the weights, so load with `trust_remote_code=True`. |
| - Sampling as the base model card recommends: temperature 1.0, top_p 0.95. |
| |
| ## Training |
| |
| - Trainer: xLLM, FSDP2, bf16, 3 GPUs. |
| - Start: the released weights of IFM/K2-Horizon-MoVA-36B-A4B. |
| - 100 steps. Sequence length 65,536, batch 1 per GPU, gradient accumulation 4: about 786k tokens per step, about 79M tokens in total. |
| - AdamW, beta1 0.9, beta2 0.95, weight decay 0.06, gradient clip 1.0. |
| - Learning rate 1.5e-5, 20 warmup steps, then constant. |
| - Chat records, where present, are trained on assistant tokens only. |
| |
| Data mix (sampling weights): |
| |
| - 44.5%: domain slices from [Avifenesh/k2h-corpus-domain-clean](https://huggingface.co/datasets/Avifenesh/k2h-corpus-domain-clean) (inference, framework, general and Linux code bundles, commits) plus pull-request text and a small slice of tuning notes that are not published. |
| - 47.7%: `replay-synth-verified` and 7.8%: `replay-math-verified` from [Avifenesh/k2h-replay-verified](https://huggingface.co/datasets/Avifenesh/k2h-replay-verified), as raw text. |
| |
| ## Evaluation |
| |
| All numbers come from the same harness as the release column (lm-evaluation-harness against a vLLM server). IFEval and chat GSM8K use the chat template with reasoning on, temperature 1.0, top_p 0.95, 14,336 new tokens, and score only the text after the reasoning block. The PR NLL is the mean token NLL on 150 held-out pull requests (530k tokens). |
|
|
| | Task | Release | This model | Change | |
| |---|---:|---:|---:| |
| | GSM8K, 5-shot, strict | 90.1 | 89.3 | -0.8 | |
| | GSM8K, chat zero-shot (reasoning on) | 96.6 | 86.7 | -9.9 | |
| | IFEval, prompt-strict (reasoning on) | 84.3 | 82.6 | -1.7 | |
| | MMLU | 77.3 | 76.9 | -0.4 | |
| | ARC-Challenge | 57.6 | 57.4 | -0.2 | |
| | ARC-Easy | 85.6 | 85.3 | -0.3 | |
| | BoolQ | 88.4 | 88.5 | +0.1 | |
| | HellaSwag (norm) | 67.6 | 68.2 | +0.6 | |
| | PIQA | 80.5 | 80.6 | +0.1 | |
| | WinoGrande | 69.1 | 69.8 | +0.7 | |
| | MBPP pass@1 | 65.2 | 71.6 | +6.4 | |
| | HumanEval pass@1 (4 samples) | 53.0 | 65.9 | +12.9 | |
| | Held-out PR NLL (lower is better) | 1.519 | 1.325 | -0.194 | |
|
|
| What the numbers say: |
|
|
| - Every general task stays level with the release except chat-format GSM8K, and coding goes up (MBPP +6.4, HumanEval +12.9). The held-out PR NLL drops from 1.519 to 1.325. |
| - IFEval 82.6 vs 84.3 is level (p=0.35, paired). |
| - Chat-format GSM8K falls 10 points: 140 of 1,319 replies end early inside the reasoning block. The cause is the math replay trained as raw text, which teaches the document-end token where a chat turn should end. Few-shot GSM8K hides this. |
| - The same data at LR 5e-6 kept chat GSM8K level (96.3). Use that learning rate, or the chat-format replay, for this data. |
|
|
| ## License |
|
|
| Apache-2.0, the license of the base model. |
|
|
