Lucasgitguy-co commited on
Commit
6802dff
·
verified ·
1 Parent(s): edc1886

Upload README.md with huggingface_hub

Browse files
Files changed (1) hide show
  1. README.md +121 -0
README.md ADDED
@@ -0,0 +1,121 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ license: apache-2.0
3
+ base_model: IFM/K2-Horizon-0.9B
4
+ base_model_relation: finetune
5
+ pipeline_tag: text-generation
6
+ library_name: transformers
7
+ language:
8
+ - en
9
+ tags:
10
+ - k2-horizon
11
+ - system-1
12
+ - no-thinking
13
+ - self-distillation
14
+ - lora
15
+ - mlx
16
+ datasets:
17
+ - Lucasgitguy-co/k2-horizon-system1-distill
18
+ ---
19
+
20
+ # K2-Horizon-0.9B-System1
21
+
22
+ [IFM/K2-Horizon-0.9B](https://huggingface.co/IFM/K2-Horizon-0.9B) is a reasoning model: it writes a long `<ifm|think>` block before every answer, and its card asks for 32K output tokens. This fine-tune is the **System 1** version. It answers right away, with no think block, and shows a line or two of working only when the task needs it. On math it uses about **5× fewer tokens** than the base model in no-think mode at the same MATH-500 accuracy.
23
+
24
+ The original thinking modes are still there. `reasoning_effort` now accepts `none` (the new default), `low`, `medium` and `high`.
25
+
26
+ | Variant | Repo |
27
+ | --- | --- |
28
+ | Safetensors (this repo, same layout and remote code as the base) | `Lucasgitguy-co/K2-Horizon-0.9B-System1` |
29
+ | MLX 8-bit / 4-bit | [`-MLX-8bit`](https://huggingface.co/Lucasgitguy-co/K2-Horizon-0.9B-System1-MLX-8bit) · [`-MLX-4bit`](https://huggingface.co/Lucasgitguy-co/K2-Horizon-0.9B-System1-MLX-4bit) |
30
+ | GGUF Q8_0 / Q4_K_M (llama.cpp, Ollama, LM Studio) | [`-GGUF`](https://huggingface.co/Lucasgitguy-co/K2-Horizon-0.9B-System1-GGUF) |
31
+ | Training data | [`k2-horizon-system1-distill`](https://huggingface.co/datasets/Lucasgitguy-co/k2-horizon-system1-distill) |
32
+
33
+ ## Results
34
+
35
+ All rows are the same model family in **no-think mode** (`reasoning_effort="none"`): the base model with an empty think block prefilled, and this fine-tune. 684 held-out prompts, temperature 0.6, top-p 0.95, 2,048-token cap, one sample each.
36
+
37
+ | | GSM8K (200) | MATH-500 (100) | HumanEval+ (164) | IFEval strict (200) | Chat (20) |
38
+ | --- | ---: | ---: | ---: | ---: | ---: |
39
+ | K2-Horizon-0.9B, no-think | 68.5 | 45.0 | **51.8** | **52.0** | 100 |
40
+ | **K2-Horizon-0.9B-System1** | **71.5** | 45.0 | 48.2 | 47.0 | 100 |
41
+
42
+ Median output tokens:
43
+
44
+ | | GSM8K | MATH-500 | HumanEval+ | IFEval | Chat |
45
+ | --- | ---: | ---: | ---: | ---: | ---: |
46
+ | K2-Horizon-0.9B, no-think | 322 | 1,580 | 219 | 105 | 36 |
47
+ | **K2-Horizon-0.9B-System1** | **183** | **317** | **161** | 215 | 113 |
48
+
49
+ The untrained base in no-think mode doesn't really answer fast. On math it just moves its reasoning into the visible answer, leaking `<ifm|think>` tags in 72 of 684 answers, and 9.5% of answers hit the 2,048-token cap. After training, leaks drop to 3 and truncations to 2.9%.
50
+
51
+ **System 2 check** (high effort, 30 math prompts, 6,144-token cap): thinking still works. Every answer still has a think block. GSM8K is 14/15 for both base and fine-tune. MATH-500 is 12/15 base vs 11/15 fine-tune, which is within noise on 15 items. Median output was 970 tokens base vs 857 fine-tune, and 1 vs 3 answers hit the cap. On hard problems high effort is much stronger than System 1, so keep using it there.
52
+
53
+ <details>
54
+ <summary>How the numbers were measured</summary>
55
+
56
+ - GSM8K: random 200 from the test split. MATH-500: random 100 of HuggingFaceH4/MATH-500. Answers are checked with `math-verify` on the last `\boxed{}`.
57
+ - HumanEval+: all 164 evalplus tasks; the extracted code is run against the plus tests.
58
+ - IFEval: random 200 of google/IFEval, scored with the official google-research checker, strict prompt-level accuracy.
59
+ - Chat: 20 everyday prompts, counted as a pass if the answer is non-empty and not a repetition loop.
60
+ - Scoring code and raw outputs: `pipeline/` in this repo.
61
+
62
+ These are small samples. A 100-item MATH-500 slice has roughly ±5 points of noise.
63
+ </details>
64
+
65
+ ## How it was made
66
+
67
+ This is self-distillation of System 2 into System 1, following *Distilling System 2 into System 1* (Yu et al., 2024) plus Qwen3-style thinking-mode fusion. Everything ran on one MacBook Air M4 with 16 GB, using MLX.
68
+
69
+ 1. **Port and check.** With this config the k2_horizon code path is plain Llama (YaRN ×16 RoPE). It was loaded in mlx-lm, and its logits match the original Transformers remote code in fp32: max |Δ| 2.4e-4 on a 16.8K-token prompt, 100% top-1 agreement (`pipeline/parity.json`).
70
+ 2. **Teacher.** The base model answered train-split prompts at `reasoning_effort="high"` with up to 8K thinking tokens. Only verified answers were kept: math checked against ground truth, code passing its tests, instruction-following passing the constraint checkers. Truncated thinking was dropped.
71
+ 3. **Targets.** About 85% of examples are System 1: the prompt, then an empty think block, then the teacher's final answer (the part after thinking, so brief working plus the result). About 15% keep the full thinking, so high effort still works.
72
+ 4. **LoRA.** Rank 16, scale 2, on all attention and MLP projections of all 28 layers. lr 1e-4, cosine schedule, 2 epochs, loss on assistant tokens only. Fused into the weights afterwards.
73
+
74
+ The release uses the 142-example set (134 train, 8 validation, mostly math). Two other variants scored worse on average and were not released:
75
+
76
+ - adding 1,000 self-distilled no-think instruction/code/chat answers: MATH-500 fell to 33 because answers got too terse;
77
+ - a 50/50 weight merge of the two: in between, on average.
78
+
79
+ ## Usage
80
+
81
+ ```python
82
+ from transformers import AutoModelForCausalLM, AutoTokenizer
83
+
84
+ model_id = "Lucasgitguy-co/K2-Horizon-0.9B-System1"
85
+ tok = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True)
86
+ model = AutoModelForCausalLM.from_pretrained(model_id, dtype="bfloat16", trust_remote_code=True)
87
+
88
+ msgs = [{"role": "user", "content": "What is 17*23?"}]
89
+ inputs = tok.apply_chat_template(msgs, add_generation_prompt=True, return_dict=True, return_tensors="pt")
90
+ inputs.pop("token_type_ids", None)
91
+ out = model.generate(**inputs, max_new_tokens=512, do_sample=True, temperature=0.6, top_p=0.95)
92
+ print(tok.decode(out[0, inputs["input_ids"].shape[1]:], skip_special_tokens=True))
93
+ # reasoning_effort="high" in apply_chat_template switches back to full thinking
94
+ ```
95
+
96
+ MLX:
97
+
98
+ ```bash
99
+ pip install mlx-lm
100
+ mlx_lm.generate --model Lucasgitguy-co/K2-Horizon-0.9B-System1-MLX-8bit --prompt "What is 17*23?"
101
+ ```
102
+
103
+ vLLM and SGLang: use the same recipes as the base model card. With the `k2_horizon` reasoning parser, no-think answers come back with an empty `reasoning_content`.
104
+
105
+ The model needs Transformers 5.x for the remote code (the base card validated 5.14–5.16). Generation should stop on `<|ifm|im_end|>` (id 64019) as well as `<|endoftext|>`; `generation_config.json` already lists both.
106
+
107
+ ## Limitations
108
+
109
+ - **Trade-off.** Instruction-following (IFEval −5) and code (HumanEval+ −3.6) are slightly below the base in no-think mode, and chat answers are a bit longer. The training set is small and mostly math. More verified non-math teacher data is the obvious next step.
110
+ - **Hard problems.** For contest math, the full thinking mode (`reasoning_effort="high"`) is still far stronger. System 1 is for quick answers.
111
+ - **Inherited.** All limitations of K2-Horizon-0.9B carry over: it can be wrong, biased or unsafe, and should be evaluated before you rely on it.
112
+
113
+ ## Credits and licence
114
+
115
+ - Base model: [IFM/K2-Horizon-0.9B](https://huggingface.co/IFM/K2-Horizon-0.9B) (Apache-2.0) by the IFM team. Its configuration and modeling code are included unchanged.
116
+ - Method: Yu et al., [*Distilling System 2 into System 1*](https://arxiv.org/abs/2407.06023) (2024); the Qwen3 technical report (thinking-mode fusion).
117
+ - Tools: [MLX / mlx-lm](https://github.com/ml-explore/mlx-lm), [llama.cpp](https://github.com/ggml-org/llama.cpp), [math-verify](https://github.com/huggingface/Math-Verify).
118
+ - Verifiers: the IFEval checker from google-research and `if_functions.py` from allenai/open-instruct (both Apache-2.0).
119
+ - Prompt sources and their licences are listed in the dataset card.
120
+
121
+ Released under Apache-2.0, like the base model.