--- license: other license_name: kailu-3-license license_link: https://huggingface.co//KaiLu-3/blob/main/LICENSE language: - en library_name: transformers pipeline_tag: text-generation inference: false tags: - kailu - kvchub - small-language-model - hybrid - short-convolution - lfm2 - conversational datasets: - HuggingFaceFW/fineweb-edu - HuggingFaceTB/smollm-corpus - DKYoon/SlimPajama-6B - HuggingFaceTB/finemath - wikimedia/wikipedia - OpenCoder-LLM/opc-annealing-corpus - HuggingFaceTB/smol-smoltalk - HuggingFaceH4/ultrafeedback_binarized - nvidia/HelpSteer2 - Anthropic/hh-rlhf --- # KaiLu 3 **A small language model with a big brain. Built from scratch by [KvcHub](https://kvchub.com).** KaiLu 3 is a 329M-parameter chat model built from scratch by [KvcHub](https://kvchub.com). It has a hybrid brain: 16 fast gated short-convolution layers read nearby words, and 8 attention layers connect the whole text. It read 6.6B tokens of textbooks, web pages, math, Wikipedia and code, then learned to chat and to prefer the answers people like. ## Model details | | | |---|---| | Parameters | 329,353,216 | | Layers | 24: 16 gated short-convolution (kernel 3), 8 attention (layers 3, 6, 9, 12, 15, 18, 21, 23) | | Width | hidden 1,024, SwiGLU feed-forward 2,816 | | Attention | 16 query heads, 4 key-value heads (grouped-query), QK-norm, RoPE θ = 1,000,000 | | Context | 1,024 tokens | | Vocabulary | 32,768 tokens (byte-level BPE) | | Embeddings | input and output shared | | Weights | bfloat16, safetensors | | Format | `Lfm2ForCausalLM` in 🤗 Transformers (4.54+). KaiLu 3 shares LFM2's layer layout only: it was trained from scratch and contains no LFM2 weights. | ## How it was made 1. **Pretraining:** 6.6B tokens, mixed as: educational web (FineWeb-Edu) 50%, synthetic textbooks (Cosmopedia v2) 18%, general web and books (SlimPajama-6B, without Books3) 12%, math (FineMath 4+) 8%, Wikipedia 6%, code (OpenCoder annealing) 6%. 2. **Supervised fine-tuning:** about 460K conversations from smol-smoltalk, one pass, learning only from the assistant replies. 3. **Preference tuning (DPO, β = 0.1):** answer pairs from UltraFeedback (binarized), HelpSteer2 preferences and Anthropic HH-RLHF (up to 60K pairs). ## Chat format KaiLu 3 learned from conversations written like this: ``` <|system|> You are KaiLu, a helpful assistant. <|user|> Hi! <|assistant|> Hello! How can I help?<|endoftext|> ``` Each reply ends with `<|endoftext|>`. ## Limitations KaiLu 3 is a small model. It can state wrong facts confidently, lose track in long conversations and write buggy code. It works best in English and within 1,024 tokens. It has had no safety testing beyond its preference tuning, so check its answers before relying on them. ## License **Free for personal use. Business use needs permission.** KaiLu 3 is released under the [KaiLu 3 License](https://huggingface.co//KaiLu-3/blob/main/LICENSE): you can use, study and modify it for free for personal and non-commercial purposes. For any business or commercial use, contact KvcHub at [hello@kvchub.com](mailto:hello@kvchub.com) first. The datasets it learned from keep their own licenses. Built by [KvcHub](https://kvchub.com).