KaiLu3 / README.md
KVC899's picture
Upload 2 files
e423197 verified
|
Raw History Blame Contribute Delete
3.23 kB
---
license: other
license_name: kailu-3-license
license_link: https://huggingface.co/<your-username>/KaiLu-3/blob/main/LICENSE
language:
- en
library_name: transformers
pipeline_tag: text-generation
inference: false
tags:
- kailu
- kvchub
- small-language-model
- hybrid
- short-convolution
- lfm2
- conversational
datasets:
- HuggingFaceFW/fineweb-edu
- HuggingFaceTB/smollm-corpus
- DKYoon/SlimPajama-6B
- HuggingFaceTB/finemath
- wikimedia/wikipedia
- OpenCoder-LLM/opc-annealing-corpus
- HuggingFaceTB/smol-smoltalk
- HuggingFaceH4/ultrafeedback_binarized
- nvidia/HelpSteer2
- Anthropic/hh-rlhf
---
# KaiLu 3
**A small language model with a big brain. Built from scratch by [KvcHub](https://kvchub.com).**
KaiLu 3 is a 329M-parameter chat model built from scratch by [KvcHub](https://kvchub.com). It has a hybrid brain: 16 fast gated short-convolution layers read nearby words, and 8 attention layers connect the whole text. It read 6.6B tokens of textbooks, web pages, math, Wikipedia and code, then learned to chat and to prefer the answers people like.
## Model details
| | |
|---|---|
| Parameters | 329,353,216 |
| Layers | 24: 16 gated short-convolution (kernel 3), 8 attention (layers 3, 6, 9, 12, 15, 18, 21, 23) |
| Width | hidden 1,024, SwiGLU feed-forward 2,816 |
| Attention | 16 query heads, 4 key-value heads (grouped-query), QK-norm, RoPE θ = 1,000,000 |
| Context | 1,024 tokens |
| Vocabulary | 32,768 tokens (byte-level BPE) |
| Embeddings | input and output shared |
| Weights | bfloat16, safetensors |
| Format | `Lfm2ForCausalLM` in 🤗 Transformers (4.54+). KaiLu 3 shares LFM2's layer layout only: it was trained from scratch and contains no LFM2 weights. |
## How it was made
1. **Pretraining:** 6.6B tokens, mixed as: educational web (FineWeb-Edu) 50%, synthetic textbooks (Cosmopedia v2) 18%, general web and books (SlimPajama-6B, without Books3) 12%, math (FineMath 4+) 8%, Wikipedia 6%, code (OpenCoder annealing) 6%.
2. **Supervised fine-tuning:** about 460K conversations from smol-smoltalk, one pass, learning only from the assistant replies.
3. **Preference tuning (DPO, β = 0.1):** answer pairs from UltraFeedback (binarized), HelpSteer2 preferences and Anthropic HH-RLHF (up to 60K pairs).
## Chat format
KaiLu 3 learned from conversations written like this:
```
<|system|>
You are KaiLu, a helpful assistant.
<|user|>
Hi!
<|assistant|>
Hello! How can I help?<|endoftext|>
```
Each reply ends with `<|endoftext|>`.
## Limitations
KaiLu 3 is a small model. It can state wrong facts confidently, lose track in long conversations and write buggy code.
It works best in English and within 1,024 tokens. It has had no safety testing beyond its preference tuning, so check
its answers before relying on them.
## License
**Free for personal use. Business use needs permission.** KaiLu 3 is released under the [KaiLu 3 License](https://huggingface.co/<your-username>/KaiLu-3/blob/main/LICENSE):
you can use, study and modify it for free for personal and non-commercial purposes. For any business or commercial use,
contact KvcHub at [hello@kvchub.com](mailto:hello@kvchub.com) first. The datasets it learned from keep their own licenses.
Built by [KvcHub](https://kvchub.com).