Text Generation
Transformers
Safetensors
GGUF
English
lfm2
kailu
kvchub
small-language-model
hybrid
short-convolution
conversational
Instructions to use KvcHub/KaiLu3 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use KvcHub/KaiLu3 with Transformers:
# Use a pipeline as a high-level helper from transformers import pipeline pipe = pipeline("text-generation", model="KvcHub/KaiLu3") messages = [ {"role": "user", "content": "Who are you?"}, ] pipe(messages)# pip install -U transformers accelerate # Load model directly from transformers import AutoTokenizer, AutoModelForCausalLM tokenizer = AutoTokenizer.from_pretrained("KvcHub/KaiLu3") model = AutoModelForCausalLM.from_pretrained("KvcHub/KaiLu3", device_map="auto") messages = [ {"role": "user", "content": "Who are you?"}, ] inputs = tokenizer.apply_chat_template( messages, add_generation_prompt=True, tokenize=True, return_dict=True, return_tensors="pt", ).to(model.device) outputs = model.generate(**inputs, max_new_tokens=256) print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:])) - Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- llama.cpp
How to use KvcHub/KaiLu3 with llama.cpp:
Install (macOS, Linux)
curl -LsSf https://llama.app/install.sh | sh # Start a local OpenAI-compatible server with a web UI: llama serve -hf KvcHub/KaiLu3 # Run inference directly in the terminal: llama cli -hf KvcHub/KaiLu3
Install from WinGet (Windows)
winget install llama.cpp # Start a local OpenAI-compatible server with a web UI: llama serve -hf KvcHub/KaiLu3 # Run inference directly in the terminal: llama cli -hf KvcHub/KaiLu3
Build from source code
git clone https://github.com/ggml-org/llama.cpp.git cd llama.cpp cmake -B build cmake --build build -j --target llama-app # Start a local OpenAI-compatible server with a web UI: ./build/bin/llama serve -hf KvcHub/KaiLu3 # Run inference directly in the terminal: ./build/bin/llama cli -hf KvcHub/KaiLu3
Use Docker
docker model run hf.co/KvcHub/KaiLu3
- Llama new
- LM Studio
- Jan
- vLLM
How to use KvcHub/KaiLu3 with vLLM:
Install from pip and serve model
# Install vLLM from pip: pip install vllm # Start the vLLM server: vllm serve "KvcHub/KaiLu3" # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:8000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "KvcHub/KaiLu3", "messages": [ { "role": "user", "content": "What is the capital of France?" } ] }'Use Docker
docker model run hf.co/KvcHub/KaiLu3
- SGLang
How to use KvcHub/KaiLu3 with SGLang:
Install from pip and serve model
# Install SGLang from pip: pip install sglang # Start the SGLang server: python3 -m sglang.launch_server \ --model-path "KvcHub/KaiLu3" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "KvcHub/KaiLu3", "messages": [ { "role": "user", "content": "What is the capital of France?" } ] }'Use Docker images
docker run --gpus all \ --shm-size 32g \ -p 30000:30000 \ -v ~/.cache/huggingface:/root/.cache/huggingface \ --env "HF_TOKEN=<secret>" \ --ipc=host \ lmsysorg/sglang:latest \ python3 -m sglang.launch_server \ --model-path "KvcHub/KaiLu3" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "KvcHub/KaiLu3", "messages": [ { "role": "user", "content": "What is the capital of France?" } ] }' - Ollama
How to use KvcHub/KaiLu3 with Ollama:
ollama run hf.co/KvcHub/KaiLu3
- Unsloth Desktop
- Docker Model Runner
How to use KvcHub/KaiLu3 with Docker Model Runner:
docker model run hf.co/KvcHub/KaiLu3
- Lemonade
How to use KvcHub/KaiLu3 with Lemonade:
Pull the model
# Download Lemonade from https://lemonade-server.ai/ lemonade pull KvcHub/KaiLu3
Run and chat with the model
lemonade run user.KaiLu3-{{QUANT_TAG}}List all available models
lemonade list
- Atomic Chat
|
Download README.md from KvcHub/KaiLu3: direct link, hf CLI and curl.
- Browser
- Download file 3.23 kB
-
https://huggingface.co/KvcHub/KaiLu3/resolve/main/README.md
- Command line
-
hf download hf://KvcHub/KaiLu3/README.md
-
curl -L -o README.md https://huggingface.co/KvcHub/KaiLu3/resolve/main/README.md
3.23 kB
| license: other | |
| license_name: kailu-3-license | |
| license_link: https://huggingface.co/<your-username>/KaiLu-3/blob/main/LICENSE | |
| language: | |
| - en | |
| library_name: transformers | |
| pipeline_tag: text-generation | |
| inference: false | |
| tags: | |
| - kailu | |
| - kvchub | |
| - small-language-model | |
| - hybrid | |
| - short-convolution | |
| - lfm2 | |
| - conversational | |
| datasets: | |
| - HuggingFaceFW/fineweb-edu | |
| - HuggingFaceTB/smollm-corpus | |
| - DKYoon/SlimPajama-6B | |
| - HuggingFaceTB/finemath | |
| - wikimedia/wikipedia | |
| - OpenCoder-LLM/opc-annealing-corpus | |
| - HuggingFaceTB/smol-smoltalk | |
| - HuggingFaceH4/ultrafeedback_binarized | |
| - nvidia/HelpSteer2 | |
| - Anthropic/hh-rlhf | |
| # KaiLu 3 | |
| **A small language model with a big brain. Built from scratch by [KvcHub](https://kvchub.com).** | |
| KaiLu 3 is a 329M-parameter chat model built from scratch by [KvcHub](https://kvchub.com). It has a hybrid brain: 16 fast gated short-convolution layers read nearby words, and 8 attention layers connect the whole text. It read 6.6B tokens of textbooks, web pages, math, Wikipedia and code, then learned to chat and to prefer the answers people like. | |
| ## Model details | |
| | | | | |
| |---|---| | |
| | Parameters | 329,353,216 | | |
| | Layers | 24: 16 gated short-convolution (kernel 3), 8 attention (layers 3, 6, 9, 12, 15, 18, 21, 23) | | |
| | Width | hidden 1,024, SwiGLU feed-forward 2,816 | | |
| | Attention | 16 query heads, 4 key-value heads (grouped-query), QK-norm, RoPE θ = 1,000,000 | | |
| | Context | 1,024 tokens | | |
| | Vocabulary | 32,768 tokens (byte-level BPE) | | |
| | Embeddings | input and output shared | | |
| | Weights | bfloat16, safetensors | | |
| | Format | `Lfm2ForCausalLM` in 🤗 Transformers (4.54+). KaiLu 3 shares LFM2's layer layout only: it was trained from scratch and contains no LFM2 weights. | | |
| ## How it was made | |
| 1. **Pretraining:** 6.6B tokens, mixed as: educational web (FineWeb-Edu) 50%, synthetic textbooks (Cosmopedia v2) 18%, general web and books (SlimPajama-6B, without Books3) 12%, math (FineMath 4+) 8%, Wikipedia 6%, code (OpenCoder annealing) 6%. | |
| 2. **Supervised fine-tuning:** about 460K conversations from smol-smoltalk, one pass, learning only from the assistant replies. | |
| 3. **Preference tuning (DPO, β = 0.1):** answer pairs from UltraFeedback (binarized), HelpSteer2 preferences and Anthropic HH-RLHF (up to 60K pairs). | |
| ## Chat format | |
| KaiLu 3 learned from conversations written like this: | |
| ``` | |
| <|system|> | |
| You are KaiLu, a helpful assistant. | |
| <|user|> | |
| Hi! | |
| <|assistant|> | |
| Hello! How can I help?<|endoftext|> | |
| ``` | |
| Each reply ends with `<|endoftext|>`. | |
| ## Limitations | |
| KaiLu 3 is a small model. It can state wrong facts confidently, lose track in long conversations and write buggy code. | |
| It works best in English and within 1,024 tokens. It has had no safety testing beyond its preference tuning, so check | |
| its answers before relying on them. | |
| ## License | |
| **Free for personal use. Business use needs permission.** KaiLu 3 is released under the [KaiLu 3 License](https://huggingface.co/<your-username>/KaiLu-3/blob/main/LICENSE): | |
| you can use, study and modify it for free for personal and non-commercial purposes. For any business or commercial use, | |
| contact KvcHub at [hello@kvchub.com](mailto:hello@kvchub.com) first. The datasets it learned from keep their own licenses. | |
| Built by [KvcHub](https://kvchub.com). | |