KaiLu3 / README.md
KVC899's picture
Upload 2 files
e423197 verified
|
Raw History Blame Contribute Delete
3.23 kB
metadata
license: other
license_name: kailu-3-license
license_link: https://huggingface.co/<your-username>/KaiLu-3/blob/main/LICENSE
language:
  - en
library_name: transformers
pipeline_tag: text-generation
inference: false
tags:
  - kailu
  - kvchub
  - small-language-model
  - hybrid
  - short-convolution
  - lfm2
  - conversational
datasets:
  - HuggingFaceFW/fineweb-edu
  - HuggingFaceTB/smollm-corpus
  - DKYoon/SlimPajama-6B
  - HuggingFaceTB/finemath
  - wikimedia/wikipedia
  - OpenCoder-LLM/opc-annealing-corpus
  - HuggingFaceTB/smol-smoltalk
  - HuggingFaceH4/ultrafeedback_binarized
  - nvidia/HelpSteer2
  - Anthropic/hh-rlhf

KaiLu 3

A small language model with a big brain. Built from scratch by KvcHub.

KaiLu 3 is a 329M-parameter chat model built from scratch by KvcHub. It has a hybrid brain: 16 fast gated short-convolution layers read nearby words, and 8 attention layers connect the whole text. It read 6.6B tokens of textbooks, web pages, math, Wikipedia and code, then learned to chat and to prefer the answers people like.

Model details

Parameters 329,353,216
Layers 24: 16 gated short-convolution (kernel 3), 8 attention (layers 3, 6, 9, 12, 15, 18, 21, 23)
Width hidden 1,024, SwiGLU feed-forward 2,816
Attention 16 query heads, 4 key-value heads (grouped-query), QK-norm, RoPE θ = 1,000,000
Context 1,024 tokens
Vocabulary 32,768 tokens (byte-level BPE)
Embeddings input and output shared
Weights bfloat16, safetensors
Format Lfm2ForCausalLM in 🤗 Transformers (4.54+). KaiLu 3 shares LFM2's layer layout only: it was trained from scratch and contains no LFM2 weights.

How it was made

  1. Pretraining: 6.6B tokens, mixed as: educational web (FineWeb-Edu) 50%, synthetic textbooks (Cosmopedia v2) 18%, general web and books (SlimPajama-6B, without Books3) 12%, math (FineMath 4+) 8%, Wikipedia 6%, code (OpenCoder annealing) 6%.
  2. Supervised fine-tuning: about 460K conversations from smol-smoltalk, one pass, learning only from the assistant replies.
  3. Preference tuning (DPO, β = 0.1): answer pairs from UltraFeedback (binarized), HelpSteer2 preferences and Anthropic HH-RLHF (up to 60K pairs).

Chat format

KaiLu 3 learned from conversations written like this:

<|system|>
You are KaiLu, a helpful assistant.
<|user|>
Hi!
<|assistant|>
Hello! How can I help?<|endoftext|>

Each reply ends with <|endoftext|>.

Limitations

KaiLu 3 is a small model. It can state wrong facts confidently, lose track in long conversations and write buggy code. It works best in English and within 1,024 tokens. It has had no safety testing beyond its preference tuning, so check its answers before relying on them.

License

Free for personal use. Business use needs permission. KaiLu 3 is released under the KaiLu 3 License: you can use, study and modify it for free for personal and non-commercial purposes. For any business or commercial use, contact KvcHub at hello@kvchub.com first. The datasets it learned from keep their own licenses.

Built by KvcHub.