KaiLu 3 Pro

KaiLu 3 Pro is a 1-billion-parameter chat model trained from scratch by KvcHub. It is small enough to run on a laptop: about 1.1 GB as a GGUF, fast on a CPU and faster on Apple Silicon or a GPU. It is open source under the Apache License 2.0.

Files

File What it is
KaiLu-3-Pro-Q8_0.gguf The whole model in one file for llama.cpp, Ollama and LM Studio: 8-bit weights, tokenizer and chat template (1.09 GB).
model.safetensors The full-precision weights (bfloat16) in one file for Transformers (2.07 GB). It also holds 6 vision_projector.* tensors from KaiLu's experimental image add-on; Transformers skips them, and they don't work on their own.
config.json, generation_config.json, tokenizer.json, tokenizer_config.json, chat_template.jinja Settings, tokenizer and chat format for Transformers.
Modelfile, template, params Ollama setup: chat format, temperature 0.2, stop words.

Use it

Download everything with hf download KvcHub/KaiLu-3-Pro --local-dir KaiLu-3-Pro, or only the GGUF with hf download KvcHub/KaiLu-3-Pro KaiLu-3-Pro-Q8_0.gguf Modelfile --local-dir KaiLu-3-Pro.

Ollama

ollama run hf.co/KvcHub/KaiLu-3-Pro

The repo's template and params files give Ollama KaiLu's chat format and settings. In a downloaded folder, ollama create kailu-3-pro -f Modelfile does the same.

llama.cpp

llama-cli -m KaiLu-3-Pro-Q8_0.gguf --jinja -c 1024 --temp 0.2
llama-server -m KaiLu-3-Pro-Q8_0.gguf --jinja -c 1024     # OpenAI-compatible API on http://localhost:8080

LM Studio: import KaiLu-3-Pro-Q8_0.gguf. The chat template is stored in the file.

Transformers (4.54 or newer)

from transformers import AutoModelForCausalLM, AutoTokenizer

path = "KvcHub/KaiLu-3-Pro"  # or a local folder
tok = AutoTokenizer.from_pretrained(path)
model = AutoModelForCausalLM.from_pretrained(path, dtype="bfloat16")

messages = [{"role": "user", "content": "What is the capital of France?"}]
prompt = tok.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
inputs = tok(prompt, return_tensors="pt")
out = model.generate(**inputs, max_new_tokens=200, do_sample=False)
print(tok.decode(out[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True))

A low temperature (0 to 0.3) gives the best answers; at 0.7 a model this size wanders.

Chat format

<|system|>
You are KaiLu, a helpful assistant.
<|user|>
Who are you?
<|assistant|>
I'm KaiLu 3 Pro, an AI assistant created by KvcHub.<|endoftext|>

If there's no system message, the template adds the default one above. Each assistant reply ends with <|endoftext|> (token 0). KaiLu also learned to reason inside <think>...</think> when asked, for example with the system message You are KaiLu, a helpful assistant. Mode: reasoning=medium. Think step-by-step inside <think>...</think> before your final answer.

Model details

Parameters 1,027,304,448
Layers 36: 24 gated short-convolution layers (kernel 3) and 12 attention layers (every third layer)
Width hidden 1,536, SwiGLU feed-forward 4,096
Attention 24 query heads, 6 key-value heads (grouped-query), QK-norm, RoPE theta 1,000,000
Context trained on 1,024-token windows
Vocabulary 32,768 tokens (byte-level BPE, trained by KvcHub)
Embeddings input and output shared
Format Lfm2ForCausalLM. KaiLu 3 Pro uses LFM2's layer layout only: it was trained from scratch and contains no LFM2 weights.

How it was made

  1. KaiLu 3 base (329M parameters): pretrained from scratch on 6.6B tokens of educational web text (FineWeb-Edu), synthetic textbooks (Cosmopedia v2), web and books (SlimPajama-6B, without Books3), math (FineMath), Wikipedia and code (OpenCoder annealing). Then fine-tuned on smol-smoltalk conversations, and preference-tuned (DPO) on UltraFeedback, HelpSteer2 and Anthropic HH-RLHF pairs.
  2. Grown to 1B: widened (hidden 1,024 to 1,536) and deepened (24 to 36 layers) with a function-preserving expansion, so it started from what the 329M model already knew.
  3. Further pretraining: including the Dolmino Mix (DCLM web text, StackExchange, peS2o papers, FLAN and math).
  4. Fine-tuning: Tulu 3 persona instruction-following, MetaMathQA, GSM8K, MMLU, ARC-Challenge, HumanEval and General-Knowledge, plus examples written by KvcHub (step-by-step reasoning in <think>, coding in several languages, and who KaiLu is).
  5. Preference tuning (DPO, beta 0.1): pairs from UltraFeedback (binarized) and Orca DPO pairs.

The datasets are listed in this card's metadata. Each keeps its own license and terms.

Limitations

  • It's a small model. It gets facts wrong, sometimes confidently (for example, it once answered "Japan is the capital of Japan"), and it can invent details. Don't rely on it for facts, medical, legal or financial advice.
  • Short memory. It was trained on 1,024-token windows. It often doesn't use what you said earlier in a chat (asked "What is my name?" right after giving it, it says it doesn't know personal details), and it may answer only part of a multi-part question.
  • English only.
  • No benchmark scores here, on purpose. The test sets of MMLU, ARC-Challenge and HumanEval, and parts of GSM8K, were used in fine-tuning, so the model's scores on those benchmarks would overstate its ability. Evaluate it on data it hasn't seen.
  • Text only. The vision_projector.* tensors need KaiLu's own image encoder pipeline, which isn't part of this release.

License

Copyright 2026 KvcHub.

Licensed under the Apache License, Version 2.0 (see LICENSE). You may use, modify and redistribute the model, including commercially, under its terms.

Downloads last month
48
Safetensors
Model size
1B params
Tensor type
BF16
·
Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support

Datasets used to train KvcHub/KaiLu-3-Pro