KVC899 commited on
Commit
51fb51c
·
verified ·
1 Parent(s): 77b08f8

Upload 8 files

Browse files
LICENSE ADDED
@@ -0,0 +1,62 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ KaiLu 3 License
2
+ Version 1.0, November 2026
3
+ Copyright (c) 2026 KVC Hub. All rights reserved except as granted below.
4
+
5
+ In short: KaiLu 3 is free for personal, non-commercial use. Any business or
6
+ commercial use needs written permission from KVC Hub first.
7
+
8
+ 1. What this license covers
9
+ "KaiLu 3" means the model weights, tokenizer, configuration files and
10
+ documentation that KVC Hub publishes under this license, and any version
11
+ made from them, including fine-tuned, merged, quantized, distilled or
12
+ converted versions ("Derivatives").
13
+
14
+ 2. Free personal use
15
+ You may download, run, study, modify and fine-tune KaiLu 3 free of charge
16
+ for personal, non-commercial purposes. This includes hobby projects,
17
+ learning, teaching, and non-commercial research.
18
+
19
+ 3. Sharing
20
+ You may share KaiLu 3 or your Derivatives for non-commercial purposes only,
21
+ and only if you:
22
+ a. include a copy of this license;
23
+ b. credit "KaiLu 3 by KVC Hub" with a link to https://kvchub.com; and
24
+ c. clearly say if you changed it.
25
+ Derivatives stay under this license.
26
+
27
+ 4. Business use needs permission
28
+ Commercial use of KaiLu 3 or its Derivatives is not allowed without a
29
+ separate written agreement with KVC Hub, made before the use starts.
30
+ Commercial use includes, for example:
31
+ a. using it in, or to build, any product or service that is sold, licensed,
32
+ subscribed to, or earns money in any way, including through ads;
33
+ b. offering it to others through an API, app or hosted service for money;
34
+ c. using it in the work of a company or organization for business purposes;
35
+ d. using it to do paid work for clients.
36
+ To ask for permission, contact KVC Hub through https://kvchub.com.
37
+
38
+ 5. Your outputs
39
+ KVC Hub claims no rights in text you create with KaiLu 3 while using it
40
+ as this license allows.
41
+
42
+ 6. Responsible use
43
+ You are responsible for how you use KaiLu 3 and what it produces. You must
44
+ not use it to break any law or to harm other people.
45
+
46
+ 7. Other licenses
47
+ KaiLu 3 learned from third-party datasets listed in its model card. Those
48
+ datasets keep their own licenses, and this license grants no rights to them.
49
+
50
+ 8. No warranty
51
+ KaiLu 3 is provided "as is", without warranties of any kind, express or
52
+ implied, including fitness for a particular purpose and accuracy. It can
53
+ make mistakes; check its answers before relying on them.
54
+
55
+ 9. Limitation of liability
56
+ To the fullest extent the law allows, KVC Hub is not liable for any damages
57
+ or losses arising from the use of, or inability to use, KaiLu 3.
58
+
59
+ 10. Ending of rights
60
+ If you break this license, your rights under it end automatically.
61
+ KVC Hub may publish new versions of this license; the version that came
62
+ with your copy of KaiLu 3 applies to that copy.
README.md ADDED
@@ -0,0 +1,84 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ license: other
3
+ license_name: kailu-3-license
4
+ license_link: https://huggingface.co/<your-username>/KaiLu-3/blob/main/LICENSE
5
+ language:
6
+ - en
7
+ library_name: transformers
8
+ pipeline_tag: text-generation
9
+ inference: false
10
+ tags:
11
+ - kailu
12
+ - kvc-hub
13
+ - small-language-model
14
+ - hybrid
15
+ - short-convolution
16
+ - lfm2
17
+ - conversational
18
+ datasets:
19
+ - HuggingFaceFW/fineweb-edu
20
+ - HuggingFaceTB/smollm-corpus
21
+ - DKYoon/SlimPajama-6B
22
+ - HuggingFaceTB/finemath
23
+ - wikimedia/wikipedia
24
+ - OpenCoder-LLM/opc-annealing-corpus
25
+ - HuggingFaceTB/smol-smoltalk
26
+ - HuggingFaceH4/ultrafeedback_binarized
27
+ - nvidia/HelpSteer2
28
+ - Anthropic/hh-rlhf
29
+ ---
30
+
31
+ # KaiLu 3
32
+
33
+ **A small language model with a big brain. Built from scratch by [KVC Hub](https://kvchub.com).**
34
+
35
+ KaiLu 3 is a 329M-parameter chat model built from scratch by [KVC Hub](https://kvchub.com). It has a hybrid brain: 16 fast gated short-convolution layers read nearby words, and 8 attention layers connect the whole text. It read 6.6B tokens of textbooks, web pages, math, Wikipedia and code, then learned to chat and to prefer the answers people like.
36
+
37
+ ## Model details
38
+
39
+ | | |
40
+ |---|---|
41
+ | Parameters | 329,353,216 |
42
+ | Layers | 24: 16 gated short-convolution (kernel 3), 8 attention (layers 3, 6, 9, 12, 15, 18, 21, 23) |
43
+ | Width | hidden 1,024, SwiGLU feed-forward 2,816 |
44
+ | Attention | 16 query heads, 4 key-value heads (grouped-query), QK-norm, RoPE θ = 1,000,000 |
45
+ | Context | 1,024 tokens |
46
+ | Vocabulary | 32,768 tokens (byte-level BPE) |
47
+ | Embeddings | input and output shared |
48
+ | Weights | bfloat16, safetensors |
49
+ | Format | `Lfm2ForCausalLM` in 🤗 Transformers (4.54+). KaiLu 3 shares LFM2's layer layout only: it was trained from scratch and contains no LFM2 weights. |
50
+
51
+ ## How it was made
52
+
53
+ 1. **Pretraining:** 6.6B tokens, mixed as: educational web (FineWeb-Edu) 50%, synthetic textbooks (Cosmopedia v2) 18%, general web and books (SlimPajama-6B, without Books3) 12%, math (FineMath 4+) 8%, Wikipedia 6%, code (OpenCoder annealing) 6%.
54
+ 2. **Supervised fine-tuning:** about 460K conversations from smol-smoltalk, one pass, learning only from the assistant replies.
55
+ 3. **Preference tuning (DPO, β = 0.1):** answer pairs from UltraFeedback (binarized), HelpSteer2 preferences and Anthropic HH-RLHF (up to 60K pairs).
56
+
57
+ ## Chat format
58
+
59
+ KaiLu 3 learned from conversations written like this:
60
+
61
+ ```
62
+ <|system|>
63
+ You are KaiLu, a helpful assistant.
64
+ <|user|>
65
+ Hi!
66
+ <|assistant|>
67
+ Hello! How can I help?<|endoftext|>
68
+ ```
69
+
70
+ Each reply ends with `<|endoftext|>`.
71
+
72
+ ## Limitations
73
+
74
+ KaiLu 3 is a small model. It can state wrong facts confidently, lose track in long conversations and write buggy code.
75
+ It works best in English and within 1,024 tokens. It has had no safety testing beyond its preference tuning, so check
76
+ its answers before relying on them.
77
+
78
+ ## License
79
+
80
+ **Free for personal use. Business use needs permission.** KaiLu 3 is released under the [KaiLu 3 License](https://huggingface.co/<your-username>/KaiLu-3/blob/main/LICENSE):
81
+ you can use, study and modify it for free for personal and non-commercial purposes. For any business or commercial use,
82
+ contact KVC Hub through [kvchub.com](https://kvchub.com) first. The datasets it learned from keep their own licenses.
83
+
84
+ Built by [KVC Hub](https://kvchub.com).
chat_template.jinja ADDED
@@ -0,0 +1,4 @@
 
 
 
 
 
1
+ {%- set roles = {'system': 'system', 'user': 'user', 'human': 'user', 'assistant': 'assistant', 'gpt': 'assistant'} -%}{%- for message in messages -%}{%- set role = roles.get((message['role'] | string) | lower) -%}{%- if role -%}{{- '<|' + role + '|>
2
+ ' + ((message['content'] or '') | string | trim) -}}{%- if role == 'assistant' -%}{{- '<|endoftext|>' -}}{%- endif -%}{{- '
3
+ ' -}}{%- endif -%}{%- endfor -%}{%- if add_generation_prompt -%}{{- '<|assistant|>
4
+ ' -}}{%- endif -%}
config.json ADDED
@@ -0,0 +1,67 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "architectures": [
3
+ "Lfm2ForCausalLM"
4
+ ],
5
+ "block_auto_adjust_ff_dim": false,
6
+ "block_ffn_dim_multiplier": 1.0,
7
+ "block_multiple_of": 256,
8
+ "bos_token_id": null,
9
+ "conv_L_cache": 3,
10
+ "conv_bias": false,
11
+ "dtype": "bfloat16",
12
+ "eos_token_id": 0,
13
+ "full_attn_idxs": [
14
+ 2,
15
+ 5,
16
+ 8,
17
+ 11,
18
+ 14,
19
+ 17,
20
+ 20,
21
+ 22
22
+ ],
23
+ "hidden_size": 1024,
24
+ "initializer_range": 0.02,
25
+ "intermediate_size": 2816,
26
+ "layer_types": [
27
+ "conv",
28
+ "conv",
29
+ "full_attention",
30
+ "conv",
31
+ "conv",
32
+ "full_attention",
33
+ "conv",
34
+ "conv",
35
+ "full_attention",
36
+ "conv",
37
+ "conv",
38
+ "full_attention",
39
+ "conv",
40
+ "conv",
41
+ "full_attention",
42
+ "conv",
43
+ "conv",
44
+ "full_attention",
45
+ "conv",
46
+ "conv",
47
+ "full_attention",
48
+ "conv",
49
+ "full_attention",
50
+ "conv"
51
+ ],
52
+ "max_position_embeddings": 1024,
53
+ "model_type": "lfm2",
54
+ "norm_eps": 1e-05,
55
+ "num_attention_heads": 16,
56
+ "num_hidden_layers": 24,
57
+ "num_key_value_heads": 4,
58
+ "pad_token_id": 0,
59
+ "rope_parameters": {
60
+ "rope_theta": 1000000.0,
61
+ "rope_type": "default"
62
+ },
63
+ "tie_word_embeddings": true,
64
+ "transformers_version": "5.22.0",
65
+ "use_cache": true,
66
+ "vocab_size": 32768
67
+ }
generation_config.json ADDED
@@ -0,0 +1,8 @@
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "do_sample": true,
3
+ "eos_token_id": 0,
4
+ "pad_token_id": 0,
5
+ "temperature": 0.7,
6
+ "top_k": 50,
7
+ "transformers_version": "5.22.0"
8
+ }
model.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:d8f78f92265af2ff068dfed9ab5981e5d294beef5d8dbd5c2946042adcea103b
3
+ size 658730768
tokenizer.json ADDED
The diff for this file is too large to render. See raw diff
 
tokenizer_config.json ADDED
@@ -0,0 +1,12 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "backend": "tokenizers",
3
+ "clean_up_tokenization_spaces": false,
4
+ "eos_token": "<|endoftext|>",
5
+ "model_input_names": [
6
+ "input_ids",
7
+ "attention_mask"
8
+ ],
9
+ "model_max_length": 1024,
10
+ "pad_token": "<|endoftext|>",
11
+ "tokenizer_class": "PreTrainedTokenizerFast"
12
+ }