judotens commited on
Commit
3e10e06
·
verified ·
1 Parent(s): 89c59a4

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +197 -0
README.md CHANGED
@@ -1,3 +1,200 @@
1
  ---
2
  license: apache-2.0
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
3
  ---
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
  ---
2
  license: apache-2.0
3
+ language:
4
+ - en
5
+ library_name: flatbuild
6
+ tags:
7
+ - language-model
8
+ - conversational
9
+ - flatseek
10
+ - flatbuild
11
+ - flatrun
12
+ - trained-from-scratch
13
+ - rope
14
+ - rmsnorm
15
+ - swiglu
16
+ - gqa
17
+ - gguf
18
+ - q4_0
19
+ pipeline_tag: text-generation
20
  ---
21
+
22
+ # Flatbot-mini-11M-Q4_0
23
+
24
+ **Try it online:** https://chat.flatseek.io
25
+
26
+ Flatbot-mini-11M-Q4_0 is the flagship compact conversational model for the **Flatseek** ecosystem.
27
+
28
+ It is an **11 million parameter** decoder-only language model trained entirely from scratch using **FlatBuild**, exported to **GGUF**, quantized to **Q4_0**, and optimized for efficient CPU inference with **FlatRun**.
29
+
30
+ The original FP32 checkpoint is approximately **135 MB**, while the **Q4_0 GGUF** version reduces the model size significantly, making it practical for lightweight local deployment without requiring a GPU.
31
+
32
+ > **Experimental model:** Flatbot-mini-11M is intended for research, education, and experimentation. Although substantially more capable than Flatbot-micro-4M, it may still produce hallucinations, factual mistakes, or inconsistent reasoning.
33
+
34
+ ---
35
+
36
+ # Architecture
37
+
38
+ | Component | Details |
39
+ |---|---|
40
+ | Architecture | Decoder-only Transformer |
41
+ | Position Encoding | RoPE |
42
+ | Normalization | RMSNorm |
43
+ | Feed Forward | SwiGLU |
44
+ | Attention | Grouped Query Attention (16 Query Heads / 4 KV Heads) |
45
+ | Weight Tying | Yes |
46
+ | Context Length | 512 |
47
+ | Parameters | ~11M |
48
+
49
+ ---
50
+
51
+ # Model Configuration
52
+
53
+ ```text
54
+ vocab_size = 1024
55
+ hidden_size = 512
56
+ num_layers = 12
57
+ num_heads = 16
58
+ num_kv_heads = 4
59
+ head_dim = 32
60
+ ffn_dim = 1408
61
+ context_length = 512
62
+ rope_theta = 10000
63
+ ```
64
+
65
+ ---
66
+
67
+ # Training
68
+
69
+ The model was trained entirely from random initialization using **FlatBuild**.
70
+
71
+ ## Dataset
72
+
73
+ | Property | Value |
74
+ |---|---:|
75
+ | Conversations | ~10,000 |
76
+ | Train Split | 95% |
77
+ | Validation Split | 5% |
78
+ | Context Length | 512 tokens |
79
+
80
+ The training dataset consists of approximately **10,000** conversational examples covering greetings, general knowledge, explanations, recommendations, question answering, follow-up conversations, and common assistant interactions.
81
+
82
+ ---
83
+
84
+ # Training Configuration
85
+
86
+ | Hyperparameter | Value |
87
+ |---|---:|
88
+ | Optimizer | AdamW |
89
+ | Learning Rate | 1e-3 |
90
+ | Scheduler | Cosine |
91
+ | Warmup | 50 steps |
92
+ | Epochs | 10 |
93
+ | Batch Size | 16 |
94
+ | Gradient Accumulation | 2 |
95
+ | Precision | FP32 |
96
+
97
+ ---
98
+
99
+ # Quantization
100
+
101
+ This release is provided as a **GGUF Q4_0** model optimized for lightweight inference.
102
+
103
+ | Item | Value |
104
+ |---|---:|
105
+ | Original Format | FP32 |
106
+ | Original Size | ~135 MB |
107
+ | Quantization | GGUF Q4_0 |
108
+ | Intended Runtime | FlatRun / llama.cpp compatible runtimes |
109
+
110
+ ---
111
+
112
+ # Features
113
+
114
+ - Trained entirely from scratch
115
+ - Custom tokenizer (1024 vocabulary)
116
+ - Native chat template
117
+ - RoPE positional embeddings
118
+ - RMSNorm
119
+ - SwiGLU
120
+ - Grouped Query Attention (GQA)
121
+ - Weight-tied embeddings
122
+ - SafeTensors export
123
+ - GGUF export
124
+ - Q4_0 quantization
125
+ - Native FlatRun compatibility
126
+
127
+ ---
128
+
129
+ # Usage
130
+
131
+ ## FlatBuild Training
132
+
133
+ ```bash
134
+ pip install flatbuild
135
+
136
+ flatbuild train configs/flatbot-mini-11M.yaml
137
+
138
+ flatbuild export \
139
+ outputs/flatbot-mini-11M-v2/*/checkpoint/final \
140
+ --format gguf \
141
+ --output flatbot-mini-11M
142
+ ```
143
+
144
+ ---
145
+
146
+ ## LM Studio
147
+
148
+ ```bash
149
+ lms import flatbot-mini-11M-Q4_0.gguf
150
+ ```
151
+
152
+ ---
153
+
154
+ ## FlatRun Inference
155
+
156
+ ```bash
157
+ pip install flatrun
158
+
159
+ flatrun chat \
160
+ --model flatbot-mini-11M-Q4_0.gguf \
161
+ --temp 0.2
162
+ ```
163
+
164
+ Example:
165
+
166
+ ```text
167
+ Detected format: gguf
168
+ Building tokenizer from GGUF metadata...
169
+ Loaded model in 0.03 s
170
+
171
+ You: Who are you?
172
+
173
+ Assistant:
174
+ I'm Flatbot, a conversational AI assistant trained from scratch using the Flatseek ecosystem. I'm designed to answer questions, explain concepts, and assist with everyday tasks.
175
+ ```
176
+
177
+ ---
178
+
179
+ # Purpose
180
+
181
+ Flatbot-mini-11M demonstrates the complete **Flatseek AI pipeline**:
182
+
183
+ 1. Build conversational datasets
184
+ 2. Train a tokenizer
185
+ 3. Configure a transformer architecture
186
+ 4. Train entirely from scratch
187
+ 5. Export SafeTensors checkpoints
188
+ 6. Convert to GGUF
189
+ 7. Quantize to Q4_0
190
+ 8. Run efficient local inference with FlatRun
191
+
192
+ The entire workflow is fully reproducible on consumer hardware and showcases how modern language models can be built without relying on pretrained foundation models.
193
+
194
+ ---
195
+
196
+ # Limitations
197
+
198
+ Flatbot-mini-11M is still a relatively small language model compared with modern LLMs containing billions of parameters. While it offers noticeably stronger conversational ability than the earlier 4M demonstration model, it remains limited in factual knowledge, reasoning depth, multilingual performance, and long-context understanding.
199
+
200
+ Its primary goal is to demonstrate an end-to-end open-source language model training, export, quantization, and inference pipeline built entirely within the Flatseek ecosystem.