judotens commited on
Commit
6936517
·
verified ·
1 Parent(s): 554d3d6

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +146 -0
README.md CHANGED
@@ -1,3 +1,149 @@
1
  ---
2
  license: apache-2.0
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
3
  ---
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
  ---
2
  license: apache-2.0
3
+ language:
4
+ - en
5
+ library_name: flatbuild
6
+ tags:
7
+ - language-model
8
+ - conversational
9
+ - transformer
10
+ - flatbuild
11
+ - flatrun
12
+ - trained-from-scratch
13
+ - rope
14
+ - rmsnorm
15
+ - swiglu
16
+ - gqa
17
+ pipeline_tag: text-generation
18
  ---
19
+
20
+ # Flatbot-4M
21
+
22
+ Flatbot-4M is the flagship demonstration model for **Flatbuild**.
23
+
24
+ It is a compact conversational language model trained entirely from scratch using the Flatbuild framework. The goal is not to compete with large language models, but to demonstrate the complete workflow of building a modern transformer—from dataset generation and tokenizer training to model training, checkpoint export, and inference.
25
+
26
+ Everything in this repository is produced by Flatbuild without relying on pretrained weights.
27
+
28
+ ## Architecture
29
+
30
+ | Component | Details |
31
+ |---|---|
32
+ | Architecture | Decoder-only Transformer |
33
+ | Position Encoding | RoPE |
34
+ | Normalization | RMSNorm |
35
+ | Feed Forward | SwiGLU |
36
+ | Attention | Grouped Query Attention (8 Query Heads / 4 KV Heads) |
37
+ | Weight Tying | Yes |
38
+ | Context Length | 384 |
39
+ | Parameters | ~4.1M |
40
+
41
+ ## Model Configuration
42
+
43
+ ```text
44
+ vocab_size = 512
45
+ hidden_size = 256
46
+ num_layers = 6
47
+ num_heads = 8
48
+ num_kv_heads = 4
49
+ head_dim = 32
50
+ ffn_dim = 768
51
+ context_length = 384
52
+ rope_theta = 10000
53
+ ```
54
+
55
+ ## Training
56
+
57
+ The model was trained completely from random initialization using Flatbuild.
58
+
59
+ ### Dataset
60
+
61
+ | Property | Value |
62
+ |---|---:|
63
+ | Conversations | ~2,500 |
64
+ | Multi-turn conversations | ~55% |
65
+ | Train Split | 95% |
66
+ | Validation Split | 5% |
67
+ | Context Length | 384 tokens |
68
+
69
+ The dataset contains general-purpose conversations including greetings, introductions, recommendations, explanations, simple reasoning, follow-up questions, and everyday dialogue.
70
+
71
+ ## Training Configuration
72
+
73
+ | Hyperparameter | Value |
74
+ |---|---:|
75
+ | Optimizer | AdamW |
76
+ | Learning Rate | 1e-3 |
77
+ | Scheduler | Cosine |
78
+ | Warmup | 20 steps |
79
+ | Epochs | 12 |
80
+ | Batch Size | 4 |
81
+ | Gradient Accumulation | 2 |
82
+ | Precision | FP32 |
83
+
84
+ ## Features
85
+
86
+ - Trained entirely from scratch
87
+ - Custom tokenizer
88
+ - Native chat template
89
+ - RoPE positional embeddings
90
+ - RMSNorm
91
+ - SwiGLU
92
+ - Grouped Query Attention (GQA)
93
+ - Weight-tied embeddings
94
+ - SafeTensors export
95
+ - GGUF export
96
+ - Native Flatrun compatibility
97
+
98
+ ## Usage
99
+
100
+ ### Flatbuild
101
+
102
+ ```bash
103
+ flatbuild generate flatseek/Flatbot-4M \
104
+ --chat \
105
+ --prompt "Hello! What can you do?"
106
+ ```
107
+
108
+ ### Python
109
+
110
+ ```python
111
+ from flatbuild import AutoModel
112
+
113
+ model = AutoModel.from_pretrained("flatseek/Flatbot-4M")
114
+
115
+ print(
116
+ model.generate(
117
+ "Hello! What can you do?",
118
+ chat=True,
119
+ )
120
+ )
121
+ ```
122
+
123
+ ### Flatrun
124
+
125
+ ```bash
126
+ flatrun model.gguf \
127
+ --chat \
128
+ --prompt "Hello!"
129
+ ```
130
+
131
+ ## Purpose
132
+
133
+ Flatbot-4M exists as an end-to-end demonstration of the Flatbuild ecosystem.
134
+
135
+ It showcases every stage of training a conversational transformer:
136
+
137
+ 1. Build a dataset
138
+ 2. Train a tokenizer
139
+ 3. Configure a transformer
140
+ 4. Train from scratch
141
+ 5. Export checkpoints
142
+ 6. Convert to GGUF
143
+ 7. Run inference with Flatrun
144
+
145
+ The complete pipeline is reproducible on consumer hardware and is intended for learning, experimentation, and research.
146
+
147
+ ## Limitations
148
+
149
+ Flatbot-4M is a small conversational language model trained on a relatively small dataset. It has limited factual knowledge and reasoning ability compared to larger models, and is intended as a demonstration of the complete Flatbuild workflow rather than a production-grade assistant.