Icold commited on
Commit
1ea617f
·
verified ·
1 Parent(s): 6a0e18e

Upload README.md with huggingface_hub

Browse files
Files changed (1) hide show
  1. README.md +666 -75
README.md CHANGED
@@ -1,75 +1,666 @@
1
- ---
2
- language:
3
- - en
4
- - ru
5
- license: mit
6
- library_name: pytorch
7
- pipeline_tag: text-generation
8
- tags:
9
- - language-model
10
- - kronecker
11
- - small-language-model
12
- - bilingual
13
- - parameter-efficiency
14
- ---
15
-
16
- # KronLLM
17
-
18
- A compact bilingual (English/Russian) language model whose matrix layers are
19
- built entirely from **Kronecker sums of rank-1 factors**. Every attention and
20
- FFN projection is a composition of `L` factors instead of a dense `d × d`
21
- weight: a layer that would need 102,400 numbers needs 15,753.
22
-
23
- This repository hosts the model card, the code and the shipped int8
24
- checkpoints. The training corpus is separate: see
25
- [Icold21/kron-corpus](https://huggingface.co/datasets/Icold21/kron-corpus) and
26
- the research log at [github.com/Icold21/kron-llm](https://github.com/Icold21/kron-llm).
27
-
28
- ## Shipped checkpoints
29
-
30
- | file | parameters | size | quality |
31
- |---|---:|---:|---|
32
- | `q8g_int8.pt` | 2,851,280 | 3.14 MB | **0.7542 bits/byte** (TinyStories val, 61.4M tokens seen) |
33
- | `q2_int8.pt` | 2,079,992 | 2.34 MB | previous model, same corpus |
34
-
35
- Int8 quantisation costs +0.0003 bpb on Q8g. The fp32 checkpoint is 11.4 MB and
36
- is not shipped.
37
-
38
- ## Usage
39
-
40
- ```python
41
- import torch
42
- from kronllm.model import KronTransformer
43
- from kronllm.quantization import load_int8
44
-
45
- ckpt = torch.load("q8g_int8.pt", map_location="cpu")
46
- model = KronTransformer(ckpt["cfg"]).eval()
47
- load_int8(model, ckpt["model"])
48
- ```
49
-
50
- The model is a research artifact: it writes short English and Russian text and
51
- is not an instruct-tuned assistant.
52
-
53
- ## Architecture at a glance
54
-
55
- | | |
56
- |---|---|
57
- | layers / width | 24 / 320, 8 heads, seq 192 |
58
- | composition | product over L=4 factors, K=6 terms, depth carry `K_win=2` |
59
- | FFN | gated (SwiGLU-style) |
60
- | tokenizer | BPE-4096, tied factorised embedding (rank 128) |
61
-
62
- Measured, not assumed: the depth carry is worth 1.1 % bpb, freezing its scalar
63
- 0.8 %, and widening the window to four layers *costs* 1.4 %. Details in the
64
- [research log](https://github.com/Icold21/kron-llm/blob/main/docs/results.md).
65
-
66
- ## Next
67
-
68
- A 10M-parameter model on a 2B-token bilingual corpus (d=512, 25 layers,
69
- K=10, 8K BPE), trained on a Kaggle T4. The corpus is published; the checkpoint
70
- will appear here when the run finishes.
71
-
72
- ## Licence
73
-
74
- Code and weights: MIT. The corpus keeps its sources' licences (see the dataset
75
- card).
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # KronLLM — a compact, fully Kronecker-parameterised language model
2
+
3
+ KronLLM is a small decoder-only language model whose matrix layers are built
4
+ entirely from **Kronecker sums of rank-1 factors**. Every attention projection
5
+ and every FFN projection is a composition of `L` such factors instead of a
6
+ dense `d × d` weight, so a layer that would need 102,400 numbers needs 15,753.
7
+ The model is designed for a **6 GB consumer GPU**: the shipped configuration
8
+ trains and samples inside 4.1 GB of VRAM.
9
+
10
+ This repository is a research log as much as a model. It contains the
11
+ architecture, the training code, the measurements that killed several plausible
12
+ ideas, and the honest numbers behind each decision. Negative results are kept
13
+ because they are the reason the current design looks the way it does —
14
+ see [§8.4](#84-what-we-tried-and-rejected).
15
+
16
+ ## At a glance
17
+
18
+ | | |
19
+ |---|---|
20
+ | Best model | **Q8g-long** — 2,851,280 parameters |
21
+ | Quality | **0.7542 bits/byte** (TinyStories val, 61.4M tokens seen) |
22
+ | Shipped artifact | [`artifacts/q8g_int8.pt`](artifacts/) — 3.29 MB, int8, +0.0003 bpb |
23
+ | Training cost | 3.3 h on a GTX 1660 Ti, 0.895 s/step, **4.09 GB** peak |
24
+ | Architecture | 24 layers, d=320, 8 heads, seq 192, BPE-4096, gated Kronecker FFN |
25
+ | Tests | 102 passed, 3 skipped (CPU only) |
26
+ | Hugging Face | [model](https://huggingface.co/Icold/kron-llm) · [corpus](https://huggingface.co/datasets/Icold/kron-corpus) |
27
+ | Corpus repo | [github.com/Icold21/kron-corpus](https://github.com/Icold21/kron-corpus) |
28
+
29
+ **Read the budget, not just the number.** TinyStories is a *development bench*,
30
+ never the target corpus. Every comparison below is made at equal tokens,
31
+ because step counts alone are misleading: Q6 trained with `batch 32` and Q8g
32
+ with `batch 24`, so 10,000 steps of one is not 10,000 steps of the other.
33
+
34
+ ### The next run: 10M parameters on 2B tokens
35
+
36
+ The current checkpoint is a development bench. The real target is a **10M
37
+ model trained on a 2B-token bilingual corpus** and shipped as an int8 artifact
38
+ under 10 MB, so it fits a 6 GB GPU and runs on CPU. The reference
39
+ configuration is 9,938,886 parameters (d=512, 25 layers, d×d = 16×32
40
+ Kronecker factors, K=10, 8K bilingual BPE). Purpose, success criteria, corpus
41
+ contract and compute plan are in [`docs/goal_10m.md`](docs/goal_10m.md); the
42
+ corpus itself is published as
43
+ [**Icold21/kron-corpus**](https://github.com/Icold21/kron-corpus). Training is
44
+ scheduled for a Kaggle T4, with the 1660 Ti as the restartable fallback.
45
+
46
+ | model | parameters | tokens | bits/byte | note |
47
+ |---|---:|---:|---:|---|
48
+ | Q2 | 2,079,992 | 61.4M | 0.8391 | previous shipped model |
49
+ | Q6 | 2,473,208 | 61.4M | 0.7644 | BPE-4096 tokenizer |
50
+ | **Q8g-long** | **2,851,280** | **61.4M** | **0.7542** | **+ gated FFN** |
51
+
52
+ Same token budget, same corpus, same seed. The difference is the tokenizer and
53
+ the gated FFN, both of which are isolated in [§8.2](#82-tokenizer-and-budget-two-effects-that-are-easy-to-confuse)
54
+ and [§8.3](#83-gated-ffn-the-one-architectural-win).
55
+
56
+ ## Contents
57
+
58
+ - [1. What "Kronecker" means here](#1-what-kronecker-means-here)
59
+ - [2. Depth recurrence](#2-depth-recurrence)
60
+ - [3. The scale defect and its fix](#3-the-scale-defect-and-its-fix)
61
+ - [4. Parameter budget](#4-parameter-budget)
62
+ - [5. Data](#5-data)
63
+ - [6. Running](#6-running)
64
+ - [7. Metrics](#7-metrics)
65
+ - [8. Results](#8-results)
66
+ - [9. Hypotheses: verdicts](#9-hypotheses-verdicts)
67
+ - [10. Tests](#10-tests)
68
+ - [11. Efficiency](#11-efficiency)
69
+ - [12. Repository layout](#12-repository-layout)
70
+ - [13. Limitations](#13-limitations)
71
+ - [14. The bilingual corpus](#14-the-bilingual-corpus)
72
+ - [15. License and citation](#15-license-and-citation)
73
+
74
+ ---
75
+
76
+ ## 1. What "Kronecker" means here
77
+
78
+ A dense projection `y = W x` stores `d_out × d_in` numbers. KronLLM instead
79
+ writes the projection as a **Kronecker sum** of `K` rank-1 factor pairs:
80
+
81
+ ```
82
+ W = s·I + c · Σ_k α_k · (a_k ⊗ b_k)
83
+ ```
84
+
85
+ where `a_k ∈ R^{n1_out}` and `b_k ∈ R^{n2_out}` are the row/column factors and
86
+ `d_out = n1_out · n2_out`. The identity term is the **anchor**: it guarantees
87
+ that the layer starts near the identity map, and the factors learn a low-rank
88
+ correction on top of it. Per module the factors cost
89
+
90
+ ```
91
+ L · K · (n1_out² + n2_out²) + 2L + 1
92
+ ```
93
+
94
+ instead of `d_out · d_in`. For the shipped model (`L=4, K=6, n1=16, n2=20,
95
+ d=320`) that is **15,753 parameters per module** instead of 102,400 — 6.5×
96
+ smaller, and the saving grows with `d`.
97
+
98
+ A composition step is therefore one `d × d` matmul that *never materialises a
99
+ dense weight*: the factors are composed into a matrix on the fly (or, in
100
+ training, batched across all modules by `KronBank` and applied as a single
101
+ GEMM).
102
+
103
+ ## 2. Depth recurrence
104
+
105
+ A module is not a single factor but a **recursion over depth**:
106
+
107
+ ```
108
+ D_0 = D_init
109
+ D_l = s·D_{l-1} + c · (a_l ⊗ b_l) · γ_l l = 1 … L
110
+ P = D_L
111
+ y = γ · P x
112
+ ```
113
+
114
+ Two things matter here.
115
+
116
+ 1. `D_l` carries a **depth state** across the `L` steps, so the composition is
117
+ not a sum of independent terms but a repeated `s·D + innovation` update.
118
+ The state also flows between transformer layers (`stop_gradient` controls
119
+ whether the gradient crosses the boundary).
120
+ 2. Because all `L` steps are linear, the whole stack collapses into a single
121
+ matrix `P` that is computed once per layer and applied as one GEMM. This is
122
+ the property that makes the architecture cheap — and it is also the property
123
+ that later experiments tried (and failed) to break; see
124
+ [§8.4.2](#842-nonlinearity-inside-the-composition).
125
+
126
+ **Measured, not assumed (E8r/E27–E30, 190K params, equal tokens).** The carry
127
+ is real but small, and it is short-range:
128
+
129
+ | variant | bpb | cost |
130
+ |---|---:|---:|
131
+ | learned σ, `K_win=2` | **1.5282** | — |
132
+ | no carry (σ=0) | 1.5455 | +1.1 % |
133
+ | σ frozen at 0.5 | 1.5409 | +0.8 % |
134
+ | `K_win=4` | 1.5496 | +1.4 % |
135
+ | `K_win=0` (whole stack) | 1.5712 | +2.8 % |
136
+
137
+ The state helps when it crosses a pair of layers and hurts when it travels
138
+ further, so this is not Universal-Transformers-style depth recurrence; making
139
+ it one is worse. Keep the cheapest form. Details in `docs/results.md` §15.
140
+
141
+ ### Parameters of one module
142
+
143
+ | component | formula | count (`d=320`) |
144
+ |---|---|---:|
145
+ | factors `a_lk`, `b_lk` | `L·K·(n1² + n2²)` = `4·6·656` | 15,744 |
146
+ | `m_l` (memory) | `L` | 4 |
147
+ | `σ_l` (scale of `D_l`) | `L` | 4 |
148
+ | `γ` (ReZero) | 1 | 1 |
149
+ | **total** | | **15,753** |
150
+
151
+ Six modules per layer (`q, k, v, o, ffn, ffn_gate`) × 24 layers = 144 modules →
152
+ **2,268,432**; embedding 4,096 × 128 = 524,288 (factorised, 582,848 with the
153
+ up-projection); RMSNorm and the rest bring the total to **2,851,280**.
154
+
155
+ ## 3. The scale defect and its fix
156
+
157
+ This section is the most important one in the repository: without these
158
+ changes the architecture from the specification **does not train at all**.
159
+ Every item is a measurement, not a matter of taste.
160
+
161
+ ### 3.1 Root defect: `gain = 1/√d` makes the model the identity
162
+
163
+ With factors initialised at `gain = 1/√d = 0.088`, and the anchor dominant:
164
+
165
+ ```
166
+ ‖D_l‖_RMS ≈ 1e-3 → W_l = s·I + D_l ≈ s·I
167
+ P = W_4 W_3 W_2 W_1 ≈ s^L · I = 0.1 · I
168
+ ```
169
+
170
+ The whole Kronecker layer degenerates into multiplication by a constant.
171
+ Measured on a real 300-step TinyStories run:
172
+
173
+ | metric | value | meaning |
174
+ |---|---|---|
175
+ | `∂L/∂D` norm | **0.000** | the recurrence receives no gradient |
176
+ | `cos(W_i, W_{i−1})` | **1.000** | all layers are the same matrix |
177
+ | `eff_rank(W)` | **128** (all layers) | full algebraic rank comes from the anchor, not the factors; effective rank is far lower |
178
+ | val PPL | **20.06** | ≈ a unigram model |
179
+ | samples | "letter soup" | — |
180
+
181
+ ### 3.2 The naive fix also breaks
182
+
183
+ With `gain = 1` (spectral norm of a factor ≈ 1) the hidden state grows across
184
+ layers as `11 → 226 → 1858 → 6830`, the attention contribution drops to
185
+ 0.2–3 %, and `std(attn logits) = 0.0098`: the model again ignores context and
186
+ only rescales the residual.
187
+
188
+ ### 3.3 What was actually done
189
+
190
+ | # | specification | implemented | reason (measurement) |
191
+ |---|---|---|---|
192
+ | 1 | `gain = 1/√d` | `D` is RMS-normalised; `gain` no longer acts in the forward pass | see 3.1: otherwise `P ≈ s^L·I` and the gradient is zero |
193
+ | 2 | one `s_init = 0.1` for QKV | per-role `s_init = 1.0`, applied as `s = s_init^{1/L}` | with `s = 0.1` and `D ≈ 1e-3` each layer halves the signal |
194
+ | 3 | — | learned `σ_l` scaling of each `D_l`, init 0.1 | sets `RMS(D)` directly; `eff_rank(D)` went 1.0 → **91.3/128** |
195
+ | 4 | — | ReZero gate `γ` on the branch output, init 0.1 | the branch starts as a perturbation; without it the residual grows 600× |
196
+ | 5 | `s` shared by QKV | its own `s` per role, optionally learnable | per-role scales differ by an order of magnitude |
197
+ | 6 | SiLU = experiment E9 | SiLU after the FFN is the default | the more useful baseline |
198
+ | 7 | `amp: fp16` | `amp: none` (fp32) | fp16 autocast was **2.4× slower** on the reference GPU |
199
+ | 8 | one composition per layer | `KronBank` fuses all 144 compositions | 439 vs 505 ms/step, identical output (test `T16`) |
200
+
201
+ Items 1–4 are one mechanism: **the scale must be normalised, not set by the
202
+ initialisation.** After the fix `eff_rank(D) = 91`, `cos(W_i, W_{i−1}) = 0.72`
203
+ instead of `1.000`, and the model trains.
204
+
205
+ ## 4. Parameter budget
206
+
207
+ ```bash
208
+ python -m kronllm.cli budget # exits 1 if a configuration exceeds 10M
209
+ ```
210
+
211
+ | model | d | layers | n1/n2 | K | L | modules | parameters |
212
+ |---|---:|---:|---|---:|---:|---:|---:|
213
+ | E-series | 128 | 12 | 8/16 | 4 | 4 | 60 | 190,108 |
214
+ | Q1 | 192 | 20 | 12/16 | 5 | 4 | 100 | 926,153 |
215
+ | Q2 / Q4 | 320 | 24 | 16/20 | 6 | 4 | 120 | 2,079,992 |
216
+ | Q6 | 320 | 24 | 16/20 | 6 | 4 | 120 | 2,473,208 |
217
+ | **Q8g** | 320 | 24 | 16/20 | 6 | 4 | 144 | **2,851,280** |
218
+
219
+ The soft ceiling is 10M parameters. Q8g uses 2.85M, i.e. 28 % of the ceiling:
220
+ the limit is not the parameter count but the 6 GB of VRAM, the 61M tokens per
221
+ run, and the throughput of a mobile GPU.
222
+
223
+ ## 5. Data
224
+
225
+ All results in this README are on **TinyStories-200K**, byte-level or BPE-4096,
226
+ and this is deliberate: it is a *development bench* that separates architecture
227
+ effects from data effects, and it fits in a few minutes of GPU time.
228
+
229
+ TinyStories is **not** the target corpus. The final model is trained on a
230
+ bilingual English/Russian corpus (60/40 by bytes, science-heavy); the corpus is
231
+ published separately as
232
+ **[Icold21/kron-corpus](https://github.com/Icold21/kron-corpus)**, and its
233
+ measured composition is summarised in [§14](#14-the-bilingual-corpus).
234
+
235
+ Every conclusion below that mentions "quality" should be read as "quality on
236
+ the development bench".
237
+
238
+ ## 6. Running
239
+
240
+ ```bash
241
+ # environment
242
+ pip install -r requirements.txt
243
+
244
+ # data: downloads TinyStories, writes a byte memmap
245
+ python scripts/prepare_data.py
246
+
247
+ # BPE-4096 tokenizer + re-tokenised corpus (~4 bytes/token)
248
+ python scripts/prepare_bpe.py --vocab 4096
249
+
250
+ # training (Q8g = the shipped configuration)
251
+ python -m kronllm.cli train --exp Q8g --steps 13333 --run-dir runs/Q8g_long
252
+
253
+ # evaluation, generation, analysis
254
+ python -m kronllm.cli eval --run runs/Q8g_long
255
+ python -m kronllm.cli generate --run runs/Q8g_long \
256
+ --prompt "Once upon a time" --temperature 1.0 --top-k 8 --samples 2
257
+ python -m kronllm.cli analyze --run runs/Q8g_long
258
+ python -m kronllm.cli quantize --run runs/Q8g_long --bits 8
259
+ ```
260
+
261
+ ### Hugging Face interface
262
+
263
+ ```python
264
+ from kronllm.hf import KronLM
265
+
266
+ model = KronLM.from_run("runs/Q8g_long")
267
+ print(model.generate("Once upon a time", max_new_tokens=100, temperature=1.0, top_k=8))
268
+ ```
269
+
270
+ ### Dashboard
271
+
272
+ `python scripts/dashboard.py` starts a local Streamlit page with the training
273
+ curves, the effective-rank spectra and side-by-side samples.
274
+
275
+ ## 7. Metrics
276
+
277
+ | metric | why it is used |
278
+ |---|---|
279
+ | **bits/byte** | the only cross-tokenizer comparison; bpb = loss / ln 2 |
280
+ | PPL | convenient for humans, but tokenizer-dependent — never compare across vocabularies |
281
+ | `eff_rank(P)` | Roy–Vetterli effective rank of the composed matrix; exact rank saturates at `d`, so the effective rank is what distinguishes compositions |
282
+ | `cos_prev_max` | maximum cosine between consecutive layer matrices; → 1.0 means depth collapse (Q5) |
283
+ | `Var σ(θ)` | spread of the per-layer scale; a collapse indicator |
284
+
285
+ ## 8. Results
286
+
287
+ ### 8.1 The shipped model
288
+
289
+ **Q8g-long**: 24 layers, `d=320`, 8 heads, seq 192, BPE-4096, gated Kronecker
290
+ FFN. 61.4M tokens (13,333 steps × 24 × 192), 0.895 s/step, **4.09 GB** peak.
291
+
292
+ ```
293
+ val loss 2.0693 PPL 7.919 bpb 0.7542 eff_rank(P) 144.8
294
+ ```
295
+
296
+ The curve was still falling at the last step (0.7553 → 0.7542), so 61.4M tokens
297
+ is a budget choice, not a measured ceiling.
298
+
299
+ Samples (`T=1.0`, `top_k=8`):
300
+
301
+ > Once upon a time there was a little girl named Lily. She was three years old
302
+ > and loved to explore. One day she went to the beach and saw a big, wide waves.
303
+ > She was excited and wanted to go on a boat to explore the water.
304
+
305
+ > Once upon a time there was a little girl named Lily. She had a big box full of
306
+ > toys and she loved to play with her toys. One day, she found a big, red book
307
+ > under a tree. She picked it up and looked at it. It was so bright and beautiful!
308
+
309
+ Artifacts:
310
+
311
+ | file | size | quality |
312
+ |---|---:|---|
313
+ | `artifacts/q8g_int8.pt` | 3.29 MB | +0.0003 bpb vs fp32 (3.55× smaller) |
314
+ | `artifacts/q2_int8.pt` | 2.34 MB | previous model, kept for comparison |
315
+ | `runs/Q8g_long/best.pt` | 11.1 MB | fp32 reference checkpoint |
316
+
317
+ int8 quantisation is per-channel symmetric and leaves the Kronecker factors
318
+ intact; the composed matrix is rebuilt in fp32 at load time, which is why the
319
+ delta is 3e-4 rather than the usual 1e-2.
320
+
321
+ ### 8.2 Tokenizer and budget: two effects that are easy to confuse
322
+
323
+ BPE-1024 compresses TinyStories to 3.07 bytes/token; BPE-4096 reaches
324
+ **3.96 bytes/token** on the same corpus. At a fixed parameter count that alone
325
+ is worth ~8 % bpb, because each token carries more text:
326
+
327
+ | model | tokenizer | parameters | steps | tokens | bpb |
328
+ |---|---|---:|---:|---:|---:|
329
+ | Q2 | BPE-1024 | 2,079,992 | 4,000 | 24.6M | 0.962 |
330
+ | Q4 | BPE-1024 | 2,079,992 | 4,000 | 24.6M | 0.9248 |
331
+ | Q6 | BPE-4096 | 2,473,208 | 4,000 | 24.6M | 0.8489 |
332
+ | Q6 | BPE-4096 | 2,473,208 | 10,000 | 61.4M | 0.7644 |
333
+ | **Q8g** | **BPE-4096** | **2,851,280** | **13,333** | **61.4M** | **0.7542** |
334
+
335
+ Q4 vs Q2 isolates the training change (full backprop instead of truncated):
336
+ **−3.9 %** at equal budget. Q6 vs Q4 isolates the tokenizer: **−8.2 %**, and
337
+ the only extra parameters are 393K in the factorised embedding.
338
+
339
+ To stay inside 6 GB with a 4,096-token vocabulary, the loss is computed in
340
+ checkpointed row chunks (`model.loss_chunk`), so the `24 × 192 × 4096` logits
341
+ tensor is never materialised. Peak VRAM is 4.09 GB.
342
+
343
+ ### 8.3 Gated FFN: the one architectural win
344
+
345
+ `ffn_gated=True` adds a second Kronecker module and multiplies it by the first
346
+ through SiLU: `f = (W_a x) · SiLU(W_b x)`. Two controls isolate the effect:
347
+
348
+ | run | layout | parameters | tokens | bpb |
349
+ |---|---|---:|---:|---:|
350
+ | Q6 | 24 layers × 5 modules | 2,473,208 | 24.6M | 0.8489 |
351
+ | **Q8w** | **20 layers × 6 modules** | **2,470,328** | **24.6M** | **0.8359** (−1.5 %) |
352
+ | Q8g | 24 layers × 6 modules | 2,851,280 | 24.6M | 0.8318 (−2.0 %) |
353
+
354
+ Q8w is the honest control: same parameters (0.1 % apart), same number of matrix
355
+ modules per token (120), same tokens. The only difference from Q6 is how they
356
+ are arranged — 20 layers with a gate versus 24 layers without. The gate wins
357
+ **1.5 %**, and that is architecture, not capacity. The remaining 0.5 % of Q8g
358
+ is the extra 378K parameters.
359
+
360
+ So the design rule is: **a gated FFN beats four extra layers at equal budget.**
361
+ Everything else in the model is unchanged from Q2.
362
+
363
+ ### 8.4 What we tried and rejected
364
+
365
+ Each entry lists the hypothesis, the measurement, and why the idea was dropped.
366
+ The weights are not kept — only the numbers, because the numbers are the result.
367
+
368
+ #### 8.4.1 Composition: sum, grouped, alternating
369
+
370
+ *Hypothesis:* a fully additive composition (`P = sI + Σ α_l D_l`) has a much
371
+ higher effective rank than the product, so it should be more expressive.
372
+
373
+ | composition | mult. depth | bpb (TinyStories) | eff_rank(P) |
374
+ |---|---:|---:|---:|
375
+ | product (E8) | 3 | **1.479** | 60.1 |
376
+ | sum (E14) | 0 | 1.5508 | 97.7 |
377
+ | dynamic gate (E15) | 0 | 1.5091 | 96.4 |
378
+ | Kron gate (E16) | 0 | 1.5504 | 97.4 |
379
+
380
+ The sum reaches rank 97.7 and still loses 4.9 % to the product. Rank was never
381
+ the bottleneck. Grouped and alternating compositions on WikiText-103 initially
382
+ looked 1.3 % better than the product (2.1561 / 2.1578 vs 2.1848), but **a second
383
+ seed reversed the sign**: product 2.1559 vs grouped 2.1624. The seed-to-seed
384
+ spread (~0.03 bpb) is larger than the effect, so the "win" was noise.
385
+
386
+ *Verdict:* product stays. The rank ordering is stable across seeds
387
+ (product 60.5 < grouped 79.2), so it remains a useful diagnostic — just not a
388
+ predictor of quality.
389
+
390
+ #### 8.4.2 Nonlinearity inside the composition
391
+
392
+ *Hypothesis:* the layer is linear by construction; putting an activation
393
+ between composition steps turns it into a small deep network inside a single
394
+ matrix, which should be a large gain per parameter.
395
+
396
+ | variant | nonlinearities per composition | bpb | cost |
397
+ |---|---:|---:|---:|
398
+ | product (E8) | 0 | **1.4789** | 3.67 MFLOP/token |
399
+ | E26, k=4 | 1 | 1.5675 | 3.67 |
400
+ | E25, k=2 | 2 | 1.5189 | 5.63 |
401
+ | E26b, additive k=4 | 1 | 1.5906 | 3.67 |
402
+
403
+ The nonlinearity cannot be fused away: `L` factors no longer collapse into one
404
+ matrix, so the layer costs `ceil(L/k)` GEMMs instead of one. Even at `k = L`,
405
+ where the FLOP count is unchanged, wall-clock time rose from 0.405 to 0.655
406
+ s/step (+62 %) — the loss of fusion, not the activation.
407
+
408
+ The interesting part is the ordering: *more* nonlinearity helped (k=2 beat k=4)
409
+ but every variant still lost to the purely linear product, and the additive
410
+ formulation — which keeps the linear skeleton — was the worst of all. The extra
411
+ branch doubles the output scale, and the model spends capacity compensating
412
+ instead of learning. `eff_rank(P)` rose to 62.3 while bpb got worse, the third
413
+ independent sign that rank is not the constraint.
414
+
415
+ *Verdict:* closed. The layer is linear, and at this scale that is a feature.
416
+
417
+ #### 8.4.3 Bilinear feature interaction (GLU)
418
+
419
+ *Hypothesis:* cross-feature products `(W_a x) ⊙ (W_b x)` give interactions that
420
+ a linear composition cannot express, at ~one module's parameter cost
421
+ (`3·d·r = 15,360` at `r=16` versus 15,753).
422
+
423
+ | run | FFN | layers | parameters | bpb |
424
+ |---|---|---:|---:|---:|
425
+ | Q6 | linear Kron | 24 | 2,473,208 | **0.8489** |
426
+ | Q9 | GLU only | 24 | 2,463,776 | 0.8740 |
427
+ | Q9k | GLU + Kron | 20 | 2,462,468 | 0.8673 |
428
+
429
+ Both lost (2.9 % and 2.2 %), and `cos_prev_max` rose to **0.939** in Q9 —
430
+ the layers became more similar, not more expressive. Adding the linear
431
+ Kronecker term back helped a little, but not enough.
432
+
433
+ A note on methodology: the first Q9 run was invalid because the generic
434
+ `std=0.02` initialisation left the bilinear branch 300× smaller than a Kron
435
+ module (output std 0.005 vs 1.76), i.e. switched off. The branch is now
436
+ initialised to unit activation and a regression test guards it. The numbers
437
+ above are from the corrected run — a false negative would have been worse than
438
+ no experiment.
439
+
440
+ *Verdict:* closed at this scale.
441
+
442
+ #### 8.4.4 Rank: `K = 24`
443
+
444
+ *Hypothesis:* more factors per module (`K=24` instead of 6) buy expressiveness
445
+ for a few extra parameters.
446
+
447
+ Q5 (`K=24`, 7.75M parameters) scored **0.9530** against Q4's **0.9248** at the
448
+ same budget, and `cos_prev_max` reached **0.99999**: every layer learned the
449
+ same matrix. The extra factors made the innovation dominate the anchor until
450
+ depth differences vanished.
451
+
452
+ *Verdict:* closed. `K=6` is the useful range at this width.
453
+
454
+ #### 8.4.5 Kronecker embedding
455
+
456
+ *Hypothesis:* the embedding is 524K of the 2.47M parameters; a Kronecker
457
+ embedding would shrink it.
458
+
459
+ Q7 scored **0.9533** against Q6's **0.8489** — 12 % worse. The factorised
460
+ `V × rank → rank × d` form is already cheap (393K for the wider vocabulary),
461
+ and the Kronecker structure costs too much in the input representation.
462
+
463
+ *Verdict:* closed; the factorised embedding stays.
464
+
465
+ #### 8.4.6 Recursive weight sharing
466
+
467
+ *Hypothesis:* reuse `n_unique` blocks for `r` passes and get depth for free.
468
+
469
+ | run | unique | passes | effective depth | parameters | bpb |
470
+ |---|---:|---:|---:|---:|---:|
471
+ | E8 | 12 | 1 | 12 | 190,108 | **1.479** |
472
+ | E19 | 6 | 2 | 12 | 111,502 | 1.717 |
473
+ | E21 | 6 | 1–2 random | 6–12 | 111,758 | 1.734 @ r=1 · 1.718 @ r=2 |
474
+ | E20 | 12 | 2 | 24 | 190,108 | **1.427** |
475
+ | E22 | 6 | 4 | 24 | 111,502 | 1.521 |
476
+
477
+ Three findings, all negative for the "free depth" hope:
478
+
479
+ 1. **Sharing costs quality.** E19 at 59 % of the parameters and the same
480
+ effective depth is 16 % worse than E8.
481
+ 2. **Depth helps, but at 2× compute.** E20 gains 3.5 % over E8 with the same
482
+ parameters and twice the FLOPs per token.
483
+ 3. **Randomised training makes recursion usable.** E21 (r ~ U{1,2}, plus a
484
+ per-pass embedding) matches E19 at r=2 and stays coherent at r=1 (1.734),
485
+ where E19 collapses to 2.554. One model, two usable depths.
486
+
487
+ *Verdict:* not a default. Recursion is a deployment/compute trade-off
488
+ (adaptive compute), not a quality upgrade.
489
+
490
+ #### 8.4.7 Input-dependent channel gates
491
+
492
+ E15 (dense dynamic gate) reached **1.5091** and E16 (Kronecker gate) **1.5504**
493
+ against E8's **1.4789**. Both cost 2.4–2.6× the step time, because the gate
494
+ depends on the input and the composition can no longer be precomputed.
495
+
496
+ *Verdict:* closed.
497
+
498
+ #### 8.4.8 What did not survive contact with the data, in one line each
499
+
500
+ - **Sum composition**: higher rank, worse quality (1.5508 vs 1.479).
501
+ - **Hybrid compositions**: a 1.3 % win on one seed, reversed on the second.
502
+ - **Dynamic gates**: 3× slower, 2 % worse.
503
+ - **Nonlinear composition**: 2.7–7.5 % worse in every form.
504
+ - **Bilinear GLU FFN**: 2.2–2.9 % worse, with layer collapse.
505
+ - **`K=24`**: 3 % worse and complete depth collapse.
506
+ - **Kronecker embedding**: 12 % worse.
507
+ - **Weight sharing**: 16 % worse at equal effective depth.
508
+ - **KV cache**: exact (2.8e-7) but no speed-up on this GPU.
509
+ - **Factorised `P` apply**: exact, but 0.20–0.31× the speed of one dense GEMM.
510
+
511
+ The pattern is consistent: **what wins is the stuff that keeps the layer
512
+ linear and fusable**; what loses is anything that breaks the single-GEMM
513
+ composition, regardless of how much rank or nonlinearity it promises.
514
+
515
+ ## 9. Hypotheses: verdicts
516
+
517
+ | # | hypothesis | verdict |
518
+ |---|---|---|
519
+ | H1 | normalising the branch scale is what makes the Kron layer trainable | **confirmed** (PPL 20.06 → 12.3 after the fix) |
520
+ | H2 | the depth recurrence adds useful state beyond independent factors | **untested** — no run removes the carry (E27/E28 pending); E8 vs E14 (1.479 vs 1.551, −4.9 %) tests the composition, not the recurrence |
521
+ | H3 | a wider tokenizer is worth more than extra parameters | **confirmed** (BPE-4096: −8.2 % at equal params) |
522
+ | H4 | effective rank is monotone in additive depth | **confirmed empirically** (60/60 modules; survives seeds) — but rank does not predict bpb |
523
+ | H5 | hybrid compositions beat the product | **rejected** (seed noise) |
524
+ | H6 | randomised recursion gives one model with usable adaptive compute | **confirmed** (r=1 1.734 vs 2.554 untrained) |
525
+ | H7 | extra depth at equal parameters is a clear win | **partially**: +3.5 %, but 2× compute |
526
+
527
+ H4 is deliberately called a hypothesis, not a theorem: effective rank is not
528
+ monotone under matrix products, and counterexamples exist for exact rank. The
529
+ observed ordering (product < alternating < grouped < sum) is empirical.
530
+
531
+ ## 10. Tests
532
+
533
+ ```bash
534
+ CUDA_VISIBLE_DEVICES=-1 python -m pytest tests
535
+ # 102 passed, 3 skipped
536
+ ```
537
+
538
+ The suite runs on CPU in ~20 s and covers: the Kronecker forward against a
539
+ naive reference, gradient flow through the recurrence, `KronBank` prelude
540
+ equivalence, the KV cache (exact to 2.8e-7), quantisation round-trips, the HF
541
+ wrapper, and regression guards for the failures described above (dead
542
+ composition scale, dead GLU branch, depth collapse).
543
+
544
+ ## 11. Efficiency
545
+
546
+ Measured on a GTX 1660 Ti (6 GB, sm75; no FlashAttention, no Triton).
547
+
548
+ | model | batch × seq | s/step | µs/token | peak VRAM |
549
+ |---|---|---|---:|---:|
550
+ | Q6 | 32 × 192 | 0.847 | 138 | 4.28 GB |
551
+ | Q8g | 24 × 192 | 0.895 | 194 | 4.09 GB |
552
+
553
+ The gated FFN adds a sixth module per layer (+20 % matmuls) and the smaller
554
+ batch loses some GPU efficiency, so cost per token rises ~40 % while peak
555
+ memory falls, because 24 layers with batch 24 fit better than 32.
556
+
557
+ Two serving optimisations are implemented and exact but not useful here:
558
+
559
+ - **KV cache** — matches the no-cache path to 2.8e-7, but the 1660 Ti is
560
+ compute-bound at these sizes, so decoding does not speed up.
561
+ - **Factorised apply** (`y = x·Pᵀ` computed factor by factor) — mathematically
562
+ identical, but only 0.20–0.31× the speed of one dense GEMM. Composing once
563
+ and multiplying once is the fast path.
564
+
565
+ `torch.compile` is unavailable in this environment (no Triton on Windows), so
566
+ all speed-ups are structural, not compiler-driven.
567
+
568
+ ## 12. Repository layout
569
+
570
+ ```
571
+ kronllm/ library: kron.py (composition), model.py, train.py,
572
+ config.py (experiments), data.py, tokenizer.py,
573
+ bilingual.py (corpus pipeline), quantization.py,
574
+ hf.py, cli.py
575
+ scripts/ data prep (TinyStories, BPE, WikiText, SFT,
576
+ build_bilingual.py, prepare_bilingual_bin.py),
577
+ dashboard
578
+ tests/ 102 tests, CPU-only
579
+ docs/ architecture.md, results.md (full tables),
580
+ data_plan.md (bilingual corpus), roadmap.md
581
+ artifacts/ q8g_int8.pt (shipped), q2_int8.pt (previous)
582
+ runs/ per-experiment logs and metrics; only the two best
583
+ checkpoints are kept (Q8g_long, Q6_10k)
584
+ notebooks/ walkthrough of the architecture and the measurements
585
+ current_progress.md rolling log of the experiment queue
586
+ ```
587
+
588
+ ## 13. Limitations
589
+
590
+ - **Development bench only.** Every quality number is TinyStories, English.
591
+ Nothing here proves the same ordering on the bilingual corpus; the corpus
592
+ pilot is built but no model has been trained on it yet.
593
+ - **One GPU, one seed.** Most experiments are single-seed; the one seed control
594
+ performed (hybrid compositions) reversed a published-looking result, which is
595
+ a warning about the rest.
596
+ - **No FlashAttention, no Triton.** Throughput is well below what the same
597
+ model would reach with a fused kernel.
598
+ - **512-token context at most.** RoPE is implemented, but training used 192–256
599
+ tokens; long-context behaviour is untested.
600
+ - **No instruction tuning in the shipped model.** S1 was an SFT probe on
601
+ Dolly-15k, not a product.
602
+ - **int8 only.** int4 is implemented but the delta at this size is not yet
603
+ worth the extra complexity.
604
+
605
+ ## 14. The bilingual corpus
606
+
607
+ The corpus now lives in its own repository:
608
+ **[Icold21/kron-corpus](https://github.com/Icold21/kron-corpus)** — a
609
+ 2B-token English/Russian corpus with a science accent, published with the full
610
+ pipeline, the per-shard manifest and the measured composition.
611
+
612
+ | stage | result |
613
+ |---|---|
614
+ | fetched text | 7.3 GB · ~795k documents |
615
+ | target size | **~2.0B tokens** (8K BPE, 3.34 bytes/token measured) |
616
+ | by language | EN 4.9 GB · RU 2.4 GB at fetch; the mix enforces 60/40 by bytes |
617
+ | domains | math 25 %, science 14 %, ML 12 %, logic 10 %, code 15 %, literature 11 %, encyclopedic 9 %, CS/engineering 3 %, humour 1 %, identity 1 % |
618
+ | dedup | MinHash (Jaccard 0.8) · held-out 2 % per language, disjoint |
619
+ | tokenizer | 8K byte-level BPE, bilingual, fertility recorded per language |
620
+ | chat template | `<\|system\|>` 8187 · `<\|user\|>` 8188 · `<\|assistant\|>` 8189 · `<\|eot\|>` 8190 · `<\|endoftext\|>` 8191 |
621
+ | identity slice | 800 plain + 850 pastiche per language, no numeric facts |
622
+
623
+ **One finding fixed the ratio policy.** Bytes per token is not comparable across
624
+ scripts: both languages receive ≈2.8 characters per token, but Cyrillic costs
625
+ two UTF-8 bytes per character. A 60/40 *byte* split therefore gives only
626
+ **33 % of tokens** to Russian, not the 45–47 % the original plan predicted. The
627
+ contract is now stated in bytes (stable, tokenizer-independent) and the token
628
+ share is recorded next to it in `tokenizer_meta.json`.
629
+
630
+ Known gaps are reported rather than hidden, and the corpus README repeats them:
631
+ logic lands at ~3 % against a 10 % quota (only formal proofs scale), ML at
632
+ ~5 % against 12 % (most prose is paywalled), and Russian code is absent by
633
+ nature. The mixer emits these as shortfalls; nothing is padded with
634
+ mislabelled text.
635
+
636
+ Next steps:
637
+
638
+ 1. train **Q10** (9,938,886 params, d=512, 25 layers, K=10, seq 512) on the
639
+ corpus on two Kaggle T4s with `torchrun --nproc_per_node=2`, one epoch =
640
+ 82k steps = 2.015B tokens, checkpoint every 500 steps;
641
+ 2. close the tokenizer question (8K vs 16K) before the run starts;
642
+ 3. package the result, then compare against public small-model baselines
643
+ before the SFT stage.
644
+
645
+ **Reproducibility details that matter for the run.** The epoch sampler is
646
+ sharded: all ranks build the same permutation from a shared seed and take
647
+ `rank::world`, so one epoch is one full pass instead of the ~63 % a random
648
+ sampler covers. DDP writes logs and checkpoints only on rank 0; `last.pt`
649
+ carries the optimizer and the step counter, so a killed Kaggle session resumes
650
+ exactly where it stopped.
651
+
652
+ The long-term plan, including the SFT families and the comparison phase, is in
653
+ [docs/roadmap.md](docs/roadmap.md).
654
+
655
+ ## 15. License and citation
656
+
657
+ MIT — see [LICENSE](LICENSE).
658
+
659
+ ```bibtex
660
+ @misc{kronllm2026,
661
+ title = {KronLLM: a compact fully Kronecker-parameterised language model},
662
+ author = {Ivan Kholodilo},
663
+ year = {2026},
664
+ url = {https://github.com/Icold21/kron-llm}
665
+ }
666
+ ```