Upload README.md with huggingface_hub
Browse files
README.md
CHANGED
|
@@ -1,75 +1,666 @@
|
|
| 1 |
-
-
|
| 2 |
-
|
| 3 |
-
|
| 4 |
-
|
| 5 |
-
|
| 6 |
-
|
| 7 |
-
|
| 8 |
-
|
| 9 |
-
|
| 10 |
-
|
| 11 |
-
|
| 12 |
-
|
| 13 |
-
|
| 14 |
-
---
|
| 15 |
-
|
| 16 |
-
#
|
| 17 |
-
|
| 18 |
-
|
| 19 |
-
|
| 20 |
-
|
| 21 |
-
|
| 22 |
-
|
| 23 |
-
|
| 24 |
-
|
| 25 |
-
|
| 26 |
-
|
| 27 |
-
|
| 28 |
-
|
| 29 |
-
|
| 30 |
-
|
| 31 |
-
|
| 32 |
-
|
| 33 |
-
|
| 34 |
-
|
| 35 |
-
|
| 36 |
-
is
|
| 37 |
-
|
| 38 |
-
|
| 39 |
-
|
| 40 |
-
|
| 41 |
-
|
| 42 |
-
|
| 43 |
-
|
| 44 |
-
|
| 45 |
-
|
| 46 |
-
model
|
| 47 |
-
|
| 48 |
-
|
| 49 |
-
|
| 50 |
-
|
| 51 |
-
|
| 52 |
-
|
| 53 |
-
|
| 54 |
-
|
| 55 |
-
|
| 56 |
-
|
| 57 |
-
|
| 58 |
-
|
| 59 |
-
|
| 60 |
-
|
| 61 |
-
|
| 62 |
-
|
| 63 |
-
|
| 64 |
-
[
|
| 65 |
-
|
| 66 |
-
|
| 67 |
-
|
| 68 |
-
|
| 69 |
-
|
| 70 |
-
|
| 71 |
-
|
| 72 |
-
|
| 73 |
-
|
| 74 |
-
|
| 75 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
# KronLLM — a compact, fully Kronecker-parameterised language model
|
| 2 |
+
|
| 3 |
+
KronLLM is a small decoder-only language model whose matrix layers are built
|
| 4 |
+
entirely from **Kronecker sums of rank-1 factors**. Every attention projection
|
| 5 |
+
and every FFN projection is a composition of `L` such factors instead of a
|
| 6 |
+
dense `d × d` weight, so a layer that would need 102,400 numbers needs 15,753.
|
| 7 |
+
The model is designed for a **6 GB consumer GPU**: the shipped configuration
|
| 8 |
+
trains and samples inside 4.1 GB of VRAM.
|
| 9 |
+
|
| 10 |
+
This repository is a research log as much as a model. It contains the
|
| 11 |
+
architecture, the training code, the measurements that killed several plausible
|
| 12 |
+
ideas, and the honest numbers behind each decision. Negative results are kept
|
| 13 |
+
because they are the reason the current design looks the way it does —
|
| 14 |
+
see [§8.4](#84-what-we-tried-and-rejected).
|
| 15 |
+
|
| 16 |
+
## At a glance
|
| 17 |
+
|
| 18 |
+
| | |
|
| 19 |
+
|---|---|
|
| 20 |
+
| Best model | **Q8g-long** — 2,851,280 parameters |
|
| 21 |
+
| Quality | **0.7542 bits/byte** (TinyStories val, 61.4M tokens seen) |
|
| 22 |
+
| Shipped artifact | [`artifacts/q8g_int8.pt`](artifacts/) — 3.29 MB, int8, +0.0003 bpb |
|
| 23 |
+
| Training cost | 3.3 h on a GTX 1660 Ti, 0.895 s/step, **4.09 GB** peak |
|
| 24 |
+
| Architecture | 24 layers, d=320, 8 heads, seq 192, BPE-4096, gated Kronecker FFN |
|
| 25 |
+
| Tests | 102 passed, 3 skipped (CPU only) |
|
| 26 |
+
| Hugging Face | [model](https://huggingface.co/Icold/kron-llm) · [corpus](https://huggingface.co/datasets/Icold/kron-corpus) |
|
| 27 |
+
| Corpus repo | [github.com/Icold21/kron-corpus](https://github.com/Icold21/kron-corpus) |
|
| 28 |
+
|
| 29 |
+
**Read the budget, not just the number.** TinyStories is a *development bench*,
|
| 30 |
+
never the target corpus. Every comparison below is made at equal tokens,
|
| 31 |
+
because step counts alone are misleading: Q6 trained with `batch 32` and Q8g
|
| 32 |
+
with `batch 24`, so 10,000 steps of one is not 10,000 steps of the other.
|
| 33 |
+
|
| 34 |
+
### The next run: 10M parameters on 2B tokens
|
| 35 |
+
|
| 36 |
+
The current checkpoint is a development bench. The real target is a **10M
|
| 37 |
+
model trained on a 2B-token bilingual corpus** and shipped as an int8 artifact
|
| 38 |
+
under 10 MB, so it fits a 6 GB GPU and runs on CPU. The reference
|
| 39 |
+
configuration is 9,938,886 parameters (d=512, 25 layers, d×d = 16×32
|
| 40 |
+
Kronecker factors, K=10, 8K bilingual BPE). Purpose, success criteria, corpus
|
| 41 |
+
contract and compute plan are in [`docs/goal_10m.md`](docs/goal_10m.md); the
|
| 42 |
+
corpus itself is published as
|
| 43 |
+
[**Icold21/kron-corpus**](https://github.com/Icold21/kron-corpus). Training is
|
| 44 |
+
scheduled for a Kaggle T4, with the 1660 Ti as the restartable fallback.
|
| 45 |
+
|
| 46 |
+
| model | parameters | tokens | bits/byte | note |
|
| 47 |
+
|---|---:|---:|---:|---|
|
| 48 |
+
| Q2 | 2,079,992 | 61.4M | 0.8391 | previous shipped model |
|
| 49 |
+
| Q6 | 2,473,208 | 61.4M | 0.7644 | BPE-4096 tokenizer |
|
| 50 |
+
| **Q8g-long** | **2,851,280** | **61.4M** | **0.7542** | **+ gated FFN** |
|
| 51 |
+
|
| 52 |
+
Same token budget, same corpus, same seed. The difference is the tokenizer and
|
| 53 |
+
the gated FFN, both of which are isolated in [§8.2](#82-tokenizer-and-budget-two-effects-that-are-easy-to-confuse)
|
| 54 |
+
and [§8.3](#83-gated-ffn-the-one-architectural-win).
|
| 55 |
+
|
| 56 |
+
## Contents
|
| 57 |
+
|
| 58 |
+
- [1. What "Kronecker" means here](#1-what-kronecker-means-here)
|
| 59 |
+
- [2. Depth recurrence](#2-depth-recurrence)
|
| 60 |
+
- [3. The scale defect and its fix](#3-the-scale-defect-and-its-fix)
|
| 61 |
+
- [4. Parameter budget](#4-parameter-budget)
|
| 62 |
+
- [5. Data](#5-data)
|
| 63 |
+
- [6. Running](#6-running)
|
| 64 |
+
- [7. Metrics](#7-metrics)
|
| 65 |
+
- [8. Results](#8-results)
|
| 66 |
+
- [9. Hypotheses: verdicts](#9-hypotheses-verdicts)
|
| 67 |
+
- [10. Tests](#10-tests)
|
| 68 |
+
- [11. Efficiency](#11-efficiency)
|
| 69 |
+
- [12. Repository layout](#12-repository-layout)
|
| 70 |
+
- [13. Limitations](#13-limitations)
|
| 71 |
+
- [14. The bilingual corpus](#14-the-bilingual-corpus)
|
| 72 |
+
- [15. License and citation](#15-license-and-citation)
|
| 73 |
+
|
| 74 |
+
---
|
| 75 |
+
|
| 76 |
+
## 1. What "Kronecker" means here
|
| 77 |
+
|
| 78 |
+
A dense projection `y = W x` stores `d_out × d_in` numbers. KronLLM instead
|
| 79 |
+
writes the projection as a **Kronecker sum** of `K` rank-1 factor pairs:
|
| 80 |
+
|
| 81 |
+
```
|
| 82 |
+
W = s·I + c · Σ_k α_k · (a_k ⊗ b_k)
|
| 83 |
+
```
|
| 84 |
+
|
| 85 |
+
where `a_k ∈ R^{n1_out}` and `b_k ∈ R^{n2_out}` are the row/column factors and
|
| 86 |
+
`d_out = n1_out · n2_out`. The identity term is the **anchor**: it guarantees
|
| 87 |
+
that the layer starts near the identity map, and the factors learn a low-rank
|
| 88 |
+
correction on top of it. Per module the factors cost
|
| 89 |
+
|
| 90 |
+
```
|
| 91 |
+
L · K · (n1_out² + n2_out²) + 2L + 1
|
| 92 |
+
```
|
| 93 |
+
|
| 94 |
+
instead of `d_out · d_in`. For the shipped model (`L=4, K=6, n1=16, n2=20,
|
| 95 |
+
d=320`) that is **15,753 parameters per module** instead of 102,400 — 6.5×
|
| 96 |
+
smaller, and the saving grows with `d`.
|
| 97 |
+
|
| 98 |
+
A composition step is therefore one `d × d` matmul that *never materialises a
|
| 99 |
+
dense weight*: the factors are composed into a matrix on the fly (or, in
|
| 100 |
+
training, batched across all modules by `KronBank` and applied as a single
|
| 101 |
+
GEMM).
|
| 102 |
+
|
| 103 |
+
## 2. Depth recurrence
|
| 104 |
+
|
| 105 |
+
A module is not a single factor but a **recursion over depth**:
|
| 106 |
+
|
| 107 |
+
```
|
| 108 |
+
D_0 = D_init
|
| 109 |
+
D_l = s·D_{l-1} + c · (a_l ⊗ b_l) · γ_l l = 1 … L
|
| 110 |
+
P = D_L
|
| 111 |
+
y = γ · P x
|
| 112 |
+
```
|
| 113 |
+
|
| 114 |
+
Two things matter here.
|
| 115 |
+
|
| 116 |
+
1. `D_l` carries a **depth state** across the `L` steps, so the composition is
|
| 117 |
+
not a sum of independent terms but a repeated `s·D + innovation` update.
|
| 118 |
+
The state also flows between transformer layers (`stop_gradient` controls
|
| 119 |
+
whether the gradient crosses the boundary).
|
| 120 |
+
2. Because all `L` steps are linear, the whole stack collapses into a single
|
| 121 |
+
matrix `P` that is computed once per layer and applied as one GEMM. This is
|
| 122 |
+
the property that makes the architecture cheap — and it is also the property
|
| 123 |
+
that later experiments tried (and failed) to break; see
|
| 124 |
+
[§8.4.2](#842-nonlinearity-inside-the-composition).
|
| 125 |
+
|
| 126 |
+
**Measured, not assumed (E8r/E27–E30, 190K params, equal tokens).** The carry
|
| 127 |
+
is real but small, and it is short-range:
|
| 128 |
+
|
| 129 |
+
| variant | bpb | cost |
|
| 130 |
+
|---|---:|---:|
|
| 131 |
+
| learned σ, `K_win=2` | **1.5282** | — |
|
| 132 |
+
| no carry (σ=0) | 1.5455 | +1.1 % |
|
| 133 |
+
| σ frozen at 0.5 | 1.5409 | +0.8 % |
|
| 134 |
+
| `K_win=4` | 1.5496 | +1.4 % |
|
| 135 |
+
| `K_win=0` (whole stack) | 1.5712 | +2.8 % |
|
| 136 |
+
|
| 137 |
+
The state helps when it crosses a pair of layers and hurts when it travels
|
| 138 |
+
further, so this is not Universal-Transformers-style depth recurrence; making
|
| 139 |
+
it one is worse. Keep the cheapest form. Details in `docs/results.md` §15.
|
| 140 |
+
|
| 141 |
+
### Parameters of one module
|
| 142 |
+
|
| 143 |
+
| component | formula | count (`d=320`) |
|
| 144 |
+
|---|---|---:|
|
| 145 |
+
| factors `a_lk`, `b_lk` | `L·K·(n1² + n2²)` = `4·6·656` | 15,744 |
|
| 146 |
+
| `m_l` (memory) | `L` | 4 |
|
| 147 |
+
| `σ_l` (scale of `D_l`) | `L` | 4 |
|
| 148 |
+
| `γ` (ReZero) | 1 | 1 |
|
| 149 |
+
| **total** | | **15,753** |
|
| 150 |
+
|
| 151 |
+
Six modules per layer (`q, k, v, o, ffn, ffn_gate`) × 24 layers = 144 modules →
|
| 152 |
+
**2,268,432**; embedding 4,096 × 128 = 524,288 (factorised, 582,848 with the
|
| 153 |
+
up-projection); RMSNorm and the rest bring the total to **2,851,280**.
|
| 154 |
+
|
| 155 |
+
## 3. The scale defect and its fix
|
| 156 |
+
|
| 157 |
+
This section is the most important one in the repository: without these
|
| 158 |
+
changes the architecture from the specification **does not train at all**.
|
| 159 |
+
Every item is a measurement, not a matter of taste.
|
| 160 |
+
|
| 161 |
+
### 3.1 Root defect: `gain = 1/√d` makes the model the identity
|
| 162 |
+
|
| 163 |
+
With factors initialised at `gain = 1/√d = 0.088`, and the anchor dominant:
|
| 164 |
+
|
| 165 |
+
```
|
| 166 |
+
‖D_l‖_RMS ≈ 1e-3 → W_l = s·I + D_l ≈ s·I
|
| 167 |
+
P = W_4 W_3 W_2 W_1 ≈ s^L · I = 0.1 · I
|
| 168 |
+
```
|
| 169 |
+
|
| 170 |
+
The whole Kronecker layer degenerates into multiplication by a constant.
|
| 171 |
+
Measured on a real 300-step TinyStories run:
|
| 172 |
+
|
| 173 |
+
| metric | value | meaning |
|
| 174 |
+
|---|---|---|
|
| 175 |
+
| `∂L/∂D` norm | **0.000** | the recurrence receives no gradient |
|
| 176 |
+
| `cos(W_i, W_{i−1})` | **1.000** | all layers are the same matrix |
|
| 177 |
+
| `eff_rank(W)` | **128** (all layers) | full algebraic rank comes from the anchor, not the factors; effective rank is far lower |
|
| 178 |
+
| val PPL | **20.06** | ≈ a unigram model |
|
| 179 |
+
| samples | "letter soup" | — |
|
| 180 |
+
|
| 181 |
+
### 3.2 The naive fix also breaks
|
| 182 |
+
|
| 183 |
+
With `gain = 1` (spectral norm of a factor ≈ 1) the hidden state grows across
|
| 184 |
+
layers as `11 → 226 → 1858 → 6830`, the attention contribution drops to
|
| 185 |
+
0.2–3 %, and `std(attn logits) = 0.0098`: the model again ignores context and
|
| 186 |
+
only rescales the residual.
|
| 187 |
+
|
| 188 |
+
### 3.3 What was actually done
|
| 189 |
+
|
| 190 |
+
| # | specification | implemented | reason (measurement) |
|
| 191 |
+
|---|---|---|---|
|
| 192 |
+
| 1 | `gain = 1/√d` | `D` is RMS-normalised; `gain` no longer acts in the forward pass | see 3.1: otherwise `P ≈ s^L·I` and the gradient is zero |
|
| 193 |
+
| 2 | one `s_init = 0.1` for QKV | per-role `s_init = 1.0`, applied as `s = s_init^{1/L}` | with `s = 0.1` and `D ≈ 1e-3` each layer halves the signal |
|
| 194 |
+
| 3 | — | learned `σ_l` scaling of each `D_l`, init 0.1 | sets `RMS(D)` directly; `eff_rank(D)` went 1.0 → **91.3/128** |
|
| 195 |
+
| 4 | — | ReZero gate `γ` on the branch output, init 0.1 | the branch starts as a perturbation; without it the residual grows 600× |
|
| 196 |
+
| 5 | `s` shared by QKV | its own `s` per role, optionally learnable | per-role scales differ by an order of magnitude |
|
| 197 |
+
| 6 | SiLU = experiment E9 | SiLU after the FFN is the default | the more useful baseline |
|
| 198 |
+
| 7 | `amp: fp16` | `amp: none` (fp32) | fp16 autocast was **2.4× slower** on the reference GPU |
|
| 199 |
+
| 8 | one composition per layer | `KronBank` fuses all 144 compositions | 439 vs 505 ms/step, identical output (test `T16`) |
|
| 200 |
+
|
| 201 |
+
Items 1–4 are one mechanism: **the scale must be normalised, not set by the
|
| 202 |
+
initialisation.** After the fix `eff_rank(D) = 91`, `cos(W_i, W_{i−1}) = 0.72`
|
| 203 |
+
instead of `1.000`, and the model trains.
|
| 204 |
+
|
| 205 |
+
## 4. Parameter budget
|
| 206 |
+
|
| 207 |
+
```bash
|
| 208 |
+
python -m kronllm.cli budget # exits 1 if a configuration exceeds 10M
|
| 209 |
+
```
|
| 210 |
+
|
| 211 |
+
| model | d | layers | n1/n2 | K | L | modules | parameters |
|
| 212 |
+
|---|---:|---:|---|---:|---:|---:|---:|
|
| 213 |
+
| E-series | 128 | 12 | 8/16 | 4 | 4 | 60 | 190,108 |
|
| 214 |
+
| Q1 | 192 | 20 | 12/16 | 5 | 4 | 100 | 926,153 |
|
| 215 |
+
| Q2 / Q4 | 320 | 24 | 16/20 | 6 | 4 | 120 | 2,079,992 |
|
| 216 |
+
| Q6 | 320 | 24 | 16/20 | 6 | 4 | 120 | 2,473,208 |
|
| 217 |
+
| **Q8g** | 320 | 24 | 16/20 | 6 | 4 | 144 | **2,851,280** |
|
| 218 |
+
|
| 219 |
+
The soft ceiling is 10M parameters. Q8g uses 2.85M, i.e. 28 % of the ceiling:
|
| 220 |
+
the limit is not the parameter count but the 6 GB of VRAM, the 61M tokens per
|
| 221 |
+
run, and the throughput of a mobile GPU.
|
| 222 |
+
|
| 223 |
+
## 5. Data
|
| 224 |
+
|
| 225 |
+
All results in this README are on **TinyStories-200K**, byte-level or BPE-4096,
|
| 226 |
+
and this is deliberate: it is a *development bench* that separates architecture
|
| 227 |
+
effects from data effects, and it fits in a few minutes of GPU time.
|
| 228 |
+
|
| 229 |
+
TinyStories is **not** the target corpus. The final model is trained on a
|
| 230 |
+
bilingual English/Russian corpus (60/40 by bytes, science-heavy); the corpus is
|
| 231 |
+
published separately as
|
| 232 |
+
**[Icold21/kron-corpus](https://github.com/Icold21/kron-corpus)**, and its
|
| 233 |
+
measured composition is summarised in [§14](#14-the-bilingual-corpus).
|
| 234 |
+
|
| 235 |
+
Every conclusion below that mentions "quality" should be read as "quality on
|
| 236 |
+
the development bench".
|
| 237 |
+
|
| 238 |
+
## 6. Running
|
| 239 |
+
|
| 240 |
+
```bash
|
| 241 |
+
# environment
|
| 242 |
+
pip install -r requirements.txt
|
| 243 |
+
|
| 244 |
+
# data: downloads TinyStories, writes a byte memmap
|
| 245 |
+
python scripts/prepare_data.py
|
| 246 |
+
|
| 247 |
+
# BPE-4096 tokenizer + re-tokenised corpus (~4 bytes/token)
|
| 248 |
+
python scripts/prepare_bpe.py --vocab 4096
|
| 249 |
+
|
| 250 |
+
# training (Q8g = the shipped configuration)
|
| 251 |
+
python -m kronllm.cli train --exp Q8g --steps 13333 --run-dir runs/Q8g_long
|
| 252 |
+
|
| 253 |
+
# evaluation, generation, analysis
|
| 254 |
+
python -m kronllm.cli eval --run runs/Q8g_long
|
| 255 |
+
python -m kronllm.cli generate --run runs/Q8g_long \
|
| 256 |
+
--prompt "Once upon a time" --temperature 1.0 --top-k 8 --samples 2
|
| 257 |
+
python -m kronllm.cli analyze --run runs/Q8g_long
|
| 258 |
+
python -m kronllm.cli quantize --run runs/Q8g_long --bits 8
|
| 259 |
+
```
|
| 260 |
+
|
| 261 |
+
### Hugging Face interface
|
| 262 |
+
|
| 263 |
+
```python
|
| 264 |
+
from kronllm.hf import KronLM
|
| 265 |
+
|
| 266 |
+
model = KronLM.from_run("runs/Q8g_long")
|
| 267 |
+
print(model.generate("Once upon a time", max_new_tokens=100, temperature=1.0, top_k=8))
|
| 268 |
+
```
|
| 269 |
+
|
| 270 |
+
### Dashboard
|
| 271 |
+
|
| 272 |
+
`python scripts/dashboard.py` starts a local Streamlit page with the training
|
| 273 |
+
curves, the effective-rank spectra and side-by-side samples.
|
| 274 |
+
|
| 275 |
+
## 7. Metrics
|
| 276 |
+
|
| 277 |
+
| metric | why it is used |
|
| 278 |
+
|---|---|
|
| 279 |
+
| **bits/byte** | the only cross-tokenizer comparison; bpb = loss / ln 2 |
|
| 280 |
+
| PPL | convenient for humans, but tokenizer-dependent — never compare across vocabularies |
|
| 281 |
+
| `eff_rank(P)` | Roy–Vetterli effective rank of the composed matrix; exact rank saturates at `d`, so the effective rank is what distinguishes compositions |
|
| 282 |
+
| `cos_prev_max` | maximum cosine between consecutive layer matrices; → 1.0 means depth collapse (Q5) |
|
| 283 |
+
| `Var σ(θ)` | spread of the per-layer scale; a collapse indicator |
|
| 284 |
+
|
| 285 |
+
## 8. Results
|
| 286 |
+
|
| 287 |
+
### 8.1 The shipped model
|
| 288 |
+
|
| 289 |
+
**Q8g-long**: 24 layers, `d=320`, 8 heads, seq 192, BPE-4096, gated Kronecker
|
| 290 |
+
FFN. 61.4M tokens (13,333 steps × 24 × 192), 0.895 s/step, **4.09 GB** peak.
|
| 291 |
+
|
| 292 |
+
```
|
| 293 |
+
val loss 2.0693 PPL 7.919 bpb 0.7542 eff_rank(P) 144.8
|
| 294 |
+
```
|
| 295 |
+
|
| 296 |
+
The curve was still falling at the last step (0.7553 → 0.7542), so 61.4M tokens
|
| 297 |
+
is a budget choice, not a measured ceiling.
|
| 298 |
+
|
| 299 |
+
Samples (`T=1.0`, `top_k=8`):
|
| 300 |
+
|
| 301 |
+
> Once upon a time there was a little girl named Lily. She was three years old
|
| 302 |
+
> and loved to explore. One day she went to the beach and saw a big, wide waves.
|
| 303 |
+
> She was excited and wanted to go on a boat to explore the water.
|
| 304 |
+
|
| 305 |
+
> Once upon a time there was a little girl named Lily. She had a big box full of
|
| 306 |
+
> toys and she loved to play with her toys. One day, she found a big, red book
|
| 307 |
+
> under a tree. She picked it up and looked at it. It was so bright and beautiful!
|
| 308 |
+
|
| 309 |
+
Artifacts:
|
| 310 |
+
|
| 311 |
+
| file | size | quality |
|
| 312 |
+
|---|---:|---|
|
| 313 |
+
| `artifacts/q8g_int8.pt` | 3.29 MB | +0.0003 bpb vs fp32 (3.55× smaller) |
|
| 314 |
+
| `artifacts/q2_int8.pt` | 2.34 MB | previous model, kept for comparison |
|
| 315 |
+
| `runs/Q8g_long/best.pt` | 11.1 MB | fp32 reference checkpoint |
|
| 316 |
+
|
| 317 |
+
int8 quantisation is per-channel symmetric and leaves the Kronecker factors
|
| 318 |
+
intact; the composed matrix is rebuilt in fp32 at load time, which is why the
|
| 319 |
+
delta is 3e-4 rather than the usual 1e-2.
|
| 320 |
+
|
| 321 |
+
### 8.2 Tokenizer and budget: two effects that are easy to confuse
|
| 322 |
+
|
| 323 |
+
BPE-1024 compresses TinyStories to 3.07 bytes/token; BPE-4096 reaches
|
| 324 |
+
**3.96 bytes/token** on the same corpus. At a fixed parameter count that alone
|
| 325 |
+
is worth ~8 % bpb, because each token carries more text:
|
| 326 |
+
|
| 327 |
+
| model | tokenizer | parameters | steps | tokens | bpb |
|
| 328 |
+
|---|---|---:|---:|---:|---:|
|
| 329 |
+
| Q2 | BPE-1024 | 2,079,992 | 4,000 | 24.6M | 0.962 |
|
| 330 |
+
| Q4 | BPE-1024 | 2,079,992 | 4,000 | 24.6M | 0.9248 |
|
| 331 |
+
| Q6 | BPE-4096 | 2,473,208 | 4,000 | 24.6M | 0.8489 |
|
| 332 |
+
| Q6 | BPE-4096 | 2,473,208 | 10,000 | 61.4M | 0.7644 |
|
| 333 |
+
| **Q8g** | **BPE-4096** | **2,851,280** | **13,333** | **61.4M** | **0.7542** |
|
| 334 |
+
|
| 335 |
+
Q4 vs Q2 isolates the training change (full backprop instead of truncated):
|
| 336 |
+
**−3.9 %** at equal budget. Q6 vs Q4 isolates the tokenizer: **−8.2 %**, and
|
| 337 |
+
the only extra parameters are 393K in the factorised embedding.
|
| 338 |
+
|
| 339 |
+
To stay inside 6 GB with a 4,096-token vocabulary, the loss is computed in
|
| 340 |
+
checkpointed row chunks (`model.loss_chunk`), so the `24 × 192 × 4096` logits
|
| 341 |
+
tensor is never materialised. Peak VRAM is 4.09 GB.
|
| 342 |
+
|
| 343 |
+
### 8.3 Gated FFN: the one architectural win
|
| 344 |
+
|
| 345 |
+
`ffn_gated=True` adds a second Kronecker module and multiplies it by the first
|
| 346 |
+
through SiLU: `f = (W_a x) · SiLU(W_b x)`. Two controls isolate the effect:
|
| 347 |
+
|
| 348 |
+
| run | layout | parameters | tokens | bpb |
|
| 349 |
+
|---|---|---:|---:|---:|
|
| 350 |
+
| Q6 | 24 layers × 5 modules | 2,473,208 | 24.6M | 0.8489 |
|
| 351 |
+
| **Q8w** | **20 layers × 6 modules** | **2,470,328** | **24.6M** | **0.8359** (−1.5 %) |
|
| 352 |
+
| Q8g | 24 layers × 6 modules | 2,851,280 | 24.6M | 0.8318 (−2.0 %) |
|
| 353 |
+
|
| 354 |
+
Q8w is the honest control: same parameters (0.1 % apart), same number of matrix
|
| 355 |
+
modules per token (120), same tokens. The only difference from Q6 is how they
|
| 356 |
+
are arranged — 20 layers with a gate versus 24 layers without. The gate wins
|
| 357 |
+
**1.5 %**, and that is architecture, not capacity. The remaining 0.5 % of Q8g
|
| 358 |
+
is the extra 378K parameters.
|
| 359 |
+
|
| 360 |
+
So the design rule is: **a gated FFN beats four extra layers at equal budget.**
|
| 361 |
+
Everything else in the model is unchanged from Q2.
|
| 362 |
+
|
| 363 |
+
### 8.4 What we tried and rejected
|
| 364 |
+
|
| 365 |
+
Each entry lists the hypothesis, the measurement, and why the idea was dropped.
|
| 366 |
+
The weights are not kept — only the numbers, because the numbers are the result.
|
| 367 |
+
|
| 368 |
+
#### 8.4.1 Composition: sum, grouped, alternating
|
| 369 |
+
|
| 370 |
+
*Hypothesis:* a fully additive composition (`P = sI + Σ α_l D_l`) has a much
|
| 371 |
+
higher effective rank than the product, so it should be more expressive.
|
| 372 |
+
|
| 373 |
+
| composition | mult. depth | bpb (TinyStories) | eff_rank(P) |
|
| 374 |
+
|---|---:|---:|---:|
|
| 375 |
+
| product (E8) | 3 | **1.479** | 60.1 |
|
| 376 |
+
| sum (E14) | 0 | 1.5508 | 97.7 |
|
| 377 |
+
| dynamic gate (E15) | 0 | 1.5091 | 96.4 |
|
| 378 |
+
| Kron gate (E16) | 0 | 1.5504 | 97.4 |
|
| 379 |
+
|
| 380 |
+
The sum reaches rank 97.7 and still loses 4.9 % to the product. Rank was never
|
| 381 |
+
the bottleneck. Grouped and alternating compositions on WikiText-103 initially
|
| 382 |
+
looked 1.3 % better than the product (2.1561 / 2.1578 vs 2.1848), but **a second
|
| 383 |
+
seed reversed the sign**: product 2.1559 vs grouped 2.1624. The seed-to-seed
|
| 384 |
+
spread (~0.03 bpb) is larger than the effect, so the "win" was noise.
|
| 385 |
+
|
| 386 |
+
*Verdict:* product stays. The rank ordering is stable across seeds
|
| 387 |
+
(product 60.5 < grouped 79.2), so it remains a useful diagnostic — just not a
|
| 388 |
+
predictor of quality.
|
| 389 |
+
|
| 390 |
+
#### 8.4.2 Nonlinearity inside the composition
|
| 391 |
+
|
| 392 |
+
*Hypothesis:* the layer is linear by construction; putting an activation
|
| 393 |
+
between composition steps turns it into a small deep network inside a single
|
| 394 |
+
matrix, which should be a large gain per parameter.
|
| 395 |
+
|
| 396 |
+
| variant | nonlinearities per composition | bpb | cost |
|
| 397 |
+
|---|---:|---:|---:|
|
| 398 |
+
| product (E8) | 0 | **1.4789** | 3.67 MFLOP/token |
|
| 399 |
+
| E26, k=4 | 1 | 1.5675 | 3.67 |
|
| 400 |
+
| E25, k=2 | 2 | 1.5189 | 5.63 |
|
| 401 |
+
| E26b, additive k=4 | 1 | 1.5906 | 3.67 |
|
| 402 |
+
|
| 403 |
+
The nonlinearity cannot be fused away: `L` factors no longer collapse into one
|
| 404 |
+
matrix, so the layer costs `ceil(L/k)` GEMMs instead of one. Even at `k = L`,
|
| 405 |
+
where the FLOP count is unchanged, wall-clock time rose from 0.405 to 0.655
|
| 406 |
+
s/step (+62 %) — the loss of fusion, not the activation.
|
| 407 |
+
|
| 408 |
+
The interesting part is the ordering: *more* nonlinearity helped (k=2 beat k=4)
|
| 409 |
+
but every variant still lost to the purely linear product, and the additive
|
| 410 |
+
formulation — which keeps the linear skeleton — was the worst of all. The extra
|
| 411 |
+
branch doubles the output scale, and the model spends capacity compensating
|
| 412 |
+
instead of learning. `eff_rank(P)` rose to 62.3 while bpb got worse, the third
|
| 413 |
+
independent sign that rank is not the constraint.
|
| 414 |
+
|
| 415 |
+
*Verdict:* closed. The layer is linear, and at this scale that is a feature.
|
| 416 |
+
|
| 417 |
+
#### 8.4.3 Bilinear feature interaction (GLU)
|
| 418 |
+
|
| 419 |
+
*Hypothesis:* cross-feature products `(W_a x) ⊙ (W_b x)` give interactions that
|
| 420 |
+
a linear composition cannot express, at ~one module's parameter cost
|
| 421 |
+
(`3·d·r = 15,360` at `r=16` versus 15,753).
|
| 422 |
+
|
| 423 |
+
| run | FFN | layers | parameters | bpb |
|
| 424 |
+
|---|---|---:|---:|---:|
|
| 425 |
+
| Q6 | linear Kron | 24 | 2,473,208 | **0.8489** |
|
| 426 |
+
| Q9 | GLU only | 24 | 2,463,776 | 0.8740 |
|
| 427 |
+
| Q9k | GLU + Kron | 20 | 2,462,468 | 0.8673 |
|
| 428 |
+
|
| 429 |
+
Both lost (2.9 % and 2.2 %), and `cos_prev_max` rose to **0.939** in Q9 —
|
| 430 |
+
the layers became more similar, not more expressive. Adding the linear
|
| 431 |
+
Kronecker term back helped a little, but not enough.
|
| 432 |
+
|
| 433 |
+
A note on methodology: the first Q9 run was invalid because the generic
|
| 434 |
+
`std=0.02` initialisation left the bilinear branch 300× smaller than a Kron
|
| 435 |
+
module (output std 0.005 vs 1.76), i.e. switched off. The branch is now
|
| 436 |
+
initialised to unit activation and a regression test guards it. The numbers
|
| 437 |
+
above are from the corrected run — a false negative would have been worse than
|
| 438 |
+
no experiment.
|
| 439 |
+
|
| 440 |
+
*Verdict:* closed at this scale.
|
| 441 |
+
|
| 442 |
+
#### 8.4.4 Rank: `K = 24`
|
| 443 |
+
|
| 444 |
+
*Hypothesis:* more factors per module (`K=24` instead of 6) buy expressiveness
|
| 445 |
+
for a few extra parameters.
|
| 446 |
+
|
| 447 |
+
Q5 (`K=24`, 7.75M parameters) scored **0.9530** against Q4's **0.9248** at the
|
| 448 |
+
same budget, and `cos_prev_max` reached **0.99999**: every layer learned the
|
| 449 |
+
same matrix. The extra factors made the innovation dominate the anchor until
|
| 450 |
+
depth differences vanished.
|
| 451 |
+
|
| 452 |
+
*Verdict:* closed. `K=6` is the useful range at this width.
|
| 453 |
+
|
| 454 |
+
#### 8.4.5 Kronecker embedding
|
| 455 |
+
|
| 456 |
+
*Hypothesis:* the embedding is 524K of the 2.47M parameters; a Kronecker
|
| 457 |
+
embedding would shrink it.
|
| 458 |
+
|
| 459 |
+
Q7 scored **0.9533** against Q6's **0.8489** — 12 % worse. The factorised
|
| 460 |
+
`V × rank → rank × d` form is already cheap (393K for the wider vocabulary),
|
| 461 |
+
and the Kronecker structure costs too much in the input representation.
|
| 462 |
+
|
| 463 |
+
*Verdict:* closed; the factorised embedding stays.
|
| 464 |
+
|
| 465 |
+
#### 8.4.6 Recursive weight sharing
|
| 466 |
+
|
| 467 |
+
*Hypothesis:* reuse `n_unique` blocks for `r` passes and get depth for free.
|
| 468 |
+
|
| 469 |
+
| run | unique | passes | effective depth | parameters | bpb |
|
| 470 |
+
|---|---:|---:|---:|---:|---:|
|
| 471 |
+
| E8 | 12 | 1 | 12 | 190,108 | **1.479** |
|
| 472 |
+
| E19 | 6 | 2 | 12 | 111,502 | 1.717 |
|
| 473 |
+
| E21 | 6 | 1–2 random | 6–12 | 111,758 | 1.734 @ r=1 · 1.718 @ r=2 |
|
| 474 |
+
| E20 | 12 | 2 | 24 | 190,108 | **1.427** |
|
| 475 |
+
| E22 | 6 | 4 | 24 | 111,502 | 1.521 |
|
| 476 |
+
|
| 477 |
+
Three findings, all negative for the "free depth" hope:
|
| 478 |
+
|
| 479 |
+
1. **Sharing costs quality.** E19 at 59 % of the parameters and the same
|
| 480 |
+
effective depth is 16 % worse than E8.
|
| 481 |
+
2. **Depth helps, but at 2× compute.** E20 gains 3.5 % over E8 with the same
|
| 482 |
+
parameters and twice the FLOPs per token.
|
| 483 |
+
3. **Randomised training makes recursion usable.** E21 (r ~ U{1,2}, plus a
|
| 484 |
+
per-pass embedding) matches E19 at r=2 and stays coherent at r=1 (1.734),
|
| 485 |
+
where E19 collapses to 2.554. One model, two usable depths.
|
| 486 |
+
|
| 487 |
+
*Verdict:* not a default. Recursion is a deployment/compute trade-off
|
| 488 |
+
(adaptive compute), not a quality upgrade.
|
| 489 |
+
|
| 490 |
+
#### 8.4.7 Input-dependent channel gates
|
| 491 |
+
|
| 492 |
+
E15 (dense dynamic gate) reached **1.5091** and E16 (Kronecker gate) **1.5504**
|
| 493 |
+
against E8's **1.4789**. Both cost 2.4–2.6× the step time, because the gate
|
| 494 |
+
depends on the input and the composition can no longer be precomputed.
|
| 495 |
+
|
| 496 |
+
*Verdict:* closed.
|
| 497 |
+
|
| 498 |
+
#### 8.4.8 What did not survive contact with the data, in one line each
|
| 499 |
+
|
| 500 |
+
- **Sum composition**: higher rank, worse quality (1.5508 vs 1.479).
|
| 501 |
+
- **Hybrid compositions**: a 1.3 % win on one seed, reversed on the second.
|
| 502 |
+
- **Dynamic gates**: 3× slower, 2 % worse.
|
| 503 |
+
- **Nonlinear composition**: 2.7–7.5 % worse in every form.
|
| 504 |
+
- **Bilinear GLU FFN**: 2.2–2.9 % worse, with layer collapse.
|
| 505 |
+
- **`K=24`**: 3 % worse and complete depth collapse.
|
| 506 |
+
- **Kronecker embedding**: 12 % worse.
|
| 507 |
+
- **Weight sharing**: 16 % worse at equal effective depth.
|
| 508 |
+
- **KV cache**: exact (2.8e-7) but no speed-up on this GPU.
|
| 509 |
+
- **Factorised `P` apply**: exact, but 0.20–0.31× the speed of one dense GEMM.
|
| 510 |
+
|
| 511 |
+
The pattern is consistent: **what wins is the stuff that keeps the layer
|
| 512 |
+
linear and fusable**; what loses is anything that breaks the single-GEMM
|
| 513 |
+
composition, regardless of how much rank or nonlinearity it promises.
|
| 514 |
+
|
| 515 |
+
## 9. Hypotheses: verdicts
|
| 516 |
+
|
| 517 |
+
| # | hypothesis | verdict |
|
| 518 |
+
|---|---|---|
|
| 519 |
+
| H1 | normalising the branch scale is what makes the Kron layer trainable | **confirmed** (PPL 20.06 → 12.3 after the fix) |
|
| 520 |
+
| H2 | the depth recurrence adds useful state beyond independent factors | **untested** — no run removes the carry (E27/E28 pending); E8 vs E14 (1.479 vs 1.551, −4.9 %) tests the composition, not the recurrence |
|
| 521 |
+
| H3 | a wider tokenizer is worth more than extra parameters | **confirmed** (BPE-4096: −8.2 % at equal params) |
|
| 522 |
+
| H4 | effective rank is monotone in additive depth | **confirmed empirically** (60/60 modules; survives seeds) — but rank does not predict bpb |
|
| 523 |
+
| H5 | hybrid compositions beat the product | **rejected** (seed noise) |
|
| 524 |
+
| H6 | randomised recursion gives one model with usable adaptive compute | **confirmed** (r=1 1.734 vs 2.554 untrained) |
|
| 525 |
+
| H7 | extra depth at equal parameters is a clear win | **partially**: +3.5 %, but 2× compute |
|
| 526 |
+
|
| 527 |
+
H4 is deliberately called a hypothesis, not a theorem: effective rank is not
|
| 528 |
+
monotone under matrix products, and counterexamples exist for exact rank. The
|
| 529 |
+
observed ordering (product < alternating < grouped < sum) is empirical.
|
| 530 |
+
|
| 531 |
+
## 10. Tests
|
| 532 |
+
|
| 533 |
+
```bash
|
| 534 |
+
CUDA_VISIBLE_DEVICES=-1 python -m pytest tests
|
| 535 |
+
# 102 passed, 3 skipped
|
| 536 |
+
```
|
| 537 |
+
|
| 538 |
+
The suite runs on CPU in ~20 s and covers: the Kronecker forward against a
|
| 539 |
+
naive reference, gradient flow through the recurrence, `KronBank` prelude
|
| 540 |
+
equivalence, the KV cache (exact to 2.8e-7), quantisation round-trips, the HF
|
| 541 |
+
wrapper, and regression guards for the failures described above (dead
|
| 542 |
+
composition scale, dead GLU branch, depth collapse).
|
| 543 |
+
|
| 544 |
+
## 11. Efficiency
|
| 545 |
+
|
| 546 |
+
Measured on a GTX 1660 Ti (6 GB, sm75; no FlashAttention, no Triton).
|
| 547 |
+
|
| 548 |
+
| model | batch × seq | s/step | µs/token | peak VRAM |
|
| 549 |
+
|---|---|---|---:|---:|
|
| 550 |
+
| Q6 | 32 × 192 | 0.847 | 138 | 4.28 GB |
|
| 551 |
+
| Q8g | 24 × 192 | 0.895 | 194 | 4.09 GB |
|
| 552 |
+
|
| 553 |
+
The gated FFN adds a sixth module per layer (+20 % matmuls) and the smaller
|
| 554 |
+
batch loses some GPU efficiency, so cost per token rises ~40 % while peak
|
| 555 |
+
memory falls, because 24 layers with batch 24 fit better than 32.
|
| 556 |
+
|
| 557 |
+
Two serving optimisations are implemented and exact but not useful here:
|
| 558 |
+
|
| 559 |
+
- **KV cache** — matches the no-cache path to 2.8e-7, but the 1660 Ti is
|
| 560 |
+
compute-bound at these sizes, so decoding does not speed up.
|
| 561 |
+
- **Factorised apply** (`y = x·Pᵀ` computed factor by factor) — mathematically
|
| 562 |
+
identical, but only 0.20–0.31× the speed of one dense GEMM. Composing once
|
| 563 |
+
and multiplying once is the fast path.
|
| 564 |
+
|
| 565 |
+
`torch.compile` is unavailable in this environment (no Triton on Windows), so
|
| 566 |
+
all speed-ups are structural, not compiler-driven.
|
| 567 |
+
|
| 568 |
+
## 12. Repository layout
|
| 569 |
+
|
| 570 |
+
```
|
| 571 |
+
kronllm/ library: kron.py (composition), model.py, train.py,
|
| 572 |
+
config.py (experiments), data.py, tokenizer.py,
|
| 573 |
+
bilingual.py (corpus pipeline), quantization.py,
|
| 574 |
+
hf.py, cli.py
|
| 575 |
+
scripts/ data prep (TinyStories, BPE, WikiText, SFT,
|
| 576 |
+
build_bilingual.py, prepare_bilingual_bin.py),
|
| 577 |
+
dashboard
|
| 578 |
+
tests/ 102 tests, CPU-only
|
| 579 |
+
docs/ architecture.md, results.md (full tables),
|
| 580 |
+
data_plan.md (bilingual corpus), roadmap.md
|
| 581 |
+
artifacts/ q8g_int8.pt (shipped), q2_int8.pt (previous)
|
| 582 |
+
runs/ per-experiment logs and metrics; only the two best
|
| 583 |
+
checkpoints are kept (Q8g_long, Q6_10k)
|
| 584 |
+
notebooks/ walkthrough of the architecture and the measurements
|
| 585 |
+
current_progress.md rolling log of the experiment queue
|
| 586 |
+
```
|
| 587 |
+
|
| 588 |
+
## 13. Limitations
|
| 589 |
+
|
| 590 |
+
- **Development bench only.** Every quality number is TinyStories, English.
|
| 591 |
+
Nothing here proves the same ordering on the bilingual corpus; the corpus
|
| 592 |
+
pilot is built but no model has been trained on it yet.
|
| 593 |
+
- **One GPU, one seed.** Most experiments are single-seed; the one seed control
|
| 594 |
+
performed (hybrid compositions) reversed a published-looking result, which is
|
| 595 |
+
a warning about the rest.
|
| 596 |
+
- **No FlashAttention, no Triton.** Throughput is well below what the same
|
| 597 |
+
model would reach with a fused kernel.
|
| 598 |
+
- **512-token context at most.** RoPE is implemented, but training used 192–256
|
| 599 |
+
tokens; long-context behaviour is untested.
|
| 600 |
+
- **No instruction tuning in the shipped model.** S1 was an SFT probe on
|
| 601 |
+
Dolly-15k, not a product.
|
| 602 |
+
- **int8 only.** int4 is implemented but the delta at this size is not yet
|
| 603 |
+
worth the extra complexity.
|
| 604 |
+
|
| 605 |
+
## 14. The bilingual corpus
|
| 606 |
+
|
| 607 |
+
The corpus now lives in its own repository:
|
| 608 |
+
**[Icold21/kron-corpus](https://github.com/Icold21/kron-corpus)** — a
|
| 609 |
+
2B-token English/Russian corpus with a science accent, published with the full
|
| 610 |
+
pipeline, the per-shard manifest and the measured composition.
|
| 611 |
+
|
| 612 |
+
| stage | result |
|
| 613 |
+
|---|---|
|
| 614 |
+
| fetched text | 7.3 GB · ~795k documents |
|
| 615 |
+
| target size | **~2.0B tokens** (8K BPE, 3.34 bytes/token measured) |
|
| 616 |
+
| by language | EN 4.9 GB · RU 2.4 GB at fetch; the mix enforces 60/40 by bytes |
|
| 617 |
+
| domains | math 25 %, science 14 %, ML 12 %, logic 10 %, code 15 %, literature 11 %, encyclopedic 9 %, CS/engineering 3 %, humour 1 %, identity 1 % |
|
| 618 |
+
| dedup | MinHash (Jaccard 0.8) · held-out 2 % per language, disjoint |
|
| 619 |
+
| tokenizer | 8K byte-level BPE, bilingual, fertility recorded per language |
|
| 620 |
+
| chat template | `<\|system\|>` 8187 · `<\|user\|>` 8188 · `<\|assistant\|>` 8189 · `<\|eot\|>` 8190 · `<\|endoftext\|>` 8191 |
|
| 621 |
+
| identity slice | 800 plain + 850 pastiche per language, no numeric facts |
|
| 622 |
+
|
| 623 |
+
**One finding fixed the ratio policy.** Bytes per token is not comparable across
|
| 624 |
+
scripts: both languages receive ≈2.8 characters per token, but Cyrillic costs
|
| 625 |
+
two UTF-8 bytes per character. A 60/40 *byte* split therefore gives only
|
| 626 |
+
**33 % of tokens** to Russian, not the 45–47 % the original plan predicted. The
|
| 627 |
+
contract is now stated in bytes (stable, tokenizer-independent) and the token
|
| 628 |
+
share is recorded next to it in `tokenizer_meta.json`.
|
| 629 |
+
|
| 630 |
+
Known gaps are reported rather than hidden, and the corpus README repeats them:
|
| 631 |
+
logic lands at ~3 % against a 10 % quota (only formal proofs scale), ML at
|
| 632 |
+
~5 % against 12 % (most prose is paywalled), and Russian code is absent by
|
| 633 |
+
nature. The mixer emits these as shortfalls; nothing is padded with
|
| 634 |
+
mislabelled text.
|
| 635 |
+
|
| 636 |
+
Next steps:
|
| 637 |
+
|
| 638 |
+
1. train **Q10** (9,938,886 params, d=512, 25 layers, K=10, seq 512) on the
|
| 639 |
+
corpus on two Kaggle T4s with `torchrun --nproc_per_node=2`, one epoch =
|
| 640 |
+
82k steps = 2.015B tokens, checkpoint every 500 steps;
|
| 641 |
+
2. close the tokenizer question (8K vs 16K) before the run starts;
|
| 642 |
+
3. package the result, then compare against public small-model baselines
|
| 643 |
+
before the SFT stage.
|
| 644 |
+
|
| 645 |
+
**Reproducibility details that matter for the run.** The epoch sampler is
|
| 646 |
+
sharded: all ranks build the same permutation from a shared seed and take
|
| 647 |
+
`rank::world`, so one epoch is one full pass instead of the ~63 % a random
|
| 648 |
+
sampler covers. DDP writes logs and checkpoints only on rank 0; `last.pt`
|
| 649 |
+
carries the optimizer and the step counter, so a killed Kaggle session resumes
|
| 650 |
+
exactly where it stopped.
|
| 651 |
+
|
| 652 |
+
The long-term plan, including the SFT families and the comparison phase, is in
|
| 653 |
+
[docs/roadmap.md](docs/roadmap.md).
|
| 654 |
+
|
| 655 |
+
## 15. License and citation
|
| 656 |
+
|
| 657 |
+
MIT — see [LICENSE](LICENSE).
|
| 658 |
+
|
| 659 |
+
```bibtex
|
| 660 |
+
@misc{kronllm2026,
|
| 661 |
+
title = {KronLLM: a compact fully Kronecker-parameterised language model},
|
| 662 |
+
author = {Ivan Kholodilo},
|
| 663 |
+
year = {2026},
|
| 664 |
+
url = {https://github.com/Icold21/kron-llm}
|
| 665 |
+
}
|
| 666 |
+
```
|