---
language:
  - en
  - ru
license: mit
library_name: pytorch
pipeline_tag: text-generation
tags:
  - language-model
  - kronecker
  - small-language-model
  - bilingual
  - parameter-efficiency
---

# KronLLM

A compact bilingual (English/Russian) language model whose matrix layers are
built entirely from **Kronecker sums of rank-1 factors**. Every attention and
FFN projection is a composition of `L` factors instead of a dense `d × d`
weight: a layer that would need 102,400 numbers needs 15,753.

This repository hosts the model card, the code and the shipped int8
checkpoints. The training corpus is separate: see
[Icold21/kron-corpus](https://huggingface.co/datasets/Icold21/kron-corpus) and
the research log at [github.com/Icold21/kron-llm](https://github.com/Icold21/kron-llm).

## Main model: Q10 (10M)

**Q10 is the model this project is about**: 9,938,886 parameters (d=512, 25
layers, 8K bilingual BPE, Kronecker K=10), trained on 2.78B tokens of bilingual
English/Russian text. It is mid-pretraining — **step 36,549 of 169,884 (21.5 %)**
of the first epoch — and reaches **1.1828 bits/byte** mixed (EN 1.4327 /
RU 0.9411) on a held-out bin that never changes; GPT-2-124M scores 1.554 on the
same blocks. The checkpoint lives on Kaggle
([`icold21/kron-q10-ckpt`](https://www.kaggle.com/datasets/icold21/kron-q10-ckpt))
and will be mirrored here when the epoch finishes.

## Experimental checkpoints (the development bench)

The two small checkpoints below are **not** the product. They are the TinyStories
bench that made the 10M configuration cheap to choose: each isolates one
decision (tokenizer, gated FFN, depth carry), and their bits/byte is not
comparable with Q10's bilingual number.

| file | parameters | size | quality |
|---|---:|---:|---|
| `q8g_int8.pt` | 2,851,280 | 3.14 MB | **0.7542 bits/byte** (TinyStories val, 61.4M tokens seen) |
| `q2_int8.pt` | 2,079,992 | 2.34 MB | previous bench model, same corpus |

Int8 quantisation costs +0.0003 bpb on Q8g. The fp32 checkpoint is 11.4 MB and
is not shipped.

## Usage (bench checkpoint)

```python
import torch
from kronllm.model import KronTransformer
from kronllm.quantization import load_int8

ckpt = torch.load("q8g_int8.pt", map_location="cpu")
model = KronTransformer(ckpt["cfg"]).eval()
load_int8(model, ckpt["model"])
```

The bench model is a research artifact: it writes short English text and is not
an instruct-tuned assistant.

## Architecture of the bench model

| | |
|---|---|
| layers / width | 24 / 320, 8 heads, seq 192 |
| composition | product over L=4 factors, K=6 terms, depth carry `K_win=2` |
| FFN | gated (SwiGLU-style) |
| tokenizer | BPE-4096, tied factorised embedding (rank 128) |

The main model keeps the same block but scales to d=512, 25 layers, K=10 and the
8K bilingual BPE, with pre-norm QK and a gated FFN.

Measured, not assumed: the depth carry is worth 1.1 % bpb, freezing its scalar
0.8 %, and widening the window to four layers *costs* 1.4 %. Details in the
[research log](https://github.com/Icold21/kron-llm/blob/main/docs/results.md).

## Training budget

At the measured 2.552 s/step on a single Kaggle T4 (batch 32, seq 512), one epoch of the 2.78B-token corpus is **120 GPU-hours**; the decay pass (30 % of the epoch, quality-selected slice plus 5 % chat) adds **35**, and SFT 4,000 steps adds **2.8**. Roughly **132 GPU-hours remain** from step 36,549 -- about 12 sessions of quota. Derivation in `docs/compute_budget.md` of the code repository.

## Next

Q10 continues on [KronCorpus](https://huggingface.co/datasets/Icold/kron-corpus)
— 2.78B tokens of bilingual English/Russian text with a science accent — on one
Kaggle T4. After the epoch: a decay pass (29 % of the epoch, quality-selected,
plus 5 % chat) and SFT. The weights will be mirrored here when the epoch
finishes; the corpus card documents every source and licence.

## Licence

Code and weights: MIT. The corpus keeps its sources' licences (see the dataset
card).