varC_long_s0 and combo_s0: weights, configs, tokenizers, ESP32-S3 packs, model card

.gitattributes

@@ -33,3 +33,5 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
 *.zip filter=lfs diff=lfs merge=lfs -text
 *.zst filter=lfs diff=lfs merge=lfs -text
 *tfevents* filter=lfs diff=lfs merge=lfs -text
+combo_s0/model.pack filter=lfs diff=lfs merge=lfs -text
+varC_long_s0/model.pack filter=lfs diff=lfs merge=lfs -text

README.md

@@ -0,0 +1,71 @@
+---
+license: mit
+language:
+- en
+datasets:
+- roneneldan/TinyStories
+tags:
+- tinyml
+- microcontroller
+- esp32
+- esp32-s3
+- edge-ai
+- quantization
+- int4
+- tinystories
+pipeline_tag: text-generation
+---
+
+# bytes-per-token: small LMs that run at ~90 tokens/s on an $8 ESP32-S3
+
+Weights for [github.com/alikup-ai/bytes-per-token](https://github.com/alikup-ai/bytes-per-token).
+The models were designed for one number: how many bytes of weights the microcontroller reads from PSRAM
+per generated token. They are trained with 4-bit quantization-aware training and run on the board in int4.
+
+| model | stored params | weights read per token (int4) | bits/byte, TinyStories valid | speed on ESP32-S3-N16R8 |
+|---|---:|---:|---:|---:|
+| `varC_long_s0` | 8.6M | 1.16 MB | **0.6253** | ~90 tok/s at the start of the context, ~81 mean over 256 tokens |
+| `combo_s0` | 3.4M | 0.90 MB | 0.7322 | ~113 tok/s at the start, ~88 mean |
+
+For reference, [slvDev/esp32-ai](https://github.com/slvDev/esp32-ai) (28.9M params) scores 0.863 bits/byte on
+the same text with the same procedure.
+
+Bits per byte: first 5,000 stories of `TinyStoriesV2-GPT4-valid.txt` (skipping the first, partial one), joined by
+`<|endoftext|>`, scored on non-overlapping 256-token windows, summed loss in bits divided by UTF-8 bytes.
+Reproduce with `python bench_pc.py` from the GitHub repo.
+
+## Architecture
+
+Llama-style decoder, 6 layers, 4 heads, context 256, RoPE, RMSNorm.
+
+- **ReLU FFN** instead of SwiGLU: ~79% of FFN neurons are zero per token, and the engine skips their rows.
+- **Per-layer embeddings (PLE)**, the idea from Gemma 3n: a per-layer token table kept in flash, read only for the current token.
+- **4-bit weights** (group-wise QAT); embeddings and output layer tied.
+
+| model | d | ffn | vocab | PLE dim |
+|---|---:|---:|---:|---:|
+| `varC_long_s0` | 128 | 256 | 8192 | 128 |
+| `combo_s0` | 128 | 320 | 4096 | 64 |
+
+## Files (per model folder)
+
+- `model.safetensors`: PyTorch weights for `train/model.py` in the GitHub repo (`LM(Config(**config["cfg"]))`).
+- `config.json`: shape, training run, measured validation bits/byte.
+- `tokenizer.json`: byte-level BPE (Hugging Face `tokenizers`), `<|endoftext|>` between stories.
+- `model.pack`: the int4 pack the ESP32-S3 engine reads, flashed at 0x90000 with the demo firmware.
+  Ready-made full images are in the [GitHub release](https://github.com/alikup-ai/bytes-per-token/releases).
+
+## Training
+
+Data: the first 300 MiB of `TinyStoriesV2-GPT4-train.txt` from
+[roneneldan/TinyStories](https://huggingface.co/datasets/roneneldan/TinyStories).
+`varC_long_s0`: 70M tokens (~0.9 epoch), `combo_s0`: 20M tokens. AdamW, lr 3e-3, batch 32 x 256, one seed,
+laptop CPU (~2.6 h for `varC_long_s0`).
+
+## Limitations
+
+- Trained only on synthetic children's stories: it continues simple English stories and does not follow instructions or answer general questions.
+- Context is 256 tokens.
+- One training seed per model; small differences between models are within seed noise (~0.003 bits/byte).
+
+License: MIT.

combo_s0/config.json

@@ -0,0 +1,26 @@
+{
+  "name": "combo_s0",
+  "cfg": {
+    "vocab": 4096,
+    "d": 128,
+    "layers": 6,
+    "heads": 4,
+    "ffn": 320,
+    "ple": 64,
+    "experts": 1,
+    "seq_len": 256,
+    "rope_theta": 10000.0,
+    "act": "relu",
+    "qbits": 4
+  },
+  "stored_params": 3375104,
+  "read_params": 1802752,
+  "bytes_per_token_4bit": 901376.0,
+  "steps": 2441,
+  "tokens": 19996672,
+  "lr": 0.003,
+  "seed": 0,
+  "batch": 32,
+  "final_val_loss": 2.0081979647332195,
+  "elapsed_s": 2419.3
+}

combo_s0/model.pack

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:97030f02d89f7ca2ddefa444a6d94e2b90627f3790008c4dfff9e4a4ab6017f6
+size 2436288

combo_s0/model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:5ea3d5098e43fa75260d17b0f821dc48f0984357df6c1ca7053f3c2027e211b7
+size 13517352

varC_long_s0/config.json

@@ -0,0 +1,27 @@
+{
+  "name": "varC_long_s0",
+  "cfg": {
+    "vocab": 8192,
+    "d": 128,
+    "layers": 6,
+    "heads": 4,
+    "ffn": 256,
+    "ple": 128,
+    "experts": 1,
+    "seq_len": 256,
+    "rope_theta": 10000.0,
+    "act": "relu",
+    "qbits": 4
+  },
+  "stored_params": 8617984,
+  "read_params": 2327424,
+  "bytes_per_token_4bit": 1163712.0,
+  "steps": 8544,
+  "tokens": 69992448,
+  "lr": 0.003,
+  "seed": 0,
+  "batch": 32,
+  "final_val_loss": 1.7688706053627863,
+  "elapsed_s": 9423.2,
+  "val_bpb": 0.6261052017431212
+}

varC_long_s0/model.pack

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:1955d394b47cbb135116ef23560482518da76fd008312915aa2f50e80972fc95
+size 5714624

varC_long_s0/model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:3dab0cfdba413b158e81479f863ce82e2b4200fdb5f21a3a5db72c945a5832cf
+size 34489144