| license: mit | |
| tags: [ternary, bitnet, bitbop, language-model] | |
| # BitBop-91M | |
| Latent-free **ternary** language model trained with **BitBop** (weights in {-1,0,+1}, no latent | |
| full-precision copy; only a bf16 flip momentum as optimiser state). Trained on a single RTX 3060 (6 GB). | |
| - Data: TinyStories (~90M tokens) | |
| - BLiMP macro-accuracy (own harness): **59.07** | |
| - Arch: d768, 12 layers, 1 full-attention layer(s) + sliding-window, NoPE, tied interface. | |
| - Recipe: per-row RMS flip, tau=(2.0,2.75); see the code/report at github.com/ValerioDolci/bitbop. | |
| Proof of concept; budget-scoped (not SOTA, no speed claim). Load with the `bitbop` package (config.json + model.pt + tokenizer.json). | |