| --- |
| license: mit |
| tags: [gated-deltanet, ternary, bitnet, looped-language-model, compute-in-memory] |
| --- |
| # GDN-Bit |
|
|
| Gated DeltaNet with ternary weights (every projection, the FFN and the tied output head are BitLinear; the |
| embedding stays float), ReLU² FFN, no short conv, one fused ternary input projection per mixer, and |
| whole-stack looping. `T1` = no loop, `T4` = the layer stack run 4 times with shared weights. |
| Code: `custom_models/gdnbit` in SSM_IMC_NATURE (branch `gdnbit`). |
|
|
| | Folder | d / L / H | Params | Data | Tokens | |
| |---|---|---|---|---| |
| | `gdnbit-250k/{T1,T4}` | 64 / 4 / 2 | 266K | TinyStories (tok512) | 6.6B (100k steps) | |
| | `gdnbit-1m/{T1,T4}` | 128 / 6 / 3 | 1.45M | TinyStories (tok512) | 6.6B (100k steps) | |
| | `gdnbit-100m/{T1,T4}` | 768 / 9 / 9 | 99M | FineWeb-Edu sample-10BT (Llama 2 32K) | 10B | |
| | `gdnbit-450m/{T1,T4}` | 1024 / 21 / 8 | 440M | FineWeb-Edu sample-100BT (Llama 3) | 100B | |
|
|
| Each run folder (`<data>-<tokens>-s<seed>`) holds an HF checkpoint (`config.json`, `model.safetensors`, |
| tokenizer files) and `eval/`. Stored BitLinear weights are the latent FP32 weights; ternary values are |
| `round(clip(w / mean|w|, -1, 1))` per matrix. |
|
|
