metadata
license: mit
tags:
- gated-deltanet
- ternary
- bitnet
- looped-language-model
- compute-in-memory
GDN-Bit
Gated DeltaNet with ternary weights (every projection, the FFN and the tied output head are BitLinear; the
embedding stays float), ReLU² FFN, no short conv, one fused ternary input projection per mixer, and
whole-stack looping. T1 = no loop, T4 = the layer stack run 4 times with shared weights.
Code: custom_models/gdnbit in SSM_IMC_NATURE (branch gdnbit).
| Folder | d / L / H | Params | Data | Tokens |
|---|---|---|---|---|
gdnbit-250k/{T1,T4} |
64 / 4 / 2 | 266K | TinyStories (tok512) | 6.6B (100k steps) |
gdnbit-1m/{T1,T4} |
128 / 6 / 3 | 1.45M | TinyStories (tok512) | 6.6B (100k steps) |
gdnbit-100m/{T1,T4} |
768 / 9 / 9 | 99M | FineWeb-Edu sample-10BT (Llama 2 32K) | 10B |
gdnbit-450m/{T1,T4} |
1024 / 21 / 8 | 440M | FineWeb-Edu sample-100BT (Llama 3) | 100B |
Each run folder (<data>-<tokens>-s<seed>) holds an HF checkpoint (config.json, model.safetensors,
tokenizer files) and eval/. Stored BitLinear weights are the latent FP32 weights; ternary values are
round(clip(w / mean|w|, -1, 1)) per matrix.