| # D270 (dense control) | |
| Part of the [Self-Drafting Latent Memory](../README.md) collection. | |
| No memory. Uniform d_ff = 2464, so its size (269.8M non-embedding) equals the memory models' active parameters per token. **Trained on 2B tokens** (3,815 steps), twice the memory models' budget. | |
| - **Architecture:** 24-layer Llama-style decoder, d = 1024, 24 query / 4 KV heads, SwiGLU, RoPE, tied embeddings, vocabulary 32,768, context 2048. Training config: `w3_dense270` in the repository. | |
| - **Validation loss:** 2.5147 (2B tokens); 2.7167 at 0.94B. | |
| - **Zero-shot (lm-eval 0.4.13):** | |
| | HellaSwag | MMLU-cloze | SciQ | WinoGrande | PIQA | ARC-Easy | | |
| |---|---|---|---|---|---| | |
| | 31.0 | 25.0 | 68.1 | 52.6 | 61.5 | 41.2 | | |
| ## Files | |
| - `model.pt`: `{"model": state_dict, "config": ..., "step": ..., "params": ...}`, weights only (no optimizer state). | |
| - `config.json`: the resolved training/model config. | |
| - `params.json`: parameter counts by component. | |
| ## Load | |
| ```python | |
| from ffnmem.infer import load_model # code from the accompanying repository | |
| model, cfg = load_model("d270/model.pt", device="cuda") | |
| ``` | |
| Research checkpoint: not instruction-tuned, not safety-tuned, not for deployment. | |