# D270 (dense control)

Part of the [Self-Drafting Latent Memory](../README.md) collection.

No memory. Uniform d_ff = 2464, so its size (269.8M non-embedding) equals the memory models' active parameters per token. **Trained on 2B tokens** (3,815 steps), twice the memory models' budget.

- **Architecture:** 24-layer Llama-style decoder, d = 1024, 24 query / 4 KV heads, SwiGLU, RoPE, tied embeddings, vocabulary 32,768, context 2048. Training config: `w3_dense270` in the repository.
- **Validation loss:** 2.5147 (2B tokens); 2.7167 at 0.94B.
- **Zero-shot (lm-eval 0.4.13):**

  | HellaSwag | MMLU-cloze | SciQ | WinoGrande | PIQA | ARC-Easy |
  |---|---|---|---|---|---|
  | 31.0 | 25.0 | 68.1 | 52.6 | 61.5 | 41.2 |

## Files

- `model.pt`: `{"model": state_dict, "config": ..., "step": ..., "params": ...}`, weights only (no optimizer state).
- `config.json`: the resolved training/model config.
- `params.json`: parameter counts by component.

## Load

```python
from ffnmem.infer import load_model   # code from the accompanying repository
model, cfg = load_model("d270/model.pt", device="cuda")
```

Research checkpoint: not instruction-tuned, not safety-tuned, not for deployment.