d270/README.md

D270 (dense control)

Part of the Self-Drafting Latent Memory collection.

No memory. Uniform d_ff = 2464, so its size (269.8M non-embedding) equals the memory models' active parameters per token. Trained on 2B tokens (3,815 steps), twice the memory models' budget.

• Architecture: 24-layer Llama-style decoder, d = 1024, 24 query / 4 KV heads, SwiGLU, RoPE, tied embeddings, vocabulary 32,768, context 2048. Training config: w3_dense270 in the repository.

• Validation loss: 2.5147 (2B tokens); 2.7167 at 0.94B.

• Zero-shot (lm-eval 0.4.13):

  HellaSwag	MMLU-cloze	SciQ	WinoGrande	PIQA	ARC-Easy
  31.0	25.0	68.1	52.6	61.5	41.2

Files

• model.pt: {"model": state_dict, "config": ..., "step": ..., "params": ...}, weights only (no optimizer state).
• config.json: the resolved training/model config.
• params.json: parameter counts by component.

Load

from ffnmem.infer import load_model   # code from the accompanying repository
model, cfg = load_model("d270/model.pt", device="cuda")

Research checkpoint: not instruction-tuned, not safety-tuned, not for deployment.