RetNet-300M (base, step 15,082)

A 310M-parameter Retentive Network language model trained from scratch on code, math and synthetic textbooks, with three additions aimed at multi-step reasoning:

  • Delta-rule memory with negative eigenvalues (beta_max = 2.0): the state can overwrite a stored value instead of only adding to it.
  • Short causal convolution (K = 4) before the q/k/v projections.
  • Calculator in the loop: arithmetic is written as <<48/2=24>>24 and the exact result is filled in at decode time.

Inference is recurrent: constant memory per generated token, no KV cache. Architecture: d_model=1024, n_layers=22, n_heads=16, vocab 32,000 (digit-split BPE, ids 1-16 reserved for post-training).

Training: about 494M tokens on a single 16 GB T4 GPU. This is a small research model: it is not instruction-tuned and will ramble or repeat itself on open-ended prompts.

Results (zero-shot, greedy)

Task Score Note
GSM8K, calculator in the loop 7.4% ± 1.6% first 256 test problems
GSM8K, no calculator 4.7% ± 1.3% first 256 test problems
MBPP pass@1 0.4% 41% syntactically valid
ARC-easy (normalized) 36.0% 500 questions, chance 25%
ARC-challenge (normalized) 21.2% 500 questions, chance 25%
HellaSwag (normalized) 36.2% 500 questions, chance 25%

Usage

This is not a transformers model; it needs the code from https://github.com/prateekstark/retnet-300m.

from retnet.checkpoint import load_run
from retnet.calculator import generate_with_calculator

model, tok, step = load_run("hf://starkprateek1/retnet-300m", "cuda")
text, calls = generate_with_calculator(
    model, tok, "Question: A baker bakes 7 trays of 24 muffins and sells 150. How many are left?\nAnswer:",
    max_new_tokens=200, stop=("\nQuestion:",))
print(text)

Files

  • model.safetensors: weights (fp32; the LM head is tied to the embedding and not stored)
  • config.json: model config and training step
  • tokenizer.json: the tokenizer (Hugging Face tokenizers format)

License

Code and weights: MIT. The training datasets keep their own licenses; check them before commercial use.

Downloads last month
22
Safetensors
Model size
0.3B params
Tensor type
F32
·
Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support

Datasets used to train starkprateek1/retnet-300m