openai/gsm8k
Benchmark • Updated • 17.6k • 1.24M • 2.09k
A 310M-parameter Retentive Network language model trained from scratch on code, math and synthetic textbooks, with three additions aimed at multi-step reasoning:
beta_max = 2.0): the state can overwrite a
stored value instead of only adding to it.K = 4) before the q/k/v projections.<<48/2=24>>24 and the exact result is filled in
at decode time.Inference is recurrent: constant memory per generated token, no KV cache.
Architecture: d_model=1024, n_layers=22, n_heads=16, vocab 32,000
(digit-split BPE, ids 1-16 reserved for post-training).
Training: about 494M tokens on a single 16 GB T4 GPU. This is a small research model: it is not instruction-tuned and will ramble or repeat itself on open-ended prompts.
| Task | Score | Note |
|---|---|---|
| GSM8K, calculator in the loop | 7.4% ± 1.6% | first 256 test problems |
| GSM8K, no calculator | 4.7% ± 1.3% | first 256 test problems |
| MBPP pass@1 | 0.4% | 41% syntactically valid |
| ARC-easy (normalized) | 36.0% | 500 questions, chance 25% |
| ARC-challenge (normalized) | 21.2% | 500 questions, chance 25% |
| HellaSwag (normalized) | 36.2% | 500 questions, chance 25% |
This is not a transformers model; it needs the code from https://github.com/prateekstark/retnet-300m.
from retnet.checkpoint import load_run
from retnet.calculator import generate_with_calculator
model, tok, step = load_run("hf://starkprateek1/retnet-300m", "cuda")
text, calls = generate_with_calculator(
model, tok, "Question: A baker bakes 7 trays of 24 muffins and sells 150. How many are left?\nAnswer:",
max_new_tokens=200, stop=("\nQuestion:",))
print(text)
model.safetensors: weights (fp32; the LM head is tied to the embedding and not stored)config.json: model config and training steptokenizer.json: the tokenizer (Hugging Face tokenizers format)Code and weights: MIT. The training datasets keep their own licenses; check them before commercial use.