RetNet-0.5B-16k β€” long-context research export (v21.2)

Status: Tier-0 partial pass β€” long-document modelling improved a lot, but needle retrieval and the effective-context test failed. This is a research export, not a claim of usable 16k context.

Base: v21.1 C-end + 20-head Ξ³ calibration (C1). 13,000 steps at 16,384 tokens (7,000 long-document + 4,500 mixed + 1,500 chat16k), 0.852 B tokens, 2Γ—RTX PRO 6000, lr 2e-5 β†’ 2e-6.

Parameters 504,179,896
Layers / width / heads 20 / 1408 / 22
Vocab / training context 20,000 ByteLevel BPE / 16,384
Decay per-head learned Ξ³, calibrated C1; GLA branch present with gla_rank=0
Storage fp32 (model.safetensors, 1.88 GiB)
SHA256 (model.safetensors) cace5697ddaf9a43ec8ae51a3761bf4fab57ca48ee89bb0aebf2a1fdfd62bd35

Measured results (val_pack, 5 files / 4,454,400 tokens, skypile excluded)

Gate Base (C-end) This export Delta Verdict
2k criterion 2.4885 2.5524 +0.0639 T2 pass (≀ +0.10), T1 fail (≀ +0.05)
16k within-doc long-tail 3.2841 2.5942 βˆ’0.690 pass (T1/T2)
by-position 4k / 8k 3.2124 / 3.2300 2.5941 / 2.5967 βˆ’0.62 / βˆ’0.63 β€”
needle@16k (120/depth, pooled) 0.1000 (chance) 0.0978 βˆ’0.002 fail (needs Ξ” β‰₯ +0.10 and abs β‰₯ 0.18)
two-arm Ξ”_marginal_long@16k +0.0418 +0.0407 βˆ’0.001 absolute pass, no gain

The two-arm metric never improved over the seed, so the export demonstrates long-document language modelling at 16k, not retrieval or effective use of the context.

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer
repo = "Ayaka157/RetNet-0.5B-16k"
tok = AutoTokenizer.from_pretrained(repo, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(repo, trust_remote_code=True, torch_dtype="float32").eval()

The chunkwise path is the supported one. For long sequences set RETNET_FAST_CHUNKWISE=1; the 16k training run needed the compiled path (1.66 it/s on 2Γ—PRO 6000, 83.1 GiB peak). Without compilation it is much slower and memory-hungry.

Limitations

  • Needle retrieval is at chance. Do not use this for retrieval, RAG, or as a long-context assistant.
  • The 2k criterion regressed by +0.0639; for chat use the released v21.3-chat instead.
  • "16k capability" is not claimed: the registered Tier-2 gate did not pass.

All checkpoints

Part of the RetNet-0.5B collection: https://huggingface.co/collections/Ayaka157/retnet-05b-from-scratch-training-line-6afc5fe4007c9acacd1cb52f

Files

model.safetensors, config.json, modeling_retnet.py, retention_fast.py, fused_ce.py, bpe_tokenizer.json, tokenizer files. Full training log and figures: https://github.com/Ayaka157/RetNet

Downloads last month
49
Safetensors
Model size
0.5B params
Tensor type
F32
Β·
Inference Providers NEW
This model isn't deployed by any Inference Provider. πŸ™‹ Ask for provider support

Collection including Ayaka157/RetNet-0.5B-16k