RetNet-0.5B v21.3-chat — post-trained chat model

Status: chat gate PASS; held-out criterion 2.5914 (seed 2.5898; +0.0016).

This model is the result of the 2026-11-12 post-training round: 8,000 steps of concise/IF SFT on 117,712 prompt-aligned rows, then 2,000 steps of low-LR replay repair with 20% general pretrain replay. It is the served-demo candidate, not a general-purpose model.

Parameters 504,179,896 (100% live)
Architecture RetNet, 20 layers, width 1408, 22 heads, value/FFN 2816
Vocabulary / context 20,000 ByteLevel BPE / 2048 training context
Decay per-head learned γ; GLA branch present with gla_rank=0
Precision model.safetensors stored fp32; load bf16 for inference
SHA256 (model.safetensors) 40a919bca43dfad776203af463598c79cd0f340b40c39a592995e518250fc54e

All checkpoints

This repo is the latest release of the family. The full training line — epoch 1-3, the repair round (v21.1 / v21.1-chat), long-context v21.2, this chat release, and the 16k / 8k research exports — is collected here:

https://huggingface.co/collections/Ayaka157/retnet-05b-from-scratch-training-line-6afc5fe4007c9acacd1cb52f

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer
tok = AutoTokenizer.from_pretrained("Ayaka157/RetNet-0.5B-v21.3-chat", trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(
    "Ayaka157/RetNet-0.5B-v21.3-chat", trust_remote_code=True, torch_dtype="bfloat16")

The play page sends this default system preset: 你是一个说话简短直接的助手:每次回答控制在两句话内,先给结论,不解释推理过程,不列清单。 Use an explicit short-answer system prompt; without one the model still stops, but answers are longer.

Measured post-training gates (2026-11-12, 2×RTX PRO 6000)

Behaviour suite, 18 prompts, concise condition:

Metric Seed v21.1-chat v21.3-chat
cap-hit 33.3% 0%
unclosed think 5.6% 0%
median reply 107 tok 63 tok

No-system condition also 0% cap, 0% unclosed, median 70 tok.

Held-out criterion (5 files, 4,454,400 tokens, skypile excluded): 2.5914 vs seed 2.5898 (+0.0016) and C-end 2.4885. Chat-val is not a gate.

Instruction-following diagnostic: 4/10 (10 fixed items, concise). This is a diagnostic, not a capability claim; arithmetic/world knowledge remain weak.

Limitations

  • Knowledge, arithmetic, code accuracy, and factual reliability remain those of a 0.5B model; the post-training improves reply shape and stopping, not knowledge.
  • This checkpoint is not a long-context/retrieval model. The 16k retrieval result is reported separately and must not be inferred from this card.
  • The 18-prompt suite is directional. A pass here is not a general human-likeness or safety claim.

Files

config.json, model.safetensors, modeling_retnet.py, retention_fast.py, fused_ce.py, bpe_tokenizer.json, and tokenizer files. Code and analysis: https://github.com/Ayaka157/RetNet

Retrieval round (negative)

The 2026-11-12 γ-frozen answer-only retrieval run reached needle/item accuracy 1.0 but collapsed the 2k criterion to 4.56 (8k) / 6.65 (16k) with negative two-arm marginal. It failed the registered gate and is not released; this chat checkpoint does not contain that retrieval training.

Downloads last month
94
Safetensors
Model size
0.5B params
Tensor type
F32
·
Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support

Collection including Ayaka157/RetNet-0.5B-v21.3-chat