--- license: mit language: - zh - en library_name: transformers pipeline_tag: text-generation tags: - retnet - retention - from-scratch --- # RetNet-100M (first chat model) The first chat model in this project, from 2026-08-20. It talks, badly. It is here because it is the starting point of a line that ends at [RetNet-0.5B](https://huggingface.co/Ayaka157/RetNet-0.5B), and because the loss curve of a model this size is worth having. Code and the engineering log: ## Configuration | | | | --- | --- | | Parameters | 100,370,064 | | Layers | 12 | | `d_model` | 768 | | Heads | 12 | | Vocab | 20000 | | Context | 512 | | Step | 16,000 | | Validation loss | 4.3768 (perplexity ≈ 79.6) | Trained on a mixed Chinese/English dialogue corpus. Peak validation loss was measured on the training machine; the exact corpus revision is recorded in the project's [`docs/12_训练状态盘点.md`](https://github.com/Ayaka157/RetNet/blob/main/docs/12_训练状态盘点.md). ## Provenance This checkpoint was trained with the handwritten implementation (`retnet/model.py`) and converted to Hugging Face format with `scripts/migrate_to_hf.py`. The conversion verifies numerical identity against the original: the maximum absolute difference in logits is **0.000e+00** at sequence lengths 1, 8 and 128, and all 172 state-dict keys load strictly with no renaming. ## Usage ```python import torch from transformers import AutoModelForCausalLM, AutoTokenizer repo = "Ayaka157/RetNet-100M" tok = AutoTokenizer.from_pretrained(repo, trust_remote_code=True) model = AutoModelForCausalLM.from_pretrained(repo, trust_remote_code=True, dtype=torch.float32).eval() ids = tok("用户:你好。\n助手:")["input_ids"] out = model.generate(torch.tensor([ids]), max_new_tokens=64, temperature=0.7) print(tok.decode(out[0].tolist()[len(ids):])) ``` `mode` in `config.json` is `parallel` for inference; the training run used `chunkwise`. The chat format is `` + optional `设定:...` persona block + `用户:...` + `助手:...` + ``, implemented in [`retnet/chat.py`](https://github.com/Ayaka157/RetNet/blob/main/retnet/chat.py). Keep prompts to the 512-token context window this model was trained on. ## License MIT.