Instructions to use Ayaka157/RetNet-0.5B-16k with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use Ayaka157/RetNet-0.5B-16k with Transformers:
# Use a pipeline as a high-level helper from transformers import pipeline pipe = pipeline("text-generation", model="Ayaka157/RetNet-0.5B-16k", trust_remote_code=True)# pip install -U transformers accelerate # Load model directly from transformers import AutoModelForCausalLM model = AutoModelForCausalLM.from_pretrained("Ayaka157/RetNet-0.5B-16k", trust_remote_code=True, device_map="auto") - Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- vLLM
How to use Ayaka157/RetNet-0.5B-16k with vLLM:
Install from pip and serve model
# Install vLLM from pip: pip install vllm # Start the vLLM server: vllm serve "Ayaka157/RetNet-0.5B-16k" # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:8000/v1/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "Ayaka157/RetNet-0.5B-16k", "prompt": "Once upon a time,", "max_tokens": 512, "temperature": 0.5 }'Use Docker
docker model run hf.co/Ayaka157/RetNet-0.5B-16k
- SGLang
How to use Ayaka157/RetNet-0.5B-16k with SGLang:
Install from pip and serve model
# Install SGLang from pip: pip install sglang # Start the SGLang server: python3 -m sglang.launch_server \ --model-path "Ayaka157/RetNet-0.5B-16k" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "Ayaka157/RetNet-0.5B-16k", "prompt": "Once upon a time,", "max_tokens": 512, "temperature": 0.5 }'Use Docker images
docker run --gpus all \ --shm-size 32g \ -p 30000:30000 \ -v ~/.cache/huggingface:/root/.cache/huggingface \ --env "HF_TOKEN=<secret>" \ --ipc=host \ lmsysorg/sglang:latest \ python3 -m sglang.launch_server \ --model-path "Ayaka157/RetNet-0.5B-16k" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "Ayaka157/RetNet-0.5B-16k", "prompt": "Once upon a time,", "max_tokens": 512, "temperature": 0.5 }' - Docker Model Runner
How to use Ayaka157/RetNet-0.5B-16k with Docker Model Runner:
docker model run hf.co/Ayaka157/RetNet-0.5B-16k
RetNet-0.5B-16k β long-context research export (v21.2)
Status: Tier-0 partial pass β long-document modelling improved a lot, but needle retrieval and the effective-context test failed. This is a research export, not a claim of usable 16k context.
Base: v21.1 C-end + 20-head Ξ³ calibration (C1). 13,000 steps at 16,384 tokens (7,000 long-document + 4,500 mixed + 1,500 chat16k), 0.852 B tokens, 2ΓRTX PRO 6000, lr 2e-5 β 2e-6.
| Parameters | 504,179,896 |
| Layers / width / heads | 20 / 1408 / 22 |
| Vocab / training context | 20,000 ByteLevel BPE / 16,384 |
| Decay | per-head learned Ξ³, calibrated C1; GLA branch present with gla_rank=0 |
| Storage | fp32 (model.safetensors, 1.88 GiB) |
| SHA256 (model.safetensors) | cace5697ddaf9a43ec8ae51a3761bf4fab57ca48ee89bb0aebf2a1fdfd62bd35 |
Measured results (val_pack, 5 files / 4,454,400 tokens, skypile excluded)
| Gate | Base (C-end) | This export | Delta | Verdict |
|---|---|---|---|---|
| 2k criterion | 2.4885 | 2.5524 | +0.0639 | T2 pass (β€ +0.10), T1 fail (β€ +0.05) |
| 16k within-doc long-tail | 3.2841 | 2.5942 | β0.690 | pass (T1/T2) |
| by-position 4k / 8k | 3.2124 / 3.2300 | 2.5941 / 2.5967 | β0.62 / β0.63 | β |
| needle@16k (120/depth, pooled) | 0.1000 (chance) | 0.0978 | β0.002 | fail (needs Ξ β₯ +0.10 and abs β₯ 0.18) |
two-arm Ξ_marginal_long@16k |
+0.0418 | +0.0407 | β0.001 | absolute pass, no gain |
The two-arm metric never improved over the seed, so the export demonstrates long-document language modelling at 16k, not retrieval or effective use of the context.
Usage
from transformers import AutoModelForCausalLM, AutoTokenizer
repo = "Ayaka157/RetNet-0.5B-16k"
tok = AutoTokenizer.from_pretrained(repo, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(repo, trust_remote_code=True, torch_dtype="float32").eval()
The chunkwise path is the supported one. For long sequences set RETNET_FAST_CHUNKWISE=1; the 16k
training run needed the compiled path (1.66 it/s on 2ΓPRO 6000, 83.1 GiB peak). Without compilation
it is much slower and memory-hungry.
Limitations
- Needle retrieval is at chance. Do not use this for retrieval, RAG, or as a long-context assistant.
- The 2k criterion regressed by +0.0639; for chat use the released v21.3-chat instead.
- "16k capability" is not claimed: the registered Tier-2 gate did not pass.
All checkpoints
Part of the RetNet-0.5B collection: https://huggingface.co/collections/Ayaka157/retnet-05b-from-scratch-training-line-6afc5fe4007c9acacd1cb52f
Files
model.safetensors, config.json, modeling_retnet.py, retention_fast.py, fused_ce.py,
bpe_tokenizer.json, tokenizer files. Full training log and figures:
https://github.com/Ayaka157/RetNet
- Downloads last month
- 49