Instructions to use Ayaka157/RetNet-143M with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use Ayaka157/RetNet-143M with Transformers:
# Use a pipeline as a high-level helper from transformers import pipeline pipe = pipeline("text-generation", model="Ayaka157/RetNet-143M", trust_remote_code=True)# pip install -U transformers accelerate # Load model directly from transformers import AutoModelForCausalLM model = AutoModelForCausalLM.from_pretrained("Ayaka157/RetNet-143M", trust_remote_code=True, device_map="auto") - Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- vLLM
How to use Ayaka157/RetNet-143M with vLLM:
Install from pip and serve model
# Install vLLM from pip: pip install vllm # Start the vLLM server: vllm serve "Ayaka157/RetNet-143M" # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:8000/v1/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "Ayaka157/RetNet-143M", "prompt": "Once upon a time,", "max_tokens": 512, "temperature": 0.5 }'Use Docker
docker model run hf.co/Ayaka157/RetNet-143M
- SGLang
How to use Ayaka157/RetNet-143M with SGLang:
Install from pip and serve model
# Install SGLang from pip: pip install sglang # Start the SGLang server: python3 -m sglang.launch_server \ --model-path "Ayaka157/RetNet-143M" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "Ayaka157/RetNet-143M", "prompt": "Once upon a time,", "max_tokens": 512, "temperature": 0.5 }'Use Docker images
docker run --gpus all \ --shm-size 32g \ -p 30000:30000 \ -v ~/.cache/huggingface:/root/.cache/huggingface \ --env "HF_TOKEN=<secret>" \ --ipc=host \ lmsysorg/sglang:latest \ python3 -m sglang.launch_server \ --model-path "Ayaka157/RetNet-143M" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "Ayaka157/RetNet-143M", "prompt": "Once upon a time,", "max_tokens": 512, "temperature": 0.5 }' - Docker Model Runner
How to use Ayaka157/RetNet-143M with Docker Model Runner:
docker model run hf.co/Ayaka157/RetNet-143M
Upload retention_fast.py with huggingface_hub
Browse files- retention_fast.py +180 -0
retention_fast.py
ADDED
|
@@ -0,0 +1,180 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""Batched-chunk chunkwise retention — mathematically identical to the reference
|
| 2 |
+
per-chunk loop in ``retnet_hf/modeling_retnet.py``, but with ~30x fewer kernel
|
| 3 |
+
launches.
|
| 4 |
+
|
| 5 |
+
Reference (per-chunk loop, 4 matmuls per chunk):
|
| 6 |
+
|
| 7 |
+
for i in chunks:
|
| 8 |
+
inner = (q_i @ k_i.T * inner_mask) @ v_i
|
| 9 |
+
cross = (q_i * cross_decay) @ state
|
| 10 |
+
out_i = inner + cross
|
| 11 |
+
state = chunk_decay * state + k_i.T @ (v_i * value_decay)
|
| 12 |
+
|
| 13 |
+
Batched form (3 matmuls total, independent of the chunk count):
|
| 14 |
+
|
| 15 |
+
1. U_i = k_i.T @ (v_i * value_decay) one batched matmul
|
| 16 |
+
2. S_prev_i = scan(U, chunk_decay, state) tiny sequential loop on
|
| 17 |
+
[B,H,kd,vd] tensors only
|
| 18 |
+
3. inner = (q_r @ k_r.T * inner_mask) @ v_r one batched matmul
|
| 19 |
+
cross = (q_r * cross_decay) @ S_prev one batched matmul
|
| 20 |
+
|
| 21 |
+
The scan in step 2 is deliberately *not* replaced by a cumsum: with
|
| 22 |
+
``chunk_decay = gamma^C`` and C=256, ``gamma^-j`` overflows fp32 for the fast
|
| 23 |
+
heads, while the log-depth pairwise scan costs more than it saves at 8 chunks.
|
| 24 |
+
"""
|
| 25 |
+
from __future__ import annotations
|
| 26 |
+
|
| 27 |
+
import torch
|
| 28 |
+
|
| 29 |
+
|
| 30 |
+
_INDEX_CACHE: dict = {}
|
| 31 |
+
|
| 32 |
+
|
| 33 |
+
def _index_tables(chunk_size: int, device):
|
| 34 |
+
"""``causal`` / ``expo`` / ``rel`` 只依赖 chunk_size,缓存掉。
|
| 35 |
+
|
| 36 |
+
原实现每次调用都重算 ``arange``/``sub``/``ge``/``clamp``,是 4 次无用的小
|
| 37 |
+
内核发射;而本算子的成本**几乎全是发射延迟**(见下)。
|
| 38 |
+
"""
|
| 39 |
+
key = (int(chunk_size), str(device))
|
| 40 |
+
tables = _INDEX_CACHE.get(key)
|
| 41 |
+
if tables is None:
|
| 42 |
+
rel = torch.arange(chunk_size, device=device)
|
| 43 |
+
delta = rel[:, None] - rel[None, :]
|
| 44 |
+
causal = delta >= 0
|
| 45 |
+
expo = delta.clamp_min(0).to(torch.float32)
|
| 46 |
+
tables = (causal, expo, rel.to(torch.float32))
|
| 47 |
+
_INDEX_CACHE[key] = tables
|
| 48 |
+
return tables
|
| 49 |
+
|
| 50 |
+
|
| 51 |
+
def _decay_tables(gamma: torch.Tensor, chunk_size: int, dtype: torch.dtype,
|
| 52 |
+
device: torch.device):
|
| 53 |
+
"""gamma: [H] (fp32)。返回 inner_mask [H,C,C], cross_decay [H,C],
|
| 54 |
+
value_decay [H,C], chunk_decay [H],均为 ``dtype``。
|
| 55 |
+
|
| 56 |
+
块内掩码用**对数空间** ``exp(log γ · delta)``,这是唯一在本模型门控范围
|
| 57 |
+
两端都安全的写法:
|
| 58 |
+
|
| 59 |
+
* 因式分解的 ``γ^i · γ^-j`` 一旦需要 ``γ^-255`` 就溢出 fp32。
|
| 60 |
+
``retnet_272m_rp`` 的 GLA 门控最小到 0.668,``0.668**-255 = 1e44 > 3.4e38``
|
| 61 |
+
→ Inf → 前向 NaN(``torch.where`` 只丢结果,不阻止 Inf 被算出来)。
|
| 62 |
+
* 朴素 ``γ ** delta`` 对 γ<1 安全(下溢到 0,那里本来就该是 0),
|
| 63 |
+
但对 γ>1 的门控不成立;对数形式两端都对。
|
| 64 |
+
|
| 65 |
+
NOTE(2026-10-10): 这个算子只有 26 万个元素却要花 **410 µs** —— 成本几乎全是
|
| 66 |
+
「22 个相互依赖的小内核」的发射延迟(Windows 上约 18 µs/次)。所以这里的
|
| 67 |
+
优化目标一律是**减少发射次数**,而不是减少计算量:
|
| 68 |
+
* 索引张量进缓存(省 4 次发射);
|
| 69 |
+
* 三张一维表复用同一个 ``γ^j``:[H,C]:``cross = γ^j·γ``、
|
| 70 |
+
``value = flip(γ^j)``、``chunk = γ^j[-1]·γ``,
|
| 71 |
+
把原来的 3 次 ``exp`` 压成 1 次。
|
| 72 |
+
"""
|
| 73 |
+
causal, expo, relf = _index_tables(chunk_size, device)
|
| 74 |
+
g = gamma.to(device=device, dtype=torch.float32)
|
| 75 |
+
# γ 可以合法地取到 0(门控被训练成「完全遗忘」),此时 log(0) = -inf,
|
| 76 |
+
# 而对角线 delta=0 会让 exp(-inf * 0) = exp(NaN) —— torch.where 只丢结果、
|
| 77 |
+
# 不阻止 NaN 被算出来,于是整层 NaN。夹一个正下限后
|
| 78 |
+
# exp(logγ·0) = 1、exp(logγ·k) 下溢到 0,正好是参考实现 gamma**delta 的
|
| 79 |
+
# 语义(0**0 = 1、0**k = 0),两条路径在 γ=0 处重新等价。
|
| 80 |
+
log_gamma = torch.log(g.clamp_min(torch.finfo(torch.float32).tiny))
|
| 81 |
+
|
| 82 |
+
inner_mask = torch.exp(log_gamma[:, None, None] * expo[None])
|
| 83 |
+
inner_mask = torch.where(causal, inner_mask,
|
| 84 |
+
torch.zeros((), device=device, dtype=torch.float32))
|
| 85 |
+
|
| 86 |
+
pow1d = torch.exp(log_gamma[:, None] * relf[None, :]) # [H,C] = γ^j
|
| 87 |
+
cross_decay = pow1d * g[:, None] # γ^(j+1)
|
| 88 |
+
value_decay = pow1d.flip(-1) # γ^(C-1-j)
|
| 89 |
+
chunk_decay = pow1d[:, -1] * g # γ^C
|
| 90 |
+
|
| 91 |
+
return (inner_mask.to(dtype), cross_decay.to(dtype),
|
| 92 |
+
value_decay.to(dtype), chunk_decay.to(dtype))
|
| 93 |
+
|
| 94 |
+
|
| 95 |
+
def chunkwise_batched(q, k, v, gamma, state, chunk_size):
|
| 96 |
+
"""Drop-in replacement for ``MultiScaleRetention._chunkwise_retention``.
|
| 97 |
+
|
| 98 |
+
q, k: [B,H,L,kd] v: [B,H,L,vd] gamma: [H] fp32
|
| 99 |
+
state: [B,H,kd,vd] or None -> (output [B,H,L,vd], new_state)
|
| 100 |
+
"""
|
| 101 |
+
B, H, L, kd = q.shape
|
| 102 |
+
vd = v.shape[-1]
|
| 103 |
+
if L % chunk_size != 0:
|
| 104 |
+
raise ValueError(f"seq_len {L} not a multiple of chunk_size {chunk_size}")
|
| 105 |
+
nc = L // chunk_size
|
| 106 |
+
C = chunk_size
|
| 107 |
+
dev = q.device
|
| 108 |
+
|
| 109 |
+
inner_mask, cross_decay, value_decay, chunk_decay = _decay_tables(
|
| 110 |
+
gamma.to(device=dev, dtype=torch.float32), C, q.dtype, dev)
|
| 111 |
+
|
| 112 |
+
has_state = state is not None
|
| 113 |
+
if state is None:
|
| 114 |
+
state = torch.zeros(B, H, kd, vd, device=dev, dtype=q.dtype)
|
| 115 |
+
|
| 116 |
+
# [B,H,nc,C,*] views — no copy (L is contiguous)
|
| 117 |
+
qr = q.view(B, H, nc, C, kd)
|
| 118 |
+
kr = k.view(B, H, nc, C, kd)
|
| 119 |
+
vr = v.view(B, H, nc, C, vd)
|
| 120 |
+
|
| 121 |
+
# (1) per-chunk state contributions, one batched matmul over all chunks
|
| 122 |
+
U = torch.matmul(kr.transpose(-1, -2),
|
| 123 |
+
vr * value_decay[None, :, None, :, None]) # [B,H,nc,kd,vd]
|
| 124 |
+
|
| 125 |
+
# (2) exclusive scan -> S_prev[i] = state before chunk i
|
| 126 |
+
#
|
| 127 |
+
# NOTE(2026-10-10): 原来的写法是 nc 次串行 ``s = decay*s + U[:,:,i]``
|
| 128 |
+
# 外加一次 ``torch.stack`` —— nc=16 时 33 次内核发射、实测 **591 µs**,
|
| 129 |
+
# 而每个参与的张量只有 0.5 MB:几乎全部是发射延迟。
|
| 130 |
+
#
|
| 131 |
+
# 关键观察是 ``decay = chunk_decay`` 在整个循环里**是常量**(它是 [H] 的
|
| 132 |
+
# 每头常数,不随 chunk 变),于是有闭式解
|
| 133 |
+
#
|
| 134 |
+
# S_prev[i] = d^i · state + Σ_{j<i} d^(i-1-j) · U_j
|
| 135 |
+
#
|
| 136 |
+
# 这正是对 ``T[h,i,j] = d_h^(i-1-j)``(下三角)的一次批量矩阵乘:
|
| 137 |
+
#
|
| 138 |
+
# S_all = T[None] @ U.reshape(B,H,nc,kd*vd) [B,H,nc+1,kd*vd]
|
| 139 |
+
#
|
| 140 |
+
# 多出来的一行 i=nc 恰好就是 new_state。**所有指数非负**,因此不存在
|
| 141 |
+
# §2.1 那种 ``γ^-j`` 溢出——d 再小也只是下溢到 0,那本就代表历史被忘光。
|
| 142 |
+
#
|
| 143 |
+
# NOTE(2026-10-10): 之前试过对数深度(Hillis-Steele)扫描把启动数从 2·nc
|
| 144 |
+
# 降到 ~2·log2(nc),**反而慢 2.8 倍**(3004 vs 1078 ms),内核数还从
|
| 145 |
+
# 10116 涨到 11472 —— 因为每轮都要 ``torch.cat`` 物化一个
|
| 146 |
+
# [B,H,nc,kd,vd] 张量,既贵又破坏 inductor 的融合。这里的闭式解不物化任何
|
| 147 |
+
# 额外的大张量(``T`` 只有 H·(nc+1)·nc 个元素),所以是另一回事。
|
| 148 |
+
# Collected in a list and stacked: writing into a torch.empty() buffer
|
| 149 |
+
# with slice assignment makes autograd emit CopySlices and leaves the
|
| 150 |
+
# untouched-in-backward buffer as uninitialised memory.
|
| 151 |
+
# 同样要夹下限:expo_t 在 (i=0, j=0) 处为 -1,clamp_min(0) 后是 0,
|
| 152 |
+
# 若 log_d 是 -inf 则 -inf*0 = NaN 会留在 T 的对角线上。
|
| 153 |
+
log_d = (torch.log(gamma.to(device=dev, dtype=torch.float32)
|
| 154 |
+
.clamp_min(torch.finfo(torch.float32).tiny))
|
| 155 |
+
* float(chunk_size))
|
| 156 |
+
idx = torch.arange(nc + 1, device=dev)
|
| 157 |
+
expo_t = idx[:, None] - torch.arange(nc, device=dev)[None, :] - 1
|
| 158 |
+
valid_t = expo_t >= 0
|
| 159 |
+
T = torch.exp(log_d[:, None, None] * expo_t.clamp_min(0).to(torch.float32)[None])
|
| 160 |
+
T = torch.where(valid_t[None], T, torch.zeros((), device=dev, dtype=torch.float32))
|
| 161 |
+
T = T.to(q.dtype) # [H,nc+1,nc]
|
| 162 |
+
|
| 163 |
+
U_flat = U.reshape(B, H, nc, kd * vd)
|
| 164 |
+
S_all = torch.matmul(T[None], U_flat) # [B,H,nc+1,kd*vd]
|
| 165 |
+
if has_state:
|
| 166 |
+
t_state = torch.exp(log_d[:, None] * idx.to(torch.float32)[None]).to(q.dtype)
|
| 167 |
+
S_all = S_all + t_state[None, :, :, None] * state.reshape(B, H, 1, kd * vd)
|
| 168 |
+
|
| 169 |
+
S_prev = S_all[:, :, :nc].reshape(B, H, nc, kd, vd)
|
| 170 |
+
new_state = S_all[:, :, nc].reshape(B, H, kd, vd)
|
| 171 |
+
|
| 172 |
+
# (3) inner chunk, one batched matmul
|
| 173 |
+
inner = torch.matmul(
|
| 174 |
+
torch.matmul(qr, kr.transpose(-1, -2)) * inner_mask[None, :, None, :, :], vr)
|
| 175 |
+
|
| 176 |
+
# (4) cross chunk, one batched matmul
|
| 177 |
+
cross = torch.matmul(qr * cross_decay[None, :, None, :, None], S_prev)
|
| 178 |
+
|
| 179 |
+
out = (inner + cross).reshape(B, H, L, vd)
|
| 180 |
+
return out, new_state
|