Instructions to use Ayaka157/RetNet-0.5B-v21.3-chat with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use Ayaka157/RetNet-0.5B-v21.3-chat with Transformers:
# Use a pipeline as a high-level helper from transformers import pipeline pipe = pipeline("text-generation", model="Ayaka157/RetNet-0.5B-v21.3-chat", trust_remote_code=True)# pip install -U transformers accelerate # Load model directly from transformers import AutoModelForCausalLM model = AutoModelForCausalLM.from_pretrained("Ayaka157/RetNet-0.5B-v21.3-chat", trust_remote_code=True, device_map="auto") - Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- vLLM
How to use Ayaka157/RetNet-0.5B-v21.3-chat with vLLM:
Install from pip and serve model
# Install vLLM from pip: pip install vllm # Start the vLLM server: vllm serve "Ayaka157/RetNet-0.5B-v21.3-chat" # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:8000/v1/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "Ayaka157/RetNet-0.5B-v21.3-chat", "prompt": "Once upon a time,", "max_tokens": 512, "temperature": 0.5 }'Use Docker
docker model run hf.co/Ayaka157/RetNet-0.5B-v21.3-chat
- SGLang
How to use Ayaka157/RetNet-0.5B-v21.3-chat with SGLang:
Install from pip and serve model
# Install SGLang from pip: pip install sglang # Start the SGLang server: python3 -m sglang.launch_server \ --model-path "Ayaka157/RetNet-0.5B-v21.3-chat" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "Ayaka157/RetNet-0.5B-v21.3-chat", "prompt": "Once upon a time,", "max_tokens": 512, "temperature": 0.5 }'Use Docker images
docker run --gpus all \ --shm-size 32g \ -p 30000:30000 \ -v ~/.cache/huggingface:/root/.cache/huggingface \ --env "HF_TOKEN=<secret>" \ --ipc=host \ lmsysorg/sglang:latest \ python3 -m sglang.launch_server \ --model-path "Ayaka157/RetNet-0.5B-v21.3-chat" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "Ayaka157/RetNet-0.5B-v21.3-chat", "prompt": "Once upon a time,", "max_tokens": 512, "temperature": 0.5 }' - Docker Model Runner
How to use Ayaka157/RetNet-0.5B-v21.3-chat with Docker Model Runner:
docker model run hf.co/Ayaka157/RetNet-0.5B-v21.3-chat
File size: 10,809 Bytes
8a1b5b3 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 | """Input-dependent GLA kernels (G=1 per-head scalar, and grouped G>1) + the gate module.
Parameterization (fixed after the Lead's review; the charter's `g=log σ(W_g x+b)` with `b=log γ` is
inconsistent because σ(log γ)≠γ):
z = W2 (W1 x) + b low-rank bottleneck, W1: d→r, W2: r→H·G, b ∈ R^{H·G}
g = min(z, 0) log-decay, fp32
exp(g) = decay per token per head/group
Function-preserving init: W2 = 0, b = log γ_h (per group) => g = log γ => exp(g) = γ, exactly the
scalar kernel's decay (the fast kernel also computes exp(log γ · Δ)). Identity measured: max|Δlogits| =
1.526e-05 (fp32 CPU, 20 layers converted, real tokens). If one insists on the log-σ form, the bias must
be logit(γ), not log γ; this file uses the direct log-decay form.
Numerics: gates, cumulative sums, masks and the scan matrices stay **fp32**; only the mask *product*
with qk and the output are cast to the activation dtype (bf16 in training). A bf16 mask is wrong: the
relative mask error reaches ~3 % at C=128 in bf16.
Variants:
* G=1 — **GLA-S**: one input-dependent scalar decay per head (cheapest; grouped GLA with one group of
all 64 key channels; NOT full per-channel GLA). Kernel: single [B,H,nc,C,C] mask, 1 qk matmul.
DELIVERED and measured: +14.7 % end-to-end fp32-gate eager upper bound (B), identity 1.5e-5.
* G>1 — grouped/per-channel: NOT DELIVERED (A's corrected design still costs +647 %/layer); the code
raises NotImplementedError by design - no crashing path and no priced G=8 branch without a tested
production kernel.
Math: Yang et al., Gated Linear Attention (arXiv:2312.06635), chunkwise form with the cumulative-gate
mask; here C=128 single-level, exact (not the sub-chunk approximation) because the gate is computed per
token before the scan. All exponents are differences of cumulative log-decays in the safe direction, so
no exp(+large) overflow appears even for γ=0.1.
Scope: **chunkwise training/evaluation + recurrent decode + chunkwise prefill** are implemented.
`MultiScaleRetention` routes `mode="chunkwise"` through this kernel and, when the recurrent GLA path is
enabled, `mode="recurrent"` through `gla_recurrent`; `RetNetForCausalLM.generate` uses chunkwise prefill
for GLA models and recurrent decode afterwards. The parallel mode remains scalar, but it is not used by
the GLA serving path. `gla_recurrent` is a per-token loop, not a fast batched decode kernel.
"""
from __future__ import annotations
import torch
class HeadGate(torch.nn.Module):
"""Low-rank input-dependent log-decay gate; parameters stay fp32 under a bf16 model."""
def __init__(self, d_model: int, n_heads: int, rank: int = 8, groups: int = 1,
gamma: torch.Tensor | None = None):
super().__init__()
self.d_model, self.n_heads, self.rank, self.groups = d_model, n_heads, rank, groups
self.w1 = torch.nn.Linear(d_model, rank, bias=False).float()
self.w2 = torch.nn.Linear(rank, n_heads * groups, bias=True).float()
torch.nn.init.normal_(self.w1.weight, std=0.02)
torch.nn.init.zeros_(self.w2.weight)
gam = gamma.detach().float().clamp(1e-4, 1 - 1e-6) if gamma is not None \
else torch.full((n_heads,), 0.9)
with torch.no_grad():
self.w2.bias.copy_(torch.log(gam).repeat_interleave(groups))
def forward(self, x): # x [B,L,d] -> fp32 log decay [B,L,H,G]
# FP32 parameters alone do not prevent AMP from downcasting Linear.
with torch.autocast(device_type=x.device.type, enabled=False):
z = self.w2(self.w1(x.float()))
return torch.clamp(z, max=0.0)
def extra_params(self) -> int:
return sum(p.numel() for p in self.parameters())
@torch.no_grad()
def init_gate_from_gamma(gate: HeadGate, gamma: torch.Tensor) -> None:
g = gamma.detach().float().clamp(1e-4, 1 - 1e-6)
gate.w2.weight.zero_()
gate.w2.bias.copy_(torch.log(g).repeat_interleave(gate.groups))
def _split(q, k, v, chunk_size):
B, H, L, kd = q.shape
C = int(chunk_size)
assert L % C == 0, (L, C)
return B, H, L, kd, v.shape[-1], L // C, C
def _causal(C, dev):
ar = torch.arange(C, device=dev)
return ar[:, None] >= ar[None, :]
def _scan_tables(Glast, nc, dev):
"""Per-(batch,head,group) data-dependent scan matrices: T[i,j]=d^(i-1-j), T2[i,j]=d^(i-j), j<i."""
P = torch.cumsum(Glast, dim=-1) # inclusive
Ce = P - Glast # exclusive
lower = torch.tril(torch.ones(nc, nc, device=dev, dtype=torch.bool), -1)
diff = Ce[..., :, None] - P[..., None, :]
diff = torch.where(lower, diff, torch.zeros((), device=dev, dtype=diff.dtype))
T = torch.exp(diff) * lower.to(diff.dtype)
Ce2 = torch.cat([torch.zeros_like(P[..., :1]), P], dim=-1) # [.., nc+1]
diff2 = Ce2[..., :, None] - P[..., None, :]
valid2 = torch.arange(nc + 1, device=dev)[:, None] > torch.arange(nc, device=dev)[None, :]
diff2 = torch.where(valid2, diff2, torch.zeros((), device=dev, dtype=diff2.dtype))
T2 = torch.exp(diff2) * valid2.to(diff2.dtype)
return T, T2, Ce, Ce2
@torch.no_grad()
def _cast(x, dt):
return x.to(dt)
def _gla_chunkwise_exact(q, k, v, g_t, state, chunk_size: int):
"""Chunkwise GLA with input-dependent grouped decay.
q,k: [B,H,L,kd]; v: [B,H,L,vd]; g_t: [B,L,H] (G=1) or [B,L,H,G] fp32 log-decay ≤0.
Returns (out [B,H,L,vd], new_state [B,H,kd,vd]).
"""
B, H, L, kd, vd, nc, C = _split(q, k, v, chunk_size)
dev, dt = q.device, q.dtype
G_ = g_t.shape[-1] if g_t.dim() == 4 else 1
if G_ == 1:
g = (g_t[..., 0] if g_t.dim() == 4 else g_t).transpose(1, 2) # [B,H,L]
g = g.reshape(B, H, nc, C).float()
qr = q.view(B, H, nc, C, kd)
kr = k.view(B, H, nc, C, kd)
vr = v.view(B, H, nc, C, vd)
Gc = torch.cumsum(g, dim=-1) # [B,H,nc,C]
causal = _causal(C, dev)
delta = Gc[..., :, None] - Gc[..., None, :]
delta = torch.where(causal, delta, torch.zeros((), device=dev))
mask = (torch.exp(delta) * causal.to(delta.dtype)).to(dt)
qk = torch.matmul(qr, kr.transpose(-1, -2))
inner = torch.matmul(qk * mask, vr)
Glast = Gc[..., -1]
val_decay = torch.exp(Glast[..., None] - Gc) # [B,H,nc,C]
U = torch.matmul(kr.transpose(-1, -2), (vr.float() * val_decay[..., None]).to(dt))
Uflat = U.reshape(B, H, nc, kd * vd).float()
T, T2, Ce, Ce2 = _scan_tables(Glast, nc, dev)
S_prev = torch.matmul(T, Uflat).view(B, H, nc, kd, vd).to(dt)
S_all = torch.matmul(T2, Uflat).view(B, H, nc + 1, kd, vd).to(dt)
if state is not None:
st = state.reshape(B, H, 1, kd, vd)
S_prev = S_prev + torch.exp(Ce)[..., None, None].to(dt) * st
S_all = S_all + torch.exp(Ce2)[..., None, None].to(dt) * st
cross = torch.matmul((qr.float() * torch.exp(Gc)[..., None]).to(dt), S_prev)
return (inner + cross).reshape(B, H, L, vd), S_all[:, :, nc]
# ---------------- grouped variant G>1: NOT DELIVERED ----------------
# The grouped kernel is not shipped in this round: the first prototype's einsum/permute ordering
# was wrong (A B5 / B B3 measured the crash) and a corrected design still costs +647 %/layer
# (A's corrected vectorised prototype: 16.928 ms vs 2.265 ms) => ~+97 % end-to-end at B=1.
# Raising loudly beats shipping a path that silently computes a different function.
raise NotImplementedError(
"gla_chunkwise: grouped G>1 is not delivered (needs a tested production kernel; "
"see GLA_PLAN_v2 §3/§9). Use groups=1 (GLA-S) or the scalar fast kernel.")
def gla_chunkwise(q, k, v, g_t, state, chunk_size: int):
"""Padded wrapper: supports prompt lengths not divisible by the chunk size.
Zero k/v/g at pad positions leave the recurrent state unchanged; the returned out is trimmed.
"""
B, H, L, kd = q.shape
vd = v.shape[-1]
C = int(chunk_size)
pad = (-L) % C
if pad == 0:
return _gla_chunkwise_exact(q, k, v, g_t, state, C)
qp = torch.nn.functional.pad(q, (0, 0, 0, pad))
kp = torch.nn.functional.pad(k, (0, 0, 0, pad))
vp = torch.nn.functional.pad(v, (0, 0, 0, pad))
if g_t.dim() == 4:
gp = torch.nn.functional.pad(g_t, (0, 0, 0, pad))
else:
gp = torch.nn.functional.pad(g_t, (0, 0, 0, pad))
out, new_state = _gla_chunkwise_exact(qp, kp, vp, gp, state, C)
return out[:, :, :L], new_state
def gla_recurrent(q, k, v, g_t, state=None):
return gla_recurrent_reference(q, k, v, g_t, state)
def gla_recurrent_reference(q, k, v, g_t, state=None):
"""Slow decode reference (G=1), with fp32 decay and persistent state."""
B, H, L, kd = q.shape
if (g_t.dim() == 4 and g_t.shape[-1] > 1):
raise NotImplementedError("recurrent reference implemented for G=1 only")
g = (g_t[..., 0] if g_t.dim() == 4 else g_t).transpose(1, 2).float()
# exp(g) near one must never pass through bf16: that would erase slow
# decay. Promote a chunkwise-prefill state once and keep subsequent
# updates in fp32; this cannot recover precision lost during prefill.
output_dtype = q.dtype
q, k, v = q.float(), k.float(), v.float()
if state is None:
state = torch.zeros(B, H, kd, v.shape[-1], device=q.device, dtype=torch.float32)
else:
state = state.float()
outs = []
for t in range(L):
state = torch.exp(g[:, :, t])[..., None, None] * state + \
k[:, :, t, :].unsqueeze(-1) * v[:, :, t, :].unsqueeze(-2)
outs.append(torch.sum(q[:, :, t, :].unsqueeze(-1) * state, dim=-2))
return torch.stack(outs, dim=2).to(output_dtype), state
def count_params(d_model=1408, n_heads=22, n_layers=20, rank=8, groups=1):
per = rank * d_model + rank * n_heads * groups + n_heads * groups
total = per * n_layers
return per, total, 100.0 * total / 504_179_896
if __name__ == "__main__":
for r, gp in ((4, 1), (8, 1), (16, 1), (8, 4), (8, 8)):
per, total, pct = count_params(rank=r, groups=gp)
print(f"low-rank rank={r:2d} G={gp}: {per:6,}/layer {total:9,} total {pct:.4f}%")
per = 1408 * 22 + 22
print(f"per-head linear (no bottleneck): {per:,}/layer {per * 20:,} total "
f"{100 * per * 20 / 504179896:.3f}%")
full = 20 * (1408 * 1408 + 1408)
print(f"full d->H*kd: {full:,} total {100 * full / 504179896:.2f}%")
|