Instructions to use Motif-Technologies/optimizer with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Kernels
How to use Motif-Technologies/optimizer with Kernels:
# !pip install kernels from kernels import get_kernel # a version (or an explicit revision) is required; see the "Files and versions" tab for the available ones kernel = get_kernel("Motif-Technologies/optimizer", version=1) - Notebooks
- Google Colab
- Kaggle
Commit ·
349b794
1
Parent(s): 8d60c30
fix(qk_clip): scale q_pe rows by full γ, not γ^(1-k_ratio) [skip-build]
Browse filesMLA logit = q_nope·k_nope (content) + q_pe·k_rope (rotary). The decoupled
rotary key k_rope is shared across heads and left unchanged, so the rotary
term is reduced by the query side alone. Since PR #32 introduced k_ratio,
wq_b scaled q_pe by γ^(1-k_ratio) uniformly with q_nope, making the rotary
term shrink by γ^(1-k_ratio) instead of γ — an under-clip for any k_ratio > 0.
Give q_pe the full γ (exponent 1) regardless of k_ratio; the k_ratio split
only applies to the content (nope) rows, whose K side is scalable. k_ratio=0
is unaffected (Q already got full γ there).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
torch-ext/optimizer/qk_clip.py
CHANGED
|
@@ -133,11 +133,15 @@ def compute_scales(p, qk_clip_state, name: str = ''):
|
|
| 133 |
|
| 134 |
Scale convention (let γ = threshold/value):
|
| 135 |
MHA/GQA wq/wk : scale = √γ on both Q and K; product = γ.
|
| 136 |
-
MLA wq_b : scale = γ^(1 - k_ratio)
|
| 137 |
-
|
|
|
|
|
|
|
|
|
|
| 138 |
MLA wkv_b : scale = γ^k_ratio applied to k_nope rows only (v rows
|
| 139 |
untouched). Returns None when k_ratio=0 (K unclipped).
|
| 140 |
-
|
|
|
|
| 141 |
"""
|
| 142 |
kind = qk_clip_state.kind
|
| 143 |
|
|
@@ -203,7 +207,8 @@ def qk_clip(p, scales, info):
|
|
| 203 |
Exponent semantics depend on kind — see compute_scales docstring.
|
| 204 |
info: QKClipInfo with kind, head_dim, and MLA sub-head dimensions.
|
| 205 |
|
| 206 |
-
MLA wq_b: γ^(1-k_ratio)
|
|
|
|
| 207 |
MLA wkv_b: γ^k_ratio applied to k_nope rows only; v rows untouched.
|
| 208 |
MHA/GQA: √γ applied to all rows in each head.
|
| 209 |
"""
|
|
@@ -220,7 +225,16 @@ def qk_clip(p, scales, info):
|
|
| 220 |
qk_head_dim = qk_nope + info.qk_rope_head_dim
|
| 221 |
W_3d = W.view(-1, qk_head_dim, W.shape[1]) # [H, qk_head_dim, in_dim]
|
| 222 |
W_3d[:, :qk_nope, :].mul_(scales.view(-1, 1, 1)) # q_nope → γ^(1-k_ratio)
|
| 223 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 224 |
|
| 225 |
elif info.kind == 'wkv_b':
|
| 226 |
qk_nope = info.qk_nope_head_dim
|
|
|
|
| 133 |
|
| 134 |
Scale convention (let γ = threshold/value):
|
| 135 |
MHA/GQA wq/wk : scale = √γ on both Q and K; product = γ.
|
| 136 |
+
MLA wq_b : scale = γ^(1 - k_ratio) for the q_nope rows. The q_pe
|
| 137 |
+
(rotary) rows instead take the full γ in qk_clip, since
|
| 138 |
+
the shared rotary key is left unchanged and the rotary
|
| 139 |
+
query must carry the whole reduction. k_ratio=0 → full
|
| 140 |
+
γ on all Q rows (K unclipped).
|
| 141 |
MLA wkv_b : scale = γ^k_ratio applied to k_nope rows only (v rows
|
| 142 |
untouched). Returns None when k_ratio=0 (K unclipped).
|
| 143 |
+
Per logit component: content = γ^(1-k_ratio)·γ^k_ratio = γ;
|
| 144 |
+
rotary = γ·1 = γ. Both are reduced by exactly γ end-to-end.
|
| 145 |
"""
|
| 146 |
kind = qk_clip_state.kind
|
| 147 |
|
|
|
|
| 207 |
Exponent semantics depend on kind — see compute_scales docstring.
|
| 208 |
info: QKClipInfo with kind, head_dim, and MLA sub-head dimensions.
|
| 209 |
|
| 210 |
+
MLA wq_b: γ^(1-k_ratio) on q_nope rows; full γ on q_pe rows (the shared
|
| 211 |
+
rotary key is unchanged, so the rotary query carries the whole γ).
|
| 212 |
MLA wkv_b: γ^k_ratio applied to k_nope rows only; v rows untouched.
|
| 213 |
MHA/GQA: √γ applied to all rows in each head.
|
| 214 |
"""
|
|
|
|
| 225 |
qk_head_dim = qk_nope + info.qk_rope_head_dim
|
| 226 |
W_3d = W.view(-1, qk_head_dim, W.shape[1]) # [H, qk_head_dim, in_dim]
|
| 227 |
W_3d[:, :qk_nope, :].mul_(scales.view(-1, 1, 1)) # q_nope → γ^(1-k_ratio)
|
| 228 |
+
# q_pe: the decoupled rotary key is shared across heads and left
|
| 229 |
+
# unchanged, so the rotary logit term q_pe·k_rope is reduced by the
|
| 230 |
+
# query side alone. The rotary query rows must therefore carry the full
|
| 231 |
+
# γ (exponent 1), independent of k_ratio; scaling them by γ^(1-k_ratio)
|
| 232 |
+
# like q_nope would under-clip the rotary logit whenever k_ratio > 0.
|
| 233 |
+
if info.k_ratio < 1.0:
|
| 234 |
+
q_pe_scales = scales**(1.0 / (1.0 - info.k_ratio)) # γ^(1-k) → γ
|
| 235 |
+
else:
|
| 236 |
+
q_pe_scales = scales # degenerate: all burden on K, Q unclipped
|
| 237 |
+
W_3d[:, qk_nope:, :].mul_(q_pe_scales.view(-1, 1, 1)) # q_pe → γ
|
| 238 |
|
| 239 |
elif info.kind == 'wkv_b':
|
| 240 |
qk_nope = info.qk_nope_head_dim
|