Kernels
superblock22 Claude Opus 4.8 commited on
Commit
349b794
·
1 Parent(s): 8d60c30

fix(qk_clip): scale q_pe rows by full γ, not γ^(1-k_ratio) [skip-build]

Browse files

MLA logit = q_nope·k_nope (content) + q_pe·k_rope (rotary). The decoupled
rotary key k_rope is shared across heads and left unchanged, so the rotary
term is reduced by the query side alone. Since PR #32 introduced k_ratio,
wq_b scaled q_pe by γ^(1-k_ratio) uniformly with q_nope, making the rotary
term shrink by γ^(1-k_ratio) instead of γ — an under-clip for any k_ratio > 0.

Give q_pe the full γ (exponent 1) regardless of k_ratio; the k_ratio split
only applies to the content (nope) rows, whose K side is scalable. k_ratio=0
is unaffected (Q already got full γ there).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

Files changed (1) hide show
  1. torch-ext/optimizer/qk_clip.py +19 -5
torch-ext/optimizer/qk_clip.py CHANGED
@@ -133,11 +133,15 @@ def compute_scales(p, qk_clip_state, name: str = ''):
133
 
134
  Scale convention (let γ = threshold/value):
135
  MHA/GQA wq/wk : scale = √γ on both Q and K; product = γ.
136
- MLA wq_b : scale = γ^(1 - k_ratio). Q carries (1-k_ratio) of the
137
- log-reduction; k_ratio=0 → full γ on Q only (K unclipped).
 
 
 
138
  MLA wkv_b : scale = γ^k_ratio applied to k_nope rows only (v rows
139
  untouched). Returns None when k_ratio=0 (K unclipped).
140
- Together wq_b × wkv_b deliver γ^(1-k_ratio) × γ^k_ratio = γ end-to-end.
 
141
  """
142
  kind = qk_clip_state.kind
143
 
@@ -203,7 +207,8 @@ def qk_clip(p, scales, info):
203
  Exponent semantics depend on kind — see compute_scales docstring.
204
  info: QKClipInfo with kind, head_dim, and MLA sub-head dimensions.
205
 
206
- MLA wq_b: γ^(1-k_ratio) applied uniformly to q_nope and q_pe rows.
 
207
  MLA wkv_b: γ^k_ratio applied to k_nope rows only; v rows untouched.
208
  MHA/GQA: √γ applied to all rows in each head.
209
  """
@@ -220,7 +225,16 @@ def qk_clip(p, scales, info):
220
  qk_head_dim = qk_nope + info.qk_rope_head_dim
221
  W_3d = W.view(-1, qk_head_dim, W.shape[1]) # [H, qk_head_dim, in_dim]
222
  W_3d[:, :qk_nope, :].mul_(scales.view(-1, 1, 1)) # q_nope → γ^(1-k_ratio)
223
- W_3d[:, qk_nope:, :].mul_(scales.view(-1, 1, 1)) # q_pe → γ^(1-k_ratio)
 
 
 
 
 
 
 
 
 
224
 
225
  elif info.kind == 'wkv_b':
226
  qk_nope = info.qk_nope_head_dim
 
133
 
134
  Scale convention (let γ = threshold/value):
135
  MHA/GQA wq/wk : scale = √γ on both Q and K; product = γ.
136
+ MLA wq_b : scale = γ^(1 - k_ratio) for the q_nope rows. The q_pe
137
+ (rotary) rows instead take the full γ in qk_clip, since
138
+ the shared rotary key is left unchanged and the rotary
139
+ query must carry the whole reduction. k_ratio=0 → full
140
+ γ on all Q rows (K unclipped).
141
  MLA wkv_b : scale = γ^k_ratio applied to k_nope rows only (v rows
142
  untouched). Returns None when k_ratio=0 (K unclipped).
143
+ Per logit component: content = γ^(1-k_ratio)·γ^k_ratio = γ;
144
+ rotary = γ·1 = γ. Both are reduced by exactly γ end-to-end.
145
  """
146
  kind = qk_clip_state.kind
147
 
 
207
  Exponent semantics depend on kind — see compute_scales docstring.
208
  info: QKClipInfo with kind, head_dim, and MLA sub-head dimensions.
209
 
210
+ MLA wq_b: γ^(1-k_ratio) on q_nope rows; full γ on q_pe rows (the shared
211
+ rotary key is unchanged, so the rotary query carries the whole γ).
212
  MLA wkv_b: γ^k_ratio applied to k_nope rows only; v rows untouched.
213
  MHA/GQA: √γ applied to all rows in each head.
214
  """
 
225
  qk_head_dim = qk_nope + info.qk_rope_head_dim
226
  W_3d = W.view(-1, qk_head_dim, W.shape[1]) # [H, qk_head_dim, in_dim]
227
  W_3d[:, :qk_nope, :].mul_(scales.view(-1, 1, 1)) # q_nope → γ^(1-k_ratio)
228
+ # q_pe: the decoupled rotary key is shared across heads and left
229
+ # unchanged, so the rotary logit term q_pe·k_rope is reduced by the
230
+ # query side alone. The rotary query rows must therefore carry the full
231
+ # γ (exponent 1), independent of k_ratio; scaling them by γ^(1-k_ratio)
232
+ # like q_nope would under-clip the rotary logit whenever k_ratio > 0.
233
+ if info.k_ratio < 1.0:
234
+ q_pe_scales = scales**(1.0 / (1.0 - info.k_ratio)) # γ^(1-k) → γ
235
+ else:
236
+ q_pe_scales = scales # degenerate: all burden on K, Q unclipped
237
+ W_3d[:, qk_nope:, :].mul_(q_pe_scales.view(-1, 1, 1)) # q_pe → γ
238
 
239
  elif info.kind == 'wkv_b':
240
  qk_nope = info.qk_nope_head_dim