deshwalmahesh commited on
Commit
8e925cf
·
verified ·
1 Parent(s): bfdad89

RoPE fp32 + gradient checkpointing + SDPA attention backend

Browse files

Three fixes to `modeling_domynedge.py`. Default CUDA `varlen` path untouched (SDPA is opt-in): no GPU regression. Checks: CPU, tiny random config, no weights (torch 2.15).

## 1. RoPE precision — resolves the `TODO` in `DomynEdgeRotaryEmbedding.forward`

- **Found:** angles round to bf16 at large positions — under `autocast(bfloat16)` the matmul runs bf16, and after `model.bfloat16()` the `inv_freq` buffer is bf16 (both noted in the TODO).
- **Why:** cos/sin drift at long context. Real val rows (bf16): prob shift vs fp32 ~8% at <1k, ~53% at 4–8k; accuracy 53% → 27% at 4–8k.
- **Fix:** disable autocast, recompute `inv_freq` in fp32 in `forward` (as the TODO proposes).
- **Proved:** cos error at pos 7000 = **0.0** (fix) vs **2.0** (old).

## 2. Gradient checkpointing — `supports_gradient_checkpointing` was `False`

- **Found:** flag `False` → `gradient_checkpointing_enable()` is a no-op.
- **Why:** 16,384-token training needs ~161 GB of activations — won't fit one 141 GB GPU without recompute.
- **Fix:** layers are stateless → set flag, route layer loop through `_gradient_checkpointing_func` when training.
- **Proved:** gradients identical to non-checkpointed (max diff **0.0**).

## 3. SDPA attention — `_supports_sdpa` was `False`

- **Found:** attention hard-wired to CUDA `torch.nn.attention.varlen`; `attn_implementation` ignored, so no portable path.
- **Why:** can't run without the flash/varlen kernel; no fp32-capable attention.
- **Fix:** opt-in `attn_implementation="sdpa"` branch — same GQA + per-layer sliding window + padding mask via `scaled_dot_product_attention`; default path unchanged.
- **Proved:** matches independent masked-attention reference to **1.5e-8**; full forward runs on CPU with finite logits (impossible on varlen). GPU `varlen`-vs-SDPA parity left to confirm on GPU.

## Hardware

| | before | after |
|---|---|---|
| H200 / A100 (flash varlen) | ✅ | ✅ unchanged + gradient checkpointing |
| T4 / older GPU (no varlen) | ❌ | ✅ `sdpa` |
| CPU | ❌ | ✅ `sdpa` (separately tested) |

Files changed (1) hide show
  1. modeling_domynedge.py +72 -21
modeling_domynedge.py CHANGED
@@ -218,18 +218,19 @@ class DomynEdgeRotaryEmbedding(torch.nn.Module):
218
 
219
  @torch.no_grad()
220
  def forward(self, x, position_ids):
221
- inv_freq_expanded = self.inv_freq[None, :, None].float().expand(position_ids.shape[0], -1, 1).to(x.device)
222
- position_ids_expanded = position_ids[:, None, :].float()
223
-
224
- # TODO: the angles lose precision in two cases, both wrong at large positions:
225
- # - under torch.autocast(bfloat16) the matmul below runs in bf16 despite the `.float()` casts;
226
- # - after model.to(torch.bfloat16) / model.bfloat16() the `inv_freq` buffer itself is cast to bf16.
227
- # Loading with `from_pretrained(dtype=torch.bfloat16)` and no autocast is not affected. Candidate fix:
228
- # recompute `inv_freq` in fp32 here, with autocast disabled, instead of reading the buffer.
229
- freqs = (inv_freq_expanded.float() @ position_ids_expanded.float()).transpose(1, 2)
230
- emb = torch.cat((freqs, freqs), dim=-1)
231
- cos = emb.cos() * self.attention_scaling
232
- sin = emb.sin() * self.attention_scaling
 
233
 
234
  return cos, sin
235
 
@@ -322,6 +323,11 @@ class DomynEdgeAttention(torch.nn.Module):
322
  cos, sin = position_embeddings
323
  q, k = self.apply_rope(q, k, cos, sin, unsqueeze_dim=2)
324
 
 
 
 
 
 
325
  if past_key_values is not None:
326
  k, v = k.transpose(1, 2), v.transpose(1, 2)
327
  k, v = past_key_values.update(k, v, self.layer_idx)
@@ -355,6 +361,44 @@ class DomynEdgeAttention(torch.nn.Module):
355
  attn_output = attn_output.view(batch_size, sequence_length, self.config.num_attention_heads * self.head_dim)
356
  return self.o_proj(attn_output)
357
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
358
 
359
  class DomynEdgeDecoderLayer(torch.nn.Module):
360
  def __init__(self, config: DomynEdgeConfig, layer_idx: int):
@@ -398,9 +442,9 @@ class DomynEdgePreTrainedModel(transformers.PreTrainedModel):
398
 
399
  config: DomynEdgeConfig
400
  base_model_prefix = "model"
401
- supports_gradient_checkpointing = False
402
  _supports_flash_attn = True
403
- _supports_sdpa = False
404
  _supports_flex_attn = False
405
 
406
  _can_compile_fullgraph = True
@@ -425,6 +469,7 @@ class DomynEdgeModel(DomynEdgePreTrainedModel):
425
  )
426
  self.norm = torch.nn.RMSNorm(config.hidden_size, eps=config.rms_norm_eps)
427
  self.rotary_emb = DomynEdgeRotaryEmbedding(config=config)
 
428
 
429
  # Initialize weights and apply final processing
430
  self.post_init()
@@ -469,13 +514,19 @@ class DomynEdgeModel(DomynEdgePreTrainedModel):
469
  )
470
 
471
  for decoder_layer in self.layers:
472
- hidden_states = decoder_layer(
473
- hidden_states,
474
- attention_mask=attention_mask,
475
- position_embeddings=position_embeddings,
476
- past_key_values=past_key_values,
477
- **kwargs,
478
- )
 
 
 
 
 
 
479
 
480
  hidden_states = self.norm(hidden_states)
481
  return transformers.modeling_outputs.BaseModelOutputWithPast(
 
218
 
219
  @torch.no_grad()
220
  def forward(self, x, position_ids):
221
+ # The angles must stay exact at large positions. Two cases would otherwise lose precision:
222
+ # (1) under torch.autocast(bfloat16) the matmul runs in bf16 despite the `.float()` casts; (2) after
223
+ # model.to(torch.bfloat16) / model.bfloat16() the `inv_freq` buffer itself is cast to bf16. We therefore
224
+ # disable autocast and recompute `inv_freq` in fp32 (tiny vector) instead of reading the buffer, so neither
225
+ # case can round the positions off. Matches standard HF rotaries, which also disable autocast here.
226
+ with torch.autocast(device_type=x.device.type, enabled=False):
227
+ inv_freq, _ = self.compute_default_rope_parameters(self.config, x.device)
228
+ inv_freq_expanded = inv_freq[None, :, None].float().expand(position_ids.shape[0], -1, 1)
229
+ position_ids_expanded = position_ids[:, None, :].float()
230
+ freqs = (inv_freq_expanded @ position_ids_expanded).transpose(1, 2)
231
+ emb = torch.cat((freqs, freqs), dim=-1)
232
+ cos = emb.cos() * self.attention_scaling
233
+ sin = emb.sin() * self.attention_scaling
234
 
235
  return cos, sin
236
 
 
323
  cos, sin = position_embeddings
324
  q, k = self.apply_rope(q, k, cos, sin, unsqueeze_dim=2)
325
 
326
+ # Opt-in SDPA path for hosts without the CUDA varlen/flash kernel (CPU, older GPUs) and for an
327
+ # fp32-capable attention. Enabled only with attn_implementation="sdpa"; the default path below is unchanged.
328
+ if getattr(self.config, "_attn_implementation", "flash_attention_2") == "sdpa":
329
+ return self._sdpa_forward(q, k, v, attention_mask, past_key_values, batch_size, sequence_length)
330
+
331
  if past_key_values is not None:
332
  k, v = k.transpose(1, 2), v.transpose(1, 2)
333
  k, v = past_key_values.update(k, v, self.layer_idx)
 
361
  attn_output = attn_output.view(batch_size, sequence_length, self.config.num_attention_heads * self.head_dim)
362
  return self.o_proj(attn_output)
363
 
364
+ def _sdpa_forward(self, q, k, v, attention_mask, past_key_values, batch_size, sequence_length):
365
+ """SDPA fallback matching the varlen path: grouped-query attention with this layer's sliding window and the
366
+ padding mask, built as an additive bias so it runs on any backend (including CPU/fp32)."""
367
+ if past_key_values is not None:
368
+ k, v = k.transpose(1, 2), v.transpose(1, 2)
369
+ k, v = past_key_values.update(k, v, self.layer_idx)
370
+ k, v = k.transpose(1, 2), v.transpose(1, 2)
371
+
372
+ q = q.transpose(1, 2) # [b, n_heads, q_len, d]
373
+ k = k.transpose(1, 2) # [b, n_kv, k_len, d]
374
+ v = v.transpose(1, 2)
375
+ q_len, k_len = q.shape[2], k.shape[2]
376
+
377
+ # Absolute positions so decode (q_len=1, k_len>q_len) and prefill both work.
378
+ qpos = torch.arange(k_len - q_len, k_len, device=q.device)[:, None]
379
+ kpos = torch.arange(k_len, device=q.device)[None, :]
380
+ left, right = self.window_size # [-1, -1] = full attention; [w, 0] = causal sliding window
381
+ allowed = torch.ones(q_len, k_len, dtype=torch.bool, device=q.device)
382
+ if right >= 0:
383
+ allowed &= (kpos - qpos) <= right
384
+ if left >= 0:
385
+ allowed &= (qpos - kpos) <= left
386
+ mask = allowed[None, None] # [1, 1, q_len, k_len]
387
+ if attention_mask is not None:
388
+ mask = mask & attention_mask[:, None, None, :].to(torch.bool)
389
+ bias = torch.zeros(mask.shape, dtype=q.dtype, device=q.device).masked_fill(~mask, float("-inf"))
390
+ # A fully-masked query row (e.g. a padding position) would make softmax produce NaN; neutralize it — those
391
+ # positions are discarded downstream.
392
+ bias = bias.masked_fill(torch.isneginf(bias).all(dim=-1, keepdim=True), 0.0)
393
+
394
+ attn_output = torch.nn.functional.scaled_dot_product_attention(
395
+ q, k, v, attn_mask=bias, scale=self.scaling, enable_gqa=self.enable_gqa
396
+ )
397
+ attn_output = attn_output.transpose(1, 2).reshape(
398
+ batch_size, sequence_length, self.config.num_attention_heads * self.head_dim
399
+ )
400
+ return self.o_proj(attn_output)
401
+
402
 
403
  class DomynEdgeDecoderLayer(torch.nn.Module):
404
  def __init__(self, config: DomynEdgeConfig, layer_idx: int):
 
442
 
443
  config: DomynEdgeConfig
444
  base_model_prefix = "model"
445
+ supports_gradient_checkpointing = True
446
  _supports_flash_attn = True
447
+ _supports_sdpa = True
448
  _supports_flex_attn = False
449
 
450
  _can_compile_fullgraph = True
 
469
  )
470
  self.norm = torch.nn.RMSNorm(config.hidden_size, eps=config.rms_norm_eps)
471
  self.rotary_emb = DomynEdgeRotaryEmbedding(config=config)
472
+ self.gradient_checkpointing = False
473
 
474
  # Initialize weights and apply final processing
475
  self.post_init()
 
514
  )
515
 
516
  for decoder_layer in self.layers:
517
+ if self.gradient_checkpointing and self.training:
518
+ # Layers are stateless, so activations can be recomputed in backward (no KV cache during training).
519
+ hidden_states = self._gradient_checkpointing_func(
520
+ decoder_layer.__call__, hidden_states, attention_mask, past_key_values, position_embeddings
521
+ )
522
+ else:
523
+ hidden_states = decoder_layer(
524
+ hidden_states,
525
+ attention_mask=attention_mask,
526
+ position_embeddings=position_embeddings,
527
+ past_key_values=past_key_values,
528
+ **kwargs,
529
+ )
530
 
531
  hidden_states = self.norm(hidden_states)
532
  return transformers.modeling_outputs.BaseModelOutputWithPast(