KitsuVp commited on
Commit
42a0881
·
verified ·
1 Parent(s): 3a4be88

Update modeling_neollm.py

Browse files
Files changed (1) hide show
  1. modeling_neollm.py +275 -167
modeling_neollm.py CHANGED
@@ -2512,91 +2512,94 @@ def _apply_repo_rope(
2512
 
2513
  class RepoGrapePositioning(nn.Module):
2514
  r"""
2515
- Minimal REPO-GRAPE-M positional operator with linear-blend coordinates.
2516
 
2517
- The implementation deliberately has no positional-mode flag. Whenever
2518
- REPO-GRAPE is active it uses exactly one coordinate map, the
2519
- ``linear-blend-repo`` form from the accompanying derivation:
2520
 
2521
  z_i^(h) = f_phi^(h)(h_i),
2522
  u_i^(h) = (1 - alpha_h) i + alpha_h z_i^(h)
2523
- = i + alpha_h (z_i^(h) - i).
2524
 
2525
- ``i`` is the actual runtime ``position_ids`` value, not an independently
2526
- generated global arange. This matters for padding and packed/reset
2527
- position streams. ``alpha_h`` is one learned scalar per query head. It
2528
- is initialized to 1, so a newly constructed model starts exactly at the
2529
- previous REPO coordinate u=z and can learn a linear anchor without an
2530
- abrupt initialization change.
2531
 
2532
- The parameter is intentionally stored directly rather than through a
2533
- sigmoid/clamp. Therefore alpha_h is initialized inside [0,1] but is not
2534
- constrained to stay there. This avoids an additional nonlinear operation,
2535
- preserves the exact alpha_h=1 initialization, and lets the optimizer learn
2536
- interpolation or extrapolation. Constraining alpha would be a different
2537
- parameterization and is not silently introduced here.
2538
 
2539
- Backward is standard autograd. If ubar_i^(h)=dL/du_i^(h), then for this
2540
- active linear-blend coordinate
2541
 
2542
- dL/dz_i^(h) = alpha_h * ubar_i^(h),
2543
- dL/dalpha_h = sum_i ubar_i^(h) * (z_i^(h) - i).
 
 
2544
 
2545
- For comparison only, the distinct ``linear-plus-repo`` residual map
 
2546
 
2547
- u_i^(h) = i + alpha_h z_i^(h)
2548
 
2549
- would instead give dL/dalpha_h = sum_i ubar_i^(h) z_i^(h). That formula
2550
- is documented to avoid confusing the two maps, but it is not another
2551
- executable mode and no flag is added for it.
 
2552
 
2553
- Storage/lifetime contract:
2554
- - ``position_blend_alpha`` is an FP32 ``nn.Parameter`` of shape [H_q].
2555
- It is optimized normally and is saved/restored in ``state_dict``.
2556
- - ``freq_log_scale`` remains an FP32 saved parameter of shape
2557
- [H_q, head_dim/2].
2558
- - ``position_ids`` is runtime metadata and is never copied into a buffer or
2559
- checkpoint.
2560
- - ``u`` and the optional IHA-expanded ``u_{i,p}`` are transient autograd
2561
- intermediates only. The IHA tensor replaces the previous ``P*z+p``
2562
- intermediate, so this change does not add another expanded coordinate
2563
- tensor.
2564
 
2565
- IHA is applied after the coordinate map. For P pseudo-slots,
 
 
2566
 
2567
- u_{i,p}^(h) = P u_i^(h) + p, p in {0,...,P-1},
2568
 
2569
- and GRAPE sees the phase
 
2570
 
2571
- phi_{b,h,i,p,r} = u_{b,h,i,p} * theta_{h,r},
2572
- theta_{h,r} = inv_freq_r * exp(s_{h,r}).
2573
 
2574
- With one key per query head, the multiplicative action obeys the exact
2575
- relative law
2576
 
2577
- (G_h(u_i)q_i)^T(G_h(u_j)k_j)
2578
- = q_i^T G_h(u_j-u_i) k_j.
 
 
 
2579
 
2580
- GQA introduces a separate geometric restriction: several query heads
2581
- share one key, although their phases phi_{b,h,i,r} can differ. A shared
2582
- key cannot realize every head-specific rotation simultaneously. For a
2583
- KV group H_m and rotary plane r, this implementation projects the requested
2584
- rotations onto the single optimal shared SO(2) rotation:
2585
 
2586
- Z_{b,m,i,r} = (1/g) * sum_{h in H_m} exp(i*phi_{b,h,i,r})
2587
- = C_{b,m,i,r} + i*S_{b,m,i,r},
 
 
2588
 
2589
- psi*_{b,m,i,r} = arg Z_{b,m,i,r}.
 
2590
 
2591
- Equivalently, when |Z| > 0,
2592
 
2593
- cos(psi*) = C/|Z|,
2594
- sin(psi*) = S/|Z|.
 
 
 
 
 
 
 
 
 
2595
 
2596
- If the circular resultant is numerically zero, the objective does not
2597
- identify a preferred shared direction; the code uses the identity rotation
2598
- as a deterministic finite convention. Under IHA, the projection is
2599
- computed independently inside every pseudo-slot.
2600
  """
2601
 
2602
  def __init__(
@@ -2628,8 +2631,13 @@ class RepoGrapePositioning(nn.Module):
2628
  requires_grad=True,
2629
  )
2630
 
2631
- # Learned log-scale over the base RoPE/GRAPE frequencies, per query
2632
- # head and rotary plane: theta_{h,r}=theta^0_r*exp(s_{h,r}).
 
 
 
 
 
2633
  self.freq_log_scale = nn.Parameter(
2634
  torch.zeros(
2635
  self.num_attention_heads, self.max_rot_half, dtype=torch.float32
@@ -2637,6 +2645,34 @@ class RepoGrapePositioning(nn.Module):
2637
  requires_grad=True,
2638
  )
2639
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
2640
  def _load_from_state_dict(
2641
  self,
2642
  state_dict,
@@ -2647,20 +2683,27 @@ class RepoGrapePositioning(nn.Module):
2647
  unexpected_keys,
2648
  error_msgs,
2649
  ):
2650
- # Backward-compatible checkpoint migration. Older REPO-GRAPE
2651
- # checkpoints do not contain alpha_h because they used u=z. Injecting
2652
- # alpha_h=1 reproduces that old forward exactly at load time, while every
2653
- # checkpoint saved by this version stores the learned parameter normally.
 
 
 
2654
  alpha_key = prefix + "position_blend_alpha"
2655
- local_state_dict = state_dict
 
 
 
2656
  if alpha_key not in state_dict:
2657
- # Use a local shallow copy so loading does not mutate the caller's
2658
- # checkpoint mapping. CPU FP32 also works when the module was
2659
- # constructed on meta before Hugging Face materializes parameters.
2660
- local_state_dict = state_dict.copy()
2661
  local_state_dict[alpha_key] = torch.ones(
2662
  self.num_attention_heads, dtype=torch.float32
2663
  )
 
 
 
 
 
2664
  super()._load_from_state_dict(
2665
  local_state_dict,
2666
  prefix,
@@ -2672,37 +2715,76 @@ class RepoGrapePositioning(nn.Module):
2672
  )
2673
 
2674
  def _apply(self, fn):
2675
- # Keep the tiny positional parameters in FP32 under model.to(dtype).
2676
- # Both are checkpointed nn.Parameters; only runtime coordinates are
2677
- # transient.
2678
  super()._apply(fn)
2679
- for p in (self.position_blend_alpha, self.freq_log_scale):
 
 
 
 
2680
  p.data = p.data.float()
2681
  if p.grad is not None:
2682
  p.grad.data = p.grad.data.float()
2683
  return self
2684
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
2685
  def transform_positions(
2686
  self,
2687
  z: torch.Tensor,
2688
  position_ids: Optional[torch.Tensor],
2689
- ) -> torch.Tensor:
2690
- r"""Build the active linear-blend REPO-GRAPE coordinate.
 
 
2691
 
2692
- The only executable map is
2693
 
2694
  u_i^(h) = (1-alpha_h) i + alpha_h z_i^(h)
2695
  = i + alpha_h (z_i^(h) - i).
2696
 
2697
- The residualized second form is used because it needs only one
2698
- subtraction, one multiplication and one addition. ``position_ids`` is
2699
- broadcast as [B,1,S], so it is not materialized once per head.
2700
 
2701
- ``u`` is evaluated in FP32 because the next operation is the FP32 phase
2702
- computation. It is not registered as a buffer and is not saved in the
2703
- checkpoint. Standard autograd gives
2704
 
2705
- dL/dalpha_h = sum_{b,i} ubar_{b,h,i} (z_{b,h,i} - i).
 
2706
  """
2707
  B, H, S = z.shape
2708
  if H != self.num_attention_heads:
@@ -2710,35 +2792,13 @@ class RepoGrapePositioning(nn.Module):
2710
  f"REPO-GRAPE expected z heads={self.num_attention_heads}, got {H}."
2711
  )
2712
 
2713
- if position_ids is None:
2714
- # Defensive fallback for direct attention calls. NeoLLMModel normally
2715
- # constructs position_ids before entering the decoder layers.
2716
- pid = torch.arange(S, device=z.device, dtype=torch.float32).view(1, S)
2717
- else:
2718
- pid = position_ids
2719
- if pid.dim() == 1:
2720
- pid = pid.unsqueeze(0)
2721
- elif pid.dim() > 2:
2722
- # Match the existing IHA RoPE convention: trainers may attach
2723
- # metadata in the final axis; column 0 is the sequential index.
2724
- pid = pid[..., 0]
2725
- if pid.dim() != 2 or pid.shape[-1] != S:
2726
- raise ValueError(
2727
- "REPO-GRAPE position_ids must have shape [S] or [B,S] "
2728
- f"matching z[...,S={S}], got {tuple(pid.shape)}."
2729
- )
2730
- if pid.shape[0] == 1 and B != 1:
2731
- pid = pid.expand(B, -1)
2732
- elif pid.shape[0] != B:
2733
- raise ValueError(
2734
- f"REPO-GRAPE position_ids batch={pid.shape[0]} does not match z batch={B}."
2735
- )
2736
- pid = pid.to(device=z.device, dtype=torch.float32)
2737
-
2738
- # Views/broadcasting do not allocate [B,H,S] copies of pid or alpha.
2739
  i = pid.unsqueeze(1)
2740
  alpha = self.position_blend_alpha.view(1, H, 1)
2741
- return i + alpha * (z.float() - i)
 
 
 
2742
 
2743
  def _query_freq(
2744
  self,
@@ -2756,11 +2816,20 @@ class RepoGrapePositioning(nn.Module):
2756
  )
2757
 
2758
  base = inv_freq.float().view(1, rot_half)
2759
- log_scale = self.freq_log_scale[:, :rot_half].float()
2760
- freq = base * torch.exp(log_scale)
2761
- freq = freq.to(device=inv_freq.device)
2762
 
2763
- return freq
 
 
 
 
 
 
 
 
 
 
 
2764
 
2765
  def apply_multiplicative(
2766
  self,
@@ -2769,41 +2838,33 @@ class RepoGrapePositioning(nn.Module):
2769
  u: torch.Tensor,
2770
  inv_freq: torch.Tensor,
2771
  attention_scaling: float,
 
 
2772
  ) -> Tuple[torch.Tensor, torch.Tensor]:
2773
- r"""
2774
- Apply REPO-GRAPE-M rotation to Q/K using final coordinates ``u``.
2775
 
2776
- ``u`` is already the linear-blend coordinate produced by
2777
- ``transform_positions`` (and, for IHA, already expanded as P*u+p).
2778
  Query heads keep their individual phases
2779
 
2780
- phi_{b,h,i,r} = u_{b,h,i} * theta_{h,r}.
2781
 
2782
- For GQA, each KV head m serves a group H_m of query heads. Its shared
2783
- rotation is the circular projection
2784
 
2785
- C = mean_{h in H_m} cos(phi_h),
2786
- S = mean_{h in H_m} sin(phi_h),
2787
- (cos psi*, sin psi*) = (C, S) / sqrt(C^2 + S^2).
2788
 
2789
- This is equivalent to psi*=arg(mean_h exp(i*phi_h)) and minimizes the
2790
- total squared discrepancy between the head-specific rotated keys and
2791
- one shared rotated key. The reduction is performed independently for
2792
- every batch element, current sequence index, rotary plane and KV group.
2793
- Because seq-expand IHA stores each pseudo-slot at a different sequence
2794
- index, its pseudo-slots remain independent automatically. No extra
2795
- mode or configuration flag is introduced.
2796
 
2797
- Args:
2798
- q: [B, H_q_eff, S, head_dim]
2799
- k: [B, H_k_eff, S, head_dim]
2800
- u: [B, H_q_base, S] final REPO-GRAPE coordinates
2801
- inv_freq: [rotary_dim/2]
2802
- attention_scaling: scaling from NeoLLMRotaryEmbedding
2803
 
2804
- Returns:
2805
- Rotated ``(q, k)`` tensors with the same shapes and dtypes as the
2806
- inputs. Channels beyond ``rotary_dim`` pass through unchanged.
 
 
 
 
2807
  """
2808
  B, H_repo, S = u.shape
2809
  H_q_eff = q.shape[1]
@@ -2849,21 +2910,46 @@ class RepoGrapePositioning(nn.Module):
2849
  cos_q_half = phase_q.cos()
2850
  sin_q_half = phase_q.sin()
2851
 
2852
- # GQA circular projection. Restore the logical head factorization
2853
- # (KV group, query-within-group, retained head-expansion slot) and
2854
- # average only over query-within-group. In the current seq-expand IHA
2855
- # path pseudo-slots live on the sequence axis, which is never reduced.
2856
- # Averaging cos/sin and normalizing the resultant is exactly equivalent
2857
- # to arg(mean(exp(i*phase))).
2858
  cos_q_grouped = cos_q_half.reshape(
2859
  B, H_k_base, q_per_k_base, P, S, rot_half
2860
  )
2861
  sin_q_grouped = sin_q_half.reshape(
2862
  B, H_k_base, q_per_k_base, P, S, rot_half
2863
  )
2864
- cos_k_half = cos_q_grouped.mean(dim=2)
2865
- sin_k_half = sin_q_grouped.mean(dim=2)
2866
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
2867
  resultant_sq = cos_k_half.square() + sin_k_half.square()
2868
  well_defined = resultant_sq > torch.finfo(resultant_sq.dtype).eps
2869
  safe_resultant_sq = torch.where(
@@ -2871,18 +2957,15 @@ class RepoGrapePositioning(nn.Module):
2871
  )
2872
  inv_resultant = torch.rsqrt(safe_resultant_sq)
2873
 
2874
- # If the resultant vanishes, every shared angle has the same objective
2875
- # value. Select identity (cos=1, sin=0) to keep forward/backward finite
2876
- # and deterministic; this is a numerical convention, not a new mode.
2877
  cos_k_half = torch.where(
2878
  well_defined,
2879
  cos_k_half * inv_resultant,
2880
- torch.ones_like(cos_k_half),
2881
  )
2882
  sin_k_half = torch.where(
2883
  well_defined,
2884
  sin_k_half * inv_resultant,
2885
- torch.zeros_like(sin_k_half),
2886
  )
2887
  cos_k_half = cos_k_half.reshape(B, H_k_eff, S, rot_half)
2888
  sin_k_half = sin_k_half.reshape(B, H_k_eff, S, rot_half)
@@ -3491,10 +3574,12 @@ class NeoLLMAttention(nn.Module):
3491
  num_key_value_heads=config.num_key_value_heads,
3492
  head_dim=self.head_dim,
3493
  )
3494
- # Availability is fixed when the module is constructed, so Dynamo
3495
- # specializes one stable route. This is deliberately not a new
3496
- # configuration flag: use_repo_grape still owns the feature, while
3497
- # unsupported inputs retain the exact PyTorch implementation.
 
 
3498
  self.use_repo_grape_triton = False
3499
  else:
3500
  self.repo_grape = None
@@ -4649,11 +4734,18 @@ class NeoLLMAttention(nn.Module):
4649
  # Pure REPO (without GRAPE) keeps the original paper behavior and
4650
  # therefore continues to use z directly.
4651
  if not _repo_grape_kernel_used:
4652
- repo_coords = (
4653
- self.repo_grape.transform_positions(z, position_ids)
4654
- if self.use_repo_grape
4655
- else z
4656
- )
 
 
 
 
 
 
 
4657
 
4658
  if not _repo_grape_kernel_used and _iha_seq_expand:
4659
  # IHA is applied *after* the coordinate map. Thus REPO-GRAPE
@@ -4671,8 +4763,19 @@ class NeoLLMAttention(nn.Module):
4671
  ).reshape(repo_coords.shape[0], repo_coords.shape[1], _S_orig * P)
4672
 
4673
  if self.use_repo_grape:
 
 
 
 
 
 
4674
  q, k = self.repo_grape.apply_multiplicative(
4675
- q, k, coords_expanded, inv_freq, attn_scaling
 
 
 
 
 
4676
  )
4677
  else:
4678
  q, k = _apply_repo_rope(
@@ -4683,7 +4786,12 @@ class NeoLLMAttention(nn.Module):
4683
  # the original REPO-RoPE path.
4684
  if self.use_repo_grape:
4685
  q, k = self.repo_grape.apply_multiplicative(
4686
- q, k, repo_coords, inv_freq, attn_scaling
 
 
 
 
 
4687
  )
4688
  else:
4689
  q, k = _apply_repo_rope(
 
2512
 
2513
  class RepoGrapePositioning(nn.Module):
2514
  r"""
2515
+ REPO-GRAPE-M positional operator with a well-conditioned GQA projection.
2516
 
2517
+ Revision implemented here
2518
+ -------------------------
2519
+ This version keeps the contextual coordinate map unchanged,
2520
 
2521
  z_i^(h) = f_phi^(h)(h_i),
2522
  u_i^(h) = (1 - alpha_h) i + alpha_h z_i^(h)
2523
+ = i + alpha_h (z_i^(h) - i),
2524
 
2525
+ and applies exactly three corrections identified by the independent
2526
+ REPO-GRAPE analysis:
 
 
 
 
2527
 
2528
+ 1. Spectral gauge fixing. The raw per-head log spectrum s_{h,r} is centered
2529
+ across the active rotary planes before exponentiation,
 
 
 
 
2530
 
2531
+ s~_{h,r} = s_{h,r} - mean_r s_{h,r},
2532
+ theta_{h,r} = inv_freq_r * exp(s~_{h,r}).
2533
 
2534
+ Hence sum_r s~_{h,r}=0 and the geometric mean of exp(s~) is one. The
2535
+ learned spectrum can reshape relative frequencies but cannot absorb a
2536
+ uniform scale that is functionally confounded with u. Autograd applies
2537
+ the matching zero-mean projection to the raw-s gradient.
2538
 
2539
+ 2. Learned GQA head importance. If H_m is the set of query heads sharing
2540
+ KV head m, one FP32 logit a_h per query head defines
2541
 
2542
+ w_h = softmax_{h in H_m}(a_h), sum_{h in H_m} w_h = 1.
2543
 
2544
+ a_h starts at zero, so the weighted circular reduction starts exactly at
2545
+ the previous uniform weighting. These are structural per-head scalars,
2546
+ not a token-dependent router, so there is no activation-sized routing
2547
+ tensor.
2548
 
2549
+ 3. Weak circular prior toward the canonical RoPE phase. Let
 
 
 
 
 
 
 
 
 
 
2550
 
2551
+ phi_{h,i,r} = u_i^(h) * theta_{h,r},
2552
+ psi0_{i,r} = i * inv_freq_r,
2553
+ Z_{m,i,r} = sum_{h in H_m} w_h exp(i*phi_{h,i,r}).
2554
 
2555
+ The shared key rotation is obtained from
2556
 
2557
+ Z_lambda = Z + lambda * exp(i*psi0),
2558
+ exp(i*psi*) = Z_lambda / |Z_lambda|.
2559
 
2560
+ This is the exact minimizer of the regularized chordal objective
 
2561
 
2562
+ sum_h w_h ||R(phi_h)k - R(psi)k||_2^2
2563
+ + lambda ||R(psi0)k - R(psi)k||_2^2.
2564
 
2565
+ When the query heads are coherent, |Z| >> lambda and the prior is almost
2566
+ invisible. When their resultant is small, the canonical RoPE direction
2567
+ supplies a weak reference. Exact cancellation Z_lambda=0 remains
2568
+ mathematically non-identifiable; at that machine-zero event the code uses
2569
+ the canonical RoPE direction as a deterministic finite representative.
2570
 
2571
+ The default prior strength is 1e-2 relative to a weighted resultant whose
2572
+ maximum magnitude is one. It is read from the optional config attribute
2573
+ ``repo_grape_circular_prior_strength`` and written back to ``config`` so the
2574
+ effective value is serialized. Setting it to zero is the clean prior
2575
+ ablation; centered spectrum and learned GQA weights remain active.
2576
 
2577
+ Runtime positions and IHA
2578
+ -------------------------
2579
+ ``i`` is the actual runtime ``position_ids`` value. With IHA sequence
2580
+ expansion, both the contextual coordinate and its canonical reference use
2581
 
2582
+ u_{i,p} = P*u_i + p,
2583
+ i_{i,p} = P*i + p,
2584
 
2585
+ so the prior points to the RoPE phase of the same pseudo-slot being rotated.
2586
 
2587
+ Storage / checkpoint contract
2588
+ -----------------------------
2589
+ - ``position_blend_alpha``: FP32 Parameter [H_q], unchanged.
2590
+ - ``freq_log_scale``: FP32 Parameter [H_q, head_dim/2], unchanged in shape;
2591
+ its effective value is centered before use.
2592
+ - ``gqa_head_weight_logits``: new FP32 Parameter [H_q], initialized to zero;
2593
+ old checkpoints receive this zero vector during strict loading.
2594
+ - ``repo_grape_circular_prior_strength``: scalar config hyperparameter, not a
2595
+ learned Parameter and not an activation-sized buffer.
2596
+ - ``u``, centered s~, phases, resultants and reference phases are transient
2597
+ autograd intermediates only.
2598
 
2599
+ The current optional CCE/Triton REPO-GRAPE kernel implements the preceding
2600
+ legacy uncentered/unweighted/unregularized contract. It therefore remains
2601
+ disabled in ``NeoLLMAttention`` until that kernel is updated in lockstep;
2602
+ silently dispatching it would change the model mathematics.
2603
  """
2604
 
2605
  def __init__(
 
2631
  requires_grad=True,
2632
  )
2633
 
2634
+ # Raw learned log-spectrum. The stored parameter keeps the historical
2635
+ # shape/state-dict key, but _query_freq fixes the common-scale gauge via
2636
+ #
2637
+ # s_tilde[h,r] = s[h,r] - mean_r(s[h,r]).
2638
+ #
2639
+ # This preserves relative per-plane spectral reshaping while removing the
2640
+ # uniform log-frequency mode that can trade against u in phi=u*theta.
2641
  self.freq_log_scale = nn.Parameter(
2642
  torch.zeros(
2643
  self.num_attention_heads, self.max_rot_half, dtype=torch.float32
 
2645
  requires_grad=True,
2646
  )
2647
 
2648
+ # Structural importance logits for the query heads inside each GQA group.
2649
+ # Softmax is group-local. Zero initialization gives exactly uniform
2650
+ # weights, so the new degree of freedom begins from the old reduction.
2651
+ self.gqa_head_weight_logits = nn.Parameter(
2652
+ torch.zeros(self.num_attention_heads, dtype=torch.float32),
2653
+ requires_grad=True,
2654
+ )
2655
+
2656
+ # Weak canonical circular prior. The weighted head resultant has |Z|<=1,
2657
+ # so lambda=1e-2 means a 1% reference vector on the same scale. Keep this
2658
+ # as a config hyperparameter rather than a Parameter to avoid optimizer
2659
+ # state while preserving run reproducibility.
2660
+ self.circular_prior_strength = float(
2661
+ getattr(config, "repo_grape_circular_prior_strength", 1.0e-2)
2662
+ )
2663
+ if (
2664
+ not math.isfinite(self.circular_prior_strength)
2665
+ or self.circular_prior_strength < 0.0
2666
+ ):
2667
+ raise ValueError(
2668
+ "repo_grape_circular_prior_strength must be a finite non-negative scalar."
2669
+ )
2670
+ setattr(
2671
+ self.config,
2672
+ "repo_grape_circular_prior_strength",
2673
+ self.circular_prior_strength,
2674
+ )
2675
+
2676
  def _load_from_state_dict(
2677
  self,
2678
  state_dict,
 
2683
  unexpected_keys,
2684
  error_msgs,
2685
  ):
2686
+ # Backward-compatible checkpoint migration.
2687
+ #
2688
+ # Very old checkpoints may lack alpha_h, and every pre-correction
2689
+ # checkpoint lacks the new GQA importance logits. alpha_h=1 preserves
2690
+ # the historical u=z initialization; a_h=0 restores uniform head weights.
2691
+ # Centering freq_log_scale intentionally changes only its common per-head
2692
+ # mode and therefore needs no new state tensor.
2693
  alpha_key = prefix + "position_blend_alpha"
2694
+ weight_key = prefix + "gqa_head_weight_logits"
2695
+ need_copy = alpha_key not in state_dict or weight_key not in state_dict
2696
+ local_state_dict = state_dict.copy() if need_copy else state_dict
2697
+
2698
  if alpha_key not in state_dict:
 
 
 
 
2699
  local_state_dict[alpha_key] = torch.ones(
2700
  self.num_attention_heads, dtype=torch.float32
2701
  )
2702
+ if weight_key not in state_dict:
2703
+ local_state_dict[weight_key] = torch.zeros(
2704
+ self.num_attention_heads, dtype=torch.float32
2705
+ )
2706
+
2707
  super()._load_from_state_dict(
2708
  local_state_dict,
2709
  prefix,
 
2715
  )
2716
 
2717
  def _apply(self, fn):
2718
+ # Keep the angular control parameters in FP32 under model.to(dtype).
2719
+ # Phases/resultants are evaluated in FP32, so quantizing these tiny state
2720
+ # tensors would save negligible memory while perturbing the geometry.
2721
  super()._apply(fn)
2722
+ for p in (
2723
+ self.position_blend_alpha,
2724
+ self.freq_log_scale,
2725
+ self.gqa_head_weight_logits,
2726
+ ):
2727
  p.data = p.data.float()
2728
  if p.grad is not None:
2729
  p.grad.data = p.grad.data.float()
2730
  return self
2731
 
2732
+ def _resolve_runtime_positions(
2733
+ self,
2734
+ batch_size: int,
2735
+ seq_len: int,
2736
+ device: torch.device,
2737
+ position_ids: Optional[torch.Tensor],
2738
+ ) -> torch.Tensor:
2739
+ """Resolve canonical runtime positions as FP32 ``[B,S]`` metadata."""
2740
+ if position_ids is None:
2741
+ return torch.arange(
2742
+ seq_len, device=device, dtype=torch.float32
2743
+ ).view(1, seq_len).expand(batch_size, -1)
2744
+
2745
+ pid = position_ids
2746
+ if pid.dim() == 1:
2747
+ pid = pid.unsqueeze(0)
2748
+ elif pid.dim() > 2:
2749
+ # Match the existing IHA convention: metadata column 0 is the
2750
+ # sequential position used by RoPE/REPO.
2751
+ pid = pid[..., 0]
2752
+ if pid.dim() != 2 or pid.shape[-1] != seq_len:
2753
+ raise ValueError(
2754
+ "REPO-GRAPE position_ids must have shape [S] or [B,S] "
2755
+ f"with S={seq_len}, got {tuple(pid.shape)}."
2756
+ )
2757
+ if pid.shape[0] == 1 and batch_size != 1:
2758
+ pid = pid.expand(batch_size, -1)
2759
+ elif pid.shape[0] != batch_size:
2760
+ raise ValueError(
2761
+ f"REPO-GRAPE position_ids batch={pid.shape[0]} does not match "
2762
+ f"batch={batch_size}."
2763
+ )
2764
+ return pid.to(device=device, dtype=torch.float32)
2765
+
2766
  def transform_positions(
2767
  self,
2768
  z: torch.Tensor,
2769
  position_ids: Optional[torch.Tensor],
2770
+ *,
2771
+ return_reference: bool = False,
2772
+ ) -> Union[torch.Tensor, Tuple[torch.Tensor, torch.Tensor]]:
2773
+ r"""Build the linear-blend coordinate and optionally return its anchor.
2774
 
2775
+ Active map:
2776
 
2777
  u_i^(h) = (1-alpha_h) i + alpha_h z_i^(h)
2778
  = i + alpha_h (z_i^(h) - i).
2779
 
2780
+ ``return_reference=True`` additionally returns canonical FP32 positions
2781
+ ``i`` as [B,S]. They are reused to construct the RoPE prior phase after
2782
+ any IHA pseudo-slot expansion.
2783
 
2784
+ Standard autograd gives
 
 
2785
 
2786
+ dL/dz_i^(h) = alpha_h * ubar_i^(h),
2787
+ dL/dalpha_h = sum_{b,i} ubar_{b,h,i}(z_{b,h,i}-i).
2788
  """
2789
  B, H, S = z.shape
2790
  if H != self.num_attention_heads:
 
2792
  f"REPO-GRAPE expected z heads={self.num_attention_heads}, got {H}."
2793
  )
2794
 
2795
+ pid = self._resolve_runtime_positions(B, S, z.device, position_ids)
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
2796
  i = pid.unsqueeze(1)
2797
  alpha = self.position_blend_alpha.view(1, H, 1)
2798
+ u = i + alpha * (z.float() - i)
2799
+ if return_reference:
2800
+ return u, pid
2801
+ return u
2802
 
2803
  def _query_freq(
2804
  self,
 
2816
  )
2817
 
2818
  base = inv_freq.float().view(1, rot_half)
2819
+ raw_log_scale = self.freq_log_scale[:, :rot_half].float()
 
 
2820
 
2821
+ # Spectral gauge fixing on the active rotary planes:
2822
+ #
2823
+ # s_tilde = s - mean_r(s),
2824
+ # theta = theta0 * exp(s_tilde).
2825
+ #
2826
+ # The centering matrix is an orthogonal projection onto the zero-mean
2827
+ # subspace, so autograd also removes the common-mode gradient.
2828
+ centered_log_scale = raw_log_scale - raw_log_scale.mean(
2829
+ dim=-1, keepdim=True
2830
+ )
2831
+ freq = base * torch.exp(centered_log_scale)
2832
+ return freq.to(device=inv_freq.device)
2833
 
2834
  def apply_multiplicative(
2835
  self,
 
2838
  u: torch.Tensor,
2839
  inv_freq: torch.Tensor,
2840
  attention_scaling: float,
2841
+ *,
2842
+ reference_positions: Optional[torch.Tensor] = None,
2843
  ) -> Tuple[torch.Tensor, torch.Tensor]:
2844
+ r"""Apply corrected REPO-GRAPE-M rotations to Q/K.
 
2845
 
 
 
2846
  Query heads keep their individual phases
2847
 
2848
+ phi_{h,i,r} = u_i^(h) * theta0_r * exp(s~_{h,r}).
2849
 
2850
+ For each GQA group H_m, structural logits a_h define
 
2851
 
2852
+ w_h = softmax_{h in H_m}(a_h),
2853
+ Z = sum_h w_h exp(i*phi_h).
 
2854
 
2855
+ The shared key uses the weakly regularized projection
 
 
 
 
 
 
2856
 
2857
+ psi0 = i * theta0,
2858
+ Z_lambda = Z + lambda exp(i*psi0),
2859
+ exp(i*psi*) = Z_lambda / |Z_lambda|.
 
 
 
2860
 
2861
+ This is still a closed-form SO(2) projection: it minimizes the weighted
2862
+ chordal key distortion plus lambda times the same distortion to the
2863
+ canonical RoPE rotation. The prior acts only on the shared GQA key;
2864
+ query phases remain fully REPO-GRAPE-specific.
2865
+
2866
+ ``reference_positions`` must describe the same final sequence axis as
2867
+ ``u``. The attention caller passes P*i+p when IHA expands pseudo-slots.
2868
  """
2869
  B, H_repo, S = u.shape
2870
  H_q_eff = q.shape[1]
 
2910
  cos_q_half = phase_q.cos()
2911
  sin_q_half = phase_q.sin()
2912
 
2913
+ # Corrected GQA circular projection. Restore the logical base-head
2914
+ # grouping and reduce only over query-within-group. Structural weights
2915
+ # are token-independent and initialized uniformly.
 
 
 
2916
  cos_q_grouped = cos_q_half.reshape(
2917
  B, H_k_base, q_per_k_base, P, S, rot_half
2918
  )
2919
  sin_q_grouped = sin_q_half.reshape(
2920
  B, H_k_base, q_per_k_base, P, S, rot_half
2921
  )
 
 
2922
 
2923
+ head_logits = self.gqa_head_weight_logits.reshape(
2924
+ H_k_base, q_per_k_base
2925
+ ).float()
2926
+ head_weights = torch.softmax(head_logits, dim=-1).view(
2927
+ 1, H_k_base, q_per_k_base, 1, 1, 1
2928
+ )
2929
+ cos_k_half = (cos_q_grouped * head_weights).sum(dim=2)
2930
+ sin_k_half = (sin_q_grouped * head_weights).sum(dim=2)
2931
+
2932
+ # Canonical RoPE prior for the same runtime/pseudo-slot positions.
2933
+ ref = self._resolve_runtime_positions(B, S, q.device, reference_positions)
2934
+ base_phase_half = ref.view(B, 1, 1, S, 1) * inv_freq.float().to(
2935
+ device=q.device
2936
+ ).view(1, 1, 1, 1, rot_half)
2937
+ base_cos_half = base_phase_half.cos()
2938
+ base_sin_half = base_phase_half.sin()
2939
+
2940
+ prior_strength = torch.as_tensor(
2941
+ self.circular_prior_strength,
2942
+ device=q.device,
2943
+ dtype=cos_k_half.dtype,
2944
+ )
2945
+ cos_k_half = cos_k_half + prior_strength * base_cos_half
2946
+ sin_k_half = sin_k_half + prior_strength * base_sin_half
2947
+
2948
+ # Project Z_lambda back onto S^1. The weak prior regularizes the usual
2949
+ # low-coherence regime but cannot forbid the exact cancellation
2950
+ # Z+lambda*exp(i*psi0)=0. At that machine-zero event every angle is a
2951
+ # minimizer of the regularized objective, so use the canonical phase as
2952
+ # a deterministic finite representative rather than identity-at-zero.
2953
  resultant_sq = cos_k_half.square() + sin_k_half.square()
2954
  well_defined = resultant_sq > torch.finfo(resultant_sq.dtype).eps
2955
  safe_resultant_sq = torch.where(
 
2957
  )
2958
  inv_resultant = torch.rsqrt(safe_resultant_sq)
2959
 
 
 
 
2960
  cos_k_half = torch.where(
2961
  well_defined,
2962
  cos_k_half * inv_resultant,
2963
+ base_cos_half.expand_as(cos_k_half),
2964
  )
2965
  sin_k_half = torch.where(
2966
  well_defined,
2967
  sin_k_half * inv_resultant,
2968
+ base_sin_half.expand_as(sin_k_half),
2969
  )
2970
  cos_k_half = cos_k_half.reshape(B, H_k_eff, S, rot_half)
2971
  sin_k_half = sin_k_half.reshape(B, H_k_eff, S, rot_half)
 
3574
  num_key_value_heads=config.num_key_value_heads,
3575
  head_dim=self.head_dim,
3576
  )
3577
+ # The currently available CCE/Triton REPO-GRAPE kernel implements
3578
+ # the legacy uncentered + uniform + no-prior projection. The model
3579
+ # now centers the spectrum, learns structural GQA weights and adds a
3580
+ # weak canonical circular prior, so that kernel MUST stay disabled
3581
+ # until its forward/backward contract is updated in lockstep.
3582
+ # Otherwise Torch and Triton would silently implement different math.
3583
  self.use_repo_grape_triton = False
3584
  else:
3585
  self.repo_grape = None
 
4734
  # Pure REPO (without GRAPE) keeps the original paper behavior and
4735
  # therefore continues to use z directly.
4736
  if not _repo_grape_kernel_used:
4737
+ if self.use_repo_grape:
4738
+ # Keep the exact canonical runtime positions alongside u.
4739
+ # They are needed only by the weak circular prior on shared
4740
+ # GQA keys and are never materialized once per query head.
4741
+ repo_coords, repo_reference_positions = (
4742
+ self.repo_grape.transform_positions(
4743
+ z, position_ids, return_reference=True
4744
+ )
4745
+ )
4746
+ else:
4747
+ repo_coords = z
4748
+ repo_reference_positions = None
4749
 
4750
  if not _repo_grape_kernel_used and _iha_seq_expand:
4751
  # IHA is applied *after* the coordinate map. Thus REPO-GRAPE
 
4763
  ).reshape(repo_coords.shape[0], repo_coords.shape[1], _S_orig * P)
4764
 
4765
  if self.use_repo_grape:
4766
+ # Match the canonical prior to the same IHA pseudo-slot:
4767
+ # u_{i,p}=P*u_i+p => i_{i,p}=P*i+p.
4768
+ reference_expanded = (
4769
+ repo_reference_positions.unsqueeze(-1) * P
4770
+ + p_offsets.view(1, 1, P)
4771
+ ).reshape(repo_reference_positions.shape[0], _S_orig * P)
4772
  q, k = self.repo_grape.apply_multiplicative(
4773
+ q,
4774
+ k,
4775
+ coords_expanded,
4776
+ inv_freq,
4777
+ attn_scaling,
4778
+ reference_positions=reference_expanded,
4779
  )
4780
  else:
4781
  q, k = _apply_repo_rope(
 
4786
  # the original REPO-RoPE path.
4787
  if self.use_repo_grape:
4788
  q, k = self.repo_grape.apply_multiplicative(
4789
+ q,
4790
+ k,
4791
+ repo_coords,
4792
+ inv_freq,
4793
+ attn_scaling,
4794
+ reference_positions=repo_reference_positions,
4795
  )
4796
  else:
4797
  q, k = _apply_repo_rope(