Text Generation
Transformers
Safetensors
PyTorch
English
neollm
causal-lm
research
bf16
attention
normalization
conversational
custom_code
Instructions to use KitsuVp/NeoLLM with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use KitsuVp/NeoLLM with Transformers:
# Use a pipeline as a high-level helper from transformers import pipeline pipe = pipeline("text-generation", model="KitsuVp/NeoLLM", trust_remote_code=True) messages = [ {"role": "user", "content": "Who are you?"}, ] pipe(messages)# pip install -U transformers accelerate # Load model directly from transformers import AutoModelForCausalLM model = AutoModelForCausalLM.from_pretrained("KitsuVp/NeoLLM", trust_remote_code=True, device_map="auto") - Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- vLLM
How to use KitsuVp/NeoLLM with vLLM:
Install from pip and serve model
# Install vLLM from pip: pip install vllm # Start the vLLM server: vllm serve "KitsuVp/NeoLLM" # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:8000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "KitsuVp/NeoLLM", "messages": [ { "role": "user", "content": "What is the capital of France?" } ] }'Use Docker
docker model run hf.co/KitsuVp/NeoLLM
- SGLang
How to use KitsuVp/NeoLLM with SGLang:
Install from pip and serve model
# Install SGLang from pip: pip install sglang # Start the SGLang server: python3 -m sglang.launch_server \ --model-path "KitsuVp/NeoLLM" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "KitsuVp/NeoLLM", "messages": [ { "role": "user", "content": "What is the capital of France?" } ] }'Use Docker images
docker run --gpus all \ --shm-size 32g \ -p 30000:30000 \ -v ~/.cache/huggingface:/root/.cache/huggingface \ --env "HF_TOKEN=<secret>" \ --ipc=host \ lmsysorg/sglang:latest \ python3 -m sglang.launch_server \ --model-path "KitsuVp/NeoLLM" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "KitsuVp/NeoLLM", "messages": [ { "role": "user", "content": "What is the capital of France?" } ] }' - Docker Model Runner
How to use KitsuVp/NeoLLM with Docker Model Runner:
docker model run hf.co/KitsuVp/NeoLLM
Update modeling_neollm.py
Browse files- modeling_neollm.py +275 -167
modeling_neollm.py
CHANGED
|
@@ -2512,91 +2512,94 @@ def _apply_repo_rope(
|
|
| 2512 |
|
| 2513 |
class RepoGrapePositioning(nn.Module):
|
| 2514 |
r"""
|
| 2515 |
-
|
| 2516 |
|
| 2517 |
-
|
| 2518 |
-
|
| 2519 |
-
|
| 2520 |
|
| 2521 |
z_i^(h) = f_phi^(h)(h_i),
|
| 2522 |
u_i^(h) = (1 - alpha_h) i + alpha_h z_i^(h)
|
| 2523 |
-
= i + alpha_h (z_i^(h) - i)
|
| 2524 |
|
| 2525 |
-
|
| 2526 |
-
|
| 2527 |
-
position streams. ``alpha_h`` is one learned scalar per query head. It
|
| 2528 |
-
is initialized to 1, so a newly constructed model starts exactly at the
|
| 2529 |
-
previous REPO coordinate u=z and can learn a linear anchor without an
|
| 2530 |
-
abrupt initialization change.
|
| 2531 |
|
| 2532 |
-
|
| 2533 |
-
|
| 2534 |
-
constrained to stay there. This avoids an additional nonlinear operation,
|
| 2535 |
-
preserves the exact alpha_h=1 initialization, and lets the optimizer learn
|
| 2536 |
-
interpolation or extrapolation. Constraining alpha would be a different
|
| 2537 |
-
parameterization and is not silently introduced here.
|
| 2538 |
|
| 2539 |
-
|
| 2540 |
-
|
| 2541 |
|
| 2542 |
-
|
| 2543 |
-
|
|
|
|
|
|
|
| 2544 |
|
| 2545 |
-
|
|
|
|
| 2546 |
|
| 2547 |
-
|
| 2548 |
|
| 2549 |
-
|
| 2550 |
-
|
| 2551 |
-
|
|
|
|
| 2552 |
|
| 2553 |
-
|
| 2554 |
-
- ``position_blend_alpha`` is an FP32 ``nn.Parameter`` of shape [H_q].
|
| 2555 |
-
It is optimized normally and is saved/restored in ``state_dict``.
|
| 2556 |
-
- ``freq_log_scale`` remains an FP32 saved parameter of shape
|
| 2557 |
-
[H_q, head_dim/2].
|
| 2558 |
-
- ``position_ids`` is runtime metadata and is never copied into a buffer or
|
| 2559 |
-
checkpoint.
|
| 2560 |
-
- ``u`` and the optional IHA-expanded ``u_{i,p}`` are transient autograd
|
| 2561 |
-
intermediates only. The IHA tensor replaces the previous ``P*z+p``
|
| 2562 |
-
intermediate, so this change does not add another expanded coordinate
|
| 2563 |
-
tensor.
|
| 2564 |
|
| 2565 |
-
|
|
|
|
|
|
|
| 2566 |
|
| 2567 |
-
|
| 2568 |
|
| 2569 |
-
|
|
|
|
| 2570 |
|
| 2571 |
-
|
| 2572 |
-
theta_{h,r} = inv_freq_r * exp(s_{h,r}).
|
| 2573 |
|
| 2574 |
-
|
| 2575 |
-
|
| 2576 |
|
| 2577 |
-
|
| 2578 |
-
|
|
|
|
|
|
|
|
|
|
| 2579 |
|
| 2580 |
-
|
| 2581 |
-
|
| 2582 |
-
|
| 2583 |
-
|
| 2584 |
-
|
| 2585 |
|
| 2586 |
-
|
| 2587 |
-
|
|
|
|
|
|
|
| 2588 |
|
| 2589 |
-
|
|
|
|
| 2590 |
|
| 2591 |
-
|
| 2592 |
|
| 2593 |
-
|
| 2594 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 2595 |
|
| 2596 |
-
|
| 2597 |
-
|
| 2598 |
-
|
| 2599 |
-
|
| 2600 |
"""
|
| 2601 |
|
| 2602 |
def __init__(
|
|
@@ -2628,8 +2631,13 @@ class RepoGrapePositioning(nn.Module):
|
|
| 2628 |
requires_grad=True,
|
| 2629 |
)
|
| 2630 |
|
| 2631 |
-
#
|
| 2632 |
-
#
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 2633 |
self.freq_log_scale = nn.Parameter(
|
| 2634 |
torch.zeros(
|
| 2635 |
self.num_attention_heads, self.max_rot_half, dtype=torch.float32
|
|
@@ -2637,6 +2645,34 @@ class RepoGrapePositioning(nn.Module):
|
|
| 2637 |
requires_grad=True,
|
| 2638 |
)
|
| 2639 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 2640 |
def _load_from_state_dict(
|
| 2641 |
self,
|
| 2642 |
state_dict,
|
|
@@ -2647,20 +2683,27 @@ class RepoGrapePositioning(nn.Module):
|
|
| 2647 |
unexpected_keys,
|
| 2648 |
error_msgs,
|
| 2649 |
):
|
| 2650 |
-
# Backward-compatible checkpoint migration.
|
| 2651 |
-
#
|
| 2652 |
-
#
|
| 2653 |
-
# checkpoint
|
|
|
|
|
|
|
|
|
|
| 2654 |
alpha_key = prefix + "position_blend_alpha"
|
| 2655 |
-
|
|
|
|
|
|
|
|
|
|
| 2656 |
if alpha_key not in state_dict:
|
| 2657 |
-
# Use a local shallow copy so loading does not mutate the caller's
|
| 2658 |
-
# checkpoint mapping. CPU FP32 also works when the module was
|
| 2659 |
-
# constructed on meta before Hugging Face materializes parameters.
|
| 2660 |
-
local_state_dict = state_dict.copy()
|
| 2661 |
local_state_dict[alpha_key] = torch.ones(
|
| 2662 |
self.num_attention_heads, dtype=torch.float32
|
| 2663 |
)
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 2664 |
super()._load_from_state_dict(
|
| 2665 |
local_state_dict,
|
| 2666 |
prefix,
|
|
@@ -2672,37 +2715,76 @@ class RepoGrapePositioning(nn.Module):
|
|
| 2672 |
)
|
| 2673 |
|
| 2674 |
def _apply(self, fn):
|
| 2675 |
-
# Keep the
|
| 2676 |
-
#
|
| 2677 |
-
#
|
| 2678 |
super()._apply(fn)
|
| 2679 |
-
for p in (
|
|
|
|
|
|
|
|
|
|
|
|
|
| 2680 |
p.data = p.data.float()
|
| 2681 |
if p.grad is not None:
|
| 2682 |
p.grad.data = p.grad.data.float()
|
| 2683 |
return self
|
| 2684 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 2685 |
def transform_positions(
|
| 2686 |
self,
|
| 2687 |
z: torch.Tensor,
|
| 2688 |
position_ids: Optional[torch.Tensor],
|
| 2689 |
-
|
| 2690 |
-
|
|
|
|
|
|
|
| 2691 |
|
| 2692 |
-
|
| 2693 |
|
| 2694 |
u_i^(h) = (1-alpha_h) i + alpha_h z_i^(h)
|
| 2695 |
= i + alpha_h (z_i^(h) - i).
|
| 2696 |
|
| 2697 |
-
|
| 2698 |
-
|
| 2699 |
-
|
| 2700 |
|
| 2701 |
-
|
| 2702 |
-
computation. It is not registered as a buffer and is not saved in the
|
| 2703 |
-
checkpoint. Standard autograd gives
|
| 2704 |
|
| 2705 |
-
dL/
|
|
|
|
| 2706 |
"""
|
| 2707 |
B, H, S = z.shape
|
| 2708 |
if H != self.num_attention_heads:
|
|
@@ -2710,35 +2792,13 @@ class RepoGrapePositioning(nn.Module):
|
|
| 2710 |
f"REPO-GRAPE expected z heads={self.num_attention_heads}, got {H}."
|
| 2711 |
)
|
| 2712 |
|
| 2713 |
-
|
| 2714 |
-
# Defensive fallback for direct attention calls. NeoLLMModel normally
|
| 2715 |
-
# constructs position_ids before entering the decoder layers.
|
| 2716 |
-
pid = torch.arange(S, device=z.device, dtype=torch.float32).view(1, S)
|
| 2717 |
-
else:
|
| 2718 |
-
pid = position_ids
|
| 2719 |
-
if pid.dim() == 1:
|
| 2720 |
-
pid = pid.unsqueeze(0)
|
| 2721 |
-
elif pid.dim() > 2:
|
| 2722 |
-
# Match the existing IHA RoPE convention: trainers may attach
|
| 2723 |
-
# metadata in the final axis; column 0 is the sequential index.
|
| 2724 |
-
pid = pid[..., 0]
|
| 2725 |
-
if pid.dim() != 2 or pid.shape[-1] != S:
|
| 2726 |
-
raise ValueError(
|
| 2727 |
-
"REPO-GRAPE position_ids must have shape [S] or [B,S] "
|
| 2728 |
-
f"matching z[...,S={S}], got {tuple(pid.shape)}."
|
| 2729 |
-
)
|
| 2730 |
-
if pid.shape[0] == 1 and B != 1:
|
| 2731 |
-
pid = pid.expand(B, -1)
|
| 2732 |
-
elif pid.shape[0] != B:
|
| 2733 |
-
raise ValueError(
|
| 2734 |
-
f"REPO-GRAPE position_ids batch={pid.shape[0]} does not match z batch={B}."
|
| 2735 |
-
)
|
| 2736 |
-
pid = pid.to(device=z.device, dtype=torch.float32)
|
| 2737 |
-
|
| 2738 |
-
# Views/broadcasting do not allocate [B,H,S] copies of pid or alpha.
|
| 2739 |
i = pid.unsqueeze(1)
|
| 2740 |
alpha = self.position_blend_alpha.view(1, H, 1)
|
| 2741 |
-
|
|
|
|
|
|
|
|
|
|
| 2742 |
|
| 2743 |
def _query_freq(
|
| 2744 |
self,
|
|
@@ -2756,11 +2816,20 @@ class RepoGrapePositioning(nn.Module):
|
|
| 2756 |
)
|
| 2757 |
|
| 2758 |
base = inv_freq.float().view(1, rot_half)
|
| 2759 |
-
|
| 2760 |
-
freq = base * torch.exp(log_scale)
|
| 2761 |
-
freq = freq.to(device=inv_freq.device)
|
| 2762 |
|
| 2763 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 2764 |
|
| 2765 |
def apply_multiplicative(
|
| 2766 |
self,
|
|
@@ -2769,41 +2838,33 @@ class RepoGrapePositioning(nn.Module):
|
|
| 2769 |
u: torch.Tensor,
|
| 2770 |
inv_freq: torch.Tensor,
|
| 2771 |
attention_scaling: float,
|
|
|
|
|
|
|
| 2772 |
) -> Tuple[torch.Tensor, torch.Tensor]:
|
| 2773 |
-
r"""
|
| 2774 |
-
Apply REPO-GRAPE-M rotation to Q/K using final coordinates ``u``.
|
| 2775 |
|
| 2776 |
-
``u`` is already the linear-blend coordinate produced by
|
| 2777 |
-
``transform_positions`` (and, for IHA, already expanded as P*u+p).
|
| 2778 |
Query heads keep their individual phases
|
| 2779 |
|
| 2780 |
-
phi_{
|
| 2781 |
|
| 2782 |
-
For
|
| 2783 |
-
rotation is the circular projection
|
| 2784 |
|
| 2785 |
-
|
| 2786 |
-
|
| 2787 |
-
(cos psi*, sin psi*) = (C, S) / sqrt(C^2 + S^2).
|
| 2788 |
|
| 2789 |
-
|
| 2790 |
-
total squared discrepancy between the head-specific rotated keys and
|
| 2791 |
-
one shared rotated key. The reduction is performed independently for
|
| 2792 |
-
every batch element, current sequence index, rotary plane and KV group.
|
| 2793 |
-
Because seq-expand IHA stores each pseudo-slot at a different sequence
|
| 2794 |
-
index, its pseudo-slots remain independent automatically. No extra
|
| 2795 |
-
mode or configuration flag is introduced.
|
| 2796 |
|
| 2797 |
-
|
| 2798 |
-
|
| 2799 |
-
|
| 2800 |
-
u: [B, H_q_base, S] final REPO-GRAPE coordinates
|
| 2801 |
-
inv_freq: [rotary_dim/2]
|
| 2802 |
-
attention_scaling: scaling from NeoLLMRotaryEmbedding
|
| 2803 |
|
| 2804 |
-
|
| 2805 |
-
|
| 2806 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
| 2807 |
"""
|
| 2808 |
B, H_repo, S = u.shape
|
| 2809 |
H_q_eff = q.shape[1]
|
|
@@ -2849,21 +2910,46 @@ class RepoGrapePositioning(nn.Module):
|
|
| 2849 |
cos_q_half = phase_q.cos()
|
| 2850 |
sin_q_half = phase_q.sin()
|
| 2851 |
|
| 2852 |
-
# GQA circular projection. Restore the logical head
|
| 2853 |
-
#
|
| 2854 |
-
#
|
| 2855 |
-
# path pseudo-slots live on the sequence axis, which is never reduced.
|
| 2856 |
-
# Averaging cos/sin and normalizing the resultant is exactly equivalent
|
| 2857 |
-
# to arg(mean(exp(i*phase))).
|
| 2858 |
cos_q_grouped = cos_q_half.reshape(
|
| 2859 |
B, H_k_base, q_per_k_base, P, S, rot_half
|
| 2860 |
)
|
| 2861 |
sin_q_grouped = sin_q_half.reshape(
|
| 2862 |
B, H_k_base, q_per_k_base, P, S, rot_half
|
| 2863 |
)
|
| 2864 |
-
cos_k_half = cos_q_grouped.mean(dim=2)
|
| 2865 |
-
sin_k_half = sin_q_grouped.mean(dim=2)
|
| 2866 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 2867 |
resultant_sq = cos_k_half.square() + sin_k_half.square()
|
| 2868 |
well_defined = resultant_sq > torch.finfo(resultant_sq.dtype).eps
|
| 2869 |
safe_resultant_sq = torch.where(
|
|
@@ -2871,18 +2957,15 @@ class RepoGrapePositioning(nn.Module):
|
|
| 2871 |
)
|
| 2872 |
inv_resultant = torch.rsqrt(safe_resultant_sq)
|
| 2873 |
|
| 2874 |
-
# If the resultant vanishes, every shared angle has the same objective
|
| 2875 |
-
# value. Select identity (cos=1, sin=0) to keep forward/backward finite
|
| 2876 |
-
# and deterministic; this is a numerical convention, not a new mode.
|
| 2877 |
cos_k_half = torch.where(
|
| 2878 |
well_defined,
|
| 2879 |
cos_k_half * inv_resultant,
|
| 2880 |
-
|
| 2881 |
)
|
| 2882 |
sin_k_half = torch.where(
|
| 2883 |
well_defined,
|
| 2884 |
sin_k_half * inv_resultant,
|
| 2885 |
-
|
| 2886 |
)
|
| 2887 |
cos_k_half = cos_k_half.reshape(B, H_k_eff, S, rot_half)
|
| 2888 |
sin_k_half = sin_k_half.reshape(B, H_k_eff, S, rot_half)
|
|
@@ -3491,10 +3574,12 @@ class NeoLLMAttention(nn.Module):
|
|
| 3491 |
num_key_value_heads=config.num_key_value_heads,
|
| 3492 |
head_dim=self.head_dim,
|
| 3493 |
)
|
| 3494 |
-
#
|
| 3495 |
-
#
|
| 3496 |
-
#
|
| 3497 |
-
#
|
|
|
|
|
|
|
| 3498 |
self.use_repo_grape_triton = False
|
| 3499 |
else:
|
| 3500 |
self.repo_grape = None
|
|
@@ -4649,11 +4734,18 @@ class NeoLLMAttention(nn.Module):
|
|
| 4649 |
# Pure REPO (without GRAPE) keeps the original paper behavior and
|
| 4650 |
# therefore continues to use z directly.
|
| 4651 |
if not _repo_grape_kernel_used:
|
| 4652 |
-
|
| 4653 |
-
|
| 4654 |
-
|
| 4655 |
-
|
| 4656 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 4657 |
|
| 4658 |
if not _repo_grape_kernel_used and _iha_seq_expand:
|
| 4659 |
# IHA is applied *after* the coordinate map. Thus REPO-GRAPE
|
|
@@ -4671,8 +4763,19 @@ class NeoLLMAttention(nn.Module):
|
|
| 4671 |
).reshape(repo_coords.shape[0], repo_coords.shape[1], _S_orig * P)
|
| 4672 |
|
| 4673 |
if self.use_repo_grape:
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 4674 |
q, k = self.repo_grape.apply_multiplicative(
|
| 4675 |
-
q,
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 4676 |
)
|
| 4677 |
else:
|
| 4678 |
q, k = _apply_repo_rope(
|
|
@@ -4683,7 +4786,12 @@ class NeoLLMAttention(nn.Module):
|
|
| 4683 |
# the original REPO-RoPE path.
|
| 4684 |
if self.use_repo_grape:
|
| 4685 |
q, k = self.repo_grape.apply_multiplicative(
|
| 4686 |
-
q,
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 4687 |
)
|
| 4688 |
else:
|
| 4689 |
q, k = _apply_repo_rope(
|
|
|
|
| 2512 |
|
| 2513 |
class RepoGrapePositioning(nn.Module):
|
| 2514 |
r"""
|
| 2515 |
+
REPO-GRAPE-M positional operator with a well-conditioned GQA projection.
|
| 2516 |
|
| 2517 |
+
Revision implemented here
|
| 2518 |
+
-------------------------
|
| 2519 |
+
This version keeps the contextual coordinate map unchanged,
|
| 2520 |
|
| 2521 |
z_i^(h) = f_phi^(h)(h_i),
|
| 2522 |
u_i^(h) = (1 - alpha_h) i + alpha_h z_i^(h)
|
| 2523 |
+
= i + alpha_h (z_i^(h) - i),
|
| 2524 |
|
| 2525 |
+
and applies exactly three corrections identified by the independent
|
| 2526 |
+
REPO-GRAPE analysis:
|
|
|
|
|
|
|
|
|
|
|
|
|
| 2527 |
|
| 2528 |
+
1. Spectral gauge fixing. The raw per-head log spectrum s_{h,r} is centered
|
| 2529 |
+
across the active rotary planes before exponentiation,
|
|
|
|
|
|
|
|
|
|
|
|
|
| 2530 |
|
| 2531 |
+
s~_{h,r} = s_{h,r} - mean_r s_{h,r},
|
| 2532 |
+
theta_{h,r} = inv_freq_r * exp(s~_{h,r}).
|
| 2533 |
|
| 2534 |
+
Hence sum_r s~_{h,r}=0 and the geometric mean of exp(s~) is one. The
|
| 2535 |
+
learned spectrum can reshape relative frequencies but cannot absorb a
|
| 2536 |
+
uniform scale that is functionally confounded with u. Autograd applies
|
| 2537 |
+
the matching zero-mean projection to the raw-s gradient.
|
| 2538 |
|
| 2539 |
+
2. Learned GQA head importance. If H_m is the set of query heads sharing
|
| 2540 |
+
KV head m, one FP32 logit a_h per query head defines
|
| 2541 |
|
| 2542 |
+
w_h = softmax_{h in H_m}(a_h), sum_{h in H_m} w_h = 1.
|
| 2543 |
|
| 2544 |
+
a_h starts at zero, so the weighted circular reduction starts exactly at
|
| 2545 |
+
the previous uniform weighting. These are structural per-head scalars,
|
| 2546 |
+
not a token-dependent router, so there is no activation-sized routing
|
| 2547 |
+
tensor.
|
| 2548 |
|
| 2549 |
+
3. Weak circular prior toward the canonical RoPE phase. Let
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 2550 |
|
| 2551 |
+
phi_{h,i,r} = u_i^(h) * theta_{h,r},
|
| 2552 |
+
psi0_{i,r} = i * inv_freq_r,
|
| 2553 |
+
Z_{m,i,r} = sum_{h in H_m} w_h exp(i*phi_{h,i,r}).
|
| 2554 |
|
| 2555 |
+
The shared key rotation is obtained from
|
| 2556 |
|
| 2557 |
+
Z_lambda = Z + lambda * exp(i*psi0),
|
| 2558 |
+
exp(i*psi*) = Z_lambda / |Z_lambda|.
|
| 2559 |
|
| 2560 |
+
This is the exact minimizer of the regularized chordal objective
|
|
|
|
| 2561 |
|
| 2562 |
+
sum_h w_h ||R(phi_h)k - R(psi)k||_2^2
|
| 2563 |
+
+ lambda ||R(psi0)k - R(psi)k||_2^2.
|
| 2564 |
|
| 2565 |
+
When the query heads are coherent, |Z| >> lambda and the prior is almost
|
| 2566 |
+
invisible. When their resultant is small, the canonical RoPE direction
|
| 2567 |
+
supplies a weak reference. Exact cancellation Z_lambda=0 remains
|
| 2568 |
+
mathematically non-identifiable; at that machine-zero event the code uses
|
| 2569 |
+
the canonical RoPE direction as a deterministic finite representative.
|
| 2570 |
|
| 2571 |
+
The default prior strength is 1e-2 relative to a weighted resultant whose
|
| 2572 |
+
maximum magnitude is one. It is read from the optional config attribute
|
| 2573 |
+
``repo_grape_circular_prior_strength`` and written back to ``config`` so the
|
| 2574 |
+
effective value is serialized. Setting it to zero is the clean prior
|
| 2575 |
+
ablation; centered spectrum and learned GQA weights remain active.
|
| 2576 |
|
| 2577 |
+
Runtime positions and IHA
|
| 2578 |
+
-------------------------
|
| 2579 |
+
``i`` is the actual runtime ``position_ids`` value. With IHA sequence
|
| 2580 |
+
expansion, both the contextual coordinate and its canonical reference use
|
| 2581 |
|
| 2582 |
+
u_{i,p} = P*u_i + p,
|
| 2583 |
+
i_{i,p} = P*i + p,
|
| 2584 |
|
| 2585 |
+
so the prior points to the RoPE phase of the same pseudo-slot being rotated.
|
| 2586 |
|
| 2587 |
+
Storage / checkpoint contract
|
| 2588 |
+
-----------------------------
|
| 2589 |
+
- ``position_blend_alpha``: FP32 Parameter [H_q], unchanged.
|
| 2590 |
+
- ``freq_log_scale``: FP32 Parameter [H_q, head_dim/2], unchanged in shape;
|
| 2591 |
+
its effective value is centered before use.
|
| 2592 |
+
- ``gqa_head_weight_logits``: new FP32 Parameter [H_q], initialized to zero;
|
| 2593 |
+
old checkpoints receive this zero vector during strict loading.
|
| 2594 |
+
- ``repo_grape_circular_prior_strength``: scalar config hyperparameter, not a
|
| 2595 |
+
learned Parameter and not an activation-sized buffer.
|
| 2596 |
+
- ``u``, centered s~, phases, resultants and reference phases are transient
|
| 2597 |
+
autograd intermediates only.
|
| 2598 |
|
| 2599 |
+
The current optional CCE/Triton REPO-GRAPE kernel implements the preceding
|
| 2600 |
+
legacy uncentered/unweighted/unregularized contract. It therefore remains
|
| 2601 |
+
disabled in ``NeoLLMAttention`` until that kernel is updated in lockstep;
|
| 2602 |
+
silently dispatching it would change the model mathematics.
|
| 2603 |
"""
|
| 2604 |
|
| 2605 |
def __init__(
|
|
|
|
| 2631 |
requires_grad=True,
|
| 2632 |
)
|
| 2633 |
|
| 2634 |
+
# Raw learned log-spectrum. The stored parameter keeps the historical
|
| 2635 |
+
# shape/state-dict key, but _query_freq fixes the common-scale gauge via
|
| 2636 |
+
#
|
| 2637 |
+
# s_tilde[h,r] = s[h,r] - mean_r(s[h,r]).
|
| 2638 |
+
#
|
| 2639 |
+
# This preserves relative per-plane spectral reshaping while removing the
|
| 2640 |
+
# uniform log-frequency mode that can trade against u in phi=u*theta.
|
| 2641 |
self.freq_log_scale = nn.Parameter(
|
| 2642 |
torch.zeros(
|
| 2643 |
self.num_attention_heads, self.max_rot_half, dtype=torch.float32
|
|
|
|
| 2645 |
requires_grad=True,
|
| 2646 |
)
|
| 2647 |
|
| 2648 |
+
# Structural importance logits for the query heads inside each GQA group.
|
| 2649 |
+
# Softmax is group-local. Zero initialization gives exactly uniform
|
| 2650 |
+
# weights, so the new degree of freedom begins from the old reduction.
|
| 2651 |
+
self.gqa_head_weight_logits = nn.Parameter(
|
| 2652 |
+
torch.zeros(self.num_attention_heads, dtype=torch.float32),
|
| 2653 |
+
requires_grad=True,
|
| 2654 |
+
)
|
| 2655 |
+
|
| 2656 |
+
# Weak canonical circular prior. The weighted head resultant has |Z|<=1,
|
| 2657 |
+
# so lambda=1e-2 means a 1% reference vector on the same scale. Keep this
|
| 2658 |
+
# as a config hyperparameter rather than a Parameter to avoid optimizer
|
| 2659 |
+
# state while preserving run reproducibility.
|
| 2660 |
+
self.circular_prior_strength = float(
|
| 2661 |
+
getattr(config, "repo_grape_circular_prior_strength", 1.0e-2)
|
| 2662 |
+
)
|
| 2663 |
+
if (
|
| 2664 |
+
not math.isfinite(self.circular_prior_strength)
|
| 2665 |
+
or self.circular_prior_strength < 0.0
|
| 2666 |
+
):
|
| 2667 |
+
raise ValueError(
|
| 2668 |
+
"repo_grape_circular_prior_strength must be a finite non-negative scalar."
|
| 2669 |
+
)
|
| 2670 |
+
setattr(
|
| 2671 |
+
self.config,
|
| 2672 |
+
"repo_grape_circular_prior_strength",
|
| 2673 |
+
self.circular_prior_strength,
|
| 2674 |
+
)
|
| 2675 |
+
|
| 2676 |
def _load_from_state_dict(
|
| 2677 |
self,
|
| 2678 |
state_dict,
|
|
|
|
| 2683 |
unexpected_keys,
|
| 2684 |
error_msgs,
|
| 2685 |
):
|
| 2686 |
+
# Backward-compatible checkpoint migration.
|
| 2687 |
+
#
|
| 2688 |
+
# Very old checkpoints may lack alpha_h, and every pre-correction
|
| 2689 |
+
# checkpoint lacks the new GQA importance logits. alpha_h=1 preserves
|
| 2690 |
+
# the historical u=z initialization; a_h=0 restores uniform head weights.
|
| 2691 |
+
# Centering freq_log_scale intentionally changes only its common per-head
|
| 2692 |
+
# mode and therefore needs no new state tensor.
|
| 2693 |
alpha_key = prefix + "position_blend_alpha"
|
| 2694 |
+
weight_key = prefix + "gqa_head_weight_logits"
|
| 2695 |
+
need_copy = alpha_key not in state_dict or weight_key not in state_dict
|
| 2696 |
+
local_state_dict = state_dict.copy() if need_copy else state_dict
|
| 2697 |
+
|
| 2698 |
if alpha_key not in state_dict:
|
|
|
|
|
|
|
|
|
|
|
|
|
| 2699 |
local_state_dict[alpha_key] = torch.ones(
|
| 2700 |
self.num_attention_heads, dtype=torch.float32
|
| 2701 |
)
|
| 2702 |
+
if weight_key not in state_dict:
|
| 2703 |
+
local_state_dict[weight_key] = torch.zeros(
|
| 2704 |
+
self.num_attention_heads, dtype=torch.float32
|
| 2705 |
+
)
|
| 2706 |
+
|
| 2707 |
super()._load_from_state_dict(
|
| 2708 |
local_state_dict,
|
| 2709 |
prefix,
|
|
|
|
| 2715 |
)
|
| 2716 |
|
| 2717 |
def _apply(self, fn):
|
| 2718 |
+
# Keep the angular control parameters in FP32 under model.to(dtype).
|
| 2719 |
+
# Phases/resultants are evaluated in FP32, so quantizing these tiny state
|
| 2720 |
+
# tensors would save negligible memory while perturbing the geometry.
|
| 2721 |
super()._apply(fn)
|
| 2722 |
+
for p in (
|
| 2723 |
+
self.position_blend_alpha,
|
| 2724 |
+
self.freq_log_scale,
|
| 2725 |
+
self.gqa_head_weight_logits,
|
| 2726 |
+
):
|
| 2727 |
p.data = p.data.float()
|
| 2728 |
if p.grad is not None:
|
| 2729 |
p.grad.data = p.grad.data.float()
|
| 2730 |
return self
|
| 2731 |
|
| 2732 |
+
def _resolve_runtime_positions(
|
| 2733 |
+
self,
|
| 2734 |
+
batch_size: int,
|
| 2735 |
+
seq_len: int,
|
| 2736 |
+
device: torch.device,
|
| 2737 |
+
position_ids: Optional[torch.Tensor],
|
| 2738 |
+
) -> torch.Tensor:
|
| 2739 |
+
"""Resolve canonical runtime positions as FP32 ``[B,S]`` metadata."""
|
| 2740 |
+
if position_ids is None:
|
| 2741 |
+
return torch.arange(
|
| 2742 |
+
seq_len, device=device, dtype=torch.float32
|
| 2743 |
+
).view(1, seq_len).expand(batch_size, -1)
|
| 2744 |
+
|
| 2745 |
+
pid = position_ids
|
| 2746 |
+
if pid.dim() == 1:
|
| 2747 |
+
pid = pid.unsqueeze(0)
|
| 2748 |
+
elif pid.dim() > 2:
|
| 2749 |
+
# Match the existing IHA convention: metadata column 0 is the
|
| 2750 |
+
# sequential position used by RoPE/REPO.
|
| 2751 |
+
pid = pid[..., 0]
|
| 2752 |
+
if pid.dim() != 2 or pid.shape[-1] != seq_len:
|
| 2753 |
+
raise ValueError(
|
| 2754 |
+
"REPO-GRAPE position_ids must have shape [S] or [B,S] "
|
| 2755 |
+
f"with S={seq_len}, got {tuple(pid.shape)}."
|
| 2756 |
+
)
|
| 2757 |
+
if pid.shape[0] == 1 and batch_size != 1:
|
| 2758 |
+
pid = pid.expand(batch_size, -1)
|
| 2759 |
+
elif pid.shape[0] != batch_size:
|
| 2760 |
+
raise ValueError(
|
| 2761 |
+
f"REPO-GRAPE position_ids batch={pid.shape[0]} does not match "
|
| 2762 |
+
f"batch={batch_size}."
|
| 2763 |
+
)
|
| 2764 |
+
return pid.to(device=device, dtype=torch.float32)
|
| 2765 |
+
|
| 2766 |
def transform_positions(
|
| 2767 |
self,
|
| 2768 |
z: torch.Tensor,
|
| 2769 |
position_ids: Optional[torch.Tensor],
|
| 2770 |
+
*,
|
| 2771 |
+
return_reference: bool = False,
|
| 2772 |
+
) -> Union[torch.Tensor, Tuple[torch.Tensor, torch.Tensor]]:
|
| 2773 |
+
r"""Build the linear-blend coordinate and optionally return its anchor.
|
| 2774 |
|
| 2775 |
+
Active map:
|
| 2776 |
|
| 2777 |
u_i^(h) = (1-alpha_h) i + alpha_h z_i^(h)
|
| 2778 |
= i + alpha_h (z_i^(h) - i).
|
| 2779 |
|
| 2780 |
+
``return_reference=True`` additionally returns canonical FP32 positions
|
| 2781 |
+
``i`` as [B,S]. They are reused to construct the RoPE prior phase after
|
| 2782 |
+
any IHA pseudo-slot expansion.
|
| 2783 |
|
| 2784 |
+
Standard autograd gives
|
|
|
|
|
|
|
| 2785 |
|
| 2786 |
+
dL/dz_i^(h) = alpha_h * ubar_i^(h),
|
| 2787 |
+
dL/dalpha_h = sum_{b,i} ubar_{b,h,i}(z_{b,h,i}-i).
|
| 2788 |
"""
|
| 2789 |
B, H, S = z.shape
|
| 2790 |
if H != self.num_attention_heads:
|
|
|
|
| 2792 |
f"REPO-GRAPE expected z heads={self.num_attention_heads}, got {H}."
|
| 2793 |
)
|
| 2794 |
|
| 2795 |
+
pid = self._resolve_runtime_positions(B, S, z.device, position_ids)
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 2796 |
i = pid.unsqueeze(1)
|
| 2797 |
alpha = self.position_blend_alpha.view(1, H, 1)
|
| 2798 |
+
u = i + alpha * (z.float() - i)
|
| 2799 |
+
if return_reference:
|
| 2800 |
+
return u, pid
|
| 2801 |
+
return u
|
| 2802 |
|
| 2803 |
def _query_freq(
|
| 2804 |
self,
|
|
|
|
| 2816 |
)
|
| 2817 |
|
| 2818 |
base = inv_freq.float().view(1, rot_half)
|
| 2819 |
+
raw_log_scale = self.freq_log_scale[:, :rot_half].float()
|
|
|
|
|
|
|
| 2820 |
|
| 2821 |
+
# Spectral gauge fixing on the active rotary planes:
|
| 2822 |
+
#
|
| 2823 |
+
# s_tilde = s - mean_r(s),
|
| 2824 |
+
# theta = theta0 * exp(s_tilde).
|
| 2825 |
+
#
|
| 2826 |
+
# The centering matrix is an orthogonal projection onto the zero-mean
|
| 2827 |
+
# subspace, so autograd also removes the common-mode gradient.
|
| 2828 |
+
centered_log_scale = raw_log_scale - raw_log_scale.mean(
|
| 2829 |
+
dim=-1, keepdim=True
|
| 2830 |
+
)
|
| 2831 |
+
freq = base * torch.exp(centered_log_scale)
|
| 2832 |
+
return freq.to(device=inv_freq.device)
|
| 2833 |
|
| 2834 |
def apply_multiplicative(
|
| 2835 |
self,
|
|
|
|
| 2838 |
u: torch.Tensor,
|
| 2839 |
inv_freq: torch.Tensor,
|
| 2840 |
attention_scaling: float,
|
| 2841 |
+
*,
|
| 2842 |
+
reference_positions: Optional[torch.Tensor] = None,
|
| 2843 |
) -> Tuple[torch.Tensor, torch.Tensor]:
|
| 2844 |
+
r"""Apply corrected REPO-GRAPE-M rotations to Q/K.
|
|
|
|
| 2845 |
|
|
|
|
|
|
|
| 2846 |
Query heads keep their individual phases
|
| 2847 |
|
| 2848 |
+
phi_{h,i,r} = u_i^(h) * theta0_r * exp(s~_{h,r}).
|
| 2849 |
|
| 2850 |
+
For each GQA group H_m, structural logits a_h define
|
|
|
|
| 2851 |
|
| 2852 |
+
w_h = softmax_{h in H_m}(a_h),
|
| 2853 |
+
Z = sum_h w_h exp(i*phi_h).
|
|
|
|
| 2854 |
|
| 2855 |
+
The shared key uses the weakly regularized projection
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 2856 |
|
| 2857 |
+
psi0 = i * theta0,
|
| 2858 |
+
Z_lambda = Z + lambda exp(i*psi0),
|
| 2859 |
+
exp(i*psi*) = Z_lambda / |Z_lambda|.
|
|
|
|
|
|
|
|
|
|
| 2860 |
|
| 2861 |
+
This is still a closed-form SO(2) projection: it minimizes the weighted
|
| 2862 |
+
chordal key distortion plus lambda times the same distortion to the
|
| 2863 |
+
canonical RoPE rotation. The prior acts only on the shared GQA key;
|
| 2864 |
+
query phases remain fully REPO-GRAPE-specific.
|
| 2865 |
+
|
| 2866 |
+
``reference_positions`` must describe the same final sequence axis as
|
| 2867 |
+
``u``. The attention caller passes P*i+p when IHA expands pseudo-slots.
|
| 2868 |
"""
|
| 2869 |
B, H_repo, S = u.shape
|
| 2870 |
H_q_eff = q.shape[1]
|
|
|
|
| 2910 |
cos_q_half = phase_q.cos()
|
| 2911 |
sin_q_half = phase_q.sin()
|
| 2912 |
|
| 2913 |
+
# Corrected GQA circular projection. Restore the logical base-head
|
| 2914 |
+
# grouping and reduce only over query-within-group. Structural weights
|
| 2915 |
+
# are token-independent and initialized uniformly.
|
|
|
|
|
|
|
|
|
|
| 2916 |
cos_q_grouped = cos_q_half.reshape(
|
| 2917 |
B, H_k_base, q_per_k_base, P, S, rot_half
|
| 2918 |
)
|
| 2919 |
sin_q_grouped = sin_q_half.reshape(
|
| 2920 |
B, H_k_base, q_per_k_base, P, S, rot_half
|
| 2921 |
)
|
|
|
|
|
|
|
| 2922 |
|
| 2923 |
+
head_logits = self.gqa_head_weight_logits.reshape(
|
| 2924 |
+
H_k_base, q_per_k_base
|
| 2925 |
+
).float()
|
| 2926 |
+
head_weights = torch.softmax(head_logits, dim=-1).view(
|
| 2927 |
+
1, H_k_base, q_per_k_base, 1, 1, 1
|
| 2928 |
+
)
|
| 2929 |
+
cos_k_half = (cos_q_grouped * head_weights).sum(dim=2)
|
| 2930 |
+
sin_k_half = (sin_q_grouped * head_weights).sum(dim=2)
|
| 2931 |
+
|
| 2932 |
+
# Canonical RoPE prior for the same runtime/pseudo-slot positions.
|
| 2933 |
+
ref = self._resolve_runtime_positions(B, S, q.device, reference_positions)
|
| 2934 |
+
base_phase_half = ref.view(B, 1, 1, S, 1) * inv_freq.float().to(
|
| 2935 |
+
device=q.device
|
| 2936 |
+
).view(1, 1, 1, 1, rot_half)
|
| 2937 |
+
base_cos_half = base_phase_half.cos()
|
| 2938 |
+
base_sin_half = base_phase_half.sin()
|
| 2939 |
+
|
| 2940 |
+
prior_strength = torch.as_tensor(
|
| 2941 |
+
self.circular_prior_strength,
|
| 2942 |
+
device=q.device,
|
| 2943 |
+
dtype=cos_k_half.dtype,
|
| 2944 |
+
)
|
| 2945 |
+
cos_k_half = cos_k_half + prior_strength * base_cos_half
|
| 2946 |
+
sin_k_half = sin_k_half + prior_strength * base_sin_half
|
| 2947 |
+
|
| 2948 |
+
# Project Z_lambda back onto S^1. The weak prior regularizes the usual
|
| 2949 |
+
# low-coherence regime but cannot forbid the exact cancellation
|
| 2950 |
+
# Z+lambda*exp(i*psi0)=0. At that machine-zero event every angle is a
|
| 2951 |
+
# minimizer of the regularized objective, so use the canonical phase as
|
| 2952 |
+
# a deterministic finite representative rather than identity-at-zero.
|
| 2953 |
resultant_sq = cos_k_half.square() + sin_k_half.square()
|
| 2954 |
well_defined = resultant_sq > torch.finfo(resultant_sq.dtype).eps
|
| 2955 |
safe_resultant_sq = torch.where(
|
|
|
|
| 2957 |
)
|
| 2958 |
inv_resultant = torch.rsqrt(safe_resultant_sq)
|
| 2959 |
|
|
|
|
|
|
|
|
|
|
| 2960 |
cos_k_half = torch.where(
|
| 2961 |
well_defined,
|
| 2962 |
cos_k_half * inv_resultant,
|
| 2963 |
+
base_cos_half.expand_as(cos_k_half),
|
| 2964 |
)
|
| 2965 |
sin_k_half = torch.where(
|
| 2966 |
well_defined,
|
| 2967 |
sin_k_half * inv_resultant,
|
| 2968 |
+
base_sin_half.expand_as(sin_k_half),
|
| 2969 |
)
|
| 2970 |
cos_k_half = cos_k_half.reshape(B, H_k_eff, S, rot_half)
|
| 2971 |
sin_k_half = sin_k_half.reshape(B, H_k_eff, S, rot_half)
|
|
|
|
| 3574 |
num_key_value_heads=config.num_key_value_heads,
|
| 3575 |
head_dim=self.head_dim,
|
| 3576 |
)
|
| 3577 |
+
# The currently available CCE/Triton REPO-GRAPE kernel implements
|
| 3578 |
+
# the legacy uncentered + uniform + no-prior projection. The model
|
| 3579 |
+
# now centers the spectrum, learns structural GQA weights and adds a
|
| 3580 |
+
# weak canonical circular prior, so that kernel MUST stay disabled
|
| 3581 |
+
# until its forward/backward contract is updated in lockstep.
|
| 3582 |
+
# Otherwise Torch and Triton would silently implement different math.
|
| 3583 |
self.use_repo_grape_triton = False
|
| 3584 |
else:
|
| 3585 |
self.repo_grape = None
|
|
|
|
| 4734 |
# Pure REPO (without GRAPE) keeps the original paper behavior and
|
| 4735 |
# therefore continues to use z directly.
|
| 4736 |
if not _repo_grape_kernel_used:
|
| 4737 |
+
if self.use_repo_grape:
|
| 4738 |
+
# Keep the exact canonical runtime positions alongside u.
|
| 4739 |
+
# They are needed only by the weak circular prior on shared
|
| 4740 |
+
# GQA keys and are never materialized once per query head.
|
| 4741 |
+
repo_coords, repo_reference_positions = (
|
| 4742 |
+
self.repo_grape.transform_positions(
|
| 4743 |
+
z, position_ids, return_reference=True
|
| 4744 |
+
)
|
| 4745 |
+
)
|
| 4746 |
+
else:
|
| 4747 |
+
repo_coords = z
|
| 4748 |
+
repo_reference_positions = None
|
| 4749 |
|
| 4750 |
if not _repo_grape_kernel_used and _iha_seq_expand:
|
| 4751 |
# IHA is applied *after* the coordinate map. Thus REPO-GRAPE
|
|
|
|
| 4763 |
).reshape(repo_coords.shape[0], repo_coords.shape[1], _S_orig * P)
|
| 4764 |
|
| 4765 |
if self.use_repo_grape:
|
| 4766 |
+
# Match the canonical prior to the same IHA pseudo-slot:
|
| 4767 |
+
# u_{i,p}=P*u_i+p => i_{i,p}=P*i+p.
|
| 4768 |
+
reference_expanded = (
|
| 4769 |
+
repo_reference_positions.unsqueeze(-1) * P
|
| 4770 |
+
+ p_offsets.view(1, 1, P)
|
| 4771 |
+
).reshape(repo_reference_positions.shape[0], _S_orig * P)
|
| 4772 |
q, k = self.repo_grape.apply_multiplicative(
|
| 4773 |
+
q,
|
| 4774 |
+
k,
|
| 4775 |
+
coords_expanded,
|
| 4776 |
+
inv_freq,
|
| 4777 |
+
attn_scaling,
|
| 4778 |
+
reference_positions=reference_expanded,
|
| 4779 |
)
|
| 4780 |
else:
|
| 4781 |
q, k = _apply_repo_rope(
|
|
|
|
| 4786 |
# the original REPO-RoPE path.
|
| 4787 |
if self.use_repo_grape:
|
| 4788 |
q, k = self.repo_grape.apply_multiplicative(
|
| 4789 |
+
q,
|
| 4790 |
+
k,
|
| 4791 |
+
repo_coords,
|
| 4792 |
+
inv_freq,
|
| 4793 |
+
attn_scaling,
|
| 4794 |
+
reference_positions=repo_reference_positions,
|
| 4795 |
)
|
| 4796 |
else:
|
| 4797 |
q, k = _apply_repo_rope(
|