| { | |
| "module_name": "hgrn2_outer", | |
| "class_name": "HGRN2OuterBlock", | |
| "category": "attention", | |
| "family": "ssm", | |
| "params": [ | |
| "config", | |
| "rank", | |
| "dim", | |
| "num_heads" | |
| ], | |
| "ops": [ | |
| "transpose", | |
| "view" | |
| ], | |
| "docstring": "HGRN-2 Outer-Product Recurrent Linear Attention Module.\n\nMathematical Mechanism:\n For multi-head queries $Q \\in \\mathbb{R}^{B \\times H \\times L \\times d_k}$, keys $K$, values $V$:\n 1. **Feature Normalization & Bounded Forget Gating:**\n $$\\tilde{q}_t = \\text{RMSNorm}(q_t), \\quad \\tilde{k}_t = \\text{RMSNorm}(k_t) / \\sqrt{d_k}$$\n $$f_t = \\exp(-\\text{softplus}(W_f x_t + b_f)) \\in (0, 1)^{B \\times H}$$\n 2. **Bidirectional Associative Matrix Recurrence:**\n $$\\vec{S}_t = f_t \\odot \\vec{S}_{t-1} + (1 - f_t) \\odot (\\tilde{k}_t \\otimes v_t)$$\n $$\\vec{z}_t = f_t \\odot \\vec{z}_{t-1} + (1 - f_t) \\odot \\tilde{k}_t$$\n $$\\overleftarrow{S}_t = f_t \\odot \\overleftarrow{S}_{t+1} + (1 - f_t) \\odot (\\tilde{k}_t \\otimes v_t)$$\n $$\\overleftarrow{z}_t = f_t \\odot \\overleftarrow{z}_{t+1} + (1 - f_t) \\odot \\tilde{k}_t$$\n 3. **Normalized Readout & Output:**\n $$\\vec{y}_t = \\frac{\\vec{S}_t \\tilde{q}_t}{\\vec{z}_t^\\top \\tilde{q}_t + \\epsilon}, \\quad \\overleftarrow{y}_t = \\frac{\\overleftarrow{S}_t \\tilde{q}_t}{\\overleftarrow{z}_t^\\top \\tilde{q}_t + \\epsilon}$$\n $$\\text{Output} = \\text{Concat}\\left(\\frac{1}{2}(\\vec{Y} + \\overleftarrow{Y})\\right) W_{\\text{out}}$$\n\nComputational Complexity:\n - Time: $\\mathcal{O}(B \\cdot H \\cdot L \\cdot d_k \\cdot d_v)$ linear in sequence length $L$.\n - Space: $\\mathcal{O}(B \\cdot H \\cdot d_k \\cdot d_v)$ recurrent state matrix memory.", | |
| "rank": 32, | |
| "d_state": 16, | |
| "num_experts": 1, | |
| "top_k": 1, | |
| "sparsity_pct": 0.0, | |
| "bits": 4, | |
| "hash_val": 153656948164252003342201920734471408603 | |
| } |
Xet Storage Details
- Size:
- 1.91 kB
- Xet hash:
- 282a465c94fda1101722fa12a4c29751e3523ab4044d5de6a09f3e49d653cc0b
·
Xet efficiently stores files, intelligently splitting them into unique chunks and accelerating uploads and downloads. More info.