tahamajs/RR / Docs /module_metadata /hgrn2_outer.json
tahamajs's picture
download
raw
1.91 kB
{
"module_name": "hgrn2_outer",
"class_name": "HGRN2OuterBlock",
"category": "attention",
"family": "ssm",
"params": [
"config",
"rank",
"dim",
"num_heads"
],
"ops": [
"transpose",
"view"
],
"docstring": "HGRN-2 Outer-Product Recurrent Linear Attention Module.\n\nMathematical Mechanism:\n For multi-head queries $Q \\in \\mathbb{R}^{B \\times H \\times L \\times d_k}$, keys $K$, values $V$:\n 1. **Feature Normalization & Bounded Forget Gating:**\n $$\\tilde{q}_t = \\text{RMSNorm}(q_t), \\quad \\tilde{k}_t = \\text{RMSNorm}(k_t) / \\sqrt{d_k}$$\n $$f_t = \\exp(-\\text{softplus}(W_f x_t + b_f)) \\in (0, 1)^{B \\times H}$$\n 2. **Bidirectional Associative Matrix Recurrence:**\n $$\\vec{S}_t = f_t \\odot \\vec{S}_{t-1} + (1 - f_t) \\odot (\\tilde{k}_t \\otimes v_t)$$\n $$\\vec{z}_t = f_t \\odot \\vec{z}_{t-1} + (1 - f_t) \\odot \\tilde{k}_t$$\n $$\\overleftarrow{S}_t = f_t \\odot \\overleftarrow{S}_{t+1} + (1 - f_t) \\odot (\\tilde{k}_t \\otimes v_t)$$\n $$\\overleftarrow{z}_t = f_t \\odot \\overleftarrow{z}_{t+1} + (1 - f_t) \\odot \\tilde{k}_t$$\n 3. **Normalized Readout & Output:**\n $$\\vec{y}_t = \\frac{\\vec{S}_t \\tilde{q}_t}{\\vec{z}_t^\\top \\tilde{q}_t + \\epsilon}, \\quad \\overleftarrow{y}_t = \\frac{\\overleftarrow{S}_t \\tilde{q}_t}{\\overleftarrow{z}_t^\\top \\tilde{q}_t + \\epsilon}$$\n $$\\text{Output} = \\text{Concat}\\left(\\frac{1}{2}(\\vec{Y} + \\overleftarrow{Y})\\right) W_{\\text{out}}$$\n\nComputational Complexity:\n - Time: $\\mathcal{O}(B \\cdot H \\cdot L \\cdot d_k \\cdot d_v)$ linear in sequence length $L$.\n - Space: $\\mathcal{O}(B \\cdot H \\cdot d_k \\cdot d_v)$ recurrent state matrix memory.",
"rank": 32,
"d_state": 16,
"num_experts": 1,
"top_k": 1,
"sparsity_pct": 0.0,
"bits": 4,
"hash_val": 153656948164252003342201920734471408603
}

Xet Storage Details

Size:
1.91 kB
·
Xet hash:
282a465c94fda1101722fa12a4c29751e3523ab4044d5de6a09f3e49d653cc0b

Xet efficiently stores files, intelligently splitting them into unique chunks and accelerating uploads and downloads. More info.