| { | |
| "module_name": "factorised_matrix_state", | |
| "class_name": "FactorisedMatrixState", | |
| "category": "attention", | |
| "family": "ssm", | |
| "params": [ | |
| "config", | |
| "rank", | |
| "d_model", | |
| "num_heads" | |
| ], | |
| "ops": [ | |
| "transpose", | |
| "view" | |
| ], | |
| "docstring": "Bidirectional Factorised Matrix State Recurrent Linear Attention.\n\nMathematical Formulation:\n 1. Multi-Head Normalized Projections:\n $$Q = X W_q, \\quad K = X W_k, \\quad V = X W_v$$\n $$\\tilde{q} = \\text{RMSNorm}(Q), \\quad \\tilde{k} = \\text{RMSNorm}(K) / \\sqrt{d_k}$$\n 2. Bidirectional Associative Matrix Recurrence with Bounded Decay:\n $$\\lambda = \\sigma(\\beta) \\in (0, 1)^{B \\times H}$$\n $$\\vec{S}_t = \\lambda \\vec{S}_{t-1} + \\tilde{k}_t \\otimes v_t, \\quad \\vec{z}_t = \\lambda \\vec{z}_{t-1} + \\tilde{k}_t$$\n $$\\overleftarrow{S}_t = \\lambda \\overleftarrow{S}_{t+1} + \\tilde{k}_t \\otimes v_t, \\quad \\overleftarrow{z}_t = \\lambda \\overleftarrow{z}_{t+1} + \\tilde{k}_t$$\n $$\\vec{Y}_t = \\frac{\\vec{S}_t \\tilde{q}_t}{\\vec{z}_t^\\top \\tilde{q}_t + \\epsilon}, \\quad \\overleftarrow{Y}_t = \\frac{\\overleftarrow{S}_t \\tilde{q}_t}{\\overleftarrow{z}_t^\\top \\tilde{q}_t + \\epsilon}$$\n 3. Low-Rank Factorized Bilinear Output Projection:\n $$Y = \\text{Concat}\\left(\\frac{1}{2}(\\vec{Y} + \\overleftarrow{Y})\\right)$$\n $$\\text{Output} = (Y L) R^\\top + Y W_{\\text{res}}$$\n\nComplexity:\n - Time: $\\mathcal{O}(B \\cdot H \\cdot L \\cdot d_k \\cdot d_v + B \\cdot L \\cdot D \\cdot r)$ linear in $L$.\n - Space: $\\mathcal{O}(B \\cdot H \\cdot d_k \\cdot d_v)$ recurrent state memory.", | |
| "rank": 32, | |
| "d_state": 16, | |
| "num_experts": 1, | |
| "top_k": 1, | |
| "sparsity_pct": 0.0, | |
| "bits": 4, | |
| "hash_val": 6878634840653279243932430398172721136 | |
| } |
Xet Storage Details
- Size:
- 1.82 kB
- Xet hash:
- 66f9bd035a8581b2640d1012712cf8e3d8a85f751610d102695400166af2ce4f
·
Xet efficiently stores files, intelligently splitting them into unique chunks and accelerating uploads and downloads. More info.