| { | |
| "module_name": "hgrn2_layer", | |
| "class_name": "HGRN2Layer", | |
| "category": "attention", | |
| "family": "ssm", | |
| "params": [ | |
| "config", | |
| "rank", | |
| "d_model", | |
| "res_clip" | |
| ], | |
| "ops": [ | |
| "silu" | |
| ], | |
| "docstring": "Hierarchically Gated Recurrent Network v2 (HGRN-2) Bidirectional Layer.\n\nMathematical Mechanism:\n For input token activations $x_t \\in \\mathbb{R}^D$:\n 1. **Dynamic Gating & Value Transformation:**\n $$f_t = \\exp(-\\text{softplus}(W_f x_t + b_f)) \\in (0, 1)^D \\quad \\text{(Log-space Bounded Retention)}$$\n $$i_t = \\sigma(W_i x_t) \\in (0, 1)^D \\quad \\text{(Input Gate)}$$\n $$v_t = W_v x_t \\in \\mathbb{R}^D, \\quad z_t = W_g x_t \\in \\mathbb{R}^D$$\n 2. **Bidirectional Linear Recurrence:**\n $$\\vec{h}_t = f_t \\odot \\vec{h}_{t-1} + i_t \\odot v_t$$\n $$\\overleftarrow{h}_t = f_t \\odot \\overleftarrow{h}_{t+1} + i_t \\odot v_t$$\n 3. **Gated Output Aggregation:**\n $$Y = \\frac{1}{2}(\\vec{h} + \\overleftarrow{h}) \\odot \\text{SiLU}(z)$$\n $$\\text{Output} = Y W_o + b_o$$\n\nComputational Complexity:\n - Time: $\\mathcal{O}(B \\cdot L \\cdot D)$ \u2014 strictly linear in sequence length $L$.\n - Space: $\\mathcal{O}(B \\cdot D)$ recurrent state memory per sequence.", | |
| "rank": 32, | |
| "d_state": 16, | |
| "num_experts": 1, | |
| "top_k": 1, | |
| "sparsity_pct": 0.0, | |
| "bits": 4, | |
| "hash_val": 153848104064096316253623281263225148722 | |
| } |
Xet Storage Details
- Size:
- 1.44 kB
- Xet hash:
- 2be721016b81edf632edf4fb6447a1dca841c3a66ca1c7b1ea297ff5d7fa6b4f
·
Xet efficiently stores files, intelligently splitting them into unique chunks and accelerating uploads and downloads. More info.