tahamajs/RR / Docs /module_metadata /hgrn2_layer.json
tahamajs's picture
download
raw
1.44 kB
{
"module_name": "hgrn2_layer",
"class_name": "HGRN2Layer",
"category": "attention",
"family": "ssm",
"params": [
"config",
"rank",
"d_model",
"res_clip"
],
"ops": [
"silu"
],
"docstring": "Hierarchically Gated Recurrent Network v2 (HGRN-2) Bidirectional Layer.\n\nMathematical Mechanism:\n For input token activations $x_t \\in \\mathbb{R}^D$:\n 1. **Dynamic Gating & Value Transformation:**\n $$f_t = \\exp(-\\text{softplus}(W_f x_t + b_f)) \\in (0, 1)^D \\quad \\text{(Log-space Bounded Retention)}$$\n $$i_t = \\sigma(W_i x_t) \\in (0, 1)^D \\quad \\text{(Input Gate)}$$\n $$v_t = W_v x_t \\in \\mathbb{R}^D, \\quad z_t = W_g x_t \\in \\mathbb{R}^D$$\n 2. **Bidirectional Linear Recurrence:**\n $$\\vec{h}_t = f_t \\odot \\vec{h}_{t-1} + i_t \\odot v_t$$\n $$\\overleftarrow{h}_t = f_t \\odot \\overleftarrow{h}_{t+1} + i_t \\odot v_t$$\n 3. **Gated Output Aggregation:**\n $$Y = \\frac{1}{2}(\\vec{h} + \\overleftarrow{h}) \\odot \\text{SiLU}(z)$$\n $$\\text{Output} = Y W_o + b_o$$\n\nComputational Complexity:\n - Time: $\\mathcal{O}(B \\cdot L \\cdot D)$ \u2014 strictly linear in sequence length $L$.\n - Space: $\\mathcal{O}(B \\cdot D)$ recurrent state memory per sequence.",
"rank": 32,
"d_state": 16,
"num_experts": 1,
"top_k": 1,
"sparsity_pct": 0.0,
"bits": 4,
"hash_val": 153848104064096316253623281263225148722
}

Xet Storage Details

Size:
1.44 kB
·
Xet hash:
2be721016b81edf632edf4fb6447a1dca841c3a66ca1c7b1ea297ff5d7fa6b4f

Xet efficiently stores files, intelligently splitting them into unique chunks and accelerating uploads and downloads. More info.