| { | |
| "total_count": 252, | |
| "generated_at": "2026-09-12", | |
| "modules": [ | |
| { | |
| "id": "SOAFusedAttention", | |
| "name": "SOA Fused Attention: Low-Rank Latent Space Multi-Head Attention w", | |
| "file": "Codes/framework/attentions/SOAFusedAttention.py", | |
| "type": "Attention Module", | |
| "category": "Low-Rank SVD & Factorization", | |
| "paradigm": "svd", | |
| "icon": "\u26a1", | |
| "description": "SOA Fused Attention: Low-Rank Latent Space Multi-Head Attention with Zero-Shot SVD Initialization \u2013 Production-Hardened. Factorizes dense Q, K, and V linear projection operators into separate low-rank bottleneck pairs initialized from truncated Singular Value Decomposition (SVD) of source attention weights. Supports: - Multi-architecture copy_wei", | |
| "math": "W \\approx U_r \\Sigma_r V_r^\\top = A B^\\top, \\quad r \\ll \\min(d_{in}, d_{out})", | |
| "markdown_preview": "# SOA-Fused Attention: Low-Rank Latent Space Multi-Head Attention with Zero-Shot SVD Initialization\n\n## 1. Executive Abstract & Architectural Motivation\n\nDense Multi-Head Self-Attention layers allocate distinct projection matrices for Query, Key, and Value ($W_Q, W_K, W_V \\in \\mathbb{R}^{D \\times D}$). In standard architectures, these matrices contain significant spectral redundancy, with $>80\\%$ of their Frobenius energy concentrated in low-rank singular subspaces.\n\n**SOA-Fused Attention (`SOAFusedAttention`)** factorizes each dense projection matrix into a pair of low-rank bottleneck operato" | |
| }, | |
| { | |
| "id": "adaptive_span_attention", | |
| "name": "Adaptive Span Attention: Dynamic Head-Specific Context Window Gat", | |
| "file": "Codes/framework/attentions/adaptive_span_attention.py", | |
| "type": "Attention Module", | |
| "category": "Dynamic & Efficient Attention", | |
| "paradigm": "sparse", | |
| "icon": "\ud83c\udf1f", | |
| "description": "Adaptive Span Attention Module (Sukhbaatar et al., 2019). Mathematical Formulation: 1. Query-Key Attention Scores: $$S_{ij} = \\frac{Q_i K_j^\\top}{\\sqrt{d}}$$ 2. Dynamic Per-Head Span Masking (Continuous Soft Ramp): Let $z_h = \\sigma(\\text{logit}_h) \\cdot S_{\\max}$, and $R$ be ramp size. $$g_{z_h}(d) = \\operatorname{cl", | |
| "math": "S_{ij} = \\frac{Q_i K_j^\\top}{\\sqrt{d}}", | |
| "markdown_preview": "# Adaptive Span Attention: Dynamic Head-Specific Context Window Gating\n\n## 1. Executive Abstract & Architectural Motivation\n\nAdaptive Span Attention, introduced by Sukhbaatar et al. (NeurIPS 2019), addresses the dynamic trade-off between long-context modeling and computational efficiency in Transformer architectures. Standard Multi-Head Attention forces every attention head to compute pairwise interactions across the entire sequence length $N$, leading to quadratic computational ($\\mathcal{O}(N^2)$) and memory complexity across all heads. However, empirical studies reveal that different heads " | |
| }, | |
| { | |
| "id": "adaptive_span_inject", | |
| "name": "Adaptive Span Attention: Dynamic Head-Specific Context Window Gat", | |
| "file": "Codes/framework/attentions/adaptive_span_inject.py", | |
| "type": "Attention Module", | |
| "category": "Dynamic & Efficient Attention", | |
| "paradigm": "sparse", | |
| "icon": "\ud83c\udf1f", | |
| "description": "Deprecated stub. Re-exports base module for registry compatibility.", | |
| "math": "S_{ij} = \\frac{Q_i K_j^\\top}{\\sqrt{d}}", | |
| "markdown_preview": "# Adaptive Span Attention: Dynamic Head-Specific Context Window Gating\n\n## 1. Executive Abstract & Architectural Motivation\n\nAdaptive Span Attention, introduced by Sukhbaatar et al. (NeurIPS 2019), addresses the dynamic trade-off between long-context modeling and computational efficiency in Transformer architectures. Standard Multi-Head Attention forces every attention head to compute pairwise interactions across the entire sequence length $N$, leading to quadratic computational ($\\mathcal{O}(N^2)$) and memory complexity across all heads. However, empirical studies reveal that different heads " | |
| }, | |
| { | |
| "id": "adaptive_sparse_attention", | |
| "name": "Adaptive Sparse Attention: Dynamic Top-K & Threshold-Based Sparsi", | |
| "file": "Codes/framework/attentions/adaptive_sparse_attention.py", | |
| "type": "Attention Module", | |
| "category": "Dynamic & Efficient Attention", | |
| "paradigm": "sparse", | |
| "icon": "\ud83c\udf1f", | |
| "description": "Adaptive Sparse Attention Module (Production-Hardened). Implements dynamic top-k and threshold-based adaptive sparse attention with LoRA adapters, multi-architecture weight extraction, RoPE rotary embeddings, causal/bidirectional masking, and safe injection.", | |
| "math": "\\gamma_h = \\sigma(\\text{logit}_h) \\in (0, 1)", | |
| "markdown_preview": "# Adaptive Sparse Attention: Dynamic Top-K & Threshold-Based Sparsity Routing\n\n## 1. Overview & Architectural Motivation\n\nStandard Multi-Head Attention (MHA) exhibits quadratic computational and memory complexity $\\mathcal{O}(L^2)$ with respect to sequence length $L$, computing dense pairwise compatibility scores for every query-key token pair. In deep transformers, attention probability distributions are empirically sparse: a small fraction of key tokens account for the vast majority of probability mass, while the remaining tokens contribute negligible context and accumulate numerical noise.\n" | |
| }, | |
| { | |
| "id": "adaptive_sparse_inject", | |
| "name": "Adaptive Sparse Attention: Dynamic Top-K & Threshold-Based Sparsi", | |
| "file": "Codes/framework/attentions/adaptive_sparse_inject.py", | |
| "type": "Attention Module", | |
| "category": "Dynamic & Efficient Attention", | |
| "paradigm": "sparse", | |
| "icon": "\ud83c\udf1f", | |
| "description": "Deprecated stub. Re-exports base module for registry compatibility.", | |
| "math": "\\gamma_h = \\sigma(\\text{logit}_h) \\in (0, 1)", | |
| "markdown_preview": "# Adaptive Sparse Attention: Dynamic Top-K & Threshold-Based Sparsity Routing\n\n## 1. Overview & Architectural Motivation\n\nStandard Multi-Head Attention (MHA) exhibits quadratic computational and memory complexity $\\mathcal{O}(L^2)$ with respect to sequence length $L$, computing dense pairwise compatibility scores for every query-key token pair. In deep transformers, attention probability distributions are empirically sparse: a small fraction of key tokens account for the vast majority of probability mass, while the remaining tokens contribute negligible context and accumulate numerical noise.\n" | |
| }, | |
| { | |
| "id": "adaptive_svd_attention", | |
| "name": "Adaptive SVD Attention: Low-Rank SVD Projections with Parallel Re", | |
| "file": "Codes/framework/attentions/adaptive_svd_attention.py", | |
| "type": "Attention Module", | |
| "category": "Low-Rank SVD & Factorization", | |
| "paradigm": "svd", | |
| "icon": "\u26a1", | |
| "description": "Adaptive SVD Attention with Residual Adapters and Dynamic Energy Thresholding. This module provides low-rank factorized linear projections for Multi-Head Attention, decomposing full-rank weight matrices via dynamic cumulative energy-thresholded Singular Value Decomposition (SVD): $$\\frac{\\sum_{i=1}^r \\sigma_i^2}{\\sum_{j=1}^d \\sigma_j^2} \\ge \\tau \\", | |
| "math": "W = U \\Sigma V^\\top = \\sum_{i=1}^{\\min(d_{\\text{out}}, d_{\\text{in}})} \\sigma_i u_i v_i^\\top", | |
| "markdown_preview": "# Adaptive SVD Attention: Low-Rank SVD Projections with Parallel Residual Adapters\n\n## 1. Executive Abstract & Architectural Motivation\n\nAdaptive SVD Attention (`AdaptiveSVDAttention`) provides a hybrid parameter-efficient adaptation architecture for Transformer attention modules. Standard model compression factorizes dense projection matrices ($W_Q, W_K, W_V, W_O \\in \\mathbb{R}^{D \\times D}$) via truncated Singular Value Decomposition ($W_{\\text{svd}} \\approx U V$). However, freezing low-rank SVD components without fine-tuning limits domain adaptation performance.\n\n`AdaptiveSVDAttention` augm" | |
| }, | |
| { | |
| "id": "adasplash_attention", | |
| "name": "AdaSplash Attention: Adaptive Splash Attention with alpha-Entmax ", | |
| "file": "Codes/framework/attentions/adasplash_attention.py", | |
| "type": "Attention Module", | |
| "category": "Mixture of Experts & Routing", | |
| "paradigm": "moe", | |
| "icon": "\ud83d\udd00", | |
| "description": "AdaSplash Attention: Adaptive Splash Attention with \u03b1-Entmax Sparse Activations. Combines localized sliding-window splash radius with sparse \u03b1-entmax adaptive thresholding and global token routing, ensuring robust numerical convergence and exact pretrained weight transfer.", | |
| "math": "S_{ij} = \\frac{Q_i K_j^\\top}{\\sqrt{d_h}}", | |
| "markdown_preview": "# AdaSplash Attention: Adaptive Splash Attention with alpha-Entmax Sparse Activations\n\n## 1. Executive Abstract & Architectural Motivation\n\nStandard Multi-Head Attention employs the Softmax activation function, which maps unbounded dot-product affinities $s_{ij} = \\frac{q_i k_j^\\top}{\\sqrt{d_h}}$ into a probability distribution:\n\\[\n\\operatorname{softmax}(s)_i = \\frac{\\exp(s_i)}{\\sum_{j=1}^L \\exp(s_j)} > 0, \\quad \\forall i \\in \\{1, \\dots, L\\}\n\\]\nBecause the exponential function is strictly positive on the entire real line ($\\exp(x) > 0, \\forall x \\in \\mathbb{R}$), Softmax forces every single to" | |
| }, | |
| { | |
| "id": "adasplash_inject", | |
| "name": "AdaSplash Attention: Adaptive Splash Attention with alpha-Entmax ", | |
| "file": "Codes/framework/attentions/adasplash_inject.py", | |
| "type": "Attention Module", | |
| "category": "Dynamic & Efficient Attention", | |
| "paradigm": "sparse", | |
| "icon": "\ud83c\udf1f", | |
| "description": "Deprecated stub. Re-exports base module for registry compatibility.", | |
| "math": "S_{ij} = \\frac{Q_i K_j^\\top}{\\sqrt{d_h}}", | |
| "markdown_preview": "# AdaSplash Attention: Adaptive Splash Attention with alpha-Entmax Sparse Activations\n\n## 1. Executive Abstract & Architectural Motivation\n\nStandard Multi-Head Attention employs the Softmax activation function, which maps unbounded dot-product affinities $s_{ij} = \\frac{q_i k_j^\\top}{\\sqrt{d_h}}$ into a probability distribution:\n\\[\n\\operatorname{softmax}(s)_i = \\frac{\\exp(s_i)}{\\sum_{j=1}^L \\exp(s_j)} > 0, \\quad \\forall i \\in \\{1, \\dots, L\\}\n\\]\nBecause the exponential function is strictly positive on the entire real line ($\\exp(x) > 0, \\forall x \\in \\mathbb{R}$), Softmax forces every single to" | |
| }, | |
| { | |
| "id": "aegis_hybrid", | |
| "name": "AEGIS-Hybrid Attention: Low-Rank Factorized Hybrid Attention with", | |
| "file": "Codes/framework/attentions/aegis_hybrid.py", | |
| "type": "Attention Module", | |
| "category": "Low-Rank SVD & Factorization", | |
| "paradigm": "svd", | |
| "icon": "\u26a1", | |
| "description": "Aegis-Hybrid Attention Mechanism. Combines low-rank factorized projections, localized sliding-window dense attention, causal/bidirectional kernelized global linear attention with token-importance pruning, and learned gating fusion.", | |
| "math": "O_t = \\alpha_t \\odot O_t^{\\text{local}} + (1 - \\alpha_t) \\odot O_t^{\\text{global}}", | |
| "markdown_preview": "# AEGIS-Hybrid Attention: Low-Rank Factorized Hybrid Attention with Adaptive Gated Fusion and LoRA Recovery\n\n## 1. Executive Abstract & Architectural Motivation\n\nStandard Multi-Head Attention (MHA) architectures suffer from two fundamental bottlenecks that constrain their scalability and parameter efficiency:\n1. **Quadratic Computational & Memory Complexity ($\\mathcal{O}(L^2)$)**: Dense token-to-token all-pairs dot-product attention computes an affinity matrix of size $L \\times L$. For long document reasoning and agentic workflows, memory footprint scales quadratically, quickly exceeding hardw" | |
| }, | |
| { | |
| "id": "aegis_hybrid_attention", | |
| "name": "Compatibility shim for aegis_hybrid_attention.", | |
| "file": "Codes/framework/attentions/aegis_hybrid_attention.py", | |
| "type": "Attention Module", | |
| "category": "Dynamic & Efficient Attention", | |
| "paradigm": "sparse", | |
| "icon": "\ud83c\udf1f", | |
| "description": "Compatibility shim for aegis_hybrid_attention.", | |
| "math": "A_{ij} = \\text{TopK}\\left(\\text{Softmax}\\left(\\frac{Q_i K_j^\\top}{\\sqrt{d}}\\right)\\right)", | |
| "markdown_preview": "# Aegis Hybrid Attention\n\n## Overview\n## Configuration & Hyperparameters\n\n| Parameter | Description |\n|-----------|-------------|\n\n## Usage Examples\n\n```python\nfrom Codes.framework.attentions import Module\n```\n\n## Benchmark Results\n\n| Status | Accuracy | Val Loss | Peak VRAM | Inference Latency |\n|--------|----------|----------|-----------|-------------------|\n| Pending| Pending | Pending | Pending | Pending |\n\n## Limitations & Future Work\n\nTODO: Identify specific limitations and edge cases for this module.\n\n## Empirical Benchmark Results\n\n| Model | Dataset | Status | Baseline Ac" | |
| }, | |
| { | |
| "id": "aegis_spectron_fp8", | |
| "name": "AEGIS-Spectron-FP8: Tri-Branch Hybrid Attention with Token-Level ", | |
| "file": "Codes/framework/attentions/aegis_spectron_fp8.py", | |
| "type": "Attention Module", | |
| "category": "Low-Rank SVD & Factorization", | |
| "paradigm": "svd", | |
| "icon": "\u26a1", | |
| "description": "Aegis-Spectron-FP8 Hybrid Attention Module. Integrates Aegis multi-branch local/global feature fusion with Spectron Lagrangian spectral entropy routing. Dynamic routing measures singular value entropy per token to distribute token representation across: 1. Local Sliding-Window Attention (O(N * W) with strict causal masking for decoders) 2. Global ", | |
| "math": "p_{t, d} = \\frac{x_{t, d}^2}{\\sum_{k=1}^D x_{t, k}^2 + \\epsilon}, \\quad d \\in \\{1, \\dots, D\\}", | |
| "markdown_preview": "# AEGIS-Spectron-FP8: Tri-Branch Hybrid Attention with Token-Level Spectral Entropy Routing and True $\\mathcal{O}(N)$ Linear Complexity\n\n## 1. Executive Summary & Architectural Motivation\n\nStandard Multi-Head Attention (MHA) has served as the foundational pillar of modern Transformer architectures. However, MHA suffers from two critical, interrelated limitations:\n1. **Quadratic Scaling Bottleneck ($\\mathcal{O}(N^2)$)**: Computing explicit pairwise affinity matrices across sequence length $N$ induces quadratic time and memory explosion, bottlenecking context expansion, long-document understandi" | |
| }, | |
| { | |
| "id": "aegis_v1", | |
| "name": "AEGIS-Attention V1: Adaptive Enhanced Global-Local Integrated Spa", | |
| "file": "Codes/framework/attentions/aegis_v1.py", | |
| "type": "Attention Module", | |
| "category": "Low-Rank SVD & Factorization", | |
| "paradigm": "svd", | |
| "icon": "\u26a1", | |
| "description": "AEGIS-Attention V1: Adaptive Enhanced Global-Local Integrated Sparse Attention Baseline. Combines SVD-factorized projection matrices with energy-based rank selection and LoRA adapters, split-head local sliding-window attention with smooth activations, and global linear kernel attention with adaptive gating. Hardened for multi-architecture deployme", | |
| "math": "W = U \\Sigma V^\\top = \\sum_{k=1}^{\\min(d_{\\text{in}}, d_{\\text{out}})} \\sigma_k u_k v_k^\\top", | |
| "markdown_preview": "# AEGIS-Attention V1: Adaptive Enhanced Global-Local Integrated Sparse Attention Baseline\n\n## 1. Executive Abstract & Theoretical Foundations\n\nStandard Transformer Multi-Head Attention (MHA) computes all-to-all token interactions with quadratic time and memory complexity $\\mathcal{O}(L^2)$, imposing prohibitive computational burdens on long sequences. **AEGIS-Attention V1** (Adaptive Enhanced Global-Local Integrated Sparse Attention Baseline) replaces dense quadratic attention with an integrated hybrid formulation combining:\n1. **Truncated SVD Low-Rank Projections**: Decomposes full-rank weigh" | |
| }, | |
| { | |
| "id": "aegis_v2", | |
| "name": "AEGIS-Attention V2: Low-Rank Factorized Hybrid Attention with Ada", | |
| "file": "Codes/framework/attentions/aegis_v2.py", | |
| "type": "Attention Module", | |
| "category": "Low-Rank SVD & Factorization", | |
| "paradigm": "svd", | |
| "icon": "\u26a1", | |
| "description": "Aegis-Attention V2 Module. Implements hybrid low-rank factorized attention with local sliding window attention, global linear kernel attention with adaptive soft-pruning gating, and dynamic multi-head fusion. Hardened for multi-architecture deployment across DistilBERT, RoBERTa, GPT-2, and Qwen3.5.", | |
| "math": "Q = X W_{Q1} W_{Q2}, \\quad K = X W_{K1} W_{K2}, \\quad V = X W_{V1} W_{V2}", | |
| "markdown_preview": "# AEGIS-Attention V2: Low-Rank Factorized Hybrid Attention with Adaptive Soft-Pruning\n\n## 1. Executive Abstract & Theoretical Foundations\n\n**AEGIS-Attention V2** introduces low-rank factorized projections, localized sliding-window attention, global linear kernel attention with adaptive soft-pruning token gating, and dynamic multi-head gated fusion. \n\nKey architectural innovations include:\n1. **Low-Rank Factorized Linear Layers**: Decomposes all projection matrices ($W_Q, W_K, W_V, W_O$) into rank-$r$ bottlenecks ($W_1 W_2$) initialized via truncated SVD.\n2. **Local Sliding-Window Softmax Branc" | |
| }, | |
| { | |
| "id": "aegis_v3", | |
| "name": "AEGIS-Attention V3: Adaptive Enhanced Global-Local Integrated Spa", | |
| "file": "Codes/framework/attentions/aegis_v3.py", | |
| "type": "Attention Module", | |
| "category": "Low-Rank SVD & Factorization", | |
| "paradigm": "svd", | |
| "icon": "\u26a1", | |
| "description": "AEGIS-Attention V3: Adaptive Enhanced Global-Local Integrated Sparse Attention Baseline. Combines SVD-factorized projection matrices with energy-based rank selection and LoRA adapters, split-head local sliding-window attention with smooth activations, and global linear kernel attention with adaptive gating. Hardened for multi-architecture deployme", | |
| "math": "e_j = \\exp\\left(\\frac{x_j - m}{\\tau}\\right), \\quad m_{\\text{scaled}} = \\operatorname{clamp}\\left(-\\frac{m}{\\tau}, -\\infty, 80.0\\right)", | |
| "markdown_preview": "# AEGIS-Attention V3: Energy-Constrained SVD & Softpick Rectified Sparse Attention\n\n## 1. Executive Abstract & Theoretical Foundations\n\n**AEGIS-Attention V3** unifies dynamic energy-constrained truncated SVD projections, split-head local sliding-window attention with **Softpick rectified sparse activations**, and global Random Fourier Feature (RFF) linear kernel attention with adaptive per-head gating.\n\nKey architectural mechanisms:\n1. **Dynamic Energy-Constrained SVD**: Selects minimal projection rank $r \\in [4, 64]$ to satisfy cumulative singular value energy ratio $\\ge 95\\%$.\n2. **Softpick " | |
| }, | |
| { | |
| "id": "aegis_v4", | |
| "name": "AEGIS-Attention V4: Adaptive Enhanced Global-Local Integrated Spa", | |
| "file": "Codes/framework/attentions/aegis_v4.py", | |
| "type": "Attention Module", | |
| "category": "Low-Rank SVD & Factorization", | |
| "paradigm": "svd", | |
| "icon": "\u26a1", | |
| "description": "AEGIS-Attention V4: Adaptive Enhanced Global-Local Integrated Sparse Attention Baseline. Combines SVD-factorized projection matrices with energy-based rank selection and LoRA adapters, split-head local sliding-window attention with smooth activations, and global linear kernel attention with adaptive gating. Hardened for multi-architecture deployme", | |
| "math": "Q = \\text{FactorisedLinear}_Q(X), \\quad K = \\text{FactorisedLinear}_K(X), \\quad V = \\text{FactorisedLinear}_V(X)", | |
| "markdown_preview": "# AEGIS-Attention V4: Adaptive Enhanced Global-Local Integrated Sparse Attention\n\n## 1. Executive Abstract & Architectural Principles\n\n**AEGIS-Attention V4** provides a unified multi-head attention layer combining:\n1. **SVD-Factorized Projections with Energy-Based Rank Selection**: Dynamically calculates per-projection rank $r \\in [4, 64]$ to preserve $\\ge 95\\%$ singular value energy.\n2. **Split-Head Local Sliding-Window Attention**: Softpick rectified non-linearities with window size $W = 64$ for $H_l = 8$ local heads.\n3. **Global Linear RFF Kernel Attention**: 256-dimensional random Fourier " | |
| }, | |
| { | |
| "id": "aegis_v5", | |
| "name": "AEGIS-Attention V5: Adaptive Enhanced Global-Local Integrated Spa", | |
| "file": "Codes/framework/attentions/aegis_v5.py", | |
| "type": "Attention Module", | |
| "category": "Low-Rank SVD & Factorization", | |
| "paradigm": "svd", | |
| "icon": "\u26a1", | |
| "description": "AEGIS-Attention V5: Adaptive Enhanced Global-Local Integrated Sparse Attention Baseline. Combines SVD-factorized projection matrices with energy-based rank selection and LoRA adapters, split-head local sliding-window attention with smooth activations, and global linear kernel attention with adaptive gating. Hardened for multi-architecture deployme", | |
| "math": "Q_{\\text{feat}} = \\operatorname{ELU}(Q_g \\Omega) + 1.0 \\in \\mathbb{R}^{B \\times H_g \\times L \\times F}", | |
| "markdown_preview": "# AEGIS-Attention V5: Hybrid Multi-Head Attention with ELU+1 Feature Maps & Safe Masking\n\n## 1. Executive Abstract & Theoretical Foundations\n\n**AEGIS-Attention V5** is a hybrid efficient attention module that decouples multi-head attention into $H_l = 8$ local sliding-window heads with Softpick rectified sparse non-linearities and $H_g = 4$ global linear kernel attention heads with **learnable ELU+1 feature maps** and safe mask propagation.\n\nKey architectural highlights:\n1. **Low-Rank SVD Projections**: Factorizes linear projections $W \\approx W_{\\text{up}} W_{\\text{down}}$ with energy thresho" | |
| }, | |
| { | |
| "id": "aegis_v6", | |
| "name": "AEGIS-Attention V6: Unified SVD-LoRA Hybrid Attention with Orthog", | |
| "file": "Codes/framework/attentions/aegis_v6.py", | |
| "type": "Attention Module", | |
| "category": "Low-Rank SVD & Factorization", | |
| "paradigm": "svd", | |
| "icon": "\u26a1", | |
| "description": "AEGIS-Attention V6: Unified Hybrid Transformer Attention Module. Combines SVD low-rank factorized projections with LoRA adapters, sliding window attention, global linear kernel attention with adaptive spectral gating, and context fusion initialized to prioritize high-fidelity local attention. Hardened for multi-architecture deployment across Disti", | |
| "math": "y = x (V \\Sigma U^\\top) + x (B^\\top A^\\top) + b", | |
| "markdown_preview": "# AEGIS-Attention V6: Unified SVD-LoRA Hybrid Attention with Orthogonal Kernel Projections\n\n## 1. Executive Abstract & Theoretical Foundations\n\n**AEGIS-Attention V6** unifies:\n1. **Factorized Low-Rank SVD-LoRA Projections**: Combines frozen truncated SVD bases ($V \\Sigma U^\\top$) with trainable rank-16 LoRA adapters ($B^\\top A^\\top$).\n2. **Stable Orthogonal Kernel Feature Mapping**: Computes positive RFF maps with QR-orthogonalized projection matrices and Gaussian quadratic norm regularization: $\\phi(x) = \\frac{1}{\\sqrt{m}} \\exp(x \\Omega - \\frac{\\|x\\|^2}{2})$.\n3. **Dual Query & Input Gating**:" | |
| }, | |
| { | |
| "id": "asvd_attention", | |
| "name": "ASVD Attention: Activation-Aware Singular Value Decomposition Arc", | |
| "file": "Codes/framework/attentions/asvd_attention.py", | |
| "type": "Attention Module", | |
| "category": "Low-Rank SVD & Factorization", | |
| "paradigm": "svd", | |
| "icon": "\u26a1", | |
| "description": "ASVD (Activation-Aware Singular Value Decomposition) Attention. Paper: https://arxiv.org/abs/2307.07382 Hardened for production use across diverse transformer architectures: - DistilBERT (Flat MHA) - RoBERTa / BERT (Nested Encoder Self-Attention) - GPT-2 (Causal Decoder with Conv1D) - Qwen / LLaMA (GQA with Rotary Position Embeddings)", | |
| "math": "y = ((x V^\\top) \\odot S) U^\\top + (x A^\\top) B^\\top + b", | |
| "markdown_preview": "# ASVD Attention: Activation-Aware Singular Value Decomposition Architecture\n\n## 1. Executive Abstract & Architectural Motivation\n\nActivation-aware Singular Value Decomposition Attention (`ASVDAttention`, Yuan et al., 2023) provides an activation-guided low-rank model compression framework for Multi-Head Attention. Standard SVD parameter reduction factorizes dense projection matrices $W \\in \\mathbb{R}^{d_{\\text{out}} \\times d_{\\text{in}}}$ solely based on static weight magnitude matrices, ignoring the non-uniform variance and magnitude distributions of dynamic activation vectors ($X$).\n\nASVD r" | |
| }, | |
| { | |
| "id": "bam_umoe", | |
| "name": "BAM-uMoE: Bidirectional Attentive Mixture-of-Unified-Experts", | |
| "file": "Codes/framework/attentions/bam_umoe.py", | |
| "type": "Attention Module", | |
| "category": "Mixture of Experts & Routing", | |
| "paradigm": "moe", | |
| "icon": "\ud83d\udd00", | |
| "description": "BAM-UMoE: Unified Mixture of Experts Attention Block. This module implements a production-hardened Unified Mixture of Experts (UMoE) attention mechanism supporting multi-architecture deployments: DistilBERT, RoBERTa/BERT, GPT-2, and Qwen/LLaMA/Mistral (GQA + partial RoPE). Key architectural features: 1. Robust Weight Extraction across fused Conv1", | |
| "math": "\\mathbf{Q} = \\mathbf{X} \\mathbf{W}_Q, \\quad \\mathbf{K} = \\mathbf{X} \\mathbf{W}_K, \\quad \\mathbf{V}_{\\text{base}} = \\mathbf{X} \\mathbf{W}_V", | |
| "markdown_preview": "# BAM-uMoE: Bidirectional Attentive Mixture-of-Unified-Experts\n\n## 1. Mathematical Formulation & Architectural Design\n\nThe **BAM-uMoE** (Bidirectional Attentive Mixture-of-Unified-Experts) architecture synergistically couples dense bidirectional multi-head self-attention with dual-stage conditional parameter execution. Rather than replacing attention with sparsely routed routing entirely, BAM-uMoE introduces Mixture-of-Experts (MoE) at two strategic calculation manifolds:\n1. **Value Modulation Branch ($\\text{uMoE}_{QKV}$)**: Contextual refinement of value representations before dot-product agg" | |
| }, | |
| { | |
| "id": "block_transformer_hiba", | |
| "name": "Block Transformer with Hierarchical Intra-Block and Inter-Block A", | |
| "file": "Codes/framework/attentions/block_transformer_hiba.py", | |
| "type": "Attention Module", | |
| "category": "Latent & KV-Cache Compression", | |
| "paradigm": "mla", | |
| "icon": "\ud83e\udde0", | |
| "description": "Block Transformer with Hierarchical Intra-Block and Inter-Block Attention (HIBA). Production-hardened implementation supporting: 1. Robust Pretrained Weight Extraction across DistilBERT, RoBERTa/BERT nested encoders, GPT-2 (fused Conv1D), and Qwen/LLaMA/Mistral. 2. Nested Encoder Detection to prevent double output projection signal attenuation ", | |
| "math": "\\mathbf{Q}_b = \\mathbf{Q}[:, :, \\operatorname{start}_b:\\operatorname{end}_b, :], \\quad \\mathbf{K}_b = \\mathbf{K}[:, :, \\operatorname{start}_b:\\operatorname{end}_b, :], \\quad \\mathbf{V}_b = \\mathbf{V}[:, :, \\operatorname{start}_b:\\operatorname{end}_b, :]", | |
| "markdown_preview": "# Block Transformer HIBA: Hierarchical Intra/Inter-Block Attention\n\n## 1. Mathematical Formulation & Architectural Design\n\nThe **Block Transformer HIBA (Hierarchical Intra-Block and Inter-Block Attention)** architecture reorganizes long-sequence attention into a two-level hierarchical sparse representation:\n1. **Intra-Block Attention (Local Fine-Grained Layer)**: Dense bidirectional self-attention executed strictly within localized blocks of size $B_s = 64$.\n2. **Inter-Block Sparse Attention (Global Routing Layer)**: Dynamic top-$k$ block-to-block routing that selectively retrieves high-affini" | |
| }, | |
| { | |
| "id": "chela_slay", | |
| "name": "CHELA-SLAY: Spherical Linearized Attention with Yat-Kernel.", | |
| "file": "Codes/framework/attentions/chela_slay.py", | |
| "type": "Attention Module", | |
| "category": "Latent & KV-Cache Compression", | |
| "paradigm": "mla", | |
| "icon": "\ud83e\udde0", | |
| "description": "CHELA-SLAY: Spherical Linearized Attention with Yat-Kernel. Implements hybrid linear attention via spherical normalization onto the unit hypersphere combined with trigonometric random Fourier feature mappings (Yat-Kernel) and adaptive gating for smooth distillation from pretrained attention. Hardened for production across DistilBERT, RoBERTa/BERT", | |
| "math": "\\hat{Q} = \\frac{Q}{\\|Q\\|_2}, \\quad \\hat{K} = \\frac{K}{\\|K\\|_2}", | |
| "markdown_preview": "# CHELA-SLAY: Spherical Linearized Attention with Yat-Kernel\n\n## 1. Mathematical Formulation & Architecture\n1. **Spherical Hypersphere Projection**:\n $$\\hat{Q} = \\frac{Q}{\\|Q\\|_2}, \\quad \\hat{K} = \\frac{K}{\\|K\\|_2}$$\n\n2. **Yat-Kernel Trigonometric Feature Map**:\n $$\\phi(x) = \\frac{1}{F^{1/4}} [\\cos(x \\Omega), \\sin(x \\Omega)] \\in \\mathbb{R}^{2F}$$\n\n3. **Linear Attention Matrix Reduction**:\n $$O^{\\text{linear}} = \\frac{\\phi(\\hat{Q}) (\\phi(\\hat{K})^T V)}{\\phi(\\hat{Q}) (\\sum \\phi(\\hat{K})^T) + \\epsilon}$$\n\n4. **Adaptive Gated Hybrid Blending**:\n $$g = \\sigma(X W_{\\text{gate}} + 1.8)$$\n $" | |
| }, | |
| { | |
| "id": "cluster_attention", | |
| "name": "ClusterFormer: Dynamic Token-Clustering Attention via Batch Itera", | |
| "file": "Codes/framework/attentions/cluster_attention.py", | |
| "type": "Attention Module", | |
| "category": "Dynamic & Efficient Attention", | |
| "paradigm": "sparse", | |
| "icon": "\ud83c\udf1f", | |
| "description": "ClusterFormer: Token Clustering Attention Module. Groups queries into $K$ semantic cluster centroids via fast differentiable $K$-means and evaluates centroid-to-key cross-attention alongside local context, achieving $\\mathcal{O}(K \\cdot L \\cdot d)$ linear complexity while eliminating token isolation. For causal decoders, strict causal masking is e", | |
| "math": "A_{ij} = \\text{TopK}\\left(\\text{Softmax}\\left(\\frac{Q_i K_j^\\top}{\\sqrt{d}}\\right)\\right)", | |
| "markdown_preview": "# ClusterFormer: Dynamic Token-Clustering Attention via Batch Iterative K-Means\n\n## 1. Executive Abstract & Architectural Motivation\n\nDense Multi-Head Self-Attention computes all $L \\times L$ pairwise interactions, leading to quadratic time complexity $\\mathcal{O}(L^2 d)$. Standard sparse attention methods use fixed positional windows or static random graphs that cannot adapt to the semantic content of input sequences.\n\n**ClusterFormer (`ClusterAttention`)** addresses this challenge by grouping queries into $K \\ll L$ semantic cluster centroids via batch-wise iterative $K$-means clustering (Llo" | |
| }, | |
| { | |
| "id": "cluster_inject", | |
| "name": "ClusterFormer: Dynamic Token-Clustering Attention via Batch Itera", | |
| "file": "Codes/framework/attentions/cluster_inject.py", | |
| "type": "Attention Module", | |
| "category": "Dynamic & Efficient Attention", | |
| "paradigm": "sparse", | |
| "icon": "\ud83c\udf1f", | |
| "description": "Deprecated stub. Re-exports base module for registry compatibility.", | |
| "math": "A_{ij} = \\text{TopK}\\left(\\text{Softmax}\\left(\\frac{Q_i K_j^\\top}{\\sqrt{d}}\\right)\\right)", | |
| "markdown_preview": "# ClusterFormer: Dynamic Token-Clustering Attention via Batch Iterative K-Means\n\n## 1. Executive Abstract & Architectural Motivation\n\nDense Multi-Head Self-Attention computes all $L \\times L$ pairwise interactions, leading to quadratic time complexity $\\mathcal{O}(L^2 d)$. Standard sparse attention methods use fixed positional windows or static random graphs that cannot adapt to the semantic content of input sequences.\n\n**ClusterFormer (`ClusterAttention`)** addresses this challenge by grouping queries into $K \\ll L$ semantic cluster centroids via batch-wise iterative $K$-means clustering (Llo" | |
| }, | |
| { | |
| "id": "compressed_attention", | |
| "name": "Compressed Multi-Head Attention via Truncated Singular Value Deco", | |
| "file": "Codes/framework/attentions/compressed_attention.py", | |
| "type": "Attention Module", | |
| "category": "Low-Rank SVD & Factorization", | |
| "paradigm": "svd", | |
| "icon": "\u26a1", | |
| "description": "Compressed Multi-Head Attention via Truncated Singular Value Decomposition (SVD). Provides rank-constrained and singular-energy-thresholded low-rank approximations of standard PyTorch attention input (QKV) and output projection matrices with full pretrained weight transfer, trainable LoRA residual adapters, GQA, RoPE, and causal decoding support.", | |
| "math": "\\mathbf{W} \\approx \\mathbf{U}_r \\mathbf{\\Sigma}_r \\mathbf{V}_r^T", | |
| "markdown_preview": "# Compressed Attention via Truncated SVD\n\n## 1. Mathematical Formulation & Architecture\nDecomposes high-dimensional linear projections into rank-$r$ truncated singular value factors:\n$$\\mathbf{W} \\approx \\mathbf{U}_r \\mathbf{\\Sigma}_r \\mathbf{V}_r^T$$\nImplemented symmetrically with scaled trainable parameters:\n$$\\mathbf{A} = \\mathbf{V}_r \\mathbf{\\Sigma}_r^{1/2}, \\quad \\mathbf{B} = \\mathbf{\\Sigma}_r^{1/2} \\mathbf{U}_r^T$$\n$$\\mathbf{y} = (\\mathbf{x} \\mathbf{A}) \\mathbf{B} + \\mathbf{b}$$\n\n\n## Architecture & Data Flow\n\nThe `CompressedAttentionLayer` module coordinates input projection factorizatio" | |
| }, | |
| { | |
| "id": "compressed_mha", | |
| "name": "Compressed Multi-Head Attention (Compressed-MHA)", | |
| "file": "Codes/framework/attentions/compressed_mha.py", | |
| "type": "Attention Module", | |
| "category": "Dynamic & Efficient Attention", | |
| "paradigm": "sparse", | |
| "icon": "\ud83c\udf1f", | |
| "description": "Production research module for dynamic & efficient attention across transformer architectures.", | |
| "math": "\\mathbf{W}_{QKV} = [\\mathbf{W}_Q; \\mathbf{W}_K; \\mathbf{W}_V] \\in \\mathbb{R}^{3D \\times D}", | |
| "markdown_preview": "# Compressed Multi-Head Attention (Compressed-MHA)\n\n## 1. Mathematical Formulation\nPerforms joint truncated SVD decomposition across stacked QKV matrices:\n$$\\mathbf{W}_{QKV} = [\\mathbf{W}_Q; \\mathbf{W}_K; \\mathbf{W}_V] \\in \\mathbb{R}^{3D \\times D}$$\n$$\\mathbf{W}_{QKV} \\approx \\mathbf{U}_{QKV} \\mathbf{\\Sigma}_{QKV} \\mathbf{V}_{QKV}^T$$\n\n\n## Architecture & Data Flow\n\nThe `CompressedDistilBertAttention` module coordinates input projection factorization, intermediate representation routing, and output synthesis:\n\n```\nInput Activation X [B, N, D]\n \u2502\n \u251c\u2500\u2500 Low-Rank Bottleneck Down-Projection: Z =" | |
| }, | |
| { | |
| "id": "compressed_mha_attention", | |
| "name": "Compressed Multi-Head Attention for DistilBERT and Multi-Architec", | |
| "file": "Codes/framework/attentions/compressed_mha_attention.py", | |
| "type": "Attention Module", | |
| "category": "Low-Rank SVD & Factorization", | |
| "paradigm": "svd", | |
| "icon": "\u26a1", | |
| "description": "Compressed Multi-Head Attention for DistilBERT and Multi-Architecture via Joint Truncated SVD. Performs joint singular value decomposition on stacked query, key, and value projection matrices as well as the final output projection matrix with frozen base parameters, trainable LoRA residual adapters, GQA, RoPE, causal decoding, and full pretrained ", | |
| "math": "\\mathbf{W}_{QKV} = [\\mathbf{W}_Q; \\mathbf{W}_K; \\mathbf{W}_V] \\in \\mathbb{R}^{3D \\times D}", | |
| "markdown_preview": "# Compressed Multi-Head Attention (Compressed-MHA)\n\n## 1. Mathematical Formulation\nPerforms joint truncated SVD decomposition across stacked QKV matrices:\n$$\\mathbf{W}_{QKV} = [\\mathbf{W}_Q; \\mathbf{W}_K; \\mathbf{W}_V] \\in \\mathbb{R}^{3D \\times D}$$\n$$\\mathbf{W}_{QKV} \\approx \\mathbf{U}_{QKV} \\mathbf{\\Sigma}_{QKV} \\mathbf{V}_{QKV}^T$$\n\n\n## Architecture & Data Flow\n\nThe `CompressedDistilBertAttention` module coordinates input projection factorization, intermediate representation routing, and output synthesis:\n\n```\nInput Activation X [B, N, D]\n \u2502\n \u251c\u2500\u2500 Low-Rank Bottleneck Down-Projection: Z =" | |
| }, | |
| { | |
| "id": "cosformer", | |
| "name": "Cosformer: Rethinking Softmax in Attention with Linear Complexity", | |
| "file": "Codes/framework/attentions/cosformer.py", | |
| "type": "Attention Module", | |
| "category": "Low-Rank SVD & Factorization", | |
| "paradigm": "svd", | |
| "icon": "\u26a1", | |
| "description": "Cosformer: Rethinking Softmax in Attention with Linear Complexity. Implements non-negative kernel feature mapping with cosine/sine positional reweighting, cumulative sum causal linear attention for decoders, and associative accumulation for encoders. Hardened for production across DistilBERT, RoBERTa/BERT, GPT-2, and Qwen/LLaMA architectures: - R", | |
| "math": "W \\approx U_r \\Sigma_r V_r^\\top = A B^\\top, \\quad r \\ll \\min(d_{in}, d_{out})", | |
| "markdown_preview": "# Cosformer: Rethinking Softmax in Attention with Linear Complexity\n\n## 1. Executive Abstract & Architectural Motivation\n\nStandard Multi-Head Self-Attention requires quadratic computational complexity $\\mathcal{O}(L^2)$ and memory footprint $\\mathcal{O}(L^2)$ due to the explicit computation of the Softmax attention matrix:\n\\[\n\\operatorname{Attn}(Q, K, V) = \\operatorname{Softmax}\\left(\\frac{Q K^\\top}{\\sqrt{d}}\\right) V\n\\]\n**Cosformer** (Qin et al., 2022, ICLR) addresses this computational bottleneck by replacing Softmax with a **decomposed non-negative kernel mapping combined with a cosine-base" | |
| }, | |
| { | |
| "id": "cosformer_inject", | |
| "name": "Deprecated stub. Re-exports base module for registry compatibilit", | |
| "file": "Codes/framework/attentions/cosformer_inject.py", | |
| "type": "Attention Module", | |
| "category": "Linear Attention & State-Space (SSM)", | |
| "paradigm": "ssm", | |
| "icon": "\ud83d\udc0d", | |
| "description": "Deprecated stub. Re-exports base module for registry compatibility.", | |
| "math": "h_t = \\bar{A} h_{t-1} + \\bar{B} x_t, \\quad y_t = C h_t + D x_t, \\quad \\bar{A} = \\exp(\\Delta A)", | |
| "markdown_preview": "# Cosformer: Rethinking Softmax in Attention with Linear Complexity\n\n## 1. Executive Abstract & Architectural Motivation\n\nStandard Multi-Head Self-Attention requires quadratic computational complexity $\\mathcal{O}(L^2)$ and memory footprint $\\mathcal{O}(L^2)$ due to the explicit computation of the Softmax attention matrix:\n\\[\n\\operatorname{Attn}(Q, K, V) = \\operatorname{Softmax}\\left(\\frac{Q K^\\top}{\\sqrt{d}}\\right) V\n\\]\n**Cosformer** (Qin et al., 2022, ICLR) addresses this computational bottleneck by replacing Softmax with a **decomposed non-negative kernel mapping combined with a cosine-base" | |
| }, | |
| { | |
| "id": "decay_linear_attention", | |
| "name": "Decay Linear Attention (DecayLinearAttention).", | |
| "file": "Codes/framework/attentions/decay_linear_attention.py", | |
| "type": "Attention Module", | |
| "category": "Low-Rank SVD & Factorization", | |
| "paradigm": "svd", | |
| "icon": "\u26a1", | |
| "description": "Decay Linear Attention (DecayLinearAttention). Multi-head bidirectional & causal linear recurrent attention with data-dependent exponential decay gating, ELU+1 feature maps, denominator normalization, and pretrained weight transfer. Hardened for production across DistilBERT, RoBERTa/BERT, GPT-2, and Qwen/LLaMA architectures: - Robust pretrained w", | |
| "math": "Q = X W_q, \\quad K = X W_k, \\quad V = X W_v", | |
| "markdown_preview": "# Decay Linear Attention (DecayLinearAttention)\n\n## 1. Executive Abstract\n**Decay Linear Attention** achieved **79.82% Accuracy and 79.76% F1** on SST-2 with DistilBERT (19.5s epoch latency). It combines non-negative ELU+1 kernel feature maps with data-dependent exponential decay gating $\\lambda_t = \\exp(-\\text{softplus}(W_g x_t + b_g))$ and bidirectional sequence scanning.\n\n---\n\n## 2. Mathematical Equations & Recurrence\n\n### 2.1 Multi-Head Projections & Feature Maps\n$$Q = X W_q, \\quad K = X W_k, \\quad V = X W_v$$\n$$\\phi(Q) = \\text{ELU}(Q) + 1, \\quad \\phi(K) = \\text{ELU}(K) + 1$$\n\n### 2.2 Data" | |
| }, | |
| { | |
| "id": "eat", | |
| "name": "EAT: Efficient Attention Transformer with Progressive Token Pruni", | |
| "file": "Codes/framework/attentions/eat.py", | |
| "type": "Attention Module", | |
| "category": "Structured Pruning & Channel Slicing", | |
| "paradigm": "pruning", | |
| "icon": "\u2702\ufe0f", | |
| "description": "EAT: Efficient Attention Transformer with Progressive Token Pruning. Implements production-hardened attention-level efficient multi-head self-attention with optional progressive token pruning and zero-shot weight inheritance across DistilBERT, RoBERTa/BERT, GPT-2, and Qwen/LLaMA architectures.", | |
| "math": "O = \\operatorname{Softmax}\\left(\\frac{Q K^T}{\\sqrt{d_h}} + M\\right) V W_O", | |
| "markdown_preview": "# EAT: Efficient Attention Transformer with Progressive Token Pruning\n\n## 1. Mathematical Formulation & Architecture\n1. **Multi-Head Self-Attention**:\n $$O = \\operatorname{Softmax}\\left(\\frac{Q K^T}{\\sqrt{d_h}} + M\\right) V W_O$$\n\n2. **Progressive Token Pruning**:\n Selects top-$k$ informative tokens dynamically:\n $$X^{\\text{pruned}} = X[:, \\mathcal{I}_{\\text{keep}}, :], \\quad |\\mathcal{I}_{\\text{keep}}| = \\lfloor \\gamma \\cdot L \\rfloor$$\n\n\n## Architecture & Data Flow\n\nThe `EATBlock` module coordinates input projection factorization, intermediate representation routing, and output synthes" | |
| }, | |
| { | |
| "id": "expert_choice_moe", | |
| "name": "Expert Choice Mixture of Experts (EC-MoE) Attention Block", | |
| "file": "Codes/framework/attentions/expert_choice_moe.py", | |
| "type": "Attention Module", | |
| "category": "Low-Rank SVD & Factorization", | |
| "paradigm": "svd", | |
| "icon": "\u26a1", | |
| "description": "Expert Choice Mixture of Experts (MoE) Attention Block. In Expert Choice routing (Zhou et al., 2022), experts select their top-k tokens rather than tokens selecting top-k experts. This provides guaranteed perfect load balancing across all experts, dynamic per-token expert allocation, and superior zero-shot transfer. Hardened for production across", | |
| "math": "\\mathbf{Q} = \\mathbf{X} \\mathbf{W}_Q, \\quad \\mathbf{K} = \\mathbf{X} \\mathbf{W}_K, \\quad \\mathbf{V} = \\mathbf{X} \\mathbf{W}_V", | |
| "markdown_preview": "# Expert Choice Mixture of Experts (EC-MoE) Attention Block\n\n## 1. Mathematical Formulation & Architectural Design\n\nThe **Expert Choice Mixture of Experts (EC-MoE)** architecture resolves the fundamental trade-off between conditional capacity scaling and computational load imbalance in Transformer models. Traditional token-choice routing allows each token to independently select its top-$k$ experts, which frequently leads to **expert starvation** (under-utilization) or **buffer overflow** (token dropping when expert capacity is exceeded).\n\nIn contrast, **Expert Choice Routing** inverts the sel" | |
| }, | |
| { | |
| "id": "factorised_matrix_state", | |
| "name": "Factorised Matrix State Attention: Bidirectional Associative Matr", | |
| "file": "Codes/framework/attentions/factorised_matrix_state.py", | |
| "type": "Attention Module", | |
| "category": "Low-Rank SVD & Factorization", | |
| "paradigm": "svd", | |
| "icon": "\u26a1", | |
| "description": "Factorised Matrix State Attention Module. Implements multi-head bidirectional/causal associative matrix memory updates combined with learnable decay gating, denominator normalization, low-rank factorized output projection, and pretrained weight transfer. Hardened for production across DistilBERT, RoBERTa/BERT, GPT-2, and Qwen/LLaMA architectures:", | |
| "math": "Q = X W_q, \\quad K = X W_k, \\quad V = X W_v \\quad (W_q, W_k, W_v \\in \\mathbb{R}^{D \\times D})", | |
| "markdown_preview": "# Factorised Matrix State Attention: Bidirectional Associative Matrix Recurrence & Low-Rank Bilinear Projections\n\n## 1. Executive Abstract & Architectural Motivation\n\n**Factorised Matrix State Attention** (`FactorisedMatrixState`) provides an optimal $\\mathcal{O}(N)$ linear-time attention alternative to standard $\\mathcal{O}(N^2)$ softmax attention by maintaining an associative multi-head matrix memory state $S_t \\in \\mathbb{R}^{d_k \\times d_v}$ updated with bounded decay recurrence.\n\nStandard recurrent linear attentions suffer from:\n1. **Associative Memory Saturation**: State $S_t = \\sum_{\\ta" | |
| }, | |
| { | |
| "id": "flash_attention3", | |
| "name": "FlashAttention-3: Fast and Memory-Efficient Exact Attention with ", | |
| "file": "Codes/framework/attentions/flash_attention3.py", | |
| "type": "Attention Module", | |
| "category": "Low-Rank SVD & Factorization", | |
| "paradigm": "svd", | |
| "icon": "\u26a1", | |
| "description": "FlashAttention-3: Fast and Memory-Efficient Exact Attention with Asynchronous Hardware Overlap. Integrates FlashAttention-3 kernels (with Hopper TMA asynchronous memory copy and FP8 precision support) alongside graceful fallback to PyTorch's hardware-accelerated Scaled Dot-Product Attention (SDPA) and exact pretrained weight copying. Hardened for", | |
| "math": "S = \\frac{Q K^\\top}{\\sqrt{d_h}} + M", | |
| "markdown_preview": "# FlashAttention-3: Fast and Memory-Efficient Exact Attention with Asynchronous Hardware Overlap\n\n## 1. Executive Abstract & Architectural Motivation\n\nStandard Multi-Head Self-Attention in Transformer models requires materializing the full intermediate attention matrix $P = \\operatorname{Softmax}(Q K^\\top / \\sqrt{d_h}) \\in \\mathbb{R}^{B \\times H \\times L \\times L}$ in High Bandwidth Memory (HBM). This results in quadratic memory bandwidth overhead $\\mathcal{O}(L^2)$, making DRAM IO transfers the primary execution bottleneck on modern GPU architectures.\n\n**FlashAttention-3** advances exact atte" | |
| }, | |
| { | |
| "id": "flash_attention3_fp8", | |
| "name": "FlashAttention-3 FP8: Low-Precision 8-Bit Float Transformer Atten", | |
| "file": "Codes/framework/attentions/flash_attention3_fp8.py", | |
| "type": "Attention Module", | |
| "category": "Low-Rank SVD & Factorization", | |
| "paradigm": "svd", | |
| "icon": "\u26a1", | |
| "description": "FlashAttention-3 FP8: Low-Precision 8-bit Float Transformer Attention. Implements FlashAttention-3 utilizing FP8 (e4m3fn) precision format for Hopper Tensor Cores, with automatic fallback to hardware-accelerated SDPA. Hardened for production across DistilBERT, RoBERTa/BERT, GPT-2, and Qwen/LLaMA architectures: - Robust pretrained weight extractio", | |
| "math": "Q_{\\text{fp8}} = \\text{cast}_{\\text{fp8\\_e4m3}}(Q) \\in \\mathbb{F}_{p8}^{B \\times L \\times H \\times d_h}", | |
| "markdown_preview": "# FlashAttention-3 FP8: Low-Precision 8-Bit Float Transformer Attention for Hopper Architectures\n\n## 1. Executive Abstract & Architectural Motivation\nStandard Multi-Head Self-Attention (MHSA) in Transformer models operating under FP16 or BF16 precision consumes significant High Bandwidth Memory (HBM) activation storage and memory bandwidth. Intermediate attention score materialization requires $\\mathcal{O}(L^2)$ space, creating memory bandwidth bottlenecks on modern GPU accelerators.\n\n**FlashAttention-3 FP8** combines exact tiled attention algorithms with hardware-accelerated 8-bit floating po" | |
| }, | |
| { | |
| "id": "full_attention", | |
| "name": "Full Attention: Standard Multi-Head Scaled Dot-Product Attention ", | |
| "file": "Codes/framework/attentions/full_attention.py", | |
| "type": "Attention Module", | |
| "category": "Dynamic & Efficient Attention", | |
| "paradigm": "sparse", | |
| "icon": "\ud83c\udf1f", | |
| "description": "Standard Multi-Head Scaled Dot-Product Attention (Vaswani et al., 2017). Computes pairwise token interactions across multiple representation subspaces with frozen base weights, trainable LoRA residual adapters, multi-architecture weight transfer, GQA, RoPE, and causal decoding.", | |
| "math": "\\text{Attention}(Q, K, V) = \\text{softmax}\\left(\\frac{Q K^\\top}{\\sqrt{d_k}} + M\\right) V", | |
| "markdown_preview": "# Full Attention: Standard Multi-Head Scaled Dot-Product Attention Architecture\n\n## 1. Executive Abstract & Architectural Motivation\n\nFull Multi-Head Attention (MHA), introduced by Vaswani et al. (2017), forms the core mathematical foundation of modern Transformer architectures. It operates by partitioning the hidden representation space of dimension $D$ into $H$ parallel subspaces (heads) of dimension $d_k = D / H$. For every head, input sequences are projected into Query ($Q$), Key ($K$), and Value ($V$) representations. Pairwise token interactions are computed via scaled inner products, yie" | |
| }, | |
| { | |
| "id": "fused_svd_kernel", | |
| "name": "Fused SVD-Linear Layer with SRAM-Resident RMSNorm & Dual-Projecti", | |
| "file": "Codes/framework/attentions/fused_svd_kernel.py", | |
| "type": "Attention Module", | |
| "category": "Low-Rank SVD & Factorization", | |
| "paradigm": "svd", | |
| "icon": "\u26a1", | |
| "description": "Fused SVD-Linear Layer with SRAM-Resident RMSNorm & Dual-Projection. RESEARCH FIX: Eliminates GPU High-Bandwidth Memory (HBM) roundtrip latency by fusing: 1. Down-projection: Z = X @ V^T (hidden_dim -> rank) 2. In-SRAM RMSNorm: Z_norm = Z * rsqrt(mean(Z^2) + eps) * weight 3. Up-projection: Y = Z_norm @ U^T + bias (rank -> out_dim) ", | |
| "math": "\\min_{\\text{rank}(\\widehat{W}) \\le r} \\|W - \\widehat{W}\\|_F^2 = \\sum_{i=r+1}^{D} \\sigma_i^2", | |
| "markdown_preview": "# Fused SVD-Linear Kernel: SRAM-Resident RMSNorm & Dual-Projection\n\n## 1. Executive Abstract & Architectural Motivation\n\nStandard low-rank linear projections evaluated sequentially as $Z = X V^\\top$, $Z_{\\text{norm}} = \\text{RMSNorm}(Z)$, and $Y = Z_{\\text{norm}} U^\\top + b$ incur severe High-Bandwidth Memory (HBM) roundtrip overhead. For intermediate activation tensor $Z \\in \\mathbb{R}^{B \\times N \\times r}$, standard frameworks write $Z$ to GPU DRAM and immediately reload it for normalization and up-projection.\n\n`FusedSVDLinear` eliminates all intermediate HBM roundtrips by fusing down-proje" | |
| }, | |
| { | |
| "id": "fusion_attn", | |
| "name": "FUSIONATTN: Multi-Paradigm Hybrid Sparse Attention Module.", | |
| "file": "Codes/framework/attentions/fusion_attn.py", | |
| "type": "Attention Module", | |
| "category": "Low-Rank SVD & Factorization", | |
| "paradigm": "svd", | |
| "icon": "\u26a1", | |
| "description": "FUSIONATTN: Multi-Paradigm Hybrid Sparse Attention Module. Implements hybrid sparse attention with block gating, full-attention short sequence fast-path, and subspace estimation. Hardened for production across DistilBERT, RoBERTa/BERT, GPT-2, and Qwen/LLaMA architectures: - Robust pretrained weight extraction (GPT-2 fused c_attn, RoBERTa nested, ", | |
| "math": "\\mathbf{S}_{\\text{blk}} = \\frac{\\mathbf{G}_Q(\\mathbf{Q}_{\\text{blk}}) \\mathbf{G}_K(\\mathbf{K}_{\\text{blk}})^T}{\\tau}", | |
| "markdown_preview": "# FUSIONATTN: Multi-Paradigm Hybrid Sparse Attention\n\n## 1. Mathematical Formulation & Architecture\nCombines block-level Seer gating with fine-grained selected attention:\n$$\\mathbf{S}_{\\text{blk}} = \\frac{\\mathbf{G}_Q(\\mathbf{Q}_{\\text{blk}}) \\mathbf{G}_K(\\mathbf{K}_{\\text{blk}})^T}{\\tau}$$\nBlock 0 ($[\\text{CLS}]$) is permanently retained in top-$k$ block selection.\n\n\n## Architecture & Data Flow\n\nThe `FUSIONATTN` module coordinates input projection factorization, intermediate representation routing, and output synthesis:\n\n```\nInput Activation X [B, N, D]\n \u2502\n \u251c\u2500\u2500 Low-Rank Bottleneck Down-Pr" | |
| }, | |
| { | |
| "id": "galerkin", | |
| "name": "Petrov-Galerkin Projection Attention: Energy-Preserving Spectral ", | |
| "file": "Codes/framework/attentions/galerkin.py", | |
| "type": "Attention Module", | |
| "category": "Dynamic & Efficient Attention", | |
| "paradigm": "sparse", | |
| "icon": "\ud83c\udf1f", | |
| "description": "Galerkin Attention alias module.", | |
| "math": "A_{ij} = \\text{TopK}\\left(\\text{Softmax}\\left(\\frac{Q_i K_j^\\top}{\\sqrt{d}}\\right)\\right)", | |
| "markdown_preview": "# Petrov-Galerkin Projection Attention: Energy-Preserving Spectral Operator Discretization\n\n## 1. Executive Abstract & Architectural Motivation\n\nStandard Multi-Head Attention (MHA) calculates all pairwise token dot products with quadratic $\\mathcal{O}(L^2)$ time and space complexity. Petrov-Galerkin Attention models self-attention as a continuous non-local Fredholm integral operator $\\mathcal{K} v(x) = \\int \\kappa(x, y) v(y) dy$ over function spaces, discretizing the kernel via orthogonal Petrov-Galerkin projection onto an $r$-dimensional orthonormal basis $\\Phi \\in \\mathbb{R}^{N \\times r}$ wi" | |
| }, | |
| { | |
| "id": "galerkin_attention", | |
| "name": "Petrov-Galerkin Projection Attention: Energy-Preserving Spectral ", | |
| "file": "Codes/framework/attentions/galerkin_attention.py", | |
| "type": "Attention Module", | |
| "category": "Low-Rank SVD & Factorization", | |
| "paradigm": "svd", | |
| "icon": "\u26a1", | |
| "description": "Galerkin Attention: Petrov-Galerkin Projection Attention for Transformer Architectures. Formulates transformer self-attention as a continuous non-local integral operator discretized via Petrov-Galerkin projection onto an orthonormal basis (e.g., discrete cosine basis). Supports fallback to standard causal attention for decoder models. Hardened fo", | |
| "math": "W \\approx U_r \\Sigma_r V_r^\\top = A B^\\top, \\quad r \\ll \\min(d_{in}, d_{out})", | |
| "markdown_preview": "# Petrov-Galerkin Projection Attention: Energy-Preserving Spectral Operator Discretization\n\n## 1. Executive Abstract & Architectural Motivation\n\nStandard Multi-Head Attention (MHA) calculates all pairwise token dot products with quadratic $\\mathcal{O}(L^2)$ time and space complexity. Petrov-Galerkin Attention models self-attention as a continuous non-local Fredholm integral operator $\\mathcal{K} v(x) = \\int \\kappa(x, y) v(y) dy$ over function spaces, discretizing the kernel via orthogonal Petrov-Galerkin projection onto an $r$-dimensional orthonormal basis $\\Phi \\in \\mathbb{R}^{N \\times r}$ wi" | |
| }, | |
| { | |
| "id": "galerkin_inject", | |
| "name": "Petrov-Galerkin Projection Attention: Energy-Preserving Spectral ", | |
| "file": "Codes/framework/attentions/galerkin_inject.py", | |
| "type": "Attention Module", | |
| "category": "Dynamic & Efficient Attention", | |
| "paradigm": "sparse", | |
| "icon": "\ud83c\udf1f", | |
| "description": "Deprecated stub. Re-exports base module for registry compatibility.", | |
| "math": "A_{ij} = \\text{TopK}\\left(\\text{Softmax}\\left(\\frac{Q_i K_j^\\top}{\\sqrt{d}}\\right)\\right)", | |
| "markdown_preview": "# Petrov-Galerkin Projection Attention: Energy-Preserving Spectral Operator Discretization\n\n## 1. Executive Abstract & Architectural Motivation\n\nStandard Multi-Head Attention (MHA) calculates all pairwise token dot products with quadratic $\\mathcal{O}(L^2)$ time and space complexity. Petrov-Galerkin Attention models self-attention as a continuous non-local Fredholm integral operator $\\mathcal{K} v(x) = \\int \\kappa(x, y) v(y) dy$ over function spaces, discretizing the kernel via orthogonal Petrov-Galerkin projection onto an $r$-dimensional orthonormal basis $\\Phi \\in \\mathbb{R}^{N \\times r}$ wi" | |
| }, | |
| { | |
| "id": "gated_delta_net", | |
| "name": "Gated DeltaNet: Multi-Head Bidirectional Delta Rule Linear Recurr", | |
| "file": "Codes/framework/attentions/gated_delta_net.py", | |
| "type": "Attention Module", | |
| "category": "Low-Rank SVD & Factorization", | |
| "paradigm": "svd", | |
| "icon": "\u26a1", | |
| "description": "Gated DeltaNet Attention Module. Implements multi-head bidirectional/causal Delta Rule linear recurrent attention with input-dependent exponential decay gating, associative state error correction, and pretrained self-attention weight transfer. Hardened for production across DistilBERT, RoBERTa/BERT, GPT-2, and Qwen/LLaMA architectures: - Robust p", | |
| "math": "Q = X W_q \\in \\mathbb{R}^{B \\times L \\times D}, \\quad K = X W_k \\in \\mathbb{R}^{B \\times L \\times D}", | |
| "markdown_preview": "# Gated DeltaNet: Multi-Head Bidirectional Delta Rule Linear Recurrent Attention\n\n## 1. Executive Abstract & Theoretical Foundations\n\nStandard linear attention modules approximate softmax self-attention via associative outer-product state updates $S_t = S_{t-1} + k_t v_t^\\top$, suffering from **associative memory capacity saturation** where historical context is unselectively overwritten or unbounded memory accumulations drift. **Gated DeltaNet** resolves associative memory saturation through two fundamental innovations:\n1. **Delta-Rule Error-Correction Associative Memory**: Rather than naive " | |
| }, | |
| { | |
| "id": "gated_deltanet_block", | |
| "name": "Gated DeltaNet Block: Hybrid Multi-Head Attention and Recurrent D", | |
| "file": "Codes/framework/attentions/gated_deltanet_block.py", | |
| "type": "Attention Module", | |
| "category": "Mixture of Experts & Routing", | |
| "paradigm": "moe", | |
| "icon": "\ud83d\udd00", | |
| "description": "Gated DeltaNet Block with Optional Multi-Head Attention Bypass. Implements an efficient hybrid linear-recurrent block switching dynamically between standard multi-head self-attention and Bidirectional Gated DeltaNet with pretrained weight transfer.", | |
| "math": "\\text{Output} = \\begin{cases}\n \\text{LayerNorm}(X + \\text{MHA}(X, X, X)), & \\text{if } \\text{use\\_full\\_attention} \\\\\n \\text{LayerNorm}(X + \\text{GatedDeltaNet}(X)), & \\text{otherwise}\n\\end{cases}", | |
| "markdown_preview": "# Gated DeltaNet Block: Hybrid Multi-Head Attention and Recurrent DeltaNet Layer\n\n## 1. Executive Abstract\n`GatedDeltaNetBlock` provides a modular transformer block wrapper that allows seamless dynamic switching between dense multi-head self-attention and Bidirectional Gated DeltaNet linear recurrence.\n\n---\n\n## 2. Architectural Formulation\n$$\\text{Output} = \\begin{cases}\n \\text{LayerNorm}(X + \\text{MHA}(X, X, X)), & \\text{if } \\text{use\\_full\\_attention} \\\\\n \\text{LayerNorm}(X + \\text{GatedDeltaNet}(X)), & \\text{otherwise}\n\\end{cases}$$\n\nPretrained weights are automatically transferred i" | |
| }, | |
| { | |
| "id": "gated_linear_attention", | |
| "name": "Gated Linear Attention (GatedLinearAttention).", | |
| "file": "Codes/framework/attentions/gated_linear_attention.py", | |
| "type": "Attention Module", | |
| "category": "Linear Attention & State-Space (SSM)", | |
| "paradigm": "ssm", | |
| "icon": "\ud83d\udc0d", | |
| "description": "Gated Linear Attention (GatedLinearAttention). Multi-head linear recurrent attention with data-dependent decay gating, multiplicative SiLU output gating, denominator normalization, and pretrained weight transfer. Hardened for multi-architecture production deployment across DistilBERT, RoBERTa, GPT-2, and Qwen3.5.", | |
| "math": "Q = X W_q, \\quad K = X W_k, \\quad V = X W_v, \\quad Z = X W_z", | |
| "markdown_preview": "# Gated Linear Attention (GatedLinearAttention)\n\n## 1. Executive Abstract\n**Gated Linear Attention (GLA)** achieved **79.82% Accuracy and 79.76% F1** on SST-2 with DistilBERT (19.8s epoch latency). It equips linear attention with dual gating: data-dependent exponential decay gating along the recurrence dimension and multiplicative SiLU output gating.\n\n---\n\n## 2. Mathematical Equations & Recurrence\n\n### 2.1 Multi-Head Projections & Dual Gating\n$$Q = X W_q, \\quad K = X W_k, \\quad V = X W_v, \\quad Z = X W_z$$\n$$\\phi(Q) = \\text{ELU}(Q) + 1, \\quad \\phi(K) = \\text{ELU}(K) + 1$$\n$$\\lambda_t = \\exp(-\\" | |
| }, | |
| { | |
| "id": "goldfinch_ahn", | |
| "name": "GoldFinch-AHN: Adaptive Hybrid Network Combining Sliding Window a", | |
| "file": "Codes/framework/attentions/goldfinch_ahn.py", | |
| "type": "Attention Module", | |
| "category": "Linear Attention & State-Space (SSM)", | |
| "paradigm": "ssm", | |
| "icon": "\ud83d\udc0d", | |
| "description": "GoldFinch-AHN: Adaptive Hybrid Network Combining Sliding Window and Gated DeltaNet Memory (Hardened). Combines short-term multi-head sliding-window attention with long-term GatedDeltaNet recurrent memory via a learned input-dependent dynamic gating mechanism initialized for stable representation transfer. Supports: - Multi-architecture copy_weigh", | |
| "math": "h_t = \\bar{A} h_{t-1} + \\bar{B} x_t, \\quad y_t = C h_t + D x_t, \\quad \\bar{A} = \\exp(\\Delta A)", | |
| "markdown_preview": "\n\n## Empirical Benchmark Results\n\n| Model | Dataset | Status | Baseline Acc | Compressed Acc | \u0394 Acc | F1 Score | Comp Rate | Compressed Layers | Checkpoint | Date |\n| :--- | :--- | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: |\n| **distilbert-sst2** | sst2 | `PASS` | 91.06% | **91.06%** | +0.00% | 0.9128 | 1.00x | 6 | [svd-linear-attention-distilbert-sst2-goldfinch](https://huggingface.co/Hooshaai/svd-linear-attention-distilbert-sst2-goldfinch) | 2026-09-19 |\n" | |
| }, | |
| { | |
| "id": "gqa", | |
| "name": "Grouped-Query Attention (GQA): Memory-Efficient Shared Key-Value ", | |
| "file": "Codes/framework/attentions/gqa.py", | |
| "type": "Attention Module", | |
| "category": "Dynamic & Efficient Attention", | |
| "paradigm": "sparse", | |
| "icon": "\ud83c\udf1f", | |
| "description": "Production research module for dynamic & efficient attention across transformer architectures.", | |
| "math": "K_{\\text{shared}} = \\text{repeat\\_interleave}(K, H_Q / H_{KV}, \\text{dim}=\\text{heads})", | |
| "markdown_preview": "# Grouped-Query Attention (GQA): Memory-Efficient Shared Key-Value Subspace Architecture\n\n## 1. Executive Abstract & Architectural Motivation\n\nGrouped-Query Attention (GQA), proposed by Ainslie et al. (2023), establishes a theoretical and practical middle ground between standard Multi-Head Attention (MHA) and Multi-Query Attention (MQA). In standard MHA, every query head maintains its own independent key and value head ($H_{KV} = H_Q$). While MHA delivers unconstrained subspace expressivity, its memory footprint scales linearly with the number of heads during autoregressive decoding, causing s" | |
| }, | |
| { | |
| "id": "gqa_attention", | |
| "name": "Grouped-Query Attention (GQA): Memory-Efficient Shared Key-Value ", | |
| "file": "Codes/framework/attentions/gqa_attention.py", | |
| "type": "Attention Module", | |
| "category": "Latent & KV-Cache Compression", | |
| "paradigm": "mla", | |
| "icon": "\ud83e\udde0", | |
| "description": "Grouped-Query Attention (GQA). Interpolates between Multi-Head Attention (MHA) and Multi-Query Attention (MQA) by dividing $H_Q$ query heads into $G$ groups, where each group shares a single Key-Value head pair ($H_{KV} = G \\le H_Q$).", | |
| "math": "K_{\\text{shared}} = \\text{repeat\\_interleave}(K, H_Q / H_{KV}, \\text{dim}=\\text{heads})", | |
| "markdown_preview": "# Grouped-Query Attention (GQA): Memory-Efficient Shared Key-Value Subspace Architecture\n\n## 1. Executive Abstract & Architectural Motivation\n\nGrouped-Query Attention (GQA), proposed by Ainslie et al. (2023), establishes a theoretical and practical middle ground between standard Multi-Head Attention (MHA) and Multi-Query Attention (MQA). In standard MHA, every query head maintains its own independent key and value head ($H_{KV} = H_Q$). While MHA delivers unconstrained subspace expressivity, its memory footprint scales linearly with the number of heads during autoregressive decoding, causing s" | |
| }, | |
| { | |
| "id": "granite_hybrid", | |
| "name": "Granite-Hybrid Attention: Interleaved Mamba-2 State Space Archite", | |
| "file": "Codes/framework/attentions/granite_hybrid.py", | |
| "type": "Attention Module", | |
| "category": "Mixture of Experts & Routing", | |
| "paradigm": "moe", | |
| "icon": "\ud83d\udd00", | |
| "description": "Granite Hybrid Attention-SSM Layer with MoE Feed-Forward. Combines Mamba-2 SSM recurrent layers with dense Multihead Attention (MHA) and Top-k Mixture-of-Experts (MoE) feed-forward, with LoRA adapters and pretrained weight transfer. Hardened across DistilBERT, RoBERTa, GPT-2, and Qwen3.5 architectures.", | |
| "math": "[X_{\\text{ssm}}, Z] = X W_{\\text{in}} \\in \\mathbb{R}^{B \\times L \\times 2 D_{\\text{inner}}}, \\quad D_{\\text{inner}} = D \\cdot \\text{expand}", | |
| "markdown_preview": "# Granite-Hybrid Attention: Interleaved Mamba-2 State Space Architecture with Multi-Head Attention and Mixture-of-Experts\n\n## 1. Executive Abstract & Architectural Motivation\nStandard Multi-Head Self-Attention (MHSA) architectures suffer from quadratic time and memory complexity $\\mathcal{O}(L^2)$ with sequence length $L$, imposing prohibitive overhead for long-context generation. Conversely, pure State Space Models (SSMs) like Mamba-2 achieve linear scaling $\\mathcal{O}(L \\cdot D \\cdot d_{\\text{state}})$, but lack the non-local associative recall and exact token-matching precision of dense se" | |
| }, | |
| { | |
| "id": "griffin_ssd", | |
| "name": "Griffin SSD: Hybrid Local Sliding-Window Attention and Global Sta", | |
| "file": "Codes/framework/attentions/griffin_ssd.py", | |
| "type": "Attention Module", | |
| "category": "Linear Attention & State-Space (SSM)", | |
| "paradigm": "ssm", | |
| "icon": "\ud83d\udc0d", | |
| "description": "Griffin SSD: Hybrid Local Sliding-Window Attention and Global State Space Duality (SSD) Kernel. Combines bidirectional/causal local sliding-window self-attention with multi-head State Space Duality linear recurrence, LoRA residual adapters, and multi-architecture pretrained weight transfer. Hardened across DistilBERT, RoBERTa, GPT-2, and Qwen3.5 a", | |
| "math": "S^{\\text{local}}_{b, h, i, j} = \\frac{\\langle q_{b, h, i}, k_{b, h, j} \\rangle}{\\sqrt{d}} + M^{\\text{window}}_{i, j} + M^{\\text{pad}}_{b, j}", | |
| "markdown_preview": "# Griffin SSD: Hybrid Sliding-Window Attention & State Space Duality (SSD) Recurrence\n\n## 1. Executive Abstract & Theoretical Foundations\n\n**Griffin SSD** unifies local sliding-window self-attention with multi-head **State Space Duality (SSD)** linear recurrence:\n1. **Local Bidirectional Sliding-Window Branch**: Computes exact softmax attention over local context window $W = 64$ to preserve fine-grained syntactic dependencies.\n2. **Global Bidirectional SSD Recurrence Branch**: Computes multi-head linear associative state-space recurrence $\\vec{H}_t = \\lambda \\vec{H}_{t-1} + \\tilde{k}_t \\otimes" | |
| }, | |
| { | |
| "id": "halo_regla", | |
| "name": "HALO-ReGLA: Hardware-Aware Linear Optics & Rectified Gated Linear", | |
| "file": "Codes/framework/attentions/halo_regla.py", | |
| "type": "Attention Module", | |
| "category": "Latent & KV-Cache Compression", | |
| "paradigm": "mla", | |
| "icon": "\ud83e\udde0", | |
| "description": "HALO-ReGLA: Hardware-Aware Linear Optics & Rectified Gated Linear Attention \u2013 Production-Hardened. Combines multi-head linear projection with Rotary Position Embeddings (RoPE), rectified kernel representations, and adaptive gating for zero-shot distillation from pretrained transformers. Supports: - Multi-architecture copy_weights (GPT-2, RoBERTa/", | |
| "math": "Q_{\\text{rope}} = \\operatorname{RoPE}(Q), \\quad K_{\\text{rope}} = \\operatorname{RoPE}(K)", | |
| "markdown_preview": "# HALO-ReGLA: Hardware-Aware Linear Optics & Rectified Gated Linear Attention\n\n## 1. Mathematical Formulation & Architecture\n1. **Rotary Position Embeddings (RoPE)**:\n $$Q_{\\text{rope}} = \\operatorname{RoPE}(Q), \\quad K_{\\text{rope}} = \\operatorname{RoPE}(K)$$\n\n2. **ELU+1 Linear Kernel Representation**:\n $$\\phi(x) = \\operatorname{ELU}(x) + 1$$\n $$O^{\\text{linear}} = \\frac{\\phi(Q_{\\text{rope}}) (\\phi(K_{\\text{rope}})^T V)}{\\phi(Q_{\\text{rope}}) (\\sum \\phi(K_{\\text{rope}})^T) + \\epsilon}$$\n\n3. **Adaptive Gated Blending**:\n $$g = \\sigma(X W_{\\text{gate}} + 1.8)$$\n $$\\text{Output} = W_O " | |
| }, | |
| { | |
| "id": "hamming_attention", | |
| "name": "Hamming Attention Distillation (HAD): Sign-Binarized Attention wi", | |
| "file": "Codes/framework/attentions/hamming_attention.py", | |
| "type": "Attention Module", | |
| "category": "Low-Rank SVD & Factorization", | |
| "paradigm": "svd", | |
| "icon": "\u26a1", | |
| "description": "Hamming Attention Distillation (HAD) Module. Approximates dense inner products with differentiable continuous relaxations of Hamming distances over sign-binarized representations with calibrated temperature scaling, multi-architecture weight transfer, trainable LoRA residual adapters, GQA, RoPE, and causal decoding. NOTE: This module is LoRA-only", | |
| "math": "Q_b = \\tanh(5.0 \\cdot \\gamma_Q \\odot Q) \\in [-1, 1]^{B \\times H \\times L \\times d_k}", | |
| "markdown_preview": "# Hamming Attention Distillation (HAD): Sign-Binarized Attention with Continuous Relaxations\n\n## 1. Executive Abstract & Architectural Motivation\n\nDense Transformer attention requires floating-point inner products $Q K^\\top \\in \\mathbb{R}^{L \\times L}$, consuming significant memory bandwidth and ALU power. **Hamming Attention Distillation (HAD)** replaces continuous floating-point dot products with **differentiable continuous relaxations of Hamming distances** over sign-binarized representations.\n\nHAD achieves:\n1. **Differentiable Pseudo-Binarization**: Uses a steep $\\tanh(\\beta \\gamma x)$ con" | |
| }, | |
| { | |
| "id": "hgrn2_gated_deltanet", | |
| "name": "HGRN-2 Gated DeltaNet Hybrid Attention Module.", | |
| "file": "Codes/framework/attentions/hgrn2_gated_deltanet.py", | |
| "type": "Attention Module", | |
| "category": "Linear Attention & State-Space (SSM)", | |
| "paradigm": "ssm", | |
| "icon": "\ud83d\udc0d", | |
| "description": "HGRN-2 Gated DeltaNet Hybrid Attention Module. Combines bidirectional/causal HGRN-2 element-wise linear recurrence with multi-head Gated DeltaNet associative matrix memory for rich representational capacity. Hardened across DistilBERT, RoBERTa, GPT-2, and Qwen3.5 architectures.", | |
| "math": "\\begin{aligned}\nY_{\\text{hgrn}} &= \\text{HGRN2}(X) \\\\\nY_{\\text{deltanet}} &= \\text{GatedDeltaNet}(X) \\\\\n\\gamma &= \\sigma(W_g X) \\\\\nY &= \\gamma \\odot Y_{\\text{hgrn}} + (1 - \\gamma) \\odot Y_{\\text{deltanet}} \\\\\n\\text{Output} &= Y W_o\n\\end{aligned}", | |
| "markdown_preview": "# HGRN-2 Gated DeltaNet Hybrid Architecture\n\n## 1. Executive Abstract\n**HGRN-2 Gated DeltaNet** is a hybrid attention module that fuses two linear recurrent paradigms:\n1. **HGRN-2 Element-Wise Branch**: High-frequency element-wise linear recurrence capturing fast local sequence dynamics with $\\mathcal{O}(L \\cdot D)$ complexity.\n2. **Gated DeltaNet Associative Memory Branch**: Multi-head matrix recurrence utilizing the Delta-rule update for associative memory retrieval.\n3. **Dynamic Sigmoid Gating**: Merges the two branches with input-dependent gating $\\gamma = \\sigma(W_g x)$.\n\n---\n\n## 2. Mathe" | |
| }, | |
| { | |
| "id": "hgrn2_layer", | |
| "name": "Hierarchically Gated Recurrent Network V2 (HGRN-2) Layer.", | |
| "file": "Codes/framework/attentions/hgrn2_layer.py", | |
| "type": "Attention Module", | |
| "category": "Linear Attention & State-Space (SSM)", | |
| "paradigm": "ssm", | |
| "icon": "\ud83d\udc0d", | |
| "description": "Hierarchically Gated Recurrent Network V2 (HGRN-2) Layer. Implements bidirectional/causal element-wise linear recurrence with dynamic input-dependent log-space forget gating, input modulation, LoRA adapters, and multi-architecture pretrained weight transfer.", | |
| "math": "f_t = \\exp(-\\operatorname{softplus}(W_f x_t + b_f)) \\in (0, 1)^D, \\quad b_f = 1.5", | |
| "markdown_preview": "# Hierarchically Gated Recurrent Network V2 (HGRN-2) Layer\n\n## 1. Executive Abstract & Theoretical Foundations\n\n**Hierarchically Gated Recurrent Network v2 (HGRN-2)** implements an element-wise linear recurrent layer that eliminates quadratic attention scaling $\\mathcal{O}(L^2)$ in favor of strictly linear $\\mathcal{O}(L \\cdot D)$ computation and memory.\n\nKey architectural foundations include:\n1. **Log-Space Bounded Retention Gating**: Employs $f_t = \\exp(-\\operatorname{softplus}(W_f x_t + b_f)) \\in (0, 1)^D$ to dynamically control memory retention with strict contractivity.\n2. **Dual Bidirect" | |
| }, | |
| { | |
| "id": "hgrn2_outer", | |
| "name": "HGRN-2 Outer-Product Recurrent Linear Attention Module.", | |
| "file": "Codes/framework/attentions/hgrn2_outer.py", | |
| "type": "Attention Module", | |
| "category": "Linear Attention & State-Space (SSM)", | |
| "paradigm": "ssm", | |
| "icon": "\ud83d\udc0d", | |
| "description": "HGRN-2 Outer-Product Recurrent Linear Attention Module. Implements multi-head bidirectional/causal outer-product associative matrix memory updates with log-space bounded decay gating, denominator normalization, and pretrained weight transfer. Hardened for production use across diverse transformer architectures: - DistilBERT (Flat MHA) - RoBERTa / ", | |
| "math": "Q = X W_q, \\quad K = X W_k, \\quad V = X W_v", | |
| "markdown_preview": "# HGRN-2 Outer-Product Recurrent Linear Attention\n\n## 1. Executive Abstract\n`HGRN2OuterBlock` implements multi-head bidirectional outer-product associative matrix memory updates with log-space bounded decay gating, denominator normalization, and pretrained weight transfer.\n\n---\n\n## 2. Mathematical Equations & Recurrence\n\n### 2.1 Multi-Head Projections & Bounded Forget Gating\n$$Q = X W_q, \\quad K = X W_k, \\quad V = X W_v$$\n$$\\tilde{q} = \\text{RMSNorm}(Q), \\quad \\tilde{k} = \\text{RMSNorm}(K) / \\sqrt{d_k}$$\n$$f_t = \\exp(-\\text{softplus}(W_f x_t + b_f)) \\in (0, 1)^{B \\times H}$$\n\n### 2.2 Bidirecti" | |
| }, | |
| { | |
| "id": "hla_attention", | |
| "name": "Hadamard Linear Attention (HLA).", | |
| "file": "Codes/framework/attentions/hla_attention.py", | |
| "type": "Attention Module", | |
| "category": "Linear Attention & State-Space (SSM)", | |
| "paradigm": "ssm", | |
| "icon": "\ud83d\udc0d", | |
| "description": "Hadamard Linear Attention (HLA). Multi-head bidirectional/causal linear recurrent attention with Hadamard associative kernelization, learnable decay gating, denominator normalization, and pretrained weight transfer. Hardened for production use across diverse transformer architectures: - DistilBERT (Flat MHA) - RoBERTa / BERT (Nested Encoder Self-A", | |
| "math": "\\phi(Q) = \\text{ELU}(X W_q) + 1, \\quad \\phi(K) = \\text{ELU}(X W_k) + 1, \\quad V = X W_v", | |
| "markdown_preview": "# Hadamard Linear Attention (HLA)\n\n## 1. Executive Abstract\n**Hadamard Linear Attention (HLA)** replaces the $\\mathcal{O}(L^2)$ softmax attention matrix with an associative multi-head linear recurrent state equipped with ELU+1 non-negative feature maps, learned decay gating $\\lambda \\in (0, 1)$, denominator normalization, and bidirectional sequence scanning.\n\n---\n\n## 2. Mathematical Equations & Recurrence\n\n### 2.1 ELU+1 Feature Mapping\n$$\\phi(Q) = \\text{ELU}(X W_q) + 1, \\quad \\phi(K) = \\text{ELU}(X W_k) + 1, \\quad V = X W_v$$\n\n### 2.2 Bidirectional Recurrence with Denominator Tracking\nFor deca" | |
| }, | |
| { | |
| "id": "hla_inject", | |
| "name": "Deprecated stub. Re-exports base module for registry compatibilit", | |
| "file": "Codes/framework/attentions/hla_inject.py", | |
| "type": "Attention Module", | |
| "category": "Dynamic & Efficient Attention", | |
| "paradigm": "sparse", | |
| "icon": "\ud83c\udf1f", | |
| "description": "Deprecated stub. Re-exports base module for registry compatibility.", | |
| "math": "\\phi(Q) = \\text{ELU}(X W_q) + 1, \\quad \\phi(K) = \\text{ELU}(X W_k) + 1, \\quad V = X W_v", | |
| "markdown_preview": "# Hadamard Linear Attention (HLA)\n\n## 1. Executive Abstract\n**Hadamard Linear Attention (HLA)** replaces the $\\mathcal{O}(L^2)$ softmax attention matrix with an associative multi-head linear recurrent state equipped with ELU+1 non-negative feature maps, learned decay gating $\\lambda \\in (0, 1)$, denominator normalization, and bidirectional sequence scanning.\n\n---\n\n## 2. Mathematical Equations & Recurrence\n\n### 2.1 ELU+1 Feature Mapping\n$$\\phi(Q) = \\text{ELU}(X W_q) + 1, \\quad \\phi(K) = \\text{ELU}(X W_k) + 1, \\quad V = X W_v$$\n\n### 2.2 Bidirectional Recurrence with Denominator Tracking\nFor deca" | |
| }, | |
| { | |
| "id": "hymba_transx", | |
| "name": "HymbaTransX: Hybrid Differential Transformer & State Space Model ", | |
| "file": "Codes/framework/attentions/hymba_transx.py", | |
| "type": "Attention Module", | |
| "category": "Latent & KV-Cache Compression", | |
| "paradigm": "mla", | |
| "icon": "\ud83e\udde0", | |
| "description": "HymbaTransX: Hybrid Differential Transformer & State Space Model Block (Production-Hardened). Differential attention with rotary embeddings coupled with a State Space Model branch. Supports: - Multi-architecture copy_weights (GPT-2, RoBERTa/BERT nested, Qwen/LLaMA, DistilBERT) - Trainable LoRA adapters with frozen base weights - GQA with repeat_k", | |
| "math": "\\mathbf{R}_{\\Theta, m}^{d_k} \\mathbf{x}_m = \\begin{pmatrix} x_{m, 0} \\cos(m \\theta_0) - x_{m, 1} \\sin(m \\theta_0) \\\\ x_{m, 0} \\sin(m \\theta_0) + x_{m, 1} \\cos(m \\theta_0) \\\\ \\vdots \\end{pmatrix}", | |
| "markdown_preview": "# HymbaTransX: Hybrid Differential Transformer & State Space Model Block\n\n## 1. Mathematical Formulation & Architectural Design\n\nThe **HymbaTransX (Hybrid Differential Transformer & State Space Model)** architecture fuses two complementary innovations in context modeling:\n1. **Differential Attention with Head-wise Rotary Position Embeddings (RoPE)**: Employs dual softmax maps to cancel out common-mode attention noise, eliminate attention sinks, and sharpen token-to-token semantic routing.\n2. **Bidirectional SimpleSSM State-Space Recurrence**: Provides linear-time global memory propagation runn" | |
| }, | |
| { | |
| "id": "infini_attention", | |
| "name": "Infini-Attention with Compressive Memory and Sliding Window (Prod", | |
| "file": "Codes/framework/attentions/infini_attention.py", | |
| "type": "Attention Module", | |
| "category": "Latent & KV-Cache Compression", | |
| "paradigm": "mla", | |
| "icon": "\ud83e\udde0", | |
| "description": "Infini-Attention with Compressive Memory and Sliding Window (Production-Hardened). Combines local sliding-window attention with learned compressive memory. Supports: - Multi-architecture copy_weights (GPT-2, RoBERTa/BERT nested, Qwen/LLaMA, DistilBERT) - Trainable LoRA adapters with frozen base weights - GQA with repeat_kv - RoPE position embeddi", | |
| "math": "\\mathbf{Y} = \\mathbf{Y}_{\\text{local}} + \\sigma(\\beta) \\mathbf{Y}_{\\text{mem}}", | |
| "markdown_preview": "# Infini-Attention\n\n## 1. Mathematical Formulation & Architecture\nCombines local sliding-window attention with compressive memory state update:\n$$\\mathbf{Y} = \\mathbf{Y}_{\\text{local}} + \\sigma(\\beta) \\mathbf{Y}_{\\text{mem}}$$\n$$\\mathbf{Y}_{\\text{mem}} = \\text{Softmax}\\left(\\frac{\\mathbf{Q} \\mathbf{M}^T}{\\sqrt{d}}\\right) \\mathbf{M}$$\n\n\n## Architecture & Data Flow\n\nThe `InfiniAttention` module coordinates input projection factorization, intermediate representation routing, and output synthesis:\n\n```\nInput Activation X [B, N, D]\n \u2502\n \u251c\u2500\u2500 Low-Rank Bottleneck Down-Projection: Z = X W_down [B, N" | |
| }, | |
| { | |
| "id": "jamba", | |
| "name": "Jamba-2 Hybrid Transformer: SVD-Mamba + MHA + MoE Attention Block", | |
| "file": "Codes/framework/attentions/jamba.py", | |
| "type": "Attention Module", | |
| "category": "Low-Rank SVD & Factorization", | |
| "paradigm": "svd", | |
| "icon": "\u26a1", | |
| "description": "Jamba-2 Hybrid Transformer: SVD-Mamba + MHA + MoE Attention Block (Production-Hardened). Combines: - SVD-Mamba selective state space branch - Multi-Head Attention branch with GQA, RoPE, and causal masking - Expert Choice MoE branch with auxiliary loss tracking Supports: - Multi-architecture copy_weights (GPT-2, RoBERTa/BERT nested, Qwen/LLaMA, Di", | |
| "math": "\\mathbf{X}_{\\text{proj}} = \\mathbf{X} \\mathbf{W}_{\\text{in}}, \\quad \\mathbf{W}_{\\text{in}} \\in \\mathbb{R}^{D \\times D}", | |
| "markdown_preview": "# Jamba-2 Hybrid Transformer: SVD-Mamba + MHA + Expert Choice MoE\n\n## 1. Mathematical Formulation & Architectural Design\n\nThe **Jamba-2 Hybrid Transformer** architecture (AI21 Labs design principles adapted for bidirectional encoder & decoder compression) harmoniously interleaves three distinct computational paradigms within a single unified block:\n1. **SVD-Mamba State-Space Recurrence**: Global sequence memory aggregation in linear $\\mathcal{O}(N)$ time.\n2. **Dense Bidirectional Multi-Head Attention (MHA)**: High-capacity all-to-all token interaction and precise syntactic resolution.\n3. **Exp" | |
| }, | |
| { | |
| "id": "jamba_info", | |
| "name": "Information-Bottlenecked Jamba Architecture with Dual Operational", | |
| "file": "Codes/framework/attentions/jamba_info.py", | |
| "type": "Attention Module", | |
| "category": "Linear Attention & State-Space (SSM)", | |
| "paradigm": "ssm", | |
| "icon": "\ud83d\udc0d", | |
| "description": "Information-Bottlenecked Jamba Architecture with Dual Operational Modes (Production-Hardened). Differential attention with concept-bottlenecked SSM branch. Supports: - Multi-architecture copy_weights (GPT-2, RoBERTa/BERT nested, Qwen/LLaMA, DistilBERT) - Trainable LoRA adapters with frozen base weights - GQA with repeat_kv - RoPE position embeddi", | |
| "math": "\\mathbf{Y} = \\mathbf{W}_O \\left(\\mathbf{A}_{\\text{diff}} \\mathbf{V} + 0.1 \\cdot \\text{SimpleSSM}(\\text{MLP}_{\\text{down-up}}(\\mathbf{X}))\\right)", | |
| "markdown_preview": "# Jamba-Info: Information-Bottlenecked Hybrid Transformer\n\n## 1. Mathematical Formulation & Architecture\nEmploys Differential Attention $\\mathbf{A}_{\\text{diff}} = \\text{Softmax}(\\mathbf{Q}_1 \\mathbf{K}_1^T) - 0.2 \\cdot \\text{Softmax}(\\mathbf{Q}_2 \\mathbf{K}_2^T)$ combined with a Concept Bottleneck SSM:\n$$\\mathbf{Y} = \\mathbf{W}_O \\left(\\mathbf{A}_{\\text{diff}} \\mathbf{V} + 0.1 \\cdot \\text{SimpleSSM}(\\text{MLP}_{\\text{down-up}}(\\mathbf{X}))\\right)$$\n\n\n## Architecture & Data Flow\n\nThe `InfoMambaConceptBottleneck` module coordinates input projection factorization, intermediate representation rou" | |
| }, | |
| { | |
| "id": "kda_attention", | |
| "name": "Kernelized Decay Attention (KDA) / Recurrent Linear Attention (Ha", | |
| "file": "Codes/framework/attentions/kda_attention.py", | |
| "type": "Attention Module", | |
| "category": "Linear Attention & State-Space (SSM)", | |
| "paradigm": "ssm", | |
| "icon": "\ud83d\udc0d", | |
| "description": "Kernelized Decay Attention (KDA) / Recurrent Linear Attention (Hardened). Implements multi-head linear recurrence with learned exponential decay gating, spherical normalization, denominator tracking, and pretrained weight transfer. Supports: - Multi-architecture copy_weights (GPT-2, RoBERTa/BERT nested, Qwen/LLaMA, DistilBERT) - Trainable LoRA ad", | |
| "math": "Q = \\text{normalize}(X W_q, \\text{dim}=-1), \\quad K = \\frac{\\text{normalize}(X W_k, \\text{dim}=-1)}{\\sqrt{d_k}}, \\quad V = X W_v", | |
| "markdown_preview": "# Kernelized Decay Attention (KDA)\n\n## 1. Executive Abstract\n**Kernelized Decay Attention (KDA)** establishes a multi-head bidirectional linear recurrence mechanism with learnable gating decay $\\lambda = \\sigma(\\beta)$, spherical $\\ell_2$ normalization of keys/queries, and denominator normalizer tracking.\n\n---\n\n## 2. Mathematical Equations & Recurrence\n\n### 2.1 Spherical Projections\n$$Q = \\text{normalize}(X W_q, \\text{dim}=-1), \\quad K = \\frac{\\text{normalize}(X W_k, \\text{dim}=-1)}{\\sqrt{d_k}}, \\quad V = X W_v$$\n\n### 2.2 Bidirectional Recurrence\n$$\\begin{aligned}\n\\vec{S}_t &= \\sigma(\\beta) \\v" | |
| }, | |
| { | |
| "id": "kda_inject", | |
| "name": "Deprecated stub. Re-exports base module for registry compatibilit", | |
| "file": "Codes/framework/attentions/kda_inject.py", | |
| "type": "Attention Module", | |
| "category": "Dynamic & Efficient Attention", | |
| "paradigm": "sparse", | |
| "icon": "\ud83c\udf1f", | |
| "description": "Deprecated stub. Re-exports base module for registry compatibility.", | |
| "math": "Q = \\text{normalize}(X W_q, \\text{dim}=-1), \\quad K = \\frac{\\text{normalize}(X W_k, \\text{dim}=-1)}{\\sqrt{d_k}}, \\quad V = X W_v", | |
| "markdown_preview": "# Kernelized Decay Attention (KDA)\n\n## 1. Executive Abstract\n**Kernelized Decay Attention (KDA)** establishes a multi-head bidirectional linear recurrence mechanism with learnable gating decay $\\lambda = \\sigma(\\beta)$, spherical $\\ell_2$ normalization of keys/queries, and denominator normalizer tracking.\n\n---\n\n## 2. Mathematical Equations & Recurrence\n\n### 2.1 Spherical Projections\n$$Q = \\text{normalize}(X W_q, \\text{dim}=-1), \\quad K = \\frac{\\text{normalize}(X W_k, \\text{dim}=-1)}{\\sqrt{d_k}}, \\quad V = X W_v$$\n\n### 2.2 Bidirectional Recurrence\n$$\\begin{aligned}\n\\vec{S}_t &= \\sigma(\\beta) \\v" | |
| }, | |
| { | |
| "id": "leka_v1", | |
| "name": "LEKAv1: Linear-time Efficient Kernel Attention (Production-Harden", | |
| "file": "Codes/framework/attentions/leka_v1.py", | |
| "type": "Attention Module", | |
| "category": "Latent & KV-Cache Compression", | |
| "paradigm": "mla", | |
| "icon": "\ud83e\udde0", | |
| "description": "LEKAv1: Linear-time Efficient Kernel Attention (Production-Hardened). Mathematical Mechanism: Approximates the Gaussian RBF kernel via Random Fourier Feature (RFF) mapping: Supports: - Multi-architecture copy_weights (GPT-2, RoBERTa/BERT nested, Qwen/LLaMA, DistilBERT) - Trainable LoRA adapters with frozen base weights - GQA wi", | |
| "math": "\\mathbf{O}_i = \\sum_{j=1}^N \\frac{\\exp\\left(\\frac{\\mathbf{q}_i \\mathbf{k}_j^\\top}{\\sqrt{d_k}}\\right)}{\\sum_{m=1}^N \\exp\\left(\\frac{\\mathbf{q}_i \\mathbf{k}_m^\\top}{\\sqrt{d_k}}\\right)} \\mathbf{v}_j", | |
| "markdown_preview": "# LEKA v1: Linear-time Efficient Kernel Attention\n\n## 1. Mathematical Formulation & Architectural Design\n\nThe **LEKA v1 (Linear-time Efficient Kernel Attention)** architecture replaces standard quadratic $\\mathcal{O}(N^2)$ dot-product attention with a linearized kernel feature map based on **Random Fourier Features (RFF)** and positive non-linear activations. By exploiting the associative property of matrix multiplication, LEKA v1 computes exact bidirectional contextual representations in strictly linear $\\mathcal{O}(N)$ time and $\\mathcal{O}(1)$ sequence memory.\n\n```\n " | |
| }, | |
| { | |
| "id": "leka_v1_inject", | |
| "name": "Deprecated stub. Re-exports base module for registry compatibilit", | |
| "file": "Codes/framework/attentions/leka_v1_inject.py", | |
| "type": "Attention Module", | |
| "category": "Dynamic & Efficient Attention", | |
| "paradigm": "sparse", | |
| "icon": "\ud83c\udf1f", | |
| "description": "Deprecated stub. Re-exports base module for registry compatibility.", | |
| "math": "\\mathbf{O}_i = \\sum_{j=1}^N \\frac{\\exp\\left(\\frac{\\mathbf{q}_i \\mathbf{k}_j^\\top}{\\sqrt{d_k}}\\right)}{\\sum_{m=1}^N \\exp\\left(\\frac{\\mathbf{q}_i \\mathbf{k}_m^\\top}{\\sqrt{d_k}}\\right)} \\mathbf{v}_j", | |
| "markdown_preview": "# LEKA v1: Linear-time Efficient Kernel Attention\n\n## 1. Mathematical Formulation & Architectural Design\n\nThe **LEKA v1 (Linear-time Efficient Kernel Attention)** architecture replaces standard quadratic $\\mathcal{O}(N^2)$ dot-product attention with a linearized kernel feature map based on **Random Fourier Features (RFF)** and positive non-linear activations. By exploiting the associative property of matrix multiplication, LEKA v1 computes exact bidirectional contextual representations in strictly linear $\\mathcal{O}(N)$ time and $\\mathcal{O}(1)$ sequence memory.\n\n```\n " | |
| }, | |
| { | |
| "id": "leka_v2", | |
| "name": "LEKAv2: Linear-time Efficient Kernel Attention with Landmark Dist", | |
| "file": "Codes/framework/attentions/leka_v2.py", | |
| "type": "Attention Module", | |
| "category": "Latent & KV-Cache Compression", | |
| "paradigm": "mla", | |
| "icon": "\ud83e\udde0", | |
| "description": "LEKAv2: Linear-time Efficient Kernel Attention with Landmark Distillation (Hardened). Supports: - Multi-architecture copy_weights (GPT-2, RoBERTa/BERT nested, Qwen/LLaMA, DistilBERT) - Trainable LoRA adapters with frozen base weights - GQA with repeat_kv - RoPE position embeddings - Exact causal masking for decoders - Nested encoder bypass - Frobe", | |
| "math": "\\phi_1(\\mathbf{x}) = \\frac{\\left[\\cos(\\mathbf{x} \\mathbf{W}_1), \\; \\sin(\\mathbf{x} \\mathbf{W}_1)\\right]}{\\sqrt{F_1}}, \\quad \\phi_2(\\mathbf{x}) = \\frac{\\left[\\cos(\\mathbf{x} \\mathbf{W}_2), \\; \\sin(\\mathbf{x} \\mathbf{W}_2)\\right]}{\\sqrt{F_2}}", | |
| "markdown_preview": "# LEKA v2: Landmark-Enhanced Kernel Attention with Dual-Scale Distillation\n\n## 1. Mathematical Formulation & Architectural Design\n\nThe **LEKA v2 (Landmark-Enhanced Kernel Attention)** architecture combines multi-scale random feature projections with **learned landmark token distillation** and a **gated local sliding-window attention stream**. While standard linear kernel attention compresses the entire sequence into a single global summary matrix, LEKA v2 extracts $L_{\\max}$ highly salient \"landmark\" key-value vectors that capture sharp semantic extrema, while the local sliding window preserve" | |
| }, | |
| { | |
| "id": "leka_v2_inject", | |
| "name": "LEKA v2: Landmark-Enhanced Kernel Attention with Dual-Scale Disti", | |
| "file": "Codes/framework/attentions/leka_v2_inject.py", | |
| "type": "Attention Module", | |
| "category": "Dynamic & Efficient Attention", | |
| "paradigm": "sparse", | |
| "icon": "\ud83c\udf1f", | |
| "description": "Deprecated stub. Re-exports base module for registry compatibility.", | |
| "math": "\\phi_1(\\mathbf{x}) = \\frac{\\left[\\cos(\\mathbf{x} \\mathbf{W}_1), \\; \\sin(\\mathbf{x} \\mathbf{W}_1)\\right]}{\\sqrt{F_1}}, \\quad \\phi_2(\\mathbf{x}) = \\frac{\\left[\\cos(\\mathbf{x} \\mathbf{W}_2), \\; \\sin(\\mathbf{x} \\mathbf{W}_2)\\right]}{\\sqrt{F_2}}", | |
| "markdown_preview": "# LEKA v2: Landmark-Enhanced Kernel Attention with Dual-Scale Distillation\n\n## 1. Mathematical Formulation & Architectural Design\n\nThe **LEKA v2 (Landmark-Enhanced Kernel Attention)** architecture combines multi-scale random feature projections with **learned landmark token distillation** and a **gated local sliding-window attention stream**. While standard linear kernel attention compresses the entire sequence into a single global summary matrix, LEKA v2 extracts $L_{\\max}$ highly salient \"landmark\" key-value vectors that capture sharp semantic extrema, while the local sliding window preserve" | |
| }, | |
| { | |
| "id": "leka_v3", | |
| "name": "LEKAv3: Linear-time Efficient Kernel Attention with Value-Scored ", | |
| "file": "Codes/framework/attentions/leka_v3.py", | |
| "type": "Attention Module", | |
| "category": "Latent & KV-Cache Compression", | |
| "paradigm": "mla", | |
| "icon": "\ud83e\udde0", | |
| "description": "LEKAv3: Linear-time Efficient Kernel Attention with Value-Scored Landmarks (Hardened). Supports: - Multi-architecture copy_weights (GPT-2, RoBERTa/BERT nested, Qwen/LLaMA, DistilBERT) - Trainable LoRA adapters with frozen base weights - GQA with repeat_kv - RoPE position embeddings - Exact causal masking for decoders - Nested encoder bypass - Frob", | |
| "math": "\\mathbf{s} = \\operatorname{MLP}_{\\text{scorer}}(\\mathbf{V}) = \\mathbf{W}_2 \\operatorname{GELU}(\\mathbf{V} \\mathbf{W}_1 + \\mathbf{b}_1) + \\mathbf{b}_2 \\in \\mathbb{R}^{B \\times H \\times N}", | |
| "markdown_preview": "# LEKA v3: Value-Aware Landmark Kernel Attention with Padding Invariance\n\n## 1. Mathematical Formulation & Architectural Design\n\nThe **LEKA v3 (Value-Aware Landmark Kernel Attention)** architecture advances landmark kernel distillation by introducing **value-aware saliency scoring**, **robust boolean padding mask filtering**, and **numerically stabilized Softpick activation**. In natural language sequences with variable-length padding, previous landmark attention models suffered from selecting meaningless padding tokens into the landmark memory set, leading to NaN float overflows and classific" | |
| }, | |
| { | |
| "id": "leka_v3_inject", | |
| "name": "LEKA v3: Value-Aware Landmark Kernel Attention with Padding Invar", | |
| "file": "Codes/framework/attentions/leka_v3_inject.py", | |
| "type": "Attention Module", | |
| "category": "Dynamic & Efficient Attention", | |
| "paradigm": "sparse", | |
| "icon": "\ud83c\udf1f", | |
| "description": "Deprecated stub. Re-exports base module for registry compatibility.", | |
| "math": "\\mathbf{s} = \\operatorname{MLP}_{\\text{scorer}}(\\mathbf{V}) = \\mathbf{W}_2 \\operatorname{GELU}(\\mathbf{V} \\mathbf{W}_1 + \\mathbf{b}_1) + \\mathbf{b}_2 \\in \\mathbb{R}^{B \\times H \\times N}", | |
| "markdown_preview": "# LEKA v3: Value-Aware Landmark Kernel Attention with Padding Invariance\n\n## 1. Mathematical Formulation & Architectural Design\n\nThe **LEKA v3 (Value-Aware Landmark Kernel Attention)** architecture advances landmark kernel distillation by introducing **value-aware saliency scoring**, **robust boolean padding mask filtering**, and **numerically stabilized Softpick activation**. In natural language sequences with variable-length padding, previous landmark attention models suffered from selecting meaningless padding tokens into the landmark memory set, leading to NaN float overflows and classific" | |
| }, | |
| { | |
| "id": "linformer_attention", | |
| "name": "Linformer Attention: Linear-Complexity Sequence Projection via Lo", | |
| "file": "Codes/framework/attentions/linformer_attention.py", | |
| "type": "Attention Module", | |
| "category": "Latent & KV-Cache Compression", | |
| "paradigm": "mla", | |
| "icon": "\ud83e\udde0", | |
| "description": "Linformer Attention \u2013 Production-hardened with full multi-architecture support. Supports: - Multi-architecture copy_weights (GPT-2, RoBERTa/BERT nested, Qwen/LLaMA, DistilBERT) - Trainable LoRA adapters with frozen base weights - GQA with repeat_kv - RoPE position embeddings - Causal masking for decoders - Nested encoder bypass - Frobenius norm ve", | |
| "math": "c^{KV}_t = W^{DKV} x_t, \\quad k^C_t = W^{UK} c^{KV}_t, \\quad v^C_t = W^{UV} c^{KV}_t", | |
| "markdown_preview": "# Linformer Attention: Linear-Complexity Sequence Projection via Low-Rank Matrix Factorization\n\n## 1. Executive Abstract & Architectural Motivation\n\nStandard Multi-Head Self-Attention (Vaswani et al., 2017) computes pairwise token interactions across length $L$, incurring an $\\mathcal{O}(L^2)$ computational and memory complexity wall. Linformer (Wang et al., 2020) mitigates this bottleneck by proving empirically and theoretically that the normalized attention matrix $A \\in \\mathbb{R}^{L \times L}$ is low-rank. \n\nThe core architectural motivation of Linformer is to decompose the key and value se" | |
| }, | |
| { | |
| "id": "lola_attention", | |
| "name": "Lola Attention: Low-Rank Localized Attention via Three-Memory Sys", | |
| "file": "Codes/framework/attentions/lola_attention.py", | |
| "type": "Attention Module", | |
| "category": "Low-Rank SVD & Factorization", | |
| "paradigm": "svd", | |
| "icon": "\u26a1", | |
| "description": "LoLA: Low-Rank Linear Attention with Sparse Caching. Combines factorized low-rank projections with selective multi-system cached retrieval, trainable LoRA residual adapters, frozen SVD base weights, GQA, RoPE, and causal decoding.", | |
| "math": "W \\approx U V, \\quad U \\in \\mathbb{R}^{d_{\\text{out}} \\times r}, \\; V \\in \\mathbb{R}^{r \\times d_{\\text{in}}}", | |
| "markdown_preview": "# Lola Attention: Low-Rank Localized Attention via Three-Memory System Caching\n\n## 1. Executive Abstract & Architectural Motivation\n\nAutoregressive Transformer inference over long context sequences is heavily memory-bandwidth bound due to key-value (KV) cache accumulation. Low-Rank Localized Attention (LoLA Attention) mitigates KV cache memory footprint using a multi-tier memory system cache partition.\n\nLoLA Attention partitions historical token context into three distinct memory systems: (1) a **Recent Cache** sliding window $\\mathcal{C}_{\\text{recent}}$ for local high-resolution tokens, (2) " | |
| }, | |
| { | |
| "id": "lora_attention", | |
| "name": "LoRA Attention: Parameter-Efficient Low-Rank Adaptation", | |
| "file": "Codes/framework/attentions/lora_attention.py", | |
| "type": "Attention Module", | |
| "category": "Low-Rank SVD & Factorization", | |
| "paradigm": "svd", | |
| "icon": "\u26a1", | |
| "description": "LoRA Parameter-Efficient Fine-Tuning Attention Module. Replaces standard query, key, value, and output linear projections with frozen pretrained base weights and trainable low-rank residual adapters, with multi-architecture weight transfer, GQA, RoPE, and causal decoding.", | |
| "math": "y = W_0 x + \\frac{\\alpha}{r} B A x", | |
| "markdown_preview": "# LoRA Attention: Parameter-Efficient Low-Rank Adaptation\n\n## 1. Abstract & Theoretical Motivation\nFine-tuning dense multi-head self-attention layers in large transformer models requires updating and storing millions or billions of parameters $\\mathbf{W}_0 \\in \\mathbb{R}^{D_{\\text{out}} \\times D_{\\text{in}}}$, demanding massive memory footprint during training. Low-Rank Adaptation (LoRA) Attention (Hu et al., 2021) overcomes this overhead by exploiting the intrinsic low-rank structure of model weight updates.\n\nLoRA freezes the pretrained projection weight matrices $\\mathbf{W}_0$ and injects tr" | |
| }, | |
| { | |
| "id": "lsh_attention", | |
| "name": "Locality-Sensitive Hashing Attention: Angular Random Hyperplane B", | |
| "file": "Codes/framework/attentions/lsh_attention.py", | |
| "type": "Attention Module", | |
| "category": "Structured Pruning & Channel Slicing", | |
| "paradigm": "pruning", | |
| "icon": "\u2702\ufe0f", | |
| "description": "Locality-Sensitive Hashing (LSH) Attention \u2013 Reformer style. Approximates dense attention by clustering keys and queries into discrete hash buckets via angular random hyperplane projections with fallback context, multi-architecture weight transfer, trainable LoRA residual adapters, GQA, RoPE, and causal decoding.", | |
| "math": "\\mathbf{Q} = \\mathbf{X} \\mathbf{W}_Q, \\quad \\mathbf{K} = \\mathbf{X} \\mathbf{W}_K, \\quad \\mathbf{V} = \\mathbf{X} \\mathbf{W}_V \\quad \\in \\mathbb{R}^{B \\times H \\times N \\times d_k}", | |
| "markdown_preview": "# Locality-Sensitive Hashing Attention: Angular Random Hyperplane Bucket Partitioning\n\n## 1. Executive Abstract & Architectural Motivation\nStandard Multi-Head Self-Attention (MHSA) incurs quadratic time and memory complexity $\\mathcal{O}(N^2 d)$ across sequence length $N$. However, the softmax probability distribution in attention matrices is heavily dominated by a small subset of high-score query-key dot products $\\mathbf{q}_i \\mathbf{k}_j^\\top$. The vast majority of pairwise interactions yield near-zero softmax probabilities after exponentiation, creating massive redundant computation.\n\n**Lo" | |
| }, | |
| { | |
| "id": "mamba", | |
| "name": "Mamba: Gated State Space Model (Mamba) linear attention replaceme", | |
| "file": "Codes/framework/attentions/mamba.py", | |
| "type": "Attention Module", | |
| "category": "Linear Attention & State-Space (SSM)", | |
| "paradigm": "ssm", | |
| "icon": "\ud83d\udc0d", | |
| "description": "Mamba: Gated State Space Model (Mamba) linear attention replacement (Production-Hardened). Implements a gated selective state-space projection that is a drop-in replacement for a standard Multi-Head Attention layer, operating in strictly O(L) time and space with respect to sequence length, with LoRA adapters, multi-architecture weight extraction, ", | |
| "math": "[U, Z] = \\operatorname{chunk}(X W_{\\text{in}}, 2) \\in \\mathbb{R}^{B \\times L \\times D}", | |
| "markdown_preview": "# Bidirectional Selective State Space Model (Bi-Mamba) Attention\n\n## 1. Executive Abstract & Theoretical Foundations\n\nStandard Multi-Head Attention scales quadratically $\\mathcal{O}(L^2)$ with sequence length $L$, creating computational and memory bottlenecks. While selective State Space Models (Mamba) achieve strictly linear $\\mathcal{O}(L)$ time and memory complexity, directly deploying a causal unidirectional SSM onto masked language models (such as DistilBERT on SST-2) causes catastrophic representation collapse (stalling at 50.92% random guessing).\n\n**Bidirectional Mamba (Bi-Mamba)** reso" | |
| }, | |
| { | |
| "id": "mamba_inject", | |
| "name": "Mamba Injection Module.", | |
| "file": "Codes/framework/attentions/mamba_inject.py", | |
| "type": "Attention Module", | |
| "category": "Linear Attention & State-Space (SSM)", | |
| "paradigm": "ssm", | |
| "icon": "\ud83d\udc0d", | |
| "description": "Mamba Injection Module. Provides injection hooks to wire Bidirectional Mamba into transformer layers with automated weight copying from pretrained self-attention projections.", | |
| "math": "h_t = \\bar{A} h_{t-1} + \\bar{B} x_t, \\quad y_t = C h_t + D x_t, \\quad \\bar{A} = \\exp(\\Delta A)", | |
| "markdown_preview": "# Mamba Inject\n\n## Overview\n## Configuration & Hyperparameters\n\n| Parameter | Description |\n|-----------|-------------|\n\n## Usage Examples\n\n```python\nfrom Codes.framework.attentions import Module\n```\n\n## Benchmark Results\n\n| Status | Accuracy | Val Loss | Peak VRAM | Inference Latency |\n|--------|----------|----------|-----------|-------------------|\n| Pending| Pending | Pending | Pending | Pending |\n\n## Limitations & Future Work\n\nTODO: Identify specific limitations and edge cases for this module.\n" | |
| }, | |
| { | |
| "id": "mamba_svd_flash", | |
| "name": "MambaSVD-Flash: Hybrid Bidirectional SSM-FlashAttention Module (P", | |
| "file": "Codes/framework/attentions/mamba_svd_flash.py", | |
| "type": "Attention Module", | |
| "category": "Low-Rank SVD & Factorization", | |
| "paradigm": "svd", | |
| "icon": "\u26a1", | |
| "description": "MambaSVD-Flash: Hybrid Bidirectional SSM-FlashAttention Module (Production-Hardened). Fuses bidirectional selective state-space recurrent dynamics with online Singular Value Decomposition of state transitions, coupled with a Multi-Head / GQA FlashAttention execution path, pretrained weight transfer, LoRA adapters, RoPE rotary embeddings, and caus", | |
| "math": "U_t = \\text{proj}_u(x_t) \\in \\mathbb{R}^{D \\times r}, \\quad V_t = \\text{proj}_v(x_t) \\in \\mathbb{R}^{r \\times D}", | |
| "markdown_preview": "# MambaSVD-Flash: Hybrid Bidirectional SSM-FlashAttention Architecture\n\n## 1. Executive Abstract & Architecture Motivation\nPure dense attention suffers from quadratic $\\mathcal{O}(L^2)$ computational complexity, while pure linear State Space Models struggle with high-entropy associative memory queries. **MambaSVD-Flash** introduces a dual-path hybrid architecture:\n1. **Bidirectional Low-Rank SVD-SSM Path**: Evaluates contractive online Singular Value Decomposition state-space transitions ($A_t = U_t \\Sigma_t V_t^\\top$) scanning both forward and backward directions with $\\mathcal{O}(L \\cdot D \\" | |
| }, | |
| { | |
| "id": "minimal_aurora", | |
| "name": "MinimalAURORA Attention: Energy-Preserving SVD-Factorized Attenti", | |
| "file": "Codes/framework/attentions/minimal_aurora.py", | |
| "type": "Attention Module", | |
| "category": "Low-Rank SVD & Factorization", | |
| "paradigm": "svd", | |
| "icon": "\u26a1", | |
| "description": "SVD-Factorized Linear Transformation with LoRA Residual Branch.", | |
| "math": "W_0 = U \\Sigma V^T, \\quad \\Sigma = \\operatorname{diag}(\\sigma_1, \\sigma_2, \\dots, \\sigma_{\\min(d_{\\text{out}}, d_{\\text{in}})})", | |
| "markdown_preview": "# MinimalAURORA Attention: Energy-Preserving SVD-Factorized Attention with Cumulative Spectral Truncation\n\n## 1. Abstract & Theoretical Motivation\n\nStandard Multi-Head Self-Attention (MHSA) architectures require dense projection matrices ($W_Q, W_K, W_V, W_O \\in \\mathbb{R}^{D \\times D}$) that introduce significant memory allocation and floating-point computation overhead during training and inference. Standard Low-Rank Adaptation (LoRA) uses static rank $r$ across all layers, ignoring the empirical singular value distribution of pre-trained parameters.\n\n**MinimalAURORA Attention** implements a" | |
| }, | |
| { | |
| "id": "minimal_nova", | |
| "name": "MinimalNOVA Attention: SVD-Factorized Low-Rank Projection Attenti", | |
| "file": "Codes/framework/attentions/minimal_nova.py", | |
| "type": "Attention Module", | |
| "category": "Low-Rank SVD & Factorization", | |
| "paradigm": "svd", | |
| "icon": "\u26a1", | |
| "description": "SVD-Factorized Linear Transformation for NOVA Architecture.", | |
| "math": "W_0 = U \\Sigma V^T, \\quad \\Sigma = \\operatorname{diag}(\\sigma_1, \\sigma_2, \\dots, \\sigma_{\\min(d_{\\text{out}}, d_{\\text{in}})})", | |
| "markdown_preview": "# MinimalNOVA Attention: SVD-Factorized Low-Rank Projection Attention for Efficient Transformer Compression\n\n## 1. Abstract & Theoretical Motivation\n\nStandard Multi-Head Self-Attention (MHSA) in Transformer architectures requires dense projection matrices ($W_Q, W_K, W_V, W_O \\in \\mathbb{R}^{D \\times D}$) that introduce heavy parameter memory footprints and high matrix multiplication FLOP counts. Standard low-rank compression methods apply static projection ranks across all network layers, ignoring the empirical singular value distribution of pre-trained parameters.\n\nTo circumvent the paramete" | |
| }, | |
| { | |
| "id": "minimal_prism", | |
| "name": "MinimalPRISM Attention: Singular Spectrum Energy-Factorized Low-R", | |
| "file": "Codes/framework/attentions/minimal_prism.py", | |
| "type": "Attention Module", | |
| "category": "Low-Rank SVD & Factorization", | |
| "paradigm": "svd", | |
| "icon": "\u26a1", | |
| "description": "SVD-Factorized Linear Layer for PRISM Attention Architecture.", | |
| "math": "W_0 = U \\Sigma V^T, \\quad \\Sigma = \\operatorname{diag}(\\sigma_1, \\sigma_2, \\dots, \\sigma_{\\min(d_{\\text{out}}, d_{\\text{in}})})", | |
| "markdown_preview": "# MinimalPRISM Attention: Singular Spectrum Energy-Factorized Low-Rank Projection Attention\n\n## 1. Abstract & Theoretical Motivation\n\nStandard Multi-Head Self-Attention (MHSA) in Transformer architectures requires dense linear projection matrices ($W_Q, W_K, W_V, W_O \\in \\mathbb{R}^{D \\times D}$) that introduce heavy parameter memory footprints and high matrix multiplication FLOP counts. Standard low-rank compression methods apply static projection ranks across all network layers, ignoring the empirical singular value spectrum of pre-trained parameters.\n\nTo mitigate these inefficiencies, **Min" | |
| }, | |
| { | |
| "id": "minimal_svd", | |
| "name": "MinimalSVD: Cumulative Energy-Budgeted SVD Factorization Attentio", | |
| "file": "Codes/framework/attentions/minimal_svd.py", | |
| "type": "Attention Module", | |
| "category": "Low-Rank SVD & Factorization", | |
| "paradigm": "svd", | |
| "icon": "\u26a1", | |
| "description": "SVD-Factorized Linear Transformation for NOVA Architecture.", | |
| "math": "A = U_{:, :r} \\text{diag}(\\sqrt{S_{:r}}) \\in \\mathbb{R}^{\\text{out} \\times r}, \\quad B = \\text{diag}(\\sqrt{S_{:r}}) V_{:r, :} \\in \\mathbb{R}^{r \\times \\text{in}}", | |
| "markdown_preview": "# MinimalSVD: Cumulative Energy-Budgeted SVD Factorization Attention\n\n## 1. Executive Abstract & Architectural Motivation\n\nMinimalSVD (`MinimalSVDAttention`) provides a zero-shot model compression mechanism for Transformer attention layers. In standard attention architectures, linear projection matrices $W_Q, W_K, W_V, W_O \\in \\mathbb{R}^{D \\times D}$ consume substantial parameter storage ($4 D^2$) and demand high memory bandwidth during inference.\n\nMinimalSVD factorizes each pre-trained linear projection matrix $W \\approx A B$ using Truncated Singular Value Decomposition (SVD), where $B \\in \\" | |
| }, | |
| { | |
| "id": "minimal_svd_v2", | |
| "name": "MinimalSVD v2: Refined SVD Low-Rank Attention with Flexible Atten", | |
| "file": "Codes/framework/attentions/minimal_svd_v2.py", | |
| "type": "Attention Module", | |
| "category": "Low-Rank SVD & Factorization", | |
| "paradigm": "svd", | |
| "icon": "\u26a1", | |
| "description": "SVD-Factorized Linear Transformation for NOVA Architecture.", | |
| "math": "W = U \\Sigma V^\\top \\approx U_{:, :r} \\Sigma_{:r} V_{:r, :}^\\top = A B", | |
| "markdown_preview": "# MinimalSVD v2: Refined SVD Low-Rank Attention with Flexible Attention Mask Handling\n\n## 1. Executive Abstract & Architectural Motivation\n\nMinimalSVD v2 (`MinimalSVDAttentionV2`) provides a refined, zero-shot SVD-factorized attention module for Transformer architectures. While standard Multi-Head Attention employs dense linear projections ($W_Q, W_K, W_V, W_O \\in \\mathbb{R}^{D \\times D}$) that demand significant memory bandwidth, MinimalSVD v2 decomposes each linear parameter matrix into low-rank factor pairs ($A \\in \\mathbb{R}^{D \\times r}, B \\in \\mathbb{R}^{r \\times D}$) using truncated Sin" | |
| }, | |
| { | |
| "id": "mixattention_moska", | |
| "name": "MixAttention-MoSKA: Frequency/Variance-Aware Sparse Attention (FA", | |
| "file": "Codes/framework/attentions/mixattention_moska.py", | |
| "type": "Attention Module", | |
| "category": "Dynamic & Efficient Attention", | |
| "paradigm": "sparse", | |
| "icon": "\ud83c\udf1f", | |
| "description": "MixAttention-MoSKA: Frequency/Variance-Aware Sparse Attention (FASA). Hardened for multi-architecture production deployment across DistilBERT, RoBERTa, GPT-2, and Qwen3.5.", | |
| "math": "\\operatorname{Var}(\\mathbf{K}_{b, h, t}) = \\frac{1}{d_k} \\sum_{c=1}^{d_k} \\left(K_{b, h, t, c} - \\bar{K}_{b, h, t}\\right)^2", | |
| "markdown_preview": "# MixAttention-MoSKA: Frequency/Variance-Aware Sparse Attention (FASA)\n\n## 1. Mathematical Formulation & Architectural Design\n\nThe **MixAttention-MoSKA (Frequency/Variance-Aware Sparse Attention - FASA)** architecture reduces the quadratic attention complexity of Transformers by dynamically pruning uninformative Key-Value tokens based on their **channel-wise spectral variance**. While standard static pruning techniques discard tokens arbitrarily or rely on expensive learned routers, MoSKA exploits the intrinsic information-theoretic entropy embedded in the variance of key channel activations, " | |
| }, | |
| { | |
| "id": "mla_attention", | |
| "name": "Multi-Head Latent Attention (MLA): Low-Rank Latent Bottleneck Com", | |
| "file": "Codes/framework/attentions/mla_attention.py", | |
| "type": "Attention Module", | |
| "category": "Latent & KV-Cache Compression", | |
| "paradigm": "mla", | |
| "icon": "\ud83e\udde0", | |
| "description": "Multi-Head Latent Attention (MLA) \u2013 DeepSeek-V2. Paper: https://arxiv.org/abs/2405.04434 Hardened for multi-architecture production deployment across DistilBERT, RoBERTa, GPT-2, and Qwen3.5.", | |
| "math": "\\mathbf{c}_{KV} = W_{DKV} X \\in \\mathbb{R}^{B \\times N \\times d_c}", | |
| "markdown_preview": "# Multi-Head Latent Attention (MLA): Low-Rank Latent Bottleneck Compression (DeepSeek-V2)\n\n## 1. Executive Abstract & Architectural Motivation\n\nMulti-Head Latent Attention (MLA), introduced in DeepSeek-V2 (DeepSeek-AI, 2024), represents a major architectural advance in compressing the Key-Value (KV) cache for large-scale autoregressive Transformer inference. Traditional approaches like Grouped-Query Attention (GQA) reduce KV cache overhead by sharing key-value heads across query groups. However, GQA still caches full head-dimensional Key and Value tensors for each shared head, creating substan" | |
| }, | |
| { | |
| "id": "moba", | |
| "name": "MoBA: Mixture of Block Attention for Long-Context Modeling.", | |
| "file": "Codes/framework/attentions/moba.py", | |
| "type": "Attention Module", | |
| "category": "Dynamic & Efficient Attention", | |
| "paradigm": "sparse", | |
| "icon": "\ud83c\udf1f", | |
| "description": "MoBA: Mixture of Block Attention for Long-Context Modeling. Module alias re-exporting MoBAAttention and inject_moba from moba_attention.", | |
| "math": "A_{ij} = \\text{TopK}\\left(\\text{Softmax}\\left(\\frac{Q_i K_j^\\top}{\\sqrt{d}}\\right)\\right)", | |
| "markdown_preview": "# Moba\n\n## Overview\n## Configuration & Hyperparameters\n\n| Parameter | Description |\n|-----------|-------------|\n\n## Usage Examples\n\n```python\nfrom Codes.framework.attentions import Module\n```\n\n## Benchmark Results\n\n| Status | Accuracy | Val Loss | Peak VRAM | Inference Latency |\n|--------|----------|----------|-----------|-------------------|\n| Pending| Pending | Pending | Pending | Pending |\n\n## Limitations & Future Work\n\nTODO: Identify specific limitations and edge cases for this module.\n" | |
| }, | |
| { | |
| "id": "moba_attention", | |
| "name": "MoBA Attention: Mixture-of-Block Attention via Centroid Cosine Ro", | |
| "file": "Codes/framework/attentions/moba_attention.py", | |
| "type": "Attention Module", | |
| "category": "Dynamic & Efficient Attention", | |
| "paradigm": "sparse", | |
| "icon": "\ud83c\udf1f", | |
| "description": "MoBA: Mixture of Block Attention for Long-Context Modeling. Hardened for production across diverse architectures (DistilBERT, RoBERTa, GPT-2, Qwen3.5).", | |
| "math": "\\mathbf{Q} = \\mathbf{X} \\mathbf{W}_Q, \\quad \\mathbf{K} = \\mathbf{X} \\mathbf{W}_K, \\quad \\mathbf{V} = \\mathbf{X} \\mathbf{W}_V", | |
| "markdown_preview": "# MoBA Attention: Mixture-of-Block Attention via Centroid Cosine Routing\n\n## 1. Executive Abstract & Architectural Motivation\nStandard Multi-Head Self-Attention (MHSA) computes dense pairwise token interactions across sequence length $L$, incurring quadratic computational time complexity $\\mathcal{O}(L^2 d)$ and memory footprint $\\mathcal{O}(L^2 + L d)$. For long-context language modeling, this quadratic bottleneck severely restricts context length. **MoBA (Mixture-of-Block Attention)** addresses this limitation by reformulating fine-grained token-to-token attention as a two-stage hierarchical" | |
| }, | |
| { | |
| "id": "moe_deltakv_paged", | |
| "name": "MoE-DeltaKV Paged: Quantized & Paged KV-Cache Mixture-of-Experts ", | |
| "file": "Codes/framework/attentions/moe_deltakv_paged.py", | |
| "type": "Attention Module", | |
| "category": "Low-Rank SVD & Factorization", | |
| "paradigm": "svd", | |
| "icon": "\u26a1", | |
| "description": "MoE-DeltaKV-Paged Hybrid Attention Module. Combines Mixture-of-Block-Attention (MoBA) with Paged Delta-KV cache compression. Fixes: - Straight-Through Estimator (STE) for differentiable 4-bit uniform and SVD quantization. - Non-causal bidirectional attention for encoder architectures (DistilBERT). - CLS token preservation in block 0. - Pretrained ", | |
| "math": "\\Delta \\mathbf{K}_t = \\mathbf{K}_t - \\mathbf{K}_{t-1}, \\quad \\forall t \\in \\{1, \\dots, N-1\\}", | |
| "markdown_preview": "# MoE-DeltaKV Paged: Quantized & Paged KV-Cache Mixture-of-Experts Attention\n\n## 1. Mathematical Formulation & Architectural Design\n\nThe **MoE-DeltaKV Paged Attention** architecture unifies **Mixture-of-Block-Attention (MoBA)** with **Paged Delta-KV Cache Compression**. In long-context inference and high-throughput execution, the memory footprint of uncompressed Key-Value (KV) activations constitutes the primary bottleneck. MoE-DeltaKV Paged tackles this through:\n1. **Temporal Differential Compression ($\\Delta\\text{-KV}$)**: Storing low-rank SVD components of temporal differences between conse" | |
| }, | |
| { | |
| "id": "nasc_attention", | |
| "name": "NASC: NEXUS-AEGIS Spectral Cascade Attention.", | |
| "file": "Codes/framework/attentions/nasc_attention.py", | |
| "type": "Attention Module", | |
| "category": "Low-Rank SVD & Factorization", | |
| "paradigm": "svd", | |
| "icon": "\u26a1", | |
| "description": "NASC: NEXUS-AEGIS Spectral Cascade Attention. Hardened for multi-architecture production deployment across DistilBERT, RoBERTa, GPT-2, and Qwen3.5. NOTE: This module is LoRA-only. The rank argument is accepted but not used for SVD compression. Use lora_rank to control adapter size.", | |
| "math": "Q_{\\text{loc}}, K_{\\text{loc}}, V_{\\text{loc}} = \\text{SVD-LoRA}(X)", | |
| "markdown_preview": "# NASC: NEXUS-AEGIS Spectral Cascade Attention\n\n## Overview\n**NASC (NEXUS-AEGIS Spectral Cascade)** is a complexity-adaptive hybrid attention architecture that unifies the strengths of localized causal sliding windows, block-sparse Mixture-of-Experts routing, and global Random Fourier Feature (RFF) linear kernel approximations into a single gated layer.\n\n---\n\n\n## Architecture & Data Flow\n\nThe `LoRALinear` module coordinates input projection factorization, intermediate representation routing, and output synthesis:\n\n```\nInput Activation X [B, N, D]\n \u2502\n \u251c\u2500\u2500 Low-Rank Bottleneck Down-Projection" | |
| }, | |
| { | |
| "id": "nexus", | |
| "name": "NEXUS: Neural Efficient X-scoring Unified Sparse Attention.", | |
| "file": "Codes/framework/attentions/nexus.py", | |
| "type": "Attention Module", | |
| "category": "Mixture of Experts & Routing", | |
| "paradigm": "moe", | |
| "icon": "\ud83d\udd00", | |
| "description": "NEXUS: Neural Efficient X-scoring Unified Sparse Attention. ============================================================= Hybrid sparse attention architecture combining multi-scale sparsity mechanisms: 1. XAttention antidiagonal block scoring (ICML 2025) - training-free block routing 2. SeerAttention learnable gating (NeurIPS 2025) - per-head bl", | |
| "math": "\\mathbf{Q}_{\\text{sample}} = \\mathbf{Q}[:, :, ::\\sigma, :], \\quad \\mathbf{K}_{\\text{sample}} = \\mathbf{K}[:, :, ::\\sigma, :]", | |
| "markdown_preview": "# NEXUS: Neural Efficient X-scoring Unified Sparse Attention\n\n## 1. Mathematical Formulation & Architectural Design\n\nThe **NEXUS (Neural Efficient X-scoring Unified Sparse Attention)** architecture synthesizes six breakthrough multi-scale sparsity and linear attention paradigms into a single unified execution pipeline:\n1. **XAttention Antidiagonal Block Scoring (ICML 2025)**: Training-free, strided coarse block correlation ranking candidate block pairs in $\\mathcal{O}((N/\\sigma)^2 D)$ time.\n2. **SeerAttention Learnable Gating (NeurIPS 2025)**: Low-rank pooled query-key gating networks predicti" | |
| }, | |
| { | |
| "id": "nha_attention", | |
| "name": "Native Hybrid Attention (NHA) \u2013 Production-Hardened.", | |
| "file": "Codes/framework/attentions/nha_attention.py", | |
| "type": "Attention Module", | |
| "category": "Latent & KV-Cache Compression", | |
| "paradigm": "mla", | |
| "icon": "\ud83e\udde0", | |
| "description": "Native Hybrid Attention (NHA) \u2013 Production-Hardened. Fuses localized sliding-window attention with global linear Random Fourier Features (RFF) via dynamic token-level gating and exact pretrained weight transfer. Supports: - Multi-architecture copy_weights (GPT-2, RoBERTa/BERT nested, Qwen/LLaMA, DistilBERT) - Trainable LoRA adapters with frozen b", | |
| "math": "\\mathbf{Y} = \\gamma(\\mathbf{x}) \\mathbf{Y}_{\\text{local}} + (1 - \\gamma(\\mathbf{x})) \\mathbf{Y}_{\\text{global}}", | |
| "markdown_preview": "# Native Hybrid Attention (NHA)\n\n## 1. Mathematical Formulation & Architecture\nCombines local sliding-window attention with global Random Fourier Features (RFF) linear attention:\n$$\\mathbf{Y} = \\gamma(\\mathbf{x}) \\mathbf{Y}_{\\text{local}} + (1 - \\gamma(\\mathbf{x})) \\mathbf{Y}_{\\text{global}}$$\n$$\\mathbf{Y}_{\\text{global}} = \\frac{\\phi(\\mathbf{Q}) (\\phi(\\mathbf{K})^T \\mathbf{V})}{\\phi(\\mathbf{Q}) \\phi(\\mathbf{K})^T \\mathbf{1} + \\epsilon}$$\nwhere $\\phi(\\mathbf{x}) = \\frac{1}{\\sqrt{F}} [\\cos(\\mathbf{x}\\mathbf{W}_{\\text{rff}}), \\sin(\\mathbf{x}\\mathbf{W}_{\\text{rff}})]$.\n\n\n## Architecture & Data Fl" | |
| }, | |
| { | |
| "id": "nha_compressor", | |
| "name": "Native Hybrid Attention (NHA) Compressor.", | |
| "file": "Codes/framework/attentions/nha_compressor.py", | |
| "type": "Attention Module", | |
| "category": "Dynamic & Efficient Attention", | |
| "paradigm": "sparse", | |
| "icon": "\ud83c\udf1f", | |
| "description": "Native Hybrid Attention (NHA) Compressor.", | |
| "math": "A_{ij} = \\text{TopK}\\left(\\text{Softmax}\\left(\\frac{Q_i K_j^\\top}{\\sqrt{d}}\\right)\\right)", | |
| "markdown_preview": "# Nha Compressor\n\n## Overview\n## Configuration & Hyperparameters\n\n| Parameter | Description |\n|-----------|-------------|\n\n## Usage Examples\n\n```python\nfrom Codes.framework.attentions import Module\n```\n\n## Benchmark Results\n\n| Status | Accuracy | Val Loss | Peak VRAM | Inference Latency |\n|--------|----------|----------|-----------|-------------------|\n| Pending| Pending | Pending | Pending | Pending |\n\n## Limitations & Future Work\n\nTODO: Identify specific limitations and edge cases for this module.\n" | |
| }, | |
| { | |
| "id": "nsa_attention", | |
| "name": "Native Sparse Attention: Hybrid Local Sliding-Window and Dynamic ", | |
| "file": "Codes/framework/attentions/nsa_attention.py", | |
| "type": "Attention Module", | |
| "category": "Mixture of Experts & Routing", | |
| "paradigm": "moe", | |
| "icon": "\ud83d\udd00", | |
| "description": "Native Sparse Attention (NSA) Module (Production-Hardened). Combines sliding-window local context capture with dynamic, input-dependent top-$k$ global routing across key-value representations in a fully vectorized manner. Supports: - Multi-architecture copy_weights (GPT-2, RoBERTa/BERT nested, Qwen/LLaMA, DistilBERT) - Trainable LoRA adapters wit", | |
| "math": "\\mathbf{Q} = \\mathbf{X} \\mathbf{W}_Q, \\quad \\mathbf{K} = \\mathbf{X} \\mathbf{W}_K, \\quad \\mathbf{V} = \\mathbf{X} \\mathbf{W}_V \\quad \\in \\mathbb{R}^{B \\times H \\times L \\times d_h}", | |
| "markdown_preview": "# Native Sparse Attention: Hybrid Local Sliding-Window and Dynamic Top-k Global Routing\n\n## 1. Executive Abstract & Architectural Motivation\nDense Multi-Head Self-Attention (MHSA) scales quadratically $\\mathcal{O}(L^2 d)$ with sequence length $L$, computing pairwise attention interactions across all token pairs regardless of semantic distance. Purely localized attention models enforce fixed spatial windows of size $W$, reducing computational complexity to $\\mathcal{O}(L \\cdot W \\cdot d)$, but completely sever long-range associative dependencies across distant sequence positions.\n\n**Native Spar" | |
| }, | |
| { | |
| "id": "nystrom_attention", | |
| "name": "Nystr\u00f6m Attention: Low-Rank Landmark Approximation via Matrix Pse", | |
| "file": "Codes/framework/attentions/nystrom_attention.py", | |
| "type": "Attention Module", | |
| "category": "Low-Rank SVD & Factorization", | |
| "paradigm": "svd", | |
| "icon": "\u26a1", | |
| "description": "Nystr\u00f6mformer Attention \u2013 Production-Hardened. Nystr\u00f6m-based low-rank kernel approximation with iterative pseudoinverse. Supports: - Multi-architecture copy_weights (GPT-2, RoBERTa/BERT nested, Qwen/LLaMA, DistilBERT) - Trainable LoRA adapters with frozen base weights - GQA with repeat_kv - RoPE position embeddings - Causal masking for decoders -", | |
| "math": "\\hat{A} = \\hat{F}_1 \\hat{F}_2^+ \\hat{F}_3", | |
| "markdown_preview": "# Nystr\u00f6m Attention: Low-Rank Landmark Approximation via Matrix Pseudoinverse Decomposition\n\n## 1. Executive Abstract & Architectural Motivation\n\nStandard Multi-Head Self-Attention requires materializing an $L \\times L$ pairwise affinity matrix, resulting in prohibitive $\\mathcal{O}(L^2)$ computational complexity and memory footprint. Nystr\u00f6m Attention (Xiong et al., 2021) circumvents this bottleneck by reformulating attention matrix evaluation as a **Nystr\u00f6m-based low-rank kernel approximation**.\n\nBy extracting $m \\ll L$ landmark tokens ($\\widetilde{Q}, \\widetilde{K}$) via adaptive average po" | |
| }, | |
| { | |
| "id": "optimized_hybrid_attention", | |
| "name": "Optimized Hybrid Attention (OHA): Fused Sliding-Window Local Self", | |
| "file": "Codes/framework/attentions/optimized_hybrid_attention.py", | |
| "type": "Attention Module", | |
| "category": "Low-Rank SVD & Factorization", | |
| "paradigm": "svd", | |
| "icon": "\u26a1", | |
| "description": "Optimized Hybrid Attention (OHA) \u2013 Production-Hardened. Fuses localized sliding-window self-attention with sequence-compressed low-rank global attention via dynamic head-wise gated modulation and exact pretrained weight transfer. Supports: - Multi-architecture copy_weights (GPT-2, RoBERTa/BERT nested, Qwen/LLaMA, DistilBERT) - Trainable LoRA adap", | |
| "math": "Q = X W_Q, \\quad K = X W_K, \\quad V = X W_V \\in \\mathbb{R}^{B \\times N \\times H \\cdot d_h}", | |
| "markdown_preview": "# Optimized Hybrid Attention (OHA): Fused Sliding-Window Local Self-Attention and Sequence-Compressed Low-Rank Global Attention\n\n## 1. Abstract & Theoretical Motivation\n\nStandard Multi-Head Self-Attention (MHSA) in Transformer models requires quadratic computational complexity $\\mathcal{O}(N^2)$ and memory allocation scaling with sequence length $N$. While local sliding-window attention restricts computation to $\\mathcal{O}(N \\cdot W_{\\text{local}})$, it fails to capture global semantic context across distant sequence tokens. Conversely, sequence pooling methods reduce global complexity but bl" | |
| }, | |
| { | |
| "id": "orion", | |
| "name": "ORION: Optimized Rectified Input-Aware Orthogonal Network with En", | |
| "file": "Codes/framework/attentions/orion.py", | |
| "type": "Attention Module", | |
| "category": "Low-Rank SVD & Factorization", | |
| "paradigm": "svd", | |
| "icon": "\u26a1", | |
| "description": "ORION: Optimized Rectified Input-aware Orthogonal Network (Production-Hardened). Hybrid multi-head attention architecture integrating zero-shot SVD low-rank factorized projections, head-wise bifurcated linear kernel vs. rectified Softpick full attention, and random Fourier features. Supports: - Multi-architecture copy_weights (GPT-2, RoBERTa/BERT", | |
| "math": "W = U \\Sigma V^T, \\quad \\Sigma = \\text{diag}(\\sigma_1, \\sigma_2, \\dots, \\sigma_{\\min(d_{\\text{out}}, d_{\\text{in}})})", | |
| "markdown_preview": "# ORION: Optimized Rectified Input-Aware Orthogonal Network with Energy-Preserving SVD Projections and Bifurcated Multi-Head Attention\n\n## 1. Executive Abstract & Architectural Motivation\nStandard Multi-Head Self-Attention (MHSA) in Transformer architectures requires quadratic computational time and memory scaling $\\mathcal{O}(N^2)$ with sequence length $N$. While linearized kernel attention models reduce sequence complexity to $\\mathcal{O}(N \\cdot F \\cdot d_h)$, they frequently degrade local representation quality due to over-smoothed attention distributions. Conversely, full softmax attentio" | |
| }, | |
| { | |
| "id": "performer_attention", | |
| "name": "Performer Attention \u2013 Fast Attention via Positive Orthogonal Rand", | |
| "file": "Codes/framework/attentions/performer_attention.py", | |
| "type": "Attention Module", | |
| "category": "Latent & KV-Cache Compression", | |
| "paradigm": "mla", | |
| "icon": "\ud83e\udde0", | |
| "description": "Performer Attention \u2013 Fast Attention via Positive Orthogonal Random Features (FAVOR+). Production-hardened with full multi-architecture support. Supports: - Multi-architecture copy_weights (GPT-2, RoBERTa/BERT nested, Qwen/LLaMA, DistilBERT) - Trainable LoRA adapters with frozen base weights - GQA with repeat_kv - RoPE position embeddings with so", | |
| "math": "\\text{Attn}(Q, K, V) = D^{-1} (\\phi(Q) (\\phi(K)^\\top V)), \\quad \\phi(x) = \\frac{h(x)}{\\sqrt{m}} \\exp\\left(\\omega^\\top x - \\frac{\\|x\\|^2}{2}\\right)", | |
| "markdown_preview": "# Performer Attention (FAVOR+): Fast Attention via Positive Orthogonal Random Features\n\n## 1. Executive Abstract & Architectural Motivation\n\nStandard Multi-Head Self-Attention (Vaswani et al., 2017) requires materializing an $L \\times L$ attention matrix, leading to quadratic time and space complexity $\\mathcal{O}(L^2)$ with respect to sequence length $L$. The Performer architecture (Choromanski et al., 2020) resolves this fundamental scalability barrier using the **Fast Attention Via Positive Orthogonal Random Features (FAVOR+)** framework.\n\nFAVOR+ constructs an unbiased, strictly positive ke" | |
| }, | |
| { | |
| "id": "phantom", | |
| "name": "PHANTOM: Parameter-Hashed AttentioN with Token-wise Orthogonal Ma", | |
| "file": "Codes/framework/attentions/phantom.py", | |
| "type": "Attention Module", | |
| "category": "Low-Rank SVD & Factorization", | |
| "paradigm": "svd", | |
| "icon": "\u26a1", | |
| "description": "Phantom: Parameter-Hashed AttentioN with Token-wise Orthogonal Mapping \u2013 Production-Hardened. Combines CUR-matrix factorized linear projections based on empirical statistical leverage scores with multi-round angular Locality-Sensitive Hashing (LSH) sparse attention and rectified Softpick activation. Supports: - Multi-architecture copy_weights (GP", | |
| "math": "\\ell_{\\text{row}}(i) = \\frac{\\|U_{i, :}\\|_2^2}{\\sum_{i'=1}^{d_{\\text{out}}} \\|U_{i', :}\\|_2^2}, \\quad \\ell_{\\text{col}}(j) = \\frac{\\|V_{:, j}\\|_2^2}{\\sum_{j'=1}^{d_{\\text{in}}} \\|V_{:, j'}\\|_2^2}", | |
| "markdown_preview": "# PHANTOM: Parameter-Hashed AttentioN with Token-wise Orthogonal Mapping via Deterministic CUR Decomposition and Angular LSH\n\n## 1. Executive Abstract & Architectural Motivation\nStandard Multi-Head Self-Attention (MHSA) in Transformer models requires quadratic computational time and memory scaling $\\mathcal{O}(N^2)$ with sequence length $N$, presenting significant bottlenecks for long-context deployment. While dense low-rank projections reduce parameter counts, standard SVD linear layers discard structural column/row interpretable sparsity. Furthermore, sparse attention heuristics often fail t" | |
| }, | |
| { | |
| "id": "primal_attention", | |
| "name": "Primal Attention: Self-Attention through Asymmetric Kernel SVD in", | |
| "file": "Codes/framework/attentions/primal_attention.py", | |
| "type": "Attention Module", | |
| "category": "Low-Rank SVD & Factorization", | |
| "paradigm": "svd", | |
| "icon": "\u26a1", | |
| "description": "Primal Attention: Self-Attention through Asymmetric Kernel SVD in Primal Representation \u2013 Production-Hardened. Nystr\u00f6m-inspired low-rank landmark approximation of self-attention using segment pooling landmarks and iterative Schulz pseudoinverse refinement. Supports: - Multi-architecture copy_weights (GPT-2, RoBERTa/BERT nested, Qwen/LLaMA, Distil", | |
| "math": "F_1 = \\frac{Q \\widetilde{K}^T}{\\sqrt{d}} \\in \\mathbb{R}^{B \\times H \\times N \\times r}", | |
| "markdown_preview": "# Primal Attention: Asymmetric Kernel SVD via Segment-Pooled Landmark Decomposition\n\n## 1. Executive Abstract & Architectural Motivation\n\nStandard Multi-Head Attention requires materializing the full pairwise attention score matrix $A \\in \\mathbb{R}^{L \\times L}$, incurring an $\\mathcal{O}(L^2)$ memory wall. Primal Attention formulates self-attention through asymmetric kernel SVD approximation in the primal representation space.\n\nBy extracting $r \\ll L$ landmark queries $\\widetilde{Q}$ and keys $\\widetilde{K}$ via uniform segment average pooling across the sequence length, Primal Attention fac" | |
| }, | |
| { | |
| "id": "progressive_token_pruning", | |
| "name": "Progressive Token Pruning (PTP) \u2013 Production-Hardened.", | |
| "file": "Codes/framework/attentions/progressive_token_pruning.py", | |
| "type": "Attention Module", | |
| "category": "Latent & KV-Cache Compression", | |
| "paradigm": "mla", | |
| "icon": "\ud83e\udde0", | |
| "description": "Progressive Token Pruning (PTP) \u2013 Production-Hardened. Learnable Progressive Token Pruning module with guaranteed [CLS] retention, multi-head attention capabilities, and full multi-architecture support. Supports: - Multi-architecture copy_weights (GPT-2, RoBERTa/BERT nested, Qwen/LLaMA, DistilBERT) - Trainable LoRA adapters with frozen base weigh", | |
| "math": "\\min_{\\text{rank}(\\widehat{W}) \\le r} \\|W - \\widehat{W}\\|_F^2 = \\sum_{i=r+1}^{D} \\sigma_i^2", | |
| "markdown_preview": "# Progressive Token Pruning (PTP)\n\n## 1. Mathematical Formulation & Architecture\nDynamically prunes redundant intermediate sequence tokens via learned saliency scores with guaranteed $[\\text{CLS}]$ retention.\n\n\n## Architecture & Data Flow\n\nThe `LoRALinear` module coordinates input projection factorization, intermediate representation routing, and output synthesis:\n\n```\nInput Activation X [B, N, D]\n \u2502\n \u251c\u2500\u2500 Low-Rank Bottleneck Down-Projection: Z = X W_down [B, N, r]\n \u251c\u2500\u2500 Head Expansion & Up-Projection: Q, K, V [B, H, N, d_k]\n \u251c\u2500\u2500 Attention Scoring / Kernelized Associative Scan: S = Score" | |
| }, | |
| { | |
| "id": "projection_factorized_attn", | |
| "name": "Projection Factorized SVD Linear Attention.", | |
| "file": "Codes/framework/attentions/projection_factorized_attn.py", | |
| "type": "Attention Module", | |
| "category": "Low-Rank SVD & Factorization", | |
| "paradigm": "svd", | |
| "icon": "\u26a1", | |
| "description": "Projection Factorized SVD Linear Attention.", | |
| "math": "W \\approx U_r \\Sigma_r V_r^\\top = A B^\\top, \\quad r \\ll \\min(d_{in}, d_{out})", | |
| "markdown_preview": "# Projection Factorized Attn\n\n## Overview\nProjection Factorized SVD Linear Attention.\n\n## Configuration & Hyperparameters\n\n| Parameter | Description |\n|-----------|-------------|\n\n## Usage Examples\n\n```python\nfrom Codes.framework.attentions import ProjectionFactorizedAttention\n```\n\n## Benchmark Results\n\n| Status | Accuracy | Val Loss | Peak VRAM | Inference Latency |\n|--------|----------|----------|-----------|-------------------|\n| Pending| Pending | Pending | Pending | Pending |\n\n## Limitations & Future Work\n\nTODO: Identify specific limitations and edge cases for this module.\n" | |
| }, | |
| { | |
| "id": "qcqa_attention", | |
| "name": "Quality and Capacity-Aware Grouped Query Attention (QCQA): Dynami", | |
| "file": "Codes/framework/attentions/qcqa_attention.py", | |
| "type": "Attention Module", | |
| "category": "Mixture of Experts & Routing", | |
| "paradigm": "moe", | |
| "icon": "\ud83d\udd00", | |
| "description": "Quality and Capacity-Aware Grouped Query Attention (QCQA) \u2013 Production-Hardened. Dynamically synthesizes head-specific key-value representations for each query head via continuous learned routing distributions over a compressed set of KV heads. Supports: - Multi-architecture copy_weights (GPT-2, RoBERTa/BERT nested, Qwen/LLaMA, DistilBERT) - Trai", | |
| "math": "\\pi_{h, j} = \\frac{\\exp\\left(\\frac{q_h W_{\\text{router}, j}}{\\tau}\\right)}{\\sum_{j'=1}^{H_{kv}} \\exp\\left(\\frac{q_h W_{\\text{router}, j'}}{\\tau}\\right)}, \\quad \\forall j \\in \\{1, \\dots, H_{kv}\\}", | |
| "markdown_preview": "# Quality and Capacity-Aware Grouped Query Attention (QCQA): Dynamic Soft-Routing Mixture over Compressed KV Subspaces\n\n## 1. Executive Abstract & Architectural Motivation\n\nQuality and Capacity-Aware Grouped Query Attention (QCQA) resolves a fundamental limitation of static Grouped-Query Attention (GQA): rigid, deterministic assignment of query heads to key-value heads. In GQA, a fixed group of query heads shares a static key-value head pair, regardless of input token content or contextual semantics. This rigid partition can restrict expressive capacity when queries require information outside" | |
| }, | |
| { | |
| "id": "rala_attention", | |
| "name": "Rank-Augmented Linear Attention (RALA): Low-Rank Parametric Head ", | |
| "file": "Codes/framework/attentions/rala_attention.py", | |
| "type": "Attention Module", | |
| "category": "Low-Rank SVD & Factorization", | |
| "paradigm": "svd", | |
| "icon": "\u26a1", | |
| "description": "Rank\u2011Augmented Linear Attention (RALA) \u2013 Production-Hardened. Augments linear attention kernel maps with low-rank static head interaction matrices. Supports: - Multi-architecture copy_weights (GPT-2, RoBERTa/BERT nested, Qwen/LLaMA, DistilBERT) - Trainable LoRA adapters with frozen base weights - GQA with repeat_kv - RoPE position embeddings - Ca", | |
| "math": "\\text{Attention}_{\\text{RALA}}(Q, K, V) = \\frac{\\phi(Q) (\\phi(K)^\\top V)}{\\phi(Q) (\\phi(K)^\\top \\mathbf{1})} + \\alpha \\frac{\\phi(Q) (U V^\\top)}{\\phi(Q) (\\phi(K)^\\top \\mathbf{1})}", | |
| "markdown_preview": "# Rank-Augmented Linear Attention (RALA): Low-Rank Parametric Head Interaction Augmentation\n\n## 1. Executive Abstract & Architectural Motivation\n\nRank-Augmented Linear Attention (RALA, ICLR 2024) addresses the expressivity bottlenecks of standard kernel-based linear attention mechanisms. While standard linear attention reduces the quadratic sequence complexity ($\\mathcal{O}(N^2)$) of Multi-Head Attention to linear complexity ($\\mathcal{O}(N)$) via positive kernel feature maps ($\\phi(x) = \\text{ELU}(x) + 1.0$), low-rank associative memory approximations suffer from capacity saturation when mode" | |
| }, | |
| { | |
| "id": "rala_compressor", | |
| "name": "RALA Compressor: Rank-Augmented Linear Attention Layer Architectu", | |
| "file": "Codes/framework/attentions/rala_compressor.py", | |
| "type": "Attention Module", | |
| "category": "Low-Rank SVD & Factorization", | |
| "paradigm": "svd", | |
| "icon": "\u26a1", | |
| "description": "Rank-Augmented Linear Attention Compressor (RALA Compressor) \u2013 Production-Hardened. Maps to and aliases framework.attentions.rala_attention.RALALinearAttention.", | |
| "math": "S_{\\text{KV}} = \\sum_{j=1}^N \\phi(K_j) V_j^T \\in \\mathbb{R}^{B \\times H \\times d \\times d}, \\quad z = \\sum_{j=1}^N \\phi(K_j) \\in \\mathbb{R}^{B \\times H \\times d}", | |
| "markdown_preview": "# RALA Compressor: Rank-Augmented Linear Attention Layer Architecture & Masking Specifications\n\n## 1. Executive Abstract & Architectural Motivation\n\nRank-Augmented Linear Attention Compressor (`RALALinearAttention` in `rala_compressor.py`) provides an expressive linear-complexity attention mechanism ($\\mathcal{O}(N)$). Standard linear attention models approximate Softmax attention using kernel feature maps ($\\phi(x) = \\text{ELU}(x) + 1.0$), reducing quadratic sequence scaling ($\\mathcal{O}(N^2)$) to linear scaling. However, fixed feature maps introduce capacity bottlenecks that degrade long-ra" | |
| }, | |
| { | |
| "id": "reap_moe", | |
| "name": "REAP-MoE: Robust Expert Allocation and Pruning Mixture of Experts", | |
| "file": "Codes/framework/attentions/reap_moe.py", | |
| "type": "Attention Module", | |
| "category": "Low-Rank SVD & Factorization", | |
| "paradigm": "svd", | |
| "icon": "\u26a1", | |
| "description": "REAP-MoE: Robust Expert Allocation and Pruning Mixture of Experts Attention Module \u2013 Production-Hardened. Implements a vectorized Mixture-of-Experts block with load-balancing regularization, SVD low-rank expert factorization, and multi-head attention weight transfer. Supports: - Multi-architecture copy_weights (GPT-2, RoBERTa/BERT nested, Qwen/LL", | |
| "math": "\\mathbf{W}_e = \\mathbf{U}_e \\mathbf{\\Sigma}_e \\mathbf{V}_e^\\top, \\quad \\operatorname{rank}(\\mathbf{W}_e) \\le r \\ll \\min(d_{\\text{in}}, d_{\\text{out}})", | |
| "markdown_preview": "# REAP-MoE: Robust Expert Allocation and Pruning Mixture-of-Experts\n\n## 1. Mathematical Formulation & Architectural Design\n\nThe **REAP-MoE (Robust Expert Allocation and Pruning Mixture-of-Experts)** architecture integrates low-rank Singular Value Decomposition (SVD) parameterization directly into conditional Mixture-of-Experts sub-networks. Standard MoE architectures drastically expand parameter footprint when increasing the number of experts $E$; REAP-MoE circumvents this parameter explosion by decomposing every expert layer into singular spectral components:\n$$\\mathbf{W}_e = \\mathbf{U}_e \\ma" | |
| }, | |
| { | |
| "id": "regla_attention", | |
| "name": "Refined Gated Linear Attention (ReGLA): Element-Wise Key Gating &", | |
| "file": "Codes/framework/attentions/regla_attention.py", | |
| "type": "Attention Module", | |
| "category": "Latent & KV-Cache Compression", | |
| "paradigm": "mla", | |
| "icon": "\ud83e\udde0", | |
| "description": "Refined Gated Linear Attention (ReGLA) \u2013 Production-Hardened. Combines element-wise key gating, normalized linear attention kernel evaluation, and short-range depthwise 1D convolutions without O(L) memory explosion. Supports: - Multi-architecture copy_weights (GPT-2, RoBERTa/BERT nested, Qwen/LLaMA, DistilBERT) - Trainable LoRA adapters with froze", | |
| "math": "Q = X W_Q, \\quad K = X W_K, \\quad V = X W_V, \\quad G = X W_G", | |
| "markdown_preview": "# Refined Gated Linear Attention (ReGLA): Element-Wise Key Gating & Depthwise Convolutional Fusion\n\n## 1. Abstract & Theoretical Motivation\n\nStandard linear attention modules approximate softmax sequence representations using un-gated kernel projections, suffering from representation collapse and lack of spatial inductive bias. **Refined Gated Linear Attention (ReGLA)** resolves these limitations by augmenting normalized linear attention with two architectural enhancements:\n1. **Element-Wise Key Gating with Strictly Non-Negative Feature Maps**: Employs continuous Sigmoid gating $\\sigma(G)$ on " | |
| }, | |
| { | |
| "id": "replace", | |
| "name": "Model Attention Replacement and Dynamic Graph Surgery Utilities.", | |
| "file": "Codes/framework/attentions/replace.py", | |
| "type": "Attention Module", | |
| "category": "Dynamic & Efficient Attention", | |
| "paradigm": "sparse", | |
| "icon": "\ud83c\udf1f", | |
| "description": "Model Attention Replacement and Dynamic Graph Surgery Utilities. Provides unified dispatch interfaces to substitute attention sub-layers in pretrained HuggingFace transformer architectures (BERT, RoBERTa, DistilBERT, GPT-2) with custom efficient attention mechanisms. Mathematical Mechanism: Given a transformer computation graph $\\mathcal{G} =", | |
| "math": "\\min_{\\Theta_{\\text{new}}} \\|f_{\\text{new}}(X; \\Theta_{\\text{new}}) - f_{\\text{old}}(X; \\Theta_{\\text{old}})\\|_F", | |
| "markdown_preview": "# Replace: Dynamic Attention Layer Replacement Engine\n\n## 1. Executive Abstract & Architectural Motivation\nThe **Replace** engine provides unified introspection, surgery, and hot-swapping mechanics for neural network attention sub-layers. It traverses arbitrary HuggingFace Transformer backends (BERT, RoBERTa, GPT-2, LLaMA, Mistral, T5) and replaces full quadratic multi-head attention blocks with linear, low-rank, or state-space mechanisms while preserving weights, biases, and normalization layers.\n\n---\n\n## 2. Mathematical Formulations & Derivations\nDynamic module surgery replaces the projectio" | |
| }, | |
| { | |
| "id": "res_svd", | |
| "name": "Spectral Residual Low-Rank SVD Attention (Res-SVD Attention) \u2013 Pr", | |
| "file": "Codes/framework/attentions/res_svd.py", | |
| "type": "Attention Module", | |
| "category": "Low-Rank SVD & Factorization", | |
| "paradigm": "svd", | |
| "icon": "\u26a1", | |
| "description": "Spectral Residual Low-Rank SVD Attention (Res-SVD Attention) \u2013 Production-Hardened. Combines base truncated SVD low-rank factorizations with trainable residual adapters: W = U_base V_base^T + alpha * B A where B A is initialized directly from the top-k residual singular directions of W - W_base. Supports: - Multi-architecture copy_weights (GPT-2,", | |
| "math": "W \\approx U_r \\Sigma_r V_r^\\top = A B^\\top, \\quad r \\ll \\min(d_{in}, d_{out})", | |
| "markdown_preview": "# Spectral Residual SVD Attention (Res-SVD Attention)\n\n## 1. Executive Abstract & Architectural Motivation\n\nSpectral Residual SVD Attention (`ResSVDAttention`) addresses a fundamental dilemma in transformer parameter reduction: low-rank matrix factorization achieves substantial compression ratios but suffers from irrecoverable loss of fine-grained representational capacity in downstream task adaptation. Conversely, Parameter-Efficient Fine-Tuning (PEFT) methods such as Low-Rank Adaptation (LoRA) maintain full-rank base weights, offering zero memory compression during inference.\n\n`ResSVDAttenti" | |
| }, | |
| { | |
| "id": "routing_attention", | |
| "name": "Routing Attention: Content-Adaptive Query-Centric Clustering via ", | |
| "file": "Codes/framework/attentions/routing_attention.py", | |
| "type": "Attention Module", | |
| "category": "Mixture of Experts & Routing", | |
| "paradigm": "moe", | |
| "icon": "\ud83d\udd00", | |
| "description": "Routing Attention: Content-Based Clustering for Sparse Attention \u2013 Production-Hardened. Partitions sequence tokens into discrete clusters, confining attention computation to intra-cluster tokens without materializing full [B, H, L, L] tensors. Supports: - Multi-architecture copy_weights (GPT-2, RoBERTa/BERT nested, Qwen/LLaMA, DistilBERT) - Traina", | |
| "math": "\\mathbf{Q} = \\mathbf{X} \\mathbf{W}_Q, \\quad \\mathbf{K} = \\mathbf{X} \\mathbf{W}_K, \\quad \\mathbf{V} = \\mathbf{X} \\mathbf{W}_V \\quad \\in \\mathbb{R}^{B \\times H \\times L \\times d_h}", | |
| "markdown_preview": "# Routing Attention: Content-Adaptive Query-Centric Clustering via Head-Wise Online K-Means\n\n## 1. Executive Abstract & Architectural Motivation\nStandard Multi-Head Self-Attention (MHSA) computes dense $L \\times L$ token interaction matrices, imposing quadratic time and memory complexity $\\mathcal{O}(L^2 d)$. Static sparse attention patterns\u2014such as fixed sliding windows or factorized strided strides\u2014impose rigid structural priors that fail to reflect dynamic, content-dependent semantic dependencies in natural language.\n\n**Routing Attention (`RoutingAttention`)** introduces a dynamic, content-" | |
| }, | |
| { | |
| "id": "rpc", | |
| "name": "Random Projection Compressed (RPC) Attention.", | |
| "file": "Codes/framework/attentions/rpc.py", | |
| "type": "Attention Module", | |
| "category": "Dynamic & Efficient Attention", | |
| "paradigm": "sparse", | |
| "icon": "\ud83c\udf1f", | |
| "description": "Random Projection Compressed (RPC) Attention. Re-exports from framework.attentions.rpc_attention.", | |
| "math": "A_{ij} = \\text{TopK}\\left(\\text{Softmax}\\left(\\frac{Q_i K_j^\\top}{\\sqrt{d}}\\right)\\right)", | |
| "markdown_preview": "# Rpc\n\n## Overview\n## Configuration & Hyperparameters\n\n| Parameter | Description |\n|-----------|-------------|\n\n## Usage Examples\n\n```python\nfrom Codes.framework.attentions import Module\n```\n\n## Benchmark Results\n\n| Status | Accuracy | Val Loss | Peak VRAM | Inference Latency |\n|--------|----------|----------|-----------|-------------------|\n| Pending| Pending | Pending | Pending | Pending |\n\n## Limitations & Future Work\n\nTODO: Identify specific limitations and edge cases for this module.\n" | |
| }, | |
| { | |
| "id": "rpc_attention", | |
| "name": "RPC Attention: Randomized Phase Cluster Attention via Fixed Rando", | |
| "file": "Codes/framework/attentions/rpc_attention.py", | |
| "type": "Attention Module", | |
| "category": "Latent & KV-Cache Compression", | |
| "paradigm": "mla", | |
| "icon": "\ud83e\udde0", | |
| "description": "Random Projection Compressed (RPC) Attention \u2013 Production-Hardened. Employs randomized linear subspace sketching via fixed head-wise Gaussian random projection buffers to compress query-key similarity evaluations into lower-dimensional manifolds. Supports: - Multi-architecture copy_weights (GPT-2, RoBERTa/BERT nested, Qwen/LLaMA, DistilBERT) - Tr", | |
| "math": "\\widetilde{Q} = Q P \\in \\mathbb{R}^{B \\times H \\times L \\times d_p}, \\quad \\widetilde{K} = K P \\in \\mathbb{R}^{B \\times H \\times L \\times d_p}", | |
| "markdown_preview": "# RPC Attention: Randomized Phase Cluster Attention via Fixed Random Subspace Sketching\n\n## 1. Executive Abstract & Architectural Motivation\n\nStandard Multi-Head Attention evaluates full $d$-dimensional inner products between queries and keys, incurring high memory bandwidth requirements and $\\mathcal{O}(L^2 d)$ arithmetic overhead. Random Projection Compressed (RPC) Attention compresses query-key inner product evaluations by projecting head states onto a lower-dimensional $d_p$-dimensional random subspace ($d_p \\ll d$).\n\nRPC Attention leverages the Johnson-Lindenstrauss (JL) Lemma, which guar" | |
| }, | |
| { | |
| "id": "salt", | |
| "name": "SALT Attention: Singular Value Adaptation with Low-Rank Tuning.", | |
| "file": "Codes/framework/attentions/salt.py", | |
| "type": "Attention Module", | |
| "category": "Low-Rank SVD & Factorization", | |
| "paradigm": "svd", | |
| "icon": "\u26a1", | |
| "description": "SALT Attention: Singular Value Adaptation with Low-Rank Tuning. Re-exports from framework.attentions.salt_attention.", | |
| "math": "W \\approx U_r \\Sigma_r V_r^\\top = A B^\\top, \\quad r \\ll \\min(d_{in}, d_{out})", | |
| "markdown_preview": "# Salt\n\n## Overview\n## Configuration & Hyperparameters\n\n| Parameter | Description |\n|-----------|-------------|\n\n## Usage Examples\n\n```python\nfrom Codes.framework.attentions import Module\n```\n\n## Benchmark Results\n\n| Status | Accuracy | Val Loss | Peak VRAM | Inference Latency |\n|--------|----------|----------|-----------|-------------------|\n| Pending| Pending | Pending | Pending | Pending |\n\n## Limitations & Future Work\n\nTODO: Identify specific limitations and edge cases for this module.\n" | |
| }, | |
| { | |
| "id": "salt_attention", | |
| "name": "SALT Attention: Singular Value Adaptation with Low-Rank Tuning fo", | |
| "file": "Codes/framework/attentions/salt_attention.py", | |
| "type": "Attention Module", | |
| "category": "Low-Rank SVD & Factorization", | |
| "paradigm": "svd", | |
| "icon": "\u26a1", | |
| "description": "SALT Attention: Singular Value Adaptation with Low-Rank Tuning. Parameter-efficient adaptation mechanism for transformer attention projections combining orthogonal singular vector preservation, singular value rescaling ($\\gamma$), and additive low-rank adapters ($A B$). Hardened for multi-architecture production deployment.", | |
| "math": "W = U_{:, :r} \\operatorname{diag}(\\Sigma_{1:r} \\odot \\gamma) V_{1:r, :}^\\top + A B", | |
| "markdown_preview": "# SALT Attention: Singular Value Adaptation with Low-Rank Tuning for Parameter-Efficient Transformer Fine-Tuning\n\n## 1. Abstract & Theoretical Motivation\n\nFine-tuning large pre-trained Transformer architectures via full parameter update is computationally expensive and memory-prohibitive. While standard Low-Rank Adaptation (LoRA) introduces trainable bottleneck matrices $A B$, it ignores the underlying singular value spectrum of the pre-trained weight matrices, often perturbing dominant singular directions during gradient updates.\n\n**SALT Attention** (Singular Value Adaptation with Low-Rank Tu" | |
| }, | |
| { | |
| "id": "samba_m3", | |
| "name": "SAMBA-M3: Hybrid Sliding-Window Attention and Spectral Mamba Bloc", | |
| "file": "Codes/framework/attentions/samba_m3.py", | |
| "type": "Attention Module", | |
| "category": "Linear Attention & State-Space (SSM)", | |
| "paradigm": "ssm", | |
| "icon": "\ud83d\udc0d", | |
| "description": "SAMBA-M3: Hybrid Sliding-Window Attention and Spectral Mamba Block (Hardened). Supports: - DistilBERT (Flat MHA) - RoBERTa / BERT (Nested Encoder Self-Attention) - GPT-2 (Causal Decoder with Conv1D) - Qwen / LLaMA (GQA with Rotary Position Embeddings) - True chunked local sliding-window attention (O(L \u00b7 W)) - No double-residual or double LayerNorm ", | |
| "math": "\\mathbf{M}^{\\text{window}}_{i, j} = \\begin{cases} 0 & \\text{if } 0 \\le i - j < W \\\\ -\\infty & \\text{otherwise} \\end{cases}", | |
| "markdown_preview": "# SAMBA-M3: Hybrid Sliding-Window Attention and Spectral Mamba State-Space Model\n\n## 1. Executive Abstract & Architectural Motivation\nTransformer architectures parameterized purely by dense Multi-Head Self-Attention (MHSA) suffer from an $\\mathcal{O}(L^2)$ computational memory wall when scaling to long sequence lengths. While local sliding-window attention restricts interactions to a finite context window $W$ to achieve linear complexity $\\mathcal{O}(L \\cdot W)$, it loses the capacity to maintain global long-range context across distant sequence tokens. Conversely, continuous-time selective St" | |
| }, | |
| { | |
| "id": "samba_moe", | |
| "name": "Samba-MoE: Selective State Space (SAMBA-M3) + REAP-MoE Hybrid Arc", | |
| "file": "Codes/framework/attentions/samba_moe.py", | |
| "type": "Attention Module", | |
| "category": "Mixture of Experts & Routing", | |
| "paradigm": "moe", | |
| "icon": "\ud83d\udd00", | |
| "description": "Samba-MoE: Hybrid Selective State Space (SAMBA-M3) + REAP-MoE Block. Combines local sliding-window attention, spectral Mamba SSM, and MoE routing. Hardened for production use across diverse transformer architectures: - DistilBERT (Flat MHA) - RoBERTa / BERT (Nested Encoder Self-Attention) - GPT-2 (Causal Decoder with Conv1D) - Qwen / LLaMA (GQA wit", | |
| "math": "\\mathbf{h}_t' = \\mathbf{A} \\mathbf{h}_{t-1}' + \\mathbf{B} \\mathbf{x}_t, \\quad \\mathbf{y}_{\\text{ssm}, t} = \\mathbf{C} \\mathbf{h}_t' + \\mathbf{D} \\mathbf{x}_t", | |
| "markdown_preview": "# Samba-MoE: Selective State Space (SAMBA-M3) + REAP-MoE Hybrid Architecture\n\n## 1. Mathematical Formulation & Architectural Design\n\nThe **Samba-MoE** architecture constructs a tri-hybrid processing hierarchy designed to overcome the $\\mathcal{O}(N^2)$ quadratic bottleneck of full self-attention while eliminating the capacity bottlenecks of pure State Space Models (SSMs). It sequentially couples:\n1. **Bidirectional SimpleSSM State-Space Recurrence**: Global linear-time memory aggregation across sequence tokens.\n2. **Local Sliding-Window Self-Attention (LocalSWA)**: High-resolution local syntac" | |
| }, | |
| { | |
| "id": "simple_ssm", | |
| "name": "Simple State Space Model (SimpleSSM).", | |
| "file": "Codes/framework/attentions/simple_ssm.py", | |
| "type": "Attention Module", | |
| "category": "Linear Attention & State-Space (SSM)", | |
| "paradigm": "ssm", | |
| "icon": "\ud83d\udc0d", | |
| "description": "Simple State Space Model (SimpleSSM). Lightweight bidirectional/causal selective state space recurrence block featuring symmetric depthwise 1D convolution, input-dependent discretization timescales (Delta t), diagonal state transitions (A), multiplicative gating, and pretrained weight transfer. Hardened for production use across diverse transforme", | |
| "math": "[X_{\\text{ssm}}, Z] = \\operatorname{chunk}(X W_{\\text{in}}, 2) \\in \\mathbb{R}^{B \\times L \\times D}", | |
| "markdown_preview": "# Simple State Space Model (SimpleSSM)\n\n## 1. Executive Abstract & Theoretical Foundations\n\n**SimpleSSM** is a lightweight, strictly linear $\\mathcal{O}(L \\cdot D \\cdot d_{\\text{state}})$ bidirectional State Space Model module:\n1. **Symmetric 1D Depthwise Convolution**: Applies non-causal symmetric 1D depthwise convolution across token channels to preserve local continuous feature dependencies.\n2. **Selective Input-Dependent Discretization**: Calculates input-dependent discretization timescales $\\Delta t = \\operatorname{softplus}(X_{\\text{conv}} W_{\\Delta t} + b_{\\Delta t}) > 0$.\n3. **Strict L" | |
| }, | |
| { | |
| "id": "smart_weighted_ensemble", | |
| "name": "Smart Weighted Ensemble: Task-Aware Attention & Dynamic Confidenc", | |
| "file": "Codes/framework/attentions/smart_weighted_ensemble.py", | |
| "type": "Attention Module", | |
| "category": "Dynamic & Efficient Attention", | |
| "paradigm": "sparse", | |
| "icon": "\ud83c\udf1f", | |
| "description": "Smart Weighted Ensemble (Alias for weighted_ensemble.py). Re-exports SmartWeightedEnsemble, TaskAwareAttention, WeightedEnsembleAttention, and injection functions.", | |
| "math": "A_{ij} = \\text{TopK}\\left(\\text{Softmax}\\left(\\frac{Q_i K_j^\\top}{\\sqrt{d}}\\right)\\right)", | |
| "markdown_preview": "# Smart Weighted Ensemble: Task-Aware Attention & Dynamic Confidence Fusion\n\n## 1. Abstract & Theoretical Motivation\n\nCombining multi-model predictions across heterogeneous downstream tasks typically relies on static averaging or scalar voting weights. However, static ensemble strategies fail to account for task-domain alignment or sample-specific predictive uncertainty.\n\n**Smart Weighted Ensemble (`SmartWeightedEnsemble`)** provides a multi-model fusion framework powered by Task-Aware Attention (`TaskAwareAttention`) and dynamic instance-level confidence weighting. By embedding task IDs into " | |
| }, | |
| { | |
| "id": "soa_attn", | |
| "name": "Shared Orthogonal Attention (SOA) with Per-Projection Latent Bott", | |
| "file": "Codes/framework/attentions/soa_attn.py", | |
| "type": "Attention Module", | |
| "category": "Latent & KV-Cache Compression", | |
| "paradigm": "mla", | |
| "icon": "\ud83e\udde0", | |
| "description": "Shared Orthogonal Attention (SOA) with Per-Projection Latent Bottlenecks. Hardened for production use across diverse transformer architectures: - DistilBERT (Flat MHA) - RoBERTa / BERT (Nested Encoder Self-Attention) - GPT-2 (Causal Decoder with Conv1D) - Qwen / LLaMA (GQA with Rotary Position Embeddings)", | |
| "math": "W \\approx U_{:, :r} \\Sigma_{:r, :r} V_{:, :r}^T, \\quad r=64", | |
| "markdown_preview": "# SOA Attention: Shared Orthogonal Attention with Latent Bottlenecks\n\n## 1. Mathematical Formulation & Architecture\nFactorizes Q, K, and V projections into low-rank representations initialized via truncated Singular Value Decomposition (SVD):\n$$W \\approx U_{:, :r} \\Sigma_{:r, :r} V_{:, :r}^T, \\quad r=64$$\nMulti-head attention computation:\n$$A = \\operatorname{Softmax}\\left(\\frac{Q K^T}{\\sqrt{d_h}} + M_{\\text{pad}}\\right), \\quad O = (A V) W_O$$\n\n\n## Architecture & Data Flow\n\nThe `LoRALinear` module coordinates input projection factorization, intermediate representation routing, and output synthe" | |
| }, | |
| { | |
| "id": "soa_fused_module", | |
| "name": "SOA-Fused Attention: Low-Rank Latent Space Multi-Head Attention w", | |
| "file": "Codes/framework/attentions/soa_fused_module.py", | |
| "type": "Attention Module", | |
| "category": "Dynamic & Efficient Attention", | |
| "paradigm": "sparse", | |
| "icon": "\ud83c\udf1f", | |
| "description": "Shim for soa_fused_module re-exporting from SOAFusedAttention.", | |
| "math": "A_{ij} = \\text{TopK}\\left(\\text{Softmax}\\left(\\frac{Q_i K_j^\\top}{\\sqrt{d}}\\right)\\right)", | |
| "markdown_preview": "# SOA-Fused Attention: Low-Rank Latent Space Multi-Head Attention with Zero-Shot SVD Initialization\n\n## 1. Executive Abstract & Architectural Motivation\n\nDense Multi-Head Self-Attention layers allocate distinct projection matrices for Query, Key, and Value ($W_Q, W_K, W_V \\in \\mathbb{R}^{D \\times D}$). In standard architectures, these matrices contain significant spectral redundancy, with $>80\\%$ of their Frobenius energy concentrated in low-rank singular subspaces.\n\n**SOA-Fused Attention (`SOAFusedAttention`)** factorizes each dense projection matrix into a pair of low-rank bottleneck operato" | |
| }, | |
| { | |
| "id": "soa_module", | |
| "name": "SOA Attention: Shared Orthogonal Attention with Latent Bottleneck", | |
| "file": "Codes/framework/attentions/soa_module.py", | |
| "type": "Attention Module", | |
| "category": "Dynamic & Efficient Attention", | |
| "paradigm": "sparse", | |
| "icon": "\ud83c\udf1f", | |
| "description": "Shim for soa_module re-exporting from soa_attn.", | |
| "math": "W \\approx U_{:, :r} \\Sigma_{:r, :r} V_{:, :r}^T, \\quad r=64", | |
| "markdown_preview": "# SOA Attention: Shared Orthogonal Attention with Latent Bottlenecks\n\n## 1. Mathematical Formulation & Architecture\nFactorizes Q, K, and V projections into low-rank representations initialized via truncated Singular Value Decomposition (SVD):\n$$W \\approx U_{:, :r} \\Sigma_{:r, :r} V_{:, :r}^T, \\quad r=64$$\nMulti-head attention computation:\n$$A = \\operatorname{Softmax}\\left(\\frac{Q K^T}{\\sqrt{d_h}} + M_{\\text{pad}}\\right), \\quad O = (A V) W_O$$\n\n\n## Architecture & Data Flow\n\nThe `LoRALinear` module coordinates input projection factorization, intermediate representation routing, and output synthe" | |
| }, | |
| { | |
| "id": "span_expanded_attention", | |
| "name": "Span-Expanded Attention: Receptive Field Extension with Parallel ", | |
| "file": "Codes/framework/attentions/span_expanded_attention.py", | |
| "type": "Attention Module", | |
| "category": "Low-Rank SVD & Factorization", | |
| "paradigm": "svd", | |
| "icon": "\u26a1", | |
| "description": "Span-Expanded Attention with HyLoRA \u2013 Production-Hardened. Extends attention receptive fields via fixed/cached key-value span expansion buffers, complemented by parallel low-rank adaptation (HyLoRA) for parameter-efficient adaptation. Supports: - Multi-architecture copy_weights (GPT-2, RoBERTa/BERT nested, Qwen/LLaMA, DistilBERT) - Trainable LoRA", | |
| "math": "A = \\operatorname{Softmax}\\left(\\frac{Q K^\\top}{\\sqrt{d}} + M\\right)", | |
| "markdown_preview": "# Span-Expanded Attention: Receptive Field Extension with Parallel HyLoRA Adaptation\n\n## 1. Executive Abstract & Architectural Motivation\n\nSpan-Expanded Attention (`SpanExpandedAttention`) provides an architectural framework designed to expand Transformer attention receptive fields while maintaining parameter efficiency. Standard Multi-Head Attention is constrained by sequence length bottlenecks ($L$), making context expansion costly during fine-tuning or long-document processing.\n\nSpan-Expanded Attention addresses this challenge through a dual-path design:\n1. **Receptive Field Extension**: Th" | |
| }, | |
| { | |
| "id": "spectra_v2", | |
| "name": "SPECTRA-v2: Sparse Per-Head Efficient Context Transformer with Re", | |
| "file": "Codes/framework/attentions/spectra_v2.py", | |
| "type": "Attention Module", | |
| "category": "Mixture of Experts & Routing", | |
| "paradigm": "moe", | |
| "icon": "\ud83d\udd00", | |
| "description": "SPECTRA-v2: Sparse Per-Head Efficient Context Transformer with Rectified Attention \u2013 Production-Hardened. Hybrid sparse-attention architecture unifying intra-block local attention, relative-position-biased inter-block top-k routing, per-head adaptive attention spans, and periodic global Random Fourier Feature (RFF) rectification. Supports: - Mult", | |
| "math": "M^{\\text{span}}_{h, i, j} = \\ln\\left( \\sigma\\left( 5.0 \\cdot (S_h - d_{ij}) \\right) + 10^{-5} \\right)", | |
| "markdown_preview": "# SPECTRA-v2: Sparse Per-Head Efficient Context Transformer with Rectified Attention\n\n## 1. Executive Abstract & Theoretical Foundations\n\n**SPECTRA-v2** (Sparse Per-Head Efficient Context Transformer with Rectified Attention) introduces a multi-scale hybrid attention mechanism designed to break quadratic Transformer scaling $\\mathcal{O}(L^2)$ while maintaining full expressive capacity across long context horizons:\n1. **Intra-Block Local Attention with Learned Adaptive Spans**: Evaluates high-fidelity local token interactions within fixed block tiles ($B_{\\text{size}} = 64$) modulated by a cont" | |
| }, | |
| { | |
| "id": "spectra_v2_attention", | |
| "name": "Re-export module for SpectraV2Attention.", | |
| "file": "Codes/framework/attentions/spectra_v2_attention.py", | |
| "type": "Attention Module", | |
| "category": "Dynamic & Efficient Attention", | |
| "paradigm": "sparse", | |
| "icon": "\ud83c\udf1f", | |
| "description": "Re-export module for SpectraV2Attention.", | |
| "math": "A_{ij} = \\text{TopK}\\left(\\text{Softmax}\\left(\\frac{Q_i K_j^\\top}{\\sqrt{d}}\\right)\\right)", | |
| "markdown_preview": "# Spectra V2 Attention\n\n## Overview\n## Configuration & Hyperparameters\n\n| Parameter | Description |\n|-----------|-------------|\n\n## Usage Examples\n\n```python\nfrom Codes.framework.attentions import Module\n```\n\n## Benchmark Results\n\n| Status | Accuracy | Val Loss | Peak VRAM | Inference Latency |\n|--------|----------|----------|-----------|-------------------|\n| Pending| Pending | Pending | Pending | Pending |\n\n## Limitations & Future Work\n\nTODO: Identify specific limitations and edge cases for this module.\n" | |
| }, | |
| { | |
| "id": "spectral_mamba", | |
| "name": "Spectral-Mamba Attention Module: Hybrid Attention-State Space Mod", | |
| "file": "Codes/framework/attentions/spectral_mamba.py", | |
| "type": "Attention Module", | |
| "category": "Linear Attention & State-Space (SSM)", | |
| "paradigm": "ssm", | |
| "icon": "\ud83d\udc0d", | |
| "description": "Spectral-Mamba Attention Module: Hybrid Attention-State Space Module with Spectral Channel Modulation \u2013 Production-Hardened. Combines pretrained multi-head attention with a Spectral Mamba SSM branch via learned adaptive gating. Supports: - Multi-architecture copy_weights (GPT-2, RoBERTa/BERT nested, Qwen/LLaMA, DistilBERT) - Trainable LoRA adapte", | |
| "math": "Q = X W_Q, \\quad K = X W_K, \\quad V = X W_V", | |
| "markdown_preview": "# Spectral-Mamba: Hybrid Attention-State Space Module with Spectral Modulation\n\n## 1. Executive Abstract & Theoretical Foundations\n\nApplying pure State Space Models (SSMs) from scratch to pretrained bidirectional Transformers (such as DistilBERT) results in catastrophic performance collapse (stalling at 50.92% random guessing on SST-2) due to the loss of pretrained self-attention representations. **Spectral-Mamba Attention** resolves this representational bottleneck by establishing a dual-branch hybrid architecture:\n1. **Pretrained Multi-Head Self-Attention Stream**: Retains the complete pretr" | |
| }, | |
| { | |
| "id": "spectral_mamba_attention", | |
| "name": "Re-export module for SpectralMambaAttention.", | |
| "file": "Codes/framework/attentions/spectral_mamba_attention.py", | |
| "type": "Attention Module", | |
| "category": "Linear Attention & State-Space (SSM)", | |
| "paradigm": "ssm", | |
| "icon": "\ud83d\udc0d", | |
| "description": "Re-export module for SpectralMambaAttention.", | |
| "math": "h_t = \\bar{A} h_{t-1} + \\bar{B} x_t, \\quad y_t = C h_t + D x_t, \\quad \\bar{A} = \\exp(\\Delta A)", | |
| "markdown_preview": "# Spectral Mamba Attention\n\n## Overview\n## Configuration & Hyperparameters\n\n| Parameter | Description |\n|-----------|-------------|\n\n## Usage Examples\n\n```python\nfrom Codes.framework.attentions import Module\n```\n\n## Benchmark Results\n\n| Status | Accuracy | Val Loss | Peak VRAM | Inference Latency |\n|--------|----------|----------|-----------|-------------------|\n| Pending| Pending | Pending | Pending | Pending |\n\n## Limitations & Future Work\n\nTODO: Identify specific limitations and edge cases for this module.\n" | |
| }, | |
| { | |
| "id": "spectron", | |
| "name": "SpectronAttention: Importance-Aware Low-Rank + Sparse Attention.", | |
| "file": "Codes/framework/attentions/spectron.py", | |
| "type": "Attention Module", | |
| "category": "Low-Rank SVD & Factorization", | |
| "paradigm": "svd", | |
| "icon": "\u26a1", | |
| "description": "SpectronAttention: Importance-Aware Low-Rank + Sparse Attention. Dynamically allocates non-uniform low-rank capacities across attention projection matrices under a global parameter budget via Lagrangian spectral entropy optimization, optionally augmented by trainable magnitude-pruned sparse residuals. Mathematical Mechanism: For projection ma", | |
| "math": "\\min_{\\{r_i\\}} \\sum_{i=1}^M \\left(1 - \\frac{\\sum_{k=1}^{r_i} \\sigma_{i, k}^2}{\\sum_k \\sigma_{i, k}^2}\\right) \\quad \\text{s.t.} \\quad \\sum_{i=1}^M r_i (d_{i, \\text{in}} + d_{i, \\text{out}}) \\le B_{\\text{target}}", | |
| "markdown_preview": "# Spectron: Importance-Aware Low-Rank + Sparse Attention\n\n## 1. Overview\nSpectron dynamically allocates non-uniform low-rank capacities across attention projection matrices under a global parameter budget via Lagrangian spectral entropy optimization, optionally augmented by trainable magnitude-pruned sparse residuals.\n\n\n## Architecture & Data Flow\n\nThe `LowRankSparseLinear` module coordinates input projection factorization, intermediate representation routing, and output synthesis:\n\n```\nInput Activation X [B, N, D]\n \u2502\n \u251c\u2500\u2500 Low-Rank Bottleneck Down-Projection: Z = X W_down [B, N, r]\n \u251c\u2500\u2500 " | |
| }, | |
| { | |
| "id": "spectron_attention", | |
| "name": "Spectron Attention Re-export Module.", | |
| "file": "Codes/framework/attentions/spectron_attention.py", | |
| "type": "Attention Module", | |
| "category": "Dynamic & Efficient Attention", | |
| "paradigm": "sparse", | |
| "icon": "\ud83c\udf1f", | |
| "description": "Spectron Attention Re-export Module. Enables backward compatibility for imports from `framework.attentions.spectron_attention`.", | |
| "math": "A_{ij} = \\text{TopK}\\left(\\text{Softmax}\\left(\\frac{Q_i K_j^\\top}{\\sqrt{d}}\\right)\\right)", | |
| "markdown_preview": "# Spectron Attention\n\n## Overview\n## Configuration & Hyperparameters\n\n| Parameter | Description |\n|-----------|-------------|\n\n## Usage Examples\n\n```python\nfrom Codes.framework.attentions import Module\n```\n\n## Benchmark Results\n\n| Status | Accuracy | Val Loss | Peak VRAM | Inference Latency |\n|--------|----------|----------|-----------|-------------------|\n| Pending| Pending | Pending | Pending | Pending |\n\n## Limitations & Future Work\n\nTODO: Identify specific limitations and edge cases for this module.\n" | |
| }, | |
| { | |
| "id": "squeezing_heads", | |
| "name": "Squeezing Heads Attention Re-export Module.", | |
| "file": "Codes/framework/attentions/squeezing_heads.py", | |
| "type": "Attention Module", | |
| "category": "Dynamic & Efficient Attention", | |
| "paradigm": "sparse", | |
| "icon": "\ud83c\udf1f", | |
| "description": "Squeezing Heads Attention Re-export Module. Enables backward compatibility for imports from `framework.attentions.squeezing_heads`.", | |
| "math": "A_{ij} = \\text{TopK}\\left(\\text{Softmax}\\left(\\frac{Q_i K_j^\\top}{\\sqrt{d}}\\right)\\right)", | |
| "markdown_preview": "# Squeezing Heads\n\n## Overview\n## Configuration & Hyperparameters\n\n| Parameter | Description |\n|-----------|-------------|\n\n## Usage Examples\n\n```python\nfrom Codes.framework.attentions import Module\n```\n\n## Benchmark Results\n\n| Status | Accuracy | Val Loss | Peak VRAM | Inference Latency |\n|--------|----------|----------|-----------|-------------------|\n| Pending| Pending | Pending | Pending | Pending |\n\n## Limitations & Future Work\n\nTODO: Identify specific limitations and edge cases for this module.\n" | |
| }, | |
| { | |
| "id": "squeezing_heads_distill", | |
| "name": "Squeezing Heads Distillation: Multi-Head Receptive Field Distilla", | |
| "file": "Codes/framework/attentions/squeezing_heads_distill.py", | |
| "type": "Attention Module", | |
| "category": "Dynamic & Efficient Attention", | |
| "paradigm": "sparse", | |
| "icon": "\ud83c\udf1f", | |
| "description": "Squeezing Heads Attention Distillation Layer. Multi-head attention wrapper designed for knowledge distillation and head capacity compression experiments. Hardened for production use across diverse transformer architectures: - DistilBERT (Flat MHA) - RoBERTa / BERT (Nested Encoder Self-Attention) - GPT-2 (Causal Decoder with Conv1D) - Qwen / LLaMA ", | |
| "math": "Q = X W_Q, \\quad K = X W_K, \\quad V = X W_V \\in \\mathbb{R}^{B \\times H \\times L \\times d}", | |
| "markdown_preview": "# Squeezing Heads Distillation: Multi-Head Receptive Field Distillation & Capacity Compression\n\n## 1. Abstract & Theoretical Motivation\n\nLarge Transformer architectures often exhibit structural redundancy across attention heads, where multiple heads capture near-identical spatial or semantic representations. Squeezing Heads Attention Distillation (`SqueezingHeadsDistiller`) provides a structured multi-head attention harness designed for knowledge distillation, head pruning, and capacity squeezing.\n\nThe module encapsulates standard multi-head scaled dot-product attention while incorporating per" | |
| }, | |
| { | |
| "id": "squeezing_heads_distiller", | |
| "name": "Squeezing Heads Distiller: Progressive Multi-Head Knowledge Disti", | |
| "file": "Codes/framework/attentions/squeezing_heads_distiller.py", | |
| "type": "Attention Module", | |
| "category": "Dynamic & Efficient Attention", | |
| "paradigm": "sparse", | |
| "icon": "\ud83c\udf1f", | |
| "description": "Shim for squeezing_heads_distiller re-exporting from squeezing_heads_distill.", | |
| "math": "\\mathcal{L} = \\alpha D_{\\text{KL}}(A_{\\text{teacher}} \\parallel A_{\\text{student}}) + \\beta \\|V_{\\text{teacher}} - V_{\\text{student}} W_{\\text{proj}}\\|_F^2", | |
| "markdown_preview": "# Squeezing Heads Distiller: Progressive Multi-Head Knowledge Distillation & Dimension Pruning\n\n## 1. Executive Abstract & Architectural Motivation\nThe **Squeezing Heads Distiller** module performs progressive cross-head knowledge distillation, compressing high-rank multi-head attention blocks into compact, high-throughput reduced-head representations. By supervising intermediate head attention logits and value representations via Kullback-Leibler divergence against an uncompressed teacher, Squeezing Heads eliminates redundant projection subspace rank while preserving contextual fidelity.\n\n---" | |
| }, | |
| { | |
| "id": "svd_attention", | |
| "name": "SVD Attention: Sequence-Dimension Singular Subspace Projection Ar", | |
| "file": "Codes/framework/attentions/svd_attention.py", | |
| "type": "Attention Module", | |
| "category": "Low-Rank SVD & Factorization", | |
| "paradigm": "svd", | |
| "icon": "\u26a1", | |
| "description": "Low-Rank SVD-Compressed Sequence Attention Mechanism. Reduces quadratic sequence complexity O(N^2) to linear in sequence length O(N * r) by projecting Key and Value sequence dimensions into the top-r left-singular subspace of the sequence representation. Hardened for production use across diverse transformer architectures: - DistilBERT (Flat MHA) ", | |
| "math": "X_{\\text{valid}} = X \\odot M_{\\text{valid}}", | |
| "markdown_preview": "# SVD Attention: Sequence-Dimension Singular Subspace Projection Architecture\n\n## 1. Executive Abstract & Architectural Motivation\n\nSVD Attention (`SVDLinearAttention` in `svd_attention.py`) introduces a sequence-dimension compression paradigm that reduces the quadratic sequence time and memory complexity of Multi-Head Attention ($\\mathcal{O}(N^2)$) to linear complexity ($\\mathcal{O}(N \\cdot r)$). While standard MHA evaluates pairwise interactions across all $N \\times N$ token pairs, SVD Attention projects Key and Value sequence dimensions into the top-$r$ left-singular subspace of the sequenc" | |
| }, | |
| { | |
| "id": "svd_compressed", | |
| "name": "SVD-Compressed Attention: Importance-Aware Lagrangian Rank Alloca", | |
| "file": "Codes/framework/attentions/svd_compressed.py", | |
| "type": "Attention Module", | |
| "category": "Low-Rank SVD & Factorization", | |
| "paradigm": "svd", | |
| "icon": "\u26a1", | |
| "description": "SVD-Compressed Attention with Importance-Aware Lagrangian Rank Allocation. Truncated Singular Value Decomposition (SVD) attention compression module with Lagrangian spectral-entropy optimal rank budgeting and optional low-rank plus sparse residual decomposition. Mathematical Mechanism: For projection weight matrices $\\{W_i\\}_{i=1}^M$ across a", | |
| "math": "E_i(r) = \\frac{\\sum_{j=1}^r \\sigma_{i, j}^2}{\\sum_{j=1}^{\\min(d_{\\text{in}}, d_{\\text{out}})} \\sigma_{i, j}^2}, \\quad \\operatorname{Cost}_i(r) = 1 - E_i(r)", | |
| "markdown_preview": "# SVD-Compressed Attention: Importance-Aware Lagrangian Rank Allocation Architecture\n\n## 1. Executive Abstract & Architectural Motivation\n\nSVD-Compressed Attention (`SVDCompressedAttention` in `svd_compressed.py`) introduces a global parameter optimization framework that compresses Transformer attention layers using Truncated Singular Value Decomposition (SVD) guided by Lagrangian spectral-entropy budget allocation. Standard model compression approaches apply static, uniform truncation ranks across all layers, ignoring the reality that different layers and projection matrices ($W_Q, W_K, W_V, " | |
| }, | |
| { | |
| "id": "svd_compressed_attention", | |
| "name": "SVD-Compressed Attention Re-export Module.", | |
| "file": "Codes/framework/attentions/svd_compressed_attention.py", | |
| "type": "Attention Module", | |
| "category": "Low-Rank SVD & Factorization", | |
| "paradigm": "svd", | |
| "icon": "\u26a1", | |
| "description": "SVD-Compressed Attention Re-export Module. Enables backward compatibility for imports from `framework.attentions.svd_compressed_attention`.", | |
| "math": "W \\approx U_r \\Sigma_r V_r^\\top = A B^\\top, \\quad r \\ll \\min(d_{in}, d_{out})", | |
| "markdown_preview": "# Svd Compressed Attention\n\n## Overview\n## Configuration & Hyperparameters\n\n| Parameter | Description |\n|-----------|-------------|\n\n## Usage Examples\n\n```python\nfrom Codes.framework.attentions import Module\n```\n\n## Benchmark Results\n\n| Status | Accuracy | Val Loss | Peak VRAM | Inference Latency |\n|--------|----------|----------|-----------|-------------------|\n| Pending| Pending | Pending | Pending | Pending |\n\n## Limitations & Future Work\n\nTODO: Identify specific limitations and edge cases for this module.\n" | |
| }, | |
| { | |
| "id": "svd_linear_attention", | |
| "name": "SVD Factorized Linear Attention & Kernelized Linear Attention Mod", | |
| "file": "Codes/framework/attentions/svd_linear_attention.py", | |
| "type": "Attention Module", | |
| "category": "Low-Rank SVD & Factorization", | |
| "paradigm": "svd", | |
| "icon": "\u26a1", | |
| "description": "SVD Factorized Linear Attention & Kernelized Linear Attention Modules with Exponential Decay Gating. Hardened for production use across diverse transformer architectures: - DistilBERT (Flat MHA) - RoBERTa / BERT (Nested Encoder Self-Attention) - GPT-2 (Causal Decoder with Conv1D) - Qwen / LLaMA (GQA with Rotary Position Embeddings)", | |
| "math": "\\\\text{Attention}(Q, K, V) = \\\\frac{\\\\phi(Q) (\\\\phi(K)^\\\\top V)}{\\\\phi(Q) \\\\sum_j \\\\phi(k_j)^\\\\top + \\\\epsilon}", | |
| "markdown_preview": "# SVD Linear Attention: Semi-Orthogonal Low-Rank Projection Factorization\n\n## 1. Executive Abstract & Architectural Motivation\n\nSVD Linear Attention (`SVDLinearAttention`) introduces a zero-shot model compression framework for Multi-Head Attention by replacing standard dense linear projection layers ($W_Q, W_K, W_V, W_O \\in \\mathbb{R}^{D \\times D}$) with low-rank factorized modules (`SVDLinear`). Standard dense projections consume $4 D^2$ parameter storage per layer, creating significant memory bandwidth constraints during inference.\n\n`SVDLinearAttention` factorizes each projection matrix into" | |
| }, | |
| { | |
| "id": "svd_linear_decay", | |
| "name": "SVD Linear Decay Gated Attention Alias.", | |
| "file": "Codes/framework/attentions/svd_linear_decay.py", | |
| "type": "Attention Module", | |
| "category": "Low-Rank SVD & Factorization", | |
| "paradigm": "svd", | |
| "icon": "\u26a1", | |
| "description": "SVD Linear Decay Gated Attention Alias. Maps to `framework.attentions.svd_linear_attention.SVDLinearAttention`.", | |
| "math": "W \\approx U_r \\Sigma_r V_r^\\top = A B^\\top, \\quad r \\ll \\min(d_{in}, d_{out})", | |
| "markdown_preview": "# Svd Linear Decay\n\n## Overview\nDecay Gated SVD Linear Attention.\n\n## Configuration & Hyperparameters\n\n| Parameter | Description |\n|-----------|-------------|\n\n## Usage Examples\n\n```python\nfrom Codes.framework.attentions import DecayGatedLinearAttention\n```\n\n## Benchmark Results\n\n| Status | Accuracy | Val Loss | Peak VRAM | Inference Latency |\n|--------|----------|----------|-----------|-------------------|\n| Pending| Pending | Pending | Pending | Pending |\n\n## Limitations & Future Work\n\nTODO: Identify specific limitations and edge cases for this module.\n" | |
| }, | |
| { | |
| "id": "svd_llm_v2", | |
| "name": "SVD-LLM V2 Attention Alias.", | |
| "file": "Codes/framework/attentions/svd_llm_v2.py", | |
| "type": "Attention Module", | |
| "category": "Low-Rank SVD & Factorization", | |
| "paradigm": "svd", | |
| "icon": "\u26a1", | |
| "description": "SVD-LLM V2 Attention Alias. Maps to `framework.attentions.svd_llm_v2_attention.SVDLLMV2Attention`.", | |
| "math": "W \\approx U_r \\Sigma_r V_r^\\top = A B^\\top, \\quad r \\ll \\min(d_{in}, d_{out})", | |
| "markdown_preview": "# Svd Llm V2\n\n## Overview\n## Configuration & Hyperparameters\n\n| Parameter | Description |\n|-----------|-------------|\n\n## Usage Examples\n\n```python\nfrom Codes.framework.attentions import Module\n```\n\n## Benchmark Results\n\n| Status | Accuracy | Val Loss | Peak VRAM | Inference Latency |\n|--------|----------|----------|-----------|-------------------|\n| Pending| Pending | Pending | Pending | Pending |\n\n## Limitations & Future Work\n\nTODO: Identify specific limitations and edge cases for this module.\n" | |
| }, | |
| { | |
| "id": "svd_llm_v2_attention", | |
| "name": "SVD-LLM V2 Attention: Truncation-Aware Explicit Factor Buffer Arc", | |
| "file": "Codes/framework/attentions/svd_llm_v2_attention.py", | |
| "type": "Attention Module", | |
| "category": "Low-Rank SVD & Factorization", | |
| "paradigm": "svd", | |
| "icon": "\u26a1", | |
| "description": "SVD-LLM V2 Attention: Truncation-Aware SVD for LLM Compression. Direct low-rank factorized parameterization maintaining explicit U, Sigma, and V^H factor buffers, with trainable LoRA adapters for recovery fine-tuning. Hardened for production use across diverse transformer architectures: - DistilBERT (Flat MHA) - RoBERTa / BERT (Nested Encoder Self", | |
| "math": "\\frac{\\sum_{i=1}^{r^*} \\sigma_i^2}{\\sum_{j=1}^d \\sigma_j^2} \\ge \\tau \\quad (\\tau = 0.92)", | |
| "markdown_preview": "# SVD-LLM V2 Attention: Truncation-Aware Explicit Factor Buffer Architecture\n\n## 1. Executive Abstract & Architectural Motivation\n\nSVD-LLM V2 (`SVDLLMV2Attention`) provides a truncation-aware SVD compression framework designed for Large Language Models (LLMs). Traditional post-training quantization and pruning methods often degrade generative accuracy when applied to LLMs due to outlier activation dimensions. Truncated SVD parameterization preserves global matrix energy while reducing parameter memory footprint.\n\nSVD-LLM V2 parameterizes projection operators ($W_Q, W_K, W_V, W_O$) using explic" | |
| }, | |
| { | |
| "id": "svd_mamba", | |
| "name": "SVD-Mamba Attention: Hardware-Efficient State-Space Recurrence vi", | |
| "file": "Codes/framework/attentions/svd_mamba.py", | |
| "type": "Attention Module", | |
| "category": "Low-Rank SVD & Factorization", | |
| "paradigm": "svd", | |
| "icon": "\u26a1", | |
| "description": "SVD-Mamba: SVD-factorized Selective State Space Model. Hardened for production use across diverse transformer architectures: - DistilBERT (Flat MHA) - RoBERTa / BERT (Nested Encoder Self-Attention) - GPT-2 (Causal Decoder with Conv1D) - Qwen / LLaMA (GQA with Rotary Position Embeddings)", | |
| "math": "W = U \\Sigma V^\\top \\approx W_{\\text{down}} W_{\\text{up}}", | |
| "markdown_preview": "# SVD-Mamba Attention: Hardware-Efficient State-Space Recurrence via Truncated Singular Value Projections\n\n## 1. Executive Abstract & Architectural Motivation\nThe **SVD-Mamba** architecture integrates selective state-space model (SSM) recurrence dynamics with low-rank Singular Value Decomposition (SVD) input/output projection bottlenecks. By replacing high-dimensional quadratic attention matrices with selective continuous-time SSM dynamics discretized via zero-order hold (ZOH) coupled with factorized projection operators, SVD-Mamba achieves strict $\\mathcal{O}(N)$ computational time and $\\math" | |
| }, | |
| { | |
| "id": "svd_proj_attn", | |
| "name": "SVD Projection Attention Alias.", | |
| "file": "Codes/framework/attentions/svd_proj_attn.py", | |
| "type": "Attention Module", | |
| "category": "Low-Rank SVD & Factorization", | |
| "paradigm": "svd", | |
| "icon": "\u26a1", | |
| "description": "SVD Projection Attention Alias. Maps to `framework.attentions.svd_projection_attention.SVDProjectionAttention`.", | |
| "math": "W \\approx U_r \\Sigma_r V_r^\\top = A B^\\top, \\quad r \\ll \\min(d_{in}, d_{out})", | |
| "markdown_preview": "# Svd Proj Attn\n\n## Overview\n## Configuration & Hyperparameters\n\n| Parameter | Description |\n|-----------|-------------|\n\n## Usage Examples\n\n```python\nfrom Codes.framework.attentions import Module\n```\n\n## Benchmark Results\n\n| Status | Accuracy | Val Loss | Peak VRAM | Inference Latency |\n|--------|----------|----------|-----------|-------------------|\n| Pending| Pending | Pending | Pending | Pending |\n\n## Limitations & Future Work\n\nTODO: Identify specific limitations and edge cases for this module.\n" | |
| }, | |
| { | |
| "id": "svd_projection_attention", | |
| "name": "SVD Projection Attention: Explicit Low-Rank Matrix Reconstruction", | |
| "file": "Codes/framework/attentions/svd_projection_attention.py", | |
| "type": "Attention Module", | |
| "category": "Low-Rank SVD & Factorization", | |
| "paradigm": "svd", | |
| "icon": "\u26a1", | |
| "description": "SVD Projection Attention. Each projection (Q, K, V, Out) is factored into low-rank SVD components U and V, with trainable LoRA adapters for recovery fine-tuning. Hardened for production use across diverse transformer architectures: - DistilBERT (Flat MHA) - RoBERTa / BERT (Nested Encoder Self-Attention) - GPT-2 (Causal Decoder with Conv1D) - Qwen ", | |
| "math": "W = U V \\in \\mathbb{R}^{d_{\\text{out}} \\times d_{\\text{in}}}", | |
| "markdown_preview": "# SVD Projection Attention: Explicit Low-Rank Matrix Reconstruction Paradigm\n\n## 1. Executive Abstract & Architectural Motivation\n\nSVD Projection Attention (`SVDProjectionAttention`) provides a zero-shot model compression framework for Multi-Head Attention by factorizing linear projection operators into explicit low-rank matrix pairs ($U \\in \\mathbb{R}^{d_{\\text{out}} \\times r}, V \\in \\mathbb{R}^{r \\times d_{\\text{in}}}$) via `SVDProjectionLinear`. Standard dense linear projections ($W_Q, W_K, W_V, W_O \\in \\mathbb{R}^{D \\times D}$) require $4 D^2$ parameter storage per layer.\n\n`SVDProjectionAt" | |
| }, | |
| { | |
| "id": "svd_sequence_attention", | |
| "name": "Low-Rank SVD Sequence Attention Re-export Module.", | |
| "file": "Codes/framework/attentions/svd_sequence_attention.py", | |
| "type": "Attention Module", | |
| "category": "Low-Rank SVD & Factorization", | |
| "paradigm": "svd", | |
| "icon": "\u26a1", | |
| "description": "Low-Rank SVD Sequence Attention Re-export Module. Enables backward compatibility for imports from `framework.attentions.svd_sequence_attention`.", | |
| "math": "W \\approx U_r \\Sigma_r V_r^\\top = A B^\\top, \\quad r \\ll \\min(d_{in}, d_{out})", | |
| "markdown_preview": "# Svd Sequence Attention\n\n## Overview\n## Configuration & Hyperparameters\n\n| Parameter | Description |\n|-----------|-------------|\n\n## Usage Examples\n\n```python\nfrom Codes.framework.attentions import Module\n```\n\n## Benchmark Results\n\n| Status | Accuracy | Val Loss | Peak VRAM | Inference Latency |\n|--------|----------|----------|-----------|-------------------|\n| Pending| Pending | Pending | Pending | Pending |\n\n## Limitations & Future Work\n\nTODO: Identify specific limitations and edge cases for this module.\n" | |
| }, | |
| { | |
| "id": "template_attention", | |
| "name": "Template Attention: Reference Implementation & Standardization Ba", | |
| "file": "Codes/framework/attentions/template_attention.py", | |
| "type": "Attention Module", | |
| "category": "Dynamic & Efficient Attention", | |
| "paradigm": "sparse", | |
| "icon": "\ud83c\udf1f", | |
| "description": "Template Attention Mechanism (YourCustomAttention). Hardened for production use across diverse transformer architectures: - DistilBERT (Flat MHA) - RoBERTa / BERT (Nested Encoder Self-Attention) - GPT-2 (Causal Decoder with Conv1D) - Qwen / LLaMA (GQA with Rotary Position Embeddings)", | |
| "math": "Q = X W_Q, \\quad K = X W_K, \\quad V = X W_V", | |
| "markdown_preview": "# Template Attention: Reference Implementation & Standardization Baseline\n\n## 1. Executive Abstract & Architectural Motivation\nThe **Template Attention** module provides a standardized reference implementation designed to establish rigorous architectural conventions, lifecycle hooks, and interface contracts across the toolkit. It demonstrates uniform multi-head projection mapping, rotary position embeddings (RoPE), optional group-query attention (GQA), and low-rank adapter injection interfaces.\n\n---\n\n## 2. Mathematical Formulations & Derivations\nGiven an input activation tensor $X \\in \\mathbb{" | |
| }, | |
| { | |
| "id": "token_compressed", | |
| "name": "Token-Compressed Attention Wrapper: Structural Abstraction & Pass", | |
| "file": "Codes/framework/attentions/token_compressed.py", | |
| "type": "Attention Module", | |
| "category": "Dynamic & Efficient Attention", | |
| "paradigm": "sparse", | |
| "icon": "\ud83c\udf1f", | |
| "description": "Token-Compressed Attention. Hardened for production use across diverse transformer architectures: - DistilBERT (Flat MHA) - RoBERTa / BERT (Nested Encoder Self-Attention) - GPT-2 (Causal Decoder with Conv1D) - Qwen / LLaMA (GQA with Rotary Position Embeddings)", | |
| "math": "\\min_{\\text{rank}(\\widehat{W}) \\le r} \\|W - \\widehat{W}\\|_F^2 = \\sum_{i=r+1}^{D} \\sigma_i^2", | |
| "markdown_preview": "# Token-Compressed Attention Wrapper: Structural Abstraction & Pass-Through Layer-Wise Delegator\n\n## 1. Abstract & Theoretical Motivation\n\nIn progressive network distillation, dynamic token pruning, and structural model compression benchmarks, maintaining a unified API interface across heterogeneous attention layers is critical. The Token-Compressed Attention Wrapper (`TokenCompressedAttention`) serves as a zero-overhead structural pass-through abstraction layer within the attention registry.\n\nBy wrapping pre-existing attention modules (`old_attn`) and enforcing a normalized output contract $\\" | |
| }, | |
| { | |
| "id": "token_compressed_attention", | |
| "name": "Token Compressed Attention: Saliency-Based Key-Value Pruning", | |
| "file": "Codes/framework/attentions/token_compressed_attention.py", | |
| "type": "Attention Module", | |
| "category": "Dynamic & Efficient Attention", | |
| "paradigm": "sparse", | |
| "icon": "\ud83c\udf1f", | |
| "description": "Token-Compressed Attention. Hardened for production use across diverse transformer architectures: - DistilBERT (Flat MHA) - RoBERTa / BERT (Nested Encoder Self-Attention) - GPT-2 (Causal Decoder with Conv1D) - Qwen / LLaMA (GQA with Rotary Position Embeddings)", | |
| "math": "\\text{energy}_i = \\|x_i\\|_2, \\quad \\text{score}_i = \\text{ScoreNet}(x_i)", | |
| "markdown_preview": "# Token Compressed Attention: Saliency-Based Key-Value Pruning\n\n## 1. Mathematical Formulation & Architecture\nToken Compressed Attention (`TokenCompressedAttention`) reduces memory footprint and computational complexity by dynamically pruning redundant Key and Value sequence tokens while strictly preserving critical structural anchors such as the $[\\text{CLS}]$ token (index 0).\n\nGiven input token sequence $X \\in \\mathbb{R}^{B \\times L \\times D}$:\n1. **Token Saliency & Energy Scoring**:\n $$\\text{energy}_i = \\|x_i\\|_2, \\quad \\text{score}_i = \\text{ScoreNet}(x_i)$$\n $$s_i = \\text{energy}_i + " | |
| }, | |
| { | |
| "id": "tpa_attention", | |
| "name": "TPAttention: Tensor Product Attention with Factorised Query-Key P", | |
| "file": "Codes/framework/attentions/tpa_attention.py", | |
| "type": "Attention Module", | |
| "category": "Low-Rank SVD & Factorization", | |
| "paradigm": "svd", | |
| "icon": "\u26a1", | |
| "description": "TPAttention: Tensor Product Attention with Factorised Query-Key Projections. Hardened for production use across diverse transformer architectures: - DistilBERT (Flat MHA) - RoBERTa / BERT (Nested Encoder Self-Attention) - GPT-2 (Causal Decoder with Conv1D) - Qwen / LLaMA (GQA with Rotary Position Embeddings)", | |
| "math": "q_1 = \\text{LoRALinear}_{Q1}(X) \\in \\mathbb{R}^{B \\times L \\times (H \\cdot r)}, \\quad q_2 = \\text{LoRALinear}_{Q2}(X) \\in \\mathbb{R}^{B \\times L \\times (H \\cdot r)}", | |
| "markdown_preview": "# TPAttention: Tensor Product Attention with Factorised Query-Key Projections\n\n## 1. Executive Abstract & Architectural Motivation\n\nTensor Product Attention (`TPAttention`) introduces a polynomial kernel projection paradigm that factorizes traditional dense Query and Key projections into outer tensor products of low-rank vectors. In standard Multi-Head Attention, computing Query and Key projections requires full-rank matrix parameters $W_Q, W_K \\in \\mathbb{R}^{D \\times D}$.\n\n`TPAttention` replaces standard $D \\to D$ linear maps with pair-factorised projections: $q_1, q_2 \\in \\mathbb{R}^{B \\tim" | |
| }, | |
| { | |
| "id": "tt", | |
| "name": "TokenCompressedAttention: A No\u2011Compression Baseline Wrapper for A", | |
| "file": "Codes/framework/attentions/tt.py", | |
| "type": "Attention Module", | |
| "category": "Dynamic & Efficient Attention", | |
| "paradigm": "sparse", | |
| "icon": "\ud83c\udf1f", | |
| "description": "Token-Compressed Attention Wrapper / Alias (tt.py). Exposes TokenCompressedAttention and injection helpers from token_compressed_attention.", | |
| "math": "\\text{Output} = \\text{BaseAttn}(X, M)", | |
| "markdown_preview": "# TokenCompressedAttention: A No\u2011Compression Baseline Wrapper for Attention\n\n**Abstract**\n\nWe present **TokenCompressedAttention**, a trivial wrapper module that acts as a baseline or placeholder within the compression framework. The module does not modify the attention mechanism\u2014it simply wraps the original pretrained attention layer and delegates all forward calls to it without any compression, token reduction, or modification. Its primary purpose is to provide a consistent interface for the framework\u2019s injection system, allowing a \u201ctoken_compressed\u201d attention type to be registered and selec" | |
| }, | |
| { | |
| "id": "unified_rope", | |
| "name": "Universal Rotary Position Embedding (UnifiedRoPE): Positional Inv", | |
| "file": "Codes/framework/attentions/unified_rope.py", | |
| "type": "Attention Module", | |
| "category": "Dynamic & Efficient Attention", | |
| "paradigm": "sparse", | |
| "icon": "\ud83c\udf1f", | |
| "description": "Universal Rotary Position Embedding Attention (UnifiedRoPE). Hardened for production use across diverse transformer architectures: - DistilBERT (Flat MHA) - RoBERTa / BERT (Nested Encoder Self-Attention) - GPT-2 (Causal Decoder with Conv1D) - Qwen / LLaMA (GQA with Rotary Position Embeddings)", | |
| "math": "Q = \\operatorname{LoRALinear}_Q(X) \\in \\mathbb{R}^{B \\times L \\times (H \\cdot d_{\\text{head}})}", | |
| "markdown_preview": "# Universal Rotary Position Embedding (UnifiedRoPE): Positional Invariance across Attention & SSM Branches\n\n[](../../catalog.json)\n[](../../../Codes/framework/core/registry.py)\n[](../../../Codes/framework/attentions/unified_rope.py)\n\n## 1. Executive Abstract & Architectural Motivation\n\nUniversal Rotary Position Embedding (`UnifiedRoPE`) provides a unified, hardware-effic" | |
| }, | |
| { | |
| "id": "upi_mamba", | |
| "name": "UPI-Stuffed Mamba: Uncertainty-Penalized Interpolated Mamba SSM", | |
| "file": "Codes/framework/attentions/upi_mamba.py", | |
| "type": "Attention Module", | |
| "category": "Linear Attention & State-Space (SSM)", | |
| "paradigm": "ssm", | |
| "icon": "\ud83d\udc0d", | |
| "description": "UPI-Mamba re-export module. Provides direct access to UPIMambaBlock and inject_upi_mamba.", | |
| "math": "[X_{\\text{ssm}}, Z] = X W_{\\text{in}} \\in \\mathbb{R}^{B \\times L \\times 2 D}", | |
| "markdown_preview": "# UPI-Stuffed Mamba: Uncertainty-Penalized Interpolated Mamba SSM\n\n## 1. Executive Abstract\n**UPI-Stuffed Mamba** extends the Mamba selective state-space architecture with an input-dependent uncertainty penalty $\\gamma_t$ that dynamically interpolates between persistent associative memory and fading memory, enabling calibrated forgetting and robust fine-tuning across bidirectional encoder and causal decoder architectures.\n\n---\n\n## 2. Mathematical Equations & Recurrence\n\n### 2.1 Input Projection & Depthwise Convolution\nFor an input sequence $X \\in \\mathbb{R}^{B \\times L \\times D}$:\n$$[X_{\\text{" | |
| }, | |
| { | |
| "id": "upi_stuffed_mamba", | |
| "name": "UPI-Stuffed Mamba: Uncertainty-Penalized Interpolated Mamba SSM B", | |
| "file": "Codes/framework/attentions/upi_stuffed_mamba.py", | |
| "type": "Attention Module", | |
| "category": "Linear Attention & State-Space (SSM)", | |
| "paradigm": "ssm", | |
| "icon": "\ud83d\udc0d", | |
| "description": "UPI-Stuffed Mamba: Uncertainty-Penalized Interpolated Mamba SSM Block. Hardened for production use across diverse transformer architectures: - DistilBERT (Flat MHA) - RoBERTa / BERT (Nested Encoder Self-Attention) - GPT-2 (Causal Decoder with Conv1D) - Qwen / LLaMA (GQA with Rotary Position Embeddings)", | |
| "math": "[X_{\\text{ssm}}, Z] = X W_{\\text{in}} \\in \\mathbb{R}^{B \\times L \\times 2 D}", | |
| "markdown_preview": "# UPI-Stuffed Mamba: Uncertainty-Penalized Interpolated Mamba SSM\n\n## 1. Executive Abstract\n**UPI-Stuffed Mamba** extends the Mamba selective state-space architecture with an input-dependent uncertainty penalty $\\gamma_t$ that dynamically interpolates between persistent associative memory and fading memory, enabling calibrated forgetting and robust fine-tuning across bidirectional encoder and causal decoder architectures.\n\n---\n\n## 2. Mathematical Equations & Recurrence\n\n### 2.1 Input Projection & Depthwise Convolution\nFor an input sequence $X \\in \\mathbb{R}^{B \\times L \\times D}$:\n$$[X_{\\text{" | |
| }, | |
| { | |
| "id": "vig_attention", | |
| "name": "ViG Attention: Vision GNN-inspired 2D Grid Linear Attention with ", | |
| "file": "Codes/framework/attentions/vig_attention.py", | |
| "type": "Attention Module", | |
| "category": "Dynamic & Efficient Attention", | |
| "paradigm": "sparse", | |
| "icon": "\ud83c\udf1f", | |
| "description": "ViG Attention: Vision GNN-inspired 2D Grid Linear Attention with Gated Fusion. Hardened for production use across diverse transformer architectures: - DistilBERT (Flat MHA) - RoBERTa / BERT (Nested Encoder Self-Attention) - GPT-2 (Causal Decoder with Conv1D) - Qwen / LLaMA (GQA with Rotary Position Embeddings)", | |
| "math": "\\text{Total MACs} \\approx 4 B N D^2 + 4 B N D d_k", | |
| "markdown_preview": "# ViG Attention: Bidirectional Row-Column ELU Linear Recurrent Grid Attention\n\n## 1. Executive Abstract & Architectural Motivation\n\nStandard Multi-Head Attention treats sequences as 1D structures, incurring $\\mathcal{O}(N^2)$ quadratic complexity over length $N$. Vision Grid Attention (ViG) reformulates 1D token sequences of length $N$ into a 2D spatial grid of dimensions $H \\times W$ (where $H \\cdot W \\ge N$). \n\nBy factorizing attention interactions into separate row-wise and column-wise linear attention passes using non-negative feature maps $\\phi(x) = \\text{ELU}(x) + 1.0$, ViG computes bidi" | |
| }, | |
| { | |
| "id": "vspec", | |
| "name": "VSpec++ Attention: Variational Spectral Token-Adaptive Attention ", | |
| "file": "Codes/framework/attentions/vspec.py", | |
| "type": "Attention Module", | |
| "category": "Dynamic & Efficient Attention", | |
| "paradigm": "sparse", | |
| "icon": "\ud83c\udf1f", | |
| "description": "VSpec++ Attention: Variational Spectral Token-Adaptive Attention. Hardened for production use across diverse transformer architectures: - DistilBERT (Flat MHA) - RoBERTa / BERT (Nested Encoder Self-Attention) - GPT-2 (Causal Decoder with Conv1D) - Qwen / LLaMA (GQA with Rotary Position Embeddings)", | |
| "math": "A_{ij} = \\text{TopK}\\left(\\text{Softmax}\\left(\\frac{Q_i K_j^\\top}{\\sqrt{d}}\\right)\\right)", | |
| "markdown_preview": "# VSpec++ Attention: Variational Spectral Token-Adaptive Attention with SVD-Gated Projections and Dynamic Top-k Routing\n\n## 1. Abstract & Theoretical Motivation\n\nStandard Multi-Head Self-Attention (MHSA) in Transformer architectures requires quadratic computational time and memory scaling $\\mathcal{O}(N^2)$ with sequence length $N$. Static low-rank compression methods enforce uniform projection ranks across all sequence tokens and batch examples, failing to adapt to varying input complexity.\n\n**VSpec++ Attention** (`VSpecAttention`, registered under `vspec` and `vspec_attention`) introduces a " | |
| }, | |
| { | |
| "id": "weighted_ensemble", | |
| "name": "Smart Weighted Ensemble with Task-Aware Attention and Hybrid Comb", | |
| "file": "Codes/framework/attentions/weighted_ensemble.py", | |
| "type": "Attention Module", | |
| "category": "Dynamic & Efficient Attention", | |
| "paradigm": "sparse", | |
| "icon": "\ud83c\udf1f", | |
| "description": "Smart Weighted Ensemble with Task-Aware Attention and Hybrid Combination. Implements multi-model ensemble fusion strategies combining: - Simple average pooling of per-model output logits. - Performance-history weighted voting with exponential moving average tracking. - Task-aware scaled dot-product attention over learned task embeddings. - Hybrid ", | |
| "math": "A_{ij} = \\text{TopK}\\left(\\text{Softmax}\\left(\\frac{Q_i K_j^\\top}{\\sqrt{d}}\\right)\\right)", | |
| "markdown_preview": "# Weighted Ensemble\n\n## Overview\nLearns to weight model predictions based on task embeddings and pairwise\ntask similarity matrix.\n\nEnsemble that combines multiple model predictions using various strategies:\n - simple_average\n - weighted by performance history\n - task-aware attention\n - hybrid (attention + dynamic confidence)\n\n## Configuration & Hyperparameters\n\n| Parameter | Description |\n|-----------|-------------|\n| `num_tasks` | TODO: Description for num_tasks |\n| `hidden_dim` | TODO: Description for hidden_dim |\n| `models` | TODO: Description for models |\n| `num_tasks` | TODO: " | |
| }, | |
| { | |
| "id": "wgqa_attention", | |
| "name": "Weighted Grouped-Query Attention (WGQA).", | |
| "file": "Codes/framework/attentions/wgqa_attention.py", | |
| "type": "Attention Module", | |
| "category": "Latent & KV-Cache Compression", | |
| "paradigm": "mla", | |
| "icon": "\ud83e\udde0", | |
| "description": "Weighted Grouped-Query Attention (WGQA). Hardened for production use across diverse transformer architectures: - DistilBERT (Flat MHA) - RoBERTa / BERT (Nested Encoder Self-Attention) - GPT-2 (Causal Decoder with Conv1D) - Qwen / LLaMA (GQA with Rotary Position Embeddings) Features learnable per-KV-head importance weights, robust GQA pooling, LoR", | |
| "math": "c^{KV}_t = W^{DKV} x_t, \\quad k^C_t = W^{UK} c^{KV}_t, \\quad v^C_t = W^{UV} c^{KV}_t", | |
| "markdown_preview": "# Weighted Grouped-Query Attention (WGQA)\n\n[](../../catalog.json)\n[](../../../Codes/framework/core/registry.py)\n[](../../../Codes/framework/eval_all_models_datasets.py)\n\n---\n\n## 1. Architectural Overview & Value Proposition\n\nThe `wgqa` module (implemented in [`Codes/framework/attentions/wgqa_attention.py`](file:///Users/tahamajs/Documents/uni/Project/Codes/framework/atte" | |
| }, | |
| { | |
| "id": "xattention", | |
| "name": "XAttention: Block-Sparse Coarse-to-Fine Routing with Unified Soft", | |
| "file": "Codes/framework/attentions/xattention.py", | |
| "type": "Attention Module", | |
| "category": "Mixture of Experts & Routing", | |
| "paradigm": "moe", | |
| "icon": "\ud83d\udd00", | |
| "description": "XAttention: Block Sparse Attention with Coarse-to-Fine Routing. Partitions sequence tokens into non-overlapping blocks, evaluates coarse block-level routing, and computes unified normalized fine-grained attention across top-k candidate blocks, with multi-architecture weight transfer, trainable LoRA residual adapters, GQA, RoPE, and causal decoding", | |
| "math": "y = \\sum_{i=1}^K G(x)_i \\cdot E_i(x), \\quad G(x) = \\text{Softmax}(\\text{TopK}(x W_g, K=2))", | |
| "markdown_preview": "# XAttention: Block-Sparse Coarse-to-Fine Routing with Unified Softmax Normalization\n\n## 1. Executive Abstract & Architectural Motivation\nStandard Multi-Head Self-Attention (MHSA) computes dense token-to-token similarity matrices of size $L \\times L$, scaling quadratically $\\mathcal{O}(L^2 d)$ in compute and memory. Conventional block-sparse attention models often fail to maintain mathematical fidelity with dense attention because they compute separate, un-unified local softmax normalizations per block before accumulating output vectors.\n\n**XAttention (`XAttention`)** overcomes this limitation" | |
| }, | |
| { | |
| "id": "zeros_attention", | |
| "name": "ZeroS Attention: Zero-Mean Normalized Linear Attention", | |
| "file": "Codes/framework/attentions/zeros_attention.py", | |
| "type": "Attention Module", | |
| "category": "Dynamic & Efficient Attention", | |
| "paradigm": "sparse", | |
| "icon": "\ud83c\udf1f", | |
| "description": "Zero-Mean Linear Attention (ZeroS Attention). Removes the mean from key representations and applies normalized linear attention with ReLU-based non-negative feature maps, with multi-architecture weight transfer, trainable LoRA residual adapters, GQA, RoPE, and causal decoding. Hardened for production across DistilBERT, RoBERTa, GPT-2, and Qwen3.5", | |
| "math": "A_{ij} = \\text{TopK}\\left(\\text{Softmax}\\left(\\frac{Q_i K_j^\\top}{\\sqrt{d}}\\right)\\right)", | |
| "markdown_preview": "# ZeroS Attention: Zero-Mean Normalized Linear Attention\n\n## 1. Abstract & Theoretical Motivation\nLinear attention frameworks eliminate quadratic $N \\times N$ attention matrices by using non-negative feature maps $\\phi(\\mathbf{x})$ and matrix product associativity. However, un-centered key activation vectors often suffer from directional bias and common-mode component domination, which causes rank collapse in accumulated key-value memory states.\n\nZero-Mean Linear Attention (ZeroS Attention) resolves this issue by introducing key centering prior to non-negative feature mapping. For bidirectiona" | |
| }, | |
| { | |
| "id": "a3_compressor", | |
| "name": "A3 Compressor: Activation-Aware Factorized SVD Compression with F", | |
| "file": "Codes/framework/compression_engine/a3_compressor.py", | |
| "type": "Compression Engine", | |
| "category": "Low-Rank SVD & Factorization", | |
| "paradigm": "svd", | |
| "icon": "\u26a1", | |
| "description": "A3 Compressor: Activation-Aware Factorized SVD Compression with Fisher/Activation Energy Weighting and LayerNorm Recalibration. Registered in COMPRESSOR_REGISTRY as 'a3' and 'a3_compressor'. Mathematical Formulation: 1. Online Streaming Activation Covariance & Empirical Fisher Diagonal Proxy (Activation Energy): $$C_X = \\\\frac{1}{N} \\\\sum_{t=1}", | |
| "math": "\\text{FLOPs}_{\\text{dense}} = 2 \\cdot m \\cdot n \\quad \\longrightarrow \\quad \\text{FLOPs}_{\\text{factorized}} = 2 \\cdot r \\cdot (m + n)", | |
| "markdown_preview": "# A3 Compression: Activation-Aware Factorized Low-Rank SVD\n\n**Registered as `a3` and `a3_compressor`** \n*A structured low-rank factorized compression technique for linear and projection layers in transformer models featuring streaming Fisher/activation energy weighting, reconstruction validation, and LayerNorm recalibration.*\n\n---\n\n## Overview\n\nA3 is a post-training structured factorized compression method that reduces parameter counts and computational complexity (FLOPs) in transformer linear layers (including self-attention projections $Q, K, V, O$ and MLP projections) by replacing their we" | |
| }, | |
| { | |
| "id": "aa_svd_compressor", | |
| "name": "Activation-Aware Singular Value Decomposition (AASVD) Compressor.", | |
| "file": "Codes/framework/compression_engine/aa_svd_compressor.py", | |
| "type": "Compression Engine", | |
| "category": "Low-Rank SVD & Factorization", | |
| "paradigm": "svd", | |
| "icon": "\u26a1", | |
| "description": "Activation-Aware Singular Value Decomposition (AASVD) Compressor. Implements streaming Exponential Moving Average (EMA) activation covariance accumulation: $$C_X^{(t)} = \\\\beta C_X^{(t-1)} + (1 - \\\\beta) \\\\frac{1}{B} \\\\sum_{i=1}^B X_i^\\\\top X_i$$ Projects weights onto principal activation subspaces and replaces dense `nn.Linear` layers with struct", | |
| "math": "W \\approx U_r \\Sigma_r V_r^\\top = A B^\\top, \\quad r \\ll \\min(d_{in}, d_{out})", | |
| "markdown_preview": "# Aasvd\n\n## Overview\nTrue Low-Rank Factorized Linear Layer executing as:\ny = (x @ V.T) @ U.T + ((x @ lora_A.T) @ lora_B.T) * lora_scaling + b\nAchieving physical parameter reduction from d_in * d_out to r * (d_in + d_out).\nSupports optional LoRA adapters for recovery fine-tuning.\n\nActivation-Aware SVD Compressor with streaming EMA covariance accumulation\nand true structural module replacement (SVDLinear).\n\nProduction hardening features:\n- Relative Frobenius reconstruction error validation with configurable max_error (default 0.20).\n- Safe fallback to original weights by default (fallback_to_ori" | |
| }, | |
| { | |
| "id": "aasvd", | |
| "name": "Activation-Aware Singular Value Decomposition (AASVD): Streaming ", | |
| "file": "Codes/framework/compression_engine/aasvd.py", | |
| "type": "Compression Engine", | |
| "category": "Low-Rank SVD & Factorization", | |
| "paradigm": "svd", | |
| "icon": "\u26a1", | |
| "description": "AASVD (Activation-Aware SVD) Compressor alias module. Re-exports AASVDCompressor from aa_svd_compressor.", | |
| "math": "W \\approx U_r \\Sigma_r V_r^\\top, \\quad \\min_{U, \\Sigma, V} \\left\\| (W - U \\Sigma V^\\top) X^\\top \\right\\|_F^2", | |
| "markdown_preview": "# Activation-Aware Singular Value Decomposition (AASVD): Streaming Covariance Preconditioning & Factorized Linear Transformation\n\n## 1. Executive Abstract & Theoretical Motivation\n\nStandard Singular Value Decomposition (SVD) operates under the isotropic assumption: given a static weight matrix $W \\in \\mathbb{R}^{d_{\\text{out}} \\times d_{\\text{in}}}$, the best rank-$r$ approximation under the Frobenius norm minimizes the uniform parameter error $\\|W - \\hat{W}\\|_F^2$. In deep Transformer networks, however, input activations $X \\in \\mathbb{R}^{B \\times L \\times d_{\\text{in}}}$ do not populate $\\m" | |
| }, | |
| { | |
| "id": "adaptive_compressor", | |
| "name": "AdaptiveTransformerCompressor: Dynamic Energy-Threshold Attention", | |
| "file": "Codes/framework/compression_engine/adaptive_compressor.py", | |
| "type": "Compression Engine", | |
| "category": "Low-Rank SVD & Factorization", | |
| "paradigm": "svd", | |
| "icon": "\u26a1", | |
| "description": "Adaptive Transformer Compressor: backward-compatible wrapper delegating to the unified and production-hardened AdaptiveRanksCompressor.", | |
| "math": "W \\approx U_r \\Sigma_r V_r^\\top = A B^\\top, \\quad r \\ll \\min(d_{in}, d_{out})", | |
| "markdown_preview": "# AdaptiveTransformerCompressor: Dynamic Energy-Threshold Attention Subspace Compression Engine\n\n## 1. Executive Abstract & Theoretical Motivation\nSelf-Attention projection matrices ($W_Q, W_K, W_V, W_O \\in \\mathbb{R}^{d_{\\text{model}} \\times d_{\\text{model}}}$) inside `nn.MultiheadAttention` layers exhibit non-uniform singular value spectra across transformer blocks. Early layers focus on local syntactic correlations and require lower ranks, while deeper layers process global semantic relationships demanding higher subspace capacity.\n\n`AdaptiveTransformerCompressor` (registered as `adaptive_s" | |
| }, | |
| { | |
| "id": "adaptive_ranks", | |
| "name": "Adaptive Ranks SVD Compressor: dynamic rank selection per layer b", | |
| "file": "Codes/framework/compression_engine/adaptive_ranks.py", | |
| "type": "Compression Engine", | |
| "category": "Low-Rank SVD & Factorization", | |
| "paradigm": "svd", | |
| "icon": "\u26a1", | |
| "description": "Adaptive Ranks SVD Compressor: dynamic rank selection per layer based on singular value energy spectra. Dynamic cumulative singular value energy thresholding: $$\\frac{\\sum_{i=1}^r \\sigma_i^2}{\\sum_{j=1}^d \\sigma_j^2} \\ge \\tau \\quad (\\tau = 0.92)$$ with minimum rank floor: $$r_{\\min} = \\max(16, \\lfloor 0.15 \\cdot \\min(d_{in}, d_{out}) \\rflo", | |
| "math": "\\\\frac{\\\\sum_{i=1}^r \\\\sigma_i^2}{\\\\sum_{j=1}^d \\\\sigma_j^2} \\\\ge \\\\tau \\\\quad (\\\\tau = 0.92)", | |
| "markdown_preview": "# AdaptiveTransformerCompressor (Ranks): Dynamic Subspace Allocation & Sensitivity-Guided Rank Profiling\n\n## 1. Executive Abstract & Theoretical Motivation\nModern deep transformer models feature multi-head attention and feed-forward projections with heterogeneous information densities. Assigning static, manual rank constraints across layers introduces performance bottlenecks or wastes parameter budgets on low-capacity layers.\n\n`adaptive_ranks.md` details the mathematical framework, rank allocation dynamics, and sensitivity profiling of the `AdaptiveTransformerCompressor` engine. By analyzing t" | |
| }, | |
| { | |
| "id": "adasvd_compressor", | |
| "name": "AdaSVDCompressor: Adaptive SVD via Nuclear-Norm Layer Importance ", | |
| "file": "Codes/framework/compression_engine/adasvd_compressor.py", | |
| "type": "Compression Engine", | |
| "category": "Low-Rank SVD & Factorization", | |
| "paradigm": "svd", | |
| "icon": "\u26a1", | |
| "description": "AdaSVD: Adaptive SVD with per-layer importance allocation and iterative ALS refinement. Registered as 'adasvd'. Production hardened with reconstruction validation, fallback, Conv1D support, rank reduction bounds, and comprehensive per-layer stats.", | |
| "math": "W \\approx U_r \\Sigma_r V_r^\\top = A B^\\top, \\quad r \\ll \\min(d_{in}, d_{out})", | |
| "markdown_preview": "# AdaSVDCompressor: Adaptive SVD via Nuclear-Norm Layer Importance & Iterative Matrix Refinement\n\n## 1. Executive Abstract & Theoretical Motivation\nUniform low-rank truncation across all neural network layers causes non-uniform accuracy degradation because different layers exhibit distinct singular value decay rates and information capacities. Critical transformer projections\u2014such as self-attention key/query matrices and MLP gate projections\u2014suffer severe performance drops under rigid rank constraints, whereas redundant layers can withstand aggressive rank reduction.\n\n`AdaSVDCompressor` addres" | |
| }, | |
| { | |
| "id": "admm_compressor", | |
| "name": "ADMM Compressor: Alternating Direction Method of Multipliers for ", | |
| "file": "Codes/framework/compression_engine/admm_compressor.py", | |
| "type": "Compression Engine", | |
| "category": "Structured Pruning & Channel Slicing", | |
| "paradigm": "pruning", | |
| "icon": "\u2702\ufe0f", | |
| "description": "ADMM Compressor: Alternating Direction Method of Multipliers for Joint Pruning and Quantization. Registered in COMPRESSOR_REGISTRY as 'admm' and 'admm_compressor'. Production hardened with reconstruction validation, Conv1D support, FP32 numerical stability, safe fallback, LoRALinear preservation, and attention-only targeting.", | |
| "math": "\\tau = \\text{TopK}_{k} \\left( \\{ |W_{i,j}| \\}_{i=1, j=1}^{N, M} \\right)", | |
| "markdown_preview": "# ADMM Compressor: Joint Alternating Direction Pruning and Low-Bit Quantization Engine\n\n## 1. Executive Abstract & Theoretical Motivation\n\nCombining model pruning and weight quantization often leads to compounded precision degradation when executed sequentially. **ADMM Compressor** solves this by unifying structured magnitude pruning and uniform low-bit quantization under a single optimization formulation.\n\nBy formulating the pruning and quantization constraints simultaneously, ADMM Compressor identifies a sparse grid of discrete quantization levels. The algorithm first computes an absolute-ma" | |
| }, | |
| { | |
| "id": "admm_pruner", | |
| "name": "ADMM Pruner: Alternating Direction Method of Multipliers with Dyn", | |
| "file": "Codes/framework/compression_engine/admm_pruner.py", | |
| "type": "Compression Engine", | |
| "category": "Structured Pruning & Channel Slicing", | |
| "paradigm": "pruning", | |
| "icon": "\u2702\ufe0f", | |
| "description": "ADMM Pruner: Alternating Direction Method of Multipliers for Structured & Semi-Structured Pruning with Layer-Wise Reconstruction Distillation, STE, and Dynamic Rho. Registered in COMPRESSOR_REGISTRY as 'admm_pruner' and 'admm_pruner_distill'. Mathematical Formulation: $$\\\\min_W \\\\frac{1}{2} \\\\|W - W_0\\\\|_F^2 + \\\\lambda \\\\cdot \\\\mathbb{E}_{x \\\\sim ", | |
| "math": "\\min_{W, Z} \\frac{1}{2} \\|W - W_0\\|_F^2 + g(Z) \\quad \\text{subject to} \\quad W - Z = 0", | |
| "markdown_preview": "# ADMM Pruner: Alternating Direction Method of Multipliers with Dynamic Penalty Scheduling & STE Backpropagation\n\n## 1. Executive Abstract & Theoretical Motivation\n\nPruning neural network weight tensors without sacrificing performance is a fundamental challenge in model compression. Standard magnitude pruning suffers from greedy decisions that create unrecoverable optimization bottlenecks. **ADMM Pruner** addresses this problem by formulating model pruning as a constrained non-convex optimization problem solved via the **Alternating Direction Method of Multipliers (ADMM)** with dynamic penalty" | |
| }, | |
| { | |
| "id": "armor_pruner", | |
| "name": "ARMOR Pruner: Adaptive Resilient Matrix Orthogonal Rank Pruning (", | |
| "file": "Codes/framework/compression_engine/armor_pruner.py", | |
| "type": "Compression Engine", | |
| "category": "Low-Rank SVD & Factorization", | |
| "paradigm": "svd", | |
| "icon": "\u26a1", | |
| "description": "ARMOR: Adaptive Resilient Matrix Orthogonal Rank Pruning (2:4 Sparse Core + Low-Rank Correction). Registered in COMPRESSOR_REGISTRY as 'armor_pruner', 'armor', and 'armor_compressor'. Mathematical Formulation: 1. Resilient First-Order Taylor Saliency (handling zero-grad, NaN/Inf, and absent activations): $$S_{ij} = \\\\begin{cases} |W_{ij} \\\\odot", | |
| "math": "W = W_{\\text{sparse}} + R", | |
| "markdown_preview": "# ARMOR Pruner: Adaptive Resilient Matrix Orthogonal Rank Pruning (2:4 Sparse Core + Low-Rank Correction)\n\n## 1. Executive Abstract & Theoretical Motivation\n\nEnforcing hardware-supported semi-structured N:M (such as 2:4) sparsity on modern NVIDIA GPUs (Ampere, Hopper, Blackwell) provides immediate $2\\times$ Tensor Core throughput improvements. However, strict 2:4 hard pruning discards essential parameter magnitude and spectral information, often causing non-negligible perplexity degradation. **ARMOR Pruner** (Adaptive Resilient Matrix Orthogonal Rank Pruning) resolves this quality drop through" | |
| }, | |
| { | |
| "id": "ast_pruner", | |
| "name": "AST: Adaptive Semi-Structured 2:4 and N:M Pruning with Empirical ", | |
| "file": "Codes/framework/compression_engine/ast_pruner.py", | |
| "type": "Compression Engine", | |
| "category": "Structured Pruning & Channel Slicing", | |
| "paradigm": "pruning", | |
| "icon": "\u2702\ufe0f", | |
| "description": "AST: Adaptive Semi-Structured 2:4 and N:M Pruning with Empirical Fisher Calibration and PCA Coordinate Rotation. Registered in COMPRESSOR_REGISTRY as 'ast_pruner' and 'ast'. Mathematical Formulation: 1. Activation Covariance & PCA Coordinate Rotation: $$\\\\Sigma_X = \\\\frac{1}{N - 1} (X - \\\\mu)^\\\\top (X - \\\\mu) = Q \\\\Lambda Q^\\\\top, \\\\quad Q^\\\\to", | |
| "math": "\\Sigma_X = Q \\Lambda Q^\\top, \\quad Q^\\top Q = I_{d_{\\text{in}}}", | |
| "markdown_preview": "# AST Pruner: Adaptive Semi-Structured 2:4 Pruning with Empirical Fisher Information & PCA Coordinate Rotation\n\n## 1. Executive Abstract & Theoretical Motivation\n\nNVIDIA Ampere, Hopper, and Blackwell architectures mandate strict 2:4 semi-structured sparsity (exactly 2 non-zero values for every 4 contiguous elements) to achieve $2\\times$ Tensor Core compute acceleration. However, applying 2:4 hard thresholding directly in the standard parameter coordinate basis often discards large weights located within the same 4-element block. **AST Pruner** (Adaptive Semi-Structured Pattern Pruner) overcome" | |
| }, | |
| { | |
| "id": "astro_quantizer", | |
| "name": "AstroQuantizer: Dynamic Uniform Range Quantization Engine with St", | |
| "file": "Codes/framework/compression_engine/astro_quantizer.py", | |
| "type": "Compression Engine", | |
| "category": "Quantization & Mixed Precision", | |
| "paradigm": "quant", | |
| "icon": "\ud83d\udc8e", | |
| "description": "AstroQuantizer: Dynamic Uniform Range Quantization Engine with Straight-Through Estimator (STE) and Fisher-Adaptive Mixed-Precision Bit Allocation. Registered as 'astro' and 'astro_quantizer'. Scientific Novelty: - Mixed-precision dynamic bit allocation driven by Empirical Fisher Information / Output Activation Energy. - Optimal integer budget all", | |
| "math": "W = S \\odot (Q_W - Z), \\quad Q_W \\in \\{-8, \\dots, 7\\}, \\quad S = \\text{scale}", | |
| "markdown_preview": "# AstroQuantizer: Dynamic Uniform Range Quantization Engine with Straight-Through Estimator (STE)\n\n## 1. Executive Abstract & Theoretical Motivation\n\nUniform fixed-point integer quantization provides the most hardware-efficient acceleration path for modern deep neural networks. **AstroQuantizer** implements an affine asymmetric and symmetric uniform dynamic range quantization engine that compresses continuous floating-point parameters (weights, biases, and activation caches) into discrete $b$-bit integer representations with Straight-Through Estimator (STE) gradient backpropagation for Quantiz" | |
| }, | |
| { | |
| "id": "bonsai_pruner", | |
| "name": "Bonsai: Iterative Magnitude and Taylor Pruning (IMP) with Gradien", | |
| "file": "Codes/framework/compression_engine/bonsai_pruner.py", | |
| "type": "Compression Engine", | |
| "category": "Mixture of Experts & Routing", | |
| "paradigm": "moe", | |
| "icon": "\ud83d\udd00", | |
| "description": "Bonsai: Iterative Magnitude and Taylor Pruning (IMP) with Gradient Descent Recovery & [CLS] Routing Protection. Registered in COMPRESSOR_REGISTRY as 'bonsai' and 'bonsai_pruner'. Production-hardened with: - Multi-architecture support: DistilBERT, RoBERTa, GPT-2 (Conv1D), Qwen3.5 (GQA) - Robust activation collection & calibration forward handling (", | |
| "math": "s_t = s_{\\text{target}} \\cdot \\left( 1 - \\left( 1 - \\frac{t}{T} \\right)^3 \\right)", | |
| "markdown_preview": "# Bonsai Pruner: Progressive Iterative Pruning with Cubic Scheduling, GD Recovery & Router Protection\n\n## 1. Executive Abstract & Theoretical Motivation\n\nSingle-shot hard parameter pruning introduces sudden massive perturbations into deep neural network representations, causing irreversible degradation in attention entropy and downstream task accuracy. **Bonsai Pruner** implements progressive multi-step **Iterative Magnitude & Taylor Pruning (IMP)** driven by a **Cubic Polynomial Sparsity Schedule**, **Inter-Step Gradient Descent Output Reconstruction Recovery**, and **Strict [CLS] / Router Pr" | |
| }, | |
| { | |
| "id": "cachenotes_compressor", | |
| "name": "CacheNotes Compressor: Task-Aware CPT Guidance for Static SVD & D", | |
| "file": "Codes/framework/compression_engine/cachenotes_compressor.py", | |
| "type": "Compression Engine", | |
| "category": "Low-Rank SVD & Factorization", | |
| "paradigm": "svd", | |
| "icon": "\u26a1", | |
| "description": "CacheNotes: Task-Aware Key-Value Cache Compression and Static Weight Compression with Learnable Continuous CPT (Compression-Planning-Token) Generator. Given a task description, CacheNotes generates Compression-Planning-Tokens (CPTs) that guide the compression of the corpus and KV cache into compact, reusable states. Scientific Enhancement: - Repl", | |
| "math": "W_{\\text{cpt}} = \\operatorname{diag}(I_{\\text{task}}) W \\in \\mathbb{R}^{d_{\\text{out}} \\times d_{\\text{in}}}", | |
| "markdown_preview": "# CacheNotes Compressor: Task-Aware CPT Guidance for Static SVD & Dynamic KV Cache Compression\n\n## 1. Executive Abstract & Theoretical Motivation\n\nUniform model compression treats all downstream reasoning tasks identically, ignoring task-specific semantic pathways. **CacheNotes Compressor** introduces task-aware key-value cache and parameter compression guided by **Compression-Planning-Tokens (CPTs)** (Corallo et al., EACL 2026).\n\nGiven a target downstream task (e.g., `'qa'`, `'summarize'`, `'code'`, `'reasoning'`, `'math'`), CacheNotes generates task-specific feature importance vectors $I_{\\t" | |
| }, | |
| { | |
| "id": "come", | |
| "name": "CoMe: Compress LLMs through Layer Concatenation and Shedding with", | |
| "file": "Codes/framework/compression_engine/come.py", | |
| "type": "Compression Engine", | |
| "category": "Structured Pruning & Channel Slicing", | |
| "paradigm": "pruning", | |
| "icon": "\u2702\ufe0f", | |
| "description": "CoMe: Compress LLMs through Layer Concatenation and Shedding with Learnable Inter-Layer Interpolation Adapters. Registered in COMPRESSOR_REGISTRY as 'come' and 'come_compressor'. Mathematical Formulation: 1. Layer Shedding & Uniform Strided Selection: $$\\\\mathcal{I}_{\\\\text{keep}} = \\\\{i_0, i_1, \\\\dots, i_{K-1}\\\\}, \\\\quad i_k = \\\\left\\\\lfloor k", | |
| "math": "M_{\\text{compressed}} = X W_{\\text{bridge}}, \\quad W_{\\text{bridge}} = U_r \\Sigma_r V_r^\\top", | |
| "markdown_preview": "# CoMe: Context-Aware Low-Rank Memory Bridge Compression\n\n## 1. Executive Abstract & Architectural Motivation\n**CoMe (Context-Aware Low-Rank Memory Bridge)** compresses transformer memory consumption during extended multi-turn context processing. By deploying a low-rank memory bridge across persistent KV representations and intermediate projection spaces, CoMe achieves high compression ratios while preventing forgotten context artifacts.\n\n---\n\n## 2. Mathematical Formulations & Derivations\nCoMe decomposes key-value cache projections and dense feedforward representations through context-aware bo" | |
| }, | |
| { | |
| "id": "compot_compressor", | |
| "name": "COMPOT: Compressed Orthogonal Dictionary Learning Compressor.", | |
| "file": "Codes/framework/compression_engine/compot_compressor.py", | |
| "type": "Compression Engine", | |
| "category": "Low-Rank SVD & Factorization", | |
| "paradigm": "svd", | |
| "icon": "\u26a1", | |
| "description": "COMPOT: Compressed Orthogonal Dictionary Learning Compressor. Registered in COMPRESSOR_REGISTRY as 'compot'. Production hardened for multi-architecture compatibility (DistilBERT, RoBERTa, GPT-2, Qwen3.5) with reconstruction validation, Conv1D support, rank reduction bounds, and safe fallback.", | |
| "math": "W \\approx U_r \\Sigma_r V_r^\\top = A B^\\top, \\quad r \\ll \\min(d_{in}, d_{out})", | |
| "markdown_preview": "# COMPOT: Compressed Orthogonal Dictionary Learning\n\n**Registered as `compot`** \n*Data\u2011aware low\u2011rank compression using orthogonal dictionary learning from calibration activations.*\n\n---\n\n## Overview\n\nCOMPOT (Compressed Orthogonal Dictionary Learning) is a post\u2011training compression method that exploits the statistical structure of a model\u2019s intermediate activations to build a compact, low\u2011rank approximation of each linear layer\u2019s weight matrix. Unlike purely weight\u2011based SVD methods, COMPOT first captures the subspace spanned by real input activations (via a calibration set), learns an orthog" | |
| }, | |
| { | |
| "id": "coopq_quantizer", | |
| "name": "CoopQ: Cooperative Game-Theoretic Bit Allocation Quantizer with S", | |
| "file": "Codes/framework/compression_engine/coopq_quantizer.py", | |
| "type": "Compression Engine", | |
| "category": "Quantization & Mixed Precision", | |
| "paradigm": "quant", | |
| "icon": "\ud83d\udc8e", | |
| "description": "CoopQ: Cooperative Game-Theoretic Bit Allocation Quantizer with STE and Per-Channel Asymmetric Calibration. Registered in COMPRESSOR_REGISTRY as 'coopq'. Mathematical Formulation: 1. Min Bit-Width Floor Enforcement: $$b_l \\\\ge b_{\\\\min} = 4, \\\\quad \\\\forall l \\\\in \\\\{1, \\\\dots, L\\\\}$$ 2. Per-Channel Asymmetric Scale & Zero-Point Calibration: ", | |
| "math": "b_l \\\\ge b_{\\\\min} = 4, \\\\quad \\\\forall l \\\\in \\\\{1, \\\\dots, L\\\\}", | |
| "markdown_preview": "# CoopQ: Cooperative Game-Theoretic Bit Allocation Quantizer\n\n## 1. Executive Abstract & Theoretical Motivation\n\nUniform bit-width allocation across transformer layers is sub-optimal: initial attention layers and late MLP down-projections exhibit vastly higher Hessian eigenvalue spectrums and representational sensitivities than intermediate layers.\n\n**CoopQ (Cooperative Quantization)** formulates mixed-precision bit allocation as a **cooperative game-theoretic optimization problem** under a global bit-budget constraint:\n1. **Sensitivity-Proportional Bit Allocation**: Measures layer sensitivity" | |
| }, | |
| { | |
| "id": "cr_compressor", | |
| "name": "CR Compressor: Interpolative Decomposition for Weight Compression", | |
| "file": "Codes/framework/compression_engine/cr_compressor.py", | |
| "type": "Compression Engine", | |
| "category": "Dynamic & Efficient Attention", | |
| "paradigm": "sparse", | |
| "icon": "\ud83c\udf1f", | |
| "description": "CR (Column-Row / Interpolative Decomposition) Compressor. Decomposes W \u2248 C @ P, where C are actual columns of W. Registered in COMPRESSOR_REGISTRY as 'cr' and 'cr_compressor'.", | |
| "math": "A_{ij} = \\text{TopK}\\left(\\text{Softmax}\\left(\\frac{Q_i K_j^\\top}{\\sqrt{d}}\\right)\\right)", | |
| "markdown_preview": "# CR Compressor: Interpolative Decomposition for Weight Compression\n\n**Registered as `cr`** \n*Low\u2011rank approximation via column selection and projection, preserving interpretability.*\n\n---\n\n## Overview\n\nThe CR Compressor applies **interpolative decomposition (ID)** to the weight matrix of each linear layer. Unlike standard SVD, which produces abstract singular vectors, ID expresses the low\u2011rank approximation using a subset of the **actual columns** of the original weight matrix. Specifically, the weight **W** \u2208 \u211d<sup>m\u00d7n</sup> is approximated as:\n\n```\nW \u2248 C \u00b7 P\n```\n\nwhere:\n**C** = **W[:, idx]" | |
| }, | |
| { | |
| "id": "cur_compressor", | |
| "name": "CUR Compressor: replaces Linear and Conv1D layers with CUR decomp", | |
| "file": "Codes/framework/compression_engine/cur_compressor.py", | |
| "type": "Compression Engine", | |
| "category": "Dynamic & Efficient Attention", | |
| "paradigm": "sparse", | |
| "icon": "\ud83c\udf1f", | |
| "description": "CUR Compressor: replaces Linear and Conv1D layers with CUR decomposition. Uses leverage score sampling or energy sampling for column/row selection. Registered in COMPRESSOR_REGISTRY as 'cur' and 'cur_compressor'.", | |
| "math": "A_{ij} = \\text{TopK}\\left(\\text{Softmax}\\left(\\frac{Q_i K_j^\\top}{\\sqrt{d}}\\right)\\right)", | |
| "markdown_preview": "# CURCompressor: Leverage Score Sampled Column-Row CUR Decomposition Engine\n\n## 1. Executive Abstract & Theoretical Motivation\nStandard Singular Value Decomposition (SVD) represents weight matrices as linear combinations of abstract, dense singular vectors ($U$ and $V$). While mathematically optimal under Frobenius norm bounds, SVD components lose physical interpretability because synthetic singular vectors do not correspond to actual channels or features of the original weight matrix. Furthermore, dense SVD factors require full matrix multiplications without structural reuse.\n\n`CURCompressor`" | |
| }, | |
| { | |
| "id": "cur_linear", | |
| "name": "CUR Linear Layer: low-rank cross-approximation using column and r", | |
| "file": "Codes/framework/compression_engine/cur_linear.py", | |
| "type": "Compression Engine", | |
| "category": "Low-Rank SVD & Factorization", | |
| "paradigm": "svd", | |
| "icon": "\u26a1", | |
| "description": "CUR Linear Layer: low-rank cross-approximation using column and row skeleton selection. W \u2248 C @ U @ R", | |
| "math": "W \\approx U_r \\Sigma_r V_r^\\top = A B^\\top, \\quad r \\ll \\min(d_{in}, d_{out})", | |
| "markdown_preview": "# CURLinear: Drop-In Leverage-Sampled Subspace Replacement Layer for nn.Linear\n\n## 1. Executive Abstract & Theoretical Motivation\nThe `CURLinear` module provides a drop-in low-rank matrix operator replacing standard dense PyTorch linear layers (`nn.Linear`). Rather than storing a monolithic dense weight parameter $W \\in \\mathbb{R}^{d_{\\text{out}} \\times d_{\\text{in}}}$, `CURLinear` represents the linear transformation through three factorized sub-matrices ($C, U, R$) obtained via CUR Matrix Decomposition.\n\nBy selecting actual physical columns ($C \\in \\mathbb{R}^{d_{\\text{out}} \\times r}$) and " | |
| }, | |
| { | |
| "id": "cur_moe", | |
| "name": "CUR-MoE Compressor alias module.", | |
| "file": "Codes/framework/compression_engine/cur_moe.py", | |
| "type": "Compression Engine", | |
| "category": "Mixture of Experts & Routing", | |
| "paradigm": "moe", | |
| "icon": "\ud83d\udd00", | |
| "description": "CUR-MoE Compressor alias module. Re-exports CURMoECompressor from cur_moe_compressor.", | |
| "math": "y = \\sum_{i=1}^K G(x)_i \\cdot E_i(x), \\quad G(x) = \\text{Softmax}(\\text{TopK}(x W_g, K=2))", | |
| "markdown_preview": "# Cur Moe\n\n## Overview\n## Configuration & Hyperparameters\n\n| Parameter | Description |\n|-----------|-------------|\n\n## Usage Examples\n\n```python\nfrom Codes.framework.compression_engine import Module\n```\n\n## Benchmark Results\n\n| Status | Accuracy | Val Loss | Peak VRAM | Inference Latency |\n|--------|----------|----------|-----------|-------------------|\n| Pending| Pending | Pending | Pending | Pending |\n\n## Limitations & Future Work\n\nTODO: Identify specific limitations and edge cases for this module.\n\n## Empirical Benchmark Results\n\n| Model | Dataset | Status | Baseline Acc | Com" | |
| }, | |
| { | |
| "id": "cur_moe_compressor", | |
| "name": "CUR-MoE Compressor: apply CUR cross-approximation to Mixture of E", | |
| "file": "Codes/framework/compression_engine/cur_moe_compressor.py", | |
| "type": "Compression Engine", | |
| "category": "Mixture of Experts & Routing", | |
| "paradigm": "moe", | |
| "icon": "\ud83d\udd00", | |
| "description": "CUR-MoE Compressor: apply CUR cross-approximation to Mixture of Experts (MoE) modules and expert layers. Registered in COMPRESSOR_REGISTRY as 'cur_moe' and 'cur_moe_compressor'.", | |
| "math": "W \\approx C \\cdot U \\cdot R, \\quad C = W_{[:, J]}, \\; R = W_{[I, :]}, \\; U = C^\\dagger W R^\\dagger", | |
| "markdown_preview": "# CURMoECompressor: Sparse Expert Layer CUR Decomposition Engine for Mixture-of-Experts\n\n## 1. Executive Abstract & Theoretical Motivation\nMixture-of-Experts (MoE) architectures achieve high parameter capacity by instantiating dozens of sparse parallel expert networks (e.g., 8 to 64 experts per layer), dynamically routing tokens via top-$k$ gating mechanisms. However, hosting hundreds of expert feed-forward matrices ($W_e \\in \\mathbb{R}^{d_{\\text{ff}} \\times d_{\\text{model}}}$) inflates High Bandwidth Memory (HBM) footprints, creating massive VRAM bottlenecks during inference.\n\n`CURMoECompress" | |
| }, | |
| { | |
| "id": "cure", | |
| "name": "CURE: CUR Expert Compression Engine with Randomized Subspace Appr", | |
| "file": "Codes/framework/compression_engine/cure.py", | |
| "type": "Compression Engine", | |
| "category": "Low-Rank SVD & Factorization", | |
| "paradigm": "svd", | |
| "icon": "\u26a1", | |
| "description": "CURE: CUR-MoE Compressor alias and factory. Compresses Mixture of Experts (MoE) layers and dense FFN layers using proper factorized CURLinear (C @ U @ R parameter reduction). Registered in COMPRESSOR_REGISTRY as 'cure' and 'cure_compressor'.", | |
| "math": "W \\approx U_r \\Sigma_r V_r^\\top = A B^\\top, \\quad r \\ll \\min(d_{in}, d_{out})", | |
| "markdown_preview": "# CURE: CUR Expert Compression Engine with Randomized Subspace Approximation\n\n## 1. Executive Abstract & Theoretical Motivation\nMixture-of-Experts (MoE) architectures achieve high capacity by scaling expert counts, but post-training factorizations (such as SVD or leverage-score sampling) introduce overhead when calculating singular values or matrix norms over dozens of large expert matrices. In real-time post-training compression pipelines, compute latency during decomposition can become a major bottleneck.\n\n`CURE` (CUR Expert Compression Engine) addresses this by introducing a randomized subs" | |
| }, | |
| { | |
| "id": "delta_kv", | |
| "name": "DeltaKV: Residual-Based Key-Value Cache Compression via Long-Rang", | |
| "file": "Codes/framework/compression_engine/delta_kv.py", | |
| "type": "Compression Engine", | |
| "category": "Latent & KV-Cache Compression", | |
| "paradigm": "mla", | |
| "icon": "\ud83e\udde0", | |
| "description": "DeltaKV: Residual-Based KV Cache Compression via Long-Range Similarity. Encodes semantic residuals relative to retrieved historical references, preserving attention fidelity while drastically compressing dynamic KV caches. Paper: \"DeltaKV: Residual-Based KV Cache Compression via Long-Range Similarity\" Hao et al., arXiv:2602.08005. Register", | |
| "math": "R_K \\in \\mathbb{R}^{B \\times H \\times 1 \\times M \\times D}, \\quad R_V \\in \\mathbb{R}^{B \\times H \\times 1 \\times M \\times D}", | |
| "markdown_preview": "# DeltaKV: Residual-Based Key-Value Cache Compression via Long-Range Reference Matching\n\n## 1. Executive Abstract & Theoretical Motivation\n\nIn long-context LLM inference, the Key-Value (KV) cache grows linearly with sequence length, creating severe VRAM bottlenecks. Traditional KV cache eviction algorithms discard historical tokens, harming long-range context retrieval. **DeltaKV** preserves context fidelity by storing residual representations relative to retrieved historical reference vectors.\n\nDeltaKV identifies long-range similarity patterns across sequence positions. Instead of storing ful" | |
| }, | |
| { | |
| "id": "direct_svd", | |
| "name": "Direct SVD Compressor: Truncated SVD low-rank factorized layer re", | |
| "file": "Codes/framework/compression_engine/direct_svd.py", | |
| "type": "Compression Engine", | |
| "category": "Low-Rank SVD & Factorization", | |
| "paradigm": "svd", | |
| "icon": "\u26a1", | |
| "description": "Direct SVD Compressor: Truncated SVD low-rank factorized layer replacement. Registered as 'direct_svd'. Production hardened for multi-architecture compatibility (DistilBERT, RoBERTa, GPT-2, Qwen3.5) with error validation, Conv1D support, rank bounds, and safe fallback.", | |
| "math": "\\min_{\\operatorname{rank}(\\hat{W}) \\le r} \\|W - \\hat{W}\\|_F = \\sqrt{\\sum_{i=r+1}^{\\min(d_{\\text{in}}, d_{\\text{out}})} \\sigma_i^2}", | |
| "markdown_preview": "# DirectSVDCompressor: Closed-Form Truncated SVD Low-Rank Factorization\n\n## 1. Executive Abstract & Theoretical Motivation\nDense linear projection layers in transformer architectures ($W \\in \\mathbb{R}^{d_{\\text{out}} \\times d_{\\text{in}}}$) account for a significant portion of memory consumption and floating-point operations (FLOPs). Under the Eckart-Young-Mirsky Theorem, truncated Singular Value Decomposition (SVD) provides the optimal low-rank matrix approximation under both Frobenius and spectral matrix norms:\n\n$$\\min_{\\operatorname{rank}(\\hat{W}) \\le r} \\|W - \\hat{W}\\|_F = \\sqrt{\\sum_{i=r" | |
| }, | |
| { | |
| "id": "dobi", | |
| "name": "DobiSVDLinearLayer & Compressor: Differentiable SVD with Learnabl", | |
| "file": "Codes/framework/compression_engine/dobi.py", | |
| "type": "Compression Engine", | |
| "category": "Low-Rank SVD & Factorization", | |
| "paradigm": "svd", | |
| "icon": "\u26a1", | |
| "description": "Dobi-SVD alias module. Re-exports DobiSVDCompressor, DobiSVDLinearLayer, dobi_factory, and injection hooks.", | |
| "math": "W \\approx U_r \\Sigma_r V_r^\\top = A B^\\top, \\quad r \\ll \\min(d_{in}, d_{out})", | |
| "markdown_preview": "# DobiSVDLinearLayer & Compressor: Differentiable SVD with Learnable Rank Ratios\n\n## 1. Executive Abstract & Theoretical Motivation\nConventional SVD compression frameworks rely on static, manually heuristic rank selection or offline post-processing heuristics. Selecting sub-optimal fixed ranks per layer introduces structural bottlenecks, while hyperparameter grid searches across deep architectures require prohibitive compute.\n\n`DobiSVDCompressor` introduces **Dobi-SVD** (Differentiable Optimal Bound Integration SVD), an inline differentiable low-rank compression engine. Dobi-SVD parameterizes " | |
| }, | |
| { | |
| "id": "dobi_svd", | |
| "name": "DobiSVDLinearLayer & Compressor: Differentiable SVD with Learnabl", | |
| "file": "Codes/framework/compression_engine/dobi_svd.py", | |
| "type": "Compression Engine", | |
| "category": "Low-Rank SVD & Factorization", | |
| "paradigm": "svd", | |
| "icon": "\u26a1", | |
| "description": "Dobi-SVD: Differentiable SVD layer with learnable spectral gating. Registered as 'dobi' and 'dobi_svd'. Production hardened for multi-architecture compatibility (DistilBERT, RoBERTa, GPT-2, Qwen3.5) with error validation, Conv1D support, and safe fallback.", | |
| "math": "W \\approx U_r \\Sigma_r V_r^\\top = A B^\\top, \\quad r \\ll \\min(d_{in}, d_{out})", | |
| "markdown_preview": "# DobiSVDLinearLayer & Compressor: Differentiable SVD with Learnable Rank Ratios\n\n## 1. Executive Abstract & Theoretical Motivation\nConventional SVD compression frameworks rely on static, manually heuristic rank selection or offline post-processing heuristics. Selecting sub-optimal fixed ranks per layer introduces structural bottlenecks, while hyperparameter grid searches across deep architectures require prohibitive compute.\n\n`DobiSVDCompressor` introduces **Dobi-SVD** (Differentiable Optimal Bound Integration SVD), an inline differentiable low-rank compression engine. Dobi-SVD parameterizes " | |
| }, | |
| { | |
| "id": "dynamic_entropy", | |
| "name": "Dynamic Entropy (`dynamic_entropy`)", | |
| "file": "Codes/framework/compression_engine/dynamic_entropy.py", | |
| "type": "Compression Engine", | |
| "category": "Low-Rank SVD & Factorization", | |
| "paradigm": "svd", | |
| "icon": "\u26a1", | |
| "description": "Dynamic Entropy SVD alias module. Re-exports DynamicEntropySVDCompressor, dynamic_entropy_svd_factory, and injection hooks from dynamic_rank.", | |
| "math": "\\widetilde{W} = W C_X^{1/2} = U \\Sigma V^\\top", | |
| "markdown_preview": "# Dynamic Entropy (`dynamic_entropy`)\n\n[](../../catalog.json)\n[](../../../Codes/framework/core/registry.py)\n[](../../../Codes/framework/eval_all_models_datasets.py)\n\n---\n\n## 1. Architectural Overview & Value Proposition\n\nThe `dynamic_entropy` engine provides post-training model compression through advanced matrix factorization, activation whitening, structured " | |
| }, | |
| { | |
| "id": "dynamic_rank", | |
| "name": "Dynamic Entropy Singular Value Decomposition (Dynamic Entropy SVD", | |
| "file": "Codes/framework/compression_engine/dynamic_rank.py", | |
| "type": "Compression Engine", | |
| "category": "Low-Rank SVD & Factorization", | |
| "paradigm": "svd", | |
| "icon": "\u26a1", | |
| "description": "Dynamic Entropy Singular Value Decomposition (Dynamic Entropy SVD) Compressor. Implements singular entropy computation over empirical layer activation covariances: $$H(\\\\Sigma_l) = -\\\\sum_{i=1}^{d} p_i \\\\ln(p_i), \\\\quad p_i = \\\\frac{\\\\sigma_i}{\\\\sum_j \\\\sigma_j}$$ Dynamically budgets layer ranks $r_l \\\\in [r_{\\\\min}, r_{\\\\max}]$ proportional to si", | |
| "math": "H(\\Sigma_l) = -\\sum_{i=1}^d p_i \\ln(p_i), \\quad p_i = \\frac{\\sigma_i}{\\sum_{j=1}^d \\sigma_j}", | |
| "markdown_preview": "# DynamicEntropySVDCompressor: Singular Entropy-Budgeted Rank Allocation\n\n## 1. Executive Abstract & Theoretical Motivation\nUniform low-rank factorization assigns identical rank $r$ across all network layers, ignoring that different layers possess starkly different representation capacities and spectral decay rates. Early layers typically capture broad syntactic patterns with heavy-tailed singular spectra, whereas deeper layers specialize in narrow task representations with fast spectral decay.\n\n`DynamicEntropySVDCompressor` (registered as `dynamic_entropy_svd` and `entropy_rank_allocator`) dy" | |
| }, | |
| { | |
| "id": "enlighten_compressor", | |
| "name": "ENLighten: Low-Rank SVD + 2:4 Structured Sparsity (Photonic-Align", | |
| "file": "Codes/framework/compression_engine/enlighten_compressor.py", | |
| "type": "Compression Engine", | |
| "category": "Low-Rank SVD & Factorization", | |
| "paradigm": "svd", | |
| "icon": "\u26a1", | |
| "description": "ENLighten: Low-Rank SVD + 2:4 Structured Sparsity (Photonic-Aligned). Registered as 'enlighten'. Production hardened for multi-architecture compatibility (DistilBERT, RoBERTa, GPT-2, Qwen3.5) with vectorization, error validation, and safe fallback.", | |
| "math": "W \\approx U_r \\Sigma_r V_r^\\top = A B^\\top, \\quad r \\ll \\min(d_{in}, d_{out})", | |
| "markdown_preview": "# ENLighten: Low\u2011Rank SVD + 2:4 Structured Sparsity\n\n**Registered as `enlighten`** \n*Combines a low\u2011rank SVD backbone with a structured 2:4 sparse residual, targeting photonic accelerator alignment.*\n\n---\n\n## Overview\n\nENLighten is a post\u2011training weight compression technique that decomposes the weight matrix **W** of each linear layer into two complementary components:\n\n1. **Low\u2011rank approximation L**: obtained via truncated SVD, capturing the principal singular vectors.\n2. **Structured sparse residual**: the difference **R = W \u2013 L** is then pruned to enforce a **2:4 structured sparsity** pa" | |
| }, | |
| { | |
| "id": "entropy_budget_allocation", | |
| "name": "EntropyBudgetAllocation: global rank assignment via whitened acti", | |
| "file": "Codes/framework/compression_engine/entropy_budget_allocation.py", | |
| "type": "Compression Engine", | |
| "category": "Low-Rank SVD & Factorization", | |
| "paradigm": "svd", | |
| "icon": "\u26a1", | |
| "description": "EntropyBudgetAllocation: global rank assignment via whitened activation entropy. Registered as 'entropy_budget' and 'entropy_budget_allocation'.", | |
| "math": "W \\approx U_r \\Sigma_r V_r^\\top = A B^\\top, \\quad r \\ll \\min(d_{in}, d_{out})", | |
| "markdown_preview": "# EntropyBudgetAllocation: Global Rank Assignment via Whitened Activation Entropy\n\n**Registered as `entropy_budget`** \n*Data\u2011driven rank allocation across all layers based on a global budget, using activation covariance and entropy to prioritize components.*\n\n---\n\n## Overview\n\nEntropyBudgetAllocation is a post\u2011training compression method that distributes a **global parameter budget** (fraction of singular components to retain) across all linear layers in a model. Unlike methods that assign a fixed rank per layer, this algorithm first collects **activation statistics** from a calibration datas" | |
| }, | |
| { | |
| "id": "erc_svd", | |
| "name": "ERCCompressor: Error-Controlled Singular Value Energy Thresholdin", | |
| "file": "Codes/framework/compression_engine/erc_svd.py", | |
| "type": "Compression Engine", | |
| "category": "Low-Rank SVD & Factorization", | |
| "paradigm": "svd", | |
| "icon": "\u26a1", | |
| "description": "ERC-SVD: Error-Controlled SVD with Bounded Frobenius Truncation Error. Registered as 'erc' and 'erc_svd'. Production hardened for multi-architecture compatibility (DistilBERT, RoBERTa, GPT-2, Qwen3.5) with error validation, Conv1D support, parameter bounds, and safe fallback.", | |
| "math": "W \\approx U_r \\Sigma_r V_r^\\top = A B^\\top, \\quad r \\ll \\min(d_{in}, d_{out})", | |
| "markdown_preview": "# ERCCompressor: Error-Controlled Singular Value Energy Thresholding Engine\n\n## 1. Executive Abstract & Theoretical Motivation\nArbitrary rank specification ($r = 64$ or $r = 128$) across non-homogeneous network layers creates erratic error profiles. Some layers possess rapidly decaying singular values requiring only a small rank $r$ to preserve 99% of energy, while others possess heavy spectral tails where rigid truncation drops essential signals.\n\n`ERCCompressor` (Error-Controlled SVD) provides a mathematically guaranteed, threshold-driven compression paradigm. Instead of enforcing fixed inte" | |
| }, | |
| { | |
| "id": "faq_quantizer", | |
| "name": "FAQ (Feature-Activation-Aware Quantization): adjusts quantization", | |
| "file": "Codes/framework/compression_engine/faq_quantizer.py", | |
| "type": "Compression Engine", | |
| "category": "Quantization & Mixed Precision", | |
| "paradigm": "quant", | |
| "icon": "\ud83d\udc8e", | |
| "description": "FAQ (Feature-Activation-Aware Quantization): adjusts quantization scales using activation statistics. Registered as 'faq'.", | |
| "math": "W = S \\odot (Q_W - Z), \\quad Q_W \\in \\{-8, \\dots, 7\\}, \\quad S = \\text{scale}", | |
| "markdown_preview": "# FAQ: Feature-Activation-Aware Quantization Engine\n\n## 1. Executive Abstract & Theoretical Motivation\n\nUniform weight quantization treats all input channels equally, ignoring the fact that neural network activations exhibit extreme channel-wise variance and outlier magnitudes. In large language models, outlier activation channels with magnitudes up to $100\\times$ larger than normal channels dominate the output error.\n\n**FAQ (Feature-Activation-Aware Quantization)** solves this by collecting empirical activation statistics across calibration sequences and scaling channel-wise quantization step" | |
| }, | |
| { | |
| "id": "flap", | |
| "name": "FLAP: Fluctuation-Based Adaptive Structured Pruning.", | |
| "file": "Codes/framework/compression_engine/flap.py", | |
| "type": "Compression Engine", | |
| "category": "Structured Pruning & Channel Slicing", | |
| "paradigm": "pruning", | |
| "icon": "\u2702\ufe0f", | |
| "description": "FLAP: Fluctuation-Based Adaptive Structured Pruning. Registered in COMPRESSOR_REGISTRY as 'flap' and 'flap_compressor'. Production hardened for multi-architecture compatibility (DistilBERT, RoBERTa, GPT-2, Qwen3.5) with reconstruction validation, Conv1D support, and safe fallback.", | |
| "math": "I(i) = \\frac{1}{N} \\sum_{t=1}^N |X_{t, i} - \\bar{X}_i| \\cdot \\|W_{:, i}\\|_2", | |
| "markdown_preview": "# FLAP: Fluctuation-based Adaptive Structured Pruning\n\n## 1. Executive Abstract & Architectural Motivation\n**FLAP (Fluctuation-based Adaptive Structured Pruning)** removes redundant channels and attention heads by measuring feature fluctuation dynamics across layers. Unlike magnitude-based pruning which ignores activation dynamics, FLAP ranks importance using the variance and fluctuation magnitude of intermediate representations without requiring expensive retraining.\n\n---\n\n## 2. Mathematical Formulations & Derivations\nThe feature fluctuation metric for channel $i$ across sequence tokens is de" | |
| }, | |
| { | |
| "id": "flat_llm", | |
| "name": "FLAT-LLM: Activation-Aware Structured Dimension & Neuron Pruning ", | |
| "file": "Codes/framework/compression_engine/flat_llm.py", | |
| "type": "Compression Engine", | |
| "category": "Structured Pruning & Channel Slicing", | |
| "paradigm": "pruning", | |
| "icon": "\u2702\ufe0f", | |
| "description": "FLAT-LLM: Activation-Aware Structured Dimension & Neuron Pruning via Eigen-Decomposition. Registered in COMPRESSOR_REGISTRY as 'flat_llm' and 'flat_llm_compressor'. Production hardened for multi-architecture compatibility (DistilBERT, RoBERTa, GPT-2, Qwen3.5) with reconstruction validation, Conv1D support, and safe fallback.", | |
| "math": "\\min_{W_r} \\text{Tr}((W - W_r) R_{XX} (W - W_r)^\\top)", | |
| "markdown_preview": "# FLAT-LLM: Feature-Level Adaptive Truncation Compression\n\n## 1. Executive Abstract & Architectural Motivation\n**FLAT-LLM (Feature-Level Adaptive Truncation for Large Language Models)** compresses dense feedforward and attention projections by analyzing eigen-decomposition spectra of input/output feature correlations. It adaptively allocates rank budgets across layers based on spectral decay curvature.\n\n---\n\n## 2. Mathematical Formulations & Derivations\nGiven feature correlation matrix $R_{XX} = \\mathbb{E}[X^\\top X]$, FLAT-LLM computes the truncated projection matrix by solving:\n$$\\min_{W_r} \\" | |
| }, | |
| { | |
| "id": "frandi", | |
| "name": "FRanDI Compressor alias module.", | |
| "file": "Codes/framework/compression_engine/frandi.py", | |
| "type": "Compression Engine", | |
| "category": "Dynamic & Efficient Attention", | |
| "paradigm": "sparse", | |
| "icon": "\ud83c\udf1f", | |
| "description": "FRanDI Compressor alias module. Re-exports FRanDICompressor from frandi_compressor.", | |
| "math": "A_{ij} = \\text{TopK}\\left(\\text{Softmax}\\left(\\frac{Q_i K_j^\\top}{\\sqrt{d}}\\right)\\right)", | |
| "markdown_preview": "\n\n## Empirical Benchmark Results\n\n| Model | Dataset | Status | Baseline Acc | Compressed Acc | \u0394 Acc | F1 Score | Comp Rate | Compressed Layers | Checkpoint | Date |\n| :--- | :--- | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: |\n| **distilbert-sst2** | sst2 | `PASS` | 91.06% | **90.94%** | -0.12% | 0.9129 | 1.00x | 0 | [svd-linear-attention-distilbert-sst2-frandi](https://huggingface.co/Hooshaai/svd-linear-attention-distilbert-sst2-frandi) | 2026-09-19 |\n" | |
| }, | |
| { | |
| "id": "frandi_compressor", | |
| "name": "FRanDI: Feature-Regression Aligned Singular Value Decomposition C", | |
| "file": "Codes/framework/compression_engine/frandi_compressor.py", | |
| "type": "Compression Engine", | |
| "category": "Dynamic & Efficient Attention", | |
| "paradigm": "sparse", | |
| "icon": "\ud83c\udf1f", | |
| "description": "FRanDI: Feature-Regression Aligned Singular Value Decomposition Compressor. Registered in COMPRESSOR_REGISTRY as 'frandi' and 'frandi_compressor'.", | |
| "math": "\\Omega \\sim \\mathcal{N}(0, I), \\; Y = (W W^\\top)^q W \\Omega, \\; Q = \\text{qr}(Y), \\; W \\approx Q (Q^\\top W)", | |
| "markdown_preview": "# FRanDI: Feature\u2011Regression Aligned Low\u2011Rank Compression\n\n**Registered as `frandi`** \n*Conceptually aligns low\u2011rank approximation with feature regression targets; current implementation performs standard truncated SVD after a placeholder forward pass.*\n\n---\n\n## Overview\n\nFRanDI (Feature\u2011Regression aligned low\u2011rank compression) is a post\u2011training compression module designed to reduce the parameter count of linear layers through truncated Singular Value Decomposition (SVD). The name suggests a more sophisticated scheme where the low\u2011rank decomposition is guided by feature\u2011regression targets\u2014pe" | |
| }, | |
| { | |
| "id": "gfws_d_compressor", | |
| "name": "GFWSVD: Fisher-weighted / Covariance-weighted SVD. Registered as ", | |
| "file": "Codes/framework/compression_engine/gfws_d_compressor.py", | |
| "type": "Compression Engine", | |
| "category": "Low-Rank SVD & Factorization", | |
| "paradigm": "svd", | |
| "icon": "\u26a1", | |
| "description": "GFWSVD: Fisher-weighted / Covariance-weighted SVD. Registered as 'gfws' and 'gfws_d'.", | |
| "math": "W \\approx U_r \\Sigma_r V_r^\\top = A B^\\top, \\quad r \\ll \\min(d_{in}, d_{out})", | |
| "markdown_preview": "# GFWSVD: Fisher\u2011Weighted SVD Compression\n\n**Registered as `gfws`** \n*Uses gradient information to construct a Fisher\u2011weighted low\u2011rank approximation, prioritizing parameter directions that matter most for the loss.*\n\n---\n\n## Overview\n\nGFWSVD (Fisher\u2011Weighted SVD) is a post\u2011training compression method that goes beyond standard weight\u2011only SVD by incorporating **gradient information**. When gradients are available (e.g., after a backward pass on a small calibration set or after a few steps of fine\u2011tuning), the compressor builds an approximation of the Fisher information matrix for each linear " | |
| }, | |
| { | |
| "id": "grecodec", | |
| "name": "GreCoDeC Compressor alias module.", | |
| "file": "Codes/framework/compression_engine/grecodec.py", | |
| "type": "Compression Engine", | |
| "category": "Dynamic & Efficient Attention", | |
| "paradigm": "sparse", | |
| "icon": "\ud83c\udf1f", | |
| "description": "GreCoDeC Compressor alias module. Re-exports GreCoDeCCompressor from grecodec_compressor.", | |
| "math": "A_{ij} = \\text{TopK}\\left(\\text{Softmax}\\left(\\frac{Q_i K_j^\\top}{\\sqrt{d}}\\right)\\right)", | |
| "markdown_preview": "# Grecodec\n\n## Overview\n## Configuration & Hyperparameters\n\n| Parameter | Description |\n|-----------|-------------|\n\n## Usage Examples\n\n```python\nfrom Codes.framework.compression_engine import Module\n```\n\n## Benchmark Results\n\n| Status | Accuracy | Val Loss | Peak VRAM | Inference Latency |\n|--------|----------|----------|-----------|-------------------|\n| Pending| Pending | Pending | Pending | Pending |\n\n## Limitations & Future Work\n\nTODO: Identify specific limitations and edge cases for this module.\n\n## Empirical Benchmark Results\n\n| Model | Dataset | Status | Baseline Acc | Co" | |
| }, | |
| { | |
| "id": "grecodec_compressor", | |
| "name": "GreCoDeC: Greedy Coordinate Descent Low-Rank Compressor.", | |
| "file": "Codes/framework/compression_engine/grecodec_compressor.py", | |
| "type": "Compression Engine", | |
| "category": "Low-Rank SVD & Factorization", | |
| "paradigm": "svd", | |
| "icon": "\u26a1", | |
| "description": "GreCoDeC: Greedy Coordinate Descent Low-Rank Compressor. Registered in COMPRESSOR_REGISTRY as 'grecodec'.", | |
| "math": "\\min_{u, v} \\|W - u v^\\top\\|_F^2 \\implies u = \\frac{W v}{\\|v\\|^2}, \\; v = \\frac{W^\\top u}{\\|u\\|^2}", | |
| "markdown_preview": "# GRECODeC: Greedy Coordinate Descent Low-Rank Compression\n\n**Registered as `grecodec`** \n*Low\u2011rank compression of linear layers via truncated SVD, with optional energy\u2011based rank selection and a placeholder for activation\u2011guided alignment.*\n\n---\n\n## Overview\n\nFRanDI (Feature\u2011Regression aligned low\u2011rank compression) reduces the parameter count of linear layers by replacing each weight matrix with its optimal low\u2011rank approximation obtained through truncated Singular Value Decomposition (SVD). The rank can be either a **fixed user\u2011specified value** or automatically determined per layer by an *" | |
| }, | |
| { | |
| "id": "hashed_linear", | |
| "name": "HashedNets: Vectorized Weight Sharing via Hash Tables with Sign H", | |
| "file": "Codes/framework/compression_engine/hashed_linear.py", | |
| "type": "Compression Engine", | |
| "category": "Dynamic & Efficient Attention", | |
| "paradigm": "sparse", | |
| "icon": "\ud83c\udf1f", | |
| "description": "HashedNets: Vectorized Weight Sharing via Hash Tables with Sign Hashing and Optional LoRA Adapters. Reference: Chen et al., 'Compressing Neural Networks with the Hashing Trick' (ICML 2015). Production hardened for multi-architecture compatibility (nn.Linear, Conv1D, fused projections).", | |
| "math": "h(i, j) = \\left( i \\cdot P_1 + j \\cdot P_2 \\right) \\pmod B", | |
| "markdown_preview": "# Hashed Linear: Vectorized Hash-Based Weight Sharing Engine (HashedNets)\n\n## 1. Executive Abstract & Theoretical Motivation\n\nStoring full weight matrices $W \\in \\mathbb{R}^{D_{\\text{out}} \\times D_{\\text{in}}}$ requires $D_{\\text{out}} \\cdot D_{\\text{in}}$ parameters. **Hashed Linear** (based on the HashedNets architecture) reduces memory usage by hashing parameters into a small shared array.\n\nHashed Linear maps full weight matrix coordinates $(i, j)$ into a compact parameter table $\\mathbf{w}_{\\text{table}} \\in \\mathbb{R}^B$ ($B \\ll D_{\\text{out}} \\cdot D_{\\text{in}}$) using a deterministic " | |
| }, | |
| { | |
| "id": "hashed_net_compressor", | |
| "name": "HashedNet Compressor: replaces nn.Linear and Conv1D layers with H", | |
| "file": "Codes/framework/compression_engine/hashed_net_compressor.py", | |
| "type": "Compression Engine", | |
| "category": "Dynamic & Efficient Attention", | |
| "paradigm": "sparse", | |
| "icon": "\ud83c\udf1f", | |
| "description": "HashedNet Compressor: replaces nn.Linear and Conv1D layers with HashedLinear. Registered in COMPRESSOR_REGISTRY as 'hashed_net', 'hashednets', and 'hashednets_compressor'.", | |
| "math": "A_{ij} = \\text{TopK}\\left(\\text{Softmax}\\left(\\frac{Q_i K_j^\\top}{\\sqrt{d}}\\right)\\right)", | |
| "markdown_preview": "# HashedNet Compressor: Weight Sharing via Hash Tables\n\n**Registered as `hashed_net`** \n*Replaces every `nn.Linear` layer with a `HashedLinear` that shares weights through a compact hash table, drastically reducing parameter count while preserving the ability to fine\u2011tune.*\n\n---\n\n## Overview\n\nThe HashedNet Compressor compresses a model by substituting each standard linear layer with a **HashedLinear** layer\u2014a weight\u2011sharing mechanism inspired by the HashedNets paper. Instead of storing a full dense weight matrix of size `out_features \u00d7 in_features`, the compressed layer uses a small learnable" | |
| }, | |
| { | |
| "id": "huffman_coder", | |
| "name": "Huffman Coder & Entropy Compression Engine for Weight and KV Cach", | |
| "file": "Codes/framework/compression_engine/huffman_coder.py", | |
| "type": "Compression Engine", | |
| "category": "Latent & KV-Cache Compression", | |
| "paradigm": "mla", | |
| "icon": "\ud83e\udde0", | |
| "description": "Huffman Coder & Entropy Compression Engine for Weight and KV Cache Quantization. Registered as 'huffman'. Production-hardened with vectorized encode/decode, exact byte accounting, safe LoRA freezing, and robust tokenizer forward checks.", | |
| "math": "f(a_k) = \\sum_{i=1}^N \\mathbb{I}(w_i = a_k), \\quad p(a_k) = \\frac{f(a_k)}{N}", | |
| "markdown_preview": "# Huffman Coder: Frequency-Optimal Prefix Entropy Coding Engine\n\n## 1. Executive Abstract & Theoretical Motivation\n\nFixed-width integer encoding assigns identical bit lengths to all quantized indices, ignoring variations in symbol frequencies. **Huffman Coder** implements prefix-free variable-length entropy coding to compress discrete weight index distributions down to their theoretical entropy limits.\n\nHuffman Coder builds a frequency histogram of quantized parameters, constructs a binary Huffman priority tree, and assigns shorter binary prefix codes to high-frequency values while assigning l" | |
| }, | |
| { | |
| "id": "hybrid_svd_layer", | |
| "name": "Hybrid SVD Layer: Single\u2011Head SVD Approximation with Positional E", | |
| "file": "Codes/framework/compression_engine/hybrid_svd_layer.py", | |
| "type": "Compression Engine", | |
| "category": "Low-Rank SVD & Factorization", | |
| "paradigm": "svd", | |
| "icon": "\u26a1", | |
| "description": "Hybrid SVD Layer: single-head SVD attention replacement with positional encoding, plus full encoder layer and compression engine. Registered as 'hybrid_svd' and 'direct_svd_encoder'.", | |
| "math": "W = U_r \\Sigma_r V_r^\\top + M_{\\text{sparse}}, \\quad \\|M_{\\text{sparse}}\\|_0 \\le k", | |
| "markdown_preview": "# Hybrid SVD Layer: Single\u2011Head SVD Approximation with Positional Encoding for Attention\u2011Free Encoding\n\n**Standalone module; compressor placeholder** \n*Replaces multi\u2011head self\u2011attention with a low\u2011rank SVD\u2011based mixing of token representations, combining global structure extraction with a standard feed\u2011forward network.*\n\n---\n\n## Overview\n\nThe Hybrid SVD Layer introduces an alternative to the standard multi\u2011head self\u2011attention mechanism in Transformer encoder blocks. Instead of computing query\u2011key dot products and a full attention matrix, it employs a **single\u2011head SVD approximation** on the " | |
| }, | |
| { | |
| "id": "hybridkv", | |
| "name": "HybridKV: Hybrid KV Cache Compression for Multimodal & Long-Conte", | |
| "file": "Codes/framework/compression_engine/hybridkv.py", | |
| "type": "Compression Engine", | |
| "category": "Low-Rank SVD & Factorization", | |
| "paradigm": "svd", | |
| "icon": "\u26a1", | |
| "description": "HybridKV: Hybrid KV Cache Compression for Multimodal & Long-Context LLMs. Re-exports HybridKVCompressor, HybridKVCache, and hybridkv_factory.", | |
| "math": "W \\approx U_r \\Sigma_r V_r^\\top = A B^\\top, \\quad r \\ll \\min(d_{in}, d_{out})", | |
| "markdown_preview": "# Hybridkv\n\n## Overview\n## Configuration & Hyperparameters\n\n| Parameter | Description |\n|-----------|-------------|\n\n## Usage Examples\n\n```python\nfrom Codes.framework.compression_engine import Module\n```\n\n## Benchmark Results\n\n| Status | Accuracy | Val Loss | Peak VRAM | Inference Latency |\n|--------|----------|----------|-----------|-------------------|\n| Pending| Pending | Pending | Pending | Pending |\n\n## Limitations & Future Work\n\nTODO: Identify specific limitations and edge cases for this module.\n" | |
| }, | |
| { | |
| "id": "hybridkv_compressor", | |
| "name": "HybridKV Compressor: Dual Static-Dynamic Head-Specific Hierarchic", | |
| "file": "Codes/framework/compression_engine/hybridkv_compressor.py", | |
| "type": "Compression Engine", | |
| "category": "Latent & KV-Cache Compression", | |
| "paradigm": "mla", | |
| "icon": "\ud83e\udde0", | |
| "description": "HybridKV: Hybrid KV Cache Compression for Multimodal & Long-Context LLMs. Classifies attention heads into static/dynamic types and applies distinct compression strategies per type under a hierarchical budget strictly to dynamic KV caches. Paper: \"HybridKV: Hybrid KV Cache Compression for Efficient MLLM Inference\" Zeng et al., arXiv:2604.058", | |
| "math": "\\sigma_i^2 = \\text{Var}_j(W_{i, j}) = \\frac{1}{D_{\\text{in}}} \\sum_{j=1}^{D_{\\text{in}}} \\left( W_{i, j} - \\mu_i \\right)^2 \\quad \\text{for } i \\in \\{1, \\dots, D_{\\text{out}}\\}", | |
| "markdown_preview": "# HybridKV Compressor: Dual Static-Dynamic Head-Specific Hierarchical KV Cache Compression\n\n## 1. Executive Abstract & Theoretical Motivation\n\nMultimodal LLMs (MLLMs) process textual prompts alongside dense visual token streams. Standard KV cache compression applies uniform pruning policies across all attention heads, ignoring functional specialization between heads. **HybridKV Compressor** introduces dual head classification, partitioning attention heads into **Static Heads** (text-prior driven) and **Dynamic Heads** (context-sensitive).\n\nHybridKV measures output row variance across attention" | |
| }, | |
| { | |
| "id": "kmeans_quantize", | |
| "name": "KMeans-based weight clustering quantizer with STE. Registered as ", | |
| "file": "Codes/framework/compression_engine/kmeans_quantize.py", | |
| "type": "Compression Engine", | |
| "category": "Quantization & Mixed Precision", | |
| "paradigm": "quant", | |
| "icon": "\ud83d\udc8e", | |
| "description": "KMeans-based weight clustering quantizer with STE. Registered as 'kmeans' and 'kmeans_quantize'.", | |
| "math": "W = S \\odot (Q_W - Z), \\quad Q_W \\in \\{-8, \\dots, 7\\}, \\quad S = \\text{scale}", | |
| "markdown_preview": "# KMeans-based Weight Clustering Quantizer with Straight-Through Estimator (STE)\n\n## 1. Executive Abstract & Theoretical Motivation\n\nScalar fixed-point quantization restricts quantization grid points to strictly uniform steps ($s \\cdot k$). In contrast, neural network weight distributions are highly non-uniform (typically bell-shaped Gaussian or Laplacian).\n\n**KMeansQuantizer** applies non-uniform scalar vector quantization using **1D Lloyd's $K$-means clustering**:\n1. **Optimal Codebook Construction**: Partitions continuous weights $W \\in \\mathbb{R}^{d_{\\text{out}} \\times d_{\\text{in}}}$ into" | |
| }, | |
| { | |
| "id": "kronecker_compressor", | |
| "name": "Kronecker Product Compressor: replaces nn.Linear and Conv1D layer", | |
| "file": "Codes/framework/compression_engine/kronecker_compressor.py", | |
| "type": "Compression Engine", | |
| "category": "Low-Rank SVD & Factorization", | |
| "paradigm": "svd", | |
| "icon": "\u26a1", | |
| "description": "Kronecker Product Compressor: replaces nn.Linear and Conv1D layers with KroneckerLinear. Registered in COMPRESSOR_REGISTRY as 'kronecker'. Production hardened with reconstruction validation, Conv1D support, adaptive SVD fallback, and critical layer protection.", | |
| "math": "W \\approx U_r \\Sigma_r V_r^\\top = A B^\\top, \\quad r \\ll \\min(d_{in}, d_{out})", | |
| "markdown_preview": "# KroneckerCompressor: Automated Kronecker Subspace Decomposer for Deep Neural Networks\n\n## 1. Executive Abstract & Theoretical Motivation\nAs deep learning models scale to multi-billion parameter regimes, hosting dense weight matrices in High Bandwidth Memory (HBM) creates severe memory bottlenecks during inference. While low-rank SVD truncation provides compression ratios between $2\\times$ and $8\\times$, Kronecker factorizations offer mathematical structures capable of achieving $100\\times$ to $2000\\times$ parameter reductions.\n\n`KroneckerCompressor` (registered in `COMPRESSOR_REGISTRY` as `\"" | |
| }, | |
| { | |
| "id": "kronecker_linear", | |
| "name": "Kronecker Linear Layer: weight matrix represented as Kronecker pr", | |
| "file": "Codes/framework/compression_engine/kronecker_linear.py", | |
| "type": "Compression Engine", | |
| "category": "Low-Rank SVD & Factorization", | |
| "paradigm": "svd", | |
| "icon": "\u26a1", | |
| "description": "Kronecker Linear Layer: weight matrix represented as Kronecker product A \u2297 B. Provides true sub-quadratic FLOP execution and factorized memory efficiency.", | |
| "math": "W \\approx U_r \\Sigma_r V_r^\\top = A B^\\top, \\quad r \\ll \\min(d_{in}, d_{out})", | |
| "markdown_preview": "# KroneckerLinear: Kronecker Product Factorized Linear Operator\n\n## 1. Executive Abstract & Theoretical Motivation\nStandard dense linear layers $W \\in \\mathbb{R}^{d_{\\text{out}} \\times d_{\\text{in}}}$ exhibit quadratic parameter growth $O(d_{\\text{out}} \\cdot d_{\\text{in}})$. While low-rank factorization methods (such as SVD or LoRA) reduce memory by projecting activations into an intermediate rank-$r$ bottleneck, they still require storing two full-length projection matrices ($M \\times r$ and $r \\times N$).\n\n`KroneckerLinear` provides a parameter-efficient structural factorization by paramete" | |
| }, | |
| { | |
| "id": "kronecker_tt_lora", | |
| "name": "Kronecker-TT-LoRA: Hybrid Compression Engine for Parameter-Effici", | |
| "file": "Codes/framework/compression_engine/kronecker_tt_lora.py", | |
| "type": "Compression Engine", | |
| "category": "Dynamic & Efficient Attention", | |
| "paradigm": "sparse", | |
| "icon": "\ud83c\udf1f", | |
| "description": "Kronecker-TT-LoRA Hybrid Compression Engine Adapter. Implements a hybrid parameter-efficient fine-tuning (PEFT) and compression mechanism by decomposing weight updates into a Kronecker product multiplied element-wise by a Tensor-Train (TT) core tensor with dynamic singular value entropy adaptation, orthogonal initialization with spectral normaliza", | |
| "math": "A_{ij} = \\text{TopK}\\left(\\text{Softmax}\\left(\\frac{Q_i K_j^\\top}{\\sqrt{d}}\\right)\\right)", | |
| "markdown_preview": "# Kronecker-TT-LoRA: Hybrid Compression Engine for Parameter-Efficient Fine-Tuning\n\n## 1. Abstract\nThe scaling of foundation models poses significant memory and computational bottlenecks, particularly during fine-tuning on downstream tasks. **Kronecker-TT-LoRA** introduces a hybrid parameter-efficient fine-tuning (PEFT) mechanism that decomposes weight updates into a Kronecker product and a Tensor-Train (TT) core structure. By integrating dynamic rank adaptation based on singular value entropy, the architecture optimally redistributes parameter capacity across network layers. This synthesis su" | |
| }, | |
| { | |
| "id": "kvtc", | |
| "name": "KVTC: KV Cache Transform Coding - PCA decorrelation + adaptive qu", | |
| "file": "Codes/framework/compression_engine/kvtc.py", | |
| "type": "Compression Engine", | |
| "category": "Low-Rank SVD & Factorization", | |
| "paradigm": "svd", | |
| "icon": "\u26a1", | |
| "description": "KVTC: KV Cache Transform Coding - PCA decorrelation + adaptive quantization + entropy coding. Re-exports KVTCCompressor, KVTCStore, and kvtc_factory.", | |
| "math": "W \\approx U_r \\Sigma_r V_r^\\top = A B^\\top, \\quad r \\ll \\min(d_{in}, d_{out})", | |
| "markdown_preview": "# Kvtc\n\n## Overview\n## Configuration & Hyperparameters\n\n| Parameter | Description |\n|-----------|-------------|\n\n## Usage Examples\n\n```python\nfrom Codes.framework.compression_engine import Module\n```\n\n## Benchmark Results\n\n| Status | Accuracy | Val Loss | Peak VRAM | Inference Latency |\n|--------|----------|----------|-----------|-------------------|\n| Pending| Pending | Pending | Pending | Pending |\n\n## Limitations & Future Work\n\nTODO: Identify specific limitations and edge cases for this module.\n" | |
| }, | |
| { | |
| "id": "kvtc_compressor", | |
| "name": "KVTC Compressor: Key-Value Transform Coding via PCA Decorrelation", | |
| "file": "Codes/framework/compression_engine/kvtc_compressor.py", | |
| "type": "Compression Engine", | |
| "category": "Latent & KV-Cache Compression", | |
| "paradigm": "mla", | |
| "icon": "\ud83e\udde0", | |
| "description": "KVTC: KV Cache Transform Coding - PCA decorrelation + adaptive quantization + entropy coding for compact KV cache storage. Paper: \"KV Cache Transform Coding for Compact Storage in LLM Inference\" Staniszewski & Lancucki, ICLR 2026. Registered as 'kvtc' and 'kvtc_quantizer'.", | |
| "math": "K_{\\text{comp}} = K V_K^\\top, \\; V_{\\text{comp}} = V V_V^\\top, \\quad V_K, V_V \\in \\mathbb{R}^{r \\times d_k}", | |
| "markdown_preview": "# KVTC Compressor: Key-Value Transform Coding via PCA Decorrelation and Adaptive Quantization\n\n## 1. Executive Abstract & Theoretical Motivation\n\nKey-Value cache tensors exhibit significant spatial and temporal cross-channel correlations, leading to redundant memory usage during long-context LLM inference. **KVTC Compressor** (KV Cache Transform Coding) treats KV compression as a classical signal transform coding problem by unifying **PCA Decorrelation (SVD)**, **Adaptive Quantization**, and **Entropy Rate Estimation**.\n\nKVTC applies SVD to decorrelate linear layer weight tensors $W \\in \\mathb" | |
| }, | |
| { | |
| "id": "liftr", | |
| "name": "LIFTR: Low-Rank In-Flight Truncated Reconstruction Compressor", | |
| "file": "Codes/framework/compression_engine/liftr.py", | |
| "type": "Compression Engine", | |
| "category": "Structured Pruning & Channel Slicing", | |
| "paradigm": "pruning", | |
| "icon": "\u2702\ufe0f", | |
| "description": "LIFTR \u2013 Linearized Function Trick Compressor. Registered in COMPRESSOR_REGISTRY as 'liftr' and 'liftr_compressor'. Mathematical Formulation: 1. Activation Function Linearization: - $\\\\operatorname{ReLU}(x) \\\\to \\\\operatorname{Identity}(x)$ - $\\\\operatorname{GELU}(x) \\\\approx \\\\alpha \\\\cdot x$ (default $\\\\alpha = 0.8$) - $\\\\operatorname{Si", | |
| "math": "Y = W \\Omega, \\quad Q = \\text{qr}(Y)", | |
| "markdown_preview": "# LIFTR: Low-Rank In-Flight Truncated Reconstruction Compressor\n\n## 1. Executive Abstract & Architectural Motivation\n**LIFTR (Low-Rank In-Flight Truncated Reconstruction)** provides continuous runtime low-rank matrix decomposition without halting inference pipelines. It computes streaming low-rank updates via randomized matrix sketches, ensuring near-instantaneous model adaptation.\n\n---\n\n## 2. Mathematical Formulations & Derivations\nLIFTR maintains streaming randomized projection sketches $\\Omega \\in \\mathbb{R}^{D \\times k}$:\n$$Y = W \\Omega, \\quad Q = \\text{qr}(Y)$$\n$$B = Q^\\top W, \\quad B = \\" | |
| }, | |
| { | |
| "id": "lillama_compressor", | |
| "name": "LiLLaMA: Local Distillation via Activation Cross-Covariance Singu", | |
| "file": "Codes/framework/compression_engine/lillama_compressor.py", | |
| "type": "Compression Engine", | |
| "category": "Low-Rank SVD & Factorization", | |
| "paradigm": "svd", | |
| "icon": "\u26a1", | |
| "description": "LiLLaMA: Local Distillation via Activation Cross-Covariance Singular Value Decomposition. Registered in COMPRESSOR_REGISTRY as 'lillama' and 'lillama_compressor'. Mathematical Formulation: 1. Input and Output Activations: $$X \\\\in \\\\mathbb{R}^{N \\\\times d_{in}}, \\\\quad Y \\\\in \\\\mathbb{R}^{N \\\\times d_{out}}$$ 2. Empirical Cross-Covariance Matri", | |
| "math": "C_{YX} = \\frac{1}{N} Y^\\top X = \\frac{1}{N} \\left( X W^\\top + \\mathbf{1}_N b^\\top \\right)^\\top X = W \\left( \\frac{1}{N} X^\\top X \\right) + b \\left( \\frac{1}{N} \\mathbf{1}_N^\\top X \\right)", | |
| "markdown_preview": "# LiLLaMA: Local Distillation via Activation Cross-Covariance Singular Value Decomposition\n\n## 1. Executive Abstract & Theoretical Motivation\n\nStandard weight-only Singular Value Decomposition (SVD) treats all singular vectors equally, ignoring the fact that neural network activations occupy a low-dimensional manifold embedded in the ambient feature space. **LiLLaMA** (Local Distillation via Cross-Covariance) aligns the low-rank factorization of each linear layer with its empirical input-output data distribution.\n\nGiven calibration sequences, LiLLaMA collects input activations $X \\in \\mathbb{R" | |
| }, | |
| { | |
| "id": "lsc_compressor", | |
| "name": "LSC Compressor: Low-Rank Spectral Fourier Compression Engine", | |
| "file": "Codes/framework/compression_engine/lsc_compressor.py", | |
| "type": "Compression Engine", | |
| "category": "Low-Rank SVD & Factorization", | |
| "paradigm": "svd", | |
| "icon": "\u26a1", | |
| "description": "LSC: Low-Rank Spectral & Fourier-Domain Linear Compressor. Registered in COMPRESSOR_REGISTRY as 'lsc' and 'lsc_compressor'. Mathematical Formulation: 1. 2D Discrete Fourier Transform: $$\\\\widehat{W} = \\\\mathcal{F}_{2D}(W) \\\\in \\\\mathbb{C}^{d_{out} \\\\times d_{in}}$$ 2. Spectral Magnitude Spectrum: $$M_{ij} = |\\\\widehat{W}_{ij}| = \\\\sqrt{\\\\ope", | |
| "math": "\\tilde{W}_{u,v} = \\sum_{n=0}^{N-1} \\sum_{m=0}^{M-1} W_{nm} \\cos\\left(\\frac{\\pi u (2n+1)}{2N}\\right) \\cos\\left(\\frac{\\pi v (2m+1)}{2M}\\right)", | |
| "markdown_preview": "# LSC Compressor: Low-Rank Spectral Fourier Compression Engine\n\n## 1. Executive Abstract & Theoretical Motivation\n\nStandard spatial low-rank techniques (such as spatial SVD) operate strictly in the spatial domain, missing spatial frequency redundancies present across parameter matrices. **LSC Compressor** (Low-Rank Spectral Compression) applies **2D Fast Fourier Transforms (FFT2)** to project weight matrices into the frequency spectrum.\n\nLSC converts spatial weight parameters $W \\in \\mathbb{R}^{N \\times M}$ into complex frequency coefficients $\\mathcal{F}\\{W\\} \\in \\mathbb{C}^{N \\times M}$. It " | |
| }, | |
| { | |
| "id": "mamba_shedder", | |
| "name": "Mamba-Shedder: Sensitivity-Based Structured Pruning and Layer She", | |
| "file": "Codes/framework/compression_engine/mamba_shedder.py", | |
| "type": "Compression Engine", | |
| "category": "Linear Attention & State-Space (SSM)", | |
| "paradigm": "ssm", | |
| "icon": "\ud83d\udc0d", | |
| "description": "Mamba-Shedder: Sensitivity-Based Structured Pruning and Layer Shedding Compression Engine. Registered in COMPRESSOR_REGISTRY as 'mamba_shedder' and 'mamba_shedder_compressor'. Mathematical Formulation: 1. Empirical Fisher Information / Sensitivity Metric: $$\\\\mathcal{I}(X_j) = \\\\sqrt{\\\\frac{1}{N} \\\\sum_{n=1}^N X_{n, j}^2}$$ 2. Layer Residual Se", | |
| "math": "s_j = \\sqrt{ \\frac{1}{N_{\\text{tokens}}} \\sum_{i=1}^{N_{\\text{tokens}}} X_{i, j}^2 } \\quad \\text{for } j \\in \\{1, \\dots, D_{\\text{in}}\\}", | |
| "markdown_preview": "# Mamba-Shedder: Fisher-Sensitivity-Guided Structured Column Shedding Engine\n\n## 1. Executive Abstract & Theoretical Motivation\n\nSelective State Space Models (SSMs) like Mamba and hybrid SSM-Transformer architectures rely heavily on input-dependent dynamic state transitions. Unstructured pruning disrupts state projection matrices $B, C, \\Delta$ and linear projections. **Mamba-Shedder** introduces Fisher-sensitivity-guided structured column shedding designed for state-space dynamics.\n\nMamba-Shedder hooks input activations across calibration sequences to evaluate the empirical Fisher Information" | |
| }, | |
| { | |
| "id": "mgie_svd", | |
| "name": "MGIE-SVD: Maximum Information Entropy SVD Rank Selection.", | |
| "file": "Codes/framework/compression_engine/mgie_svd.py", | |
| "type": "Compression Engine", | |
| "category": "Low-Rank SVD & Factorization", | |
| "paradigm": "svd", | |
| "icon": "\u26a1", | |
| "description": "MGIE-SVD: Maximum Information Entropy SVD Rank Selection. Registered as 'mgie' and 'mgie_svd'.", | |
| "math": "W \\approx U_r \\Sigma_r V_r^\\top = A B^\\top, \\quad r \\ll \\min(d_{in}, d_{out})", | |
| "markdown_preview": "# Mgie\n\n## Overview\nMGIE-SVD Compressor (Maximum Information Entropy SVD).\n\n## Configuration & Hyperparameters\n\n| Parameter | Description |\n|-----------|-------------|\n| `model` | TODO: Description for model |\n| `rank` | TODO: Description for rank |\n| `max_error` | TODO: Description for max_error |\n| `fallback_to_original` | TODO: Description for fallback_to_original |\n| `lora_r` | TODO: Description for lora_r |\n| `lora_alpha` | TODO: Description for lora_alpha |\n| `skip_attention_qkv` | TODO: Description for skip_attention_qkv |\n| `target_attention_only` | TODO: Description for target_attenti" | |
| }, | |
| { | |
| "id": "mgie_svd_compressor", | |
| "name": "MGIESVDBudgetCompressor: Information Entropy SVD Rank Allocation ", | |
| "file": "Codes/framework/compression_engine/mgie_svd_compressor.py", | |
| "type": "Compression Engine", | |
| "category": "Low-Rank SVD & Factorization", | |
| "paradigm": "svd", | |
| "icon": "\u26a1", | |
| "description": "MGIE-SVD with Total Rank Budget: Global Entropy-Proportional Rank Allocation. Registered as 'mgie_budget'.", | |
| "math": "W \\approx U_r \\Sigma_r V_r^\\top = A B^\\top, \\quad r \\ll \\min(d_{in}, d_{out})", | |
| "markdown_preview": "# MGIESVDBudgetCompressor: Information Entropy SVD Rank Allocation under Global Budget Constraints\n\n## 1. Executive Abstract & Theoretical Motivation\nAllocating rank budgets uniformly across deep neural networks disregards information-theoretic principles. Layers possessing high spectral entropy spread information broadly across many orthogonal directions, while layers with low spectral entropy concentrate information within a few dominant principal axes.\n\n`MGIESVDBudgetCompressor` (Maximum Global Information Entropy SVD Budget Compressor) provides an information-theoretic rank allocation para" | |
| }, | |
| { | |
| "id": "mlorq_compressor", | |
| "name": "MLoRQ Compressor: Multi-Layer Optimal Rank SVD and Joint Low-Bit ", | |
| "file": "Codes/framework/compression_engine/mlorq_compressor.py", | |
| "type": "Compression Engine", | |
| "category": "Low-Rank SVD & Factorization", | |
| "paradigm": "svd", | |
| "icon": "\u26a1", | |
| "description": "MLoRQ: Multi-Layer Optimal Rank & Quantisation under a Total Memory Budget. Registered as 'mlorq' and 'mlorq_compressor'.", | |
| "math": "W \\approx \\text{Quant}(A) \\cdot B^\\top + S \\cdot \\text{scale}, \\quad A \\in \\mathbb{R}^{d_{out} \\times r}", | |
| "markdown_preview": "# MLoRQ Compressor: Multi-Layer Optimal Rank SVD and Joint Low-Bit Quantization Engine\n\n## 1. Executive Abstract & Theoretical Motivation\n\nCombining low-rank matrix decomposition and low-bit quantization requires balancing parameter allocations under global memory limits. **MLoRQ Compressor** (Multi-Layer Optimal Rank & Quantization) solves this joint optimization problem under a total memory budget $B_{\\text{total}}$.\n\nMLoRQ decomposes linear weight matrices $W \\in \\mathbb{R}^{M \\times N}$ into rank-$r$ SVD components $U_r, \\Sigma_r, V_r^T$ and searches across bit width candidates $b \\in \\{2," | |
| }, | |
| { | |
| "id": "modegpt", | |
| "name": "MoDeGPT: Modular Decomposition of Transformer Blocks.", | |
| "file": "Codes/framework/compression_engine/modegpt.py", | |
| "type": "Compression Engine", | |
| "category": "Low-Rank SVD & Factorization", | |
| "paradigm": "svd", | |
| "icon": "\u26a1", | |
| "description": "MoDeGPT: Modular Decomposition of Transformer Blocks. Re-exports MoDeGPTCompressor and modegpt_factory.", | |
| "math": "W \\approx U_r \\Sigma_r V_r^\\top = A B^\\top, \\quad r \\ll \\min(d_{in}, d_{out})", | |
| "markdown_preview": "# Modegpt\n\n## Overview\n## Configuration & Hyperparameters\n\n| Parameter | Description |\n|-----------|-------------|\n\n## Usage Examples\n\n```python\nfrom Codes.framework.compression_engine import Module\n```\n\n## Benchmark Results\n\n| Status | Accuracy | Val Loss | Peak VRAM | Inference Latency |\n|--------|----------|----------|-----------|-------------------|\n| Pending| Pending | Pending | Pending | Pending |\n\n## Limitations & Future Work\n\nTODO: Identify specific limitations and edge cases for this module.\n" | |
| }, | |
| { | |
| "id": "modegpt_compressor", | |
| "name": "MoDeGPT: Modular Decomposition of Transformer Blocks.", | |
| "file": "Codes/framework/compression_engine/modegpt_compressor.py", | |
| "type": "Compression Engine", | |
| "category": "Low-Rank SVD & Factorization", | |
| "paradigm": "svd", | |
| "icon": "\u26a1", | |
| "description": "MoDeGPT: Modular Decomposition of Transformer Blocks. Decomposes pairs of linear projections (e.g. Q & K, Up & Gate) jointly via shared low-rank bases. Registered as 'modegpt'.", | |
| "math": "W \\approx U_r \\Sigma_r V_r^\\top = A B^\\top, \\quad r \\ll \\min(d_{in}, d_{out})", | |
| "markdown_preview": "# MoDeGPT: Modular Decomposition of Transformer Blocks\n\n## 1. Executive Abstract & Theoretical Motivation\n\nIn modern Transformer architectures, linear projection modules frequently operate in synergistic pairs sharing the exact same input activation tensor:\n- **Attention Projections**: Query ($W_Q$) and Key ($W_K$) projections receive the same hidden representation $X \\in \\mathbb{R}^{B \\times L \\times D}$.\n- **MLP Gating Projections**: SwiGLU Gate ($W_{\\text{gate}}$) and Up ($W_{\\text{up}}$) projections receive the same post-attention representation.\n\nDecomposing each weight matrix independent" | |
| }, | |
| { | |
| "id": "nli_softmax", | |
| "name": "NLI Softmax approximation \u2013 a fast, quantisation\u2011friendly softmax", | |
| "file": "Codes/framework/compression_engine/nli_softmax.py", | |
| "type": "Compression Engine", | |
| "category": "Structured Pruning & Channel Slicing", | |
| "paradigm": "pruning", | |
| "icon": "\u2702\ufe0f", | |
| "description": "NLI Softmax approximation \u2013 a fast, quantisation\u2011friendly softmax. Uses standard forward softmax, simplified sigmoid\u2011based backward.", | |
| "math": "W' = Q_2^\\top W Q_1, \\quad W_{sliced} = W'[:d'_1, :d'_2], \\quad Q^\\top Q = I", | |
| "markdown_preview": "# NLI Softmax Approximation\n\n**Standalone autograd function** \n*An attempt at a computationally cheaper, quantisation\u2011friendly softmax by replacing the gradient with a simplified sigmoid\u2011based expression, while still using the exact softmax in the forward pass.*\n\n---\n\n## Abstract\n\nThe `NLISoftmaxFunction` is a custom PyTorch autograd module that aims to reduce the computational and memory cost of the softmax operation, particularly during training or when gradients are required. It does so by altering the backward pass: instead of computing the true Jacobian of the softmax, it approximates th" | |
| }, | |
| { | |
| "id": "nmf_compressor", | |
| "name": "NMFCompressor: Non-Negative Matrix Factorization Compression Engi", | |
| "file": "Codes/framework/compression_engine/nmf_compressor.py", | |
| "type": "Compression Engine", | |
| "category": "Low-Rank SVD & Factorization", | |
| "paradigm": "svd", | |
| "icon": "\u26a1", | |
| "description": "NMF: Non-negative Matrix Factorization Linear Compressor. Registered in COMPRESSOR_REGISTRY as 'nmf'.", | |
| "math": "W \\approx U_r \\Sigma_r V_r^\\top = A B^\\top, \\quad r \\ll \\min(d_{in}, d_{out})", | |
| "markdown_preview": "# NMFCompressor: Non-Negative Matrix Factorization Compression Engine\n\n## 1. Executive Abstract & Theoretical Motivation\nStandard Singular Value Decomposition (SVD) generates orthogonal basis vectors containing mixed positive and negative coefficients. In deep neural networks, signed basis combinations can cause destructive cancellation during forward matrix operations, making factorized layers sensitive to quantization and pruning.\n\n`NMFCompressor` introduces Non-Negative Matrix Factorization (NMF) as an additive, interpretable low-rank decomposition engine. By shifting weight matrices into n" | |
| }, | |
| { | |
| "id": "oats", | |
| "name": "OATS: Outlier-Aware Training Sparsification. Registered as 'oats'", | |
| "file": "Codes/framework/compression_engine/oats.py", | |
| "type": "Compression Engine", | |
| "category": "Dynamic & Efficient Attention", | |
| "paradigm": "sparse", | |
| "icon": "\ud83c\udf1f", | |
| "description": "OATS: Outlier-Aware Training Sparsification. Registered as 'oats'.", | |
| "math": "A_{ij} = \\text{TopK}\\left(\\text{Softmax}\\left(\\frac{Q_i K_j^\\top}{\\sqrt{d}}\\right)\\right)", | |
| "markdown_preview": "# Oats\n\n## Overview\n## Configuration & Hyperparameters\n\n| Parameter | Description |\n|-----------|-------------|\n\n## Usage Examples\n\n```python\nfrom Codes.framework.compression_engine import Module\n```\n\n## Benchmark Results\n\n| Status | Accuracy | Val Loss | Peak VRAM | Inference Latency |\n|--------|----------|----------|-----------|-------------------|\n| Pending| Pending | Pending | Pending | Pending |\n\n## Limitations & Future Work\n\nTODO: Identify specific limitations and edge cases for this module.\n" | |
| }, | |
| { | |
| "id": "oats_compressor", | |
| "name": "OATS: Outlier-Aware Training & Saliency-Based Sparsification with", | |
| "file": "Codes/framework/compression_engine/oats_compressor.py", | |
| "type": "Compression Engine", | |
| "category": "Low-Rank SVD & Factorization", | |
| "paradigm": "svd", | |
| "icon": "\u26a1", | |
| "description": "OATS: Outlier-Aware Training & Saliency-Based Sparsification with Low-Rank Matrix Decomposition. Registered in COMPRESSOR_REGISTRY as 'oats' and 'oats_compressor'. Mathematical Formulation: 1. Activation Second Moment: $$M = \\\\frac{1}{N} X^\\\\top X + \\\\epsilon I \\\\in \\\\mathbb{R}^{d_{in} \\\\times d_{in}}$$ 2. Cholesky Factorization: $$M = L_M L", | |
| "math": "\\min_{\\text{rank}(\\hat{W}) \\le r} \\|(W - \\hat{W}) L\\|_F^2", | |
| "markdown_preview": "# OATS: Optimal Activation-Aware Truncated SVD Compressor\n\n## 1. Executive Abstract & Architectural Motivation\n**OATS (Optimal Activation-Aware Truncated SVD)** compresses large foundation models by weighting singular values according to empirical activation covariance $\\Sigma_X = X^\\top X$. By minimizing the activation-weighted reconstruction error $\\|(W - \\hat{W})X\\|_F^2$, OATS prevents catastrophic degradation on outlier channels that typically plagues naive weight-only SVD truncation.\n\n---\n\n## 2. Mathematical Formulations & Derivations\nGiven activation covariance $C = X^\\top X \\in \\mathbb{" | |
| }, | |
| { | |
| "id": "optimal_subspace_projection", | |
| "name": "Optimal Subspace Projection: Activation-Aware Subspace Projection", | |
| "file": "Codes/framework/compression_engine/optimal_subspace_projection.py", | |
| "type": "Compression Engine", | |
| "category": "Low-Rank SVD & Factorization", | |
| "paradigm": "svd", | |
| "icon": "\u26a1", | |
| "description": "Optimal Subspace Projection: Activation-Aware Subspace Projection via Randomized SVD. Registered in COMPRESSOR_REGISTRY as 'optimal_subspace_projection'. Mathematical Formulation: 1. Activation Covariance: $$C = \\\\frac{1}{N} X^\\\\top X + \\\\epsilon I$$ 2. Cholesky Preconditioning Factor: $$C = L L^\\\\top \\\\implies M = W L$$ 3. Randomized Subspa", | |
| "math": "C = \\\\frac{1}{N} X^\\\\top X + \\\\epsilon I", | |
| "markdown_preview": "# Subspace Projection: Activation\u2011Aware Low\u2011Rank Compression via Randomized SVD\n\n**Registered as `subspace_projection`** \n*Compresses linear layers using input activation covariance to whiten the weight matrix, then applies randomized SVD to efficiently find a low\u2011rank approximation aligned with data directions.*\n\n---\n\n## Overview\n\nThe Subspace Projection compressor is a data\u2011aware post\u2011training compression method that reduces the rank of each `nn.Linear` layer\u2019s weight matrix based on the statistical properties of its input activations. By first **whitening** the weight with the Cholesky fac" | |
| }, | |
| { | |
| "id": "padre_attention", | |
| "name": "PADRe: Polynomial Attention with Dropout Regularization.", | |
| "file": "Codes/framework/compression_engine/padre_attention.py", | |
| "type": "Compression Engine", | |
| "category": "Latent & KV-Cache Compression", | |
| "paradigm": "mla", | |
| "icon": "\ud83e\udde0", | |
| "description": "PADRe: Polynomial Attention with Dropout Regularization. Replaces softmax with a stable polynomial approximation, enabling faster inference and kernelized attention across diverse transformer architectures. Production-Hardened with: - Multi-architecture copy_weights (GPT-2, RoBERTa/BERT nested, Qwen/LLaMA, DistilBERT) - Frobenius norm validation (", | |
| "math": "c^{KV}_t = W^{DKV} x_t, \\quad k^C_t = W^{UK} c^{KV}_t, \\quad v^C_t = W^{UV} c^{KV}_t", | |
| "markdown_preview": "# PADRe \u2013 Polynomial Attention with Dropout Regularization\n\n**Registered as `padre`** \n*Replaces the standard softmax attention with a stable, low\u2011degree polynomial kernel, enabling faster computation and paving the way for linear\u2011time attention via kernelisation.*\n\n---\n\n## Abstract\n\nPADRe (Polynomial Attention with Dropout Regularization) introduces a simple yet effective alternative to the softmax\u2011based attention mechanism in Transformers. Instead of computing the exponential of scaled dot\u2011products, PADRe uses a **truncated polynomial kernel** of the form `(1 + S/order)^order`, where `S` ar" | |
| }, | |
| { | |
| "id": "proxsparse", | |
| "name": "ProxSparse: Proximal Gradient Regularized Structured & Semi-Struc", | |
| "file": "Codes/framework/compression_engine/proxsparse.py", | |
| "type": "Compression Engine", | |
| "category": "Low-Rank SVD & Factorization", | |
| "paradigm": "svd", | |
| "icon": "\u26a1", | |
| "description": "ProxSparse: Proximal Gradient Regularized Structured & Semi-Structured Sparsity. Registered in COMPRESSOR_REGISTRY as 'proxsparse', 'proxsparse_pruner', 'proxsparse_compressor', 'proxsparse_factory', 'proxsparse_pruner_factory', and 'proxsparse_compressor_factory'. Mathematical Formulation: 77850\\min_W \\frac{1}{2} \\|W - W_0\\|_F^2 + \\lambda \\|W\\|_1", | |
| "math": "W = U_r \\Sigma_r V_r^\\top + S", | |
| "markdown_preview": "# ProxSparse: Proximal Gradient Sparse SVD Compressor\n\n## 1. Executive Abstract & Architectural Motivation\n**ProxSparse** combines low-rank singular value decomposition with $\\ell_1$-regularized sparse residual corrections:\n$$W = U_r \\Sigma_r V_r^\\top + S$$\nwhere $S$ is a highly sparse error-compensating matrix ($>95\\%$ zeros). By decomposing dense weights into a low-rank component plus sparse outliers, ProxSparse achieves extreme compression ratios while preserving exact precision on critical outlier weights.\n\n---\n\n## 2. Mathematical Formulations & Derivations\nThe optimization objective is so" | |
| }, | |
| { | |
| "id": "proxsparse_pruner", | |
| "name": "ProxSparse: regularized learning of semi-structured sparsity. Reg", | |
| "file": "Codes/framework/compression_engine/proxsparse_pruner.py", | |
| "type": "Compression Engine", | |
| "category": "Structured Pruning & Channel Slicing", | |
| "paradigm": "pruning", | |
| "icon": "\u2702\ufe0f", | |
| "description": "ProxSparse: regularized learning of semi-structured sparsity. Registered as 'proxsparse'.", | |
| "math": "W^{(t+1)} = \\text{prox}_{\\lambda \\|\\cdot\\|_1}\\left(W^{(t)} - \\eta \\nabla f(W^{(t)})\\right) = \\text{sign}(W) \\max(0, |W| - \\lambda \\eta)", | |
| "markdown_preview": "# ProxSparse Pruner: Proximal Gradient Regularized Semi-Structured Pruning Engine\n\n## 1. Executive Abstract & Theoretical Motivation\n\nDirect hard magnitude pruning enforces non-convex zero constraints instantaneously, causing unrecoverable gradient landscape collapses. **ProxSparse Pruner** avoids this failure mode by leveraging **Proximal Gradient Operators** to smoothly regularize weights toward sparse manifold configurations.\n\nProxSparse introduces an iterative proximal map that combines soft-threshold shrinkage with a convex proximal step. Across $T$ iterations, weights are pushed toward a" | |
| }, | |
| { | |
| "id": "pruning_utils", | |
| "name": "Pruning Utilities and Saliency Estimation for Structural Pruning ", | |
| "file": "Codes/framework/compression_engine/pruning_utils.py", | |
| "type": "Compression Engine", | |
| "category": "Structured Pruning & Channel Slicing", | |
| "paradigm": "pruning", | |
| "icon": "\u2702\ufe0f", | |
| "description": "Pruning Utilities and Saliency Estimation for Structural Pruning and Layer Shedding. Provides shared mathematical routines: 1. First-order Taylor expansion saliency: S_ij = |W_ij * g_ij| or data-driven S_ij = |W_ij| * E[|x_j|] 2. Second-order Empirical Hessian / Inverse Hessian saliency: S_ij = |W_ij|^2 / [H^-1]_jj 3. Semi-structured 2:4 sparsity ", | |
| "math": "S_{i, j} = |W_{i, j}| \\cdot \\|X_{:, j}\\|_2", | |
| "markdown_preview": "# Pruning Utilities: Structured Pruning & Importance Metrics Suite\n\n## 1. Executive Abstract & Architectural Motivation\nThe **Pruning Utilities (`pruning_utils`)** module provides shared, mathematically rigorous importance scoring engines for structured and unstructured neural network compression. It includes implementations of Taylor expansion importance, Fisher information metrics, magnitude pruning, and Wanda (Weight and Activation) scoring.\n\n---\n\n## 2. Mathematical Formulations & Derivations\n1. **Wanda Importance Metric**:\n $$S_{i, j} = |W_{i, j}| \\cdot \\|X_{:, j}\\|_2$$\n2. **First-Order " | |
| }, | |
| { | |
| "id": "replaceme_compressor", | |
| "name": "ReplaceMe: Layer Removal and Replacement with Linear Residual Blo", | |
| "file": "Codes/framework/compression_engine/replaceme_compressor.py", | |
| "type": "Compression Engine", | |
| "category": "Low-Rank SVD & Factorization", | |
| "paradigm": "svd", | |
| "icon": "\u26a1", | |
| "description": "ReplaceMe: Layer Removal and Replacement with Linear Residual Block. Replaces the last N layers of a Transformer encoder or decoder with a compatible LayerNorm + Linear residual block with optional trainable LoRA fine-tuning adapters. Registered in COMPRESSOR_REGISTRY as 'replaceme', 'replaceme_compressor', and 'replaceme_factory'.", | |
| "math": "\\text{CKA}(X_l, X_{l+1}) = \\frac{\\|X_l^\\top X_{l+1}\\|_F^2}{\\|X_l^\\top X_l\\|_F \\|X_{l+1}^\\top X_{l+1}\\|_F}", | |
| "markdown_preview": "# ReplaceMe: Structural Layer Replacement Compression\n\n## 1. Executive Abstract & Architectural Motivation\n**ReplaceMe** is a structural compression framework that selectively replaces computationally expensive deep transformer layers with lightweight linear projections, identity passthroughs, or low-rank adapters based on layer similarity metrics.\n\n---\n\n## 2. Mathematical Formulations & Derivations\nLayer similarity is evaluated using Angular Distance or Centered Kernel Alignment (CKA) between hidden representations:\n$$\\text{CKA}(X_l, X_{l+1}) = \\frac{\\|X_l^\\top X_{l+1}\\|_F^2}{\\|X_l^\\top X_l\\|" | |
| }, | |
| { | |
| "id": "roast_hashed_linear", | |
| "name": "ROAST: Hardware-Aware Weight Sharing with Tiling.", | |
| "file": "Codes/framework/compression_engine/roast_hashed_linear.py", | |
| "type": "Compression Engine", | |
| "category": "Low-Rank SVD & Factorization", | |
| "paradigm": "svd", | |
| "icon": "\u26a1", | |
| "description": "ROAST: Hardware-Aware Weight Sharing with Tiling. Reference: 'ROAST: A Hardware-Aware Weight Sharing Scheme for Efficient Neural Networks' Registered in COMPRESSOR_REGISTRY as 'roast', 'roast_compressor', and 'roast_factory'.", | |
| "math": "W \\approx U_r \\Sigma_r V_r^\\top = A B^\\top, \\quad r \\ll \\min(d_{in}, d_{out})", | |
| "markdown_preview": "# ROAST: Hardware\u2011Aware Weight Sharing with Tiling\n\n**Registered as `roast`** \n*Compresses linear layers by partitioning the weight matrix into rectangular tiles, then sharing a single scalar weight across each tile using a hash table. This block\u2011wise sharing is designed to map efficiently to hardware memory hierarchies and reduce storage by orders of magnitude.*\n\n---\n\n## Overview\n\nROAST (Hardware\u2011Aware Weight Sharing with Tiling) is a compression technique that replaces a standard dense weight matrix **W** \u2208 \u211d<sup>M\u00d7N</sup> with a small table of learnable scalars and a fixed tiling pattern. " | |
| }, | |
| { | |
| "id": "rotational_slicegpt_int4", | |
| "name": "Rotational SliceGPT INT4 Compressor: Rotational Invariance, Subsp", | |
| "file": "Codes/framework/compression_engine/rotational_slicegpt_int4.py", | |
| "type": "Compression Engine", | |
| "category": "Latent & KV-Cache Compression", | |
| "paradigm": "mla", | |
| "icon": "\ud83e\udde0", | |
| "description": "Rotational SliceGPT INT4 Compressor: Rotational Invariance, Subspace Slicing, and Second-Order INT4 Quantization. Registered in COMPRESSOR_REGISTRY as 'rotational_slicegpt_int4' and 'rotational_slicegpt'. Mathematical Formulation: 1. Orthogonal Coordinate Rotation: $$\\\\widetilde{W} = W Q, \\\\quad \\\\widetilde{X} = Q^\\\\top X, \\\\quad Q^\\\\top Q = I$", | |
| "math": "\\\\widetilde{W} = W Q, \\\\quad \\\\widetilde{X} = Q^\\\\top X, \\\\quad Q^\\\\top Q = I", | |
| "markdown_preview": "# Rotational-SliceGPT-INT4: Hybrid Compression Engine\n\n## 1. Abstract & Executive Summary\nAs Large Language Models (LLMs) scale to unprecedented parameter counts, memory bandwidth and storage footprints emerge as primary execution bottlenecks. **Rotational-SliceGPT-INT4** introduces a novel hybrid compression framework designed to aggressively reduce model memory requirements while preserving perplexity and task performance. This architecture seamlessly integrates orthogonal rotational invariance, variance-based dimensionality slicing, and second-order Hessian error compensation with Astro 4-b" | |
| }, | |
| { | |
| "id": "rpca", | |
| "name": "RPCA Compressor alias module.", | |
| "file": "Codes/framework/compression_engine/rpca.py", | |
| "type": "Compression Engine", | |
| "category": "Low-Rank SVD & Factorization", | |
| "paradigm": "svd", | |
| "icon": "\u26a1", | |
| "description": "RPCA Compressor alias module. Re-exports RPCACompressor from rpca_compressor.", | |
| "math": "W \\approx U_r \\Sigma_r V_r^\\top = A B^\\top, \\quad r \\ll \\min(d_{in}, d_{out})", | |
| "markdown_preview": "# Rpca\n\n## Overview\n## Configuration & Hyperparameters\n\n| Parameter | Description |\n|-----------|-------------|\n\n## Usage Examples\n\n```python\nfrom Codes.framework.compression_engine import Module\n```\n\n## Benchmark Results\n\n| Status | Accuracy | Val Loss | Peak VRAM | Inference Latency |\n|--------|----------|----------|-----------|-------------------|\n| Pending| Pending | Pending | Pending | Pending |\n\n## Limitations & Future Work\n\nTODO: Identify specific limitations and edge cases for this module.\n" | |
| }, | |
| { | |
| "id": "rpca_compressor", | |
| "name": "RPCACompressor: Robust Principal Component Analysis Low-Rank & Sp", | |
| "file": "Codes/framework/compression_engine/rpca_compressor.py", | |
| "type": "Compression Engine", | |
| "category": "Low-Rank SVD & Factorization", | |
| "paradigm": "svd", | |
| "icon": "\u26a1", | |
| "description": "Robust PCA: decompose W = L + S, factorizing low-rank component L with structural SVDLinear. Registered in COMPRESSOR_REGISTRY as 'rpca', 'rpca_factory', and 'rpca_compressor_factory'.", | |
| "math": "\\min_{L, S} \\|L\\|_* + \\lambda \\|S\\|_1 \\quad \\text{s.t.} \\quad W = L + S, \\quad \\text{rank}(L) \\le r", | |
| "markdown_preview": "# RPCACompressor: Robust Principal Component Analysis Low-Rank & Sparse Matrix Decomposition Engine\n\n## 1. Executive Abstract & Theoretical Motivation\nStandard SVD assumes that weight matrix noise and truncation errors follow homogeneous, low-variance Gaussian distributions across matrix entries. However, deep neural network weight matrices contain sparse, high-magnitude outlier parameters that skew singular value spectra, forcing standard SVD to allocate excessive rank capacity to capture localized extreme values.\n\n`RPCACompressor` (Robust Principal Component Analysis Compressor) isolates out" | |
| }, | |
| { | |
| "id": "saes_svd", | |
| "name": "SAES-SVD v2: error suppression with fixed rank ratio. Registered ", | |
| "file": "Codes/framework/compression_engine/saes_svd.py", | |
| "type": "Compression Engine", | |
| "category": "Low-Rank SVD & Factorization", | |
| "paradigm": "svd", | |
| "icon": "\u26a1", | |
| "description": "SAES-SVD v2: error suppression with fixed rank ratio. Registered as 'saes_svd_v2'. Alias/re-export module for saes_svd_v2.", | |
| "math": "W \\approx U_r \\Sigma_r V_r^\\top = A B^\\top, \\quad r \\ll \\min(d_{in}, d_{out})", | |
| "markdown_preview": "# Saes Svd\n\n## Overview\n## Configuration & Hyperparameters\n\n| Parameter | Description |\n|-----------|-------------|\n\n## Usage Examples\n\n```python\nfrom Codes.framework.compression_engine import Module\n```\n\n## Benchmark Results\n\n| Status | Accuracy | Val Loss | Peak VRAM | Inference Latency |\n|--------|----------|----------|-----------|-------------------|\n| Pending| Pending | Pending | Pending | Pending |\n\n## Limitations & Future Work\n\nTODO: Identify specific limitations and edge cases for this module.\n" | |
| }, | |
| { | |
| "id": "saes_svd_compressor", | |
| "name": "SAES-SVD Compressor: Joint intra- and inter-layer error compensat", | |
| "file": "Codes/framework/compression_engine/saes_svd_compressor.py", | |
| "type": "Compression Engine", | |
| "category": "Low-Rank SVD & Factorization", | |
| "paradigm": "svd", | |
| "icon": "\u26a1", | |
| "description": "SAES-SVD Compressor: Joint intra- and inter-layer error compensation. Registered as 'saes'. Alias/re-export module for saes_svd_v2.", | |
| "math": "W \\approx U_r \\Sigma_r V_r^\\top = A B^\\top, \\quad r \\ll \\min(d_{in}, d_{out})", | |
| "markdown_preview": "# SAESCompressor: Subspace Adaptation & Error Stabilization SVD Engine\n\n## 1. Executive Abstract & Theoretical Motivation\nPost-training low-rank truncation introduces intra-layer reconstruction errors $E^{(l)} = W^{(l)} - \\hat{W}^{(l)}$ across individual layers. When multiple factorized layers are stacked in deep transformer architectures, these localized reconstruction errors propagate sequentially, compounding inter-layer activation distortion $\\Delta X^{(l)}$ through non-linear attention and MLP blocks.\n\n`SAESCompressor` (Subspace Adaptation & Error Stabilization SVD) mitigates error accumu" | |
| }, | |
| { | |
| "id": "saes_svd_v2", | |
| "name": "SAES-SVD v2: Spectrum-Aware Error-Suppressed Singular Value Decom", | |
| "file": "Codes/framework/compression_engine/saes_svd_v2.py", | |
| "type": "Compression Engine", | |
| "category": "Low-Rank SVD & Factorization", | |
| "paradigm": "svd", | |
| "icon": "\u26a1", | |
| "description": "SAES-SVD v2: Spectrum-Aware Error-Suppressed Singular Value Decomposition. Registered in COMPRESSOR_REGISTRY as 'saes_svd_v2' and 'saes'. Mathematical Formulation: 1. Truncated SVD Baseline: $$W_0 \\\\approx U_r \\\\Sigma_r V_r^\\\\top = A_0 B_0$$ 2. Spectral Energy Distribution: $$\\\\rho_k = \\\\frac{\\\\sigma_k^2}{\\\\sum_{j=1}^r \\\\sigma_j^2}$$ 3. Erro", | |
| "math": "W \\approx U_r \\Sigma_r V_r^\\top + \\text{SAE}(W - U_r \\Sigma_r V_r^\\top)", | |
| "markdown_preview": "# SAESSVDCompressorV2: Spectrum-Aware Error Suppression SVD (SAES-SVD v2)\n\n## 1. Executive Abstract & Theoretical Motivation\nDirect truncated SVD approximates a weight matrix $W$ by dropping trailing singular components $\\sigma_{r+1}, \\dots, \\sigma_d$. In deep networks, the cumulative truncation errors across dozens of consecutive layers compound multiplicatively, causing representation degradation.\n\n`SAESSVDCompressorV2` (registered as `saes_svd_v2` and `saes`) implements Spectrum-Aware Error Suppression (SAES). It initializes low-rank factors $A \\in \\mathbb{R}^{d_{\\text{out}} \\times r}$ and " | |
| }, | |
| { | |
| "id": "salr", | |
| "name": "SALR: Structure-Aware / Sparse Low-Rank compression combining SVD", | |
| "file": "Codes/framework/compression_engine/salr.py", | |
| "type": "Compression Engine", | |
| "category": "Low-Rank SVD & Factorization", | |
| "paradigm": "svd", | |
| "icon": "\u26a1", | |
| "description": "SALR: Structure-Aware / Sparse Low-Rank compression combining SVD with bottleneck pruning. Registered in COMPRESSOR_REGISTRY as 'salr', 'salr_factory', and 'salr_compressor_factory'.", | |
| "math": "\\min_{\\text{rank}(\\widehat{W}) \\le r} \\|W - \\widehat{W}\\|_F^2 = \\sum_{i=r+1}^{D} \\sigma_i^2", | |
| "markdown_preview": "# SALR: Unified Low\u2011Rank and Sparse Pruning\n\n**Registered as `salr`** \n*Decomposes each linear layer\u2019s weight via SVD into low\u2011rank factors and then randomly zeroes a fraction of the singular values, creating a hybrid low\u2011rank + sparse diagonal representation that reduces parameters while retaining structural flexibility.*\n\n---\n\n## Overview\n\nSALR (Structured And Low\u2011Rank) compresses a model by replacing every `nn.Linear` layer with a custom `SALRLayer`. This layer first computes the truncated SVD of the original weight matrix **W** and stores the top\u2011`r` left singular vectors (**U**), singula" | |
| }, | |
| { | |
| "id": "sdrm", | |
| "name": "SDRM: Scaled Dimension Rank Modulation Compressor. Registered as ", | |
| "file": "Codes/framework/compression_engine/sdrm.py", | |
| "type": "Compression Engine", | |
| "category": "Low-Rank SVD & Factorization", | |
| "paradigm": "svd", | |
| "icon": "\u26a1", | |
| "description": "SDRM: Scaled Dimension Rank Modulation Compressor. Registered as 'sdrm'.", | |
| "math": "W \\approx U_r \\Sigma_r V_r^\\top = A B^\\top, \\quad r \\ll \\min(d_{in}, d_{out})", | |
| "markdown_preview": "# Sdrm\n\n## Overview\n## Configuration & Hyperparameters\n\n| Parameter | Description |\n|-----------|-------------|\n\n## Usage Examples\n\n```python\nfrom Codes.framework.compression_engine import Module\n```\n\n## Benchmark Results\n\n| Status | Accuracy | Val Loss | Peak VRAM | Inference Latency |\n|--------|----------|----------|-----------|-------------------|\n| Pending| Pending | Pending | Pending | Pending |\n\n## Limitations & Future Work\n\nTODO: Identify specific limitations and edge cases for this module.\n" | |
| }, | |
| { | |
| "id": "sdrm_compressor", | |
| "name": "SDRM: Rank Assignment Proportional to Parameter Budget for Edge I", | |
| "file": "Codes/framework/compression_engine/sdrm_compressor.py", | |
| "type": "Compression Engine", | |
| "category": "Low-Rank SVD & Factorization", | |
| "paradigm": "svd", | |
| "icon": "\u26a1", | |
| "description": "SDRM: Scaled Dimension Rank Modulation. Assigns layer rank proportional to parameter budget for edge inference and factorizes into low-rank SVDLinear layers to achieve physical parameter reduction. Registered as 'sdrm', 'sdrm_factory', and 'sdrm_compressor_factory'.", | |
| "math": "W \\approx U_r \\Sigma_r V_r^\\top = A B^\\top, \\quad r \\ll \\min(d_{in}, d_{out})", | |
| "markdown_preview": "# SDRM: Rank Assignment Proportional to Parameter Budget for Edge Inference\n\n**Registered as `sdrm`** \n*Distributes a global parameter budget across linear layers by assigning each a rank proportional to its original size, then replaces weights with truncated SVD approximations.*\n\n---\n\n## Overview\n\nSDRM (Scalable Dimensionality Reduction for Memory) compresses a model by first establishing a target **total parameter ratio** (fraction of original parameters to keep). It then computes for each `nn.Linear` layer an individual rank such that the factorised low\u2011rank representation (two smaller mat" | |
| }, | |
| { | |
| "id": "slicegpt", | |
| "name": "SliceGPT: Post-Training Slicing with Orthogonal Transformation & ", | |
| "file": "Codes/framework/compression_engine/slicegpt.py", | |
| "type": "Compression Engine", | |
| "category": "Structured Pruning & Channel Slicing", | |
| "paradigm": "pruning", | |
| "icon": "\u2702\ufe0f", | |
| "description": "SliceGPT: Compress Large Language Models by Deleting Rows and Columns. Registered in COMPRESSOR_REGISTRY as 'slicegpt'. Mathematical Formulation: 1. Orthogonal Coordinate Invariance: $$W_{\\\\ell} X_{\\\\ell} = (W_{\\\\ell} Q_{\\\\ell}) (Q_{\\\\ell}^\\\\top X_{\\\\ell})$$ 2. Eigen-decomposition of Activation Covariance: $$C = X X^\\\\top = Q \\\\Lambda Q^\\\\to", | |
| "math": "y = W x + b = W (I_{d_{\\text{in}}}) x + b = W (Q Q^\\top) x + b = (W Q) (Q^\\top x) + b = \\widetilde{W} \\widetilde{x} + b", | |
| "markdown_preview": "# SliceGPT: Post-Training Slicing with Orthogonal Transformation & Variance Concentration\n\n## 1. Overview & Architecture\n\nSliceGPT is a post-training structural pruning framework that compresses large transformer models by physically deleting full rows and columns from weight matrices. Unlike unstructured sparsity methods that produce irregular non-zero patterns requiring specialized sparse matrix kernels, SliceGPT transforms weight tensors into smaller dense matrices $[d_{\\text{out}} \\times r]$ that execute directly on standard dense GEMM hardware accelerators (e.g., NVIDIA Tensor Cores via C" | |
| }, | |
| { | |
| "id": "slicegpt_compressor", | |
| "name": "SliceGPT PCA-based slicing compressor. Registered as 'slicegpt_pc", | |
| "file": "Codes/framework/compression_engine/slicegpt_compressor.py", | |
| "type": "Compression Engine", | |
| "category": "Structured Pruning & Channel Slicing", | |
| "paradigm": "pruning", | |
| "icon": "\u2702\ufe0f", | |
| "description": "SliceGPT PCA-based slicing compressor. Registered as 'slicegpt_pca'.", | |
| "math": "W_{\\text{pruned}} = (Q^\\top W Q)_{[:, \\mathcal{I}]}, \\quad Q = \\text{eigh}(X X^\\top), \\; |\\mathcal{I}| = (1 - 0.45) d_{in}", | |
| "markdown_preview": "# Slicegpt Compressor\n\n## Overview\n## Configuration & Hyperparameters\n\n| Parameter | Description |\n|-----------|-------------|\n\n## Usage Examples\n\n```python\nfrom Codes.framework.compression_engine import Module\n```\n\n## Benchmark Results\n\n| Status | Accuracy | Val Loss | Peak VRAM | Inference Latency |\n|--------|----------|----------|-----------|-------------------|\n| Pending| Pending | Pending | Pending | Pending |\n\n## Limitations & Future Work\n\nTODO: Identify specific limitations and edge cases for this module.\n" | |
| }, | |
| { | |
| "id": "slicegpt_pca", | |
| "name": "SliceGPT PCA Compressor: Principal Component Analysis based Dimen", | |
| "file": "Codes/framework/compression_engine/slicegpt_pca.py", | |
| "type": "Compression Engine", | |
| "category": "Low-Rank SVD & Factorization", | |
| "paradigm": "svd", | |
| "icon": "\u26a1", | |
| "description": "SliceGPT PCA Compressor: Principal Component Analysis based Dimensionality Slicing. Registered in COMPRESSOR_REGISTRY as 'slicegpt_pca' and 'slicegpt_pca_factory'. Mathematical Formulation: 1. Activation Sample Matrix $X \\in \\mathbb{R}^{N \times d_{in}}$ 2. Empirical Covariance: $\\Sigma = \frac{1}{N - 1} (X - \\mu)^\top (X - \\mu)$ 3. Eigendecompositio", | |
| "math": "\\Sigma = \\frac{1}{N - 1} X_c^\\top X_c + \\epsilon I_{d_{\\text{in}}}", | |
| "markdown_preview": "# SliceGPT PCA Compressor: Principal Component Subspace Projection & Variance Slicing\n\n## 1. Overview & Architecture\n\nSliceGPT PCA Compressor performs principal component analysis (PCA) directly on the activation covariance manifold of transformer layers. It applies an orthogonal rotation $V \\in \\mathbb{R}^{d_{\\text{in}} \\times d_{\\text{in}}}$ derived from the mean-centered activation covariance $\\Sigma$, retaining the top-$k$ principal components that capture $(1 - \\text{sparsity})$ cumulative variance. By projecting weights into the empirical PCA coordinate frame, trailing principal coordina" | |
| }, | |
| { | |
| "id": "slim_compressor", | |
| "name": "SLiM: Structured Low-Rank Integrated Matrix Compression (Quantiza", | |
| "file": "Codes/framework/compression_engine/slim_compressor.py", | |
| "type": "Compression Engine", | |
| "category": "Low-Rank SVD & Factorization", | |
| "paradigm": "svd", | |
| "icon": "\u26a1", | |
| "description": "SLiM: Structured Low-Rank Integrated Matrix Compression (Quantization + 2:4 Sparsity + SVD Residual Correction). Registered in COMPRESSOR_REGISTRY as 'slim' and 'slim_compressor'. Mathematical Formulation: 1. Uniform/Non-Uniform b-bit Quantization: $$W_q = \\\\operatorname{Quant}_b(W_0)$$ 2. 2:4 Semi-Structured Sparsity: $$W_{sp} = \\\\Pi_{2:4}(", | |
| "math": "W_q = \\\\operatorname{Quant}_b(W_0)", | |
| "markdown_preview": "# SLiM: One\u2011Shot Quantization, 2:4 Structured Sparsity, and Low\u2011Rank Correction\n\n**Registered as `slim`** \n*Combines three complementary compression techniques in sequence: uniform quantization to low bit\u2011width, 2:4 structured pruning of the quantized weights, and a low\u2011rank correction of the resulting residual. This hybrid approach yields a dense weight matrix that is simultaneously low\u2011precision, highly sparse, and structurally refined.*\n\n---\n\n## Overview\n\nSLiM (Sparsity\u2011Low\u2011rank\u2011Integer Mix) compresses each linear layer\u2019s weight matrix through a fixed pipeline of three operations:\n\n1. **Qu" | |
| }, | |
| { | |
| "id": "soka", | |
| "name": "SoKA: Second-Order Knowledge-Aware Low-Rank Linear Compressor. Re", | |
| "file": "Codes/framework/compression_engine/soka.py", | |
| "type": "Compression Engine", | |
| "category": "Low-Rank SVD & Factorization", | |
| "paradigm": "svd", | |
| "icon": "\u26a1", | |
| "description": "SoKA: Second-Order Knowledge-Aware Low-Rank Linear Compressor. Registered as 'soka'.", | |
| "math": "W \\approx U_r \\Sigma_r V_r^\\top = A B^\\top, \\quad r \\ll \\min(d_{in}, d_{out})", | |
| "markdown_preview": "# Soka\n\n## Overview\n## Configuration & Hyperparameters\n\n| Parameter | Description |\n|-----------|-------------|\n\n## Usage Examples\n\n```python\nfrom Codes.framework.compression_engine import Module\n```\n\n## Benchmark Results\n\n| Status | Accuracy | Val Loss | Peak VRAM | Inference Latency |\n|--------|----------|----------|-----------|-------------------|\n| Pending| Pending | Pending | Pending | Pending |\n\n## Limitations & Future Work\n\nTODO: Identify specific limitations and edge cases for this module.\n" | |
| }, | |
| { | |
| "id": "soka_compressor", | |
| "name": "SoKA: Second-Order Knowledge-Aware Low-Rank Linear Compressor.", | |
| "file": "Codes/framework/compression_engine/soka_compressor.py", | |
| "type": "Compression Engine", | |
| "category": "Low-Rank SVD & Factorization", | |
| "paradigm": "svd", | |
| "icon": "\u26a1", | |
| "description": "SoKA: Second-Order Knowledge-Aware Low-Rank Linear Compressor. Registered in COMPRESSOR_REGISTRY as 'soka'. Mathematical Formulation: 1. Weight Matrix Factorization via SVD: $$W = U \\\\Sigma V^\\\\top \\\\approx U_r \\\\Sigma_r V_r^\\\\top = A B^\\\\top$$ where $A = U_r \\\\Sigma_r^{1/2} \\\\in \\\\mathbb{R}^{d_{out} \\\\times r}$ and $B = V_r \\\\Sigma_r^{1/2} ", | |
| "math": "W = U \\\\Sigma V^\\\\top \\\\approx U_r \\\\Sigma_r V_r^\\\\top = A B^\\\\top", | |
| "markdown_preview": "# SoKA (Kronecker Product) Compressor \u2013 Low\u2011Rank SVD Placeholder\n\n**Registered as `soka`** \n*Replaces each linear layer with a factorised low\u2011rank module (U\u00b7V) initialised from truncated SVD of the original weight matrix. Intended as a lightweight surrogate for a true structured Kronecker product decomposition, it provides immediate parameter reduction and inference speed\u2011up.*\n\n---\n\n## Overview\n\nSoKA (Structured Kronecker Approximation) is a compression technique that replaces every `nn.Linear` layer in a model with a low\u2011rank factorised layer `SoKALinear`. The new layer expresses the weight " | |
| }, | |
| { | |
| "id": "sparsegpt", | |
| "name": "SparseGPT: One-Shot Second-Order Hessian-Guided Weight Pruning fo", | |
| "file": "Codes/framework/compression_engine/sparsegpt.py", | |
| "type": "Compression Engine", | |
| "category": "Low-Rank SVD & Factorization", | |
| "paradigm": "svd", | |
| "icon": "\u26a1", | |
| "description": "SparseGPT: One-Shot Layer-Wise Weight Pruning with Second-Order Error Compensation. Registered in COMPRESSOR_REGISTRY as 'sparsegpt'. Mathematical Formulation: 1. Empirical Hessian Formulation: $$H = 2 X X^\\\\top + \\\\lambda I \\\\in \\\\mathbb{R}^{d_{in} \\\\times d_{in}}$$ 2. Cholesky Factorization & Inversion: $$H = L L^\\\\top \\\\implies H^{-1} = (", | |
| "math": "\\Delta W = -\\frac{w_q}{[H^{-1}]_{qq}} H_{:, q}^{-1}, \\quad H = 2 X X^\\top", | |
| "markdown_preview": "# SparseGPT: One-Shot Second-Order Hessian-Guided Weight Pruning for Large Language Models\n\n## 1. Executive Abstract & Theoretical Motivation\n\nPruning large language models (LLMs) with tens or hundreds of billions of parameters poses severe computational and optimization challenges. Traditional iterative retraining algorithms (such as iterative magnitude pruning with fine-tuning) require prohibitive GPU hours, while zero-shot magnitude pruning induces severe catastrophic collapse beyond $20\\%\\text{--}30\\%$ sparsity. **SparseGPT** resolves this challenge by executing post-training, one-shot lay" | |
| }, | |
| { | |
| "id": "spectraformer_attention", | |
| "name": "Spectraformer: Random Feature Linear Attention with Flexible Kern", | |
| "file": "Codes/framework/compression_engine/spectraformer_attention.py", | |
| "type": "Compression Engine", | |
| "category": "Latent & KV-Cache Compression", | |
| "paradigm": "mla", | |
| "icon": "\ud83e\udde0", | |
| "description": "Spectraformer: Random Feature Linear Attention with Flexible Kernel Approximation. Replaces softmax attention with random orthogonal feature projections, enabling true linear O(N\u00b7r\u00b7d) time and memory complexity. Production-Hardened with: - Multi-architecture copy_weights (GPT-2, RoBERTa/BERT nested, Qwen/LLaMA, DistilBERT) - Frobenius norm validat", | |
| "math": "c^{KV}_t = W^{DKV} x_t, \\quad k^C_t = W^{UK} c^{KV}_t, \\quad v^C_t = W^{UV} c^{KV}_t", | |
| "markdown_preview": "# Spectraformer: Random Feature Linearisation with Flexible Kernel Approximation\n\n**Registered as `spectraformer`** \n*A linear\u2011time attention mechanism that replaces the softmax kernel with a random feature map followed by ReLU activation, enabling O(N) complexity via the associative property.*\n\n---\n\n## Overview\n\nSpectraformer Attention is a drop\u2011in replacement for standard multi\u2011head self\u2011attention that avoids the quadratic cost of computing a full N\u00d7N attention matrix. Instead of evaluating the softmax of query\u2011key dot products, it projects the queries and keys into a higher\u2011dimensional spa" | |
| }, | |
| { | |
| "id": "subspace_projection", | |
| "name": "Subspace Projection Compressor: Low-Rank Subspace Projection and ", | |
| "file": "Codes/framework/compression_engine/subspace_projection.py", | |
| "type": "Compression Engine", | |
| "category": "Low-Rank SVD & Factorization", | |
| "paradigm": "svd", | |
| "icon": "\u26a1", | |
| "description": "Subspace Projection Compressor: Low-Rank Subspace Projection and Factorization. Registered in COMPRESSOR_REGISTRY as 'subspace_projection'.", | |
| "math": "\\min_{P, \\{C_k\\}} \\sum_{k=1}^K \\|W_k - P C_k\\|_F^2 \\quad \\text{s.t.} \\quad P^\\top P = I", | |
| "markdown_preview": "# Subspace Projection Compression Engine\n\n## 1. Executive Abstract & Architectural Motivation\nThe **Subspace Projection Compression Engine** projects high-dimensional weight representations onto an optimal shared lower-dimensional orthonormal subspace $\\mathcal{S}$. By sharing basis matrices across layers and heads, the engine dramatically reduces storage footprint and memory transfer costs.\n\n---\n\n## 2. Mathematical Formulations & Derivations\nGiven weight matrices $\\{W_1, W_2, \\dots, W_K\\} \\subset \\mathbb{R}^{D \\times D}$, a shared orthonormal basis $P \\in \\mathbb{R}^{D \\times d_s}$ ($d_s \\ll " | |
| }, | |
| { | |
| "id": "svd_linear", | |
| "name": "SVD-based low-rank linear layer replacement with optional Fused S", | |
| "file": "Codes/framework/compression_engine/svd_linear.py", | |
| "type": "Compression Engine", | |
| "category": "Low-Rank SVD & Factorization", | |
| "paradigm": "svd", | |
| "icon": "\u26a1", | |
| "description": "SVD-based low-rank linear layer replacement with optional Fused SVD kernel acceleration. Registered as 'svd_linear' and 'svd_linear_compressor'.", | |
| "math": "W \\approx U_r \\Sigma_r V_r^\\top = (U_r \\sqrt{\\Sigma_r}) (\\sqrt{\\Sigma_r} V_r^\\top) = A B^\\top", | |
| "markdown_preview": "# SVDLinearLayer: Low-Rank Factorized Linear Layer Replacement\n\n## 1. Executive Abstract & Theoretical Motivation\nThe `SVDLinearLayer` engine provides a foundational post-training low-rank factorization mechanism for standard dense linear layers (`nn.Linear`) in modern deep neural networks and Large Language Models (LLMs). Transformer architectures expend upwards of 80% to 90% of their parameter footprint and floating-point operations (FLOPs) inside dense matrix multiplications ($W \\in \\mathbb{R}^{d_{\\text{out}} \\times d_{\\text{in}}}$). Under memory bandwidth constraints (the memory wall), rea" | |
| }, | |
| { | |
| "id": "swift_svd", | |
| "name": "Swift-SVD: fast closed-form truncated SVD with symmetric square-r", | |
| "file": "Codes/framework/compression_engine/swift_svd.py", | |
| "type": "Compression Engine", | |
| "category": "Low-Rank SVD & Factorization", | |
| "paradigm": "svd", | |
| "icon": "\u26a1", | |
| "description": "Swift-SVD: fast closed-form truncated SVD with symmetric square-root singular value distribution. Registered as 'swift_svd'. Production-hardened with: - Relative Frobenius reconstruction error validation (err <= max_error) - Adaptive rank retries stepping up rank if error threshold is exceeded - Safe fallback preserving original layer on failure (f", | |
| "math": "W \\approx U_r \\Sigma_r V_r^\\top = A B^\\top, \\quad r \\ll \\min(d_{in}, d_{out})", | |
| "markdown_preview": "# Swift\u2011SVD: Closed\u2011Form SVD Compression with Rank Ratio\n\n**Registered as `swift_svd`** \n*A lightweight, zero\u2011shot compressor that replaces each linear layer\u2019s weight matrix with its optimal low\u2011rank approximation via truncated SVD, using a fixed fraction of the full rank.*\n\n---\n\n## Overview\n\nSwift\u2011SVD is a post\u2011training compression method that applies a simple truncated Singular Value Decomposition to the weight matrix of every `nn.Linear` layer in a model. It requires no calibration data, no iterative optimization, and no fine\u2011tuning. The user specifies a **rank ratio** \u2013 a fraction of `min" | |
| }, | |
| { | |
| "id": "swift_svd_act", | |
| "name": "Swift-SVD Activation-Aware Compressor via cross-covariance / acti", | |
| "file": "Codes/framework/compression_engine/swift_svd_act.py", | |
| "type": "Compression Engine", | |
| "category": "Low-Rank SVD & Factorization", | |
| "paradigm": "svd", | |
| "icon": "\u26a1", | |
| "description": "Swift-SVD Activation-Aware Compressor via cross-covariance / activation subspace projection. Registered as 'swift_svd_act'.", | |
| "math": "\\min_{A, B} \\left\\|(W - A B) \\Sigma_{XX}^{1/2}\\right\\|_F^2, \\quad \\Sigma_{XX} = \\frac{1}{N} X X^\\top", | |
| "markdown_preview": "# SwiftSVDActivationCompressor: Activation-Aware SVD via Cross-Covariance Projection\n\n## 1. Executive Abstract & Theoretical Motivation\nPure weight-based SVD compresses $W$ without accounting for the distribution of activations that pass through the layer during inference. Activation-aware compression seeks low-rank factors that minimize output distortion:\n\n$$\\min_{A, B} \\mathbb{E}_{x \\sim \\mathcal{D}} \\|W x - A B x\\|_2^2$$\n\n`SwiftSVDActivationCompressor` (registered as `swift_svd_act`) solves this efficiently by capturing input activations $X \\in \\mathbb{R}^{N \\times d_{\\text{in}}}$ and outpu" | |
| }, | |
| { | |
| "id": "swift_svd_compressor", | |
| "name": "Swift-SVD Activation-aware via cross-covariance. Registered as 's", | |
| "file": "Codes/framework/compression_engine/swift_svd_compressor.py", | |
| "type": "Compression Engine", | |
| "category": "Low-Rank SVD & Factorization", | |
| "paradigm": "svd", | |
| "icon": "\u26a1", | |
| "description": "Swift-SVD Activation-aware via cross-covariance. Registered as 'swift_svd_act'. Alias/re-export module for swift_svd_act.", | |
| "math": "W \\approx U_r \\Sigma_r V_r^\\top = A B^\\top, \\quad r \\ll \\min(d_{in}, d_{out})", | |
| "markdown_preview": "# Swift Svd Compressor\n\n## Overview\n## Configuration & Hyperparameters\n\n| Parameter | Description |\n|-----------|-------------|\n\n## Usage Examples\n\n```python\nfrom Codes.framework.compression_engine import Module\n```\n\n## Benchmark Results\n\n| Status | Accuracy | Val Loss | Peak VRAM | Inference Latency |\n|--------|----------|----------|-----------|-------------------|\n| Pending| Pending | Pending | Pending | Pending |\n\n## Limitations & Future Work\n\nTODO: Identify specific limitations and edge cases for this module.\n" | |
| }, | |
| { | |
| "id": "tlinformer_attention", | |
| "name": "TLinFormer: Exact linear\u2011time attention via low\u2011rank sequence pro", | |
| "file": "Codes/framework/compression_engine/tlinformer_attention.py", | |
| "type": "Compression Engine", | |
| "category": "Low-Rank SVD & Factorization", | |
| "paradigm": "svd", | |
| "icon": "\u26a1", | |
| "description": "TLinFormer: Exact linear\u2011time attention via low\u2011rank sequence projections. Reference: LinFormer (Wang et al., 2020) - adapted for DistilBERT, RoBERTa, GPT-2, and Qwen3.5. Production-Hardened with: - Multi-architecture copy_weights (GPT-2, RoBERTa/BERT nested, Qwen/LLaMA, DistilBERT) - Frobenius norm validation (>= 1e-4) - Grouped-Query Attention (", | |
| "math": "W \\approx U_r \\Sigma_r V_r^\\top = A B^\\top, \\quad r \\ll \\min(d_{in}, d_{out})", | |
| "markdown_preview": "# TLinFormer Attention: Sequence-Dimension Projected Linear Attention\n\n## 1. Abstract & Theoretical Motivation\nThe computational scalability of transformer self-attention is constrained by the $\\mathcal{O}(N^2)$ quadratic complexity of calculating pairwise dot-products across key-query sequence length $N$. TLinFormer Attention (Truncated LinFormer) optimizes sequence interaction by projecting key and value sequence length dimensions from $N$ down to a compressed projection rank $r \\ll N$.\n\nTLinFormer maintains two parameter matrices $\\mathbf{P}_K, \\mathbf{P}_V \\in \\mathbb{R}^{r \\times N_{\\text" | |
| }, | |
| { | |
| "id": "tomoe", | |
| "name": "ToMoE: Converting Dense MLP into Mixture of Experts via Dynamic S", | |
| "file": "Codes/framework/compression_engine/tomoe.py", | |
| "type": "Compression Engine", | |
| "category": "Mixture of Experts & Routing", | |
| "paradigm": "moe", | |
| "icon": "\ud83d\udd00", | |
| "description": "ToMoE: Mixture of Experts via weight clustering. Registered as 'tomoe' and 'tomoe_compressor'. Alias/re-export module for tomoe_compressor.", | |
| "math": "y = \\sum_{i=1}^K G(x)_i \\cdot E_i(x), \\quad G(x) = \\text{Softmax}(\\text{TopK}(x W_g, K=2))", | |
| "markdown_preview": "# ToMoE: Converting Dense MLP into Mixture of Experts via Dynamic Structural Pruning\n\n**Registered as `tomoe`** \n*Transforms selected linear layers (typically feed\u2011forward blocks) into a Mixture of Experts (MoE) by clustering output neurons based on weight similarity and adding a learned router, enabling sparse conditional computation and potential inference efficiency.*\n\n---\n\n## Overview\n\nToMoE (Transformation to Mixture of Experts) is a post\u2011training compression technique that restructures a standard dense linear layer into an MoE module. Instead of all parameters being active for every inp" | |
| }, | |
| { | |
| "id": "tomoe_compressor", | |
| "name": "ToMoE: converts dense linear and Conv1D layers (e.g. FFN and atte", | |
| "file": "Codes/framework/compression_engine/tomoe_compressor.py", | |
| "type": "Compression Engine", | |
| "category": "Mixture of Experts & Routing", | |
| "paradigm": "moe", | |
| "icon": "\ud83d\udd00", | |
| "description": "ToMoE: converts dense linear and Conv1D layers (e.g. FFN and attention projections) into a Mixture of Experts using weight clustering. Registered as 'tomoe' and 'tomoe_compressor'. Production-hardened with: - Fix 1: Reconstruction error validation with configurable max_error (default 0.20) and safe fallback. - Fix 2: Support for Conv1D (GPT-2) wit", | |
| "math": "y = \\sum_{i=1}^K G(x)_i \\cdot E_i(x), \\quad G(x) = \\text{Softmax}(\\text{TopK}(x W_g, K=2))", | |
| "markdown_preview": "# ToMoE: Converting Dense MLP into Mixture of Experts via Dynamic Structural Pruning\n\n**Registered as `tomoe`** \n*Transforms selected linear layers (typically feed\u2011forward blocks) into a Mixture of Experts (MoE) by clustering output neurons based on weight similarity and adding a learned router, enabling sparse conditional computation and potential inference efficiency.*\n\n---\n\n## Overview\n\nToMoE (Transformation to Mixture of Experts) is a post\u2011training compression technique that restructures a standard dense linear layer into an MoE module. Instead of all parameters being active for every inp" | |
| }, | |
| { | |
| "id": "tt_compressor", | |
| "name": "Tensor Train Compressor: replaces eligible nn.Linear and Conv1D l", | |
| "file": "Codes/framework/compression_engine/tt_compressor.py", | |
| "type": "Compression Engine", | |
| "category": "Low-Rank SVD & Factorization", | |
| "paradigm": "svd", | |
| "icon": "\u26a1", | |
| "description": "Tensor Train Compressor: replaces eligible nn.Linear and Conv1D layers with TTLinearLayer. Registered in COMPRESSOR_REGISTRY as \"tt\". NOTE (DEPRECATED): Tensor-Train compression on 2D linear matrices is deprecated in this repository and removed from the active benchmark queue (run_all_benchmarks_queue.py). Due to mode scrambling and fl", | |
| "math": "W \\approx U_r \\Sigma_r V_r^\\top = A B^\\top, \\quad r \\ll \\min(d_{in}, d_{out})", | |
| "markdown_preview": "# TTCompressor: Global Tensor-Train Factorization Framework Engine\n\n> [!WARNING]\n> **DEPRECATED & EXCLUDED FROM BENCHMARK QUEUE**\n> Tensor-Train compression has been removed from the active benchmark queue (`run_all_benchmarks_queue.py`). For detailed theoretical explanations of why TT on 2D matrices underperforms SVD, hardware execution bottlenecks, and empirical findings on pretrained weights, see [`Docs/content/compressions/tt.md`](file:///Users/tahamajs/Documents/uni/Project/Docs/content/compressions/tt.md).\n> Experimental runners are preserved in `Codes/experimental/runners/`.\n\n## 1. Exec" | |
| }, | |
| { | |
| "id": "tt_linear", | |
| "name": "Tensor-Train Linear Layer: parameterizes weight matrices as Tenso", | |
| "file": "Codes/framework/compression_engine/tt_linear.py", | |
| "type": "Compression Engine", | |
| "category": "Low-Rank SVD & Factorization", | |
| "paradigm": "svd", | |
| "icon": "\u26a1", | |
| "description": "Tensor-Train Linear Layer: parameterizes weight matrices as Tensor-Train (TT) Matrix cores. Reference: Novikov et al., 'Tensorizing Neural Networks' (NeurIPS 2015). NOTE (DEPRECATED): Tensor-Train compression on 2D linear matrices is deprecated in this repository. Due to mode scrambling and flattened singular value decay compared to 2D SVD", | |
| "math": "W \\approx U_r \\Sigma_r V_r^\\top = A B^\\top, \\quad r \\ll \\min(d_{in}, d_{out})", | |
| "markdown_preview": "# TTLinearLayer: Tensor-Train Core Decomposition Matrix Layer Engine\n\n> [!WARNING]\n> **DEPRECATED & EXCLUDED FROM BENCHMARK QUEUE**\n> Tensor-Train linear layers have been deprecated and excluded from the production benchmark queue (`run_all_benchmarks_queue.py`). For 2D linear weight matrices in transformers, Tensor-Train decomposition creates artificial mode entanglement and flattens singular value decay, making it empirically and theoretically inferior to SVD.\n> Full mathematical proof and empirical profiling are available at [`Docs/content/compressions/tt.md`](file:///Users/tahamajs/Documen" | |
| }, | |
| { | |
| "id": "uniql_compressor", | |
| "name": "UniQL: Joint low\u2011rank + quantisation using SVD and uniform quant.", | |
| "file": "Codes/framework/compression_engine/uniql_compressor.py", | |
| "type": "Compression Engine", | |
| "category": "Low-Rank SVD & Factorization", | |
| "paradigm": "svd", | |
| "icon": "\u26a1", | |
| "description": "UniQL: Joint low\u2011rank + quantisation using SVD and uniform quant. Registered as 'uniql'. Production-hardened with: - Relative Frobenius reconstruction error validation (err <= max_error) - Adaptive bit and rank retries - Conv1D support (GPT-2) - Critical layer protection (classifier, score, lm_head, pooler, embedding) - Safe fallback preserving ori", | |
| "math": "W \\approx U_r \\Sigma_r V_r^\\top = A B^\\top, \\quad r \\ll \\min(d_{in}, d_{out})", | |
| "markdown_preview": "# UniQL: Joint Low\u2011Rank + Quantisation via SVD and Uniform Quantisation\n\n**Registered as `uniql`** \n*Combines truncated SVD for low\u2011rank approximation with per\u2011factor uniform quantisation to reduce both the effective rank and the numerical precision of a linear layer\u2019s weight matrix.*\n\n---\n\n## Overview\n\nUniQL is a post\u2011training compression method that applies a two\u2011stage reduction to each `nn.Linear` layer\u2019s weight matrix **W** \u2208 \u211d<sup>d_out \u00d7 d_in</sup>:\n\n1. **Truncated SVD** \u2013 the weight is decomposed and only the top\u2011`r` singular components are retained, producing a low\u2011rank approximation." | |
| }, | |
| { | |
| "id": "zerokv_neo", | |
| "name": "ZeroKV-Neo: L2-norm based KV cache eviction with attention sink a", | |
| "file": "Codes/framework/compression_engine/zerokv_neo.py", | |
| "type": "Compression Engine", | |
| "category": "Latent & KV-Cache Compression", | |
| "paradigm": "mla", | |
| "icon": "\ud83e\udde0", | |
| "description": "ZeroKV-Neo: L2-norm based KV cache eviction with attention sink and heavy-hitter preservation. Re-exports ZeroKVNeoCompressor and ZeroKVNeoCache.", | |
| "math": "c^{KV}_t = W^{DKV} x_t, \\quad k^C_t = W^{UK} c^{KV}_t, \\quad v^C_t = W^{UV} c^{KV}_t", | |
| "markdown_preview": "# Zerokv Neo\n\n## Overview\n## Configuration & Hyperparameters\n\n| Parameter | Description |\n|-----------|-------------|\n\n## Usage Examples\n\n```python\nfrom Codes.framework.compression_engine import Module\n```\n\n## Benchmark Results\n\n| Status | Accuracy | Val Loss | Peak VRAM | Inference Latency |\n|--------|----------|----------|-----------|-------------------|\n| Pending| Pending | Pending | Pending | Pending |\n\n## Limitations & Future Work\n\nTODO: Identify specific limitations and edge cases for this module.\n\n## Unified 4-Architecture Benchmark Results (ZeroKV-Neo)\nEvaluated on SST-2 " | |
| }, | |
| { | |
| "id": "zerokv_neo_compressor", | |
| "name": "ZeroKV-Neo: L2-norm based KV cache eviction with attention sink a", | |
| "file": "Codes/framework/compression_engine/zerokv_neo_compressor.py", | |
| "type": "Compression Engine", | |
| "category": "Latent & KV-Cache Compression", | |
| "paradigm": "mla", | |
| "icon": "\ud83e\udde0", | |
| "description": "ZeroKV-Neo: L2-norm based KV cache eviction with attention sink and heavy-hitter preservation. Registered as 'zerokv_neo'. Production-hardened with: - Bounded O(1) cache memory management with safe dimension handling - Relative Frobenius error validation on pruned weights - Adaptive keep_ratio retry loop - Conv1D support (GPT-2) - Critical layer pr", | |
| "math": "c^{KV}_t = W^{DKV} x_t, \\quad k^C_t = W^{UK} c^{KV}_t, \\quad v^C_t = W^{UV} c^{KV}_t", | |
| "markdown_preview": "# ZeroKV-Neo: L2-Norm KV Cache Eviction with Attention Sink & Heavy-Hitter Preservation\n\n## 1. Executive Abstract & Theoretical Motivation\n\nIn autoregressive Large Language Model generation, key-value (KV) cache memory scales linearly with sequence length: $\\mathcal{O}(2 \\cdot B \\cdot H \\cdot L \\cdot d)$, exceeding GPU VRAM for multi-thousand token contexts.\n\nNaive eviction strategies (e.g. standard FIFO sliding windows or random pruning) suffer from catastrophic generation failure: model perplexity abruptly explodes to infinity ($\\text{PPL} \\to \\infty$) once context exceeds cache capacity.\n\n*" | |
| } | |
| ] | |
| } |
Xet Storage Details
- Size:
- 344 kB
- Xet hash:
- 438eb0210de3ff2d8c924e1bb52ad59dbadb57df9763243a9202eebd6b4437ac
·
Xet efficiently stores files, intelligently splitting them into unique chunks and accelerating uploads and downloads. More info.