Update README.md

README.md

@@ -33,6 +33,10 @@ model_type: qwen3_5_text
 
 <div align="center">
 
+# PentaCoder-9B
+
+**Frontier-Calibrated 9B Hybrid Gated DeltaNet Causal Engine for Autonomous Software Engineering**
+
 [![License](https://img.shields.io/badge/License-Apache%202.0-6E56CF?style=for-the-badge)](https://opensource.org/licenses/Apache-2.0)
 [![Library](https://img.shields.io/badge/Library-transformers-FFD21E?style=for-the-badge&logo=huggingface&logoColor=black)](https://github.com/huggingface/transformers)
 [![Merge Method](https://img.shields.io/badge/Merge%20Method-GeoDELLA--HG-27AE60?style=for-the-badge)](#merge-methodology--mathematical-formulation)
@@ -54,6 +58,7 @@ Most sub-10B coding models fail in realistic agentic environments due to a commo
 
 > [!IMPORTANT]
 > The result is a lean, blisteringly fast 9B pure-text causal engine with a native **256k context window** that runs comfortably on consumer GPUs.
+
 ---
 
 ### Contents
@@ -75,14 +80,14 @@ Most sub-10B coding models fail in realistic agentic environments due to a commo
 | :--- | :---: |
 | **Total Parameters** | 8.8B (Pure Text Backbone) |
 | **Architecture Type** | Hybrid Recurrent-Attention Causal LM (`qwen3_5_text`) |
-| **Hidden Dimension** ($d_{\text{model}}$) | 4096 |
-| **Intermediate Dimension** ($d_{\text{mlp}}$) | 12288 (SwiGLU) |
+| **Hidden Dimension** (*d*<sub>model</sub>) | 4096 |
+| **Intermediate Dimension** (*d*<sub>mlp</sub>) | 12288 (SwiGLU) |
 | **Decoder Layers** | 32 |
-| **Attention Layout** | 8 Blocks $\times$ (3 Gated DeltaNet Linear Layers : 1 Gated Softmax Layer) |
+| **Attention Layout** | 8 Blocks × (3 Gated DeltaNet Linear Layers : 1 Gated Softmax Layer) |
 | **Full Attention Layers** | Layers 3, 7, 11, 15, 19, 23, 27, 31 |
-| **Linear Attention Configuration** | 16 Key Heads / 32 Value Heads ($d_k = d_v = 128$) |
-| **Full Attention Configuration** | 16 Query Heads / 4 Key-Value Heads (GQA, $d_h = 256$) |
-| **Rotary Position Embedding (RoPE)** | 1D Partial RoPE ($\theta = 10^7$, Partial Factor = 0.25 $\rightarrow$ 64 dimensions) |
+| **Linear Attention Configuration** | 16 Key Heads / 32 Value Heads (*d*<sub>k</sub> = *d*<sub>v</sub> = 128) |
+| **Full Attention Configuration** | 16 Query Heads / 4 Key-Value Heads (GQA, *d*<sub>h</sub> = 256) |
+| **Rotary Position Embedding (RoPE)** | 1D Partial RoPE (θ = 10⁷, Factor = 0.25 → 64 dimensions) |
 | **Context Window Length** | 262,144 tokens (256k) |
 | **Native Precision** | `bfloat16` |
 | **Vocabulary Size** | 248,320 (Padded for Fill-In-The-Middle and Tool Tokens) |
@@ -91,11 +96,11 @@ Most sub-10B coding models fail in realistic agentic environments due to a commo
 
 ## Composition & Donor Weighting
 
-The foundational weights of [Qwen/Qwen3.5-9B](https://huggingface.co/Qwen/Qwen3.5-9B) serve as the shared topological base ($W_0$). Five specialized donor checkpoints provide non-overlapping task vectors mapped across normalized layer depth $u \in [0, 1]$:
+The foundational weights of [Qwen/Qwen3.5-9B](https://huggingface.co/Qwen/Qwen3.5-9B) serve as the shared topological base (*W*₀). Five specialized donor checkpoints provide non-overlapping task vectors mapped across normalized layer depth *u* ∈ [0, 1]:
 
 | Model | Primary Focus | Depth Target |
 | :--- | :--- | :---: |
-| [Qwen/Qwen3.5-9B](https://huggingface.co/Qwen/Qwen3.5-9B) | Base pre-trained manifold and state-space anchors | Global ($W_0$) |
+| [Qwen/Qwen3.5-9B](https://huggingface.co/Qwen/Qwen3.5-9B) | Base pre-trained manifold and state-space anchors | Global (*W*₀) |
 | [empero-ai/Qwen3.8-9B-Distill](https://huggingface.co/empero-ai/Qwen3.8-9B-Distill) | Abstract token synthesis, reasoning structure, syntax | Lower & Mid Decoders |
 | [Jackrong/Qwopus3.5-9B-Coder](https://huggingface.co/Jackrong/Qwopus3.5-9B-Coder) | Typing discipline, algorithm design, functional purity | Mid Decoders (Bell Curve) |
 | [ornith-ai/Ornith-1.5-9B](https://huggingface.co/ornith-ai/Ornith-1.5-9B) | Circuit-breaker recovery, environment feedback integration | Upper-Mid Decoders |
@@ -110,41 +115,61 @@ The merge was executed using the **GeoDELLA-Coder** pipeline, which solves weigh
 
 ### 1. Task Vector Formulation
 
-For each donor checkpoint $k \in \{1, \dots, 5\}$, the parameter update delta $\tau_k$ is computed relative to the base anchor $W_0$:
+For each donor checkpoint *k* ∈ {1, ..., 5}, the parameter update delta τ<sub>*k*</sub> is computed relative to the base anchor *W*₀:
 
-$$\tau_k = D_k - W_0, \quad k \in \{\text{Qwen3.8}, \text{Qwopus}, \text{Ornith}, \text{OxCoder}, \text{MiMo}\}$$
+$$
+\tau_k = D_k - W_0, \quad k \in \{\text{Qwen3.8}, \text{Qwopus}, \text{Ornith}, \text{OxCoder}, \text{MiMo}\}
+$$
 
 ### 2. Hybrid-Aware Continuous Depth Modulation
 
-Task vector mixing coefficients are continuously modulated over normalized depth $u = \frac{l}{L-1}$, where $l \in [0, 31]$ and $L = 32$:
+Task vector mixing coefficients are continuously modulated over normalized depth *u* = *l* / (*L* - 1), where *l* ∈ [0, 31] and *L* = 32:
 
-$$u_{\text{qwen38}}(u) = 0.15 + 0.35 \cos^2\left(\frac{\pi}{2} u\right) + 0.15 \sin^2(\pi u)$$
+$$
+u_{\text{qwen38}}(u) = 0.15 + 0.35 \cos^2\left(\frac{\pi}{2} u\right) + 0.15 \sin^2(\pi u)
+$$
 
-$$u_{\text{qwopus}}(u) = 0.05 + 0.35 \sin^2(\pi u)$$
+$$
+u_{\text{qwopus}}(u) = 0.05 + 0.35 \sin^2(\pi u)
+$$
 
-$$u_{\text{ornith}}(u) = 0.05 + 0.35 \sin^2\left(\frac{\pi}{2} u\right)$$
+$$
+u_{\text{ornith}}(u) = 0.05 + 0.35 \sin^2\left(\frac{\pi}{2} u\right)
+$$
 
-$$u_{\text{oxcoder}}(u) = 0.05 + 0.25 \sin^2\left(\frac{\pi}{2} u\right)$$
+$$
+u_{\text{oxcoder}}(u) = 0.05 + 0.25 \sin^2\left(\frac{\pi}{2} u\right)
+$$
 
-$$u_{\text{mimo}}(u) = 0.05 + 0.15 \sin(\pi u)$$
+$$
+u_{\text{mimo}}(u) = 0.05 + 0.15 \sin(\pi u)
+$$
 
-The initial coefficients are normalized to form a partition of unity:
+The initial coefficients are normalized to form a partition of unity across all layers:
 
-$$w_k(l) = \frac{u_k(u)}{\sum_{j=1}^5 u_j(u)}, \quad \sum_{k=1}^5 w_k(l) = 1.0$$
+$$
+w_k(l) = \frac{u_k(u)}{\sum_{j=1}^5 u_j(u)}, \quad \sum_{k=1}^5 w_k(l) = 1.0
+$$
 
 ### 3. Dynamic Gram-Matrix Task De-Biasing
 
-Fine-tuned models frequently share underlying distillation datasets, leading to collinearity that can drown out specialized task vectors. To correct for this over-representation, the empirical Gram correlation matrix $G \in \mathbb{R}^{K \times K}$ is evaluated per tensor:
+Fine-tuned models frequently share underlying distillation datasets, leading to collinearity that can drown out specialized task vectors. To correct for this over-representation, the empirical Gram correlation matrix *G* ∈ ℝ<sup>*K* × *K*</sup> is evaluated per tensor:
 
-$$G_{ij} = \frac{|\langle \text{vec}(\tau_i), \text{vec}(\tau_j) \rangle|}{\|\tau_i\|_2 \|\tau_j\|_2}$$
+$$
+G_{ij} = \frac{|\langle \text{vec}(\tau_i), \text{vec}(\tau_j) \rangle|}{\|\tau_i\|_2 \|\tau_j\|_2}
+$$
 
-A uniqueness coefficient $U_k$ is derived from the inverse column sum of task vector correlations:
+A uniqueness coefficient *U*<sub>*k*</sub> is derived from the inverse column sum of task vector correlations:
 
-$$U_k = \frac{1}{\sum_{j=1}^K G_{kj}}$$
+$$
+U_k = \frac{1}{\sum_{j=1}^K G_{kj}}
+$$
 
 The dynamic weights are then re-balanced and normalized:
 
-$$\tilde{w}_k = \frac{w_k U_k}{\sum_{j=1}^K w_j U_j}$$
+$$
+\tilde{w}_k = \frac{w_k U_k}{\sum_{j=1}^K w_j U_j}
+$$
 
 This prevents dataset overlap from suppressing distinct algorithmic representations.
 
@@ -153,82 +178,112 @@ This prevents dataset overlap from suppressing distinct algorithmic representati
 Qwen 3.5 9B features asymmetric head ratios in both its linear attention and full attention layers. Standard fused tensor merging causes cross-head pollution by treating routing projections identically to memory projections. 
 
 Fused projection tensors are sliced into their functional sub-matrices prior to merging:
-- **Gated DeltaNet Layers ($8192 \times d_{\text{model}}$):** Sliced into Query ($2048$), Key ($2048$), and Value ($4096$).
-- **Gated Attention Layers ($6144 \times d_{\text{model}}$):** Sliced into Query ($4096$), Key ($1024$), and Value ($1024$).
+- **Gated DeltaNet Layers (8192 × *d*<sub>model</sub>):** Sliced into Query (2048), Key (2048), and Value (4096).
+- **Gated Attention Layers (6144 × *d*<sub>model</sub>):** Sliced into Query (4096), Key (1024), and Value (1024).
 
-Query and Key slices are processed with a conservative retention density ($\rho = 0.95$) to preserve sharp context routing. Value matrices are processed with adaptive MLP density ($\rho = 0.70$) to maximize conceptual synthesis. The components are then re-concatenated along the head dimension.
+Query and Key slices are processed with a conservative retention density (ρ = 0.95) to preserve sharp context routing. Value matrices are processed with adaptive MLP density (ρ = 0.70) to maximize conceptual synthesis. The components are then re-concatenated along the head dimension.
 
 ### 5. Neuron-Coherent Row Gating
 
 To eliminate destructive interference in Feed-Forward Networks (MLPs), task vectors are gated at the single-neuron (row) level:
 
-$$\bar{\tau} = \frac{1}{K} \sum_{k=1}^K \tau_k$$
+$$
+\bar{\tau} = \frac{1}{K} \sum_{k=1}^K \tau_k
+$$
 
-For each row $r$ of donor delta $\tau_k$, the directional alignment with the consensus mean is evaluated:
+For each row *r* of donor delta τ<sub>*k*</sub>, the directional alignment with the consensus mean is evaluated:
 
-$$\cos \theta_{k, r} = \frac{\langle \tau_{k, r}, \bar{\tau}_r \rangle}{\|\tau_{k, r}\|_2 \|\bar{\tau}_r\|_2 + \epsilon}$$
+$$
+\cos \theta_{k, r} = \frac{\langle \tau_{k, r}, \bar{\tau}_r \rangle}{\|\tau_{k, r}\|_2 \|\bar{\tau}_r\|_2 + \epsilon}
+$$
 
-Rows exhibiting severe directional opposition ($\cos \theta_{k, r} < -0.10$) are masked out:
+Rows exhibiting severe directional opposition (cos θ<sub>*k*, *r*</sub> < -0.10) are masked out:
 
-$$\hat{\tau}_{k, r} = \tau_{k, r} \cdot \mathbb{I}\left(\cos \theta_{k, r} \ge -0.10\right)$$
+$$
+\hat{\tau}_{k, r} = \tau_{k, r} \cdot \mathbb{I}\left(\cos \theta_{k, r} \ge -0.10\right)
+$$
 
 This eliminates opposing gradient vectors that produce incoherent syntax generation.
 
 ### 6. Heavy-Tailed DELLA Adaptive Rescaling
 
-Surviving parameters undergo non-linear magnitude-based sampling. Using parameter rank indices $R_{k, ij} \in [0, 1]$ sorted by absolute magnitude, a Pareto-style retention probability $p_{k, ij}$ is established:
+Surviving parameters undergo non-linear magnitude-based sampling. Using parameter rank indices *R*<sub>*k*, *ij*</sub> ∈ [0, 1] sorted by absolute magnitude, a Pareto-style retention probability *p*<sub>*k*, *ij*</sub> is established:
 
-$$p_{k, ij} = p_{\min} + (p_{\max} - p_{\min}) \cdot \sqrt{R_{k, ij}}$$
+$$
+p_{k, ij} = p_{\min} + (p_{\max} - p_{\min}) \cdot \sqrt{R_{k, ij}}
+$$
 
 Parameters are sampled via a Bernoulli trial and rescaled by their inverse survival probability:
 
-$$M_{k, ij} \sim \text{Bernoulli}(p_{k, ij})$$
+$$
+M_{k, ij} \sim \text{Bernoulli}(p_{k, ij})
+$$
 
-$$\tilde{\tau}_{k, ij} = \frac{\hat{\tau}_{k, ij} \odot M_{k, ij}}{p_{k, ij}}$$
+$$
+\tilde{\tau}_{k, ij} = \frac{\hat{\tau}_{k, ij} \odot M_{k, ij}}{p_{k, ij}}
+$$
 
 ### 7. Coordinate-Wise Sign Consensus & Model Stock Scaling
 
 Directional consensus is determined via weighted sign agreement:
 
-$$\Gamma = \operatorname{sgn}\left(\sum_{k=1}^K \tilde{w}_k \tilde{\tau}_k\right)$$
+$$
+\Gamma = \operatorname{sgn}\left(\sum_{k=1}^K \tilde{w}_k \tilde{\tau}_k\right)
+$$
 
-$$A_k = \mathbb{I}\left(\operatorname{sgn}(\tilde{\tau}_k) = \Gamma\right) \odot \mathbb{I}\left(\tilde{\tau}_k \neq 0\right)$$
+$$
+A_k = \mathbb{I}\left(\operatorname{sgn}(\tilde{\tau}_k) = \Gamma\right) \odot \mathbb{I}\left(\tilde{\tau}_k \neq 0\right)
+$$
 
-$$\Delta_{\text{consensus}} = \frac{\sum_{k=1}^K \tilde{\tau}_k \odot A_k}{\sum_{k=1}^K A_k + \epsilon}$$
+$$
+\Delta_{\text{consensus}} = \frac{\sum_{k=1}^K \tilde{\tau}_k \odot A_k}{\sum_{k=1}^K A_k + \epsilon}
+$$
 
-The aggregated delta is projected onto the non-linear manifold using the Model Stock analytic scaling factor $t^*$:
+The aggregated delta is projected onto the non-linear manifold using the Model Stock analytic scaling factor *t*<sup>*</sup>:
 
-$$\bar{\rho} = \frac{2}{K(K-1)} \sum_{i < j} \frac{\langle \text{vec}(\tau_i), \text{vec}(\tau_j) \rangle}{\|\tau_i\|_2 \|\tau_j\|_2}$$
+$$
+\bar{\rho} = \frac{2}{K(K-1)} \sum_{i < j} \frac{\langle \text{vec}(\tau_i), \text{vec}(\tau_j) \rangle}{\|\tau_i\|_2 \|\tau_j\|_2}
+$$
 
-$$t^* = \frac{K \bar{\rho}}{1 + (K - 1)\bar{\rho}}$$
+$$
+t^* = \frac{K \bar{\rho}}{1 + (K - 1)\bar{\rho}}
+$$
 
-$$\Delta_{\text{final}} = t^* \cdot \Delta_{\text{consensus}}$$
+$$
+\Delta_{\text{final}} = t^* \cdot \Delta_{\text{consensus}}
+$$
 
 ### 8. Spectral Norm & Attention Entropy Anchoring
 
-Attention projection matrices ($W_{\text{attn}}$) are vulnerable to spectral explosion during merges, which contracts attention entropy and leads to repetitive generation loops. 
+Attention projection matrices (*W*<sub>attn</sub>) are vulnerable to spectral explosion during merges, which contracts attention entropy and leads to repetitive generation loops. 
 
-The dominant singular value $\sigma(W)$ is calculated via a three-iteration deterministic power iteration:
+The dominant singular value σ(*W*) is calculated via a three-iteration deterministic power iteration:
 
-$$v^{(t+1)} = \frac{W^T u^{(t)}}{\|W^T u^{(t)}\|_2}, \quad u^{(t+1)} = \frac{W v^{(t+1)}}{\|W v^{(t+1)}\|_2}$$
+$$
+v^{(t+1)} = \frac{W^T u^{(t)}}{\|W^T u^{(t)}\|_2}, \quad u^{(t+1)} = \frac{W v^{(t+1)}}{\|W v^{(t+1)}\|_2}
+$$
 
-$$\sigma(W) \approx {u^{(3)}}^T W v^{(3)}$$
+$$
+\sigma(W) \approx {u^{(3)}}^T W v^{(3)}
+$$
 
 If the merged spectral radius grows more than 5% relative to the base model, it is scaled down:
 
-$$W_{\text{final}} = \begin{cases} W_{\text{merged}} \cdot \left(\frac{1.05 \cdot \sigma(W_0)}{\sigma(W_{\text{merged}})}\right) & \text{if } \sigma(W_{\text{merged}}) > 1.05 \cdot \sigma(W_0) \\ W_{\text{merged}} & \text{otherwise} \end{cases}$$
+$$
+W_{\text{final}} = \begin{cases} W_{\text{merged}} \cdot \left(\frac{1.05 \cdot \sigma(W_0)}{\sigma(W_{\text{merged}})}\right) & \text{if } \sigma(W_{\text{merged}}) > 1.05 \cdot \sigma(W_0) \\ W_{\text{merged}} & \text{otherwise} \end{cases}
+$$
 
 ---
 
 ## Layer-Stratified Component Policies
 
-| Parameter Class | Target Identifiers | Applied Policy | Density ($\rho$) | Mathematical Constraints |
+| Parameter Class | Target Identifiers | Applied Policy | Density (ρ) | Mathematical Constraints |
 | :--- | :--- | :---: | :---: | :--- |
 | **Embeddings & LM Head** | `embed_tokens`, `lm_head` | Low-Memory Streaming Blend | 1.0 | Convex iterative accumulation; vocab dimension aligned to 248,320. |
 | **Linear State-Space Projections** | `linear_attn.in_proj_qkv` | Asymmetric GQA DELLA | 0.95 (QK) / 0.70 (V) | Sub-tensor slicing; separate routing and associative memory passes. |
-| **Self-Attention Projections** | `self_attn.qkv_proj`, `o_proj` | Asymmetric GQA + Spectral Anchor | 0.95 (QK) / 0.70 (V) | Power-iteration clipping prevents $\sigma > 1.05 \sigma_0$. |
-| **Feed-Forward Blocks** | `mlp.gate_proj`, `up_proj`, `down_proj` | Neuron-Gated HG-DELLA | 0.50 – 0.70 | Row-wise cosine filtering ($\cos \theta \ge -0.10$); sign consensus. |
-| **Recurrent Gates & Normalization** | `A_log`, `norm`, `conv1d` | Convex Parameter Blend | 1.0 | Preservation of $A_{\text{log}} \le 0$ to guarantee BIBO stability. |
+| **Self-Attention Projections** | `self_attn.qkv_proj`, `o_proj` | Asymmetric GQA + Spectral Anchor | 0.95 (QK) / 0.70 (V) | Power-iteration clipping prevents σ > 1.05 σ₀. |
+| **Feed-Forward Blocks** | `mlp.gate_proj`, `up_proj`, `down_proj` | Neuron-Gated HG-DELLA | 0.50 – 0.70 | Row-wise cosine filtering (cos θ ≥ -0.10); sign consensus. |
+| **Recurrent Gates & Normalization** | `A_log`, `norm`, `conv1d` | Convex Parameter Blend | 1.0 | Preservation of *A*<sub>log</sub> ≤ 0 to guarantee BIBO stability. |
 
 ---