Create README.md

README.md

@@ -0,0 +1,343 @@
+---
+base_model:
+  - Qwen/Qwen3.5-9B
+  - XiaomiMiMo/MiMo-V2.6-Distill-Qwen-9B
+  - ornith-ai/Ornith-1.5-9B
+  - Jackrong/Qwopus3.5-9B-Coder
+  - OrionLLM/OxCoder-9B
+  - empero-ai/Qwen3.8-9B-Distill
+base_model_relation: merge
+library_name: transformers
+tags:
+  - merge
+  - ties
+  - della
+  - model-stock
+  - geodella
+  - qwen
+  - qwen3.5
+  - causal-lm
+  - deltanet
+  - linear-attention
+  - agentic
+  - reasoning
+  - code
+  - swe-bench
+license: apache-2.0
+language:
+  - en
+  - zh
+pipeline_tag: text-generation
+model_type: qwen3_5_text
+---
+
+<div align="center">
+
+[![License](https://img.shields.io/badge/License-Apache%202.0-6E56CF?style=for-the-badge)](https://opensource.org/licenses/Apache-2.0)
+[![Library](https://img.shields.io/badge/Library-transformers-FFD21E?style=for-the-badge&logo=huggingface&logoColor=black)](https://github.com/huggingface/transformers)
+[![Merge Method](https://img.shields.io/badge/Merge%20Method-GeoDELLA--HG-27AE60?style=for-the-badge)](#merge-methodology--mathematical-formulation)
+[![Architecture](https://img.shields.io/badge/Architecture-Qwen%203.5%209B%20Dense-2D9CDB?style=for-the-badge)](#architectural-specifications)
+[![Attention](https://img.shields.io/badge/Hybrid-Gated%20DeltaNet%20(3:1)-EB5757?style=for-the-badge)](#architectural-specifications)
+[![Context](https://img.shields.io/badge/Context%20Window-256k-F2994A?style=for-the-badge)](#architectural-specifications)
+
+</div>
+
+> [!NOTE]
+> This repository provides quantized GGUF builds of [**pragmaticcs/PentaCoder-9B**](https://huggingface.co/pragmaticcs/PentaCoder-9B)
+
+Most sub-10B coding models fail in realistic agentic environments due to a common trade-off: aggressive fine-tuning on synthetic coding instructions improves immediate benchmark pass rates but introduces brittle syntactic degradation and catastrophic repetition loops when shell commands or compiler checks fail.
+
+**PentaCoder** addresses these limitations by uniting five specialized post-trained checkpoints of [**Qwen 3.5 9B**](https://huggingface.co/Qwen/Qwen3.5-9B) via **GeoDELLA** (Geometric Drop-and-Rescale with Task-Covariance De-Biasing and Spectral Norm Anchoring). The model synthesizes the distinct mathematical distributions of each donor:
+
+- **Algorithmic Correctness & Architectural Decomposition** from [**Qwopus3.5-Coder**](https://huggingface.co/Jackrong/Qwopus3.5-9B-Coder) (Claude 3.5 Opus distillation trajectories).
+- **Multi-Turn SWE-bench Planning & Tool Protocol Integrity** from [**MiMo-V2.6-Distill**](https://huggingface.co/XiaomiMiMo/MiMo-V2.6-Distill-Qwen-9B) (Large-scale agentic execution traces).
+- **Terminal Execution Discipline & Error-Recovery Heuristics** from [**Ornith-1.5**](https://huggingface.co/ornith-ai/Ornith-1.5-9B) (Reinforcement learning for anti-looping).
+- **Low-Level Systems Implementation & Runtime Robustness** from [**OxCoder**](https://huggingface.co/OrionLLM/OxCoder-9B) (Deep API, CLI, and operational coding specialization).
+- **Abstract Structural Reasoning & Syntax Grounding** from [**Qwen3.8-Distill**](https://huggingface.co/empero-ai/Qwen3.8-9B-Distill) (Distilled frontier chain-of-thought representations).
+
+> [!IMPORTANT]
+> The result is a lean, blisteringly fast 9B pure-text causal engine with a native **256k context window** that runs comfortably on consumer GPUs.
+
+---
+
+### Contents
+
+- [Architectural Specifications](#architectural-specifications)
+- [Composition & Donor Weighting](#composition--donor-weighting)
+- [Merge Methodology & Mathematical Formulation](#merge-methodology--mathematical-formulation)
+- [Layer-Stratified Component Policies](#layer-stratified-component-policies)
+- [Agentic Chat Template & Operational Directives](#agentic-chat-template--operational-directives)
+- [Recommended Generation Parameters](#recommended-generation-parameters)
+- [How to Use](#how-to-use)
+- [Citation & References](#citation--references)
+
+---
+
+## Architectural Specifications
+
+| Parameter | Specification |
+| :--- | :---: |
+| **Total Parameters** | 8.8B (Pure Text Backbone) |
+| **Architecture Type** | Hybrid Recurrent-Attention Causal LM (`qwen3_5_text`) |
+| **Hidden Dimension** (*d*<sub>model</sub>) | 4096 |
+| **Intermediate Dimension** (*d*<sub>mlp</sub>) | 12288 (SwiGLU) |
+| **Decoder Layers** | 32 |
+| **Attention Layout** | 8 Blocks × (3 Gated DeltaNet Linear Layers : 1 Gated Softmax Layer) |
+| **Full Attention Layers** | Layers 3, 7, 11, 15, 19, 23, 27, 31 |
+| **Linear Attention Configuration** | 16 Key Heads / 32 Value Heads (*d*<sub>k</sub> = *d*<sub>v</sub> = 128) |
+| **Full Attention Configuration** | 16 Query Heads / 4 Key-Value Heads (GQA, *d*<sub>h</sub> = 256) |
+| **Rotary Position Embedding (RoPE)** | 1D Partial RoPE (θ = 10⁷, Factor = 0.25 → 64 dimensions) |
+| **Context Window Length** | 262,144 tokens (256k) |
+| **Native Precision** | `bfloat16` |
+| **Vocabulary Size** | 248,320 (Padded for Fill-In-The-Middle and Tool Tokens) |
+
+---
+
+## Composition & Donor Weighting
+
+The foundational weights of [Qwen/Qwen3.5-9B](https://huggingface.co/Qwen/Qwen3.5-9B) serve as the shared topological base (*W*₀). Five specialized donor checkpoints provide non-overlapping task vectors mapped across normalized layer depth *u* ∈ [0, 1]:
+
+| Model | Primary Focus | Depth Target |
+| :--- | :--- | :---: |
+| [Qwen/Qwen3.5-9B](https://huggingface.co/Qwen/Qwen3.5-9B) | Base pre-trained manifold and state-space anchors | Global (*W*₀) |
+| [empero-ai/Qwen3.8-9B-Distill](https://huggingface.co/empero-ai/Qwen3.8-9B-Distill) | Abstract token synthesis, reasoning structure, syntax | Lower & Mid Decoders |
+| [Jackrong/Qwopus3.5-9B-Coder](https://huggingface.co/Jackrong/Qwopus3.5-9B-Coder) | Typing discipline, algorithm design, functional purity | Mid Decoders (Bell Curve) |
+| [ornith-ai/Ornith-1.5-9B](https://huggingface.co/ornith-ai/Ornith-1.5-9B) | Circuit-breaker recovery, environment feedback integration | Upper-Mid Decoders |
+| [OrionLLM/OxCoder-9B](https://huggingface.co/OrionLLM/OxCoder-9B) | Systems engineering, runtime bug localization, CLI tooling | Deep Layers (Ascending Ramp) |
+| [XiaomiMiMo/MiMo-V2.6-Distill-Qwen-9B](https://huggingface.co/XiaomiMiMo/MiMo-V2.6-Distill-Qwen-9B) | SWE-bench multi-step planning, long-context tool invocation | Broad Central Decoders |
+
+---
+
+## Merge Methodology & Mathematical Formulation
+
+The merge was executed using the **GeoDELLA-Coder** pipeline, which solves weight interference through dynamic task-correlation de-biasing, row-wise sign consensus, sub-tensor GQA disentanglement, and power-iteration spectral norm stabilization.
+
+### 1. Task Vector Formulation
+
+For each donor checkpoint *k* ∈ {1, ..., 5}, the parameter update delta τ<sub>*k*</sub> is computed relative to the base anchor *W*₀:
+
+$$
+\tau_k = D_k - W_0, \quad k \in \{\text{Qwen3.8}, \text{Qwopus}, \text{Ornith}, \text{OxCoder}, \text{MiMo}\}
+$$
+
+### 2. Hybrid-Aware Continuous Depth Modulation
+
+Task vector mixing coefficients are continuously modulated over normalized depth *u* = *l* / (*L* - 1), where *l* ∈ [0, 31] and *L* = 32:
+
+$$
+u_{\text{qwen38}}(u) = 0.15 + 0.35 \cos^2\left(\frac{\pi}{2} u\right) + 0.15 \sin^2(\pi u)
+$$
+
+$$
+u_{\text{qwopus}}(u) = 0.05 + 0.35 \sin^2(\pi u)
+$$
+
+$$
+u_{\text{ornith}}(u) = 0.05 + 0.35 \sin^2\left(\frac{\pi}{2} u\right)
+$$
+
+$$
+u_{\text{oxcoder}}(u) = 0.05 + 0.25 \sin^2\left(\frac{\pi}{2} u\right)
+$$
+
+$$
+u_{\text{mimo}}(u) = 0.05 + 0.15 \sin(\pi u)
+$$
+
+The initial coefficients are normalized to form a partition of unity across all layers:
+
+$$
+w_k(l) = \frac{u_k(u)}{\sum_{j=1}^5 u_j(u)}, \quad \sum_{k=1}^5 w_k(l) = 1.0
+$$
+
+### 3. Dynamic Gram-Matrix Task De-Biasing
+
+Fine-tuned models frequently share underlying distillation datasets, leading to collinearity that can drown out specialized task vectors. To correct for this over-representation, the empirical Gram correlation matrix *G* ∈ ℝ<sup>*K* × *K*</sup> is evaluated per tensor:
+
+$$
+G_{ij} = \frac{|\langle \text{vec}(\tau_i), \text{vec}(\tau_j) \rangle|}{\|\tau_i\|_2 \|\tau_j\|_2}
+$$
+
+A uniqueness coefficient *U*<sub>*k*</sub> is derived from the inverse column sum of task vector correlations:
+
+$$
+U_k = \frac{1}{\sum_{j=1}^K G_{kj}}
+$$
+
+The dynamic weights are then re-balanced and normalized:
+
+$$
+\tilde{w}_k = \frac{w_k U_k}{\sum_{j=1}^K w_j U_j}
+$$
+
+This prevents dataset overlap from suppressing distinct algorithmic representations.
+
+### 4. Asymmetric GQA Disentangled QKV Slicing
+
+Qwen 3.5 9B features asymmetric head ratios in both its linear attention and full attention layers. Standard fused tensor merging causes cross-head pollution by treating routing projections identically to memory projections. 
+
+Fused projection tensors are sliced into their functional sub-matrices prior to merging:
+- **Gated DeltaNet Layers (8192 × *d*<sub>model</sub>):** Sliced into Query (2048), Key (2048), and Value (4096).
+- **Gated Attention Layers (6144 × *d*<sub>model</sub>):** Sliced into Query (4096), Key (1024), and Value (1024).
+
+Query and Key slices are processed with a conservative retention density (ρ = 0.95) to preserve sharp context routing. Value matrices are processed with adaptive MLP density (ρ = 0.70) to maximize conceptual synthesis. The components are then re-concatenated along the head dimension.
+
+### 5. Neuron-Coherent Row Gating
+
+To eliminate destructive interference in Feed-Forward Networks (MLPs), task vectors are gated at the single-neuron (row) level:
+
+$$
+\bar{\tau} = \frac{1}{K} \sum_{k=1}^K \tau_k
+$$
+
+For each row *r* of donor delta τ<sub>*k*</sub>, the directional alignment with the consensus mean is evaluated:
+
+$$
+\cos \theta_{k, r} = \frac{\langle \tau_{k, r}, \bar{\tau}_r \rangle}{\|\tau_{k, r}\|_2 \|\bar{\tau}_r\|_2 + \epsilon}
+$$
+
+Rows exhibiting severe directional opposition (cos θ<sub>*k*, *r*</sub> < -0.10) are masked out:
+
+$$
+\hat{\tau}_{k, r} = \tau_{k, r} \cdot \mathbb{I}\left(\cos \theta_{k, r} \ge -0.10\right)
+$$
+
+This eliminates opposing gradient vectors that produce incoherent syntax generation.
+
+### 6. Heavy-Tailed DELLA Adaptive Rescaling
+
+Surviving parameters undergo non-linear magnitude-based sampling. Using parameter rank indices *R*<sub>*k*, *ij*</sub> ∈ [0, 1] sorted by absolute magnitude, a Pareto-style retention probability *p*<sub>*k*, *ij*</sub> is established:
+
+$$
+p_{k, ij} = p_{\min} + (p_{\max} - p_{\min}) \cdot \sqrt{R_{k, ij}}
+$$
+
+Parameters are sampled via a Bernoulli trial and rescaled by their inverse survival probability:
+
+$$
+M_{k, ij} \sim \text{Bernoulli}(p_{k, ij})
+$$
+
+$$
+\tilde{\tau}_{k, ij} = \frac{\hat{\tau}_{k, ij} \odot M_{k, ij}}{p_{k, ij}}
+$$
+
+### 7. Coordinate-Wise Sign Consensus & Model Stock Scaling
+
+Directional consensus is determined via weighted sign agreement:
+
+$$
+\Gamma = \operatorname{sgn}\left(\sum_{k=1}^K \tilde{w}_k \tilde{\tau}_k\right)
+$$
+
+$$
+A_k = \mathbb{I}\left(\operatorname{sgn}(\tilde{\tau}_k) = \Gamma\right) \odot \mathbb{I}\left(\tilde{\tau}_k \neq 0\right)
+$$
+
+$$
+\Delta_{\text{consensus}} = \frac{\sum_{k=1}^K \tilde{\tau}_k \odot A_k}{\sum_{k=1}^K A_k + \epsilon}
+$$
+
+The aggregated delta is projected onto the non-linear manifold using the Model Stock analytic scaling factor *t*<sup>*</sup>:
+
+$$
+\bar{\rho} = \frac{2}{K(K-1)} \sum_{i < j} \frac{\langle \text{vec}(\tau_i), \text{vec}(\tau_j) \rangle}{\|\tau_i\|_2 \|\tau_j\|_2}
+$$
+
+$$
+t^* = \frac{K \bar{\rho}}{1 + (K - 1)\bar{\rho}}
+$$
+
+$$
+\Delta_{\text{final}} = t^* \cdot \Delta_{\text{consensus}}
+$$
+
+### 8. Spectral Norm & Attention Entropy Anchoring
+
+Attention projection matrices (*W*<sub>attn</sub>) are vulnerable to spectral explosion during merges, which contracts attention entropy and leads to repetitive generation loops. 
+
+The dominant singular value σ(*W*) is calculated via a three-iteration deterministic power iteration:
+
+$$
+v^{(t+1)} = \frac{W^T u^{(t)}}{\|W^T u^{(t)}\|_2}, \quad u^{(t+1)} = \frac{W v^{(t+1)}}{\|W v^{(t+1)}\|_2}
+$$
+
+$$
+\sigma(W) \approx {u^{(3)}}^T W v^{(3)}
+$$
+
+If the merged spectral radius grows more than 5% relative to the base model, it is scaled down:
+
+$$
+W_{\text{final}} = \begin{cases} W_{\text{merged}} \cdot \left(\frac{1.05 \cdot \sigma(W_0)}{\sigma(W_{\text{merged}})}\right) & \text{if } \sigma(W_{\text{merged}}) > 1.05 \cdot \sigma(W_0) \\ W_{\text{merged}} & \text{otherwise} \end{cases}
+$$
+
+---
+
+## Layer-Stratified Component Policies
+
+| Parameter Class | Target Identifiers | Applied Policy | Density (ρ) | Mathematical Constraints |
+| :--- | :--- | :---: | :---: | :--- |
+| **Embeddings & LM Head** | `embed_tokens`, `lm_head` | Low-Memory Streaming Blend | 1.0 | Convex iterative accumulation; vocab dimension aligned to 248,320. |
+| **Linear State-Space Projections** | `linear_attn.in_proj_qkv` | Asymmetric GQA DELLA | 0.95 (QK) / 0.70 (V) | Sub-tensor slicing; separate routing and associative memory passes. |
+| **Self-Attention Projections** | `self_attn.qkv_proj`, `o_proj` | Asymmetric GQA + Spectral Anchor | 0.95 (QK) / 0.70 (V) | Power-iteration clipping prevents σ > 1.05 σ₀. |
+| **Feed-Forward Blocks** | `mlp.gate_proj`, `up_proj`, `down_proj` | Neuron-Gated HG-DELLA | 0.50 – 0.70 | Row-wise cosine filtering (cos θ ≥ -0.10); sign consensus. |
+| **Recurrent Gates & Normalization** | `A_log`, `norm`, `conv1d` | Convex Parameter Blend | 1.0 | Preservation of *A*<sub>log</sub> ≤ 0 to guarantee BIBO stability. |
+
+---
+
+## Agentic Chat Template & Operational Directives
+
+This model uses the [Improved Chat Template for Qwen 3.x by Olivia Rossi](https://huggingface.co/OliviaRossi/Improved-Chat-Template-for-Qwen-3.x) to support multi-tier Chain-of-Thought (CoT) reasoning, dual-format agentic tool execution, automatic error-recovery heuristics, and strict token-waste elimination.
+
+---
+
+## Recommended Generation Parameters
+
+For deterministic software engineering and complex reasoning benchmarks:
+
+| Parameter | Recommended Value | Description |
+| :--- | :---: | :--- |
+| **Temperature** | `0.6` | Balances strict syntactic validity with algorithmic path exploration. |
+| **Top-P** | `0.95` | Eliminates low-probability token tails while preserving alternative logic paths. |
+| **Top-K** | `20` | Restricts token candidate pools to prevent architectural syntax drift. |
+| **Min-P** | `0.0` (Off) | Disabled in favor of explicit Top-K / Top-P governance. |
+| **Repetition Penalty** | `1.0` (Off) | Disabled to prevent syntax degradation in repetitive code patterns (indentation, braces). |
+| **Presence Penalty** | `0.0` | Prevents naming mutations across long-context symbol resolution. |
+
+---
+
+## Citation & References
+
+- [Qwen/Qwen3.5-9B](https://huggingface.co/Qwen/Qwen3.5-9B)
+- [XiaomiMiMo/MiMo-V2.6-Distill-Qwen-9B](https://huggingface.co/XiaomiMiMo/MiMo-V2.6-Distill-Qwen-9B)
+- [ornith-ai/Ornith-1.5-9B](https://huggingface.co/ornith-ai/Ornith-1.5-9B)
+- [Jackrong/Qwopus3.5-9B-Coder](https://huggingface.co/Jackrong/Qwopus3.5-9B-Coder)
+- [OrionLLM/OxCoder-9B](https://huggingface.co/OrionLLM/OxCoder-9B)
+- [empero-ai/Qwen3.8-9B-Distill](https://huggingface.co/empero-ai/Qwen3.8-9B-Distill)
+- [Improved Chat Template for Qwen 3.x](https://huggingface.co/OliviaRossi/Improved-Chat-Template-for-Qwen-3.x)
+
+```bibtex
+@inproceedings{yadav2023ties,
+  title={Resolving Interference When Merging Models},
+  author={Yadav, Prateek and Tam, Derek and Choshen, Leshem and Raffel, Colin and Bansal, Mohit},
+  booktitle={Advances in Neural Information Processing Systems (NeurIPS)},
+  volume={36},
+  pages={7093--7115},
+  year={2023}
+}
+
+@article{deep2024della,
+  title={DELLA-Merging: Reducing Interference in Model Merging through Magnitude-Based Sampling},
+  author={Deep, Pala Tej and Bhardwaj, Rishabh and Poria, Soujanya},
+  journal={arXiv preprint arXiv:2406.11617},
+  year={2024}
+}
+
+@article{jang2024modelstock,
+  title={Model Stock: All We Need Is just a Few Fine-Tuned Models},
+  author={Jang, Dong-Hwan and Yoon, Sang-Doo and Song, Gyeong-Moon},
+  journal={arXiv preprint arXiv:2403.19522},
+  year={2024}
+}
+```