| ---
|
| license: apache-2.0
|
| language:
|
| - en
|
| tags:
|
| - text-generation
|
| - transformer
|
| - 32-layers
|
| - deepseek-v4
|
| - kimi
|
| - mla
|
| - moe
|
| - muon-optimizer
|
| - hyper-connections
|
| - from-scratch
|
| - research
|
| - viuai
|
| library_name: pytorch
|
| pipeline_tag: text-generation
|
| model-index:
|
| - name: ViuAI-UltraDeep-32L
|
| results: []
|
| ---
|
|
|
| # 🚀 ViuAI-UltraDeep (32 Layers)
|
|
|
| **An ultra-deep ~8.96M parameter MoE language model with DeepSeek-V4 MLA, Kimi Fine-Grained MoE, and mHC Hyper-Connections.**
|
|
|
| ViuAI-UltraDeep pushes the limits of sequential reasoning by stacking **32 Transformer Blocks** with:
|
| - **32 Reasoning Layers (Depth)**: Equivalent depth to frontier 8B models (Llama-3 has 32 layers).
|
| - **Multi-Head Latent Attention (MLA)**: Low-rank Query & KV Joint Compression (80%-90% KV-cache reduction).
|
| - **Fine-Grained MoE (DeepSeek / Kimi Style)**: 1 Always-Active Shared Expert + 8 Routed Experts (Top-2 active per token).
|
| - **Manifold-Constrained Hyper-Connections (mHC)**: Learnable residual scale vectors per layer that preserve gradient fidelity across all 32 layers.
|
| - **52.6% MoE Compute Sparsity**: Total capacity is ~8.96M parameters, but compute cost per token is only **~4.25M active parameters**!
|
| - **Muon Optimizer**: Newton-Schulz 5th-order orthogonalization for 1.5x-2x faster training convergence.
|
| - **Strict Weight Tying**: Input embedding and output LM head share weights.
|
|
|
| ---
|
|
|
| ## 📊 Model Specifications
|
|
|
| | Hyperparameter | Value | Description |
|
| |---|---|---|
|
| | **Layers (Depth)** | **32** | 32 Sequential Thinking Steps per Token |
|
| | **Total Model Parameters** | **8,964,736 (~8.96M)** | Full knowledge capacity across all 32 layers & experts |
|
| | **Active Parameters per Token** | **4,246,144 (~4.25M)** | 1 Shared + 2 Routed Experts executed per layer |
|
| | **Compute Sparsity** | **52.6%** | Over half the expert parameters bypassed per token |
|
| | **Memory Footprint (FP32)** | **~34.20 MB** | Extremely compact for a 32-layer LLM |
|
| | **Hidden Size ($d_{\text{model}}$)** | **128** | Backbone representation dimension |
|
| | **Attention Mechanism** | **DeepSeek MLA** | Multi-Head Latent Attention |
|
| | **Query LoRA Rank ($q_{\text{lora}}$)** | **48** | Low-rank Query compression |
|
| | **KV LoRA Rank ($kv_{\text{lora}}$)** | **32** | Low-rank Joint KV compression |
|
| | **Shared Experts** | **1 per layer** | Preserves core grammatical and semantic structures |
|
| | **Routed Experts** | **8 per layer** | Specialized fine-grained subnetworks |
|
| | **Top-K Routing** | **2 per layer** | Dynamic sparse gating |
|
| | **Expert FFN Dimension** | **64** | SwiGLU mini-expert hidden dimension |
|
| | **Vocabulary** | **4,096** | Custom BPE with Weight Tying |
|
| | **Context Length** | **512 tokens** | Supported sequence length |
|
| | **Residual Architecture** | **mHC** | Learnable Hyper-Connection residual scaling |
|
| | **Training Optimizer** | **Muon + AdamW** | Newton-Schulz matrix polar decomposition |
|
|
|
| ---
|
|
|
| ## 🚀 Quick Start
|
|
|
| ```python
|
| import torch
|
| from viuai.model.config import ViuAIConfig
|
| from viuai.model.viuai_model import ViuAIForCausalLM
|
|
|
| # Load 32-layer model
|
| config = ViuAIConfig(num_layers=32)
|
| model = ViuAIForCausalLM(config)
|
|
|
| print(f"Total Parameters: {model.count_parameters():,}")
|
| # Output: 8,964,736 (~8.96M)
|
|
|
| print(f"Active per Token: {model.count_active_parameters():,}")
|
| # Output: 4,246,144 (~4.25M)
|
| ```
|
|
|
| ---
|
|
|
| **Built with ❤️ by [ViuAI](https://huggingface.co/ViuAI)**
|
|
|
