File size: 3,488 Bytes
7f1c419 96a059a 9e7ff42 7f1c419 96a059a 7f1c419 96a059a 7f1c419 96a059a 7f1c419 96a059a 7f1c419 7324bad 7f1c419 7324bad 7f1c419 96a059a 9e7ff42 96a059a 9e7ff42 7324bad 96a059a 9e7ff42 7f1c419 7324bad 7f1c419 96a059a 7f1c419 96a059a 9e7ff42 96a059a 7324bad 96a059a 7f1c419 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 | ---
license: apache-2.0
language:
- en
tags:
- text-generation
- transformer
- 32-layers
- deepseek-v4
- kimi
- mla
- moe
- muon-optimizer
- hyper-connections
- from-scratch
- research
- viuai
library_name: pytorch
pipeline_tag: text-generation
model-index:
- name: ViuAI-UltraDeep-32L
results: []
---
# 🚀 ViuAI-UltraDeep (32 Layers)
**An ultra-deep ~8.96M parameter MoE language model with DeepSeek-V4 MLA, Kimi Fine-Grained MoE, and mHC Hyper-Connections.**
ViuAI-UltraDeep pushes the limits of sequential reasoning by stacking **32 Transformer Blocks** with:
- **32 Reasoning Layers (Depth)**: Equivalent depth to frontier 8B models (Llama-3 has 32 layers).
- **Multi-Head Latent Attention (MLA)**: Low-rank Query & KV Joint Compression (80%-90% KV-cache reduction).
- **Fine-Grained MoE (DeepSeek / Kimi Style)**: 1 Always-Active Shared Expert + 8 Routed Experts (Top-2 active per token).
- **Manifold-Constrained Hyper-Connections (mHC)**: Learnable residual scale vectors per layer that preserve gradient fidelity across all 32 layers.
- **52.6% MoE Compute Sparsity**: Total capacity is ~8.96M parameters, but compute cost per token is only **~4.25M active parameters**!
- **Muon Optimizer**: Newton-Schulz 5th-order orthogonalization for 1.5x-2x faster training convergence.
- **Strict Weight Tying**: Input embedding and output LM head share weights.
---
## 📊 Model Specifications
| Hyperparameter | Value | Description |
|---|---|---|
| **Layers (Depth)** | **32** | 32 Sequential Thinking Steps per Token |
| **Total Model Parameters** | **8,964,736 (~8.96M)** | Full knowledge capacity across all 32 layers & experts |
| **Active Parameters per Token** | **4,246,144 (~4.25M)** | 1 Shared + 2 Routed Experts executed per layer |
| **Compute Sparsity** | **52.6%** | Over half the expert parameters bypassed per token |
| **Memory Footprint (FP32)** | **~34.20 MB** | Extremely compact for a 32-layer LLM |
| **Hidden Size ($d_{\text{model}}$)** | **128** | Backbone representation dimension |
| **Attention Mechanism** | **DeepSeek MLA** | Multi-Head Latent Attention |
| **Query LoRA Rank ($q_{\text{lora}}$)** | **48** | Low-rank Query compression |
| **KV LoRA Rank ($kv_{\text{lora}}$)** | **32** | Low-rank Joint KV compression |
| **Shared Experts** | **1 per layer** | Preserves core grammatical and semantic structures |
| **Routed Experts** | **8 per layer** | Specialized fine-grained subnetworks |
| **Top-K Routing** | **2 per layer** | Dynamic sparse gating |
| **Expert FFN Dimension** | **64** | SwiGLU mini-expert hidden dimension |
| **Vocabulary** | **4,096** | Custom BPE with Weight Tying |
| **Context Length** | **512 tokens** | Supported sequence length |
| **Residual Architecture** | **mHC** | Learnable Hyper-Connection residual scaling |
| **Training Optimizer** | **Muon + AdamW** | Newton-Schulz matrix polar decomposition |
---
## 🚀 Quick Start
```python
import torch
from viuai.model.config import ViuAIConfig
from viuai.model.viuai_model import ViuAIForCausalLM
# Load 32-layer model
config = ViuAIConfig(num_layers=32)
model = ViuAIForCausalLM(config)
print(f"Total Parameters: {model.count_parameters():,}")
# Output: 8,964,736 (~8.96M)
print(f"Active per Token: {model.count_active_parameters():,}")
# Output: 4,246,144 (~4.25M)
```
---
**Built with ❤️ by [ViuAI](https://huggingface.co/ViuAI)**
|