---
license: apache-2.0
language:
  - en
tags:
  - text-generation
  - transformer
  - 32-layers
  - deepseek-v4
  - kimi
  - mla
  - moe
  - muon-optimizer
  - hyper-connections
  - from-scratch
  - research
  - viuai
library_name: pytorch
pipeline_tag: text-generation
model-index:
  - name: ViuAI-UltraDeep-32L
    results: []
---

# 🚀 ViuAI-UltraDeep (32 Layers)

**An ultra-deep ~8.96M parameter MoE language model with DeepSeek-V4 MLA, Kimi Fine-Grained MoE, and mHC Hyper-Connections.**

ViuAI-UltraDeep pushes the limits of sequential reasoning by stacking **32 Transformer Blocks** with:
- **32 Reasoning Layers (Depth)**: Equivalent depth to frontier 8B models (Llama-3 has 32 layers).
- **Multi-Head Latent Attention (MLA)**: Low-rank Query & KV Joint Compression (80%-90% KV-cache reduction).
- **Fine-Grained MoE (DeepSeek / Kimi Style)**: 1 Always-Active Shared Expert + 8 Routed Experts (Top-2 active per token).
- **Manifold-Constrained Hyper-Connections (mHC)**: Learnable residual scale vectors per layer that preserve gradient fidelity across all 32 layers.
- **52.6% MoE Compute Sparsity**: Total capacity is ~8.96M parameters, but compute cost per token is only **~4.25M active parameters**!
- **Muon Optimizer**: Newton-Schulz 5th-order orthogonalization for 1.5x-2x faster training convergence.
- **Strict Weight Tying**: Input embedding and output LM head share weights.

---

## 📊 Model Specifications

| Hyperparameter | Value | Description |
|---|---|---|
| **Layers (Depth)** | **32** | 32 Sequential Thinking Steps per Token |
| **Total Model Parameters** | **8,964,736 (~8.96M)** | Full knowledge capacity across all 32 layers & experts |
| **Active Parameters per Token** | **4,246,144 (~4.25M)** | 1 Shared + 2 Routed Experts executed per layer |
| **Compute Sparsity** | **52.6%** | Over half the expert parameters bypassed per token |
| **Memory Footprint (FP32)** | **~34.20 MB** | Extremely compact for a 32-layer LLM |
| **Hidden Size ($d_{\text{model}}$)** | **128** | Backbone representation dimension |
| **Attention Mechanism** | **DeepSeek MLA** | Multi-Head Latent Attention |
| **Query LoRA Rank ($q_{\text{lora}}$)** | **48** | Low-rank Query compression |
| **KV LoRA Rank ($kv_{\text{lora}}$)** | **32** | Low-rank Joint KV compression |
| **Shared Experts** | **1 per layer** | Preserves core grammatical and semantic structures |
| **Routed Experts** | **8 per layer** | Specialized fine-grained subnetworks |
| **Top-K Routing** | **2 per layer** | Dynamic sparse gating |
| **Expert FFN Dimension** | **64** | SwiGLU mini-expert hidden dimension |
| **Vocabulary** | **4,096** | Custom BPE with Weight Tying |
| **Context Length** | **512 tokens** | Supported sequence length |
| **Residual Architecture** | **mHC** | Learnable Hyper-Connection residual scaling |
| **Training Optimizer** | **Muon + AdamW** | Newton-Schulz matrix polar decomposition |

---

## 🚀 Quick Start

```python
import torch
from viuai.model.config import ViuAIConfig
from viuai.model.viuai_model import ViuAIForCausalLM

# Load 32-layer model
config = ViuAIConfig(num_layers=32)
model = ViuAIForCausalLM(config)

print(f"Total Parameters:  {model.count_parameters():,}")
# Output: 8,964,736 (~8.96M)

print(f"Active per Token:  {model.count_active_parameters():,}")
# Output: 4,246,144 (~4.25M)
```

---

**Built with ❤️ by [ViuAI](https://huggingface.co/ViuAI)**