File size: 3,522 Bytes
0d31479 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 | ---
license: apache-2.0
language:
- en
tags:
- text-generation
- transformer
- 32-layers
- deepseek-v4
- kimi
- mla
- moe
- muon-optimizer
- hyper-connections
- from-scratch
- research
- viuai
library_name: pytorch
pipeline_tag: text-generation
model-index:
- name: ViuAI-10M
results: []
---
# 🚀 ViuAI-10M (UltraDeep 32-Layer Architecture)
**An ultra-deep ~8.96M (~10M class) parameter MoE language model built from scratch with DeepSeek-V4 MLA, Kimi Fine-Grained MoE, and mHC Hyper-Connections.**
ViuAI-10M pushes the boundaries of efficient deep learning by stacking **32 Transformer Blocks** with:
- **32 Reasoning Layers (Depth)**: Equivalent depth to frontier 8B models (Llama-3 has 32 layers).
- **Multi-Head Latent Attention (MLA)**: Low-rank Query & KV Joint Compression (80%-90% KV-cache reduction).
- **Fine-Grained MoE (DeepSeek / Kimi Style)**: 1 Always-Active Shared Expert + 8 Routed Experts (Top-2 active per token).
- **Manifold-Constrained Hyper-Connections (mHC)**: Learnable residual scale vectors per layer that preserve gradient fidelity across all 32 layers.
- **52.6% MoE Compute Sparsity**: Total capacity is ~8.96M parameters, but compute cost per token is only **~4.25M active parameters**!
- **Muon Optimizer**: Newton-Schulz 5th-order orthogonalization for 1.5x-2x faster training convergence.
- **Strict Weight Tying**: Input embedding and output LM head share weights.
---
## 📊 Model Specifications
| Hyperparameter | Value | Description |
|---|---|---|
| **Layers (Depth)** | **32** | 32 Sequential Thinking Steps per Token |
| **Total Model Parameters** | **8,964,736 (~8.96M)** | Full knowledge capacity across all 32 layers & experts |
| **Active Parameters per Token** | **4,246,144 (~4.25M)** | 1 Shared + 2 Routed Experts executed per layer |
| **Compute Sparsity** | **52.6%** | Over half the expert parameters bypassed per token |
| **Memory Footprint (FP32)** | **~34.20 MB** | Extremely compact for a 32-layer LLM |
| **Hidden Size ($d_{\text{model}}$)** | **128** | Backbone representation dimension |
| **Attention Mechanism** | **DeepSeek MLA** | Multi-Head Latent Attention |
| **Query LoRA Rank ($q_{\text{lora}}$)** | **48** | Low-rank Query compression |
| **KV LoRA Rank ($kv_{\text{lora}}$)** | **32** | Low-rank Joint KV compression |
| **Shared Experts** | **1 per layer** | Preserves core grammatical and semantic structures |
| **Routed Experts** | **8 per layer** | Specialized fine-grained subnetworks |
| **Top-K Routing** | **2 per layer** | Dynamic sparse gating |
| **Expert FFN Dimension** | **64** | SwiGLU mini-expert hidden dimension |
| **Vocabulary** | **4,096** | Custom BPE with Weight Tying |
| **Context Length** | **512 tokens** | Supported sequence length |
| **Residual Architecture** | **mHC** | Learnable Hyper-Connection residual scaling |
| **Training Optimizer** | **Muon + AdamW** | Newton-Schulz matrix polar decomposition |
---
## 🚀 Quick Start
```python
import torch
from viuai.model.config import ViuAIConfig
from viuai.model.viuai_model import ViuAIForCausalLM
# Load 10M model
config = ViuAIConfig(num_layers=32)
model = ViuAIForCausalLM(config)
print(f"Total Parameters: {model.count_parameters():,}")
# Output: 8,964,736 (~8.96M)
print(f"Active per Token: {model.count_active_parameters():,}")
# Output: 4,246,144 (~4.25M)
```
---
**Built with ❤️ by [ViuAI](https://huggingface.co/ViuAI)**
|