README.md

metadata

license: apache-2.0
language:
  - en
tags:
  - text-generation
  - transformer
  - 32-layers
  - deepseek-v4
  - kimi
  - mla
  - moe
  - muon-optimizer
  - hyper-connections
  - from-scratch
  - research
  - viuai
library_name: pytorch
pipeline_tag: text-generation
model-index:
  - name: ViuAI-10M
    results: []

🚀 ViuAI-10M (UltraDeep 32-Layer Architecture)

An ultra-deep 8.96M (10M class) parameter MoE language model built from scratch with DeepSeek-V4 MLA, Kimi Fine-Grained MoE, and mHC Hyper-Connections.

ViuAI-10M pushes the boundaries of efficient deep learning by stacking 32 Transformer Blocks with:

• 32 Reasoning Layers (Depth): Equivalent depth to frontier 8B models (Llama-3 has 32 layers).
• Multi-Head Latent Attention (MLA): Low-rank Query & KV Joint Compression (80%-90% KV-cache reduction).
• Fine-Grained MoE (DeepSeek / Kimi Style): 1 Always-Active Shared Expert + 8 Routed Experts (Top-2 active per token).
• Manifold-Constrained Hyper-Connections (mHC): Learnable residual scale vectors per layer that preserve gradient fidelity across all 32 layers.
• 52.6% MoE Compute Sparsity: Total capacity is 8.96M parameters, but compute cost per token is only **4.25M active parameters**!
• Muon Optimizer: Newton-Schulz 5th-order orthogonalization for 1.5x-2x faster training convergence.
• Strict Weight Tying: Input embedding and output LM head share weights.

---

📊 Model Specifications

Hyperparameter	Value	Description
Layers (Depth)	32	32 Sequential Thinking Steps per Token
Total Model Parameters	8,964,736 (~8.96M)	Full knowledge capacity across all 32 layers & experts
Active Parameters per Token	4,246,144 (~4.25M)	1 Shared + 2 Routed Experts executed per layer
Compute Sparsity	52.6%	Over half the expert parameters bypassed per token
Memory Footprint (FP32)	~34.20 MB	Extremely compact for a 32-layer LLM
Hidden Size ($d_{\text{model}}$)	128	Backbone representation dimension
Attention Mechanism	DeepSeek MLA	Multi-Head Latent Attention
Query LoRA Rank ($q_{\text{lora}}$)	48	Low-rank Query compression
KV LoRA Rank ($kv_{\text{lora}}$)	32	Low-rank Joint KV compression
Shared Experts	1 per layer	Preserves core grammatical and semantic structures
Routed Experts	8 per layer	Specialized fine-grained subnetworks
Top-K Routing	2 per layer	Dynamic sparse gating
Expert FFN Dimension	64	SwiGLU mini-expert hidden dimension
Vocabulary	4,096	Custom BPE with Weight Tying
Context Length	512 tokens	Supported sequence length
Residual Architecture	mHC	Learnable Hyper-Connection residual scaling
Training Optimizer	Muon + AdamW	Newton-Schulz matrix polar decomposition

---

🚀 Quick Start

import torch
from viuai.model.config import ViuAIConfig
from viuai.model.viuai_model import ViuAIForCausalLM

# Load 10M model
config = ViuAIConfig(num_layers=32)
model = ViuAIForCausalLM(config)

print(f"Total Parameters:  {model.count_parameters():,}")
# Output: 8,964,736 (~8.96M)

print(f"Active per Token:  {model.count_active_parameters():,}")
# Output: 4,246,144 (~4.25M)

---

Built with ❤️ by ViuAI