| --- |
| language: |
| - hi |
| - en |
| license: apache-2.0 |
| tags: |
| - dense |
| - transformer |
| - indic |
| - hindi |
| - hinglish |
| - english |
| - gqa |
| - gemma-4 |
| - pytorch |
| - text-generation |
| pipeline_tag: text-generation |
| --- |
| |
| # ViuMini-Dense-360M (Sovereign 32-Layer Dense Transformer) |
|
|
| **ViuMini-Dense-360M** is a sovereign Indian Small Language Model (SLM) featuring a **32-Layer Deep Dense Transformer** architecture optimized for hierarchical multi-step reasoning, Indic multilingual fluency (Hindi, Hinglish, English), and high-speed on-device inference (mobile phones, laptops, and consumer GPUs). |
|
|
| Unlike sparse MoE models where routing instability and inactive parameters limit depth and reasoning density, **ViuMini-Dense-360M** dedicates **100% of its 366.6 Million parameters** to every single token, delivering 2.2× higher active cognitive capacity per forward pass with rock-solid gradient stability. |
|
|
| --- |
|
|
| ## 🏛️ Architecture Specifications |
|
|
| | Specification | Value | Technical Rationale | |
| |---|---|---| |
| | **Total Parameters** | **366,617,536 (~366.6M)** | Exact SmolLM-360M blueprint scaled for Indic | |
| | **Active Parameters / Token** | **366.6M (100% Active)** | No routing overhead, no expert collapse | |
| | **Layers (`n_layers`)** | **32 Deep Layers** | Ultra-deep hierarchical reasoning abstraction | |
| | **Hidden Dimension (`dim`)** | **960** | Optimal representation width for SLM | |
| | **Attention Heads (`n_heads`)** | **15 Query Heads** | Head dimension $d_{\text{head}} = 64$ ($15 \times 64 = 960$) | |
| | **KV Heads (`n_kv_heads`)** | **5 KV Heads (GQA)** | 3:1 Grouped-Query Attention saving 66% KV cache | |
| | **Feed-Forward Dimension (`hidden_dim`)**| **2,624** | SwiGLU MLP ($h = 64 \times \lceil 2592/64 \rceil$, `mult=2.7`, `multiple_of=64`) | |
| | **Vocabulary Size** | **48,000 Tokens** | Custom BPE optimized for Devanagari, Hinglish & Code | |
| | **Embeddings** | **Weight-Tied** | Input embeddings tied to LM output head (saves 46M params) | |
| | **Attention Stability** | **Gemma 4 Pure QK-Norm** | RMSNorm on Q and K; zero gradient saturation | |
| | **Positional Encoding** | **Rotary RoPE ($\theta = 500,000$)** | High-frequency positional precision, long-context ready | |
| | **Context Length (`max_seq_len`)** | **2,048 Tokens** | 3-Stage Curriculum: $512 \rightarrow 1,024 \rightarrow 2,048$ tokens | |
| | **Optimizer** | **PagedAdamW8bit** | Stable, decoupled weight decay, zero OOM spikes | |
| | **Learning Rate** | **$2.5 \times 10^{-4}$ (Cosine)** | 1,000 warmup steps, grad clip 1.0 | |
|
|
| --- |
|
|
| ## 🚀 Why 32-Layer Dense beats 242M MoE |
|
|
| 1. **2.2× Higher Cognitive Capacity per Token**: |
| - Old `ViuMini-MoE-242M` only activated ~166M parameters per token (with 76M parameters sitting idle). |
| - **ViuMini-Dense-360M activates all 366.6M parameters on every token**. |
| 2. **32 Layers of Hierarchical Reasoning**: |
| - 32 sequential layers provide deep multi-step deduction, grammar parsing, and factual recall compared to shallower configurations. |
| 3. **Rock-Solid Training Stability**: |
| - MoE router collapse (where routers get stuck picking the same 1–2 experts) and auxiliary loss fighting are completely eliminated. |
| - PagedAdamW8bit with $2.5 \times 10^{-4}$ learning rate and Gemma 4 QK-Norm guarantees clean loss convergence from the initial theoretical loss ($\ln(48,000) \approx 10.78$). |
| 4. **Zero-Friction Deployment (1-Click GGUF / Ollama)**: |
| - Dense models run natively in `llama.cpp`, Ollama, vLLM, MLC-LLM, and LM Studio without custom MoE routing kernels. |
| - 4-bit quantized footprint is under **220 MB RAM**, running at **100+ tokens/second** directly on mobile devices and laptops. |
|
|
| --- |
|
|
| ## 📚 Pretraining Dataset |
|
|
| - **Dataset Repo**: `ViuAI/viu-mini-raw-pretrain` |
| - **Total Volume**: **126.5B+ authentic tokens** across **2,109 verified Parquet files** |
| - **Domain Coverage**: |
| - *Hindi & Devanagari*: Valmiki Ramayana, Mahabharata, 16 Mahapuranas, 4 Vedas, Hindi literature & news. |
| - *Hinglish & Colloquial*: Bollywood cinema screenplays, conversational dialogues, social chat. |
| - *Reasoning & Code*: DeepSeek-R1 math, Orca math, GSM8K, Python code & algorithms. |
| - *Indian Governance, Law & Finance*: Constitution of India, BNS/BNSS/BSA 2023 statutes, Supreme Court judgments, RBI circulars & financial news. |
| - *Parallel Translation*: AI4Bharat Samanantar (26 Million English $\leftrightarrow$ Hindi sentence pairs). |
|
|
| --- |
|
|
| ## ⚡ Kaggle Dual Tesla T4 Pretraining |
|
|
| Pretraining is fully automated and session-proof on Kaggle Dual Tesla T4 (2×16GB): |
|
|
| ```bash |
| # Clone the repository |
| git clone https://huggingface.co/ViuAI/ViuMini-Dense-360M /kaggle/working/ViuMini-Dense-360M |
| cd /kaggle/working/ViuMini-Dense-360M |
| |
| # Run pretraining using the dedicated Kaggle config |
| torchrun --nproc_per_node=2 model/scripts/train.py --config model/configs/train_kaggle_t4.yaml |
| ``` |
|
|
| - **Curriculum Batching**: Constant **65,536 tokens/step** across Dual T4: |
| - *Stage 1 (0–50k steps)*: seq 512, micro-batch 8 per GPU, accum 8 $\rightarrow$ $512 \times 8 \times 16 = 65,536$ tok/step |
| - *Stage 2 (50k–200k steps)*: seq 1024, micro-batch 4 per GPU, accum 8 $\rightarrow$ $1024 \times 4 \times 16 = 65,536$ tok/step |
| - *Stage 3 (200k+ steps)*: seq 2048, micro-batch 2 per GPU, accum 8 $\rightarrow$ $2048 \times 2 \times 16 = 65,536$ tok/step |
| - **Throughput & Cadence**: ~8,500 – 10,000 tokens/sec across Dual T4. |
| - Every **1,000 steps** $\approx$ **65.5 Million tokens** ($\approx$ **2.14 hours**), pushed directly to Hugging Face Hub. |
| - Full 126.5B dataset pretraining: $\approx$ 1,930,000 steps. |
| - **VRAM Utilization**: ~3.4 GB / 16 GB per T4 (12.6 GB safety headroom, zero OOM risk). |
|
|
| --- |
|
|
| ## 📁 Repository Structure |
|
|
| ``` |
| ViuMini-Dense-360M/ |
| ├── model/ |
| │ ├── configs/ |
| │ │ ├── viu1_dense_config.yaml # 32L Dense model architecture (366.6M params) |
| │ │ ├── train_kaggle_t4.yaml # Kaggle Dual T4 DDP training config |
| │ │ ├── train_config.yaml # Universal pretraining config |
| │ │ └── config.json # Hugging Face compatible architecture config |
| │ └── scripts/ |
| │ ├── viu1_dense.py # ViuMini Transformer engine (32L Dense + QK-Norm + GQA) |
| │ ├── train.py # Distributed DDP pretrainer with HF streaming & auto-push |
| │ └── push_to_hf.py # Hub sync utility |
| ├── tokenizer/ |
| │ ├── configs/ |
| │ └── outputs/ # Custom 48,000 Indic BPE tokenizer |
| ├── docs/ |
| │ ├── KAGGLE_RUN.md # Single-cell copy-paste Kaggle guide |
| │ └── PROGRESS.md # Development milestones & audit logs |
| └── README.md |
| ``` |
|
|
| --- |
|
|
| ## 📜 License |
| Apache-2.0. Sovereign AI initiative by ViuAI. |
|
|
