| license: apache-2.0 | |
| tags: | |
| - pytorch | |
| - causal-lm | |
| - mixture-of-experts | |
| - from-scratch | |
| - intelligence-os | |
| pipeline_tag: text-generation | |
| # Intelligence_OS (1.5 Billion Parameter Sparse Architecture) | |
| Intelligence_OS is a decoder-only sparse mixture-of-experts language model designed from scratch for maximum quality per joule and compute-efficient training. | |
| ## Model Details | |
| - **Architecture**: Decoder-only Sparse MoE | |
| - **Parameters**: 1,518,659,584 (theoretical 1.519B class) | |
| - **Layers**: 26 | |
| - **Hidden Dimension**: 2,048 | |
| - **Attention**: Grouped-Query Attention (16 Q heads / 4 KV heads) | |
| - **Position Encoding**: Rotary Position Embedding (RoPE) | |
| - **Feed-Forward**: 4 SwiGLU Experts per layer with Top-2 and Adaptive Routing | |
| - **Embeddings**: Tied input/output token embeddings | |
| ## Training Innovations | |
| - **Low-Memory Optimization (LOMO)**: Layer-wise in-place updates keeping peak VRAM under 1.5 GB. | |
| - **Zero-GPU Non-Neural AI Engine**: Holographic Associative Memory capable of 85k tokens/s. | |