metadata
license: apache-2.0
tags:
- pytorch
- causal-lm
- mixture-of-experts
- from-scratch
- intelligence-os
pipeline_tag: text-generation
Intelligence_OS (1.5 Billion Parameter Sparse Architecture)
Intelligence_OS is a decoder-only sparse mixture-of-experts language model designed from scratch for maximum quality per joule and compute-efficient training.
Model Details
- Architecture: Decoder-only Sparse MoE
- Parameters: 1,518,659,584 (theoretical 1.519B class)
- Layers: 26
- Hidden Dimension: 2,048
- Attention: Grouped-Query Attention (16 Q heads / 4 KV heads)
- Position Encoding: Rotary Position Embedding (RoPE)
- Feed-Forward: 4 SwiGLU Experts per layer with Top-2 and Adaptive Routing
- Embeddings: Tied input/output token embeddings
Training Innovations
- Low-Memory Optimization (LOMO): Layer-wise in-place updates keeping peak VRAM under 1.5 GB.
- Zero-GPU Non-Neural AI Engine: Holographic Associative Memory capable of 85k tokens/s.