---
license: apache-2.0
tags:
- pytorch
- causal-lm
- mixture-of-experts
- from-scratch
- intelligence-os
pipeline_tag: text-generation
---

# Intelligence_OS (1.5 Billion Parameter Sparse Architecture)

Intelligence_OS is a decoder-only sparse mixture-of-experts language model designed from scratch for maximum quality per joule and compute-efficient training.

## Model Details
- **Architecture**: Decoder-only Sparse MoE
- **Parameters**: 1,518,659,584 (theoretical 1.519B class)
- **Layers**: 26
- **Hidden Dimension**: 2,048
- **Attention**: Grouped-Query Attention (16 Q heads / 4 KV heads)
- **Position Encoding**: Rotary Position Embedding (RoPE)
- **Feed-Forward**: 4 SwiGLU Experts per layer with Top-2 and Adaptive Routing
- **Embeddings**: Tied input/output token embeddings

## Training Innovations
- **Low-Memory Optimization (LOMO)**: Layer-wise in-place updates keeping peak VRAM under 1.5 GB.
- **Zero-GPU Non-Neural AI Engine**: Holographic Associative Memory capable of 85k tokens/s.