YAML Metadata Warning:empty or missing yaml metadata in repo card
Check out the documentation for more information.
808 AI
A modern 150M-parameter decoder-only language model.
Engineered by Sadat Sami · Released under the Apache License 2.0
Overview
808 AI is an open-source language model designed and engineered by Sadat Sami as part of his research into efficient, modern language model architectures. It is a compact decoder-only transformer built for fine-tuning on conversational and instruction-following tasks, and can be trained end-to-end on a single workstation GPU.
Architectural Highlights
- Grouped Query Attention (GQA) — 12 query heads, 4 key/value heads.
- Query-Key Normalization (QK-Norm) — RMSNorm on q/k heads before RoPE.
- Layer-wise Sliding Window Attention — alternating local/global pattern.
- Rotary Position Embeddings (RoPE) — with optional linear scaling.
- SwiGLU Feed-Forward Networks.
- Pre-norm Residual Topology.
- Tied Input/Output Embeddings.
- FlashAttention-2 Ready via PyTorch SDPA.
Model Size
| Component | Value |
|---|---|
| Parameters | ~150M |
| Hidden size | 768 |
| Layers | 20 |
| Query heads | 12 |
| Key/value heads | 4 |
| Head dimension | 64 |
| Intermediate size | 2048 |
| Vocabulary size | 32,000 |
| Max position | 4096 |
Installation
pip install torch transformers
Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support