| --- |
| license: mit |
| tags: |
| - sparseflow |
| - sparse-attention |
| - efficient-nlp |
| datasets: |
| - gsm8k |
| - lighteval/MATH |
| - allenai/ai2_arc |
| - tau/commonsense_qa |
| - piqa |
| - allenai/sciq |
| - trivia_qa |
| - nq_open |
| - wikitext |
| --- |
| |
| # SparseFlow v8 |
|
|
| Efficient language model with **sparse attention** and **persistent memory**. |
|
|
| ## 📊 REAL Measured Metrics |
|
|
| | Metric | Value | |
| |--------|-------| |
| | Parameters | 71,359,746 | |
| | Perplexity | 14.77 | |
| | Attention Sparsity | 87.5% | |
| | Channel Sparsity | 75.0% | |
| | Peak Memory | 3.67 GB | |
|
|
| ## 🏗️ Architecture |
|
|
| - **Sparse Token Attention**: Attends to top-64 tokens per position |
| - **Sparse Channel FFN**: Activates top-128 channels |
| - **Persistent Memory**: 20,000 memory vectors |
| - **8 Transformer layers** with 512 dim |
|
|
| ## 📚 Training Data |
|
|
| Open source datasets only: |
| - GSM8K, MATH (mathematics) |
| - ARC, OpenBookQA, SciQ (science & reasoning) |
| - CommonsenseQA, PIQA (common sense) |
| - TriviaQA, Natural Questions (factual) |
| - WikiText-103 (language modeling) |
|
|
| ## 👨💻 Author |
|
|
| **Logo (Mike Amega)** — Ame Web Studio |
|
|
