Upload README.md with huggingface_hub
Browse files
README.md
ADDED
|
@@ -0,0 +1,73 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
---
|
| 2 |
+
license: mit
|
| 3 |
+
library_name: transformers
|
| 4 |
+
tags:
|
| 5 |
+
- vector-quantization
|
| 6 |
+
- vq2
|
| 7 |
+
- moe
|
| 8 |
+
- qwen
|
| 9 |
+
- compression
|
| 10 |
+
- triton
|
| 11 |
+
- paged-streaming
|
| 12 |
+
pipeline_tag: text-generation
|
| 13 |
+
---
|
| 14 |
+
|
| 15 |
+
# ⚡ VQ2 & LEGION AI: Model Hub
|
| 16 |
+
|
| 17 |
+
Official repository for compiled binary containers of **VQ2** and **LEGION AI**.
|
| 18 |
+
|
| 19 |
+
* **GitHub Code & Docs**: [https://github.com/Unoalss/CompressAI](https://github.com/Unoalss/CompressAI)
|
| 20 |
+
* **Architecture**: Vector Quantization with Incoherence (RHT), Hessian LDLQ compensation, and Paged MoE Streaming.
|
| 21 |
+
|
| 22 |
+
---
|
| 23 |
+
|
| 24 |
+
## 📦 Hosted Models
|
| 25 |
+
|
| 26 |
+
### 1. 🏆 `qwen35b_mixed_b2.vq2m` (Production Candidate MoE - 10.11 GB)
|
| 27 |
+
* **Base Model**: `Qwen3.8-35B-A3B-Distill` (35.5B parameters, 40 layers, 256 experts per layer).
|
| 28 |
+
* **Compression**: 86.0% smaller than BF16 (67 GB -> 10.11 GB / 9.41 GiB).
|
| 29 |
+
* **Quality**: **0.9974 cosine similarity** and **90.6% top-1 agreement** vs dense BF16.
|
| 30 |
+
* **Memory**: Runs in **< 1.25 GB active VRAM** and **~3.0 GB RAM** via LEGION Paged Streaming.
|
| 31 |
+
* **SHA-256**: `2ADB36E37F387842A521D2F3E07A29B77A7D9240997411372E0975586AEB8EAE`
|
| 32 |
+
|
| 33 |
+
### 2. ⚡ `qwen3.5_2b_vq2_560mb.vq2r` (Ultra-compact 2B RDO - 567.9 MB)
|
| 34 |
+
* **Base Model**: `Qwen3.5-2B` (2 Billion parameters).
|
| 35 |
+
* **Compression**: 53.5% smaller than GGUF Q4_K_M (1221 MB -> 567 MB) and 84.2% smaller than BF16.
|
| 36 |
+
* **Memory**: Loads into GPU VRAM in **0.61 seconds**, occupying only **582 MiB** of active VRAM.
|
| 37 |
+
|
| 38 |
+
### 3. 🔑 `codebooks.pt` (4.47 MB)
|
| 39 |
+
* Shared 8-dimensional spherical codebooks for on-the-fly decompression.
|
| 40 |
+
|
| 41 |
+
---
|
| 42 |
+
|
| 43 |
+
## 📥 How to Download & Run
|
| 44 |
+
|
| 45 |
+
Clone the engine from GitHub:
|
| 46 |
+
```bash
|
| 47 |
+
git clone https://github.com/Unoalss/CompressAI.git
|
| 48 |
+
cd CompressAI
|
| 49 |
+
pip install -r requirements.txt
|
| 50 |
+
pip install -e .
|
| 51 |
+
```
|
| 52 |
+
|
| 53 |
+
Download the models:
|
| 54 |
+
```bash
|
| 55 |
+
# MoE 35B B2 Model (10 GB)
|
| 56 |
+
huggingface-cli download ArthurLss/CompressAI qwen35b_mixed_b2.vq2m --local-dir vq2_35b_out
|
| 57 |
+
|
| 58 |
+
# 2B Model (567 MB)
|
| 59 |
+
huggingface-cli download ArthurLss/CompressAI qwen3.5_2b_vq2_560mb.vq2r --local-dir VQ2_MARCO_FINAL
|
| 60 |
+
|
| 61 |
+
# Shared Codebooks (4.47 MB)
|
| 62 |
+
huggingface-cli download ArthurLss/CompressAI codebooks.pt --local-dir vq2_out
|
| 63 |
+
```
|
| 64 |
+
|
| 65 |
+
Launch the LEGION chatbot interface:
|
| 66 |
+
```bash
|
| 67 |
+
# Windows
|
| 68 |
+
INICIAR_LEGION.bat
|
| 69 |
+
|
| 70 |
+
# Linux / Mac
|
| 71 |
+
python legion/server.py
|
| 72 |
+
```
|
| 73 |
+
Open `http://127.0.0.1:7777` in your browser!
|