Upload README.md with huggingface_hub
Browse files
README.md
ADDED
|
@@ -0,0 +1,105 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
---
|
| 2 |
+
base_model: Qwen/Qwen3-Coder-Next
|
| 3 |
+
base_model_relation: quantized
|
| 4 |
+
quantized_by: 6block
|
| 5 |
+
license: apache-2.0
|
| 6 |
+
license_link: https://huggingface.co/Qwen/Qwen3-Coder-Next/blob/main/LICENSE
|
| 7 |
+
language:
|
| 8 |
+
- en
|
| 9 |
+
- zh
|
| 10 |
+
pipeline_tag: text-generation
|
| 11 |
+
tags:
|
| 12 |
+
- compressed-tensors
|
| 13 |
+
- gptq
|
| 14 |
+
- w4a16
|
| 15 |
+
- vllm
|
| 16 |
+
- qwen3-next
|
| 17 |
+
- moe
|
| 18 |
+
- code
|
| 19 |
+
- quantized
|
| 20 |
+
---
|
| 21 |
+
|
| 22 |
+
# Qwen3-Coder-Next-GPTQ
|
| 23 |
+
|
| 24 |
+
GPTQ (error-compensated weight quantization) (W4A16) quantization of
|
| 25 |
+
[Qwen/Qwen3-Coder-Next](https://huggingface.co/Qwen/Qwen3-Coder-Next), made with
|
| 26 |
+
[llm-compressor](https://github.com/vllm-project/llm-compressor) in the **compressed-tensors** format
|
| 27 |
+
for efficient serving with vLLM and SGLang.
|
| 28 |
+
|
| 29 |
+
Chinese version: [README_zh.md](https://huggingface.co/6block/Qwen3-Coder-Next-GPTQ/blob/main/README_zh.md)
|
| 30 |
+
|
| 31 |
+
## Why W4A16 (and not the official FP8)
|
| 32 |
+
|
| 33 |
+
Qwen publishes only an FP8 checkpoint for this model. This repo is the **4-bit (W4A16)** build:
|
| 34 |
+
**~40 GiB** on disk versus ~159 GiB for the bf16 master (≈4× smaller) and about half the size of the
|
| 35 |
+
FP8 release. That brings the 80B-A3B coder onto a single 24–48 GB GPU while the MoE routing is left
|
| 36 |
+
untouched (see the protected layers below), so generation tracks the base model.
|
| 37 |
+
|
| 38 |
+
Weights only are quantized — activations stay 16-bit (W4A16), which is what keeps quality close to the
|
| 39 |
+
original on a sparse MoE where the wrong rounding on a router would cascade.
|
| 40 |
+
|
| 41 |
+
## Model
|
| 42 |
+
|
| 43 |
+
| | |
|
| 44 |
+
|---|---|
|
| 45 |
+
| Base model | [Qwen/Qwen3-Coder-Next](https://huggingface.co/Qwen/Qwen3-Coder-Next) |
|
| 46 |
+
| Architecture | `qwen3_next` MoE — 80B total, ~3B active, 48 layers, 512 experts, top-10 routing |
|
| 47 |
+
| Context | 256K |
|
| 48 |
+
| Method | GPTQ (error-compensated weight quantization), W4A16, group size 128 |
|
| 49 |
+
| Format | compressed-tensors (safetensors) |
|
| 50 |
+
| Size | ~40 GiB |
|
| 51 |
+
| Tool | [llm-compressor](https://github.com/vllm-project/llm-compressor) |
|
| 52 |
+
| License | Apache-2.0 (inherited from base model) |
|
| 53 |
+
|
| 54 |
+
## What is quantized, and what is protected
|
| 55 |
+
|
| 56 |
+
GPTQ quantizes weights column-by-column and compensates the residual error into the not-yet-quantized columns using second-order (Hessian) information. It was calibrated on a 256-sample mixed code / English / Chinese corpus (2048 tokens
|
| 57 |
+
each). On a 512-expert MoE the routing tensors are the fragile part, so they are kept at full precision:
|
| 58 |
+
|
| 59 |
+
| Tensors | Kept | Reason |
|
| 60 |
+
|---|---|---|
|
| 61 |
+
| `*.mlp.gate` (router) | bf16 | a compressed router sends tokens to the wrong experts |
|
| 62 |
+
| `lm_head` | bf16 | the output projection; a global 4-bit type would squeeze it hardest |
|
| 63 |
+
| everything else (expert + attention linears) | **W4A16** | the bulk of the weights, where 4-bit pays off |
|
| 64 |
+
|
| 65 |
+
## Quality (perplexity)
|
| 66 |
+
|
| 67 |
+
Perplexity on wikitext-2-raw (test), `n_ctx = 512`, 12 chunks, measured with vLLM — the same recipe
|
| 68 |
+
across all three, so the numbers are directly comparable. Lower is better; the bf16 row is the ceiling.
|
| 69 |
+
|
| 70 |
+
| Weights | PPL | vs bf16 |
|
| 71 |
+
|---|---|---|
|
| 72 |
+
| bf16 (master, reference) | 7.74 | — |
|
| 73 |
+
| **this repo (GPTQ W4A16)** | **9.34** | +20.5% |
|
| 74 |
+
|
| 75 |
+
## Serving
|
| 76 |
+
|
| 77 |
+
```bash
|
| 78 |
+
vllm serve 6block/Qwen3-Coder-Next-GPTQ
|
| 79 |
+
```
|
| 80 |
+
|
| 81 |
+
The quantization scheme is declared in `config.json` and detected automatically — no extra flags. Works
|
| 82 |
+
with any engine that reads compressed-tensors (vLLM, SGLang). At ~40 GiB a single 48 GB GPU serves it
|
| 83 |
+
comfortably; on 24 GB use tensor/pipeline parallel or offload.
|
| 84 |
+
|
| 85 |
+
## Prompt format
|
| 86 |
+
|
| 87 |
+
```
|
| 88 |
+
<|im_start|>system
|
| 89 |
+
{system_prompt}<|im_end|>
|
| 90 |
+
<|im_start|>user
|
| 91 |
+
{prompt}<|im_end|>
|
| 92 |
+
<|im_start|>assistant
|
| 93 |
+
```
|
| 94 |
+
|
| 95 |
+
## Other quantizations
|
| 96 |
+
|
| 97 |
+
| Format | Repo | Note |
|
| 98 |
+
|---|---|---|
|
| 99 |
+
| AWQ (W4A16) | [6block/Qwen3-Coder-Next-AWQ](https://huggingface.co/6block/Qwen3-Coder-Next-AWQ) | the other 4-bit build |
|
| 100 |
+
| FP8 (official) | [Qwen/Qwen3-Coder-Next-FP8](https://huggingface.co/Qwen/Qwen3-Coder-Next-FP8) | 8-bit, ~2× this size |
|
| 101 |
+
|
| 102 |
+
---
|
| 103 |
+
|
| 104 |
+
*Quantized by 6block with llm-compressor. Weights are unmodified apart from quantization; all model
|
| 105 |
+
behavior and license obligations follow the base model.*
|