| --- |
| base_model: Qwen/Qwen3-Coder-Next |
| base_model_relation: quantized |
| quantized_by: 6block |
| license: apache-2.0 |
| license_link: https://huggingface.co/Qwen/Qwen3-Coder-Next/blob/main/LICENSE |
| language: |
| - en |
| - zh |
| pipeline_tag: text-generation |
| tags: |
| - compressed-tensors |
| - awq |
| - w4a16 |
| - vllm |
| - qwen3-next |
| - moe |
| - code |
| - quantized |
| --- |
| |
| # Qwen3-Coder-Next-AWQ |
|
|
| AWQ (activation-aware weight quantization) (W4A16) quantization of |
| [Qwen/Qwen3-Coder-Next](https://huggingface.co/Qwen/Qwen3-Coder-Next), made with |
| [llm-compressor](https://github.com/vllm-project/llm-compressor) in the **compressed-tensors** format |
| for efficient serving with vLLM and SGLang. |
|
|
| Chinese version: [README_zh.md](https://huggingface.co/6block/Qwen3-Coder-Next-AWQ/blob/main/README_zh.md) |
|
|
| ## Why W4A16 (and not the official FP8) |
|
|
| Qwen publishes only an FP8 checkpoint for this model. This repo is the **4-bit (W4A16)** build: |
| **~40 GiB** on disk versus ~159 GiB for the bf16 master (≈4× smaller) and about half the size of the |
| FP8 release. That brings the 80B-A3B coder onto a single 24–48 GB GPU while the MoE routing is left |
| untouched (see the protected layers below), so generation tracks the base model. |
|
|
| Weights only are quantized — activations stay 16-bit (W4A16), which is what keeps quality close to the |
| original on a sparse MoE where the wrong rounding on a router would cascade. |
|
|
| ## Model |
|
|
| | | | |
| |---|---| |
| | Base model | [Qwen/Qwen3-Coder-Next](https://huggingface.co/Qwen/Qwen3-Coder-Next) | |
| | Architecture | `qwen3_next` MoE — 80B total, ~3B active, 48 layers, 512 experts, top-10 routing | |
| | Context | 256K | |
| | Method | AWQ (activation-aware weight quantization), W4A16, group size 128 | |
| | Format | compressed-tensors (safetensors) | |
| | Size | ~40 GiB | |
| | Tool | [llm-compressor](https://github.com/vllm-project/llm-compressor) | |
| | License | Apache-2.0 (inherited from base model) | |
|
|
| ## What is quantized, and what is protected |
|
|
| AWQ scales each weight channel by its activation importance before 4-bit rounding, so the quantization error lands on the channels that matter least. It was calibrated on a 256-sample mixed code / English / Chinese corpus (2048 tokens |
| each). On a 512-expert MoE the routing tensors are the fragile part, so they are kept at full precision: |
|
|
| | Tensors | Kept | Reason | |
| |---|---|---| |
| | `*.mlp.gate` (router) | bf16 | a compressed router sends tokens to the wrong experts | |
| | `lm_head` | bf16 | the output projection; a global 4-bit type would squeeze it hardest | |
| | everything else (expert + attention linears) | **W4A16** | the bulk of the weights, where 4-bit pays off | |
|
|
| ## Quality (perplexity) |
|
|
| Perplexity on wikitext-2-raw (test), `n_ctx = 512`, 12 chunks, measured with vLLM — the same recipe |
| across all three, so the numbers are directly comparable. Lower is better; the bf16 row is the ceiling. |
|
|
| | Weights | PPL | vs bf16 | |
| |---|---|---| |
| | bf16 (master, reference) | 7.74 | — | |
| | **this repo (AWQ W4A16)** | **12.53** | +61.8% | |
|
|
| ## Serving |
|
|
| ```bash |
| vllm serve 6block/Qwen3-Coder-Next-AWQ |
| ``` |
|
|
| The quantization scheme is declared in `config.json` and detected automatically — no extra flags. Works |
| with any engine that reads compressed-tensors (vLLM, SGLang). At ~40 GiB a single 48 GB GPU serves it |
| comfortably; on 24 GB use tensor/pipeline parallel or offload. |
|
|
| ## Prompt format |
|
|
| ``` |
| <|im_start|>system |
| {system_prompt}<|im_end|> |
| <|im_start|>user |
| {prompt}<|im_end|> |
| <|im_start|>assistant |
| ``` |
|
|
| ## Other quantizations |
|
|
| | Format | Repo | Note | |
| |---|---|---| |
| | GPTQ (W4A16) | [6block/Qwen3-Coder-Next-GPTQ](https://huggingface.co/6block/Qwen3-Coder-Next-GPTQ) | the other 4-bit build | |
| | FP8 (official) | [Qwen/Qwen3-Coder-Next-FP8](https://huggingface.co/Qwen/Qwen3-Coder-Next-FP8) | 8-bit, ~2× this size | |
|
|
| --- |
|
|
| *Quantized by 6block with llm-compressor. Weights are unmodified apart from quantization; all model |
| behavior and license obligations follow the base model.* |
|
|
