Blame for README.md

---
base_model: Qwen/Qwen3-Coder-Next
base_model_relation: quantized
quantized_by: 6block
license: apache-2.0
license_link: https://huggingface.co/Qwen/Qwen3-Coder-Next/blob/main/LICENSE
language:
- en
- zh
pipeline_tag: text-generation
tags:
- compressed-tensors
- awq
- w4a16
- vllm
- qwen3-next
- moe
- code
- quantized
---

# Qwen3-Coder-Next-AWQ

AWQ (activation-aware weight quantization) (W4A16) quantization of
[Qwen/Qwen3-Coder-Next](https://huggingface.co/Qwen/Qwen3-Coder-Next), made with
[llm-compressor](https://github.com/vllm-project/llm-compressor) in the **compressed-tensors** format
for efficient serving with vLLM and SGLang.

Chinese version: [README_zh.md](https://huggingface.co/6block/Qwen3-Coder-Next-AWQ/blob/main/README_zh.md)

## Why W4A16 (and not the official FP8)

Qwen publishes only an FP8 checkpoint for this model. This repo is the **4-bit (W4A16)** build:
**~40 GiB** on disk versus ~159 GiB for the bf16 master (≈4× smaller) and about half the size of the
FP8 release. That brings the 80B-A3B coder onto a single 24–48 GB GPU while the MoE routing is left
untouched (see the protected layers below), so generation tracks the base model.

Weights only are quantized — activations stay 16-bit (W4A16), which is what keeps quality close to the
original on a sparse MoE where the wrong rounding on a router would cascade.

## Model

| | |
|---|---|
| Base model | [Qwen/Qwen3-Coder-Next](https://huggingface.co/Qwen/Qwen3-Coder-Next) |
| Architecture | `qwen3_next` MoE — 80B total, ~3B active, 48 layers, 512 experts, top-10 routing |
| Context | 256K |
| Method | AWQ (activation-aware weight quantization), W4A16, group size 128 |
| Format | compressed-tensors (safetensors) |
| Size | ~40 GiB |
| Tool | [llm-compressor](https://github.com/vllm-project/llm-compressor) |
| License | Apache-2.0 (inherited from base model) |

## What is quantized, and what is protected

AWQ scales each weight channel by its activation importance before 4-bit rounding, so the quantization error lands on the channels that matter least. It was calibrated on a 256-sample mixed code / English / Chinese corpus (2048 tokens
each). On a 512-expert MoE the routing tensors are the fragile part, so they are kept at full precision:

| Tensors | Kept | Reason |
|---|---|---|
| `*.mlp.gate` (router) | bf16 | a compressed router sends tokens to the wrong experts |
| `lm_head` | bf16 | the output projection; a global 4-bit type would squeeze it hardest |
| everything else (expert + attention linears) | **W4A16** | the bulk of the weights, where 4-bit pays off |

## Quality (perplexity)

Perplexity on wikitext-2-raw (test), `n_ctx = 512`, 12 chunks, measured with vLLM — the same recipe
across all three, so the numbers are directly comparable. Lower is better; the bf16 row is the ceiling.

| Weights | PPL | vs bf16 |
|---|---|---|
| bf16 (master, reference) | 7.74 | — |
| **this repo (AWQ W4A16)** | **12.53** | +61.8% |

## Serving

```bash
vllm serve 6block/Qwen3-Coder-Next-AWQ
```

The quantization scheme is declared in `config.json` and detected automatically — no extra flags. Works
with any engine that reads compressed-tensors (vLLM, SGLang). At ~40 GiB a single 48 GB GPU serves it
comfortably; on 24 GB use tensor/pipeline parallel or offload.

## Prompt format

```
<|im_start|>system
{system_prompt}<|im_end|>
<|im_start|>user
{prompt}<|im_end|>
<|im_start|>assistant
```

## Other quantizations

| Format | Repo | Note |
|---|---|---|
| GPTQ (W4A16) | [6block/Qwen3-Coder-Next-GPTQ](https://huggingface.co/6block/Qwen3-Coder-Next-GPTQ) | the other 4-bit build |
| FP8 (official) | [Qwen/Qwen3-Coder-Next-FP8](https://huggingface.co/Qwen/Qwen3-Coder-Next-FP8) | 8-bit, ~2× this size |

---

*Quantized by 6block with llm-compressor. Weights are unmodified apart from quantization; all model
behavior and license obligations follow the base model.*