bonujel2 commited on
Commit
bc9cdab
·
verified ·
1 Parent(s): 37b8702

Upload README.md with huggingface_hub

Browse files
Files changed (1) hide show
  1. README.md +105 -0
README.md ADDED
@@ -0,0 +1,105 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ base_model: Qwen/Qwen3-Coder-Next
3
+ base_model_relation: quantized
4
+ quantized_by: 6block
5
+ license: apache-2.0
6
+ license_link: https://huggingface.co/Qwen/Qwen3-Coder-Next/blob/main/LICENSE
7
+ language:
8
+ - en
9
+ - zh
10
+ pipeline_tag: text-generation
11
+ tags:
12
+ - compressed-tensors
13
+ - gptq
14
+ - w4a16
15
+ - vllm
16
+ - qwen3-next
17
+ - moe
18
+ - code
19
+ - quantized
20
+ ---
21
+
22
+ # Qwen3-Coder-Next-GPTQ
23
+
24
+ GPTQ (error-compensated weight quantization) (W4A16) quantization of
25
+ [Qwen/Qwen3-Coder-Next](https://huggingface.co/Qwen/Qwen3-Coder-Next), made with
26
+ [llm-compressor](https://github.com/vllm-project/llm-compressor) in the **compressed-tensors** format
27
+ for efficient serving with vLLM and SGLang.
28
+
29
+ Chinese version: [README_zh.md](https://huggingface.co/6block/Qwen3-Coder-Next-GPTQ/blob/main/README_zh.md)
30
+
31
+ ## Why W4A16 (and not the official FP8)
32
+
33
+ Qwen publishes only an FP8 checkpoint for this model. This repo is the **4-bit (W4A16)** build:
34
+ **~40 GiB** on disk versus ~159 GiB for the bf16 master (≈4× smaller) and about half the size of the
35
+ FP8 release. That brings the 80B-A3B coder onto a single 24–48 GB GPU while the MoE routing is left
36
+ untouched (see the protected layers below), so generation tracks the base model.
37
+
38
+ Weights only are quantized — activations stay 16-bit (W4A16), which is what keeps quality close to the
39
+ original on a sparse MoE where the wrong rounding on a router would cascade.
40
+
41
+ ## Model
42
+
43
+ | | |
44
+ |---|---|
45
+ | Base model | [Qwen/Qwen3-Coder-Next](https://huggingface.co/Qwen/Qwen3-Coder-Next) |
46
+ | Architecture | `qwen3_next` MoE — 80B total, ~3B active, 48 layers, 512 experts, top-10 routing |
47
+ | Context | 256K |
48
+ | Method | GPTQ (error-compensated weight quantization), W4A16, group size 128 |
49
+ | Format | compressed-tensors (safetensors) |
50
+ | Size | ~40 GiB |
51
+ | Tool | [llm-compressor](https://github.com/vllm-project/llm-compressor) |
52
+ | License | Apache-2.0 (inherited from base model) |
53
+
54
+ ## What is quantized, and what is protected
55
+
56
+ GPTQ quantizes weights column-by-column and compensates the residual error into the not-yet-quantized columns using second-order (Hessian) information. It was calibrated on a 256-sample mixed code / English / Chinese corpus (2048 tokens
57
+ each). On a 512-expert MoE the routing tensors are the fragile part, so they are kept at full precision:
58
+
59
+ | Tensors | Kept | Reason |
60
+ |---|---|---|
61
+ | `*.mlp.gate` (router) | bf16 | a compressed router sends tokens to the wrong experts |
62
+ | `lm_head` | bf16 | the output projection; a global 4-bit type would squeeze it hardest |
63
+ | everything else (expert + attention linears) | **W4A16** | the bulk of the weights, where 4-bit pays off |
64
+
65
+ ## Quality (perplexity)
66
+
67
+ Perplexity on wikitext-2-raw (test), `n_ctx = 512`, 12 chunks, measured with vLLM — the same recipe
68
+ across all three, so the numbers are directly comparable. Lower is better; the bf16 row is the ceiling.
69
+
70
+ | Weights | PPL | vs bf16 |
71
+ |---|---|---|
72
+ | bf16 (master, reference) | 7.74 | — |
73
+ | **this repo (GPTQ W4A16)** | **9.34** | +20.5% |
74
+
75
+ ## Serving
76
+
77
+ ```bash
78
+ vllm serve 6block/Qwen3-Coder-Next-GPTQ
79
+ ```
80
+
81
+ The quantization scheme is declared in `config.json` and detected automatically — no extra flags. Works
82
+ with any engine that reads compressed-tensors (vLLM, SGLang). At ~40 GiB a single 48 GB GPU serves it
83
+ comfortably; on 24 GB use tensor/pipeline parallel or offload.
84
+
85
+ ## Prompt format
86
+
87
+ ```
88
+ <|im_start|>system
89
+ {system_prompt}<|im_end|>
90
+ <|im_start|>user
91
+ {prompt}<|im_end|>
92
+ <|im_start|>assistant
93
+ ```
94
+
95
+ ## Other quantizations
96
+
97
+ | Format | Repo | Note |
98
+ |---|---|---|
99
+ | AWQ (W4A16) | [6block/Qwen3-Coder-Next-AWQ](https://huggingface.co/6block/Qwen3-Coder-Next-AWQ) | the other 4-bit build |
100
+ | FP8 (official) | [Qwen/Qwen3-Coder-Next-FP8](https://huggingface.co/Qwen/Qwen3-Coder-Next-FP8) | 8-bit, ~2× this size |
101
+
102
+ ---
103
+
104
+ *Quantized by 6block with llm-compressor. Weights are unmodified apart from quantization; all model
105
+ behavior and license obligations follow the base model.*