Add Qwen3 32B w4a8 and w6a8

Qwen3-32B/NOTICE.md

@@ -1,25 +1,42 @@
 # Notice
 
-`qwen3_32b_int8_convrot.safetensors` is a modified copy of Qwen3-32B.
+`qwen3_32b_int8_convrot.safetensors`, `qwen3_32b_w6a8.safetensors` and
+`qwen3_32b_w4a8.safetensors` are modified copies of Qwen3-32B.
 
 | | |
 |---|---|
 | Work | Qwen3-32B, by the Qwen team, Alibaba Cloud |
 | Source | https://huggingface.co/Qwen/Qwen3-32B, revision `9216db5781bf21249d130ec9da846c4624c16137` |
 | Licence | Apache License 2.0, in [`LICENSE`](LICENSE). Copyright 2024 Alibaba Cloud |
-| Modified by | WAS, 2026-10-27 |
+| Modified by | WAS, 2026-10-27 and 2026-10-28 |
 
-## Changes
+## Changes in every file
 
-- The 17 `model-*.safetensors` shards are joined into one file. Tensor names are unchanged.
-- Every linear layer in the 64 transformer layers (`q_proj`, `k_proj`, `v_proj`, `o_proj`,
-  `gate_proj`, `up_proj`, `down_proj`), the embedding `model.embed_tokens` and the output layer
-  `lm_head` are quantized to int8: each row is rotated by a 256-wide Hadamard block, then rounded
-  to the nearest of 255 levels scaled by the row's largest magnitude. Each layer stores
-  `weight` (int8), `weight_scale` (float32, one per row) and a `comfy_quant` tag reading
+- The 17 `model-*.safetensors` shards are joined into one file. Tensor names are unchanged,
+  apart from the split below.
+- The embedding `model.embed_tokens` and the output layer `lm_head` are quantized to int8: each
+  row is rotated by a 256-wide Hadamard block, then rounded to the nearest of 255 levels scaled by
+  the row's largest magnitude. Each stores `weight` (int8), `weight_scale` (float32, one per row)
+  and a `comfy_quant` tag reading
   `{"format": "int8_tensorwise", "convrot": true, "convrot_groupsize": 256}`.
 - `input_layernorm`, `post_attention_layernorm`, `q_norm`, `k_norm` and the final `norm` are
   kept in bfloat16 as released.
 - The tokenizer and configuration files are not included; ComfyUI supplies the Qwen tokenizer.
 
+## The linear layers
+
+The seven linears of each of the 64 layers (`q_proj`, `k_proj`, `v_proj`, `o_proj`,
+`gate_proj`, `up_proj`, `down_proj`):
+
+| File | Format | Stored per layer |
+|---|---|---|
+| `qwen3_32b_int8_convrot.safetensors` | int8, as the embedding above | `weight`, `weight_scale`, `comfy_quant` |
+| `qwen3_32b_w6a8.safetensors` | 6-bit codes in groups of 32 inputs, rows rotated by a 256-wide Hadamard block; `{"format": "w6a8_int8", "group_size": 32, "convrot": true, "convrot_groupsize": 256}` | `weight`, `weight_s_channel`, `weight_s_rel` (float8), `comfy_quant` |
+| `qwen3_32b_w4a8.safetensors` | 4-bit codes from a 16-entry codebook in groups of 16 inputs, rows rotated by a 256-wide Hadamard block; `{"format": "asym_w4a8_int8", "group_size": 16, "convrot": true, "convrot_groupsize": 256}` | `weight`, `weight_codebook`, `weight_s_channel`, `weight_s_rel` (float8), `comfy_quant` |
+
+In the w6a8 and w4a8 files each `mlp.down_proj` (5,120 outputs, 25,600 inputs) is split into its
+first and last 12,800 input columns, stored as `mlp.down_proj.parts.0` and
+`mlp.down_proj.parts.1` and quantized separately. The layer's output is the sum of the two
+halves' outputs.
+
 No weights were trained or fine-tuned.

Qwen3-32B/qwen3_32b_w4a8.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:e8010915064353333e2f5e324d066aeeeb0690960e0fb40f2f9fb4cef33bd653
+size 19132294160

Qwen3-32B/qwen3_32b_w6a8.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:b3daea4c7bde12d439e1acdfe718481e03155d5690ca4af5a15e78c0c0a460bd
+size 25958488592

README.md

@@ -7,6 +7,8 @@ tags:
 - comfyui
 - qwen3
 - int8
+- w4a8
+- w6a8
 - convrot
 - text-generation
 ---
@@ -22,26 +24,40 @@ which hands a CLIP to core **Generate Text**, **Fast Generate Text** and **LLM F
 
 Put them in `ComfyUI/models/text_encoders`.
 
-| Model | File | Size | Weights | Licence |
-|---|---|--:|---|---|
-| [Qwen3 32B](https://huggingface.co/Qwen/Qwen3-32B) | [`Qwen3-32B/qwen3_32b_int8_convrot.safetensors`](Qwen3-32B/qwen3_32b_int8_convrot.safetensors) | 32.8 GB | int8 convrot | [Apache-2.0](Qwen3-32B/LICENSE), [notice](Qwen3-32B/NOTICE.md) |
+| Model | File | Size | Weights | Writing speed, RTX 4090 24 GB |
+|---|---|--:|---|--:|
+| [Qwen3 32B](https://huggingface.co/Qwen/Qwen3-32B) | [`Qwen3-32B/qwen3_32b_w4a8.safetensors`](Qwen3-32B/qwen3_32b_w4a8.safetensors) | 19.1 GB | 4-bit weights in groups of 16, 8-bit activations | 27.6 tokens/s |
+| | [`Qwen3-32B/qwen3_32b_w6a8.safetensors`](Qwen3-32B/qwen3_32b_w6a8.safetensors) | 26.0 GB | 6-bit weights in groups of 32, 8-bit activations | 3.3 tokens/s |
+| | [`Qwen3-32B/qwen3_32b_int8_convrot.safetensors`](Qwen3-32B/qwen3_32b_int8_convrot.safetensors) | 32.8 GB | 8-bit weights | 1.5 tokens/s |
+
+Speeds are Fast Generate Text's graph decode, greedy, with ComfyUI's `--reserve-vram 2`. A file
+that fits on the graphics card runs from it; a larger one streams part of its weights from system
+memory for every token, which is why w6a8 and int8 are slow on a 24 GB card. On a 32 GB card the
+w6a8 file fits. Licence for all three: [Apache-2.0](Qwen3-32B/LICENSE),
+[notice](Qwen3-32B/NOTICE.md).
+
+| | w4a8 | w6a8 | int8 convrot |
+|---|--:|--:|--:|
+| Mean weight error of the layer linears | 7.3% | 2.3% | 0.91% |
 
 ## Qwen3 32B
 
 Qwen's dense 32B model with its own output layer, all 64 layers, 40,960 tokens of context, and
 the hybrid thinking switch: Generate Text's `thinking` input turns reasoning on, and off it answers
-straight away.
+straight away. All three files gave the same greedy answer to the same question.
 
-- Every linear layer, the embedding and the output layer are int8, each row rotated by a
-  256-wide Hadamard block before it was rounded (int8 convrot, as Comfy-Org quantizes its Qwen
-  text encoders). Norms stay bfloat16. The mean weight error is 0.91%, the worst layer 1.09%.
+- Every layer's linears are quantized, each row rotated by a 256-wide Hadamard block before
+  rounding (ConvRot, as Comfy-Org quantizes its Qwen text encoders). The embedding and the output
+  layer are int8 in every file. Norms stay bfloat16.
 - Tensor names are Qwen's own (`model.layers.0.self_attn.q_proj.weight`, `lm_head.weight`), with
   a `comfy_quant` tag beside each quantized layer.
+- In the w4a8 and w6a8 files each layer's `mlp.down_proj`, 25,600 inputs wide, is stored as two
+  12,800-wide column halves, `mlp.down_proj.parts.0` and `mlp.down_proj.parts.1`, whose outputs
+  add up to the layer's. ComfyUI's 4-bit and 6-bit kernels take a layer up to 17,408 inputs wide.
+  Language Model Loader adds the halves back up.
 - ComfyUI's **Load CLIP** does not read a 32B Qwen3: it takes a 5120-wide Qwen3 for Mistral 24B.
   **Language Model Loader** builds it on ComfyUI's own Qwen3 model.
-- The file is larger than a 24 GB graphics card, so part of it streams from system memory for
-  every token. On an RTX 4090 with 96 GB of system memory, Fast Generate Text writes about 1.5
-  tokens a second.
+- w4a8 and w6a8 need an RTX 30 series card or newer; int8 convrot an RTX 20 series or newer.
 - It reads text only. MiniMax H3's `qwen3vl_32b` text encoder is a different file: the first 50
   layers of Qwen3-VL 32B with no output layer, which conditions H3 and cannot write text.
 
@@ -50,6 +66,6 @@ Tested with ComfyUI 0.42.
 ## License
 
 Qwen3 32B is by the Qwen team at Alibaba Cloud and is licensed under the Apache License 2.0, a copy
-of which is in [`Qwen3-32B/LICENSE`](Qwen3-32B/LICENSE). The file here is modified from
+of which is in [`Qwen3-32B/LICENSE`](Qwen3-32B/LICENSE). The files here are modified from
 [Qwen/Qwen3-32B](https://huggingface.co/Qwen/Qwen3-32B) by WAS: what changed is listed in
 [`Qwen3-32B/NOTICE.md`](Qwen3-32B/NOTICE.md).