| --- |
| license: apache-2.0 |
| base_model: Qwen/Qwen3-Coder-Next |
| tags: |
| - qwen |
| - qwen3 |
| - coder |
| - cq |
| - cq_quant |
| - quantization |
| - 4bit |
| - vllm |
| - text-generation |
| pipeline_tag: text-generation |
| --- |
| |
| # Qwen3-Coder-Next-CQ-4bit |
|
|
| > **80B-parameter Mixture-of-Experts coding model · 3B activated parameters · Coded Quantization (CQ) 4-bit** |
| > |
| > **Note on Hugging Face's model-size display:** Hugging Face may report a smaller stored parameter count for this repository because CQ stores the weights in a packed 4-bit representation. The underlying model is **Qwen3-Coder-Next**, with approximately **80B total parameters and 3B activated parameters per token**. The smaller number reflects the packed storage tensors counted by the Hub, not the parameter count of the original model architecture. |
|
|
| **Qwen3-Coder-Next quantized to 4-bit with Coded Quantization (CQ).** |
|
|
| **ThousandML:** https://thousandml.com |
|
|
| CQ is a **calibration-free 4-bit quantization method** designed to preserve the behavior of the original model without requiring a calibration dataset. Because CQ does not optimize the quantization around a fixed calibration corpus, it is designed to generalize beyond calibration-specific data and workloads. |
|
|
| This repository contains **Qwen3-Coder-Next** quantized with **Coded Quantization (CQ) 4-bit**. |
|
|
| ## Why CQ? |
|
|
| **Calibration-free.** CQ does not require a representative calibration dataset before quantization. |
|
|
| This avoids tying the quantization procedure to a particular calibration corpus and is intended to provide better generalization beyond the data that would otherwise have been used for calibration. |
|
|
| **4-bit inference.** Model weights are quantized to 4-bit for substantially lower weight memory requirements than BF16/FP16 deployment. |
|
|
| **Native vLLM integration.** CQ models run through the CQ vLLM quantization plugin and custom CUDA kernels included in the Thousandml CQ Docker image. |
|
|
| ## Model |
|
|
| | | | |
| |---|---| |
| | **Base model** | `Qwen/Qwen3-Coder-Next` | |
| | **Original model parameters** | **~80B total / ~3B activated** | |
| | **Model type** | Mixture-of-Experts coding model | |
| | **Quantization** | **Coded Quantization (CQ) 4-bit** | |
| | **Calibration** | **None — calibration-free** | |
| | **Runtime** | vLLM + CQ custom kernels | |
| | **Quantization name** | `cq` | |
| | **Context length** | **262,144 tokens** | |
|
|
| ## Run with Docker |
|
|
| The recommended way to run this model is with the CQ Docker image, which contains the compatible vLLM installation, CQ quantization plugin, and CQ CUDA kernels. |
|
|
| ### 1. Make sure NVIDIA Docker support works |
|
|
| ```bash |
| docker run --rm --gpus all \ |
| nvidia/cuda:13.0.2-base-ubuntu22.04 \ |
| nvidia-smi |
| ``` |
|
|
| Your GPU should appear in the output. |
|
|
| ### 2. Start Qwen3-Coder-Next-CQ-4bit |
|
|
| ```bash |
| docker run --rm --gpus all \ |
| -p 8000:8000 \ |
| docker.io/thousandml/cq:4bit-main \ |
| vllm serve thousandml/Qwen3-Coder-Next-CQ-4bit \ |
| --quantization cq |
| ``` |
|
|
| The model is then available through vLLM's OpenAI-compatible API on port `8000`. |
|
|
| ### WSL2 |
|
|
| If you are running Docker through WSL2 and vLLM reports that UVA is unavailable, enable vLLM's WSL2 pinned-memory support: |
|
|
| ```bash |
| docker run --rm --gpus all \ |
| -e VLLM_WSL2_ENABLE_PIN_MEMORY=1 \ |
| -p 8000:8000 \ |
| docker.io/thousandml/cq:4bit-main \ |
| vllm serve thousandml/Qwen3-Coder-Next-CQ-4bit \ |
| --quantization cq |
| ``` |
|
|
| ## Send a request |
|
|
| Once the server is ready: |
|
|
| ```bash |
| curl http://localhost:8000/v1/chat/completions \ |
| -H "Content-Type: application/json" \ |
| -d '{ |
| "model": "thousandml/Qwen3-Coder-Next-CQ-4bit", |
| "messages": [ |
| { |
| "role": "user", |
| "content": "Write a Python function that performs binary search on a sorted list." |
| } |
| ] |
| }' |
| ``` |
|
|
| ## Tool calling |
|
|
| Qwen3-Coder-Next is designed for coding and agentic workflows and can be served with vLLM's automatic tool-choice support. |
|
|
| Example: |
|
|
| ```bash |
| docker run --rm --gpus all \ |
| -p 8000:8000 \ |
| docker.io/thousandml/cq:4bit-main \ |
| vllm serve thousandml/Qwen3-Coder-Next-CQ-4bit \ |
| --quantization cq \ |
| --enable-auto-tool-choice \ |
| --tool-call-parser qwen3_coder |
| ``` |
|
|
| ## Hugging Face authentication |
|
|
| For higher Hugging Face Hub rate limits, pass your Hugging Face token into the container: |
|
|
| ```bash |
| docker run --rm --gpus all \ |
| -e HF_TOKEN="$HF_TOKEN" \ |
| -p 8000:8000 \ |
| docker.io/thousandml/cq:4bit-main \ |
| vllm serve thousandml/Qwen3-Coder-Next-CQ-4bit \ |
| --quantization cq |
| ``` |
|
|
| On WSL2, combine it with: |
|
|
| ```bash |
| -e VLLM_WSL2_ENABLE_PIN_MEMORY=1 |
| ``` |
|
|
| ## About Coded Quantization (CQ) |
|
|
| Learn more at **https://thousandml.com**. |
|
|
| **Coded Quantization (CQ)** is a calibration-free quantization approach for efficient LLM inference. |
|
|
| Traditional post-training quantization methods can depend on a calibration dataset to estimate quantization parameters. The resulting quantization can therefore be influenced by the distribution and composition of that calibration data. |
|
|
| CQ removes that calibration-data requirement. The goal is to retain strong fidelity to the original model while allowing the quantized representation to generalize beyond any particular calibration corpus. |
|
|
| ## Base model |
|
|
| This checkpoint is derived from **Qwen/Qwen3-Coder-Next**, a Mixture-of-Experts coding model designed for software development, agentic coding, tool use, and long-context workflows. |
|
|
| Please refer to the original Qwen model card for architecture details, capabilities, usage guidance, limitations, and upstream licensing information. |
|
|
| ## License |
|
|
| The base model is released under the **Apache 2.0 License**. See the repository license and the original Qwen model card for applicable terms. |
