Configuration Parsing Warning:In config.json: "quantization_config.modules_to_not_convert" must be an array

Qwen3-4B-AWQ-INT4

INT4 quantization of Qwen/Qwen3-4B. Built to run on a single 6 GB+ consumer GPU.

Footprint

Source params 4B
Quantized weights ~2.5 GB on disk
Inference VRAM (incl. KV cache @ 32K context) ~6 GB

Fits any 6 GB+ consumer card: RTX 2060 / 3050 / 4050 / mobile GPUs / older laptops. The smallest tier in the kit.

Bench

Scored on drawais/needle-1M-bench-mvp (50K-token haystack, real arxiv text):

Metric Score
Overall recall 70.0%
Paper-anchored 60.0%
Synthetic codes 80.0%

Quick start

vllm serve drawais/Qwen3-4B-AWQ-INT4 --quantization awq_marlin --max-model-len 32768
from transformers import AutoTokenizer, AutoModelForCausalLM
tok = AutoTokenizer.from_pretrained("drawais/Qwen3-4B-AWQ-INT4")
model = AutoModelForCausalLM.from_pretrained("drawais/Qwen3-4B-AWQ-INT4", device_map="auto")

Context length

Native: 40,960 tokens. For longer contexts, enable YaRN rope-scaling per the base model's config.

License

Apache 2.0 (inherits from base model).

Downloads last month
15
Safetensors
Model size
4B params
Tensor type
I32
ยท
BF16
ยท
Inference Providers NEW
This model isn't deployed by any Inference Provider. ๐Ÿ™‹ Ask for provider support

Model tree for drawais/Qwen3-4B-AWQ-INT4

Finetuned
Qwen/Qwen3-4B
Quantized
(397)
this model

Collection including drawais/Qwen3-4B-AWQ-INT4