Azimml/Qwen3-8B-trellis-3bit-webgpu

Qwen3-8B — 3-bit trellis-quantized for WebGPU

A 3-bit trellis-coded quantization (TCQ) of Qwen3-8B, packed to run its full forward pass in a web browser on WebGPU — no CUDA, no server. Runs on a 4 GB GPU: verified live in Chrome on a laptop RTX 3050 Ti, where it completed "The capital of France is" → "Paris. The capital of Italy is Rome. The capital of Germany is Berlin."

This is part of Trellis WebGPU — the first trellis-coded quantization decoder (the QTIP / EXL3 quality tier) to run outside CUDA. See the repo for the quantizer, the WGSL kernels, and the full verification harness.

Quality (WikiText-2 perplexity, no fine-tuning)

	fp16	TCQ 3-bit (this model)	vs fp16
Qwen3-8B	9.02	10.34	1.15×

Trellis-coded quantization is the current quality frontier for low-bit LLM weights, beating scalar quantization (GPTQ / AWQ / GGUF) at equal bitrate. The K=2 quantizer in this project reproduces the QTIP paper's published rate–distortion (MSE 0.0739 vs 0.0733).

This packed model

• 36 layers, packed to ~3 bits/weight (2.9 GB on disk).
• runs in a browser on a 4 GB GPU — verified live in Chrome on a laptop RTX 3050 Ti (4 GB), generating the completion above without running out of memory (256-token KV cache; maxSeq is configurable for longer contexts).
• Verified: the full model runs through the exact shipping WGSL shaders and generates coherent, factually-correct text; kernels match NumPy to <1e-6; 135M logits are top-5 exact vs PyTorch.

Format

This is not a standard transformers checkpoint. It is a packed 3-bit format (manifest.json + sharded .bin weights + IP metadata) designed for the WebGPU runtime in the Trellis WebGPU repo. To run it:

git clone https://github.com/Azimml/trellis-webgpu
# place these files under web/model_8b/ , then:
cd web && python3 -m http.server 8000   # open index.html in a WebGPU browser
# or verify headlessly on your GPU:
python scripts/run_packed_headless.py web/model_8b "The capital of France is" 40

License & credit

Quantized derivative of Qwen3-8B (Apache-2.0), distributed under the base model's license. Method: QTIP (Tseng et al., NeurIPS 2024) and EXL3 (turboderp). Quantization + WebGPU port: independent reimplementation.