areen09/qwen-engine-models-qwenbin

Unified Qwen Inference Models (.qwenbin v2)

Pre-quantized high-performance model weights formatted for the native CUDA C++ inference engine.

Quantization Format

• Scheme: W4A16 Group-128
• Weights: 4-bit unsigned integer (symmetric per group of 128)
• Scales: FP16 (1 float16 scale per 128 weights)
• Activations: FP16
• Layout: Warp-interleaved for optimal tensor core memory access

Available Models

1. Qwen 2.5-1.5B (qwen2.5-1.5b-w4a16.qwenbin ~1.0 GB)
2. Qwen 3.5-4B (qwen3.5-4b-w4a16.qwenbin ~2.4 GB)
3. Qwen 3.8-27B TP=2 (qwen3.8-27b-tp2-rank0-w4a16.qwenbin ~5.2 GB, qwen3.8-27b-tp2-rank1-w4a16.qwenbin ~5.2 GB)

Quickstart CLI Command

Single GPU (Kaggle Tesla T4 or Local RTX)

./build/bin/cli_bench --model models/qwen2.5-1.5b-w4a16.qwenbin --device 0 --prompt "Hello, tell me about yourself."
./build/bin/cli_bench --model models/qwen3.5-4b-w4a16.qwenbin --device 0 --prompt "Hello, tell me about yourself."

Dual-GPU 2-Way Tensor Parallelism (Dual Tesla T4 cuda:0 & cuda:1)

./build/bin/cli_bench --model models/qwen3.8-27b-tp2-rank0-w4a16.qwenbin --devices 0,1 --prompt "Write a short poem about speed."