fp8 quant for qwen3.5 models, nearly half memory decrease, speedup 30%, vllm serve can run
🤝 Open to Collab
HyperAI
Hyper-AI
AI & ML interests
lightvl is a lightweight Vision-Language Model (VLM) quantization toolkit supporting FP8, INT8, FP8-Block. It integrates with vLLM for high-throughput inference and supports Qwen3-VL, Qwen3.5, InternVL-Chat, and Gemma-4 models.
fast quant your model step by step:
1、 pip3 install lightvl
2、 lightvl YOUR_HF_MODEL_PATH
Organizations
None yet
fp8 quant for qwen3.5 models, nearly half memory decrease, speedup 30%, vllm serve can run
fp8 quant for qwen3-vl-embedding models, nearly half memory decrease, speedup 30%, vllm serve can run
fp8 quant for qwen3-vl models, nearly half memory decrease, speedup 30%, vllm serve can run
fp8 quant for gemma-4 models, nearly half memory decrease, speedup 30%, vllm serve can run