Spaces:
Running on Zero
Running on Zero
Commit History
-
Migrate to AWQ quantization with FlashAttention-2
- Replace BitsAndBytes 8-bit with AWQ 4-bit quantization (primary path) - Add FlashAttention-2 support for optimized attention - Enable TF32 math for Ampere+ GPUs - Add CUDA kernel warmup on startup to reduce first-token latency - Update model names to reflect AWQ optimization - Add graceful fallback chain: AWQ -> BitsAndBytes -> bf16/fp16/fp32 - Update requirements.txt with flash-attn>=2.5.0 - Update README with performance optimizations documentation06b4cf5
