tacodevs's picture
FP8_DYNAMIC W8A8 of ReadyArt/Wicked-Stuff-31B-Final (llm-compressor)
f6fccfd verified
|
Raw History Blame Contribute Delete
904 Bytes
---
license: apache-2.0
base_model:
- ReadyArt/Wicked-Stuff-31B-Final
base_model_relation: quantized
tags:
- fp8
- llm-compressor
- compressed-tensors
- gemma4
- roleplay
---
# Wicked-Stuff-31B-Final FP8-Dynamic (W8A8)
FP8 quantization of [ReadyArt/Wicked-Stuff-31B-Final](https://huggingface.co/ReadyArt/Wicked-Stuff-31B-Final) (Gemma 4 31B).
- **Scheme:** `FP8_DYNAMIC` W8A8: per-channel FP8 weights, dynamic per-token FP8 activations, on all `Linear` layers (llm-compressor). Vision, audio, embeddings and `lm_head` stay BF16. No calibration data needed.
- **Hardware:** Hopper (H100/H200) and Blackwell; Ada (RTX 4090/L40S) also has FP8.
```bash
vllm serve tacodevs/Wicked-Stuff-31B-Final-FP8-Dynamic --max-model-len 32768 --kv-cache-dtype fp8 \
--reasoning-parser gemma4 --enable-auto-tool-choice --tool-call-parser gemma4
```
Thinking: pass `chat_template_kwargs: {"enable_thinking": true}`.