Buckets:

HuggingFaceDocBuilder's picture
|
download
raw
2.1 kB
# EETQ
[Easy and Efficient Quantization for Transformers (EETQ)](https://github.com/NetEase-FuXi/EETQ) quantizes weights to int8 with one scale per output channel, and runs the matmul with int8 weights and 16-bit activations. Use it for 8-bit weight-only quantization on NVIDIA GPUs without calibration data.
The table below summarizes what EETQ supports in Transformers.
| | |
|---|---|
| Config | [EetqConfig](/docs/transformers/pr_49444/en/main_classes/quantization#transformers.EetqConfig) |
| `quant_method` | `eetq` |
| Workflow | Quantize while loading |
| Bits | 8 |
| Hardware | CUDA |
| Save | ✅ |
| Fine-tune | ✅ |
| Auto-compile | ❌ |
## Install the EETQ kernels
Transformers downloads the EETQ kernels from the [kernels-community/quantization-eetq](https://huggingface.co/kernels-community/quantization-eetq) repository on the Hub, so you don't need to build EETQ. Install the [kernels](https://huggingface.co/docs/kernels/index) and [Accelerate](https://huggingface.co/docs/accelerate/index) libraries.
```bash
pip install -U "transformers[kernels]" accelerate
```
## Quantize a model with EETQ
Create an [EetqConfig](/docs/transformers/pr_49444/en/main_classes/quantization#transformers.EetqConfig) and pass it to [from_pretrained()](/docs/transformers/pr_49444/en/main_classes/model#transformers.PreTrainedModel.from_pretrained). `weights` sets the weight dtype and only accepts `"int8"`, which is the default.
```py
import torch
from transformers import AutoModelForCausalLM, EetqConfig
quantization_config = EetqConfig("int8")
model = AutoModelForCausalLM.from_pretrained(
"Qwen/Qwen3-8B",
quantization_config=quantization_config,
dtype=torch.float16,
device_map="cuda",
)
```
Load the model with `dtype=torch.float16` for the best efficiency. EETQ stores its scales in fp16.
## Limitations
- EETQ needs a CUDA GPU. Loading without one raises a `RuntimeError`.
- EETQ can't offload to the CPU or disk. A `device_map` that includes `"cpu"` or `"disk"` raises a `ValueError`.
## Resources
- [EETQ](https://github.com/NetEase-FuXi/EETQ) repository

Xet Storage Details

Size:
2.1 kB
·
Xet hash:
64f0f285e342b4d1a26ddb4ecdad4a913a891ab89fa7bb9acc80632d69842228

Xet efficiently stores files, intelligently splitting them into unique chunks and accelerating uploads and downloads. More info.