Buckets:
| # AQLM | |
| Additive Quantization of Language Models ([AQLM](https://huggingface.co/papers/2401.06118)) quantizes multiple weights together and takes advantage of interdependencies between them. AQLM represents groups of 8-16 weights as a sum of multiple vector codes. | |
| AQLM also supports fine-tuning with [LoRA](https://huggingface.co/docs/peft/package_reference/lora) with the [PEFT](https://huggingface.co/docs/peft) library, and is fully compatible with [torch.compile](https://pytorch.org/tutorials/intermediate/torch_compile_tutorial.html) for even faster inference and training. | |
| Run the command below to install the AQLM library with kernel support for both GPU and CPU inference and training. | |
| ```bash | |
| pip install aqlm[gpu,cpu] | |
| ``` | |
| Load an AQLM-quantized model with [from_pretrained()](/docs/transformers/pr_46771/en/main_classes/model#transformers.PreTrainedModel.from_pretrained). | |
| ```python | |
| from transformers import AutoTokenizer, AutoModelForCausalLM | |
| quantized_model = AutoModelForCausalLM.from_pretrained( | |
| "ISTA-DASLab/Mixtral-8x7b-AQLM-2Bit-1x16-hf", | |
| dtype="auto", | |
| device_map="auto" | |
| ) | |
| ``` | |
| ## Configurations | |
| AQLM quantization setups vary mainly in the number of codebooks used, as well as codebook sizes in bits. The most popular setups and supported inference kernels are shown below. | |
| | Kernel | Number of codebooks | Codebook size, bits | Notation | Accuracy | Speedup | Fast GPU inference | Fast CPU inference | | |
| |---|---------------------|---------------------|----------|-------------|-------------|--------------------|--------------------| | |
| | Triton | K | N | KxN | - | Up to ~0.7x | ✅ | ❌ | | |
| | CUDA | 1 | 16 | 1x16 | Best | Up to ~1.3x | ✅ | ❌ | | |
| | CUDA | 2 | 8 | 2x8 | OK | Up to ~3.0x | ✅ | ❌ | | |
| | Numba | K | 8 | Kx8 | Good | Up to ~4.0x | ❌ | ✅ | | |
| ## Resources | |
| Run the AQLM demo [notebook](https://colab.research.google.com/drive/1-xZmBRXT5Fm3Ghn4Mwa2KRypORXb855X?usp=sharing) for more examples of how to quantize a model, push a quantized model to the Hub, and more. | |
| For more example demo notebooks, visit the AQLM [repository](https://github.com/Vahe1994/AQLM). | |
Xet Storage Details
- Size:
- 2.45 kB
- Xet hash:
- 6994ff9c09c335b3b3fcd61af55ed5d271b355b8cb046a94eaa8f579e8581f8d
·
Xet efficiently stores files, intelligently splitting them into unique chunks and accelerating uploads and downloads. More info.