transformers/pr_48387/en/quantization/aqlm.md

AQLM

Additive Quantization of Language Models (AQLM) quantizes multiple weights together and takes advantage of interdependencies between them. AQLM represents groups of 8-16 weights as a sum of multiple vector codes.

AQLM also supports fine-tuning with LoRA with the PEFT library, and is fully compatible with torch.compile for even faster inference and training.

Run the command below to install the AQLM library with kernel support for both GPU and CPU inference and training. AQLM only works with Python 3.10+.

pip install aqlm[gpu,cpu]

Load an AQLM-quantized model with from_pretrained().

from transformers import AutoTokenizer, AutoModelForCausalLM

quantized_model = AutoModelForCausalLM.from_pretrained(
    "ISTA-DASLab/Mixtral-8x7b-AQLM-2Bit-1x16-hf",
    dtype="auto", 
    device_map="auto"
)

Configurations

AQLM quantization setups vary mainly in the number of codebooks used, as well as codebook sizes in bits. The most popular setups and supported inference kernels are shown below.

Kernel	Number of codebooks	Codebook size, bits	Notation	Accuracy	Speedup	Fast GPU inference	Fast CPU inference
Triton	K	N	KxN	-	Up to ~0.7x	✅	❌
CUDA	1	16	1x16	Best	Up to ~1.3x	✅	❌
CUDA	2	8	2x8	OK	Up to ~3.0x	✅	❌
Numba	K	8	Kx8	Good	Up to ~4.0x	❌	✅

Resources

Run the AQLM demo notebook for more examples of how to quantize a model, push a quantized model to the Hub, and more.

For more example demo notebooks, visit the AQLM repository.