Buckets:
| # Compile a model | |
| Quantization saves memory, but dequantizing weights inside every matmul can make generation slower than in bf16. [torch.compile](../perf_torch_compile) fuses that work into optimized kernels and recovers much of the speed. | |
| ## Compile automatically in generate | |
| [generate()](/docs/transformers/pr_43265/en/main_classes/text_generation#transformers.GenerationMixin.generate) compiles the model's forward pass automatically when all of these hold. | |
| - The quantization method supports compilation ([compressed-tensors](./compressed_tensors), [Fine-grained FP8](./finegrained_fp8), [GGUF](./gguf), [NVFP4](./nvfp4), and [torchao](./torchao)). | |
| - The model uses a cache with a fixed size, like the [static cache](../kv_cache#fixed-size-cache). | |
| - The model runs on CUDA, XPU, Neuron, or TPU. | |
| - No part of the model is offloaded to the CPU or disk. | |
| Pass `cache_implementation="static"` to [generate()](/docs/transformers/pr_43265/en/main_classes/text_generation#transformers.GenerationMixin.generate). The example below uses [torchao](./torchao) int8 weight-only quantization. | |
| ```py | |
| import torch | |
| from torchao.quantization import Int8WeightOnlyConfig | |
| from transformers import AutoModelForCausalLM, AutoTokenizer, TorchAoConfig | |
| model_id = "Qwen/Qwen3-0.6B" | |
| model = AutoModelForCausalLM.from_pretrained( | |
| model_id, | |
| quantization_config=TorchAoConfig(quant_type=Int8WeightOnlyConfig()), | |
| device_map="cuda", | |
| ) | |
| tokenizer = AutoTokenizer.from_pretrained(model_id) | |
| inputs = tokenizer("The capital of France is", return_tensors="pt").to(model.device) | |
| outputs = model.generate(**inputs, max_new_tokens=32, cache_implementation="static") | |
| print(tokenizer.decode(outputs[0], skip_special_tokens=True)) | |
| ``` | |
| The first calls are slow while the model compiles. Later calls with the same shapes reuse the compiled kernels. | |
| Pass `disable_compile=True` to [generate()](/docs/transformers/pr_43265/en/main_classes/text_generation#transformers.GenerationMixin.generate) to turn automatic compilation off. | |
Xet Storage Details
- Size:
- 2.01 kB
- Xet hash:
- 7eacfd2e7dfe710d1fb5c7ee807352a119a7c797177942b2cdfa06227aaaefe6
·
Xet efficiently stores files, intelligently splitting them into unique chunks and accelerating uploads and downloads. More info.