Buckets:

|
download
raw
2.01 kB
# Compile a model
Quantization saves memory, but dequantizing weights inside every matmul can make generation slower than in bf16. [torch.compile](../perf_torch_compile) fuses that work into optimized kernels and recovers much of the speed.
## Compile automatically in generate
[generate()](/docs/transformers/pr_48566/en/main_classes/text_generation#transformers.GenerationMixin.generate) compiles the model's forward pass automatically when all of these hold.
- The quantization method supports compilation ([compressed-tensors](./compressed_tensors), [Fine-grained FP8](./finegrained_fp8), [GGUF](./gguf), [NVFP4](./nvfp4), and [torchao](./torchao)).
- The model uses a cache with a fixed size, like the [static cache](../kv_cache#fixed-size-cache).
- The model runs on CUDA, XPU, Neuron, or TPU.
- No part of the model is offloaded to the CPU or disk.
Pass `cache_implementation="static"` to [generate()](/docs/transformers/pr_48566/en/main_classes/text_generation#transformers.GenerationMixin.generate). The example below uses [torchao](./torchao) int8 weight-only quantization.
```py
import torch
from torchao.quantization import Int8WeightOnlyConfig
from transformers import AutoModelForCausalLM, AutoTokenizer, TorchAoConfig
model_id = "Qwen/Qwen3-0.6B"
model = AutoModelForCausalLM.from_pretrained(
model_id,
quantization_config=TorchAoConfig(quant_type=Int8WeightOnlyConfig()),
device_map="cuda",
)
tokenizer = AutoTokenizer.from_pretrained(model_id)
inputs = tokenizer("The capital of France is", return_tensors="pt").to(model.device)
outputs = model.generate(**inputs, max_new_tokens=32, cache_implementation="static")
print(tokenizer.decode(outputs[0], skip_special_tokens=True))
```
The first calls are slow while the model compiles. Later calls with the same shapes reuse the compiled kernels.
Pass `disable_compile=True` to [generate()](/docs/transformers/pr_48566/en/main_classes/text_generation#transformers.GenerationMixin.generate) to turn automatic compilation off.

Xet Storage Details

Size:
2.01 kB
·
Xet hash:
a8dc43f10254567a89ef76db79da4993a477c1bdf985cc5fc1103a24403254bd

Xet efficiently stores files, intelligently splitting them into unique chunks and accelerating uploads and downloads. More info.