Buckets:

|
download
raw
2.01 kB

Compile a model

Quantization saves memory, but dequantizing weights inside every matmul can make generation slower than in bf16. torch.compile fuses that work into optimized kernels and recovers much of the speed.

Compile automatically in generate

generate() compiles the model's forward pass automatically when all of these hold.

Pass cache_implementation="static" to generate(). The example below uses torchao int8 weight-only quantization.

import torch
from torchao.quantization import Int8WeightOnlyConfig
from transformers import AutoModelForCausalLM, AutoTokenizer, TorchAoConfig

model_id = "Qwen/Qwen3-0.6B"
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    quantization_config=TorchAoConfig(quant_type=Int8WeightOnlyConfig()),
    device_map="cuda",
)
tokenizer = AutoTokenizer.from_pretrained(model_id)

inputs = tokenizer("The capital of France is", return_tensors="pt").to(model.device)
outputs = model.generate(**inputs, max_new_tokens=32, cache_implementation="static")
print(tokenizer.decode(outputs[0], skip_special_tokens=True))

The first calls are slow while the model compiles. Later calls with the same shapes reuse the compiled kernels.

Pass disable_compile=True to generate() to turn automatic compilation off.

Xet Storage Details

Size:
2.01 kB
·
Xet hash:
a8dc43f10254567a89ef76db79da4993a477c1bdf985cc5fc1103a24403254bd

Xet efficiently stores files, intelligently splitting them into unique chunks and accelerating uploads and downloads. More info.