Save and share a model

Saving a model after quantizing it while loading lets you, and anyone you share it with, skip the quantization step next time. Calibration methods like GPTQ benefit the most because calibration can take minutes to hours.

Save the model

Call save_pretrained() on the quantized model to write the quantized weights and add the quantization_config to config.json.

from transformers import AutoModelForCausalLM, BitsAndBytesConfig

quantization_config = BitsAndBytesConfig(load_in_4bit=True)
model = AutoModelForCausalLM.from_pretrained(
    "Qwen/Qwen3-0.6B",
    quantization_config=quantization_config,
    device_map="auto",
)
model.save_pretrained("Qwen3-0.6B-bnb-4bit")

Save the tokenizer to the same directory so the checkpoint is complete.

Push the model to the Hub

Call push_to_hub() to upload the quantized weights and config to a Hub repository. Log in with hf auth login first.

model.push_to_hub("Qwen3-0.6B-bnb-4bit")

Load the saved model

Load the saved checkpoint with from_pretrained() and no quantization config. Transformers reads the method from the quantization_config in config.json and loads the weights already quantized.

from transformers import AutoModelForCausalLM

model = AutoModelForCausalLM.from_pretrained("your-username/Qwen3-0.6B-bnb-4bit", device_map="auto")

The method’s library must be installed wherever the checkpoint is loaded.

Methods that can’t save

Some methods can’t save their quantized weights, and save_pretrained() raises a ValueError for them.

A quantized model with a PEFT adapter attached is an exception. save_pretrained() saves only the adapter weights, so it works with any method.

To save a model in full precision instead, dequantize it first. See Dequantize a model.