Buckets:

|
download
raw
2.88 kB

Save and share a model

Saving a model after quantizing it while loading lets you, and anyone you share it with, skip the quantization step next time. Calibration methods like GPTQ benefit the most because calibration can take minutes to hours.

Save the model

Call save_pretrained() on the quantized model to write the quantized weights and add the quantization_config to config.json.

from transformers import AutoModelForCausalLM, BitsAndBytesConfig

quantization_config = BitsAndBytesConfig(load_in_4bit=True)
model = AutoModelForCausalLM.from_pretrained(
    "Qwen/Qwen3-0.6B",
    quantization_config=quantization_config,
    device_map="auto",
)
model.save_pretrained("Qwen3-0.6B-bnb-4bit")

Save the tokenizer to the same directory so the checkpoint is complete.

Push the model to the Hub

Call push_to_hub() to upload the quantized weights and config to a Hub repository. Log in with hf auth login first.

model.push_to_hub("Qwen3-0.6B-bnb-4bit")

Load the saved model

Load the saved checkpoint with from_pretrained() and no quantization config. Transformers reads the method from the quantization_config in config.json and loads the weights already quantized.

from transformers import AutoModelForCausalLM

model = AutoModelForCausalLM.from_pretrained("your-username/Qwen3-0.6B-bnb-4bit", device_map="auto")

The method's library must be installed wherever the checkpoint is loaded.

Methods that can't save

Some methods can't save their quantized weights, and save_pretrained() raises a ValueError for them.

Method Why
GGUF Weights load from a GGUF file. Save in GGUF format with llama.cpp tools instead.
NVFP4 Saving isn't supported. Quantize the model again when you load it.
Quanto Saving isn't supported through Transformers.
Quark Saving isn't supported through Transformers.
AWQ with an ExLlama backend Saving isn't supported with the ExLlama backends. Load the model with another backend to save it.

A quantized model with a PEFT adapter attached is an exception. save_pretrained() saves only the adapter weights, so it works with any method.

To save a model in full precision instead, dequantize it first. See Dequantize a model.

Xet Storage Details

Size:
2.88 kB
·
Xet hash:
52c083611557209bfd19121d100f61ca8d629bec0ae8e5920e5d98dec1daaf06

Xet efficiently stores files, intelligently splitting them into unique chunks and accelerating uploads and downloads. More info.