Buckets:
| <meta charset="utf-8" /><meta name="hf:doc:metadata" content="{"title":"Dequantize a model","local":"dequantize-a-model","sections":[{"title":"Dequantize after loading","local":"dequantize-after-loading","sections":[],"depth":2},{"title":"Dequantize while loading","local":"dequantize-while-loading","sections":[],"depth":2}],"depth":1}"/> | |
| <link href="/docs/transformers/main/en/_app/immutable/entry/start.BKs81iV6.js" rel="modulepreload"> | |
| <link href="/docs/transformers/main/en/_app/immutable/chunks/NUJjOGEr.js" rel="modulepreload"> | |
| <link href="/docs/transformers/main/en/_app/immutable/chunks/Cdpslwgk.js" rel="modulepreload"> | |
| <link href="/docs/transformers/main/en/_app/immutable/entry/app.BhvEH3Bp.js" rel="modulepreload"> | |
| <link href="/docs/transformers/main/en/_app/immutable/chunks/DRF6BKTD.js" rel="modulepreload"> | |
| <link href="/docs/transformers/main/en/_app/immutable/chunks/DOLBYqcr.js" rel="modulepreload"> | |
| <link href="/docs/transformers/main/en/_app/immutable/chunks/DsnmJJEf.js" rel="modulepreload"> | |
| <link href="/docs/transformers/main/en/_app/immutable/chunks/C7xMRJKQ.js" rel="modulepreload"> | |
| <link href="/docs/transformers/main/en/_app/immutable/nodes/0.CjenUr70.js" rel="modulepreload"> | |
| <link href="/docs/transformers/main/en/_app/immutable/chunks/D0Q3SQ6Z.js" rel="modulepreload"> | |
| <link href="/docs/transformers/main/en/_app/immutable/nodes/4.sFeee406.js" rel="modulepreload"> | |
| <link href="/docs/transformers/main/en/_app/immutable/chunks/CWISPsAZ.js" rel="modulepreload"> | |
| <!--ui5vy5--><meta name="hf:doc:metadata" content="{"title":"Dequantize a model","local":"dequantize-a-model","sections":[{"title":"Dequantize after loading","local":"dequantize-after-loading","sections":[],"depth":2},{"title":"Dequantize while loading","local":"dequantize-while-loading","sections":[],"depth":2}],"depth":1}"/><!----> | |
| <link href="/docs/transformers/main/en/_app/immutable/assets/0.tn0RQdqM.css" rel="modulepreload"> <!--[--><!--[0--><!--[--><!--[0--><!--[--><p></p> <div class="items-center shrink-0 min-w-[100px] max-sm:min-w-[50px] justify-end ml-auto flex" style="float: inline-end; margin-inline-start: 10px; display: inline-flex; position: relative; z-index: 10;"><div dir="ltr" class="inline-flex rounded-md max-sm:rounded-sm"><button class="inline-flex items-center gap-1 h-7 max-sm:h-7 px-2 max-sm:px-1.5 text-sm font-medium text-gray-800 border border-r-0 rounded-l-md max-sm:rounded-l-sm border-gray-200 bg-white hover:shadow-inner dark:border-gray-850 dark:bg-gray-950 dark:text-gray-200 dark:hover:bg-gray-800" aria-live="polite"><span class="inline-flex items-center justify-center rounded-md p-0.5 max-sm:p-0 hover:text-gray-800 dark:hover:text-gray-200"><svg class="sm:size-3.5 size-3" xmlns="http://www.w3.org/2000/svg" aria-hidden="true" fill="currentColor" focusable="false" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 32 32"><path d="M28,10V28H10V10H28m0-2H10a2,2,0,0,0-2,2V28a2,2,0,0,0,2,2H28a2,2,0,0,0,2-2V10a2,2,0,0,0-2-2Z" transform="translate(0)"></path><path d="M4,18H2V4A2,2,0,0,1,4,2H18V4H4Z" transform="translate(0)"></path><rect fill="none" width="32" height="32"></rect></svg><!----></span> <span dir="auto">Copy page</span></button> <button class="inline-flex items-center justify-center w-6 max-sm:w-5 h-7 max-sm:h-7 disabled:pointer-events-none text-sm text-gray-500 hover:text-gray-700 dark:hover:text-white rounded-r-md max-sm:rounded-r-sm border border-l transition border-gray-200 bg-white hover:shadow-inner dark:border-gray-850 dark:bg-gray-950 dark:text-gray-200 dark:hover:bg-gray-800" aria-haspopup="menu" aria-expanded="false" aria-label="Open copy menu"><svg class="transition-transform text-gray-400 overflow-visible sm:size-3.5 size-3 rotate-0" width="1em" height="1em" viewBox="0 0 12 7" fill="none" xmlns="http://www.w3.org/2000/svg"><path d="M1 1L6 6L11 1" stroke="currentColor"></path></svg><!----></button></div> <!--[-1--><!--]--></div><!----> <!--[0--><h1 class="relative group"><a id="dequantize-a-model" class="header-link block pr-1.5 text-lg no-hover:hidden with-hover:absolute with-hover:p-1.5 with-hover:opacity-0 with-hover:group-hover:opacity-100 with-hover:right-full" href="#dequantize-a-model"><span><svg xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" aria-hidden="true" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 256 256"><path d="M167.594 88.393a8.001 8.001 0 0 1 0 11.314l-67.882 67.882a8 8 0 1 1-11.314-11.315l67.882-67.881a8.003 8.003 0 0 1 11.314 0zm-28.287 84.86l-28.284 28.284a40 40 0 0 1-56.567-56.567l28.284-28.284a8 8 0 0 0-11.315-11.315l-28.284 28.284a56 56 0 0 0 79.196 79.197l28.285-28.285a8 8 0 1 0-11.315-11.314zM212.852 43.14a56.002 56.002 0 0 0-79.196 0l-28.284 28.284a8 8 0 1 0 11.314 11.314l28.284-28.284a40 40 0 0 1 56.568 56.567l-28.285 28.285a8 8 0 0 0 11.315 11.314l28.284-28.284a56.065 56.065 0 0 0 0-79.196z" fill="currentColor"></path></svg><!----></span></a> <span><!---->Dequantize a model<!----></span></h1><!--]--><!----> <p>Dequantizing converts quantized weights back to a higher-precision dtype like bf16. Dequantize a model to fully fine-tune it, to run it on hardware the method’s kernels don’t support, or to convert it to another format. The result uses as much memory as an unquantized model, and it doesn’t recover the accuracy lost to rounding.</p> <p>Transformers dequantizes in two ways, depending on the method.</p> <table><thead><tr><th>Method</th><th>How to dequantize</th></tr></thead><tbody><tr><td><a href="./bitsandbytes">bitsandbytes</a>, <a href="./higgs">HIGGS</a></td><td>Call <a href="/docs/transformers/pr_49594/en/main_classes/model#transformers.PreTrainedModel.dequantize">dequantize()</a> after loading.</td></tr><tr><td><a href="./compressed_tensors">compressed-tensors</a>, <a href="./finegrained_fp8">Fine-grained FP8</a>, <a href="./gguf">GGUF</a>, <a href="./metal">Metal</a>, <a href="./mxfp4">MXFP4</a></td><td>Pass <code>dequantize=True</code> to the method’s config when loading.</td></tr></tbody></table> <p>Other methods raise a <code>NotImplementedError</code> from <a href="/docs/transformers/pr_49594/en/main_classes/model#transformers.PreTrainedModel.dequantize">dequantize()</a>.</p> <!--[1--><h2 class="relative group"><a id="dequantize-after-loading" class="header-link block pr-1.5 text-lg no-hover:hidden with-hover:absolute with-hover:p-1.5 with-hover:opacity-0 with-hover:group-hover:opacity-100 with-hover:right-full" href="#dequantize-after-loading"><span><svg xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" aria-hidden="true" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 256 256"><path d="M167.594 88.393a8.001 8.001 0 0 1 0 11.314l-67.882 67.882a8 8 0 1 1-11.314-11.315l67.882-67.881a8.003 8.003 0 0 1 11.314 0zm-28.287 84.86l-28.284 28.284a40 40 0 0 1-56.567-56.567l28.284-28.284a8 8 0 0 0-11.315-11.315l-28.284 28.284a56 56 0 0 0 79.196 79.197l28.285-28.285a8 8 0 1 0-11.315-11.314zM212.852 43.14a56.002 56.002 0 0 0-79.196 0l-28.284 28.284a8 8 0 1 0 11.314 11.314l28.284-28.284a40 40 0 0 1 56.568 56.567l-28.285 28.285a8 8 0 0 0 11.315 11.314l28.284-28.284a56.065 56.065 0 0 0 0-79.196z" fill="currentColor"></path></svg><!----></span></a> <span><!---->Dequantize after loading<!----></span></h2><!--]--><!----> <p>Call <a href="/docs/transformers/pr_49594/en/main_classes/model#transformers.PreTrainedModel.dequantize">dequantize()</a> on a loaded model. It defaults to the dtype the model was loaded in. Pass <code>dtype</code> to choose another.</p> <div class="code-block relative "><div class="absolute top-2.5 right-4"><button class="inline-flex items-center relative text-sm focus:text-green-500 cursor-pointer focus:outline-none transition duration-200 ease-in-out opacity-0 mx-0.5 text-gray-600 " title="code excerpt" type="button"><svg xmlns="http://www.w3.org/2000/svg" aria-hidden="true" fill="currentColor" focusable="false" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 32 32"><path d="M28,10V28H10V10H28m0-2H10a2,2,0,0,0-2,2V28a2,2,0,0,0,2,2H28a2,2,0,0,0,2-2V10a2,2,0,0,0-2-2Z" transform="translate(0)"></path><path d="M4,18H2V4A2,2,0,0,1,4,2H18V4H4Z" transform="translate(0)"></path><rect fill="none" width="32" height="32"></rect></svg><!----> <div class=" absolute pointer-events-none transition-opacity bg-black text-white py-1 px-2 leading-tight rounded font-normal shadow left-1/2 top-full transform -translate-x-1/2 translate-y-2 opacity-0 "><div class="absolute bottom-full left-1/2 transform -translate-x-1/2 w-0 h-0 border-black border-4 border-t-0" style="border-left-color: transparent; border-right-color: transparent;"></div> Copied</div><!----></button><!----></div> <pre class="language-py "><!----><span class="hljs-keyword">import</span> torch | |
| <span class="hljs-keyword">from</span> transformers <span class="hljs-keyword">import</span> AutoModelForCausalLM, BitsAndBytesConfig | |
| model = AutoModelForCausalLM.from_pretrained( | |
| <span class="hljs-string">"Qwen/Qwen3-0.6B"</span>, | |
| quantization_config=BitsAndBytesConfig(load_in_4bit=<span class="hljs-literal">True</span>), | |
| device_map=<span class="hljs-string">"auto"</span>, | |
| ) | |
| model = model.dequantize(dtype=torch.bfloat16)<!----></pre></div><!----> <p><a href="/docs/transformers/pr_49594/en/main_classes/model#transformers.PreTrainedModel.dequantize">dequantize()</a> also removes the <code>quantization_config</code>, so <a href="/docs/transformers/pr_49594/en/main_classes/model#transformers.PreTrainedModel.save_pretrained">save_pretrained()</a> saves a regular full-precision checkpoint.</p> <!--[1--><h2 class="relative group"><a id="dequantize-while-loading" class="header-link block pr-1.5 text-lg no-hover:hidden with-hover:absolute with-hover:p-1.5 with-hover:opacity-0 with-hover:group-hover:opacity-100 with-hover:right-full" href="#dequantize-while-loading"><span><svg xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" aria-hidden="true" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 256 256"><path d="M167.594 88.393a8.001 8.001 0 0 1 0 11.314l-67.882 67.882a8 8 0 1 1-11.314-11.315l67.882-67.881a8.003 8.003 0 0 1 11.314 0zm-28.287 84.86l-28.284 28.284a40 40 0 0 1-56.567-56.567l28.284-28.284a8 8 0 0 0-11.315-11.315l-28.284 28.284a56 56 0 0 0 79.196 79.197l28.285-28.285a8 8 0 1 0-11.315-11.314zM212.852 43.14a56.002 56.002 0 0 0-79.196 0l-28.284 28.284a8 8 0 1 0 11.314 11.314l28.284-28.284a40 40 0 0 1 56.568 56.567l-28.285 28.285a8 8 0 0 0 11.315 11.314l28.284-28.284a56.065 56.065 0 0 0 0-79.196z" fill="currentColor"></path></svg><!----></span></a> <span><!---->Dequantize while loading<!----></span></h2><!--]--><!----> <p>Pass <code>dequantize=True</code> to the config of a method that supports it. The weights are dequantized as they load, so the quantized kernels are never used. The example below loads the MXFP4 weights of gpt-oss in bf16.</p> <div class="code-block relative "><div class="absolute top-2.5 right-4"><button class="inline-flex items-center relative text-sm focus:text-green-500 cursor-pointer focus:outline-none transition duration-200 ease-in-out opacity-0 mx-0.5 text-gray-600 " title="code excerpt" type="button"><svg xmlns="http://www.w3.org/2000/svg" aria-hidden="true" fill="currentColor" focusable="false" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 32 32"><path d="M28,10V28H10V10H28m0-2H10a2,2,0,0,0-2,2V28a2,2,0,0,0,2,2H28a2,2,0,0,0,2-2V10a2,2,0,0,0-2-2Z" transform="translate(0)"></path><path d="M4,18H2V4A2,2,0,0,1,4,2H18V4H4Z" transform="translate(0)"></path><rect fill="none" width="32" height="32"></rect></svg><!----> <div class=" absolute pointer-events-none transition-opacity bg-black text-white py-1 px-2 leading-tight rounded font-normal shadow left-1/2 top-full transform -translate-x-1/2 translate-y-2 opacity-0 "><div class="absolute bottom-full left-1/2 transform -translate-x-1/2 w-0 h-0 border-black border-4 border-t-0" style="border-left-color: transparent; border-right-color: transparent;"></div> Copied</div><!----></button><!----></div> <pre class="language-py "><!----><span class="hljs-keyword">from</span> transformers <span class="hljs-keyword">import</span> AutoModelForCausalLM, Mxfp4Config | |
| model = AutoModelForCausalLM.from_pretrained( | |
| <span class="hljs-string">"openai/gpt-oss-20b"</span>, | |
| quantization_config=Mxfp4Config(dequantize=<span class="hljs-literal">True</span>), | |
| device_map=<span class="hljs-string">"auto"</span>, | |
| )<!----></pre></div><!----> <p>Some methods dequantize automatically when their kernels can’t run. <a href="./gguf">GGUF</a>, for example, falls back to dequantizing when no kernel supports the file’s quantization types.</p> <a class="!text-gray-400 !no-underline text-sm flex items-center not-prose mt-4" href="https://github.com/huggingface/transformers/blob/main/docs/source/en/quantization/dequantize.md" target="_blank"><svg class="mr-1" xmlns="http://www.w3.org/2000/svg" aria-hidden="true" fill="currentColor" focusable="false" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 32 32"><path d="M31,16l-7,7l-1.41-1.41L28.17,16l-5.58-5.59L24,9l7,7z"></path><path d="M1,16l7-7l1.41,1.41L3.83,16l5.58,5.59L8,23l-7-7z"></path><path d="M12.419,25.484L17.639,6.552l1.932,0.518L14.351,26.002z"></path></svg><!----> <span><span class="underline">Update</span> on GitHub</span></a><!----> <p></p><!--]--><!----><!--]--><!--]--><!--]--> <!--[-1--><!--]--><!--]--> | |
| <script> | |
| { | |
| __sveltekit_zv0m7q = { | |
| base: "/docs/transformers/main/en", | |
| assets: "/docs/transformers/main/en" | |
| }; | |
| const element = document.currentScript.parentElement; | |
| Promise.all([ | |
| import("/docs/transformers/main/en/_app/immutable/entry/start.BKs81iV6.js"), | |
| import("/docs/transformers/main/en/_app/immutable/entry/app.BhvEH3Bp.js") | |
| ]).then(([kit, app]) => { | |
| kit.start(app, element, { | |
| node_ids: [0, 4], | |
| data: [null,null], | |
| form: null, | |
| error: null | |
| }); | |
| }); | |
| } | |
| </script> | |
Xet Storage Details
- Size:
- 14.6 kB
- Xet hash:
- 73bc172a2b96e6b2bf2a6545307b690a7ed8dd42b493d34f0a30f3ee2d87baa6
·
Xet efficiently stores files, intelligently splitting them into unique chunks and accelerating uploads and downloads. More info.