Buckets:
| <meta charset="utf-8" /><meta name="hf:doc:metadata" content="{"title":"Keep modules in full precision","local":"keep-modules-in-full-precision","sections":[{"title":"Default skipped modules","local":"default-skipped-modules","sections":[],"depth":2},{"title":"Pass a skip list","local":"pass-a-skip-list","sections":[],"depth":2},{"title":"Match module names","local":"match-module-names","sections":[],"depth":2},{"title":"Choose which modules to skip","local":"choose-which-modules-to-skip","sections":[],"depth":2}],"depth":1}"/> | |
| <link href="/docs/transformers/main/en/_app/immutable/entry/start.DYIO-UII.js" rel="modulepreload"> | |
| <link href="/docs/transformers/main/en/_app/immutable/chunks/DSYYoqAA.js" rel="modulepreload"> | |
| <link href="/docs/transformers/main/en/_app/immutable/chunks/BovLmb3z.js" rel="modulepreload"> | |
| <link href="/docs/transformers/main/en/_app/immutable/entry/app.D2_KizbC.js" rel="modulepreload"> | |
| <link href="/docs/transformers/main/en/_app/immutable/chunks/WyYpyc-L.js" rel="modulepreload"> | |
| <link href="/docs/transformers/main/en/_app/immutable/chunks/DYBfcveH.js" rel="modulepreload"> | |
| <link href="/docs/transformers/main/en/_app/immutable/chunks/DOLBYqcr.js" rel="modulepreload"> | |
| <link href="/docs/transformers/main/en/_app/immutable/chunks/DsnmJJEf.js" rel="modulepreload"> | |
| <link href="/docs/transformers/main/en/_app/immutable/nodes/0.ByO_lNbj.js" rel="modulepreload"> | |
| <link href="/docs/transformers/main/en/_app/immutable/chunks/ZtBhN-Ei.js" rel="modulepreload"> | |
| <link href="/docs/transformers/main/en/_app/immutable/nodes/725.Dr5DOixD.js" rel="modulepreload"> | |
| <link href="/docs/transformers/main/en/_app/immutable/chunks/D-dvW--N.js" rel="modulepreload"> | |
| <!--13baw3s--><meta name="hf:doc:metadata" content="{"title":"Keep modules in full precision","local":"keep-modules-in-full-precision","sections":[{"title":"Default skipped modules","local":"default-skipped-modules","sections":[],"depth":2},{"title":"Pass a skip list","local":"pass-a-skip-list","sections":[],"depth":2},{"title":"Match module names","local":"match-module-names","sections":[],"depth":2},{"title":"Choose which modules to skip","local":"choose-which-modules-to-skip","sections":[],"depth":2}],"depth":1}"/><!----> | |
| <link href="/docs/transformers/main/en/_app/immutable/assets/0.tn0RQdqM.css" rel="modulepreload"> <!--[--><!--[0--><!--[--><!--[0--><!--[--><p></p> <div class="items-center shrink-0 min-w-[100px] max-sm:min-w-[50px] justify-end ml-auto flex" style="float: inline-end; margin-inline-start: 10px; display: inline-flex; position: relative; z-index: 10;"><div dir="ltr" class="inline-flex rounded-md max-sm:rounded-sm"><button class="inline-flex items-center gap-1 h-7 max-sm:h-7 px-2 max-sm:px-1.5 text-sm font-medium text-gray-800 border border-r-0 rounded-l-md max-sm:rounded-l-sm border-gray-200 bg-white hover:shadow-inner dark:border-gray-850 dark:bg-gray-950 dark:text-gray-200 dark:hover:bg-gray-800" aria-live="polite"><span class="inline-flex items-center justify-center rounded-md p-0.5 max-sm:p-0 hover:text-gray-800 dark:hover:text-gray-200"><svg class="sm:size-3.5 size-3" xmlns="http://www.w3.org/2000/svg" aria-hidden="true" fill="currentColor" focusable="false" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 32 32"><path d="M28,10V28H10V10H28m0-2H10a2,2,0,0,0-2,2V28a2,2,0,0,0,2,2H28a2,2,0,0,0,2-2V10a2,2,0,0,0-2-2Z" transform="translate(0)"></path><path d="M4,18H2V4A2,2,0,0,1,4,2H18V4H4Z" transform="translate(0)"></path><rect fill="none" width="32" height="32"></rect></svg><!----></span> <span dir="auto">Copy page</span></button> <button class="inline-flex items-center justify-center w-6 max-sm:w-5 h-7 max-sm:h-7 disabled:pointer-events-none text-sm text-gray-500 hover:text-gray-700 dark:hover:text-white rounded-r-md max-sm:rounded-r-sm border border-l transition border-gray-200 bg-white hover:shadow-inner dark:border-gray-850 dark:bg-gray-950 dark:text-gray-200 dark:hover:bg-gray-800" aria-haspopup="menu" aria-expanded="false" aria-label="Open copy menu"><svg class="transition-transform text-gray-400 overflow-visible sm:size-3.5 size-3 rotate-0" width="1em" height="1em" viewBox="0 0 12 7" fill="none" xmlns="http://www.w3.org/2000/svg"><path d="M1 1L6 6L11 1" stroke="currentColor"></path></svg><!----></button></div> <!--[-1--><!--]--></div><!----> <!--[0--><h1 class="relative group"><a id="keep-modules-in-full-precision" class="header-link block pr-1.5 text-lg no-hover:hidden with-hover:absolute with-hover:p-1.5 with-hover:opacity-0 with-hover:group-hover:opacity-100 with-hover:right-full" href="#keep-modules-in-full-precision"><span><svg xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" aria-hidden="true" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 256 256"><path d="M167.594 88.393a8.001 8.001 0 0 1 0 11.314l-67.882 67.882a8 8 0 1 1-11.314-11.315l67.882-67.881a8.003 8.003 0 0 1 11.314 0zm-28.287 84.86l-28.284 28.284a40 40 0 0 1-56.567-56.567l28.284-28.284a8 8 0 0 0-11.315-11.315l-28.284 28.284a56 56 0 0 0 79.196 79.197l28.285-28.285a8 8 0 1 0-11.315-11.314zM212.852 43.14a56.002 56.002 0 0 0-79.196 0l-28.284 28.284a8 8 0 1 0 11.314 11.314l28.284-28.284a40 40 0 0 1 56.568 56.567l-28.285 28.285a8 8 0 0 0 11.315 11.314l28.284-28.284a56.065 56.065 0 0 0 0-79.196z" fill="currentColor"></path></svg><!----></span></a> <span><!---->Keep modules in full precision<!----></span></h1><!--]--><!----> <p>Some modules lose more accuracy than others when quantized. Keeping them in their original precision costs a little memory and can recover much of the accuracy. This applies when you quantize a model while loading. A pre-quantized checkpoint already records which modules it skipped.</p> <!--[1--><h2 class="relative group"><a id="default-skipped-modules" class="header-link block pr-1.5 text-lg no-hover:hidden with-hover:absolute with-hover:p-1.5 with-hover:opacity-0 with-hover:group-hover:opacity-100 with-hover:right-full" href="#default-skipped-modules"><span><svg xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" aria-hidden="true" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 256 256"><path d="M167.594 88.393a8.001 8.001 0 0 1 0 11.314l-67.882 67.882a8 8 0 1 1-11.314-11.315l67.882-67.881a8.003 8.003 0 0 1 11.314 0zm-28.287 84.86l-28.284 28.284a40 40 0 0 1-56.567-56.567l28.284-28.284a8 8 0 0 0-11.315-11.315l-28.284 28.284a56 56 0 0 0 79.196 79.197l28.285-28.285a8 8 0 1 0-11.315-11.314zM212.852 43.14a56.002 56.002 0 0 0-79.196 0l-28.284 28.284a8 8 0 1 0 11.314 11.314l28.284-28.284a40 40 0 0 1 56.568 56.567l-28.285 28.285a8 8 0 0 0 11.315 11.314l28.284-28.284a56.065 56.065 0 0 0 0-79.196z" fill="currentColor"></path></svg><!----></span></a> <span><!---->Default skipped modules<!----></span></h2><!--]--><!----> <p>If you don’t pass a skip list, most methods keep these modules unquantized.</p> <ul><li>The output embeddings, usually <code>lm_head</code>.</li> <li>Weights tied to other weights, like input embeddings that share weights with <code>lm_head</code>.</li> <li>The model’s last module.</li> <li>Modules the model lists in <code>_keep_in_fp32_modules</code>.</li></ul> <p>A few methods set their own default instead. <a href="./hqq">HQQ</a>, <a href="./fp_quant">FP-Quant</a>, and <a href="./fouroversix">Four Over Six</a> skip only <code>lm_head</code>. <a href="./sinq">SINQ</a> skips only the modules in <code>_keep_in_fp32_modules</code>.</p> <!--[1--><h2 class="relative group"><a id="pass-a-skip-list" class="header-link block pr-1.5 text-lg no-hover:hidden with-hover:absolute with-hover:p-1.5 with-hover:opacity-0 with-hover:group-hover:opacity-100 with-hover:right-full" href="#pass-a-skip-list"><span><svg xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" aria-hidden="true" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 256 256"><path d="M167.594 88.393a8.001 8.001 0 0 1 0 11.314l-67.882 67.882a8 8 0 1 1-11.314-11.315l67.882-67.881a8.003 8.003 0 0 1 11.314 0zm-28.287 84.86l-28.284 28.284a40 40 0 0 1-56.567-56.567l28.284-28.284a8 8 0 0 0-11.315-11.315l-28.284 28.284a56 56 0 0 0 79.196 79.197l28.285-28.285a8 8 0 1 0-11.315-11.314zM212.852 43.14a56.002 56.002 0 0 0-79.196 0l-28.284 28.284a8 8 0 1 0 11.314 11.314l28.284-28.284a40 40 0 0 1 56.568 56.567l-28.285 28.285a8 8 0 0 0 11.315 11.314l28.284-28.284a56.065 56.065 0 0 0 0-79.196z" fill="currentColor"></path></svg><!----></span></a> <span><!---->Pass a skip list<!----></span></h2><!--]--><!----> <p>Pass the module names to the skip-list argument of the method’s config. The example below keeps the <code>lm_head</code> and the MLP down projections in bf16 with <a href="./bitsandbytes">bitsandbytes</a>.</p> <div class="code-block relative "><div class="absolute top-2.5 right-4"><button class="inline-flex items-center relative text-sm focus:text-green-500 cursor-pointer focus:outline-none transition duration-200 ease-in-out opacity-0 mx-0.5 text-gray-600 " title="code excerpt" type="button"><svg xmlns="http://www.w3.org/2000/svg" aria-hidden="true" fill="currentColor" focusable="false" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 32 32"><path d="M28,10V28H10V10H28m0-2H10a2,2,0,0,0-2,2V28a2,2,0,0,0,2,2H28a2,2,0,0,0,2-2V10a2,2,0,0,0-2-2Z" transform="translate(0)"></path><path d="M4,18H2V4A2,2,0,0,1,4,2H18V4H4Z" transform="translate(0)"></path><rect fill="none" width="32" height="32"></rect></svg><!----> <div class=" absolute pointer-events-none transition-opacity bg-black text-white py-1 px-2 leading-tight rounded font-normal shadow left-1/2 top-full transform -translate-x-1/2 translate-y-2 opacity-0 "><div class="absolute bottom-full left-1/2 transform -translate-x-1/2 w-0 h-0 border-black border-4 border-t-0" style="border-left-color: transparent; border-right-color: transparent;"></div> Copied</div><!----></button><!----></div> <pre class="language-py "><!----><span class="hljs-keyword">from</span> transformers <span class="hljs-keyword">import</span> AutoModelForCausalLM, BitsAndBytesConfig | |
| quantization_config = BitsAndBytesConfig( | |
| load_in_4bit=<span class="hljs-literal">True</span>, | |
| llm_int8_skip_modules=[<span class="hljs-string">"lm_head"</span>, <span class="hljs-string">"mlp.down_proj"</span>], | |
| ) | |
| model = AutoModelForCausalLM.from_pretrained( | |
| <span class="hljs-string">"Qwen/Qwen3-0.6B"</span>, | |
| quantization_config=quantization_config, | |
| device_map=<span class="hljs-string">"auto"</span>, | |
| )<!----></pre></div><!----> <blockquote class="warning"><p>A skip list replaces the default skipped modules instead of adding to them, although modules in <code>_keep_in_fp32_modules</code> stay skipped. Include <code>lm_head</code> in your list if you still want it unquantized. <a href="./awq">AWQ</a> is the exception and adds your list to the defaults.</p></blockquote> <p>The argument’s name depends on the method.</p> <table><thead><tr><th>Argument</th><th>Methods</th></tr></thead><tbody><tr><td><code>modules_to_not_convert</code></td><td>Most methods, including AWQ, EETQ, FBGEMM FP8, Fine-grained FP8, Four Over Six, FP-Quant, HIGGS, Metal, MXFP4, NVFP4, Quanto, SINQ, and torchao</td></tr><tr><td><code>llm_int8_skip_modules</code></td><td>bitsandbytes, in both 4-bit and 8-bit mode</td></tr><tr><td><code>skip_modules</code></td><td>HQQ</td></tr></tbody></table> <p><a href="./gptq">GPTQ</a> takes the opposite approach. <code>modules_in_block_to_quantize</code> lists the modules to quantize, and everything else in the block is left unquantized.</p> <!--[1--><h2 class="relative group"><a id="match-module-names" class="header-link block pr-1.5 text-lg no-hover:hidden with-hover:absolute with-hover:p-1.5 with-hover:opacity-0 with-hover:group-hover:opacity-100 with-hover:right-full" href="#match-module-names"><span><svg xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" aria-hidden="true" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 256 256"><path d="M167.594 88.393a8.001 8.001 0 0 1 0 11.314l-67.882 67.882a8 8 0 1 1-11.314-11.315l67.882-67.881a8.003 8.003 0 0 1 11.314 0zm-28.287 84.86l-28.284 28.284a40 40 0 0 1-56.567-56.567l28.284-28.284a8 8 0 0 0-11.315-11.315l-28.284 28.284a56 56 0 0 0 79.196 79.197l28.285-28.285a8 8 0 1 0-11.315-11.314zM212.852 43.14a56.002 56.002 0 0 0-79.196 0l-28.284 28.284a8 8 0 1 0 11.314 11.314l28.284-28.284a40 40 0 0 1 56.568 56.567l-28.285 28.285a8 8 0 0 0 11.315 11.314l28.284-28.284a56.065 56.065 0 0 0 0-79.196z" fill="currentColor"></path></svg><!----></span></a> <span><!---->Match module names<!----></span></h2><!--]--><!----> <p>Most methods match each entry in the skip list against the full module names from <code>model.named_modules()</code>, such as <code>model.layers.3.mlp.down_proj</code>. <a href="./hqq">HQQ</a> is the exception and matches layer names exactly. An entry matches in one of three ways.</p> <ul><li>As a suffix. <code>mlp.down_proj</code> matches the down projection in every layer.</li> <li>As a prefix that ends at a module boundary. <code>model.layers.1</code> matches <code>model.layers.1.mlp</code> but not <code>model.layers.11.mlp</code>.</li> <li>As a regular expression with the same boundary. <code>model\.layers\.(0|1)</code> matches the first two layers.</li></ul> <p>Print the module names to find the right entries.</p> <div class="code-block relative "><div class="absolute top-2.5 right-4"><button class="inline-flex items-center relative text-sm focus:text-green-500 cursor-pointer focus:outline-none transition duration-200 ease-in-out opacity-0 mx-0.5 text-gray-600 " title="code excerpt" type="button"><svg xmlns="http://www.w3.org/2000/svg" aria-hidden="true" fill="currentColor" focusable="false" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 32 32"><path d="M28,10V28H10V10H28m0-2H10a2,2,0,0,0-2,2V28a2,2,0,0,0,2,2H28a2,2,0,0,0,2-2V10a2,2,0,0,0-2-2Z" transform="translate(0)"></path><path d="M4,18H2V4A2,2,0,0,1,4,2H18V4H4Z" transform="translate(0)"></path><rect fill="none" width="32" height="32"></rect></svg><!----> <div class=" absolute pointer-events-none transition-opacity bg-black text-white py-1 px-2 leading-tight rounded font-normal shadow left-1/2 top-full transform -translate-x-1/2 translate-y-2 opacity-0 "><div class="absolute bottom-full left-1/2 transform -translate-x-1/2 w-0 h-0 border-black border-4 border-t-0" style="border-left-color: transparent; border-right-color: transparent;"></div> Copied</div><!----></button><!----></div> <pre class="language-py "><!----><span class="hljs-keyword">for</span> name, _ <span class="hljs-keyword">in</span> model.named_modules(): | |
| <span class="hljs-built_in">print</span>(name)<!----></pre></div><!----> <!--[1--><h2 class="relative group"><a id="choose-which-modules-to-skip" class="header-link block pr-1.5 text-lg no-hover:hidden with-hover:absolute with-hover:p-1.5 with-hover:opacity-0 with-hover:group-hover:opacity-100 with-hover:right-full" href="#choose-which-modules-to-skip"><span><svg xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" aria-hidden="true" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 256 256"><path d="M167.594 88.393a8.001 8.001 0 0 1 0 11.314l-67.882 67.882a8 8 0 1 1-11.314-11.315l67.882-67.881a8.003 8.003 0 0 1 11.314 0zm-28.287 84.86l-28.284 28.284a40 40 0 0 1-56.567-56.567l28.284-28.284a8 8 0 0 0-11.315-11.315l-28.284 28.284a56 56 0 0 0 79.196 79.197l28.285-28.285a8 8 0 1 0-11.315-11.314zM212.852 43.14a56.002 56.002 0 0 0-79.196 0l-28.284 28.284a8 8 0 1 0 11.314 11.314l28.284-28.284a40 40 0 0 1 56.568 56.567l-28.285 28.285a8 8 0 0 0 11.315 11.314l28.284-28.284a56.065 56.065 0 0 0 0-79.196z" fill="currentColor"></path></svg><!----></span></a> <span><!---->Choose which modules to skip<!----></span></h2><!--]--><!----> <p>Skip the modules where rounding error affects accuracy the most for the least memory saved. Start with the defaults, measure accuracy on your task, and add modules only if it drops.</p> <ul><li><code>lm_head</code> produces the logits for every token in the vocabulary, so errors there directly change which token is picked. It’s skipped by default. On models with large vocabularies it’s also one of the biggest single layers, so quantizing it saves real memory if accuracy holds up.</li> <li>MoE router or gate layers, like <code>mlp.gate</code> in Qwen3-MoE or <code>mlp.router</code> in gpt-oss, decide which experts each token goes to. They’re tiny, and a rounding error can send a token to the wrong expert, so they’re almost always worth skipping.</li> <li>Vision encoders and multimodal projectors in vision-language models are small next to the language model, and errors there distort every image token. Skip them if image understanding degrades.</li> <li>The first and last few decoder layers often have more activation outliers than the middle layers. Skip them when the quantized model’s accuracy is noticeably worse than in bf16.</li></ul> <p>Every module you skip stays in its original precision and costs memory. Check that the model still fits after you add to the skip list.</p> <a class="!text-gray-400 !no-underline text-sm flex items-center not-prose mt-4" href="https://github.com/huggingface/transformers/blob/main/docs/source/en/quantization/skip_modules.md" target="_blank"><svg class="mr-1" xmlns="http://www.w3.org/2000/svg" aria-hidden="true" fill="currentColor" focusable="false" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 32 32"><path d="M31,16l-7,7l-1.41-1.41L28.17,16l-5.58-5.59L24,9l7,7z"></path><path d="M1,16l7-7l1.41,1.41L3.83,16l5.58,5.59L8,23l-7-7z"></path><path d="M12.419,25.484L17.639,6.552l1.932,0.518L14.351,26.002z"></path></svg><!----> <span><span class="underline">Update</span> on GitHub</span></a><!----> <p></p><!--]--><!----><!--]--><!--]--><!--]--> <!--[-1--><!--]--><!--]--> | |
| <script> | |
| { | |
| __sveltekit_1m6yd5j = { | |
| base: "/docs/transformers/main/en", | |
| assets: "/docs/transformers/main/en" | |
| }; | |
| const element = document.currentScript.parentElement; | |
| Promise.all([ | |
| import("/docs/transformers/main/en/_app/immutable/entry/start.DYIO-UII.js"), | |
| import("/docs/transformers/main/en/_app/immutable/entry/app.D2_KizbC.js") | |
| ]).then(([kit, app]) => { | |
| kit.start(app, element, { | |
| node_ids: [0, 725], | |
| data: [null,null], | |
| form: null, | |
| error: null | |
| }); | |
| }); | |
| } | |
| </script> | |
Xet Storage Details
- Size:
- 19.3 kB
- Xet hash:
- 6024cf19a7321f2e8c7c39c28263752575e356fdc4463f307e990a5ed6020af7
·
Xet efficiently stores files, intelligently splitting them into unique chunks and accelerating uploads and downloads. More info.