| ---
|
| license: apache-2.0
|
| base_model: microsoft/AesCode-32B
|
| base_model_relation: quantized
|
| quantized_by: WeReCooking
|
| pipeline_tag: image-text-to-text
|
| tags:
|
| - exl3
|
| - qwen3_vl
|
| - html
|
| ---
|
|
|
| <style>
|
| .container-dark { font-family: -apple-system, BlinkMacSystemFont, "Segoe UI", Roboto, Arial, sans-serif; line-height: 1.6; color: #d4d4d4; }
|
| a { color: #569cd6; text-decoration: none; font-weight: 600; }
|
| a:hover { text-decoration: underline; }
|
| .card-dark { background-color: #252526; border-radius: 12px; padding: 24px; margin-bottom: 20px; box-shadow: 0 4px 12px rgba(0,0,0,0.3); border: 1px solid #3c3c3c; }
|
| .card-dark h1 { font-size: 2.2em; color: #ffffff; text-align: center; margin-bottom: 10px; }
|
| .card-dark .subtitle { text-align: center; font-size: 1.1em; color: #a0a0a0; }
|
| .card-dark h2 { font-size: 1.5em; margin-top: 0; padding-bottom: 10px; border-bottom: 1px solid #3c3c3c; color: #c586c0; }
|
| .styled-table { display: table; border: none; width: 100%; font-size: 0.95em; }
|
| .styled-table thead th { background-color: #333333; color: #c586c0; text-align: left; padding: 12px 15px; }
|
| .styled-table td { padding: 0; border-bottom: 1px solid #3c3c3c; }
|
| .styled-table tbody tr { transition: background-color 0.1s ease; }
|
| .styled-table tbody tr:hover { background-color: #3a3a3a; }
|
| .styled-table tr:last-child td { border-bottom: none; }
|
| .styled-table td a { display: block; padding: 12px 15px; }
|
| .styled-table td a.fake-link { text-decoration:none; color:inherit; }
|
| details { margin-top: 20px; border: 1px solid #3c3c3c; border-radius: 8px; overflow: hidden; }
|
| summary { cursor: pointer; padding: 12px 18px; background-color: #6A5ACD; font-weight: 600; display: flex; align-items: center; gap: 10px; justify-content: space-between; list-style: none; }
|
| summary::-webkit-details-marker { display: none; }
|
| summary:hover { filter: brightness(1.1); }
|
| summary::after { content: ''; display: inline-block; width: 8px; height: 8px; border-bottom: 2px solid white; border-right: 2px solid white; transform: rotate(45deg); transition: transform 0.3s ease; }
|
| details[open] > summary::after { transform: rotate(225deg); }
|
| .details-content { padding: 18px; }
|
| </style>
|
|
|
| <div class="container-dark">
|
|
|
| <div class="card-dark">
|
| <h1>AesCode-32B EXL3</h1>
|
| <p class="subtitle">
|
| EXL3 quants of <a href="https://huggingface.co/microsoft/AesCode-32B">microsoft/AesCode-32B</a>
|
| using <a href="https://github.com/turboderp-org/exllamav3/">exllamav3</a> v1.6.1
|
| </p>
|
| </div>
|
|
|
| <div class="card-dark">
|
| <h2>KL Divergence vs VRAM</h2>
|
| <img src="kld_plot.png" alt="KLD plot" style="width:100%; border-radius: 8px;" />
|
| <p class="subtitle">Reference: BF16 original (PPL 6.843). Noise floor KLD 0.000151. Measured with exllamav3 <code>eval/qbench.py</code> on wikitext-2 (20 rows, 2048 ctx). VRAM = decoder + lm_head weights.</p>
|
| <details>
|
| <summary>KLD spread (median, IQR, noise floor)</summary>
|
| <div class="details-content">
|
| <img src="kld_spread_plot.png" alt="KLD spread" style="width:100%; border-radius: 8px;" />
|
| </div>
|
| </details>
|
| </div>
|
|
|
| <div class="card-dark">
|
| <h2>Quants</h2>
|
| <table class="styled-table">
|
| <thead>
|
| <tr><th>Branch</th><th>BPW</th><th>Head</th><th>VRAM (GiB)</th><th>KLD mean</th><th>KLD median</th><th>PPL</th><th>Type</th></tr>
|
| </thead>
|
| <tbody>
|
| <tr>
|
| <td><a href="https://huggingface.co/WeReCooking/AesCode-32B-EXL3/tree/3.0bpw_H6">3.0bpw_H6</a></td>
|
| <td><a class="fake-link" href="https://huggingface.co/WeReCooking/AesCode-32B-EXL3/tree/3.0bpw_H6">3.0</a></td>
|
| <td><a class="fake-link" href="https://huggingface.co/WeReCooking/AesCode-32B-EXL3/tree/3.0bpw_H6">6</a></td>
|
| <td><a class="fake-link" href="https://huggingface.co/WeReCooking/AesCode-32B-EXL3/tree/3.0bpw_H6">11.5</a></td>
|
| <td><a class="fake-link" href="https://huggingface.co/WeReCooking/AesCode-32B-EXL3/tree/3.0bpw_H6">0.0360</a></td>
|
| <td><a class="fake-link" href="https://huggingface.co/WeReCooking/AesCode-32B-EXL3/tree/3.0bpw_H6">0.01522</a></td>
|
| <td><a class="fake-link" href="https://huggingface.co/WeReCooking/AesCode-32B-EXL3/tree/3.0bpw_H6">6.987</a></td>
|
| <td><a class="fake-link" href="https://huggingface.co/WeReCooking/AesCode-32B-EXL3/tree/3.0bpw_H6">base</a></td>
|
| </tr>
|
| <tr>
|
| <td><a href="https://huggingface.co/WeReCooking/AesCode-32B-EXL3/tree/3.73bpw_H6">3.73bpw_H6</a></td>
|
| <td><a class="fake-link" href="https://huggingface.co/WeReCooking/AesCode-32B-EXL3/tree/3.73bpw_H6">3.73</a></td>
|
| <td><a class="fake-link" href="https://huggingface.co/WeReCooking/AesCode-32B-EXL3/tree/3.73bpw_H6">6</a></td>
|
| <td><a class="fake-link" href="https://huggingface.co/WeReCooking/AesCode-32B-EXL3/tree/3.73bpw_H6">14.1</a></td>
|
| <td><a class="fake-link" href="https://huggingface.co/WeReCooking/AesCode-32B-EXL3/tree/3.73bpw_H6">0.0157</a></td>
|
| <td><a class="fake-link" href="https://huggingface.co/WeReCooking/AesCode-32B-EXL3/tree/3.73bpw_H6">0.00616</a></td>
|
| <td><a class="fake-link" href="https://huggingface.co/WeReCooking/AesCode-32B-EXL3/tree/3.73bpw_H6">6.902</a></td>
|
| <td><a class="fake-link" href="https://huggingface.co/WeReCooking/AesCode-32B-EXL3/tree/3.73bpw_H6">optimized</a></td>
|
| </tr>
|
| <tr>
|
| <td><a href="https://huggingface.co/WeReCooking/AesCode-32B-EXL3/tree/4.0bpw_H6">4.0bpw_H6</a></td>
|
| <td><a class="fake-link" href="https://huggingface.co/WeReCooking/AesCode-32B-EXL3/tree/4.0bpw_H6">4.0</a></td>
|
| <td><a class="fake-link" href="https://huggingface.co/WeReCooking/AesCode-32B-EXL3/tree/4.0bpw_H6">6</a></td>
|
| <td><a class="fake-link" href="https://huggingface.co/WeReCooking/AesCode-32B-EXL3/tree/4.0bpw_H6">15.1</a></td>
|
| <td><a class="fake-link" href="https://huggingface.co/WeReCooking/AesCode-32B-EXL3/tree/4.0bpw_H6">0.0093</a></td>
|
| <td><a class="fake-link" href="https://huggingface.co/WeReCooking/AesCode-32B-EXL3/tree/4.0bpw_H6">0.00392</a></td>
|
| <td><a class="fake-link" href="https://huggingface.co/WeReCooking/AesCode-32B-EXL3/tree/4.0bpw_H6">6.876</a></td>
|
| <td><a class="fake-link" href="https://huggingface.co/WeReCooking/AesCode-32B-EXL3/tree/4.0bpw_H6">base</a></td>
|
| </tr>
|
| <tr>
|
| <td><a href="https://huggingface.co/WeReCooking/AesCode-32B-EXL3/tree/4.54bpw_H6">4.54bpw_H6</a></td>
|
| <td><a class="fake-link" href="https://huggingface.co/WeReCooking/AesCode-32B-EXL3/tree/4.54bpw_H6">4.54</a></td>
|
| <td><a class="fake-link" href="https://huggingface.co/WeReCooking/AesCode-32B-EXL3/tree/4.54bpw_H6">6</a></td>
|
| <td><a class="fake-link" href="https://huggingface.co/WeReCooking/AesCode-32B-EXL3/tree/4.54bpw_H6">17.0</a></td>
|
| <td><a class="fake-link" href="https://huggingface.co/WeReCooking/AesCode-32B-EXL3/tree/4.54bpw_H6">0.0065</a></td>
|
| <td><a class="fake-link" href="https://huggingface.co/WeReCooking/AesCode-32B-EXL3/tree/4.54bpw_H6">0.00260</a></td>
|
| <td><a class="fake-link" href="https://huggingface.co/WeReCooking/AesCode-32B-EXL3/tree/4.54bpw_H6">6.863</a></td>
|
| <td><a class="fake-link" href="https://huggingface.co/WeReCooking/AesCode-32B-EXL3/tree/4.54bpw_H6">optimized</a></td>
|
| </tr>
|
| <tr>
|
| <td><a href="https://huggingface.co/WeReCooking/AesCode-32B-EXL3/tree/5.0bpw_H6">5.0bpw_H6</a></td>
|
| <td><a class="fake-link" href="https://huggingface.co/WeReCooking/AesCode-32B-EXL3/tree/5.0bpw_H6">5.0</a></td>
|
| <td><a class="fake-link" href="https://huggingface.co/WeReCooking/AesCode-32B-EXL3/tree/5.0bpw_H6">6</a></td>
|
| <td><a class="fake-link" href="https://huggingface.co/WeReCooking/AesCode-32B-EXL3/tree/5.0bpw_H6">18.7</a></td>
|
| <td><a class="fake-link" href="https://huggingface.co/WeReCooking/AesCode-32B-EXL3/tree/5.0bpw_H6">0.0027</a></td>
|
| <td><a class="fake-link" href="https://huggingface.co/WeReCooking/AesCode-32B-EXL3/tree/5.0bpw_H6">0.00122</a></td>
|
| <td><a class="fake-link" href="https://huggingface.co/WeReCooking/AesCode-32B-EXL3/tree/5.0bpw_H6">6.871</a></td>
|
| <td><a class="fake-link" href="https://huggingface.co/WeReCooking/AesCode-32B-EXL3/tree/5.0bpw_H6">base</a></td>
|
| </tr>
|
| </tbody>
|
| </table>
|
| <p class="subtitle">Bases are direct converts. Optimized = KLD-guided tensor mix of the bases + attention recompiled at 5bpw; on this model they land on the base size/KLD curve, so treat them as in-between sizes and pick by VRAM. Vision tower 6bpw in every branch.</p>
|
| </div>
|
|
|
| <div class="card-dark">
|
| <h2>Download and run</h2>
|
| <details>
|
| <summary>Download</summary>
|
| <div class="details-content">
|
| <pre><code>pip install -U "huggingface_hub"
|
| hf download WeReCooking/AesCode-32B-EXL3 --revision 4.0bpw_H6 --local-dir ./AesCode-32B-EXL3</code></pre>
|
| </div>
|
| </details>
|
| <details>
|
| <summary>Run with exllamav3</summary>
|
| <div class="details-content">
|
| <b>Text chat, official exllamav3 CLI</b> (model card sampling). Install exllamav3 first (<a href="https://github.com/turboderp-org/exllamav3/releases">prebuilt wheels</a>):
|
| <pre><code>git clone https://github.com/turboderp-org/exllamav3 && cd exllamav3
|
| pip install pyperclip prompt_toolkit
|
| python examples/chat.py -m ../AesCode-32B-EXL3 -mode chatml -cs 24576 -maxr 12000 -temp 0.8 -topp 0.95 -topk 20 -minp 0</code></pre>
|
| <b>One shot to file:</b>
|
| <pre><code>python examples/chat.py -m ../AesCode-32B-EXL3 -mode chatml -cs 24576 -maxr 12000 -temp 0.8 -topp 0.95 -topk 20 -minp 0 -prompt "Create a portrait poster infographic about coffee origins as one complete HTML page." -save page.md</code></pre>
|
| <b>With a reference image</b> (how AesCode is meant to be used, based on <code>examples/multimodal.py</code>):
|
| <pre><code>from exllamav3 import Cache, ComboSampler, Config, Generator, Model, Tokenizer
|
| from PIL import Image
|
|
|
| config = Config.from_directory("AesCode-32B-EXL3")
|
| vision = Model.from_config(config, component="vision"); vision.load()
|
| model = Model.from_config(config); cache = Cache(model, max_num_tokens=24576); model.load()
|
| tok = Tokenizer.from_config(config)
|
| emb = vision.get_image_embeddings(tokenizer=tok, image=Image.open("reference.png"))
|
| prompt = ("<|im_start|>user\n" + emb.text_alias + "Create a landscape dashboard ... as one complete HTML page."
|
| "<|im_end|>\n<|im_start|>assistant\n")
|
| gen = Generator(model=model, cache=cache, tokenizer=tok)
|
| print(gen.generate(prompt=prompt, embeddings=[emb], encode_special_tokens=True, completion_only=True,
|
| max_new_tokens=12000, stop_conditions=["<|im_end|>"],
|
| sampler=ComboSampler(temperature=0.8, top_p=0.95, top_k=20)))</code></pre>
|
| <p>The repo's training system prompt and spec template (<a href="https://github.com/microsoft/AesCode/blob/main/inference/prompts.py">inference/prompts.py</a>) give the best results.</p>
|
| </div>
|
| </details>
|
| </div>
|
|
|
| <div class="card-dark">
|
| <h2>Example output (4.0bpw_H6)</h2>
|
| <img src="example_dashboard.jpg" alt="Reference image vs EXL3 output" style="width:100%; border-radius: 8px;" />
|
| <p class="subtitle">Repo example <code>hospital_dashboard</code> with the training prompt template and reference image. Exact text 11/11, chart data 5/5, table 0/1, 1 element outside the 1280x720 canvas. 5097 tokens at 58.0 tok/s on an RTX 5090.</p>
|
| </div>
|
|
|
| <div class="card-dark">
|
| <h2>Build Details</h2>
|
| <details>
|
| <summary>How these were made</summary>
|
| <div class="details-content">
|
| <p><b>Base quants:</b> <code>convert.py -b <bpw></code> (3.0, 4.0, 5.0), head 6bpw, vision 6bpw. Calibration: 250 rows x 2048 (default) for 3.0, 128 rows x 2048 for 4.0/5.0 (32 GB RAM build box)</p>
|
| <p><b>KLD measurement:</b> <code>util/measure.py -r <bf16> -i <3.0> <4.0> <5.0></code> (<code>measurement.json</code> on main)</p>
|
| <p><b>Optimized:</b> <code>util/optimize.py -i <3.0> <4.0> <5.0> -m measurement.json -b <target></code> then <code>util/recompile.py -or override.yaml</code> with <code>*.self_attn.*</code> from 5.0bpw</p>
|
| <p><b>Eval:</b> <code>eval/qbench.py</code>, BF16 streamed as reference</p>
|
| <p>Docs: <a href="https://github.com/turboderp-org/exllamav3/blob/master/doc/convert.md">exllamav3 convert.md</a></p>
|
| </div>
|
| </details>
|
| </div>
|
|
|
| <div class="card-dark">
|
| <h2>Files</h2>
|
| <p><code>main</code> branch: <code>measurement.json</code>, KLD plots, example image, README</p>
|
| <p>Each bpw branch: quantized shards (text + vision) + config + tokenizer + processor configs</p>
|
| </div>
|
|
|
| </div>
|
|
|
