File size: 13,106 Bytes
9021aa9 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 | ---
license: apache-2.0
base_model: microsoft/AesCode-32B
base_model_relation: quantized
quantized_by: WeReCooking
pipeline_tag: image-text-to-text
tags:
- exl3
- qwen3_vl
- html
---
<style>
.container-dark { font-family: -apple-system, BlinkMacSystemFont, "Segoe UI", Roboto, Arial, sans-serif; line-height: 1.6; color: #d4d4d4; }
a { color: #569cd6; text-decoration: none; font-weight: 600; }
a:hover { text-decoration: underline; }
.card-dark { background-color: #252526; border-radius: 12px; padding: 24px; margin-bottom: 20px; box-shadow: 0 4px 12px rgba(0,0,0,0.3); border: 1px solid #3c3c3c; }
.card-dark h1 { font-size: 2.2em; color: #ffffff; text-align: center; margin-bottom: 10px; }
.card-dark .subtitle { text-align: center; font-size: 1.1em; color: #a0a0a0; }
.card-dark h2 { font-size: 1.5em; margin-top: 0; padding-bottom: 10px; border-bottom: 1px solid #3c3c3c; color: #c586c0; }
.styled-table { display: table; border: none; width: 100%; font-size: 0.95em; }
.styled-table thead th { background-color: #333333; color: #c586c0; text-align: left; padding: 12px 15px; }
.styled-table td { padding: 0; border-bottom: 1px solid #3c3c3c; }
.styled-table tbody tr { transition: background-color 0.1s ease; }
.styled-table tbody tr:hover { background-color: #3a3a3a; }
.styled-table tr:last-child td { border-bottom: none; }
.styled-table td a { display: block; padding: 12px 15px; }
.styled-table td a.fake-link { text-decoration:none; color:inherit; }
details { margin-top: 20px; border: 1px solid #3c3c3c; border-radius: 8px; overflow: hidden; }
summary { cursor: pointer; padding: 12px 18px; background-color: #6A5ACD; font-weight: 600; display: flex; align-items: center; gap: 10px; justify-content: space-between; list-style: none; }
summary::-webkit-details-marker { display: none; }
summary:hover { filter: brightness(1.1); }
summary::after { content: ''; display: inline-block; width: 8px; height: 8px; border-bottom: 2px solid white; border-right: 2px solid white; transform: rotate(45deg); transition: transform 0.3s ease; }
details[open] > summary::after { transform: rotate(225deg); }
.details-content { padding: 18px; }
</style>
<div class="container-dark">
<div class="card-dark">
<h1>AesCode-32B EXL3</h1>
<p class="subtitle">
EXL3 quants of <a href="https://huggingface.co/microsoft/AesCode-32B">microsoft/AesCode-32B</a>
using <a href="https://github.com/turboderp-org/exllamav3/">exllamav3</a> v1.6.1
</p>
</div>
<div class="card-dark">
<h2>KL Divergence vs VRAM</h2>
<img src="kld_plot.png" alt="KLD plot" style="width:100%; border-radius: 8px;" />
<p class="subtitle">Reference: BF16 original (PPL 6.843). Noise floor KLD 0.000151. Measured with exllamav3 <code>eval/qbench.py</code> on wikitext-2 (20 rows, 2048 ctx). VRAM = decoder + lm_head weights.</p>
<details>
<summary>KLD spread (median, IQR, noise floor)</summary>
<div class="details-content">
<img src="kld_spread_plot.png" alt="KLD spread" style="width:100%; border-radius: 8px;" />
</div>
</details>
</div>
<div class="card-dark">
<h2>Quants</h2>
<table class="styled-table">
<thead>
<tr><th>Branch</th><th>BPW</th><th>Head</th><th>VRAM (GiB)</th><th>KLD mean</th><th>KLD median</th><th>PPL</th><th>Type</th></tr>
</thead>
<tbody>
<tr>
<td><a href="https://huggingface.co/WeReCooking/AesCode-32B-EXL3/tree/3.0bpw_H6">3.0bpw_H6</a></td>
<td><a class="fake-link" href="https://huggingface.co/WeReCooking/AesCode-32B-EXL3/tree/3.0bpw_H6">3.0</a></td>
<td><a class="fake-link" href="https://huggingface.co/WeReCooking/AesCode-32B-EXL3/tree/3.0bpw_H6">6</a></td>
<td><a class="fake-link" href="https://huggingface.co/WeReCooking/AesCode-32B-EXL3/tree/3.0bpw_H6">11.5</a></td>
<td><a class="fake-link" href="https://huggingface.co/WeReCooking/AesCode-32B-EXL3/tree/3.0bpw_H6">0.0360</a></td>
<td><a class="fake-link" href="https://huggingface.co/WeReCooking/AesCode-32B-EXL3/tree/3.0bpw_H6">0.01522</a></td>
<td><a class="fake-link" href="https://huggingface.co/WeReCooking/AesCode-32B-EXL3/tree/3.0bpw_H6">6.987</a></td>
<td><a class="fake-link" href="https://huggingface.co/WeReCooking/AesCode-32B-EXL3/tree/3.0bpw_H6">base</a></td>
</tr>
<tr>
<td><a href="https://huggingface.co/WeReCooking/AesCode-32B-EXL3/tree/3.73bpw_H6">3.73bpw_H6</a></td>
<td><a class="fake-link" href="https://huggingface.co/WeReCooking/AesCode-32B-EXL3/tree/3.73bpw_H6">3.73</a></td>
<td><a class="fake-link" href="https://huggingface.co/WeReCooking/AesCode-32B-EXL3/tree/3.73bpw_H6">6</a></td>
<td><a class="fake-link" href="https://huggingface.co/WeReCooking/AesCode-32B-EXL3/tree/3.73bpw_H6">14.1</a></td>
<td><a class="fake-link" href="https://huggingface.co/WeReCooking/AesCode-32B-EXL3/tree/3.73bpw_H6">0.0157</a></td>
<td><a class="fake-link" href="https://huggingface.co/WeReCooking/AesCode-32B-EXL3/tree/3.73bpw_H6">0.00616</a></td>
<td><a class="fake-link" href="https://huggingface.co/WeReCooking/AesCode-32B-EXL3/tree/3.73bpw_H6">6.902</a></td>
<td><a class="fake-link" href="https://huggingface.co/WeReCooking/AesCode-32B-EXL3/tree/3.73bpw_H6">optimized</a></td>
</tr>
<tr>
<td><a href="https://huggingface.co/WeReCooking/AesCode-32B-EXL3/tree/4.0bpw_H6">4.0bpw_H6</a></td>
<td><a class="fake-link" href="https://huggingface.co/WeReCooking/AesCode-32B-EXL3/tree/4.0bpw_H6">4.0</a></td>
<td><a class="fake-link" href="https://huggingface.co/WeReCooking/AesCode-32B-EXL3/tree/4.0bpw_H6">6</a></td>
<td><a class="fake-link" href="https://huggingface.co/WeReCooking/AesCode-32B-EXL3/tree/4.0bpw_H6">15.1</a></td>
<td><a class="fake-link" href="https://huggingface.co/WeReCooking/AesCode-32B-EXL3/tree/4.0bpw_H6">0.0093</a></td>
<td><a class="fake-link" href="https://huggingface.co/WeReCooking/AesCode-32B-EXL3/tree/4.0bpw_H6">0.00392</a></td>
<td><a class="fake-link" href="https://huggingface.co/WeReCooking/AesCode-32B-EXL3/tree/4.0bpw_H6">6.876</a></td>
<td><a class="fake-link" href="https://huggingface.co/WeReCooking/AesCode-32B-EXL3/tree/4.0bpw_H6">base</a></td>
</tr>
<tr>
<td><a href="https://huggingface.co/WeReCooking/AesCode-32B-EXL3/tree/4.54bpw_H6">4.54bpw_H6</a></td>
<td><a class="fake-link" href="https://huggingface.co/WeReCooking/AesCode-32B-EXL3/tree/4.54bpw_H6">4.54</a></td>
<td><a class="fake-link" href="https://huggingface.co/WeReCooking/AesCode-32B-EXL3/tree/4.54bpw_H6">6</a></td>
<td><a class="fake-link" href="https://huggingface.co/WeReCooking/AesCode-32B-EXL3/tree/4.54bpw_H6">17.0</a></td>
<td><a class="fake-link" href="https://huggingface.co/WeReCooking/AesCode-32B-EXL3/tree/4.54bpw_H6">0.0065</a></td>
<td><a class="fake-link" href="https://huggingface.co/WeReCooking/AesCode-32B-EXL3/tree/4.54bpw_H6">0.00260</a></td>
<td><a class="fake-link" href="https://huggingface.co/WeReCooking/AesCode-32B-EXL3/tree/4.54bpw_H6">6.863</a></td>
<td><a class="fake-link" href="https://huggingface.co/WeReCooking/AesCode-32B-EXL3/tree/4.54bpw_H6">optimized</a></td>
</tr>
<tr>
<td><a href="https://huggingface.co/WeReCooking/AesCode-32B-EXL3/tree/5.0bpw_H6">5.0bpw_H6</a></td>
<td><a class="fake-link" href="https://huggingface.co/WeReCooking/AesCode-32B-EXL3/tree/5.0bpw_H6">5.0</a></td>
<td><a class="fake-link" href="https://huggingface.co/WeReCooking/AesCode-32B-EXL3/tree/5.0bpw_H6">6</a></td>
<td><a class="fake-link" href="https://huggingface.co/WeReCooking/AesCode-32B-EXL3/tree/5.0bpw_H6">18.7</a></td>
<td><a class="fake-link" href="https://huggingface.co/WeReCooking/AesCode-32B-EXL3/tree/5.0bpw_H6">0.0027</a></td>
<td><a class="fake-link" href="https://huggingface.co/WeReCooking/AesCode-32B-EXL3/tree/5.0bpw_H6">0.00122</a></td>
<td><a class="fake-link" href="https://huggingface.co/WeReCooking/AesCode-32B-EXL3/tree/5.0bpw_H6">6.871</a></td>
<td><a class="fake-link" href="https://huggingface.co/WeReCooking/AesCode-32B-EXL3/tree/5.0bpw_H6">base</a></td>
</tr>
</tbody>
</table>
<p class="subtitle">Bases are direct converts. Optimized = KLD-guided tensor mix of the bases + attention recompiled at 5bpw; on this model they land on the base size/KLD curve, so treat them as in-between sizes and pick by VRAM. Vision tower 6bpw in every branch.</p>
</div>
<div class="card-dark">
<h2>Download and run</h2>
<details>
<summary>Download</summary>
<div class="details-content">
<pre><code>pip install -U "huggingface_hub"
hf download WeReCooking/AesCode-32B-EXL3 --revision 4.0bpw_H6 --local-dir ./AesCode-32B-EXL3</code></pre>
</div>
</details>
<details>
<summary>Run with exllamav3</summary>
<div class="details-content">
<b>Text chat, official exllamav3 CLI</b> (model card sampling). Install exllamav3 first (<a href="https://github.com/turboderp-org/exllamav3/releases">prebuilt wheels</a>):
<pre><code>git clone https://github.com/turboderp-org/exllamav3 && cd exllamav3
pip install pyperclip prompt_toolkit
python examples/chat.py -m ../AesCode-32B-EXL3 -mode chatml -cs 24576 -maxr 12000 -temp 0.8 -topp 0.95 -topk 20 -minp 0</code></pre>
<b>One shot to file:</b>
<pre><code>python examples/chat.py -m ../AesCode-32B-EXL3 -mode chatml -cs 24576 -maxr 12000 -temp 0.8 -topp 0.95 -topk 20 -minp 0 -prompt "Create a portrait poster infographic about coffee origins as one complete HTML page." -save page.md</code></pre>
<b>With a reference image</b> (how AesCode is meant to be used, based on <code>examples/multimodal.py</code>):
<pre><code>from exllamav3 import Cache, ComboSampler, Config, Generator, Model, Tokenizer
from PIL import Image
config = Config.from_directory("AesCode-32B-EXL3")
vision = Model.from_config(config, component="vision"); vision.load()
model = Model.from_config(config); cache = Cache(model, max_num_tokens=24576); model.load()
tok = Tokenizer.from_config(config)
emb = vision.get_image_embeddings(tokenizer=tok, image=Image.open("reference.png"))
prompt = ("<|im_start|>user\n" + emb.text_alias + "Create a landscape dashboard ... as one complete HTML page."
"<|im_end|>\n<|im_start|>assistant\n")
gen = Generator(model=model, cache=cache, tokenizer=tok)
print(gen.generate(prompt=prompt, embeddings=[emb], encode_special_tokens=True, completion_only=True,
max_new_tokens=12000, stop_conditions=["<|im_end|>"],
sampler=ComboSampler(temperature=0.8, top_p=0.95, top_k=20)))</code></pre>
<p>The repo's training system prompt and spec template (<a href="https://github.com/microsoft/AesCode/blob/main/inference/prompts.py">inference/prompts.py</a>) give the best results.</p>
</div>
</details>
</div>
<div class="card-dark">
<h2>Example output (4.0bpw_H6)</h2>
<img src="example_dashboard.jpg" alt="Reference image vs EXL3 output" style="width:100%; border-radius: 8px;" />
<p class="subtitle">Repo example <code>hospital_dashboard</code> with the training prompt template and reference image. Exact text 11/11, chart data 5/5, table 0/1, 1 element outside the 1280x720 canvas. 5097 tokens at 58.0 tok/s on an RTX 5090.</p>
</div>
<div class="card-dark">
<h2>Build Details</h2>
<details>
<summary>How these were made</summary>
<div class="details-content">
<p><b>Base quants:</b> <code>convert.py -b <bpw></code> (3.0, 4.0, 5.0), head 6bpw, vision 6bpw. Calibration: 250 rows x 2048 (default) for 3.0, 128 rows x 2048 for 4.0/5.0 (32 GB RAM build box)</p>
<p><b>KLD measurement:</b> <code>util/measure.py -r <bf16> -i <3.0> <4.0> <5.0></code> (<code>measurement.json</code> on main)</p>
<p><b>Optimized:</b> <code>util/optimize.py -i <3.0> <4.0> <5.0> -m measurement.json -b <target></code> then <code>util/recompile.py -or override.yaml</code> with <code>*.self_attn.*</code> from 5.0bpw</p>
<p><b>Eval:</b> <code>eval/qbench.py</code>, BF16 streamed as reference</p>
<p>Docs: <a href="https://github.com/turboderp-org/exllamav3/blob/master/doc/convert.md">exllamav3 convert.md</a></p>
</div>
</details>
</div>
<div class="card-dark">
<h2>Files</h2>
<p><code>main</code> branch: <code>measurement.json</code>, KLD plots, example image, README</p>
<p>Each bpw branch: quantized shards (text + vision) + config + tokenizer + processor configs</p>
</div>
</div>
|