---
quantized_by: bartowski
pipeline_tag: image-text-to-text
base_model_relation: quantized
license: apache-2.0
language:
- en
tags:
- code-generation
- html
- multimodal
- reinforcement-learning
- design
base_model: microsoft/AesCode-32B
---
## Llamacpp imatrix Quantizations of AesCode-32B by microsoft
Using llama.cpp release b11490 for quantization.
Original model: https://huggingface.co/microsoft/AesCode-32B
**Model details:**
- Parameter count: 33B (source checkpoint)
- Input support: text, image (with mmproj file) - [details](#multimodal)
- Speculative decoding: no
- imatrix: yes - [details](#imatrix)
[How to run](#how-to-run)
## Prompt format
```
<|im_start|>system
{system_prompt}<|im_end|>
<|im_start|>user
{prompt}<|im_end|>
<|im_start|>assistant
```
Prompt format with tool definitions
```
<|im_start|>system
{system_prompt}
# Tools
You may call one or more functions to assist with the user query.
You are provided with function signatures within XML tags:
{"type": "function", "function": {"name": "get_stock_price", "description": "Get the current stock price", "parameters": {"type": "object", "properties": {"symbol": {"type": "string", "description": "The stock symbol, e.g. AAPL, GOOG"}}, "required": ["symbol"]}}}
For each function call, return a json object with function name and arguments within XML tags:
{"name": , "arguments": }
<|im_end|>
<|im_start|>user
{prompt}<|im_end|>
<|im_start|>assistant
```
Click to view download instructions
First, make sure you have the Hugging Face CLI installed:
```
pip install -U "huggingface_hub[cli]"
```
Download a specific file:
```
hf download bartowski/AesCode-32B-GGUF --include "AesCode-32B-Q4_K_M.gguf" --local-dir ./
```
The files marked `true` in the Split column above are stored as multiple parts in a folder. To download all the parts to a local folder, run:
```
hf download bartowski/AesCode-32B-GGUF --include "AesCode-32B-bf16/*" --local-dir ./
```
You can either specify a new local-dir (AesCode-32B-bf16) or download them all in place (./)
## How to run
These quants run with [llama.cpp](https://github.com/ggml-org/llama.cpp) - installable in one line via [llama.app](https://llama.app/):
```
curl -LsSf https://llama.app/install.sh | sh
llama-server -hf bartowski/AesCode-32B-GGUF:Q4_K_M
```
llama-server includes a built-in chat web UI, served at http://localhost:8080 by default.
These quants were made with llama.cpp release b11490 - if this model's architecture is newly supported, you'll need that release or newer to run them.
They also work in: [LM Studio](https://lmstudio.ai/) · [koboldcpp](https://github.com/LostRuins/koboldcpp) · [ramalama](https://github.com/containers/ramalama) · [Jan AI](https://www.jan.ai/) · [Text Generation Web UI](https://github.com/oobabooga/text-generation-webui) · [LoLLMs](https://github.com/ParisNeo/lollms) · [Atomic Chat](https://atomic.chat/)
## Multimodal
This model supports image input. Alongside the quants, this repo includes the multimodal projector files [mmproj-AesCode-32B-bf16.gguf](https://huggingface.co/bartowski/AesCode-32B-GGUF/blob/main/mmproj-AesCode-32B-bf16.gguf) and [mmproj-AesCode-32B-f16.gguf](https://huggingface.co/bartowski/AesCode-32B-GGUF/blob/main/mmproj-AesCode-32B-f16.gguf), which pair with any quant above.
llama.cpp downloads the mmproj automatically when using `-hf` as shown above; if you're loading files manually, pass it with `--mmproj`.
## imatrix
All quants made using imatrix option, with a calibration corpus rendered through this model's own chat template. The corpus pairs plain prose with tool-calling and reasoning conversations ([corpus source data](https://gist.github.com/bartowski1182/e26453c0404e24eb317543ec5360f87a)), encoded exactly as this model sees them at inference and processed with `--parse-special`, so chat-format special tokens contribute to the importance matrix. The corpus rendered for this model is included in this repo: [AesCode-32B-calibration-v6.txt](https://huggingface.co/bartowski/AesCode-32B-GGUF/blob/main/AesCode-32B-calibration-v6.txt). The imatrix is available here: [AesCode-32B-imatrix.gguf](https://huggingface.co/bartowski/AesCode-32B-GGUF/blob/main/AesCode-32B-imatrix.gguf).
Calibration render details
```json
{
"recipe": "calibration-v6",
"encoder": "chat_template",
"library_versions": {
"transformers": "5.9.0",
"tokenizers": "0.22.2",
"tiktoken": "0.14.0",
"blobfile": "3.3.0",
"huggingface_hub": "1.31.0"
},
"chunk_size": 512,
"prose_chunks": 218,
"tool_chunks": 268,
"total_chunks": 486,
"n_conversations": 164,
"conversation_token_lengths": [
319,
847,
573,
1180,
806,
635,
719,
2084,
1087,
410,
612,
967,
810,
1200,
481,
669,
2076,
771,
696,
633,
695,
367,
316,
637,
667,
615,
1009,
580,
547,
785,
1530,
807,
637,
477,
622,
1229,
999,
455,
509,
718,
425,
1262,
1075,
703,
244,
778,
645,
646,
984,
1545,
879,
1713,
680,
944,
536,
2326,
2079,
450,
2395,
489,
666,
456,
332,
377,
1978,
518,
607,
661,
636,
729,
579,
716,
651,
559,
706,
689,
595,
1115,
401,
786,
1583,
816,
265,
130,
345,
731,
2760,
1947,
357,
1277,
472,
621,
500,
718,
317,
691,
398,
791,
621,
846,
1036,
799,
636,
846,
827,
539,
617,
368,
483,
1158,
2144,
2082,
328,
318,
1248,
847,
844,
2071,
1357,
613,
295,
581,
1115,
1091,
1406,
1115,
522,
676,
508,
242,
651,
2069,
406,
926,
418,
435,
925,
248,
1071,
458,
147,
124,
1977,
452,
650,
1695,
1386,
2020,
2057,
467,
636,
497,
528,
620,
563,
358,
786,
548,
404,
1095,
617,
516,
739,
1011
]
}
```
## Embed/output weights
Some of these quants (Q3_K_XL, Q4_K_L etc) are the standard quantization method with the embeddings and output weights quantized to Q8_0 instead of what they would normally default to.
## ARM/AVX information
llama.cpp automatically "repacks" weights into an interleaved layout at load time for faster inference on ARM and AVX machines - details in [this PR](https://github.com/ggml-org/llama.cpp/pull/9921). This once required downloading special Q4_0_4_4/4_8/8_8 files; those are long gone. Online repacking now covers Q4_0, IQ4_NL, and most K-quants, so no special quant choice is needed for CPU inference.
## Which file should I choose?
Click here for details
An older (early 2024) but still useful write-up with charts comparing quant performances is provided by Artefact2 [here](https://gist.github.com/Artefact2/b5f810600771265fc1e39442288e8ec9)
The first thing to figure out is how big a model you can run. To do this, you'll need to figure out how much RAM and/or VRAM you have.
If you want your model running as FAST as possible, you'll want to fit the whole thing on your GPU's VRAM. Aim for a quant with a file size 1-2GB smaller than your GPU's total VRAM.
If you want the absolute maximum quality, add both your system RAM and your GPU's VRAM together, then similarly grab a quant with a file size 1-2GB Smaller than that total.
Hugging Face can also do this math for you: add your hardware in your [Local Apps settings](https://huggingface.co/settings/local-apps) and the model page will show which files fit.
Next, you'll need to decide if you want to use an 'I-quant' or a 'K-quant'.
If you don't want to think too much, grab one of the K-quants. These are in format 'QX_K_X', like Q5_K_M.
If you want to get more into the weeds, you can check out this extremely useful feature chart:
[llama.cpp feature matrix](https://github.com/ggml-org/llama.cpp/wiki/Feature-matrix)
But basically, if you're aiming for below Q4, and you're running cuBLAS (Nvidia) or rocBLAS (AMD), you should look towards the I-quants. These are in format IQX_X, like IQ3_M. These are newer and offer better performance for their size.
These I-quants can also be used on CPU, but will be slower than their K-quant equivalent, so speed vs performance is a tradeoff you'll have to decide.
## Credits
Thank you kalomaze and Dampf for assistance in creating the imatrix calibration dataset.
Thank you ZeroWw for the inspiration to experiment with embed/output.
Want to support my work? Visit my ko-fi page here: https://ko-fi.com/bartowski