How to Run Aion 1.0 Instruct (2.3B) Locally

Importance-matrix GGUF files of Microsoft's Aion 1.0 Instruct, the small model behind the coming Windows on-device SLM, for llama.cpp.

Atomic Chat Discord GitHub X
  • Built by the team behind Atomic Chat, the app that runs open-weight models locally and offline.
  • Without refusals: Aion-1.0-Instruct-2.3B-Uncensored-GGUF (baked files and a LoRA for these quants, with the measurements).
  • Needs llama.cpp with Aion support. Upstream llama.cpp, LM Studio and Ollama do not load Aion yet: build llama.cpp with our port, aion-port.patch in this repo (instructions below).

Aion 1.0 Instruct is a 2.3B small language model from Microsoft's Windows Applied Sciences Group, MIT-licensed. Microsoft's card says a further optimized version of it, with a content moderation model, becomes the Windows OS small language model by the end of January 2027. It is trained for summarization, rewriting, document question answering and chat, with a 32K context and eight languages (English, German, Spanish, Portuguese, Italian, French, Chinese, Japanese). It uses a new architecture that llama.cpp does not support yet, so we wrote the support ourselves and checked it against Microsoft's reference code. This repo has GGUF files for llama.cpp built on that port, quantized with an importance matrix.

Pick a file

AD (Atomic Dynamic) files quantize the feed-forward weights with an importance matrix and keep the parts the model is sensitive to higher: attention K/V at Q8_0, the first and last blocks one step up, the tied embedding (which is also the output head) at Q5_K to Q8_0. File names are Aion-1.0-Instruct-2.3B-<name>.gguf.

file GB on disk bpw role mean KLD 99th pct KLD same top token
BF16 4.59 16.0 the original weights, the reference 0 0 100%
Q8_0 2.44 8.50 closest to the original 0.0029 0.020 96.6%
AD-Q5_K_M 1.78 6.18 near Q8_0 for 0.66 GB less 0.0062 0.039 95.1%
AD-Q4_K_M 1.58 5.48 our pick 0.0131 0.081 93.0%
AD-IQ3_S 1.29 4.49 smallest we publish 0.0705 0.401 84.1%

The KV cache is small too: at 32K tokens about 1.1 GB at F16, because only the 7 full-attention layers keep all 32K tokens (0.94 GB) and the 21 sliding-window layers keep 2,048 (0.18 GB).

"Same top token" is how often the file picks the same next token as the BF16 original; "mean KLD" is the KL divergence of its next-token distribution from BF16's. Both are measured with llama-perplexity on our 30-language neutral corpus (80 chunks of 4096 tokens) against the BF16 GGUF, on llama.cpp bafa0420 with our port, 1x A10 24 GB. All numbers, logs and the importance matrix are in AtomicChat/Aion-1.0-Instruct-2.3B-GGUF-metrics.

We also built AD-Q6_K (2.14 GB, KLD 0.0054, 95.3%) and AD-IQ4_XS (1.53 GB, KLD 0.0193, 91.4%) and do not publish them: the first is barely closer than AD-Q5_K_M for 0.36 GB more, the second saves 50 MB over AD-Q4_K_M at half again its divergence.

Against llama.cpp's own preset. No other GGUF of Aion existed when we published. On the same stand, llama.cpp's Q4_K_M preset without an importance matrix (1.47 GB) measures mean KLD 0.112 and picks the original's token 80.4% of the time. AD-Q4_K_M is 0.11 GB larger and has 88% lower divergence (0.0131, 93.0%); AD-IQ3_S is 0.18 GB smaller than that preset and still closer to the original (0.0705, 84.1%).

One part of every file is quantized without calibration: the tied embedding, 18% of the parameters. llama.cpp reads it as a lookup table for the input and does not record the output head in the importance matrix, so it gets a type that holds up without one (Q8_0 to Q5_K).

Run it

Build llama.cpp at bafa0420 with aion-port.patch from this repo:

git clone https://github.com/ggml-org/llama.cpp && cd llama.cpp
git checkout bafa0420c1e57edf04d71587d32fe57d45df7b95
curl -LO https://huggingface.co/AtomicChat/Aion-1.0-Instruct-2.3B-GGUF/resolve/main/aion-port.patch
git apply aion-port.patch
cmake -B build -DGGML_CUDA=ON   # Mac: drop -DGGML_CUDA=ON (Metal is on by default)
cmake --build build -j --config Release --target llama-server llama-cli

The chat template (system preamble with the current date, tools) is embedded, so pass --jinja:

./build/bin/llama-server -hf AtomicChat/Aion-1.0-Instruct-2.3B-GGUF:AD-Q4_K_M --jinja -c 32768 -ngl 99

Tool calls come back as OpenAI-style tool_calls. Microsoft's card notes that the model is not trained for world knowledge; it is meant for working on text you give it.

Speed on llama.cpp bafa0420 with the port, 1x A10 24 GB, llama-bench with flash attention:

file prompt, 512 tokens prompt, 4096 tokens generation
Q8_0 9,171 t/s 8,383 t/s 158 t/s
AD-Q5_K_M 8,637 t/s 7,914 t/s 196 t/s
AD-Q4_K_M 9,059 t/s 8,239 t/s 210 t/s
AD-IQ3_S 9,455 t/s 8,550 t/s 217 t/s

Checked by hand in llama-server --jinja with AD-Q4_K_M and AD-IQ3_S: a one-sentence summary that stops on its own, answers in Russian and Chinese, and a tool call returned as tool_calls. With AD-Q4_K_M at -c 32768 the model found a sentence planted at 10%, 50% and 90% depth of a 19.9K-token document (3 of 3).

When llama.cpp adds Aion upstream, we will re-check these files against the merged version.

How we built them

  • Port: a new llama.cpp architecture aion: Qwen3's decoder math (per-head QK RMSNorm, SwiGLU, tied embeddings) plus Aion's hybrid attention: 21 of 28 layers use a 2048-token sliding window, 7 use full attention, and only the full-attention layers apply YaRN (x16 from 2048) to RoPE. Checked against Microsoft's transformers code on the same tokens (4 chunks of 4096, so the window and YaRN are exercised): the F32 GGUF matches the fp32 reference at mean KLD 0.00005 and the same top token 99.6% of the time; the BF16 GGUF at 0.00009 / 99.3%. The tokenizer matches Hugging Face's on 5,599 test strings.
  • Base: BF16 GGUF converted from Microsoft's BF16 safetensors (revision 1423a073); the values are copied, not rounded.
  • Importance matrix: computed on the BF16 over our public calibration build (4.86M tokens: agentic, code, reasoning, 30 languages, long context, a sweep of the vocabulary), rendered through Aion's own chat template and tokenized with --parse-special.
  • Layout: profile dense-llama of our quantization pipeline. The FFN carries most of the bytes and takes the target type, the first and last blocks one step higher; attention K/V stay Q8_0; the tied embedding, which is also the output head, takes the head's type (Q8_0 or Q6_K, Q5_K on the 3-bit file).

Model facts

parameters 2.29B (0.41B of them the tied embedding / head)
layers 28: 21 sliding-window (2048), 7 full attention
attention GQA, 24 query heads, 8 KV heads, head dim 128, QK RMSNorm
hidden / FFN 2048 / 8192 (SwiGLU)
RoPE theta 32000; YaRN x16 from 2048 on the full-attention layers
context 32K
vocabulary 200,029 (o200k)
license MIT (LICENSE in this repo, from the base model)

Quantized by AtomicChat.

Downloads last month
418
GGUF
Model size
2B params
Architecture
aion
Hardware compatibility
Log In to add your hardware

3-bit

4-bit

5-bit

8-bit

16-bit

Inference Providers NEW
This model isn't deployed by any Inference Provider. ๐Ÿ™‹ Ask for provider support

Model tree for AtomicChat/Aion-1.0-Instruct-2.3B-GGUF

Quantized
(4)
this model

Collection including AtomicChat/Aion-1.0-Instruct-2.3B-GGUF