Instructions to use AtomicChat/Aion-1.0-Instruct-2.3B-GGUF with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- llama.cpp
How to use AtomicChat/Aion-1.0-Instruct-2.3B-GGUF with llama.cpp:
Install (macOS, Linux)
curl -LsSf https://llama.app/install.sh | sh # Start a local OpenAI-compatible server with a web UI: llama serve -hf AtomicChat/Aion-1.0-Instruct-2.3B-GGUF:Q4_K_M # Run inference directly in the terminal: llama cli -hf AtomicChat/Aion-1.0-Instruct-2.3B-GGUF:Q4_K_M
Install from WinGet (Windows)
winget install llama.cpp # Start a local OpenAI-compatible server with a web UI: llama serve -hf AtomicChat/Aion-1.0-Instruct-2.3B-GGUF:Q4_K_M # Run inference directly in the terminal: llama cli -hf AtomicChat/Aion-1.0-Instruct-2.3B-GGUF:Q4_K_M
Build from source code
git clone https://github.com/ggml-org/llama.cpp.git cd llama.cpp cmake -B build cmake --build build -j --target llama-app # Start a local OpenAI-compatible server with a web UI: ./build/bin/llama serve -hf AtomicChat/Aion-1.0-Instruct-2.3B-GGUF:Q4_K_M # Run inference directly in the terminal: ./build/bin/llama cli -hf AtomicChat/Aion-1.0-Instruct-2.3B-GGUF:Q4_K_M
Use Docker
docker model run hf.co/AtomicChat/Aion-1.0-Instruct-2.3B-GGUF:Q4_K_M
- Llama new
- LM Studio
- Jan
- vLLM
How to use AtomicChat/Aion-1.0-Instruct-2.3B-GGUF with vLLM:
Install from pip and serve model
# Install vLLM from pip: pip install vllm # Start the vLLM server: vllm serve "AtomicChat/Aion-1.0-Instruct-2.3B-GGUF" # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:8000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "AtomicChat/Aion-1.0-Instruct-2.3B-GGUF", "messages": [ { "role": "user", "content": "What is the capital of France?" } ] }'Use Docker
docker model run hf.co/AtomicChat/Aion-1.0-Instruct-2.3B-GGUF:Q4_K_M
- Ollama
How to use AtomicChat/Aion-1.0-Instruct-2.3B-GGUF with Ollama:
ollama run hf.co/AtomicChat/Aion-1.0-Instruct-2.3B-GGUF:Q4_K_M
- Unsloth Desktop
- Pi
How to use AtomicChat/Aion-1.0-Instruct-2.3B-GGUF with Pi:
Start the llama.cpp server
# Install llama.cpp: brew install llama.cpp # Start a local OpenAI-compatible server: llama serve -hf AtomicChat/Aion-1.0-Instruct-2.3B-GGUF:Q4_K_M
Configure the model in Pi
# Install Pi: npm install -g @earendil-works/pi-coding-agent # Add to ~/.pi/agent/models.json: { "providers": { "llama-cpp": { "baseUrl": "http://localhost:9931/v1", "api": "openai-completions", "apiKey": "none", "models": [ { "id": "AtomicChat/Aion-1.0-Instruct-2.3B-GGUF:Q4_K_M" } ] } } }Run Pi
# Start Pi in your project directory: pi
- Docker Model Runner
How to use AtomicChat/Aion-1.0-Instruct-2.3B-GGUF with Docker Model Runner:
docker model run hf.co/AtomicChat/Aion-1.0-Instruct-2.3B-GGUF:Q4_K_M
- Lemonade
How to use AtomicChat/Aion-1.0-Instruct-2.3B-GGUF with Lemonade:
Pull the model
# Download Lemonade from https://lemonade-server.ai/ lemonade pull AtomicChat/Aion-1.0-Instruct-2.3B-GGUF:Q4_K_M
Run and chat with the model
lemonade run user.Aion-1.0-Instruct-2.3B-GGUF-Q4_K_M
List all available models
lemonade list
- Hermes Agent
How to use AtomicChat/Aion-1.0-Instruct-2.3B-GGUF with Hermes Agent:
Start the llama.cpp server
# Install llama.cpp: brew install llama.cpp # Start a local OpenAI-compatible server: llama serve -hf AtomicChat/Aion-1.0-Instruct-2.3B-GGUF:Q4_K_M
Configure Hermes
# Install Hermes: curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash hermes setup # Point Hermes at the local server: hermes config set model.provider custom hermes config set model.base_url http://127.0.0.1:9931/v1 hermes config set model.default AtomicChat/Aion-1.0-Instruct-2.3B-GGUF:Q4_K_M
Run Hermes
hermes
- Atomic Chat
- OpenClaw
How to use AtomicChat/Aion-1.0-Instruct-2.3B-GGUF with OpenClaw:
Start the llama.cpp server
# Install llama.cpp: brew install llama.cpp # Start a local OpenAI-compatible server: llama serve -hf AtomicChat/Aion-1.0-Instruct-2.3B-GGUF:Q4_K_M
Configure OpenClaw
# Install OpenClaw: npm install -g openclaw@latest # Register the local server and set it as the default model: openclaw onboard --non-interactive --mode local \ --auth-choice custom-api-key \ --custom-base-url http://127.0.0.1:9931/v1 \ --custom-model-id "AtomicChat/Aion-1.0-Instruct-2.3B-GGUF:Q4_K_M" \ --custom-provider-id llama-cpp \ --custom-compatibility openai \ --custom-text-input \ --accept-risk \ --skip-health
Run OpenClaw
openclaw agent --local --agent main --message "Hello from Hugging Face"
How to Run Aion 1.0 Instruct (2.3B) Locally
Importance-matrix GGUF files of Microsoft's Aion 1.0 Instruct, the small model behind the coming Windows on-device SLM, for llama.cpp.
- Built by the team behind Atomic Chat, the app that runs open-weight models locally and offline.
- Without refusals: Aion-1.0-Instruct-2.3B-Uncensored-GGUF (baked files and a LoRA for these quants, with the measurements).
- Needs llama.cpp with Aion support. Upstream llama.cpp, LM Studio and Ollama do not load Aion yet: build llama.cpp with our port,
aion-port.patchin this repo (instructions below).
Aion 1.0 Instruct is a 2.3B small language model from Microsoft's Windows Applied Sciences Group, MIT-licensed. Microsoft's card says a further optimized version of it, with a content moderation model, becomes the Windows OS small language model by the end of January 2027. It is trained for summarization, rewriting, document question answering and chat, with a 32K context and eight languages (English, German, Spanish, Portuguese, Italian, French, Chinese, Japanese). It uses a new architecture that llama.cpp does not support yet, so we wrote the support ourselves and checked it against Microsoft's reference code. This repo has GGUF files for llama.cpp built on that port, quantized with an importance matrix.
Pick a file
AD (Atomic Dynamic) files quantize the feed-forward weights with an importance matrix and keep the parts the model is sensitive to higher: attention K/V at Q8_0, the first and last blocks one step up, the tied embedding (which is also the output head) at Q5_K to Q8_0. File names are Aion-1.0-Instruct-2.3B-<name>.gguf.
| file | GB on disk | bpw | role | mean KLD | 99th pct KLD | same top token |
|---|---|---|---|---|---|---|
BF16 |
4.59 | 16.0 | the original weights, the reference | 0 | 0 | 100% |
Q8_0 |
2.44 | 8.50 | closest to the original | 0.0029 | 0.020 | 96.6% |
AD-Q5_K_M |
1.78 | 6.18 | near Q8_0 for 0.66 GB less | 0.0062 | 0.039 | 95.1% |
AD-Q4_K_M |
1.58 | 5.48 | our pick | 0.0131 | 0.081 | 93.0% |
AD-IQ3_S |
1.29 | 4.49 | smallest we publish | 0.0705 | 0.401 | 84.1% |
The KV cache is small too: at 32K tokens about 1.1 GB at F16, because only the 7 full-attention layers keep all 32K tokens (0.94 GB) and the 21 sliding-window layers keep 2,048 (0.18 GB).
"Same top token" is how often the file picks the same next token as the BF16 original; "mean KLD" is the KL divergence of its next-token distribution from BF16's. Both are measured with llama-perplexity on our 30-language neutral corpus (80 chunks of 4096 tokens) against the BF16 GGUF, on llama.cpp bafa0420 with our port, 1x A10 24 GB. All numbers, logs and the importance matrix are in AtomicChat/Aion-1.0-Instruct-2.3B-GGUF-metrics.
We also built AD-Q6_K (2.14 GB, KLD 0.0054, 95.3%) and AD-IQ4_XS (1.53 GB, KLD 0.0193, 91.4%) and do not publish them: the first is barely closer than AD-Q5_K_M for 0.36 GB more, the second saves 50 MB over AD-Q4_K_M at half again its divergence.
Against llama.cpp's own preset. No other GGUF of Aion existed when we published. On the same stand, llama.cpp's Q4_K_M preset without an importance matrix (1.47 GB) measures mean KLD 0.112 and picks the original's token 80.4% of the time. AD-Q4_K_M is 0.11 GB larger and has 88% lower divergence (0.0131, 93.0%); AD-IQ3_S is 0.18 GB smaller than that preset and still closer to the original (0.0705, 84.1%).
One part of every file is quantized without calibration: the tied embedding, 18% of the parameters. llama.cpp reads it as a lookup table for the input and does not record the output head in the importance matrix, so it gets a type that holds up without one (Q8_0 to Q5_K).
Run it
Build llama.cpp at bafa0420 with aion-port.patch from this repo:
git clone https://github.com/ggml-org/llama.cpp && cd llama.cpp
git checkout bafa0420c1e57edf04d71587d32fe57d45df7b95
curl -LO https://huggingface.co/AtomicChat/Aion-1.0-Instruct-2.3B-GGUF/resolve/main/aion-port.patch
git apply aion-port.patch
cmake -B build -DGGML_CUDA=ON # Mac: drop -DGGML_CUDA=ON (Metal is on by default)
cmake --build build -j --config Release --target llama-server llama-cli
The chat template (system preamble with the current date, tools) is embedded, so pass --jinja:
./build/bin/llama-server -hf AtomicChat/Aion-1.0-Instruct-2.3B-GGUF:AD-Q4_K_M --jinja -c 32768 -ngl 99
Tool calls come back as OpenAI-style tool_calls. Microsoft's card notes that the model is not trained for world knowledge; it is meant for working on text you give it.
Speed on llama.cpp bafa0420 with the port, 1x A10 24 GB, llama-bench with flash attention:
| file | prompt, 512 tokens | prompt, 4096 tokens | generation |
|---|---|---|---|
Q8_0 |
9,171 t/s | 8,383 t/s | 158 t/s |
AD-Q5_K_M |
8,637 t/s | 7,914 t/s | 196 t/s |
AD-Q4_K_M |
9,059 t/s | 8,239 t/s | 210 t/s |
AD-IQ3_S |
9,455 t/s | 8,550 t/s | 217 t/s |
Checked by hand in llama-server --jinja with AD-Q4_K_M and AD-IQ3_S: a one-sentence summary that stops on its own, answers in Russian and Chinese, and a tool call returned as tool_calls. With AD-Q4_K_M at -c 32768 the model found a sentence planted at 10%, 50% and 90% depth of a 19.9K-token document (3 of 3).
When llama.cpp adds Aion upstream, we will re-check these files against the merged version.
How we built them
- Port: a new llama.cpp architecture
aion: Qwen3's decoder math (per-head QK RMSNorm, SwiGLU, tied embeddings) plus Aion's hybrid attention: 21 of 28 layers use a 2048-token sliding window, 7 use full attention, and only the full-attention layers apply YaRN (x16 from 2048) to RoPE. Checked against Microsoft's transformers code on the same tokens (4 chunks of 4096, so the window and YaRN are exercised): the F32 GGUF matches the fp32 reference at mean KLD 0.00005 and the same top token 99.6% of the time; the BF16 GGUF at 0.00009 / 99.3%. The tokenizer matches Hugging Face's on 5,599 test strings. - Base: BF16 GGUF converted from Microsoft's BF16 safetensors (revision
1423a073); the values are copied, not rounded. - Importance matrix: computed on the BF16 over our public calibration build (4.86M tokens: agentic, code, reasoning, 30 languages, long context, a sweep of the vocabulary), rendered through Aion's own chat template and tokenized with
--parse-special. - Layout: profile
dense-llamaof our quantization pipeline. The FFN carries most of the bytes and takes the target type, the first and last blocks one step higher; attention K/V stay Q8_0; the tied embedding, which is also the output head, takes the head's type (Q8_0 or Q6_K, Q5_K on the 3-bit file).
Model facts
| parameters | 2.29B (0.41B of them the tied embedding / head) |
| layers | 28: 21 sliding-window (2048), 7 full attention |
| attention | GQA, 24 query heads, 8 KV heads, head dim 128, QK RMSNorm |
| hidden / FFN | 2048 / 8192 (SwiGLU) |
| RoPE | theta 32000; YaRN x16 from 2048 on the full-attention layers |
| context | 32K |
| vocabulary | 200,029 (o200k) |
| license | MIT (LICENSE in this repo, from the base model) |
Quantized by AtomicChat.
- Downloads last month
- 418
3-bit
4-bit
5-bit
8-bit
16-bit
Model tree for AtomicChat/Aion-1.0-Instruct-2.3B-GGUF
Base model
microsoft/Aion-1.0-Instruct-2.3B


