How to use from
llama.cpp
Install (macOS, Linux)
curl -LsSf https://llama.app/install.sh | sh
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf AtomicChat/Maxwell-V1-Flash-GGUF:
# Run inference directly in the terminal:
llama cli -hf AtomicChat/Maxwell-V1-Flash-GGUF:
Install from WinGet (Windows)
winget install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf AtomicChat/Maxwell-V1-Flash-GGUF:
# Run inference directly in the terminal:
llama cli -hf AtomicChat/Maxwell-V1-Flash-GGUF:
Build from source code
git clone https://github.com/ggml-org/llama.cpp.git
cd llama.cpp
cmake -B build
cmake --build build -j --target llama-app
# Start a local OpenAI-compatible server with a web UI:
./build/bin/llama serve -hf AtomicChat/Maxwell-V1-Flash-GGUF:
# Run inference directly in the terminal:
./build/bin/llama cli -hf AtomicChat/Maxwell-V1-Flash-GGUF:
Use Docker
docker model run hf.co/AtomicChat/Maxwell-V1-Flash-GGUF:
Quick Links

How to Run Maxwell-V1.0-Flash Locally

The first GGUF of Maxwell-V1.0-Flash. We wrote its llama.cpp support. Built from the Maxwell team's original BF16 weights.

Atomic Chat Discord GitHub X
  • Built by the team behind Atomic Chat, the app that runs open-weight models locally and offline.
  • Needs llama.cpp with Maxwell support: our port (maxwell-moe-port.patch in this repo) on llama.cpp d3c3e242c. Stock llama.cpp, LM Studio and Ollama do not load these files yet.

Maxwell-V1.0-Flash is a 35B-total / 3B-active Mixture-of-Experts model by the Maxwell team, pre-trained on ~26T multilingual tokens and post-trained for reasoning, search and agentic work, Apache-2.0. Its tokenizer is built for Thai, Lao, Myanmar, Khmer and Shan as much as for CJK and Latin text. This repo has GGUF files of it for llama.cpp.

Pick a file

AD (Atomic Dynamic) files keep everything but the routed experts at Q8_0 (the mHC mixing matrices stay F32) and quantize the experts, which hold ~94% of the weights, with an importance matrix computed on the BF16 weights over our public 4.8M-token calibration build, rendered with Maxwell's own chat template. The name says the expert types: AD-Q5_K-Q4_K has Q5_K down projections and Q4_K gate/up projections. File names are Maxwell-V1-Flash-<name>.gguf; for the split BF16, point llama.cpp at the first part.

file GB on disk bpw suggested hardware mean KLD 99th pct KLD same top token
BF16 (2 parts, folder BF16/) 72.0 16.02 the original weights, bit-exact 0 0 100%
Q8_0 38.4 8.54 64 GB Macs, 48 GB of GPU memory 0.0371 0.431 93.6%
AD-Q6_K 30.2 6.72 48 GB Macs, 2x 24 GB GPUs 0.0482 0.563 92.6%
AD-Q6_K-Q5_K 27.2 6.05 32 GB GPUs (RTX 5090) 0.0697 0.826 90.9%
AD-Q5_K-Q4_K 22.9 5.09 32 GB with a long context, 32 GB Macs 0.1383 1.649 87.3%
AD-Q4_K-IQ4_XS 20.8 4.62 24 GB GPUs (RTX 3090 / 4090) 0.1736 2.045 85.7%

"Same top token" is how often the file picks the same next token as the BF16 original; "mean KLD" is the KL divergence of its next-token distribution from BF16's. Both are measured with llama-perplexity on our 30-language neutral corpus (114 chunks of 4096 tokens), llama.cpp d3c3e242c with the port, 2x H100 80 GB. Read them with the section below: for this model they include a rounding floor of about 0.03. The suggested hardware leaves room for a moderate context.

This model loses more to quantization than most: at 4.6 bits per weight the file picks the original's token 85.7% of the time, where models of this class usually stay above 95%. The plain llama.cpp Q4_K_M preset without an importance matrix (21.9 GB) measures 0.281 mean KLD and 81.6% same top token; AD-Q4_K-IQ4_XS is 1.1 GB smaller with 38% lower mean KLD. Below 20 GB our layouts fell under 80% same top token (18.1 GB: 79.4%, 15.9 GB: 74.8%, 14.0 GB: 68.5%), so we do not publish them. All numbers, logs and the importance matrix are in AtomicChat/Maxwell-V1-Flash-GGUF-metrics.

Speed, one run of llama-bench with flash attention: AD-Q4_K-IQ4_XS on 1x H100 80 GB processes a 512-token prompt at 6,398 tokens/s and generates 173 tokens/s with an empty context; 5,374 and 150 at 8K tokens of context; 3,804 and 132 at 32K. Checked by hand in llama-server --jinja with the same file: a reasoning answer, thinking off, a tool call and an answer in Thai.

How precise this model is to run

Maxwell is unusually sensitive to rounding. The same BF16 weights give a different next-token distribution when llama.cpp multiplies them with bf16-rounded inputs than when it runs everything in F32: mean KLD 0.030, the same top token 93.2% of the time (full model, 3 chunks of 4096 tokens). The KV cache type makes no difference, and no single tensor group is the cause: keeping all attention, the head, the shared and dense FFNs in F32 brings it only to 0.023. Our reading is that small differences anywhere flip which of the 256 experts the router picks near its top-12 boundary, layer after layer. For scale: the vendor's transformers code in bf16 and our BF16 file differ by mean KLD 0.044 on the same tokens.

So the numbers in the table, measured against the BF16 file as for every AtomicChat release, include this floor: even Q8_0 reads 0.037. They still rank the files by how far quantization moves them.

Run it

Build llama.cpp at d3c3e242c with the port from this repo:

git clone https://github.com/ggml-org/llama.cpp
cd llama.cpp
git checkout d3c3e242c16b8cbf51400fad410cd69b22d64f35
git apply /path/to/maxwell-moe-port.patch
cmake -B build -DGGML_CUDA=ON          # Mac: drop -DGGML_CUDA=ON (Metal is on by default)
cmake --build build -j --config Release --target llama-server llama-cli

Then serve a file. The chat template (reasoning, tools) is embedded, so pass --jinja; the sampling below is the vendor's:

./build/bin/llama-server -m Maxwell-V1-Flash-AD-Q4_K-IQ4_XS.gguf --jinja -c 32768 -ngl 99 \
    --temp 0.6 --top-p 0.95 --top-k 20

Thinking is on by default. To turn it off for a request, send "chat_template_kwargs": {"enable_thinking": false}, or start the server with --chat-template-kwargs '{"enable_thinking": false}'.

llama.cpp support

We wrote the maxwell-moe architecture for llama.cpp:

  • 45 blocks: three dense SwiGLU blocks, then 42 blocks with 256 routed experts (12 active, sigmoid scores with a selection bias) plus one shared expert.
  • GQA with 32 query and 4 KV heads, head size 128, per-head RMSNorm on Q and K, RoPE.
  • Manifold-constrained hyper-connections: four residual streams mixed by Sinkhorn-normalized matrices, collapsed by a learned head (the arithmetic of DeepSeek-V4's mHC in llama.cpp).
  • Its own pre-tokenizer: separate splits for newlines, Han, and Thai/Lao/Myanmar/Khmer runs before a GPT-4o-style word split. It matches the original tokenizer on all 5,599 texts we compared (30 languages, code, edge cases, a code-point sweep).

What it does not do yet:

  • Block-sparse attention. The original attends, per KV group, to 8 prefix and 8 local blocks of 128 tokens plus 16 blocks picked by a learned indexer. Up to 4,096 tokens that is every block, so llama.cpp's dense attention is exact there; the vendor's own vLLM build also attends densely below 16K tokens. Beyond 16K tokens these files attend to the whole context where the original attends to its selected blocks. The indexer weights are in the files for a later sparse path.
  • MTP. The multi-token-prediction head is left out.

How we checked it against the Maxwell team's modeling_maxwell_moe.py (fp32), with llama.cpp in F32 on the same tokens:

check (2 chunks of 512 tokens, our neutral corpus) mean KLD same top token
the first block of Maxwell-V1.0-Flash + its head 0.000002 100.0%
the first 3 blocks (dense FFN) + head 0.000002 99.6%
the first 6 blocks (3 of them MoE) + head 0.000055 100.0%
for scale: the vendor's code on CPU against itself on GPU, 6 blocks 0.000002 100.0%

The small growth with the MoE blocks comes from exact ties in the router: its selection bias is ~20.2, so in fp32 the score differences below ~2e-6 vanish and 1.8% of tokens tie at the top-12 boundary in the first MoE block. PyTorch and ggml break such ties differently.

test-llama-archs -a maxwell-moe passes on CUDA and CPU, and llama-server --jinja parses reasoning and tool calls from the embedded template.

Model facts

parameters 35B total, ~3B active per token (both without embeddings)
layers 45 (3 dense FFN + 42 MoE)
hidden / expert FFN 2048 / 512 (shared expert 1024, dense FFN 6144)
experts 256 routed, 12 active, 1 shared
attention 32 query heads, 4 KV heads, head dim 128; block-sparse in the original
context 256K per the model card (config: 524,288)
vocabulary 200,704
license Apache-2.0 (LICENSE in this repo, from the base model)

Quantized by AtomicChat.

Downloads last month
10,974
GGUF
Model size
36B params
Architecture
maxwell-moe
Hardware compatibility
Log In to add your hardware

4-bit

6-bit

8-bit

16-bit

Inference Providers NEW
This model isn't deployed by any Inference Provider. ๐Ÿ™‹ Ask for provider support

Model tree for AtomicChat/Maxwell-V1-Flash-GGUF

Quantized
(5)
this model

Space using AtomicChat/Maxwell-V1-Flash-GGUF 1

Collection including AtomicChat/Maxwell-V1-Flash-GGUF