How to Run Maxwell-V1.0-Flash Locally

The first GGUF of Maxwell-V1.0-Flash. We wrote its llama.cpp support. Built from the Maxwell team's original BF16 weights.

Atomic Chat Discord GitHub X
  • Built by the team behind Atomic Chat, the app that runs open-weight models locally and offline.
  • Needs llama.cpp with Maxwell support: our port (maxwell-moe-port.patch in this repo) on llama.cpp d3c3e242c. Stock llama.cpp, LM Studio and Ollama do not load these files yet.

Maxwell-V1.0-Flash is a 35B-total / 3B-active Mixture-of-Experts model by the Maxwell team, pre-trained on ~26T multilingual tokens and post-trained for reasoning, search and agentic work, Apache-2.0. Its tokenizer is built for Thai, Lao, Myanmar, Khmer and Shan as much as for CJK and Latin text. This repo has GGUF files of it for llama.cpp.

Pick a file

File names are Maxwell-V1-Flash-<name>.gguf; for the split BF16, point llama.cpp at the first part.

file GB on disk bpw suggested hardware mean KLD 99th pct KLD same top token
BF16 (2 parts, folder BF16/) 72.0 16.02 the original weights, bit-exact 0 0 100%
Q8_0 38.4 8.54 64 GB Macs, 48 GB of GPU memory 0.0371 0.431 93.6%

"Same top token" is how often the file picks the same next token as the BF16 original; "mean KLD" is the KL divergence of its next-token distribution from BF16's. Both are measured with llama-perplexity on our 30-language neutral corpus (114 chunks of 4096 tokens), llama.cpp d3c3e242c with the port, 2x H100 80 GB. Read them with the section below: for this model they include a rounding floor of about 0.03.

For comparison, llama.cpp's plain Q4_K_M preset without an importance matrix (21.9 GB) measures 0.281 mean KLD and 81.6% same top token. We do not publish it.

How precise this model is to run

Maxwell is unusually sensitive to rounding. The same BF16 weights give a different next-token distribution when llama.cpp multiplies them with bf16-rounded inputs than when it runs everything in F32: mean KLD 0.030, the same top token 93.2% of the time (full model, 3 chunks of 4096 tokens). The KV cache type makes no difference, and no single tensor group is the cause: keeping all attention, the head, the shared and dense FFNs in F32 brings it only to 0.023. Our reading is that small differences anywhere flip which of the 256 experts the router picks near its top-12 boundary, layer after layer. For scale: the vendor's transformers code in bf16 and our BF16 file differ by mean KLD 0.044 on the same tokens.

So the numbers in the table, measured against the BF16 file as for every AtomicChat release, include this floor: even Q8_0 reads 0.037. They still rank the files by how far quantization moves them.

Run it

Build llama.cpp at d3c3e242c with the port from this repo:

git clone https://github.com/ggml-org/llama.cpp
cd llama.cpp
git checkout d3c3e242c16b8cbf51400fad410cd69b22d64f35
git apply /path/to/maxwell-moe-port.patch
cmake -B build -DGGML_CUDA=ON          # Mac: drop -DGGML_CUDA=ON (Metal is on by default)
cmake --build build -j --config Release --target llama-server llama-cli

Then serve a file. The chat template (reasoning, tools) is embedded, so pass --jinja; the sampling below is the vendor's:

./build/bin/llama-server -m Maxwell-V1-Flash-AD-Q4_K-IQ4_XS.gguf --jinja -c 32768 -ngl 99 \
    --temp 0.6 --top-p 0.95 --top-k 20

Thinking is on by default. To turn it off for a request, send "chat_template_kwargs": {"enable_thinking": false}, or start the server with --chat-template-kwargs '{"enable_thinking": false}'.

llama.cpp support

We wrote the maxwell-moe architecture for llama.cpp:

  • 45 blocks: three dense SwiGLU blocks, then 42 blocks with 256 routed experts (12 active, sigmoid scores with a selection bias) plus one shared expert.
  • GQA with 32 query and 4 KV heads, head size 128, per-head RMSNorm on Q and K, RoPE.
  • Manifold-constrained hyper-connections: four residual streams mixed by Sinkhorn-normalized matrices, collapsed by a learned head (the arithmetic of DeepSeek-V4's mHC in llama.cpp).
  • Its own pre-tokenizer: separate splits for newlines, Han, and Thai/Lao/Myanmar/Khmer runs before a GPT-4o-style word split. It matches the original tokenizer on all 5,599 texts we compared (30 languages, code, edge cases, a code-point sweep).

What it does not do yet:

  • Block-sparse attention. The original attends, per KV group, to 8 prefix and 8 local blocks of 128 tokens plus 16 blocks picked by a learned indexer. Up to 4,096 tokens that is every block, so llama.cpp's dense attention is exact there; the vendor's own vLLM build also attends densely below 16K tokens. Beyond 16K tokens these files attend to the whole context where the original attends to its selected blocks. The indexer weights are in the files for a later sparse path.
  • MTP. The multi-token-prediction head is left out.

How we checked it against the Maxwell team's modeling_maxwell_moe.py (fp32), with llama.cpp in F32 on the same tokens:

check (2 chunks of 512 tokens, our neutral corpus) mean KLD same top token
the first block of Maxwell-V1.0-Flash + its head 0.000002 100.0%
the first 3 blocks (dense FFN) + head 0.000002 99.6%
the first 6 blocks (3 of them MoE) + head 0.000055 100.0%
for scale: the vendor's code on CPU against itself on GPU, 6 blocks 0.000002 100.0%

The small growth with the MoE blocks comes from exact ties in the router: its selection bias is ~20.2, so in fp32 the score differences below ~2e-6 vanish and 1.8% of tokens tie at the top-12 boundary in the first MoE block. PyTorch and ggml break such ties differently.

test-llama-archs -a maxwell-moe passes on CUDA and CPU, and llama-server --jinja parses reasoning and tool calls from the embedded template.

Model facts

parameters 35B total, ~3B active per token (both without embeddings)
layers 45 (3 dense FFN + 42 MoE)
hidden / expert FFN 2048 / 512 (shared expert 1024, dense FFN 6144)
experts 256 routed, 12 active, 1 shared
attention 32 query heads, 4 KV heads, head dim 128; block-sparse in the original
context 256K per the model card (config: 524,288)
vocabulary 200,704
license Apache-2.0 (LICENSE in this repo, from the base model)

Quantized by AtomicChat.

Downloads last month
-
GGUF
Model size
36B params
Architecture
maxwell-moe
Hardware compatibility
Log In to add your hardware

2-bit

4-bit

6-bit

8-bit

16-bit

Inference Providers NEW
This model isn't deployed by any Inference Provider. ๐Ÿ™‹ Ask for provider support

Model tree for AtomicChat/Maxwell-V1-Flash-GGUF

Quantized
(1)
this model

Collection including AtomicChat/Maxwell-V1-Flash-GGUF