nanbeige4.2-3b-mxfp8-mlx

MLX MX FP8 (micro-scaled 8-bit float) quantization of Nanbeige/Nanbeige4.2-3B, converted with mlx-lm.

Nanbeige4.2-3B is a compact agentic model built on Nanbeige4.2-3B-Base. Its Looped Transformer architecture reuses the same transformer layers across multiple passes to increase effective capacity without adding parameters. With only 3B non-embedding parameters, it outperforms larger models (Qwen3.5-9B, Gemma4-12B) on general-agent, code-agent, and reasoning benchmarks (per the base model's card: SWE-Bench Verified 63.6, GPQA-Diamond 87.4, HMMT-Feb-2026 82.8).

Architecture

Base model	Nanbeige4.2-3B (Looped Transformer, nanbeige arch, 22 unique layers × 2 loops, ~4B total / 3B non-embedding params)
Quantization	MX FP8 (micro-scaled 8-bit float)
Bits per weight	8.250
Disk size	4.0 GB (from ~7.8 GB bf16)
Format	MLX (safetensors)

New architecture support — requires a PR branch of mlx-lm for now

Nanbeige4.2-3B uses a Looped Transformer architecture (model_type: "nanbeige"): the same 22 decoder layers run twice (num_loops: 2), with a full RMSNorm applied between and after each pass — a way to add effective depth without adding parameters (3B non-embedding params, ~4B total). This architecture had no mlx-lm support at all prior to this release. A minimal, from-scratch MLX port (mlx_lm/models/nanbeige.py) was written for this conversion, implementing the loop mechanism with per-loop KV cache isolation (each loop's attention only attends to its own loop's keys/values) and covering the subset of the (very large) upstream NanbeigeConfig that this checkpoint actually uses — the full reference implementation also defines n-gram embeddings, hyper-connections, and depth-attention, none of which are active here (num_loops=2 is the only non-default architectural flag Nanbeige4.2-3B sets).

This is not yet in an official mlx-lm release. A PR adding this support is open against ml-explore/mlx-lm. Until it merges, install mlx-lm from the PR branch to load this model:

pip install git+https://github.com/SahilChachra/mlx-lm.git@add-nanbeige-support

Usage

from mlx_lm import load, generate

model, tokenizer = load("sahilchachra/nanbeige4.2-3b-mxfp8-mlx")

messages = [{"role": "user", "content": "Which number is bigger, 9.11 or 9.8?"}]
prompt = tokenizer.apply_chat_template(messages, add_generation_prompt=True, tokenize=False)
out = generate(model, tokenizer, prompt=prompt, max_tokens=2048, verbose=False)
print(out)

The chat template supports enable_thinking (default on — set False for non-thinking mode) and preserve_thinking (whether prior turns' reasoning is kept in multi-turn contexts; the base model card recommends False for general chat/QA and True for multi-turn tool-use/agentic workflows). Per the base model's recommendations: temperature 1.0 for agentic/tool-use tasks, 0.6 for reasoning/chat tasks; eos_token_id is 166101.

CLI:

mlx_lm.generate --model sahilchachra/nanbeige4.2-3b-mxfp8-mlx \
  --prompt "Which number is bigger, 9.11 or 9.8?" \
  --max-tokens 2048

Smoke test

Verified via mlx_lm.generate through the model's chat template, on an Apple M4 Pro:

Prompt: "Explain the difference between TCP and UDP in two sentences."
Output (mxfp4): "...We need to provide an explanation that is concise, exactly two
sentences... TCP (Transmission Control Protocol) is a connection-oriented, reliable,
error-correcting protocol... UDP (User Datagram Protocol) is a connectionless,
unreliable, but lightweight protocol..."

Prompt: "Write a haiku about the ocean at night."
Output (mxfp8): "...haiku structure: three lines with syllable counts: 5, 7, 5...
Elements to include: night setting, ocean imagery (waves, stars reflected,
moonlight, darkness)..."

All four quantized variants preserve the base model's thinking-native behavior (private reasoning before the final answer) intact through quantization.

Decode speed

Measured with mlx_lm.generate on 5 GPQA-Diamond prompts (the same benchmark the base model reports its 87.4 score on), Apple M4 Pro, max_tokens=400:

Variant	Bits/weight	Disk size	Decode speed	vs FP16
fp16	16	7.8 GB	16.6 tok/s	baseline
mxfp4	4.250	2.1 GB	50.0 tok/s	3.0x
int4	4.500	2.2 GB	49.2 tok/s	3.0x
mxfp8	8.250	4.0 GB	30.2 tok/s	1.8x
int8	8.500	4.2 GB	29.7 tok/s	1.8x

4-bit and 8-bit each cluster tightly regardless of MX-float vs affine-int encoding — pick based on quality tolerance (smaller/faster 4-bit vs higher-fidelity 8-bit) rather than the encoding scheme.

A note on accuracy: we attempted a full 25-sample GPQA-Diamond accuracy run across all variants, but this is a thinking-native model that reasons at length on hard graduate-level questions — most responses didn't reach a final answer within a 2048-token budget (across all variants), making the resulting accuracy numbers a measurement of response-length variance rather than model quality. We don't have a reliable accuracy comparison to publish here as a result; treat the decode-speed numbers above as the trustworthy signal from this exercise.

A note on correctness validation: this quantization required first porting Nanbeige's Looped Transformer architecture to mlx-lm (see "New architecture support" below). That port was validated against the official modeling_nanbeige.py reference implementation (run via transformers, float32) on held-out prompts: computed logits from both implementations for the same inputs and compared next-token predictions and full logit distributions. After aligning RoPE frequency initialization between the two implementations, results matched closely — logit cosine similarity ≥0.9999 and identical next-token predictions on every test prompt.

Other MLX variants

• nanbeige4.2-3b-mxfp4-mlx
• nanbeige4.2-3b-int4-mlx
• nanbeige4.2-3b-mxfp8-mlx
• nanbeige4.2-3b-int8-mlx

optiq-5bpw was not produced for this release — it requires the mlx-optiq package, which wasn't available in the conversion environment.

Credits

• Base model: Nanbeige/Nanbeige4.2-3B
• Quantization tooling: mlx-lm