Issingggggg/MiniCPM5-2B-SFT-MLX-4bit

MiniCPM5-2B-SFT — MLX 4-bit

A 4-bit MLX quantization of openbmb/MiniCPM5-2B-SFT for Apple Silicon. 1.3 GB on disk, runs fully offline on a Mac.

This is a converted copy, not a new model: no fine-tuning, no weight changes beyond quantization. All credit for the base capability belongs to the MiniCPM team.

Quantization details

Method	mlx_lm.convert (affine, group size 64)
Bits	4 (~4.501 bits per weight)
Size	1.3 GB (vs 4.7 GB bf16)
Architecture	LlamaForCausalLM, 42 layers, hidden 2048, 16 heads, vocab 130560
Context	131,072 tokens

Reproduce it yourself:

mlx_lm.convert --hf-path openbmb/MiniCPM5-2B-SFT \
               --mlx-path ./MiniCPM5-2B-SFT-MLX-4bit \
               -q --q-bits 4 --q-group-size 64 --dtype bfloat16

Usage

pip install mlx-lm

from mlx_lm import load, generate

model, tokenizer = load("Issingggggg/MiniCPM5-2B-SFT-MLX-4bit")
prompt = tokenizer.apply_chat_template(
    [{"role": "user", "content": "Write a Python function that checks if a list is monotonic."}],
    tokenize=False, add_generation_prompt=True,
)
print(generate(model, tokenizer, prompt=prompt, max_tokens=400, verbose=False))

Note: passing a raw string to mlx_lm.generate does not apply the chat template — the model then degenerates into raw continuation and tends to parrot the prompt. Always template instruct-style requests.

Measured results

Evaluated with the harness in flux-spark on HumanEval's 164 tasks, greedy decoding, max_tokens=400:

Setting	Pass rate
Single shot	72.6%
best-of-3 sampling + test filtering	84.8%
+ reflection-based repair	87.8%
Single shot, HumanEval+ (~80× tests)	68.9%
Full loop, HumanEval+	79.9%

Numbers are from this quantized artifact specifically, measured on one machine. They are reproductions of the base model's capability under a pipeline, not a claim that this quantization is better than the original.

Limitations

• 2B parameters. Single-function generation is its comfort zone; multi-file work, long contexts and project-level refactors are out of reach.
• Outputs are often preceded by "thinking" text; strip it or instruct explicitly.
• 4-bit quantization costs accuracy: single-shot is ~11.5 pp below bf16 (p<0.003), though a verification loop recovers most of it (−2.4 pp, not significant with 164 tasks).
• No safety tuning was added by this conversion; inherit the base model's behavior.

License and attribution

Base model openbmb/MiniCPM5-2B-SFT is released under Apache-2.0; this derivative keeps the same license. If you use it, please cite the original MiniCPM work as well.