GEHIRNS/magi-1.2-2B

---

Introduction

magi-1.2-2B is Qwen3.5-2B in 1.23 GiB at a 32K context, with the highest capability density we have measured: ρ = 1.305, against 1.149 for magi-1.1-2B and 0.385 for the unquantized BF16 model, all measured under the same settings. It matches magi-1.1-2B's score (67.1% against 66.1%) in 131 MiB less memory and thinks 27% less.

It is three steps on top of Qwen/Qwen3.5-2B. The first is magi-1.1-2B's; the other two are new:

1. magi-1.1-2B's fine-tune (LoRA): the model trained on its own correct, short reasoning traces, so it finishes its thinking.
2. One round of preference training against repetition loops. Under greedy decoding the quantized 2B still gets stuck repeating itself until the thinking budget runs out: magi-1.1-2B hits the 4,096-token budget on 81% of benchmark tasks. Each of 224 training pairs sets a correct, short answer against the model's own looping trace on the same prompt, cut a few repeats after the loop starts. Only the repeated text is penalized, not the reasoning before it. Budget hits fall to 48%.
3. Mixed precision fitted to the 2B. magi-1.1-2B uses the magi-1.0 rules, written for Qwen3.5-4B. Here every tensor group was measured on the 2B: one quantization step up and one down for each of 115 groups, scored by KL divergence from the BF16 model on held-out reasoning and chat text. Bits were then moved to where they buy the most (39 groups changed), and the importance matrix was recomputed from the 2B's own reasoning traces. The file is 66 MiB smaller than magi-1.1-2B's.

Capability density

Capability density is ρ = −log₂(1 − score) / GiB: how many times the benchmark error rate is halved per gibibyte of memory. All models here were measured the same way (llama.cpp, q4_0 KV cache, memory at a 32K context from llama-fit-params). The magi-1.0-4B card's chart used Ollama's memory figures, so its numbers are not comparable with these.

Benchmarks

Every column was run on one RTX 5070 (12 GB) with llama.cpp b11800 (llama-server) under one protocol:

• Tasks: 2,313, run with thinking on and a 4,096-token thinking budget (budget forcing), greedy decoding.
• Server: q4_0 KV cache with flash attention and micro-batch 128 for all three.
• BF16 model: Unsloth's BF16 GGUF of Qwen3.5-2B (unsloth/Qwen3.5-2B-GGUF, Qwen3.5-2B-BF16.gguf), the unquantized starting point of both magi builds.
• Code grading: by executing tests (EvalPlus for HumanEval+ and MBPP+). LiveCodeBench uses 150 problems from contests between 2025-01-18 and 2025-04-06.

Memory and ρ. Memory is the resident size llama-fit-params projects for a 32K context with one sequence: weights, KV cache, recurrent state and compute buffers, GPU plus host, without the CUDA context.

Benchmark	Qwen3.5-2B BF16 4.72 GiB	magi-1.1-2B 1.36 GiB	magi-1.2-2B 1.23 GiB
Efficiency
Memory at 32K (GiB)	4.72	1.36	1.23
Benchmark score	71.66%	66.06%	67.09%
Capability density ρ	0.385	1.149	1.305
Decode speed (tok/s)	not measured	325	333
Mean thinking tokens	3,018	3,409	2,499
Hit the 4K thinking budget	62.4%	80.7%	48.2%
Traces ending in a loop	49.2%	77.4%	38.8%
Coding
HumanEval+	56.10	46.95	50.61
MBPP+	55.64	44.61	44.86
LiveCodeBench	11.33	1.33	8.00
Math
GSM8K	82.67	78.00	76.67
Science and Knowledge
ARC-Challenge	94.67	93.33	96.00
MMLU	74.50	64.00	65.50
OpenBookQA	86.50	77.00	82.50
SciQ	95.00	94.00	92.00
TruthfulQA	75.50	77.50	76.00
Commonsense
HellaSwag	60.00	50.50	50.50
WinoGrande	68.00	65.50	62.50
Tool Use
Function calling	100.00	100.00	100.00

magi-1.2-2B against magi-1.1-2B: on the same 2,313 tasks, +1.03 points (paired, standard error 0.81; 207 tasks gained, 187 lost), so the scores are within noise of each other. What changes is the cost: 131 MiB less memory, 27% fewer thinking tokens, and budget hits from 81% to 48% of tasks.

magi-1.2-2B against the BF16 model: 4.56 points lower (paired, standard error 0.83; 134 tasks gained, 259 lost) in 3.8× less memory. Largest gaps, in points: MBPP+ −10.8, HellaSwag −9.5, MMLU −9.0.

Decode speed: single-stream generation speed (llama-bench, 128 tokens, q4_0 KV), both magi builds measured back to back on the same GPU.

Recipe

Per-tensor types, read from the file (Qwen3.5-2B: 24 layers, 18 Gated DeltaNet and 6 full attention):

Tensor group	Types (tensors)	Share of parameters	bits/weight
FFN gate, up, down (all 24 layers)	IQ3_XXS ×61, IQ4_XS ×10, IQ2_S ×1	48.1%	3.22
Token embedding = tied output head	Q3_K	27.0%	3.44
DeltaNet input projection attn_qkv (18)	IQ3_XXS ×9, Q4_K ×9	12.0%	3.78
Full-attention q, k, v, output (6 layers)	Q4_K ×16, Q5_K ×6, IQ3_XXS ×2	4.7%	4.37
Gated-attention output gate (18 DeltaNet layers)	IQ3_XXS ×9, IQ2_S ×6, IQ4_XS ×3	4.0%	3.09
DeltaNet output projection ssm_out (18)	Q4_K ×12, Q5_K ×5, IQ3_XXS ×1	4.0%	4.70
DeltaNet gates ssm_alpha, ssm_beta	Q8_0 ×36	<0.1%	8.50
Norms, convolution, A, Δt bias	F32 ×133	<0.1%	32.00

The quantizer is llama-quantize --pure from the PrismML-Eng/llama.cpp fork, with base type IQ2_XXS and a per-tensor override file. The starting point was the magi-1.0 rules; the 39 measured moves changed one group each by one step. The importance matrix was computed on the stage-1 model (BF16) from 445 correct, finished reasoning traces that Qwen3.5-2B wrote for training-split prompts (no benchmark questions, and none of the traces used in training). File SHA-256: 12501b4d40affd417dd4b36660ce0a21ee386ba4a918f9864e5baab09c570551 (788.0 MiB).

Training

• Stage 1, magi-1.1-2B's adapter: LoRA rank 16, α 32, on all MLP projections and the q/k/v/o projections of the 6 full-attention layers (96 modules), trained for 2 epochs at learning rate 1e-4 on 600 of the model's own correct, short reasoning traces. See the magi-1.1-2B card.
• Stage 2, preference training (DPO with an NLL term): the same adapter, trained further.
  • Loss: DPO, β 0.1, plus the chosen answer's per-token negative log-likelihood (weight 1.0). The reference model is the stage-1 model.
  • Optimisation: learning rate 2e-5 (cosine), 2 epochs, 8 pairs per step, bf16.
  • Data: 224 pairs, plus 10 held out. On the held-out pairs, every chosen answer ended up preferred, and the chosen answers became more likely, not less.
  • Chosen side: a correct, finished answer, the shortest of 4 samples at temperature 0.6 from an 8-bit (Q8_0) build of the stage-1 model. Median 706 thinking tokens.
  • Rejected side: the quantized model's own greedy trace on the same prompt. It is cut after the third occurrence of its final repeating unit and scored only from the first repeat on (median 141 scored tokens).
  • Benchmark exclusion: benchmark questions were excluded by id, normalized text and hash.
• Merge: the adapter is merged into the BF16 weights before quantization.
• Prompt sources: the training/validation splits below. The licenses are as declared on each dataset's Hugging Face card.

Dataset	Split	Declared license
openai/gsm8k	train	MIT
cais/mmlu	validation, dev	MIT
google-research-datasets/mbpp	train, validation, prompt	CC BY 4.0
allenai/ai2_arc	train	CC BY-SA 4.0
allenai/sciq	train	CC BY-NC 3.0
allenai/openbookqa	train	unknown
Rowan/hellaswag	train	not declared
allenai/winogrande	train	not declared

The training rows hold the model's own outputs, not the datasets' answer texts. Some sources carry non-commercial or share-alike terms. If those terms matter for your use, check them.

How to run

llama.cpp (the evaluated configuration; uses the Qwen3.5 chat template embedded in the file):

llama-server -m magi-1.2-2B.gguf -c 32768 --jinja --flash-attn on --cache-type-k q4_0 --cache-type-v q4_0 -ub 128

Ollama:

ollama create magi-1.2-2B -f Modelfile
ollama run magi-1.2-2B

The benchmark used greedy decoding. For interactive use with thinking enabled, Qwen recommends temperature 0.6, top_p 0.95 and top_k 20, which also makes repetition loops much rarer.

Limitations

• Below the BF16 model by 4.6 points overall (largest gaps: MBPP+, HellaSwag, MMLU). Competitive programming is mostly out of reach at 2B: LiveCodeBench 8.0%, against 11.3% for BF16.
• Still loops under greedy decoding: 48% of benchmark tasks hit the thinking budget, so use sampling for interactive work.
• Selected on this benchmark. Both new steps were kept by decisions made on this benchmark (a 250-task screen, then the full set). Read the score difference to magi-1.1-2B as a tie. The memory and thinking-length savings are large compared with that noise.
• Narrow evaluation: one runtime (llama.cpp b11800), one benchmark, one GPU, each configuration run once.
• The name is ours. "2B" follows Qwen's naming; the file holds 1,881,825,088 parameters. Text only: the base model's vision encoder is not included.

Related

• GEHIRNS/magi-1.1-2B: stage 1 alone with the magi-1.0 rules (the middle column above).
• GEHIRNS/magi-1.0-2B: the magi-1.0 recipe on Qwen3.5-2B, no training.
• GEHIRNS/magi-1.1-4B and GEHIRNS/magi-1.0-4B: the same line on Qwen3.5-4B.
• Paper on the recipe: Eason Sun, Where the Bits Go, Zenodo, 2026. DOI: 10.5281/zenodo.23630199.

License

The weights are Apache 2.0, inherited from Qwen/Qwen3.5-2B; the license text is included in this repository. See Training for the terms of the datasets the prompts came from.