avlp12/Solar-Open2-250B-Alis-MLX-Dynamic

Solar-Open2-250B — Alis MLX Dynamic (DWQ) Builds

Built with Solar — this is a derivative of upstage/Solar-Open2-250B, distributed under the Upstage Solar License. A copy of the license ships in every branch.

MLX (Apple Silicon) port of Solar Open 2 (250B-A15B Hybrid-Attention MoE) in three quantization tiers, produced with alis-dwq — data-aware quantization with teacher-guided distillation (DWQ). Both distilled builds are fine-tuned against the Q8 reference under a per-slice acceptance gate, with an EN sentinel gate as the final safety net.

✅ Status: all three tiers final. Q v3 (4-bit/g32 + DWQ, lr 5e-7 × 3 rounds): EN loop cleared with teacher-level diversity (distinct4 0.399 vs teacher 0.411), code KLD better than its own base. F v2 (2.34 bpw, 73.3 GB): KLD 0.42 (−76% vs the legacy 2-bit path), all loops clean, EN sentinel ACCEPT. A fourth tier (M, 4-bit/g64) was trained twice (Q-teacher and Q8-teacher) but is not shipped — it lost every metric to Q v3 at the same memory tier.

Builds (branches)

One repo, three weights — pick a branch by machine memory:

branch	build	recipe	measured bpw	size	fits
main	Q v3 — daily driver, best quality	4-bit / g32 + DWQ (Q8 teacher, per-slice gate, lr 5e-7 × 3 rounds)	5.001	156.5 GB (145.7 GiB, 31 shards)	192 GiB stock (tight) / 256 GiB comfortable
t96	F v2 — smallest usable	2-bit / g128 experts only + clip + DWQ rank 8 (Q8 teacher); chokepoints 8b, embed+head 6b, attn/shared 4b via quant_predicate_builder	2.344	73.3 GB (68.3 GiB, 15 shards) + 4.0 GB LoRA adapters	96 GiB stock (tight) / 128 GiB comfortable
t384	T — Q8 reference / teacher	8-bit / g64	8.500	265.9 GB (247.7 GiB, 62 shards)	384 GiB stock / 512 GiB for full 1M ctx

Sizes are decimal GB (10⁹ bytes, as shown on the HF page) with binary GiB (2³⁰ bytes) in parentheses. Measured bpw is computed on the weight shards over 250.29B params; the t96 adapters (4.0 GB) ship separately and are not fused into the shards.

Every build keeps all MoE router gates at 8-bit / g64 (48/48 layers) — router quantization was found to be the dominant quality risk, so only expert/attention weights are quantized per the tier recipe. Why no mid tier? An M build (4-bit/g64, 131–141 GB) was distilled twice (Q-teacher and Q8-teacher arms). The Q8 arm fixed its EN regression but still lost every metric to Q v3 at the same memory tier (≥160 GB machines), so it is retained locally as a control, not shipped.

Install & run

⚠️ Requires the solar_open2 architecture, which is not yet in PyPI mlx-lm (0.31.3). Stock pip install mlx-lm will fail with ValueError: Model type solar_open2 not supported. Two options:

# option A — the fork that adds solar_open2 (recommended; the fork's default branch is `solar-open2`)
pip install "git+https://github.com/avlp12/mlx-lm.git"

Option B — keep stock mlx-lm and drop the bundled solar_open2.py (shipped in every branch) into .../site-packages/mlx_lm/models/. Same file, same result.

# main = Q v3 (recommended)
mlx_lm.generate --model avlp12/Solar-Open2-250B-Alis-MLX-Dynamic \
  --prompt "안녕하세요, 간단히 자기소개를 해주세요." --max-tokens 512

Branch selection (t96 = F v2, t384 = T Q8) uses the Python API — the CLI has no --revision flag:

from mlx_lm import load, generate

model, tok = load("avlp12/Solar-Open2-250B-Alis-MLX-Dynamic", revision="t96")   # F v2 (77 GB)
# model, tok = load("avlp12/Solar-Open2-250B-Alis-MLX-Dynamic", revision="t384") # T Q8 (266 GB)
print(generate(model, tok, prompt="Explain MoE routing in three sentences.", max_tokens=512, verbose=False))

CLI-only alternative: hf download avlp12/Solar-Open2-250B-Alis-MLX-Dynamic --revision t96, then point --model at the downloaded snapshot directory.

F v2 LoRA adapters (t96 branch only): the DWQ compensator adapters ship un-fused in alis_adapters/ (rank 8). Download that folder and pass --adapter-path <local alis_adapters dir> to use them. All numbers in this card are measured without the adapters (same convention as the control arm), so a fused deployment may do better.

Chat, reasoning & sampling. The chat template (chat_template.jinja) ships in every branch; mlx_lm applies it automatically in chat/server mode (raw mlx_lm.generate with a plain string prompt does not — apply the template yourself). Client-side settings follow the base model card: temperature 1.0 / top_p 1.0 for both reasoning modes — avoid ~0.7-class temperatures, which measurably raise loop/runaway-thinking risk on quantized builds. reasoning_effort defaults to "high" (thinking ON); use "none" for a direct answer (--chat-template-args '{"reasoning_effort":"none"}' on mlx_lm.server, or the apply_chat_template kwarg). Reasoning markers are dedicated tokens <|think:start|> / <|think:end|> (ids 130/131) — there is no </think> or "Final Answer" convention. Split reasoning from the final answer on <|think:end|>, and do not add it as a stop token: the answer comes after it, and termination is <|im:end|> (id 129, already in generation_config.json). Budget max_tokens for thinking + answer together — thinking typically runs 0.5–2K tokens on chat prompts, so ≥4K total is recommended; an empty answer or a reply cut off mid-thought means the budget ran out inside the reasoning block. Server + LoRA adapters (F v2): fork revisions ≤ 9d67a0c silently dropped --adapter-path in mlx_lm.server — the adapter lookup ran after the model-name remap, always missed, and the server then served the bare 2-bit build. Fixed in 7937fda (branch solar-open2); update the fork, or pass "adapters": "<local alis_adapters dir>" in the request body as a workaround.

Hardware compatibility (memory planning)

Weights are loaded fully into GPU-wired unified memory; context costs extra on top. KV cache ≈ 48 KiB/token fp16 — only 12 of 48 layers (the softmax GQA layers, every 4th) hold a KV cache; the 36 KDA linear-attention layers carry a fixed-size recurrent state (~150 MiB total) that does not grow with context. → ~4.8 GiB per 100K tokens · 256K ≈ 12 GiB · 1M ≈ 48 GiB.

Default GPU wired limit on macOS ≈ 80% of RAM; raise it with sudo sysctl iogpu.wired_limit_mb=<MB> (revert by lowering again; needs admin). Build footprints below use exact GiB: F v2 68.3 GiB (+3.9 GiB adapters) · Q v3 145.7 GiB · T 247.7 GiB.

machine RAM	default GPU limit (~80%)	what fits stock	with wired-limit bump
96 GiB	~77 GiB	F v2 — 한계 직전 (여유 ~5 GiB ≈ 100K tok ctx), bump 권장	88 GiB: F v2 + ~16 GiB ctx (330K tok)
128 GiB	~103 GiB	F v2 + 31 GiB ctx (640K tok)	—
192 GiB	~154 GiB	Q v3 — 한계 직전 (여유 ~8 GiB ≈ 170K tok ctx), bump 권장	176 GiB: Q v3 + ~30 GiB ctx (630K tok)
256 GiB	~206 GiB	Q v3 + ~60 GiB ctx (full 1M tok)	—
384 GiB	~309 GiB	T (Q8) + ~61 GiB ctx (full 1M tok)	—
512 GiB	~412 GiB	T (Q8) + full 1M ctx	—

Rule of thumb: pick the largest build whose weights fit your machine with the context you need — quality scales with bits (see the measured table below). DWQ training peak observed: 275 GB wired+compressed during Q v3 distillation (training-only; inference needs are far lower).

Note on the HF sidebar panel: the native Hardware compatibility widget indexes only the repo's root model (main = Q v3, shown as "4-bit · MLX · 156 GB") — unlike GGUF repos, where every quant file gets its own row, an MLX repo gets a single row and branch builds (t96, t384) are not indexed. Use the table above for F/T hardware sizing.

Measured quality (vs Q8 reference, §2-10 eval, ref = T)

Metric	T (ref)	Q v3 (main)	F v2 (t96)
KLD — overall	0 (ref)	0.06369 (flip 0.0928)	0.41976 (flip 0.2122)
KLD — English	0 (ref)	0.04858 (flip 0.0713, clean)	0.30345 (flip 0.1514, clean)
KLD — code	0 (ref)	0.02811 (flip 0.0439)	0.20571 (flip 0.1426, clean)
KLD — Korean	0 (ref)	0.11438 (flip 0.1631)	0.75013 (flip 0.3535, clean)
loop probe (256)	—	clean (all slices, distinct4 EN 0.399) ✅	clean (all slices, 0.356) ✅
kv probe (4)	—	selfKL=0.02305, flip=0.0557	selfKL=0.02195, flip=0.0495
DWQ valid loss (init → final)	—	0.01597 → 0.01381 (−13.5%, 3/3 accepted)	0.12102 → 0.07242 (−40.2%)
EN sentinel gate (eps=0, cap 1%)	—	mean −8.8% (paired 3/16)	ACCEPT (−15.9%, no breach) ✅

Q tier journey (disclosed): the 4-bit/g32 base shipped a greedy EN loop (len23). Attempt 1 (Q-dwq, legacy scalar gate) made it worse (len27) — the scalar gate's blind spot that motivated the per-slice gate. Attempt 2 (Q v2, gate + Q8 teacher, lr 1e-6 × 6 rounds) cleared the loop but collapsed EN diversity (distinct4 0.221) and regressed KLD +6.7% — over-distillation. Attempt 3 (Q v3, same gate/teacher but lr 5e-7 × 3 rounds) keeps the loop fix, restores EN diversity to 0.399 (97% of the Q8 teacher's 0.411), brings KLD EN back to base level (+0.7%), and improves code KLD −5.1% below its own base. The residual trade-off is Korean KLD +8.4% vs base (0.10553 → 0.11438) — see Known Limitations.

F tier: the legacy path (uniform 2-bit/g128 base + clip + DWQ) failed the usability gate (KLD 1.23, all slices LOOPED) — a uniform 2-bit predicate destroys the KDA chokepoints and embed/head beyond DWQ recovery. F v2 re-quantizes with quant_predicate_builder (chokepoint 8b / embed+head 6b / attn+shared 4b / experts 2b, 578/578 per-layer entries verified) and distills from the Q8 teacher under a per-slice acceptance gate with rank-8 LoRA compensators. Result: KLD 1.73 (legacy base) → 1.23 (legacy DWQ) → 0.42 (F v2 + DWQ), all loop probes clean, EN sentinel ACCEPT. The LoRA adapters ship separately (alis_adapters/); the numbers above are measured without them (same convention as the legacy arm), so a fused deployment may do better.

DWQ training detail: 200 calibration samples (180 train / 20 valid), expert-routing coverage 99.8%; teacher = Q8 reference for all shipped builds; deepest-first layer order, 3 rounds (Q v3) / 6 rounds (F v2); per-slice acceptance gate (EN ceiling ε=1%) + EN sentinel gate (paired, cap 1%).

Quantization quality (pre-distillation measurements)

Effective bit utilization (code-entropy audit, §2-2):

Metric	M (4-bit/g64)	F (2-bit/g128 clipped)
Effective bits/weight	3.72 (93% of nominal)	1.65 (82% of nominal)
Low-entropy groups	0.0%	0.5% (concentrated in L0, worst 31.7%)
Weak (<50% util) experts	0 / 144 stacks	0 / 144 stacks

F clip requantization gate (T-dequant clipping before 2-bit requant):

Held-out PPL	Before clip	After clip	Δ
wikitext	110.56	49.51	−55.2%
code	21.81	11.63	−46.7%
Korean	23.94	18.93	−20.9%

Base model (bf16, official card numbers)

Solar Open 2 is Upstage's 250B-A15B open-weight model: Hybrid-Attention MoE — [Softmax, Linear×3] × 12 (only 12 of 48 layers hold a KV cache), NoPE on linear-attention layers, 321 experts (320 routed top-8 + 1 shared), vocab 196,608, 1M-token context, ~12T pre-training tokens, English/Korean/Japanese.

Benchmark	Solar Open 2 (bf16)
MMLU-Pro	86.2
GPQA-Diamond	86.3
SWE-Bench Verified	70.4
AIME 2026	95.7
KMMLU-Pro	78.4

(Official numbers from the base model card; quantized-build deltas are tracked in the KLD table above.)

Known Limitations

• Q v3 carries a disclosed Korean KLD regression (+8.4% vs its 4-bit/g32 base: 0.10553 → 0.11438) — the price of clearing the base's EN greedy loop while restoring EN diversity. English and code KLD are at or below base level; if your workload is Korean-critical and you can tolerate a known EN loop, prefer the T (Q8) tier or the base model.
• F v2 is a 2.34 bpw build: Korean KLD (0.75) is materially higher than the Q tier — expected at this bit budget; prefer Q v3 for Korean-heavy workloads. The DWQ LoRA adapters ship un-fused (alis_adapters/); all reported numbers are without them.
• Reasoning-block runaway (root-caused, disclosed): an external integrator reported F v2 thinking that never closed (22.6K chars, no <|think:end|>). This was traced to a serving bug, not the weights: fork mlx_lm.server ≤ 9d67a0c silently dropped --adapter-path, so that serve ran the bare 2-bit build without its DWQ LoRA compensators (fixed in 7937fda, see Install & run). With adapters actually applied, thinking terminated in 5/5 direct-path runs across both teams — at the official temp 1.0 / top_p 1.0 and at 0.7/0.95. Guidance stands regardless: official sampling, generous max_tokens, reasoning_effort="none" for short direct answers — and treat the adapters as part of the deployed F v2 configuration (a bare serve can ramble).
• Architecture support: solar_open2 is not yet in PyPI mlx-lm (0.31.3) — use the fork (default branch solar-open2; the adapter-drop fix is 7937fda) or the bundled solar_open2.py.
• Calibration coverage saturates at 200 samples (99.8% expert routing), but median tokens/expert (4,314) is ~19% of the 753B reference run; very rare experts may be under-distilled.
• Quantized builds inherit the base model's knowledge cutoff (2026-02) and language focus (EN/KO/JA).

License

Distributed under the Upstage Solar License (same as the base model). Per the license's derivative-model terms:

• Model name carries the "Solar" prefix ✅
• "Built with Solar" attribution displayed above ✅
• A copy of the Upstage Solar License is included in this repo ✅

Credits

• Base model: Upstage — Solar-Open2-250B (Upstage Solar License).
• MLX & mlx-lm: Apple ml-explore — solar_open2 architecture port (KDA hybrid-attention MoE, NoPE, sigmoid-gated GQA): Alis (avlp12) (fork).
• Mixed-precision DWQ quantization, per-slice acceptance gate + EN sentinel gate, F-tier quant_predicate_builder recipe (chokepoint 8b / embed+head 6b / attn+shared 4b / experts 2b): Alis (avlp12) (alis-dwq).

Citation

@misc{solar-open-2-2026,
  title={Solar Open 2 Technical Report},
  author={Upstage AI},
  year={2026},
  url={https://huggingface.co/upstage/Solar-Open2-250B}
}