Sahuagin/Ornith-1.0-35B-GGUF-router-intact

Ornith-1.0-35B — router-intact asymmetric GGUF quants (2/3/4-bit experts)

Sub-Q4 GGUF quantizations of DeepReinforce's Ornith-1.0-35B that the official GGUF repo does not ship, built with asymmetric expert quantization: the ~93% of parameters in the routed experts are crushed to 2/3/4-bit while everything the model steers with is kept at high precision.

This is a re-quantization of DeepReinforce's Ornith-1.0-35B using the asymmetric-expert-quantization technique pioneered by DeepSeek (FP4-experts/FP8-rest in V3/V4) and popularized for GGUF by Salvatore Sanfilippo (antirez). The only contribution here is the specific application — an importance matrix calibrated on agentic coding data — and the benchmarks below. All credit for the model belongs to DeepReinforce, for the technique to DeepSeek and antirez, and for the tooling to Georgi Gerganov and the llama.cpp contributors (binaries built by Unsloth).

Files

file	experts	everything else	size	fits 24GB card
ornith-35b-expq4_K.gguf	q4_K	q8_0 (router f32)	20 GB	yes (32k ctx)
ornith-35b-expq3_K.gguf	q3_K	q8_0 (router f32)	16 GB	yes (64k+ ctx)
ornith-35b-expq2_K.gguf	q2_K	q8_0 (router f32)	13 GB	yes (~131k ctx)
imatrix-ornith35b.dat	—	importance matrix used above	184 MB	—

Precisely: routed experts (ffn_{gate,up,down}_exps) take the low-bit type; the router (ffn_gate_inp, kept f32), attention, shared expert (ffn_*_shexp), embeddings and output head stay at q8_0. The intuition: experts store knowledge, the preserved tensors carry control flow (routing, attention, format discipline) — so agentic/tool-use ability survives far below the usual quantization floor.

Measured behavior (private 34-case real-PR review suite + 30-run agentic tool-loop bench)

rung	code review (score / recall)	agentic tool-loop	notes
q4	0.954 on a 5-case public-style suite; 0.78 / 0.85 on the 34-case suite	73%	ties gemma4:31b / gpt-oss:20b as a local reviewer
q3	0.929	77%
q2	0.897	80%	agentic ability statistically identical to q4

Treat these as relative signals (private fixtures), not leaderboard numbers. The stable findings across the study: recall stays perfect on seeded defects at every rung; review precision degrades mildly with expert bits; agentic tool-use does not degrade at all down to 2-bit experts.

Recipe (llama.cpp)

# importance matrix on a RAM-fitting quant (activation stats are robust to base precision)
llama-imatrix -m ornith-q5.gguf -f calibration.txt -o imatrix.dat --chunks 400

# asymmetric quant from the bf16 GGUF: experts low-bit, rest q8_0, router stays f32
llama-quantize --imatrix imatrix.dat --tensor-type exps=q2_K ornith-bf16.gguf out.gguf q8_0

The included imatrix was computed on an agentic Rust-coding corpus (public code; per-tensor activation statistics only — no text is recoverable from it). For a general-purpose re-quant, substitute a broad calibration set.

Serving notes

• Ornith-1.0-35B is a hybrid Transformer/SSM MoE (KV on 10 of 40 layers). llama.cpp pins it to n_seq_max = 1 regardless of parallel settings: one loaded instance serves one request at a time. For concurrency, run N instances (one per GPU) — every rung here fits a 24 GB card.
• KV cache is tiny (~340 MB at 32k, q8_0); the context-scaling cost is the compute buffer. Cap context (e.g. 32k) unless you need more; q2 holds ~131k on a single 24 GB card.
• ollama: FROM <gguf> with RENDERER ornith / PARSER ornith, stop <|im_end|>, temperature 0.6, and a num_ctx that matches your VRAM plan.

Attribution

• Model: Ornith-1.0-35B by DeepReinforce (MIT), post-trained on the Gemma 4 / Qwen 3.5 lineage.
• Technique: asymmetric expert quantization — DeepSeek (V3/V4 FP4-experts serving) and antirez's DS4 2-bit-experts GGUF work.
• Tooling: llama.cpp (llama-quantize --tensor-type, llama-imatrix) by Georgi Gerganov and contributors; binaries built by Unsloth.

License: MIT, inherited from the base model.