sizzlebop/PrimeMind-9B-GGUF

PrimeMind-9B GGUF

This repository provides GGUF quantizations and multimodal vision projector weights for CrowdMind/PrimeMind-9B, a fine-tuned version of Qwen3.5-9B optimized for compact, information-dense reasoning.

PrimeMind-9B uses <think> tags to structure its thought process before generating final responses, cutting verbose reasoning filler while preserving mathematical and visual deduction accuracy.

All GGUF files were dequantized from the original packed BitsAndBytes NF4 4-bit weights into native full-precision BF16 safetensors, converted using llama.cpp at native BF16 precision, and quantized into standard k-quant formats. The vision encoder and projection adapter are provided as a standalone mmproj file for multimodal vision tasks.

---

Available Files and Quantizations

File	Quant Type	Size	Description / Recommendation
PrimeMind-9B-BF16.gguf	BF16	16.69 GB	Full precision base text model. Highest fidelity reference weights.
PrimeMind-9B-Q8_0.gguf	Q8_0	8.87 GB	Near-lossless 8-bit quantization. Recommended for critical precision.
PrimeMind-9B-Q6_K.gguf	Q6_K	6.85 GB	High quality retention with minimal degradation. Excellent balance.
PrimeMind-9B-Q5_K_M.gguf	Q5_K_M	6.02 GB	Balanced compression for reasoning and code extraction.
PrimeMind-9B-Q4_K_M.gguf	Q4_K_M	5.24 GB	Fast, lightweight 4-bit quant. Recommended default for local workstation inference.
PrimeMind-9B-Q3_K_M.gguf	Q3_K_M	4.31 GB	Lower memory footprint for hardware with limited VRAM.
PrimeMind-9B-Q2_K.gguf	Q2_K	3.56 GB	Maximum compression for testing on low-memory hardware.
mmproj-PrimeMind-9B-BF16.gguf	BF16 (Vision)	0.86 GB	Multimodal projector (27-layer ViT vision tower + projector). Required for image inputs.

---

Technical Specifications

• Base Architecture: Qwen3_5ForConditionalGeneration (model_type: qwen3_5)
• Text Backbone: qwen3_5_text (32 hidden layers, hybrid DeltaNet linear attention + full attention)
• Vision Tower: qwen3_5_vision (27-layer ViT, hidden size 1152, patch size 16, spatial merge size 2)
• Parameters: ~9B active
• Context Length: Up to 262,144 tokens
• Vocabulary Size: 248,320 tokens
• Native Precision: bfloat16
• License: Apache 2.0

---

Training Details

• Base Model: Qwen/Qwen3.5-9B (multimodal, 5.8B active params)
• Fine-Tuning Method: LoRA SFT (rank 64, alpha 128, dropout 0.05)
• Target Modules: q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj
• Datasets:
  • catsaresupercool/synthetic-caveman-thinking (600 math/reasoning examples)
  • nibauman/objectnav-sft-claude-caveman (600 navigation reasoning examples with images)
• Total Training Samples: 1,200 (600 text + 600 multimodal)
• Training Loss: 5.59 -> 0.78

---

Prompt Format and Compressed Reasoning

PrimeMind-9B uses standard ChatML syntax with an explicit <think> reasoning block:

<|im_start|>system
You are a helpful assistant.<|im_end|>
<|im_start|>user
What is 25 + 37?<|im_end|>
<|im_start|>assistant
<think>
25 + 37 = 62
</think>

62<|im_end|>

---

How to Use

1. Multimodal Inference (Text + Image)

Pass both the model and the mmproj projector to llama-qwen2vl-cli:

llama-qwen2vl-cli \
    -m ./PrimeMind-9B-Q4_K_M.gguf \
    --mmproj ./mmproj-PrimeMind-9B-BF16.gguf \
    --image ./room_scene.png \
    -p "Where is the blue mug on the desk?" \
    -n 256

2. Text Inference via llama-cli

Run single-turn reasoning tasks directly:

llama-cli \
    -m ./PrimeMind-9B-Q4_K_M.gguf \
    -p "<|im_start|>system\nYou are a helpful assistant.<|im_end|>\n<|im_start|>user\nSolve: 14 * 17<|im_end|>\n<|im_start|>assistant\n<think>\n" \
    -n 256 \
    -st

3. OpenAI-Compatible Server via llama-server

Start a local server supporting both text and image endpoints:

llama-server \
    -m ./PrimeMind-9B-Q4_K_M.gguf \
    --mmproj ./mmproj-PrimeMind-9B-BF16.gguf \
    --host 0.0.0.0 \
    --port 8080 \
    -c 8192

4. Ollama Modelfile

Create a Modelfile:

FROM ./PrimeMind-9B-Q4_K_M.gguf
TEMPLATE """{{ if .System }}<|im_start|>system
{{ .System }}<|im_end|>
{{ end }}{{ if .Prompt }}<|im_start|>user
{{ .Prompt }}<|im_end|>
{{ end }}<|im_start|>assistant
<think>
"""
PARAMETER stop "<|im_end|>"
PARAMETER stop "<|im_start|>"

Build and run:

ollama create primemind-9b -f Modelfile
ollama run primemind-9b

---

Conversion Notes

PrimeMind-9B was originally published with 1,074 packed BitsAndBytes NF4 4-bit weight tensors. Direct conversion with convert_hf_to_gguf.py fails on packed NF4 shapes. Prior to GGUF export, all weights were reconstructed to full BF16 precision using bitsandbytes.functional.dequantize_4bit, verifying exact match across all 760 layer tensors before writing GGUF binaries.