kd13/Type-o1-mini-instruct-GGUF

Type-o1-mini-instruct - GGUF

GGUF quantizations of kd13/Type-o1-mini-instruct, a compact general-purpose instruct model (~1B parameters) for everyday assistant use.

Converted with llama.cpp. The IQ quant was produced with an importance matrix; the rest are static quants.

Read the Usage section before running these files. This model uses a custom chat template, so llama.cpp requires the --jinja flag. Without it you will get this custom template is not supported.

Provided quants

Sorted by size, which is not the same as sorted by quality. IQ-quants are often preferable to non-IQ quants of a similar size.

Link	Type	Size/GB	Notes
GGUF	Q2_K	0.6
GGUF	Q3_K_S	0.6
GGUF	Q3_K_M	0.7	lower quality
GGUF	Q3_K_L	0.7
GGUF	IQ4_XS	0.7
GGUF	Q4_K_S	0.8	fast, recommended
GGUF	Q4_K_M	0.8	fast, recommended
GGUF	Q5_K_S	0.9
GGUF	Q5_K_M	0.9
GGUF	Q6_K	1.0	very good quality
GGUF	Q8_0	1.3	fast, best quality
GGUF	f16	2.5	16 bpw, overkill

Which one should I pick?

For a model this small the practical range is Q4_K_M through Q8_0. A 1B model has little redundancy to give up, so the very low-bit quants lose more than they would on a 7B. Q2_K and Q3_K_S are included for completeness rather than as recommendations.

Chat template

This model was fine-tuned on a custom template, not the standard Llama 3 header format. Each message is wrapped as:

''' <|begin_of_text|>{role} {content}<|end_of_text|> '''

and generation is prompted with a trailing <|begin_of_text|>assistant\n. The full Jinja template is embedded in every GGUF file in this repo, so any runtime with Jinja support applies it automatically.

Because this format is not one of llama.cpp's built-in recognised templates, its C++ template matcher will reject it. Passing --jinja tells llama.cpp to use the embedded Jinja template instead, which is what you want.

Usage

llama.cpp

llama-completion -m Type-o1-mini-instruct.Q4_K_M.gguf --jinja \
  -sys "You are a helpful assistant." \
  -p "Explain photosynthesis in two sentences."

Recent llama.cpp builds renamed llama-cli to llama-completion; on older builds use llama-cli with the same flags. For raw text completion with no template applied at all, add -no-cnv and drop --jinja.

Server:

llama-server -m Type-o1-mini-instruct.Q4_K_M.gguf --jinja -c 4096

Omitting --jinja produces this custom template is not supported — that is a template-matching error, not a corrupt file.

Ollama

ollama run hf.co/kd13/Type-o1-mini-instruct-GGUF:Q4_K_M