Qwen3.5-4B-RustCoder-GGUF

GGUF quantizations of alan13367/Qwen3.5-4B-RustCoder: Qwen/Qwen3.5-4B fine-tuned to write Rust that compiles, passes tests and is clippy-clean, with its thinking mode kept. They are for llama.cpp and other GGUF runtimes. Text only: no vision projector is included.

File	Bits per weight	Size	Use
Qwen3.5-4B-RustCoder-Q8_0.gguf	8.5	4.5 GB	Closest to the evaluated model
Qwen3.5-4B-RustCoder-Q4_K_M.gguf	5.1	2.7 GB	Smaller and faster, slightly less accurate (see Checks)

Other versions:

• Qwen3.5-4B-RustCoder: bf16 safetensors, for transformers.
• Qwen3.5-4B-RustCoder-MLX-8bit: MLX, for Apple Silicon. This is the checkpoint that was trained and evaluated.

Results of the fine-tune

Pass rate on held-out Rust tasks with hidden tests: 4 samples per task, thinking on, no system prompt. These were measured on the MLX 8-bit checkpoint. The GGUF files were not run on these tiers.

Pass rate	Easy (29 tasks)	Hard (40)	Quality (29)	Dev (82)
Qwen3.5-4B (8-bit)	41.4%	7.5%	8.6%	20.7%
Qwen3.5-4B-RustCoder	51.7%	8.8%	17.2%	22.9%

Pooled over all 180 tasks, the gain over the base is +4.3 points (95% CI +0.6 to +8.2). The data, the training and the limitations are in the main model card.

Usage

With llama.cpp (a build with Qwen3.5 support):

llama-server -hf alan13367/Qwen3.5-4B-RustCoder-GGUF:Q8_0 -c 8192 \
  --temp 0.6 --top-p 0.95 --top-k 20

Then send chat requests to http://localhost:8080/v1/chat/completions. Use :Q4_K_M for the smaller file. Other GGUF runtimes, such as Ollama and LM Studio, can load these files if their bundled llama.cpp supports the qwen35 architecture. They were not tested here.

Recommended:

• Use no system prompt. Training and evaluation used none.
• Leave thinking on. The chat template thinks by default.
• Allow about 4096 output tokens and a context of at least 8192. Answers that finish take a median of about 1.2k tokens. An answer that runs to the cap is usually stuck in a reasoning loop, so resampling helps more than a longer cap.
• Use temperature 0.6, top-p 0.95 and top-k 20. These are the settings behind the results.

Checks

Both files were converted from the bf16 weights with llama.cpp's convert_hf_to_gguf.py (build b12110), then quantized with llama-quantize, without an importance matrix. Tested with llama.cpp on CPU.

Perplexity on six of the model's own training answers, rendered with its chat template (llama-perplexity, 2048-token chunks):

File	Perplexity	Loss per token
bf16 (not uploaded)	1.3632 ± 0.019	0.310
Q8_0	1.3630 ± 0.019	0.310
Q4_K_M	1.3760 ± 0.019	0.319

Q8_0 is indistinguishable from bf16. Q4_K_M costs about 1% in perplexity. The bf16 weights have a loss of 0.24 to 0.44 on the same kind of answers in transformers, which matches the training loss.

Neither file has been run on the evaluation tiers yet.

License

Apache 2.0, the license of the base model Qwen/Qwen3.5-4B.