Qwen3.5-4B-RustCoder

Qwen/Qwen3.5-4B fine-tuned to write Rust that compiles, passes tests and is clippy-clean. It keeps its thinking mode. These are bf16 safetensors in the original Qwen3.5 layout, for transformers and the tools that read Hugging Face checkpoints.

For Apple Silicon, the MLX 8-bit version is alan13367/Qwen3.5-4B-RustCoder-MLX-8bit. That is the checkpoint that was trained and evaluated, and these weights are the same model (see About these weights).

Every training answer was checked by a gate that builds the code, runs hidden tests and lints it. Answers that failed it were not used.

Results

Pass rate: the share of answers whose code passes the task's hidden tests. Each task was sampled 4 times, with thinking on and no system prompt. The test tiers were never trained on.

Pass rate	Easy (29 tasks)	Hard (40)	Quality (29)	Dev (82)
Qwen3.5-4B (8-bit)	41.4%	7.5%	8.6%	20.7%
Qwen3.5-4B-RustCoder	51.7%	8.8%	17.2%	22.9%

• Pooled over all 180 tasks: +4.3 points over the base (95% CI +0.6 to +8.2) and +0.046 graded reward (CI +0.012 to +0.081).
• The tiers are small. A single tier only detects a gain of about 9 points (hard, quality) or 14 points (easy), so only the pooled gain is established.
• Quality covers fixing bugs, refactoring, writing tests and new domains. A refactor counts only if it is also lint-clean. Dev is the tier used to pick checkpoints.
• At least 1 of the 4 attempts passes on 82.8% of easy tasks and 22.5% of hard tasks.

General ability, against the base (subsets of each benchmark):

	MMLU (200)	ARC-Challenge (150)	GSM8K (100)	HellaSwag (150)
Qwen3.5-4B (8-bit)	69.5%	89.3%	92.0%	80.7%
Qwen3.5-4B-RustCoder	69.0%	90.0%	92.0%	82.0%

On general text, its token distribution stays close to the base: KL 0.026 nats/token and 94.2% top-1 agreement.

Usage

With transformers (a version with Qwen3.5 support):

from transformers import AutoModelForImageTextToText, AutoTokenizer

repo = "alan13367/Qwen3.5-4B-RustCoder"
tok = AutoTokenizer.from_pretrained(repo)
model = AutoModelForImageTextToText.from_pretrained(repo, dtype="auto", device_map="auto")
messages = [{"role": "user", "content": "Write in Rust: pub fn median(xs: &mut [i64]) -> Option<f64>"}]
inputs = tok.apply_chat_template(messages, add_generation_prompt=True, enable_thinking=True,
                                 return_tensors="pt", return_dict=True).to(model.device)
out = model.generate(**inputs, max_new_tokens=4096, do_sample=True, temperature=0.6, top_p=0.95, top_k=20)
print(tok.decode(out[0, inputs["input_ids"].shape[1]:], skip_special_tokens=True))

Recommended:

• Use no system prompt. Training and evaluation used none. A long style prompt sent the base model into runaway reasoning.
• Leave thinking on and cap the output at about 4096 tokens. Answers that finish take a median of about 1.2k tokens. An attempt that runs to the cap is usually stuck in a reasoning loop, so resampling helps more than raising the cap.
• Use temperature 0.6, top-p 0.95 and top-k 20. These are the settings behind the numbers above.

Training

LoRA SFT, one epoch. Rank 16, alpha 32, dropout 0.05, on the last 16 decoder layers. Learning rate 1e-4, batches of up to 4096 tokens. In total 739 samples, 784k trained tokens, 402 steps. Validation loss was 0.411.

The data is mostly the model's own work, filtered by the gate:

Source	Records	What it is
Own passing attempts	238	The base model's own attempts at the training tasks, kept only if they pass the hidden tests and the test lints
Self-repairs	277	The base model fixing its own failed attempts, given the compiler or test output as the next user turn, kept only if the fix passes
Teacher traces	175	Passing traces from DeepSeek V4.1 Flash. 159 of them repair the 4B's own failed attempts. Only traces within the 4B's own length and style were kept (completion of at most 2048 tokens, little second-guessing)
Replay	88	The base model's own answers to general, non-Rust prompts, to keep its general ability

What the experiments behind this checkpoint showed:

• The model learns best from its own style. Reference solutions with model-written reasoning made it worse (easy 41.4% → 26.7%). It imitated their dense style and compiled less often.
• Unfiltered teacher traces also made it worse (easy 34.5%). Their long, second-guessing reasoning taught it to never leave thinking: 177 of 720 answers ran to the cap.
• Filtering the teacher traces fixed that. This checkpoint has 36 of 720 answers that run to the cap, against 82 for the base.
• The loss covers only the final reply. In repair records, the failed attempt and the feedback are context, not training targets.

About these weights

• Where they come from. The fine-tune was trained and merged on the 8-bit MLX model. These weights are that checkpoint dequantized to bf16, so they are exactly the evaluated model: every weight is its 8-bit value. They do not get back the precision the 8-bit quantization lost. Weights the fine-tune left alone differ from the original Qwen3.5-4B bf16 by about 0.7% (relative error), the size of the 8-bit quantization error.
• Checks. Each tensor's name, shape and dtype matches Qwen/Qwen3.5-4B. Tensors that were never quantized (norms, conv1d, A_log, the vision tower) match the original exactly.
• Tokenizer and config. Both are Qwen/Qwen3.5-4B's. The tokenizer gives the same ids as the one used in training.
• No MTP head. The 8-bit checkpoint the fine-tune was built on has no multi-token-prediction head, so MTP speculative decoding (vLLM, SGLang) is not available.

Limitations

• It is a 4B model and Rust is hard for it. Many of its answers still fail to compile (about half did for the base model and the previous fine-tune), mostly with type, trait-bound and method errors in its own code (E0308, E0277, E0599). Harder multi-part tasks rarely pass (hard tier 8.8%). It does much better with a compiler in the loop: given the error output it fixes a share of its failures, and the best of 4 attempts passes far more often than a single one.
• It still over-thinks sometimes. Cap the output, and resample when an answer gets stuck.
• Only the text side was trained. The vision tower is the base's, unchanged and untested here.
• The evaluation sets are small (29 to 82 tasks per tier), so each per-tier number has a wide margin. Treat the gains as modest.
• It was trained and evaluated on standalone tasks: a function, a type or a small module checked by hidden tests, plus bug fixes, refactors and test-writing. It has not been measured on multi-file projects or long-context editing.

License

Apache 2.0, the license of the base model Qwen/Qwen3.5-4B.