helmsman-4b

A merged, fine-tuned 4B model built on Qwen/Qwen3-4B-Instruct-2507, distilled to answer questions grounded in a small document set (a specialist for the Unhosted local-inference daemon). Trained with QLoRA, then merged into the base weights and shipped as a standalone model (full safetensors + a Q4_K_M GGUF) — no adapter needed at load time.

Honesty note (2026-09-26): an earlier version of this card described the repo as a LoRA adapter. It is not — the LoRA was merged and this repo ships the full merged model (model.safetensors, ~8 GB) and a quantized GGUF. library_name and the load examples below reflect that.

TL;DR

• Base: Qwen/Qwen3-4B-Instruct-2507 (4B params, 262K context)
• Method: QLoRA (r=16, α=32) SFT → merged into the base (this repo is the merge)
• Formats: model.safetensors (fp16, transformers) · helmsman-4b-Q4_K_M.gguf (llama.cpp / Ollama)
• Training data: 305 synthetic (prompt, response) pairs, teacher = Claude (Opus-class)
• Hardware / time: Apple Silicon, bf16, ~30 min/run
• License: AGPL-3.0-or-later (inherits the base model's terms + the daemon's)

What it's good at

Answering questions grounded in the document set it was distilled on — a focused, on-device specialist that runs fast in 4-bit on a laptop.

What it's NOT for

• Open-ended chat unrelated to the training documents — here it is no better than the base model.
• Tool use / function calling — it was not trained for it.
• Anything safety-critical. This is a distillation: the teacher's failure modes and any defects in its outputs carry through.

Use it

Transformers (fp16 safetensors):

from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

mid = "sinhaankur/helmsman-4b"
tok = AutoTokenizer.from_pretrained(mid)
model = AutoModelForCausalLM.from_pretrained(mid, torch_dtype=torch.float16, device_map="auto")

msgs = [{"role": "user", "content": "Summarise the key points of the document."}]
ids = tok.apply_chat_template(msgs, add_generation_prompt=True, return_tensors="pt").to(model.device)
out = model.generate(ids, max_new_tokens=512, temperature=0.7, top_p=0.8, top_k=20)
print(tok.decode(out[0][ids.shape[-1]:], skip_special_tokens=True))

Ollama (Q4_K_M GGUF, ~2.5 GB — runs on a laptop):

# Modelfile
printf 'FROM ./helmsman-4b-Q4_K_M.gguf\nPARAMETER temperature 0.7\nPARAMETER top_p 0.8\nPARAMETER top_k 20\n' > Modelfile
ollama create helmsman-4b -f Modelfile
ollama run helmsman-4b "Summarise the key points of the document."

llama.cpp:

llama-cli -hf sinhaankur/helmsman-4b:Q4_K_M -p "Summarise the key points of the document."

Training recipe (how it was made)

1. Synthetic data — gen_data.py generated 305 (prompt, response) pairs against a Claude (Opus-class) teacher, one chat-completion per source document.
2. SFT — 3 epochs, lr 2e-4, effective batch 16, cosine schedule, 5% warmup.
3. QLoRA — base in 4-bit NF4, compute dtype bf16, double quantization; LoRA on all attention + MLP projections; then merged into the base and exported to GGUF.
4. Seed 42 for reproducibility.

The gen_data.py / train.py recipe lives in the Unhosted distillation project (unhosted-ai); the raw train.jsonl is not shipped in this repo (an earlier card claimed it was — corrected here).

Evaluation

Metric	This model
LLM-as-judge (0–46)	32.5 / 46
Normalized	71%

Eval set: 46 held-out prompts generated with a different seed than training, so the trainer never saw them. This measures in-distribution grounded QA only — see "What it's NOT for" for the honest limits.