nerve4/Qwen2.5-Coder-7B-GGUF

Qwen2.5-Coder-7B (base) — GGUF Q4_K_M

A Q4_K_M GGUF of Qwen/Qwen2.5-Coder-7B, the base (pre-trained, not instruction-tuned) code model, for code completion and fill-in-the-middle (FIM) in editors and local tools.

The upstream publisher ships GGUF files only for the Instruct models. Base models are the better choice for completion — they continue and infill code instead of answering — so this repo provides the base model in GGUF, converted directly from the official weights with no other changes.

File	qwen2.5-coder-7b-q4_k_m.gguf — 4.36 GiB (4683073376 bytes)
Quantization	Q4_K_M (4-bit k-quant, medium)
Base model	Qwen/Qwen2.5-Coder-7B @ 0396a76181e127dfc13e5c5ec48a8cee09938b02
Format	GGUF, converted with llama.cpp b11518
Licence	Apache-2.0, same as the upstream weights (LICENSE, NOTICE)
SHA-256	c1096c17c7f49aff6f82ec4b1ff36b87a52893a4a37e2cf644019e192930dcab

Intended use

• Inline code completion and fill-in-the-middle in an editor (Continue, llama.vscode, other FIM clients), or any tool that sends a prefix and a suffix.
• Plain code continuation from a prompt.

Not intended for chat or instructions. It is a base model: it does not follow requests like "write a function that…" reliably. For chat use the Instruct GGUF from the publisher.

Prompt format (FIM)

The tokenizer keeps Qwen2.5-Coder's FIM tokens. The model writes the code that belongs between prefix and suffix:

<|fim_prefix|>{code before the cursor}<|fim_suffix|>{code after the cursor}<|fim_middle|>

For context from other files of the same project, Qwen2.5-Coder also understands <|repo_name|> and <|file_sep|> (see the upstream model card).

Use

llama.cpp — server with the FIM endpoint (/infill):

llama-server -m qwen2.5-coder-7b-q4_k_m.gguf -c 4096 -ngl 99 --port 8012
curl -s http://127.0.0.1:8012/infill -d '{
  "input_prefix": "def fibonacci(n):\n    ",
  "input_suffix": "\n\nprint(fibonacci(10))\n",
  "n_predict": 64
}'

Raw completion (any OpenAI-compatible /v1/completions server, e.g. llama-server): send the FIM prompt above as prompt, with a low temperature and a stop on <|endoftext|>.

Ollama:

cat > Modelfile <<'EOF'
FROM ./qwen2.5-coder-7b-q4_k_m.gguf
TEMPLATE """{{ .Prompt }}"""
PARAMETER stop <|endoftext|>
EOF
ollama create qwen2.5-coder-7b-base -f Modelfile

Send FIM prompts in raw mode ("raw": true on /api/generate).

Editors: point a FIM / autocomplete provider at one of the servers above (for example Continue's autocomplete model with a llama.cpp or Ollama provider).

Nerve4 (a local AI gateway) installs this file by its hash and serves it for completions.

Hardware

The 4.36 GiB file plus a 4096-token context runs fully on an 8 GB GPU (measured on a GTX 1070 Ti: about 4.6 GiB of VRAM with all layers on the GPU). With less VRAM, offload part of the layers (-ngl); it also runs on the CPU, more slowly.

Verify

sha256sum qwen2.5-coder-7b-q4_k_m.gguf
# c1096c17c7f49aff6f82ec4b1ff36b87a52893a4a37e2cf644019e192930dcab

How it was made (reproducible)

Every input is pinned: the source weights by commit and per-file hash, the converter and quantizer by llama.cpp commit 1f8fa52318b594b373e7447a3c2bb36cdba9515f (b11518). With that llama.cpp checkout, built normally (keep the folder name hf: the converter may record it in the metadata, and the hash must match byte for byte):

hf download Qwen/Qwen2.5-Coder-7B --revision 0396a76181e127dfc13e5c5ec48a8cee09938b02 --local-dir hf
python convert_hf_to_gguf.py hf --outtype bf16 --outfile model-bf16.gguf
./llama-quantize model-bf16.gguf qwen2.5-coder-7b-q4_k_m.gguf Q4_K_M
sha256sum qwen2.5-coder-7b-q4_k_m.gguf     # expected: c1096c17c7f49aff6f82ec4b1ff36b87a52893a4a37e2cf644019e192930dcab

Upstream weight files read by the conversion (SHA-256 as listed by Hugging Face at that commit):

File	Size (bytes)	SHA-256
model-00001-of-00004.safetensors	4877660776	2296315c41ccc5e956a3800a30e9c32ce14f2228dc833f81e090891caa7d60d0
model-00002-of-00004.safetensors	4932751008	9db3a8f68a410e972e19e15f39fce99a22df8f3044311b9f09dd70d1646205e2
model-00003-of-00004.safetensors	4330865200	292de283f4dce64c46fa0a810e396e0c7d0f04b6896b14fe6197a978935a1890
model-00004-of-00004.safetensors	1089994880	0167e1c1b01ce71c87dd479157695cc1667342b9d2d75d2a64a648dfbaf04b8f

Changes from the upstream model

Format conversion (safetensors → GGUF, via bf16) and Q4_K_M quantization only. No fine-tuning, no merged adapters, no change to the tokenizer or the vocabulary.

Limitations

• Quantization to Q4_K_M costs a little accuracy compared with the bf16 weights.
• Everything that applies to the upstream base model applies here: it can produce incorrect, insecure or outdated code — review what it suggests.

Licence and credits

The weights are by the Qwen team, released under Apache-2.0; this conversion keeps that licence (see LICENSE and NOTICE). Conversion by the Nerve4 project.

@article{hui2024qwen2,
  title   = {Qwen2.5-Coder Technical Report},
  author  = {Hui, Binyuan and Yang, Jian and Cui, Zeyu and others},
  journal = {arXiv preprint arXiv:2409.12186},
  year    = {2024}
}