Instructions to use minsore/Quill-Gen-1 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use minsore/Quill-Gen-1 with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="minsore/Quill-Gen-1")

# pip install -U transformers accelerate
# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("minsore/Quill-Gen-1")
model = AutoModelForCausalLM.from_pretrained("minsore/Quill-Gen-1", device_map="auto")

Notebooks
Google Colab
Kaggle
Local Apps Settings

llama.cpp

How to use minsore/Quill-Gen-1 with llama.cpp:

Install (macOS, Linux)

curl -LsSf https://llama.app/install.sh | sh
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf minsore/Quill-Gen-1:Q4_K_M
# Run inference directly in the terminal:
llama cli -hf minsore/Quill-Gen-1:Q4_K_M

Install from WinGet (Windows)

winget install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf minsore/Quill-Gen-1:Q4_K_M
# Run inference directly in the terminal:
llama cli -hf minsore/Quill-Gen-1:Q4_K_M

Use pre-built binary

# Download pre-built binary from:
# https://github.com/ggerganov/llama.cpp/releases
# Start a local OpenAI-compatible server with a web UI:
./llama-server -hf minsore/Quill-Gen-1:Q4_K_M
# Run inference directly in the terminal:
./llama-cli -hf minsore/Quill-Gen-1:Q4_K_M

Build from source code

git clone https://github.com/ggerganov/llama.cpp.git
cd llama.cpp
cmake -B build
cmake --build build -j --target llama-server llama-cli
# Start a local OpenAI-compatible server with a web UI:
./build/bin/llama-server -hf minsore/Quill-Gen-1:Q4_K_M
# Run inference directly in the terminal:
./build/bin/llama-cli -hf minsore/Quill-Gen-1:Q4_K_M

Use Docker

docker model run hf.co/minsore/Quill-Gen-1:Q4_K_M

LM Studio
Jan

vLLM

How to use minsore/Quill-Gen-1 with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "minsore/Quill-Gen-1"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "minsore/Quill-Gen-1",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker

docker model run hf.co/minsore/Quill-Gen-1:Q4_K_M

SGLang

How to use minsore/Quill-Gen-1 with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "minsore/Quill-Gen-1" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "minsore/Quill-Gen-1",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "minsore/Quill-Gen-1" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "minsore/Quill-Gen-1",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Ollama
How to use minsore/Quill-Gen-1 with Ollama:
```
ollama run hf.co/minsore/Quill-Gen-1:Q4_K_M
```
Unsloth Desktop
Docker Model Runner
How to use minsore/Quill-Gen-1 with Docker Model Runner:
```
docker model run hf.co/minsore/Quill-Gen-1:Q4_K_M
```

Lemonade

How to use minsore/Quill-Gen-1 with Lemonade:

Pull the model

# Download Lemonade from https://lemonade-server.ai/
lemonade pull minsore/Quill-Gen-1:Q4_K_M

Run and chat with the model

lemonade run user.Quill-Gen-1-Q4_K_M

List all available models

lemonade list

Atomic Chat

Quill-Gen-1 / README.md

sollamon

Make AI Fly!

9e1a8c3 verified 21 days ago

preview code

Raw History Blame Contribute Delete

5.91 kB

---
language:
  - en
  - code
license: apache-2.0
library_name: transformers
pipeline_tag: text-generation
base_model: Qwen/Qwen2.5-Coder-0.5B-Base
tags:
  - code
  - python
  - code-generation
  - text-generation
  - qwen2.5
  - qwen
  - llama.cpp
  - gguf
  - unsloth
  - qlora
  - lora
  - small-models
  - edge-ai
  - local-ai
  - minsore
  - ukrainian
  - experimental
  - 0.5b
  - generation
model-index:
  - name: Quill Gen 1
    results:
      - task:
          type: text-generation
          name: Code Generation
        dataset:
          type: openai_humaneval
          name: HumanEval@50
        metrics:
          - type: pass@1
            value: 80.0
            name: Pass@1
---
# 🟢 Quill Gen 1

Lightweight Python code generation model built on Qwen2.5-Coder-0.5B.

Quill Gen 1 is a 0.5B parameter experimental model for generating Python functions from natural-language descriptions. It is the younger sibling of Pepper 1 Preview — same task, one third the size. Trained as an experiment in small-model code generation, it reaches 80% on HumanEval@50 while running comfortably on consumer hardware.

> ⚠️ **This is NOT a FIM model.** Quill Gen 1 cannot do fill-in-the-middle completion. For autocomplete use [Quill 1 Preview](https://huggingface.co/Minsore/quill-1-preview).

    Part of the Minsore family · minsore.com
## ✨ Highlights

- 🧠 **Instruction-tuned** — writes Python functions from plain descriptions
- 📦 **Tiny** — 0.5B params, ~400 MB in Q4_K_M
- 🎯 **Purpose-built** — code generation only, not chat, not FIM
- ⚡ **Fast** — designed for low-VRAM setups
- 🧪 **Experimental** — a proof of concept for small-model generation
- 🆓 Apache 2.0 — same license as base model

## 📊 Benchmarks

![Quill Gen comparison](assets/quill_gen_1_benchmarks.png)

Evaluated against 0.5B–1.5B code models. All runs used `temperature=0.0`, `max_tokens=512`, `--chat-template none`.

| Benchmark | Quill Gen 1 | Pepper 1 Preview | Quill 1 Preview | Qwen2.5-Coder 0.5B |
|---|---|---|---|---|
| HumanEval@50 | **80.0%** | 82.0% | 54.0% | 52.0% |
| HumanEval-Infilling (EditSim) | 0.161 | — | **0.423** | 0.045 |
| HumanEval-Infilling (Exact Match) | 0% | — | **16%** | 0% |
| Delulu FIM (EditSim) | 0.048 | — | **0.318** | 0.035 |

> 📌 **Key insight:** Quill Gen 1 reaches **80% HumanEval@50** — nearly matching Pepper 1 Preview (82%) at **one third the parameter count**. On FIM-specific tasks it underperforms significantly, which is expected: this model was trained for generation, not autocomplete.

## 🚀 Quick Start

### llama.cpp

```bash
llama-server -m quill-gen-1.Q4_K_M.gguf \
  --port 8080 \
  -ngl 99 \
  -c 768 \
  --chat-template none
```

**Requirements:** any GPU with ≥2 GB VRAM (full offload), or partial CPU offload as fallback.

### Request

```bash
curl http://localhost:8080/completion \
  -H "Content-Type: application/json" \
  -d '{
    "prompt": "### Instruction:\nWrite a Python function that checks if a number is prime.\n\n### Response:\n",
    "n_predict": 256,
    "temperature": 0.0
  }'
```

### Python

```python
import requests
def generate(instruction, max_tokens=256):
    prompt = f"### Instruction:\n{instruction}\n\n### Response:\n"
    r = requests.post("http://localhost:8080/completion", json={
        "prompt": prompt,
        "n_predict": max_tokens,
        "temperature": 0.0,
        "stop": ["### Instruction:", "<|endoftext|>"],
    })
    return r.json()["content"]
print(generate("Write a Python function that reverses a string."))
# → def reverse_string(s):
#       return s[::-1]
```

## ⚙️ Recommended Settings

| Parameter | Value | Notes |
|---|---|---|
| `--chat-template` | `none` | Required. Quill Gen 1 is not a chat model |
| `n_predict` | 128–256 | Works best on short completions |
| `temperature` | `0.0` | Deterministic; use `0.2` for variation |
| `repeat_penalty` | `1.1` | Prevents repetition |
| `-c` | `768` | Matches training context length |

## ⚠️ Limitations

- **Generation only** — does not support fill-in-the-middle, tool calling, or chat
- **Python-only** — trained exclusively on Python code
- **Small context** — 768 tokens; long files are truncated
- **Weak on FIM** — EditSim 0.16, Exact Match 0% (use Quill 1 instead)
- **Occasional over-explanation** — may include comments when only code is requested
- **Number looping** — may repeat large integers on some prompts (use `repeat_penalty=1.1`)

## 🧬 Training Details

| | |
|---|---|
| Base model | Qwen2.5-Coder-0.5B-Base |
| Method | QLoRA (r=16, α=16) |
| Data | FIM-converted Python instruction data |
| Epochs | 1 |
| Context | 768 tokens |
| Optimizer | `paged_adamw_8bit` |

## 📁 Files

| File | Size | Description |
|---|---|---|
| `quill-gen-1.Q4_K_M.gguf` | ~400 MB | Ready to use with llama.cpp (recommended) |
| `model.safetensors` | ~1 GB | Full-precision merged weights |
| `config.json` | — | Model config |
| `tokenizer.json` | — | Tokenizer |
| `tokenizer_config.json` | — | Tokenizer config |
| `generation_config.json` | — | Generation defaults (optional) |

## 🗺️ Roadmap

- **Quill 2** — FIM autocomplete, fixed suffix handling, JS/TS/Rust support
- **Pepper 2** — improved MBPP and LiveCodeBench
- **Symphony** — flagship agentic code model (3B MoE)

## 📜 License

Apache 2.0 — same as the base Qwen2.5-Coder-0.5B model.

## 🙏 Credits

- Base model: [Qwen2.5-Coder-0.5B-Base](https://huggingface.co/Qwen/Qwen2.5-Coder-0.5B-Base) by Alibaba Cloud
- Training framework: Unsloth
- Inference: llama.cpp

## 📬 Contact

Minsore — Ukrainian AI lab building open language models.

- 🌐 [minsore.com](https://minsore.com)
- 🤗 [huggingface.co/Minsore](https://huggingface.co/Minsore)
- 💬 Built by [@Sollamon](https://huggingface.co/Sollamon)

⭐ If Quill Gen 1 is useful, star the repo and share your results.

	---
	language:
	- en
	- code
	license: apache-2.0
	library_name: transformers
	pipeline_tag: text-generation
	base_model: Qwen/Qwen2.5-Coder-0.5B-Base
	tags:
	- code
	- python
	- code-generation
	- text-generation
	- qwen2.5
	- qwen
	- llama.cpp
	- gguf
	- unsloth
	- qlora
	- lora
	- small-models
	- edge-ai
	- local-ai
	- minsore
	- ukrainian
	- experimental
	- 0.5b
	- generation
	model-index:
	- name: Quill Gen 1
	results:
	- task:
	type: text-generation
	name: Code Generation
	dataset:
	type: openai_humaneval
	name: HumanEval@50
	metrics:
	- type: pass@1
	value: 80.0
	name: Pass@1
	---

	# 🟢 Quill Gen 1

	Lightweight Python code generation model built on Qwen2.5-Coder-0.5B.

	Quill Gen 1 is a 0.5B parameter experimental model for generating Python functions from natural-language descriptions. It is the younger sibling of Pepper 1 Preview — same task, one third the size. Trained as an experiment in small-model code generation, it reaches 80% on HumanEval@50 while running comfortably on consumer hardware.

	> ⚠️ This is NOT a FIM model. Quill Gen 1 cannot do fill-in-the-middle completion. For autocomplete use [Quill 1 Preview](https://huggingface.co/Minsore/quill-1-preview).

	Part of the Minsore family · minsore.com

	## ✨ Highlights

	- 🧠 Instruction-tuned — writes Python functions from plain descriptions
	- 📦 Tiny — 0.5B params, ~400 MB in Q4_K_M
	- 🎯 Purpose-built — code generation only, not chat, not FIM
	- ⚡ Fast — designed for low-VRAM setups
	- 🧪 Experimental — a proof of concept for small-model generation
	- 🆓 Apache 2.0 — same license as base model

	## 📊 Benchmarks

	![Quill Gen comparison](assets/quill_gen_1_benchmarks.png)

	Evaluated against 0.5B–1.5B code models. All runs used `temperature=0.0`, `max_tokens=512`, `--chat-template none`.

	\| Benchmark \| Quill Gen 1 \| Pepper 1 Preview \| Quill 1 Preview \| Qwen2.5-Coder 0.5B \|
	\|---\|---\|---\|---\|---\|
	\| HumanEval@50 \| 80.0% \| 82.0% \| 54.0% \| 52.0% \|
	\| HumanEval-Infilling (EditSim) \| 0.161 \| — \| 0.423 \| 0.045 \|
	\| HumanEval-Infilling (Exact Match) \| 0% \| — \| 16% \| 0% \|
	\| Delulu FIM (EditSim) \| 0.048 \| — \| 0.318 \| 0.035 \|

	> 📌 Key insight: Quill Gen 1 reaches 80% HumanEval@50 — nearly matching Pepper 1 Preview (82%) at one third the parameter count. On FIM-specific tasks it underperforms significantly, which is expected: this model was trained for generation, not autocomplete.

	## 🚀 Quick Start

	### llama.cpp

	```bash
	llama-server -m quill-gen-1.Q4_K_M.gguf \
	--port 8080 \
	-ngl 99 \
	-c 768 \
	--chat-template none
	```

	Requirements: any GPU with ≥2 GB VRAM (full offload), or partial CPU offload as fallback.

	### Request

	```bash
	curl http://localhost:8080/completion \
	-H "Content-Type: application/json" \
	-d '{
	"prompt": "### Instruction:\nWrite a Python function that checks if a number is prime.\n\n### Response:\n",
	"n_predict": 256,
	"temperature": 0.0
	}'
	```

	### Python

	```python
	import requests

	def generate(instruction, max_tokens=256):
	prompt = f"### Instruction:\n{instruction}\n\n### Response:\n"
	r = requests.post("http://localhost:8080/completion", json={
	"prompt": prompt,
	"n_predict": max_tokens,
	"temperature": 0.0,
	"stop": ["### Instruction:", "<\|endoftext\|>"],
	})
	return r.json()["content"]

	print(generate("Write a Python function that reverses a string."))
	# → def reverse_string(s):
	# return s[::-1]
	```

	## ⚙️ Recommended Settings

	\| Parameter \| Value \| Notes \|
	\|---\|---\|---\|
	\| `--chat-template` \| `none` \| Required. Quill Gen 1 is not a chat model \|
	\| `n_predict` \| 128–256 \| Works best on short completions \|
	\| `temperature` \| `0.0` \| Deterministic; use `0.2` for variation \|
	\| `repeat_penalty` \| `1.1` \| Prevents repetition \|
	\| `-c` \| `768` \| Matches training context length \|

	## ⚠️ Limitations

	- Generation only — does not support fill-in-the-middle, tool calling, or chat
	- Python-only — trained exclusively on Python code
	- Small context — 768 tokens; long files are truncated
	- Weak on FIM — EditSim 0.16, Exact Match 0% (use Quill 1 instead)
	- Occasional over-explanation — may include comments when only code is requested
	- Number looping — may repeat large integers on some prompts (use `repeat_penalty=1.1`)

	## 🧬 Training Details

	\| \| \|
	\|---\|---\|
	\| Base model \| Qwen2.5-Coder-0.5B-Base \|
	\| Method \| QLoRA (r=16, α=16) \|
	\| Data \| FIM-converted Python instruction data \|
	\| Epochs \| 1 \|
	\| Context \| 768 tokens \|
	\| Optimizer \| `paged_adamw_8bit` \|

	## 📁 Files

	\| File \| Size \| Description \|
	\|---\|---\|---\|
	\| `quill-gen-1.Q4_K_M.gguf` \| ~400 MB \| Ready to use with llama.cpp (recommended) \|
	\| `model.safetensors` \| ~1 GB \| Full-precision merged weights \|
	\| `config.json` \| — \| Model config \|
	\| `tokenizer.json` \| — \| Tokenizer \|
	\| `tokenizer_config.json` \| — \| Tokenizer config \|
	\| `generation_config.json` \| — \| Generation defaults (optional) \|

	## 🗺️ Roadmap

	- Quill 2 — FIM autocomplete, fixed suffix handling, JS/TS/Rust support
	- Pepper 2 — improved MBPP and LiveCodeBench
	- Symphony — flagship agentic code model (3B MoE)

	## 📜 License

	Apache 2.0 — same as the base Qwen2.5-Coder-0.5B model.

	## 🙏 Credits

	- Base model: [Qwen2.5-Coder-0.5B-Base](https://huggingface.co/Qwen/Qwen2.5-Coder-0.5B-Base) by Alibaba Cloud
	- Training framework: Unsloth
	- Inference: llama.cpp

	## 📬 Contact

	Minsore — Ukrainian AI lab building open language models.

	- 🌐 [minsore.com](https://minsore.com)
	- 🤗 [huggingface.co/Minsore](https://huggingface.co/Minsore)
	- 💬 Built by [@Sollamon](https://huggingface.co/Sollamon)

	⭐ If Quill Gen 1 is useful, star the repo and share your results.