File size: 6,756 Bytes
bf386c5
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
5ecafff
bf386c5
 
 
 
 
 
 
 
5ecafff
bf386c5
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
5ecafff
 
 
 
 
 
bf386c5
5ecafff
bf386c5
 
 
 
5ecafff
 
 
 
 
bf386c5
 
 
5ecafff
bf386c5
 
 
 
 
 
 
 
 
 
 
5ecafff
 
bf386c5
 
 
 
 
 
 
 
 
 
5ecafff
 
 
 
 
 
 
 
 
bf386c5
5ecafff
bf386c5
5ecafff
bf386c5
5ecafff
bf386c5
 
5ecafff
bf386c5
 
 
 
 
 
 
 
5ecafff
bf386c5
 
 
 
 
 
 
 
 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
---
base_model: Qwen/Qwen2.5-0.5B-Instruct
library_name: peft
license: apache-2.0
datasets:
- bernabepuente/python-instruction-dataset
language:
- en
pipeline_tag: text-generation
tags:
- lora
- peft
- sft
- python
- code
- qwen2
---

# Qwen2.5-0.5B-Instruct · Python assistant LoRA (learning project)

A LoRA adapter for [Qwen/Qwen2.5-0.5B-Instruct](https://huggingface.co/Qwen/Qwen2.5-0.5B-Instruct), supervised fine-tuned on the small
[bernabepuente/python-instruction-dataset](https://huggingface.co/datasets/bernabepuente/python-instruction-dataset) dataset to answer Python questions: writing code from instructions,
explaining concepts, and fixing bugs.

**This is a practical learning project, not a state-of-the-art coding model.** It's a 0.5B-parameter model trained on about 259 examples.
Larger coding models (e.g. Qwen2.5-Coder) will do much better. Always run and test the code it produces.

## How to use

```python
from peft import AutoPeftModelForCausalLM
from transformers import AutoTokenizer

repo = "ali4566544/qwen2.5-0.5b-instruct-python-lora"
model = AutoPeftModelForCausalLM.from_pretrained(repo, torch_dtype="auto", device_map="auto")
tokenizer = AutoTokenizer.from_pretrained(repo)

messages = [
    {"role": "system", "content": 'You are a helpful Python coding assistant. Write clear, correct, idiomatic Python, explain the code concisely, and point out bugs and how to fix them when asked.'},
    {"role": "user", "content": "Write a function that removes duplicates from a list while keeping order."},
]
inputs = tokenizer.apply_chat_template(messages, add_generation_prompt=True, return_tensors="pt").to(model.device)
out = model.generate(inputs, max_new_tokens=512, do_sample=False)
print(tokenizer.decode(out[0, inputs.shape[1]:], skip_special_tokens=True))
```

The adapter was trained **with the system prompt above**. Use it for coding questions. Without it, the model behaves closer to the base model.
To merge the weights into a standalone model: `model.merge_and_unload().save_pretrained(...)`.

## Training data

* Source: `bernabepuente/python-instruction-dataset`, 462 rows. Fields used: `instruction`, optional `input` (non-empty in only 10 cleaned rows), and `output`.
  **The `thought` field was not used.**
* Cleaning (335 rows kept): dropped 6 empty/very short outputs,
  22 truncated outputs (unbalanced code fences), 61 outputs
  cut off after a trailing `:`, 25 outputs with raw code that couldn't be fenced safely,
  13 outputs whose `python` code blocks fail `ast.parse`, and
  0 exact duplicates. In 68 outputs, raw code that
  parses as Python was wrapped in ```` ```python ```` fences.
* Split (seed 42): instructions were clustered by word overlap (Jaccard ≥ 0.6) so that paraphrased questions never straddle splits, giving
  train 259 / validation 42 / test 34 rows. The exact `row_id`s are in `split.json`.
* **Topic coverage is narrow** (keyword counts on cleaned instructions):

| Topic | Examples |
|---|---|
| exceptions / errors | 88 |
| web (FastAPI/Flask/Django) | 77 |
| pandas / DataFrames | 64 |
| logging / monitoring | 55 |
| Docker / deployment | 51 |
| async / asyncio | 2 |
| debugging / fixing bugs | 2 |
| decorators | 1 |
| testing | 1 |
| type hints | 0 |

Async, decorators, type hints and debugging are barely represented, so the fine-tune can't have learned much about them from this data.
The reference answers were **not** verified for correctness beyond the Python syntax check. The dataset declares **no license**.
Check its terms before reusing this adapter commercially.

## Training procedure

LoRA (r=16, alpha=32, dropout=0.05) on all attention and MLP projections. Loss only on assistant tokens.
lr 0.0001 (cosine, warmup 0.05), effective batch
16, up to 3 epochs with early stopping on validation loss
(best checkpoint kept: eval loss 1.3612 at step 40 of 51).
Max length 1536 tokens, fp16 mixed precision on Tesla T4, 1.6 min.
Software: torch 2.11.0+cu130, transformers 4.57.1, peft 0.17.1, datasets 4.8.5.

![loss curve](loss_curve.png)

## Evaluation

Base = the same model with the adapter disabled. Greedy decoding, same prompts. Full details are in `eval_results.json`.

| Metric | Base | Fine-tuned |
|---|---|---|
| Test loss on reference answers ↓ | 1.490 | 1.241 |
| Test perplexity ↓ | 4.43 | 3.46 |
| ROUGE-L F1 vs reference ↑ | 0.134 | 0.346 |
| Answers containing a fenced code block | 76.5% | 38.2% |
| Generated `python` blocks that parse ↑ | 100.0% of 43 blocks | 100.0% of 12 blocks |
| Answers finished within token limit ↑ | 29.4% | 100.0% |
| Mean answer length (chars) | 2023 | 299 |
| **Executable benchmark (pass@1, greedy)** ↑ | 11/16 | 7/16 |
|   type hints | 2/2 | 2/2 |
|   decorators | 2/2 | 0/2 |
|   async | 0/2 | 0/2 |
|   pandas | 2/3 | 2/3 |
|   exceptions | 2/2 | 2/2 |
|   debugging | 1/3 | 0/3 |
|   basics | 2/2 | 1/2 |
| General probes, auto-checks ↑ | 4/4 | 3/4 |

* **Test loss / ROUGE-L** use the 34 held-out dataset questions. They measure how closely the model matches the dataset's answers and style, *not* whether the code is correct.
  The test questions come from the same narrow topics as training, so they're in-distribution.
* **Executable benchmark**: 16 small hand-written tasks with unit tests (type hints, decorators, async, pandas, exceptions, debugging, basics).
  Each task is worth 6.2 points, so a difference of 1–2 tasks is within noise. This is *not* HumanEval/MBPP.
  For a standard number, run e.g. [EvalPlus](https://github.com/evalplus/evalplus) on both models.
* **General probes**: 6 everyday questions without the coding system prompt, used to spot obvious damage to general chat ability. That's a smoke test, not a benchmark.

## Limitations

* **Tiny, narrow dataset.** About 259 training examples, concentrated on Docker, pandas, exceptions, logging and web frameworks. Expect little or no improvement on other topics.
* **Small model.** 0.5B parameters. It makes logic errors, invents APIs and package options, and can write code that looks right but isn't. Review and test everything.
* **Unverified references.** The training answers come from a public dataset that wasn't checked for correctness or security. The model may reproduce its mistakes,
  e.g. outdated Dockerfile practices.
* **Limited evaluation.** The small test set, a 16-task hand-written benchmark and 6 general probes can't capture regressions on other skills or languages.
* English only. No additional safety tuning beyond what the base model has.

## Reproducibility

The notebook that produced this adapter fixes seed 42, records library versions, and saves the data split (`split.json`) and every metric (`eval_results.json`).