File size: 6,998 Bytes
5c9fb91
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
f8b34ba
86bd38d
f8b34ba
86bd38d
f8b34ba
86bd38d
f8b34ba
86bd38d
f8b34ba
f4eb51e
86bd38d
 
f8b34ba
86bd38d
f8b34ba
86bd38d
 
 
 
 
 
 
 
f8b34ba
 
86bd38d
f8b34ba
 
86bd38d
 
 
 
 
 
 
 
 
 
 
 
f8b34ba
86bd38d
 
 
 
 
 
 
 
f8b34ba
86bd38d
 
f8b34ba
86bd38d
f8b34ba
 
 
 
 
 
 
 
 
 
 
 
 
 
 
86bd38d
 
f4eb51e
86bd38d
 
 
 
f8b34ba
86bd38d
f8b34ba
86bd38d
f8b34ba
86bd38d
f8b34ba
86bd38d
f8b34ba
 
 
86bd38d
 
f8b34ba
86bd38d
f8b34ba
 
86bd38d
 
 
 
 
 
 
 
9cff495
cee9c0f
 
86bd38d
9cff495
cee9c0f
9cff495
86bd38d
 
 
9cff495
c8c87e0
9cff495
 
 
c8c87e0
9cff495
86bd38d
 
 
 
 
 
 
9cff495
86bd38d
cee9c0f
86bd38d
 
 
9cff495
 
 
 
cee9c0f
9cff495
86bd38d
 
9cff495
 
 
 
 
 
 
 
 
 
86bd38d
9cff495
86bd38d
caecdc8
f8b34ba
86bd38d
 
 
9cff495
86bd38d
 
9cff495
 
 
 
86bd38d
 
9cff495
86bd38d
 
 
 
f8b34ba
86bd38d
9cff495
 
 
 
 
 
 
86bd38d
 
 
 
 
 
 
 
 
9cff495
f8b34ba
9cff495
 
 
 
 
 
 
86bd38d
 
 
 
 
f8b34ba
 
 
 
 
 
5c9fb91
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
---
language:
- en
base_model:
- Qwen/Qwen2.5-Coder-1.5B-Instruct
pipeline_tag: text-generation
library_name: peft
tags:
- pyspark
- apache-spark
- qwen2.5-coder
- qlora
- lora
- code-generation
- data-engineering
- fine-tuned
---
# PySparkQwen: [**Qwen/Qwen2.5-Coder-1.5B-Instruct**](https://huggingface.co/Qwen/Qwen2.5-Coder-1.5B-Instruct) Fine-Tuning with QLoRA

QLoRA fine-tuning of [**Qwen/Qwen2.5-Coder-1.5B-Instruct**](https://huggingface.co/Qwen/Qwen2.5-Coder-1.5B-Instruct) for PySpark code generation.

**PySparkQwen** is a PySpark coding model, that is used for data engineering tasks, including DataFrame transformations, joins, Spark SQL, structured streaming, Delta Lake, performance optimization, MLlib, and distributed processing.

**Given:**

- PySpark task (prompt)
- DataFrame or table schemas, column names (and/or relevant runtime constraints)


**Training note:** The base model was loaded with 4-bit NF4 quantization and adapted locally using parameter-efficient LoRA weights. This repository uses a **PEFT adapter**, not a standalone copy of the base model.

## Example

**DataFrame context:**

```text
DataFrame: orders
Columns: customer_id (string), order_id (string), amount (double)
```

**Prompt:**
```text
Calculate total revenue per customer and return the five customers with the highest revenue.
```
**Expected Model Output:**

```python
from pyspark.sql import functions as F

result = (
    orders.groupBy("customer_id")
    .agg(F.sum("amount").alias("total_revenue"))
    .orderBy(F.col("total_revenue").desc())
    .limit(5)
)
```

## Dataset: Synthetic PySpark Code Tasks


| Dataset statistic | Count |
|---|---:|
| Total examples | 900 |
| Training examples | 600 |
| Validation examples | 150 |
| Held-out test examples | 150 |
| Task categories | 15 |
| Algorithm families | 180 |

Dataset is **balanced across 15 task categories**, with **60 examples per category** (40 training, 10 validation, and 10 test). It covers:

1. DataFrame creation and transformations
2. Data ingestion and file formats
3. Data cleaning and preprocessing
4. Filtering, sorting, and aggregation
5. Joins and window functions
6. Spark SQL
7. UDFs and Pandas UDFs
8. Structured Streaming
9. Performance optimization and caching
10. Partitioning and repartitioning
11. Machine learning with MLlib
12. Nested and complex data types
13. Error handling and debugging
14. Delta Lake operations
15. Distributed computing and cluster execution


Note: All 900 reference snippets passed Python `ast.parse` syntax checks.


## Model Evaluation

The original Qwen2.5-Coder model and the fine-tuned adapter were evaluated on the same **150 held-out synthetic tasks**, using the same prompt construction and greedy decoding (`do_sample=False`) with an output limit of 384 new tokens.

**Greedy decoding** selects the highest-probability next token at each step, making this comparison deterministic under the same runtime conditions. It does not search for the globally best full program.

### Held-out Test Set (150 Examples)

| Metric | Base Qwen2.5-Coder-1.5B-Instruct | Fine-tuned PySparkQwen-Coder |
|---|---:|---:|
| Evaluated examples | 150 | 150 |
| Python AST-valid outputs | 90.00% (135/150) | **98.67% (148/150)** |
| Mean character-sequence similarity to reference | 0.1168 | **0.6621** |


## Model Limitations

- **Missing context:** In rare cases (without context): The model may invent columns, tables, file paths, or dependencies if the task or schema is incomplete.
- **Review before use:** Generated code should be reviewed and tested in a controlled Spark environment before running against production data.

## Example Import (PEFT)

```python
import torch
from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig

# Model repositories
base_model = "Qwen/Qwen2.5-Coder-1.5B-Instruct"
adapter_model = "AaronTekle/PySparkQwen"

# Load the original Qwen2.5-Coder tokenizer
tokenizer = AutoTokenizer.from_pretrained(base_model)

if tokenizer.pad_token is None:
    tokenizer.pad_token = tokenizer.eos_token

# Configure 4-bit quantization for lower GPU memory usage
compute_dtype = (
    torch.bfloat16
    if torch.cuda.is_bf16_supported()
    else torch.float16
)

quantization = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_use_double_quant=True,
    bnb_4bit_compute_dtype=compute_dtype,
)

# Load the base Qwen2.5-Coder model
base_model = AutoModelForCausalLM.from_pretrained(
    base_model,
    quantization_config=quantization,
    device_map="auto",
)

# Load the fine-tuned PySparkQwen LoRA adapter
model = PeftModel.from_pretrained(
    base_model,
    adapter_model,
)
model.eval()

# Example PySpark coding task
context = (
    "DataFrame orders: customer_id string, "
    "order_id string, amount double"
)

question = (
    "Calculate total revenue per customer "
    "and show the top five customers."
)

# Define coding assistant behavior
system_prompt = (
    "You are a PySpark data engineering coding assistant. "
    "Return only executable Python/PySpark code with necessary imports. "
    "Prefer Spark DataFrame APIs over pandas and RDDs. "
    "Use the provided schemas and names without inventing tables or columns. "
)

messages = [
    {"role": "system", "content": system_prompt},
    {
        "role": "user",
        "content": f"{question}\n\nContext / schema:\n{context}",
    },
]

# Apply Qwen2.5-Coder's native instruct chat template
prompt = tokenizer.apply_chat_template(
    messages,
    tokenize=False,
    add_generation_prompt=True,
)

# Tokenize the prompt and move tensors to the model's device
inputs = tokenizer(
    prompt,
    return_tensors="pt",
).to(model.get_input_embeddings().weight.device)

# Generate PySpark code using greedy decoding
with torch.inference_mode():
    output_ids = model.generate(
        **inputs,
        max_new_tokens=384,
        do_sample=False,
        pad_token_id=tokenizer.pad_token_id,
        eos_token_id=tokenizer.convert_tokens_to_ids("<|im_end|>"),
    )

# Decode only newly generated tokens
new_tokens = output_ids[0][inputs["input_ids"].shape[-1]:]

generated_code = tokenizer.decode(
    new_tokens,
    skip_special_tokens=True,
).strip()

print("Generated PySpark Code:")
print(generated_code)
```

## References

- [Qwen Team. Qwen2.5-Coder-1.5B-Instruct Model Card](https://huggingface.co/Qwen/Qwen2.5-Coder-1.5B-Instruct)
- [Hui et al. Qwen2.5-Coder Technical Report (2024)](https://arxiv.org/abs/2409.12186)
- [Hu et al. LoRA: Low-Rank Adaptation of Large Language Models (2021)](https://arxiv.org/abs/2106.09685)
- [Dettmers et al. QLoRA: Efficient Finetuning of Quantized LLMs (2023)](https://arxiv.org/abs/2305.14314)
- [Hugging Face. TRL SFTTrainer](https://huggingface.co/docs/trl/sft_trainer)
- [Hugging Face. PEFT LoRA](https://huggingface.co/docs/peft/en/package_reference/lora)
- [Hugging Face. PEFT Adapters in Transformers](https://huggingface.co/docs/transformers/peft)