Text Generation
PEFT
Safetensors
English
pyspark
apache-spark
qwen2.5-coder
qlora
lora
code-generation
data-engineering
fine-tuned
Instructions to use AaronTekle/PySparkQwen with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- PEFT
How to use AaronTekle/PySparkQwen with PEFT:
from peft import PeftModel from transformers import AutoModelForCausalLM base_model = AutoModelForCausalLM.from_pretrained("Qwen/Qwen2.5-Coder-1.5B-Instruct") model = PeftModel.from_pretrained(base_model, "AaronTekle/PySparkQwen") - Notebooks
- Google Colab
- Kaggle
File size: 6,998 Bytes
5c9fb91 f8b34ba 86bd38d f8b34ba 86bd38d f8b34ba 86bd38d f8b34ba 86bd38d f8b34ba f4eb51e 86bd38d f8b34ba 86bd38d f8b34ba 86bd38d f8b34ba 86bd38d f8b34ba 86bd38d f8b34ba 86bd38d f8b34ba 86bd38d f8b34ba 86bd38d f8b34ba 86bd38d f4eb51e 86bd38d f8b34ba 86bd38d f8b34ba 86bd38d f8b34ba 86bd38d f8b34ba 86bd38d f8b34ba 86bd38d f8b34ba 86bd38d f8b34ba 86bd38d 9cff495 cee9c0f 86bd38d 9cff495 cee9c0f 9cff495 86bd38d 9cff495 c8c87e0 9cff495 c8c87e0 9cff495 86bd38d 9cff495 86bd38d cee9c0f 86bd38d 9cff495 cee9c0f 9cff495 86bd38d 9cff495 86bd38d 9cff495 86bd38d caecdc8 f8b34ba 86bd38d 9cff495 86bd38d 9cff495 86bd38d 9cff495 86bd38d f8b34ba 86bd38d 9cff495 86bd38d 9cff495 f8b34ba 9cff495 86bd38d f8b34ba 5c9fb91 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 | ---
language:
- en
base_model:
- Qwen/Qwen2.5-Coder-1.5B-Instruct
pipeline_tag: text-generation
library_name: peft
tags:
- pyspark
- apache-spark
- qwen2.5-coder
- qlora
- lora
- code-generation
- data-engineering
- fine-tuned
---
# PySparkQwen: [**Qwen/Qwen2.5-Coder-1.5B-Instruct**](https://huggingface.co/Qwen/Qwen2.5-Coder-1.5B-Instruct) Fine-Tuning with QLoRA
QLoRA fine-tuning of [**Qwen/Qwen2.5-Coder-1.5B-Instruct**](https://huggingface.co/Qwen/Qwen2.5-Coder-1.5B-Instruct) for PySpark code generation.
**PySparkQwen** is a PySpark coding model, that is used for data engineering tasks, including DataFrame transformations, joins, Spark SQL, structured streaming, Delta Lake, performance optimization, MLlib, and distributed processing.
**Given:**
- PySpark task (prompt)
- DataFrame or table schemas, column names (and/or relevant runtime constraints)
**Training note:** The base model was loaded with 4-bit NF4 quantization and adapted locally using parameter-efficient LoRA weights. This repository uses a **PEFT adapter**, not a standalone copy of the base model.
## Example
**DataFrame context:**
```text
DataFrame: orders
Columns: customer_id (string), order_id (string), amount (double)
```
**Prompt:**
```text
Calculate total revenue per customer and return the five customers with the highest revenue.
```
**Expected Model Output:**
```python
from pyspark.sql import functions as F
result = (
orders.groupBy("customer_id")
.agg(F.sum("amount").alias("total_revenue"))
.orderBy(F.col("total_revenue").desc())
.limit(5)
)
```
## Dataset: Synthetic PySpark Code Tasks
| Dataset statistic | Count |
|---|---:|
| Total examples | 900 |
| Training examples | 600 |
| Validation examples | 150 |
| Held-out test examples | 150 |
| Task categories | 15 |
| Algorithm families | 180 |
Dataset is **balanced across 15 task categories**, with **60 examples per category** (40 training, 10 validation, and 10 test). It covers:
1. DataFrame creation and transformations
2. Data ingestion and file formats
3. Data cleaning and preprocessing
4. Filtering, sorting, and aggregation
5. Joins and window functions
6. Spark SQL
7. UDFs and Pandas UDFs
8. Structured Streaming
9. Performance optimization and caching
10. Partitioning and repartitioning
11. Machine learning with MLlib
12. Nested and complex data types
13. Error handling and debugging
14. Delta Lake operations
15. Distributed computing and cluster execution
Note: All 900 reference snippets passed Python `ast.parse` syntax checks.
## Model Evaluation
The original Qwen2.5-Coder model and the fine-tuned adapter were evaluated on the same **150 held-out synthetic tasks**, using the same prompt construction and greedy decoding (`do_sample=False`) with an output limit of 384 new tokens.
**Greedy decoding** selects the highest-probability next token at each step, making this comparison deterministic under the same runtime conditions. It does not search for the globally best full program.
### Held-out Test Set (150 Examples)
| Metric | Base Qwen2.5-Coder-1.5B-Instruct | Fine-tuned PySparkQwen-Coder |
|---|---:|---:|
| Evaluated examples | 150 | 150 |
| Python AST-valid outputs | 90.00% (135/150) | **98.67% (148/150)** |
| Mean character-sequence similarity to reference | 0.1168 | **0.6621** |
## Model Limitations
- **Missing context:** In rare cases (without context): The model may invent columns, tables, file paths, or dependencies if the task or schema is incomplete.
- **Review before use:** Generated code should be reviewed and tested in a controlled Spark environment before running against production data.
## Example Import (PEFT)
```python
import torch
from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig
# Model repositories
base_model = "Qwen/Qwen2.5-Coder-1.5B-Instruct"
adapter_model = "AaronTekle/PySparkQwen"
# Load the original Qwen2.5-Coder tokenizer
tokenizer = AutoTokenizer.from_pretrained(base_model)
if tokenizer.pad_token is None:
tokenizer.pad_token = tokenizer.eos_token
# Configure 4-bit quantization for lower GPU memory usage
compute_dtype = (
torch.bfloat16
if torch.cuda.is_bf16_supported()
else torch.float16
)
quantization = BitsAndBytesConfig(
load_in_4bit=True,
bnb_4bit_quant_type="nf4",
bnb_4bit_use_double_quant=True,
bnb_4bit_compute_dtype=compute_dtype,
)
# Load the base Qwen2.5-Coder model
base_model = AutoModelForCausalLM.from_pretrained(
base_model,
quantization_config=quantization,
device_map="auto",
)
# Load the fine-tuned PySparkQwen LoRA adapter
model = PeftModel.from_pretrained(
base_model,
adapter_model,
)
model.eval()
# Example PySpark coding task
context = (
"DataFrame orders: customer_id string, "
"order_id string, amount double"
)
question = (
"Calculate total revenue per customer "
"and show the top five customers."
)
# Define coding assistant behavior
system_prompt = (
"You are a PySpark data engineering coding assistant. "
"Return only executable Python/PySpark code with necessary imports. "
"Prefer Spark DataFrame APIs over pandas and RDDs. "
"Use the provided schemas and names without inventing tables or columns. "
)
messages = [
{"role": "system", "content": system_prompt},
{
"role": "user",
"content": f"{question}\n\nContext / schema:\n{context}",
},
]
# Apply Qwen2.5-Coder's native instruct chat template
prompt = tokenizer.apply_chat_template(
messages,
tokenize=False,
add_generation_prompt=True,
)
# Tokenize the prompt and move tensors to the model's device
inputs = tokenizer(
prompt,
return_tensors="pt",
).to(model.get_input_embeddings().weight.device)
# Generate PySpark code using greedy decoding
with torch.inference_mode():
output_ids = model.generate(
**inputs,
max_new_tokens=384,
do_sample=False,
pad_token_id=tokenizer.pad_token_id,
eos_token_id=tokenizer.convert_tokens_to_ids("<|im_end|>"),
)
# Decode only newly generated tokens
new_tokens = output_ids[0][inputs["input_ids"].shape[-1]:]
generated_code = tokenizer.decode(
new_tokens,
skip_special_tokens=True,
).strip()
print("Generated PySpark Code:")
print(generated_code)
```
## References
- [Qwen Team. Qwen2.5-Coder-1.5B-Instruct Model Card](https://huggingface.co/Qwen/Qwen2.5-Coder-1.5B-Instruct)
- [Hui et al. Qwen2.5-Coder Technical Report (2024)](https://arxiv.org/abs/2409.12186)
- [Hu et al. LoRA: Low-Rank Adaptation of Large Language Models (2021)](https://arxiv.org/abs/2106.09685)
- [Dettmers et al. QLoRA: Efficient Finetuning of Quantized LLMs (2023)](https://arxiv.org/abs/2305.14314)
- [Hugging Face. TRL SFTTrainer](https://huggingface.co/docs/trl/sft_trainer)
- [Hugging Face. PEFT LoRA](https://huggingface.co/docs/peft/en/package_reference/lora)
- [Hugging Face. PEFT Adapters in Transformers](https://huggingface.co/docs/transformers/peft) |