PySparkQwen / README.md
AaronTekle's picture
Update README.md
6f6b9c3 verified
|
Raw History Blame Contribute Delete
7 kB
---
language:
- en
base_model:
- Qwen/Qwen2.5-Coder-1.5B-Instruct
pipeline_tag: text-generation
library_name: peft
tags:
- pyspark
- apache-spark
- qwen2.5-coder
- qlora
- lora
- code-generation
- data-engineering
- fine-tuned
---
# PySparkQwen: [**Qwen/Qwen2.5-Coder-1.5B-Instruct**](https://huggingface.co/Qwen/Qwen2.5-Coder-1.5B-Instruct) Fine-Tuning with QLoRA
QLoRA fine-tuning of [**Qwen/Qwen2.5-Coder-1.5B-Instruct**](https://huggingface.co/Qwen/Qwen2.5-Coder-1.5B-Instruct) for PySpark code generation.
**PySparkQwen** is a PySpark coding model, that is used for data engineering tasks, including DataFrame transformations, joins, Spark SQL, structured streaming, Delta Lake, performance optimization, MLlib, and distributed processing.
**Given:**
- PySpark task (prompt)
- DataFrame or table schemas, column names (and/or relevant runtime constraints)
**Training note:** The base model was loaded with 4-bit NF4 quantization and adapted locally using parameter-efficient LoRA weights. This repository uses a **PEFT adapter**, not a standalone copy of the base model.
## Example
**DataFrame context:**
```text
DataFrame: orders
Columns: customer_id (string), order_id (string), amount (double)
```
**Prompt:**
```text
Calculate total revenue per customer and return the five customers with the highest revenue.
```
**Expected Model Output:**
```python
from pyspark.sql import functions as F
result = (
orders.groupBy("customer_id")
.agg(F.sum("amount").alias("total_revenue"))
.orderBy(F.col("total_revenue").desc())
.limit(5)
)
```
## Dataset: Synthetic PySpark Code Tasks
| Dataset statistic | Count |
|---|---:|
| Total examples | 900 |
| Training examples | 600 |
| Validation examples | 150 |
| Held-out test examples | 150 |
| Task categories | 15 |
| Algorithm families | 180 |
Dataset is **balanced across 15 task categories**, with **60 examples per category** (40 training, 10 validation, and 10 test). It covers:
1. DataFrame creation and transformations
2. Data ingestion and file formats
3. Data cleaning and preprocessing
4. Filtering, sorting, and aggregation
5. Joins and window functions
6. Spark SQL
7. UDFs and Pandas UDFs
8. Structured Streaming
9. Performance optimization and caching
10. Partitioning and repartitioning
11. Machine learning with MLlib
12. Nested and complex data types
13. Error handling and debugging
14. Delta Lake operations
15. Distributed computing and cluster execution
Note: All 900 reference snippets passed Python `ast.parse` syntax checks.
## Model Evaluation
The original Qwen2.5-Coder model and the fine-tuned adapter were evaluated on the same **150 held-out synthetic tasks**, using the same prompt construction and greedy decoding (`do_sample=False`) with an output limit of 384 new tokens.
**Greedy decoding** selects the highest-probability next token at each step, making this comparison deterministic under the same runtime conditions. It does not search for the globally best full program.
### Held-out Test Set (150 Examples)
| Metric | Base Qwen2.5-Coder-1.5B-Instruct | Fine-tuned PySparkQwen-Coder |
|---|---:|---:|
| Evaluated examples | 150 | 150 |
| Python AST-valid outputs | 90.00% (135/150) | **98.67% (148/150)** |
| Mean character-sequence similarity to reference | 0.1168 | **0.6621** |
## Model Limitations
- **Missing context:** In rare cases (without context): The model may invent columns, tables, file paths, or dependencies if the task or schema is incomplete.
- **Review before use:** Generated code should be reviewed and tested in a controlled Spark environment before running against production data.
## Example Import (PEFT)
```python
import torch
from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig
# Model repositories
base_model = "Qwen/Qwen2.5-Coder-1.5B-Instruct"
adapter_model = "AaronTekle/PySparkQwen"
# Load the original Qwen2.5-Coder tokenizer
tokenizer = AutoTokenizer.from_pretrained(base_model)
if tokenizer.pad_token is None:
tokenizer.pad_token = tokenizer.eos_token
# Configure 4-bit quantization for lower GPU memory usage
compute_dtype = (
torch.bfloat16
if torch.cuda.is_bf16_supported()
else torch.float16
)
quantization = BitsAndBytesConfig(
load_in_4bit=True,
bnb_4bit_quant_type="nf4",
bnb_4bit_use_double_quant=True,
bnb_4bit_compute_dtype=compute_dtype,
)
# Load the base Qwen2.5-Coder model
base_model = AutoModelForCausalLM.from_pretrained(
base_model,
quantization_config=quantization,
device_map="auto",
)
# Load the fine-tuned PySparkQwen LoRA adapter
model = PeftModel.from_pretrained(
base_model,
adapter_model,
)
model.eval()
# Example PySpark coding task
context = (
"DataFrame orders: customer_id string, "
"order_id string, amount double"
)
question = (
"Calculate total revenue per customer "
"and show the top five customers."
)
# Define coding assistant behavior
system_prompt = (
"You are a PySpark data engineering coding assistant. "
"Return only executable Python/PySpark code with necessary imports. "
"Prefer Spark DataFrame APIs over pandas and RDDs. "
"Use the provided schemas and names without inventing tables or columns. "
)
messages = [
{"role": "system", "content": system_prompt},
{
"role": "user",
"content": f"{question}\n\nContext / schema:\n{context}",
},
]
# Apply Qwen2.5-Coder's native instruct chat template
prompt = tokenizer.apply_chat_template(
messages,
tokenize=False,
add_generation_prompt=True,
)
# Tokenize the prompt and move tensors to the model's device
inputs = tokenizer(
prompt,
return_tensors="pt",
).to(model.get_input_embeddings().weight.device)
# Generate PySpark code using greedy decoding
with torch.inference_mode():
output_ids = model.generate(
**inputs,
max_new_tokens=384,
do_sample=False,
pad_token_id=tokenizer.pad_token_id,
eos_token_id=tokenizer.convert_tokens_to_ids("<|im_end|>"),
)
# Decode only newly generated tokens
new_tokens = output_ids[0][inputs["input_ids"].shape[-1]:]
generated_code = tokenizer.decode(
new_tokens,
skip_special_tokens=True,
).strip()
print("Generated PySpark Code:")
print(generated_code)
```
## References
- [Qwen Team. Qwen2.5-Coder-1.5B-Instruct Model Card](https://huggingface.co/Qwen/Qwen2.5-Coder-1.5B-Instruct)
- [Hui et al. Qwen2.5-Coder Technical Report (2024)](https://arxiv.org/abs/2409.12186)
- [Hu et al. LoRA: Low-Rank Adaptation of Large Language Models (2021)](https://arxiv.org/abs/2106.09685)
- [Dettmers et al. QLoRA: Efficient Finetuning of Quantized LLMs (2023)](https://arxiv.org/abs/2305.14314)
- [Hugging Face. TRL SFTTrainer](https://huggingface.co/docs/trl/sft_trainer)
- [Hugging Face. PEFT LoRA](https://huggingface.co/docs/peft/en/package_reference/lora)
- [Hugging Face. PEFT Adapters in Transformers](https://huggingface.co/docs/transformers/peft)