Text Generation
PEFT
Safetensors
English
pyspark
apache-spark
qwen2.5-coder
qlora
lora
code-generation
data-engineering
fine-tuned
Instructions to use AaronTekle/PySparkQwen with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- PEFT
How to use AaronTekle/PySparkQwen with PEFT:
from peft import PeftModel from transformers import AutoModelForCausalLM base_model = AutoModelForCausalLM.from_pretrained("Qwen/Qwen2.5-Coder-1.5B-Instruct") model = PeftModel.from_pretrained(base_model, "AaronTekle/PySparkQwen") - Notebooks
- Google Colab
- Kaggle
Upload README.md
Browse files
README.md
ADDED
|
@@ -0,0 +1,219 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
# SparkQwen (Spark-QLoRA, [Qwen/Qwen3-1.7B](https://huggingface.co/Qwen/Qwen3-1.7B) )
|
| 2 |
+
|
| 3 |
+
QLoRA fine-tuning of [**Qwen/Qwen3-1.7B**](https://huggingface.co/Qwen/Qwen3-1.7B) for generating Spark/PySpark code.
|
| 4 |
+
|
| 5 |
+
SparkQwen is a lightweight coding language model for day-to-day data engineering tasks, including DataFrame manipulations, joins, Spark SQL, structured streaming, Delta Lake operations, optimization techniques, MLlib, and more.
|
| 6 |
+
|
| 7 |
+
given:
|
| 8 |
+
|
| 9 |
+
- PySpark prompts/task
|
| 10 |
+
- relevant DataFrame schemas, column names, (and/or runtime context, if necessary)
|
| 11 |
+
|
| 12 |
+
the model then generates the relevant Spark/PySpark code
|
| 13 |
+
|
| 14 |
+
**Training Note:** Trained locally using 4-bit quantization and parameter-efficient LoRA adapters
|
| 15 |
+
|
| 16 |
+
|
| 17 |
+
## Example (Dataset Context)
|
| 18 |
+
|
| 19 |
+
**DataFrame context:**
|
| 20 |
+
|
| 21 |
+
```text
|
| 22 |
+
DataFrame: orders
|
| 23 |
+
Columns: customer_id (string), order_id (string), amount (double)
|
| 24 |
+
```
|
| 25 |
+
|
| 26 |
+
**example prompt:**
|
| 27 |
+
|
| 28 |
+
Calculate total revenue per customer and return the five customers with the highest revenue.
|
| 29 |
+
|
| 30 |
+
**example output :**
|
| 31 |
+
|
| 32 |
+
```python
|
| 33 |
+
from pyspark.sql import functions as F
|
| 34 |
+
|
| 35 |
+
result = (
|
| 36 |
+
orders.groupBy("customer_id")
|
| 37 |
+
.agg(F.sum("amount").alias("total_revenue"))
|
| 38 |
+
.orderBy(F.col("total_revenue").desc())
|
| 39 |
+
.limit(5)
|
| 40 |
+
)
|
| 41 |
+
```
|
| 42 |
+
|
| 43 |
+
## Why QLoRA
|
| 44 |
+
|
| 45 |
+
QLoRA combines low-rank adaptation (LoRA) with a quantized, frozen pretrained model. Only the added adapter parameters are updated during training, reducing memory requirements compared with full fine-tuning.
|
| 46 |
+
|
| 47 |
+
base model was loaded in 4-bit NF4 with double quantization. LoRA adapters were trained on attention and feed-forward projection layers:
|
| 48 |
+
|
| 49 |
+
`q_proj`, `k_proj`, `v_proj`, `o_proj`, `gate_proj`, `up_proj`, `down_proj`.
|
| 50 |
+
|
| 51 |
+
**17,432,576 of 1,738,007,552 parameters were trainable (1.0030%).** The reported total includes the adapter parameters.
|
| 52 |
+
|
| 53 |
+
| LoRA setting | Value |
|
| 54 |
+
|---|---|
|
| 55 |
+
| Task type | Causal language modeling |
|
| 56 |
+
| Base quantization | 4-bit NF4 with double quantization |
|
| 57 |
+
|
| 58 |
+
## Model
|
| 59 |
+
|
| 60 |
+
| Component | Value |
|
| 61 |
+
|---|---|
|
| 62 |
+
| Base model | `Qwen/Qwen3-1.7B` |
|
| 63 |
+
| Fine-tuning | PEFT QLoRA, TRL `SFTTrainer` |
|
| 64 |
+
| Libraries | Transformers, PEFT, TRL, bitsandbytes, PyTorch |
|
| 65 |
+
| Training output | PEFT LoRA adapter |
|
| 66 |
+
|
| 67 |
+
|
| 68 |
+
## Dataset (Local Generated Synthetic Spark Dataset)
|
| 69 |
+
|
| 70 |
+
|
| 71 |
+
| Dataset statistic | Count |
|
| 72 |
+
|---|---:|
|
| 73 |
+
| Total examples | 900 |
|
| 74 |
+
| Training examples | 600 |
|
| 75 |
+
| Validation examples | 150 |
|
| 76 |
+
| Held-out test examples | 150 |
|
| 77 |
+
| PySpark task categories | 15 |
|
| 78 |
+
| Algorithm families | 180 |
|
| 79 |
+
|
| 80 |
+
|
| 81 |
+
**Categories:**
|
| 82 |
+
- DataFrame transformations
|
| 83 |
+
- ingestion and file formats
|
| 84 |
+
- cleaning and preprocessing
|
| 85 |
+
- filtering, sorting, and aggregation
|
| 86 |
+
- joins and windows
|
| 87 |
+
- Spark SQL
|
| 88 |
+
- UDFs and Pandas UDFs
|
| 89 |
+
- structured streaming
|
| 90 |
+
- performance optimization
|
| 91 |
+
- partitioning and repartitioning
|
| 92 |
+
- MLlib
|
| 93 |
+
- nested and complex types
|
| 94 |
+
- debugging; Delta Lake
|
| 95 |
+
- distributed cluster execution
|
| 96 |
+
|
| 97 |
+
note: All 900 reference snippets passed Python AST syntax parsing. (means that your dataset is structurally clean and ready for analysis, and or model training (without needing syntax-level preprocessing)
|
| 98 |
+
## Training
|
| 99 |
+
|
| 100 |
+
|
| 101 |
+
### Loss by Epoch
|
| 102 |
+
|
| 103 |
+
| Epoch | Step | Training loss | Validation loss |
|
| 104 |
+
|---|---:|---:|---:|
|
| 105 |
+
| 1 | 75 | 0.1145 | 0.5008 |
|
| 106 |
+
| 2 | 150 | 0.0213 | 0.5720 |
|
| 107 |
+
| 3 | 225 | 0.0098 | 0.6047 |
|
| 108 |
+
|
| 109 |
+
|
| 110 |
+
## Model Evaluation
|
| 111 |
+
|
| 112 |
+
Both models were evaluated on the same 150 held-out synthetic test tasks under the same loading and generation settings: non-thinking prompts, greedy decoding (b/c it's deterministic, computationally fast, and removes sampling randomness), and a maximum of 384 new tokens
|
| 113 |
+
- [**greedy decoding**](https://machinelearningmastery.com/decoding-strategies-and-output-control/): deterministic text generation strategy in ML where a model picks the single token with the highest predicted probability at each step of an autoregressive sequence (model that predicts the next element in a sequence by conditioning on its own past outputs or lagged values)
|
| 114 |
+
- **note:** greedy decoding called "greedy" because it always grabs the immediate best option (the token with the highest probability) at every single step without caring about the long-term consequence for the rest of the sentence
|
| 115 |
+
- **immediate local gain:** at each step, the language model looks at all possible next words and "greedily" picks the single highest-scoring one
|
| 116 |
+
- **no long-term planning:** greedy decoding algorithms never look ahead or evaluates whether choosing that local favorite will trap the model in a repetitive loop, a dead end, or a awkward phrasing two or three words later
|
| 117 |
+
- **myopic behavior:** just like a greedy person who only thinks about right now rather than the future, the algorithm maximizes step-by-step payoff at the expense of the global (whole-sentence) quality.
|
| 118 |
+
|
| 119 |
+
**Note (AST):** Python AST is a built-in standard library tool that parses Python source code into an Abstract Syntax Tree (AST). The tree represents code as structured Python objects rather than plain text, allowing you to inspect, analyze, and modify programs before they are compiled into bytecode.
|
| 120 |
+
|
| 121 |
+
**Note:**
|
| 122 |
+
- **Mean Character-Sequence:** AVG similarity score computed across a set of textual information when each is compared character-by-character against a ground-truth reference sequence
|
| 123 |
+
|
| 124 |
+
| Metric | Base Qwen3-1.7B | Qwen3 + PySpark adapter |
|
| 125 |
+
|---|---:|---:|
|
| 126 |
+
| Syntax-valid outputs (Python AST (Abstract Syntax Tree)) | 58.67% (88/150) | **95.33%** (143/150) |
|
| 127 |
+
| Mean character-sequence similarity to reference | 0.1009 | 0.6354 |
|
| 128 |
+
|
| 129 |
+
|
| 130 |
+
|
| 131 |
+
## Model Limits
|
| 132 |
+
- **Note:** Rare Cases (if context is not provided): model may invent (hallucinate) columns, tables, file paths, or dependencies when insufficient context is provided
|
| 133 |
+
- Training used 600 synthetic examples from 120 algorithm families; the model has not been validated on production tasks
|
| 134 |
+
- Held out test tasks come from the same synthetic generation method as the training data, even though algorithm families are separated by split
|
| 135 |
+
- Delta Lake operations, streaming jobs, cloud connectors, and distributed execution require compatible runtime configuration and should be validated before use
|
| 136 |
+
|
| 137 |
+
## Example Import (PEFT)
|
| 138 |
+
|
| 139 |
+
**Note 1:** Example prompt requires a compatible CUDA GPU, PyTorch, Transformers, PEFT, Accelerate, and bitsandbytes. For larger GPUs, loading the unquantized base model is also possible
|
| 140 |
+
|
| 141 |
+
**Note 2:** this model uses an adapter, not the full Qwen3 base model
|
| 142 |
+
|
| 143 |
+
```python
|
| 144 |
+
import torch
|
| 145 |
+
from peft import PeftModel
|
| 146 |
+
from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig
|
| 147 |
+
|
| 148 |
+
base_id = "Qwen/Qwen3-1.7B"
|
| 149 |
+
adapter_id = "AaronTekle/SparkQwen"
|
| 150 |
+
|
| 151 |
+
tokenizer = AutoTokenizer.from_pretrained(base_id)
|
| 152 |
+
if tokenizer.pad_token is None:
|
| 153 |
+
tokenizer.pad_token = tokenizer.eos_token
|
| 154 |
+
|
| 155 |
+
# Use 4-bit loading for lower GPU memory consumption
|
| 156 |
+
compute_dtype = torch.bfloat16 if torch.cuda.is_bf16_supported() else torch.float16
|
| 157 |
+
quantization = BitsAndBytesConfig(
|
| 158 |
+
load_in_4bit=True,
|
| 159 |
+
bnb_4bit_quant_type="nf4",
|
| 160 |
+
bnb_4bit_use_double_quant=True,
|
| 161 |
+
bnb_4bit_compute_dtype=compute_dtype,
|
| 162 |
+
)
|
| 163 |
+
|
| 164 |
+
base_model = AutoModelForCausalLM.from_pretrained(
|
| 165 |
+
base_id,
|
| 166 |
+
quantization_config=quantization,
|
| 167 |
+
device_map="auto",
|
| 168 |
+
)
|
| 169 |
+
model = PeftModel.from_pretrained(base_model, adapter_id)
|
| 170 |
+
model.eval()
|
| 171 |
+
|
| 172 |
+
context = "DataFrame orders: customer_id string, order_id string, amount double"
|
| 173 |
+
question = "Calculate total revenue per customer and show the top five customers."
|
| 174 |
+
|
| 175 |
+
system_prompt = (
|
| 176 |
+
"You are a Spark/PySpark data engineering coding assistant. "
|
| 177 |
+
"Return only executable Spark/PySpark code with necessary imports. "
|
| 178 |
+
"Prefer Spark DataFrame APIs over pandas and RDDs. "
|
| 179 |
+
"Use the provided schemas and names without inventing tables or columns. "
|
| 180 |
+
"Do not add Markdown fences or prose."
|
| 181 |
+
)
|
| 182 |
+
|
| 183 |
+
messages = [
|
| 184 |
+
{"role": "system", "content": system_prompt},
|
| 185 |
+
{"role": "user", "content": f"{question}\n\nContext / schema:\n{context}"},
|
| 186 |
+
]
|
| 187 |
+
|
| 188 |
+
prompt = tokenizer.apply_chat_template(
|
| 189 |
+
messages,
|
| 190 |
+
tokenize=False,
|
| 191 |
+
add_generation_prompt=True,
|
| 192 |
+
enable_thinking=False,
|
| 193 |
+
)
|
| 194 |
+
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
|
| 195 |
+
|
| 196 |
+
with torch.inference_mode():
|
| 197 |
+
output_ids = model.generate(
|
| 198 |
+
**inputs,
|
| 199 |
+
max_new_tokens=384,
|
| 200 |
+
do_sample=False,
|
| 201 |
+
pad_token_id=tokenizer.pad_token_id,
|
| 202 |
+
eos_token_id=tokenizer.convert_tokens_to_ids("<|im_end|>"),
|
| 203 |
+
)
|
| 204 |
+
|
| 205 |
+
generated_code = tokenizer.decode(
|
| 206 |
+
output_ids[0][inputs["input_ids"].shape[-1]:],
|
| 207 |
+
skip_special_tokens=True,
|
| 208 |
+
).strip()
|
| 209 |
+
print(generated_code)
|
| 210 |
+
```
|
| 211 |
+
|
| 212 |
+
## References
|
| 213 |
+
|
| 214 |
+
- [Qwen Team. Qwen3-1.7B Model Card](https://huggingface.co/Qwen/Qwen3-1.7B)
|
| 215 |
+
- [Yang, A. et al. Qwen3 Technical Report (2025)](https://arxiv.org/abs/2505.09388)
|
| 216 |
+
- [Hu, E. J. et al. LoRA: Low-Rank Adaptation of Large Language Models (2021)](https://arxiv.org/abs/2106.09685)
|
| 217 |
+
- [Dettmers, T. et al. QLoRA: Efficient Finetuning of Quantized LLMs (2023)](https://arxiv.org/abs/2305.14314)
|
| 218 |
+
- [Hugging Face. Parameter-Efficient Fine-Tuning](https://huggingface.co/docs/transformers/peft)
|
| 219 |
+
- [Hugging Face. PEFT LoRA](https://huggingface.co/docs/peft/en/package_reference/lora)
|