Text Generation
PEFT
Safetensors
English
pyspark
apache-spark
qwen2.5-coder
qlora
lora
code-generation
data-engineering
fine-tuned
Instructions to use AaronTekle/PySparkQwen with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- PEFT
How to use AaronTekle/PySparkQwen with PEFT:
from peft import PeftModel from transformers import AutoModelForCausalLM base_model = AutoModelForCausalLM.from_pretrained("Qwen/Qwen2.5-Coder-1.5B-Instruct") model = PeftModel.from_pretrained(base_model, "AaronTekle/PySparkQwen") - Notebooks
- Google Colab
- Kaggle
Update README.md
Browse files
README.md
CHANGED
|
@@ -15,11 +15,11 @@ tags:
|
|
| 15 |
- fine-tuned
|
| 16 |
- qwen
|
| 17 |
---
|
| 18 |
-
#
|
| 19 |
|
| 20 |
QLoRA fine-tuning of [**Qwen/Qwen3-1.7B**](https://huggingface.co/Qwen/Qwen3-1.7B) for generating Spark/PySpark code.
|
| 21 |
|
| 22 |
-
|
| 23 |
|
| 24 |
given:
|
| 25 |
|
|
@@ -213,11 +213,10 @@ question = (
|
|
| 213 |
|
| 214 |
# Define coding assistant behavior
|
| 215 |
system_prompt = (
|
| 216 |
-
"You are a
|
| 217 |
-
"Return
|
| 218 |
"Prefer Spark DataFrame APIs over pandas and RDDs. "
|
| 219 |
"Use the provided schemas and names without inventing tables or columns. "
|
| 220 |
-
"Do not add Markdown fences or prose."
|
| 221 |
)
|
| 222 |
|
| 223 |
messages = [
|
|
|
|
| 15 |
- fine-tuned
|
| 16 |
- qwen
|
| 17 |
---
|
| 18 |
+
# PySparkQwen (PySparkQwen-QLoRA, [Qwen/Qwen3-1.7B](https://huggingface.co/Qwen/Qwen3-1.7B) )
|
| 19 |
|
| 20 |
QLoRA fine-tuning of [**Qwen/Qwen3-1.7B**](https://huggingface.co/Qwen/Qwen3-1.7B) for generating Spark/PySpark code.
|
| 21 |
|
| 22 |
+
PySparkQwen is a lightweight coding language model for day-to-day data engineering tasks, including DataFrame manipulations, joins, Spark SQL, structured streaming, Delta Lake operations, optimization techniques, MLlib, and more.
|
| 23 |
|
| 24 |
given:
|
| 25 |
|
|
|
|
| 213 |
|
| 214 |
# Define coding assistant behavior
|
| 215 |
system_prompt = (
|
| 216 |
+
"You are a PySpark data engineering coding assistant. "
|
| 217 |
+
"Return executable PySpark code with necessary imports. "
|
| 218 |
"Prefer Spark DataFrame APIs over pandas and RDDs. "
|
| 219 |
"Use the provided schemas and names without inventing tables or columns. "
|
|
|
|
| 220 |
)
|
| 221 |
|
| 222 |
messages = [
|