Text Generation
PEFT
Safetensors
English
pyspark
apache-spark
qwen2.5-coder
qlora
lora
code-generation
data-engineering
fine-tuned
Instructions to use AaronTekle/PySparkQwen with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- PEFT
How to use AaronTekle/PySparkQwen with PEFT:
from peft import PeftModel from transformers import AutoModelForCausalLM base_model = AutoModelForCausalLM.from_pretrained("Qwen/Qwen2.5-Coder-1.5B-Instruct") model = PeftModel.from_pretrained(base_model, "AaronTekle/PySparkQwen") - Notebooks
- Google Colab
- Kaggle
|
Download README.md from AaronTekle/PySparkQwen: direct link, hf CLI and curl.
- Browser
- Download file 7 kB
-
https://huggingface.co/AaronTekle/PySparkQwen/resolve/main/README.md
- Command line
-
hf download hf://AaronTekle/PySparkQwen/README.md
-
curl -L -o README.md https://huggingface.co/AaronTekle/PySparkQwen/resolve/main/README.md
7 kB
| language: | |
| - en | |
| base_model: | |
| - Qwen/Qwen2.5-Coder-1.5B-Instruct | |
| pipeline_tag: text-generation | |
| library_name: peft | |
| tags: | |
| - pyspark | |
| - apache-spark | |
| - qwen2.5-coder | |
| - qlora | |
| - lora | |
| - code-generation | |
| - data-engineering | |
| - fine-tuned | |
| # PySparkQwen: [**Qwen/Qwen2.5-Coder-1.5B-Instruct**](https://huggingface.co/Qwen/Qwen2.5-Coder-1.5B-Instruct) Fine-Tuning with QLoRA | |
| QLoRA fine-tuning of [**Qwen/Qwen2.5-Coder-1.5B-Instruct**](https://huggingface.co/Qwen/Qwen2.5-Coder-1.5B-Instruct) for PySpark code generation. | |
| **PySparkQwen** is a PySpark coding model, that is used for data engineering tasks, including DataFrame transformations, joins, Spark SQL, structured streaming, Delta Lake, performance optimization, MLlib, and distributed processing. | |
| **Given:** | |
| - PySpark task (prompt) | |
| - DataFrame or table schemas, column names (and/or relevant runtime constraints) | |
| **Training note:** The base model was loaded with 4-bit NF4 quantization and adapted locally using parameter-efficient LoRA weights. This repository uses a **PEFT adapter**, not a standalone copy of the base model. | |
| ## Example | |
| **DataFrame context:** | |
| ```text | |
| DataFrame: orders | |
| Columns: customer_id (string), order_id (string), amount (double) | |
| ``` | |
| **Prompt:** | |
| ```text | |
| Calculate total revenue per customer and return the five customers with the highest revenue. | |
| ``` | |
| **Expected Model Output:** | |
| ```python | |
| from pyspark.sql import functions as F | |
| result = ( | |
| orders.groupBy("customer_id") | |
| .agg(F.sum("amount").alias("total_revenue")) | |
| .orderBy(F.col("total_revenue").desc()) | |
| .limit(5) | |
| ) | |
| ``` | |
| ## Dataset: Synthetic PySpark Code Tasks | |
| | Dataset statistic | Count | | |
| |---|---:| | |
| | Total examples | 900 | | |
| | Training examples | 600 | | |
| | Validation examples | 150 | | |
| | Held-out test examples | 150 | | |
| | Task categories | 15 | | |
| | Algorithm families | 180 | | |
| Dataset is **balanced across 15 task categories**, with **60 examples per category** (40 training, 10 validation, and 10 test). It covers: | |
| 1. DataFrame creation and transformations | |
| 2. Data ingestion and file formats | |
| 3. Data cleaning and preprocessing | |
| 4. Filtering, sorting, and aggregation | |
| 5. Joins and window functions | |
| 6. Spark SQL | |
| 7. UDFs and Pandas UDFs | |
| 8. Structured Streaming | |
| 9. Performance optimization and caching | |
| 10. Partitioning and repartitioning | |
| 11. Machine learning with MLlib | |
| 12. Nested and complex data types | |
| 13. Error handling and debugging | |
| 14. Delta Lake operations | |
| 15. Distributed computing and cluster execution | |
| Note: All 900 reference snippets passed Python `ast.parse` syntax checks. | |
| ## Model Evaluation | |
| The original Qwen2.5-Coder model and the fine-tuned adapter were evaluated on the same **150 held-out synthetic tasks**, using the same prompt construction and greedy decoding (`do_sample=False`) with an output limit of 384 new tokens. | |
| **Greedy decoding** selects the highest-probability next token at each step, making this comparison deterministic under the same runtime conditions. It does not search for the globally best full program. | |
| ### Held-out Test Set (150 Examples) | |
| | Metric | Base Qwen2.5-Coder-1.5B-Instruct | Fine-tuned PySparkQwen-Coder | | |
| |---|---:|---:| | |
| | Evaluated examples | 150 | 150 | | |
| | Python AST-valid outputs | 90.00% (135/150) | **98.67% (148/150)** | | |
| | Mean character-sequence similarity to reference | 0.1168 | **0.6621** | | |
| ## Model Limitations | |
| - **Missing context:** In rare cases (without context): The model may invent columns, tables, file paths, or dependencies if the task or schema is incomplete. | |
| - **Review before use:** Generated code should be reviewed and tested in a controlled Spark environment before running against production data. | |
| ## Example Import (PEFT) | |
| ```python | |
| import torch | |
| from peft import PeftModel | |
| from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig | |
| # Model repositories | |
| base_model = "Qwen/Qwen2.5-Coder-1.5B-Instruct" | |
| adapter_model = "AaronTekle/PySparkQwen" | |
| # Load the original Qwen2.5-Coder tokenizer | |
| tokenizer = AutoTokenizer.from_pretrained(base_model) | |
| if tokenizer.pad_token is None: | |
| tokenizer.pad_token = tokenizer.eos_token | |
| # Configure 4-bit quantization for lower GPU memory usage | |
| compute_dtype = ( | |
| torch.bfloat16 | |
| if torch.cuda.is_bf16_supported() | |
| else torch.float16 | |
| ) | |
| quantization = BitsAndBytesConfig( | |
| load_in_4bit=True, | |
| bnb_4bit_quant_type="nf4", | |
| bnb_4bit_use_double_quant=True, | |
| bnb_4bit_compute_dtype=compute_dtype, | |
| ) | |
| # Load the base Qwen2.5-Coder model | |
| base_model = AutoModelForCausalLM.from_pretrained( | |
| base_model, | |
| quantization_config=quantization, | |
| device_map="auto", | |
| ) | |
| # Load the fine-tuned PySparkQwen LoRA adapter | |
| model = PeftModel.from_pretrained( | |
| base_model, | |
| adapter_model, | |
| ) | |
| model.eval() | |
| # Example PySpark coding task | |
| context = ( | |
| "DataFrame orders: customer_id string, " | |
| "order_id string, amount double" | |
| ) | |
| question = ( | |
| "Calculate total revenue per customer " | |
| "and show the top five customers." | |
| ) | |
| # Define coding assistant behavior | |
| system_prompt = ( | |
| "You are a PySpark data engineering coding assistant. " | |
| "Return only executable Python/PySpark code with necessary imports. " | |
| "Prefer Spark DataFrame APIs over pandas and RDDs. " | |
| "Use the provided schemas and names without inventing tables or columns. " | |
| ) | |
| messages = [ | |
| {"role": "system", "content": system_prompt}, | |
| { | |
| "role": "user", | |
| "content": f"{question}\n\nContext / schema:\n{context}", | |
| }, | |
| ] | |
| # Apply Qwen2.5-Coder's native instruct chat template | |
| prompt = tokenizer.apply_chat_template( | |
| messages, | |
| tokenize=False, | |
| add_generation_prompt=True, | |
| ) | |
| # Tokenize the prompt and move tensors to the model's device | |
| inputs = tokenizer( | |
| prompt, | |
| return_tensors="pt", | |
| ).to(model.get_input_embeddings().weight.device) | |
| # Generate PySpark code using greedy decoding | |
| with torch.inference_mode(): | |
| output_ids = model.generate( | |
| **inputs, | |
| max_new_tokens=384, | |
| do_sample=False, | |
| pad_token_id=tokenizer.pad_token_id, | |
| eos_token_id=tokenizer.convert_tokens_to_ids("<|im_end|>"), | |
| ) | |
| # Decode only newly generated tokens | |
| new_tokens = output_ids[0][inputs["input_ids"].shape[-1]:] | |
| generated_code = tokenizer.decode( | |
| new_tokens, | |
| skip_special_tokens=True, | |
| ).strip() | |
| print("Generated PySpark Code:") | |
| print(generated_code) | |
| ``` | |
| ## References | |
| - [Qwen Team. Qwen2.5-Coder-1.5B-Instruct Model Card](https://huggingface.co/Qwen/Qwen2.5-Coder-1.5B-Instruct) | |
| - [Hui et al. Qwen2.5-Coder Technical Report (2024)](https://arxiv.org/abs/2409.12186) | |
| - [Hu et al. LoRA: Low-Rank Adaptation of Large Language Models (2021)](https://arxiv.org/abs/2106.09685) | |
| - [Dettmers et al. QLoRA: Efficient Finetuning of Quantized LLMs (2023)](https://arxiv.org/abs/2305.14314) | |
| - [Hugging Face. TRL SFTTrainer](https://huggingface.co/docs/trl/sft_trainer) | |
| - [Hugging Face. PEFT LoRA](https://huggingface.co/docs/peft/en/package_reference/lora) | |
| - [Hugging Face. PEFT Adapters in Transformers](https://huggingface.co/docs/transformers/peft) |