--- language: - en base_model: - Qwen/Qwen2.5-Coder-1.5B-Instruct pipeline_tag: text-generation library_name: peft tags: - pyspark - apache-spark - qwen2.5-coder - qlora - lora - code-generation - data-engineering - fine-tuned --- # PySparkQwen: [**Qwen/Qwen2.5-Coder-1.5B-Instruct**](https://huggingface.co/Qwen/Qwen2.5-Coder-1.5B-Instruct) Fine-Tuning with QLoRA QLoRA fine-tuning of [**Qwen/Qwen2.5-Coder-1.5B-Instruct**](https://huggingface.co/Qwen/Qwen2.5-Coder-1.5B-Instruct) for PySpark code generation. **PySparkQwen** is a PySpark coding model, that is used for data engineering tasks, including DataFrame transformations, joins, Spark SQL, structured streaming, Delta Lake, performance optimization, MLlib, and distributed processing. **Given:** - PySpark task (prompt) - DataFrame or table schemas, column names (and/or relevant runtime constraints) **Training note:** The base model was loaded with 4-bit NF4 quantization and adapted locally using parameter-efficient LoRA weights. This repository uses a **PEFT adapter**, not a standalone copy of the base model. ## Example **DataFrame context:** ```text DataFrame: orders Columns: customer_id (string), order_id (string), amount (double) ``` **Prompt:** ```text Calculate total revenue per customer and return the five customers with the highest revenue. ``` **Expected Model Output:** ```python from pyspark.sql import functions as F result = ( orders.groupBy("customer_id") .agg(F.sum("amount").alias("total_revenue")) .orderBy(F.col("total_revenue").desc()) .limit(5) ) ``` ## Dataset: Synthetic PySpark Code Tasks | Dataset statistic | Count | |---|---:| | Total examples | 900 | | Training examples | 600 | | Validation examples | 150 | | Held-out test examples | 150 | | Task categories | 15 | | Algorithm families | 180 | Dataset is **balanced across 15 task categories**, with **60 examples per category** (40 training, 10 validation, and 10 test). It covers: 1. DataFrame creation and transformations 2. Data ingestion and file formats 3. Data cleaning and preprocessing 4. Filtering, sorting, and aggregation 5. Joins and window functions 6. Spark SQL 7. UDFs and Pandas UDFs 8. Structured Streaming 9. Performance optimization and caching 10. Partitioning and repartitioning 11. Machine learning with MLlib 12. Nested and complex data types 13. Error handling and debugging 14. Delta Lake operations 15. Distributed computing and cluster execution Note: All 900 reference snippets passed Python `ast.parse` syntax checks. ## Model Evaluation The original Qwen2.5-Coder model and the fine-tuned adapter were evaluated on the same **150 held-out synthetic tasks**, using the same prompt construction and greedy decoding (`do_sample=False`) with an output limit of 384 new tokens. **Greedy decoding** selects the highest-probability next token at each step, making this comparison deterministic under the same runtime conditions. It does not search for the globally best full program. ### Held-out Test Set (150 Examples) | Metric | Base Qwen2.5-Coder-1.5B-Instruct | Fine-tuned PySparkQwen-Coder | |---|---:|---:| | Evaluated examples | 150 | 150 | | Python AST-valid outputs | 90.00% (135/150) | **98.67% (148/150)** | | Mean character-sequence similarity to reference | 0.1168 | **0.6621** | ## Model Limitations - **Missing context:** In rare cases (without context): The model may invent columns, tables, file paths, or dependencies if the task or schema is incomplete. - **Review before use:** Generated code should be reviewed and tested in a controlled Spark environment before running against production data. ## Example Import (PEFT) ```python import torch from peft import PeftModel from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig # Model repositories base_model = "Qwen/Qwen2.5-Coder-1.5B-Instruct" adapter_model = "AaronTekle/PySparkQwen" # Load the original Qwen2.5-Coder tokenizer tokenizer = AutoTokenizer.from_pretrained(base_model) if tokenizer.pad_token is None: tokenizer.pad_token = tokenizer.eos_token # Configure 4-bit quantization for lower GPU memory usage compute_dtype = ( torch.bfloat16 if torch.cuda.is_bf16_supported() else torch.float16 ) quantization = BitsAndBytesConfig( load_in_4bit=True, bnb_4bit_quant_type="nf4", bnb_4bit_use_double_quant=True, bnb_4bit_compute_dtype=compute_dtype, ) # Load the base Qwen2.5-Coder model base_model = AutoModelForCausalLM.from_pretrained( base_model, quantization_config=quantization, device_map="auto", ) # Load the fine-tuned PySparkQwen LoRA adapter model = PeftModel.from_pretrained( base_model, adapter_model, ) model.eval() # Example PySpark coding task context = ( "DataFrame orders: customer_id string, " "order_id string, amount double" ) question = ( "Calculate total revenue per customer " "and show the top five customers." ) # Define coding assistant behavior system_prompt = ( "You are a PySpark data engineering coding assistant. " "Return only executable Python/PySpark code with necessary imports. " "Prefer Spark DataFrame APIs over pandas and RDDs. " "Use the provided schemas and names without inventing tables or columns. " ) messages = [ {"role": "system", "content": system_prompt}, { "role": "user", "content": f"{question}\n\nContext / schema:\n{context}", }, ] # Apply Qwen2.5-Coder's native instruct chat template prompt = tokenizer.apply_chat_template( messages, tokenize=False, add_generation_prompt=True, ) # Tokenize the prompt and move tensors to the model's device inputs = tokenizer( prompt, return_tensors="pt", ).to(model.get_input_embeddings().weight.device) # Generate PySpark code using greedy decoding with torch.inference_mode(): output_ids = model.generate( **inputs, max_new_tokens=384, do_sample=False, pad_token_id=tokenizer.pad_token_id, eos_token_id=tokenizer.convert_tokens_to_ids("<|im_end|>"), ) # Decode only newly generated tokens new_tokens = output_ids[0][inputs["input_ids"].shape[-1]:] generated_code = tokenizer.decode( new_tokens, skip_special_tokens=True, ).strip() print("Generated PySpark Code:") print(generated_code) ``` ## References - [Qwen Team. Qwen2.5-Coder-1.5B-Instruct Model Card](https://huggingface.co/Qwen/Qwen2.5-Coder-1.5B-Instruct) - [Hui et al. Qwen2.5-Coder Technical Report (2024)](https://arxiv.org/abs/2409.12186) - [Hu et al. LoRA: Low-Rank Adaptation of Large Language Models (2021)](https://arxiv.org/abs/2106.09685) - [Dettmers et al. QLoRA: Efficient Finetuning of Quantized LLMs (2023)](https://arxiv.org/abs/2305.14314) - [Hugging Face. TRL SFTTrainer](https://huggingface.co/docs/trl/sft_trainer) - [Hugging Face. PEFT LoRA](https://huggingface.co/docs/peft/en/package_reference/lora) - [Hugging Face. PEFT Adapters in Transformers](https://huggingface.co/docs/transformers/peft)