AaronTekle commited on
Commit
86bd38d
·
verified ·
1 Parent(s): 35ee81a

Upload README.md

Browse files
Files changed (1) hide show
  1. README.md +219 -0
README.md ADDED
@@ -0,0 +1,219 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # SparkQwen (Spark-QLoRA, [Qwen/Qwen3-1.7B](https://huggingface.co/Qwen/Qwen3-1.7B) )
2
+
3
+ QLoRA fine-tuning of [**Qwen/Qwen3-1.7B**](https://huggingface.co/Qwen/Qwen3-1.7B) for generating Spark/PySpark code.
4
+
5
+ SparkQwen is a lightweight coding language model for day-to-day data engineering tasks, including DataFrame manipulations, joins, Spark SQL, structured streaming, Delta Lake operations, optimization techniques, MLlib, and more.
6
+
7
+ given:
8
+
9
+ - PySpark prompts/task
10
+ - relevant DataFrame schemas, column names, (and/or runtime context, if necessary)
11
+
12
+ the model then generates the relevant Spark/PySpark code
13
+
14
+ **Training Note:** Trained locally using 4-bit quantization and parameter-efficient LoRA adapters
15
+
16
+
17
+ ## Example (Dataset Context)
18
+
19
+ **DataFrame context:**
20
+
21
+ ```text
22
+ DataFrame: orders
23
+ Columns: customer_id (string), order_id (string), amount (double)
24
+ ```
25
+
26
+ **example prompt:**
27
+
28
+ Calculate total revenue per customer and return the five customers with the highest revenue.
29
+
30
+ **example output :**
31
+
32
+ ```python
33
+ from pyspark.sql import functions as F
34
+
35
+ result = (
36
+ orders.groupBy("customer_id")
37
+ .agg(F.sum("amount").alias("total_revenue"))
38
+ .orderBy(F.col("total_revenue").desc())
39
+ .limit(5)
40
+ )
41
+ ```
42
+
43
+ ## Why QLoRA
44
+
45
+ QLoRA combines low-rank adaptation (LoRA) with a quantized, frozen pretrained model. Only the added adapter parameters are updated during training, reducing memory requirements compared with full fine-tuning.
46
+
47
+ base model was loaded in 4-bit NF4 with double quantization. LoRA adapters were trained on attention and feed-forward projection layers:
48
+
49
+ `q_proj`, `k_proj`, `v_proj`, `o_proj`, `gate_proj`, `up_proj`, `down_proj`.
50
+
51
+ **17,432,576 of 1,738,007,552 parameters were trainable (1.0030%).** The reported total includes the adapter parameters.
52
+
53
+ | LoRA setting | Value |
54
+ |---|---|
55
+ | Task type | Causal language modeling |
56
+ | Base quantization | 4-bit NF4 with double quantization |
57
+
58
+ ## Model
59
+
60
+ | Component | Value |
61
+ |---|---|
62
+ | Base model | `Qwen/Qwen3-1.7B` |
63
+ | Fine-tuning | PEFT QLoRA, TRL `SFTTrainer` |
64
+ | Libraries | Transformers, PEFT, TRL, bitsandbytes, PyTorch |
65
+ | Training output | PEFT LoRA adapter |
66
+
67
+
68
+ ## Dataset (Local Generated Synthetic Spark Dataset)
69
+
70
+
71
+ | Dataset statistic | Count |
72
+ |---|---:|
73
+ | Total examples | 900 |
74
+ | Training examples | 600 |
75
+ | Validation examples | 150 |
76
+ | Held-out test examples | 150 |
77
+ | PySpark task categories | 15 |
78
+ | Algorithm families | 180 |
79
+
80
+
81
+ **Categories:**
82
+ - DataFrame transformations
83
+ - ingestion and file formats
84
+ - cleaning and preprocessing
85
+ - filtering, sorting, and aggregation
86
+ - joins and windows
87
+ - Spark SQL
88
+ - UDFs and Pandas UDFs
89
+ - structured streaming
90
+ - performance optimization
91
+ - partitioning and repartitioning
92
+ - MLlib
93
+ - nested and complex types
94
+ - debugging; Delta Lake
95
+ - distributed cluster execution
96
+
97
+ note: All 900 reference snippets passed Python AST syntax parsing. (means that your dataset is structurally clean and ready for analysis, and or model training (without needing syntax-level preprocessing)
98
+ ## Training
99
+
100
+
101
+ ### Loss by Epoch
102
+
103
+ | Epoch | Step | Training loss | Validation loss |
104
+ |---|---:|---:|---:|
105
+ | 1 | 75 | 0.1145 | 0.5008 |
106
+ | 2 | 150 | 0.0213 | 0.5720 |
107
+ | 3 | 225 | 0.0098 | 0.6047 |
108
+
109
+
110
+ ## Model Evaluation
111
+
112
+ Both models were evaluated on the same 150 held-out synthetic test tasks under the same loading and generation settings: non-thinking prompts, greedy decoding (b/c it's deterministic, computationally fast, and removes sampling randomness), and a maximum of 384 new tokens
113
+ - [**greedy decoding**](https://machinelearningmastery.com/decoding-strategies-and-output-control/): deterministic text generation strategy in ML where a model picks the single token with the highest predicted probability at each step of an autoregressive sequence (model that predicts the next element in a sequence by conditioning on its own past outputs or lagged values)
114
+ - **note:** greedy decoding called "greedy" because it always grabs the immediate best option (the token with the highest probability) at every single step without caring about the long-term consequence for the rest of the sentence
115
+ - **immediate local gain:** at each step, the language model looks at all possible next words and "greedily" picks the single highest-scoring one
116
+ - **no long-term planning:** greedy decoding algorithms never look ahead or evaluates whether choosing that local favorite will trap the model in a repetitive loop, a dead end, or a awkward phrasing two or three words later
117
+ - **myopic behavior:** just like a greedy person who only thinks about right now rather than the future, the algorithm maximizes step-by-step payoff at the expense of the global (whole-sentence) quality.
118
+
119
+ **Note (AST):** Python AST is a built-in standard library tool that parses Python source code into an Abstract Syntax Tree (AST). The tree represents code as structured Python objects rather than plain text, allowing you to inspect, analyze, and modify programs before they are compiled into bytecode.
120
+
121
+ **Note:**
122
+ - **Mean Character-Sequence:** AVG similarity score computed across a set of textual information when each is compared character-by-character against a ground-truth reference sequence
123
+
124
+ | Metric | Base Qwen3-1.7B | Qwen3 + PySpark adapter |
125
+ |---|---:|---:|
126
+ | Syntax-valid outputs (Python AST (Abstract Syntax Tree)) | 58.67% (88/150) | **95.33%** (143/150) |
127
+ | Mean character-sequence similarity to reference | 0.1009 | 0.6354 |
128
+
129
+
130
+
131
+ ## Model Limits
132
+ - **Note:** Rare Cases (if context is not provided): model may invent (hallucinate) columns, tables, file paths, or dependencies when insufficient context is provided
133
+ - Training used 600 synthetic examples from 120 algorithm families; the model has not been validated on production tasks
134
+ - Held out test tasks come from the same synthetic generation method as the training data, even though algorithm families are separated by split
135
+ - Delta Lake operations, streaming jobs, cloud connectors, and distributed execution require compatible runtime configuration and should be validated before use
136
+
137
+ ## Example Import (PEFT)
138
+
139
+ **Note 1:** Example prompt requires a compatible CUDA GPU, PyTorch, Transformers, PEFT, Accelerate, and bitsandbytes. For larger GPUs, loading the unquantized base model is also possible
140
+
141
+ **Note 2:** this model uses an adapter, not the full Qwen3 base model
142
+
143
+ ```python
144
+ import torch
145
+ from peft import PeftModel
146
+ from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig
147
+
148
+ base_id = "Qwen/Qwen3-1.7B"
149
+ adapter_id = "AaronTekle/SparkQwen"
150
+
151
+ tokenizer = AutoTokenizer.from_pretrained(base_id)
152
+ if tokenizer.pad_token is None:
153
+ tokenizer.pad_token = tokenizer.eos_token
154
+
155
+ # Use 4-bit loading for lower GPU memory consumption
156
+ compute_dtype = torch.bfloat16 if torch.cuda.is_bf16_supported() else torch.float16
157
+ quantization = BitsAndBytesConfig(
158
+ load_in_4bit=True,
159
+ bnb_4bit_quant_type="nf4",
160
+ bnb_4bit_use_double_quant=True,
161
+ bnb_4bit_compute_dtype=compute_dtype,
162
+ )
163
+
164
+ base_model = AutoModelForCausalLM.from_pretrained(
165
+ base_id,
166
+ quantization_config=quantization,
167
+ device_map="auto",
168
+ )
169
+ model = PeftModel.from_pretrained(base_model, adapter_id)
170
+ model.eval()
171
+
172
+ context = "DataFrame orders: customer_id string, order_id string, amount double"
173
+ question = "Calculate total revenue per customer and show the top five customers."
174
+
175
+ system_prompt = (
176
+ "You are a Spark/PySpark data engineering coding assistant. "
177
+ "Return only executable Spark/PySpark code with necessary imports. "
178
+ "Prefer Spark DataFrame APIs over pandas and RDDs. "
179
+ "Use the provided schemas and names without inventing tables or columns. "
180
+ "Do not add Markdown fences or prose."
181
+ )
182
+
183
+ messages = [
184
+ {"role": "system", "content": system_prompt},
185
+ {"role": "user", "content": f"{question}\n\nContext / schema:\n{context}"},
186
+ ]
187
+
188
+ prompt = tokenizer.apply_chat_template(
189
+ messages,
190
+ tokenize=False,
191
+ add_generation_prompt=True,
192
+ enable_thinking=False,
193
+ )
194
+ inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
195
+
196
+ with torch.inference_mode():
197
+ output_ids = model.generate(
198
+ **inputs,
199
+ max_new_tokens=384,
200
+ do_sample=False,
201
+ pad_token_id=tokenizer.pad_token_id,
202
+ eos_token_id=tokenizer.convert_tokens_to_ids("<|im_end|>"),
203
+ )
204
+
205
+ generated_code = tokenizer.decode(
206
+ output_ids[0][inputs["input_ids"].shape[-1]:],
207
+ skip_special_tokens=True,
208
+ ).strip()
209
+ print(generated_code)
210
+ ```
211
+
212
+ ## References
213
+
214
+ - [Qwen Team. Qwen3-1.7B Model Card](https://huggingface.co/Qwen/Qwen3-1.7B)
215
+ - [Yang, A. et al. Qwen3 Technical Report (2025)](https://arxiv.org/abs/2505.09388)
216
+ - [Hu, E. J. et al. LoRA: Low-Rank Adaptation of Large Language Models (2021)](https://arxiv.org/abs/2106.09685)
217
+ - [Dettmers, T. et al. QLoRA: Efficient Finetuning of Quantized LLMs (2023)](https://arxiv.org/abs/2305.14314)
218
+ - [Hugging Face. Parameter-Efficient Fine-Tuning](https://huggingface.co/docs/transformers/peft)
219
+ - [Hugging Face. PEFT LoRA](https://huggingface.co/docs/peft/en/package_reference/lora)