base_model: Qwen/Qwen2.5-Coder-7B-Instruct
library_name: peft
LoRA adapters for natural-language CAD program editing (IE 643 course project, team Tired Perceptron)
patch_lora/: the model replies with a unified diff that edits the CadQuery program (ours).rewrite_lora/: the same training, but the model replies with the whole edited program (baseline).
Training (identical for both): Qwen2.5-Coder-7B-Instruct, LoRA r=16, alpha=32, dropout 0.05 on q/k/v/o/gate/up/down,
lr 2e-4 cosine, 3% warmup, 1 epoch over 27,839 triplets (dataset repo jajamken/DLTP_project), effective batch 32, bf16,
loss only on the reply. Prompt format: see prompts.py in the project code.
from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel
base = AutoModelForCausalLM.from_pretrained("Qwen/Qwen2.5-Coder-7B-Instruct", dtype="bfloat16", device_map="cuda")
model = PeftModel.from_pretrained(base, "jajamken/DLTP_project", subfolder="patch_lora")
Stage 2: further training of the patch model
All stage-2 adapters continue from patch_lora/ with the same prompt and LoRA shape. Their training programs are
restyled to look like BenchCAD's (values written inside the calls, parameter names only in a comment header,
show_object(result) at the end).
| folder | starts from | training |
|---|---|---|
patch_lora/ |
Qwen2.5-Coder-7B-Instruct | stage 1: SFT on 27,839 triplets |
patch_lora_v2/ |
patch_lora/ |
SFT on 2,871 restyled programs, 2 epochs (lr 2e-4 cosine, batch 32) |
patch_lora_v2_sft/ |
patch_lora_v2/ |
SFT on 1,503 restyled prompts, 2 epochs, same recipe: the control for GRPO |
patch_lora_v2_grpo/ |
patch_lora_v2/ |
GRPO on the same 1,503 prompts, 2 epochs (188 steps of 16 prompts x 8 sampled replies, lr 2e-5 constant, no KL); reward = 0.5 x normalized IoU of the edited solid with the target, + 0.5 when it is the right shape |
The 1,503 prompts are those where 8 sampled replies of patch_lora_v2 do not all get the same reward (GRPO learns
nothing from a group of equal rewards). patch_lora_v2_sft and patch_lora_v2_grpo see the same prompts, so the
difference between them is the training method. To load one, change subfolder in the snippet above, e.g.
subfolder="patch_lora_v2_sft".
Results (greedy decoding, normalized exact IoU, higher is better)
| adapter | BenchCAD edit-bench (748) | T1 | T2 | T3 | T4 | T5 | our test split (2,800) |
|---|---|---|---|---|---|---|---|
patch_lora/ |
0.119 | 0.401 | 0.010 | 0.035 | 0.039 | 0.017 | 0.947 |
patch_lora_v2/ |
0.314 | 0.450 | 0.855 | 0.300 | 0.117 | 0.053 | 0.831 |
patch_lora_v2_sft/ |
0.314 | 0.414 | 0.795 | 0.378 | 0.139 | 0.043 | 0.879 |
patch_lora_v2_grpo/ |
0.469 | 0.742 | 0.878 | 0.513 | 0.277 | 0.080 | 0.936 |
Normalized IoU per task = clip((IoU(edited, target) - IoU(original, target)) / (1 - IoU(original, target)), 0, 1); a patch
or program that fails counts as 0. Scored with our own scorer, not BenchCAD's harness: exact OpenCascade IoU (not
BenchCAD's voxel IoU), and on BenchCAD the part is the solid the program renders (the last object passed to
show_object; BenchCAD's harness exports the result variable instead). So these numbers are not on the scale of the
BenchCAD paper's table. GRPO vs SFT on the same prompts (_grpo vs _sft), paired per BenchCAD task: 172 tasks better,
44 worse (sign test p = 4.5e-19).