SparkFlashXGemma4

LoRA adapters for Gemma 4 31B IT (QAT, w4a16), fine-tuned to act as an autonomous coding agent: locate the relevant code, make a minimal change, and emit a unified diff.

These are PEFT LoRA adapters, not a full model. The base weights are the competition checkpoint gemma-4-31b-it-qat-w4a16-ct (INT4 weights, group-32 symmetric scales, 16-bit activations, ~16–17 GB in 4-bit) and are not included here.

Adapters

name	rank	params	size	role	target
main_lora	16	122,429,440	233.6 MB	root coding agent	410 language-model projections
tool_lora	8	61,214,720	116.9 MB	read-only code_analyzer	410 language-model projections

Both target q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj across all 60 decoder layers (vision tower excluded), with rsLoRA scaling (alpha = 2r).

What each adapter learns

• main_lora — problem statement → a three-line plan (LOCATE / CAUSE / CHANGE) followed by a fenced unified diff, because submit_patch() takes a diff and the harness grants a limited tool-call budget per task.
• tool_lora — problem statement → a localization report (FILES + SYMBOLS with line ranges and the enclosing symbol). It is trained on targets derived from the diff with the patch removed, so the read-only analyzer learns to find code without learning to emit it.

Training

Trained from google/gemma-4-31b-it-qat-q4_0-unquantized — the bf16 QAT weights, which the model card designates for research/fine-tuning (the compressed-tensors w4a16 file is the vLLM serving artifact and has no quantized forward path). The base is quantized to NF4 + double quantization at load time on 2× Tesla T4, per QLoRA. The adapter is shape-identical to one trained against the serving checkpoint.

	main_lora	tool_lora
steps	26 (2 epochs, eff. batch 8)	26
wall clock	1.10 h @ 120.4 s/step	0.62 h @ 70.4 s/step
final train loss	0.624	0.759
eval loss (25 held-out tasks)	0.857 → 0.804 → 0.791 → 0.791	0.850 → 0.778 → 0.759 → 0.759

Held-out eval loss decreases monotonically in both runs — the adapters learn a transferable output format rather than memorising the 104 training examples.

Method references

• QLoRA (arXiv:2305.14314) — NF4, double quantization, 8-bit optimizer state
• LoRA (arXiv:2106.09685) — frozen base, low-rank A/B only
• rsLoRA (arXiv:2312.03732) — alpha/sqrt(r) so rank 16 is trainable
• ZeRO (arXiv:1910.02054) — optimizer state as the binding memory constraint
• LoftQ (arXiv:2310.08659) / QuAILoRA (arXiv:2410.14713) — evaluated, not used (a QAT-trained base has little quantization residual to recover)
• PiSSA (arXiv:2404.02948) / DoRA (arXiv:2402.09353) — rejected; PiSSA replaces the frozen base weight, DoRA is not loadable by vLLM's LoRA path

Two implementation details worth flagging, both verified at runtime:

1. Chunked cross-entropy. With a 262,144-token vocabulary, materialising [seq, vocab] logits needs ~2.7 GB at seq 1024, which does not fit alongside the 4-bit base on a 14.5 GB GPU. The loss is computed in checkpointed chunks over supervised positions only. It was checked against the model's own loss (abs_diff 0.0) before being enabled, which also confirms Gemma 4's final_logit_softcapping = 30.0 is being reproduced.
2. Gemma 4 wraps vision-tower projections in Gemma4ClippableLinear, which PEFT refuses to target. LoRA targets are resolved from the live module tree rather than hard-coded, and the vision tower is excluded.

Data

Only the competition dev split (tasks.jsonl, 129 tasks with gold patches), seed-42 split into 104 train / 25 validation. No external agent-trajectory corpus was used. Prompts are rendered by the model's own chat template with add_generation_prompt=True, so the trained response format matches the served format; loss is masked to the assistant turn only.

Usage

Requires ~20 GB VRAM for the 4-bit base (it does not fit on a 6 GB laptop GPU).

import torch
from transformers import (AutoProcessor, AutoModelForCausalLM,
                          BitsAndBytesConfig)
from peft import PeftModel

BASE = "google/gemma-4-31b-it-qat-w4a16-ct"
REPO = "<this-repo>"

bnb = BitsAndBytesConfig(load_in_4bit=True, bnb_4bit_quant_type="nf4",
                         bnb_4bit_use_double_quant=True,
                         bnb_4bit_compute_dtype=torch.float16)

model = AutoModelForCausalLM.from_pretrained(
    BASE, quantization_config=bnb, dtype=torch.float16, device_map="auto")
model = PeftModel.from_pretrained(model, f"{REPO}/main_lora")
processor = AutoProcessor.from_pretrained(BASE)

messages = [{"role": "user", "content": "Fix the failing test in tests/test_util.py"}]
text = processor.apply_chat_template(messages, tokenize=False,
                                     add_generation_prompt=True,
                                     enable_thinking=False)
inputs = processor(text=text, return_tensors="pt").to(model.device)
out = model.generate(**inputs, max_new_tokens=1024)
print(processor.decode(out[0][inputs["input_ids"].shape[-1]:],
                       skip_special_tokens=False))

Serving with vLLM multi-LoRA:

vllm serve google/gemma-4-31b-it-qat-w4a16-ct \
  --enable-lora --max-lora-rank 128 --max-loras 8 \
  --lora-modules main_lora=./main_lora tool_lora=./tool_lora

Honest limitations

• 104 training examples against 122 M trainable parameters: heavily over-parameterised. Mitigated with rank 16 (not 128), 2 epochs, lr 1e-4, gradient clipping, and a held-out split — but the eval curve is the only evidence that this is not pure memorisation.
• The adapters shape output format, not capability. The base model's SWE ability is unchanged, and a mis-shaped adapter could only reduce resolution rate, never raise it above the base.
• Trained against NF4 quantization of the base; served by vLLM on int4-group-32. Both are 4-bit approximations of the same QAT weights, so a small numeric difference remains between training and serving.