-
Upload 11 files
# 🧠 Fine-Tuning Spec Sheet (Gemma 3 4B + LoRA)
## 📌 Base Model
- **Model ID:** `google/gemma-3-4b-it`
- **Model Type:** Causal Language Model
- **Precision:** float32
- **Device Mapping:** `auto`
- **Attention Implementation:** `eager`
- **Tokenizer Padding:** `right`
- **Special Token Handling:**
- `pad_token` is set to `eos_token` if not defined
---
## 🔧 LoRA Configuration
- **r (rank):** 8
- **lora_alpha:** 32
- **lora_dropout:** 0.05
- **bias:** `"none"`
- **target_modules:** `["q_proj", "k_proj", "v_proj"]`
- **task_type:** `TaskType.CAUSAL_LM`
- **Framework:** PEFT (`get_peft_model`)
---
## 🏋️ SFT Training Configuration
- **output_dir:** `./outputs`
- **per_device_train_batch_size:** 1
- **gradient_accumulation_steps:** 8
- **num_train_epochs:** 3
- **logging_steps:** 5
- **learning_rate:** 5e-5
- **warmup_steps:** 50
- **optim:** `adamw_torch`
- **save_strategy:** `epoch`
- **report_to:** `"none"` (no external logger)
- **fp16:** `True`
- **max_seq_length:** 512
- **p…