Spaces:
Sleeping
Sleeping
File size: 7,398 Bytes
dfbf500 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 | # Lab 2 - Fine-Tuning a Large Language Model
## Overview
In this lab we fine-tuned the open-source base model `unsloth/Llama-3.2-3B-Instruct` on Maxime Labonne's FineTome-100k instruction dataset using LoRA (Low-Rank Adaptation) with QLoRA quantization via Unsloth and `trl.SFTTrainer`. The training used the following hyperparameters:
| Hyperparameter | Value |
|----------------|-------|
| `per_device_train_batch_size` | 2 |
| `gradient_accumulation_steps` | 4 |
| `num_train_epochs` | 1 |
| `learning_rate` | 2e-4 |
| `warmup_steps` | 5 |
| `weight_decay` | 0.01 |
| `optimizer` | adamw_8bit |
---
## Evaluation Methodology
We evaluated both the base model and the fine-tuned model on **100 held-out examples** from a 15% test split of FineTome-100k. For each example we used the conversation history as input and treated the last assistant turn as the reference answer.
### Metrics Used
We used **ROUGE scores** (Recall-Oriented Understudy for Gisting Evaluation) to measure the overlap between model-generated responses and reference answers:
- **ROUGE-1**: Measures unigram (single word) overlap between the generated text and reference. Higher scores indicate better word-level similarity.
- **ROUGE-2**: Measures bigram (two consecutive words) overlap. This captures phrase-level similarity and is more sensitive to word order.
- **ROUGE-L**: Measures the longest common subsequence between generated and reference text. This captures sentence-level structure and fluency.
All scores use the F-measure (harmonic mean of precision and recall) and range from 0 to 1, where higher is better.
### Generation Settings
We used **deterministic generation** (greedy decoding with `do_sample=False`) for both models to ensure reproducible and comparable results. Each response was limited to 256 new tokens.
---
## Results
The fine-tuned model clearly outperformed the base model across all ROUGE metrics:
| Metric | Base Model | Fine-Tuned Model | Improvement |
|--------|------------|------------------|-------------|
| ROUGE-1 | 0.4732 | 0.5323 | **+12.5%** |
| ROUGE-2 | 0.2255 | 0.2849 | **+26.4%** |
| ROUGE-L | 0.2856 | 0.3521 | **+23.3%** |
These results demonstrate that even a single epoch of LoRA fine-tuning on FineTome-100k produces substantial improvements in response quality. The largest gain was in ROUGE-2 (+26.4%), indicating that the fine-tuned model better captures phrase-level patterns from the training data.
### Response Length Analysis
| Metric | Base Model | Fine-Tuned Model | Reference |
|--------|------------|------------------|-----------|
| Mean length (words) | 165.0 | 155.7 | 216.3 |
| Median length (words) | 176.0 | 165.5 | 199.0 |
The fine-tuned model produces slightly more concise responses while achieving higher ROUGE scores, suggesting improved information density.
---
## Improving Model Performance
### (a) Model-Centric Approach
A model-centric approach keeps the data fixed and focuses on changing the model architecture, training configuration, or optimization procedure. Below are concrete strategies for further improvement:
#### Hyperparameter Tuning
- **Learning rate**: Sweep over values such as `1e-4`, `2e-4`, `5e-4` to find the optimal learning rate. Our current setting of `2e-4` is a reasonable default but may not be optimal.
- **Learning rate schedule**: Experiment with cosine decay or cosine with warm restarts instead of constant learning rate.
- **Training epochs**: Train for 2–3 epochs with early stopping based on validation loss to potentially improve convergence.
- **Batch size**: Increase effective batch size (via `gradient_accumulation_steps`) within memory constraints for more stable gradients.
- **Warmup steps**: Adjust warmup duration (e.g., 10–100 steps) to improve training stability.
- **Weight decay**: Test different regularization strengths (e.g., `0.001`, `0.01`, `0.1`) to control overfitting.
#### LoRA Configuration
- **Rank (r)**: Increase LoRA rank (e.g., from 16 to 32 or 64) to allow more expressive adapter updates, at the cost of increased memory.
- **Alpha scaling**: Adjust the LoRA alpha parameter to control the magnitude of adapter contributions.
- **Target modules**: Experiment with applying LoRA to different layer types (attention only, MLP layers, or both) and different layer ranges.
#### Model Architecture
- **Base model selection**: Compare different foundation models such as `Llama-3.2-1B-Instruct` (faster inference) or `Llama-3.1-8B-Instruct` (potentially higher quality but slower).
- **Quantization**: Compare 4-bit (QLoRA) vs 8-bit quantization to understand the quality-speed tradeoff.
#### Training Procedure
- **Gradient clipping**: Add gradient clipping to prevent exploding gradients and improve training stability.
- **Mixed precision**: Ensure optimal use of mixed precision training for faster iteration.
### (b) Data-Centric Approach
A data-centric approach keeps the model and training loop mostly fixed and focuses on improving or extending the training data. Below are concrete strategies:
#### Data Quality Improvements
- **Filter low-quality examples**: Remove very short, unclear, or noisy instruction-response pairs from FineTome-100k to increase average signal per batch.
- **Deduplicate**: Remove near-duplicate examples that may cause the model to overfit to specific patterns.
- **Balance task types**: If the target application focuses on specific capabilities (e.g., reasoning, coding, explanation), up-sample those categories and down-sample less relevant ones.
#### Additional Data Sources
Augment FineTome-100k with other high-quality open-source instruction datasets:
| Dataset | Focus Area | Potential Benefit |
|---------|------------|-------------------|
| OpenAssistant Conversations | Multi-turn dialogue | Improved conversational ability |
| GSM8K / MetaMath | Math reasoning | Better mathematical problem-solving |
| CodeAlpaca / Code-Feedback | Programming tasks | Improved code generation |
| FLAN Collection | Diverse NLP tasks | Broader task coverage |
| UltraChat | Long-form dialogue | Better handling of extended conversations |
#### Domain-Specific Fine-Tuning
- **Curriculum learning**: Start training on general instructions, then gradually shift to more specialized or difficult examples.
- **Task-specific adapters**: Train separate LoRA adapters for different domains (math, code, creative writing) and select the appropriate adapter at inference time.
#### Data Alignment
- **Match UI format**: If the final application expects specific output formats (e.g., step-by-step reasoning, JSON responses), construct or filter training examples that demonstrate these formats.
- **User feedback loop**: In production, log anonymized user interactions (if permitted) to create a fine-tuning set that reflects real usage patterns.
---
## Conclusion
Our fine-tuning pipeline demonstrates measurable improvements over the base model, with ROUGE scores increasing by 12–26% on a held-out test set. The model-centric and data-centric strategies outlined above provide clear directions for further performance gains. The most promising next steps would be:
1. **Hyperparameter sweep** on learning rate and number of epochs
2. **Increase LoRA rank** to allow more expressive updates
3. **Mix in domain-specific datasets** (e.g., math reasoning or code) to improve performance on specialized tasks
|