| --- |
| base_model: unsloth/meta-llama-3.1-8b-instruct-bnb-4bit |
| library_name: peft |
| --- |
| |
| # SCoReLoRA: Self-Correct via Reinforcement Learning |
|
|
| SCoReLoRA is an innovative approach to fine-tuning language models using Low-Rank Adaptation (LoRA) combined with reinforcement learning techniques for self-correction. This method aims to improve the model's ability to generate more accurate and refined responses through a two-stage training process. |
|
|
| ## Features |
|
|
| - Implements a two-stage training process for self-correction |
| - Utilizes reinforcement learning to improve model outputs |
| - Compatible with Hugging Face's Transformers library and PEFT |
| - Supports quantized models for efficient fine-tuning |
| - Includes evaluation metrics for self-correction performance |
|
|
| ## How It Works |
|
|
| SCoreLora uses a two-stage training process: |
|
|
| 1. **Stage I**: The model is trained to generate initial responses and then correct them, minimizing the KL divergence between the base model and the fine-tuned model. |
|
|
| 2. **Stage II**: The model is further trained using reinforcement learning techniques, with rewards based on the quality of self-corrections. |
|
|
| The training process utilizes shaped rewards and KL divergence to balance between improvement and staying close to the original model's behavior. |
|
|
| ## Evaluation |
|
|
| The implementation includes functions to evaluate the model's self-correction capabilities, measuring metrics such as: |
|
|
| - Accuracy before and after correction |
| - Improvement rate |
| - Rate of successful corrections |
| - Rate of erroneous corrections |
|
|
| ## Reference |
|
|
| - [Training Language Models to Self-Correct via Reinforcement Learning](https://arxiv.org/abs/2409.12917) |
