Update README.md

808df0f verified over 1 year ago

4.76 kB

---
license: apache-2.0
language:
- en
base_model:
- unsloth/Qwen2.5-3B-Instruct-unsloth-bnb-4bit
library_name: adapter-transformers
tags:
- text-generation-inference
- transformers
- unsloth
- qwen2
- trl
- grpo
---
# Uploaded  model

- **Developed by:** Akshint47
- **License:** apache-2.0
- **Finetuned from model :** unsloth/qwen2.5-3b-instruct-unsloth-bnb-4bit

This qwen2 model was trained 2x faster with [Unsloth](https://github.com/unslothai/unsloth) and Huggingface's TRL library.

[<img src="https://raw.githubusercontent.com/unslothai/unsloth/main/images/unsloth%20made%20with%20love.png" width="200"/>](https://github.com/unslothai/unsloth)
# Fine-Tuning Qwen2.5-3B-Instruct with GRPO for GSM8K Dataset

This notebook demonstrates the process of fine-tuning the **Qwen2.5-3B-Instruct** model using **GRPO (Generalized Reward Policy Optimization)** on the **GSM8K** dataset. The goal is to improve the model's ability to solve mathematical reasoning problems by leveraging reinforcement learning with custom reward functions.

## Overview

The notebook is structured as follows:

1. **Installation**: Installs necessary libraries such as `unsloth`, `vllm`, and `trl` for efficient fine-tuning and inference.
2. **Unsloth Setup**: Configures the environment for faster fine-tuning using Unsloth's `PatchFastRL` and loads the Qwen2.5-3B-Instruct model with LoRA (Low-Rank Adaptation) for parameter-efficient fine-tuning.
3. **Data Preparation**: Loads and preprocesses the GSM8K dataset, formatting it for training with a system prompt and XML-style reasoning and answer format.
4. **Reward Functions**: Defines custom reward functions to evaluate the model's responses, including:
   - **Correctness Reward**: Checks if the extracted answer matches the ground truth.
   - **Format Reward**: Ensures the response follows the specified XML format.
   - **Integer Reward**: Verifies if the extracted answer is an integer.
   - **XML Count Reward**: Evaluates the completeness of the XML structure in the response.
5. **GRPO Training**: Configures and runs the GRPO trainer with vLLM for fast inference, using the defined reward functions to optimize the model's performance.
6. **Training Progress**: Monitors the training progress, including rewards, completion length, and KL divergence, to ensure the model is improving over time.

## Key Features

- **Efficient Fine-Tuning**: Utilizes Unsloth and LoRA to fine-tune the model with reduced memory usage and faster training times.
- **Custom Reward Functions**: Implements multiple reward functions to guide the model towards generating correct and well-formatted responses.
- **vLLM Integration**: Leverages vLLM for fast inference during training, enabling efficient generation of multiple responses for reward calculation.
- **GSM8K Dataset**: Focuses on improving the model's performance on mathematical reasoning tasks, specifically the GSM8K dataset.

## Requirements

- Python 3.11
- Libraries: `unsloth`, `vllm`, `trl`, `torch`, `transformers`

## Installation

To set up the environment, run:

```bash
pip install unsloth vllm trl
```

## Usage
- **Load the Model**: The notebook loads the Qwen2.5-3B-Instruct model with LoRA for fine-tuning.

- **Prepare the Dataset**: The GSM8K dataset is loaded and formatted with a system prompt and XML-style reasoning and answer format.

- **Define Reward Functions**: Custom reward functions are defined to evaluate the model's responses.

- **Train the Model**: The GRPO trainer is configured and run to fine-tune the model using the defined reward functions.

- **Monitor Progress**: The training progress is monitored, including rewards, completion length, and KL divergence.

## Results
- The training process is designed to improve the model's ability to generate correct and well-formatted responses to mathematical reasoning problems. The reward functions guide the model towards better performance, and the training progress is logged for analysis.

## Future Work
- **Hyperparameter Tuning**: Experiment with different learning rates, batch sizes, and reward weights to optimize performance.

- **Additional Datasets**: Extend the fine-tuning process to other datasets to improve the model's generalization capabilities.

- **Advanced Reward Functions**: Implement more sophisticated reward functions to further refine the model's responses.

## Acknowledgments
- **Unsloth**: For providing tools to speed up fine-tuning.

- **vLLM**: For enabling fast inference during training.

- **Hugging Face**: For the trl library and the GSM8K dataset.

- Special thanks to @sudhir2016 sir for mentoring me for developing such a prominent fine-tuning model.

## License
This project is licensed under the MIT License. See the LICENSE file for details.

	---
	license: apache-2.0
	language:
	- en
	base_model:
	- unsloth/Qwen2.5-3B-Instruct-unsloth-bnb-4bit
	library_name: adapter-transformers
	tags:
	- text-generation-inference
	- transformers
	- unsloth
	- qwen2
	- trl
	- grpo
	---

	# Uploaded model

	- Developed by: Akshint47
	- License: apache-2.0
	- Finetuned from model : unsloth/qwen2.5-3b-instruct-unsloth-bnb-4bit

	This qwen2 model was trained 2x faster with [Unsloth](https://github.com/unslothai/unsloth) and Huggingface's TRL library.

	[<img src="https://raw.githubusercontent.com/unslothai/unsloth/main/images/unsloth%20made%20with%20love.png" width="200"/>](https://github.com/unslothai/unsloth)
	# Fine-Tuning Qwen2.5-3B-Instruct with GRPO for GSM8K Dataset

	This notebook demonstrates the process of fine-tuning the Qwen2.5-3B-Instruct model using GRPO (Generalized Reward Policy Optimization) on the GSM8K dataset. The goal is to improve the model's ability to solve mathematical reasoning problems by leveraging reinforcement learning with custom reward functions.

	## Overview

	The notebook is structured as follows:

	1. Installation: Installs necessary libraries such as `unsloth`, `vllm`, and `trl` for efficient fine-tuning and inference.
	2. Unsloth Setup: Configures the environment for faster fine-tuning using Unsloth's `PatchFastRL` and loads the Qwen2.5-3B-Instruct model with LoRA (Low-Rank Adaptation) for parameter-efficient fine-tuning.
	3. Data Preparation: Loads and preprocesses the GSM8K dataset, formatting it for training with a system prompt and XML-style reasoning and answer format.
	4. Reward Functions: Defines custom reward functions to evaluate the model's responses, including:
	- Correctness Reward: Checks if the extracted answer matches the ground truth.
	- Format Reward: Ensures the response follows the specified XML format.
	- Integer Reward: Verifies if the extracted answer is an integer.
	- XML Count Reward: Evaluates the completeness of the XML structure in the response.
	5. GRPO Training: Configures and runs the GRPO trainer with vLLM for fast inference, using the defined reward functions to optimize the model's performance.
	6. Training Progress: Monitors the training progress, including rewards, completion length, and KL divergence, to ensure the model is improving over time.

	## Key Features

	- Efficient Fine-Tuning: Utilizes Unsloth and LoRA to fine-tune the model with reduced memory usage and faster training times.
	- Custom Reward Functions: Implements multiple reward functions to guide the model towards generating correct and well-formatted responses.
	- vLLM Integration: Leverages vLLM for fast inference during training, enabling efficient generation of multiple responses for reward calculation.
	- GSM8K Dataset: Focuses on improving the model's performance on mathematical reasoning tasks, specifically the GSM8K dataset.

	## Requirements

	- Python 3.11
	- Libraries: `unsloth`, `vllm`, `trl`, `torch`, `transformers`

	## Installation

	To set up the environment, run:

	```bash
	pip install unsloth vllm trl

	```

	## Usage
	- Load the Model: The notebook loads the Qwen2.5-3B-Instruct model with LoRA for fine-tuning.

	- Prepare the Dataset: The GSM8K dataset is loaded and formatted with a system prompt and XML-style reasoning and answer format.

	- Define Reward Functions: Custom reward functions are defined to evaluate the model's responses.

	- Train the Model: The GRPO trainer is configured and run to fine-tune the model using the defined reward functions.

	- Monitor Progress: The training progress is monitored, including rewards, completion length, and KL divergence.

	## Results
	- The training process is designed to improve the model's ability to generate correct and well-formatted responses to mathematical reasoning problems. The reward functions guide the model towards better performance, and the training progress is logged for analysis.

	## Future Work
	- Hyperparameter Tuning: Experiment with different learning rates, batch sizes, and reward weights to optimize performance.

	- Additional Datasets: Extend the fine-tuning process to other datasets to improve the model's generalization capabilities.

	- Advanced Reward Functions: Implement more sophisticated reward functions to further refine the model's responses.

	## Acknowledgments
	- Unsloth: For providing tools to speed up fine-tuning.

	- vLLM: For enabling fast inference during training.

	- Hugging Face: For the trl library and the GSM8K dataset.

	- Special thanks to @sudhir2016 sir for mentoring me for developing such a prominent fine-tuning model.

	## License
	This project is licensed under the MIT License. See the LICENSE file for details.