| --- |
| license: apache-2.0 |
| library_name: transformers |
| tags: |
| - reward-model |
| - rlhf |
| - alignment |
| --- |
| |
| # BestRewardModel |
|
|
| <div align="center"> |
| <img src="figures/training_curve.png" width="70%" alt="Training Curve" /> |
| </div> |
|
|
| ## Model Description |
|
|
| This is a reward model trained for RLHF alignment, selected from multiple experimental runs based on validation accuracy and reward alignment quality. |
|
|
| ## Selection Criteria |
|
|
| The best checkpoint was chosen according to: |
| - **Highest `val_accuracy`** among all final checkpoints |
| - **Minimum `reward_alignment_score` threshold of 0.80** |
| |
| Only checkpoints satisfying **both** conditions were eligible. |
| |
| ## Training Runs Comparison |
| |
| | Run | Base Model | Learning Rate | Final Step | Val Accuracy | Reward Alignment | Train Loss | |
| |-----|-----------|--------------|------------|-------------|-----------------|------------| |
| | run_gpt2_base_lr1e4 | GPT-2 Base | 1e-4 | 1000 | 0.907 | 0.876 | 0.115 | |
| | run_gpt2_base_lr5e5 | GPT-2 Base | 5e-5 | 1000 | 0.870 | 0.839 | 0.207 | |
| | run_gpt2_large_lr1e4 | GPT-2 Large | 1e-4 | 1000 | 0.958 | 0.928 | 0.061 | |
| | run_gpt2_large_lr5e5 | GPT-2 Large | 5e-5 | 1000 | 0.901 | 0.854 | 0.159 | |
| | run_deberta_lr1e4 | DeBERTa-v2 | 1e-4 | 1000 | 0.837 | 0.827 | 0.301 | |
| |
| ## Best Run Metrics |
| |
| | Metric | Value | |
| |--------|-------| |
| | Run Name | run_gpt2_large_lr1e4 | |
| | Val Accuracy | 0.958 | |
| | Reward Alignment Score | 0.928 | |
| | Final Train Loss | 0.061 | |
| |
| ## Intended Uses |
| |
| This model is intended for use as a reward model in RLHF pipelines to score and rank model outputs based on human preference alignment. |
| |
| ## How to Use |
| |
| ```python |
| from transformers import AutoModelForSequenceClassification, AutoTokenizer |
| |
| model = AutoModelForSequenceClassification.from_pretrained("BestRewardModel-TestRepo") |
| tokenizer = AutoTokenizer.from_pretrained("BestRewardModel-TestRepo") |
| |
| inputs = tokenizer("prompt", "response", return_tensors="pt") |
| score = model(**inputs).logits[0].item() |
| ``` |
| |
| <div align="center"> |
| <img src="figures/reward_dist.png" width="60%" alt="Reward Distribution" /> |
| </div> |
| |
| ## License |
| |
| Apache-2.0 |
| |
