---
license: apache-2.0
library_name: transformers
tags:
- text-classification
- roberta
- hyperparameter-sweep
---
# SweepBestModel

<div align="center">
  <img src="figures/fig1.png" width="70%" alt="SweepBestModel overview" />
</div>

## Overview

SweepBestModel is a RoBERTa-base model fine-tuned for sequence classification through a systematic hyperparameter sweep. We explored learning rates and weight decay values to find the optimal configuration.

## Training Configuration

| Run | Learning Rate | Weight Decay | Best Checkpoint | Best F1 |
|---|---|---|---|---|
| run_lr2e-5_wd0.01 | 2e-5 | 0.01 | \u2014 | {RESULT} |
| run_lr5e-5_wd0.01 | 5e-5 | 0.01 | \u2014 | {RESULT} |
| run_lr1e-4_wd0.01 | 1e-4 | 0.01 | \u2014 | {RESULT} |
| run_lr2e-5_wd0.1  | 2e-5 | 0.1  | \u2014 | {RESULT} |

## Sweep Results

<div align="center">

| Run | Learning Rate | Weight Decay | Best Eval F1 |
|---|---|---|---|
| run_lr2e-5_wd0.01 | 2e-5 | 0.01 | 0.827 |
| run_lr5e-5_wd0.01 | 5e-5 | 0.01 | 0.856 |
| run_lr1e-4_wd0.01 | 1e-4 | 0.01 | 0.793 |
| run_lr2e-5_wd0.1  | 2e-5 | 0.1  | 0.741 |

</div>

<p align="center">
  <img width="60%" src="figures/fig2.png">
</p>

The best performing configuration used a learning rate of 5e-5 with weight decay 0.01, achieving the highest F1 score across all sweep runs.

## Usage

```python
from transformers import AutoModelForSequenceClassification, AutoTokenizer

model = AutoModelForSequenceClassification.from_pretrained("SweepBest-TestRepo")
tokenizer = AutoTokenizer.from_pretrained("SweepBest-TestRepo")
```

## License

This model is released under the Apache 2.0 license.