---
library_name: peft
license: mit
base_model: roberta-base
pipeline_tag: text-classification
tags:
- base_model:adapter:roberta-base
- lora
- peft
- transformers
- torchao
- safetensors
- text-classification
- emotion-classification
- int8
metrics:
- accuracy
- matthews_correlation
- f1
- precision
- recall
model-index:
- name: peft-roberta-base
  results:
  - task:
      type: text-classification
      name: Emotion Classification
    dataset:
      type: custom
      name: Combined Emotions Datasets
    metrics:
    - type: accuracy
      value: 0.9745
      name: Accuracy
    - type: matthews_correlation
      value: 0.9662
      name: Matthews Correlation
    - type: f1
      value: 0.9609
      name: F1
    - type: precision
      value: 0.9550
      name: Precision
    - type: recall
      value: 0.9671
      name: Recall
---

# peft-roberta-base

This repository contains a PEFT/LoRA adapter for
[roberta-base](https://huggingface.co/roberta-base),
fine-tuned for six-class emotion classification.

The model was developed as part of the
[Emotional Sentiment Benchmark: Quantised Transformers Pipeline](https://github.com/Wb-az/peft-qlora-text-classification),
which compares ModernBERT, RoBERTa and Facebook OPT-350M under a common
parameter-efficient training and evaluation framework.

It achieves the following results on the evaluation set:

- Loss: 0.0933
- Accuracy: 0.9745
- Matthews Correlation: 0.9662
- F1: 0.9609
- Precision: 0.9550
- Recall: 0.9671

## Model description

This model adapts `roberta-base` for six-class sequence classification using
LoRA-based Parameter-Efficient Fine-Tuning.

The same training and inference methodology was applied across the three
benchmark architectures: ModernBERT, RoBERTa and Facebook OPT-350M.

The shared configuration uses:

- TorchAO 8-bit weight-only quantization of eligible backbone linear layers
- exclusion of the task-specific classification head from quantization
- frozen backbone parameters
- LoRA-based fine-tuning
- class-imbalance interventions during training
- the same inference reconstruction order

The adapter weights are stored in SafeTensors format.

For the full shared methodology and implementation details, see:

[ModernBERT model card](https://huggingface.co/Wb-az/peft-modernbert-base/blob/main/README.md)

## Repository contents

This repository provides the task-specific PEFT adapter package for the
RoBERTa emotion classifier.

The original `roberta-base` weights are not included and must be loaded
separately when reconstructing the classifier.

## Intended uses & limitations

This adapter is intended primarily for:

- emotion-classification research
- PEFT and LoRA experimentation
- comparison of transformer architectures
- resource-conscious NLP benchmarking

It is not intended as an authoritative measure of human emotional state or for
clinical, psychological, employment, legal or other high-stakes applications.

The model also inherits limitations and potential biases from the underlying
`roberta-base` model and the tweet-based training data.

For the shared benchmark limitations and evaluation considerations, see:

[ModernBERT model card](https://huggingface.co/Wb-az/peft-modernbert-base/blob/main/README.md)

## Training and evaluation data

RoBERTa was trained and evaluated using the same prepared benchmark dataset as
the ModernBERT model.

The benchmark combines two tweet-based emotion datasets covering six classes:
sadness, joy, love, anger, fear and surprise. The combined corpus contains more
than 450,000 source examples and includes substantial class imbalance.

For full dataset provenance, source links, label mapping and preparation
details, see:

[ModernBERT model card](https://huggingface.co/Wb-az/peft-modernbert-base/blob/main/README.md)


## Training procedure

### Training hyperparameters

The following hyperparameters were used during training:
- learning_rate: 0.0001
- train_batch_size: 8
- eval_batch_size: 8
- seed: 42
- gradient_accumulation_steps: 4
- total_train_batch_size: 32
- optimizer: Use OptimizerNames.ADAMW_TORCH_FUSED with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
- lr_scheduler_type: linear
- num_epochs: 1

### Training results

| Training Loss | Epoch  | Step | Validation Loss | Accuracy | Matthews Correlation | F1     | Precision | Recall |
|:-------------:|:------:|:----:|:---------------:|:--------:|:--------------------:|:------:|:---------:|:------:|
| 1.1892        | 0.1977 | 1400 | 0.2318          | 0.9216   | 0.8977               | 0.8968 | 0.8726    | 0.9270 |
| 0.6786        | 0.3954 | 2800 | 0.1395          | 0.9621   | 0.9497               | 0.9438 | 0.9332    | 0.9567 |
| 0.6029        | 0.5931 | 4200 | 0.1098          | 0.9696   | 0.9597               | 0.9558 | 0.9491    | 0.9629 |
| 0.5632        | 0.7908 | 5600 | 0.0951          | 0.9742   | 0.9658               | 0.9602 | 0.9539    | 0.9672 |
| 0.5216        | 0.9885 | 7000 | 0.0933          | 0.9745   | 0.9662               | 0.9609 | 0.9550    | 0.9671 |


### Framework versions

- PEFT 0.18.1
- Transformers 5.2.0
- Pytorch 2.10.0+cu128
- Datasets 4.0.0
- Tokenizers 0.22.2


## How to use for inference

## How to use for inference

The RoBERTa adapter is reconstructed using the same inference pipeline used
across the ModernBERT, RoBERTa and OPT-350M benchmark models.

The PEFT configuration is first used to identify the original base model.
A six-class sequence-classification model is then created. If quantized
inference is requested, TorchAO quantization is applied before the PEFT adapter
is attached.

### Quantization

When `quantized=True`, the pipeline uses `Int8WeightOnlyConfig(version=2)` and
applies TorchAO quantization to eligible `nn.Linear` layers. Classification and
other task-specific head modules are excluded using the shared `HEAD_PREFIXES`
filter.

The same quantization methodology was used across the benchmark architectures.

> **Note:** Quantization is optional for inference. Set `quantized=False` to
> omit the TorchAO INT8 step and attach the PEFT adapter to the non-quantized
> base model.

### Example

```python
from peft import PeftConfig, PeftModel

ADAPTER_ID = "Wb-az/peft-roberta-base"

peft_cfg = PeftConfig.from_pretrained(ADAPTER_ID)
base_ckpt = peft_cfg.base_model_name_or_path

base = seq_class_init(
    base_ckpt,
    num_labels=6,
    id2label=id2label,
    label2id=label2id,
    device=device,
    quantized=True,
)

model = PeftModel.from_pretrained(
    base,
    ADAPTER_ID,
)

tokenizer = build_tokenizer(ADAPTER_ID)

model.eval()

```

The shared implementation is available in
[hf_pipeline.py](https://github.com/Wb-az/peft-qlora-text-classification/blob/main/utilities/hf_pipeline.py)