peft-modernbert-base

This repository contains a PEFT/LoRA adapter, stored in SafeTensors format, together with the tokenizer and associated configuration files for answerdotai/ModernBERT-base, fine-tuned for six-class emotion classification.

This is not a merged standalone ModernBERT checkpoint. The answerdotai/ModernBERT-base model must be loaded separately and combined with the adapter for inference.

The model was developed as part of the Emotional Sentiment Benchmark: Quantised Transformers Pipeline, which compares ModernBERT, RoBERTa and Facebook OPT-350M using a parameter-efficient training and evaluation pipeline.

It achieves the following results on the evaluation set:

• Loss: 0.0410
• Accuracy: 0.9887
• Matthews Correlation: 0.9850
• F1: 0.9760
• Precision: 0.9730
• Recall: 0.9792

Model description

This model uses answerdotai/ModernBERT-base as the underlying transformer architecture and applies Parameter-Efficient Fine-Tuning using LoRA.

The training configuration combines:

• TorchAO 8-bit weight-only quantization of eligible linear layers
• exclusion of the task-specific classification head from quantization
• frozen backbone parameters
• LoRA-based fine-tuning
• class-imbalance interventions during training

The task-specific classification head remains at full precision.

The published Hugging Face repository contains the trained PEFT adapter and the tokenizer used for the model. The base ModernBERT model is obtained separately.

The six target classes are:

Label	Emotion
0	sadness
1	joy
2	love
3	anger
4	fear
5	surprise

Repository contents

This repository provides a complete PEFT adapter package for the trained six-class ModernBERT emotion classifier, including:

• trained PEFT/LoRA adapter weights in SafeTensors format
• PEFT adapter configuration
• tokenizer files and tokenizer configuration
• task and model metadata
• model card and training results

The repository contains the complete task-specific adapter package, but it does not contain the original answerdotai/ModernBERT-base weights.

The ModernBERT base model must therefore be loaded separately and combined with this adapter for inference.

Intended uses & limitations

Intended uses

This model is intended primarily for:

• research and experimentation in text emotion classification
• benchmarking parameter-efficient transformer fine-tuning
• evaluation of resource-conscious NLP pipelines
• comparison of transformer architectures under a common PEFT workflow
• educational and research use involving LoRA and weight-only quantization

Limitations

The model should not be treated as a general-purpose or authoritative emotion detection system.

Important limitations include:

• The training data are derived from tweets and may not generalise to other types of text.
• Human emotions are represented as six discrete categories, which simplifies subjective and context-dependent emotional expression.
• The source data are class-imbalanced, although imbalance mitigation was applied during training.
• High aggregate performance does not guarantee equivalent performance across every class or population subgroup.
• Social-media data may contain demographic, linguistic, cultural and annotation biases inherited from the source datasets.
• Predicted probabilities have not been established as calibrated confidence estimates.
• The model has not been validated for clinical, psychological, employment, legal or other high-stakes uses.
• This repository contains an adapter rather than a complete standalone model. Users must load the compatible ModernBERT base model separately.

Predictions should be interpreted as model classifications rather than objective measurements of an individual's emotional state.

Training and evaluation data

The project uses two tweet-based emotion datasets.

Emotions dataset

The first dataset contains 416,808 tweets with a single integer emotion label.

The six labels are:

• sadness (0)
• joy (1)
• love (2)
• anger (3)
• fear (4)
• surprise (5)

Source:

Emotions Dataset on Kaggle

emotions.csv

The second dataset contains 40,000 tweets with emotional text, tweet ID, author information and a single categorical emotion label.

A subset of this dataset was previously used in a project for Microsoft's Cortana Intelligence Gallery. The dataset was added to Data.World by CrowdFlower on July 15, 2016.

Source:

Emotion Dataset on Data.World

The resulting benchmark corpus contains more than 450,000 source examples.

The data exhibit substantial class imbalance, with approximately a 10:1 difference between the largest and smallest classes. Class imbalance was addressed during model development using a combination of resampling and weighted cross-entropy loss.

Evaluation was performed on held-out data not used for model training.

Users wishing to reproduce or redistribute the datasets should review the licensing and usage conditions provided by the original dataset publishers. The Apache 2.0 licence associated with this model repository does not replace the terms governing the source datasets.

Training procedure

The ModernBERT training pipeline used:

1. TorchAO 8-bit weight-only quantization of eligible linear layers
2. exclusion of the classification head from quantization
3. freezing of backbone parameters
4. LoRA adapter training
5. weighted loss and resampling to address class imbalance

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 0.0001
• train_batch_size: 8
• eval_batch_size: 8
• seed: 42
• gradient_accumulation_steps: 4
• total_train_batch_size: 32
• optimizer: ADAMW_TORCH_FUSED with betas=(0.9, 0.999) and epsilon=1e-08
• lr_scheduler_type: linear
• num_epochs: 1

Training results

Training Loss	Epoch	Step	Validation Loss	Accuracy	Matthews Correlation	F1	Precision	Recall
0.5348	0.1977	1400	0.0968	0.9722	0.9631	0.9568	0.9528	0.9609
0.2975	0.3954	2800	0.0728	0.9808	0.9745	0.9637	0.9559	0.9725
0.2385	0.5931	4200	0.0518	0.9865	0.9821	0.9731	0.9685	0.9780
0.2500	0.7908	5600	0.0443	0.9882	0.9843	0.9752	0.9709	0.9803
0.1968	0.9885	7000	0.0410	0.9887	0.9850	0.9760	0.9730	0.9792

How to use for inference

This repository contains the trained PEFT/LoRA adapter and tokenizer.

The answerdotai/ModernBERT-base model must be loaded separately. To reproduce the benchmark inference configuration, the eligible backbone linear layers are first quantized using TorchAO 8-bit weight-only quantization. The classification head is excluded from quantization and remains at full precision. The trained PEFT adapter is then attached to the quantized base model.

Install dependencies

pip install "transformers==5.2.0" "peft==0.18.1" "torchao==0.16.0" torch

Quantization function

import torch
import torch.nn as nn

from torchao.quantization import quantize_, Int8WeightOnlyConfig


HEAD_PREFIXES = (
    "classifier", "score", "lm_head", "head",
    "qa_outputs", "classification_head", "pre_classifier"
)


def model_quant(
    model: nn.Module,
    device=None,
    head_prefixes=HEAD_PREFIXES
) -> nn.Module:

    general_config = Int8WeightOnlyConfig(version=2)

    def filter_fn(mod: nn.Module, fqn: str) -> bool:
        is_linear = isinstance(mod, nn.Linear)
        is_head = any(
            fqn == p or fqn.startswith(p + ".")
            for p in head_prefixes
        )
        return is_linear and not is_head

    quantize_(
        model,
        general_config,
        filter_fn,
        device=device,
    )

    return model

import torch
import torch.nn as nn

from transformers import AutoTokenizer, AutoModelForSequenceClassification
from peft import PeftModel
from torchao.quantization import quantize_, Int8WeightOnlyConfig


BASE_MODEL_ID = "answerdotai/ModernBERT-base"
ADAPTER_ID = "Wb-az/peft-modernbert-base"

HEAD_PREFIXES = (
    "classifier",
    "score",
)

id2label = {
    0: "sadness",
    1: "joy",
    2: "love",
    3: "anger",
    4: "fear",
    5: "surprise",
}

label2id = {label: idx for idx, label in id2label.items()}


def model_quant(model, device=None, head_prefixes=HEAD_PREFIXES):
    config = Int8WeightOnlyConfig(version=2)

    def filter_fn(mod: nn.Module, fqn: str) -> bool:
        is_linear = isinstance(mod, nn.Linear)
        is_head = any(
            fqn == prefix or fqn.startswith(prefix + ".")
            for prefix in head_prefixes
        )
        return is_linear and not is_head

    quantize_(
        model,
        config,
        filter_fn,
        device=device,
    )

    return model


# Tokenizer is included in this repository
tokenizer = AutoTokenizer.from_pretrained(ADAPTER_ID)

# Load the ModernBERT base model separately
base_model = AutoModelForSequenceClassification.from_pretrained(
    BASE_MODEL_ID,
    num_labels=6,
    id2label=id2label,
    label2id=label2id,
)

device = "cuda" if torch.cuda.is_available() else "cpu"

# Reproduce the benchmark INT8 backbone configuration
base_model = model_quant(
    base_model,
    device=device,
)

# Attach the trained PEFT/LoRA adapter
model = PeftModel.from_pretrained(
    base_model,
    ADAPTER_ID,
)

model.eval()

Run a prediction

text = "I can't believe it worked, I'm so happy!"

inputs = tokenizer(
    text,
    return_tensors="pt",
    truncation=True,
    padding=True,
)

model_device = next(model.parameters()).device

inputs = {
    key: value.to(model_device)
    for key, value in inputs.items()
}

with torch.no_grad():
    outputs = model(**inputs)

predicted_id = outputs.logits.argmax(
    dim=-1
).item()

print(id2label[predicted_id])

Note: Quantization is optional for inference.
If you do not want to use the TorchAO INT8 configuration, omit the quantization step and attach the PEFT adapter directly to the full-precision answerdotai/ModernBERT-base classification model.

Load ModernBERT-base | v Create six-class classification model | v [Optional] Apply TorchAO INT8 weight-only quantization to eligible backbone Linear layers | v Keep task-specific head at full precision | v Attach Wb-az/peft-modernbert-base adapter | v Run inference using the included tokenizer