peft-opt-350m

This repository contains a PEFT/LoRA adapter for facebook/opt-350m, fine-tuned for six-class emotion classification.

The adapter was developed as part of the Emotional Sentiment Benchmark: Quantised Transformers Pipeline, which compares ModernBERT, RoBERTa and OPT-350M under a common parameter-efficient training and evaluation framework.

It achieves the following results on the evaluation set:

• Loss: 0.1054
• Accuracy: 0.9808
• Matthews Correlation: 0.9744
• F1: 0.9654
• Precision: 0.9620
• Recall: 0.9691

Model description

This model adapts facebook/opt-350m, a decoder-only transformer, for six-class sequence classification using LoRA.

The benchmark configuration uses:

• TorchAO 8-bit weight-only quantization of eligible backbone linear layers
• exclusion of the task-specific classification head from quantization
• frozen backbone parameters
• LoRA-based parameter-efficient fine-tuning
• class-imbalance interventions during training

The six target classes are:

Label	Emotion
0	sadness
1	joy
2	love
3	anger
4	fear
5	surprise

For the full benchmark methodology and implementation details, see:

peft-modernbert-base,

Repository contents

This repository provides the task-specific PEFT adapter package for the OPT-350M emotion classifier.

The original facebook/opt-350m base-model weights are not included and must be loaded separately.

Intended uses & limitations

This adapter is intended for:

• emotion-classification research
• PEFT and LoRA experimentation
• comparison of decoder-only and encoder-based transformer architectures
• resource-conscious transformer benchmarking

It is not intended as an authoritative measure of human emotional state or for clinical, psychological, employment, legal or other high-stakes use.

OPT also inherits limitations and potential biases from its underlying facebook/opt-350m base model.

For the shared benchmark limitations, dataset considerations and evaluation methodology, see the companion ModernBERT model card: Wb-az/peft-modernbert-base

Training and evaluation data

OPT-350M was trained and evaluated using the same prepared dataset, class labels, train/evaluation framework and imbalance interventions used for the ModernBERT benchmark.

The benchmark combines two tweet-based emotion datasets containing more than 450,000 source examples across:

• sadness
• joy
• love
• anger
• fear
• surprise

Full dataset provenance, source links, class-imbalance information and usage considerations are documented here:

ModernBERT model card – Training and evaluation data Wb-az/peft-modernbert-base

Training procedure

OPT-350M used the same common benchmark training configuration as ModernBERT.

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 0.0001
• train_batch_size: 8
• eval_batch_size: 8
• seed: 42
• gradient_accumulation_steps: 4
• total_train_batch_size: 32
• optimizer: Use OptimizerNames.ADAMW_TORCH_FUSED with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
• lr_scheduler_type: linear
• num_epochs: 1

Training results

Training Loss	Epoch	Step	Validation Loss	Accuracy	Matthews Correlation	F1	Precision	Recall
4.3269	0.1977	1400	0.5872	0.8729	0.8336	0.8248	0.8057	0.8566
3.3421	0.3954	2800	0.4799	0.9599	0.9469	0.9387	0.9298	0.9494
1.3997	0.5931	4200	0.2904	0.9703	0.9608	0.9487	0.9502	0.9508
1.1424	0.7908	5600	0.1853	0.9762	0.9684	0.9577	0.9495	0.9688
0.8520	0.9885	7000	0.1054	0.9808	0.9744	0.9654	0.9620	0.9691

Framework versions

• Transformers 5.2.0
• PyTorch 2.10.0+cu128
• TorchAO 0.16.0
• Datasets 4.0.0
• Tokenizers 0.22.2

Full project

Source code, shared methodology, statistical analysis and cross-model benchmark results are available at:

Wb-az/peft-qlora-text-classification

How to use for inference

The OPT model is distributed as a PEFT/LoRA adapter and is combined with the facebook/opt-350m base model at inference time.

The project loader reads the required base checkpoint from the PEFT configuration, constructs the sequence-classification model, optionally applies the TorchAO INT8 configuration, and then attaches the trained adapter.

def load_seqcls_with_adapter(
    adapter_dir,
    num_labels,
    id2label,
    label2id,
    device,
    quantized=False,
):
    peft_cfg = PeftConfig.from_pretrained(adapter_dir)
    base_ckpt = peft_cfg.base_model_name_or_path

    base = seq_class_init(
        base_ckpt,
        num_labels,
        id2label,
        label2id,
        device,
        quantized=quantized,
    )

    # Basic stability adjustment used for OPT
    for name, module in base.named_modules():
        if isinstance(module, nn.Linear):
            module.to(dtype)

    model = PeftModel.from_pretrained(
        base,
        adapter_dir,
    )

    tokenizer = build_tokenizer(
        adapter_dir
    )
    return model, tokenizer, base_ckpt

# For the quantized benchmark configuration:
model, tokenizer, base_ckpt = load_seqcls_with_adapter(
    "Wb-az/peft-opt-350m",
    num_labels=6,
    id2label=id2label,
    label2id=label2id,
    device=device,
    quantized=True, # For full-precision inference quantized=False
)
  
model.eval()

# Example

text = "I can't believe it worked, I'm so happy!"
inputs = tokenizer(
    text,
    return_tensors="pt",
    truncation=True,
    padding=True,
)

inputs = {
    key: value.to(device)
    for key, value in inputs.items()
}

with torch.no_grad():
    logits = model(**inputs).logits

predicted_id = logits.argmax(dim=-1).item()
print(id2label[predicted_id])