Mamba3 GoEmotions

Overview

Model fine-tuned from Mamba3-SISO-187M on the GoEmotions dataset for multi-label emotion classification.

The model uses a custom classifier developed by me specifically for Mamba3.

Classifier implementation: GitHub repository.

Dataset

GoEmotions is a dataset of Reddit comments annotated with 27 emotion categories and neutral. Each sample can have one or multiple emotion labels. A threshold of 0.5 is used for prediction. For more details, see the original paper.

Training

The model was fine-tuned using the following configuration:

Parameter Value
Pretrained model Mamba3-SISO-187M
Tokenizer Llama 3.1-8B
Classification head Custom RoBERTa-style head
Pooling Last non-padding token
Learning rate 4.38 × 10⁻⁵
Weight decay 0.01497
Batch size 16
Warmup ratio 0.01828
Learning-rate scheduler Cosine
Maximum gradient norm 1.0
Dropout 0.1491
Positive-weight clamp 10.0
Maximum sequence length 256
Random seed 42
Decision threshold 0.5

Usage

Clone Classifier implementation and install requirements.txt.

import torch
from src.model_loader import load_model

model = load_model("FeDuck113/mamba3-go_emotions")
predictions = model.predict("I am so grateful for your help!")
print(predictions)

Fine-tuning

The loaded model can be fine-tuned with the Hugging Face Trainer. This example continues training on GoEmotions

import torch
from transformers import Trainer, TrainingArguments

from src.model_loader import load_model
from src.utils.load_datasets import load_go_emotions

model = load_model("FeDuck113/mamba3-go_emotions")
tokenizer = model.tokenizer
train_ds, val_ds, _, pos_weight, _ = load_go_emotions(
    model.tokenize_batch, num_labels=model.num_labels, return_label_names=True
)
model.pos_weight = torch.as_tensor(pos_weight, dtype=torch.float32).clamp(max=10.0)

training_args = TrainingArguments(...)
trainer = Trainer(...)
trainer.train()

Metrics

Overall results

Split Examples Subset accuracy Macro precision Macro recall Macro F1 Micro F1 Macro AP Hamming loss
Validation 5,426 0.2326 0.3935 0.6752 0.4857 0.5297 0.5008 0.0567
Test 5,427 0.2091 0.3767 0.6649 0.4700 0.5225 0.4849 0.0579

Сomparison

Model Precision Recall F1 Source
Mamba3 GoEmotions (this work) 37.67 66.49 47.00 This repository
RoBERTa GoEmotions 57.52 39.62 45.03 SamLowe/roberta-base-go_emotions
UltraBERT GoEmotions 65.67 42.98 49.67 cirimus/ultrabert-380M-go-emotions

Per-class test results

Emotion Support Precision Recall F1 AP
admiration 504 0.4619 0.8671 0.6028 0.7260
amusement 264 0.6295 0.9205 0.7477 0.8060
anger 198 0.2762 0.6515 0.3880 0.3955
annoyance 320 0.2032 0.6750 0.3124 0.3202
approval 351 0.2479 0.6695 0.3618 0.4344
caring 135 0.2877 0.6222 0.3934 0.3634
confusion 153 0.2359 0.7386 0.3576 0.4011
curiosity 284 0.4091 0.8873 0.5600 0.5123
desire 83 0.4128 0.5422 0.4688 0.4513
disappointment 151 0.1555 0.4768 0.2345 0.2523
disapproval 267 0.2600 0.6816 0.3764 0.3597
disgust 123 0.3255 0.6748 0.4392 0.4985
embarrassment 37 0.4500 0.4865 0.4675 0.4045
excitement 103 0.2651 0.5534 0.3585 0.4209
fear 78 0.4745 0.8333 0.6047 0.7170
gratitude 352 0.7557 0.9403 0.8380 0.9553
grief 6 0.5000 0.5000 0.5000 0.3611
joy 161 0.3296 0.7329 0.4547 0.5408
love 238 0.6324 0.9034 0.7439 0.8430
nervousness 23 0.1739 0.3478 0.2319 0.2009
optimism 186 0.3664 0.6559 0.4701 0.5719
pride 16 0.4286 0.3750 0.4000 0.2552
realization 145 0.2090 0.3517 0.2622 0.2213
relief 11 0.2000 0.3636 0.2581 0.1376
remorse 56 0.5426 0.9107 0.6800 0.6147
sadness 156 0.3567 0.7500 0.4835 0.5773
surprise 141 0.3731 0.6879 0.4838 0.5332
neutral 1,787 0.5840 0.8170 0.6811 0.7027
Downloads last month
13
Safetensors
Model size
0.2B params
Tensor type
F32
·
BF16
·
Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support

Dataset used to train FeDuck113/mamba3-go_emotions