emotion-distilbert

A fine-tuned DistilBERT model that classifies English text into one of six emotions: joy, sadness, anger, fear, surprise, disgust.

It was trained on GoEmotions (Reddit comments), with the 27 fine-grained labels mapped to the 6 Ekman emotions.

Usage

from transformers import pipeline

clf = pipeline("text-classification", model="mai-data-ai/emotion-distilbert", top_k=None)
print(clf("I am so happy and grateful for everything you did for me!"))

Or with the model directly:

import torch
from transformers import AutoTokenizer, AutoModelForSequenceClassification

name = "mai-data-ai/emotion-distilbert"
tok = AutoTokenizer.from_pretrained(name)
model = AutoModelForSequenceClassification.from_pretrained(name).eval()

enc = tok("Stop wasting my time, this is so annoying", return_tensors="pt",
          truncation=True, max_length=64)
with torch.no_grad():
    probs = torch.softmax(model(**enc).logits, dim=-1)[0]
print({model.config.id2label[i]: round(p.item(), 3) for i, p in enumerate(probs)})

Data

GoEmotions (simplified config) labels were mapped to six Ekman emotions:

Ekman emotion GoEmotions labels
anger anger, annoyance, disapproval
disgust disgust
fear fear, nervousness
joy joy, amusement, approval, excitement, gratitude, love, optimism, relief, pride, admiration, desire, caring
sadness sadness, disappointment, embarrassment, grief, remorse
surprise surprise, realization, confusion, curiosity

Only comments whose labels map to exactly one Ekman emotion were kept. Comments with mixed emotions, and those labeled only neutral, were dropped.

Split Comments
Train 28,104
Validation 3,527
Test 3,539

Train class distribution: joy 15,809 · anger 4,590 · surprise 4,160 · sadness 2,455 · fear 566 · disgust 524.

Training

  • Base model: distilbert-base-uncased
  • Loss: cross-entropy with class weights (balanced), to compensate for the imbalance
  • Learning rate 2e-5, batch size 32, 4 epochs, weight decay 0.01, 350 warmup steps
  • Max sequence length 64, seed 42
  • Early stopping (patience 2); best checkpoint chosen by validation macro-F1
  • Trained with the Hugging Face Trainer

Results

Test set (3,539 comments):

Metric Score
Accuracy 0.7934
Macro F1 0.6962
Emotion Precision Recall F1 Support
joy 0.937 0.854 0.894 1,937
sadness 0.615 0.674 0.643 298
anger 0.675 0.704 0.689 609
fear 0.625 0.765 0.688 85
surprise 0.675 0.771 0.720 533
disgust 0.490 0.610 0.543 77

Comparison with models trained from scratch (GloVe 100d)

Model Accuracy Macro F1
DistilBERT (this model) 0.7934 0.6962
BiLSTM + Attention (trainable embeddings) 0.7426 0.6526
LSTM (trainable embeddings) 0.7251 0.6359
GRU (trainable embeddings) 0.7321 0.6289
GRU (frozen embeddings) 0.7098 0.6164
BiLSTM + Attention (frozen embeddings) 0.7056 0.6164
LSTM (frozen embeddings) 0.6917 0.5867

Limitations

  • Rare classes are weaker. Disgust (524 training examples) is the hardest class; fear is also small. Class weights trade precision for recall on these classes.
  • Domain. Trained on English Reddit comments; performance may drop on other styles (formal text, other languages, long documents).
  • No neutral class. The model always picks one of the six emotions, even for text with no clear emotion.
  • Label mapping. Grouping 27 fine-grained labels into 6 emotions is a design choice (e.g. "approval" counts as joy), and the model inherits the annotation noise of GoEmotions.
  • Single run. Results come from one run per model, so small gaps between models should not be over-interpreted.
  • Not intended for high-stakes decisions about individuals.

Author

Mai Reda — GitHub · LinkedIn

Downloads last month
-
Safetensors
Model size
67M params
Tensor type
F32
·
Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support

Model tree for mai-data-ai/emotion-distilbert

Finetuned
(12,888)
this model

Dataset used to train mai-data-ai/emotion-distilbert

Evaluation results