Ido-shraga/mmbert-sentiment-lora

mmBERT per-language sentiment LoRA adapters (stage 2)

Per-language LoRA + classification head adapters trained on top of the frozen stage-1 checkpoint Ido-shraga/mmbert-base-sentiment-adapted (mmBERT-base fully fine-tuned on combined 3-class tweet sentiment). Base weights are frozen; each adapter trains LoRA (r=16, α=32, dropout 0.1) on Wqkv/Wo/Wi plus the classification head (modules_to_save), ~1.3% of parameters.

All adapters share one label space: negative / neutral / positive.

Adapters and test results

Adapter (folder)	Training data	Test macro-F1	Test accuracy
en-super	super_tweeteval tweet_sentiment (5-point scale folded to 3-class), 26.6K rows	0.6299	0.6432
arabic	tweet_sentiment_multilingual arabic	0.6388	0.6322
english	tweet_sentiment_multilingual english	0.6920	0.6885
french	tweet_sentiment_multilingual french	0.7023	0.7023
german	tweet_sentiment_multilingual german	0.7451	0.7448
hindi	tweet_sentiment_multilingual hindi	0.5204	0.5195
italian	tweet_sentiment_multilingual italian	0.6380	0.6345
portuguese	tweet_sentiment_multilingual portuguese	0.6954	0.6943
spanish	tweet_sentiment_multilingual spanish	0.6616	0.6586

Notes:

• en-super scores are on the folded 3-class labels and are not comparable to the official SuperTweetEval leaderboard metric (1 − MAE^M on the 5-point scale).
• hindi is the clear outlier (0.52 vs 0.63–0.75); the other languages cluster around 0.63–0.75.

Hyperparameters (per adapter)

• en-super: 2 epochs (1,666 steps); others: 5 epochs (290 steps)
• batch 32, max length 128, bf16, lr 1e-4 linear, LoRA r=16 / α=32 / dropout 0.1, bias="none"
• Attention: PyTorch SDPA (flash-attention kernels) on A10G

Usage

import torch
from transformers import AutoModelForSequenceClassification, AutoTokenizer
from peft import PeftModel

base = AutoModelForSequenceClassification.from_pretrained(
    "Ido-shraga/mmbert-base-sentiment-adapted", dtype=torch.bfloat16
)
model = PeftModel.from_pretrained(
    base, "Ido-shraga/mmbert-sentiment-lora", subfolder="german"
)
tok = AutoTokenizer.from_pretrained("Ido-shraga/mmbert-base-sentiment-adapted")

inputs = tok("Das neue Update ist wirklich großartig!", return_tensors="pt")
print(model(**inputs).logits.argmax(-1))  # tensor: 2 (positive)

References

• mmBERT: arXiv 2509.06888 · SuperTweetEval: arXiv 2310.14757