ProPrak01/deberta-v3-financial-sentiment

DeBERTa-v3 Multi-Dataset Financial Sentiment

A financial sentiment model built on DeBERTa-v3-base, fine-tuned jointly on three standard benchmarks: Financial PhraseBank, Twitter Financial News, and FiQA. One shared backbone, two heads -- a 3-class classifier and a continuous regression head.

It beats the current best public multi-dataset financial sentiment model (pmatorras/financial-sentiment-analysis) on two of the three benchmarks, and reports bootstrap confidence intervals plus a continuous MAE/MSE baseline for FiQA that the comparison model doesn't report.

Source code: github.com/ProPrak01/deberta-v3-financial-sentiment Author: Prakash Jha

Results

Evaluated once, on each dataset's locked test split (Twitter has no official test split, so its validation split is used instead -- same as pmatorras's methodology).

Dataset	Macro-F1 (95% CI)	Accuracy (95% CI)	MAE (95% CI)	pmatorras accuracy
Financial PhraseBank (test, n=970)	0.865 (0.839-0.886)	0.871 (0.849-0.890)	--	0.959
Twitter Financial News (validation, n=2388)	0.875 (0.860-0.890)	0.897 (0.885-0.910)	--	0.833
FiQA Sentiment (test, n=234)	0.649 (0.590-0.718)	0.863	0.152 (0.131-0.174)	0.815

Beats pmatorras on Twitter (+6.4 points) and FiQA (+4.8 points), falls short on PhraseBank (87.1% vs 95.9%). More on that gap below.

Architecture

Shared DeBERTa-v3-base backbone, mean-pooled over non-padded tokens, fine-tuned end to end. Two heads sit on top: a 3-class classification head (cross-entropy) trained on PhraseBank and Twitter, and a single-output regression head (MSE loss) trained on FiQA's continuous scores.

Labels are unified across PhraseBank and Twitter as 0=negative, 1=neutral, 2=positive. Twitter's native Bearish/Bullish/Neutral labels get remapped into this scheme.

One thing worth calling out: Twitter has 9,543 training examples, PhraseBank has 3,100, FiQA has 822. A plain round-robin sampler lets the largest dataset dominate every epoch once the smaller ones run out. This model uses a balanced interleaved sampler instead, cycling the smaller datasets so all three contribute through the whole epoch rather than getting crowded out.

Training

Base model: microsoft/deberta-v3-base.

Dataset	Train	Valid	Test	Text field	Label
Financial PhraseBank	3,100	776	970	sentence	3-class, news headlines
Twitter Financial News	9,543	2,388	--	text	3-class, social media
FiQA Sentiment	822	117	234	sentence	continuous score [-1, 1], aspect-based

Hyperparameters: learning rate 1e-5 for the main run, 5e-6 for a later continuation pass; batch size 16; AdamW with eps=1e-6 and weight decay 0.01; 10% linear warmup; max sequence length 128 tokens. Trained 4 epochs, then continued for up to 3 more epochs focused on closing the PhraseBank gap, early-stopped once PhraseBank stopped improving without hurting the other two. Trained on a single T4 GPU (Google Colab).

Checkpoints were kept based on a composite score -- the average of PhraseBank F1, Twitter F1, and a normalized FiQA score (1 / (1 + MAE)) -- rather than raw combined validation loss. Raw loss let PhraseBank's smaller, noisier validation set (776 examples) drive early stopping even when Twitter and FiQA were both still improving.

Evaluation

Each dataset was evaluated once, at the end, on its locked test split. 95% bootstrap confidence intervals (1,000 resamples) are reported on Macro-F1 and MAE, since FiQA's test set is small (234 examples) and a single point estimate isn't enough to claim a real difference between models. FiQA's continuous scores are also bucketed into 3 classes (>0.1 positive, <-0.1 negative, else neutral) so accuracy/F1 can be reported the same way as the other two datasets.

The PhraseBank gap

87.1% vs pmatorras's 95.9% is a real gap, not a bug. The train/test split we used has no meaningful overlap (2 shared sentences out of 4,842 total, checked directly), so it's not a leakage artifact on our end.

Looking at the confusion matrix, the errors are almost entirely at the neutral/positive boundary:

              pred_neg  pred_neu  pred_pos
true_negative:    114        11        0
true_neutral:      19       491       55
true_positive:      4        36      240

Negative and positive are almost never confused with each other -- the mistakes are all adjacent-class, which looks more like genuine linguistic ambiguity (mildly-positive-but-flat headlines) than a model that hasn't learned the task.

Our best guess at the cause: pmatorras's backbone is FinBERT, which was pretrained on financial news text close in style to PhraseBank's formal, 2014-era headlines. That's likely an advantage specific to this dataset. DeBERTa's broader pretraining seems to help more on Twitter and FiQA's informal, modern text, which is exactly where this model wins. We haven't verified this by training a FinBERT baseline ourselves, so treat it as our working theory, not a confirmed finding. We also don't know for certain whether pmatorras used the same PhraseBank train/test split -- some versions on HuggingFace filter by annotator agreement, which changes difficulty.

We did try to close the gap: a continuation run that gave PhraseBank more relative training exposure and a higher loss weight, capped so it couldn't hurt Twitter or FiQA beyond a point. That recovered a small real gain (86.5% to 87.1%) before flattening out -- more epochs on the same data didn't help further, which suggests the remaining gap isn't just a training-budget problem.

Other limitations: English only, no real-time market or price data (this is a text model, not a trading signal), FiQA's test set is small enough that its confidence intervals are wide (0.590-0.718 on Macro-F1), and Twitter's evaluation is on its validation split rather than a true held-out test set.

Intended use

Sentiment classification and continuous scoring for financial news headlines, social media posts about stocks and markets, and aspect-based sentiment on financial statements.

Not intended for automated trading decisions or real-time market prediction, or any use where accuracy needs to exceed what the reported confidence intervals support.

Usage

import torch
import torch.nn as nn
from huggingface_hub import hf_hub_download
from transformers import AutoTokenizer, AutoModel

class MultiTaskDeberta(nn.Module):
    def __init__(self, model_name, num_classes=3):
        super().__init__()
        self.backbone = AutoModel.from_pretrained(model_name)
        hidden = self.backbone.config.hidden_size
        self.dropout = nn.Dropout(0.1)
        self.cls_head = nn.Linear(hidden, num_classes)
        self.reg_head = nn.Linear(hidden, 1)

    def forward(self, input_ids, attention_mask):
        out = self.backbone(input_ids=input_ids, attention_mask=attention_mask)
        mask = attention_mask.unsqueeze(-1).float()
        pooled = (out.last_hidden_state * mask).sum(1) / mask.sum(1).clamp(min=1e-9)
        pooled = self.dropout(pooled)
        return self.cls_head(pooled), self.reg_head(pooled).squeeze(-1)

model_id = "ProPrak01/deberta-v3-financial-sentiment"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = MultiTaskDeberta("microsoft/deberta-v3-base")
weights_path = hf_hub_download(repo_id=model_id, filename="pytorch_model.bin")
model.load_state_dict(torch.load(weights_path, map_location="cpu"))
model.eval()

texts = ["Company reports record profits and strong revenue growth"]
enc = tokenizer(texts, padding=True, truncation=True, max_length=128, return_tensors="pt")
with torch.no_grad():
    cls_logits, reg_score = model(enc["input_ids"], enc["attention_mask"])

label_names = ["negative", "neutral", "positive"]
pred_class = label_names[cls_logits.argmax(-1).item()]
pred_score = reg_score.item()
print(pred_class, pred_score)

Reproducibility

The base training pipeline -- the balanced interleaved sampler, the multi-task model, the training loop, and the bootstrap evaluation -- is in train_and_evaluate.ipynb in the linked repo, run end to end on a T4 GPU with the outputs kept in the notebook.

The published weights here go one step further: after the run in the notebook, a second, shorter continuation phase gave PhraseBank more relative training exposure and a higher loss weight (capped so it couldn't hurt Twitter or FiQA), which is the source of the PhraseBank gain described above. That continuation phase isn't in the notebook, so re-running it as-is gets you a very similar but not identical model -- slightly better on Twitter, slightly behind on PhraseBank and FiQA than the numbers reported here. Both are legitimate results from the same codebase; the numbers on this card are from the better-performing one.

Comparison to pmatorras/financial-sentiment-analysis

	This model	pmatorras
Backbone	DeBERTa-v3-base, 128K vocab, disentangled attention	FinBERT, BERT-base, 30K vocab, 2019
PhraseBank accuracy	87.1%	95.9%
Twitter accuracy	89.7%	83.3%
FiQA accuracy	86.3%	81.5%
FiQA MAE / bootstrap CIs	0.152 (0.131-0.174)	not reported
Regression head	yes	not reported

Citation

If you use this model, please cite the underlying datasets: Financial PhraseBank (Malo et al., 2014), Twitter Financial News Sentiment (zeroshot/twitter-financial-news-sentiment), and FiQA Sentiment (TheFinAI/fiqa-sentiment-classification), along with the base model, DeBERTa-v3 (He et al., 2021).