File size: 13,505 Bytes
dfd384d | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 269 270 271 272 273 274 275 276 277 278 279 280 281 282 283 284 285 286 287 288 289 290 291 292 293 294 295 296 297 298 299 300 301 302 303 304 305 306 307 308 309 310 311 312 313 314 315 316 317 318 319 320 321 322 323 324 325 326 327 328 329 330 331 332 333 334 335 336 337 338 339 340 341 342 343 344 345 346 347 348 349 350 351 352 353 354 355 356 357 358 359 360 361 362 363 364 365 366 367 368 369 370 371 372 373 374 375 376 377 378 379 380 381 382 383 384 385 386 387 388 389 390 391 392 393 394 395 396 397 398 399 400 401 402 403 404 405 406 407 408 409 410 411 412 413 414 415 416 417 418 419 420 421 422 423 424 425 426 427 428 429 430 431 432 433 434 435 436 437 438 439 440 441 442 443 444 445 446 447 448 449 450 451 452 453 454 455 456 457 458 459 460 461 462 463 464 465 466 467 468 469 470 471 472 473 474 475 476 477 478 479 480 481 482 483 484 485 486 487 488 489 490 491 492 493 494 495 496 497 498 499 500 501 502 503 504 505 506 507 508 509 510 511 512 513 514 515 516 517 518 519 520 521 522 523 524 525 526 527 528 529 530 | ---
language:
- en
library_name: transformers
license: mit
tags:
- sentiment-analysis
- emotion-classification
- text-classification
- multi-label-classification
- deberta-v3
- nlp
- pytorch
pipeline_tag: text-classification
---
# Sentimental Analyzer V3
**Sentimental Analyzer V3** is a multi-task Natural Language Processing (NLP) model designed to analyze English text through two complementary tasks:
1. **5-class sentiment classification**
2. **28-class multi-label emotion classification**
The model uses `microsoft/deberta-v3-base` as its transformer backbone and includes a frozen production calibration pipeline for emotion predictions.
## Model Overview
| Property | Value |
|---|---|
| Model | Sentimental Analyzer V3 |
| Backbone | `microsoft/deberta-v3-base` |
| Framework | PyTorch |
| Maximum sequence length | 256 |
| Sentiment classes | 5 |
| Emotion classes | 28 |
| Sentiment task | Single-label classification |
| Emotion task | Multi-label classification |
| Production calibration | `HYBRID_ISOTONIC_GLOBAL` |
| Calibrated emotion classes | 11 |
| Raw emotion classes | 17 |
| Global emotion threshold | 0.35 |
| Fallback strategy | `highest_calibrated_probability` |
## What the Model Does
Given an English text or comment, the model produces:
- One sentiment class from five possible sentiment categories.
- One or more emotion labels from 28 possible emotion categories, with a fallback selection when no class crosses the production threshold.
- A probability distribution over the five sentiment classes.
- A probability for each of the 28 emotion classes.
The model is designed for text analytics, exploratory NLP, research, educational demonstrations, and applications that analyze user feedback.
## Sentiment Classification
The model predicts one of five sentiment categories:
| ID | Sentiment |
|---:|---|
| 0 | Negative |
| 1 | Neutral |
| 2 | Positive |
| 3 | Very Negative |
| 4 | Very Positive |
Sentiment classification is a single-label classification task. The sentiment probabilities are normalized using softmax.
## Emotion Classification
The model performs multi-label emotion classification.
| ID | Emotion |
|---:|---|
| 0 | admiration |
| 1 | amusement |
| 2 | anger |
| 3 | annoyance |
| 4 | approval |
| 5 | caring |
| 6 | confusion |
| 7 | curiosity |
| 8 | desire |
| 9 | disappointment |
| 10 | disapproval |
| 11 | disgust |
| 12 | embarrassment |
| 13 | excitement |
| 14 | fear |
| 15 | gratitude |
| 16 | grief |
| 17 | joy |
| 18 | love |
| 19 | nervousness |
| 20 | neutral |
| 21 | optimism |
| 22 | pride |
| 23 | realization |
| 24 | relief |
| 25 | remorse |
| 26 | sadness |
| 27 | surprise |
Multiple emotions can be associated with the same text.
## Architecture
The model uses `microsoft/deberta-v3-base` as the transformer encoder.
```text
Input Text
|
v
DeBERTa-v3-base Encoder
|
v
Attention Pooling
|
v
Shared Representation
/ \
/ \
v v
Sentiment Head Emotion Head
| |
v v
5 Sentiments 28 Emotions
```
### Attention Pooling
The sequence-level hidden representations produced by DeBERTa are aggregated using learned attention pooling. Padding tokens are masked before the attention softmax so that padded positions do not contribute to the pooled representation.
### Shared Representation
The pooled representation is passed through:
- Linear projection
- GELU activation
- Layer normalization
- Dropout
The resulting representation is shared by the two task-specific prediction heads.
### Sentiment Head
The sentiment head produces five logits. Softmax is applied during inference to obtain the sentiment probability distribution.
### Emotion Head
The emotion head produces 28 logits. Sigmoid probabilities are used because emotion classification is multi-label.
## Production Calibration
The production emotion pipeline uses the frozen strategy:
`HYBRID_ISOTONIC_GLOBAL`
```text
Raw Emotion Logits
|
v
Sigmoid Probabilities
|
v
Class-wise Isotonic Calibration
|
v
Global Threshold = 0.35
|
v
Final Emotion Labels
```
### Calibration Configuration
- Calibration strategy: `HYBRID_ISOTONIC_GLOBAL`
- Calibrated emotion classes: 11
- Raw emotion classes: 17
- Minimum calibration support: 10
- Global threshold: 0.35
- Fallback: `highest_calibrated_probability`
The following 11 emotion classes use isotonic calibration:
- admiration
- anger
- annoyance
- approval
- disappointment
- disapproval
- disgust
- excitement
- joy
- love
- neutral
The following 17 rare classes retain their raw sigmoid probabilities:
- amusement
- caring
- confusion
- curiosity
- desire
- embarrassment
- fear
- gratitude
- grief
- nervousness
- optimism
- pride
- realization
- relief
- remorse
- sadness
- surprise
Calibration parameters and the threshold were selected using validation data. The final test set was not used for calibration or threshold selection.
## Evaluation
The final V3 checkpoint was evaluated on an untouched real-world test split containing **253 examples**.
### Sentiment Results
| Metric | Score |
|---|---:|
| Accuracy | 0.8024 |
| Macro F1 | 0.6499 |
| Weighted F1 | 0.7864 |
### Emotion Results
| Metric | Score |
|---|---:|
| Micro F1 | 0.5770 |
| Macro F1 | 0.3026 |
| Weighted F1 | 0.5968 |
| Macro Precision | 0.2791 |
| Macro Recall | 0.4198 |
These results correspond to the frozen V3 checkpoint and production calibration configuration.
## Evaluation Interpretation
The sentiment task shows stronger performance than the 28-class emotion task.
The emotion task is more challenging because:
- It is multi-label.
- Some emotions are substantially rarer than others.
- Emotion boundaries can be subjective.
- Multiple emotions can coexist in the same text.
- Some classes have limited calibration support.
Macro-level emotion metrics are particularly sensitive to performance on rare classes.
## Dataset and Labeling Limitation
The real-world adaptation dataset used during development contains **LLM/pseudo-labeled examples rather than a fully human-annotated gold-standard dataset**.
Therefore, the reported real-world adaptation results should be interpreted as evaluation against the project's labeling methodology rather than definitive human-gold benchmark performance.
The model may perform differently on text that differs substantially from the adaptation data, including different domains, writing styles, topics, vocabulary, slang, cultural contexts, levels of context, or forms of sarcasm and implicit meaning.
## Intended Use
Sentimental Analyzer V3 is intended for:
- Sentiment analysis of English comments and short text.
- Emotion analysis of user feedback.
- NLP research and experimentation.
- Educational demonstrations.
- Exploratory analysis of text datasets.
- Aggregate analysis of customer or user feedback.
- Integration into applications through an inference API.
The model can be used as an inference component behind a web application or API.
## Limitations
This model should not be considered a perfect representation of human emotion.
Important limitations include:
- Emotion classification can be subjective.
- Some emotion classes are substantially rarer than others.
- Multi-label emotion annotation can be difficult to validate consistently.
- Sarcasm and implicit meaning may be challenging.
- Domain shift may reduce performance.
- Rare emotion classes have limited calibration support.
- The real-world adaptation labels are not fully human-gold.
- The model has primarily been developed and evaluated for English text.
- Model predictions are statistical estimates and should not be treated as objective facts about a person's feelings.
- Predictions should not be interpreted as psychological, medical, or clinical assessments.
## Example
### Input
```text
I absolutely love this product! It is amazing.
```
### Example Production Output
```json
{
"sentiment": "Very Positive",
"sentiment_confidence": 0.987,
"emotions": [
"joy"
]
}
```
The exact probability values depend on the frozen checkpoint and production calibration artifacts.
## Model Artifacts
```text
Sentimental-Analyzer-V3/
│
├── README.md
├── config.json
│
├── model/
│ └── best_v3_model.pt
│
├── calibration/
│ └── v3_isotonic_calibrators.pkl
│
└── labels/
├── sentiment_labels.json
└── emotion_labels.json
```
### Artifact Descriptions
- `model/best_v3_model.pt`: Frozen V3 PyTorch checkpoint containing the trained model parameters.
- `calibration/v3_isotonic_calibrators.pkl`: Frozen emotion calibration parameters used by the production inference pipeline.
- `labels/sentiment_labels.json`: Mapping between sentiment class IDs and sentiment labels.
- `labels/emotion_labels.json`: Mapping between emotion class IDs and emotion labels.
- `config.json`: Frozen production configuration containing model, class, calibration, decision, selection, and checkpoint integrity information.
## Checkpoint Integrity
The published checkpoint corresponds to the frozen V3 production candidate.
SHA-256:
```text
4d3f5a97d352460d41fa884fd337b2de6f58549139135cad1b4b79beefcb2078
```
The checkpoint was verified against this SHA-256 hash before publication.
## Frozen Production Configuration
```text
Version:
V3_FINAL_CANDIDATE
Model:
microsoft/deberta-v3-base
Maximum sequence length:
256
Sentiment classes:
5
Emotion classes:
28
Calibration strategy:
HYBRID_ISOTONIC_GLOBAL
Calibrated emotion classes:
11
Raw emotion classes:
17
Minimum calibration support:
10
Global emotion threshold:
0.35
Fallback:
highest_calibrated_probability
Selection dataset:
validation
Test used for selection:
false
Threshold source:
validation
Calibration source:
validation
```
No additional training or calibration is performed during production inference.
## Inference Behavior
```text
Input text
|
v
Tokenization
|
v
DeBERTa-v3-base
|
v
Shared representation
|
+-------------------------+
| |
v v
Sentiment logits Emotion logits
| |
v v
Softmax Sigmoid
| |
v v
Sentiment label Frozen calibration
|
v
Threshold = 0.35
|
v
Emotion labels
```
The model is loaded in evaluation mode and its parameters are frozen during inference.
## API Integration
The model can be exposed through a FastAPI service.
A production API can provide:
```text
POST /predict
```
Example request:
```json
{
"text": "I absolutely love this product!"
}
```
Example response:
```json
{
"sentiment": "Very Positive",
"sentiment_confidence": 0.98,
"sentiment_probabilities": {
"Negative": 0.00,
"Neutral": 0.00,
"Positive": 0.01,
"Very Negative": 0.00,
"Very Positive": 0.98
},
"emotions": [
"joy"
],
"emotion_probabilities": {
"joy": 0.76
}
}
```
The API layer can be deployed separately from this model repository so that API users do not need to download the model checkpoint directly.
## Reproducibility
The V3 production candidate is frozen.
The following components are fixed:
- Model checkpoint
- Model architecture
- Tokenizer configuration
- Maximum sequence length
- Sentiment label mapping
- Emotion label mapping
- Emotion ordering
- Emotion calibration artifacts
- Global emotion threshold
- Fallback strategy
- Production configuration
The checkpoint can be verified using the published SHA-256 hash.
## Responsible Use
This model produces statistical predictions from textual patterns learned during training and adaptation.
Its outputs should not be presented as certain facts about:
- A person's actual emotional state
- A person's intentions
- A person's personality
- A person's psychological condition
- A person's medical or clinical status
Applications using this model should provide appropriate context and should avoid presenting model predictions as definitive judgments.
## Project Status
**V3 production candidate — frozen.**
The model, calibration configuration, and production inference behavior have been verified before publication.
Training and calibration are not performed during production inference.
## License
MIT
|