Mental-Health RoBERTa Classifier (ONNX int8)
A roberta-base model fine-tuned to sort English text into 7 mental-health categories, exported to ONNX and quantized to dynamic int8 for fast CPU inference.
It is the Diagnostic Agent in the Multi-Agent Mental Health Chatbot final-year project. There it labels each user message and supplies the Suicidal probability to the crisis safety net.
β οΈ This is not a diagnostic tool. It is a research and educational prototype. Its outputs are not clinical assessments and must not replace professional care or emergency services.
Labels
| id | label |
|---|---|
| 0 | Normal |
| 1 | Depression |
| 2 | Suicidal |
| 3 | Anxiety |
| 4 | Bipolar |
| 5 | Stress |
| 6 | Personality disorder |
Quick start
pip install "optimum[onnxruntime]" transformers
import torch
from transformers import AutoTokenizer
from optimum.onnxruntime import ORTModelForSequenceClassification
repo = "Fz0212/mh-roberta-classifier-onnx-int8"
tokenizer = AutoTokenizer.from_pretrained(repo)
model = ORTModelForSequenceClassification.from_pretrained(
repo, file_name="model_quantized.onnx"
)
text = "I can't sleep and I keep worrying about everything."
inputs = tokenizer(text, return_tensors="pt", truncation=True, max_length=256)
probs = torch.softmax(model(**inputs).logits, dim=-1)[0]
label = model.config.id2label[int(probs.argmax())]
print(label, round(float(probs.max()), 4))
Or use the pipeline API:
from transformers import pipeline
clf = pipeline("text-classification", model=model, tokenizer=tokenizer, top_k=None)
print(clf("I feel hopeless and tired all the time."))
Files
| File | Description |
|---|---|
model_quantized.onnx |
Dynamic int8 quantized ONNX graph (~120 MB) |
config.json |
Model config with id2label / label2id |
ort_config.json |
ONNX Runtime quantization settings |
tokenizer.json, vocab.json, merges.txt, tokenizer_config.json, special_tokens_map.json |
RoBERTa BPE tokenizer |
Training
Base model: FacebookAI/roberta-base. It beat BERT, DistilBERT and Mental-BERT on test macro F1.
Data: sentimentAnalysisMH.csv, a 7-class dataset of short English posts labeled by mental-health status. Preprocessing:
- removed URLs and HTML tags, and collapsed whitespace
- dropped empty and duplicate texts so no text appears in both train and test
- made a stratified 70 / 10 / 20 train / validation / test split (seed 42); the test set has 10,213 samples
Hyperparameters
| Setting | Value |
|---|---|
| Epochs | 3 (early stopping, patience 2) |
| Learning rate | 1.5e-5 |
| Optimizer | AdamW, weight decay 0.05 |
| Batch size | 16 train / 32 eval |
| Loss | Class-weighted cross-entropy, label smoothing 0.1 |
| Dropout | 0.3 (hidden + attention) |
| Max sequence length | 256 |
| Model selection | Macro F1 on validation |
Quantization
The model was exported with optimum.onnxruntime and then quantized with ORTQuantizer, using these settings:
AutoQuantizationConfig.avx512_vnni(is_static=False, per_channel=False)- dynamic quantization (
IntegerOps,QOperatorformat) - weights as symmetric
QInt8, activations as asymmetricQUInt8
Evaluation
Fine-tuned PyTorch model (test set, max_len 256)
| Metric | Value |
|---|---|
| Accuracy | 0.8274 |
| Macro F1 | 0.8010 |
Per-class results:
| Class | Precision | Recall | F1 | Support |
|---|---|---|---|---|
| Normal | 0.9822 | 0.9105 | 0.9450 | 3208 |
| Anxiety | 0.8066 | 0.9115 | 0.8558 | 723 |
| Bipolar | 0.8595 | 0.8320 | 0.8455 | 500 |
| Depression | 0.8405 | 0.7106 | 0.7701 | 3017 |
| Suicidal | 0.6875 | 0.8445 | 0.7579 | 2128 |
| Stress | 0.6565 | 0.8472 | 0.7398 | 458 |
| Personality disorder | 0.6868 | 0.6983 | 0.6925 | 179 |
PyTorch fp32 vs ONNX int8 (CPU, test set, max_len 128)
| Metric | PyTorch fp32 | ONNX int8 | Change |
|---|---|---|---|
| Size | 478.9 MB | 124.3 MB | 3.85Γ smaller |
| Accuracy | 0.8185 | 0.8161 | β0.23 pp |
| Macro F1 | 0.7849 | 0.7800 | β0.49 pp |
| Latency (batch 1, median) | 37.92 ms | 17.35 ms | 2.19Γ faster |
| Throughput (batch 32) | 26.4 ex/s | 66.6 ex/s | 2.52Γ faster |
The benchmark used a shorter max length (128) than training (256), so its accuracy is slightly lower than the table above.
Intended use
- Research and teaching on mental-health text classification
- One signal inside a larger system with human-designed safeguards. In the source project, a message is escalated to a crisis-helpline reply when
P(Suicidal) β₯ 0.8or a self-harm regex matches.
Out-of-scope use
- Diagnosing or screening real individuals
- Making decisions about a person's care, insurance, employment or legal status
- Acting as the only crisis-detection mechanism
- Non-English text or long clinical documents
Limitations and bias
- Depression and Suicidal are often confused. Depression recall is 0.71 and Suicidal precision is 0.69, so many depressive posts are flagged as Suicidal. This errs on the side of caution, but it produces false alarms.
- Rare classes are weaker. Personality disorder has only 179 test samples and F1 0.69.
- Domain shift: the training data is mostly short social-media posts, so performance on chat messages, formal writing or other dialects may be lower.
- Label noise: labels come from the source dataset, not from clinicians.
- Quantization costs about 0.5 pp of macro F1 compared with fp32.
Citation
@misc{mh-roberta-onnx-int8,
author = {Lim Feng Zhi},
title = {Mental-Health RoBERTa Classifier (ONNX int8)},
year = {2026},
url = {https://huggingface.co/Fz0212/mh-roberta-classifier-onnx-int8}
}
- Downloads last month
- 13
Model tree for Fz0212/mh-roberta-classifier-onnx-int8
Base model
FacebookAI/roberta-baseEvaluation results
- Accuracy (ONNX int8, max_len 128) on Sentiment Analysis for Mental Health (sentimentAnalysisMH.csv)test set self-reported0.816
- Macro F1 (ONNX int8, max_len 128) on Sentiment Analysis for Mental Health (sentimentAnalysisMH.csv)test set self-reported0.780