Mental-Health RoBERTa Classifier (ONNX int8)

A roberta-base model fine-tuned to sort English text into 7 mental-health categories, exported to ONNX and quantized to dynamic int8 for fast CPU inference.

It is the Diagnostic Agent in the Multi-Agent Mental Health Chatbot final-year project. There it labels each user message and supplies the Suicidal probability to the crisis safety net.

⚠️ This is not a diagnostic tool. It is a research and educational prototype. Its outputs are not clinical assessments and must not replace professional care or emergency services.

Labels

id label
0 Normal
1 Depression
2 Suicidal
3 Anxiety
4 Bipolar
5 Stress
6 Personality disorder

Quick start

pip install "optimum[onnxruntime]" transformers
import torch
from transformers import AutoTokenizer
from optimum.onnxruntime import ORTModelForSequenceClassification

repo = "Fz0212/mh-roberta-classifier-onnx-int8"
tokenizer = AutoTokenizer.from_pretrained(repo)
model = ORTModelForSequenceClassification.from_pretrained(
    repo, file_name="model_quantized.onnx"
)

text = "I can't sleep and I keep worrying about everything."
inputs = tokenizer(text, return_tensors="pt", truncation=True, max_length=256)
probs = torch.softmax(model(**inputs).logits, dim=-1)[0]

label = model.config.id2label[int(probs.argmax())]
print(label, round(float(probs.max()), 4))

Or use the pipeline API:

from transformers import pipeline
clf = pipeline("text-classification", model=model, tokenizer=tokenizer, top_k=None)
print(clf("I feel hopeless and tired all the time."))

Files

File Description
model_quantized.onnx Dynamic int8 quantized ONNX graph (~120 MB)
config.json Model config with id2label / label2id
ort_config.json ONNX Runtime quantization settings
tokenizer.json, vocab.json, merges.txt, tokenizer_config.json, special_tokens_map.json RoBERTa BPE tokenizer

Training

Base model: FacebookAI/roberta-base. It beat BERT, DistilBERT and Mental-BERT on test macro F1.

Data: sentimentAnalysisMH.csv, a 7-class dataset of short English posts labeled by mental-health status. Preprocessing:

  • removed URLs and HTML tags, and collapsed whitespace
  • dropped empty and duplicate texts so no text appears in both train and test
  • made a stratified 70 / 10 / 20 train / validation / test split (seed 42); the test set has 10,213 samples

Hyperparameters

Setting Value
Epochs 3 (early stopping, patience 2)
Learning rate 1.5e-5
Optimizer AdamW, weight decay 0.05
Batch size 16 train / 32 eval
Loss Class-weighted cross-entropy, label smoothing 0.1
Dropout 0.3 (hidden + attention)
Max sequence length 256
Model selection Macro F1 on validation

Quantization

The model was exported with optimum.onnxruntime and then quantized with ORTQuantizer, using these settings:

  • AutoQuantizationConfig.avx512_vnni(is_static=False, per_channel=False)
  • dynamic quantization (IntegerOps, QOperator format)
  • weights as symmetric QInt8, activations as asymmetric QUInt8

Evaluation

Fine-tuned PyTorch model (test set, max_len 256)

Metric Value
Accuracy 0.8274
Macro F1 0.8010

Per-class results:

Class Precision Recall F1 Support
Normal 0.9822 0.9105 0.9450 3208
Anxiety 0.8066 0.9115 0.8558 723
Bipolar 0.8595 0.8320 0.8455 500
Depression 0.8405 0.7106 0.7701 3017
Suicidal 0.6875 0.8445 0.7579 2128
Stress 0.6565 0.8472 0.7398 458
Personality disorder 0.6868 0.6983 0.6925 179

PyTorch fp32 vs ONNX int8 (CPU, test set, max_len 128)

Metric PyTorch fp32 ONNX int8 Change
Size 478.9 MB 124.3 MB 3.85Γ— smaller
Accuracy 0.8185 0.8161 βˆ’0.23 pp
Macro F1 0.7849 0.7800 βˆ’0.49 pp
Latency (batch 1, median) 37.92 ms 17.35 ms 2.19Γ— faster
Throughput (batch 32) 26.4 ex/s 66.6 ex/s 2.52Γ— faster

The benchmark used a shorter max length (128) than training (256), so its accuracy is slightly lower than the table above.

Intended use

  • Research and teaching on mental-health text classification
  • One signal inside a larger system with human-designed safeguards. In the source project, a message is escalated to a crisis-helpline reply when P(Suicidal) β‰₯ 0.8 or a self-harm regex matches.

Out-of-scope use

  • Diagnosing or screening real individuals
  • Making decisions about a person's care, insurance, employment or legal status
  • Acting as the only crisis-detection mechanism
  • Non-English text or long clinical documents

Limitations and bias

  • Depression and Suicidal are often confused. Depression recall is 0.71 and Suicidal precision is 0.69, so many depressive posts are flagged as Suicidal. This errs on the side of caution, but it produces false alarms.
  • Rare classes are weaker. Personality disorder has only 179 test samples and F1 0.69.
  • Domain shift: the training data is mostly short social-media posts, so performance on chat messages, formal writing or other dialects may be lower.
  • Label noise: labels come from the source dataset, not from clinicians.
  • Quantization costs about 0.5 pp of macro F1 compared with fp32.

Citation

@misc{mh-roberta-onnx-int8,
  author = {Lim Feng Zhi},
  title  = {Mental-Health RoBERTa Classifier (ONNX int8)},
  year   = {2026},
  url    = {https://huggingface.co/Fz0212/mh-roberta-classifier-onnx-int8}
}
Downloads last month
13
Inference Providers NEW
This model isn't deployed by any Inference Provider. πŸ™‹ Ask for provider support

Model tree for Fz0212/mh-roberta-classifier-onnx-int8

Quantized
(20)
this model

Evaluation results

  • Accuracy (ONNX int8, max_len 128) on Sentiment Analysis for Mental Health (sentimentAnalysisMH.csv)
    test set self-reported
    0.816
  • Macro F1 (ONNX int8, max_len 128) on Sentiment Analysis for Mental Health (sentimentAnalysisMH.csv)
    test set self-reported
    0.780