File size: 4,978 Bytes
3abb143 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 | ---
base_model: bert-base-uncased
library_name: peft
tags:
- mental-health
- burnout-detection
- lora
- dual-head-classifier
- workplace-wellbeing
---
# SentinelAI BERT Filter - LoRA Adapters
LoRA fine-tuned BERT model for employee mental health classification in workplace messages. Part of the SentinelAI system for automated burnout detection via Slack message analysis.
## Model Description
- **Base Model:** bert-base-uncased (110M parameters)
- **Fine-tuning Method:** LoRA (Low-Rank Adaptation)
- **Task:** Dual-head classification (category + severity)
- **Trainable Parameters:** 303,371 / 109,785,611 (0.28%)
- **Developed by:** Team Rocket Number One, King's College London
- **License:** MIT (project-specific)
## Architecture
**Dual-Head Classifier:**
- **Category Head:** 7-class classification
- neutral, humor_sarcasm, stress, burnout, depression, harassment, suicidal_ideation
- **Severity Head:** 4-stage classification
- none, early, middle, late
- **Binary Routing:** 5 risk categories trigger escalation to LLM agents
## Training Details
### Dataset
- **Total Examples:** 7,000 (mixed dataset for quality + diversity)
- 5,000 from v0.1 (natural Slack-style phrasing)
- 2,000 from v0.2 (lexically diverse, synonym-enhanced)
- **Splits:** 80% train (5,600), 10% val (700), 10% test (700)
- **Lexical Diversity:** TTR 0.35 (exceeds 0.3 quality threshold)
- **Clinical Grounding:** Maslach Burnout Inventory (MBI), DSM-5, UK Equality Act 2010
### Hyperparameters
```yaml
LoRA Configuration:
r: 8
lora_alpha: 16
lora_dropout: 0.1
target_modules: ["query", "value"]
task_type: FEATURE_EXTRACTION
Training:
epochs: 3
batch_size: 16
learning_rate: 3e-4
optimizer: AdamW
scheduler: Linear warmup + decay
max_sequence_length: 128
loss_function: CrossEntropyLoss (category + severity summed)
```
### Hardware & Performance
- **GPU:** NVIDIA GeForce GTX 1080 (8GB VRAM)
- **Training Time:** ~3 minutes (1 min/epoch)
- **Training Regime:** fp32
## Results
### Test Set Performance
| Metric | Score |
| :----- | :---- |
| **Category Accuracy** | 76.29% |
| **Severity Accuracy** | 78.29% |
| **Test Loss** | 1.1840 |
**Performance Context:**
- Category: 5.3x better than random (7-class baseline: 14.3%)
- Severity: 3.1x better than random (4-class baseline: 25%)
- Low/no overfitting: Test accuracy matches validation accuracy
## Usage
### Loading the Model (Production Pattern)
The repository uses a centralised **Model Factory** to handle architecture initialisation and weight loading. It includes **Auto-Download** logic that pulls the latest checkpoint from Hugging Face Hub if it is not found locally.
```python
import torch
from services.model_factory import load_production_model
# Inference
device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
# This will:
# 1. Initialise DualHeadBERTClassifier
# 2. Apply LoRA adapters
# 3. Check for 'dual_head_classifier.pt' locally
# 4. If missing, download latest from OguzhanKOG/sentinelai-bert-filter
# 5. Load trained weights and return model in eval mode
model = load_production_model(device=device)
# Model is ready for inference
message = "I'm completely overwhelmed with work and can't sleep anymore"
# ... standard tokenization using config.MODEL_NAME ...
```
### Configuration
All parameters (LoRA rank, Alpha, Model Backbone, Paths) are centralised in `filter/config.py`. To change the backbone or parameters across the entire service, update this file only.
## Limitations
- **Synthetic Training Data:** Model trained on generated examples, not real workplace messages
- **English Only:** No multilingual support
- **Context Window:** Limited to 128 tokens (Slack message-sized)
- **Not a Clinical Tool:** Designed for workplace wellbeing monitoring, not medical diagnosis
- **Bias Risk:** May reflect biases in synthetic data generation process
## Intended Use
**Primary Use Case:** Fast, cost-effective gatekeeper filter in SentinelAI architecture. Routes high-risk messages to expensive LLM agents for detailed analysis, while filtering out low-risk neutral messages.
**Architecture Position:**
```text
Slack Message → BERT Filter (this model) → [if risk] → LLM Agent Analysis → HR Alert
```
**Not Intended For:**
- Clinical diagnosis or medical decision-making
- Standalone mental health assessment
- Real-time crisis intervention (human oversight required)
- Legal or disciplinary actions without human review
## Training Logs
Full training metrics available in `training_log.json`:
- Epoch-by-epoch train/val losses
- Category and severity accuracies per epoch
- Final test set evaluation results
## Repository
Full implementation available in the project repository (Private).
Branch: `feature/filter`
### Framework versions
- PEFT 0.18.1
|