Buckets:
| # Policy classifier: ModernBERT-large vs Qwen3-0.6B, in-domain and held-out domains | |
| 3 November 2026 | |
| ## Hypothesis | |
| A message-only policy classifier (followed / violated / not applicable) trained on generated support conversations should still work on business domains it has never seen. We test this by holding out 2 of the 12 domains entirely, and compare two backbones of similar cost: the encoder ModernBERT-large and the decoder Qwen3-0.6B. | |
| ## What did we do | |
| Four training runs: each model on two versions of the same 30,000 labelled pairs. | |
| **Data** | |
| | Split version | Train | Validation | Test | | |
| | --- | ---: | ---: | ---: | | |
| | Original (random by case, all 12 domains) | 27,000 | 3,000 | none | | |
| | Domain split (10 training domains, 2 held out) | 22,204 | 2,575 | 5,221 | | |
| - Held-out test domains: insurance claims and travel booking. They were chosen because their label mix (about 38% followed, 25% violated, 37% not applicable) is closest to the whole dataset. | |
| - Training domains: e-commerce returns, healthcare intake, edtech tutoring, recruiting, telecom support, real estate, banking support, HR helpdesk, fitness and nutrition, IT helpdesk. | |
| - Validation in the domain split is one whole case from each training domain (154 to 353 pairs per domain), so no conversation appears in both train and validation. | |
| **Training configuration** (identical for all four runs except where noted) | |
| | Setting | ModernBERT-large | Qwen3-0.6B | | |
| | --- | --- | --- | | |
| | Classification head | mean-pooled head + classifier | score on last non-pad token | | |
| | LoRA target layers | attention QKV, MLP in/out | all attention and MLP projections | | |
| | Max sequence length | 2,400 | 2,352 | | |
| | Adapter | LoRA, rank 32, alpha 64, dropout 0.05 (no DoRA) | same | | |
| | Batch size | 64, no accumulation | same | | |
| | Epochs | 1.5 | same | | |
| | Learning rate | 2e-4, linear decay, 5% warmup | same | | |
| | Weight decay / gradient clip | 0.01 / 1.0 | same | | |
| | Precision | BF16, gradient checkpointing on | same | | |
| | Seed | 7 | same | | |
| | Optimiser steps | 633 (original), 521 (domain split) | same | | |
| | Hardware | 1x H100 NVL (the two domain-split runs shared it) | same | | |
| All numbers below come from the final checkpoint of each run. Training metrics were logged to Weights & Biases in offline mode and have not been synced yet. | |
| ## What did we find | |
| **Core findings** | |
| - Both models carry over to unseen domains with a small drop: Qwen loses about 1 accuracy point from validation to the held-out test, ModernBERT about 2. | |
| - Qwen3-0.6B is better than ModernBERT-large on every summary metric, in both split versions. The largest gap is on the held-out test (macro F1 0.829 vs 0.812). | |
| - Violation detection is the weak class for both models: violated recall is 0.69 (ModernBERT) and 0.75 (Qwen) on domain-split validation. Missed violations are mostly predicted as followed. | |
| - On the original split the two models are within 0.5 points of each other, which is within noise for 3,000 validation pairs. | |
| **Domain split: summary** | |
| | Model | Split | Pairs | Accuracy | Macro F1 | | |
| | --- | --- | ---: | ---: | ---: | | |
| | ModernBERT-large | Validation (seen domains) | 2,575 | 0.8392 | 0.8296 | | |
| | ModernBERT-large | Test (unseen domains) | 5,221 | 0.8159 | 0.8116 | | |
| | Qwen3-0.6B | Validation (seen domains) | 2,575 | 0.8439 | 0.8376 | | |
| | Qwen3-0.6B | Test (unseen domains) | 5,221 | 0.8324 | 0.8291 | | |
| **Domain split: per class** | |
| | Model | Split | Class | Precision | Recall | F1 | Support | | |
| | --- | --- | --- | ---: | ---: | ---: | ---: | | |
| | ModernBERT-large | Validation | followed | 0.7788 | 0.8950 | 0.8328 | 952 | | |
| | ModernBERT-large | Validation | violated | 0.8544 | 0.6880 | 0.7622 | 657 | | |
| | ModernBERT-large | Validation | not applicable | 0.9002 | 0.8872 | 0.8936 | 966 | | |
| | ModernBERT-large | Test | followed | 0.8184 | 0.8083 | 0.8133 | 1,946 | | |
| | ModernBERT-large | Test | violated | 0.7668 | 0.7849 | 0.7757 | 1,311 | | |
| | ModernBERT-large | Test | not applicable | 0.8472 | 0.8442 | 0.8457 | 1,964 | | |
| | Qwen3-0.6B | Validation | followed | 0.8031 | 0.8613 | 0.8312 | 952 | | |
| | Qwen3-0.6B | Validation | violated | 0.8419 | 0.7458 | 0.7910 | 657 | | |
| | Qwen3-0.6B | Validation | not applicable | 0.8879 | 0.8934 | 0.8906 | 966 | | |
| | Qwen3-0.6B | Test | followed | 0.8115 | 0.8294 | 0.8203 | 1,946 | | |
| | Qwen3-0.6B | Test | violated | 0.8043 | 0.7994 | 0.8018 | 1,311 | | |
| | Qwen3-0.6B | Test | not applicable | 0.8730 | 0.8574 | 0.8651 | 1,964 | | |
| **Original split (validation only, 3,000 pairs)** | |
| | Model | Class | Precision | Recall | F1 | Support | | |
| | --- | --- | ---: | ---: | ---: | ---: | | |
| | ModernBERT-large | followed | 0.8186 | 0.8215 | 0.8201 | 1,115 | | |
| | ModernBERT-large | violated | 0.8071 | 0.7714 | 0.7888 | 770 | | |
| | ModernBERT-large | not applicable | 0.8524 | 0.8753 | 0.8637 | 1,115 | | |
| | ModernBERT-large | **accuracy / macro F1** | | | **0.8287 / 0.8242** | | | |
| | Qwen3-0.6B | followed | 0.8197 | 0.8152 | 0.8174 | 1,115 | | |
| | Qwen3-0.6B | violated | 0.8359 | 0.7740 | 0.8038 | 770 | | |
| | Qwen3-0.6B | not applicable | 0.8430 | 0.8906 | 0.8661 | 1,115 | | |
| | Qwen3-0.6B | **accuracy / macro F1** | | | **0.8327 / 0.8291** | | | |
| Confusion matrices, original split (rows = true label, columns = predicted followed / violated / not applicable): | |
| | Model | True label | followed | violated | not applicable | | |
| | --- | --- | ---: | ---: | ---: | | |
| | ModernBERT-large | followed | 916 | 84 | 115 | | |
| | ModernBERT-large | violated | 122 | 594 | 54 | | |
| | ModernBERT-large | not applicable | 81 | 58 | 976 | | |
| | Qwen3-0.6B | followed | 909 | 74 | 132 | | |
| | Qwen3-0.6B | violated | 121 | 596 | 53 | | |
| | Qwen3-0.6B | not applicable | 79 | 43 | 993 | | |
| Original-split and domain-split numbers use different validation sets and should not be compared with each other. Each result is a single seed. | |
| ## Next steps | |
| To be agreed before writing. | |
Xet Storage Details
- Size:
- 5.77 kB
- Xet hash:
- f93159aeff6f1315c62e37f2fd907bc21c9efbda18a1849f62872ed21658e4e2
·
Xet efficiently stores files, intelligently splitting them into unique chunks and accelerating uploads and downloads. More info.