03-11-progress/reports/policy_classifier_domain_split.md

Policy classifier: ModernBERT-large vs Qwen3-0.6B, in-domain and held-out domains

3 November 2026

Hypothesis

A message-only policy classifier (followed / violated / not applicable) trained on generated support conversations should still work on business domains it has never seen. We test this by holding out 2 of the 12 domains entirely, and compare two backbones of similar cost: the encoder ModernBERT-large and the decoder Qwen3-0.6B.

What did we do

Four training runs: each model on two versions of the same 30,000 labelled pairs.

Data

Split version	Train	Validation	Test
Original (random by case, all 12 domains)	27,000	3,000	none
Domain split (10 training domains, 2 held out)	22,204	2,575	5,221

• Held-out test domains: insurance claims and travel booking. They were chosen because their label mix (about 38% followed, 25% violated, 37% not applicable) is closest to the whole dataset.
• Training domains: e-commerce returns, healthcare intake, edtech tutoring, recruiting, telecom support, real estate, banking support, HR helpdesk, fitness and nutrition, IT helpdesk.
• Validation in the domain split is one whole case from each training domain (154 to 353 pairs per domain), so no conversation appears in both train and validation.

Training configuration (identical for all four runs except where noted)

Setting	ModernBERT-large	Qwen3-0.6B
Classification head	mean-pooled head + classifier	score on last non-pad token
LoRA target layers	attention QKV, MLP in/out	all attention and MLP projections
Max sequence length	2,400	2,352
Adapter	LoRA, rank 32, alpha 64, dropout 0.05 (no DoRA)	same
Batch size	64, no accumulation	same
Epochs	1.5	same
Learning rate	2e-4, linear decay, 5% warmup	same
Weight decay / gradient clip	0.01 / 1.0	same
Precision	BF16, gradient checkpointing on	same
Seed	7	same
Optimiser steps	633 (original), 521 (domain split)	same
Hardware	1x H100 NVL (the two domain-split runs shared it)	same

All numbers below come from the final checkpoint of each run. Training metrics were logged to Weights & Biases in offline mode and have not been synced yet.

What did we find

Core findings

• Both models carry over to unseen domains with a small drop: Qwen loses about 1 accuracy point from validation to the held-out test, ModernBERT about 2.
• Qwen3-0.6B is better than ModernBERT-large on every summary metric, in both split versions. The largest gap is on the held-out test (macro F1 0.829 vs 0.812).
• Violation detection is the weak class for both models: violated recall is 0.69 (ModernBERT) and 0.75 (Qwen) on domain-split validation. Missed violations are mostly predicted as followed.
• On the original split the two models are within 0.5 points of each other, which is within noise for 3,000 validation pairs.

Domain split: summary

Model	Split	Pairs	Accuracy	Macro F1
ModernBERT-large	Validation (seen domains)	2,575	0.8392	0.8296
ModernBERT-large	Test (unseen domains)	5,221	0.8159	0.8116
Qwen3-0.6B	Validation (seen domains)	2,575	0.8439	0.8376
Qwen3-0.6B	Test (unseen domains)	5,221	0.8324	0.8291

Domain split: per class

Model	Split	Class	Precision	Recall	F1	Support
ModernBERT-large	Validation	followed	0.7788	0.8950	0.8328	952
ModernBERT-large	Validation	violated	0.8544	0.6880	0.7622	657
ModernBERT-large	Validation	not applicable	0.9002	0.8872	0.8936	966
ModernBERT-large	Test	followed	0.8184	0.8083	0.8133	1,946
ModernBERT-large	Test	violated	0.7668	0.7849	0.7757	1,311
ModernBERT-large	Test	not applicable	0.8472	0.8442	0.8457	1,964
Qwen3-0.6B	Validation	followed	0.8031	0.8613	0.8312	952
Qwen3-0.6B	Validation	violated	0.8419	0.7458	0.7910	657
Qwen3-0.6B	Validation	not applicable	0.8879	0.8934	0.8906	966
Qwen3-0.6B	Test	followed	0.8115	0.8294	0.8203	1,946
Qwen3-0.6B	Test	violated	0.8043	0.7994	0.8018	1,311
Qwen3-0.6B	Test	not applicable	0.8730	0.8574	0.8651	1,964

Original split (validation only, 3,000 pairs)

Model	Class	Precision	Recall	F1	Support
ModernBERT-large	followed	0.8186	0.8215	0.8201	1,115
ModernBERT-large	violated	0.8071	0.7714	0.7888	770
ModernBERT-large	not applicable	0.8524	0.8753	0.8637	1,115
ModernBERT-large	accuracy / macro F1			0.8287 / 0.8242
Qwen3-0.6B	followed	0.8197	0.8152	0.8174	1,115
Qwen3-0.6B	violated	0.8359	0.7740	0.8038	770
Qwen3-0.6B	not applicable	0.8430	0.8906	0.8661	1,115
Qwen3-0.6B	accuracy / macro F1			0.8327 / 0.8291

Confusion matrices, original split (rows = true label, columns = predicted followed / violated / not applicable):

Model	True label	followed	violated	not applicable
ModernBERT-large	followed	916	84	115
ModernBERT-large	violated	122	594	54
ModernBERT-large	not applicable	81	58	976
Qwen3-0.6B	followed	909	74	132
Qwen3-0.6B	violated	121	596	53
Qwen3-0.6B	not applicable	79	43	993

Original-split and domain-split numbers use different validation sets and should not be compared with each other. Each result is a single seed.

Next steps

To be agreed before writing.