cev-141m

Calibrated multi-label intent classifier for HR query routing.

Cev is a lightweight classifier that predicts which capability buckets an HR assistant query requires — with intrinsically calibrated probabilities via Brier score training. No post-hoc temperature scaling needed.

Architecture

Component	Detail
Backbone	microsoft/deberta-v3-small (141M, frozen)
Head	2-layer MLP · GELU · Dropout → sigmoid
Trainable params	297K (head only)
Training loss	Brier score — proper scoring rule, calibration is baked in
Labels	4 hr capability buckets (multi-label)

DeBERTa-v3's disentangled attention (separate content and position encodings) gives stronger representations for low-data classification than standard BERT-family models.

Capabilities

Label	Meaning
person_lookup	Query involves identifying or retrieving an employee
skill_assessment	Query involves skill gaps, competencies, or growth data
learning_discovery	Query involves finding courses, training, or learning content
assignment	Query involves enrolling or assigning learning to an employee

Labels are non-exclusive — a single query can require multiple capabilities.

Usage

from huggingface_hub import hf_hub_download
from transformers import AutoTokenizer
import importlib, sys

# Pull the model file from the hub
hf_hub_download("Nehaa/cev-141m", "modeling_cev.py", local_dir=".")
spec = importlib.util.spec_from_file_location("modeling_cev", "./modeling_cev.py")
mod = importlib.util.module_from_spec(spec); spec.loader.exec_module(mod)
CevModel = mod.CevModel

model = CevModel.load("Nehaa/cev-141m")
tokenizer = AutoTokenizer.from_pretrained("microsoft/deberta-v3-small")

queries = [
    "What skill gaps does Janet Wise have?",
    "Assign Tom some sales training to close his strategy gap",
    "find courses on business analysis for jwise",
]

results = model.predict(queries, tokenizer)
for q, r in zip(queries, results):
    print(q)
    print("  probs:    ", r["probabilities"])
    print("  predicted:", r["predicted"])

Example output:

What skill gaps does Janet Wise have?
  probs:     {'person_lookup': 0.973, 'skill_assessment': 0.961, 'learning_discovery': 0.241, 'assignment': 0.031}
  predicted: {'person_lookup': 1, 'skill_assessment': 1, 'learning_discovery': 0, 'assignment': 0}

Assign Tom some sales training to close his strategy gap
  probs:     {'person_lookup': 0.971, 'skill_assessment': 0.958, 'learning_discovery': 0.897, 'assignment': 0.884}
  predicted: {'person_lookup': 1, 'skill_assessment': 1, 'learning_discovery': 1, 'assignment': 1}

find courses on business analysis for jwise
  probs:     {'person_lookup': 0.965, 'skill_assessment': 0.951, 'learning_discovery': 0.912, 'assignment': 0.072}
  predicted: {'person_lookup': 1, 'skill_assessment': 1, 'learning_discovery': 1, 'assignment': 0}

Why Brier Loss?

Standard classifiers minimise cross-entropy, which optimises rank order but not probability calibration. A model can be 90% accurate but systematically overconfident (all positives predicted at p=0.99 instead of the true ~0.85).

The Brier score mean((p - y)²) is a proper scoring rule — it is uniquely minimised when predicted probabilities match empirical frequencies. This means calibration is an objective, not an afterthought.

Cev inherits this idea from Jev, which uses RLCD (Reinforcement Learning for Calibrated Decisions) with a similar proper scoring objective. The difference: Jev is a large general-domain causal model; Cev is a small domain-specific encoder trained from scratch on ~750 examples.

Training Details

• Data: ~750 hr query examples (10× paraphrase augmentation from 75 seed utterances)
• Augmentation: Haiku-generated paraphrases varying tone, formality, and phrasing
• Split: 90/10 train/val
• Epochs: 100 (head only; encoder frozen throughout)
• Optimiser: AdamW · lr=5e-4 · weight_decay=0.01
• Scheduler: OneCycleLR
• Label smoothing: 0.05 (improves calibration on minority classes)
• Hardware: Apple Silicon CPU (~5 min embedding cache + ~2 min MLP training)

Evaluation

Evaluated on 83 held-out examples:

Capability	F1	ECE
person_lookup	1.000	— ¹
skill_assessment	1.000	— ¹
learning_discovery	0.915	0.103
assignment	0.906	0.142
Mean	0.955

¹ ECE is undefined for capabilities with no negative examples in the held-out set (all 83 val queries required person lookup and skill assessment).

Mean F1: 0.955 across all 4 capabilities.

Relation to Kev / Dev

	Kev	Dev	Cev
Backbone	Qwen2.5-0.5B (causal)	ModernBERT-large	DeBERTa-v3-small
Params	494M frozen + 8.8M LoRA	399M	141M frozen + 297K head
Training loss	Cross-entropy	Cross-entropy	Brier score
Calibration	Post-hoc	Post-hoc	Intrinsic
Domain	General	General	HR-specific
Training examples	9k+	15k+	~750

Cev's contribution: same calibration principle as Jev (proper scoring rule), on a domain-specific task, trained from ~750 examples in under 10 minutes on a laptop.

License

Apache 2.0