Nasir6/intentprism-modernbert

IntentPrism: multi-intent router for chatbots

One message can trigger several intents. IntentPrism is a fine-tuned ModernBERT-base multi-label classifier that detects every intent in a user message, with an independent calibrated probability for each, so each intent can be sent to the right agent.

"tell me about your company and book an appointment" -> company_info (1.00) + book_appointment (1.00)

What it does

• Multi-intent: a sigmoid head gives each of 156 intents its own score, so two or more can fire at once.
• Typo-robust: trained with up to 5 typos per sentence ("tel me abot yuor compnay and bok an apointment" still works).
• Out-of-scope aware: no intent firing means out of scope (for example "is there a hostel for girls").
• Vague-query aware: one-word queries like "fees?" are sent to a clarifying question.
• Calibrated: temperature scaling and a tuned threshold (see calibration.json).

Labels (156)

• 150 intents from CLINC150 (banking, travel, productivity, and so on)
• 6 custom company intents: company_info, book_appointment, fee_info, contact_support, working_hours, office_location

Quick start

import json, numpy as np, torch
from scipy.special import expit
from transformers import AutoTokenizer, AutoModelForSequenceClassification
from huggingface_hub import hf_hub_download

REPO = "Nasir6/intentprism-modernbert"
tok = AutoTokenizer.from_pretrained(REPO)
model = AutoModelForSequenceClassification.from_pretrained(REPO).eval()
cal = json.load(open(hf_hub_download(REPO, "calibration.json")))

def predict(text):
    enc = tok(text, return_tensors="pt", truncation=True, max_length=96)
    enc.pop("token_type_ids", None)
    with torch.no_grad():
        logits = model(**enc).logits[0].numpy()
    p = expit(logits / cal["temperature"])
    return {cal["intents"][i]: round(float(p[i]), 3) for i in np.where(p >= cal["threshold"])[0]}

print(predict("tell me about your company and book an appointment"))

Training

• Base model: answerdotai/ModernBERT-base
• Data: CLINC150 + templated custom intents + synthetic multi-intent mixes (2-3 intents joined with connectors) + typo copies + vague queries and hard out-of-scope negatives (both labeled "no intent")
• Loss: binary cross-entropy (multi-label), fp16, 6 epochs
• Calibration: temperature fitted on validation, global threshold chosen by balanced exact-match

Results

On a small hand-written test of 11 queries, a single-label baseline (top-1 of this model) passed 3 of 11 and the multi-label router with rules passed 11 of 11. Treat this as a sanity check, not a benchmark: the queries are few and several were written while fixing bugs.

Limitations

• Custom intents come from templated text, so real user messages may score lower.
• Several fixes are rule-based safety nets (window recheck, fee-conflict rule, company-word check), included in router.py, not learned by the model alone.
• English only. Tuned for short chat messages, max 96 tokens.
• Not evaluated on naturally written multi-intent traffic.

Intended use

Routing user messages to agents or tools in chatbots. Not for safety-critical decisions.