University Intent Router

A ModernBERT-base classifier that routes queries for a university assistant into one of four routes and returns a confidence score for every route. It understands English and Roman Urdu.

Routes

Label Meaning Example
general_info Fees, admissions, programs, eligibility, scholarships, facilities, timings, location bs cs ki fee kitni hai
booking Book, reschedule or cancel a campus visit, counselor meeting, test slot or appointment book a campus visit for saturday
media_link Requests for university images, photos or videos hostel ki pictures bhejo
oos Anything outside these three routes what is the weather in lahore

How to Use

import torch
from transformers import AutoTokenizer, AutoModelForSequenceClassification

repo = "Nasir6/university-intent-router"
tokenizer = AutoTokenizer.from_pretrained(repo)
model = AutoModelForSequenceClassification.from_pretrained(repo).eval()

query = "hostel ki pictures bhejo"
enc = tokenizer(query, return_tensors="pt", truncation=True, max_length=64)
with torch.no_grad():
    probs = torch.softmax(model(**enc).logits, dim=-1)[0] * 100

scores = sorted(((model.config.id2label[i], round(p.item(), 2)) for i, p in enumerate(probs)), key=lambda x: -x[1])
route, confidence = scores[0]
needs_review = confidence < 70.0
print(route, confidence, needs_review, scores)

Output:

media_link 99.06 False [('media_link', 99.06), ('general_info', 0.47), ('booking', 0.36), ('oos', 0.11)]

Recommended rule: if the top confidence is below 70.0%, do not route blindly. Ask the user a clarification question or send the query to an LLM fallback. In testing, this flag caught the wrong predictions while leaving correct ones untouched.

Training Data

There was no public dataset for these routes, so the data was built from a system prompt that defines each route:

  • Templates with slots (program, facility, topic, day, place) generate natural queries in English and Roman Urdu
  • Seed examples written by hand for each route
  • Contrast pairs: the same object appears in different routes (hostel info vs hostel visit booking vs hostel pictures), so the model learns intent instead of keywords
  • Hard negatives: oos queries that look like routes, such as "uber book kar do" or "dubai ki ticket book karo"
  • CLINC OOS: out-of-scope queries and queries from unrelated CLINC intents form the oos class
  • Typo copies: about 30% of rows get realistic keyboard typos

Rows per source:

label clinc_intent clinc_oos seed template typo
general_info 0 0 25 850 261
booking 0 0 27 1040 335
media_link 0 0 23 822 242
oos 1188 1148 25 139 394

The split is done by template group, not by row, so the validation and test sets contain sentence patterns the model never saw during training.

label train validation test
general_info 795 171 170
booking 982 210 210
media_link 761 163 163
oos 2026 434 434

Training Procedure

Setting Value
Base model answerdotai/ModernBERT-base
Loss Cross-entropy
Optimizer AdamW
Learning rate 3e-05 with warmup and linear decay
Weight decay 0.01
Batch size 32
Max length 64 tokens
Max epochs 10 with early stopping (patience 2)
Best epoch 4 (lowest validation loss 0.5938)
Precision fp16

Evaluation

The most important test is a hand-written real-world eval set of 80 natural queries (20 per route), never used for training. Four models were compared on it:

  • Baseline v1 and Model A: TF-IDF + Logistic Regression and ModernBERT on the original data
  • Baseline v2 and Model B: the same two models on the improved data (this repo is Model B)
metric Baseline v1 Model A Baseline v2 Model B
accuracy 0.5 0.525 0.6 0.7
macro_f1 0.502 0.53 0.614 0.702
f1_general_info 0.308 0.429 0.562 0.581
f1_booking 0.571 0.533 0.621 0.81
f1_media_link 0.667 0.667 0.727 0.788
f1_oos 0.464 0.491 0.545 0.63

Key findings:

  • Template test scores overestimate real performance. The first model scored 0.918 on template test data but only 0.525 on natural queries.
  • Improving the data gave far more than changing the model: +17.5 accuracy points for ModernBERT versus +2.5 from switching TF-IDF to ModernBERT.
  • On the improved data, ModernBERT beats the TF-IDF baseline by 10 points.
  • Model B's mistakes usually have low confidence, so the review flag can catch them.

Limitations

  • Not production grade yet: about 70% accuracy on natural queries. The target for production is around 90%.
  • Synthetic training data: real user queries are not included yet.
  • Unseen topics fall into oos: questions about topics missing from training (for example department heads or dress code) are often predicted as oos.
  • Indirect requests: media or booking requests without clear keywords ("what does it look like inside", "put my name down") can be missed.
  • Small eval set: 80 queries, so each query changes accuracy by 1.25%. The numbers show a trend, not a final score.
  • Scripts: trained on English and Roman Urdu only. Urdu script is not supported.

Intended Use

Routing step in a university chatbot or WhatsApp assistant, followed by a handler per route. Use as a pilot with the review flag on and predictions logged.

Not intended for decisions that affect students without a human or fallback check.

Next Steps

  • Add labeled real user queries from chat logs
  • Generate diverse paraphrases with an LLM from the system prompt
  • Build a larger frozen eval set of 200 to 400 queries
  • Log low-confidence predictions and retrain on corrected labels
Downloads last month
11
Safetensors
Model size
0.1B params
Tensor type
F32
ยท
Inference Providers NEW
This model isn't deployed by any Inference Provider. ๐Ÿ™‹ Ask for provider support

Model tree for Nasir6/university-intent-router

Finetuned
(1,605)
this model