Instructions to use Nasir6/university-intent-router with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use Nasir6/university-intent-router with Transformers:
# Use a pipeline as a high-level helper from transformers import pipeline pipe = pipeline("text-classification", model="Nasir6/university-intent-router")# pip install -U transformers accelerate # Load model directly from transformers import AutoTokenizer, AutoModelForSequenceClassification tokenizer = AutoTokenizer.from_pretrained("Nasir6/university-intent-router") model = AutoModelForSequenceClassification.from_pretrained("Nasir6/university-intent-router", device_map="auto") - Notebooks
- Google Colab
- Kaggle
University Intent Router
A ModernBERT-base classifier that routes queries for a university assistant into one of four routes and returns a confidence score for every route. It understands English and Roman Urdu.
Routes
| Label | Meaning | Example |
|---|---|---|
| general_info | Fees, admissions, programs, eligibility, scholarships, facilities, timings, location | bs cs ki fee kitni hai |
| booking | Book, reschedule or cancel a campus visit, counselor meeting, test slot or appointment | book a campus visit for saturday |
| media_link | Requests for university images, photos or videos | hostel ki pictures bhejo |
| oos | Anything outside these three routes | what is the weather in lahore |
How to Use
import torch
from transformers import AutoTokenizer, AutoModelForSequenceClassification
repo = "Nasir6/university-intent-router"
tokenizer = AutoTokenizer.from_pretrained(repo)
model = AutoModelForSequenceClassification.from_pretrained(repo).eval()
query = "hostel ki pictures bhejo"
enc = tokenizer(query, return_tensors="pt", truncation=True, max_length=64)
with torch.no_grad():
probs = torch.softmax(model(**enc).logits, dim=-1)[0] * 100
scores = sorted(((model.config.id2label[i], round(p.item(), 2)) for i, p in enumerate(probs)), key=lambda x: -x[1])
route, confidence = scores[0]
needs_review = confidence < 70.0
print(route, confidence, needs_review, scores)
Output:
media_link 99.06 False [('media_link', 99.06), ('general_info', 0.47), ('booking', 0.36), ('oos', 0.11)]
Recommended rule: if the top confidence is below 70.0%, do not route blindly. Ask the user a clarification question or send the query to an LLM fallback. In testing, this flag caught the wrong predictions while leaving correct ones untouched.
Training Data
There was no public dataset for these routes, so the data was built from a system prompt that defines each route:
- Templates with slots (program, facility, topic, day, place) generate natural queries in English and Roman Urdu
- Seed examples written by hand for each route
- Contrast pairs: the same object appears in different routes (hostel info vs hostel visit booking vs hostel pictures), so the model learns intent instead of keywords
- Hard negatives: oos queries that look like routes, such as "uber book kar do" or "dubai ki ticket book karo"
- CLINC OOS: out-of-scope queries and queries from unrelated CLINC intents form the oos class
- Typo copies: about 30% of rows get realistic keyboard typos
Rows per source:
| label | clinc_intent | clinc_oos | seed | template | typo |
|---|---|---|---|---|---|
| general_info | 0 | 0 | 25 | 850 | 261 |
| booking | 0 | 0 | 27 | 1040 | 335 |
| media_link | 0 | 0 | 23 | 822 | 242 |
| oos | 1188 | 1148 | 25 | 139 | 394 |
The split is done by template group, not by row, so the validation and test sets contain sentence patterns the model never saw during training.
| label | train | validation | test |
|---|---|---|---|
| general_info | 795 | 171 | 170 |
| booking | 982 | 210 | 210 |
| media_link | 761 | 163 | 163 |
| oos | 2026 | 434 | 434 |
Training Procedure
| Setting | Value |
|---|---|
| Base model | answerdotai/ModernBERT-base |
| Loss | Cross-entropy |
| Optimizer | AdamW |
| Learning rate | 3e-05 with warmup and linear decay |
| Weight decay | 0.01 |
| Batch size | 32 |
| Max length | 64 tokens |
| Max epochs | 10 with early stopping (patience 2) |
| Best epoch | 4 (lowest validation loss 0.5938) |
| Precision | fp16 |
Evaluation
The most important test is a hand-written real-world eval set of 80 natural queries (20 per route), never used for training. Four models were compared on it:
- Baseline v1 and Model A: TF-IDF + Logistic Regression and ModernBERT on the original data
- Baseline v2 and Model B: the same two models on the improved data (this repo is Model B)
| metric | Baseline v1 | Model A | Baseline v2 | Model B |
|---|---|---|---|---|
| accuracy | 0.5 | 0.525 | 0.6 | 0.7 |
| macro_f1 | 0.502 | 0.53 | 0.614 | 0.702 |
| f1_general_info | 0.308 | 0.429 | 0.562 | 0.581 |
| f1_booking | 0.571 | 0.533 | 0.621 | 0.81 |
| f1_media_link | 0.667 | 0.667 | 0.727 | 0.788 |
| f1_oos | 0.464 | 0.491 | 0.545 | 0.63 |
Key findings:
- Template test scores overestimate real performance. The first model scored 0.918 on template test data but only 0.525 on natural queries.
- Improving the data gave far more than changing the model: +17.5 accuracy points for ModernBERT versus +2.5 from switching TF-IDF to ModernBERT.
- On the improved data, ModernBERT beats the TF-IDF baseline by 10 points.
- Model B's mistakes usually have low confidence, so the review flag can catch them.
Limitations
- Not production grade yet: about 70% accuracy on natural queries. The target for production is around 90%.
- Synthetic training data: real user queries are not included yet.
- Unseen topics fall into oos: questions about topics missing from training (for example department heads or dress code) are often predicted as oos.
- Indirect requests: media or booking requests without clear keywords ("what does it look like inside", "put my name down") can be missed.
- Small eval set: 80 queries, so each query changes accuracy by 1.25%. The numbers show a trend, not a final score.
- Scripts: trained on English and Roman Urdu only. Urdu script is not supported.
Intended Use
Routing step in a university chatbot or WhatsApp assistant, followed by a handler per route. Use as a pilot with the review flag on and predictions logged.
Not intended for decisions that affect students without a human or fallback check.
Next Steps
- Add labeled real user queries from chat logs
- Generate diverse paraphrases with an LLM from the system prompt
- Build a larger frozen eval set of 200 to 400 queries
- Log low-confidence predictions and retrain on corrected labels
- Downloads last month
- 11
Model tree for Nasir6/university-intent-router
Base model
answerdotai/ModernBERT-base