PolyAI/banking77
Viewer β’ Updated β’ 13.1k β’ 22.8k β’ 126
State-of-the-art 77-class customer service intent classifier powered by ModernBERT, achieving 95.40% test accuracy on complex financial banking queries.
transformers, sentence-transformers, peft, and standard inference pipelines.Evaluated on the official held-out test split under standard production evaluation protocols:
| Benchmark Metric | Score / Value |
|---|---|
| Pretrained Baseline Zero-Shot Accuracy | 1.40% |
| Pretrained Baseline Macro-F1 | 0.04% |
| Fine-Tuned Held-Out Test Accuracy | 95.40% |
| Absolute Accuracy Gain | +94.00% |
| Fine-Tuned Held-Out Macro-F1 | 95.19% |
| Held-Out Test Loss | 0.1813 |
| Total Intent Classes | 77 |
| Evaluation Throughput | 487.3 samples/sec |
π‘ Benchmark Takeaway: Fine-tuning establishes substantial gains over the zero-shot base checkpoint, dramatically suppressing false predictions and improving domain confidence.
Install the required dependencies:
pip install transformers torch sentence-transformers peft accelerate
from transformers import pipeline
# Load fine-tuned 77-class banking intent classifier
classifier = pipeline("text-classification", model="vennyvirtuoso/modernbert-banking77-intent", device_map="auto")
queries = [
"Why is there an extra $15 fee charged on my international wire transfer?",
"I received my card today, how do I activate it so I can start making purchases?",
"My card was swallowed by the ATM machine at the airport, what should I do?"
]
results = classifier(queries)
print(f"\nQuery | Predicted Intent | Score ")
print("-" * 105)
for q, res in zip(queries, results):
print(f"{q[:62] + '...':<65} | {res['label']:<28} | {res['score']:>6.2%}")
Query | Predicted Intent | Score
---------------------------------------------------------------------------------------------------------
Why is there an extra $15 fee charged on my international wire... | transfer_fee_charged | 99.12%
I received my card today, how do I activate it so I can start ... | activate_my_card | 99.85%
My card was swallowed by the ATM machine at the airport, what ... | card_swallowed | 99.41%
| Parameter | Specification |
|---|---|
| Base Foundation Model | answerdotai/ModernBERT-base |
| Fine-Tuned Checkpoint | vennyvirtuoso/modernbert-banking77-intent |
| Task Category | Banking Intent Classification |
| Training Dataset | PolyAI/banking77 |
| Precision | Mixed Precision (bfloat16 / fp16) |
| License | Apache 2.0 (Permissive Commercial & Research) |
transformers or Text Embeddings/Classification Inference (TEI).This model is open-sourced under the Apache 2.0 license. We express gratitude to the open-source AI community and authors of the base model and dataset for releasing permissive open weights and data.
Base model
answerdotai/ModernBERT-base