Instructions to use VibeHash/laya-banking77 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Laya
How to use VibeHash/laya-banking77 with Laya:
# No code snippets available yet for this library. # To use this model, check the repository files and the library's documentation. # Want to help? PRs adding snippets are welcome at: # https://github.com/huggingface/huggingface.js
- Notebooks
- Google Colab
- Kaggle
laya-banking77
A fine-tune of Laya that sorts bank customer messages into the 77 intents of Banking77. It gives a probability for every intent in one forward pass, and its confidence can be trusted. So it can answer the messages it is sure about by itself and hand the rest to a person.
Results
All 3,080 messages of Banking77's test set (40 per intent), one fine-tune (seed 0), one RTX 4070 SUPER, 2026-11-13.
| Untuned Laya | This model | |
|---|---|---|
| Right | 1,354 of 3,080 (44.0%) | 2,817 of 3,080 (91.5%) |
| 95% confidence interval | 42.2 to 45.7% | 90.4 to 92.4% |
| Mean stated confidence | 0.566 | 0.917 |
| Calibration error (ECE, lower is better) | 0.160 | 0.035 |
When it says 0.92, it is right about 92% of the time.
Answering only when sure. During the fine-tune, Laya held back 400 training messages and fitted a threshold on them, aiming at 99% right among the answers above it. That threshold is 0.979. Applied once to the test:
- The model answers 1,523 of the 3,080 messages (49.4%) by itself, and 1,505 of those are right (98.8%).
- The other 1,557 go to a person, each with the model's best guess.
With the cut chosen on the test itself, which describes this test and is optimistic for new messages: 95% right while answering 91.9% of messages, 98% right at 70.2%, 99% right at 46.3%.
Messages about something else. Banking77 has no "none of these" intent, so the model always picks one of the 77. On the 1,000 queries that CLINC150 labels out of scope (the stock market, screen brightness, algebra), its mean confidence is 0.165, and 7 of the 1,000 (0.7%) pass the threshold. Three of those seven are banking questions after all: "can i deposit a check", "what is the fee for a cash advance", "i need to update my address".
Speed. One message alone: median 22 ms, 95th percentile 24 ms. In batches of 32: about 10 ms a message.
Against published results. The Banking77 paper (Casanueva et al. 2020, Table 3, full training data) reports 93.66% for fine-tuned BERT and 93.36% for USE+ConveRT. This model is about two points behind on accuracy alone, and its whole 95% interval sits below 93.66%. The paper does not report calibration or answering only when sure. This run used Laya's default fine-tune settings and was not tuned for accuracy.
Where it is weakest. Fiat currency support (75% right), declined transfer, pending transfer, and balance not updated after bank transfer (77.5% each). The most common mix-ups are pairs a person could also confuse: fiat currency support and exchange via app, top up reverted and top up failed, card arrival and card delivery estimate.
Use
pip install laya==0.4.31
import json
import laya
from huggingface_hub import hf_hub_download
REPO = "VibeHash/laya-banking77"
model = laya.load(REPO)
with open(hf_hub_download(REPO, "questions.json")) as f:
questions = json.load(f)
result = model.predict("Why was I charged a fee for withdrawing cash?", questions, min_confidence=0.979)
answer = result["answers"]["intent"]
if answer["abstention"] == "abstained":
print("send to a person; best guess:", answer["choice"])
else:
print(answer["choice"], answer["answer_confidence"])
What it returned on the RTX 4070 SUPER:
| Message | Answer | Confidence | At 0.979 |
|---|---|---|---|
| How do I activate my new card? | activate my card | 0.993 | answered |
| Why was I charged a fee for withdrawing cash? | cash withdrawal charge | 0.996 | answered |
| My card payment was declined at the shop | declined card payment | 0.988 | answered |
| I still haven't received my new card, it's been two weeks | card arrival | 0.974 | to a person |
| What's the weather like tomorrow? | extra charge on statement | 0.060 | to a person |
- Ask the question in
questions.jsonexactly as written: the same instruction and the same 77 options in the same order.labels.jsonmaps each option back to Banking77's label name. min_confidence=0.979is the threshold fitted in training. Leave it out to get an answer for every message, or pick your own cut from the trade-off above.predict_batch(texts, questions, batch_size=32)scores many messages in shared forward passes.- To make sure you load exactly these weights, pass
expected_sha256={"model.safetensors": "<hash from SHA256SUMS>"}tolaya.load.SHA256SUMSlists every file.
Limits
- English bank customer messages in the style of one bank's app. Other banks, other products and other languages were not tested.
- It always picks one of the 77 intents. The threshold is the guard against off-topic messages, and about 0.7% of them got past it in the test above.
- One fine-tune. How much a second seed would differ was not measured.
- The threshold was fitted on 400 messages. On the test it gave 98.8% right, a little under the 99% it aimed at.
Training
Laya's own laya.train.finetune at its default settings (4 epochs, effective batch 64, RLCD loss), with two
changes, both fixed before any fine-tuned model was scored:
head_max_len512 instead of 192. The 77 options take 356 tokens, and at 192 Laya cuts them short until four pairs read the same.target_error0.01 instead of 0.10, so the threshold aims at 99% right.
It trained on 9,603 messages; Laya held back the other 400 to fit its temperature and the threshold. Training took
26 minutes on one RTX 4070 SUPER. The test was read only by the scorer, through laya.load(...).predict_batch, the
same call a user makes. 7 of the 3,080 test messages also appear in the training data; they are left in, as in the
official split.
Reproduce
The code that built and scored this model is in code/. Data, weights and runs go under ~/.laya-banking77/.
hf download VibeHash/laya-banking77 --include "code/*" --local-dir laya-banking77
cd laya-banking77/code
pip install torch --index-url https://download.pytorch.org/whl/cu126 # or the build for your GPU
pip install -r requirements.txt
python -m banking77.data # Banking77 at a pinned revision; writes and locks the test
python -m banking77.train --seed 0 # 26 minutes on an RTX 4070 SUPER
python -m banking77.evaluate --model ~/.laya-banking77/runs/seed0/model --name seed0
python -m banking77.offtopic --model ~/.laya-banking77/runs/seed0/model --name seed0
Every source is pinned by revision and SHA-256, and the scorer refuses a test file whose SHA-256 changed.
eval.json and offtopic.json hold the full scores behind this card.
Licence and credits
Apache 2.0. A fine-tune of Laya by Convai Innovations (Apache 2.0); please cite their preprint,
doi:10.20944/preprints202611.0781.v1. Trained on Banking77 by
PolyAI (CC BY 4.0): Casanueva et al., Efficient Intent Detection with Dual Sentence Encoders, NLP4ConvAI 2020.
The off-topic test uses CLINC150 (CC BY 3.0): Larson et al., An Evaluation Dataset for Intent Classification and
Out-of-Scope Prediction, EMNLP-IJCNLP 2019. See NOTICE.
- Downloads last month
- 10
Model tree for VibeHash/laya-banking77
Base model
convaiinnovations/layaDataset used to train VibeHash/laya-banking77
Evaluation results
- Accuracy on Banking77test set self-reported0.915
- Expected calibration error (15 bins) on Banking77test set self-reported0.035