laya-banking77

A fine-tune of Laya that sorts bank customer messages into the 77 intents of Banking77. It gives a probability for every intent in one forward pass, and its confidence can be trusted. So it can answer the messages it is sure about by itself and hand the rest to a person.

Results

All 3,080 messages of Banking77's test set (40 per intent), one fine-tune (seed 0), one RTX 4070 SUPER, 2026-11-13.

Untuned Laya This model
Right 1,354 of 3,080 (44.0%) 2,817 of 3,080 (91.5%)
95% confidence interval 42.2 to 45.7% 90.4 to 92.4%
Mean stated confidence 0.566 0.917
Calibration error (ECE, lower is better) 0.160 0.035

When it says 0.92, it is right about 92% of the time.

Answering only when sure. During the fine-tune, Laya held back 400 training messages and fitted a threshold on them, aiming at 99% right among the answers above it. That threshold is 0.979. Applied once to the test:

  • The model answers 1,523 of the 3,080 messages (49.4%) by itself, and 1,505 of those are right (98.8%).
  • The other 1,557 go to a person, each with the model's best guess.

With the cut chosen on the test itself, which describes this test and is optimistic for new messages: 95% right while answering 91.9% of messages, 98% right at 70.2%, 99% right at 46.3%.

Messages about something else. Banking77 has no "none of these" intent, so the model always picks one of the 77. On the 1,000 queries that CLINC150 labels out of scope (the stock market, screen brightness, algebra), its mean confidence is 0.165, and 7 of the 1,000 (0.7%) pass the threshold. Three of those seven are banking questions after all: "can i deposit a check", "what is the fee for a cash advance", "i need to update my address".

Speed. One message alone: median 22 ms, 95th percentile 24 ms. In batches of 32: about 10 ms a message.

Against published results. The Banking77 paper (Casanueva et al. 2020, Table 3, full training data) reports 93.66% for fine-tuned BERT and 93.36% for USE+ConveRT. This model is about two points behind on accuracy alone, and its whole 95% interval sits below 93.66%. The paper does not report calibration or answering only when sure. This run used Laya's default fine-tune settings and was not tuned for accuracy.

Where it is weakest. Fiat currency support (75% right), declined transfer, pending transfer, and balance not updated after bank transfer (77.5% each). The most common mix-ups are pairs a person could also confuse: fiat currency support and exchange via app, top up reverted and top up failed, card arrival and card delivery estimate.

Use

pip install laya==0.4.31
import json

import laya
from huggingface_hub import hf_hub_download

REPO = "VibeHash/laya-banking77"
model = laya.load(REPO)
with open(hf_hub_download(REPO, "questions.json")) as f:
    questions = json.load(f)

result = model.predict("Why was I charged a fee for withdrawing cash?", questions, min_confidence=0.979)
answer = result["answers"]["intent"]
if answer["abstention"] == "abstained":
    print("send to a person; best guess:", answer["choice"])
else:
    print(answer["choice"], answer["answer_confidence"])

What it returned on the RTX 4070 SUPER:

Message Answer Confidence At 0.979
How do I activate my new card? activate my card 0.993 answered
Why was I charged a fee for withdrawing cash? cash withdrawal charge 0.996 answered
My card payment was declined at the shop declined card payment 0.988 answered
I still haven't received my new card, it's been two weeks card arrival 0.974 to a person
What's the weather like tomorrow? extra charge on statement 0.060 to a person
  • Ask the question in questions.json exactly as written: the same instruction and the same 77 options in the same order. labels.json maps each option back to Banking77's label name.
  • min_confidence=0.979 is the threshold fitted in training. Leave it out to get an answer for every message, or pick your own cut from the trade-off above.
  • predict_batch(texts, questions, batch_size=32) scores many messages in shared forward passes.
  • To make sure you load exactly these weights, pass expected_sha256={"model.safetensors": "<hash from SHA256SUMS>"} to laya.load. SHA256SUMS lists every file.

Limits

  • English bank customer messages in the style of one bank's app. Other banks, other products and other languages were not tested.
  • It always picks one of the 77 intents. The threshold is the guard against off-topic messages, and about 0.7% of them got past it in the test above.
  • One fine-tune. How much a second seed would differ was not measured.
  • The threshold was fitted on 400 messages. On the test it gave 98.8% right, a little under the 99% it aimed at.

Training

Laya's own laya.train.finetune at its default settings (4 epochs, effective batch 64, RLCD loss), with two changes, both fixed before any fine-tuned model was scored:

  • head_max_len 512 instead of 192. The 77 options take 356 tokens, and at 192 Laya cuts them short until four pairs read the same.
  • target_error 0.01 instead of 0.10, so the threshold aims at 99% right.

It trained on 9,603 messages; Laya held back the other 400 to fit its temperature and the threshold. Training took 26 minutes on one RTX 4070 SUPER. The test was read only by the scorer, through laya.load(...).predict_batch, the same call a user makes. 7 of the 3,080 test messages also appear in the training data; they are left in, as in the official split.

Reproduce

The code that built and scored this model is in code/. Data, weights and runs go under ~/.laya-banking77/.

hf download VibeHash/laya-banking77 --include "code/*" --local-dir laya-banking77
cd laya-banking77/code
pip install torch --index-url https://download.pytorch.org/whl/cu126   # or the build for your GPU
pip install -r requirements.txt
python -m banking77.data                  # Banking77 at a pinned revision; writes and locks the test
python -m banking77.train --seed 0        # 26 minutes on an RTX 4070 SUPER
python -m banking77.evaluate --model ~/.laya-banking77/runs/seed0/model --name seed0
python -m banking77.offtopic --model ~/.laya-banking77/runs/seed0/model --name seed0

Every source is pinned by revision and SHA-256, and the scorer refuses a test file whose SHA-256 changed. eval.json and offtopic.json hold the full scores behind this card.

Licence and credits

Apache 2.0. A fine-tune of Laya by Convai Innovations (Apache 2.0); please cite their preprint, doi:10.20944/preprints202611.0781.v1. Trained on Banking77 by PolyAI (CC BY 4.0): Casanueva et al., Efficient Intent Detection with Dual Sentence Encoders, NLP4ConvAI 2020. The off-topic test uses CLINC150 (CC BY 3.0): Larson et al., An Evaluation Dataset for Intent Classification and Out-of-Scope Prediction, EMNLP-IJCNLP 2019. See NOTICE.

Downloads last month
10
Safetensors
Model size
0.4B params
Tensor type
F16
ยท
Inference Providers NEW
This model isn't deployed by any Inference Provider. ๐Ÿ™‹ Ask for provider support

Model tree for VibeHash/laya-banking77

Finetuned
(228)
this model

Dataset used to train VibeHash/laya-banking77

Evaluation results