Text Classification
Laya
Safetensors
English
banking77
intent-classification
calibrated
selective-prediction
Eval Results (legacy)
Instructions to use VibeHash/laya-banking77 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Laya
How to use VibeHash/laya-banking77 with Laya:
# No code snippets available yet for this library. # To use this model, check the repository files and the library's documentation. # Want to help? PRs adding snippets are welcome at: # https://github.com/huggingface/huggingface.js
- Notebooks
- Google Colab
- Kaggle
File size: 7,845 Bytes
4e09985 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 | ---
license: apache-2.0
base_model: convaiinnovations/laya
datasets:
- PolyAI/banking77
language:
- en
pipeline_tag: text-classification
tags:
- laya
- banking77
- intent-classification
- calibrated
- selective-prediction
model-index:
- name: laya-banking77
results:
- task:
type: text-classification
name: Intent classification
dataset:
name: Banking77
type: PolyAI/banking77
split: test
revision: 1846984189f2bc0ac51c6fc3a600fa9e1714e0c6
metrics:
- type: accuracy
value: 0.9146
name: Accuracy
- type: ece
value: 0.0355
name: Expected calibration error (15 bins)
---
# laya-banking77
A fine-tune of [Laya](https://huggingface.co/convaiinnovations/laya) that sorts bank customer messages into the 77
intents of [Banking77](https://huggingface.co/datasets/PolyAI/banking77). It gives a probability for every intent in
one forward pass, and its confidence can be trusted. So it can answer the messages it is sure about by itself and
hand the rest to a person.
## Results
All 3,080 messages of Banking77's test set (40 per intent), one fine-tune (seed 0), one RTX 4070 SUPER, 2026-11-13.
| | Untuned Laya | This model |
|---|---|---|
| Right | 1,354 of 3,080 (44.0%) | 2,817 of 3,080 (91.5%) |
| 95% confidence interval | 42.2 to 45.7% | 90.4 to 92.4% |
| Mean stated confidence | 0.566 | 0.917 |
| Calibration error (ECE, lower is better) | 0.160 | 0.035 |
When it says 0.92, it is right about 92% of the time.
**Answering only when sure.** During the fine-tune, Laya held back 400 training messages and fitted a threshold on
them, aiming at 99% right among the answers above it. That threshold is **0.979**. Applied once to the test:
- The model answers 1,523 of the 3,080 messages (49.4%) by itself, and 1,505 of those are right (98.8%).
- The other 1,557 go to a person, each with the model's best guess.
With the cut chosen on the test itself, which describes this test and is optimistic for new messages: 95% right
while answering 91.9% of messages, 98% right at 70.2%, 99% right at 46.3%.
**Messages about something else.** Banking77 has no "none of these" intent, so the model always picks one of the
77. On the 1,000 queries that [CLINC150](https://huggingface.co/datasets/clinc/clinc_oos) labels out of scope
(the stock market, screen brightness, algebra), its mean confidence is 0.165, and 7 of the 1,000 (0.7%) pass the
threshold. Three of those seven are banking questions after all: "can i deposit a check", "what is the fee for a
cash advance", "i need to update my address".
**Speed.** One message alone: median 22 ms, 95th percentile 24 ms. In batches of 32: about 10 ms a message.
**Against published results.** The Banking77 paper (Casanueva et al. 2020, Table 3, full training data) reports
93.66% for fine-tuned BERT and 93.36% for USE+ConveRT. This model is about two points behind on accuracy alone,
and its whole 95% interval sits below 93.66%. The paper does not report calibration or answering only when sure.
This run used Laya's default fine-tune settings and was not tuned for accuracy.
**Where it is weakest.** Fiat currency support (75% right), declined transfer, pending transfer, and balance not
updated after bank transfer (77.5% each). The most common mix-ups are pairs a person could also confuse: fiat
currency support and exchange via app, top up reverted and top up failed, card arrival and card delivery estimate.
## Use
```bash
pip install laya==0.4.31
```
```python
import json
import laya
from huggingface_hub import hf_hub_download
REPO = "VibeHash/laya-banking77"
model = laya.load(REPO)
with open(hf_hub_download(REPO, "questions.json")) as f:
questions = json.load(f)
result = model.predict("Why was I charged a fee for withdrawing cash?", questions, min_confidence=0.979)
answer = result["answers"]["intent"]
if answer["abstention"] == "abstained":
print("send to a person; best guess:", answer["choice"])
else:
print(answer["choice"], answer["answer_confidence"])
```
What it returned on the RTX 4070 SUPER:
| Message | Answer | Confidence | At 0.979 |
|---|---|---|---|
| How do I activate my new card? | activate my card | 0.993 | answered |
| Why was I charged a fee for withdrawing cash? | cash withdrawal charge | 0.996 | answered |
| My card payment was declined at the shop | declined card payment | 0.988 | answered |
| I still haven't received my new card, it's been two weeks | card arrival | 0.974 | to a person |
| What's the weather like tomorrow? | extra charge on statement | 0.060 | to a person |
- Ask the question in `questions.json` exactly as written: the same instruction and the same 77 options in the same
order. `labels.json` maps each option back to Banking77's label name.
- `min_confidence=0.979` is the threshold fitted in training. Leave it out to get an answer for every message, or
pick your own cut from the trade-off above.
- `predict_batch(texts, questions, batch_size=32)` scores many messages in shared forward passes.
- To make sure you load exactly these weights, pass
`expected_sha256={"model.safetensors": "<hash from SHA256SUMS>"}` to `laya.load`. `SHA256SUMS` lists every file.
## Limits
- English bank customer messages in the style of one bank's app. Other banks, other products and other languages
were not tested.
- It always picks one of the 77 intents. The threshold is the guard against off-topic messages, and about 0.7% of
them got past it in the test above.
- One fine-tune. How much a second seed would differ was not measured.
- The threshold was fitted on 400 messages. On the test it gave 98.8% right, a little under the 99% it aimed at.
## Training
Laya's own `laya.train.finetune` at its default settings (4 epochs, effective batch 64, RLCD loss), with two
changes, both fixed before any fine-tuned model was scored:
- `head_max_len` 512 instead of 192. The 77 options take 356 tokens, and at 192 Laya cuts them short until four
pairs read the same.
- `target_error` 0.01 instead of 0.10, so the threshold aims at 99% right.
It trained on 9,603 messages; Laya held back the other 400 to fit its temperature and the threshold. Training took
26 minutes on one RTX 4070 SUPER. The test was read only by the scorer, through `laya.load(...).predict_batch`, the
same call a user makes. 7 of the 3,080 test messages also appear in the training data; they are left in, as in the
official split.
## Reproduce
The code that built and scored this model is in `code/`. Data, weights and runs go under `~/.laya-banking77/`.
```bash
hf download VibeHash/laya-banking77 --include "code/*" --local-dir laya-banking77
cd laya-banking77/code
pip install torch --index-url https://download.pytorch.org/whl/cu126 # or the build for your GPU
pip install -r requirements.txt
python -m banking77.data # Banking77 at a pinned revision; writes and locks the test
python -m banking77.train --seed 0 # 26 minutes on an RTX 4070 SUPER
python -m banking77.evaluate --model ~/.laya-banking77/runs/seed0/model --name seed0
python -m banking77.offtopic --model ~/.laya-banking77/runs/seed0/model --name seed0
```
Every source is pinned by revision and SHA-256, and the scorer refuses a test file whose SHA-256 changed.
`eval.json` and `offtopic.json` hold the full scores behind this card.
## Licence and credits
Apache 2.0. A fine-tune of Laya by Convai Innovations (Apache 2.0); please cite their preprint,
[doi:10.20944/preprints202611.0781.v1](https://doi.org/10.20944/preprints202611.0781.v1). Trained on Banking77 by
PolyAI (CC BY 4.0): Casanueva et al., *Efficient Intent Detection with Dual Sentence Encoders*, NLP4ConvAI 2020.
The off-topic test uses CLINC150 (CC BY 3.0): Larson et al., *An Evaluation Dataset for Intent Classification and
Out-of-Scope Prediction*, EMNLP-IJCNLP 2019. See `NOTICE`.
|