File size: 7,845 Bytes
4e09985
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
---
license: apache-2.0
base_model: convaiinnovations/laya
datasets:
- PolyAI/banking77
language:
- en
pipeline_tag: text-classification
tags:
- laya
- banking77
- intent-classification
- calibrated
- selective-prediction
model-index:
- name: laya-banking77
  results:
  - task:
      type: text-classification
      name: Intent classification
    dataset:
      name: Banking77
      type: PolyAI/banking77
      split: test
      revision: 1846984189f2bc0ac51c6fc3a600fa9e1714e0c6
    metrics:
    - type: accuracy
      value: 0.9146
      name: Accuracy
    - type: ece
      value: 0.0355
      name: Expected calibration error (15 bins)
---

# laya-banking77

A fine-tune of [Laya](https://huggingface.co/convaiinnovations/laya) that sorts bank customer messages into the 77
intents of [Banking77](https://huggingface.co/datasets/PolyAI/banking77). It gives a probability for every intent in
one forward pass, and its confidence can be trusted. So it can answer the messages it is sure about by itself and
hand the rest to a person.

## Results

All 3,080 messages of Banking77's test set (40 per intent), one fine-tune (seed 0), one RTX 4070 SUPER, 2026-11-13.

| | Untuned Laya | This model |
|---|---|---|
| Right | 1,354 of 3,080 (44.0%) | 2,817 of 3,080 (91.5%) |
| 95% confidence interval | 42.2 to 45.7% | 90.4 to 92.4% |
| Mean stated confidence | 0.566 | 0.917 |
| Calibration error (ECE, lower is better) | 0.160 | 0.035 |

When it says 0.92, it is right about 92% of the time.

**Answering only when sure.** During the fine-tune, Laya held back 400 training messages and fitted a threshold on
them, aiming at 99% right among the answers above it. That threshold is **0.979**. Applied once to the test:

- The model answers 1,523 of the 3,080 messages (49.4%) by itself, and 1,505 of those are right (98.8%).
- The other 1,557 go to a person, each with the model's best guess.

With the cut chosen on the test itself, which describes this test and is optimistic for new messages: 95% right
while answering 91.9% of messages, 98% right at 70.2%, 99% right at 46.3%.

**Messages about something else.** Banking77 has no "none of these" intent, so the model always picks one of the
77. On the 1,000 queries that [CLINC150](https://huggingface.co/datasets/clinc/clinc_oos) labels out of scope
(the stock market, screen brightness, algebra), its mean confidence is 0.165, and 7 of the 1,000 (0.7%) pass the
threshold. Three of those seven are banking questions after all: "can i deposit a check", "what is the fee for a
cash advance", "i need to update my address".

**Speed.** One message alone: median 22 ms, 95th percentile 24 ms. In batches of 32: about 10 ms a message.

**Against published results.** The Banking77 paper (Casanueva et al. 2020, Table 3, full training data) reports
93.66% for fine-tuned BERT and 93.36% for USE+ConveRT. This model is about two points behind on accuracy alone,
and its whole 95% interval sits below 93.66%. The paper does not report calibration or answering only when sure.
This run used Laya's default fine-tune settings and was not tuned for accuracy.

**Where it is weakest.** Fiat currency support (75% right), declined transfer, pending transfer, and balance not
updated after bank transfer (77.5% each). The most common mix-ups are pairs a person could also confuse: fiat
currency support and exchange via app, top up reverted and top up failed, card arrival and card delivery estimate.

## Use

```bash
pip install laya==0.4.31
```

```python
import json

import laya
from huggingface_hub import hf_hub_download

REPO = "VibeHash/laya-banking77"
model = laya.load(REPO)
with open(hf_hub_download(REPO, "questions.json")) as f:
    questions = json.load(f)

result = model.predict("Why was I charged a fee for withdrawing cash?", questions, min_confidence=0.979)
answer = result["answers"]["intent"]
if answer["abstention"] == "abstained":
    print("send to a person; best guess:", answer["choice"])
else:
    print(answer["choice"], answer["answer_confidence"])
```

What it returned on the RTX 4070 SUPER:

| Message | Answer | Confidence | At 0.979 |
|---|---|---|---|
| How do I activate my new card? | activate my card | 0.993 | answered |
| Why was I charged a fee for withdrawing cash? | cash withdrawal charge | 0.996 | answered |
| My card payment was declined at the shop | declined card payment | 0.988 | answered |
| I still haven't received my new card, it's been two weeks | card arrival | 0.974 | to a person |
| What's the weather like tomorrow? | extra charge on statement | 0.060 | to a person |

- Ask the question in `questions.json` exactly as written: the same instruction and the same 77 options in the same
  order. `labels.json` maps each option back to Banking77's label name.
- `min_confidence=0.979` is the threshold fitted in training. Leave it out to get an answer for every message, or
  pick your own cut from the trade-off above.
- `predict_batch(texts, questions, batch_size=32)` scores many messages in shared forward passes.
- To make sure you load exactly these weights, pass
  `expected_sha256={"model.safetensors": "<hash from SHA256SUMS>"}` to `laya.load`. `SHA256SUMS` lists every file.

## Limits

- English bank customer messages in the style of one bank's app. Other banks, other products and other languages
  were not tested.
- It always picks one of the 77 intents. The threshold is the guard against off-topic messages, and about 0.7% of
  them got past it in the test above.
- One fine-tune. How much a second seed would differ was not measured.
- The threshold was fitted on 400 messages. On the test it gave 98.8% right, a little under the 99% it aimed at.

## Training

Laya's own `laya.train.finetune` at its default settings (4 epochs, effective batch 64, RLCD loss), with two
changes, both fixed before any fine-tuned model was scored:

- `head_max_len` 512 instead of 192. The 77 options take 356 tokens, and at 192 Laya cuts them short until four
  pairs read the same.
- `target_error` 0.01 instead of 0.10, so the threshold aims at 99% right.

It trained on 9,603 messages; Laya held back the other 400 to fit its temperature and the threshold. Training took
26 minutes on one RTX 4070 SUPER. The test was read only by the scorer, through `laya.load(...).predict_batch`, the
same call a user makes. 7 of the 3,080 test messages also appear in the training data; they are left in, as in the
official split.

## Reproduce

The code that built and scored this model is in `code/`. Data, weights and runs go under `~/.laya-banking77/`.

```bash
hf download VibeHash/laya-banking77 --include "code/*" --local-dir laya-banking77
cd laya-banking77/code
pip install torch --index-url https://download.pytorch.org/whl/cu126   # or the build for your GPU
pip install -r requirements.txt
python -m banking77.data                  # Banking77 at a pinned revision; writes and locks the test
python -m banking77.train --seed 0        # 26 minutes on an RTX 4070 SUPER
python -m banking77.evaluate --model ~/.laya-banking77/runs/seed0/model --name seed0
python -m banking77.offtopic --model ~/.laya-banking77/runs/seed0/model --name seed0
```

Every source is pinned by revision and SHA-256, and the scorer refuses a test file whose SHA-256 changed.
`eval.json` and `offtopic.json` hold the full scores behind this card.

## Licence and credits

Apache 2.0. A fine-tune of Laya by Convai Innovations (Apache 2.0); please cite their preprint,
[doi:10.20944/preprints202611.0781.v1](https://doi.org/10.20944/preprints202611.0781.v1). Trained on Banking77 by
PolyAI (CC BY 4.0): Casanueva et al., *Efficient Intent Detection with Dual Sentence Encoders*, NLP4ConvAI 2020.
The off-topic test uses CLINC150 (CC BY 3.0): Larson et al., *An Evaluation Dataset for Intent Classification and
Out-of-Scope Prediction*, EMNLP-IJCNLP 2019. See `NOTICE`.