--- license: apache-2.0 base_model: convaiinnovations/laya datasets: - PolyAI/banking77 language: - en pipeline_tag: text-classification tags: - laya - banking77 - intent-classification - calibrated - selective-prediction model-index: - name: laya-banking77 results: - task: type: text-classification name: Intent classification dataset: name: Banking77 type: PolyAI/banking77 split: test revision: 1846984189f2bc0ac51c6fc3a600fa9e1714e0c6 metrics: - type: accuracy value: 0.9146 name: Accuracy - type: ece value: 0.0355 name: Expected calibration error (15 bins) --- # laya-banking77 A fine-tune of [Laya](https://huggingface.co/convaiinnovations/laya) that sorts bank customer messages into the 77 intents of [Banking77](https://huggingface.co/datasets/PolyAI/banking77). It gives a probability for every intent in one forward pass, and its confidence can be trusted. So it can answer the messages it is sure about by itself and hand the rest to a person. ## Results All 3,080 messages of Banking77's test set (40 per intent), one fine-tune (seed 0), one RTX 4070 SUPER, 2026-11-13. | | Untuned Laya | This model | |---|---|---| | Right | 1,354 of 3,080 (44.0%) | 2,817 of 3,080 (91.5%) | | 95% confidence interval | 42.2 to 45.7% | 90.4 to 92.4% | | Mean stated confidence | 0.566 | 0.917 | | Calibration error (ECE, lower is better) | 0.160 | 0.035 | When it says 0.92, it is right about 92% of the time. **Answering only when sure.** During the fine-tune, Laya held back 400 training messages and fitted a threshold on them, aiming at 99% right among the answers above it. That threshold is **0.979**. Applied once to the test: - The model answers 1,523 of the 3,080 messages (49.4%) by itself, and 1,505 of those are right (98.8%). - The other 1,557 go to a person, each with the model's best guess. With the cut chosen on the test itself, which describes this test and is optimistic for new messages: 95% right while answering 91.9% of messages, 98% right at 70.2%, 99% right at 46.3%. **Messages about something else.** Banking77 has no "none of these" intent, so the model always picks one of the 77. On the 1,000 queries that [CLINC150](https://huggingface.co/datasets/clinc/clinc_oos) labels out of scope (the stock market, screen brightness, algebra), its mean confidence is 0.165, and 7 of the 1,000 (0.7%) pass the threshold. Three of those seven are banking questions after all: "can i deposit a check", "what is the fee for a cash advance", "i need to update my address". **Speed.** One message alone: median 22 ms, 95th percentile 24 ms. In batches of 32: about 10 ms a message. **Against published results.** The Banking77 paper (Casanueva et al. 2020, Table 3, full training data) reports 93.66% for fine-tuned BERT and 93.36% for USE+ConveRT. This model is about two points behind on accuracy alone, and its whole 95% interval sits below 93.66%. The paper does not report calibration or answering only when sure. This run used Laya's default fine-tune settings and was not tuned for accuracy. **Where it is weakest.** Fiat currency support (75% right), declined transfer, pending transfer, and balance not updated after bank transfer (77.5% each). The most common mix-ups are pairs a person could also confuse: fiat currency support and exchange via app, top up reverted and top up failed, card arrival and card delivery estimate. ## Use ```bash pip install laya==0.4.31 ``` ```python import json import laya from huggingface_hub import hf_hub_download REPO = "VibeHash/laya-banking77" model = laya.load(REPO) with open(hf_hub_download(REPO, "questions.json")) as f: questions = json.load(f) result = model.predict("Why was I charged a fee for withdrawing cash?", questions, min_confidence=0.979) answer = result["answers"]["intent"] if answer["abstention"] == "abstained": print("send to a person; best guess:", answer["choice"]) else: print(answer["choice"], answer["answer_confidence"]) ``` What it returned on the RTX 4070 SUPER: | Message | Answer | Confidence | At 0.979 | |---|---|---|---| | How do I activate my new card? | activate my card | 0.993 | answered | | Why was I charged a fee for withdrawing cash? | cash withdrawal charge | 0.996 | answered | | My card payment was declined at the shop | declined card payment | 0.988 | answered | | I still haven't received my new card, it's been two weeks | card arrival | 0.974 | to a person | | What's the weather like tomorrow? | extra charge on statement | 0.060 | to a person | - Ask the question in `questions.json` exactly as written: the same instruction and the same 77 options in the same order. `labels.json` maps each option back to Banking77's label name. - `min_confidence=0.979` is the threshold fitted in training. Leave it out to get an answer for every message, or pick your own cut from the trade-off above. - `predict_batch(texts, questions, batch_size=32)` scores many messages in shared forward passes. - To make sure you load exactly these weights, pass `expected_sha256={"model.safetensors": ""}` to `laya.load`. `SHA256SUMS` lists every file. ## Limits - English bank customer messages in the style of one bank's app. Other banks, other products and other languages were not tested. - It always picks one of the 77 intents. The threshold is the guard against off-topic messages, and about 0.7% of them got past it in the test above. - One fine-tune. How much a second seed would differ was not measured. - The threshold was fitted on 400 messages. On the test it gave 98.8% right, a little under the 99% it aimed at. ## Training Laya's own `laya.train.finetune` at its default settings (4 epochs, effective batch 64, RLCD loss), with two changes, both fixed before any fine-tuned model was scored: - `head_max_len` 512 instead of 192. The 77 options take 356 tokens, and at 192 Laya cuts them short until four pairs read the same. - `target_error` 0.01 instead of 0.10, so the threshold aims at 99% right. It trained on 9,603 messages; Laya held back the other 400 to fit its temperature and the threshold. Training took 26 minutes on one RTX 4070 SUPER. The test was read only by the scorer, through `laya.load(...).predict_batch`, the same call a user makes. 7 of the 3,080 test messages also appear in the training data; they are left in, as in the official split. ## Reproduce The code that built and scored this model is in `code/`. Data, weights and runs go under `~/.laya-banking77/`. ```bash hf download VibeHash/laya-banking77 --include "code/*" --local-dir laya-banking77 cd laya-banking77/code pip install torch --index-url https://download.pytorch.org/whl/cu126 # or the build for your GPU pip install -r requirements.txt python -m banking77.data # Banking77 at a pinned revision; writes and locks the test python -m banking77.train --seed 0 # 26 minutes on an RTX 4070 SUPER python -m banking77.evaluate --model ~/.laya-banking77/runs/seed0/model --name seed0 python -m banking77.offtopic --model ~/.laya-banking77/runs/seed0/model --name seed0 ``` Every source is pinned by revision and SHA-256, and the scorer refuses a test file whose SHA-256 changed. `eval.json` and `offtopic.json` hold the full scores behind this card. ## Licence and credits Apache 2.0. A fine-tune of Laya by Convai Innovations (Apache 2.0); please cite their preprint, [doi:10.20944/preprints202611.0781.v1](https://doi.org/10.20944/preprints202611.0781.v1). Trained on Banking77 by PolyAI (CC BY 4.0): Casanueva et al., *Efficient Intent Detection with Dual Sentence Encoders*, NLP4ConvAI 2020. The off-topic test uses CLINC150 (CC BY 3.0): Larson et al., *An Evaluation Dataset for Intent Classification and Out-of-Scope Prediction*, EMNLP-IJCNLP 2019. See `NOTICE`.