Text Classification
Laya
Safetensors
English
banking77
intent-classification
calibrated
selective-prediction
Eval Results (legacy)
Instructions to use VibeHash/laya-banking77 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Laya
How to use VibeHash/laya-banking77 with Laya:
# No code snippets available yet for this library. # To use this model, check the repository files and the library's documentation. # Want to help? PRs adding snippets are welcome at: # https://github.com/huggingface/huggingface.js
- Notebooks
- Google Colab
- Kaggle
|
Download README.md from VibeHash/laya-banking77: direct link, hf CLI and curl.
- Browser
- Download file 7.85 kB
-
https://huggingface.co/VibeHash/laya-banking77/resolve/main/README.md
- Command line
-
hf download hf://VibeHash/laya-banking77/README.md
-
curl -L -o README.md https://huggingface.co/VibeHash/laya-banking77/resolve/main/README.md
7.85 kB
| license: apache-2.0 | |
| base_model: convaiinnovations/laya | |
| datasets: | |
| - PolyAI/banking77 | |
| language: | |
| - en | |
| pipeline_tag: text-classification | |
| tags: | |
| - laya | |
| - banking77 | |
| - intent-classification | |
| - calibrated | |
| - selective-prediction | |
| model-index: | |
| - name: laya-banking77 | |
| results: | |
| - task: | |
| type: text-classification | |
| name: Intent classification | |
| dataset: | |
| name: Banking77 | |
| type: PolyAI/banking77 | |
| split: test | |
| revision: 1846984189f2bc0ac51c6fc3a600fa9e1714e0c6 | |
| metrics: | |
| - type: accuracy | |
| value: 0.9146 | |
| name: Accuracy | |
| - type: ece | |
| value: 0.0355 | |
| name: Expected calibration error (15 bins) | |
| # laya-banking77 | |
| A fine-tune of [Laya](https://huggingface.co/convaiinnovations/laya) that sorts bank customer messages into the 77 | |
| intents of [Banking77](https://huggingface.co/datasets/PolyAI/banking77). It gives a probability for every intent in | |
| one forward pass, and its confidence can be trusted. So it can answer the messages it is sure about by itself and | |
| hand the rest to a person. | |
| ## Results | |
| All 3,080 messages of Banking77's test set (40 per intent), one fine-tune (seed 0), one RTX 4070 SUPER, 2026-11-13. | |
| | | Untuned Laya | This model | | |
| |---|---|---| | |
| | Right | 1,354 of 3,080 (44.0%) | 2,817 of 3,080 (91.5%) | | |
| | 95% confidence interval | 42.2 to 45.7% | 90.4 to 92.4% | | |
| | Mean stated confidence | 0.566 | 0.917 | | |
| | Calibration error (ECE, lower is better) | 0.160 | 0.035 | | |
| When it says 0.92, it is right about 92% of the time. | |
| **Answering only when sure.** During the fine-tune, Laya held back 400 training messages and fitted a threshold on | |
| them, aiming at 99% right among the answers above it. That threshold is **0.979**. Applied once to the test: | |
| - The model answers 1,523 of the 3,080 messages (49.4%) by itself, and 1,505 of those are right (98.8%). | |
| - The other 1,557 go to a person, each with the model's best guess. | |
| With the cut chosen on the test itself, which describes this test and is optimistic for new messages: 95% right | |
| while answering 91.9% of messages, 98% right at 70.2%, 99% right at 46.3%. | |
| **Messages about something else.** Banking77 has no "none of these" intent, so the model always picks one of the | |
| 77. On the 1,000 queries that [CLINC150](https://huggingface.co/datasets/clinc/clinc_oos) labels out of scope | |
| (the stock market, screen brightness, algebra), its mean confidence is 0.165, and 7 of the 1,000 (0.7%) pass the | |
| threshold. Three of those seven are banking questions after all: "can i deposit a check", "what is the fee for a | |
| cash advance", "i need to update my address". | |
| **Speed.** One message alone: median 22 ms, 95th percentile 24 ms. In batches of 32: about 10 ms a message. | |
| **Against published results.** The Banking77 paper (Casanueva et al. 2020, Table 3, full training data) reports | |
| 93.66% for fine-tuned BERT and 93.36% for USE+ConveRT. This model is about two points behind on accuracy alone, | |
| and its whole 95% interval sits below 93.66%. The paper does not report calibration or answering only when sure. | |
| This run used Laya's default fine-tune settings and was not tuned for accuracy. | |
| **Where it is weakest.** Fiat currency support (75% right), declined transfer, pending transfer, and balance not | |
| updated after bank transfer (77.5% each). The most common mix-ups are pairs a person could also confuse: fiat | |
| currency support and exchange via app, top up reverted and top up failed, card arrival and card delivery estimate. | |
| ## Use | |
| ```bash | |
| pip install laya==0.4.31 | |
| ``` | |
| ```python | |
| import json | |
| import laya | |
| from huggingface_hub import hf_hub_download | |
| REPO = "VibeHash/laya-banking77" | |
| model = laya.load(REPO) | |
| with open(hf_hub_download(REPO, "questions.json")) as f: | |
| questions = json.load(f) | |
| result = model.predict("Why was I charged a fee for withdrawing cash?", questions, min_confidence=0.979) | |
| answer = result["answers"]["intent"] | |
| if answer["abstention"] == "abstained": | |
| print("send to a person; best guess:", answer["choice"]) | |
| else: | |
| print(answer["choice"], answer["answer_confidence"]) | |
| ``` | |
| What it returned on the RTX 4070 SUPER: | |
| | Message | Answer | Confidence | At 0.979 | | |
| |---|---|---|---| | |
| | How do I activate my new card? | activate my card | 0.993 | answered | | |
| | Why was I charged a fee for withdrawing cash? | cash withdrawal charge | 0.996 | answered | | |
| | My card payment was declined at the shop | declined card payment | 0.988 | answered | | |
| | I still haven't received my new card, it's been two weeks | card arrival | 0.974 | to a person | | |
| | What's the weather like tomorrow? | extra charge on statement | 0.060 | to a person | | |
| - Ask the question in `questions.json` exactly as written: the same instruction and the same 77 options in the same | |
| order. `labels.json` maps each option back to Banking77's label name. | |
| - `min_confidence=0.979` is the threshold fitted in training. Leave it out to get an answer for every message, or | |
| pick your own cut from the trade-off above. | |
| - `predict_batch(texts, questions, batch_size=32)` scores many messages in shared forward passes. | |
| - To make sure you load exactly these weights, pass | |
| `expected_sha256={"model.safetensors": "<hash from SHA256SUMS>"}` to `laya.load`. `SHA256SUMS` lists every file. | |
| ## Limits | |
| - English bank customer messages in the style of one bank's app. Other banks, other products and other languages | |
| were not tested. | |
| - It always picks one of the 77 intents. The threshold is the guard against off-topic messages, and about 0.7% of | |
| them got past it in the test above. | |
| - One fine-tune. How much a second seed would differ was not measured. | |
| - The threshold was fitted on 400 messages. On the test it gave 98.8% right, a little under the 99% it aimed at. | |
| ## Training | |
| Laya's own `laya.train.finetune` at its default settings (4 epochs, effective batch 64, RLCD loss), with two | |
| changes, both fixed before any fine-tuned model was scored: | |
| - `head_max_len` 512 instead of 192. The 77 options take 356 tokens, and at 192 Laya cuts them short until four | |
| pairs read the same. | |
| - `target_error` 0.01 instead of 0.10, so the threshold aims at 99% right. | |
| It trained on 9,603 messages; Laya held back the other 400 to fit its temperature and the threshold. Training took | |
| 26 minutes on one RTX 4070 SUPER. The test was read only by the scorer, through `laya.load(...).predict_batch`, the | |
| same call a user makes. 7 of the 3,080 test messages also appear in the training data; they are left in, as in the | |
| official split. | |
| ## Reproduce | |
| The code that built and scored this model is in `code/`. Data, weights and runs go under `~/.laya-banking77/`. | |
| ```bash | |
| hf download VibeHash/laya-banking77 --include "code/*" --local-dir laya-banking77 | |
| cd laya-banking77/code | |
| pip install torch --index-url https://download.pytorch.org/whl/cu126 # or the build for your GPU | |
| pip install -r requirements.txt | |
| python -m banking77.data # Banking77 at a pinned revision; writes and locks the test | |
| python -m banking77.train --seed 0 # 26 minutes on an RTX 4070 SUPER | |
| python -m banking77.evaluate --model ~/.laya-banking77/runs/seed0/model --name seed0 | |
| python -m banking77.offtopic --model ~/.laya-banking77/runs/seed0/model --name seed0 | |
| ``` | |
| Every source is pinned by revision and SHA-256, and the scorer refuses a test file whose SHA-256 changed. | |
| `eval.json` and `offtopic.json` hold the full scores behind this card. | |
| ## Licence and credits | |
| Apache 2.0. A fine-tune of Laya by Convai Innovations (Apache 2.0); please cite their preprint, | |
| [doi:10.20944/preprints202611.0781.v1](https://doi.org/10.20944/preprints202611.0781.v1). Trained on Banking77 by | |
| PolyAI (CC BY 4.0): Casanueva et al., *Efficient Intent Detection with Dual Sentence Encoders*, NLP4ConvAI 2020. | |
| The off-topic test uses CLINC150 (CC BY 3.0): Larson et al., *An Evaluation Dataset for Intent Classification and | |
| Out-of-Scope Prediction*, EMNLP-IJCNLP 2019. See `NOTICE`. | |