Add held-out test results (LaBSE 88.35 pooled macro-F1, per-language). Flag the tamilish column as a corpus defect, not a model result: the tamilish test rendering is 60.3% OOV against tamilish train.

390e04b verified about 2 months ago

preview code

Raw History Blame Contribute Delete

10.3 kB

---
language:
- en
- si
- ta
license: cc-by-4.0
base_model: sentence-transformers/LaBSE
pipeline_tag: text-classification
tags:
- banking
- intent-classification
- labse
- multilingual
- code-mixed
- sinhala
- tamil
metrics:
- f1
---
# Swift-Support LaBSE Intent Classifier (v1.0)

Fine-tuned **LaBSE** for 77-way banking intent classification across English, Sinhala, Tamil
and their romanized code-mixed forms (Singlish, Tanglish). Part of the **Swift** support-ticket
triage project.

> **Card revised 2026-09-08.** The previous version reported a 3-epoch training run and
> explained the Indic specialists' failure with a mechanism the data does not support. Both are
> corrected below, and the correction is kept visible rather than quietly overwritten. See
> *What changed and why*.

## Model details

| | |
|---|---|
| Base model | `sentence-transformers/LaBSE` (501k vocabulary) |
| Task | 77-way intent classification (BANKING77 taxonomy) |
| Languages | English, Sinhala, Tamil, Singlish (romanized Sinhala), Tanglish (romanized Tamil) |
| Metric | **macro-F1** over 77 classes |
| Split | frozen, identity `e7b5934392cd` — drawn once on ticket `id`, then fanned to all five tracks |
| Training data | [`Swift-Support/swift-support-tickets-1.0`](https://huggingface.co/datasets/Swift-Support/swift-support-tickets-1.0) |
| Licence | CC-BY-4.0 |

**The split matters and is not the official BANKING77 split.** The same ticket exists five
times under one `id`. Scores here are macro-F1 pooled over all five tracks of the frozen
split, so they are not comparable to BANKING77 leaderboard numbers, which are English-only
accuracy on the official split.

## Evaluation

Six encoders, identical budget (6 epochs, lr 2e-5, batch 32, max length 128, fp16, seed 42),
fitted on `train` and scored on `dev`.

### Pooled and per-track macro-F1 (dev, %)

| model | English | Sinhala | Tamil | Singlish | Tanglish | **pooled** |
|---|---:|---:|---:|---:|---:|---:|
| **mmBERT** | 93.97 | 93.71 | 93.75 | 92.08 | 91.22 | **92.94** |
| **LaBSE** (this model) | 93.09 | 94.05 | 93.42 | 92.40 | 91.70 | **92.93** |
| XLM-R base | 92.68 | 93.61 | 92.40 | 90.75 | 91.55 | 92.19 |
| IndicBERT | 91.83 | 90.00 | 92.90 | 90.69 | 89.82 | 91.04 |
| TwHIN-BERT | 91.20 | 92.12 | 90.87 | 89.99 | 89.96 | 90.82 |
| MuRIL | 90.03 | **76.33** | 90.09 | 88.79 | 90.23 | 87.03 |

**LaBSE and mmBERT are statistically indistinguishable on this task.** Paired bootstrap over
1,000 resamples: Δ = −0.0001, 95% CI [−0.0045, +0.0046], p = 0.944. This card previously
called LaBSE "the intent champion". It is not; it ties. Either model is a defensible choice,
and mmBERT is the smaller of the two.

**These are dev numbers.** The held-out test numbers are now available — see below.

## Held-out test results

Fitted on `train+dev` (49,990 rows), scored **once** on the frozen split's test set
(15,395 rows). `epoch_selection = final-epoch` with `best_epoch = 6 of 6` for all three, so
no epoch-selection bias is possible.

| model | pooled | english | sinhala | tamil | singlish | tamilish |
|---|---:|---:|---:|---:|---:|---:|
| **LaBSE** (this model) | **88.35** | 94.12 | **93.19** | **93.29** | **90.34** | 69.28 |
| XLM-R base | 88.01 | 94.02 | 92.44 | 91.51 | 89.87 | **70.67** |
| mmBERT | 86.80 | 93.74 | 91.23 | 91.47 | 90.13 | 65.66 |

On dev, mmBERT and LaBSE tied and XLM-R trailed. On test the order is LaBSE > XLM-R >
mmBERT. **This is a point estimate, not a tested difference** — that run wrote no per-row
predictions, so no paired test is possible. Do not read it as mmBERT being beaten.

### ⚠️ The tamilish column is a corpus defect, not a model result

Tamilish scores 69.28 on test against **91.70 on dev**, a drop no other track shows
(english, sinhala and tamil move 1–3 points; singlish 2). The cause is a vocabulary
discontinuity between the train and test renderings of that one track:

| track | OOV vs train (dev) | **OOV vs train (test)** |
|---|---:|---:|
| english | 14.8% | 20.3% |
| singlish | 13.3% | 27.8% |
| sinhala | 13.7% | 30.2% |
| tamil | 21.6% | 29.8% |
| **tamilish** | 22.3% | **60.3%** |

The tamilish rendering of the BANKING77 **test** file was produced by a different
transliteration process from the tamilish rendering of the **train** file. Tamilish test
text also averages more characters but fewer words than tamilish train, and is the only
track with duplicate test strings.

**Do not quote 69.28 as this model's ability to handle romanized Tamil.** It measures a
train/test mismatch in the corpus. The effect is task-dependent — priority on tamilish is
0.8142 against english's 0.9229, a gap rather than a collapse — because a 3-way task
tolerates vocabulary drift that a 77-way task does not. This is being fixed for corpus
v1.1.

## What changed and why

**1. The old table was trained for 3 epochs and understated the whole roster.** Five of six
models were still improving when training stopped at 6, so even these numbers are a lower
bound. The effect on MuRIL was severe — its Tamil score was reported at 66.01% and is 90.09%
at an adequate budget.

**2. The old explanation of the Indic specialists' failure was backwards.** The previous card
said MuRIL and IndicBERT "failed outright on the pooled and code-mixed tracks because their
smaller vocabularies couldn't handle heavy romanization or English slang".

The data says the opposite. MuRIL's romanized tracks are among its *best*: Tanglish 90.23,
Singlish 88.79. It fails on exactly one track — **native-script Sinhala, at 76.33**, 13.7
points below its own English. The mechanism is in its tokenizer:

| MuRIL tokenizer | English | Sinhala | Tamil | Singlish | Tanglish |
|---|---:|---:|---:|---:|---:|
| `[UNK]` rate (% of characters) | 0.0 | **64.5** | 0.0 | 0.0 | 0.0 |

MuRIL maps **64.5% of Sinhala characters to `[UNK]`** — the characters are destroyed before
the first transformer layer, and no amount of fine-tuning recovers them. Every other model in
the roster is at 0.0% on every track.

So the correct claim is not "specialists cannot handle romanization". It is **"check the
tokenizer's `[UNK]` rate on your script before fine-tuning anything"**. Romanization is
handled fine by every model here; script coverage is what varies.

Note also that token *fertility* (tokens per word) does **not** predict downstream quality
across this roster — Spearman ρ = −0.035 against per-language deficit, versus −0.521 for
`[UNK]` rate. Fertility is a cost; `[UNK]` is destruction. They are often conflated.

## Intended use and limitations

**Intended:** routing inbound banking support tickets to one of 77 intents, as a triage aid
with a human in the loop.

**Out of scope:** non-banking domains (the taxonomy and vocabulary are banking-specific);
languages or scripts outside the five tracks; any customer-facing decision taken without
review.

**Performance is not uniform across tracks.** Tanglish trails English on every system
measured. Expect worse service for customers writing in transliteration — which, in this
setting, correlates with who those customers are.

**Intent labels are BANKING77's human gold**, unlike this project's sentiment and priority
labels, which are LLM-generated. That makes intent the most trustworthy of the three tasks
here.

## Usage

```python
from transformers import pipeline
classifier = pipeline("text-classification", model="Swift-Support/labse-intent-1.0")
print(classifier("I lost my credit card yesterday, please help me cancel it"))
```

---

## Citation

**Citing BANKING77 alone is not sufficient attribution for this corpus.** BANKING77 supplies
the English text and the 77-way intent labels — 20% of the rows and one of the three label
columns. The rest is this project's work:

| | |
|---|---|
| Rows whose text is **new** (Sinhala, Tamil, Singlish, Tanglish) | **52,308 of 65,385 — 80%** |
| Rows inherited from BANKING77 (English) | 13,077 — 20% |
| Label columns inherited | 1 (`category` / intent) |
| Label columns **added** | 2 (`sentiment`, `priority`) — 26,154 new per-ticket assignments |
| Split | the frozen `id`-level split is this project's, not BANKING77's |

If you use the translated tracks, the sentiment or priority labels, the frozen split, or any
of the models, **cite this work**. If you additionally use the English text or the intent
labels, cite BANKING77 as well. Both licences are CC-BY-4.0, so attribution is a licence
condition, not a courtesy.

### This work — the corpus

```bibtex
@misc{swift_tickets_2026,
  author       = {Sithija Seneviratne and Ruththiragayan Sutharsan and Shazan Shaheed},
  title        = {Swift Support Tickets: A Multilingual Ticket-Triage Corpus for Sinhala,
                  Tamil and English, Including Romanized Code-Mixed Text},
  year         = {2026},
  publisher    = {Hugging Face},
  howpublished = {\url{https://huggingface.co/datasets/Swift-Support/swift-support-tickets-1.0}},
  note         = {Derived from BANKING77. Adds four language tracks and the sentiment and
                  priority label columns. Paper in preparation.}
}
```

### This work — the models

```bibtex
@misc{swift_models_2026,
  author       = {Sithija Seneviratne and Ruththiragayan Sutharsan and Shazan Shaheed},
  title        = {Swift: Multilingual Baselines for Support-Ticket Triage in Sinhala,
                  Tamil and English},
  year         = {2026},
  publisher    = {Hugging Face},
  howpublished = {\url{https://huggingface.co/Swift-Support}},
  note         = {Intent, sentiment and priority classifiers. Paper in preparation.}
}
```

### BANKING77 — the source of the English text and the intent labels

```bibtex
@inproceedings{casanueva2020banking77,
  author    = {Casanueva, I{\~n}igo and Tem{\v c}inas, Tadas and Gerz, Daniela and
               Henderson, Matthew and Vuli{\'c}, Ivan},
  title     = {Efficient Intent Detection with Dual Sentence Encoders},
  booktitle = {Proceedings of the 2nd Workshop on Natural Language Processing for
               Conversational AI},
  pages     = {38--45},
  year      = {2020},
  publisher = {Association for Computational Linguistics},
  doi       = {10.18653/v1/2020.nlp4convai-1.5},
  url       = {https://aclanthology.org/2020.nlp4convai-1.5/}
}
```

	---
	language:
	- en
	- si
	- ta
	license: cc-by-4.0
	base_model: sentence-transformers/LaBSE
	pipeline_tag: text-classification
	tags:
	- banking
	- intent-classification
	- labse
	- multilingual
	- code-mixed
	- sinhala
	- tamil
	metrics:
	- f1
	---

	# Swift-Support LaBSE Intent Classifier (v1.0)

	Fine-tuned LaBSE for 77-way banking intent classification across English, Sinhala, Tamil
	and their romanized code-mixed forms (Singlish, Tanglish). Part of the Swift support-ticket
	triage project.

	> Card revised 2026-09-08. The previous version reported a 3-epoch training run and
	> explained the Indic specialists' failure with a mechanism the data does not support. Both are
	> corrected below, and the correction is kept visible rather than quietly overwritten. See
	> What changed and why.

	## Model details

	\| \| \|
	\|---\|---\|
	\| Base model \| `sentence-transformers/LaBSE` (501k vocabulary) \|
	\| Task \| 77-way intent classification (BANKING77 taxonomy) \|
	\| Languages \| English, Sinhala, Tamil, Singlish (romanized Sinhala), Tanglish (romanized Tamil) \|
	\| Metric \| macro-F1 over 77 classes \|
	\| Split \| frozen, identity `e7b5934392cd` — drawn once on ticket `id`, then fanned to all five tracks \|
	\| Training data \| [`Swift-Support/swift-support-tickets-1.0`](https://huggingface.co/datasets/Swift-Support/swift-support-tickets-1.0) \|
	\| Licence \| CC-BY-4.0 \|

	The split matters and is not the official BANKING77 split. The same ticket exists five
	times under one `id`. Scores here are macro-F1 pooled over all five tracks of the frozen
	split, so they are not comparable to BANKING77 leaderboard numbers, which are English-only
	accuracy on the official split.

	## Evaluation

	Six encoders, identical budget (6 epochs, lr 2e-5, batch 32, max length 128, fp16, seed 42),
	fitted on `train` and scored on `dev`.

	### Pooled and per-track macro-F1 (dev, %)

	\| model \| English \| Sinhala \| Tamil \| Singlish \| Tanglish \| pooled \|
	\|---\|---:\|---:\|---:\|---:\|---:\|---:\|
	\| mmBERT \| 93.97 \| 93.71 \| 93.75 \| 92.08 \| 91.22 \| 92.94 \|
	\| LaBSE (this model) \| 93.09 \| 94.05 \| 93.42 \| 92.40 \| 91.70 \| 92.93 \|
	\| XLM-R base \| 92.68 \| 93.61 \| 92.40 \| 90.75 \| 91.55 \| 92.19 \|
	\| IndicBERT \| 91.83 \| 90.00 \| 92.90 \| 90.69 \| 89.82 \| 91.04 \|
	\| TwHIN-BERT \| 91.20 \| 92.12 \| 90.87 \| 89.99 \| 89.96 \| 90.82 \|
	\| MuRIL \| 90.03 \| 76.33 \| 90.09 \| 88.79 \| 90.23 \| 87.03 \|

	LaBSE and mmBERT are statistically indistinguishable on this task. Paired bootstrap over
	1,000 resamples: Δ = −0.0001, 95% CI [−0.0045, +0.0046], p = 0.944. This card previously
	called LaBSE "the intent champion". It is not; it ties. Either model is a defensible choice,
	and mmBERT is the smaller of the two.

	These are dev numbers. The held-out test numbers are now available — see below.

	## Held-out test results

	Fitted on `train+dev` (49,990 rows), scored once on the frozen split's test set
	(15,395 rows). `epoch_selection = final-epoch` with `best_epoch = 6 of 6` for all three, so
	no epoch-selection bias is possible.

	\| model \| pooled \| english \| sinhala \| tamil \| singlish \| tamilish \|
	\|---\|---:\|---:\|---:\|---:\|---:\|---:\|
	\| LaBSE (this model) \| 88.35 \| 94.12 \| 93.19 \| 93.29 \| 90.34 \| 69.28 \|
	\| XLM-R base \| 88.01 \| 94.02 \| 92.44 \| 91.51 \| 89.87 \| 70.67 \|
	\| mmBERT \| 86.80 \| 93.74 \| 91.23 \| 91.47 \| 90.13 \| 65.66 \|

	On dev, mmBERT and LaBSE tied and XLM-R trailed. On test the order is LaBSE > XLM-R >
	mmBERT. This is a point estimate, not a tested difference — that run wrote no per-row
	predictions, so no paired test is possible. Do not read it as mmBERT being beaten.

	### ⚠️ The tamilish column is a corpus defect, not a model result

	Tamilish scores 69.28 on test against 91.70 on dev, a drop no other track shows
	(english, sinhala and tamil move 1–3 points; singlish 2). The cause is a vocabulary
	discontinuity between the train and test renderings of that one track:

	\| track \| OOV vs train (dev) \| OOV vs train (test) \|
	\|---\|---:\|---:\|
	\| english \| 14.8% \| 20.3% \|
	\| singlish \| 13.3% \| 27.8% \|
	\| sinhala \| 13.7% \| 30.2% \|
	\| tamil \| 21.6% \| 29.8% \|
	\| tamilish \| 22.3% \| 60.3% \|

	The tamilish rendering of the BANKING77 test file was produced by a different
	transliteration process from the tamilish rendering of the train file. Tamilish test
	text also averages more characters but fewer words than tamilish train, and is the only
	track with duplicate test strings.

	Do not quote 69.28 as this model's ability to handle romanized Tamil. It measures a
	train/test mismatch in the corpus. The effect is task-dependent — priority on tamilish is
	0.8142 against english's 0.9229, a gap rather than a collapse — because a 3-way task
	tolerates vocabulary drift that a 77-way task does not. This is being fixed for corpus
	v1.1.

	## What changed and why

	1. The old table was trained for 3 epochs and understated the whole roster. Five of six
	models were still improving when training stopped at 6, so even these numbers are a lower
	bound. The effect on MuRIL was severe — its Tamil score was reported at 66.01% and is 90.09%
	at an adequate budget.

	2. The old explanation of the Indic specialists' failure was backwards. The previous card
	said MuRIL and IndicBERT "failed outright on the pooled and code-mixed tracks because their
	smaller vocabularies couldn't handle heavy romanization or English slang".

	The data says the opposite. MuRIL's romanized tracks are among its best: Tanglish 90.23,
	Singlish 88.79. It fails on exactly one track — native-script Sinhala, at 76.33, 13.7
	points below its own English. The mechanism is in its tokenizer:

	\| MuRIL tokenizer \| English \| Sinhala \| Tamil \| Singlish \| Tanglish \|
	\|---\|---:\|---:\|---:\|---:\|---:\|
	\| `[UNK]` rate (% of characters) \| 0.0 \| 64.5 \| 0.0 \| 0.0 \| 0.0 \|

	MuRIL maps 64.5% of Sinhala characters to `[UNK]` — the characters are destroyed before
	the first transformer layer, and no amount of fine-tuning recovers them. Every other model in
	the roster is at 0.0% on every track.

	So the correct claim is not "specialists cannot handle romanization". It is **"check the
	tokenizer's `[UNK]` rate on your script before fine-tuning anything"**. Romanization is
	handled fine by every model here; script coverage is what varies.

	Note also that token fertility (tokens per word) does not predict downstream quality
	across this roster — Spearman ρ = −0.035 against per-language deficit, versus −0.521 for
	`[UNK]` rate. Fertility is a cost; `[UNK]` is destruction. They are often conflated.

	## Intended use and limitations

	Intended: routing inbound banking support tickets to one of 77 intents, as a triage aid
	with a human in the loop.

	Out of scope: non-banking domains (the taxonomy and vocabulary are banking-specific);
	languages or scripts outside the five tracks; any customer-facing decision taken without
	review.

	Performance is not uniform across tracks. Tanglish trails English on every system
	measured. Expect worse service for customers writing in transliteration — which, in this
	setting, correlates with who those customers are.

	Intent labels are BANKING77's human gold, unlike this project's sentiment and priority
	labels, which are LLM-generated. That makes intent the most trustworthy of the three tasks
	here.

	## Usage

	```python
	from transformers import pipeline

	classifier = pipeline("text-classification", model="Swift-Support/labse-intent-1.0")
	print(classifier("I lost my credit card yesterday, please help me cancel it"))
	```

	---

	## Citation

	Citing BANKING77 alone is not sufficient attribution for this corpus. BANKING77 supplies
	the English text and the 77-way intent labels — 20% of the rows and one of the three label
	columns. The rest is this project's work:

	\| \| \|
	\|---\|---\|
	\| Rows whose text is new (Sinhala, Tamil, Singlish, Tanglish) \| 52,308 of 65,385 — 80% \|
	\| Rows inherited from BANKING77 (English) \| 13,077 — 20% \|
	\| Label columns inherited \| 1 (`category` / intent) \|
	\| Label columns added \| 2 (`sentiment`, `priority`) — 26,154 new per-ticket assignments \|
	\| Split \| the frozen `id`-level split is this project's, not BANKING77's \|

	If you use the translated tracks, the sentiment or priority labels, the frozen split, or any
	of the models, cite this work. If you additionally use the English text or the intent
	labels, cite BANKING77 as well. Both licences are CC-BY-4.0, so attribution is a licence
	condition, not a courtesy.

	### This work — the corpus

	```bibtex
	@misc{swift_tickets_2026,
	author = {Sithija Seneviratne and Ruththiragayan Sutharsan and Shazan Shaheed},
	title = {Swift Support Tickets: A Multilingual Ticket-Triage Corpus for Sinhala,
	Tamil and English, Including Romanized Code-Mixed Text},
	year = {2026},
	publisher = {Hugging Face},
	howpublished = {\url{https://huggingface.co/datasets/Swift-Support/swift-support-tickets-1.0}},
	note = {Derived from BANKING77. Adds four language tracks and the sentiment and
	priority label columns. Paper in preparation.}
	}
	```

	### This work — the models

	```bibtex
	@misc{swift_models_2026,
	author = {Sithija Seneviratne and Ruththiragayan Sutharsan and Shazan Shaheed},
	title = {Swift: Multilingual Baselines for Support-Ticket Triage in Sinhala,
	Tamil and English},
	year = {2026},
	publisher = {Hugging Face},
	howpublished = {\url{https://huggingface.co/Swift-Support}},
	note = {Intent, sentiment and priority classifiers. Paper in preparation.}
	}
	```

	### BANKING77 — the source of the English text and the intent labels

	```bibtex
	@inproceedings{casanueva2020banking77,
	author = {Casanueva, I{\~n}igo and Tem{\v c}inas, Tadas and Gerz, Daniela and
	Henderson, Matthew and Vuli{\'c}, Ivan},
	title = {Efficient Intent Detection with Dual Sentence Encoders},
	booktitle = {Proceedings of the 2nd Workshop on Natural Language Processing for
	Conversational AI},
	pages = {38--45},
	year = {2020},
	publisher = {Association for Computational Linguistics},
	doi = {10.18653/v1/2020.nlp4convai-1.5},
	url = {https://aclanthology.org/2020.nlp4convai-1.5/}
	}
	```