README.md

---
language:
- en
- si
- ta
license: cc-by-4.0
base_model: sentence-transformers/LaBSE
pipeline_tag: text-classification
tags:
- banking
- priority-classification
- ticket-triage
- labse
- multilingual
- code-mixed
- sinhala
- tamil
metrics:
- f1
---

# Swift-Support LaBSE Priority Classifier (v1.0)

A fine-tuned **LaBSE** (Language-Agnostic BERT Sentence Embedding) model that assigns an
escalation priority — **Low / Medium / High** — to a banking support ticket written in any of five
language tracks. Built for the **Swift** support-ticket triage project, alongside
[`Swift-Support/labse-intent-1.0`](https://huggingface.co/Swift-Support/labse-intent-1.0).

## Model details

* **Base architecture:** `sentence-transformers/LaBSE` (471M parameters, 501k vocabulary)
* **Task:** 3-class text classification (priority / urgency)
* **Classes:** `Low`, `Medium`, `High`
* **Languages:** English, Sinhala, Tamil, Singlish (romanized Sinhala), Tanglish (romanized Tamil)
* **Regime:** one multilingual model over all five tracks — *not* five per-language models

The headline metric is **macro-F1**, never accuracy: the class distribution is roughly
55% Low / 36% Medium / 9% High, so accuracy flatters a model that neglects `High`.

## Evaluation

Trained on `train+dev` (49,990 rows = 9,998 tickets × 5 languages), scored **once** on the
held-out test set (15,395 rows = 3,079 tickets × 5 languages). Frozen split `e7b5934392cd`; test
tickets come from the official BANKING77 test file and were never used for model selection.

**Pooled test macro-F1: 0.8901** (accuracy 0.9008)

Per-class F1: `Low` 0.9206 · `Medium` 0.8760 · `High` 0.8735

### Against the alternatives (pooled test macro-F1)

| model | macro-F1 |
|---|---:|
| `gemma-3-1b` multitask (shared head) | 0.8904 |
| **LaBSE (this model)** | **0.8901** |
| `gemma-3-1b` (LoRA, single-task) | 0.8898 |
| mmBERT | 0.8887 |
| XLM-RoBERTa base | 0.8872 |
| TF-IDF + LinearSVC (classical champion) | 0.8722 |
| TF-IDF + logistic regression | 0.8683 |

The classical champion's 95% CI is [0.8605, 0.8831], so this model clears its upper bound — a
real, if modest, win. The Gemma multitask variant is a statistical tie, not a better model.

### Per language, on test

| track | LaBSE (this model) | classical TF-IDF | delta |
|---|---:|---:|---:|
| English | **0.9229** | 0.9032 | +0.0197 |
| Sinhala | **0.9179** | 0.8745 | +0.0434 |
| Singlish (romanized) | 0.8817 | **0.8915** | −0.0098 |
| Tamil | **0.9130** | 0.8905 | +0.0225 |
| Tanglish (romanized) | **0.8142** | 0.7994 | +0.0148 |
| **ALL (pooled)** | **0.8901** | 0.8722 | +0.0179 |

Two things worth stating plainly:

1. **The classical baseline still wins on Singlish.** LaBSE gives back most of its native-script
   advantage on romanized text — a pattern that also shows up in linear probing, where LaBSE has
   the *largest* native-minus-romanized gap of any backbone in the roster.
2. **Tanglish is the weak track**, 7–10 points below every other language for both model families.

## ⚠️ The label ceiling — read this before quoting 0.89

The training labels were generated by an LLM prompt, not by human annotators. On a 500-ticket
benchmark set that *was* annotated by hand, those prompt labels agree with human judgement at
**0.7722 macro-F1** (95% CI [0.7263, 0.8147]; raw agreement 0.804, Cohen's κ = 0.644).

This does **not** cap the number above — against the prompt labels a model could in principle
reach 1.0. It caps what the number *means*. This model has learned the labeling rule well; the
rule itself agrees with a human 77% of the time. **Quoting 0.89 as "priority accuracy" overstates
what a human reviewer would call correct.** Any external write-up should state the 0.7722 figure
alongside it.

## Usage

```python
import torch
from transformers import AutoModelForSequenceClassification, AutoTokenizer

repo = "Swift-Support/labse-priority-1.0"
tok = AutoTokenizer.from_pretrained(repo)
model = AutoModelForSequenceClassification.from_pretrained(repo).eval()

texts = ["Someone has taken money from my account and nobody is helping me!",
         "How do I activate my new card?"]

with torch.no_grad():
    batch = tok(texts, return_tensors="pt", padding=True, truncation=True, max_length=128)
    probs = model(**batch).logits.softmax(-1)

for text, p in zip(texts, probs):
    print(model.config.id2label[int(p.argmax())], f"{p.max():.3f}", "|", text)
```

Two things that will silently corrupt results if you get them wrong:

* **`max_length=128` must match training.** It is not stored in the checkpoint.
* **Read the label from `model.config.id2label`, never from a hardcoded index.** This checkpoint
  carries an explicit mapping (`0: Low, 1: Medium, 2: High`). A wrong index guess does not raise —
  it silently returns the wrong priority.

Cost: ~1.9 GB resident, roughly 100–300 ms per ticket on CPU. Load the model once at process
start, never per request.

## Training

| | |
|---|---|
| base | `sentence-transformers/LaBSE` |
| fit portion | `train+dev`, 49,990 rows |
| epochs | 3 (best epoch: 3 of 3) |
| learning rate | 2e-5 |
| batch size | 32 |
| max sequence length | 128 |
| class imbalance | `class_weight` (balanced) |
| precision | fp16 |
| hardware | Kaggle T4, ~112 rows/s, 22 min wall |

`best_epoch = 3 of 3` — the model was still improving when training stopped, which is what a
consistent labeling target looks like.

## Limitations

* **Romanized text is synthetic.** Singlish is rule-generated from Sinhala and Tanglish is
  machine-translated, so both are cleaner and more regular than text a human would type. The
  Singlish and Tanglish numbers above are an optimistic upper bound, and no romanized-specific
  conclusion from this model should be trusted until it is re-measured on human-typed data.
* **Labels are LLM-generated** — see the label ceiling section.
* **Domain-bound.** Derived from BANKING77; behaviour outside retail-banking support is untested.
* **Not calibrated.** The softmax scores are not probabilities you should threshold on without
  re-calibrating; a threshold tuned by cross-validation on a sibling task failed to transfer to
  test in this project.
* Trained and evaluated only on the five tracks listed. LaBSE covers 109 languages, but nothing
  here measures the other 104.

## Citation & provenance

Derived from [BANKING77](https://huggingface.co/datasets/PolyAI/banking77) (PolyAI, CC-BY-4.0),
translated into Sinhala and Tamil and romanized into Singlish and Tanglish. Priority labels were
generated by an LLM prompt and benchmarked against human annotation as described above.

Training data: [`Swift-Support/swift-support-tickets-1.0`](https://huggingface.co/datasets/Swift-Support/swift-support-tickets-1.0)

### How to cite

**Citing BANKING77 alone is not sufficient attribution for this corpus.** BANKING77 supplies
the English text and the 77-way intent labels — 20% of the rows and one of the three label
columns. The rest is this project's work:

| | |
|---|---|
| Rows whose text is **new** (Sinhala, Tamil, Singlish, Tanglish) | **52,308 of 65,385 — 80%** |
| Rows inherited from BANKING77 (English) | 13,077 — 20% |
| Label columns inherited | 1 (`category` / intent) |
| Label columns **added** | 2 (`sentiment`, `priority`) — 26,154 new per-ticket assignments |
| Split | the frozen `id`-level split is this project's, not BANKING77's |

If you use the translated tracks, the sentiment or priority labels, the frozen split, or any
of the models, **cite this work**. If you additionally use the English text or the intent
labels, cite BANKING77 as well. Both licences are CC-BY-4.0, so attribution is a licence
condition, not a courtesy.

### This work — the corpus

```bibtex
@misc{swift_tickets_2026,
  author       = {Sithija Seneviratne and Ruththiragayan Sutharsan and Shazan Shaheed},
  title        = {Swift Support Tickets: A Multilingual Ticket-Triage Corpus for Sinhala,
                  Tamil and English, Including Romanized Code-Mixed Text},
  year         = {2026},
  publisher    = {Hugging Face},
  howpublished = {\url{https://huggingface.co/datasets/Swift-Support/swift-support-tickets-1.0}},
  note         = {Derived from BANKING77. Adds four language tracks and the sentiment and
                  priority label columns. Paper in preparation.}
}
```

### This work — the models

```bibtex
@misc{swift_models_2026,
  author       = {Sithija Seneviratne and Ruththiragayan Sutharsan and Shazan Shaheed},
  title        = {Swift: Multilingual Baselines for Support-Ticket Triage in Sinhala,
                  Tamil and English},
  year         = {2026},
  publisher    = {Hugging Face},
  howpublished = {\url{https://huggingface.co/Swift-Support}},
  note         = {Intent, sentiment and priority classifiers. Paper in preparation.}
}
```

### BANKING77 — the source of the English text and the intent labels

```bibtex
@inproceedings{casanueva2020banking77,
  author    = {Casanueva, I{\~n}igo and Tem{\v c}inas, Tadas and Gerz, Daniela and
               Henderson, Matthew and Vuli{\'c}, Ivan},
  title     = {Efficient Intent Detection with Dual Sentence Encoders},
  booktitle = {Proceedings of the 2nd Workshop on Natural Language Processing for
               Conversational AI},
  pages     = {38--45},
  year      = {2020},
  publisher = {Association for Computational Linguistics},
  doi       = {10.18653/v1/2020.nlp4convai-1.5},
  url       = {https://aclanthology.org/2020.nlp4convai-1.5/}
}
```