---
license: apache-2.0
language:
- fa
base_model: convaiinnovations/laya
tags:
- laya
- persian
- farsi
- text-classification
- customer-support
- decision-model
- fine-tuned
---
# laya-fa-support
**Fine-tuned by [Ali Jahani](https://t.me/tarfandoonchannel)** · Telegram: [t.me/tarfandoonchannel](https://t.me/tarfandoonchannel)
A **Persian customer-support triage** model, fine-tuned from
[Laya](https://github.com/NandhaKishorM/laya)'s `laya-multilingual` checkpoint. It reads a
Persian support message and decides which department should handle it (`billing`,
`technical`, `cancel` or `other`) in a single forward pass. It returns a probability for every
option and generates no text, so there is nothing to parse and nothing to hallucinate.
## Results at a glance
These scores come from the held-out [`persian_fa` benchmark](https://github.com/alipyth/laya-persian-benchmark): 64 Persian
support messages the model never saw during training, asked with prompts it never saw
during training.
| Prompt | Base `laya-multilingual` | **This model** | Change |
|---|---|---|---|
| Persian instructions | 36/64 (56.2%) | **51/64 (79.7%)** | +15 cases |
| English instructions | 41/64 (64.1%) | **44/64 (68.8%)** | +3 cases |
## Quick start
```bash
pip install laya
```
```python
import laya
agent = laya.load("tarfandoon/laya-fa-support") # add device="cuda" for a GPU
question = {
"department": {
"type": "choice",
"instructions": "این پیام پشتیبانی باید به کدام بخش ارجاع داده شود؟",
"criteria": {
"billing": "پرداخت، کسر وجه، استرداد پول، فاکتور، قیمت و هزینه‌ی اشتراک",
"technical": "خطا، باگ، کرش، مشکل ورود یا رمز، کندی و قطعی برنامه",
"cancel": "کاربر می‌خواهد اشتراک را لغو کند، تمدید را متوقف کند یا حساب را ببندد یا حذف کند",
"other": "هر چیز دیگر: تشکر، پیشنهاد، همکاری، استخدام، سؤال عمومی",
},
}
}
result = agent.predict({"message": "سلام، دو بار پول از حسابم کم شده ولی اشتراکم فعال نشده"}, question)
answer = result["answers"]["department"]
print(answer["choice"]) # one of: billing, technical, cancel, other
print(answer["probabilities"]) # probability for every department
print(answer["answer_confidence"]) # calibrated probability of the chosen answer
```
### Many messages at once
```python
messages = ["اپ باز نمیشه", "mikham eshterakam ro laghv konam", "مرسی از پشتیبانی‌تون"]
results = agent.predict_batch([{"message": m} for m in messages], question)
for m, r in zip(messages, results):
print(r["answers"]["department"]["choice"], "<-", m)
```
### Send uncertain cases to a human
```python
THRESHOLD = 0.6 # choose this on your own labelled data
if answer["answer_confidence"] >= THRESHOLD:
route_to(answer["choice"])
else:
send_to_human(answer)
```
### Tips
- **Keep the four option keys** (`billing`, `technical`, `cancel`, `other`) and **give every
option a full description**, like the example above. The wording can change, because the
model was trained on several paraphrases in Persian and English. **Persian instructions
work best** for this model (see the results).
- ⚠️ **Do not use one-word descriptions** such as `"billing": "مالی"`. In a spot check on 8 new
messages, the full prompt above got 7/8 right, while one-word descriptions got 5/8. The
one-word version was also confidently wrong: it labelled «اپ باز نمیشه» as `cancel` with
0.92 confidence.
- The input can be formal or colloquial Persian, Finglish, or Persian mixed with English words.
Arabic ي/ك and missing half-spaces are handled.
- On a CPU a decision takes roughly 150-200 ms. On a GPU it is much faster.
## Full benchmark results: before vs after
Base model: `convaiinnovations/laya` / `multilingual` @ `55cf4c4ebb4ebe31b2550e8bdf3bd21b99753851`. Scores are from the first
of three repeats, and every repeat gave identical predictions. Neither model saw these 64
messages or these two prompts during training.
### Persian instructions (`choice_fa`)
| Metric | Base | This model | Change |
|---|---|---|---|
| **Accuracy** | 36/64 | **51/64** | **+15** |
| Macro-F1 | 0.513 | 0.795 | |
| False `cancel` (would wrongly close an account) | 4 | 2 | |
| Missed `cancel` | 9 | 3 | |
| Mean answer confidence | 0.84 | 0.69 | |
| ECE (calibration error, lower is better) | 0.273 | 0.103 | |
| Family | Base | This model | Change |
|---|---|---|---|
| formal | 6/8 | 8/8 | **+2** |
| colloquial | 5/8 | 7/8 | **+2** |
| finglish | 4/8 | 4/8 | 0 |
| orthography | 4/8 | 6/8 | **+2** |
| code_mixed | 5/8 | 8/8 | **+3** |
| negation | 3/8 | 6/8 | **+3** |
| sarcasm_taarof | 5/8 | 6/8 | **+1** |
| digits | 4/8 | 6/8 | **+2** |
| Label (recall) | Base | This model |
|---|---|---|
| `billing` | 13/16 | 14/16 |
| `technical` | 14/16 | 14/16 |
| `cancel` | 7/16 | 13/16 |
| `other` | 2/16 | 10/16 |
Confusion matrices (rows = gold, columns = predicted)
**Base**
| | billing | technical | cancel | other |
|---|---|---|---|---|
| billing | 13 | 2 | 1 | 0 |
| technical | 1 | 14 | 1 | 0 |
| cancel | 1 | 8 | 7 | 0 |
| other | 6 | 6 | 2 | 2 |
**This model**
| | billing | technical | cancel | other |
|---|---|---|---|---|
| billing | 14 | 0 | 1 | 1 |
| technical | 1 | 14 | 1 | 0 |
| cancel | 1 | 2 | 13 | 0 |
| other | 4 | 2 | 0 | 10 |
Confusion matrices (rows = gold, columns = predicted)
**Base**
| | billing | technical | cancel | other |
|---|---|---|---|---|
| billing | 10 | 3 | 2 | 1 |
| technical | 1 | 15 | 0 | 0 |
| cancel | 0 | 6 | 10 | 0 |
| other | 4 | 6 | 0 | 6 |
**This model**
| | billing | technical | cancel | other |
|---|---|---|---|---|
| billing | 10 | 2 | 2 | 2 |
| technical | 1 | 14 | 1 | 0 |
| cancel | 0 | 4 | 10 | 2 |
| other | 3 | 3 | 0 | 10 |