Update model card

c01b8da verified about 1 month ago

10.2 kB

---
license: apache-2.0
language:
- fa
base_model: convaiinnovations/laya
tags:
- laya
- persian
- farsi
- text-classification
- customer-support
- decision-model
- fine-tuned
---

# laya-fa-support

**Fine-tuned by [Ali Jahani](https://t.me/tarfandoonchannel)** · Telegram: [t.me/tarfandoonchannel](https://t.me/tarfandoonchannel)

A **Persian customer-support triage** model, fine-tuned from
[Laya](https://github.com/NandhaKishorM/laya)'s `laya-multilingual` checkpoint. It reads a
Persian support message and decides which department should handle it (`billing`,
`technical`, `cancel` or `other`) in a single forward pass. It returns a probability for every
option and generates no text, so there is nothing to parse and nothing to hallucinate.

## Results at a glance

These scores come from the held-out [`persian_fa` benchmark](https://github.com/alipyth/laya-persian-benchmark): 64 Persian
support messages the model never saw during training, asked with prompts it never saw
during training.

| Prompt | Base `laya-multilingual` | **This model** | Change |
|---|---|---|---|
| Persian instructions | 36/64 (56.2%) | **51/64 (79.7%)** | +15 cases |
| English instructions | 41/64 (64.1%) | **44/64 (68.8%)** | +3 cases |

## Quick start

```bash
pip install laya
```

```python
import laya

agent = laya.load("tarfandoon/laya-fa-support")            # add device="cuda" for a GPU

question = {
    "department": {
        "type": "choice",
        "instructions": "این پیام پشتیبانی باید به کدام بخش ارجاع داده شود؟",
        "criteria": {
            "billing": "پرداخت، کسر وجه، استرداد پول، فاکتور، قیمت و هزینه‌ی اشتراک",
            "technical": "خطا، باگ، کرش، مشکل ورود یا رمز، کندی و قطعی برنامه",
            "cancel": "کاربر می‌خواهد اشتراک را لغو کند، تمدید را متوقف کند یا حساب را ببندد یا حذف کند",
            "other": "هر چیز دیگر: تشکر، پیشنهاد، همکاری، استخدام، سؤال عمومی",
        },
    }
}

result = agent.predict({"message": "سلام، دو بار پول از حسابم کم شده ولی اشتراکم فعال نشده"}, question)
answer = result["answers"]["department"]
print(answer["choice"])              # one of: billing, technical, cancel, other
print(answer["probabilities"])       # probability for every department
print(answer["answer_confidence"])   # calibrated probability of the chosen answer
```

### Many messages at once

```python
messages = ["اپ باز نمیشه", "mikham eshterakam ro laghv konam", "مرسی از پشتیبانی‌تون"]
results = agent.predict_batch([{"message": m} for m in messages], question)
for m, r in zip(messages, results):
    print(r["answers"]["department"]["choice"], "<-", m)
```

### Send uncertain cases to a human

```python
THRESHOLD = 0.6   # choose this on your own labelled data
if answer["answer_confidence"] >= THRESHOLD:
    route_to(answer["choice"])
else:
    send_to_human(answer)
```

### Tips

- **Keep the four option keys** (`billing`, `technical`, `cancel`, `other`) and **give every
  option a full description**, like the example above. The wording can change, because the
  model was trained on several paraphrases in Persian and English. **Persian instructions
  work best** for this model (see the results).
- ⚠️ **Do not use one-word descriptions** such as `"billing": "مالی"`. In a spot check on 8 new
  messages, the full prompt above got 7/8 right, while one-word descriptions got 5/8. The
  one-word version was also confidently wrong: it labelled «اپ باز نمیشه» as `cancel` with
  0.92 confidence.
- The input can be formal or colloquial Persian, Finglish, or Persian mixed with English words.
  Arabic ي/ك and missing half-spaces are handled.
- On a CPU a decision takes roughly 150-200 ms. On a GPU it is much faster.

## Full benchmark results: before vs after

Base model: `convaiinnovations/laya` / `multilingual` @ `55cf4c4ebb4ebe31b2550e8bdf3bd21b99753851`. Scores are from the first
of three repeats, and every repeat gave identical predictions. Neither model saw these 64
messages or these two prompts during training.

### Persian instructions (`choice_fa`)

| Metric | Base | This model | Change |
|---|---|---|---|
| **Accuracy** | 36/64 | **51/64** | **+15** |
| Macro-F1 | 0.513 | 0.795 | |
| False `cancel` (would wrongly close an account) | 4 | 2 | |
| Missed `cancel` | 9 | 3 | |
| Mean answer confidence | 0.84 | 0.69 | |
| ECE (calibration error, lower is better) | 0.273 | 0.103 | |

| Family | Base | This model | Change |
|---|---|---|---|
| formal | 6/8 | 8/8 | **+2** |
| colloquial | 5/8 | 7/8 | **+2** |
| finglish | 4/8 | 4/8 | 0 |
| orthography | 4/8 | 6/8 | **+2** |
| code_mixed | 5/8 | 8/8 | **+3** |
| negation | 3/8 | 6/8 | **+3** |
| sarcasm_taarof | 5/8 | 6/8 | **+1** |
| digits | 4/8 | 6/8 | **+2** |

| Label (recall) | Base | This model |
|---|---|---|
| `billing` | 13/16 | 14/16 |
| `technical` | 14/16 | 14/16 |
| `cancel` | 7/16 | 13/16 |
| `other` | 2/16 | 10/16 |

<details><summary>Confusion matrices (rows = gold, columns = predicted)</summary>

**Base**

| | billing | technical | cancel | other |
|---|---|---|---|---|
| billing | 13 | 2 | 1 | 0 |
| technical | 1 | 14 | 1 | 0 |
| cancel | 1 | 8 | 7 | 0 |
| other | 6 | 6 | 2 | 2 |

**This model**

| | billing | technical | cancel | other |
|---|---|---|---|---|
| billing | 14 | 0 | 1 | 1 |
| technical | 1 | 14 | 1 | 0 |
| cancel | 1 | 2 | 13 | 0 |
| other | 4 | 2 | 0 | 10 |

</details>

### English instructions (`choice_en`)

| Metric | Base | This model | Change |
|---|---|---|---|
| **Accuracy** | 41/64 | **44/64** | **+3** |
| Macro-F1 | 0.633 | 0.685 | |
| False `cancel` (would wrongly close an account) | 2 | 3 | |
| Missed `cancel` | 6 | 6 | |
| Mean answer confidence | 0.82 | 0.71 | |
| ECE (calibration error, lower is better) | 0.250 | 0.108 | |

| Family | Base | This model | Change |
|---|---|---|---|
| formal | 5/8 | 8/8 | **+3** |
| colloquial | 7/8 | 6/8 | **-1** |
| finglish | 4/8 | 5/8 | **+1** |
| orthography | 5/8 | 5/8 | 0 |
| code_mixed | 6/8 | 6/8 | 0 |
| negation | 3/8 | 4/8 | **+1** |
| sarcasm_taarof | 6/8 | 5/8 | **-1** |
| digits | 5/8 | 5/8 | 0 |

| Label (recall) | Base | This model |
|---|---|---|
| `billing` | 10/16 | 10/16 |
| `technical` | 15/16 | 14/16 |
| `cancel` | 10/16 | 10/16 |
| `other` | 6/16 | 10/16 |

<details><summary>Confusion matrices (rows = gold, columns = predicted)</summary>

**Base**

| | billing | technical | cancel | other |
|---|---|---|---|---|
| billing | 10 | 3 | 2 | 1 |
| technical | 1 | 15 | 0 | 0 |
| cancel | 0 | 6 | 10 | 0 |
| other | 4 | 6 | 0 | 6 |

**This model**

| | billing | technical | cancel | other |
|---|---|---|---|---|
| billing | 10 | 2 | 2 | 2 |
| technical | 1 | 14 | 1 | 0 |
| cancel | 0 | 4 | 10 | 2 |
| other | 3 | 3 | 0 | 10 |

</details>


## Training

| | |
|---|---|
| Base checkpoint | `laya-multilingual` (mmBERT-base, 322M parameters) @ `55cf4c4ebb4ebe31b2550e8bdf3bd21b99753851` |
| Method | `top` mode: decision head + last 4 encoder layers. Token embeddings frozen. |
| Objective | cross-entropy on the gold option, AdamW, fp16 autocast |
| Data | 200 training and 40 dev Persian support messages (50 and 10 per label), all synthetic |
| Prompts | 4 paraphrased questions (2 Persian, 2 English), **none of them the benchmark prompts** |
| Robustness | options shuffled with probability 0.5; random ي/ك, half-space and digit variants |
| Model selection | best epoch by dev NLL (epoch 6) |
| Calibration | one choice temperature fitted on dev (1.27) |
| Hardware | a single NVIDIA GeForce GTX 1650 (4 GB), about 10 minutes |
| Software | laya 0.3.21, torch 2.14.0+cu126, seed 42 |

The dev set has 40 messages x 4 training prompts. Dev accuracy went from 0.562
before training to 0.700 at the selected epoch.

| Epoch | Train loss | Dev accuracy | Dev NLL |
|---|---|---|---|
| 0 (before) | - | 0.562 | 1.555 |
| 1 | 1.077 | 0.581 | 0.999 |
| 2 | 0.870 | 0.619 | 0.917 |
| 3 | 0.800 | 0.656 | 0.896 |
| 4 | 0.800 | 0.663 | 0.852 |
| 5 | 0.728 | 0.700 | 0.839 |
| 6 ✅ selected | 0.733 | 0.700 | 0.810 |
| 7 | 0.700 | 0.706 | 0.818 |
| 8 | 0.639 | 0.706 | 0.820 |

A near-duplicate check (character 3-gram Jaccard < 0.5) guarantees that no training or dev
message copies a benchmark message.

## Limitations

- **The data is small and synthetic.** The training set and the benchmark were drafted the
  same way, with AI assistance. Real customer traffic will be harder, so expect a smaller
  gain there than the benchmark shows. Evaluate on your own messages before relying on the
  model.
- With Persian instructions, these families did **not** improve: finglish (4/8).
- With English instructions, these families did **not** improve: colloquial (6/8), orthography (5/8), code_mixed (6/8), sarcasm_taarof (5/8), digits (5/8).
- With English instructions the overall change is small (+3 of 64), which is within noise for 64 cases.
- **It is sensitive to how the options are described.** Use descriptive option texts (see
  Tips). Terse one-word options made it much worse, with high confidence.
- **It is specialised to one 4-way decision.** Other Laya tasks and languages were not
  re-evaluated and may have degraded. Use the original Laya checkpoints for those.
- Confidence was calibrated on 160 dev rows only. Refit the threshold or temperature on your
  own data before gating decisions on it.

## Links

- 📊 Benchmark (data, code, all raw results): https://github.com/alipyth/laya-persian-benchmark
- 🛠️ Training code and data: https://github.com/alipyth/laya_fine_tune_fa
- 🧠 Laya: https://github.com/NandhaKishorM/laya (Apache-2.0)
- 📣 Ali Jahani: https://t.me/tarfandoonchannel

## License

Apache-2.0, the same as the base Laya weights.

	---
	license: apache-2.0
	language:
	- fa
	base_model: convaiinnovations/laya
	tags:
	- laya
	- persian
	- farsi
	- text-classification
	- customer-support
	- decision-model
	- fine-tuned
	---

	# laya-fa-support

	Fine-tuned by [Ali Jahani](https://t.me/tarfandoonchannel) · Telegram: [t.me/tarfandoonchannel](https://t.me/tarfandoonchannel)

	A Persian customer-support triage model, fine-tuned from
	[Laya](https://github.com/NandhaKishorM/laya)'s `laya-multilingual` checkpoint. It reads a
	Persian support message and decides which department should handle it (`billing`,
	`technical`, `cancel` or `other`) in a single forward pass. It returns a probability for every
	option and generates no text, so there is nothing to parse and nothing to hallucinate.

	## Results at a glance

	These scores come from the held-out [`persian_fa` benchmark](https://github.com/alipyth/laya-persian-benchmark): 64 Persian
	support messages the model never saw during training, asked with prompts it never saw
	during training.

	\| Prompt \| Base `laya-multilingual` \| This model \| Change \|
	\|---\|---\|---\|---\|
	\| Persian instructions \| 36/64 (56.2%) \| 51/64 (79.7%) \| +15 cases \|
	\| English instructions \| 41/64 (64.1%) \| 44/64 (68.8%) \| +3 cases \|

	## Quick start

	```bash
	pip install laya
	```

	```python
	import laya

	agent = laya.load("tarfandoon/laya-fa-support") # add device="cuda" for a GPU

	question = {
	"department": {
	"type": "choice",
	"instructions": "این پیام پشتیبانی باید به کدام بخش ارجاع داده شود؟",
	"criteria": {
	"billing": "پرداخت، کسر وجه، استرداد پول، فاکتور، قیمت و هزینه‌ی اشتراک",
	"technical": "خطا، باگ، کرش، مشکل ورود یا رمز، کندی و قطعی برنامه",
	"cancel": "کاربر می‌خواهد اشتراک را لغو کند، تمدید را متوقف کند یا حساب را ببندد یا حذف کند",
	"other": "هر چیز دیگر: تشکر، پیشنهاد، همکاری، استخدام، سؤال عمومی",
	},
	}
	}

	result = agent.predict({"message": "سلام، دو بار پول از حسابم کم شده ولی اشتراکم فعال نشده"}, question)
	answer = result["answers"]["department"]
	print(answer["choice"]) # one of: billing, technical, cancel, other
	print(answer["probabilities"]) # probability for every department
	print(answer["answer_confidence"]) # calibrated probability of the chosen answer
	```

	### Many messages at once

	```python
	messages = ["اپ باز نمیشه", "mikham eshterakam ro laghv konam", "مرسی از پشتیبانی‌تون"]
	results = agent.predict_batch([{"message": m} for m in messages], question)
	for m, r in zip(messages, results):
	print(r["answers"]["department"]["choice"], "<-", m)
	```

	### Send uncertain cases to a human

	```python
	THRESHOLD = 0.6 # choose this on your own labelled data
	if answer["answer_confidence"] >= THRESHOLD:
	route_to(answer["choice"])
	else:
	send_to_human(answer)
	```

	### Tips

	- Keep the four option keys (`billing`, `technical`, `cancel`, `other`) and **give every
	option a full description**, like the example above. The wording can change, because the
	model was trained on several paraphrases in Persian and English. **Persian instructions
	work best** for this model (see the results).
	- ⚠️ Do not use one-word descriptions such as `"billing": "مالی"`. In a spot check on 8 new
	messages, the full prompt above got 7/8 right, while one-word descriptions got 5/8. The
	one-word version was also confidently wrong: it labelled «اپ باز نمیشه» as `cancel` with
	0.92 confidence.
	- The input can be formal or colloquial Persian, Finglish, or Persian mixed with English words.
	Arabic ي/ك and missing half-spaces are handled.
	- On a CPU a decision takes roughly 150-200 ms. On a GPU it is much faster.

	## Full benchmark results: before vs after

	Base model: `convaiinnovations/laya` / `multilingual` @ `55cf4c4ebb4ebe31b2550e8bdf3bd21b99753851`. Scores are from the first
	of three repeats, and every repeat gave identical predictions. Neither model saw these 64
	messages or these two prompts during training.

	### Persian instructions (`choice_fa`)

	\| Metric \| Base \| This model \| Change \|
	\|---\|---\|---\|---\|
	\| Accuracy \| 36/64 \| 51/64 \| +15 \|
	\| Macro-F1 \| 0.513 \| 0.795 \| \|
	\| False `cancel` (would wrongly close an account) \| 4 \| 2 \| \|
	\| Missed `cancel` \| 9 \| 3 \| \|
	\| Mean answer confidence \| 0.84 \| 0.69 \| \|
	\| ECE (calibration error, lower is better) \| 0.273 \| 0.103 \| \|

	\| Family \| Base \| This model \| Change \|
	\|---\|---\|---\|---\|
	\| formal \| 6/8 \| 8/8 \| +2 \|
	\| colloquial \| 5/8 \| 7/8 \| +2 \|
	\| finglish \| 4/8 \| 4/8 \| 0 \|
	\| orthography \| 4/8 \| 6/8 \| +2 \|
	\| code_mixed \| 5/8 \| 8/8 \| +3 \|
	\| negation \| 3/8 \| 6/8 \| +3 \|
	\| sarcasm_taarof \| 5/8 \| 6/8 \| +1 \|
	\| digits \| 4/8 \| 6/8 \| +2 \|

	\| Label (recall) \| Base \| This model \|
	\|---\|---\|---\|
	\| `billing` \| 13/16 \| 14/16 \|
	\| `technical` \| 14/16 \| 14/16 \|
	\| `cancel` \| 7/16 \| 13/16 \|
	\| `other` \| 2/16 \| 10/16 \|

	<details><summary>Confusion matrices (rows = gold, columns = predicted)</summary>

	Base

	\| \| billing \| technical \| cancel \| other \|
	\|---\|---\|---\|---\|---\|
	\| billing \| 13 \| 2 \| 1 \| 0 \|
	\| technical \| 1 \| 14 \| 1 \| 0 \|
	\| cancel \| 1 \| 8 \| 7 \| 0 \|
	\| other \| 6 \| 6 \| 2 \| 2 \|

	This model

	\| \| billing \| technical \| cancel \| other \|
	\|---\|---\|---\|---\|---\|
	\| billing \| 14 \| 0 \| 1 \| 1 \|
	\| technical \| 1 \| 14 \| 1 \| 0 \|
	\| cancel \| 1 \| 2 \| 13 \| 0 \|
	\| other \| 4 \| 2 \| 0 \| 10 \|

	</details>

	### English instructions (`choice_en`)

	\| Metric \| Base \| This model \| Change \|
	\|---\|---\|---\|---\|
	\| Accuracy \| 41/64 \| 44/64 \| +3 \|
	\| Macro-F1 \| 0.633 \| 0.685 \| \|
	\| False `cancel` (would wrongly close an account) \| 2 \| 3 \| \|
	\| Missed `cancel` \| 6 \| 6 \| \|
	\| Mean answer confidence \| 0.82 \| 0.71 \| \|
	\| ECE (calibration error, lower is better) \| 0.250 \| 0.108 \| \|

	\| Family \| Base \| This model \| Change \|
	\|---\|---\|---\|---\|
	\| formal \| 5/8 \| 8/8 \| +3 \|
	\| colloquial \| 7/8 \| 6/8 \| -1 \|
	\| finglish \| 4/8 \| 5/8 \| +1 \|
	\| orthography \| 5/8 \| 5/8 \| 0 \|
	\| code_mixed \| 6/8 \| 6/8 \| 0 \|
	\| negation \| 3/8 \| 4/8 \| +1 \|
	\| sarcasm_taarof \| 6/8 \| 5/8 \| -1 \|
	\| digits \| 5/8 \| 5/8 \| 0 \|

	\| Label (recall) \| Base \| This model \|
	\|---\|---\|---\|
	\| `billing` \| 10/16 \| 10/16 \|
	\| `technical` \| 15/16 \| 14/16 \|
	\| `cancel` \| 10/16 \| 10/16 \|
	\| `other` \| 6/16 \| 10/16 \|

	<details><summary>Confusion matrices (rows = gold, columns = predicted)</summary>

	Base

	\| \| billing \| technical \| cancel \| other \|
	\|---\|---\|---\|---\|---\|
	\| billing \| 10 \| 3 \| 2 \| 1 \|
	\| technical \| 1 \| 15 \| 0 \| 0 \|
	\| cancel \| 0 \| 6 \| 10 \| 0 \|
	\| other \| 4 \| 6 \| 0 \| 6 \|

	This model

	\| \| billing \| technical \| cancel \| other \|
	\|---\|---\|---\|---\|---\|
	\| billing \| 10 \| 2 \| 2 \| 2 \|
	\| technical \| 1 \| 14 \| 1 \| 0 \|
	\| cancel \| 0 \| 4 \| 10 \| 2 \|
	\| other \| 3 \| 3 \| 0 \| 10 \|

	</details>


	## Training

	\| \| \|
	\|---\|---\|
	\| Base checkpoint \| `laya-multilingual` (mmBERT-base, 322M parameters) @ `55cf4c4ebb4ebe31b2550e8bdf3bd21b99753851` \|
	\| Method \| `top` mode: decision head + last 4 encoder layers. Token embeddings frozen. \|
	\| Objective \| cross-entropy on the gold option, AdamW, fp16 autocast \|
	\| Data \| 200 training and 40 dev Persian support messages (50 and 10 per label), all synthetic \|
	\| Prompts \| 4 paraphrased questions (2 Persian, 2 English), none of them the benchmark prompts \|
	\| Robustness \| options shuffled with probability 0.5; random ي/ك, half-space and digit variants \|
	\| Model selection \| best epoch by dev NLL (epoch 6) \|
	\| Calibration \| one choice temperature fitted on dev (1.27) \|
	\| Hardware \| a single NVIDIA GeForce GTX 1650 (4 GB), about 10 minutes \|
	\| Software \| laya 0.3.21, torch 2.14.0+cu126, seed 42 \|

	The dev set has 40 messages x 4 training prompts. Dev accuracy went from 0.562
	before training to 0.700 at the selected epoch.

	\| Epoch \| Train loss \| Dev accuracy \| Dev NLL \|
	\|---\|---\|---\|---\|
	\| 0 (before) \| - \| 0.562 \| 1.555 \|
	\| 1 \| 1.077 \| 0.581 \| 0.999 \|
	\| 2 \| 0.870 \| 0.619 \| 0.917 \|
	\| 3 \| 0.800 \| 0.656 \| 0.896 \|
	\| 4 \| 0.800 \| 0.663 \| 0.852 \|
	\| 5 \| 0.728 \| 0.700 \| 0.839 \|
	\| 6 ✅ selected \| 0.733 \| 0.700 \| 0.810 \|
	\| 7 \| 0.700 \| 0.706 \| 0.818 \|
	\| 8 \| 0.639 \| 0.706 \| 0.820 \|

	A near-duplicate check (character 3-gram Jaccard < 0.5) guarantees that no training or dev
	message copies a benchmark message.

	## Limitations

	- The data is small and synthetic. The training set and the benchmark were drafted the
	same way, with AI assistance. Real customer traffic will be harder, so expect a smaller
	gain there than the benchmark shows. Evaluate on your own messages before relying on the
	model.
	- With Persian instructions, these families did not improve: finglish (4/8).
	- With English instructions, these families did not improve: colloquial (6/8), orthography (5/8), code_mixed (6/8), sarcasm_taarof (5/8), digits (5/8).
	- With English instructions the overall change is small (+3 of 64), which is within noise for 64 cases.
	- It is sensitive to how the options are described. Use descriptive option texts (see
	Tips). Terse one-word options made it much worse, with high confidence.
	- It is specialised to one 4-way decision. Other Laya tasks and languages were not
	re-evaluated and may have degraded. Use the original Laya checkpoints for those.
	- Confidence was calibrated on 160 dev rows only. Refit the threshold or temperature on your
	own data before gating decisions on it.

	## Links

	- 📊 Benchmark (data, code, all raw results): https://github.com/alipyth/laya-persian-benchmark
	- 🛠️ Training code and data: https://github.com/alipyth/laya_fine_tune_fa
	- 🧠 Laya: https://github.com/NandhaKishorM/laya (Apache-2.0)
	- 📣 Ali Jahani: https://t.me/tarfandoonchannel

	## License

	Apache-2.0, the same as the base Laya weights.