File size: 4,415 Bytes
a90e526 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 | ---
license: apache-2.0
base_model: convaiinnovations/laya
language: [en]
tags:
- laya
- decision-model
- text-classification
- agent-routing
- system-1
library_name: laya
pipeline_tag: text-classification
---
# agent-triage-classifier
A fine-tuned [laya](https://huggingface.co/convaiinnovations/laya) checkpoint that decides **what a personal-assistant agent should do
with a request**, not what the request is about. It is the companion of
[agent-task-classifier](https://huggingface.co/kumarabd1996/agent-task-classifier), which says what *shape* of work a request is
(plan, research, diagnose, ...); this one says whether to answer, ask, start a long task, schedule it, or act on existing records.
Same interface as every laya checkpoint: text in, one typed `choice` decision out, one forward pass, no generation. English, ~0.2 s/query on CPU.
## The six decisions
| decision | meaning |
|---|---|
| `answer` | a question, remark, greeting, thanks or status check: reply directly, no task |
| `ask` | the request cannot be done without one detail only the person has (a destination, a recipient, an amount, a time) |
| `start_task` | work wanted now that is a long multi-step procedure: planning a gameweek, a trip or a party, setting up a goal, budget or routine, reviewing history. Also a scheduled job that has come due |
| `schedule` | something to record or do later: a reminder, task, event or watch, or any request naming a later time or a repeat |
| `act` | change something that already exists (move, rename, complete, delete) or a short procedure done inline (a lookup, a conversion) |
| `unsure` | none of the above fits clearly: hand over to a stronger model |
## Usage
```python
import json
import laya
from huggingface_hub import hf_hub_download
agent = laya.load("kumarabd1996/agent-triage-classifier", device="cpu")
questions = json.load(open(hf_hub_download("kumarabd1996/agent-triage-classifier", "questions.json")))
out = agent.predict("remind me to call Sam at 5pm tomorrow", questions)["answers"]["decision"]
print(out["choice"], out["probabilities"]) # schedule, with the whole distribution
```
For a follow-up, pass `"Earlier request: <previous request>\nNew request: <this request>"`. The probabilities are calibrated enough to pick
a threshold; below it, treat the answer as `unsure`.
## Training data (874 train / 97 dev rows)
| source | rows | notes |
|---|---|---|
| synthetic | ~490 | hand-written templates over personal-assistant domains (sports-fantasy planning, goals, finance, health, trips, reminders and watches, tool questions), with voice-transcript style noise |
| [MASSIVE](https://huggingface.co/datasets/SetFit/amazon_massive_intent_en-US) en-US | 312 | intents mapped onto the six decisions (alarm/calendar/list creation -> `schedule`; device and playback control -> `act`; queries and Q&A -> `answer`); CC BY 4.0, FitzGerald et al. 2022 |
| hand-labelled real messages | 76 | one user's assistant sessions, labelled by hand; the messages are not published |
No public dataset matched `start_task` or `ask` as defined here, so those come from the synthetic and real rows only. Full fine-tune from the
English laya base, 6 epochs, effective batch 16.
## Results
- Own dev split (97 rows from the same sources): accuracy 0.887 (untrained base 0.196), ECE 0.063.
- 46 triage cases from the author's agent evaluation (23 dev + 23 held-out): **41/46**, 21/23 and 20/23. **Treat this as optimistic**: those cases
and the synthetic rows come from the same domain and the same author, and near-duplicates are likely. For comparison, LLM triage on the dev
half scored 68-75% in the same harness. Independent validation on fresh traffic is still to come (this model is first run in shadow mode).
## Limitations
- It sees only the message (and optionally the one before it). It does not see which skills matched, whether a task is already running, or
the date, all of which matter to the real decision.
- Weakest on a named later time attached to a long task ("in 2 min, plan my gameweek"): `start_task` and `schedule` can come out nearly tied.
Read the probabilities, not just the label.
- English only; skewed to the domains above; replies that only make sense with the assistant's previous message ("yes, go ahead") were left
out of training and are not reliable.
- A routing hint for an agent, not a safety or moderation classifier.
|