| --- |
| license: apache-2.0 |
| base_model: convaiinnovations/laya |
| language: [en] |
| tags: |
| - laya |
| - decision-model |
| - text-classification |
| - agent-routing |
| - system-1 |
| library_name: laya |
| pipeline_tag: text-classification |
| --- |
| |
| # agent-triage-classifier |
|
|
| A fine-tuned [laya](https://huggingface.co/convaiinnovations/laya) checkpoint that decides **what a personal-assistant agent should do |
| with a request**, not what the request is about. It is the companion of |
| [agent-task-classifier](https://huggingface.co/kumarabd1996/agent-task-classifier), which says what *shape* of work a request is |
| (plan, research, diagnose, ...); this one says whether to answer, ask, start a long task, schedule it, or act on existing records. |
|
|
| Same interface as every laya checkpoint: text in, one typed `choice` decision out, one forward pass, no generation. English, ~0.2 s/query on CPU. |
|
|
| ## The six decisions |
|
|
| | decision | meaning | |
| |---|---| |
| | `answer` | a question, remark, greeting, thanks or status check: reply directly, no task | |
| | `ask` | the request cannot be done without one detail only the person has (a destination, a recipient, an amount, a time) | |
| | `start_task` | work wanted now that is a long multi-step procedure: planning a gameweek, a trip or a party, setting up a goal, budget or routine, reviewing history. Also a scheduled job that has come due | |
| | `schedule` | something to record or do later: a reminder, task, event or watch, or any request naming a later time or a repeat | |
| | `act` | change something that already exists (move, rename, complete, delete) or a short procedure done inline (a lookup, a conversion) | |
| | `unsure` | none of the above fits clearly: hand over to a stronger model | |
|
|
| ## Usage |
|
|
| ```python |
| import json |
| import laya |
| from huggingface_hub import hf_hub_download |
| |
| agent = laya.load("kumarabd1996/agent-triage-classifier", device="cpu") |
| questions = json.load(open(hf_hub_download("kumarabd1996/agent-triage-classifier", "questions.json"))) |
| out = agent.predict("remind me to call Sam at 5pm tomorrow", questions)["answers"]["decision"] |
| print(out["choice"], out["probabilities"]) # schedule, with the whole distribution |
| ``` |
|
|
| For a follow-up, pass `"Earlier request: <previous request>\nNew request: <this request>"`. The probabilities are calibrated enough to pick |
| a threshold; below it, treat the answer as `unsure`. |
|
|
| ## Training data (874 train / 97 dev rows) |
|
|
| | source | rows | notes | |
| |---|---|---| |
| | synthetic | ~490 | hand-written templates over personal-assistant domains (sports-fantasy planning, goals, finance, health, trips, reminders and watches, tool questions), with voice-transcript style noise | |
| | [MASSIVE](https://huggingface.co/datasets/SetFit/amazon_massive_intent_en-US) en-US | 312 | intents mapped onto the six decisions (alarm/calendar/list creation -> `schedule`; device and playback control -> `act`; queries and Q&A -> `answer`); CC BY 4.0, FitzGerald et al. 2022 | |
| | hand-labelled real messages | 76 | one user's assistant sessions, labelled by hand; the messages are not published | |
|
|
| No public dataset matched `start_task` or `ask` as defined here, so those come from the synthetic and real rows only. Full fine-tune from the |
| English laya base, 6 epochs, effective batch 16. |
|
|
| ## Results |
|
|
| - Own dev split (97 rows from the same sources): accuracy 0.887 (untrained base 0.196), ECE 0.063. |
| - 46 triage cases from the author's agent evaluation (23 dev + 23 held-out): **41/46**, 21/23 and 20/23. **Treat this as optimistic**: those cases |
| and the synthetic rows come from the same domain and the same author, and near-duplicates are likely. For comparison, LLM triage on the dev |
| half scored 68-75% in the same harness. Independent validation on fresh traffic is still to come (this model is first run in shadow mode). |
|
|
| ## Limitations |
|
|
| - It sees only the message (and optionally the one before it). It does not see which skills matched, whether a task is already running, or |
| the date, all of which matter to the real decision. |
| - Weakest on a named later time attached to a long task ("in 2 min, plan my gameweek"): `start_task` and `schedule` can come out nearly tied. |
| Read the probabilities, not just the label. |
| - English only; skewed to the domains above; replies that only make sense with the assistant's previous message ("yes, go ahead") were left |
| out of training and are not reliable. |
| - A routing hint for an agent, not a safety or moderation classifier. |
|
|
