agent-triage-classifier

A fine-tuned laya checkpoint that decides what a personal-assistant agent should do with a request, not what the request is about. It is the companion of agent-task-classifier, which says what shape of work a request is (plan, research, diagnose, ...); this one says whether to answer, ask, start a long task, schedule it, or act on existing records.

Same interface as every laya checkpoint: text in, one typed choice decision out, one forward pass, no generation. English, ~0.2 s/query on CPU.

The six decisions

decision	meaning
answer	a question, remark, greeting, thanks or status check: reply directly, no task
ask	the request cannot be done without one detail only the person has (a destination, a recipient, an amount, a time)
start_task	work wanted now that is a long multi-step procedure: planning a gameweek, a trip or a party, setting up a goal, budget or routine, reviewing history. Also a scheduled job that has come due
schedule	something to record or do later: a reminder, task, event or watch, or any request naming a later time or a repeat
act	change something that already exists (move, rename, complete, delete) or a short procedure done inline (a lookup, a conversion)
unsure	none of the above fits clearly: hand over to a stronger model

Usage

import json
import laya
from huggingface_hub import hf_hub_download

agent = laya.load("kumarabd1996/agent-triage-classifier", device="cpu")
questions = json.load(open(hf_hub_download("kumarabd1996/agent-triage-classifier", "questions.json")))
out = agent.predict("remind me to call Sam at 5pm tomorrow", questions)["answers"]["decision"]
print(out["choice"], out["probabilities"])      # schedule, with the whole distribution

For a follow-up, pass "Earlier request: <previous request>\nNew request: <this request>". The probabilities are calibrated enough to pick a threshold; below it, treat the answer as unsure.

Training data (874 train / 97 dev rows)

source	rows	notes
synthetic	~490	hand-written templates over personal-assistant domains (sports-fantasy planning, goals, finance, health, trips, reminders and watches, tool questions), with voice-transcript style noise
MASSIVE en-US	312	intents mapped onto the six decisions (alarm/calendar/list creation -> schedule; device and playback control -> act; queries and Q&A -> answer); CC BY 4.0, FitzGerald et al. 2022
hand-labelled real messages	76	one user's assistant sessions, labelled by hand; the messages are not published

No public dataset matched start_task or ask as defined here, so those come from the synthetic and real rows only. Full fine-tune from the English laya base, 6 epochs, effective batch 16.

Results

• Own dev split (97 rows from the same sources): accuracy 0.887 (untrained base 0.196), ECE 0.063.
• 46 triage cases from the author's agent evaluation (23 dev + 23 held-out): 41/46, 21/23 and 20/23. Treat this as optimistic: those cases and the synthetic rows come from the same domain and the same author, and near-duplicates are likely. For comparison, LLM triage on the dev half scored 68-75% in the same harness. Independent validation on fresh traffic is still to come (this model is first run in shadow mode).

Limitations

• It sees only the message (and optionally the one before it). It does not see which skills matched, whether a task is already running, or the date, all of which matter to the real decision.
• Weakest on a named later time attached to a long task ("in 2 min, plan my gameweek"): start_task and schedule can come out nearly tied. Read the probabilities, not just the label.
• English only; skewed to the domains above; replies that only make sense with the assistant's previous message ("yes, go ahead") were left out of training and are not reliable.
• A routing hint for an agent, not a safety or moderation classifier.