kumarabd1996

laya loop-taxonomy v2: nine labels incl. plan (47/54 held out, plan 11/13)

226ce32 verified 2 days ago

4.7 kB

---
license: apache-2.0
base_model: convaiinnovations/laya
language: [en]
tags:
  - laya
  - decision-model
  - text-classification
  - agent-routing
  - system-1
library_name: laya
pipeline_tag: text-classification
---
# agent-task-classifier

*(laya loop-taxonomy v2 — nine labels)*

A fine-tuned [laya](https://huggingface.co/convaiinnovations/laya) checkpoint that classifies a
user request to a personal-assistant agent by **the agent-loop shape it demands** — the skeleton
of the high-level process required to fulfil it — rather than by topic.

Same interface as every laya checkpoint: text in, one typed `choice` decision out, one forward
pass, no generation. English, ~0.3 s/query on CPU.

**v2 changes the label set** (v1 had eleven labels; `git` history keeps it). v1 → v2:
`make`→`produce`; `react`→`stabilize`; `arrange`, `acquire`, `execute`→`transact`;
`tend`→`conversation` (reviewing a standing concern is a lookup; the wake that raises it is a
trigger, not a request); **`plan` is new** — setting up or changing a person's own goals, routines,
budgets and trackers.

## The nine labels

| label | the loop it demands |
|---|---|
| `conversation` | questions, quick asks, lookups, a reported measurement, a single reminder, chatter — the default, safe fallback |
| `companion` | inner-life support with no outcome in the world (reflection, therapy) |
| `learn` | build the user's own ability over sessions (practice, tutoring) |
| `diagnose` | find the cause of a specific malfunction or anomaly, including in the user's own data |
| `research` | assemble understanding from investigation (multi-source, comparison, synthesis) |
| `produce` | make an artifact judged against a standard (document, spreadsheet, script, letter) |
| `transact` | a commitment or possession in the world: book, buy, pay, transfer, or complete a procedure |
| `stabilize` | urgent or deteriorating situation — act before full understanding |
| `plan` | set up or change the person's own standing structure: a goal, routine, budget/limit, or tracker |

Boundary rule: two requests share a label iff they demand the same loop skeleton — same phases,
same iteration driver, same exit criterion. Topic is irrelevant. Labels are checked in the order
`stabilize`, `transact`, `plan`, `produce`, `diagnose`, `research`, `learn`, `companion`,
`conversation`; the first match wins. `plan` is *not* a single reminder ("remind me to call Sam"),
a lookup ("how much did I spend on dining"), a reported measurement ("I weighed 82 kg"), a document
to write ("build me a budget spreadsheet") or a payment ("transfer $500 to savings").

## Usage

```python
import laya
agent = laya.load("kumarabd1996/agent-task-classifier", device="cpu")
questions = json.load(open(hf_hub_download("kumarabd1996/agent-task-classifier", "questions.json")))
out = agent.predict("I want to keep my dining spending under $200 a month", questions)
out["answers"]["loop"]["choice"]         # "plan"
out["answers"]["loop"]["probabilities"]  # calibrated (temperatures fitted at train time)
```

Continuations: pass the request being continued in the state, the format the model was trained on:

```
Earlier request: I want to keep my dining spending under $200 a month
New request: All
```

## Evaluation (held out, never trained on)

54 items: the 24 taxonomy boundary drills (relabelled to v2) plus 30 hand-written personal-records
and finance cases.

| | accuracy | mean confidence | ECE |
|---|---|---|---|
| untuned laya | 24.1% (13/54) | 0.31 | 0.19 |
| **this model** | **87.0% (47/54)** | 0.96 | 0.09 |

`plan`: 11/13. Nothing outside `plan` was predicted as `plan`. The two misses were implicit
phrasings ("Don't let me overspend on clothes this quarter", "I keep meaning to exercise more, can
you help me actually do it?"). Remaining misses are older boundary drills (an implied purchase,
"is X worth it?" as research, a money transfer) and the model is confident when it is wrong (~0.95),
so a confidence floor does not catch them. 54 items is a small sample: treat the numbers as a
smoke test, not a benchmark.

## Training

Full fine-tune of laya on 4,754 examples (400 held back to fit calibration temperatures), 4 epochs,
effective batch 64. Data: [kumarabd1996/agent-task-classifier-dataset](https://huggingface.co/datasets/kumarabd1996/agent-task-classifier-dataset).

## Limitations

- English only; nine fixed labels; out-of-taxonomy requests resolve to `conversation`.
- Much of the training data is synthetic, so template phrasing biases it; implicit or indirect
  requests ("I keep meaning to…") are the weak spot.
- Calibrated on a small held-out slice; probabilities are not reliable above ~0.9.

	---
	license: apache-2.0
	base_model: convaiinnovations/laya
	language: [en]
	tags:
	- laya
	- decision-model
	- text-classification
	- agent-routing
	- system-1
	library_name: laya
	pipeline_tag: text-classification
	---

	# agent-task-classifier

	(laya loop-taxonomy v2 — nine labels)

	A fine-tuned [laya](https://huggingface.co/convaiinnovations/laya) checkpoint that classifies a
	user request to a personal-assistant agent by the agent-loop shape it demands — the skeleton
	of the high-level process required to fulfil it — rather than by topic.

	Same interface as every laya checkpoint: text in, one typed `choice` decision out, one forward
	pass, no generation. English, ~0.3 s/query on CPU.

	v2 changes the label set (v1 had eleven labels; `git` history keeps it). v1 → v2:
	`make`→`produce`; `react`→`stabilize`; `arrange`, `acquire`, `execute`→`transact`;
	`tend`→`conversation` (reviewing a standing concern is a lookup; the wake that raises it is a
	trigger, not a request); `plan` is new — setting up or changing a person's own goals, routines,
	budgets and trackers.

	## The nine labels

	\| label \| the loop it demands \|
	\|---\|---\|
	\| `conversation` \| questions, quick asks, lookups, a reported measurement, a single reminder, chatter — the default, safe fallback \|
	\| `companion` \| inner-life support with no outcome in the world (reflection, therapy) \|
	\| `learn` \| build the user's own ability over sessions (practice, tutoring) \|
	\| `diagnose` \| find the cause of a specific malfunction or anomaly, including in the user's own data \|
	\| `research` \| assemble understanding from investigation (multi-source, comparison, synthesis) \|
	\| `produce` \| make an artifact judged against a standard (document, spreadsheet, script, letter) \|
	\| `transact` \| a commitment or possession in the world: book, buy, pay, transfer, or complete a procedure \|
	\| `stabilize` \| urgent or deteriorating situation — act before full understanding \|
	\| `plan` \| set up or change the person's own standing structure: a goal, routine, budget/limit, or tracker \|

	Boundary rule: two requests share a label iff they demand the same loop skeleton — same phases,
	same iteration driver, same exit criterion. Topic is irrelevant. Labels are checked in the order
	`stabilize`, `transact`, `plan`, `produce`, `diagnose`, `research`, `learn`, `companion`,
	`conversation`; the first match wins. `plan` is not a single reminder ("remind me to call Sam"),
	a lookup ("how much did I spend on dining"), a reported measurement ("I weighed 82 kg"), a document
	to write ("build me a budget spreadsheet") or a payment ("transfer $500 to savings").

	## Usage

	```python
	import laya

	agent = laya.load("kumarabd1996/agent-task-classifier", device="cpu")
	questions = json.load(open(hf_hub_download("kumarabd1996/agent-task-classifier", "questions.json")))
	out = agent.predict("I want to keep my dining spending under $200 a month", questions)
	out["answers"]["loop"]["choice"] # "plan"
	out["answers"]["loop"]["probabilities"] # calibrated (temperatures fitted at train time)
	```

	Continuations: pass the request being continued in the state, the format the model was trained on:

	```
	Earlier request: I want to keep my dining spending under $200 a month
	New request: All
	```

	## Evaluation (held out, never trained on)

	54 items: the 24 taxonomy boundary drills (relabelled to v2) plus 30 hand-written personal-records
	and finance cases.

	\| \| accuracy \| mean confidence \| ECE \|
	\|---\|---\|---\|---\|
	\| untuned laya \| 24.1% (13/54) \| 0.31 \| 0.19 \|
	\| this model \| 87.0% (47/54) \| 0.96 \| 0.09 \|

	`plan`: 11/13. Nothing outside `plan` was predicted as `plan`. The two misses were implicit
	phrasings ("Don't let me overspend on clothes this quarter", "I keep meaning to exercise more, can
	you help me actually do it?"). Remaining misses are older boundary drills (an implied purchase,
	"is X worth it?" as research, a money transfer) and the model is confident when it is wrong (~0.95),
	so a confidence floor does not catch them. 54 items is a small sample: treat the numbers as a
	smoke test, not a benchmark.

	## Training

	Full fine-tune of laya on 4,754 examples (400 held back to fit calibration temperatures), 4 epochs,
	effective batch 64. Data: [kumarabd1996/agent-task-classifier-dataset](https://huggingface.co/datasets/kumarabd1996/agent-task-classifier-dataset).

	## Limitations

	- English only; nine fixed labels; out-of-taxonomy requests resolve to `conversation`.
	- Much of the training data is synthetic, so template phrasing biases it; implicit or indirect
	requests ("I keep meaning to…") are the weak spot.
	- Calibrated on a small held-out slice; probabilities are not reliable above ~0.9.