laya loop-taxonomy v2: nine labels incl. plan (47/54 held out, plan 11/13)

README.md

@@ -14,7 +14,7 @@ pipeline_tag: text-classification
 
 # agent-task-classifier
 
-*(laya loop-taxonomy v1)*
+*(laya loop-taxonomy v2 — nine labels)*
 
 A fine-tuned [laya](https://huggingface.co/convaiinnovations/laya) checkpoint that classifies a
 user request to a personal-assistant agent by **the agent-loop shape it demands** — the skeleton
@@ -23,25 +23,32 @@ of the high-level process required to fulfil it — rather than by topic.
 Same interface as every laya checkpoint: text in, one typed `choice` decision out, one forward
 pass, no generation. English, ~0.3 s/query on CPU.
 
-## The eleven labels
+**v2 changes the label set** (v1 had eleven labels; `git` history keeps it). v1 → v2:
+`make`→`produce`; `react`→`stabilize`; `arrange`, `acquire`, `execute`→`transact`;
+`tend`→`conversation` (reviewing a standing concern is a lookup; the wake that raises it is a
+trigger, not a request); **`plan` is new** — setting up or changing a person's own goals, routines,
+budgets and trackers.
+
+## The nine labels
 
 | label | the loop it demands |
 |---|---|
-| `conversation` | answer/act directly — the default, safe fallback |
-| `diagnose` | find the cause of a specific malfunction or anomaly |
-| `research` | assemble understanding from investigation (multi-source, synthesis) |
-| `arrange` | commit a real-world arrangement — book, reserve, schedule, hire |
-| `acquire` | obtain a purchasable good, and verify it arrived |
-| `execute` | complete a known, externally-defined procedure (renew, file, claim) |
-| `make` | produce an artifact judged against a standard (tests, spec, taste) |
-| `tend` | review/maintain a standing concern (budget, health, home, wardrobe) |
-| `learn` | build the user's own ability over sessions (practice, tutoring) |
+| `conversation` | questions, quick asks, lookups, a reported measurement, a single reminder, chatter — the default, safe fallback |
 | `companion` | inner-life support with no outcome in the world (reflection, therapy) |
-| `react` | urgent/deteriorating situation — act before full understanding |
+| `learn` | build the user's own ability over sessions (practice, tutoring) |
+| `diagnose` | find the cause of a specific malfunction or anomaly, including in the user's own data |
+| `research` | assemble understanding from investigation (multi-source, comparison, synthesis) |
+| `produce` | make an artifact judged against a standard (document, spreadsheet, script, letter) |
+| `transact` | a commitment or possession in the world: book, buy, pay, transfer, or complete a procedure |
+| `stabilize` | urgent or deteriorating situation — act before full understanding |
+| `plan` | set up or change the person's own standing structure: a goal, routine, budget/limit, or tracker |
 
 Boundary rule: two requests share a label iff they demand the same loop skeleton — same phases,
-same iteration driver, same exit criterion. Topic is irrelevant (`conversation` spans finance,
-wardrobe, memories; `make` spans code, letters, meal plans).
+same iteration driver, same exit criterion. Topic is irrelevant. Labels are checked in the order
+`stabilize`, `transact`, `plan`, `produce`, `diagnose`, `research`, `learn`, `companion`,
+`conversation`; the first match wins. `plan` is *not* a single reminder ("remind me to call Sam"),
+a lookup ("how much did I spend on dining"), a reported measurement ("I weighed 82 kg"), a document
+to write ("build me a budget spreadsheet") or a payment ("transfer $500 to savings").
 
 ## Usage
 
@@ -49,103 +56,44 @@ wardrobe, memories; `make` spans code, letters, meal plans).
 import laya
 
 agent = laya.load("kumarabd1996/agent-task-classifier", device="cpu")
-questions = {
-    "loop": {
-        "type": "choice",
-        "instructions": "What does this request (or the request it continues) demand?",
-        "criteria": { ... },  # shipped as questions.json in this repo
-    }
-}
-out = agent.predict("the user's message", questions)
-out["answers"]["loop"]["choice"]         # e.g. "arrange"
+questions = json.load(open(hf_hub_download("kumarabd1996/agent-task-classifier", "questions.json")))
+out = agent.predict("I want to keep my dining spending under $200 a month", questions)
+out["answers"]["loop"]["choice"]         # "plan"
 out["answers"]["loop"]["probabilities"]  # calibrated (temperatures fitted at train time)
 ```
 
 Continuations: pass the request being continued in the state, the format the model was trained on:
 
 ```
-Earlier request: What are my biggest categories?
-New request: Continue the bulk update
+Earlier request: I want to keep my dining spending under $200 a month
+New request: All
 ```
 
-## Serving
+## Evaluation (held out, never trained on)
 
-A ready-made HTTP image (CPU, weights baked in) is published from
-[realnighthawk/classifier](https://github.com/realnighthawk/classifier):
+54 items: the 24 taxonomy boundary drills (relabelled to v2) plus 30 hand-written personal-records
+and finance cases.
 
-```
-docker run -p 8000:8000 ghcr.io/realnighthawk/classifier:latest
-curl localhost:8000/classify -H 'content-type: application/json' \
-  -d '{"text":"book me a table for two at 8pm friday"}'
-# {"label":"arrange","probabilities":{...}}
-```
+| | accuracy | mean confidence | ECE |
+|---|---|---|---|
+| untuned laya | 24.1% (13/54) | 0.31 | 0.19 |
+| **this model** | **87.0% (47/54)** | 0.96 | 0.09 |
 
-`POST /classify` takes `{text, earlier?}`; `earlier` is the request being continued.
+`plan`: 11/13. Nothing outside `plan` was predicted as `plan`. The two misses were implicit
+phrasings ("Don't let me overspend on clothes this quarter", "I keep meaning to exercise more, can
+you help me actually do it?"). Remaining misses are older boundary drills (an implied purchase,
+"is X worth it?" as research, a money transfer) and the model is confident when it is wrong (~0.95),
+so a confidence floor does not catch them. 54 items is a small sample: treat the numbers as a
+smoke test, not a benchmark.
 
 ## Training
 
-- **Base**: `convaiinnovations/laya` (English checkpoint; ModernBERT-large + decision head), Apache-2.0.
-- **Corpus**: 5,280 items, 8 sources — MultiWOZ 2.1 booking utterances (~990, arrange), MASSIVE
-  en-US slices (~920, acquire/arrange/conversation), OASST2 prompts + continuations (~1,060,
-  teacher-labeled), synthetic items (~2,315) covering all eleven shapes (labels by construction,
-  incl. 34 hand-written boundary cases), and 1,056 continuation-form items
-  (`Earlier request: … / New request: …`). Per-class caps keep the largest class ≤ ~28%.
-  Released as a dataset: `kumarabd1996/agent-task-classifier-dataset`.
-- **Hyperparameters**: 4 epochs, effective batch 64 (micro 4 × accum 16), encoder lr 2.5e-5 /
-  head lr 1e-4, RLCD objective (default), option-order augmentation, max-len 256.
-- **Hardware**: one Mac, CPU only (~2 hours). A frozen-encoder variant failed to learn the
-  taxonomy (37% in-distribution); encoder adaptation is required — matching laya's own findings.
-- Temperatures fitted automatically on a held-out calibration slice (choice temperature ≈ 3.82).
-
-## Evaluation
-
-Boundary drills: 24 held-out hand-written cases targeting the taxonomy's hard boundaries
-(info-vs-procedure, understanding-vs-commitment, urgency-vs-worry, artifact-vs-investigation).
-
-| | accuracy | ECE | mean confidence |
-|---|---|---|---|
-| base `convaiinnovations/laya` (zero-shot) | 0.29 (7/24) | 0.54 | 0.83 (miscalibrated) |
-| **this checkpoint** | **0.83 (20/24)** | **0.13** | 0.96 |
-
-The 4 remaining misses, all at genuine boundaries: "Is Bali worth visiting in December?"
-(research, predicted arrange), "What does renewing a passport require?" (conversation,
-predicted execute), "Why have I been so tired lately?" (diagnose, predicted conversation),
-"Sign me up for the best EV lease deal you find." (arrange, predicted conversation).
+Full fine-tune of laya on 4,754 examples (400 held back to fit calibration temperatures), 4 epochs,
+effective batch 64. Data: [kumarabd1996/agent-task-classifier-dataset](https://huggingface.co/datasets/kumarabd1996/agent-task-classifier-dataset).
 
 ## Limitations
 
-- Trained largely on synthetic + public chat corpora; real user traffic is under-represented.
-- English only (the base multilingual checkpoint was not used).
-- The eleven labels are fixed; requests outside them resolve to `conversation` by design.
-- Continuations require the caller to supply the continued request in the state.
-- Not a safety classifier — do not use for guardrails or moderation.
-
-## Related work
-
-Small-model routing exists on two other axes, both public on the Hub:
-
-- **Capability buckets** — e.g. [ai-systems/agent-task-classifier](https://huggingface.co/ai-systems/agent-task-classifier),
-  a TF-IDF reference model over tool/subsystem categories (browser, retrieval, verification, …).
-  It answers *which capability a task needs*.
-- **Model-choice routing** — e.g. [subhash4face/agentic-ai-routing-task-classifier](https://huggingface.co/subhash4face/agentic-ai-routing-task-classifier),
-  a DeBERTa fine-tune over LLM task types (QA, coding, summarization, long-context, …).
-  It answers *which model should handle a prompt*.
-
-This model answers a third question: **which execution loop the task demands** — the only one of the
-three that decides how the agent's own work should proceed (phases, iteration driver, exit
-criteria) rather than which resource to engage. Capability and loop are orthogonal — a `diagnose`
-task may need retrieval tools, a `conversation` task may need none — and in an agent harness the
-loop decision is the routing decision that determines everything downstream.
-
-## Training data provenance
-
-MultiWOZ 2.1 (research use, MIT-licensed distribution) · MASSIVE en-US (CC-BY-4.0, Amazon) ·
-OASST2 (Apache-2.0, OpenAssistant) · synthetic/generated items (Apache-2.0 with this release).
-No user/private data was used in training.
-
-## Files
-
-`model.safetensors` (weights, incl. encoder) · `encoder/config.json` · `tokenizer/` ·
-`questions.json` (the taxonomy question, ready to pass to `predict`) ·
-`rl_agent_config.json` (fitted temperatures, training config) · `train_report.json`
-(before/after metrics).
+- English only; nine fixed labels; out-of-taxonomy requests resolve to `conversation`.
+- Much of the training data is synthetic, so template phrasing biases it; implicit or indirect
+  requests ("I keep meaning to…") are the weak spot.
+- Calibrated on a small held-out slice; probabilities are not reliable above ~0.9.

model.safetensors

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:04eb1c795796ddc492cfc661f229e1438e6bca9651081a839b387aa8437d4321
+oid sha256:fca19e77f4f6462c84d0adc568e2982d6e3a290e1f778334320c891aad8a5c09
 size 842609220

questions.json

@@ -3,17 +3,15 @@
     "type": "choice",
     "instructions": "What does this request (or the request it continues) demand?",
     "criteria": {
-      "react": "Urgent or deteriorating situation where delay has real consequences (safety, health, money, a missed critical window).",
-      "acquire": "The outcome is possession of a purchasable good \u2014 buy, order, restock, replace.",
-      "arrange": "The outcome is a commitment in the world with other parties or the calendar \u2014 book, reserve, schedule, hire, plan-to-commit.",
-      "execute": "The request names a known, externally-defined procedure to be completed to submission or acceptance \u2014 renew, file, claim, apply, submit.",
-      "make": "The outcome is an artifact the assistant produces, judged against a standard (tests, a spec, taste).",
-      "diagnose": "The outcome is the cause of a specific malfunction or anomaly.",
-      "research": "The outcome is understanding assembled from investigation \u2014 multi-source, current, or synthesis-shaped.",
-      "learn": "The outcome is the user's own ability, built over sessions \u2014 practice, tutoring, review.",
-      "tend": "The request concerns the ongoing state of a standing concern \u2014 review, maintain, adjust; recurring or wake-arrived.",
-      "companion": "Inner-life support with no outcome in the world \u2014 explicitly reflective or support-seeking.",
-      "conversation": "Nothing above fits \u2014 questions, quick asks, lookups, chatter, micro-tasks."
+      "stabilize": "Urgent or deteriorating situation where delay has real consequences (safety, health, money, a missed critical window) and a cheap, reversible action plausibly stops the worsening.",
+      "transact": "The outcome is a commitment or possession in the world: a booking, reservation, purchase, payment, transfer, or a named procedure completed to acceptance.",
+      "plan": "The outcome is the person's own standing structure, set up or changed in their personal records: a goal, a routine, a budget or spending limit, a tracker. Reversible, nobody else involved.",
+      "produce": "The outcome is an artifact the assistant produces, judged against a standard (tests, a spec, taste): a document, spreadsheet, script, letter.",
+      "diagnose": "The outcome is the cause of a specific malfunction or anomaly, including causes in the person's own data.",
+      "research": "The outcome is understanding assembled from investigation: multi-source, current, comparison or synthesis-shaped.",
+      "learn": "The outcome is the person's own ability, built over sessions: practice, tutoring, review.",
+      "companion": "Inner-life support with no outcome in the world: explicitly reflective or support-seeking.",
+      "conversation": "None of the above: questions, quick asks, lookups, reports of a measurement, a single reminder or note, chatter."
     }
   }
 }

rl_agent_config.json

@@ -11,7 +11,7 @@
   "amp_dtype": "bf16",
   "model_name": "rl-agent",
   "temperature": [
-    3.8229823112487793,
+    3.9046413898468018,
     1.0,
     1.0
   ],
@@ -23,8 +23,8 @@
     "fine_tuned_from_checkpoint": true,
     "laya_train": {
       "epochs": 4,
-      "micro_batch": 4,
-      "grad_accum": 16,
+      "micro_batch": 2,
+      "grad_accum": 32,
       "encoder_lr": 2.5e-05,
       "head_lr": 0.0001,
       "min_lr": 1e-06,
@@ -55,17 +55,17 @@
       "label_column": "label",
       "question_id": "label",
       "instructions": null,
-      "eval_data": "corpus/eval_drills.jsonl",
+      "eval_data": "corpus_v2/eval.jsonl",
       "target_error": 0.1,
       "min_abstain_n": 10
     },
     "laya_train_calibration": {
       "choice": {
         "items": 400,
-        "temperature": 3.8229823112487793,
+        "temperature": 3.9046413898468018,
         "issues": [],
-        "accuracy": 0.9475,
-        "mean_confidence": 0.9591468240120362
+        "accuracy": 0.9575,
+        "mean_confidence": 0.9617928987797445
       },
       "score": {
         "items": 0,
@@ -79,66 +79,66 @@
       }
     },
     "train_report": {
-      "eval_source": "corpus/eval_drills.jsonl",
+      "eval_source": "corpus_v2/eval.jsonl",
       "eval_mode": "held_out",
       "is_held_out": true,
-      "eval_items": 24,
+      "eval_items": 54,
       "note": "Metrics reflect independent held-out evaluation.",
       "before": {
-        "items": 24,
-        "loss": 7.6903,
-        "accuracy": 0.2917,
-        "mean_confidence": 0.8302,
-        "ece": 0.5385,
-        "brier": 1.1763,
-        "brier_top1": 0.4885,
+        "items": 54,
+        "loss": 1.8596,
+        "accuracy": 0.2407,
+        "mean_confidence": 0.3148,
+        "ece": 0.1869,
+        "brier": 0.7831,
+        "brier_top1": 0.1308,
         "by_type": {
           "choice": {
-            "items": 24,
-            "loss": 7.6903,
-            "accuracy": 0.2917,
-            "mean_confidence": 0.8302,
-            "ece": 0.5385,
-            "brier": 1.1763,
-            "brier_top1": 0.4885
+            "items": 54,
+            "loss": 1.8596,
+            "accuracy": 0.2407,
+            "mean_confidence": 0.3148,
+            "ece": 0.1869,
+            "brier": 0.7831,
+            "brier_top1": 0.1308
           }
         }
       },
       "after": {
-        "items": 24,
-        "loss": 0.7962,
-        "accuracy": 0.8333,
-        "mean_confidence": 0.9643,
-        "ece": 0.131,
-        "brier": 0.315,
-        "brier_top1": 0.1524,
+        "items": 54,
+        "loss": 0.6661,
+        "accuracy": 0.8704,
+        "mean_confidence": 0.9589,
+        "ece": 0.0886,
+        "brier": 0.2397,
+        "brier_top1": 0.1096,
         "by_type": {
           "choice": {
-            "items": 24,
-            "loss": 0.7962,
-            "accuracy": 0.8333,
-            "mean_confidence": 0.9643,
-            "ece": 0.131,
-            "brier": 0.315,
-            "brier_top1": 0.1524
+            "items": 54,
+            "loss": 0.6661,
+            "accuracy": 0.8704,
+            "mean_confidence": 0.9589,
+            "ece": 0.0886,
+            "brier": 0.2397,
+            "brier_top1": 0.1096
           }
         }
       },
       "comparison": {
-        "delta_accuracy": 0.5416,
-        "delta_loss": -6.8941,
-        "delta_ece": -0.4075,
-        "delta_brier": -0.8613,
-        "delta_brier_top1": -0.3361,
-        "delta_mean_confidence": 0.1341
+        "delta_accuracy": 0.6297,
+        "delta_loss": -1.1935,
+        "delta_ece": -0.0983,
+        "delta_brier": -0.5434,
+        "delta_brier_top1": -0.0212,
+        "delta_mean_confidence": 0.6441
       },
       "calibration": {
         "choice": {
           "items": 400,
-          "temperature": 3.8229823112487793,
+          "temperature": 3.9046413898468018,
           "issues": [],
-          "accuracy": 0.9475,
-          "mean_confidence": 0.9591468240120362
+          "accuracy": 0.9575,
+          "mean_confidence": 0.9617928987797445
         },
         "score": {
           "items": 0,
@@ -152,12 +152,12 @@
         }
       },
       "abstention_thresholds": {
-        "choice:11+": 0.3049
+        "choice:6-10": 0.5204
       },
       "training": {
         "epochs": 4,
-        "micro_batch": 4,
-        "grad_accum": 16,
+        "micro_batch": 2,
+        "grad_accum": 32,
         "encoder_lr": 2.5e-05,
         "head_lr": 0.0001,
         "min_lr": 1e-06,
@@ -188,13 +188,13 @@
         "label_column": "label",
         "question_id": "label",
         "instructions": null,
-        "eval_data": "corpus/eval_drills.jsonl",
+        "eval_data": "corpus_v2/eval.jsonl",
         "target_error": 0.1,
         "min_abstain_n": 10
       }
     },
     "abstention_thresholds": {
-      "choice:11+": 0.3049
+      "choice:6-10": 0.5204
     }
   },
   "fine_tuned": true

train_report.json

@@ -1,64 +1,64 @@
 {
-  "eval_source": "corpus/eval_drills.jsonl",
+  "eval_source": "corpus_v2/eval.jsonl",
   "eval_mode": "held_out",
   "is_held_out": true,
-  "eval_items": 24,
+  "eval_items": 54,
   "note": "Metrics reflect independent held-out evaluation.",
   "before": {
-    "items": 24,
-    "loss": 7.6903,
-    "accuracy": 0.2917,
-    "mean_confidence": 0.8302,
-    "ece": 0.5385,
-    "brier": 1.1763,
-    "brier_top1": 0.4885,
+    "items": 54,
+    "loss": 1.8596,
+    "accuracy": 0.2407,
+    "mean_confidence": 0.3148,
+    "ece": 0.1869,
+    "brier": 0.7831,
+    "brier_top1": 0.1308,
     "by_type": {
       "choice": {
-        "items": 24,
-        "loss": 7.6903,
-        "accuracy": 0.2917,
-        "mean_confidence": 0.8302,
-        "ece": 0.5385,
-        "brier": 1.1763,
-        "brier_top1": 0.4885
+        "items": 54,
+        "loss": 1.8596,
+        "accuracy": 0.2407,
+        "mean_confidence": 0.3148,
+        "ece": 0.1869,
+        "brier": 0.7831,
+        "brier_top1": 0.1308
       }
     }
   },
   "after": {
-    "items": 24,
-    "loss": 0.7962,
-    "accuracy": 0.8333,
-    "mean_confidence": 0.9643,
-    "ece": 0.131,
-    "brier": 0.315,
-    "brier_top1": 0.1524,
+    "items": 54,
+    "loss": 0.6661,
+    "accuracy": 0.8704,
+    "mean_confidence": 0.9589,
+    "ece": 0.0886,
+    "brier": 0.2397,
+    "brier_top1": 0.1096,
     "by_type": {
       "choice": {
-        "items": 24,
-        "loss": 0.7962,
-        "accuracy": 0.8333,
-        "mean_confidence": 0.9643,
-        "ece": 0.131,
-        "brier": 0.315,
-        "brier_top1": 0.1524
+        "items": 54,
+        "loss": 0.6661,
+        "accuracy": 0.8704,
+        "mean_confidence": 0.9589,
+        "ece": 0.0886,
+        "brier": 0.2397,
+        "brier_top1": 0.1096
       }
     }
   },
   "comparison": {
-    "delta_accuracy": 0.5416,
-    "delta_loss": -6.8941,
-    "delta_ece": -0.4075,
-    "delta_brier": -0.8613,
-    "delta_brier_top1": -0.3361,
-    "delta_mean_confidence": 0.1341
+    "delta_accuracy": 0.6297,
+    "delta_loss": -1.1935,
+    "delta_ece": -0.0983,
+    "delta_brier": -0.5434,
+    "delta_brier_top1": -0.0212,
+    "delta_mean_confidence": 0.6441
   },
   "calibration": {
     "choice": {
       "items": 400,
-      "temperature": 3.8229823112487793,
+      "temperature": 3.9046413898468018,
       "issues": [],
-      "accuracy": 0.9475,
-      "mean_confidence": 0.9591468240120362
+      "accuracy": 0.9575,
+      "mean_confidence": 0.9617928987797445
     },
     "score": {
       "items": 0,
@@ -72,12 +72,12 @@
     }
   },
   "abstention_thresholds": {
-    "choice:11+": 0.3049
+    "choice:6-10": 0.5204
   },
   "training": {
     "epochs": 4,
-    "micro_batch": 4,
-    "grad_accum": 16,
+    "micro_batch": 2,
+    "grad_accum": 32,
     "encoder_lr": 2.5e-05,
     "head_lr": 0.0001,
     "min_lr": 1e-06,
@@ -108,7 +108,7 @@
     "label_column": "label",
     "question_id": "label",
     "instructions": null,
-    "eval_data": "corpus/eval_drills.jsonl",
+    "eval_data": "corpus_v2/eval.jsonl",
     "target_error": 0.1,
     "min_abstain_n": 10
   }