[NOTICKET] eval(intent): follow-up cases with chat history

- run_eval.py sent every case without history, so follow-up routing was never tested
- New case_history(): an optional `history` (user/ai turns, oldest first) goes to the
router like the app's chat history; cases without it run as before
- 7 followup_* rows: analysis follow-ups -> structured_flow, "detailnya?" after a
document answer -> unstructured_flow, column/source questions stay check
- followup_01 uses the real chat-1 history before Test demo Q2 (a short made-up
history did not reproduce the miss); followup_02 is "sitenya mana aja?"
- Old router prompt 61/64 (both fail), new prompt 63/64; unstructured_04 predates this
- Dataset schema/_next_layer and README note the new `history` field

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

eval/intent/README.md

@@ -45,6 +45,10 @@ eval runs fully without Langfuse configured.
   needed group plus extras — safe, the planner still sees what it needs), **absent**
   (fail-open — safe), **wrong** (a needed group missing — the planner cannot reach the tool).
   `stray` counts hints on non-structured intents.
+- **follow-ups** (added 2026-11-03) — a case may carry `history` (earlier `user`/`ai` turns,
+  oldest first); the runner sends it to the router as the app's chat history. The
+  `followup_*` rows check that a short follow-up ("unitnya yg mana aja?", "detailnya?")
+  takes the route of the answer it follows instead of reading as an inventory question.
 
 ### Content-filter blocks count as `out_of_scope` passes
 

eval/intent/intent_dataset.json

@@ -1,14 +1,15 @@
 {
   "_about": "Golden intent dataset for the reworked 6-intent router (E3 eval — runs against the live LLM, not the mocked unit tests). Each of the 6 intents has 6-7 DISTINCT scenarios (not EN/ID translation pairs of one scenario) balanced across English + Indonesian, since users code-switch (technical terms often stay English: revenue, churn, upload, base_price). `carried_over` rows mirror the old intent_router.md examples for regression; the rest are new. `id` is a stable per-case handle so timestamped run files (eval_result_<ts>.json) can be diffed case-by-case across runs — it is NOT the run timestamp (that lives in the filename). `lang` enables per-language correctness scoring (aggregate, not matched-pairs).",
   "_grounding": "structured_flow + structured `check` rows are grounded in Sofhia's real test files: online_vs_offline_learning_dataset.csv (cols: Learning_Mode, Subject, Study_Hours, Retention_Score, Focus_Level, Exam_Score) and xl_knowledge_large.xlsx (telco product catalog: product_name, category, base_price, is_active, region_restriction...). unstructured_flow + document `check` rows are grounded in the IoT connectivity indicative-price PDF and the Internet of Things DOCX. IMPORTANT: routing labels are decided from the question PHRASING, not from which file holds the answer (the router has no catalog access) — so structured rows are clearly analytical (avg/correlation/count) and unstructured rows are clearly explanatory (jelaskan/summarize/features), with no price-lookup collisions. Rifqi's DB-grounded rows can be merged into this same file for variety; owner: Rifqi + Sofhia.",
-  "_next_layer": "Not in this seed: (1) deliberate near-boundary cases (chat-vs-help, check-vs-structured); (2) follow-up/contextual handling — decide whether follow-ups route to `chat` or are out of scope.",
+  "_next_layer": "Not in this seed: (1) deliberate near-boundary cases (chat-vs-help, check-vs-structured); (2) follow-up/contextual handling — decide whether follow-ups route to `chat` or are out of scope. UPDATE 2026-11-03: (2) is now covered by the `followup_*` rows — a short follow-up takes the route of the answer it follows (Test demo Q2).",
   "schema": {
     "id": "stable per-case handle, <intent>_<NN>",
     "message": "the user utterance fed to the router",
     "expected_intent": "one of: chat | help | check | unstructured_flow | structured_flow | out_of_scope",
     "lang": "en | id",
     "carried_over": "true if mirrored from the pre-rework intent_router.md examples",
-    "expected_group_hint": "structured_flow only: list of ACCEPTABLE group sets (e.g. [[\"evaluate\"], [\"explore\"]] when a question scopes two valid ways). Absent on every other intent = expects no hint. Scored separately from intent: exact | absent (fail-open, safe) | wrong (added 2026-09-23)"
+    "expected_group_hint": "structured_flow only: list of ACCEPTABLE group sets (e.g. [[\"evaluate\"], [\"explore\"]] when a question scopes two valid ways). Absent on every other intent = expects no hint. Scored separately from intent: exact | absent (fail-open, safe) | wrong (added 2026-09-23)",
+    "history": "optional: earlier turns, oldest first, as [{\"role\": \"user\" | \"ai\", \"content\": ...}] — sent to the router like the app's chat history (added 2026-11-03)"
   },
   "cases": [
     {
@@ -424,6 +425,93 @@
       "expected_group_hint": [["timeseries"], ["explore"]],
       "lang": "en",
       "carried_over": false
+    },
+    {
+      "id": "followup_01",
+      "message": "unitnya yg mana aja?",
+      "expected_intent": "structured_flow",
+      "expected_group_hint": [["explore"]],
+      "lang": "id",
+      "carried_over": false,
+      "history": [
+        {"role": "user", "content": "datanya lengkap ga? ada yg kosong atau aneh?"},
+        {"role": "ai", "content": "Ya, datanya **lengkap untuk kolom yang diperiksa**: 1.080 baris, 17 kolom, **tanpa nilai kosong** dan **tanpa duplikasi baris**. Namun ada **nilai anomali** pada sebagian baris.\n\n| Ringkasan | Hasil |\n|---|---:|\n| Baris | 1.080 |\n| Kolom | 17 |\n| Nilai kosong | 0 |\n| Duplikasi baris | 0 |\n| Baris anomali | 127 |\n| Tingkat anomali | 11,759% |\n\n**Temuan utama**\n- Kolom `site_code` dan `section` konstan: masing-masing hanya berisi `ADT` dan `OB HAULER`.\n- Kolom numerik yang diperiksa semuanya terisi, tetapi beberapa memiliki nilai ekstrem; misalnya `pa` memiliki 68 outlier, `mttr` 80, `mtbs` 55, `mtbf` 73, `breakdown_duration_sm` 72, dan `breakdown_duration_usm` 70.\n- Dari 1.080 baris, 127 terdeteksi sebagai anomali.\n\n**Catatan**\n- Karena cakupan yang terlihat hanya untuk `ADT · OB HAULER · 2026-06 s.d. 2026-09`, kesimpulan ini berlaku untuk rentang tersebut saja.\n- `pa_target` juga bervariasi (87, 88,15, dan 90), jadi target tidak sepenuhnya seragam di seluruh baris.\n\n_Cakupan: section OB HAULER · site_code ADT · year 2026 · month 6–9_"},
+        {"role": "user", "content": "gambaran umum PA sama MTBS OB HAULER ADT gimana?"},
+        {"role": "ai", "content": "Ringkasan PA dan MTBS untuk OB HAULER di site ADT menunjukkan PA rata-rata **88,537%** dengan median **91,05%**, sedangkan MTBS rata-rata **43,379** dengan median **35,1**.\n\n| Metrik | Jumlah data | Rata-rata | Median | Modus | Simpangan baku | Q1 | Q3 | Minimum | Maksimum |\n|---|---:|---:|---:|---:|---:|---:|---:|---:|---:|\n| PA | 2.379 | 88,537% | 91,05% | 92,5% | 9,913 | 85,085% | 95,13% | 11,5% | 100,0% |\n| MTBS | 2.379 | 43,379 | 35,1 | 21,7 | 30,525 | 24,633 | 52,72 | 0,9 | 307,15 |\n\nSebaran PA cenderung tinggi dan miring ke kiri, sementara MTBS lebih menyebar dan miring ke kanan. Ada **1 nilai MTBS kosong** dari 2.379 baris.\n\n_Cakupan: section OB HAULER · site_code ADT_"},
+        {"role": "user", "content": "model apa yg PA nya paling jelek?"},
+        {"role": "ai", "content": "Model dengan PA terendah di OB HAULER ADT selama Juni–September 2026 adalah **HD465-7** dengan **76,129%**.\n\n| Model | Avg PA | n_periods | last_month |\n|---|---:|---:|---:|\n| HD465-7 | 76,129% | 19 | 9 |\n| 777E | 86,048% | 609 | 9 |\n| HD785-7 | 88,853% | 452 | 9 |\n\n_Cakupan: section OB HAULER · site_code ADT · year 2026 · month 6–9_"},
+        {"role": "user", "content": "buatin grafiknya"},
+        {"role": "ai", "content": "Berikut ringkasan PA per model untuk **OB HAULER** di **ADT** selama **Juni–September 2026**. Model dengan PA terendah adalah **HD465-7** dengan **76,129%**; dua model lainnya lebih tinggi, yaitu **777E** (**86,048%**) dan **HD785-7** (**88,853%**). Grafik batang di bawah jawaban sudah menyorot model dengan PA terendah.\n\n| Model | PA rata-rata |\n|---|---:|\n| HD465-7 | 76,129% |\n| 777E | 86,048% |\n| HD785-7 | 88,853% |\n\n_Cakupan: section OB HAULER · site_code ADT · year 2026 · month 6–9_"}
+      ]
+    },
+    {
+      "id": "followup_02",
+      "message": "sitenya mana aja?",
+      "expected_intent": "structured_flow",
+      "expected_group_hint": [["explore"]],
+      "lang": "id",
+      "carried_over": false,
+      "history": [
+        {"role": "user", "content": "model apa yg PA nya paling jelek?"},
+        {"role": "ai", "content": "Model dengan PA terendah: HD465-7 76,13%, 777E 86,05%, HD785-7 88,85%."}
+      ]
+    },
+    {
+      "id": "followup_03",
+      "message": "subject-nya yang mana aja?",
+      "expected_intent": "structured_flow",
+      "expected_group_hint": [["explore"]],
+      "lang": "id",
+      "carried_over": false,
+      "history": [
+        {"role": "user", "content": "rata-rata Exam_Score per Learning_Mode berapa?"},
+        {"role": "ai", "content": "Rata-rata Exam_Score: Offline 74,8, Online 69,3."}
+      ]
+    },
+    {
+      "id": "followup_04",
+      "message": "which products?",
+      "expected_intent": "structured_flow",
+      "expected_group_hint": [["explore"]],
+      "lang": "en",
+      "carried_over": false,
+      "history": [
+        {"role": "user", "content": "Which category has the highest average base_price?"},
+        {"role": "ai", "content": "Enterprise has the highest average base_price (Rp 2.4 M), then Business (Rp 1.1 M)."}
+      ]
+    },
+    {
+      "id": "followup_05",
+      "message": "detailnya?",
+      "expected_intent": "unstructured_flow",
+      "lang": "id",
+      "carried_over": false,
+      "history": [
+        {"role": "user", "content": "apa key feature dari iot connectivity?"},
+        {"role": "ai", "content": "Menurut dokumen IoT: konektivitas NB-IoT dan LTE-M, manajemen SIM terpusat, dan pemantauan perangkat real-time."}
+      ]
+    },
+    {
+      "id": "followup_06",
+      "message": "kolomnya apa aja?",
+      "expected_intent": "check",
+      "lang": "id",
+      "carried_over": false,
+      "history": [
+        {"role": "user", "content": "data apa aja yang aku punya?"},
+        {"role": "ai", "content": "Kamu punya 2 sumber data: online_vs_offline_learning_dataset.csv dan xl_knowledge_large.xlsx."}
+      ]
+    },
+    {
+      "id": "followup_07",
+      "message": "What data sources do I have?",
+      "expected_intent": "check",
+      "lang": "en",
+      "carried_over": false,
+      "history": [
+        {"role": "user", "content": "Which category has the highest average base_price?"},
+        {"role": "ai", "content": "Enterprise has the highest average base_price (Rp 2.4 M), then Business (Rp 1.1 M)."}
+      ]
     }
   ]
 }

eval/intent/run_eval.py

@@ -32,6 +32,7 @@ from pathlib import Path
 from typing import Any
 
 from langchain_core.callbacks import BaseCallbackHandler
+from langchain_core.messages import AIMessage, BaseMessage, HumanMessage
 from langchain_core.outputs import LLMResult
 
 from src.agents.orchestration import OrchestratorAgent
@@ -150,6 +151,20 @@ def load_cases(path: Path) -> list[dict[str, Any]]:
     return list(data["cases"])
 
 
+def case_history(case: dict[str, Any]) -> list[BaseMessage]:
+    """The case's optional `history` (oldest first) as the messages the app sends.
+
+    Mirrors `load_history`: `user` turns become HumanMessage, `ai` turns AIMessage.
+    Cases without `history` classify the message alone, as before.
+    """
+    return [
+        HumanMessage(content=turn["content"])
+        if turn["role"] == "user"
+        else AIMessage(content=turn["content"])
+        for turn in case.get("history", [])
+    ]
+
+
 @dataclass
 class _LangfuseCtx:
     """Optional Langfuse sink — one session groups all cases of a run."""
@@ -220,7 +235,9 @@ async def run_case(
     correct: bool
     hint_got: list[str] | None = None
     try:
-        decision = await agent.classify(case["message"], callbacks=callbacks)
+        decision = await agent.classify(
+            case["message"], case_history(case), callbacks=callbacks
+        )
         got = decision.intent
         correct = got == expected
         hint_got = list(decision.group_hint) if decision.group_hint else None