eval(chat_sim): doc + mixed suites with bound documents

UNSTRUCTURED_V2_PLAN U8. Suite.document_ids binds v2-ingested documents in the
shape Go's buildBoundSource materialises; unowned or un-indexed documents are
refused.

doc suite (BUMA standard): 4/4 after rephrasing turn 3 — a figure answer
carried asset://4f72f8b170d1787b (Gambar 2.1), resolved via image-token to
200 image/jpeg. mixed suite is defined but not run: dedorch data_catalog has
0 rows, so padb cannot be bound.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

eval/chat_sim/README.md

@@ -53,3 +53,28 @@ uv run python -m eval.chat_sim.run_chat --plain          # no ANSI colors
   output is shown (annotated `masked→cloud` where the cloud trace would redact it).
 - The first scripted question may route to `chat` rather than `help` — that's the live
   router's real call, shown transparently.
+
+## Document suites — `doc` and `mixed` (added 2026-09-23)
+
+`Suite.document_ids` binds v2-ingested documents (`documents.id`, ingested through
+`/api/v2/document/*`) next to the suite's data source. `setup_analysis` writes them into
+the analysis-scope row as `{source_id: documents.id, source_type: "unstructured"}` — the
+shape Go's `buildBoundSource` materialises — and **refuses** a document that is not owned
+by `--user-id` or whose v2 ingestion did not complete. A suite may have documents and no
+data source (`doc`).
+
+```bash
+./.venv/Scripts/python.exe -m eval.chat_sim.run_chat --suite doc --no-report
+./.venv/Scripts/python.exe -m eval.chat_sim.run_chat --suite mixed --no-report
+```
+
+- **`doc`** — the BUMA standard alone: check → definition (ID) → a figure answer that must
+  carry an `asset://` marker → a formula (EN). Document turns write no `report_inputs`,
+  so `--no-report` (the report would 409 by design).
+- **`mixed`** — `padb` + the BUMA standard: check lists both → two turns needing the
+  document's definition *and* computed numbers → one numbers-only turn that must NOT
+  retrieve the document.
+- The document id is `BUMA_DOCUMENT_ID` in `run_chat.py`; re-ingesting mints a new one.
+- **Phrase figure turns by topic, not by number, and not as "show me the chart"** — see
+  the notes on the `doc` suite's turn 3 (UNSTRUCTURED_V2_PLAN §4.4).
+

eval/chat_sim/results/transcript_doc_2026-09-23_114510.json

@@ -0,0 +1,69 @@
+{
+  "run": {
+    "timestamp": "2026-09-23T11:45:10",
+    "suite": "doc",
+    "source_id": null,
+    "source_name": "Standar Parameter Produksi (BUMA).pdf",
+    "notes": "Document only. Turn 1 = check lists the document; 2 + 4 = document Q&A (ID/EN); 3 = a figure answer, which must carry an asset:// marker (U6).",
+    "user_id": "78e8df76-14a1-43ec-95bc-498b4a029701",
+    "analysis_id": "04ddbd55-562e-46b5-99f4-840d4f23d1ee",
+    "objective": "Understand how the company standard defines and calculates its production parameters.",
+    "turns": 4,
+    "tokens": {
+      "input": 41148,
+      "output": 731,
+      "total": 41879
+    }
+  },
+  "turns": [
+    {
+      "n": 1,
+      "question": "Dokumen apa saja yang ada di analisis ini?",
+      "intent": "check",
+      "confidence": 0.98,
+      "rewritten": "What documents are available in this analysis?",
+      "answer": "Kamu punya 1 dokumen:\n\n- Standar Parameter Produksi (BUMA).pdf",
+      "tools": [],
+      "tokens_in": 4430,
+      "tokens_out": 59,
+      "ms": 15924
+    },
+    {
+      "n": 2,
+      "question": "Apa definisi Physical Availability menurut standar ini, dan bagaimana rumusnya?",
+      "intent": "unstructured_flow",
+      "confidence": 0.97,
+      "rewritten": "What is the definition of Physical Availability according to this standard, and what is the formula?",
+      "answer": "Menurut standar ini, **Physical Availability (PA)** adalah **ketersediaan fisik suatu equipment/unit**, yaitu proporsi waktu equipment/unit tersebut berada dalam kondisi **available (siap pakai)** selama periode tertentu.\n\nRumus umumnya:\n\n**PA = (Total Hours - Breakdown) / Total Hours × 100%**\n\nUntuk grouping/composite, rumus dasarnya:\n\n**PA_ij = (Total Hours_ij - Breakdown_ij) / Total Hours_ij × 100%**\n\nStandar juga menyebutkan bahwa nilai PA hasil perhitungan tersebut ekuivalen dengan **rata-rata tertimbang** PA masing-masing type/class unit dengan **Quantity (Qty)** sebagai faktor pembobot.",
+      "tools": [],
+      "tokens_in": 7053,
+      "tokens_out": 217,
+      "ms": 30580
+    },
+    {
+      "n": 3,
+      "question": "Show me the waterfall chart that breaks down gain/loss per production parameter.",
+      "intent": "structured_flow",
+      "confidence": 0.96,
+      "rewritten": "Show me the waterfall chart that breaks down gain/loss per production parameter.",
+      "answer": "I can't answer that from the data sources connected to this analysis. The catalog does not expose any production-parameter measure or gain/loss column that can be used to build a waterfall breakdown. The available approved measures are Qty, PA, UA, produksi, and PTY, but none of them represent gain/loss per parameter, and no charting workflow exists for a waterfall chart specifically. The nearest supported analyses are a grouped breakdown of one of the approved measures by an existing category, or a trend/summary of PA, UA, Qty, produksi, or PTY if you want a chartable metric. You can bind a source that holds this data, or ask me what's available (try /help or \"what data do I have?\").",
+      "tools": [],
+      "tokens_in": 22556,
+      "tokens_out": 253,
+      "ms": 40392
+    },
+    {
+      "n": 4,
+      "question": "How is Utilization of Availability (UA) calculated in this standard?",
+      "intent": "unstructured_flow",
+      "confidence": 0.98,
+      "rewritten": "How is Utilization of Availability (UA) calculated in this standard?",
+      "answer": "Dalam standar ini, **Utilization of Availability (UA)** adalah tingkat efektivitas penggunaan alat yang tersedia secara fisik, yaitu seberapa lama unit digunakan secara efektif untuk bekerja (**working hours**) selama berada dalam kondisi siap dioperasikan (**available**).\n\nRumusnya:\n\n**UA = Working Hours / (Total Hours - Breakdown)**\n\nUntuk **Grouping (Composite)** type/class unit, UA dihitung dengan pembobotan berdasarkan **Quantity (Qty)**:\n\n**UA_ij = (Σ Qty_ij × UA_ij) / (Σ Qty_ij)**\n\nCatatan: UA disesuaikan berdasarkan kategori aktivitas operasional (**activity**).",
+      "tools": [],
+      "tokens_in": 7109,
+      "tokens_out": 202,
+      "ms": 18336
+    }
+  ],
+  "report_markdown": null
+}

eval/chat_sim/results/transcript_doc_2026-09-23_114643.json

@@ -0,0 +1,33 @@
+{
+  "run": {
+    "timestamp": "2026-09-23T11:46:43",
+    "suite": "doc",
+    "source_id": null,
+    "source_name": "Standar Parameter Produksi (BUMA).pdf",
+    "notes": "Document only. Turn 1 = check lists the document; 2 + 4 = document Q&A (ID/EN); 3 = a figure answer, which must carry an asset:// marker (U6).",
+    "user_id": "78e8df76-14a1-43ec-95bc-498b4a029701",
+    "analysis_id": "26d45658-8624-4dda-a70d-fb27f7963074",
+    "objective": "Understand how the company standard defines and calculates its production parameters.",
+    "turns": 1,
+    "tokens": {
+      "input": 6046,
+      "output": 150,
+      "total": 6196
+    }
+  },
+  "turns": [
+    {
+      "n": 1,
+      "question": "What does Figure 2.1 in the standard show? Include the figure.",
+      "intent": "unstructured_flow",
+      "confidence": 0.93,
+      "rewritten": "What does Figure 2.1 in the standard show? Include the figure.",
+      "answer": "I can’t see Figure 2.1 in the provided pages. The available context only shows notes on pages 6–7 and a table titled **“Perhitungan Gain/Loss per Model Unit”** on page 6, but not the figure itself.\n\nIf you want, I can still help summarize the surrounding section or look for another page/chunk that contains the figure marker.",
+      "tools": [],
+      "tokens_in": 6046,
+      "tokens_out": 150,
+      "ms": 30777
+    }
+  ],
+  "report_markdown": null
+}

eval/chat_sim/results/transcript_doc_2026-09-23_114849.json

@@ -0,0 +1,33 @@
+{
+  "run": {
+    "timestamp": "2026-09-23T11:48:49",
+    "suite": "doc",
+    "source_id": null,
+    "source_name": "Standar Parameter Produksi (BUMA).pdf",
+    "notes": "Document only. Turn 1 = check lists the document; 2 + 4 = document Q&A (ID/EN); 3 = a figure answer, which must carry an asset:// marker (U6).",
+    "user_id": "78e8df76-14a1-43ec-95bc-498b4a029701",
+    "analysis_id": "27a9f75d-0344-48c5-8163-2907c421bb5c",
+    "objective": "Understand how the company standard defines and calculates its production parameters.",
+    "turns": 1,
+    "tokens": {
+      "input": 6411,
+      "output": 229,
+      "total": 6640
+    }
+  },
+  "turns": [
+    {
+      "n": 1,
+      "question": "Apa isi Gambar 2.1 pada bagian Waterfall Analysis di standar? Sertakan gambarnya.",
+      "intent": "unstructured_flow",
+      "confidence": 0.91,
+      "rewritten": "What is the content of Figure 2.1 in the Waterfall Analysis section of the standard? Include the image.",
+      "answer": "Gambar 2.1 pada bagian *Waterfall Analysis* adalah **“Analisis Gain/Loss Berdasarkan Setiap Parameter Produksi”**. Keterangan di bawahnya menjelaskan bahwa analisis ini menampilkan kontribusi Gain/Loss dari setiap parameter produksi berdasarkan hasil perhitungan Gain/Loss Production.\n\n![](asset://4f72f8b170d1787b)\n\nKeterangan tambahan yang ada di halaman itu:\n- **Other Activity** adalah kondisi ketika alokasi equipment/unit pada pelaksanaan aktual tidak sesuai dengan aktivitas yang telah direncanakan.\n- Contohnya, equipment/unit yang direncanakan untuk *waste removal* dipakai untuk *coal mining* pada pelaksanaan aktual, atau sebaliknya.",
+      "tools": [],
+      "tokens_in": 6411,
+      "tokens_out": 229,
+      "ms": 29287
+    }
+  ],
+  "report_markdown": null
+}

eval/chat_sim/run_chat.py

@@ -106,6 +106,11 @@ class Suite(NamedTuple):
     objective: str
     turns: list[str]
     notes: str
+    # v2-ingested documents (`documents.id`) bound alongside `source_id` — added
+    # 2026-09-23 for UNSTRUCTURED_V2_PLAN U8. They are written into the analysis-scope
+    # catalog in exactly the shape Go's `buildBoundSource` materialises for a bound
+    # pdf/docx/txt: {source_id: documents.id, source_type: "unstructured", ...}.
+    document_ids: tuple[str, ...] = ()
 
 
 SUITES: dict[str, Suite] = {}
@@ -247,6 +252,57 @@ SUITES["cycle"] = Suite(
 )
 
 
+# Documents v2 suites (UNSTRUCTURED_V2_PLAN U8, 2026-09-23). The document is the BUMA
+# "Standar Parameter Produksi" PDF (9 pages, 28 chunks, 2 figures), ingested through
+# the real v2 routes for DEFAULT_USER_ID. It DEFINES PA/UA/Pty and their formulas and
+# carries two figures (Gambar 2.1 waterfall p.8, Gambar 2.2 p.9) — it states NO
+# numeric PA target, so no turn asks for one. Re-ingest -> update the id below.
+BUMA_DOCUMENT_ID = "0b4160db-327c-461a-b30d-348f94718f1b"
+
+# Unstructured only: no data source, one bound document.
+SUITES["doc"] = Suite(
+    source_id=None,
+    name="Standar Parameter Produksi (BUMA).pdf",
+    objective="Understand how the company standard defines and calculates its production "
+              "parameters.",
+    notes="Document only. Turn 1 = check lists the document; 2 + 4 = document Q&A (ID/EN); "
+          "3 = a figure answer, which must carry an asset:// marker (U6).",
+    turns=[
+        "Dokumen apa saja yang ada di analisis ini?",
+        "Apa definisi Physical Availability menurut standar ini, dan bagaimana rumusnya?",
+        # Phrased as a question ABOUT the document's figure, NAMING ITS TOPIC. Two
+        # failures on 2026-09-23 set this wording: "Show me the waterfall chart ..."
+        # routed to structured_flow (the router reads "chart" as a plotting request)
+        # and was refused; "What does Figure 2.1 show?" routed correctly but dense
+        # retrieval cannot match a bare figure NUMBER (top-5 = pp.6-7, figure on p.8).
+        # Naming the topic ranks the figure chunk 1st-2nd in EN and ID.
+        "Apa isi Gambar 2.1 pada bagian Waterfall Analysis di standar? Sertakan gambarnya.",
+        "How is Utilization of Availability (UA) calculated in this standard?",
+    ],
+    document_ids=(BUMA_DOCUMENT_ID,),
+)
+
+# Structured + unstructured: padb (Postgres) with the standard bound beside it.
+SUITES["mixed"] = Suite(
+    source_id=SUITES["padb"].source_id,
+    name="pa example (postgres) + Standar Parameter Produksi (BUMA).pdf",
+    objective="Understand fleet availability across 2025, read against how the company "
+              "standard defines it.",
+    notes="Turn 1 = check lists data AND document; 2 = mixed (definition + monthly trend); "
+          "3 = mixed EN (UA vs PA definition + site comparison); 4 = numbers only, must "
+          "NOT retrieve the document.",
+    turns=[
+        "Data dan dokumen apa saja yang ada di analisis ini?",
+        "Bagaimana standar mendefinisikan Physical Availability, dan bagaimana tren PA "
+        "bulanan kita sepanjang 2025?",
+        "According to the standard, what does PA measure that UA does not — and how does "
+        "PA compare between our two sites?",
+        "Rata-rata PA per model unit berapa?",
+    ],
+    document_ids=(BUMA_DOCUMENT_ID,),
+)
+
+
 # Default scripted conversation. Chosen to exercise every router intent against the
 # real Titanic columns (Survived, Sex, Pclass, Age, Fare, Embarked).
 DEFAULT_TURNS = SUITES["pa"].turns
@@ -555,6 +611,37 @@ def save_transcript(
 # ───────────────────────────── setup / turns ─────────────────────────────────
 
 
+async def _document_sources(user_id: str, document_ids: tuple[str, ...]) -> list:
+    """Catalog sources for bound documents, shaped like Go's `buildBoundSource`.
+
+    Only documents this user owns whose v2 ingestion completed — a document that
+    never indexed would bind, retrieve nothing, and make a green-looking run.
+    """
+    if not document_ids:
+        return []
+    from datetime import UTC, datetime
+
+    from sqlalchemy import select
+
+    from src.catalog.models import Source
+    from src.db.postgres.connection import AsyncSessionLocal
+    from src.db.postgres.models import Document
+
+    async with AsyncSessionLocal() as s:
+        rows = (await s.execute(
+            select(Document).where(
+                Document.id.in_(document_ids),
+                Document.user_id == user_id,
+                Document.status == "completed",
+            )
+        )).scalars().all()
+    return [
+        Source(source_id=d.id, source_type="unstructured", name=d.filename,
+               location_ref=f"document://{d.id}", updated_at=datetime.now(UTC))
+        for d in rows
+    ]
+
+
 async def setup_analysis(user_id: str, suite: Suite, bind_source: bool) -> str:
     """Create a fresh analysis session (state row) + optionally bind it to the source.
 
@@ -587,20 +674,29 @@ async def setup_analysis(user_id: str, suite: Suite, bind_source: bool) -> str:
             # user's catalog restricted to the bound source. structured_flow reads it via
             # CatalogStore.get_by_analysis (the data_sources binding table was removed;
             # in production Go materializes B from analyses.data_bind).
-            user_cat = await CatalogStore().get(user_id)
-            bound = [
-                s for s in (user_cat.sources if user_cat else [])
-                if s.source_id == suite.source_id
-            ]
-            if not bound:
-                # Hard stop, not a warning. Falling open to the whole catalog here
-                # means every answer is about a DIFFERENT dataset than the suite
-                # was written for, and the run still looks green. Better to refuse.
+            bound = []
+            if suite.source_id:
+                user_cat = await CatalogStore().get(user_id)
+                bound = [
+                    s for s in (user_cat.sources if user_cat else [])
+                    if s.source_id == suite.source_id
+                ]
+                if not bound:
+                    # Hard stop, not a warning. Falling open to the whole catalog here
+                    # means every answer is about a DIFFERENT dataset than the suite
+                    # was written for, and the run still looks green. Better to refuse.
+                    raise RuntimeError(
+                        f"source {suite.source_id} ({suite.name}) is not in "
+                        f"{user_id}'s catalog — wrong --user-id for this suite?"
+                    )
+            docs = await _document_sources(user_id, suite.document_ids)
+            if len(docs) != len(suite.document_ids):
                 raise RuntimeError(
-                    f"source {suite.source_id} ({suite.name}) is not in "
-                    f"{user_id}'s catalog — wrong --user-id for this suite?"
+                    f"only {len(docs)} of {len(suite.document_ids)} suite documents are "
+                    f"v2-ingested and owned by {user_id} — re-ingest via /api/v2/document/*"
                 )
-            else:
+            bound = [*bound, *docs]
+            if bound:
                 scoped = CatalogModel(
                     user_id=user_id, generated_at=datetime.now(UTC), sources=bound,
                 )
@@ -610,8 +706,9 @@ async def setup_analysis(user_id: str, suite: Suite, bind_source: bool) -> str:
                         catalog_payload=scoped.model_dump(mode="json"),
                     ))
                     await s.commit()
-                print(f"  bound source: {suite.name} ({suite.source_id})  "
-                      f"{c('dim', '→ structured_flow scoped to it (analysis catalog)')}")
+                for s_ in bound:
+                    print(f"  bound {s_.source_type}: {s_.name} ({s_.source_id})")
+                print(c("dim", "  → turns scoped to these (analysis catalog)"))
         except Exception as e:  # noqa: BLE001 — fail-open to whole catalog
             print(c("warn", f"  binding skipped ({type(e).__name__}: {e}) — "
                             f"fail-open to whole catalog"))
@@ -756,7 +853,7 @@ async def amain(args: argparse.Namespace) -> None:
         enable_tracing=False, enable_gate=False
     )
     suite = SUITES[args.suite]
-    if suite.source_id is None:
+    if suite.source_id is None and not suite.document_ids:
         # Refused here, before a single token is spent: the source is not in
         # dedorch, so structured_flow would silently fall back to the whole
         # catalog and every answer would be about the WRONG data -- a green run