Rifqi Hafizuddin
Claude Opus 5
[NOTICKET] docs: sync REPO_STATUS / DEV_PLAN / contract for the 8 Sep list (K24)
be227a8 | # Data Eyond — Current Development Plan (post 2026-06-24 → 2026-06-30 checkpoints) | |
| <!-- doc-status --> | |
| > ✅ **LIVE — CANONICAL: what is in flight.** Sprint plan, task table and locked decisions (§1/§1.5 are locked; §6 lists what is still open). Built state: [REPO_STATUS.md](REPO_STATUS.md) · all other docs: [docs/](docs/). | |
| **Purpose:** context file for Claude Code sessions working on the current sprint. | |
| **Branch:** `pr/5` · **Snapshot:** 2026-06-30. | |
| **Companion:** [REPO_STATUS.md](REPO_STATUS.md) describes the repo's *current built state*; this file | |
| describes the *in-flight plan* that changes it. The **active sprint is pr/5** ([§0](#0-current-sprint--pr5-observability--endpoint-restructure)); sections §1–§6 are the prior | |
| 2026-06-24/25 pivot (now largely ✅), kept for context. | |
| --- | |
| ## 0. Current sprint — pr/5: Observability + Endpoint Restructure | |
| From the **2026-06-30 checkpoint**. Direction: **Python → generation/AI-only**; Go owns the analysis | |
| lifecycle + data plane. Endpoint contract sent to Harry on 2026-06-30: | |
| [API_ENDPOINTS_RESTRUCTURE.md](docs/archive/API_ENDPOINTS_RESTRUCTURE.md) (chat→v2, tools regroup, observability — | |
| observability marked tentative). REPO_STATUS carries a matching `pr/5` direction banner. | |
| Mentor's task order: **unwire → regroup endpoints → add tools (retrieve-data + observability)**; share | |
| the endpoint contract *before* coding the tools. Status legend: ⬜ not started · 🔄 in progress · ✅ done · | |
| ⛔ blocked · 🔎 verify · ⏸️ deferred. | |
| | Phase | Task | Owner | Status | Notes | | |
| |---|---|---|---|---| | |
| | **P0 — contract** | Draft + send endpoint contract to Harry (chat v2 · tools group · observability) | Rifqi + Sofhia | ✅ | `API_ENDPOINTS_RESTRUCTURE.md` sent 2026-06-30 (before-noon deadline met). Observability section flagged tentative. | | |
| | **1 — unwire** | Unwire `users`(login)/`document`/`room`/`db_client`/`data_catalog`/`analysis` from `main` + Swagger | Sofhia | ✅ | **KM-686**, commit `0b2d678`. Commented, not deleted; `chat`/`report`/`tools` kept mounted. Resolves the analysis-CRUD scope Q — whole `analysis` router unwired (Go owns it). | | |
| | **2 — v2 + regroup** | Create `src/api/v2/` and move the chat pilot there | Rifqi | ✅ | New `src/api/v2/__init__.py` + `src/api/v2/chat.py` (`POST /api/v2/chat/stream`), mounted in `main.py`. Only chat in v2; v1 `/chat/stream` kept mounted until FE moves over. Routes import-verified. | | |
| | **2 — v2 + regroup** | Chat: `room_id` → **`analysis_id`** (request field + handler + history) | Rifqi | ✅ | v2 `ChatRequest{user_id, analysis_id, message}`; reuses warm `ChatHandler` + v1 cache/history helpers; `done` returns `{message_id}` (always minted Python-side, server-authoritative — open-Q #1 resolved in pr/6). Persistence kept transitionally → still ties to #25 (`analyses_messages`); ruff-clean. | | |
| | **2 — v2 + regroup** | Move report under tools → `/api/v1/tools/report` (+ version routes) | Rifqi | ✅ | report router re-prefixed `/api/v1` → `/api/v1/tools` (all 3 routes move together), tag → `Tools`; old `/api/v1/report` gone. Same functionality, new home. Import-verified. | | |
| | **2 — v2 + regroup** | Move help under tools → `POST /api/v1/tools/help` (dedicated endpoint) | Sofhia | ✅ | New `src/api/v1/help.py` (SSE: `sources:[]`→`chunk`→`done{message_id}`) + additive `ChatHandler.stream_help()` (reuses HelpAgent+state+readiness, no router). Generative-only (no persist). **Router `help` intent KEPT** — both paths live by design. message_id always minted Python-side, server-authoritative (open-Q #1 resolved, pr/6). Import-verified. | | |
| | **2 — v2 + regroup** | Tools list → `/api/v1/tools/list` | Sofhia | ✅ | Renamed route `GET /api/v1/tools` → `GET /api/v1/tools/list` ([tools.py:133](src/api/v1/tools.py:133)). | | |
| | **2 — v2 + regroup** | FE: slash menu = `/help` only; report = right-side button | Mentor (FE) | ⬜ | Coordination note, not Python work. | | |
| | **3 — tools + obs** | Finish `help` so it actually **calls** (not just lists) + test | Sofhia | ⬜ | Mentor: help currently only lists tools. Core #2 after chat. | | |
| | **3 — tools + obs** | Traceability **scratchpad** accumulating in the chat agent | Rifqi + Sofhia | ✅ | **KM-691.** `TraceabilityScratchpad` + `TraceabilityToolInvoker` (`src/traceability/`) capture planning / tool I/O / sources during the run; flushed one row before every `done` (all 8 sites; error turns = no row). Renamed observability→traceability (vs. Langfuse). | | |
| | **3 — tools + obs** | Audit `report_inputs` — covers planning + tool I/O + source? add cols / new store | Rifqi | ✅ | **KM-691.** Chose a dedicated store: `message_traceability` = 1 JSONB row per message (Python-owned, like `report_inputs`; DDL run manually against dedorch, handed to Harry). Langfuse kept for engineering. | | |
| | **3 — tools + obs** | Build `GET /api/v1/traceability` (one merged response) | Rifqi | ✅ | **KM-691.** `src/api/v1/traceability.py` → store.get → payload/404. Intent-based source rules (greeting/help/refusals = none; retrieve = required); full planning only on slow path. Contract §7 updated. | | |
| | **3 — tools + obs** | Keep stream **text-only**; traceability is a separate parallel call | Rifqi | ✅ | **KM-691.** No trace data in the SSE stream; the FE fetches `/traceability` on `done`. | | |
| | **3 — tools + obs** | Resolve `message_id` correlation (stream ↔ traceability) with Harry | Rifqi ↔ Harry | ✅ | **RESOLVED (pr/6):** Python is the **sole minter** — `message_id` dropped from the `/api/v2/chat/stream` + `/api/v1/tools/help` request bodies; always minted server-side (server-authoritative, FE-security) and returned on `done`. Any caller-sent `message_id` is ignored. Contract open-Q #1 closed. **Updated 2026-07-09 (pr/13):** id format changed from `msg_<hex>` to a canonical UUID string (`str(uuid.uuid4())`, both mint sites) to mirror Go's `analyses_messages.id` shape — the value is still independently Python-minted (not the real row id), only format-compatible for a future swap. | | |
| | **4 — biz questions** | Get Go folder; confirm `business_questions` in create-analysis (max 5); sync Python | Harry/Mentor → Rifqi | ⬜ | Go currently missing the field ("lagi difixing"). Python already models objective + business_questions. | | |
| | **deferred** | Report formats: PPT (preferred) / PDF / infographic on download | — | ⏸️ | MD is fine for the FE preview stage now. | | |
| | **deferred** | Charts (Plotly→JSON) + images tables | — | ⏸️ | Carried from §4 #26/#27. | | |
| **Next up:** Phase 2 Python work is **done** (chat→v2 `analysis_id`; `help`/`report`/`list` regrouped | |
| under `/api/v1/tools/`). The `message_id` correlation contract is now settled in **pr/6** (Python sole | |
| minter, stream-only). The **Phase 3 traceability build** — scratchpad + `GET /api/v1/traceability` | |
| (contract §7) — is **done (KM-691)**. Next is **Phase 4** (business questions, Go-blocked). | |
| --- | |
| ## 0.5. pr/13 sprint — agent-quality fixes (2026-07-08 live-test review) | |
| Findings from the scoped live sessions (mining analysis, 2026-07-07/08 traces): the planner | |
| force-mapped absent measures (`pa` aliased as "revenue"), top-N ranked raw rows (duplicate models), | |
| `analyze_trend` collapsed integer months into a single 1970-01 bucket, an invalid grouped IR reached | |
| Postgres, failed retrievals wrote all-null traceability sources, and numeric catalog samples arrive | |
| base64-mangled from Go. Fix tasks (same status legend as §0): | |
| | # | Task | Owner | Status | Note | | |
| |---|---|---|---|---| | |
| | Q1 | IR validator: reject bare selects under `group_by` (planner retry self-corrects) | Rifqi | ✅ | `query/ir/validator.py` | | |
| | Q2 | Planner **infeasible** path: `TaskList.infeasible_reason` + deterministic EN/ID data-gap reply | Rifqi | ✅ | schemas/validator/coordinator/refusals + planner.md "When the catalog cannot answer"; refusal wording → Rifqi to review | | |
| | Q3 | `analyze_trend`: integer year/month handling (epoch-parse bug) | Rifqi | ✅ | `temporal.py` + 5 local tests | | |
| | Q4 | Planner few-shots: top-N (Example G) + infeasible (Example H) + entity-vs-row ranking rule | Rifqi | ✅ | live-tested 2026-07-08: backlog top-3 correct via single-IR group+sum; "best PA performance" correct in-process (avg-per-model, assumption recorded). Stale-server trace was a false alarm | | |
| | Q5 | Catalog numeric `sample_values` base64-decode stopgap (`catalog/sample_decode.py`) | Rifqi | ✅ | self-disabling; **primary fix = Go marshaling — DDL-free handoff to Harry** | | |
| | Q6 | Traceability null-source suppression + `check_data` `-1` row-count hiding | Rifqi | ✅ | `scratchpad.py` / `data_access.py` | | |
| | Q7 | `analyze_merge` two-table combine tool (unblocks "worst A + biggest B" questions) | tool owner | ✅ | tool shipped by Sofia (8abf635, KM-703); planner slice done 2026-07-09: `_validate_data_source` guards `data_right`, two-retrieve→merge few-shot (Example I), planner.md "Two measures per entity" bullet | | |
| | Q8 | Report v2: business-question answer section, unresolved/excluded sections, evidence tables from `results_snapshot`, caveat dedupe, single language | Rifqi/Sofhia | ✅ | done 2026-07-09: still exactly ONE LLM call (extended to also draft `bq_answers`, index-based record refs, deterministic fallback = v1 behavior); evidence tables from table-kind outputs (≤3/record, ≤10 rows, ≤8 cols, `check_*` skipped); reply language via `detect_reply_language` on objective+BQs; verified in-process against live analysis 935a091e | | |
| | Q9 | Record-curation endpoint (`GET …/records` + `exclude_record_ids`) + readiness GET for the FE delta guard | Rifqi ↔ FE | ✅ | done 2026-07-09: `GET /tools/report/{analysis_id}/records` + `/readiness` (registered before `/{version}` — int-coercion route-order trap), `exclude_record_ids` on POST; contract updated same change; FE wiring pending (Rifqi → FE) | | |
| | Q10 | Traceability `data_used` layer — resolve IR ids → real names for the FE (users couldn't map `c_…`/`t_…` ids back to their data; aggregate aliases like `total_revenue` looked like real columns) | Rifqi ↔ FE | ✅ | done 2026-07-13 (pr/15): new `src/traceability/resolve.py` builds `data_used[]` (real source/table/column names; joins; plain-language filters; `columns_read` vs `output_columns` with `computed`+`formula`), `tool_calls[].summary`, `sources[]` gains `source_name`+all tables; **ids kept but machine-only (FE must not render)**; deterministic no-LLM, never-throw; catalog threaded to the scratchpad at the composition root. Contract + `TRACEABILITY_FE_HANDOFF.md` updated; FE wiring pending (Rifqi → FE). Additive/non-breaking | | |
| | Q11 | IR wart: `OrderByClause.column_id` may hold a SELECT **alias** (a computed output), not a catalog column_id | Rifqi | 🔎 | surfaced by Q10 — the resolver tolerates it (`kind: "computed"` fallback), but the IR field name is misleading. Consider an explicit `by_alias` field or renaming. Low priority; no functional bug | | |
| ## 0.6. pr/16 sprint — Spine v2: W2 charts + W1 checkpoint (SPINE_V2_PLAN, approved 2026-07-13) | |
| Scope approved 2026-07-13 (Rifqi + Sofia): build W2 (`render_chart` + chart store + `GET /charts` | |
| + planner viz slice) and W1 (S1a quality checkpoint). **W3 (activate deferred `analyze_*`) deferred | |
| at approval — do not start until Rifqi re-opens.** W4 (S1b repair) stays gated on INV-6 sign-off + | |
| S1a telemetry. Design + handoff source: `SPINE_V2_PLAN.md`. | |
| | # | Task | Owner | Status | Note | | |
| |---|---|---|---|---| | |
| | V1 | `render_chart` tool slice: `visualization.py` (Plotly-JSON `dataeyond.chart.v1` envelope, no plotly dep) + `ToolOutput.kind` `"chart"` + registry + invoker | Rifqi (Sofia signed off on the tool-layer edit) | ✅ | done 2026-07-13; deterministic spec builder (bar/line/pie/scatter, fixed style preset), traceability scratchpad summarizes chart outputs compactly (point_count, not the raw arrays) | | |
| | V2 | Chart store + API: `MessageChartRow` (`message_charts`) + `src/charts/store.py` (never-throw save) + write site in `_run_slow_path` + `GET /api/v1/charts` + contract §charts | Rifqi | ✅ | done 2026-07-13; empty list = valid 200; `done` event unchanged (no `chart_count` — open, Harry); FE fetches unconditionally on `done` | | |
| | V3 | Planner viz slice: recipe table + "Charts only on explicit ask" rule (planner.md), Example J (viz tail) + Example K (viz-infeasible), validator Check 10 (`render_chart.data` must be table-kind), assembler chart one-liner guard | Rifqi | ✅ | done 2026-07-13; behavioral matrix verified in-process (real LLM): explicit ask EN/ID → chart tail; plain question → no chart; absent dimension → infeasible (Example K + prompt guard added after the first smoke force-mapped `status` AS "region") | | |
| | V4 | W1 S1a quality checkpoint: `slow_path/checkpoint.py` (CK1–CK6) + `RunAssessment` schemas + coordinator call site + "Execution assessment" block in the assembler input + `refusals.run_failure_message` | Rifqi | ✅ | done 2026-07-13; 13 local tests (one per CK rule + never-throw + prompt + coordinator); CK1 all-failed → deterministic honest failure, **no assembler call**; clean run renders nothing (zero behavior change); every flag logs `repair_candidate` (S1b evidence) | | |
| | V5 | `message_charts` DDL: run manually against dedorch (block for the live e2e chart test), then hand the schema to Harry for the dedorch migration | Rifqi → Harry | ✅ | Rifqi ran the DDL 2026-07-13; **live e2e ALL PASS** same day (real v2 endpoint: viz turn → chart row keyed by `done` message_id → `GET /charts` valid v1 envelope; chartless → 200 empty; injected `render_chart` failure → answer streams as a table, no row). **Remaining: send Harry the migration handoff** (schema + contract §charts) | | |
| | V6 | Restore the `eval.chat_sim` harness — `eval/chat_sim/*.py` is missing from disk AND git (only `__pycache__` remains); §7B prompt gate can't run | Rifqi | ✅ | restored by Rifqi 2026-07-13 (accidental delete). ⚠️ its hard-coded `DEFAULT_USER_ID`/`TITANIC_SOURCE_ID` are stale (that user has no catalog; the Titanic blobs are gone) — update the constants before the next full run | | |
| | V7 | Local `.env` lagged Go's Supabase-S3 data plane: `storage_provider=azure_blob` + empty `supabase_s3_*` made EVERY local tabular retrieve fail `BlobNotFound` | Rifqi | ✅ | found during the e2e (masked as "data not available" by the honest-degrade path); Rifqi set the six values 2026-07-13. Gotcha documented in REPO_STATUS §13 | | |
| | V8 | Lead review of `GET /charts`: lookup by `message_id` alone + tri-state response marker (`status: success \| empty \| not_found` + `message`) instead of a bare list | Rifqi (lead ask) | ✅ | done 2026-07-14; `not_found` vs `empty` decided against the turn's traceability row (PK lookup); always HTTP 200; contract §charts updated. ⚠️ additive DDL for the new lookup: `CREATE INDEX IF NOT EXISTS idx_message_charts_message ON message_charts (message_id);` (run manually + include in Harry's migration). Traceability GET still takes both params — aligning it is open | | |
| | V9 | Report chart embedding: `AnalysisReport.charts` (verbatim envelopes per record) + `## EDA` section with ` ```plotly ` fences (content = the **full v1 envelope**, pretty-printed — the FE hook's verified shape); `has_successful_analysis` extended so a successful `render_chart` counts (chart-only sessions satisfy the report floor) | Rifqi | ✅ | done 2026-07-14; first cut emitted bare `{data, layout}` — FE test showed the hook parses the full envelope, fixed same day; live reports v3 (wrong fence) → **v4 (correct)** for analysis `7be50846…` (3 charts embedded); suite **381 passed, 7 skipped** (+5 chart-embed tests; same 2 pre-existing failures) | | |
| Full-suite evidence for this sprint: **376 passed, 7 skipped** (+13 new checkpoint tests; the 2 | |
| failures — `test_chat_handler::test_structured_flow_runs_slow_path`, | |
| `test_reader::test_structured_read_falls_back_to_user_scope_when_no_analysis_row` — reproduce at | |
| HEAD before this diff, i.e. pre-existing). Ruff clean on all touched paths; `import main` OK. | |
| ## 1. The direction change (locked decisions from 2026-06-24) | |
| 1. **"Problem statement" is replaced by two user-entered fields: `objective` + `business_questions`.** | |
| User fills them at onboarding; **both mandatory to submit; NO agent validation.** | |
| 2. The **gate (`problem_validated`) and the `problem_statement` skill/intent are removed** (comment out, don't delete). | |
| 3. **Report is records-based** (reads persisted `AnalysisRecord`s) — **decided and pushed** (KM-674). | |
| It is formal markdown: title, date, "generated by {user}", objective, business questions, | |
| findings, insights. **NOT gated** on whether business questions were answered. | |
| 4. **`owner_id` → `user_id`** everywhere (Harry mirrors in dedorch/Go). | |
| 5. **State writes go through a request to Go**, not direct Python DB writes. | |
| 6. **FE-callable surface = 4 endpoints:** `call_agent` (chat/stream), `list_skills` (`GET /tools`), | |
| **skill: help**, **skill: report**. `problem_statement` removed; `check_data` not FE-facing from | |
| Python (Go provides it); analysis CRUD not needed from FE (comment, don't delete). | |
| 7. Deliverables for Harry: (a) API endpoint doc (MD); (b) full Python project doc (MD → PDF/Word BRD). | |
| 8. Integration tested via Swagger `/docs` on the HF Python build (simulating FE manually). Target ~Wed. | |
| ## 1.5. 2026-06-25 checkpoint deltas | |
| Confirms the 2026-06-24 direction and adds these concrete changes (folded into §4 as tasks 21–28): | |
| 1. **Rename `analysis_records` → `report_inputs`** (DONE #21) — names the table by purpose (the rows | |
| report generation reads); avoids clashing with Go's `analyses_messages` and with Langfuse | |
| observability. **Stays Python-owned**; finalized schema handed to Harry so his dedorch migration | |
| creates it post-`SKIP_INIT_DB` (#22, resolves #16). Write scope = **one row per slow-path analysis | |
| run** (decided — not per-agent-call telemetry; that stays Langfuse). | |
| 2. **`analyses` table (Go) — `status`, `data_bind` + `data_bind_version`, `report_collection`** (id+version). | |
| **Verified 2026-06-25: these + `user_id` are ALREADY present in dedorch `analyses`.** Plus Harry drops | |
| the duplicate/wrong singular `analysis` table. (→ #3) | |
| 3. **`analyses_messages` (Go) = the analysis chat room** (user Q + agent A) — replaces the now-**deprecated** | |
| `chat_messages`/`rooms`; Python's chat read/write must migrate here before cutover. (→ #25) | |
| 4. **Reports: Go owns ALL writes.** Report stays a **skill** (no router intent): FE → Go → Python; | |
| Python only returns content. Input = the records table (now `agent_observability`); edit-mode may | |
| also need the last report. (→ #7/#18/#24) | |
| 5. **Markdown minimum now:** tables, **bold**, *italic*, horizontal separators — optimize that before | |
| anything fancier. (→ #23) | |
| 6. **Deferred:** charts (prefer **Plotly→JSON** in a future `chart` table over matplotlib PNGs) and | |
| images (image table keyed by analysis/message/report + originals in a bucket). (→ #26/#27) | |
| 7. **Near-term:** the remove-`problem_statement` work isn't on HF yet → **PR + deploy + test in the | |
| playground** (#13). Harry stabilizes Go ~Fri; FE manual testing ~Mon. **Keep it playground-able.** | |
| 8. **UI research** (no dedicated UI person): new-analysis form (title/objective/business_questions), | |
| knowledge menu (user-level vs analysis-level binding), report artifacts panel + version selector; | |
| interview + old analysis UI removed. (→ #28) | |
| ## 2. What is already done (KM-674, pushed on `pr/4`) | |
| Report layer adapted to the new goal shape: | |
| - `report/schemas.py::ProblemStatement` → `objective: str` + `business_questions: list[str]` | |
| (old `target_value`/`scope`/`metric_direction`/`target_metric` dropped). Class name kept for now | |
| (rename to `ReportGoal` once the upstream AnalysisState rename lands). | |
| - `report/generator.py` renders **Objective** + numbered **Business Questions** + a | |
| **"generated by {user}"** line. | |
| - `api/v1/report.py::_problem_statement_from` is **tolerant**: prefers new `objective` / | |
| `business_questions` from state, falls back to legacy `problem_statement` — works before AND after | |
| Harry's migration. | |
| - `config/prompts/report_summary.md` updated to objective + business questions. | |
| - Report stays **records-based**; the floor gate (`problem_validated`) was deliberately left for task #2. | |
| **This tolerant-migration pattern (getattr fallback) is the model for tasks #2 and #4.** | |
| ## 3. Assessment — gaps & contradictions to resolve before building | |
| These came out of reviewing the plan against the actual code. They are folded into the task table (§4) as tasks 15–19. | |
| - **G1 (→ task 15). Records-based reports need the slow path ON.** `AnalysisRecord`s persist only in | |
| `chat_handler._run_slow_path`, which runs only when `ENABLE_SLOW_PATH=true`. Default is off → no | |
| records → `POST /report` 409s. The Swagger demo can't show a non-empty report unless slow path is | |
| flipped on and a `structured_flow` question is run first. `BusinessContext` is still a stub but the | |
| slow path runs fine on it. | |
| - **G2 (→ task 16). `analysis_records` ownership is now required and collides with `SKIP_INIT_DB`.** | |
| It's created today by Python `create_all` (`db/postgres/init_db.py`), is in no dedorch/Go migration, | |
| and after the dedorch cutover (`SKIP_INIT_DB=true`) Python stops running `create_all` → the table | |
| won't exist → reports break. Decide: dedorch migration (Harry) OR a Python carve-out that creates | |
| just this one table even under `SKIP_INIT_DB`. *(Resolved 2026-06-25 — see §1.5.1 / #16 / #22.)* | |
| - **G3 (→ task 17). `chat_history` in the report contract is vestigial.** Records-based generation | |
| reads records by `analysis_id`; it never uses chat history. Drop `chat_history` from the report | |
| skill contract, or mark it reserved/unused. | |
| - **G4 (→ task 4 note). Make #2/#4 tolerant of both state shapes.** If Harry drops | |
| `problem_validated`/`owner_id` from dedorch before Python stops reading them, Python's gate + | |
| state_store break. Use the same `getattr` tolerance KM-674 used. The `owner_id`→`user_id` rename | |
| also touches `api/v1/analysis.py` (`_serialize_state`, `list_analyses`, `get_analysis`), not just | |
| the model + state_store. | |
| - **G5 (→ task 18). "State writes via Go" is bigger than `report_id`.** Python still writes state in | |
| `/analysis/create` (state + room + bindings, plus the data-first gate and soon the mandatory-field | |
| check) and in `state_store.ensure` per turn. If creation moves to Go (consistent with commenting | |
| analysis CRUD), then Go owns ALL state writes + both creation gates, and Python's `ensure` must | |
| become a **read-only get** (Go must guarantee the row exists before any chat turn). | |
| - **G6 (smaller).** | |
| - Removing `problem_statement` (task 1) means neutering it in 4 places: the `Intent` literal | |
| (`agents/orchestration.py`), the router prompt (`config/prompts/intent_router.md`), the handler, | |
| and the gate's redirect *target*. Do it with task 2. | |
| - "generated by {user}" currently prints the raw `user_id`; a formal report wants a name — source | |
| from `users.fullname` or have Go pass a display name (task 19). | |
| - The meeting's outline (background / EDA / insights) isn't fully in the renderer; map those | |
| sections onto the record fields deliberately (task 5 follow-up). | |
| - The full project doc (task 11) should reuse [REPO_STATUS.md](REPO_STATUS.md), not restart. | |
| ## 4. Task table | |
| Status legend: ⬜ not started · 🔄 in progress · ✅ done · ⛔ blocked · 🔎 verify · ⏸️ deferred. | |
| | # | Task | Owner | Status | Note | | |
| |---|---|---|---|---| | |
| | 1 | Comment out `problem_statement` skill **+ `Intent` literal + router prompt + gate redirect target**; remove `/problem-statement` from `list_tools` | Rifqi | ✅ | Done 2026-06-25 (one commit w/ #2). Unwired in `orchestration.py`, `intent_router.md`, `chat_handler.py`, `tools.py`; `problem_statement.py` kept intact | | |
| | 2 | Drop `problem_validated`: gate neutered; `is_report_ready`/`report_floor` → **≥1 completed analysis** only, no-LLM | Rifqi | ✅ | Done 2026-06-25. `gate.py` no-op, gate call site commented in `chat_handler.py`, `report_floor` drops the goal check. Tests updated (`test_gate`/`test_chat_handler`/`test_readiness`). Suite: **284 passed, 7 skipped**; ruff clean | | |
| | 3 | dedorch `analyses` migration: drop `problem_statement`/`problem_validated`, add `objective` + `business_questions` | Harry | 🔄 | **Verified dedorch 2026-06-25:** `analyses` (plural) ALREADY has `user_id` + `status` + `data_bind` + `data_bind_version` + `report_collection` → those parts done. **Remaining:** drop `problem_statement`/`problem_validated` + add `objective`/`business_questions`. Singular `analysis` = deprecated duplicate to drop | | |
| | 4 | Update Python `analyses` model + `state_store` + `analysis.py` to match dedorch; `owner_id`→`user_id` | Rifqi/Sofhia | ✅ | Done 2026-06-26. `owner_id`→`user_id` + added `status`/`data_bind`/`data_bind_version`/`report_collection` (DB-only, not in the `AnalysisState` pydantic) across `models.py`/`gate.py`/`state_store.py`/`analysis.py` + 3 local tests; also `report_inputs` `id`/`analysis_id` → `uuid`. Kept `problem_statement`/`problem_validated`; `objective`/`business_questions` wait on Harry's #3. Suite **284 passed** | | |
| | 5 | Report generator → `objective`+`business_questions`, "generated by {user}", formal outline | Sofhia | ✅ | Goal-shape (KM-674) + author name (#19) + outline (KM-680): Objective → Business Questions → Executive Summary → Key Findings → EDA → Notes & Limitations → How This Was Analyzed | | |
| | 6 | Report skill input contract: `analysis_id` + `user_id` (no `chat_history`) | Sofhia/Rifqi | ✅ | No-op: `POST /report` already takes only analysis_id + user_id (records-based). Documented in API_ENDPOINTS.md §5. *(Edit-mode input revisited in #24.)* | | |
| | 7 | `report_id` state update via request to Go, not direct DB | Sofhia + Harry | ⬜ | Needs Go endpoint. **Checkpoint:** Go owns ALL `reports` writes; Python stops any direct insert/update and only returns content; report stays a **skill** (no intent). See #18 | | |
| | 8 | Expose/confirm 4 FE endpoints; comment `check_data` + analysis CRUD | Sofhia | ✅ | KM-678: `list_tools` trimmed to `/help` + `/report` (analytics/check/retrieve commented in the **menu**). `help` confirmed as a `call_agent` intent — no own endpoint. Analysis CRUD endpoint left **registered**: "comment the rest" was about the FE slash menu, not killing HTTP routes Go needs | | |
| | 9 | Verify `analysis_id` in `call_agent` contract | Sofhia | ✅ | Verified: no separate field — carried as `room_id` (`analysis_id == room_id`), per REPO_STATUS §4/§11. Action for Go: send the id as `room_id` | | |
| | 10 | API endpoint doc (MD), 4 endpoints, for Go integration | Rifqi + Sofhia | ✅ | Done 2026-06-25 — `API_ENDPOINTS.md` (repo root). 4 FE surfaces with request/response **examples** (chat SSE transcript, report 201/409 JSON, version list), schemas, §9 full 32-route inventory + task-8 reading | | |
| | 11 | Full Python project doc (MD → PDF/Word BRD) | Rifqi | ✅ | Done 2026-06-26 — `PROJECT_BRD.md` (repo root): purpose/context, FR-1..9 capabilities, lifecycle, architecture, data model, API (→ API_ENDPOINTS), NFRs, integrations, open items. Reuses REPO_STATUS/API_ENDPOINTS; convert to PDF/Word for distribution | | |
| | 12 | Reconcile/open the `list_tools` PR cleanly (stacked commits) | Rifqi | ✅ | N/A — we develop directly on the single active branch `pr/4` (KM-652 + KM-678 already stacked there); no separate PR to reconcile | | |
| | 13 | Deploy HF Python build (remove-`problem_statement` work) → test 4 endpoints via Swagger / playground | Sofhia + Harry | 🔄 | **Unblocked (#15 ✅).** Remove-PS work is on `pr/4` but **not on HF `main` yet** → PR + deploy, then manual test. Harry stabilizes Go ~Fri; FE testing ~Mon | | |
| | 14 | `analysis_records` home | Rifqi + Sofhia + lead | ✅ | **Resolved 2026-06-25:** stays Python-owned, **renamed** (→ #21); schema handed to Harry so the dedorch migration creates it post-cutover (→ #22). Not moved to Go | | |
| | 15 | Flip `ENABLE_SLOW_PATH=true` + verify an `AnalysisRecord` persists from a `structured_flow` question | Rifqi | ✅ | Verified locally 2026-06-25 (in-process). structured_flow on Titanic.csv → 3-task plan `check_data→retrieve_data→analyze_aggregate` (all success) → AnalysisRecord persisted (substantive) → `report_floor` pass → report generates (201). HF env-flip + Swagger run folds into #13 | | |
| | 16 | Decide `analysis_records` creation under `SKIP_INIT_DB` | Rifqi + Harry | ✅ | **Resolved 2026-06-25:** Python defines it; **Harry's dedorch migration creates it** on env-move (Python still creates locally meanwhile) → exists post-cutover. Execution = #22 | | |
| | 17 | Reconcile report contract with records-based: remove/flag `chat_history` | Sofhia/Rifqi | ✅ | Nothing to remove — `chat_history` was never in the report contract/code (only in help.md). Confirmed via grep; API_ENDPOINTS.md §5 documents the clean contract | | |
| | 18 | Confirm Go owns ALL analysis-state writes + both creation gates; make Python `state_store.ensure` read-only | Rifqi + Harry | ⬜ | **Confirmed by 2026-06-25 checkpoint** (Python read-only; Go owns writes + new tables). Execution pending Go endpoints | | |
| | 19 | Decide report author display-name source (`users.fullname` vs Go-passed name) | Sofhia | ✅ | Done 2026-06-25. `AnalysisReport.user_name`; `generator` renders `user_name or user_id`; `api/v1/report.py::_resolve_user_name` reads `users.fullname` never-throw (fallback `user_id`). Decided: resolve in Python (unblocked); swap to Go-passed name later if preferred | | |
| | 20 | **Help handoff:** update `handlers/help.py` + `help.md` — drop the `problem_validated` tier + `define_problem_statement` action (the skill it points at is gone as of #1) | Sofhia | ✅ | Done 2026-06-25. `help.py`: actions = `ask_analysis_question` (always) + `generate_report` (if ready); renders objective/business_questions (getattr-tolerant). `help.md` v1→v2: 3 tiers, no `/problem_statement`, `/generate report`→`/report`. Local test_help updated → 11 pass | | |
| | 21 | Rename `analysis_records` → **`report_inputs`** (table, ORM `ReportInputRow`, store `*ReportInputStore`) | Rifqi | ✅ | Done 2026-06-26. `sed` rename across 9 files; Pydantic `AnalysisRecord` kept; columns stay String (pure rename — uuid+FK is the #22 Harry schema). Name `report_inputs` (purpose; avoids Langfuse/`analyses_messages` clash). Write scope = one row per slow-path run. Suite **284 passed** | | |
| | 22 | Finalize `report_inputs` schema → hand to Harry for the dedorch migration | Rifqi → Harry | ✅ | **DDL ready** (uuid `id`/`analysis_id` + FK→`analyses(id)`; `user_id`/`plan_id` text; `data` jsonb = serialized `AnalysisRecord`, shape documented). dedorch has empty `analysis_records` → rename. Resolves #16. ~~**Action: send Harry the DDL + `data` shape**~~ **RESOLVED 2026-07-22:** Harry's `0001_create_core_schema.sql` now carries `report_inputs` with exactly this shape — FK to `analyses(id)` included. Python's `ReportInputRow` diffed against the live Neon table: **column-for-column identical, no drift.** The two remaining un-migrated Python tables (`message_traceability`, `message_charts`) split out as → #32 | | |
| | 23 | Report markdown formatting: tables, **bold**, *italic*, horizontal separators | Sofhia | ✅ | Done 2026-06-25. Added `---` separators between header + each section in `_render_markdown`. Tables (EDA) / bold (method labels) / italic (meta + citations) already emitted. Relaxed `report_summary.md` to allow inline `**bold**`/`*italic*` for emphasis (kept no-headings/no-bullets so it doesn't duplicate the section structure / Key Findings). Compile + ruff clean | | |
| | 24 | Clarify report input contract: records table (+ `last_report` for edit mode?) | Rifqi/Sofhia ↔ Harry | ⬜ new | Edit-mode input left open at the checkpoint | | |
| | 25 | Migrate Python chat path to Go `analyses_messages` (+ `analyses`) | Rifqi ↔ Harry | ✅ | Done 2026-07-02. Read path already on `analyses_messages` (commit `0066161`). This change makes Python **read-only**: removed the `save_messages` calls from `/api/v2/chat/stream` so **Go is the sole writer** — fixes the double-write both Go+Python were producing. `load_history` still reads `analyses_messages`. v1 `/chat/stream` is unwired so left untouched | | |
| | 26 | **Charts (DEFERRED):** store Plotly JSON in a future `chart` table (not matplotlib PNG) | — | 🔎 | **Landed 2026-07-13 (pr/16, §0.6 V1–V3):** `render_chart` + `message_charts` + `GET /api/v1/charts`, Plotly JSON as decided. 🔎 pending the dedorch DDL run (V5) + live e2e | | |
| | 27 | **Images (DEFERRED):** image table (id, analysis_id, msg/report ref, order) + originals in a bucket | — | ⏸️ | Maintenance-heavy; parked | | |
| | 28 | **UI research** (FE): new-analysis form, knowledge menu (user vs analysis level), report artifacts + version selector | Team | ⬜ new | No dedicated UI person; interview + old analysis UI removed | | |
| | 29 | **LLM env quad rename `__4o` → `__54m`** + set the four `azureai__*__54m` secrets on the HF Space | Rifqi | 🔄 | Code done 2026-07-14 (pr/17): settings quad + 9 call sites + 2 eval runners renamed; deployment is `gpt-5.4-mini`. Suite **381 passed, 2 failed (both pre-existing), 7 skipped** — same as baseline; ruff clean on touched files; `import main` 0. **Hard rename — no `__4o` fallback**, so HF fails loudly on the first LLM call until the four `__54m` secrets are set. Root cause it fixes: HF silently ran GPT-4o while local ran 5.4-mini → same question, same catalog, planner hallucinated catalog ids on HF only. **Action: set HF secrets, redeploy, re-run the failing question.** Local `.env` still carries the now-dead `__4o` keys — safe to delete | | |
| | 30 | **Neon `reports.user_id` NOT NULL** — reconcile the ORM + `ReportStore` write | Rifqi | ✅ | Done 2026-07-22 (pr/18, commit `13f74e1`). `ReportStore.save` never wrote `user_id`; on Neon that column is `text NOT NULL` → **every `POST /api/v1/tools/report` 500'd** with `NotNullViolationError`. Fix: `models.py` adds `user_id = Column(Text)` (**nullable** on purpose — Python matches the loosest deployment shape, §7D getattr-tolerance) + `store.py` writes `report.user_id`, which the endpoint already required (`user_id: str = Query(...)`) and threaded via `generator.generate`. Live-tested by Rifqi → **201**. Suite 381 passed / 2 pre-existing failures / 7 skipped; `import main` 0; ruff unchanged (3 pre-existing E501 in `models.py`, same count on HEAD). **Correction 2026-07-22 (after pulling Go `dd37b38`):** the commit message and the first version of this row said the column was *not* in Go migrations `0001–0004`. **That was wrong** — `0001_create_core_schema.sql` has declared `reports.user_id TEXT NOT NULL` since the schema was written. Real root cause → #31 | | |
| | 31 | **Go migration set is not convergent** — fresh vs migrated dedorch DBs get different NOT NULL constraints | Rifqi → Harry | ⬜ new | Root cause of #30, verified in the Go source 2026-07-22. `0001_create_core_schema.sql` creates `reports.user_id TEXT NOT NULL` and `analyses_messages.user_id TEXT NOT NULL`; `0002_cleanup_legacy_schema.sql` (L95, L92) and `0004_replace_chat_with_analysis_scope.sql` (L61, L57) retrofit the same two columns onto pre-existing DBs as **nullable** `ALTER TABLE … ADD COLUMN IF NOT EXISTS user_id TEXT`. Because `CREATE TABLE IF NOT EXISTS` no-ops on an existing table, **the same migration set yields two different schemas** — the old dedorch DB got nullable (hiding Python's missing write for months), Neon got NOT NULL. **Ask Harry:** make the retrofit converge (backfill + `ALTER COLUMN … SET NOT NULL`, or add the constraint in a new migration) so every instance matches `0001`. Until then, `information_schema` on the target instance — not the migration files — is the only reliable schema source (REPO_STATUS §13). **Action: Rifqi raises with Harry** | | |
| | 32 | **`message_traceability` + `message_charts` are in no Go migration** — hand the DDL to Harry | Rifqi → Harry | ⬜ new | Found by the 2026-07-22 drift scan. Both tables exist on Neon **only because they were created by hand** (2026-07-06 / 2026-07-13); neither appears in `0001–0006`. Any newly provisioned dedorch instance will be missing them → traceability flush and chart persist fail (both never-throw, so they degrade **silently** — no 500 like #30 to make it visible). DDL for `message_charts` is in `SPINE_V2_PLAN.md` §4.4. This is the surviving half of #22. **Action: Rifqi sends Harry both DDL blocks** | | |
| ## 0.7. pr/19 — code review remediation (2026-07-23) | |
| From the end-to-end review in `CODE_REVIEW_2026-07-23.md` (findings are cited as **F-n** there) | |
| plus the live report bug on analysis `966224d4…`. Same status legend as §0. | |
| | # | Task | Owner | Status | Note | | |
| |---|---|---|---|---| | |
| | 33 | **Report body vs floor split** — `has_reportable_result` for the body, `has_successful_analysis` stays the floor | Rifqi | ✅ | Shipped 2026-07-23. Root cause: planner R2/R2b make `analyze_*` optional, so a correct analyze-free run was classed non-substantive, dropped from the body, and its business question rendered **"Unanswered"**. Live-verified by Rifqi on `966224d4…` | | |
| | 34 | **Report floor extension** — a successful `retrieve_data` **that returned rows** clears the floor | Rifqi | ✅ | Same root cause; fixes a hard **409** for a session where every question is R2/R2b-shaped. Guardrail-adjacent, authorised 2026-07-23. NOT the "Floor Fixer" failure mode: the floor still asks "did we produce a real result" — empty retrievals, `check_*`-only and fully-failed runs all still fail it | | |
| | 35 | **`GET …/records` `substantive` flag** repointed to the body predicate + contract §records updated | Rifqi | ✅ | The curation list was contradicting the artifact it curates. Behavioral, non-breaking: no field added/removed/retyped | | |
| | 36 | **CK5b** — quality checkpoint covers analyze-free plans | Rifqi | ✅ | CK5 only inspected `analyze_*` tasks, so an all-null aggregate column on an R2/R2b plan reached the answer unflagged. `check_*` excluded (uncounted tables legitimately carry nulls) | | |
| | 37 | **F-2 service-secret gate** — `X-Dataeyond-Service-Secret`, router-level dependency | Rifqi | ⏸️ | Shipped inert 2026-07-23. **UNWIRED 2026-07-27 (lead decision).** Verified in `E2E-Frontend-Data-Eyond/src/services/agenticApi.ts` that the **browser SPA is the only caller** of Python and we don't own it — it sends only `Content-Type`, and cannot be changed by us to send the header. So the gate could never be armed without a 401 outage: a wired-but-unarmable gate is a footgun (any operator setting `dataeyond__service__secret` breaks prod). Guard dependency removed from all six router mounts in `main.py`; `service_auth.py` kept in-tree but parked (comment-out-don't-delete); the `dataeyond_service_secret` setting commented. A prepared FE proxy (Node `server.js` holding the secret and injecting the header server-side) is the way to arm it later without exposing the secret to the browser — but that needs FE-repo access we don't have. **Net: the live surface is unauthenticated by design; real auth = #43. F-1 tenant predicates (#38) stay** as defensive-in-depth. | | |
| | 38 | **F-1 tenant scoping** — `user_id` predicate on the six analysis-keyed reads | Rifqi | ✅ | `CatalogStore.get_by_analysis` filtered on `analysis_id` alone where Go filters on both; the catalog payload carries the owner's `user_id`, so `DbExecutor`'s ownership check compared the victim's id against itself and passed → cross-tenant **query execution against a customer DB**. Defence-in-depth only until #37 is armed (`user_id` is caller-supplied, and `GET /traceability` leaks it) | | |
| | 39 | **Stale tests resolved** — the two long-standing suite failures | Rifqi | ✅ | Both encoded the pre-2026-07-13 user-scope fallback that `reader.py` deliberately removed. Not product bugs. Suite is now **394 passed / 0 failed / 7 skipped** — fully green for the first time | | |
| | 40 | ~~**F-3** — scope `GET /charts` + `GET /traceability` by `user_id`~~ | Rifqi | ⛔ | **RESOLVED 2026-07-23 — DECLINED by Rifqi. Do not re-open without his sign-off.** Both endpoints keep their existing lookup keys: `/traceability` by `(analysis_id, message_id)`, `/charts` by `message_id` alone (the 2026-07-13 lead decision). F-3 proposed adding a `user_id` parameter to both; an optional-param version was implemented on 2026-07-23 and then **reverted in full** (endpoints, charts-store predicates, and the contract notes) once the decision was restated — no FE change is required and none should be requested. **Accepted consequence:** both endpoints remain unauthenticated capability URLs over real customer data (`charts[].spec.plotly.data` is actual table values; the traceability payload carries 5-row previews, the executed SQL, and the owner's `user_id`). **The service-secret gate (#37) is therefore the only control protecting them** — which raises #37 from important to load-bearing. The stores' optional `user_id` parameters are kept as dead capability for a future Go-forwarded identity (#43); `PostgresChartStore` was returned to its message_id-only form. | | |
| | 41 | **F-12 / F-13** — bound the planner catalog render and the tabular blob read | Rifqi | ✅ | Shipped 2026-07-23. **F-13:** neither storage backend could report an object size, so `object_size()` was added to both (S3 `head_object`→`ContentLength`, Azure `get_blob_properties().size`, each returning None rather than raising) and `TabularExecutor` now refuses >500 MB **before** downloading, with a post-download byte check as the fallback when the probe is unavailable. **F-12:** `render()` gained TWO ceilings — `_MAX_TABLES=150` and `_MAX_CATALOG_CHARS=250_000` — because a table-count cap alone leaves wide tables unbounded (20 tables × 300 cols is as fatal as 400 × 30). Measured after: 400×30 went from ~241k tokens to ~63k; 100×30 (~45k tokens) still renders **in full**, so no realistic catalog is touched. Truncation emits an explicit "N more tables not shown" line so the planner knows it saw a subset, and logs — that log is the signal to retune. 12 new tests | | |
| | 42 | **F-20 observability** — `degraded_seam=<name>` on every never-throw / silent-drop path | Rifqi | 🔄 | The 2026-07-23 report bug was invisible by construction: the record was dropped with zero logging. **Partially shipped 2026-07-23** — the 10 seams where silent degradation is user-visible now emit a stable `degraded_seam` field (+ `repr(e)` instead of `str(e)`, so an empty-`str()` Fernet error is no longer a blank log): `input_guard_fail_open`, `analysis_catalog_read`, `report_floor_record_read`, `traceability_persist`, `traceability_flush`, `chart_persist` (×2), `report_input_persist` (×2), `analysis_state_ensure`. **Remaining:** the other ~76 `except Exception` sites, most of which are in unwired routers (`db_client`, `data_catalog`, `users`) or non-live paths — deliberately not swept, since a blanket edit across unwired code is exactly the drive-by §7A forbids. Control flow unchanged throughout (§5.4) | | |
| | 43 | **Go identity contract** — what does Go forward, and when? | Rifqi ↔ Harry | ⬜ | **Now the sole path to caller auth** after #37 was unwired 2026-07-27. Python cannot authenticate the caller alone; either Go forwards a verified per-user token, or the FE (once we can change it) forwards the Go bearer token it already holds (`orchestrationApi.ts` shows the FE has one). Until then the live surface is unauthenticated and the #38 predicates are defensive only. Options for arming interim protection when FE access returns: the `server.js` BFF proxy (secret server-side) or route agentic calls through Go | | |
| | 44 | **F-17 / F-18 / F-25 compiler-parity batch** | Rifqi | ✅ new | Shipped 2026-07-23. Three execution-verified review findings that had **no task row** — the tracker jumped from #39 to #40 and lost them. **F-17 (High):** the bare-select check was gated on `if ir.group_by`, so a mixed select with `group_by=[]` passed; Postgres then failed loudly but the pandas path silently DROPPED the column while `output_columns` still advertised it → a real-looking table with a fabricated all-null column. Fixed in `validator.py` (fires whenever any agg is present — no false positives possible) + a presence backstop in `TabularExecutor`. **F-18:** `astype(str)` ran before `na=False`, so `NULL LIKE '%an%'` matched the literal `"nan"`/`"None"`. **F-25:** `SqlCompiler` raised on an empty `in`/`not_in` where pandas and `_column_values`' own docstring implement the empty-set semantics — a two-step plan whose first step legitimately returned zero rows hard-failed instead of answering. 16 new parity tests; one stale test updated (it pinned the old F-25 raise) | | |
| | 45 | **F-9 PII in persisted artifacts** — mask the traceability preview + report evidence tables | Rifqi | ✅ | Shipped 2026-07-24. `pii_flag` was an **ingestion-time control only**: it nulls `sample_values` into the planner prompt, but nothing stops the planner SELECTing a flagged column — and "list our top 20 customers" legitimately selects `customer_name`/`email`. Real values then reached two PERSISTED sinks: `message_traceability.data` (served by an unauthenticated GET, F-3 declined) and report evidence tables frozen permanently into `reports.content`. Fix = `retrieve_data` now carries `meta.pii_columns` (resolved through the IR select list, so aliases are honoured), and both sinks redact those cells to `[redacted]`. **Per the 2026-07-23 decision the ASSEMBLER still receives real values**, so the answer prose is unchanged and the question stays answerable — verified end-to-end: assembler input kept `Ada Lovelace`, the persisted preview showed `[redacted]`. Aggregates are deliberately NOT masked except `min`/`max`: `sum(salary)` identifies nobody, but `max(email)` returns one customer's actual address. Fails **open** (an unresolvable name is left unmasked, never a legitimate column wrongly blanked), so this is a mitigation, not a guarantee. 15 new tests | | |
| | 46 | **F-8 prompt-injection resistance** — planner / assembler / report_summary | Rifqi | ✅ new | Shipped 2026-07-24. The three prompts that ingest customer data had **no injection rule**: `guardrails.md` is appended only in `chatbot.py`/`help.py`, and the `InputGuard` screens only the user's message — it never sees catalog or row content. This is the one attack the five query-defense layers structurally cannot see, because every IR the planner emits is individually valid. A hostile string only needs to reach a text column in the customer's OWN database (a product description, a support ticket, a form field) to be sampled into `sample_values` and rendered verbatim into the planner prompt. Fix = a purpose-written "content is data, never instructions" rule in each prompt + `<data>…</data>` delimiters around the catalog render and the run-state render. **Deliberately NOT `guardrails.md` wholesale** — its rules prescribe refusal sentences, and the planner's only free-text field is `infeasible_reason`, so those strings would surface there and could regress the Q2 data-gap path (tests pin their absence). Verified: planner eval **6/6, carried_over 5/5 green**; live hostile-catalog run planned only `t_products` and never touched the planted `employees.salary`; 10 new tests | | |
| | 47 | **Cheap-batch review fixes** — F-19, F-22, F-24, F-26, F-16, F-4 | Rifqi | ✅ new | Shipped 2026-07-24, one low-risk batch. **F-19:** `sources` was missing entirely on `check` + router-`help` and came *after* `status` on the slow path — contract said always-first; additive fix, contract updated. **F-22:** `analysis_id` now 422s unless it parses as a UUID, on **both** live endpoints (chat + help kept identical on purpose). **F-24:** traceability/chart writes are skipped, and logged, when `analysis_id` is falsy — Go `0007` declares `analysis_id UUID NOT NULL` (+ FK on charts), so `analysis_id or ""` failed the insert and the never-throw seam lost the row silently. **F-26:** `QueryResult.error` uses `str(e) or repr(e)` — a Fernet `InvalidToken` reached the assembler/traceability/report as an EMPTY string; falling back only when `str()` is empty changes no existing message. **F-16:** retrieval cache key gains `redis_prefix` (two envs on one Redis cross-served results). **F-4:** non-Postgres sources now refused at the executor — **zero blast radius today** (Go's `isSupportedActive` allows only `postgres`), a tripwire so nobody re-enables a path that has no read-only session and no `statement_timeout`; legacy branch commented out per house convention, orphaned import commented with it | | |
| | 48 | **floor_08 — floor/body disagreement** | Rifqi | ✅ | **Decided + shipped 2026-07-24 (lead).** #34's row-producing arm was unconditional, so a plan that HAS an `analyze_*` step which FAILED still cleared the floor on the strength of its upstream fetch — while the body rejected it. Because the "Attempted, Unresolved" section is commented out, such a run left no trace: as a session's only run it produced an empty report with the business question "Unanswered" (the #33 bug via another door). Fix = the arm now applies **only when the plan has no analysis step**, mirroring `has_reportable_result`; the shared `_plan_has_analysis` helper means the two predicates can no longer drift on that question. `floor_08` flips to `expected_ready: false`. The INTENDED asymmetry is preserved and re-verified: a zero-row retrieval still fails the floor but passes the body (floor_03). Readiness eval **17/17** | | |
| | 49 | **F-5 timeout does not stop the customer's query** | Rifqi | ✅ new | Shipped 2026-07-24. `asyncio.wait_for` cancels the awaiting **coroutine**; the `to_thread` worker is not cancellable and runs to completion, holding a thread and a connection on the **customer's** database after we already answered "timed out". Two fixes. **(a) Dedicated bounded pool** — DB work moves to its own `ThreadPoolExecutor(50, 'dbexec')` via `run_in_executor`; abandoned workers previously accumulated in the shared default pool (`min(32, cpu_count+4)`) alongside every other `to_thread` caller, notably the tabular Parquet loader, so a few slow customer queries could stall unrelated work process-wide. **(b) Session hardening is no longer best-effort** — and a **latent bug** was found while reading it: both SETs shared one `try`, so a `statement_timeout` failure **skipped `default_transaction_read_only` entirely** and the connection served queries in a WRITABLE session behind a `logger.warning`. Now independent: read-only **fails the connection** if it cannot be set (a writable session against a customer DB is not something to degrade into); `statement_timeout` logs at **error** with `degraded_seam` but does not refuse service, since it is their I/O at risk rather than our correctness. ⚠️ **Blast radius to watch:** if any deployment currently fails that SET silently, its sources now fail loudly instead. Believed impossible (Neon accepts it as a SET — the existing comment documents this), but it is the one judgement call here. 7 new tests | | |
| **Reading `eval/readiness/results/` (note for future sessions).** Four files are dated | |
| 2026-07-23. `…_150632.json` scores **4/15 (26.7%)** — that is **not** a product | |
| regression. It is the run that exposed the eval harness itself being broken by | |
| `fd4865b` (`_FakeRecord` had no `results_snapshot` for #34, `_FakeStore` took no | |
| `user_id` for #38; both errors were swallowed by `report_floor`'s fail-closed seam, so | |
| every case reported "not ready"). `…_150859` (15/15) is post-harness-fix, `…_150948` | |
| and `…_152615` (17/17) add the two cases that exercise #34. **The current baseline is | |
| `…_152615.json` — 17/17.** The intermediate files are kept as the audit trail for the | |
| drift; per §7F no result file is ever deleted or overwritten. | |
| **Not re-raised:** F-4 (non-Postgres read-only/timeout gap) was **downgraded to latent** — | |
| Go's `database_clients.Service.Create` enforces `isSupportedActive`, and only `postgres` is | |
| `active`, so no such source can be registered today. | |
| **Decided 2026-07-27 (lead) — F-2 gate unwired, not armed.** The service-secret gate (#37) was | |
| removed from the router mounts because the sole caller is a browser SPA we don't own and can't | |
| change to send the header; arming it would 401 the whole app. The live surface is **unauthenticated | |
| by design** until #43 (Go-forwarded identity). Not a gap to re-raise — it is a recorded posture. | |
| CORS was left at `["*"]` on purpose (tightening it needs the FE origin as config, which we chose | |
| not to set for now). | |
| ## 0.8. Knowledge pipeline — test integration (2026-08-19 → **building, 2026-08-21**) | |
| > ## Updated 2026-09-11 — the open-queue pass (K17–K22) | |
| > | |
| > Six items off `knowledge_pipeline_runs/SESSION_HANDOFF.md` §4, **zero API spend**: every number | |
| > below comes from replaying the six persisted paid runs or re-running the free stages on the three | |
| > v2 artifacts. Detail in REPO_STATUS §14; per-item rows K17–K22 in the table below. | |
| > | |
| > **The headline is Step 2**, the item everyone agreed on and nobody had done: the review queue put | |
| > 24 unresolved key parameters in front of Komatsu's `Engine` (108 mentions, rank 25). It is rank 1 | |
| > now, with nothing dropped. | |
| > | |
| > **Two findings worth reading as findings, not as changes:** | |
| > 1. **The spend ceiling was gating on a partial number.** `estimate_cost` summed only the glossary | |
| > branch under a field named `estimated_prompt_tokens`. Real undercount 8.8%–18.4%; Open Pit is at | |
| > **78%** of the ceiling, not 71%. | |
| > 2. **Escalation has never worked.** 48 calls across three documents and two configurations, | |
| > **0 definitions** recovered. Retired by constant, loop left in place. | |
| > | |
| > **What is deliberately NOT switched on:** genre routing. The classifier is right on all three | |
| > documents, but the label sets are unscoreable without a second gold set and branch-skipping rests | |
| > on a single document, so `ROUTE_LABELS` / `ROUTE_BRANCHES` are both off and the verdict is recorded | |
| > rather than acted on. **Flipping either is Rifqi's call and one line.** | |
| > | |
| > **Still open and unchanged:** the paid Step 0 band (needs authorised repeated runs), Phase 4's | |
| > vision call, Step 8, Step 7, R4, and E5 — Sofhia still has not seen the `src/knowledge_parsing/` | |
| > diff. | |
| > ## Updated 2026-09-11 — v2: clustering fix + parsing 0.4.0 (asset pointers) | |
| > | |
| > Two changes, measured on three documents (BUMA standard · Komatsu shop manual · Open Pit textbook | |
| > Ch.1) at **zero API spend** for the free stages — the parse cache fingerprint is unchanged, so | |
| > re-parses reuse the cached Mistral responses. Full write-up: | |
| > `knowledge_pipeline_runs/CHANGELOG_v2.md`. | |
| > | |
| > **The fuzzy clustering pass is RETIRED (K13).** `token_set_ratio` returns exactly 100 when one token | |
| > set is a subset of the other, so a head noun always absorbed its own compounds and the threshold of | |
| > 92 was **inert**. 131 of its 140 merges were subset matches (all wrong); 6 were plurals, now folded | |
| > deterministically; 3 were distinct metrics wrongly merged. Token-subset merges **15/18/97 → 0**, and | |
| > post-cluster gold recall **0.8293 → 0.8537** — the gold term `EWH`, which `Working Hours` had | |
| > absorbed, is recovered. `_fuzzy_match` is parked, not deleted. | |
| > | |
| > **Parsing `schema_version` 0.4.0 (K14).** A figure is no longer its own chunk. It was, with an EMPTY | |
| > `text`, which made it unreachable by construction — **13 figures across three documents, each | |
| > described by a vision model we pay for, and ZERO reaching any prompt.** Now the `` | |
| > reference is folded into the prose chunk at its reading-order position and the figure travels as an | |
| > `Asset` (content-addressed, persisted bytes, verbatim caption, unwrapped description). Image chunks | |
| > **2/3/8 → 0/0/0**; recall unmoved (E1 and E1c both identical v1→v2). Tables deliberately stay their | |
| > own chunk. `adapter.py` now carries `assets`, `referenced_by` and **`table_html`**, which parsing had | |
| > emitted for months while the adapter dropped it. | |
| > | |
| > **New metric E1c (post-cluster recall)**, and **E2's compression kill line REMOVED** — over-merging | |
| > *raises* compression, so the metric was rewarding the defect it should have exposed. | |
| > | |
| > **Two settings changes measured and REJECTED:** `WINDOW_WORDS` 256 (truncates 46 of 99 Open Pit | |
| > windows — the "flat recall" justifying it was measured on BUMA, the one document whose truncation | |
| > count does not vary with window size) and `SPAN_SCORE_THRESHOLD` 0.15 (+50% clusters against a +20% | |
| > bar). `EVIDENCE_K` raised 3 → 5 on Rifqi's call; measured cost +1.8% to +15% evidence tokens, and on | |
| > the one gold-set document top-5 recovers the same 16/18 as top-3. | |
| > | |
| > **Export for the team:** `knowledge_pipeline_runs/06_export_knowledge_parsing_v0.4.0/`, verified to | |
| > import and validate a fixture with no repo on `sys.path`. ⚠️ `src/knowledge_parsing/` is Sofhia's | |
| > half — she needs the diff before it goes out. | |
| > | |
| > **Paid comparison landed 2026-09-11 (K15).** Definitions 7→10 / 0→0 / 25→49; queue junk rows down | |
| > 37–65%; Komatsu still 0, as predicted. Caveats: the `source_wording` gain is largely tautological, | |
| > and **no definition cited a figure caption on any document** — assets are reachable but unused, which | |
| > points at the caption-to-filter path (built, flag off) as the next free experiment. Open Pit now sits | |
| > at **71% of the 2M spend ceiling** for one chapter. | |
| > **Updated 2026-08-21.** The banner here said *"Nothing has landed in this repo"* — that is no | |
| > longer true. **Both halves are now in-tree** (`src/knowledge_parsing/` + `src/knowledge_extraction/` | |
| > + `eval/knowledge/`, ~5,360 lines across commits `024c30a…a7bdeae`), and three dependencies were | |
| > added. ~~Still true, and the part not to misread: **no HTTP surface, no database table, and nothing | |
| > mounted in `main.py`** — both halves are offline CLIs.~~ **Superseded 2026-09-07: there is now an | |
| > HTTP surface (8 routes) and five tables.** See the 2026-09-07 delta below and REPO_STATUS §14. | |
| > **Updated 2026-09-07 — what this pipeline is for, and what Data Eyond actually needs.** Recorded | |
| > because it changes the definition of done. The review loop, the admin ingestion role and the | |
| > curated-artifact lifecycle belong to a **separate B2B knowledge product**, where a company admin | |
| > configures the company's knowledge and an expert reviews candidates before the company's normal | |
| > users consume them. **This repo is where that concept is being tested cheaply, on real documents.** | |
| > **Data Eyond's own version is one added step:** a user clicks a button and knowledge is extracted | |
| > from **all of that user's documents in one batch** — no admin role, no review queue, no curation UI. | |
| > Full framing: [knowledge_pipeline_context.md](docs/knowledge/knowledge_pipeline_context.md) §1.5/§1.6. | |
| > | |
| > Consequences already folded into the docs: | |
| > - **Multi-document semantics settled** — entries **append, never merge**; `entity_id` stays | |
| > document-scoped; cross-document overlap is *classified* (`new`/`duplicate`/`conflicting`) and | |
| > surfaced, never silently collapsed; `DomainContext` stays per document. A canonical company-level | |
| > concept layer is **deferred to Product B** — it needs an expert to rule on conflicts, and Data | |
| > Eyond's flow has none. | |
| > - **Persistence contract revised to v2** (five tables): Q1 closed — `documents.id` is `text` PK, | |
| > verified by read-only introspection of live dedorch; **`knowledge_jobs` added** (batch envelope, | |
| > and a parse failure has nowhere else to live); and **`scope_id` — company scope — added to every | |
| > table** (Rifqi, 2026-09-07): knowledge belongs to the company, not the uploader, and a per-user key | |
| > means two employees' documents never diff against each other. dedorch has **no tenant entity** | |
| > (15 tables, none of them one) and `users.company` is nullable free text, so the value is provisional | |
| > — normalised `users.company`, else `user:<user_id>` — and **`scope_id` is deliberately excluded from | |
| > the `entity_id` derivation**, which is what keeps re-pointing it a value migration rather than a | |
| > re-key that would destroy expert approvals. Raised to Harry as **Q6**: we want a real `companies` | |
| > table. | |
| > - **N7 closed.** "MinerU cannot install in the Space" predates the Mistral backend; Mistral OCR is | |
| > a hosted API. **Corrected 2026-09-07 — "void" was too strong:** the `python-docx`/`pypdf` conflict | |
| > was making `uv lock` fail for the *whole project*, silently blocking every dependency change. Closed | |
| > by **dropping MinerU** from the extra (`26ba1ab`), not by lifting the pin. **N7b done** in the same | |
| > change: torch pinned to the Linux CPU wheel, 36 `nvidia-*` packages and `triton` gone from the lock. | |
| > GLiNER *is* the term filter, so torch cannot become an optional extra if extraction runs in the Space. | |
| > - **D4 re-opened.** "Admin/service-auth" was right for Product B; Data Eyond's caller is the browser | |
| > SPA, which is the #37 problem. Rate limit + pre-flight spend ceiling + idempotency bound the damage | |
| > but do not identify the caller — Rifqi's call, see the surface plan §6. | |
| > - **Q5 for Harry:** `message_traceability` and `message_charts` exist in dedorch (80 and 6 rows, | |
| > `idx_message_charts_message` present) but were created by hand and are **not in Go's migration | |
| > set** — a fresh environment comes up without them and both write paths fail silently. | |
| Context and intent: [knowledge_pipeline_context.md](docs/knowledge/knowledge_pipeline_context.md). Task breakdown, | |
| owners and verified statuses: [KNOWLEDGE_PIPELINE_TODO.md](docs/knowledge/KNOWLEDGE_PIPELINE_TODO.md). | |
| **Shape.** A six-stage pipeline (parse → free CPU/regex filter → one LLM call per *term cluster* → | |
| deterministic span validation → diff vs. the active glossary → frequency-sorted expert review) that | |
| turns client documents into reviewable knowledge entries, ultimately served over MCP. It splits into | |
| two independently runnable halves — **parsing (Sofhia, Mistral OCR; MinerU dropped 2026-09-07)** and **knowledge extraction (Rifqi)** — meeting only at a **persisted, versioned | |
| parsed-document artifact**. Extraction never | |
| takes a file path; that constraint is the whole point of the split. | |
| **Where it started (audited 2026-08-19).** A working **prototype** (`kex`, ~2,763 LOC, 21 passing | |
| tests, 8-command CLI, four experiments against real Azure calls) exists outside version control at | |
| `Documents/Dataeyond_catalog/knowledge-pipeline/`. It is **not the product and was not ported** — | |
| it answered the design questions; extraction v2 was rebuilt here. What carried forward is the | |
| *evidence* (gold set, experiment results) and the *tuned constants*, not the code. | |
| Measured on the BUMA standard (9 pages): GLiNER Indonesian recall **0.854** (kill line 0.70, PASS — | |
| this closes the context doc's headline open risk), clustering compression **2.56×** (kill 2.0, PASS), | |
| nano schema-fill precision **0.75** (kill 0.80, **FAIL**), span-validation rejection **1.00** with | |
| 0.00 false rejections (kill 0.90, PASS). Cost **$0.0069** for the document. | |
| **Where it stands (2026-08-21).** Both halves run end to end as offline CLIs, on the same document: | |
| | | | | |
| |---|---| | |
| | `python -m src.knowledge_parsing.run --input <folder>` | PDF/DOCX → versioned `ParsedDocument` | | |
| | `python -m src.knowledge_extraction.cli <artifact.json> --extract` | artifact → entries + review queue | | |
| The seam holds from both sides: `ParsedDocument` is the only shape crossing it, extraction takes no | |
| file path, and local seam tests import `knowledge_parsing.contracts` directly so a rename on either | |
| side fails loudly. **Rebuild verified against the frozen baseline: term-filter recall 0.8537, | |
| identical to the prototype** (`eval/knowledge/results/v2_term_filter_2026-08-19_144318.json`). | |
| A live 3-call pilot on `gpt-5.4-nano` returned correct Indonesian definitions with 0 span | |
| rejections and 7,168 of 8,029 prompt tokens served from cache. | |
| **Not done, and not to be read as done:** no persistence (stage output is JSON on disk — D2 still | |
| owes Harry a DDL block) and no HTTP surface. ~~no full-document extraction run, and **E3 has never | |
| been scored on v2**~~ — **RESOLVED 2026-08-27 (K9):** the full-document run is routine now and E3 | |
| scores **1.000** on a committed fixture; see the K9 row for how to read that number. | |
| | # | Task | Owner | Status | Note | | |
| |---|---|---|---|---| | |
| | P1 | Freeze the `kex` prototype (read-only snapshot) | Rifqi | ✅ | **2026-08-19: kept as-is, unversioned — Rifqi's call.** Risk accepted, and much reduced once P2/P3 lifted the two things that could not be recreated | | |
| | P2 | Harvest the prototype's tuned constants | Rifqi | ✅ | **Done** → `KNOWLEDGE_PIPELINE_CALIBRATION.md`, 10 sections: every value with the reason it has that value, plus §7 negative findings and §9/§10 the deviations made while rebuilding | | |
| | P3 | Migrate the evidence into `eval/knowledge/` | Rifqi | ✅ | **Done.** Gold set (41 terms + 15 rules), ported scorer, frozen baseline. Migration proved faithful: the ported scorer re-derives the prototype numbers exactly | | |
| | S1 | Agree the parsed-artifact schema | Sofhia + Rifqi | ✅ | **Settled 2026-08-21.** `src/knowledge_parsing/contracts.py` (`schema_version 0.2.0`; **0.3.0** as of the 2026-08-27 fixture) is the shape of record and the extraction adapter consumes it directly. Envelope, verbatim heading, 0-based `page_idx`, prose `text` with `latex`/`table_html` beside it | | |
| | S5 | Keep the seam off the file path | Sofhia + Rifqi | ✅ | **Satisfied both sides.** `ParsedDocument` carries no handle to the parser, and `knowledge_extraction.adapter` accepts only the artifact. Locked by local seam tests importing the parsing contract directly | | |
| | X* | Build extraction v2 in this repo | Rifqi | ✅ | **Done 2026-08-21** — `src/knowledge_extraction/`: adapter · filters (span/cue/legend) · cluster · rank · extract (4 branches) · validate · diff · queue · service facade · CLI. Recall matches the frozen baseline exactly | | |
| | D2 | Artifact + candidate-entry persistence | Rifqi → Harry | ✅ (Python) / ⬜ (Go) | **Built and live 2026-09-07.** Five tables created in dedorch (`docs/knowledge/knowledge_tables.sql`), ORM models, `knowledge_store.py`, `PostgresJobStore`. Verified end to end: 88 entry rows persisted from the BUMA standard. **⚠️ Harry must still fold the SQL into Go's migration set** — the tables exist only because they were run by hand, the same gap that left `message_traceability`/`message_charts` out | | |
| | D4 | Admin ingest surface | Rifqi | ✅ | **Built 2026-09-07.** 8 routes under `/api/v1/knowledge` (contract §Knowledge): ingest + poll, re-extract, entries, queue, reviews. Async batch (202 + poll, not SSE), company-scoped, rate-limited, spend-gated, single-flight. **Auth posture recorded, not solved:** unauthenticated like the rest of the surface, and `POST …/reviews` accepts content that reaches a later prompt — the strongest case yet for #43 | | |
| | R1 | Fix the literal-source-wording violation | Rifqi | ✅ | **Fixed and live-verified 2026-08-21.** `source_wording` is read from the document's own heading rather than asked of the model (asked directly, it quoted the normalised page-1 form); the span haystack includes headings; the queue flags the mismatch. PA now surfaces as *"source wording differs from the expanded name"* | | |
| | D3 | Provision a mini deployment | Rifqi | ⏸️ | **Deferred 2026-08-19: staying on nano.** The `__54n` quad is wired in `src/config/settings.py`. Accepted knowingly — nano measured 0.75 against a 0.80 line, and `rule`/`summary` run on the tier whose failure mode is least detectable | | |
| | D6 | Expert review of the gold set | Rifqi → Mas Beta | ⏸️ | **Deferred: provisional gold is good enough for now.** Standing caveat on every score, E1 and E3 both move when it is reviewed | | |
| | D5 | v2 module name | Rifqi | ✅ | **`src/knowledge_extraction/`**, beside Sofhia's `src/knowledge_parsing/`. Neither collides with the existing `src/knowledge/` OCR→pgvector service, which is untouched | | |
| | D1 | New dependencies | Rifqi | ✅ | **Added 2026-08-21.** `rapidfuzz`, `gliner`, `torch>=2.6` as main deps; **`mineru==3.4.4` as an optional extra** (`.[knowledge-parsing]`) so the deployed Space never builds it. **Updated 2026-09-07 (`26ba1ab`): `mineru` dropped** — it made `uv lock` fail project-wide; `torch` pinned to the Linux CPU wheel. The §6 gliner/torch question is now RESOLVED | | |
| | K9 | Full-document extraction run + score E3 on v2 | Rifqi | ✅ | **Done — latest run 2026-08-27** (`eval/knowledge/results/v2_full_2026-08-27_133757.json`, 90 calls, 170 s). E1 **0.8293** / E2 **2.556×** / E3 **1.000** — all three above their kill lines; rule recall 0.4286 (6/14, tracked not gated). **The prototype's failing experiment now passes**, but read E3 as *precision on a scoreable base of 6*: 57 of 63 entries abstain, so recall against the 18 gold terms with criteria is 0.33. 0 fields rejected by the span check | | |
| | K10 | Fixture artifact committed | Rifqi | ✅ | **Done 2026-08-27** — `eval/knowledge/fixtures/parsed_STD_2026_006_MNO.json` (BUMA standard, MinerU 3.4.4 `hybrid`, `schema_version 0.3.0`, 31 chunks). The eval and the extraction CLI both run with MinerU absent. The Open Pit textbook fixture is still open (S7 in the checklist) | | |
| | K11 | Output schema v2 (Harry's contract review) | Rifqi | 🔎 | **Landed 2026-08-26, unevaluated.** Deterministic entity ids, glossary references formulas instead of restating them, rules link to formulas and terms, 16 fields cut (50 → 42). Contract: [docs/knowledge/KNOWLEDGE_OUTPUT_CONTRACT.md](docs/knowledge/KNOWLEDGE_OUTPUT_CONTRACT.md); detail in the checklist as X24. Fixed two live defects — neither `cluster_id` (a frequency ordinal) nor `rule_id` (model-written) survived a re-run, which breaks an expert's approvals, not just links. Suite 497 passed / 7 skipped; **no eval run** — the prompt edits join X20's in the unevaluated queue | | |
| | K12 | Output schema v3 — `DomainContext` | Rifqi | ✅ | **Done 2026-09-02** (`7f47c61` · `6f5d619` · `e2865ac` · `172d8dc`; checklist **X26**). `BriefContext` → **`DomainContext`**, renamed while nothing is persisted — afterwards an id-prefix change is a migration that loses an expert's approvals. **The branch is no longer the pipeline's one unverifiable output:** it was made guardable by changing the question rather than the model — the prompt asks it to *locate* the document's own statement of purpose (`purpose` → `purpose_verbatim`) instead of composing a summary, so `GUARDED_FIELDS["summary"]` goes `()` → `("title", "purpose_verbatim")`. Adds derived `outline`, `subdomains` and entry counts; `key_parameters` gains a corroborating `term_id`, and an unresolved one earns its own review-queue row. **Contract change for Harry:** queue rows now carry `row_kind`, and a `key_parameter` row has no `term_id`, definition or mention count. Measured on the BUMA fixture (nano, 12 calls): 0 fields rejected, fabricated purpose refused, key-parameter linking 0/5 → 4/5 after a bug the smoke caught. Suite 541 passed / 7 skipped | | |
| | K13 | **Clustering: retire the fuzzy pass** | Rifqi | ✅ | **Done 2026-09-10.** `token_set_ratio` is blind to the subset case (returns 100), so the threshold never applied and head nouns ate their compounds. Measured across three documents: 131 of 140 merges were subset matches. Now `exact (plural-folded) → abbreviation ↔ expansion` only; `depluralise()` handles the 6 legitimate plural merges. Subset merges **15/18/97 → 0**; E1c_strict **0.8293 → 0.8537** (`EWH` recovered). New metric **E1c** added to `score.py`/`run_eval.py`; **E2's kill line removed** (it rises when the defect fires). Suite 676 passed / 7 skipped | | |
| | K14 | **Parsing 0.4.0 — asset pointers** | Rifqi (Sofhia's half) | ✅ | **Done 2026-09-10.** Figures fold into the prose chunk as `` at reading-order position; `Asset` carries a content-addressed id, persisted bytes, a **verbatim** caption and a **model-written** description that never enters `text`. Image chunks **2/3/8 → 0/0/0**, 0 dangling refs, 0 description leaks, captions 17/17 preserved, **recall unchanged**. Tables keep their own chunk + two-way pointers. Seam: `adapter.py` carries `assets`/`referenced_by`/**`table_html`**; `SUPPORTED_SCHEMA` → (0,4), so parsing and extraction ship together. Found and fixed a Mistral bug: `image_id` mislabelled on one block, collapsing two figures onto one hash. ⚠️ **Sofhia needs the diff before the team export goes out** | | |
| | K15 | Paid v1-vs-v2 comparison | Rifqi | ✅ | **Done 2026-09-11**, 3 runs, ~9¢. Definitions **7→10 / 0→0 / 25→49** — Open Pit nearly doubled; junk review-queue rows −37% (Komatsu) / −65% (Open Pit); cache rate +7% on all three; 0 retries. **Komatsu stays at 0 definitions**, as predicted: a troubleshooting manual states symptoms, not definitions, and no plumbing change reaches that. Two caveats recorded rather than buried: `source_wording`'s headline +1608% is mostly **tautological** (informative 12→101, not 13→222), and **`definitions_citing_a_caption` = 0 on every document** — the assets are reachable but nothing consumed them, most likely because captions do not feed the term filter (flag off by decision). ⚠️ Cost: Open Pit is now **71% of the 2M pre-flight ceiling** (was 56%) for one chapter. Three variables moved at once, so no single number is attributable. Write-up: `knowledge_pipeline_runs/V1_VS_V2_PAID_COMPARISON.md` | | |
| | K16 | Caption hypothesis for the 0 caption-citations | Rifqi | ✅ | **Tested 2026-09-11, HYPOTHESIS FALSIFIED.** Both candidate explanations for `definitions_citing_a_caption = 0` are false: captions ALREADY reach the term filter (they are part of `Chunk.text`; 2/2 BUMA and 3/8 Open Pit captions already produce mentions), and figure-bearing chunks are ALREADY selected (a captioned chunk is in the top-5 for 22/83 BUMA and 64/427 Open Pit clusters). Built the missing ranker signal `asset_caption_names_term` to test it properly: at weight 2.5 it changed the top-5 of **1 cluster out of 603** and moved none from no-figure to has-figure. **The real reason: not one caption in the corpus is a definition** — they are figure titles ("Figure 1.3. Relative ability to influence costs"). A model declining to quote one is CORRECT. Signal ships at **weight 0.0** with the measurement beside it. ⚠️ Also corrects a false claim in the earlier docs: the caption-to-filter flag was described as "built, off by decision" — it was never built. Redirects the remaining figure value to Phase 4 (vision call on the bytes), which is now the only untested path | | |
| | K17 | **Step 2 — review queue ordering + key-parameter heads** | Rifqi | ✅ | **Done 2026-09-11.** Unresolved key parameters held their own sort tier, so every one outranked every routine term; the branch emits them in bulk on the documents it understands least, so the promotion fired hardest where least deserved. Now spliced at most **1 per 5** term rows. Replayed on the three persisted v2 runs: Komatsu `Engine` (108 mentions) **rank 25 → 1**, Open Pit `Resource` **8 → 1**, first junk row **1 → 6**, **row counts identical** (nothing dropped), BUMA unchanged (0 unresolved). `term` now carries a head (first `.` `:` or newline, only past 60 chars — "No. of units" survives) with the full text in new **`surface_full`**. Contract: OUTPUT_CONTRACT §9 | | |
| | K18 | **Step 0 — measurement hardening** | Rifqi | 🔄 | **Harness done 2026-09-11, the paid band is NOT run.** `eval/knowledge/invariants.py`: asserts on **fields and invariants, never exact text** — span locatable, entity ids present and unique, a rejected field still null (rejected-never-repaired, checkable only from outside), page in range, queue ranks contiguous, no leaked sort keys. `--runs N` repeats **only the paid stage** (free stages are deterministic) and reports min/median/max with the **median** as the value, plus whether all runs minted identical entity ids — the K11 defect class, unwatched since K11. `EXPECTED_MODEL_SNAPSHOT` turns X25's recording into an **assertion** (exit 6). New exit codes: 5 invariant, 6 snapshot. **Needs Rifqi's go-ahead to produce a real band** | | |
| | K19 | **Step 6 — spend ceiling in cluster units** | Rifqi | ✅ | **Done 2026-09-11.** Found first: `estimate_cost` summed **only the glossary branch** under a field named `estimated_prompt_tokens`, so the ceiling read a partial number as a total — undercount **BUMA 18.4% · Komatsu 8.8% · Open Pit 15.2%**, moving Open Pit 67.7% → **78.0%** of the 2M ceiling. Now all four branches (glossary exact, the rest approximated from prefix + chunk text, both labelled). Second ceiling **`MAX_CLUSTERS = 750`** — exact, free, and what drives spend; at ~3,200 tokens/cluster the token ceiling still binds first for these documents, so this guards the *other* shape (many cheap clusters, 494 calls ≈ 15 min). Refusals name the ceiling and the declined call count; `IngestJob.estimated_cost` (declared since the job model, **never populated**) now carries the per-document breakdown; a wholly-refused batch settles **`blocked`**, which the contract documented and the code could not produce | | |
| | K20 | **Step 5 — genre classification** | Rifqi | 🔄 | **Classifier done and correct on all three documents; routing deliberately OFF.** `src/knowledge_extraction/genre.py` → `standard` · `diagnostic` · `conceptual` · `unknown` from four free signals (heading coverage · tabular ratio · abbrev pairs · rule density), which separate in *different directions* — measured BUMA 46%/29%/5/0.57, Komatsu 1%/74%/0/0.01, Open Pit 86%/7%/0/1.48. **`ROUTE_LABELS` and `ROUTE_BRANCHES` are both `False`**: the genre label sets are unscoreable without a second gold set (more mentions ≠ better recall) and branch-skipping rests on one document. Today it is observability — `well_served=False` warns *before* spend that a shop manual yields almost nothing, and heading coverage raises a per-document warning (**P5**, partly: the warning exists, a parsing-side fallback does not). **Flipping either lever is Rifqi's call** | | |
| | K21 | **Finding 6 — escalation retired** | Rifqi | ✅ | **Done 2026-09-11, with the number Finding 6 asked for.** Across all six persisted paid runs (v1 K=3 and v2 K=5 × three documents): **48 escalation calls, 0 definitions recovered** — `extraction_status == "escalated"` appears zero times in every file. Why it cannot work as built: escalation hands the model the *next* k evidence chunks, ranked **below** ones it already declined, and 16 of 18 gold definitions sit at rank ≤3. `MAX_ESCALATION_ROUNDS = 0`; loop and call site untouched. Reviving it means changing the **question** (different prompt or tier), not the evidence | | |
| | K22 | **The over-cap window (pre-existing)** | Rifqi | ✅ | **Done 2026-09-11.** `WINDOW_WORDS` caps words; GLiNER truncates on **subwords** and warns rather than failing. Measured with GLiNER's own tokeniser: **1 window of 368** exceeds the 384-token cap — BUMA `STD_2026_006_MNO::0024`, 130 words / 738 chars → **409 tokens** — losing its tail at *every* window size including 130, exactly as the 2026-09-10 sweep suspected and left. Windows are now checked against the real tokeniser and split only when they do not fit (a character budget would have to assume the tightest ratio seen, 1.57, and chop up the 367 that fit). Halves overlap by `SPAN_TOKEN_CAP` words so no term straddles a cut. Fixture: 3 splits, mentions **186 → 208**, clusters 74 → 80 (**+8% spend**), **E1 0.8293 and E1c unchanged** | | |
| | K23 | Step 8 + Step 7 design proposals | Rifqi | 💡 | **Written 2026-09-11, nothing built, both need sign-off.** [TABLE_BRANCH_PROPOSAL.md](docs/knowledge/TABLE_BRANCH_PROPOSAL.md) — deterministic (row header, column header, cell mark) → assertion, one call per **table** (59 on Komatsu, fewer than the 93 the glossary branch spends to abstain 93 times), provenance at the cell; five decisions listed, **T4 (what `table_html` shapes the parser emits) is free and comes first**. [TERM_HIERARCHY_PROPOSAL.md](docs/knowledge/TERM_HIERARCHY_PROPOSAL.md) — parent/child **with** cross-document identity because SciCo's point is they are one problem; the dominating constraint is that **hierarchy must not enter `entity_id`**, or every correction invalidates expert approvals. Sequenced behind **R4** (still not asked) and Step 0 | | |
| | K24 | **The 2026-09-08 discussion list — structure, incremental ingest, review** | Rifqi | ✅ | **Done 2026-09-14 (pr/21, 7 commits).** *Struktur data 7/8:* `label` carries a readable `TYPE_SUBJECT` key (`keys.py`) — prefixes from the `kind` vocabulary, rules borrow the term they govern plus a digest since a rule has no canonical surface, and the key is a **label, never an identity** (`entity_id` stays the hash approvals hang on). `content`/`provenance`/`doc_ids` promoted out of `payload`; `get_by_key()` + `glossary_index()` make lookup O(1) with no model call. **m7 parked** — the `kind` CHECK would reject `'summary'`. *Ingest bertahap 7/7:* same `doc_id` + new `content_hash` = a new VERSION (reuse check moved to any-version, or idempotency would become a paid no-op); an approved term skips the paid call; staleness compares **only the ruled-on fields** (`mutation.py`) because comparing the payload re-opens everything on every ingest and comparing document `content_hash` flips untouched entries; `impacted_by_document()`; withdrawal marks and keeps rather than deletes. *Sistem review 4/4:* `review_mode` at ingest, `decided_by` + `authorised_by` so an LLM-mode approval can say whose decision it was, and `review_diff()` naming which fields moved. **7 columns added to dedorch** (§6/§7 of `knowledge_tables.sql`, run by hand **after** the handoff went to Harry, verified read-only against live — Harry still needs both in the migration set). Suite **816 passed / 7 skipped** (was 742), ruff clean on touched paths, `import main` 0. **Inert until the review queue has rows:** reuse and staleness both key on approved entries, and `knowledge_reviews` is at 0 | | |
| **Not decided, do not assume:** whether the admin ingest surface is HTTP at all (the offline CLI is | |
| the honest milestone and both halves are there now); and GLiNER + LangExtract ensembling stays ⏸️ | |
| flagged as scope creep. **`kex` was not ported** — it answered the design questions; v2 was built | |
| here against this repo's patterns. | |
| ## 5. Critical path & sequencing | |
| - **Critical path:** ~~#22 (send Harry the `report_inputs` schema)~~ **✅ resolved 2026-07-22** — now **#32** (`message_traceability` + `message_charts` DDL to Harry) and **#31** (non-convergent migration set). HF deploy (#13) for the playground. (#4 ✅, #21 ✅; Harry's #3 no longer blocks us — Python is getattr-tolerant.) | |
| - **Parallelizable now:** #31 + #32 (both are Harry handoffs). (#4 ✅, #11 ✅, #22 ✅ done.) | |
| - **Harry-blocked / coordinated:** #3 (now 🔄, blocks #4), #7 (Go endpoint), #18 (Go state ownership), #24 (contract). **#25 = chat-path migration to `analyses_messages` — a cutover blocker.** | |
| - **Demo gate (playground, #13):** deploy the remove-`problem_statement` work to HF — slow path (#15 ✅) | |
| and the report path are verified locally, and #16 is resolved (#22 hands Harry the schema). **Keep it | |
| playground-able.** | |
| ## 6. Decisions still open (need the team / Harry / lead) | |
| - ~~`analysis_records`: dedorch-owned vs Python-owned (#16/#14).~~ RESOLVED: Python-owned + renamed **`report_inputs`** (#21 done); Harry's migration creates it (#22). | |
| - ~~Whether `help` is its own endpoint or via `call_agent` (#8).~~ RESOLVED: `help` is a `call_agent` intent (no own endpoint). | |
| - ~~Author display-name source for the report (#19).~~ RESOLVED: Python resolves `users.fullname` (fallback `user_id`); swap to a Go-passed name later if preferred. | |
| - ~~Keep vs drop `chat_history` in the report contract (#17).~~ RESOLVED: never in the contract; report is records-based (analysis_id + user_id only). | |
| - Confirm Go takes over analysis creation + both creation gates (data-first + mandatory fields) (#18). | |
| - **Report input for edit mode** — does Python need the last report content? (#24) | |
| - ~~`report_inputs` write scope — every agent call vs slow-path-only? (#21)~~ RESOLVED: one row per slow-path run (telemetry stays Langfuse). | |
| - **Python history source** — confirm Go's `analysis_message` (#25). | |
| - **`done.chart_count`** — additive field on the SSE `done` event so the FE can skip `GET /charts` | |
| on chartless turns? (Harry; SPINE_V2_PLAN §4.5.) Until decided the FE fetches unconditionally. | |
| - **W3 re-open timing** (deferred `analyze_*` activation) — Rifqi (deferred at the 2026-07-13 approval). | |
| - **INV-6 relaxation for S1b targeted repair** — team, only after S1a `repair_candidate` telemetry | |
| shows a meaningful hit-rate (SPINE_V2_PLAN §6). | |
| - ~~**Knowledge pipeline — v2 module name** (§0.8 D5).~~ **RESOLVED 2026-08-21:** | |
| `src/knowledge_extraction/` and `src/knowledge_parsing/`; the existing `src/knowledge/` | |
| OCR→pgvector service is untouched. | |
| - ~~**Knowledge pipeline — should `gliner`/`torch` be an optional extra too?** (§0.8 D1)~~ | |
| **RESOLVED 2026-09-07 (`26ba1ab`): no — they cannot be, and the size problem was solved another | |
| way.** GLiNER *is* the term filter (`use_span_filter=False` yields zero mentions), so extraction in | |
| the Space requires torch **installed**; a lazily-imported extra that is absent is a 500, not a | |
| graceful skip. What was actually costly was not torch but its CUDA tail: the default PyPI Linux | |
| wheel drags the whole runtime as `nvidia-*` dependencies. Pinning the `+cpu` build from PyTorch's | |
| own index removed **36 `nvidia-*` packages and `triton`** from `uv.lock` with zero behavioural | |
| change. `mineru` — the comparison this question was built on — was **dropped** in the same change, | |
| because it made `uv lock` fail project-wide. | |
| - **Knowledge pipeline — admin ingest surface** (§0.8 D4): whether it is HTTP at all, and if so its | |
| auth posture. A write surface triggered by an admin is a different risk class from the current | |
| unauthenticated read-only chat surface (§0.7) — raise it, do not inherit the posture by default. | |