DEV_PLAN.md

Data Eyond — Current Development Plan (post 2026-06-24 → 2026-06-30 checkpoints)

✅ LIVE — CANONICAL: what is in flight. Sprint plan, task table and locked decisions (§1/§1.5 are locked; §6 lists what is still open). Built state: REPO_STATUS.md · all other docs: docs/.

Purpose: context file for Claude Code sessions working on the current sprint. Branch: pr/5 · Snapshot: 2026-06-30. Companion: REPO_STATUS.md describes the repo's current built state; this file describes the in-flight plan that changes it. The active sprint is pr/5 (§0); sections §1–§6 are the prior 2026-06-24/25 pivot (now largely ✅), kept for context.

---

0. Current sprint — pr/5: Observability + Endpoint Restructure

From the 2026-06-30 checkpoint. Direction: Python → generation/AI-only; Go owns the analysis lifecycle + data plane. Endpoint contract sent to Harry on 2026-06-30: API_ENDPOINTS_RESTRUCTURE.md (chat→v2, tools regroup, observability — observability marked tentative). REPO_STATUS carries a matching pr/5 direction banner.

Mentor's task order: unwire → regroup endpoints → add tools (retrieve-data + observability); share the endpoint contract before coding the tools. Status legend: ⬜ not started · 🔄 in progress · ✅ done · ⛔ blocked · 🔎 verify · ⏸️ deferred.

Phase	Task	Owner	Status	Notes
P0 — contract	Draft + send endpoint contract to Harry (chat v2 · tools group · observability)	Rifqi + Sofhia	✅	API_ENDPOINTS_RESTRUCTURE.md sent 2026-06-30 (before-noon deadline met). Observability section flagged tentative.
1 — unwire	Unwire users(login)/document/room/db_client/data_catalog/analysis from main + Swagger	Sofhia	✅	KM-686, commit 0b2d678. Commented, not deleted; chat/report/tools kept mounted. Resolves the analysis-CRUD scope Q — whole analysis router unwired (Go owns it).
2 — v2 + regroup	Create src/api/v2/ and move the chat pilot there	Rifqi	✅	New src/api/v2/__init__.py + src/api/v2/chat.py (POST /api/v2/chat/stream), mounted in main.py. Only chat in v2; v1 /chat/stream kept mounted until FE moves over. Routes import-verified.
2 — v2 + regroup	Chat: room_id → analysis_id (request field + handler + history)	Rifqi	✅	v2 ChatRequest{user_id, analysis_id, message}; reuses warm ChatHandler + v1 cache/history helpers; done returns {message_id} (always minted Python-side, server-authoritative — open-Q #1 resolved in pr/6). Persistence kept transitionally → still ties to #25 (analyses_messages); ruff-clean.
2 — v2 + regroup	Move report under tools → /api/v1/tools/report (+ version routes)	Rifqi	✅	report router re-prefixed /api/v1 → /api/v1/tools (all 3 routes move together), tag → Tools; old /api/v1/report gone. Same functionality, new home. Import-verified.
2 — v2 + regroup	Move help under tools → POST /api/v1/tools/help (dedicated endpoint)	Sofhia	✅	New src/api/v1/help.py (SSE: sources:[]→chunk→done{message_id}) + additive ChatHandler.stream_help() (reuses HelpAgent+state+readiness, no router). Generative-only (no persist). Router help intent KEPT — both paths live by design. message_id always minted Python-side, server-authoritative (open-Q #1 resolved, pr/6). Import-verified.
2 — v2 + regroup	Tools list → /api/v1/tools/list	Sofhia	✅	Renamed route GET /api/v1/tools → GET /api/v1/tools/list (tools.py:133).
2 — v2 + regroup	FE: slash menu = /help only; report = right-side button	Mentor (FE)	⬜	Coordination note, not Python work.
3 — tools + obs	Finish help so it actually calls (not just lists) + test	Sofhia	⬜	Mentor: help currently only lists tools. Core #2 after chat.
3 — tools + obs	Traceability scratchpad accumulating in the chat agent	Rifqi + Sofhia	✅	KM-691. TraceabilityScratchpad + TraceabilityToolInvoker (src/traceability/) capture planning / tool I/O / sources during the run; flushed one row before every done (all 8 sites; error turns = no row). Renamed observability→traceability (vs. Langfuse).
3 — tools + obs	Audit report_inputs — covers planning + tool I/O + source? add cols / new store	Rifqi	✅	KM-691. Chose a dedicated store: message_traceability = 1 JSONB row per message (Python-owned, like report_inputs; DDL run manually against dedorch, handed to Harry). Langfuse kept for engineering.
3 — tools + obs	Build GET /api/v1/traceability (one merged response)	Rifqi	✅	KM-691. src/api/v1/traceability.py → store.get → payload/404. Intent-based source rules (greeting/help/refusals = none; retrieve = required); full planning only on slow path. Contract §7 updated.
3 — tools + obs	Keep stream text-only; traceability is a separate parallel call	Rifqi	✅	KM-691. No trace data in the SSE stream; the FE fetches /traceability on done.
3 — tools + obs	Resolve message_id correlation (stream ↔ traceability) with Harry	Rifqi ↔ Harry	✅	RESOLVED (pr/6): Python is the sole minter — message_id dropped from the /api/v2/chat/stream + /api/v1/tools/help request bodies; always minted server-side (server-authoritative, FE-security) and returned on done. Any caller-sent message_id is ignored. Contract open-Q #1 closed. Updated 2026-07-09 (pr/13): id format changed from msg_<hex> to a canonical UUID string (str(uuid.uuid4()), both mint sites) to mirror Go's analyses_messages.id shape — the value is still independently Python-minted (not the real row id), only format-compatible for a future swap.
4 — biz questions	Get Go folder; confirm business_questions in create-analysis (max 5); sync Python	Harry/Mentor → Rifqi	⬜	Go currently missing the field ("lagi difixing"). Python already models objective + business_questions.
deferred	Report formats: PPT (preferred) / PDF / infographic on download	—	⏸️	MD is fine for the FE preview stage now.
deferred	Charts (Plotly→JSON) + images tables	—	⏸️	Carried from §4 #26/#27.

Next up: Phase 2 Python work is done (chat→v2 analysis_id; help/report/list regrouped under /api/v1/tools/). The message_id correlation contract is now settled in pr/6 (Python sole minter, stream-only). The Phase 3 traceability build — scratchpad + GET /api/v1/traceability (contract §7) — is done (KM-691). Next is Phase 4 (business questions, Go-blocked).

---

0.5. pr/13 sprint — agent-quality fixes (2026-07-08 live-test review)

Findings from the scoped live sessions (mining analysis, 2026-07-07/08 traces): the planner force-mapped absent measures (pa aliased as "revenue"), top-N ranked raw rows (duplicate models), analyze_trend collapsed integer months into a single 1970-01 bucket, an invalid grouped IR reached Postgres, failed retrievals wrote all-null traceability sources, and numeric catalog samples arrive base64-mangled from Go. Fix tasks (same status legend as §0):

#	Task	Owner	Status	Note
Q1	IR validator: reject bare selects under group_by (planner retry self-corrects)	Rifqi	✅	query/ir/validator.py
Q2	Planner infeasible path: TaskList.infeasible_reason + deterministic EN/ID data-gap reply	Rifqi	✅	schemas/validator/coordinator/refusals + planner.md "When the catalog cannot answer"; refusal wording → Rifqi to review
Q3	analyze_trend: integer year/month handling (epoch-parse bug)	Rifqi	✅	temporal.py + 5 local tests
Q4	Planner few-shots: top-N (Example G) + infeasible (Example H) + entity-vs-row ranking rule	Rifqi	✅	live-tested 2026-07-08: backlog top-3 correct via single-IR group+sum; "best PA performance" correct in-process (avg-per-model, assumption recorded). Stale-server trace was a false alarm
Q5	Catalog numeric sample_values base64-decode stopgap (catalog/sample_decode.py)	Rifqi	✅	self-disabling; primary fix = Go marshaling — DDL-free handoff to Harry
Q6	Traceability null-source suppression + check_data -1 row-count hiding	Rifqi	✅	scratchpad.py / data_access.py
Q7	analyze_merge two-table combine tool (unblocks "worst A + biggest B" questions)	tool owner	✅	tool shipped by Sofia (8abf635, KM-703); planner slice done 2026-07-09: _validate_data_source guards data_right, two-retrieve→merge few-shot (Example I), planner.md "Two measures per entity" bullet
Q8	Report v2: business-question answer section, unresolved/excluded sections, evidence tables from results_snapshot, caveat dedupe, single language	Rifqi/Sofhia	✅	done 2026-07-09: still exactly ONE LLM call (extended to also draft bq_answers, index-based record refs, deterministic fallback = v1 behavior); evidence tables from table-kind outputs (≤3/record, ≤10 rows, ≤8 cols, check_* skipped); reply language via detect_reply_language on objective+BQs; verified in-process against live analysis 935a091e
Q9	Record-curation endpoint (GET …/records + exclude_record_ids) + readiness GET for the FE delta guard	Rifqi ↔ FE	✅	done 2026-07-09: GET /tools/report/{analysis_id}/records + /readiness (registered before /{version} — int-coercion route-order trap), exclude_record_ids on POST; contract updated same change; FE wiring pending (Rifqi → FE)
Q10	Traceability data_used layer — resolve IR ids → real names for the FE (users couldn't map c_…/t_… ids back to their data; aggregate aliases like total_revenue looked like real columns)	Rifqi ↔ FE	✅	done 2026-07-13 (pr/15): new src/traceability/resolve.py builds data_used[] (real source/table/column names; joins; plain-language filters; columns_read vs output_columns with computed+formula), tool_calls[].summary, sources[] gains source_name+all tables; ids kept but machine-only (FE must not render); deterministic no-LLM, never-throw; catalog threaded to the scratchpad at the composition root. Contract + TRACEABILITY_FE_HANDOFF.md updated; FE wiring pending (Rifqi → FE). Additive/non-breaking
Q11	IR wart: OrderByClause.column_id may hold a SELECT alias (a computed output), not a catalog column_id	Rifqi	🔎	surfaced by Q10 — the resolver tolerates it (kind: "computed" fallback), but the IR field name is misleading. Consider an explicit by_alias field or renaming. Low priority; no functional bug

0.6. pr/16 sprint — Spine v2: W2 charts + W1 checkpoint (SPINE_V2_PLAN, approved 2026-07-13)

Scope approved 2026-07-13 (Rifqi + Sofia): build W2 (render_chart + chart store + GET /charts

• planner viz slice) and W1 (S1a quality checkpoint). W3 (activate deferred analyze_*) deferred at approval — do not start until Rifqi re-opens. W4 (S1b repair) stays gated on INV-6 sign-off + S1a telemetry. Design + handoff source: SPINE_V2_PLAN.md.

#	Task	Owner	Status	Note
V1	render_chart tool slice: visualization.py (Plotly-JSON dataeyond.chart.v1 envelope, no plotly dep) + ToolOutput.kind "chart" + registry + invoker	Rifqi (Sofia signed off on the tool-layer edit)	✅	done 2026-07-13; deterministic spec builder (bar/line/pie/scatter, fixed style preset), traceability scratchpad summarizes chart outputs compactly (point_count, not the raw arrays)
V2	Chart store + API: MessageChartRow (message_charts) + src/charts/store.py (never-throw save) + write site in _run_slow_path + GET /api/v1/charts + contract §charts	Rifqi	✅	done 2026-07-13; empty list = valid 200; done event unchanged (no chart_count — open, Harry); FE fetches unconditionally on done
V3	Planner viz slice: recipe table + "Charts only on explicit ask" rule (planner.md), Example J (viz tail) + Example K (viz-infeasible), validator Check 10 (render_chart.data must be table-kind), assembler chart one-liner guard	Rifqi	✅	done 2026-07-13; behavioral matrix verified in-process (real LLM): explicit ask EN/ID → chart tail; plain question → no chart; absent dimension → infeasible (Example K + prompt guard added after the first smoke force-mapped status AS "region")
V4	W1 S1a quality checkpoint: slow_path/checkpoint.py (CK1–CK6) + RunAssessment schemas + coordinator call site + "Execution assessment" block in the assembler input + refusals.run_failure_message	Rifqi	✅	done 2026-07-13; 13 local tests (one per CK rule + never-throw + prompt + coordinator); CK1 all-failed → deterministic honest failure, no assembler call; clean run renders nothing (zero behavior change); every flag logs repair_candidate (S1b evidence)
V5	message_charts DDL: run manually against dedorch (block for the live e2e chart test), then hand the schema to Harry for the dedorch migration	Rifqi → Harry	✅	Rifqi ran the DDL 2026-07-13; live e2e ALL PASS same day (real v2 endpoint: viz turn → chart row keyed by done message_id → GET /charts valid v1 envelope; chartless → 200 empty; injected render_chart failure → answer streams as a table, no row). Remaining: send Harry the migration handoff (schema + contract §charts)
V6	Restore the eval.chat_sim harness — eval/chat_sim/*.py is missing from disk AND git (only __pycache__ remains); §7B prompt gate can't run	Rifqi	✅	restored by Rifqi 2026-07-13 (accidental delete). ⚠️ its hard-coded DEFAULT_USER_ID/TITANIC_SOURCE_ID are stale (that user has no catalog; the Titanic blobs are gone) — update the constants before the next full run
V7	Local .env lagged Go's Supabase-S3 data plane: storage_provider=azure_blob + empty supabase_s3_* made EVERY local tabular retrieve fail BlobNotFound	Rifqi	✅	found during the e2e (masked as "data not available" by the honest-degrade path); Rifqi set the six values 2026-07-13. Gotcha documented in REPO_STATUS §13
V8	Lead review of GET /charts: lookup by message_id alone + tri-state response marker (status: success | empty | not_found + message) instead of a bare list	Rifqi (lead ask)	✅	done 2026-07-14; not_found vs empty decided against the turn's traceability row (PK lookup); always HTTP 200; contract §charts updated. ⚠️ additive DDL for the new lookup: CREATE INDEX IF NOT EXISTS idx_message_charts_message ON message_charts (message_id); (run manually + include in Harry's migration). Traceability GET still takes both params — aligning it is open
V9	Report chart embedding: AnalysisReport.charts (verbatim envelopes per record) + ## EDA section with ```plotly fences (content = the **full v1 envelope**, pretty-printed — the FE hook's verified shape); has_successful_analysis extended so a successful render_chart counts (chart-only sessions satisfy the report floor)	Rifqi	✅	done 2026-07-14; first cut emitted bare {data, layout} — FE test showed the hook parses the full envelope, fixed same day; live reports v3 (wrong fence) → v4 (correct) for analysis 7be50846… (3 charts embedded); suite 381 passed, 7 skipped (+5 chart-embed tests; same 2 pre-existing failures)

Full-suite evidence for this sprint: 376 passed, 7 skipped (+13 new checkpoint tests; the 2 failures — test_chat_handler::test_structured_flow_runs_slow_path, test_reader::test_structured_read_falls_back_to_user_scope_when_no_analysis_row — reproduce at HEAD before this diff, i.e. pre-existing). Ruff clean on all touched paths; import main OK.

1. The direction change (locked decisions from 2026-06-24)

1. "Problem statement" is replaced by two user-entered fields: objective + business_questions. User fills them at onboarding; both mandatory to submit; NO agent validation.
2. The gate (problem_validated) and the problem_statement skill/intent are removed (comment out, don't delete).
3. Report is records-based (reads persisted AnalysisRecords) — decided and pushed (KM-674). It is formal markdown: title, date, "generated by {user}", objective, business questions, findings, insights. NOT gated on whether business questions were answered.
4. owner_id → user_id everywhere (Harry mirrors in dedorch/Go).
5. State writes go through a request to Go, not direct Python DB writes.
6. FE-callable surface = 4 endpoints: call_agent (chat/stream), list_skills (GET /tools), skill: help, skill: report. problem_statement removed; check_data not FE-facing from Python (Go provides it); analysis CRUD not needed from FE (comment, don't delete).
7. Deliverables for Harry: (a) API endpoint doc (MD); (b) full Python project doc (MD → PDF/Word BRD).
8. Integration tested via Swagger /docs on the HF Python build (simulating FE manually). Target ~Wed.

1.5. 2026-06-25 checkpoint deltas

Confirms the 2026-06-24 direction and adds these concrete changes (folded into §4 as tasks 21–28):

1. Rename analysis_records → report_inputs (DONE #21) — names the table by purpose (the rows report generation reads); avoids clashing with Go's analyses_messages and with Langfuse observability. Stays Python-owned; finalized schema handed to Harry so his dedorch migration creates it post-SKIP_INIT_DB (#22, resolves #16). Write scope = one row per slow-path analysis run (decided — not per-agent-call telemetry; that stays Langfuse).
2. analyses table (Go) — status, data_bind + data_bind_version, report_collection (id+version). Verified 2026-06-25: these + user_id are ALREADY present in dedorch analyses. Plus Harry drops the duplicate/wrong singular analysis table. (→ #3)
3. analyses_messages (Go) = the analysis chat room (user Q + agent A) — replaces the now-deprecated chat_messages/rooms; Python's chat read/write must migrate here before cutover. (→ #25)
4. Reports: Go owns ALL writes. Report stays a skill (no router intent): FE → Go → Python; Python only returns content. Input = the records table (now agent_observability); edit-mode may also need the last report. (→ #7/#18/#24)
5. Markdown minimum now: tables, bold, italic, horizontal separators — optimize that before anything fancier. (→ #23)
6. Deferred: charts (prefer Plotly→JSON in a future chart table over matplotlib PNGs) and images (image table keyed by analysis/message/report + originals in a bucket). (→ #26/#27)
7. Near-term: the remove-problem_statement work isn't on HF yet → PR + deploy + test in the playground (#13). Harry stabilizes Go ~Fri; FE manual testing ~Mon. Keep it playground-able.
8. UI research (no dedicated UI person): new-analysis form (title/objective/business_questions), knowledge menu (user-level vs analysis-level binding), report artifacts panel + version selector; interview + old analysis UI removed. (→ #28)

2. What is already done (KM-674, pushed on pr/4)

Report layer adapted to the new goal shape:

• report/schemas.py::ProblemStatement → objective: str + business_questions: list[str] (old target_value/scope/metric_direction/target_metric dropped). Class name kept for now (rename to ReportGoal once the upstream AnalysisState rename lands).
• report/generator.py renders Objective + numbered Business Questions + a "generated by {user}" line.
• api/v1/report.py::_problem_statement_from is tolerant: prefers new objective / business_questions from state, falls back to legacy problem_statement — works before AND after Harry's migration.
• config/prompts/report_summary.md updated to objective + business questions.
• Report stays records-based; the floor gate (problem_validated) was deliberately left for task #2.

This tolerant-migration pattern (getattr fallback) is the model for tasks #2 and #4.

3. Assessment — gaps & contradictions to resolve before building

These came out of reviewing the plan against the actual code. They are folded into the task table (§4) as tasks 15–19.

• G1 (→ task 15). Records-based reports need the slow path ON. AnalysisRecords persist only in chat_handler._run_slow_path, which runs only when ENABLE_SLOW_PATH=true. Default is off → no records → POST /report 409s. The Swagger demo can't show a non-empty report unless slow path is flipped on and a structured_flow question is run first. BusinessContext is still a stub but the slow path runs fine on it.
• G2 (→ task 16). analysis_records ownership is now required and collides with SKIP_INIT_DB. It's created today by Python create_all (db/postgres/init_db.py), is in no dedorch/Go migration, and after the dedorch cutover (SKIP_INIT_DB=true) Python stops running create_all → the table won't exist → reports break. Decide: dedorch migration (Harry) OR a Python carve-out that creates just this one table even under SKIP_INIT_DB. (Resolved 2026-06-25 — see §1.5.1 / #16 / #22.)
• G3 (→ task 17). chat_history in the report contract is vestigial. Records-based generation reads records by analysis_id; it never uses chat history. Drop chat_history from the report skill contract, or mark it reserved/unused.
• G4 (→ task 4 note). Make #2/#4 tolerant of both state shapes. If Harry drops problem_validated/owner_id from dedorch before Python stops reading them, Python's gate + state_store break. Use the same getattr tolerance KM-674 used. The owner_id→user_id rename also touches api/v1/analysis.py (_serialize_state, list_analyses, get_analysis), not just the model + state_store.
• G5 (→ task 18). "State writes via Go" is bigger than report_id. Python still writes state in /analysis/create (state + room + bindings, plus the data-first gate and soon the mandatory-field check) and in state_store.ensure per turn. If creation moves to Go (consistent with commenting analysis CRUD), then Go owns ALL state writes + both creation gates, and Python's ensure must become a read-only get (Go must guarantee the row exists before any chat turn).
• G6 (smaller).
  • Removing problem_statement (task 1) means neutering it in 4 places: the Intent literal (agents/orchestration.py), the router prompt (config/prompts/intent_router.md), the handler, and the gate's redirect target. Do it with task 2.
  • "generated by {user}" currently prints the raw user_id; a formal report wants a name — source from users.fullname or have Go pass a display name (task 19).
  • The meeting's outline (background / EDA / insights) isn't fully in the renderer; map those sections onto the record fields deliberately (task 5 follow-up).
  • The full project doc (task 11) should reuse REPO_STATUS.md, not restart.

4. Task table

Status legend: ⬜ not started · 🔄 in progress · ✅ done · ⛔ blocked · 🔎 verify · ⏸️ deferred.

#	Task	Owner	Status	Note
1	Comment out problem_statement skill + Intent literal + router prompt + gate redirect target; remove /problem-statement from list_tools	Rifqi	✅	Done 2026-06-25 (one commit w/ #2). Unwired in orchestration.py, intent_router.md, chat_handler.py, tools.py; problem_statement.py kept intact
2	Drop problem_validated: gate neutered; is_report_ready/report_floor → ≥1 completed analysis only, no-LLM	Rifqi	✅	Done 2026-06-25. gate.py no-op, gate call site commented in chat_handler.py, report_floor drops the goal check. Tests updated (test_gate/test_chat_handler/test_readiness). Suite: 284 passed, 7 skipped; ruff clean
3	dedorch analyses migration: drop problem_statement/problem_validated, add objective + business_questions	Harry	🔄	Verified dedorch 2026-06-25: analyses (plural) ALREADY has user_id + status + data_bind + data_bind_version + report_collection → those parts done. Remaining: drop problem_statement/problem_validated + add objective/business_questions. Singular analysis = deprecated duplicate to drop
4	Update Python analyses model + state_store + analysis.py to match dedorch; owner_id→user_id	Rifqi/Sofhia	✅	Done 2026-06-26. owner_id→user_id + added status/data_bind/data_bind_version/report_collection (DB-only, not in the AnalysisState pydantic) across models.py/gate.py/state_store.py/analysis.py + 3 local tests; also report_inputs id/analysis_id → uuid. Kept problem_statement/problem_validated; objective/business_questions wait on Harry's #3. Suite 284 passed
5	Report generator → objective+business_questions, "generated by {user}", formal outline	Sofhia	✅	Goal-shape (KM-674) + author name (#19) + outline (KM-680): Objective → Business Questions → Executive Summary → Key Findings → EDA → Notes & Limitations → How This Was Analyzed
6	Report skill input contract: analysis_id + user_id (no chat_history)	Sofhia/Rifqi	✅	No-op: POST /report already takes only analysis_id + user_id (records-based). Documented in API_ENDPOINTS.md §5. (Edit-mode input revisited in #24.)
7	report_id state update via request to Go, not direct DB	Sofhia + Harry	⬜	Needs Go endpoint. Checkpoint: Go owns ALL reports writes; Python stops any direct insert/update and only returns content; report stays a skill (no intent). See #18
8	Expose/confirm 4 FE endpoints; comment check_data + analysis CRUD	Sofhia	✅	KM-678: list_tools trimmed to /help + /report (analytics/check/retrieve commented in the menu). help confirmed as a call_agent intent — no own endpoint. Analysis CRUD endpoint left registered: "comment the rest" was about the FE slash menu, not killing HTTP routes Go needs
9	Verify analysis_id in call_agent contract	Sofhia	✅	Verified: no separate field — carried as room_id (analysis_id == room_id), per REPO_STATUS §4/§11. Action for Go: send the id as room_id
10	API endpoint doc (MD), 4 endpoints, for Go integration	Rifqi + Sofhia	✅	Done 2026-06-25 — API_ENDPOINTS.md (repo root). 4 FE surfaces with request/response examples (chat SSE transcript, report 201/409 JSON, version list), schemas, §9 full 32-route inventory + task-8 reading
11	Full Python project doc (MD → PDF/Word BRD)	Rifqi	✅	Done 2026-06-26 — PROJECT_BRD.md (repo root): purpose/context, FR-1..9 capabilities, lifecycle, architecture, data model, API (→ API_ENDPOINTS), NFRs, integrations, open items. Reuses REPO_STATUS/API_ENDPOINTS; convert to PDF/Word for distribution
12	Reconcile/open the list_tools PR cleanly (stacked commits)	Rifqi	✅	N/A — we develop directly on the single active branch pr/4 (KM-652 + KM-678 already stacked there); no separate PR to reconcile
13	Deploy HF Python build (remove-problem_statement work) → test 4 endpoints via Swagger / playground	Sofhia + Harry	🔄	Unblocked (#15 ✅). Remove-PS work is on pr/4 but not on HF main yet → PR + deploy, then manual test. Harry stabilizes Go ~Fri; FE testing ~Mon
14	analysis_records home	Rifqi + Sofhia + lead	✅	Resolved 2026-06-25: stays Python-owned, renamed (→ #21); schema handed to Harry so the dedorch migration creates it post-cutover (→ #22). Not moved to Go
15	Flip ENABLE_SLOW_PATH=true + verify an AnalysisRecord persists from a structured_flow question	Rifqi	✅	Verified locally 2026-06-25 (in-process). structured_flow on Titanic.csv → 3-task plan check_data→retrieve_data→analyze_aggregate (all success) → AnalysisRecord persisted (substantive) → report_floor pass → report generates (201). HF env-flip + Swagger run folds into #13
16	Decide analysis_records creation under SKIP_INIT_DB	Rifqi + Harry	✅	Resolved 2026-06-25: Python defines it; Harry's dedorch migration creates it on env-move (Python still creates locally meanwhile) → exists post-cutover. Execution = #22
17	Reconcile report contract with records-based: remove/flag chat_history	Sofhia/Rifqi	✅	Nothing to remove — chat_history was never in the report contract/code (only in help.md). Confirmed via grep; API_ENDPOINTS.md §5 documents the clean contract
18	Confirm Go owns ALL analysis-state writes + both creation gates; make Python state_store.ensure read-only	Rifqi + Harry	⬜	Confirmed by 2026-06-25 checkpoint (Python read-only; Go owns writes + new tables). Execution pending Go endpoints
19	Decide report author display-name source (users.fullname vs Go-passed name)	Sofhia	✅	Done 2026-06-25. AnalysisReport.user_name; generator renders user_name or user_id; api/v1/report.py::_resolve_user_name reads users.fullname never-throw (fallback user_id). Decided: resolve in Python (unblocked); swap to Go-passed name later if preferred
20	Help handoff: update handlers/help.py + help.md — drop the problem_validated tier + define_problem_statement action (the skill it points at is gone as of #1)	Sofhia	✅	Done 2026-06-25. help.py: actions = ask_analysis_question (always) + generate_report (if ready); renders objective/business_questions (getattr-tolerant). help.md v1→v2: 3 tiers, no /problem_statement, /generate report→/report. Local test_help updated → 11 pass
21	Rename analysis_records → report_inputs (table, ORM ReportInputRow, store *ReportInputStore)	Rifqi	✅	Done 2026-06-26. sed rename across 9 files; Pydantic AnalysisRecord kept; columns stay String (pure rename — uuid+FK is the #22 Harry schema). Name report_inputs (purpose; avoids Langfuse/analyses_messages clash). Write scope = one row per slow-path run. Suite 284 passed
22	Finalize report_inputs schema → hand to Harry for the dedorch migration	Rifqi → Harry	✅	DDL ready (uuid id/analysis_id + FK→analyses(id); user_id/plan_id text; data jsonb = serialized AnalysisRecord, shape documented). dedorch has empty analysis_records → rename. Resolves #16. Action: send Harry the DDL + data shape RESOLVED 2026-07-22: Harry's 0001_create_core_schema.sql now carries report_inputs with exactly this shape — FK to analyses(id) included. Python's ReportInputRow diffed against the live Neon table: column-for-column identical, no drift. The two remaining un-migrated Python tables (message_traceability, message_charts) split out as → #32
23	Report markdown formatting: tables, bold, italic, horizontal separators	Sofhia	✅	Done 2026-06-25. Added --- separators between header + each section in _render_markdown. Tables (EDA) / bold (method labels) / italic (meta + citations) already emitted. Relaxed report_summary.md to allow inline **bold**/*italic* for emphasis (kept no-headings/no-bullets so it doesn't duplicate the section structure / Key Findings). Compile + ruff clean
24	Clarify report input contract: records table (+ last_report for edit mode?)	Rifqi/Sofhia ↔ Harry	⬜ new	Edit-mode input left open at the checkpoint
25	Migrate Python chat path to Go analyses_messages (+ analyses)	Rifqi ↔ Harry	✅	Done 2026-07-02. Read path already on analyses_messages (commit 0066161). This change makes Python read-only: removed the save_messages calls from /api/v2/chat/stream so Go is the sole writer — fixes the double-write both Go+Python were producing. load_history still reads analyses_messages. v1 /chat/stream is unwired so left untouched
26	Charts (DEFERRED): store Plotly JSON in a future chart table (not matplotlib PNG)	—	🔎	Landed 2026-07-13 (pr/16, §0.6 V1–V3): render_chart + message_charts + GET /api/v1/charts, Plotly JSON as decided. 🔎 pending the dedorch DDL run (V5) + live e2e
27	Images (DEFERRED): image table (id, analysis_id, msg/report ref, order) + originals in a bucket	—	⏸️	Maintenance-heavy; parked
28	UI research (FE): new-analysis form, knowledge menu (user vs analysis level), report artifacts + version selector	Team	⬜ new	No dedicated UI person; interview + old analysis UI removed
29	LLM env quad rename __4o → __54m + set the four azureai__*__54m secrets on the HF Space	Rifqi	🔄	Code done 2026-07-14 (pr/17): settings quad + 9 call sites + 2 eval runners renamed; deployment is gpt-5.4-mini. Suite 381 passed, 2 failed (both pre-existing), 7 skipped — same as baseline; ruff clean on touched files; import main 0. Hard rename — no __4o fallback, so HF fails loudly on the first LLM call until the four __54m secrets are set. Root cause it fixes: HF silently ran GPT-4o while local ran 5.4-mini → same question, same catalog, planner hallucinated catalog ids on HF only. Action: set HF secrets, redeploy, re-run the failing question. Local .env still carries the now-dead __4o keys — safe to delete
30	Neon reports.user_id NOT NULL — reconcile the ORM + ReportStore write	Rifqi	✅	Done 2026-07-22 (pr/18, commit 13f74e1). ReportStore.save never wrote user_id; on Neon that column is text NOT NULL → every POST /api/v1/tools/report 500'd with NotNullViolationError. Fix: models.py adds user_id = Column(Text) (nullable on purpose — Python matches the loosest deployment shape, §7D getattr-tolerance) + store.py writes report.user_id, which the endpoint already required (user_id: str = Query(...)) and threaded via generator.generate. Live-tested by Rifqi → 201. Suite 381 passed / 2 pre-existing failures / 7 skipped; import main 0; ruff unchanged (3 pre-existing E501 in models.py, same count on HEAD). Correction 2026-07-22 (after pulling Go dd37b38): the commit message and the first version of this row said the column was not in Go migrations 0001–0004. That was wrong — 0001_create_core_schema.sql has declared reports.user_id TEXT NOT NULL since the schema was written. Real root cause → #31
31	Go migration set is not convergent — fresh vs migrated dedorch DBs get different NOT NULL constraints	Rifqi → Harry	⬜ new	Root cause of #30, verified in the Go source 2026-07-22. 0001_create_core_schema.sql creates reports.user_id TEXT NOT NULL and analyses_messages.user_id TEXT NOT NULL; 0002_cleanup_legacy_schema.sql (L95, L92) and 0004_replace_chat_with_analysis_scope.sql (L61, L57) retrofit the same two columns onto pre-existing DBs as nullable ALTER TABLE … ADD COLUMN IF NOT EXISTS user_id TEXT. Because CREATE TABLE IF NOT EXISTS no-ops on an existing table, the same migration set yields two different schemas — the old dedorch DB got nullable (hiding Python's missing write for months), Neon got NOT NULL. Ask Harry: make the retrofit converge (backfill + ALTER COLUMN … SET NOT NULL, or add the constraint in a new migration) so every instance matches 0001. Until then, information_schema on the target instance — not the migration files — is the only reliable schema source (REPO_STATUS §13). Action: Rifqi raises with Harry
32	message_traceability + message_charts are in no Go migration — hand the DDL to Harry	Rifqi → Harry	⬜ new	Found by the 2026-07-22 drift scan. Both tables exist on Neon only because they were created by hand (2026-07-06 / 2026-07-13); neither appears in 0001–0006. Any newly provisioned dedorch instance will be missing them → traceability flush and chart persist fail (both never-throw, so they degrade silently — no 500 like #30 to make it visible). DDL for message_charts is in SPINE_V2_PLAN.md §4.4. This is the surviving half of #22. Action: Rifqi sends Harry both DDL blocks

0.7. pr/19 — code review remediation (2026-07-23)

From the end-to-end review in CODE_REVIEW_2026-07-23.md (findings are cited as F-n there) plus the live report bug on analysis 966224d4…. Same status legend as §0.

#	Task	Owner	Status	Note
33	Report body vs floor split — has_reportable_result for the body, has_successful_analysis stays the floor	Rifqi	✅	Shipped 2026-07-23. Root cause: planner R2/R2b make analyze_* optional, so a correct analyze-free run was classed non-substantive, dropped from the body, and its business question rendered "Unanswered". Live-verified by Rifqi on 966224d4…
34	Report floor extension — a successful retrieve_data that returned rows clears the floor	Rifqi	✅	Same root cause; fixes a hard 409 for a session where every question is R2/R2b-shaped. Guardrail-adjacent, authorised 2026-07-23. NOT the "Floor Fixer" failure mode: the floor still asks "did we produce a real result" — empty retrievals, check_*-only and fully-failed runs all still fail it
35	GET …/records substantive flag repointed to the body predicate + contract §records updated	Rifqi	✅	The curation list was contradicting the artifact it curates. Behavioral, non-breaking: no field added/removed/retyped
36	CK5b — quality checkpoint covers analyze-free plans	Rifqi	✅	CK5 only inspected analyze_* tasks, so an all-null aggregate column on an R2/R2b plan reached the answer unflagged. check_* excluded (uncounted tables legitimately carry nulls)
37	F-2 service-secret gate — X-Dataeyond-Service-Secret, router-level dependency	Rifqi	⏸️	Shipped inert 2026-07-23. UNWIRED 2026-07-27 (lead decision). Verified in E2E-Frontend-Data-Eyond/src/services/agenticApi.ts that the browser SPA is the only caller of Python and we don't own it — it sends only Content-Type, and cannot be changed by us to send the header. So the gate could never be armed without a 401 outage: a wired-but-unarmable gate is a footgun (any operator setting dataeyond__service__secret breaks prod). Guard dependency removed from all six router mounts in main.py; service_auth.py kept in-tree but parked (comment-out-don't-delete); the dataeyond_service_secret setting commented. A prepared FE proxy (Node server.js holding the secret and injecting the header server-side) is the way to arm it later without exposing the secret to the browser — but that needs FE-repo access we don't have. Net: the live surface is unauthenticated by design; real auth = #43. F-1 tenant predicates (#38) stay as defensive-in-depth.
38	F-1 tenant scoping — user_id predicate on the six analysis-keyed reads	Rifqi	✅	CatalogStore.get_by_analysis filtered on analysis_id alone where Go filters on both; the catalog payload carries the owner's user_id, so DbExecutor's ownership check compared the victim's id against itself and passed → cross-tenant query execution against a customer DB. Defence-in-depth only until #37 is armed (user_id is caller-supplied, and GET /traceability leaks it)
39	Stale tests resolved — the two long-standing suite failures	Rifqi	✅	Both encoded the pre-2026-07-13 user-scope fallback that reader.py deliberately removed. Not product bugs. Suite is now 394 passed / 0 failed / 7 skipped — fully green for the first time
40	F-3 — scope GET /charts + GET /traceability by user_id	Rifqi	⛔	RESOLVED 2026-07-23 — DECLINED by Rifqi. Do not re-open without his sign-off. Both endpoints keep their existing lookup keys: /traceability by (analysis_id, message_id), /charts by message_id alone (the 2026-07-13 lead decision). F-3 proposed adding a user_id parameter to both; an optional-param version was implemented on 2026-07-23 and then reverted in full (endpoints, charts-store predicates, and the contract notes) once the decision was restated — no FE change is required and none should be requested. Accepted consequence: both endpoints remain unauthenticated capability URLs over real customer data (charts[].spec.plotly.data is actual table values; the traceability payload carries 5-row previews, the executed SQL, and the owner's user_id). The service-secret gate (#37) is therefore the only control protecting them — which raises #37 from important to load-bearing. The stores' optional user_id parameters are kept as dead capability for a future Go-forwarded identity (#43); PostgresChartStore was returned to its message_id-only form.
41	F-12 / F-13 — bound the planner catalog render and the tabular blob read	Rifqi	✅	Shipped 2026-07-23. F-13: neither storage backend could report an object size, so object_size() was added to both (S3 head_object→ContentLength, Azure get_blob_properties().size, each returning None rather than raising) and TabularExecutor now refuses >500 MB before downloading, with a post-download byte check as the fallback when the probe is unavailable. F-12: render() gained TWO ceilings — _MAX_TABLES=150 and _MAX_CATALOG_CHARS=250_000 — because a table-count cap alone leaves wide tables unbounded (20 tables × 300 cols is as fatal as 400 × 30). Measured after: 400×30 went from 241k tokens to ~63k; 100×30 (45k tokens) still renders in full, so no realistic catalog is touched. Truncation emits an explicit "N more tables not shown" line so the planner knows it saw a subset, and logs — that log is the signal to retune. 12 new tests
42	F-20 observability — degraded_seam=<name> on every never-throw / silent-drop path	Rifqi	🔄	The 2026-07-23 report bug was invisible by construction: the record was dropped with zero logging. Partially shipped 2026-07-23 — the 10 seams where silent degradation is user-visible now emit a stable degraded_seam field (+ repr(e) instead of str(e), so an empty-str() Fernet error is no longer a blank log): input_guard_fail_open, analysis_catalog_read, report_floor_record_read, traceability_persist, traceability_flush, chart_persist (×2), report_input_persist (×2), analysis_state_ensure. Remaining: the other ~76 except Exception sites, most of which are in unwired routers (db_client, data_catalog, users) or non-live paths — deliberately not swept, since a blanket edit across unwired code is exactly the drive-by §7A forbids. Control flow unchanged throughout (§5.4)
43	Go identity contract — what does Go forward, and when?	Rifqi ↔ Harry	⬜	Now the sole path to caller auth after #37 was unwired 2026-07-27. Python cannot authenticate the caller alone; either Go forwards a verified per-user token, or the FE (once we can change it) forwards the Go bearer token it already holds (orchestrationApi.ts shows the FE has one). Until then the live surface is unauthenticated and the #38 predicates are defensive only. Options for arming interim protection when FE access returns: the server.js BFF proxy (secret server-side) or route agentic calls through Go
44	F-17 / F-18 / F-25 compiler-parity batch	Rifqi	✅ new	Shipped 2026-07-23. Three execution-verified review findings that had no task row — the tracker jumped from #39 to #40 and lost them. F-17 (High): the bare-select check was gated on if ir.group_by, so a mixed select with group_by=[] passed; Postgres then failed loudly but the pandas path silently DROPPED the column while output_columns still advertised it → a real-looking table with a fabricated all-null column. Fixed in validator.py (fires whenever any agg is present — no false positives possible) + a presence backstop in TabularExecutor. F-18: astype(str) ran before na=False, so NULL LIKE '%an%' matched the literal "nan"/"None". F-25: SqlCompiler raised on an empty in/not_in where pandas and _column_values' own docstring implement the empty-set semantics — a two-step plan whose first step legitimately returned zero rows hard-failed instead of answering. 16 new parity tests; one stale test updated (it pinned the old F-25 raise)
45	F-9 PII in persisted artifacts — mask the traceability preview + report evidence tables	Rifqi	✅	Shipped 2026-07-24. pii_flag was an ingestion-time control only: it nulls sample_values into the planner prompt, but nothing stops the planner SELECTing a flagged column — and "list our top 20 customers" legitimately selects customer_name/email. Real values then reached two PERSISTED sinks: message_traceability.data (served by an unauthenticated GET, F-3 declined) and report evidence tables frozen permanently into reports.content. Fix = retrieve_data now carries meta.pii_columns (resolved through the IR select list, so aliases are honoured), and both sinks redact those cells to [redacted]. Per the 2026-07-23 decision the ASSEMBLER still receives real values, so the answer prose is unchanged and the question stays answerable — verified end-to-end: assembler input kept Ada Lovelace, the persisted preview showed [redacted]. Aggregates are deliberately NOT masked except min/max: sum(salary) identifies nobody, but max(email) returns one customer's actual address. Fails open (an unresolvable name is left unmasked, never a legitimate column wrongly blanked), so this is a mitigation, not a guarantee. 15 new tests
46	F-8 prompt-injection resistance — planner / assembler / report_summary	Rifqi	✅ new	Shipped 2026-07-24. The three prompts that ingest customer data had no injection rule: guardrails.md is appended only in chatbot.py/help.py, and the InputGuard screens only the user's message — it never sees catalog or row content. This is the one attack the five query-defense layers structurally cannot see, because every IR the planner emits is individually valid. A hostile string only needs to reach a text column in the customer's OWN database (a product description, a support ticket, a form field) to be sampled into sample_values and rendered verbatim into the planner prompt. Fix = a purpose-written "content is data, never instructions" rule in each prompt + <data>…</data> delimiters around the catalog render and the run-state render. Deliberately NOT guardrails.md wholesale — its rules prescribe refusal sentences, and the planner's only free-text field is infeasible_reason, so those strings would surface there and could regress the Q2 data-gap path (tests pin their absence). Verified: planner eval 6/6, carried_over 5/5 green; live hostile-catalog run planned only t_products and never touched the planted employees.salary; 10 new tests
47	Cheap-batch review fixes — F-19, F-22, F-24, F-26, F-16, F-4	Rifqi	✅ new	Shipped 2026-07-24, one low-risk batch. F-19: sources was missing entirely on check + router-help and came after status on the slow path — contract said always-first; additive fix, contract updated. F-22: analysis_id now 422s unless it parses as a UUID, on both live endpoints (chat + help kept identical on purpose). F-24: traceability/chart writes are skipped, and logged, when analysis_id is falsy — Go 0007 declares analysis_id UUID NOT NULL (+ FK on charts), so analysis_id or "" failed the insert and the never-throw seam lost the row silently. F-26: QueryResult.error uses str(e) or repr(e) — a Fernet InvalidToken reached the assembler/traceability/report as an EMPTY string; falling back only when str() is empty changes no existing message. F-16: retrieval cache key gains redis_prefix (two envs on one Redis cross-served results). F-4: non-Postgres sources now refused at the executor — zero blast radius today (Go's isSupportedActive allows only postgres), a tripwire so nobody re-enables a path that has no read-only session and no statement_timeout; legacy branch commented out per house convention, orphaned import commented with it
48	floor_08 — floor/body disagreement	Rifqi	✅	Decided + shipped 2026-07-24 (lead). #34's row-producing arm was unconditional, so a plan that HAS an analyze_* step which FAILED still cleared the floor on the strength of its upstream fetch — while the body rejected it. Because the "Attempted, Unresolved" section is commented out, such a run left no trace: as a session's only run it produced an empty report with the business question "Unanswered" (the #33 bug via another door). Fix = the arm now applies only when the plan has no analysis step, mirroring has_reportable_result; the shared _plan_has_analysis helper means the two predicates can no longer drift on that question. floor_08 flips to expected_ready: false. The INTENDED asymmetry is preserved and re-verified: a zero-row retrieval still fails the floor but passes the body (floor_03). Readiness eval 17/17
49	F-5 timeout does not stop the customer's query	Rifqi	✅ new	Shipped 2026-07-24. asyncio.wait_for cancels the awaiting coroutine; the to_thread worker is not cancellable and runs to completion, holding a thread and a connection on the customer's database after we already answered "timed out". Two fixes. (a) Dedicated bounded pool — DB work moves to its own ThreadPoolExecutor(50, 'dbexec') via run_in_executor; abandoned workers previously accumulated in the shared default pool (min(32, cpu_count+4)) alongside every other to_thread caller, notably the tabular Parquet loader, so a few slow customer queries could stall unrelated work process-wide. (b) Session hardening is no longer best-effort — and a latent bug was found while reading it: both SETs shared one try, so a statement_timeout failure skipped default_transaction_read_only entirely and the connection served queries in a WRITABLE session behind a logger.warning. Now independent: read-only fails the connection if it cannot be set (a writable session against a customer DB is not something to degrade into); statement_timeout logs at error with degraded_seam but does not refuse service, since it is their I/O at risk rather than our correctness. ⚠️ Blast radius to watch: if any deployment currently fails that SET silently, its sources now fail loudly instead. Believed impossible (Neon accepts it as a SET — the existing comment documents this), but it is the one judgement call here. 7 new tests

Reading eval/readiness/results/ (note for future sessions). Four files are dated 2026-07-23. …_150632.json scores 4/15 (26.7%) — that is not a product regression. It is the run that exposed the eval harness itself being broken by fd4865b (_FakeRecord had no results_snapshot for #34, _FakeStore took no user_id for #38; both errors were swallowed by report_floor's fail-closed seam, so every case reported "not ready"). …_150859 (15/15) is post-harness-fix, …_150948 and …_152615 (17/17) add the two cases that exercise #34. The current baseline is …_152615.json — 17/17. The intermediate files are kept as the audit trail for the drift; per §7F no result file is ever deleted or overwritten.

Not re-raised: F-4 (non-Postgres read-only/timeout gap) was downgraded to latent — Go's database_clients.Service.Create enforces isSupportedActive, and only postgres is active, so no such source can be registered today.

Decided 2026-07-27 (lead) — F-2 gate unwired, not armed. The service-secret gate (#37) was removed from the router mounts because the sole caller is a browser SPA we don't own and can't change to send the header; arming it would 401 the whole app. The live surface is unauthenticated by design until #43 (Go-forwarded identity). Not a gap to re-raise — it is a recorded posture. CORS was left at ["*"] on purpose (tightening it needs the FE origin as config, which we chose not to set for now).

0.8. Knowledge pipeline — test integration (2026-08-19 → building, 2026-08-21)

Updated 2026-09-11 — the open-queue pass (K17–K22)

Six items off knowledge_pipeline_runs/SESSION_HANDOFF.md §4, zero API spend: every number below comes from replaying the six persisted paid runs or re-running the free stages on the three v2 artifacts. Detail in REPO_STATUS §14; per-item rows K17–K22 in the table below.

The headline is Step 2, the item everyone agreed on and nobody had done: the review queue put 24 unresolved key parameters in front of Komatsu's Engine (108 mentions, rank 25). It is rank 1 now, with nothing dropped.

Two findings worth reading as findings, not as changes:

1. The spend ceiling was gating on a partial number. estimate_cost summed only the glossary branch under a field named estimated_prompt_tokens. Real undercount 8.8%–18.4%; Open Pit is at 78% of the ceiling, not 71%.
2. Escalation has never worked. 48 calls across three documents and two configurations, 0 definitions recovered. Retired by constant, loop left in place.

What is deliberately NOT switched on: genre routing. The classifier is right on all three documents, but the label sets are unscoreable without a second gold set and branch-skipping rests on a single document, so ROUTE_LABELS / ROUTE_BRANCHES are both off and the verdict is recorded rather than acted on. Flipping either is Rifqi's call and one line.

Still open and unchanged: the paid Step 0 band (needs authorised repeated runs), Phase 4's vision call, Step 8, Step 7, R4, and E5 — Sofhia still has not seen the src/knowledge_parsing/ diff.

Updated 2026-09-11 — v2: clustering fix + parsing 0.4.0 (asset pointers)

Two changes, measured on three documents (BUMA standard · Komatsu shop manual · Open Pit textbook Ch.1) at zero API spend for the free stages — the parse cache fingerprint is unchanged, so re-parses reuse the cached Mistral responses. Full write-up: knowledge_pipeline_runs/CHANGELOG_v2.md.

The fuzzy clustering pass is RETIRED (K13). token_set_ratio returns exactly 100 when one token set is a subset of the other, so a head noun always absorbed its own compounds and the threshold of 92 was inert. 131 of its 140 merges were subset matches (all wrong); 6 were plurals, now folded deterministically; 3 were distinct metrics wrongly merged. Token-subset merges 15/18/97 → 0, and post-cluster gold recall 0.8293 → 0.8537 — the gold term EWH, which Working Hours had absorbed, is recovered. _fuzzy_match is parked, not deleted.

Parsing schema_version 0.4.0 (K14). A figure is no longer its own chunk. It was, with an EMPTY text, which made it unreachable by construction — 13 figures across three documents, each described by a vision model we pay for, and ZERO reaching any prompt. Now the ![](asset://<id>) reference is folded into the prose chunk at its reading-order position and the figure travels as an Asset (content-addressed, persisted bytes, verbatim caption, unwrapped description). Image chunks 2/3/8 → 0/0/0; recall unmoved (E1 and E1c both identical v1→v2). Tables deliberately stay their own chunk. adapter.py now carries assets, referenced_by and table_html, which parsing had emitted for months while the adapter dropped it.

New metric E1c (post-cluster recall), and E2's compression kill line REMOVED — over-merging raises compression, so the metric was rewarding the defect it should have exposed.

Two settings changes measured and REJECTED: WINDOW_WORDS 256 (truncates 46 of 99 Open Pit windows — the "flat recall" justifying it was measured on BUMA, the one document whose truncation count does not vary with window size) and SPAN_SCORE_THRESHOLD 0.15 (+50% clusters against a +20% bar). EVIDENCE_K raised 3 → 5 on Rifqi's call; measured cost +1.8% to +15% evidence tokens, and on the one gold-set document top-5 recovers the same 16/18 as top-3.

Export for the team: knowledge_pipeline_runs/06_export_knowledge_parsing_v0.4.0/, verified to import and validate a fixture with no repo on sys.path. ⚠️ src/knowledge_parsing/ is Sofhia's half — she needs the diff before it goes out.

Paid comparison landed 2026-09-11 (K15). Definitions 7→10 / 0→0 / 25→49; queue junk rows down 37–65%; Komatsu still 0, as predicted. Caveats: the source_wording gain is largely tautological, and no definition cited a figure caption on any document — assets are reachable but unused, which points at the caption-to-filter path (built, flag off) as the next free experiment. Open Pit now sits at 71% of the 2M spend ceiling for one chapter.

Updated 2026-08-21. The banner here said "Nothing has landed in this repo" — that is no longer true. Both halves are now in-tree (src/knowledge_parsing/ + src/knowledge_extraction/

• eval/knowledge/, ~5,360 lines across commits 024c30a…a7bdeae), and three dependencies were added. Still true, and the part not to misread: no HTTP surface, no database table, and nothing mounted in main.py — both halves are offline CLIs. Superseded 2026-09-07: there is now an HTTP surface (8 routes) and five tables. See the 2026-09-07 delta below and REPO_STATUS §14.

Updated 2026-09-07 — what this pipeline is for, and what Data Eyond actually needs. Recorded because it changes the definition of done. The review loop, the admin ingestion role and the curated-artifact lifecycle belong to a separate B2B knowledge product, where a company admin configures the company's knowledge and an expert reviews candidates before the company's normal users consume them. This repo is where that concept is being tested cheaply, on real documents. Data Eyond's own version is one added step: a user clicks a button and knowledge is extracted from all of that user's documents in one batch — no admin role, no review queue, no curation UI. Full framing: knowledge_pipeline_context.md §1.5/§1.6.

Consequences already folded into the docs:

• Multi-document semantics settled — entries append, never merge; entity_id stays document-scoped; cross-document overlap is classified (new/duplicate/conflicting) and surfaced, never silently collapsed; DomainContext stays per document. A canonical company-level concept layer is deferred to Product B — it needs an expert to rule on conflicts, and Data Eyond's flow has none.
• Persistence contract revised to v2 (five tables): Q1 closed — documents.id is text PK, verified by read-only introspection of live dedorch; knowledge_jobs added (batch envelope, and a parse failure has nowhere else to live); and scope_id — company scope — added to every table (Rifqi, 2026-09-07): knowledge belongs to the company, not the uploader, and a per-user key means two employees' documents never diff against each other. dedorch has no tenant entity (15 tables, none of them one) and users.company is nullable free text, so the value is provisional — normalised users.company, else user:<user_id> — and scope_id is deliberately excluded from the entity_id derivation, which is what keeps re-pointing it a value migration rather than a re-key that would destroy expert approvals. Raised to Harry as Q6: we want a real companies table.
• N7 closed. "MinerU cannot install in the Space" predates the Mistral backend; Mistral OCR is a hosted API. Corrected 2026-09-07 — "void" was too strong: the python-docx/pypdf conflict was making uv lock fail for the whole project, silently blocking every dependency change. Closed by dropping MinerU from the extra (26ba1ab), not by lifting the pin. N7b done in the same change: torch pinned to the Linux CPU wheel, 36 nvidia-* packages and triton gone from the lock. GLiNER is the term filter, so torch cannot become an optional extra if extraction runs in the Space.
• D4 re-opened. "Admin/service-auth" was right for Product B; Data Eyond's caller is the browser SPA, which is the #37 problem. Rate limit + pre-flight spend ceiling + idempotency bound the damage but do not identify the caller — Rifqi's call, see the surface plan §6.
• Q5 for Harry: message_traceability and message_charts exist in dedorch (80 and 6 rows, idx_message_charts_message present) but were created by hand and are not in Go's migration set — a fresh environment comes up without them and both write paths fail silently.

Context and intent: knowledge_pipeline_context.md. Task breakdown, owners and verified statuses: KNOWLEDGE_PIPELINE_TODO.md.

Shape. A six-stage pipeline (parse → free CPU/regex filter → one LLM call per term cluster → deterministic span validation → diff vs. the active glossary → frequency-sorted expert review) that turns client documents into reviewable knowledge entries, ultimately served over MCP. It splits into two independently runnable halves — parsing (Sofhia, Mistral OCR; MinerU dropped 2026-09-07) and knowledge extraction (Rifqi) — meeting only at a persisted, versioned parsed-document artifact. Extraction never takes a file path; that constraint is the whole point of the split.

Where it started (audited 2026-08-19). A working prototype (kex, ~2,763 LOC, 21 passing tests, 8-command CLI, four experiments against real Azure calls) exists outside version control at Documents/Dataeyond_catalog/knowledge-pipeline/. It is not the product and was not ported — it answered the design questions; extraction v2 was rebuilt here. What carried forward is the evidence (gold set, experiment results) and the tuned constants, not the code. Measured on the BUMA standard (9 pages): GLiNER Indonesian recall 0.854 (kill line 0.70, PASS — this closes the context doc's headline open risk), clustering compression 2.56× (kill 2.0, PASS), nano schema-fill precision 0.75 (kill 0.80, FAIL), span-validation rejection 1.00 with 0.00 false rejections (kill 0.90, PASS). Cost $0.0069 for the document.

Where it stands (2026-08-21). Both halves run end to end as offline CLIs, on the same document:

python -m src.knowledge_parsing.run --input <folder>	PDF/DOCX → versioned ParsedDocument
python -m src.knowledge_extraction.cli <artifact.json> --extract	artifact → entries + review queue

The seam holds from both sides: ParsedDocument is the only shape crossing it, extraction takes no file path, and local seam tests import knowledge_parsing.contracts directly so a rename on either side fails loudly. Rebuild verified against the frozen baseline: term-filter recall 0.8537, identical to the prototype (eval/knowledge/results/v2_term_filter_2026-08-19_144318.json). A live 3-call pilot on gpt-5.4-nano returned correct Indonesian definitions with 0 span rejections and 7,168 of 8,029 prompt tokens served from cache.

Not done, and not to be read as done: no persistence (stage output is JSON on disk — D2 still owes Harry a DDL block) and no HTTP surface. no full-document extraction run, and E3 has never been scored on v2 — RESOLVED 2026-08-27 (K9): the full-document run is routine now and E3 scores 1.000 on a committed fixture; see the K9 row for how to read that number.

#	Task	Owner	Status	Note
P1	Freeze the kex prototype (read-only snapshot)	Rifqi	✅	2026-08-19: kept as-is, unversioned — Rifqi's call. Risk accepted, and much reduced once P2/P3 lifted the two things that could not be recreated
P2	Harvest the prototype's tuned constants	Rifqi	✅	Done → KNOWLEDGE_PIPELINE_CALIBRATION.md, 10 sections: every value with the reason it has that value, plus §7 negative findings and §9/§10 the deviations made while rebuilding
P3	Migrate the evidence into eval/knowledge/	Rifqi	✅	Done. Gold set (41 terms + 15 rules), ported scorer, frozen baseline. Migration proved faithful: the ported scorer re-derives the prototype numbers exactly
S1	Agree the parsed-artifact schema	Sofhia + Rifqi	✅	Settled 2026-08-21. src/knowledge_parsing/contracts.py (schema_version 0.2.0; 0.3.0 as of the 2026-08-27 fixture) is the shape of record and the extraction adapter consumes it directly. Envelope, verbatim heading, 0-based page_idx, prose text with latex/table_html beside it
S5	Keep the seam off the file path	Sofhia + Rifqi	✅	Satisfied both sides. ParsedDocument carries no handle to the parser, and knowledge_extraction.adapter accepts only the artifact. Locked by local seam tests importing the parsing contract directly
X*	Build extraction v2 in this repo	Rifqi	✅	Done 2026-08-21 — src/knowledge_extraction/: adapter · filters (span/cue/legend) · cluster · rank · extract (4 branches) · validate · diff · queue · service facade · CLI. Recall matches the frozen baseline exactly
D2	Artifact + candidate-entry persistence	Rifqi → Harry	✅ (Python) / ⬜ (Go)	Built and live 2026-09-07. Five tables created in dedorch (docs/knowledge/knowledge_tables.sql), ORM models, knowledge_store.py, PostgresJobStore. Verified end to end: 88 entry rows persisted from the BUMA standard. ⚠️ Harry must still fold the SQL into Go's migration set — the tables exist only because they were run by hand, the same gap that left message_traceability/message_charts out
D4	Admin ingest surface	Rifqi	✅	Built 2026-09-07. 8 routes under /api/v1/knowledge (contract §Knowledge): ingest + poll, re-extract, entries, queue, reviews. Async batch (202 + poll, not SSE), company-scoped, rate-limited, spend-gated, single-flight. Auth posture recorded, not solved: unauthenticated like the rest of the surface, and POST …/reviews accepts content that reaches a later prompt — the strongest case yet for #43
R1	Fix the literal-source-wording violation	Rifqi	✅	Fixed and live-verified 2026-08-21. source_wording is read from the document's own heading rather than asked of the model (asked directly, it quoted the normalised page-1 form); the span haystack includes headings; the queue flags the mismatch. PA now surfaces as "source wording differs from the expanded name"
D3	Provision a mini deployment	Rifqi	⏸️	Deferred 2026-08-19: staying on nano. The __54n quad is wired in src/config/settings.py. Accepted knowingly — nano measured 0.75 against a 0.80 line, and rule/summary run on the tier whose failure mode is least detectable
D6	Expert review of the gold set	Rifqi → Mas Beta	⏸️	Deferred: provisional gold is good enough for now. Standing caveat on every score, E1 and E3 both move when it is reviewed
D5	v2 module name	Rifqi	✅	src/knowledge_extraction/, beside Sofhia's src/knowledge_parsing/. Neither collides with the existing src/knowledge/ OCR→pgvector service, which is untouched

| D1 | New dependencies | Rifqi | ✅ | Added 2026-08-21. rapidfuzz, gliner, torch>=2.6 as main deps; mineru==3.4.4 as an optional extra (.[knowledge-parsing]) so the deployed Space never builds it. Updated 2026-09-07 (26ba1ab): mineru dropped — it made uv lock fail project-wide; torch pinned to the Linux CPU wheel. The §6 gliner/torch question is now RESOLVED | | K9 | Full-document extraction run + score E3 on v2 | Rifqi | ✅ | Done — latest run 2026-08-27 (eval/knowledge/results/v2_full_2026-08-27_133757.json, 90 calls, 170 s). E1 0.8293 / E2 2.556× / E3 1.000 — all three above their kill lines; rule recall 0.4286 (6/14, tracked not gated). The prototype's failing experiment now passes, but read E3 as precision on a scoreable base of 6: 57 of 63 entries abstain, so recall against the 18 gold terms with criteria is 0.33. 0 fields rejected by the span check | | K10 | Fixture artifact committed | Rifqi | ✅ | Done 2026-08-27 — eval/knowledge/fixtures/parsed_STD_2026_006_MNO.json (BUMA standard, MinerU 3.4.4 hybrid, schema_version 0.3.0, 31 chunks). The eval and the extraction CLI both run with MinerU absent. The Open Pit textbook fixture is still open (S7 in the checklist) | | K11 | Output schema v2 (Harry's contract review) | Rifqi | 🔎 | Landed 2026-08-26, unevaluated. Deterministic entity ids, glossary references formulas instead of restating them, rules link to formulas and terms, 16 fields cut (50 → 42). Contract: docs/knowledge/KNOWLEDGE_OUTPUT_CONTRACT.md; detail in the checklist as X24. Fixed two live defects — neither cluster_id (a frequency ordinal) nor rule_id (model-written) survived a re-run, which breaks an expert's approvals, not just links. Suite 497 passed / 7 skipped; no eval run — the prompt edits join X20's in the unevaluated queue | | K12 | Output schema v3 — DomainContext | Rifqi | ✅ | Done 2026-09-02 (7f47c61 · 6f5d619 · e2865ac · 172d8dc; checklist X26). BriefContext → DomainContext, renamed while nothing is persisted — afterwards an id-prefix change is a migration that loses an expert's approvals. The branch is no longer the pipeline's one unverifiable output: it was made guardable by changing the question rather than the model — the prompt asks it to locate the document's own statement of purpose (purpose → purpose_verbatim) instead of composing a summary, so GUARDED_FIELDS["summary"] goes () → ("title", "purpose_verbatim"). Adds derived outline, subdomains and entry counts; key_parameters gains a corroborating term_id, and an unresolved one earns its own review-queue row. Contract change for Harry: queue rows now carry row_kind, and a key_parameter row has no term_id, definition or mention count. Measured on the BUMA fixture (nano, 12 calls): 0 fields rejected, fabricated purpose refused, key-parameter linking 0/5 → 4/5 after a bug the smoke caught. Suite 541 passed / 7 skipped | | K13 | Clustering: retire the fuzzy pass | Rifqi | ✅ | Done 2026-09-10. token_set_ratio is blind to the subset case (returns 100), so the threshold never applied and head nouns ate their compounds. Measured across three documents: 131 of 140 merges were subset matches. Now exact (plural-folded) → abbreviation ↔ expansion only; depluralise() handles the 6 legitimate plural merges. Subset merges 15/18/97 → 0; E1c_strict 0.8293 → 0.8537 (EWH recovered). New metric E1c added to score.py/run_eval.py; E2's kill line removed (it rises when the defect fires). Suite 676 passed / 7 skipped | | K14 | Parsing 0.4.0 — asset pointers | Rifqi (Sofhia's half) | ✅ | Done 2026-09-10. Figures fold into the prose chunk as ![](asset://<id>) at reading-order position; Asset carries a content-addressed id, persisted bytes, a verbatim caption and a model-written description that never enters text. Image chunks 2/3/8 → 0/0/0, 0 dangling refs, 0 description leaks, captions 17/17 preserved, recall unchanged. Tables keep their own chunk + two-way pointers. Seam: adapter.py carries assets/referenced_by/table_html; SUPPORTED_SCHEMA → (0,4), so parsing and extraction ship together. Found and fixed a Mistral bug: image_id mislabelled on one block, collapsing two figures onto one hash. ⚠️ Sofhia needs the diff before the team export goes out | | K15 | Paid v1-vs-v2 comparison | Rifqi | ✅ | Done 2026-09-11, 3 runs, ~9¢. Definitions 7→10 / 0→0 / 25→49 — Open Pit nearly doubled; junk review-queue rows −37% (Komatsu) / −65% (Open Pit); cache rate +7% on all three; 0 retries. Komatsu stays at 0 definitions, as predicted: a troubleshooting manual states symptoms, not definitions, and no plumbing change reaches that. Two caveats recorded rather than buried: source_wording's headline +1608% is mostly tautological (informative 12→101, not 13→222), and definitions_citing_a_caption = 0 on every document — the assets are reachable but nothing consumed them, most likely because captions do not feed the term filter (flag off by decision). ⚠️ Cost: Open Pit is now 71% of the 2M pre-flight ceiling (was 56%) for one chapter. Three variables moved at once, so no single number is attributable. Write-up: knowledge_pipeline_runs/V1_VS_V2_PAID_COMPARISON.md | | K16 | Caption hypothesis for the 0 caption-citations | Rifqi | ✅ | Tested 2026-09-11, HYPOTHESIS FALSIFIED. Both candidate explanations for definitions_citing_a_caption = 0 are false: captions ALREADY reach the term filter (they are part of Chunk.text; 2/2 BUMA and 3/8 Open Pit captions already produce mentions), and figure-bearing chunks are ALREADY selected (a captioned chunk is in the top-5 for 22/83 BUMA and 64/427 Open Pit clusters). Built the missing ranker signal asset_caption_names_term to test it properly: at weight 2.5 it changed the top-5 of 1 cluster out of 603 and moved none from no-figure to has-figure. The real reason: not one caption in the corpus is a definition — they are figure titles ("Figure 1.3. Relative ability to influence costs"). A model declining to quote one is CORRECT. Signal ships at weight 0.0 with the measurement beside it. ⚠️ Also corrects a false claim in the earlier docs: the caption-to-filter flag was described as "built, off by decision" — it was never built. Redirects the remaining figure value to Phase 4 (vision call on the bytes), which is now the only untested path |

| K17 | Step 2 — review queue ordering + key-parameter heads | Rifqi | ✅ | Done 2026-09-11. Unresolved key parameters held their own sort tier, so every one outranked every routine term; the branch emits them in bulk on the documents it understands least, so the promotion fired hardest where least deserved. Now spliced at most 1 per 5 term rows. Replayed on the three persisted v2 runs: Komatsu Engine (108 mentions) rank 25 → 1, Open Pit Resource 8 → 1, first junk row 1 → 6, row counts identical (nothing dropped), BUMA unchanged (0 unresolved). term now carries a head (first . : or newline, only past 60 chars — "No. of units" survives) with the full text in new surface_full. Contract: OUTPUT_CONTRACT §9 | | K18 | Step 0 — measurement hardening | Rifqi | 🔄 | Harness done 2026-09-11, the paid band is NOT run. eval/knowledge/invariants.py: asserts on fields and invariants, never exact text — span locatable, entity ids present and unique, a rejected field still null (rejected-never-repaired, checkable only from outside), page in range, queue ranks contiguous, no leaked sort keys. --runs N repeats only the paid stage (free stages are deterministic) and reports min/median/max with the median as the value, plus whether all runs minted identical entity ids — the K11 defect class, unwatched since K11. EXPECTED_MODEL_SNAPSHOT turns X25's recording into an assertion (exit 6). New exit codes: 5 invariant, 6 snapshot. Needs Rifqi's go-ahead to produce a real band | | K19 | Step 6 — spend ceiling in cluster units | Rifqi | ✅ | Done 2026-09-11. Found first: estimate_cost summed only the glossary branch under a field named estimated_prompt_tokens, so the ceiling read a partial number as a total — undercount BUMA 18.4% · Komatsu 8.8% · Open Pit 15.2%, moving Open Pit 67.7% → 78.0% of the 2M ceiling. Now all four branches (glossary exact, the rest approximated from prefix + chunk text, both labelled). Second ceiling MAX_CLUSTERS = 750 — exact, free, and what drives spend; at ~3,200 tokens/cluster the token ceiling still binds first for these documents, so this guards the other shape (many cheap clusters, 494 calls ≈ 15 min). Refusals name the ceiling and the declined call count; IngestJob.estimated_cost (declared since the job model, never populated) now carries the per-document breakdown; a wholly-refused batch settles blocked, which the contract documented and the code could not produce | | K20 | Step 5 — genre classification | Rifqi | 🔄 | Classifier done and correct on all three documents; routing deliberately OFF. src/knowledge_extraction/genre.py → standard · diagnostic · conceptual · unknown from four free signals (heading coverage · tabular ratio · abbrev pairs · rule density), which separate in different directions — measured BUMA 46%/29%/5/0.57, Komatsu 1%/74%/0/0.01, Open Pit 86%/7%/0/1.48. ROUTE_LABELS and ROUTE_BRANCHES are both False: the genre label sets are unscoreable without a second gold set (more mentions ≠ better recall) and branch-skipping rests on one document. Today it is observability — well_served=False warns before spend that a shop manual yields almost nothing, and heading coverage raises a per-document warning (P5, partly: the warning exists, a parsing-side fallback does not). Flipping either lever is Rifqi's call | | K21 | Finding 6 — escalation retired | Rifqi | ✅ | Done 2026-09-11, with the number Finding 6 asked for. Across all six persisted paid runs (v1 K=3 and v2 K=5 × three documents): 48 escalation calls, 0 definitions recovered — extraction_status == "escalated" appears zero times in every file. Why it cannot work as built: escalation hands the model the next k evidence chunks, ranked below ones it already declined, and 16 of 18 gold definitions sit at rank ≤3. MAX_ESCALATION_ROUNDS = 0; loop and call site untouched. Reviving it means changing the question (different prompt or tier), not the evidence | | K22 | The over-cap window (pre-existing) | Rifqi | ✅ | Done 2026-09-11. WINDOW_WORDS caps words; GLiNER truncates on subwords and warns rather than failing. Measured with GLiNER's own tokeniser: 1 window of 368 exceeds the 384-token cap — BUMA STD_2026_006_MNO::0024, 130 words / 738 chars → 409 tokens — losing its tail at every window size including 130, exactly as the 2026-09-10 sweep suspected and left. Windows are now checked against the real tokeniser and split only when they do not fit (a character budget would have to assume the tightest ratio seen, 1.57, and chop up the 367 that fit). Halves overlap by SPAN_TOKEN_CAP words so no term straddles a cut. Fixture: 3 splits, mentions 186 → 208, clusters 74 → 80 (+8% spend), E1 0.8293 and E1c unchanged | | K23 | Step 8 + Step 7 design proposals | Rifqi | 💡 | Written 2026-09-11, nothing built, both need sign-off. TABLE_BRANCH_PROPOSAL.md — deterministic (row header, column header, cell mark) → assertion, one call per table (59 on Komatsu, fewer than the 93 the glossary branch spends to abstain 93 times), provenance at the cell; five decisions listed, T4 (what table_html shapes the parser emits) is free and comes first. TERM_HIERARCHY_PROPOSAL.md — parent/child with cross-document identity because SciCo's point is they are one problem; the dominating constraint is that hierarchy must not enter entity_id, or every correction invalidates expert approvals. Sequenced behind R4 (still not asked) and Step 0 | | K24 | The 2026-09-08 discussion list — structure, incremental ingest, review | Rifqi | ✅ | Done 2026-09-14 (pr/21, 7 commits). Struktur data 7/8: label carries a readable TYPE_SUBJECT key (keys.py) — prefixes from the kind vocabulary, rules borrow the term they govern plus a digest since a rule has no canonical surface, and the key is a label, never an identity (entity_id stays the hash approvals hang on). content/provenance/doc_ids promoted out of payload; get_by_key() + glossary_index() make lookup O(1) with no model call. m7 parked — the kind CHECK would reject 'summary'. Ingest bertahap 7/7: same doc_id + new content_hash = a new VERSION (reuse check moved to any-version, or idempotency would become a paid no-op); an approved term skips the paid call; staleness compares only the ruled-on fields (mutation.py) because comparing the payload re-opens everything on every ingest and comparing document content_hash flips untouched entries; impacted_by_document(); withdrawal marks and keeps rather than deletes. Sistem review 4/4: review_mode at ingest, decided_by + authorised_by so an LLM-mode approval can say whose decision it was, and review_diff() naming which fields moved. 7 columns added to dedorch (§6/§7 of knowledge_tables.sql, run by hand after the handoff went to Harry, verified read-only against live — Harry still needs both in the migration set). Suite 816 passed / 7 skipped (was 742), ruff clean on touched paths, import main 0. Inert until the review queue has rows: reuse and staleness both key on approved entries, and knowledge_reviews is at 0 |

Not decided, do not assume: whether the admin ingest surface is HTTP at all (the offline CLI is the honest milestone and both halves are there now); and GLiNER + LangExtract ensembling stays ⏸️ flagged as scope creep. kex was not ported — it answered the design questions; v2 was built here against this repo's patterns.

5. Critical path & sequencing

• Critical path: #22 (send Harry the report_inputs schema) ✅ resolved 2026-07-22 — now #32 (message_traceability + message_charts DDL to Harry) and #31 (non-convergent migration set). HF deploy (#13) for the playground. (#4 ✅, #21 ✅; Harry's #3 no longer blocks us — Python is getattr-tolerant.)
• Parallelizable now: #31 + #32 (both are Harry handoffs). (#4 ✅, #11 ✅, #22 ✅ done.)
• Harry-blocked / coordinated: #3 (now 🔄, blocks #4), #7 (Go endpoint), #18 (Go state ownership), #24 (contract). #25 = chat-path migration to analyses_messages — a cutover blocker.
• Demo gate (playground, #13): deploy the remove-problem_statement work to HF — slow path (#15 ✅) and the report path are verified locally, and #16 is resolved (#22 hands Harry the schema). Keep it playground-able.

6. Decisions still open (need the team / Harry / lead)

• analysis_records: dedorch-owned vs Python-owned (#16/#14). RESOLVED: Python-owned + renamed report_inputs (#21 done); Harry's migration creates it (#22).

• Whether help is its own endpoint or via call_agent (#8). RESOLVED: help is a call_agent intent (no own endpoint).

• Author display-name source for the report (#19). RESOLVED: Python resolves users.fullname (fallback user_id); swap to a Go-passed name later if preferred.

• Keep vs drop chat_history in the report contract (#17). RESOLVED: never in the contract; report is records-based (analysis_id + user_id only).

• Confirm Go takes over analysis creation + both creation gates (data-first + mandatory fields) (#18).

• Report input for edit mode — does Python need the last report content? (#24)

• report_inputs write scope — every agent call vs slow-path-only? (#21) RESOLVED: one row per slow-path run (telemetry stays Langfuse).

• Python history source — confirm Go's analysis_message (#25).

• done.chart_count — additive field on the SSE done event so the FE can skip GET /charts on chartless turns? (Harry; SPINE_V2_PLAN §4.5.) Until decided the FE fetches unconditionally.

• W3 re-open timing (deferred analyze_* activation) — Rifqi (deferred at the 2026-07-13 approval).

• INV-6 relaxation for S1b targeted repair — team, only after S1a repair_candidate telemetry shows a meaningful hit-rate (SPINE_V2_PLAN §6).

• Knowledge pipeline — v2 module name (§0.8 D5). RESOLVED 2026-08-21: src/knowledge_extraction/ and src/knowledge_parsing/; the existing src/knowledge/ OCR→pgvector service is untouched.

• Knowledge pipeline — should gliner/torch be an optional extra too? (§0.8 D1) RESOLVED 2026-09-07 (26ba1ab): no — they cannot be, and the size problem was solved another way. GLiNER is the term filter (use_span_filter=False yields zero mentions), so extraction in the Space requires torch installed; a lazily-imported extra that is absent is a 500, not a graceful skip. What was actually costly was not torch but its CUDA tail: the default PyPI Linux wheel drags the whole runtime as nvidia-* dependencies. Pinning the +cpu build from PyTorch's own index removed 36 nvidia-* packages and triton from uv.lock with zero behavioural change. mineru — the comparison this question was built on — was dropped in the same change, because it made uv lock fail project-wide.

• Knowledge pipeline — admin ingest surface (§0.8 D4): whether it is HTTP at all, and if so its auth posture. A write surface triggered by an admin is a different risk class from the current unauthenticated read-only chat surface (§0.7) — raise it, do not inherit the posture by default.