Commit History
-
eval: mixed T3 re-checked 3x with full records; label calls by output
Rifqi Hafizuddin +1: Claude Opus 5.5T3 ("according to the standard...") x3 via --turns 3: 2/3 plans add retrieve_knowledge, 1/3 answer from the approved domain card (stated in its assumptions); identical correct site figures; no retries. Retrieve-then-plan kept nothing in 3/3 (EN question vs ID standard: 0.584-0.643 > 0.5 cut). chat_sim labelled LLM calls by order, so the input-guard call shifted every label by one; labels now come from each call's output fields. The three new transcripts were relabelled from their saved outputs (outputs untouched). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>1cd7221 -
eval: keep full plans, every attempt and the run log for paid runs
Rifqi Hafizuddin +1: Claude Opus 5.5eval.planner saves each case's full TaskList and every attempt the model returned with the error that rejected it; chat_sim saves every LLM call's full output (each planner attempt included) and each retrieval's query + cosine distances. Both tee their console output to results/logs/ under the result's timestamp. Adds the rescued logs of the 2026-09-23 D/E runs (planner 155532 + 184105, mixed 160605). No LLM behaviour change; 0 tokens. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>5959da8 -
[NOTICKET] docs: pr/23 part 2 D + E build record, traceability contract note
Rifqi Hafizuddin +1: Claude Opus 5.5Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>153b12f -
feat(planner): retrieve-then-plan, per-turn recipes, leaner prompt
Rifqi Hafizuddin +1: Claude Opus 5.5E1: bound documents are retrieved before planning (top-3, cosine < 0.5) and passed to the planner and assembler as a <data> Document context block; never-throw; planner span Langfuse-masked when present (E2). D1 question last; D2 recipe table + close pairs rendered per turn for visible tools only (planner/recipes.py, verbatim); D3 joins rule only with a database source; D5 compact exemplar JSON. D4 (reorder) tried and reverted. planner.md 6,299 -> 5,182 tokens; per-case prompt -17%. eval.planner final config 39/40, carried_over 22/22. chat_sim evidence recorder now captures table-kind analyze_* values. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>c089593 -
feat(router): intents defined once, 21 few-shots, sharper group_hint groups
Rifqi Hafizuddin +1: Claude Opus 5.5C1: drop the restating Routing rules section. C2: 11 definition-restating few-shots removed (kept as eval rows); explore/grouping/evaluate sharpened. eval.intent 57/57 both runs; group_hint 12 exact / 3 superset / 0 wrong; router prompt 3,484 -> 3,164 tokens. Scorer gains a superset bucket. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>1dfbf6d -
eval: scoped/DB/document planner cases, group_hint scoring, faithfulness, cache
Rifqi Hafizuddin +1: Claude Opus 5.5PR23 part 2 section A. - eval.planner: per-case group_hint (scoped like live), a DB fixture with an FK (has_join), document-context cases (filter_value; skipped until the planner accepts document_context), cache_read logging, --only. - eval.intent: expected_group_hint on the 15 structured cases, scored exact / absent / wrong / stray; cache_read logging. - chat_sim: each turn records bounded tool/document evidence. - eval/faithfulness: deterministic check that answer numbers appear in that evidence (ID + EN formats, ratio<->percent). - eval/tools baseline: 35/35 cases, 14/14 tools. Smokes: planner 2/2 (scoped 15.4k vs ~24k unscoped; join emitted), router subset 5/5 intent + 5/5 group_hint. Prompt caching measured live: planner 6,144 cached tokens on the 2nd call, router 73%. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>bff85a5 -
fix(chatbot): reply in the turn's language, not the documents'
Rifqi Hafizuddin +1: Claude Opus 5.5An English question about an Indonesian document was answered in Indonesian. detect_reply_language already returned English; the model followed the Indonesian excerpts, and the rule only warned about English context pulling the reply. The rule is now language-neutral: this turn's language wins (explicit "answer in X" still overrides). Doc smoke (turns 2,4): ID -> ID, EN -> EN (transcript_doc_2026-09-23_141212.json, ~14k). 65 related tests pass. Docs: DEV_PLAN §6 (c), REPO_STATUS §9, plan §4.4. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>b6150d5 -
[NOTICKET] docs: mixed suite result, check + router fixes, U8 done
Rifqi Hafizuddin +1: Claude Opus 5.5- UNSTRUCTURED_V2_PLAN §4.5: the mixed run (T2/T3 one plan with document + data, T4 numbers-only without the document, T1 check bug -> fixed) and the router fix with its phrasing-rule caveat. - DEV_PLAN §0.10 U8 done, U9 drafted; §6 follow-ups (a) and (d) resolved. - REPO_STATUS §9: router + check behaviour, known limits. - transcript_mixed_2026-09-23_132033.json (run by Rifqi after the catalog rebuild). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>0e1ad6e -
feat(router): a figure IN a document vs a chart FROM data
Rifqi Hafizuddin +1: Claude Opus 5.5"Show me the waterfall chart …" routed to structured_flow and was refused (U8 doc suite). One disambiguation bullet + a contrastive few-shot pair: a chart/figure that already exists in a document -> unstructured_flow (looked up), a chart computed from the user's data -> structured_flow (drawn). eval.intent 57/57 (eval_result_2026-09-23_132736.json, 215k) vs 52/52 at 2026-09-17_113055: 2 mirror rows for the new few-shots + 3 distinct test rows. A chart ask that names no document still routes to structured_flow — correct under the dataset's phrasing rule (the router cannot see the catalog). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>770b988 -
eval(chat_sim): doc + mixed suites with bound documents
Rifqi Hafizuddin +1: Claude Opus 5.5UNSTRUCTURED_V2_PLAN U8. Suite.document_ids binds v2-ingested documents in the shape Go's buildBoundSource materialises; unowned or un-indexed documents are refused. doc suite (BUMA standard): 4/4 after rephrasing turn 3 — a figure answer carried asset://4f72f8b170d1787b (Gambar 2.1), resolved via image-token to 200 image/jpeg. mixed suite is defined but not run: dedorch data_catalog has 0 rows, so padb cannot be bound. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>b6e060b -
feat(prompts): mixed document+data path, figure markers
Rifqi Hafizuddin +1: Claude Opus 5.5UNSTRUCTURED_V2_PLAN U6 + U7. No new intent; router untouched. - planner.md: mixing bullet says when to add retrieve_knowledge (the question also asks what a document says) and when not; its output is never data. - assembler.md rule 6: document excerpts are context, a number in one is the document's statement, never a computed result. - chatbot_system.md rule 6: copy a relevant asset:// marker exactly. - eval/planner: 2 cases with a per-case `documents` list (other prompts unchanged) + --only. eval.planner 32/33, carried_over 22/22 (planner_result_2026-09-23_113854.json, 864k) vs 31/31 at 2026-09-21_145101; the one fail (rank_units_most_breakdown_id, not carried_over) passed 3/3 on re-run = known variance. Gate smoke 2/2. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>39019d5 -
/feat ds tools (#22)
- feat(tools): analytical-tools foundation — spec_builder + ml executor (ee1a76a466c6ea6dc52f337d0fd9534ca0271a86) - feat(tools): shared chart kit + auto-chart builders (535f17340950e070556441140276c47d074606e2) - feat(tools,spine): wire tool auto-charts to chart persistence (522e582bab372b645577789cbbd59d5105a66632) - build(deps): add statsmodels + explicit scikit-learn pin (4d76bd0e57f5f7b1bbb0b77e49970063d1898a1b) - [NOTICKET] docs: analytical-tools foundation — conventions, DEV_PLAN 0.9, status sync (a7f0bca51ed8ff08784cca3fd4179dce56ef7622) - [NOTICKET] eval(planner): skip cases whose tool is withheld, don't fail them (845284b4a3b32df142565de23386dee739d0f733) - [NOTICKET] feat(tools): analyze_forecast (ETS) — first ML tool, DEFERRED (dae102f68fc36cf755e7d53df10affb4ec7edaa6) - [NOTICKET] planner: rebalance few-shot exemplars EN/ID, demo the reply-language rule (fb30aec99586daf12a4799087ae538e5e862451c) - [NOTICKET] feat(tools): analyze_anomaly (Isolation Forest) — second ML tool,…e201ae4
-
/fix parsing and term extract (#21)
- [NOTICKET] fix: retire the fuzzy clustering pass; add E1c post-cluster recall (2cc21c92688d662c5cfdfeeb6784bb85b98575ca) - [NOTICKET] feat: parsing schema 0.4.0 — asset pointers for figures and tables (e662ac089b741c90e19b9f51f4933fb97a1225bc) - [NOTICKET] docs: sync REPO_STATUS / DEV_PLAN / contracts for pipeline v2 (c3c184d83bec4a4d42e9ad3f7265560802817e5b) - [NOTICKET] docs: close K15 with the paid v1-vs-v2 comparison (62782fed9292f67d7b376783920a07197d6776e9) - [NOTICKET] fix: test the caption hypothesis; ship the signal at weight 0.0 (36a166e1844d186d6cfb8fff9ea070f3dc69b220) - [NOTICKET] fix: cap key-parameter pre-emption in the review queue (Step 2) (4e3a6aefef7024db3ea4b7946fd7771a6c269601) - [NOTICKET] feat: assert on invariants, report a band, classify the genre (Steps 0 + 5) (f1d64b113db0289601806635a9b154201a6127fc) - [NOTICKET] fix: gate spend in cluster units and surface the estimate (Step 6) (cdb5fe82cc5f2893ccbb5e28282e8dc46a27e061) - [NOTICKET] fix: retire escalation…f07443e
-
/feat knowledge management (#20)
- [NOTICKET] docs: knowledge pipeline test integration — context, audited task checklist, DEV_PLAN §0.8 (8e69dc6c638509f4dd7b6775d520fb4aa4efbd3e) - [NOTICKET] docs: knowledge pipeline — reframe kex as prototype, not port target (7cce5ffc9ab88791ac139dbef8e88e124a3d049e) - [NOTICKET] feat(eval): knowledge-extraction gold set, scorer + frozen prototype baseline (60cf4a885ef31a8aa64916f5a70ac5a0e8062c5f) - [NOTICKET] chore: add missing planner and readiness eval results (bd6659b3bf523597e4482b04ab1fcff5e1284d25) - [NOTICKET] feat(knowledge_extraction): free stages — models, seam adapter, filters, cluster, ranking (024c30abd1189119d33508eded334fe5a3ed5011) - [NOTICKET] fix(knowledge_extraction): move runner into the package; fix two ranking defects (72605ff6876a7936ab685d417c766ce820f6214b) - [NOTICKET] feat(knowledge_extraction): span filter — E1 recall 0.8537, matches the frozen baseline (3a016343398753cf730d50548c0e92ff47c6716c) - [NOTICKET] feat(knowledge_extraction): paid extraction …b68816f
-
/fix planner count and report (#19)
- /fix planner count question (11de970ff65be9b4f81f773cfb355d3dc57358d7) - [noticket] fix report bug where some answered analysis isnt included in report summary answers (8dec1825cf803ad28f4ff0400b9c373250a7d90a) - [NOTICKET] fix: split report floor from report body; scope analysis reads by user_id (fd4865b20dfc46782706be9aab318f909b9903c3) - [NOTICKET] docs: sync report floor/body + security posture; add review remediation tracker (fc5200c994441ec505ea7a82469da577afe170a5) - [NOTICKET] test: planner eval harness (rule-compliance, --selfcheck/--rescore) (4292522aa3fa3a7af8f933307334cb7e5eb0b377) - [NOTICKET] eval: repair readiness harness; commit intent/help/readiness baselines (08a0a4ad5eacc6c90df22fab5be69f2d33579f29) - [NOTICKET] fix: compiler parity — F-17 fabricated column, F-18 LIKE nulls, F-25 empty in/not_in (8f736f90f1958960a83847d332b45b13fa346168) - [NOTICKET] feat: bound catalog render + blob read; scope charts/traceability; mark degraded seams (0cb7d53cb6f7f7e7b687dd6d9c27…f282b15
-
/fix change model to gpt 5.4 mini (#17)
- [NOTICKET] chore: rename LLM env quad __4o -> __54m (GPT-5.4-mini) (790ef0c7aaac6e76eb01b0ef9a3398ff4249416d)d65c41d -
fix/ check and new chart tool (#16)
- check: table-level column drill + drop misleading "name a table" hint (ef0c7b1c5aa49a244b02f1b5eea15cc77dcf2bcb) - [KM-715] charts: render_chart tool + message_charts store + GET /api/v1/charts + planner viz slice (87cfcf884d923ee4bcd722e0e4ef4d90d7eaedca) - [KM-715] checkpoint: S1a deterministic quality checkpoint between runner and assembler (f3c8de34dc19a573031b67fd4f48dcfbadcdfed8) - [KM-715] docs: sync REPO_STATUS/DEV_PLAN + land SPINE_V2_PLAN (W1/W2 done) (434eacbf2ac8324188ebe5e32775dbc9d2a3f49a) - planner: value-handoff (${tN.col} → list) for tabular lookup/anti-join (54070ac1407b751000e49dfe7f102a2be5ab1ff8) - [KM-715] charts: GET /charts by message_id only + tri-state status marker (b8d15761addf4d499c9631df64e7693aa1d61fa2) - [KM-715] report: embed charts as plotly EDA fences; render_chart counts toward the floor (89db8be6b6f11e86ad0776d2ee07f2306e3b6079) - [KM-715] docs: contract + status deltas for the charts lead-review round (96347ee7c55a4eb39a906409ec214463e25c5e35) - …5a60e93
-
/fix help and tools analyze (#14)
- [KM-711] Retire /report slash command; report is Generate-button only (f7f7b172da0dc16adac0a7555f6d34a704cc2050) - fix(analyze_trend): handle value_column == date_column (count events per period) (55e3327f7b2e21a954b317e143f46bcea8b5ada7) - [KM-711] Docs: drop /report from list_skills catalog examples (0772863b296e2ed3e2d559df05e98540764e58b0) - fix(planner): reject inlined table-ref as analyze_* data (Pattern A) (776fae3c7c6e3d3a07c22bef10898f6b1c33c701) - fix(analyze_aggregate): reject non-string group_by (CASE/binning) keys (b40fa76e2fa68572dad0d8532e07076207095d6b) - fix(planner): reply-language for infeasible_reason + answer-only-asked rule (4d89d7d93938e41892979cd20e5ccf0c97441dac) Co-authored-by: Sofhia Az-Zahra <sofhiaazzhr@users.noreply.huggingface.co>17e31c6
-
/fix validator and report (#10)
- /fix report: remove record_id hash citations, EDA (for future tool) change to data source, transform to mdx-safe (b1b101730ccf5628e43e80f2ad654f78fbed5207) - [NOTICKET] fix(planner): repair near-miss catalog ids in QueryIR before validation (aacd4d13def258a9815761aecbd7b246a8dae9fc) - /fix report: group findings by analysis, dedup, code-span-safe MDX escaping (029b48894e9f79af6c0d81c00a2d811a982f87ae) - [NOTICKET] fix(tools): log swallowed analytics tool errors in AnalyticsToolInvoker (93d9381b2a5f5efe058c913fe0b793ce994a9097) - fix(planner): harden plan validation to reduce retry thrash on structured questions (1619dd3c0fa74825010c7af6d4b375e3b9404847) - [NOTICKET] fix(tools): coerce numeric-string columns before analyze_* compute (6bda333866a1a8ccf1b3fe6fcee7a8e1832f28d7) - fix(planner): correct top-N guidance — in-query ranking IS supported (99aa4141aed8241a625d290fb170f74192a6d831) - feat(guardrails): add input guard, out_of_scope intent, and refusal handling (0bbc27bc40ba20e7a8e…0e5fdb5
-
/fix check and help tool (#8)
- /fix data_catalog model + CatalogStore to dedorch schema (data → catalog_payload) (09857fc52486354f53e752e3ea329b064a9e4c4b) - [NOTICKET] fix(help): lock reply language to user's language on button path (cb7f5ac484ad4a510e68faf5291091958b6d2f42) - /fix default SKIP_INIT_DB=true for dedorch cutover (skip init_db on Go-owned schema) (781b93c6b8744209a1101629b5170175e9212592) - /fix surface catalog foreign keys to the planner (FK-backed joins) (446c57524e932fcdbb0ddecc049189c443b6ae97) - [NOTICKET] feat(eval): add help-skill eval harness + goal-based language fallback (cbdbefb7ab0ffa36949713f18422213db480a959) - [NOTICKET] fix(eval): update readiness eval to post-KM-652 floor (drop problem_validated) (6f46e81bdeb1eb987f9dac8af0e22a975474cbb1) - /fix infer FKs for FK-less dedorch catalog + log repr(e) in db executor (3669290474583ae011f5ed5e50b94dad146fbc77) - Merge eval/help + readiness eval fix into PR #8 (5f4818a8e38f93988ab5756f3a06c94645826b15) - /docs REPO_STATUS: FK inference (FK-…3743cfe
-
feat/Knowledge & Data Tools (#3)
- [KM-641] refactor(tools): trim analytics registry to 4 active tools (66e2e4df66f1abf7eaa7e442d9900d428696ece4) - [NOTICKET] docs: add checkpoint plan for 2026-06-17 sync (8c1bb63ebdee02f9db0da9fc2b2269823b0a7688) - [KM-642] refactor(tools): rename data-access tools to data/knowledge naming (c38c0c29d3e6b245f67d03f2c35e451900e1abdb) - [KM-643] feat(tools): merge data sources into data_check + add knowledge_check (4bd5f1efdc560a64092cc8df672a7990fa949b59) - [NOTICKET] docs: record KM-642/643 in checkpoint plan + PROGRESS (4c57588b90fe0cac8d0d4d22cfe5701fc9eb52dd) - [NOTICKET] fix(chat): scope response cache by user_id + drop TTL to 1h (R5) (b701e95d300c7d2480a3608b2d4da6dfe102b19a) - [NOTICKET] feat(slow-path): persist AnalysisRecord via PostgresAnalysisStore (09f845188226dd01633cf74525caa569e4a7d37a) - [NOTICKET] Adopt verb-first skill naming (2d6406def7ed45edd1226e96645ba2a5cdbfa3d0) - [NOTICKET] docs: sync checkpoint plan — verb-first tool names + report-as-API (3b399f84d73b0d98bd99…0721bb4