Commit History
4 days ago
-
[NOTICKET] fix(planner): EMA dry-run fixes — analysis scope, site-code lookup, in-query aggregation, NULLS LAST
Rifqi Hafizuddin +1: Claude Opus 5.5From a 9-turn replay of a real EMA session on maintenance views (Neon): - compiler: ORDER BY ... NULLS LAST (a NULL group ranked first under DESC) - the analysis objective/business questions reach the planner as a `# Analysis context` section inside <data>; today's date (WIB) in `# Question` - planner.md: aggregate in the IR, relative periods, site codes via the lookup (never a typed id, bare codes matched to lookup values), "why" uses the cause table, driver questions compare groups in the IR, rankings return 10 not 1, "infeasible is about columns, never rows"; client codes replaced by placeholders - examples: A/E aggregate in the IR, D asks for a median, P notes its row count, new X (ranking without N), Y (driver on a large table), Z (code lookup) - validator: 8i capped-slice importance, 8j typed lookup ids (code columns only), 8k top-1 rankings, check 11 rejects handoffs from metadata steps - assembler: shared top values, correct cell labels, period per row - eval/planne…a0e68ee -
[NOTICKET] feat(catalog): infer star-schema joins to same-named dimension keys
Rifqi Hafizuddin +1: Claude Opus 5.5fk_inference now also links `<base>_id` to a dimension table named with a dim_/vw_dim_/vw_/tbl_ prefix whose key has the same name (e.g. fact.site_id -> vw_dim_site.site_id), as in BUMA's EMA views. Safeguards: single unambiguous candidate only, identical data types, same-name keys refused when catalog stats show duplicates, a table's own key is never treated as a FK, fact tables sharing a key are not linked, declared FKs still win. Composite keys remain unsupported. Verified: ruff clean; 1029 passed, 7 skipped (baseline 1016/7 + 13 new local tests); import main OK. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>8c74a3d
26 days ago
-
[KM-717] eval(planner): data-dictionary cases, and live Neon knowledge tables verified
Rifqi Hafizuddin +1: Claude Opus 5.5eval.planner gains db_dict / db_dict_injected fixture variants: the DB fixture plus data-dictionary entries and two opaque measures (ua, bd_uns) whose meaning exists only in their dictionary line. Three new `dictionary` cases: - dict_meaning_join_en: pick ua, avg, join to equipment - dict_sum_by_model_id: pick bd_uns from its Indonesian business meaning, sum, join (shaped like demo step 3) - dict_injection_ignored_en: a dictionary entry orders the planner to refuse; it must plan normally (prompt-injection regression guard) The existing 40 cases keep their exact prompts. Run (--only the three): 3/3 on the first attempt, 0 retries, 59,588 tokens (planner_result_2026-09-30_093130.json + log). One sample each, no no-dictionary control. Docs: DEV_PLAN P2-I2 done; D2 closed. Live Neon knowledge tables match Go 0008 (86 columns, nullability, all 14 constraints; read-only check run by Rifqi). REPO_STATUS §9/§14, knowledge_tables.sql, KNOWLEDGE_PERSISTENCE_CONTRACT.md and eval/planner/REA…97b27d1 -
[KM-717] feat(planner): read the catalog data dictionary into the planner's catalog summary
Rifqi Hafizuddin +1: Claude Opus 5.5Go 887d98a (KM-717) stores user-entered data-dictionary fields on each catalog column. Python's Column model silently dropped them (pydantic extra="ignore"), so none reached the planner. - catalog/models.py: Column declares the eight optional fields (description, business_meaning, unit_of_measure, classify, aggregation, is_key, data_classification, validation_rules). - planner/inputs.py: CatalogSummary renders one capped (300-char) "dictionary:" line under a column that has an entry, inside the existing <data> block. Governance fields are not rendered; PII columns show the entry but still no values. - A catalog without entries renders byte-identical (eval.planner fixtures, 5,109 bytes), so committed eval results stand; planner.md unchanged. Docs: Go migration 0008 (66e70ca) adopts the five knowledge tables. REPO_STATUS §8/§9/§12/§14, DEV_PLAN D2 and P2-I1..I3, knowledge_tables.sql, KNOWLEDGE_PERSISTENCE_CONTRACT.md and docs/README.md updated. Verified: 1016 passed, 7 ski…2d70efe
-
/fix planner and unstructured pipeline (#23)
- feat(documents): v2 unstructured ingestion, image-token, scoped retrieval (b848f739c23c9ea152c5bcc5ea4b9d8b4ca64afa) - feat(prompts): mixed document+data path, figure markers (39019d57e159b510fda1f5bd813d051e1ba7c05c) - eval(chat_sim): doc + mixed suites with bound documents (b6e060b328c8dbbdb19f3da466fe97f62dab09aa) - [NOTICKET] docs: unstructured v2 plan, contract, status sync (6702119c2705ef9a99a4a0ba0adaab9ac7a28c54) - fix(check): list data and documents when both are asked (2b7c3bd79fe73b693634e583be46c812b6c74d3c) - [NOTICKET] docs: mixed suite result, check + router fixes, U8 done (0e1ad6e143e30ec20960d3a2c5ae22190aa512ff) - feat(router): a figure IN a document vs a chart FROM data (770b98855d640c727a402b0db1252638364c20d7) - fix(chatbot): reply in the turn's language, not the documents' (b6150d58b991fb5d98b0bb2278956c938d80d4c0) - fix(query,planner): stable capped rows, plan-time handoff check, trimmed records (ea9848db5ec39e32067a2c3dbc9453f1a94736d2) - eval: scoped/DB/docum…7124acf
-
/feat ds tools (#22)
- feat(tools): analytical-tools foundation — spec_builder + ml executor (ee1a76a466c6ea6dc52f337d0fd9534ca0271a86) - feat(tools): shared chart kit + auto-chart builders (535f17340950e070556441140276c47d074606e2) - feat(tools,spine): wire tool auto-charts to chart persistence (522e582bab372b645577789cbbd59d5105a66632) - build(deps): add statsmodels + explicit scikit-learn pin (4d76bd0e57f5f7b1bbb0b77e49970063d1898a1b) - [NOTICKET] docs: analytical-tools foundation — conventions, DEV_PLAN 0.9, status sync (a7f0bca51ed8ff08784cca3fd4179dce56ef7622) - [NOTICKET] eval(planner): skip cases whose tool is withheld, don't fail them (845284b4a3b32df142565de23386dee739d0f733) - [NOTICKET] feat(tools): analyze_forecast (ETS) — first ML tool, DEFERRED (dae102f68fc36cf755e7d53df10affb4ec7edaa6) - [NOTICKET] planner: rebalance few-shot exemplars EN/ID, demo the reply-language rule (fb30aec99586daf12a4799087ae538e5e862451c) - [NOTICKET] feat(tools): analyze_anomaly (Isolation Forest) — second ML tool,…e201ae4
-
/fix parsing and term extract (#21)
- [NOTICKET] fix: retire the fuzzy clustering pass; add E1c post-cluster recall (2cc21c92688d662c5cfdfeeb6784bb85b98575ca) - [NOTICKET] feat: parsing schema 0.4.0 — asset pointers for figures and tables (e662ac089b741c90e19b9f51f4933fb97a1225bc) - [NOTICKET] docs: sync REPO_STATUS / DEV_PLAN / contracts for pipeline v2 (c3c184d83bec4a4d42e9ad3f7265560802817e5b) - [NOTICKET] docs: close K15 with the paid v1-vs-v2 comparison (62782fed9292f67d7b376783920a07197d6776e9) - [NOTICKET] fix: test the caption hypothesis; ship the signal at weight 0.0 (36a166e1844d186d6cfb8fff9ea070f3dc69b220) - [NOTICKET] fix: cap key-parameter pre-emption in the review queue (Step 2) (4e3a6aefef7024db3ea4b7946fd7771a6c269601) - [NOTICKET] feat: assert on invariants, report a band, classify the genre (Steps 0 + 5) (f1d64b113db0289601806635a9b154201a6127fc) - [NOTICKET] fix: gate spend in cluster units and surface the estimate (Step 6) (cdb5fe82cc5f2893ccbb5e28282e8dc46a27e061) - [NOTICKET] fix: retire escalation…f07443e
-
/feat knowledge management (#20)
- [NOTICKET] docs: knowledge pipeline test integration — context, audited task checklist, DEV_PLAN §0.8 (8e69dc6c638509f4dd7b6775d520fb4aa4efbd3e) - [NOTICKET] docs: knowledge pipeline — reframe kex as prototype, not port target (7cce5ffc9ab88791ac139dbef8e88e124a3d049e) - [NOTICKET] feat(eval): knowledge-extraction gold set, scorer + frozen prototype baseline (60cf4a885ef31a8aa64916f5a70ac5a0e8062c5f) - [NOTICKET] chore: add missing planner and readiness eval results (bd6659b3bf523597e4482b04ab1fcff5e1284d25) - [NOTICKET] feat(knowledge_extraction): free stages — models, seam adapter, filters, cluster, ranking (024c30abd1189119d33508eded334fe5a3ed5011) - [NOTICKET] fix(knowledge_extraction): move runner into the package; fix two ranking defects (72605ff6876a7936ab685d417c766ce820f6214b) - [NOTICKET] feat(knowledge_extraction): span filter — E1 recall 0.8537, matches the frozen baseline (3a016343398753cf730d50548c0e92ff47c6716c) - [NOTICKET] feat(knowledge_extraction): paid extraction …b68816f