Commit History
-
[NOTICKET][AI] feat(chat): wire Orchestrator -> slow path at the endpoint (env-gated)
Rifqi Hafizuddin +1: Claude Opus 4.8The slow path (Planner -> TaskRunner -> Assembler) was reachable in ChatHandler but never enabled from the live endpoint. Wire it so the team can test end-to-end from /chat/stream, behind an env flag so it stays opt-in until the lead's real BusinessContext lands (it's still a stub). - settings.enable_slow_path (env ENABLE_SLOW_PATH, default False). - api/v1/chat.py passes it to the shared ChatHandler. - ENABLE_SLOW_PATH=true → structured intents route through the slow path and stream status progress + the assembled answer; default keeps the single-query QueryService path. Fast/unstructured paths unchanged. Verified: toggle flips handler._enable_slow_path; wiring tests pass; live run through ChatHandler.handle confirmed earlier this session. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>636290e -
[NOTICKET][AI] refactor(planner): R11 - single-source placeholder regex + name-checked data-access specs
Rifqi Hafizuddin +1: Claude Opus 4.8Comments are not a sync mechanism. Remove the two duplicated definitions on the agent side. - PLACEHOLDER_RE: defined once in planner/schemas.py (it's part of the ToolCall ${t<id>} placeholder convention); planner/validator.py and slow_path/task_runner.py now import it instead of each compiling their own copy "kept in sync" by comment. - planner/registry.py still holds local data-access spec BODIES (stub pending KM-465 #4) but derives/validates their NAMES against the canonical src.tools.data_access.DATA_ACCESS_TOOLS: _data_access_slice() raises at default_registry() if a tool is added/renamed/removed upstream, instead of drifting silently. Registry output unchanged (same 12 tools, same order). Tests: 51 passed (planner + slow_path); ruff clean; registry builds. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>0f2b85e -
[NOTICKET] refactor(tools): single source of truth for data-access tool names + clarify input_schema is presence-only
R11 (TAB slice): the data-access tool-name set was duplicated between invoker.py and planner/registry.py, synced only by a keep-in-lockstep comment. A 5th/renamed tool added in one place but not the other would silently mis-route (registry knows it, CompositeToolInvoker rejects it). Define canonical DATA_ACCESS_TOOLS once in tools/data_access.py; invoker.py imports it. (planner/registry.py + _PLACEHOLDER_RE dedup are the agent-side remainder.) T1: ToolSpec.input_schema comment said "validates ToolCall.args" but TaskRunner._validate_args is presence-only; properties types are documentation, not enforced. Clarified so nobody assumes type-safety. Doc/comment + a constant move; no behavior change. 41 tool tests pass, ruff clean. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>90e80f9 -
[NOTICKET] perf(slow-path): DB3 pre-connect + R4 progress events
Rifqi Hafizuddin +1: Claude Opus 4.8DB3 (DB owner): DbExecutor.prewarm(catalog, user_id) warms the pooled engine for schema sources, fired (best-effort, never raises) at slow-path entry so the cold first-query TCP+TLS+auth handshake overlaps the ~4s Planner LLM call. Gated to the default path (skipped when a coordinator factory is injected, so tests stay hermetic). Reuses DbExecutor's existing client resolution + the DB2 engine cache. R4 (agent): SlowPathCoordinator.run gained an optional `progress` callback; ChatHandler bridges it to SSE `status` events via an asyncio queue, and chat.py forwards them. The slow path no longer streams ~13s of silence — live timeline now shows Planning -> Running N steps -> Composing, max wire gap ~4.6s, fixing proxy idle-timeout + UX. Status events only appear when the coordinator calls back, so the existing wiring test's event sequence is unchanged (fake coordinator just accepts **kwargs). Token-streaming the Assembler answer is deferred: it would require splitting the Assembler into a …735c421 -
[NOTICKET][DB] perf(query): DB2 - pooled engine cache for user DBs
Rifqi Hafizuddin +1: Claude Opus 4.8DbExecutor built a fresh engine and disposed it on every query, paying a full TCP+TLS+auth handshake per call (~6-8s measured, dominating slow-path latency). engine_scope's connect-once-dispose is right for ingestion, wrong for the query path. - New UserEngineCache (src/database_client/engine.py): process-wide cache of pooled engines keyed by client_id + a hash of the decrypted creds (rotation auto- invalidates the key). Bounded LRU (50) + 600s idle TTL; pool_pre_ping + pool_recycle=300; small pool (1 + 2 overflow). invalidate(client_id) disposes eagerly. Scope: postgres/supabase only; other db_types keep the legacy per-call path, so nothing regresses. - DbExecutor._run_sync reuses the warm connection (no per-query SET, no dispose). - Read-only + statement_timeout moved from libpq startup `options` to a per- connection `connect` event: Neon's transaction pooler rejects default_transaction_read_only as a startup parameter (caught in a live run) but accepts it as a SET. …4c911cb -
[NOTICKET][DB] fix(query): R2 - always compile a bounded LIMIT
Rifqi Hafizuddin +1: Claude Opus 4.8An IR with no limit compiled to a LIMIT-less SELECT, and DbExecutor materialized the entire result set before capping at 10k rows - an unbounded fetch that can OOM on a large user table. - SqlCompiler now bounds every query: an explicit IR limit is honored (clamped to MAX_RESULT_ROWS=10000); an unbounded query gets LIMIT cap+1 so the executor can distinguish exactly-the-cap from more-rows-existed and flag truncation. - CompiledSql.row_cap carries the effective cap; DbExecutor caps + derives truncated from it and drops its own _ROW_HARD_CAP constant. - test_sql.py: no-limit golden strings get the safety bound; +3 cases (no-limit bounded, explicit-limit clamp, row_cap). Restored S608 to tests/** ruff ignore (documented in PR3-DB but missing from pyproject) - golden tests assert literal SQL. Also logs R13: a pre-existing, unrelated test_prompt failure found while running R2. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>6974c5b -
[NOTICKET][AI] perf(chat): PR-DB1 - memoizing catalog reader + shared ChatHandler
Rifqi Hafizuddin +1: Claude Opus 4.8Cut redundant catalog-DB round-trips in the slow path and stop rebuilding the Azure LLM clients per request. - MemoizingCatalogReader (catalog/reader.py): request-scoped wrapper that caches each read by source_hint. ChatHandler builds one per request and threads it into the slow-path DataAccessToolInvoker, so the planner's catalog load is reused by describe_source / query_structured instead of re-reading from the catalog DB. Measured live: describe_source 3.5s -> ~2.0s; catalog reads/request ~5 -> ~2. - chat.py: one module-level ChatHandler singleton instead of per-request, keeping the Orchestrator/Chatbot Azure clients (and TLS pools) warm across requests. Does not touch the external query_structured handshake (that is the engine-cache work, DB2). Also logs the principal-review fix tracker (R1-R12, DB1-DB3) in PROGRESS. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>4c62146 -
[NOTICKET][AI] Planner: drop parallelizable_with from Task
Rifqi Hafizuddin +1: Claude Opus 4.8Remove the unused parallelizable_with field and its validator check (5). TaskRunner derives parallelism from depends_on alone, so the field was redundant. Strips it from schema, validator, all few-shots (A-D), and the planner prompt. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>272a186 -
[NOTICKET] refactor(catalog): single SAMPLE_LIMIT constant shared across introspection paths
Hoist the per-column sample cap into catalog.introspect.base.SAMPLE_LIMIT so the tabular and DB introspection paths reference one source of truth instead of two separate literals (tabular head(3) + extractor SAMPLE_LIMIT=3) that can drift. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>6d46ba5
-
[KM-626][AI] Planner few-shot: add analyze_aggregate (Example D) to fix arg shape
Rifqi Hafizuddin +1: Claude Opus 4.8Live test surfaced an agent-side bug: for group-by questions the Planner emitted analyze_aggregate args as a nested list of metric specs, but the tool expects `aggregations` as an OBJECT {column: [funcs]} with `group_by` as a SEPARATE array. The tool then did aggregations.keys() on a list -> AttributeError. There was no few-shot for analyze_aggregate, so the LLM guessed the wrong structure. Add Example D ("average and total order value per region") demonstrating the exact shape: aggregations={"revenue": ["mean","sum"]}, group_by=["region"]. Verified live: the same question that failed now plans the correct shape ({"base_price": ["mean"]}, group_by=["category"]) and analyze_aggregate succeeds end-to-end. Planner suite green (32 passed). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>f07e524 -
[KM-631][AI] Langfuse tracing: tokens + latency across the chat pipeline
Rifqi Hafizuddin +1: Claude Opus 4.8Wire Langfuse (was scaffolded-only) so each chat request emits one trace with per-call token counts + latency, plus tool spans. Gated and PII-aware. - New src/observability/langfuse/tracing.py: RequestTracer (one trace/request), NullTracer (no-op), TracingToolInvoker (metadata-only span per tool call), _redact MaskFunction. Best-effort: never raises; pipeline unchanged if Langfuse is down or disabled. - Each LLM call (Orchestrator, Planner incl. retries, Assembler, Chatbot) gained an optional `callbacks` param, passed to the LangChain call only when present (so existing fakes/tests are untouched). Coordinator threads planner/assembler callbacks through. - ChatHandler: enable_tracing flag (default OFF so tests never hit Langfuse), creates one trace per request, wraps the tool invoker for spans, ends the trace. - PII policy for Cloud: UNMASKED = Orchestrator + Planner (PII-safe inputs); MASKED (tokens+latency only) = Assembler + Chatbot (see real rows / doc chunks); to…8cb3714 -
fix(tools): align analyze_descriptive required args + normalize Timestamp in aggregate
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>4bb7623 -
[KM-624] Compute Functions: fix non-JSON-safe scalars in mode & top_value
Two latent bugs in the same class as the Decimal crash: numpy/pandas scalars leaked into tool output and would break JSON serialization (SSE / persistence). - analyze_descriptive: `mode` of an integer column returned numpy.int64 (not JSON-serializable); a datetime column returned a pandas.Timestamp. Added a _clean pass (int64 -> int, Timestamp -> ISO string). - analyze_profile: `top_value` of a datetime column returned a pandas.Timestamp that failed json.dumps. Hardened _clean to coerce Timestamp -> ISO string. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>6981ed3 -
[KM-630] Data-Access Tools: coerce DB Decimal to float
DB NUMERIC columns arrive as decimal.Decimal (asyncpg). This broke analyze_contribution with a `float + Decimal` TypeError, and the same Decimal would also break JSON serialization (SSE / analysis_record) of query_structured output. - query_structured: _json_safe coerces Decimal -> float when building ToolOutput.rows, so the output is JSON-safe at the source. - _materialize: _coerce_decimals normalizes Decimal object-columns to float64 so the whole analyze_* family receives consistent float input. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>1195870 -
[KM-626][AI] Planner few-shot: make describe_source success_criteria checkable
Rifqi Hafizuddin +1: Claude Opus 4.8Live end-to-end test against real user data exposed a Planner robustness bug: the validator requires success_criteria to contain a measurable signal word, but the three describe_source examples modeled "describe_source returns the orders table..." (no signal word). The LLM imitated that phrasing and failed its own validation (3 retries exhausted). Reworded to "Produced the orders table schema..." so the examples are consistent with the validator. After the fix the same question plans in one shot (no retries). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>f87e5ec -
[KM-626][AI] Slow-path wiring prep: context + persistence seams, retrieve_documents few-shot
Rifqi Hafizuddin +1: Claude Opus 4.8Three small steps that de-couple the live wiring from the remaining blockers so flipping enable_slow_path on later is a one-liner. - BusinessContext seam: new get_business_context(user_id) (planner/business_context.py) is the single place context is read — returns a stub today, swap the body when the lead's real source lands. ChatHandler now reads through it. - AnalysisStore seam: new slow_path/store.py (Protocol + NullAnalysisStore no-op that logs only). ChatHandler persists the analysis_record through it after streaming (never breaks the answer on failure). Real Postgres-backed store in the catalog DB (Neon dataeyond) is a TODO — table not created yet. - Planner few-shot: add Example C (retrieve_documents) — a mixed structured + unstructured plan showing an independent document-retrieval branch (NL query, no ${t} placeholder, runs in parallel). The Planner had no retrieve_documents example. Verified: slow_path + planner suites green (52 passed); live RUN_PLANNER_EVAL …3b27b6b -
[KM-626][AI] ChatHandler: gated structured->slow-path wiring
Rifqi Hafizuddin +1: Claude Opus 4.8Add an off-by-default slow analytical path to ChatHandler. When enable_slow_path=True, a `structured` intent routes to SlowPathCoordinator (Planner -> TaskRunner -> Assembler) instead of the single-query QueryService path, streaming the assembled chat_answer as SSE. - New keyword-only ctor params: enable_slow_path (default False) and an injectable slow_path_coordinator_factory for tests. - _get_slow_path_coordinator is the composition root: builds a per-request CompositeToolInvoker (data-access needs user_id + CatalogReader) so the slow_path/ package stays tool-agnostic (INV-7). - Flag defaults off and chat.py constructs ChatHandler() unchanged, so live behavior is identical. Gated on two stubs before flipping on: _stub_business_context (-> lead's real BusinessContext) and analysis_record persistence (no store yet). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>50df057 -
[KM-630] Data-Access Tools: review fixes
- describe_source: add table_row_count column (the registry tool description advertises row counts; output now matches the contract). - retrieve_documents: coerce top_k to int with a fallback to 5, so a non-int top_k from the Planner can't reach the retriever unguarded. - Refresh the module docstring to list all four tools (query_structured and retrieve_documents were missing). Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>37b8384 -
[KM-630] Data-Access Tools
DataAccessToolInvoker (src/tools/data_access.py) — never-throwing invoker for the data-access family, constructed per-request with the authenticated user_id and a CatalogReader (dependency injection; INV-7 keeps the agent layer tool-agnostic). Implements all four tools: - list_sources — user's data sources (id, name, type, table count). - describe_source — tables/columns of one source (metadata only; exposes pii_flag, never sample_values). - query_structured — runs a pre-built QueryIR (validate -> dispatch -> execute, skipping the planner) and returns ToolOutput(kind="table"); this is the Pattern A handoff the analyze_* tools consume. - retrieve_documents — dense retrieval over unstructured sources; optional source_id is a best-effort metadata post-filter (see TODO). CompositeToolInvoker (src/tools/invoker.py) — one invoke(tool_name, args) dispatching the whole tool surface: routes the four data-a…148e33b
-
[KM-626][AI] Planner: consume teammate's real analytics registry
Rifqi Hafizuddin +1: Claude Opus 4.8After the tool team landed the canonical contracts + real analytics registry (KM-627/628), reconcile our planner-facing registry to the real thing instead of the stub: - default_registry() now composes the REAL analytics slice (src/tools/registry.py ::analytics_registry()) + a local stub for only the 4 data-access tools (still pending KM-465 #4). - Drops our 8 analyze_* stub entries; removes the one drift (their analyze_descriptive requires only ["data"], column_ids optional) and adopts the real prompt-style tool descriptions. - Pattern A is now confirmed (KM-465); no agent-code change needed (INV-7). Verified: 44 planner+slow_path tests green, ruff clean. No changes to src/tools (tool team owned). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>c08395e -
[KM-629] AnalyticsToolInvoker (wrapper layer)
Implement the ToolInvoker Protocol (src/agents/slow_path/invoker.py) for the analyze_* family: invoke(tool_name, args) -> ToolOutput. - Dispatch tool_name -> (compute fn, output_kind); unknown name -> error envelope. - Materialize the Pattern A `data` arg (already resolved by the TaskRunner to the upstream ToolOutput kind="table") into a DataFrame; also accepts a raw DataFrame or {columns, rows} dict. - Pass remaining args as kwargs (names match compute signatures 1:1); wrap the result in ToolOutput with the tool's declared kind. - Never-throw: unknown tool, bad/missing data, or a compute exception all return ToolOutput(kind="error"), honoring the §8.4 seam guarantee. Verified locally (tests are gitignored): 10 invoker tests + full tools suite 82 passed; ruff + mypy strict clean. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>421cf11 -
[KM-628] Analytics tool registry
Real registry of the 8 analyze_* family tools (src/tools/registry.py), built on the canonical ToolSpec (KM-627) and the prompt-style DESCRIPTION constants (KM-625). Replaces the analytics slice of the agent team's stub registry. - Pattern A: each tool takes data="${t<id>}" placeholder (no self-fetch). - input_schema.required mirrors each compute fn's no-default params; descriptive relaxes column_ids to optional (compute treats None = all columns). - output_kind per tool; analyze_comparison = stats. The 4 data-access tools + default_registry() composition land with KM-465 #4. Verified: registry builds (8 tools), ruff + mypy strict clean. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>4abb099 -
[KM-627] Canonical ToolSpec + ToolOutput
Move ToolSpec/ToolRegistry/ToolOutput to src/tools/contracts.py as the single source of truth (owned by the tool team, KM-465). planner/contracts.py now re-exports them so the planner/slow_path importers are unchanged; shapes are identical to the prior stubs (zero field change). Records the data-flow decision: Pattern A (analyze_* take data="${t<id>}" placeholder, not self-fetch by source_id) and analyze_comparison.output_kind = "stats", aligning with the planner registry. Verified: imports OK, ruff + mypy strict clean, 72 analytics tests green. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>0cb4e6d -
[KM-625] Prompt-style Tool Descriptions
Add a DESCRIPTION constant to each of the 8 analytics compute tools, written prompt-style (Summary / USE WHEN / DON'T USE WHEN -> other tool / example questions) for the Planner to decide which tool to pick. English body with Indonesian trigger-word equivalents in parentheses. comparison also documents the group_a baseline / comparison-direction rule. Final destination is ToolSpec.description once the wrapper layer is built. No behavior change; 72 tests, ruff, and mypy strict all green.db217d6 -
[KM-626][AI] Slow-path agent: Assembler + Coordinator
Rifqi Hafizuddin +1: Claude Opus 4.8The write-up half of the slow path (AGENT_ARCHITECTURE_CONTEXT_new.md §7.5, §8.3). - assembler.py + prompt.py + config/prompts/assembler.md: single LLM call -> AssemblerNarrative; code merges it with the RunState to build the AnalysisRecord (results_snapshot / tasks_run / metadata copied from RunState, never re-authored by the model, so the record stays a faithful source of truth). chat_answer is the first output field so it streams via SSE. Chain construction mirrors the planner service. - coordinator.py: SlowPathCoordinator wires Planner -> TaskRunner -> Assembler. Built and tested, but NOT yet wired into the live ChatHandler. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>e4337a8 -
[KM-626][AI] Slow-path agent: seam contracts + TaskRunner
Rifqi Hafizuddin +1: Claude Opus 4.8The execution half of the slow path (AGENT_ARCHITECTURE_CONTEXT_new.md §7.4, §8.2/§8.4). Deterministic, 0 LLM, tool-agnostic (INV-7). - schemas.py: TaskResult, RunState (the blackboard); TaskSummary, AnalysisRecord, AssembledOutput, AssemblerNarrative. Reuses planner ToolOutput. - invoker.py: ToolInvoker Protocol only -- the runtime seam the runner calls; the tool team owns the implementation (KM-418). - errors.py: SlowPathError, AssemblerError. - task_runner.py: wave-based dependency execution (asyncio.gather), ${t<id>} placeholder resolution, internal validate_args, never-throw invoke, status labeling, degrade-and-continue. No replanning, no mid-run LLM. Lives in agents/slow_path/ -- NOT "orchestrator" (that name is the entry dispatcher in agents/orchestration.py). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>ac310de -
[KM-624][AI] Planner: realign stub registry + examples to composite analyze_* tools
Rifqi Hafizuddin +1: Claude Opus 4.8Team decision: v1 uses composite "family" tools (analyze_*), not the atomic compute_* set. Realign the planner-facing stub to the real KM-624 inventory so the Planner plans against tools that exist. - registry.py: replace the 9 atomic entries (compute_median/stddev/percentile/ mode, date_trunc, ...) with 12 composite entries -- 4 data-access (query_structured, retrieve_documents, list_sources, describe_source) + 8 analyze_* (descriptive, aggregate, comparison, contribution, profile, correlation, segment, trend). Each analyze_* takes a `data` "${t<id>}" placeholder (Pattern A, assumed pending the tool team). - examples.py: Example A -> analyze_contribution; Example B -> analyze_trend (drops the removed date_trunc/compute_stddev chain). - planner.md: rewrite the "compute_* tools" bullet as data-access vs analytics. Validator/prompt/service unchanged (generic over the registry). Planner tests updated locally (tests/ is gitignored): 32 passing + 1 gated, ruff clean. Co-Autho…93a24da -
[KM-624] Implement Compute Functions Tools
78c598c
-
[KM-567][AI] Planner agent: fix PII leak + placeholder/retry robustness
Rifqi Hafizuddin +1: Claude Opus 4.8Review follow-ups on the Planner agent: - inputs.py: suppress a PII column's top_values when building CatalogSummary. Previously only sample_values were nulled, so a low-cardinality PII column's top_values (same class of data) could reach the planner prompt. Now both are suppressed. (Regression test added.) - validator.py: a ${tN} placeholder is now valid when its target is a transitive ancestor, not only a direct depends_on. Cycle detection moved ahead of the placeholder check so ancestors are well-defined on a DAG. This stops valid plans (target reached via an intermediate task) being rejected. - service.py: floor max_retries at 1 (mirrors QueryService) so a misconfigured value can't skip the LLM call entirely. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>6b3637d -
[KM-567][AI] Planner agent: validator + service
Rifqi Hafizuddin +1: Claude Opus 4.8- validator.py: PlannerValidator runs the 8 checks from §7.3 (tools in registry, catalog refs exist, DAG valid / no cycles, parallelism consistent, within task cap, checkable success_criteria, args valid, inline query_structured IR via the existing IRValidator). Raises PlannerValidationError with self-correctable messages. - service.py: PlannerService + plan_analysis(). LLM chain mirrors query/planner/service.py; validate-and-retry loop (max 3) mirrors QueryService. Takes the full Catalog, derives the PII-safe CatalogSummary for the prompt. Static plan only — no replanning. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>72306d0 -
[KM-567][AI] Planner agent: prompt + few-shot examples
Rifqi Hafizuddin +1: Claude Opus 4.8- prompt.py: build_planner_prompt assembles per-call human content (business context + condensed catalog + tool list + constraints + examples + question, plus prior error on retry). - config/prompts/planner.md: system prompt encoding INV-1/6/7 and the planning principles. - examples.py: two few-shot TaskLists (A exploratory revenue-by-category; B descriptive monthly-trend-by-region with date_trunc), built from the real TaskList schema so they cannot drift from the output contract. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>83ba6b1 -
[KM-567][AI] Planner agent: contracts, schemas, inputs, registry
Rifqi Hafizuddin +1: Claude Opus 4.8Phase 3 slow-path Planner foundation (AGENT_ARCHITECTURE_CONTEXT_new.md §7.3). - contracts.py: STUB BusinessContext / ToolSpec / ToolRegistry / ToolOutput (to reconcile with lead + tool team KM-608). - schemas.py: TaskList / Task / ToolCall output contract. No replan schemas. - inputs.py: CatalogSummary (condensed, PII sample_values nulled, from_catalog builder) + Constraints. - registry.py: STUB v1 P0 tool registry (query_structured, retrieve_documents, list/describe_source, compute_median/stddev/percentile/mode, date_trunc). - errors.py: PlannerError / PlannerValidationError. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>be96601
-
[KM-582][DED][AI] Fix Retrieval in Agentic Service
ticket: https://bukittechnology.atlassian.net/browse/KM-582 fix: replace LangChain ORM retrieval with raw SQL and fix pgvector collection name - Rewrite DocumentRetriever.retrieve() using raw SQL cosine/manhattan queries instead of LangChain PGVector ORM, bypassing asyncpg type-mapping issues (id String vs UUID column, jsonb_path_match binding quirks) - Fix _COLLECTION_NAME from "document_embeddings" to "documents" to match the collection name set by the Golang ingestion service - Fix collection_name in vector_store.py to match consistently - Fix Redis chat cache to store {response, sources} dict so cached replies also populate message_sources table - Add cache management endpoints: DELETE /chat/cache, /chat/cache/room/{id}, /retrieval/cache/{user_id} - Invalidate retrieval cache automatically after document processing - Update intent_router prompt: route topical/knowledge questions to unstructured even without explicit document mention; prefer unstructured when ambiguous…61c746f
-
fix build error in importing packages langchain_postgres
0e9263a -
Fix schema table langchain_pg_embedding
079dea0
-
[NOTICKET] fix: add SKIP_INIT_DB flag and remove pymongo dependency
- Add SKIP_INIT_DB env variable to skip database initialization on startup, useful for local dev when DB is not available - Remove pymongo dependency (unused, replaced by PostgreSQL-based user storage) - Comment out unused MongoDB settings in Settings class - Update .gitignore: add API_CONTRACT_AGENT*.md and migratego/ - Remove .vscode/launch.json (superseded by uv run fastapi dev)19bd037
-
feat/Catalog Retrieval System (#1)
- initial commit (8557a2045ca2c0873e07efdf6227064b83e4a1e5) - initial structure (efc0c0a8b2444014cbe74f04617c3c974d9915be) - [KM-553] initialize shared contract (f31f6731692b0199d8cb5c5d29dffa466d6d4ca0) - [KM-553] migrate retrieval layer and remove obsolete rag/tools modules (df220ea47c96eb8a1efa07c6eebef4604fc308d6) - [NOTICKET] catalog enricher + structured pipeline + on_db_registered (77ea82d29d86400c5ed30ac154be8df6a7e897eb) - Merge commit 'refs/pr/1' of https://huggingface.co/spaces/DataEyond/Agentic-Service-Data-Eyond-Catalog into pr/1 (0ae8717c7efce2cda14fe4b8caecd774459b1f6e) - [NOTICKET] SqlCompiler (Postgres) + DbExecutor (212dad360714508c1aa607995039e6818fa40f78) - Merge commit 'refs/pr/1' of https://huggingface.co/spaces/DataEyond/Agentic-Service-Data-Eyond-Catalog into pr/1 (faf8a4f6d7e06ccf1ce84c2e7f4990c79f27f843) - TabularIntrospector + on_tabular_uploaded trigger + 31 tests (ca425208969b72a20ed76f86ef0faa1a0e912339) - [KM 556] add columns field to QueryResult + update…6bff5d9
-
[KM-438][KM-439] Improve Retrieval and Querying feature (#15)
- [noticket] add gitignore (c87f27f458854787102eebc110b178c2572ef8a6) - [NOTICKET]: add document pipeline, simplify document API (fb871f3d7db9c5a9c95a012761c4180a432e7e4b) - [NOTICKET]: update folder document_pipelines after pipelines (a4cf97ab3890e35831c0043a1d411c17d84f374b) - [NOTICKET][DB] refactor code to new repo (7f3bb9782bb9b6ace69e6be35377c12757a2f81d) - [KM-441] add mean and median (9b5933420821187e73bb72a896e28d7adf6d6137) - [NOTICKET] new metadata format for cleaner code (6b590d94c398ac4966a0e376f2809aa52c5ba022) - update document (5a69e0eecf80dd807565dbc369374e41450c37fc) - delete duplicate file (3848d7b262745b758069e3ab8089437ced42a45f) - edit document for new pipeline (425e021053c4de65e2598b0f5554b7c1044a904c) - [NOTICKET]: add CSV and XLSX file type (31920c3bc6f9faf9e07095a02fc1faf08adc246e) - [DB] fix/rename db_pipeline.py (d913315c356d741cc7a5a5b1f8e36ba41397f2ab) - [NOTICKET][DB] menyesuaikan format struktur db_pipeline sesuai dengan file lain (e13a9017e0f7fd26ef9b8a…c93ec90
-
Make a query for tabular (XLSX and CSV) (#14)
- [NOTICKET] add software to gitignore (d43ecb180036339cf287e93f1d9116bd6eff9b9d) - [NOTICKET] add pyarrow (e50eadc82e160012e3319267f5bfe084cd9034d4) - [KM-515][document] Make Query for Tabular Type (XLSX & CSV) (695ca0a154a68077c51499ed83b7507b988be065) - [KM-455][document] decided methods retrieval for document (8c9cc79223eb1d96c8622d129219a83a4ba2500b) Co-authored-by: Sofhia Az-Zahra <sofhiaazzhr@users.noreply.huggingface.co>8802920
-
Merge dev_new to main (improve retriever and add querying) (#13)
- [noticket] add gitignore (c87f27f458854787102eebc110b178c2572ef8a6) - [NOTICKET]: add document pipeline, simplify document API (fb871f3d7db9c5a9c95a012761c4180a432e7e4b) - [NOTICKET]: update folder document_pipelines after pipelines (a4cf97ab3890e35831c0043a1d411c17d84f374b) - [NOTICKET][DB] refactor code to new repo (7f3bb9782bb9b6ace69e6be35377c12757a2f81d) - [KM-441] add mean and median (9b5933420821187e73bb72a896e28d7adf6d6137) - [NOTICKET] new metadata format for cleaner code (6b590d94c398ac4966a0e376f2809aa52c5ba022) - update document (5a69e0eecf80dd807565dbc369374e41450c37fc) - delete duplicate file (3848d7b262745b758069e3ab8089437ced42a45f) - edit document for new pipeline (425e021053c4de65e2598b0f5554b7c1044a904c) - [NOTICKET]: add CSV and XLSX file type (31920c3bc6f9faf9e07095a02fc1faf08adc246e) - [DB] fix/rename db_pipeline.py (d913315c356d741cc7a5a5b1f8e36ba41397f2ab) - [NOTICKET][DB] menyesuaikan format struktur db_pipeline sesuai dengan file lain (e13a9017e0f7fd26ef9b8a…52999bc
-
[NOTICKET] fixed single source into multiple sources (#12)
- [NOTICKET] fixed single source into multiple sources (9d826225424dd8fb0044ce6f4a010ab8eac1075b) - [NOTICKET] remove source_hint line (d1f034edff5a6a78c0a7b9f3402ca2a46baa4f43) Co-authored-by: Sofhia Az-Zahra <sofhiaazzhr@users.noreply.huggingface.co>c8e3046 -
[NOTICKET] add updated_at metadata & prevent duplicate while adding new embedding (#10)
- [NOTICKET] add updated_at & delete old embedding before adding new embedding ingestion (da83e0111d3fd220fcfdd717018ce4794415e020)bd0029e
-
[NOTICKET] add total token logging (#9)
- [NOTICKET] add token logging (34eb194ed4c3f31ccd968390d1c05ca9547edafe)d973099
-
[KM-496] [DED][AI] Fix Response data structure for field sources (db flow)
5c4067e -
[NOTICKET] [document]: add doctypes endpoint (#7)
- [NOTIKCET][document]: add doctypes endpoint and 10MB file size limit (9694cc0635e4b59a0d6291e613d83262ee3277f3) Co-authored-by: Sofhia Az-Zahra <sofhiaazzhr@users.noreply.huggingface.co>b1f1ccd -
[NOTICKET][DB] add endpoints for dbtypes (#6)
- [noticket] add gitignore (c87f27f458854787102eebc110b178c2572ef8a6) - [NOTICKET]: add document pipeline, simplify document API (fb871f3d7db9c5a9c95a012761c4180a432e7e4b) - [NOTICKET]: update folder document_pipelines after pipelines (a4cf97ab3890e35831c0043a1d411c17d84f374b) - [NOTICKET][DB] refactor code to new repo (7f3bb9782bb9b6ace69e6be35377c12757a2f81d) - [KM-441] add mean and median (9b5933420821187e73bb72a896e28d7adf6d6137) - [NOTICKET] new metadata format for cleaner code (6b590d94c398ac4966a0e376f2809aa52c5ba022) - update document (5a69e0eecf80dd807565dbc369374e41450c37fc) - delete duplicate file (3848d7b262745b758069e3ab8089437ced42a45f) - edit document for new pipeline (425e021053c4de65e2598b0f5554b7c1044a904c) - [NOTICKET]: add CSV and XLSX file type (31920c3bc6f9faf9e07095a02fc1faf08adc246e) - [DB] fix/rename db_pipeline.py (d913315c356d741cc7a5a5b1f8e36ba41397f2ab) - [NOTICKET][DB] menyesuaikan format struktur db_pipeline sesuai dengan file lain (e13a9017e0f7fd26ef9b8a…c7d59cf
-
[KM-493] [DED][AI] Fix source extraction in chat
changes: - Add .vscode/launch.json with debugpy config for FastAPI via uvicorn on port 7860 - Fix _extract_sources in chat.py to read metadata from nested `data` key - Add debug log for extracted sources tikcet: https://bukittechnology.atlassian.net/browse/KM-4937da6db5 -
[KM-436][KM-437] add and modify for database and document pipeline (#3)
- [noticket] add gitignore (c87f27f458854787102eebc110b178c2572ef8a6) - [NOTICKET]: add document pipeline, simplify document API (fb871f3d7db9c5a9c95a012761c4180a432e7e4b) - [NOTICKET]: update folder document_pipelines after pipelines (a4cf97ab3890e35831c0043a1d411c17d84f374b) - [NOTICKET][DB] refactor code to new repo (7f3bb9782bb9b6ace69e6be35377c12757a2f81d) - [KM-441] add mean and median (9b5933420821187e73bb72a896e28d7adf6d6137) - [NOTICKET] new metadata format for cleaner code (6b590d94c398ac4966a0e376f2809aa52c5ba022) - update document (5a69e0eecf80dd807565dbc369374e41450c37fc) - delete duplicate file (3848d7b262745b758069e3ab8089437ced42a45f) - edit document for new pipeline (425e021053c4de65e2598b0f5554b7c1044a904c) - [NOTICKET]: add CSV and XLSX file type (31920c3bc6f9faf9e07095a02fc1faf08adc246e) - [DB] fix/rename db_pipeline.py (d913315c356d741cc7a5a5b1f8e36ba41397f2ab) - [NOTICKET][DB] menyesuaikan format struktur db_pipeline sesuai dengan file lain (e13a9017e0f7fd26ef9b8a…2ba0613
-
[NOTICKET] Init db_client endpoint
a7d8798 -
[KM-467] [DED][AI] Orchestration Agent - Init
- https://bukittechnology.atlassian.net/browse/KM-467027123c