Commit History
-
[NOTICKET][AI] feat(chat): wire Orchestrator -> slow path at the endpoint (env-gated)
Rifqi Hafizuddin +1: Claude Opus 4.8The slow path (Planner -> TaskRunner -> Assembler) was reachable in ChatHandler but never enabled from the live endpoint. Wire it so the team can test end-to-end from /chat/stream, behind an env flag so it stays opt-in until the lead's real BusinessContext lands (it's still a stub). - settings.enable_slow_path (env ENABLE_SLOW_PATH, default False). - api/v1/chat.py passes it to the shared ChatHandler. - ENABLE_SLOW_PATH=true → structured intents route through the slow path and stream status progress + the assembled answer; default keeps the single-query QueryService path. Fast/unstructured paths unchanged. Verified: toggle flips handler._enable_slow_path; wiring tests pass; live run through ChatHandler.handle confirmed earlier this session. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>636290e -
[NOTICKET] perf(slow-path): DB3 pre-connect + R4 progress events
Rifqi Hafizuddin +1: Claude Opus 4.8DB3 (DB owner): DbExecutor.prewarm(catalog, user_id) warms the pooled engine for schema sources, fired (best-effort, never raises) at slow-path entry so the cold first-query TCP+TLS+auth handshake overlaps the ~4s Planner LLM call. Gated to the default path (skipped when a coordinator factory is injected, so tests stay hermetic). Reuses DbExecutor's existing client resolution + the DB2 engine cache. R4 (agent): SlowPathCoordinator.run gained an optional `progress` callback; ChatHandler bridges it to SSE `status` events via an asyncio queue, and chat.py forwards them. The slow path no longer streams ~13s of silence — live timeline now shows Planning -> Running N steps -> Composing, max wire gap ~4.6s, fixing proxy idle-timeout + UX. Status events only appear when the coordinator calls back, so the existing wiring test's event sequence is unchanged (fake coordinator just accepts **kwargs). Token-streaming the Assembler answer is deferred: it would require splitting the Assembler into a …735c421 -
[NOTICKET][AI] perf(chat): PR-DB1 - memoizing catalog reader + shared ChatHandler
Rifqi Hafizuddin +1: Claude Opus 4.8Cut redundant catalog-DB round-trips in the slow path and stop rebuilding the Azure LLM clients per request. - MemoizingCatalogReader (catalog/reader.py): request-scoped wrapper that caches each read by source_hint. ChatHandler builds one per request and threads it into the slow-path DataAccessToolInvoker, so the planner's catalog load is reused by describe_source / query_structured instead of re-reading from the catalog DB. Measured live: describe_source 3.5s -> ~2.0s; catalog reads/request ~5 -> ~2. - chat.py: one module-level ChatHandler singleton instead of per-request, keeping the Orchestrator/Chatbot Azure clients (and TLS pools) warm across requests. Does not touch the external query_structured handshake (that is the engine-cache work, DB2). Also logs the principal-review fix tracker (R1-R12, DB1-DB3) in PROGRESS. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>4c62146
-
[KM-631][AI] Langfuse tracing: tokens + latency across the chat pipeline
Rifqi Hafizuddin +1: Claude Opus 4.8Wire Langfuse (was scaffolded-only) so each chat request emits one trace with per-call token counts + latency, plus tool spans. Gated and PII-aware. - New src/observability/langfuse/tracing.py: RequestTracer (one trace/request), NullTracer (no-op), TracingToolInvoker (metadata-only span per tool call), _redact MaskFunction. Best-effort: never raises; pipeline unchanged if Langfuse is down or disabled. - Each LLM call (Orchestrator, Planner incl. retries, Assembler, Chatbot) gained an optional `callbacks` param, passed to the LangChain call only when present (so existing fakes/tests are untouched). Coordinator threads planner/assembler callbacks through. - ChatHandler: enable_tracing flag (default OFF so tests never hit Langfuse), creates one trace per request, wraps the tool invoker for spans, ends the trace. - PII policy for Cloud: UNMASKED = Orchestrator + Planner (PII-safe inputs); MASKED (tokens+latency only) = Assembler + Chatbot (see real rows / doc chunks); to…8cb3714
-
[KM-582][DED][AI] Fix Retrieval in Agentic Service
ticket: https://bukittechnology.atlassian.net/browse/KM-582 fix: replace LangChain ORM retrieval with raw SQL and fix pgvector collection name - Rewrite DocumentRetriever.retrieve() using raw SQL cosine/manhattan queries instead of LangChain PGVector ORM, bypassing asyncpg type-mapping issues (id String vs UUID column, jsonb_path_match binding quirks) - Fix _COLLECTION_NAME from "document_embeddings" to "documents" to match the collection name set by the Golang ingestion service - Fix collection_name in vector_store.py to match consistently - Fix Redis chat cache to store {response, sources} dict so cached replies also populate message_sources table - Add cache management endpoints: DELETE /chat/cache, /chat/cache/room/{id}, /retrieval/cache/{user_id} - Invalidate retrieval cache automatically after document processing - Update intent_router prompt: route topical/knowledge questions to unstructured even without explicit document mention; prefer unstructured when ambiguous…61c746f
-
feat/Catalog Retrieval System (#1)
- initial commit (8557a2045ca2c0873e07efdf6227064b83e4a1e5) - initial structure (efc0c0a8b2444014cbe74f04617c3c974d9915be) - [KM-553] initialize shared contract (f31f6731692b0199d8cb5c5d29dffa466d6d4ca0) - [KM-553] migrate retrieval layer and remove obsolete rag/tools modules (df220ea47c96eb8a1efa07c6eebef4604fc308d6) - [NOTICKET] catalog enricher + structured pipeline + on_db_registered (77ea82d29d86400c5ed30ac154be8df6a7e897eb) - Merge commit 'refs/pr/1' of https://huggingface.co/spaces/DataEyond/Agentic-Service-Data-Eyond-Catalog into pr/1 (0ae8717c7efce2cda14fe4b8caecd774459b1f6e) - [NOTICKET] SqlCompiler (Postgres) + DbExecutor (212dad360714508c1aa607995039e6818fa40f78) - Merge commit 'refs/pr/1' of https://huggingface.co/spaces/DataEyond/Agentic-Service-Data-Eyond-Catalog into pr/1 (faf8a4f6d7e06ccf1ce84c2e7f4990c79f27f843) - TabularIntrospector + on_tabular_uploaded trigger + 31 tests (ca425208969b72a20ed76f86ef0faa1a0e912339) - [KM 556] add columns field to QueryResult + update…6bff5d9
-
[KM-438][KM-439] Improve Retrieval and Querying feature (#15)
- [noticket] add gitignore (c87f27f458854787102eebc110b178c2572ef8a6) - [NOTICKET]: add document pipeline, simplify document API (fb871f3d7db9c5a9c95a012761c4180a432e7e4b) - [NOTICKET]: update folder document_pipelines after pipelines (a4cf97ab3890e35831c0043a1d411c17d84f374b) - [NOTICKET][DB] refactor code to new repo (7f3bb9782bb9b6ace69e6be35377c12757a2f81d) - [KM-441] add mean and median (9b5933420821187e73bb72a896e28d7adf6d6137) - [NOTICKET] new metadata format for cleaner code (6b590d94c398ac4966a0e376f2809aa52c5ba022) - update document (5a69e0eecf80dd807565dbc369374e41450c37fc) - delete duplicate file (3848d7b262745b758069e3ab8089437ced42a45f) - edit document for new pipeline (425e021053c4de65e2598b0f5554b7c1044a904c) - [NOTICKET]: add CSV and XLSX file type (31920c3bc6f9faf9e07095a02fc1faf08adc246e) - [DB] fix/rename db_pipeline.py (d913315c356d741cc7a5a5b1f8e36ba41397f2ab) - [NOTICKET][DB] menyesuaikan format struktur db_pipeline sesuai dengan file lain (e13a9017e0f7fd26ef9b8a…c93ec90
-
Merge dev_new to main (improve retriever and add querying) (#13)
- [noticket] add gitignore (c87f27f458854787102eebc110b178c2572ef8a6) - [NOTICKET]: add document pipeline, simplify document API (fb871f3d7db9c5a9c95a012761c4180a432e7e4b) - [NOTICKET]: update folder document_pipelines after pipelines (a4cf97ab3890e35831c0043a1d411c17d84f374b) - [NOTICKET][DB] refactor code to new repo (7f3bb9782bb9b6ace69e6be35377c12757a2f81d) - [KM-441] add mean and median (9b5933420821187e73bb72a896e28d7adf6d6137) - [NOTICKET] new metadata format for cleaner code (6b590d94c398ac4966a0e376f2809aa52c5ba022) - update document (5a69e0eecf80dd807565dbc369374e41450c37fc) - delete duplicate file (3848d7b262745b758069e3ab8089437ced42a45f) - edit document for new pipeline (425e021053c4de65e2598b0f5554b7c1044a904c) - [NOTICKET]: add CSV and XLSX file type (31920c3bc6f9faf9e07095a02fc1faf08adc246e) - [DB] fix/rename db_pipeline.py (d913315c356d741cc7a5a5b1f8e36ba41397f2ab) - [NOTICKET][DB] menyesuaikan format struktur db_pipeline sesuai dengan file lain (e13a9017e0f7fd26ef9b8a…52999bc
-
[NOTICKET] fixed single source into multiple sources (#12)
- [NOTICKET] fixed single source into multiple sources (9d826225424dd8fb0044ce6f4a010ab8eac1075b) - [NOTICKET] remove source_hint line (d1f034edff5a6a78c0a7b9f3402ca2a46baa4f43) Co-authored-by: Sofhia Az-Zahra <sofhiaazzhr@users.noreply.huggingface.co>c8e3046
-
[KM-496] [DED][AI] Fix Response data structure for field sources (db flow)
5c4067e -
[NOTICKET] [document]: add doctypes endpoint (#7)
- [NOTIKCET][document]: add doctypes endpoint and 10MB file size limit (9694cc0635e4b59a0d6291e613d83262ee3277f3) Co-authored-by: Sofhia Az-Zahra <sofhiaazzhr@users.noreply.huggingface.co>b1f1ccd -
[NOTICKET][DB] add endpoints for dbtypes (#6)
- [noticket] add gitignore (c87f27f458854787102eebc110b178c2572ef8a6) - [NOTICKET]: add document pipeline, simplify document API (fb871f3d7db9c5a9c95a012761c4180a432e7e4b) - [NOTICKET]: update folder document_pipelines after pipelines (a4cf97ab3890e35831c0043a1d411c17d84f374b) - [NOTICKET][DB] refactor code to new repo (7f3bb9782bb9b6ace69e6be35377c12757a2f81d) - [KM-441] add mean and median (9b5933420821187e73bb72a896e28d7adf6d6137) - [NOTICKET] new metadata format for cleaner code (6b590d94c398ac4966a0e376f2809aa52c5ba022) - update document (5a69e0eecf80dd807565dbc369374e41450c37fc) - delete duplicate file (3848d7b262745b758069e3ab8089437ced42a45f) - edit document for new pipeline (425e021053c4de65e2598b0f5554b7c1044a904c) - [NOTICKET]: add CSV and XLSX file type (31920c3bc6f9faf9e07095a02fc1faf08adc246e) - [DB] fix/rename db_pipeline.py (d913315c356d741cc7a5a5b1f8e36ba41397f2ab) - [NOTICKET][DB] menyesuaikan format struktur db_pipeline sesuai dengan file lain (e13a9017e0f7fd26ef9b8a…c7d59cf
-
[KM-493] [DED][AI] Fix source extraction in chat
changes: - Add .vscode/launch.json with debugpy config for FastAPI via uvicorn on port 7860 - Fix _extract_sources in chat.py to read metadata from nested `data` key - Add debug log for extracted sources tikcet: https://bukittechnology.atlassian.net/browse/KM-4937da6db5 -
[KM-436][KM-437] add and modify for database and document pipeline (#3)
- [noticket] add gitignore (c87f27f458854787102eebc110b178c2572ef8a6) - [NOTICKET]: add document pipeline, simplify document API (fb871f3d7db9c5a9c95a012761c4180a432e7e4b) - [NOTICKET]: update folder document_pipelines after pipelines (a4cf97ab3890e35831c0043a1d411c17d84f374b) - [NOTICKET][DB] refactor code to new repo (7f3bb9782bb9b6ace69e6be35377c12757a2f81d) - [KM-441] add mean and median (9b5933420821187e73bb72a896e28d7adf6d6137) - [NOTICKET] new metadata format for cleaner code (6b590d94c398ac4966a0e376f2809aa52c5ba022) - update document (5a69e0eecf80dd807565dbc369374e41450c37fc) - delete duplicate file (3848d7b262745b758069e3ab8089437ced42a45f) - edit document for new pipeline (425e021053c4de65e2598b0f5554b7c1044a904c) - [NOTICKET]: add CSV and XLSX file type (31920c3bc6f9faf9e07095a02fc1faf08adc246e) - [DB] fix/rename db_pipeline.py (d913315c356d741cc7a5a5b1f8e36ba41397f2ab) - [NOTICKET][DB] menyesuaikan format struktur db_pipeline sesuai dengan file lain (e13a9017e0f7fd26ef9b8a…2ba0613
-
[NOTICKET] Init db_client endpoint
a7d8798 -
[KM-467] [DED][AI] Orchestration Agent - Init
- https://bukittechnology.atlassian.net/browse/KM-467027123c