Commit History
-
[KM-564] fix source, now shows name instead of id. added diff retrieval vs catalog
Rifqi Hafizuddin96598f8 -
add source to stream output
Rifqi Hafizuddin277f7e2 -
[KM-565] sync catalog on document and db client deletion
- add CatalogStore.remove_source() to drop a single source by source_id - add on_tabular_deleted / on_db_deleted triggers - call triggers from delete endpoints so catalog stays in sync Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>8b4794c -
[KM-565] fix catalog timestamps, count_distinct logic, and query debug logging
- generated_at set once on first insert, never overwritten on upsert - updated_at only changes when catalog data jsonb actually differs - query_planner prompt: disambiguate count vs count_distinct - tabular executor: add _render_query helper for debug logging - query service: log select/filters/group_by on each IR attempt Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>887c1c4
-
[KM-564] fix render and sql error
Rifqi Hafizuddin2e601ec -
Merge commit 'refs/pr/1' of https://huggingface.co/spaces/DataEyond/Agentic-Service-Data-Eyond-Catalog into pr/1
Rifqi Hafizuddinc4754f3 -
[KM-564] edit agent logic
Rifqi Hafizuddin35b204d -
[NOTICKET] add column updated_at on catalog
2109c4e -
[KM-564] Edit column description in catalog to reduce token & ingestion time
Rifqi Hafizuddin430c361 -
[NOTICKET] cleanup PR: wire ChatHandler, drop Phase 1 remnants, implement catalog rebuild
- wire chat.py to ChatHandler; remove inline intent routing, retrieval, and answer generation - drop Phase 1 dual-write in db_client /ingest: only on_db_registered runs now - implement on_catalog_rebuild_requested + POST /api/v1/data-catalog/rebuild endpoint - delete dead modules: agents/chatbot (Phase 1), pipeline/orchestrator, query/base, api/v1/knowledge, config/agents/ - rename answer_agent.py -> chatbot.py, AnswerAgent -> ChatbotAgent - fix ChatHandler to use RetrievalRouter (restores Redis retrieval cache) - remove dead _build_csv_documents/_build_excel_documents from processing_service - add top_values to ColumnStats for low-cardinality columns (query planner context) - replace deprecated @app.on_event("startup") with lifespan context manager Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>0700e2b
-
[KM-556][KM-562] fix tabular catalog pipeline: trigger wiring, parquet upload, lazy Azure init, backfill script
- Wire on_db_registered in /database-clients/{id}/ingest endpoint so DB registration now builds the catalog (was silently skipped before). - CSV/XLSX no longer ingested into vector store (embedding wasted); parquet upload preserved via _upload_parquet() so TabularExecutor can still read blobs at query time. - on_document_uploaded implemented (was raise NotImplementedError); opens its own AsyncSessionLocal session, calls document_pipeline.process(). - Lazy-init Azure/PGVector clients in retrieval/document.py via @functools.cache getters; RetrievalRouter holds DocumentRetriever lazily. FastAPI no longer crashes at startup if embedding env vars are missing. - _normalize_chunks in chat_handler.py now handles RetrievalResult objects (previously silently dropped, causing empty context for unstructured path). - Cache invalidation: retrieval_router.invalidate_cache(user_id) called after document process and delete; Redis failure is logged, not raised. - Add scripts/build_initia…59b14af -
[KM-560][KM-561] drop catalog LLM enrichment + rename store to data_catalog + add /data-catalog index endpoint
Rifqi Hafizuddin +1: Claude Opus 4.7- Remove CatalogEnricher entirely (cost cut). Planner reads stats + sample rows + column names directly. Deletes src/catalog/enricher.py, config/prompts/catalog_enricher.md, and the StructuredPipeline enrich step. render_source moves to src/catalog/render.py. - Rename jsonb table catalogs -> data_catalog. Class name unchanged; table created fresh on next init_db (no rows in prod yet, no migration needed). - Add GET /api/v1/data-catalog/{user_id} returning list[CatalogIndexEntry] (source_id, source_type, name, location_ref, table_count, updated_at). Wired in main.py. Lightweight summary intended for the catalog refresher. - Update ARCHITECTURE.md / REPO_CONTEXT.md / PROGRESS.md: LLM call sites now three (was four), ingestion flow is introspect -> validate -> upsert. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>2d6eca0
-
fixes: chain singleton, parquet logger, SSE event consistency, remove double DB fetch
ce20d89 -
[KM-556] delete Phase 1 remnants: query/executors/, query_executor.py, orchestration.py
b272cc7 -
[KM-556] rewire chat stream endpoint to Phase 2 IntentRouter + AnswerAgent + QueryService
5670888 -
[KM-556] wire document process endpoint to on_tabular_uploaded[KM-556] wire document process endpoint to on_tabular_uploaded
0000030 -
[KM-556] move parquet_service from knowledge/ to storage/
cd06c0f -
[KM-556] flatten document_pipeline folder into single module
7b2c669 -
fix planner prompt template parsing
f34af71 -
Merge DB bundle (PR2b/4/5/6/7) into pr/1 + reconcile PROGRESS.md
efc301b -
[NOTICKET] IntentRouter + planner + dispatcher + QueryService + AnswerAgent + ChatHandler
Rifqi Hafizuddin +1: Claude Opus 4.7Bundles every Both-PR work item that doesn't require teammate's tabular modules. Phase 1 endpoints and chatbot.py are intentionally NOT touched -- cleanup PR will rewire /chat/stream and rename answer_agent.py -> chatbot.py. - IntentRouter: classify chat / unstructured / structured; history-aware rewritten_query via Pydantic structured output. Prompt with full ruleset + few-shot in config/prompts/intent_router.md. - Planner prompt: build_planner_prompt(question, catalog, previous_error) reuses catalog.enricher.render_source so DB and tabular sources render identically across enricher and planner. System prompt has hard constraints + DB and tabular few-shot in config/prompts/query_planner.md. - QueryPlannerService: Azure OpenAI structured output -> QueryIR. Injectable chain. Supports retry via previous_error arg. - ExecutorDispatcher: pick(ir) routes by source.source_type. Lazy executor imports keep module import-safe. Caches per source_type. Tests inject factories. - Q…f65aee0 -
detect XLSX by filename, not table count
307a7b1
-
eat(query): PandasCompiler + TabularExecutor (PR3-TAB)
c749544 -
[KM 556] add columns field to QueryResult + update DbExecutor
04e5c48 -
TabularIntrospector + on_tabular_uploaded trigger + 31 tests
ca42520 -
Merge commit 'refs/pr/1' of https://huggingface.co/spaces/DataEyond/Agentic-Service-Data-Eyond-Catalog into pr/1
faf8a4f -
[NOTICKET] SqlCompiler (Postgres) + DbExecutor
Rifqi Hafizuddin +1: Claude Opus 4.7- SqlCompiler: deterministic IR -> (sql, params dict) for Postgres dialect (Supabase reuses). Double-quoted identifiers from catalog, named parameters for values. Supports all 12 whitelisted filter ops, 6 aggs (count/count_distinct/sum/avg/min/max), count(*), alias-aware order_by. CompiledSql.params changed from list to dict[str, Any]. - DbExecutor: layered safety -- source_type check + dbclient:// parse + client owner match + sqlglot SELECT/no-DML guard + Postgres session default_transaction_read_only + statement_timeout=30s + asyncio.wait_for backstop + 10k row cap. Runs sync engine via db_pipeline_service.engine_scope inside asyncio.to_thread. Never raises -- populates QueryResult.error. - pandas.py compiler stub: fixed pre-existing UP035 (Callable import). - pyproject.toml: added S608 to tests/** ruff ignore (test assertions on literal SQL strings are a false positive). Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>212dad3 -
Merge commit 'refs/pr/1' of https://huggingface.co/spaces/DataEyond/Agentic-Service-Data-Eyond-Catalog into pr/1
0ae8717 -
[NOTICKET] catalog enricher + structured pipeline + on_db_registered
Rifqi Hafizuddin +1: Claude Opus 4.7- CatalogEnricher: Azure OpenAI GPT-4o with structured output (EnrichmentResponse keyed by stable IDs), render_source helper reusable by the planner prompt, apply_descriptions merger, injectable structured_chain for tests - catalog_enricher.md system prompt with style rules + few-shot - StructuredPipeline: introspect -> enrich -> merge -> validate -> upsert; source-type-agnostic (caller supplies introspector); default_structured_pipeline() factory lazy-imports prod deps - on_db_registered trigger wired; tabular/document/rebuild stubs preserved with implementation notes for the teammate - Table.foreign_keys + ForeignKey model; DB introspector now extracts FKs from extractor data (was discarded) - REPO_CONTEXT.md catalog schema diagram reflects FK addition - PROGRESS.md persistent checklist for the team Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>77ea82d -
[KM-553] migrate retrieval layer and remove obsolete rag/tools modules
- Replace src/rag/ with src/retrieval/: implement DocumentRetriever (MMR/cosine/euclidean/manhattan), simplified RetrievalRouter (unstructured-only, no schema leg, Redis cache preserved), and shared RetrievalResult/BaseRetriever base - Remove src/tools/ (orphaned LangChain @tool wrapper, never called by production code) - Update RetrievalResult imports in chat.py, query/base.py, query/executors/db_executor.py, query/executors/tabular.py, query/query_executor.py from src.rag.base to src.retrieval.base - Wire chat.py to new retrieval_router (aliased as retriever, no call-site changes) - Delete dead stubs: src/query/service.py, src/models/user_info.py, src/pipeline/document_pipeline.py (flat, shadowed by subfolder) Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>df220ea -
[KM-553] initialize shared contract
Rifqi Hafizuddinf31f673 -
initial structure
efc0c0a -
initial commit
8557a20
-
[KM-438][KM-439] Improve Retrieval and Querying feature (#15)
- [noticket] add gitignore (c87f27f458854787102eebc110b178c2572ef8a6) - [NOTICKET]: add document pipeline, simplify document API (fb871f3d7db9c5a9c95a012761c4180a432e7e4b) - [NOTICKET]: update folder document_pipelines after pipelines (a4cf97ab3890e35831c0043a1d411c17d84f374b) - [NOTICKET][DB] refactor code to new repo (7f3bb9782bb9b6ace69e6be35377c12757a2f81d) - [KM-441] add mean and median (9b5933420821187e73bb72a896e28d7adf6d6137) - [NOTICKET] new metadata format for cleaner code (6b590d94c398ac4966a0e376f2809aa52c5ba022) - update document (5a69e0eecf80dd807565dbc369374e41450c37fc) - delete duplicate file (3848d7b262745b758069e3ab8089437ced42a45f) - edit document for new pipeline (425e021053c4de65e2598b0f5554b7c1044a904c) - [NOTICKET]: add CSV and XLSX file type (31920c3bc6f9faf9e07095a02fc1faf08adc246e) - [DB] fix/rename db_pipeline.py (d913315c356d741cc7a5a5b1f8e36ba41397f2ab) - [NOTICKET][DB] menyesuaikan format struktur db_pipeline sesuai dengan file lain (e13a9017e0f7fd26ef9b8a…c93ec90
-
Make a query for tabular (XLSX and CSV) (#14)
- [NOTICKET] add software to gitignore (d43ecb180036339cf287e93f1d9116bd6eff9b9d) - [NOTICKET] add pyarrow (e50eadc82e160012e3319267f5bfe084cd9034d4) - [KM-515][document] Make Query for Tabular Type (XLSX & CSV) (695ca0a154a68077c51499ed83b7507b988be065) - [KM-455][document] decided methods retrieval for document (8c9cc79223eb1d96c8622d129219a83a4ba2500b) Co-authored-by: Sofhia Az-Zahra <sofhiaazzhr@users.noreply.huggingface.co>8802920
-
Merge dev_new to main (improve retriever and add querying) (#13)
- [noticket] add gitignore (c87f27f458854787102eebc110b178c2572ef8a6) - [NOTICKET]: add document pipeline, simplify document API (fb871f3d7db9c5a9c95a012761c4180a432e7e4b) - [NOTICKET]: update folder document_pipelines after pipelines (a4cf97ab3890e35831c0043a1d411c17d84f374b) - [NOTICKET][DB] refactor code to new repo (7f3bb9782bb9b6ace69e6be35377c12757a2f81d) - [KM-441] add mean and median (9b5933420821187e73bb72a896e28d7adf6d6137) - [NOTICKET] new metadata format for cleaner code (6b590d94c398ac4966a0e376f2809aa52c5ba022) - update document (5a69e0eecf80dd807565dbc369374e41450c37fc) - delete duplicate file (3848d7b262745b758069e3ab8089437ced42a45f) - edit document for new pipeline (425e021053c4de65e2598b0f5554b7c1044a904c) - [NOTICKET]: add CSV and XLSX file type (31920c3bc6f9faf9e07095a02fc1faf08adc246e) - [DB] fix/rename db_pipeline.py (d913315c356d741cc7a5a5b1f8e36ba41397f2ab) - [NOTICKET][DB] menyesuaikan format struktur db_pipeline sesuai dengan file lain (e13a9017e0f7fd26ef9b8a…52999bc
-
[NOTICKET] fixed single source into multiple sources (#12)
- [NOTICKET] fixed single source into multiple sources (9d826225424dd8fb0044ce6f4a010ab8eac1075b) - [NOTICKET] remove source_hint line (d1f034edff5a6a78c0a7b9f3402ca2a46baa4f43) Co-authored-by: Sofhia Az-Zahra <sofhiaazzhr@users.noreply.huggingface.co>c8e3046 -
[NOTICKET] add updated_at metadata & prevent duplicate while adding new embedding (#10)
- [NOTICKET] add updated_at & delete old embedding before adding new embedding ingestion (da83e0111d3fd220fcfdd717018ce4794415e020)bd0029e
-
[NOTICKET] add total token logging (#9)
- [NOTICKET] add token logging (34eb194ed4c3f31ccd968390d1c05ca9547edafe)d973099
-
[KM-496] [DED][AI] Fix Response data structure for field sources (db flow)
5c4067e -
[NOTICKET] [document]: add doctypes endpoint (#7)
- [NOTIKCET][document]: add doctypes endpoint and 10MB file size limit (9694cc0635e4b59a0d6291e613d83262ee3277f3) Co-authored-by: Sofhia Az-Zahra <sofhiaazzhr@users.noreply.huggingface.co>b1f1ccd -
[NOTICKET][DB] add endpoints for dbtypes (#6)
- [noticket] add gitignore (c87f27f458854787102eebc110b178c2572ef8a6) - [NOTICKET]: add document pipeline, simplify document API (fb871f3d7db9c5a9c95a012761c4180a432e7e4b) - [NOTICKET]: update folder document_pipelines after pipelines (a4cf97ab3890e35831c0043a1d411c17d84f374b) - [NOTICKET][DB] refactor code to new repo (7f3bb9782bb9b6ace69e6be35377c12757a2f81d) - [KM-441] add mean and median (9b5933420821187e73bb72a896e28d7adf6d6137) - [NOTICKET] new metadata format for cleaner code (6b590d94c398ac4966a0e376f2809aa52c5ba022) - update document (5a69e0eecf80dd807565dbc369374e41450c37fc) - delete duplicate file (3848d7b262745b758069e3ab8089437ced42a45f) - edit document for new pipeline (425e021053c4de65e2598b0f5554b7c1044a904c) - [NOTICKET]: add CSV and XLSX file type (31920c3bc6f9faf9e07095a02fc1faf08adc246e) - [DB] fix/rename db_pipeline.py (d913315c356d741cc7a5a5b1f8e36ba41397f2ab) - [NOTICKET][DB] menyesuaikan format struktur db_pipeline sesuai dengan file lain (e13a9017e0f7fd26ef9b8a…c7d59cf
-
[KM-493] [DED][AI] Fix source extraction in chat
changes: - Add .vscode/launch.json with debugpy config for FastAPI via uvicorn on port 7860 - Fix _extract_sources in chat.py to read metadata from nested `data` key - Add debug log for extracted sources tikcet: https://bukittechnology.atlassian.net/browse/KM-4937da6db5 -
[KM-436][KM-437] add and modify for database and document pipeline (#3)
- [noticket] add gitignore (c87f27f458854787102eebc110b178c2572ef8a6) - [NOTICKET]: add document pipeline, simplify document API (fb871f3d7db9c5a9c95a012761c4180a432e7e4b) - [NOTICKET]: update folder document_pipelines after pipelines (a4cf97ab3890e35831c0043a1d411c17d84f374b) - [NOTICKET][DB] refactor code to new repo (7f3bb9782bb9b6ace69e6be35377c12757a2f81d) - [KM-441] add mean and median (9b5933420821187e73bb72a896e28d7adf6d6137) - [NOTICKET] new metadata format for cleaner code (6b590d94c398ac4966a0e376f2809aa52c5ba022) - update document (5a69e0eecf80dd807565dbc369374e41450c37fc) - delete duplicate file (3848d7b262745b758069e3ab8089437ced42a45f) - edit document for new pipeline (425e021053c4de65e2598b0f5554b7c1044a904c) - [NOTICKET]: add CSV and XLSX file type (31920c3bc6f9faf9e07095a02fc1faf08adc246e) - [DB] fix/rename db_pipeline.py (d913315c356d741cc7a5a5b1f8e36ba41397f2ab) - [NOTICKET][DB] menyesuaikan format struktur db_pipeline sesuai dengan file lain (e13a9017e0f7fd26ef9b8a…2ba0613
-
[NOTICKET] Init db_client endpoint
a7d8798 -
[KM-467] [DED][AI] Orchestration Agent - Init
- https://bukittechnology.atlassian.net/browse/KM-467027123c