Commit History
26 days ago
-
feat(documents): PaddleOCR-VL parser for v2 processing, server toggle
Rifqi Hafizuddin +1: Claude Opus 5.5POST /api/v2/document/process now parses with PaddleOCR-VL-1.6 (Baidu AI Studio job API) by default; DOCUMENT_PARSER_BACKEND=paddleocr|mistral picks the parser server-side. New paddle_ocr.py (multipart submit -> poll state -> JSONL; figures downloaded at parse time) and normalize_paddle.py (same ParsedDocument; a title before a table becomes its caption; figures matched by label+bbox since the job API leaves image blocks empty). DOCX is refused on PaddleOCR with a readable error. Rows record their parser; chart images reach image-token. Knowledge pipeline default unchanged (Mistral). Live 2026-09-24: 64-page Komatsu manual -> completed, 80 chunks, 27 figures, signed figure URL 200. Needs PADDLEOCR_ACCESS_TOKEN. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>7ee2b45
27 days ago
-
feat(documents): v2 unstructured ingestion, image-token, scoped retrieval
Rifqi Hafizuddin +1: Claude Opus 5.5UNSTRUCTURED_V2_PLAN U1-U5 + the U7 planner-catalog change. - /api/v2/document*: doctypes, upload, process, list for pdf/docx/txt, mirroring Go's shapes; delete stays on Go (U-D7). Parsing v2 (Mistral OCR) -> figures to the private Supabase bucket -> embeddings in the existing `documents` collection tagged parser_version=v2. - POST /api/v2/document/image-token: 1h presigned URLs for asset:// markers, bound-document + owner + key-prefix checks, never persisted. - Retrieval is v2-only and analysis-scoped in SQL (no scope row -> none); scope is part of the Redis key. retrieve_knowledge pushes its scope down, drops a source_id post-filter that never matched, and whitelists metadata. - structured_flow adds the analysis's bound documents to the planner catalog. Live: BUMA PDF -> 28 chunks, 2 figures; figure signed and fetched (200 image/jpeg). Suite 939 passed / 7 skipped; ruff adds nothing; import main OK. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>b848f73
-
Merge dev_new to main (improve retriever and add querying) (#13)
- [noticket] add gitignore (c87f27f458854787102eebc110b178c2572ef8a6) - [NOTICKET]: add document pipeline, simplify document API (fb871f3d7db9c5a9c95a012761c4180a432e7e4b) - [NOTICKET]: update folder document_pipelines after pipelines (a4cf97ab3890e35831c0043a1d411c17d84f374b) - [NOTICKET][DB] refactor code to new repo (7f3bb9782bb9b6ace69e6be35377c12757a2f81d) - [KM-441] add mean and median (9b5933420821187e73bb72a896e28d7adf6d6137) - [NOTICKET] new metadata format for cleaner code (6b590d94c398ac4966a0e376f2809aa52c5ba022) - update document (5a69e0eecf80dd807565dbc369374e41450c37fc) - delete duplicate file (3848d7b262745b758069e3ab8089437ced42a45f) - edit document for new pipeline (425e021053c4de65e2598b0f5554b7c1044a904c) - [NOTICKET]: add CSV and XLSX file type (31920c3bc6f9faf9e07095a02fc1faf08adc246e) - [DB] fix/rename db_pipeline.py (d913315c356d741cc7a5a5b1f8e36ba41397f2ab) - [NOTICKET][DB] menyesuaikan format struktur db_pipeline sesuai dengan file lain (e13a9017e0f7fd26ef9b8a…52999bc
-
[KM-467] [DED][AI] Orchestration Agent - Init
- https://bukittechnology.atlassian.net/browse/KM-467027123c