Commit History
-
[NOTICKET] feat(knowledge-parsing): wire Mistral OCR backend end-to-end
Makes Mistral OCR the primary parser (MinerU kept as a fallback), runnable as `python -m src.knowledge_parsing.run --backend mistral` with no GPU. - config: backend="mistral" default + Mistral settings (include_blocks, bbox_annotation, table_format). fingerprint() adds the Mistral settings ONLY on that backend, so existing MinerU cache keys stay byte-identical. - mistral_ocr.py (new): the API call — upload -> ocr.process, key from env/.env, file_id with a signed-url fallback, optional per-image descriptions. - parse.py: _parse_mistral caches the response like the MinerU path; cache_key uses the model id as the parser version for Mistral. - run.py: dispatches to normalise_mistral for the Mistral backend (skips the MinerU-only content-drift check and word-boundary vocabulary), stamps parser_name/normaliser_version per backend. - pyproject: mistralai in the knowledge-parsing extra. NOTE: it needs pydantic>=2.11.2, which the pinned pydantic==2.10.3 does not satisfy; the repo…6856ea8
-
[NOTICKET] feat(knowledge-parsing): one runner, in Python rather than shell
Running the pipeline meant six environment variables and an interpreter path from another repo, typed out every time. This wraps it: py scripts/parse.py # Windows python3 scripts/parse.py # Linux Python, not shell, and that was learned the hard way. A bash version and a PowerShell version were both written first, and between them hit two traps in twenty minutes: - CRLF. core.autocrlf=true rewrites .sh on Windows checkout, and bash then dies on line one with `set: pipefail` — a message that names nothing. - A trailing backslash. The default input path contains spaces, so PowerShell quotes it, the final backslash escapes the closing quote, and the NEXT argument is swallowed: `--backend pipeline` ended up inside `--input`. `subprocess` takes an argument list and never goes through a shell, so neither is possible here. One file instead of two, and readable by anyone on the team without knowing both shells. The backend is chosen from…74acbf7
-
[NOTICKET] feat(knowledge-parsing): document parsing half of the knowledge pipeline
Implements §4 B2/B5/B6 and closes the four open seam items (S2, S3, S4, S6b/c/d) against the contracts.py draft circulated 2026-08-19. Additive and flag-gated: src/knowledge/ (Tesseract -> pgvector) is untouched. Seam — src/knowledge_parsing/contracts.py, pydantic per S6e: - ParsedDocument envelope: content_hash, n_pages, version, parser_name/version/ backend/config, schema_version, raw_output_dir. parser_backend is read from MinerU's _middle.json, so it records what actually ran, not what was configured — a MinerU change and a prompt change stay distinguishable (S2). - Heading structure is READ, not derived. Verified in MinerU's source: pipeline and vlm run identical text_level logic, and bbox is emitted by both. Heading availability is document-dependent, not backend-dependent — the earlier claim came from sampling the McGraw-Hill handbook, which has no numbered headings (S3). - source_wording on TermRecord; heading carried verbatim (S4). - page_idx/page_idxs, 0-based, n…5840d20 -
[NOTICKET] feat(knowledge_extraction): span filter — E1 recall 0.8537, matches the frozen baseline
Rifqi Hafizuddin +1: Claude Fable 5Adds the term-filter stage, the last free stage before the paid extraction call. Dependencies (approved): gliner>=0.2.13 (resolved 0.2.28), torch>=2.6 (resolved 2.11.0+cpu). Install verified clean — `import torchgen` succeeds, so not the corrupt-wheel failure recorded in the calibration doc. filters/span_filter.py runs the model on EVERY chunk, unconditionally. There is deliberately no relevance gate in front of it: a pre-filter's false negatives drop content before extraction sees it, and dropped content never reaches expert review to be caught. "Zero candidate spans" is the relevance signal instead. Two mechanical traps handled, both of which silently cost recall: - the model truncates long inputs and WARNS rather than failing, so chunks are fed as overlapping 130-word windows with character offsets remapped. Offsets are walked over the original spacing, never re-joined — re-joining would shift every offset in a chunk containing newlines and break every span check. - the ~12…3a01634 -
[NOTICKET] fix(knowledge_extraction): move runner into the package; fix two ranking defects
Rifqi Hafizuddin +1: Claude Fable 5Runner relocated scripts/run_knowledge_extraction.py -> the package as src/knowledge_extraction/cli.py. scripts/ is gitignored, and this runner is the pipeline's operator entry point, so it has to ship with the module: uv run --no-sync python -m src.knowledge_extraction.cli <artifact.json> pyproject: narrow the T201 lint exemption to that one CLI file rather than all of scripts/, so the ignore traces to this change and nothing else. Two ranking defects found while verifying against the real 9-page BUMA standard (both recorded as deviations V1/V2 in KNOWLEDGE_PIPELINE_CALIBRATION.md §9): V1 — a heading that names the term now counts as a mention at offset 0 for cue-proximity scoring. Indonesian standards are written "2.1.3 Physical of Availability (PA)" / "Adalah ketersediaan fisik...": the heading names the term and the body opens with the definitional cue WITHOUT repeating it, so the definition section earned no cue bonus at all. Measured before the fix, PA's own definition se…72605ff -
[NOTICKET] feat(knowledge_extraction): free stages — models, seam adapter, filters, cluster, ranking
Rifqi Hafizuddin +1: Claude Fable 5First slice of extraction v2, rebuilt in this repo (the kex prototype is not being ported). All stages here are free: zero API calls, CPU only. New module src/knowledge_extraction/ (named to avoid colliding with src/knowledge/, the existing OCR->pgvector service, which is untouched): - models.py pydantic v2 contracts. Encodes the three invariants: content fields Optional, subdomain_tags as an enum, provenance.span mandatory. Adds source_wording so the literal source wording survives instead of being normalised away. - adapter.py THE ONLY file that knows the seam's shape. Stages depend on an internal Chunk subset, so the artifact contract can settle either way without touching a stage. Verified against both live shapes: Sofhia's bare-list draft with its `kind` discriminator, and the document-level envelope proposed in the checklist.…024c30a
-
feat/Knowledge & Data Tools (#3)
- [KM-641] refactor(tools): trim analytics registry to 4 active tools (66e2e4df66f1abf7eaa7e442d9900d428696ece4) - [NOTICKET] docs: add checkpoint plan for 2026-06-17 sync (8c1bb63ebdee02f9db0da9fc2b2269823b0a7688) - [KM-642] refactor(tools): rename data-access tools to data/knowledge naming (c38c0c29d3e6b245f67d03f2c35e451900e1abdb) - [KM-643] feat(tools): merge data sources into data_check + add knowledge_check (4bd5f1efdc560a64092cc8df672a7990fa949b59) - [NOTICKET] docs: record KM-642/643 in checkpoint plan + PROGRESS (4c57588b90fe0cac8d0d4d22cfe5701fc9eb52dd) - [NOTICKET] fix(chat): scope response cache by user_id + drop TTL to 1h (R5) (b701e95d300c7d2480a3608b2d4da6dfe102b19a) - [NOTICKET] feat(slow-path): persist AnalysisRecord via PostgresAnalysisStore (09f845188226dd01633cf74525caa569e4a7d37a) - [NOTICKET] Adopt verb-first skill naming (2d6406def7ed45edd1226e96645ba2a5cdbfa3d0) - [NOTICKET] docs: sync checkpoint plan — verb-first tool names + report-as-API (3b399f84d73b0d98bd99…0721bb4
-
feat/Planner Agent (#2)
- [KM-567][AI] Planner agent: contracts, schemas, inputs, registry (be96601ad46c00e8e789768dd6e89c2db827bde4) - [KM-567][AI] Planner agent: prompt + few-shot examples (83ba6b1743370145454d9315224eeb7b2d12ac18) - [KM-567][AI] Planner agent: validator + service (72306d0ce92c76d60117a20c701c7f8185bcfeef) - [KM-567] docs: record Phase 3 Planner agent in PROGRESS.md (ba2fa8892392028c8d13646c09d1155bbf34e703) - [KM-567][AI] Planner agent: fix PII leak + placeholder/retry robustness (6b3637dda097f16cd98d4b9ad6094fdaed6d1346) - [KM-624] Implement Compute Functions Tools (78c598c22d2f2a14229a83c3f43fa98c5d520ae7) - [KM-624][AI] Planner: realign stub registry + examples to composite analyze_* tools (93a24dabf48d8121e981cc470c835cb10badb62a) - [KM-624] docs: record tool taxonomy + ownership revision in PROGRESS (f3461142542f6216aff23b30a1321dfb4654afdc) - [KM-626][AI] Slow-path agent: seam contracts + TaskRunner (ac310de551d0888ccfef31e981e22d88f54c6a38) - [KM-626][AI] Slow-path agent: Assembler…81e5fe7
-
[NOTICKET] fix: add SKIP_INIT_DB flag and remove pymongo dependency
- Add SKIP_INIT_DB env variable to skip database initialization on startup, useful for local dev when DB is not available - Remove pymongo dependency (unused, replaced by PostgreSQL-based user storage) - Comment out unused MongoDB settings in Settings class - Update .gitignore: add API_CONTRACT_AGENT*.md and migratego/ - Remove .vscode/launch.json (superseded by uv run fastapi dev)19bd037
-
feat/Catalog Retrieval System (#1)
- initial commit (8557a2045ca2c0873e07efdf6227064b83e4a1e5) - initial structure (efc0c0a8b2444014cbe74f04617c3c974d9915be) - [KM-553] initialize shared contract (f31f6731692b0199d8cb5c5d29dffa466d6d4ca0) - [KM-553] migrate retrieval layer and remove obsolete rag/tools modules (df220ea47c96eb8a1efa07c6eebef4604fc308d6) - [NOTICKET] catalog enricher + structured pipeline + on_db_registered (77ea82d29d86400c5ed30ac154be8df6a7e897eb) - Merge commit 'refs/pr/1' of https://huggingface.co/spaces/DataEyond/Agentic-Service-Data-Eyond-Catalog into pr/1 (0ae8717c7efce2cda14fe4b8caecd774459b1f6e) - [NOTICKET] SqlCompiler (Postgres) + DbExecutor (212dad360714508c1aa607995039e6818fa40f78) - Merge commit 'refs/pr/1' of https://huggingface.co/spaces/DataEyond/Agentic-Service-Data-Eyond-Catalog into pr/1 (faf8a4f6d7e06ccf1ce84c2e7f4990c79f27f843) - TabularIntrospector + on_tabular_uploaded trigger + 31 tests (ca425208969b72a20ed76f86ef0faa1a0e912339) - [KM 556] add columns field to QueryResult + update…6bff5d9
-
[KM-438][KM-439] Improve Retrieval and Querying feature (#15)
- [noticket] add gitignore (c87f27f458854787102eebc110b178c2572ef8a6) - [NOTICKET]: add document pipeline, simplify document API (fb871f3d7db9c5a9c95a012761c4180a432e7e4b) - [NOTICKET]: update folder document_pipelines after pipelines (a4cf97ab3890e35831c0043a1d411c17d84f374b) - [NOTICKET][DB] refactor code to new repo (7f3bb9782bb9b6ace69e6be35377c12757a2f81d) - [KM-441] add mean and median (9b5933420821187e73bb72a896e28d7adf6d6137) - [NOTICKET] new metadata format for cleaner code (6b590d94c398ac4966a0e376f2809aa52c5ba022) - update document (5a69e0eecf80dd807565dbc369374e41450c37fc) - delete duplicate file (3848d7b262745b758069e3ab8089437ced42a45f) - edit document for new pipeline (425e021053c4de65e2598b0f5554b7c1044a904c) - [NOTICKET]: add CSV and XLSX file type (31920c3bc6f9faf9e07095a02fc1faf08adc246e) - [DB] fix/rename db_pipeline.py (d913315c356d741cc7a5a5b1f8e36ba41397f2ab) - [NOTICKET][DB] menyesuaikan format struktur db_pipeline sesuai dengan file lain (e13a9017e0f7fd26ef9b8a…c93ec90
-
Make a query for tabular (XLSX and CSV) (#14)
- [NOTICKET] add software to gitignore (d43ecb180036339cf287e93f1d9116bd6eff9b9d) - [NOTICKET] add pyarrow (e50eadc82e160012e3319267f5bfe084cd9034d4) - [KM-515][document] Make Query for Tabular Type (XLSX & CSV) (695ca0a154a68077c51499ed83b7507b988be065) - [KM-455][document] decided methods retrieval for document (8c9cc79223eb1d96c8622d129219a83a4ba2500b) Co-authored-by: Sofhia Az-Zahra <sofhiaazzhr@users.noreply.huggingface.co>8802920
-
Merge dev_new to main (improve retriever and add querying) (#13)
- [noticket] add gitignore (c87f27f458854787102eebc110b178c2572ef8a6) - [NOTICKET]: add document pipeline, simplify document API (fb871f3d7db9c5a9c95a012761c4180a432e7e4b) - [NOTICKET]: update folder document_pipelines after pipelines (a4cf97ab3890e35831c0043a1d411c17d84f374b) - [NOTICKET][DB] refactor code to new repo (7f3bb9782bb9b6ace69e6be35377c12757a2f81d) - [KM-441] add mean and median (9b5933420821187e73bb72a896e28d7adf6d6137) - [NOTICKET] new metadata format for cleaner code (6b590d94c398ac4966a0e376f2809aa52c5ba022) - update document (5a69e0eecf80dd807565dbc369374e41450c37fc) - delete duplicate file (3848d7b262745b758069e3ab8089437ced42a45f) - edit document for new pipeline (425e021053c4de65e2598b0f5554b7c1044a904c) - [NOTICKET]: add CSV and XLSX file type (31920c3bc6f9faf9e07095a02fc1faf08adc246e) - [DB] fix/rename db_pipeline.py (d913315c356d741cc7a5a5b1f8e36ba41397f2ab) - [NOTICKET][DB] menyesuaikan format struktur db_pipeline sesuai dengan file lain (e13a9017e0f7fd26ef9b8a…52999bc
-
[NOTICKET] fix error in imports
Rifqi Hafizuddin9d87441 -
[KM-436][KM-437] add and modify for database and document pipeline (#3)
- [noticket] add gitignore (c87f27f458854787102eebc110b178c2572ef8a6) - [NOTICKET]: add document pipeline, simplify document API (fb871f3d7db9c5a9c95a012761c4180a432e7e4b) - [NOTICKET]: update folder document_pipelines after pipelines (a4cf97ab3890e35831c0043a1d411c17d84f374b) - [NOTICKET][DB] refactor code to new repo (7f3bb9782bb9b6ace69e6be35377c12757a2f81d) - [KM-441] add mean and median (9b5933420821187e73bb72a896e28d7adf6d6137) - [NOTICKET] new metadata format for cleaner code (6b590d94c398ac4966a0e376f2809aa52c5ba022) - update document (5a69e0eecf80dd807565dbc369374e41450c37fc) - delete duplicate file (3848d7b262745b758069e3ab8089437ced42a45f) - edit document for new pipeline (425e021053c4de65e2598b0f5554b7c1044a904c) - [NOTICKET]: add CSV and XLSX file type (31920c3bc6f9faf9e07095a02fc1faf08adc246e) - [DB] fix/rename db_pipeline.py (d913315c356d741cc7a5a5b1f8e36ba41397f2ab) - [NOTICKET][DB] menyesuaikan format struktur db_pipeline sesuai dengan file lain (e13a9017e0f7fd26ef9b8a…2ba0613
-
[KM-467] [DED][AI] Orchestration Agent - Init
- https://bukittechnology.atlassian.net/browse/KM-467027123c