Commit History
-
[NOTICKET] feat(knowledge-parsing): add 1-based page_no (S6b)
A computed `page_no` (= page_idx + 1) on Chunk, so the human/review-UI page number lives in the artifact instead of each consumer converting it. - Derived, never stored: cannot drift from page_idx, and neither normaliser has to set it. page_idx stays 0-based and raw. - Additive, schema 0.3.2 -> 0.3.3; older artifacts and consumers are unaffected. - Closes S6b (open since 2026-08-19); reverses the earlier "convert in the UI" note now that a real reviewer is about to read these. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>717452f -
[NOTICKET] feat(knowledge-parsing): wire Mistral OCR backend end-to-end
Makes Mistral OCR the primary parser (MinerU kept as a fallback), runnable as `python -m src.knowledge_parsing.run --backend mistral` with no GPU. - config: backend="mistral" default + Mistral settings (include_blocks, bbox_annotation, table_format). fingerprint() adds the Mistral settings ONLY on that backend, so existing MinerU cache keys stay byte-identical. - mistral_ocr.py (new): the API call — upload -> ocr.process, key from env/.env, file_id with a signed-url fallback, optional per-image descriptions. - parse.py: _parse_mistral caches the response like the MinerU path; cache_key uses the model id as the parser version for Mistral. - run.py: dispatches to normalise_mistral for the Mistral backend (skips the MinerU-only content-drift check and word-boundary vocabulary), stamps parser_name/normaliser_version per backend. - pyproject: mistralai in the knowledge-parsing extra. NOTE: it needs pydantic>=2.11.2, which the pinned pydantic==2.10.3 does not satisfy; the repo…6856ea8 -
[NOTICKET] feat(knowledge-parsing): Mistral OCR block normalizer (section default)
Mistral OCR (include_blocks=True) block stream -> per-section Chunk[], the counterpart of normalize.py for the Mistral backend. - Reuses normalize._heading and render.py so section boundaries and prose rendering match the MinerU path; artifact shape is identical (seam unchanged, extraction never learns which parser ran). - Strips markdown markup Mistral adds (not present in the source PDF) so the extraction span-check still matches quotes literally; whitespace kept verbatim. - granularity="section" (default) populates heading_path/heading/section_no, which extraction needs for outline + source_wording; "page" is recall-only. - Validated on the BUMA fixture through extraction's own adapter + span filter: section recall 0.878, page 0.951 (both above MinerU section 0.829). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>81b4f4e
-
[NOTICKET] refactor(knowledge-extraction): BriefContext -> DomainContext
Rifqi Hafizuddin +1: Claude Opus 5The workstream diagram has always called this box "Domain Knowledge", while the model and its fields described a short orientation card. Rename the entity to match the intent before anything is persisted: once an expert has approved rows keyed on these ids, changing the `b_` prefix is an id migration that loses the approvals - the exact failure the deterministic-id scheme exists to prevent. BriefContext -> DomainContext brief_id -> domain_id (prefix b_ -> d_) ExtractionResult.brief -> .domain brief_context.json -> domain_context.json Deliberately unchanged: - `Branch = "summary"` and GUARDED_FIELDS["summary"]. The branch literal is the measurement key in CallUsage.branch / RejectedField.branch; renaming it would break comparability with the committed eval baseline. - prompts/summary.txt. Editing prompt text needs the eval path, and the rewrite belongs with the v3 schema change, not with a rename. No field, prompt or guarding change - the …7f47c61
-
[NOTICKET] feat(knowledge-parsing): carry the document's human-readable title
`doc_id` is on its way to becoming an identifier rather than a label. Every `term_id` / `formula_id` / `rule_id` / `brief_id` downstream is a digest of it, so it must stop depending on what someone typed into a Save dialog — which it does today, being the file stem. A long name makes the cost visible: chunk_id: "STD_2026_006_MNO_ Rev.0.0 - Production Parameter and ECA::0000" That name is embedded in all 31 chunks and then in every entry derived from them. Renaming `Rev.0.0` to `Rev 0.0` re-keys the lot. `source_title` is the field the name moves to. It equals `doc_id` today, and that is the point: the two are now separate, so the identifier can change without the human-readable name going with it. `Provenance` is what a reviewing expert actually reads, and it carries `doc_id` — without this field they would be shown `f10807178b20531e` instead of the standard's name. Added ahead of the switch because it is needed whichever identifier wins (content hash or the Go service's upload…df571be
-
[NOTICKET] feat(knowledge-parsing): commit the reference artifact, and stop the stamp churning
Three changes, all aimed at one problem: the two halves of the pipeline could not reliably agree on which artifact they were measuring. **Metadata now precedes `chunks` in the artifact.** Pydantic serialises in declaration order, so it decides what a person sees on opening the file. With `chunks` first, checking which parser or normaliser produced an artifact meant scrolling past thousands of lines to the end — and that is a routine check now, not a rare one. JSON objects are unordered by specification and every consumer reads by key, so this changes nothing for code and `schema_version` is unchanged: no field, type or meaning moved. **`contracts.py` is out of the normaliser hash.** The artifact's SHAPE is already versioned by `schema_version`, so hashing it again counted the same change twice. The reordering above proved the cost: the stamp moved while the chunks came out byte-identical, which is a false "your artifact is stale". A signal that cries wolf gets ignored, and this one e…a0e59e4 -
[NOTICKET] feat(knowledge-parsing): stamp the artifact with the code that built it
A chunks.json is a frozen snapshot of the normalisation code at the moment it ran. When that code is fixed, existing artifacts do not update themselves — and nothing in their metadata said so. `parser_config` fingerprints MinerU's settings; `schema_version` versions the artifact's SHAPE, which a normalisation bugfix does not change. So a stale artifact and a correct one declared identical metadata, and the only observable difference was the character count. That cost a day. A pre-fix artifact reached the extraction half, term recall came back 0.7073 against a 0.8049 baseline, and it read as a regression in extraction's own work. It was neither side's setup: same filter code, different input. Verified here — scoring that artifact reproduces 169 mentions and 0.7073 exactly, against 186 and 0.8293 for the rebuilt one. `normaliser_version` is a hash of the modules that decide what a chunk CONTAINS: normalize.py, render.py, checks.py, contracts.py. A source hash rather than a hand-mainta…c64a0cb -
[NOTICKET] docs(knowledge-parsing): naming convention, and why it is a stopgap
The file name is currently a primary key, not a label: `doc_id` is the file stem, `chunk_id` is built from it, and `term_id` / `formula_id` / `rule_id` / `brief_id` are all digests of `doc_id` in `knowledge_extraction/ids.py`. Renaming a file after ingestion therefore re-keys every identifier an expert's approve/reject decisions hang on. That was worth writing down, because nothing in the code says it. The convention itself is ordinary — stable doc code, machine-readable revision suffix, no spaces, never `(1)`/`(2)`, never rename after ingestion. `(2)` gets its own line because it means "downloaded twice", not "version 2", and ingested as-is it becomes a second legitimate document with a full set of identifiers. What matters more is the section saying it is a STOPGAP. It holds only because we control the file names while parsing is an admin-triggered batch. Behind the upload endpoint that is planned after eval and the playground it stops holding: people will upload `Copy of SOP final…942f14e -
[NOTICKET] fix(knowledge-parsing): long names, name clashes, and a GPU check in the wrong place
Four defects, all found by asking what happens when a real person uploads a real file rather than the two well-named documents in the test folder. **Long file names failed the parse.** MinerU names its output folder after `pdf_file_names` and then writes images into it under a full SHA-256 filename. Measured on this repo the fixed part of the path costs 217 characters, leaving 43 for the name — and the BUMA standard's own name is 56. `_mineru_name` now passes a bounded name instead. Head AND tail are kept, because the names that collide in practice share a long prefix and differ at the end ("… and ECA" versus "… and ECA (2)"); keeping only the head would render exactly those indistinguishable. Verified with a real parse of the 56-character name: 239 characters, 31 chunks, where before it died on MAX_PATH. Uniqueness does not depend on that name — the parent folder is the cache key, which is the full SHA-256 of the file contents. It is cosmetic, exists so the cache can be browsed whil…9083dd1
-
[NOTICKET] fix(knowledge-parsing): normalize.py silently dropped every list block
MinerU keeps list content in `list_items` and gives such items no `text` field. They therefore fell through the generic text path, which reads `item["text"]`, came back empty, and were skipped — no error, no warning, no trace. This is the same shape of bug the chart/image branch was written to prevent, and that branch's comment describes it exactly: "pictures have no such field — so BOTH its caption and its description vanish in silence". Lists have no such field either, and nobody noticed. Measured cost on the BUMA standard: two dropped blocks took five gold terms with them — Joint survey, Truck count, EWH, Controllable, Hierarki Level Analysis. None of the five were "missed by the model"; none were in the artifact at all. Term-filter recall 0.7073 -> 0.8293, +0.1220. List content joins the running text chunk rather than opening its own, because it is prose inside its section. It deliberately skips heading detection: a short bullet can look like a heading, and letting one open a se…6edfbe7 -
[NOTICKET] docs(knowledge-parsing): settle determinism; finish the English pass
Determinism was the one thing left open on the backend decision: the VLM family is not deterministic in general, so a frozen baseline might drift and no eval number could be trusted. It does not drift. Two runs over the same documents, with the parse cache moved aside so both genuinely re-parsed (from_cache: false on all four records), produced 0 differences across 50 chunks. All four image descriptions were present in both runs and character-for-character identical (192, 256, 77, 146 chars) — those are the only model-written content in the artifact and were the likeliest to wobble. Existing baselines therefore stay valid and need no re-measurement, and hybrid/high is final rather than provisional. Marked settled in config.py, calibration §13, and REPO_STATUS §14. The parsing README was still Indonesian in the how-to-use sections written last week, while the .py files had already moved to English; it is now English throughout. Its console example was also stale — it showed the old I…d5ed50c
-
[NOTICKET] refactor(knowledge-parsing): English throughout the module
The parsing half was written in Indonesian — identifiers, docstrings, comments, manifest keys and console output. Everything else in the repo is English, and this module is the newest work, so it is the one to bring into line rather than the older code around it. Renamed across the module: parse_dokumen -> parse_document, normalisasi -> normalise, pulihkan_batas_kata -> restore_word_boundaries, kosakata_layak -> trustworthy_vocabulary, periksa_items -> check_items, CatatanDokumen -> DocumentRecord, and the locals with them. Docstrings and comments were REWRITTEN rather than translated word for word. Several carry the reasoning behind a decision — why `\frac` is unwrapped before the braced wrappers, why a heading never enters `text`, why the digit threshold is 3 — and a mechanical translation would have flattened exactly the part worth keeping. Manifest keys move too: detik_per_halaman -> seconds_per_page, peringatan -> warnings, jenis -> kind, galat -> error. That is a data change, …7f153d6
-
[NOTICKET] docs(knowledge-parsing): how to run it, and how to read what comes back
The README still said `python -m src.knowledge_parsing.run` with no environment set, and pointed at `uv sync --extra knowledge-parsing` — a command that fails. Four sections that answer questions actually asked this week. **How to run it**, and what a successful run looks like, with real console output. First-time readers cannot tell whether `⚠ 3 peringatan` is routine or alarming, and that is the one line worth reading: warnings are normal, `ok (cache)` means nothing was re-parsed, and only `ok / gagal` needs attention. **Backend**, with the measured table and why hybrid/high won — including that `vlm` is identical but slower and `pipeline` fuses the multiplication sign. Stability is marked open, not settled. **Running without a GPU**: the runner drops to `pipeline` on its own, and the result is valid but LOWER FIDELITY — no image descriptions, multiplication copied as a letter. Easy to forget, so it is stated where it will be seen. **Installing MinerU**, with an explicit do-not-…b3eae0f -
[NOTICKET] feat(knowledge-parsing): default to hybrid/high, the measured choice
All four backends run on the same GPU, same two documents, 29 pages: hybrid/high 4.85 s/page image descriptions YES hybrid/medium 1.88 s/page descriptions FORCE-DISABLED by MinerU pipeline 2.67 s/page never analyses images at all vlm 6.96 s/page output IDENTICAL to hybrid/high, 100% Two findings decided it. `vlm` and `hybrid/high` produce the same LaTeX and the same descriptions, so `vlm` only costs more time for nothing. And `pipeline` loses twice: slower than hybrid/medium, and the only backend that copies the multiplication sign as a literal `x`, which fuses `MOHH x Qty x PA` into one token and takes five terms with it. The `high` / `medium` gap is $6.89 against $2.67 across the whole 6,800-page BUMA corpus — $4, too small to trade image descriptions for. `effort` moves to "high" in the same commit, and that is not cosmetic: MinerU force-disables image analysis at medium, so flipping the backend alone would have produced hybrid WI…52e1bd4 -
[NOTICKET] fix(knowledge): settle the chunk_id separator on `::`
Rifqi Hafizuddin +1: Claude Opus 5The two halves minted provenance keys differently: the parsing half emits `<doc_id>::<seq>`, while the extraction adapter fell back to `<doc_id>#<seq>` for artifacts arriving without one. Nothing broke in practice — the adapter prefers the supplied id, so real runs always carried `::` — but the two shapes could diverge the moment a fixture or an older artifact took the fallback. `::` wins: it is what the producer actually emits, the calibration doc already uses it in prose, and `#` is URL-fragment syntax that would bite once these ids become MCP resource identifiers. The `KNOWLEDGE_OUTPUT_CONTRACT.md` cited as the source of `#` does not exist in the repo or its history. The separator is now stated in the contract next to the field, because this is the second time an undocumented detail of the seam's shape has cost both halves a round trip. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>e8cc26c -
[NOTICKET] feat(knowledge-parsing): carry image descriptions, apart from the text
MinerU describes images and charts, but only on some backends. Read from its own source (is_image_analysis_option_visible): never on `pipeline`, always on `vlm`, and on `hybrid` only at effort=high — medium disables it outright. Two faults kept that description from ever arriving. `normalize.py` had no branch for `image` items at all. Different backends label the same picture differently: what `pipeline` calls a chart, `vlm` calls an image. Image items fell through to the generic text path, which reads only item["text"] — a field pictures do not have — so caption AND description were dropped in silence. That is most of what looked like "vlm loses chart detection" in yesterday's comparison: vlm found them, we discarded them. The second is worse and was waiting to fire. `_teks_chart` appended `content` straight into Chunk.text. Harmless today because `pipeline` always leaves it empty — but the moment the backend flips to vlm, model-written prose starts flowing into the field the extra…c113bf8 -
Merge commit 'refs/pr/20' of https://huggingface.co/spaces/DataEyond/Agentic-Service-Data-Eyond-Catalog into pr/20
6e78f1e -
[NOTICKET] docs: reorganise into docs/ and give every doc a status banner
Rifqi Hafizuddin +1: Claude Opus 5The root carried 14 markdown files and no way to tell which were authoritative. That knowledge lived in CLAUDE.md §1 and in Rifqi's head, which works for us and fails for anyone else — and one of the stale ones already has a named failure mode ("The Stale-Doc Believer") for being implemented from. Root now holds four: REPO_STATUS, DEV_PLAN, API_CONTRACT_BE_PYTHON, README. Everything else moves under docs/, by git mv so history follows: docs/knowledge/ context, TODO, calibration, parsing output contract docs/plans/ SPINE_V2_PLAN (approved, W3 deferred), DS_TOOLS_PLAN (proposal) docs/archive/ API_ENDPOINTS_RESTRUCTURE, CODE_REVIEW_2026-07-23, PROJECT_BRD docs/reference/ API_CONTRACT_BE_GOLANG (the Go repo's, a copy that can drift) Every doc gains a one-line status banner under its H1, marked <!-- doc-status -->: LIVE / LIVE PLAN / PROPOSAL / ARCHIVED / REFERENCE. Archived ones name what superseded them — API_ENDPOINTS_RESTRUCTURE now opens with "DO NOT IMPLEMENT FROM…f1342ce -
[NOTICKET] fix(knowledge-parsing): let the source prove itself before trusting it
Word-boundary recovery treats the PDF text layer as the arbiter. That is safe when the layer is intact and dangerous when it is HALF there: a legitimate word looks unknown, and gets split. An empty text layer was already safe — no vocabulary, no recovery. The gap was the middle ground, and nothing measured it. So the page now has to prove itself: how many of MinerU's own prose words on that page are actually found in its text layer. A low score means the two read the same page differently, and a source that disagrees with the parser is not fit to arbitrate it. Pages scoring below 90%, or carrying too little prose to judge at all, lose their vocabulary and skip recovery — left damaged but honest, rather than repaired into something the document never said. This replaces a constant tuned against one document with a measure that assesses each page on its own evidence. Measured across three documents, two of them a McGraw-Hill textbook in English with nothing in common with the BUMA st…22c6ad1 -
[NOTICKET] fix(knowledge-parsing): read word boundaries back from the source PDF
MinerU writes formulas one character at a time ("T o t a l H o u r s"), and once those are rejoined the word boundaries are gone: "Total Hours" and "TotalHours" are no longer distinguishable from the LaTeX alone. Everything fuses into a single token. The multiplication written as a letter "x" is not a separate bug. It is the same one — an "x" between terms is just another spaced character, so "MOHH x Qty x PA x UA x Pty" collapses to "MOHHxQtyxPAxUAxPty". Two obvious fixes were tried and rejected: - Rewriting a lone "x" to a multiplication sign guesses, and guesses wrong on any document with a variable named x, silently. It also leaves "TotalHours" untouched, which is the larger half of the damage. - Emitting a warning fires on 7 of 11 equations. A warning that lights up on 64% of items is not actionable, and recovers nothing. The source PDF still holds the boundaries: its text layer reads "MOHH x Qty x PA x UA x Pty", properly spaced. So the answer is read from t…e047f7a -
[NOTICKET] fix(knowledge-parsing): render_latex was deleting division (X21)
Rifqi's three faults, all reproduced against the document's own 11 equations and all fixed. Two of his numbers were off, and the corrections matter: - It hit 4 of 11 equations, not 2 — PA, PA_ij, UA_ij and PTY_ij. Not just PA and UA: every composite form was affected too. - The nested braces come from mathrm{} and sum _{ij}, NOT from text{}, which this standard never uses. A fix written against text{} would have repaired the synthetic example and no real equation. (C), the serious one: the frac rewrite needed a brace-free argument, could not match a nested numerator, and the unmatched frac then fell through to the generic "drop leftover commands" pass. The division was deleted, leaving a formula that reads as a product and states something the document does not. The ordering decision this needed: braced wrappers are unwrapped BEFORE frac, and the unwrap consumes its braces. The old pass removed only the command name and left { } behind — those leftover braces were wh…a356e3c -
[NOTICKET] fix(knowledge-parsing): stop the warning marker from failing documents
A Windows console defaults to cp1252, which has no "warning" glyph. Printing the quality-warning line raised UnicodeEncodeError INSIDE the per-document try block, so a document that carried warnings was recorded as FAILED — even though its parse had already succeeded and been written to the cache. The effect was the exact inverse of the intent: clean documents passed, and the ones whose quality needed attention were the ones thrown away. The same throw in the run summary then took down the whole run with exit code 1. Reproduced on the 9-page BUMA standard, which carries three warnings: it was reported as "gagal" while its 31 chunks sat complete on disk. stdout and stderr are now reconfigured to UTF-8 at CLI entry, so the run no longer depends on the host codepage. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>df84654 -
[NOTICKET] fix(knowledge-parsing): send MinerU a backend name it accepts
The GPU flip config.py advertises did not work. MinerU dispatches on `backend == "pipeline"`, then `startswith("vlm-")` and `startswith("hybrid-")`. A bare "vlm" matches none of them, so MinerU did nothing at all, silently — and the failure surfaced here as "content_list not found", which points at the wrong place entirely. Valid names are pipeline / vlm-engine / hybrid-engine. The short names stay in the config because they are part of the cache key and read better; the translation happens at the boundary, just before MinerU is touched. Nobody would have caught this until the day a GPU arrived and everyone was waiting on the result. Also: the temp directory used with_suffix(), which REPLACES the last suffix rather than appending. Cache folders end in the MinerU version ("...-3.4.4"), so ".4" was thrown away and two versions collided on one temp directory that the next line deletes without asking. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>8e93c60
-
[NOTICKET] docs(knowledge-parsing): module README + refresh the parsing/seam statuses
README follows the per-module convention already used by src/catalog, src/query, src/retrieval and src/security, and records the three things that are easy to get wrong and expensive to discover: text stays verbatim, page_idx is 0-based, and markup never goes in text (with the recall numbers behind that last one). KNOWLEDGE_PIPELINE_TODO.md — parsing (B) and seam (S) rows only; P/X/D/E/F and S7/S8 are untouched: - B6, B8, S2, S3, S4 -> done. Contract landed, envelope carries parser identity read back from MinerU's _middle.json, section-structure claim corrected against MinerU's source, source_wording added. - B4, B5, S1, S5, S6 -> in progress, deliberately NOT marked done: B5's acceptance test needs the Open Pit textbook (S7) before the NSR cross-page case can be claimed; S5 is satisfied on the parsing side but is Rifqi's to close; B4's language question is answered but text quality is not yet judged. Marking unproven work done is what makes a checklist useless. - S6: (c)…f0cf168
-
[NOTICKET] fix(knowledge-parsing): render markup to prose in text; only numbered headings split sections
Both defects were measured by the extraction side on the same document and gold set, changing only the parse: PyMuPDF (plain text) recall 0.8537 35/41 MinerU, raw markup in text recall 0.7561 31/41 MinerU, formula + table rendered recall 0.8293 34/41 1. Chunk.text now carries readable prose, not markup. The term filter is an NER model reading prose. MinerU writes formulas with every character spaced out — "{ \mathrm { P u r c h a s i n g ~ c o s t s } }" — and tables as HTML; neither yields a single mention for terms the same model finds easily in plain text. render.py rejoins spelled-out words and numbers, folds subscripts, maps operators to non-letter glyphs (a literal "x" for \times would merge into the token beside it), and flattens tables to " | "-separated rows. The markup is kept, not discarded: Chunk.latex and Chunk.table_html hold it verbatim, because the formula branch needs exactly that form. Found while verifying: table cells carr…2cb1336 -
[NOTICKET] fix(knowledge-parsing): keep the heading line out of Chunk.text
The heading was briefly copied into `text` as its first segment, to stop the chunk that defines a term from containing no mention of that term — the BUMA standard names the term in the heading and opens the body with "Adalah ..." without repeating it. The extraction half already solves this, and better, because it measured it on the document: `validate/span_check.py` builds its haystack as heading + text, and `rank/evidence.py` treats a heading that names the term as a mention at offset 0. So copying it into `text` only causes harm: the heading appears twice in that haystack, and its occurrence becomes a real mention, inflating mention_count. That count is what the frozen prototype baseline is compared against (169 mentions -> 66 clusters), so v2 numbers would drift for a reason that has nothing to do with extraction quality. `heading` stays verbatim, which is what both halves actually needed. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>15028da -
[NOTICKET] feat(knowledge-parsing): document parsing half of the knowledge pipeline
Implements §4 B2/B5/B6 and closes the four open seam items (S2, S3, S4, S6b/c/d) against the contracts.py draft circulated 2026-08-19. Additive and flag-gated: src/knowledge/ (Tesseract -> pgvector) is untouched. Seam — src/knowledge_parsing/contracts.py, pydantic per S6e: - ParsedDocument envelope: content_hash, n_pages, version, parser_name/version/ backend/config, schema_version, raw_output_dir. parser_backend is read from MinerU's _middle.json, so it records what actually ran, not what was configured — a MinerU change and a prompt change stay distinguishable (S2). - Heading structure is READ, not derived. Verified in MinerU's source: pipeline and vlm run identical text_level logic, and bbox is emitted by both. Heading availability is document-dependent, not backend-dependent — the earlier claim came from sampling the McGraw-Hill handbook, which has no numbered headings (S3). - source_wording on TermRecord; heading carried verbatim (S4). - page_idx/page_idxs, 0-based, n…5840d20