Commit History
-
[NOTICKET] refactor(knowledge-parsing): English throughout the module
The parsing half was written in Indonesian — identifiers, docstrings, comments, manifest keys and console output. Everything else in the repo is English, and this module is the newest work, so it is the one to bring into line rather than the older code around it. Renamed across the module: parse_dokumen -> parse_document, normalisasi -> normalise, pulihkan_batas_kata -> restore_word_boundaries, kosakata_layak -> trustworthy_vocabulary, periksa_items -> check_items, CatatanDokumen -> DocumentRecord, and the locals with them. Docstrings and comments were REWRITTEN rather than translated word for word. Several carry the reasoning behind a decision — why `\frac` is unwrapped before the braced wrappers, why a heading never enters `text`, why the digit threshold is 3 — and a mechanical translation would have flattened exactly the part worth keeping. Manifest keys move too: detik_per_halaman -> seconds_per_page, peringatan -> warnings, jenis -> kind, galat -> error. That is a data change, …7f153d6
-
[NOTICKET] fix(knowledge-parsing): let the source prove itself before trusting it
Word-boundary recovery treats the PDF text layer as the arbiter. That is safe when the layer is intact and dangerous when it is HALF there: a legitimate word looks unknown, and gets split. An empty text layer was already safe — no vocabulary, no recovery. The gap was the middle ground, and nothing measured it. So the page now has to prove itself: how many of MinerU's own prose words on that page are actually found in its text layer. A low score means the two read the same page differently, and a source that disagrees with the parser is not fit to arbitrate it. Pages scoring below 90%, or carrying too little prose to judge at all, lose their vocabulary and skip recovery — left damaged but honest, rather than repaired into something the document never said. This replaces a constant tuned against one document with a measure that assesses each page on its own evidence. Measured across three documents, two of them a McGraw-Hill textbook in English with nothing in common with the BUMA st…22c6ad1 -
[NOTICKET] fix(knowledge-parsing): read word boundaries back from the source PDF
MinerU writes formulas one character at a time ("T o t a l H o u r s"), and once those are rejoined the word boundaries are gone: "Total Hours" and "TotalHours" are no longer distinguishable from the LaTeX alone. Everything fuses into a single token. The multiplication written as a letter "x" is not a separate bug. It is the same one — an "x" between terms is just another spaced character, so "MOHH x Qty x PA x UA x Pty" collapses to "MOHHxQtyxPAxUAxPty". Two obvious fixes were tried and rejected: - Rewriting a lone "x" to a multiplication sign guesses, and guesses wrong on any document with a variable named x, silently. It also leaves "TotalHours" untouched, which is the larger half of the damage. - Emitting a warning fires on 7 of 11 equations. A warning that lights up on 64% of items is not actionable, and recovers nothing. The source PDF still holds the boundaries: its text layer reads "MOHH x Qty x PA x UA x Pty", properly spaced. So the answer is read from t…e047f7a
-
[NOTICKET] feat(knowledge-parsing): document parsing half of the knowledge pipeline
Implements §4 B2/B5/B6 and closes the four open seam items (S2, S3, S4, S6b/c/d) against the contracts.py draft circulated 2026-08-19. Additive and flag-gated: src/knowledge/ (Tesseract -> pgvector) is untouched. Seam — src/knowledge_parsing/contracts.py, pydantic per S6e: - ParsedDocument envelope: content_hash, n_pages, version, parser_name/version/ backend/config, schema_version, raw_output_dir. parser_backend is read from MinerU's _middle.json, so it records what actually ran, not what was configured — a MinerU change and a prompt change stay distinguishable (S2). - Heading structure is READ, not derived. Verified in MinerU's source: pipeline and vlm run identical text_level logic, and bbox is emitted by both. Heading availability is document-dependent, not backend-dependent — the earlier claim came from sampling the McGraw-Hill handbook, which has no numbered headings (S3). - source_wording on TermRecord; heading carried verbatim (S4). - page_idx/page_idxs, 0-based, n…5840d20