Commit History
-
[NOTICKET] fix(knowledge_extraction): close the notation rules against real backend markup
Rifqi Hafizuddin +1: Claude Opus 5Sofhia supplied the same 11 equations as MinerU transcribed them under four backends -- 44 real strings, committed as a fixture. Running the shipped normaliser over them reproduced her figures exactly: 11/11 for vlm/medium, 10/11 for hybrid/medium, 8/11 for pipeline, against hybrid/high as reference. Her independent check and this code agree on both the counts and which equations differ, which is the point of her having run it separately. The remaining gaps were in the command inventory, and the inventory is now evidence rather than guesswork. \mathsf and \displaystyle occur in this corpus and were missing -- \mathsf sets the whole of equation 7 under pipeline, including its multiplication sign, and \displaystyle appears in equation 4. Both are handled now, and the wrapped-x rule generalises across every presentational wrapper rather than being \mathrm specific. Equation 4 needed two more, and one of them was a decision Sofhia deliberately left open. The array ALIGNMENT SPEC arrives…01c76dd -
[NOTICKET] fix(knowledge_extraction): rule.txt few-shots stop contaminating the gold set (X23)
Rifqi Hafizuddin +1: Claude Opus 5Two of the four worked examples were lifted from the reference corpus, and they contaminated the measurement in opposite directions. Example 2 used "Waterfall Analysis dapat di-review melalui dua metode" and taught that it is NOT a rule. The gold set records that exact sentence as WATERFALL_TWO_METHODS. The prompt was instructing the model to fail one gold rule by construction, so the branch could not score above 14/15. Example 1 was the inverse and had gone unnoticed: its evidence is gold rule PTY_PRODUCTION_SOURCE verbatim (joint survey / truck count). That is a free hit inflating the score, not capping it. Both are now synthetic, which is the reasoning examples 3 and 4 already carried in the prompt itself. The closing note is generalised: all four are synthetic, so neither an answer nor a NON-answer for a real document leaks in. The gold set was deliberately NOT touched, following the X22 precedent -- the prompt is tuned constantly and the gold is the measuring instrument, so th…7e8614f -
[NOTICKET] feat(knowledge_extraction): record which guarantee formula_latex carries
Rifqi Hafizuddin +1: Claude Opus 5Three paths leave formula_latex populated and they are not equally strong: located in the source markup, no markup to check against, or an operator the source cannot express. Until now the output could not tell them apart, so a verified formula and an unverifiable one reached the expert as identical rows. That gap is not theoretical. MinerU's pipeline backend writes multiplication as a bare letter x, so every product formula from such an artifact is unverifiable -- including the Production formula this contract uses as its own worked example. Every artifact on Sofhia's machine is pipeline, because she has no GPU. FormulaEntry gains latex_verification: verified | unverified_no_markup | unverified_operator, null when there is no claim to characterise. Set by the span check, never by the model -- the model cannot know which guard ran, and a field it could write would be a claim rather than a record. _mark_latex is a never-throw seam like the rest of validation: an entry type without th…f81a227 -
[NOTICKET] fix(knowledge_extraction): compare formula_latex by notation, not identity
Rifqi Hafizuddin +1: Claude Opus 59c8a6e4 gave formula_latex the right haystack and the wrong test. Sofhia ran it against real artifacts and found `Production = MOHH \times Qty \times ...` rejected: the `pipeline` backend transcribes the multiplication sign as a bare letter x, so the claim had nothing to match. Probing further, the problem was wider than the case she hit. Against a source that DOES carry \times, the guard also rejected a model that dropped \mathrm{}, one that used \cdot, and one that wrote idiomatic LaTeX -- 4 of 5 plausible correct outputs. It passed only when the model mirrored MinerU's notation character for character. The test that shipped green passed only because I hand-wrote the fixture to mirror the source. That is the same category error as the original defect, one level up. First the haystack was the wrong FORMAT; then it was the wrong NOTION OF CORRECTNESS. The prompt asks the model to transcribe INTO LaTeX, which is a translation, and identity is not how you check a translation. normalis…a998ea4 -
[NOTICKET] fix(knowledge_extraction): formula_latex is checked against Chunk.latex
Rifqi Hafizuddin +1: Claude Opus 5The field was guarded in name only. It sat in GUARDED_FIELDS but not in _TRANSCRIBED, so its value was never searched in the source -- only its provenance.span was. A structurally wrong transcription with a valid prose span passed, which is exactly the shape X21 produced: a \frac that lost its division reads as a product and states something the document does not. Adding the field to _TRANSCRIBED was not the fix. `text` carries rendered prose and can never contain LaTeX, so that would have nulled every formula entry and presented as a model failure rather than a validation bug. The haystack was the problem, not the check. Chunk.latex is now the haystack. The parsing half has always emitted it; extraction's adapter dropped it on the floor, which is what left the claim unprovable. It now crosses the seam and is compared with whitespace removed ENTIRELY rather than collapsed. That is a different notion of equality, not a weaker one. In LaTeX, whitespace outside \text{} is not significa…9c8a6e4 -
[NOTICKET] docs(knowledge-parsing): settle determinism; finish the English pass
Determinism was the one thing left open on the backend decision: the VLM family is not deterministic in general, so a frozen baseline might drift and no eval number could be trusted. It does not drift. Two runs over the same documents, with the parse cache moved aside so both genuinely re-parsed (from_cache: false on all four records), produced 0 differences across 50 chunks. All four image descriptions were present in both runs and character-for-character identical (192, 256, 77, 146 chars) — those are the only model-written content in the artifact and were the likeliest to wobble. Existing baselines therefore stay valid and need no re-measurement, and hybrid/high is final rather than provisional. Marked settled in config.py, calibration §13, and REPO_STATUS §14. The parsing README was still Indonesian in the how-to-use sections written last week, while the .py files had already moved to English; it is now English throughout. Its console example was also stale — it showed the old I…d5ed50c -
[NOTICKET] feat(knowledge_extraction): output schema v2 — entity ids, links, 16 fields cut
Rifqi Hafizuddin +1: Claude Fable 5Harry's review of the output contract asked for four things: an id on every entity, a formula referenced rather than restated, rules related to formulas, and a smaller schema. All four are adopted. Two defects surfaced while adding the ids, and they matter more than the ids do. Neither existing identifier survived a re-run: `cluster_id` was a frequency-rank ordinal (`#c000` over a list sorted by mention count), so one changed count reshuffled every id after it, and `rule_id` was written by the model, so the same rule could return a different slug next run. The product is an expert working a review queue — if Mas Beta approves 40 entries and a prompt is retuned, unstable ids lose which 40. Approval continuity, not linking, is why these exist. New `ids.py` mints term/formula/rule/brief ids as sha256 over content: never positional, never model-supplied. New `link.py` resolves every reference after the branches run — deterministic, no LLM, no spend — and returns dangle counts rather than…49dfe4d -
[NOTICKET] fix(knowledge_extraction): rule.txt excludes bare equations (X20)
Rifqi Hafizuddin +1: Claude Opus 5The rule branch was spending calls re-extracting equations the formula branch already captures, and dropping the operational prose it exists for. 4 of 8 v2 rule statements were bare equations; in the prototype 0 of 7 were. The cause is not the model. Before rendering, formulas reached it as mangled character-spaced LaTeX and did not read as statements. Rendered to clean prose they read exactly like prescriptive rules, and rule.txt never excluded them -- it defined a rule as "a statement that constrains or directs behaviour", which a formula arguably is. Rendering gained +7.3 recall points on the term filter, so the fix belongs in the prompt, not in undoing the rendering. Three edits. A rule is now PROSE, with a paragraph saying an equation belongs to the formula branch and that "calculation policy" means prose stating how, when or on what basis a calculation applies. A new requirement 3 sets statement to null when the only prescriptive content is a formula. And -- the clause that mat…296cdd7
-
[NOTICKET] refactor(knowledge-parsing): English throughout the module
The parsing half was written in Indonesian — identifiers, docstrings, comments, manifest keys and console output. Everything else in the repo is English, and this module is the newest work, so it is the one to bring into line rather than the older code around it. Renamed across the module: parse_dokumen -> parse_document, normalisasi -> normalise, pulihkan_batas_kata -> restore_word_boundaries, kosakata_layak -> trustworthy_vocabulary, periksa_items -> check_items, CatatanDokumen -> DocumentRecord, and the locals with them. Docstrings and comments were REWRITTEN rather than translated word for word. Several carry the reasoning behind a decision — why `\frac` is unwrapped before the braced wrappers, why a heading never enters `text`, why the digit threshold is 3 — and a mechanical translation would have flattened exactly the part worth keeping. Manifest keys move too: detik_per_halaman -> seconds_per_page, peringatan -> warnings, jenis -> kind, galat -> error. That is a data change, …7f153d6
-
[NOTICKET] docs(knowledge-parsing): how to run it, and how to read what comes back
The README still said `python -m src.knowledge_parsing.run` with no environment set, and pointed at `uv sync --extra knowledge-parsing` — a command that fails. Four sections that answer questions actually asked this week. **How to run it**, and what a successful run looks like, with real console output. First-time readers cannot tell whether `⚠ 3 peringatan` is routine or alarming, and that is the one line worth reading: warnings are normal, `ok (cache)` means nothing was re-parsed, and only `ok / gagal` needs attention. **Backend**, with the measured table and why hybrid/high won — including that `vlm` is identical but slower and `pipeline` fuses the multiplication sign. Stability is marked open, not settled. **Running without a GPU**: the runner drops to `pipeline` on its own, and the result is valid but LOWER FIDELITY — no image descriptions, multiplication copied as a letter. Easy to forget, so it is stated where it will be seen. **Installing MinerU**, with an explicit do-not-…b3eae0f -
[NOTICKET] feat(knowledge-parsing): default to hybrid/high, the measured choice
All four backends run on the same GPU, same two documents, 29 pages: hybrid/high 4.85 s/page image descriptions YES hybrid/medium 1.88 s/page descriptions FORCE-DISABLED by MinerU pipeline 2.67 s/page never analyses images at all vlm 6.96 s/page output IDENTICAL to hybrid/high, 100% Two findings decided it. `vlm` and `hybrid/high` produce the same LaTeX and the same descriptions, so `vlm` only costs more time for nothing. And `pipeline` loses twice: slower than hybrid/medium, and the only backend that copies the multiplication sign as a literal `x`, which fuses `MOHH x Qty x PA` into one token and takes five terms with it. The `high` / `medium` gap is $6.89 against $2.67 across the whole 6,800-page BUMA corpus — $4, too small to trade image descriptions for. `effort` moves to "high" in the same commit, and that is not cosmetic: MinerU force-disables image analysis at medium, so flipping the backend alone would have produced hybrid WI…52e1bd4 -
[NOTICKET] fix(knowledge_extraction): refuse an artifact that will not name its backend
Rifqi Hafizuddin +1: Claude Opus 5MinerU's `pipeline` and `vlm` backends emit different text from the same PDF. The same equation arrives as a semantic `\times` from one and as a spaced literal `x` from the other, and word boundaries that survive in `\text{...}` under `vlm` are gone under `pipeline`. Two extraction results measured on different backends are therefore not comparable, and nothing downstream can tell them apart after the fact. This is not hypothetical: the two halves spent a week comparing numbers taken from two different parses and read the disagreement as a rendering bug. Each side was measuring correctly, on a different artifact. The provenance already existed and was walked around. `ParsedDocument` carries `parser_backend`, and the parsing half reads it from MinerU's own `_middle.json` rather than from config, precisely so the recorded backend is the one actually used. It never reached extraction because what crossed the seam was a bare `content_list.json`, stripped of its envelope. So the CLI now …4db5f14 -
[NOTICKET] fix(knowledge): settle the chunk_id separator on `::`
Rifqi Hafizuddin +1: Claude Opus 5The two halves minted provenance keys differently: the parsing half emits `<doc_id>::<seq>`, while the extraction adapter fell back to `<doc_id>#<seq>` for artifacts arriving without one. Nothing broke in practice — the adapter prefers the supplied id, so real runs always carried `::` — but the two shapes could diverge the moment a fixture or an older artifact took the fallback. `::` wins: it is what the producer actually emits, the calibration doc already uses it in prose, and `#` is URL-fragment syntax that would bite once these ids become MCP resource identifiers. The `KNOWLEDGE_OUTPUT_CONTRACT.md` cited as the source of `#` does not exist in the repo or its history. The separator is now stated in the contract next to the field, because this is the second time an undocumented detail of the seam's shape has cost both halves a round trip. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>e8cc26c -
[NOTICKET] feat(knowledge-parsing): carry image descriptions, apart from the text
MinerU describes images and charts, but only on some backends. Read from its own source (is_image_analysis_option_visible): never on `pipeline`, always on `vlm`, and on `hybrid` only at effort=high — medium disables it outright. Two faults kept that description from ever arriving. `normalize.py` had no branch for `image` items at all. Different backends label the same picture differently: what `pipeline` calls a chart, `vlm` calls an image. Image items fell through to the generic text path, which reads only item["text"] — a field pictures do not have — so caption AND description were dropped in silence. That is most of what looked like "vlm loses chart detection" in yesterday's comparison: vlm found them, we discarded them. The second is worse and was waiting to fire. `_teks_chart` appended `content` straight into Chunk.text. Harmless today because `pipeline` always leaves it empty — but the moment the backend flips to vlm, model-written prose starts flowing into the field the extra…c113bf8 -
Merge commit 'refs/pr/20' of https://huggingface.co/spaces/DataEyond/Agentic-Service-Data-Eyond-Catalog into pr/20
6e78f1e -
[NOTICKET] docs: reorganise into docs/ and give every doc a status banner
Rifqi Hafizuddin +1: Claude Opus 5The root carried 14 markdown files and no way to tell which were authoritative. That knowledge lived in CLAUDE.md §1 and in Rifqi's head, which works for us and fails for anyone else — and one of the stale ones already has a named failure mode ("The Stale-Doc Believer") for being implemented from. Root now holds four: REPO_STATUS, DEV_PLAN, API_CONTRACT_BE_PYTHON, README. Everything else moves under docs/, by git mv so history follows: docs/knowledge/ context, TODO, calibration, parsing output contract docs/plans/ SPINE_V2_PLAN (approved, W3 deferred), DS_TOOLS_PLAN (proposal) docs/archive/ API_ENDPOINTS_RESTRUCTURE, CODE_REVIEW_2026-07-23, PROJECT_BRD docs/reference/ API_CONTRACT_BE_GOLANG (the Go repo's, a copy that can drift) Every doc gains a one-line status banner under its H1, marked <!-- doc-status -->: LIVE / LIVE PLAN / PROPOSAL / ARCHIVED / REFERENCE. Archived ones name what superseded them — API_ENDPOINTS_RESTRUCTURE now opens with "DO NOT IMPLEMENT FROM…f1342ce -
[NOTICKET] fix(knowledge-parsing): let the source prove itself before trusting it
Word-boundary recovery treats the PDF text layer as the arbiter. That is safe when the layer is intact and dangerous when it is HALF there: a legitimate word looks unknown, and gets split. An empty text layer was already safe — no vocabulary, no recovery. The gap was the middle ground, and nothing measured it. So the page now has to prove itself: how many of MinerU's own prose words on that page are actually found in its text layer. A low score means the two read the same page differently, and a source that disagrees with the parser is not fit to arbitrate it. Pages scoring below 90%, or carrying too little prose to judge at all, lose their vocabulary and skip recovery — left damaged but honest, rather than repaired into something the document never said. This replaces a constant tuned against one document with a measure that assesses each page on its own evidence. Measured across three documents, two of them a McGraw-Hill textbook in English with nothing in common with the BUMA st…22c6ad1 -
[NOTICKET] fix(knowledge-parsing): read word boundaries back from the source PDF
MinerU writes formulas one character at a time ("T o t a l H o u r s"), and once those are rejoined the word boundaries are gone: "Total Hours" and "TotalHours" are no longer distinguishable from the LaTeX alone. Everything fuses into a single token. The multiplication written as a letter "x" is not a separate bug. It is the same one — an "x" between terms is just another spaced character, so "MOHH x Qty x PA x UA x Pty" collapses to "MOHHxQtyxPAxUAxPty". Two obvious fixes were tried and rejected: - Rewriting a lone "x" to a multiplication sign guesses, and guesses wrong on any document with a variable named x, silently. It also leaves "TotalHours" untouched, which is the larger half of the damage. - Emitting a warning fires on 7 of 11 equations. A warning that lights up on 64% of items is not actionable, and recovers nothing. The source PDF still holds the boundaries: its text layer reads "MOHH x Qty x PA x UA x Pty", properly spaced. So the answer is read from t…e047f7a -
[NOTICKET] fix(knowledge-parsing): render_latex was deleting division (X21)
Rifqi's three faults, all reproduced against the document's own 11 equations and all fixed. Two of his numbers were off, and the corrections matter: - It hit 4 of 11 equations, not 2 — PA, PA_ij, UA_ij and PTY_ij. Not just PA and UA: every composite form was affected too. - The nested braces come from mathrm{} and sum _{ij}, NOT from text{}, which this standard never uses. A fix written against text{} would have repaired the synthetic example and no real equation. (C), the serious one: the frac rewrite needed a brace-free argument, could not match a nested numerator, and the unmatched frac then fell through to the generic "drop leftover commands" pass. The division was deleted, leaving a formula that reads as a product and states something the document does not. The ordering decision this needed: braced wrappers are unwrapped BEFORE frac, and the unwrap consumes its braces. The old pass removed only the command name and left { } behind — those leftover braces were wh…a356e3c -
[NOTICKET] fix(knowledge-parsing): stop the warning marker from failing documents
A Windows console defaults to cp1252, which has no "warning" glyph. Printing the quality-warning line raised UnicodeEncodeError INSIDE the per-document try block, so a document that carried warnings was recorded as FAILED — even though its parse had already succeeded and been written to the cache. The effect was the exact inverse of the intent: clean documents passed, and the ones whose quality needed attention were the ones thrown away. The same throw in the run summary then took down the whole run with exit code 1. Reproduced on the 9-page BUMA standard, which carries three warnings: it was reported as "gagal" while its 31 chunks sat complete on disk. stdout and stderr are now reconfigured to UTF-8 at CLI entry, so the run no longer depends on the host codepage. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>df84654 -
[NOTICKET] fix(knowledge-parsing): send MinerU a backend name it accepts
The GPU flip config.py advertises did not work. MinerU dispatches on `backend == "pipeline"`, then `startswith("vlm-")` and `startswith("hybrid-")`. A bare "vlm" matches none of them, so MinerU did nothing at all, silently — and the failure surfaced here as "content_list not found", which points at the wrong place entirely. Valid names are pipeline / vlm-engine / hybrid-engine. The short names stay in the config because they are part of the cache key and read better; the translation happens at the boundary, just before MinerU is touched. Nobody would have caught this until the day a GPU arrived and everyone was waiting on the result. Also: the temp directory used with_suffix(), which REPLACES the last suffix rather than appending. Cache folders end in the MinerU version ("...-3.4.4"), so ".4" was thrown away and two versions collided on one temp directory that the next line deletes without asking. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>8e93c60 -
[NOTICKET] eval(knowledge): full document through both halves — E3 0.90 PASS; fix legend blank-line bug
Rifqi Hafizuddin +1: Claude Fable 5First full-document run with the parsing module building the artifact and the extraction module running every branch. Result committed beside the frozen baseline: eval/knowledge/results/v2_full_document_2026-08-24_093051.json E1 term-filter recall 0.8049 (prototype 0.8537) PASS, kill 0.70 E3 schema fill 0.90 (prototype 0.75) PASS, kill 0.80 rule-branch recall 0.0667 (prototype 0.2667) REGRESSION E3 is the headline: the one experiment that FAILED in the prototype now clears its line. It needs the same care the original number did — the scoreable base is 10 entries, so one adjudication moves it ~10 points, and nano is not deterministic at temperature 0. The single failure (produksi) is the same one the prototype recorded and reads as a gold-adjudication case, not a model error. E1 sits below the baseline because the CHUNKING differs, not the filter: the parsing module splits to a token budget (31 chunks) where the earlier comparison used 14, so th…ec7d3d4
-
[NOTICKET] fix(knowledge_extraction): show the model the section heading
Rifqi Hafizuddin +1: Claude Fable 5The evidence block sent to the model carried only `chunk.text`, so the section heading never reached it. Two consequences, both silent: - Indonesian standards name the term in the heading and open the body with the definition — "2.1.3 Physical of Availability (PA)" / "Adalah ketersediaan fisik…" — and the body often never repeats the term. The model was being asked to define a term its evidence never named. - It broke the invariant that what the model reads is exactly what the span check searches. `validate.evidence_text` already composed heading + text, so a field quoting a section title would have been rejected as unlocatable — and the model could not have quoted one anyway. Found while verifying the claim in the flow illustration: "Physical of Availability" appeared in the prompt only because the prompt file's own worked example contains that string, never from the document being processed. The deterministic `source_wording` fallback masked the effect, which is why th…a7bdeae -
[NOTICKET] docs(knowledge-parsing): module README + refresh the parsing/seam statuses
README follows the per-module convention already used by src/catalog, src/query, src/retrieval and src/security, and records the three things that are easy to get wrong and expensive to discover: text stays verbatim, page_idx is 0-based, and markup never goes in text (with the recall numbers behind that last one). KNOWLEDGE_PIPELINE_TODO.md — parsing (B) and seam (S) rows only; P/X/D/E/F and S7/S8 are untouched: - B6, B8, S2, S3, S4 -> done. Contract landed, envelope carries parser identity read back from MinerU's _middle.json, section-structure claim corrected against MinerU's source, source_wording added. - B4, B5, S1, S5, S6 -> in progress, deliberately NOT marked done: B5's acceptance test needs the Open Pit textbook (S7) before the NSR cross-page case can be claimed; S5 is satisfied on the parsing side but is Rifqi's to close; B4's language question is answered but text quality is not yet judged. Marking unproven work done is what makes a checklist useless. - S6: (c)…f0cf168 -
[NOTICKET] feat(knowledge_extraction): consume the parsing half's ParsedDocument
Rifqi Hafizuddin +1: Claude Fable 5Wires the adapter to the contract Sofhia landed in src/knowledge_parsing, so the seam is exercised with the real types on both sides rather than a hand-written dict. - accepts a ParsedDocument instance directly, not just a dict - page_idx / page_idxs / page_idx_end are now the primary field names; the older draft and prototype names still load so existing fixtures keep working - page numbers pass through 0-BASED with no conversion anywhere. Converting is the review UI's job, done once at display time — an off-by-one here would be invisible until an expert opened the wrong page - parser_backend is folded into parser_version, because the same MinerU build emits different text from `pipeline` and `vlm`; a shift in extraction output has to stay attributable to one or the other New local seam tests import src.knowledge_parsing.contracts directly, so a field rename on either side fails there instead of silently degrading a corpus run. They pin the three properties the seam exists…f132cc1
-
[NOTICKET] fix(knowledge-parsing): render markup to prose in text; only numbered headings split sections
Both defects were measured by the extraction side on the same document and gold set, changing only the parse: PyMuPDF (plain text) recall 0.8537 35/41 MinerU, raw markup in text recall 0.7561 31/41 MinerU, formula + table rendered recall 0.8293 34/41 1. Chunk.text now carries readable prose, not markup. The term filter is an NER model reading prose. MinerU writes formulas with every character spaced out — "{ \mathrm { P u r c h a s i n g ~ c o s t s } }" — and tables as HTML; neither yields a single mention for terms the same model finds easily in plain text. render.py rejoins spelled-out words and numbers, folds subscripts, maps operators to non-letter glyphs (a literal "x" for \times would merge into the token beside it), and flattens tables to " | "-separated rows. The markup is kept, not discarded: Chunk.latex and Chunk.table_html hold it verbatim, because the formula branch needs exactly that form. Found while verifying: table cells carr…2cb1336 -
[NOTICKET] fix(knowledge-parsing): keep the heading line out of Chunk.text
The heading was briefly copied into `text` as its first segment, to stop the chunk that defines a term from containing no mention of that term — the BUMA standard names the term in the heading and opens the body with "Adalah ..." without repeating it. The extraction half already solves this, and better, because it measured it on the document: `validate/span_check.py` builds its haystack as heading + text, and `rank/evidence.py` treats a heading that names the term as a mention at offset 0. So copying it into `text` only causes harm: the heading appears twice in that haystack, and its occurrence becomes a real mention, inflating mention_count. That count is what the frozen prototype baseline is compared against (169 mentions -> 66 clusters), so v2 numbers would drift for a reason that has nothing to do with extraction quality. `heading` stays verbatim, which is what both halves actually needed. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>15028da -
[NOTICKET] feat(knowledge-parsing): document parsing half of the knowledge pipeline
Implements §4 B2/B5/B6 and closes the four open seam items (S2, S3, S4, S6b/c/d) against the contracts.py draft circulated 2026-08-19. Additive and flag-gated: src/knowledge/ (Tesseract -> pgvector) is untouched. Seam — src/knowledge_parsing/contracts.py, pydantic per S6e: - ParsedDocument envelope: content_hash, n_pages, version, parser_name/version/ backend/config, schema_version, raw_output_dir. parser_backend is read from MinerU's _middle.json, so it records what actually ran, not what was configured — a MinerU change and a prompt change stay distinguishable (S2). - Heading structure is READ, not derived. Verified in MinerU's source: pipeline and vlm run identical text_level logic, and bbox is emitted by both. Heading availability is document-dependent, not backend-dependent — the earlier claim came from sampling the McGraw-Hill handbook, which has no numbered headings (S3). - source_wording on TermRecord; heading carried verbatim (S4). - page_idx/page_idxs, 0-based, n…5840d20 -
[NOTICKET] feat(knowledge_extraction): paid extraction stage + validate, diff, queue
Rifqi Hafizuddin +1: Claude Fable 5Completes every stage that does not depend on the seam. The pipeline now runs end to end: artifact -> filters -> cluster -> rank -> LLM -> validate -> diff -> frequency-sorted review queue. extract/ client on the nano __54n quad; structured output is PROBED not assumed (json_schema, falling back to json_object + validate-retry, recording which applied); cached tokens read from the API, never modelled. Four branches, one call per TERM CLUSTER. Prompts stay as files so the fixed prefix is byte-identical and caching keeps engaging. validate/ span check (whitespace-only normalisation, never repairs, unlocatable provenance rejects every guarded field), escalation, conflict detection by token overlap — explainable to the reviewer, and it never picks a winner. diff/ new / duplicate / conflicting against an explicitly supplied active glossary, so the duplicate and conflicting paths are reachable. The …ab5ea78 -
[NOTICKET] feat(knowledge_extraction): span filter — E1 recall 0.8537, matches the frozen baseline
Rifqi Hafizuddin +1: Claude Fable 5Adds the term-filter stage, the last free stage before the paid extraction call. Dependencies (approved): gliner>=0.2.13 (resolved 0.2.28), torch>=2.6 (resolved 2.11.0+cpu). Install verified clean — `import torchgen` succeeds, so not the corrupt-wheel failure recorded in the calibration doc. filters/span_filter.py runs the model on EVERY chunk, unconditionally. There is deliberately no relevance gate in front of it: a pre-filter's false negatives drop content before extraction sees it, and dropped content never reaches expert review to be caught. "Zero candidate spans" is the relevance signal instead. Two mechanical traps handled, both of which silently cost recall: - the model truncates long inputs and WARNS rather than failing, so chunks are fed as overlapping 130-word windows with character offsets remapped. Offsets are walked over the original spacing, never re-joined — re-joining would shift every offset in a chunk containing newlines and break every span check. - the ~12…3a01634 -
[NOTICKET] fix(knowledge_extraction): move runner into the package; fix two ranking defects
Rifqi Hafizuddin +1: Claude Fable 5Runner relocated scripts/run_knowledge_extraction.py -> the package as src/knowledge_extraction/cli.py. scripts/ is gitignored, and this runner is the pipeline's operator entry point, so it has to ship with the module: uv run --no-sync python -m src.knowledge_extraction.cli <artifact.json> pyproject: narrow the T201 lint exemption to that one CLI file rather than all of scripts/, so the ignore traces to this change and nothing else. Two ranking defects found while verifying against the real 9-page BUMA standard (both recorded as deviations V1/V2 in KNOWLEDGE_PIPELINE_CALIBRATION.md §9): V1 — a heading that names the term now counts as a mention at offset 0 for cue-proximity scoring. Indonesian standards are written "2.1.3 Physical of Availability (PA)" / "Adalah ketersediaan fisik...": the heading names the term and the body opens with the definitional cue WITHOUT repeating it, so the definition section earned no cue bonus at all. Measured before the fix, PA's own definition se…72605ff -
[NOTICKET] feat(knowledge_extraction): free stages — models, seam adapter, filters, cluster, ranking
Rifqi Hafizuddin +1: Claude Fable 5First slice of extraction v2, rebuilt in this repo (the kex prototype is not being ported). All stages here are free: zero API calls, CPU only. New module src/knowledge_extraction/ (named to avoid colliding with src/knowledge/, the existing OCR->pgvector service, which is untouched): - models.py pydantic v2 contracts. Encodes the three invariants: content fields Optional, subdomain_tags as an enum, provenance.span mandatory. Adds source_wording so the literal source wording survives instead of being normalised away. - adapter.py THE ONLY file that knows the seam's shape. Stages depend on an internal Chunk subset, so the artifact contract can settle either way without touching a stage. Verified against both live shapes: Sofhia's bare-list draft with its `kind` discriminator, and the document-level envelope proposed in the checklist.…024c30a -
/fix planner count and report (#19)
- /fix planner count question (11de970ff65be9b4f81f773cfb355d3dc57358d7) - [noticket] fix report bug where some answered analysis isnt included in report summary answers (8dec1825cf803ad28f4ff0400b9c373250a7d90a) - [NOTICKET] fix: split report floor from report body; scope analysis reads by user_id (fd4865b20dfc46782706be9aab318f909b9903c3) - [NOTICKET] docs: sync report floor/body + security posture; add review remediation tracker (fc5200c994441ec505ea7a82469da577afe170a5) - [NOTICKET] test: planner eval harness (rule-compliance, --selfcheck/--rescore) (4292522aa3fa3a7af8f933307334cb7e5eb0b377) - [NOTICKET] eval: repair readiness harness; commit intent/help/readiness baselines (08a0a4ad5eacc6c90df22fab5be69f2d33579f29) - [NOTICKET] fix: compiler parity — F-17 fabricated column, F-18 LIKE nulls, F-25 empty in/not_in (8f736f90f1958960a83847d332b45b13fa346168) - [NOTICKET] feat: bound catalog render + blob read; scope charts/traceability; mark degraded seams (0cb7d53cb6f7f7e7b687dd6d9c27…f282b15
-
/fix report not working (#18)
- [NOTICKET] fix: write reports.user_id (Neon NOT NULL column broke report persistence) (13f74e11df6058800f71e539fe135eeba7576f24) - [NOTICKET] docs: correct 13f74e1 root cause — Go migrations are non-convergent (86d0805130637d66082c5a75b8da71fae36002ad)9070d67
-
/fix change model to gpt 5.4 mini (#17)
- [NOTICKET] chore: rename LLM env quad __4o -> __54m (GPT-5.4-mini) (790ef0c7aaac6e76eb01b0ef9a3398ff4249416d)d65c41d -
fix/ check and new chart tool (#16)
- check: table-level column drill + drop misleading "name a table" hint (ef0c7b1c5aa49a244b02f1b5eea15cc77dcf2bcb) - [KM-715] charts: render_chart tool + message_charts store + GET /api/v1/charts + planner viz slice (87cfcf884d923ee4bcd722e0e4ef4d90d7eaedca) - [KM-715] checkpoint: S1a deterministic quality checkpoint between runner and assembler (f3c8de34dc19a573031b67fd4f48dcfbadcdfed8) - [KM-715] docs: sync REPO_STATUS/DEV_PLAN + land SPINE_V2_PLAN (W1/W2 done) (434eacbf2ac8324188ebe5e32775dbc9d2a3f49a) - planner: value-handoff (${tN.col} → list) for tabular lookup/anti-join (54070ac1407b751000e49dfe7f102a2be5ab1ff8) - [KM-715] charts: GET /charts by message_id only + tri-state status marker (b8d15761addf4d499c9631df64e7693aa1d61fa2) - [KM-715] report: embed charts as plotly EDA fences; render_chart counts toward the floor (89db8be6b6f11e86ad0776d2ee07f2306e3b6079) - [KM-715] docs: contract + status deltas for the charts lead-review round (96347ee7c55a4eb39a906409ec214463e25c5e35) - …5a60e93
-
/fix traceability (#15)
- [KM-691] Traceability: add resolved data_used layer for the FE (a646735e33f6ac725d0756a88956b0aec15fe495) - check: don't leak user-scope catalog when analysis catalog missing (c0ba82938383169b41059c4125765db5275885e7) Co-authored-by: Sofhia Az-Zahra <sofhiaazzhr@users.noreply.huggingface.co>32abc41 -
/fix help and tools analyze (#14)
- [KM-711] Retire /report slash command; report is Generate-button only (f7f7b172da0dc16adac0a7555f6d34a704cc2050) - fix(analyze_trend): handle value_column == date_column (count events per period) (55e3327f7b2e21a954b317e143f46bcea8b5ada7) - [KM-711] Docs: drop /report from list_skills catalog examples (0772863b296e2ed3e2d559df05e98540764e58b0) - fix(planner): reject inlined table-ref as analyze_* data (Pattern A) (776fae3c7c6e3d3a07c22bef10898f6b1c33c701) - fix(analyze_aggregate): reject non-string group_by (CASE/binning) keys (b40fa76e2fa68572dad0d8532e07076207095d6b) - fix(planner): reply-language for infeasible_reason + answer-only-asked rule (4d89d7d93938e41892979cd20e5ccf0c97441dac) Co-authored-by: Sofhia Az-Zahra <sofhiaazzhr@users.noreply.huggingface.co>17e31c6
-
/fix planner and report (#13)
- [KM-644] planner: infeasible plan outcome + entity-ranking few-shots (089b69d9b6111e813307c6afbd3e7613bb823f9d) - [KM-644] harden query path: grouped-select rule, int-date trends, sample decode, trace guard (74ad5261736bda0bd72bccdb41565ea67313993a) - [KM-703] add analyze_merge tool: two-table combine (analytics.combine) (8abf635532366d670367557a3ffa03d42371d831) - [KM-567] planner support for analyze_merge: data_right guard + two-retrieve merge few-shot (a029a843247e6ea4d3c79a4e2c1b7571ba4cda69) - [KM-644] report v2: BQ answers, unresolved/excluded sections, evidence tables + curation/readiness endpoints (4ceb058d85c5a2a289443b9411d2b345027fa226) - [NOTICKET] refactor: mint message_id as UUID to mirror analyses_messages id shape (4253733b6b8db0449a82442a915c732ddd55f528) - [KM-644] report: drop Notes/Unresolved/Method sections from rendered markdown (compact report) (eef6de811a139402c79fd74ad4adfc00cf7aff01) - /edit md file (8198cc188ff05a2798ae2056b90b6e56e1bea87b) Co-authored-by…49b0848
-
fix/ check (#12)
- [NOTICKET] check: scope to analysis catalog + list database tables (db2a10f61e1f6ff9579deaaeafed8fdf0cdf5bb7) - feat(planner/help): graceful degradation for gated analytics capabilities (efc0af7c05308acb564bffdd7d8ab8abd2dd2e81) - Scope structured_flow + report to analysis catalog; drop dead data_sources binding (c333ec84a96b171484af7a87bdf07fd1fb0e9c69) - update md file (c5e2e1b1adcba46b56eaf6bbb85d9b8bb0a0d277) - [NOTICKET] check: read documents from analysis-scope catalog (6977394de88d7944dd2504beeed3e00eafe7857a) Co-authored-by: Sofhia Az-Zahra <sofhiaazzhr@users.noreply.huggingface.co>b9dfc76
-
/feat traceability (#11)
- [NOTICKET] Fix check skill: reply language + tabular inventory routing (1367f41269cf9ef42fd91f311b24d4edf5885c7b) - [KM-691] Develop traceability (23cc2079989fef931a6e146980fe6bf528ee6e92) - [NOTICKET] Humanize check output: hide UUIDs, add lead-in, list format (2bdda8fc3fc4b4e17b4df7fb53aee5abede45add) - [NOTICKET] check: schema-detail view for column/type questions (9e248d9cc5688d6ee622be523a6bedddf5010995) - [KM-691] Make sources trace-authoritative; drop stream sources (b43ecc77f26c1537cbbfddb6800ec4ff62d16441) - [KM-691] Bound traceability payload + reject contradictory IR filter ranges (db6db73d6ab0c2175447946742038443d67b2f78) - /fix planner can now plan quarter comparison (839e32c8d6d5895029072074ab7aba8bb4fcdb6e) Co-authored-by: Sofhia Az-Zahra <sofhiaazzhr@users.noreply.huggingface.co>f873f92
-
/fix validator and report (#10)
- /fix report: remove record_id hash citations, EDA (for future tool) change to data source, transform to mdx-safe (b1b101730ccf5628e43e80f2ad654f78fbed5207) - [NOTICKET] fix(planner): repair near-miss catalog ids in QueryIR before validation (aacd4d13def258a9815761aecbd7b246a8dae9fc) - /fix report: group findings by analysis, dedup, code-span-safe MDX escaping (029b48894e9f79af6c0d81c00a2d811a982f87ae) - [NOTICKET] fix(tools): log swallowed analytics tool errors in AnalyticsToolInvoker (93d9381b2a5f5efe058c913fe0b793ce994a9097) - fix(planner): harden plan validation to reduce retry thrash on structured questions (1619dd3c0fa74825010c7af6d4b375e3b9404847) - [NOTICKET] fix(tools): coerce numeric-string columns before analyze_* compute (6bda333866a1a8ccf1b3fe6fcee7a8e1832f28d7) - fix(planner): correct top-N guidance — in-query ranking IS supported (99aa4141aed8241a625d290fb170f74192a6d831) - feat(guardrails): add input guard, out_of_scope intent, and refusal handling (0bbc27bc40ba20e7a8e…0e5fdb5
-
/fix analysis + report_inputs and storage update (#9)
- fix(chat): drop ENABLE_SLOW_PATH flag + make chat persistence read-only (b8b76c4fb8db21de183cc6f80c0acba0d35bdad1) - [NOTICKET] feat(storage): read tabular Parquet from Supabase S3 via provider toggle (3f48226adaf2ed107a4eb2c0373cbb73fc62f552) Co-authored-by: Sofhia Az-Zahra <sofhiaazzhr@users.noreply.huggingface.co>079782d -
/fix check and help tool (#8)
- /fix data_catalog model + CatalogStore to dedorch schema (data → catalog_payload) (09857fc52486354f53e752e3ea329b064a9e4c4b) - [NOTICKET] fix(help): lock reply language to user's language on button path (cb7f5ac484ad4a510e68faf5291091958b6d2f42) - /fix default SKIP_INIT_DB=true for dedorch cutover (skip init_db on Go-owned schema) (781b93c6b8744209a1101629b5170175e9212592) - /fix surface catalog foreign keys to the planner (FK-backed joins) (446c57524e932fcdbb0ddecc049189c443b6ae97) - [NOTICKET] feat(eval): add help-skill eval harness + goal-based language fallback (cbdbefb7ab0ffa36949713f18422213db480a959) - [NOTICKET] fix(eval): update readiness eval to post-KM-652 floor (drop problem_validated) (6f46e81bdeb1eb987f9dac8af0e22a975474cbb1) - /fix infer FKs for FK-less dedorch catalog + log repr(e) in db executor (3669290474583ae011f5ed5e50b94dad146fbc77) - Merge eval/help + readiness eval fix into PR #8 (5f4818a8e38f93988ab5756f3a06c94645826b15) - /docs REPO_STATUS: FK inference (FK-…3743cfe
-
/fix v2 chat stream api update to analyses_messages (#7)
- [NOTICKET] fix(db): reconcile analyses schema + migrate chat to analyses_messages (283eb0e4653acfe9311f42e3bb12d0490b8c9751)0066161 -
/fix message_id for chat/stream Endpoint (#6)
- [NOTICKET] refactor(api): server-mint message_id only; unwire v1 chat (729086150b982cbd747829074658bc798d8e18f4)d8e7745 -
feat/ Endpoint Restructure (#5)
- [KM-686] Unwire Non-AI Routers from Main + Swagger (0b2d6785801ecc4d58a32580b1a9ef6238059de2) - [NOTICKET] docs: pr/5 sprint tracker + endpoint-restructure contract + REPO_STATUS banner (ce7be178428cadf1884a77d3d1f1398dccf32f6e) - [KM-687] Add Dedicated /tools/help Endpoint + Regroup Tools List (a881d38851470bf685990fdb554417da4fc1eccd) - [KM-688] feat(api): v2 chat pilot + regroup report under /tools; adopt lead API contracts (ad2ce619b51c234a8cd1c3772417a9a542a9a40d) Co-authored-by: Sofhia Az-Zahra <sofhiaazzhr@users.noreply.huggingface.co>3bacc1d
-
feat/Analysis State & Report Rework (#4)
- [NOTICKET] Add project repo documentation (ade9c0c635b1bb6f171a3f4e710e6d0f9bae28f3) - [KM-674] feat(report): adapt report generator to objective + business_questions (9bd8ec51fdcd5057b5e8ba6b219a7d9dce945144) - [NOTICKET] Add revision plan md file (e0384e1dc600b2496f96cbba152e616d65650f07) - chore(docs): tidy root markdown — archive stale docs, delete superseded ones (4835fb85915bd69079fb43d7c602774d514e02ae) - [KM-652] refactor: remove problem_statement skill + problem_validated gate (057e0b24b56dd697603d183f508cb1e8a87268fd) - [KM-678] feat(tools): trim FE slash menu to /help + /report (fed17ccc98ec523088c482673132763826f86701) - [NOTICKET] docs: sync DEV_PLAN status (#8/#9/#20) (e936cfafdfde1385fe16bd7e3796e7d46c193197) - [KM-679] fix(help): drop problem_statement refs after KM-652 (0bec4725db572d492634ced6a80ef6e65d075edc) - [KM-680] feat(report): show author display name (users.fullname) (9bccc122878fbdbae0ac1bd0b4316f5ea8af658c) - [NOTICKET] docs: add API_ENDPOINTS.md (FE surf…0e02a0f
-
feat/Knowledge & Data Tools (#3)
- [KM-641] refactor(tools): trim analytics registry to 4 active tools (66e2e4df66f1abf7eaa7e442d9900d428696ece4) - [NOTICKET] docs: add checkpoint plan for 2026-06-17 sync (8c1bb63ebdee02f9db0da9fc2b2269823b0a7688) - [KM-642] refactor(tools): rename data-access tools to data/knowledge naming (c38c0c29d3e6b245f67d03f2c35e451900e1abdb) - [KM-643] feat(tools): merge data sources into data_check + add knowledge_check (4bd5f1efdc560a64092cc8df672a7990fa949b59) - [NOTICKET] docs: record KM-642/643 in checkpoint plan + PROGRESS (4c57588b90fe0cac8d0d4d22cfe5701fc9eb52dd) - [NOTICKET] fix(chat): scope response cache by user_id + drop TTL to 1h (R5) (b701e95d300c7d2480a3608b2d4da6dfe102b19a) - [NOTICKET] feat(slow-path): persist AnalysisRecord via PostgresAnalysisStore (09f845188226dd01633cf74525caa569e4a7d37a) - [NOTICKET] Adopt verb-first skill naming (2d6406def7ed45edd1226e96645ba2a5cdbfa3d0) - [NOTICKET] docs: sync checkpoint plan — verb-first tool names + report-as-API (3b399f84d73b0d98bd99…0721bb4
-
/fix redis connection code for general redis connection
Rifqi Hafizuddin0810be5