Commit History
-
[NOTICKET] fix(knowledge_extraction): close the notation rules against real backend markup
Rifqi Hafizuddin +1: Claude Opus 5Sofhia supplied the same 11 equations as MinerU transcribed them under four backends -- 44 real strings, committed as a fixture. Running the shipped normaliser over them reproduced her figures exactly: 11/11 for vlm/medium, 10/11 for hybrid/medium, 8/11 for pipeline, against hybrid/high as reference. Her independent check and this code agree on both the counts and which equations differ, which is the point of her having run it separately. The remaining gaps were in the command inventory, and the inventory is now evidence rather than guesswork. \mathsf and \displaystyle occur in this corpus and were missing -- \mathsf sets the whole of equation 7 under pipeline, including its multiplication sign, and \displaystyle appears in equation 4. Both are handled now, and the wrapped-x rule generalises across every presentational wrapper rather than being \mathrm specific. Equation 4 needed two more, and one of them was a decision Sofhia deliberately left open. The array ALIGNMENT SPEC arrives…01c76dd -
[NOTICKET] eval(knowledge): the runner the harness has been missing
Rifqi Hafizuddin +1: Claude Opus 5The 2026-08-24 figures were assembled by hand. That is why the rule-branch number in the committed result file is 1/15 measured with a scorer that has since been fixed, and why nobody could re-derive it. This is the reproducible version, following the house convention the other four harnesses already use: module mode, timestamped results, never overwritten. It takes a ParsedDocument artifact and never a PDF. MinerU is not invoked and is not required -- that is the seam doing its job, and it is what makes the pending X20 and X23 measurements possible on a machine with no GPU and no parser installed. Free by default: E1 term-filter recall and E2 compression, plus the funnel, with zero API calls. --extract adds the paid branches, E3 schema fill and rule recall. --dry-run prints the token estimate and the per-branch cache-floor status. --mock exercises every branch with no spend, and stamps the result NOT SCOREABLE so a fabricated number can never be quoted as a measurement. Two refusal…1edf61c
-
[NOTICKET] fix(eval/knowledge): stop the scorer treating `_` as a word character (X22)
Rifqi Hafizuddin +1: Claude Opus 5`norm()` flattened every punctuation mark to a space except the underscore, because `_` is inside `\w`. The gold writes notation bare (`QtyA`) while the renderer folds the LaTeX subscript and emits `Qty_A`. Those normalised to `qtya` and `qty_a`, the substring test failed, and a correct entry was scored as a miss — the instrument was wrong, not the extraction. Underscores are deleted rather than spaced. Spacing only turns `qty_a` into `qty a`, which still does not contain `qtya`; deletion is what makes the two notations meet. The gold is NOT edited. Verified: none of the 15 rules contains an underscore, so no gold fragment changes under the new normaliser and matches can only move from miss to hit. The alternative — rewriting the gold to `Qty_A` — was rejected because it binds the frozen instrument to one renderer's current output, and rendering has already changed once this week and will change again with the backend decision. Two rules use the affected notation, GAINLOSS_QTY_ZERO_…abae344 -
[NOTICKET] docs: reorganise into docs/ and give every doc a status banner
Rifqi Hafizuddin +1: Claude Opus 5The root carried 14 markdown files and no way to tell which were authoritative. That knowledge lived in CLAUDE.md §1 and in Rifqi's head, which works for us and fails for anyone else — and one of the stale ones already has a named failure mode ("The Stale-Doc Believer") for being implemented from. Root now holds four: REPO_STATUS, DEV_PLAN, API_CONTRACT_BE_PYTHON, README. Everything else moves under docs/, by git mv so history follows: docs/knowledge/ context, TODO, calibration, parsing output contract docs/plans/ SPINE_V2_PLAN (approved, W3 deferred), DS_TOOLS_PLAN (proposal) docs/archive/ API_ENDPOINTS_RESTRUCTURE, CODE_REVIEW_2026-07-23, PROJECT_BRD docs/reference/ API_CONTRACT_BE_GOLANG (the Go repo's, a copy that can drift) Every doc gains a one-line status banner under its H1, marked <!-- doc-status -->: LIVE / LIVE PLAN / PROPOSAL / ARCHIVED / REFERENCE. Archived ones name what superseded them — API_ENDPOINTS_RESTRUCTURE now opens with "DO NOT IMPLEMENT FROM…f1342ce -
[NOTICKET] eval(knowledge): full document through both halves — E3 0.90 PASS; fix legend blank-line bug
Rifqi Hafizuddin +1: Claude Fable 5First full-document run with the parsing module building the artifact and the extraction module running every branch. Result committed beside the frozen baseline: eval/knowledge/results/v2_full_document_2026-08-24_093051.json E1 term-filter recall 0.8049 (prototype 0.8537) PASS, kill 0.70 E3 schema fill 0.90 (prototype 0.75) PASS, kill 0.80 rule-branch recall 0.0667 (prototype 0.2667) REGRESSION E3 is the headline: the one experiment that FAILED in the prototype now clears its line. It needs the same care the original number did — the scoreable base is 10 entries, so one adjudication moves it ~10 points, and nano is not deterministic at temperature 0. The single failure (produksi) is the same one the prototype recorded and reads as a gold-adjudication case, not a model error. E1 sits below the baseline because the CHUNKING differs, not the filter: the parsing module splits to a token budget (31 chunks) where the earlier comparison used 14, so th…ec7d3d4
-
[NOTICKET] eval(knowledge): run the pipeline on MinerU-parsed input — parser comparison
Rifqi Hafizuddin +1: Claude Fable 5First run of the extraction half on Rifqi's MinerU output. Same document, same gold set, same pipeline code, so the only variable is the parse. PyMuPDF (what the frozen baseline used) E1 0.8537 35/41 MinerU, raw markup E1 0.7561 31/41 MinerU, formulas + tables rendered to prose E1 0.8293 34/41 The span model reads prose, not markup. MinerU's structure helps the formula branch and hurts the filter unless rendered first: - Terms wrapped in LaTeX are invisible. "\text {Total Hours} - \text {Breakdown}" produces no mentions for either, though both are gold terms found in the plain-text parse. Unwrapping \text/\mathrm/\frac and collapsing MinerU's character-spaced output recovers them (+7.3 points). - Raw <table> HTML sat in 8 of 14 chunks before rendering. The residual ~2.4 points are CHUNKING, not content: MinerU marks non-section labels (Keterangan:, Keterangan grafik:) as text_level titles, so opening a section at each splits a legend…6038b2b -
[NOTICKET] feat(knowledge_extraction): span filter — E1 recall 0.8537, matches the frozen baseline
Rifqi Hafizuddin +1: Claude Fable 5Adds the term-filter stage, the last free stage before the paid extraction call. Dependencies (approved): gliner>=0.2.13 (resolved 0.2.28), torch>=2.6 (resolved 2.11.0+cpu). Install verified clean — `import torchgen` succeeds, so not the corrupt-wheel failure recorded in the calibration doc. filters/span_filter.py runs the model on EVERY chunk, unconditionally. There is deliberately no relevance gate in front of it: a pre-filter's false negatives drop content before extraction sees it, and dropped content never reaches expert review to be caught. "Zero candidate spans" is the relevance signal instead. Two mechanical traps handled, both of which silently cost recall: - the model truncates long inputs and WARNS rather than failing, so chunks are fed as overlapping 130-word windows with character offsets remapped. Offsets are walked over the original spacing, never re-joined — re-joining would shift every offset in a chunk containing newlines and break every span check. - the ~12…3a01634 -
[NOTICKET] feat(eval): knowledge-extraction gold set, scorer + frozen prototype baseline
Rifqi Hafizuddin +1: Claude Fable 5P2 + P3 + P4 from the knowledge-pipeline plan. Lifts the two things out of the kex prototype that could not be recreated if the machine died — the calibration knowledge and the measurement baseline — without porting any pipeline code. eval/knowledge/ (new, follows the eval/{intent,help,readiness,planner} convention): - knowledge_gold.yaml 41 terms + 15 rules, verbatim from the prototype. Still provisional (not expert-reviewed); accepted as good enough so the rebuild is not blocked on expert time. - score.py ported; only change is dropping the prototype's sys.path bootstrap. Pipeline-independent: scores plain lists, so it works against the prototype's artifacts and v2 alike. - results/baseline_prototype_2026-08-13_145132.json FROZEN. The numbers v2 must match or beat. - README.md provenance, what the scorer refuses to conflate, and how …60cf4a8