Commit History
-
[NOTICKET] eval(knowledge): commit the BUMA fixture artifact; K9/K10 close
Rifqi Hafizuddin +1: Claude Opus 5The 2026-08-27T13:28 re-parse of STD_2026_006_MNO becomes the committed fixture (S7/K10). Same document and content_hash as the 08-26 parse, same parser_config, five corrected chunk boundaries — of which ::0030 is the one that matters: it previously held only "Keterangan grafik:" and now carries the hierarchy-level paragraph, moving Controllable / Hierarki Level Analysis from E1 miss to hit. One paid run on it (90 calls, 170 s): E1 0.8293 / E2 2.556x / E3 1.000, all three above their kill lines, rule recall 0.4286 (6/14). 0 fields rejected by the span check, rules_without_term_link 0. Two caveats recorded beside the numbers rather than left for a reader to rediscover: E3's precision 1.000 sits on a scoreable base of 6 (57 of 63 entries abstain, recall 0.33 against the 18 gold terms with criteria), and the rule-recall delta is partly a denominator change — the same 6 rules matched as in the 08:29 run, the gold base went 15 to 14 when X23 closed. The harness now runs with data/knowled…e5f4abe -
[NOTICKET] feat(knowledge_extraction): cue class for declarative composition (X24b cause 3)
Rifqi HafizuddinThe rule branch could not see three gold rules because the cue taxonomy had no slot for the voice they are written in. It had conditional cues (jika, apabila, harus) and definitional ones (adalah, merupakan). This corpus states most of its calculation policy in a third voice -- declarative composition: "Nilai PA ... EKUIVALEN DENGAN rata-rata tertimbang ... dengan Quantity sebagai FAKTOR PEMBOBOT" "Perhitungan Pty ... DIPEROLEH DARI hasil pembobotan ..." "Gain/Loss grouping ... merupakan PENJUMLAHAN DARI Gain/Loss model unit" Those are calculation rules by the extraction prompt's own definition -- how a value is computed, sourced or aggregated -- and no existing cue reached any of them. Not a tuning gap; a missing class. Five patterns, chosen by measuring which chunks each reaches rather than by intuition, every one load-bearing for a previously invisible gold rule. Candidates 8 -> 15. \bdihitung menggunakan\b was REJECTED despite matching 5 chunks: it precedes every formu…e6d9d29
-
[NOTICKET] eval(knowledge): rule-branch noise band — X20 confirmed, X23 answered
Rifqi HafizuddinFive repeat runs at identical config, plus the two from earlier today: seven runs of the same pipeline on the same artifact at temperature 0. 3/15 x1 (the run carrying the latex-guard defect) 4/15 x5 6/15 x1 min 3, median 4, max 6, mean 4.14 The branch is much less noisy than one pair of runs suggested. Five of seven land on exactly 4/15. The rule from here on is: quote the median and the band, never a single run. Per-rule stability turned out to be the useful view, not the total. X20 IS CONFIRMED, and by the rule it names. GAINLOSS_UNITS -- the units sentence that shared a chunk with a Gain/Loss equation, the exact case the fix was written for -- fires in 6 of 7 runs against 0 at the 3/15 baseline. That is a stable +1 sitting outside the noise band, not a lucky sample. Three other rules fire 7/7 and two fire 1/7; those two are the whole of the variance. X23'S GOLD QUESTION IS ANSWERED. With the few-shot that forbade it removed, WATERFALL_TWO_METHODS still f…f754631 -
[NOTICKET] fix(knowledge_extraction): the latex guard was rejecting correct transcriptions
Rifqi Hafizuddin +1: Claude Opus 5First real run of the new runner, on Sofhia's hybrid artifact, and it caught a defect I shipped two commits ago. 4 of 6 formulas had formula_latex nulled -- Qty twice, PA and UA -- and all four rejections were FALSE. The cause is the same identity trap, one level deeper. The model reads RENDERED PROSE, and the document writes `Qty = (INPR Hours) / (Total Hours)`. So it answers with a slash and the document's own parentheses, while the source markup carries `\frac{INPR Hours}{Total Hours}` with neither. Two spellings of one division, and the guard called them different formulas. Three fixes, all notational: Division canonicalises to the prose spelling: `\frac{A}{B}` -> `(A)/(B)`, so a slash and a \frac converge instead of diverging. The old `frac(A)(B)` form was also unsafe on its own terms -- it used parentheses as delimiters, so a literal paren in the numerator collided with the delimiter and `\frac{(a)}{(b)}` could never match `\frac{a}{b}`. Redundant grouping parens collapse: `(…22eb3d4
-
[NOTICKET] eval(knowledge): full document through both halves — E3 0.90 PASS; fix legend blank-line bug
Rifqi Hafizuddin +1: Claude Fable 5First full-document run with the parsing module building the artifact and the extraction module running every branch. Result committed beside the frozen baseline: eval/knowledge/results/v2_full_document_2026-08-24_093051.json E1 term-filter recall 0.8049 (prototype 0.8537) PASS, kill 0.70 E3 schema fill 0.90 (prototype 0.75) PASS, kill 0.80 rule-branch recall 0.0667 (prototype 0.2667) REGRESSION E3 is the headline: the one experiment that FAILED in the prototype now clears its line. It needs the same care the original number did — the scoreable base is 10 entries, so one adjudication moves it ~10 points, and nano is not deterministic at temperature 0. The single failure (produksi) is the same one the prototype recorded and reads as a gold-adjudication case, not a model error. E1 sits below the baseline because the CHUNKING differs, not the filter: the parsing module splits to a token budget (31 chunks) where the earlier comparison used 14, so th…ec7d3d4
-
[NOTICKET] eval(knowledge): run the pipeline on MinerU-parsed input — parser comparison
Rifqi Hafizuddin +1: Claude Fable 5First run of the extraction half on Rifqi's MinerU output. Same document, same gold set, same pipeline code, so the only variable is the parse. PyMuPDF (what the frozen baseline used) E1 0.8537 35/41 MinerU, raw markup E1 0.7561 31/41 MinerU, formulas + tables rendered to prose E1 0.8293 34/41 The span model reads prose, not markup. MinerU's structure helps the formula branch and hurts the filter unless rendered first: - Terms wrapped in LaTeX are invisible. "\text {Total Hours} - \text {Breakdown}" produces no mentions for either, though both are gold terms found in the plain-text parse. Unwrapping \text/\mathrm/\frac and collapsing MinerU's character-spaced output recovers them (+7.3 points). - Raw <table> HTML sat in 8 of 14 chunks before rendering. The residual ~2.4 points are CHUNKING, not content: MinerU marks non-section labels (Keterangan:, Keterangan grafik:) as text_level titles, so opening a section at each splits a legend…6038b2b -
[NOTICKET] feat(knowledge_extraction): span filter — E1 recall 0.8537, matches the frozen baseline
Rifqi Hafizuddin +1: Claude Fable 5Adds the term-filter stage, the last free stage before the paid extraction call. Dependencies (approved): gliner>=0.2.13 (resolved 0.2.28), torch>=2.6 (resolved 2.11.0+cpu). Install verified clean — `import torchgen` succeeds, so not the corrupt-wheel failure recorded in the calibration doc. filters/span_filter.py runs the model on EVERY chunk, unconditionally. There is deliberately no relevance gate in front of it: a pre-filter's false negatives drop content before extraction sees it, and dropped content never reaches expert review to be caught. "Zero candidate spans" is the relevance signal instead. Two mechanical traps handled, both of which silently cost recall: - the model truncates long inputs and WARNS rather than failing, so chunks are fed as overlapping 130-word windows with character offsets remapped. Offsets are walked over the original spacing, never re-joined — re-joining would shift every offset in a chunk containing newlines and break every span check. - the ~12…3a01634 -
[NOTICKET] feat(eval): knowledge-extraction gold set, scorer + frozen prototype baseline
Rifqi Hafizuddin +1: Claude Fable 5P2 + P3 + P4 from the knowledge-pipeline plan. Lifts the two things out of the kex prototype that could not be recreated if the machine died — the calibration knowledge and the measurement baseline — without porting any pipeline code. eval/knowledge/ (new, follows the eval/{intent,help,readiness,planner} convention): - knowledge_gold.yaml 41 terms + 15 rules, verbatim from the prototype. Still provisional (not expert-reviewed); accepted as good enough so the rebuild is not blocked on expert time. - score.py ported; only change is dropping the prototype's sys.path bootstrap. Pipeline-independent: scores plain lists, so it works against the prototype's artifacts and v2 alike. - results/baseline_prototype_2026-08-13_145132.json FROZEN. The numbers v2 must match or beat. - README.md provenance, what the scorer refuses to conflate, and how …60cf4a8