Commit History
-
[NOTICKET] feat(knowledge-eval): freeze the Mistral baseline (M1)
Rifqi Hafizuddin +1: Claude Opus 5Milestone 1 - the FULL pipeline is now proven on Mistral, not just parsing and the term filter. Every kill line passes and three of four metrics are the best recorded. E1 term-filter recall 0.8780 (kill 0.70) E2 review compression 2.6970 (kill 2.00) E3 glossary schema fill 1.0000 (kill 0.80) rule recall 0.5000 (8/16, tracked not gated) 92 calls, 156,386 prompt tokens, 62% cached, 0 fields rejected by the span check, latex_verification {'verified': 5}, key_parameters_unresolved 0. The three features flagged as at-risk on a parser swap all work: - `outline` 13 entries, including the verbatim "Physical of Availability (PA)" - `source_wording` 13/13 headings locatable; PA surfaces as "Physical of Availability (PA)" against full_name "Physical of Availability" - the locked discrepancy behaviour survives Mistral + markdown stripping - formula guard: all 5 formulas `verified`, none `unverified_no_markup` Two gates ran before spendi…a358448
-
[NOTICKET] eval(knowledge): X20 measures net -1; close X23 on expert ruling; instrument X25
Rifqi HafizuddinThree things, all from the same session. X20 -- FIRST CONTROLLED A/B IN THIS PROJECT, AND IT SAYS X20 COSTS A RULE. Arms alternated inside one session so drift hit both equally. Arm B reverts X20's three edits and KEEPS X23's synthetic examples, so the only variable is X20 itself. Rule branch only: 15 calls per run, ~35s, five rounds each. X20 in 5/14, every one of 5 runs X20 reverted 6/14, every one of 5 runs Zero variance in both arms, and exactly one rule differs: GAINLOSS_QTY_ZERO_PLAN, 0% with X20 and 100% without. The cause is X20's own final clause. That gold rule is a genuine conditional whose CONDITION is written symbolically -- "Jika (Qty_P(M) = 0, Qty_A(M) > 0), maka:" -- and X20 ends with "a statement that is mostly symbols, = and numbers is an equation". The heuristic cannot tell a computation from a symbolic condition, so it nulls a real rule. X20 is also not buying what it was written to buy. GAINLOSS_UNITS, the rule it targets, fires 0% in BOTH arms …e6ed414
-
[NOTICKET] eval(knowledge): the runner the harness has been missing
Rifqi Hafizuddin +1: Claude Opus 5The 2026-08-24 figures were assembled by hand. That is why the rule-branch number in the committed result file is 1/15 measured with a scorer that has since been fixed, and why nobody could re-derive it. This is the reproducible version, following the house convention the other four harnesses already use: module mode, timestamped results, never overwritten. It takes a ParsedDocument artifact and never a PDF. MinerU is not invoked and is not required -- that is the seam doing its job, and it is what makes the pending X20 and X23 measurements possible on a machine with no GPU and no parser installed. Free by default: E1 term-filter recall and E2 compression, plus the funnel, with zero API calls. --extract adds the paid branches, E3 schema fill and rule recall. --dry-run prints the token estimate and the per-branch cache-floor status. --mock exercises every branch with no spend, and stamps the result NOT SCOREABLE so a fabricated number can never be quoted as a measurement. Two refusal…1edf61c