eval/knowledge/README.md

eval/knowledge — knowledge-extraction eval harness

Scores the knowledge-extraction pipeline against a gold set built from BUMA STD/2026/006/MNO Rev.0.0 — Production Parameter & ECA (9 pages, majority Bahasa Indonesia).

Migrated from the kex prototype on 2026-08-19. The prototype itself is not being ported — it answered the design questions and stays where it is. This directory carries the parts that survive it: the gold set, the scorer, and a frozen baseline. Plan: KNOWLEDGE_PIPELINE_TODO.md, tuned constants: KNOWLEDGE_PIPELINE_CALIBRATION.md.

Why this landed before any v2 pipeline code

Without a standing baseline, "new and improved" is unfalsifiable — and one of the four experiments already failed (E3, schema fill). Quality is what this work gets judged on, so the yardstick ships first and every v2 stage is measured from its first commit rather than retroactively.

Contents

Path	What it is
knowledge_gold.yaml	41 terms + 14 rules. Provisional — bootstrapped by reading the source PDF, not expert-reviewed
score.py	Precision / recall / F1 per branch. Pipeline-independent: takes plain lists of surfaces and entry dicts, so it scores the prototype's artifacts and v2 alike
fixtures/parsed_STD_2026_006_MNO.json	Committed fixture artifact (2026-08-27, S7/K10). The BUMA standard parsed by MinerU 3.4.4 hybrid, schema_version 0.3.0, 31 chunks. Every score below is measured on it, and it is what makes this harness runnable with MinerU not installed
results/baseline_prototype_2026-08-13_145132.json	Frozen. The prototype's measured numbers. Never regenerate — add new timestamped files beside it

run_eval.py — the runner. Added 2026-08-26, following the house convention (module mode, timestamped results, never overwritten) used by eval/{intent,help,readiness,planner}/. The 2026-08-24 figures were assembled by hand; this is the reproducible version.

Running it

It takes a ParsedDocument artifact, never a PDF — MinerU is not invoked and is not required. That is the seam doing its job: point it at the chunks.json written by python -m src.knowledge_parsing.run — or at the committed fixture, which needs no parser at all: eval/knowledge/fixtures/parsed_STD_2026_006_MNO.json.

# free — E1 recall, E2 compression, the funnel. No API calls, no spend
./.venv/Scripts/python.exe -m eval.knowledge.run_eval <artifact.json>

# cost estimate before spending anything
./.venv/Scripts/python.exe -m eval.knowledge.run_eval <artifact.json> --dry-run

# the paid branches: E3 schema fill and rule recall
./.venv/Scripts/python.exe -m eval.knowledge.run_eval <artifact.json> --extract

# every branch, no spend, no credentials — plumbing only, NOT scoreable
./.venv/Scripts/python.exe -m eval.knowledge.run_eval <artifact.json> --extract --mock

Two things it refuses to do, both because a number that cannot be compared is worse than no number:

• It will not run on an artifact with no parser_backend. The same MinerU build emits different text from pipeline and hybrid/high, so results measured on different backends are not comparable and nothing downstream can tell them apart afterwards. --allow-unlabelled-artifact overrides and stamps the result file so the caveat travels with it.
• It never overwrites a result file, and refuses rather than clobbering if a name collides.

It also reports the latex_verification breakdown, which is the cheapest read on whether an artifact's markup can support the formula guard at all — a run that is all unverified_operator was parsed with a backend that cannot express multiplication. That figure costs nothing: the marker is set during validation, not by a model call.

The baseline to beat

Measured by the prototype on the document above, run 20260813-145132, deployment gpt-5.4-nano:

Experiment	Question	Result	Kill line	Verdict
E1	GLiNER recall on Bahasa Indonesia technical prose	0.854	0.70	PASS
E2	Does clustering cut expert review burden?	2.56× (169 → 66)	2.0×	PASS
E3	Is nano sufficient for schema fill?	0.75	0.80	FAIL
E4	Does verbatim-span validation catch fabrication?	1.00 (false-rejection 0.00)	0.90	PASS

Funnel: 9 pages → 13 chunks → 169 mentions → 66 clusters → 83 LLM calls → 66 entries → 66 queue items. Cost $0.0069; 125,184 of 155,313 prompt tokens served from cache.

Where v2 stands — results/v2_full_2026-08-27_133757.json

Measured on the committed fixture, deployment gpt-5.4-nano, one run:

Experiment	Result	Kill line	Verdict	vs v2_full_document_2026-08-24_093051.json
E1 term-filter recall	0.8293	0.70	PASS	+0.0244
E2 compression	2.556×	2.0	PASS	−0.254
E3 glossary schema fill (precision)	1.0000	0.80	PASS	+0.10
rule recall	0.4286 (6/14)	— tracked	—	+0.2286

Funnel: 9 pages → 31 chunks → 186 mentions → 63 clusters → 90 LLM calls → 63 glossary + 15 rule + 6 formula entries. 153,318 prompt tokens, 98,816 cached (64%); 0 fields rejected by the span check; rules_without_term_link 0.

Two caveats to carry when quoting these. E3's precision 1.00 sits on a scoreable base of 6 — 57 of 63 entries abstain with a null definition, so recall against the 18 gold terms with criteria is 0.33. And the rule-recall delta is partly a denominator change: the same 6 rules matched as in the 2026-08-27 08:29 run, but the gold base went 15 → 14 when X23 was closed on the expert ruling.

Two things the scorer refuses to do

It never conflates term-filter recall with extraction precision. They are different failure modes with different fixes — recall is fixed at the filter stage (GLiNER labels), precision at the extraction stage (model tier, prompt). E1 is the recall number specifically.

It never counts abstention as an error. For a term the document does not define, null is the correct answer. Scoring is restricted to the scoreable subset: entries that produced a definition, whose term is in the gold set, and whose gold record carries definition_contains to check against. Counting the rest as errors would measure gold coverage while claiming to measure model accuracy. Coverage is reported separately in Score.as_dict().

Reading E3 carefully

The scoreable base is 8 entries, so each adjudication moves the number ~12 points. nano is also not deterministic at temperature=0 — consecutive runs of the identical pipeline scored 0.75 and 0.625. Treat E3 as a signal, not a measurement, until the gold set is reviewed.

Every failure is listed verbatim under experiments.E3.scoreable_basis.failures in the baseline file, for Mas Beta to adjudicate. Some are genuine extraction errors; others are cases where the document carries two valid definitions and the provisional gold names only one. Gold was not edited after seeing output, so the figure is a lower bound.

Gold-set conventions (keep these when extending it)

• term is the string a reader would look up.
• full_name is the literal source wording, never normalised. The standard writes "Physical of Availability (PA)" in the heading and "Physical Availability" in the legend — both are recorded as variants. This is deliberate: the pipeline is required to surface that discrepancy to the expert rather than silently correct it.
• definition_contains are substrings that must appear in a correct definition. Substring matching, not exact, so a correct-but-differently-worded extraction is not scored as a miss.
• The gold set is partial by design — the expert is the labelling bottleneck, so scoring reports coverage rather than blocking on a complete file.

Status of the gold set

status: provisional, labelled_by: claude-bootstrap, labelled_on: 2026-08-13. It has not been reviewed by Mas Beta. Every score derived from it carries that caveat, and both E1 and E3 will move when it is reviewed. Accepted as good enough for now (2026-08-19, Rifqi) so the rebuild is not blocked on expert time.