eval/knowledge/README.md

# eval/knowledge — knowledge-extraction eval harness

Scores the knowledge-extraction pipeline against a gold set built from
**BUMA `STD/2026/006/MNO Rev.0.0` — Production Parameter & ECA** (9 pages, majority Bahasa
Indonesia).

Migrated from the `kex` prototype on 2026-08-19. **The prototype itself is not being ported** — it
answered the design questions and stays where it is. This directory carries the parts that survive
it: the gold set, the scorer, and a frozen baseline. Plan: [KNOWLEDGE_PIPELINE_TODO.md](../../docs/knowledge/KNOWLEDGE_PIPELINE_TODO.md),
tuned constants: [KNOWLEDGE_PIPELINE_CALIBRATION.md](../../docs/knowledge/KNOWLEDGE_PIPELINE_CALIBRATION.md).

## Why this landed before any v2 pipeline code

Without a standing baseline, "new and improved" is unfalsifiable — and one of the four experiments
already **failed** (E3, schema fill). Quality is what this work gets judged on, so the yardstick
ships first and every v2 stage is measured from its first commit rather than retroactively.

## Contents

| Path | What it is |
|---|---|
| `knowledge_gold.yaml` | 41 terms + 14 rules. **Provisional** — bootstrapped by reading the source PDF, *not* expert-reviewed |
| `score.py` | Precision / recall / F1 per branch. Pipeline-independent: takes plain lists of surfaces and entry dicts, so it scores the prototype's artifacts and v2 alike |
| `fixtures/parsed_STD_2026_006_MNO.json` | **Committed fixture artifact (2026-08-27, S7/K10).** The BUMA standard parsed by MinerU 3.4.4 `hybrid`, `schema_version 0.3.0`, 31 chunks. Every score below is measured on it, and it is what makes this harness runnable **with MinerU not installed** |
| `results/baseline_prototype_2026-08-13_145132.json` | **Frozen.** The prototype's measured numbers. Never regenerate — add new timestamped files beside it |

`run_eval.py` — **the runner. Added 2026-08-26**, following the house convention (module mode,
timestamped results, never overwritten) used by `eval/{intent,help,readiness,planner}/`. The
2026-08-24 figures were assembled by hand; this is the reproducible version.

## Running it

**It takes a `ParsedDocument` artifact, never a PDF — MinerU is not invoked and is not required.**
That is the seam doing its job: point it at the `chunks.json` written by
`python -m src.knowledge_parsing.run` — or at the committed fixture, which needs no parser at all:
`eval/knowledge/fixtures/parsed_STD_2026_006_MNO.json`.

```bash
# free — E1 recall, E2 compression, the funnel. No API calls, no spend
./.venv/Scripts/python.exe -m eval.knowledge.run_eval <artifact.json>

# cost estimate before spending anything
./.venv/Scripts/python.exe -m eval.knowledge.run_eval <artifact.json> --dry-run

# the paid branches: E3 schema fill and rule recall
./.venv/Scripts/python.exe -m eval.knowledge.run_eval <artifact.json> --extract

# every branch, no spend, no credentials — plumbing only, NOT scoreable
./.venv/Scripts/python.exe -m eval.knowledge.run_eval <artifact.json> --extract --mock
```

Two things it refuses to do, both because a number that cannot be compared is worse than no number:

- **It will not run on an artifact with no `parser_backend`.** The same MinerU build emits different
  text from `pipeline` and `hybrid/high`, so results measured on different backends are not
  comparable and nothing downstream can tell them apart afterwards. `--allow-unlabelled-artifact`
  overrides and stamps the result file so the caveat travels with it.
- **It never overwrites a result file**, and refuses rather than clobbering if a name collides.

It also reports the `latex_verification` breakdown, which is the cheapest read on whether an
artifact's markup can support the formula guard at all — a run that is all `unverified_operator`
was parsed with a backend that cannot express multiplication. That figure costs nothing: the marker
is set during validation, not by a model call.

## The baseline to beat

Measured by the prototype on the document above, run `20260813-145132`, deployment `gpt-5.4-nano`:

| Experiment | Question | Result | Kill line | Verdict |
|---|---|---|---|---|
| **E1** | GLiNER recall on Bahasa Indonesia technical prose | **0.854** | 0.70 | **PASS** |
| **E2** | Does clustering cut expert review burden? | **2.56×** (169 → 66) | 2.0× | **PASS** |
| **E3** | Is nano sufficient for schema fill? | **0.75** | 0.80 | **FAIL** |
| **E4** | Does verbatim-span validation catch fabrication? | **1.00** (false-rejection 0.00) | 0.90 | **PASS** |

Funnel: 9 pages → 13 chunks → 169 mentions → 66 clusters → 83 LLM calls → 66 entries → 66 queue
items. Cost $0.0069; 125,184 of 155,313 prompt tokens served from cache.

### Where v2 stands — `results/v2_full_2026-08-27_133757.json`

Measured on the committed fixture, deployment `gpt-5.4-nano`, one run:

| Experiment | Result | Kill line | Verdict | vs `v2_full_document_2026-08-24_093051.json` |
|---|---|---|---|---|
| **E1** term-filter recall | **0.8293** | 0.70 | **PASS** | +0.0244 |
| **E2** compression | **2.556×** | 2.0 | **PASS** | −0.254 |
| **E3** glossary schema fill (precision) | **1.0000** | 0.80 | **PASS** | +0.10 |
| **rule recall** | **0.4286** (6/14) | — tracked | — | +0.2286 |

Funnel: 9 pages → 31 chunks → 186 mentions → 63 clusters → 90 LLM calls → 63 glossary + 15 rule +
6 formula entries. 153,318 prompt tokens, 98,816 cached (64%); 0 fields rejected by the span check;
`rules_without_term_link` 0.

**Two caveats to carry when quoting these.** E3's precision 1.00 sits on a **scoreable base of 6** —
57 of 63 entries abstain with a null definition, so recall against the 18 gold terms with criteria is
0.33. And the rule-recall delta is partly a denominator change: the same **6** rules matched as in
the 2026-08-27 08:29 run, but the gold base went 15 → 14 when X23 was closed on the expert ruling.

## Two things the scorer refuses to do

**It never conflates term-filter recall with extraction precision.** They are different failure
modes with different fixes — recall is fixed at the filter stage (GLiNER labels), precision at the
extraction stage (model tier, prompt). E1 is the recall number specifically.

**It never counts abstention as an error.** For a term the document does not define, `null` is the
correct answer. Scoring is restricted to the *scoreable* subset: entries that produced a definition,
whose term is in the gold set, and whose gold record carries `definition_contains` to check against.
Counting the rest as errors would measure gold coverage while claiming to measure model accuracy.
Coverage is reported separately in `Score.as_dict()`.

## Reading E3 carefully

The scoreable base is **8 entries**, so each adjudication moves the number ~12 points. nano is also
not deterministic at `temperature=0` — consecutive runs of the identical pipeline scored 0.75 and
0.625. Treat E3 as a signal, not a measurement, until the gold set is reviewed.

Every failure is listed verbatim under `experiments.E3.scoreable_basis.failures` in the baseline
file, for Mas Beta to adjudicate. Some are genuine extraction errors; others are cases where the
document carries two valid definitions and the provisional gold names only one. **Gold was not
edited after seeing output**, so the figure is a lower bound.

## Gold-set conventions (keep these when extending it)

- `term` is the string a reader would look up.
- `full_name` is the **literal source wording**, never normalised. The standard writes *"Physical of
  Availability (PA)"* in the heading and *"Physical Availability"* in the legend — both are recorded
  as variants. This is deliberate: the pipeline is required to surface that discrepancy to the
  expert rather than silently correct it.
- `definition_contains` are substrings that **must** appear in a correct definition. Substring
  matching, not exact, so a correct-but-differently-worded extraction is not scored as a miss.
- The gold set is **partial by design** — the expert is the labelling bottleneck, so scoring reports
  coverage rather than blocking on a complete file.

## Status of the gold set

`status: provisional`, `labelled_by: claude-bootstrap`, `labelled_on: 2026-08-13`. It has **not**
been reviewed by Mas Beta. Every score derived from it carries that caveat, and both E1 and E3 will
move when it is reviewed. Accepted as good enough for now (2026-08-19, Rifqi) so the rebuild is not
blocked on expert time.