Commit History
-
[NOTICKET] feat(knowledge-parsing): commit the reference artifact, and stop the stamp churning
Three changes, all aimed at one problem: the two halves of the pipeline could not reliably agree on which artifact they were measuring. **Metadata now precedes `chunks` in the artifact.** Pydantic serialises in declaration order, so it decides what a person sees on opening the file. With `chunks` first, checking which parser or normaliser produced an artifact meant scrolling past thousands of lines to the end — and that is a routine check now, not a rare one. JSON objects are unordered by specification and every consumer reads by key, so this changes nothing for code and `schema_version` is unchanged: no field, type or meaning moved. **`contracts.py` is out of the normaliser hash.** The artifact's SHAPE is already versioned by `schema_version`, so hashing it again counted the same change twice. The reordering above proved the cost: the stamp moved while the chunks came out byte-identical, which is a false "your artifact is stale". A signal that cries wolf gets ignored, and this one e…a0e59e4 -
[NOTICKET] feat(knowledge-parsing): stamp the artifact with the code that built it
A chunks.json is a frozen snapshot of the normalisation code at the moment it ran. When that code is fixed, existing artifacts do not update themselves — and nothing in their metadata said so. `parser_config` fingerprints MinerU's settings; `schema_version` versions the artifact's SHAPE, which a normalisation bugfix does not change. So a stale artifact and a correct one declared identical metadata, and the only observable difference was the character count. That cost a day. A pre-fix artifact reached the extraction half, term recall came back 0.7073 against a 0.8049 baseline, and it read as a regression in extraction's own work. It was neither side's setup: same filter code, different input. Verified here — scoring that artifact reproduces 169 mentions and 0.7073 exactly, against 186 and 0.8293 for the rebuilt one. `normaliser_version` is a hash of the modules that decide what a chunk CONTAINS: normalize.py, render.py, checks.py, contracts.py. A source hash rather than a hand-mainta…c64a0cb
-
[NOTICKET] fix(knowledge-parsing): normalize.py silently dropped every list block
MinerU keeps list content in `list_items` and gives such items no `text` field. They therefore fell through the generic text path, which reads `item["text"]`, came back empty, and were skipped — no error, no warning, no trace. This is the same shape of bug the chart/image branch was written to prevent, and that branch's comment describes it exactly: "pictures have no such field — so BOTH its caption and its description vanish in silence". Lists have no such field either, and nobody noticed. Measured cost on the BUMA standard: two dropped blocks took five gold terms with them — Joint survey, Truck count, EWH, Controllable, Hierarki Level Analysis. None of the five were "missed by the model"; none were in the artifact at all. Term-filter recall 0.7073 -> 0.8293, +0.1220. List content joins the running text chunk rather than opening its own, because it is prose inside its section. It deliberately skips heading detection: a short bullet can look like a heading, and letting one open a se…6edfbe7