docs/knowledge/KNOWLEDGE_PARSING_OUTPUT_CONTRACT.md

# Knowledge Pipeline — Parsing Output Contract

<!-- doc-status -->
> 🔄 **LIVE — the seam.** Output contract of the parsing half, consumed by `src/knowledge_extraction/adapter.py`. Source of truth is `src/knowledge_parsing/contracts.py`; this doc describes it.

**Source of truth:** `src/knowledge_parsing/contracts.py`
**Producer:** `python -m src.knowledge_parsing.run --input data/knowledge_docs/`
**Reference run:** `20260824-100720` · BUMA `STD_2026_006_MNO` · 9 pages · MinerU `3.4.4`, backend `pipeline`
**Status:** offline artifacts only — no endpoint, no table. Values below are **literal from that run**, not illustrative.

Companion to `KNOWLEDGE_OUTPUT_CONTRACT.md` (extraction). This is the half **upstream** of it:
parsing emits `ParsedDocument`, extraction consumes it and never touches the parser.

> ## ⚠️ schema_version **0.4.0** — asset pointers (2026-09-10)
>
> **A figure is no longer its own chunk.** Its markdown reference is folded into the prose chunk at
> its reading-order position, and the figure travels as an `Asset` on that chunk. This is a **MINOR**
> bump, not a patch: chunk boundaries change, so a 0.3.x consumer would silently read a different
> document shape. `adapter.check_schema_version` rejects a newer minor by design — parsing and
> extraction ship together.
>
> **Why.** Before 0.4.0 an image became its own chunk with an **empty `text`**, which made it
> unreachable by construction: no text → no term mentions → joins no cluster → never selected as
> evidence. Measured across three documents: **13 figures, every one described by a vision model we
> pay for, and ZERO appearing in any cluster's evidence.** Meanwhile the prose that said
> "Figure 1.1" had its `![](…)` reference stripped, so nothing connected the sentence to the picture.
>
> **What changed**
>
> | | 0.3.3 | 0.4.0 |
> |---|---|---|
> | A figure | its own chunk, `text=""`, unreachable | folded into the prose chunk as `![](asset://<id>)`, carried in `Chunk.assets` |
> | A table | its own chunk | **unchanged** — its linearised text earns its evidence slot (74.6% of the Komatsu manual's top-3 evidence is tabular). Gains an `Asset` + two-way pointers |
> | Asset bytes | returned by the API, **written nowhere**; `images[]` pointed at files that did not exist | content-addressed and persisted; `Asset.storage_key` |
> | Asset identity | the parser's page-local filename | **`sha256(bytes)[:16]`** |
> | A caption | landed in a *neighbouring* prose chunk, unlinked | attached to the `Asset` **and** left in the prose text (it is verbatim document content) |
> | A description | stored raw, JSON braces included | unwrapped; `description_source` records what produced it |
>
> **Measured effect** (same three documents, no re-parse — the cache fingerprint is unchanged):
> chunks 32→28, 85→80, 59→44 · **image chunks 2/3/8 → 0/0/0** · empty-text chunks → 0 ·
> 13 assets persisted · 0 dangling references · **0 descriptions leaked into `text`** ·
> captions still 17/17 in text · recall unmoved.
>
> **The invariant that must not break:** a model-written `description` never enters `Chunk.text`.
> `text` is the haystack the span check searches, so generated prose in there would let a
> fabrication pass the control built to catch it.

> **Current state (2026-09-02) — additive changes since the reference run below.** The default
> parser is now **Mistral OCR** (`parser_name: "mistral-ocr"`, `parser_backend: "mistral-ocr"`),
> API-based and needing **no GPU**. The artifact shape is **identical whichever parser produced it** —
> extraction never learns which ran, which is what made the swap possible at all.
>
> **2026-09-07:** **MinerU was DROPPED 2026-09-07** (Rifqi): commented out of the `knowledge-parsing` extra and installed nowhere — not the Space, not the dev venv (verified). It is **not a fallback**; there is one parser, Mistral OCR. Its code paths in `src/knowledge_parsing/` are untouched (comment-out-don't-delete) and every MinerU measurement in these docs stands as record. Dropping it was also forced: `mineru==3.4.4` pinned against this project's `python-docx` and `pypdf`, which made `uv lock` fail for the whole project. The `parser_name` / `parser_backend` fields stay in the contract:
> committed artifacts record `mineru`, and a reader must keep distinguishing them.
> `schema_version` is now **`0.3.3`**, all changes additive (null/derived on older artifacts):
> `source_title` (0.3.2) and `normaliser_version` (0.3.1) at the document level, and a computed
> 1-based **`page_no`** (0.3.3) on each Chunk. The reference JSON below is still the literal
> 2026-08-24 **MinerU** run, so it predates these fields.

---

## Transport reality — read this first

There is **no endpoint and no table yet.** The CLI writes one directory per run:

```
data/knowledge_runs/<run_id>/
├── manifest.json                    run record: config, timings, quality warnings
├── failures.jsonl                   documents that failed, with tracebacks
└── <doc_id>/
    ├── chunks.json                  ← the artifact: one ParsedDocument
    └── mineru-source.txt            pointer to the untouched MinerU output
```

**Unlike the extraction payloads, this artifact is not a bare array.** `chunks.json` is a single
self-describing object that carries its own provenance — parser name, version, backend, config
fingerprint, source hash. The envelope proposed at the end of the extraction contract already exists
here, and is worth reusing rather than inventing a second shape.

Three conventions hold:

- **`page_idx` is 0-based**, exactly as MinerU reports it, with no conversion anywhere. Converting to
  1-based is the UI's job, done once at display, so the artifact always matches the raw MinerU output
  kept beside it in the cache.
- **`text` is VERBATIM.** Extraction's span check locates quoted spans literally inside it. Reflowing
  or normalising it makes the lookup fail and the field goes silently null — which presents as a bad
  model, not as a parser bug.
- **Markup never goes in `text`.** Formulas and tables are rendered to readable prose for `text`, and
  preserved in their original form in `latex` / `table_html`.

---

## 1 · `chunks.json` → `ParsedDocument` *(single object)*

```json
{
  "doc_id": "STD_2026_006_MNO",
  "chunks": [ "… Chunk[] …" ],
  "source_path": "data\\knowledge_docs\\STD_2026_006_MNO.pdf",
  "content_hash": "f10807178b20531e86237a2640ff4125d259aa68e5e9d9f910c0c2365cd92833",
  "n_pages": 9,
  "parser_name": "mineru",
  "parser_version": "3.4.4",
  "parser_backend": "pipeline",
  "parser_config": "b367868551b9",
  "version": 1,
  "schema_version": "0.3.0",
  "created_at": "2026-08-24T10:07:21",
  "raw_output_dir": "…/data/knowledge_cache/parse/f10807178b20531e-b367868551b9-3.4.4"
}
```

| Field | Type | Req | Notes |
|---|---|---|---|
| `doc_id` | string | ✅ | Source filename stem |
| `chunks` | Chunk[] | ✅ | 31 on the reference run |
| `source_path` | string | ✅ | |
| `content_hash` | string | ✅ | sha256 of the **source file** — a changed document forces a new artifact |
| `n_pages` | int | ✅ | |
| `parser_name` | string | ✅ | Default `"mineru"`. The field exists so a Tesseract / Azure DI path is distinguishable |
| `parser_version` | string \| null | — | e.g. `"3.4.4"` |
| `parser_backend` | string \| null | — | `pipeline` · `vlm` · `hybrid`. **Read back from MinerU's `_middle.json`** — what actually ran, not what was requested |
| `parser_config` | string \| null | — | 12-char fingerprint of the settings that affect output |
| `source_title` | string \| null | — | **(0.3.2)** The document's human-readable name. Equals `doc_id` today; a separate field so the two can diverge once `doc_id` becomes a content hash |
| `normaliser_version` | string \| null | — | **(0.3.1)** Fingerprint of the normalisation code that produced these chunks. `parser_config` covers the parser's settings; this covers our half |
| `version` | int | ✅ | Version of **this artifact for this document**; bumped on re-parse |
| `schema_version` | string | ✅ | Versions **the contract**. `0.3.0` on this reference run; **current `0.4.0`** (see the banner above) |
| `created_at` | string \| null | — | ISO 8601, seconds precision |
| `raw_output_dir` | string \| null | — | Untouched MinerU output, kept content-addressed and never copied |

> **Why provenance is mandatory rather than nice-to-have:** without
> `parser_version` / `parser_backend`, a MinerU upgrade and a prompt change are
> indistinguishable when extraction results shift.

---

## 2 · `Chunk`

```json
{
  "chunk_id": "STD_2026_006_MNO::0003",
  "doc_id": "STD_2026_006_MNO",
  "kind": "text",
  "text": "Adalah serangkaian kegiatan yang dilakukan untuk menciptakan nilai tambah terhadap sumber daya …",
  "page_idx": 1,
  "page_idxs": [1],
  "section_no": "2.1.1",
  "heading": "Production",
  "chapters": [],
  "heading_path": ["2.1.1. Production"],
  "has_formula": true,
  "is_tabular": false,
  "source_items": [6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16],
  "bbox": [122, 293, 275, 307],
  "images": ["images/76503982720c45fc9d387accb74d5f5c5083f4f3109ed7fbf6a4da345bc20e0f.jpg"],
  "latex": ["$$\nP r o d u c t i o n = M O H H x Q t y x P A x U A x P t y\n$$"],
  "table_html": null
}
```

| Field | Type | Req | Notes |
|---|---|---|---|
| `chunk_id` | string | ✅ | **Format `<doc_id>::<seq>`**, `seq` zero-padded to 4. Note the `::` separator — see *Known gaps* |
| `doc_id` | string | ✅ | |
| `kind` | enum | ✅ | `text` · `table` · `chart` · `equation` · `image`. **Backends disagree on the label** — a figure `pipeline` calls a chart, `vlm` calls an image. Branch on both, never on one |
| `text` | string | ✅ | **VERBATIM.** Excludes the section heading line — that lives in `heading`. Consumers needing both compose them |
| `page_idx` | int | ✅ | **0-based.** First page of the chunk |
| `page_idxs` | int[] | ✅ | Every page the chunk spans. Default `[]` |
| `page_no` | int | ✅ | **(0.3.3) 1-based**, computed `page_idx + 1`. The number a human / review UI reads; derived, never stored, so it cannot drift from `page_idx` |
| `section_no` | string \| null | — | e.g. `"2.1.3"`; null when the document is unnumbered |
| `heading` | string \| null | — | This chunk's own section title, **verbatim as written** — `"Physical of Availability (PA)"`, not the tidied `"Physical Availability"` |
| `chapters` | string[] | ✅ | Running header per page spanned. Default `[]` |
| `heading_path` | string[] | ✅ | Breadcrumb of enclosing headings, outermost first |
| `has_formula` | bool | ✅ | Cheap downstream filter |
| `is_tabular` | bool | ✅ | |
| `source_items` | int[] | ✅ | Indices into MinerU's `content_list.json` — the trace back |
| `bbox` | int[] \| null | — | `[x0, y0, x1, y1]` of the first source item. Carried for a curation UI; **nothing in the pipeline reasons about it** — ordering uses item sequence, never coordinates |
| `images` | string[] | ✅ | ⚠️ **DEPRECATED 0.4.0** — superseded by `assets`. Still populated for one version so a 0.3.x reader degrades rather than breaking. Holds the parser's own filenames, which are **not reliable**: Mistral reported `image_id: img-2.jpeg` on a block whose markdown said `img-1.jpeg`, so two figures collided and one image's bytes were lost. Read `assets` |
| `assets` | Asset[] | ✅ | **New 0.4.0.** Figures and tables this chunk contains or points at. Default `[]` |
| `referenced_by` | string[] | ✅ | **New 0.4.0.** `chunk_id`s pointing AT this chunk. Populated on table chunks. Default `[]` |
| `latex` | string[] | ✅ | Source markup for formulas, kept beside the rendered prose. ⚠️ **LOAD-BEARING** — extraction's formula guard validates against this field. Reflowing it or leaving it empty weakens the check with no error. See below |
| `table_html` | string \| null | — | Source markup for tables |
| `generated_description` | string \| null | — | ⚠️ **MODEL-WRITTEN, NOT FROM THE DOCUMENT.** MinerU's description of an image or chart. **Never span-checkable** — treat as a model claim about a picture, never as evidence. See below |

## 2b · `Asset` *(new at 0.4.0)*

```json
{
  "asset_id": "d2e7b4ffe31006ec",
  "kind": "figure",
  "storage_key": "assets/d2e7b4ffe31006ec.jpeg",
  "page_idx": 2,
  "page_no": 3,
  "bbox": [93, 562, 434, 907],
  "caption": "Figure 1.1. The relationship between exploration information, resources and reserves (SME, 1991).",
  "description": "The diagram illustrates the classification of mineral resources and reserves based on increasing levels of geological knowledge…",
  "description_source": "mistral-ocr/bbox_annotation"
}
```

| Field | Type | Req | Notes |
|---|---|---|---|
| `asset_id` | string | ✅ | **`sha256(bytes)[:16]`.** Not the parser's id — see the `images` row. Content addressing makes a re-parse idempotent and de-duplicates a figure that appears twice (Open Pit has one such pair) |
| `kind` | enum | ✅ | `figure` · `chart` · `diagram` · `table` — one vocabulary for both cases |
| `storage_key` | string \| null | — | Where the bytes live. `null` for a table (no bytes of its own) and whenever persistence was skipped or failed — writing assets is a **never-throw** seam |
| `page_idx` | int | ✅ | 0-based, as the parser reports |
| `page_no` | int | ✅ | **Computed** `page_idx + 1`. The number a human reads; derived so it cannot drift |
| `bbox` | int[] \| null | — | `[x0,y0,x1,y1]` of the **asset**, so a UI can highlight the figure. Distinct from the host chunk's `bbox`, which stays where the text is |
| `caption` | string \| null | — | **VERBATIM** from the document's own caption block. Quotable as evidence, span-checkable. Also left in the prose `text`, where it already landed — the asset gets a copy, not a move |
| `description` | string \| null | — | ⚠️ **MODEL-WRITTEN.** Never span-checkable, never evidence, and **never in `text`** |
| `description_source` | string \| null | — | e.g. `mistral-ocr/bbox_annotation`, so a later model swap is traceable rather than silently changing what the field means |

**A figure's reference appears inline in its host chunk's `text`**, at the reading-order position the
document put it:

```text
…so closely and the geological character is so well defined that size, shape, depth
and mineral content of the resource are well established.

![](asset://d2e7b4ffe31006ec)

Figure 1.1. The relationship between exploration information, resources and reserves (SME, 1991).

– Indicated. Quantity and grade…
```

Two consumer obligations:

- **Resolve `asset://<id>` against that chunk's own `assets`.** Every reference is guaranteed to
  resolve locally (0 dangling across the three test documents).
- **Switch on the caption/description distinction.** A caption may be quoted and used as a provenance
  span; a description may not, and a span taken from one is rejected by the validator.

---

### Why `text` is rendered but markup is kept

The term filter downstream is an NER model reading prose. MinerU writes formulas character-spaced
(`P u r c h a s i n g ~ c o s t s`) and tables as HTML, and neither yields a single mention.
Measured on the same document and gold set, changing only the parse:

| `text` contains | Recall |
|---|---|
| raw markup | 0.7561 |
| plain text | 0.8537 |
| markup rendered back to prose | 0.8293 |

The markup is not discarded, because the formula branch needs exactly that form.

### `latex` is load-bearing — changing it weakens a guardrail silently

**As of 2026-08-26 this field carries weight it did not carry before.** Extraction's
formula guard validates a model's `formula_latex` claim against `Chunk.latex` —
**not** against `text`, which never contains markup. Until that landed, `latex`
was carried but unread; a parser change touching it would have broken nothing.
That is no longer true.

The failure mode is the dangerous kind. If this field is reflowed, "tidied", or
left unpopulated, the guard does not error — it quietly has less to check against,
and formula claims it can no longer verify are recorded as unverifiable rather
than caught. Fewer things fail; nothing looks wrong.

Two obligations follow, and they run **opposite** to the rule for `text`:

- **Do not clean it.** MinerU's character-spacing (`P r o d u c t i o n`), its
  `\mathrm{}` wrappers and its `~` spacing are all expected — extraction
  canonicalises before comparing, so cosmetic differences are already tolerated.
  Pre-cleaning here removes information the canonicaliser was built to absorb.
- **Do not drop it.** An artifact with no `latex` leaves every formula claim
  unverifiable. That is handled gracefully, but it is a guard switched off.

What extraction tolerates is **notation**; what it will not tolerate is **structure**.
Presentational wrappers, `\cdot` versus `\times`, and brace style are all erased
before comparison. A missing `\frac` is not — that is a different formula, and
catching it is the point of the guard.

**Worked evidence.** The `pipeline` backend writes multiplication as a bare letter
`x`, with no `\times` anywhere. Extraction cannot then tell a multiplication sign
from a variable, so every product formula from such an artifact comes back
`latex_verification: "unverified_operator"` — correctly, since the source genuinely
cannot express the operator. This is a real state, not a hypothetical: every
artifact produced without a GPU is a `pipeline` artifact. It is one more reason the
production backend is `hybrid/high` (see `KNOWLEDGE_PIPELINE_CALIBRATION.md` §13).

**Cheap health check, no paid run needed.** `latex_verification` is set during
validation, not by a model call, so counting `verified` against
`unverified_operator` in `formulas.json` measures directly whether an artifact's
markup is rich enough to support the guard.

### What decides a chunk boundary — **not the page**

A page break is a printing artifact, not a meaning boundary. Cutting there would sever a definition
that happens to straddle one, and the evidence ranker downstream reasons about sections. So chunking
follows the document's own structure, in this priority:

| | Opens a new chunk | |
|---|---|---|
| 1 | **A numbered section heading** | the primary boundary |
| 2 | **A table or a chart** | one MinerU item → one chunk |
| 3 | **6 000 characters (~1 500 tokens)** | safety net **only** for documents where no heading is detected; without it one chunk could swallow a whole document, blunting evidence ranking and inflating the summary branch's token share. Splits at an item boundary, so `text` stays verbatim |

Measured on the reference run — the spread is content-driven, not fixed:

| | |
|---|---|
| Chunks per page | **1 to 7** (page 0 → 1, page 5 → 7) |
| Chunks spanning >1 page | 0 of 31 here, though `page_idxs` is a list precisely because it can happen |
| `text` length | median **48 words**, max 444, one empty (the caption-less chart) |

Rule 3 therefore almost never fires on a well-structured standard — the headings do the work. It
matters for the mid-chapter textbook pages, which carry no headings at all.

---

## 3 · Reference run — what 9 pages actually produced

| | |
|---|---|
| MinerU items | 118 — 90 text, 14 table, 11 equation, 3 chart |
| Chunks emitted | **31** — 15 `text`, 13 `table`, 3 `chart`, **0 `equation`** |
| Images | 27 unique (**3.0 per page**) |
| Quality warnings | 3 |
| Parse time | 385.5 s (**42.8 s/page**, CPU) |

> The chunk count differs from the `13` quoted in the extraction contract's reference run
> (`20260813-145132`). That run predates the section-aware chunker built 2026-08-19; **31 is
> current.** Anything calibrated against 13 chunks should be re-checked.

### Quality warnings — `manifest.json`

Warnings never fail a document; they are recorded so quality cannot drop silently across hundreds of
documents. The reference run's three:

| `jenis` | Page | Detail |
|---|---|---|
| `latex_tanpa_angka` | 1 | `P r o d u c t i o n = M O H H x Q t y x P A x U A x P t y` — symbolic, so no digits |
| `tabel_kosong` | 6 | 1 of 14 tables detected but empty |
| `chart_tanpa_konteks` | 7 | Chart with neither content nor caption |

The check that matters most — **every number in a formula cross-checked against the source PDF's own
text layer** — returned clean. It exists because a CPU-pipeline parse once wrote `5600` as `55600`
with no error and a perfectly valid-looking formula.

---

### `generated_description` — why it is not in `text`

`text` is the haystack the extraction guardrail searches to prove a quote is verbatim. Generated
prose placed there would let a hallucinated sentence pass the very check built to catch
hallucination — silently, and with no code change to blame. A caption **printed in the document** is
verbatim and stays in `text`; a description **invented about the picture** does not.

Whether it is populated at all depends on the backend, read from MinerU's own source
(`is_image_analysis_option_visible`):

| Backend | Describes images |
|---|---|
| `pipeline` | never — `content` is always empty |
| `hybrid` + `effort=medium` | **disabled outright** |
| `hybrid` + `effort=high` | yes |
| `vlm` | yes, always |

So an artifact from `pipeline` carries `null` here throughout. That is the backend's behaviour, not
a parsing failure — and one more reason `parser_backend` has to travel with the data.

---

## Known gaps — do not design around these without asking

**`chunk_id` separator disagrees with the extraction contract.** This artifact emits
`STD_2026_006_MNO::0003`. `KNOWLEDGE_OUTPUT_CONTRACT.md` shows `STD_2026_006_MNO#0007`. Provenance
back-references break if a consumer parses on the wrong separator. **Needs one call, then both docs
updated.**

**`kind="equation"` is declared but never emitted.** All 11 equations arrive as `latex` entries on
`text` chunks with `has_formula=true`. Either the enum member goes, or the chunker should emit them —
but consumers must not branch on `kind == "equation"` today.

**`heading_path` carries page-number pollution.** On the reference run its root is frequently
`"1. 7"` — page 7's number caught as a numbered heading — appearing in 7 of 14 distinct paths. And
`"2. PENJELASAN PARAMETER"` never appears as a root although it should. `heading_path` feeds the
evidence ranker, so this is not cosmetic. Open against B5.

**`chapters` is empty on every chunk** of the reference run. The field is populated only when MinerU
reports running headers; this document has none detected.

---

## Caveat on the sample values

Field shapes, types and defaults are read from `src/knowledge_parsing/contracts.py`. Unlike the
extraction contract, **the values here are literal** — taken from
`data/knowledge_runs/20260824-100720/STD_2026_006_MNO/chunks.json`, with long strings truncated and
marked `…`.