Blame for docs/knowledge/KNOWLEDGE_PARSING_OUTPUT_CONTRACT.md

# Knowledge Pipeline — Parsing Output Contract

<!-- doc-status -->
> 🔄 **LIVE — the seam.** Output contract of the parsing half, consumed by `src/knowledge_extraction/adapter.py`. Source of truth is `src/knowledge_parsing/contracts.py`; this doc describes it.

**Source of truth:** `src/knowledge_parsing/contracts.py`
**Producer:** `python -m src.knowledge_parsing.run --input data/knowledge_docs/`
**Reference run:** `20260824-100720` · BUMA `STD_2026_006_MNO` · 9 pages · MinerU `3.4.4`, backend `pipeline`
**Status:** offline artifacts only — no endpoint, no table. Values below are **literal from that run**, not illustrative.

Companion to `KNOWLEDGE_OUTPUT_CONTRACT.md` (extraction). This is the half **upstream** of it:
parsing emits `ParsedDocument`, extraction consumes it and never touches the parser.

---

## Transport reality — read this first

There is **no endpoint and no table yet.** The CLI writes one directory per run:

```
data/knowledge_runs/<run_id>/
├── manifest.json                    run record: config, timings, quality warnings
├── failures.jsonl                   documents that failed, with tracebacks
└── <doc_id>/
    ├── chunks.json                  ← the artifact: one ParsedDocument
    └── mineru-source.txt            pointer to the untouched MinerU output
```

**Unlike the extraction payloads, this artifact is not a bare array.** `chunks.json` is a single
self-describing object that carries its own provenance — parser name, version, backend, config
fingerprint, source hash. The envelope proposed at the end of the extraction contract already exists
here, and is worth reusing rather than inventing a second shape.

Three conventions hold:

- **`page_idx` is 0-based**, exactly as MinerU reports it, with no conversion anywhere. Converting to
  1-based is the UI's job, done once at display, so the artifact always matches the raw MinerU output
  kept beside it in the cache.
- **`text` is VERBATIM.** Extraction's span check locates quoted spans literally inside it. Reflowing
  or normalising it makes the lookup fail and the field goes silently null — which presents as a bad
  model, not as a parser bug.
- **Markup never goes in `text`.** Formulas and tables are rendered to readable prose for `text`, and
  preserved in their original form in `latex` / `table_html`.

---

## 1 · `chunks.json` → `ParsedDocument` *(single object)*

```json
{
  "doc_id": "STD_2026_006_MNO",
  "chunks": [ "… Chunk[] …" ],
  "source_path": "data\\knowledge_docs\\STD_2026_006_MNO.pdf",
  "content_hash": "f10807178b20531e86237a2640ff4125d259aa68e5e9d9f910c0c2365cd92833",
  "n_pages": 9,
  "parser_name": "mineru",
  "parser_version": "3.4.4",
  "parser_backend": "pipeline",
  "parser_config": "b367868551b9",
  "version": 1,
  "schema_version": "0.3.0",
  "created_at": "2026-08-24T10:07:21",
  "raw_output_dir": "…/data/knowledge_cache/parse/f10807178b20531e-b367868551b9-3.4.4"
}
```

| Field | Type | Req | Notes |
|---|---|---|---|
| `doc_id` | string | ✅ | Source filename stem |
| `chunks` | Chunk[] | ✅ | 31 on the reference run |
| `source_path` | string | ✅ | |
| `content_hash` | string | ✅ | sha256 of the **source file** — a changed document forces a new artifact |
| `n_pages` | int | ✅ | |
| `parser_name` | string | ✅ | Default `"mineru"`. The field exists so a Tesseract / Azure DI path is distinguishable |
| `parser_version` | string \| null | — | e.g. `"3.4.4"` |
| `parser_backend` | string \| null | — | `pipeline` · `vlm` · `hybrid`. **Read back from MinerU's `_middle.json`** — what actually ran, not what was requested |
| `parser_config` | string \| null | — | 12-char fingerprint of the settings that affect output |
| `version` | int | ✅ | Version of **this artifact for this document**; bumped on re-parse |
| `schema_version` | string | ✅ | Versions **the contract**. Currently `0.3.0` |
| `created_at` | string \| null | — | ISO 8601, seconds precision |
| `raw_output_dir` | string \| null | — | Untouched MinerU output, kept content-addressed and never copied |

> **Why provenance is mandatory rather than nice-to-have:** without
> `parser_version` / `parser_backend`, a MinerU upgrade and a prompt change are
> indistinguishable when extraction results shift.

---

## 2 · `Chunk`

```json
{
  "chunk_id": "STD_2026_006_MNO::0003",
  "doc_id": "STD_2026_006_MNO",
  "kind": "text",
  "text": "Adalah serangkaian kegiatan yang dilakukan untuk menciptakan nilai tambah terhadap sumber daya …",
  "page_idx": 1,
  "page_idxs": [1],
  "section_no": "2.1.1",
  "heading": "Production",
  "chapters": [],
  "heading_path": ["2.1.1. Production"],
  "has_formula": true,
  "is_tabular": false,
  "source_items": [6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16],
  "bbox": [122, 293, 275, 307],
  "images": ["images/76503982720c45fc9d387accb74d5f5c5083f4f3109ed7fbf6a4da345bc20e0f.jpg"],
  "latex": ["$$\nP r o d u c t i o n = M O H H x Q t y x P A x U A x P t y\n$$"],
  "table_html": null
}
```

| Field | Type | Req | Notes |
|---|---|---|---|
| `chunk_id` | string | ✅ | **Format `<doc_id>::<seq>`**, `seq` zero-padded to 4. Note the `::` separator — see *Known gaps* |
| `doc_id` | string | ✅ | |
| `kind` | enum | ✅ | `text` · `table` · `chart` · `equation` · `image`. **Backends disagree on the label** — a figure `pipeline` calls a chart, `vlm` calls an image. Branch on both, never on one |
| `text` | string | ✅ | **VERBATIM.** Excludes the section heading line — that lives in `heading`. Consumers needing both compose them |
| `page_idx` | int | ✅ | **0-based.** First page of the chunk |
| `page_idxs` | int[] | ✅ | Every page the chunk spans. Default `[]` |
| `section_no` | string \| null | — | e.g. `"2.1.3"`; null when the document is unnumbered |
| `heading` | string \| null | — | This chunk's own section title, **verbatim as written** — `"Physical of Availability (PA)"`, not the tidied `"Physical Availability"` |
| `chapters` | string[] | ✅ | Running header per page spanned. Default `[]` |
| `heading_path` | string[] | ✅ | Breadcrumb of enclosing headings, outermost first |
| `has_formula` | bool | ✅ | Cheap downstream filter |
| `is_tabular` | bool | ✅ | |
| `source_items` | int[] | ✅ | Indices into MinerU's `content_list.json` — the trace back |
| `bbox` | int[] \| null | — | `[x0, y0, x1, y1]` of the first source item. Carried for a curation UI; **nothing in the pipeline reasons about it** — ordering uses item sequence, never coordinates |
| `images` | string[] | ✅ | Paths relative to `raw_output_dir`. Formula images, table/chart crops |
| `latex` | string[] | ✅ | Source markup for formulas, kept beside the rendered prose. ⚠️ **LOAD-BEARING** — extraction's formula guard validates against this field. Reflowing it or leaving it empty weakens the check with no error. See below |
| `table_html` | string \| null | — | Source markup for tables |
| `generated_description` | string \| null | — | ⚠️ **MODEL-WRITTEN, NOT FROM THE DOCUMENT.** MinerU's description of an image or chart. **Never span-checkable** — treat as a model claim about a picture, never as evidence. See below |

### Why `text` is rendered but markup is kept

The term filter downstream is an NER model reading prose. MinerU writes formulas character-spaced
(`P u r c h a s i n g ~ c o s t s`) and tables as HTML, and neither yields a single mention.
Measured on the same document and gold set, changing only the parse:

| `text` contains | Recall |
|---|---|
| raw markup | 0.7561 |
| plain text | 0.8537 |
| markup rendered back to prose | 0.8293 |

The markup is not discarded, because the formula branch needs exactly that form.

### `latex` is load-bearing — changing it weakens a guardrail silently

**As of 2026-08-26 this field carries weight it did not carry before.** Extraction's
formula guard validates a model's `formula_latex` claim against `Chunk.latex` —
**not** against `text`, which never contains markup. Until that landed, `latex`
was carried but unread; a parser change touching it would have broken nothing.
That is no longer true.

The failure mode is the dangerous kind. If this field is reflowed, "tidied", or
left unpopulated, the guard does not error — it quietly has less to check against,
and formula claims it can no longer verify are recorded as unverifiable rather
than caught. Fewer things fail; nothing looks wrong.

Two obligations follow, and they run **opposite** to the rule for `text`:

- **Do not clean it.** MinerU's character-spacing (`P r o d u c t i o n`), its
  `\mathrm{}` wrappers and its `~` spacing are all expected — extraction
  canonicalises before comparing, so cosmetic differences are already tolerated.
  Pre-cleaning here removes information the canonicaliser was built to absorb.
- **Do not drop it.** An artifact with no `latex` leaves every formula claim
  unverifiable. That is handled gracefully, but it is a guard switched off.

What extraction tolerates is **notation**; what it will not tolerate is **structure**.
Presentational wrappers, `\cdot` versus `\times`, and brace style are all erased
before comparison. A missing `\frac` is not — that is a different formula, and
catching it is the point of the guard.

**Worked evidence.** The `pipeline` backend writes multiplication as a bare letter
`x`, with no `\times` anywhere. Extraction cannot then tell a multiplication sign
from a variable, so every product formula from such an artifact comes back
`latex_verification: "unverified_operator"` — correctly, since the source genuinely
cannot express the operator. This is a real state, not a hypothetical: every
artifact produced without a GPU is a `pipeline` artifact. It is one more reason the
production backend is `hybrid/high` (see `KNOWLEDGE_PIPELINE_CALIBRATION.md` §13).

**Cheap health check, no paid run needed.** `latex_verification` is set during
validation, not by a model call, so counting `verified` against
`unverified_operator` in `formulas.json` measures directly whether an artifact's
markup is rich enough to support the guard.

### What decides a chunk boundary — **not the page**

A page break is a printing artifact, not a meaning boundary. Cutting there would sever a definition
that happens to straddle one, and the evidence ranker downstream reasons about sections. So chunking
follows the document's own structure, in this priority:

| | Opens a new chunk | |
|---|---|---|
| 1 | **A numbered section heading** | the primary boundary |
| 2 | **A table or a chart** | one MinerU item → one chunk |
| 3 | **6 000 characters (~1 500 tokens)** | safety net **only** for documents where no heading is detected; without it one chunk could swallow a whole document, blunting evidence ranking and inflating the summary branch's token share. Splits at an item boundary, so `text` stays verbatim |

Measured on the reference run — the spread is content-driven, not fixed:

| | |
|---|---|
| Chunks per page | **1 to 7** (page 0 → 1, page 5 → 7) |
| Chunks spanning >1 page | 0 of 31 here, though `page_idxs` is a list precisely because it can happen |
| `text` length | median **48 words**, max 444, one empty (the caption-less chart) |

Rule 3 therefore almost never fires on a well-structured standard — the headings do the work. It
matters for the mid-chapter textbook pages, which carry no headings at all.

---

## 3 · Reference run — what 9 pages actually produced

| | |
|---|---|
| MinerU items | 118 — 90 text, 14 table, 11 equation, 3 chart |
| Chunks emitted | **31** — 15 `text`, 13 `table`, 3 `chart`, **0 `equation`** |
| Images | 27 unique (**3.0 per page**) |
| Quality warnings | 3 |
| Parse time | 385.5 s (**42.8 s/page**, CPU) |

> The chunk count differs from the `13` quoted in the extraction contract's reference run
> (`20260813-145132`). That run predates the section-aware chunker built 2026-08-19; **31 is
> current.** Anything calibrated against 13 chunks should be re-checked.

### Quality warnings — `manifest.json`

Warnings never fail a document; they are recorded so quality cannot drop silently across hundreds of
documents. The reference run's three:

| `jenis` | Page | Detail |
|---|---|---|
| `latex_tanpa_angka` | 1 | `P r o d u c t i o n = M O H H x Q t y x P A x U A x P t y` — symbolic, so no digits |
| `tabel_kosong` | 6 | 1 of 14 tables detected but empty |
| `chart_tanpa_konteks` | 7 | Chart with neither content nor caption |

The check that matters most — **every number in a formula cross-checked against the source PDF's own
text layer** — returned clean. It exists because a CPU-pipeline parse once wrote `5600` as `55600`
with no error and a perfectly valid-looking formula.

---

### `generated_description` — why it is not in `text`

`text` is the haystack the extraction guardrail searches to prove a quote is verbatim. Generated
prose placed there would let a hallucinated sentence pass the very check built to catch
hallucination — silently, and with no code change to blame. A caption **printed in the document** is
verbatim and stays in `text`; a description **invented about the picture** does not.

Whether it is populated at all depends on the backend, read from MinerU's own source
(`is_image_analysis_option_visible`):

| Backend | Describes images |
|---|---|
| `pipeline` | never — `content` is always empty |
| `hybrid` + `effort=medium` | **disabled outright** |
| `hybrid` + `effort=high` | yes |
| `vlm` | yes, always |

So an artifact from `pipeline` carries `null` here throughout. That is the backend's behaviour, not
a parsing failure — and one more reason `parser_backend` has to travel with the data.

---

## Known gaps — do not design around these without asking

**`chunk_id` separator disagrees with the extraction contract.** This artifact emits
`STD_2026_006_MNO::0003`. `KNOWLEDGE_OUTPUT_CONTRACT.md` shows `STD_2026_006_MNO#0007`. Provenance
back-references break if a consumer parses on the wrong separator. **Needs one call, then both docs
updated.**

**`kind="equation"` is declared but never emitted.** All 11 equations arrive as `latex` entries on
`text` chunks with `has_formula=true`. Either the enum member goes, or the chunker should emit them —
but consumers must not branch on `kind == "equation"` today.

**`heading_path` carries page-number pollution.** On the reference run its root is frequently
`"1. 7"` — page 7's number caught as a numbered heading — appearing in 7 of 14 distinct paths. And
`"2. PENJELASAN PARAMETER"` never appears as a root although it should. `heading_path` feeds the
evidence ranker, so this is not cosmetic. Open against B5.

**`chapters` is empty on every chunk** of the reference run. The field is populated only when MinerU
reports running headers; this document has none detected.

---

## Caveat on the sample values

Field shapes, types and defaults are read from `src/knowledge_parsing/contracts.py`. Unlike the
extraction contract, **the values here are literal** — taken from
`data/knowledge_runs/20260824-100720/STD_2026_006_MNO/chunks.json`, with long strings truncated and
marked `…`.