docs/knowledge/KNOWLEDGE_PARSING_OUTPUT_CONTRACT.md

Knowledge Pipeline — Parsing Output Contract

🔄 LIVE — the seam. Output contract of the parsing half, consumed by src/knowledge_extraction/adapter.py. Source of truth is src/knowledge_parsing/contracts.py; this doc describes it.

Source of truth: src/knowledge_parsing/contracts.py Producer: python -m src.knowledge_parsing.run --input data/knowledge_docs/ Reference run: 20260824-100720 · BUMA STD_2026_006_MNO · 9 pages · MinerU 3.4.4, backend pipeline Status: offline artifacts only — no endpoint, no table. Values below are literal from that run, not illustrative.

Companion to KNOWLEDGE_OUTPUT_CONTRACT.md (extraction). This is the half upstream of it: parsing emits ParsedDocument, extraction consumes it and never touches the parser.

---

Transport reality — read this first

There is no endpoint and no table yet. The CLI writes one directory per run:

data/knowledge_runs/<run_id>/
├── manifest.json                    run record: config, timings, quality warnings
├── failures.jsonl                   documents that failed, with tracebacks
└── <doc_id>/
    ├── chunks.json                  ← the artifact: one ParsedDocument
    └── mineru-source.txt            pointer to the untouched MinerU output

Unlike the extraction payloads, this artifact is not a bare array. chunks.json is a single self-describing object that carries its own provenance — parser name, version, backend, config fingerprint, source hash. The envelope proposed at the end of the extraction contract already exists here, and is worth reusing rather than inventing a second shape.

Three conventions hold:

• page_idx is 0-based, exactly as MinerU reports it, with no conversion anywhere. Converting to 1-based is the UI's job, done once at display, so the artifact always matches the raw MinerU output kept beside it in the cache.
• text is VERBATIM. Extraction's span check locates quoted spans literally inside it. Reflowing or normalising it makes the lookup fail and the field goes silently null — which presents as a bad model, not as a parser bug.
• Markup never goes in text. Formulas and tables are rendered to readable prose for text, and preserved in their original form in latex / table_html.

---

1 · chunks.json → ParsedDocument (single object)

{
  "doc_id": "STD_2026_006_MNO",
  "chunks": [ "… Chunk[] …" ],
  "source_path": "data\\knowledge_docs\\STD_2026_006_MNO.pdf",
  "content_hash": "f10807178b20531e86237a2640ff4125d259aa68e5e9d9f910c0c2365cd92833",
  "n_pages": 9,
  "parser_name": "mineru",
  "parser_version": "3.4.4",
  "parser_backend": "pipeline",
  "parser_config": "b367868551b9",
  "version": 1,
  "schema_version": "0.3.0",
  "created_at": "2026-08-24T10:07:21",
  "raw_output_dir": "…/data/knowledge_cache/parse/f10807178b20531e-b367868551b9-3.4.4"
}

Field	Type	Req	Notes
doc_id	string	✅	Source filename stem
chunks	Chunk[]	✅	31 on the reference run
source_path	string	✅
content_hash	string	✅	sha256 of the source file — a changed document forces a new artifact
n_pages	int	✅
parser_name	string	✅	Default "mineru". The field exists so a Tesseract / Azure DI path is distinguishable
parser_version	string | null	—	e.g. "3.4.4"
parser_backend	string | null	—	pipeline · vlm · hybrid. Read back from MinerU's _middle.json — what actually ran, not what was requested
parser_config	string | null	—	12-char fingerprint of the settings that affect output
version	int	✅	Version of this artifact for this document; bumped on re-parse
schema_version	string	✅	Versions the contract. Currently 0.3.0
created_at	string | null	—	ISO 8601, seconds precision
raw_output_dir	string | null	—	Untouched MinerU output, kept content-addressed and never copied

Why provenance is mandatory rather than nice-to-have: without parser_version / parser_backend, a MinerU upgrade and a prompt change are indistinguishable when extraction results shift.

---

2 · Chunk

{
  "chunk_id": "STD_2026_006_MNO::0003",
  "doc_id": "STD_2026_006_MNO",
  "kind": "text",
  "text": "Adalah serangkaian kegiatan yang dilakukan untuk menciptakan nilai tambah terhadap sumber daya …",
  "page_idx": 1,
  "page_idxs": [1],
  "section_no": "2.1.1",
  "heading": "Production",
  "chapters": [],
  "heading_path": ["2.1.1. Production"],
  "has_formula": true,
  "is_tabular": false,
  "source_items": [6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16],
  "bbox": [122, 293, 275, 307],
  "images": ["images/76503982720c45fc9d387accb74d5f5c5083f4f3109ed7fbf6a4da345bc20e0f.jpg"],
  "latex": ["$$\nP r o d u c t i o n = M O H H x Q t y x P A x U A x P t y\n$$"],
  "table_html": null
}

Field	Type	Req	Notes
chunk_id	string	✅	Format <doc_id>::<seq>, seq zero-padded to 4. Note the :: separator — see Known gaps
doc_id	string	✅
kind	enum	✅	text · table · chart · equation · image. Backends disagree on the label — a figure pipeline calls a chart, vlm calls an image. Branch on both, never on one
text	string	✅	VERBATIM. Excludes the section heading line — that lives in heading. Consumers needing both compose them
page_idx	int	✅	0-based. First page of the chunk
page_idxs	int[]	✅	Every page the chunk spans. Default []
section_no	string | null	—	e.g. "2.1.3"; null when the document is unnumbered
heading	string | null	—	This chunk's own section title, verbatim as written — "Physical of Availability (PA)", not the tidied "Physical Availability"
chapters	string[]	✅	Running header per page spanned. Default []
heading_path	string[]	✅	Breadcrumb of enclosing headings, outermost first
has_formula	bool	✅	Cheap downstream filter
is_tabular	bool	✅
source_items	int[]	✅	Indices into MinerU's content_list.json — the trace back
bbox	int[] | null	—	[x0, y0, x1, y1] of the first source item. Carried for a curation UI; nothing in the pipeline reasons about it — ordering uses item sequence, never coordinates
images	string[]	✅	Paths relative to raw_output_dir. Formula images, table/chart crops
latex	string[]	✅	Source markup for formulas, kept beside the rendered prose. ⚠️ LOAD-BEARING — extraction's formula guard validates against this field. Reflowing it or leaving it empty weakens the check with no error. See below
table_html	string | null	—	Source markup for tables
generated_description	string | null	—	⚠️ MODEL-WRITTEN, NOT FROM THE DOCUMENT. MinerU's description of an image or chart. Never span-checkable — treat as a model claim about a picture, never as evidence. See below

Why text is rendered but markup is kept

The term filter downstream is an NER model reading prose. MinerU writes formulas character-spaced (P u r c h a s i n g ~ c o s t s) and tables as HTML, and neither yields a single mention. Measured on the same document and gold set, changing only the parse:

text contains	Recall
raw markup	0.7561
plain text	0.8537
markup rendered back to prose	0.8293

The markup is not discarded, because the formula branch needs exactly that form.

latex is load-bearing — changing it weakens a guardrail silently

As of 2026-08-26 this field carries weight it did not carry before. Extraction's formula guard validates a model's formula_latex claim against Chunk.latex — not against text, which never contains markup. Until that landed, latex was carried but unread; a parser change touching it would have broken nothing. That is no longer true.

The failure mode is the dangerous kind. If this field is reflowed, "tidied", or left unpopulated, the guard does not error — it quietly has less to check against, and formula claims it can no longer verify are recorded as unverifiable rather than caught. Fewer things fail; nothing looks wrong.

Two obligations follow, and they run opposite to the rule for text:

• Do not clean it. MinerU's character-spacing (P r o d u c t i o n), its \mathrm{} wrappers and its ~ spacing are all expected — extraction canonicalises before comparing, so cosmetic differences are already tolerated. Pre-cleaning here removes information the canonicaliser was built to absorb.
• Do not drop it. An artifact with no latex leaves every formula claim unverifiable. That is handled gracefully, but it is a guard switched off.

What extraction tolerates is notation; what it will not tolerate is structure. Presentational wrappers, \cdot versus \times, and brace style are all erased before comparison. A missing \frac is not — that is a different formula, and catching it is the point of the guard.

Worked evidence. The pipeline backend writes multiplication as a bare letter x, with no \times anywhere. Extraction cannot then tell a multiplication sign from a variable, so every product formula from such an artifact comes back latex_verification: "unverified_operator" — correctly, since the source genuinely cannot express the operator. This is a real state, not a hypothetical: every artifact produced without a GPU is a pipeline artifact. It is one more reason the production backend is hybrid/high (see KNOWLEDGE_PIPELINE_CALIBRATION.md §13).

Cheap health check, no paid run needed. latex_verification is set during validation, not by a model call, so counting verified against unverified_operator in formulas.json measures directly whether an artifact's markup is rich enough to support the guard.

What decides a chunk boundary — not the page

A page break is a printing artifact, not a meaning boundary. Cutting there would sever a definition that happens to straddle one, and the evidence ranker downstream reasons about sections. So chunking follows the document's own structure, in this priority:

	Opens a new chunk
1	A numbered section heading	the primary boundary
2	A table or a chart	one MinerU item → one chunk
3	6 000 characters (~1 500 tokens)	safety net only for documents where no heading is detected; without it one chunk could swallow a whole document, blunting evidence ranking and inflating the summary branch's token share. Splits at an item boundary, so text stays verbatim

Measured on the reference run — the spread is content-driven, not fixed:

Chunks per page	1 to 7 (page 0 → 1, page 5 → 7)
Chunks spanning >1 page	0 of 31 here, though page_idxs is a list precisely because it can happen
text length	median 48 words, max 444, one empty (the caption-less chart)

Rule 3 therefore almost never fires on a well-structured standard — the headings do the work. It matters for the mid-chapter textbook pages, which carry no headings at all.

---

3 · Reference run — what 9 pages actually produced

MinerU items	118 — 90 text, 14 table, 11 equation, 3 chart
Chunks emitted	31 — 15 text, 13 table, 3 chart, 0 equation
Images	27 unique (3.0 per page)
Quality warnings	3
Parse time	385.5 s (42.8 s/page, CPU)

The chunk count differs from the 13 quoted in the extraction contract's reference run (20260813-145132). That run predates the section-aware chunker built 2026-08-19; 31 is current. Anything calibrated against 13 chunks should be re-checked.

Quality warnings — manifest.json

Warnings never fail a document; they are recorded so quality cannot drop silently across hundreds of documents. The reference run's three:

jenis	Page	Detail
latex_tanpa_angka	1	P r o d u c t i o n = M O H H x Q t y x P A x U A x P t y — symbolic, so no digits
tabel_kosong	6	1 of 14 tables detected but empty
chart_tanpa_konteks	7	Chart with neither content nor caption

The check that matters most — every number in a formula cross-checked against the source PDF's own text layer — returned clean. It exists because a CPU-pipeline parse once wrote 5600 as 55600 with no error and a perfectly valid-looking formula.

---

generated_description — why it is not in text

text is the haystack the extraction guardrail searches to prove a quote is verbatim. Generated prose placed there would let a hallucinated sentence pass the very check built to catch hallucination — silently, and with no code change to blame. A caption printed in the document is verbatim and stays in text; a description invented about the picture does not.

Whether it is populated at all depends on the backend, read from MinerU's own source (is_image_analysis_option_visible):

Backend	Describes images
pipeline	never — content is always empty
hybrid + effort=medium	disabled outright
hybrid + effort=high	yes
vlm	yes, always

So an artifact from pipeline carries null here throughout. That is the backend's behaviour, not a parsing failure — and one more reason parser_backend has to travel with the data.

---

Known gaps — do not design around these without asking

chunk_id separator disagrees with the extraction contract. This artifact emits STD_2026_006_MNO::0003. KNOWLEDGE_OUTPUT_CONTRACT.md shows STD_2026_006_MNO#0007. Provenance back-references break if a consumer parses on the wrong separator. Needs one call, then both docs updated.

kind="equation" is declared but never emitted. All 11 equations arrive as latex entries on text chunks with has_formula=true. Either the enum member goes, or the chunker should emit them — but consumers must not branch on kind == "equation" today.

heading_path carries page-number pollution. On the reference run its root is frequently "1. 7" — page 7's number caught as a numbered heading — appearing in 7 of 14 distinct paths. And "2. PENJELASAN PARAMETER" never appears as a root although it should. heading_path feeds the evidence ranker, so this is not cosmetic. Open against B5.

chapters is empty on every chunk of the reference run. The field is populated only when MinerU reports running headers; this document has none detected.

---

Caveat on the sample values

Field shapes, types and defaults are read from src/knowledge_parsing/contracts.py. Unlike the extraction contract, the values here are literal — taken from data/knowledge_runs/20260824-100720/STD_2026_006_MNO/chunks.json, with long strings truncated and marked ….