docs/knowledge/TABLE_BRANCH_PROPOSAL.md

Table branch — design proposal (Step 8)

💡 PROPOSAL — nothing is built from this. Written 2026-09-11 against the open queue in knowledge_pipeline_runs/SESSION_HANDOFF.md §4.6. Needs Rifqi's sign-off before any code. Roll-up: DEV_PLAN.md §0.8 · built state: REPO_STATUS.md §14.

Owner: Rifqi · Size: weeks · Depends on: parsing 0.4.0 (shipped), genre classification (shipped 2026-09-11, advisory) · Blocks: any useful result on diagnostic documents.

---

1 · The problem, stated as a number

The Komatsu HD785-7 troubleshooting manual is 74% tabular — 59 tables across 80 chunks — and the pipeline extracts zero knowledge entries from it. Two paid runs, v1 and v2, both returned 0 definitions from 75 and 93 clusters respectively. That is not a tuning failure and no prompt change reaches it: a shop manual does not define its terms, it asserts relationships between them, and it does so in a matrix.

The genre classifier (src/knowledge_extraction/genre.py) now names this document diagnostic and reports well_served = False before anything is spent. This proposal is what would make that answer change.

2 · What the document actually contains

A troubleshooting table is a symptom × cause matrix. Each marked cell is one assertion:

Symptom \ Cause	Fuel filter clogged	Injector worn	Air cleaner blocked
Engine does not start	●	●
Black exhaust smoke		●	●
Output drops	●		●

Three assertions come out of the first row alone, and none of them is a definition:

"Engine does not start" may be caused by "Fuel filter clogged".

The unit of knowledge here is the (row header, column header, cell mark) triple, and it is recoverable deterministically — the mark is in the parsed HTML, the headers are in the parsed HTML, nothing needs to be inferred.

3 · The shape of the branch

Deterministic extraction, LLM only to phrase. This is the inversion of every existing branch, and it is the whole point: the assertion's existence is read off the table, so the model cannot fabricate a relationship that is not in the document — it can only phrase one that is.

table chunk ──► parse table_html ──► (row header, col header, cell mark) triples
                                              │
                                              ├── deterministic: the assertion EXISTS
                                              │
                                              ▼
                                    one LLM call per TABLE (not per cell)
                                              │
                                              ▼
                                    natural-language phrasing + typing
                                              │
                                              ▼
                                    span check against the cell's own headers

Provenance points at the cell, which is stronger than anything the pipeline produces today: the span is the row header plus the column header, both verbatim document text, both locatable by the existing span_present.

3.1 · New entity

class RelationEntry(BaseModel):
    relation_id: str          # sha256 over (doc_id, table asset_id, row, col) — positional
                              # within a table, but the table's asset_id is content-addressed,
                              # so a re-parse of identical bytes re-mints identical ids.
    kind: Literal["symptom_cause", "component_action", "condition_effect"]
    subject: str              # verbatim row header
    object: str               # verbatim column header
    statement: str | None     # model-phrased; span-checked against subject + object + caption
    table_asset_id: str
    provenance: Provenance    # span = the two headers; page from the chunk
    term_ids: list[str]       # links into the glossary where headers match known terms

3.2 · Why one call per table, not per cell

Komatsu's 59 tables carry on the order of 10³ marked cells. One call per cell is a different cost class entirely and would blow the pre-flight ceiling by itself. One call per table hands the model the linearised table plus the extracted triples and asks it to phrase each one — 59 calls, which is less than the 93 the glossary branch currently spends to abstain 93 times.

4 · What has to be decided before coding

#	Question	Why it blocks
T1	Is a relation a first-class entry kind (a fifth kind in knowledge_entries) or a new table?	Persistence contract + Harry's migration. A fifth kind is additive; a table is a DDL handoff
T2	What does a reviewer do with 1,000 relations?	The review queue is frequency-sorted and finishable by construction. 1,000 relations is not finishable. Likely answer: review per TABLE, not per relation — one approve covers the matrix
T3	Does a mark type carry meaning (● vs ○ vs ▲)?	Komatsu uses more than one glyph. If they mean "primary/secondary cause" the triple is really a quad, and getting this wrong silently inverts assertions
T4	Which table_html shapes does the parser actually emit — merged cells, multi-row headers?	Determines whether header resolution is 20 lines or a small parser. Measure before designing
T5	Does this ship behind the genre router (diagnostic only) or for every document?	BUMA is 25% tabular and its tables are data, not relation matrices. Running this on them produces noise

T4 is the one to do first and it is free: inspect the 59 Komatsu tables and count the shapes.

5 · Kill lines, agreed before the first paid run

Copying the discipline that made the clustering defect findable — a metric that moves the wrong way is worse than no metric (E2), so these are stated as recall against a human-read sample:

Metric	What it measures	Kill line
T-E1	Triples extracted ÷ triples a human reads off the same 5 tables	≥ 0.90 — it is deterministic; anything less is a parser bug
T-E2	Phrasings whose span check passes	≥ 0.95
T-E3	Phrasings a human judges to invert or distort the relation	0 — this is the one that matters, and it is the reason the mark glyph question (T3) is not cosmetic
T-cost	Calls per document	≤ 1 per table

T-E3 has no automated proxy. It needs Mas Beta or an equivalent reader on a sample, which makes the R4 conversation (does "found but undefined" help an expert?) worth having in the same sitting.

6 · What this does NOT solve

A relation matrix is not a glossary. After this branch ships, the Komatsu manual still yields zero definitions — correctly, because it contains none. If the product's promise is "a glossary from your documents", a diagnostic document will always disappoint it, and the honest place to say so is the ingest response, which the genre classifier now makes possible.

---

7 · Groundwork already landed (2026-09-11)

Two pieces of this branch's plumbing exist before the branch does, and both are measured:

The pointer is populated and durable. 0.4.0 gives every table chunk an Asset whose id comes from synthetic_asset_id(doc_id, chunk_id, item_index) — stable document coordinates, so it survives a re-parse — plus referenced_by naming the prose chunk that introduces it, and the reverse pointer on that prose chunk's assets.

One linked partner now joins the evidence window (EVIDENCE_LINK_PARTNERS = 1, in rank/evidence.py). The measurement that motivated it: across 603 clusters on the three test documents, zero had both sides of a linked pair in their top-5, so a table reached the model without the prose naming it, and vice versa. 31/83, 55/93 and 70/427 clusters now gain a partner, at +1.6%/+4.1%/+4.2% of prompt spend.

What this does not do, and the branch must: table_html crosses the seam into Chunk.table_html and nothing reads it — the prompt shows the linearised render_table text from Chunk.text instead. That is deliberate: the span check validates against heading + text only, so putting HTML in front of the model would let it quote markup that is then nulled as unlocatable. A branch that wants the real cell structure has to extend the span-checked text too, or carry provenance that points at a cell rather than a character span (§3.1's cell_ref does the latter, which is the cleaner of the two).

---

8 · Correction and re-diagnosis of the Komatsu tables (2026-09-11)

Correction. An earlier reading of this corpus (same day, recorded in the session notes) said the troubleshooting charts had lost their column headers, on the evidence that only 4 of 59 tabular chunks contain a <th>. That count is accurate and the conclusion drawn from it was wrong. The headers are present; they are emitted as ordinary <td> cells in a separate chunk.

What Mistral actually produces

Each chart is split into three or four sibling tables, and the first cell of each names the fragment. Across the 59 tabular chunks:

first cell	count	what it holds
Causes	17	the column headers — the cause list
Troubleshooting	16	check rows
Questions	15	interview/symptom rows
Remedy	10	the action row

So the (row header, column header, cell mark) triple is recoverable, but its three parts live in different chunks. The branch's first job is re-assembly, not OCR recovery — the fragments are adjacent, same-page, and regularly labelled.

The defect that is real

Marks survive in only 20 of 59 tabular chunks. Doc-wide there are 278 marks (18 ◎, 137 ○, 25 △, 98 ●), and they cluster on the pages where the matrix fragment kept them; on pages 9-10 the Questions and Troubleshooting fragments come through with row labels and zero marks. Column structure and rowspan/colspan are preserved where the marks are.

A chart whose marks were dropped carries no symptom-to-cause link at all, so roughly two thirds of the document's assertions are currently unrecoverable from the artifact. That, not the headers, is what gates this branch, and it has to be fixed at the parsing layer.

Free evidence that the content is reachable

Adding a fault code label to the span filter (see config/labels.yaml) surfaced 66 cause phrases as clusters on this document — "Clogged fuel filter and/or strainer", "Improper injection timing", "Defective alternator (regulator section)" — plus 17 service codes. Those are the column headers, read out of the Causes fragments. The raw material for the branch is already in the artifact and already findable; what is missing is the mark matrix that connects it to the symptoms.