# Table branch — design proposal (Step 8)

<!-- doc-status -->
> 💡 **PROPOSAL — nothing is built from this.** Written 2026-09-11 against the open queue in
> `knowledge_pipeline_runs/SESSION_HANDOFF.md` §4.6. Needs Rifqi's sign-off before any code.
> Roll-up: [DEV_PLAN.md](../../DEV_PLAN.md) §0.8 · built state: [REPO_STATUS.md](../../REPO_STATUS.md) §14.

**Owner:** Rifqi · **Size:** weeks · **Depends on:** parsing 0.4.0 (shipped), genre classification
(shipped 2026-09-11, advisory) · **Blocks:** any useful result on diagnostic documents.

---

## 1 · The problem, stated as a number

The Komatsu HD785-7 troubleshooting manual is **74% tabular** — 59 tables across 80 chunks — and the
pipeline extracts **zero** knowledge entries from it. Two paid runs, v1 and v2, both returned **0
definitions** from 75 and 93 clusters respectively. That is not a tuning failure and no prompt change
reaches it: a shop manual does not define its terms, it **asserts relationships between them**, and
it does so in a matrix.

The genre classifier (`src/knowledge_extraction/genre.py`) now names this document `diagnostic` and
reports `well_served = False` before anything is spent. This proposal is what would make that answer
change.

## 2 · What the document actually contains

A troubleshooting table is a **symptom × cause matrix**. Each marked cell is one assertion:

| Symptom \ Cause | Fuel filter clogged | Injector worn | Air cleaner blocked |
|---|---|---|---|
| Engine does not start | ● | ● | |
| Black exhaust smoke | | ● | ● |
| Output drops | ● | | ● |

Three assertions come out of the first row alone, and **none of them is a definition**:

> *"Engine does not start" may be caused by "Fuel filter clogged".*

The unit of knowledge here is the **(row header, column header, cell mark) triple**, and it is
recoverable **deterministically** — the mark is in the parsed HTML, the headers are in the parsed
HTML, nothing needs to be inferred.

## 3 · The shape of the branch

**Deterministic extraction, LLM only to phrase.** This is the inversion of every existing branch, and
it is the whole point: the assertion's *existence* is read off the table, so the model cannot
fabricate a relationship that is not in the document — it can only phrase one that is.

```
table chunk ──► parse table_html ──► (row header, col header, cell mark) triples
                                              │
                                              ├── deterministic: the assertion EXISTS
                                              │
                                              ▼
                                    one LLM call per TABLE (not per cell)
                                              │
                                              ▼
                                    natural-language phrasing + typing
                                              │
                                              ▼
                                    span check against the cell's own headers
```

**Provenance points at the cell**, which is stronger than anything the pipeline produces today: the
span is the row header plus the column header, both verbatim document text, both locatable by the
existing `span_present`.

### 3.1 · New entity

```python
class RelationEntry(BaseModel):
    relation_id: str          # sha256 over (doc_id, table asset_id, row, col) — positional
                              # within a table, but the table's asset_id is content-addressed,
                              # so a re-parse of identical bytes re-mints identical ids.
    kind: Literal["symptom_cause", "component_action", "condition_effect"]
    subject: str              # verbatim row header
    object: str               # verbatim column header
    statement: str | None     # model-phrased; span-checked against subject + object + caption
    table_asset_id: str
    provenance: Provenance    # span = the two headers; page from the chunk
    term_ids: list[str]       # links into the glossary where headers match known terms
```

### 3.2 · Why one call per table, not per cell

Komatsu's 59 tables carry on the order of 10³ marked cells. One call per cell is a different cost
class entirely and would blow the pre-flight ceiling by itself. One call per table hands the model
the linearised table plus the extracted triples and asks it to phrase each one — 59 calls, which is
**less than the 93 the glossary branch currently spends to abstain 93 times.**

## 4 · What has to be decided before coding

| # | Question | Why it blocks |
|---|---|---|
| T1 | Is a relation a first-class **entry kind** (a fifth `kind` in `knowledge_entries`) or a new table? | Persistence contract + Harry's migration. A fifth kind is additive; a table is a DDL handoff |
| T2 | What does a reviewer **do** with 1,000 relations? | The review queue is frequency-sorted and finishable by construction. 1,000 relations is not finishable. Likely answer: review per TABLE, not per relation — one approve covers the matrix |
| T3 | Does a mark type carry meaning (`●` vs `○` vs `▲`)? | Komatsu uses more than one glyph. If they mean "primary/secondary cause" the triple is really a quad, and getting this wrong silently inverts assertions |
| T4 | Which `table_html` shapes does the parser actually emit — merged cells, multi-row headers? | Determines whether header resolution is 20 lines or a small parser. **Measure before designing** |
| T5 | Does this ship behind the genre router (`diagnostic` only) or for every document? | BUMA is 25% tabular and its tables are *data*, not relation matrices. Running this on them produces noise |

**T4 is the one to do first** and it is free: inspect the 59 Komatsu tables and count the shapes.

## 5 · Kill lines, agreed before the first paid run

Copying the discipline that made the clustering defect findable — a metric that moves the *wrong* way
is worse than no metric (E2), so these are stated as recall against a **human-read sample**:

| Metric | What it measures | Kill line |
|---|---|---|
| **T-E1** | Triples extracted ÷ triples a human reads off the same 5 tables | ≥ 0.90 — it is deterministic; anything less is a parser bug |
| **T-E2** | Phrasings whose span check passes | ≥ 0.95 |
| **T-E3** | Phrasings a human judges to *invert or distort* the relation | 0 — this is the one that matters, and it is the reason the mark glyph question (T3) is not cosmetic |
| **T-cost** | Calls per document | ≤ 1 per table |

**T-E3 has no automated proxy.** It needs Mas Beta or an equivalent reader on a sample, which makes
the R4 conversation (does "found but undefined" help an expert?) worth having in the same sitting.

## 6 · What this does NOT solve

A relation matrix is not a glossary. After this branch ships, the Komatsu manual still yields **zero
definitions** — correctly, because it contains none. If the product's promise is "a glossary from
your documents", a diagnostic document will always disappoint it, and the honest place to say so is
the ingest response, which the genre classifier now makes possible.

---

## 7 · Groundwork already landed (2026-09-11)

Two pieces of this branch's plumbing exist before the branch does, and both are measured:

**The pointer is populated and durable.** 0.4.0 gives every table chunk an `Asset` whose id comes
from `synthetic_asset_id(doc_id, chunk_id, item_index)` — stable document coordinates, so it survives
a re-parse — plus `referenced_by` naming the prose chunk that introduces it, and the reverse pointer
on that prose chunk's `assets`.

**One linked partner now joins the evidence window** (`EVIDENCE_LINK_PARTNERS = 1`, in `rank/evidence.py`).
The measurement that motivated it: across 603 clusters on the three test documents, **zero** had both
sides of a linked pair in their top-5, so a table reached the model without the prose naming it, and
vice versa. 31/83, 55/93 and 70/427 clusters now gain a partner, at +1.6%/+4.1%/+4.2% of prompt spend.

**What this does *not* do, and the branch must:** `table_html` crosses the seam into
`Chunk.table_html` and **nothing reads it** — the prompt shows the linearised `render_table` text
from `Chunk.text` instead. That is deliberate: the span check validates against `heading + text`
only, so putting HTML in front of the model would let it quote markup that is then nulled as
unlocatable. A branch that wants the real cell structure has to extend the span-checked text too, or
carry provenance that points at a cell rather than a character span (§3.1's `cell_ref` does the
latter, which is the cleaner of the two).


---

## 8 · Correction and re-diagnosis of the Komatsu tables (2026-09-11)

> **Correction.** An earlier reading of this corpus (same day, recorded in the session
> notes) said the troubleshooting charts had **lost their column headers**, on the
> evidence that only 4 of 59 tabular chunks contain a `<th>`. That count is accurate
> and the conclusion drawn from it was **wrong**. The headers are present; they are
> emitted as ordinary `<td>` cells in a **separate chunk**.

### What Mistral actually produces

Each chart is split into three or four sibling tables, and the first cell of each names
the fragment. Across the 59 tabular chunks:

| first cell | count | what it holds |
|---|---|---|
| `Causes` | 17 | **the column headers** — the cause list |
| `Troubleshooting` | 16 | check rows |
| `Questions` | 15 | interview/symptom rows |
| `Remedy` | 10 | the action row |

So the (row header, column header, cell mark) triple is recoverable, but its three parts
live in **different chunks**. The branch's first job is re-assembly, not OCR recovery —
the fragments are adjacent, same-page, and regularly labelled.

### The defect that is real

**Marks survive in only 20 of 59 tabular chunks.** Doc-wide there are 278 marks
(18 ◎, 137 ○, 25 △, 98 ●), and they cluster on the pages where the matrix fragment
kept them; on pages 9-10 the `Questions` and `Troubleshooting` fragments come through
with row labels and **zero marks**. Column structure and `rowspan`/`colspan` are
preserved where the marks are.

A chart whose marks were dropped carries no symptom-to-cause link at all, so roughly
two thirds of the document's assertions are currently unrecoverable from the artifact.
**That, not the headers, is what gates this branch**, and it has to be fixed at the
parsing layer.

### Free evidence that the content is reachable

Adding a `fault code` label to the span filter (see `config/labels.yaml`) surfaced
**66 cause phrases** as clusters on this document — *"Clogged fuel filter and/or
strainer"*, *"Improper injection timing"*, *"Defective alternator (regulator section)"* —
plus 17 service codes. Those are the column headers, read out of the `Causes`
fragments. The raw material for the branch is already in the artifact and already
findable; what is missing is the mark matrix that connects it to the symptoms.