[NOTICKET] feat(knowledge-parsing): carry image descriptions, apart from the text

MinerU describes images and charts, but only on some backends. Read from its
own source (is_image_analysis_option_visible): never on `pipeline`, always on
`vlm`, and on `hybrid` only at effort=high — medium disables it outright.

Two faults kept that description from ever arriving.

`normalize.py` had no branch for `image` items at all. Different backends label
the same picture differently: what `pipeline` calls a chart, `vlm` calls an
image. Image items fell through to the generic text path, which reads only
item["text"] — a field pictures do not have — so caption AND description were
dropped in silence. That is most of what looked like "vlm loses chart
detection" in yesterday's comparison: vlm found them, we discarded them.

The second is worse and was waiting to fire. `_teks_chart` appended `content`
straight into Chunk.text. Harmless today because `pipeline` always leaves it
empty — but the moment the backend flips to vlm, model-written prose starts
flowing into the field the extraction guardrail searches to prove a quote is
verbatim. A hallucinated sentence would pass the very check built to catch
hallucination, silently, with no code change to blame.

So generated text now has its own field, `Chunk.generated_description`, and
never enters `text`. A caption printed in the document is verbatim and stays;
a description invented about the picture does not. Consumers must treat it as
a model claim about an image — it cannot be span-checked against anything.

Contract moves to schema 0.3.0: `image` joins ChunkKind, and the new field is
additive. Also adds `--effort` to the CLI, without which hybrid could only ever
run at its default.

Verified on `pipeline`: 31 chunks unchanged, generated_description null
throughout, chart captions still in text — exactly as it should be for a
backend that never describes anything.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

src/knowledge_parsing/contracts.py

@@ -41,9 +41,9 @@ from typing import Literal
 
 from pydantic import BaseModel, Field
 
-ChunkKind = Literal["text", "table", "chart", "equation"]
+ChunkKind = Literal["text", "table", "chart", "equation", "image"]
 
-SCHEMA_VERSION = "0.2.0"
+SCHEMA_VERSION = "0.3.0"
 
 
 class Chunk(BaseModel):
@@ -121,6 +121,21 @@ class Chunk(BaseModel):
     latex: list[str] = Field(default_factory=list)
     table_html: str | None = None
 
+    # ⚠️ MODEL-WRITTEN, NOT FROM THE DOCUMENT. MinerU's `vlm` backend (and
+    # `hybrid` at effort=high) describes images and charts; that description
+    # arrives in the item's `content` field.
+    #
+    # It is kept OUT of `text` deliberately. `text` is the haystack the
+    # extraction guardrail searches to prove a quote is verbatim; putting
+    # generated prose there would let a hallucinated sentence pass the very
+    # check that exists to catch hallucination. A caption printed in the
+    # document is verbatim and belongs in `text`; a description invented about
+    # the picture does not.
+    #
+    # Any consumer using this must treat it as a model claim about the image,
+    # never as evidence — it cannot be span-checked against anything.
+    generated_description: str | None = None
+
 
 class ParsedDocument(BaseModel):
     """The artifact itself — one parsed document, self-describing.

src/knowledge_parsing/normalize.py

@@ -105,12 +105,22 @@ def _teks_tabel(item: dict[str, Any], kosakata: frozenset[str] | None = None) ->
     return "\n\n".join(b for b in bagian if b)
 
 
-def _teks_chart(item: dict[str, Any]) -> str:
-    bagian = list(item.get("chart_caption") or [])
-    if (item.get("content") or "").strip():
-        bagian.append(item["content"])
-    bagian += list(item.get("chart_footnote") or [])
-    return "\n\n".join(b for b in bagian if b)
+def _teks_gambar(item: dict[str, Any], jenis: str) -> tuple[str, str | None]:
+    """(teks verbatim, deskripsi karangan model) untuk satu item chart/image.
+
+    `content` SENGAJA dipisahkan dari teks. Di backend `pipeline` field itu
+    selalu kosong, jadi selama ini tidak kelihatan — tapi `vlm` dan
+    `hybrid --effort high` mengisinya dengan deskripsi hasil analisis gambar.
+
+    Menaruhnya di `text` berarti kalimat karangan model ikut jadi jerami yang
+    ditelusuri span-check extraction, sehingga halusinasi lolos lewat
+    pemeriksaan yang justru dibuat untuk menangkapnya. Caption yang benar-benar
+    tercetak di dokumen tetap verbatim, dan tetap masuk `text`.
+    """
+    bagian = list(item.get(f"{jenis}_caption") or [])
+    bagian += list(item.get(f"{jenis}_footnote") or [])
+    deskripsi = (item.get("content") or "").strip() or None
+    return "\n\n".join(b for b in bagian if b), deskripsi
 
 
 def normalisasi(
@@ -190,10 +200,15 @@ def normalisasi(
                 chunks.append(c)
             continue
 
-        if tipe == "chart":
+        # `image` berdampingan dengan `chart`: backend berbeda memberi label
+        # berbeda untuk gambar yang sama — `pipeline` menyebut chart apa yang
+        # `vlm` sebut image. Tanpa cabang ini item `image` jatuh ke jalur teks
+        # umum, yang cuma membaca `item["text"]`, dan gambar tidak punya field
+        # itu — jadi caption DAN deskripsinya hilang tanpa suara.
+        if tipe in {"chart", "image"}:
             tutup()
-            c = buka("chart", page)
-            c.text = _teks_chart(item)
+            c = buka(tipe, page)
+            c.text, c.generated_description = _teks_gambar(item, tipe)
             c.source_items = [i]
             c.bbox = item.get("bbox")
             if item.get("img_path"):

src/knowledge_parsing/run.py

@@ -164,6 +164,10 @@ def main() -> None:
     p.add_argument("--backend", choices=["pipeline", "vlm", "hybrid"],
                    help="pipeline=CPU, vlm=butuh GPU (target resmi)")
     p.add_argument("--lang", default=None)
+    # Hanya berlaku untuk `hybrid`. Bukan sekadar cepat vs teliti: MinerU
+    # MEMATIKAN PAKSA analisis gambar di `medium` (`is_image_analysis_option_visible`),
+    # jadi deskripsi gambar cuma ada di `high` — dan selalu ada di `vlm`.
+    p.add_argument("--effort", choices=["medium", "high"], default=None)
     p.add_argument("--no-resume", action="store_true", help="proses ulang semua")
     p.add_argument("--start-page", type=int, default=None)
     p.add_argument("--end-page", type=int, default=None)
@@ -176,6 +180,8 @@ def main() -> None:
         cfg.backend = a.backend
     if a.lang:
         cfg.lang = a.lang
+    if a.effort:
+        cfg.effort = a.effort
     if a.no_resume:
         cfg.resume = False
     if a.start_page is not None: