src/knowledge_parsing/README.md

# knowledge_parsing

Document parsing — the first half of the knowledge pipeline. PDF/DOCX in, a
versioned `ParsedDocument` out.

The second half (`src/knowledge_extraction/`) consumes that artifact and never
the parser itself, which is what keeps MinerU swappable: a Tesseract or Azure
Document Intelligence path emits the same artifact and extraction never learns
which one ran.

Additive and flag-gated. The existing unstructured path (`src/knowledge/`,
Tesseract OCR → chunk → pgvector) is untouched and keeps running as-is.

> **Current (2026-09-02) — the default parser is now Mistral OCR.** `--backend mistral` (the config
> default) is an API with **no GPU**: `run --backend mistral` produces the artifact in ~8 s. The
> MinerU backends (`hybrid`/`vlm`/`pipeline`, everything below about GPU) are now the **fallback** —
> both emit the identical `ParsedDocument`. Mistral needs `MISTRAL_API_KEY` (read from the repo
> `.env`); its normalizer is `normalize_mistral.py`, chunking **section**-level by default. Recall on
> the BUMA fixture: section **0.878** (`eval/parsing/README.md`).

> **2026-09-24 — `paddleocr` backend, v2 document processing only.** `paddle_ocr.py` calls
> Baidu AI Studio's PaddleOCR-VL job API (multipart submit → poll `state` until `done` →
> JSONL `result.layoutParsingResults[]`, one per page); figures come back as hosted URLs and are
> downloaded at parse time into `markdown.images_base64`, so the cache is self-contained.
> `normalize_paddle.py` turns the layout blocks into the same `ParsedDocument`. Selected by the
> app setting `DOCUMENT_PARSER_BACKEND`; the default here and the CLI's `--backend` choices are
> unchanged, so offline knowledge runs never reach it. PDF and images only. Needs
> `PADDLEOCR_ACCESS_TOKEN`.

## Files

- `contracts.py` — **the seam.** `Chunk`, `ParsedDocument`, and the declared
  `Mention` / `TermRecord` handoff shapes. The one file both halves must agree on
- `config.py` — all settings, and the single place the GPU backend flips
- `parse.py` — parser wrapper (Mistral or MinerU) with a content-addressed parse cache
- `mistral_ocr.py` — the Mistral OCR API call (upload → `ocr.process`); no GPU
- `normalize.py` — MinerU's flat item list → meaningful chunks
- `normalize_mistral.py` — Mistral's block stream → chunks (the Mistral counterpart of `normalize.py`)
- `render.py` — LaTeX/HTML → readable prose for `Chunk.text` (shared by both normalizers)
- `checks.py` — quality warnings, including silent-corruption detection
- `manifest.py` / `report.py` — per-run record and throughput extrapolation
- `run.py` — batch CLI: one folder in, artifacts + manifest out

## Use

```bash
py scripts/parse.py                     # Windows
python3 scripts/parse.py                # Linux
py scripts/parse.py data/other          # a different folder
py scripts/parse.py --backend vlm       # force a specific backend
```

The script sets the env vars that are required and picks a backend. Calling the
module directly works too, but then those env vars are yours to set — see
`scripts/parse.py` for the list.

> The wrapper is **Python, not shell**, deliberately. Bash and PowerShell
> versions were written first and inside twenty minutes hit two traps that are
> specific to shells: CRLF line endings making bash fail on its very first line,
> and a trailing backslash on a path swallowing the argument AFTER it.
> `subprocess` takes a list of arguments and never goes through a shell, so
> neither is possible.

This is a development aid. **Production calls the module directly** — parsing is
an admin-triggered offline batch, not something a person runs from a terminal.

A successful run looks like this:

```
>> No GPU detected — falling back to pipeline
   (no image descriptions; multiplication copied as the letter 'x')
Run 20260826-131546 | backend=pipeline | 2 documents

[1/2] Maintenance Engineering Handbook (1)-1 … ok (cache) | 1 chunk
[2/2] STD_2026_006_MNO … ok 385.5s | 31 chunks | ⚠ 3 warnings

--- summary ---
  ok / failed     : 2 / 0
  total pages     : 10
  ⚠ quality warnings: 3 — see manifest.json
```

How to read it:

- **`ok (cache)`** means that document was not re-parsed. The cache key is
  document content + settings + MinerU version, so only what genuinely changed
  gets redone. Cached documents are **excluded** from the seconds/page average.
- **`⚠ n warnings` is NORMAL, not a failure.** The document is still
  processed. Warnings flag quality worth a human look — formulas with no
  numbers, empty tables, charts with no context, and most importantly **numbers
  that are absent from the source PDF**. Details land in `manifest.json`.
- **`ok / failed`** is the line to watch. Failed documents are recorded in full,
  with their traces, in `failures.jsonl`, and one failure does not stop the rest.

Throughput numbers and their extrapolation:

```bash
uv run --no-sync python -m src.knowledge_parsing.report --scale 6800
```

### Backend

> These are the **MinerU** backends — now the **fallback** (see the Mistral note at the top).
> They still apply whenever `--backend hybrid/vlm/pipeline` is chosen.

Among the MinerU backends the default is **`hybrid` + `effort=high`**, decided 2026-08-24 after all
four were measured on the same GPU over the same 29 pages:

| Backend | s/page | Image descriptions | Multiplication sign |
|---|---|---|---|
| `hybrid/high` ⭐ | 4.85 | **yes** | `\times` |
| `hybrid/medium` | 1.88 | force-disabled by MinerU | `\times` |
| `pipeline` | 2.67 | never | the letter `x` — **fuses tokens** |
| `vlm` | 6.96 | yes | `\times` |

`vlm` and `hybrid/high` produce **100% identical** output, but `vlm` is slower.
`pipeline` is the only one that copies the multiplication sign as a literal `x`,
which fuses `MOHH x Qty x PA` into a single token and takes five terms with it.

**Determinism: settled 2026-08-26.** Two genuine re-parses produced 0 differences
across 50 chunks — image descriptions included, character for character. Frozen
baselines do not drift. Parse *time* varies (model loading); the output does not.

### Running on a laptop with no GPU

`hybrid` and `vlm` need a GPU. Without one the script falls back to `pipeline` on
its own and says so. That output is **valid but lower quality** — no image
descriptions, and the multiplication sign copied as a letter. Do not use it as a
reference measurement.

Calling the module directly without `--backend` stops with a message naming the
command to use instead, rather than an opaque torch error.

### Installing MinerU

MinerU is an **extra**, not a core dependency — the agent service never parses a
document during a request, so the deployed Space does not carry it. Importing
this package does not import MinerU or torch; only `parse.py` does, when called.

```bash
UV_CACHE_DIR=D:/uv-cache uv pip install --python .venv mineru==3.4.4
```

⚠️ **Not `uv sync --extra knowledge-parsing`.** That fails: `mineru` needs
`python-docx>=1.2.0` while this project pins `1.1.2`. Locking it means raising
the shared pin, and that is a team decision — not taken yet.

So `mineru` is **absent from `uv.lock`**. Two consequences:

- **A plain `uv run` uninstalls MinerU** — it syncs the venv to the lock first.
  Use `uv run --no-sync`, this repo's standing convention (see CLAUDE.md);
  `scripts/parse.py` already does.
- If anyone runs `uv sync`, MinerU disappears. Reinstall with the command above;
  `scripts/parse.py` prints that command itself when it happens.

### Where the large files live

All kept off the nearly-full `C:` drive:

| | |
|---|---|
| Model weights | `D:/mineru-models/hf`, `D:/mineru-models/modelscope` |
| Temp files | `D:/mineru-models/tmp` |
| `uv` cache | `D:/uv-cache` |
| Parse cache + run output | `data/knowledge_cache/`, `data/knowledge_runs/` (gitignored) |

## Naming documents — a convention, and why it is temporary

**The file name is currently a primary key, not a label.** `doc_id` is the file
name stem; `chunk_id` is built from `doc_id`; and `term_id`, `formula_id`,
`rule_id` and `domain_id` are all digests of `doc_id` (`knowledge_extraction/ids.py`).
Renaming a file after ingestion therefore changes every identifier derived from
it, and an expert's approve/reject decisions are keyed to those identifiers.

Until that changes, name documents like this:

```
<DOC-CODE>_Rev<N.N>.pdf

STD-2026-006-MNO_Rev0.0.pdf
MEH-2019_Ch03.pdf
```

- **Stable prefix, varying suffix.** The doc code stays fixed across revisions so
  they can eventually be recognised as versions of one document; `_Rev1.0` is the
  part that moves, and it is machine-readable rather than only human-readable.
- **No spaces.** They force quoting in every command, and a trailing backslash on
  a quoted path has already swallowed an argument once in this repo.
- **Never `(1)` / `(2)`.** That suffix means "downloaded twice", not "version 2".
  Ingested as-is it becomes a second legitimate document with its own full set of
  identifiers. If the content is identical, delete one — `run.py` warns when two
  inputs share a hash. If the content genuinely differs, it is a revision: give it
  a real `_Rev<N>`.
- **Never rename after ingestion.**

The long human title is not lost by any of this — it stays inside the document,
and normally appears as the first chunk's `heading`.

### Why this is a stopgap, not a design

The convention holds only because *we* control the file names while parsing is an
admin-triggered batch. The moment this sits behind an upload endpoint — which is
planned, after eval and the playground — it stops holding: people will upload
`Copy of SOP final FIX (2).pdf`, and permanent knowledge-base identifiers should
never have depended on what someone typed into a Save dialog.

The fix is to stop deriving identity from the name. `ParsedDocument` already
carries `content_hash`, a SHA-256 of the file, so the identifier exists — it is
simply not the one being used:

```
doc_id        content_hash      key, stable, machine-facing
source_title  the file name     label, may change freely
```

That is a seam change (`doc_id` reaches `chunk_id`, `Provenance` and `ids.py`), so
it needs agreeing rather than doing. It is also cheapest now: the migration cost
rises with every document ingested and every expert decision recorded, because
re-keying orphans all of them. One consequence to handle when it lands —
`Provenance` currently shows `doc_id` to the reviewer, so a display title has to
travel with it or experts will be reading hashes.

`parse._mineru_name` exists for the same reason and is deliberately minimal: once
`doc_id` is a hash, every path is short by construction and that function is
deleted.

## Three things that are easy to get wrong

**`Chunk.text` must stay verbatim.** Extraction's guardrail locates quoted spans
literally inside it. Reflowing or normalising the text makes the lookup fail and
the field goes silently null — which presents as a bad model, not as a parser
bug.

**`page_idx` is 0-based**, exactly as MinerU reports it, with no conversion
anywhere in the pipeline. Converting to 1-based is the UI's job, done once at
display time, so the artifact always matches the raw MinerU output kept beside
it in the cache.

**Markup never goes in `text`.** The term filter is an NER model reading prose;
MinerU writes formulas character-spaced (`P u r c h a s i n g ~ c o s t s`) and
tables as HTML, and neither yields a single mention. Measured on one document and
gold set, changing only the parse: raw markup scored recall 0.7561 against 0.8537
for plain text; rendering it back recovered 0.8293. The markup is preserved in
`Chunk.latex` and `Chunk.table_html`, because the formula branch needs that form.

## Output layout

```
data/knowledge_cache/parse/<hash>-<config>-<version>/   untouched MinerU output
data/knowledge_runs/<run_id>/
├── manifest.json     backend, versions, timings, quality warnings
├── failures.jsonl    documents that failed, with traces
└── <doc>/chunks.json the artifact
```

Both are gitignored — they are data, not code.

See [`docs/knowledge/KNOWLEDGE_PIPELINE_TODO.md`](../../docs/knowledge/KNOWLEDGE_PIPELINE_TODO.md)
for the task breakdown and the seam discussion, and
[`docs/knowledge/KNOWLEDGE_PIPELINE_CALIBRATION.md`](../../docs/knowledge/KNOWLEDGE_PIPELINE_CALIBRATION.md)
§4 for the chunking constants this module follows.