Commit History
-
[NOTICKET] feat(knowledge_extraction): record which guarantee formula_latex carries
Rifqi Hafizuddin +1: Claude Opus 5Three paths leave formula_latex populated and they are not equally strong: located in the source markup, no markup to check against, or an operator the source cannot express. Until now the output could not tell them apart, so a verified formula and an unverifiable one reached the expert as identical rows. That gap is not theoretical. MinerU's pipeline backend writes multiplication as a bare letter x, so every product formula from such an artifact is unverifiable -- including the Production formula this contract uses as its own worked example. Every artifact on Sofhia's machine is pipeline, because she has no GPU. FormulaEntry gains latex_verification: verified | unverified_no_markup | unverified_operator, null when there is no claim to characterise. Set by the span check, never by the model -- the model cannot know which guard ran, and a field it could write would be a claim rather than a record. _mark_latex is a never-throw seam like the rest of validation: an entry type without th…f81a227 -
[NOTICKET] fix(knowledge_extraction): formula_latex is checked against Chunk.latex
Rifqi Hafizuddin +1: Claude Opus 5The field was guarded in name only. It sat in GUARDED_FIELDS but not in _TRANSCRIBED, so its value was never searched in the source -- only its provenance.span was. A structurally wrong transcription with a valid prose span passed, which is exactly the shape X21 produced: a \frac that lost its division reads as a product and states something the document does not. Adding the field to _TRANSCRIBED was not the fix. `text` carries rendered prose and can never contain LaTeX, so that would have nulled every formula entry and presented as a model failure rather than a validation bug. The haystack was the problem, not the check. Chunk.latex is now the haystack. The parsing half has always emitted it; extraction's adapter dropped it on the floor, which is what left the claim unprovable. It now crosses the seam and is compared with whitespace removed ENTIRELY rather than collapsed. That is a different notion of equality, not a weaker one. In LaTeX, whitespace outside \text{} is not significa…9c8a6e4 -
[NOTICKET] feat(knowledge_extraction): output schema v2 — entity ids, links, 16 fields cut
Rifqi Hafizuddin +1: Claude Fable 5Harry's review of the output contract asked for four things: an id on every entity, a formula referenced rather than restated, rules related to formulas, and a smaller schema. All four are adopted. Two defects surfaced while adding the ids, and they matter more than the ids do. Neither existing identifier survived a re-run: `cluster_id` was a frequency-rank ordinal (`#c000` over a list sorted by mention count), so one changed count reshuffled every id after it, and `rule_id` was written by the model, so the same rule could return a different slug next run. The product is an expert working a review queue — if Mas Beta approves 40 entries and a prompt is retuned, unstable ids lose which 40. Approval continuity, not linking, is why these exist. New `ids.py` mints term/formula/rule/brief ids as sha256 over content: never positional, never model-supplied. New `link.py` resolves every reference after the branches run — deterministic, no LLM, no spend — and returns dangle counts rather than…49dfe4d
-
[NOTICKET] fix(knowledge_extraction): refuse an artifact that will not name its backend
Rifqi Hafizuddin +1: Claude Opus 5MinerU's `pipeline` and `vlm` backends emit different text from the same PDF. The same equation arrives as a semantic `\times` from one and as a spaced literal `x` from the other, and word boundaries that survive in `\text{...}` under `vlm` are gone under `pipeline`. Two extraction results measured on different backends are therefore not comparable, and nothing downstream can tell them apart after the fact. This is not hypothetical: the two halves spent a week comparing numbers taken from two different parses and read the disagreement as a rendering bug. Each side was measuring correctly, on a different artifact. The provenance already existed and was walked around. `ParsedDocument` carries `parser_backend`, and the parsing half reads it from MinerU's own `_middle.json` rather than from config, precisely so the recorded backend is the one actually used. It never reached extraction because what crossed the seam was a bare `content_list.json`, stripped of its envelope. So the CLI now …4db5f14
-
[NOTICKET] feat(knowledge_extraction): paid extraction stage + validate, diff, queue
Rifqi Hafizuddin +1: Claude Fable 5Completes every stage that does not depend on the seam. The pipeline now runs end to end: artifact -> filters -> cluster -> rank -> LLM -> validate -> diff -> frequency-sorted review queue. extract/ client on the nano __54n quad; structured output is PROBED not assumed (json_schema, falling back to json_object + validate-retry, recording which applied); cached tokens read from the API, never modelled. Four branches, one call per TERM CLUSTER. Prompts stay as files so the fixed prefix is byte-identical and caching keeps engaging. validate/ span check (whitespace-only normalisation, never repairs, unlocatable provenance rejects every guarded field), escalation, conflict detection by token overlap — explainable to the reviewer, and it never picks a winner. diff/ new / duplicate / conflicting against an explicitly supplied active glossary, so the duplicate and conflicting paths are reachable. The …ab5ea78 -
[NOTICKET] feat(knowledge_extraction): free stages — models, seam adapter, filters, cluster, ranking
Rifqi Hafizuddin +1: Claude Fable 5First slice of extraction v2, rebuilt in this repo (the kex prototype is not being ported). All stages here are free: zero API calls, CPU only. New module src/knowledge_extraction/ (named to avoid colliding with src/knowledge/, the existing OCR->pgvector service, which is untouched): - models.py pydantic v2 contracts. Encodes the three invariants: content fields Optional, subdomain_tags as an enum, provenance.span mandatory. Adds source_wording so the literal source wording survives instead of being normalised away. - adapter.py THE ONLY file that knows the seam's shape. Stages depend on an internal Chunk subset, so the artifact contract can settle either way without touching a stage. Verified against both live shapes: Sofhia's bare-list draft with its `kind` discriminator, and the document-level envelope proposed in the checklist.…024c30a