# Knowledge Pipeline โ€” Context for the Data Eyond Python Repo (19 Aug 2026) > ๐Ÿ”„ **LIVE โ€” intent & settled decisions.** Why the pipeline is shaped this way; does not track status. Status lives in [DEV_PLAN.md](../../DEV_PLAN.md) ยง0.8, built state in [REPO_STATUS.md](../../REPO_STATUS.md) ยง14. > **Who this is for:** a Claude Code session working in `Agentic-Service-Data-Eyond-Catalog` > (the Python agentic service). This doc carries *context and intent* only โ€” the why, the > ownership split, the shape of the pipeline, and the decisions already settled. It does not > prescribe folder layout, module names, or endpoint signatures; work those out against > `CLAUDE.md`, `REPO_STATUS.md` and the existing subsystem patterns in the repo. > --- ## 1. What we are doing and why now Data Eyond is an AI data-analyst platform. The strategic thesis from the 30 July exec review is that **domain knowledge is the differentiator, not architecture** โ€” orchestrators and connectors are replicable in a quarter, model capability is rented, only encoded domain knowledge compounds. The end goal is an **MCP product** exposing modular domain knowledge that EMA and later clients connect to. That knowledge currently comes from experts typing it in. **Mas Beta** (plant & maintenance) is the single validation bottleneck โ€” nothing becomes an artifact without him. The knowledge pipeline exists to change the expert's job from *authoring* to *reviewing*: the pipeline reads the client's own documents (standards, SOPs, handbooks) and proposes candidate knowledge entries; the expert approves, edits or rejects them. **Why it lands in this repo now.** We want to integrate the pipeline into Data Eyond to test it end-to-end on real documents, in the place where the resulting knowledge will actually be consumed. This is a **test integration**, not a productionisation โ€” treat it as a new, gated subsystem alongside the existing document/catalog pipelines, not as a rewrite of them. --- ## 1.5 Two products, one pipeline (recorded 2026-09-07) **This is an experiment, and naming what it is an experiment *for* changes what "done" means.** The review loop, the admin ingestion step and the curated-artifact lifecycle belong to a **separate B2B product**, not to Data Eyond. In that product a company's **admin** configures the company's knowledge: they onboard documents, run extraction, and work the review queue until the result is a set of trusted knowledge artifacts. The company's **normal users** never touch that surface โ€” they upload limited data and run analyses with Data-Eyond-shaped tools against knowledge somebody else curated. Admin-configures / users-consume is the whole shape of that product, and the expert review queue is the mechanism that makes it trustworthy. **Data Eyond does not need the full product.** Its version is a single added step: a user clicks one button and the pipeline extracts knowledge from **many documents at once** โ€” not one file at a time, no admin role, no curation UI, and none of Product B's permissions model. **But the review step stays.** Not because Data Eyond's users need it, but because *we* do: the point of running this here is to feel the whole loop end to end โ€” candidate โ†’ queue โ†’ a human ruling โ†’ an approved set โ€” and learn where it is wrong before Product B is built on it. A pipeline that stops at "here are 63 candidates" never surfaces the problems that only appear when somebody actually works the queue. So `knowledge_reviews` is written, not merely created, and the review surface is built to the *minimum* that makes the loop real rather than to Product B's specification. | | Product B (the B2B knowledge product) | Data Eyond today | |---|---|---| | Who ingests | Company admin, as configuration | Any user, one button | | Granularity | Per document, deliberate | **All the user's documents, one batch** | | Review | Expert queue โ€” the point of the product | **Kept, as a simulation** โ€” end to end, minimal surface | | Who owns the knowledge | The company | The company โ€” **not the uploader** | | Scope key | Company | **Company** (`scope_id`) โ€” decided 2026-09-07 | | Consumers | The company's normal users, over MCP | The analytics path, later | **Knowledge is the company's, not the uploader's.** Data Eyond is sold to a company whose employees are its users, so the production standard one employee uploads and the maintenance SOP another uploads are one body of knowledge and belong in one glossary. Every persisted row therefore carries a `scope_id` tenant key beside `user_id` (which records who uploaded it). Under a per-user key those two documents would never diff against each other and cross-document dedup would never fire โ€” which is most of the value. Interim value and the invariant that keeps it cheap to re-point: [KNOWLEDGE_PERSISTENCE_CONTRACT.md](KNOWLEDGE_PERSISTENCE_CONTRACT.md) revision note 3. **Why this repo builds it anyway.** Data Eyond is where we can run the thing end to end on real documents and learn from the mistakes cheaply. The review tables and the diff exist here because **the experiment is the deliverable** โ€” we are buying the lessons, not the feature. What Data Eyond consumes is the artifact; what we keep is what we learned about producing it. **Two consequences that bind the build:** 1. **The batch is the unit, not the file.** One click over N documents is one job with N documents inside it, and the multi-document semantics below (ยง1.6) are the primary path, not an edge case. 2. **Nothing gets curated automatically.** With no expert in the Data Eyond flow, a cross-document disagreement cannot be *resolved* โ€” only *surfaced*. Silently picking a winner would be the one failure this pipeline was built to avoid. ## 1.6 What happens with multiple documents **Entries append; they never merge.** Each document is parsed, filtered, clustered and extracted independently, and `entity_id` is scoped per document by design (`ids.py`) โ€” the same term in two documents gets two ids. That is deliberate: deciding that BUMA's "PA" and a textbook's "PA" are the same concept is a judgement about the world, not a string comparison, which is the same reasoning that makes the pipeline record *"Physical **of** Availability"* verbatim instead of normalising it. Cross-document overlap is **classified, not collapsed**. `diff/glossary_diff.py` matches candidates against an active glossary on the term surface and labels each `new`, `duplicate` or `conflicting`. So a second mining-production PDF whose "PA" agrees comes back `duplicate`, and one that disagrees comes back `conflicting` โ€” visible, attributed to its source, and left standing. **`DomainContext` stays per document.** Its `purpose_verbatim` is a span located *in that document*; there is no such thing as a located company-wide purpose. N documents produce N domain contexts, and the corpus-level view is a **list of them**, never a composed summary โ€” composing one is precisely what the v3 redesign moved away from when it made the branch guardable. **Company scope is what makes the diff earn its keep.** One user's own documents rarely overlap, so at per-user scope `diff_status` would almost never fire. Across a company's uploads the collisions are real โ€” two documents defining PA, one agreeing and one not โ€” which is exactly the signal worth surfacing. **What we are deliberately not building yet:** a canonical company-level concept layer that unifies `t_abc` in doc A with `t_xyz` in doc B under one approved entry. It is the right shape for Product B and it needs an expert to rule on conflicts, so it waits until Product B is real. Until then `diff_status` on each entry carries the cross-document relationship and that is enough. --- ## 2. Ownership โ€” two items, two people, one seam The pipeline splits cleanly in two, and the split is deliberate: **they are separate, independently runnable, independently swappable pipelines.** | Half | Owner | Scope | |---|---|---| | **โ‘  Parsing (Mistral OCR)** | **Sofhia Az-Zahra** | PDF/DOCX in โ†’ structured, typed extracted content out. **Update 2026-09-02:** the parser is now **Mistral OCR** (API, **no GPU**). **2026-09-07:** MinerU dropped entirely โ€” one parser, installed nowhere else โ€” so the swappability the seam promises (below) is now proven with a real second parser. Both add formula/image/chart/diagram detection with type labels. (MinerU history: it replaced the text+table-only extraction and was presented 13 Aug as the prerequisite for Scenario 2.) | | **โ‘ก Knowledge extraction** | **Rifqi (me)** | Parsed content in โ†’ validated candidate knowledge entries out. Term filtering, clustering, evidence ranking, schema-guided LLM attribute fill, span validation, diff against the active glossary version, and the frequency-sorted review queue. | **The seam.** The two halves must meet only at a **persisted, versioned parsed-document artifact** โ€” extraction consumes that artifact, never a raw file, never the parser's API. This is the single most important structural constraint in the whole plan, because: - MinerU can be swapped (or fall back to the existing Tesseract / Azure Document Intelligence paths) without extraction knowing; - extraction can be re-run cheaply on an already-parsed corpus when a prompt or schema changes โ€” and it will be, often, because prompt iteration is the main development loop here; - Sofhia and I can work and test independently, against a fixture artifact, without blocking each other; - parsing is the slow/expensive-to-rerun half; extraction is the fast-to-iterate half. Keeping the boundary hard means we never re-parse to fix an extraction bug. Anything that makes the two halves import each other's internals, or that makes extraction take a file path, breaks the point of the split. --- ## 3. The pipeline Six stages. The cost story matters as much as the flow: almost everything is free CPU/regex work, and the LLM appears in exactly one stage. ```mermaid flowchart TD A["Admin / expert triggers ingest
not continuous, not per-user"] --> B subgraph P["โ‘  PARSING โ€” Sofhia"] B["MinerU
text ยท tables ยท formulas ยท figures ยท charts"] --> C["Section-aware chunking
split on numbered headings 2.1.1, 2.1.2 โ€ฆ"] end C --> SEAM[("Parsed-document artifact
versioned ยท persisted
THE SEAM")] SEAM --> D subgraph F["โ‘ก TERM FILTER โ€” CPU / regex, free"] D["GLiNER span filter on ALL chunks โ†’ term mentions"] --> H E["Discourse-cue regex โ†’ rule-of-thumb candidates"] --> H G["Legend-block regex โ†’ formula vars + abbreviation dictionary"] --> H G2["Section pass โ†’ summary units"] --> H H["Normalise + cluster mentions
PA ยท P.A. ยท Physical Availability โ†’ 1 cluster"] --> I["Evidence ranking
cue ยท heading ยท legend ยท formula ยท formatting ยท first-occurrence"] I --> J["Top-3 evidence chunks per cluster"] end SEAM --> E SEAM --> G SEAM --> G2 J --> K subgraph X["โ‘ข KNOWLEDGE EXTRACTION โ€” the only paid stage"] K["1 LLM call per TERM CLUSTER
(not per mention, not per chunk)"] L["1 call per cue chunk ยท 1 per unique formula ยท 1 per document (summary)"] end K --> M L --> M subgraph V["โ‘ฃ VALIDATION โ€” deterministic"] M["Verbatim-span check โ†’ reject field if span not locatable"] --> N["Null definition โ†’ escalate to next 3 evidence chunks
max 2 rounds โ†’ else flag 'term found, no definition located'"] N --> O["Conflicting definitions โ†’ definition_conflict + variants[]"] end O --> Q["โ‘ค DIFF vs active glossary version
duplicate (skip) ยท new ยท conflicting"] Q --> R subgraph RV["โ‘ฅ REVIEW โ€” human"] R["Queue sorted by mention frequency
PA 250 โ†’ UA 180 โ†’ โ€ฆ โ†’ long tail"] --> S["Expert (Mas Beta) approves / edits / rejects"] S --> T["Glossary ยท Interpretation Pack ยท Domain Context ยท Formula"] T --> U["Versioning + approval audit trail"] end U --> W["MCP server โ€” the consumption surface"] style P fill:#e8f4ff,stroke:#4a90d9 style F fill:#e6f7e6,stroke:#4caf50 style X fill:#dbe9ff,stroke:#2f6fd0 style V fill:#f0e6ff,stroke:#8e5fd0 style RV fill:#fff0e0,stroke:#e08b3c style SEAM fill:#fffbe0,stroke:#c9a227,stroke-width:3px ``` Colour convention, carried from the diagrams shown to the team: **green = free** (CPU/regex โ€” most of the pipeline) ยท **blue = LLM** (the only paid part) ยท **purple = deterministic code** ยท **orange = human**. ### The five-box version (how this is explained to execs) ```mermaid flowchart LR A["PARSE
MinerU"] --> B["FILTER
free, CPU"] --> C["EXTRACT
LLM, per term cluster"] --> D["VALIDATE + DIFF
deterministic"] --> E["EXPERT REVIEW
frequency-sorted queue"] style A fill:#e8f4ff,stroke:#4a90d9 style B fill:#e6f7e6,stroke:#4caf50 style C fill:#dbe9ff,stroke:#2f6fd0 style D fill:#f0e6ff,stroke:#8e5fd0 style E fill:#fff0e0,stroke:#e08b3c ``` ### Ownership view ```mermaid flowchart LR subgraph S["Sofhia"] P["Parsing pipeline
MinerU + backends"] end subgraph R["Rifqi"] K["Extraction pipeline
filter โ†’ LLM โ†’ validate โ†’ diff โ†’ queue"] end subgraph H["Mas Har / Mas Beta"] C["Curation UI + expert review
versioning, approval"] end P -->|"versioned parsed artifact
(the only contract)"| K K -->|"candidate entries + provenance"| C C -->|"active versions"| M["MCP server"] ``` --- ## 4. Decisions already settled โ€” do not reopen these 1. **The LLM call unit is the term cluster** โ€” not the chunk, not the mention. Per-chunk works for a 9-page document and breaks at 1,000+ pages: "PA" mentioned 250ร— would produce 250 near-identical candidates. Clustering first cuts expert review burden ~6.2ร— (3,125 raw extractions โ†’ ~500 term entries on a 1,000-page document). The token saving is trivial (~$0.45); **the review-burden reduction is the justification.** 2. **Conflict detection depends on the clustering.** Two contradictory definitions of the same term can only be spotted because all evidence for that term arrives in one call. Two separate calls would never meet, and the conflict would surface as two silently-accepted entries. 3. **No standalone relevance gate.** A binary relevance classifier before the term filter is a second imperfect classifier whose false negatives drop content *before* extraction sees it โ€” and dropped content never reaches expert review to be caught. GLiNER is cheap enough (CPU, ~400MB) to run unconditionally on every chunk; "zero candidate spans" becomes the relevance signal as a byproduct of looking, rather than a judgment made before looking. 4. **Evidence ranking is allowed even though it is also a filter** โ€” because no term is dropped (only which passages feed the call is narrowed), all mentions stay in provenance so a bad top-3 is visible, and the escalation loop self-corrects. The rejected relevance gate had none of those three properties. This distinction is the reasoning to preserve if the ranking is ever revisited. 5. **`provenance.span` is mandatory and verbatim.** A field whose span cannot be located in the source is rejected. This is the primary anti-hallucination control and the thing that makes expert review *finishable* โ€” the reviewer checks a quote against a page, not a claim against their memory. 6. **All content fields are Optional in the schema.** The model must be able to answer null; a required field is a fabrication invitation for smaller models. `subdomain_tags` is an enum โ€” classification, not generation. 7. **Frequency-sorted review queue.** Directly targets the Mas Beta bottleneck: validate the ~40 terms carrying the document's meaning first, let the long tail wait. 8. **Model tier is per branch.** Glossary and Formula are extraction tasks well-guarded by span verification โ†’ a small/cheap tier is fine. Rule-of-thumb was a reasoning/generation task where a small model's failure mode is least detectable โ†’ larger tier. Test before committing the small tier on the rule-of-thumb branch. **Amended 2026-09-02:** Summary no longer belongs in that second group. It was reclassified from *generation* to *extraction* by rewriting the prompt to locate the document's own statement of purpose rather than compose one about it, so it is now span-guarded like glossary and formula. The argument for a larger tier on that branch has weakened accordingly โ€” locating a sentence asks far less of a model than writing one. 9. **GLiNER-hybrid stays the glossary-branch default.** Pure NER can only fill 2 of 6 schema fields (definition, formula and interpretation are not spans โ€” they are assembled from text that may be pages apart). Pure small-LLM shows recall 0.93โ€“1.00 at precision 0.15โ€“0.43; perfect recall is the failure signature of a model fabricating a value for every field. 10. **The pipeline records the literal source wording.** It recorded "Physical of Availability" from the BUMA standard rather than the textbook's "Physical Availability" โ€” surfacing the discrepancy to the expert instead of silently normalising it. Keep that behaviour. --- ## 5. Deployment reality โ€” this shapes every engineering trade-off The pipeline runs **once at setup (bulk corpus ingest)** and **occasionally when an admin or expert adds a file**. It is **not continuous** and **not per-user** โ€” normal engineers cannot add data. Consequences, in the order they matter: - **Extraction quality and expert-review efficiency dominate.** A missed term has no self-correcting mechanism, because engineers can't add data to fill a gap later. - **Maintainability matters more than it looks.** A glue chain invoked every few months is exactly the thing nobody remembers how to debug. Prefer boring, inspectable stages over clever ones. - **API cost is negligible and latency is irrelevant.** ~$8โ€“12 for a 6,000-page corpus; an incremental single 9-page file is ~1.5 US cents. Do not spend engineering time optimising tokens โ€” engineering time on the review experience is worth more than any token saving available here. - **No GPU required.** GLiNER is a CPU encoder model (~1.6 seq/s Python, ~6.7 seq/s via the Rust port on an i9 8-core). ~15,000 chunks โ‰ˆ 2.6h Python / ~40min Rust for a job that runs a few times a year. The LLM half is an API call. (Runtime as reported to the team on 13 Aug: ~11 s/page GLiNER, ~26 s/page LLM โ€” with the caveat that **cost scales on term count, not page count.**) - **Two ingest modes to build:** *bulk setup ingest* (corpus-scale โ€” the only place corpus-frequency statistics are usable as a candidate-term booster) and *incremental single-file add* (no corpus context; diff against the active glossary version). - **The summary branch is the quiet cost risk** โ€” only ~300 calls but a quarter of all input tokens, because summarisation can't be filtered; it needs whole documents. --- ## 6. Where the output goes Approved candidates become the project's **four artifacts** (see the `context` doc for the full architecture): 1. **Domain knowledge** โ€” system-prompt material, capped ~1500 tokens: operating and equipment hierarchy, time convention (WITA), seasonality, most-important glossary terms, high-level business process, and agent limitations. 2. **Data dictionary** โ€” from DB column profiling; used for query generation. Pak Ricky connected the extraction work directly to **automating this**. 3. **Interpretation pack** โ€” expert-curated interpretation logic, action rules, benchmark history, tied to a use case ("PA missed" reads differently from "PA missed but MTBS achieved"). Revived on 13 Aug after earlier removal for being too normative โ€” reintroduce carefully. 4. **Skill registry** โ€” query rules and expert-approved formulas. Approval is mandatory because the same term computes differently per company (MTTR at BUMA = breakdown duration รท breakdown frequency; MTTR in IT = mean time to resolve). All four carry **versioning plus an approval audit trail**, and the expert chooses which version is active. Delivery format is **MCP** โ€” skill registry โ†’ MCP skills; data dictionary โ†’ MCP on demand; interpretation pack โ†’ attached to the skill; domain knowledge โ†’ system prompt but preferably served via MCP so owners can change it without redeploying the engine. --- ## 7. Grounding material Two real sample documents anchor every design claim above, and should anchor test fixtures too: - **BUMA STD/2026/006/MNO Rev.0.0** โ€” Production Parameter & ECA. 9 pages, majority Bahasa Indonesia. Carries: PA (glossary, p.4), Other Activity + the controllability hierarchy (rule-of-thumb, pp.8โ€“9), the MOHH/Qty/PA/UA/Pty legend block (formula branch, p.2). - **Open Pit Mine Planning & Design** (textbook excerpt) โ€” carries NSR as a **cross-page definition**: intro p.92, formula p.93, interpretive remark p.94 (ยง2.3.5). This is the case that justifies section-aware chunking over fixed token windows โ€” the whole definition must stay inside one semantic unit. **Known open risk:** GLiNER multi-v2.1's Indonesian performance is unverified, and the BUMA standard is majority Bahasa Indonesia. This is the standing "Indonesian degradation" question, now with a concrete test case. Verify before trusting recall numbers. **Flagged as scope creep, proposed not decided:** ensembling GLiNER + LangExtract on every ingest. Cheap in API terms, but it adds a reconciliation/dedup step between two candidate sets that may disagree โ€” new engineering surface for uncertain gain. --- ## 8. Integration guidance for this repo Deliberately shape-agnostic โ€” decide the specifics against `CLAUDE.md` and the existing subsystem patterns, not against this doc. - **Both halves are new subsystems, not modifications** to the existing document/catalog pipelines. The current unstructured path (Tesseract OCR โ†’ chunk โ†’ pgvector) stays as it is; the MinerU path is additive and, at least initially, gated. - **Follow the repo's existing multi-stage precedent.** The query subsystem (IR โ†’ validator โ†’ compiler โ†’ executor, orchestrated from a service facade, triggered by thin entry points) is the closest structural analogue to what both halves need. - **Respect the hard boundaries.** Go owns the dedorch schema โ€” any new table needs a Harry-ready DDL handoff, never DDL executed from Python. Any new endpoint on the live surface needs a contract-doc entry, and an admin ingest surface is a different risk class than the read-only chat surface โ€” raise the authentication question rather than assuming. - **Build the offline path first.** Because this runs a few times a year and is triggered by an admin, a script-driven run over a fixture document is the honest first milestone; HTTP endpoints are a convenience layer over it, not the other way round. - **Fixtures over live parsing in tests.** A committed parsed-artifact fixture from one of the two sample documents lets the extraction half be tested without MinerU installed at all โ€” which is the seam doing its job.