docs/knowledge/knowledge_pipeline_context.md

# Knowledge Pipeline — Context for the Data Eyond Python Repo (19 Aug 2026)

<!-- doc-status -->
> 🔄 **LIVE — intent & settled decisions.** Why the pipeline is shaped this way; does not track status. Status lives in [DEV_PLAN.md](../../DEV_PLAN.md) §0.8, built state in [REPO_STATUS.md](../../REPO_STATUS.md) §14.

> **Who this is for:** a Claude Code session working in `Agentic-Service-Data-Eyond-Catalog`
> (the Python agentic service). This doc carries *context and intent* only — the why, the
> ownership split, the shape of the pipeline, and the decisions already settled. It does not
> prescribe folder layout, module names, or endpoint signatures; work those out against
> `CLAUDE.md`, `REPO_STATUS.md` and the existing subsystem patterns in the repo.
>
---

## 1. What we are doing and why now

Data Eyond is an AI data-analyst platform. The strategic thesis from the 30 July exec review
is that **domain knowledge is the differentiator, not architecture** — orchestrators and
connectors are replicable in a quarter, model capability is rented, only encoded domain
knowledge compounds. The end goal is an **MCP product** exposing modular domain knowledge
that EMA and later clients connect to.

That knowledge currently comes from experts typing it in. **Mas Beta** (plant & maintenance)
is the single validation bottleneck — nothing becomes an artifact without him. The knowledge
pipeline exists to change the expert's job from *authoring* to *reviewing*: the pipeline reads
the client's own documents (standards, SOPs, handbooks) and proposes candidate knowledge
entries; the expert approves, edits or rejects them.

**Why it lands in this repo now.** We want to integrate the pipeline into Data Eyond to test
it end-to-end on real documents, in the place where the resulting knowledge will actually be
consumed. This is a **test integration**, not a productionisation — treat it as a new, gated
subsystem alongside the existing document/catalog pipelines, not as a rewrite of them.

---

## 1.5 Two products, one pipeline (recorded 2026-09-07)

**This is an experiment, and naming what it is an experiment *for* changes what "done" means.**

The review loop, the admin ingestion step and the curated-artifact lifecycle belong to a
**separate B2B product**, not to Data Eyond. In that product a company's **admin** configures the
company's knowledge: they onboard documents, run extraction, and work the review queue until the
result is a set of trusted knowledge artifacts. The company's **normal users** never touch that
surface — they upload limited data and run analyses with Data-Eyond-shaped tools against knowledge
somebody else curated. Admin-configures / users-consume is the whole shape of that product, and the
expert review queue is the mechanism that makes it trustworthy.

**Data Eyond does not need the full product.** Its version is a single added step: a user clicks one
button and the pipeline extracts knowledge from **many documents at once** — not one file at a time,
no admin role, no curation UI, and none of Product B's permissions model.

**But the review step stays.** Not because Data Eyond's users need it, but because *we* do: the point
of running this here is to feel the whole loop end to end — candidate → queue → a human ruling → an
approved set — and learn where it is wrong before Product B is built on it. A pipeline that stops at
"here are 63 candidates" never surfaces the problems that only appear when somebody actually works
the queue. So `knowledge_reviews` is written, not merely created, and the review surface is built to
the *minimum* that makes the loop real rather than to Product B's specification.

| | Product B (the B2B knowledge product) | Data Eyond today |
|---|---|---|
| Who ingests | Company admin, as configuration | Any user, one button |
| Granularity | Per document, deliberate | **All the user's documents, one batch** |
| Review | Expert queue — the point of the product | **Kept, as a simulation** — end to end, minimal surface |
| Who owns the knowledge | The company | The company — **not the uploader** |
| Scope key | Company | **Company** (`scope_id`) — decided 2026-09-07 |
| Consumers | The company's normal users, over MCP | The analytics path, later |

**Knowledge is the company's, not the uploader's.** Data Eyond is sold to a company whose employees
are its users, so the production standard one employee uploads and the maintenance SOP another
uploads are one body of knowledge and belong in one glossary. Every persisted row therefore carries
a `scope_id` tenant key beside `user_id` (which records who uploaded it). Under a per-user key those
two documents would never diff against each other and cross-document dedup would never fire — which
is most of the value. Interim value and the invariant that keeps it cheap to re-point:
[KNOWLEDGE_PERSISTENCE_CONTRACT.md](KNOWLEDGE_PERSISTENCE_CONTRACT.md) revision note 3.

**Why this repo builds it anyway.** Data Eyond is where we can run the thing end to end on real
documents and learn from the mistakes cheaply. The review tables and the diff exist here because
**the experiment is the deliverable** — we are buying the lessons, not the feature. What Data Eyond
consumes is the artifact; what we keep is what we learned about producing it.

**Two consequences that bind the build:**

1. **The batch is the unit, not the file.** One click over N documents is one job with N documents
   inside it, and the multi-document semantics below (§1.6) are the primary path, not an edge case.
2. **Nothing gets curated automatically.** With no expert in the Data Eyond flow, a cross-document
   disagreement cannot be *resolved* — only *surfaced*. Silently picking a winner would be the one
   failure this pipeline was built to avoid.

## 1.6 What happens with multiple documents

**Entries append; they never merge.** Each document is parsed, filtered, clustered and extracted
independently, and `entity_id` is scoped per document by design (`ids.py`) — the same term in two
documents gets two ids. That is deliberate: deciding that BUMA's "PA" and a textbook's "PA" are the
same concept is a judgement about the world, not a string comparison, which is the same reasoning
that makes the pipeline record *"Physical **of** Availability"* verbatim instead of normalising it.

Cross-document overlap is **classified, not collapsed**. `diff/glossary_diff.py` matches candidates
against an active glossary on the term surface and labels each `new`, `duplicate` or `conflicting`.
So a second mining-production PDF whose "PA" agrees comes back `duplicate`, and one that disagrees
comes back `conflicting` — visible, attributed to its source, and left standing.

**`DomainContext` stays per document.** Its `purpose_verbatim` is a span located *in that document*;
there is no such thing as a located company-wide purpose. N documents produce N domain contexts, and
the corpus-level view is a **list of them**, never a composed summary — composing one is precisely
what the v3 redesign moved away from when it made the branch guardable.

**Company scope is what makes the diff earn its keep.** One user's own documents rarely overlap, so
at per-user scope `diff_status` would almost never fire. Across a company's uploads the collisions are
real — two documents defining PA, one agreeing and one not — which is exactly the signal worth
surfacing.

**What we are deliberately not building yet:** a canonical company-level concept layer that unifies
`t_abc` in doc A with `t_xyz` in doc B under one approved entry. It is the right shape for Product B
and it needs an expert to rule on conflicts, so it waits until Product B is real. Until then
`diff_status` on each entry carries the cross-document relationship and that is enough.

---

## 2. Ownership — two items, two people, one seam

The pipeline splits cleanly in two, and the split is deliberate: **they are separate,
independently runnable, independently swappable pipelines.**

| Half | Owner | Scope |
|---|---|---|
| **① Parsing (Mistral OCR)** | **Sofhia Az-Zahra** | PDF/DOCX in → structured, typed extracted content out. **Update 2026-09-02:** the parser is now **Mistral OCR** (API, **no GPU**). **2026-09-07:** MinerU dropped entirely — one parser, installed nowhere else — so the swappability the seam promises (below) is now proven with a real second parser. Both add formula/image/chart/diagram detection with type labels. (MinerU history: it replaced the text+table-only extraction and was presented 13 Aug as the prerequisite for Scenario 2.) |
| **② Knowledge extraction** | **Rifqi (me)** | Parsed content in → validated candidate knowledge entries out. Term filtering, clustering, evidence ranking, schema-guided LLM attribute fill, span validation, diff against the active glossary version, and the frequency-sorted review queue. |

**The seam.** The two halves must meet only at a **persisted, versioned parsed-document
artifact** — extraction consumes that artifact, never a raw file, never the parser's API.
This is the single most important structural constraint in the whole plan, because:

- MinerU can be swapped (or fall back to the existing Tesseract / Azure Document Intelligence
  paths) without extraction knowing;
- extraction can be re-run cheaply on an already-parsed corpus when a prompt or schema changes
  — and it will be, often, because prompt iteration is the main development loop here;
- Sofhia and I can work and test independently, against a fixture artifact, without blocking
  each other;
- parsing is the slow/expensive-to-rerun half; extraction is the fast-to-iterate half. Keeping
  the boundary hard means we never re-parse to fix an extraction bug.

Anything that makes the two halves import each other's internals, or that makes extraction
take a file path, breaks the point of the split.

---

## 3. The pipeline

Six stages. The cost story matters as much as the flow: almost everything is free CPU/regex
work, and the LLM appears in exactly one stage.

```mermaid
flowchart TD
    A["Admin / expert triggers ingest<br/><i>not continuous, not per-user</i>"] --> B

    subgraph P["① PARSING — Sofhia"]
        B["MinerU<br/>text · tables · formulas · figures · charts"] --> C["Section-aware chunking<br/>split on numbered headings 2.1.1, 2.1.2 …"]
    end

    C --> SEAM[("Parsed-document artifact<br/>versioned · persisted<br/><b>THE SEAM</b>")]
    SEAM --> D

    subgraph F["② TERM FILTER — CPU / regex, free"]
        D["GLiNER span filter on ALL chunks → term mentions"] --> H
        E["Discourse-cue regex → rule-of-thumb candidates"] --> H
        G["Legend-block regex → formula vars + abbreviation dictionary"] --> H
        G2["Section pass → summary units"] --> H
        H["Normalise + cluster mentions<br/>PA · P.A. · Physical Availability → 1 cluster"] --> I["Evidence ranking<br/>cue · heading · legend · formula · formatting · first-occurrence"]
        I --> J["Top-3 evidence chunks per cluster"]
    end

    SEAM --> E
    SEAM --> G
    SEAM --> G2

    J --> K

    subgraph X["③ KNOWLEDGE EXTRACTION — the only paid stage"]
        K["1 LLM call per TERM CLUSTER<br/>(not per mention, not per chunk)"]
        L["1 call per cue chunk · 1 per unique formula · 1 per document (summary)"]
    end

    K --> M
    L --> M

    subgraph V["④ VALIDATION — deterministic"]
        M["Verbatim-span check → reject field if span not locatable"] --> N["Null definition → escalate to next 3 evidence chunks<br/>max 2 rounds → else flag 'term found, no definition located'"]
        N --> O["Conflicting definitions → definition_conflict + variants[]"]
    end

    O --> Q["⑤ DIFF vs active glossary version<br/>duplicate (skip) · new · conflicting"]
    Q --> R

    subgraph RV["⑥ REVIEW — human"]
        R["Queue sorted by mention frequency<br/>PA 250 → UA 180 → … → long tail"] --> S["Expert (Mas Beta) approves / edits / rejects"]
        S --> T["Glossary · Interpretation Pack · Domain Context · Formula"]
        T --> U["Versioning + approval audit trail"]
    end

    U --> W["MCP server — the consumption surface"]

    style P fill:#e8f4ff,stroke:#4a90d9
    style F fill:#e6f7e6,stroke:#4caf50
    style X fill:#dbe9ff,stroke:#2f6fd0
    style V fill:#f0e6ff,stroke:#8e5fd0
    style RV fill:#fff0e0,stroke:#e08b3c
    style SEAM fill:#fffbe0,stroke:#c9a227,stroke-width:3px
```

Colour convention, carried from the diagrams shown to the team:
**green = free** (CPU/regex — most of the pipeline) · **blue = LLM** (the only paid part) ·
**purple = deterministic code** · **orange = human**.

### The five-box version (how this is explained to execs)

```mermaid
flowchart LR
    A["PARSE<br/>MinerU"] --> B["FILTER<br/>free, CPU"] --> C["EXTRACT<br/>LLM, per term cluster"] --> D["VALIDATE + DIFF<br/>deterministic"] --> E["EXPERT REVIEW<br/>frequency-sorted queue"]
    style A fill:#e8f4ff,stroke:#4a90d9
    style B fill:#e6f7e6,stroke:#4caf50
    style C fill:#dbe9ff,stroke:#2f6fd0
    style D fill:#f0e6ff,stroke:#8e5fd0
    style E fill:#fff0e0,stroke:#e08b3c
```

### Ownership view

```mermaid
flowchart LR
    subgraph S["Sofhia"]
        P["Parsing pipeline<br/>MinerU + backends"]
    end
    subgraph R["Rifqi"]
        K["Extraction pipeline<br/>filter → LLM → validate → diff → queue"]
    end
    subgraph H["Mas Har / Mas Beta"]
        C["Curation UI + expert review<br/>versioning, approval"]
    end
    P -->|"versioned parsed artifact<br/>(the only contract)"| K
    K -->|"candidate entries + provenance"| C
    C -->|"active versions"| M["MCP server"]
```

---

## 4. Decisions already settled — do not reopen these

1. **The LLM call unit is the term cluster** — not the chunk, not the mention. Per-chunk works
   for a 9-page document and breaks at 1,000+ pages: "PA" mentioned 250× would produce 250
   near-identical candidates. Clustering first cuts expert review burden ~6.2× (3,125 raw
   extractions → ~500 term entries on a 1,000-page document). The token saving is trivial
   (~$0.45); **the review-burden reduction is the justification.**
2. **Conflict detection depends on the clustering.** Two contradictory definitions of the same
   term can only be spotted because all evidence for that term arrives in one call. Two
   separate calls would never meet, and the conflict would surface as two silently-accepted
   entries.
3. **No standalone relevance gate.** A binary relevance classifier before the term filter is a
   second imperfect classifier whose false negatives drop content *before* extraction sees it —
   and dropped content never reaches expert review to be caught. GLiNER is cheap enough (CPU,
   ~400MB) to run unconditionally on every chunk; "zero candidate spans" becomes the relevance
   signal as a byproduct of looking, rather than a judgment made before looking.
4. **Evidence ranking is allowed even though it is also a filter** — because no term is
   dropped (only which passages feed the call is narrowed), all mentions stay in provenance so
   a bad top-3 is visible, and the escalation loop self-corrects. The rejected relevance gate
   had none of those three properties. This distinction is the reasoning to preserve if the
   ranking is ever revisited.
5. **`provenance.span` is mandatory and verbatim.** A field whose span cannot be located in the
   source is rejected. This is the primary anti-hallucination control and the thing that makes
   expert review *finishable* — the reviewer checks a quote against a page, not a claim against
   their memory.
6. **All content fields are Optional in the schema.** The model must be able to answer null; a
   required field is a fabrication invitation for smaller models. `subdomain_tags` is an enum —
   classification, not generation.
7. **Frequency-sorted review queue.** Directly targets the Mas Beta bottleneck: validate the
   ~40 terms carrying the document's meaning first, let the long tail wait.
8. **Model tier is per branch.** Glossary and Formula are extraction tasks well-guarded by span
   verification → a small/cheap tier is fine. Rule-of-thumb was a reasoning/generation task where a
   small model's failure mode is least detectable → larger tier. Test before committing the small
   tier on the rule-of-thumb branch.
   **Amended 2026-09-02:** Summary no longer belongs in that second group. It was reclassified from
   *generation* to *extraction* by rewriting the prompt to locate the document's own statement of
   purpose rather than compose one about it, so it is now span-guarded like glossary and formula.
   The argument for a larger tier on that branch has weakened accordingly — locating a sentence
   asks far less of a model than writing one.
9. **GLiNER-hybrid stays the glossary-branch default.** Pure NER can only fill 2 of 6 schema
   fields (definition, formula and interpretation are not spans — they are assembled from text
   that may be pages apart). Pure small-LLM shows recall 0.93–1.00 at precision 0.15–0.43;
   perfect recall is the failure signature of a model fabricating a value for every field.
10. **The pipeline records the literal source wording.** It recorded "Physical of Availability"
    from the BUMA standard rather than the textbook's "Physical Availability" — surfacing the
    discrepancy to the expert instead of silently normalising it. Keep that behaviour.

---

## 5. Deployment reality — this shapes every engineering trade-off

The pipeline runs **once at setup (bulk corpus ingest)** and **occasionally when an admin or
expert adds a file**. It is **not continuous** and **not per-user** — normal engineers cannot
add data. Consequences, in the order they matter:

- **Extraction quality and expert-review efficiency dominate.** A missed term has no
  self-correcting mechanism, because engineers can't add data to fill a gap later.
- **Maintainability matters more than it looks.** A glue chain invoked every few months is
  exactly the thing nobody remembers how to debug. Prefer boring, inspectable stages over
  clever ones.
- **API cost is negligible and latency is irrelevant.** ~$8–12 for a 6,000-page corpus;
  an incremental single 9-page file is ~1.5 US cents. Do not spend engineering time optimising
  tokens — engineering time on the review experience is worth more than any token saving
  available here.
- **No GPU required.** GLiNER is a CPU encoder model (~1.6 seq/s Python, ~6.7 seq/s via the
  Rust port on an i9 8-core). ~15,000 chunks ≈ 2.6h Python / ~40min Rust for a job that runs a
  few times a year. The LLM half is an API call. (Runtime as reported to the team on 13 Aug:
  ~11 s/page GLiNER, ~26 s/page LLM — with the caveat that **cost scales on term count, not
  page count.**)
- **Two ingest modes to build:** *bulk setup ingest* (corpus-scale — the only place
  corpus-frequency statistics are usable as a candidate-term booster) and *incremental
  single-file add* (no corpus context; diff against the active glossary version).
- **The summary branch is the quiet cost risk** — only ~300 calls but a quarter of all input
  tokens, because summarisation can't be filtered; it needs whole documents.

---

## 6. Where the output goes

Approved candidates become the project's **four artifacts** (see the `context` doc for the
full architecture):

1. **Domain knowledge** — system-prompt material, capped ~1500 tokens: operating and equipment
   hierarchy, time convention (WITA), seasonality, most-important glossary terms, high-level
   business process, and agent limitations.
2. **Data dictionary** — from DB column profiling; used for query generation. Pak Ricky
   connected the extraction work directly to **automating this**.
3. **Interpretation pack** — expert-curated interpretation logic, action rules, benchmark
   history, tied to a use case ("PA missed" reads differently from "PA missed but MTBS
   achieved"). Revived on 13 Aug after earlier removal for being too normative — reintroduce
   carefully.
4. **Skill registry** — query rules and expert-approved formulas. Approval is mandatory because
   the same term computes differently per company (MTTR at BUMA = breakdown duration ÷
   breakdown frequency; MTTR in IT = mean time to resolve).

All four carry **versioning plus an approval audit trail**, and the expert chooses which
version is active. Delivery format is **MCP** — skill registry → MCP skills; data dictionary →
MCP on demand; interpretation pack → attached to the skill; domain knowledge → system prompt
but preferably served via MCP so owners can change it without redeploying the engine.

---

## 7. Grounding material

Two real sample documents anchor every design claim above, and should anchor test fixtures too:

- **BUMA STD/2026/006/MNO Rev.0.0** — Production Parameter & ECA. 9 pages, majority Bahasa
  Indonesia. Carries: PA (glossary, p.4), Other Activity + the controllability hierarchy
  (rule-of-thumb, pp.8–9), the MOHH/Qty/PA/UA/Pty legend block (formula branch, p.2).
- **Open Pit Mine Planning & Design** (textbook excerpt) — carries NSR as a **cross-page
  definition**: intro p.92, formula p.93, interpretive remark p.94 (§2.3.5). This is the case
  that justifies section-aware chunking over fixed token windows — the whole definition must
  stay inside one semantic unit.

**Known open risk:** GLiNER multi-v2.1's Indonesian performance is unverified, and the BUMA
standard is majority Bahasa Indonesia. This is the standing "Indonesian degradation" question,
now with a concrete test case. Verify before trusting recall numbers.

**Flagged as scope creep, proposed not decided:** ensembling GLiNER + LangExtract on every
ingest. Cheap in API terms, but it adds a reconciliation/dedup step between two candidate sets
that may disagree — new engineering surface for uncertain gain.

---

## 8. Integration guidance for this repo

Deliberately shape-agnostic — decide the specifics against `CLAUDE.md` and the existing
subsystem patterns, not against this doc.

- **Both halves are new subsystems, not modifications** to the existing document/catalog
  pipelines. The current unstructured path (Tesseract OCR → chunk → pgvector) stays as it is;
  the MinerU path is additive and, at least initially, gated.
- **Follow the repo's existing multi-stage precedent.** The query subsystem (IR → validator →
  compiler → executor, orchestrated from a service facade, triggered by thin entry points) is
  the closest structural analogue to what both halves need.
- **Respect the hard boundaries.** Go owns the dedorch schema — any new table needs a
  Harry-ready DDL handoff, never DDL executed from Python. Any new endpoint on the live surface
  needs a contract-doc entry, and an admin ingest surface is a different risk class than the
  read-only chat surface — raise the authentication question rather than assuming.
- **Build the offline path first.** Because this runs a few times a year and is triggered by an
  admin, a script-driven run over a fixture document is the honest first milestone; HTTP
  endpoints are a convenience layer over it, not the other way round.
- **Fixtures over live parsing in tests.** A committed parsed-artifact fixture from one of the
  two sample documents lets the extraction half be tested without MinerU installed at all —
  which is the seam doing its job.