docs/README.md

# Docs index

<!-- doc-status -->
> ✅ **LIVE — the map.** Reorganised 2026-08-24. Every doc in this repo carries a **status banner
> under its H1**; read it before trusting the body.

**Status legend:** ✅ LIVE (canonical) · 🔄 LIVE PLAN (in flight) · 💡 PROPOSAL (not approved) ·
⛔ ARCHIVED (superseded, kept for the record) · 📎 REFERENCE (another repo's, can drift)

---

## Start here — repo root

The root deliberately holds only four docs. If you read nothing else, read the first three.

| Doc | Status | Is |
|---|---|---|
| [../REPO_STATUS.md](../REPO_STATUS.md) | ✅ | **What is built.** Canonical current state |
| [../DEV_PLAN.md](../DEV_PLAN.md) | ✅ | **What is in flight.** Sprint plan, task table, locked decisions |
| [../API_CONTRACT_BE_PYTHON.md](../API_CONTRACT_BE_PYTHON.md) | ✅ | **The contract of record** for the FE-facing surface |
| [../README.md](../README.md) | — | HuggingFace Space card. The YAML frontmatter is load-bearing for deployment — don't restructure it |

`CLAUDE.md` also sits in the root but is **gitignored** (local operating manual, not shipped).

**Trust order when sources disagree:** code > REPO_STATUS / DEV_PLAN > everything else.

---

## `knowledge/` — the knowledge-pipeline workstream

Client documents → candidate knowledge entries an expert reviews rather than authors. Two halves
(parsing, Sofhia · extraction, Rifqi) meeting at one versioned artifact.

| Doc | Status | Read it when |
|---|---|---|
| [knowledge/knowledge_pipeline_context.md](knowledge/knowledge_pipeline_context.md) | 🔄 | You need the *why* — ownership split, settled decisions, what not to reopen |
| [knowledge/KNOWLEDGE_PIPELINE_TODO.md](knowledge/KNOWLEDGE_PIPELINE_TODO.md) | 🔄 | You want per-task status and owners. **Start here** |
| [knowledge/KNOWLEDGE_PIPELINE_CALIBRATION.md](knowledge/KNOWLEDGE_PIPELINE_CALIBRATION.md) | 🔄 | **Before changing any constant.** Every tuned value with the reason it has that value |
| [knowledge/KNOWLEDGE_PARSING_OUTPUT_CONTRACT.md](knowledge/KNOWLEDGE_PARSING_OUTPUT_CONTRACT.md) | 🔄 | You're working either side of the seam |
| [knowledge/KNOWLEDGE_SURFACE_PLAN.md](knowledge/KNOWLEDGE_SURFACE_PLAN.md) | 🔄 | Persistence + endpoints: who owns what, phased tasks, blockers. Approved 2026-09-01 |
| [knowledge/KNOWLEDGE_CONSUMPTION_PLAN.md](knowledge/KNOWLEDGE_CONSUMPTION_PLAN.md) | 💡 | How agents READ the entries (N14). Which agent gets what, and why. **Proposal, nothing built** |
| [knowledge/TABLE_BRANCH_PROPOSAL.md](knowledge/TABLE_BRANCH_PROPOSAL.md) | 💡 | Step 8 — the symptom x cause matrix a shop manual actually holds. Deterministic triples, LLM only to phrase. **Proposal, nothing built** |
| [knowledge/TERM_HIERARCHY_PROPOSAL.md](knowledge/TERM_HIERARCHY_PROPOSAL.md) | 💡 | Step 7 + R7 — parent/child terms and cross-document identity, designed together. **Touches `entity_id`; read before building** |
| [knowledge/KNOWLEDGE_DOMAIN_CONTEXT_V4.md](knowledge/KNOWLEDGE_DOMAIN_CONTEXT_V4.md) | 💡 | What a *real* domain context is. v3's object is a document brief wearing the name. **Proposal, nothing built** |
| [knowledge/KNOWLEDGE_PERSISTENCE_CONTRACT.md](knowledge/KNOWLEDGE_PERSISTENCE_CONTRACT.md) | 💡 | The four-table DDL sent to Harry 2026-09-01. **Pending — no ORM written against it yet** |
| [knowledge/KNOWLEDGE_DOMAIN_CONTEXT_V3.md](knowledge/KNOWLEDGE_DOMAIN_CONTEXT_V3.md) | ✅ | `DomainContext` design reasoning. **Shipped in full 2026-09-02** — the live shape is the output contract §8 |

Roll-up status: DEV_PLAN §0.8 · built state: REPO_STATUS §14 · eval harness: [../eval/knowledge/README.md](../eval/knowledge/README.md)

---

## `plans/` — design plans

| Doc | Status | Note |
|---|---|---|
| [plans/SPINE_V2_PLAN.md](plans/SPINE_V2_PLAN.md) | 🔄 | Approved 2026-07-13; W2+W1 shipped. **W3 deferred — do not start.** Holds the `message_charts` DDL (§4.4) |
| [plans/DS_TOOLS_PLAN.md](plans/DS_TOOLS_PLAN.md) | 💡 | Design study only. **Nothing was built from it**; don't implement without sign-off |

---

## `tools/` — the analytical-tools workstream

The `analyze_*` tool surface: forecast, importance, anomaly, cluster, A/B testing, segmentation,
pareto. Tools are **pure functions, no base class** — the "framework" is `spec_builder` +
`ml/executor` + the shared chart kit.

| Doc | Status | Read it when |
|---|---|---|
| [tools/ANALYTICAL_TOOLS_CONVENTIONS.md](tools/ANALYTICAL_TOOLS_CONVENTIONS.md) | ✅ | **Before writing any tool.** The InputModel → ToolSpec flow, the auto-chart convention, `run_fit`, the three registrations, and the rules that aren't negotiable |

Roll-up status: DEV_PLAN §0.9 · built state: REPO_STATUS §9 · chart envelope: [plans/SPINE_V2_PLAN.md](plans/SPINE_V2_PLAN.md) §4.2

---

## `testing/` — what running the thing actually found

End-to-end dry-runs of the `structured_flow` path against real sources. These runs are
expensive (~170k-290k tokens each) and found twelve bugs that 881 passing unit tests did
not, because each was an interaction between the model's choices and a real data shape.

| Doc | Status | Read it when |
|---|---|---|
| [testing/AGENTIC_DRY_RUN_LOG.md](testing/AGENTIC_DRY_RUN_LOG.md) | 📎 | **Before running a dry-run** (it costs real money, and §6 halves it), and before assuming a green run means a working turn — seven of the twelve bugs produced a confident WRONG answer rather than an error |

Suites + shapes: `eval/chat_sim/run_chat.py` · per-run answers: `eval/chat_sim/results/`

---

## `archive/` — superseded, kept for the record

Nothing here is safe to implement from. Each says what replaced it.

| Doc | Superseded by |
|---|---|
| [archive/API_ENDPOINTS_RESTRUCTURE.md](archive/API_ENDPOINTS_RESTRUCTURE.md) | [../API_CONTRACT_BE_PYTHON.md](../API_CONTRACT_BE_PYTHON.md). The frozen 2026-06-30 snapshot sent to Harry; says "observability" (now traceability) and cites the removed `ENABLE_SLOW_PATH` |
| [archive/CODE_REVIEW_2026-07-23.md](archive/CODE_REVIEW_2026-07-23.md) | DEV_PLAN §0.7 for live **F-n** status. The review body still explains *why* each fix exists |
| [archive/PROJECT_BRD.md](archive/PROJECT_BRD.md) | REPO_STATUS (built state) + API_CONTRACT_BE_PYTHON (contract). Written against `pr/4` |

---

## `reference/` — other repos

| Doc | Note |
|---|---|
| [reference/API_CONTRACT_BE_GOLANG.md](reference/API_CONTRACT_BE_GOLANG.md) | The **Go** Orchestrator's contract. We don't edit that repo; this is a copy and can drift. Verify against the Go source before relying on it. Last synced 2026-07-01 |

---

## Subsystem docs (live beside the code)

`src/*/README.md` — catalog · knowledge_parsing · query · retrieval · security.
`eval/*/README.md` — chat_sim · help · intent · knowledge · planner · readiness.

Eval READMEs are reliable; follow their run and commit conventions exactly. Subsystem READMEs can
lag — code wins.

---

## Conventions

- **Every doc gets a status banner** under its H1, marked `<!-- doc-status -->`. New docs too.
- **Archive, don't delete.** A deleted doc leaves a dead link and no explanation; an archived one
  says what replaced it. (Repo convention: deletion needs Rifqi's explicit sign-off.)
- **Docs are part of the change** — a behaviour, surface or schema change updates the affected doc
  in the same unit of work.