PHASE_2_INGESTION.md

# PHASE_2_INGESTION.md — Phase 2: Document Ingestion Pipeline
## RAG-Based Technical Documentation Assistant

---

## 1. Phase Goal

*   **Business Goal**: Process and store technical source documents so they are available for vector search queries.
*   **Technical Goal**: Create loader utilities for local files and URLs, construct a recursive character text splitter, build local sentence-transformer and ChromaDB vector store wrappers, and implement a CLI seed script.
*   **Completion Criteria**: Running the ingestion script loads all test markdown documents inside the corpus directory, generates embeddings, stores them inside ChromaDB, and registers them in SQLite database.

---

## 2. Scope

### Included
*   Abstract interfaces for loaders, embeddings, and vector stores.
*   Local Markdown/PDF/TXT loader and remote URL fetch loader.
*   Recursive splitting utility ensuring code-block integrity.
*   Embedding model adapter loading `sentence-transformers/all-MiniLM-L6-v2` locally.
*   ChromaDB vector store adapter configured for persistent database operations.
*   Document registration repository (`app/repositories/document_repository.py`).
*   Ingestion manager service orchestrating the process.
*   Seed script `ingestion/ingest_corpus.py`.
*   Standard markdown corpus documents (`fastapi_tutorial.md`, `pydantic_v2.md`, `langgraph_concepts.md`, `langchain_tools.md`).

### Excluded
*   LangGraph nodes and StateGraph assembly.
*   FastAPI endpoints.
*   Feedback storage interactions.

---

## 3. Dependencies

*   Phase 1 completed successfully.
*   Database tables initialized.

---

## 4. Deliverables

*   `app/infrastructure/vector_store/base.py`
*   `app/infrastructure/vector_store/chroma.py`
*   `app/infrastructure/embeddings/base.py`
*   `app/infrastructure/embeddings/sentence_transformers.py`
*   `app/infrastructure/document_loader/base.py`
*   `app/infrastructure/document_loader/file_loader.py`
*   `app/infrastructure/document_loader/url_loader.py`
*   `app/repositories/document_repository.py`
*   `app/services/ingestion_service.py`
*   `app/utils/chunking.py`
*   `app/utils/hashing.py`
*   `ingestion/ingest_corpus.py`
*   `corpus/` (containing target markdown files)

---

## 5. Sub-Phases

### Phase 2.1: Infrastructure Adapters (Embeddings & Vector Store)
*   **Goal**: Create wrappers around local sentence transformers and ChromaDB.
*   **Tasks**:
    1. Define base interfaces for embedding model and vector store.
    2. Write sentence-transformer implementation loading the model locally.
    3. Write ChromaDB vector store adapter. Configure PersistentClient with similarity metric `cosine`.
*   **Files**:
    - `app/infrastructure/embeddings/base.py`
    - `app/infrastructure/embeddings/sentence_transformers.py`
    - `app/infrastructure/vector_store/base.py`
    - `app/infrastructure/vector_store/chroma.py`
*   **Acceptance Criteria**: Running a simple python script instantiates the embedding model and performs local cosine similarity searches using ChromaDB.
*   **Verification**: Embed a single text string and verify the resulting vector dimension is 384.

---

### Phase 2.2: Document Loaders & Splitters
*   **Goal**: Extract raw text from files or URLs and partition them into semantically consistent chunks.
*   **Tasks**:
    1. Define base interface for document loaders.
    2. Write local file loader that parses `.md`, `.txt`, `.html`, `.pdf`.
    3. Write remote URL loader that fetches and parses web content.
    4. Write `app/utils/chunking.py` utilizing `RecursiveCharacterTextSplitter` with separators prioritizing code fences and paragraph boundaries.
    5. Write `app/utils/hashing.py` computing SHA256 checksums to detect duplicate uploads.
*   **Files**:
    - `app/infrastructure/document_loader/base.py`
    - `app/infrastructure/document_loader/file_loader.py`
    - `app/infrastructure/document_loader/url_loader.py`
    - `app/utils/chunking.py`
    - `app/utils/hashing.py`
*   **Acceptance Criteria**: Text chunks respect boundary constraints and do not break code formatting blocks. Duplicate files are identified by their SHA256 hashes.
*   **Verification**: Parse a test file and assert the chunk size and counts match criteria.

---

### Phase 2.3: Ingestion Service & CLI Runner
*   **Goal**: Coordinate loading, chunking, embedding, database storage, and registry transactions.
*   **Tasks**:
    1. Write document database repository (`app/repositories/document_repository.py`).
    2. Write orchestration manager service (`app/services/ingestion_service.py`) tracking task states.
    3. Write corpus runner CLI `ingestion/ingest_corpus.py` loading target documents.
*   **Files**:
    - `app/repositories/document_repository.py`
    - `app/services/ingestion_service.py`
    - `ingestion/ingest_corpus.py`
*   **Acceptance Criteria**: CLI execution indexes all markdown docs in the corpus directory, inserts meta records in SQLite, and writes chunks to ChromaDB.
*   **Verification**: Run CLI pipeline and run verification assertions.

---

## 6. AI Build Prompt (`AI_BUILD_PROMPT.md`)

```markdown
# AI Build Prompt: Phase 2 (Ingestion)

## Goal
Implement the document ingestion pipeline. This includes loading, chunking, embedding, vector store storage, duplicate checking, and catalog registry.

## Files to Create/Modify
- **app/infrastructure/embeddings/base.py**: Abstract class EmbeddingModelBase.
- **app/infrastructure/embeddings/sentence_transformers.py**: Implements EmbeddingModelBase using local `sentence-transformers/all-MiniLM-L6-v2`.
- **app/infrastructure/vector_store/base.py**: Abstract class VectorStoreBase.
- **app/infrastructure/vector_store/chroma.py**: Implements VectorStoreBase using `chromadb.PersistentClient`.
- **app/infrastructure/document_loader/base.py**: Abstract class DocumentLoaderBase.
- **app/infrastructure/document_loader/file_loader.py**: File loader supporting `.md`, `.txt`, `.html` (and optionally simple `.pdf` parsing).
- **app/infrastructure/document_loader/url_loader.py**: URL loader fetching HTML content using `httpx` and stripping tags using `BeautifulSoup` (or regex).
- **app/utils/chunking.py**: Function `split_text(text: str, chunk_size: int, overlap: int) -> List[str]` wrapping `RecursiveCharacterTextSplitter`.
- **app/utils/hashing.py**: Hash helper: `calculate_sha256(content: bytes) -> str`.
- **app/repositories/document_repository.py**: Database operations on `documents` table: `insert_document()`, `get_document_by_hash()`, `list_documents()`, `delete_document()`.
- **app/services/ingestion_service.py**: Ingestion manager coordinating document loaders, splitting, hashing, embedding generations, ChromaDB inserts, and SQLite registrations. Handles duplicate checks.
- **ingestion/ingest_corpus.py**: Python CLI script that loops through the files in `corpus/` and invokes IngestionService to seed the vector store.

## Constraints
- Ensure ChromaDB persistently writes to config property `CHROMA_PERSIST_DIR`.
- Embedding models must load locally without making online network calls during queries.
- Recursive character text splitter should use separators `["\n\n", "\n", "```", ".", " ", ""]` to keep code blocks intact.

## Acceptance Criteria
- Run a verification script verifying indexed files count in SQLite database and vectors count in ChromaDB matches.
```

---

## 7. Verification Package

### Manual Verification
1. Setup seed corpus files in `corpus/` folder.
2. Run ingestion command:
   ```bash
   python ingestion/ingest_corpus.py
   ```
3. Run verification check query:
   ```bash
   python -c "import sqlite3; conn = sqlite3.connect('data/registry.db'); print(conn.cursor().execute('SELECT filename, chunk_count, status FROM documents').fetchall())"
   ```

### Expected Results
*   The ingest runner prints indexing status for all documents.
*   The check query output displays a list containing filenames, their respective chunk counts, and `indexed` status.
*   ChromaDB directory `./chroma_db` is created containing index assets.

### Failure Conditions
*   ChromaDB query fails with database mismatch errors.
*   Duplicate files are ingested twice instead of being skipped.

---

## 8. Review Gates

- [ ] local sentence-transformer models load without network errors.
- [ ] Document hash deduplication handles uploads correctly.
- [ ] Markdown files parsed and chunk size remains within limits.
- [ ] SQLite registry records track files states.
- [ ] Clean log records printed during chunking and vector storage.