PHASE_1_FOUNDATION.md

PHASE_1_FOUNDATION.md — Phase 1: Project Foundation

RAG-Based Technical Documentation Assistant

---

1. Phase Goal

• Business Goal: Establish a robust, standard project layout and execution runtime that ensures clean separation of concerns, structured configuration loading, and error traceability.
• Technical Goal: Scaffolding the python workspace directory, defining Pydantic settings loading, structured logging, custom domain exceptions, and SQLite database utility.
• Completion Criteria: A developer can launch the workspace, load configuration properties, and execute a shell command to initialize SQL tables idempotently.

---

2. Scope

Included

• Folder structure layout according to FOLDER_STRUCTURE.md.
• Virtual environment configuration files (pyproject.toml, requirements.txt, requirements-dev.txt).
• Environment settings setup utilizing pydantic-settings (loading .env).
• Structured logging configuration (JSON logs wrapper).
• Custom system exceptions hierarchy (BaseException, LLMError, DBError, etc.).
• SQLite connection utility and DDL schema initialization.

Excluded

• FastAPI endpoint routers.
• LangGraph workflow compilation.
• Vector database storage setup.
• ChromaDB indexing or SentenceTransformer models loading.

---

3. Dependencies

• Python 3.11 installed on the system.
• Visual Studio Build Tools (for any local compilation requirements).

---

4. Deliverables

• requirements.txt
• requirements-dev.txt
• pyproject.toml
• app/config.py
• app/core/logging.py
• app/core/exceptions.py
• app/core/database.py

---

5. Sub-Phases

Phase 1.1: Dependency Scaffolding & Setup

• Goal: Create environment configuration files and initialize standard directory folders.
• Tasks:
  1. Create directory tree structure (app/api/routes, app/api/schemas, app/workflow/nodes, app/infrastructure/vector_store, app/infrastructure/embeddings, app/infrastructure/llm, app/infrastructure/document_loader, app/repositories, app/core, app/utils, ingestion, corpus, tests/unit, tests/integration, tests/api, data, chroma_db).
  2. Write requirements.txt listing all baseline packages.
  3. Write requirements-dev.txt listing development tools.
  4. Write pyproject.toml configuration for linters (Ruff/MyPy) and pytest settings.
• Files:
  • requirements.txt
  • requirements-dev.txt
  • pyproject.toml
• Acceptance Criteria: Running pip install works, directories are properly initialized, and config files match expectations.
• Verification: Ensure python environment activates and dependencies resolve.

---

Phase 1.2: Configuration & Logging

• Goal: Load settings from environment variables and setup JSON logs.
• Tasks:
  1. Implement app/config.py using pydantic-settings.
  2. Implement app/core/logging.py with custom formatting for structured logs.
  3. Write .env.example file.
• Files:
  • app/config.py
  • app/core/logging.py
  • .env.example
• Acceptance Criteria: Configurations can be loaded from .env or system environment, defaulting correctly, and logger outputs JSON lines.
• Verification: Check settings outputs using a python test script.

---

Phase 1.3: Custom Exceptions & SQLite Database Setup

• Goal: Define error mappings and build SQLite connection factory to setup databases.
• Tasks:
  1. Implement app/core/exceptions.py mapping domain errors.
  2. Implement app/core/database.py containing connection context and tables setup.
• Files:
  • app/core/exceptions.py
  • app/core/database.py
• Acceptance Criteria: Database initialization script executes idempotently, creating documents and feedback tables with indices.
• Verification: Executing table creation script creates ./data/registry.db and ./data/feedback.db.

---

6. AI Build Prompt (AI_BUILD_PROMPT.md)

# AI Build Prompt: Phase 1 (Foundation)

## Goal
Build the foundation infrastructure for the RAG-Based Technical Documentation Assistant. This includes directory layout, configurations, logging, exceptions, and SQLite storage initialization.

## Files to Create
1. **requirements.txt**: Pin dependencies:
   - fastapi>=0.111.0
   - uvicorn[standard]>=0.30.0
   - langgraph>=0.1.0
   - langchain>=0.2.0
   - langchain-community>=0.2.0
   - chromadb>=0.5.0
   - sentence-transformers>=3.0.0
   - pydantic>=2.7.0
   - pydantic-settings>=2.2.0
   - python-dotenv>=1.0.0
   - groq>=0.9.0
   - openai>=1.30.0
   - google-genai>=0.1.0
   - python-multipart>=0.0.9
   - aiofiles>=23.2.0
   - httpx>=0.27.0
2. **requirements-dev.txt**: pytest, pytest-asyncio, pytest-cov, ruff, black
3. **pyproject.toml**: Configure Ruff, Pytest (asyncio auto mode), MyPy
4. **app/config.py**: Class Settings inheriting from BaseSettings:
   - LLM_PROVIDER (str, default "google")
   - LLM_MODEL (str, default "gemini-2.5-flash")
   - GEMINI_API_KEY (Optional[str])
   - GROQ_API_KEY (Optional[str])
   - OPENAI_API_KEY (Optional[str])
   - EMBEDDING_MODEL (str, default "sentence-transformers/all-MiniLM-L6-v2")
   - CHROMA_PERSIST_DIR (str, default "./chroma_db")
   - TOP_K (int, default 5)
   - MAX_RETRIES (int, default 2)
   - CHUNK_SIZE (int, default 512)
   - CHUNK_OVERLAP (int, default 64)
5. **app/core/logging.py**: Configure structured JSON log output formatting to stdout.
6. **app/core/exceptions.py**: Class hierarchy for custom exceptions:
   - RAGException (Base)
   - LLMProviderError
   - VectorStoreError
   - DatabaseError
   - IngestionError
7. **app/core/database.py**: Connection factory for SQLite. Define schema tables:
   - documents: id (TEXT PK), filename (TEXT), source_url (TEXT), file_hash (TEXT), file_size_bytes (INTEGER), file_type (TEXT), chunk_count (INTEGER), status (TEXT), ingestion_timestamp (TEXT), embedding_model (TEXT), error_message (TEXT).
   - feedback: id (INTEGER PK), feedback_id (TEXT UNIQUE), query (TEXT), answer (TEXT), rating (TEXT), comment (TEXT), sources_used (TEXT), session_id (TEXT), query_type (TEXT), retry_count (INTEGER), response_time_ms (INTEGER), created_at (TEXT).
   Write an idempotent function `initialize_db()` that runs on module load/startup.

## Constraints
- Do not import any non-existent packages.
- Ensure type annotations are strictly followed.
- Use native sqlite3 module for simplicity and to avoid bloated ORM dependencies.

## Acceptance Criteria
- Run a simple script to verify database creation.
- Check that configuration settings parse standard `.env` properties.

---

7. Verification Package

Manual Verification

1. Install all dependencies:

   python -m venv .venv
   .venv\Scripts\activate
   pip install -r requirements.txt -r requirements-dev.txt
   

2. Run database initialization check:

   python -c "from app.core.database import initialize_db; initialize_db(); print('DB Ready')"
   

Expected Results

• The script should execute cleanly, print "DB Ready".
• data/registry.db and data/feedback.db (or a single data/app.db containing both tables) should be created successfully in the folder structure.

Failure Conditions

• SQLite script crashes with SQL syntax errors.
• Config initialization fails because of missing environment variables (ensure optional values are handled correctly).

---

8. Review Gates

• All required files created successfully.
• Requirements pinned with exact or minimum versions.
• Idempotent DB initialization logic verified manually.
• No secrets stored directly in app/config.py.
• Log output formatting verified.
• Python linter commands (ruff check .) return no major issues.