# Document Chat Documentation An **Agentic Retrieval-Augmented Generation (RAG)** system built with **LangGraph**, featuring **parent–child chunking**, **hybrid dense + sparse retrieval**, and Groq-powered chat with a Gradio frontend. Embeddings use ZeroGPU on HF Spaces and CPU locally; no Ollama server is required. The learning notebooks retain their Ollama setup. ## Table of Contents [Quick Start](#quick-start) | [Hugging Face Deployment](#hugging-face-spaces-recommended) | [Architecture Overview](#architecture-overview) | [Project Structure](#project-structure) | [Configuration Guide](#configuration-guide) | [Common Customizations](#common-customizations) | [Observability](#observability) | [Advanced Topics](#advanced-topics) | [Troubleshooting](#troubleshooting) --- ## Quick Start ### Installation From the repository root, use `uv` (Windows PowerShell): ```powershell uv venv --python 3.13 .venv uv pip install --python .venv\Scripts\python.exe --torch-backend cpu -r requirements.txt Copy-Item project\.env.example project\.env ``` Set `GROQ_API_KEY` in `project/.env`. Get a key from https://console.groq.com/keys. Never commit `.env` or expose the key in frontend code. The root `requirements.txt` is the standalone runtime dependency manifest. HF Gradio mounts only this file during installation, so it must not include other local requirement files. Notebook/evaluation tools are in `requirements-notebooks.txt`. The optional `project/requirements.txt` redirects local installs to the root manifest; Docker installs the root file directly. ### Running the Application Start the Gradio interface locally: ```powershell uv run --no-project --python .venv\Scripts\python.exe project\app.py ``` The application will be available at `http://localhost:7860` (default Gradio port). > Chat memory is isolated per browser session and kept in memory, so it is lost > on restart. Documents are a **shared library**, not private per-user storage: > anyone with app access can query, upload, or clear them. Use a private Space > or set both `GRADIO_AUTH_USERNAME` and `GRADIO_AUTH_PASSWORD` as secrets. > This also protects your Groq quota. Run one app process: local Qdrant cannot > be opened by multiple workers at the same path. ### Prerequisites - Python 3.11+ - A Groq API key; outbound access to Groq and Hugging Face model downloads --- ## Architecture Overview This system implements an advanced RAG pipeline with the following key features: - **Parent-Child Chunking**: Documents are split into small child chunks (for precise retrieval) linked to larger parent chunks (for rich context) - **Hybrid Search**: Combines dense embeddings and sparse (BM25) retrieval for optimal results - **LangGraph Agent**: Orchestrates query rewriting, retrieval, and response generation - **Compact Chat Memory**: Rewrites follow-ups using both a rolling summary and bounded recent conversation history - **Clarification Memory**: Preserves an unresolved query across human-in-the-loop pauses, combines its clarifications, then discards those temporary replies after the answer - **Document-Aware Query Analysis**: Receives the current uploaded filenames on every turn, resolves single-document references, and allows new requests to replace pending clarifications - **Document Inventory Answers**: Lists uploaded filenames from the library without vector retrieval or a GPU allocation - **Groq Chat**: Tool calling, structured query analysis, and streamed responses through `ChatGroq` - **Vector Storage**: Uses Qdrant for efficient similarity search ### Data Flow ``` PDF → Markdown Conversion → Parent/Child Chunking → Vector Indexing → Agent Retrieval → LLM Response ``` --- ## Project Structure ### Entry Point & Configuration | File | Purpose | |------|---------| | `project/app.py` | Application entry point, launches Gradio UI | | `project/config.py` | **Central configuration hub** - edit this for provider/model/chunking changes | | `project/utils.py` | PDF conversion and cached context-token estimation with an offline-safe fallback | | `project/document_chunker.py` | Parent/child splitting logic with cleaning and merging rules | | `Dockerfile` | uv-based CPU runtime for the backend and Gradio frontend | ### Core System | File | Purpose | |------|---------| | `project/core/rag_system.py` | System bootstrap - creates managers and compiles LangGraph agent | | `project/core/document_manager.py` | Document ingestion pipeline (convert, chunk, index) | | `project/core/chat_interface.py` | Streams the aggregated answer while separating query analysis and tool activity from internal node output | | `project/core/observability.py` | Optional Langfuse tracing — callback handler lifecycle | ### Database Layer | File | Purpose | |------|---------| | `project/db/vector_db_manager.py` | Qdrant client wrapper with inference-free schema validation | | `project/db/dense_embeddings.py` | MiniLM embeddings with ZeroGPU inference on Spaces and CPU inference locally | | `project/db/parent_store_manager.py` | File-backed storage for parent chunks | ### RAG Agent (LangGraph) | File | Purpose | |------|---------| | `project/rag_agent/graph.py` | Graph builder and compilation logic | | `project/rag_agent/graph_state.py` | Shared and per-agent state, including rolling memory, pending clarification, and answer accumulation/reset logic | | `project/rag_agent/nodes.py` | Node implementations (summarize, rewrite, agent execution, aggregate) | | `project/rag_agent/edges.py` | Conditional edge routing logic (e.g., routing based on query clarity) | | `project/rag_agent/tools.py` | Retrieval tools (`search_child_chunks`, `retrieve_parent_chunks`) | | `project/rag_agent/prompts.py` | System prompts for agent behavior | | `project/rag_agent/schemas.py` | Structured output schemas (Pydantic models) | ### User Interface | File | Purpose | |------|---------| | `project/ui/css.py` | Custom CSS styling for the Gradio interface | | `project/ui/gradio_app.py` | Gradio UI implementation with document upload and chat | --- ## Configuration Guide All primary settings are in `project/config.py`. Key parameters: ### Directory Configuration ```python MARKDOWN_DIR = "markdown_docs" # Storage for converted PDF → Markdown files PARENT_STORE_PATH = "parent_store" # File-backed storage for parent chunks QDRANT_DB_PATH = "qdrant_db" # Local Qdrant vector database path ``` ### Qdrant Configuration ```python CHILD_COLLECTION = "document_child_chunks" # Collection name for child chunks SPARSE_VECTOR_NAME = "sparse" # Named sparse vector field (BM25) ``` ### Model Configuration ```python # Default: single model configuration DENSE_MODEL = "sentence-transformers/all-MiniLM-L6-v2" SPARSE_MODEL = "Qdrant/bm25" LLM_MODEL = os.environ.get("GROQ_MODEL", "openai/gpt-oss-20b") JUDGE_MODEL = "ministral-3:3b-instruct-2512-q8_0" LLM_TEMPERATURE = 0 # 0 = deterministic, 1 = creative LLM_SEED = 42 LLM_TIMEOUT = 60 LLM_MAX_RETRIES = 2 ``` `GROQ_API_KEY` is required at startup. `DENSE_MODEL` and `DATA_DIR` can also be set as environment variables. `JUDGE_MODEL` remains the local Ollama judge for the evaluation notebook, not the deployed app. Query analysis explicitly uses function calling, so it does not require Groq's model-specific JSON Schema mode. ### Retrieval Configuration ```python RETRIEVAL_SCORE_THRESHOLD = 0.4 # Lower = more recall, higher = more precision DEFAULT_RETRIEVAL_K = 7 # Default number of child chunks used by retrieval and evaluation CHILD_CHUNK_SEPARATOR = "\n\n\n\n" # Keeps ranked child results separable for evaluation ``` ### Agent Configuration ```python # Hard limits to prevent infinite loops MAX_TOOL_CALLS = 8 # Maximum tool calls per agent run MAX_ITERATIONS = 10 # Maximum agent loop iterations GRAPH_RECURSION_LIMIT = 50 # Maximum number of steps before hitting a stop condition MAIN_HISTORY_MESSAGES_TO_KEEP = 4 # Raw messages retained after each answer; minimum 2 # Context compression thresholds BASE_TOKEN_THRESHOLD = 2000 # Initial token threshold for compression TOKEN_GROWTH_FACTOR = 0.9 # Multiplier applied after each compression ``` ### Terminal Execution Logging ```python EXECUTION_LOGGING_ENABLED = False # Print graph steps, state previews, tool calls, and outputs EXECUTION_LOG_MAX_CHARS = 1200 # Maximum characters shown for long values EXECUTION_LOG_USE_COLOR = True # Use ANSI colors in the terminal ``` ### Text Splitter Configuration ```python CHILD_CHUNK_SIZE = 500 # Size of chunks used for retrieval CHILD_CHUNK_OVERLAP = 100 # Overlap between chunks (prevents context loss) MIN_PARENT_SIZE = 2000 # Target minimum; very short documents may remain smaller MAX_PARENT_SIZE = 4000 # Maximum parent chunk size # Markdown header splitting strategy HEADERS_TO_SPLIT_ON = [ ("#", "H1"), ("##", "H2"), ("###", "H3") ] ``` ### Langfuse Observability (Optional) ```python import os LANGFUSE_ENABLED = os.environ.get("LANGFUSE_ENABLED", "false").lower() == "true" LANGFUSE_PUBLIC_KEY = "" # From your Langfuse project settings LANGFUSE_SECRET_KEY = "" # From your Langfuse project settings LANGFUSE_BASE_URL = "http://localhost:3000" # Langfuse Cloud or self-hosted URL ``` --- ## Common Customizations ### 1. Switching LLM Provider (Single Provider) > **Performance Note:** LLMs with 8B+ parameters typically offer superior reasoning, context comprehension, and response quality compared to smaller models. This applies to both proprietary and open-source models, as long as they **support native tool/function calling,** which is required for agentic RAG workflows. If you want to switch from Groq to another provider (e.g., Google Gemini), follow these steps: **Step 1:** Install the provider's SDK ```bash uv pip install langchain-google-genai ``` **Step 2:** Set environment variable ```bash export GOOGLE_API_KEY="your-google-key" ``` **Step 3:** Update `project/config.py` ```python LLM_MODEL = "gemini-2.5-pro" LLM_TEMPERATURE = 0 LLM_SEED = 42 ``` **Step 4:** Modify `project/core/rag_system.py` Replace: ```python llm = ChatGroq(model=config.LLM_MODEL, temperature=config.LLM_TEMPERATURE, timeout=config.LLM_TIMEOUT, max_retries=config.LLM_MAX_RETRIES, model_kwargs={"seed": config.LLM_SEED}) ``` With: ```python from langchain_google_genai import ChatGoogleGenerativeAI llm = ChatGoogleGenerativeAI(model=config.LLM_MODEL, temperature=config.LLM_TEMPERATURE) ``` ### 2. Optional Multi-Provider Configuration The application ships with a Groq implementation. The following is an optional customization example, not the shipped configuration. Remove the Groq-only startup key check if implementing a different provider. This approach allows you to maintain multiple provider configurations and switch between them easily. **Step 1:** Install required SDKs ```bash uv pip install langchain-openai langchain-anthropic langchain-google-genai ``` **Step 2:** Set environment variables ```bash export OPENAI_API_KEY="your-openai-key" export ANTHROPIC_API_KEY="your-anthropic-key" export GOOGLE_API_KEY="your-google-key" ``` **Step 3:** Update `project/config.py` with multi-provider configuration ```python # --- Multi-Provider LLM Configuration --- LLM_CONFIGS = { "ollama": { "model": "granite4.1:8b", "url":"http://localhost:11434", "temperature": 0 }, "openai": { "model": "gpt-5.2", "temperature": 0 }, "anthropic": { "model": "claude-sonnet-4-6", "temperature": 0 }, "google": { "model": "gemini-2.5-flash", "temperature": 0 } } # Switch providers by changing this single line ACTIVE_LLM_CONFIG = "ollama" ``` **Step 4:** Modify `project/core/rag_system.py` in the `initialize()` method Replace the existing LLM initialization with: ```python def initialize(self): self.vector_db.create_collection(self.collection_name) collection = self.vector_db.get_collection(self.collection_name) # Load active configuration active_config = config.LLM_CONFIGS[config.ACTIVE_LLM_CONFIG] model = active_config["model"] temperature = active_config["temperature"] if config.ACTIVE_LLM_CONFIG == "ollama": from langchain_ollama import ChatOllama llm = ChatOllama(model=model, temperature=temperature, base_url=active_config["url"]) elif config.ACTIVE_LLM_CONFIG == "openai": from langchain_openai import ChatOpenAI llm = ChatOpenAI(model=model, temperature=temperature) elif config.ACTIVE_LLM_CONFIG == "anthropic": from langchain_anthropic import ChatAnthropic llm = ChatAnthropic(model=model, temperature=temperature) elif config.ACTIVE_LLM_CONFIG == "google": from langchain_google_genai import ChatGoogleGenerativeAI llm = ChatGoogleGenerativeAI(model=model, temperature=temperature) else: raise ValueError(f"Unsupported LLM provider: {config.ACTIVE_LLM_CONFIG}") # Continue with tool and graph initialization tools = ToolFactory(collection).create_tools() self.agent_graph = create_agent_graph(llm, tools) ``` **Switching Providers:** Simply change `ACTIVE_LLM_CONFIG` in `config.py`: ```python ACTIVE_LLM_CONFIG = "google" # Switch to Gemini Pro # ACTIVE_LLM_CONFIG = "anthropic" # Or Claude Sonnet # ACTIVE_LLM_CONFIG = "openai" # Or GPT-4o ``` --- **Provider Reference Table:** | Provider | Environment Variable | Import Statement | Example Models | |----------|---------------------|------------------|----------------| | Groq | `GROQ_API_KEY` | `from langchain_groq import ChatGroq` | `openai/gpt-oss-20b` | | OpenAI | `OPENAI_API_KEY` | `from langchain_openai import ChatOpenAI` | `gpt-5.2`, `gpt-5-mini` | | Anthropic | `ANTHROPIC_API_KEY` | `from langchain_anthropic import ChatAnthropic` | `claude-opus-4-6`, `claude-sonnet-4-6` | | Google | `GOOGLE_API_KEY` | `from langchain_google_genai import ChatGoogleGenerativeAI` | `gemini-2.5-pro`, `gemini-2.5-flash` | | Ollama | None (local) | `from langchain_ollama import ChatOllama` | `granite4.1:8b`, `llama3.1:8b-instruct-q6_K` | --- ### 3. Changing Embedding Models **Why change?** Trade-offs between speed, cost, and quality. **Step 1:** Update `project/config.py` ```python # Default: Compact model (ZeroGPU on HF, CPU locally) DENSE_MODEL = "sentence-transformers/all-MiniLM-L6-v2" # Example: Alternative Sentence Transformers model # DENSE_MODEL = "sentence-transformers/all-mpnet-base-v2" # Example: Gemma embeddings (Google's open model) # DENSE_MODEL = "google/embeddinggemma-300m" # Optional: Qwen embeddings (Alibaba's multilingual model) # DENSE_MODEL = "Qwen/Qwen3-Embedding-0.6B" SPARSE_MODEL = "Qdrant/bm25" # Usually no need to change ``` **Step 2:** Re-index your documents ⚠️ **Important:** Changing embeddings requires re-indexing. In the educational UI, use **Clear All**, restart if the old collection prevents startup, then upload the documents again. **Implementation Details** (in `project/db/vector_db_manager.py`): ```python from langchain_huggingface import HuggingFaceEmbeddings from langchain_qdrant import FastEmbedSparse import config from db.dense_embeddings import DenseEmbeddings self.__dense_embeddings = DenseEmbeddings(model_name=config.DENSE_MODEL) self.__sparse_embeddings = FastEmbedSparse(model_name=config.SPARSE_MODEL) ``` **Popular Embedding Models:** | Model | Context Size | Vector Dimension | Speed | Quality | Use Case | |-------|--------------|------------------|-------|---------|----------| | all-MiniLM-L6-v2 | 256 tokens | 384 | Fast | Good | Default compact embedding model | | all-mpnet-base-v2 | 512 tokens | 768 | Medium | Excellent | High-accuracy semantic search | | bge-large-en-v1.5 | 512 tokens | 1024 | Slow | Best | Production-grade retrieval on GPU | | google/embeddinggemma-300m | 2048 tokens | 768 | Fast | Very Good | Lightweight, efficient multilingual retrieval | | Qwen/Qwen3-Embedding-0.6B | 32768 tokens | 1024 | Medium | Excellent | Optional multilingual/code retrieval model | | Qwen/Qwen3-Embedding-8B | 32768 tokens | 4096 | Very Slow | Excellent / SOTA | Heavy GPU-oriented retrieval experiments | --- ### 4. Adjusting Chunking Strategy **Why adjust?** Balance between retrieval precision and context richness. > Before re-indexing, inspect Markdown output and chunk boundaries to verify the document structure. **Step 1:** Update chunk sizes in `project/config.py` ```python # For short, factual queries (e.g., technical documentation) CHILD_CHUNK_SIZE = 300 CHILD_CHUNK_OVERLAP = 60 MIN_PARENT_SIZE = 1500 MAX_PARENT_SIZE = 8000 # For narrative or contextual queries (e.g., legal documents) # CHILD_CHUNK_SIZE = 800 # CHILD_CHUNK_OVERLAP = 150 # MIN_PARENT_SIZE = 3000 # MAX_PARENT_SIZE = 15000 ``` **Step 2 (Optional):** Replace the splitter in `project/document_chunker.py` **Default (Character-based):** ```python from langchain_text_splitters import RecursiveCharacterTextSplitter self.__child_splitter = RecursiveCharacterTextSplitter( chunk_size=config.CHILD_CHUNK_SIZE, chunk_overlap=config.CHILD_CHUNK_OVERLAP ) ``` **Alternative (Sentence-aware):** ```python from langchain_text_splitters import SentenceTransformersTokenTextSplitter self.__child_splitter = SentenceTransformersTokenTextSplitter( chunk_size=config.CHILD_CHUNK_SIZE, chunk_overlap=config.CHILD_CHUNK_OVERLAP ) ``` **Step 3:** Re-run ingestion Use **Clear All** in the Gradio interface, then upload the documents again. Existing documents are intentionally skipped to avoid accidental duplicates. **Chunking Guidelines:** > ⚠️ **Disclaimer:** These are empirical guidelines. Optimal sizes depend on: > - **Child chunk** → embedding model's context window (e.g. 256 tokens for all-MiniLM-L6-v2, 512 for bge-large-en-v1.5): child size should not exceed it > - **Parent chunk** → generative model's context window (e.g. 8K, 32K, 128K tokens): parent must fit within the context sent to the LLM alongside the query > > Always validate values empirically on your own corpus. The chunker enforces `MAX_PARENT_SIZE`, deduplicates merged header metadata, and rebalances small neighboring chunks when possible. A document shorter than `MIN_PARENT_SIZE` remains a single smaller chunk. | Document Type | Child Size | Parent Size | Reasoning | |---------------|-----------|-------------|-----------| | Technical Docs | 300-500 | 2000-4000 | Precise lookups, code snippets | | Legal Contracts | 600-1000 | 5000-15000 | Context-heavy, definitions | | Research Papers | 400-600 | 3000-8000 | Balance of precision and context | | FAQs / Knowledge Base | 200-400 | 1500-4000 | Short, focused answers | --- ### 5. Agent Configuration Tune agent behavior in `project/config.py`: ```python # Hard limits to prevent infinite loops MAX_TOOL_CALLS = 8 # Maximum tool calls per agent run MAX_ITERATIONS = 10 # Maximum agent loop iterations GRAPH_RECURSION_LIMIT = 50 # Maximum number of steps before hitting a stop condition MAIN_HISTORY_MESSAGES_TO_KEEP = 4 # Raw messages retained after each answer; minimum 2 # Context compression thresholds BASE_TOKEN_THRESHOLD = 2000 # Initial token threshold for compression TOKEN_GROWTH_FACTOR = 0.9 # Multiplier applied after each compression ``` | Parameter | Effect | |-----------|--------| | `MAX_TOOL_CALLS` | Maximum cumulative tool calls requested; a request that would exceed it is not executed | | `MAX_ITERATIONS` | Maximum LLM reasoning iterations; a final answer at the boundary is still accepted | | `GRAPH_RECURSION_LIMIT` | Increase for complex [graphs](https://docs.langchain.com/oss/python/langgraph/errors/GRAPH_RECURSION_LIMIT) | | `BASE_TOKEN_THRESHOLD` | Delay compression by increasing this value | | `TOKEN_GROWTH_FACTOR` | Lower values compress more aggressively | --- ## Observability Optional tracing with [Langfuse](https://langfuse.com) captures every LLM call, tool invocation, and graph transition. It is useful for debugging agent behavior, tracking costs, and evaluating retrieval quality. ### Enabling Langfuse 1. Sign up on [Langfuse Cloud](https://cloud.langfuse.com/), create an organization, then create a project and generate API keys from the project settings. 2. Set environment variables (or copy `.env.example` to `.env`): ```bash export LANGFUSE_ENABLED=true export LANGFUSE_PUBLIC_KEY=pk-lf-... export LANGFUSE_SECRET_KEY=sk-lf-... export LANGFUSE_BASE_URL=https://cloud.langfuse.com # or your self-hosted URL ``` 3. Run the app normally. Traces appear in your [Langfuse dashboard](https://cloud.langfuse.com/). To disable tracing, set `LANGFUSE_ENABLED=false` or leave the variables unset. The application runs identically either way. For additional details on integrating Langfuse with LangChain or LangGraph, see the official [documentation](https://docs.langchain.com/oss/python/integrations/providers/langfuse). ### What gets traced | Component | Traced operations | |-----------|-------------------| | Graph nodes | `summarize_history`, `rewrite_query`, `orchestrator`, `compress_context`, `fallback_response`, `aggregate_answers` | | Tools | `search_child_chunks`, `retrieve_parent_chunks` (arguments + results) | | Structured output | `QueryAnalysis` parsing in the rewrite step | | Subgraph fan-out | Parallel agent invocations via `Send()` | ### Hosting options - **Langfuse Cloud** — sign up at [cloud.langfuse.com](https://cloud.langfuse.com) and check the current plan limits there. - **Self-hosted** — MIT-licensed, deploy with Docker Compose. See the [official self-hosting docs](https://langfuse.com/self-hosting). For a conceptual tracing guide and platform context, see [`notebooks/observability.ipynb`](../notebooks/observability.ipynb). --- ## Advanced Topics ### Customizing the RAG Agent **Location:** `project/rag_agent/` **Add/Remove Nodes:** Edit `graph.py` and `nodes.py` Example: Adding a fact-checking node ```python # In nodes.py def fact_check_node(state): # Your fact-checking logic return {"fact_checked": True} # In graph.py builder.add_node("fact_check", fact_check_node) builder.add_edge("retrieve", "fact_check") ``` **Modify Conditional Routing:** Edit `edges.py` to change graph flow logic Example from the system - routing based on query clarity: ```python def route_after_rewrite(state: State) -> Literal["request_clarification", "agent"]: """Routes to human input if question unclear, otherwise processes all rewritten queries""" if not state.get("questionIsClear", False): return "request_clarification" else: # Fan-out: send each rewritten question to parallel processing return [ Send("agent", {"question": query, "question_index": idx, "messages": []}) for idx, query in enumerate(state["rewrittenQuestions"]) ] ``` This pattern allows the agent to either request clarification from the user or fan-out multiple query variations for parallel retrieval. **Modify Prompts:** Edit `prompts.py` to change agent behavior and response style **Add Custom Tools:** Extend `tools.py` with new retrieval strategies or external integrations ### Replacing Storage Backends **Vector Database:** - Default: Local Qdrant - Alternatives: Remote Qdrant Cloud, Pinecone, Weaviate - Edit: `project/db/vector_db_manager.py` **Parent Store:** - Default: JSON file - Alternatives: PostgreSQL, MongoDB, S3 - Edit: `project/db/parent_store_manager.py` ### Extending the UI **Location:** `project/ui/gradio_app.py` Add runtime settings, admin panels, or analytics: ```python with gr.Accordion("Advanced Settings", open=False): provider_dropdown = gr.Dropdown( choices=["openai", "anthropic", "google", "ollama"], label="LLM Provider" ) ``` ### Docker Deployment The root `Dockerfile` installs only runtime dependencies using uv and CPU PyTorch. It serves the existing Gradio frontend and backend together; there is no separate frontend build or Ollama process. MiniLM reduces memory use, but PyTorch and PDF indexing still need more than a 512 MB host. If selecting Qwen embeddings instead, allow several GB of RAM; 8 GB is a starting budget. Model weights download on first startup, so the initial start takes longer. #### Build and Run ```powershell docker build -t agentic-rag . docker run --name rag-assistant --env-file project\.env -p 7860:7860 agentic-rag ``` For persistent documents and indexes: ```powershell docker run --name rag-assistant --env-file project\.env -e DATA_DIR=/data -v rag-data:/data -p 7860:7860 agentic-rag ``` Once running, open `http://localhost:7860`. ### Hugging Face Spaces (Recommended) 1. Create a **Gradio Space** and select **ZeroGPU Free** if offered for your account. Do not select Docker. The root README already specifies `sdk: gradio`, `app_file: project/app.py`, the Gradio version, and Python 3.12.12 (supported by ZeroGPU). 2. Upload the root `README.md`, `requirements.txt`, `LICENSE`, and the full `project` folder (including `project/requirements.txt`). Do not upload `.env`, `.venv`, local documents, or local database files. 3. Under **Settings > Variables and secrets**, add `GROQ_API_KEY` as a **secret**, set `GROQ_MODEL=openai/gpt-oss-20b` as a variable (or remove an old override to use this default), and add both authentication values as secrets if the Space is public. 4. Wait for the build and first model download, then open the **App** tab. Upload a small Markdown/PDF file under **Documents**, click **Add Documents**, and ask a question under **Chat**. The public Gradio API is named `/chat`. Document-management callbacks are UI-only (`api_visibility="private"`), so API clients do not select the file upload callback when sending text. This is not per-user document isolation. After a rebuild, hard-refresh open app tabs to discard stale event IDs and queue sessions. A queue-stream 404 can mean that its session no longer exists; it is not evidence of a missing document or a Groq failure. The app loads MiniLM on **CUDA** at startup when HF sets `SPACES_ZERO_GPU=true`. Dense embedding inference runs inside `@spaces.GPU(duration=30)`, both during document indexing and query retrieval. Only texts and embedding results cross the worker boundary; local Qdrant writes and Groq calls stay outside GPU tasks. Startup obtains the vector dimension from model metadata and validates the Qdrant schema without making an inference request. ZeroGPU requires a registered GPU function during startup; a CPU-only app is not sufficient on this hardware tier. Embedding requests consume the visitor's GPU quota and may queue. Large indexing jobs can require multiple allocations. CPU embedding inference is retained for local and non-ZeroGPU deployments. Verify first startup and indexing in the Space logs. The UI is branded **Document Chat**, with a teal-accented header, a compact chat panel, and the same **Documents** and **Chat** workflow. Per the [ZeroGPU documentation](https://huggingface.co/docs/hub/spaces-zerogpu), free personal accounts in good standing (verified email and older than 30 days) can host up to two ZeroGPU Spaces. Availability is account-dependent. Gradio Spaces manage dependency installation from root `requirements.txt`; uv remains the tool for local installation and the optional Docker build. Documents and indexes use ephemeral local disk and may be lost on restart/rebuild. Chat memory is always in memory. Do not use this demo as durable storage. `DATA_DIR` can point to an existing writable mount on hosts that provide one, but does not provision persistence. #### Render (Alternative) Create a **Docker Web Service** using the root Dockerfile with the repository root as the build context. Set `GROQ_API_KEY` as a secret. The app honors Render's `PORT` and listens on `0.0.0.0` in Docker. Choose a paid instance with enough RAM for the embedding model, and attach a disk at `/data` with `DATA_DIR=/data` if documents must survive deploys. Use one instance/worker. The app defaults to `DENSE_MODEL=sentence-transformers/all-MiniLM-L6-v2`; this trades retrieval quality for lower memory use compared with Qwen. Render's free instances sleep and lose local files; do not rely on them for this model-heavy app. See [free-service limits](https://render.com/docs/free). #### Vercel Not recommended for the full app: it depends on a long-running Gradio server, in-memory LangGraph checkpoints, local Qdrant files, and model downloads. Vercel can host a future separate JavaScript frontend, but the current Python service should remain on Spaces or Render. --- ## Troubleshooting | Issue | Cause | Solution | |-------|-------|----------| | "Model not found" error | Incorrect model name for provider | Verify `LLM_MODEL` matches provider's API (e.g., `gpt-4o-mini` not `gpt4-mini`) | | Low-quality retrieval results | Poor embedding model or chunk configuration | Re-index with better embeddings or adjust chunk sizes | | Vector size mismatch | Existing Qdrant collection was built with a different embedding model | Clear the collection and re-index documents after changing `DENSE_MODEL` | | Slow response times | Large embedding model or high `top_k` value | Use smaller embedding models (e.g., all-MiniLM-L6-v2) or reduce `top_k` in retrieval tools | | API rate limits exceeded | Too many requests to external provider | Add retry logic with exponential backoff or switch to local Ollama models | | Out of memory errors | Large document set or embedding model | Use smaller embeddings, reduce batch size, or enable GPU acceleration | | Empty retrieval results | Collection not indexed or wrong collection name | Verify documents are uploaded and `CHILD_COLLECTION` name matches in config | | Import errors after provider switch | Missing SDK installation | Install required package: `pip install langchain-{provider}` | | Inconsistent answers across runs | High temperature setting | Set `LLM_TEMPERATURE = 0` in config for deterministic responses |