agentic-rag / project /README.md
devank2000's picture
fix2
3366ea9 verified
|
Raw History Blame Contribute Delete
30.3 kB

A newer version of the Gradio SDK is available: 6.32.0

Upgrade

Document Chat Documentation

An Agentic Retrieval-Augmented Generation (RAG) system built with LangGraph, featuring parent–child chunking, hybrid dense + sparse retrieval, and Groq-powered chat with a Gradio frontend. Embeddings use ZeroGPU on HF Spaces and CPU locally; no Ollama server is required. The learning notebooks retain their Ollama setup.

Table of Contents

Quick Start | Hugging Face Deployment | Architecture Overview | Project Structure | Configuration Guide | Common Customizations | Observability | Advanced Topics | Troubleshooting


Quick Start

Installation

From the repository root, use uv (Windows PowerShell):

uv venv --python 3.13 .venv
uv pip install --python .venv\Scripts\python.exe --torch-backend cpu -r requirements.txt
Copy-Item project\.env.example project\.env

Set GROQ_API_KEY in project/.env. Get a key from https://console.groq.com/keys. Never commit .env or expose the key in frontend code. The root requirements.txt is the standalone runtime dependency manifest. HF Gradio mounts only this file during installation, so it must not include other local requirement files. Notebook/evaluation tools are in requirements-notebooks.txt. The optional project/requirements.txt redirects local installs to the root manifest; Docker installs the root file directly.

Running the Application

Start the Gradio interface locally:

uv run --no-project --python .venv\Scripts\python.exe project\app.py

The application will be available at http://localhost:7860 (default Gradio port).

Chat memory is isolated per browser session and kept in memory, so it is lost on restart. Documents are a shared library, not private per-user storage: anyone with app access can query, upload, or clear them. Use a private Space or set both GRADIO_AUTH_USERNAME and GRADIO_AUTH_PASSWORD as secrets. This also protects your Groq quota. Run one app process: local Qdrant cannot be opened by multiple workers at the same path.

Prerequisites

  • Python 3.11+
  • A Groq API key; outbound access to Groq and Hugging Face model downloads

Architecture Overview

This system implements an advanced RAG pipeline with the following key features:

  • Parent-Child Chunking: Documents are split into small child chunks (for precise retrieval) linked to larger parent chunks (for rich context)
  • Hybrid Search: Combines dense embeddings and sparse (BM25) retrieval for optimal results
  • LangGraph Agent: Orchestrates query rewriting, retrieval, and response generation
  • Compact Chat Memory: Rewrites follow-ups using both a rolling summary and bounded recent conversation history
  • Clarification Memory: Preserves an unresolved query across human-in-the-loop pauses, combines its clarifications, then discards those temporary replies after the answer
  • Document-Aware Query Analysis: Receives the current uploaded filenames on every turn, resolves single-document references, and allows new requests to replace pending clarifications
  • Document Inventory Answers: Lists uploaded filenames from the library without vector retrieval or a GPU allocation
  • Groq Chat: Tool calling, structured query analysis, and streamed responses through ChatGroq
  • Vector Storage: Uses Qdrant for efficient similarity search

Data Flow

PDF → Markdown Conversion → Parent/Child Chunking → Vector Indexing → Agent Retrieval → LLM Response

Project Structure

Entry Point & Configuration

File Purpose
project/app.py Application entry point, launches Gradio UI
project/config.py Central configuration hub - edit this for provider/model/chunking changes
project/utils.py PDF conversion and cached context-token estimation with an offline-safe fallback
project/document_chunker.py Parent/child splitting logic with cleaning and merging rules
Dockerfile uv-based CPU runtime for the backend and Gradio frontend

Core System

File Purpose
project/core/rag_system.py System bootstrap - creates managers and compiles LangGraph agent
project/core/document_manager.py Document ingestion pipeline (convert, chunk, index)
project/core/chat_interface.py Streams the aggregated answer while separating query analysis and tool activity from internal node output
project/core/observability.py Optional Langfuse tracing — callback handler lifecycle

Database Layer

File Purpose
project/db/vector_db_manager.py Qdrant client wrapper with inference-free schema validation
project/db/dense_embeddings.py MiniLM embeddings with ZeroGPU inference on Spaces and CPU inference locally
project/db/parent_store_manager.py File-backed storage for parent chunks

RAG Agent (LangGraph)

File Purpose
project/rag_agent/graph.py Graph builder and compilation logic
project/rag_agent/graph_state.py Shared and per-agent state, including rolling memory, pending clarification, and answer accumulation/reset logic
project/rag_agent/nodes.py Node implementations (summarize, rewrite, agent execution, aggregate)
project/rag_agent/edges.py Conditional edge routing logic (e.g., routing based on query clarity)
project/rag_agent/tools.py Retrieval tools (search_child_chunks, retrieve_parent_chunks)
project/rag_agent/prompts.py System prompts for agent behavior
project/rag_agent/schemas.py Structured output schemas (Pydantic models)

User Interface

File Purpose
project/ui/css.py Custom CSS styling for the Gradio interface
project/ui/gradio_app.py Gradio UI implementation with document upload and chat

Configuration Guide

All primary settings are in project/config.py. Key parameters:

Directory Configuration

MARKDOWN_DIR = "markdown_docs"        # Storage for converted PDF → Markdown files
PARENT_STORE_PATH = "parent_store"    # File-backed storage for parent chunks
QDRANT_DB_PATH = "qdrant_db"          # Local Qdrant vector database path

Qdrant Configuration

CHILD_COLLECTION = "document_child_chunks"  # Collection name for child chunks
SPARSE_VECTOR_NAME = "sparse"               # Named sparse vector field (BM25)

Model Configuration

# Default: single model configuration
DENSE_MODEL = "sentence-transformers/all-MiniLM-L6-v2"
SPARSE_MODEL = "Qdrant/bm25"
LLM_MODEL = os.environ.get("GROQ_MODEL", "openai/gpt-oss-20b")
JUDGE_MODEL = "ministral-3:3b-instruct-2512-q8_0"
LLM_TEMPERATURE = 0  # 0 = deterministic, 1 = creative
LLM_SEED = 42
LLM_TIMEOUT = 60
LLM_MAX_RETRIES = 2

GROQ_API_KEY is required at startup. DENSE_MODEL and DATA_DIR can also be set as environment variables. JUDGE_MODEL remains the local Ollama judge for the evaluation notebook, not the deployed app. Query analysis explicitly uses function calling, so it does not require Groq's model-specific JSON Schema mode.

Retrieval Configuration

RETRIEVAL_SCORE_THRESHOLD = 0.4  # Lower = more recall, higher = more precision
DEFAULT_RETRIEVAL_K = 7          # Default number of child chunks used by retrieval and evaluation
CHILD_CHUNK_SEPARATOR = "\n\n<CHILD_CHUNK_BOUNDARY>\n\n"  # Keeps ranked child results separable for evaluation

Agent Configuration

# Hard limits to prevent infinite loops
MAX_TOOL_CALLS = 8       # Maximum tool calls per agent run
MAX_ITERATIONS = 10      # Maximum agent loop iterations
GRAPH_RECURSION_LIMIT = 50 # Maximum number of steps before hitting a stop condition
MAIN_HISTORY_MESSAGES_TO_KEEP = 4  # Raw messages retained after each answer; minimum 2

# Context compression thresholds
BASE_TOKEN_THRESHOLD = 2000     # Initial token threshold for compression
TOKEN_GROWTH_FACTOR = 0.9       # Multiplier applied after each compression

Terminal Execution Logging

EXECUTION_LOGGING_ENABLED = False  # Print graph steps, state previews, tool calls, and outputs
EXECUTION_LOG_MAX_CHARS = 1200     # Maximum characters shown for long values
EXECUTION_LOG_USE_COLOR = True     # Use ANSI colors in the terminal

Text Splitter Configuration

CHILD_CHUNK_SIZE = 500              # Size of chunks used for retrieval
CHILD_CHUNK_OVERLAP = 100           # Overlap between chunks (prevents context loss)
MIN_PARENT_SIZE = 2000              # Target minimum; very short documents may remain smaller
MAX_PARENT_SIZE = 4000             # Maximum parent chunk size

# Markdown header splitting strategy
HEADERS_TO_SPLIT_ON = [
    ("#", "H1"),
    ("##", "H2"),
    ("###", "H3")
]

Langfuse Observability (Optional)

import os

LANGFUSE_ENABLED = os.environ.get("LANGFUSE_ENABLED", "false").lower() == "true"
LANGFUSE_PUBLIC_KEY = ""               # From your Langfuse project settings
LANGFUSE_SECRET_KEY = ""               # From your Langfuse project settings
LANGFUSE_BASE_URL = "http://localhost:3000"  # Langfuse Cloud or self-hosted URL

Common Customizations

1. Switching LLM Provider (Single Provider)

Performance Note: LLMs with 8B+ parameters typically offer superior reasoning, context comprehension, and response quality compared to smaller models. This applies to both proprietary and open-source models, as long as they support native tool/function calling, which is required for agentic RAG workflows.

If you want to switch from Groq to another provider (e.g., Google Gemini), follow these steps:

Step 1: Install the provider's SDK

uv pip install langchain-google-genai

Step 2: Set environment variable

export GOOGLE_API_KEY="your-google-key"

Step 3: Update project/config.py

LLM_MODEL = "gemini-2.5-pro"
LLM_TEMPERATURE = 0
LLM_SEED = 42

Step 4: Modify project/core/rag_system.py

Replace:

llm = ChatGroq(model=config.LLM_MODEL, temperature=config.LLM_TEMPERATURE,
               timeout=config.LLM_TIMEOUT, max_retries=config.LLM_MAX_RETRIES,
               model_kwargs={"seed": config.LLM_SEED})

With:

from langchain_google_genai import ChatGoogleGenerativeAI

llm = ChatGoogleGenerativeAI(model=config.LLM_MODEL, temperature=config.LLM_TEMPERATURE)

2. Optional Multi-Provider Configuration

The application ships with a Groq implementation. The following is an optional customization example, not the shipped configuration. Remove the Groq-only startup key check if implementing a different provider.

This approach allows you to maintain multiple provider configurations and switch between them easily.

Step 1: Install required SDKs

uv pip install langchain-openai langchain-anthropic langchain-google-genai

Step 2: Set environment variables

export OPENAI_API_KEY="your-openai-key"
export ANTHROPIC_API_KEY="your-anthropic-key"
export GOOGLE_API_KEY="your-google-key"

Step 3: Update project/config.py with multi-provider configuration

# --- Multi-Provider LLM Configuration ---
LLM_CONFIGS = {
    "ollama": {
        "model": "granite4.1:8b",
        "url":"http://localhost:11434",
        "temperature": 0
    },
    "openai": {
        "model": "gpt-5.2",
        "temperature": 0
    },
    "anthropic": {
        "model": "claude-sonnet-4-6",
        "temperature": 0
    },
    "google": {
        "model": "gemini-2.5-flash",
        "temperature": 0
    }
}

# Switch providers by changing this single line
ACTIVE_LLM_CONFIG = "ollama"

Step 4: Modify project/core/rag_system.py in the initialize() method

Replace the existing LLM initialization with:

def initialize(self):
    self.vector_db.create_collection(self.collection_name)
    collection = self.vector_db.get_collection(self.collection_name)
    
    # Load active configuration
    active_config = config.LLM_CONFIGS[config.ACTIVE_LLM_CONFIG]
    model = active_config["model"]
    temperature = active_config["temperature"]
    
    if config.ACTIVE_LLM_CONFIG == "ollama":
        from langchain_ollama import ChatOllama
        llm = ChatOllama(model=model, temperature=temperature, base_url=active_config["url"])
        
    elif config.ACTIVE_LLM_CONFIG == "openai":
        from langchain_openai import ChatOpenAI
        llm = ChatOpenAI(model=model, temperature=temperature)
        
    elif config.ACTIVE_LLM_CONFIG == "anthropic":
        from langchain_anthropic import ChatAnthropic
        llm = ChatAnthropic(model=model, temperature=temperature)
        
    elif config.ACTIVE_LLM_CONFIG == "google":
        from langchain_google_genai import ChatGoogleGenerativeAI
        llm = ChatGoogleGenerativeAI(model=model, temperature=temperature)
        
    else:
        raise ValueError(f"Unsupported LLM provider: {config.ACTIVE_LLM_CONFIG}")
    
    # Continue with tool and graph initialization
    tools = ToolFactory(collection).create_tools()
    self.agent_graph = create_agent_graph(llm, tools)

Switching Providers: Simply change ACTIVE_LLM_CONFIG in config.py:

ACTIVE_LLM_CONFIG = "google"  # Switch to Gemini Pro
# ACTIVE_LLM_CONFIG = "anthropic"  # Or Claude Sonnet
# ACTIVE_LLM_CONFIG = "openai"  # Or GPT-4o

Provider Reference Table:

Provider Environment Variable Import Statement Example Models
Groq GROQ_API_KEY from langchain_groq import ChatGroq openai/gpt-oss-20b
OpenAI OPENAI_API_KEY from langchain_openai import ChatOpenAI gpt-5.2, gpt-5-mini
Anthropic ANTHROPIC_API_KEY from langchain_anthropic import ChatAnthropic claude-opus-4-6, claude-sonnet-4-6
Google GOOGLE_API_KEY from langchain_google_genai import ChatGoogleGenerativeAI gemini-2.5-pro, gemini-2.5-flash
Ollama None (local) from langchain_ollama import ChatOllama granite4.1:8b, llama3.1:8b-instruct-q6_K

3. Changing Embedding Models

Why change? Trade-offs between speed, cost, and quality.

Step 1: Update project/config.py

# Default: Compact model (ZeroGPU on HF, CPU locally)
DENSE_MODEL = "sentence-transformers/all-MiniLM-L6-v2"

# Example: Alternative Sentence Transformers model
# DENSE_MODEL = "sentence-transformers/all-mpnet-base-v2"

# Example: Gemma embeddings (Google's open model)
# DENSE_MODEL = "google/embeddinggemma-300m"

# Optional: Qwen embeddings (Alibaba's multilingual model)
# DENSE_MODEL = "Qwen/Qwen3-Embedding-0.6B"

SPARSE_MODEL = "Qdrant/bm25"  # Usually no need to change

Step 2: Re-index your documents

⚠️ Important: Changing embeddings requires re-indexing. In the educational UI, use Clear All, restart if the old collection prevents startup, then upload the documents again.

Implementation Details (in project/db/vector_db_manager.py):

from langchain_huggingface import HuggingFaceEmbeddings
from langchain_qdrant import FastEmbedSparse
import config

from db.dense_embeddings import DenseEmbeddings

self.__dense_embeddings = DenseEmbeddings(model_name=config.DENSE_MODEL)
self.__sparse_embeddings = FastEmbedSparse(model_name=config.SPARSE_MODEL)

Popular Embedding Models:

Model Context Size Vector Dimension Speed Quality Use Case
all-MiniLM-L6-v2 256 tokens 384 Fast Good Default compact embedding model
all-mpnet-base-v2 512 tokens 768 Medium Excellent High-accuracy semantic search
bge-large-en-v1.5 512 tokens 1024 Slow Best Production-grade retrieval on GPU
google/embeddinggemma-300m 2048 tokens 768 Fast Very Good Lightweight, efficient multilingual retrieval
Qwen/Qwen3-Embedding-0.6B 32768 tokens 1024 Medium Excellent Optional multilingual/code retrieval model
Qwen/Qwen3-Embedding-8B 32768 tokens 4096 Very Slow Excellent / SOTA Heavy GPU-oriented retrieval experiments

4. Adjusting Chunking Strategy

Why adjust? Balance between retrieval precision and context richness.

Before re-indexing, inspect Markdown output and chunk boundaries to verify the document structure.

Step 1: Update chunk sizes in project/config.py

# For short, factual queries (e.g., technical documentation)
CHILD_CHUNK_SIZE = 300
CHILD_CHUNK_OVERLAP = 60
MIN_PARENT_SIZE = 1500
MAX_PARENT_SIZE = 8000

# For narrative or contextual queries (e.g., legal documents)
# CHILD_CHUNK_SIZE = 800
# CHILD_CHUNK_OVERLAP = 150
# MIN_PARENT_SIZE = 3000
# MAX_PARENT_SIZE = 15000

Step 2 (Optional): Replace the splitter in project/document_chunker.py

Default (Character-based):

from langchain_text_splitters import RecursiveCharacterTextSplitter

self.__child_splitter = RecursiveCharacterTextSplitter(
    chunk_size=config.CHILD_CHUNK_SIZE,
    chunk_overlap=config.CHILD_CHUNK_OVERLAP
)

Alternative (Sentence-aware):

from langchain_text_splitters import SentenceTransformersTokenTextSplitter

self.__child_splitter = SentenceTransformersTokenTextSplitter(
    chunk_size=config.CHILD_CHUNK_SIZE,
    chunk_overlap=config.CHILD_CHUNK_OVERLAP
)

Step 3: Re-run ingestion

Use Clear All in the Gradio interface, then upload the documents again. Existing documents are intentionally skipped to avoid accidental duplicates.

Chunking Guidelines:

⚠️ Disclaimer: These are empirical guidelines. Optimal sizes depend on:

  • Child chunk → embedding model's context window (e.g. 256 tokens for all-MiniLM-L6-v2, 512 for bge-large-en-v1.5): child size should not exceed it
  • Parent chunk → generative model's context window (e.g. 8K, 32K, 128K tokens): parent must fit within the context sent to the LLM alongside the query

Always validate values empirically on your own corpus.

The chunker enforces MAX_PARENT_SIZE, deduplicates merged header metadata, and rebalances small neighboring chunks when possible. A document shorter than MIN_PARENT_SIZE remains a single smaller chunk.

Document Type Child Size Parent Size Reasoning
Technical Docs 300-500 2000-4000 Precise lookups, code snippets
Legal Contracts 600-1000 5000-15000 Context-heavy, definitions
Research Papers 400-600 3000-8000 Balance of precision and context
FAQs / Knowledge Base 200-400 1500-4000 Short, focused answers

5. Agent Configuration

Tune agent behavior in project/config.py:

# Hard limits to prevent infinite loops
MAX_TOOL_CALLS = 8       # Maximum tool calls per agent run
MAX_ITERATIONS = 10      # Maximum agent loop iterations
GRAPH_RECURSION_LIMIT = 50 # Maximum number of steps before hitting a stop condition
MAIN_HISTORY_MESSAGES_TO_KEEP = 4  # Raw messages retained after each answer; minimum 2

# Context compression thresholds
BASE_TOKEN_THRESHOLD = 2000     # Initial token threshold for compression
TOKEN_GROWTH_FACTOR = 0.9       # Multiplier applied after each compression
Parameter Effect
MAX_TOOL_CALLS Maximum cumulative tool calls requested; a request that would exceed it is not executed
MAX_ITERATIONS Maximum LLM reasoning iterations; a final answer at the boundary is still accepted
GRAPH_RECURSION_LIMIT Increase for complex graphs
BASE_TOKEN_THRESHOLD Delay compression by increasing this value
TOKEN_GROWTH_FACTOR Lower values compress more aggressively

Observability

Optional tracing with Langfuse captures every LLM call, tool invocation, and graph transition. It is useful for debugging agent behavior, tracking costs, and evaluating retrieval quality.

Enabling Langfuse

  1. Sign up on Langfuse Cloud, create an organization, then create a project and generate API keys from the project settings.
  2. Set environment variables (or copy .env.example to .env):
export LANGFUSE_ENABLED=true
export LANGFUSE_PUBLIC_KEY=pk-lf-...
export LANGFUSE_SECRET_KEY=sk-lf-...
export LANGFUSE_BASE_URL=https://cloud.langfuse.com   # or your self-hosted URL
  1. Run the app normally. Traces appear in your Langfuse dashboard.

To disable tracing, set LANGFUSE_ENABLED=false or leave the variables unset. The application runs identically either way.

For additional details on integrating Langfuse with LangChain or LangGraph, see the official documentation.

What gets traced

Component Traced operations
Graph nodes summarize_history, rewrite_query, orchestrator, compress_context, fallback_response, aggregate_answers
Tools search_child_chunks, retrieve_parent_chunks (arguments + results)
Structured output QueryAnalysis parsing in the rewrite step
Subgraph fan-out Parallel agent invocations via Send()

Hosting options

For a conceptual tracing guide and platform context, see notebooks/observability.ipynb.


Advanced Topics

Customizing the RAG Agent

Location: project/rag_agent/

Add/Remove Nodes: Edit graph.py and nodes.py

Example: Adding a fact-checking node

# In nodes.py
def fact_check_node(state):
    # Your fact-checking logic
    return {"fact_checked": True}

# In graph.py
builder.add_node("fact_check", fact_check_node)
builder.add_edge("retrieve", "fact_check")

Modify Conditional Routing: Edit edges.py to change graph flow logic

Example from the system - routing based on query clarity:

def route_after_rewrite(state: State) -> Literal["request_clarification", "agent"]:
    """Routes to human input if question unclear, otherwise processes all rewritten queries"""
    if not state.get("questionIsClear", False):
        return "request_clarification"
    else:
        # Fan-out: send each rewritten question to parallel processing
        return [
            Send("agent", {"question": query, "question_index": idx, "messages": []})
            for idx, query in enumerate(state["rewrittenQuestions"])
        ]

This pattern allows the agent to either request clarification from the user or fan-out multiple query variations for parallel retrieval.

Modify Prompts: Edit prompts.py to change agent behavior and response style

Add Custom Tools: Extend tools.py with new retrieval strategies or external integrations

Replacing Storage Backends

Vector Database:

  • Default: Local Qdrant
  • Alternatives: Remote Qdrant Cloud, Pinecone, Weaviate
  • Edit: project/db/vector_db_manager.py

Parent Store:

  • Default: JSON file
  • Alternatives: PostgreSQL, MongoDB, S3
  • Edit: project/db/parent_store_manager.py

Extending the UI

Location: project/ui/gradio_app.py

Add runtime settings, admin panels, or analytics:

with gr.Accordion("Advanced Settings", open=False):
    provider_dropdown = gr.Dropdown(
        choices=["openai", "anthropic", "google", "ollama"],
        label="LLM Provider"
    )

Docker Deployment

The root Dockerfile installs only runtime dependencies using uv and CPU PyTorch. It serves the existing Gradio frontend and backend together; there is no separate frontend build or Ollama process. MiniLM reduces memory use, but PyTorch and PDF indexing still need more than a 512 MB host. If selecting Qwen embeddings instead, allow several GB of RAM; 8 GB is a starting budget. Model weights download on first startup, so the initial start takes longer.

Build and Run

docker build -t agentic-rag .
docker run --name rag-assistant --env-file project\.env -p 7860:7860 agentic-rag

For persistent documents and indexes:

docker run --name rag-assistant --env-file project\.env -e DATA_DIR=/data -v rag-data:/data -p 7860:7860 agentic-rag

Once running, open http://localhost:7860.

Hugging Face Spaces (Recommended)

  1. Create a Gradio Space and select ZeroGPU Free if offered for your account. Do not select Docker. The root README already specifies sdk: gradio, app_file: project/app.py, the Gradio version, and Python 3.12.12 (supported by ZeroGPU).
  2. Upload the root README.md, requirements.txt, LICENSE, and the full project folder (including project/requirements.txt). Do not upload .env, .venv, local documents, or local database files.
  3. Under Settings > Variables and secrets, add GROQ_API_KEY as a secret, set GROQ_MODEL=openai/gpt-oss-20b as a variable (or remove an old override to use this default), and add both authentication values as secrets if the Space is public.
  4. Wait for the build and first model download, then open the App tab. Upload a small Markdown/PDF file under Documents, click Add Documents, and ask a question under Chat.

The public Gradio API is named /chat. Document-management callbacks are UI-only (api_visibility="private"), so API clients do not select the file upload callback when sending text. This is not per-user document isolation. After a rebuild, hard-refresh open app tabs to discard stale event IDs and queue sessions. A queue-stream 404 can mean that its session no longer exists; it is not evidence of a missing document or a Groq failure.

The app loads MiniLM on CUDA at startup when HF sets SPACES_ZERO_GPU=true. Dense embedding inference runs inside @spaces.GPU(duration=30), both during document indexing and query retrieval. Only texts and embedding results cross the worker boundary; local Qdrant writes and Groq calls stay outside GPU tasks. Startup obtains the vector dimension from model metadata and validates the Qdrant schema without making an inference request.

ZeroGPU requires a registered GPU function during startup; a CPU-only app is not sufficient on this hardware tier. Embedding requests consume the visitor's GPU quota and may queue. Large indexing jobs can require multiple allocations. CPU embedding inference is retained for local and non-ZeroGPU deployments. Verify first startup and indexing in the Space logs.

The UI is branded Document Chat, with a teal-accented header, a compact chat panel, and the same Documents and Chat workflow.

Per the ZeroGPU documentation, free personal accounts in good standing (verified email and older than 30 days) can host up to two ZeroGPU Spaces. Availability is account-dependent. Gradio Spaces manage dependency installation from root requirements.txt; uv remains the tool for local installation and the optional Docker build.

Documents and indexes use ephemeral local disk and may be lost on restart/rebuild. Chat memory is always in memory. Do not use this demo as durable storage. DATA_DIR can point to an existing writable mount on hosts that provide one, but does not provision persistence.

Render (Alternative)

Create a Docker Web Service using the root Dockerfile with the repository root as the build context. Set GROQ_API_KEY as a secret. The app honors Render's PORT and listens on 0.0.0.0 in Docker. Choose a paid instance with enough RAM for the embedding model, and attach a disk at /data with DATA_DIR=/data if documents must survive deploys. Use one instance/worker. The app defaults to DENSE_MODEL=sentence-transformers/all-MiniLM-L6-v2; this trades retrieval quality for lower memory use compared with Qwen. Render's free instances sleep and lose local files; do not rely on them for this model-heavy app. See free-service limits.

Vercel

Not recommended for the full app: it depends on a long-running Gradio server, in-memory LangGraph checkpoints, local Qdrant files, and model downloads. Vercel can host a future separate JavaScript frontend, but the current Python service should remain on Spaces or Render.


Troubleshooting

Issue Cause Solution
"Model not found" error Incorrect model name for provider Verify LLM_MODEL matches provider's API (e.g., gpt-4o-mini not gpt4-mini)
Low-quality retrieval results Poor embedding model or chunk configuration Re-index with better embeddings or adjust chunk sizes
Vector size mismatch Existing Qdrant collection was built with a different embedding model Clear the collection and re-index documents after changing DENSE_MODEL
Slow response times Large embedding model or high top_k value Use smaller embedding models (e.g., all-MiniLM-L6-v2) or reduce top_k in retrieval tools
API rate limits exceeded Too many requests to external provider Add retry logic with exponential backoff or switch to local Ollama models
Out of memory errors Large document set or embedding model Use smaller embeddings, reduce batch size, or enable GPU acceleration
Empty retrieval results Collection not indexed or wrong collection name Verify documents are uploaded and CHILD_COLLECTION name matches in config
Import errors after provider switch Missing SDK installation Install required package: pip install langchain-{provider}
Inconsistent answers across runs High temperature setting Set LLM_TEMPERATURE = 0 in config for deterministic responses