Spaces:
Sleeping
Download project/README.md from devank2000/agentic-rag: direct link, hf CLI and curl.
- Browser
- Download file 30.3 kB
-
https://huggingface.co/spaces/devank2000/agentic-rag/resolve/main/project/README.md
- Command line
-
hf download hf://spaces/devank2000/agentic-rag/project/README.md
-
curl -L -o README.md https://huggingface.co/spaces/devank2000/agentic-rag/resolve/main/project/README.md
A newer version of the Gradio SDK is available: 6.32.0
Document Chat Documentation
An Agentic Retrieval-Augmented Generation (RAG) system built with LangGraph, featuring parent–child chunking, hybrid dense + sparse retrieval, and Groq-powered chat with a Gradio frontend. Embeddings use ZeroGPU on HF Spaces and CPU locally; no Ollama server is required. The learning notebooks retain their Ollama setup.
Table of Contents
Quick Start | Hugging Face Deployment | Architecture Overview | Project Structure | Configuration Guide | Common Customizations | Observability | Advanced Topics | Troubleshooting
Quick Start
Installation
From the repository root, use uv (Windows PowerShell):
uv venv --python 3.13 .venv
uv pip install --python .venv\Scripts\python.exe --torch-backend cpu -r requirements.txt
Copy-Item project\.env.example project\.env
Set GROQ_API_KEY in project/.env. Get a key from
https://console.groq.com/keys. Never commit .env or expose the key in frontend
code. The root requirements.txt is the standalone runtime dependency manifest.
HF Gradio mounts only this file during installation, so it must not include
other local requirement files. Notebook/evaluation tools are in
requirements-notebooks.txt. The optional project/requirements.txt redirects
local installs to the root manifest; Docker installs the root file directly.
Running the Application
Start the Gradio interface locally:
uv run --no-project --python .venv\Scripts\python.exe project\app.py
The application will be available at http://localhost:7860 (default Gradio port).
Chat memory is isolated per browser session and kept in memory, so it is lost on restart. Documents are a shared library, not private per-user storage: anyone with app access can query, upload, or clear them. Use a private Space or set both
GRADIO_AUTH_USERNAMEandGRADIO_AUTH_PASSWORDas secrets. This also protects your Groq quota. Run one app process: local Qdrant cannot be opened by multiple workers at the same path.
Prerequisites
- Python 3.11+
- A Groq API key; outbound access to Groq and Hugging Face model downloads
Architecture Overview
This system implements an advanced RAG pipeline with the following key features:
- Parent-Child Chunking: Documents are split into small child chunks (for precise retrieval) linked to larger parent chunks (for rich context)
- Hybrid Search: Combines dense embeddings and sparse (BM25) retrieval for optimal results
- LangGraph Agent: Orchestrates query rewriting, retrieval, and response generation
- Compact Chat Memory: Rewrites follow-ups using both a rolling summary and bounded recent conversation history
- Clarification Memory: Preserves an unresolved query across human-in-the-loop pauses, combines its clarifications, then discards those temporary replies after the answer
- Document-Aware Query Analysis: Receives the current uploaded filenames on every turn, resolves single-document references, and allows new requests to replace pending clarifications
- Document Inventory Answers: Lists uploaded filenames from the library without vector retrieval or a GPU allocation
- Groq Chat: Tool calling, structured query analysis, and streamed responses through
ChatGroq - Vector Storage: Uses Qdrant for efficient similarity search
Data Flow
PDF → Markdown Conversion → Parent/Child Chunking → Vector Indexing → Agent Retrieval → LLM Response
Project Structure
Entry Point & Configuration
| File | Purpose |
|---|---|
project/app.py |
Application entry point, launches Gradio UI |
project/config.py |
Central configuration hub - edit this for provider/model/chunking changes |
project/utils.py |
PDF conversion and cached context-token estimation with an offline-safe fallback |
project/document_chunker.py |
Parent/child splitting logic with cleaning and merging rules |
Dockerfile |
uv-based CPU runtime for the backend and Gradio frontend |
Core System
| File | Purpose |
|---|---|
project/core/rag_system.py |
System bootstrap - creates managers and compiles LangGraph agent |
project/core/document_manager.py |
Document ingestion pipeline (convert, chunk, index) |
project/core/chat_interface.py |
Streams the aggregated answer while separating query analysis and tool activity from internal node output |
project/core/observability.py |
Optional Langfuse tracing — callback handler lifecycle |
Database Layer
| File | Purpose |
|---|---|
project/db/vector_db_manager.py |
Qdrant client wrapper with inference-free schema validation |
project/db/dense_embeddings.py |
MiniLM embeddings with ZeroGPU inference on Spaces and CPU inference locally |
project/db/parent_store_manager.py |
File-backed storage for parent chunks |
RAG Agent (LangGraph)
| File | Purpose |
|---|---|
project/rag_agent/graph.py |
Graph builder and compilation logic |
project/rag_agent/graph_state.py |
Shared and per-agent state, including rolling memory, pending clarification, and answer accumulation/reset logic |
project/rag_agent/nodes.py |
Node implementations (summarize, rewrite, agent execution, aggregate) |
project/rag_agent/edges.py |
Conditional edge routing logic (e.g., routing based on query clarity) |
project/rag_agent/tools.py |
Retrieval tools (search_child_chunks, retrieve_parent_chunks) |
project/rag_agent/prompts.py |
System prompts for agent behavior |
project/rag_agent/schemas.py |
Structured output schemas (Pydantic models) |
User Interface
| File | Purpose |
|---|---|
project/ui/css.py |
Custom CSS styling for the Gradio interface |
project/ui/gradio_app.py |
Gradio UI implementation with document upload and chat |
Configuration Guide
All primary settings are in project/config.py. Key parameters:
Directory Configuration
MARKDOWN_DIR = "markdown_docs" # Storage for converted PDF → Markdown files
PARENT_STORE_PATH = "parent_store" # File-backed storage for parent chunks
QDRANT_DB_PATH = "qdrant_db" # Local Qdrant vector database path
Qdrant Configuration
CHILD_COLLECTION = "document_child_chunks" # Collection name for child chunks
SPARSE_VECTOR_NAME = "sparse" # Named sparse vector field (BM25)
Model Configuration
# Default: single model configuration
DENSE_MODEL = "sentence-transformers/all-MiniLM-L6-v2"
SPARSE_MODEL = "Qdrant/bm25"
LLM_MODEL = os.environ.get("GROQ_MODEL", "openai/gpt-oss-20b")
JUDGE_MODEL = "ministral-3:3b-instruct-2512-q8_0"
LLM_TEMPERATURE = 0 # 0 = deterministic, 1 = creative
LLM_SEED = 42
LLM_TIMEOUT = 60
LLM_MAX_RETRIES = 2
GROQ_API_KEY is required at startup. DENSE_MODEL and DATA_DIR can also be
set as environment variables. JUDGE_MODEL remains the local Ollama judge for
the evaluation notebook, not the deployed app. Query analysis explicitly uses
function calling, so it does not require Groq's model-specific JSON Schema mode.
Retrieval Configuration
RETRIEVAL_SCORE_THRESHOLD = 0.4 # Lower = more recall, higher = more precision
DEFAULT_RETRIEVAL_K = 7 # Default number of child chunks used by retrieval and evaluation
CHILD_CHUNK_SEPARATOR = "\n\n<CHILD_CHUNK_BOUNDARY>\n\n" # Keeps ranked child results separable for evaluation
Agent Configuration
# Hard limits to prevent infinite loops
MAX_TOOL_CALLS = 8 # Maximum tool calls per agent run
MAX_ITERATIONS = 10 # Maximum agent loop iterations
GRAPH_RECURSION_LIMIT = 50 # Maximum number of steps before hitting a stop condition
MAIN_HISTORY_MESSAGES_TO_KEEP = 4 # Raw messages retained after each answer; minimum 2
# Context compression thresholds
BASE_TOKEN_THRESHOLD = 2000 # Initial token threshold for compression
TOKEN_GROWTH_FACTOR = 0.9 # Multiplier applied after each compression
Terminal Execution Logging
EXECUTION_LOGGING_ENABLED = False # Print graph steps, state previews, tool calls, and outputs
EXECUTION_LOG_MAX_CHARS = 1200 # Maximum characters shown for long values
EXECUTION_LOG_USE_COLOR = True # Use ANSI colors in the terminal
Text Splitter Configuration
CHILD_CHUNK_SIZE = 500 # Size of chunks used for retrieval
CHILD_CHUNK_OVERLAP = 100 # Overlap between chunks (prevents context loss)
MIN_PARENT_SIZE = 2000 # Target minimum; very short documents may remain smaller
MAX_PARENT_SIZE = 4000 # Maximum parent chunk size
# Markdown header splitting strategy
HEADERS_TO_SPLIT_ON = [
("#", "H1"),
("##", "H2"),
("###", "H3")
]
Langfuse Observability (Optional)
import os
LANGFUSE_ENABLED = os.environ.get("LANGFUSE_ENABLED", "false").lower() == "true"
LANGFUSE_PUBLIC_KEY = "" # From your Langfuse project settings
LANGFUSE_SECRET_KEY = "" # From your Langfuse project settings
LANGFUSE_BASE_URL = "http://localhost:3000" # Langfuse Cloud or self-hosted URL
Common Customizations
1. Switching LLM Provider (Single Provider)
Performance Note: LLMs with 8B+ parameters typically offer superior reasoning, context comprehension, and response quality compared to smaller models. This applies to both proprietary and open-source models, as long as they support native tool/function calling, which is required for agentic RAG workflows.
If you want to switch from Groq to another provider (e.g., Google Gemini), follow these steps:
Step 1: Install the provider's SDK
uv pip install langchain-google-genai
Step 2: Set environment variable
export GOOGLE_API_KEY="your-google-key"
Step 3: Update project/config.py
LLM_MODEL = "gemini-2.5-pro"
LLM_TEMPERATURE = 0
LLM_SEED = 42
Step 4: Modify project/core/rag_system.py
Replace:
llm = ChatGroq(model=config.LLM_MODEL, temperature=config.LLM_TEMPERATURE,
timeout=config.LLM_TIMEOUT, max_retries=config.LLM_MAX_RETRIES,
model_kwargs={"seed": config.LLM_SEED})
With:
from langchain_google_genai import ChatGoogleGenerativeAI
llm = ChatGoogleGenerativeAI(model=config.LLM_MODEL, temperature=config.LLM_TEMPERATURE)
2. Optional Multi-Provider Configuration
The application ships with a Groq implementation. The following is an optional customization example, not the shipped configuration. Remove the Groq-only startup key check if implementing a different provider.
This approach allows you to maintain multiple provider configurations and switch between them easily.
Step 1: Install required SDKs
uv pip install langchain-openai langchain-anthropic langchain-google-genai
Step 2: Set environment variables
export OPENAI_API_KEY="your-openai-key"
export ANTHROPIC_API_KEY="your-anthropic-key"
export GOOGLE_API_KEY="your-google-key"
Step 3: Update project/config.py with multi-provider configuration
# --- Multi-Provider LLM Configuration ---
LLM_CONFIGS = {
"ollama": {
"model": "granite4.1:8b",
"url":"http://localhost:11434",
"temperature": 0
},
"openai": {
"model": "gpt-5.2",
"temperature": 0
},
"anthropic": {
"model": "claude-sonnet-4-6",
"temperature": 0
},
"google": {
"model": "gemini-2.5-flash",
"temperature": 0
}
}
# Switch providers by changing this single line
ACTIVE_LLM_CONFIG = "ollama"
Step 4: Modify project/core/rag_system.py in the initialize() method
Replace the existing LLM initialization with:
def initialize(self):
self.vector_db.create_collection(self.collection_name)
collection = self.vector_db.get_collection(self.collection_name)
# Load active configuration
active_config = config.LLM_CONFIGS[config.ACTIVE_LLM_CONFIG]
model = active_config["model"]
temperature = active_config["temperature"]
if config.ACTIVE_LLM_CONFIG == "ollama":
from langchain_ollama import ChatOllama
llm = ChatOllama(model=model, temperature=temperature, base_url=active_config["url"])
elif config.ACTIVE_LLM_CONFIG == "openai":
from langchain_openai import ChatOpenAI
llm = ChatOpenAI(model=model, temperature=temperature)
elif config.ACTIVE_LLM_CONFIG == "anthropic":
from langchain_anthropic import ChatAnthropic
llm = ChatAnthropic(model=model, temperature=temperature)
elif config.ACTIVE_LLM_CONFIG == "google":
from langchain_google_genai import ChatGoogleGenerativeAI
llm = ChatGoogleGenerativeAI(model=model, temperature=temperature)
else:
raise ValueError(f"Unsupported LLM provider: {config.ACTIVE_LLM_CONFIG}")
# Continue with tool and graph initialization
tools = ToolFactory(collection).create_tools()
self.agent_graph = create_agent_graph(llm, tools)
Switching Providers: Simply change ACTIVE_LLM_CONFIG in config.py:
ACTIVE_LLM_CONFIG = "google" # Switch to Gemini Pro
# ACTIVE_LLM_CONFIG = "anthropic" # Or Claude Sonnet
# ACTIVE_LLM_CONFIG = "openai" # Or GPT-4o
Provider Reference Table:
| Provider | Environment Variable | Import Statement | Example Models |
|---|---|---|---|
| Groq | GROQ_API_KEY |
from langchain_groq import ChatGroq |
openai/gpt-oss-20b |
| OpenAI | OPENAI_API_KEY |
from langchain_openai import ChatOpenAI |
gpt-5.2, gpt-5-mini |
| Anthropic | ANTHROPIC_API_KEY |
from langchain_anthropic import ChatAnthropic |
claude-opus-4-6, claude-sonnet-4-6 |
GOOGLE_API_KEY |
from langchain_google_genai import ChatGoogleGenerativeAI |
gemini-2.5-pro, gemini-2.5-flash |
|
| Ollama | None (local) | from langchain_ollama import ChatOllama |
granite4.1:8b, llama3.1:8b-instruct-q6_K |
3. Changing Embedding Models
Why change? Trade-offs between speed, cost, and quality.
Step 1: Update project/config.py
# Default: Compact model (ZeroGPU on HF, CPU locally)
DENSE_MODEL = "sentence-transformers/all-MiniLM-L6-v2"
# Example: Alternative Sentence Transformers model
# DENSE_MODEL = "sentence-transformers/all-mpnet-base-v2"
# Example: Gemma embeddings (Google's open model)
# DENSE_MODEL = "google/embeddinggemma-300m"
# Optional: Qwen embeddings (Alibaba's multilingual model)
# DENSE_MODEL = "Qwen/Qwen3-Embedding-0.6B"
SPARSE_MODEL = "Qdrant/bm25" # Usually no need to change
Step 2: Re-index your documents
⚠️ Important: Changing embeddings requires re-indexing. In the educational UI, use Clear All, restart if the old collection prevents startup, then upload the documents again.
Implementation Details (in project/db/vector_db_manager.py):
from langchain_huggingface import HuggingFaceEmbeddings
from langchain_qdrant import FastEmbedSparse
import config
from db.dense_embeddings import DenseEmbeddings
self.__dense_embeddings = DenseEmbeddings(model_name=config.DENSE_MODEL)
self.__sparse_embeddings = FastEmbedSparse(model_name=config.SPARSE_MODEL)
Popular Embedding Models:
| Model | Context Size | Vector Dimension | Speed | Quality | Use Case |
|---|---|---|---|---|---|
| all-MiniLM-L6-v2 | 256 tokens | 384 | Fast | Good | Default compact embedding model |
| all-mpnet-base-v2 | 512 tokens | 768 | Medium | Excellent | High-accuracy semantic search |
| bge-large-en-v1.5 | 512 tokens | 1024 | Slow | Best | Production-grade retrieval on GPU |
| google/embeddinggemma-300m | 2048 tokens | 768 | Fast | Very Good | Lightweight, efficient multilingual retrieval |
| Qwen/Qwen3-Embedding-0.6B | 32768 tokens | 1024 | Medium | Excellent | Optional multilingual/code retrieval model |
| Qwen/Qwen3-Embedding-8B | 32768 tokens | 4096 | Very Slow | Excellent / SOTA | Heavy GPU-oriented retrieval experiments |
4. Adjusting Chunking Strategy
Why adjust? Balance between retrieval precision and context richness.
Before re-indexing, inspect Markdown output and chunk boundaries to verify the document structure.
Step 1: Update chunk sizes in project/config.py
# For short, factual queries (e.g., technical documentation)
CHILD_CHUNK_SIZE = 300
CHILD_CHUNK_OVERLAP = 60
MIN_PARENT_SIZE = 1500
MAX_PARENT_SIZE = 8000
# For narrative or contextual queries (e.g., legal documents)
# CHILD_CHUNK_SIZE = 800
# CHILD_CHUNK_OVERLAP = 150
# MIN_PARENT_SIZE = 3000
# MAX_PARENT_SIZE = 15000
Step 2 (Optional): Replace the splitter in project/document_chunker.py
Default (Character-based):
from langchain_text_splitters import RecursiveCharacterTextSplitter
self.__child_splitter = RecursiveCharacterTextSplitter(
chunk_size=config.CHILD_CHUNK_SIZE,
chunk_overlap=config.CHILD_CHUNK_OVERLAP
)
Alternative (Sentence-aware):
from langchain_text_splitters import SentenceTransformersTokenTextSplitter
self.__child_splitter = SentenceTransformersTokenTextSplitter(
chunk_size=config.CHILD_CHUNK_SIZE,
chunk_overlap=config.CHILD_CHUNK_OVERLAP
)
Step 3: Re-run ingestion
Use Clear All in the Gradio interface, then upload the documents again. Existing documents are intentionally skipped to avoid accidental duplicates.
Chunking Guidelines:
⚠️ Disclaimer: These are empirical guidelines. Optimal sizes depend on:
- Child chunk → embedding model's context window (e.g. 256 tokens for all-MiniLM-L6-v2, 512 for bge-large-en-v1.5): child size should not exceed it
- Parent chunk → generative model's context window (e.g. 8K, 32K, 128K tokens): parent must fit within the context sent to the LLM alongside the query
Always validate values empirically on your own corpus.
The chunker enforces MAX_PARENT_SIZE, deduplicates merged header metadata, and rebalances small neighboring chunks when possible. A document shorter than MIN_PARENT_SIZE remains a single smaller chunk.
| Document Type | Child Size | Parent Size | Reasoning |
|---|---|---|---|
| Technical Docs | 300-500 | 2000-4000 | Precise lookups, code snippets |
| Legal Contracts | 600-1000 | 5000-15000 | Context-heavy, definitions |
| Research Papers | 400-600 | 3000-8000 | Balance of precision and context |
| FAQs / Knowledge Base | 200-400 | 1500-4000 | Short, focused answers |
5. Agent Configuration
Tune agent behavior in project/config.py:
# Hard limits to prevent infinite loops
MAX_TOOL_CALLS = 8 # Maximum tool calls per agent run
MAX_ITERATIONS = 10 # Maximum agent loop iterations
GRAPH_RECURSION_LIMIT = 50 # Maximum number of steps before hitting a stop condition
MAIN_HISTORY_MESSAGES_TO_KEEP = 4 # Raw messages retained after each answer; minimum 2
# Context compression thresholds
BASE_TOKEN_THRESHOLD = 2000 # Initial token threshold for compression
TOKEN_GROWTH_FACTOR = 0.9 # Multiplier applied after each compression
| Parameter | Effect |
|---|---|
MAX_TOOL_CALLS |
Maximum cumulative tool calls requested; a request that would exceed it is not executed |
MAX_ITERATIONS |
Maximum LLM reasoning iterations; a final answer at the boundary is still accepted |
GRAPH_RECURSION_LIMIT |
Increase for complex graphs |
BASE_TOKEN_THRESHOLD |
Delay compression by increasing this value |
TOKEN_GROWTH_FACTOR |
Lower values compress more aggressively |
Observability
Optional tracing with Langfuse captures every LLM call, tool invocation, and graph transition. It is useful for debugging agent behavior, tracking costs, and evaluating retrieval quality.
Enabling Langfuse
- Sign up on Langfuse Cloud, create an organization, then create a project and generate API keys from the project settings.
- Set environment variables (or copy
.env.exampleto.env):
export LANGFUSE_ENABLED=true
export LANGFUSE_PUBLIC_KEY=pk-lf-...
export LANGFUSE_SECRET_KEY=sk-lf-...
export LANGFUSE_BASE_URL=https://cloud.langfuse.com # or your self-hosted URL
- Run the app normally. Traces appear in your Langfuse dashboard.
To disable tracing, set LANGFUSE_ENABLED=false or leave the variables unset. The application runs identically either way.
For additional details on integrating Langfuse with LangChain or LangGraph, see the official documentation.
What gets traced
| Component | Traced operations |
|---|---|
| Graph nodes | summarize_history, rewrite_query, orchestrator, compress_context, fallback_response, aggregate_answers |
| Tools | search_child_chunks, retrieve_parent_chunks (arguments + results) |
| Structured output | QueryAnalysis parsing in the rewrite step |
| Subgraph fan-out | Parallel agent invocations via Send() |
Hosting options
- Langfuse Cloud — sign up at cloud.langfuse.com and check the current plan limits there.
- Self-hosted — MIT-licensed, deploy with Docker Compose. See the official self-hosting docs.
For a conceptual tracing guide and platform context, see notebooks/observability.ipynb.
Advanced Topics
Customizing the RAG Agent
Location: project/rag_agent/
Add/Remove Nodes: Edit graph.py and nodes.py
Example: Adding a fact-checking node
# In nodes.py
def fact_check_node(state):
# Your fact-checking logic
return {"fact_checked": True}
# In graph.py
builder.add_node("fact_check", fact_check_node)
builder.add_edge("retrieve", "fact_check")
Modify Conditional Routing: Edit edges.py to change graph flow logic
Example from the system - routing based on query clarity:
def route_after_rewrite(state: State) -> Literal["request_clarification", "agent"]:
"""Routes to human input if question unclear, otherwise processes all rewritten queries"""
if not state.get("questionIsClear", False):
return "request_clarification"
else:
# Fan-out: send each rewritten question to parallel processing
return [
Send("agent", {"question": query, "question_index": idx, "messages": []})
for idx, query in enumerate(state["rewrittenQuestions"])
]
This pattern allows the agent to either request clarification from the user or fan-out multiple query variations for parallel retrieval.
Modify Prompts: Edit prompts.py to change agent behavior and response style
Add Custom Tools: Extend tools.py with new retrieval strategies or external integrations
Replacing Storage Backends
Vector Database:
- Default: Local Qdrant
- Alternatives: Remote Qdrant Cloud, Pinecone, Weaviate
- Edit:
project/db/vector_db_manager.py
Parent Store:
- Default: JSON file
- Alternatives: PostgreSQL, MongoDB, S3
- Edit:
project/db/parent_store_manager.py
Extending the UI
Location: project/ui/gradio_app.py
Add runtime settings, admin panels, or analytics:
with gr.Accordion("Advanced Settings", open=False):
provider_dropdown = gr.Dropdown(
choices=["openai", "anthropic", "google", "ollama"],
label="LLM Provider"
)
Docker Deployment
The root Dockerfile installs only runtime dependencies using uv and CPU
PyTorch. It serves the existing Gradio frontend and backend together; there is
no separate frontend build or Ollama process. MiniLM reduces memory use,
but PyTorch and PDF indexing still need more than a 512 MB host. If selecting
Qwen embeddings instead, allow several GB of RAM; 8 GB is a starting budget.
Model weights download on first startup, so the initial start takes longer.
Build and Run
docker build -t agentic-rag .
docker run --name rag-assistant --env-file project\.env -p 7860:7860 agentic-rag
For persistent documents and indexes:
docker run --name rag-assistant --env-file project\.env -e DATA_DIR=/data -v rag-data:/data -p 7860:7860 agentic-rag
Once running, open http://localhost:7860.
Hugging Face Spaces (Recommended)
- Create a Gradio Space and select ZeroGPU Free if offered for your
account. Do not select Docker. The root README already specifies
sdk: gradio,app_file: project/app.py, the Gradio version, and Python 3.12.12 (supported by ZeroGPU). - Upload the root
README.md,requirements.txt,LICENSE, and the fullprojectfolder (includingproject/requirements.txt). Do not upload.env,.venv, local documents, or local database files. - Under Settings > Variables and secrets, add
GROQ_API_KEYas a secret, setGROQ_MODEL=openai/gpt-oss-20bas a variable (or remove an old override to use this default), and add both authentication values as secrets if the Space is public. - Wait for the build and first model download, then open the App tab. Upload a small Markdown/PDF file under Documents, click Add Documents, and ask a question under Chat.
The public Gradio API is named /chat. Document-management callbacks are
UI-only (api_visibility="private"), so API clients do not select the file
upload callback when sending text. This is not per-user document isolation.
After a rebuild, hard-refresh open app tabs to discard stale event IDs and
queue sessions. A queue-stream 404 can mean that its session no longer exists;
it is not evidence of a missing document or a Groq failure.
The app loads MiniLM on CUDA at startup when HF sets SPACES_ZERO_GPU=true.
Dense embedding inference runs inside @spaces.GPU(duration=30), both during
document indexing and query retrieval. Only texts and embedding results cross
the worker boundary; local Qdrant writes and Groq calls stay outside GPU tasks.
Startup obtains the vector dimension from model metadata and validates the
Qdrant schema without making an inference request.
ZeroGPU requires a registered GPU function during startup; a CPU-only app is not sufficient on this hardware tier. Embedding requests consume the visitor's GPU quota and may queue. Large indexing jobs can require multiple allocations. CPU embedding inference is retained for local and non-ZeroGPU deployments. Verify first startup and indexing in the Space logs.
The UI is branded Document Chat, with a teal-accented header, a compact chat panel, and the same Documents and Chat workflow.
Per the ZeroGPU documentation,
free personal accounts in good standing (verified email and older than 30 days)
can host up to two ZeroGPU Spaces. Availability is account-dependent.
Gradio Spaces manage dependency installation from root requirements.txt;
uv remains the tool for local installation and the optional Docker build.
Documents and indexes use ephemeral local disk and may be lost on
restart/rebuild. Chat memory is always in memory. Do not use this demo as
durable storage. DATA_DIR can point to an existing writable mount on hosts
that provide one, but does not provision persistence.
Render (Alternative)
Create a Docker Web Service using the root Dockerfile with the repository
root as the build context. Set GROQ_API_KEY as a secret. The app honors
Render's PORT and listens on 0.0.0.0 in Docker. Choose a paid instance with
enough RAM for the embedding model, and attach a disk at /data with
DATA_DIR=/data if documents must survive deploys. Use one instance/worker.
The app defaults to DENSE_MODEL=sentence-transformers/all-MiniLM-L6-v2;
this trades retrieval quality for lower memory use compared with Qwen.
Render's free instances sleep and lose local files; do not rely on them for
this model-heavy app. See free-service limits.
Vercel
Not recommended for the full app: it depends on a long-running Gradio server, in-memory LangGraph checkpoints, local Qdrant files, and model downloads. Vercel can host a future separate JavaScript frontend, but the current Python service should remain on Spaces or Render.
Troubleshooting
| Issue | Cause | Solution |
|---|---|---|
| "Model not found" error | Incorrect model name for provider | Verify LLM_MODEL matches provider's API (e.g., gpt-4o-mini not gpt4-mini) |
| Low-quality retrieval results | Poor embedding model or chunk configuration | Re-index with better embeddings or adjust chunk sizes |
| Vector size mismatch | Existing Qdrant collection was built with a different embedding model | Clear the collection and re-index documents after changing DENSE_MODEL |
| Slow response times | Large embedding model or high top_k value |
Use smaller embedding models (e.g., all-MiniLM-L6-v2) or reduce top_k in retrieval tools |
| API rate limits exceeded | Too many requests to external provider | Add retry logic with exponential backoff or switch to local Ollama models |
| Out of memory errors | Large document set or embedding model | Use smaller embeddings, reduce batch size, or enable GPU acceleration |
| Empty retrieval results | Collection not indexed or wrong collection name | Verify documents are uploaded and CHILD_COLLECTION name matches in config |
| Import errors after provider switch | Missing SDK installation | Install required package: pip install langchain-{provider} |
| Inconsistent answers across runs | High temperature setting | Set LLM_TEMPERATURE = 0 in config for deterministic responses |