Spaces:
Sleeping
Sleeping
File size: 30,277 Bytes
b3b0ccd 717b14e b3b0ccd 717b14e b3b0ccd 717b14e b3b0ccd 45b0535 b3b0ccd 717b14e b3b0ccd 3366ea9 b3b0ccd 3366ea9 b3b0ccd 717b14e b3b0ccd 717b14e b3b0ccd 717b14e b3b0ccd 3366ea9 b3b0ccd 3366ea9 717b14e b3b0ccd | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 269 270 271 272 273 274 275 276 277 278 279 280 281 282 283 284 285 286 287 288 289 290 291 292 293 294 295 296 297 298 299 300 301 302 303 304 305 306 307 308 309 310 311 312 313 314 315 316 317 318 319 320 321 322 323 324 325 326 327 328 329 330 331 332 333 334 335 336 337 338 339 340 341 342 343 344 345 346 347 348 349 350 351 352 353 354 355 356 357 358 359 360 361 362 363 364 365 366 367 368 369 370 371 372 373 374 375 376 377 378 379 380 381 382 383 384 385 386 387 388 389 390 391 392 393 394 395 396 397 398 399 400 401 402 403 404 405 406 407 408 409 410 411 412 413 414 415 416 417 418 419 420 421 422 423 424 425 426 427 428 429 430 431 432 433 434 435 436 437 438 439 440 441 442 443 444 445 446 447 448 449 450 451 452 453 454 455 456 457 458 459 460 461 462 463 464 465 466 467 468 469 470 471 472 473 474 475 476 477 478 479 480 481 482 483 484 485 486 487 488 489 490 491 492 493 494 495 496 497 498 499 500 501 502 503 504 505 506 507 508 509 510 511 512 513 514 515 516 517 518 519 520 521 522 523 524 525 526 527 528 529 530 531 532 533 534 535 536 537 538 539 540 541 542 543 544 545 546 547 548 549 550 551 552 553 554 555 556 557 558 559 560 561 562 563 564 565 566 567 568 569 570 571 572 573 574 575 576 577 578 579 580 581 582 583 584 585 586 587 588 589 590 591 592 593 594 595 596 597 598 599 600 601 602 603 604 605 606 607 608 609 610 611 612 613 614 615 616 617 618 619 620 621 622 623 624 625 626 627 628 629 630 631 632 633 634 635 636 637 638 639 640 641 642 643 644 645 646 647 648 649 650 651 652 653 654 655 656 657 658 659 660 661 662 663 664 665 666 667 668 669 670 671 672 673 674 675 676 677 678 679 680 681 682 683 684 685 686 687 688 689 690 691 692 693 694 695 696 697 698 699 700 701 702 703 704 705 706 707 708 709 710 711 712 713 714 715 716 717 718 719 720 721 722 723 724 725 726 727 728 729 730 731 732 733 734 735 736 737 738 739 740 | # Document Chat Documentation
An **Agentic Retrieval-Augmented Generation (RAG)** system built with **LangGraph**, featuring **parent–child chunking**, **hybrid dense + sparse retrieval**, and Groq-powered chat with a Gradio frontend. Embeddings use ZeroGPU on HF Spaces and CPU locally; no Ollama server is required. The learning notebooks retain their Ollama setup.
## Table of Contents
[Quick Start](#quick-start) | [Hugging Face Deployment](#hugging-face-spaces-recommended) | [Architecture Overview](#architecture-overview) | [Project Structure](#project-structure) | [Configuration Guide](#configuration-guide) | [Common Customizations](#common-customizations) | [Observability](#observability) | [Advanced Topics](#advanced-topics) | [Troubleshooting](#troubleshooting)
---
## Quick Start
### Installation
From the repository root, use `uv` (Windows PowerShell):
```powershell
uv venv --python 3.13 .venv
uv pip install --python .venv\Scripts\python.exe --torch-backend cpu -r requirements.txt
Copy-Item project\.env.example project\.env
```
Set `GROQ_API_KEY` in `project/.env`. Get a key from
https://console.groq.com/keys. Never commit `.env` or expose the key in frontend
code. The root `requirements.txt` is the standalone runtime dependency manifest.
HF Gradio mounts only this file during installation, so it must not include
other local requirement files. Notebook/evaluation tools are in
`requirements-notebooks.txt`. The optional `project/requirements.txt` redirects
local installs to the root manifest; Docker installs the root file directly.
### Running the Application
Start the Gradio interface locally:
```powershell
uv run --no-project --python .venv\Scripts\python.exe project\app.py
```
The application will be available at `http://localhost:7860` (default Gradio port).
> Chat memory is isolated per browser session and kept in memory, so it is lost
> on restart. Documents are a **shared library**, not private per-user storage:
> anyone with app access can query, upload, or clear them. Use a private Space
> or set both `GRADIO_AUTH_USERNAME` and `GRADIO_AUTH_PASSWORD` as secrets.
> This also protects your Groq quota. Run one app process: local Qdrant cannot
> be opened by multiple workers at the same path.
### Prerequisites
- Python 3.11+
- A Groq API key; outbound access to Groq and Hugging Face model downloads
---
## Architecture Overview
This system implements an advanced RAG pipeline with the following key features:
- **Parent-Child Chunking**: Documents are split into small child chunks (for precise retrieval) linked to larger parent chunks (for rich context)
- **Hybrid Search**: Combines dense embeddings and sparse (BM25) retrieval for optimal results
- **LangGraph Agent**: Orchestrates query rewriting, retrieval, and response generation
- **Compact Chat Memory**: Rewrites follow-ups using both a rolling summary and bounded recent conversation history
- **Clarification Memory**: Preserves an unresolved query across human-in-the-loop pauses, combines its clarifications, then discards those temporary replies after the answer
- **Document-Aware Query Analysis**: Receives the current uploaded filenames on every turn, resolves single-document references, and allows new requests to replace pending clarifications
- **Document Inventory Answers**: Lists uploaded filenames from the library without vector retrieval or a GPU allocation
- **Groq Chat**: Tool calling, structured query analysis, and streamed responses through `ChatGroq`
- **Vector Storage**: Uses Qdrant for efficient similarity search
### Data Flow
```
PDF → Markdown Conversion → Parent/Child Chunking → Vector Indexing → Agent Retrieval → LLM Response
```
---
## Project Structure
### Entry Point & Configuration
| File | Purpose |
|------|---------|
| `project/app.py` | Application entry point, launches Gradio UI |
| `project/config.py` | **Central configuration hub** - edit this for provider/model/chunking changes |
| `project/utils.py` | PDF conversion and cached context-token estimation with an offline-safe fallback |
| `project/document_chunker.py` | Parent/child splitting logic with cleaning and merging rules |
| `Dockerfile` | uv-based CPU runtime for the backend and Gradio frontend |
### Core System
| File | Purpose |
|------|---------|
| `project/core/rag_system.py` | System bootstrap - creates managers and compiles LangGraph agent |
| `project/core/document_manager.py` | Document ingestion pipeline (convert, chunk, index) |
| `project/core/chat_interface.py` | Streams the aggregated answer while separating query analysis and tool activity from internal node output |
| `project/core/observability.py` | Optional Langfuse tracing — callback handler lifecycle |
### Database Layer
| File | Purpose |
|------|---------|
| `project/db/vector_db_manager.py` | Qdrant client wrapper with inference-free schema validation |
| `project/db/dense_embeddings.py` | MiniLM embeddings with ZeroGPU inference on Spaces and CPU inference locally |
| `project/db/parent_store_manager.py` | File-backed storage for parent chunks |
### RAG Agent (LangGraph)
| File | Purpose |
|------|---------|
| `project/rag_agent/graph.py` | Graph builder and compilation logic |
| `project/rag_agent/graph_state.py` | Shared and per-agent state, including rolling memory, pending clarification, and answer accumulation/reset logic |
| `project/rag_agent/nodes.py` | Node implementations (summarize, rewrite, agent execution, aggregate) |
| `project/rag_agent/edges.py` | Conditional edge routing logic (e.g., routing based on query clarity) |
| `project/rag_agent/tools.py` | Retrieval tools (`search_child_chunks`, `retrieve_parent_chunks`) |
| `project/rag_agent/prompts.py` | System prompts for agent behavior |
| `project/rag_agent/schemas.py` | Structured output schemas (Pydantic models) |
### User Interface
| File | Purpose |
|------|---------|
| `project/ui/css.py` | Custom CSS styling for the Gradio interface |
| `project/ui/gradio_app.py` | Gradio UI implementation with document upload and chat |
---
## Configuration Guide
All primary settings are in `project/config.py`. Key parameters:
### Directory Configuration
```python
MARKDOWN_DIR = "markdown_docs" # Storage for converted PDF → Markdown files
PARENT_STORE_PATH = "parent_store" # File-backed storage for parent chunks
QDRANT_DB_PATH = "qdrant_db" # Local Qdrant vector database path
```
### Qdrant Configuration
```python
CHILD_COLLECTION = "document_child_chunks" # Collection name for child chunks
SPARSE_VECTOR_NAME = "sparse" # Named sparse vector field (BM25)
```
### Model Configuration
```python
# Default: single model configuration
DENSE_MODEL = "sentence-transformers/all-MiniLM-L6-v2"
SPARSE_MODEL = "Qdrant/bm25"
LLM_MODEL = os.environ.get("GROQ_MODEL", "openai/gpt-oss-20b")
JUDGE_MODEL = "ministral-3:3b-instruct-2512-q8_0"
LLM_TEMPERATURE = 0 # 0 = deterministic, 1 = creative
LLM_SEED = 42
LLM_TIMEOUT = 60
LLM_MAX_RETRIES = 2
```
`GROQ_API_KEY` is required at startup. `DENSE_MODEL` and `DATA_DIR` can also be
set as environment variables. `JUDGE_MODEL` remains the local Ollama judge for
the evaluation notebook, not the deployed app. Query analysis explicitly uses
function calling, so it does not require Groq's model-specific JSON Schema mode.
### Retrieval Configuration
```python
RETRIEVAL_SCORE_THRESHOLD = 0.4 # Lower = more recall, higher = more precision
DEFAULT_RETRIEVAL_K = 7 # Default number of child chunks used by retrieval and evaluation
CHILD_CHUNK_SEPARATOR = "\n\n<CHILD_CHUNK_BOUNDARY>\n\n" # Keeps ranked child results separable for evaluation
```
### Agent Configuration
```python
# Hard limits to prevent infinite loops
MAX_TOOL_CALLS = 8 # Maximum tool calls per agent run
MAX_ITERATIONS = 10 # Maximum agent loop iterations
GRAPH_RECURSION_LIMIT = 50 # Maximum number of steps before hitting a stop condition
MAIN_HISTORY_MESSAGES_TO_KEEP = 4 # Raw messages retained after each answer; minimum 2
# Context compression thresholds
BASE_TOKEN_THRESHOLD = 2000 # Initial token threshold for compression
TOKEN_GROWTH_FACTOR = 0.9 # Multiplier applied after each compression
```
### Terminal Execution Logging
```python
EXECUTION_LOGGING_ENABLED = False # Print graph steps, state previews, tool calls, and outputs
EXECUTION_LOG_MAX_CHARS = 1200 # Maximum characters shown for long values
EXECUTION_LOG_USE_COLOR = True # Use ANSI colors in the terminal
```
### Text Splitter Configuration
```python
CHILD_CHUNK_SIZE = 500 # Size of chunks used for retrieval
CHILD_CHUNK_OVERLAP = 100 # Overlap between chunks (prevents context loss)
MIN_PARENT_SIZE = 2000 # Target minimum; very short documents may remain smaller
MAX_PARENT_SIZE = 4000 # Maximum parent chunk size
# Markdown header splitting strategy
HEADERS_TO_SPLIT_ON = [
("#", "H1"),
("##", "H2"),
("###", "H3")
]
```
### Langfuse Observability (Optional)
```python
import os
LANGFUSE_ENABLED = os.environ.get("LANGFUSE_ENABLED", "false").lower() == "true"
LANGFUSE_PUBLIC_KEY = "" # From your Langfuse project settings
LANGFUSE_SECRET_KEY = "" # From your Langfuse project settings
LANGFUSE_BASE_URL = "http://localhost:3000" # Langfuse Cloud or self-hosted URL
```
---
## Common Customizations
### 1. Switching LLM Provider (Single Provider)
> **Performance Note:** LLMs with 8B+ parameters typically offer superior reasoning, context comprehension, and response quality compared to smaller models. This applies to both proprietary and open-source models, as long as they **support native tool/function calling,** which is required for agentic RAG workflows.
If you want to switch from Groq to another provider (e.g., Google Gemini), follow these steps:
**Step 1:** Install the provider's SDK
```bash
uv pip install langchain-google-genai
```
**Step 2:** Set environment variable
```bash
export GOOGLE_API_KEY="your-google-key"
```
**Step 3:** Update `project/config.py`
```python
LLM_MODEL = "gemini-2.5-pro"
LLM_TEMPERATURE = 0
LLM_SEED = 42
```
**Step 4:** Modify `project/core/rag_system.py`
Replace:
```python
llm = ChatGroq(model=config.LLM_MODEL, temperature=config.LLM_TEMPERATURE,
timeout=config.LLM_TIMEOUT, max_retries=config.LLM_MAX_RETRIES,
model_kwargs={"seed": config.LLM_SEED})
```
With:
```python
from langchain_google_genai import ChatGoogleGenerativeAI
llm = ChatGoogleGenerativeAI(model=config.LLM_MODEL, temperature=config.LLM_TEMPERATURE)
```
### 2. Optional Multi-Provider Configuration
The application ships with a Groq implementation. The following is an optional
customization example, not the shipped configuration. Remove the Groq-only
startup key check if implementing a different provider.
This approach allows you to maintain multiple provider configurations and switch between them easily.
**Step 1:** Install required SDKs
```bash
uv pip install langchain-openai langchain-anthropic langchain-google-genai
```
**Step 2:** Set environment variables
```bash
export OPENAI_API_KEY="your-openai-key"
export ANTHROPIC_API_KEY="your-anthropic-key"
export GOOGLE_API_KEY="your-google-key"
```
**Step 3:** Update `project/config.py` with multi-provider configuration
```python
# --- Multi-Provider LLM Configuration ---
LLM_CONFIGS = {
"ollama": {
"model": "granite4.1:8b",
"url":"http://localhost:11434",
"temperature": 0
},
"openai": {
"model": "gpt-5.2",
"temperature": 0
},
"anthropic": {
"model": "claude-sonnet-4-6",
"temperature": 0
},
"google": {
"model": "gemini-2.5-flash",
"temperature": 0
}
}
# Switch providers by changing this single line
ACTIVE_LLM_CONFIG = "ollama"
```
**Step 4:** Modify `project/core/rag_system.py` in the `initialize()` method
Replace the existing LLM initialization with:
```python
def initialize(self):
self.vector_db.create_collection(self.collection_name)
collection = self.vector_db.get_collection(self.collection_name)
# Load active configuration
active_config = config.LLM_CONFIGS[config.ACTIVE_LLM_CONFIG]
model = active_config["model"]
temperature = active_config["temperature"]
if config.ACTIVE_LLM_CONFIG == "ollama":
from langchain_ollama import ChatOllama
llm = ChatOllama(model=model, temperature=temperature, base_url=active_config["url"])
elif config.ACTIVE_LLM_CONFIG == "openai":
from langchain_openai import ChatOpenAI
llm = ChatOpenAI(model=model, temperature=temperature)
elif config.ACTIVE_LLM_CONFIG == "anthropic":
from langchain_anthropic import ChatAnthropic
llm = ChatAnthropic(model=model, temperature=temperature)
elif config.ACTIVE_LLM_CONFIG == "google":
from langchain_google_genai import ChatGoogleGenerativeAI
llm = ChatGoogleGenerativeAI(model=model, temperature=temperature)
else:
raise ValueError(f"Unsupported LLM provider: {config.ACTIVE_LLM_CONFIG}")
# Continue with tool and graph initialization
tools = ToolFactory(collection).create_tools()
self.agent_graph = create_agent_graph(llm, tools)
```
**Switching Providers:** Simply change `ACTIVE_LLM_CONFIG` in `config.py`:
```python
ACTIVE_LLM_CONFIG = "google" # Switch to Gemini Pro
# ACTIVE_LLM_CONFIG = "anthropic" # Or Claude Sonnet
# ACTIVE_LLM_CONFIG = "openai" # Or GPT-4o
```
---
**Provider Reference Table:**
| Provider | Environment Variable | Import Statement | Example Models |
|----------|---------------------|------------------|----------------|
| Groq | `GROQ_API_KEY` | `from langchain_groq import ChatGroq` | `openai/gpt-oss-20b` |
| OpenAI | `OPENAI_API_KEY` | `from langchain_openai import ChatOpenAI` | `gpt-5.2`, `gpt-5-mini` |
| Anthropic | `ANTHROPIC_API_KEY` | `from langchain_anthropic import ChatAnthropic` | `claude-opus-4-6`, `claude-sonnet-4-6` |
| Google | `GOOGLE_API_KEY` | `from langchain_google_genai import ChatGoogleGenerativeAI` | `gemini-2.5-pro`, `gemini-2.5-flash` |
| Ollama | None (local) | `from langchain_ollama import ChatOllama` | `granite4.1:8b`, `llama3.1:8b-instruct-q6_K` |
---
### 3. Changing Embedding Models
**Why change?** Trade-offs between speed, cost, and quality.
**Step 1:** Update `project/config.py`
```python
# Default: Compact model (ZeroGPU on HF, CPU locally)
DENSE_MODEL = "sentence-transformers/all-MiniLM-L6-v2"
# Example: Alternative Sentence Transformers model
# DENSE_MODEL = "sentence-transformers/all-mpnet-base-v2"
# Example: Gemma embeddings (Google's open model)
# DENSE_MODEL = "google/embeddinggemma-300m"
# Optional: Qwen embeddings (Alibaba's multilingual model)
# DENSE_MODEL = "Qwen/Qwen3-Embedding-0.6B"
SPARSE_MODEL = "Qdrant/bm25" # Usually no need to change
```
**Step 2:** Re-index your documents
⚠️ **Important:** Changing embeddings requires re-indexing. In the educational UI, use **Clear All**, restart if the old collection prevents startup, then upload the documents again.
**Implementation Details** (in `project/db/vector_db_manager.py`):
```python
from langchain_huggingface import HuggingFaceEmbeddings
from langchain_qdrant import FastEmbedSparse
import config
from db.dense_embeddings import DenseEmbeddings
self.__dense_embeddings = DenseEmbeddings(model_name=config.DENSE_MODEL)
self.__sparse_embeddings = FastEmbedSparse(model_name=config.SPARSE_MODEL)
```
**Popular Embedding Models:**
| Model | Context Size | Vector Dimension | Speed | Quality | Use Case |
|-------|--------------|------------------|-------|---------|----------|
| all-MiniLM-L6-v2 | 256 tokens | 384 | Fast | Good | Default compact embedding model |
| all-mpnet-base-v2 | 512 tokens | 768 | Medium | Excellent | High-accuracy semantic search |
| bge-large-en-v1.5 | 512 tokens | 1024 | Slow | Best | Production-grade retrieval on GPU |
| google/embeddinggemma-300m | 2048 tokens | 768 | Fast | Very Good | Lightweight, efficient multilingual retrieval |
| Qwen/Qwen3-Embedding-0.6B | 32768 tokens | 1024 | Medium | Excellent | Optional multilingual/code retrieval model |
| Qwen/Qwen3-Embedding-8B | 32768 tokens | 4096 | Very Slow | Excellent / SOTA | Heavy GPU-oriented retrieval experiments |
---
### 4. Adjusting Chunking Strategy
**Why adjust?** Balance between retrieval precision and context richness.
> Before re-indexing, inspect Markdown output and chunk boundaries to verify the document structure.
**Step 1:** Update chunk sizes in `project/config.py`
```python
# For short, factual queries (e.g., technical documentation)
CHILD_CHUNK_SIZE = 300
CHILD_CHUNK_OVERLAP = 60
MIN_PARENT_SIZE = 1500
MAX_PARENT_SIZE = 8000
# For narrative or contextual queries (e.g., legal documents)
# CHILD_CHUNK_SIZE = 800
# CHILD_CHUNK_OVERLAP = 150
# MIN_PARENT_SIZE = 3000
# MAX_PARENT_SIZE = 15000
```
**Step 2 (Optional):** Replace the splitter in `project/document_chunker.py`
**Default (Character-based):**
```python
from langchain_text_splitters import RecursiveCharacterTextSplitter
self.__child_splitter = RecursiveCharacterTextSplitter(
chunk_size=config.CHILD_CHUNK_SIZE,
chunk_overlap=config.CHILD_CHUNK_OVERLAP
)
```
**Alternative (Sentence-aware):**
```python
from langchain_text_splitters import SentenceTransformersTokenTextSplitter
self.__child_splitter = SentenceTransformersTokenTextSplitter(
chunk_size=config.CHILD_CHUNK_SIZE,
chunk_overlap=config.CHILD_CHUNK_OVERLAP
)
```
**Step 3:** Re-run ingestion
Use **Clear All** in the Gradio interface, then upload the documents again. Existing documents are intentionally skipped to avoid accidental duplicates.
**Chunking Guidelines:**
> ⚠️ **Disclaimer:** These are empirical guidelines. Optimal sizes depend on:
> - **Child chunk** → embedding model's context window (e.g. 256 tokens for all-MiniLM-L6-v2, 512 for bge-large-en-v1.5): child size should not exceed it
> - **Parent chunk** → generative model's context window (e.g. 8K, 32K, 128K tokens): parent must fit within the context sent to the LLM alongside the query
>
> Always validate values empirically on your own corpus.
The chunker enforces `MAX_PARENT_SIZE`, deduplicates merged header metadata, and rebalances small neighboring chunks when possible. A document shorter than `MIN_PARENT_SIZE` remains a single smaller chunk.
| Document Type | Child Size | Parent Size | Reasoning |
|---------------|-----------|-------------|-----------|
| Technical Docs | 300-500 | 2000-4000 | Precise lookups, code snippets |
| Legal Contracts | 600-1000 | 5000-15000 | Context-heavy, definitions |
| Research Papers | 400-600 | 3000-8000 | Balance of precision and context |
| FAQs / Knowledge Base | 200-400 | 1500-4000 | Short, focused answers |
---
### 5. Agent Configuration
Tune agent behavior in `project/config.py`:
```python
# Hard limits to prevent infinite loops
MAX_TOOL_CALLS = 8 # Maximum tool calls per agent run
MAX_ITERATIONS = 10 # Maximum agent loop iterations
GRAPH_RECURSION_LIMIT = 50 # Maximum number of steps before hitting a stop condition
MAIN_HISTORY_MESSAGES_TO_KEEP = 4 # Raw messages retained after each answer; minimum 2
# Context compression thresholds
BASE_TOKEN_THRESHOLD = 2000 # Initial token threshold for compression
TOKEN_GROWTH_FACTOR = 0.9 # Multiplier applied after each compression
```
| Parameter | Effect |
|-----------|--------|
| `MAX_TOOL_CALLS` | Maximum cumulative tool calls requested; a request that would exceed it is not executed |
| `MAX_ITERATIONS` | Maximum LLM reasoning iterations; a final answer at the boundary is still accepted |
| `GRAPH_RECURSION_LIMIT` | Increase for complex [graphs](https://docs.langchain.com/oss/python/langgraph/errors/GRAPH_RECURSION_LIMIT) |
| `BASE_TOKEN_THRESHOLD` | Delay compression by increasing this value |
| `TOKEN_GROWTH_FACTOR` | Lower values compress more aggressively |
---
## Observability
Optional tracing with [Langfuse](https://langfuse.com) captures every LLM call, tool invocation, and graph transition. It is useful for debugging agent behavior, tracking costs, and evaluating retrieval quality.
### Enabling Langfuse
1. Sign up on [Langfuse Cloud](https://cloud.langfuse.com/), create an organization, then create a project and generate API keys from the project settings.
2. Set environment variables (or copy `.env.example` to `.env`):
```bash
export LANGFUSE_ENABLED=true
export LANGFUSE_PUBLIC_KEY=pk-lf-...
export LANGFUSE_SECRET_KEY=sk-lf-...
export LANGFUSE_BASE_URL=https://cloud.langfuse.com # or your self-hosted URL
```
3. Run the app normally. Traces appear in your [Langfuse dashboard](https://cloud.langfuse.com/).
To disable tracing, set `LANGFUSE_ENABLED=false` or leave the variables unset. The application runs identically either way.
For additional details on integrating Langfuse with LangChain or LangGraph, see the official [documentation](https://docs.langchain.com/oss/python/integrations/providers/langfuse).
### What gets traced
| Component | Traced operations |
|-----------|-------------------|
| Graph nodes | `summarize_history`, `rewrite_query`, `orchestrator`, `compress_context`, `fallback_response`, `aggregate_answers` |
| Tools | `search_child_chunks`, `retrieve_parent_chunks` (arguments + results) |
| Structured output | `QueryAnalysis` parsing in the rewrite step |
| Subgraph fan-out | Parallel agent invocations via `Send()` |
### Hosting options
- **Langfuse Cloud** — sign up at [cloud.langfuse.com](https://cloud.langfuse.com) and check the current plan limits there.
- **Self-hosted** — MIT-licensed, deploy with Docker Compose. See the [official self-hosting docs](https://langfuse.com/self-hosting).
For a conceptual tracing guide and platform context, see [`notebooks/observability.ipynb`](../notebooks/observability.ipynb).
---
## Advanced Topics
### Customizing the RAG Agent
**Location:** `project/rag_agent/`
**Add/Remove Nodes:** Edit `graph.py` and `nodes.py`
Example: Adding a fact-checking node
```python
# In nodes.py
def fact_check_node(state):
# Your fact-checking logic
return {"fact_checked": True}
# In graph.py
builder.add_node("fact_check", fact_check_node)
builder.add_edge("retrieve", "fact_check")
```
**Modify Conditional Routing:** Edit `edges.py` to change graph flow logic
Example from the system - routing based on query clarity:
```python
def route_after_rewrite(state: State) -> Literal["request_clarification", "agent"]:
"""Routes to human input if question unclear, otherwise processes all rewritten queries"""
if not state.get("questionIsClear", False):
return "request_clarification"
else:
# Fan-out: send each rewritten question to parallel processing
return [
Send("agent", {"question": query, "question_index": idx, "messages": []})
for idx, query in enumerate(state["rewrittenQuestions"])
]
```
This pattern allows the agent to either request clarification from the user or fan-out multiple query variations for parallel retrieval.
**Modify Prompts:** Edit `prompts.py` to change agent behavior and response style
**Add Custom Tools:** Extend `tools.py` with new retrieval strategies or external integrations
### Replacing Storage Backends
**Vector Database:**
- Default: Local Qdrant
- Alternatives: Remote Qdrant Cloud, Pinecone, Weaviate
- Edit: `project/db/vector_db_manager.py`
**Parent Store:**
- Default: JSON file
- Alternatives: PostgreSQL, MongoDB, S3
- Edit: `project/db/parent_store_manager.py`
### Extending the UI
**Location:** `project/ui/gradio_app.py`
Add runtime settings, admin panels, or analytics:
```python
with gr.Accordion("Advanced Settings", open=False):
provider_dropdown = gr.Dropdown(
choices=["openai", "anthropic", "google", "ollama"],
label="LLM Provider"
)
```
### Docker Deployment
The root `Dockerfile` installs only runtime dependencies using uv and CPU
PyTorch. It serves the existing Gradio frontend and backend together; there is
no separate frontend build or Ollama process. MiniLM reduces memory use,
but PyTorch and PDF indexing still need more than a 512 MB host. If selecting
Qwen embeddings instead, allow several GB of RAM; 8 GB is a starting budget.
Model weights download on first startup, so the initial start takes longer.
#### Build and Run
```powershell
docker build -t agentic-rag .
docker run --name rag-assistant --env-file project\.env -p 7860:7860 agentic-rag
```
For persistent documents and indexes:
```powershell
docker run --name rag-assistant --env-file project\.env -e DATA_DIR=/data -v rag-data:/data -p 7860:7860 agentic-rag
```
Once running, open `http://localhost:7860`.
### Hugging Face Spaces (Recommended)
1. Create a **Gradio Space** and select **ZeroGPU Free** if offered for your
account. Do not select Docker. The root README already specifies
`sdk: gradio`, `app_file: project/app.py`, the Gradio version, and Python
3.12.12 (supported by ZeroGPU).
2. Upload the root `README.md`, `requirements.txt`, `LICENSE`, and the full
`project` folder (including `project/requirements.txt`).
Do not upload `.env`, `.venv`, local documents, or local database files.
3. Under **Settings > Variables and secrets**, add `GROQ_API_KEY` as a
**secret**, set `GROQ_MODEL=openai/gpt-oss-20b` as a variable (or remove an
old override to use this default), and add both authentication
values as secrets if the Space is public.
4. Wait for the build and first model download, then open the **App** tab.
Upload a small Markdown/PDF file under **Documents**, click **Add Documents**,
and ask a question under **Chat**.
The public Gradio API is named `/chat`. Document-management callbacks are
UI-only (`api_visibility="private"`), so API clients do not select the file
upload callback when sending text. This is not per-user document isolation.
After a rebuild, hard-refresh open app tabs to discard stale event IDs and
queue sessions. A queue-stream 404 can mean that its session no longer exists;
it is not evidence of a missing document or a Groq failure.
The app loads MiniLM on **CUDA** at startup when HF sets `SPACES_ZERO_GPU=true`.
Dense embedding inference runs inside `@spaces.GPU(duration=30)`, both during
document indexing and query retrieval. Only texts and embedding results cross
the worker boundary; local Qdrant writes and Groq calls stay outside GPU tasks.
Startup obtains the vector dimension from model metadata and validates the
Qdrant schema without making an inference request.
ZeroGPU requires a registered GPU function during startup; a CPU-only app is
not sufficient on this hardware tier. Embedding requests consume the visitor's
GPU quota and may queue. Large indexing jobs can require multiple allocations.
CPU embedding inference is retained for local and non-ZeroGPU deployments.
Verify first startup and indexing in the Space logs.
The UI is branded **Document Chat**, with a teal-accented header, a compact
chat panel, and the same **Documents** and **Chat** workflow.
Per the [ZeroGPU documentation](https://huggingface.co/docs/hub/spaces-zerogpu),
free personal accounts in good standing (verified email and older than 30 days)
can host up to two ZeroGPU Spaces. Availability is account-dependent.
Gradio Spaces manage dependency installation from root `requirements.txt`;
uv remains the tool for local installation and the optional Docker build.
Documents and indexes use ephemeral local disk and may be lost on
restart/rebuild. Chat memory is always in memory. Do not use this demo as
durable storage. `DATA_DIR` can point to an existing writable mount on hosts
that provide one, but does not provision persistence.
#### Render (Alternative)
Create a **Docker Web Service** using the root Dockerfile with the repository
root as the build context. Set `GROQ_API_KEY` as a secret. The app honors
Render's `PORT` and listens on `0.0.0.0` in Docker. Choose a paid instance with
enough RAM for the embedding model, and attach a disk at `/data` with
`DATA_DIR=/data` if documents must survive deploys. Use one instance/worker.
The app defaults to `DENSE_MODEL=sentence-transformers/all-MiniLM-L6-v2`;
this trades retrieval quality for lower memory use compared with Qwen.
Render's free instances sleep and lose local files; do not rely on them for
this model-heavy app. See [free-service limits](https://render.com/docs/free).
#### Vercel
Not recommended for the full app: it depends on a long-running Gradio server,
in-memory LangGraph checkpoints, local Qdrant files, and model downloads.
Vercel can host a future separate JavaScript frontend, but the current Python
service should remain on Spaces or Render.
---
## Troubleshooting
| Issue | Cause | Solution |
|-------|-------|----------|
| "Model not found" error | Incorrect model name for provider | Verify `LLM_MODEL` matches provider's API (e.g., `gpt-4o-mini` not `gpt4-mini`) |
| Low-quality retrieval results | Poor embedding model or chunk configuration | Re-index with better embeddings or adjust chunk sizes |
| Vector size mismatch | Existing Qdrant collection was built with a different embedding model | Clear the collection and re-index documents after changing `DENSE_MODEL` |
| Slow response times | Large embedding model or high `top_k` value | Use smaller embedding models (e.g., all-MiniLM-L6-v2) or reduce `top_k` in retrieval tools |
| API rate limits exceeded | Too many requests to external provider | Add retry logic with exponential backoff or switch to local Ollama models |
| Out of memory errors | Large document set or embedding model | Use smaller embeddings, reduce batch size, or enable GPU acceleration |
| Empty retrieval results | Collection not indexed or wrong collection name | Verify documents are uploaded and `CHILD_COLLECTION` name matches in config |
| Import errors after provider switch | Missing SDK installation | Install required package: `pip install langchain-{provider}` |
| Inconsistent answers across runs | High temperature setting | Set `LLM_TEMPERATURE = 0` in config for deterministic responses |
|