RAG-PDF-Project / EXAMPLES_AND_TESTS.md
RAG PDF Assistant
Initial commit: Production-ready RAG PDF Assistant with HuggingFace integration
1b31f0c
|
Raw History Blame Contribute Delete
12 kB
# RAG PDF Project - Usage Examples & Test Cases
## 🎯 Getting Started with Examples
### Setup Check
**Step 1: Verify Installation**
```bash
# Check Python version
python --version # Should be 3.10+
# Check dependencies
pip list | grep -E "gradio|faiss|transformers|sentence-transformers|pypdf"
# Expected output:
# faiss-cpu
# gradio
# sentence-transformers
# transformers
# pypdf
```
**Step 2: Verify Project Structure**
```
rag_pdf_project/
β”œβ”€β”€ app.py βœ…
β”œβ”€β”€ main.py βœ…
β”œβ”€β”€ requirements.txt βœ…
β”œβ”€β”€ configs/settings.py βœ…
β”œβ”€β”€ data/documents/1.pdf βœ…
β”œβ”€β”€ ingestion/ βœ…
β”œβ”€β”€ retrieval/ βœ…
β”œβ”€β”€ generation/ βœ…
└── pipeline/ βœ…
```
**Step 3: Start the Application**
```bash
python app.py
# Output: * Running on local URL: http://127.0.0.1:7860
```
---
## πŸ“‹ Test Cases & Examples
### Test Case 1: Load Default PDF
**Action**:
1. Open http://127.0.0.1:7860
2. Click "Use Default PDF" button
3. Wait for "Status" box to update
**Expected Result**:
```
Status: Ready. Loaded default document: 1.pdf
```
**What Happens Internally**:
```
1. PDF loaded: 1.pdf
2. Text extracted: ~5000 words
3. Split into chunks: ~17 chunks (300 words each)
4. Embedded: 17 Γ— 384-dim vectors created
5. Indexed: FAISS index ready for search
6. UI State: index, chunks, document_name updated
```
---
### Test Case 2: Ask a Simple Question
**Scenario**: Analyze a document about "Machine Learning"
**Question**: "What is the main topic?"
**Expected Flow**:
```
INPUT: "What is the main topic?"
β”‚
β”œβ”€ Embed: "What is the main topic?" β†’ [0.23, -0.15, ..., 0.42]
β”‚
β”œβ”€ Retrieve: Find 3 most similar chunks
β”‚ └─ Chunk 1: "This document covers machine learning..."
β”‚ └─ Chunk 2: "ML is a subset of artificial intelligence..."
β”‚ └─ Chunk 5: "Key topics include supervised learning..."
β”‚
β”œβ”€ Build Context:
β”‚ "This document covers machine learning...
β”‚ ML is a subset of artificial intelligence...
β”‚ Key topics include supervised learning..."
β”‚
β”œβ”€ Build Prompt:
β”‚ "Answer the question using only the context below...
β”‚ Context: [above text]...
β”‚ Question: What is the main topic?
β”‚ Answer:"
β”‚
β”œβ”€ Generate Answer (T5):
β”‚ "The main topic is machine learning,
β”‚ which is a subset of artificial intelligence..."
β”‚
OUTPUT: Display in "Answer" box
```
---
### Test Case 3: Upload Custom PDF
**Action**:
1. Click "Upload PDF"
2. Select a PDF file from your computer
3. Click "Load Uploaded PDF"
4. Wait for status update
**File Requirements**:
- Format: `.pdf` only
- Size: Any size (tested up to 100MB)
- Content: Text extractable (not scanned images)
**Expected Status**:
```
Ready. Loaded uploaded document: your_file.pdf
```
---
### Test Case 4: Multi-turn Conversation
**Scenario**: Ask follow-up questions
**Q1**: "What are the main concepts?"
```
Answer: "The main concepts include..."
```
**Q2**: "Explain concept A in detail"
```
Context: Retrieved chunks about concept A
Answer: "Concept A is..." (focused on specific concept)
```
**Q3**: "How does this relate to the introduction?"
```
Context: Retrieved chunks relevant to relationship
Answer: "This relates to the introduction by..."
```
**Key Point**: Each question is independent (new retrieval), but user context is maintained through conversation history in the UI.
---
## πŸ§ͺ Practical Examples
### Example 1: Analyzing a Research Paper
**Document**: `research_paper.pdf` (10 pages)
**Questions & Expected Answers**:
```
Q: "What problem does this paper solve?"
A: "This paper addresses the challenge of [problem]
by proposing a novel approach that..."
Q: "What is the methodology?"
A: "The methodology involves three main steps:
1. Data preparation...
2. Model training...
3. Evaluation..."
Q: "What are the key results?"
A: "The key results show that the proposed method
achieves [metric] improvements over baseline..."
Q: "What are future directions?"
A: "Future work could focus on [directions mentioned in paper]..."
```
### Example 2: Analyzing Product Documentation
**Document**: `product_manual.pdf` (50 pages)
**Questions & Expected Answers**:
```
Q: "How do I install this product?"
A: "Installation involves the following steps:
1. Unpack the product...
2. Connect to power...
3. Configure settings..."
Q: "What should I do if I get error code X?"
A: "Error code X usually indicates...
Try the following solutions:
1. Restart the device...
2. Check connections..."
Q: "What are the specifications?"
A: "The product specifications are:
- Power: 110V AC...
- Operating temperature: 0-40Β°C...
- Weight: 5kg..."
```
### Example 3: Analyzing Financial Reports
**Document**: `annual_report.pdf` (100 pages)
**Questions & Expected Answers**:
```
Q: "What is the company's revenue for Q3?"
A: "Q3 revenue was $[amount], representing
a [percentage] increase/decrease from Q2..."
Q: "What are the main risks mentioned?"
A: "The company identifies the following key risks:
1. Market volatility...
2. Supply chain disruptions...
3. Regulatory changes..."
Q: "What was discussed in the board letter?"
A: "The board letter highlighted...
Our strategic priorities include..."
```
---
## πŸ“Š Performance Monitoring
### Monitor Query Performance
**Timing Test**:
```
1. Note the time before asking question
2. Ask: "What is this document about?"
3. Note the time when answer appears
4. Calculate elapsed time
Expected: 0.5-2 seconds (depends on answer length)
```
**Performance Breakdown** (typical):
```
- Embedding query: 50-100ms
- FAISS retrieval: 1-5ms
- Prompt building: 1-2ms
- T5 generation: 500-1000ms
─────────────────────────────
- Total: 550-1100ms (~1 second average)
```
### Monitor Quality
**Quality Checklist**:
```
βœ“ Answer is grounded in document (not hallucinated)
βœ“ Answer is relevant to the question
βœ“ Answer is factually accurate
βœ“ Answer is concise but complete
βœ“ No duplicate or nonsensical text
```
**If answers are poor**:
1. Check the document has relevant content
2. Try a different question
3. Increase TOP_K to get more context
4. Check chunk quality (no corrupted text)
---
## πŸ” Debugging Examples
### Debug Case 1: Wrong Answer Format
**Symptom**: Answer contains control characters or formatting issues
**Debug Steps**:
```python
# In generation/llm.py, add debug logging:
def generate(prompt):
# ... existing code ...
output_ids = model.generate(**inputs, max_new_tokens=200)
raw_output = tokenizer.decode(output_ids[0]) # Add this
print(f"DEBUG: Raw output: {repr(raw_output)}") # Add this
clean_output = raw_output.strip()
return clean_output
```
**Solution**: May need to adjust tokenizer decode parameters
### Debug Case 2: No Relevant Chunks Retrieved
**Symptom**: Answer says "I do not know" or is irrelevant
**Debug Steps**:
```python
# Add debug output in retrieval/retriever.py:
def retrieve(index, query_vector):
distances, indices = index.search(query_vector, TOP_K)
print(f"DEBUG: Retrieved indices: {indices[0]}")
print(f"DEBUG: Distances: {distances[0]}")
return indices[0]
```
**Solution**:
- Increase TOP_K from 3 to 5
- Check document has relevant content
- Try different question phrasing
### Debug Case 3: Slow Generation
**Symptom**: Takes >3 seconds per query
**Debug Steps**:
```python
import time
def rag_answer(question, index, chunks):
t0 = time.time()
query_vector = embed([question])
print(f"Embedding: {time.time()-t0:.3f}s")
t1 = time.time()
ids = retrieve(index, query_vector)
print(f"Retrieval: {time.time()-t1:.3f}s")
# ... build context ...
t2 = time.time()
answer = generate(prompt)
print(f"Generation: {time.time()-t2:.3f}s")
return answer
```
**Solution**:
- If generation is slow: Use CPU acceleration or reduce model size
- If retrieval is slow: Check FAISS index size
---
## βœ… Validation Checklist
### Pre-Launch Checklist
- [ ] All files in place (project structure intact)
- [ ] requirements.txt has all dependencies
- [ ] Python version is 3.10 or higher
- [ ] Sample PDF exists in data/documents/1.pdf
- [ ] All settings in configs/settings.py are valid
- [ ] No import errors when running `python app.py`
### Functionality Checklist
- [ ] Web UI loads at http://127.0.0.1:7860
- [ ] "Use Default PDF" button works
- [ ] PDF loading shows status message
- [ ] Can ask questions after loading PDF
- [ ] Answers appear in "Answer" box
- [ ] Questions can be cleared
- [ ] Can upload custom PDFs
- [ ] Answer quality is acceptable
### Performance Checklist
- [ ] First query takes <2 seconds
- [ ] Subsequent queries take <1 second
- [ ] No memory leaks (RAM stable after multiple queries)
- [ ] Models load on first use only
- [ ] UI remains responsive while generating
---
## πŸŽ“ Learning Experiments
### Experiment 1: Chunk Size Impact
**Test**: How does chunk size affect answer quality?
```python
# Edit configs/settings.py and test:
CHUNK_SIZE = 100 # Smaller chunks
CHUNK_SIZE = 300 # Default
CHUNK_SIZE = 500 # Larger chunks
# Compare answers for same question
# Smaller = more focused, Larger = more context
```
### Experiment 2: TOP_K Impact
**Test**: How many retrieved chunks are optimal?
```python
# Edit configs/settings.py and test:
TOP_K = 1 # Minimal context
TOP_K = 3 # Default
TOP_K = 5 # Rich context
TOP_K = 10 # Very rich context
# Observe: Quality vs. inference time tradeoff
```
### Experiment 3: Different LLM Models
**Test**: Compare different generation models
```python
# Edit configs/settings.py and test:
GEN_MODEL = 'google/flan-t5-small' # 250M, fast
GEN_MODEL = 'google/flan-t5-base' # 580M, balanced
GEN_MODEL = 'google/flan-t5-large' # 770M, quality
# Compare: answer quality vs. speed
# Note: Larger models need more memory
```
### Experiment 4: Different Embedding Models
**Test**: Compare embedding models
```python
# Edit configs/settings.py and test:
EMBED_MODEL = 'all-MiniLM-L6-v2' # Fast, English only
EMBED_MODEL = 'paraphrase-multilingual-MiniLM-L12-v2' # Default
EMBED_MODEL = 'all-mpnet-base-v2' # High quality, slower
# Compare: retrieval relevance and speed
```
---
## 🚨 Troubleshooting Decision Tree
```
Application won't start?
β”œβ”€ YES β†’ Check Python version: python --version
β”‚ └─ < 3.10? Install Python 3.10+
β”‚ └─ >= 3.10? Install dependencies: pip install -r requirements.txt
β”‚
β”œβ”€ Web UI won't load?
β”‚ β”œβ”€ Port 7860 already in use?
β”‚ β”‚ └─ Kill process or use different port
β”‚ └─ Check terminal for error messages
β”‚
β”œβ”€ Can't load PDF?
β”‚ β”œβ”€ Does 1.pdf exist?
β”‚ β”‚ └─ NO? Place PDF in data/documents/
β”‚ β”œβ”€ Is it a valid PDF?
β”‚ β”‚ └─ Try with different PDF
β”‚ └─ Enough disk space?
β”‚ └─ Check available storage
β”‚
β”œβ”€ No answer generated?
β”‚ β”œβ”€ Did you load a PDF first?
β”‚ β”‚ └─ Click "Use Default PDF" or upload one
β”‚ β”œβ”€ Is the question empty?
β”‚ β”‚ └─ Type a question
β”‚ └─ Out of memory?
β”‚ └─ Free up RAM or use smaller model
β”‚
└─ Answer is irrelevant?
β”œβ”€ Is it in the document?
β”‚ └─ PDF may not contain answer
β”œβ”€ Try different question phrasing
└─ Increase TOP_K in settings.py
```
---
## πŸ“ž Getting Help
**For issues, check**:
1. QUICK_START.md - Basic usage guide
2. TECHNICAL_DOCS.md - Architecture details
3. Terminal output - Error messages
4. configs/settings.py - Configuration issues
**Common fixes**:
- Reload page: Ctrl+R or Cmd+R
- Restart app: Kill terminal and run app.py again
- Clear browser cache: Ctrl+Shift+Delete
---
**You're now ready to use the RAG PDF Assistant! πŸš€**