docs/PROJECT_DOCUMENTATION.md

RAG-Based Chatbot - Detailed Project Documentation

Table of Contents

1. Project Overview
2. Architecture
3. Technology Stack
4. System Components
5. Implementation Details
6. Deployment Guide
7. Performance Optimization
8. Troubleshooting
9. Future Enhancements

---

1. Project Overview

1.1 Purpose

This project implements a Retrieval-Augmented Generation (RAG) chatbot that combines the power of large language models with a custom knowledge base to provide accurate, context-aware responses to user queries.

1.2 Key Features

• Semantic Search: Uses vector embeddings for intelligent document retrieval
• Context-Aware Responses: Generates answers based on retrieved relevant documents
• Open-Source Stack: Built entirely with open-source tools and models
• Easy Deployment: Ready for deployment on Hugging Face Spaces
• Customizable: Easy to modify knowledge base and models

1.3 Use Cases

• Customer support chatbots
• Internal knowledge base assistants
• Educational Q&A systems
• Documentation helpers
• Domain-specific information retrieval

---

2. Architecture

2.1 High-Level Architecture

User Query
    ↓
[Streamlit UI]
    ↓
[Query Embedding]
    ↓
[FAISS Vector Search] ← [Vector Database]
    ↓                        ↑
[Retrieved Documents]    [Knowledge Base]
    ↓                        ↑
[Prompt Construction]    [Document Chunks]
    ↓
[Language Model (TinyLlama)]
    ↓
[Response Generation]
    ↓
[Display to User]

2.2 Data Flow

1. Initialization Phase:

   • Load knowledge base documents
   • Split documents into chunks
   • Generate embeddings for each chunk
   • Store embeddings in FAISS vector database
   • Load language model into memory

2. Query Phase:

   • User submits a question
   • Question is embedded using the same embedding model
   • FAISS performs similarity search to find top-k relevant chunks
   • Retrieved chunks are formatted as context
   • Context + query are sent to the language model
   • Model generates response
   • Response is displayed with source documents

2.3 Component Interaction

┌─────────────────┐
│   Streamlit UI  │
└────────┬────────┘
         │
    ┌────▼────┐
    │  Cache  │ (st.cache_resource)
    └────┬────┘
         │
    ┌────▼──────────────────┐
    │  Vector Store Manager │
    │  - Load documents     │
    │  - Create embeddings  │
    │  - Build FAISS index  │
    └────┬──────────────────┘
         │
    ┌────▼──────────────┐
    │  LLM Manager      │
    │  - Load model     │
    │  - Load tokenizer │
    └────┬──────────────┘
         │
    ┌────▼──────────────┐
    │  RAG Pipeline     │
    │  - Retrieve docs  │
    │  - Generate       │
    └───────────────────┘

---

3. Technology Stack

3.1 Core Technologies

Component	Technology	Version	Purpose
UI Framework	Streamlit	1.31.0	Web interface
LLM Framework	LangChain	0.1.0	RAG orchestration
Embeddings	Sentence Transformers	2.3.1	Text embeddings
Vector DB	FAISS	1.7.4	Similarity search
LLM	TinyLlama	1.1B	Text generation
ML Framework	PyTorch	2.1.2	Model inference
Transformers	Hugging Face	4.36.2	Model loading

3.2 Why These Technologies?

Streamlit:

• Rapid prototyping
• Built-in caching
• Easy deployment
• Interactive widgets

LangChain:

• Standardized RAG patterns
• Document loaders
• Text splitters
• Retriever abstractions

FAISS:

• Fast similarity search
• CPU-friendly
• Scalable to millions of vectors
• No external dependencies

TinyLlama:

• Small model size (1.1B parameters)
• Good performance for size
• Fast inference on CPU
• Open-source license

Sentence Transformers:

• High-quality embeddings
• Pre-trained models
• Efficient encoding
• Wide model selection

---

4. System Components

4.1 Document Processing Pipeline

4.1.1 Document Loading

loader = TextLoader(kb_path, encoding='utf-8')
documents = loader.load()

• Loads text files from the knowledge base
• Handles UTF-8 encoding
• Returns Document objects with metadata

4.1.2 Text Splitting

splitter = RecursiveCharacterTextSplitter(
    chunk_size=500,
    chunk_overlap=50
)
texts = splitter.split_documents(documents)

• chunk_size: Maximum characters per chunk
• chunk_overlap: Characters shared between chunks
• Preserves context across chunk boundaries
• Splits on natural boundaries (paragraphs, sentences)

4.1.3 Embedding Generation

embeddings = HuggingFaceEmbeddings(
    model_name="sentence-transformers/all-MiniLM-L6-v2"
)

• Model: all-MiniLM-L6-v2
• Embedding dimension: 384
• Fast encoding speed
• Good semantic understanding

4.1.4 Vector Store Creation

db = FAISS.from_documents(texts, embeddings)

• Creates FAISS index
• Stores document chunks with embeddings
• Enables fast similarity search

4.2 Language Model Component

4.2.1 Model Selection

• TinyLlama-1.1B-Chat-v1.0
  • Parameters: 1.1 billion
  • Context length: 2048 tokens
  • Training: Instruction-tuned for chat
  • License: Apache 2.0

4.2.2 Model Loading

tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForCausalLM.from_pretrained(
    model_name,
    torch_dtype=torch.float32,
    low_cpu_mem_usage=True
)

• Uses float32 for CPU compatibility
• Optimized memory loading
• Automatic device mapping

4.2.3 Generation Parameters

outputs = model.generate(
    inputs.input_ids,
    max_new_tokens=256,
    temperature=0.7,
    top_p=0.9,
    do_sample=True,
    pad_token_id=tokenizer.eos_token_id
)

• max_new_tokens: Maximum response length
• temperature: Randomness (0.7 = balanced)
• top_p: Nucleus sampling threshold
• do_sample: Enable sampling for variety

4.3 RAG Pipeline

4.3.1 Retrieval

retriever = db.as_retriever(search_kwargs={"k": 3})
docs = retriever.get_relevant_documents(query)

• Retrieves top-3 most relevant chunks
• Uses cosine similarity
• Returns Document objects

4.3.2 Context Formation

context = "\n\n".join([doc.page_content for doc in docs])

• Concatenates retrieved documents
• Separates with double newlines
• Maintains readability

4.3.3 Prompt Construction

prompt = f"""Context: {context}

Question: {query}

Answer:"""

• Simple, effective format
• Clear separation of context and question
• Guides model to use context

---

5. Implementation Details

5.1 Caching Strategy

@st.cache_resource
def load_vector_store():
    # ...
    
@st.cache_resource
def load_llm():
    # ...

Benefits:

• Models loaded only once
• Persists across reruns
• Faster subsequent queries
• Reduced memory usage

5.2 Session State Management

if 'chat_history' not in st.session_state:
    st.session_state.chat_history = []

Stored Data:

• Chat history
• Query-answer pairs
• Source documents

5.3 Error Handling

try:
    # Operation
except Exception as e:
    st.error(f"Error: {str(e)}")
    return None

Error Types Handled:

• File not found
• Model loading failures
• Generation errors
• Encoding issues

5.4 UI Components

5.4.1 Input Section

query = st.text_input("Ask a question:", key="user_input")
if st.button("Submit") and query:
    # Process query

5.4.2 Response Display

with st.expander(f"Q: {chat['query']}", expanded=(i==0)):
    st.markdown("### Answer")
    st.write(chat['answer'])
    st.markdown("### Source Documents")

5.4.3 Loading States

with st.spinner("Thinking..."):
    answer, sources = rag_chat(query, db, model, tokenizer)

---

6. Deployment Guide

6.1 Local Deployment

Prerequisites

• Python 3.8+
• 8GB RAM minimum
• 10GB disk space

Steps

# Clone repository
git clone <repo-url>
cd chatbot

# Create virtual environment
python -m venv venv
source venv/bin/activate  # Windows: venv\Scripts\activate

# Install dependencies
pip install -r requirements.txt

# Run application
streamlit run app.py

6.2 Hugging Face Spaces Deployment

Method 1: Web Interface

1. Create Space

   • Go to huggingface.co/spaces
   • Click "Create new Space"
   • Select Streamlit SDK
   • Choose hardware tier

2. Upload Files

   • app.py
   • requirements.txt
   • README.md
   • data/knowledge_base.txt

3. Configure

   • Set visibility (public/private)
   • Add description
   • Add tags

4. Deploy

   • Automatic build on file upload
   • Monitor build logs
   • Access via provided URL

Method 2: Git Push

# Create Space on Hugging Face
# Clone Space repository
git clone https://huggingface.co/spaces/<username>/<space-name>

# Copy files
cp -r * <space-name>/

# Commit and push
cd <space-name>
git add .
git commit -m "Deploy RAG chatbot"
git push

6.3 Docker Deployment

Create Dockerfile:

FROM python:3.10-slim

WORKDIR /app

COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt

COPY . .

EXPOSE 8501

CMD ["streamlit", "run", "app.py", "--server.port=8501", "--server.address=0.0.0.0"]

Build and run:

docker build -t rag-chatbot .
docker run -p 8501:8501 rag-chatbot

---

7. Performance Optimization

7.1 Model Optimization

Use Quantization

from transformers import BitsAndBytesConfig

quantization_config = BitsAndBytesConfig(
    load_in_8bit=True,
    llm_int8_threshold=6.0
)

model = AutoModelForCausalLM.from_pretrained(
    model_name,
    quantization_config=quantization_config
)

Use Smaller Models

• TinyLlama-1.1B (current)
• DistilGPT2 (faster, less capable)
• Flan-T5-small (good for Q&A)

7.2 Vector Store Optimization

Reduce Chunk Size

splitter = RecursiveCharacterTextSplitter(
    chunk_size=300,  # Smaller chunks
    chunk_overlap=30
)

Use GPU for Embeddings

embeddings = HuggingFaceEmbeddings(
    model_name="sentence-transformers/all-MiniLM-L6-v2",
    model_kwargs={'device': 'cuda'}  # If GPU available
)

7.3 Caching Strategies

Cache Embeddings

import pickle

# Save
with open('embeddings.pkl', 'wb') as f:
    pickle.dump(db, f)

# Load
with open('embeddings.pkl', 'rb') as f:
    db = pickle.load(f)

7.4 Response Time Optimization

Optimization	Impact	Difficulty
Use GPU	5-10x faster	Easy
Quantize model	2x faster	Medium
Reduce max_tokens	Linear speedup	Easy
Cache embeddings	Faster startup	Easy
Smaller model	2-5x faster	Easy

---

8. Troubleshooting

8.1 Common Issues

Issue: Model fails to load

Symptoms: Error during model initialization Solutions:

• Check internet connection
• Verify model name is correct
• Try smaller model
• Increase timeout

Issue: Out of memory

Symptoms: Crash during model loading or generation Solutions:

• Use smaller model
• Reduce batch size
• Enable quantization
• Upgrade hardware

Issue: Slow responses

Symptoms: Long wait times for answers Solutions:

• Reduce max_new_tokens
• Use GPU hardware
• Optimize retrieval (fewer docs)
• Use smaller model

Issue: Poor answer quality

Symptoms: Irrelevant or incorrect answers Solutions:

• Improve knowledge base content
• Increase chunk overlap
• Retrieve more documents (k=5)
• Use better model
• Adjust temperature

8.2 Debugging

Enable Logging

import logging

logging.basicConfig(level=logging.DEBUG)
logger = logging.getLogger(__name__)

logger.debug(f"Retrieved docs: {len(docs)}")
logger.debug(f"Context length: {len(context)}")

Inspect Retrieved Documents

for i, doc in enumerate(docs):
    st.write(f"Doc {i}: {doc.page_content[:100]}...")
    st.write(f"Score: {doc.metadata.get('score', 'N/A')}")

---

9. Future Enhancements

9.1 Short-term Improvements

1. Conversation Memory

   • Store chat history
   • Include in context
   • Multi-turn conversations

2. Better UI

   • Chat-style interface
   • Markdown rendering
   • Code syntax highlighting

3. Multiple File Support

   • PDF loader
   • Word documents
   • Web scraping

4. Advanced Retrieval

   • Hybrid search (keyword + semantic)
   • Re-ranking
   • Query expansion

9.2 Medium-term Enhancements

1. User Authentication

   • Login system
   • User-specific history
   • Access control

2. Analytics Dashboard

   • Query statistics
   • Response quality metrics
   • Usage patterns

3. API Endpoint

   • REST API
   • Webhook support
   • Integration capabilities

4. Multi-language Support

   • Translation
   • Language detection
   • Multilingual embeddings

9.3 Long-term Vision

1. Advanced RAG

   • Graph RAG
   • Multi-hop reasoning
   • Self-reflection

2. Fine-tuning

   • Domain-specific models
   • RLHF integration
   • Continuous learning

3. Enterprise Features

   • Role-based access
   • Audit logs
   • Compliance tools

4. Scalability

   • Distributed vector store
   • Load balancing
   • Caching layer

---

10. Appendix

10.1 File Structure

chatbot/
├── app.py                          # Main application
├── requirements.txt                # Dependencies
├── README.md                       # User documentation
├── .gitignore                      # Git ignore rules
├── data/
│   └── knowledge_base.txt         # Knowledge base
├── docs/
│   └── PROJECT_DOCUMENTATION.md   # This file
└── tests/                         # Future: Unit tests

10.2 Dependencies Explained

• streamlit: Web UI framework
• langchain: RAG orchestration
• langchain-community: Community integrations
• transformers: Model loading and inference
• torch: Deep learning framework
• sentence-transformers: Embedding models
• faiss-cpu: Vector similarity search
• huggingface-hub: Model downloads
• accelerate: Model optimization

10.3 Useful Resources

• Streamlit Documentation
• LangChain Documentation
• Hugging Face Transformers
• FAISS Documentation
• RAG Paper

10.4 License

This project is released under the MIT License. See LICENSE file for details.

10.5 Contributing

Contributions are welcome! Please:

1. Fork the repository
2. Create a feature branch
3. Make your changes
4. Submit a pull request

10.6 Support

For issues or questions:

• Open a GitHub issue
• Check existing documentation
• Review troubleshooting section

---

Document Version: 1.0
Last Updated: August 2026
Author: Chetan S P
Project: RAG-Based Chatbot