Harshit Temkar commited on
Commit
f35185e
Β·
1 Parent(s): eb89f47

Create README.md for DocuMind AI

Browse files

Add comprehensive README for DocuMind AI project.

Files changed (1) hide show
  1. README.md +363 -0
README.md ADDED
@@ -0,0 +1,363 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ <div align="center">
2
+
3
+ # 🧠 DocuMind AI
4
+
5
+ ### *Chat with your documents. Get cited, grounded answers.*
6
+
7
+ [![Python](https://img.shields.io/badge/Python-3.10+-3776AB?style=for-the-badge&logo=python&logoColor=white)](https://python.org)
8
+ [![FastAPI](https://img.shields.io/badge/FastAPI-0.100+-009688?style=for-the-badge&logo=fastapi&logoColor=white)](https://fastapi.tiangolo.com)
9
+ [![LLaMA3](https://img.shields.io/badge/LLaMA3-Groq_API-FF6B35?style=for-the-badge&logo=meta&logoColor=white)](https://groq.com)
10
+ [![ChromaDB](https://img.shields.io/badge/ChromaDB-Vector_Store-E85D04?style=for-the-badge)](https://www.trychroma.com)
11
+ [![Docker](https://img.shields.io/badge/Docker-Ready-2496ED?style=for-the-badge&logo=docker&logoColor=white)](https://docker.com)
12
+ [![License: MIT](https://img.shields.io/badge/License-MIT-yellow?style=for-the-badge)](LICENSE)
13
+
14
+ **DocuMind AI** is a production-grade Retrieval-Augmented Generation (RAG) system that lets you upload PDFs and ask intelligent questions β€” receiving accurate, cited answers powered by LLaMA3 via Groq.
15
+
16
+ [Features](#-features) β€’ [Architecture](#-architecture) β€’ [Tech Stack](#-tech-stack) β€’ [Getting Started](#-getting-started) β€’ [API Docs](#-api-reference) β€’ [Project Structure](#-project-structure)
17
+
18
+ ---
19
+
20
+ </div>
21
+
22
+ ## 🎯 What Problem Does This Solve?
23
+
24
+ > People waste an average of **2.5 hours per day** searching through documents for answers.
25
+
26
+ DocuMind AI eliminates this by turning any PDF into an intelligent, queryable knowledge base. Upload your research papers, reports, or manuals β€” and simply *ask*.
27
+
28
+ ---
29
+
30
+ ## ✨ Features
31
+
32
+ - πŸ“„ **PDF Ingestion** β€” Upload and process any PDF document instantly
33
+ - πŸ” **Semantic Search** β€” Finds meaning, not just keywords, using vector embeddings
34
+ - πŸ€– **LLM-Powered Answers** β€” Grounded responses via LLaMA3 (8B) through Groq API
35
+ - πŸ“Ž **Citation-Based Output** β€” Every answer includes source references from your document
36
+ - πŸ“Š **Confidence Scoring** β€” Know how reliable each answer is
37
+ - πŸ—„οΈ **Persistent Vector Store** β€” ChromaDB storage survives restarts
38
+ - ⚑ **REST API** β€” Full FastAPI backend with Swagger UI
39
+ - πŸ–₯️ **Chat UI** β€” Clean, responsive vanilla JS frontend
40
+ - 🐳 **Docker Ready** β€” One-command deployment
41
+
42
+ ---
43
+ ## UI Interface
44
+
45
+ ### πŸ–₯️ Chat Interface
46
+ ![Chat UI](docs/chat-ui.png)
47
+
48
+ ### πŸ“„ PDF Upload
49
+ ![Upload](docs/upload.png)
50
+
51
+ ### πŸ€– AI Response with Sources
52
+ ![Response](docs/response1.png)
53
+ ![Response](docs/response2.png)
54
+ ![Response](docs/response3.png)
55
+
56
+ ---
57
+
58
+ ## πŸ’‘ Use Cases
59
+
60
+ | Use Case | Example |
61
+ |----------|---------|
62
+ | πŸ“š **Research & Academia** | Upload a research paper and ask "What methodology did the authors use?" |
63
+ | 🏒 **Business Reports** | Query quarterly reports β€” "What were the key revenue drivers this quarter?" |
64
+ | βš–οΈ **Legal Documents** | Extract clauses from contracts β€” "What are the termination conditions?" |
65
+ | πŸ› οΈ **Technical Manuals** | Ask product docs β€” "How do I reset the device to factory settings?" |
66
+ | πŸ₯ **Medical Literature** | Summarize clinical studies β€” "What were the side effects observed?" |
67
+ | πŸŽ“ **Study Assistant** | Upload lecture notes and quiz yourself interactively |
68
+
69
+ > **Tip:** DocuMind works best with text-heavy PDFs. Scanned image-only PDFs may yield lower accuracy without OCR preprocessing.
70
+
71
+ ## πŸ—οΈ Architecture
72
+
73
+ ### End-to-End RAG Pipeline
74
+
75
+ ```
76
+ πŸ“„ PDF Upload
77
+ β”‚
78
+ β–Ό
79
+ β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
80
+ β”‚ Text Extraction β”‚ ← PyMuPDF
81
+ β””β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”˜
82
+ β”‚
83
+ β–Ό
84
+ β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
85
+ β”‚ Chunking β”‚ ← Size: 500 tokens | Overlap: 50
86
+ β””β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”˜
87
+ β”‚
88
+ β–Ό
89
+ β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
90
+ β”‚ Embeddings β”‚ ← all-MiniLM-L6-v2 (dim: 384)
91
+ β””β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”˜
92
+ β”‚
93
+ β–Ό
94
+ β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
95
+ β”‚ Vector Storage β”‚ ← ChromaDB (persistent)
96
+ β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜
97
+
98
+ πŸ”Ž At Query Time:
99
+
100
+ ❓ User Question
101
+ β”‚
102
+ β–Ό
103
+ β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
104
+ β”‚ Query Embedding β”‚ ← Same model
105
+ β””β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”˜
106
+ β”‚
107
+ β–Ό
108
+ β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
109
+ β”‚ Similarity Searchβ”‚ ← FAISS cosine similarity | Top-5 chunks
110
+ β””β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”˜
111
+ β”‚
112
+ β–Ό
113
+ β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
114
+ β”‚ LLM Generation β”‚ ← Groq LLaMA3 8B
115
+ β””β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”˜
116
+ β”‚
117
+ β–Ό
118
+ βœ… Answer + Sources + Confidence Score
119
+ ```
120
+
121
+ ---
122
+
123
+ ## πŸ› οΈ Tech Stack
124
+
125
+ | Layer | Technology |
126
+ |-------|-----------|
127
+ | **Backend** | FastAPI, Python 3.10+ |
128
+ | **Embeddings** | `sentence-transformers` (`all-MiniLM-L6-v2`) |
129
+ | **Vector DB** | ChromaDB (persistent) |
130
+ | **Similarity Search** | FAISS (cosine) |
131
+ | **LLM** | LLaMA3 8B via Groq API |
132
+ | **PDF Processing** | PyMuPDF |
133
+ | **Frontend** | Vanilla HTML, CSS, JavaScript |
134
+ | **Infrastructure** | Docker + docker-compose |
135
+
136
+ ---
137
+
138
+ ## πŸš€ Getting Started
139
+
140
+ ### Prerequisites
141
+
142
+ - Python 3.10+
143
+ - Docker & docker-compose (optional but recommended)
144
+ - A [Groq API key](https://console.groq.com)
145
+
146
+ ### 1. Clone the Repository
147
+
148
+ ```bash
149
+ git clone https://github.com/yourusername/documind-ai.git
150
+ cd documind-ai
151
+ ```
152
+
153
+ ### 2. Configure Environment
154
+
155
+ ```bash
156
+ cp .env.example .env
157
+ ```
158
+
159
+ Edit `.env` and add your credentials:
160
+
161
+ ```env
162
+ GROQ_API_KEY=your_groq_api_key_here
163
+ ```
164
+
165
+ ### 3a. Run with Docker (Recommended)
166
+
167
+ ```bash
168
+ docker-compose up --build
169
+ ```
170
+
171
+ ### 3b. Run Locally
172
+
173
+ ```bash
174
+ # Create virtual environment
175
+ python -m venv venv
176
+ source venv/bin/activate # Windows: venv\Scripts\activate
177
+
178
+ # Install dependencies
179
+ pip install -r requirements.txt
180
+
181
+ # Start the server
182
+ uvicorn app.main:app --reload --port 8000
183
+ ```
184
+
185
+ ### 4. Open the App
186
+
187
+ - **Chat UI:** [http://localhost:8000](http://localhost:8000)
188
+ - **Swagger API Docs:** [http://localhost:8000/docs](http://localhost:8000/docs)
189
+
190
+ ---
191
+
192
+ ## πŸ“‘ API Reference
193
+
194
+ | Endpoint | Method | Description |
195
+ |----------|--------|-------------|
196
+ | `/api/health` | `GET` | System health check |
197
+ | `/api/upload` | `POST` | Upload a PDF document |
198
+ | `/api/documents` | `GET` | List all uploaded documents |
199
+ | `/api/query` | `POST` | Ask a question ⭐ |
200
+ | `/docs` | `GET` | Swagger UI |
201
+
202
+ ### Query Example
203
+
204
+ **Request:**
205
+ ```json
206
+ POST /api/query
207
+ {
208
+ "question": "What are the main findings of the report?",
209
+ "top_k": 5
210
+ }
211
+ ```
212
+
213
+ **Response:**
214
+ ```json
215
+ {
216
+ "answer": "The report concludes that...",
217
+ "sources": [
218
+ { "page": 4, "chunk": "...relevant excerpt..." },
219
+ { "page": 7, "chunk": "...another source..." }
220
+ ],
221
+ "confidence": 0.91
222
+ }
223
+ ```
224
+
225
+ ---
226
+
227
+ ## πŸ“ Project Structure
228
+
229
+ ```
230
+ documind-ai/
231
+ β”œβ”€β”€ app/
232
+ β”‚ β”œβ”€β”€ main.py # FastAPI app entry point
233
+ β”‚ β”œβ”€β”€ routes.py # API endpoints
234
+ β”‚ β”œβ”€β”€ schemas.py # Request/Response models
235
+ β”‚ β”œβ”€β”€ config.py # Environment configuration
236
+ β”‚ └── logger.py # Logging setup
237
+ β”‚
238
+ β”œβ”€β”€ services/
239
+ β”‚ β”œβ”€β”€ pdf_processor.py # PDF extraction + chunking
240
+ β”‚ β”œβ”€β”€ embeddings.py # Embedding model wrapper
241
+ β”‚ β”œβ”€β”€ vector_store.py # ChromaDB operations
242
+ β”‚ └── rag_engine.py # ⭐ Full pipeline orchestration
243
+ β”‚
244
+ β”œβ”€β”€ frontend/
245
+ β”‚ β”œβ”€β”€ index.html # Chat UI
246
+ β”‚ β”œβ”€β”€ style.css # Styling
247
+ β”‚ └── app.js # Frontend logic
248
+ β”‚
249
+ β”œβ”€β”€ data/
250
+ β”‚ β”œβ”€β”€ uploads/ # Uploaded PDFs (gitignored)
251
+ β”‚ └── vectorstore/ # ChromaDB embeddings (gitignored)
252
+ β”‚
253
+ β”œβ”€β”€ notebooks/ # Experiment notebooks
254
+ β”œβ”€β”€ tests/ # API test suite
255
+ β”œβ”€β”€ docs/ # Visual analysis & diagrams
256
+ β”œβ”€β”€ docker-compose.yml
257
+ β”œβ”€β”€ Dockerfile
258
+ β”œβ”€β”€ requirements.txt
259
+ β”œβ”€β”€ .env.example
260
+ └── README.md
261
+ ```
262
+
263
+ ---
264
+
265
+ ## βš™οΈ Configuration
266
+
267
+ | Variable | Description | Default |
268
+ |----------|-------------|---------|
269
+ | `GROQ_API_KEY` | Your Groq API key | *required* |
270
+ | `CHUNK_SIZE` | Token size per chunk | `500` |
271
+ | `CHUNK_OVERLAP` | Overlap between chunks | `50` |
272
+ | `TOP_K` | Top chunks to retrieve | `5` |
273
+ | `EMBEDDING_MODEL` | Sentence transformer model | `all-MiniLM-L6-v2` |
274
+
275
+ ---
276
+
277
+ ## πŸ§ͺ Running Tests
278
+
279
+ ```bash
280
+ pytest tests/ -v
281
+ ```
282
+
283
+ ---
284
+
285
+ ## ⚠️ Before Pushing to GitHub
286
+
287
+ Make sure your `.gitignore` includes:
288
+
289
+ ```gitignore
290
+ # Environment
291
+ .env
292
+ venv/
293
+ __pycache__/
294
+
295
+ # Data (can be large)
296
+ data/uploads/
297
+ data/vectorstore/
298
+
299
+ # OS
300
+ .DS_Store
301
+ ```
302
+
303
+ ---
304
+
305
+ ## πŸ—ΊοΈ Roadmap
306
+
307
+ - [ ] Multi-document cross-referencing
308
+ - [ ] Chat history persistence
309
+ - [ ] Support for DOCX, TXT, and web URLs
310
+ - [ ] Authentication & user sessions
311
+ - [ ] Streaming responses (SSE)
312
+ - [ ] Deploy to cloud (AWS / GCP / Railway)
313
+
314
+ ---
315
+
316
+ ---
317
+
318
+ ## πŸ“ˆ Performance & Limitations
319
+
320
+ ### Benchmarks
321
+
322
+ | Metric | Value |
323
+ |--------|-------|
324
+ | Avg. query response time | ~1.5–3s (Groq API) |
325
+ | Embedding model size | ~90MB (`all-MiniLM-L6-v2`) |
326
+ | Max recommended PDF size | ~50 pages (for optimal chunking) |
327
+ | Retrieval accuracy (top-5) | High for text-dense, well-structured PDFs |
328
+ | Supported file type | PDF only (v1.0) |
329
+
330
+ ### Known Limitations
331
+
332
+ - **Scanned PDFs** β€” Image-only PDFs are not supported without an OCR preprocessing step. PyMuPDF extracts embedded text only.
333
+ - **Large documents** β€” PDFs exceeding ~100 pages may increase ingestion time and reduce retrieval precision without tuning chunk size.
334
+ - **Confidence scores are approximate** β€” The confidence value reflects embedding similarity, not factual correctness. Always verify critical answers against the source.
335
+ - **No chat memory** β€” Each query is independent; the system does not retain conversation context between questions.
336
+ - **Single document per query** β€” Cross-document querying is not yet supported (see Roadmap).
337
+ - **Language** β€” Optimized for English-language documents. Multilingual support is untested.
338
+
339
+ ## 🀝 Contributing
340
+
341
+ Contributions are welcome! Please open an issue first to discuss what you'd like to change.
342
+
343
+ 1. Fork the repository
344
+ 2. Create your feature branch (`git checkout -b feature/amazing-feature`)
345
+ 3. Commit your changes (`git commit -m 'Add amazing feature'`)
346
+ 4. Push to the branch (`git push origin feature/amazing-feature`)
347
+ 5. Open a Pull Request
348
+
349
+ ---
350
+
351
+ ## πŸ“„ License
352
+
353
+ This project is licensed under the MIT License β€” see the [LICENSE](LICENSE) file for details.
354
+
355
+ ---
356
+
357
+ <div align="center">
358
+
359
+ **Built with ❀️ using FastAPI, LLaMA3, and ChromaDB**
360
+
361
+ *If this project helped you, consider giving it a ⭐ on GitHub!*
362
+
363
+ </div>