File size: 5,986 Bytes
d3d09d6
 
 
 
 
c9e39ab
3134060
 
d3d09d6
 
 
b28d261
d3d09d6
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
c9e39ab
 
d3d09d6
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
2a14ed8
d3d09d6
2a14ed8
 
d3d09d6
 
 
c9e39ab
d3d09d6
c9e39ab
d3d09d6
c9e39ab
 
 
d3d09d6
c9e39ab
 
 
d3d09d6
c9e39ab
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
---
title: Production Document Question Answering (DQA)
emoji: πŸ“„
colorFrom: blue
colorTo: indigo
sdk: gradio
sdk_version: 6.32.0
python_version: '3.12'
app_file: app.py
pinned: false
license: mit
short_description: Grounded Document QA with citations & verification
---

# Production Document Question Answering (DQA) System

A production-grade, modular Document Question Answering (DQA) engine designed to answer complex user questions across multi-page **PDF**, **DOCX**, and **TXT** documents with high precision, zero hallucination tolerance, direct quote evidence, and strict page and section citations.

---

## πŸ—οΈ Architecture & Pipeline Flow

```mermaid
flowchart TD
    subgraph Ingestion ["1. Layout-Aware Ingestion & Chunking"]
        A[User Documents: PDF, DOCX, TXT] --> B{Layout Parser}
        B -->|PDF| B1[pdfplumber: Text + Tables + Fonts]
        B -->|Scanned PDF Page| B2[pdf2image + Tesseract OCR Fallback]
        B -->|DOCX| B3[python-docx: Heading Styles + Tables]
        B -->|TXT/MD| B4[Markdown/Capitalized Header Parser]
        B1 & B2 & B3 & B4 --> C[Structure-Aware Chunking: 300-500 tokens, 10-15% overlap]
        C -->|Page No + Section Title Metadata| D[(Document Chunks)]
    end

    subgraph QueryHandling ["2. Query Rewriting & Decomposition"]
        Q[User Query] --> QR{History Present?}
        QR -->|Yes| Q1[Rewrite to Standalone Search Query]
        QR -->|No| Q2[Original Query]
        Q1 & Q2 --> QD{Complex / Comparative?}
        QD -->|Yes| Q3[Decompose into Sub-Queries]
        QD -->|No| Q4[Single Query]
    end

    subgraph Retrieval ["3. Hybrid Search & Reranking"]
        D --> E1[Dense Embeddings: BAAI/bge-small-en-v1.5 + FAISS]
        D --> E2[Sparse Lexical Index: BM25Okapi]
        Q3 & Q4 --> E1 & E2
        E1 -->|Top 20 Dense| F[Reciprocal Rank Fusion RRF k=60]
        E2 -->|Top 20 BM25| F
        F --> G[Top 20 Unified Candidates]
        G --> H[Cross-Encoder Reranker: ms-marco-MiniLM-L-6-v2]
        H --> I[Top 5 High-Precision Chunks]
    end

    subgraph GenerationVerification ["4. Grounded Generation & Second-Pass Verification"]
        I --> J[Grounded LLM Generator: HF Inference API / Offline Engine]
        J -->|Evidence Quotes + Citations [Page X, Section Y]| K[Candidate Answer]
        J -->|If not in context| L["Not found in the document."]
        K --> M[Second-Pass Claim Verifier]
        M -->|Entailment check against retrieved chunks| N{All claims supported?}
        N -->|Yes| O[Final Answer with Citations]
        N -->|No| P[Flag unsupported claims with ⚠️ & Output Faithfulness Audit]
    end
```

---

## πŸš€ Key Modules & Design

| Module | Responsibility |
|---|---|
| [`config.py`](file:///home/ramakrishna/Desktop/Projects/DQA_Project/config.py) | Centralized dataclass configuration (models, chunk sizes, weights, thresholds, paths). |
| [`ingest.py`](file:///home/ramakrishna/Desktop/Projects/DQA_Project/ingest.py) | Layout-aware document parsing, markdown table extraction, OCR fallback, and structure-aware chunking (300–500 tokens, 10–15% overlap) with `[Page, Section]` metadata preservation. |
| [`retrieve.py`](file:///home/ramakrishna/Desktop/Projects/DQA_Project/retrieve.py) | Hybrid retriever combining FAISS cosine vector search and BM25Okapi via Reciprocal Rank Fusion (RRF), Cross-Encoder reranking (`ms-marco-MiniLM-L-6-v2`), query rewriting, and query decomposition. |
| [`generate.py`](file:///home/ramakrishna/Desktop/Projects/DQA_Project/generate.py) | Strict context-grounded generator quoting evidence and generating `[Page X, Section Y]` citations. Multi-backend: Hugging Face Serverless Inference API, OpenAI-compatible APIs, and offline deterministic grounding engine. |
| [`verify.py`](file:///home/ramakrishna/Desktop/Projects/DQA_Project/verify.py) | Second-pass verifier extracting atomic factual claims and validating them against retrieved passages via cross-encoder entailment, flagging or removing unsupported claims, and reporting Faithfulness Scores. |
| [`evaluate.py`](file:///home/ramakrishna/Desktop/Projects/DQA_Project/evaluate.py) | 50-pair benchmark evaluation suite testing Recall@k, Answer Faithfulness, Answer Correctness, and Hallucination Rate across 4 baseline/proposed configurations. |
| [`app.py`](file:///home/ramakrishna/Desktop/Projects/DQA_Project/app.py) | Gradio interactive web application with file upload, real-time token streaming, citation accordion, verification audit display, and prompt chips. (Default for Hugging Face Spaces). |
| [`streamlit_app.py`](file:///home/ramakrishna/Desktop/Projects/DQA_Project/streamlit_app.py) | Alternative Streamlit web interface for users who prefer Streamlit. |

---

## πŸ“Š Benchmark & Evaluation Results (50 Q&A Pairs)

The system was evaluated against a rigorous test set of **50 Q&A pairs** across multi-page PDF, DOCX, and TXT documents (35 answerable factual, table lookup, cross-page synthesis questions + 15 negative unanswerable controls):

| Pipeline Configuration | Recall@5 | Recall@20 | Answer Correctness | Answer Faithfulness | Hallucination Rate |
|---|---|---|---|---|---|
| **1. Baseline: Plain Dense (FAISS only)** | 100.0% | 100.0% | 79.2% | 92.0% | 8.0% |
| **2. Baseline: Plain BM25 (Lexical only)** | 100.0% | 100.0% | 79.3% | 92.0% | 8.0% |
| **3. Baseline: Hybrid Search (No Rerank)** | 100.0% | 100.0% | 79.3% | 92.0% | 8.0% |
| **4. Proposed: Hybrid + Reranker + Verification** | **100.0%** | **100.0%** | **79.3%** | **98.0%** | **2.0%** |

---

## πŸ€— Live on Hugging Face Spaces (Gradio)

This repository is deployed to Hugging Face Spaces at:
**[https://huggingface.co/spaces/Varun2007/document-qa-system](https://huggingface.co/spaces/Varun2007/document-qa-system)**

---

## πŸ› οΈ Running Locally

### Gradio (Default)
```bash
python3 app.py
```
Open `http://localhost:7860` in your browser.

### Streamlit (Alternative)
```bash
streamlit run streamlit_app.py
```
Open `http://localhost:8501` in your browser.