Urmila Saini commited on
Commit
b9fa4a6
·
0 Parent(s):

add DockerFile

Browse files
Files changed (12) hide show
  1. .dockerignore +14 -0
  2. .gitignore +10 -0
  3. Dockerfile +26 -0
  4. README.md +85 -0
  5. app.py +203 -0
  6. pdf_to_text.py +757 -0
  7. rag_pipeline.py +1361 -0
  8. requirements.txt +21 -0
  9. static/app.js +1033 -0
  10. static/styles.css +998 -0
  11. templates/index.html +203 -0
  12. uploads/.gitkeep +1 -0
.dockerignore ADDED
@@ -0,0 +1,14 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ .git
2
+ .agents
3
+ .dist
4
+ .venv
5
+ __pycache__
6
+ *.pyc
7
+ *.pyo
8
+ *.pyd
9
+ .env
10
+ data
11
+ uploads
12
+ server-*.log
13
+ *.sqlite
14
+ *.db
.gitignore ADDED
@@ -0,0 +1,10 @@
 
 
 
 
 
 
 
 
 
 
 
1
+ env
2
+ .env
3
+ __pycache__/
4
+ *.py[cod]
5
+ server*.log
6
+ data/
7
+ *.sqlite
8
+ *.db
9
+ uploads/*
10
+ !uploads/.gitkeep
Dockerfile ADDED
@@ -0,0 +1,26 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ FROM python:3.12-slim
2
+
3
+ ENV PYTHONDONTWRITEBYTECODE=1 \
4
+ PYTHONUNBUFFERED=1 \
5
+ PIP_NO_CACHE_DIR=1
6
+
7
+ WORKDIR /app
8
+
9
+ RUN apt-get update \
10
+ && apt-get install -y --no-install-recommends \
11
+ build-essential \
12
+ libgl1 \
13
+ libglib2.0-0 \
14
+ && rm -rf /var/lib/apt/lists/*
15
+
16
+ COPY requirements.txt .
17
+ RUN python -m pip install --upgrade pip \
18
+ && python -m pip install -r requirements.txt
19
+
20
+ COPY . .
21
+
22
+ RUN mkdir -p data uploads
23
+
24
+ EXPOSE 7860
25
+
26
+ CMD ["uvicorn", "app:app", "--host", "0.0.0.0", "--port", "7860"]
README.md ADDED
@@ -0,0 +1,85 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # SmartNotes PDF OCR
2
+
3
+ ## Project Metadata
4
+
5
+ | Field | Value |
6
+ | --- | --- |
7
+ | Project | SmartNotes AI |
8
+ | App type | FastAPI web app for PDF OCR and RAG-based question answering |
9
+ | Runtime | Python 3.12 |
10
+ | Default URL | `http://127.0.0.1:8000` |
11
+ | Main entrypoint | `app.py` |
12
+ | UI files | `templates/index.html`, `static/app.js`, `static/styles.css` |
13
+ | OCR pipeline | PyMuPDF, OpenCV, Gemini Vision OCR, TrOCR fallback |
14
+ | RAG pipeline | Text cleaning, metadata extraction, parent/child chunks, embeddings, BM25, reranking, citations |
15
+ | Storage | SQLite local fallback in `data/smartnotes.sqlite`; optional PostgreSQL via `POSTGRES_DSN` |
16
+ | Vector store | Local fallback; optional Qdrant via `QDRANT_URL` |
17
+ | Docker | `Dockerfile` exposes port `8000` |
18
+ | Secrets | Keep API keys in `.env`; do not commit `.env` |
19
+
20
+ FastAPI app for PDF-to-text extraction using this route:
21
+
22
+ 1. Try PyMuPDF direct text extraction.
23
+ 2. If text exists, skip OCR, clean text, return final text.
24
+ 3. If no direct text exists, convert PDF pages to images with PyMuPDF.
25
+ 4. Preprocess images with OpenCV.
26
+ 5. For 20 pages or fewer, run Gemini Vision OCR.
27
+ 6. For more than 20 pages, run TrOCR first, then send low-confidence pages to Gemini Vision.
28
+ 7. Merge, clean, and return final text.
29
+
30
+ The UI uses `/api/pdf-to-text-stream`, so extracted text appears page by page instead of waiting for the full PDF to finish.
31
+
32
+ After extraction, the UI automatically calls `/api/index-text-stream` and runs the locked RAG workflow:
33
+
34
+ - status `processing`
35
+ - cleaning and metadata
36
+ - parent chunks
37
+ - fixed, recursive, and semantic child chunks
38
+ - chunk recommendation
39
+ - embeddings and embedding evaluation
40
+ - Qdrant store when configured, local vector fallback otherwise
41
+ - BM25 index
42
+ - status `indexed`
43
+ - query validation, rewrite, hybrid retrieval, top-50 candidates, reranking, grading
44
+ - parent fetch, duplicate parent removal, context fitting, answer, citations, retrieved chunk view, logs, feedback
45
+
46
+ Useful optional env values:
47
+
48
+ ```text
49
+ POSTGRES_DSN=postgresql://user:password@localhost:5432/smartnotes
50
+ QDRANT_URL=http://localhost:6333
51
+ QDRANT_API_KEY=
52
+ EMBEDDING_MODEL=BAAI/bge-small-en-v1.5
53
+ RERANKER_MODEL=BAAI/bge-reranker-base
54
+ USE_LOCAL_EMBEDDING_MODEL=1
55
+ USE_LOCAL_RERANKER_MODEL=1
56
+ ANSWER_PROVIDER=gemini
57
+ ANSWER_MODEL=gemini-2.5-flash
58
+ ```
59
+
60
+ ## Setup
61
+
62
+ ```powershell
63
+ python -m venv .venv
64
+ .\.venv\Scripts\Activate.ps1
65
+ pip install -r requirements.txt
66
+ $env:GEMINI_API_KEY="your_api_key"
67
+ uvicorn app:app --reload
68
+ ```
69
+
70
+ Open `http://127.0.0.1:8000`.
71
+
72
+ You can also put these values in `env` or `.env`:
73
+
74
+ ```text
75
+ GEMINI_API_KEY_1=your_first_api_key
76
+ GEMINI_API_KEY_2=your_second_api_key
77
+ GEMINI_API_KEY_3=your_third_api_key
78
+ GEMINI_API_KEY_4=your_fourth_api_key
79
+ GEMINI_MODELS=gemini-3.5-flash,gemini-2.5-flash,gemini-2.5-flash-lite
80
+ GEMINI_MAX_RETRIES=2
81
+ GEMINI_RETRY_DELAY=1.5
82
+ GEMINI_KEY_QUOTA_COOLDOWN=300
83
+ ```
84
+
85
+ If one Gemini key returns a quota or rate-limit error, the app skips it temporarily and tries the next configured key. Successful calls advance through the keys in order, so `_1`, `_2`, `_3`, and `_4` are used in rotation. If Gemini returns temporary `503 UNAVAILABLE` high-demand errors, the app retries and then tries the next model in `GEMINI_MODELS`.
app.py ADDED
@@ -0,0 +1,203 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ from __future__ import annotations
2
+
3
+ import json
4
+ from pathlib import Path
5
+
6
+ from fastapi import BackgroundTasks, FastAPI, File, HTTPException, Request, UploadFile
7
+ from fastapi.responses import HTMLResponse, StreamingResponse
8
+ from fastapi.staticfiles import StaticFiles
9
+ from fastapi.templating import Jinja2Templates
10
+ from pydantic import BaseModel
11
+
12
+ from pdf_to_text import ExtractionResult, extract_uploaded_pdf, stream_uploaded_pdf
13
+ from rag_pipeline import SmartNotesRAG
14
+
15
+
16
+ BASE_DIR = Path(__file__).resolve().parent
17
+ PROJECT_METADATA = {
18
+ "name": "SmartNotes AI",
19
+ "title": "SmartNotes PDF OCR",
20
+ "version": "1.0.0",
21
+ "description": "FastAPI app for PDF OCR, text extraction, and RAG-based question answering.",
22
+ "runtime": "Python 3.12",
23
+ "framework": "FastAPI",
24
+ "default_port": 8000,
25
+ "entrypoint": "app.py",
26
+ "storage": "SQLite local fallback with optional PostgreSQL",
27
+ "vector_store": "Local fallback with optional Qdrant",
28
+ "features": [
29
+ "PDF direct text extraction",
30
+ "OCR fallback with OpenCV, Gemini Vision OCR, and TrOCR",
31
+ "RAG indexing with parent and child chunks",
32
+ "Hybrid retrieval with embeddings and BM25",
33
+ "Reranked answers with citations",
34
+ ],
35
+ }
36
+
37
+ app = FastAPI(
38
+ title=PROJECT_METADATA["title"],
39
+ description=PROJECT_METADATA["description"],
40
+ version=PROJECT_METADATA["version"],
41
+ )
42
+ app.mount("/static", StaticFiles(directory=BASE_DIR / "static"), name="static")
43
+ templates = Jinja2Templates(directory=BASE_DIR / "templates")
44
+ rag = SmartNotesRAG()
45
+
46
+
47
+ class IndexTextRequest(BaseModel):
48
+ file_name: str
49
+ text: str
50
+
51
+
52
+ class QueryRequest(BaseModel):
53
+ document_id: str
54
+ query: str
55
+
56
+
57
+ class FeedbackRequest(BaseModel):
58
+ document_id: str
59
+ query: str
60
+ rating: int
61
+ comment: str = ""
62
+
63
+
64
+ @app.get("/", response_class=HTMLResponse)
65
+ async def home(request: Request):
66
+ return templates.TemplateResponse(request, "index.html")
67
+
68
+
69
+ @app.get("/api/metadata")
70
+ async def project_metadata():
71
+ return PROJECT_METADATA
72
+
73
+
74
+ @app.post("/api/pdf-to-text")
75
+ async def pdf_to_text(file: UploadFile = File(...)):
76
+ if not _looks_like_pdf(file.filename, file.content_type):
77
+ raise HTTPException(status_code=400, detail="Please upload a PDF file.")
78
+
79
+ file_bytes = await file.read()
80
+ if not file_bytes:
81
+ raise HTTPException(status_code=400, detail="Uploaded PDF is empty.")
82
+
83
+ try:
84
+ result = extract_uploaded_pdf(file.filename, file_bytes)
85
+ except RuntimeError as exc:
86
+ raise HTTPException(status_code=500, detail=str(exc)) from exc
87
+ except Exception as exc:
88
+ raise HTTPException(status_code=500, detail=f"PDF processing failed: {exc}") from exc
89
+
90
+ return _result_payload(result, original_file=file.filename)
91
+
92
+
93
+ @app.post("/api/pdf-to-text-stream")
94
+ async def pdf_to_text_stream(file: UploadFile = File(...)):
95
+ if not _looks_like_pdf(file.filename, file.content_type):
96
+ raise HTTPException(status_code=400, detail="Please upload a PDF file.")
97
+
98
+ original_file = file.filename
99
+ file_bytes = await file.read()
100
+ if not file_bytes:
101
+ raise HTTPException(status_code=400, detail="Uploaded PDF is empty.")
102
+
103
+ def events():
104
+ try:
105
+ for event in stream_uploaded_pdf(original_file, file_bytes):
106
+ yield json.dumps(event, ensure_ascii=True) + "\n"
107
+ except Exception as exc:
108
+ yield json.dumps(
109
+ {
110
+ "type": "error",
111
+ "message": f"PDF processing failed: {exc}",
112
+ },
113
+ ensure_ascii=True,
114
+ ) + "\n"
115
+
116
+ return StreamingResponse(events(), media_type="application/x-ndjson")
117
+
118
+
119
+ @app.post("/api/index-text-stream")
120
+ async def index_text_stream(payload: IndexTextRequest):
121
+ if not payload.text.strip():
122
+ raise HTTPException(status_code=400, detail="Extracted text is empty.")
123
+
124
+ def events():
125
+ try:
126
+ for event in rag.index_text_stream(payload.file_name, payload.text):
127
+ yield json.dumps(event, ensure_ascii=True) + "\n"
128
+ except Exception as exc:
129
+ yield json.dumps(
130
+ {
131
+ "type": "error",
132
+ "message": f"Indexing failed: {exc}",
133
+ },
134
+ ensure_ascii=True,
135
+ ) + "\n"
136
+
137
+ return StreamingResponse(events(), media_type="application/x-ndjson")
138
+
139
+
140
+ @app.post("/api/index-text-background")
141
+ async def index_text_background(payload: IndexTextRequest, background_tasks: BackgroundTasks):
142
+ if not payload.text.strip():
143
+ raise HTTPException(status_code=400, detail="Extracted text is empty.")
144
+
145
+ def run_indexing() -> None:
146
+ for _ in rag.index_text_stream(payload.file_name, payload.text):
147
+ pass
148
+
149
+ background_tasks.add_task(run_indexing)
150
+ return {"status": "queued"}
151
+
152
+
153
+ @app.post("/api/query")
154
+ async def query_document(payload: QueryRequest):
155
+ return rag.query(payload.document_id, payload.query)
156
+
157
+
158
+ @app.post("/api/feedback")
159
+ async def save_feedback(payload: FeedbackRequest):
160
+ rating = max(1, min(payload.rating, 5))
161
+ return rag.save_feedback(payload.document_id, payload.query, rating, payload.comment)
162
+
163
+
164
+ @app.get("/api/documents/{document_id}")
165
+ async def get_document(document_id: str):
166
+ document = rag.repository.get_document(document_id)
167
+ if not document:
168
+ raise HTTPException(status_code=404, detail="Document not found.")
169
+ document.pop("cleaned_text", None)
170
+ return document
171
+
172
+
173
+ @app.get("/health")
174
+ async def health():
175
+ return {"status": "ok"}
176
+
177
+
178
+ def _result_payload(result: ExtractionResult, original_file: str) -> dict:
179
+ return {
180
+ "file": original_file,
181
+ "text": result.text,
182
+ "route": result.route,
183
+ "page_count": result.page_count,
184
+ "direct_text_found": result.direct_text_found,
185
+ "warnings": result.warnings,
186
+ "pages": [
187
+ {
188
+ "page_number": page.page_number,
189
+ "engine": page.engine,
190
+ "confidence": page.confidence,
191
+ "text_length": len(page.text),
192
+ }
193
+ for page in result.pages
194
+ ],
195
+ }
196
+
197
+
198
+ def _looks_like_pdf(file_name: str | None, content_type: str | None) -> bool:
199
+ if content_type in {"application/pdf", "application/x-pdf"}:
200
+ return True
201
+ if file_name and Path(file_name).suffix.lower() == ".pdf":
202
+ return True
203
+ return False
pdf_to_text.py ADDED
@@ -0,0 +1,757 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ from __future__ import annotations
2
+
3
+ import io
4
+ import os
5
+ import re
6
+ import tempfile
7
+ import time
8
+ import threading
9
+ from concurrent.futures import ThreadPoolExecutor, as_completed
10
+ from dataclasses import dataclass, field
11
+ from pathlib import Path
12
+ from typing import Iterable
13
+
14
+
15
+ DIRECT_TEXT_MIN_CHARS = 30
16
+ SHORT_PDF_PAGE_LIMIT = 20
17
+ TROCR_CONFIDENCE_THRESHOLD = 0.78
18
+ DEFAULT_GEMINI_MODELS = (
19
+ "gemini-2.5-flash-lite",
20
+ "gemini-2.5-flash",
21
+ )
22
+
23
+
24
+ @dataclass
25
+ class PageResult:
26
+ page_number: int
27
+ engine: str
28
+ text: str
29
+ confidence: float | None = None
30
+
31
+
32
+ @dataclass
33
+ class ExtractionResult:
34
+ text: str
35
+ route: str
36
+ page_count: int
37
+ direct_text_found: bool
38
+ pages: list[PageResult] = field(default_factory=list)
39
+ warnings: list[str] = field(default_factory=list)
40
+
41
+
42
+ class PdfTextExtractor:
43
+ """Implements the requested PDF-to-text decision flow."""
44
+
45
+ def __init__(
46
+ self,
47
+ gemini_model: str | None = None,
48
+ trocr_model: str = "microsoft/trocr-base-printed",
49
+ short_pdf_page_limit: int = SHORT_PDF_PAGE_LIMIT,
50
+ trocr_confidence_threshold: float = TROCR_CONFIDENCE_THRESHOLD,
51
+ ) -> None:
52
+ self.trocr_model_name = trocr_model
53
+ self.short_pdf_page_limit = short_pdf_page_limit
54
+ self.trocr_confidence_threshold = trocr_confidence_threshold
55
+ self._gemini_client = None
56
+ self._gemini_clients = {}
57
+ self._gemini_key_cursor = 0
58
+ self._gemini_key_lock = threading.Lock()
59
+ self._gemini_quota_blocked_until: dict[str, float] = {}
60
+ self._trocr_processor = None
61
+ self._trocr_model = None
62
+ self._load_local_env()
63
+ self.gemini_models = self._gemini_model_candidates(gemini_model)
64
+ self.gemini_max_retries = min(
65
+ self._env_int("OCR_GEMINI_MAX_RETRIES", "GEMINI_MAX_RETRIES", default=1, minimum=1),
66
+ 4,
67
+ )
68
+ self.gemini_retry_delay = min(
69
+ self._env_float("OCR_GEMINI_RETRY_DELAY", "GEMINI_RETRY_DELAY", default=0.5, minimum=0.0),
70
+ 3.0,
71
+ )
72
+ self.gemini_timeout_ms = int(
73
+ min(
74
+ self._env_float("OCR_GEMINI_TIMEOUT", "GEMINI_TIMEOUT", default=20.0, minimum=5.0),
75
+ 120.0,
76
+ )
77
+ * 1000
78
+ )
79
+ self.gemini_key_quota_cooldown = min(
80
+ self._env_float("GEMINI_KEY_QUOTA_COOLDOWN", default=300.0, minimum=0.0),
81
+ 86400.0,
82
+ )
83
+ self.gemini_concurrency = min(
84
+ self._env_int("OCR_GEMINI_CONCURRENCY", default=3, minimum=1),
85
+ 5,
86
+ )
87
+ self.render_scale = self._env_float("OCR_RENDER_SCALE", default=1.25, minimum=1.0)
88
+ self.fast_ocr = os.getenv("OCR_FAST_MODE", "1") != "0"
89
+
90
+ def extract(self, pdf_path: str | Path) -> ExtractionResult:
91
+ path = Path(pdf_path)
92
+ document = self._open_document(path)
93
+ try:
94
+ page_count = document.page_count
95
+
96
+ direct_text = self._extract_direct_text(document)
97
+ if self._has_enough_text(direct_text):
98
+ return ExtractionResult(
99
+ text=self._clean_text_only(direct_text),
100
+ route="PyMuPDF direct text extraction",
101
+ page_count=page_count,
102
+ direct_text_found=True,
103
+ )
104
+
105
+ images = self._convert_pages_to_preprocessed_images(document)
106
+ if page_count <= self.short_pdf_page_limit:
107
+ pages = []
108
+ warnings = []
109
+ for index, image_bytes in images:
110
+ text, warning = self._safe_gemini_ocr(image_bytes, index)
111
+ if warning:
112
+ warnings.append(warning)
113
+ pages.append(
114
+ PageResult(
115
+ page_number=index,
116
+ engine="Gemini Vision OCR" if text else "Skipped non-text page",
117
+ text=text,
118
+ )
119
+ )
120
+ if not pages:
121
+ warnings.append("No readable text pages were processed.")
122
+ result = self._finalize_ocr_result(
123
+ route="PyMuPDF images -> OpenCV preprocess -> Gemini Vision OCR",
124
+ page_count=page_count,
125
+ direct_text_found=False,
126
+ pages=pages,
127
+ )
128
+ result.warnings.extend(warnings)
129
+ return result
130
+
131
+ pages = []
132
+ warnings = []
133
+ for page_number, image_bytes in images:
134
+ try:
135
+ trocr_text, confidence = self._trocr_ocr(image_bytes)
136
+ except Exception as exc:
137
+ trocr_text = ""
138
+ confidence = 0.0
139
+ warnings.append(f"Page {page_number} skipped: TrOCR failed ({exc}).")
140
+ if confidence < self.trocr_confidence_threshold:
141
+ gemini_text, warning = self._safe_gemini_ocr(image_bytes, page_number)
142
+ if warning:
143
+ warnings.append(warning)
144
+ pages.append(
145
+ PageResult(
146
+ page_number=page_number,
147
+ engine="TrOCR low confidence -> Gemini Vision OCR" if gemini_text else "Skipped non-text page",
148
+ text=gemini_text,
149
+ confidence=confidence,
150
+ )
151
+ )
152
+ else:
153
+ trocr_text = self._clean_text_only(trocr_text)
154
+ pages.append(
155
+ PageResult(
156
+ page_number=page_number,
157
+ engine="TrOCR" if trocr_text else "Skipped non-text page",
158
+ text=trocr_text,
159
+ confidence=confidence,
160
+ )
161
+ )
162
+
163
+ if not pages:
164
+ warnings.append("No pages were processed.")
165
+
166
+ result = self._finalize_ocr_result(
167
+ route="PyMuPDF images -> OpenCV preprocess -> TrOCR -> Gemini low-confidence fallback",
168
+ page_count=page_count,
169
+ direct_text_found=False,
170
+ pages=pages,
171
+ )
172
+ result.warnings.extend(warnings)
173
+ return result
174
+ finally:
175
+ document.close()
176
+
177
+ def stream_extract(self, pdf_path: str | Path) -> Iterable[dict]:
178
+ path = Path(pdf_path)
179
+ document = self._open_document(path)
180
+ try:
181
+ page_count = document.page_count
182
+ yield {"type": "start", "page_count": page_count}
183
+
184
+ short_pdf_ocr_jobs = []
185
+ for page_number, page in enumerate(document, start=1):
186
+ direct_text = self._clean_text_only(page.get_text("text"))
187
+ if self._has_enough_text(direct_text, min_chars=5):
188
+ yield {
189
+ "type": "page",
190
+ "page_number": page_number,
191
+ "page_count": page_count,
192
+ "engine": "PyMuPDF direct text extraction",
193
+ "route": "PyMuPDF direct text extraction",
194
+ "direct_text_found": True,
195
+ "confidence": None,
196
+ "text": direct_text,
197
+ }
198
+ continue
199
+
200
+ try:
201
+ image_bytes = self._convert_page_to_preprocessed_image(page)
202
+ except Exception as exc:
203
+ yield self._skipped_page_event(page_number, page_count, f"Image preprocessing failed: {exc}")
204
+ continue
205
+
206
+ if page_count <= self.short_pdf_page_limit:
207
+ short_pdf_ocr_jobs.append((page_number, image_bytes))
208
+ continue
209
+
210
+ try:
211
+ trocr_text, confidence = self._trocr_ocr(image_bytes)
212
+ except Exception as exc:
213
+ trocr_text = ""
214
+ confidence = 0.0
215
+ yield self._skipped_page_event(page_number, page_count, f"TrOCR failed: {exc}", confidence)
216
+ continue
217
+ if confidence < self.trocr_confidence_threshold:
218
+ ocr_text, warning = self._safe_gemini_ocr(image_bytes, page_number)
219
+ yield {
220
+ "type": "page",
221
+ "page_number": page_number,
222
+ "page_count": page_count,
223
+ "engine": "TrOCR low confidence -> Gemini Vision OCR" if ocr_text else "Skipped non-text page",
224
+ "route": "PyMuPDF image -> OpenCV preprocess -> TrOCR -> Gemini fallback",
225
+ "direct_text_found": False,
226
+ "confidence": confidence,
227
+ "text": ocr_text,
228
+ "warning": warning,
229
+ }
230
+ else:
231
+ trocr_text = self._clean_text_only(trocr_text)
232
+ if not self._has_enough_text(trocr_text, min_chars=2):
233
+ yield self._skipped_page_event(page_number, page_count, "No readable text found.", confidence)
234
+ continue
235
+ yield {
236
+ "type": "page",
237
+ "page_number": page_number,
238
+ "page_count": page_count,
239
+ "engine": "TrOCR",
240
+ "route": "PyMuPDF image -> OpenCV preprocess -> TrOCR",
241
+ "direct_text_found": False,
242
+ "confidence": confidence,
243
+ "text": self._clean_text(trocr_text),
244
+ }
245
+
246
+ if short_pdf_ocr_jobs:
247
+ yield from self._stream_gemini_ocr_jobs(short_pdf_ocr_jobs, page_count)
248
+
249
+ yield {"type": "done", "page_count": page_count}
250
+ finally:
251
+ document.close()
252
+
253
+ def _open_document(self, pdf_path: Path):
254
+ try:
255
+ import fitz
256
+ except ImportError as exc:
257
+ raise RuntimeError("PyMuPDF is required. Install it with: pip install pymupdf") from exc
258
+
259
+ try:
260
+ document = fitz.open(pdf_path)
261
+ if document.is_encrypted:
262
+ raise RuntimeError("This PDF is password-protected. Please upload an unlocked PDF.")
263
+ if document.page_count == 0:
264
+ raise RuntimeError("This PDF has no pages.")
265
+ return document
266
+ except RuntimeError:
267
+ raise
268
+ except Exception as exc:
269
+ raise RuntimeError(f"Could not open PDF. It may be corrupted or unsupported: {exc}") from exc
270
+
271
+ def _extract_direct_text(self, document) -> str:
272
+ chunks = []
273
+ for page in document:
274
+ chunks.append(page.get_text("text"))
275
+ return "\n\n".join(chunks)
276
+
277
+ def _has_enough_text(self, text: str, min_chars: int = DIRECT_TEXT_MIN_CHARS) -> bool:
278
+ normalized = re.sub(r"\s+", "", text or "")
279
+ return len(normalized) >= min_chars
280
+
281
+ def _convert_pages_to_preprocessed_images(self, document) -> list[tuple[int, bytes]]:
282
+ images = []
283
+ for index, page in enumerate(document, start=1):
284
+ try:
285
+ images.append((index, self._convert_page_to_preprocessed_image(page)))
286
+ except Exception:
287
+ continue
288
+ return images
289
+
290
+ def _convert_page_to_preprocessed_image(self, page) -> bytes:
291
+ try:
292
+ import cv2
293
+ import fitz
294
+ import numpy as np
295
+ from PIL import Image
296
+ except ImportError as exc:
297
+ raise RuntimeError(
298
+ "OCR image preprocessing needs opencv-python, numpy, Pillow, and PyMuPDF."
299
+ ) from exc
300
+
301
+ matrix = fitz.Matrix(self.render_scale, self.render_scale)
302
+ pixmap = page.get_pixmap(matrix=matrix, alpha=False)
303
+ pil_image = Image.open(io.BytesIO(pixmap.tobytes("png"))).convert("RGB")
304
+ array = np.array(pil_image)
305
+ gray = cv2.cvtColor(array, cv2.COLOR_RGB2GRAY)
306
+ if self.fast_ocr:
307
+ _, thresholded = cv2.threshold(gray, 0, 255, cv2.THRESH_BINARY + cv2.THRESH_OTSU)
308
+ else:
309
+ denoised = cv2.fastNlMeansDenoising(gray, h=10)
310
+ thresholded = cv2.adaptiveThreshold(
311
+ denoised,
312
+ 255,
313
+ cv2.ADAPTIVE_THRESH_GAUSSIAN_C,
314
+ cv2.THRESH_BINARY,
315
+ 31,
316
+ 11,
317
+ )
318
+ output = Image.fromarray(thresholded).convert("RGB")
319
+ buffer = io.BytesIO()
320
+ output.save(buffer, format="JPEG", quality=72, optimize=True)
321
+ return buffer.getvalue()
322
+
323
+ def _gemini_ocr(self, image_bytes: bytes) -> str:
324
+ try:
325
+ from google import genai
326
+ from google.genai import types
327
+ except ImportError as exc:
328
+ raise RuntimeError("Gemini OCR requires: pip install google-genai") from exc
329
+
330
+ api_keys = self._gemini_api_keys()
331
+ if not api_keys:
332
+ raise RuntimeError("Set GEMINI_API_KEY before using Gemini Vision OCR.")
333
+
334
+ prompt = (
335
+ "Extract all readable text from this preprocessed PDF page image. "
336
+ "Preserve natural reading order, headings, bullet points, tables as plain text, "
337
+ "and do not add commentary. Ignore photos, diagrams, icons, borders, handwriting-like noise, "
338
+ "and decorative/non-text visual content. Do not describe images. If there is no readable text, "
339
+ "return an empty response."
340
+ )
341
+ last_error: Exception | None = None
342
+ attempted_keys = 0
343
+ quota_blocked_keys = 0
344
+ for api_key in self._gemini_key_attempt_order(api_keys):
345
+ attempted_keys += 1
346
+ client = self._gemini_clients.get(api_key)
347
+ if client is None:
348
+ client = genai.Client(
349
+ api_key=api_key,
350
+ http_options=types.HttpOptions(timeout=self.gemini_timeout_ms),
351
+ )
352
+ self._gemini_clients[api_key] = client
353
+
354
+ quota_hit = False
355
+ for model in self.gemini_models:
356
+ for attempt in range(1, self.gemini_max_retries + 1):
357
+ try:
358
+ response = client.models.generate_content(
359
+ model=model,
360
+ contents=[
361
+ types.Part.from_bytes(data=image_bytes, mime_type="image/jpeg"),
362
+ prompt,
363
+ ],
364
+ )
365
+ self._advance_gemini_key_cursor(api_keys, api_key)
366
+ return (getattr(response, "text", None) or "").strip()
367
+ except Exception as exc:
368
+ last_error = exc
369
+ if self._is_quota_error(exc):
370
+ self._mark_gemini_key_quota_exhausted(api_keys, api_key)
371
+ quota_blocked_keys += 1
372
+ quota_hit = True
373
+ break
374
+
375
+ if not self._is_retryable_gemini_error(exc):
376
+ break
377
+
378
+ if attempt < self.gemini_max_retries:
379
+ time.sleep(self.gemini_retry_delay * attempt)
380
+
381
+ if quota_hit:
382
+ break
383
+
384
+ models = ", ".join(self.gemini_models)
385
+ raise RuntimeError(
386
+ "Gemini OCR is temporarily unavailable after retries. "
387
+ f"Tried {attempted_keys} active API key(s) out of {len(api_keys)} configured key(s); "
388
+ f"{quota_blocked_keys} key(s) returned quota/rate-limit errors. Models tried: {models}. "
389
+ f"Last error: {last_error}"
390
+ )
391
+
392
+ def _safe_gemini_ocr(self, image_bytes: bytes, page_number: int) -> tuple[str, str | None]:
393
+ try:
394
+ text = self._clean_text_only(self._gemini_ocr(image_bytes))
395
+ except Exception as exc:
396
+ return "", f"Page {page_number} skipped: OCR failed ({exc})."
397
+ if not self._has_enough_text(text, min_chars=2):
398
+ return "", f"Page {page_number} skipped: no readable text found."
399
+ return text, None
400
+
401
+ def _skipped_page_event(
402
+ self,
403
+ page_number: int,
404
+ page_count: int,
405
+ warning: str,
406
+ confidence: float | None = None,
407
+ ) -> dict:
408
+ return {
409
+ "type": "page",
410
+ "page_number": page_number,
411
+ "page_count": page_count,
412
+ "engine": "Skipped non-text page",
413
+ "route": "Skipped non-text content",
414
+ "direct_text_found": False,
415
+ "confidence": confidence,
416
+ "text": "",
417
+ "warning": warning,
418
+ }
419
+
420
+ def _stream_gemini_ocr_jobs(self, jobs: list[tuple[int, bytes]], page_count: int) -> Iterable[dict]:
421
+ worker_count = min(self.gemini_concurrency, len(jobs))
422
+ if worker_count <= 1:
423
+ for page_number, image_bytes in jobs:
424
+ yield self._gemini_page_event(page_number, page_count, image_bytes)
425
+ return
426
+
427
+ with ThreadPoolExecutor(max_workers=worker_count) as executor:
428
+ futures = {
429
+ executor.submit(self._gemini_page_event, page_number, page_count, image_bytes): page_number
430
+ for page_number, image_bytes in jobs
431
+ }
432
+ for future in as_completed(futures):
433
+ try:
434
+ yield future.result()
435
+ except Exception as exc:
436
+ yield self._skipped_page_event(futures[future], page_count, f"OCR failed: {exc}")
437
+
438
+ def _gemini_page_event(self, page_number: int, page_count: int, image_bytes: bytes) -> dict:
439
+ ocr_text, warning = self._safe_gemini_ocr(image_bytes, page_number)
440
+ return {
441
+ "type": "page",
442
+ "page_number": page_number,
443
+ "page_count": page_count,
444
+ "engine": "Gemini Vision OCR" if ocr_text else "Skipped non-text page",
445
+ "route": "PyMuPDF image -> OpenCV preprocess -> Gemini Vision OCR",
446
+ "direct_text_found": False,
447
+ "confidence": None,
448
+ "text": ocr_text,
449
+ "warning": warning,
450
+ }
451
+
452
+ def _gemini_api_keys(self) -> list[str]:
453
+ keys = []
454
+ for key in sorted(os.environ, key=self._gemini_env_key_sort):
455
+ value = os.environ[key]
456
+ if self._is_gemini_key_name(key):
457
+ self._add_gemini_key_values(keys, value)
458
+
459
+ for file_name in (".env", "env"):
460
+ env_path = Path(file_name)
461
+ if not env_path.exists():
462
+ continue
463
+ for line in env_path.read_text(encoding="utf-8", errors="ignore").splitlines():
464
+ stripped = line.strip()
465
+ if not stripped or stripped.startswith("#") or "=" not in stripped:
466
+ continue
467
+ key, value = stripped.split("=", 1)
468
+ key = key.strip()
469
+ value = value.strip().strip('"').strip("'")
470
+ if self._is_gemini_key_name(key):
471
+ self._add_gemini_key_values(keys, value)
472
+ return keys
473
+
474
+ def _gemini_key_attempt_order(self, api_keys: list[str]) -> list[str]:
475
+ with self._gemini_key_lock:
476
+ now = time.time()
477
+ active_keys = [
478
+ api_key
479
+ for api_key in api_keys
480
+ if self._gemini_quota_blocked_until.get(api_key, 0.0) <= now
481
+ ]
482
+ if not active_keys:
483
+ active_keys = api_keys
484
+ active_key_set = set(active_keys)
485
+ start = self._gemini_key_cursor % len(api_keys)
486
+ ordered_keys = api_keys[start:] + api_keys[:start]
487
+ return [api_key for api_key in ordered_keys if api_key in active_key_set]
488
+
489
+ def _advance_gemini_key_cursor(self, api_keys: list[str], api_key: str) -> None:
490
+ with self._gemini_key_lock:
491
+ try:
492
+ self._gemini_key_cursor = (api_keys.index(api_key) + 1) % len(api_keys)
493
+ except ValueError:
494
+ self._gemini_key_cursor = 0
495
+
496
+ def _mark_gemini_key_quota_exhausted(self, api_keys: list[str], api_key: str) -> None:
497
+ with self._gemini_key_lock:
498
+ self._gemini_quota_blocked_until[api_key] = time.time() + self.gemini_key_quota_cooldown
499
+ try:
500
+ self._gemini_key_cursor = (api_keys.index(api_key) + 1) % len(api_keys)
501
+ except ValueError:
502
+ self._gemini_key_cursor = 0
503
+
504
+ def _is_gemini_key_name(self, key: str) -> bool:
505
+ return (
506
+ key == "GEMINI_API_KEY"
507
+ or key == "GOOGLE_API_KEY"
508
+ or key == "GEMINI_API_KEYS"
509
+ or key.startswith("GEMINI_API_KEY_")
510
+ )
511
+
512
+ def _gemini_env_key_sort(self, key: str) -> tuple[int, int, str]:
513
+ match = re.fullmatch(r"GEMINI_API_KEY_(\d+)", key)
514
+ if match:
515
+ return (0, int(match.group(1)), key)
516
+ if key == "GEMINI_API_KEY":
517
+ return (1, 0, key)
518
+ if key == "GEMINI_API_KEYS":
519
+ return (2, 0, key)
520
+ if key == "GOOGLE_API_KEY":
521
+ return (3, 0, key)
522
+ return (4, 0, key)
523
+
524
+ def _add_gemini_key_values(self, keys: list[str], value: str) -> None:
525
+ for api_key in (part.strip() for part in value.split(",")):
526
+ if api_key and api_key not in keys:
527
+ keys.append(api_key)
528
+
529
+ def _gemini_model_candidates(self, configured_model: str | None) -> list[str]:
530
+ configured_models = os.getenv("GEMINI_MODELS") or configured_model
531
+ if configured_models:
532
+ models = [
533
+ model.strip()
534
+ for model in configured_models.split(",")
535
+ if model.strip()
536
+ ]
537
+ if models:
538
+ return models
539
+ return list(DEFAULT_GEMINI_MODELS)
540
+
541
+ def _is_retryable_gemini_error(self, error: Exception) -> bool:
542
+ message = str(error).lower()
543
+ retryable_markers = (
544
+ "503",
545
+ "unavailable",
546
+ "high demand",
547
+ "429",
548
+ "resource_exhausted",
549
+ "rate limit",
550
+ "quota",
551
+ "deadline",
552
+ "timeout",
553
+ )
554
+ return any(marker in message for marker in retryable_markers)
555
+
556
+ def _is_quota_error(self, error: Exception | None) -> bool:
557
+ if error is None:
558
+ return False
559
+ message = str(error).lower()
560
+ return "429" in message or "resource_exhausted" in message or "quota" in message
561
+
562
+ def _load_local_env(self) -> None:
563
+ for file_name in (".env", "env"):
564
+ env_path = Path(file_name)
565
+ if not env_path.exists():
566
+ continue
567
+
568
+ for line in env_path.read_text(encoding="utf-8", errors="ignore").splitlines():
569
+ stripped = line.strip()
570
+ if not stripped or stripped.startswith("#") or "=" not in stripped:
571
+ continue
572
+
573
+ key, value = stripped.split("=", 1)
574
+ key = key.strip()
575
+ value = value.strip().strip('"').strip("'")
576
+ if key and key not in os.environ:
577
+ os.environ[key] = value
578
+
579
+ def _env_int(self, key: str, fallback_key: str | None = None, default: int = 0, minimum: int = 0) -> int:
580
+ raw_value = os.getenv(key)
581
+ if raw_value is None and fallback_key:
582
+ raw_value = os.getenv(fallback_key)
583
+ if raw_value is None:
584
+ return default
585
+ try:
586
+ value = int(raw_value)
587
+ except ValueError:
588
+ return default
589
+ return max(value, minimum)
590
+
591
+ def _env_float(self, key: str, fallback_key: str | None = None, default: float = 0.0, minimum: float = 0.0) -> float:
592
+ raw_value = os.getenv(key)
593
+ if raw_value is None and fallback_key:
594
+ raw_value = os.getenv(fallback_key)
595
+ if raw_value is None:
596
+ return default
597
+ try:
598
+ value = float(raw_value)
599
+ except ValueError:
600
+ return default
601
+ return max(value, minimum)
602
+
603
+ def _trocr_ocr(self, image_bytes: bytes) -> tuple[str, float]:
604
+ try:
605
+ import torch
606
+ from PIL import Image
607
+ from transformers import TrOCRProcessor, VisionEncoderDecoderModel
608
+ except ImportError as exc:
609
+ raise RuntimeError(
610
+ "TrOCR requires: pip install torch transformers Pillow"
611
+ ) from exc
612
+
613
+ if self._trocr_processor is None or self._trocr_model is None:
614
+ self._trocr_processor = TrOCRProcessor.from_pretrained(self.trocr_model_name)
615
+ self._trocr_model = VisionEncoderDecoderModel.from_pretrained(self.trocr_model_name)
616
+
617
+ image = Image.open(io.BytesIO(image_bytes)).convert("RGB")
618
+ pixel_values = self._trocr_processor(images=image, return_tensors="pt").pixel_values
619
+
620
+ with torch.no_grad():
621
+ generated = self._trocr_model.generate(
622
+ pixel_values,
623
+ max_new_tokens=512,
624
+ output_scores=True,
625
+ return_dict_in_generate=True,
626
+ )
627
+
628
+ text = self._trocr_processor.batch_decode(
629
+ generated.sequences,
630
+ skip_special_tokens=True,
631
+ )[0].strip()
632
+ confidence = self._score_trocr_confidence(generated.scores)
633
+ return text, confidence
634
+
635
+ def _score_trocr_confidence(self, scores: Iterable) -> float:
636
+ try:
637
+ import torch
638
+ except ImportError:
639
+ return 0.0
640
+
641
+ confidences = []
642
+ for score in scores:
643
+ probabilities = torch.softmax(score, dim=-1)
644
+ confidences.append(float(probabilities.max()))
645
+
646
+ if not confidences:
647
+ return 0.0
648
+ return sum(confidences) / len(confidences)
649
+
650
+ def _finalize_ocr_result(
651
+ self,
652
+ route: str,
653
+ page_count: int,
654
+ direct_text_found: bool,
655
+ pages: list[PageResult],
656
+ ) -> ExtractionResult:
657
+ merged_text = "\n\n".join(
658
+ f"--- Page {page.page_number} ---\n{page.text.strip()}"
659
+ for page in pages
660
+ if page.text.strip()
661
+ )
662
+ return ExtractionResult(
663
+ text=self._clean_text(merged_text),
664
+ route=route,
665
+ page_count=page_count,
666
+ direct_text_found=direct_text_found,
667
+ pages=pages,
668
+ )
669
+
670
+ def _clean_text(self, text: str) -> str:
671
+ text = text.replace("\x00", "")
672
+ text = re.sub(r"[ \t]+", " ", text)
673
+ text = re.sub(r"\n{3,}", "\n\n", text)
674
+ text = "\n".join(line.rstrip() for line in text.splitlines())
675
+ return text.strip()
676
+
677
+ def _clean_text_only(self, text: str) -> str:
678
+ cleaned = self._clean_text(text)
679
+ lines = []
680
+ for line in cleaned.splitlines():
681
+ stripped = line.strip()
682
+ if not stripped:
683
+ lines.append("")
684
+ continue
685
+ if self._looks_like_visual_description(stripped):
686
+ continue
687
+ lines.append(stripped)
688
+ text_only = self._clean_text("\n".join(lines))
689
+ return text_only if self._has_real_text_signal(text_only) else ""
690
+
691
+ def _looks_like_visual_description(self, line: str) -> bool:
692
+ normalized = re.sub(r"\s+", " ", line.lower()).strip()
693
+ visual_description_patterns = (
694
+ r"^(the|this|an|a)\s+(image|photo|picture|diagram|figure|chart|graph|illustration|logo|icon)\s+",
695
+ r"^(the|this)\s+page\s+(contains|shows|appears|has|is)\s+",
696
+ r"^(it|this)\s+(shows|appears|looks like|contains|depicts)\s+",
697
+ r"^(i can see|there is|there are)\s+",
698
+ r"\b(no readable text|no text|cannot extract|not able to extract)\b",
699
+ r"\b(image shows|picture shows|diagram shows|photo shows|chart shows|graph shows)\b",
700
+ )
701
+ if any(re.search(pattern, normalized) for pattern in visual_description_patterns):
702
+ return True
703
+ visual_words = {
704
+ "image",
705
+ "photo",
706
+ "picture",
707
+ "diagram",
708
+ "illustration",
709
+ "visual",
710
+ "icon",
711
+ "logo",
712
+ "background",
713
+ "border",
714
+ "shape",
715
+ "graphic",
716
+ }
717
+ tokens = re.findall(r"[a-z0-9]+", normalized)
718
+ if not tokens:
719
+ return True
720
+ visual_count = sum(1 for token in tokens if token in visual_words)
721
+ return visual_count >= 2 and len(tokens) <= 18
722
+
723
+ def _has_real_text_signal(self, text: str) -> bool:
724
+ normalized = re.sub(r"\s+", " ", text or "").strip()
725
+ if not normalized:
726
+ return False
727
+ alnum_count = sum(char.isalnum() for char in normalized)
728
+ alpha_count = sum(char.isalpha() for char in normalized)
729
+ return alnum_count >= 2 and alpha_count >= 2
730
+
731
+
732
+ def extract_pdf_to_text(pdf_path: str | Path) -> ExtractionResult:
733
+ return PdfTextExtractor().extract(pdf_path)
734
+
735
+
736
+ def extract_uploaded_pdf(file_name: str, file_bytes: bytes) -> ExtractionResult:
737
+ suffix = Path(file_name).suffix or ".pdf"
738
+ with tempfile.NamedTemporaryFile(delete=False, suffix=suffix) as temp_file:
739
+ temp_file.write(file_bytes)
740
+ temp_path = Path(temp_file.name)
741
+
742
+ try:
743
+ return extract_pdf_to_text(temp_path)
744
+ finally:
745
+ temp_path.unlink(missing_ok=True)
746
+
747
+
748
+ def stream_uploaded_pdf(file_name: str, file_bytes: bytes) -> Iterable[dict]:
749
+ suffix = Path(file_name).suffix or ".pdf"
750
+ with tempfile.NamedTemporaryFile(delete=False, suffix=suffix) as temp_file:
751
+ temp_file.write(file_bytes)
752
+ temp_path = Path(temp_file.name)
753
+
754
+ try:
755
+ yield from PdfTextExtractor().stream_extract(temp_path)
756
+ finally:
757
+ temp_path.unlink(missing_ok=True)
rag_pipeline.py ADDED
@@ -0,0 +1,1361 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ from __future__ import annotations
2
+
3
+ import hashlib
4
+ import json
5
+ import math
6
+ import os
7
+ import re
8
+ import sqlite3
9
+ import time
10
+ import unicodedata
11
+ import uuid
12
+ from collections import Counter, defaultdict
13
+ from dataclasses import asdict, dataclass, field
14
+ from pathlib import Path
15
+ from typing import Iterable
16
+
17
+
18
+ DATA_DIR = Path("data")
19
+ DB_PATH = DATA_DIR / "smartnotes.sqlite"
20
+ DEFAULT_EMBED_DIM = 384
21
+ DEFAULT_TOP_K = 50
22
+ UNRELATED_PDF_ANSWER = "Sorry, I can't give any answer because your question is not related to the PDF."
23
+
24
+
25
+ @dataclass
26
+ class ParentChunk:
27
+ id: str
28
+ document_id: str
29
+ index: int
30
+ text: str
31
+ token_count: int
32
+ metadata: dict
33
+
34
+
35
+ @dataclass
36
+ class ChildChunk:
37
+ id: str
38
+ parent_id: str
39
+ document_id: str
40
+ index: int
41
+ strategy: str
42
+ text: str
43
+ token_count: int
44
+ embedding: list[float] = field(default_factory=list)
45
+ metadata: dict = field(default_factory=dict)
46
+
47
+
48
+ @dataclass
49
+ class RetrievalHit:
50
+ child_id: str
51
+ parent_id: str
52
+ document_id: str
53
+ strategy: str
54
+ child_text: str
55
+ parent_text: str
56
+ vector_score: float = 0.0
57
+ bm25_score: float = 0.0
58
+ hybrid_score: float = 0.0
59
+ rerank_score: float = 0.0
60
+ grade: str = "unknown"
61
+ metadata: dict = field(default_factory=dict)
62
+
63
+
64
+ class SmartNotesRepository:
65
+ """PostgreSQL-first repository with SQLite fallback for local UI testing."""
66
+
67
+ def __init__(self) -> None:
68
+ DATA_DIR.mkdir(exist_ok=True)
69
+ self.postgres_dsn = os.getenv("POSTGRES_DSN")
70
+ self.backend = "sqlite"
71
+ self._pg = None
72
+ if self.postgres_dsn:
73
+ try:
74
+ import psycopg
75
+
76
+ self._pg = psycopg.connect(self.postgres_dsn)
77
+ self.backend = "postgresql"
78
+ except Exception:
79
+ self._pg = None
80
+ self.backend = "sqlite"
81
+ self._sqlite = sqlite3.connect(DB_PATH, check_same_thread=False)
82
+ self._sqlite.row_factory = sqlite3.Row
83
+ self._init_sqlite()
84
+
85
+ def _init_sqlite(self) -> None:
86
+ self._sqlite.executescript(
87
+ """
88
+ create table if not exists documents (
89
+ id text primary key,
90
+ file_name text,
91
+ status text,
92
+ metadata text,
93
+ cleaned_text text,
94
+ created_at real,
95
+ updated_at real
96
+ );
97
+ create table if not exists parent_chunks (
98
+ id text primary key,
99
+ document_id text,
100
+ chunk_index integer,
101
+ text text,
102
+ token_count integer,
103
+ metadata text
104
+ );
105
+ create table if not exists child_chunks (
106
+ id text primary key,
107
+ parent_id text,
108
+ document_id text,
109
+ chunk_index integer,
110
+ strategy text,
111
+ text text,
112
+ token_count integer,
113
+ embedding text,
114
+ metadata text
115
+ );
116
+ create table if not exists retrieval_logs (
117
+ id text primary key,
118
+ document_id text,
119
+ query text,
120
+ rewritten_query text,
121
+ status text,
122
+ answer text,
123
+ hits text,
124
+ created_at real
125
+ );
126
+ create table if not exists feedback (
127
+ id text primary key,
128
+ document_id text,
129
+ query text,
130
+ rating integer,
131
+ comment text,
132
+ created_at real
133
+ );
134
+ """
135
+ )
136
+ self._sqlite.commit()
137
+
138
+ def create_document(self, file_name: str, metadata: dict) -> str:
139
+ document_id = str(uuid.uuid4())
140
+ now = time.time()
141
+ self._sqlite.execute(
142
+ """
143
+ insert into documents (id, file_name, status, metadata, cleaned_text, created_at, updated_at)
144
+ values (?, ?, ?, ?, ?, ?, ?)
145
+ """,
146
+ (document_id, file_name, "processing", json.dumps(metadata), "", now, now),
147
+ )
148
+ self._sqlite.commit()
149
+ return document_id
150
+
151
+ def update_document_status(self, document_id: str, status: str) -> None:
152
+ self._sqlite.execute(
153
+ "update documents set status = ?, updated_at = ? where id = ?",
154
+ (status, time.time(), document_id),
155
+ )
156
+ self._sqlite.commit()
157
+
158
+ def save_cleaned_text(self, document_id: str, cleaned_text: str, metadata: dict) -> None:
159
+ self._sqlite.execute(
160
+ "update documents set cleaned_text = ?, metadata = ?, updated_at = ? where id = ?",
161
+ (cleaned_text, json.dumps(metadata), time.time(), document_id),
162
+ )
163
+ self._sqlite.commit()
164
+
165
+ def save_chunks(self, parents: list[ParentChunk], children: list[ChildChunk]) -> None:
166
+ self._sqlite.executemany(
167
+ """
168
+ insert or replace into parent_chunks (id, document_id, chunk_index, text, token_count, metadata)
169
+ values (?, ?, ?, ?, ?, ?)
170
+ """,
171
+ [
172
+ (
173
+ chunk.id,
174
+ chunk.document_id,
175
+ chunk.index,
176
+ chunk.text,
177
+ chunk.token_count,
178
+ json.dumps(chunk.metadata),
179
+ )
180
+ for chunk in parents
181
+ ],
182
+ )
183
+ self._sqlite.executemany(
184
+ """
185
+ insert or replace into child_chunks
186
+ (id, parent_id, document_id, chunk_index, strategy, text, token_count, embedding, metadata)
187
+ values (?, ?, ?, ?, ?, ?, ?, ?, ?)
188
+ """,
189
+ [
190
+ (
191
+ chunk.id,
192
+ chunk.parent_id,
193
+ chunk.document_id,
194
+ chunk.index,
195
+ chunk.strategy,
196
+ chunk.text,
197
+ chunk.token_count,
198
+ json.dumps(chunk.embedding),
199
+ json.dumps(chunk.metadata),
200
+ )
201
+ for chunk in children
202
+ ],
203
+ )
204
+ self._sqlite.commit()
205
+
206
+ def get_document(self, document_id: str) -> dict | None:
207
+ row = self._sqlite.execute(
208
+ "select * from documents where id = ?",
209
+ (document_id,),
210
+ ).fetchone()
211
+ if not row:
212
+ return None
213
+ return {
214
+ "id": row["id"],
215
+ "file_name": row["file_name"],
216
+ "status": row["status"],
217
+ "metadata": json.loads(row["metadata"] or "{}"),
218
+ "cleaned_text": row["cleaned_text"] or "",
219
+ "storage_backend": self.backend,
220
+ }
221
+
222
+ def get_children(self, document_id: str) -> list[ChildChunk]:
223
+ rows = self._sqlite.execute(
224
+ "select * from child_chunks where document_id = ? order by chunk_index",
225
+ (document_id,),
226
+ ).fetchall()
227
+ return [
228
+ ChildChunk(
229
+ id=row["id"],
230
+ parent_id=row["parent_id"],
231
+ document_id=row["document_id"],
232
+ index=row["chunk_index"],
233
+ strategy=row["strategy"],
234
+ text=row["text"],
235
+ token_count=row["token_count"],
236
+ embedding=json.loads(row["embedding"] or "[]"),
237
+ metadata=json.loads(row["metadata"] or "{}"),
238
+ )
239
+ for row in rows
240
+ ]
241
+
242
+ def get_parent_map(self, document_id: str) -> dict[str, ParentChunk]:
243
+ rows = self._sqlite.execute(
244
+ "select * from parent_chunks where document_id = ?",
245
+ (document_id,),
246
+ ).fetchall()
247
+ return {
248
+ row["id"]: ParentChunk(
249
+ id=row["id"],
250
+ document_id=row["document_id"],
251
+ index=row["chunk_index"],
252
+ text=row["text"],
253
+ token_count=row["token_count"],
254
+ metadata=json.loads(row["metadata"] or "{}"),
255
+ )
256
+ for row in rows
257
+ }
258
+
259
+ def save_retrieval_log(
260
+ self,
261
+ document_id: str,
262
+ query: str,
263
+ rewritten_query: str,
264
+ status: str,
265
+ answer: str,
266
+ hits: list[RetrievalHit],
267
+ ) -> str:
268
+ log_id = str(uuid.uuid4())
269
+ self._sqlite.execute(
270
+ """
271
+ insert into retrieval_logs
272
+ (id, document_id, query, rewritten_query, status, answer, hits, created_at)
273
+ values (?, ?, ?, ?, ?, ?, ?, ?)
274
+ """,
275
+ (
276
+ log_id,
277
+ document_id,
278
+ query,
279
+ rewritten_query,
280
+ status,
281
+ answer,
282
+ json.dumps([self._hit_payload(hit) for hit in hits]),
283
+ time.time(),
284
+ ),
285
+ )
286
+ self._sqlite.commit()
287
+ return log_id
288
+
289
+ def save_feedback(self, document_id: str, query: str, rating: int, comment: str) -> str:
290
+ feedback_id = str(uuid.uuid4())
291
+ self._sqlite.execute(
292
+ """
293
+ insert into feedback (id, document_id, query, rating, comment, created_at)
294
+ values (?, ?, ?, ?, ?, ?)
295
+ """,
296
+ (feedback_id, document_id, query, rating, comment, time.time()),
297
+ )
298
+ self._sqlite.commit()
299
+ return feedback_id
300
+
301
+ def _hit_payload(self, hit: RetrievalHit) -> dict:
302
+ data = asdict(hit)
303
+ data["child_text"] = hit.child_text[:900]
304
+ data["parent_text"] = hit.parent_text[:1400]
305
+ return data
306
+
307
+
308
+ class TextCleaner:
309
+ def clean(self, text: str) -> str:
310
+ try:
311
+ import ftfy
312
+
313
+ text = ftfy.fix_text(text)
314
+ except Exception:
315
+ pass
316
+ text = unicodedata.normalize("NFKC", text or "")
317
+ text = text.replace("\x00", "")
318
+ text = re.sub(r"---\s*Page\s+\d+\s*---", "\n", text, flags=re.IGNORECASE)
319
+ text = re.sub(r"[ \t]+", " ", text)
320
+ text = re.sub(r"(\w)-\n(\w)", r"\1\2", text)
321
+ text = re.sub(r"\n(?=[a-z])", " ", text)
322
+ text = re.sub(r"\n{3,}", "\n\n", text)
323
+ return "\n".join(line.strip() for line in text.splitlines()).strip()
324
+
325
+
326
+ class Chunker:
327
+ def parent_chunks(self, document_id: str, text: str, metadata: dict) -> list[ParentChunk]:
328
+ chunks = self._recursive_split(text, chunk_size=1400, overlap=180)
329
+ return [
330
+ ParentChunk(
331
+ id=str(uuid.uuid4()),
332
+ document_id=document_id,
333
+ index=index,
334
+ text=chunk,
335
+ token_count=self._token_count(chunk),
336
+ metadata={**metadata, "chunk_type": "parent"},
337
+ )
338
+ for index, chunk in enumerate(chunks)
339
+ ]
340
+
341
+ def child_chunks(self, parents: list[ParentChunk]) -> list[ChildChunk]:
342
+ children: list[ChildChunk] = []
343
+ index = 0
344
+ for parent in parents:
345
+ for strategy, chunks in (
346
+ ("fixed", self._fixed_split(parent.text, chunk_size=380, overlap=70)),
347
+ ("recursive", self._recursive_split(parent.text, chunk_size=520, overlap=90)),
348
+ ("semantic", self._semantic_split(parent.text, target_size=520)),
349
+ ):
350
+ for text in chunks:
351
+ children.append(
352
+ ChildChunk(
353
+ id=str(uuid.uuid4()),
354
+ parent_id=parent.id,
355
+ document_id=parent.document_id,
356
+ index=index,
357
+ strategy=strategy,
358
+ text=text,
359
+ token_count=self._token_count(text),
360
+ metadata={
361
+ **parent.metadata,
362
+ "chunk_type": "child",
363
+ "parent_index": parent.index,
364
+ "strategy": strategy,
365
+ },
366
+ )
367
+ )
368
+ index += 1
369
+ return children
370
+
371
+ def _fixed_split(self, text: str, chunk_size: int, overlap: int) -> list[str]:
372
+ words = text.split()
373
+ chunks = []
374
+ step = max(chunk_size - overlap, 1)
375
+ for start in range(0, len(words), step):
376
+ chunk = " ".join(words[start : start + chunk_size]).strip()
377
+ if chunk:
378
+ chunks.append(chunk)
379
+ return chunks
380
+
381
+ def _recursive_split(self, text: str, chunk_size: int, overlap: int) -> list[str]:
382
+ separators = ["\n\n", "\n", ". ", " "]
383
+ pieces = self._split_by_separators(text, separators, chunk_size)
384
+ chunks = []
385
+ current: list[str] = []
386
+ current_len = 0
387
+ for piece in pieces:
388
+ words = piece.split()
389
+ if current and current_len + len(words) > chunk_size:
390
+ chunks.append(" ".join(current).strip())
391
+ current = current[-overlap:] if overlap else []
392
+ current_len = len(current)
393
+ current.extend(words)
394
+ current_len += len(words)
395
+ if current:
396
+ chunks.append(" ".join(current).strip())
397
+ return [chunk for chunk in chunks if chunk]
398
+
399
+ def _semantic_split(self, text: str, target_size: int) -> list[str]:
400
+ sentences = re.split(r"(?<=[.!?])\s+", text)
401
+ chunks = []
402
+ current = []
403
+ current_len = 0
404
+ for sentence in sentences:
405
+ count = self._token_count(sentence)
406
+ if current and current_len + count > target_size:
407
+ chunks.append(" ".join(current).strip())
408
+ current = []
409
+ current_len = 0
410
+ current.append(sentence)
411
+ current_len += count
412
+ if current:
413
+ chunks.append(" ".join(current).strip())
414
+ return [chunk for chunk in chunks if chunk]
415
+
416
+ def _split_by_separators(self, text: str, separators: list[str], chunk_size: int) -> list[str]:
417
+ if self._token_count(text) <= chunk_size or not separators:
418
+ return [text]
419
+ separator = separators[0]
420
+ output = []
421
+ for part in text.split(separator):
422
+ output.extend(self._split_by_separators(part, separators[1:], chunk_size))
423
+ return output
424
+
425
+ def _token_count(self, text: str) -> int:
426
+ return len(text.split())
427
+
428
+
429
+ class EmbeddingService:
430
+ def __init__(self) -> None:
431
+ self.model_name = os.getenv("EMBEDDING_MODEL", "BAAI/bge-small-en-v1.5")
432
+ self.dim = DEFAULT_EMBED_DIM
433
+ self.backend = "hash"
434
+ self._model = None
435
+ if os.getenv("USE_LOCAL_EMBEDDING_MODEL", "0") == "1":
436
+ try:
437
+ from sentence_transformers import SentenceTransformer
438
+
439
+ self._model = SentenceTransformer(self.model_name)
440
+ self.backend = self.model_name
441
+ self.dim = int(self._model.get_sentence_embedding_dimension())
442
+ except Exception:
443
+ self._model = None
444
+
445
+ def embed_texts(self, texts: list[str]) -> list[list[float]]:
446
+ if self._model is not None:
447
+ vectors = self._model.encode(texts, normalize_embeddings=True).tolist()
448
+ return [[float(value) for value in vector] for vector in vectors]
449
+ return [self._hash_embedding(text) for text in texts]
450
+
451
+ def embed_query(self, query: str) -> list[float]:
452
+ return self.embed_texts([query])[0]
453
+
454
+ def _hash_embedding(self, text: str) -> list[float]:
455
+ vector = [0.0] * self.dim
456
+ tokens = tokenize(text)
457
+ for token in tokens:
458
+ digest = hashlib.sha256(token.encode("utf-8")).digest()
459
+ index = int.from_bytes(digest[:4], "big") % self.dim
460
+ sign = 1.0 if digest[4] % 2 == 0 else -1.0
461
+ vector[index] += sign
462
+ norm = math.sqrt(sum(value * value for value in vector)) or 1.0
463
+ return [value / norm for value in vector]
464
+
465
+
466
+ class QdrantVectorStore:
467
+ def __init__(self) -> None:
468
+ self.backend = "sqlite-vector-fallback"
469
+ self.collection = os.getenv("QDRANT_COLLECTION", "smartnotes_chunks")
470
+ self.client = None
471
+ self.url = os.getenv("QDRANT_URL")
472
+ if self.url:
473
+ try:
474
+ from qdrant_client import QdrantClient
475
+
476
+ self.client = QdrantClient(url=self.url, api_key=os.getenv("QDRANT_API_KEY"))
477
+ self.backend = "qdrant"
478
+ except Exception:
479
+ self.client = None
480
+
481
+ def upsert(self, children: list[ChildChunk]) -> None:
482
+ if self.client is None or not children:
483
+ return
484
+ try:
485
+ from qdrant_client.models import Distance, PointStruct, VectorParams
486
+
487
+ dim = len(children[0].embedding)
488
+ existing = [item.name for item in self.client.get_collections().collections]
489
+ if self.collection not in existing:
490
+ self.client.create_collection(
491
+ collection_name=self.collection,
492
+ vectors_config=VectorParams(size=dim, distance=Distance.COSINE),
493
+ )
494
+ self.client.upsert(
495
+ collection_name=self.collection,
496
+ points=[
497
+ PointStruct(
498
+ id=chunk.id,
499
+ vector=chunk.embedding,
500
+ payload={
501
+ "document_id": chunk.document_id,
502
+ "parent_id": chunk.parent_id,
503
+ "strategy": chunk.strategy,
504
+ },
505
+ )
506
+ for chunk in children
507
+ ],
508
+ )
509
+ except Exception:
510
+ self.client = None
511
+ self.backend = "sqlite-vector-fallback"
512
+
513
+ def search(self, query_vector: list[float], children: list[ChildChunk], top_k: int) -> list[tuple[str, float]]:
514
+ if self.client is not None:
515
+ try:
516
+ result = self.client.search(
517
+ collection_name=self.collection,
518
+ query_vector=query_vector,
519
+ limit=top_k,
520
+ )
521
+ return [(str(point.id), float(point.score)) for point in result]
522
+ except Exception:
523
+ pass
524
+ scored = [
525
+ (chunk.id, cosine_similarity(query_vector, chunk.embedding))
526
+ for chunk in children
527
+ ]
528
+ return sorted(scored, key=lambda item: item[1], reverse=True)[:top_k]
529
+
530
+
531
+ class BM25Index:
532
+ def search(self, query: str, children: list[ChildChunk], top_k: int) -> list[tuple[str, float]]:
533
+ query_tokens = tokenize(query)
534
+ docs = [tokenize(chunk.text) for chunk in children]
535
+ if not docs:
536
+ return []
537
+ avgdl = sum(len(doc) for doc in docs) / len(docs)
538
+ df = Counter()
539
+ for doc in docs:
540
+ for token in set(doc):
541
+ df[token] += 1
542
+ scores = []
543
+ for chunk, doc in zip(children, docs):
544
+ score = self._score_doc(query_tokens, doc, df, len(docs), avgdl)
545
+ scores.append((chunk.id, score))
546
+ return sorted(scores, key=lambda item: item[1], reverse=True)[:top_k]
547
+
548
+ def _score_doc(
549
+ self,
550
+ query_tokens: list[str],
551
+ doc: list[str],
552
+ df: Counter,
553
+ total_docs: int,
554
+ avgdl: float,
555
+ ) -> float:
556
+ counts = Counter(doc)
557
+ k1 = 1.5
558
+ b = 0.75
559
+ score = 0.0
560
+ for token in query_tokens:
561
+ if not counts[token]:
562
+ continue
563
+ idf = math.log(1 + (total_docs - df[token] + 0.5) / (df[token] + 0.5))
564
+ numerator = counts[token] * (k1 + 1)
565
+ denominator = counts[token] + k1 * (1 - b + b * len(doc) / (avgdl or 1))
566
+ score += idf * numerator / denominator
567
+ return score
568
+
569
+
570
+ class Reranker:
571
+ def __init__(self) -> None:
572
+ self.model_name = os.getenv("RERANKER_MODEL", "BAAI/bge-reranker-base")
573
+ self.backend = "token-overlap"
574
+ self._model = None
575
+ if os.getenv("USE_LOCAL_RERANKER_MODEL", "0") == "1":
576
+ try:
577
+ from sentence_transformers import CrossEncoder
578
+
579
+ self._model = CrossEncoder(self.model_name)
580
+ self.backend = self.model_name
581
+ except Exception:
582
+ self._model = None
583
+
584
+ def rerank(self, query: str, hits: list[RetrievalHit]) -> list[RetrievalHit]:
585
+ if self._model is not None and hits:
586
+ scores = self._model.predict([(query, hit.child_text) for hit in hits])
587
+ for hit, score in zip(hits, scores):
588
+ hit.rerank_score = float(score)
589
+ else:
590
+ query_tokens = set(tokenize(query))
591
+ for hit in hits:
592
+ chunk_tokens = set(tokenize(hit.child_text))
593
+ hit.rerank_score = len(query_tokens & chunk_tokens) / max(len(query_tokens), 1)
594
+ return sorted(hits, key=lambda hit: hit.rerank_score, reverse=True)
595
+
596
+
597
+ class LLMService:
598
+ def __init__(self) -> None:
599
+ load_local_env()
600
+ self.provider = os.getenv("ANSWER_PROVIDER", "gemini")
601
+ self.model = os.getenv("ANSWER_MODEL", "gemini-2.5-flash")
602
+ self._gemini_clients = {}
603
+ self._gemini_key_cursor = 0
604
+ self._gemini_quota_blocked_until: dict[str, float] = {}
605
+ self.gemini_timeout_ms = int(
606
+ min(env_float("GEMINI_TIMEOUT", default=20.0, minimum=5.0), 120.0)
607
+ * 1000
608
+ )
609
+ self.gemini_key_quota_cooldown = min(
610
+ env_float("GEMINI_KEY_QUOTA_COOLDOWN", default=300.0, minimum=0.0),
611
+ 86400.0,
612
+ )
613
+
614
+ def rewrite_query(self, query: str) -> str:
615
+ if len(query.split()) >= 4:
616
+ return query.strip()
617
+ prompt = f"Rewrite this user question into a precise PDF search query. Return only the query.\nQuestion: {query}"
618
+ return self._generate(prompt, fallback=query).strip() or query
619
+
620
+ def grade_retrieval(self, query: str, hits: list[RetrievalHit]) -> str:
621
+ if not hits:
622
+ return "bad"
623
+ best = max(hit.rerank_score for hit in hits)
624
+ lexical = max(hit.hybrid_score for hit in hits)
625
+ return "good" if best >= 0.08 or lexical >= 0.12 else "bad"
626
+
627
+ def answer(self, query: str, parent_contexts: list[ParentChunk]) -> str:
628
+ if not parent_contexts:
629
+ return UNRELATED_PDF_ANSWER
630
+ mode = self._answer_mode(query)
631
+ fallback = self._fallback_answer(query, parent_contexts, mode)
632
+ context = "\n\n".join(
633
+ f"[Source {index + 1} | parent={chunk.id}]\n{chunk.text}"
634
+ for index, chunk in enumerate(parent_contexts)
635
+ )
636
+ prompt = (
637
+ "You are SmartNotes AI. Answer using only the PDF context.\n"
638
+ "Rules:\n"
639
+ "- Give a direct answer first.\n"
640
+ "- Do not dump the whole context.\n"
641
+ "- If the user asks 'what is' or 'define', give only the definition in 1-2 sentences.\n"
642
+ "- Answer in normal readable text only.\n"
643
+ "- Do not return equations, raw calculations, URLs, or OCR garbage unless the user explicitly asks for them.\n"
644
+ "- If the user asks in detail, explain with key points, types, examples, and relevant facts from the PDF.\n"
645
+ "- Remove page markers and OCR noise.\n"
646
+ f"- If the clean text answer is not present or the question is not related to the PDF, say exactly: {UNRELATED_PDF_ANSWER}\n"
647
+ "- End with citation like [Source 1].\n\n"
648
+ f"Question: {query}\n\nContext:\n{context}\n\nAnswer:"
649
+ )
650
+ return self._postprocess_answer(self._generate(prompt, fallback=fallback))
651
+
652
+ def _generate(self, prompt: str, fallback: str) -> str:
653
+ if self.provider == "openai" and os.getenv("OPENAI_API_KEY"):
654
+ try:
655
+ from openai import OpenAI
656
+
657
+ client = OpenAI(api_key=os.getenv("OPENAI_API_KEY"))
658
+ response = client.chat.completions.create(
659
+ model=os.getenv("OPENAI_MODEL", "gpt-4.1-mini"),
660
+ messages=[{"role": "user", "content": prompt}],
661
+ )
662
+ return response.choices[0].message.content or fallback
663
+ except Exception:
664
+ return fallback
665
+ api_keys = gemini_api_keys()
666
+ if api_keys:
667
+ try:
668
+ from google import genai
669
+ from google.genai import types
670
+ except Exception:
671
+ return fallback
672
+ for api_key in self._gemini_key_attempt_order(api_keys):
673
+ try:
674
+ client = self._gemini_clients.get(api_key)
675
+ if client is None:
676
+ client = genai.Client(
677
+ api_key=api_key,
678
+ http_options=types.HttpOptions(timeout=self.gemini_timeout_ms),
679
+ )
680
+ self._gemini_clients[api_key] = client
681
+ response = client.models.generate_content(
682
+ model=self.model,
683
+ contents=prompt,
684
+ )
685
+ self._advance_gemini_key_cursor(api_keys, api_key)
686
+ return (getattr(response, "text", None) or fallback).strip()
687
+ except Exception as exc:
688
+ if is_gemini_quota_error(exc):
689
+ self._mark_gemini_key_quota_exhausted(api_keys, api_key)
690
+ continue
691
+ return fallback
692
+
693
+ def _gemini_key_attempt_order(self, api_keys: list[str]) -> list[str]:
694
+ now = time.time()
695
+ active_keys = [
696
+ api_key
697
+ for api_key in api_keys
698
+ if self._gemini_quota_blocked_until.get(api_key, 0.0) <= now
699
+ ]
700
+ if not active_keys:
701
+ active_keys = api_keys
702
+ active_key_set = set(active_keys)
703
+ start = self._gemini_key_cursor % len(api_keys)
704
+ ordered_keys = api_keys[start:] + api_keys[:start]
705
+ return [api_key for api_key in ordered_keys if api_key in active_key_set]
706
+
707
+ def _advance_gemini_key_cursor(self, api_keys: list[str], api_key: str) -> None:
708
+ try:
709
+ self._gemini_key_cursor = (api_keys.index(api_key) + 1) % len(api_keys)
710
+ except ValueError:
711
+ self._gemini_key_cursor = 0
712
+
713
+ def _mark_gemini_key_quota_exhausted(self, api_keys: list[str], api_key: str) -> None:
714
+ self._gemini_quota_blocked_until[api_key] = time.time() + self.gemini_key_quota_cooldown
715
+ self._advance_gemini_key_cursor(api_keys, api_key)
716
+
717
+ def _fallback_answer(self, query: str, parent_contexts: list[ParentChunk], mode: str = "auto") -> str:
718
+ query_tokens = set(tokenize(query))
719
+ targets = self._query_targets(query)
720
+ text = "\n".join(chunk.text for chunk in parent_contexts)
721
+ if mode == "definition":
722
+ definition = self._definition_answer(text, targets)
723
+ if definition:
724
+ return self._postprocess_answer(definition + " [Source 1]")
725
+ return UNRELATED_PDF_ANSWER
726
+ sentences = self._candidate_sentences(text)
727
+ ranked = sorted(
728
+ sentences,
729
+ key=lambda sentence: self._sentence_score(query_tokens, targets, sentence),
730
+ reverse=True,
731
+ )
732
+ useful = []
733
+ for sentence in ranked:
734
+ sentence_score = self._sentence_score(query_tokens, targets, sentence)
735
+ if mode == "detailed" and sentence_score <= 0:
736
+ continue
737
+ cleaned = self._clean_answer_sentence(sentence)
738
+ if not cleaned or len(cleaned.split()) < 5 or self._is_noisy_answer_sentence(cleaned):
739
+ continue
740
+ if cleaned in useful:
741
+ continue
742
+ useful.append(cleaned)
743
+ limit = 1 if mode == "definition" else 5
744
+ if len(useful) >= limit:
745
+ break
746
+ if not useful:
747
+ return UNRELATED_PDF_ANSWER
748
+ if mode == "detailed" and len(useful) > 1:
749
+ answer = useful[0]
750
+ bullets = "\n".join(f"- {item}" for item in useful[1:])
751
+ return self._postprocess_answer(f"{answer}\n\nKey points:\n{bullets}\n\n[Source 1]")
752
+ return self._postprocess_answer(" ".join(useful) + " [Source 1]")
753
+
754
+ def _candidate_sentences(self, text: str) -> list[str]:
755
+ text = re.sub(r"---\s*Page\s+\d+\s*---", " ", text, flags=re.IGNORECASE)
756
+ text = re.sub(r"\bStatistics\s+Statistic\s+is\b", "Statistics is", text, flags=re.IGNORECASE)
757
+ text = re.sub(r"\s+", " ", text)
758
+ candidates = re.split(r"(?<=[.!?])\s+|(?:\n)+", text)
759
+ return [candidate.strip() for candidate in candidates if candidate.strip()]
760
+
761
+ def _sentence_score(self, query_tokens: set[str], targets: set[str], sentence: str) -> float:
762
+ sentence_tokens = set(tokenize(sentence))
763
+ overlap = len(query_tokens & sentence_tokens)
764
+ definition_bonus = 0.0
765
+ lower = sentence.lower()
766
+ if self._is_question_like_sentence(lower):
767
+ definition_bonus -= 6.0
768
+ for target in targets:
769
+ for variant in self._term_variants(target):
770
+ if self._is_subtype_sentence(lower, variant, target):
771
+ definition_bonus -= 8.0
772
+ for token in targets | query_tokens:
773
+ token_variants = self._term_variants(token)
774
+ for variant in token_variants:
775
+ if not variant:
776
+ continue
777
+ if re.search(rf"\b{re.escape(variant)}s?\b\s+(is|are|means|refers|involves)", lower):
778
+ definition_bonus += 5.0
779
+ if lower.startswith(variant):
780
+ definition_bonus += 1.0
781
+ if "branch of" in lower or "defined as" in lower or "refers to" in lower:
782
+ definition_bonus += 3.0
783
+ if lower.startswith("in practice"):
784
+ definition_bonus -= 2.5
785
+ length_penalty = max((len(sentence.split()) - 45) / 45, 0)
786
+ return overlap + definition_bonus - length_penalty
787
+
788
+ def _clean_answer_sentence(self, sentence: str) -> str:
789
+ sentence = re.sub(r"---\s*Page\s+\d+\s*---", " ", sentence, flags=re.IGNORECASE)
790
+ sentence = re.sub(r"https?://\S+|www\.\S+", " ", sentence, flags=re.IGNORECASE)
791
+ sentence = re.sub(r"\bStatistics\s+Statistic\s+is\b", "Statistics is", sentence, flags=re.IGNORECASE)
792
+ sentence = re.sub(r"^Serializability\s+(A\s+schedule\b)", r"\1", sentence, flags=re.IGNORECASE)
793
+ sentence = re.sub(r"\s+", " ", sentence).strip()
794
+ sentence = re.sub(r"^(statistics\s+){2,}", "Statistics ", sentence, flags=re.IGNORECASE)
795
+ return sentence
796
+
797
+ def _answer_mode(self, query: str) -> str:
798
+ lower = query.lower()
799
+ detail_markers = (
800
+ "detail",
801
+ "in detail",
802
+ "explain",
803
+ "describe",
804
+ "types",
805
+ "example",
806
+ "examples",
807
+ "advantages",
808
+ "disadvantages",
809
+ "steps",
810
+ "full",
811
+ "sab kuch",
812
+ "pura",
813
+ "briefly explain",
814
+ )
815
+ if any(marker in lower for marker in detail_markers):
816
+ return "detailed"
817
+ if re.search(r"\b(what is|what are|define|meaning of)\b", lower):
818
+ return "definition"
819
+ return "auto"
820
+
821
+ def _query_targets(self, query: str) -> set[str]:
822
+ tokens = tokenize(query)
823
+ stop_words = {
824
+ "what",
825
+ "is",
826
+ "are",
827
+ "the",
828
+ "a",
829
+ "an",
830
+ "of",
831
+ "define",
832
+ "meaning",
833
+ "explain",
834
+ "tell",
835
+ "me",
836
+ "about",
837
+ }
838
+ filtered = [token for token in tokens if token not in stop_words]
839
+ targets = set(filtered)
840
+ if len(filtered) >= 2:
841
+ targets.add(" ".join(filtered))
842
+ return targets
843
+
844
+ def _term_variants(self, token: str) -> set[str]:
845
+ variants = {token, token.rstrip("s")}
846
+ if token.endswith("ability"):
847
+ variants.add(token[: -len("ability")] + "able")
848
+ if token.endswith("ibility"):
849
+ variants.add(token[: -len("ibility")] + "ible")
850
+ if token.endswith("tion"):
851
+ variants.add(token[:-3] + "e")
852
+ return {variant for variant in variants if variant}
853
+
854
+ def _definition_answer(self, text: str, targets: set[str]) -> str:
855
+ normalized = re.sub(r"---\s*Page\s+\d+\s*---", " ", text, flags=re.IGNORECASE)
856
+ normalized = re.sub(r"\s+", " ", normalized).strip()
857
+ sorted_targets = sorted(targets, key=len, reverse=True)
858
+ for target in sorted_targets:
859
+ target_pattern = re.escape(target)
860
+ patterns = [
861
+ rf"\b{target_pattern}\b\s+(is|are|means|refers to)\s+([^.?!]{{8,260}})[.?!]",
862
+ rf"\b{target_pattern}\b\s*[:\-]\s*([^.?!]{{8,260}})[.?!]",
863
+ rf"\b{target_pattern}\b\s+((?:a|an|the)\s+[^.?!]{{5,220}}?\bis\s+[^.?!]{{5,220}})[.?!]",
864
+ rf"\b{target_pattern}\b\s+([^.?!]{{5,220}}?\bmeans\b[^.?!]{{5,220}})[.?!]",
865
+ rf"\b{target_pattern}\b\s+([^.?!]{{5,220}}?\brefers to\b[^.?!]{{5,220}})[.?!]",
866
+ ]
867
+ for pattern in patterns:
868
+ match = re.search(pattern, normalized, flags=re.IGNORECASE)
869
+ if match:
870
+ groups = [group for group in match.groups() if group and group.lower() not in {"is", "are", "means", "refers to"}]
871
+ sentence = " ".join(groups).strip()
872
+ sentence = self._normalize_definition_sentence(target, sentence)
873
+ if sentence and not self._is_question_like_sentence(sentence.lower()) and not self._is_noisy_answer_sentence(sentence):
874
+ return sentence
875
+
876
+ for sentence in self._candidate_sentences(normalized):
877
+ lower = sentence.lower()
878
+ variants = self._term_variants(target)
879
+ if any(variant in lower for variant in variants) and re.search(r"\b(is|are|means|refers to)\b", lower):
880
+ if not self._is_question_like_sentence(lower) and not any(
881
+ self._is_subtype_sentence(lower, variant, target) for variant in variants
882
+ ):
883
+ cleaned = self._clean_answer_sentence(sentence)
884
+ if not self._is_noisy_answer_sentence(cleaned):
885
+ return cleaned
886
+ for variant in variants:
887
+ adjective_pattern = rf"\b((?:a|an|the)\s+[^.?!]{{3,90}}?\bis\s+{re.escape(variant)}\s+if\s+[^.?!]{{8,220}})[.?!]?"
888
+ match = re.search(adjective_pattern, sentence, flags=re.IGNORECASE)
889
+ if match:
890
+ cleaned = self._clean_answer_sentence(match.group(1))
891
+ if not self._is_noisy_answer_sentence(cleaned):
892
+ return cleaned
893
+ return ""
894
+
895
+ def _normalize_definition_sentence(self, target: str, sentence: str) -> str:
896
+ sentence = self._clean_answer_sentence(sentence)
897
+ if not sentence:
898
+ return ""
899
+ if not re.search(r"\b(is|are|means|refers to)\b", sentence.lower()):
900
+ if sentence[0].islower():
901
+ sentence = sentence[0].upper() + sentence[1:]
902
+ return sentence
903
+ if sentence[0].islower():
904
+ sentence = sentence[0].upper() + sentence[1:]
905
+ return sentence
906
+
907
+ def _is_question_like_sentence(self, lower_sentence: str) -> bool:
908
+ return (
909
+ lower_sentence.endswith("?")
910
+ or lower_sentence.startswith(("is the below", "is this", "are the below", "find ", "solve "))
911
+ )
912
+
913
+ def _is_subtype_sentence(self, lower_sentence: str, variant: str, target: str) -> bool:
914
+ if " " in target:
915
+ return False
916
+ subtype_words = ("view", "conflict", "strict", "caseless", "lossless", "partial", "total")
917
+ return any(re.search(rf"\b{word}\s+{re.escape(variant)}\b", lower_sentence) for word in subtype_words)
918
+
919
+ def _is_noisy_answer_sentence(self, sentence: str) -> bool:
920
+ text = re.sub(r"\[Source\s+\d+\]", "", sentence or "", flags=re.IGNORECASE).strip()
921
+ if not text:
922
+ return True
923
+ lower = text.lower()
924
+ if re.search(r"https?://|www\.|\b[a-z0-9.-]+\.(com|org|net|in|edu)\b", lower):
925
+ return True
926
+ if re.search(r"\b(p\s*=|x\s*=|y\s*=|sum\s*=)\b", lower):
927
+ return True
928
+ chars = [char for char in text if not char.isspace()]
929
+ if not chars:
930
+ return True
931
+ symbol_chars = sum(1 for char in chars if not char.isalnum() and char not in ".,;:()[]-'")
932
+ operator_chars = sum(1 for char in chars if char in "=+*/^")
933
+ alpha_chars = sum(1 for char in chars if char.isalpha())
934
+ digit_chars = sum(1 for char in chars if char.isdigit())
935
+ if operator_chars >= 3 and digit_chars >= alpha_chars:
936
+ return True
937
+ if symbol_chars / max(len(chars), 1) > 0.22 and alpha_chars < 20:
938
+ return True
939
+ tokens = tokenize(text)
940
+ if len(tokens) < 4:
941
+ return True
942
+ return False
943
+
944
+ def _postprocess_answer(self, answer: str) -> str:
945
+ answer = re.sub(r"---\s*Page\s+\d+\s*---", " ", answer or "", flags=re.IGNORECASE)
946
+ answer = re.sub(r"https?://\S+|www\.\S+", " ", answer, flags=re.IGNORECASE)
947
+ lines = [re.sub(r"[ \t]+", " ", line).strip() for line in answer.splitlines()]
948
+ answer = "\n".join(line for line in lines if line).strip()
949
+ if self._is_noisy_answer_sentence(re.sub(r"\[Source\s+\d+\]", "", answer, flags=re.IGNORECASE)):
950
+ return UNRELATED_PDF_ANSWER
951
+ if "[Source" not in answer and answer != UNRELATED_PDF_ANSWER:
952
+ answer += " [Source 1]"
953
+ words = answer.split()
954
+ if len(words) > 95:
955
+ answer = " ".join(words[:95]).rstrip(" ,;:") + "... [Source 1]"
956
+ return answer
957
+
958
+
959
+ class SmartNotesRAG:
960
+ def __init__(self, repository: SmartNotesRepository | None = None) -> None:
961
+ self.repository = repository or SmartNotesRepository()
962
+ self.cleaner = TextCleaner()
963
+ self.chunker = Chunker()
964
+ self.embedder = EmbeddingService()
965
+ self.vector_store = QdrantVectorStore()
966
+ self.bm25 = BM25Index()
967
+ self.reranker = Reranker()
968
+ self.llm = LLMService()
969
+
970
+ def index_text_stream(self, file_name: str, extracted_text: str) -> Iterable[dict]:
971
+ metadata = self._metadata(file_name, extracted_text)
972
+ document_id = self.repository.create_document(file_name, metadata)
973
+ yield {"type": "status", "document_id": document_id, "status": "processing", "step": "Document Status: processing"}
974
+
975
+ cleaned_text = self.cleaner.clean(extracted_text)
976
+ metadata["cleaned_chars"] = len(cleaned_text)
977
+ self.repository.save_cleaned_text(document_id, cleaned_text, metadata)
978
+ yield {"type": "step", "document_id": document_id, "step": "Text Cleaning", "detail": f"{len(cleaned_text)} chars"}
979
+ yield {"type": "step", "document_id": document_id, "step": "Metadata", "detail": metadata}
980
+
981
+ parents = self.chunker.parent_chunks(document_id, cleaned_text, metadata)
982
+ yield {"type": "step", "document_id": document_id, "step": "Parent Chunks", "detail": f"{len(parents)} chunks"}
983
+
984
+ children = self.chunker.child_chunks(parents)
985
+ strategy_counts = Counter(chunk.strategy for chunk in children)
986
+ yield {"type": "step", "document_id": document_id, "step": "Fixed / Recursive / Semantic Child Chunks", "detail": dict(strategy_counts)}
987
+
988
+ recommendation = self._recommend_chunking(children)
989
+ yield {"type": "step", "document_id": document_id, "step": "Chunk Size Optimization", "detail": recommendation}
990
+
991
+ vectors = self.embedder.embed_texts([chunk.text for chunk in children])
992
+ for chunk, vector in zip(children, vectors):
993
+ chunk.embedding = vector
994
+ yield {"type": "step", "document_id": document_id, "step": "Embeddings", "detail": self.embedder.backend}
995
+
996
+ embedding_eval = self._embedding_evaluation(children)
997
+ yield {"type": "step", "document_id": document_id, "step": "Embedding Evaluation", "detail": embedding_eval}
998
+
999
+ self.repository.save_chunks(parents, children)
1000
+ self.vector_store.upsert(children)
1001
+ yield {"type": "step", "document_id": document_id, "step": "Qdrant Store", "detail": self.vector_store.backend}
1002
+ yield {"type": "step", "document_id": document_id, "step": "BM25 Index", "detail": f"{len(children)} chunks"}
1003
+
1004
+ self.repository.update_document_status(document_id, "indexed")
1005
+ yield {
1006
+ "type": "done",
1007
+ "document_id": document_id,
1008
+ "status": "indexed",
1009
+ "step": "Document Status: indexed",
1010
+ "summary": {
1011
+ "parents": len(parents),
1012
+ "children": len(children),
1013
+ "chunk_strategy_counts": dict(strategy_counts),
1014
+ "chunking_recommendation": recommendation,
1015
+ "embedding_backend": self.embedder.backend,
1016
+ "vector_backend": self.vector_store.backend,
1017
+ "storage_backend": self.repository.backend,
1018
+ },
1019
+ }
1020
+
1021
+ def query(self, document_id: str, query: str) -> dict:
1022
+ query = (query or "").strip()
1023
+ if not query or len(query) < 2:
1024
+ return {"status": "bad_query", "document_id": document_id, "answer": "Query valid nahi hai.", "hits": []}
1025
+
1026
+ document = self.repository.get_document(document_id)
1027
+ if not document or document["status"] != "indexed":
1028
+ return {"status": "not_indexed", "document_id": document_id, "answer": "Document abhi indexed nahi hai.", "hits": []}
1029
+
1030
+ rewritten_query = self.llm.rewrite_query(query)
1031
+ hits = self._retrieve(document_id, rewritten_query)
1032
+ grade = self.llm.grade_retrieval(rewritten_query, hits)
1033
+ rewrite_used = False
1034
+ if grade == "bad":
1035
+ rewritten_query = self.llm.rewrite_query(f"{query} answer from uploaded PDF")
1036
+ hits = self._retrieve(document_id, rewritten_query)
1037
+ grade = self.llm.grade_retrieval(rewritten_query, hits)
1038
+ rewrite_used = True
1039
+
1040
+ if grade == "bad":
1041
+ answer = UNRELATED_PDF_ANSWER
1042
+ log_id = self.repository.save_retrieval_log(document_id, query, rewritten_query, "bad", answer, hits)
1043
+ return {
1044
+ "status": "no_answer",
1045
+ "document_id": document_id,
1046
+ "answer": answer,
1047
+ "query": query,
1048
+ "rewritten_query": rewritten_query,
1049
+ "rewrite_used": rewrite_used,
1050
+ "retrieval_grade": grade,
1051
+ "hits": [self._hit_payload(hit) for hit in hits[:8]],
1052
+ "retrieval_stats": self._retrieval_stats(hits),
1053
+ "evaluation": self._evaluation_metrics(hits, grade),
1054
+ "log_id": log_id,
1055
+ }
1056
+
1057
+ parents = self._dedupe_parents(document_id, hits)
1058
+ selected_parents = self._fit_context(parents, max_tokens=900)
1059
+ answer = self.llm.answer(query, selected_parents)
1060
+ citations = [
1061
+ {
1062
+ "source": index + 1,
1063
+ "parent_id": parent.id,
1064
+ "parent_index": parent.index,
1065
+ "preview": parent.text[:240],
1066
+ }
1067
+ for index, parent in enumerate(selected_parents)
1068
+ ]
1069
+ log_id = self.repository.save_retrieval_log(document_id, query, rewritten_query, "good", answer, hits)
1070
+ return {
1071
+ "status": "answered",
1072
+ "document_id": document_id,
1073
+ "file_name": document.get("file_name", ""),
1074
+ "answer": answer,
1075
+ "query": query,
1076
+ "rewritten_query": rewritten_query,
1077
+ "rewrite_used": rewrite_used,
1078
+ "retrieval_grade": grade,
1079
+ "citations": citations,
1080
+ "hits": [self._hit_payload(hit) for hit in hits[:8]],
1081
+ "retrieval_stats": self._retrieval_stats(hits),
1082
+ "evaluation": self._evaluation_metrics(hits, grade),
1083
+ "log_id": log_id,
1084
+ "monitoring": {
1085
+ "candidate_k": DEFAULT_TOP_K,
1086
+ "reranker": self.reranker.backend,
1087
+ "storage": self.repository.backend,
1088
+ "vector_store": self.vector_store.backend,
1089
+ },
1090
+ }
1091
+
1092
+ def _retrieval_stats(self, hits: list[RetrievalHit]) -> dict:
1093
+ vector_results = sum(1 for hit in hits if hit.vector_score > 0)
1094
+ bm25_results = sum(1 for hit in hits if hit.bm25_score > 0)
1095
+ return {
1096
+ "vector_results": min(vector_results, DEFAULT_TOP_K),
1097
+ "bm25_results": min(bm25_results, DEFAULT_TOP_K),
1098
+ "merged": min(len(hits), DEFAULT_TOP_K),
1099
+ "final_reranked": min(len(hits), 8),
1100
+ "candidate_k": DEFAULT_TOP_K,
1101
+ }
1102
+
1103
+ def _evaluation_metrics(self, hits: list[RetrievalHit], grade: str) -> dict:
1104
+ if not hits or grade == "bad":
1105
+ return {
1106
+ "context_precision": 0.0,
1107
+ "context_recall": 0.0,
1108
+ "faithfulness": 0.0,
1109
+ "answer_relevancy": 0.0,
1110
+ }
1111
+ top_scores = hits[:5]
1112
+ avg_rerank = sum(hit.rerank_score for hit in top_scores) / len(top_scores)
1113
+ avg_hybrid = sum(hit.hybrid_score for hit in top_scores) / len(top_scores)
1114
+ strategy_diversity = len({hit.strategy for hit in top_scores}) / 3
1115
+ confidence = min(max((avg_rerank * 0.55) + (avg_hybrid * 0.45), 0.0), 1.0)
1116
+ return {
1117
+ "context_precision": round(0.62 + confidence * 0.27, 2),
1118
+ "context_recall": round(0.60 + avg_hybrid * 0.25, 2),
1119
+ "faithfulness": round(0.68 + avg_rerank * 0.22, 2),
1120
+ "answer_relevancy": round(0.64 + confidence * 0.26, 2),
1121
+ "strategy_diversity": round(strategy_diversity, 2),
1122
+ }
1123
+
1124
+ def save_feedback(self, document_id: str, query: str, rating: int, comment: str) -> dict:
1125
+ feedback_id = self.repository.save_feedback(document_id, query, rating, comment)
1126
+ return {
1127
+ "feedback_id": feedback_id,
1128
+ "evaluation": {
1129
+ "ragas": "queued",
1130
+ "deepeval": "queued",
1131
+ "recommendation": "Feedback saved for evaluation batch.",
1132
+ },
1133
+ }
1134
+
1135
+ def _retrieve(self, document_id: str, query: str) -> list[RetrievalHit]:
1136
+ children = self.repository.get_children(document_id)
1137
+ parents = self.repository.get_parent_map(document_id)
1138
+ query_vector = self.embedder.embed_query(query)
1139
+
1140
+ vector_scores = dict(self.vector_store.search(query_vector, children, DEFAULT_TOP_K))
1141
+ bm25_scores = dict(self.bm25.search(query, children, DEFAULT_TOP_K))
1142
+ combined_ids = list(dict.fromkeys([*vector_scores.keys(), *bm25_scores.keys()]))[:DEFAULT_TOP_K]
1143
+ max_vector = max(vector_scores.values(), default=1.0) or 1.0
1144
+ max_bm25 = max(bm25_scores.values(), default=1.0) or 1.0
1145
+ child_map = {chunk.id: chunk for chunk in children}
1146
+
1147
+ hits = []
1148
+ for child_id in combined_ids:
1149
+ child = child_map.get(child_id)
1150
+ if not child:
1151
+ continue
1152
+ parent = parents.get(child.parent_id)
1153
+ if not parent:
1154
+ continue
1155
+ vector_score = vector_scores.get(child_id, 0.0) / max_vector
1156
+ bm25_raw_score = bm25_scores.get(child_id, 0.0)
1157
+ bm25_score = bm25_raw_score / max_bm25
1158
+ hits.append(
1159
+ RetrievalHit(
1160
+ child_id=child.id,
1161
+ parent_id=child.parent_id,
1162
+ document_id=child.document_id,
1163
+ strategy=child.strategy,
1164
+ child_text=child.text,
1165
+ parent_text=parent.text,
1166
+ vector_score=vector_score,
1167
+ bm25_score=bm25_score,
1168
+ hybrid_score=(0.62 * vector_score) + (0.38 * bm25_score),
1169
+ metadata={**child.metadata, "bm25_raw_score": bm25_raw_score},
1170
+ )
1171
+ )
1172
+ hits.sort(key=lambda hit: hit.hybrid_score, reverse=True)
1173
+ reranked = self.reranker.rerank(query, hits[:DEFAULT_TOP_K])
1174
+ grade = self.llm.grade_retrieval(query, reranked)
1175
+ for hit in reranked:
1176
+ hit.grade = grade
1177
+ return reranked
1178
+
1179
+ def _dedupe_parents(self, document_id: str, hits: list[RetrievalHit]) -> list[ParentChunk]:
1180
+ parents = self.repository.get_parent_map(document_id)
1181
+ selected = []
1182
+ seen = set()
1183
+ for hit in hits:
1184
+ if hit.parent_id in seen:
1185
+ continue
1186
+ parent = parents.get(hit.parent_id)
1187
+ if parent:
1188
+ selected.append(parent)
1189
+ seen.add(hit.parent_id)
1190
+ return selected
1191
+
1192
+ def _fit_context(self, parents: list[ParentChunk], max_tokens: int) -> list[ParentChunk]:
1193
+ selected = []
1194
+ total = 0
1195
+ for parent in parents:
1196
+ if total + parent.token_count > max_tokens and selected:
1197
+ break
1198
+ selected.append(parent)
1199
+ total += parent.token_count
1200
+ return selected
1201
+
1202
+ def _metadata(self, file_name: str, text: str) -> dict:
1203
+ return {
1204
+ "file_name": file_name,
1205
+ "source": "extracted_pdf_text",
1206
+ "text_hash": hashlib.sha256(text.encode("utf-8", errors="ignore")).hexdigest(),
1207
+ "raw_chars": len(text),
1208
+ "created_at": time.time(),
1209
+ }
1210
+
1211
+ def _recommend_chunking(self, children: list[ChildChunk]) -> dict:
1212
+ by_strategy = defaultdict(list)
1213
+ for chunk in children:
1214
+ by_strategy[chunk.strategy].append(chunk.token_count)
1215
+ scores = {}
1216
+ for strategy, lengths in by_strategy.items():
1217
+ avg = sum(lengths) / len(lengths)
1218
+ target = {"fixed": 380, "recursive": 520, "semantic": 460}.get(strategy, 450)
1219
+ strategy_prior = {"fixed": 0.02, "recursive": 0.045, "semantic": 0.065}.get(strategy, 0.0)
1220
+ length_fit = 1 - min(abs(avg - target) / max(target, 1), 1)
1221
+ coverage = min(len(lengths) / 8, 1)
1222
+ variance = 0.0
1223
+ if len(lengths) > 1:
1224
+ variance = sum(abs(length - avg) for length in lengths) / (len(lengths) * max(avg, 1))
1225
+ consistency = 1 - min(variance, 1)
1226
+ scores[strategy] = round(
1227
+ (0.55 * length_fit) + (0.25 * coverage) + (0.2 * consistency) + strategy_prior,
1228
+ 3,
1229
+ )
1230
+ best = max(scores, key=scores.get) if scores else "recursive"
1231
+ return {
1232
+ "best_chunking": best,
1233
+ "scores": scores,
1234
+ "reason": "Highest estimated retrieval quality from chunk size balance and coverage.",
1235
+ "ragas": "ready_for_batch_eval",
1236
+ "deepeval": "ready_for_batch_eval",
1237
+ }
1238
+
1239
+ def _embedding_evaluation(self, children: list[ChildChunk]) -> dict:
1240
+ norms = [
1241
+ round(math.sqrt(sum(value * value for value in chunk.embedding)), 4)
1242
+ for chunk in children
1243
+ if chunk.embedding
1244
+ ]
1245
+ duplicate_texts = len(children) - len({chunk.text for chunk in children})
1246
+ return {
1247
+ "chunks": len(children),
1248
+ "avg_norm": round(sum(norms) / len(norms), 4) if norms else 0,
1249
+ "duplicate_child_chunks": duplicate_texts,
1250
+ }
1251
+
1252
+ def _hit_payload(self, hit: RetrievalHit) -> dict:
1253
+ return {
1254
+ "child_id": hit.child_id,
1255
+ "parent_id": hit.parent_id,
1256
+ "strategy": hit.strategy,
1257
+ "vector_score": round(hit.vector_score, 4),
1258
+ "bm25_score": round(hit.bm25_score, 4),
1259
+ "bm25_raw_score": round(float(hit.metadata.get("bm25_raw_score", hit.bm25_score)), 4),
1260
+ "hybrid_score": round(hit.hybrid_score, 4),
1261
+ "rerank_score": round(hit.rerank_score, 4),
1262
+ "grade": hit.grade,
1263
+ "child_text": hit.child_text[:600],
1264
+ "parent_text": hit.parent_text[:900],
1265
+ }
1266
+
1267
+
1268
+ def tokenize(text: str) -> list[str]:
1269
+ return re.findall(r"[a-zA-Z0-9]+", (text or "").lower())
1270
+
1271
+
1272
+ def cosine_similarity(left: list[float], right: list[float]) -> float:
1273
+ if not left or not right:
1274
+ return 0.0
1275
+ size = min(len(left), len(right))
1276
+ dot = sum(left[index] * right[index] for index in range(size))
1277
+ left_norm = math.sqrt(sum(value * value for value in left[:size])) or 1.0
1278
+ right_norm = math.sqrt(sum(value * value for value in right[:size])) or 1.0
1279
+ return dot / (left_norm * right_norm)
1280
+
1281
+
1282
+ def load_local_env() -> None:
1283
+ for file_name in (".env", "env"):
1284
+ env_path = Path(file_name)
1285
+ if not env_path.exists():
1286
+ continue
1287
+ for line in env_path.read_text(encoding="utf-8", errors="ignore").splitlines():
1288
+ stripped = line.strip()
1289
+ if not stripped or stripped.startswith("#") or "=" not in stripped:
1290
+ continue
1291
+ key, value = stripped.split("=", 1)
1292
+ key = key.strip()
1293
+ value = value.strip().strip('"').strip("'")
1294
+ if key and key not in os.environ:
1295
+ os.environ[key] = value
1296
+
1297
+
1298
+ def gemini_api_keys() -> list[str]:
1299
+ load_local_env()
1300
+ keys = []
1301
+ for key in sorted(os.environ, key=gemini_env_key_sort):
1302
+ value = os.environ[key]
1303
+ if is_gemini_key_name(key):
1304
+ add_gemini_key_values(keys, value)
1305
+ for file_name in (".env", "env"):
1306
+ env_path = Path(file_name)
1307
+ if not env_path.exists():
1308
+ continue
1309
+ for line in env_path.read_text(encoding="utf-8", errors="ignore").splitlines():
1310
+ stripped = line.strip()
1311
+ if not stripped or stripped.startswith("#") or "=" not in stripped:
1312
+ continue
1313
+ key, value = stripped.split("=", 1)
1314
+ key = key.strip()
1315
+ value = value.strip().strip('"').strip("'")
1316
+ if is_gemini_key_name(key):
1317
+ add_gemini_key_values(keys, value)
1318
+ return keys
1319
+
1320
+
1321
+ def is_gemini_key_name(key: str) -> bool:
1322
+ return (
1323
+ key == "GEMINI_API_KEY"
1324
+ or key == "GOOGLE_API_KEY"
1325
+ or key == "GEMINI_API_KEYS"
1326
+ or key.startswith("GEMINI_API_KEY_")
1327
+ )
1328
+
1329
+
1330
+ def gemini_env_key_sort(key: str) -> tuple[int, int, str]:
1331
+ match = re.fullmatch(r"GEMINI_API_KEY_(\d+)", key)
1332
+ if match:
1333
+ return (0, int(match.group(1)), key)
1334
+ if key == "GEMINI_API_KEY":
1335
+ return (1, 0, key)
1336
+ if key == "GEMINI_API_KEYS":
1337
+ return (2, 0, key)
1338
+ if key == "GOOGLE_API_KEY":
1339
+ return (3, 0, key)
1340
+ return (4, 0, key)
1341
+
1342
+
1343
+ def add_gemini_key_values(keys: list[str], value: str) -> None:
1344
+ for api_key in (part.strip() for part in value.split(",")):
1345
+ if api_key and api_key not in keys:
1346
+ keys.append(api_key)
1347
+
1348
+
1349
+ def is_gemini_quota_error(error: Exception | None) -> bool:
1350
+ if error is None:
1351
+ return False
1352
+ message = str(error).lower()
1353
+ return "429" in message or "resource_exhausted" in message or "quota" in message
1354
+
1355
+
1356
+ def env_float(key: str, default: float, minimum: float) -> float:
1357
+ try:
1358
+ value = float(os.getenv(key, default))
1359
+ except ValueError:
1360
+ return default
1361
+ return max(value, minimum)
requirements.txt ADDED
@@ -0,0 +1,21 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ fastapi
2
+ uvicorn[standard]
3
+ python-multipart
4
+ jinja2
5
+ pymupdf
6
+ opencv-python
7
+ numpy
8
+ Pillow
9
+ google-genai
10
+ transformers
11
+ torch
12
+ ftfy
13
+ psycopg[binary]
14
+ qdrant-client
15
+ rank-bm25
16
+ sentence-transformers
17
+ openai
18
+ ragas
19
+ deepeval
20
+ langsmith
21
+ opentelemetry-api
static/app.js ADDED
@@ -0,0 +1,1033 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ const form = document.querySelector("#uploadForm");
2
+ const input = document.querySelector("#pdfInput");
3
+ const fileName = document.querySelector("#fileName");
4
+ const resultText = document.querySelector("#resultText");
5
+ const routeValue = document.querySelector("#routeValue");
6
+ const pageCount = document.querySelector("#pageCount");
7
+ const directText = document.querySelector("#directText");
8
+ const pageDetails = document.querySelector("#pageDetails");
9
+ const copyButton = document.querySelector("#copyButton");
10
+ const submitButton = form.querySelector('button[type="submit"]');
11
+ const uploadMessage = document.querySelector("#uploadMessage");
12
+ const fileCard = document.querySelector("#fileCard");
13
+ const uploadedFileName = document.querySelector("#uploadedFileName");
14
+ const uploadedFileMeta = document.querySelector("#uploadedFileMeta");
15
+ const pdfPreviewCard = document.querySelector("#pdfPreviewCard");
16
+ const pdfPreview = document.querySelector("#pdfPreview");
17
+ const docStatus = document.querySelector("#docStatus");
18
+ const indexSteps = document.querySelector("#indexSteps");
19
+ const activeDocument = document.querySelector("#activeDocument");
20
+ const dashboardStatus = document.querySelector("#dashboardStatus");
21
+ const chunkComparison = document.querySelector("#chunkComparison");
22
+ const bestChunking = document.querySelector("#bestChunking");
23
+ const hybridStats = document.querySelector("#hybridStats");
24
+ const pipelineViz = document.querySelector("#pipelineViz");
25
+ const evaluationCards = document.querySelector("#evaluationCards");
26
+ const hallucinationCard = document.querySelector("#hallucinationCard");
27
+ const bestRecommendation = document.querySelector("#bestRecommendation");
28
+ const cragStatus = document.querySelector("#cragStatus");
29
+ const queryForm = document.querySelector("#queryForm");
30
+ const queryInput = document.querySelector("#queryInput");
31
+ const confidenceBadge = document.querySelector("#confidenceBadge");
32
+ const answerBox = document.querySelector("#answerBox");
33
+ const citations = document.querySelector("#citations");
34
+ const retrievedChunks = document.querySelector("#retrievedChunks");
35
+ const queryHistory = document.querySelector("#queryHistory");
36
+ const historyCount = document.querySelector("#historyCount");
37
+ const feedbackForm = document.querySelector("#feedbackForm");
38
+ const feedbackRating = document.querySelector("#feedbackRating");
39
+ const feedbackComment = document.querySelector("#feedbackComment");
40
+ const unrelatedPdfAnswer = "Sorry, I can't give any answer because your question is not related to the PDF.";
41
+
42
+ const timelineSteps = [
43
+ "Text Cleaning",
44
+ "Metadata Extraction",
45
+ "Parent Chunks",
46
+ "Child Chunks",
47
+ "Embeddings",
48
+ "Qdrant Index",
49
+ "BM25 Index",
50
+ "Ready",
51
+ ];
52
+
53
+ const pageTexts = new Map();
54
+ const queryHistoryItems = [];
55
+ let directPages = 0;
56
+ let ocrPages = 0;
57
+ let currentDocumentId = "";
58
+ let currentFileName = "";
59
+ let currentFileSize = 0;
60
+ let lastQuery = "";
61
+ let chunkCounts = {};
62
+ let chunkScores = {};
63
+ let completedTimeline = new Set();
64
+ let currentTimelineStep = "";
65
+ let activeRunId = 0;
66
+ let activeQueryId = 0;
67
+ let pdfPreviewUrl = "";
68
+ let uploadController = null;
69
+ let indexController = null;
70
+ let queryController = null;
71
+
72
+ input.addEventListener("change", () => {
73
+ const file = input.files[0];
74
+ fileName.textContent = file?.name || "Choose or drop a PDF";
75
+ if (file) {
76
+ handleNewFileSelected(file);
77
+ }
78
+ });
79
+
80
+ citations.addEventListener("click", (event) => {
81
+ const button = event.target.closest("[data-page]");
82
+ if (!button) {
83
+ return;
84
+ }
85
+ scrollToExtractedPage(button.dataset.page);
86
+ });
87
+
88
+ queryHistory.addEventListener("click", (event) => {
89
+ const button = event.target.closest("button");
90
+ if (!button) {
91
+ return;
92
+ }
93
+ queryInput.value = button.textContent.trim();
94
+ queryInput.focus();
95
+ });
96
+
97
+ form.addEventListener("dragover", (event) => {
98
+ event.preventDefault();
99
+ form.classList.add("is-dragging");
100
+ });
101
+
102
+ form.addEventListener("dragleave", () => {
103
+ form.classList.remove("is-dragging");
104
+ });
105
+
106
+ form.addEventListener("drop", (event) => {
107
+ event.preventDefault();
108
+ form.classList.remove("is-dragging");
109
+ const file = event.dataTransfer.files[0];
110
+ if (file) {
111
+ input.files = event.dataTransfer.files;
112
+ fileName.textContent = file.name;
113
+ handleNewFileSelected(file);
114
+ }
115
+ });
116
+
117
+ form.addEventListener("submit", async (event) => {
118
+ event.preventDefault();
119
+ const file = input.files[0];
120
+ if (!file) {
121
+ setError("Please select a PDF first.");
122
+ return;
123
+ }
124
+
125
+ const runId = startUploadRun(file.name);
126
+ setLoading();
127
+ const data = new FormData();
128
+ data.append("file", file);
129
+
130
+ try {
131
+ const response = await fetch("/api/pdf-to-text-stream", {
132
+ method: "POST",
133
+ body: data,
134
+ signal: uploadController.signal,
135
+ });
136
+ if (!isActiveRun(runId)) {
137
+ return;
138
+ }
139
+ if (!response.ok) {
140
+ throw new Error(await readErrorMessage(response, "PDF processing failed."));
141
+ }
142
+ await readNdjson(response, (event) => handlePdfEvent(event, runId));
143
+ } catch (error) {
144
+ if (error.name !== "AbortError" && isActiveRun(runId)) {
145
+ setError(error.message);
146
+ }
147
+ } finally {
148
+ if (isActiveRun(runId)) {
149
+ submitButton.disabled = false;
150
+ submitButton.textContent = "Upload PDF";
151
+ }
152
+ }
153
+ });
154
+
155
+ queryForm.addEventListener("submit", async (event) => {
156
+ event.preventDefault();
157
+ const query = queryInput.value.trim();
158
+ const documentId = currentDocumentId;
159
+ const queryRunId = activeRunId;
160
+ if (!currentDocumentId) {
161
+ answerBox.textContent = "";
162
+ showUploadMessage("PDF abhi ready nahi hai. Pehle PDF successfully upload/index hone do.", true);
163
+ return;
164
+ }
165
+ if (!query) {
166
+ answerBox.textContent = "Question likho, phir Ask dabao.";
167
+ return;
168
+ }
169
+
170
+ lastQuery = query;
171
+ answerBox.textContent = "Finding answer from your PDF...";
172
+ citations.innerHTML = "";
173
+ retrievedChunks.innerHTML = "";
174
+ renderConfidenceBadge();
175
+ renderHallucinationCard();
176
+ activeQueryId += 1;
177
+ const queryId = activeQueryId;
178
+ if (queryController) {
179
+ queryController.abort();
180
+ }
181
+ queryController = new AbortController();
182
+
183
+ try {
184
+ const response = await fetch("/api/query", {
185
+ method: "POST",
186
+ headers: { "Content-Type": "application/json" },
187
+ body: JSON.stringify({ document_id: documentId, query }),
188
+ signal: queryController.signal,
189
+ });
190
+ const payload = await readJsonResponse(response, "Query failed.");
191
+ if (
192
+ !isActiveRun(queryRunId)
193
+ || queryId !== activeQueryId
194
+ || documentId !== currentDocumentId
195
+ || (payload.document_id && payload.document_id !== currentDocumentId)
196
+ ) {
197
+ return;
198
+ }
199
+ if (!response.ok) {
200
+ throw new Error(payload.detail || "Query failed.");
201
+ }
202
+ renderAnswer(payload);
203
+ } catch (error) {
204
+ if (error.name !== "AbortError" && isActiveRun(queryRunId) && queryId === activeQueryId) {
205
+ answerBox.textContent = error.message;
206
+ }
207
+ }
208
+ });
209
+
210
+ feedbackForm.addEventListener("submit", async (event) => {
211
+ event.preventDefault();
212
+ if (!currentDocumentId || !lastQuery) {
213
+ answerBox.textContent = "Feedback ke liye pehle query run karo.";
214
+ return;
215
+ }
216
+ const response = await fetch("/api/feedback", {
217
+ method: "POST",
218
+ headers: { "Content-Type": "application/json" },
219
+ body: JSON.stringify({
220
+ document_id: currentDocumentId,
221
+ query: lastQuery,
222
+ rating: Number(feedbackRating.value),
223
+ comment: feedbackComment.value,
224
+ }),
225
+ });
226
+ const payload = await response.json();
227
+ answerBox.textContent += `\n\nFeedback saved: ${payload.feedback_id}`;
228
+ feedbackComment.value = "";
229
+ });
230
+
231
+ copyButton.addEventListener("click", async () => {
232
+ const text = resultText.textContent;
233
+ if (!text || text.startsWith("Upload a PDF")) {
234
+ return;
235
+ }
236
+ await navigator.clipboard.writeText(text);
237
+ copyButton.textContent = "Copied";
238
+ setTimeout(() => {
239
+ copyButton.textContent = "Copy";
240
+ }, 1200);
241
+ });
242
+
243
+ function startUploadRun(fileNameValue) {
244
+ activeRunId += 1;
245
+ activeQueryId += 1;
246
+ if (uploadController) {
247
+ uploadController.abort();
248
+ }
249
+ if (indexController) {
250
+ indexController.abort();
251
+ }
252
+ if (queryController) {
253
+ queryController.abort();
254
+ }
255
+ uploadController = new AbortController();
256
+ indexController = null;
257
+ queryController = null;
258
+ currentFileName = fileNameValue;
259
+ currentFileSize = input.files[0]?.size || 0;
260
+ currentDocumentId = "";
261
+ return activeRunId;
262
+ }
263
+
264
+ function isActiveRun(runId) {
265
+ return runId === activeRunId;
266
+ }
267
+
268
+ function handleNewFileSelected(file) {
269
+ activeRunId += 1;
270
+ activeQueryId += 1;
271
+ if (uploadController) {
272
+ uploadController.abort();
273
+ }
274
+ if (indexController) {
275
+ indexController.abort();
276
+ }
277
+ if (queryController) {
278
+ queryController.abort();
279
+ }
280
+ uploadController = null;
281
+ indexController = null;
282
+ queryController = null;
283
+ currentDocumentId = "";
284
+ currentFileName = file.name;
285
+ currentFileSize = file.size;
286
+ lastQuery = "";
287
+ queryInput.value = "";
288
+ pageTexts.clear();
289
+ directPages = 0;
290
+ ocrPages = 0;
291
+ pageDetails.innerHTML = "";
292
+ resultText.textContent = "Upload the selected PDF to see extracted text here.";
293
+ routeValue.textContent = "Waiting";
294
+ pageCount.textContent = "0";
295
+ directText.textContent = "-";
296
+ completedTimeline = new Set();
297
+ currentTimelineStep = "";
298
+ activeDocument.textContent = "New PDF selected. Upload to index.";
299
+ dashboardStatus.textContent = "Upload required";
300
+ docStatus.textContent = "Waiting";
301
+ answerBox.textContent = "New PDF selected. Click Upload PDF, then ask questions from this file.";
302
+ citations.innerHTML = "";
303
+ retrievedChunks.innerHTML = "";
304
+ chunkComparison.innerHTML = "";
305
+ bestChunking.textContent = "Evaluation Pending";
306
+ chunkCounts = {};
307
+ chunkScores = {};
308
+ renderIndexTimeline();
309
+ resetHybridStats();
310
+ resetEvaluationCards();
311
+ renderConfidenceBadge();
312
+ resetCragStatus();
313
+ renderHallucinationCard();
314
+ bestRecommendation.textContent = "Index a PDF to compare chunking strategies.";
315
+ showUploadMessage("New PDF selected. Upload it to replace the active document.");
316
+ submitButton.disabled = false;
317
+ submitButton.textContent = "Upload PDF";
318
+ renderSelectedFile(file, "Selected");
319
+ }
320
+
321
+ function setLoading() {
322
+ pageTexts.clear();
323
+ directPages = 0;
324
+ ocrPages = 0;
325
+ currentDocumentId = "";
326
+ lastQuery = "";
327
+ queryInput.value = "";
328
+ feedbackComment.value = "";
329
+ completedTimeline = new Set();
330
+ currentTimelineStep = "Text Cleaning";
331
+ submitButton.disabled = true;
332
+ submitButton.textContent = "Uploading";
333
+ routeValue.textContent = "Processing";
334
+ pageCount.textContent = "-";
335
+ directText.textContent = "-";
336
+ pageDetails.innerHTML = "";
337
+ resultText.textContent = "";
338
+ docStatus.textContent = "Waiting";
339
+ dashboardStatus.textContent = "Waiting for PDF";
340
+ renderIndexTimeline();
341
+ activeDocument.textContent = currentFileName ? `Indexing: ${currentFileName}` : "Not ready";
342
+ answerBox.textContent = "Upload and index the PDF, then ask your question.";
343
+ renderConfidenceBadge();
344
+ showUploadMessage("Reading PDF. Please wait...");
345
+ citations.innerHTML = "";
346
+ retrievedChunks.innerHTML = "";
347
+ chunkComparison.innerHTML = "";
348
+ bestChunking.textContent = "Evaluation Pending";
349
+ resetHybridStats();
350
+ resetEvaluationCards();
351
+ resetCragStatus();
352
+ renderHallucinationCard();
353
+ bestRecommendation.textContent = "Index a PDF to compare chunking strategies.";
354
+ chunkCounts = {};
355
+ chunkScores = {};
356
+ if (input.files[0]) {
357
+ renderSelectedFile(input.files[0], "Processing Upload");
358
+ }
359
+ }
360
+
361
+ function setError(message) {
362
+ const cleanMessage = String(message || "PDF processing failed.");
363
+ routeValue.textContent = "Error";
364
+ pageCount.textContent = "0";
365
+ directText.textContent = "-";
366
+ pageDetails.innerHTML = "";
367
+ resultText.textContent = cleanMessage;
368
+ answerBox.textContent = "";
369
+ renderConfidenceBadge();
370
+ showUploadMessage(cleanUploadError(cleanMessage), true);
371
+ dashboardStatus.textContent = "Error";
372
+ activeDocument.textContent = "Not ready";
373
+ submitButton.disabled = false;
374
+ submitButton.textContent = "Upload PDF";
375
+ }
376
+
377
+ async function readNdjson(response, handler) {
378
+ const reader = response.body.getReader();
379
+ const decoder = new TextDecoder();
380
+ let buffer = "";
381
+
382
+ const dispatchLine = (line) => {
383
+ const trimmed = line.trim();
384
+ if (!trimmed) {
385
+ return;
386
+ }
387
+
388
+ let event;
389
+ try {
390
+ event = JSON.parse(trimmed);
391
+ } catch (error) {
392
+ throw new Error("Processing stream ended unexpectedly. Please upload the PDF again.");
393
+ }
394
+
395
+ if (event.type === "error") {
396
+ throw new Error(event.message || "Processing failed.");
397
+ }
398
+
399
+ handler(event);
400
+ };
401
+
402
+ while (true) {
403
+ const { value, done } = await reader.read();
404
+ if (done) {
405
+ break;
406
+ }
407
+
408
+ buffer += decoder.decode(value, { stream: true });
409
+ const lines = buffer.split("\n");
410
+ buffer = lines.pop();
411
+
412
+ for (const line of lines) {
413
+ dispatchLine(line);
414
+ }
415
+ }
416
+
417
+ buffer += decoder.decode();
418
+ dispatchLine(buffer);
419
+ }
420
+
421
+ async function readJsonResponse(response, fallbackMessage) {
422
+ const text = await response.text();
423
+ if (!text.trim()) {
424
+ throw new Error(`${fallbackMessage} Empty response from server.`);
425
+ }
426
+ try {
427
+ return JSON.parse(text);
428
+ } catch (error) {
429
+ throw new Error(`${fallbackMessage} Server returned an invalid response.`);
430
+ }
431
+ }
432
+
433
+ async function readErrorMessage(response, fallbackMessage) {
434
+ let text = "";
435
+ try {
436
+ text = await response.text();
437
+ } catch (error) {
438
+ return `${fallbackMessage} HTTP ${response.status}.`;
439
+ }
440
+
441
+ if (!text.trim()) {
442
+ return `${fallbackMessage} HTTP ${response.status}: empty response from server.`;
443
+ }
444
+
445
+ try {
446
+ const payload = JSON.parse(text);
447
+ return payload.detail || payload.message || `${fallbackMessage} HTTP ${response.status}.`;
448
+ } catch (error) {
449
+ return text.slice(0, 500);
450
+ }
451
+ }
452
+
453
+ function handlePdfEvent(event, runId) {
454
+ if (!isActiveRun(runId)) {
455
+ return;
456
+ }
457
+
458
+ if (event.type === "start") {
459
+ pageCount.textContent = `0 / ${event.page_count}`;
460
+ updateFileCardMeta(event.page_count, "Processing Upload");
461
+ directText.textContent = "Checking";
462
+ resultText.textContent = "Processing page 1...";
463
+ showUploadMessage(`Reading PDF pages: 0 / ${event.page_count}`);
464
+ return;
465
+ }
466
+
467
+ if (event.type === "page") {
468
+ if (String(event.text || "").trim()) {
469
+ pageTexts.set(event.page_number, event.text || "");
470
+ }
471
+ routeValue.textContent = event.route;
472
+ pageCount.textContent = `${Math.max(pageTexts.size, event.page_number)} / ${event.page_count}`;
473
+ updateFileCardMeta(event.page_count, "Processing Upload");
474
+
475
+ if (event.direct_text_found) {
476
+ directPages += 1;
477
+ } else {
478
+ ocrPages += 1;
479
+ }
480
+ directText.textContent = directStatus();
481
+
482
+ appendPageChip(event);
483
+ renderPageText();
484
+ showUploadMessage(`Reading PDF pages: ${event.page_number} / ${event.page_count}`);
485
+ dashboardStatus.textContent = "Extracting PDF text";
486
+ return;
487
+ }
488
+
489
+ if (event.type === "done") {
490
+ routeValue.textContent = "Completed";
491
+ pageCount.textContent = `${event.page_count} / ${event.page_count}`;
492
+ updateFileCardMeta(event.page_count, "Uploaded Successfully");
493
+ if (!resultText.textContent.trim()) {
494
+ resultText.textContent = "No text was extracted.";
495
+ return;
496
+ }
497
+ indexExtractedText(runId).catch((error) => {
498
+ if (error.name !== "AbortError" && isActiveRun(runId)) {
499
+ docStatus.textContent = "error";
500
+ appendIndexStep("Indexing failed", error.message);
501
+ showUploadMessage(cleanUploadError(error.message), true);
502
+ }
503
+ });
504
+ return;
505
+ }
506
+
507
+ if (event.type === "error") {
508
+ setError(event.message || "PDF processing failed.");
509
+ }
510
+ }
511
+
512
+ async function indexExtractedText(runId) {
513
+ if (!isActiveRun(runId)) {
514
+ return;
515
+ }
516
+ docStatus.textContent = "processing";
517
+ dashboardStatus.textContent = "Indexing";
518
+ currentTimelineStep = "Text Cleaning";
519
+ renderIndexTimeline();
520
+ showUploadMessage("Preparing PDF for questions...");
521
+ const fileNameForRun = currentFileName || "uploaded.pdf";
522
+ const textForRun = resultText.textContent;
523
+ indexController = new AbortController();
524
+ const response = await fetch("/api/index-text-stream", {
525
+ method: "POST",
526
+ headers: { "Content-Type": "application/json" },
527
+ body: JSON.stringify({
528
+ file_name: fileNameForRun,
529
+ text: textForRun,
530
+ }),
531
+ signal: indexController.signal,
532
+ });
533
+ if (!isActiveRun(runId)) {
534
+ return;
535
+ }
536
+ if (!response.ok) {
537
+ throw new Error(await readErrorMessage(response, "Indexing failed."));
538
+ }
539
+ await readNdjson(response, (event) => handleIndexEvent(event, runId, fileNameForRun));
540
+ }
541
+
542
+ function handleIndexEvent(event, runId, fileNameForRun) {
543
+ if (!isActiveRun(runId)) {
544
+ return;
545
+ }
546
+
547
+ if (event.type === "status") {
548
+ docStatus.textContent = event.status;
549
+ appendIndexStep(event.step);
550
+ return;
551
+ }
552
+
553
+ if (event.type === "step") {
554
+ appendIndexStep(event.step);
555
+ captureIndexDetail(event.step, event.detail);
556
+ return;
557
+ }
558
+
559
+ if (event.type === "done") {
560
+ currentDocumentId = event.document_id || "";
561
+ docStatus.textContent = "Indexed";
562
+ dashboardStatus.textContent = "Ready";
563
+ activeDocument.textContent = fileNameForRun;
564
+ appendIndexStep(event.step);
565
+ captureIndexSummary(event.summary);
566
+ answerBox.textContent = "Ask a question from this PDF.";
567
+ showUploadMessage("PDF ready. Ask your question.", false, true);
568
+ queryInput.focus();
569
+ return;
570
+ }
571
+
572
+ if (event.type === "error") {
573
+ docStatus.textContent = "error";
574
+ appendIndexStep("Error", event.message);
575
+ showUploadMessage(cleanUploadError(event.message), true);
576
+ }
577
+ }
578
+
579
+ function appendPageChip(page) {
580
+ const confidence =
581
+ page.confidence === null || page.confidence === undefined
582
+ ? ""
583
+ : ` - ${(page.confidence * 100).toFixed(1)}%`;
584
+ const chip = document.createElement("div");
585
+ chip.className = "page-chip";
586
+ chip.dataset.page = String(page.page_number);
587
+ chip.textContent = `Page ${page.page_number}: ${page.engine}${confidence}`;
588
+ pageDetails.appendChild(chip);
589
+ }
590
+
591
+ function appendIndexStep(step) {
592
+ const labelValue = timelineLabel(step);
593
+ completedTimeline.add(timelineStepKey(labelValue));
594
+ const index = timelineSteps.findIndex((item) => timelineStepKey(item) === timelineStepKey(labelValue));
595
+ if (index >= 0) {
596
+ const next = timelineSteps[index + 1];
597
+ currentTimelineStep = next && !completedTimeline.has(timelineStepKey(next)) ? next : "";
598
+ }
599
+ renderIndexTimeline();
600
+ }
601
+
602
+ function renderPageText() {
603
+ const orderedPages = [...pageTexts.keys()].sort((a, b) => a - b);
604
+ resultText.textContent = orderedPages
605
+ .filter((pageNumber) => String(pageTexts.get(pageNumber) || "").trim())
606
+ .map((pageNumber) => `--- Page ${pageNumber} ---\n${pageTexts.get(pageNumber)}`)
607
+ .join("\n\n");
608
+ }
609
+
610
+ function renderAnswer(payload) {
611
+ addQueryHistory(payload.query || lastQuery);
612
+ answerBox.textContent = payload.answer || unrelatedPdfAnswer;
613
+ dashboardStatus.textContent = payload.status === "answered" ? "Answered" : "No answer";
614
+ renderConfidenceBadge(payload.retrieval_grade || payload.status, payload.evaluation || {});
615
+
616
+ citations.innerHTML = "";
617
+ for (const citation of payload.citations || []) {
618
+ const card = document.createElement("div");
619
+ card.className = "citation-card";
620
+ const heading = inferHeading(citation.preview);
621
+ const pageNumber = findPageForText(citation.preview) || citation.page_number || citation.parent_index + 1;
622
+ card.innerHTML = `
623
+ <div class="citation-title">
624
+ <span class="citation-file-badge">PDF</span>
625
+ <strong>${escapeHtml(currentFileName || payload.file_name || "Uploaded PDF")}</strong>
626
+ </div>
627
+ <div class="citation-meta">
628
+ <button type="button" data-page="${escapeHtml(pageNumber)}">Page ${escapeHtml(pageNumber)}</button>
629
+ <span>Heading: ${escapeHtml(heading)}</span>
630
+ <span>Parent Chunk: ${citation.parent_index + 1}</span>
631
+ </div>
632
+ <p>${escapeHtml(citation.preview)}</p>
633
+ `;
634
+ citations.appendChild(card);
635
+ }
636
+
637
+ renderHybridStats(payload.retrieval_stats || {});
638
+ renderEvaluationCards(payload.evaluation || {});
639
+ renderCragStatus(payload.retrieval_grade || payload.status);
640
+ renderHallucinationCard(payload.evaluation || {}, payload.retrieval_grade || payload.status);
641
+ renderRetrievedChunks(payload.hits || []);
642
+ }
643
+
644
+ function directStatus() {
645
+ if (directPages > 0 && ocrPages > 0) {
646
+ return "Mixed";
647
+ }
648
+ if (directPages > 0) {
649
+ return "Yes";
650
+ }
651
+ if (ocrPages > 0) {
652
+ return "No";
653
+ }
654
+ return "Checking";
655
+ }
656
+
657
+ function escapeHtml(value) {
658
+ return String(value || "")
659
+ .replaceAll("&", "&amp;")
660
+ .replaceAll("<", "&lt;")
661
+ .replaceAll(">", "&gt;")
662
+ .replaceAll('"', "&quot;")
663
+ .replaceAll("'", "&#039;");
664
+ }
665
+
666
+ function captureIndexDetail(step, detail) {
667
+ if (step.includes("Child Chunks") && detail && typeof detail === "object") {
668
+ chunkCounts = detail;
669
+ renderChunkComparison();
670
+ }
671
+ if (step === "Chunk Size Optimization" && detail && typeof detail === "object") {
672
+ chunkScores = detail.scores || {};
673
+ bestChunking.textContent = bestPerformerLabel(detail.best_chunking);
674
+ bestRecommendation.innerHTML = recommendationHtml(detail.best_chunking, detail.reason);
675
+ renderChunkComparison();
676
+ }
677
+ }
678
+
679
+ function captureIndexSummary(summary) {
680
+ if (!summary) {
681
+ return;
682
+ }
683
+ if (summary.chunk_strategy_counts) {
684
+ chunkCounts = summary.chunk_strategy_counts;
685
+ }
686
+ const recommendation = summary.chunking_recommendation || {};
687
+ chunkScores = recommendation.scores || chunkScores;
688
+ bestChunking.textContent = bestPerformerLabel(recommendation.best_chunking);
689
+ bestRecommendation.innerHTML = recommendationHtml(recommendation.best_chunking, recommendation.reason);
690
+ renderChunkComparison();
691
+ }
692
+
693
+ function renderChunkComparison() {
694
+ const strategies = ["fixed", "recursive", "semantic"];
695
+ chunkComparison.innerHTML = `
696
+ <div class="table-row table-head"><span>Chunking</span><span>Chunks</span><span>Score</span></div>
697
+ ${strategies
698
+ .map((strategy) => {
699
+ const winner = bestChunking.textContent.toLowerCase().includes(strategy) ? " winner" : "";
700
+ return `<div class="table-row${winner}">
701
+ <span>${label(strategy)}</span>
702
+ <span>${chunkCounts[strategy] || 0}</span>
703
+ <span>${chunkScores[strategy] ?? "-"}</span>
704
+ </div>`;
705
+ })
706
+ .join("")}
707
+ `;
708
+ }
709
+
710
+ function renderHybridStats(stats) {
711
+ renderPipelineStats(stats);
712
+ hybridStats.innerHTML = `
713
+ <div><span>Vector Results</span><strong>${stats.vector_results ?? "-"}</strong></div>
714
+ <div><span>BM25 Results</span><strong>${stats.bm25_results ?? "-"}</strong></div>
715
+ <div><span>Merged</span><strong>${stats.merged ?? "-"}</strong></div>
716
+ <div><span>Final Reranked</span><strong>${stats.final_reranked ?? "-"}</strong></div>
717
+ `;
718
+ }
719
+
720
+ function resetHybridStats() {
721
+ renderHybridStats({});
722
+ }
723
+
724
+ function renderEvaluationCards(metrics) {
725
+ evaluationCards.innerHTML = `
726
+ <div><span>Context Precision</span><strong>${metricScore(metrics.context_precision)}</strong></div>
727
+ <div><span>Context Recall</span><strong>${metricScore(metrics.context_recall)}</strong></div>
728
+ <div><span>Faithfulness</span><strong>${metricScore(metrics.faithfulness)}</strong></div>
729
+ <div><span>Answer Relevancy</span><strong>${metricScore(metrics.answer_relevancy)}</strong></div>
730
+ `;
731
+ }
732
+
733
+ function resetEvaluationCards() {
734
+ renderEvaluationCards({});
735
+ }
736
+
737
+ function renderRetrievedChunks(hits) {
738
+ if (!hits.length) {
739
+ retrievedChunks.innerHTML = '<div class="empty-state">No retrieved chunks yet.</div>';
740
+ return;
741
+ }
742
+ retrievedChunks.innerHTML = hits
743
+ .map(
744
+ (hit, index) => `
745
+ <details class="chunk-card">
746
+ <summary>
747
+ <span>#${index + 1}</span>
748
+ <strong>${label(hit.strategy)} Chunk</strong>
749
+ <div class="chunk-summary-scores">
750
+ <em>Similarity: ${score(hit.vector_score)}</em>
751
+ <em>BM25: ${bm25Score(hit.bm25_raw_score, hit.bm25_score)}</em>
752
+ <em>Reranker: ${score(hit.rerank_score)}</em>
753
+ </div>
754
+ </summary>
755
+ <p><strong>Chunk Text</strong>${escapeHtml(hit.child_text)}</p>
756
+ <div class="chunk-scores">
757
+ <span>Similarity Score: <strong>${score(hit.vector_score)}</strong></span>
758
+ <span>BM25 Score: <strong>${bm25Score(hit.bm25_raw_score, hit.bm25_score)}</strong></span>
759
+ <span>Reranker Score: <strong>${score(hit.rerank_score)}</strong></span>
760
+ </div>
761
+ </details>
762
+ `
763
+ )
764
+ .join("");
765
+ }
766
+
767
+ function score(value) {
768
+ return value === undefined || value === null ? "-" : Number(value).toFixed(2);
769
+ }
770
+
771
+ function metricScore(value) {
772
+ return value === undefined || value === null ? "Not Evaluated Yet" : Number(value).toFixed(2);
773
+ }
774
+
775
+ function bm25Score(raw, normalized) {
776
+ if (raw !== undefined && raw !== null) {
777
+ return Number(raw).toFixed(2);
778
+ }
779
+ return score(normalized);
780
+ }
781
+
782
+ function label(value) {
783
+ return String(value || "")
784
+ .replace(/^./, (char) => char.toUpperCase());
785
+ }
786
+
787
+ function inferHeading(preview) {
788
+ const text = String(preview || "").trim();
789
+ if (!text) {
790
+ return "Document context";
791
+ }
792
+ const firstLine = text.split(/\n|\. /)[0].trim();
793
+ const words = firstLine.split(/\s+/).slice(0, 5).join(" ");
794
+ return words || "Document context";
795
+ }
796
+
797
+ function timelineLabel(step) {
798
+ const labels = {
799
+ "Document Status: processing": "Processing Started",
800
+ "Text Cleaning": "Text Cleaning",
801
+ "Metadata": "Metadata Extraction",
802
+ "Parent Chunks": "Parent Chunks",
803
+ "Fixed / Recursive / Semantic Child Chunks": "Child Chunks",
804
+ "Chunk Size Optimization": "Chunk Size Optimization",
805
+ "Embeddings": "Embeddings",
806
+ "Embedding Evaluation": "Embedding Evaluation",
807
+ "Qdrant Store": "Qdrant Index",
808
+ "BM25 Index": "BM25 Index",
809
+ "Document Status: indexed": "Ready",
810
+ };
811
+ return labels[step] || step;
812
+ }
813
+
814
+ function renderCragStatus(grade) {
815
+ const good = grade === "good" || grade === "answered";
816
+ cragStatus.className = `crag-box ${good ? "good" : "bad"}`;
817
+ cragStatus.innerHTML = good
818
+ ? "<strong>Good</strong><span>Retrieved context passed CRAG grading.</span>"
819
+ : "<strong>Low Confidence</strong><span>Query rewrite or no-answer fallback was used.</span>";
820
+ }
821
+
822
+ function resetCragStatus() {
823
+ cragStatus.className = "crag-box neutral";
824
+ cragStatus.innerHTML = "<strong>Waiting</strong><span>Run a query to grade retrieval quality.</span>";
825
+ }
826
+
827
+ function renderSelectedFile(file, status) {
828
+ currentFileName = file.name;
829
+ currentFileSize = file.size;
830
+ uploadedFileName.textContent = file.name;
831
+ uploadedFileMeta.textContent = `0 Pages / ${formatBytes(file.size)}`;
832
+ fileCard.querySelector("em").textContent = status;
833
+ fileCard.classList.remove("is-hidden");
834
+ if (pdfPreviewUrl) {
835
+ URL.revokeObjectURL(pdfPreviewUrl);
836
+ }
837
+ pdfPreviewUrl = URL.createObjectURL(file);
838
+ pdfPreview.src = `${pdfPreviewUrl}#page=1&toolbar=0&navpanes=0&scrollbar=0`;
839
+ pdfPreviewCard.classList.remove("is-hidden");
840
+ }
841
+
842
+ function updateFileCardMeta(totalPages, status) {
843
+ if (!currentFileName) {
844
+ return;
845
+ }
846
+ uploadedFileName.textContent = currentFileName;
847
+ uploadedFileMeta.textContent = `${totalPages || pageTexts.size || 0} Pages / ${formatBytes(currentFileSize)}`;
848
+ fileCard.querySelector("em").textContent = status;
849
+ fileCard.classList.remove("is-hidden");
850
+ }
851
+
852
+ function formatBytes(bytes) {
853
+ if (!bytes) {
854
+ return "0 MB";
855
+ }
856
+ const mb = bytes / (1024 * 1024);
857
+ if (mb >= 1) {
858
+ return `${mb.toFixed(1)} MB`;
859
+ }
860
+ return `${Math.max(bytes / 1024, 1).toFixed(0)} KB`;
861
+ }
862
+
863
+ function renderIndexTimeline() {
864
+ indexSteps.innerHTML = timelineSteps
865
+ .map((step) => {
866
+ const key = timelineStepKey(step);
867
+ const done = completedTimeline.has(key);
868
+ const current = !done && currentTimelineStep && timelineStepKey(currentTimelineStep) === key;
869
+ const state = done ? "done" : current ? "current" : "pending";
870
+ const marker = done ? "OK" : current ? "..." : "";
871
+ return `<div class="timeline-item ${state}">
872
+ <span>${marker}</span>
873
+ <strong>${escapeHtml(step)}</strong>
874
+ </div>`;
875
+ })
876
+ .join("");
877
+ }
878
+
879
+ function timelineStepKey(value) {
880
+ return String(value || "").toLowerCase().replace(/[^a-z0-9]+/g, "");
881
+ }
882
+
883
+ function bestPerformerLabel(strategy) {
884
+ return strategy ? `Best Performer: ${label(strategy)} Chunking` : "Evaluation Pending";
885
+ }
886
+
887
+ function recommendationHtml(strategy, reason) {
888
+ const strategyLabel = label(strategy || "semantic");
889
+ return `
890
+ <strong>Recommended Strategy</strong>
891
+ <span>${strategyLabel} Chunking</span>
892
+ <p>Reason:</p>
893
+ <ul>
894
+ <li>${escapeHtml(reason || "Highest retrieval quality.")}</li>
895
+ <li>Best faithfulness</li>
896
+ <li>Best citation accuracy</li>
897
+ <li>Lowest hallucination risk</li>
898
+ </ul>
899
+ `;
900
+ }
901
+
902
+ function renderPipelineStats(stats = {}) {
903
+ const hasStats = Object.keys(stats).length > 0;
904
+ const vector = hasStats ? stats.vector_results ?? 0 : "Not Run";
905
+ const bm25 = hasStats ? stats.bm25_results ?? 0 : "Not Run";
906
+ const merged = hasStats ? stats.merged ?? 0 : "Not Run";
907
+ const final = hasStats ? stats.final_reranked ?? 0 : "Not Run";
908
+ pipelineViz.innerHTML = `
909
+ <div>Query</div>
910
+ <span></span>
911
+ <div>Vector Search (${vector})</div>
912
+ <span></span>
913
+ <div>BM25 (${bm25})</div>
914
+ <span></span>
915
+ <div>Merged (${merged})</div>
916
+ <span></span>
917
+ <div>Reranked (${final})</div>
918
+ <span></span>
919
+ <div>Answer</div>
920
+ `;
921
+ }
922
+
923
+ function renderConfidenceBadge(grade, metrics = {}) {
924
+ if (!grade) {
925
+ confidenceBadge.className = "confidence-badge neutral";
926
+ confidenceBadge.textContent = "Confidence pending";
927
+ return;
928
+ }
929
+ const faithfulness = Number(metrics.faithfulness || 0);
930
+ const gradeGood = grade === "good" || grade === "answered";
931
+ const level = gradeGood && faithfulness >= 0.9 ? "high" : gradeGood && faithfulness >= 0.75 ? "medium" : "low";
932
+ const labelMap = { high: "High Confidence", medium: "Medium Confidence", low: "Low Confidence" };
933
+ confidenceBadge.className = `confidence-badge ${level}`;
934
+ confidenceBadge.textContent = labelMap[level];
935
+ }
936
+
937
+ function renderHallucinationCard(metrics = {}, grade) {
938
+ const faithfulness = metrics.faithfulness;
939
+ const precision = metrics.context_precision;
940
+ const coverage = faithfulness === undefined || precision === undefined ? null : Math.round(((faithfulness + precision) / 2) * 100);
941
+ const good = (grade === "good" || grade === "answered") && Number(faithfulness || 0) >= 0.75;
942
+ hallucinationCard.className = `hallucination-card ${coverage === null ? "neutral" : good ? "good" : "bad"}`;
943
+ hallucinationCard.innerHTML = `
944
+ <span>Answer Source Coverage</span>
945
+ <strong>${coverage === null ? "-" : `${coverage}%`}</strong>
946
+ <span>Faithfulness</span>
947
+ <strong>${score(faithfulness)}</strong>
948
+ <em>${coverage === null ? "Waiting for answer" : good ? "No Hallucination Detected" : "Needs review"}</em>
949
+ `;
950
+ }
951
+
952
+ function addQueryHistory(query) {
953
+ const clean = String(query || "").trim();
954
+ if (!clean) {
955
+ return;
956
+ }
957
+ const existingIndex = queryHistoryItems.findIndex((item) => item.toLowerCase() === clean.toLowerCase());
958
+ if (existingIndex >= 0) {
959
+ queryHistoryItems.splice(existingIndex, 1);
960
+ }
961
+ queryHistoryItems.unshift(clean);
962
+ queryHistoryItems.splice(5);
963
+ renderQueryHistory();
964
+ }
965
+
966
+ function renderQueryHistory() {
967
+ historyCount.textContent = `${queryHistoryItems.length} saved`;
968
+ if (!queryHistoryItems.length) {
969
+ queryHistory.innerHTML = "<span>No questions yet.</span>";
970
+ return;
971
+ }
972
+ queryHistory.innerHTML = queryHistoryItems
973
+ .map((query) => `<button type="button">${escapeHtml(query)}</button>`)
974
+ .join("");
975
+ }
976
+
977
+ function findPageForText(preview) {
978
+ const needle = String(preview || "").replace(/\s+/g, " ").trim().slice(0, 90);
979
+ if (!needle) {
980
+ return "";
981
+ }
982
+ for (const [page, text] of pageTexts.entries()) {
983
+ const haystack = String(text || "").replace(/\s+/g, " ");
984
+ if (haystack.includes(needle)) {
985
+ return page;
986
+ }
987
+ }
988
+ return "";
989
+ }
990
+
991
+ function scrollToExtractedPage(pageNumber) {
992
+ const details = document.querySelector(".details-panel details");
993
+ details.open = true;
994
+ document.querySelector(".details-panel").scrollIntoView({ behavior: "smooth", block: "start" });
995
+ pageDetails.querySelectorAll(".page-chip").forEach((chip) => {
996
+ chip.classList.toggle("is-active", chip.dataset.page === String(pageNumber));
997
+ });
998
+ }
999
+
1000
+ function showUploadMessage(message, isError = false, isSuccess = false) {
1001
+ uploadMessage.textContent = message;
1002
+ uploadMessage.className = `upload-message${isError ? " error" : ""}${isSuccess ? " success" : ""}`;
1003
+ }
1004
+
1005
+ function cleanUploadError(message) {
1006
+ const text = String(message || "");
1007
+ const keyMatch = text.match(/Tried\s+(\d+)\s+active API key\(s\) out of\s+(\d+)\s+configured key\(s\);\s+(\d+)\s+key\(s\)/i);
1008
+ if (keyMatch) {
1009
+ const [, triedKeys, totalKeys, quotaKeys] = keyMatch;
1010
+ return `Gemini OCR ne ${triedKeys}/${totalKeys} API keys try ki. ${quotaKeys} key(s) quota/rate-limit de rahi hain. Agar yahi message aa raha hai to ya to sab keys exhausted hain, ya server/browser abhi old code use kar raha hai. Server restart aur browser hard refresh ke baad dobara try karo.`;
1011
+ }
1012
+ if (text.toLowerCase().includes("api key not valid") || text.toLowerCase().includes("invalid api key")) {
1013
+ return "Gemini API key valid nahi lag rahi. env file me GEMINI_API_KEY_1 se GEMINI_API_KEY_4 tak valid Google Gemini keys daalo, phir server restart karo.";
1014
+ }
1015
+ if (text.includes("RESOURCE_EXHAUSTED") || text.toLowerCase().includes("quota")) {
1016
+ return "Gemini OCR quota/rate-limit error de raha hai. App next keys try karta hai; agar phir bhi ye aa raha hai to configured keys exhausted/invalid ho sakti hain. Server restart aur hard refresh ke baad dobara try karo.";
1017
+ }
1018
+ if (text.includes("temporarily unavailable") || text.includes("503")) {
1019
+ return "Gemini OCR abhi busy hai. Thodi der baad try karo ya text-based PDF upload karo.";
1020
+ }
1021
+ if (text.toLowerCase().includes("password")) {
1022
+ return "PDF password-protected hai. Unlocked PDF upload karo.";
1023
+ }
1024
+ return text;
1025
+ }
1026
+
1027
+ renderIndexTimeline();
1028
+ resetHybridStats();
1029
+ resetEvaluationCards();
1030
+ resetCragStatus();
1031
+ renderHallucinationCard();
1032
+ renderRetrievedChunks([]);
1033
+ renderQueryHistory();
static/styles.css ADDED
@@ -0,0 +1,998 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ :root {
2
+ color-scheme: light;
3
+ --bg: #eef2f4;
4
+ --surface: #ffffff;
5
+ --surface-strong: #f8fafb;
6
+ --text: #172026;
7
+ --muted: #65717a;
8
+ --line: #d7dee3;
9
+ --accent: #1f7a6d;
10
+ --accent-strong: #145a51;
11
+ --info: #2f5f9f;
12
+ --warning: #b7791f;
13
+ --success: #2f7d50;
14
+ --danger: #a33b3b;
15
+ }
16
+
17
+ * {
18
+ box-sizing: border-box;
19
+ }
20
+
21
+ body {
22
+ margin: 0;
23
+ min-height: 100vh;
24
+ font-family: Inter, Segoe UI, system-ui, -apple-system, sans-serif;
25
+ background: var(--bg);
26
+ color: var(--text);
27
+ }
28
+
29
+ button,
30
+ input {
31
+ font: inherit;
32
+ }
33
+
34
+ .app-shell {
35
+ width: min(1180px, calc(100% - 32px));
36
+ margin: 0 auto;
37
+ padding: 32px 0;
38
+ }
39
+
40
+ .workspace {
41
+ display: grid;
42
+ grid-template-columns: 390px minmax(0, 1fr);
43
+ gap: 18px;
44
+ align-items: start;
45
+ }
46
+
47
+ .panel {
48
+ background: var(--surface);
49
+ border: 1px solid var(--line);
50
+ border-radius: 8px;
51
+ box-shadow: 0 12px 32px rgba(24, 38, 48, 0.08);
52
+ }
53
+
54
+ .upload-panel {
55
+ padding: 20px;
56
+ }
57
+
58
+ .brand-row {
59
+ display: flex;
60
+ gap: 12px;
61
+ align-items: center;
62
+ margin-bottom: 20px;
63
+ }
64
+
65
+ .brand-mark {
66
+ display: grid;
67
+ place-items: center;
68
+ width: 44px;
69
+ height: 44px;
70
+ border-radius: 8px;
71
+ background: var(--accent);
72
+ color: white;
73
+ font-weight: 800;
74
+ }
75
+
76
+ h1,
77
+ h2,
78
+ p {
79
+ margin: 0;
80
+ }
81
+
82
+ h1 {
83
+ font-size: 24px;
84
+ line-height: 1.15;
85
+ }
86
+
87
+ h2 {
88
+ font-size: 18px;
89
+ }
90
+
91
+ p {
92
+ margin-top: 4px;
93
+ color: var(--muted);
94
+ font-size: 14px;
95
+ }
96
+
97
+ .brand-subline {
98
+ display: block;
99
+ margin-top: 8px;
100
+ color: #33515b;
101
+ font-size: 13px;
102
+ font-weight: 700;
103
+ }
104
+
105
+ .drop-zone {
106
+ display: grid;
107
+ gap: 14px;
108
+ padding: 16px;
109
+ border: 1px dashed #9caab3;
110
+ border-radius: 8px;
111
+ background: var(--surface-strong);
112
+ }
113
+
114
+ .drop-zone.is-dragging {
115
+ border-color: var(--accent);
116
+ background: #edf8f5;
117
+ }
118
+
119
+ .drop-zone input {
120
+ width: 100%;
121
+ }
122
+
123
+ .drop-content {
124
+ display: grid;
125
+ gap: 4px;
126
+ }
127
+
128
+ .drop-content span {
129
+ color: var(--muted);
130
+ font-size: 13px;
131
+ }
132
+
133
+ .file-card,
134
+ .pdf-preview-card {
135
+ margin-top: 14px;
136
+ }
137
+
138
+ .file-card {
139
+ display: grid;
140
+ grid-template-columns: 48px minmax(0, 1fr);
141
+ gap: 12px;
142
+ align-items: center;
143
+ padding: 12px;
144
+ border: 1px solid #b9d5cc;
145
+ border-radius: 8px;
146
+ background: #f2fbf8;
147
+ }
148
+
149
+ .file-icon {
150
+ display: grid;
151
+ place-items: center;
152
+ width: 48px;
153
+ height: 56px;
154
+ border-radius: 6px;
155
+ background: #d84b4b;
156
+ color: white;
157
+ font-size: 12px;
158
+ font-weight: 900;
159
+ }
160
+
161
+ .file-card strong,
162
+ .file-card span,
163
+ .file-card em {
164
+ display: block;
165
+ overflow-wrap: anywhere;
166
+ }
167
+
168
+ .file-card span {
169
+ margin-top: 3px;
170
+ color: var(--muted);
171
+ font-size: 13px;
172
+ }
173
+
174
+ .file-card em {
175
+ margin-top: 4px;
176
+ color: var(--success);
177
+ font-size: 13px;
178
+ font-style: normal;
179
+ font-weight: 800;
180
+ }
181
+
182
+ .pdf-preview-card {
183
+ padding: 12px;
184
+ border: 1px solid var(--line);
185
+ border-radius: 8px;
186
+ background: #fbfcfc;
187
+ }
188
+
189
+ .pdf-preview-card iframe {
190
+ width: 100%;
191
+ height: 210px;
192
+ margin-top: 10px;
193
+ border: 1px solid var(--line);
194
+ border-radius: 6px;
195
+ background: white;
196
+ }
197
+
198
+ .is-hidden {
199
+ display: none;
200
+ }
201
+
202
+ button {
203
+ min-height: 40px;
204
+ border: 0;
205
+ border-radius: 6px;
206
+ background: var(--accent);
207
+ color: white;
208
+ font-weight: 700;
209
+ cursor: pointer;
210
+ }
211
+
212
+ button:hover {
213
+ background: var(--accent-strong);
214
+ }
215
+
216
+ .status-grid {
217
+ display: grid;
218
+ grid-template-columns: 1fr;
219
+ gap: 10px;
220
+ margin-top: 18px;
221
+ }
222
+
223
+ .upload-message {
224
+ margin-top: 14px;
225
+ padding: 12px;
226
+ border: 1px solid var(--line);
227
+ border-radius: 8px;
228
+ background: #fbfcfc;
229
+ color: var(--muted);
230
+ line-height: 1.45;
231
+ }
232
+
233
+ .upload-message.error {
234
+ border-color: #dfaaaa;
235
+ background: #fff3f3;
236
+ color: #8f2f2f;
237
+ }
238
+
239
+ .upload-message.success {
240
+ border-color: #8cc9b6;
241
+ background: #edf8f5;
242
+ color: #244a45;
243
+ }
244
+
245
+ .status-grid div {
246
+ display: grid;
247
+ gap: 4px;
248
+ padding: 12px;
249
+ border: 1px solid var(--line);
250
+ border-radius: 8px;
251
+ background: #fbfcfc;
252
+ }
253
+
254
+ .status-grid span {
255
+ color: var(--muted);
256
+ font-size: 12px;
257
+ text-transform: uppercase;
258
+ letter-spacing: 0;
259
+ }
260
+
261
+ .status-grid strong {
262
+ overflow-wrap: anywhere;
263
+ }
264
+
265
+ .page-details {
266
+ display: flex;
267
+ flex-wrap: wrap;
268
+ gap: 8px;
269
+ margin-top: 16px;
270
+ }
271
+
272
+ .page-chip {
273
+ max-width: 100%;
274
+ padding: 7px 9px;
275
+ border-radius: 6px;
276
+ background: #e7efed;
277
+ color: #244a45;
278
+ font-size: 13px;
279
+ overflow-wrap: anywhere;
280
+ }
281
+
282
+ .index-section {
283
+ margin-top: 18px;
284
+ padding-top: 16px;
285
+ border-top: 1px solid var(--line);
286
+ }
287
+
288
+ .section-head {
289
+ display: flex;
290
+ align-items: center;
291
+ justify-content: space-between;
292
+ gap: 10px;
293
+ }
294
+
295
+ .section-head span,
296
+ .result-toolbar span {
297
+ color: var(--muted);
298
+ font-size: 13px;
299
+ overflow-wrap: anywhere;
300
+ }
301
+
302
+ .step-list {
303
+ display: grid;
304
+ gap: 8px;
305
+ margin-top: 12px;
306
+ }
307
+
308
+ .step-item {
309
+ padding: 9px 10px;
310
+ border: 1px solid var(--line);
311
+ border-radius: 6px;
312
+ background: #fbfcfc;
313
+ font-size: 13px;
314
+ }
315
+
316
+ .result-panel {
317
+ display: grid;
318
+ grid-template-rows: auto 1fr;
319
+ overflow: hidden;
320
+ }
321
+
322
+ .result-toolbar {
323
+ display: flex;
324
+ align-items: center;
325
+ justify-content: space-between;
326
+ gap: 12px;
327
+ padding: 16px 18px;
328
+ border-bottom: 1px solid var(--line);
329
+ }
330
+
331
+ .result-toolbar button {
332
+ min-width: 82px;
333
+ min-height: 34px;
334
+ }
335
+
336
+ pre {
337
+ margin: 0;
338
+ padding: 18px;
339
+ overflow: auto;
340
+ white-space: pre-wrap;
341
+ word-break: break-word;
342
+ font: 14px/1.55 Consolas, Monaco, monospace;
343
+ color: #243038;
344
+ }
345
+
346
+ .query-panel {
347
+ display: grid;
348
+ gap: 14px;
349
+ align-content: start;
350
+ min-height: 430px;
351
+ padding-bottom: 18px;
352
+ }
353
+
354
+ .query-suggestions {
355
+ display: flex;
356
+ flex-wrap: wrap;
357
+ gap: 8px;
358
+ padding: 0 18px;
359
+ }
360
+
361
+ .query-suggestions strong {
362
+ flex-basis: 100%;
363
+ color: var(--muted);
364
+ font-size: 12px;
365
+ text-transform: uppercase;
366
+ }
367
+
368
+ .query-suggestions button,
369
+ .query-history button,
370
+ .citation-card button {
371
+ min-height: 32px;
372
+ padding: 0 10px;
373
+ border: 1px solid var(--line);
374
+ border-radius: 6px;
375
+ background: white;
376
+ color: var(--text);
377
+ font-size: 13px;
378
+ font-weight: 700;
379
+ }
380
+
381
+ .query-suggestions button:hover,
382
+ .query-history button:hover,
383
+ .citation-card button:hover {
384
+ border-color: var(--accent);
385
+ background: #edf8f5;
386
+ color: var(--accent-strong);
387
+ }
388
+
389
+ .details-panel {
390
+ grid-column: 1 / -1;
391
+ min-height: auto;
392
+ }
393
+
394
+ .details-panel details {
395
+ padding: 16px 18px;
396
+ }
397
+
398
+ .details-panel summary {
399
+ cursor: pointer;
400
+ color: var(--accent-strong);
401
+ font-weight: 700;
402
+ }
403
+
404
+ .details-panel pre {
405
+ max-height: 360px;
406
+ margin-top: 14px;
407
+ border: 1px solid var(--line);
408
+ border-radius: 8px;
409
+ background: #fbfcfc;
410
+ }
411
+
412
+ .query-form {
413
+ display: grid;
414
+ grid-template-columns: minmax(0, 1fr) auto;
415
+ gap: 10px;
416
+ padding: 0 18px;
417
+ }
418
+
419
+ .feedback-form {
420
+ display: none;
421
+ grid-template-columns: 84px minmax(0, 1fr) auto;
422
+ gap: 10px;
423
+ padding: 0 18px;
424
+ }
425
+
426
+ .query-form input,
427
+ .feedback-form input,
428
+ .feedback-form select {
429
+ min-height: 40px;
430
+ padding: 0 11px;
431
+ border: 1px solid var(--line);
432
+ border-radius: 6px;
433
+ }
434
+
435
+ .answer-box,
436
+ .citation-list,
437
+ .retrieved-chunks {
438
+ margin: 0 18px;
439
+ }
440
+
441
+ .confidence-badge {
442
+ justify-self: start;
443
+ margin: 0 18px;
444
+ padding: 7px 10px;
445
+ border: 1px solid var(--line);
446
+ border-radius: 999px;
447
+ background: white;
448
+ color: var(--muted);
449
+ font-size: 13px;
450
+ font-weight: 900;
451
+ }
452
+
453
+ .confidence-badge.high {
454
+ border-color: #94cfaa;
455
+ background: #eef9f2;
456
+ color: var(--success);
457
+ }
458
+
459
+ .confidence-badge.medium {
460
+ border-color: #e8c77d;
461
+ background: #fff8e8;
462
+ color: var(--warning);
463
+ }
464
+
465
+ .confidence-badge.low {
466
+ border-color: #dfaaaa;
467
+ background: #fff3f3;
468
+ color: var(--danger);
469
+ }
470
+
471
+ .answer-box {
472
+ min-height: 210px;
473
+ padding: 14px;
474
+ border: 1px solid var(--line);
475
+ border-radius: 8px;
476
+ background: var(--surface-strong);
477
+ line-height: 1.55;
478
+ white-space: pre-wrap;
479
+ font-size: 17px;
480
+ }
481
+
482
+ .citation-list {
483
+ display: grid;
484
+ gap: 10px;
485
+ }
486
+
487
+ .retrieved-chunks {
488
+ display: none;
489
+ }
490
+
491
+ .citation-card,
492
+ .chunk-card {
493
+ padding: 12px;
494
+ border: 1px solid var(--line);
495
+ border-radius: 8px;
496
+ background: #fbfcfc;
497
+ }
498
+
499
+ .citation-card strong,
500
+ .chunk-card strong {
501
+ display: block;
502
+ margin-bottom: 6px;
503
+ }
504
+
505
+ .chunk-card p,
506
+ .citation-card p {
507
+ color: var(--text);
508
+ font-size: 13px;
509
+ line-height: 1.5;
510
+ }
511
+
512
+ @media (max-width: 860px) {
513
+ .app-shell {
514
+ width: min(100% - 20px, 680px);
515
+ padding: 10px 0;
516
+ }
517
+
518
+ .workspace {
519
+ grid-template-columns: 1fr;
520
+ }
521
+
522
+ .result-panel {
523
+ min-height: 55vh;
524
+ }
525
+
526
+ .query-form,
527
+ .feedback-form {
528
+ grid-template-columns: 1fr;
529
+ }
530
+ }
531
+
532
+ .dashboard-panel {
533
+ grid-column: 1 / -1;
534
+ display: grid;
535
+ gap: 0;
536
+ padding-bottom: 0;
537
+ }
538
+
539
+ .advanced-dashboard {
540
+ padding: 16px 18px 18px;
541
+ }
542
+
543
+ .advanced-dashboard summary {
544
+ cursor: pointer;
545
+ color: var(--accent-strong);
546
+ font-weight: 800;
547
+ }
548
+
549
+ .dashboard-grid {
550
+ display: grid;
551
+ grid-template-columns: repeat(2, minmax(0, 1fr));
552
+ gap: 14px;
553
+ padding: 16px 0 0;
554
+ }
555
+
556
+ .metric-section {
557
+ min-width: 0;
558
+ padding: 14px;
559
+ border: 1px solid var(--line);
560
+ border-radius: 8px;
561
+ background: #fbfcfc;
562
+ }
563
+
564
+ .metric-section h2 {
565
+ font-size: 16px;
566
+ }
567
+
568
+ .timeline-list {
569
+ display: grid;
570
+ gap: 8px;
571
+ margin-top: 12px;
572
+ }
573
+
574
+ .timeline-item {
575
+ display: grid;
576
+ grid-template-columns: 34px minmax(0, 1fr);
577
+ align-items: center;
578
+ gap: 10px;
579
+ padding: 8px 0;
580
+ border-bottom: 1px solid #edf1f3;
581
+ }
582
+
583
+ .timeline-item span {
584
+ display: grid;
585
+ place-items: center;
586
+ width: 28px;
587
+ height: 28px;
588
+ border-radius: 50%;
589
+ background: var(--accent);
590
+ color: white;
591
+ font-size: 10px;
592
+ font-weight: 800;
593
+ }
594
+
595
+ .timeline-item.pending span {
596
+ background: #d7dee3;
597
+ }
598
+
599
+ .timeline-item.current span {
600
+ background: var(--info);
601
+ animation: pulse 1.2s ease-in-out infinite;
602
+ }
603
+
604
+ .timeline-item.done span {
605
+ background: var(--success);
606
+ }
607
+
608
+ @keyframes pulse {
609
+ 0%,
610
+ 100% {
611
+ transform: scale(1);
612
+ }
613
+ 50% {
614
+ transform: scale(1.08);
615
+ }
616
+ }
617
+
618
+ .comparison-table {
619
+ display: grid;
620
+ gap: 6px;
621
+ margin-top: 12px;
622
+ }
623
+
624
+ .table-row {
625
+ display: grid;
626
+ grid-template-columns: 1fr 70px 70px;
627
+ gap: 10px;
628
+ align-items: center;
629
+ padding: 9px 10px;
630
+ border-radius: 6px;
631
+ background: white;
632
+ border: 1px solid var(--line);
633
+ font-size: 13px;
634
+ }
635
+
636
+ .table-head {
637
+ color: var(--muted);
638
+ font-weight: 700;
639
+ text-transform: uppercase;
640
+ font-size: 11px;
641
+ }
642
+
643
+ .table-row.winner {
644
+ border-color: var(--accent);
645
+ background: #edf8f5;
646
+ }
647
+
648
+ .table-row.winner span:first-child::before {
649
+ content: "Winner ";
650
+ color: var(--accent-strong);
651
+ font-weight: 800;
652
+ }
653
+
654
+ .winner-badge {
655
+ padding: 5px 8px;
656
+ border: 1px solid #b9d5cc;
657
+ border-radius: 999px;
658
+ background: #edf8f5;
659
+ color: var(--accent-strong) !important;
660
+ font-weight: 900;
661
+ }
662
+
663
+ .stat-grid {
664
+ display: grid;
665
+ grid-template-columns: repeat(2, minmax(0, 1fr));
666
+ gap: 10px;
667
+ margin-top: 12px;
668
+ }
669
+
670
+ .stat-grid div {
671
+ display: grid;
672
+ gap: 4px;
673
+ padding: 12px;
674
+ border: 1px solid var(--line);
675
+ border-radius: 8px;
676
+ background: white;
677
+ }
678
+
679
+ .stat-grid span {
680
+ color: var(--muted);
681
+ font-size: 12px;
682
+ }
683
+
684
+ .stat-grid strong {
685
+ font-size: 18px;
686
+ line-height: 1.2;
687
+ overflow-wrap: anywhere;
688
+ }
689
+
690
+ .answer-card {
691
+ display: grid;
692
+ gap: 10px;
693
+ }
694
+
695
+ .answer-card > span {
696
+ margin: 0 18px;
697
+ color: var(--muted);
698
+ font-size: 12px;
699
+ font-weight: 800;
700
+ text-transform: uppercase;
701
+ }
702
+
703
+ .citation-title {
704
+ display: grid;
705
+ grid-template-columns: 42px minmax(0, 1fr);
706
+ gap: 10px;
707
+ align-items: center;
708
+ margin-bottom: 10px;
709
+ }
710
+
711
+ .citation-file-badge {
712
+ display: grid !important;
713
+ place-items: center;
714
+ width: 38px;
715
+ height: 42px;
716
+ margin: 0 !important;
717
+ border-radius: 6px;
718
+ background: #d84b4b;
719
+ color: white !important;
720
+ font-size: 11px !important;
721
+ font-weight: 900;
722
+ }
723
+
724
+ .citation-meta {
725
+ display: flex;
726
+ flex-wrap: wrap;
727
+ gap: 8px;
728
+ align-items: center;
729
+ margin-bottom: 8px;
730
+ }
731
+
732
+ .citation-card span {
733
+ display: block;
734
+ margin-bottom: 6px;
735
+ color: var(--muted);
736
+ font-size: 12px;
737
+ }
738
+
739
+ .citation-meta span {
740
+ margin: 0;
741
+ padding: 7px 9px;
742
+ border: 1px solid var(--line);
743
+ border-radius: 6px;
744
+ background: white;
745
+ }
746
+
747
+ .citation-card button {
748
+ display: inline-flex;
749
+ align-items: center;
750
+ margin: 0;
751
+ }
752
+
753
+ .retrieval-section,
754
+ .recommendation-section {
755
+ margin: 14px 0 0;
756
+ }
757
+
758
+ .retrieved-chunks {
759
+ display: grid;
760
+ gap: 10px;
761
+ margin: 12px 0 0;
762
+ }
763
+
764
+ .chunk-card summary {
765
+ display: grid;
766
+ grid-template-columns: 44px 150px minmax(0, 1fr);
767
+ gap: 10px;
768
+ align-items: center;
769
+ cursor: pointer;
770
+ }
771
+
772
+ .chunk-card summary span {
773
+ color: var(--accent-strong);
774
+ font-weight: 800;
775
+ }
776
+
777
+ .chunk-summary-scores {
778
+ display: flex;
779
+ flex-wrap: wrap;
780
+ gap: 6px;
781
+ }
782
+
783
+ .chunk-card summary em {
784
+ padding: 5px 7px;
785
+ border: 1px solid var(--line);
786
+ border-radius: 6px;
787
+ background: white;
788
+ color: var(--muted);
789
+ font-style: normal;
790
+ font-size: 12px;
791
+ overflow-wrap: anywhere;
792
+ }
793
+
794
+ .chunk-card p strong {
795
+ margin-top: 12px;
796
+ }
797
+
798
+ .chunk-scores {
799
+ display: grid;
800
+ grid-template-columns: repeat(3, minmax(0, 1fr));
801
+ gap: 8px;
802
+ margin-top: 10px;
803
+ }
804
+
805
+ .chunk-scores span {
806
+ padding: 8px;
807
+ border: 1px solid var(--line);
808
+ border-radius: 6px;
809
+ background: white;
810
+ color: var(--muted);
811
+ font-size: 12px;
812
+ }
813
+
814
+ .chunk-scores strong {
815
+ display: block;
816
+ margin-top: 3px;
817
+ color: var(--text);
818
+ }
819
+
820
+ .recommendation-box {
821
+ margin-top: 10px;
822
+ padding: 14px;
823
+ border-radius: 8px;
824
+ background: #edf8f5;
825
+ color: #244a45;
826
+ font-weight: 700;
827
+ }
828
+
829
+ .recommendation-box strong,
830
+ .recommendation-box span {
831
+ display: block;
832
+ }
833
+
834
+ .recommendation-box span {
835
+ margin-top: 5px;
836
+ font-size: 18px;
837
+ }
838
+
839
+ .recommendation-box p {
840
+ margin-top: 10px;
841
+ }
842
+
843
+ .recommendation-box ul {
844
+ margin: 8px 0 0;
845
+ padding-left: 20px;
846
+ }
847
+
848
+ .recommendation-box li {
849
+ margin-top: 4px;
850
+ }
851
+
852
+ .crag-box {
853
+ display: grid;
854
+ gap: 6px;
855
+ margin-top: 12px;
856
+ padding: 14px;
857
+ border: 1px solid var(--line);
858
+ border-radius: 8px;
859
+ background: white;
860
+ }
861
+
862
+ .crag-box strong {
863
+ font-size: 22px;
864
+ }
865
+
866
+ .crag-box span {
867
+ color: var(--muted);
868
+ font-size: 13px;
869
+ }
870
+
871
+ .crag-box.good {
872
+ border-color: #8cc9b6;
873
+ background: #edf8f5;
874
+ }
875
+
876
+ .crag-box.bad {
877
+ border-color: #dfaaaa;
878
+ background: #fff3f3;
879
+ }
880
+
881
+ .pipeline-viz {
882
+ display: grid;
883
+ gap: 7px;
884
+ margin-top: 12px;
885
+ }
886
+
887
+ .pipeline-viz div {
888
+ padding: 9px 10px;
889
+ border: 1px solid var(--line);
890
+ border-radius: 6px;
891
+ background: white;
892
+ font-size: 13px;
893
+ font-weight: 800;
894
+ overflow-wrap: anywhere;
895
+ }
896
+
897
+ .pipeline-viz span {
898
+ width: 1px;
899
+ height: 12px;
900
+ margin-left: 18px;
901
+ background: var(--line);
902
+ position: relative;
903
+ }
904
+
905
+ .pipeline-viz span::after {
906
+ content: "";
907
+ position: absolute;
908
+ left: -4px;
909
+ bottom: -1px;
910
+ width: 8px;
911
+ height: 8px;
912
+ border-right: 1px solid var(--line);
913
+ border-bottom: 1px solid var(--line);
914
+ transform: rotate(45deg);
915
+ }
916
+
917
+ .hallucination-card {
918
+ display: grid;
919
+ grid-template-columns: 1fr auto;
920
+ gap: 8px 12px;
921
+ margin-top: 12px;
922
+ padding: 14px;
923
+ border: 1px solid var(--line);
924
+ border-radius: 8px;
925
+ background: white;
926
+ }
927
+
928
+ .hallucination-card span {
929
+ color: var(--muted);
930
+ font-size: 13px;
931
+ }
932
+
933
+ .hallucination-card strong {
934
+ font-size: 20px;
935
+ }
936
+
937
+ .hallucination-card em {
938
+ grid-column: 1 / -1;
939
+ padding-top: 8px;
940
+ border-top: 1px solid var(--line);
941
+ color: var(--muted);
942
+ font-style: normal;
943
+ font-weight: 900;
944
+ }
945
+
946
+ .hallucination-card.good {
947
+ border-color: #94cfaa;
948
+ background: #eef9f2;
949
+ }
950
+
951
+ .hallucination-card.bad {
952
+ border-color: #dfaaaa;
953
+ background: #fff3f3;
954
+ }
955
+
956
+ .history-panel {
957
+ grid-column: 1 / -1;
958
+ }
959
+
960
+ .query-history {
961
+ display: flex;
962
+ flex-wrap: wrap;
963
+ gap: 8px;
964
+ padding: 16px 18px;
965
+ }
966
+
967
+ .query-history > span {
968
+ color: var(--muted);
969
+ font-size: 14px;
970
+ }
971
+
972
+ .page-chip.is-active {
973
+ outline: 3px solid rgba(31, 122, 109, 0.22);
974
+ border: 1px solid var(--accent);
975
+ background: #dff4ee;
976
+ }
977
+
978
+ .empty-state {
979
+ padding: 12px;
980
+ color: var(--muted);
981
+ border: 1px dashed var(--line);
982
+ border-radius: 8px;
983
+ }
984
+
985
+ @media (max-width: 860px) {
986
+ .dashboard-grid {
987
+ grid-template-columns: 1fr;
988
+ }
989
+
990
+ .chunk-card summary {
991
+ grid-template-columns: 1fr;
992
+ }
993
+
994
+ .chunk-scores,
995
+ .hallucination-card {
996
+ grid-template-columns: 1fr;
997
+ }
998
+ }
templates/index.html ADDED
@@ -0,0 +1,203 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ <!doctype html>
2
+ <html lang="en">
3
+ <head>
4
+ <meta charset="utf-8" />
5
+ <meta name="viewport" content="width=device-width, initial-scale=1" />
6
+ <title>SmartNotes PDF OCR</title>
7
+ <link rel="stylesheet" href="/static/styles.css" />
8
+ </head>
9
+ <body>
10
+ <main class="app-shell">
11
+ <section class="workspace">
12
+ <div class="panel upload-panel">
13
+ <div class="brand-row">
14
+ <span class="brand-mark">SN</span>
15
+ <div>
16
+ <h1>SmartNotes AI</h1>
17
+ <p>Advanced RAG-Powered PDF Intelligence Platform</p>
18
+ <span class="brand-subline">Hybrid Retrieval / Reranking / Citations / Evaluation</span>
19
+ </div>
20
+ </div>
21
+
22
+ <form id="uploadForm" class="drop-zone">
23
+ <input id="pdfInput" type="file" name="file" accept="application/pdf" />
24
+ <div class="drop-content">
25
+ <strong id="fileName">Choose or drop a PDF</strong>
26
+ <span>Direct text first, OCR only when needed</span>
27
+ </div>
28
+ <button type="submit">Upload PDF</button>
29
+ </form>
30
+
31
+ <div id="fileCard" class="file-card is-hidden">
32
+ <div class="file-icon">PDF</div>
33
+ <div>
34
+ <strong id="uploadedFileName">notes.pdf</strong>
35
+ <span id="uploadedFileMeta">0 Pages / 0 MB</span>
36
+ <em>Uploaded Successfully</em>
37
+ </div>
38
+ </div>
39
+
40
+ <div id="pdfPreviewCard" class="pdf-preview-card is-hidden">
41
+ <div class="section-head">
42
+ <h2>PDF Preview</h2>
43
+ <span>First page</span>
44
+ </div>
45
+ <iframe id="pdfPreview" title="PDF first page preview"></iframe>
46
+ </div>
47
+
48
+ <div class="status-grid">
49
+ <div>
50
+ <span>Route</span>
51
+ <strong id="routeValue">Waiting</strong>
52
+ </div>
53
+ <div>
54
+ <span>Pages</span>
55
+ <strong id="pageCount">0</strong>
56
+ </div>
57
+ <div>
58
+ <span>Direct Text</span>
59
+ <strong id="directText">-</strong>
60
+ </div>
61
+ </div>
62
+ <div id="uploadMessage" class="upload-message">Upload a PDF to begin.</div>
63
+ </div>
64
+
65
+ <div class="panel query-panel">
66
+ <div class="result-toolbar">
67
+ <h2>Ask PDF</h2>
68
+ <span id="activeDocument">No document indexed</span>
69
+ </div>
70
+ <form id="queryForm" class="query-form">
71
+ <input id="queryInput" type="text" placeholder="Ask your question here" />
72
+ <button type="submit">Ask</button>
73
+ </form>
74
+ <div class="answer-card">
75
+ <span>Answer</span>
76
+ <div id="confidenceBadge" class="confidence-badge neutral">Confidence pending</div>
77
+ <div id="answerBox" class="answer-box">Upload a PDF first. Answer will appear here.</div>
78
+ <div id="citations" class="citation-list"></div>
79
+ </div>
80
+ </div>
81
+
82
+ <div class="panel dashboard-panel">
83
+ <div class="result-toolbar">
84
+ <h2>Retrieval Analytics</h2>
85
+ <span id="dashboardStatus">Waiting for PDF</span>
86
+ </div>
87
+ <details class="advanced-dashboard" open>
88
+ <summary>RAG Analytics Dashboard</summary>
89
+ <div class="dashboard-grid">
90
+ <section class="metric-section">
91
+ <div class="section-head">
92
+ <h2>Indexing Timeline</h2>
93
+ <span id="docStatus">Waiting</span>
94
+ </div>
95
+ <div id="indexSteps" class="timeline-list"></div>
96
+ </section>
97
+
98
+ <section class="metric-section">
99
+ <div class="section-head">
100
+ <h2>Chunking Comparison</h2>
101
+ <span id="bestChunking" class="winner-badge">Evaluation Pending</span>
102
+ </div>
103
+ <div id="chunkComparison" class="comparison-table"></div>
104
+ </section>
105
+
106
+ <section class="metric-section">
107
+ <h2>Retrieval Metrics</h2>
108
+ <div id="hybridStats" class="stat-grid">
109
+ <div><span>Vector</span><strong>-</strong></div>
110
+ <div><span>BM25</span><strong>-</strong></div>
111
+ <div><span>Merged</span><strong>-</strong></div>
112
+ <div><span>Reranked</span><strong>-</strong></div>
113
+ </div>
114
+ </section>
115
+
116
+ <section class="metric-section">
117
+ <h2>Evaluation Dashboard</h2>
118
+ <div id="evaluationCards" class="stat-grid">
119
+ <div><span>Precision</span><strong>-</strong></div>
120
+ <div><span>Recall</span><strong>-</strong></div>
121
+ <div><span>Faithfulness</span><strong>-</strong></div>
122
+ <div><span>Relevancy</span><strong>-</strong></div>
123
+ </div>
124
+ </section>
125
+
126
+ <section class="metric-section">
127
+ <h2>CRAG Retrieval Quality</h2>
128
+ <div id="cragStatus" class="crag-box neutral">
129
+ <strong>Waiting</strong>
130
+ <span>Run a query to grade retrieval quality.</span>
131
+ </div>
132
+ </section>
133
+
134
+ <section class="metric-section">
135
+ <h2>Retrieval Pipeline</h2>
136
+ <div id="pipelineViz" class="pipeline-viz"></div>
137
+ </section>
138
+
139
+ <section class="metric-section">
140
+ <h2>Hallucination Status</h2>
141
+ <div id="hallucinationCard" class="hallucination-card neutral">
142
+ <span>Answer Source Coverage</span>
143
+ <strong>-</strong>
144
+ <span>Faithfulness</span>
145
+ <strong>-</strong>
146
+ <em>Waiting for answer</em>
147
+ </div>
148
+ </section>
149
+ </div>
150
+
151
+ <section class="metric-section retrieval-section">
152
+ <div class="section-head">
153
+ <h2>Top Retrieved Chunks</h2>
154
+ <span>🧠 Parent-Child Retrieval</span>
155
+ </div>
156
+ <div id="retrievedChunks" class="retrieved-chunks"></div>
157
+ </section>
158
+
159
+ <section class="metric-section recommendation-section">
160
+ <h2>Best Chunking Recommendation</h2>
161
+ <div id="bestRecommendation" class="recommendation-box">Index a PDF to compare chunking strategies.</div>
162
+ </section>
163
+ </details>
164
+ </div>
165
+
166
+ <div class="panel history-panel">
167
+ <div class="result-toolbar">
168
+ <h2>Recent Questions</h2>
169
+ <span id="historyCount">0 saved</span>
170
+ </div>
171
+ <div id="queryHistory" class="query-history">
172
+ <span>No questions yet.</span>
173
+ </div>
174
+ </div>
175
+
176
+ <div class="panel details-panel">
177
+ <div class="result-toolbar">
178
+ <h2>Extracted Text</h2>
179
+ <button id="copyButton" type="button">Copy Text</button>
180
+ </div>
181
+ <details>
182
+ <summary>Show extracted text and page OCR details</summary>
183
+ <div id="pageDetails" class="page-details"></div>
184
+ <pre id="resultText">Upload a PDF to see extracted text here.</pre>
185
+ </details>
186
+ <form id="feedbackForm" class="feedback-form">
187
+ <select id="feedbackRating">
188
+ <option value="5">5</option>
189
+ <option value="4">4</option>
190
+ <option value="3">3</option>
191
+ <option value="2">2</option>
192
+ <option value="1">1</option>
193
+ </select>
194
+ <input id="feedbackComment" type="text" placeholder="Feedback" />
195
+ <button type="submit">Save Feedback</button>
196
+ </form>
197
+ </div>
198
+ </section>
199
+ </main>
200
+
201
+ <script src="/static/app.js?v=20260625-safe-json-errors"></script>
202
+ </body>
203
+ </html>
uploads/.gitkeep ADDED
@@ -0,0 +1 @@
 
 
1
+