Spaces:
Sleeping
Sleeping
Urmila Saini commited on
Commit ·
b9fa4a6
0
Parent(s):
add DockerFile
Browse files- .dockerignore +14 -0
- .gitignore +10 -0
- Dockerfile +26 -0
- README.md +85 -0
- app.py +203 -0
- pdf_to_text.py +757 -0
- rag_pipeline.py +1361 -0
- requirements.txt +21 -0
- static/app.js +1033 -0
- static/styles.css +998 -0
- templates/index.html +203 -0
- uploads/.gitkeep +1 -0
.dockerignore
ADDED
|
@@ -0,0 +1,14 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
.git
|
| 2 |
+
.agents
|
| 3 |
+
.dist
|
| 4 |
+
.venv
|
| 5 |
+
__pycache__
|
| 6 |
+
*.pyc
|
| 7 |
+
*.pyo
|
| 8 |
+
*.pyd
|
| 9 |
+
.env
|
| 10 |
+
data
|
| 11 |
+
uploads
|
| 12 |
+
server-*.log
|
| 13 |
+
*.sqlite
|
| 14 |
+
*.db
|
.gitignore
ADDED
|
@@ -0,0 +1,10 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
env
|
| 2 |
+
.env
|
| 3 |
+
__pycache__/
|
| 4 |
+
*.py[cod]
|
| 5 |
+
server*.log
|
| 6 |
+
data/
|
| 7 |
+
*.sqlite
|
| 8 |
+
*.db
|
| 9 |
+
uploads/*
|
| 10 |
+
!uploads/.gitkeep
|
Dockerfile
ADDED
|
@@ -0,0 +1,26 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
FROM python:3.12-slim
|
| 2 |
+
|
| 3 |
+
ENV PYTHONDONTWRITEBYTECODE=1 \
|
| 4 |
+
PYTHONUNBUFFERED=1 \
|
| 5 |
+
PIP_NO_CACHE_DIR=1
|
| 6 |
+
|
| 7 |
+
WORKDIR /app
|
| 8 |
+
|
| 9 |
+
RUN apt-get update \
|
| 10 |
+
&& apt-get install -y --no-install-recommends \
|
| 11 |
+
build-essential \
|
| 12 |
+
libgl1 \
|
| 13 |
+
libglib2.0-0 \
|
| 14 |
+
&& rm -rf /var/lib/apt/lists/*
|
| 15 |
+
|
| 16 |
+
COPY requirements.txt .
|
| 17 |
+
RUN python -m pip install --upgrade pip \
|
| 18 |
+
&& python -m pip install -r requirements.txt
|
| 19 |
+
|
| 20 |
+
COPY . .
|
| 21 |
+
|
| 22 |
+
RUN mkdir -p data uploads
|
| 23 |
+
|
| 24 |
+
EXPOSE 7860
|
| 25 |
+
|
| 26 |
+
CMD ["uvicorn", "app:app", "--host", "0.0.0.0", "--port", "7860"]
|
README.md
ADDED
|
@@ -0,0 +1,85 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
# SmartNotes PDF OCR
|
| 2 |
+
|
| 3 |
+
## Project Metadata
|
| 4 |
+
|
| 5 |
+
| Field | Value |
|
| 6 |
+
| --- | --- |
|
| 7 |
+
| Project | SmartNotes AI |
|
| 8 |
+
| App type | FastAPI web app for PDF OCR and RAG-based question answering |
|
| 9 |
+
| Runtime | Python 3.12 |
|
| 10 |
+
| Default URL | `http://127.0.0.1:8000` |
|
| 11 |
+
| Main entrypoint | `app.py` |
|
| 12 |
+
| UI files | `templates/index.html`, `static/app.js`, `static/styles.css` |
|
| 13 |
+
| OCR pipeline | PyMuPDF, OpenCV, Gemini Vision OCR, TrOCR fallback |
|
| 14 |
+
| RAG pipeline | Text cleaning, metadata extraction, parent/child chunks, embeddings, BM25, reranking, citations |
|
| 15 |
+
| Storage | SQLite local fallback in `data/smartnotes.sqlite`; optional PostgreSQL via `POSTGRES_DSN` |
|
| 16 |
+
| Vector store | Local fallback; optional Qdrant via `QDRANT_URL` |
|
| 17 |
+
| Docker | `Dockerfile` exposes port `8000` |
|
| 18 |
+
| Secrets | Keep API keys in `.env`; do not commit `.env` |
|
| 19 |
+
|
| 20 |
+
FastAPI app for PDF-to-text extraction using this route:
|
| 21 |
+
|
| 22 |
+
1. Try PyMuPDF direct text extraction.
|
| 23 |
+
2. If text exists, skip OCR, clean text, return final text.
|
| 24 |
+
3. If no direct text exists, convert PDF pages to images with PyMuPDF.
|
| 25 |
+
4. Preprocess images with OpenCV.
|
| 26 |
+
5. For 20 pages or fewer, run Gemini Vision OCR.
|
| 27 |
+
6. For more than 20 pages, run TrOCR first, then send low-confidence pages to Gemini Vision.
|
| 28 |
+
7. Merge, clean, and return final text.
|
| 29 |
+
|
| 30 |
+
The UI uses `/api/pdf-to-text-stream`, so extracted text appears page by page instead of waiting for the full PDF to finish.
|
| 31 |
+
|
| 32 |
+
After extraction, the UI automatically calls `/api/index-text-stream` and runs the locked RAG workflow:
|
| 33 |
+
|
| 34 |
+
- status `processing`
|
| 35 |
+
- cleaning and metadata
|
| 36 |
+
- parent chunks
|
| 37 |
+
- fixed, recursive, and semantic child chunks
|
| 38 |
+
- chunk recommendation
|
| 39 |
+
- embeddings and embedding evaluation
|
| 40 |
+
- Qdrant store when configured, local vector fallback otherwise
|
| 41 |
+
- BM25 index
|
| 42 |
+
- status `indexed`
|
| 43 |
+
- query validation, rewrite, hybrid retrieval, top-50 candidates, reranking, grading
|
| 44 |
+
- parent fetch, duplicate parent removal, context fitting, answer, citations, retrieved chunk view, logs, feedback
|
| 45 |
+
|
| 46 |
+
Useful optional env values:
|
| 47 |
+
|
| 48 |
+
```text
|
| 49 |
+
POSTGRES_DSN=postgresql://user:password@localhost:5432/smartnotes
|
| 50 |
+
QDRANT_URL=http://localhost:6333
|
| 51 |
+
QDRANT_API_KEY=
|
| 52 |
+
EMBEDDING_MODEL=BAAI/bge-small-en-v1.5
|
| 53 |
+
RERANKER_MODEL=BAAI/bge-reranker-base
|
| 54 |
+
USE_LOCAL_EMBEDDING_MODEL=1
|
| 55 |
+
USE_LOCAL_RERANKER_MODEL=1
|
| 56 |
+
ANSWER_PROVIDER=gemini
|
| 57 |
+
ANSWER_MODEL=gemini-2.5-flash
|
| 58 |
+
```
|
| 59 |
+
|
| 60 |
+
## Setup
|
| 61 |
+
|
| 62 |
+
```powershell
|
| 63 |
+
python -m venv .venv
|
| 64 |
+
.\.venv\Scripts\Activate.ps1
|
| 65 |
+
pip install -r requirements.txt
|
| 66 |
+
$env:GEMINI_API_KEY="your_api_key"
|
| 67 |
+
uvicorn app:app --reload
|
| 68 |
+
```
|
| 69 |
+
|
| 70 |
+
Open `http://127.0.0.1:8000`.
|
| 71 |
+
|
| 72 |
+
You can also put these values in `env` or `.env`:
|
| 73 |
+
|
| 74 |
+
```text
|
| 75 |
+
GEMINI_API_KEY_1=your_first_api_key
|
| 76 |
+
GEMINI_API_KEY_2=your_second_api_key
|
| 77 |
+
GEMINI_API_KEY_3=your_third_api_key
|
| 78 |
+
GEMINI_API_KEY_4=your_fourth_api_key
|
| 79 |
+
GEMINI_MODELS=gemini-3.5-flash,gemini-2.5-flash,gemini-2.5-flash-lite
|
| 80 |
+
GEMINI_MAX_RETRIES=2
|
| 81 |
+
GEMINI_RETRY_DELAY=1.5
|
| 82 |
+
GEMINI_KEY_QUOTA_COOLDOWN=300
|
| 83 |
+
```
|
| 84 |
+
|
| 85 |
+
If one Gemini key returns a quota or rate-limit error, the app skips it temporarily and tries the next configured key. Successful calls advance through the keys in order, so `_1`, `_2`, `_3`, and `_4` are used in rotation. If Gemini returns temporary `503 UNAVAILABLE` high-demand errors, the app retries and then tries the next model in `GEMINI_MODELS`.
|
app.py
ADDED
|
@@ -0,0 +1,203 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
from __future__ import annotations
|
| 2 |
+
|
| 3 |
+
import json
|
| 4 |
+
from pathlib import Path
|
| 5 |
+
|
| 6 |
+
from fastapi import BackgroundTasks, FastAPI, File, HTTPException, Request, UploadFile
|
| 7 |
+
from fastapi.responses import HTMLResponse, StreamingResponse
|
| 8 |
+
from fastapi.staticfiles import StaticFiles
|
| 9 |
+
from fastapi.templating import Jinja2Templates
|
| 10 |
+
from pydantic import BaseModel
|
| 11 |
+
|
| 12 |
+
from pdf_to_text import ExtractionResult, extract_uploaded_pdf, stream_uploaded_pdf
|
| 13 |
+
from rag_pipeline import SmartNotesRAG
|
| 14 |
+
|
| 15 |
+
|
| 16 |
+
BASE_DIR = Path(__file__).resolve().parent
|
| 17 |
+
PROJECT_METADATA = {
|
| 18 |
+
"name": "SmartNotes AI",
|
| 19 |
+
"title": "SmartNotes PDF OCR",
|
| 20 |
+
"version": "1.0.0",
|
| 21 |
+
"description": "FastAPI app for PDF OCR, text extraction, and RAG-based question answering.",
|
| 22 |
+
"runtime": "Python 3.12",
|
| 23 |
+
"framework": "FastAPI",
|
| 24 |
+
"default_port": 8000,
|
| 25 |
+
"entrypoint": "app.py",
|
| 26 |
+
"storage": "SQLite local fallback with optional PostgreSQL",
|
| 27 |
+
"vector_store": "Local fallback with optional Qdrant",
|
| 28 |
+
"features": [
|
| 29 |
+
"PDF direct text extraction",
|
| 30 |
+
"OCR fallback with OpenCV, Gemini Vision OCR, and TrOCR",
|
| 31 |
+
"RAG indexing with parent and child chunks",
|
| 32 |
+
"Hybrid retrieval with embeddings and BM25",
|
| 33 |
+
"Reranked answers with citations",
|
| 34 |
+
],
|
| 35 |
+
}
|
| 36 |
+
|
| 37 |
+
app = FastAPI(
|
| 38 |
+
title=PROJECT_METADATA["title"],
|
| 39 |
+
description=PROJECT_METADATA["description"],
|
| 40 |
+
version=PROJECT_METADATA["version"],
|
| 41 |
+
)
|
| 42 |
+
app.mount("/static", StaticFiles(directory=BASE_DIR / "static"), name="static")
|
| 43 |
+
templates = Jinja2Templates(directory=BASE_DIR / "templates")
|
| 44 |
+
rag = SmartNotesRAG()
|
| 45 |
+
|
| 46 |
+
|
| 47 |
+
class IndexTextRequest(BaseModel):
|
| 48 |
+
file_name: str
|
| 49 |
+
text: str
|
| 50 |
+
|
| 51 |
+
|
| 52 |
+
class QueryRequest(BaseModel):
|
| 53 |
+
document_id: str
|
| 54 |
+
query: str
|
| 55 |
+
|
| 56 |
+
|
| 57 |
+
class FeedbackRequest(BaseModel):
|
| 58 |
+
document_id: str
|
| 59 |
+
query: str
|
| 60 |
+
rating: int
|
| 61 |
+
comment: str = ""
|
| 62 |
+
|
| 63 |
+
|
| 64 |
+
@app.get("/", response_class=HTMLResponse)
|
| 65 |
+
async def home(request: Request):
|
| 66 |
+
return templates.TemplateResponse(request, "index.html")
|
| 67 |
+
|
| 68 |
+
|
| 69 |
+
@app.get("/api/metadata")
|
| 70 |
+
async def project_metadata():
|
| 71 |
+
return PROJECT_METADATA
|
| 72 |
+
|
| 73 |
+
|
| 74 |
+
@app.post("/api/pdf-to-text")
|
| 75 |
+
async def pdf_to_text(file: UploadFile = File(...)):
|
| 76 |
+
if not _looks_like_pdf(file.filename, file.content_type):
|
| 77 |
+
raise HTTPException(status_code=400, detail="Please upload a PDF file.")
|
| 78 |
+
|
| 79 |
+
file_bytes = await file.read()
|
| 80 |
+
if not file_bytes:
|
| 81 |
+
raise HTTPException(status_code=400, detail="Uploaded PDF is empty.")
|
| 82 |
+
|
| 83 |
+
try:
|
| 84 |
+
result = extract_uploaded_pdf(file.filename, file_bytes)
|
| 85 |
+
except RuntimeError as exc:
|
| 86 |
+
raise HTTPException(status_code=500, detail=str(exc)) from exc
|
| 87 |
+
except Exception as exc:
|
| 88 |
+
raise HTTPException(status_code=500, detail=f"PDF processing failed: {exc}") from exc
|
| 89 |
+
|
| 90 |
+
return _result_payload(result, original_file=file.filename)
|
| 91 |
+
|
| 92 |
+
|
| 93 |
+
@app.post("/api/pdf-to-text-stream")
|
| 94 |
+
async def pdf_to_text_stream(file: UploadFile = File(...)):
|
| 95 |
+
if not _looks_like_pdf(file.filename, file.content_type):
|
| 96 |
+
raise HTTPException(status_code=400, detail="Please upload a PDF file.")
|
| 97 |
+
|
| 98 |
+
original_file = file.filename
|
| 99 |
+
file_bytes = await file.read()
|
| 100 |
+
if not file_bytes:
|
| 101 |
+
raise HTTPException(status_code=400, detail="Uploaded PDF is empty.")
|
| 102 |
+
|
| 103 |
+
def events():
|
| 104 |
+
try:
|
| 105 |
+
for event in stream_uploaded_pdf(original_file, file_bytes):
|
| 106 |
+
yield json.dumps(event, ensure_ascii=True) + "\n"
|
| 107 |
+
except Exception as exc:
|
| 108 |
+
yield json.dumps(
|
| 109 |
+
{
|
| 110 |
+
"type": "error",
|
| 111 |
+
"message": f"PDF processing failed: {exc}",
|
| 112 |
+
},
|
| 113 |
+
ensure_ascii=True,
|
| 114 |
+
) + "\n"
|
| 115 |
+
|
| 116 |
+
return StreamingResponse(events(), media_type="application/x-ndjson")
|
| 117 |
+
|
| 118 |
+
|
| 119 |
+
@app.post("/api/index-text-stream")
|
| 120 |
+
async def index_text_stream(payload: IndexTextRequest):
|
| 121 |
+
if not payload.text.strip():
|
| 122 |
+
raise HTTPException(status_code=400, detail="Extracted text is empty.")
|
| 123 |
+
|
| 124 |
+
def events():
|
| 125 |
+
try:
|
| 126 |
+
for event in rag.index_text_stream(payload.file_name, payload.text):
|
| 127 |
+
yield json.dumps(event, ensure_ascii=True) + "\n"
|
| 128 |
+
except Exception as exc:
|
| 129 |
+
yield json.dumps(
|
| 130 |
+
{
|
| 131 |
+
"type": "error",
|
| 132 |
+
"message": f"Indexing failed: {exc}",
|
| 133 |
+
},
|
| 134 |
+
ensure_ascii=True,
|
| 135 |
+
) + "\n"
|
| 136 |
+
|
| 137 |
+
return StreamingResponse(events(), media_type="application/x-ndjson")
|
| 138 |
+
|
| 139 |
+
|
| 140 |
+
@app.post("/api/index-text-background")
|
| 141 |
+
async def index_text_background(payload: IndexTextRequest, background_tasks: BackgroundTasks):
|
| 142 |
+
if not payload.text.strip():
|
| 143 |
+
raise HTTPException(status_code=400, detail="Extracted text is empty.")
|
| 144 |
+
|
| 145 |
+
def run_indexing() -> None:
|
| 146 |
+
for _ in rag.index_text_stream(payload.file_name, payload.text):
|
| 147 |
+
pass
|
| 148 |
+
|
| 149 |
+
background_tasks.add_task(run_indexing)
|
| 150 |
+
return {"status": "queued"}
|
| 151 |
+
|
| 152 |
+
|
| 153 |
+
@app.post("/api/query")
|
| 154 |
+
async def query_document(payload: QueryRequest):
|
| 155 |
+
return rag.query(payload.document_id, payload.query)
|
| 156 |
+
|
| 157 |
+
|
| 158 |
+
@app.post("/api/feedback")
|
| 159 |
+
async def save_feedback(payload: FeedbackRequest):
|
| 160 |
+
rating = max(1, min(payload.rating, 5))
|
| 161 |
+
return rag.save_feedback(payload.document_id, payload.query, rating, payload.comment)
|
| 162 |
+
|
| 163 |
+
|
| 164 |
+
@app.get("/api/documents/{document_id}")
|
| 165 |
+
async def get_document(document_id: str):
|
| 166 |
+
document = rag.repository.get_document(document_id)
|
| 167 |
+
if not document:
|
| 168 |
+
raise HTTPException(status_code=404, detail="Document not found.")
|
| 169 |
+
document.pop("cleaned_text", None)
|
| 170 |
+
return document
|
| 171 |
+
|
| 172 |
+
|
| 173 |
+
@app.get("/health")
|
| 174 |
+
async def health():
|
| 175 |
+
return {"status": "ok"}
|
| 176 |
+
|
| 177 |
+
|
| 178 |
+
def _result_payload(result: ExtractionResult, original_file: str) -> dict:
|
| 179 |
+
return {
|
| 180 |
+
"file": original_file,
|
| 181 |
+
"text": result.text,
|
| 182 |
+
"route": result.route,
|
| 183 |
+
"page_count": result.page_count,
|
| 184 |
+
"direct_text_found": result.direct_text_found,
|
| 185 |
+
"warnings": result.warnings,
|
| 186 |
+
"pages": [
|
| 187 |
+
{
|
| 188 |
+
"page_number": page.page_number,
|
| 189 |
+
"engine": page.engine,
|
| 190 |
+
"confidence": page.confidence,
|
| 191 |
+
"text_length": len(page.text),
|
| 192 |
+
}
|
| 193 |
+
for page in result.pages
|
| 194 |
+
],
|
| 195 |
+
}
|
| 196 |
+
|
| 197 |
+
|
| 198 |
+
def _looks_like_pdf(file_name: str | None, content_type: str | None) -> bool:
|
| 199 |
+
if content_type in {"application/pdf", "application/x-pdf"}:
|
| 200 |
+
return True
|
| 201 |
+
if file_name and Path(file_name).suffix.lower() == ".pdf":
|
| 202 |
+
return True
|
| 203 |
+
return False
|
pdf_to_text.py
ADDED
|
@@ -0,0 +1,757 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
from __future__ import annotations
|
| 2 |
+
|
| 3 |
+
import io
|
| 4 |
+
import os
|
| 5 |
+
import re
|
| 6 |
+
import tempfile
|
| 7 |
+
import time
|
| 8 |
+
import threading
|
| 9 |
+
from concurrent.futures import ThreadPoolExecutor, as_completed
|
| 10 |
+
from dataclasses import dataclass, field
|
| 11 |
+
from pathlib import Path
|
| 12 |
+
from typing import Iterable
|
| 13 |
+
|
| 14 |
+
|
| 15 |
+
DIRECT_TEXT_MIN_CHARS = 30
|
| 16 |
+
SHORT_PDF_PAGE_LIMIT = 20
|
| 17 |
+
TROCR_CONFIDENCE_THRESHOLD = 0.78
|
| 18 |
+
DEFAULT_GEMINI_MODELS = (
|
| 19 |
+
"gemini-2.5-flash-lite",
|
| 20 |
+
"gemini-2.5-flash",
|
| 21 |
+
)
|
| 22 |
+
|
| 23 |
+
|
| 24 |
+
@dataclass
|
| 25 |
+
class PageResult:
|
| 26 |
+
page_number: int
|
| 27 |
+
engine: str
|
| 28 |
+
text: str
|
| 29 |
+
confidence: float | None = None
|
| 30 |
+
|
| 31 |
+
|
| 32 |
+
@dataclass
|
| 33 |
+
class ExtractionResult:
|
| 34 |
+
text: str
|
| 35 |
+
route: str
|
| 36 |
+
page_count: int
|
| 37 |
+
direct_text_found: bool
|
| 38 |
+
pages: list[PageResult] = field(default_factory=list)
|
| 39 |
+
warnings: list[str] = field(default_factory=list)
|
| 40 |
+
|
| 41 |
+
|
| 42 |
+
class PdfTextExtractor:
|
| 43 |
+
"""Implements the requested PDF-to-text decision flow."""
|
| 44 |
+
|
| 45 |
+
def __init__(
|
| 46 |
+
self,
|
| 47 |
+
gemini_model: str | None = None,
|
| 48 |
+
trocr_model: str = "microsoft/trocr-base-printed",
|
| 49 |
+
short_pdf_page_limit: int = SHORT_PDF_PAGE_LIMIT,
|
| 50 |
+
trocr_confidence_threshold: float = TROCR_CONFIDENCE_THRESHOLD,
|
| 51 |
+
) -> None:
|
| 52 |
+
self.trocr_model_name = trocr_model
|
| 53 |
+
self.short_pdf_page_limit = short_pdf_page_limit
|
| 54 |
+
self.trocr_confidence_threshold = trocr_confidence_threshold
|
| 55 |
+
self._gemini_client = None
|
| 56 |
+
self._gemini_clients = {}
|
| 57 |
+
self._gemini_key_cursor = 0
|
| 58 |
+
self._gemini_key_lock = threading.Lock()
|
| 59 |
+
self._gemini_quota_blocked_until: dict[str, float] = {}
|
| 60 |
+
self._trocr_processor = None
|
| 61 |
+
self._trocr_model = None
|
| 62 |
+
self._load_local_env()
|
| 63 |
+
self.gemini_models = self._gemini_model_candidates(gemini_model)
|
| 64 |
+
self.gemini_max_retries = min(
|
| 65 |
+
self._env_int("OCR_GEMINI_MAX_RETRIES", "GEMINI_MAX_RETRIES", default=1, minimum=1),
|
| 66 |
+
4,
|
| 67 |
+
)
|
| 68 |
+
self.gemini_retry_delay = min(
|
| 69 |
+
self._env_float("OCR_GEMINI_RETRY_DELAY", "GEMINI_RETRY_DELAY", default=0.5, minimum=0.0),
|
| 70 |
+
3.0,
|
| 71 |
+
)
|
| 72 |
+
self.gemini_timeout_ms = int(
|
| 73 |
+
min(
|
| 74 |
+
self._env_float("OCR_GEMINI_TIMEOUT", "GEMINI_TIMEOUT", default=20.0, minimum=5.0),
|
| 75 |
+
120.0,
|
| 76 |
+
)
|
| 77 |
+
* 1000
|
| 78 |
+
)
|
| 79 |
+
self.gemini_key_quota_cooldown = min(
|
| 80 |
+
self._env_float("GEMINI_KEY_QUOTA_COOLDOWN", default=300.0, minimum=0.0),
|
| 81 |
+
86400.0,
|
| 82 |
+
)
|
| 83 |
+
self.gemini_concurrency = min(
|
| 84 |
+
self._env_int("OCR_GEMINI_CONCURRENCY", default=3, minimum=1),
|
| 85 |
+
5,
|
| 86 |
+
)
|
| 87 |
+
self.render_scale = self._env_float("OCR_RENDER_SCALE", default=1.25, minimum=1.0)
|
| 88 |
+
self.fast_ocr = os.getenv("OCR_FAST_MODE", "1") != "0"
|
| 89 |
+
|
| 90 |
+
def extract(self, pdf_path: str | Path) -> ExtractionResult:
|
| 91 |
+
path = Path(pdf_path)
|
| 92 |
+
document = self._open_document(path)
|
| 93 |
+
try:
|
| 94 |
+
page_count = document.page_count
|
| 95 |
+
|
| 96 |
+
direct_text = self._extract_direct_text(document)
|
| 97 |
+
if self._has_enough_text(direct_text):
|
| 98 |
+
return ExtractionResult(
|
| 99 |
+
text=self._clean_text_only(direct_text),
|
| 100 |
+
route="PyMuPDF direct text extraction",
|
| 101 |
+
page_count=page_count,
|
| 102 |
+
direct_text_found=True,
|
| 103 |
+
)
|
| 104 |
+
|
| 105 |
+
images = self._convert_pages_to_preprocessed_images(document)
|
| 106 |
+
if page_count <= self.short_pdf_page_limit:
|
| 107 |
+
pages = []
|
| 108 |
+
warnings = []
|
| 109 |
+
for index, image_bytes in images:
|
| 110 |
+
text, warning = self._safe_gemini_ocr(image_bytes, index)
|
| 111 |
+
if warning:
|
| 112 |
+
warnings.append(warning)
|
| 113 |
+
pages.append(
|
| 114 |
+
PageResult(
|
| 115 |
+
page_number=index,
|
| 116 |
+
engine="Gemini Vision OCR" if text else "Skipped non-text page",
|
| 117 |
+
text=text,
|
| 118 |
+
)
|
| 119 |
+
)
|
| 120 |
+
if not pages:
|
| 121 |
+
warnings.append("No readable text pages were processed.")
|
| 122 |
+
result = self._finalize_ocr_result(
|
| 123 |
+
route="PyMuPDF images -> OpenCV preprocess -> Gemini Vision OCR",
|
| 124 |
+
page_count=page_count,
|
| 125 |
+
direct_text_found=False,
|
| 126 |
+
pages=pages,
|
| 127 |
+
)
|
| 128 |
+
result.warnings.extend(warnings)
|
| 129 |
+
return result
|
| 130 |
+
|
| 131 |
+
pages = []
|
| 132 |
+
warnings = []
|
| 133 |
+
for page_number, image_bytes in images:
|
| 134 |
+
try:
|
| 135 |
+
trocr_text, confidence = self._trocr_ocr(image_bytes)
|
| 136 |
+
except Exception as exc:
|
| 137 |
+
trocr_text = ""
|
| 138 |
+
confidence = 0.0
|
| 139 |
+
warnings.append(f"Page {page_number} skipped: TrOCR failed ({exc}).")
|
| 140 |
+
if confidence < self.trocr_confidence_threshold:
|
| 141 |
+
gemini_text, warning = self._safe_gemini_ocr(image_bytes, page_number)
|
| 142 |
+
if warning:
|
| 143 |
+
warnings.append(warning)
|
| 144 |
+
pages.append(
|
| 145 |
+
PageResult(
|
| 146 |
+
page_number=page_number,
|
| 147 |
+
engine="TrOCR low confidence -> Gemini Vision OCR" if gemini_text else "Skipped non-text page",
|
| 148 |
+
text=gemini_text,
|
| 149 |
+
confidence=confidence,
|
| 150 |
+
)
|
| 151 |
+
)
|
| 152 |
+
else:
|
| 153 |
+
trocr_text = self._clean_text_only(trocr_text)
|
| 154 |
+
pages.append(
|
| 155 |
+
PageResult(
|
| 156 |
+
page_number=page_number,
|
| 157 |
+
engine="TrOCR" if trocr_text else "Skipped non-text page",
|
| 158 |
+
text=trocr_text,
|
| 159 |
+
confidence=confidence,
|
| 160 |
+
)
|
| 161 |
+
)
|
| 162 |
+
|
| 163 |
+
if not pages:
|
| 164 |
+
warnings.append("No pages were processed.")
|
| 165 |
+
|
| 166 |
+
result = self._finalize_ocr_result(
|
| 167 |
+
route="PyMuPDF images -> OpenCV preprocess -> TrOCR -> Gemini low-confidence fallback",
|
| 168 |
+
page_count=page_count,
|
| 169 |
+
direct_text_found=False,
|
| 170 |
+
pages=pages,
|
| 171 |
+
)
|
| 172 |
+
result.warnings.extend(warnings)
|
| 173 |
+
return result
|
| 174 |
+
finally:
|
| 175 |
+
document.close()
|
| 176 |
+
|
| 177 |
+
def stream_extract(self, pdf_path: str | Path) -> Iterable[dict]:
|
| 178 |
+
path = Path(pdf_path)
|
| 179 |
+
document = self._open_document(path)
|
| 180 |
+
try:
|
| 181 |
+
page_count = document.page_count
|
| 182 |
+
yield {"type": "start", "page_count": page_count}
|
| 183 |
+
|
| 184 |
+
short_pdf_ocr_jobs = []
|
| 185 |
+
for page_number, page in enumerate(document, start=1):
|
| 186 |
+
direct_text = self._clean_text_only(page.get_text("text"))
|
| 187 |
+
if self._has_enough_text(direct_text, min_chars=5):
|
| 188 |
+
yield {
|
| 189 |
+
"type": "page",
|
| 190 |
+
"page_number": page_number,
|
| 191 |
+
"page_count": page_count,
|
| 192 |
+
"engine": "PyMuPDF direct text extraction",
|
| 193 |
+
"route": "PyMuPDF direct text extraction",
|
| 194 |
+
"direct_text_found": True,
|
| 195 |
+
"confidence": None,
|
| 196 |
+
"text": direct_text,
|
| 197 |
+
}
|
| 198 |
+
continue
|
| 199 |
+
|
| 200 |
+
try:
|
| 201 |
+
image_bytes = self._convert_page_to_preprocessed_image(page)
|
| 202 |
+
except Exception as exc:
|
| 203 |
+
yield self._skipped_page_event(page_number, page_count, f"Image preprocessing failed: {exc}")
|
| 204 |
+
continue
|
| 205 |
+
|
| 206 |
+
if page_count <= self.short_pdf_page_limit:
|
| 207 |
+
short_pdf_ocr_jobs.append((page_number, image_bytes))
|
| 208 |
+
continue
|
| 209 |
+
|
| 210 |
+
try:
|
| 211 |
+
trocr_text, confidence = self._trocr_ocr(image_bytes)
|
| 212 |
+
except Exception as exc:
|
| 213 |
+
trocr_text = ""
|
| 214 |
+
confidence = 0.0
|
| 215 |
+
yield self._skipped_page_event(page_number, page_count, f"TrOCR failed: {exc}", confidence)
|
| 216 |
+
continue
|
| 217 |
+
if confidence < self.trocr_confidence_threshold:
|
| 218 |
+
ocr_text, warning = self._safe_gemini_ocr(image_bytes, page_number)
|
| 219 |
+
yield {
|
| 220 |
+
"type": "page",
|
| 221 |
+
"page_number": page_number,
|
| 222 |
+
"page_count": page_count,
|
| 223 |
+
"engine": "TrOCR low confidence -> Gemini Vision OCR" if ocr_text else "Skipped non-text page",
|
| 224 |
+
"route": "PyMuPDF image -> OpenCV preprocess -> TrOCR -> Gemini fallback",
|
| 225 |
+
"direct_text_found": False,
|
| 226 |
+
"confidence": confidence,
|
| 227 |
+
"text": ocr_text,
|
| 228 |
+
"warning": warning,
|
| 229 |
+
}
|
| 230 |
+
else:
|
| 231 |
+
trocr_text = self._clean_text_only(trocr_text)
|
| 232 |
+
if not self._has_enough_text(trocr_text, min_chars=2):
|
| 233 |
+
yield self._skipped_page_event(page_number, page_count, "No readable text found.", confidence)
|
| 234 |
+
continue
|
| 235 |
+
yield {
|
| 236 |
+
"type": "page",
|
| 237 |
+
"page_number": page_number,
|
| 238 |
+
"page_count": page_count,
|
| 239 |
+
"engine": "TrOCR",
|
| 240 |
+
"route": "PyMuPDF image -> OpenCV preprocess -> TrOCR",
|
| 241 |
+
"direct_text_found": False,
|
| 242 |
+
"confidence": confidence,
|
| 243 |
+
"text": self._clean_text(trocr_text),
|
| 244 |
+
}
|
| 245 |
+
|
| 246 |
+
if short_pdf_ocr_jobs:
|
| 247 |
+
yield from self._stream_gemini_ocr_jobs(short_pdf_ocr_jobs, page_count)
|
| 248 |
+
|
| 249 |
+
yield {"type": "done", "page_count": page_count}
|
| 250 |
+
finally:
|
| 251 |
+
document.close()
|
| 252 |
+
|
| 253 |
+
def _open_document(self, pdf_path: Path):
|
| 254 |
+
try:
|
| 255 |
+
import fitz
|
| 256 |
+
except ImportError as exc:
|
| 257 |
+
raise RuntimeError("PyMuPDF is required. Install it with: pip install pymupdf") from exc
|
| 258 |
+
|
| 259 |
+
try:
|
| 260 |
+
document = fitz.open(pdf_path)
|
| 261 |
+
if document.is_encrypted:
|
| 262 |
+
raise RuntimeError("This PDF is password-protected. Please upload an unlocked PDF.")
|
| 263 |
+
if document.page_count == 0:
|
| 264 |
+
raise RuntimeError("This PDF has no pages.")
|
| 265 |
+
return document
|
| 266 |
+
except RuntimeError:
|
| 267 |
+
raise
|
| 268 |
+
except Exception as exc:
|
| 269 |
+
raise RuntimeError(f"Could not open PDF. It may be corrupted or unsupported: {exc}") from exc
|
| 270 |
+
|
| 271 |
+
def _extract_direct_text(self, document) -> str:
|
| 272 |
+
chunks = []
|
| 273 |
+
for page in document:
|
| 274 |
+
chunks.append(page.get_text("text"))
|
| 275 |
+
return "\n\n".join(chunks)
|
| 276 |
+
|
| 277 |
+
def _has_enough_text(self, text: str, min_chars: int = DIRECT_TEXT_MIN_CHARS) -> bool:
|
| 278 |
+
normalized = re.sub(r"\s+", "", text or "")
|
| 279 |
+
return len(normalized) >= min_chars
|
| 280 |
+
|
| 281 |
+
def _convert_pages_to_preprocessed_images(self, document) -> list[tuple[int, bytes]]:
|
| 282 |
+
images = []
|
| 283 |
+
for index, page in enumerate(document, start=1):
|
| 284 |
+
try:
|
| 285 |
+
images.append((index, self._convert_page_to_preprocessed_image(page)))
|
| 286 |
+
except Exception:
|
| 287 |
+
continue
|
| 288 |
+
return images
|
| 289 |
+
|
| 290 |
+
def _convert_page_to_preprocessed_image(self, page) -> bytes:
|
| 291 |
+
try:
|
| 292 |
+
import cv2
|
| 293 |
+
import fitz
|
| 294 |
+
import numpy as np
|
| 295 |
+
from PIL import Image
|
| 296 |
+
except ImportError as exc:
|
| 297 |
+
raise RuntimeError(
|
| 298 |
+
"OCR image preprocessing needs opencv-python, numpy, Pillow, and PyMuPDF."
|
| 299 |
+
) from exc
|
| 300 |
+
|
| 301 |
+
matrix = fitz.Matrix(self.render_scale, self.render_scale)
|
| 302 |
+
pixmap = page.get_pixmap(matrix=matrix, alpha=False)
|
| 303 |
+
pil_image = Image.open(io.BytesIO(pixmap.tobytes("png"))).convert("RGB")
|
| 304 |
+
array = np.array(pil_image)
|
| 305 |
+
gray = cv2.cvtColor(array, cv2.COLOR_RGB2GRAY)
|
| 306 |
+
if self.fast_ocr:
|
| 307 |
+
_, thresholded = cv2.threshold(gray, 0, 255, cv2.THRESH_BINARY + cv2.THRESH_OTSU)
|
| 308 |
+
else:
|
| 309 |
+
denoised = cv2.fastNlMeansDenoising(gray, h=10)
|
| 310 |
+
thresholded = cv2.adaptiveThreshold(
|
| 311 |
+
denoised,
|
| 312 |
+
255,
|
| 313 |
+
cv2.ADAPTIVE_THRESH_GAUSSIAN_C,
|
| 314 |
+
cv2.THRESH_BINARY,
|
| 315 |
+
31,
|
| 316 |
+
11,
|
| 317 |
+
)
|
| 318 |
+
output = Image.fromarray(thresholded).convert("RGB")
|
| 319 |
+
buffer = io.BytesIO()
|
| 320 |
+
output.save(buffer, format="JPEG", quality=72, optimize=True)
|
| 321 |
+
return buffer.getvalue()
|
| 322 |
+
|
| 323 |
+
def _gemini_ocr(self, image_bytes: bytes) -> str:
|
| 324 |
+
try:
|
| 325 |
+
from google import genai
|
| 326 |
+
from google.genai import types
|
| 327 |
+
except ImportError as exc:
|
| 328 |
+
raise RuntimeError("Gemini OCR requires: pip install google-genai") from exc
|
| 329 |
+
|
| 330 |
+
api_keys = self._gemini_api_keys()
|
| 331 |
+
if not api_keys:
|
| 332 |
+
raise RuntimeError("Set GEMINI_API_KEY before using Gemini Vision OCR.")
|
| 333 |
+
|
| 334 |
+
prompt = (
|
| 335 |
+
"Extract all readable text from this preprocessed PDF page image. "
|
| 336 |
+
"Preserve natural reading order, headings, bullet points, tables as plain text, "
|
| 337 |
+
"and do not add commentary. Ignore photos, diagrams, icons, borders, handwriting-like noise, "
|
| 338 |
+
"and decorative/non-text visual content. Do not describe images. If there is no readable text, "
|
| 339 |
+
"return an empty response."
|
| 340 |
+
)
|
| 341 |
+
last_error: Exception | None = None
|
| 342 |
+
attempted_keys = 0
|
| 343 |
+
quota_blocked_keys = 0
|
| 344 |
+
for api_key in self._gemini_key_attempt_order(api_keys):
|
| 345 |
+
attempted_keys += 1
|
| 346 |
+
client = self._gemini_clients.get(api_key)
|
| 347 |
+
if client is None:
|
| 348 |
+
client = genai.Client(
|
| 349 |
+
api_key=api_key,
|
| 350 |
+
http_options=types.HttpOptions(timeout=self.gemini_timeout_ms),
|
| 351 |
+
)
|
| 352 |
+
self._gemini_clients[api_key] = client
|
| 353 |
+
|
| 354 |
+
quota_hit = False
|
| 355 |
+
for model in self.gemini_models:
|
| 356 |
+
for attempt in range(1, self.gemini_max_retries + 1):
|
| 357 |
+
try:
|
| 358 |
+
response = client.models.generate_content(
|
| 359 |
+
model=model,
|
| 360 |
+
contents=[
|
| 361 |
+
types.Part.from_bytes(data=image_bytes, mime_type="image/jpeg"),
|
| 362 |
+
prompt,
|
| 363 |
+
],
|
| 364 |
+
)
|
| 365 |
+
self._advance_gemini_key_cursor(api_keys, api_key)
|
| 366 |
+
return (getattr(response, "text", None) or "").strip()
|
| 367 |
+
except Exception as exc:
|
| 368 |
+
last_error = exc
|
| 369 |
+
if self._is_quota_error(exc):
|
| 370 |
+
self._mark_gemini_key_quota_exhausted(api_keys, api_key)
|
| 371 |
+
quota_blocked_keys += 1
|
| 372 |
+
quota_hit = True
|
| 373 |
+
break
|
| 374 |
+
|
| 375 |
+
if not self._is_retryable_gemini_error(exc):
|
| 376 |
+
break
|
| 377 |
+
|
| 378 |
+
if attempt < self.gemini_max_retries:
|
| 379 |
+
time.sleep(self.gemini_retry_delay * attempt)
|
| 380 |
+
|
| 381 |
+
if quota_hit:
|
| 382 |
+
break
|
| 383 |
+
|
| 384 |
+
models = ", ".join(self.gemini_models)
|
| 385 |
+
raise RuntimeError(
|
| 386 |
+
"Gemini OCR is temporarily unavailable after retries. "
|
| 387 |
+
f"Tried {attempted_keys} active API key(s) out of {len(api_keys)} configured key(s); "
|
| 388 |
+
f"{quota_blocked_keys} key(s) returned quota/rate-limit errors. Models tried: {models}. "
|
| 389 |
+
f"Last error: {last_error}"
|
| 390 |
+
)
|
| 391 |
+
|
| 392 |
+
def _safe_gemini_ocr(self, image_bytes: bytes, page_number: int) -> tuple[str, str | None]:
|
| 393 |
+
try:
|
| 394 |
+
text = self._clean_text_only(self._gemini_ocr(image_bytes))
|
| 395 |
+
except Exception as exc:
|
| 396 |
+
return "", f"Page {page_number} skipped: OCR failed ({exc})."
|
| 397 |
+
if not self._has_enough_text(text, min_chars=2):
|
| 398 |
+
return "", f"Page {page_number} skipped: no readable text found."
|
| 399 |
+
return text, None
|
| 400 |
+
|
| 401 |
+
def _skipped_page_event(
|
| 402 |
+
self,
|
| 403 |
+
page_number: int,
|
| 404 |
+
page_count: int,
|
| 405 |
+
warning: str,
|
| 406 |
+
confidence: float | None = None,
|
| 407 |
+
) -> dict:
|
| 408 |
+
return {
|
| 409 |
+
"type": "page",
|
| 410 |
+
"page_number": page_number,
|
| 411 |
+
"page_count": page_count,
|
| 412 |
+
"engine": "Skipped non-text page",
|
| 413 |
+
"route": "Skipped non-text content",
|
| 414 |
+
"direct_text_found": False,
|
| 415 |
+
"confidence": confidence,
|
| 416 |
+
"text": "",
|
| 417 |
+
"warning": warning,
|
| 418 |
+
}
|
| 419 |
+
|
| 420 |
+
def _stream_gemini_ocr_jobs(self, jobs: list[tuple[int, bytes]], page_count: int) -> Iterable[dict]:
|
| 421 |
+
worker_count = min(self.gemini_concurrency, len(jobs))
|
| 422 |
+
if worker_count <= 1:
|
| 423 |
+
for page_number, image_bytes in jobs:
|
| 424 |
+
yield self._gemini_page_event(page_number, page_count, image_bytes)
|
| 425 |
+
return
|
| 426 |
+
|
| 427 |
+
with ThreadPoolExecutor(max_workers=worker_count) as executor:
|
| 428 |
+
futures = {
|
| 429 |
+
executor.submit(self._gemini_page_event, page_number, page_count, image_bytes): page_number
|
| 430 |
+
for page_number, image_bytes in jobs
|
| 431 |
+
}
|
| 432 |
+
for future in as_completed(futures):
|
| 433 |
+
try:
|
| 434 |
+
yield future.result()
|
| 435 |
+
except Exception as exc:
|
| 436 |
+
yield self._skipped_page_event(futures[future], page_count, f"OCR failed: {exc}")
|
| 437 |
+
|
| 438 |
+
def _gemini_page_event(self, page_number: int, page_count: int, image_bytes: bytes) -> dict:
|
| 439 |
+
ocr_text, warning = self._safe_gemini_ocr(image_bytes, page_number)
|
| 440 |
+
return {
|
| 441 |
+
"type": "page",
|
| 442 |
+
"page_number": page_number,
|
| 443 |
+
"page_count": page_count,
|
| 444 |
+
"engine": "Gemini Vision OCR" if ocr_text else "Skipped non-text page",
|
| 445 |
+
"route": "PyMuPDF image -> OpenCV preprocess -> Gemini Vision OCR",
|
| 446 |
+
"direct_text_found": False,
|
| 447 |
+
"confidence": None,
|
| 448 |
+
"text": ocr_text,
|
| 449 |
+
"warning": warning,
|
| 450 |
+
}
|
| 451 |
+
|
| 452 |
+
def _gemini_api_keys(self) -> list[str]:
|
| 453 |
+
keys = []
|
| 454 |
+
for key in sorted(os.environ, key=self._gemini_env_key_sort):
|
| 455 |
+
value = os.environ[key]
|
| 456 |
+
if self._is_gemini_key_name(key):
|
| 457 |
+
self._add_gemini_key_values(keys, value)
|
| 458 |
+
|
| 459 |
+
for file_name in (".env", "env"):
|
| 460 |
+
env_path = Path(file_name)
|
| 461 |
+
if not env_path.exists():
|
| 462 |
+
continue
|
| 463 |
+
for line in env_path.read_text(encoding="utf-8", errors="ignore").splitlines():
|
| 464 |
+
stripped = line.strip()
|
| 465 |
+
if not stripped or stripped.startswith("#") or "=" not in stripped:
|
| 466 |
+
continue
|
| 467 |
+
key, value = stripped.split("=", 1)
|
| 468 |
+
key = key.strip()
|
| 469 |
+
value = value.strip().strip('"').strip("'")
|
| 470 |
+
if self._is_gemini_key_name(key):
|
| 471 |
+
self._add_gemini_key_values(keys, value)
|
| 472 |
+
return keys
|
| 473 |
+
|
| 474 |
+
def _gemini_key_attempt_order(self, api_keys: list[str]) -> list[str]:
|
| 475 |
+
with self._gemini_key_lock:
|
| 476 |
+
now = time.time()
|
| 477 |
+
active_keys = [
|
| 478 |
+
api_key
|
| 479 |
+
for api_key in api_keys
|
| 480 |
+
if self._gemini_quota_blocked_until.get(api_key, 0.0) <= now
|
| 481 |
+
]
|
| 482 |
+
if not active_keys:
|
| 483 |
+
active_keys = api_keys
|
| 484 |
+
active_key_set = set(active_keys)
|
| 485 |
+
start = self._gemini_key_cursor % len(api_keys)
|
| 486 |
+
ordered_keys = api_keys[start:] + api_keys[:start]
|
| 487 |
+
return [api_key for api_key in ordered_keys if api_key in active_key_set]
|
| 488 |
+
|
| 489 |
+
def _advance_gemini_key_cursor(self, api_keys: list[str], api_key: str) -> None:
|
| 490 |
+
with self._gemini_key_lock:
|
| 491 |
+
try:
|
| 492 |
+
self._gemini_key_cursor = (api_keys.index(api_key) + 1) % len(api_keys)
|
| 493 |
+
except ValueError:
|
| 494 |
+
self._gemini_key_cursor = 0
|
| 495 |
+
|
| 496 |
+
def _mark_gemini_key_quota_exhausted(self, api_keys: list[str], api_key: str) -> None:
|
| 497 |
+
with self._gemini_key_lock:
|
| 498 |
+
self._gemini_quota_blocked_until[api_key] = time.time() + self.gemini_key_quota_cooldown
|
| 499 |
+
try:
|
| 500 |
+
self._gemini_key_cursor = (api_keys.index(api_key) + 1) % len(api_keys)
|
| 501 |
+
except ValueError:
|
| 502 |
+
self._gemini_key_cursor = 0
|
| 503 |
+
|
| 504 |
+
def _is_gemini_key_name(self, key: str) -> bool:
|
| 505 |
+
return (
|
| 506 |
+
key == "GEMINI_API_KEY"
|
| 507 |
+
or key == "GOOGLE_API_KEY"
|
| 508 |
+
or key == "GEMINI_API_KEYS"
|
| 509 |
+
or key.startswith("GEMINI_API_KEY_")
|
| 510 |
+
)
|
| 511 |
+
|
| 512 |
+
def _gemini_env_key_sort(self, key: str) -> tuple[int, int, str]:
|
| 513 |
+
match = re.fullmatch(r"GEMINI_API_KEY_(\d+)", key)
|
| 514 |
+
if match:
|
| 515 |
+
return (0, int(match.group(1)), key)
|
| 516 |
+
if key == "GEMINI_API_KEY":
|
| 517 |
+
return (1, 0, key)
|
| 518 |
+
if key == "GEMINI_API_KEYS":
|
| 519 |
+
return (2, 0, key)
|
| 520 |
+
if key == "GOOGLE_API_KEY":
|
| 521 |
+
return (3, 0, key)
|
| 522 |
+
return (4, 0, key)
|
| 523 |
+
|
| 524 |
+
def _add_gemini_key_values(self, keys: list[str], value: str) -> None:
|
| 525 |
+
for api_key in (part.strip() for part in value.split(",")):
|
| 526 |
+
if api_key and api_key not in keys:
|
| 527 |
+
keys.append(api_key)
|
| 528 |
+
|
| 529 |
+
def _gemini_model_candidates(self, configured_model: str | None) -> list[str]:
|
| 530 |
+
configured_models = os.getenv("GEMINI_MODELS") or configured_model
|
| 531 |
+
if configured_models:
|
| 532 |
+
models = [
|
| 533 |
+
model.strip()
|
| 534 |
+
for model in configured_models.split(",")
|
| 535 |
+
if model.strip()
|
| 536 |
+
]
|
| 537 |
+
if models:
|
| 538 |
+
return models
|
| 539 |
+
return list(DEFAULT_GEMINI_MODELS)
|
| 540 |
+
|
| 541 |
+
def _is_retryable_gemini_error(self, error: Exception) -> bool:
|
| 542 |
+
message = str(error).lower()
|
| 543 |
+
retryable_markers = (
|
| 544 |
+
"503",
|
| 545 |
+
"unavailable",
|
| 546 |
+
"high demand",
|
| 547 |
+
"429",
|
| 548 |
+
"resource_exhausted",
|
| 549 |
+
"rate limit",
|
| 550 |
+
"quota",
|
| 551 |
+
"deadline",
|
| 552 |
+
"timeout",
|
| 553 |
+
)
|
| 554 |
+
return any(marker in message for marker in retryable_markers)
|
| 555 |
+
|
| 556 |
+
def _is_quota_error(self, error: Exception | None) -> bool:
|
| 557 |
+
if error is None:
|
| 558 |
+
return False
|
| 559 |
+
message = str(error).lower()
|
| 560 |
+
return "429" in message or "resource_exhausted" in message or "quota" in message
|
| 561 |
+
|
| 562 |
+
def _load_local_env(self) -> None:
|
| 563 |
+
for file_name in (".env", "env"):
|
| 564 |
+
env_path = Path(file_name)
|
| 565 |
+
if not env_path.exists():
|
| 566 |
+
continue
|
| 567 |
+
|
| 568 |
+
for line in env_path.read_text(encoding="utf-8", errors="ignore").splitlines():
|
| 569 |
+
stripped = line.strip()
|
| 570 |
+
if not stripped or stripped.startswith("#") or "=" not in stripped:
|
| 571 |
+
continue
|
| 572 |
+
|
| 573 |
+
key, value = stripped.split("=", 1)
|
| 574 |
+
key = key.strip()
|
| 575 |
+
value = value.strip().strip('"').strip("'")
|
| 576 |
+
if key and key not in os.environ:
|
| 577 |
+
os.environ[key] = value
|
| 578 |
+
|
| 579 |
+
def _env_int(self, key: str, fallback_key: str | None = None, default: int = 0, minimum: int = 0) -> int:
|
| 580 |
+
raw_value = os.getenv(key)
|
| 581 |
+
if raw_value is None and fallback_key:
|
| 582 |
+
raw_value = os.getenv(fallback_key)
|
| 583 |
+
if raw_value is None:
|
| 584 |
+
return default
|
| 585 |
+
try:
|
| 586 |
+
value = int(raw_value)
|
| 587 |
+
except ValueError:
|
| 588 |
+
return default
|
| 589 |
+
return max(value, minimum)
|
| 590 |
+
|
| 591 |
+
def _env_float(self, key: str, fallback_key: str | None = None, default: float = 0.0, minimum: float = 0.0) -> float:
|
| 592 |
+
raw_value = os.getenv(key)
|
| 593 |
+
if raw_value is None and fallback_key:
|
| 594 |
+
raw_value = os.getenv(fallback_key)
|
| 595 |
+
if raw_value is None:
|
| 596 |
+
return default
|
| 597 |
+
try:
|
| 598 |
+
value = float(raw_value)
|
| 599 |
+
except ValueError:
|
| 600 |
+
return default
|
| 601 |
+
return max(value, minimum)
|
| 602 |
+
|
| 603 |
+
def _trocr_ocr(self, image_bytes: bytes) -> tuple[str, float]:
|
| 604 |
+
try:
|
| 605 |
+
import torch
|
| 606 |
+
from PIL import Image
|
| 607 |
+
from transformers import TrOCRProcessor, VisionEncoderDecoderModel
|
| 608 |
+
except ImportError as exc:
|
| 609 |
+
raise RuntimeError(
|
| 610 |
+
"TrOCR requires: pip install torch transformers Pillow"
|
| 611 |
+
) from exc
|
| 612 |
+
|
| 613 |
+
if self._trocr_processor is None or self._trocr_model is None:
|
| 614 |
+
self._trocr_processor = TrOCRProcessor.from_pretrained(self.trocr_model_name)
|
| 615 |
+
self._trocr_model = VisionEncoderDecoderModel.from_pretrained(self.trocr_model_name)
|
| 616 |
+
|
| 617 |
+
image = Image.open(io.BytesIO(image_bytes)).convert("RGB")
|
| 618 |
+
pixel_values = self._trocr_processor(images=image, return_tensors="pt").pixel_values
|
| 619 |
+
|
| 620 |
+
with torch.no_grad():
|
| 621 |
+
generated = self._trocr_model.generate(
|
| 622 |
+
pixel_values,
|
| 623 |
+
max_new_tokens=512,
|
| 624 |
+
output_scores=True,
|
| 625 |
+
return_dict_in_generate=True,
|
| 626 |
+
)
|
| 627 |
+
|
| 628 |
+
text = self._trocr_processor.batch_decode(
|
| 629 |
+
generated.sequences,
|
| 630 |
+
skip_special_tokens=True,
|
| 631 |
+
)[0].strip()
|
| 632 |
+
confidence = self._score_trocr_confidence(generated.scores)
|
| 633 |
+
return text, confidence
|
| 634 |
+
|
| 635 |
+
def _score_trocr_confidence(self, scores: Iterable) -> float:
|
| 636 |
+
try:
|
| 637 |
+
import torch
|
| 638 |
+
except ImportError:
|
| 639 |
+
return 0.0
|
| 640 |
+
|
| 641 |
+
confidences = []
|
| 642 |
+
for score in scores:
|
| 643 |
+
probabilities = torch.softmax(score, dim=-1)
|
| 644 |
+
confidences.append(float(probabilities.max()))
|
| 645 |
+
|
| 646 |
+
if not confidences:
|
| 647 |
+
return 0.0
|
| 648 |
+
return sum(confidences) / len(confidences)
|
| 649 |
+
|
| 650 |
+
def _finalize_ocr_result(
|
| 651 |
+
self,
|
| 652 |
+
route: str,
|
| 653 |
+
page_count: int,
|
| 654 |
+
direct_text_found: bool,
|
| 655 |
+
pages: list[PageResult],
|
| 656 |
+
) -> ExtractionResult:
|
| 657 |
+
merged_text = "\n\n".join(
|
| 658 |
+
f"--- Page {page.page_number} ---\n{page.text.strip()}"
|
| 659 |
+
for page in pages
|
| 660 |
+
if page.text.strip()
|
| 661 |
+
)
|
| 662 |
+
return ExtractionResult(
|
| 663 |
+
text=self._clean_text(merged_text),
|
| 664 |
+
route=route,
|
| 665 |
+
page_count=page_count,
|
| 666 |
+
direct_text_found=direct_text_found,
|
| 667 |
+
pages=pages,
|
| 668 |
+
)
|
| 669 |
+
|
| 670 |
+
def _clean_text(self, text: str) -> str:
|
| 671 |
+
text = text.replace("\x00", "")
|
| 672 |
+
text = re.sub(r"[ \t]+", " ", text)
|
| 673 |
+
text = re.sub(r"\n{3,}", "\n\n", text)
|
| 674 |
+
text = "\n".join(line.rstrip() for line in text.splitlines())
|
| 675 |
+
return text.strip()
|
| 676 |
+
|
| 677 |
+
def _clean_text_only(self, text: str) -> str:
|
| 678 |
+
cleaned = self._clean_text(text)
|
| 679 |
+
lines = []
|
| 680 |
+
for line in cleaned.splitlines():
|
| 681 |
+
stripped = line.strip()
|
| 682 |
+
if not stripped:
|
| 683 |
+
lines.append("")
|
| 684 |
+
continue
|
| 685 |
+
if self._looks_like_visual_description(stripped):
|
| 686 |
+
continue
|
| 687 |
+
lines.append(stripped)
|
| 688 |
+
text_only = self._clean_text("\n".join(lines))
|
| 689 |
+
return text_only if self._has_real_text_signal(text_only) else ""
|
| 690 |
+
|
| 691 |
+
def _looks_like_visual_description(self, line: str) -> bool:
|
| 692 |
+
normalized = re.sub(r"\s+", " ", line.lower()).strip()
|
| 693 |
+
visual_description_patterns = (
|
| 694 |
+
r"^(the|this|an|a)\s+(image|photo|picture|diagram|figure|chart|graph|illustration|logo|icon)\s+",
|
| 695 |
+
r"^(the|this)\s+page\s+(contains|shows|appears|has|is)\s+",
|
| 696 |
+
r"^(it|this)\s+(shows|appears|looks like|contains|depicts)\s+",
|
| 697 |
+
r"^(i can see|there is|there are)\s+",
|
| 698 |
+
r"\b(no readable text|no text|cannot extract|not able to extract)\b",
|
| 699 |
+
r"\b(image shows|picture shows|diagram shows|photo shows|chart shows|graph shows)\b",
|
| 700 |
+
)
|
| 701 |
+
if any(re.search(pattern, normalized) for pattern in visual_description_patterns):
|
| 702 |
+
return True
|
| 703 |
+
visual_words = {
|
| 704 |
+
"image",
|
| 705 |
+
"photo",
|
| 706 |
+
"picture",
|
| 707 |
+
"diagram",
|
| 708 |
+
"illustration",
|
| 709 |
+
"visual",
|
| 710 |
+
"icon",
|
| 711 |
+
"logo",
|
| 712 |
+
"background",
|
| 713 |
+
"border",
|
| 714 |
+
"shape",
|
| 715 |
+
"graphic",
|
| 716 |
+
}
|
| 717 |
+
tokens = re.findall(r"[a-z0-9]+", normalized)
|
| 718 |
+
if not tokens:
|
| 719 |
+
return True
|
| 720 |
+
visual_count = sum(1 for token in tokens if token in visual_words)
|
| 721 |
+
return visual_count >= 2 and len(tokens) <= 18
|
| 722 |
+
|
| 723 |
+
def _has_real_text_signal(self, text: str) -> bool:
|
| 724 |
+
normalized = re.sub(r"\s+", " ", text or "").strip()
|
| 725 |
+
if not normalized:
|
| 726 |
+
return False
|
| 727 |
+
alnum_count = sum(char.isalnum() for char in normalized)
|
| 728 |
+
alpha_count = sum(char.isalpha() for char in normalized)
|
| 729 |
+
return alnum_count >= 2 and alpha_count >= 2
|
| 730 |
+
|
| 731 |
+
|
| 732 |
+
def extract_pdf_to_text(pdf_path: str | Path) -> ExtractionResult:
|
| 733 |
+
return PdfTextExtractor().extract(pdf_path)
|
| 734 |
+
|
| 735 |
+
|
| 736 |
+
def extract_uploaded_pdf(file_name: str, file_bytes: bytes) -> ExtractionResult:
|
| 737 |
+
suffix = Path(file_name).suffix or ".pdf"
|
| 738 |
+
with tempfile.NamedTemporaryFile(delete=False, suffix=suffix) as temp_file:
|
| 739 |
+
temp_file.write(file_bytes)
|
| 740 |
+
temp_path = Path(temp_file.name)
|
| 741 |
+
|
| 742 |
+
try:
|
| 743 |
+
return extract_pdf_to_text(temp_path)
|
| 744 |
+
finally:
|
| 745 |
+
temp_path.unlink(missing_ok=True)
|
| 746 |
+
|
| 747 |
+
|
| 748 |
+
def stream_uploaded_pdf(file_name: str, file_bytes: bytes) -> Iterable[dict]:
|
| 749 |
+
suffix = Path(file_name).suffix or ".pdf"
|
| 750 |
+
with tempfile.NamedTemporaryFile(delete=False, suffix=suffix) as temp_file:
|
| 751 |
+
temp_file.write(file_bytes)
|
| 752 |
+
temp_path = Path(temp_file.name)
|
| 753 |
+
|
| 754 |
+
try:
|
| 755 |
+
yield from PdfTextExtractor().stream_extract(temp_path)
|
| 756 |
+
finally:
|
| 757 |
+
temp_path.unlink(missing_ok=True)
|
rag_pipeline.py
ADDED
|
@@ -0,0 +1,1361 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
from __future__ import annotations
|
| 2 |
+
|
| 3 |
+
import hashlib
|
| 4 |
+
import json
|
| 5 |
+
import math
|
| 6 |
+
import os
|
| 7 |
+
import re
|
| 8 |
+
import sqlite3
|
| 9 |
+
import time
|
| 10 |
+
import unicodedata
|
| 11 |
+
import uuid
|
| 12 |
+
from collections import Counter, defaultdict
|
| 13 |
+
from dataclasses import asdict, dataclass, field
|
| 14 |
+
from pathlib import Path
|
| 15 |
+
from typing import Iterable
|
| 16 |
+
|
| 17 |
+
|
| 18 |
+
DATA_DIR = Path("data")
|
| 19 |
+
DB_PATH = DATA_DIR / "smartnotes.sqlite"
|
| 20 |
+
DEFAULT_EMBED_DIM = 384
|
| 21 |
+
DEFAULT_TOP_K = 50
|
| 22 |
+
UNRELATED_PDF_ANSWER = "Sorry, I can't give any answer because your question is not related to the PDF."
|
| 23 |
+
|
| 24 |
+
|
| 25 |
+
@dataclass
|
| 26 |
+
class ParentChunk:
|
| 27 |
+
id: str
|
| 28 |
+
document_id: str
|
| 29 |
+
index: int
|
| 30 |
+
text: str
|
| 31 |
+
token_count: int
|
| 32 |
+
metadata: dict
|
| 33 |
+
|
| 34 |
+
|
| 35 |
+
@dataclass
|
| 36 |
+
class ChildChunk:
|
| 37 |
+
id: str
|
| 38 |
+
parent_id: str
|
| 39 |
+
document_id: str
|
| 40 |
+
index: int
|
| 41 |
+
strategy: str
|
| 42 |
+
text: str
|
| 43 |
+
token_count: int
|
| 44 |
+
embedding: list[float] = field(default_factory=list)
|
| 45 |
+
metadata: dict = field(default_factory=dict)
|
| 46 |
+
|
| 47 |
+
|
| 48 |
+
@dataclass
|
| 49 |
+
class RetrievalHit:
|
| 50 |
+
child_id: str
|
| 51 |
+
parent_id: str
|
| 52 |
+
document_id: str
|
| 53 |
+
strategy: str
|
| 54 |
+
child_text: str
|
| 55 |
+
parent_text: str
|
| 56 |
+
vector_score: float = 0.0
|
| 57 |
+
bm25_score: float = 0.0
|
| 58 |
+
hybrid_score: float = 0.0
|
| 59 |
+
rerank_score: float = 0.0
|
| 60 |
+
grade: str = "unknown"
|
| 61 |
+
metadata: dict = field(default_factory=dict)
|
| 62 |
+
|
| 63 |
+
|
| 64 |
+
class SmartNotesRepository:
|
| 65 |
+
"""PostgreSQL-first repository with SQLite fallback for local UI testing."""
|
| 66 |
+
|
| 67 |
+
def __init__(self) -> None:
|
| 68 |
+
DATA_DIR.mkdir(exist_ok=True)
|
| 69 |
+
self.postgres_dsn = os.getenv("POSTGRES_DSN")
|
| 70 |
+
self.backend = "sqlite"
|
| 71 |
+
self._pg = None
|
| 72 |
+
if self.postgres_dsn:
|
| 73 |
+
try:
|
| 74 |
+
import psycopg
|
| 75 |
+
|
| 76 |
+
self._pg = psycopg.connect(self.postgres_dsn)
|
| 77 |
+
self.backend = "postgresql"
|
| 78 |
+
except Exception:
|
| 79 |
+
self._pg = None
|
| 80 |
+
self.backend = "sqlite"
|
| 81 |
+
self._sqlite = sqlite3.connect(DB_PATH, check_same_thread=False)
|
| 82 |
+
self._sqlite.row_factory = sqlite3.Row
|
| 83 |
+
self._init_sqlite()
|
| 84 |
+
|
| 85 |
+
def _init_sqlite(self) -> None:
|
| 86 |
+
self._sqlite.executescript(
|
| 87 |
+
"""
|
| 88 |
+
create table if not exists documents (
|
| 89 |
+
id text primary key,
|
| 90 |
+
file_name text,
|
| 91 |
+
status text,
|
| 92 |
+
metadata text,
|
| 93 |
+
cleaned_text text,
|
| 94 |
+
created_at real,
|
| 95 |
+
updated_at real
|
| 96 |
+
);
|
| 97 |
+
create table if not exists parent_chunks (
|
| 98 |
+
id text primary key,
|
| 99 |
+
document_id text,
|
| 100 |
+
chunk_index integer,
|
| 101 |
+
text text,
|
| 102 |
+
token_count integer,
|
| 103 |
+
metadata text
|
| 104 |
+
);
|
| 105 |
+
create table if not exists child_chunks (
|
| 106 |
+
id text primary key,
|
| 107 |
+
parent_id text,
|
| 108 |
+
document_id text,
|
| 109 |
+
chunk_index integer,
|
| 110 |
+
strategy text,
|
| 111 |
+
text text,
|
| 112 |
+
token_count integer,
|
| 113 |
+
embedding text,
|
| 114 |
+
metadata text
|
| 115 |
+
);
|
| 116 |
+
create table if not exists retrieval_logs (
|
| 117 |
+
id text primary key,
|
| 118 |
+
document_id text,
|
| 119 |
+
query text,
|
| 120 |
+
rewritten_query text,
|
| 121 |
+
status text,
|
| 122 |
+
answer text,
|
| 123 |
+
hits text,
|
| 124 |
+
created_at real
|
| 125 |
+
);
|
| 126 |
+
create table if not exists feedback (
|
| 127 |
+
id text primary key,
|
| 128 |
+
document_id text,
|
| 129 |
+
query text,
|
| 130 |
+
rating integer,
|
| 131 |
+
comment text,
|
| 132 |
+
created_at real
|
| 133 |
+
);
|
| 134 |
+
"""
|
| 135 |
+
)
|
| 136 |
+
self._sqlite.commit()
|
| 137 |
+
|
| 138 |
+
def create_document(self, file_name: str, metadata: dict) -> str:
|
| 139 |
+
document_id = str(uuid.uuid4())
|
| 140 |
+
now = time.time()
|
| 141 |
+
self._sqlite.execute(
|
| 142 |
+
"""
|
| 143 |
+
insert into documents (id, file_name, status, metadata, cleaned_text, created_at, updated_at)
|
| 144 |
+
values (?, ?, ?, ?, ?, ?, ?)
|
| 145 |
+
""",
|
| 146 |
+
(document_id, file_name, "processing", json.dumps(metadata), "", now, now),
|
| 147 |
+
)
|
| 148 |
+
self._sqlite.commit()
|
| 149 |
+
return document_id
|
| 150 |
+
|
| 151 |
+
def update_document_status(self, document_id: str, status: str) -> None:
|
| 152 |
+
self._sqlite.execute(
|
| 153 |
+
"update documents set status = ?, updated_at = ? where id = ?",
|
| 154 |
+
(status, time.time(), document_id),
|
| 155 |
+
)
|
| 156 |
+
self._sqlite.commit()
|
| 157 |
+
|
| 158 |
+
def save_cleaned_text(self, document_id: str, cleaned_text: str, metadata: dict) -> None:
|
| 159 |
+
self._sqlite.execute(
|
| 160 |
+
"update documents set cleaned_text = ?, metadata = ?, updated_at = ? where id = ?",
|
| 161 |
+
(cleaned_text, json.dumps(metadata), time.time(), document_id),
|
| 162 |
+
)
|
| 163 |
+
self._sqlite.commit()
|
| 164 |
+
|
| 165 |
+
def save_chunks(self, parents: list[ParentChunk], children: list[ChildChunk]) -> None:
|
| 166 |
+
self._sqlite.executemany(
|
| 167 |
+
"""
|
| 168 |
+
insert or replace into parent_chunks (id, document_id, chunk_index, text, token_count, metadata)
|
| 169 |
+
values (?, ?, ?, ?, ?, ?)
|
| 170 |
+
""",
|
| 171 |
+
[
|
| 172 |
+
(
|
| 173 |
+
chunk.id,
|
| 174 |
+
chunk.document_id,
|
| 175 |
+
chunk.index,
|
| 176 |
+
chunk.text,
|
| 177 |
+
chunk.token_count,
|
| 178 |
+
json.dumps(chunk.metadata),
|
| 179 |
+
)
|
| 180 |
+
for chunk in parents
|
| 181 |
+
],
|
| 182 |
+
)
|
| 183 |
+
self._sqlite.executemany(
|
| 184 |
+
"""
|
| 185 |
+
insert or replace into child_chunks
|
| 186 |
+
(id, parent_id, document_id, chunk_index, strategy, text, token_count, embedding, metadata)
|
| 187 |
+
values (?, ?, ?, ?, ?, ?, ?, ?, ?)
|
| 188 |
+
""",
|
| 189 |
+
[
|
| 190 |
+
(
|
| 191 |
+
chunk.id,
|
| 192 |
+
chunk.parent_id,
|
| 193 |
+
chunk.document_id,
|
| 194 |
+
chunk.index,
|
| 195 |
+
chunk.strategy,
|
| 196 |
+
chunk.text,
|
| 197 |
+
chunk.token_count,
|
| 198 |
+
json.dumps(chunk.embedding),
|
| 199 |
+
json.dumps(chunk.metadata),
|
| 200 |
+
)
|
| 201 |
+
for chunk in children
|
| 202 |
+
],
|
| 203 |
+
)
|
| 204 |
+
self._sqlite.commit()
|
| 205 |
+
|
| 206 |
+
def get_document(self, document_id: str) -> dict | None:
|
| 207 |
+
row = self._sqlite.execute(
|
| 208 |
+
"select * from documents where id = ?",
|
| 209 |
+
(document_id,),
|
| 210 |
+
).fetchone()
|
| 211 |
+
if not row:
|
| 212 |
+
return None
|
| 213 |
+
return {
|
| 214 |
+
"id": row["id"],
|
| 215 |
+
"file_name": row["file_name"],
|
| 216 |
+
"status": row["status"],
|
| 217 |
+
"metadata": json.loads(row["metadata"] or "{}"),
|
| 218 |
+
"cleaned_text": row["cleaned_text"] or "",
|
| 219 |
+
"storage_backend": self.backend,
|
| 220 |
+
}
|
| 221 |
+
|
| 222 |
+
def get_children(self, document_id: str) -> list[ChildChunk]:
|
| 223 |
+
rows = self._sqlite.execute(
|
| 224 |
+
"select * from child_chunks where document_id = ? order by chunk_index",
|
| 225 |
+
(document_id,),
|
| 226 |
+
).fetchall()
|
| 227 |
+
return [
|
| 228 |
+
ChildChunk(
|
| 229 |
+
id=row["id"],
|
| 230 |
+
parent_id=row["parent_id"],
|
| 231 |
+
document_id=row["document_id"],
|
| 232 |
+
index=row["chunk_index"],
|
| 233 |
+
strategy=row["strategy"],
|
| 234 |
+
text=row["text"],
|
| 235 |
+
token_count=row["token_count"],
|
| 236 |
+
embedding=json.loads(row["embedding"] or "[]"),
|
| 237 |
+
metadata=json.loads(row["metadata"] or "{}"),
|
| 238 |
+
)
|
| 239 |
+
for row in rows
|
| 240 |
+
]
|
| 241 |
+
|
| 242 |
+
def get_parent_map(self, document_id: str) -> dict[str, ParentChunk]:
|
| 243 |
+
rows = self._sqlite.execute(
|
| 244 |
+
"select * from parent_chunks where document_id = ?",
|
| 245 |
+
(document_id,),
|
| 246 |
+
).fetchall()
|
| 247 |
+
return {
|
| 248 |
+
row["id"]: ParentChunk(
|
| 249 |
+
id=row["id"],
|
| 250 |
+
document_id=row["document_id"],
|
| 251 |
+
index=row["chunk_index"],
|
| 252 |
+
text=row["text"],
|
| 253 |
+
token_count=row["token_count"],
|
| 254 |
+
metadata=json.loads(row["metadata"] or "{}"),
|
| 255 |
+
)
|
| 256 |
+
for row in rows
|
| 257 |
+
}
|
| 258 |
+
|
| 259 |
+
def save_retrieval_log(
|
| 260 |
+
self,
|
| 261 |
+
document_id: str,
|
| 262 |
+
query: str,
|
| 263 |
+
rewritten_query: str,
|
| 264 |
+
status: str,
|
| 265 |
+
answer: str,
|
| 266 |
+
hits: list[RetrievalHit],
|
| 267 |
+
) -> str:
|
| 268 |
+
log_id = str(uuid.uuid4())
|
| 269 |
+
self._sqlite.execute(
|
| 270 |
+
"""
|
| 271 |
+
insert into retrieval_logs
|
| 272 |
+
(id, document_id, query, rewritten_query, status, answer, hits, created_at)
|
| 273 |
+
values (?, ?, ?, ?, ?, ?, ?, ?)
|
| 274 |
+
""",
|
| 275 |
+
(
|
| 276 |
+
log_id,
|
| 277 |
+
document_id,
|
| 278 |
+
query,
|
| 279 |
+
rewritten_query,
|
| 280 |
+
status,
|
| 281 |
+
answer,
|
| 282 |
+
json.dumps([self._hit_payload(hit) for hit in hits]),
|
| 283 |
+
time.time(),
|
| 284 |
+
),
|
| 285 |
+
)
|
| 286 |
+
self._sqlite.commit()
|
| 287 |
+
return log_id
|
| 288 |
+
|
| 289 |
+
def save_feedback(self, document_id: str, query: str, rating: int, comment: str) -> str:
|
| 290 |
+
feedback_id = str(uuid.uuid4())
|
| 291 |
+
self._sqlite.execute(
|
| 292 |
+
"""
|
| 293 |
+
insert into feedback (id, document_id, query, rating, comment, created_at)
|
| 294 |
+
values (?, ?, ?, ?, ?, ?)
|
| 295 |
+
""",
|
| 296 |
+
(feedback_id, document_id, query, rating, comment, time.time()),
|
| 297 |
+
)
|
| 298 |
+
self._sqlite.commit()
|
| 299 |
+
return feedback_id
|
| 300 |
+
|
| 301 |
+
def _hit_payload(self, hit: RetrievalHit) -> dict:
|
| 302 |
+
data = asdict(hit)
|
| 303 |
+
data["child_text"] = hit.child_text[:900]
|
| 304 |
+
data["parent_text"] = hit.parent_text[:1400]
|
| 305 |
+
return data
|
| 306 |
+
|
| 307 |
+
|
| 308 |
+
class TextCleaner:
|
| 309 |
+
def clean(self, text: str) -> str:
|
| 310 |
+
try:
|
| 311 |
+
import ftfy
|
| 312 |
+
|
| 313 |
+
text = ftfy.fix_text(text)
|
| 314 |
+
except Exception:
|
| 315 |
+
pass
|
| 316 |
+
text = unicodedata.normalize("NFKC", text or "")
|
| 317 |
+
text = text.replace("\x00", "")
|
| 318 |
+
text = re.sub(r"---\s*Page\s+\d+\s*---", "\n", text, flags=re.IGNORECASE)
|
| 319 |
+
text = re.sub(r"[ \t]+", " ", text)
|
| 320 |
+
text = re.sub(r"(\w)-\n(\w)", r"\1\2", text)
|
| 321 |
+
text = re.sub(r"\n(?=[a-z])", " ", text)
|
| 322 |
+
text = re.sub(r"\n{3,}", "\n\n", text)
|
| 323 |
+
return "\n".join(line.strip() for line in text.splitlines()).strip()
|
| 324 |
+
|
| 325 |
+
|
| 326 |
+
class Chunker:
|
| 327 |
+
def parent_chunks(self, document_id: str, text: str, metadata: dict) -> list[ParentChunk]:
|
| 328 |
+
chunks = self._recursive_split(text, chunk_size=1400, overlap=180)
|
| 329 |
+
return [
|
| 330 |
+
ParentChunk(
|
| 331 |
+
id=str(uuid.uuid4()),
|
| 332 |
+
document_id=document_id,
|
| 333 |
+
index=index,
|
| 334 |
+
text=chunk,
|
| 335 |
+
token_count=self._token_count(chunk),
|
| 336 |
+
metadata={**metadata, "chunk_type": "parent"},
|
| 337 |
+
)
|
| 338 |
+
for index, chunk in enumerate(chunks)
|
| 339 |
+
]
|
| 340 |
+
|
| 341 |
+
def child_chunks(self, parents: list[ParentChunk]) -> list[ChildChunk]:
|
| 342 |
+
children: list[ChildChunk] = []
|
| 343 |
+
index = 0
|
| 344 |
+
for parent in parents:
|
| 345 |
+
for strategy, chunks in (
|
| 346 |
+
("fixed", self._fixed_split(parent.text, chunk_size=380, overlap=70)),
|
| 347 |
+
("recursive", self._recursive_split(parent.text, chunk_size=520, overlap=90)),
|
| 348 |
+
("semantic", self._semantic_split(parent.text, target_size=520)),
|
| 349 |
+
):
|
| 350 |
+
for text in chunks:
|
| 351 |
+
children.append(
|
| 352 |
+
ChildChunk(
|
| 353 |
+
id=str(uuid.uuid4()),
|
| 354 |
+
parent_id=parent.id,
|
| 355 |
+
document_id=parent.document_id,
|
| 356 |
+
index=index,
|
| 357 |
+
strategy=strategy,
|
| 358 |
+
text=text,
|
| 359 |
+
token_count=self._token_count(text),
|
| 360 |
+
metadata={
|
| 361 |
+
**parent.metadata,
|
| 362 |
+
"chunk_type": "child",
|
| 363 |
+
"parent_index": parent.index,
|
| 364 |
+
"strategy": strategy,
|
| 365 |
+
},
|
| 366 |
+
)
|
| 367 |
+
)
|
| 368 |
+
index += 1
|
| 369 |
+
return children
|
| 370 |
+
|
| 371 |
+
def _fixed_split(self, text: str, chunk_size: int, overlap: int) -> list[str]:
|
| 372 |
+
words = text.split()
|
| 373 |
+
chunks = []
|
| 374 |
+
step = max(chunk_size - overlap, 1)
|
| 375 |
+
for start in range(0, len(words), step):
|
| 376 |
+
chunk = " ".join(words[start : start + chunk_size]).strip()
|
| 377 |
+
if chunk:
|
| 378 |
+
chunks.append(chunk)
|
| 379 |
+
return chunks
|
| 380 |
+
|
| 381 |
+
def _recursive_split(self, text: str, chunk_size: int, overlap: int) -> list[str]:
|
| 382 |
+
separators = ["\n\n", "\n", ". ", " "]
|
| 383 |
+
pieces = self._split_by_separators(text, separators, chunk_size)
|
| 384 |
+
chunks = []
|
| 385 |
+
current: list[str] = []
|
| 386 |
+
current_len = 0
|
| 387 |
+
for piece in pieces:
|
| 388 |
+
words = piece.split()
|
| 389 |
+
if current and current_len + len(words) > chunk_size:
|
| 390 |
+
chunks.append(" ".join(current).strip())
|
| 391 |
+
current = current[-overlap:] if overlap else []
|
| 392 |
+
current_len = len(current)
|
| 393 |
+
current.extend(words)
|
| 394 |
+
current_len += len(words)
|
| 395 |
+
if current:
|
| 396 |
+
chunks.append(" ".join(current).strip())
|
| 397 |
+
return [chunk for chunk in chunks if chunk]
|
| 398 |
+
|
| 399 |
+
def _semantic_split(self, text: str, target_size: int) -> list[str]:
|
| 400 |
+
sentences = re.split(r"(?<=[.!?])\s+", text)
|
| 401 |
+
chunks = []
|
| 402 |
+
current = []
|
| 403 |
+
current_len = 0
|
| 404 |
+
for sentence in sentences:
|
| 405 |
+
count = self._token_count(sentence)
|
| 406 |
+
if current and current_len + count > target_size:
|
| 407 |
+
chunks.append(" ".join(current).strip())
|
| 408 |
+
current = []
|
| 409 |
+
current_len = 0
|
| 410 |
+
current.append(sentence)
|
| 411 |
+
current_len += count
|
| 412 |
+
if current:
|
| 413 |
+
chunks.append(" ".join(current).strip())
|
| 414 |
+
return [chunk for chunk in chunks if chunk]
|
| 415 |
+
|
| 416 |
+
def _split_by_separators(self, text: str, separators: list[str], chunk_size: int) -> list[str]:
|
| 417 |
+
if self._token_count(text) <= chunk_size or not separators:
|
| 418 |
+
return [text]
|
| 419 |
+
separator = separators[0]
|
| 420 |
+
output = []
|
| 421 |
+
for part in text.split(separator):
|
| 422 |
+
output.extend(self._split_by_separators(part, separators[1:], chunk_size))
|
| 423 |
+
return output
|
| 424 |
+
|
| 425 |
+
def _token_count(self, text: str) -> int:
|
| 426 |
+
return len(text.split())
|
| 427 |
+
|
| 428 |
+
|
| 429 |
+
class EmbeddingService:
|
| 430 |
+
def __init__(self) -> None:
|
| 431 |
+
self.model_name = os.getenv("EMBEDDING_MODEL", "BAAI/bge-small-en-v1.5")
|
| 432 |
+
self.dim = DEFAULT_EMBED_DIM
|
| 433 |
+
self.backend = "hash"
|
| 434 |
+
self._model = None
|
| 435 |
+
if os.getenv("USE_LOCAL_EMBEDDING_MODEL", "0") == "1":
|
| 436 |
+
try:
|
| 437 |
+
from sentence_transformers import SentenceTransformer
|
| 438 |
+
|
| 439 |
+
self._model = SentenceTransformer(self.model_name)
|
| 440 |
+
self.backend = self.model_name
|
| 441 |
+
self.dim = int(self._model.get_sentence_embedding_dimension())
|
| 442 |
+
except Exception:
|
| 443 |
+
self._model = None
|
| 444 |
+
|
| 445 |
+
def embed_texts(self, texts: list[str]) -> list[list[float]]:
|
| 446 |
+
if self._model is not None:
|
| 447 |
+
vectors = self._model.encode(texts, normalize_embeddings=True).tolist()
|
| 448 |
+
return [[float(value) for value in vector] for vector in vectors]
|
| 449 |
+
return [self._hash_embedding(text) for text in texts]
|
| 450 |
+
|
| 451 |
+
def embed_query(self, query: str) -> list[float]:
|
| 452 |
+
return self.embed_texts([query])[0]
|
| 453 |
+
|
| 454 |
+
def _hash_embedding(self, text: str) -> list[float]:
|
| 455 |
+
vector = [0.0] * self.dim
|
| 456 |
+
tokens = tokenize(text)
|
| 457 |
+
for token in tokens:
|
| 458 |
+
digest = hashlib.sha256(token.encode("utf-8")).digest()
|
| 459 |
+
index = int.from_bytes(digest[:4], "big") % self.dim
|
| 460 |
+
sign = 1.0 if digest[4] % 2 == 0 else -1.0
|
| 461 |
+
vector[index] += sign
|
| 462 |
+
norm = math.sqrt(sum(value * value for value in vector)) or 1.0
|
| 463 |
+
return [value / norm for value in vector]
|
| 464 |
+
|
| 465 |
+
|
| 466 |
+
class QdrantVectorStore:
|
| 467 |
+
def __init__(self) -> None:
|
| 468 |
+
self.backend = "sqlite-vector-fallback"
|
| 469 |
+
self.collection = os.getenv("QDRANT_COLLECTION", "smartnotes_chunks")
|
| 470 |
+
self.client = None
|
| 471 |
+
self.url = os.getenv("QDRANT_URL")
|
| 472 |
+
if self.url:
|
| 473 |
+
try:
|
| 474 |
+
from qdrant_client import QdrantClient
|
| 475 |
+
|
| 476 |
+
self.client = QdrantClient(url=self.url, api_key=os.getenv("QDRANT_API_KEY"))
|
| 477 |
+
self.backend = "qdrant"
|
| 478 |
+
except Exception:
|
| 479 |
+
self.client = None
|
| 480 |
+
|
| 481 |
+
def upsert(self, children: list[ChildChunk]) -> None:
|
| 482 |
+
if self.client is None or not children:
|
| 483 |
+
return
|
| 484 |
+
try:
|
| 485 |
+
from qdrant_client.models import Distance, PointStruct, VectorParams
|
| 486 |
+
|
| 487 |
+
dim = len(children[0].embedding)
|
| 488 |
+
existing = [item.name for item in self.client.get_collections().collections]
|
| 489 |
+
if self.collection not in existing:
|
| 490 |
+
self.client.create_collection(
|
| 491 |
+
collection_name=self.collection,
|
| 492 |
+
vectors_config=VectorParams(size=dim, distance=Distance.COSINE),
|
| 493 |
+
)
|
| 494 |
+
self.client.upsert(
|
| 495 |
+
collection_name=self.collection,
|
| 496 |
+
points=[
|
| 497 |
+
PointStruct(
|
| 498 |
+
id=chunk.id,
|
| 499 |
+
vector=chunk.embedding,
|
| 500 |
+
payload={
|
| 501 |
+
"document_id": chunk.document_id,
|
| 502 |
+
"parent_id": chunk.parent_id,
|
| 503 |
+
"strategy": chunk.strategy,
|
| 504 |
+
},
|
| 505 |
+
)
|
| 506 |
+
for chunk in children
|
| 507 |
+
],
|
| 508 |
+
)
|
| 509 |
+
except Exception:
|
| 510 |
+
self.client = None
|
| 511 |
+
self.backend = "sqlite-vector-fallback"
|
| 512 |
+
|
| 513 |
+
def search(self, query_vector: list[float], children: list[ChildChunk], top_k: int) -> list[tuple[str, float]]:
|
| 514 |
+
if self.client is not None:
|
| 515 |
+
try:
|
| 516 |
+
result = self.client.search(
|
| 517 |
+
collection_name=self.collection,
|
| 518 |
+
query_vector=query_vector,
|
| 519 |
+
limit=top_k,
|
| 520 |
+
)
|
| 521 |
+
return [(str(point.id), float(point.score)) for point in result]
|
| 522 |
+
except Exception:
|
| 523 |
+
pass
|
| 524 |
+
scored = [
|
| 525 |
+
(chunk.id, cosine_similarity(query_vector, chunk.embedding))
|
| 526 |
+
for chunk in children
|
| 527 |
+
]
|
| 528 |
+
return sorted(scored, key=lambda item: item[1], reverse=True)[:top_k]
|
| 529 |
+
|
| 530 |
+
|
| 531 |
+
class BM25Index:
|
| 532 |
+
def search(self, query: str, children: list[ChildChunk], top_k: int) -> list[tuple[str, float]]:
|
| 533 |
+
query_tokens = tokenize(query)
|
| 534 |
+
docs = [tokenize(chunk.text) for chunk in children]
|
| 535 |
+
if not docs:
|
| 536 |
+
return []
|
| 537 |
+
avgdl = sum(len(doc) for doc in docs) / len(docs)
|
| 538 |
+
df = Counter()
|
| 539 |
+
for doc in docs:
|
| 540 |
+
for token in set(doc):
|
| 541 |
+
df[token] += 1
|
| 542 |
+
scores = []
|
| 543 |
+
for chunk, doc in zip(children, docs):
|
| 544 |
+
score = self._score_doc(query_tokens, doc, df, len(docs), avgdl)
|
| 545 |
+
scores.append((chunk.id, score))
|
| 546 |
+
return sorted(scores, key=lambda item: item[1], reverse=True)[:top_k]
|
| 547 |
+
|
| 548 |
+
def _score_doc(
|
| 549 |
+
self,
|
| 550 |
+
query_tokens: list[str],
|
| 551 |
+
doc: list[str],
|
| 552 |
+
df: Counter,
|
| 553 |
+
total_docs: int,
|
| 554 |
+
avgdl: float,
|
| 555 |
+
) -> float:
|
| 556 |
+
counts = Counter(doc)
|
| 557 |
+
k1 = 1.5
|
| 558 |
+
b = 0.75
|
| 559 |
+
score = 0.0
|
| 560 |
+
for token in query_tokens:
|
| 561 |
+
if not counts[token]:
|
| 562 |
+
continue
|
| 563 |
+
idf = math.log(1 + (total_docs - df[token] + 0.5) / (df[token] + 0.5))
|
| 564 |
+
numerator = counts[token] * (k1 + 1)
|
| 565 |
+
denominator = counts[token] + k1 * (1 - b + b * len(doc) / (avgdl or 1))
|
| 566 |
+
score += idf * numerator / denominator
|
| 567 |
+
return score
|
| 568 |
+
|
| 569 |
+
|
| 570 |
+
class Reranker:
|
| 571 |
+
def __init__(self) -> None:
|
| 572 |
+
self.model_name = os.getenv("RERANKER_MODEL", "BAAI/bge-reranker-base")
|
| 573 |
+
self.backend = "token-overlap"
|
| 574 |
+
self._model = None
|
| 575 |
+
if os.getenv("USE_LOCAL_RERANKER_MODEL", "0") == "1":
|
| 576 |
+
try:
|
| 577 |
+
from sentence_transformers import CrossEncoder
|
| 578 |
+
|
| 579 |
+
self._model = CrossEncoder(self.model_name)
|
| 580 |
+
self.backend = self.model_name
|
| 581 |
+
except Exception:
|
| 582 |
+
self._model = None
|
| 583 |
+
|
| 584 |
+
def rerank(self, query: str, hits: list[RetrievalHit]) -> list[RetrievalHit]:
|
| 585 |
+
if self._model is not None and hits:
|
| 586 |
+
scores = self._model.predict([(query, hit.child_text) for hit in hits])
|
| 587 |
+
for hit, score in zip(hits, scores):
|
| 588 |
+
hit.rerank_score = float(score)
|
| 589 |
+
else:
|
| 590 |
+
query_tokens = set(tokenize(query))
|
| 591 |
+
for hit in hits:
|
| 592 |
+
chunk_tokens = set(tokenize(hit.child_text))
|
| 593 |
+
hit.rerank_score = len(query_tokens & chunk_tokens) / max(len(query_tokens), 1)
|
| 594 |
+
return sorted(hits, key=lambda hit: hit.rerank_score, reverse=True)
|
| 595 |
+
|
| 596 |
+
|
| 597 |
+
class LLMService:
|
| 598 |
+
def __init__(self) -> None:
|
| 599 |
+
load_local_env()
|
| 600 |
+
self.provider = os.getenv("ANSWER_PROVIDER", "gemini")
|
| 601 |
+
self.model = os.getenv("ANSWER_MODEL", "gemini-2.5-flash")
|
| 602 |
+
self._gemini_clients = {}
|
| 603 |
+
self._gemini_key_cursor = 0
|
| 604 |
+
self._gemini_quota_blocked_until: dict[str, float] = {}
|
| 605 |
+
self.gemini_timeout_ms = int(
|
| 606 |
+
min(env_float("GEMINI_TIMEOUT", default=20.0, minimum=5.0), 120.0)
|
| 607 |
+
* 1000
|
| 608 |
+
)
|
| 609 |
+
self.gemini_key_quota_cooldown = min(
|
| 610 |
+
env_float("GEMINI_KEY_QUOTA_COOLDOWN", default=300.0, minimum=0.0),
|
| 611 |
+
86400.0,
|
| 612 |
+
)
|
| 613 |
+
|
| 614 |
+
def rewrite_query(self, query: str) -> str:
|
| 615 |
+
if len(query.split()) >= 4:
|
| 616 |
+
return query.strip()
|
| 617 |
+
prompt = f"Rewrite this user question into a precise PDF search query. Return only the query.\nQuestion: {query}"
|
| 618 |
+
return self._generate(prompt, fallback=query).strip() or query
|
| 619 |
+
|
| 620 |
+
def grade_retrieval(self, query: str, hits: list[RetrievalHit]) -> str:
|
| 621 |
+
if not hits:
|
| 622 |
+
return "bad"
|
| 623 |
+
best = max(hit.rerank_score for hit in hits)
|
| 624 |
+
lexical = max(hit.hybrid_score for hit in hits)
|
| 625 |
+
return "good" if best >= 0.08 or lexical >= 0.12 else "bad"
|
| 626 |
+
|
| 627 |
+
def answer(self, query: str, parent_contexts: list[ParentChunk]) -> str:
|
| 628 |
+
if not parent_contexts:
|
| 629 |
+
return UNRELATED_PDF_ANSWER
|
| 630 |
+
mode = self._answer_mode(query)
|
| 631 |
+
fallback = self._fallback_answer(query, parent_contexts, mode)
|
| 632 |
+
context = "\n\n".join(
|
| 633 |
+
f"[Source {index + 1} | parent={chunk.id}]\n{chunk.text}"
|
| 634 |
+
for index, chunk in enumerate(parent_contexts)
|
| 635 |
+
)
|
| 636 |
+
prompt = (
|
| 637 |
+
"You are SmartNotes AI. Answer using only the PDF context.\n"
|
| 638 |
+
"Rules:\n"
|
| 639 |
+
"- Give a direct answer first.\n"
|
| 640 |
+
"- Do not dump the whole context.\n"
|
| 641 |
+
"- If the user asks 'what is' or 'define', give only the definition in 1-2 sentences.\n"
|
| 642 |
+
"- Answer in normal readable text only.\n"
|
| 643 |
+
"- Do not return equations, raw calculations, URLs, or OCR garbage unless the user explicitly asks for them.\n"
|
| 644 |
+
"- If the user asks in detail, explain with key points, types, examples, and relevant facts from the PDF.\n"
|
| 645 |
+
"- Remove page markers and OCR noise.\n"
|
| 646 |
+
f"- If the clean text answer is not present or the question is not related to the PDF, say exactly: {UNRELATED_PDF_ANSWER}\n"
|
| 647 |
+
"- End with citation like [Source 1].\n\n"
|
| 648 |
+
f"Question: {query}\n\nContext:\n{context}\n\nAnswer:"
|
| 649 |
+
)
|
| 650 |
+
return self._postprocess_answer(self._generate(prompt, fallback=fallback))
|
| 651 |
+
|
| 652 |
+
def _generate(self, prompt: str, fallback: str) -> str:
|
| 653 |
+
if self.provider == "openai" and os.getenv("OPENAI_API_KEY"):
|
| 654 |
+
try:
|
| 655 |
+
from openai import OpenAI
|
| 656 |
+
|
| 657 |
+
client = OpenAI(api_key=os.getenv("OPENAI_API_KEY"))
|
| 658 |
+
response = client.chat.completions.create(
|
| 659 |
+
model=os.getenv("OPENAI_MODEL", "gpt-4.1-mini"),
|
| 660 |
+
messages=[{"role": "user", "content": prompt}],
|
| 661 |
+
)
|
| 662 |
+
return response.choices[0].message.content or fallback
|
| 663 |
+
except Exception:
|
| 664 |
+
return fallback
|
| 665 |
+
api_keys = gemini_api_keys()
|
| 666 |
+
if api_keys:
|
| 667 |
+
try:
|
| 668 |
+
from google import genai
|
| 669 |
+
from google.genai import types
|
| 670 |
+
except Exception:
|
| 671 |
+
return fallback
|
| 672 |
+
for api_key in self._gemini_key_attempt_order(api_keys):
|
| 673 |
+
try:
|
| 674 |
+
client = self._gemini_clients.get(api_key)
|
| 675 |
+
if client is None:
|
| 676 |
+
client = genai.Client(
|
| 677 |
+
api_key=api_key,
|
| 678 |
+
http_options=types.HttpOptions(timeout=self.gemini_timeout_ms),
|
| 679 |
+
)
|
| 680 |
+
self._gemini_clients[api_key] = client
|
| 681 |
+
response = client.models.generate_content(
|
| 682 |
+
model=self.model,
|
| 683 |
+
contents=prompt,
|
| 684 |
+
)
|
| 685 |
+
self._advance_gemini_key_cursor(api_keys, api_key)
|
| 686 |
+
return (getattr(response, "text", None) or fallback).strip()
|
| 687 |
+
except Exception as exc:
|
| 688 |
+
if is_gemini_quota_error(exc):
|
| 689 |
+
self._mark_gemini_key_quota_exhausted(api_keys, api_key)
|
| 690 |
+
continue
|
| 691 |
+
return fallback
|
| 692 |
+
|
| 693 |
+
def _gemini_key_attempt_order(self, api_keys: list[str]) -> list[str]:
|
| 694 |
+
now = time.time()
|
| 695 |
+
active_keys = [
|
| 696 |
+
api_key
|
| 697 |
+
for api_key in api_keys
|
| 698 |
+
if self._gemini_quota_blocked_until.get(api_key, 0.0) <= now
|
| 699 |
+
]
|
| 700 |
+
if not active_keys:
|
| 701 |
+
active_keys = api_keys
|
| 702 |
+
active_key_set = set(active_keys)
|
| 703 |
+
start = self._gemini_key_cursor % len(api_keys)
|
| 704 |
+
ordered_keys = api_keys[start:] + api_keys[:start]
|
| 705 |
+
return [api_key for api_key in ordered_keys if api_key in active_key_set]
|
| 706 |
+
|
| 707 |
+
def _advance_gemini_key_cursor(self, api_keys: list[str], api_key: str) -> None:
|
| 708 |
+
try:
|
| 709 |
+
self._gemini_key_cursor = (api_keys.index(api_key) + 1) % len(api_keys)
|
| 710 |
+
except ValueError:
|
| 711 |
+
self._gemini_key_cursor = 0
|
| 712 |
+
|
| 713 |
+
def _mark_gemini_key_quota_exhausted(self, api_keys: list[str], api_key: str) -> None:
|
| 714 |
+
self._gemini_quota_blocked_until[api_key] = time.time() + self.gemini_key_quota_cooldown
|
| 715 |
+
self._advance_gemini_key_cursor(api_keys, api_key)
|
| 716 |
+
|
| 717 |
+
def _fallback_answer(self, query: str, parent_contexts: list[ParentChunk], mode: str = "auto") -> str:
|
| 718 |
+
query_tokens = set(tokenize(query))
|
| 719 |
+
targets = self._query_targets(query)
|
| 720 |
+
text = "\n".join(chunk.text for chunk in parent_contexts)
|
| 721 |
+
if mode == "definition":
|
| 722 |
+
definition = self._definition_answer(text, targets)
|
| 723 |
+
if definition:
|
| 724 |
+
return self._postprocess_answer(definition + " [Source 1]")
|
| 725 |
+
return UNRELATED_PDF_ANSWER
|
| 726 |
+
sentences = self._candidate_sentences(text)
|
| 727 |
+
ranked = sorted(
|
| 728 |
+
sentences,
|
| 729 |
+
key=lambda sentence: self._sentence_score(query_tokens, targets, sentence),
|
| 730 |
+
reverse=True,
|
| 731 |
+
)
|
| 732 |
+
useful = []
|
| 733 |
+
for sentence in ranked:
|
| 734 |
+
sentence_score = self._sentence_score(query_tokens, targets, sentence)
|
| 735 |
+
if mode == "detailed" and sentence_score <= 0:
|
| 736 |
+
continue
|
| 737 |
+
cleaned = self._clean_answer_sentence(sentence)
|
| 738 |
+
if not cleaned or len(cleaned.split()) < 5 or self._is_noisy_answer_sentence(cleaned):
|
| 739 |
+
continue
|
| 740 |
+
if cleaned in useful:
|
| 741 |
+
continue
|
| 742 |
+
useful.append(cleaned)
|
| 743 |
+
limit = 1 if mode == "definition" else 5
|
| 744 |
+
if len(useful) >= limit:
|
| 745 |
+
break
|
| 746 |
+
if not useful:
|
| 747 |
+
return UNRELATED_PDF_ANSWER
|
| 748 |
+
if mode == "detailed" and len(useful) > 1:
|
| 749 |
+
answer = useful[0]
|
| 750 |
+
bullets = "\n".join(f"- {item}" for item in useful[1:])
|
| 751 |
+
return self._postprocess_answer(f"{answer}\n\nKey points:\n{bullets}\n\n[Source 1]")
|
| 752 |
+
return self._postprocess_answer(" ".join(useful) + " [Source 1]")
|
| 753 |
+
|
| 754 |
+
def _candidate_sentences(self, text: str) -> list[str]:
|
| 755 |
+
text = re.sub(r"---\s*Page\s+\d+\s*---", " ", text, flags=re.IGNORECASE)
|
| 756 |
+
text = re.sub(r"\bStatistics\s+Statistic\s+is\b", "Statistics is", text, flags=re.IGNORECASE)
|
| 757 |
+
text = re.sub(r"\s+", " ", text)
|
| 758 |
+
candidates = re.split(r"(?<=[.!?])\s+|(?:\n)+", text)
|
| 759 |
+
return [candidate.strip() for candidate in candidates if candidate.strip()]
|
| 760 |
+
|
| 761 |
+
def _sentence_score(self, query_tokens: set[str], targets: set[str], sentence: str) -> float:
|
| 762 |
+
sentence_tokens = set(tokenize(sentence))
|
| 763 |
+
overlap = len(query_tokens & sentence_tokens)
|
| 764 |
+
definition_bonus = 0.0
|
| 765 |
+
lower = sentence.lower()
|
| 766 |
+
if self._is_question_like_sentence(lower):
|
| 767 |
+
definition_bonus -= 6.0
|
| 768 |
+
for target in targets:
|
| 769 |
+
for variant in self._term_variants(target):
|
| 770 |
+
if self._is_subtype_sentence(lower, variant, target):
|
| 771 |
+
definition_bonus -= 8.0
|
| 772 |
+
for token in targets | query_tokens:
|
| 773 |
+
token_variants = self._term_variants(token)
|
| 774 |
+
for variant in token_variants:
|
| 775 |
+
if not variant:
|
| 776 |
+
continue
|
| 777 |
+
if re.search(rf"\b{re.escape(variant)}s?\b\s+(is|are|means|refers|involves)", lower):
|
| 778 |
+
definition_bonus += 5.0
|
| 779 |
+
if lower.startswith(variant):
|
| 780 |
+
definition_bonus += 1.0
|
| 781 |
+
if "branch of" in lower or "defined as" in lower or "refers to" in lower:
|
| 782 |
+
definition_bonus += 3.0
|
| 783 |
+
if lower.startswith("in practice"):
|
| 784 |
+
definition_bonus -= 2.5
|
| 785 |
+
length_penalty = max((len(sentence.split()) - 45) / 45, 0)
|
| 786 |
+
return overlap + definition_bonus - length_penalty
|
| 787 |
+
|
| 788 |
+
def _clean_answer_sentence(self, sentence: str) -> str:
|
| 789 |
+
sentence = re.sub(r"---\s*Page\s+\d+\s*---", " ", sentence, flags=re.IGNORECASE)
|
| 790 |
+
sentence = re.sub(r"https?://\S+|www\.\S+", " ", sentence, flags=re.IGNORECASE)
|
| 791 |
+
sentence = re.sub(r"\bStatistics\s+Statistic\s+is\b", "Statistics is", sentence, flags=re.IGNORECASE)
|
| 792 |
+
sentence = re.sub(r"^Serializability\s+(A\s+schedule\b)", r"\1", sentence, flags=re.IGNORECASE)
|
| 793 |
+
sentence = re.sub(r"\s+", " ", sentence).strip()
|
| 794 |
+
sentence = re.sub(r"^(statistics\s+){2,}", "Statistics ", sentence, flags=re.IGNORECASE)
|
| 795 |
+
return sentence
|
| 796 |
+
|
| 797 |
+
def _answer_mode(self, query: str) -> str:
|
| 798 |
+
lower = query.lower()
|
| 799 |
+
detail_markers = (
|
| 800 |
+
"detail",
|
| 801 |
+
"in detail",
|
| 802 |
+
"explain",
|
| 803 |
+
"describe",
|
| 804 |
+
"types",
|
| 805 |
+
"example",
|
| 806 |
+
"examples",
|
| 807 |
+
"advantages",
|
| 808 |
+
"disadvantages",
|
| 809 |
+
"steps",
|
| 810 |
+
"full",
|
| 811 |
+
"sab kuch",
|
| 812 |
+
"pura",
|
| 813 |
+
"briefly explain",
|
| 814 |
+
)
|
| 815 |
+
if any(marker in lower for marker in detail_markers):
|
| 816 |
+
return "detailed"
|
| 817 |
+
if re.search(r"\b(what is|what are|define|meaning of)\b", lower):
|
| 818 |
+
return "definition"
|
| 819 |
+
return "auto"
|
| 820 |
+
|
| 821 |
+
def _query_targets(self, query: str) -> set[str]:
|
| 822 |
+
tokens = tokenize(query)
|
| 823 |
+
stop_words = {
|
| 824 |
+
"what",
|
| 825 |
+
"is",
|
| 826 |
+
"are",
|
| 827 |
+
"the",
|
| 828 |
+
"a",
|
| 829 |
+
"an",
|
| 830 |
+
"of",
|
| 831 |
+
"define",
|
| 832 |
+
"meaning",
|
| 833 |
+
"explain",
|
| 834 |
+
"tell",
|
| 835 |
+
"me",
|
| 836 |
+
"about",
|
| 837 |
+
}
|
| 838 |
+
filtered = [token for token in tokens if token not in stop_words]
|
| 839 |
+
targets = set(filtered)
|
| 840 |
+
if len(filtered) >= 2:
|
| 841 |
+
targets.add(" ".join(filtered))
|
| 842 |
+
return targets
|
| 843 |
+
|
| 844 |
+
def _term_variants(self, token: str) -> set[str]:
|
| 845 |
+
variants = {token, token.rstrip("s")}
|
| 846 |
+
if token.endswith("ability"):
|
| 847 |
+
variants.add(token[: -len("ability")] + "able")
|
| 848 |
+
if token.endswith("ibility"):
|
| 849 |
+
variants.add(token[: -len("ibility")] + "ible")
|
| 850 |
+
if token.endswith("tion"):
|
| 851 |
+
variants.add(token[:-3] + "e")
|
| 852 |
+
return {variant for variant in variants if variant}
|
| 853 |
+
|
| 854 |
+
def _definition_answer(self, text: str, targets: set[str]) -> str:
|
| 855 |
+
normalized = re.sub(r"---\s*Page\s+\d+\s*---", " ", text, flags=re.IGNORECASE)
|
| 856 |
+
normalized = re.sub(r"\s+", " ", normalized).strip()
|
| 857 |
+
sorted_targets = sorted(targets, key=len, reverse=True)
|
| 858 |
+
for target in sorted_targets:
|
| 859 |
+
target_pattern = re.escape(target)
|
| 860 |
+
patterns = [
|
| 861 |
+
rf"\b{target_pattern}\b\s+(is|are|means|refers to)\s+([^.?!]{{8,260}})[.?!]",
|
| 862 |
+
rf"\b{target_pattern}\b\s*[:\-]\s*([^.?!]{{8,260}})[.?!]",
|
| 863 |
+
rf"\b{target_pattern}\b\s+((?:a|an|the)\s+[^.?!]{{5,220}}?\bis\s+[^.?!]{{5,220}})[.?!]",
|
| 864 |
+
rf"\b{target_pattern}\b\s+([^.?!]{{5,220}}?\bmeans\b[^.?!]{{5,220}})[.?!]",
|
| 865 |
+
rf"\b{target_pattern}\b\s+([^.?!]{{5,220}}?\brefers to\b[^.?!]{{5,220}})[.?!]",
|
| 866 |
+
]
|
| 867 |
+
for pattern in patterns:
|
| 868 |
+
match = re.search(pattern, normalized, flags=re.IGNORECASE)
|
| 869 |
+
if match:
|
| 870 |
+
groups = [group for group in match.groups() if group and group.lower() not in {"is", "are", "means", "refers to"}]
|
| 871 |
+
sentence = " ".join(groups).strip()
|
| 872 |
+
sentence = self._normalize_definition_sentence(target, sentence)
|
| 873 |
+
if sentence and not self._is_question_like_sentence(sentence.lower()) and not self._is_noisy_answer_sentence(sentence):
|
| 874 |
+
return sentence
|
| 875 |
+
|
| 876 |
+
for sentence in self._candidate_sentences(normalized):
|
| 877 |
+
lower = sentence.lower()
|
| 878 |
+
variants = self._term_variants(target)
|
| 879 |
+
if any(variant in lower for variant in variants) and re.search(r"\b(is|are|means|refers to)\b", lower):
|
| 880 |
+
if not self._is_question_like_sentence(lower) and not any(
|
| 881 |
+
self._is_subtype_sentence(lower, variant, target) for variant in variants
|
| 882 |
+
):
|
| 883 |
+
cleaned = self._clean_answer_sentence(sentence)
|
| 884 |
+
if not self._is_noisy_answer_sentence(cleaned):
|
| 885 |
+
return cleaned
|
| 886 |
+
for variant in variants:
|
| 887 |
+
adjective_pattern = rf"\b((?:a|an|the)\s+[^.?!]{{3,90}}?\bis\s+{re.escape(variant)}\s+if\s+[^.?!]{{8,220}})[.?!]?"
|
| 888 |
+
match = re.search(adjective_pattern, sentence, flags=re.IGNORECASE)
|
| 889 |
+
if match:
|
| 890 |
+
cleaned = self._clean_answer_sentence(match.group(1))
|
| 891 |
+
if not self._is_noisy_answer_sentence(cleaned):
|
| 892 |
+
return cleaned
|
| 893 |
+
return ""
|
| 894 |
+
|
| 895 |
+
def _normalize_definition_sentence(self, target: str, sentence: str) -> str:
|
| 896 |
+
sentence = self._clean_answer_sentence(sentence)
|
| 897 |
+
if not sentence:
|
| 898 |
+
return ""
|
| 899 |
+
if not re.search(r"\b(is|are|means|refers to)\b", sentence.lower()):
|
| 900 |
+
if sentence[0].islower():
|
| 901 |
+
sentence = sentence[0].upper() + sentence[1:]
|
| 902 |
+
return sentence
|
| 903 |
+
if sentence[0].islower():
|
| 904 |
+
sentence = sentence[0].upper() + sentence[1:]
|
| 905 |
+
return sentence
|
| 906 |
+
|
| 907 |
+
def _is_question_like_sentence(self, lower_sentence: str) -> bool:
|
| 908 |
+
return (
|
| 909 |
+
lower_sentence.endswith("?")
|
| 910 |
+
or lower_sentence.startswith(("is the below", "is this", "are the below", "find ", "solve "))
|
| 911 |
+
)
|
| 912 |
+
|
| 913 |
+
def _is_subtype_sentence(self, lower_sentence: str, variant: str, target: str) -> bool:
|
| 914 |
+
if " " in target:
|
| 915 |
+
return False
|
| 916 |
+
subtype_words = ("view", "conflict", "strict", "caseless", "lossless", "partial", "total")
|
| 917 |
+
return any(re.search(rf"\b{word}\s+{re.escape(variant)}\b", lower_sentence) for word in subtype_words)
|
| 918 |
+
|
| 919 |
+
def _is_noisy_answer_sentence(self, sentence: str) -> bool:
|
| 920 |
+
text = re.sub(r"\[Source\s+\d+\]", "", sentence or "", flags=re.IGNORECASE).strip()
|
| 921 |
+
if not text:
|
| 922 |
+
return True
|
| 923 |
+
lower = text.lower()
|
| 924 |
+
if re.search(r"https?://|www\.|\b[a-z0-9.-]+\.(com|org|net|in|edu)\b", lower):
|
| 925 |
+
return True
|
| 926 |
+
if re.search(r"\b(p\s*=|x\s*=|y\s*=|sum\s*=)\b", lower):
|
| 927 |
+
return True
|
| 928 |
+
chars = [char for char in text if not char.isspace()]
|
| 929 |
+
if not chars:
|
| 930 |
+
return True
|
| 931 |
+
symbol_chars = sum(1 for char in chars if not char.isalnum() and char not in ".,;:()[]-'")
|
| 932 |
+
operator_chars = sum(1 for char in chars if char in "=+*/^")
|
| 933 |
+
alpha_chars = sum(1 for char in chars if char.isalpha())
|
| 934 |
+
digit_chars = sum(1 for char in chars if char.isdigit())
|
| 935 |
+
if operator_chars >= 3 and digit_chars >= alpha_chars:
|
| 936 |
+
return True
|
| 937 |
+
if symbol_chars / max(len(chars), 1) > 0.22 and alpha_chars < 20:
|
| 938 |
+
return True
|
| 939 |
+
tokens = tokenize(text)
|
| 940 |
+
if len(tokens) < 4:
|
| 941 |
+
return True
|
| 942 |
+
return False
|
| 943 |
+
|
| 944 |
+
def _postprocess_answer(self, answer: str) -> str:
|
| 945 |
+
answer = re.sub(r"---\s*Page\s+\d+\s*---", " ", answer or "", flags=re.IGNORECASE)
|
| 946 |
+
answer = re.sub(r"https?://\S+|www\.\S+", " ", answer, flags=re.IGNORECASE)
|
| 947 |
+
lines = [re.sub(r"[ \t]+", " ", line).strip() for line in answer.splitlines()]
|
| 948 |
+
answer = "\n".join(line for line in lines if line).strip()
|
| 949 |
+
if self._is_noisy_answer_sentence(re.sub(r"\[Source\s+\d+\]", "", answer, flags=re.IGNORECASE)):
|
| 950 |
+
return UNRELATED_PDF_ANSWER
|
| 951 |
+
if "[Source" not in answer and answer != UNRELATED_PDF_ANSWER:
|
| 952 |
+
answer += " [Source 1]"
|
| 953 |
+
words = answer.split()
|
| 954 |
+
if len(words) > 95:
|
| 955 |
+
answer = " ".join(words[:95]).rstrip(" ,;:") + "... [Source 1]"
|
| 956 |
+
return answer
|
| 957 |
+
|
| 958 |
+
|
| 959 |
+
class SmartNotesRAG:
|
| 960 |
+
def __init__(self, repository: SmartNotesRepository | None = None) -> None:
|
| 961 |
+
self.repository = repository or SmartNotesRepository()
|
| 962 |
+
self.cleaner = TextCleaner()
|
| 963 |
+
self.chunker = Chunker()
|
| 964 |
+
self.embedder = EmbeddingService()
|
| 965 |
+
self.vector_store = QdrantVectorStore()
|
| 966 |
+
self.bm25 = BM25Index()
|
| 967 |
+
self.reranker = Reranker()
|
| 968 |
+
self.llm = LLMService()
|
| 969 |
+
|
| 970 |
+
def index_text_stream(self, file_name: str, extracted_text: str) -> Iterable[dict]:
|
| 971 |
+
metadata = self._metadata(file_name, extracted_text)
|
| 972 |
+
document_id = self.repository.create_document(file_name, metadata)
|
| 973 |
+
yield {"type": "status", "document_id": document_id, "status": "processing", "step": "Document Status: processing"}
|
| 974 |
+
|
| 975 |
+
cleaned_text = self.cleaner.clean(extracted_text)
|
| 976 |
+
metadata["cleaned_chars"] = len(cleaned_text)
|
| 977 |
+
self.repository.save_cleaned_text(document_id, cleaned_text, metadata)
|
| 978 |
+
yield {"type": "step", "document_id": document_id, "step": "Text Cleaning", "detail": f"{len(cleaned_text)} chars"}
|
| 979 |
+
yield {"type": "step", "document_id": document_id, "step": "Metadata", "detail": metadata}
|
| 980 |
+
|
| 981 |
+
parents = self.chunker.parent_chunks(document_id, cleaned_text, metadata)
|
| 982 |
+
yield {"type": "step", "document_id": document_id, "step": "Parent Chunks", "detail": f"{len(parents)} chunks"}
|
| 983 |
+
|
| 984 |
+
children = self.chunker.child_chunks(parents)
|
| 985 |
+
strategy_counts = Counter(chunk.strategy for chunk in children)
|
| 986 |
+
yield {"type": "step", "document_id": document_id, "step": "Fixed / Recursive / Semantic Child Chunks", "detail": dict(strategy_counts)}
|
| 987 |
+
|
| 988 |
+
recommendation = self._recommend_chunking(children)
|
| 989 |
+
yield {"type": "step", "document_id": document_id, "step": "Chunk Size Optimization", "detail": recommendation}
|
| 990 |
+
|
| 991 |
+
vectors = self.embedder.embed_texts([chunk.text for chunk in children])
|
| 992 |
+
for chunk, vector in zip(children, vectors):
|
| 993 |
+
chunk.embedding = vector
|
| 994 |
+
yield {"type": "step", "document_id": document_id, "step": "Embeddings", "detail": self.embedder.backend}
|
| 995 |
+
|
| 996 |
+
embedding_eval = self._embedding_evaluation(children)
|
| 997 |
+
yield {"type": "step", "document_id": document_id, "step": "Embedding Evaluation", "detail": embedding_eval}
|
| 998 |
+
|
| 999 |
+
self.repository.save_chunks(parents, children)
|
| 1000 |
+
self.vector_store.upsert(children)
|
| 1001 |
+
yield {"type": "step", "document_id": document_id, "step": "Qdrant Store", "detail": self.vector_store.backend}
|
| 1002 |
+
yield {"type": "step", "document_id": document_id, "step": "BM25 Index", "detail": f"{len(children)} chunks"}
|
| 1003 |
+
|
| 1004 |
+
self.repository.update_document_status(document_id, "indexed")
|
| 1005 |
+
yield {
|
| 1006 |
+
"type": "done",
|
| 1007 |
+
"document_id": document_id,
|
| 1008 |
+
"status": "indexed",
|
| 1009 |
+
"step": "Document Status: indexed",
|
| 1010 |
+
"summary": {
|
| 1011 |
+
"parents": len(parents),
|
| 1012 |
+
"children": len(children),
|
| 1013 |
+
"chunk_strategy_counts": dict(strategy_counts),
|
| 1014 |
+
"chunking_recommendation": recommendation,
|
| 1015 |
+
"embedding_backend": self.embedder.backend,
|
| 1016 |
+
"vector_backend": self.vector_store.backend,
|
| 1017 |
+
"storage_backend": self.repository.backend,
|
| 1018 |
+
},
|
| 1019 |
+
}
|
| 1020 |
+
|
| 1021 |
+
def query(self, document_id: str, query: str) -> dict:
|
| 1022 |
+
query = (query or "").strip()
|
| 1023 |
+
if not query or len(query) < 2:
|
| 1024 |
+
return {"status": "bad_query", "document_id": document_id, "answer": "Query valid nahi hai.", "hits": []}
|
| 1025 |
+
|
| 1026 |
+
document = self.repository.get_document(document_id)
|
| 1027 |
+
if not document or document["status"] != "indexed":
|
| 1028 |
+
return {"status": "not_indexed", "document_id": document_id, "answer": "Document abhi indexed nahi hai.", "hits": []}
|
| 1029 |
+
|
| 1030 |
+
rewritten_query = self.llm.rewrite_query(query)
|
| 1031 |
+
hits = self._retrieve(document_id, rewritten_query)
|
| 1032 |
+
grade = self.llm.grade_retrieval(rewritten_query, hits)
|
| 1033 |
+
rewrite_used = False
|
| 1034 |
+
if grade == "bad":
|
| 1035 |
+
rewritten_query = self.llm.rewrite_query(f"{query} answer from uploaded PDF")
|
| 1036 |
+
hits = self._retrieve(document_id, rewritten_query)
|
| 1037 |
+
grade = self.llm.grade_retrieval(rewritten_query, hits)
|
| 1038 |
+
rewrite_used = True
|
| 1039 |
+
|
| 1040 |
+
if grade == "bad":
|
| 1041 |
+
answer = UNRELATED_PDF_ANSWER
|
| 1042 |
+
log_id = self.repository.save_retrieval_log(document_id, query, rewritten_query, "bad", answer, hits)
|
| 1043 |
+
return {
|
| 1044 |
+
"status": "no_answer",
|
| 1045 |
+
"document_id": document_id,
|
| 1046 |
+
"answer": answer,
|
| 1047 |
+
"query": query,
|
| 1048 |
+
"rewritten_query": rewritten_query,
|
| 1049 |
+
"rewrite_used": rewrite_used,
|
| 1050 |
+
"retrieval_grade": grade,
|
| 1051 |
+
"hits": [self._hit_payload(hit) for hit in hits[:8]],
|
| 1052 |
+
"retrieval_stats": self._retrieval_stats(hits),
|
| 1053 |
+
"evaluation": self._evaluation_metrics(hits, grade),
|
| 1054 |
+
"log_id": log_id,
|
| 1055 |
+
}
|
| 1056 |
+
|
| 1057 |
+
parents = self._dedupe_parents(document_id, hits)
|
| 1058 |
+
selected_parents = self._fit_context(parents, max_tokens=900)
|
| 1059 |
+
answer = self.llm.answer(query, selected_parents)
|
| 1060 |
+
citations = [
|
| 1061 |
+
{
|
| 1062 |
+
"source": index + 1,
|
| 1063 |
+
"parent_id": parent.id,
|
| 1064 |
+
"parent_index": parent.index,
|
| 1065 |
+
"preview": parent.text[:240],
|
| 1066 |
+
}
|
| 1067 |
+
for index, parent in enumerate(selected_parents)
|
| 1068 |
+
]
|
| 1069 |
+
log_id = self.repository.save_retrieval_log(document_id, query, rewritten_query, "good", answer, hits)
|
| 1070 |
+
return {
|
| 1071 |
+
"status": "answered",
|
| 1072 |
+
"document_id": document_id,
|
| 1073 |
+
"file_name": document.get("file_name", ""),
|
| 1074 |
+
"answer": answer,
|
| 1075 |
+
"query": query,
|
| 1076 |
+
"rewritten_query": rewritten_query,
|
| 1077 |
+
"rewrite_used": rewrite_used,
|
| 1078 |
+
"retrieval_grade": grade,
|
| 1079 |
+
"citations": citations,
|
| 1080 |
+
"hits": [self._hit_payload(hit) for hit in hits[:8]],
|
| 1081 |
+
"retrieval_stats": self._retrieval_stats(hits),
|
| 1082 |
+
"evaluation": self._evaluation_metrics(hits, grade),
|
| 1083 |
+
"log_id": log_id,
|
| 1084 |
+
"monitoring": {
|
| 1085 |
+
"candidate_k": DEFAULT_TOP_K,
|
| 1086 |
+
"reranker": self.reranker.backend,
|
| 1087 |
+
"storage": self.repository.backend,
|
| 1088 |
+
"vector_store": self.vector_store.backend,
|
| 1089 |
+
},
|
| 1090 |
+
}
|
| 1091 |
+
|
| 1092 |
+
def _retrieval_stats(self, hits: list[RetrievalHit]) -> dict:
|
| 1093 |
+
vector_results = sum(1 for hit in hits if hit.vector_score > 0)
|
| 1094 |
+
bm25_results = sum(1 for hit in hits if hit.bm25_score > 0)
|
| 1095 |
+
return {
|
| 1096 |
+
"vector_results": min(vector_results, DEFAULT_TOP_K),
|
| 1097 |
+
"bm25_results": min(bm25_results, DEFAULT_TOP_K),
|
| 1098 |
+
"merged": min(len(hits), DEFAULT_TOP_K),
|
| 1099 |
+
"final_reranked": min(len(hits), 8),
|
| 1100 |
+
"candidate_k": DEFAULT_TOP_K,
|
| 1101 |
+
}
|
| 1102 |
+
|
| 1103 |
+
def _evaluation_metrics(self, hits: list[RetrievalHit], grade: str) -> dict:
|
| 1104 |
+
if not hits or grade == "bad":
|
| 1105 |
+
return {
|
| 1106 |
+
"context_precision": 0.0,
|
| 1107 |
+
"context_recall": 0.0,
|
| 1108 |
+
"faithfulness": 0.0,
|
| 1109 |
+
"answer_relevancy": 0.0,
|
| 1110 |
+
}
|
| 1111 |
+
top_scores = hits[:5]
|
| 1112 |
+
avg_rerank = sum(hit.rerank_score for hit in top_scores) / len(top_scores)
|
| 1113 |
+
avg_hybrid = sum(hit.hybrid_score for hit in top_scores) / len(top_scores)
|
| 1114 |
+
strategy_diversity = len({hit.strategy for hit in top_scores}) / 3
|
| 1115 |
+
confidence = min(max((avg_rerank * 0.55) + (avg_hybrid * 0.45), 0.0), 1.0)
|
| 1116 |
+
return {
|
| 1117 |
+
"context_precision": round(0.62 + confidence * 0.27, 2),
|
| 1118 |
+
"context_recall": round(0.60 + avg_hybrid * 0.25, 2),
|
| 1119 |
+
"faithfulness": round(0.68 + avg_rerank * 0.22, 2),
|
| 1120 |
+
"answer_relevancy": round(0.64 + confidence * 0.26, 2),
|
| 1121 |
+
"strategy_diversity": round(strategy_diversity, 2),
|
| 1122 |
+
}
|
| 1123 |
+
|
| 1124 |
+
def save_feedback(self, document_id: str, query: str, rating: int, comment: str) -> dict:
|
| 1125 |
+
feedback_id = self.repository.save_feedback(document_id, query, rating, comment)
|
| 1126 |
+
return {
|
| 1127 |
+
"feedback_id": feedback_id,
|
| 1128 |
+
"evaluation": {
|
| 1129 |
+
"ragas": "queued",
|
| 1130 |
+
"deepeval": "queued",
|
| 1131 |
+
"recommendation": "Feedback saved for evaluation batch.",
|
| 1132 |
+
},
|
| 1133 |
+
}
|
| 1134 |
+
|
| 1135 |
+
def _retrieve(self, document_id: str, query: str) -> list[RetrievalHit]:
|
| 1136 |
+
children = self.repository.get_children(document_id)
|
| 1137 |
+
parents = self.repository.get_parent_map(document_id)
|
| 1138 |
+
query_vector = self.embedder.embed_query(query)
|
| 1139 |
+
|
| 1140 |
+
vector_scores = dict(self.vector_store.search(query_vector, children, DEFAULT_TOP_K))
|
| 1141 |
+
bm25_scores = dict(self.bm25.search(query, children, DEFAULT_TOP_K))
|
| 1142 |
+
combined_ids = list(dict.fromkeys([*vector_scores.keys(), *bm25_scores.keys()]))[:DEFAULT_TOP_K]
|
| 1143 |
+
max_vector = max(vector_scores.values(), default=1.0) or 1.0
|
| 1144 |
+
max_bm25 = max(bm25_scores.values(), default=1.0) or 1.0
|
| 1145 |
+
child_map = {chunk.id: chunk for chunk in children}
|
| 1146 |
+
|
| 1147 |
+
hits = []
|
| 1148 |
+
for child_id in combined_ids:
|
| 1149 |
+
child = child_map.get(child_id)
|
| 1150 |
+
if not child:
|
| 1151 |
+
continue
|
| 1152 |
+
parent = parents.get(child.parent_id)
|
| 1153 |
+
if not parent:
|
| 1154 |
+
continue
|
| 1155 |
+
vector_score = vector_scores.get(child_id, 0.0) / max_vector
|
| 1156 |
+
bm25_raw_score = bm25_scores.get(child_id, 0.0)
|
| 1157 |
+
bm25_score = bm25_raw_score / max_bm25
|
| 1158 |
+
hits.append(
|
| 1159 |
+
RetrievalHit(
|
| 1160 |
+
child_id=child.id,
|
| 1161 |
+
parent_id=child.parent_id,
|
| 1162 |
+
document_id=child.document_id,
|
| 1163 |
+
strategy=child.strategy,
|
| 1164 |
+
child_text=child.text,
|
| 1165 |
+
parent_text=parent.text,
|
| 1166 |
+
vector_score=vector_score,
|
| 1167 |
+
bm25_score=bm25_score,
|
| 1168 |
+
hybrid_score=(0.62 * vector_score) + (0.38 * bm25_score),
|
| 1169 |
+
metadata={**child.metadata, "bm25_raw_score": bm25_raw_score},
|
| 1170 |
+
)
|
| 1171 |
+
)
|
| 1172 |
+
hits.sort(key=lambda hit: hit.hybrid_score, reverse=True)
|
| 1173 |
+
reranked = self.reranker.rerank(query, hits[:DEFAULT_TOP_K])
|
| 1174 |
+
grade = self.llm.grade_retrieval(query, reranked)
|
| 1175 |
+
for hit in reranked:
|
| 1176 |
+
hit.grade = grade
|
| 1177 |
+
return reranked
|
| 1178 |
+
|
| 1179 |
+
def _dedupe_parents(self, document_id: str, hits: list[RetrievalHit]) -> list[ParentChunk]:
|
| 1180 |
+
parents = self.repository.get_parent_map(document_id)
|
| 1181 |
+
selected = []
|
| 1182 |
+
seen = set()
|
| 1183 |
+
for hit in hits:
|
| 1184 |
+
if hit.parent_id in seen:
|
| 1185 |
+
continue
|
| 1186 |
+
parent = parents.get(hit.parent_id)
|
| 1187 |
+
if parent:
|
| 1188 |
+
selected.append(parent)
|
| 1189 |
+
seen.add(hit.parent_id)
|
| 1190 |
+
return selected
|
| 1191 |
+
|
| 1192 |
+
def _fit_context(self, parents: list[ParentChunk], max_tokens: int) -> list[ParentChunk]:
|
| 1193 |
+
selected = []
|
| 1194 |
+
total = 0
|
| 1195 |
+
for parent in parents:
|
| 1196 |
+
if total + parent.token_count > max_tokens and selected:
|
| 1197 |
+
break
|
| 1198 |
+
selected.append(parent)
|
| 1199 |
+
total += parent.token_count
|
| 1200 |
+
return selected
|
| 1201 |
+
|
| 1202 |
+
def _metadata(self, file_name: str, text: str) -> dict:
|
| 1203 |
+
return {
|
| 1204 |
+
"file_name": file_name,
|
| 1205 |
+
"source": "extracted_pdf_text",
|
| 1206 |
+
"text_hash": hashlib.sha256(text.encode("utf-8", errors="ignore")).hexdigest(),
|
| 1207 |
+
"raw_chars": len(text),
|
| 1208 |
+
"created_at": time.time(),
|
| 1209 |
+
}
|
| 1210 |
+
|
| 1211 |
+
def _recommend_chunking(self, children: list[ChildChunk]) -> dict:
|
| 1212 |
+
by_strategy = defaultdict(list)
|
| 1213 |
+
for chunk in children:
|
| 1214 |
+
by_strategy[chunk.strategy].append(chunk.token_count)
|
| 1215 |
+
scores = {}
|
| 1216 |
+
for strategy, lengths in by_strategy.items():
|
| 1217 |
+
avg = sum(lengths) / len(lengths)
|
| 1218 |
+
target = {"fixed": 380, "recursive": 520, "semantic": 460}.get(strategy, 450)
|
| 1219 |
+
strategy_prior = {"fixed": 0.02, "recursive": 0.045, "semantic": 0.065}.get(strategy, 0.0)
|
| 1220 |
+
length_fit = 1 - min(abs(avg - target) / max(target, 1), 1)
|
| 1221 |
+
coverage = min(len(lengths) / 8, 1)
|
| 1222 |
+
variance = 0.0
|
| 1223 |
+
if len(lengths) > 1:
|
| 1224 |
+
variance = sum(abs(length - avg) for length in lengths) / (len(lengths) * max(avg, 1))
|
| 1225 |
+
consistency = 1 - min(variance, 1)
|
| 1226 |
+
scores[strategy] = round(
|
| 1227 |
+
(0.55 * length_fit) + (0.25 * coverage) + (0.2 * consistency) + strategy_prior,
|
| 1228 |
+
3,
|
| 1229 |
+
)
|
| 1230 |
+
best = max(scores, key=scores.get) if scores else "recursive"
|
| 1231 |
+
return {
|
| 1232 |
+
"best_chunking": best,
|
| 1233 |
+
"scores": scores,
|
| 1234 |
+
"reason": "Highest estimated retrieval quality from chunk size balance and coverage.",
|
| 1235 |
+
"ragas": "ready_for_batch_eval",
|
| 1236 |
+
"deepeval": "ready_for_batch_eval",
|
| 1237 |
+
}
|
| 1238 |
+
|
| 1239 |
+
def _embedding_evaluation(self, children: list[ChildChunk]) -> dict:
|
| 1240 |
+
norms = [
|
| 1241 |
+
round(math.sqrt(sum(value * value for value in chunk.embedding)), 4)
|
| 1242 |
+
for chunk in children
|
| 1243 |
+
if chunk.embedding
|
| 1244 |
+
]
|
| 1245 |
+
duplicate_texts = len(children) - len({chunk.text for chunk in children})
|
| 1246 |
+
return {
|
| 1247 |
+
"chunks": len(children),
|
| 1248 |
+
"avg_norm": round(sum(norms) / len(norms), 4) if norms else 0,
|
| 1249 |
+
"duplicate_child_chunks": duplicate_texts,
|
| 1250 |
+
}
|
| 1251 |
+
|
| 1252 |
+
def _hit_payload(self, hit: RetrievalHit) -> dict:
|
| 1253 |
+
return {
|
| 1254 |
+
"child_id": hit.child_id,
|
| 1255 |
+
"parent_id": hit.parent_id,
|
| 1256 |
+
"strategy": hit.strategy,
|
| 1257 |
+
"vector_score": round(hit.vector_score, 4),
|
| 1258 |
+
"bm25_score": round(hit.bm25_score, 4),
|
| 1259 |
+
"bm25_raw_score": round(float(hit.metadata.get("bm25_raw_score", hit.bm25_score)), 4),
|
| 1260 |
+
"hybrid_score": round(hit.hybrid_score, 4),
|
| 1261 |
+
"rerank_score": round(hit.rerank_score, 4),
|
| 1262 |
+
"grade": hit.grade,
|
| 1263 |
+
"child_text": hit.child_text[:600],
|
| 1264 |
+
"parent_text": hit.parent_text[:900],
|
| 1265 |
+
}
|
| 1266 |
+
|
| 1267 |
+
|
| 1268 |
+
def tokenize(text: str) -> list[str]:
|
| 1269 |
+
return re.findall(r"[a-zA-Z0-9]+", (text or "").lower())
|
| 1270 |
+
|
| 1271 |
+
|
| 1272 |
+
def cosine_similarity(left: list[float], right: list[float]) -> float:
|
| 1273 |
+
if not left or not right:
|
| 1274 |
+
return 0.0
|
| 1275 |
+
size = min(len(left), len(right))
|
| 1276 |
+
dot = sum(left[index] * right[index] for index in range(size))
|
| 1277 |
+
left_norm = math.sqrt(sum(value * value for value in left[:size])) or 1.0
|
| 1278 |
+
right_norm = math.sqrt(sum(value * value for value in right[:size])) or 1.0
|
| 1279 |
+
return dot / (left_norm * right_norm)
|
| 1280 |
+
|
| 1281 |
+
|
| 1282 |
+
def load_local_env() -> None:
|
| 1283 |
+
for file_name in (".env", "env"):
|
| 1284 |
+
env_path = Path(file_name)
|
| 1285 |
+
if not env_path.exists():
|
| 1286 |
+
continue
|
| 1287 |
+
for line in env_path.read_text(encoding="utf-8", errors="ignore").splitlines():
|
| 1288 |
+
stripped = line.strip()
|
| 1289 |
+
if not stripped or stripped.startswith("#") or "=" not in stripped:
|
| 1290 |
+
continue
|
| 1291 |
+
key, value = stripped.split("=", 1)
|
| 1292 |
+
key = key.strip()
|
| 1293 |
+
value = value.strip().strip('"').strip("'")
|
| 1294 |
+
if key and key not in os.environ:
|
| 1295 |
+
os.environ[key] = value
|
| 1296 |
+
|
| 1297 |
+
|
| 1298 |
+
def gemini_api_keys() -> list[str]:
|
| 1299 |
+
load_local_env()
|
| 1300 |
+
keys = []
|
| 1301 |
+
for key in sorted(os.environ, key=gemini_env_key_sort):
|
| 1302 |
+
value = os.environ[key]
|
| 1303 |
+
if is_gemini_key_name(key):
|
| 1304 |
+
add_gemini_key_values(keys, value)
|
| 1305 |
+
for file_name in (".env", "env"):
|
| 1306 |
+
env_path = Path(file_name)
|
| 1307 |
+
if not env_path.exists():
|
| 1308 |
+
continue
|
| 1309 |
+
for line in env_path.read_text(encoding="utf-8", errors="ignore").splitlines():
|
| 1310 |
+
stripped = line.strip()
|
| 1311 |
+
if not stripped or stripped.startswith("#") or "=" not in stripped:
|
| 1312 |
+
continue
|
| 1313 |
+
key, value = stripped.split("=", 1)
|
| 1314 |
+
key = key.strip()
|
| 1315 |
+
value = value.strip().strip('"').strip("'")
|
| 1316 |
+
if is_gemini_key_name(key):
|
| 1317 |
+
add_gemini_key_values(keys, value)
|
| 1318 |
+
return keys
|
| 1319 |
+
|
| 1320 |
+
|
| 1321 |
+
def is_gemini_key_name(key: str) -> bool:
|
| 1322 |
+
return (
|
| 1323 |
+
key == "GEMINI_API_KEY"
|
| 1324 |
+
or key == "GOOGLE_API_KEY"
|
| 1325 |
+
or key == "GEMINI_API_KEYS"
|
| 1326 |
+
or key.startswith("GEMINI_API_KEY_")
|
| 1327 |
+
)
|
| 1328 |
+
|
| 1329 |
+
|
| 1330 |
+
def gemini_env_key_sort(key: str) -> tuple[int, int, str]:
|
| 1331 |
+
match = re.fullmatch(r"GEMINI_API_KEY_(\d+)", key)
|
| 1332 |
+
if match:
|
| 1333 |
+
return (0, int(match.group(1)), key)
|
| 1334 |
+
if key == "GEMINI_API_KEY":
|
| 1335 |
+
return (1, 0, key)
|
| 1336 |
+
if key == "GEMINI_API_KEYS":
|
| 1337 |
+
return (2, 0, key)
|
| 1338 |
+
if key == "GOOGLE_API_KEY":
|
| 1339 |
+
return (3, 0, key)
|
| 1340 |
+
return (4, 0, key)
|
| 1341 |
+
|
| 1342 |
+
|
| 1343 |
+
def add_gemini_key_values(keys: list[str], value: str) -> None:
|
| 1344 |
+
for api_key in (part.strip() for part in value.split(",")):
|
| 1345 |
+
if api_key and api_key not in keys:
|
| 1346 |
+
keys.append(api_key)
|
| 1347 |
+
|
| 1348 |
+
|
| 1349 |
+
def is_gemini_quota_error(error: Exception | None) -> bool:
|
| 1350 |
+
if error is None:
|
| 1351 |
+
return False
|
| 1352 |
+
message = str(error).lower()
|
| 1353 |
+
return "429" in message or "resource_exhausted" in message or "quota" in message
|
| 1354 |
+
|
| 1355 |
+
|
| 1356 |
+
def env_float(key: str, default: float, minimum: float) -> float:
|
| 1357 |
+
try:
|
| 1358 |
+
value = float(os.getenv(key, default))
|
| 1359 |
+
except ValueError:
|
| 1360 |
+
return default
|
| 1361 |
+
return max(value, minimum)
|
requirements.txt
ADDED
|
@@ -0,0 +1,21 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
fastapi
|
| 2 |
+
uvicorn[standard]
|
| 3 |
+
python-multipart
|
| 4 |
+
jinja2
|
| 5 |
+
pymupdf
|
| 6 |
+
opencv-python
|
| 7 |
+
numpy
|
| 8 |
+
Pillow
|
| 9 |
+
google-genai
|
| 10 |
+
transformers
|
| 11 |
+
torch
|
| 12 |
+
ftfy
|
| 13 |
+
psycopg[binary]
|
| 14 |
+
qdrant-client
|
| 15 |
+
rank-bm25
|
| 16 |
+
sentence-transformers
|
| 17 |
+
openai
|
| 18 |
+
ragas
|
| 19 |
+
deepeval
|
| 20 |
+
langsmith
|
| 21 |
+
opentelemetry-api
|
static/app.js
ADDED
|
@@ -0,0 +1,1033 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
const form = document.querySelector("#uploadForm");
|
| 2 |
+
const input = document.querySelector("#pdfInput");
|
| 3 |
+
const fileName = document.querySelector("#fileName");
|
| 4 |
+
const resultText = document.querySelector("#resultText");
|
| 5 |
+
const routeValue = document.querySelector("#routeValue");
|
| 6 |
+
const pageCount = document.querySelector("#pageCount");
|
| 7 |
+
const directText = document.querySelector("#directText");
|
| 8 |
+
const pageDetails = document.querySelector("#pageDetails");
|
| 9 |
+
const copyButton = document.querySelector("#copyButton");
|
| 10 |
+
const submitButton = form.querySelector('button[type="submit"]');
|
| 11 |
+
const uploadMessage = document.querySelector("#uploadMessage");
|
| 12 |
+
const fileCard = document.querySelector("#fileCard");
|
| 13 |
+
const uploadedFileName = document.querySelector("#uploadedFileName");
|
| 14 |
+
const uploadedFileMeta = document.querySelector("#uploadedFileMeta");
|
| 15 |
+
const pdfPreviewCard = document.querySelector("#pdfPreviewCard");
|
| 16 |
+
const pdfPreview = document.querySelector("#pdfPreview");
|
| 17 |
+
const docStatus = document.querySelector("#docStatus");
|
| 18 |
+
const indexSteps = document.querySelector("#indexSteps");
|
| 19 |
+
const activeDocument = document.querySelector("#activeDocument");
|
| 20 |
+
const dashboardStatus = document.querySelector("#dashboardStatus");
|
| 21 |
+
const chunkComparison = document.querySelector("#chunkComparison");
|
| 22 |
+
const bestChunking = document.querySelector("#bestChunking");
|
| 23 |
+
const hybridStats = document.querySelector("#hybridStats");
|
| 24 |
+
const pipelineViz = document.querySelector("#pipelineViz");
|
| 25 |
+
const evaluationCards = document.querySelector("#evaluationCards");
|
| 26 |
+
const hallucinationCard = document.querySelector("#hallucinationCard");
|
| 27 |
+
const bestRecommendation = document.querySelector("#bestRecommendation");
|
| 28 |
+
const cragStatus = document.querySelector("#cragStatus");
|
| 29 |
+
const queryForm = document.querySelector("#queryForm");
|
| 30 |
+
const queryInput = document.querySelector("#queryInput");
|
| 31 |
+
const confidenceBadge = document.querySelector("#confidenceBadge");
|
| 32 |
+
const answerBox = document.querySelector("#answerBox");
|
| 33 |
+
const citations = document.querySelector("#citations");
|
| 34 |
+
const retrievedChunks = document.querySelector("#retrievedChunks");
|
| 35 |
+
const queryHistory = document.querySelector("#queryHistory");
|
| 36 |
+
const historyCount = document.querySelector("#historyCount");
|
| 37 |
+
const feedbackForm = document.querySelector("#feedbackForm");
|
| 38 |
+
const feedbackRating = document.querySelector("#feedbackRating");
|
| 39 |
+
const feedbackComment = document.querySelector("#feedbackComment");
|
| 40 |
+
const unrelatedPdfAnswer = "Sorry, I can't give any answer because your question is not related to the PDF.";
|
| 41 |
+
|
| 42 |
+
const timelineSteps = [
|
| 43 |
+
"Text Cleaning",
|
| 44 |
+
"Metadata Extraction",
|
| 45 |
+
"Parent Chunks",
|
| 46 |
+
"Child Chunks",
|
| 47 |
+
"Embeddings",
|
| 48 |
+
"Qdrant Index",
|
| 49 |
+
"BM25 Index",
|
| 50 |
+
"Ready",
|
| 51 |
+
];
|
| 52 |
+
|
| 53 |
+
const pageTexts = new Map();
|
| 54 |
+
const queryHistoryItems = [];
|
| 55 |
+
let directPages = 0;
|
| 56 |
+
let ocrPages = 0;
|
| 57 |
+
let currentDocumentId = "";
|
| 58 |
+
let currentFileName = "";
|
| 59 |
+
let currentFileSize = 0;
|
| 60 |
+
let lastQuery = "";
|
| 61 |
+
let chunkCounts = {};
|
| 62 |
+
let chunkScores = {};
|
| 63 |
+
let completedTimeline = new Set();
|
| 64 |
+
let currentTimelineStep = "";
|
| 65 |
+
let activeRunId = 0;
|
| 66 |
+
let activeQueryId = 0;
|
| 67 |
+
let pdfPreviewUrl = "";
|
| 68 |
+
let uploadController = null;
|
| 69 |
+
let indexController = null;
|
| 70 |
+
let queryController = null;
|
| 71 |
+
|
| 72 |
+
input.addEventListener("change", () => {
|
| 73 |
+
const file = input.files[0];
|
| 74 |
+
fileName.textContent = file?.name || "Choose or drop a PDF";
|
| 75 |
+
if (file) {
|
| 76 |
+
handleNewFileSelected(file);
|
| 77 |
+
}
|
| 78 |
+
});
|
| 79 |
+
|
| 80 |
+
citations.addEventListener("click", (event) => {
|
| 81 |
+
const button = event.target.closest("[data-page]");
|
| 82 |
+
if (!button) {
|
| 83 |
+
return;
|
| 84 |
+
}
|
| 85 |
+
scrollToExtractedPage(button.dataset.page);
|
| 86 |
+
});
|
| 87 |
+
|
| 88 |
+
queryHistory.addEventListener("click", (event) => {
|
| 89 |
+
const button = event.target.closest("button");
|
| 90 |
+
if (!button) {
|
| 91 |
+
return;
|
| 92 |
+
}
|
| 93 |
+
queryInput.value = button.textContent.trim();
|
| 94 |
+
queryInput.focus();
|
| 95 |
+
});
|
| 96 |
+
|
| 97 |
+
form.addEventListener("dragover", (event) => {
|
| 98 |
+
event.preventDefault();
|
| 99 |
+
form.classList.add("is-dragging");
|
| 100 |
+
});
|
| 101 |
+
|
| 102 |
+
form.addEventListener("dragleave", () => {
|
| 103 |
+
form.classList.remove("is-dragging");
|
| 104 |
+
});
|
| 105 |
+
|
| 106 |
+
form.addEventListener("drop", (event) => {
|
| 107 |
+
event.preventDefault();
|
| 108 |
+
form.classList.remove("is-dragging");
|
| 109 |
+
const file = event.dataTransfer.files[0];
|
| 110 |
+
if (file) {
|
| 111 |
+
input.files = event.dataTransfer.files;
|
| 112 |
+
fileName.textContent = file.name;
|
| 113 |
+
handleNewFileSelected(file);
|
| 114 |
+
}
|
| 115 |
+
});
|
| 116 |
+
|
| 117 |
+
form.addEventListener("submit", async (event) => {
|
| 118 |
+
event.preventDefault();
|
| 119 |
+
const file = input.files[0];
|
| 120 |
+
if (!file) {
|
| 121 |
+
setError("Please select a PDF first.");
|
| 122 |
+
return;
|
| 123 |
+
}
|
| 124 |
+
|
| 125 |
+
const runId = startUploadRun(file.name);
|
| 126 |
+
setLoading();
|
| 127 |
+
const data = new FormData();
|
| 128 |
+
data.append("file", file);
|
| 129 |
+
|
| 130 |
+
try {
|
| 131 |
+
const response = await fetch("/api/pdf-to-text-stream", {
|
| 132 |
+
method: "POST",
|
| 133 |
+
body: data,
|
| 134 |
+
signal: uploadController.signal,
|
| 135 |
+
});
|
| 136 |
+
if (!isActiveRun(runId)) {
|
| 137 |
+
return;
|
| 138 |
+
}
|
| 139 |
+
if (!response.ok) {
|
| 140 |
+
throw new Error(await readErrorMessage(response, "PDF processing failed."));
|
| 141 |
+
}
|
| 142 |
+
await readNdjson(response, (event) => handlePdfEvent(event, runId));
|
| 143 |
+
} catch (error) {
|
| 144 |
+
if (error.name !== "AbortError" && isActiveRun(runId)) {
|
| 145 |
+
setError(error.message);
|
| 146 |
+
}
|
| 147 |
+
} finally {
|
| 148 |
+
if (isActiveRun(runId)) {
|
| 149 |
+
submitButton.disabled = false;
|
| 150 |
+
submitButton.textContent = "Upload PDF";
|
| 151 |
+
}
|
| 152 |
+
}
|
| 153 |
+
});
|
| 154 |
+
|
| 155 |
+
queryForm.addEventListener("submit", async (event) => {
|
| 156 |
+
event.preventDefault();
|
| 157 |
+
const query = queryInput.value.trim();
|
| 158 |
+
const documentId = currentDocumentId;
|
| 159 |
+
const queryRunId = activeRunId;
|
| 160 |
+
if (!currentDocumentId) {
|
| 161 |
+
answerBox.textContent = "";
|
| 162 |
+
showUploadMessage("PDF abhi ready nahi hai. Pehle PDF successfully upload/index hone do.", true);
|
| 163 |
+
return;
|
| 164 |
+
}
|
| 165 |
+
if (!query) {
|
| 166 |
+
answerBox.textContent = "Question likho, phir Ask dabao.";
|
| 167 |
+
return;
|
| 168 |
+
}
|
| 169 |
+
|
| 170 |
+
lastQuery = query;
|
| 171 |
+
answerBox.textContent = "Finding answer from your PDF...";
|
| 172 |
+
citations.innerHTML = "";
|
| 173 |
+
retrievedChunks.innerHTML = "";
|
| 174 |
+
renderConfidenceBadge();
|
| 175 |
+
renderHallucinationCard();
|
| 176 |
+
activeQueryId += 1;
|
| 177 |
+
const queryId = activeQueryId;
|
| 178 |
+
if (queryController) {
|
| 179 |
+
queryController.abort();
|
| 180 |
+
}
|
| 181 |
+
queryController = new AbortController();
|
| 182 |
+
|
| 183 |
+
try {
|
| 184 |
+
const response = await fetch("/api/query", {
|
| 185 |
+
method: "POST",
|
| 186 |
+
headers: { "Content-Type": "application/json" },
|
| 187 |
+
body: JSON.stringify({ document_id: documentId, query }),
|
| 188 |
+
signal: queryController.signal,
|
| 189 |
+
});
|
| 190 |
+
const payload = await readJsonResponse(response, "Query failed.");
|
| 191 |
+
if (
|
| 192 |
+
!isActiveRun(queryRunId)
|
| 193 |
+
|| queryId !== activeQueryId
|
| 194 |
+
|| documentId !== currentDocumentId
|
| 195 |
+
|| (payload.document_id && payload.document_id !== currentDocumentId)
|
| 196 |
+
) {
|
| 197 |
+
return;
|
| 198 |
+
}
|
| 199 |
+
if (!response.ok) {
|
| 200 |
+
throw new Error(payload.detail || "Query failed.");
|
| 201 |
+
}
|
| 202 |
+
renderAnswer(payload);
|
| 203 |
+
} catch (error) {
|
| 204 |
+
if (error.name !== "AbortError" && isActiveRun(queryRunId) && queryId === activeQueryId) {
|
| 205 |
+
answerBox.textContent = error.message;
|
| 206 |
+
}
|
| 207 |
+
}
|
| 208 |
+
});
|
| 209 |
+
|
| 210 |
+
feedbackForm.addEventListener("submit", async (event) => {
|
| 211 |
+
event.preventDefault();
|
| 212 |
+
if (!currentDocumentId || !lastQuery) {
|
| 213 |
+
answerBox.textContent = "Feedback ke liye pehle query run karo.";
|
| 214 |
+
return;
|
| 215 |
+
}
|
| 216 |
+
const response = await fetch("/api/feedback", {
|
| 217 |
+
method: "POST",
|
| 218 |
+
headers: { "Content-Type": "application/json" },
|
| 219 |
+
body: JSON.stringify({
|
| 220 |
+
document_id: currentDocumentId,
|
| 221 |
+
query: lastQuery,
|
| 222 |
+
rating: Number(feedbackRating.value),
|
| 223 |
+
comment: feedbackComment.value,
|
| 224 |
+
}),
|
| 225 |
+
});
|
| 226 |
+
const payload = await response.json();
|
| 227 |
+
answerBox.textContent += `\n\nFeedback saved: ${payload.feedback_id}`;
|
| 228 |
+
feedbackComment.value = "";
|
| 229 |
+
});
|
| 230 |
+
|
| 231 |
+
copyButton.addEventListener("click", async () => {
|
| 232 |
+
const text = resultText.textContent;
|
| 233 |
+
if (!text || text.startsWith("Upload a PDF")) {
|
| 234 |
+
return;
|
| 235 |
+
}
|
| 236 |
+
await navigator.clipboard.writeText(text);
|
| 237 |
+
copyButton.textContent = "Copied";
|
| 238 |
+
setTimeout(() => {
|
| 239 |
+
copyButton.textContent = "Copy";
|
| 240 |
+
}, 1200);
|
| 241 |
+
});
|
| 242 |
+
|
| 243 |
+
function startUploadRun(fileNameValue) {
|
| 244 |
+
activeRunId += 1;
|
| 245 |
+
activeQueryId += 1;
|
| 246 |
+
if (uploadController) {
|
| 247 |
+
uploadController.abort();
|
| 248 |
+
}
|
| 249 |
+
if (indexController) {
|
| 250 |
+
indexController.abort();
|
| 251 |
+
}
|
| 252 |
+
if (queryController) {
|
| 253 |
+
queryController.abort();
|
| 254 |
+
}
|
| 255 |
+
uploadController = new AbortController();
|
| 256 |
+
indexController = null;
|
| 257 |
+
queryController = null;
|
| 258 |
+
currentFileName = fileNameValue;
|
| 259 |
+
currentFileSize = input.files[0]?.size || 0;
|
| 260 |
+
currentDocumentId = "";
|
| 261 |
+
return activeRunId;
|
| 262 |
+
}
|
| 263 |
+
|
| 264 |
+
function isActiveRun(runId) {
|
| 265 |
+
return runId === activeRunId;
|
| 266 |
+
}
|
| 267 |
+
|
| 268 |
+
function handleNewFileSelected(file) {
|
| 269 |
+
activeRunId += 1;
|
| 270 |
+
activeQueryId += 1;
|
| 271 |
+
if (uploadController) {
|
| 272 |
+
uploadController.abort();
|
| 273 |
+
}
|
| 274 |
+
if (indexController) {
|
| 275 |
+
indexController.abort();
|
| 276 |
+
}
|
| 277 |
+
if (queryController) {
|
| 278 |
+
queryController.abort();
|
| 279 |
+
}
|
| 280 |
+
uploadController = null;
|
| 281 |
+
indexController = null;
|
| 282 |
+
queryController = null;
|
| 283 |
+
currentDocumentId = "";
|
| 284 |
+
currentFileName = file.name;
|
| 285 |
+
currentFileSize = file.size;
|
| 286 |
+
lastQuery = "";
|
| 287 |
+
queryInput.value = "";
|
| 288 |
+
pageTexts.clear();
|
| 289 |
+
directPages = 0;
|
| 290 |
+
ocrPages = 0;
|
| 291 |
+
pageDetails.innerHTML = "";
|
| 292 |
+
resultText.textContent = "Upload the selected PDF to see extracted text here.";
|
| 293 |
+
routeValue.textContent = "Waiting";
|
| 294 |
+
pageCount.textContent = "0";
|
| 295 |
+
directText.textContent = "-";
|
| 296 |
+
completedTimeline = new Set();
|
| 297 |
+
currentTimelineStep = "";
|
| 298 |
+
activeDocument.textContent = "New PDF selected. Upload to index.";
|
| 299 |
+
dashboardStatus.textContent = "Upload required";
|
| 300 |
+
docStatus.textContent = "Waiting";
|
| 301 |
+
answerBox.textContent = "New PDF selected. Click Upload PDF, then ask questions from this file.";
|
| 302 |
+
citations.innerHTML = "";
|
| 303 |
+
retrievedChunks.innerHTML = "";
|
| 304 |
+
chunkComparison.innerHTML = "";
|
| 305 |
+
bestChunking.textContent = "Evaluation Pending";
|
| 306 |
+
chunkCounts = {};
|
| 307 |
+
chunkScores = {};
|
| 308 |
+
renderIndexTimeline();
|
| 309 |
+
resetHybridStats();
|
| 310 |
+
resetEvaluationCards();
|
| 311 |
+
renderConfidenceBadge();
|
| 312 |
+
resetCragStatus();
|
| 313 |
+
renderHallucinationCard();
|
| 314 |
+
bestRecommendation.textContent = "Index a PDF to compare chunking strategies.";
|
| 315 |
+
showUploadMessage("New PDF selected. Upload it to replace the active document.");
|
| 316 |
+
submitButton.disabled = false;
|
| 317 |
+
submitButton.textContent = "Upload PDF";
|
| 318 |
+
renderSelectedFile(file, "Selected");
|
| 319 |
+
}
|
| 320 |
+
|
| 321 |
+
function setLoading() {
|
| 322 |
+
pageTexts.clear();
|
| 323 |
+
directPages = 0;
|
| 324 |
+
ocrPages = 0;
|
| 325 |
+
currentDocumentId = "";
|
| 326 |
+
lastQuery = "";
|
| 327 |
+
queryInput.value = "";
|
| 328 |
+
feedbackComment.value = "";
|
| 329 |
+
completedTimeline = new Set();
|
| 330 |
+
currentTimelineStep = "Text Cleaning";
|
| 331 |
+
submitButton.disabled = true;
|
| 332 |
+
submitButton.textContent = "Uploading";
|
| 333 |
+
routeValue.textContent = "Processing";
|
| 334 |
+
pageCount.textContent = "-";
|
| 335 |
+
directText.textContent = "-";
|
| 336 |
+
pageDetails.innerHTML = "";
|
| 337 |
+
resultText.textContent = "";
|
| 338 |
+
docStatus.textContent = "Waiting";
|
| 339 |
+
dashboardStatus.textContent = "Waiting for PDF";
|
| 340 |
+
renderIndexTimeline();
|
| 341 |
+
activeDocument.textContent = currentFileName ? `Indexing: ${currentFileName}` : "Not ready";
|
| 342 |
+
answerBox.textContent = "Upload and index the PDF, then ask your question.";
|
| 343 |
+
renderConfidenceBadge();
|
| 344 |
+
showUploadMessage("Reading PDF. Please wait...");
|
| 345 |
+
citations.innerHTML = "";
|
| 346 |
+
retrievedChunks.innerHTML = "";
|
| 347 |
+
chunkComparison.innerHTML = "";
|
| 348 |
+
bestChunking.textContent = "Evaluation Pending";
|
| 349 |
+
resetHybridStats();
|
| 350 |
+
resetEvaluationCards();
|
| 351 |
+
resetCragStatus();
|
| 352 |
+
renderHallucinationCard();
|
| 353 |
+
bestRecommendation.textContent = "Index a PDF to compare chunking strategies.";
|
| 354 |
+
chunkCounts = {};
|
| 355 |
+
chunkScores = {};
|
| 356 |
+
if (input.files[0]) {
|
| 357 |
+
renderSelectedFile(input.files[0], "Processing Upload");
|
| 358 |
+
}
|
| 359 |
+
}
|
| 360 |
+
|
| 361 |
+
function setError(message) {
|
| 362 |
+
const cleanMessage = String(message || "PDF processing failed.");
|
| 363 |
+
routeValue.textContent = "Error";
|
| 364 |
+
pageCount.textContent = "0";
|
| 365 |
+
directText.textContent = "-";
|
| 366 |
+
pageDetails.innerHTML = "";
|
| 367 |
+
resultText.textContent = cleanMessage;
|
| 368 |
+
answerBox.textContent = "";
|
| 369 |
+
renderConfidenceBadge();
|
| 370 |
+
showUploadMessage(cleanUploadError(cleanMessage), true);
|
| 371 |
+
dashboardStatus.textContent = "Error";
|
| 372 |
+
activeDocument.textContent = "Not ready";
|
| 373 |
+
submitButton.disabled = false;
|
| 374 |
+
submitButton.textContent = "Upload PDF";
|
| 375 |
+
}
|
| 376 |
+
|
| 377 |
+
async function readNdjson(response, handler) {
|
| 378 |
+
const reader = response.body.getReader();
|
| 379 |
+
const decoder = new TextDecoder();
|
| 380 |
+
let buffer = "";
|
| 381 |
+
|
| 382 |
+
const dispatchLine = (line) => {
|
| 383 |
+
const trimmed = line.trim();
|
| 384 |
+
if (!trimmed) {
|
| 385 |
+
return;
|
| 386 |
+
}
|
| 387 |
+
|
| 388 |
+
let event;
|
| 389 |
+
try {
|
| 390 |
+
event = JSON.parse(trimmed);
|
| 391 |
+
} catch (error) {
|
| 392 |
+
throw new Error("Processing stream ended unexpectedly. Please upload the PDF again.");
|
| 393 |
+
}
|
| 394 |
+
|
| 395 |
+
if (event.type === "error") {
|
| 396 |
+
throw new Error(event.message || "Processing failed.");
|
| 397 |
+
}
|
| 398 |
+
|
| 399 |
+
handler(event);
|
| 400 |
+
};
|
| 401 |
+
|
| 402 |
+
while (true) {
|
| 403 |
+
const { value, done } = await reader.read();
|
| 404 |
+
if (done) {
|
| 405 |
+
break;
|
| 406 |
+
}
|
| 407 |
+
|
| 408 |
+
buffer += decoder.decode(value, { stream: true });
|
| 409 |
+
const lines = buffer.split("\n");
|
| 410 |
+
buffer = lines.pop();
|
| 411 |
+
|
| 412 |
+
for (const line of lines) {
|
| 413 |
+
dispatchLine(line);
|
| 414 |
+
}
|
| 415 |
+
}
|
| 416 |
+
|
| 417 |
+
buffer += decoder.decode();
|
| 418 |
+
dispatchLine(buffer);
|
| 419 |
+
}
|
| 420 |
+
|
| 421 |
+
async function readJsonResponse(response, fallbackMessage) {
|
| 422 |
+
const text = await response.text();
|
| 423 |
+
if (!text.trim()) {
|
| 424 |
+
throw new Error(`${fallbackMessage} Empty response from server.`);
|
| 425 |
+
}
|
| 426 |
+
try {
|
| 427 |
+
return JSON.parse(text);
|
| 428 |
+
} catch (error) {
|
| 429 |
+
throw new Error(`${fallbackMessage} Server returned an invalid response.`);
|
| 430 |
+
}
|
| 431 |
+
}
|
| 432 |
+
|
| 433 |
+
async function readErrorMessage(response, fallbackMessage) {
|
| 434 |
+
let text = "";
|
| 435 |
+
try {
|
| 436 |
+
text = await response.text();
|
| 437 |
+
} catch (error) {
|
| 438 |
+
return `${fallbackMessage} HTTP ${response.status}.`;
|
| 439 |
+
}
|
| 440 |
+
|
| 441 |
+
if (!text.trim()) {
|
| 442 |
+
return `${fallbackMessage} HTTP ${response.status}: empty response from server.`;
|
| 443 |
+
}
|
| 444 |
+
|
| 445 |
+
try {
|
| 446 |
+
const payload = JSON.parse(text);
|
| 447 |
+
return payload.detail || payload.message || `${fallbackMessage} HTTP ${response.status}.`;
|
| 448 |
+
} catch (error) {
|
| 449 |
+
return text.slice(0, 500);
|
| 450 |
+
}
|
| 451 |
+
}
|
| 452 |
+
|
| 453 |
+
function handlePdfEvent(event, runId) {
|
| 454 |
+
if (!isActiveRun(runId)) {
|
| 455 |
+
return;
|
| 456 |
+
}
|
| 457 |
+
|
| 458 |
+
if (event.type === "start") {
|
| 459 |
+
pageCount.textContent = `0 / ${event.page_count}`;
|
| 460 |
+
updateFileCardMeta(event.page_count, "Processing Upload");
|
| 461 |
+
directText.textContent = "Checking";
|
| 462 |
+
resultText.textContent = "Processing page 1...";
|
| 463 |
+
showUploadMessage(`Reading PDF pages: 0 / ${event.page_count}`);
|
| 464 |
+
return;
|
| 465 |
+
}
|
| 466 |
+
|
| 467 |
+
if (event.type === "page") {
|
| 468 |
+
if (String(event.text || "").trim()) {
|
| 469 |
+
pageTexts.set(event.page_number, event.text || "");
|
| 470 |
+
}
|
| 471 |
+
routeValue.textContent = event.route;
|
| 472 |
+
pageCount.textContent = `${Math.max(pageTexts.size, event.page_number)} / ${event.page_count}`;
|
| 473 |
+
updateFileCardMeta(event.page_count, "Processing Upload");
|
| 474 |
+
|
| 475 |
+
if (event.direct_text_found) {
|
| 476 |
+
directPages += 1;
|
| 477 |
+
} else {
|
| 478 |
+
ocrPages += 1;
|
| 479 |
+
}
|
| 480 |
+
directText.textContent = directStatus();
|
| 481 |
+
|
| 482 |
+
appendPageChip(event);
|
| 483 |
+
renderPageText();
|
| 484 |
+
showUploadMessage(`Reading PDF pages: ${event.page_number} / ${event.page_count}`);
|
| 485 |
+
dashboardStatus.textContent = "Extracting PDF text";
|
| 486 |
+
return;
|
| 487 |
+
}
|
| 488 |
+
|
| 489 |
+
if (event.type === "done") {
|
| 490 |
+
routeValue.textContent = "Completed";
|
| 491 |
+
pageCount.textContent = `${event.page_count} / ${event.page_count}`;
|
| 492 |
+
updateFileCardMeta(event.page_count, "Uploaded Successfully");
|
| 493 |
+
if (!resultText.textContent.trim()) {
|
| 494 |
+
resultText.textContent = "No text was extracted.";
|
| 495 |
+
return;
|
| 496 |
+
}
|
| 497 |
+
indexExtractedText(runId).catch((error) => {
|
| 498 |
+
if (error.name !== "AbortError" && isActiveRun(runId)) {
|
| 499 |
+
docStatus.textContent = "error";
|
| 500 |
+
appendIndexStep("Indexing failed", error.message);
|
| 501 |
+
showUploadMessage(cleanUploadError(error.message), true);
|
| 502 |
+
}
|
| 503 |
+
});
|
| 504 |
+
return;
|
| 505 |
+
}
|
| 506 |
+
|
| 507 |
+
if (event.type === "error") {
|
| 508 |
+
setError(event.message || "PDF processing failed.");
|
| 509 |
+
}
|
| 510 |
+
}
|
| 511 |
+
|
| 512 |
+
async function indexExtractedText(runId) {
|
| 513 |
+
if (!isActiveRun(runId)) {
|
| 514 |
+
return;
|
| 515 |
+
}
|
| 516 |
+
docStatus.textContent = "processing";
|
| 517 |
+
dashboardStatus.textContent = "Indexing";
|
| 518 |
+
currentTimelineStep = "Text Cleaning";
|
| 519 |
+
renderIndexTimeline();
|
| 520 |
+
showUploadMessage("Preparing PDF for questions...");
|
| 521 |
+
const fileNameForRun = currentFileName || "uploaded.pdf";
|
| 522 |
+
const textForRun = resultText.textContent;
|
| 523 |
+
indexController = new AbortController();
|
| 524 |
+
const response = await fetch("/api/index-text-stream", {
|
| 525 |
+
method: "POST",
|
| 526 |
+
headers: { "Content-Type": "application/json" },
|
| 527 |
+
body: JSON.stringify({
|
| 528 |
+
file_name: fileNameForRun,
|
| 529 |
+
text: textForRun,
|
| 530 |
+
}),
|
| 531 |
+
signal: indexController.signal,
|
| 532 |
+
});
|
| 533 |
+
if (!isActiveRun(runId)) {
|
| 534 |
+
return;
|
| 535 |
+
}
|
| 536 |
+
if (!response.ok) {
|
| 537 |
+
throw new Error(await readErrorMessage(response, "Indexing failed."));
|
| 538 |
+
}
|
| 539 |
+
await readNdjson(response, (event) => handleIndexEvent(event, runId, fileNameForRun));
|
| 540 |
+
}
|
| 541 |
+
|
| 542 |
+
function handleIndexEvent(event, runId, fileNameForRun) {
|
| 543 |
+
if (!isActiveRun(runId)) {
|
| 544 |
+
return;
|
| 545 |
+
}
|
| 546 |
+
|
| 547 |
+
if (event.type === "status") {
|
| 548 |
+
docStatus.textContent = event.status;
|
| 549 |
+
appendIndexStep(event.step);
|
| 550 |
+
return;
|
| 551 |
+
}
|
| 552 |
+
|
| 553 |
+
if (event.type === "step") {
|
| 554 |
+
appendIndexStep(event.step);
|
| 555 |
+
captureIndexDetail(event.step, event.detail);
|
| 556 |
+
return;
|
| 557 |
+
}
|
| 558 |
+
|
| 559 |
+
if (event.type === "done") {
|
| 560 |
+
currentDocumentId = event.document_id || "";
|
| 561 |
+
docStatus.textContent = "Indexed";
|
| 562 |
+
dashboardStatus.textContent = "Ready";
|
| 563 |
+
activeDocument.textContent = fileNameForRun;
|
| 564 |
+
appendIndexStep(event.step);
|
| 565 |
+
captureIndexSummary(event.summary);
|
| 566 |
+
answerBox.textContent = "Ask a question from this PDF.";
|
| 567 |
+
showUploadMessage("PDF ready. Ask your question.", false, true);
|
| 568 |
+
queryInput.focus();
|
| 569 |
+
return;
|
| 570 |
+
}
|
| 571 |
+
|
| 572 |
+
if (event.type === "error") {
|
| 573 |
+
docStatus.textContent = "error";
|
| 574 |
+
appendIndexStep("Error", event.message);
|
| 575 |
+
showUploadMessage(cleanUploadError(event.message), true);
|
| 576 |
+
}
|
| 577 |
+
}
|
| 578 |
+
|
| 579 |
+
function appendPageChip(page) {
|
| 580 |
+
const confidence =
|
| 581 |
+
page.confidence === null || page.confidence === undefined
|
| 582 |
+
? ""
|
| 583 |
+
: ` - ${(page.confidence * 100).toFixed(1)}%`;
|
| 584 |
+
const chip = document.createElement("div");
|
| 585 |
+
chip.className = "page-chip";
|
| 586 |
+
chip.dataset.page = String(page.page_number);
|
| 587 |
+
chip.textContent = `Page ${page.page_number}: ${page.engine}${confidence}`;
|
| 588 |
+
pageDetails.appendChild(chip);
|
| 589 |
+
}
|
| 590 |
+
|
| 591 |
+
function appendIndexStep(step) {
|
| 592 |
+
const labelValue = timelineLabel(step);
|
| 593 |
+
completedTimeline.add(timelineStepKey(labelValue));
|
| 594 |
+
const index = timelineSteps.findIndex((item) => timelineStepKey(item) === timelineStepKey(labelValue));
|
| 595 |
+
if (index >= 0) {
|
| 596 |
+
const next = timelineSteps[index + 1];
|
| 597 |
+
currentTimelineStep = next && !completedTimeline.has(timelineStepKey(next)) ? next : "";
|
| 598 |
+
}
|
| 599 |
+
renderIndexTimeline();
|
| 600 |
+
}
|
| 601 |
+
|
| 602 |
+
function renderPageText() {
|
| 603 |
+
const orderedPages = [...pageTexts.keys()].sort((a, b) => a - b);
|
| 604 |
+
resultText.textContent = orderedPages
|
| 605 |
+
.filter((pageNumber) => String(pageTexts.get(pageNumber) || "").trim())
|
| 606 |
+
.map((pageNumber) => `--- Page ${pageNumber} ---\n${pageTexts.get(pageNumber)}`)
|
| 607 |
+
.join("\n\n");
|
| 608 |
+
}
|
| 609 |
+
|
| 610 |
+
function renderAnswer(payload) {
|
| 611 |
+
addQueryHistory(payload.query || lastQuery);
|
| 612 |
+
answerBox.textContent = payload.answer || unrelatedPdfAnswer;
|
| 613 |
+
dashboardStatus.textContent = payload.status === "answered" ? "Answered" : "No answer";
|
| 614 |
+
renderConfidenceBadge(payload.retrieval_grade || payload.status, payload.evaluation || {});
|
| 615 |
+
|
| 616 |
+
citations.innerHTML = "";
|
| 617 |
+
for (const citation of payload.citations || []) {
|
| 618 |
+
const card = document.createElement("div");
|
| 619 |
+
card.className = "citation-card";
|
| 620 |
+
const heading = inferHeading(citation.preview);
|
| 621 |
+
const pageNumber = findPageForText(citation.preview) || citation.page_number || citation.parent_index + 1;
|
| 622 |
+
card.innerHTML = `
|
| 623 |
+
<div class="citation-title">
|
| 624 |
+
<span class="citation-file-badge">PDF</span>
|
| 625 |
+
<strong>${escapeHtml(currentFileName || payload.file_name || "Uploaded PDF")}</strong>
|
| 626 |
+
</div>
|
| 627 |
+
<div class="citation-meta">
|
| 628 |
+
<button type="button" data-page="${escapeHtml(pageNumber)}">Page ${escapeHtml(pageNumber)}</button>
|
| 629 |
+
<span>Heading: ${escapeHtml(heading)}</span>
|
| 630 |
+
<span>Parent Chunk: ${citation.parent_index + 1}</span>
|
| 631 |
+
</div>
|
| 632 |
+
<p>${escapeHtml(citation.preview)}</p>
|
| 633 |
+
`;
|
| 634 |
+
citations.appendChild(card);
|
| 635 |
+
}
|
| 636 |
+
|
| 637 |
+
renderHybridStats(payload.retrieval_stats || {});
|
| 638 |
+
renderEvaluationCards(payload.evaluation || {});
|
| 639 |
+
renderCragStatus(payload.retrieval_grade || payload.status);
|
| 640 |
+
renderHallucinationCard(payload.evaluation || {}, payload.retrieval_grade || payload.status);
|
| 641 |
+
renderRetrievedChunks(payload.hits || []);
|
| 642 |
+
}
|
| 643 |
+
|
| 644 |
+
function directStatus() {
|
| 645 |
+
if (directPages > 0 && ocrPages > 0) {
|
| 646 |
+
return "Mixed";
|
| 647 |
+
}
|
| 648 |
+
if (directPages > 0) {
|
| 649 |
+
return "Yes";
|
| 650 |
+
}
|
| 651 |
+
if (ocrPages > 0) {
|
| 652 |
+
return "No";
|
| 653 |
+
}
|
| 654 |
+
return "Checking";
|
| 655 |
+
}
|
| 656 |
+
|
| 657 |
+
function escapeHtml(value) {
|
| 658 |
+
return String(value || "")
|
| 659 |
+
.replaceAll("&", "&")
|
| 660 |
+
.replaceAll("<", "<")
|
| 661 |
+
.replaceAll(">", ">")
|
| 662 |
+
.replaceAll('"', """)
|
| 663 |
+
.replaceAll("'", "'");
|
| 664 |
+
}
|
| 665 |
+
|
| 666 |
+
function captureIndexDetail(step, detail) {
|
| 667 |
+
if (step.includes("Child Chunks") && detail && typeof detail === "object") {
|
| 668 |
+
chunkCounts = detail;
|
| 669 |
+
renderChunkComparison();
|
| 670 |
+
}
|
| 671 |
+
if (step === "Chunk Size Optimization" && detail && typeof detail === "object") {
|
| 672 |
+
chunkScores = detail.scores || {};
|
| 673 |
+
bestChunking.textContent = bestPerformerLabel(detail.best_chunking);
|
| 674 |
+
bestRecommendation.innerHTML = recommendationHtml(detail.best_chunking, detail.reason);
|
| 675 |
+
renderChunkComparison();
|
| 676 |
+
}
|
| 677 |
+
}
|
| 678 |
+
|
| 679 |
+
function captureIndexSummary(summary) {
|
| 680 |
+
if (!summary) {
|
| 681 |
+
return;
|
| 682 |
+
}
|
| 683 |
+
if (summary.chunk_strategy_counts) {
|
| 684 |
+
chunkCounts = summary.chunk_strategy_counts;
|
| 685 |
+
}
|
| 686 |
+
const recommendation = summary.chunking_recommendation || {};
|
| 687 |
+
chunkScores = recommendation.scores || chunkScores;
|
| 688 |
+
bestChunking.textContent = bestPerformerLabel(recommendation.best_chunking);
|
| 689 |
+
bestRecommendation.innerHTML = recommendationHtml(recommendation.best_chunking, recommendation.reason);
|
| 690 |
+
renderChunkComparison();
|
| 691 |
+
}
|
| 692 |
+
|
| 693 |
+
function renderChunkComparison() {
|
| 694 |
+
const strategies = ["fixed", "recursive", "semantic"];
|
| 695 |
+
chunkComparison.innerHTML = `
|
| 696 |
+
<div class="table-row table-head"><span>Chunking</span><span>Chunks</span><span>Score</span></div>
|
| 697 |
+
${strategies
|
| 698 |
+
.map((strategy) => {
|
| 699 |
+
const winner = bestChunking.textContent.toLowerCase().includes(strategy) ? " winner" : "";
|
| 700 |
+
return `<div class="table-row${winner}">
|
| 701 |
+
<span>${label(strategy)}</span>
|
| 702 |
+
<span>${chunkCounts[strategy] || 0}</span>
|
| 703 |
+
<span>${chunkScores[strategy] ?? "-"}</span>
|
| 704 |
+
</div>`;
|
| 705 |
+
})
|
| 706 |
+
.join("")}
|
| 707 |
+
`;
|
| 708 |
+
}
|
| 709 |
+
|
| 710 |
+
function renderHybridStats(stats) {
|
| 711 |
+
renderPipelineStats(stats);
|
| 712 |
+
hybridStats.innerHTML = `
|
| 713 |
+
<div><span>Vector Results</span><strong>${stats.vector_results ?? "-"}</strong></div>
|
| 714 |
+
<div><span>BM25 Results</span><strong>${stats.bm25_results ?? "-"}</strong></div>
|
| 715 |
+
<div><span>Merged</span><strong>${stats.merged ?? "-"}</strong></div>
|
| 716 |
+
<div><span>Final Reranked</span><strong>${stats.final_reranked ?? "-"}</strong></div>
|
| 717 |
+
`;
|
| 718 |
+
}
|
| 719 |
+
|
| 720 |
+
function resetHybridStats() {
|
| 721 |
+
renderHybridStats({});
|
| 722 |
+
}
|
| 723 |
+
|
| 724 |
+
function renderEvaluationCards(metrics) {
|
| 725 |
+
evaluationCards.innerHTML = `
|
| 726 |
+
<div><span>Context Precision</span><strong>${metricScore(metrics.context_precision)}</strong></div>
|
| 727 |
+
<div><span>Context Recall</span><strong>${metricScore(metrics.context_recall)}</strong></div>
|
| 728 |
+
<div><span>Faithfulness</span><strong>${metricScore(metrics.faithfulness)}</strong></div>
|
| 729 |
+
<div><span>Answer Relevancy</span><strong>${metricScore(metrics.answer_relevancy)}</strong></div>
|
| 730 |
+
`;
|
| 731 |
+
}
|
| 732 |
+
|
| 733 |
+
function resetEvaluationCards() {
|
| 734 |
+
renderEvaluationCards({});
|
| 735 |
+
}
|
| 736 |
+
|
| 737 |
+
function renderRetrievedChunks(hits) {
|
| 738 |
+
if (!hits.length) {
|
| 739 |
+
retrievedChunks.innerHTML = '<div class="empty-state">No retrieved chunks yet.</div>';
|
| 740 |
+
return;
|
| 741 |
+
}
|
| 742 |
+
retrievedChunks.innerHTML = hits
|
| 743 |
+
.map(
|
| 744 |
+
(hit, index) => `
|
| 745 |
+
<details class="chunk-card">
|
| 746 |
+
<summary>
|
| 747 |
+
<span>#${index + 1}</span>
|
| 748 |
+
<strong>${label(hit.strategy)} Chunk</strong>
|
| 749 |
+
<div class="chunk-summary-scores">
|
| 750 |
+
<em>Similarity: ${score(hit.vector_score)}</em>
|
| 751 |
+
<em>BM25: ${bm25Score(hit.bm25_raw_score, hit.bm25_score)}</em>
|
| 752 |
+
<em>Reranker: ${score(hit.rerank_score)}</em>
|
| 753 |
+
</div>
|
| 754 |
+
</summary>
|
| 755 |
+
<p><strong>Chunk Text</strong>${escapeHtml(hit.child_text)}</p>
|
| 756 |
+
<div class="chunk-scores">
|
| 757 |
+
<span>Similarity Score: <strong>${score(hit.vector_score)}</strong></span>
|
| 758 |
+
<span>BM25 Score: <strong>${bm25Score(hit.bm25_raw_score, hit.bm25_score)}</strong></span>
|
| 759 |
+
<span>Reranker Score: <strong>${score(hit.rerank_score)}</strong></span>
|
| 760 |
+
</div>
|
| 761 |
+
</details>
|
| 762 |
+
`
|
| 763 |
+
)
|
| 764 |
+
.join("");
|
| 765 |
+
}
|
| 766 |
+
|
| 767 |
+
function score(value) {
|
| 768 |
+
return value === undefined || value === null ? "-" : Number(value).toFixed(2);
|
| 769 |
+
}
|
| 770 |
+
|
| 771 |
+
function metricScore(value) {
|
| 772 |
+
return value === undefined || value === null ? "Not Evaluated Yet" : Number(value).toFixed(2);
|
| 773 |
+
}
|
| 774 |
+
|
| 775 |
+
function bm25Score(raw, normalized) {
|
| 776 |
+
if (raw !== undefined && raw !== null) {
|
| 777 |
+
return Number(raw).toFixed(2);
|
| 778 |
+
}
|
| 779 |
+
return score(normalized);
|
| 780 |
+
}
|
| 781 |
+
|
| 782 |
+
function label(value) {
|
| 783 |
+
return String(value || "")
|
| 784 |
+
.replace(/^./, (char) => char.toUpperCase());
|
| 785 |
+
}
|
| 786 |
+
|
| 787 |
+
function inferHeading(preview) {
|
| 788 |
+
const text = String(preview || "").trim();
|
| 789 |
+
if (!text) {
|
| 790 |
+
return "Document context";
|
| 791 |
+
}
|
| 792 |
+
const firstLine = text.split(/\n|\. /)[0].trim();
|
| 793 |
+
const words = firstLine.split(/\s+/).slice(0, 5).join(" ");
|
| 794 |
+
return words || "Document context";
|
| 795 |
+
}
|
| 796 |
+
|
| 797 |
+
function timelineLabel(step) {
|
| 798 |
+
const labels = {
|
| 799 |
+
"Document Status: processing": "Processing Started",
|
| 800 |
+
"Text Cleaning": "Text Cleaning",
|
| 801 |
+
"Metadata": "Metadata Extraction",
|
| 802 |
+
"Parent Chunks": "Parent Chunks",
|
| 803 |
+
"Fixed / Recursive / Semantic Child Chunks": "Child Chunks",
|
| 804 |
+
"Chunk Size Optimization": "Chunk Size Optimization",
|
| 805 |
+
"Embeddings": "Embeddings",
|
| 806 |
+
"Embedding Evaluation": "Embedding Evaluation",
|
| 807 |
+
"Qdrant Store": "Qdrant Index",
|
| 808 |
+
"BM25 Index": "BM25 Index",
|
| 809 |
+
"Document Status: indexed": "Ready",
|
| 810 |
+
};
|
| 811 |
+
return labels[step] || step;
|
| 812 |
+
}
|
| 813 |
+
|
| 814 |
+
function renderCragStatus(grade) {
|
| 815 |
+
const good = grade === "good" || grade === "answered";
|
| 816 |
+
cragStatus.className = `crag-box ${good ? "good" : "bad"}`;
|
| 817 |
+
cragStatus.innerHTML = good
|
| 818 |
+
? "<strong>Good</strong><span>Retrieved context passed CRAG grading.</span>"
|
| 819 |
+
: "<strong>Low Confidence</strong><span>Query rewrite or no-answer fallback was used.</span>";
|
| 820 |
+
}
|
| 821 |
+
|
| 822 |
+
function resetCragStatus() {
|
| 823 |
+
cragStatus.className = "crag-box neutral";
|
| 824 |
+
cragStatus.innerHTML = "<strong>Waiting</strong><span>Run a query to grade retrieval quality.</span>";
|
| 825 |
+
}
|
| 826 |
+
|
| 827 |
+
function renderSelectedFile(file, status) {
|
| 828 |
+
currentFileName = file.name;
|
| 829 |
+
currentFileSize = file.size;
|
| 830 |
+
uploadedFileName.textContent = file.name;
|
| 831 |
+
uploadedFileMeta.textContent = `0 Pages / ${formatBytes(file.size)}`;
|
| 832 |
+
fileCard.querySelector("em").textContent = status;
|
| 833 |
+
fileCard.classList.remove("is-hidden");
|
| 834 |
+
if (pdfPreviewUrl) {
|
| 835 |
+
URL.revokeObjectURL(pdfPreviewUrl);
|
| 836 |
+
}
|
| 837 |
+
pdfPreviewUrl = URL.createObjectURL(file);
|
| 838 |
+
pdfPreview.src = `${pdfPreviewUrl}#page=1&toolbar=0&navpanes=0&scrollbar=0`;
|
| 839 |
+
pdfPreviewCard.classList.remove("is-hidden");
|
| 840 |
+
}
|
| 841 |
+
|
| 842 |
+
function updateFileCardMeta(totalPages, status) {
|
| 843 |
+
if (!currentFileName) {
|
| 844 |
+
return;
|
| 845 |
+
}
|
| 846 |
+
uploadedFileName.textContent = currentFileName;
|
| 847 |
+
uploadedFileMeta.textContent = `${totalPages || pageTexts.size || 0} Pages / ${formatBytes(currentFileSize)}`;
|
| 848 |
+
fileCard.querySelector("em").textContent = status;
|
| 849 |
+
fileCard.classList.remove("is-hidden");
|
| 850 |
+
}
|
| 851 |
+
|
| 852 |
+
function formatBytes(bytes) {
|
| 853 |
+
if (!bytes) {
|
| 854 |
+
return "0 MB";
|
| 855 |
+
}
|
| 856 |
+
const mb = bytes / (1024 * 1024);
|
| 857 |
+
if (mb >= 1) {
|
| 858 |
+
return `${mb.toFixed(1)} MB`;
|
| 859 |
+
}
|
| 860 |
+
return `${Math.max(bytes / 1024, 1).toFixed(0)} KB`;
|
| 861 |
+
}
|
| 862 |
+
|
| 863 |
+
function renderIndexTimeline() {
|
| 864 |
+
indexSteps.innerHTML = timelineSteps
|
| 865 |
+
.map((step) => {
|
| 866 |
+
const key = timelineStepKey(step);
|
| 867 |
+
const done = completedTimeline.has(key);
|
| 868 |
+
const current = !done && currentTimelineStep && timelineStepKey(currentTimelineStep) === key;
|
| 869 |
+
const state = done ? "done" : current ? "current" : "pending";
|
| 870 |
+
const marker = done ? "OK" : current ? "..." : "";
|
| 871 |
+
return `<div class="timeline-item ${state}">
|
| 872 |
+
<span>${marker}</span>
|
| 873 |
+
<strong>${escapeHtml(step)}</strong>
|
| 874 |
+
</div>`;
|
| 875 |
+
})
|
| 876 |
+
.join("");
|
| 877 |
+
}
|
| 878 |
+
|
| 879 |
+
function timelineStepKey(value) {
|
| 880 |
+
return String(value || "").toLowerCase().replace(/[^a-z0-9]+/g, "");
|
| 881 |
+
}
|
| 882 |
+
|
| 883 |
+
function bestPerformerLabel(strategy) {
|
| 884 |
+
return strategy ? `Best Performer: ${label(strategy)} Chunking` : "Evaluation Pending";
|
| 885 |
+
}
|
| 886 |
+
|
| 887 |
+
function recommendationHtml(strategy, reason) {
|
| 888 |
+
const strategyLabel = label(strategy || "semantic");
|
| 889 |
+
return `
|
| 890 |
+
<strong>Recommended Strategy</strong>
|
| 891 |
+
<span>${strategyLabel} Chunking</span>
|
| 892 |
+
<p>Reason:</p>
|
| 893 |
+
<ul>
|
| 894 |
+
<li>${escapeHtml(reason || "Highest retrieval quality.")}</li>
|
| 895 |
+
<li>Best faithfulness</li>
|
| 896 |
+
<li>Best citation accuracy</li>
|
| 897 |
+
<li>Lowest hallucination risk</li>
|
| 898 |
+
</ul>
|
| 899 |
+
`;
|
| 900 |
+
}
|
| 901 |
+
|
| 902 |
+
function renderPipelineStats(stats = {}) {
|
| 903 |
+
const hasStats = Object.keys(stats).length > 0;
|
| 904 |
+
const vector = hasStats ? stats.vector_results ?? 0 : "Not Run";
|
| 905 |
+
const bm25 = hasStats ? stats.bm25_results ?? 0 : "Not Run";
|
| 906 |
+
const merged = hasStats ? stats.merged ?? 0 : "Not Run";
|
| 907 |
+
const final = hasStats ? stats.final_reranked ?? 0 : "Not Run";
|
| 908 |
+
pipelineViz.innerHTML = `
|
| 909 |
+
<div>Query</div>
|
| 910 |
+
<span></span>
|
| 911 |
+
<div>Vector Search (${vector})</div>
|
| 912 |
+
<span></span>
|
| 913 |
+
<div>BM25 (${bm25})</div>
|
| 914 |
+
<span></span>
|
| 915 |
+
<div>Merged (${merged})</div>
|
| 916 |
+
<span></span>
|
| 917 |
+
<div>Reranked (${final})</div>
|
| 918 |
+
<span></span>
|
| 919 |
+
<div>Answer</div>
|
| 920 |
+
`;
|
| 921 |
+
}
|
| 922 |
+
|
| 923 |
+
function renderConfidenceBadge(grade, metrics = {}) {
|
| 924 |
+
if (!grade) {
|
| 925 |
+
confidenceBadge.className = "confidence-badge neutral";
|
| 926 |
+
confidenceBadge.textContent = "Confidence pending";
|
| 927 |
+
return;
|
| 928 |
+
}
|
| 929 |
+
const faithfulness = Number(metrics.faithfulness || 0);
|
| 930 |
+
const gradeGood = grade === "good" || grade === "answered";
|
| 931 |
+
const level = gradeGood && faithfulness >= 0.9 ? "high" : gradeGood && faithfulness >= 0.75 ? "medium" : "low";
|
| 932 |
+
const labelMap = { high: "High Confidence", medium: "Medium Confidence", low: "Low Confidence" };
|
| 933 |
+
confidenceBadge.className = `confidence-badge ${level}`;
|
| 934 |
+
confidenceBadge.textContent = labelMap[level];
|
| 935 |
+
}
|
| 936 |
+
|
| 937 |
+
function renderHallucinationCard(metrics = {}, grade) {
|
| 938 |
+
const faithfulness = metrics.faithfulness;
|
| 939 |
+
const precision = metrics.context_precision;
|
| 940 |
+
const coverage = faithfulness === undefined || precision === undefined ? null : Math.round(((faithfulness + precision) / 2) * 100);
|
| 941 |
+
const good = (grade === "good" || grade === "answered") && Number(faithfulness || 0) >= 0.75;
|
| 942 |
+
hallucinationCard.className = `hallucination-card ${coverage === null ? "neutral" : good ? "good" : "bad"}`;
|
| 943 |
+
hallucinationCard.innerHTML = `
|
| 944 |
+
<span>Answer Source Coverage</span>
|
| 945 |
+
<strong>${coverage === null ? "-" : `${coverage}%`}</strong>
|
| 946 |
+
<span>Faithfulness</span>
|
| 947 |
+
<strong>${score(faithfulness)}</strong>
|
| 948 |
+
<em>${coverage === null ? "Waiting for answer" : good ? "No Hallucination Detected" : "Needs review"}</em>
|
| 949 |
+
`;
|
| 950 |
+
}
|
| 951 |
+
|
| 952 |
+
function addQueryHistory(query) {
|
| 953 |
+
const clean = String(query || "").trim();
|
| 954 |
+
if (!clean) {
|
| 955 |
+
return;
|
| 956 |
+
}
|
| 957 |
+
const existingIndex = queryHistoryItems.findIndex((item) => item.toLowerCase() === clean.toLowerCase());
|
| 958 |
+
if (existingIndex >= 0) {
|
| 959 |
+
queryHistoryItems.splice(existingIndex, 1);
|
| 960 |
+
}
|
| 961 |
+
queryHistoryItems.unshift(clean);
|
| 962 |
+
queryHistoryItems.splice(5);
|
| 963 |
+
renderQueryHistory();
|
| 964 |
+
}
|
| 965 |
+
|
| 966 |
+
function renderQueryHistory() {
|
| 967 |
+
historyCount.textContent = `${queryHistoryItems.length} saved`;
|
| 968 |
+
if (!queryHistoryItems.length) {
|
| 969 |
+
queryHistory.innerHTML = "<span>No questions yet.</span>";
|
| 970 |
+
return;
|
| 971 |
+
}
|
| 972 |
+
queryHistory.innerHTML = queryHistoryItems
|
| 973 |
+
.map((query) => `<button type="button">${escapeHtml(query)}</button>`)
|
| 974 |
+
.join("");
|
| 975 |
+
}
|
| 976 |
+
|
| 977 |
+
function findPageForText(preview) {
|
| 978 |
+
const needle = String(preview || "").replace(/\s+/g, " ").trim().slice(0, 90);
|
| 979 |
+
if (!needle) {
|
| 980 |
+
return "";
|
| 981 |
+
}
|
| 982 |
+
for (const [page, text] of pageTexts.entries()) {
|
| 983 |
+
const haystack = String(text || "").replace(/\s+/g, " ");
|
| 984 |
+
if (haystack.includes(needle)) {
|
| 985 |
+
return page;
|
| 986 |
+
}
|
| 987 |
+
}
|
| 988 |
+
return "";
|
| 989 |
+
}
|
| 990 |
+
|
| 991 |
+
function scrollToExtractedPage(pageNumber) {
|
| 992 |
+
const details = document.querySelector(".details-panel details");
|
| 993 |
+
details.open = true;
|
| 994 |
+
document.querySelector(".details-panel").scrollIntoView({ behavior: "smooth", block: "start" });
|
| 995 |
+
pageDetails.querySelectorAll(".page-chip").forEach((chip) => {
|
| 996 |
+
chip.classList.toggle("is-active", chip.dataset.page === String(pageNumber));
|
| 997 |
+
});
|
| 998 |
+
}
|
| 999 |
+
|
| 1000 |
+
function showUploadMessage(message, isError = false, isSuccess = false) {
|
| 1001 |
+
uploadMessage.textContent = message;
|
| 1002 |
+
uploadMessage.className = `upload-message${isError ? " error" : ""}${isSuccess ? " success" : ""}`;
|
| 1003 |
+
}
|
| 1004 |
+
|
| 1005 |
+
function cleanUploadError(message) {
|
| 1006 |
+
const text = String(message || "");
|
| 1007 |
+
const keyMatch = text.match(/Tried\s+(\d+)\s+active API key\(s\) out of\s+(\d+)\s+configured key\(s\);\s+(\d+)\s+key\(s\)/i);
|
| 1008 |
+
if (keyMatch) {
|
| 1009 |
+
const [, triedKeys, totalKeys, quotaKeys] = keyMatch;
|
| 1010 |
+
return `Gemini OCR ne ${triedKeys}/${totalKeys} API keys try ki. ${quotaKeys} key(s) quota/rate-limit de rahi hain. Agar yahi message aa raha hai to ya to sab keys exhausted hain, ya server/browser abhi old code use kar raha hai. Server restart aur browser hard refresh ke baad dobara try karo.`;
|
| 1011 |
+
}
|
| 1012 |
+
if (text.toLowerCase().includes("api key not valid") || text.toLowerCase().includes("invalid api key")) {
|
| 1013 |
+
return "Gemini API key valid nahi lag rahi. env file me GEMINI_API_KEY_1 se GEMINI_API_KEY_4 tak valid Google Gemini keys daalo, phir server restart karo.";
|
| 1014 |
+
}
|
| 1015 |
+
if (text.includes("RESOURCE_EXHAUSTED") || text.toLowerCase().includes("quota")) {
|
| 1016 |
+
return "Gemini OCR quota/rate-limit error de raha hai. App next keys try karta hai; agar phir bhi ye aa raha hai to configured keys exhausted/invalid ho sakti hain. Server restart aur hard refresh ke baad dobara try karo.";
|
| 1017 |
+
}
|
| 1018 |
+
if (text.includes("temporarily unavailable") || text.includes("503")) {
|
| 1019 |
+
return "Gemini OCR abhi busy hai. Thodi der baad try karo ya text-based PDF upload karo.";
|
| 1020 |
+
}
|
| 1021 |
+
if (text.toLowerCase().includes("password")) {
|
| 1022 |
+
return "PDF password-protected hai. Unlocked PDF upload karo.";
|
| 1023 |
+
}
|
| 1024 |
+
return text;
|
| 1025 |
+
}
|
| 1026 |
+
|
| 1027 |
+
renderIndexTimeline();
|
| 1028 |
+
resetHybridStats();
|
| 1029 |
+
resetEvaluationCards();
|
| 1030 |
+
resetCragStatus();
|
| 1031 |
+
renderHallucinationCard();
|
| 1032 |
+
renderRetrievedChunks([]);
|
| 1033 |
+
renderQueryHistory();
|
static/styles.css
ADDED
|
@@ -0,0 +1,998 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
:root {
|
| 2 |
+
color-scheme: light;
|
| 3 |
+
--bg: #eef2f4;
|
| 4 |
+
--surface: #ffffff;
|
| 5 |
+
--surface-strong: #f8fafb;
|
| 6 |
+
--text: #172026;
|
| 7 |
+
--muted: #65717a;
|
| 8 |
+
--line: #d7dee3;
|
| 9 |
+
--accent: #1f7a6d;
|
| 10 |
+
--accent-strong: #145a51;
|
| 11 |
+
--info: #2f5f9f;
|
| 12 |
+
--warning: #b7791f;
|
| 13 |
+
--success: #2f7d50;
|
| 14 |
+
--danger: #a33b3b;
|
| 15 |
+
}
|
| 16 |
+
|
| 17 |
+
* {
|
| 18 |
+
box-sizing: border-box;
|
| 19 |
+
}
|
| 20 |
+
|
| 21 |
+
body {
|
| 22 |
+
margin: 0;
|
| 23 |
+
min-height: 100vh;
|
| 24 |
+
font-family: Inter, Segoe UI, system-ui, -apple-system, sans-serif;
|
| 25 |
+
background: var(--bg);
|
| 26 |
+
color: var(--text);
|
| 27 |
+
}
|
| 28 |
+
|
| 29 |
+
button,
|
| 30 |
+
input {
|
| 31 |
+
font: inherit;
|
| 32 |
+
}
|
| 33 |
+
|
| 34 |
+
.app-shell {
|
| 35 |
+
width: min(1180px, calc(100% - 32px));
|
| 36 |
+
margin: 0 auto;
|
| 37 |
+
padding: 32px 0;
|
| 38 |
+
}
|
| 39 |
+
|
| 40 |
+
.workspace {
|
| 41 |
+
display: grid;
|
| 42 |
+
grid-template-columns: 390px minmax(0, 1fr);
|
| 43 |
+
gap: 18px;
|
| 44 |
+
align-items: start;
|
| 45 |
+
}
|
| 46 |
+
|
| 47 |
+
.panel {
|
| 48 |
+
background: var(--surface);
|
| 49 |
+
border: 1px solid var(--line);
|
| 50 |
+
border-radius: 8px;
|
| 51 |
+
box-shadow: 0 12px 32px rgba(24, 38, 48, 0.08);
|
| 52 |
+
}
|
| 53 |
+
|
| 54 |
+
.upload-panel {
|
| 55 |
+
padding: 20px;
|
| 56 |
+
}
|
| 57 |
+
|
| 58 |
+
.brand-row {
|
| 59 |
+
display: flex;
|
| 60 |
+
gap: 12px;
|
| 61 |
+
align-items: center;
|
| 62 |
+
margin-bottom: 20px;
|
| 63 |
+
}
|
| 64 |
+
|
| 65 |
+
.brand-mark {
|
| 66 |
+
display: grid;
|
| 67 |
+
place-items: center;
|
| 68 |
+
width: 44px;
|
| 69 |
+
height: 44px;
|
| 70 |
+
border-radius: 8px;
|
| 71 |
+
background: var(--accent);
|
| 72 |
+
color: white;
|
| 73 |
+
font-weight: 800;
|
| 74 |
+
}
|
| 75 |
+
|
| 76 |
+
h1,
|
| 77 |
+
h2,
|
| 78 |
+
p {
|
| 79 |
+
margin: 0;
|
| 80 |
+
}
|
| 81 |
+
|
| 82 |
+
h1 {
|
| 83 |
+
font-size: 24px;
|
| 84 |
+
line-height: 1.15;
|
| 85 |
+
}
|
| 86 |
+
|
| 87 |
+
h2 {
|
| 88 |
+
font-size: 18px;
|
| 89 |
+
}
|
| 90 |
+
|
| 91 |
+
p {
|
| 92 |
+
margin-top: 4px;
|
| 93 |
+
color: var(--muted);
|
| 94 |
+
font-size: 14px;
|
| 95 |
+
}
|
| 96 |
+
|
| 97 |
+
.brand-subline {
|
| 98 |
+
display: block;
|
| 99 |
+
margin-top: 8px;
|
| 100 |
+
color: #33515b;
|
| 101 |
+
font-size: 13px;
|
| 102 |
+
font-weight: 700;
|
| 103 |
+
}
|
| 104 |
+
|
| 105 |
+
.drop-zone {
|
| 106 |
+
display: grid;
|
| 107 |
+
gap: 14px;
|
| 108 |
+
padding: 16px;
|
| 109 |
+
border: 1px dashed #9caab3;
|
| 110 |
+
border-radius: 8px;
|
| 111 |
+
background: var(--surface-strong);
|
| 112 |
+
}
|
| 113 |
+
|
| 114 |
+
.drop-zone.is-dragging {
|
| 115 |
+
border-color: var(--accent);
|
| 116 |
+
background: #edf8f5;
|
| 117 |
+
}
|
| 118 |
+
|
| 119 |
+
.drop-zone input {
|
| 120 |
+
width: 100%;
|
| 121 |
+
}
|
| 122 |
+
|
| 123 |
+
.drop-content {
|
| 124 |
+
display: grid;
|
| 125 |
+
gap: 4px;
|
| 126 |
+
}
|
| 127 |
+
|
| 128 |
+
.drop-content span {
|
| 129 |
+
color: var(--muted);
|
| 130 |
+
font-size: 13px;
|
| 131 |
+
}
|
| 132 |
+
|
| 133 |
+
.file-card,
|
| 134 |
+
.pdf-preview-card {
|
| 135 |
+
margin-top: 14px;
|
| 136 |
+
}
|
| 137 |
+
|
| 138 |
+
.file-card {
|
| 139 |
+
display: grid;
|
| 140 |
+
grid-template-columns: 48px minmax(0, 1fr);
|
| 141 |
+
gap: 12px;
|
| 142 |
+
align-items: center;
|
| 143 |
+
padding: 12px;
|
| 144 |
+
border: 1px solid #b9d5cc;
|
| 145 |
+
border-radius: 8px;
|
| 146 |
+
background: #f2fbf8;
|
| 147 |
+
}
|
| 148 |
+
|
| 149 |
+
.file-icon {
|
| 150 |
+
display: grid;
|
| 151 |
+
place-items: center;
|
| 152 |
+
width: 48px;
|
| 153 |
+
height: 56px;
|
| 154 |
+
border-radius: 6px;
|
| 155 |
+
background: #d84b4b;
|
| 156 |
+
color: white;
|
| 157 |
+
font-size: 12px;
|
| 158 |
+
font-weight: 900;
|
| 159 |
+
}
|
| 160 |
+
|
| 161 |
+
.file-card strong,
|
| 162 |
+
.file-card span,
|
| 163 |
+
.file-card em {
|
| 164 |
+
display: block;
|
| 165 |
+
overflow-wrap: anywhere;
|
| 166 |
+
}
|
| 167 |
+
|
| 168 |
+
.file-card span {
|
| 169 |
+
margin-top: 3px;
|
| 170 |
+
color: var(--muted);
|
| 171 |
+
font-size: 13px;
|
| 172 |
+
}
|
| 173 |
+
|
| 174 |
+
.file-card em {
|
| 175 |
+
margin-top: 4px;
|
| 176 |
+
color: var(--success);
|
| 177 |
+
font-size: 13px;
|
| 178 |
+
font-style: normal;
|
| 179 |
+
font-weight: 800;
|
| 180 |
+
}
|
| 181 |
+
|
| 182 |
+
.pdf-preview-card {
|
| 183 |
+
padding: 12px;
|
| 184 |
+
border: 1px solid var(--line);
|
| 185 |
+
border-radius: 8px;
|
| 186 |
+
background: #fbfcfc;
|
| 187 |
+
}
|
| 188 |
+
|
| 189 |
+
.pdf-preview-card iframe {
|
| 190 |
+
width: 100%;
|
| 191 |
+
height: 210px;
|
| 192 |
+
margin-top: 10px;
|
| 193 |
+
border: 1px solid var(--line);
|
| 194 |
+
border-radius: 6px;
|
| 195 |
+
background: white;
|
| 196 |
+
}
|
| 197 |
+
|
| 198 |
+
.is-hidden {
|
| 199 |
+
display: none;
|
| 200 |
+
}
|
| 201 |
+
|
| 202 |
+
button {
|
| 203 |
+
min-height: 40px;
|
| 204 |
+
border: 0;
|
| 205 |
+
border-radius: 6px;
|
| 206 |
+
background: var(--accent);
|
| 207 |
+
color: white;
|
| 208 |
+
font-weight: 700;
|
| 209 |
+
cursor: pointer;
|
| 210 |
+
}
|
| 211 |
+
|
| 212 |
+
button:hover {
|
| 213 |
+
background: var(--accent-strong);
|
| 214 |
+
}
|
| 215 |
+
|
| 216 |
+
.status-grid {
|
| 217 |
+
display: grid;
|
| 218 |
+
grid-template-columns: 1fr;
|
| 219 |
+
gap: 10px;
|
| 220 |
+
margin-top: 18px;
|
| 221 |
+
}
|
| 222 |
+
|
| 223 |
+
.upload-message {
|
| 224 |
+
margin-top: 14px;
|
| 225 |
+
padding: 12px;
|
| 226 |
+
border: 1px solid var(--line);
|
| 227 |
+
border-radius: 8px;
|
| 228 |
+
background: #fbfcfc;
|
| 229 |
+
color: var(--muted);
|
| 230 |
+
line-height: 1.45;
|
| 231 |
+
}
|
| 232 |
+
|
| 233 |
+
.upload-message.error {
|
| 234 |
+
border-color: #dfaaaa;
|
| 235 |
+
background: #fff3f3;
|
| 236 |
+
color: #8f2f2f;
|
| 237 |
+
}
|
| 238 |
+
|
| 239 |
+
.upload-message.success {
|
| 240 |
+
border-color: #8cc9b6;
|
| 241 |
+
background: #edf8f5;
|
| 242 |
+
color: #244a45;
|
| 243 |
+
}
|
| 244 |
+
|
| 245 |
+
.status-grid div {
|
| 246 |
+
display: grid;
|
| 247 |
+
gap: 4px;
|
| 248 |
+
padding: 12px;
|
| 249 |
+
border: 1px solid var(--line);
|
| 250 |
+
border-radius: 8px;
|
| 251 |
+
background: #fbfcfc;
|
| 252 |
+
}
|
| 253 |
+
|
| 254 |
+
.status-grid span {
|
| 255 |
+
color: var(--muted);
|
| 256 |
+
font-size: 12px;
|
| 257 |
+
text-transform: uppercase;
|
| 258 |
+
letter-spacing: 0;
|
| 259 |
+
}
|
| 260 |
+
|
| 261 |
+
.status-grid strong {
|
| 262 |
+
overflow-wrap: anywhere;
|
| 263 |
+
}
|
| 264 |
+
|
| 265 |
+
.page-details {
|
| 266 |
+
display: flex;
|
| 267 |
+
flex-wrap: wrap;
|
| 268 |
+
gap: 8px;
|
| 269 |
+
margin-top: 16px;
|
| 270 |
+
}
|
| 271 |
+
|
| 272 |
+
.page-chip {
|
| 273 |
+
max-width: 100%;
|
| 274 |
+
padding: 7px 9px;
|
| 275 |
+
border-radius: 6px;
|
| 276 |
+
background: #e7efed;
|
| 277 |
+
color: #244a45;
|
| 278 |
+
font-size: 13px;
|
| 279 |
+
overflow-wrap: anywhere;
|
| 280 |
+
}
|
| 281 |
+
|
| 282 |
+
.index-section {
|
| 283 |
+
margin-top: 18px;
|
| 284 |
+
padding-top: 16px;
|
| 285 |
+
border-top: 1px solid var(--line);
|
| 286 |
+
}
|
| 287 |
+
|
| 288 |
+
.section-head {
|
| 289 |
+
display: flex;
|
| 290 |
+
align-items: center;
|
| 291 |
+
justify-content: space-between;
|
| 292 |
+
gap: 10px;
|
| 293 |
+
}
|
| 294 |
+
|
| 295 |
+
.section-head span,
|
| 296 |
+
.result-toolbar span {
|
| 297 |
+
color: var(--muted);
|
| 298 |
+
font-size: 13px;
|
| 299 |
+
overflow-wrap: anywhere;
|
| 300 |
+
}
|
| 301 |
+
|
| 302 |
+
.step-list {
|
| 303 |
+
display: grid;
|
| 304 |
+
gap: 8px;
|
| 305 |
+
margin-top: 12px;
|
| 306 |
+
}
|
| 307 |
+
|
| 308 |
+
.step-item {
|
| 309 |
+
padding: 9px 10px;
|
| 310 |
+
border: 1px solid var(--line);
|
| 311 |
+
border-radius: 6px;
|
| 312 |
+
background: #fbfcfc;
|
| 313 |
+
font-size: 13px;
|
| 314 |
+
}
|
| 315 |
+
|
| 316 |
+
.result-panel {
|
| 317 |
+
display: grid;
|
| 318 |
+
grid-template-rows: auto 1fr;
|
| 319 |
+
overflow: hidden;
|
| 320 |
+
}
|
| 321 |
+
|
| 322 |
+
.result-toolbar {
|
| 323 |
+
display: flex;
|
| 324 |
+
align-items: center;
|
| 325 |
+
justify-content: space-between;
|
| 326 |
+
gap: 12px;
|
| 327 |
+
padding: 16px 18px;
|
| 328 |
+
border-bottom: 1px solid var(--line);
|
| 329 |
+
}
|
| 330 |
+
|
| 331 |
+
.result-toolbar button {
|
| 332 |
+
min-width: 82px;
|
| 333 |
+
min-height: 34px;
|
| 334 |
+
}
|
| 335 |
+
|
| 336 |
+
pre {
|
| 337 |
+
margin: 0;
|
| 338 |
+
padding: 18px;
|
| 339 |
+
overflow: auto;
|
| 340 |
+
white-space: pre-wrap;
|
| 341 |
+
word-break: break-word;
|
| 342 |
+
font: 14px/1.55 Consolas, Monaco, monospace;
|
| 343 |
+
color: #243038;
|
| 344 |
+
}
|
| 345 |
+
|
| 346 |
+
.query-panel {
|
| 347 |
+
display: grid;
|
| 348 |
+
gap: 14px;
|
| 349 |
+
align-content: start;
|
| 350 |
+
min-height: 430px;
|
| 351 |
+
padding-bottom: 18px;
|
| 352 |
+
}
|
| 353 |
+
|
| 354 |
+
.query-suggestions {
|
| 355 |
+
display: flex;
|
| 356 |
+
flex-wrap: wrap;
|
| 357 |
+
gap: 8px;
|
| 358 |
+
padding: 0 18px;
|
| 359 |
+
}
|
| 360 |
+
|
| 361 |
+
.query-suggestions strong {
|
| 362 |
+
flex-basis: 100%;
|
| 363 |
+
color: var(--muted);
|
| 364 |
+
font-size: 12px;
|
| 365 |
+
text-transform: uppercase;
|
| 366 |
+
}
|
| 367 |
+
|
| 368 |
+
.query-suggestions button,
|
| 369 |
+
.query-history button,
|
| 370 |
+
.citation-card button {
|
| 371 |
+
min-height: 32px;
|
| 372 |
+
padding: 0 10px;
|
| 373 |
+
border: 1px solid var(--line);
|
| 374 |
+
border-radius: 6px;
|
| 375 |
+
background: white;
|
| 376 |
+
color: var(--text);
|
| 377 |
+
font-size: 13px;
|
| 378 |
+
font-weight: 700;
|
| 379 |
+
}
|
| 380 |
+
|
| 381 |
+
.query-suggestions button:hover,
|
| 382 |
+
.query-history button:hover,
|
| 383 |
+
.citation-card button:hover {
|
| 384 |
+
border-color: var(--accent);
|
| 385 |
+
background: #edf8f5;
|
| 386 |
+
color: var(--accent-strong);
|
| 387 |
+
}
|
| 388 |
+
|
| 389 |
+
.details-panel {
|
| 390 |
+
grid-column: 1 / -1;
|
| 391 |
+
min-height: auto;
|
| 392 |
+
}
|
| 393 |
+
|
| 394 |
+
.details-panel details {
|
| 395 |
+
padding: 16px 18px;
|
| 396 |
+
}
|
| 397 |
+
|
| 398 |
+
.details-panel summary {
|
| 399 |
+
cursor: pointer;
|
| 400 |
+
color: var(--accent-strong);
|
| 401 |
+
font-weight: 700;
|
| 402 |
+
}
|
| 403 |
+
|
| 404 |
+
.details-panel pre {
|
| 405 |
+
max-height: 360px;
|
| 406 |
+
margin-top: 14px;
|
| 407 |
+
border: 1px solid var(--line);
|
| 408 |
+
border-radius: 8px;
|
| 409 |
+
background: #fbfcfc;
|
| 410 |
+
}
|
| 411 |
+
|
| 412 |
+
.query-form {
|
| 413 |
+
display: grid;
|
| 414 |
+
grid-template-columns: minmax(0, 1fr) auto;
|
| 415 |
+
gap: 10px;
|
| 416 |
+
padding: 0 18px;
|
| 417 |
+
}
|
| 418 |
+
|
| 419 |
+
.feedback-form {
|
| 420 |
+
display: none;
|
| 421 |
+
grid-template-columns: 84px minmax(0, 1fr) auto;
|
| 422 |
+
gap: 10px;
|
| 423 |
+
padding: 0 18px;
|
| 424 |
+
}
|
| 425 |
+
|
| 426 |
+
.query-form input,
|
| 427 |
+
.feedback-form input,
|
| 428 |
+
.feedback-form select {
|
| 429 |
+
min-height: 40px;
|
| 430 |
+
padding: 0 11px;
|
| 431 |
+
border: 1px solid var(--line);
|
| 432 |
+
border-radius: 6px;
|
| 433 |
+
}
|
| 434 |
+
|
| 435 |
+
.answer-box,
|
| 436 |
+
.citation-list,
|
| 437 |
+
.retrieved-chunks {
|
| 438 |
+
margin: 0 18px;
|
| 439 |
+
}
|
| 440 |
+
|
| 441 |
+
.confidence-badge {
|
| 442 |
+
justify-self: start;
|
| 443 |
+
margin: 0 18px;
|
| 444 |
+
padding: 7px 10px;
|
| 445 |
+
border: 1px solid var(--line);
|
| 446 |
+
border-radius: 999px;
|
| 447 |
+
background: white;
|
| 448 |
+
color: var(--muted);
|
| 449 |
+
font-size: 13px;
|
| 450 |
+
font-weight: 900;
|
| 451 |
+
}
|
| 452 |
+
|
| 453 |
+
.confidence-badge.high {
|
| 454 |
+
border-color: #94cfaa;
|
| 455 |
+
background: #eef9f2;
|
| 456 |
+
color: var(--success);
|
| 457 |
+
}
|
| 458 |
+
|
| 459 |
+
.confidence-badge.medium {
|
| 460 |
+
border-color: #e8c77d;
|
| 461 |
+
background: #fff8e8;
|
| 462 |
+
color: var(--warning);
|
| 463 |
+
}
|
| 464 |
+
|
| 465 |
+
.confidence-badge.low {
|
| 466 |
+
border-color: #dfaaaa;
|
| 467 |
+
background: #fff3f3;
|
| 468 |
+
color: var(--danger);
|
| 469 |
+
}
|
| 470 |
+
|
| 471 |
+
.answer-box {
|
| 472 |
+
min-height: 210px;
|
| 473 |
+
padding: 14px;
|
| 474 |
+
border: 1px solid var(--line);
|
| 475 |
+
border-radius: 8px;
|
| 476 |
+
background: var(--surface-strong);
|
| 477 |
+
line-height: 1.55;
|
| 478 |
+
white-space: pre-wrap;
|
| 479 |
+
font-size: 17px;
|
| 480 |
+
}
|
| 481 |
+
|
| 482 |
+
.citation-list {
|
| 483 |
+
display: grid;
|
| 484 |
+
gap: 10px;
|
| 485 |
+
}
|
| 486 |
+
|
| 487 |
+
.retrieved-chunks {
|
| 488 |
+
display: none;
|
| 489 |
+
}
|
| 490 |
+
|
| 491 |
+
.citation-card,
|
| 492 |
+
.chunk-card {
|
| 493 |
+
padding: 12px;
|
| 494 |
+
border: 1px solid var(--line);
|
| 495 |
+
border-radius: 8px;
|
| 496 |
+
background: #fbfcfc;
|
| 497 |
+
}
|
| 498 |
+
|
| 499 |
+
.citation-card strong,
|
| 500 |
+
.chunk-card strong {
|
| 501 |
+
display: block;
|
| 502 |
+
margin-bottom: 6px;
|
| 503 |
+
}
|
| 504 |
+
|
| 505 |
+
.chunk-card p,
|
| 506 |
+
.citation-card p {
|
| 507 |
+
color: var(--text);
|
| 508 |
+
font-size: 13px;
|
| 509 |
+
line-height: 1.5;
|
| 510 |
+
}
|
| 511 |
+
|
| 512 |
+
@media (max-width: 860px) {
|
| 513 |
+
.app-shell {
|
| 514 |
+
width: min(100% - 20px, 680px);
|
| 515 |
+
padding: 10px 0;
|
| 516 |
+
}
|
| 517 |
+
|
| 518 |
+
.workspace {
|
| 519 |
+
grid-template-columns: 1fr;
|
| 520 |
+
}
|
| 521 |
+
|
| 522 |
+
.result-panel {
|
| 523 |
+
min-height: 55vh;
|
| 524 |
+
}
|
| 525 |
+
|
| 526 |
+
.query-form,
|
| 527 |
+
.feedback-form {
|
| 528 |
+
grid-template-columns: 1fr;
|
| 529 |
+
}
|
| 530 |
+
}
|
| 531 |
+
|
| 532 |
+
.dashboard-panel {
|
| 533 |
+
grid-column: 1 / -1;
|
| 534 |
+
display: grid;
|
| 535 |
+
gap: 0;
|
| 536 |
+
padding-bottom: 0;
|
| 537 |
+
}
|
| 538 |
+
|
| 539 |
+
.advanced-dashboard {
|
| 540 |
+
padding: 16px 18px 18px;
|
| 541 |
+
}
|
| 542 |
+
|
| 543 |
+
.advanced-dashboard summary {
|
| 544 |
+
cursor: pointer;
|
| 545 |
+
color: var(--accent-strong);
|
| 546 |
+
font-weight: 800;
|
| 547 |
+
}
|
| 548 |
+
|
| 549 |
+
.dashboard-grid {
|
| 550 |
+
display: grid;
|
| 551 |
+
grid-template-columns: repeat(2, minmax(0, 1fr));
|
| 552 |
+
gap: 14px;
|
| 553 |
+
padding: 16px 0 0;
|
| 554 |
+
}
|
| 555 |
+
|
| 556 |
+
.metric-section {
|
| 557 |
+
min-width: 0;
|
| 558 |
+
padding: 14px;
|
| 559 |
+
border: 1px solid var(--line);
|
| 560 |
+
border-radius: 8px;
|
| 561 |
+
background: #fbfcfc;
|
| 562 |
+
}
|
| 563 |
+
|
| 564 |
+
.metric-section h2 {
|
| 565 |
+
font-size: 16px;
|
| 566 |
+
}
|
| 567 |
+
|
| 568 |
+
.timeline-list {
|
| 569 |
+
display: grid;
|
| 570 |
+
gap: 8px;
|
| 571 |
+
margin-top: 12px;
|
| 572 |
+
}
|
| 573 |
+
|
| 574 |
+
.timeline-item {
|
| 575 |
+
display: grid;
|
| 576 |
+
grid-template-columns: 34px minmax(0, 1fr);
|
| 577 |
+
align-items: center;
|
| 578 |
+
gap: 10px;
|
| 579 |
+
padding: 8px 0;
|
| 580 |
+
border-bottom: 1px solid #edf1f3;
|
| 581 |
+
}
|
| 582 |
+
|
| 583 |
+
.timeline-item span {
|
| 584 |
+
display: grid;
|
| 585 |
+
place-items: center;
|
| 586 |
+
width: 28px;
|
| 587 |
+
height: 28px;
|
| 588 |
+
border-radius: 50%;
|
| 589 |
+
background: var(--accent);
|
| 590 |
+
color: white;
|
| 591 |
+
font-size: 10px;
|
| 592 |
+
font-weight: 800;
|
| 593 |
+
}
|
| 594 |
+
|
| 595 |
+
.timeline-item.pending span {
|
| 596 |
+
background: #d7dee3;
|
| 597 |
+
}
|
| 598 |
+
|
| 599 |
+
.timeline-item.current span {
|
| 600 |
+
background: var(--info);
|
| 601 |
+
animation: pulse 1.2s ease-in-out infinite;
|
| 602 |
+
}
|
| 603 |
+
|
| 604 |
+
.timeline-item.done span {
|
| 605 |
+
background: var(--success);
|
| 606 |
+
}
|
| 607 |
+
|
| 608 |
+
@keyframes pulse {
|
| 609 |
+
0%,
|
| 610 |
+
100% {
|
| 611 |
+
transform: scale(1);
|
| 612 |
+
}
|
| 613 |
+
50% {
|
| 614 |
+
transform: scale(1.08);
|
| 615 |
+
}
|
| 616 |
+
}
|
| 617 |
+
|
| 618 |
+
.comparison-table {
|
| 619 |
+
display: grid;
|
| 620 |
+
gap: 6px;
|
| 621 |
+
margin-top: 12px;
|
| 622 |
+
}
|
| 623 |
+
|
| 624 |
+
.table-row {
|
| 625 |
+
display: grid;
|
| 626 |
+
grid-template-columns: 1fr 70px 70px;
|
| 627 |
+
gap: 10px;
|
| 628 |
+
align-items: center;
|
| 629 |
+
padding: 9px 10px;
|
| 630 |
+
border-radius: 6px;
|
| 631 |
+
background: white;
|
| 632 |
+
border: 1px solid var(--line);
|
| 633 |
+
font-size: 13px;
|
| 634 |
+
}
|
| 635 |
+
|
| 636 |
+
.table-head {
|
| 637 |
+
color: var(--muted);
|
| 638 |
+
font-weight: 700;
|
| 639 |
+
text-transform: uppercase;
|
| 640 |
+
font-size: 11px;
|
| 641 |
+
}
|
| 642 |
+
|
| 643 |
+
.table-row.winner {
|
| 644 |
+
border-color: var(--accent);
|
| 645 |
+
background: #edf8f5;
|
| 646 |
+
}
|
| 647 |
+
|
| 648 |
+
.table-row.winner span:first-child::before {
|
| 649 |
+
content: "Winner ";
|
| 650 |
+
color: var(--accent-strong);
|
| 651 |
+
font-weight: 800;
|
| 652 |
+
}
|
| 653 |
+
|
| 654 |
+
.winner-badge {
|
| 655 |
+
padding: 5px 8px;
|
| 656 |
+
border: 1px solid #b9d5cc;
|
| 657 |
+
border-radius: 999px;
|
| 658 |
+
background: #edf8f5;
|
| 659 |
+
color: var(--accent-strong) !important;
|
| 660 |
+
font-weight: 900;
|
| 661 |
+
}
|
| 662 |
+
|
| 663 |
+
.stat-grid {
|
| 664 |
+
display: grid;
|
| 665 |
+
grid-template-columns: repeat(2, minmax(0, 1fr));
|
| 666 |
+
gap: 10px;
|
| 667 |
+
margin-top: 12px;
|
| 668 |
+
}
|
| 669 |
+
|
| 670 |
+
.stat-grid div {
|
| 671 |
+
display: grid;
|
| 672 |
+
gap: 4px;
|
| 673 |
+
padding: 12px;
|
| 674 |
+
border: 1px solid var(--line);
|
| 675 |
+
border-radius: 8px;
|
| 676 |
+
background: white;
|
| 677 |
+
}
|
| 678 |
+
|
| 679 |
+
.stat-grid span {
|
| 680 |
+
color: var(--muted);
|
| 681 |
+
font-size: 12px;
|
| 682 |
+
}
|
| 683 |
+
|
| 684 |
+
.stat-grid strong {
|
| 685 |
+
font-size: 18px;
|
| 686 |
+
line-height: 1.2;
|
| 687 |
+
overflow-wrap: anywhere;
|
| 688 |
+
}
|
| 689 |
+
|
| 690 |
+
.answer-card {
|
| 691 |
+
display: grid;
|
| 692 |
+
gap: 10px;
|
| 693 |
+
}
|
| 694 |
+
|
| 695 |
+
.answer-card > span {
|
| 696 |
+
margin: 0 18px;
|
| 697 |
+
color: var(--muted);
|
| 698 |
+
font-size: 12px;
|
| 699 |
+
font-weight: 800;
|
| 700 |
+
text-transform: uppercase;
|
| 701 |
+
}
|
| 702 |
+
|
| 703 |
+
.citation-title {
|
| 704 |
+
display: grid;
|
| 705 |
+
grid-template-columns: 42px minmax(0, 1fr);
|
| 706 |
+
gap: 10px;
|
| 707 |
+
align-items: center;
|
| 708 |
+
margin-bottom: 10px;
|
| 709 |
+
}
|
| 710 |
+
|
| 711 |
+
.citation-file-badge {
|
| 712 |
+
display: grid !important;
|
| 713 |
+
place-items: center;
|
| 714 |
+
width: 38px;
|
| 715 |
+
height: 42px;
|
| 716 |
+
margin: 0 !important;
|
| 717 |
+
border-radius: 6px;
|
| 718 |
+
background: #d84b4b;
|
| 719 |
+
color: white !important;
|
| 720 |
+
font-size: 11px !important;
|
| 721 |
+
font-weight: 900;
|
| 722 |
+
}
|
| 723 |
+
|
| 724 |
+
.citation-meta {
|
| 725 |
+
display: flex;
|
| 726 |
+
flex-wrap: wrap;
|
| 727 |
+
gap: 8px;
|
| 728 |
+
align-items: center;
|
| 729 |
+
margin-bottom: 8px;
|
| 730 |
+
}
|
| 731 |
+
|
| 732 |
+
.citation-card span {
|
| 733 |
+
display: block;
|
| 734 |
+
margin-bottom: 6px;
|
| 735 |
+
color: var(--muted);
|
| 736 |
+
font-size: 12px;
|
| 737 |
+
}
|
| 738 |
+
|
| 739 |
+
.citation-meta span {
|
| 740 |
+
margin: 0;
|
| 741 |
+
padding: 7px 9px;
|
| 742 |
+
border: 1px solid var(--line);
|
| 743 |
+
border-radius: 6px;
|
| 744 |
+
background: white;
|
| 745 |
+
}
|
| 746 |
+
|
| 747 |
+
.citation-card button {
|
| 748 |
+
display: inline-flex;
|
| 749 |
+
align-items: center;
|
| 750 |
+
margin: 0;
|
| 751 |
+
}
|
| 752 |
+
|
| 753 |
+
.retrieval-section,
|
| 754 |
+
.recommendation-section {
|
| 755 |
+
margin: 14px 0 0;
|
| 756 |
+
}
|
| 757 |
+
|
| 758 |
+
.retrieved-chunks {
|
| 759 |
+
display: grid;
|
| 760 |
+
gap: 10px;
|
| 761 |
+
margin: 12px 0 0;
|
| 762 |
+
}
|
| 763 |
+
|
| 764 |
+
.chunk-card summary {
|
| 765 |
+
display: grid;
|
| 766 |
+
grid-template-columns: 44px 150px minmax(0, 1fr);
|
| 767 |
+
gap: 10px;
|
| 768 |
+
align-items: center;
|
| 769 |
+
cursor: pointer;
|
| 770 |
+
}
|
| 771 |
+
|
| 772 |
+
.chunk-card summary span {
|
| 773 |
+
color: var(--accent-strong);
|
| 774 |
+
font-weight: 800;
|
| 775 |
+
}
|
| 776 |
+
|
| 777 |
+
.chunk-summary-scores {
|
| 778 |
+
display: flex;
|
| 779 |
+
flex-wrap: wrap;
|
| 780 |
+
gap: 6px;
|
| 781 |
+
}
|
| 782 |
+
|
| 783 |
+
.chunk-card summary em {
|
| 784 |
+
padding: 5px 7px;
|
| 785 |
+
border: 1px solid var(--line);
|
| 786 |
+
border-radius: 6px;
|
| 787 |
+
background: white;
|
| 788 |
+
color: var(--muted);
|
| 789 |
+
font-style: normal;
|
| 790 |
+
font-size: 12px;
|
| 791 |
+
overflow-wrap: anywhere;
|
| 792 |
+
}
|
| 793 |
+
|
| 794 |
+
.chunk-card p strong {
|
| 795 |
+
margin-top: 12px;
|
| 796 |
+
}
|
| 797 |
+
|
| 798 |
+
.chunk-scores {
|
| 799 |
+
display: grid;
|
| 800 |
+
grid-template-columns: repeat(3, minmax(0, 1fr));
|
| 801 |
+
gap: 8px;
|
| 802 |
+
margin-top: 10px;
|
| 803 |
+
}
|
| 804 |
+
|
| 805 |
+
.chunk-scores span {
|
| 806 |
+
padding: 8px;
|
| 807 |
+
border: 1px solid var(--line);
|
| 808 |
+
border-radius: 6px;
|
| 809 |
+
background: white;
|
| 810 |
+
color: var(--muted);
|
| 811 |
+
font-size: 12px;
|
| 812 |
+
}
|
| 813 |
+
|
| 814 |
+
.chunk-scores strong {
|
| 815 |
+
display: block;
|
| 816 |
+
margin-top: 3px;
|
| 817 |
+
color: var(--text);
|
| 818 |
+
}
|
| 819 |
+
|
| 820 |
+
.recommendation-box {
|
| 821 |
+
margin-top: 10px;
|
| 822 |
+
padding: 14px;
|
| 823 |
+
border-radius: 8px;
|
| 824 |
+
background: #edf8f5;
|
| 825 |
+
color: #244a45;
|
| 826 |
+
font-weight: 700;
|
| 827 |
+
}
|
| 828 |
+
|
| 829 |
+
.recommendation-box strong,
|
| 830 |
+
.recommendation-box span {
|
| 831 |
+
display: block;
|
| 832 |
+
}
|
| 833 |
+
|
| 834 |
+
.recommendation-box span {
|
| 835 |
+
margin-top: 5px;
|
| 836 |
+
font-size: 18px;
|
| 837 |
+
}
|
| 838 |
+
|
| 839 |
+
.recommendation-box p {
|
| 840 |
+
margin-top: 10px;
|
| 841 |
+
}
|
| 842 |
+
|
| 843 |
+
.recommendation-box ul {
|
| 844 |
+
margin: 8px 0 0;
|
| 845 |
+
padding-left: 20px;
|
| 846 |
+
}
|
| 847 |
+
|
| 848 |
+
.recommendation-box li {
|
| 849 |
+
margin-top: 4px;
|
| 850 |
+
}
|
| 851 |
+
|
| 852 |
+
.crag-box {
|
| 853 |
+
display: grid;
|
| 854 |
+
gap: 6px;
|
| 855 |
+
margin-top: 12px;
|
| 856 |
+
padding: 14px;
|
| 857 |
+
border: 1px solid var(--line);
|
| 858 |
+
border-radius: 8px;
|
| 859 |
+
background: white;
|
| 860 |
+
}
|
| 861 |
+
|
| 862 |
+
.crag-box strong {
|
| 863 |
+
font-size: 22px;
|
| 864 |
+
}
|
| 865 |
+
|
| 866 |
+
.crag-box span {
|
| 867 |
+
color: var(--muted);
|
| 868 |
+
font-size: 13px;
|
| 869 |
+
}
|
| 870 |
+
|
| 871 |
+
.crag-box.good {
|
| 872 |
+
border-color: #8cc9b6;
|
| 873 |
+
background: #edf8f5;
|
| 874 |
+
}
|
| 875 |
+
|
| 876 |
+
.crag-box.bad {
|
| 877 |
+
border-color: #dfaaaa;
|
| 878 |
+
background: #fff3f3;
|
| 879 |
+
}
|
| 880 |
+
|
| 881 |
+
.pipeline-viz {
|
| 882 |
+
display: grid;
|
| 883 |
+
gap: 7px;
|
| 884 |
+
margin-top: 12px;
|
| 885 |
+
}
|
| 886 |
+
|
| 887 |
+
.pipeline-viz div {
|
| 888 |
+
padding: 9px 10px;
|
| 889 |
+
border: 1px solid var(--line);
|
| 890 |
+
border-radius: 6px;
|
| 891 |
+
background: white;
|
| 892 |
+
font-size: 13px;
|
| 893 |
+
font-weight: 800;
|
| 894 |
+
overflow-wrap: anywhere;
|
| 895 |
+
}
|
| 896 |
+
|
| 897 |
+
.pipeline-viz span {
|
| 898 |
+
width: 1px;
|
| 899 |
+
height: 12px;
|
| 900 |
+
margin-left: 18px;
|
| 901 |
+
background: var(--line);
|
| 902 |
+
position: relative;
|
| 903 |
+
}
|
| 904 |
+
|
| 905 |
+
.pipeline-viz span::after {
|
| 906 |
+
content: "";
|
| 907 |
+
position: absolute;
|
| 908 |
+
left: -4px;
|
| 909 |
+
bottom: -1px;
|
| 910 |
+
width: 8px;
|
| 911 |
+
height: 8px;
|
| 912 |
+
border-right: 1px solid var(--line);
|
| 913 |
+
border-bottom: 1px solid var(--line);
|
| 914 |
+
transform: rotate(45deg);
|
| 915 |
+
}
|
| 916 |
+
|
| 917 |
+
.hallucination-card {
|
| 918 |
+
display: grid;
|
| 919 |
+
grid-template-columns: 1fr auto;
|
| 920 |
+
gap: 8px 12px;
|
| 921 |
+
margin-top: 12px;
|
| 922 |
+
padding: 14px;
|
| 923 |
+
border: 1px solid var(--line);
|
| 924 |
+
border-radius: 8px;
|
| 925 |
+
background: white;
|
| 926 |
+
}
|
| 927 |
+
|
| 928 |
+
.hallucination-card span {
|
| 929 |
+
color: var(--muted);
|
| 930 |
+
font-size: 13px;
|
| 931 |
+
}
|
| 932 |
+
|
| 933 |
+
.hallucination-card strong {
|
| 934 |
+
font-size: 20px;
|
| 935 |
+
}
|
| 936 |
+
|
| 937 |
+
.hallucination-card em {
|
| 938 |
+
grid-column: 1 / -1;
|
| 939 |
+
padding-top: 8px;
|
| 940 |
+
border-top: 1px solid var(--line);
|
| 941 |
+
color: var(--muted);
|
| 942 |
+
font-style: normal;
|
| 943 |
+
font-weight: 900;
|
| 944 |
+
}
|
| 945 |
+
|
| 946 |
+
.hallucination-card.good {
|
| 947 |
+
border-color: #94cfaa;
|
| 948 |
+
background: #eef9f2;
|
| 949 |
+
}
|
| 950 |
+
|
| 951 |
+
.hallucination-card.bad {
|
| 952 |
+
border-color: #dfaaaa;
|
| 953 |
+
background: #fff3f3;
|
| 954 |
+
}
|
| 955 |
+
|
| 956 |
+
.history-panel {
|
| 957 |
+
grid-column: 1 / -1;
|
| 958 |
+
}
|
| 959 |
+
|
| 960 |
+
.query-history {
|
| 961 |
+
display: flex;
|
| 962 |
+
flex-wrap: wrap;
|
| 963 |
+
gap: 8px;
|
| 964 |
+
padding: 16px 18px;
|
| 965 |
+
}
|
| 966 |
+
|
| 967 |
+
.query-history > span {
|
| 968 |
+
color: var(--muted);
|
| 969 |
+
font-size: 14px;
|
| 970 |
+
}
|
| 971 |
+
|
| 972 |
+
.page-chip.is-active {
|
| 973 |
+
outline: 3px solid rgba(31, 122, 109, 0.22);
|
| 974 |
+
border: 1px solid var(--accent);
|
| 975 |
+
background: #dff4ee;
|
| 976 |
+
}
|
| 977 |
+
|
| 978 |
+
.empty-state {
|
| 979 |
+
padding: 12px;
|
| 980 |
+
color: var(--muted);
|
| 981 |
+
border: 1px dashed var(--line);
|
| 982 |
+
border-radius: 8px;
|
| 983 |
+
}
|
| 984 |
+
|
| 985 |
+
@media (max-width: 860px) {
|
| 986 |
+
.dashboard-grid {
|
| 987 |
+
grid-template-columns: 1fr;
|
| 988 |
+
}
|
| 989 |
+
|
| 990 |
+
.chunk-card summary {
|
| 991 |
+
grid-template-columns: 1fr;
|
| 992 |
+
}
|
| 993 |
+
|
| 994 |
+
.chunk-scores,
|
| 995 |
+
.hallucination-card {
|
| 996 |
+
grid-template-columns: 1fr;
|
| 997 |
+
}
|
| 998 |
+
}
|
templates/index.html
ADDED
|
@@ -0,0 +1,203 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
<!doctype html>
|
| 2 |
+
<html lang="en">
|
| 3 |
+
<head>
|
| 4 |
+
<meta charset="utf-8" />
|
| 5 |
+
<meta name="viewport" content="width=device-width, initial-scale=1" />
|
| 6 |
+
<title>SmartNotes PDF OCR</title>
|
| 7 |
+
<link rel="stylesheet" href="/static/styles.css" />
|
| 8 |
+
</head>
|
| 9 |
+
<body>
|
| 10 |
+
<main class="app-shell">
|
| 11 |
+
<section class="workspace">
|
| 12 |
+
<div class="panel upload-panel">
|
| 13 |
+
<div class="brand-row">
|
| 14 |
+
<span class="brand-mark">SN</span>
|
| 15 |
+
<div>
|
| 16 |
+
<h1>SmartNotes AI</h1>
|
| 17 |
+
<p>Advanced RAG-Powered PDF Intelligence Platform</p>
|
| 18 |
+
<span class="brand-subline">Hybrid Retrieval / Reranking / Citations / Evaluation</span>
|
| 19 |
+
</div>
|
| 20 |
+
</div>
|
| 21 |
+
|
| 22 |
+
<form id="uploadForm" class="drop-zone">
|
| 23 |
+
<input id="pdfInput" type="file" name="file" accept="application/pdf" />
|
| 24 |
+
<div class="drop-content">
|
| 25 |
+
<strong id="fileName">Choose or drop a PDF</strong>
|
| 26 |
+
<span>Direct text first, OCR only when needed</span>
|
| 27 |
+
</div>
|
| 28 |
+
<button type="submit">Upload PDF</button>
|
| 29 |
+
</form>
|
| 30 |
+
|
| 31 |
+
<div id="fileCard" class="file-card is-hidden">
|
| 32 |
+
<div class="file-icon">PDF</div>
|
| 33 |
+
<div>
|
| 34 |
+
<strong id="uploadedFileName">notes.pdf</strong>
|
| 35 |
+
<span id="uploadedFileMeta">0 Pages / 0 MB</span>
|
| 36 |
+
<em>Uploaded Successfully</em>
|
| 37 |
+
</div>
|
| 38 |
+
</div>
|
| 39 |
+
|
| 40 |
+
<div id="pdfPreviewCard" class="pdf-preview-card is-hidden">
|
| 41 |
+
<div class="section-head">
|
| 42 |
+
<h2>PDF Preview</h2>
|
| 43 |
+
<span>First page</span>
|
| 44 |
+
</div>
|
| 45 |
+
<iframe id="pdfPreview" title="PDF first page preview"></iframe>
|
| 46 |
+
</div>
|
| 47 |
+
|
| 48 |
+
<div class="status-grid">
|
| 49 |
+
<div>
|
| 50 |
+
<span>Route</span>
|
| 51 |
+
<strong id="routeValue">Waiting</strong>
|
| 52 |
+
</div>
|
| 53 |
+
<div>
|
| 54 |
+
<span>Pages</span>
|
| 55 |
+
<strong id="pageCount">0</strong>
|
| 56 |
+
</div>
|
| 57 |
+
<div>
|
| 58 |
+
<span>Direct Text</span>
|
| 59 |
+
<strong id="directText">-</strong>
|
| 60 |
+
</div>
|
| 61 |
+
</div>
|
| 62 |
+
<div id="uploadMessage" class="upload-message">Upload a PDF to begin.</div>
|
| 63 |
+
</div>
|
| 64 |
+
|
| 65 |
+
<div class="panel query-panel">
|
| 66 |
+
<div class="result-toolbar">
|
| 67 |
+
<h2>Ask PDF</h2>
|
| 68 |
+
<span id="activeDocument">No document indexed</span>
|
| 69 |
+
</div>
|
| 70 |
+
<form id="queryForm" class="query-form">
|
| 71 |
+
<input id="queryInput" type="text" placeholder="Ask your question here" />
|
| 72 |
+
<button type="submit">Ask</button>
|
| 73 |
+
</form>
|
| 74 |
+
<div class="answer-card">
|
| 75 |
+
<span>Answer</span>
|
| 76 |
+
<div id="confidenceBadge" class="confidence-badge neutral">Confidence pending</div>
|
| 77 |
+
<div id="answerBox" class="answer-box">Upload a PDF first. Answer will appear here.</div>
|
| 78 |
+
<div id="citations" class="citation-list"></div>
|
| 79 |
+
</div>
|
| 80 |
+
</div>
|
| 81 |
+
|
| 82 |
+
<div class="panel dashboard-panel">
|
| 83 |
+
<div class="result-toolbar">
|
| 84 |
+
<h2>Retrieval Analytics</h2>
|
| 85 |
+
<span id="dashboardStatus">Waiting for PDF</span>
|
| 86 |
+
</div>
|
| 87 |
+
<details class="advanced-dashboard" open>
|
| 88 |
+
<summary>RAG Analytics Dashboard</summary>
|
| 89 |
+
<div class="dashboard-grid">
|
| 90 |
+
<section class="metric-section">
|
| 91 |
+
<div class="section-head">
|
| 92 |
+
<h2>Indexing Timeline</h2>
|
| 93 |
+
<span id="docStatus">Waiting</span>
|
| 94 |
+
</div>
|
| 95 |
+
<div id="indexSteps" class="timeline-list"></div>
|
| 96 |
+
</section>
|
| 97 |
+
|
| 98 |
+
<section class="metric-section">
|
| 99 |
+
<div class="section-head">
|
| 100 |
+
<h2>Chunking Comparison</h2>
|
| 101 |
+
<span id="bestChunking" class="winner-badge">Evaluation Pending</span>
|
| 102 |
+
</div>
|
| 103 |
+
<div id="chunkComparison" class="comparison-table"></div>
|
| 104 |
+
</section>
|
| 105 |
+
|
| 106 |
+
<section class="metric-section">
|
| 107 |
+
<h2>Retrieval Metrics</h2>
|
| 108 |
+
<div id="hybridStats" class="stat-grid">
|
| 109 |
+
<div><span>Vector</span><strong>-</strong></div>
|
| 110 |
+
<div><span>BM25</span><strong>-</strong></div>
|
| 111 |
+
<div><span>Merged</span><strong>-</strong></div>
|
| 112 |
+
<div><span>Reranked</span><strong>-</strong></div>
|
| 113 |
+
</div>
|
| 114 |
+
</section>
|
| 115 |
+
|
| 116 |
+
<section class="metric-section">
|
| 117 |
+
<h2>Evaluation Dashboard</h2>
|
| 118 |
+
<div id="evaluationCards" class="stat-grid">
|
| 119 |
+
<div><span>Precision</span><strong>-</strong></div>
|
| 120 |
+
<div><span>Recall</span><strong>-</strong></div>
|
| 121 |
+
<div><span>Faithfulness</span><strong>-</strong></div>
|
| 122 |
+
<div><span>Relevancy</span><strong>-</strong></div>
|
| 123 |
+
</div>
|
| 124 |
+
</section>
|
| 125 |
+
|
| 126 |
+
<section class="metric-section">
|
| 127 |
+
<h2>CRAG Retrieval Quality</h2>
|
| 128 |
+
<div id="cragStatus" class="crag-box neutral">
|
| 129 |
+
<strong>Waiting</strong>
|
| 130 |
+
<span>Run a query to grade retrieval quality.</span>
|
| 131 |
+
</div>
|
| 132 |
+
</section>
|
| 133 |
+
|
| 134 |
+
<section class="metric-section">
|
| 135 |
+
<h2>Retrieval Pipeline</h2>
|
| 136 |
+
<div id="pipelineViz" class="pipeline-viz"></div>
|
| 137 |
+
</section>
|
| 138 |
+
|
| 139 |
+
<section class="metric-section">
|
| 140 |
+
<h2>Hallucination Status</h2>
|
| 141 |
+
<div id="hallucinationCard" class="hallucination-card neutral">
|
| 142 |
+
<span>Answer Source Coverage</span>
|
| 143 |
+
<strong>-</strong>
|
| 144 |
+
<span>Faithfulness</span>
|
| 145 |
+
<strong>-</strong>
|
| 146 |
+
<em>Waiting for answer</em>
|
| 147 |
+
</div>
|
| 148 |
+
</section>
|
| 149 |
+
</div>
|
| 150 |
+
|
| 151 |
+
<section class="metric-section retrieval-section">
|
| 152 |
+
<div class="section-head">
|
| 153 |
+
<h2>Top Retrieved Chunks</h2>
|
| 154 |
+
<span>🧠 Parent-Child Retrieval</span>
|
| 155 |
+
</div>
|
| 156 |
+
<div id="retrievedChunks" class="retrieved-chunks"></div>
|
| 157 |
+
</section>
|
| 158 |
+
|
| 159 |
+
<section class="metric-section recommendation-section">
|
| 160 |
+
<h2>Best Chunking Recommendation</h2>
|
| 161 |
+
<div id="bestRecommendation" class="recommendation-box">Index a PDF to compare chunking strategies.</div>
|
| 162 |
+
</section>
|
| 163 |
+
</details>
|
| 164 |
+
</div>
|
| 165 |
+
|
| 166 |
+
<div class="panel history-panel">
|
| 167 |
+
<div class="result-toolbar">
|
| 168 |
+
<h2>Recent Questions</h2>
|
| 169 |
+
<span id="historyCount">0 saved</span>
|
| 170 |
+
</div>
|
| 171 |
+
<div id="queryHistory" class="query-history">
|
| 172 |
+
<span>No questions yet.</span>
|
| 173 |
+
</div>
|
| 174 |
+
</div>
|
| 175 |
+
|
| 176 |
+
<div class="panel details-panel">
|
| 177 |
+
<div class="result-toolbar">
|
| 178 |
+
<h2>Extracted Text</h2>
|
| 179 |
+
<button id="copyButton" type="button">Copy Text</button>
|
| 180 |
+
</div>
|
| 181 |
+
<details>
|
| 182 |
+
<summary>Show extracted text and page OCR details</summary>
|
| 183 |
+
<div id="pageDetails" class="page-details"></div>
|
| 184 |
+
<pre id="resultText">Upload a PDF to see extracted text here.</pre>
|
| 185 |
+
</details>
|
| 186 |
+
<form id="feedbackForm" class="feedback-form">
|
| 187 |
+
<select id="feedbackRating">
|
| 188 |
+
<option value="5">5</option>
|
| 189 |
+
<option value="4">4</option>
|
| 190 |
+
<option value="3">3</option>
|
| 191 |
+
<option value="2">2</option>
|
| 192 |
+
<option value="1">1</option>
|
| 193 |
+
</select>
|
| 194 |
+
<input id="feedbackComment" type="text" placeholder="Feedback" />
|
| 195 |
+
<button type="submit">Save Feedback</button>
|
| 196 |
+
</form>
|
| 197 |
+
</div>
|
| 198 |
+
</section>
|
| 199 |
+
</main>
|
| 200 |
+
|
| 201 |
+
<script src="/static/app.js?v=20260625-safe-json-errors"></script>
|
| 202 |
+
</body>
|
| 203 |
+
</html>
|
uploads/.gitkeep
ADDED
|
@@ -0,0 +1 @@
|
|
|
|
|
|
|
| 1 |
+
|