Goutam112 commited on
Commit
a42a7e2
Β·
1 Parent(s): f99e19b

solved error

Browse files
README.md CHANGED
@@ -3,62 +3,96 @@
3
  [![Python](https://img.shields.io/badge/Python-3.10%2B-blue.svg)](https://www.python.org/)
4
  [![Streamlit](https://img.shields.io/badge/Streamlit-1.40%2B-FF4B4B.svg)](https://streamlit.io/)
5
  [![EasyOCR](https://img.shields.io/badge/EasyOCR-PyTorch%20GPU-green.svg)](https://github.com/JaidedAI/EasyOCR)
 
6
  [![License](https://img.shields.io/badge/License-MIT-purple.svg)](LICENSE)
7
 
8
- **DocIQ** is an Intelligent Document Processing (IDP) platform built with Streamlit, EasyOCR, OpenCV, Sentence-Transformers, and PyMuPDF. It transforms raw document images, receipts, invoices, and scans into clean, structured, searchable, and semantically indexed data.
 
 
9
 
10
  ---
11
 
12
- ## πŸš€ Key Features
13
-
14
- ### 1. πŸ–ΌοΈ Computer Vision Preprocessing Pipeline (`src/preprocessor.py`)
15
- - **Auto-Deskew**: Automatically detects document tilt using Hough line transform and corrects orientation.
16
- - **Adaptive Contrast (CLAHE)**: Enhances faded or low-contrast document scans.
17
- - **Bilateral Denoising**: Smooths scanner grain while preserving crisp character edges.
18
- - **Multiple Modes**:
19
- - `auto`: Deskew + CLAHE + Denoise
20
- - `binarize`: High-contrast black & white document thresholding (Otsu adaptive)
21
- - `warp`: Perspective transformation for flattened documents
22
- - `raw`: Process image as uploaded
23
-
24
- ### 2. ⚑ Accelerated OCR Inference (`src/ocr_engine.py`)
25
- - Powered by **EasyOCR** with PyTorch CUDA GPU acceleration.
26
- - Built-in multi-lingual support: **English (`en`)**, **Hindi (`hi`)**, French, German, Spanish, Chinese, and Arabic.
27
- - Caching with `@st.cache_resource` for zero-redundancy model loading across sessions.
28
- - Detailed bounding box coordinates `[x_min, y_min, x_max, y_max]`, confidence scores, and reading order.
29
-
30
- ### 3. 🎨 Visual Canvas & Spatial Inspection (`src/visualizer.py`)
31
- - **Confidence Bounding Boxes**: Visual color cues (Green β‰₯80%, Yellow 50–80%, Red <50%).
32
- - **Confidence Heatmap**: Continuous thermal density overlay showing spatial OCR certainty.
33
- - **Reading Order Vector Flow**: Numbered direction arrows indicating extracted line-by-line reading flow.
34
- - **Interactive Visual Keyword Search**: Search any keyword to immediately highlight its spatial bounding box in gold.
35
-
36
- ### 4. πŸ—‚οΈ Structured Entity & Invoice Extraction (`src/extractor.py`)
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
37
  - **Automated Document Classification**: Categorizes document as *Invoice / Receipt*, *Identity Document*, *Medical / Prescription*, *Legal / Contract*, or *Academic*.
38
- - **Regex & Context Parsing**:
39
- - Dates (multiple international formats)
40
- - Monetary amounts & currencies (`$`, `β‚Ή`, `€`, `Β£`)
41
- - Contact information (emails, phones, URLs)
42
- - Invoice & purchase order numbers, subtotals, tax, discounts, and due dates
43
  - **PII Detection & Redaction**: Detects sensitive credentials (Aadhaar, PAN, Credit Cards, SSN) with optional 1-click `β–ˆβ–ˆβ–ˆβ–ˆ` redaction.
44
 
45
- ### 5. 🧬 Semantic Embedding Vector Space (`src/embedder.py`)
46
- - Embeds extracted words or sentences using `sentence-transformers/all-MiniLM-L6-v2`.
47
- - Projects 384-dimensional dense vectors down to 2D via **PCA**, **t-SNE**, or **UMAP**.
48
- - Clusters related concepts using **K-Means** with interactive Plotly scatter plots.
49
- - Computes pairwise **Cosine Similarity Heatmaps** to discover semantic connections in document content.
50
-
51
- ### 6. πŸ“€ Multi-Format Export Engine (`src/exporter.py`)
52
- - **Searchable PDF**: Embedded invisible text layer enabling native `Ctrl+F` search across standard PDF viewers.
53
- - **Annotated Audit PDF**: Complete executive report with bounding box visual overview, confidence metrics, and entity tables.
54
- - **Structured JSON**: Full word coordinates, confidence scores, and detected metadata for downstream NLP pipelines.
55
- - **CSV / Excel**: Tabular word list with positions and confidence.
56
  - **Plain Text**: Formatted clean text with optional PII redaction.
57
- - **All-In-One ZIP Archive**: 1-click download of all formats bundled together.
58
 
59
- ### 7. ✨ Preset Samples & Camera Scanner
60
- - **Preset Test Documents**: Built-in Commercial Invoice and Retail Receipt generators (`src/sample_docs.py`) for instant testing without needing local image files.
61
- - **Live Camera Scanner**: Snap physical documents in real-time via camera/webcam.
 
62
 
63
  ---
64
 
@@ -66,27 +100,23 @@
66
 
67
  ```
68
  Text_Extracted/
69
- β”œβ”€β”€ app.py # Streamlit UI, visual canvas, search, embeddings, and tabs
70
- β”œβ”€β”€ requirements.txt # Project dependencies
71
- β”œβ”€β”€ README.md # Project documentation
72
  └── src/
73
  β”œβ”€β”€ __init__.py # Package initialization
74
- β”œβ”€β”€ preprocessor.py # OpenCV deskew, contrast CLAHE, binarization, perspective warp
75
- β”œβ”€β”€ ocr_engine.py # EasyOCR GPU inference engine and result normalizer
76
- β”œβ”€β”€ visualizer.py # Bounding box overlays, heatmaps, search highlighter
77
- β”œβ”€β”€ extractor.py # Entity extraction (dates, amounts, contacts, invoice fields, PII)
78
  β”œβ”€β”€ embedder.py # Sentence-transformers embedding & 2D dimensionality reduction
79
- β”œβ”€β”€ exporter.py # Searchable PDF, Annotated PDF, JSON, CSV, TXT, and ZIP bundle
80
  └── sample_docs.py # Synthetic document generator for 1-click demo testing
81
  ```
82
 
83
  ---
84
 
85
- ## βš™οΈ Installation & Setup
86
-
87
- ### Prerequisites
88
- - Python 3.10+
89
- - (Optional) NVIDIA GPU with CUDA for accelerated neural OCR inference
90
 
91
  ### 1. Clone the Repository
92
  ```bash
@@ -108,8 +138,8 @@ Open **http://localhost:8501** in your browser to begin.
108
  ---
109
 
110
  ## πŸ›‘οΈ Privacy & Compliance
111
- - **Local Execution**: All OCR inference, image preprocessing, and embedding calculations happen locally on your machine.
112
- - **PII Redaction**: Sensitive IDs and financial numbers can be stripped or masked before exporting.
113
 
114
  ---
115
 
 
3
  [![Python](https://img.shields.io/badge/Python-3.10%2B-blue.svg)](https://www.python.org/)
4
  [![Streamlit](https://img.shields.io/badge/Streamlit-1.40%2B-FF4B4B.svg)](https://streamlit.io/)
5
  [![EasyOCR](https://img.shields.io/badge/EasyOCR-PyTorch%20GPU-green.svg)](https://github.com/JaidedAI/EasyOCR)
6
+ [![Theme](https://img.shields.io/badge/Theme-Light%20Modern-38bdf8.svg)](https://streamlit.io/)
7
  [![License](https://img.shields.io/badge/License-MIT-purple.svg)](LICENSE)
8
 
9
+ **DocIQ** is an Intelligent Document Processing (IDP) platform built with Streamlit, EasyOCR, OpenCV, Sentence-Transformers, and PyMuPDF. It transforms raw document images, live camera captures, receipts, invoices, and scans into clean, structured, searchable, and semantically indexed data.
10
+
11
+ Featuring a modern light-mode interface, specialized live-camera enhancement (shadow removal & unsharp mask blur correction), comprehensive OCR accuracy measurement, and one-click multi-format export.
12
 
13
  ---
14
 
15
+ ## 🎯 OCR Accuracy Measurement & Quality Metrics
16
+
17
+ DocIQ includes a built-in mathematical accuracy engine (`compute_accuracy_metrics` in `src/ocr_engine.py`) to quantify recognition fidelity before documents enter downstream workflows:
18
+
19
+ $$\text{Composite Accuracy Index} = (0.70 \times \text{Avg Confidence}) + (0.30 \times \text{Lexical Validity})$$
20
+
21
+ ### πŸ“Š Quality Grading Scale
22
+
23
+ | Grade | Accuracy Range | Quality Level | Recommended Action |
24
+ | :---: | :---: | :---: | :--- |
25
+ | **A+** | **90.0% – 100%** | Exceptional / Studio Scan | Production ready; automated ingestion |
26
+ | **A** | **80.0% – 89.9%** | High Accuracy | Production ready; standard documents |
27
+ | **B** | **68.0% – 79.9%** | Good / Readable | Acceptable; minor camera noise or tilt |
28
+ | **C** | **50.0% – 67.9%** | Moderate | Manual review recommended; poor lighting |
29
+ | **D** | **< 50.0%** | Low Accuracy | Retake capture with improved lighting |
30
+
31
+ ### πŸ“ˆ Evaluated Metrics & Indicators
32
+
33
+ - **Overall Accuracy Score (%)**: Composite index weighting deep neural confidence and linguistic token validity.
34
+ - **Confidence Distribution Breakdown**:
35
+ - 🟒 **High Confidence (β‰₯ 80%)**: Verified character accuracy.
36
+ - 🟑 **Medium Confidence (50% – 79%)**: Legible text with minor ambiguity.
37
+ - πŸ”΄ **Low Confidence (< 50%)**: Borderline characters or background artifacts.
38
+ - **Lexical Validity (%)**: Ratio of extracted tokens containing valid alphanumeric structures vs. non-linguistic noise.
39
+ - **Reading Order Flow**: Directional vector connectivity ensuring natural left-to-right, top-to-bottom line sequencing.
40
+
41
+ ---
42
+
43
+ ## πŸ“Έ Live Photo & Camera Optimization Pipeline
44
+
45
+ Camera photos from smartphones and webcams present real-world challenges that cause standard OCR engines to fail. DocIQ integrates an automated camera pipeline (`src/preprocessor.py`):
46
+
47
+ ```mermaid
48
+ graph TD
49
+ Raw[Raw Camera Photo] --> EXIF[EXIF Orientation Normalization]
50
+ EXIF --> Rescale[Bicubic Resolution Rescaling]
51
+ Rescale --> Shadow[Background Shadow & Glare Removal]
52
+ Shadow --> CLAHE[Adaptive CLAHE Contrast Boost]
53
+ CLAHE --> Sharpen[Unsharp Masking Blur Correction]
54
+ Sharpen --> Denoise[Bilateral Edge-Preserving Denoising]
55
+ Denoise --> MultiPass[Two-Pass High-Sensitivity Neural OCR]
56
+ MultiPass --> Output[High-Fidelity Text & Bounding Boxes]
57
+ ```
58
+
59
+ 1. **EXIF Orientation Correction**: Automatically transposes mobile photos that are rotated 90Β° or 270Β° by smartphone gyroscope metadata.
60
+ 2. **Shadow & Glare Removal**: Morphological background approximation divides out uneven ambient room lighting and cast shadows.
61
+ 3. **Unsharp Mask Sharpening**: Restores crisp stroke boundaries for slightly out-of-focus camera lenses.
62
+ 4. **Two-Pass Neural Recognition**: If initial detection returns low word volume, DocIQ automatically launches a secondary high-sensitivity recognition pass.
63
+
64
+ ---
65
+
66
+ ## πŸš€ Complete Feature Suite
67
+
68
+ ### 1. πŸ–ΌοΈ Visual Canvas & Spatial Inspection (`src/visualizer.py`)
69
+ - **Bounding Boxes**: Confidence-coded overlays (Green β‰₯80%, Yellow 50–79%, Red <50%).
70
+ - **Confidence Heatmap**: Thermal spatial density map displaying model certainty across page regions.
71
+ - **Reading Order**: Numbered vectors tracing the natural human reading sequence.
72
+ - **Interactive Visual Search**: Search any keyword to immediately illuminate its exact bounding box in gold.
73
+
74
+ ### 2. πŸ—‚οΈ Structured Entity & Invoice Extraction (`src/extractor.py`)
75
  - **Automated Document Classification**: Categorizes document as *Invoice / Receipt*, *Identity Document*, *Medical / Prescription*, *Legal / Contract*, or *Academic*.
76
+ - **Key-Value Parsing**: Invoice numbers, PO numbers, dates, monetary totals, subtotals, tax, discounts, emails, phones, and URLs.
 
 
 
 
77
  - **PII Detection & Redaction**: Detects sensitive credentials (Aadhaar, PAN, Credit Cards, SSN) with optional 1-click `β–ˆβ–ˆβ–ˆβ–ˆ` redaction.
78
 
79
+ ### 3. 🧬 Semantic Embedding Vector Space (`src/embedder.py`)
80
+ - Transforms recognized text into 384-dimensional dense vectors using `sentence-transformers/all-MiniLM-L6-v2`.
81
+ - Projects embeddings to 2D via **PCA** or **t-SNE** with interactive **K-Means** clustering.
82
+ - Pairwise **Cosine Similarity Heatmaps** revealing conceptual relationships across document segments.
83
+
84
+ ### 4. πŸ“€ Multi-Format Export Center (`src/exporter.py`)
85
+ - **Searchable PDF**: Image combined with an invisible selectable text layer for native `Ctrl+F` search in Acrobat, Chrome, and Preview.
86
+ - **Annotated Audit PDF**: Comprehensive executive report with bounding box visual map, accuracy metrics, and word-by-word audit trail.
87
+ - **Structured JSON**: Machine-readable coordinates, confidence scores, and parsed entities.
88
+ - **CSV / Excel**: Tabular spreadsheet of words, coordinates, and confidence ratings.
 
89
  - **Plain Text**: Formatted clean text with optional PII redaction.
90
+ - **All-In-One ZIP Archive**: Single-click package containing all export formats.
91
 
92
+ ### 5. ✨ Multi-Input Sources
93
+ - **Upload Image**: Drag-and-drop support for JPG, PNG, and WEBP.
94
+ - **Preset Sample Documents**: Built-in Commercial Invoice and Retail Receipt generators (`src/sample_docs.py`) for instant testing without local files.
95
+ - **Camera Scanner**: Live webcam/mobile camera capture.
96
 
97
  ---
98
 
 
100
 
101
  ```
102
  Text_Extracted/
103
+ β”œβ”€β”€ app.py # Modern Light-Mode UI, accuracy gauge, canvas tabs, export center
104
+ β”œβ”€β”€ requirements.txt # Dependencies (Streamlit, EasyOCR, PyTorch, OpenCV, ReportLab)
105
+ β”œβ”€β”€ README.md # Documentation, architecture, and accuracy evaluation guide
106
  └── src/
107
  β”œβ”€β”€ __init__.py # Package initialization
108
+ β”œβ”€β”€ preprocessor.py # EXIF fix, shadow removal, unsharp mask, deskew, CLAHE
109
+ β”œβ”€β”€ ocr_engine.py # EasyOCR GPU engine, two-pass recognition, accuracy metrics
110
+ β”œβ”€β”€ visualizer.py # Bounding boxes, confidence heatmap, reading order flow
111
+ β”œβ”€β”€ extractor.py # Key-value invoice fields, dates, amounts, PII detection
112
  β”œβ”€β”€ embedder.py # Sentence-transformers embedding & 2D dimensionality reduction
113
+ β”œβ”€β”€ exporter.py # Searchable PDF, Audit PDF, JSON, CSV, TXT, and ZIP bundle
114
  └── sample_docs.py # Synthetic document generator for 1-click demo testing
115
  ```
116
 
117
  ---
118
 
119
+ ## βš™οΈ Installation & Quick Start
 
 
 
 
120
 
121
  ### 1. Clone the Repository
122
  ```bash
 
138
  ---
139
 
140
  ## πŸ›‘οΈ Privacy & Compliance
141
+ - **100% Local Inference**: All OCR processing, image filtering, and vector embeddings execute entirely on your local machine.
142
+ - **PII Redaction**: Sensitive personal credentials can be automatically masked (`β–ˆβ–ˆβ–ˆβ–ˆ`) before exporting.
143
 
144
  ---
145
 
app.py CHANGED
@@ -1,16 +1,7 @@
1
  """
2
- app.py β€” Advanced Intelligent Document Processing (IDP) Platform
3
- ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
4
- Features:
5
- 1. OpenCV auto-deskew / contrast / denoising preprocessing
6
- 2. GPU-accelerated EasyOCR with bounding box visualization
7
- 3. Confidence heatmap & reading-order visualization
8
- 4. Interactive keyword search with spatial highlighting
9
- 5. PII detection & one-click redaction
10
- 6. Structured entity extraction (dates, amounts, invoice fields)
11
- 7. Embedding vector space visualization (PCA / t-SNE)
12
- 8. Semantic similarity heatmap
13
- 9. Multi-format export: Searchable PDF, Annotated PDF, JSON, CSV, TXT
14
  """
15
 
16
  import streamlit as st
@@ -26,8 +17,25 @@ import sys
26
  # Make src importable
27
  sys.path.insert(0, os.path.dirname(__file__))
28
 
 
 
 
 
 
 
 
 
 
 
29
  from src.preprocessor import full_preprocess
30
- from src.ocr_engine import run_ocr, get_full_text, get_statistics
 
 
 
 
 
 
 
31
  from src.visualizer import draw_bounding_boxes, search_and_highlight, create_confidence_heatmap, draw_reading_order
32
  from src.extractor import extract_all, redact_pii
33
  from src.embedder import build_embedding_data
@@ -46,225 +54,287 @@ from src.sample_docs import SAMPLE_DOCUMENTS
46
  # Page Config
47
  # ──────────────────────────────────────────────
48
  st.set_page_config(
49
- page_title="DocIQ β€” Advanced Intelligent Document Processing",
50
  page_icon="🧠",
51
  layout="wide",
52
  initial_sidebar_state="expanded",
53
  )
54
 
55
  # ──────────────────────────────────────────────
56
- # Custom CSS: Dark Premium Theme
57
  # ──────────────────────────────────────────────
58
  st.markdown("""
59
  <style>
60
- @import url('https://fonts.googleapis.com/css2?family=Inter:wght@300;400;500;600;700;800&display=swap');
61
 
62
- *, *::before, *::after { font-family: 'Inter', sans-serif !important; box-sizing: border-box; }
 
 
 
63
 
64
- /* Dark Background */
65
- .stApp { background: linear-gradient(135deg, #0a0e1a 0%, #0f172a 50%, #0d1117 100%) !important; }
 
 
 
66
  .stApp > header { background: transparent !important; }
67
 
68
- /* Sidebar */
69
  section[data-testid="stSidebar"] {
70
- background: rgba(15,23,42,0.95) !important;
71
- border-right: 1px solid rgba(99,102,241,0.25) !important;
 
 
 
 
72
  }
73
 
74
- /* Cards */
75
  .doc-card {
76
- background: linear-gradient(145deg, rgba(30,41,59,0.95), rgba(15,23,42,0.98));
77
- border: 1px solid rgba(99,102,241,0.3);
78
  border-radius: 16px;
79
- padding: 1.4rem 1.6rem;
80
  margin-bottom: 1rem;
81
- backdrop-filter: blur(20px);
82
- transition: border-color 0.3s ease, transform 0.2s ease;
83
  }
84
  .doc-card:hover {
85
- border-color: rgba(99,102,241,0.6);
86
- transform: translateY(-2px);
87
  }
88
 
89
- /* Hero header */
90
  .hero-header {
91
- background: linear-gradient(135deg, #6366f1 0%, #8b5cf6 40%, #06b6d4 100%);
92
  -webkit-background-clip: text;
93
  -webkit-text-fill-color: transparent;
94
  background-clip: text;
95
- font-size: 2.8rem;
96
  font-weight: 800;
97
- letter-spacing: -1px;
98
- line-height: 1.1;
99
  }
100
 
101
  .hero-sub {
102
- color: #94a3b8;
103
  font-size: 1.05rem;
104
  font-weight: 400;
105
- margin-top: 0.4rem;
106
- margin-bottom: 2rem;
107
  }
108
 
109
- /* Badge */
110
  .badge {
111
  display: inline-block;
112
- padding: 0.25rem 0.75rem;
113
  border-radius: 999px;
114
- font-size: 0.72rem;
115
- font-weight: 600;
116
- letter-spacing: 0.05em;
117
  text-transform: uppercase;
118
  }
119
- .badge-green { background: rgba(34,197,94,0.15); color: #22c55e; border: 1px solid rgba(34,197,94,0.3); }
120
- .badge-yellow { background: rgba(234,179,8,0.15); color: #eab308; border: 1px solid rgba(234,179,8,0.3); }
121
- .badge-red { background: rgba(239,68,68,0.15); color: #ef4444; border: 1px solid rgba(239,68,68,0.3); }
122
- .badge-indigo { background: rgba(99,102,241,0.15); color: #818cf8; border: 1px solid rgba(99,102,241,0.3); }
123
- .badge-cyan { background: rgba(6,182,212,0.15); color: #22d3ee; border: 1px solid rgba(6,182,212,0.3); }
124
 
125
- /* Metric cards */
126
  .metric-card {
127
- background: rgba(30,41,59,0.7);
128
- border: 1px solid rgba(99,102,241,0.2);
129
- border-radius: 12px;
130
- padding: 1.2rem;
131
  text-align: center;
132
- transition: all 0.25s ease;
 
133
  }
134
- .metric-card:hover { border-color: rgba(99,102,241,0.5); background: rgba(30,41,59,0.95); }
135
- .metric-value { font-size: 2rem; font-weight: 700; color: #818cf8; line-height: 1; }
136
- .metric-label { font-size: 0.78rem; color: #64748b; margin-top: 0.3rem; text-transform: uppercase; letter-spacing: 0.05em; }
137
-
138
- /* Entity chip */
139
- .entity-chip {
140
- display: inline-block;
141
- margin: 2px 3px;
142
- padding: 3px 10px;
143
- border-radius: 6px;
144
- font-size: 0.82rem;
145
- font-weight: 500;
146
  }
 
 
147
 
148
- /* Text areas */
149
- textarea { background: rgba(15,23,42,0.9) !important; color: #e2e8f0 !important; border: 1px solid rgba(99,102,241,0.3) !important; border-radius: 10px !important; }
 
 
 
 
 
 
 
150
 
151
  /* Tabs */
152
- .stTabs [role="tablist"] { gap: 4px; background: rgba(15,23,42,0.8); border-radius: 12px; padding: 4px; border: 1px solid rgba(99,102,241,0.2); }
153
- .stTabs [role="tab"] { border-radius: 8px !important; color: #94a3b8 !important; font-weight: 500 !important; padding: 6px 16px !important; }
154
- .stTabs [aria-selected="true"] { background: linear-gradient(135deg, #6366f1, #8b5cf6) !important; color: white !important; }
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
155
 
156
- /* Buttons */
157
  .stButton > button {
158
- background: linear-gradient(135deg, #6366f1, #8b5cf6) !important;
159
  border: none !important;
160
  color: white !important;
161
  font-weight: 600 !important;
162
  border-radius: 10px !important;
163
- padding: 0.5rem 1.5rem !important;
164
  transition: all 0.2s ease !important;
165
- box-shadow: 0 4px 15px rgba(99,102,241,0.3) !important;
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
166
  }
167
- .stButton > button:hover { transform: translateY(-2px) !important; box-shadow: 0 6px 20px rgba(99,102,241,0.5) !important; }
168
 
169
- /* File uploader */
170
  [data-testid="stFileUploader"] {
171
- background: rgba(15,23,42,0.8) !important;
172
- border: 2px dashed rgba(99,102,241,0.4) !important;
173
  border-radius: 16px !important;
 
174
  }
175
 
176
  /* Scrollbar */
177
- ::-webkit-scrollbar { width: 6px; height: 6px; }
178
- ::-webkit-scrollbar-track { background: transparent; }
179
- ::-webkit-scrollbar-thumb { background: rgba(99,102,241,0.4); border-radius: 3px; }
180
-
181
- /* Divider */
182
- hr { border-color: rgba(99,102,241,0.2) !important; }
183
-
184
- /* selectbox / slider labels */
185
- label { color: #94a3b8 !important; }
186
  </style>
187
  """, unsafe_allow_html=True)
188
 
189
 
190
  # ──────────────────────────────────────────────
191
- # Sidebar
192
  # ──────────────────────────────────────────────
193
  with st.sidebar:
194
  st.markdown("""
195
- <div style="text-align:center; padding: 1rem 0 0.5rem;">
196
- <div style="font-size:2.5rem;">🧠</div>
197
- <div style="font-size:1.2rem; font-weight:700; background:linear-gradient(135deg,#6366f1,#06b6d4);
198
- -webkit-background-clip:text; -webkit-text-fill-color:transparent; background-clip:text;">DocIQ</div>
199
- <div style="font-size:0.7rem; color:#475569; margin-top:2px;">Intelligent Document Processing</div>
200
  </div>
201
- <hr style="margin: 0.8rem 0;">
202
  """, unsafe_allow_html=True)
203
 
204
- st.markdown("### βš™οΈ OCR Settings")
205
  languages = st.multiselect(
206
  "Languages",
207
  ["en", "hi", "fr", "de", "es", "zh_sim", "ar"],
208
  default=["en", "hi"],
209
- help="Select languages present in your document"
210
  )
211
 
212
  confidence_threshold = st.slider(
213
- "Confidence Threshold",
214
- min_value=0.1, max_value=0.99, value=0.3, step=0.05,
215
- help="Discard text detected below this confidence"
216
  )
217
 
218
- st.markdown("### πŸ–ΌοΈ Preprocessing")
219
  preprocess_mode = st.selectbox(
220
- "Mode",
221
- ["auto", "raw", "binarize", "warp"],
 
 
 
 
 
 
 
 
 
 
 
 
 
222
  index=0,
223
- help="auto=deskew+CLAHE+denoise | binarize=B&W | warp=perspective flatten"
224
  )
225
 
226
- st.markdown("### πŸ”’ Embedding Settings")
227
- embed_unit = st.selectbox("Text Unit", ["word", "sentence", "line"], index=0)
228
- embed_reduction = st.selectbox("Reduction Method", ["PCA", "t-SNE", "UMAP"], index=0)
229
- embed_clusters = st.slider("K-Means Clusters", 2, 10, 5)
 
230
 
231
- st.markdown("### πŸ›‘οΈ Privacy")
232
- auto_redact = st.toggle("Auto-detect & Redact PII", value=False)
 
 
 
 
 
233
 
234
- st.markdown("<hr>", unsafe_allow_html=True)
235
  st.markdown("""
236
- <div style="font-size:0.7rem; color:#334155; text-align:center;">
237
- GPU Accelerated Β· EasyOCR 1.7.2<br>
238
- sentence-transformers Β· PyMuPDF
239
  </div>
240
  """, unsafe_allow_html=True)
241
 
242
 
243
  # ──────────────────────────────────────────────
244
- # Main Content
245
  # ──────────────────────────────────────────────
246
  st.markdown("""
247
  <div class="hero-header">DocIQ β€” Intelligent Document Processing</div>
248
  <div class="hero-sub">
249
- Upload any document image β†’ Auto-enhance β†’ OCR β†’ Visualize Embeddings β†’ Extract Entities β†’ Export
250
  </div>
251
  """, unsafe_allow_html=True)
252
 
253
- # Input Mode Tabs
 
 
 
254
  input_tab1, input_tab2, input_tab3 = st.tabs([
255
- "πŸ“ Upload File",
256
  "✨ Preset Sample Documents",
257
- "πŸ“Έ Camera Scanner",
258
  ])
259
 
260
  pil_img = None
261
  doc_name = "document.png"
 
262
 
263
  with input_tab1:
264
  uploaded_file = st.file_uploader(
265
- "Drop your document image here (JPG, PNG, WEBP)",
266
  type=["jpg", "jpeg", "png", "webp"],
267
- help="Supports scanned documents, invoices, receipts, ID cards, forms, and business records.",
268
  key="file_uploader_input"
269
  )
270
  if uploaded_file is not None:
@@ -273,19 +343,19 @@ with input_tab1:
273
 
274
  with input_tab2:
275
  st.markdown("""
276
- <div style="color:#94a3b8; font-size:0.88rem; margin-bottom: 0.8rem;">
277
- No file handy? Pick a pre-configured synthetic sample to test the entire OCR, entity extraction, and embedding pipeline immediately:
278
  </div>
279
  """, unsafe_allow_html=True)
280
 
281
- sc1, sc2 = st.columns([1, 1])
282
  for idx, (s_name, s_fn) in enumerate(SAMPLE_DOCUMENTS.items()):
283
  target_col = sc1 if idx % 2 == 0 else sc2
284
  with target_col:
285
  st.markdown(f"""
286
- <div class="doc-card" style="padding:1rem;">
287
- <div style="font-weight:700; color:#818cf8; font-size:1rem; margin-bottom:4px;">πŸ“„ {s_name}</div>
288
- <div style="color:#64748b; font-size:0.78rem; margin-bottom:10px;">Includes tabular line items, totals, dates, and contact info.</div>
289
  </div>
290
  """, unsafe_allow_html=True)
291
  if st.button(f"⚑ Load {s_name}", key=f"btn_sample_{idx}", use_container_width=True):
@@ -295,111 +365,198 @@ with input_tab2:
295
  if active_s and active_s in SAMPLE_DOCUMENTS and pil_img is None:
296
  pil_img = SAMPLE_DOCUMENTS[active_s]()
297
  doc_name = f"{active_s.lower().replace(' ', '_')}.png"
298
- st.success(f"βœ“ Loaded sample: **{active_s}**")
299
 
300
  with input_tab3:
301
- cam_file = st.camera_input("Take a photo of a document", key="camera_scanner_input")
302
- if cam_file is not None and pil_img is None:
 
 
 
 
 
 
303
  pil_img = Image.open(cam_file).convert("RGB")
304
  doc_name = "camera_capture.png"
 
 
305
 
 
 
 
306
  if pil_img is None:
307
- # Landing feature grid
308
  st.markdown("<br>", unsafe_allow_html=True)
309
  cols = st.columns(4)
310
  features = [
311
- ("πŸ”", "Visual OCR", "Bounding boxes with confidence heatmap & reading-order"),
312
- ("🧬", "Embedding Space", "See your words as vector clusters in 2D space"),
313
- ("πŸ›‘οΈ", "PII Redaction", "Auto-detect Aadhaar, PAN, credit cards & redact"),
314
- ("πŸ“Š", "Entity Extraction", "Dates, amounts, invoice fields, emails auto-parsed"),
315
- ("πŸ“„", "Searchable PDF", "Export with invisible text layer for native search"),
316
- ("πŸ”Ž", "Visual Search", "Search a keyword and light up its bounding box"),
317
- ("🌑️", "Confidence Map", "Heatmap overlay showing OCR certainty spatially"),
318
- ("⚑", "GPU Accelerated", "Torch CUDA 12.8 inference, model cached per session"),
319
  ]
320
  for i, (icon, title, desc) in enumerate(features):
321
  with cols[i % 4]:
322
  st.markdown(f"""
323
- <div class="doc-card" style="min-height:120px;">
324
- <div style="font-size:1.8rem; margin-bottom:6px;">{icon}</div>
325
- <div style="font-weight:600; color:#e2e8f0; font-size:0.9rem;">{title}</div>
326
- <div style="color:#64748b; font-size:0.75rem; margin-top:4px; line-height:1.4;">{desc}</div>
327
  </div>
328
  """, unsafe_allow_html=True)
329
  st.stop()
330
 
331
- with st.spinner("🎨 Preprocessing image..."):
332
- processed_img, preprocess_meta = full_preprocess(pil_img, mode=preprocess_mode)
333
 
334
  # ──────────────────────────────────────────────
335
- # OCR Inference
 
 
 
 
 
 
 
 
 
336
  # ──────────────────────────────────────────────
337
- with st.spinner("🧠 Running OCR (GPU accelerated)..."):
338
- ocr_results = run_ocr(processed_img, languages=languages, confidence_threshold=confidence_threshold)
 
 
 
 
 
339
 
340
  if not ocr_results:
341
- st.warning("⚠️ No text detected. Try lowering the confidence threshold or switching to 'binarize' mode.")
342
  st.stop()
343
 
344
  full_text = get_full_text(ocr_results)
345
- stats = get_statistics(ocr_results)
346
 
347
  # PII Redaction
348
  redacted_text, pii_count = redact_pii(full_text) if auto_redact else (full_text, 0)
349
  display_text = redacted_text if auto_redact else full_text
350
 
351
  # Entity Extraction
352
- with st.spinner("πŸ” Extracting entities..."):
353
  entities = extract_all(full_text)
354
 
 
355
  # ──────────────────────────────────────────────
356
- # Metrics Row
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
357
  # ──────────────────────────────────────────────
358
- st.markdown("<br>", unsafe_allow_html=True)
359
  m1, m2, m3, m4, m5 = st.columns(5)
360
  metrics = [
361
- (m1, "Words", stats["total_words"], "badge-indigo"),
362
- (m2, "Avg Confidence", f"{stats['avg_confidence']:.1%}", "badge-green"),
363
- (m3, "High Conf β‰₯80%", stats["high_conf_count"], "badge-green"),
364
- (m4, "PII Detected", pii_count if auto_redact else "β€”", "badge-red" if pii_count > 0 else "badge-indigo"),
365
- (m5, "Doc Type", entities.get("document_type", "Unknown")[:18], "badge-cyan"),
366
  ]
367
- for col, label, value, badge_class in metrics:
368
  with col:
369
  st.markdown(f"""
370
  <div class="metric-card">
371
- <div class="metric-value" style="font-size:1.4rem;">{value}</div>
372
  <div class="metric-label">{label}</div>
373
  </div>
374
  """, unsafe_allow_html=True)
375
 
376
  st.markdown("<br>", unsafe_allow_html=True)
377
 
 
378
  # ──────────────────────────────────────────────
379
- # Tabs
380
  # ──────────────────────────────────────────────
381
  tab_vis, tab_search, tab_text, tab_entities, tab_embed, tab_export = st.tabs([
382
  "πŸ–ΌοΈ Visual Canvas",
383
- "πŸ”Ž Visual Search",
384
- "πŸ“ Extracted Text",
385
  "πŸ—‚οΈ Entities & PII",
386
  "🧬 Embedding Space",
387
- "πŸ“€ Export",
388
  ])
389
 
 
390
  # ────────────────
391
  # TAB 1: Visual Canvas
392
  # ────────────────
393
  with tab_vis:
394
  c1, c2 = st.columns([3, 1])
395
  with c2:
 
396
  view_mode = st.radio(
397
  "Canvas View",
398
- ["Bounding Boxes", "Confidence Heatmap", "Reading Order", "Original"],
399
- index=0
 
400
  )
401
- show_labels = st.toggle("Word Labels", value=True)
402
- show_conf = st.toggle("Confidence %", value=True)
 
 
 
 
 
 
 
 
 
 
403
 
404
  with c1:
405
  if view_mode == "Bounding Boxes":
@@ -408,40 +565,40 @@ with tab_vis:
408
  show_labels=show_labels,
409
  show_confidence=show_conf,
410
  )
411
- st.image(annotated, use_container_width=True, caption="Bounding Boxes (Greenβ‰₯80%, Yellow 50-80%, Red<50%)")
412
  elif view_mode == "Confidence Heatmap":
413
  heatmap_img = create_confidence_heatmap(processed_img, ocr_results)
414
- st.image(heatmap_img, use_container_width=True, caption="Confidence Heatmap (Blue=High, Red=Low)")
415
- elif view_mode == "Reading Order":
416
  order_img = draw_reading_order(processed_img, ocr_results)
417
- st.image(order_img, use_container_width=True, caption="Reading Order (numbered arrows)")
418
  else:
419
  col_o, col_p = st.columns(2)
420
  with col_o:
421
- st.image(pil_img, caption="Original", use_container_width=True)
422
  with col_p:
423
- st.image(processed_img, caption=f"Preprocessed ({preprocess_mode})", use_container_width=True)
424
 
425
- # Preprocessing steps info
426
  st.markdown(f"""
427
- <div class="doc-card" style="margin-top:0.5rem;">
428
- <span style="color:#64748b; font-size:0.8rem;">Preprocessing applied: </span>
429
- {''.join(f'<span class="badge badge-cyan" style="margin-left:4px;">{s}</span>' for s in preprocess_meta.get('steps', []))}
430
  </div>
431
  """, unsafe_allow_html=True)
432
 
433
 
434
  # ────────────────
435
- # TAB 2: Visual Search
436
  # ────────────────
437
  with tab_search:
438
  search_col, opts_col = st.columns([3, 1])
439
  with opts_col:
440
- case_sensitive = st.toggle("Case Sensitive", value=False)
441
  with search_col:
442
  keyword = st.text_input(
443
- "πŸ”Ž Search keyword in document",
444
- placeholder="Type a word or phrase...",
445
  label_visibility="collapsed"
446
  )
447
 
@@ -452,71 +609,71 @@ with tab_search:
452
 
453
  if matched_texts:
454
  st.markdown(f"""
455
- <div class="doc-card" style="background:rgba(34,197,94,0.08); border-color:rgba(34,197,94,0.4);">
456
- <span style="color:#22c55e; font-weight:600;">βœ“ Found {len(matched_texts)} match(es)</span>
457
- <br>
458
- {' '.join(f'<span class="entity-chip" style="background:rgba(234,179,8,0.15);color:#eab308;border:1px solid rgba(234,179,8,0.3);">{t}</span>' for t in matched_texts[:20])}
 
459
  </div>
460
  """, unsafe_allow_html=True)
461
  else:
462
  st.markdown(f"""
463
- <div class="doc-card" style="background:rgba(239,68,68,0.08); border-color:rgba(239,68,68,0.4);">
464
- <span style="color:#ef4444; font-weight:600;">βœ— No matches for "{keyword}"</span>
465
  </div>
466
  """, unsafe_allow_html=True)
467
 
468
- st.image(highlighted_img, use_container_width=True,
469
- caption=f"Search Results: '{keyword}' highlighted in gold")
470
  else:
471
- st.info("πŸ‘† Enter a keyword above to highlight its position on the document canvas.")
472
  st.image(processed_img, use_container_width=True, caption="Document Preview")
473
 
474
 
475
  # ────────────────
476
- # TAB 3: Extracted Text
477
  # ────────────────
478
  with tab_text:
479
  c1, c2 = st.columns([2, 1])
480
  with c1:
481
  st.markdown(f"""<div class="doc-card">
482
  <div style="display:flex; justify-content:space-between; align-items:center; margin-bottom:8px;">
483
- <span style="font-weight:600; color:#e2e8f0;">
484
- Extracted Text
485
  {'<span class="badge badge-red" style="margin-left:8px;">PII REDACTED</span>' if auto_redact and pii_count > 0 else ''}
486
  </span>
487
- <span style="font-size:0.75rem; color:#64748b;">
488
- {len(display_text.split())} words Β· {len(display_text)} chars
489
  </span>
490
  </div>
491
  </div>""", unsafe_allow_html=True)
492
  st.text_area(
493
  "Extracted Text Content",
494
  value=display_text,
495
- height=380,
496
  label_visibility="collapsed",
497
- help="Select all and Ctrl+C to copy, or edit text directly."
498
  )
499
 
500
  with c2:
501
- st.markdown("#### πŸ“Š Word Confidence Table")
502
  df = pd.DataFrame([{
503
  "Text": r["text"],
504
  "Confidence": f"{r['confidence']:.1%}",
505
  "Width": r["width"],
506
  "Height": r["height"],
507
  } for r in sorted(ocr_results, key=lambda x: x["confidence"], reverse=True)])
508
- st.dataframe(df, use_container_width=True, height=420)
509
 
510
 
511
  # ────────────────
512
  # TAB 4: Entities & PII
513
  # ────────────────
514
  with tab_entities:
515
- doc_type = entities.get("document_type", "Unknown")
516
  st.markdown(f"""
517
- <div class="doc-card" style="background:linear-gradient(135deg,rgba(99,102,241,0.12),rgba(139,92,246,0.08));">
518
- <span style="color:#94a3b8; font-size:0.8rem;">CLASSIFIED AS</span>
519
- <div style="font-size:1.5rem; font-weight:700; color:#818cf8; margin-top:4px;">πŸ“‹ {doc_type}</div>
520
  </div>
521
  """, unsafe_allow_html=True)
522
 
@@ -526,31 +683,31 @@ with tab_entities:
526
  # Dates
527
  dates = entities.get("dates", [])
528
  st.markdown(f"""<div class="doc-card">
529
- <div style="font-weight:600;color:#e2e8f0;margin-bottom:8px;">πŸ“… Dates ({len(dates)})</div>
530
- {''.join(f'<span class="entity-chip" style="background:rgba(6,182,212,0.12);color:#22d3ee;border:1px solid rgba(6,182,212,0.3);">{d}</span>' for d in dates) if dates else '<span style="color:#475569;font-size:0.82rem;">None detected</span>'}
531
  </div>""", unsafe_allow_html=True)
532
 
533
  # Amounts
534
  amounts = entities.get("amounts", [])
535
  st.markdown(f"""<div class="doc-card">
536
- <div style="font-weight:600;color:#e2e8f0;margin-bottom:8px;">πŸ’° Amounts ({len(amounts)})</div>
537
- {''.join(f'<span class="entity-chip" style="background:rgba(34,197,94,0.12);color:#22c55e;border:1px solid rgba(34,197,94,0.3);">{a}</span>' for a in amounts) if amounts else '<span style="color:#475569;font-size:0.82rem;">None detected</span>'}
538
  </div>""", unsafe_allow_html=True)
539
 
540
- # Contact
541
  emails = entities.get("emails", [])
542
  phones = entities.get("phones", [])
543
  urls = entities.get("urls", [])
544
  st.markdown(f"""<div class="doc-card">
545
- <div style="font-weight:600;color:#e2e8f0;margin-bottom:8px;">πŸ“¬ Contact Info</div>
546
- {''.join(f'<span class="entity-chip" style="background:rgba(99,102,241,0.12);color:#818cf8;border:1px solid rgba(99,102,241,0.3);">βœ‰ {e}</span>' for e in emails)}
547
- {''.join(f'<span class="entity-chip" style="background:rgba(139,92,246,0.12);color:#a78bfa;border:1px solid rgba(139,92,246,0.3);">πŸ“ž {p}</span>' for p in phones)}
548
- {''.join(f'<span class="entity-chip" style="background:rgba(6,182,212,0.12);color:#22d3ee;border:1px solid rgba(6,182,212,0.3);">πŸ”— {u[:30]}...</span>' for u in urls[:3])}
549
- {'<span style="color:#475569;font-size:0.82rem;">None detected</span>' if not emails and not phones and not urls else ''}
550
  </div>""", unsafe_allow_html=True)
551
 
552
  with col_b:
553
- # Invoice fields
554
  inv = entities.get("invoice_fields", {})
555
  FIELD_ICONS = {
556
  "invoice_number": "🧾",
@@ -569,62 +726,66 @@ with tab_entities:
569
  icon = FIELD_ICONS.get(k, "πŸ“Œ")
570
  title = k.replace('_', ' ').title()
571
  inv_rows.append(f"""
572
- <div style="display:flex; justify-content:space-between; align-items:center; padding:9px 12px; margin-bottom:6px; background:rgba(15,23,42,0.6); border:1px solid rgba(99,102,241,0.18); border-radius:10px;">
573
- <span style="color:#94a3b8; font-size:0.84rem; display:flex; align-items:center; gap:6px;">
574
  <span>{icon}</span>
575
- <strong style="color:#cbd5e1; font-weight:500;">{title}</strong>
576
  </span>
577
- <span style="color:#38bdf8; font-size:0.86rem; font-weight:600; font-family:monospace; background:rgba(56,189,248,0.1); padding:2px 8px; border-radius:6px; border:1px solid rgba(56,189,248,0.25);">
578
  {v}
579
  </span>
580
  </div>""")
581
  inv_html = "".join(inv_rows)
582
  st.markdown(f"""<div class="doc-card">
583
- <div style="font-weight:600; color:#e2e8f0; margin-bottom:12px; display:flex; align-items:center; justify-content:space-between;">
584
- <span>🧾 Key Document Fields</span>
585
- <span class="badge badge-cyan">{len(inv)} detected</span>
586
  </div>
587
  {inv_html}
588
  </div>""", unsafe_allow_html=True)
589
  else:
590
  st.markdown("""<div class="doc-card">
591
- <div style="font-weight:600; color:#e2e8f0; margin-bottom:6px;">🧾 Key Document Fields</div>
592
- <div style="color:#64748b; font-size:0.82rem;">No structured invoice or PO fields detected in this document.</div>
593
  </div>""", unsafe_allow_html=True)
594
 
595
- # PII
596
  pii = entities.get("pii", {})
597
  pii_found = {k: v for k, v in pii.items() if v}
598
  if pii_found:
599
  pii_html = "".join(f"""
600
- <div style="display:flex;justify-content:space-between;padding:6px 0;border-bottom:1px solid rgba(239,68,68,0.1);">
601
- <span style="color:#ef4444;font-size:0.82rem;">⚠ {k.upper()}</span>
602
- <span style="color:#fca5a5;font-size:0.82rem;">{len(v)} found</span>
603
  </div>""" for k, v in pii_found.items())
604
- st.markdown(f"""<div class="doc-card" style="border-color:rgba(239,68,68,0.3); background:rgba(239,68,68,0.05);">
605
- <div style="font-weight:600;color:#ef4444;margin-bottom:8px;">πŸ›‘οΈ PII Detected</div>
606
  {pii_html}
607
- <div style="margin-top:10px;font-size:0.75rem;color:#64748b;">Enable "Auto-redact PII" in sidebar to redact from exported text.</div>
 
 
608
  </div>""", unsafe_allow_html=True)
609
  else:
610
- st.markdown("""<div class="doc-card" style="border-color:rgba(34,197,94,0.3); background:rgba(34,197,94,0.05);">
611
- <div style="font-weight:600;color:#22c55e;margin-bottom:4px;">βœ… No PII Detected</div>
612
- <div style="font-size:0.8rem;color:#64748b;">No Aadhaar, PAN, credit card, or SSN patterns found.</div>
613
  </div>""", unsafe_allow_html=True)
614
 
615
 
616
  # ────────────────
617
- # TAB 5: Embedding Space
618
  # ────────────���───
619
  with tab_embed:
620
  st.markdown("""<div class="doc-card">
621
- <div style="font-weight:600;color:#e2e8f0;margin-bottom:4px;">🧬 Semantic Embedding Vector Space</div>
622
- <div style="color:#64748b;font-size:0.82rem;">Each point represents a text unit projected to 2D using dimensionality reduction.
623
- Semantically similar words/phrases cluster together. Colors = K-Means clusters.</div>
 
 
624
  </div>""", unsafe_allow_html=True)
625
 
626
- if st.button("⚑ Generate Embedding Visualization", type="primary"):
627
- with st.spinner("πŸ”’ Embedding text with sentence-transformers..."):
628
  embed_data = build_embedding_data(
629
  ocr_results,
630
  text_unit=embed_unit,
@@ -636,33 +797,13 @@ with tab_embed:
636
  coords = np.array(embed_data["coords_2d"])
637
  texts = embed_data["texts"]
638
  labels = embed_data["cluster_labels"]
639
- dim = embed_data["embedding_dim"]
640
-
641
- # Metric row
642
- mc1, mc2, mc3 = st.columns(3)
643
- for col, val, lab in [
644
- (mc1, embed_data["n_texts"], f"{embed_unit.title()}s Embedded"),
645
- (mc2, dim, "Embedding Dimensions"),
646
- (mc3, embed_data["reduction_method"], "Reduction Method"),
647
- ]:
648
- with col:
649
- st.markdown(f"""<div class="metric-card">
650
- <div class="metric-value">{val}</div>
651
- <div class="metric-label">{lab}</div>
652
- </div>""", unsafe_allow_html=True)
653
-
654
- st.markdown("<br>", unsafe_allow_html=True)
655
-
656
- # Scatter plot
657
  CLUSTER_COLORS = [
658
- "#6366f1", "#22d3ee", "#22c55e", "#eab308", "#f97316",
659
- "#ec4899", "#a78bfa", "#34d399", "#fb923c", "#38bdf8"
660
  ]
661
- color_list = [CLUSTER_COLORS[l % len(CLUSTER_COLORS)] for l in labels]
662
 
663
  fig = go.Figure()
664
-
665
- # Plot each cluster as separate trace for legend
666
  unique_labels = sorted(set(labels))
667
  for cluster_id in unique_labels:
668
  idx = [i for i, l in enumerate(labels) if l == cluster_id]
@@ -671,7 +812,6 @@ with tab_embed:
671
  y_vals = [coords[i, 1] for i in idx]
672
  color = CLUSTER_COLORS[cluster_id % len(CLUSTER_COLORS)]
673
 
674
- # Ellipse hull (convex hull approximation via scatter)
675
  fig.add_trace(go.Scatter(
676
  x=x_vals,
677
  y=y_vals,
@@ -679,132 +819,91 @@ with tab_embed:
679
  name=f"Cluster {cluster_id + 1}",
680
  marker=dict(
681
  color=color,
682
- size=9,
683
- opacity=0.85,
684
- line=dict(width=1.5, color="rgba(255,255,255,0.3)"),
685
  ),
686
- text=[t[:20] for t in cluster_texts],
687
  textposition="top center",
688
- textfont=dict(size=9, color="rgba(255,255,255,0.7)"),
689
- hovertemplate="<b>%{customdata}</b><br>x=%{x:.3f}, y=%{y:.3f}<extra></extra>",
690
  customdata=cluster_texts,
691
  ))
692
 
693
  fig.update_layout(
694
  title=dict(
695
- text=f"<b>Embedding Space</b> β€” {embed_unit.title()}s via {embed_reduction}",
696
- font=dict(size=16, color="#e2e8f0"),
697
  ),
698
- paper_bgcolor="rgba(15,23,42,0)",
699
- plot_bgcolor="rgba(15,23,42,0.7)",
700
- font=dict(color="#94a3b8", size=11),
701
  xaxis=dict(
702
- title=f"{embed_reduction} Dim 1",
703
- gridcolor="rgba(99,102,241,0.1)",
704
- zerolinecolor="rgba(99,102,241,0.2)",
705
  title_font_color="#64748b",
706
  ),
707
  yaxis=dict(
708
- title=f"{embed_reduction} Dim 2",
709
- gridcolor="rgba(99,102,241,0.1)",
710
- zerolinecolor="rgba(99,102,241,0.2)",
711
  title_font_color="#64748b",
712
  ),
713
  legend=dict(
714
- bgcolor="rgba(30,41,59,0.8)",
715
- bordercolor="rgba(99,102,241,0.3)",
716
  borderwidth=1,
717
- font=dict(color="#94a3b8"),
718
  ),
719
  height=520,
720
  margin=dict(l=10, r=10, t=50, b=10),
721
  )
722
  st.plotly_chart(fig, use_container_width=True)
723
 
724
- # Embedding raw values table
725
- with st.expander("πŸ”’ Raw Embedding Vectors (first 8 dimensions)"):
726
- emb_array = embed_data["embeddings"]
727
- n_show = min(20, len(texts))
728
- table_data = {
729
- "Text": texts[:n_show],
730
- "Cluster": [f"C{l+1}" for l in labels[:n_show]],
731
- }
732
- n_dims = min(8, emb_array.shape[1])
733
- for d in range(n_dims):
734
- table_data[f"dim_{d+1}"] = [f"{emb_array[i, d]:.4f}" for i in range(n_show)]
735
- st.dataframe(pd.DataFrame(table_data), use_container_width=True)
736
-
737
- # Cosine Similarity Heatmap
738
  sim_matrix = embed_data.get("similarity_matrix")
739
- if sim_matrix is not None and len(texts) <= 40:
740
  st.markdown("#### 🌑️ Semantic Similarity Heatmap")
741
- n = len(texts)
742
- short_labels = [t[:15] + ("…" if len(t) > 15 else "") for t in texts]
743
  fig_heat = px.imshow(
744
  sim_matrix,
745
  x=short_labels,
746
  y=short_labels,
747
- color_continuous_scale=[
748
- [0.0, "rgba(15,23,42,1)"],
749
- [0.3, "rgba(67,56,202,1)"],
750
- [0.6, "rgba(99,102,241,1)"],
751
- [0.8, "rgba(139,92,246,1)"],
752
- [1.0, "rgba(6,182,212,1)"],
753
- ],
754
  zmin=0, zmax=1,
755
  aspect="auto",
756
- title="<b>Cosine Similarity Between Text Units</b>",
757
  )
758
  fig_heat.update_layout(
759
- paper_bgcolor="rgba(15,23,42,0)",
760
- plot_bgcolor="rgba(15,23,42,0.7)",
761
- font=dict(color="#94a3b8", size=9),
762
- height=500,
763
  margin=dict(l=10, r=10, t=50, b=10),
764
  )
765
  st.plotly_chart(fig_heat, use_container_width=True)
766
-
767
- # Top similar pairs
768
- similar_pairs = embed_data.get("similar_pairs", [])
769
- if similar_pairs:
770
- st.markdown("#### πŸ”— Top Semantically Similar Pairs")
771
- pairs_df = pd.DataFrame([{
772
- "Text A": p["text_i"],
773
- "Text B": p["text_j"],
774
- "Similarity": f"{p['similarity']:.1%}",
775
- } for p in similar_pairs[:10]])
776
- st.dataframe(pairs_df, use_container_width=True)
777
-
778
  else:
779
- st.warning("⚠️ Not enough text to generate embeddings. Try processing an image with more text.")
780
  else:
781
- st.markdown("""<div style="text-align:center;padding:3rem;color:#475569;">
782
- <div style="font-size:3rem; margin-bottom:1rem;">🧬</div>
783
- <div style="font-size:1rem;color:#64748b;">Click the button above to generate the embedding space visualization.<br>
784
- This uses <code>sentence-transformers/all-MiniLM-L6-v2</code> to convert each text unit into a 384-dim vector,
785
- then reduces to 2D for visualization.</div>
786
- </div>""", unsafe_allow_html=True)
787
 
788
 
789
  # ────────────────
790
- # TAB 6: Export
791
  # ────────────────
792
  with tab_export:
793
  st.markdown("""<div class="doc-card">
794
- <div style="font-weight:600;color:#e2e8f0;margin-bottom:4px;">πŸ“€ Export Formats & Downloads</div>
795
- <div style="color:#64748b;font-size:0.82rem;">Download your processed document in multiple formats or grab the all-in-one ZIP archive.</div>
796
  </div>""", unsafe_allow_html=True)
797
 
798
  base_doc_stem = os.path.splitext(doc_name)[0]
799
 
800
- # One-click all formats bundle
801
- st.markdown("""
802
- <div style="margin-bottom:12px;">
803
- <span style="font-size:0.88rem; font-weight:600; color:#818cf8;">πŸ“¦ All-In-One Bundle</span>
804
- </div>
805
- """, unsafe_allow_html=True)
806
- if st.button("πŸ—œοΈ Prepare Complete Export Bundle (ZIP)", type="primary", use_container_width=True):
807
- with st.spinner("Bundling all formats (PDFs, JSON, CSV, TXT) into ZIP archive..."):
808
  zip_bytes = export_bundle_zip(processed_img, ocr_results, entities, display_text, base_name=base_doc_stem)
809
  st.download_button(
810
  "⬇️ Download All Formats (.ZIP Archive)",
@@ -814,19 +913,15 @@ with tab_export:
814
  use_container_width=True,
815
  )
816
 
817
- st.markdown("<hr style='margin:1.2rem 0;'>", unsafe_allow_html=True)
818
- st.markdown("""
819
- <div style="margin-bottom:12px;">
820
- <span style="font-size:0.88rem; font-weight:600; color:#cbd5e1;">πŸ“„ Individual Formats</span>
821
- </div>
822
- """, unsafe_allow_html=True)
823
 
824
  ec1, ec2 = st.columns(2)
825
 
826
  with ec1:
827
  # Searchable PDF
828
  if st.button("πŸ“„ Generate Searchable PDF", use_container_width=True):
829
- with st.spinner("Generating searchable PDF..."):
830
  pdf_bytes = export_searchable_pdf(processed_img, ocr_results, doc_name)
831
  st.download_button(
832
  "⬇️ Download Searchable PDF",
@@ -836,23 +931,23 @@ with tab_export:
836
  use_container_width=True,
837
  )
838
 
839
- # Annotated PDF
840
- if st.button("πŸ“Š Generate Annotated Report PDF", use_container_width=True):
841
- with st.spinner("Generating annotated PDF report..."):
842
  annotated_img = draw_bounding_boxes(processed_img, ocr_results, show_labels=True, show_confidence=True)
843
  pdf_bytes = export_annotated_pdf(annotated_img, ocr_results, entities, title="DocIQ OCR Report")
844
  st.download_button(
845
- "⬇️ Download Annotated PDF",
846
  data=pdf_bytes,
847
- file_name=f"{base_doc_stem}_report.pdf",
848
  mime="application/pdf",
849
  use_container_width=True,
850
  )
851
 
852
  # JSON
853
- json_bytes = export_json(ocr_results, entities, {"filename": doc_name})
854
  st.download_button(
855
- "⬇️ Download Structured JSON",
856
  data=json_bytes,
857
  file_name=f"{base_doc_stem}_data.json",
858
  mime="application/json",
@@ -863,7 +958,7 @@ with tab_export:
863
  # CSV
864
  csv_bytes = export_csv(ocr_results, entities)
865
  st.download_button(
866
- "⬇️ Download CSV / Excel",
867
  data=csv_bytes,
868
  file_name=f"{base_doc_stem}_words.csv",
869
  mime="text/csv",
@@ -873,14 +968,14 @@ with tab_export:
873
  # Plain Text
874
  txt_bytes = export_txt(ocr_results, full_text=display_text)
875
  st.download_button(
876
- f"⬇️ Download Plain Text {'(Redacted)' if auto_redact else ''}",
877
  data=txt_bytes,
878
  file_name=f"{base_doc_stem}_text.txt",
879
  mime="text/plain",
880
  use_container_width=True,
881
  )
882
 
883
- # Export entity summary
884
  st.download_button(
885
  "⬇️ Download Entity Summary (JSON)",
886
  data=export_json([], entities, {"filename": doc_name, "type": "entities_only"}),
@@ -890,12 +985,12 @@ with tab_export:
890
  )
891
 
892
  st.markdown("<br>", unsafe_allow_html=True)
893
- st.markdown("""<div class="doc-card" style="background:rgba(99,102,241,0.06);">
894
- <div style="font-weight:600;color:#818cf8;margin-bottom:8px;">πŸ“‹ Export Format Guide</div>
895
- <div style="display:grid;grid-template-columns:1fr 1fr;gap:8px;">
896
- <div><span style="color:#22d3ee;font-weight:500;">Searchable PDF</span><br><span style="color:#475569;font-size:0.78rem;">Image + invisible text layer. Ctrl+F search works natively in any PDF viewer.</span></div>
897
- <div><span style="color:#22d3ee;font-weight:500;">Annotated PDF</span><br><span style="color:#475569;font-size:0.78rem;">Bounding box image + entity table + confidence table. Great for audit trails.</span></div>
898
- <div><span style="color:#22d3ee;font-weight:500;">Structured JSON</span><br><span style="color:#475569;font-size:0.78rem;">Full word coordinates, confidence, entities. Ready for downstream NLP pipelines.</span></div>
899
- <div><span style="color:#22d3ee;font-weight:500;">CSV / Excel</span><br><span style="color:#475569;font-size:0.78rem;">Tabular word data + entity section. Open directly in Excel or Google Sheets.</span></div>
900
  </div>
901
  </div>""", unsafe_allow_html=True)
 
1
  """
2
+ app.py β€” DocIQ: Advanced Intelligent Document Processing (IDP) Platform
3
+ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
4
+ Light Theme Edition with Live Camera Optimization & Comprehensive OCR Accuracy Metrics
 
 
 
 
 
 
 
 
 
5
  """
6
 
7
  import streamlit as st
 
17
  # Make src importable
18
  sys.path.insert(0, os.path.dirname(__file__))
19
 
20
+ import importlib
21
+
22
+ # Ensure fresh imports across Streamlit hot reloads
23
+ for mod in ["src.preprocessor", "src.ocr_engine", "src.visualizer", "src.extractor", "src.embedder", "src.exporter", "src.sample_docs"]:
24
+ if mod in sys.modules:
25
+ try:
26
+ importlib.reload(sys.modules[mod])
27
+ except Exception:
28
+ pass
29
+
30
  from src.preprocessor import full_preprocess
31
+
32
+ try:
33
+ from src.ocr_engine import run_ocr, get_full_text, compute_accuracy_metrics
34
+ except ImportError:
35
+ if "src.ocr_engine" in sys.modules:
36
+ del sys.modules["src.ocr_engine"]
37
+ import src.ocr_engine
38
+ from src.ocr_engine import run_ocr, get_full_text, compute_accuracy_metrics
39
  from src.visualizer import draw_bounding_boxes, search_and_highlight, create_confidence_heatmap, draw_reading_order
40
  from src.extractor import extract_all, redact_pii
41
  from src.embedder import build_embedding_data
 
54
  # Page Config
55
  # ──────────────────────────────────────────────
56
  st.set_page_config(
57
+ page_title="DocIQ β€” Intelligent Document Processing",
58
  page_icon="🧠",
59
  layout="wide",
60
  initial_sidebar_state="expanded",
61
  )
62
 
63
  # ──────────────────────────────────────────────
64
+ # Custom CSS: Modern Clean Light Theme
65
  # ──────────────────────────────────────────────
66
  st.markdown("""
67
  <style>
68
+ @import url('https://fonts.googleapis.com/css2?family=Plus+Jakarta+Sans:wght@300;400;500;600;700;800&display=swap');
69
 
70
+ *, *::before, *::after {
71
+ font-family: 'Plus Jakarta Sans', sans-serif !important;
72
+ box-sizing: border-box;
73
+ }
74
 
75
+ /* Light Background */
76
+ .stApp {
77
+ background: linear-gradient(180deg, #f8fafc 0%, #f1f5f9 100%) !important;
78
+ color: #0f172a !important;
79
+ }
80
  .stApp > header { background: transparent !important; }
81
 
82
+ /* Sidebar Light Styling */
83
  section[data-testid="stSidebar"] {
84
+ background: #ffffff !important;
85
+ border-right: 1px solid #e2e8f0 !important;
86
+ box-shadow: 2px 0 12px rgba(0, 0, 0, 0.02) !important;
87
+ }
88
+ section[data-testid="stSidebar"] * {
89
+ color: #334155 !important;
90
  }
91
 
92
+ /* Light Cards */
93
  .doc-card {
94
+ background: #ffffff;
95
+ border: 1px solid #e2e8f0;
96
  border-radius: 16px;
97
+ padding: 1.3rem 1.5rem;
98
  margin-bottom: 1rem;
99
+ box-shadow: 0 4px 12px -2px rgba(15, 23, 42, 0.05), 0 2px 6px -1px rgba(15, 23, 42, 0.02);
100
+ transition: all 0.25s ease;
101
  }
102
  .doc-card:hover {
103
+ border-color: #cbd5e1;
104
+ box-shadow: 0 8px 20px -4px rgba(15, 23, 42, 0.08);
105
  }
106
 
107
+ /* Hero Header */
108
  .hero-header {
109
+ background: linear-gradient(135deg, #1e1b4b 0%, #312e81 40%, #4338ca 100%);
110
  -webkit-background-clip: text;
111
  -webkit-text-fill-color: transparent;
112
  background-clip: text;
113
+ font-size: 2.6rem;
114
  font-weight: 800;
115
+ letter-spacing: -0.8px;
116
+ line-height: 1.15;
117
  }
118
 
119
  .hero-sub {
120
+ color: #64748b;
121
  font-size: 1.05rem;
122
  font-weight: 400;
123
+ margin-top: 0.3rem;
124
+ margin-bottom: 1.5rem;
125
  }
126
 
127
+ /* Badges */
128
  .badge {
129
  display: inline-block;
130
+ padding: 0.3rem 0.75rem;
131
  border-radius: 999px;
132
+ font-size: 0.74rem;
133
+ font-weight: 700;
134
+ letter-spacing: 0.04em;
135
  text-transform: uppercase;
136
  }
137
+ .badge-green { background: #dcfce7; color: #15803d; border: 1px solid #bbf7d0; }
138
+ .badge-yellow { background: #fef9c3; color: #a16207; border: 1px solid #fef08a; }
139
+ .badge-red { background: #fee2e2; color: #b91c1c; border: 1px solid #fecaca; }
140
+ .badge-indigo { background: #e0e7ff; color: #4338ca; border: 1px solid #c7d2fe; }
141
+ .badge-cyan { background: #e0f2fe; color: #0369a1; border: 1px solid #bae6fd; }
142
 
143
+ /* Metric Cards */
144
  .metric-card {
145
+ background: #ffffff;
146
+ border: 1px solid #e2e8f0;
147
+ border-radius: 14px;
148
+ padding: 1.1rem 1rem;
149
  text-align: center;
150
+ box-shadow: 0 2px 8px rgba(15, 23, 42, 0.04);
151
+ transition: all 0.2s ease;
152
  }
153
+ .metric-card:hover {
154
+ transform: translateY(-2px);
155
+ box-shadow: 0 6px 16px rgba(15, 23, 42, 0.08);
156
+ border-color: #cbd5e1;
 
 
 
 
 
 
 
 
157
  }
158
+ .metric-value { font-size: 1.85rem; font-weight: 800; color: #4338ca; line-height: 1.1; }
159
+ .metric-label { font-size: 0.75rem; color: #64748b; margin-top: 0.3rem; text-transform: uppercase; font-weight: 600; letter-spacing: 0.05em; }
160
 
161
+ /* Accuracy Meter Box */
162
+ .accuracy-banner {
163
+ background: linear-gradient(135deg, #ffffff 0%, #f8fafc 100%);
164
+ border: 1px solid #cbd5e1;
165
+ border-radius: 16px;
166
+ padding: 1.3rem 1.6rem;
167
+ margin-bottom: 1.2rem;
168
+ box-shadow: 0 4px 14px rgba(15, 23, 42, 0.04);
169
+ }
170
 
171
  /* Tabs */
172
+ .stTabs [role="tablist"] {
173
+ gap: 6px;
174
+ background: #ffffff;
175
+ border-radius: 12px;
176
+ padding: 5px;
177
+ border: 1px solid #e2e8f0;
178
+ box-shadow: 0 2px 6px rgba(15, 23, 42, 0.03);
179
+ }
180
+ .stTabs [role="tab"] {
181
+ border-radius: 8px !important;
182
+ color: #64748b !important;
183
+ font-weight: 600 !important;
184
+ padding: 8px 18px !important;
185
+ border: none !important;
186
+ }
187
+ .stTabs [aria-selected="true"] {
188
+ background: #4f46e5 !important;
189
+ color: #ffffff !important;
190
+ box-shadow: 0 2px 8px rgba(79, 70, 229, 0.3) !important;
191
+ }
192
 
193
+ /* Primary Buttons */
194
  .stButton > button {
195
+ background: linear-gradient(135deg, #4f46e5, #6366f1) !important;
196
  border: none !important;
197
  color: white !important;
198
  font-weight: 600 !important;
199
  border-radius: 10px !important;
200
+ padding: 0.55rem 1.4rem !important;
201
  transition: all 0.2s ease !important;
202
+ box-shadow: 0 3px 10px rgba(79, 70, 229, 0.25) !important;
203
+ }
204
+ .stButton > button:hover {
205
+ transform: translateY(-1px) !important;
206
+ box-shadow: 0 6px 18px rgba(79, 70, 229, 0.35) !important;
207
+ }
208
+
209
+ /* Text Areas & Inputs */
210
+ textarea, input[type="text"] {
211
+ background: #ffffff !important;
212
+ color: #0f172a !important;
213
+ border: 1px solid #cbd5e1 !important;
214
+ border-radius: 10px !important;
215
+ }
216
+ textarea:focus, input[type="text"]:focus {
217
+ border-color: #6366f1 !important;
218
+ box-shadow: 0 0 0 3px rgba(99, 102, 241, 0.15) !important;
219
  }
 
220
 
221
+ /* File Uploader Light */
222
  [data-testid="stFileUploader"] {
223
+ background: #ffffff !important;
224
+ border: 2px dashed #cbd5e1 !important;
225
  border-radius: 16px !important;
226
+ padding: 1rem !important;
227
  }
228
 
229
  /* Scrollbar */
230
+ ::-webkit-scrollbar { width: 7px; height: 7px; }
231
+ ::-webkit-scrollbar-track { background: #f1f5f9; }
232
+ ::-webkit-scrollbar-thumb { background: #cbd5e1; border-radius: 4px; }
233
+ ::-webkit-scrollbar-thumb:hover { background: #94a3b8; }
 
 
 
 
 
234
  </style>
235
  """, unsafe_allow_html=True)
236
 
237
 
238
  # ──────────────────────────────────────────────
239
+ # Sidebar Controls
240
  # ──────────────────────────────────────────────
241
  with st.sidebar:
242
  st.markdown("""
243
+ <div style="text-align:center; padding: 0.8rem 0 0.4rem;">
244
+ <div style="font-size:2.4rem;">🧠</div>
245
+ <div style="font-size:1.3rem; font-weight:800; color:#1e1b4b; letter-spacing:-0.5px;">DocIQ</div>
246
+ <div style="font-size:0.75rem; color:#64748b; font-weight:500;">Intelligent Document Processing</div>
 
247
  </div>
248
+ <hr style="margin: 0.8rem 0; border-color:#e2e8f0;">
249
  """, unsafe_allow_html=True)
250
 
251
+ st.markdown("#### βš™οΈ OCR Settings")
252
  languages = st.multiselect(
253
  "Languages",
254
  ["en", "hi", "fr", "de", "es", "zh_sim", "ar"],
255
  default=["en", "hi"],
256
+ help="Select languages present in your document (en=English, hi=Hindi)"
257
  )
258
 
259
  confidence_threshold = st.slider(
260
+ "Confidence Filter",
261
+ min_value=0.10, max_value=0.95, value=0.25, step=0.05,
262
+ help="Discard text detected below this confidence score"
263
  )
264
 
265
+ st.markdown("#### πŸ–ΌοΈ Image Preprocessing")
266
  preprocess_mode = st.selectbox(
267
+ "Processing Mode",
268
+ [
269
+ "camera_enhance",
270
+ "auto",
271
+ "binarize",
272
+ "warp",
273
+ "raw"
274
+ ],
275
+ format_func=lambda x: {
276
+ "camera_enhance": "πŸ“Έ Camera & Mobile Capture (Shadows & Blur Fix)",
277
+ "auto": "⚑ Auto Enhancement (Deskew + CLAHE)",
278
+ "binarize": "βšͺ High Contrast Black & White",
279
+ "warp": "πŸ“ Perspective Flatten (4-Corner)",
280
+ "raw": "Original Unmodified Image",
281
+ }.get(x, x),
282
  index=0,
283
+ help="Use 'Camera & Mobile Capture' for live camera photos with shadows or tilt."
284
  )
285
 
286
+ camera_opt = st.checkbox(
287
+ "πŸ“Έ Two-Pass Live Photo Optimization",
288
+ value=True,
289
+ help="Runs multi-pass recognition with high sensitivity for difficult lighting, uneven angles, or camera glare."
290
+ )
291
 
292
+ st.markdown("#### πŸ›‘οΈ Privacy & Compliance")
293
+ auto_redact = st.toggle("Auto-Redact PII (Aadhaar, PAN, Cards)", value=False)
294
+
295
+ st.markdown("#### πŸ”’ Semantic Settings")
296
+ embed_unit = st.selectbox("Text Unit", ["word", "sentence", "line"], index=0)
297
+ embed_reduction = st.selectbox("Reduction Method", ["PCA", "t-SNE"], index=0)
298
+ embed_clusters = st.slider("K-Means Clusters", 2, 8, 4)
299
 
300
+ st.markdown("<hr style='margin:1rem 0; border-color:#e2e8f0;'>", unsafe_allow_html=True)
301
  st.markdown("""
302
+ <div style="font-size:0.75rem; color:#64748b; text-align:center; line-height:1.5;">
303
+ <strong>GPU Accelerated</strong> Β· EasyOCR Engine<br>
304
+ OpenCV Preprocessing Β· Sentence-Transformers
305
  </div>
306
  """, unsafe_allow_html=True)
307
 
308
 
309
  # ──────────────────────────────────────────────
310
+ # Main Header
311
  # ──────────────────────────────────────────────
312
  st.markdown("""
313
  <div class="hero-header">DocIQ β€” Intelligent Document Processing</div>
314
  <div class="hero-sub">
315
+ Capture or upload any document β†’ Auto-enhance lighting & blur β†’ Extract text & entities β†’ Measure accuracy β†’ Export
316
  </div>
317
  """, unsafe_allow_html=True)
318
 
319
+
320
+ # ──────────────────────────────────────────────
321
+ # Multi-Input Modes: Upload | Preset Sample | Camera Scanner
322
+ # ──────────────────────────────────────────────
323
  input_tab1, input_tab2, input_tab3 = st.tabs([
324
+ "πŸ“ Upload Image",
325
  "✨ Preset Sample Documents",
326
+ "πŸ“Έ Live Camera Scanner",
327
  ])
328
 
329
  pil_img = None
330
  doc_name = "document.png"
331
+ is_camera_input = False
332
 
333
  with input_tab1:
334
  uploaded_file = st.file_uploader(
335
+ "Drop a document image here (JPG, PNG, WEBP)",
336
  type=["jpg", "jpeg", "png", "webp"],
337
+ help="Upload scanned receipts, invoices, IDs, medical notes, or book pages.",
338
  key="file_uploader_input"
339
  )
340
  if uploaded_file is not None:
 
343
 
344
  with input_tab2:
345
  st.markdown("""
346
+ <div style="color:#64748b; font-size:0.88rem; margin-bottom: 0.8rem;">
347
+ No document image on hand? Click a pre-configured sample to test the complete OCR, accuracy metrics, and export pipeline:
348
  </div>
349
  """, unsafe_allow_html=True)
350
 
351
+ sc1, sc2 = st.columns(2)
352
  for idx, (s_name, s_fn) in enumerate(SAMPLE_DOCUMENTS.items()):
353
  target_col = sc1 if idx % 2 == 0 else sc2
354
  with target_col:
355
  st.markdown(f"""
356
+ <div class="doc-card" style="padding:1.1rem;">
357
+ <div style="font-weight:700; color:#1e1b4b; font-size:1.05rem; margin-bottom:4px;">πŸ“„ {s_name}</div>
358
+ <div style="color:#64748b; font-size:0.8rem; margin-bottom:12px;">Includes line items, monetary values, dates, and contact data.</div>
359
  </div>
360
  """, unsafe_allow_html=True)
361
  if st.button(f"⚑ Load {s_name}", key=f"btn_sample_{idx}", use_container_width=True):
 
365
  if active_s and active_s in SAMPLE_DOCUMENTS and pil_img is None:
366
  pil_img = SAMPLE_DOCUMENTS[active_s]()
367
  doc_name = f"{active_s.lower().replace(' ', '_')}.png"
368
+ st.success(f"βœ“ Loaded sample document: **{active_s}**")
369
 
370
  with input_tab3:
371
+ st.markdown("""
372
+ <div style="color:#64748b; font-size:0.88rem; margin-bottom: 0.5rem;">
373
+ Take a live photo of a document using your webcam or mobile camera.
374
+ DocIQ automatically normalizes EXIF orientation, compensates for shadows, and sharpens text.
375
+ </div>
376
+ """, unsafe_allow_html=True)
377
+ cam_file = st.camera_input("Capture document with camera", key="camera_scanner_input")
378
+ if cam_file is not None:
379
  pil_img = Image.open(cam_file).convert("RGB")
380
  doc_name = "camera_capture.png"
381
+ is_camera_input = True
382
+
383
 
384
+ # ──────────────────────────────────────────────
385
+ # Empty State: Landing Feature Grid
386
+ # ──────────────────────────────────────────────
387
  if pil_img is None:
 
388
  st.markdown("<br>", unsafe_allow_html=True)
389
  cols = st.columns(4)
390
  features = [
391
+ ("🎯", "OCR Accuracy Gauge", "Precise character & word reliability scores with quality grade"),
392
+ ("πŸ“Έ", "Live Photo Ready", "Shadow removal, blur sharpening, and EXIF orientation fix"),
393
+ ("πŸ”", "Interactive Canvas", "Confidence color bounding boxes, reading order, and heatmaps"),
394
+ ("πŸ›‘οΈ", "PII Redaction", "Auto-detect Aadhaar, PAN, credit cards with 1-click masking"),
395
+ ("πŸ“Š", "Structured Parsing", "Auto-classify document and extract dates, amounts, invoices"),
396
+ ("🧬", "Embedding Space", "Cluster words in 2D vector space with similarity matrix"),
397
+ ("πŸ“„", "Searchable PDF", "Export with invisible text layer for native Ctrl+F search"),
398
+ ("πŸ“¦", "All-In-One ZIP", "Single-click bundle with PDFs, JSON, CSV, and TXT"),
399
  ]
400
  for i, (icon, title, desc) in enumerate(features):
401
  with cols[i % 4]:
402
  st.markdown(f"""
403
+ <div class="doc-card" style="min-height:130px;">
404
+ <div style="font-size:1.8rem; margin-bottom:8px;">{icon}</div>
405
+ <div style="font-weight:700; color:#0f172a; font-size:0.95rem;">{title}</div>
406
+ <div style="color:#64748b; font-size:0.78rem; margin-top:4px; line-height:1.45;">{desc}</div>
407
  </div>
408
  """, unsafe_allow_html=True)
409
  st.stop()
410
 
 
 
411
 
412
  # ──────────────────────────────────────────────
413
+ # Preprocessing Pipeline
414
+ # ──────────────────────────────────────────────
415
+ with st.spinner("🎨 Applying image enhancement pipeline..."):
416
+ # If camera capture, default to camera_enhance mode for optimal results
417
+ effective_mode = "camera_enhance" if is_camera_input and preprocess_mode == "auto" else preprocess_mode
418
+ processed_img, preprocess_meta = full_preprocess(pil_img, mode=effective_mode)
419
+
420
+
421
+ # ──────────────────────────────────────────────
422
+ # OCR Neural Inference
423
  # ──────────────────────────────────────────────
424
+ with st.spinner("🧠 Running neural OCR inference (GPU accelerated)..."):
425
+ ocr_results = run_ocr(
426
+ processed_img,
427
+ languages=languages,
428
+ confidence_threshold=confidence_threshold,
429
+ camera_mode=(camera_opt or is_camera_input)
430
+ )
431
 
432
  if not ocr_results:
433
+ st.warning("⚠️ No text detected. Try switching the processing mode to 'Camera & Mobile Capture' in the sidebar or lowering the confidence filter.")
434
  st.stop()
435
 
436
  full_text = get_full_text(ocr_results)
437
+ accuracy_stats = compute_accuracy_metrics(ocr_results, full_text=full_text)
438
 
439
  # PII Redaction
440
  redacted_text, pii_count = redact_pii(full_text) if auto_redact else (full_text, 0)
441
  display_text = redacted_text if auto_redact else full_text
442
 
443
  # Entity Extraction
444
+ with st.spinner("πŸ” Extracting structured entities..."):
445
  entities = extract_all(full_text)
446
 
447
+
448
  # ──────────────────────────────────────────────
449
+ # Accuracy & Quality Banner (Light Theme)
450
+ # ──────────────────────────────────────────────
451
+ acc_val = accuracy_stats["overall_accuracy"]
452
+ acc_grade = accuracy_stats["quality_grade"]
453
+ acc_label = accuracy_stats["reliability_label"]
454
+ acc_color = accuracy_stats["reliability_color"]
455
+
456
+ st.markdown(f"""
457
+ <div class="accuracy-banner">
458
+ <div style="display:flex; justify-content:space-between; align-items:center; flex-wrap:wrap; gap:16px;">
459
+ <div style="display:flex; align-items:center; gap:18px;">
460
+ <div style="width:68px; height:68px; border-radius:50%; background:{acc_color}18; border:3px solid {acc_color}; display:flex; align-items:center; justify-content:center;">
461
+ <span style="font-size:1.6rem; font-weight:800; color:{acc_color};">{acc_grade}</span>
462
+ </div>
463
+ <div>
464
+ <div style="font-size:0.78rem; text-transform:uppercase; font-weight:700; color:#64748b; letter-spacing:0.05em;">
465
+ OCR Accuracy & Quality Score
466
+ </div>
467
+ <div style="font-size:1.8rem; font-weight:800; color:#0f172a; line-height:1.2;">
468
+ {acc_val}% <span style="font-size:1rem; font-weight:600; color:{acc_color}; margin-left:6px;">β€’ {acc_label}</span>
469
+ </div>
470
+ </div>
471
+ </div>
472
+ <div style="display:flex; gap:16px; align-items:center; flex-wrap:wrap;">
473
+ <div style="text-align:right;">
474
+ <div style="font-size:0.75rem; color:#64748b; font-weight:600;">High Confidence (β‰₯80%)</div>
475
+ <div style="font-size:1.15rem; font-weight:700; color:#15803d;">{accuracy_stats['high_conf_pct']}% ({accuracy_stats['high_conf_count']} words)</div>
476
+ </div>
477
+ <div style="text-align:right; border-left:1px solid #e2e8f0; padding-left:16px;">
478
+ <div style="font-size:0.75rem; color:#64748b; font-weight:600;">Lexical Validity</div>
479
+ <div style="font-size:1.15rem; font-weight:700; color:#4338ca;">{accuracy_stats['lexical_validity']}%</div>
480
+ </div>
481
+ </div>
482
+ </div>
483
+ <!-- Confidence Distribution Bar -->
484
+ <div style="margin-top:14px; background:#e2e8f0; height:8px; border-radius:999px; overflow:hidden; display:flex;">
485
+ <div style="width:{accuracy_stats['high_conf_pct']}%; background:#22c55e;" title="High confidence"></div>
486
+ <div style="width:{accuracy_stats['medium_conf_pct']}%; background:#eab308;" title="Moderate confidence"></div>
487
+ <div style="width:{accuracy_stats['low_conf_pct']}%; background:#ef4444;" title="Low confidence"></div>
488
+ </div>
489
+ <div style="display:flex; justify-content:space-between; margin-top:6px; font-size:0.72rem; color:#64748b;">
490
+ <span>🟒 High: {accuracy_stats['high_conf_pct']}%</span>
491
+ <span>🟑 Medium: {accuracy_stats['medium_conf_pct']}%</span>
492
+ <span>πŸ”΄ Low: {accuracy_stats['low_conf_pct']}%</span>
493
+ <span>Total: {accuracy_stats['total_words']} words ({accuracy_stats['total_characters']} characters)</span>
494
+ </div>
495
+ </div>
496
+ """, unsafe_allow_html=True)
497
+
498
+
499
+ # ──────────────────────────────────────────────
500
+ # Metrics Grid
501
  # ──────────────────────────────────────────────
 
502
  m1, m2, m3, m4, m5 = st.columns(5)
503
  metrics = [
504
+ (m1, "Recognized Words", accuracy_stats["total_words"]),
505
+ (m2, "Avg Confidence", f"{accuracy_stats['avg_confidence']:.1%}"),
506
+ (m3, "High Quality Words", accuracy_stats["high_conf_count"]),
507
+ (m4, "PII Detected", pii_count if auto_redact else "β€”"),
508
+ (m5, "Document Type", entities.get("document_type", "General")[:18]),
509
  ]
510
+ for col, label, value in metrics:
511
  with col:
512
  st.markdown(f"""
513
  <div class="metric-card">
514
+ <div class="metric-value">{value}</div>
515
  <div class="metric-label">{label}</div>
516
  </div>
517
  """, unsafe_allow_html=True)
518
 
519
  st.markdown("<br>", unsafe_allow_html=True)
520
 
521
+
522
  # ──────────────────────────────────────────────
523
+ # Main Application Tabs (Light Mode)
524
  # ──────────────────────────────────────────────
525
  tab_vis, tab_search, tab_text, tab_entities, tab_embed, tab_export = st.tabs([
526
  "πŸ–ΌοΈ Visual Canvas",
527
+ "πŸ”Ž Visual Keyword Search",
528
+ "πŸ“ Extracted Text & Confidence",
529
  "πŸ—‚οΈ Entities & PII",
530
  "🧬 Embedding Space",
531
+ "πŸ“€ Multi-Format Export",
532
  ])
533
 
534
+
535
  # ────────────────
536
  # TAB 1: Visual Canvas
537
  # ────────────────
538
  with tab_vis:
539
  c1, c2 = st.columns([3, 1])
540
  with c2:
541
+ st.markdown("##### 🎨 Canvas Layer")
542
  view_mode = st.radio(
543
  "Canvas View",
544
+ ["Bounding Boxes", "Confidence Heatmap", "Reading Order Flow", "Side-by-Side Comparison"],
545
+ index=0,
546
+ label_visibility="collapsed"
547
  )
548
+ st.markdown("---")
549
+ show_labels = st.toggle("Show Word Labels", value=True)
550
+ show_conf = st.toggle("Show Confidence %", value=True)
551
+
552
+ st.markdown("""
553
+ <div style="background:#f8fafc; border:1px solid #e2e8f0; border-radius:12px; padding:12px; margin-top:16px;">
554
+ <div style="font-size:0.75rem; font-weight:700; color:#334155; margin-bottom:6px;">COLOR LEGEND</div>
555
+ <div style="font-size:0.78rem; color:#15803d; margin-bottom:4px;">🟒 Green: β‰₯ 80% Confidence</div>
556
+ <div style="font-size:0.78rem; color:#a16207; margin-bottom:4px;">🟑 Yellow: 50% – 79%</div>
557
+ <div style="font-size:0.78rem; color:#b91c1c;">πŸ”΄ Red: &lt; 50% Confidence</div>
558
+ </div>
559
+ """, unsafe_allow_html=True)
560
 
561
  with c1:
562
  if view_mode == "Bounding Boxes":
 
565
  show_labels=show_labels,
566
  show_confidence=show_conf,
567
  )
568
+ st.image(annotated, use_container_width=True, caption="Bounding Boxes with Confidence Color Coding")
569
  elif view_mode == "Confidence Heatmap":
570
  heatmap_img = create_confidence_heatmap(processed_img, ocr_results)
571
+ st.image(heatmap_img, use_container_width=True, caption="Confidence Heatmap (Blue=High Certainty, Red=Low Certainty)")
572
+ elif view_mode == "Reading Order Flow":
573
  order_img = draw_reading_order(processed_img, ocr_results)
574
+ st.image(order_img, use_container_width=True, caption="Reconstructed Reading Order with Directional Arrows")
575
  else:
576
  col_o, col_p = st.columns(2)
577
  with col_o:
578
+ st.image(pil_img, caption="Original Input", use_container_width=True)
579
  with col_p:
580
+ st.image(processed_img, caption=f"Enhanced Preprocessed ({effective_mode})", use_container_width=True)
581
 
582
+ # Preprocessing tags
583
  st.markdown(f"""
584
+ <div class="doc-card" style="margin-top:0.8rem; padding:0.8rem 1.2rem;">
585
+ <span style="color:#64748b; font-size:0.82rem; font-weight:600;">Active Preprocessing Enhancements: </span>
586
+ {''.join(f'<span class="badge badge-cyan" style="margin-left:6px;">{s.replace("_", " ").title()}</span>' for s in preprocess_meta.get('steps', []))}
587
  </div>
588
  """, unsafe_allow_html=True)
589
 
590
 
591
  # ────────────────
592
+ # TAB 2: Visual Keyword Search
593
  # ────────────────
594
  with tab_search:
595
  search_col, opts_col = st.columns([3, 1])
596
  with opts_col:
597
+ case_sensitive = st.toggle("Case Sensitive Search", value=False)
598
  with search_col:
599
  keyword = st.text_input(
600
+ "Search document keyword",
601
+ placeholder="Type any word or phrase (e.g. invoice, total, patient)...",
602
  label_visibility="collapsed"
603
  )
604
 
 
609
 
610
  if matched_texts:
611
  st.markdown(f"""
612
+ <div class="doc-card" style="background:#f0fdf4; border-color:#bbf7d0;">
613
+ <span style="color:#15803d; font-weight:700; font-size:1rem;">βœ“ Found {len(matched_texts)} matching occurrence(s)</span>
614
+ <div style="margin-top:8px;">
615
+ {' '.join(f'<span class="badge badge-yellow" style="margin-right:4px;">{t}</span>' for t in matched_texts[:20])}
616
+ </div>
617
  </div>
618
  """, unsafe_allow_html=True)
619
  else:
620
  st.markdown(f"""
621
+ <div class="doc-card" style="background:#fef2f2; border-color:#fecaca;">
622
+ <span style="color:#b91c1c; font-weight:700;">βœ— No text matching "{keyword}" was found in the document.</span>
623
  </div>
624
  """, unsafe_allow_html=True)
625
 
626
+ st.image(highlighted_img, use_container_width=True, caption=f"Spatial Search Overlay: '{keyword}' highlighted in gold")
 
627
  else:
628
+ st.info("πŸ‘† Enter a word or phrase above to instantly highlight its exact physical position on the document.")
629
  st.image(processed_img, use_container_width=True, caption="Document Preview")
630
 
631
 
632
  # ────────────────
633
+ # TAB 3: Extracted Text & Quality Table
634
  # ────────────────
635
  with tab_text:
636
  c1, c2 = st.columns([2, 1])
637
  with c1:
638
  st.markdown(f"""<div class="doc-card">
639
  <div style="display:flex; justify-content:space-between; align-items:center; margin-bottom:8px;">
640
+ <span style="font-weight:700; color:#0f172a; font-size:1.05rem;">
641
+ Extracted Text Content
642
  {'<span class="badge badge-red" style="margin-left:8px;">PII REDACTED</span>' if auto_redact and pii_count > 0 else ''}
643
  </span>
644
+ <span style="font-size:0.8rem; color:#64748b; font-weight:600;">
645
+ {len(display_text.split())} words Β· {len(display_text)} characters
646
  </span>
647
  </div>
648
  </div>""", unsafe_allow_html=True)
649
  st.text_area(
650
  "Extracted Text Content",
651
  value=display_text,
652
+ height=400,
653
  label_visibility="collapsed",
654
+ help="Select all and copy with Ctrl+C, or edit freely."
655
  )
656
 
657
  with c2:
658
+ st.markdown("#### πŸ“Š Word Recognition Table")
659
  df = pd.DataFrame([{
660
  "Text": r["text"],
661
  "Confidence": f"{r['confidence']:.1%}",
662
  "Width": r["width"],
663
  "Height": r["height"],
664
  } for r in sorted(ocr_results, key=lambda x: x["confidence"], reverse=True)])
665
+ st.dataframe(df, use_container_width=True, height=430)
666
 
667
 
668
  # ────────────────
669
  # TAB 4: Entities & PII
670
  # ────────────────
671
  with tab_entities:
672
+ doc_type = entities.get("document_type", "General Document")
673
  st.markdown(f"""
674
+ <div class="doc-card" style="background:linear-gradient(135deg, #eef2ff 0%, #e0e7ff 100%); border-color:#c7d2fe;">
675
+ <span style="color:#4338ca; font-size:0.78rem; font-weight:700; text-transform:uppercase; letter-spacing:0.05em;">CLASSIFIED DOCUMENT TYPE</span>
676
+ <div style="font-size:1.6rem; font-weight:800; color:#1e1b4b; margin-top:4px;">πŸ“‹ {doc_type}</div>
677
  </div>
678
  """, unsafe_allow_html=True)
679
 
 
683
  # Dates
684
  dates = entities.get("dates", [])
685
  st.markdown(f"""<div class="doc-card">
686
+ <div style="font-weight:700; color:#0f172a; margin-bottom:8px;">πŸ“… Detected Dates ({len(dates)})</div>
687
+ {''.join(f'<span class="badge badge-cyan" style="margin:2px 3px;">{d}</span>' for d in dates) if dates else '<span style="color:#94a3b8; font-size:0.84rem;">None detected</span>'}
688
  </div>""", unsafe_allow_html=True)
689
 
690
  # Amounts
691
  amounts = entities.get("amounts", [])
692
  st.markdown(f"""<div class="doc-card">
693
+ <div style="font-weight:700; color:#0f172a; margin-bottom:8px;">πŸ’° Monetary Amounts ({len(amounts)})</div>
694
+ {''.join(f'<span class="badge badge-green" style="margin:2px 3px;">{a}</span>' for a in amounts) if amounts else '<span style="color:#94a3b8; font-size:0.84rem;">None detected</span>'}
695
  </div>""", unsafe_allow_html=True)
696
 
697
+ # Contact Info
698
  emails = entities.get("emails", [])
699
  phones = entities.get("phones", [])
700
  urls = entities.get("urls", [])
701
  st.markdown(f"""<div class="doc-card">
702
+ <div style="font-weight:700; color:#0f172a; margin-bottom:8px;">πŸ“¬ Contact Information</div>
703
+ {''.join(f'<span class="badge badge-indigo" style="margin:2px 3px;">βœ‰ {e}</span>' for e in emails)}
704
+ {''.join(f'<span class="badge badge-indigo" style="margin:2px 3px;">πŸ“ž {p}</span>' for p in phones)}
705
+ {''.join(f'<span class="badge badge-cyan" style="margin:2px 3px;">πŸ”— {u[:28]}...</span>' for u in urls[:3])}
706
+ {'<span style="color:#94a3b8; font-size:0.84rem;">None detected</span>' if not emails and not phones and not urls else ''}
707
  </div>""", unsafe_allow_html=True)
708
 
709
  with col_b:
710
+ # Key Document Fields
711
  inv = entities.get("invoice_fields", {})
712
  FIELD_ICONS = {
713
  "invoice_number": "🧾",
 
726
  icon = FIELD_ICONS.get(k, "πŸ“Œ")
727
  title = k.replace('_', ' ').title()
728
  inv_rows.append(f"""
729
+ <div style="display:flex; justify-content:space-between; align-items:center; padding:10px 14px; margin-bottom:8px; background:#f8fafc; border:1px solid #e2e8f0; border-radius:10px;">
730
+ <span style="color:#475569; font-size:0.86rem; display:flex; align-items:center; gap:8px;">
731
  <span>{icon}</span>
732
+ <strong style="color:#1e293b; font-weight:600;">{title}</strong>
733
  </span>
734
+ <span style="color:#0369a1; font-size:0.88rem; font-weight:700; font-family:monospace; background:#e0f2fe; padding:3px 10px; border-radius:6px; border:1px solid #bae6fd;">
735
  {v}
736
  </span>
737
  </div>""")
738
  inv_html = "".join(inv_rows)
739
  st.markdown(f"""<div class="doc-card">
740
+ <div style="font-weight:700; color:#0f172a; margin-bottom:12px; display:flex; align-items:center; justify-content:space-between;">
741
+ <span>🧾 Extracted Key-Value Fields</span>
742
+ <span class="badge badge-indigo">{len(inv)} detected</span>
743
  </div>
744
  {inv_html}
745
  </div>""", unsafe_allow_html=True)
746
  else:
747
  st.markdown("""<div class="doc-card">
748
+ <div style="font-weight:700; color:#0f172a; margin-bottom:6px;">🧾 Key-Value Fields</div>
749
+ <div style="color:#94a3b8; font-size:0.84rem;">No invoice or purchase order key-value fields detected.</div>
750
  </div>""", unsafe_allow_html=True)
751
 
752
+ # PII Detected
753
  pii = entities.get("pii", {})
754
  pii_found = {k: v for k, v in pii.items() if v}
755
  if pii_found:
756
  pii_html = "".join(f"""
757
+ <div style="display:flex; justify-content:space-between; padding:8px 0; border-bottom:1px solid #fee2e2;">
758
+ <span style="color:#b91c1c; font-size:0.86rem; font-weight:600;">⚠ {k.upper()}</span>
759
+ <span style="color:#b91c1c; font-size:0.84rem; font-weight:700;">{len(v)} detected</span>
760
  </div>""" for k, v in pii_found.items())
761
+ st.markdown(f"""<div class="doc-card" style="border-color:#fecaca; background:#fff5f5;">
762
+ <div style="font-weight:700; color:#b91c1c; margin-bottom:8px;">πŸ›‘οΈ Sensitive PII Detected</div>
763
  {pii_html}
764
+ <div style="margin-top:10px; font-size:0.78rem; color:#64748b;">
765
+ Enable "Auto-Redact PII" in the sidebar to mask these identifiers before export.
766
+ </div>
767
  </div>""", unsafe_allow_html=True)
768
  else:
769
+ st.markdown("""<div class="doc-card" style="border-color:#bbf7d0; background:#f0fdf4;">
770
+ <div style="font-weight:700; color:#15803d; margin-bottom:4px;">βœ… No Sensitive PII Found</div>
771
+ <div style="font-size:0.82rem; color:#64748b;">No Aadhaar, PAN, SSN, or credit card patterns detected.</div>
772
  </div>""", unsafe_allow_html=True)
773
 
774
 
775
  # ────────────────
776
+ # TAB 5: Embedding Space (Light Theme)
777
  # ────────────���───
778
  with tab_embed:
779
  st.markdown("""<div class="doc-card">
780
+ <div style="font-weight:700; color:#0f172a; margin-bottom:4px;">🧬 Semantic Vector Space Visualization</div>
781
+ <div style="color:#64748b; font-size:0.85rem;">
782
+ Converts text units into 384-dimensional dense semantic vectors using <code>all-MiniLM-L6-v2</code>,
783
+ then projects to 2D space. Words with related meanings cluster together automatically.
784
+ </div>
785
  </div>""", unsafe_allow_html=True)
786
 
787
+ if st.button("⚑ Generate Embedding Space", type="primary"):
788
+ with st.spinner("πŸ”’ Calculating sentence embeddings and 2D projection..."):
789
  embed_data = build_embedding_data(
790
  ocr_results,
791
  text_unit=embed_unit,
 
797
  coords = np.array(embed_data["coords_2d"])
798
  texts = embed_data["texts"]
799
  labels = embed_data["cluster_labels"]
800
+
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
801
  CLUSTER_COLORS = [
802
+ "#4f46e5", "#0284c7", "#16a34a", "#ca8a04", "#ea580c",
803
+ "#db2777", "#7c3aed", "#059669", "#d97706", "#2563eb"
804
  ]
 
805
 
806
  fig = go.Figure()
 
 
807
  unique_labels = sorted(set(labels))
808
  for cluster_id in unique_labels:
809
  idx = [i for i, l in enumerate(labels) if l == cluster_id]
 
812
  y_vals = [coords[i, 1] for i in idx]
813
  color = CLUSTER_COLORS[cluster_id % len(CLUSTER_COLORS)]
814
 
 
815
  fig.add_trace(go.Scatter(
816
  x=x_vals,
817
  y=y_vals,
 
819
  name=f"Cluster {cluster_id + 1}",
820
  marker=dict(
821
  color=color,
822
+ size=10,
823
+ opacity=0.9,
824
+ line=dict(width=1.5, color="#ffffff"),
825
  ),
826
+ text=[t[:18] for t in cluster_texts],
827
  textposition="top center",
828
+ textfont=dict(size=10, color="#1e293b"),
829
+ hovertemplate="<b>%{customdata}</b><br>x=%{x:.2f}, y=%{y:.2f}<extra></extra>",
830
  customdata=cluster_texts,
831
  ))
832
 
833
  fig.update_layout(
834
  title=dict(
835
+ text=f"<b>Semantic Embedding Space</b> β€” {embed_unit.title()}s via {embed_reduction}",
836
+ font=dict(size=16, color="#0f172a"),
837
  ),
838
+ paper_bgcolor="#ffffff",
839
+ plot_bgcolor="#f8fafc",
840
+ font=dict(color="#334155", size=11),
841
  xaxis=dict(
842
+ title=f"{embed_reduction} Axis 1",
843
+ gridcolor="#e2e8f0",
844
+ zerolinecolor="#cbd5e1",
845
  title_font_color="#64748b",
846
  ),
847
  yaxis=dict(
848
+ title=f"{embed_reduction} Axis 2",
849
+ gridcolor="#e2e8f0",
850
+ zerolinecolor="#cbd5e1",
851
  title_font_color="#64748b",
852
  ),
853
  legend=dict(
854
+ bgcolor="#ffffff",
855
+ bordercolor="#e2e8f0",
856
  borderwidth=1,
857
+ font=dict(color="#334155"),
858
  ),
859
  height=520,
860
  margin=dict(l=10, r=10, t=50, b=10),
861
  )
862
  st.plotly_chart(fig, use_container_width=True)
863
 
864
+ # Cosine similarity heatmap
 
 
 
 
 
 
 
 
 
 
 
 
 
865
  sim_matrix = embed_data.get("similarity_matrix")
866
+ if sim_matrix is not None and len(texts) <= 35:
867
  st.markdown("#### 🌑️ Semantic Similarity Heatmap")
868
+ short_labels = [t[:14] + ("…" if len(t) > 14 else "") for t in texts]
 
869
  fig_heat = px.imshow(
870
  sim_matrix,
871
  x=short_labels,
872
  y=short_labels,
873
+ color_continuous_scale="Viridis",
 
 
 
 
 
 
874
  zmin=0, zmax=1,
875
  aspect="auto",
876
+ title="<b>Cosine Similarity Heatmap</b>",
877
  )
878
  fig_heat.update_layout(
879
+ paper_bgcolor="#ffffff",
880
+ plot_bgcolor="#f8fafc",
881
+ font=dict(color="#334155", size=9),
882
+ height=480,
883
  margin=dict(l=10, r=10, t=50, b=10),
884
  )
885
  st.plotly_chart(fig_heat, use_container_width=True)
 
 
 
 
 
 
 
 
 
 
 
 
886
  else:
887
+ st.warning("⚠️ Document text is too brief to compute clusters. Try an image with more paragraphs.")
888
  else:
889
+ st.info("πŸ‘† Click the button above to project words into semantic vector space.")
 
 
 
 
 
890
 
891
 
892
  # ────────────────
893
+ # TAB 6: Multi-Format Export
894
  # ────────────────
895
  with tab_export:
896
  st.markdown("""<div class="doc-card">
897
+ <div style="font-weight:700; color:#0f172a; margin-bottom:4px;">πŸ“€ Multi-Format Export Center</div>
898
+ <div style="color:#64748b; font-size:0.85rem;">Download your processed document in standard formats or grab the complete archive.</div>
899
  </div>""", unsafe_allow_html=True)
900
 
901
  base_doc_stem = os.path.splitext(doc_name)[0]
902
 
903
+ # One-click bundle ZIP
904
+ st.markdown("##### πŸ“¦ All-In-One Bundle")
905
+ if st.button("πŸ—œοΈ Generate Complete Export Package (.ZIP)", type="primary", use_container_width=True):
906
+ with st.spinner("Bundling Searchable PDF, Audit Report, JSON, CSV, and Plain Text..."):
 
 
 
 
907
  zip_bytes = export_bundle_zip(processed_img, ocr_results, entities, display_text, base_name=base_doc_stem)
908
  st.download_button(
909
  "⬇️ Download All Formats (.ZIP Archive)",
 
913
  use_container_width=True,
914
  )
915
 
916
+ st.markdown("<hr style='margin:1.4rem 0; border-color:#e2e8f0;'>", unsafe_allow_html=True)
917
+ st.markdown("##### πŸ“„ Individual Formats")
 
 
 
 
918
 
919
  ec1, ec2 = st.columns(2)
920
 
921
  with ec1:
922
  # Searchable PDF
923
  if st.button("πŸ“„ Generate Searchable PDF", use_container_width=True):
924
+ with st.spinner("Generating searchable PDF with invisible text layer..."):
925
  pdf_bytes = export_searchable_pdf(processed_img, ocr_results, doc_name)
926
  st.download_button(
927
  "⬇️ Download Searchable PDF",
 
931
  use_container_width=True,
932
  )
933
 
934
+ # Annotated Audit PDF
935
+ if st.button("πŸ“Š Generate Annotated Audit Report (PDF)", use_container_width=True):
936
+ with st.spinner("Generating annotated audit PDF report..."):
937
  annotated_img = draw_bounding_boxes(processed_img, ocr_results, show_labels=True, show_confidence=True)
938
  pdf_bytes = export_annotated_pdf(annotated_img, ocr_results, entities, title="DocIQ OCR Report")
939
  st.download_button(
940
+ "⬇️ Download Annotated PDF Report",
941
  data=pdf_bytes,
942
+ file_name=f"{base_doc_stem}_audit_report.pdf",
943
  mime="application/pdf",
944
  use_container_width=True,
945
  )
946
 
947
  # JSON
948
+ json_bytes = export_json(ocr_results, entities, {"filename": doc_name, "accuracy": accuracy_stats})
949
  st.download_button(
950
+ "⬇️ Download Structured JSON Data",
951
  data=json_bytes,
952
  file_name=f"{base_doc_stem}_data.json",
953
  mime="application/json",
 
958
  # CSV
959
  csv_bytes = export_csv(ocr_results, entities)
960
  st.download_button(
961
+ "⬇️ Download Words Table (CSV / Excel)",
962
  data=csv_bytes,
963
  file_name=f"{base_doc_stem}_words.csv",
964
  mime="text/csv",
 
968
  # Plain Text
969
  txt_bytes = export_txt(ocr_results, full_text=display_text)
970
  st.download_button(
971
+ f"⬇️ Download Clean Plain Text {'(Redacted)' if auto_redact else ''}",
972
  data=txt_bytes,
973
  file_name=f"{base_doc_stem}_text.txt",
974
  mime="text/plain",
975
  use_container_width=True,
976
  )
977
 
978
+ # Entity Summary JSON
979
  st.download_button(
980
  "⬇️ Download Entity Summary (JSON)",
981
  data=export_json([], entities, {"filename": doc_name, "type": "entities_only"}),
 
985
  )
986
 
987
  st.markdown("<br>", unsafe_allow_html=True)
988
+ st.markdown("""<div class="doc-card" style="background:#f8fafc;">
989
+ <div style="font-weight:700; color:#334155; margin-bottom:8px;">πŸ“‹ Export Formats Reference</div>
990
+ <div style="display:grid; grid-template-columns:1fr 1fr; gap:10px;">
991
+ <div><strong style="color:#0284c7;">Searchable PDF</strong><br><span style="color:#64748b; font-size:0.8rem;">Contains hidden text layer. Native Ctrl+F search works in Adobe Acrobat, Chrome, and Preview.</span></div>
992
+ <div><strong style="color:#0284c7;">Annotated PDF Report</strong><br><span style="color:#64748b; font-size:0.8rem;">Visual bounding box document map, accuracy scores, and word-by-word audit trail.</span></div>
993
+ <div><strong style="color:#0284c7;">Structured JSON</strong><br><span style="color:#64748b; font-size:0.8rem;">Precise pixel coordinates, confidence ratings, and parsed entities for downstream pipelines.</span></div>
994
+ <div><strong style="color:#0284c7;">CSV / Excel</strong><br><span style="color:#64748b; font-size:0.8rem;">Tabular dataset ready to import directly into Microsoft Excel, Google Sheets, or Pandas.</span></div>
995
  </div>
996
  </div>""", unsafe_allow_html=True)
src/__pycache__/extractor.cpython-310.pyc CHANGED
Binary files a/src/__pycache__/extractor.cpython-310.pyc and b/src/__pycache__/extractor.cpython-310.pyc differ
 
src/__pycache__/ocr_engine.cpython-310.pyc CHANGED
Binary files a/src/__pycache__/ocr_engine.cpython-310.pyc and b/src/__pycache__/ocr_engine.cpython-310.pyc differ
 
src/__pycache__/preprocessor.cpython-310.pyc CHANGED
Binary files a/src/__pycache__/preprocessor.cpython-310.pyc and b/src/__pycache__/preprocessor.cpython-310.pyc differ