Claude commited on
Commit
a20dd9f
·
unverified ·
1 Parent(s): 2a9e166

Add chat assistant to tailor OCR extraction settings conversationally

Browse files

Adds a chat tab where users describe in plain language what to flag and
pull (e.g. "only flag overdue invoices and pull SAP PO numbers"). A
Hugging Face-hosted chat model (chat_config.py) parses that into
categories/custom keywords/export-filter settings, which drive the same
controls used by the Upload & Run tab. Falls back gracefully with a
clear message if no HF token is configured, so manual controls still
work standalone.

Files changed (4) hide show
  1. README.md +32 -5
  2. app.py +92 -27
  3. chat_config.py +128 -0
  4. requirements.txt +2 -1
README.md CHANGED
@@ -4,7 +4,7 @@ emoji: 🧾
4
  colorFrom: blue
5
  colorTo: indigo
6
  sdk: gradio
7
- sdk_version: 4.44.0
8
  app_file: app.py
9
  pinned: false
10
  license: mit
@@ -25,23 +25,48 @@ and key fields out of paper-heavy workflows:
25
 
26
  ## How it works
27
 
28
- 1. Upload one or more PDFs or images.
29
- 2. Each page's text is pulled directly from the PDF when it has a text
 
 
 
 
 
 
 
30
  layer; scanned/image-only pages are rendered and run through
31
  Tesseract OCR.
32
- 3. The text is scanned against keyword sets for the categories you pick
33
  (or your own custom keywords), and structured fields (PO/SAP/invoice
34
  numbers, dates, dollar amounts, percentages, certification codes) are
35
  pulled out with regex.
36
- 4. Results are shown in a table and exported as a CSV, one row per page
37
  that has a hit (or every page, if you turn that filter off).
38
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
39
  ## Local development
40
 
41
  ```bash
42
  pip install -r requirements.txt
43
  # Tesseract must also be installed on the system, e.g.:
44
  # apt-get install tesseract-ocr
 
45
  python app.py
46
  ```
47
 
@@ -51,3 +76,5 @@ python app.py
51
  - `ocr_engine.py` — PDF/image loading and OCR (PyMuPDF + Tesseract).
52
  - `extractors.py` — keyword notification scanning and structured field
53
  extraction (regex-based, easy to extend per domain).
 
 
 
4
  colorFrom: blue
5
  colorTo: indigo
6
  sdk: gradio
7
+ sdk_version: 5.9.1
8
  app_file: app.py
9
  pinned: false
10
  license: mit
 
25
 
26
  ## How it works
27
 
28
+ 1. **Tailor extraction (chat tab)** — describe in plain language what you
29
+ want flagged and pulled (e.g. *"only flag overdue invoices over $10k
30
+ and non-compliance notices, and pull SAP PO numbers"*). A Hugging
31
+ Face-hosted chat model turns that into extraction settings
32
+ (categories, custom keywords, export filter), which populate the
33
+ Upload & Run tab automatically. You can also skip the chat and set
34
+ these by hand.
35
+ 2. Upload one or more PDFs or images.
36
+ 3. Each page's text is pulled directly from the PDF when it has a text
37
  layer; scanned/image-only pages are rendered and run through
38
  Tesseract OCR.
39
+ 4. The text is scanned against keyword sets for the categories you pick
40
  (or your own custom keywords), and structured fields (PO/SAP/invoice
41
  numbers, dates, dollar amounts, percentages, certification codes) are
42
  pulled out with regex.
43
+ 5. Results are shown in a table and exported as a CSV, one row per page
44
  that has a hit (or every page, if you turn that filter off).
45
 
46
+ ## Chat assistant setup
47
+
48
+ The chat tab calls the Hugging Face Inference API, so it needs a token:
49
+
50
+ - **On a Space:** add a repo secret named `HF_TOKEN` (Settings ->
51
+ Variables and secrets) with a token that has Inference API access.
52
+ - **Locally / as a fallback:** paste a token into the "Hugging Face
53
+ token" field in the chat tab; it's only used for that session.
54
+
55
+ The model defaults to `HuggingFaceH4/zephyr-7b-beta` (ungated). Override
56
+ it with the `HF_CHAT_MODEL` environment variable/secret if you'd rather
57
+ use a different instruct model your token can access.
58
+
59
+ If the chat call fails or no token is available, the Upload & Run tab
60
+ still works fully via its manual checkboxes/textboxes — the chat is a
61
+ convenience layer on top, not a dependency.
62
+
63
  ## Local development
64
 
65
  ```bash
66
  pip install -r requirements.txt
67
  # Tesseract must also be installed on the system, e.g.:
68
  # apt-get install tesseract-ocr
69
+ export HF_TOKEN=hf_... # optional, enables the chat tab
70
  python app.py
71
  ```
72
 
 
76
  - `ocr_engine.py` — PDF/image loading and OCR (PyMuPDF + Tesseract).
77
  - `extractors.py` — keyword notification scanning and structured field
78
  extraction (regex-based, easy to extend per domain).
79
+ - `chat_config.py` — chat assistant that turns natural-language
80
+ instructions into extraction settings via a Hugging Face chat model.
app.py CHANGED
@@ -12,10 +12,17 @@ from datetime import datetime
12
  import gradio as gr
13
  import pandas as pd
14
 
 
15
  import extractors
16
  import ocr_engine
17
 
18
- CATEGORY_CHOICES = ["construction", "compliance", "finance", "general"]
 
 
 
 
 
 
19
 
20
  COLUMNS = [
21
  "file_name", "page", "text_source", "ocr_confidence",
@@ -107,6 +114,26 @@ def process_files(files, categories, custom_keywords_str, only_matches):
107
  return status, df, csv_path, "\n".join(full_text_log)
108
 
109
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
110
  with gr.Blocks(title="OCR Notification & Data Extractor") as demo:
111
  gr.Markdown(
112
  "# OCR Notification & Data Extractor\n"
@@ -117,33 +144,71 @@ with gr.Blocks(title="OCR Notification & Data Extractor") as demo:
117
  "codes), and exports everything to CSV."
118
  )
119
 
120
- with gr.Row():
121
- with gr.Column(scale=1):
122
- files_input = gr.Files(
123
- label="PDF or image files",
124
- file_types=[".pdf", ".png", ".jpg", ".jpeg", ".tif", ".tiff", ".bmp"],
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
125
  )
126
- categories_input = gr.CheckboxGroup(
127
- choices=CATEGORY_CHOICES,
128
- value=CATEGORY_CHOICES,
129
- label="Notification categories",
130
- )
131
- custom_keywords_input = gr.Textbox(
132
- label="Custom keywords (comma-separated, optional)",
133
- placeholder="e.g. re-inspection, warranty claim, hold payment",
134
- )
135
- only_matches_input = gr.Checkbox(
136
- value=True,
137
- label="Only export pages with a notification hit or extracted field",
138
- )
139
- run_button = gr.Button("Run OCR & Extract", variant="primary")
140
-
141
- with gr.Column(scale=2):
142
- status_output = gr.Textbox(label="Status", lines=3, interactive=False)
143
- table_output = gr.Dataframe(label="Extracted records", headers=COLUMNS, wrap=True)
144
- csv_output = gr.File(label="Download CSV")
145
- with gr.Accordion("Full OCR text (debug)", open=False):
146
- text_output = gr.Textbox(label="Raw extracted text", lines=20)
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
147
 
148
  run_button.click(
149
  fn=process_files,
 
12
  import gradio as gr
13
  import pandas as pd
14
 
15
+ import chat_config
16
  import extractors
17
  import ocr_engine
18
 
19
+ CATEGORY_CHOICES = chat_config.CATEGORY_CHOICES
20
+
21
+ DEFAULT_SETTINGS = {
22
+ "categories": CATEGORY_CHOICES,
23
+ "custom_keywords": [],
24
+ "only_matches": True,
25
+ }
26
 
27
  COLUMNS = [
28
  "file_name", "page", "text_source", "ocr_confidence",
 
114
  return status, df, csv_path, "\n".join(full_text_log)
115
 
116
 
117
+ def handle_chat(history, message, settings, hf_token):
118
+ history = history or []
119
+ if not message or not message.strip():
120
+ return (
121
+ history, "", settings, settings,
122
+ settings["categories"], ", ".join(settings["custom_keywords"]), settings["only_matches"],
123
+ )
124
+
125
+ reply, new_settings = chat_config.chat_update(history, message.strip(), settings, hf_token)
126
+ history = history + [
127
+ {"role": "user", "content": message.strip()},
128
+ {"role": "assistant", "content": reply},
129
+ ]
130
+ custom_kw_str = ", ".join(new_settings["custom_keywords"])
131
+ return (
132
+ history, "", new_settings, new_settings,
133
+ new_settings["categories"], custom_kw_str, new_settings["only_matches"],
134
+ )
135
+
136
+
137
  with gr.Blocks(title="OCR Notification & Data Extractor") as demo:
138
  gr.Markdown(
139
  "# OCR Notification & Data Extractor\n"
 
144
  "codes), and exports everything to CSV."
145
  )
146
 
147
+ settings_state = gr.State(dict(DEFAULT_SETTINGS))
148
+
149
+ with gr.Tab("1. Tailor extraction (chat)"):
150
+ gr.Markdown(
151
+ "Describe what you want flagged and pulled, in plain language, "
152
+ "e.g. *\"Only flag overdue invoices over $10k and non-compliance "
153
+ "notices, and pull SAP PO numbers\"*. This updates the settings "
154
+ "used in the **Upload & Run** tab — you can still edit them by "
155
+ "hand there too."
156
+ )
157
+ hf_token_input = gr.Textbox(
158
+ label="Hugging Face token (optional if HF_TOKEN is set as a Space secret)",
159
+ type="password",
160
+ placeholder="hf_...",
161
+ )
162
+ chatbot = gr.Chatbot(height=350, label="Extraction assistant")
163
+ with gr.Row():
164
+ chat_input = gr.Textbox(
165
+ label="Message", scale=4,
166
+ placeholder="e.g. Only flag overdue invoices and pull PO numbers, ignore construction stuff",
167
  )
168
+ chat_send = gr.Button("Send", scale=1, variant="primary")
169
+ settings_display = gr.JSON(label="Active extraction settings", value=DEFAULT_SETTINGS)
170
+
171
+ with gr.Tab("2. Upload & Run"):
172
+ with gr.Row():
173
+ with gr.Column(scale=1):
174
+ files_input = gr.Files(
175
+ label="PDF or image files",
176
+ file_types=[".pdf", ".png", ".jpg", ".jpeg", ".tif", ".tiff", ".bmp"],
177
+ )
178
+ categories_input = gr.CheckboxGroup(
179
+ choices=CATEGORY_CHOICES,
180
+ value=CATEGORY_CHOICES,
181
+ label="Notification categories",
182
+ )
183
+ custom_keywords_input = gr.Textbox(
184
+ label="Custom keywords (comma-separated, optional)",
185
+ placeholder="e.g. re-inspection, warranty claim, hold payment",
186
+ )
187
+ only_matches_input = gr.Checkbox(
188
+ value=True,
189
+ label="Only export pages with a notification hit or extracted field",
190
+ )
191
+ run_button = gr.Button("Run OCR & Extract", variant="primary")
192
+
193
+ with gr.Column(scale=2):
194
+ status_output = gr.Textbox(label="Status", lines=3, interactive=False)
195
+ table_output = gr.Dataframe(label="Extracted records", headers=COLUMNS, wrap=True)
196
+ csv_output = gr.File(label="Download CSV")
197
+ with gr.Accordion("Full OCR text (debug)", open=False):
198
+ text_output = gr.Textbox(label="Raw extracted text", lines=20)
199
+
200
+ chat_send.click(
201
+ fn=handle_chat,
202
+ inputs=[chatbot, chat_input, settings_state, hf_token_input],
203
+ outputs=[chatbot, chat_input, settings_state, settings_display,
204
+ categories_input, custom_keywords_input, only_matches_input],
205
+ )
206
+ chat_input.submit(
207
+ fn=handle_chat,
208
+ inputs=[chatbot, chat_input, settings_state, hf_token_input],
209
+ outputs=[chatbot, chat_input, settings_state, settings_display,
210
+ categories_input, custom_keywords_input, only_matches_input],
211
+ )
212
 
213
  run_button.click(
214
  fn=process_files,
chat_config.py ADDED
@@ -0,0 +1,128 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ """Conversational assistant that turns free-form instructions into
2
+ extraction settings (categories, custom keywords, export filter) using a
3
+ Hugging Face-hosted chat model.
4
+ """
5
+ import json
6
+ import os
7
+ import re
8
+
9
+ from huggingface_hub import InferenceClient
10
+
11
+ import extractors
12
+
13
+ CATEGORY_CHOICES = list(extractors.KEYWORD_SETS.keys())
14
+
15
+ DEFAULT_MODEL = os.environ.get("HF_CHAT_MODEL", "HuggingFaceH4/zephyr-7b-beta")
16
+
17
+ SYSTEM_PROMPT = f"""You are a configuration assistant for a document OCR/data-extraction tool.
18
+
19
+ The tool can flag document pages using built-in keyword categories, each with a
20
+ preset list of "notification" phrases for that domain:
21
+ - construction: change order, delay notice, punch list, non-conformance, RFI,
22
+ submittal rejected, back charge, stop work, safety violation, schedule slip, etc.
23
+ - compliance: non-compliant, violation, deficiency, corrective action, audit
24
+ finding, expired, recall, citation, penalty, failed inspection, etc.
25
+ - finance: past due, overdue, payment rejected, invoice disputed, credit hold,
26
+ chargeback, insufficient funds, write-off, collections, late fee, etc.
27
+ - general: urgent, cancelled, rejected, approved, pending approval, on hold.
28
+
29
+ It always extracts these structured fields wherever present, regardless of
30
+ category: PO numbers, SAP document numbers, invoice numbers, certification
31
+ codes (ISO/OSHA/ASTM/ANSI), dates, dollar amounts, percentages.
32
+
33
+ The user will describe in plain language what they want flagged or pulled.
34
+ Given their message and the current settings, respond with:
35
+ 1. A short (1-3 sentence) conversational reply confirming what you changed,
36
+ or asking a clarifying question if the request is ambiguous.
37
+ 2. On its own line, a fenced json block with the FULL updated settings:
38
+ ```json
39
+ {{"categories": ["finance"], "custom_keywords": ["warranty claim"], "only_matches": true}}
40
+ ```
41
+
42
+ Rules:
43
+ - "categories" must only contain values from {CATEGORY_CHOICES}.
44
+ - "custom_keywords" is a list of extra free-text phrases/terms to flag that
45
+ aren't already covered by a category (empty list if none requested).
46
+ - "only_matches" is true if only pages with a hit should be exported, false
47
+ to export every page regardless of match.
48
+ - Always include all three keys, carrying over prior values the user didn't
49
+ ask to change.
50
+ - Never invent a category, field, or capability that isn't listed above.
51
+ """
52
+
53
+
54
+ def _extract_json_block(text):
55
+ match = re.search(r"```json\s*(\{.*?\})\s*```", text, re.DOTALL)
56
+ if not match:
57
+ match = re.search(r"(\{.*\})", text, re.DOTALL)
58
+ if not match:
59
+ return None
60
+ try:
61
+ return json.loads(match.group(1))
62
+ except json.JSONDecodeError:
63
+ return None
64
+
65
+
66
+ def _sanitize_settings(parsed, current_settings):
67
+ categories = current_settings.get("categories", [])
68
+ custom_keywords = current_settings.get("custom_keywords", [])
69
+ only_matches = current_settings.get("only_matches", True)
70
+
71
+ if isinstance(parsed, dict):
72
+ if isinstance(parsed.get("categories"), list):
73
+ categories = [c for c in parsed["categories"] if c in CATEGORY_CHOICES]
74
+ if isinstance(parsed.get("custom_keywords"), list):
75
+ custom_keywords = [str(k).strip() for k in parsed["custom_keywords"] if str(k).strip()]
76
+ if isinstance(parsed.get("only_matches"), bool):
77
+ only_matches = parsed["only_matches"]
78
+
79
+ return {
80
+ "categories": categories,
81
+ "custom_keywords": custom_keywords,
82
+ "only_matches": only_matches,
83
+ }
84
+
85
+
86
+ def chat_update(history, user_message, current_settings, hf_token=None):
87
+ """Send the conversation to the HF-hosted model and return (reply, new_settings).
88
+
89
+ `history` is a list of {"role": ..., "content": ...} dicts (gr.Chatbot's
90
+ "messages" format). On any failure, the reply explains the error and
91
+ current_settings are returned unchanged.
92
+ """
93
+ token = hf_token or os.environ.get("HF_TOKEN")
94
+ if not token:
95
+ return (
96
+ "I need a Hugging Face token to reach the chat model. Set the "
97
+ "HF_TOKEN secret on this Space, or paste a token in the field above.",
98
+ current_settings,
99
+ )
100
+
101
+ messages = [{"role": "system", "content": SYSTEM_PROMPT}]
102
+ messages.append({
103
+ "role": "system",
104
+ "content": f"Current settings: {json.dumps(current_settings)}",
105
+ })
106
+ messages.extend(history)
107
+ messages.append({"role": "user", "content": user_message})
108
+
109
+ try:
110
+ client = InferenceClient(token=token, provider="auto")
111
+ response = client.chat_completion(
112
+ messages=messages, model=DEFAULT_MODEL, max_tokens=400, temperature=0.2
113
+ )
114
+ content = response.choices[0].message.content
115
+ except Exception as exc:
116
+ return (
117
+ f"The chat model call failed ({exc}). Check that HF_TOKEN is valid "
118
+ f"and that the model '{DEFAULT_MODEL}' is available to your token "
119
+ "(set the HF_CHAT_MODEL env var to switch models).",
120
+ current_settings,
121
+ )
122
+
123
+ parsed = _extract_json_block(content)
124
+ new_settings = _sanitize_settings(parsed, current_settings)
125
+ reply_text = re.sub(r"```json.*?```", "", content, flags=re.DOTALL).strip()
126
+ if not reply_text:
127
+ reply_text = "Updated the extraction settings."
128
+ return reply_text, new_settings
requirements.txt CHANGED
@@ -1,5 +1,6 @@
1
- gradio>=4.44.0
2
  pymupdf>=1.24.0
3
  pytesseract>=0.3.10
4
  pillow>=10.0.0
5
  pandas>=2.0.0
 
 
1
+ gradio>=5.0.0
2
  pymupdf>=1.24.0
3
  pytesseract>=0.3.10
4
  pillow>=10.0.0
5
  pandas>=2.0.0
6
+ huggingface_hub>=0.25.0