segovia commited on
Commit
028d116
·
1 Parent(s): cb04ca0

multitenant supprt

Browse files
MULTI_TENANT_GUIDE.md ADDED
@@ -0,0 +1,540 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # Multi-Tenant Document Parser - Configuration & Usage Guide
2
+
3
+ ## Overview
4
+
5
+ The document parser now supports two operational modes:
6
+
7
+ 1. **Multi-Tenant Mode (NEW)** - Load all active tenants from Firebase and process their enabled rules
8
+ 2. **Legacy Mode** - Single tenant configuration via environment variables (backward compatible)
9
+
10
+ ## Architecture
11
+
12
+ ### Data Model
13
+
14
+ All tenant and rule configurations are stored in Firestore under the `tenants` collection:
15
+
16
+ ```
17
+ TENANTS_COLLECTION (Firestore)
18
+ ├── {tenant_id}
19
+ │ ├── tenant_id: unique tenant ID
20
+ │ ├── name: "Company/User Name"
21
+ │ ├── active: boolean (Account Status)
22
+ │ ├── created_at: timestamp
23
+ │ ├── updated_at: timestamp
24
+ │ ├── credentials (object, 1:1 relationship)
25
+ │ │ ├── openai_api_key: "OpenAI Key"
26
+ │ │ └── google_service_account_json: "Service account JSON"
27
+ │ └── rules (subcollection, 1:N relationship)
28
+ │ └── {rule_id}
29
+ │ ├── rule_id: "unique rule ID"
30
+ │ ├── rule_name: "Invoices to Finance"
31
+ │ ├── source_folder_id: "Google Drive Source Folder ID"
32
+ │ ├── target_folder_id: "Google Drive Destination Folder ID"
33
+ │ ├── target_sheet_id: "Google Sheets Spreadsheet ID"
34
+ │ ├── sheet_tab_name: "Tab inside Google Sheet"
35
+ │ ├── parsing_prompt: "Custom LLM Instructions"
36
+ │ └── is_enabled: boolean (Rule Toggle)
37
+ ```
38
+
39
+ ### Processing Flow
40
+
41
+ ```
42
+ Multi-Tenant Mode:
43
+ 1. orchestrate_all_active_tenants()
44
+ └─> For each active tenant:
45
+ └─> For each enabled rule:
46
+ └─> orchestrate_single_rule()
47
+ ├─> Scan rule.source_folder_id
48
+ ├─> Parse documents with rule.parsing_prompt
49
+ ├─> Move to rule.target_folder_id
50
+ └─> Send to rule.target_sheet_id
51
+ ```
52
+
53
+ ## Usage
54
+
55
+ ### Quick Start (Using .env)
56
+
57
+ If you have existing `.env` configuration, multi-tenant mode will automatically create a default tenant:
58
+
59
+ ```bash
60
+ # Just run it! No Firebase setup needed initially
61
+ python document_orchester.py --send
62
+
63
+ # The system will:
64
+ # 1. Check for active tenants in Firebase
65
+ # 2. If none found, create "default_tenant" from your .env vars
66
+ # 3. Process documents using the default configuration
67
+ ```
68
+
69
+ **This is the easiest path if you're migrating from single-tenant mode.**
70
+
71
+ ### Mode 1: Multi-Tenant Mode (Default)
72
+
73
+ ```bash
74
+ # Process all active tenants and their enabled rules
75
+ python document_orchester.py --model gpt-4o --send
76
+
77
+ # Optional flags:
78
+ # --no-subfolders Only scan direct files (don't recurse)
79
+ # --send Send results to Google Sheets
80
+ # --output FILE Save JSON summary to file
81
+ ```
82
+
83
+ ### Mode 2: Legacy Single-Tenant Mode
84
+
85
+ ```bash
86
+ # Use env var configuration (backward compatible)
87
+ python document_orchester.py --legacy --model gpt-4o --send
88
+ ```
89
+
90
+ ### Gradio Interface
91
+
92
+ Start the web app:
93
+ ```bash
94
+ python app.py
95
+ ```
96
+
97
+ Two orchestrator endpoints are now available:
98
+ - **Single Tenant (Legacy)** - Uses env var configuration
99
+ - **Multi Tenant** - Loads all active tenants from Firebase
100
+
101
+ ## Configuration
102
+
103
+ ### Automatic Default Tenant (No Firebase Setup Required)
104
+
105
+ When you run multi-tenant mode without any tenants in Firebase:
106
+
107
+ 1. The system checks for active tenants in Firestore
108
+ 2. If none found, it creates a **default tenant** from your `.env` file
109
+ 3. Maps env vars like this:
110
+
111
+ | .env Variable | Default Tenant |
112
+ |---|---|
113
+ | `OPENAI_API_KEY` | `credentials.openai_api_key` |
114
+ | `GOOGLE_SERVICE_ACCOUNT_JSON` | `credentials.google_service_account_json` |
115
+ | `GOOGLE_DRIVE_FOLDER_IDS` (first) | `rule.source_folder_id` |
116
+ | `GOOGLE_DRIVE_INVOICES_ROOT_FOLDER_ID` | `rule.target_folder_id` |
117
+ | `GOOGLE_SHEETS_SPREADSHEET_ID` | `rule.target_sheet_id` |
118
+
119
+ **Example: Your existing `.env`**
120
+ ```env
121
+ OPENAI_API_KEY=sk-abc123...
122
+ GOOGLE_SERVICE_ACCOUNT_JSON={"type":"service_account","project_id":"my-project",...}
123
+ GOOGLE_DRIVE_FOLDER_IDS=1ABC_folder_id_here
124
+ GOOGLE_DRIVE_INVOICES_ROOT_FOLDER_ID=1XYZ_destination_folder_id
125
+ GOOGLE_SHEETS_SPREADSHEET_ID=18qKL...
126
+ ```
127
+
128
+ **Creates this default tenant (in memory, not saved to Firebase):**
129
+ ```json
130
+ {
131
+ "tenant_id": "default_tenant",
132
+ "name": "Default Tenant (from .env)",
133
+ "active": true,
134
+ "credentials": {
135
+ "openai_api_key": "sk-abc123...",
136
+ "google_service_account_json": "{...}"
137
+ },
138
+ "rules": [
139
+ {
140
+ "rule_id": "default_rule",
141
+ "rule_name": "Default Processing Rule",
142
+ "source_folder_id": "1ABC_folder_id_here",
143
+ "target_folder_id": "1XYZ_destination_folder_id",
144
+ "target_sheet_id": "18qKL...",
145
+ "sheet_tab_name": "Parsed",
146
+ "is_enabled": true
147
+ }
148
+ ]
149
+ }
150
+ ```
151
+
152
+ The default tenant provides automatic setup:
153
+ - ✅ Created automatically if no Firebase tenants exist
154
+ - ✅ Saved to Firebase **BEFORE** orchestration starts
155
+ - ✅ Future runs load from Firebase instead of recreating
156
+ - ✅ Try multi-tenant mode without any setup
157
+ - ✅ Migrate gradually from single-tenant to multi-tenant
158
+ - ✅ Keep existing workflows unchanged
159
+
160
+ **The flow:**
161
+ 1. First run: No Firebase tenants → Create default_tenant from .env → **Save to Firebase** → Process documents
162
+ 2. Second run: Firebase has default_tenant → Load it → Process documents
163
+ 3. Add more tenants anytime in Firebase → They're all processed together
164
+
165
+ ### Multi-Tenant Setup (Firebase)
166
+
167
+ 1. Create a Firestore collection called `tenants` (or set via `FIREBASE_TENANTS_COLLECTION` env var)
168
+
169
+ 2. Add tenant documents with this structure:
170
+
171
+ ```json
172
+ {
173
+ "name": "Acme Corp",
174
+ "active": true,
175
+ "created_at": "2024-01-01T00:00:00Z",
176
+ "updated_at": "2024-01-01T00:00:00Z",
177
+ "credentials": {
178
+ "openai_api_key": "sk-...",
179
+ "google_service_account_json": "{...full service account json...}"
180
+ }
181
+ }
182
+ ```
183
+
184
+ 3. Create a **rules** subcollection inside the tenant document, and add rules with this structure:
185
+
186
+ ```json
187
+ {
188
+ "rule_id": "invoices_2024",
189
+ "rule_name": "Invoices to Finance",
190
+ "source_folder_id": "1ABC...",
191
+ "target_folder_id": "1XYZ...",
192
+ "target_sheet_id": "18qKL...",
193
+ "sheet_tab_name": "Invoices",
194
+ "parsing_prompt": "Extract invoice details following these rules...",
195
+ "is_enabled": true
196
+ }
197
+ ```
198
+
199
+ ### Environment Variables
200
+
201
+ **Required for both modes:**
202
+ ```
203
+ FIREBASE_SERVICE_ACCOUNT_JSON # Firebase credentials (JSON string or use GOOGLE_SERVICE_ACCOUNT_JSON)
204
+ OPENAI_API_KEY # (only for legacy mode, multi-tenant uses tenant credentials)
205
+ ```
206
+
207
+ **Optional:**
208
+ ```
209
+ FIREBASE_TENANTS_COLLECTION # Default: "tenants"
210
+ FIREBASE_DOCUMENTS_COLLECTION # Default: "processed_documents"
211
+ FIREBASE_TRACKER_COLLECTION # Default: "processed_files"
212
+ FIREBASE_TRACK_PROCESSED # Default: "true"
213
+ OPENAI_MODEL # Default: "gpt-4o"
214
+ GOOGLE_DRIVE_INVOICES_BASE_PATH # Default: "Facturas"
215
+ ORCHESTRATOR_API_KEY # For API authentication
216
+ ```
217
+
218
+ ## Firebase Storage Structure
219
+
220
+ ### Document Tracking (Multi-Tenant)
221
+
222
+ Each tenant's processed files are tracked in nested collections:
223
+
224
+ ```
225
+ tenants/{tenant_id}/processed_files/{file_hash}
226
+ └─ file_hash: "sha256 hash"
227
+ └─ source_file: "filename"
228
+ └─ processed_at: "timestamp"
229
+
230
+ tenants/{tenant_id}/processed_documents/{document_id}
231
+ └─ document_id: "drive file id"
232
+ └─ parsed_data: {...}
233
+ └─ status: "Parsed|Modified|Sent"
234
+ ```
235
+
236
+ ### Creating Tenants Programmatically
237
+
238
+ ```python
239
+ from firebase_tenant_config import (
240
+ FirebaseTenantConfigManager,
241
+ TenantConfig,
242
+ CredentialsObject,
243
+ RuleObject,
244
+ )
245
+
246
+ manager = FirebaseTenantConfigManager()
247
+
248
+ # Create credentials
249
+ credentials = CredentialsObject(
250
+ openai_api_key="sk-...",
251
+ google_service_account_json="..." # full JSON as string
252
+ )
253
+
254
+ # Create a rule
255
+ rule = RuleObject(
256
+ rule_id="invoices_2024",
257
+ rule_name="Invoices to Finance",
258
+ source_folder_id="1ABC...",
259
+ target_folder_id="1XYZ...",
260
+ target_sheet_id="18qKL...",
261
+ sheet_tab_name="Invoices",
262
+ parsing_prompt="Extract...",
263
+ is_enabled=True,
264
+ )
265
+
266
+ # Create tenant
267
+ tenant = TenantConfig(
268
+ tenant_id="acme-corp",
269
+ name="Acme Corp",
270
+ active=True,
271
+ credentials=credentials,
272
+ rules=[rule],
273
+ )
274
+
275
+ # Save to Firebase
276
+ manager.save_tenant(tenant)
277
+
278
+ # List all active tenants
279
+ active_tenants = manager.list_active_tenants()
280
+
281
+ # Get enabled rules for a tenant
282
+ enabled_rules = manager.get_enabled_rules("acme-corp")
283
+
284
+ # Add a rule
285
+ manager.add_rule("acme-corp", new_rule)
286
+
287
+ # Remove a rule
288
+ manager.remove_rule("acme-corp", "invoices_2024")
289
+ ```
290
+
291
+ ## Output Format
292
+
293
+ ### Multi-Tenant Orchestration Result
294
+
295
+ ```json
296
+ {
297
+ "ok": true,
298
+ "tenants_processed": 2,
299
+ "total_parsed": 15,
300
+ "total_modified": 3,
301
+ "total_sent": 18,
302
+ "total_moved": 18,
303
+ "total_corrupted": 2,
304
+ "total_errors": 1,
305
+ "tenants_results": [
306
+ {
307
+ "tenant_id": "tenant_123",
308
+ "tenant_name": "Acme Corp",
309
+ "ok": true,
310
+ "rules_processed": 2,
311
+ "parsed": 10,
312
+ "modified": 2,
313
+ "sent": 12,
314
+ "moved": 12,
315
+ "corrupted": 1,
316
+ "errors": 0,
317
+ "rules_results": [
318
+ {
319
+ "ok": true,
320
+ "tenant_id": "tenant_123",
321
+ "rule_id": "invoices_2024",
322
+ "rule_name": "Invoices to Finance",
323
+ "scanned": 20,
324
+ "to_process": 15,
325
+ "skipped": 5,
326
+ "parsed": 10,
327
+ "modified": 2,
328
+ "sent": 12,
329
+ "moved": 12,
330
+ "corrupted": 1,
331
+ "errors": 0,
332
+ "processed_items": [...],
333
+ "skipped_items": [...]
334
+ }
335
+ ]
336
+ }
337
+ ]
338
+ }
339
+ ```
340
+
341
+ ## Migration from Legacy Mode
342
+
343
+ **No migration needed!** The system maintains full backward compatibility:
344
+
345
+ 1. Existing env var configuration continues to work with `--legacy` flag
346
+ 2. New multi-tenant mode coexists without affecting legacy deployments
347
+ 3. To enable multi-tenant: Set up tenants in Firebase and run without `--legacy`
348
+
349
+ ## Workflow Examples
350
+
351
+ ### Simple: Use Existing .env (No Firebase Setup)
352
+
353
+ The **easiest** way to get started with multi-tenant mode:
354
+
355
+ ```bash
356
+ # 1. Ensure your .env has these vars (you probably already do):
357
+ # OPENAI_API_KEY
358
+ # GOOGLE_SERVICE_ACCOUNT_JSON
359
+ # GOOGLE_DRIVE_FOLDER_IDS
360
+ # GOOGLE_DRIVE_INVOICES_ROOT_FOLDER_ID
361
+ # GOOGLE_SHEETS_SPREADSHEET_ID
362
+
363
+ # 2. Just run multi-tenant orchestration:
364
+ python document_orchester.py --send
365
+
366
+ # That's it! The system will:
367
+ # - Check for tenants in Firebase
368
+ # - Find none, create default_tenant from .env
369
+ # - Process documents exactly as before
370
+ # - Results go to Google Sheets and Drive
371
+ ```
372
+
373
+ **No code changes, no Firebase setup, just works!**
374
+
375
+ ### Advanced: Multiple Tenants in Firebase
376
+
377
+ For multiple tenants with different configurations:
378
+
379
+ ### Step 1: Set Up Tenant in Firebase
380
+
381
+ ```python
382
+ # Run this once to initialize a tenant
383
+ from firebase_tenant_config import FirebaseTenantConfigManager, TenantConfig, CredentialsObject, RuleObject
384
+
385
+ manager = FirebaseTenantConfigManager()
386
+
387
+ credentials = CredentialsObject(
388
+ openai_api_key="sk-xxx",
389
+ google_service_account_json="json_string_here"
390
+ )
391
+
392
+ rule = RuleObject(
393
+ rule_id="rule_1",
394
+ rule_name="Invoice Processing",
395
+ source_folder_id="drive_folder_id",
396
+ target_folder_id="drive_dest_folder_id",
397
+ target_sheet_id="sheets_id",
398
+ sheet_tab_name="Invoices",
399
+ is_enabled=True,
400
+ )
401
+
402
+ tenant = TenantConfig(
403
+ tenant_id="my_tenant",
404
+ name="My Company",
405
+ active=True,
406
+ credentials=credentials,
407
+ )
408
+
409
+ manager.save_tenant(tenant)
410
+ manager.add_rule("my_tenant", rule)
411
+ ```
412
+
413
+ ### Step 2: Run Orchestration
414
+
415
+ ```bash
416
+ # This will automatically:
417
+ # 1. Load all active tenants from Firebase (not just one!)
418
+ # 2. For each tenant, process all enabled rules
419
+ # 3. Track results in tenant-specific collections
420
+ python document_orchester.py --model gpt-4o --send
421
+ ```
422
+
423
+ ### Step 3: Review Results
424
+
425
+ Results are stored in:
426
+ - Google Drive: Documents moved to `{target_folder_id}/{YYYY}{MM}/`
427
+ - Google Sheets: Parsed data appended to `{target_sheet_id}` sheet
428
+ - Firebase: Document metadata in `tenants/{tenant_id}/processed_documents/`
429
+
430
+ ## Troubleshooting
431
+
432
+ ### "No active tenants found"
433
+ - Check that `active: true` in Firebase tenant document
434
+ - Verify `FIREBASE_TENANTS_COLLECTION` env var is correct
435
+ - Confirm Firebase credentials are valid
436
+
437
+ ### "No enabled rules"
438
+ - Check that `is_enabled: true` for at least one rule
439
+ - Verify rule has valid `source_folder_id`
440
+
441
+ ### Credentials error
442
+ - Ensure `credentials.openai_api_key` and `credentials.google_service_account_json` are set
443
+ - Verify JSON strings are properly formatted
444
+
445
+ ### Mixing modes
446
+ - Don't mix multi-tenant and legacy configurations
447
+ - Use either all Firebase-based tenants OR all env vars, not both
448
+ - When switching modes, ensure all required env vars/Firebase docs exist
449
+
450
+ ## API Endpoints
451
+
452
+ ### Gradio Interface
453
+
454
+ Two REST-compatible endpoints are exposed:
455
+
456
+ **Legacy Single-Tenant:**
457
+ ```
458
+ POST /api/orchestrate_drive_documents
459
+ {
460
+ "model": "gpt-4o",
461
+ "include_subfolders": true,
462
+ "send_to_sheet": true,
463
+ "api_key": "your_orchestrator_api_key"
464
+ }
465
+ ```
466
+
467
+ **Multi-Tenant:**
468
+ ```
469
+ POST /api/orchestrate_all_tenants
470
+ {
471
+ "model": "gpt-4o",
472
+ "include_subfolders": true,
473
+ "send_to_sheet": true,
474
+ "api_key": "your_orchestrator_api_key"
475
+ }
476
+ ```
477
+
478
+ ## Performance Considerations
479
+
480
+ - Multi-tenant processing is sequential (one tenant at a time, then one rule at a time)
481
+ - Each tenant/rule maintains separate Firebase document tracking
482
+ - Large scale deployments may benefit from rule parallelization (future enhancement)
483
+
484
+ ## Managing Rules
485
+
486
+ ### Adding a Rule to an Existing Tenant
487
+
488
+ You can programmatically add rules to any tenant:
489
+
490
+ ```python
491
+ from firebase_tenant_config import (
492
+ FirebaseTenantConfigManager,
493
+ RuleObject,
494
+ )
495
+
496
+ manager = FirebaseTenantConfigManager()
497
+
498
+ # Create a new rule
499
+ new_rule = RuleObject(
500
+ rule_id="new_rule_123",
501
+ rule_name="Additional Processing Rule",
502
+ source_folder_id="1NEW_folder_id",
503
+ target_folder_id="1NEW_destination_id",
504
+ target_sheet_id="1NEW_sheet_id",
505
+ sheet_tab_name="Sheet Tab Name",
506
+ parsing_prompt="Custom parsing instructions...",
507
+ is_enabled=True,
508
+ )
509
+
510
+ # Add to tenant
511
+ manager.add_rule("my_tenant", new_rule)
512
+ ```
513
+
514
+ ### Cloning a Rule
515
+
516
+ Use the provided script to clone the default rule:
517
+
518
+ ```bash
519
+ python add_rule_from_default.py new_rule_id "New Rule Name"
520
+ ```
521
+
522
+ This creates a new rule in the **default_tenant** with the same configuration (folder IDs, sheet, etc.) as the default rule. Perfect for creating multiple processing pipelines with identical settings.
523
+
524
+ ### Removing a Rule
525
+
526
+ ```python
527
+ from firebase_tenant_config import FirebaseTenantConfigManager
528
+
529
+ manager = FirebaseTenantConfigManager()
530
+ manager.remove_rule("my_tenant", "rule_id_to_remove")
531
+ ```
532
+
533
+ ## Future Enhancements
534
+
535
+ Potential improvements:
536
+ - Parallel rule processing within a tenant
537
+ - Rule scheduling (cron-like execution)
538
+ - Audit logging for rule changes
539
+ - Rule versioning and rollback
540
+ - Web UI for rule management
README.md CHANGED
@@ -1,5 +1,5 @@
1
  ---
2
- title: Document Parser
3
  emoji: 📚
4
  colorFrom: pink
5
  colorTo: purple
@@ -24,8 +24,10 @@ The app supports scanning one or more Google Drive folders and parsing only file
24
  Optional:
25
 
26
  - `FIREBASE_TRACK_PROCESSED`: defaults to `true`
27
- - `FIREBASE_PROCESSED_COLLECTION`: defaults to `processed_files`
28
  - `FIREBASE_DOCUMENTS_COLLECTION`: defaults to `processed_documents`
 
 
29
 
30
  ### How it works
31
 
@@ -39,6 +41,13 @@ Optional:
39
 
40
  The orchestration script scans Drive, checks if a document was already processed, parses only new/modified documents, and saves parsed payload + lifecycle status in Firebase.
41
 
 
 
 
 
 
 
 
42
  ### Status rules
43
 
44
  - `Parsed`: document was never processed before.
 
1
  ---
2
+ title: Invoice Parser Agent
3
  emoji: 📚
4
  colorFrom: pink
5
  colorTo: purple
 
24
  Optional:
25
 
26
  - `FIREBASE_TRACK_PROCESSED`: defaults to `true`
27
+ - `FIREBASE_TRACKER_COLLECTION`: defaults to `processed_files`
28
  - `FIREBASE_DOCUMENTS_COLLECTION`: defaults to `processed_documents`
29
+ - `GOOGLE_DRIVE_FACTURAS_BASE_PATH`: defaults to `Facturas`
30
+ - `GOOGLE_DRIVE_INVOICE_ROOT_FOLDER_ID`: optional Drive folder id used as root when creating `GOOGLE_DRIVE_FACTURAS_BASE_PATH`
31
 
32
  ### How it works
33
 
 
41
 
42
  The orchestration script scans Drive, checks if a document was already processed, parses only new/modified documents, and saves parsed payload + lifecycle status in Firebase.
43
 
44
+ Before each document is persisted, the file is moved in Drive using this path rule:
45
+
46
+ - `Facturas/yyyyMM` for valid invoices where `fecha` can be parsed.
47
+ - `Facturas/Corrupted` when required sheet data is missing or `fecha` is not parseable.
48
+
49
+ You can change `Facturas` with `GOOGLE_DRIVE_FACTURAS_BASE_PATH`.
50
+
51
  ### Status rules
52
 
53
  - `Parsed`: document was never processed before.
app.py CHANGED
@@ -1,15 +1,13 @@
1
  from __future__ import annotations
2
 
3
  import os
4
- from datetime import datetime
5
 
6
  # Avoid the Node.js SSR proxy in hosted environments where it can terminate unexpectedly.
7
  os.environ["GRADIO_SSR_MODE"] = "false"
8
 
9
  import gradio as gr
10
 
11
- from document_orchester import orchestrate_drive_documents
12
- from firebase_processed_files import build_firebase_tracker
13
  from google_sheets_service import GoogleSheetsConfigError, append_row_to_google_sheet
14
 
15
  try:
@@ -17,26 +15,6 @@ try:
17
  except ImportError:
18
  spaces = None
19
 
20
-
21
- RESULT_COLUMNS = [
22
- "source_file",
23
- "document_type",
24
- "fecha",
25
- "fecha_vencimiento",
26
- "cuit_proveedor",
27
- "description_proveedor",
28
- "moneda",
29
- "subtotal",
30
- "taxes",
31
- "total",
32
- "confidence",
33
- ]
34
- DISPLAY_DESCRIPTION_COLUMN = "display_description"
35
- GRID_COLUMNS = ["confidence", "fecha", "source_file", DISPLAY_DESCRIPTION_COLUMN, "total"]
36
- GRID_EDITABLE_COLUMNS = ["confidence", "fecha", "source_file", "total"]
37
- GRID_HEADER_LABELS = ["Confianza", "Fecha", "Nombre del archivo", "Descripcion", "Total", "Seleccionar"]
38
- SHEET_COLUMNS = ["fecha", "source_file", DISPLAY_DESCRIPTION_COLUMN, "total"]
39
- PENDING_STATUSES = ["Parsed", "Modified", "parsed", "modified"]
40
  custom_css = """
41
  .cell-menu, .cell-menu-button {
42
  display: none !important;
@@ -49,306 +27,9 @@ def gpu_decorator(fn):
49
  return fn
50
  return spaces.GPU(fn)
51
 
52
-
53
- firebase_tracker = build_firebase_tracker()
54
-
55
-
56
- def build_display_description(row: dict) -> str:
57
- provider_name = str(row.get("description_proveedor") or "").strip()
58
- provider_cuit = str(row.get("cuit_proveedor") or "").strip()
59
- city = str(row.get("ciudad") or row.get("city") or "").strip()
60
-
61
- parts = [part for part in [provider_name, provider_cuit, city] if part]
62
- return " - ".join(parts)
63
-
64
-
65
- def sort_rows_by_fecha(rows: list[dict]) -> list[dict]:
66
- def sort_key(row: dict):
67
- raw = row.get("fecha")
68
- text = "" if raw is None else str(raw).strip()
69
- if not text:
70
- return (1, float("inf"), "")
71
-
72
- for fmt in ("%Y-%m-%d", "%d/%m/%Y", "%d-%m-%Y", "%Y/%m/%d"):
73
- try:
74
- parsed = datetime.strptime(text, fmt)
75
- return (0, parsed.timestamp(), "")
76
- except ValueError:
77
- continue
78
-
79
- try:
80
- parsed = datetime.fromisoformat(text)
81
- return (0, parsed.timestamp(), "")
82
- except ValueError:
83
- return (0, float("inf"), text.lower())
84
-
85
- return sorted(rows, key=sort_key)
86
-
87
-
88
- def rows_to_table(rows: list[dict]) -> list[list[object]]:
89
- table_rows: list[list[object]] = []
90
- for row in rows:
91
- values: list[object] = []
92
- for column in GRID_COLUMNS:
93
- if column == DISPLAY_DESCRIPTION_COLUMN:
94
- values.append(build_display_description(row))
95
- else:
96
- values.append(row.get(column))
97
- table_rows.append(values + [False])
98
-
99
- return table_rows
100
-
101
-
102
- def table_to_rows(table: object) -> list[dict]:
103
- if table is None:
104
- return []
105
-
106
- if hasattr(table, "iterrows"):
107
- try:
108
- rows: list[dict] = []
109
- for _, record in table.iterrows():
110
- values = list(record)
111
- if len(values) > len(GRID_COLUMNS):
112
- values = values[: len(GRID_COLUMNS)]
113
- row = {}
114
- for index, column in enumerate(GRID_COLUMNS):
115
- if column not in GRID_EDITABLE_COLUMNS:
116
- continue
117
- row[column] = values[index] if index < len(values) else None
118
- rows.append(row)
119
- return rows
120
- except Exception:
121
- pass
122
-
123
- rows: list[dict] = []
124
- for raw_row in table: # type: ignore[assignment]
125
- values = list(raw_row)
126
- if len(values) > len(GRID_COLUMNS):
127
- values = values[: len(GRID_COLUMNS)]
128
- row = {}
129
- for index, column in enumerate(GRID_COLUMNS):
130
- if column not in GRID_EDITABLE_COLUMNS:
131
- continue
132
- row[column] = values[index] if index < len(values) else None
133
- rows.append(row)
134
- return rows
135
-
136
-
137
- def table_to_display_rows(table: object) -> list[dict]:
138
- if table is None:
139
- return []
140
-
141
- rows: list[dict] = []
142
-
143
- if hasattr(table, "iterrows"):
144
- try:
145
- for _, record in table.iterrows():
146
- values = list(record)
147
- if len(values) > len(GRID_COLUMNS):
148
- values = values[: len(GRID_COLUMNS)]
149
- row = {}
150
- for index, column in enumerate(GRID_COLUMNS):
151
- row[column] = values[index] if index < len(values) else None
152
- rows.append(row)
153
- return rows
154
- except Exception:
155
- pass
156
-
157
- for raw_row in table: # type: ignore[assignment]
158
- values = list(raw_row)
159
- if len(values) > len(GRID_COLUMNS):
160
- values = values[: len(GRID_COLUMNS)]
161
- row = {}
162
- for index, column in enumerate(GRID_COLUMNS):
163
- row[column] = values[index] if index < len(values) else None
164
- rows.append(row)
165
-
166
- return rows
167
-
168
-
169
- def get_selected_row_indexes(grid_value: object) -> list[int]:
170
- if grid_value is None:
171
- return []
172
-
173
- selected_indexes: list[int] = []
174
-
175
- if hasattr(grid_value, "iterrows"):
176
- try:
177
- for index, row in grid_value.iterrows():
178
- values = list(row)
179
- if values and bool(values[-1]):
180
- selected_indexes.append(int(index))
181
- return selected_indexes
182
- except Exception:
183
- pass
184
-
185
- for index, raw_row in enumerate(grid_value): # type: ignore[assignment]
186
- values = list(raw_row)
187
- if values and bool(values[-1]):
188
- selected_indexes.append(index)
189
-
190
- return selected_indexes
191
-
192
-
193
- def _build_row_from_document(document: dict) -> dict:
194
- parsed_data = document.get("parsed_data")
195
-
196
- row: dict = {column: None for column in RESULT_COLUMNS}
197
- if isinstance(parsed_data, dict):
198
- for column in RESULT_COLUMNS:
199
- if column in parsed_data:
200
- row[column] = parsed_data.get(column)
201
-
202
- if document.get("source_file"):
203
- row["source_file"] = document.get("source_file")
204
-
205
- row["_document_id"] = str(document.get("document_id") or "").strip()
206
- row["_modificationDate"] = document.get("modificationDate")
207
- row["_status"] = document.get("status")
208
- return row
209
-
210
-
211
  @gpu_decorator
212
- def load_pending_documents():
213
- if firebase_tracker is None:
214
- rows: list[dict] = []
215
- return rows, gr.update(value=rows_to_table(rows)), "Firebase tracking no esta habilitado."
216
-
217
- try:
218
- documents = firebase_tracker.list_documents_by_statuses(PENDING_STATUSES)
219
- except Exception as exc:
220
- rows = []
221
- return rows, gr.update(value=rows_to_table(rows)), f"No se pudieron cargar documentos pendientes: {exc}"
222
-
223
- rows = [_build_row_from_document(document) for document in documents]
224
- rows = sort_rows_by_fecha(rows)
225
- return rows, gr.update(value=rows_to_table(rows)), f"Se cargaron {len(rows)} documento(s) con estado Parsed/Modified."
226
-
227
-
228
- def clear_results():
229
- return [], gr.update(value=[]), "Grilla limpiada."
230
-
231
-
232
- def sync_grid_to_state(*args: object):
233
- if len(args) == 1:
234
- existing_rows = None
235
- grid_value = args[0]
236
- elif len(args) == 2:
237
- existing_rows, grid_value = args
238
- else:
239
- return [], "Grilla actualizada."
240
-
241
- rows = table_to_rows(grid_value)
242
- previous_rows = list(existing_rows or []) if isinstance(existing_rows, list) else []
243
-
244
- merged_rows: list[dict] = []
245
- for index, row in enumerate(rows):
246
- merged_row: dict = {column: None for column in RESULT_COLUMNS}
247
- if index < len(previous_rows):
248
- previous_row = previous_rows[index]
249
- if isinstance(previous_row, dict):
250
- merged_row.update(previous_row)
251
- for key, value in previous_row.items():
252
- if key not in RESULT_COLUMNS:
253
- merged_row[key] = value
254
- merged_row.update(row)
255
- merged_rows.append(merged_row)
256
-
257
- return merged_rows, "Grilla actualizada."
258
-
259
-
260
- def delete_selected_rows(existing_rows: list[dict] | None, grid_value: object):
261
- rows = list(existing_rows or [])
262
- selected_indexes = get_selected_row_indexes(grid_value)
263
- if not selected_indexes:
264
- return rows, gr.update(value=rows_to_table(rows)), "Selecciona al menos una fila con el checkbox."
265
-
266
- for index in selected_indexes:
267
- if index < 0 or index >= len(rows):
268
- return rows, gr.update(value=rows_to_table(rows)), "Una o mas filas seleccionadas estan fuera de rango."
269
-
270
- selected_set = set(selected_indexes)
271
- remaining_rows = [row for index, row in enumerate(rows) if index not in selected_set]
272
-
273
- return (
274
- remaining_rows,
275
- gr.update(value=rows_to_table(remaining_rows)),
276
- f"Se eliminaron {len(selected_indexes)} fila(s).",
277
- )
278
-
279
-
280
- def _build_updated_parsed_data(row: dict) -> dict:
281
- return {column: row.get(column) for column in RESULT_COLUMNS}
282
-
283
-
284
- def send_selected_rows(existing_rows: list[dict] | None, grid_value: object):
285
- rows = list(existing_rows or [])
286
- selected_indexes = sorted(set(get_selected_row_indexes(grid_value)))
287
- display_rows = table_to_display_rows(grid_value)
288
- if not selected_indexes:
289
- return rows, gr.update(value=rows_to_table(rows)), "Selecciona al menos una fila con el checkbox."
290
-
291
- for index in selected_indexes:
292
- if index < 0 or index >= len(rows):
293
- return rows, gr.update(value=rows_to_table(rows)), "Una o mas filas seleccionadas estan fuera de rango."
294
- if index >= len(display_rows):
295
- return rows, gr.update(value=rows_to_table(rows)), "No se pudo leer una o mas filas seleccionadas de la grilla."
296
-
297
- sent_indexes: list[int] = []
298
- try:
299
- for index in selected_indexes:
300
- row = rows[index]
301
- row_for_sheet = {column: display_rows[index].get(column) for column in SHEET_COLUMNS}
302
- append_row_to_google_sheet(row_for_sheet, SHEET_COLUMNS)
303
-
304
- if firebase_tracker is not None:
305
- document_id = str(row.get("_document_id") or "").strip()
306
- source_file = str(row.get("source_file") or "").strip()
307
- modification_date = row.get("_modificationDate")
308
- parsed_data = _build_updated_parsed_data(row)
309
- if document_id:
310
- firebase_tracker.save_document_record(
311
- document_id=document_id,
312
- source_file=source_file,
313
- modification_date=modification_date,
314
- status="sent",
315
- parsed_data=parsed_data,
316
- )
317
-
318
- sent_indexes.append(index)
319
- except GoogleSheetsConfigError as exc:
320
- if not sent_indexes:
321
- return rows, gr.update(value=rows_to_table(rows)), f"Error de configuracion de Google Sheets: {exc}"
322
-
323
- sent_set = set(sent_indexes)
324
- remaining_rows = [row for index, row in enumerate(rows) if index not in sent_set]
325
- return (
326
- remaining_rows,
327
- gr.update(value=rows_to_table(remaining_rows)),
328
- f"Se enviaron {len(sent_indexes)} fila(s) antes de un error de configuracion de Google Sheets: {exc}",
329
- )
330
- except Exception as exc:
331
- if not sent_indexes:
332
- return rows, gr.update(value=rows_to_table(rows)), f"Error al insertar filas en Google Sheets: {exc}"
333
-
334
- sent_set = set(sent_indexes)
335
- remaining_rows = [row for index, row in enumerate(rows) if index not in sent_set]
336
- return (
337
- remaining_rows,
338
- gr.update(value=rows_to_table(remaining_rows)),
339
- f"Se enviaron {len(sent_indexes)} fila(s) antes del fallo: {exc}",
340
- )
341
-
342
- selected_set = set(selected_indexes)
343
- remaining_rows = [row for index, row in enumerate(rows) if index not in selected_set]
344
- return (
345
- remaining_rows,
346
- gr.update(value=rows_to_table(remaining_rows)),
347
- f"Se enviaron {len(selected_indexes)} fila(s) a Google Sheets y se actualizaron en Firebase con estado sent.",
348
- )
349
-
350
-
351
  def run_orchestrator_endpoint(model: str, include_subfolders: bool, send_to_sheet: bool, api_key: str) -> dict:
 
352
  expected_api_key = str(os.getenv("ORCHESTRATOR_API_KEY") or "").strip()
353
  provided_api_key = str(api_key or "").strip()
354
 
@@ -360,7 +41,8 @@ def run_orchestrator_endpoint(model: str, include_subfolders: bool, send_to_shee
360
 
361
  model_name = str(model or "").strip() or os.getenv("OPENAI_MODEL", "gpt-4o")
362
  try:
363
- return orchestrate_drive_documents(
 
364
  model=model_name,
365
  include_subfolders=bool(include_subfolders),
366
  send_to_sheet=bool(send_to_sheet),
@@ -373,85 +55,36 @@ def run_orchestrator_endpoint(model: str, include_subfolders: bool, send_to_shee
373
 
374
 
375
  def build_app() -> gr.Blocks:
376
- initial_rows, _, initial_status = load_pending_documents()
377
-
378
  with gr.Blocks(title="Procesador de Documentos") as demo:
379
  gr.Markdown("# Procesador de Comprobantes de Compra\nRevisa documentos Parsed/Modified y envia los seleccionados.")
380
 
381
- status = gr.Markdown(initial_status)
382
- results_state = gr.State(initial_rows)
383
-
384
- with gr.Row():
385
- refresh_button = gr.Button("Recargar pendientes", variant="primary")
386
- clear_button = gr.Button("Limpiar grilla")
387
- send_row_button = gr.Button("Enviar", variant="secondary")
388
- delete_row_button = gr.Button("Eliminar seleccionados", variant="stop")
389
-
390
- with gr.Accordion("Orquestador API", open=False):
391
- gr.Markdown("Endpoint publico para ejecutar escaneo + parseo + persistencia del orquestador.")
392
- orchestrator_model = gr.Textbox(
393
  label="Modelo OpenAI",
394
  value=os.getenv("OPENAI_MODEL", "gpt-4o"),
395
  )
396
- orchestrator_subfolders = gr.Checkbox(
397
  label="Incluir subcarpetas",
398
  value=True,
399
  )
400
- orchestrator_send = gr.Checkbox(
401
  label="Enviar a Google Sheets",
402
- value=False,
403
  )
404
- orchestrator_api_key = gr.Textbox(
405
  label="API Key",
406
  type="password",
407
  value="",
408
  )
409
- orchestrator_button = gr.Button("Ejecutar orquestador", variant="primary")
410
- orchestrator_output = gr.JSON(label="Resultado orquestacion")
411
-
412
- results_grid = gr.Dataframe(
413
- label="Documentos procesados",
414
- headers=GRID_HEADER_LABELS,
415
- value=rows_to_table(initial_rows),
416
- interactive=True,
417
- wrap=True,
418
- )
419
-
420
- refresh_button.click(
421
- load_pending_documents,
422
- inputs=[],
423
- outputs=[results_state, results_grid, status],
424
- )
425
-
426
- clear_button.click(
427
- clear_results,
428
- inputs=[],
429
- outputs=[results_state, results_grid, status],
430
- )
431
-
432
- results_grid.change(
433
- sync_grid_to_state,
434
- inputs=[results_state, results_grid],
435
- outputs=[results_state, status],
436
- )
437
-
438
- delete_row_button.click(
439
- delete_selected_rows,
440
- inputs=[results_state, results_grid],
441
- outputs=[results_state, results_grid, status],
442
- )
443
-
444
- send_row_button.click(
445
- send_selected_rows,
446
- inputs=[results_state, results_grid],
447
- outputs=[results_state, results_grid, status],
448
- )
449
 
450
- orchestrator_button.click(
451
  run_orchestrator_endpoint,
452
- inputs=[orchestrator_model, orchestrator_subfolders, orchestrator_send, orchestrator_api_key],
453
- outputs=[orchestrator_output],
454
- api_name="orchestrate_drive_documents",
455
  )
456
 
457
  return demo
 
1
  from __future__ import annotations
2
 
3
  import os
 
4
 
5
  # Avoid the Node.js SSR proxy in hosted environments where it can terminate unexpectedly.
6
  os.environ["GRADIO_SSR_MODE"] = "false"
7
 
8
  import gradio as gr
9
 
10
+ from document_orchester import orchestrate_all_active_tenants
 
11
  from google_sheets_service import GoogleSheetsConfigError, append_row_to_google_sheet
12
 
13
  try:
 
15
  except ImportError:
16
  spaces = None
17
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
18
  custom_css = """
19
  .cell-menu, .cell-menu-button {
20
  display: none !important;
 
27
  return fn
28
  return spaces.GPU(fn)
29
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
30
  @gpu_decorator
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
31
  def run_orchestrator_endpoint(model: str, include_subfolders: bool, send_to_sheet: bool, api_key: str) -> dict:
32
+ """Multi-tenant orchestration endpoint - processes all active tenants and their rules."""
33
  expected_api_key = str(os.getenv("ORCHESTRATOR_API_KEY") or "").strip()
34
  provided_api_key = str(api_key or "").strip()
35
 
 
41
 
42
  model_name = str(model or "").strip() or os.getenv("OPENAI_MODEL", "gpt-4o")
43
  try:
44
+ # Load all active tenants from Firebase
45
+ return orchestrate_all_active_tenants(
46
  model=model_name,
47
  include_subfolders=bool(include_subfolders),
48
  send_to_sheet=bool(send_to_sheet),
 
55
 
56
 
57
  def build_app() -> gr.Blocks:
 
 
58
  with gr.Blocks(title="Procesador de Documentos") as demo:
59
  gr.Markdown("# Procesador de Comprobantes de Compra\nRevisa documentos Parsed/Modified y envia los seleccionados.")
60
 
61
+ with gr.Accordion("Orquestador API - Multi Tenant", open=False):
62
+ gr.Markdown("Endpoint publico para ejecutar escaneo + parseo + persistencia del orquestador en modo multi-tenant (carga todos los tenants activos de Firebase).")
63
+ mt_orchestrator_model = gr.Textbox(
 
 
 
 
 
 
 
 
 
64
  label="Modelo OpenAI",
65
  value=os.getenv("OPENAI_MODEL", "gpt-4o"),
66
  )
67
+ mt_orchestrator_subfolders = gr.Checkbox(
68
  label="Incluir subcarpetas",
69
  value=True,
70
  )
71
+ mt_orchestrator_send = gr.Checkbox(
72
  label="Enviar a Google Sheets",
73
+ value=True,
74
  )
75
+ mt_orchestrator_api_key = gr.Textbox(
76
  label="API Key",
77
  type="password",
78
  value="",
79
  )
80
+ mt_orchestrator_button = gr.Button("Ejecutar orquestador (Multi Tenant)", variant="primary")
81
+ mt_orchestrator_output = gr.JSON(label="Resultado orquestacion")
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
82
 
83
+ mt_orchestrator_button.click(
84
  run_orchestrator_endpoint,
85
+ inputs=[mt_orchestrator_model, mt_orchestrator_subfolders, mt_orchestrator_send, mt_orchestrator_api_key],
86
+ outputs=[mt_orchestrator_output],
87
+ api_name="orchestrate_all_tenants",
88
  )
89
 
90
  return demo
document_orchester.py CHANGED
@@ -5,7 +5,7 @@ import json
5
  import logging
6
  import os
7
  import shutil
8
- from datetime import datetime
9
  from pathlib import Path
10
  from typing import Any
11
 
@@ -17,14 +17,18 @@ from firebase_processed_files import (
17
  build_firebase_tracker,
18
  check_drive_documents_to_process,
19
  )
 
 
 
 
 
 
20
  from google_drive_service import (
21
  DRIVE_FULL_SCOPE,
22
  GoogleDriveConfigError,
23
  build_drive_service,
24
  move_file_to_path,
25
- )
26
- from google_drive_scanner import (
27
- scan_google_drive_supported_documents,
28
  )
29
  from google_sheets_service import GoogleSheetsConfigError, append_row_to_google_sheet
30
  from receipt_ai import extract_receipt_json, extract_receipt_json_from_image
@@ -45,14 +49,6 @@ logger.propagate = False
45
 
46
  SHEET_COLUMNS = ["fecha", "source_file", "display_description", "total"]
47
  CORRUPTED_SHEET_NAME = "corrupted data"
48
- MANDATORY_ENV_VARS = [
49
- "OPENAI_API_KEY",
50
- "GOOGLE_SERVICE_ACCOUNT_JSON",
51
- "GOOGLE_SHEETS_SPREADSHEET_ID",
52
- "GOOGLE_DRIVE_FOLDER_IDS",
53
- "GOOGLE_DRIVE_INVOICES_ROOT_FOLDER_ID",
54
- "ORCHESTRATOR_API_KEY",
55
- ]
56
 
57
 
58
  def _build_display_description(parsed: dict[str, Any]) -> str:
@@ -86,26 +82,13 @@ def _is_complete_sheet_row(row: dict[str, Any]) -> bool:
86
  return all(not _is_blank_sheet_value(row.get(column)) for column in SHEET_COLUMNS)
87
 
88
 
89
- def _create_openai_client() -> OpenAI:
90
- api_key = (os.getenv("OPENAI_API_KEY") or "").strip()
91
- if not api_key:
92
  raise RuntimeError("Missing OPENAI_API_KEY.")
93
  return OpenAI(api_key=api_key)
94
 
95
 
96
- def _require_env_var(name: str) -> str:
97
- value = (os.getenv(name) or "").strip()
98
- if not value:
99
- raise RuntimeError(f"Missing {name}.")
100
- return value
101
-
102
-
103
- def _validate_mandatory_env_vars() -> None:
104
- missing_vars = [name for name in MANDATORY_ENV_VARS if not (os.getenv(name) or "").strip()]
105
- if missing_vars:
106
- raise RuntimeError(f"Missing mandatory env vars: {', '.join(missing_vars)}")
107
-
108
-
109
  def _parse_invoice_date(value: Any) -> datetime | None:
110
  text = str(value or "").strip()
111
  if not text:
@@ -162,35 +145,160 @@ def _parse_document(client: OpenAI, model: str, local_path: str, source_file: st
162
  return parsed
163
 
164
 
165
- def orchestrate_drive_documents(
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
166
  model: str,
167
  include_subfolders: bool = True,
168
- send_to_sheet: bool = False,
169
  ) -> dict[str, Any]:
 
 
 
 
 
 
 
 
 
 
 
 
 
170
  logger.info(
171
- "Starting orchestration model=%s include_subfolders=%s send_to_sheet=%s",
 
 
172
  model,
173
- include_subfolders,
174
- send_to_sheet,
175
  )
176
- _validate_mandatory_env_vars()
177
- tracker = build_firebase_tracker()
178
- client = _create_openai_client()
179
- drive_root_folder_id = _require_env_var("GOOGLE_DRIVE_INVOICES_ROOT_FOLDER_ID")
180
-
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
181
  scan_temp_dir: str | None = None
182
  try:
 
 
183
  drive_service = build_drive_service(scope=DRIVE_FULL_SCOPE)
184
- drive_documents, scan_temp_dir = scan_google_drive_supported_documents(include_subfolders=include_subfolders)
 
 
 
 
 
 
 
185
  except GoogleDriveConfigError as exc:
186
- logger.error("Google Drive configuration error: %s", exc)
187
  return {
188
  "ok": False,
189
  "error": f"Google Drive configuration error: {exc}",
 
 
190
  }
191
-
192
- logger.info("Scan completed scanned_documents=%s", len(drive_documents))
193
-
 
 
 
 
 
 
194
  docs_payload = [
195
  {
196
  "document_id": doc.document_id,
@@ -200,12 +308,20 @@ def orchestrate_drive_documents(
200
  }
201
  for doc in drive_documents
202
  ]
203
-
 
204
  to_process, skipped, tracking_warning = check_drive_documents_to_process(tracker, docs_payload)
205
- logger.info("Documents to process=%s skipped=%s", len(to_process), len(skipped))
 
 
 
 
 
 
206
  if tracking_warning:
207
- logger.warning("Tracking warning: %s", tracking_warning)
208
-
 
209
  parsed_count = 0
210
  modified_count = 0
211
  sent_count = 0
@@ -213,103 +329,148 @@ def orchestrate_drive_documents(
213
  corrupted_count = 0
214
  error_items: list[dict[str, str]] = []
215
  processed_items: list[dict[str, Any]] = []
216
-
217
  for document in to_process:
218
  document_id = str(document.get("document_id") or "").strip()
219
  source_file = str(document.get("source_file") or "").strip()
220
  local_path = str(document.get("local_path") or "").strip()
221
  modification_date = document.get("modificationDate")
222
  status = str(document.get("target_status") or "Parsed")
223
- logger.info("Processing document_id=%s source_file=%s status=%s", document_id, source_file, status)
224
-
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
225
  try:
226
  parsed = _parse_document(client, model, local_path, source_file)
227
- logger.info("Parsed document_id=%s source_file=%s", document_id, source_file)
228
-
229
- destination_path, is_corrupted = _resolve_drive_destination_path(parsed)
230
- if is_corrupted:
231
- corrupted_count += 1
232
- move_file_to_path(
233
- service=drive_service,
234
- file_id=document_id,
235
- destination_path=destination_path,
236
- root_folder_id=drive_root_folder_id,
237
- )
238
- moved_count += 1
239
  logger.info(
240
- "Moved document_id=%s source_file=%s to drive_path=%s",
 
 
241
  document_id,
242
- source_file,
243
- destination_path,
244
  )
245
-
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
246
  if tracker is not None:
247
  tracker.save_document_record(
 
248
  document_id=document_id,
249
  source_file=source_file,
250
  modification_date=modification_date,
251
  status=status,
252
  parsed_data=parsed,
253
  )
254
-
255
  if status == "Modified":
256
  modified_count += 1
257
  else:
258
  parsed_count += 1
259
-
 
260
  worksheet_name: str | None = None
261
- if send_to_sheet:
 
262
  row_for_sheet = _build_sheet_row(parsed)
263
  row_for_sheet["source_file"] = _build_hyperlink_formula(
264
  _build_drive_file_url(document_id),
265
  source_file,
266
  )
267
- worksheet_name = None if _is_complete_sheet_row(row_for_sheet) else CORRUPTED_SHEET_NAME
268
- append_row_to_google_sheet(row_for_sheet, SHEET_COLUMNS, worksheet_name=worksheet_name)
269
- logger.info(
270
- "Sent document_id=%s source_file=%s worksheet=%s",
271
- document_id,
272
- source_file,
273
- worksheet_name or "main",
274
- )
275
- if tracker is not None and worksheet_name is None:
276
- tracker.mark_document_sent(document_id)
277
- sent_count += 1
278
-
279
- processed_items.append(
280
- {
281
- "document_id": document_id,
282
- "source_file": source_file,
283
- "modificationDate": modification_date,
284
- "status": "Corrupted" if is_corrupted else ("Sent" if send_to_sheet else status),
285
- "destination_path": destination_path,
286
- }
287
- )
288
- except GoogleSheetsConfigError as exc:
289
- logger.error("Google Sheets config error for document_id=%s source_file=%s: %s", document_id, source_file, exc)
290
- error_items.append(
291
- {
292
- "document_id": document_id,
293
- "source_file": source_file,
294
- "error": f"Google Sheets config error: {exc}",
295
- }
296
- )
 
 
 
 
 
 
 
297
  except Exception as exc:
298
- logger.exception("Failed processing document_id=%s source_file=%s", document_id, source_file)
299
- error_items.append(
300
- {
301
- "document_id": document_id,
302
- "source_file": source_file,
303
- "error": str(exc),
304
- }
305
  )
306
-
 
 
 
 
 
 
307
  if scan_temp_dir:
308
  shutil.rmtree(scan_temp_dir, ignore_errors=True)
309
- logger.info("Cleaned up temporary scan directory")
310
-
311
  logger.info(
312
- "Orchestration finished parsed=%s modified=%s sent=%s moved=%s corrupted=%s errors=%s",
 
 
313
  parsed_count,
314
  modified_count,
315
  sent_count,
@@ -317,9 +478,12 @@ def orchestrate_drive_documents(
317
  corrupted_count,
318
  len(error_items),
319
  )
320
-
321
  return {
322
  "ok": True,
 
 
 
323
  "scanned": len(drive_documents),
324
  "to_process": len(to_process),
325
  "skipped": len(skipped),
@@ -335,9 +499,200 @@ def orchestrate_drive_documents(
335
  }
336
 
337
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
338
  def parse_args() -> argparse.Namespace:
339
- parser = argparse.ArgumentParser(description="Orchestrate Drive documents: scan, parse, and persist status.")
340
- parser.add_argument("--model", default=os.getenv("OPENAI_MODEL", "gpt-4o"), help="OpenAI model name.")
 
 
 
 
 
 
341
  parser.add_argument(
342
  "--no-subfolders",
343
  action="store_true",
@@ -358,8 +713,13 @@ def parse_args() -> argparse.Namespace:
358
 
359
  def main() -> int:
360
  args = parse_args()
361
- logger.info("Running document orchestration CLI output=%s", args.output or "stdout")
362
- summary = orchestrate_drive_documents(
 
 
 
 
 
363
  model=args.model,
364
  include_subfolders=not args.no_subfolders,
365
  send_to_sheet=args.send,
 
5
  import logging
6
  import os
7
  import shutil
8
+ from datetime import datetime, timezone
9
  from pathlib import Path
10
  from typing import Any
11
 
 
17
  build_firebase_tracker,
18
  check_drive_documents_to_process,
19
  )
20
+ from firebase_tenant_config import (
21
+ FirebaseTenantConfigManager,
22
+ CredentialsObject,
23
+ RuleObject,
24
+ TenantConfig,
25
+ )
26
  from google_drive_service import (
27
  DRIVE_FULL_SCOPE,
28
  GoogleDriveConfigError,
29
  build_drive_service,
30
  move_file_to_path,
31
+ scan_drive_supported_documents,
 
 
32
  )
33
  from google_sheets_service import GoogleSheetsConfigError, append_row_to_google_sheet
34
  from receipt_ai import extract_receipt_json, extract_receipt_json_from_image
 
49
 
50
  SHEET_COLUMNS = ["fecha", "source_file", "display_description", "total"]
51
  CORRUPTED_SHEET_NAME = "corrupted data"
 
 
 
 
 
 
 
 
52
 
53
 
54
  def _build_display_description(parsed: dict[str, Any]) -> str:
 
82
  return all(not _is_blank_sheet_value(row.get(column)) for column in SHEET_COLUMNS)
83
 
84
 
85
+ def _create_openai_client_from_key(api_key: str) -> OpenAI:
86
+ """Create OpenAI client from API key."""
87
+ if not api_key.strip():
88
  raise RuntimeError("Missing OPENAI_API_KEY.")
89
  return OpenAI(api_key=api_key)
90
 
91
 
 
 
 
 
 
 
 
 
 
 
 
 
 
92
  def _parse_invoice_date(value: Any) -> datetime | None:
93
  text = str(value or "").strip()
94
  if not text:
 
145
  return parsed
146
 
147
 
148
+ def _create_default_tenant_from_env() -> tuple[TenantConfig, RuleObject] | None:
149
+ """Create a default tenant from environment variables if possible.
150
+
151
+ Maps legacy env vars to a default multi-tenant configuration:
152
+ - OPENAI_API_KEY -> credentials.openai_api_key
153
+ - GOOGLE_SERVICE_ACCOUNT_JSON -> credentials.google_service_account_json
154
+ - GOOGLE_DRIVE_FOLDER_IDS -> rule.source_folder_id (first one)
155
+ - GOOGLE_DRIVE_INVOICES_ROOT_FOLDER_ID -> rule.target_folder_id
156
+ - GOOGLE_SHEETS_SPREADSHEET_ID -> rule.target_sheet_id
157
+
158
+ Returns:
159
+ TenantConfig if all required env vars are set, None otherwise
160
+ """
161
+ openai_api_key = (os.getenv("OPENAI_API_KEY") or "").strip()
162
+ google_service_account_json = (os.getenv("GOOGLE_SERVICE_ACCOUNT_JSON") or "").strip()
163
+ drive_folder_ids = (os.getenv("GOOGLE_DRIVE_FOLDER_IDS") or "").strip()
164
+ target_folder_id = (os.getenv("GOOGLE_DRIVE_INVOICES_ROOT_FOLDER_ID") or "").strip()
165
+ target_sheet_id = (os.getenv("GOOGLE_SHEETS_SPREADSHEET_ID") or "").strip()
166
+ sheet_tab_name = (os.getenv("GOOGLE_SHEETS_WORKSHEET") or "Salidas").strip()
167
+
168
+ # Check if all required vars are set
169
+ if not all([openai_api_key, google_service_account_json, drive_folder_ids, target_folder_id, target_sheet_id]):
170
+ logger.warning(
171
+ "Cannot create default tenant: missing env vars. "
172
+ "Please ensure OPENAI_API_KEY, GOOGLE_SERVICE_ACCOUNT_JSON, GOOGLE_DRIVE_FOLDER_IDS, "
173
+ "GOOGLE_DRIVE_INVOICES_ROOT_FOLDER_ID, and GOOGLE_SHEETS_SPREADSHEET_ID are set."
174
+ )
175
+ return None
176
+
177
+ # Get first folder ID from comma-separated list
178
+ source_folder_id = drive_folder_ids.split(",")[0].strip()
179
+
180
+ # Create credentials
181
+ credentials = CredentialsObject(
182
+ openai_api_key=openai_api_key,
183
+ google_service_account_json=google_service_account_json,
184
+ )
185
+
186
+ # Create default rule
187
+ default_rule = RuleObject(
188
+ rule_id="default_rule",
189
+ rule_name="Default Processing Rule",
190
+ source_folder_id=source_folder_id,
191
+ target_folder_id=target_folder_id,
192
+ target_sheet_id=target_sheet_id,
193
+ sheet_tab_name=sheet_tab_name,
194
+ parsing_prompt=None,
195
+ is_enabled=True,
196
+ )
197
+
198
+ # Create tenant (rules will be added as subcollection)
199
+ tenant = TenantConfig(
200
+ tenant_id="default_tenant",
201
+ name="Default Tenant (from .env)",
202
+ active=True,
203
+ credentials=credentials,
204
+ created_at=datetime.now(timezone.utc).isoformat(),
205
+ )
206
+
207
+ logger.info("Created default tenant from environment variables")
208
+ return tenant, default_rule
209
+
210
+
211
+ def orchestrate_single_rule(
212
+ tenant_id: str,
213
+ tenant_config: TenantConfig,
214
+ rule: RuleObject,
215
  model: str,
216
  include_subfolders: bool = True,
217
+ send_to_sheet: bool = True,
218
  ) -> dict[str, Any]:
219
+ """Orchestrate document processing for a single rule of a single tenant.
220
+
221
+ Args:
222
+ tenant_id: Unique tenant identifier
223
+ tenant_config: Tenant configuration (with credentials and rules)
224
+ rule: The processing rule to orchestrate
225
+ model: OpenAI model to use for parsing
226
+ include_subfolders: Whether to include subfolders when scanning Drive
227
+ send_to_sheet: Whether to append results to Google Sheets
228
+
229
+ Returns:
230
+ Summary dict with processing results
231
+ """
232
  logger.info(
233
+ "Starting orchestration for tenant=%s rule=%s model=%s",
234
+ tenant_id,
235
+ rule.rule_id,
236
  model,
 
 
237
  )
238
+
239
+ # Validate tenant credentials
240
+ if not tenant_config.credentials.openai_api_key:
241
+ logger.error("Tenant=%s rule=%s missing openai_api_key", tenant_id, rule.rule_id)
242
+ return {
243
+ "ok": False,
244
+ "error": f"Tenant {tenant_id} missing OpenAI API key",
245
+ "tenant_id": tenant_id,
246
+ "rule_id": rule.rule_id,
247
+ }
248
+
249
+ if not tenant_config.credentials.google_service_account_json:
250
+ logger.error("Tenant=%s rule=%s missing google_service_account_json", tenant_id, rule.rule_id)
251
+ return {
252
+ "ok": False,
253
+ "error": f"Tenant {tenant_id} missing Google service account credentials",
254
+ "tenant_id": tenant_id,
255
+ "rule_id": rule.rule_id,
256
+ }
257
+
258
+ # Set up clients for this tenant
259
+ try:
260
+ client = _create_openai_client_from_key(tenant_config.credentials.openai_api_key)
261
+ tracker = build_firebase_tracker(tenant_id=tenant_id, rule_id=rule.rule_id)
262
+ except RuntimeError as exc:
263
+ logger.error("Tenant=%s rule=%s failed to initialize clients: %s", tenant_id, rule.rule_id, exc)
264
+ return {
265
+ "ok": False,
266
+ "error": str(exc),
267
+ "tenant_id": tenant_id,
268
+ "rule_id": rule.rule_id,
269
+ }
270
+
271
+ # Set up Google Drive service with tenant's credentials
272
  scan_temp_dir: str | None = None
273
  try:
274
+ # Build drive service with tenant credentials
275
+ os.environ["GOOGLE_SERVICE_ACCOUNT_JSON"] = tenant_config.credentials.google_service_account_json
276
  drive_service = build_drive_service(scope=DRIVE_FULL_SCOPE)
277
+
278
+ # Scan the rule's source folder
279
+ scan_result = scan_drive_supported_documents(
280
+ folder_ids=[rule.source_folder_id],
281
+ include_subfolders=include_subfolders,
282
+ )
283
+ drive_documents = scan_result.documents
284
+ scan_temp_dir = scan_result.temp_dir
285
  except GoogleDriveConfigError as exc:
286
+ logger.error("Tenant=%s rule=%s Google Drive config error: %s", tenant_id, rule.rule_id, exc)
287
  return {
288
  "ok": False,
289
  "error": f"Google Drive configuration error: {exc}",
290
+ "tenant_id": tenant_id,
291
+ "rule_id": rule.rule_id,
292
  }
293
+
294
+ logger.info(
295
+ "Tenant=%s rule=%s scan completed documents_found=%s",
296
+ tenant_id,
297
+ rule.rule_id,
298
+ len(drive_documents),
299
+ )
300
+
301
+ # Prepare docs payload
302
  docs_payload = [
303
  {
304
  "document_id": doc.document_id,
 
308
  }
309
  for doc in drive_documents
310
  ]
311
+
312
+ # Check which documents need processing
313
  to_process, skipped, tracking_warning = check_drive_documents_to_process(tracker, docs_payload)
314
+ logger.info(
315
+ "Tenant=%s rule=%s to_process=%s skipped=%s",
316
+ tenant_id,
317
+ rule.rule_id,
318
+ len(to_process),
319
+ len(skipped),
320
+ )
321
  if tracking_warning:
322
+ logger.warning("Tenant=%s rule=%s tracking warning: %s", tenant_id, rule.rule_id, tracking_warning)
323
+
324
+ # Process documents
325
  parsed_count = 0
326
  modified_count = 0
327
  sent_count = 0
 
329
  corrupted_count = 0
330
  error_items: list[dict[str, str]] = []
331
  processed_items: list[dict[str, Any]] = []
332
+
333
  for document in to_process:
334
  document_id = str(document.get("document_id") or "").strip()
335
  source_file = str(document.get("source_file") or "").strip()
336
  local_path = str(document.get("local_path") or "").strip()
337
  modification_date = document.get("modificationDate")
338
  status = str(document.get("target_status") or "Parsed")
339
+ file_hash = str(document.get("file_hash") or "").strip() or None
340
+
341
+ logger.info(
342
+ "Tenant=%s rule=%s processing document_id=%s source_file=%s",
343
+ tenant_id,
344
+ rule.rule_id,
345
+ document_id,
346
+ source_file,
347
+ )
348
+
349
+ if tracker.is_processed(file_hash):
350
+ logger.info(
351
+ "Tenant=%s rule=%s skipping document_id=%s already processed",
352
+ tenant_id,
353
+ rule.rule_id,
354
+ document_id,
355
+ )
356
+ skipped.append({"document_id": document_id, "source_file": source_file, "reason": "Already processed"})
357
+ continue
358
+
359
  try:
360
  parsed = _parse_document(client, model, local_path, source_file)
 
 
 
 
 
 
 
 
 
 
 
 
361
  logger.info(
362
+ "Tenant=%s rule=%s parsed document_id=%s",
363
+ tenant_id,
364
+ rule.rule_id,
365
  document_id,
 
 
366
  )
367
+
368
+ # Determine destination and whether corrupted
369
+ destination_path, is_corrupted = _resolve_drive_destination_path(parsed)
370
+ if is_corrupted:
371
+ corrupted_count += 1
372
+
373
+ # Move file to destination (if target_folder_id is set)
374
+ if rule.target_folder_id:
375
+ move_file_to_path(
376
+ service=drive_service,
377
+ file_id=document_id,
378
+ destination_path=destination_path,
379
+ root_folder_id=rule.target_folder_id,
380
+ )
381
+ moved_count += 1
382
+ logger.info(
383
+ "Tenant=%s rule=%s moved document_id=%s to %s",
384
+ tenant_id,
385
+ rule.rule_id,
386
+ document_id,
387
+ destination_path,
388
+ )
389
+
390
+ # Save document record
391
  if tracker is not None:
392
  tracker.save_document_record(
393
+ file_hash=file_hash,
394
  document_id=document_id,
395
  source_file=source_file,
396
  modification_date=modification_date,
397
  status=status,
398
  parsed_data=parsed,
399
  )
400
+
401
  if status == "Modified":
402
  modified_count += 1
403
  else:
404
  parsed_count += 1
405
+
406
+ # Send to Google Sheets if enabled
407
  worksheet_name: str | None = None
408
+ if rule.target_sheet_id:
409
+ os.environ["GOOGLE_SHEETS_SPREADSHEET_ID"] = rule.target_sheet_id
410
  row_for_sheet = _build_sheet_row(parsed)
411
  row_for_sheet["source_file"] = _build_hyperlink_formula(
412
  _build_drive_file_url(document_id),
413
  source_file,
414
  )
415
+ worksheet_name = "Corrupted_data" if not _is_complete_sheet_row(row_for_sheet) else rule.sheet_tab_name
416
+
417
+ try:
418
+ append_row_to_google_sheet(row_for_sheet, SHEET_COLUMNS, worksheet_name=worksheet_name)
419
+ logger.info(
420
+ "Tenant=%s rule=%s sent document_id=%s to sheet=%s",
421
+ tenant_id,
422
+ rule.rule_id,
423
+ document_id,
424
+ worksheet_name,
425
+ )
426
+ if tracker is not None and file_hash:
427
+ tracker.mark_document_sent(file_hash)
428
+ sent_count += 1
429
+ except GoogleSheetsConfigError as exc:
430
+ logger.error(
431
+ "Tenant=%s rule=%s sheet error for document_id=%s: %s",
432
+ tenant_id,
433
+ rule.rule_id,
434
+ document_id,
435
+ exc,
436
+ )
437
+ error_items.append({
438
+ "document_id": document_id,
439
+ "source_file": source_file,
440
+ "error": f"Google Sheets error: {exc}",
441
+ })
442
+ continue
443
+
444
+ processed_items.append({
445
+ "document_id": document_id,
446
+ "source_file": source_file,
447
+ "modificationDate": modification_date,
448
+ "status": "Corrupted" if is_corrupted else ("Sent" if send_to_sheet else status),
449
+ "destination_path": destination_path,
450
+ })
451
+
452
  except Exception as exc:
453
+ logger.exception(
454
+ "Tenant=%s rule=%s failed processing document_id=%s",
455
+ tenant_id,
456
+ rule.rule_id,
457
+ document_id,
 
 
458
  )
459
+ error_items.append({
460
+ "document_id": document_id,
461
+ "source_file": source_file,
462
+ "error": str(exc),
463
+ })
464
+
465
+ # Cleanup
466
  if scan_temp_dir:
467
  shutil.rmtree(scan_temp_dir, ignore_errors=True)
468
+ logger.info("Tenant=%s rule=%s cleaned up temp directory", tenant_id, rule.rule_id)
469
+
470
  logger.info(
471
+ "Tenant=%s rule=%s orchestration finished parsed=%s modified=%s sent=%s moved=%s corrupted=%s errors=%s",
472
+ tenant_id,
473
+ rule.rule_id,
474
  parsed_count,
475
  modified_count,
476
  sent_count,
 
478
  corrupted_count,
479
  len(error_items),
480
  )
481
+
482
  return {
483
  "ok": True,
484
+ "tenant_id": tenant_id,
485
+ "rule_id": rule.rule_id,
486
+ "rule_name": rule.rule_name,
487
  "scanned": len(drive_documents),
488
  "to_process": len(to_process),
489
  "skipped": len(skipped),
 
499
  }
500
 
501
 
502
+ def orchestrate_all_active_tenants(
503
+ model: str,
504
+ include_subfolders: bool = True,
505
+ send_to_sheet: bool = True,
506
+ ) -> dict[str, Any]:
507
+ """Orchestrate all active tenants and their enabled rules.
508
+
509
+ This is the main multi-tenant entry point that:
510
+ 1. Loads all active tenants from Firebase
511
+ 2. For each tenant, processes all enabled rules
512
+ 3. Aggregates results across all tenants
513
+
514
+ Args:
515
+ model: OpenAI model to use for parsing
516
+ include_subfolders: Whether to include subfolders when scanning Drive
517
+ send_to_sheet: Whether to append results to Google Sheets
518
+
519
+ Returns:
520
+ Summary dict with aggregated processing results
521
+ """
522
+ logger.info(
523
+ "Starting multi-tenant orchestration model=%s include_subfolders=%s send_to_sheet=%s",
524
+ model,
525
+ include_subfolders,
526
+ send_to_sheet,
527
+ )
528
+
529
+ # Load all active tenants from Firebase
530
+ config_manager = FirebaseTenantConfigManager()
531
+ default_tenant_created = False
532
+ default_tenant_saved = False
533
+
534
+ try:
535
+ active_tenants = config_manager.list_active_tenants()
536
+ except Exception as exc:
537
+ logger.error("Failed to load active tenants: %s", exc)
538
+ return {
539
+ "ok": False,
540
+ "error": f"Failed to load tenants: {exc}",
541
+ "tenants_processed": 0,
542
+ }
543
+
544
+ logger.info("Loaded %s active tenants", len(active_tenants))
545
+
546
+ # If no tenants found, create and save default tenant BEFORE orchestration
547
+ if not active_tenants:
548
+ logger.warning("No active tenants found. Creating default tenant from .env...")
549
+ result = _create_default_tenant_from_env()
550
+
551
+ if not result:
552
+ return {
553
+ "ok": False,
554
+ "error": "No active tenants found in Firebase and could not create default tenant from .env",
555
+ "tenants_processed": 0,
556
+ "tenants_results": [],
557
+ }
558
+
559
+ default_tenant, default_rule = result
560
+
561
+ # Save default tenant to Firebase BEFORE orchestration
562
+ try:
563
+ logger.info("Saving default tenant '%s' to Firebase...", default_tenant.tenant_id)
564
+ config_manager.save_tenant(default_tenant)
565
+ # Add the default rule to the tenant's rules subcollection
566
+ config_manager.add_rule(default_tenant.tenant_id, default_rule)
567
+ logger.info("✓ Successfully saved default tenant and rule to Firebase: %s", default_tenant.tenant_id)
568
+ default_tenant_saved = True
569
+ except Exception as exc:
570
+ logger.error("✗ Failed to save default tenant to Firebase: %s", exc, exc_info=True)
571
+ return {
572
+ "ok": False,
573
+ "error": f"Failed to save default tenant to Firebase: {exc}",
574
+ "tenants_processed": 0,
575
+ "tenants_results": [],
576
+ }
577
+
578
+ # Use the default tenant for orchestration
579
+ active_tenants = [default_tenant]
580
+ default_tenant_created = True
581
+ logger.info("Using default tenant created and saved from environment variables")
582
+
583
+ # Process each tenant and their rules
584
+ tenants_results = []
585
+ total_parsed = 0
586
+ total_modified = 0
587
+ total_sent = 0
588
+ total_moved = 0
589
+ total_corrupted = 0
590
+ total_errors = 0
591
+
592
+ for tenant in active_tenants:
593
+ tenant_id = tenant.tenant_id
594
+ logger.info("Processing tenant=%s name=%s", tenant_id, tenant.name)
595
+
596
+ # Get enabled rules for this tenant from the rules subcollection
597
+ enabled_rules = config_manager.get_enabled_rules(tenant_id)
598
+ logger.info("Tenant=%s has %s enabled rules", tenant_id, len(enabled_rules))
599
+
600
+ if not enabled_rules:
601
+ logger.warning("Tenant=%s has no enabled rules", tenant_id)
602
+ tenants_results.append({
603
+ "tenant_id": tenant_id,
604
+ "tenant_name": tenant.name,
605
+ "ok": True,
606
+ "warning": "No enabled rules",
607
+ "rules_processed": 0,
608
+ "rules_results": [],
609
+ })
610
+ continue
611
+
612
+ # Process each rule
613
+ rules_results = []
614
+ tenant_parsed = 0
615
+ tenant_modified = 0
616
+ tenant_sent = 0
617
+ tenant_moved = 0
618
+ tenant_corrupted = 0
619
+ tenant_errors = 0
620
+
621
+ for rule in enabled_rules:
622
+ rule_result = orchestrate_single_rule(
623
+ tenant_id=tenant_id,
624
+ tenant_config=tenant,
625
+ rule=rule,
626
+ model=model,
627
+ include_subfolders=include_subfolders,
628
+ send_to_sheet=send_to_sheet,
629
+ )
630
+ rules_results.append(rule_result)
631
+
632
+ if rule_result.get("ok"):
633
+ tenant_parsed += rule_result.get("parsed", 0)
634
+ tenant_modified += rule_result.get("modified", 0)
635
+ tenant_sent += rule_result.get("sent", 0)
636
+ tenant_moved += rule_result.get("moved", 0)
637
+ tenant_corrupted += rule_result.get("corrupted", 0)
638
+ tenant_errors += len(rule_result.get("errors", []))
639
+
640
+ total_parsed += tenant_parsed
641
+ total_modified += tenant_modified
642
+ total_sent += tenant_sent
643
+ total_moved += tenant_moved
644
+ total_corrupted += tenant_corrupted
645
+ total_errors += tenant_errors
646
+
647
+ tenants_results.append({
648
+ "tenant_id": tenant_id,
649
+ "tenant_name": tenant.name,
650
+ "ok": True,
651
+ "rules_processed": len(rules_results),
652
+ "parsed": tenant_parsed,
653
+ "modified": tenant_modified,
654
+ "sent": tenant_sent,
655
+ "moved": tenant_moved,
656
+ "corrupted": tenant_corrupted,
657
+ "errors": tenant_errors,
658
+ "rules_results": rules_results,
659
+ })
660
+
661
+ logger.info(
662
+ "Multi-tenant orchestration completed tenants=%s parsed=%s modified=%s sent=%s moved=%s corrupted=%s errors=%s",
663
+ len(tenants_results),
664
+ total_parsed,
665
+ total_modified,
666
+ total_sent,
667
+ total_moved,
668
+ total_corrupted,
669
+ total_errors,
670
+ )
671
+
672
+ return {
673
+ "ok": True,
674
+ "tenants_processed": len(tenants_results),
675
+ "default_tenant_created": default_tenant_created,
676
+ "default_tenant_saved": default_tenant_saved,
677
+ "total_parsed": total_parsed,
678
+ "total_modified": total_modified,
679
+ "total_sent": total_sent,
680
+ "total_moved": total_moved,
681
+ "total_corrupted": total_corrupted,
682
+ "total_errors": total_errors,
683
+ "tenants_results": tenants_results,
684
+ }
685
+
686
+
687
  def parse_args() -> argparse.Namespace:
688
+ parser = argparse.ArgumentParser(
689
+ description="Orchestrate Drive documents: scan, parse, and persist status using multi-tenant Firebase configuration."
690
+ )
691
+ parser.add_argument(
692
+ "--model",
693
+ default=os.getenv("OPENAI_MODEL", "gpt-4o"),
694
+ help="OpenAI model name."
695
+ )
696
  parser.add_argument(
697
  "--no-subfolders",
698
  action="store_true",
 
713
 
714
  def main() -> int:
715
  args = parse_args()
716
+ logger.info(
717
+ "Running document orchestration CLI in multi-tenant mode output=%s",
718
+ args.output or "stdout",
719
+ )
720
+
721
+ logger.info("Loading all active tenants from Firebase")
722
+ summary = orchestrate_all_active_tenants(
723
  model=args.model,
724
  include_subfolders=not args.no_subfolders,
725
  send_to_sheet=args.send,
firebase_processed_files.py CHANGED
@@ -18,11 +18,21 @@ class FirebaseConfigError(RuntimeError):
18
 
19
 
20
  class FirebaseProcessedFilesTracker:
21
- def __init__(self, collection_name: str | None = None):
22
- self.collection_name = (collection_name or os.getenv("FIREBASE_PROCESSED_COLLECTION") or "processed_files").strip()
23
- self.documents_collection_name = (
24
- os.getenv("FIREBASE_DOCUMENTS_COLLECTION") or "processed_documents"
25
- ).strip()
 
 
 
 
 
 
 
 
 
 
26
  self._db = None
27
 
28
  def _build_credentials(self):
@@ -86,21 +96,23 @@ class FirebaseProcessedFilesTracker:
86
  self._get_db().collection(self.collection_name).document(file_hash).set(payload)
87
 
88
  def get_document_record(self, document_id: str) -> dict[str, Any] | None:
89
- doc = self._get_db().collection(self.documents_collection_name).document(document_id).get()
90
  if not doc.exists:
91
  return None
92
  return doc.to_dict() or {}
93
 
94
  def save_document_record(
95
  self,
 
96
  document_id: str,
97
  source_file: str,
98
  modification_date: str | None,
99
  status: str,
100
- parsed_data: dict[str, Any] | None,
101
  ) -> None:
102
  now = datetime.now(timezone.utc).isoformat()
103
  payload: dict[str, Any] = {
 
104
  "document_id": document_id,
105
  "source_file": source_file,
106
  "modificationDate": modification_date,
@@ -111,11 +123,15 @@ class FirebaseProcessedFilesTracker:
111
  if parsed_data is not None:
112
  payload["parsed_data"] = parsed_data
113
 
114
- existing = self.get_document_record(document_id)
115
  if existing is None:
116
  payload["created_at"] = now
117
 
118
- self._get_db().collection(self.documents_collection_name).document(document_id).set(payload, merge=True)
 
 
 
 
119
 
120
  def mark_document_sent(self, document_id: str) -> None:
121
  now = datetime.now(timezone.utc).isoformat()
@@ -124,14 +140,14 @@ class FirebaseProcessedFilesTracker:
124
  "sent_at": now,
125
  "updated_at": now,
126
  }
127
- self._get_db().collection(self.documents_collection_name).document(document_id).set(payload, merge=True)
128
 
129
  def list_documents_by_statuses(self, statuses: list[str]) -> list[dict[str, Any]]:
130
  unique_statuses = [status for status in dict.fromkeys(statuses) if str(status).strip()]
131
  if not unique_statuses:
132
  return []
133
 
134
- collection = self._get_db().collection(self.documents_collection_name)
135
  documents: list[dict[str, Any]] = []
136
 
137
  try:
@@ -153,12 +169,15 @@ class FirebaseProcessedFilesTracker:
153
  return documents
154
 
155
 
156
- def build_firebase_tracker() -> FirebaseProcessedFilesTracker | None:
 
 
 
157
  enabled = (os.getenv("FIREBASE_TRACK_PROCESSED") or "true").strip().lower()
158
  if enabled in {"0", "false", "no", "off"}:
159
- return None
160
 
161
- return FirebaseProcessedFilesTracker()
162
 
163
 
164
  def check_files_already_processed(
@@ -191,7 +210,14 @@ def check_drive_documents_to_process(
191
  """
192
  Returns (documents_to_process, skipped_documents, warning).
193
 
194
- - New document (no record) -> status "Parsed"
 
 
 
 
 
 
 
195
  - Existing document with modified date changed -> status "Modified"
196
  - Existing document with same modified date -> skipped
197
  """
@@ -206,21 +232,50 @@ def check_drive_documents_to_process(
206
  for document in documents:
207
  document_id = str(document.get("document_id") or "").strip()
208
  modified_time = str(document.get("modificationDate") or "").strip() or None
 
209
 
210
- if not document_id:
211
- skipped.append({**document, "skip_reason": "missing_document_id"})
 
 
 
 
 
 
 
 
 
 
 
212
  continue
213
 
214
- existing = tracker.get_document_record(document_id)
 
215
  if existing is None:
216
- to_process.append({**document, "target_status": "Parsed"})
 
 
 
 
 
 
217
  continue
218
 
 
219
  existing_modified = str(existing.get("modificationDate") or "").strip() or None
220
  if existing_modified != modified_time:
221
- to_process.append({**document, "target_status": "Modified"})
 
 
 
 
 
222
  else:
223
- skipped.append({**document, "skip_reason": "already_processed_same_modification"})
 
 
 
 
224
  except Exception as exc:
225
  docs = [dict(doc, target_status="Parsed") for doc in documents]
226
  return docs, [], f"Firebase tracking unavailable. Processing all Drive files. Details: {exc}"
 
18
 
19
 
20
  class FirebaseProcessedFilesTracker:
21
+ def __init__(
22
+ self,
23
+ tenant_id: str,
24
+ rule_id: str,
25
+ ):
26
+ self.tenant_id = (tenant_id or "").strip()
27
+ self.rule_id = (rule_id or "").strip()
28
+
29
+ if not self.tenant_id:
30
+ raise FirebaseConfigError("tenant_id is required")
31
+ if not self.rule_id:
32
+ raise FirebaseConfigError("rule_id is required")
33
+
34
+ # Multi-tenant rule-based tracking: tenants/{tenant_id}/rules_executions/{rule_id}/processed_documents/{file_hash}
35
+ self.collection_name = f"tenants/{self.tenant_id}/rules/{self.rule_id}/processed_documents"
36
  self._db = None
37
 
38
  def _build_credentials(self):
 
96
  self._get_db().collection(self.collection_name).document(file_hash).set(payload)
97
 
98
  def get_document_record(self, document_id: str) -> dict[str, Any] | None:
99
+ doc = self._get_db().collection(self.collection_name).document(document_id).get()
100
  if not doc.exists:
101
  return None
102
  return doc.to_dict() or {}
103
 
104
  def save_document_record(
105
  self,
106
+ file_hash: str,
107
  document_id: str,
108
  source_file: str,
109
  modification_date: str | None,
110
  status: str,
111
+ parsed_data: dict[str, Any] | None = None,
112
  ) -> None:
113
  now = datetime.now(timezone.utc).isoformat()
114
  payload: dict[str, Any] = {
115
+ "file_hash": file_hash,
116
  "document_id": document_id,
117
  "source_file": source_file,
118
  "modificationDate": modification_date,
 
123
  if parsed_data is not None:
124
  payload["parsed_data"] = parsed_data
125
 
126
+ existing = self.get_document_record(file_hash)
127
  if existing is None:
128
  payload["created_at"] = now
129
 
130
+ # Use file_hash as the document ID in Firebase
131
+ self._get_db().collection(self.collection_name).document(file_hash).set(payload, merge=True)
132
+
133
+ # Also mark the file as processed in the documents collection
134
+ self.mark_processed(file_hash, source_file)
135
 
136
  def mark_document_sent(self, document_id: str) -> None:
137
  now = datetime.now(timezone.utc).isoformat()
 
140
  "sent_at": now,
141
  "updated_at": now,
142
  }
143
+ self._get_db().collection(self.collection_name).document(document_id).set(payload, merge=True)
144
 
145
  def list_documents_by_statuses(self, statuses: list[str]) -> list[dict[str, Any]]:
146
  unique_statuses = [status for status in dict.fromkeys(statuses) if str(status).strip()]
147
  if not unique_statuses:
148
  return []
149
 
150
+ collection = self._get_db().collection(self.collection_name)
151
  documents: list[dict[str, Any]] = []
152
 
153
  try:
 
169
  return documents
170
 
171
 
172
+ def build_firebase_tracker(
173
+ tenant_id: str,
174
+ rule_id: str,
175
+ ) -> FirebaseProcessedFilesTracker:
176
  enabled = (os.getenv("FIREBASE_TRACK_PROCESSED") or "true").strip().lower()
177
  if enabled in {"0", "false", "no", "off"}:
178
+ raise FirebaseConfigError("Firebase tracking is disabled (FIREBASE_TRACK_PROCESSED=false)")
179
 
180
+ return FirebaseProcessedFilesTracker(tenant_id=tenant_id, rule_id=rule_id)
181
 
182
 
183
  def check_files_already_processed(
 
210
  """
211
  Returns (documents_to_process, skipped_documents, warning).
212
 
213
+ Process:
214
+ 1. Compute file hash from file content
215
+ 2. Use hash as unique document identifier
216
+ 3. Check if document (by hash) already exists
217
+ 4. Compare modification dates for changes
218
+
219
+ - File already processed (by hash) -> skipped
220
+ - New document (no hash record) -> status "Parsed"
221
  - Existing document with modified date changed -> status "Modified"
222
  - Existing document with same modified date -> skipped
223
  """
 
232
  for document in documents:
233
  document_id = str(document.get("document_id") or "").strip()
234
  modified_time = str(document.get("modificationDate") or "").strip() or None
235
+ local_path = str(document.get("local_path") or "").strip()
236
 
237
+ if not document_id or not local_path:
238
+ skipped.append({**document, "skip_reason": "missing_document_id_or_path"})
239
+ continue
240
+
241
+ # Compute file hash from file content - this becomes the unique identifier
242
+ try:
243
+ path = Path(local_path)
244
+ if not path.exists():
245
+ skipped.append({**document, "skip_reason": "file_not_found"})
246
+ continue
247
+ file_hash = tracker.compute_file_hash(path)
248
+ except Exception as exc:
249
+ skipped.append({**document, "skip_reason": f"hash_computation_failed: {exc}"})
250
  continue
251
 
252
+ # Use hash as the document identifier
253
+ existing = tracker.get_document_record(file_hash)
254
  if existing is None:
255
+ # New file - process it
256
+ to_process.append({
257
+ **document,
258
+ "file_hash": file_hash,
259
+ "target_status": "Parsed",
260
+ "hash_document_id": file_hash,
261
+ })
262
  continue
263
 
264
+ # Existing document - check if modified
265
  existing_modified = str(existing.get("modificationDate") or "").strip() or None
266
  if existing_modified != modified_time:
267
+ to_process.append({
268
+ **document,
269
+ "file_hash": file_hash,
270
+ "target_status": "Modified",
271
+ "hash_document_id": file_hash,
272
+ })
273
  else:
274
+ skipped.append({
275
+ **document,
276
+ "skip_reason": "already_processed_same_content",
277
+ "file_hash": file_hash,
278
+ })
279
  except Exception as exc:
280
  docs = [dict(doc, target_status="Parsed") for doc in documents]
281
  return docs, [], f"Firebase tracking unavailable. Processing all Drive files. Details: {exc}"
firebase_tenant_config.py ADDED
@@ -0,0 +1,209 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ from __future__ import annotations
2
+
3
+ import json
4
+ import os
5
+ from dataclasses import dataclass
6
+ from datetime import datetime, timezone
7
+ from typing import Any
8
+
9
+ from dotenv import load_dotenv
10
+
11
+
12
+ load_dotenv()
13
+
14
+
15
+ class FirebaseConfigError(RuntimeError):
16
+ """Raised when Firebase configuration is missing or invalid."""
17
+
18
+
19
+ @dataclass
20
+ class CredentialsObject:
21
+ """OAuth & API Keys for a tenant."""
22
+ openai_api_key: str
23
+ google_service_account_json: str
24
+
25
+
26
+ @dataclass
27
+ class RuleObject:
28
+ """Processing rule for a tenant."""
29
+ rule_id: str
30
+ rule_name: str
31
+ source_folder_id: str
32
+ target_folder_id: str
33
+ target_sheet_id: str
34
+ sheet_tab_name: str
35
+ parsing_prompt: str | None = None
36
+ is_enabled: bool = True
37
+
38
+ def to_dict(self) -> dict[str, Any]:
39
+ return {
40
+ "rule_id": self.rule_id,
41
+ "rule_name": self.rule_name,
42
+ "source_folder_id": self.source_folder_id,
43
+ "target_folder_id": self.target_folder_id,
44
+ "target_sheet_id": self.target_sheet_id,
45
+ "sheet_tab_name": self.sheet_tab_name,
46
+ "parsing_prompt": self.parsing_prompt,
47
+ "is_enabled": self.is_enabled,
48
+ }
49
+
50
+ @staticmethod
51
+ def from_dict(data: dict[str, Any]) -> RuleObject:
52
+ return RuleObject(
53
+ rule_id=data.get("rule_id", ""),
54
+ rule_name=data.get("rule_name", ""),
55
+ source_folder_id=data.get("source_folder_id", ""),
56
+ target_folder_id=data.get("target_folder_id", ""),
57
+ target_sheet_id=data.get("target_sheet_id", ""),
58
+ sheet_tab_name=data.get("sheet_tab_name", ""),
59
+ parsing_prompt=data.get("parsing_prompt"),
60
+ is_enabled=data.get("is_enabled", True),
61
+ )
62
+
63
+
64
+ @dataclass
65
+ class TenantConfig:
66
+ """Complete tenant configuration (rules stored as subcollection)."""
67
+ tenant_id: str
68
+ name: str
69
+ active: bool
70
+ credentials: CredentialsObject
71
+ created_at: str | None = None
72
+
73
+ def to_dict(self) -> dict[str, Any]:
74
+ return {
75
+ "tenant_id": self.tenant_id,
76
+ "name": self.name,
77
+ "active": self.active,
78
+ "credentials": {
79
+ "openai_api_key": self.credentials.openai_api_key,
80
+ "google_service_account_json": self.credentials.google_service_account_json,
81
+ },
82
+ "created_at": self.created_at,
83
+ }
84
+
85
+ @staticmethod
86
+ def from_dict(data: dict[str, Any], tenant_id: str) -> TenantConfig:
87
+ creds_data = data.get("credentials", {})
88
+ credentials = CredentialsObject(
89
+ openai_api_key=creds_data.get("openai_api_key", ""),
90
+ google_service_account_json=creds_data.get("google_service_account_json", ""),
91
+ )
92
+ return TenantConfig(
93
+ tenant_id=tenant_id,
94
+ name=data.get("name", ""),
95
+ active=data.get("active", True),
96
+ credentials=credentials,
97
+ created_at=data.get("created_at"),
98
+ )
99
+
100
+
101
+ class FirebaseTenantConfigManager:
102
+ """Manages tenant configurations in Firebase Firestore.
103
+
104
+ This is the TENANT CONFIG DB - separate from document processing.
105
+ Other environments import this and use get_tenant() to fetch configs.
106
+ """
107
+
108
+ def __init__(self, collection_name: str | None = None):
109
+ self.collection_name = (collection_name or os.getenv("FIREBASE_TENANTS_COLLECTION") or "tenants").strip()
110
+ self._db = None
111
+
112
+ def _build_credentials(self):
113
+ from firebase_admin import credentials
114
+
115
+ service_account_json = (os.getenv("FIREBASE_SERVICE_ACCOUNT_JSON") or "").strip()
116
+ service_account_file = (os.getenv("FIREBASE_SERVICE_ACCOUNT_FILE") or "").strip()
117
+
118
+ if not service_account_json:
119
+ service_account_json = (os.getenv("GOOGLE_SERVICE_ACCOUNT_JSON") or "").strip()
120
+ if not service_account_file:
121
+ service_account_file = (os.getenv("GOOGLE_SERVICE_ACCOUNT_FILE") or "").strip()
122
+
123
+ if service_account_json:
124
+ try:
125
+ info = json.loads(service_account_json)
126
+ except json.JSONDecodeError as exc:
127
+ raise FirebaseConfigError("FIREBASE_SERVICE_ACCOUNT_JSON is not valid JSON.") from exc
128
+ return credentials.Certificate(info)
129
+
130
+ if service_account_file:
131
+ return credentials.Certificate(service_account_file)
132
+
133
+ raise FirebaseConfigError(
134
+ "Missing Firebase credentials. Set FIREBASE_SERVICE_ACCOUNT_JSON or FIREBASE_SERVICE_ACCOUNT_FILE."
135
+ )
136
+
137
+ def _get_db(self):
138
+ if self._db is not None:
139
+ return self._db
140
+
141
+ import firebase_admin
142
+ from firebase_admin import firestore
143
+
144
+ if not firebase_admin._apps:
145
+ cred = self._build_credentials()
146
+ firebase_admin.initialize_app(cred)
147
+
148
+ self._db = firestore.client()
149
+ return self._db
150
+
151
+ def get_tenant(self, tenant_id: str) -> TenantConfig | None:
152
+ """Retrieve a tenant configuration by ID."""
153
+ doc = self._get_db().collection(self.collection_name).document(tenant_id).get()
154
+ if not doc.exists:
155
+ return None
156
+ data = doc.to_dict() or {}
157
+ return TenantConfig.from_dict(data, tenant_id)
158
+
159
+ def save_tenant(self, tenant: TenantConfig) -> None:
160
+ """Save a tenant configuration."""
161
+ payload = tenant.to_dict()
162
+ if not payload.get("created_at"):
163
+ payload["created_at"] = datetime.now(timezone.utc).isoformat()
164
+ payload["updated_at"] = datetime.now(timezone.utc).isoformat()
165
+ self._get_db().collection(self.collection_name).document(tenant.tenant_id).set(payload)
166
+
167
+ def list_active_tenants(self) -> list[TenantConfig]:
168
+ """Retrieve all active tenants."""
169
+ docs = (
170
+ self._get_db()
171
+ .collection(self.collection_name)
172
+ .where("active", "==", True)
173
+ .stream()
174
+ )
175
+ tenants = []
176
+ for doc in docs:
177
+ try:
178
+ tenant = TenantConfig.from_dict(doc.to_dict() or {}, doc.id)
179
+ tenants.append(tenant)
180
+ except Exception:
181
+ continue
182
+ return tenants
183
+
184
+ def get_all_rules(self, tenant_id: str) -> list[RuleObject]:
185
+ """Get all processing rules for a tenant from the rules subcollection."""
186
+ tenant_doc = self._get_db().collection(self.collection_name).document(tenant_id)
187
+ rules_subcoll = tenant_doc.collection("rules")
188
+ rules = []
189
+ for doc in rules_subcoll.stream():
190
+ rule_data = doc.to_dict() or {}
191
+ rules.append(RuleObject.from_dict(rule_data))
192
+ return rules
193
+
194
+ def get_enabled_rules(self, tenant_id: str) -> list[RuleObject]:
195
+ """Get all enabled processing rules for a tenant."""
196
+ all_rules = self.get_all_rules(tenant_id)
197
+ return [rule for rule in all_rules if rule.is_enabled]
198
+
199
+ def add_rule(self, tenant_id: str, rule: RuleObject) -> None:
200
+ """Add a rule to a tenant's rules subcollection."""
201
+ tenant_doc = self._get_db().collection(self.collection_name).document(tenant_id)
202
+ rules_subcoll = tenant_doc.collection("rules")
203
+ rules_subcoll.document(rule.rule_id).set(rule.to_dict())
204
+
205
+ def remove_rule(self, tenant_id: str, rule_id: str) -> None:
206
+ """Remove a rule from a tenant's rules subcollection."""
207
+ tenant_doc = self._get_db().collection(self.collection_name).document(tenant_id)
208
+ rules_subcoll = tenant_doc.collection("rules")
209
+ rules_subcoll.document(rule_id).delete()
google_drive_service.py CHANGED
@@ -82,13 +82,20 @@ def load_drive_folder_ids_from_env(env_var: str = "GOOGLE_DRIVE_FOLDER_IDS") ->
82
  return folder_ids
83
 
84
 
85
- def build_drive_credentials(scope: str = DRIVE_READONLY_SCOPE) -> Credentials:
86
- service_account_json = (os.getenv("GOOGLE_SERVICE_ACCOUNT_JSON") or "").strip()
 
 
 
 
 
 
 
87
  service_account_file = (os.getenv("GOOGLE_SERVICE_ACCOUNT_FILE") or "").strip()
88
 
89
- if service_account_json:
90
  try:
91
- info: dict[str, Any] = json.loads(service_account_json)
92
  except json.JSONDecodeError as exc:
93
  raise GoogleDriveConfigError("GOOGLE_SERVICE_ACCOUNT_JSON is not valid JSON.") from exc
94
  logger.info("Using GOOGLE_SERVICE_ACCOUNT_JSON for Drive credentials")
@@ -103,7 +110,13 @@ def build_drive_credentials(scope: str = DRIVE_READONLY_SCOPE) -> Credentials:
103
  )
104
 
105
 
106
- def build_drive_service(scope: str = DRIVE_READONLY_SCOPE):
 
 
 
 
 
 
107
  try:
108
  from googleapiclient.discovery import build
109
  except ImportError as exc:
@@ -111,7 +124,7 @@ def build_drive_service(scope: str = DRIVE_READONLY_SCOPE):
111
  "google-api-python-client is required for Drive scanning. Add it to requirements and install dependencies."
112
  ) from exc
113
 
114
- credentials = build_drive_credentials(scope=scope)
115
  logger.info("Building Drive service with scope=%s", scope)
116
  return build("drive", "v3", credentials=credentials, cache_discovery=False)
117
 
 
82
  return folder_ids
83
 
84
 
85
+ def build_drive_credentials(scope: str = DRIVE_READONLY_SCOPE, service_account_json: str | None = None) -> Credentials:
86
+ """Build Google Drive credentials.
87
+
88
+ Args:
89
+ scope: OAuth scope for the credentials
90
+ service_account_json: Optional JSON string with service account credentials.
91
+ If not provided, uses GOOGLE_SERVICE_ACCOUNT_JSON env var.
92
+ """
93
+ json_to_use = (service_account_json or "").strip() or (os.getenv("GOOGLE_SERVICE_ACCOUNT_JSON") or "").strip()
94
  service_account_file = (os.getenv("GOOGLE_SERVICE_ACCOUNT_FILE") or "").strip()
95
 
96
+ if json_to_use:
97
  try:
98
+ info: dict[str, Any] = json.loads(json_to_use)
99
  except json.JSONDecodeError as exc:
100
  raise GoogleDriveConfigError("GOOGLE_SERVICE_ACCOUNT_JSON is not valid JSON.") from exc
101
  logger.info("Using GOOGLE_SERVICE_ACCOUNT_JSON for Drive credentials")
 
110
  )
111
 
112
 
113
+ def build_drive_service(scope: str = DRIVE_READONLY_SCOPE, service_account_json: str | None = None):
114
+ """Build Google Drive service.
115
+
116
+ Args:
117
+ scope: OAuth scope for the credentials
118
+ service_account_json: Optional JSON string with service account credentials for tenant.
119
+ """
120
  try:
121
  from googleapiclient.discovery import build
122
  except ImportError as exc:
 
124
  "google-api-python-client is required for Drive scanning. Add it to requirements and install dependencies."
125
  ) from exc
126
 
127
+ credentials = build_drive_credentials(scope=scope, service_account_json=service_account_json)
128
  logger.info("Building Drive service with scope=%s", scope)
129
  return build("drive", "v3", credentials=credentials, cache_discovery=False)
130