Spaces:
Sleeping
Sleeping
segovia commited on
Commit ·
028d116
1
Parent(s): cb04ca0
multitenant supprt
Browse files- MULTI_TENANT_GUIDE.md +540 -0
- README.md +11 -2
- app.py +17 -384
- document_orchester.py +475 -115
- firebase_processed_files.py +76 -21
- firebase_tenant_config.py +209 -0
- google_drive_service.py +19 -6
MULTI_TENANT_GUIDE.md
ADDED
|
@@ -0,0 +1,540 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
# Multi-Tenant Document Parser - Configuration & Usage Guide
|
| 2 |
+
|
| 3 |
+
## Overview
|
| 4 |
+
|
| 5 |
+
The document parser now supports two operational modes:
|
| 6 |
+
|
| 7 |
+
1. **Multi-Tenant Mode (NEW)** - Load all active tenants from Firebase and process their enabled rules
|
| 8 |
+
2. **Legacy Mode** - Single tenant configuration via environment variables (backward compatible)
|
| 9 |
+
|
| 10 |
+
## Architecture
|
| 11 |
+
|
| 12 |
+
### Data Model
|
| 13 |
+
|
| 14 |
+
All tenant and rule configurations are stored in Firestore under the `tenants` collection:
|
| 15 |
+
|
| 16 |
+
```
|
| 17 |
+
TENANTS_COLLECTION (Firestore)
|
| 18 |
+
├── {tenant_id}
|
| 19 |
+
│ ├── tenant_id: unique tenant ID
|
| 20 |
+
│ ├── name: "Company/User Name"
|
| 21 |
+
│ ├── active: boolean (Account Status)
|
| 22 |
+
│ ├── created_at: timestamp
|
| 23 |
+
│ ├── updated_at: timestamp
|
| 24 |
+
│ ├── credentials (object, 1:1 relationship)
|
| 25 |
+
│ │ ├── openai_api_key: "OpenAI Key"
|
| 26 |
+
│ │ └── google_service_account_json: "Service account JSON"
|
| 27 |
+
│ └── rules (subcollection, 1:N relationship)
|
| 28 |
+
│ └── {rule_id}
|
| 29 |
+
│ ├── rule_id: "unique rule ID"
|
| 30 |
+
│ ├── rule_name: "Invoices to Finance"
|
| 31 |
+
│ ├── source_folder_id: "Google Drive Source Folder ID"
|
| 32 |
+
│ ├── target_folder_id: "Google Drive Destination Folder ID"
|
| 33 |
+
│ ├── target_sheet_id: "Google Sheets Spreadsheet ID"
|
| 34 |
+
│ ├── sheet_tab_name: "Tab inside Google Sheet"
|
| 35 |
+
│ ├── parsing_prompt: "Custom LLM Instructions"
|
| 36 |
+
│ └── is_enabled: boolean (Rule Toggle)
|
| 37 |
+
```
|
| 38 |
+
|
| 39 |
+
### Processing Flow
|
| 40 |
+
|
| 41 |
+
```
|
| 42 |
+
Multi-Tenant Mode:
|
| 43 |
+
1. orchestrate_all_active_tenants()
|
| 44 |
+
└─> For each active tenant:
|
| 45 |
+
└─> For each enabled rule:
|
| 46 |
+
└─> orchestrate_single_rule()
|
| 47 |
+
├─> Scan rule.source_folder_id
|
| 48 |
+
├─> Parse documents with rule.parsing_prompt
|
| 49 |
+
├─> Move to rule.target_folder_id
|
| 50 |
+
└─> Send to rule.target_sheet_id
|
| 51 |
+
```
|
| 52 |
+
|
| 53 |
+
## Usage
|
| 54 |
+
|
| 55 |
+
### Quick Start (Using .env)
|
| 56 |
+
|
| 57 |
+
If you have existing `.env` configuration, multi-tenant mode will automatically create a default tenant:
|
| 58 |
+
|
| 59 |
+
```bash
|
| 60 |
+
# Just run it! No Firebase setup needed initially
|
| 61 |
+
python document_orchester.py --send
|
| 62 |
+
|
| 63 |
+
# The system will:
|
| 64 |
+
# 1. Check for active tenants in Firebase
|
| 65 |
+
# 2. If none found, create "default_tenant" from your .env vars
|
| 66 |
+
# 3. Process documents using the default configuration
|
| 67 |
+
```
|
| 68 |
+
|
| 69 |
+
**This is the easiest path if you're migrating from single-tenant mode.**
|
| 70 |
+
|
| 71 |
+
### Mode 1: Multi-Tenant Mode (Default)
|
| 72 |
+
|
| 73 |
+
```bash
|
| 74 |
+
# Process all active tenants and their enabled rules
|
| 75 |
+
python document_orchester.py --model gpt-4o --send
|
| 76 |
+
|
| 77 |
+
# Optional flags:
|
| 78 |
+
# --no-subfolders Only scan direct files (don't recurse)
|
| 79 |
+
# --send Send results to Google Sheets
|
| 80 |
+
# --output FILE Save JSON summary to file
|
| 81 |
+
```
|
| 82 |
+
|
| 83 |
+
### Mode 2: Legacy Single-Tenant Mode
|
| 84 |
+
|
| 85 |
+
```bash
|
| 86 |
+
# Use env var configuration (backward compatible)
|
| 87 |
+
python document_orchester.py --legacy --model gpt-4o --send
|
| 88 |
+
```
|
| 89 |
+
|
| 90 |
+
### Gradio Interface
|
| 91 |
+
|
| 92 |
+
Start the web app:
|
| 93 |
+
```bash
|
| 94 |
+
python app.py
|
| 95 |
+
```
|
| 96 |
+
|
| 97 |
+
Two orchestrator endpoints are now available:
|
| 98 |
+
- **Single Tenant (Legacy)** - Uses env var configuration
|
| 99 |
+
- **Multi Tenant** - Loads all active tenants from Firebase
|
| 100 |
+
|
| 101 |
+
## Configuration
|
| 102 |
+
|
| 103 |
+
### Automatic Default Tenant (No Firebase Setup Required)
|
| 104 |
+
|
| 105 |
+
When you run multi-tenant mode without any tenants in Firebase:
|
| 106 |
+
|
| 107 |
+
1. The system checks for active tenants in Firestore
|
| 108 |
+
2. If none found, it creates a **default tenant** from your `.env` file
|
| 109 |
+
3. Maps env vars like this:
|
| 110 |
+
|
| 111 |
+
| .env Variable | Default Tenant |
|
| 112 |
+
|---|---|
|
| 113 |
+
| `OPENAI_API_KEY` | `credentials.openai_api_key` |
|
| 114 |
+
| `GOOGLE_SERVICE_ACCOUNT_JSON` | `credentials.google_service_account_json` |
|
| 115 |
+
| `GOOGLE_DRIVE_FOLDER_IDS` (first) | `rule.source_folder_id` |
|
| 116 |
+
| `GOOGLE_DRIVE_INVOICES_ROOT_FOLDER_ID` | `rule.target_folder_id` |
|
| 117 |
+
| `GOOGLE_SHEETS_SPREADSHEET_ID` | `rule.target_sheet_id` |
|
| 118 |
+
|
| 119 |
+
**Example: Your existing `.env`**
|
| 120 |
+
```env
|
| 121 |
+
OPENAI_API_KEY=sk-abc123...
|
| 122 |
+
GOOGLE_SERVICE_ACCOUNT_JSON={"type":"service_account","project_id":"my-project",...}
|
| 123 |
+
GOOGLE_DRIVE_FOLDER_IDS=1ABC_folder_id_here
|
| 124 |
+
GOOGLE_DRIVE_INVOICES_ROOT_FOLDER_ID=1XYZ_destination_folder_id
|
| 125 |
+
GOOGLE_SHEETS_SPREADSHEET_ID=18qKL...
|
| 126 |
+
```
|
| 127 |
+
|
| 128 |
+
**Creates this default tenant (in memory, not saved to Firebase):**
|
| 129 |
+
```json
|
| 130 |
+
{
|
| 131 |
+
"tenant_id": "default_tenant",
|
| 132 |
+
"name": "Default Tenant (from .env)",
|
| 133 |
+
"active": true,
|
| 134 |
+
"credentials": {
|
| 135 |
+
"openai_api_key": "sk-abc123...",
|
| 136 |
+
"google_service_account_json": "{...}"
|
| 137 |
+
},
|
| 138 |
+
"rules": [
|
| 139 |
+
{
|
| 140 |
+
"rule_id": "default_rule",
|
| 141 |
+
"rule_name": "Default Processing Rule",
|
| 142 |
+
"source_folder_id": "1ABC_folder_id_here",
|
| 143 |
+
"target_folder_id": "1XYZ_destination_folder_id",
|
| 144 |
+
"target_sheet_id": "18qKL...",
|
| 145 |
+
"sheet_tab_name": "Parsed",
|
| 146 |
+
"is_enabled": true
|
| 147 |
+
}
|
| 148 |
+
]
|
| 149 |
+
}
|
| 150 |
+
```
|
| 151 |
+
|
| 152 |
+
The default tenant provides automatic setup:
|
| 153 |
+
- ✅ Created automatically if no Firebase tenants exist
|
| 154 |
+
- ✅ Saved to Firebase **BEFORE** orchestration starts
|
| 155 |
+
- ✅ Future runs load from Firebase instead of recreating
|
| 156 |
+
- ✅ Try multi-tenant mode without any setup
|
| 157 |
+
- ✅ Migrate gradually from single-tenant to multi-tenant
|
| 158 |
+
- ✅ Keep existing workflows unchanged
|
| 159 |
+
|
| 160 |
+
**The flow:**
|
| 161 |
+
1. First run: No Firebase tenants → Create default_tenant from .env → **Save to Firebase** → Process documents
|
| 162 |
+
2. Second run: Firebase has default_tenant → Load it → Process documents
|
| 163 |
+
3. Add more tenants anytime in Firebase → They're all processed together
|
| 164 |
+
|
| 165 |
+
### Multi-Tenant Setup (Firebase)
|
| 166 |
+
|
| 167 |
+
1. Create a Firestore collection called `tenants` (or set via `FIREBASE_TENANTS_COLLECTION` env var)
|
| 168 |
+
|
| 169 |
+
2. Add tenant documents with this structure:
|
| 170 |
+
|
| 171 |
+
```json
|
| 172 |
+
{
|
| 173 |
+
"name": "Acme Corp",
|
| 174 |
+
"active": true,
|
| 175 |
+
"created_at": "2024-01-01T00:00:00Z",
|
| 176 |
+
"updated_at": "2024-01-01T00:00:00Z",
|
| 177 |
+
"credentials": {
|
| 178 |
+
"openai_api_key": "sk-...",
|
| 179 |
+
"google_service_account_json": "{...full service account json...}"
|
| 180 |
+
}
|
| 181 |
+
}
|
| 182 |
+
```
|
| 183 |
+
|
| 184 |
+
3. Create a **rules** subcollection inside the tenant document, and add rules with this structure:
|
| 185 |
+
|
| 186 |
+
```json
|
| 187 |
+
{
|
| 188 |
+
"rule_id": "invoices_2024",
|
| 189 |
+
"rule_name": "Invoices to Finance",
|
| 190 |
+
"source_folder_id": "1ABC...",
|
| 191 |
+
"target_folder_id": "1XYZ...",
|
| 192 |
+
"target_sheet_id": "18qKL...",
|
| 193 |
+
"sheet_tab_name": "Invoices",
|
| 194 |
+
"parsing_prompt": "Extract invoice details following these rules...",
|
| 195 |
+
"is_enabled": true
|
| 196 |
+
}
|
| 197 |
+
```
|
| 198 |
+
|
| 199 |
+
### Environment Variables
|
| 200 |
+
|
| 201 |
+
**Required for both modes:**
|
| 202 |
+
```
|
| 203 |
+
FIREBASE_SERVICE_ACCOUNT_JSON # Firebase credentials (JSON string or use GOOGLE_SERVICE_ACCOUNT_JSON)
|
| 204 |
+
OPENAI_API_KEY # (only for legacy mode, multi-tenant uses tenant credentials)
|
| 205 |
+
```
|
| 206 |
+
|
| 207 |
+
**Optional:**
|
| 208 |
+
```
|
| 209 |
+
FIREBASE_TENANTS_COLLECTION # Default: "tenants"
|
| 210 |
+
FIREBASE_DOCUMENTS_COLLECTION # Default: "processed_documents"
|
| 211 |
+
FIREBASE_TRACKER_COLLECTION # Default: "processed_files"
|
| 212 |
+
FIREBASE_TRACK_PROCESSED # Default: "true"
|
| 213 |
+
OPENAI_MODEL # Default: "gpt-4o"
|
| 214 |
+
GOOGLE_DRIVE_INVOICES_BASE_PATH # Default: "Facturas"
|
| 215 |
+
ORCHESTRATOR_API_KEY # For API authentication
|
| 216 |
+
```
|
| 217 |
+
|
| 218 |
+
## Firebase Storage Structure
|
| 219 |
+
|
| 220 |
+
### Document Tracking (Multi-Tenant)
|
| 221 |
+
|
| 222 |
+
Each tenant's processed files are tracked in nested collections:
|
| 223 |
+
|
| 224 |
+
```
|
| 225 |
+
tenants/{tenant_id}/processed_files/{file_hash}
|
| 226 |
+
└─ file_hash: "sha256 hash"
|
| 227 |
+
└─ source_file: "filename"
|
| 228 |
+
└─ processed_at: "timestamp"
|
| 229 |
+
|
| 230 |
+
tenants/{tenant_id}/processed_documents/{document_id}
|
| 231 |
+
└─ document_id: "drive file id"
|
| 232 |
+
└─ parsed_data: {...}
|
| 233 |
+
└─ status: "Parsed|Modified|Sent"
|
| 234 |
+
```
|
| 235 |
+
|
| 236 |
+
### Creating Tenants Programmatically
|
| 237 |
+
|
| 238 |
+
```python
|
| 239 |
+
from firebase_tenant_config import (
|
| 240 |
+
FirebaseTenantConfigManager,
|
| 241 |
+
TenantConfig,
|
| 242 |
+
CredentialsObject,
|
| 243 |
+
RuleObject,
|
| 244 |
+
)
|
| 245 |
+
|
| 246 |
+
manager = FirebaseTenantConfigManager()
|
| 247 |
+
|
| 248 |
+
# Create credentials
|
| 249 |
+
credentials = CredentialsObject(
|
| 250 |
+
openai_api_key="sk-...",
|
| 251 |
+
google_service_account_json="..." # full JSON as string
|
| 252 |
+
)
|
| 253 |
+
|
| 254 |
+
# Create a rule
|
| 255 |
+
rule = RuleObject(
|
| 256 |
+
rule_id="invoices_2024",
|
| 257 |
+
rule_name="Invoices to Finance",
|
| 258 |
+
source_folder_id="1ABC...",
|
| 259 |
+
target_folder_id="1XYZ...",
|
| 260 |
+
target_sheet_id="18qKL...",
|
| 261 |
+
sheet_tab_name="Invoices",
|
| 262 |
+
parsing_prompt="Extract...",
|
| 263 |
+
is_enabled=True,
|
| 264 |
+
)
|
| 265 |
+
|
| 266 |
+
# Create tenant
|
| 267 |
+
tenant = TenantConfig(
|
| 268 |
+
tenant_id="acme-corp",
|
| 269 |
+
name="Acme Corp",
|
| 270 |
+
active=True,
|
| 271 |
+
credentials=credentials,
|
| 272 |
+
rules=[rule],
|
| 273 |
+
)
|
| 274 |
+
|
| 275 |
+
# Save to Firebase
|
| 276 |
+
manager.save_tenant(tenant)
|
| 277 |
+
|
| 278 |
+
# List all active tenants
|
| 279 |
+
active_tenants = manager.list_active_tenants()
|
| 280 |
+
|
| 281 |
+
# Get enabled rules for a tenant
|
| 282 |
+
enabled_rules = manager.get_enabled_rules("acme-corp")
|
| 283 |
+
|
| 284 |
+
# Add a rule
|
| 285 |
+
manager.add_rule("acme-corp", new_rule)
|
| 286 |
+
|
| 287 |
+
# Remove a rule
|
| 288 |
+
manager.remove_rule("acme-corp", "invoices_2024")
|
| 289 |
+
```
|
| 290 |
+
|
| 291 |
+
## Output Format
|
| 292 |
+
|
| 293 |
+
### Multi-Tenant Orchestration Result
|
| 294 |
+
|
| 295 |
+
```json
|
| 296 |
+
{
|
| 297 |
+
"ok": true,
|
| 298 |
+
"tenants_processed": 2,
|
| 299 |
+
"total_parsed": 15,
|
| 300 |
+
"total_modified": 3,
|
| 301 |
+
"total_sent": 18,
|
| 302 |
+
"total_moved": 18,
|
| 303 |
+
"total_corrupted": 2,
|
| 304 |
+
"total_errors": 1,
|
| 305 |
+
"tenants_results": [
|
| 306 |
+
{
|
| 307 |
+
"tenant_id": "tenant_123",
|
| 308 |
+
"tenant_name": "Acme Corp",
|
| 309 |
+
"ok": true,
|
| 310 |
+
"rules_processed": 2,
|
| 311 |
+
"parsed": 10,
|
| 312 |
+
"modified": 2,
|
| 313 |
+
"sent": 12,
|
| 314 |
+
"moved": 12,
|
| 315 |
+
"corrupted": 1,
|
| 316 |
+
"errors": 0,
|
| 317 |
+
"rules_results": [
|
| 318 |
+
{
|
| 319 |
+
"ok": true,
|
| 320 |
+
"tenant_id": "tenant_123",
|
| 321 |
+
"rule_id": "invoices_2024",
|
| 322 |
+
"rule_name": "Invoices to Finance",
|
| 323 |
+
"scanned": 20,
|
| 324 |
+
"to_process": 15,
|
| 325 |
+
"skipped": 5,
|
| 326 |
+
"parsed": 10,
|
| 327 |
+
"modified": 2,
|
| 328 |
+
"sent": 12,
|
| 329 |
+
"moved": 12,
|
| 330 |
+
"corrupted": 1,
|
| 331 |
+
"errors": 0,
|
| 332 |
+
"processed_items": [...],
|
| 333 |
+
"skipped_items": [...]
|
| 334 |
+
}
|
| 335 |
+
]
|
| 336 |
+
}
|
| 337 |
+
]
|
| 338 |
+
}
|
| 339 |
+
```
|
| 340 |
+
|
| 341 |
+
## Migration from Legacy Mode
|
| 342 |
+
|
| 343 |
+
**No migration needed!** The system maintains full backward compatibility:
|
| 344 |
+
|
| 345 |
+
1. Existing env var configuration continues to work with `--legacy` flag
|
| 346 |
+
2. New multi-tenant mode coexists without affecting legacy deployments
|
| 347 |
+
3. To enable multi-tenant: Set up tenants in Firebase and run without `--legacy`
|
| 348 |
+
|
| 349 |
+
## Workflow Examples
|
| 350 |
+
|
| 351 |
+
### Simple: Use Existing .env (No Firebase Setup)
|
| 352 |
+
|
| 353 |
+
The **easiest** way to get started with multi-tenant mode:
|
| 354 |
+
|
| 355 |
+
```bash
|
| 356 |
+
# 1. Ensure your .env has these vars (you probably already do):
|
| 357 |
+
# OPENAI_API_KEY
|
| 358 |
+
# GOOGLE_SERVICE_ACCOUNT_JSON
|
| 359 |
+
# GOOGLE_DRIVE_FOLDER_IDS
|
| 360 |
+
# GOOGLE_DRIVE_INVOICES_ROOT_FOLDER_ID
|
| 361 |
+
# GOOGLE_SHEETS_SPREADSHEET_ID
|
| 362 |
+
|
| 363 |
+
# 2. Just run multi-tenant orchestration:
|
| 364 |
+
python document_orchester.py --send
|
| 365 |
+
|
| 366 |
+
# That's it! The system will:
|
| 367 |
+
# - Check for tenants in Firebase
|
| 368 |
+
# - Find none, create default_tenant from .env
|
| 369 |
+
# - Process documents exactly as before
|
| 370 |
+
# - Results go to Google Sheets and Drive
|
| 371 |
+
```
|
| 372 |
+
|
| 373 |
+
**No code changes, no Firebase setup, just works!**
|
| 374 |
+
|
| 375 |
+
### Advanced: Multiple Tenants in Firebase
|
| 376 |
+
|
| 377 |
+
For multiple tenants with different configurations:
|
| 378 |
+
|
| 379 |
+
### Step 1: Set Up Tenant in Firebase
|
| 380 |
+
|
| 381 |
+
```python
|
| 382 |
+
# Run this once to initialize a tenant
|
| 383 |
+
from firebase_tenant_config import FirebaseTenantConfigManager, TenantConfig, CredentialsObject, RuleObject
|
| 384 |
+
|
| 385 |
+
manager = FirebaseTenantConfigManager()
|
| 386 |
+
|
| 387 |
+
credentials = CredentialsObject(
|
| 388 |
+
openai_api_key="sk-xxx",
|
| 389 |
+
google_service_account_json="json_string_here"
|
| 390 |
+
)
|
| 391 |
+
|
| 392 |
+
rule = RuleObject(
|
| 393 |
+
rule_id="rule_1",
|
| 394 |
+
rule_name="Invoice Processing",
|
| 395 |
+
source_folder_id="drive_folder_id",
|
| 396 |
+
target_folder_id="drive_dest_folder_id",
|
| 397 |
+
target_sheet_id="sheets_id",
|
| 398 |
+
sheet_tab_name="Invoices",
|
| 399 |
+
is_enabled=True,
|
| 400 |
+
)
|
| 401 |
+
|
| 402 |
+
tenant = TenantConfig(
|
| 403 |
+
tenant_id="my_tenant",
|
| 404 |
+
name="My Company",
|
| 405 |
+
active=True,
|
| 406 |
+
credentials=credentials,
|
| 407 |
+
)
|
| 408 |
+
|
| 409 |
+
manager.save_tenant(tenant)
|
| 410 |
+
manager.add_rule("my_tenant", rule)
|
| 411 |
+
```
|
| 412 |
+
|
| 413 |
+
### Step 2: Run Orchestration
|
| 414 |
+
|
| 415 |
+
```bash
|
| 416 |
+
# This will automatically:
|
| 417 |
+
# 1. Load all active tenants from Firebase (not just one!)
|
| 418 |
+
# 2. For each tenant, process all enabled rules
|
| 419 |
+
# 3. Track results in tenant-specific collections
|
| 420 |
+
python document_orchester.py --model gpt-4o --send
|
| 421 |
+
```
|
| 422 |
+
|
| 423 |
+
### Step 3: Review Results
|
| 424 |
+
|
| 425 |
+
Results are stored in:
|
| 426 |
+
- Google Drive: Documents moved to `{target_folder_id}/{YYYY}{MM}/`
|
| 427 |
+
- Google Sheets: Parsed data appended to `{target_sheet_id}` sheet
|
| 428 |
+
- Firebase: Document metadata in `tenants/{tenant_id}/processed_documents/`
|
| 429 |
+
|
| 430 |
+
## Troubleshooting
|
| 431 |
+
|
| 432 |
+
### "No active tenants found"
|
| 433 |
+
- Check that `active: true` in Firebase tenant document
|
| 434 |
+
- Verify `FIREBASE_TENANTS_COLLECTION` env var is correct
|
| 435 |
+
- Confirm Firebase credentials are valid
|
| 436 |
+
|
| 437 |
+
### "No enabled rules"
|
| 438 |
+
- Check that `is_enabled: true` for at least one rule
|
| 439 |
+
- Verify rule has valid `source_folder_id`
|
| 440 |
+
|
| 441 |
+
### Credentials error
|
| 442 |
+
- Ensure `credentials.openai_api_key` and `credentials.google_service_account_json` are set
|
| 443 |
+
- Verify JSON strings are properly formatted
|
| 444 |
+
|
| 445 |
+
### Mixing modes
|
| 446 |
+
- Don't mix multi-tenant and legacy configurations
|
| 447 |
+
- Use either all Firebase-based tenants OR all env vars, not both
|
| 448 |
+
- When switching modes, ensure all required env vars/Firebase docs exist
|
| 449 |
+
|
| 450 |
+
## API Endpoints
|
| 451 |
+
|
| 452 |
+
### Gradio Interface
|
| 453 |
+
|
| 454 |
+
Two REST-compatible endpoints are exposed:
|
| 455 |
+
|
| 456 |
+
**Legacy Single-Tenant:**
|
| 457 |
+
```
|
| 458 |
+
POST /api/orchestrate_drive_documents
|
| 459 |
+
{
|
| 460 |
+
"model": "gpt-4o",
|
| 461 |
+
"include_subfolders": true,
|
| 462 |
+
"send_to_sheet": true,
|
| 463 |
+
"api_key": "your_orchestrator_api_key"
|
| 464 |
+
}
|
| 465 |
+
```
|
| 466 |
+
|
| 467 |
+
**Multi-Tenant:**
|
| 468 |
+
```
|
| 469 |
+
POST /api/orchestrate_all_tenants
|
| 470 |
+
{
|
| 471 |
+
"model": "gpt-4o",
|
| 472 |
+
"include_subfolders": true,
|
| 473 |
+
"send_to_sheet": true,
|
| 474 |
+
"api_key": "your_orchestrator_api_key"
|
| 475 |
+
}
|
| 476 |
+
```
|
| 477 |
+
|
| 478 |
+
## Performance Considerations
|
| 479 |
+
|
| 480 |
+
- Multi-tenant processing is sequential (one tenant at a time, then one rule at a time)
|
| 481 |
+
- Each tenant/rule maintains separate Firebase document tracking
|
| 482 |
+
- Large scale deployments may benefit from rule parallelization (future enhancement)
|
| 483 |
+
|
| 484 |
+
## Managing Rules
|
| 485 |
+
|
| 486 |
+
### Adding a Rule to an Existing Tenant
|
| 487 |
+
|
| 488 |
+
You can programmatically add rules to any tenant:
|
| 489 |
+
|
| 490 |
+
```python
|
| 491 |
+
from firebase_tenant_config import (
|
| 492 |
+
FirebaseTenantConfigManager,
|
| 493 |
+
RuleObject,
|
| 494 |
+
)
|
| 495 |
+
|
| 496 |
+
manager = FirebaseTenantConfigManager()
|
| 497 |
+
|
| 498 |
+
# Create a new rule
|
| 499 |
+
new_rule = RuleObject(
|
| 500 |
+
rule_id="new_rule_123",
|
| 501 |
+
rule_name="Additional Processing Rule",
|
| 502 |
+
source_folder_id="1NEW_folder_id",
|
| 503 |
+
target_folder_id="1NEW_destination_id",
|
| 504 |
+
target_sheet_id="1NEW_sheet_id",
|
| 505 |
+
sheet_tab_name="Sheet Tab Name",
|
| 506 |
+
parsing_prompt="Custom parsing instructions...",
|
| 507 |
+
is_enabled=True,
|
| 508 |
+
)
|
| 509 |
+
|
| 510 |
+
# Add to tenant
|
| 511 |
+
manager.add_rule("my_tenant", new_rule)
|
| 512 |
+
```
|
| 513 |
+
|
| 514 |
+
### Cloning a Rule
|
| 515 |
+
|
| 516 |
+
Use the provided script to clone the default rule:
|
| 517 |
+
|
| 518 |
+
```bash
|
| 519 |
+
python add_rule_from_default.py new_rule_id "New Rule Name"
|
| 520 |
+
```
|
| 521 |
+
|
| 522 |
+
This creates a new rule in the **default_tenant** with the same configuration (folder IDs, sheet, etc.) as the default rule. Perfect for creating multiple processing pipelines with identical settings.
|
| 523 |
+
|
| 524 |
+
### Removing a Rule
|
| 525 |
+
|
| 526 |
+
```python
|
| 527 |
+
from firebase_tenant_config import FirebaseTenantConfigManager
|
| 528 |
+
|
| 529 |
+
manager = FirebaseTenantConfigManager()
|
| 530 |
+
manager.remove_rule("my_tenant", "rule_id_to_remove")
|
| 531 |
+
```
|
| 532 |
+
|
| 533 |
+
## Future Enhancements
|
| 534 |
+
|
| 535 |
+
Potential improvements:
|
| 536 |
+
- Parallel rule processing within a tenant
|
| 537 |
+
- Rule scheduling (cron-like execution)
|
| 538 |
+
- Audit logging for rule changes
|
| 539 |
+
- Rule versioning and rollback
|
| 540 |
+
- Web UI for rule management
|
README.md
CHANGED
|
@@ -1,5 +1,5 @@
|
|
| 1 |
---
|
| 2 |
-
title:
|
| 3 |
emoji: 📚
|
| 4 |
colorFrom: pink
|
| 5 |
colorTo: purple
|
|
@@ -24,8 +24,10 @@ The app supports scanning one or more Google Drive folders and parsing only file
|
|
| 24 |
Optional:
|
| 25 |
|
| 26 |
- `FIREBASE_TRACK_PROCESSED`: defaults to `true`
|
| 27 |
-
- `
|
| 28 |
- `FIREBASE_DOCUMENTS_COLLECTION`: defaults to `processed_documents`
|
|
|
|
|
|
|
| 29 |
|
| 30 |
### How it works
|
| 31 |
|
|
@@ -39,6 +41,13 @@ Optional:
|
|
| 39 |
|
| 40 |
The orchestration script scans Drive, checks if a document was already processed, parses only new/modified documents, and saves parsed payload + lifecycle status in Firebase.
|
| 41 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 42 |
### Status rules
|
| 43 |
|
| 44 |
- `Parsed`: document was never processed before.
|
|
|
|
| 1 |
---
|
| 2 |
+
title: Invoice Parser Agent
|
| 3 |
emoji: 📚
|
| 4 |
colorFrom: pink
|
| 5 |
colorTo: purple
|
|
|
|
| 24 |
Optional:
|
| 25 |
|
| 26 |
- `FIREBASE_TRACK_PROCESSED`: defaults to `true`
|
| 27 |
+
- `FIREBASE_TRACKER_COLLECTION`: defaults to `processed_files`
|
| 28 |
- `FIREBASE_DOCUMENTS_COLLECTION`: defaults to `processed_documents`
|
| 29 |
+
- `GOOGLE_DRIVE_FACTURAS_BASE_PATH`: defaults to `Facturas`
|
| 30 |
+
- `GOOGLE_DRIVE_INVOICE_ROOT_FOLDER_ID`: optional Drive folder id used as root when creating `GOOGLE_DRIVE_FACTURAS_BASE_PATH`
|
| 31 |
|
| 32 |
### How it works
|
| 33 |
|
|
|
|
| 41 |
|
| 42 |
The orchestration script scans Drive, checks if a document was already processed, parses only new/modified documents, and saves parsed payload + lifecycle status in Firebase.
|
| 43 |
|
| 44 |
+
Before each document is persisted, the file is moved in Drive using this path rule:
|
| 45 |
+
|
| 46 |
+
- `Facturas/yyyyMM` for valid invoices where `fecha` can be parsed.
|
| 47 |
+
- `Facturas/Corrupted` when required sheet data is missing or `fecha` is not parseable.
|
| 48 |
+
|
| 49 |
+
You can change `Facturas` with `GOOGLE_DRIVE_FACTURAS_BASE_PATH`.
|
| 50 |
+
|
| 51 |
### Status rules
|
| 52 |
|
| 53 |
- `Parsed`: document was never processed before.
|
app.py
CHANGED
|
@@ -1,15 +1,13 @@
|
|
| 1 |
from __future__ import annotations
|
| 2 |
|
| 3 |
import os
|
| 4 |
-
from datetime import datetime
|
| 5 |
|
| 6 |
# Avoid the Node.js SSR proxy in hosted environments where it can terminate unexpectedly.
|
| 7 |
os.environ["GRADIO_SSR_MODE"] = "false"
|
| 8 |
|
| 9 |
import gradio as gr
|
| 10 |
|
| 11 |
-
from document_orchester import
|
| 12 |
-
from firebase_processed_files import build_firebase_tracker
|
| 13 |
from google_sheets_service import GoogleSheetsConfigError, append_row_to_google_sheet
|
| 14 |
|
| 15 |
try:
|
|
@@ -17,26 +15,6 @@ try:
|
|
| 17 |
except ImportError:
|
| 18 |
spaces = None
|
| 19 |
|
| 20 |
-
|
| 21 |
-
RESULT_COLUMNS = [
|
| 22 |
-
"source_file",
|
| 23 |
-
"document_type",
|
| 24 |
-
"fecha",
|
| 25 |
-
"fecha_vencimiento",
|
| 26 |
-
"cuit_proveedor",
|
| 27 |
-
"description_proveedor",
|
| 28 |
-
"moneda",
|
| 29 |
-
"subtotal",
|
| 30 |
-
"taxes",
|
| 31 |
-
"total",
|
| 32 |
-
"confidence",
|
| 33 |
-
]
|
| 34 |
-
DISPLAY_DESCRIPTION_COLUMN = "display_description"
|
| 35 |
-
GRID_COLUMNS = ["confidence", "fecha", "source_file", DISPLAY_DESCRIPTION_COLUMN, "total"]
|
| 36 |
-
GRID_EDITABLE_COLUMNS = ["confidence", "fecha", "source_file", "total"]
|
| 37 |
-
GRID_HEADER_LABELS = ["Confianza", "Fecha", "Nombre del archivo", "Descripcion", "Total", "Seleccionar"]
|
| 38 |
-
SHEET_COLUMNS = ["fecha", "source_file", DISPLAY_DESCRIPTION_COLUMN, "total"]
|
| 39 |
-
PENDING_STATUSES = ["Parsed", "Modified", "parsed", "modified"]
|
| 40 |
custom_css = """
|
| 41 |
.cell-menu, .cell-menu-button {
|
| 42 |
display: none !important;
|
|
@@ -49,306 +27,9 @@ def gpu_decorator(fn):
|
|
| 49 |
return fn
|
| 50 |
return spaces.GPU(fn)
|
| 51 |
|
| 52 |
-
|
| 53 |
-
firebase_tracker = build_firebase_tracker()
|
| 54 |
-
|
| 55 |
-
|
| 56 |
-
def build_display_description(row: dict) -> str:
|
| 57 |
-
provider_name = str(row.get("description_proveedor") or "").strip()
|
| 58 |
-
provider_cuit = str(row.get("cuit_proveedor") or "").strip()
|
| 59 |
-
city = str(row.get("ciudad") or row.get("city") or "").strip()
|
| 60 |
-
|
| 61 |
-
parts = [part for part in [provider_name, provider_cuit, city] if part]
|
| 62 |
-
return " - ".join(parts)
|
| 63 |
-
|
| 64 |
-
|
| 65 |
-
def sort_rows_by_fecha(rows: list[dict]) -> list[dict]:
|
| 66 |
-
def sort_key(row: dict):
|
| 67 |
-
raw = row.get("fecha")
|
| 68 |
-
text = "" if raw is None else str(raw).strip()
|
| 69 |
-
if not text:
|
| 70 |
-
return (1, float("inf"), "")
|
| 71 |
-
|
| 72 |
-
for fmt in ("%Y-%m-%d", "%d/%m/%Y", "%d-%m-%Y", "%Y/%m/%d"):
|
| 73 |
-
try:
|
| 74 |
-
parsed = datetime.strptime(text, fmt)
|
| 75 |
-
return (0, parsed.timestamp(), "")
|
| 76 |
-
except ValueError:
|
| 77 |
-
continue
|
| 78 |
-
|
| 79 |
-
try:
|
| 80 |
-
parsed = datetime.fromisoformat(text)
|
| 81 |
-
return (0, parsed.timestamp(), "")
|
| 82 |
-
except ValueError:
|
| 83 |
-
return (0, float("inf"), text.lower())
|
| 84 |
-
|
| 85 |
-
return sorted(rows, key=sort_key)
|
| 86 |
-
|
| 87 |
-
|
| 88 |
-
def rows_to_table(rows: list[dict]) -> list[list[object]]:
|
| 89 |
-
table_rows: list[list[object]] = []
|
| 90 |
-
for row in rows:
|
| 91 |
-
values: list[object] = []
|
| 92 |
-
for column in GRID_COLUMNS:
|
| 93 |
-
if column == DISPLAY_DESCRIPTION_COLUMN:
|
| 94 |
-
values.append(build_display_description(row))
|
| 95 |
-
else:
|
| 96 |
-
values.append(row.get(column))
|
| 97 |
-
table_rows.append(values + [False])
|
| 98 |
-
|
| 99 |
-
return table_rows
|
| 100 |
-
|
| 101 |
-
|
| 102 |
-
def table_to_rows(table: object) -> list[dict]:
|
| 103 |
-
if table is None:
|
| 104 |
-
return []
|
| 105 |
-
|
| 106 |
-
if hasattr(table, "iterrows"):
|
| 107 |
-
try:
|
| 108 |
-
rows: list[dict] = []
|
| 109 |
-
for _, record in table.iterrows():
|
| 110 |
-
values = list(record)
|
| 111 |
-
if len(values) > len(GRID_COLUMNS):
|
| 112 |
-
values = values[: len(GRID_COLUMNS)]
|
| 113 |
-
row = {}
|
| 114 |
-
for index, column in enumerate(GRID_COLUMNS):
|
| 115 |
-
if column not in GRID_EDITABLE_COLUMNS:
|
| 116 |
-
continue
|
| 117 |
-
row[column] = values[index] if index < len(values) else None
|
| 118 |
-
rows.append(row)
|
| 119 |
-
return rows
|
| 120 |
-
except Exception:
|
| 121 |
-
pass
|
| 122 |
-
|
| 123 |
-
rows: list[dict] = []
|
| 124 |
-
for raw_row in table: # type: ignore[assignment]
|
| 125 |
-
values = list(raw_row)
|
| 126 |
-
if len(values) > len(GRID_COLUMNS):
|
| 127 |
-
values = values[: len(GRID_COLUMNS)]
|
| 128 |
-
row = {}
|
| 129 |
-
for index, column in enumerate(GRID_COLUMNS):
|
| 130 |
-
if column not in GRID_EDITABLE_COLUMNS:
|
| 131 |
-
continue
|
| 132 |
-
row[column] = values[index] if index < len(values) else None
|
| 133 |
-
rows.append(row)
|
| 134 |
-
return rows
|
| 135 |
-
|
| 136 |
-
|
| 137 |
-
def table_to_display_rows(table: object) -> list[dict]:
|
| 138 |
-
if table is None:
|
| 139 |
-
return []
|
| 140 |
-
|
| 141 |
-
rows: list[dict] = []
|
| 142 |
-
|
| 143 |
-
if hasattr(table, "iterrows"):
|
| 144 |
-
try:
|
| 145 |
-
for _, record in table.iterrows():
|
| 146 |
-
values = list(record)
|
| 147 |
-
if len(values) > len(GRID_COLUMNS):
|
| 148 |
-
values = values[: len(GRID_COLUMNS)]
|
| 149 |
-
row = {}
|
| 150 |
-
for index, column in enumerate(GRID_COLUMNS):
|
| 151 |
-
row[column] = values[index] if index < len(values) else None
|
| 152 |
-
rows.append(row)
|
| 153 |
-
return rows
|
| 154 |
-
except Exception:
|
| 155 |
-
pass
|
| 156 |
-
|
| 157 |
-
for raw_row in table: # type: ignore[assignment]
|
| 158 |
-
values = list(raw_row)
|
| 159 |
-
if len(values) > len(GRID_COLUMNS):
|
| 160 |
-
values = values[: len(GRID_COLUMNS)]
|
| 161 |
-
row = {}
|
| 162 |
-
for index, column in enumerate(GRID_COLUMNS):
|
| 163 |
-
row[column] = values[index] if index < len(values) else None
|
| 164 |
-
rows.append(row)
|
| 165 |
-
|
| 166 |
-
return rows
|
| 167 |
-
|
| 168 |
-
|
| 169 |
-
def get_selected_row_indexes(grid_value: object) -> list[int]:
|
| 170 |
-
if grid_value is None:
|
| 171 |
-
return []
|
| 172 |
-
|
| 173 |
-
selected_indexes: list[int] = []
|
| 174 |
-
|
| 175 |
-
if hasattr(grid_value, "iterrows"):
|
| 176 |
-
try:
|
| 177 |
-
for index, row in grid_value.iterrows():
|
| 178 |
-
values = list(row)
|
| 179 |
-
if values and bool(values[-1]):
|
| 180 |
-
selected_indexes.append(int(index))
|
| 181 |
-
return selected_indexes
|
| 182 |
-
except Exception:
|
| 183 |
-
pass
|
| 184 |
-
|
| 185 |
-
for index, raw_row in enumerate(grid_value): # type: ignore[assignment]
|
| 186 |
-
values = list(raw_row)
|
| 187 |
-
if values and bool(values[-1]):
|
| 188 |
-
selected_indexes.append(index)
|
| 189 |
-
|
| 190 |
-
return selected_indexes
|
| 191 |
-
|
| 192 |
-
|
| 193 |
-
def _build_row_from_document(document: dict) -> dict:
|
| 194 |
-
parsed_data = document.get("parsed_data")
|
| 195 |
-
|
| 196 |
-
row: dict = {column: None for column in RESULT_COLUMNS}
|
| 197 |
-
if isinstance(parsed_data, dict):
|
| 198 |
-
for column in RESULT_COLUMNS:
|
| 199 |
-
if column in parsed_data:
|
| 200 |
-
row[column] = parsed_data.get(column)
|
| 201 |
-
|
| 202 |
-
if document.get("source_file"):
|
| 203 |
-
row["source_file"] = document.get("source_file")
|
| 204 |
-
|
| 205 |
-
row["_document_id"] = str(document.get("document_id") or "").strip()
|
| 206 |
-
row["_modificationDate"] = document.get("modificationDate")
|
| 207 |
-
row["_status"] = document.get("status")
|
| 208 |
-
return row
|
| 209 |
-
|
| 210 |
-
|
| 211 |
@gpu_decorator
|
| 212 |
-
def load_pending_documents():
|
| 213 |
-
if firebase_tracker is None:
|
| 214 |
-
rows: list[dict] = []
|
| 215 |
-
return rows, gr.update(value=rows_to_table(rows)), "Firebase tracking no esta habilitado."
|
| 216 |
-
|
| 217 |
-
try:
|
| 218 |
-
documents = firebase_tracker.list_documents_by_statuses(PENDING_STATUSES)
|
| 219 |
-
except Exception as exc:
|
| 220 |
-
rows = []
|
| 221 |
-
return rows, gr.update(value=rows_to_table(rows)), f"No se pudieron cargar documentos pendientes: {exc}"
|
| 222 |
-
|
| 223 |
-
rows = [_build_row_from_document(document) for document in documents]
|
| 224 |
-
rows = sort_rows_by_fecha(rows)
|
| 225 |
-
return rows, gr.update(value=rows_to_table(rows)), f"Se cargaron {len(rows)} documento(s) con estado Parsed/Modified."
|
| 226 |
-
|
| 227 |
-
|
| 228 |
-
def clear_results():
|
| 229 |
-
return [], gr.update(value=[]), "Grilla limpiada."
|
| 230 |
-
|
| 231 |
-
|
| 232 |
-
def sync_grid_to_state(*args: object):
|
| 233 |
-
if len(args) == 1:
|
| 234 |
-
existing_rows = None
|
| 235 |
-
grid_value = args[0]
|
| 236 |
-
elif len(args) == 2:
|
| 237 |
-
existing_rows, grid_value = args
|
| 238 |
-
else:
|
| 239 |
-
return [], "Grilla actualizada."
|
| 240 |
-
|
| 241 |
-
rows = table_to_rows(grid_value)
|
| 242 |
-
previous_rows = list(existing_rows or []) if isinstance(existing_rows, list) else []
|
| 243 |
-
|
| 244 |
-
merged_rows: list[dict] = []
|
| 245 |
-
for index, row in enumerate(rows):
|
| 246 |
-
merged_row: dict = {column: None for column in RESULT_COLUMNS}
|
| 247 |
-
if index < len(previous_rows):
|
| 248 |
-
previous_row = previous_rows[index]
|
| 249 |
-
if isinstance(previous_row, dict):
|
| 250 |
-
merged_row.update(previous_row)
|
| 251 |
-
for key, value in previous_row.items():
|
| 252 |
-
if key not in RESULT_COLUMNS:
|
| 253 |
-
merged_row[key] = value
|
| 254 |
-
merged_row.update(row)
|
| 255 |
-
merged_rows.append(merged_row)
|
| 256 |
-
|
| 257 |
-
return merged_rows, "Grilla actualizada."
|
| 258 |
-
|
| 259 |
-
|
| 260 |
-
def delete_selected_rows(existing_rows: list[dict] | None, grid_value: object):
|
| 261 |
-
rows = list(existing_rows or [])
|
| 262 |
-
selected_indexes = get_selected_row_indexes(grid_value)
|
| 263 |
-
if not selected_indexes:
|
| 264 |
-
return rows, gr.update(value=rows_to_table(rows)), "Selecciona al menos una fila con el checkbox."
|
| 265 |
-
|
| 266 |
-
for index in selected_indexes:
|
| 267 |
-
if index < 0 or index >= len(rows):
|
| 268 |
-
return rows, gr.update(value=rows_to_table(rows)), "Una o mas filas seleccionadas estan fuera de rango."
|
| 269 |
-
|
| 270 |
-
selected_set = set(selected_indexes)
|
| 271 |
-
remaining_rows = [row for index, row in enumerate(rows) if index not in selected_set]
|
| 272 |
-
|
| 273 |
-
return (
|
| 274 |
-
remaining_rows,
|
| 275 |
-
gr.update(value=rows_to_table(remaining_rows)),
|
| 276 |
-
f"Se eliminaron {len(selected_indexes)} fila(s).",
|
| 277 |
-
)
|
| 278 |
-
|
| 279 |
-
|
| 280 |
-
def _build_updated_parsed_data(row: dict) -> dict:
|
| 281 |
-
return {column: row.get(column) for column in RESULT_COLUMNS}
|
| 282 |
-
|
| 283 |
-
|
| 284 |
-
def send_selected_rows(existing_rows: list[dict] | None, grid_value: object):
|
| 285 |
-
rows = list(existing_rows or [])
|
| 286 |
-
selected_indexes = sorted(set(get_selected_row_indexes(grid_value)))
|
| 287 |
-
display_rows = table_to_display_rows(grid_value)
|
| 288 |
-
if not selected_indexes:
|
| 289 |
-
return rows, gr.update(value=rows_to_table(rows)), "Selecciona al menos una fila con el checkbox."
|
| 290 |
-
|
| 291 |
-
for index in selected_indexes:
|
| 292 |
-
if index < 0 or index >= len(rows):
|
| 293 |
-
return rows, gr.update(value=rows_to_table(rows)), "Una o mas filas seleccionadas estan fuera de rango."
|
| 294 |
-
if index >= len(display_rows):
|
| 295 |
-
return rows, gr.update(value=rows_to_table(rows)), "No se pudo leer una o mas filas seleccionadas de la grilla."
|
| 296 |
-
|
| 297 |
-
sent_indexes: list[int] = []
|
| 298 |
-
try:
|
| 299 |
-
for index in selected_indexes:
|
| 300 |
-
row = rows[index]
|
| 301 |
-
row_for_sheet = {column: display_rows[index].get(column) for column in SHEET_COLUMNS}
|
| 302 |
-
append_row_to_google_sheet(row_for_sheet, SHEET_COLUMNS)
|
| 303 |
-
|
| 304 |
-
if firebase_tracker is not None:
|
| 305 |
-
document_id = str(row.get("_document_id") or "").strip()
|
| 306 |
-
source_file = str(row.get("source_file") or "").strip()
|
| 307 |
-
modification_date = row.get("_modificationDate")
|
| 308 |
-
parsed_data = _build_updated_parsed_data(row)
|
| 309 |
-
if document_id:
|
| 310 |
-
firebase_tracker.save_document_record(
|
| 311 |
-
document_id=document_id,
|
| 312 |
-
source_file=source_file,
|
| 313 |
-
modification_date=modification_date,
|
| 314 |
-
status="sent",
|
| 315 |
-
parsed_data=parsed_data,
|
| 316 |
-
)
|
| 317 |
-
|
| 318 |
-
sent_indexes.append(index)
|
| 319 |
-
except GoogleSheetsConfigError as exc:
|
| 320 |
-
if not sent_indexes:
|
| 321 |
-
return rows, gr.update(value=rows_to_table(rows)), f"Error de configuracion de Google Sheets: {exc}"
|
| 322 |
-
|
| 323 |
-
sent_set = set(sent_indexes)
|
| 324 |
-
remaining_rows = [row for index, row in enumerate(rows) if index not in sent_set]
|
| 325 |
-
return (
|
| 326 |
-
remaining_rows,
|
| 327 |
-
gr.update(value=rows_to_table(remaining_rows)),
|
| 328 |
-
f"Se enviaron {len(sent_indexes)} fila(s) antes de un error de configuracion de Google Sheets: {exc}",
|
| 329 |
-
)
|
| 330 |
-
except Exception as exc:
|
| 331 |
-
if not sent_indexes:
|
| 332 |
-
return rows, gr.update(value=rows_to_table(rows)), f"Error al insertar filas en Google Sheets: {exc}"
|
| 333 |
-
|
| 334 |
-
sent_set = set(sent_indexes)
|
| 335 |
-
remaining_rows = [row for index, row in enumerate(rows) if index not in sent_set]
|
| 336 |
-
return (
|
| 337 |
-
remaining_rows,
|
| 338 |
-
gr.update(value=rows_to_table(remaining_rows)),
|
| 339 |
-
f"Se enviaron {len(sent_indexes)} fila(s) antes del fallo: {exc}",
|
| 340 |
-
)
|
| 341 |
-
|
| 342 |
-
selected_set = set(selected_indexes)
|
| 343 |
-
remaining_rows = [row for index, row in enumerate(rows) if index not in selected_set]
|
| 344 |
-
return (
|
| 345 |
-
remaining_rows,
|
| 346 |
-
gr.update(value=rows_to_table(remaining_rows)),
|
| 347 |
-
f"Se enviaron {len(selected_indexes)} fila(s) a Google Sheets y se actualizaron en Firebase con estado sent.",
|
| 348 |
-
)
|
| 349 |
-
|
| 350 |
-
|
| 351 |
def run_orchestrator_endpoint(model: str, include_subfolders: bool, send_to_sheet: bool, api_key: str) -> dict:
|
|
|
|
| 352 |
expected_api_key = str(os.getenv("ORCHESTRATOR_API_KEY") or "").strip()
|
| 353 |
provided_api_key = str(api_key or "").strip()
|
| 354 |
|
|
@@ -360,7 +41,8 @@ def run_orchestrator_endpoint(model: str, include_subfolders: bool, send_to_shee
|
|
| 360 |
|
| 361 |
model_name = str(model or "").strip() or os.getenv("OPENAI_MODEL", "gpt-4o")
|
| 362 |
try:
|
| 363 |
-
|
|
|
|
| 364 |
model=model_name,
|
| 365 |
include_subfolders=bool(include_subfolders),
|
| 366 |
send_to_sheet=bool(send_to_sheet),
|
|
@@ -373,85 +55,36 @@ def run_orchestrator_endpoint(model: str, include_subfolders: bool, send_to_shee
|
|
| 373 |
|
| 374 |
|
| 375 |
def build_app() -> gr.Blocks:
|
| 376 |
-
initial_rows, _, initial_status = load_pending_documents()
|
| 377 |
-
|
| 378 |
with gr.Blocks(title="Procesador de Documentos") as demo:
|
| 379 |
gr.Markdown("# Procesador de Comprobantes de Compra\nRevisa documentos Parsed/Modified y envia los seleccionados.")
|
| 380 |
|
| 381 |
-
|
| 382 |
-
|
| 383 |
-
|
| 384 |
-
with gr.Row():
|
| 385 |
-
refresh_button = gr.Button("Recargar pendientes", variant="primary")
|
| 386 |
-
clear_button = gr.Button("Limpiar grilla")
|
| 387 |
-
send_row_button = gr.Button("Enviar", variant="secondary")
|
| 388 |
-
delete_row_button = gr.Button("Eliminar seleccionados", variant="stop")
|
| 389 |
-
|
| 390 |
-
with gr.Accordion("Orquestador API", open=False):
|
| 391 |
-
gr.Markdown("Endpoint publico para ejecutar escaneo + parseo + persistencia del orquestador.")
|
| 392 |
-
orchestrator_model = gr.Textbox(
|
| 393 |
label="Modelo OpenAI",
|
| 394 |
value=os.getenv("OPENAI_MODEL", "gpt-4o"),
|
| 395 |
)
|
| 396 |
-
|
| 397 |
label="Incluir subcarpetas",
|
| 398 |
value=True,
|
| 399 |
)
|
| 400 |
-
|
| 401 |
label="Enviar a Google Sheets",
|
| 402 |
-
value=
|
| 403 |
)
|
| 404 |
-
|
| 405 |
label="API Key",
|
| 406 |
type="password",
|
| 407 |
value="",
|
| 408 |
)
|
| 409 |
-
|
| 410 |
-
|
| 411 |
-
|
| 412 |
-
results_grid = gr.Dataframe(
|
| 413 |
-
label="Documentos procesados",
|
| 414 |
-
headers=GRID_HEADER_LABELS,
|
| 415 |
-
value=rows_to_table(initial_rows),
|
| 416 |
-
interactive=True,
|
| 417 |
-
wrap=True,
|
| 418 |
-
)
|
| 419 |
-
|
| 420 |
-
refresh_button.click(
|
| 421 |
-
load_pending_documents,
|
| 422 |
-
inputs=[],
|
| 423 |
-
outputs=[results_state, results_grid, status],
|
| 424 |
-
)
|
| 425 |
-
|
| 426 |
-
clear_button.click(
|
| 427 |
-
clear_results,
|
| 428 |
-
inputs=[],
|
| 429 |
-
outputs=[results_state, results_grid, status],
|
| 430 |
-
)
|
| 431 |
-
|
| 432 |
-
results_grid.change(
|
| 433 |
-
sync_grid_to_state,
|
| 434 |
-
inputs=[results_state, results_grid],
|
| 435 |
-
outputs=[results_state, status],
|
| 436 |
-
)
|
| 437 |
-
|
| 438 |
-
delete_row_button.click(
|
| 439 |
-
delete_selected_rows,
|
| 440 |
-
inputs=[results_state, results_grid],
|
| 441 |
-
outputs=[results_state, results_grid, status],
|
| 442 |
-
)
|
| 443 |
-
|
| 444 |
-
send_row_button.click(
|
| 445 |
-
send_selected_rows,
|
| 446 |
-
inputs=[results_state, results_grid],
|
| 447 |
-
outputs=[results_state, results_grid, status],
|
| 448 |
-
)
|
| 449 |
|
| 450 |
-
|
| 451 |
run_orchestrator_endpoint,
|
| 452 |
-
inputs=[
|
| 453 |
-
outputs=[
|
| 454 |
-
api_name="
|
| 455 |
)
|
| 456 |
|
| 457 |
return demo
|
|
|
|
| 1 |
from __future__ import annotations
|
| 2 |
|
| 3 |
import os
|
|
|
|
| 4 |
|
| 5 |
# Avoid the Node.js SSR proxy in hosted environments where it can terminate unexpectedly.
|
| 6 |
os.environ["GRADIO_SSR_MODE"] = "false"
|
| 7 |
|
| 8 |
import gradio as gr
|
| 9 |
|
| 10 |
+
from document_orchester import orchestrate_all_active_tenants
|
|
|
|
| 11 |
from google_sheets_service import GoogleSheetsConfigError, append_row_to_google_sheet
|
| 12 |
|
| 13 |
try:
|
|
|
|
| 15 |
except ImportError:
|
| 16 |
spaces = None
|
| 17 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 18 |
custom_css = """
|
| 19 |
.cell-menu, .cell-menu-button {
|
| 20 |
display: none !important;
|
|
|
|
| 27 |
return fn
|
| 28 |
return spaces.GPU(fn)
|
| 29 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 30 |
@gpu_decorator
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 31 |
def run_orchestrator_endpoint(model: str, include_subfolders: bool, send_to_sheet: bool, api_key: str) -> dict:
|
| 32 |
+
"""Multi-tenant orchestration endpoint - processes all active tenants and their rules."""
|
| 33 |
expected_api_key = str(os.getenv("ORCHESTRATOR_API_KEY") or "").strip()
|
| 34 |
provided_api_key = str(api_key or "").strip()
|
| 35 |
|
|
|
|
| 41 |
|
| 42 |
model_name = str(model or "").strip() or os.getenv("OPENAI_MODEL", "gpt-4o")
|
| 43 |
try:
|
| 44 |
+
# Load all active tenants from Firebase
|
| 45 |
+
return orchestrate_all_active_tenants(
|
| 46 |
model=model_name,
|
| 47 |
include_subfolders=bool(include_subfolders),
|
| 48 |
send_to_sheet=bool(send_to_sheet),
|
|
|
|
| 55 |
|
| 56 |
|
| 57 |
def build_app() -> gr.Blocks:
|
|
|
|
|
|
|
| 58 |
with gr.Blocks(title="Procesador de Documentos") as demo:
|
| 59 |
gr.Markdown("# Procesador de Comprobantes de Compra\nRevisa documentos Parsed/Modified y envia los seleccionados.")
|
| 60 |
|
| 61 |
+
with gr.Accordion("Orquestador API - Multi Tenant", open=False):
|
| 62 |
+
gr.Markdown("Endpoint publico para ejecutar escaneo + parseo + persistencia del orquestador en modo multi-tenant (carga todos los tenants activos de Firebase).")
|
| 63 |
+
mt_orchestrator_model = gr.Textbox(
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 64 |
label="Modelo OpenAI",
|
| 65 |
value=os.getenv("OPENAI_MODEL", "gpt-4o"),
|
| 66 |
)
|
| 67 |
+
mt_orchestrator_subfolders = gr.Checkbox(
|
| 68 |
label="Incluir subcarpetas",
|
| 69 |
value=True,
|
| 70 |
)
|
| 71 |
+
mt_orchestrator_send = gr.Checkbox(
|
| 72 |
label="Enviar a Google Sheets",
|
| 73 |
+
value=True,
|
| 74 |
)
|
| 75 |
+
mt_orchestrator_api_key = gr.Textbox(
|
| 76 |
label="API Key",
|
| 77 |
type="password",
|
| 78 |
value="",
|
| 79 |
)
|
| 80 |
+
mt_orchestrator_button = gr.Button("Ejecutar orquestador (Multi Tenant)", variant="primary")
|
| 81 |
+
mt_orchestrator_output = gr.JSON(label="Resultado orquestacion")
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 82 |
|
| 83 |
+
mt_orchestrator_button.click(
|
| 84 |
run_orchestrator_endpoint,
|
| 85 |
+
inputs=[mt_orchestrator_model, mt_orchestrator_subfolders, mt_orchestrator_send, mt_orchestrator_api_key],
|
| 86 |
+
outputs=[mt_orchestrator_output],
|
| 87 |
+
api_name="orchestrate_all_tenants",
|
| 88 |
)
|
| 89 |
|
| 90 |
return demo
|
document_orchester.py
CHANGED
|
@@ -5,7 +5,7 @@ import json
|
|
| 5 |
import logging
|
| 6 |
import os
|
| 7 |
import shutil
|
| 8 |
-
from datetime import datetime
|
| 9 |
from pathlib import Path
|
| 10 |
from typing import Any
|
| 11 |
|
|
@@ -17,14 +17,18 @@ from firebase_processed_files import (
|
|
| 17 |
build_firebase_tracker,
|
| 18 |
check_drive_documents_to_process,
|
| 19 |
)
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 20 |
from google_drive_service import (
|
| 21 |
DRIVE_FULL_SCOPE,
|
| 22 |
GoogleDriveConfigError,
|
| 23 |
build_drive_service,
|
| 24 |
move_file_to_path,
|
| 25 |
-
|
| 26 |
-
from google_drive_scanner import (
|
| 27 |
-
scan_google_drive_supported_documents,
|
| 28 |
)
|
| 29 |
from google_sheets_service import GoogleSheetsConfigError, append_row_to_google_sheet
|
| 30 |
from receipt_ai import extract_receipt_json, extract_receipt_json_from_image
|
|
@@ -45,14 +49,6 @@ logger.propagate = False
|
|
| 45 |
|
| 46 |
SHEET_COLUMNS = ["fecha", "source_file", "display_description", "total"]
|
| 47 |
CORRUPTED_SHEET_NAME = "corrupted data"
|
| 48 |
-
MANDATORY_ENV_VARS = [
|
| 49 |
-
"OPENAI_API_KEY",
|
| 50 |
-
"GOOGLE_SERVICE_ACCOUNT_JSON",
|
| 51 |
-
"GOOGLE_SHEETS_SPREADSHEET_ID",
|
| 52 |
-
"GOOGLE_DRIVE_FOLDER_IDS",
|
| 53 |
-
"GOOGLE_DRIVE_INVOICES_ROOT_FOLDER_ID",
|
| 54 |
-
"ORCHESTRATOR_API_KEY",
|
| 55 |
-
]
|
| 56 |
|
| 57 |
|
| 58 |
def _build_display_description(parsed: dict[str, Any]) -> str:
|
|
@@ -86,26 +82,13 @@ def _is_complete_sheet_row(row: dict[str, Any]) -> bool:
|
|
| 86 |
return all(not _is_blank_sheet_value(row.get(column)) for column in SHEET_COLUMNS)
|
| 87 |
|
| 88 |
|
| 89 |
-
def
|
| 90 |
-
|
| 91 |
-
if not api_key:
|
| 92 |
raise RuntimeError("Missing OPENAI_API_KEY.")
|
| 93 |
return OpenAI(api_key=api_key)
|
| 94 |
|
| 95 |
|
| 96 |
-
def _require_env_var(name: str) -> str:
|
| 97 |
-
value = (os.getenv(name) or "").strip()
|
| 98 |
-
if not value:
|
| 99 |
-
raise RuntimeError(f"Missing {name}.")
|
| 100 |
-
return value
|
| 101 |
-
|
| 102 |
-
|
| 103 |
-
def _validate_mandatory_env_vars() -> None:
|
| 104 |
-
missing_vars = [name for name in MANDATORY_ENV_VARS if not (os.getenv(name) or "").strip()]
|
| 105 |
-
if missing_vars:
|
| 106 |
-
raise RuntimeError(f"Missing mandatory env vars: {', '.join(missing_vars)}")
|
| 107 |
-
|
| 108 |
-
|
| 109 |
def _parse_invoice_date(value: Any) -> datetime | None:
|
| 110 |
text = str(value or "").strip()
|
| 111 |
if not text:
|
|
@@ -162,35 +145,160 @@ def _parse_document(client: OpenAI, model: str, local_path: str, source_file: st
|
|
| 162 |
return parsed
|
| 163 |
|
| 164 |
|
| 165 |
-
def
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 166 |
model: str,
|
| 167 |
include_subfolders: bool = True,
|
| 168 |
-
send_to_sheet: bool =
|
| 169 |
) -> dict[str, Any]:
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 170 |
logger.info(
|
| 171 |
-
"Starting orchestration
|
|
|
|
|
|
|
| 172 |
model,
|
| 173 |
-
include_subfolders,
|
| 174 |
-
send_to_sheet,
|
| 175 |
)
|
| 176 |
-
|
| 177 |
-
|
| 178 |
-
|
| 179 |
-
|
| 180 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 181 |
scan_temp_dir: str | None = None
|
| 182 |
try:
|
|
|
|
|
|
|
| 183 |
drive_service = build_drive_service(scope=DRIVE_FULL_SCOPE)
|
| 184 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 185 |
except GoogleDriveConfigError as exc:
|
| 186 |
-
logger.error("Google Drive
|
| 187 |
return {
|
| 188 |
"ok": False,
|
| 189 |
"error": f"Google Drive configuration error: {exc}",
|
|
|
|
|
|
|
| 190 |
}
|
| 191 |
-
|
| 192 |
-
logger.info(
|
| 193 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 194 |
docs_payload = [
|
| 195 |
{
|
| 196 |
"document_id": doc.document_id,
|
|
@@ -200,12 +308,20 @@ def orchestrate_drive_documents(
|
|
| 200 |
}
|
| 201 |
for doc in drive_documents
|
| 202 |
]
|
| 203 |
-
|
|
|
|
| 204 |
to_process, skipped, tracking_warning = check_drive_documents_to_process(tracker, docs_payload)
|
| 205 |
-
logger.info(
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 206 |
if tracking_warning:
|
| 207 |
-
logger.warning("
|
| 208 |
-
|
|
|
|
| 209 |
parsed_count = 0
|
| 210 |
modified_count = 0
|
| 211 |
sent_count = 0
|
|
@@ -213,103 +329,148 @@ def orchestrate_drive_documents(
|
|
| 213 |
corrupted_count = 0
|
| 214 |
error_items: list[dict[str, str]] = []
|
| 215 |
processed_items: list[dict[str, Any]] = []
|
| 216 |
-
|
| 217 |
for document in to_process:
|
| 218 |
document_id = str(document.get("document_id") or "").strip()
|
| 219 |
source_file = str(document.get("source_file") or "").strip()
|
| 220 |
local_path = str(document.get("local_path") or "").strip()
|
| 221 |
modification_date = document.get("modificationDate")
|
| 222 |
status = str(document.get("target_status") or "Parsed")
|
| 223 |
-
|
| 224 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 225 |
try:
|
| 226 |
parsed = _parse_document(client, model, local_path, source_file)
|
| 227 |
-
logger.info("Parsed document_id=%s source_file=%s", document_id, source_file)
|
| 228 |
-
|
| 229 |
-
destination_path, is_corrupted = _resolve_drive_destination_path(parsed)
|
| 230 |
-
if is_corrupted:
|
| 231 |
-
corrupted_count += 1
|
| 232 |
-
move_file_to_path(
|
| 233 |
-
service=drive_service,
|
| 234 |
-
file_id=document_id,
|
| 235 |
-
destination_path=destination_path,
|
| 236 |
-
root_folder_id=drive_root_folder_id,
|
| 237 |
-
)
|
| 238 |
-
moved_count += 1
|
| 239 |
logger.info(
|
| 240 |
-
"
|
|
|
|
|
|
|
| 241 |
document_id,
|
| 242 |
-
source_file,
|
| 243 |
-
destination_path,
|
| 244 |
)
|
| 245 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 246 |
if tracker is not None:
|
| 247 |
tracker.save_document_record(
|
|
|
|
| 248 |
document_id=document_id,
|
| 249 |
source_file=source_file,
|
| 250 |
modification_date=modification_date,
|
| 251 |
status=status,
|
| 252 |
parsed_data=parsed,
|
| 253 |
)
|
| 254 |
-
|
| 255 |
if status == "Modified":
|
| 256 |
modified_count += 1
|
| 257 |
else:
|
| 258 |
parsed_count += 1
|
| 259 |
-
|
|
|
|
| 260 |
worksheet_name: str | None = None
|
| 261 |
-
if
|
|
|
|
| 262 |
row_for_sheet = _build_sheet_row(parsed)
|
| 263 |
row_for_sheet["source_file"] = _build_hyperlink_formula(
|
| 264 |
_build_drive_file_url(document_id),
|
| 265 |
source_file,
|
| 266 |
)
|
| 267 |
-
worksheet_name =
|
| 268 |
-
|
| 269 |
-
|
| 270 |
-
|
| 271 |
-
|
| 272 |
-
|
| 273 |
-
|
| 274 |
-
|
| 275 |
-
|
| 276 |
-
|
| 277 |
-
|
| 278 |
-
|
| 279 |
-
|
| 280 |
-
|
| 281 |
-
|
| 282 |
-
|
| 283 |
-
|
| 284 |
-
|
| 285 |
-
|
| 286 |
-
|
| 287 |
-
|
| 288 |
-
|
| 289 |
-
|
| 290 |
-
|
| 291 |
-
|
| 292 |
-
|
| 293 |
-
|
| 294 |
-
|
| 295 |
-
|
| 296 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 297 |
except Exception as exc:
|
| 298 |
-
logger.exception(
|
| 299 |
-
|
| 300 |
-
|
| 301 |
-
|
| 302 |
-
|
| 303 |
-
"error": str(exc),
|
| 304 |
-
}
|
| 305 |
)
|
| 306 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 307 |
if scan_temp_dir:
|
| 308 |
shutil.rmtree(scan_temp_dir, ignore_errors=True)
|
| 309 |
-
logger.info("
|
| 310 |
-
|
| 311 |
logger.info(
|
| 312 |
-
"
|
|
|
|
|
|
|
| 313 |
parsed_count,
|
| 314 |
modified_count,
|
| 315 |
sent_count,
|
|
@@ -317,9 +478,12 @@ def orchestrate_drive_documents(
|
|
| 317 |
corrupted_count,
|
| 318 |
len(error_items),
|
| 319 |
)
|
| 320 |
-
|
| 321 |
return {
|
| 322 |
"ok": True,
|
|
|
|
|
|
|
|
|
|
| 323 |
"scanned": len(drive_documents),
|
| 324 |
"to_process": len(to_process),
|
| 325 |
"skipped": len(skipped),
|
|
@@ -335,9 +499,200 @@ def orchestrate_drive_documents(
|
|
| 335 |
}
|
| 336 |
|
| 337 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 338 |
def parse_args() -> argparse.Namespace:
|
| 339 |
-
parser = argparse.ArgumentParser(
|
| 340 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 341 |
parser.add_argument(
|
| 342 |
"--no-subfolders",
|
| 343 |
action="store_true",
|
|
@@ -358,8 +713,13 @@ def parse_args() -> argparse.Namespace:
|
|
| 358 |
|
| 359 |
def main() -> int:
|
| 360 |
args = parse_args()
|
| 361 |
-
logger.info(
|
| 362 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 363 |
model=args.model,
|
| 364 |
include_subfolders=not args.no_subfolders,
|
| 365 |
send_to_sheet=args.send,
|
|
|
|
| 5 |
import logging
|
| 6 |
import os
|
| 7 |
import shutil
|
| 8 |
+
from datetime import datetime, timezone
|
| 9 |
from pathlib import Path
|
| 10 |
from typing import Any
|
| 11 |
|
|
|
|
| 17 |
build_firebase_tracker,
|
| 18 |
check_drive_documents_to_process,
|
| 19 |
)
|
| 20 |
+
from firebase_tenant_config import (
|
| 21 |
+
FirebaseTenantConfigManager,
|
| 22 |
+
CredentialsObject,
|
| 23 |
+
RuleObject,
|
| 24 |
+
TenantConfig,
|
| 25 |
+
)
|
| 26 |
from google_drive_service import (
|
| 27 |
DRIVE_FULL_SCOPE,
|
| 28 |
GoogleDriveConfigError,
|
| 29 |
build_drive_service,
|
| 30 |
move_file_to_path,
|
| 31 |
+
scan_drive_supported_documents,
|
|
|
|
|
|
|
| 32 |
)
|
| 33 |
from google_sheets_service import GoogleSheetsConfigError, append_row_to_google_sheet
|
| 34 |
from receipt_ai import extract_receipt_json, extract_receipt_json_from_image
|
|
|
|
| 49 |
|
| 50 |
SHEET_COLUMNS = ["fecha", "source_file", "display_description", "total"]
|
| 51 |
CORRUPTED_SHEET_NAME = "corrupted data"
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 52 |
|
| 53 |
|
| 54 |
def _build_display_description(parsed: dict[str, Any]) -> str:
|
|
|
|
| 82 |
return all(not _is_blank_sheet_value(row.get(column)) for column in SHEET_COLUMNS)
|
| 83 |
|
| 84 |
|
| 85 |
+
def _create_openai_client_from_key(api_key: str) -> OpenAI:
|
| 86 |
+
"""Create OpenAI client from API key."""
|
| 87 |
+
if not api_key.strip():
|
| 88 |
raise RuntimeError("Missing OPENAI_API_KEY.")
|
| 89 |
return OpenAI(api_key=api_key)
|
| 90 |
|
| 91 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 92 |
def _parse_invoice_date(value: Any) -> datetime | None:
|
| 93 |
text = str(value or "").strip()
|
| 94 |
if not text:
|
|
|
|
| 145 |
return parsed
|
| 146 |
|
| 147 |
|
| 148 |
+
def _create_default_tenant_from_env() -> tuple[TenantConfig, RuleObject] | None:
|
| 149 |
+
"""Create a default tenant from environment variables if possible.
|
| 150 |
+
|
| 151 |
+
Maps legacy env vars to a default multi-tenant configuration:
|
| 152 |
+
- OPENAI_API_KEY -> credentials.openai_api_key
|
| 153 |
+
- GOOGLE_SERVICE_ACCOUNT_JSON -> credentials.google_service_account_json
|
| 154 |
+
- GOOGLE_DRIVE_FOLDER_IDS -> rule.source_folder_id (first one)
|
| 155 |
+
- GOOGLE_DRIVE_INVOICES_ROOT_FOLDER_ID -> rule.target_folder_id
|
| 156 |
+
- GOOGLE_SHEETS_SPREADSHEET_ID -> rule.target_sheet_id
|
| 157 |
+
|
| 158 |
+
Returns:
|
| 159 |
+
TenantConfig if all required env vars are set, None otherwise
|
| 160 |
+
"""
|
| 161 |
+
openai_api_key = (os.getenv("OPENAI_API_KEY") or "").strip()
|
| 162 |
+
google_service_account_json = (os.getenv("GOOGLE_SERVICE_ACCOUNT_JSON") or "").strip()
|
| 163 |
+
drive_folder_ids = (os.getenv("GOOGLE_DRIVE_FOLDER_IDS") or "").strip()
|
| 164 |
+
target_folder_id = (os.getenv("GOOGLE_DRIVE_INVOICES_ROOT_FOLDER_ID") or "").strip()
|
| 165 |
+
target_sheet_id = (os.getenv("GOOGLE_SHEETS_SPREADSHEET_ID") or "").strip()
|
| 166 |
+
sheet_tab_name = (os.getenv("GOOGLE_SHEETS_WORKSHEET") or "Salidas").strip()
|
| 167 |
+
|
| 168 |
+
# Check if all required vars are set
|
| 169 |
+
if not all([openai_api_key, google_service_account_json, drive_folder_ids, target_folder_id, target_sheet_id]):
|
| 170 |
+
logger.warning(
|
| 171 |
+
"Cannot create default tenant: missing env vars. "
|
| 172 |
+
"Please ensure OPENAI_API_KEY, GOOGLE_SERVICE_ACCOUNT_JSON, GOOGLE_DRIVE_FOLDER_IDS, "
|
| 173 |
+
"GOOGLE_DRIVE_INVOICES_ROOT_FOLDER_ID, and GOOGLE_SHEETS_SPREADSHEET_ID are set."
|
| 174 |
+
)
|
| 175 |
+
return None
|
| 176 |
+
|
| 177 |
+
# Get first folder ID from comma-separated list
|
| 178 |
+
source_folder_id = drive_folder_ids.split(",")[0].strip()
|
| 179 |
+
|
| 180 |
+
# Create credentials
|
| 181 |
+
credentials = CredentialsObject(
|
| 182 |
+
openai_api_key=openai_api_key,
|
| 183 |
+
google_service_account_json=google_service_account_json,
|
| 184 |
+
)
|
| 185 |
+
|
| 186 |
+
# Create default rule
|
| 187 |
+
default_rule = RuleObject(
|
| 188 |
+
rule_id="default_rule",
|
| 189 |
+
rule_name="Default Processing Rule",
|
| 190 |
+
source_folder_id=source_folder_id,
|
| 191 |
+
target_folder_id=target_folder_id,
|
| 192 |
+
target_sheet_id=target_sheet_id,
|
| 193 |
+
sheet_tab_name=sheet_tab_name,
|
| 194 |
+
parsing_prompt=None,
|
| 195 |
+
is_enabled=True,
|
| 196 |
+
)
|
| 197 |
+
|
| 198 |
+
# Create tenant (rules will be added as subcollection)
|
| 199 |
+
tenant = TenantConfig(
|
| 200 |
+
tenant_id="default_tenant",
|
| 201 |
+
name="Default Tenant (from .env)",
|
| 202 |
+
active=True,
|
| 203 |
+
credentials=credentials,
|
| 204 |
+
created_at=datetime.now(timezone.utc).isoformat(),
|
| 205 |
+
)
|
| 206 |
+
|
| 207 |
+
logger.info("Created default tenant from environment variables")
|
| 208 |
+
return tenant, default_rule
|
| 209 |
+
|
| 210 |
+
|
| 211 |
+
def orchestrate_single_rule(
|
| 212 |
+
tenant_id: str,
|
| 213 |
+
tenant_config: TenantConfig,
|
| 214 |
+
rule: RuleObject,
|
| 215 |
model: str,
|
| 216 |
include_subfolders: bool = True,
|
| 217 |
+
send_to_sheet: bool = True,
|
| 218 |
) -> dict[str, Any]:
|
| 219 |
+
"""Orchestrate document processing for a single rule of a single tenant.
|
| 220 |
+
|
| 221 |
+
Args:
|
| 222 |
+
tenant_id: Unique tenant identifier
|
| 223 |
+
tenant_config: Tenant configuration (with credentials and rules)
|
| 224 |
+
rule: The processing rule to orchestrate
|
| 225 |
+
model: OpenAI model to use for parsing
|
| 226 |
+
include_subfolders: Whether to include subfolders when scanning Drive
|
| 227 |
+
send_to_sheet: Whether to append results to Google Sheets
|
| 228 |
+
|
| 229 |
+
Returns:
|
| 230 |
+
Summary dict with processing results
|
| 231 |
+
"""
|
| 232 |
logger.info(
|
| 233 |
+
"Starting orchestration for tenant=%s rule=%s model=%s",
|
| 234 |
+
tenant_id,
|
| 235 |
+
rule.rule_id,
|
| 236 |
model,
|
|
|
|
|
|
|
| 237 |
)
|
| 238 |
+
|
| 239 |
+
# Validate tenant credentials
|
| 240 |
+
if not tenant_config.credentials.openai_api_key:
|
| 241 |
+
logger.error("Tenant=%s rule=%s missing openai_api_key", tenant_id, rule.rule_id)
|
| 242 |
+
return {
|
| 243 |
+
"ok": False,
|
| 244 |
+
"error": f"Tenant {tenant_id} missing OpenAI API key",
|
| 245 |
+
"tenant_id": tenant_id,
|
| 246 |
+
"rule_id": rule.rule_id,
|
| 247 |
+
}
|
| 248 |
+
|
| 249 |
+
if not tenant_config.credentials.google_service_account_json:
|
| 250 |
+
logger.error("Tenant=%s rule=%s missing google_service_account_json", tenant_id, rule.rule_id)
|
| 251 |
+
return {
|
| 252 |
+
"ok": False,
|
| 253 |
+
"error": f"Tenant {tenant_id} missing Google service account credentials",
|
| 254 |
+
"tenant_id": tenant_id,
|
| 255 |
+
"rule_id": rule.rule_id,
|
| 256 |
+
}
|
| 257 |
+
|
| 258 |
+
# Set up clients for this tenant
|
| 259 |
+
try:
|
| 260 |
+
client = _create_openai_client_from_key(tenant_config.credentials.openai_api_key)
|
| 261 |
+
tracker = build_firebase_tracker(tenant_id=tenant_id, rule_id=rule.rule_id)
|
| 262 |
+
except RuntimeError as exc:
|
| 263 |
+
logger.error("Tenant=%s rule=%s failed to initialize clients: %s", tenant_id, rule.rule_id, exc)
|
| 264 |
+
return {
|
| 265 |
+
"ok": False,
|
| 266 |
+
"error": str(exc),
|
| 267 |
+
"tenant_id": tenant_id,
|
| 268 |
+
"rule_id": rule.rule_id,
|
| 269 |
+
}
|
| 270 |
+
|
| 271 |
+
# Set up Google Drive service with tenant's credentials
|
| 272 |
scan_temp_dir: str | None = None
|
| 273 |
try:
|
| 274 |
+
# Build drive service with tenant credentials
|
| 275 |
+
os.environ["GOOGLE_SERVICE_ACCOUNT_JSON"] = tenant_config.credentials.google_service_account_json
|
| 276 |
drive_service = build_drive_service(scope=DRIVE_FULL_SCOPE)
|
| 277 |
+
|
| 278 |
+
# Scan the rule's source folder
|
| 279 |
+
scan_result = scan_drive_supported_documents(
|
| 280 |
+
folder_ids=[rule.source_folder_id],
|
| 281 |
+
include_subfolders=include_subfolders,
|
| 282 |
+
)
|
| 283 |
+
drive_documents = scan_result.documents
|
| 284 |
+
scan_temp_dir = scan_result.temp_dir
|
| 285 |
except GoogleDriveConfigError as exc:
|
| 286 |
+
logger.error("Tenant=%s rule=%s Google Drive config error: %s", tenant_id, rule.rule_id, exc)
|
| 287 |
return {
|
| 288 |
"ok": False,
|
| 289 |
"error": f"Google Drive configuration error: {exc}",
|
| 290 |
+
"tenant_id": tenant_id,
|
| 291 |
+
"rule_id": rule.rule_id,
|
| 292 |
}
|
| 293 |
+
|
| 294 |
+
logger.info(
|
| 295 |
+
"Tenant=%s rule=%s scan completed documents_found=%s",
|
| 296 |
+
tenant_id,
|
| 297 |
+
rule.rule_id,
|
| 298 |
+
len(drive_documents),
|
| 299 |
+
)
|
| 300 |
+
|
| 301 |
+
# Prepare docs payload
|
| 302 |
docs_payload = [
|
| 303 |
{
|
| 304 |
"document_id": doc.document_id,
|
|
|
|
| 308 |
}
|
| 309 |
for doc in drive_documents
|
| 310 |
]
|
| 311 |
+
|
| 312 |
+
# Check which documents need processing
|
| 313 |
to_process, skipped, tracking_warning = check_drive_documents_to_process(tracker, docs_payload)
|
| 314 |
+
logger.info(
|
| 315 |
+
"Tenant=%s rule=%s to_process=%s skipped=%s",
|
| 316 |
+
tenant_id,
|
| 317 |
+
rule.rule_id,
|
| 318 |
+
len(to_process),
|
| 319 |
+
len(skipped),
|
| 320 |
+
)
|
| 321 |
if tracking_warning:
|
| 322 |
+
logger.warning("Tenant=%s rule=%s tracking warning: %s", tenant_id, rule.rule_id, tracking_warning)
|
| 323 |
+
|
| 324 |
+
# Process documents
|
| 325 |
parsed_count = 0
|
| 326 |
modified_count = 0
|
| 327 |
sent_count = 0
|
|
|
|
| 329 |
corrupted_count = 0
|
| 330 |
error_items: list[dict[str, str]] = []
|
| 331 |
processed_items: list[dict[str, Any]] = []
|
| 332 |
+
|
| 333 |
for document in to_process:
|
| 334 |
document_id = str(document.get("document_id") or "").strip()
|
| 335 |
source_file = str(document.get("source_file") or "").strip()
|
| 336 |
local_path = str(document.get("local_path") or "").strip()
|
| 337 |
modification_date = document.get("modificationDate")
|
| 338 |
status = str(document.get("target_status") or "Parsed")
|
| 339 |
+
file_hash = str(document.get("file_hash") or "").strip() or None
|
| 340 |
+
|
| 341 |
+
logger.info(
|
| 342 |
+
"Tenant=%s rule=%s processing document_id=%s source_file=%s",
|
| 343 |
+
tenant_id,
|
| 344 |
+
rule.rule_id,
|
| 345 |
+
document_id,
|
| 346 |
+
source_file,
|
| 347 |
+
)
|
| 348 |
+
|
| 349 |
+
if tracker.is_processed(file_hash):
|
| 350 |
+
logger.info(
|
| 351 |
+
"Tenant=%s rule=%s skipping document_id=%s already processed",
|
| 352 |
+
tenant_id,
|
| 353 |
+
rule.rule_id,
|
| 354 |
+
document_id,
|
| 355 |
+
)
|
| 356 |
+
skipped.append({"document_id": document_id, "source_file": source_file, "reason": "Already processed"})
|
| 357 |
+
continue
|
| 358 |
+
|
| 359 |
try:
|
| 360 |
parsed = _parse_document(client, model, local_path, source_file)
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 361 |
logger.info(
|
| 362 |
+
"Tenant=%s rule=%s parsed document_id=%s",
|
| 363 |
+
tenant_id,
|
| 364 |
+
rule.rule_id,
|
| 365 |
document_id,
|
|
|
|
|
|
|
| 366 |
)
|
| 367 |
+
|
| 368 |
+
# Determine destination and whether corrupted
|
| 369 |
+
destination_path, is_corrupted = _resolve_drive_destination_path(parsed)
|
| 370 |
+
if is_corrupted:
|
| 371 |
+
corrupted_count += 1
|
| 372 |
+
|
| 373 |
+
# Move file to destination (if target_folder_id is set)
|
| 374 |
+
if rule.target_folder_id:
|
| 375 |
+
move_file_to_path(
|
| 376 |
+
service=drive_service,
|
| 377 |
+
file_id=document_id,
|
| 378 |
+
destination_path=destination_path,
|
| 379 |
+
root_folder_id=rule.target_folder_id,
|
| 380 |
+
)
|
| 381 |
+
moved_count += 1
|
| 382 |
+
logger.info(
|
| 383 |
+
"Tenant=%s rule=%s moved document_id=%s to %s",
|
| 384 |
+
tenant_id,
|
| 385 |
+
rule.rule_id,
|
| 386 |
+
document_id,
|
| 387 |
+
destination_path,
|
| 388 |
+
)
|
| 389 |
+
|
| 390 |
+
# Save document record
|
| 391 |
if tracker is not None:
|
| 392 |
tracker.save_document_record(
|
| 393 |
+
file_hash=file_hash,
|
| 394 |
document_id=document_id,
|
| 395 |
source_file=source_file,
|
| 396 |
modification_date=modification_date,
|
| 397 |
status=status,
|
| 398 |
parsed_data=parsed,
|
| 399 |
)
|
| 400 |
+
|
| 401 |
if status == "Modified":
|
| 402 |
modified_count += 1
|
| 403 |
else:
|
| 404 |
parsed_count += 1
|
| 405 |
+
|
| 406 |
+
# Send to Google Sheets if enabled
|
| 407 |
worksheet_name: str | None = None
|
| 408 |
+
if rule.target_sheet_id:
|
| 409 |
+
os.environ["GOOGLE_SHEETS_SPREADSHEET_ID"] = rule.target_sheet_id
|
| 410 |
row_for_sheet = _build_sheet_row(parsed)
|
| 411 |
row_for_sheet["source_file"] = _build_hyperlink_formula(
|
| 412 |
_build_drive_file_url(document_id),
|
| 413 |
source_file,
|
| 414 |
)
|
| 415 |
+
worksheet_name = "Corrupted_data" if not _is_complete_sheet_row(row_for_sheet) else rule.sheet_tab_name
|
| 416 |
+
|
| 417 |
+
try:
|
| 418 |
+
append_row_to_google_sheet(row_for_sheet, SHEET_COLUMNS, worksheet_name=worksheet_name)
|
| 419 |
+
logger.info(
|
| 420 |
+
"Tenant=%s rule=%s sent document_id=%s to sheet=%s",
|
| 421 |
+
tenant_id,
|
| 422 |
+
rule.rule_id,
|
| 423 |
+
document_id,
|
| 424 |
+
worksheet_name,
|
| 425 |
+
)
|
| 426 |
+
if tracker is not None and file_hash:
|
| 427 |
+
tracker.mark_document_sent(file_hash)
|
| 428 |
+
sent_count += 1
|
| 429 |
+
except GoogleSheetsConfigError as exc:
|
| 430 |
+
logger.error(
|
| 431 |
+
"Tenant=%s rule=%s sheet error for document_id=%s: %s",
|
| 432 |
+
tenant_id,
|
| 433 |
+
rule.rule_id,
|
| 434 |
+
document_id,
|
| 435 |
+
exc,
|
| 436 |
+
)
|
| 437 |
+
error_items.append({
|
| 438 |
+
"document_id": document_id,
|
| 439 |
+
"source_file": source_file,
|
| 440 |
+
"error": f"Google Sheets error: {exc}",
|
| 441 |
+
})
|
| 442 |
+
continue
|
| 443 |
+
|
| 444 |
+
processed_items.append({
|
| 445 |
+
"document_id": document_id,
|
| 446 |
+
"source_file": source_file,
|
| 447 |
+
"modificationDate": modification_date,
|
| 448 |
+
"status": "Corrupted" if is_corrupted else ("Sent" if send_to_sheet else status),
|
| 449 |
+
"destination_path": destination_path,
|
| 450 |
+
})
|
| 451 |
+
|
| 452 |
except Exception as exc:
|
| 453 |
+
logger.exception(
|
| 454 |
+
"Tenant=%s rule=%s failed processing document_id=%s",
|
| 455 |
+
tenant_id,
|
| 456 |
+
rule.rule_id,
|
| 457 |
+
document_id,
|
|
|
|
|
|
|
| 458 |
)
|
| 459 |
+
error_items.append({
|
| 460 |
+
"document_id": document_id,
|
| 461 |
+
"source_file": source_file,
|
| 462 |
+
"error": str(exc),
|
| 463 |
+
})
|
| 464 |
+
|
| 465 |
+
# Cleanup
|
| 466 |
if scan_temp_dir:
|
| 467 |
shutil.rmtree(scan_temp_dir, ignore_errors=True)
|
| 468 |
+
logger.info("Tenant=%s rule=%s cleaned up temp directory", tenant_id, rule.rule_id)
|
| 469 |
+
|
| 470 |
logger.info(
|
| 471 |
+
"Tenant=%s rule=%s orchestration finished parsed=%s modified=%s sent=%s moved=%s corrupted=%s errors=%s",
|
| 472 |
+
tenant_id,
|
| 473 |
+
rule.rule_id,
|
| 474 |
parsed_count,
|
| 475 |
modified_count,
|
| 476 |
sent_count,
|
|
|
|
| 478 |
corrupted_count,
|
| 479 |
len(error_items),
|
| 480 |
)
|
| 481 |
+
|
| 482 |
return {
|
| 483 |
"ok": True,
|
| 484 |
+
"tenant_id": tenant_id,
|
| 485 |
+
"rule_id": rule.rule_id,
|
| 486 |
+
"rule_name": rule.rule_name,
|
| 487 |
"scanned": len(drive_documents),
|
| 488 |
"to_process": len(to_process),
|
| 489 |
"skipped": len(skipped),
|
|
|
|
| 499 |
}
|
| 500 |
|
| 501 |
|
| 502 |
+
def orchestrate_all_active_tenants(
|
| 503 |
+
model: str,
|
| 504 |
+
include_subfolders: bool = True,
|
| 505 |
+
send_to_sheet: bool = True,
|
| 506 |
+
) -> dict[str, Any]:
|
| 507 |
+
"""Orchestrate all active tenants and their enabled rules.
|
| 508 |
+
|
| 509 |
+
This is the main multi-tenant entry point that:
|
| 510 |
+
1. Loads all active tenants from Firebase
|
| 511 |
+
2. For each tenant, processes all enabled rules
|
| 512 |
+
3. Aggregates results across all tenants
|
| 513 |
+
|
| 514 |
+
Args:
|
| 515 |
+
model: OpenAI model to use for parsing
|
| 516 |
+
include_subfolders: Whether to include subfolders when scanning Drive
|
| 517 |
+
send_to_sheet: Whether to append results to Google Sheets
|
| 518 |
+
|
| 519 |
+
Returns:
|
| 520 |
+
Summary dict with aggregated processing results
|
| 521 |
+
"""
|
| 522 |
+
logger.info(
|
| 523 |
+
"Starting multi-tenant orchestration model=%s include_subfolders=%s send_to_sheet=%s",
|
| 524 |
+
model,
|
| 525 |
+
include_subfolders,
|
| 526 |
+
send_to_sheet,
|
| 527 |
+
)
|
| 528 |
+
|
| 529 |
+
# Load all active tenants from Firebase
|
| 530 |
+
config_manager = FirebaseTenantConfigManager()
|
| 531 |
+
default_tenant_created = False
|
| 532 |
+
default_tenant_saved = False
|
| 533 |
+
|
| 534 |
+
try:
|
| 535 |
+
active_tenants = config_manager.list_active_tenants()
|
| 536 |
+
except Exception as exc:
|
| 537 |
+
logger.error("Failed to load active tenants: %s", exc)
|
| 538 |
+
return {
|
| 539 |
+
"ok": False,
|
| 540 |
+
"error": f"Failed to load tenants: {exc}",
|
| 541 |
+
"tenants_processed": 0,
|
| 542 |
+
}
|
| 543 |
+
|
| 544 |
+
logger.info("Loaded %s active tenants", len(active_tenants))
|
| 545 |
+
|
| 546 |
+
# If no tenants found, create and save default tenant BEFORE orchestration
|
| 547 |
+
if not active_tenants:
|
| 548 |
+
logger.warning("No active tenants found. Creating default tenant from .env...")
|
| 549 |
+
result = _create_default_tenant_from_env()
|
| 550 |
+
|
| 551 |
+
if not result:
|
| 552 |
+
return {
|
| 553 |
+
"ok": False,
|
| 554 |
+
"error": "No active tenants found in Firebase and could not create default tenant from .env",
|
| 555 |
+
"tenants_processed": 0,
|
| 556 |
+
"tenants_results": [],
|
| 557 |
+
}
|
| 558 |
+
|
| 559 |
+
default_tenant, default_rule = result
|
| 560 |
+
|
| 561 |
+
# Save default tenant to Firebase BEFORE orchestration
|
| 562 |
+
try:
|
| 563 |
+
logger.info("Saving default tenant '%s' to Firebase...", default_tenant.tenant_id)
|
| 564 |
+
config_manager.save_tenant(default_tenant)
|
| 565 |
+
# Add the default rule to the tenant's rules subcollection
|
| 566 |
+
config_manager.add_rule(default_tenant.tenant_id, default_rule)
|
| 567 |
+
logger.info("✓ Successfully saved default tenant and rule to Firebase: %s", default_tenant.tenant_id)
|
| 568 |
+
default_tenant_saved = True
|
| 569 |
+
except Exception as exc:
|
| 570 |
+
logger.error("✗ Failed to save default tenant to Firebase: %s", exc, exc_info=True)
|
| 571 |
+
return {
|
| 572 |
+
"ok": False,
|
| 573 |
+
"error": f"Failed to save default tenant to Firebase: {exc}",
|
| 574 |
+
"tenants_processed": 0,
|
| 575 |
+
"tenants_results": [],
|
| 576 |
+
}
|
| 577 |
+
|
| 578 |
+
# Use the default tenant for orchestration
|
| 579 |
+
active_tenants = [default_tenant]
|
| 580 |
+
default_tenant_created = True
|
| 581 |
+
logger.info("Using default tenant created and saved from environment variables")
|
| 582 |
+
|
| 583 |
+
# Process each tenant and their rules
|
| 584 |
+
tenants_results = []
|
| 585 |
+
total_parsed = 0
|
| 586 |
+
total_modified = 0
|
| 587 |
+
total_sent = 0
|
| 588 |
+
total_moved = 0
|
| 589 |
+
total_corrupted = 0
|
| 590 |
+
total_errors = 0
|
| 591 |
+
|
| 592 |
+
for tenant in active_tenants:
|
| 593 |
+
tenant_id = tenant.tenant_id
|
| 594 |
+
logger.info("Processing tenant=%s name=%s", tenant_id, tenant.name)
|
| 595 |
+
|
| 596 |
+
# Get enabled rules for this tenant from the rules subcollection
|
| 597 |
+
enabled_rules = config_manager.get_enabled_rules(tenant_id)
|
| 598 |
+
logger.info("Tenant=%s has %s enabled rules", tenant_id, len(enabled_rules))
|
| 599 |
+
|
| 600 |
+
if not enabled_rules:
|
| 601 |
+
logger.warning("Tenant=%s has no enabled rules", tenant_id)
|
| 602 |
+
tenants_results.append({
|
| 603 |
+
"tenant_id": tenant_id,
|
| 604 |
+
"tenant_name": tenant.name,
|
| 605 |
+
"ok": True,
|
| 606 |
+
"warning": "No enabled rules",
|
| 607 |
+
"rules_processed": 0,
|
| 608 |
+
"rules_results": [],
|
| 609 |
+
})
|
| 610 |
+
continue
|
| 611 |
+
|
| 612 |
+
# Process each rule
|
| 613 |
+
rules_results = []
|
| 614 |
+
tenant_parsed = 0
|
| 615 |
+
tenant_modified = 0
|
| 616 |
+
tenant_sent = 0
|
| 617 |
+
tenant_moved = 0
|
| 618 |
+
tenant_corrupted = 0
|
| 619 |
+
tenant_errors = 0
|
| 620 |
+
|
| 621 |
+
for rule in enabled_rules:
|
| 622 |
+
rule_result = orchestrate_single_rule(
|
| 623 |
+
tenant_id=tenant_id,
|
| 624 |
+
tenant_config=tenant,
|
| 625 |
+
rule=rule,
|
| 626 |
+
model=model,
|
| 627 |
+
include_subfolders=include_subfolders,
|
| 628 |
+
send_to_sheet=send_to_sheet,
|
| 629 |
+
)
|
| 630 |
+
rules_results.append(rule_result)
|
| 631 |
+
|
| 632 |
+
if rule_result.get("ok"):
|
| 633 |
+
tenant_parsed += rule_result.get("parsed", 0)
|
| 634 |
+
tenant_modified += rule_result.get("modified", 0)
|
| 635 |
+
tenant_sent += rule_result.get("sent", 0)
|
| 636 |
+
tenant_moved += rule_result.get("moved", 0)
|
| 637 |
+
tenant_corrupted += rule_result.get("corrupted", 0)
|
| 638 |
+
tenant_errors += len(rule_result.get("errors", []))
|
| 639 |
+
|
| 640 |
+
total_parsed += tenant_parsed
|
| 641 |
+
total_modified += tenant_modified
|
| 642 |
+
total_sent += tenant_sent
|
| 643 |
+
total_moved += tenant_moved
|
| 644 |
+
total_corrupted += tenant_corrupted
|
| 645 |
+
total_errors += tenant_errors
|
| 646 |
+
|
| 647 |
+
tenants_results.append({
|
| 648 |
+
"tenant_id": tenant_id,
|
| 649 |
+
"tenant_name": tenant.name,
|
| 650 |
+
"ok": True,
|
| 651 |
+
"rules_processed": len(rules_results),
|
| 652 |
+
"parsed": tenant_parsed,
|
| 653 |
+
"modified": tenant_modified,
|
| 654 |
+
"sent": tenant_sent,
|
| 655 |
+
"moved": tenant_moved,
|
| 656 |
+
"corrupted": tenant_corrupted,
|
| 657 |
+
"errors": tenant_errors,
|
| 658 |
+
"rules_results": rules_results,
|
| 659 |
+
})
|
| 660 |
+
|
| 661 |
+
logger.info(
|
| 662 |
+
"Multi-tenant orchestration completed tenants=%s parsed=%s modified=%s sent=%s moved=%s corrupted=%s errors=%s",
|
| 663 |
+
len(tenants_results),
|
| 664 |
+
total_parsed,
|
| 665 |
+
total_modified,
|
| 666 |
+
total_sent,
|
| 667 |
+
total_moved,
|
| 668 |
+
total_corrupted,
|
| 669 |
+
total_errors,
|
| 670 |
+
)
|
| 671 |
+
|
| 672 |
+
return {
|
| 673 |
+
"ok": True,
|
| 674 |
+
"tenants_processed": len(tenants_results),
|
| 675 |
+
"default_tenant_created": default_tenant_created,
|
| 676 |
+
"default_tenant_saved": default_tenant_saved,
|
| 677 |
+
"total_parsed": total_parsed,
|
| 678 |
+
"total_modified": total_modified,
|
| 679 |
+
"total_sent": total_sent,
|
| 680 |
+
"total_moved": total_moved,
|
| 681 |
+
"total_corrupted": total_corrupted,
|
| 682 |
+
"total_errors": total_errors,
|
| 683 |
+
"tenants_results": tenants_results,
|
| 684 |
+
}
|
| 685 |
+
|
| 686 |
+
|
| 687 |
def parse_args() -> argparse.Namespace:
|
| 688 |
+
parser = argparse.ArgumentParser(
|
| 689 |
+
description="Orchestrate Drive documents: scan, parse, and persist status using multi-tenant Firebase configuration."
|
| 690 |
+
)
|
| 691 |
+
parser.add_argument(
|
| 692 |
+
"--model",
|
| 693 |
+
default=os.getenv("OPENAI_MODEL", "gpt-4o"),
|
| 694 |
+
help="OpenAI model name."
|
| 695 |
+
)
|
| 696 |
parser.add_argument(
|
| 697 |
"--no-subfolders",
|
| 698 |
action="store_true",
|
|
|
|
| 713 |
|
| 714 |
def main() -> int:
|
| 715 |
args = parse_args()
|
| 716 |
+
logger.info(
|
| 717 |
+
"Running document orchestration CLI in multi-tenant mode output=%s",
|
| 718 |
+
args.output or "stdout",
|
| 719 |
+
)
|
| 720 |
+
|
| 721 |
+
logger.info("Loading all active tenants from Firebase")
|
| 722 |
+
summary = orchestrate_all_active_tenants(
|
| 723 |
model=args.model,
|
| 724 |
include_subfolders=not args.no_subfolders,
|
| 725 |
send_to_sheet=args.send,
|
firebase_processed_files.py
CHANGED
|
@@ -18,11 +18,21 @@ class FirebaseConfigError(RuntimeError):
|
|
| 18 |
|
| 19 |
|
| 20 |
class FirebaseProcessedFilesTracker:
|
| 21 |
-
def __init__(
|
| 22 |
-
self
|
| 23 |
-
|
| 24 |
-
|
| 25 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 26 |
self._db = None
|
| 27 |
|
| 28 |
def _build_credentials(self):
|
|
@@ -86,21 +96,23 @@ class FirebaseProcessedFilesTracker:
|
|
| 86 |
self._get_db().collection(self.collection_name).document(file_hash).set(payload)
|
| 87 |
|
| 88 |
def get_document_record(self, document_id: str) -> dict[str, Any] | None:
|
| 89 |
-
doc = self._get_db().collection(self.
|
| 90 |
if not doc.exists:
|
| 91 |
return None
|
| 92 |
return doc.to_dict() or {}
|
| 93 |
|
| 94 |
def save_document_record(
|
| 95 |
self,
|
|
|
|
| 96 |
document_id: str,
|
| 97 |
source_file: str,
|
| 98 |
modification_date: str | None,
|
| 99 |
status: str,
|
| 100 |
-
parsed_data: dict[str, Any] | None,
|
| 101 |
) -> None:
|
| 102 |
now = datetime.now(timezone.utc).isoformat()
|
| 103 |
payload: dict[str, Any] = {
|
|
|
|
| 104 |
"document_id": document_id,
|
| 105 |
"source_file": source_file,
|
| 106 |
"modificationDate": modification_date,
|
|
@@ -111,11 +123,15 @@ class FirebaseProcessedFilesTracker:
|
|
| 111 |
if parsed_data is not None:
|
| 112 |
payload["parsed_data"] = parsed_data
|
| 113 |
|
| 114 |
-
existing = self.get_document_record(
|
| 115 |
if existing is None:
|
| 116 |
payload["created_at"] = now
|
| 117 |
|
| 118 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
| 119 |
|
| 120 |
def mark_document_sent(self, document_id: str) -> None:
|
| 121 |
now = datetime.now(timezone.utc).isoformat()
|
|
@@ -124,14 +140,14 @@ class FirebaseProcessedFilesTracker:
|
|
| 124 |
"sent_at": now,
|
| 125 |
"updated_at": now,
|
| 126 |
}
|
| 127 |
-
self._get_db().collection(self.
|
| 128 |
|
| 129 |
def list_documents_by_statuses(self, statuses: list[str]) -> list[dict[str, Any]]:
|
| 130 |
unique_statuses = [status for status in dict.fromkeys(statuses) if str(status).strip()]
|
| 131 |
if not unique_statuses:
|
| 132 |
return []
|
| 133 |
|
| 134 |
-
collection = self._get_db().collection(self.
|
| 135 |
documents: list[dict[str, Any]] = []
|
| 136 |
|
| 137 |
try:
|
|
@@ -153,12 +169,15 @@ class FirebaseProcessedFilesTracker:
|
|
| 153 |
return documents
|
| 154 |
|
| 155 |
|
| 156 |
-
def build_firebase_tracker(
|
|
|
|
|
|
|
|
|
|
| 157 |
enabled = (os.getenv("FIREBASE_TRACK_PROCESSED") or "true").strip().lower()
|
| 158 |
if enabled in {"0", "false", "no", "off"}:
|
| 159 |
-
|
| 160 |
|
| 161 |
-
return FirebaseProcessedFilesTracker()
|
| 162 |
|
| 163 |
|
| 164 |
def check_files_already_processed(
|
|
@@ -191,7 +210,14 @@ def check_drive_documents_to_process(
|
|
| 191 |
"""
|
| 192 |
Returns (documents_to_process, skipped_documents, warning).
|
| 193 |
|
| 194 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 195 |
- Existing document with modified date changed -> status "Modified"
|
| 196 |
- Existing document with same modified date -> skipped
|
| 197 |
"""
|
|
@@ -206,21 +232,50 @@ def check_drive_documents_to_process(
|
|
| 206 |
for document in documents:
|
| 207 |
document_id = str(document.get("document_id") or "").strip()
|
| 208 |
modified_time = str(document.get("modificationDate") or "").strip() or None
|
|
|
|
| 209 |
|
| 210 |
-
if not document_id:
|
| 211 |
-
skipped.append({**document, "skip_reason": "
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 212 |
continue
|
| 213 |
|
| 214 |
-
|
|
|
|
| 215 |
if existing is None:
|
| 216 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 217 |
continue
|
| 218 |
|
|
|
|
| 219 |
existing_modified = str(existing.get("modificationDate") or "").strip() or None
|
| 220 |
if existing_modified != modified_time:
|
| 221 |
-
to_process.append({
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 222 |
else:
|
| 223 |
-
skipped.append({
|
|
|
|
|
|
|
|
|
|
|
|
|
| 224 |
except Exception as exc:
|
| 225 |
docs = [dict(doc, target_status="Parsed") for doc in documents]
|
| 226 |
return docs, [], f"Firebase tracking unavailable. Processing all Drive files. Details: {exc}"
|
|
|
|
| 18 |
|
| 19 |
|
| 20 |
class FirebaseProcessedFilesTracker:
|
| 21 |
+
def __init__(
|
| 22 |
+
self,
|
| 23 |
+
tenant_id: str,
|
| 24 |
+
rule_id: str,
|
| 25 |
+
):
|
| 26 |
+
self.tenant_id = (tenant_id or "").strip()
|
| 27 |
+
self.rule_id = (rule_id or "").strip()
|
| 28 |
+
|
| 29 |
+
if not self.tenant_id:
|
| 30 |
+
raise FirebaseConfigError("tenant_id is required")
|
| 31 |
+
if not self.rule_id:
|
| 32 |
+
raise FirebaseConfigError("rule_id is required")
|
| 33 |
+
|
| 34 |
+
# Multi-tenant rule-based tracking: tenants/{tenant_id}/rules_executions/{rule_id}/processed_documents/{file_hash}
|
| 35 |
+
self.collection_name = f"tenants/{self.tenant_id}/rules/{self.rule_id}/processed_documents"
|
| 36 |
self._db = None
|
| 37 |
|
| 38 |
def _build_credentials(self):
|
|
|
|
| 96 |
self._get_db().collection(self.collection_name).document(file_hash).set(payload)
|
| 97 |
|
| 98 |
def get_document_record(self, document_id: str) -> dict[str, Any] | None:
|
| 99 |
+
doc = self._get_db().collection(self.collection_name).document(document_id).get()
|
| 100 |
if not doc.exists:
|
| 101 |
return None
|
| 102 |
return doc.to_dict() or {}
|
| 103 |
|
| 104 |
def save_document_record(
|
| 105 |
self,
|
| 106 |
+
file_hash: str,
|
| 107 |
document_id: str,
|
| 108 |
source_file: str,
|
| 109 |
modification_date: str | None,
|
| 110 |
status: str,
|
| 111 |
+
parsed_data: dict[str, Any] | None = None,
|
| 112 |
) -> None:
|
| 113 |
now = datetime.now(timezone.utc).isoformat()
|
| 114 |
payload: dict[str, Any] = {
|
| 115 |
+
"file_hash": file_hash,
|
| 116 |
"document_id": document_id,
|
| 117 |
"source_file": source_file,
|
| 118 |
"modificationDate": modification_date,
|
|
|
|
| 123 |
if parsed_data is not None:
|
| 124 |
payload["parsed_data"] = parsed_data
|
| 125 |
|
| 126 |
+
existing = self.get_document_record(file_hash)
|
| 127 |
if existing is None:
|
| 128 |
payload["created_at"] = now
|
| 129 |
|
| 130 |
+
# Use file_hash as the document ID in Firebase
|
| 131 |
+
self._get_db().collection(self.collection_name).document(file_hash).set(payload, merge=True)
|
| 132 |
+
|
| 133 |
+
# Also mark the file as processed in the documents collection
|
| 134 |
+
self.mark_processed(file_hash, source_file)
|
| 135 |
|
| 136 |
def mark_document_sent(self, document_id: str) -> None:
|
| 137 |
now = datetime.now(timezone.utc).isoformat()
|
|
|
|
| 140 |
"sent_at": now,
|
| 141 |
"updated_at": now,
|
| 142 |
}
|
| 143 |
+
self._get_db().collection(self.collection_name).document(document_id).set(payload, merge=True)
|
| 144 |
|
| 145 |
def list_documents_by_statuses(self, statuses: list[str]) -> list[dict[str, Any]]:
|
| 146 |
unique_statuses = [status for status in dict.fromkeys(statuses) if str(status).strip()]
|
| 147 |
if not unique_statuses:
|
| 148 |
return []
|
| 149 |
|
| 150 |
+
collection = self._get_db().collection(self.collection_name)
|
| 151 |
documents: list[dict[str, Any]] = []
|
| 152 |
|
| 153 |
try:
|
|
|
|
| 169 |
return documents
|
| 170 |
|
| 171 |
|
| 172 |
+
def build_firebase_tracker(
|
| 173 |
+
tenant_id: str,
|
| 174 |
+
rule_id: str,
|
| 175 |
+
) -> FirebaseProcessedFilesTracker:
|
| 176 |
enabled = (os.getenv("FIREBASE_TRACK_PROCESSED") or "true").strip().lower()
|
| 177 |
if enabled in {"0", "false", "no", "off"}:
|
| 178 |
+
raise FirebaseConfigError("Firebase tracking is disabled (FIREBASE_TRACK_PROCESSED=false)")
|
| 179 |
|
| 180 |
+
return FirebaseProcessedFilesTracker(tenant_id=tenant_id, rule_id=rule_id)
|
| 181 |
|
| 182 |
|
| 183 |
def check_files_already_processed(
|
|
|
|
| 210 |
"""
|
| 211 |
Returns (documents_to_process, skipped_documents, warning).
|
| 212 |
|
| 213 |
+
Process:
|
| 214 |
+
1. Compute file hash from file content
|
| 215 |
+
2. Use hash as unique document identifier
|
| 216 |
+
3. Check if document (by hash) already exists
|
| 217 |
+
4. Compare modification dates for changes
|
| 218 |
+
|
| 219 |
+
- File already processed (by hash) -> skipped
|
| 220 |
+
- New document (no hash record) -> status "Parsed"
|
| 221 |
- Existing document with modified date changed -> status "Modified"
|
| 222 |
- Existing document with same modified date -> skipped
|
| 223 |
"""
|
|
|
|
| 232 |
for document in documents:
|
| 233 |
document_id = str(document.get("document_id") or "").strip()
|
| 234 |
modified_time = str(document.get("modificationDate") or "").strip() or None
|
| 235 |
+
local_path = str(document.get("local_path") or "").strip()
|
| 236 |
|
| 237 |
+
if not document_id or not local_path:
|
| 238 |
+
skipped.append({**document, "skip_reason": "missing_document_id_or_path"})
|
| 239 |
+
continue
|
| 240 |
+
|
| 241 |
+
# Compute file hash from file content - this becomes the unique identifier
|
| 242 |
+
try:
|
| 243 |
+
path = Path(local_path)
|
| 244 |
+
if not path.exists():
|
| 245 |
+
skipped.append({**document, "skip_reason": "file_not_found"})
|
| 246 |
+
continue
|
| 247 |
+
file_hash = tracker.compute_file_hash(path)
|
| 248 |
+
except Exception as exc:
|
| 249 |
+
skipped.append({**document, "skip_reason": f"hash_computation_failed: {exc}"})
|
| 250 |
continue
|
| 251 |
|
| 252 |
+
# Use hash as the document identifier
|
| 253 |
+
existing = tracker.get_document_record(file_hash)
|
| 254 |
if existing is None:
|
| 255 |
+
# New file - process it
|
| 256 |
+
to_process.append({
|
| 257 |
+
**document,
|
| 258 |
+
"file_hash": file_hash,
|
| 259 |
+
"target_status": "Parsed",
|
| 260 |
+
"hash_document_id": file_hash,
|
| 261 |
+
})
|
| 262 |
continue
|
| 263 |
|
| 264 |
+
# Existing document - check if modified
|
| 265 |
existing_modified = str(existing.get("modificationDate") or "").strip() or None
|
| 266 |
if existing_modified != modified_time:
|
| 267 |
+
to_process.append({
|
| 268 |
+
**document,
|
| 269 |
+
"file_hash": file_hash,
|
| 270 |
+
"target_status": "Modified",
|
| 271 |
+
"hash_document_id": file_hash,
|
| 272 |
+
})
|
| 273 |
else:
|
| 274 |
+
skipped.append({
|
| 275 |
+
**document,
|
| 276 |
+
"skip_reason": "already_processed_same_content",
|
| 277 |
+
"file_hash": file_hash,
|
| 278 |
+
})
|
| 279 |
except Exception as exc:
|
| 280 |
docs = [dict(doc, target_status="Parsed") for doc in documents]
|
| 281 |
return docs, [], f"Firebase tracking unavailable. Processing all Drive files. Details: {exc}"
|
firebase_tenant_config.py
ADDED
|
@@ -0,0 +1,209 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
from __future__ import annotations
|
| 2 |
+
|
| 3 |
+
import json
|
| 4 |
+
import os
|
| 5 |
+
from dataclasses import dataclass
|
| 6 |
+
from datetime import datetime, timezone
|
| 7 |
+
from typing import Any
|
| 8 |
+
|
| 9 |
+
from dotenv import load_dotenv
|
| 10 |
+
|
| 11 |
+
|
| 12 |
+
load_dotenv()
|
| 13 |
+
|
| 14 |
+
|
| 15 |
+
class FirebaseConfigError(RuntimeError):
|
| 16 |
+
"""Raised when Firebase configuration is missing or invalid."""
|
| 17 |
+
|
| 18 |
+
|
| 19 |
+
@dataclass
|
| 20 |
+
class CredentialsObject:
|
| 21 |
+
"""OAuth & API Keys for a tenant."""
|
| 22 |
+
openai_api_key: str
|
| 23 |
+
google_service_account_json: str
|
| 24 |
+
|
| 25 |
+
|
| 26 |
+
@dataclass
|
| 27 |
+
class RuleObject:
|
| 28 |
+
"""Processing rule for a tenant."""
|
| 29 |
+
rule_id: str
|
| 30 |
+
rule_name: str
|
| 31 |
+
source_folder_id: str
|
| 32 |
+
target_folder_id: str
|
| 33 |
+
target_sheet_id: str
|
| 34 |
+
sheet_tab_name: str
|
| 35 |
+
parsing_prompt: str | None = None
|
| 36 |
+
is_enabled: bool = True
|
| 37 |
+
|
| 38 |
+
def to_dict(self) -> dict[str, Any]:
|
| 39 |
+
return {
|
| 40 |
+
"rule_id": self.rule_id,
|
| 41 |
+
"rule_name": self.rule_name,
|
| 42 |
+
"source_folder_id": self.source_folder_id,
|
| 43 |
+
"target_folder_id": self.target_folder_id,
|
| 44 |
+
"target_sheet_id": self.target_sheet_id,
|
| 45 |
+
"sheet_tab_name": self.sheet_tab_name,
|
| 46 |
+
"parsing_prompt": self.parsing_prompt,
|
| 47 |
+
"is_enabled": self.is_enabled,
|
| 48 |
+
}
|
| 49 |
+
|
| 50 |
+
@staticmethod
|
| 51 |
+
def from_dict(data: dict[str, Any]) -> RuleObject:
|
| 52 |
+
return RuleObject(
|
| 53 |
+
rule_id=data.get("rule_id", ""),
|
| 54 |
+
rule_name=data.get("rule_name", ""),
|
| 55 |
+
source_folder_id=data.get("source_folder_id", ""),
|
| 56 |
+
target_folder_id=data.get("target_folder_id", ""),
|
| 57 |
+
target_sheet_id=data.get("target_sheet_id", ""),
|
| 58 |
+
sheet_tab_name=data.get("sheet_tab_name", ""),
|
| 59 |
+
parsing_prompt=data.get("parsing_prompt"),
|
| 60 |
+
is_enabled=data.get("is_enabled", True),
|
| 61 |
+
)
|
| 62 |
+
|
| 63 |
+
|
| 64 |
+
@dataclass
|
| 65 |
+
class TenantConfig:
|
| 66 |
+
"""Complete tenant configuration (rules stored as subcollection)."""
|
| 67 |
+
tenant_id: str
|
| 68 |
+
name: str
|
| 69 |
+
active: bool
|
| 70 |
+
credentials: CredentialsObject
|
| 71 |
+
created_at: str | None = None
|
| 72 |
+
|
| 73 |
+
def to_dict(self) -> dict[str, Any]:
|
| 74 |
+
return {
|
| 75 |
+
"tenant_id": self.tenant_id,
|
| 76 |
+
"name": self.name,
|
| 77 |
+
"active": self.active,
|
| 78 |
+
"credentials": {
|
| 79 |
+
"openai_api_key": self.credentials.openai_api_key,
|
| 80 |
+
"google_service_account_json": self.credentials.google_service_account_json,
|
| 81 |
+
},
|
| 82 |
+
"created_at": self.created_at,
|
| 83 |
+
}
|
| 84 |
+
|
| 85 |
+
@staticmethod
|
| 86 |
+
def from_dict(data: dict[str, Any], tenant_id: str) -> TenantConfig:
|
| 87 |
+
creds_data = data.get("credentials", {})
|
| 88 |
+
credentials = CredentialsObject(
|
| 89 |
+
openai_api_key=creds_data.get("openai_api_key", ""),
|
| 90 |
+
google_service_account_json=creds_data.get("google_service_account_json", ""),
|
| 91 |
+
)
|
| 92 |
+
return TenantConfig(
|
| 93 |
+
tenant_id=tenant_id,
|
| 94 |
+
name=data.get("name", ""),
|
| 95 |
+
active=data.get("active", True),
|
| 96 |
+
credentials=credentials,
|
| 97 |
+
created_at=data.get("created_at"),
|
| 98 |
+
)
|
| 99 |
+
|
| 100 |
+
|
| 101 |
+
class FirebaseTenantConfigManager:
|
| 102 |
+
"""Manages tenant configurations in Firebase Firestore.
|
| 103 |
+
|
| 104 |
+
This is the TENANT CONFIG DB - separate from document processing.
|
| 105 |
+
Other environments import this and use get_tenant() to fetch configs.
|
| 106 |
+
"""
|
| 107 |
+
|
| 108 |
+
def __init__(self, collection_name: str | None = None):
|
| 109 |
+
self.collection_name = (collection_name or os.getenv("FIREBASE_TENANTS_COLLECTION") or "tenants").strip()
|
| 110 |
+
self._db = None
|
| 111 |
+
|
| 112 |
+
def _build_credentials(self):
|
| 113 |
+
from firebase_admin import credentials
|
| 114 |
+
|
| 115 |
+
service_account_json = (os.getenv("FIREBASE_SERVICE_ACCOUNT_JSON") or "").strip()
|
| 116 |
+
service_account_file = (os.getenv("FIREBASE_SERVICE_ACCOUNT_FILE") or "").strip()
|
| 117 |
+
|
| 118 |
+
if not service_account_json:
|
| 119 |
+
service_account_json = (os.getenv("GOOGLE_SERVICE_ACCOUNT_JSON") or "").strip()
|
| 120 |
+
if not service_account_file:
|
| 121 |
+
service_account_file = (os.getenv("GOOGLE_SERVICE_ACCOUNT_FILE") or "").strip()
|
| 122 |
+
|
| 123 |
+
if service_account_json:
|
| 124 |
+
try:
|
| 125 |
+
info = json.loads(service_account_json)
|
| 126 |
+
except json.JSONDecodeError as exc:
|
| 127 |
+
raise FirebaseConfigError("FIREBASE_SERVICE_ACCOUNT_JSON is not valid JSON.") from exc
|
| 128 |
+
return credentials.Certificate(info)
|
| 129 |
+
|
| 130 |
+
if service_account_file:
|
| 131 |
+
return credentials.Certificate(service_account_file)
|
| 132 |
+
|
| 133 |
+
raise FirebaseConfigError(
|
| 134 |
+
"Missing Firebase credentials. Set FIREBASE_SERVICE_ACCOUNT_JSON or FIREBASE_SERVICE_ACCOUNT_FILE."
|
| 135 |
+
)
|
| 136 |
+
|
| 137 |
+
def _get_db(self):
|
| 138 |
+
if self._db is not None:
|
| 139 |
+
return self._db
|
| 140 |
+
|
| 141 |
+
import firebase_admin
|
| 142 |
+
from firebase_admin import firestore
|
| 143 |
+
|
| 144 |
+
if not firebase_admin._apps:
|
| 145 |
+
cred = self._build_credentials()
|
| 146 |
+
firebase_admin.initialize_app(cred)
|
| 147 |
+
|
| 148 |
+
self._db = firestore.client()
|
| 149 |
+
return self._db
|
| 150 |
+
|
| 151 |
+
def get_tenant(self, tenant_id: str) -> TenantConfig | None:
|
| 152 |
+
"""Retrieve a tenant configuration by ID."""
|
| 153 |
+
doc = self._get_db().collection(self.collection_name).document(tenant_id).get()
|
| 154 |
+
if not doc.exists:
|
| 155 |
+
return None
|
| 156 |
+
data = doc.to_dict() or {}
|
| 157 |
+
return TenantConfig.from_dict(data, tenant_id)
|
| 158 |
+
|
| 159 |
+
def save_tenant(self, tenant: TenantConfig) -> None:
|
| 160 |
+
"""Save a tenant configuration."""
|
| 161 |
+
payload = tenant.to_dict()
|
| 162 |
+
if not payload.get("created_at"):
|
| 163 |
+
payload["created_at"] = datetime.now(timezone.utc).isoformat()
|
| 164 |
+
payload["updated_at"] = datetime.now(timezone.utc).isoformat()
|
| 165 |
+
self._get_db().collection(self.collection_name).document(tenant.tenant_id).set(payload)
|
| 166 |
+
|
| 167 |
+
def list_active_tenants(self) -> list[TenantConfig]:
|
| 168 |
+
"""Retrieve all active tenants."""
|
| 169 |
+
docs = (
|
| 170 |
+
self._get_db()
|
| 171 |
+
.collection(self.collection_name)
|
| 172 |
+
.where("active", "==", True)
|
| 173 |
+
.stream()
|
| 174 |
+
)
|
| 175 |
+
tenants = []
|
| 176 |
+
for doc in docs:
|
| 177 |
+
try:
|
| 178 |
+
tenant = TenantConfig.from_dict(doc.to_dict() or {}, doc.id)
|
| 179 |
+
tenants.append(tenant)
|
| 180 |
+
except Exception:
|
| 181 |
+
continue
|
| 182 |
+
return tenants
|
| 183 |
+
|
| 184 |
+
def get_all_rules(self, tenant_id: str) -> list[RuleObject]:
|
| 185 |
+
"""Get all processing rules for a tenant from the rules subcollection."""
|
| 186 |
+
tenant_doc = self._get_db().collection(self.collection_name).document(tenant_id)
|
| 187 |
+
rules_subcoll = tenant_doc.collection("rules")
|
| 188 |
+
rules = []
|
| 189 |
+
for doc in rules_subcoll.stream():
|
| 190 |
+
rule_data = doc.to_dict() or {}
|
| 191 |
+
rules.append(RuleObject.from_dict(rule_data))
|
| 192 |
+
return rules
|
| 193 |
+
|
| 194 |
+
def get_enabled_rules(self, tenant_id: str) -> list[RuleObject]:
|
| 195 |
+
"""Get all enabled processing rules for a tenant."""
|
| 196 |
+
all_rules = self.get_all_rules(tenant_id)
|
| 197 |
+
return [rule for rule in all_rules if rule.is_enabled]
|
| 198 |
+
|
| 199 |
+
def add_rule(self, tenant_id: str, rule: RuleObject) -> None:
|
| 200 |
+
"""Add a rule to a tenant's rules subcollection."""
|
| 201 |
+
tenant_doc = self._get_db().collection(self.collection_name).document(tenant_id)
|
| 202 |
+
rules_subcoll = tenant_doc.collection("rules")
|
| 203 |
+
rules_subcoll.document(rule.rule_id).set(rule.to_dict())
|
| 204 |
+
|
| 205 |
+
def remove_rule(self, tenant_id: str, rule_id: str) -> None:
|
| 206 |
+
"""Remove a rule from a tenant's rules subcollection."""
|
| 207 |
+
tenant_doc = self._get_db().collection(self.collection_name).document(tenant_id)
|
| 208 |
+
rules_subcoll = tenant_doc.collection("rules")
|
| 209 |
+
rules_subcoll.document(rule_id).delete()
|
google_drive_service.py
CHANGED
|
@@ -82,13 +82,20 @@ def load_drive_folder_ids_from_env(env_var: str = "GOOGLE_DRIVE_FOLDER_IDS") ->
|
|
| 82 |
return folder_ids
|
| 83 |
|
| 84 |
|
| 85 |
-
def build_drive_credentials(scope: str = DRIVE_READONLY_SCOPE) -> Credentials:
|
| 86 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 87 |
service_account_file = (os.getenv("GOOGLE_SERVICE_ACCOUNT_FILE") or "").strip()
|
| 88 |
|
| 89 |
-
if
|
| 90 |
try:
|
| 91 |
-
info: dict[str, Any] = json.loads(
|
| 92 |
except json.JSONDecodeError as exc:
|
| 93 |
raise GoogleDriveConfigError("GOOGLE_SERVICE_ACCOUNT_JSON is not valid JSON.") from exc
|
| 94 |
logger.info("Using GOOGLE_SERVICE_ACCOUNT_JSON for Drive credentials")
|
|
@@ -103,7 +110,13 @@ def build_drive_credentials(scope: str = DRIVE_READONLY_SCOPE) -> Credentials:
|
|
| 103 |
)
|
| 104 |
|
| 105 |
|
| 106 |
-
def build_drive_service(scope: str = DRIVE_READONLY_SCOPE):
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 107 |
try:
|
| 108 |
from googleapiclient.discovery import build
|
| 109 |
except ImportError as exc:
|
|
@@ -111,7 +124,7 @@ def build_drive_service(scope: str = DRIVE_READONLY_SCOPE):
|
|
| 111 |
"google-api-python-client is required for Drive scanning. Add it to requirements and install dependencies."
|
| 112 |
) from exc
|
| 113 |
|
| 114 |
-
credentials = build_drive_credentials(scope=scope)
|
| 115 |
logger.info("Building Drive service with scope=%s", scope)
|
| 116 |
return build("drive", "v3", credentials=credentials, cache_discovery=False)
|
| 117 |
|
|
|
|
| 82 |
return folder_ids
|
| 83 |
|
| 84 |
|
| 85 |
+
def build_drive_credentials(scope: str = DRIVE_READONLY_SCOPE, service_account_json: str | None = None) -> Credentials:
|
| 86 |
+
"""Build Google Drive credentials.
|
| 87 |
+
|
| 88 |
+
Args:
|
| 89 |
+
scope: OAuth scope for the credentials
|
| 90 |
+
service_account_json: Optional JSON string with service account credentials.
|
| 91 |
+
If not provided, uses GOOGLE_SERVICE_ACCOUNT_JSON env var.
|
| 92 |
+
"""
|
| 93 |
+
json_to_use = (service_account_json or "").strip() or (os.getenv("GOOGLE_SERVICE_ACCOUNT_JSON") or "").strip()
|
| 94 |
service_account_file = (os.getenv("GOOGLE_SERVICE_ACCOUNT_FILE") or "").strip()
|
| 95 |
|
| 96 |
+
if json_to_use:
|
| 97 |
try:
|
| 98 |
+
info: dict[str, Any] = json.loads(json_to_use)
|
| 99 |
except json.JSONDecodeError as exc:
|
| 100 |
raise GoogleDriveConfigError("GOOGLE_SERVICE_ACCOUNT_JSON is not valid JSON.") from exc
|
| 101 |
logger.info("Using GOOGLE_SERVICE_ACCOUNT_JSON for Drive credentials")
|
|
|
|
| 110 |
)
|
| 111 |
|
| 112 |
|
| 113 |
+
def build_drive_service(scope: str = DRIVE_READONLY_SCOPE, service_account_json: str | None = None):
|
| 114 |
+
"""Build Google Drive service.
|
| 115 |
+
|
| 116 |
+
Args:
|
| 117 |
+
scope: OAuth scope for the credentials
|
| 118 |
+
service_account_json: Optional JSON string with service account credentials for tenant.
|
| 119 |
+
"""
|
| 120 |
try:
|
| 121 |
from googleapiclient.discovery import build
|
| 122 |
except ImportError as exc:
|
|
|
|
| 124 |
"google-api-python-client is required for Drive scanning. Add it to requirements and install dependencies."
|
| 125 |
) from exc
|
| 126 |
|
| 127 |
+
credentials = build_drive_credentials(scope=scope, service_account_json=service_account_json)
|
| 128 |
logger.info("Building Drive service with scope=%s", scope)
|
| 129 |
return build("drive", "v3", credentials=credentials, cache_discovery=False)
|
| 130 |
|