File size: 14,821 Bytes
028d116
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
# Multi-Tenant Document Parser - Configuration & Usage Guide

## Overview

The document parser now supports two operational modes:

1. **Multi-Tenant Mode (NEW)** - Load all active tenants from Firebase and process their enabled rules
2. **Legacy Mode** - Single tenant configuration via environment variables (backward compatible)

## Architecture

### Data Model

All tenant and rule configurations are stored in Firestore under the `tenants` collection:

```
TENANTS_COLLECTION (Firestore)
β”œβ”€β”€ {tenant_id}
β”‚   β”œβ”€β”€ tenant_id: unique tenant ID
β”‚   β”œβ”€β”€ name: "Company/User Name"
β”‚   β”œβ”€β”€ active: boolean (Account Status)
β”‚   β”œβ”€β”€ created_at: timestamp
β”‚   β”œβ”€β”€ updated_at: timestamp
β”‚   β”œβ”€β”€ credentials (object, 1:1 relationship)
β”‚   β”‚   β”œβ”€β”€ openai_api_key: "OpenAI Key"
β”‚   β”‚   └── google_service_account_json: "Service account JSON"
β”‚   └── rules (subcollection, 1:N relationship)
β”‚       └── {rule_id}
β”‚           β”œβ”€β”€ rule_id: "unique rule ID"
β”‚           β”œβ”€β”€ rule_name: "Invoices to Finance"
β”‚           β”œβ”€β”€ source_folder_id: "Google Drive Source Folder ID"
β”‚           β”œβ”€β”€ target_folder_id: "Google Drive Destination Folder ID"
β”‚           β”œβ”€β”€ target_sheet_id: "Google Sheets Spreadsheet ID"
β”‚           β”œβ”€β”€ sheet_tab_name: "Tab inside Google Sheet"
β”‚           β”œβ”€β”€ parsing_prompt: "Custom LLM Instructions"
β”‚           └── is_enabled: boolean (Rule Toggle)
```

### Processing Flow

```
Multi-Tenant Mode:
1. orchestrate_all_active_tenants()
   └─> For each active tenant:
       └─> For each enabled rule:
           └─> orchestrate_single_rule()
               β”œβ”€> Scan rule.source_folder_id
               β”œβ”€> Parse documents with rule.parsing_prompt
               β”œβ”€> Move to rule.target_folder_id
               └─> Send to rule.target_sheet_id
```

## Usage

### Quick Start (Using .env)

If you have existing `.env` configuration, multi-tenant mode will automatically create a default tenant:

```bash
# Just run it! No Firebase setup needed initially
python document_orchester.py --send

# The system will:
# 1. Check for active tenants in Firebase
# 2. If none found, create "default_tenant" from your .env vars
# 3. Process documents using the default configuration
```

**This is the easiest path if you're migrating from single-tenant mode.**

### Mode 1: Multi-Tenant Mode (Default)

```bash
# Process all active tenants and their enabled rules
python document_orchester.py --model gpt-4o --send

# Optional flags:
# --no-subfolders    Only scan direct files (don't recurse)
# --send             Send results to Google Sheets
# --output FILE      Save JSON summary to file
```

### Mode 2: Legacy Single-Tenant Mode

```bash
# Use env var configuration (backward compatible)
python document_orchester.py --legacy --model gpt-4o --send
```

### Gradio Interface

Start the web app:
```bash
python app.py
```

Two orchestrator endpoints are now available:
- **Single Tenant (Legacy)** - Uses env var configuration
- **Multi Tenant** - Loads all active tenants from Firebase

## Configuration

### Automatic Default Tenant (No Firebase Setup Required)

When you run multi-tenant mode without any tenants in Firebase:

1. The system checks for active tenants in Firestore
2. If none found, it creates a **default tenant** from your `.env` file
3. Maps env vars like this:

| .env Variable | Default Tenant |
|---|---|
| `OPENAI_API_KEY` | `credentials.openai_api_key` |
| `GOOGLE_SERVICE_ACCOUNT_JSON` | `credentials.google_service_account_json` |
| `GOOGLE_DRIVE_FOLDER_IDS` (first) | `rule.source_folder_id` |
| `GOOGLE_DRIVE_INVOICES_ROOT_FOLDER_ID` | `rule.target_folder_id` |
| `GOOGLE_SHEETS_SPREADSHEET_ID` | `rule.target_sheet_id` |

**Example: Your existing `.env`**
```env
OPENAI_API_KEY=sk-abc123...
GOOGLE_SERVICE_ACCOUNT_JSON={"type":"service_account","project_id":"my-project",...}
GOOGLE_DRIVE_FOLDER_IDS=1ABC_folder_id_here
GOOGLE_DRIVE_INVOICES_ROOT_FOLDER_ID=1XYZ_destination_folder_id
GOOGLE_SHEETS_SPREADSHEET_ID=18qKL...
```

**Creates this default tenant (in memory, not saved to Firebase):**
```json
{
  "tenant_id": "default_tenant",
  "name": "Default Tenant (from .env)",
  "active": true,
  "credentials": {
    "openai_api_key": "sk-abc123...",
    "google_service_account_json": "{...}"
  },
  "rules": [
    {
      "rule_id": "default_rule",
      "rule_name": "Default Processing Rule",
      "source_folder_id": "1ABC_folder_id_here",
      "target_folder_id": "1XYZ_destination_folder_id",
      "target_sheet_id": "18qKL...",
      "sheet_tab_name": "Parsed",
      "is_enabled": true
    }
  ]
}
```

The default tenant provides automatic setup:
- βœ… Created automatically if no Firebase tenants exist
- βœ… Saved to Firebase **BEFORE** orchestration starts
- βœ… Future runs load from Firebase instead of recreating
- βœ… Try multi-tenant mode without any setup
- βœ… Migrate gradually from single-tenant to multi-tenant
- βœ… Keep existing workflows unchanged

**The flow:**
1. First run: No Firebase tenants β†’ Create default_tenant from .env β†’ **Save to Firebase** β†’ Process documents
2. Second run: Firebase has default_tenant β†’ Load it β†’ Process documents
3. Add more tenants anytime in Firebase β†’ They're all processed together

### Multi-Tenant Setup (Firebase)

1. Create a Firestore collection called `tenants` (or set via `FIREBASE_TENANTS_COLLECTION` env var)

2. Add tenant documents with this structure:

```json
{
  "name": "Acme Corp",
  "active": true,
  "created_at": "2024-01-01T00:00:00Z",
  "updated_at": "2024-01-01T00:00:00Z",
  "credentials": {
    "openai_api_key": "sk-...",
    "google_service_account_json": "{...full service account json...}"
  }
}
```

3. Create a **rules** subcollection inside the tenant document, and add rules with this structure:

```json
{
  "rule_id": "invoices_2024",
  "rule_name": "Invoices to Finance",
  "source_folder_id": "1ABC...",
  "target_folder_id": "1XYZ...",
  "target_sheet_id": "18qKL...",
  "sheet_tab_name": "Invoices",
  "parsing_prompt": "Extract invoice details following these rules...",
  "is_enabled": true
}
```

### Environment Variables

**Required for both modes:**
```
FIREBASE_SERVICE_ACCOUNT_JSON    # Firebase credentials (JSON string or use GOOGLE_SERVICE_ACCOUNT_JSON)
OPENAI_API_KEY                   # (only for legacy mode, multi-tenant uses tenant credentials)
```

**Optional:**
```
FIREBASE_TENANTS_COLLECTION      # Default: "tenants"
FIREBASE_DOCUMENTS_COLLECTION    # Default: "processed_documents"
FIREBASE_TRACKER_COLLECTION      # Default: "processed_files"
FIREBASE_TRACK_PROCESSED         # Default: "true"
OPENAI_MODEL                     # Default: "gpt-4o"
GOOGLE_DRIVE_INVOICES_BASE_PATH  # Default: "Facturas"
ORCHESTRATOR_API_KEY             # For API authentication
```

## Firebase Storage Structure

### Document Tracking (Multi-Tenant)

Each tenant's processed files are tracked in nested collections:

```
tenants/{tenant_id}/processed_files/{file_hash}
└─ file_hash: "sha256 hash"
└─ source_file: "filename"
└─ processed_at: "timestamp"

tenants/{tenant_id}/processed_documents/{document_id}
└─ document_id: "drive file id"
└─ parsed_data: {...}
└─ status: "Parsed|Modified|Sent"
```

### Creating Tenants Programmatically

```python
from firebase_tenant_config import (
    FirebaseTenantConfigManager,
    TenantConfig,
    CredentialsObject,
    RuleObject,
)

manager = FirebaseTenantConfigManager()

# Create credentials
credentials = CredentialsObject(
    openai_api_key="sk-...",
    google_service_account_json="..." # full JSON as string
)

# Create a rule
rule = RuleObject(
    rule_id="invoices_2024",
    rule_name="Invoices to Finance",
    source_folder_id="1ABC...",
    target_folder_id="1XYZ...",
    target_sheet_id="18qKL...",
    sheet_tab_name="Invoices",
    parsing_prompt="Extract...",
    is_enabled=True,
)

# Create tenant
tenant = TenantConfig(
    tenant_id="acme-corp",
    name="Acme Corp",
    active=True,
    credentials=credentials,
    rules=[rule],
)

# Save to Firebase
manager.save_tenant(tenant)

# List all active tenants
active_tenants = manager.list_active_tenants()

# Get enabled rules for a tenant
enabled_rules = manager.get_enabled_rules("acme-corp")

# Add a rule
manager.add_rule("acme-corp", new_rule)

# Remove a rule
manager.remove_rule("acme-corp", "invoices_2024")
```

## Output Format

### Multi-Tenant Orchestration Result

```json
{
  "ok": true,
  "tenants_processed": 2,
  "total_parsed": 15,
  "total_modified": 3,
  "total_sent": 18,
  "total_moved": 18,
  "total_corrupted": 2,
  "total_errors": 1,
  "tenants_results": [
    {
      "tenant_id": "tenant_123",
      "tenant_name": "Acme Corp",
      "ok": true,
      "rules_processed": 2,
      "parsed": 10,
      "modified": 2,
      "sent": 12,
      "moved": 12,
      "corrupted": 1,
      "errors": 0,
      "rules_results": [
        {
          "ok": true,
          "tenant_id": "tenant_123",
          "rule_id": "invoices_2024",
          "rule_name": "Invoices to Finance",
          "scanned": 20,
          "to_process": 15,
          "skipped": 5,
          "parsed": 10,
          "modified": 2,
          "sent": 12,
          "moved": 12,
          "corrupted": 1,
          "errors": 0,
          "processed_items": [...],
          "skipped_items": [...]
        }
      ]
    }
  ]
}
```

## Migration from Legacy Mode

**No migration needed!** The system maintains full backward compatibility:

1. Existing env var configuration continues to work with `--legacy` flag
2. New multi-tenant mode coexists without affecting legacy deployments
3. To enable multi-tenant: Set up tenants in Firebase and run without `--legacy`

## Workflow Examples

### Simple: Use Existing .env (No Firebase Setup)

The **easiest** way to get started with multi-tenant mode:

```bash
# 1. Ensure your .env has these vars (you probably already do):
#    OPENAI_API_KEY
#    GOOGLE_SERVICE_ACCOUNT_JSON
#    GOOGLE_DRIVE_FOLDER_IDS
#    GOOGLE_DRIVE_INVOICES_ROOT_FOLDER_ID
#    GOOGLE_SHEETS_SPREADSHEET_ID

# 2. Just run multi-tenant orchestration:
python document_orchester.py --send

# That's it! The system will:
# - Check for tenants in Firebase
# - Find none, create default_tenant from .env
# - Process documents exactly as before
# - Results go to Google Sheets and Drive
```

**No code changes, no Firebase setup, just works!**

### Advanced: Multiple Tenants in Firebase

For multiple tenants with different configurations:

### Step 1: Set Up Tenant in Firebase

```python
# Run this once to initialize a tenant
from firebase_tenant_config import FirebaseTenantConfigManager, TenantConfig, CredentialsObject, RuleObject

manager = FirebaseTenantConfigManager()

credentials = CredentialsObject(
    openai_api_key="sk-xxx",
    google_service_account_json="json_string_here"
)

rule = RuleObject(
    rule_id="rule_1",
    rule_name="Invoice Processing",
    source_folder_id="drive_folder_id",
    target_folder_id="drive_dest_folder_id",
    target_sheet_id="sheets_id",
    sheet_tab_name="Invoices",
    is_enabled=True,
)

tenant = TenantConfig(
    tenant_id="my_tenant",
    name="My Company",
    active=True,
    credentials=credentials,
)

manager.save_tenant(tenant)
manager.add_rule("my_tenant", rule)
```

### Step 2: Run Orchestration

```bash
# This will automatically:
# 1. Load all active tenants from Firebase (not just one!)
# 2. For each tenant, process all enabled rules
# 3. Track results in tenant-specific collections
python document_orchester.py --model gpt-4o --send
```

### Step 3: Review Results

Results are stored in:
- Google Drive: Documents moved to `{target_folder_id}/{YYYY}{MM}/`
- Google Sheets: Parsed data appended to `{target_sheet_id}` sheet
- Firebase: Document metadata in `tenants/{tenant_id}/processed_documents/`

## Troubleshooting

### "No active tenants found"
- Check that `active: true` in Firebase tenant document
- Verify `FIREBASE_TENANTS_COLLECTION` env var is correct
- Confirm Firebase credentials are valid

### "No enabled rules"
- Check that `is_enabled: true` for at least one rule
- Verify rule has valid `source_folder_id`

### Credentials error
- Ensure `credentials.openai_api_key` and `credentials.google_service_account_json` are set
- Verify JSON strings are properly formatted

### Mixing modes
- Don't mix multi-tenant and legacy configurations
- Use either all Firebase-based tenants OR all env vars, not both
- When switching modes, ensure all required env vars/Firebase docs exist

## API Endpoints

### Gradio Interface

Two REST-compatible endpoints are exposed:

**Legacy Single-Tenant:**
```
POST /api/orchestrate_drive_documents
{
  "model": "gpt-4o",
  "include_subfolders": true,
  "send_to_sheet": true,
  "api_key": "your_orchestrator_api_key"
}
```

**Multi-Tenant:**
```
POST /api/orchestrate_all_tenants
{
  "model": "gpt-4o",
  "include_subfolders": true,
  "send_to_sheet": true,
  "api_key": "your_orchestrator_api_key"
}
```

## Performance Considerations

- Multi-tenant processing is sequential (one tenant at a time, then one rule at a time)
- Each tenant/rule maintains separate Firebase document tracking
- Large scale deployments may benefit from rule parallelization (future enhancement)

## Managing Rules

### Adding a Rule to an Existing Tenant

You can programmatically add rules to any tenant:

```python
from firebase_tenant_config import (
    FirebaseTenantConfigManager,
    RuleObject,
)

manager = FirebaseTenantConfigManager()

# Create a new rule
new_rule = RuleObject(
    rule_id="new_rule_123",
    rule_name="Additional Processing Rule",
    source_folder_id="1NEW_folder_id",
    target_folder_id="1NEW_destination_id",
    target_sheet_id="1NEW_sheet_id",
    sheet_tab_name="Sheet Tab Name",
    parsing_prompt="Custom parsing instructions...",
    is_enabled=True,
)

# Add to tenant
manager.add_rule("my_tenant", new_rule)
```

### Cloning a Rule

Use the provided script to clone the default rule:

```bash
python add_rule_from_default.py new_rule_id "New Rule Name"
```

This creates a new rule in the **default_tenant** with the same configuration (folder IDs, sheet, etc.) as the default rule. Perfect for creating multiple processing pipelines with identical settings.

### Removing a Rule

```python
from firebase_tenant_config import FirebaseTenantConfigManager

manager = FirebaseTenantConfigManager()
manager.remove_rule("my_tenant", "rule_id_to_remove")
```

## Future Enhancements

Potential improvements:
- Parallel rule processing within a tenant
- Rule scheduling (cron-like execution)
- Audit logging for rule changes
- Rule versioning and rollback
- Web UI for rule management