# Multi-Tenant Document Parser - Configuration & Usage Guide ## Overview The document parser now supports two operational modes: 1. **Multi-Tenant Mode (NEW)** - Load all active tenants from Firebase and process their enabled rules 2. **Legacy Mode** - Single tenant configuration via environment variables (backward compatible) ## Architecture ### Data Model All tenant and rule configurations are stored in Firestore under the `tenants` collection: ``` TENANTS_COLLECTION (Firestore) ├── {tenant_id} │ ├── tenant_id: unique tenant ID │ ├── name: "Company/User Name" │ ├── active: boolean (Account Status) │ ├── created_at: timestamp │ ├── updated_at: timestamp │ ├── credentials (object, 1:1 relationship) │ │ ├── openai_api_key: "OpenAI Key" │ │ └── google_service_account_json: "Service account JSON" │ └── rules (subcollection, 1:N relationship) │ └── {rule_id} │ ├── rule_id: "unique rule ID" │ ├── rule_name: "Invoices to Finance" │ ├── source_folder_id: "Google Drive Source Folder ID" │ ├── target_folder_id: "Google Drive Destination Folder ID" │ ├── target_sheet_id: "Google Sheets Spreadsheet ID" │ ├── sheet_tab_name: "Tab inside Google Sheet" │ ├── parsing_prompt: "Custom LLM Instructions" │ └── is_enabled: boolean (Rule Toggle) ``` ### Processing Flow ``` Multi-Tenant Mode: 1. orchestrate_all_active_tenants() └─> For each active tenant: └─> For each enabled rule: └─> orchestrate_single_rule() ├─> Scan rule.source_folder_id ├─> Parse documents with rule.parsing_prompt ├─> Move to rule.target_folder_id └─> Send to rule.target_sheet_id ``` ## Usage ### Quick Start (Using .env) If you have existing `.env` configuration, multi-tenant mode will automatically create a default tenant: ```bash # Just run it! No Firebase setup needed initially python document_orchester.py --send # The system will: # 1. Check for active tenants in Firebase # 2. If none found, create "default_tenant" from your .env vars # 3. Process documents using the default configuration ``` **This is the easiest path if you're migrating from single-tenant mode.** ### Mode 1: Multi-Tenant Mode (Default) ```bash # Process all active tenants and their enabled rules python document_orchester.py --model gpt-4o --send # Optional flags: # --no-subfolders Only scan direct files (don't recurse) # --send Send results to Google Sheets # --output FILE Save JSON summary to file ``` ### Mode 2: Legacy Single-Tenant Mode ```bash # Use env var configuration (backward compatible) python document_orchester.py --legacy --model gpt-4o --send ``` ### Gradio Interface Start the web app: ```bash python app.py ``` Two orchestrator endpoints are now available: - **Single Tenant (Legacy)** - Uses env var configuration - **Multi Tenant** - Loads all active tenants from Firebase ## Configuration ### Automatic Default Tenant (No Firebase Setup Required) When you run multi-tenant mode without any tenants in Firebase: 1. The system checks for active tenants in Firestore 2. If none found, it creates a **default tenant** from your `.env` file 3. Maps env vars like this: | .env Variable | Default Tenant | |---|---| | `OPENAI_API_KEY` | `credentials.openai_api_key` | | `GOOGLE_SERVICE_ACCOUNT_JSON` | `credentials.google_service_account_json` | | `GOOGLE_DRIVE_FOLDER_IDS` (first) | `rule.source_folder_id` | | `GOOGLE_DRIVE_INVOICES_ROOT_FOLDER_ID` | `rule.target_folder_id` | | `GOOGLE_SHEETS_SPREADSHEET_ID` | `rule.target_sheet_id` | **Example: Your existing `.env`** ```env OPENAI_API_KEY=sk-abc123... GOOGLE_SERVICE_ACCOUNT_JSON={"type":"service_account","project_id":"my-project",...} GOOGLE_DRIVE_FOLDER_IDS=1ABC_folder_id_here GOOGLE_DRIVE_INVOICES_ROOT_FOLDER_ID=1XYZ_destination_folder_id GOOGLE_SHEETS_SPREADSHEET_ID=18qKL... ``` **Creates this default tenant (in memory, not saved to Firebase):** ```json { "tenant_id": "default_tenant", "name": "Default Tenant (from .env)", "active": true, "credentials": { "openai_api_key": "sk-abc123...", "google_service_account_json": "{...}" }, "rules": [ { "rule_id": "default_rule", "rule_name": "Default Processing Rule", "source_folder_id": "1ABC_folder_id_here", "target_folder_id": "1XYZ_destination_folder_id", "target_sheet_id": "18qKL...", "sheet_tab_name": "Parsed", "is_enabled": true } ] } ``` The default tenant provides automatic setup: - ✅ Created automatically if no Firebase tenants exist - ✅ Saved to Firebase **BEFORE** orchestration starts - ✅ Future runs load from Firebase instead of recreating - ✅ Try multi-tenant mode without any setup - ✅ Migrate gradually from single-tenant to multi-tenant - ✅ Keep existing workflows unchanged **The flow:** 1. First run: No Firebase tenants → Create default_tenant from .env → **Save to Firebase** → Process documents 2. Second run: Firebase has default_tenant → Load it → Process documents 3. Add more tenants anytime in Firebase → They're all processed together ### Multi-Tenant Setup (Firebase) 1. Create a Firestore collection called `tenants` (or set via `FIREBASE_TENANTS_COLLECTION` env var) 2. Add tenant documents with this structure: ```json { "name": "Acme Corp", "active": true, "created_at": "2024-01-01T00:00:00Z", "updated_at": "2024-01-01T00:00:00Z", "credentials": { "openai_api_key": "sk-...", "google_service_account_json": "{...full service account json...}" } } ``` 3. Create a **rules** subcollection inside the tenant document, and add rules with this structure: ```json { "rule_id": "invoices_2024", "rule_name": "Invoices to Finance", "source_folder_id": "1ABC...", "target_folder_id": "1XYZ...", "target_sheet_id": "18qKL...", "sheet_tab_name": "Invoices", "parsing_prompt": "Extract invoice details following these rules...", "is_enabled": true } ``` ### Environment Variables **Required for both modes:** ``` FIREBASE_SERVICE_ACCOUNT_JSON # Firebase credentials (JSON string or use GOOGLE_SERVICE_ACCOUNT_JSON) OPENAI_API_KEY # (only for legacy mode, multi-tenant uses tenant credentials) ``` **Optional:** ``` FIREBASE_TENANTS_COLLECTION # Default: "tenants" FIREBASE_DOCUMENTS_COLLECTION # Default: "processed_documents" FIREBASE_TRACKER_COLLECTION # Default: "processed_files" FIREBASE_TRACK_PROCESSED # Default: "true" OPENAI_MODEL # Default: "gpt-4o" GOOGLE_DRIVE_INVOICES_BASE_PATH # Default: "Facturas" ORCHESTRATOR_API_KEY # For API authentication ``` ## Firebase Storage Structure ### Document Tracking (Multi-Tenant) Each tenant's processed files are tracked in nested collections: ``` tenants/{tenant_id}/processed_files/{file_hash} └─ file_hash: "sha256 hash" └─ source_file: "filename" └─ processed_at: "timestamp" tenants/{tenant_id}/processed_documents/{document_id} └─ document_id: "drive file id" └─ parsed_data: {...} └─ status: "Parsed|Modified|Sent" ``` ### Creating Tenants Programmatically ```python from firebase_tenant_config import ( FirebaseTenantConfigManager, TenantConfig, CredentialsObject, RuleObject, ) manager = FirebaseTenantConfigManager() # Create credentials credentials = CredentialsObject( openai_api_key="sk-...", google_service_account_json="..." # full JSON as string ) # Create a rule rule = RuleObject( rule_id="invoices_2024", rule_name="Invoices to Finance", source_folder_id="1ABC...", target_folder_id="1XYZ...", target_sheet_id="18qKL...", sheet_tab_name="Invoices", parsing_prompt="Extract...", is_enabled=True, ) # Create tenant tenant = TenantConfig( tenant_id="acme-corp", name="Acme Corp", active=True, credentials=credentials, rules=[rule], ) # Save to Firebase manager.save_tenant(tenant) # List all active tenants active_tenants = manager.list_active_tenants() # Get enabled rules for a tenant enabled_rules = manager.get_enabled_rules("acme-corp") # Add a rule manager.add_rule("acme-corp", new_rule) # Remove a rule manager.remove_rule("acme-corp", "invoices_2024") ``` ## Output Format ### Multi-Tenant Orchestration Result ```json { "ok": true, "tenants_processed": 2, "total_parsed": 15, "total_modified": 3, "total_sent": 18, "total_moved": 18, "total_corrupted": 2, "total_errors": 1, "tenants_results": [ { "tenant_id": "tenant_123", "tenant_name": "Acme Corp", "ok": true, "rules_processed": 2, "parsed": 10, "modified": 2, "sent": 12, "moved": 12, "corrupted": 1, "errors": 0, "rules_results": [ { "ok": true, "tenant_id": "tenant_123", "rule_id": "invoices_2024", "rule_name": "Invoices to Finance", "scanned": 20, "to_process": 15, "skipped": 5, "parsed": 10, "modified": 2, "sent": 12, "moved": 12, "corrupted": 1, "errors": 0, "processed_items": [...], "skipped_items": [...] } ] } ] } ``` ## Migration from Legacy Mode **No migration needed!** The system maintains full backward compatibility: 1. Existing env var configuration continues to work with `--legacy` flag 2. New multi-tenant mode coexists without affecting legacy deployments 3. To enable multi-tenant: Set up tenants in Firebase and run without `--legacy` ## Workflow Examples ### Simple: Use Existing .env (No Firebase Setup) The **easiest** way to get started with multi-tenant mode: ```bash # 1. Ensure your .env has these vars (you probably already do): # OPENAI_API_KEY # GOOGLE_SERVICE_ACCOUNT_JSON # GOOGLE_DRIVE_FOLDER_IDS # GOOGLE_DRIVE_INVOICES_ROOT_FOLDER_ID # GOOGLE_SHEETS_SPREADSHEET_ID # 2. Just run multi-tenant orchestration: python document_orchester.py --send # That's it! The system will: # - Check for tenants in Firebase # - Find none, create default_tenant from .env # - Process documents exactly as before # - Results go to Google Sheets and Drive ``` **No code changes, no Firebase setup, just works!** ### Advanced: Multiple Tenants in Firebase For multiple tenants with different configurations: ### Step 1: Set Up Tenant in Firebase ```python # Run this once to initialize a tenant from firebase_tenant_config import FirebaseTenantConfigManager, TenantConfig, CredentialsObject, RuleObject manager = FirebaseTenantConfigManager() credentials = CredentialsObject( openai_api_key="sk-xxx", google_service_account_json="json_string_here" ) rule = RuleObject( rule_id="rule_1", rule_name="Invoice Processing", source_folder_id="drive_folder_id", target_folder_id="drive_dest_folder_id", target_sheet_id="sheets_id", sheet_tab_name="Invoices", is_enabled=True, ) tenant = TenantConfig( tenant_id="my_tenant", name="My Company", active=True, credentials=credentials, ) manager.save_tenant(tenant) manager.add_rule("my_tenant", rule) ``` ### Step 2: Run Orchestration ```bash # This will automatically: # 1. Load all active tenants from Firebase (not just one!) # 2. For each tenant, process all enabled rules # 3. Track results in tenant-specific collections python document_orchester.py --model gpt-4o --send ``` ### Step 3: Review Results Results are stored in: - Google Drive: Documents moved to `{target_folder_id}/{YYYY}{MM}/` - Google Sheets: Parsed data appended to `{target_sheet_id}` sheet - Firebase: Document metadata in `tenants/{tenant_id}/processed_documents/` ## Troubleshooting ### "No active tenants found" - Check that `active: true` in Firebase tenant document - Verify `FIREBASE_TENANTS_COLLECTION` env var is correct - Confirm Firebase credentials are valid ### "No enabled rules" - Check that `is_enabled: true` for at least one rule - Verify rule has valid `source_folder_id` ### Credentials error - Ensure `credentials.openai_api_key` and `credentials.google_service_account_json` are set - Verify JSON strings are properly formatted ### Mixing modes - Don't mix multi-tenant and legacy configurations - Use either all Firebase-based tenants OR all env vars, not both - When switching modes, ensure all required env vars/Firebase docs exist ## API Endpoints ### Gradio Interface Two REST-compatible endpoints are exposed: **Legacy Single-Tenant:** ``` POST /api/orchestrate_drive_documents { "model": "gpt-4o", "include_subfolders": true, "send_to_sheet": true, "api_key": "your_orchestrator_api_key" } ``` **Multi-Tenant:** ``` POST /api/orchestrate_all_tenants { "model": "gpt-4o", "include_subfolders": true, "send_to_sheet": true, "api_key": "your_orchestrator_api_key" } ``` ## Performance Considerations - Multi-tenant processing is sequential (one tenant at a time, then one rule at a time) - Each tenant/rule maintains separate Firebase document tracking - Large scale deployments may benefit from rule parallelization (future enhancement) ## Managing Rules ### Adding a Rule to an Existing Tenant You can programmatically add rules to any tenant: ```python from firebase_tenant_config import ( FirebaseTenantConfigManager, RuleObject, ) manager = FirebaseTenantConfigManager() # Create a new rule new_rule = RuleObject( rule_id="new_rule_123", rule_name="Additional Processing Rule", source_folder_id="1NEW_folder_id", target_folder_id="1NEW_destination_id", target_sheet_id="1NEW_sheet_id", sheet_tab_name="Sheet Tab Name", parsing_prompt="Custom parsing instructions...", is_enabled=True, ) # Add to tenant manager.add_rule("my_tenant", new_rule) ``` ### Cloning a Rule Use the provided script to clone the default rule: ```bash python add_rule_from_default.py new_rule_id "New Rule Name" ``` This creates a new rule in the **default_tenant** with the same configuration (folder IDs, sheet, etc.) as the default rule. Perfect for creating multiple processing pipelines with identical settings. ### Removing a Rule ```python from firebase_tenant_config import FirebaseTenantConfigManager manager = FirebaseTenantConfigManager() manager.remove_rule("my_tenant", "rule_id_to_remove") ``` ## Future Enhancements Potential improvements: - Parallel rule processing within a tenant - Rule scheduling (cron-like execution) - Audit logging for rule changes - Rule versioning and rollback - Web UI for rule management