Spaces:
Sleeping
Sleeping
|
Download MULTI_TENANT_GUIDE.md from breakpointsoftware/document-parser: direct link, hf CLI and curl.
- Browser
- Download file 14.8 kB
-
https://huggingface.co/spaces/breakpointsoftware/document-parser/resolve/main/MULTI_TENANT_GUIDE.md
- Command line
-
hf download hf://spaces/breakpointsoftware/document-parser/MULTI_TENANT_GUIDE.md
-
curl -L -o MULTI_TENANT_GUIDE.md https://huggingface.co/spaces/breakpointsoftware/document-parser/resolve/main/MULTI_TENANT_GUIDE.md
14.8 kB
| # Multi-Tenant Document Parser - Configuration & Usage Guide | |
| ## Overview | |
| The document parser now supports two operational modes: | |
| 1. **Multi-Tenant Mode (NEW)** - Load all active tenants from Firebase and process their enabled rules | |
| 2. **Legacy Mode** - Single tenant configuration via environment variables (backward compatible) | |
| ## Architecture | |
| ### Data Model | |
| All tenant and rule configurations are stored in Firestore under the `tenants` collection: | |
| ``` | |
| TENANTS_COLLECTION (Firestore) | |
| βββ {tenant_id} | |
| β βββ tenant_id: unique tenant ID | |
| β βββ name: "Company/User Name" | |
| β βββ active: boolean (Account Status) | |
| β βββ created_at: timestamp | |
| β βββ updated_at: timestamp | |
| β βββ credentials (object, 1:1 relationship) | |
| β β βββ openai_api_key: "OpenAI Key" | |
| β β βββ google_service_account_json: "Service account JSON" | |
| β βββ rules (subcollection, 1:N relationship) | |
| β βββ {rule_id} | |
| β βββ rule_id: "unique rule ID" | |
| β βββ rule_name: "Invoices to Finance" | |
| β βββ source_folder_id: "Google Drive Source Folder ID" | |
| β βββ target_folder_id: "Google Drive Destination Folder ID" | |
| β βββ target_sheet_id: "Google Sheets Spreadsheet ID" | |
| β βββ sheet_tab_name: "Tab inside Google Sheet" | |
| β βββ parsing_prompt: "Custom LLM Instructions" | |
| β βββ is_enabled: boolean (Rule Toggle) | |
| ``` | |
| ### Processing Flow | |
| ``` | |
| Multi-Tenant Mode: | |
| 1. orchestrate_all_active_tenants() | |
| ββ> For each active tenant: | |
| ββ> For each enabled rule: | |
| ββ> orchestrate_single_rule() | |
| ββ> Scan rule.source_folder_id | |
| ββ> Parse documents with rule.parsing_prompt | |
| ββ> Move to rule.target_folder_id | |
| ββ> Send to rule.target_sheet_id | |
| ``` | |
| ## Usage | |
| ### Quick Start (Using .env) | |
| If you have existing `.env` configuration, multi-tenant mode will automatically create a default tenant: | |
| ```bash | |
| # Just run it! No Firebase setup needed initially | |
| python document_orchester.py --send | |
| # The system will: | |
| # 1. Check for active tenants in Firebase | |
| # 2. If none found, create "default_tenant" from your .env vars | |
| # 3. Process documents using the default configuration | |
| ``` | |
| **This is the easiest path if you're migrating from single-tenant mode.** | |
| ### Mode 1: Multi-Tenant Mode (Default) | |
| ```bash | |
| # Process all active tenants and their enabled rules | |
| python document_orchester.py --model gpt-4o --send | |
| # Optional flags: | |
| # --no-subfolders Only scan direct files (don't recurse) | |
| # --send Send results to Google Sheets | |
| # --output FILE Save JSON summary to file | |
| ``` | |
| ### Mode 2: Legacy Single-Tenant Mode | |
| ```bash | |
| # Use env var configuration (backward compatible) | |
| python document_orchester.py --legacy --model gpt-4o --send | |
| ``` | |
| ### Gradio Interface | |
| Start the web app: | |
| ```bash | |
| python app.py | |
| ``` | |
| Two orchestrator endpoints are now available: | |
| - **Single Tenant (Legacy)** - Uses env var configuration | |
| - **Multi Tenant** - Loads all active tenants from Firebase | |
| ## Configuration | |
| ### Automatic Default Tenant (No Firebase Setup Required) | |
| When you run multi-tenant mode without any tenants in Firebase: | |
| 1. The system checks for active tenants in Firestore | |
| 2. If none found, it creates a **default tenant** from your `.env` file | |
| 3. Maps env vars like this: | |
| | .env Variable | Default Tenant | | |
| |---|---| | |
| | `OPENAI_API_KEY` | `credentials.openai_api_key` | | |
| | `GOOGLE_SERVICE_ACCOUNT_JSON` | `credentials.google_service_account_json` | | |
| | `GOOGLE_DRIVE_FOLDER_IDS` (first) | `rule.source_folder_id` | | |
| | `GOOGLE_DRIVE_INVOICES_ROOT_FOLDER_ID` | `rule.target_folder_id` | | |
| | `GOOGLE_SHEETS_SPREADSHEET_ID` | `rule.target_sheet_id` | | |
| **Example: Your existing `.env`** | |
| ```env | |
| OPENAI_API_KEY=sk-abc123... | |
| GOOGLE_SERVICE_ACCOUNT_JSON={"type":"service_account","project_id":"my-project",...} | |
| GOOGLE_DRIVE_FOLDER_IDS=1ABC_folder_id_here | |
| GOOGLE_DRIVE_INVOICES_ROOT_FOLDER_ID=1XYZ_destination_folder_id | |
| GOOGLE_SHEETS_SPREADSHEET_ID=18qKL... | |
| ``` | |
| **Creates this default tenant (in memory, not saved to Firebase):** | |
| ```json | |
| { | |
| "tenant_id": "default_tenant", | |
| "name": "Default Tenant (from .env)", | |
| "active": true, | |
| "credentials": { | |
| "openai_api_key": "sk-abc123...", | |
| "google_service_account_json": "{...}" | |
| }, | |
| "rules": [ | |
| { | |
| "rule_id": "default_rule", | |
| "rule_name": "Default Processing Rule", | |
| "source_folder_id": "1ABC_folder_id_here", | |
| "target_folder_id": "1XYZ_destination_folder_id", | |
| "target_sheet_id": "18qKL...", | |
| "sheet_tab_name": "Parsed", | |
| "is_enabled": true | |
| } | |
| ] | |
| } | |
| ``` | |
| The default tenant provides automatic setup: | |
| - β Created automatically if no Firebase tenants exist | |
| - β Saved to Firebase **BEFORE** orchestration starts | |
| - β Future runs load from Firebase instead of recreating | |
| - β Try multi-tenant mode without any setup | |
| - β Migrate gradually from single-tenant to multi-tenant | |
| - β Keep existing workflows unchanged | |
| **The flow:** | |
| 1. First run: No Firebase tenants β Create default_tenant from .env β **Save to Firebase** β Process documents | |
| 2. Second run: Firebase has default_tenant β Load it β Process documents | |
| 3. Add more tenants anytime in Firebase β They're all processed together | |
| ### Multi-Tenant Setup (Firebase) | |
| 1. Create a Firestore collection called `tenants` (or set via `FIREBASE_TENANTS_COLLECTION` env var) | |
| 2. Add tenant documents with this structure: | |
| ```json | |
| { | |
| "name": "Acme Corp", | |
| "active": true, | |
| "created_at": "2024-01-01T00:00:00Z", | |
| "updated_at": "2024-01-01T00:00:00Z", | |
| "credentials": { | |
| "openai_api_key": "sk-...", | |
| "google_service_account_json": "{...full service account json...}" | |
| } | |
| } | |
| ``` | |
| 3. Create a **rules** subcollection inside the tenant document, and add rules with this structure: | |
| ```json | |
| { | |
| "rule_id": "invoices_2024", | |
| "rule_name": "Invoices to Finance", | |
| "source_folder_id": "1ABC...", | |
| "target_folder_id": "1XYZ...", | |
| "target_sheet_id": "18qKL...", | |
| "sheet_tab_name": "Invoices", | |
| "parsing_prompt": "Extract invoice details following these rules...", | |
| "is_enabled": true | |
| } | |
| ``` | |
| ### Environment Variables | |
| **Required for both modes:** | |
| ``` | |
| FIREBASE_SERVICE_ACCOUNT_JSON # Firebase credentials (JSON string or use GOOGLE_SERVICE_ACCOUNT_JSON) | |
| OPENAI_API_KEY # (only for legacy mode, multi-tenant uses tenant credentials) | |
| ``` | |
| **Optional:** | |
| ``` | |
| FIREBASE_TENANTS_COLLECTION # Default: "tenants" | |
| FIREBASE_DOCUMENTS_COLLECTION # Default: "processed_documents" | |
| FIREBASE_TRACKER_COLLECTION # Default: "processed_files" | |
| FIREBASE_TRACK_PROCESSED # Default: "true" | |
| OPENAI_MODEL # Default: "gpt-4o" | |
| GOOGLE_DRIVE_INVOICES_BASE_PATH # Default: "Facturas" | |
| ORCHESTRATOR_API_KEY # For API authentication | |
| ``` | |
| ## Firebase Storage Structure | |
| ### Document Tracking (Multi-Tenant) | |
| Each tenant's processed files are tracked in nested collections: | |
| ``` | |
| tenants/{tenant_id}/processed_files/{file_hash} | |
| ββ file_hash: "sha256 hash" | |
| ββ source_file: "filename" | |
| ββ processed_at: "timestamp" | |
| tenants/{tenant_id}/processed_documents/{document_id} | |
| ββ document_id: "drive file id" | |
| ββ parsed_data: {...} | |
| ββ status: "Parsed|Modified|Sent" | |
| ``` | |
| ### Creating Tenants Programmatically | |
| ```python | |
| from firebase_tenant_config import ( | |
| FirebaseTenantConfigManager, | |
| TenantConfig, | |
| CredentialsObject, | |
| RuleObject, | |
| ) | |
| manager = FirebaseTenantConfigManager() | |
| # Create credentials | |
| credentials = CredentialsObject( | |
| openai_api_key="sk-...", | |
| google_service_account_json="..." # full JSON as string | |
| ) | |
| # Create a rule | |
| rule = RuleObject( | |
| rule_id="invoices_2024", | |
| rule_name="Invoices to Finance", | |
| source_folder_id="1ABC...", | |
| target_folder_id="1XYZ...", | |
| target_sheet_id="18qKL...", | |
| sheet_tab_name="Invoices", | |
| parsing_prompt="Extract...", | |
| is_enabled=True, | |
| ) | |
| # Create tenant | |
| tenant = TenantConfig( | |
| tenant_id="acme-corp", | |
| name="Acme Corp", | |
| active=True, | |
| credentials=credentials, | |
| rules=[rule], | |
| ) | |
| # Save to Firebase | |
| manager.save_tenant(tenant) | |
| # List all active tenants | |
| active_tenants = manager.list_active_tenants() | |
| # Get enabled rules for a tenant | |
| enabled_rules = manager.get_enabled_rules("acme-corp") | |
| # Add a rule | |
| manager.add_rule("acme-corp", new_rule) | |
| # Remove a rule | |
| manager.remove_rule("acme-corp", "invoices_2024") | |
| ``` | |
| ## Output Format | |
| ### Multi-Tenant Orchestration Result | |
| ```json | |
| { | |
| "ok": true, | |
| "tenants_processed": 2, | |
| "total_parsed": 15, | |
| "total_modified": 3, | |
| "total_sent": 18, | |
| "total_moved": 18, | |
| "total_corrupted": 2, | |
| "total_errors": 1, | |
| "tenants_results": [ | |
| { | |
| "tenant_id": "tenant_123", | |
| "tenant_name": "Acme Corp", | |
| "ok": true, | |
| "rules_processed": 2, | |
| "parsed": 10, | |
| "modified": 2, | |
| "sent": 12, | |
| "moved": 12, | |
| "corrupted": 1, | |
| "errors": 0, | |
| "rules_results": [ | |
| { | |
| "ok": true, | |
| "tenant_id": "tenant_123", | |
| "rule_id": "invoices_2024", | |
| "rule_name": "Invoices to Finance", | |
| "scanned": 20, | |
| "to_process": 15, | |
| "skipped": 5, | |
| "parsed": 10, | |
| "modified": 2, | |
| "sent": 12, | |
| "moved": 12, | |
| "corrupted": 1, | |
| "errors": 0, | |
| "processed_items": [...], | |
| "skipped_items": [...] | |
| } | |
| ] | |
| } | |
| ] | |
| } | |
| ``` | |
| ## Migration from Legacy Mode | |
| **No migration needed!** The system maintains full backward compatibility: | |
| 1. Existing env var configuration continues to work with `--legacy` flag | |
| 2. New multi-tenant mode coexists without affecting legacy deployments | |
| 3. To enable multi-tenant: Set up tenants in Firebase and run without `--legacy` | |
| ## Workflow Examples | |
| ### Simple: Use Existing .env (No Firebase Setup) | |
| The **easiest** way to get started with multi-tenant mode: | |
| ```bash | |
| # 1. Ensure your .env has these vars (you probably already do): | |
| # OPENAI_API_KEY | |
| # GOOGLE_SERVICE_ACCOUNT_JSON | |
| # GOOGLE_DRIVE_FOLDER_IDS | |
| # GOOGLE_DRIVE_INVOICES_ROOT_FOLDER_ID | |
| # GOOGLE_SHEETS_SPREADSHEET_ID | |
| # 2. Just run multi-tenant orchestration: | |
| python document_orchester.py --send | |
| # That's it! The system will: | |
| # - Check for tenants in Firebase | |
| # - Find none, create default_tenant from .env | |
| # - Process documents exactly as before | |
| # - Results go to Google Sheets and Drive | |
| ``` | |
| **No code changes, no Firebase setup, just works!** | |
| ### Advanced: Multiple Tenants in Firebase | |
| For multiple tenants with different configurations: | |
| ### Step 1: Set Up Tenant in Firebase | |
| ```python | |
| # Run this once to initialize a tenant | |
| from firebase_tenant_config import FirebaseTenantConfigManager, TenantConfig, CredentialsObject, RuleObject | |
| manager = FirebaseTenantConfigManager() | |
| credentials = CredentialsObject( | |
| openai_api_key="sk-xxx", | |
| google_service_account_json="json_string_here" | |
| ) | |
| rule = RuleObject( | |
| rule_id="rule_1", | |
| rule_name="Invoice Processing", | |
| source_folder_id="drive_folder_id", | |
| target_folder_id="drive_dest_folder_id", | |
| target_sheet_id="sheets_id", | |
| sheet_tab_name="Invoices", | |
| is_enabled=True, | |
| ) | |
| tenant = TenantConfig( | |
| tenant_id="my_tenant", | |
| name="My Company", | |
| active=True, | |
| credentials=credentials, | |
| ) | |
| manager.save_tenant(tenant) | |
| manager.add_rule("my_tenant", rule) | |
| ``` | |
| ### Step 2: Run Orchestration | |
| ```bash | |
| # This will automatically: | |
| # 1. Load all active tenants from Firebase (not just one!) | |
| # 2. For each tenant, process all enabled rules | |
| # 3. Track results in tenant-specific collections | |
| python document_orchester.py --model gpt-4o --send | |
| ``` | |
| ### Step 3: Review Results | |
| Results are stored in: | |
| - Google Drive: Documents moved to `{target_folder_id}/{YYYY}{MM}/` | |
| - Google Sheets: Parsed data appended to `{target_sheet_id}` sheet | |
| - Firebase: Document metadata in `tenants/{tenant_id}/processed_documents/` | |
| ## Troubleshooting | |
| ### "No active tenants found" | |
| - Check that `active: true` in Firebase tenant document | |
| - Verify `FIREBASE_TENANTS_COLLECTION` env var is correct | |
| - Confirm Firebase credentials are valid | |
| ### "No enabled rules" | |
| - Check that `is_enabled: true` for at least one rule | |
| - Verify rule has valid `source_folder_id` | |
| ### Credentials error | |
| - Ensure `credentials.openai_api_key` and `credentials.google_service_account_json` are set | |
| - Verify JSON strings are properly formatted | |
| ### Mixing modes | |
| - Don't mix multi-tenant and legacy configurations | |
| - Use either all Firebase-based tenants OR all env vars, not both | |
| - When switching modes, ensure all required env vars/Firebase docs exist | |
| ## API Endpoints | |
| ### Gradio Interface | |
| Two REST-compatible endpoints are exposed: | |
| **Legacy Single-Tenant:** | |
| ``` | |
| POST /api/orchestrate_drive_documents | |
| { | |
| "model": "gpt-4o", | |
| "include_subfolders": true, | |
| "send_to_sheet": true, | |
| "api_key": "your_orchestrator_api_key" | |
| } | |
| ``` | |
| **Multi-Tenant:** | |
| ``` | |
| POST /api/orchestrate_all_tenants | |
| { | |
| "model": "gpt-4o", | |
| "include_subfolders": true, | |
| "send_to_sheet": true, | |
| "api_key": "your_orchestrator_api_key" | |
| } | |
| ``` | |
| ## Performance Considerations | |
| - Multi-tenant processing is sequential (one tenant at a time, then one rule at a time) | |
| - Each tenant/rule maintains separate Firebase document tracking | |
| - Large scale deployments may benefit from rule parallelization (future enhancement) | |
| ## Managing Rules | |
| ### Adding a Rule to an Existing Tenant | |
| You can programmatically add rules to any tenant: | |
| ```python | |
| from firebase_tenant_config import ( | |
| FirebaseTenantConfigManager, | |
| RuleObject, | |
| ) | |
| manager = FirebaseTenantConfigManager() | |
| # Create a new rule | |
| new_rule = RuleObject( | |
| rule_id="new_rule_123", | |
| rule_name="Additional Processing Rule", | |
| source_folder_id="1NEW_folder_id", | |
| target_folder_id="1NEW_destination_id", | |
| target_sheet_id="1NEW_sheet_id", | |
| sheet_tab_name="Sheet Tab Name", | |
| parsing_prompt="Custom parsing instructions...", | |
| is_enabled=True, | |
| ) | |
| # Add to tenant | |
| manager.add_rule("my_tenant", new_rule) | |
| ``` | |
| ### Cloning a Rule | |
| Use the provided script to clone the default rule: | |
| ```bash | |
| python add_rule_from_default.py new_rule_id "New Rule Name" | |
| ``` | |
| This creates a new rule in the **default_tenant** with the same configuration (folder IDs, sheet, etc.) as the default rule. Perfect for creating multiple processing pipelines with identical settings. | |
| ### Removing a Rule | |
| ```python | |
| from firebase_tenant_config import FirebaseTenantConfigManager | |
| manager = FirebaseTenantConfigManager() | |
| manager.remove_rule("my_tenant", "rule_id_to_remove") | |
| ``` | |
| ## Future Enhancements | |
| Potential improvements: | |
| - Parallel rule processing within a tenant | |
| - Rule scheduling (cron-like execution) | |
| - Audit logging for rule changes | |
| - Rule versioning and rollback | |
| - Web UI for rule management | |