document-parser / MULTI_TENANT_GUIDE.md
segovia
multitenant supprt
028d116
|
Raw History Blame Contribute Delete
14.8 kB
# Multi-Tenant Document Parser - Configuration & Usage Guide
## Overview
The document parser now supports two operational modes:
1. **Multi-Tenant Mode (NEW)** - Load all active tenants from Firebase and process their enabled rules
2. **Legacy Mode** - Single tenant configuration via environment variables (backward compatible)
## Architecture
### Data Model
All tenant and rule configurations are stored in Firestore under the `tenants` collection:
```
TENANTS_COLLECTION (Firestore)
β”œβ”€β”€ {tenant_id}
β”‚ β”œβ”€β”€ tenant_id: unique tenant ID
β”‚ β”œβ”€β”€ name: "Company/User Name"
β”‚ β”œβ”€β”€ active: boolean (Account Status)
β”‚ β”œβ”€β”€ created_at: timestamp
β”‚ β”œβ”€β”€ updated_at: timestamp
β”‚ β”œβ”€β”€ credentials (object, 1:1 relationship)
β”‚ β”‚ β”œβ”€β”€ openai_api_key: "OpenAI Key"
β”‚ β”‚ └── google_service_account_json: "Service account JSON"
β”‚ └── rules (subcollection, 1:N relationship)
β”‚ └── {rule_id}
β”‚ β”œβ”€β”€ rule_id: "unique rule ID"
β”‚ β”œβ”€β”€ rule_name: "Invoices to Finance"
β”‚ β”œβ”€β”€ source_folder_id: "Google Drive Source Folder ID"
β”‚ β”œβ”€β”€ target_folder_id: "Google Drive Destination Folder ID"
β”‚ β”œβ”€β”€ target_sheet_id: "Google Sheets Spreadsheet ID"
β”‚ β”œβ”€β”€ sheet_tab_name: "Tab inside Google Sheet"
β”‚ β”œβ”€β”€ parsing_prompt: "Custom LLM Instructions"
β”‚ └── is_enabled: boolean (Rule Toggle)
```
### Processing Flow
```
Multi-Tenant Mode:
1. orchestrate_all_active_tenants()
└─> For each active tenant:
└─> For each enabled rule:
└─> orchestrate_single_rule()
β”œβ”€> Scan rule.source_folder_id
β”œβ”€> Parse documents with rule.parsing_prompt
β”œβ”€> Move to rule.target_folder_id
└─> Send to rule.target_sheet_id
```
## Usage
### Quick Start (Using .env)
If you have existing `.env` configuration, multi-tenant mode will automatically create a default tenant:
```bash
# Just run it! No Firebase setup needed initially
python document_orchester.py --send
# The system will:
# 1. Check for active tenants in Firebase
# 2. If none found, create "default_tenant" from your .env vars
# 3. Process documents using the default configuration
```
**This is the easiest path if you're migrating from single-tenant mode.**
### Mode 1: Multi-Tenant Mode (Default)
```bash
# Process all active tenants and their enabled rules
python document_orchester.py --model gpt-4o --send
# Optional flags:
# --no-subfolders Only scan direct files (don't recurse)
# --send Send results to Google Sheets
# --output FILE Save JSON summary to file
```
### Mode 2: Legacy Single-Tenant Mode
```bash
# Use env var configuration (backward compatible)
python document_orchester.py --legacy --model gpt-4o --send
```
### Gradio Interface
Start the web app:
```bash
python app.py
```
Two orchestrator endpoints are now available:
- **Single Tenant (Legacy)** - Uses env var configuration
- **Multi Tenant** - Loads all active tenants from Firebase
## Configuration
### Automatic Default Tenant (No Firebase Setup Required)
When you run multi-tenant mode without any tenants in Firebase:
1. The system checks for active tenants in Firestore
2. If none found, it creates a **default tenant** from your `.env` file
3. Maps env vars like this:
| .env Variable | Default Tenant |
|---|---|
| `OPENAI_API_KEY` | `credentials.openai_api_key` |
| `GOOGLE_SERVICE_ACCOUNT_JSON` | `credentials.google_service_account_json` |
| `GOOGLE_DRIVE_FOLDER_IDS` (first) | `rule.source_folder_id` |
| `GOOGLE_DRIVE_INVOICES_ROOT_FOLDER_ID` | `rule.target_folder_id` |
| `GOOGLE_SHEETS_SPREADSHEET_ID` | `rule.target_sheet_id` |
**Example: Your existing `.env`**
```env
OPENAI_API_KEY=sk-abc123...
GOOGLE_SERVICE_ACCOUNT_JSON={"type":"service_account","project_id":"my-project",...}
GOOGLE_DRIVE_FOLDER_IDS=1ABC_folder_id_here
GOOGLE_DRIVE_INVOICES_ROOT_FOLDER_ID=1XYZ_destination_folder_id
GOOGLE_SHEETS_SPREADSHEET_ID=18qKL...
```
**Creates this default tenant (in memory, not saved to Firebase):**
```json
{
"tenant_id": "default_tenant",
"name": "Default Tenant (from .env)",
"active": true,
"credentials": {
"openai_api_key": "sk-abc123...",
"google_service_account_json": "{...}"
},
"rules": [
{
"rule_id": "default_rule",
"rule_name": "Default Processing Rule",
"source_folder_id": "1ABC_folder_id_here",
"target_folder_id": "1XYZ_destination_folder_id",
"target_sheet_id": "18qKL...",
"sheet_tab_name": "Parsed",
"is_enabled": true
}
]
}
```
The default tenant provides automatic setup:
- βœ… Created automatically if no Firebase tenants exist
- βœ… Saved to Firebase **BEFORE** orchestration starts
- βœ… Future runs load from Firebase instead of recreating
- βœ… Try multi-tenant mode without any setup
- βœ… Migrate gradually from single-tenant to multi-tenant
- βœ… Keep existing workflows unchanged
**The flow:**
1. First run: No Firebase tenants β†’ Create default_tenant from .env β†’ **Save to Firebase** β†’ Process documents
2. Second run: Firebase has default_tenant β†’ Load it β†’ Process documents
3. Add more tenants anytime in Firebase β†’ They're all processed together
### Multi-Tenant Setup (Firebase)
1. Create a Firestore collection called `tenants` (or set via `FIREBASE_TENANTS_COLLECTION` env var)
2. Add tenant documents with this structure:
```json
{
"name": "Acme Corp",
"active": true,
"created_at": "2024-01-01T00:00:00Z",
"updated_at": "2024-01-01T00:00:00Z",
"credentials": {
"openai_api_key": "sk-...",
"google_service_account_json": "{...full service account json...}"
}
}
```
3. Create a **rules** subcollection inside the tenant document, and add rules with this structure:
```json
{
"rule_id": "invoices_2024",
"rule_name": "Invoices to Finance",
"source_folder_id": "1ABC...",
"target_folder_id": "1XYZ...",
"target_sheet_id": "18qKL...",
"sheet_tab_name": "Invoices",
"parsing_prompt": "Extract invoice details following these rules...",
"is_enabled": true
}
```
### Environment Variables
**Required for both modes:**
```
FIREBASE_SERVICE_ACCOUNT_JSON # Firebase credentials (JSON string or use GOOGLE_SERVICE_ACCOUNT_JSON)
OPENAI_API_KEY # (only for legacy mode, multi-tenant uses tenant credentials)
```
**Optional:**
```
FIREBASE_TENANTS_COLLECTION # Default: "tenants"
FIREBASE_DOCUMENTS_COLLECTION # Default: "processed_documents"
FIREBASE_TRACKER_COLLECTION # Default: "processed_files"
FIREBASE_TRACK_PROCESSED # Default: "true"
OPENAI_MODEL # Default: "gpt-4o"
GOOGLE_DRIVE_INVOICES_BASE_PATH # Default: "Facturas"
ORCHESTRATOR_API_KEY # For API authentication
```
## Firebase Storage Structure
### Document Tracking (Multi-Tenant)
Each tenant's processed files are tracked in nested collections:
```
tenants/{tenant_id}/processed_files/{file_hash}
└─ file_hash: "sha256 hash"
└─ source_file: "filename"
└─ processed_at: "timestamp"
tenants/{tenant_id}/processed_documents/{document_id}
└─ document_id: "drive file id"
└─ parsed_data: {...}
└─ status: "Parsed|Modified|Sent"
```
### Creating Tenants Programmatically
```python
from firebase_tenant_config import (
FirebaseTenantConfigManager,
TenantConfig,
CredentialsObject,
RuleObject,
)
manager = FirebaseTenantConfigManager()
# Create credentials
credentials = CredentialsObject(
openai_api_key="sk-...",
google_service_account_json="..." # full JSON as string
)
# Create a rule
rule = RuleObject(
rule_id="invoices_2024",
rule_name="Invoices to Finance",
source_folder_id="1ABC...",
target_folder_id="1XYZ...",
target_sheet_id="18qKL...",
sheet_tab_name="Invoices",
parsing_prompt="Extract...",
is_enabled=True,
)
# Create tenant
tenant = TenantConfig(
tenant_id="acme-corp",
name="Acme Corp",
active=True,
credentials=credentials,
rules=[rule],
)
# Save to Firebase
manager.save_tenant(tenant)
# List all active tenants
active_tenants = manager.list_active_tenants()
# Get enabled rules for a tenant
enabled_rules = manager.get_enabled_rules("acme-corp")
# Add a rule
manager.add_rule("acme-corp", new_rule)
# Remove a rule
manager.remove_rule("acme-corp", "invoices_2024")
```
## Output Format
### Multi-Tenant Orchestration Result
```json
{
"ok": true,
"tenants_processed": 2,
"total_parsed": 15,
"total_modified": 3,
"total_sent": 18,
"total_moved": 18,
"total_corrupted": 2,
"total_errors": 1,
"tenants_results": [
{
"tenant_id": "tenant_123",
"tenant_name": "Acme Corp",
"ok": true,
"rules_processed": 2,
"parsed": 10,
"modified": 2,
"sent": 12,
"moved": 12,
"corrupted": 1,
"errors": 0,
"rules_results": [
{
"ok": true,
"tenant_id": "tenant_123",
"rule_id": "invoices_2024",
"rule_name": "Invoices to Finance",
"scanned": 20,
"to_process": 15,
"skipped": 5,
"parsed": 10,
"modified": 2,
"sent": 12,
"moved": 12,
"corrupted": 1,
"errors": 0,
"processed_items": [...],
"skipped_items": [...]
}
]
}
]
}
```
## Migration from Legacy Mode
**No migration needed!** The system maintains full backward compatibility:
1. Existing env var configuration continues to work with `--legacy` flag
2. New multi-tenant mode coexists without affecting legacy deployments
3. To enable multi-tenant: Set up tenants in Firebase and run without `--legacy`
## Workflow Examples
### Simple: Use Existing .env (No Firebase Setup)
The **easiest** way to get started with multi-tenant mode:
```bash
# 1. Ensure your .env has these vars (you probably already do):
# OPENAI_API_KEY
# GOOGLE_SERVICE_ACCOUNT_JSON
# GOOGLE_DRIVE_FOLDER_IDS
# GOOGLE_DRIVE_INVOICES_ROOT_FOLDER_ID
# GOOGLE_SHEETS_SPREADSHEET_ID
# 2. Just run multi-tenant orchestration:
python document_orchester.py --send
# That's it! The system will:
# - Check for tenants in Firebase
# - Find none, create default_tenant from .env
# - Process documents exactly as before
# - Results go to Google Sheets and Drive
```
**No code changes, no Firebase setup, just works!**
### Advanced: Multiple Tenants in Firebase
For multiple tenants with different configurations:
### Step 1: Set Up Tenant in Firebase
```python
# Run this once to initialize a tenant
from firebase_tenant_config import FirebaseTenantConfigManager, TenantConfig, CredentialsObject, RuleObject
manager = FirebaseTenantConfigManager()
credentials = CredentialsObject(
openai_api_key="sk-xxx",
google_service_account_json="json_string_here"
)
rule = RuleObject(
rule_id="rule_1",
rule_name="Invoice Processing",
source_folder_id="drive_folder_id",
target_folder_id="drive_dest_folder_id",
target_sheet_id="sheets_id",
sheet_tab_name="Invoices",
is_enabled=True,
)
tenant = TenantConfig(
tenant_id="my_tenant",
name="My Company",
active=True,
credentials=credentials,
)
manager.save_tenant(tenant)
manager.add_rule("my_tenant", rule)
```
### Step 2: Run Orchestration
```bash
# This will automatically:
# 1. Load all active tenants from Firebase (not just one!)
# 2. For each tenant, process all enabled rules
# 3. Track results in tenant-specific collections
python document_orchester.py --model gpt-4o --send
```
### Step 3: Review Results
Results are stored in:
- Google Drive: Documents moved to `{target_folder_id}/{YYYY}{MM}/`
- Google Sheets: Parsed data appended to `{target_sheet_id}` sheet
- Firebase: Document metadata in `tenants/{tenant_id}/processed_documents/`
## Troubleshooting
### "No active tenants found"
- Check that `active: true` in Firebase tenant document
- Verify `FIREBASE_TENANTS_COLLECTION` env var is correct
- Confirm Firebase credentials are valid
### "No enabled rules"
- Check that `is_enabled: true` for at least one rule
- Verify rule has valid `source_folder_id`
### Credentials error
- Ensure `credentials.openai_api_key` and `credentials.google_service_account_json` are set
- Verify JSON strings are properly formatted
### Mixing modes
- Don't mix multi-tenant and legacy configurations
- Use either all Firebase-based tenants OR all env vars, not both
- When switching modes, ensure all required env vars/Firebase docs exist
## API Endpoints
### Gradio Interface
Two REST-compatible endpoints are exposed:
**Legacy Single-Tenant:**
```
POST /api/orchestrate_drive_documents
{
"model": "gpt-4o",
"include_subfolders": true,
"send_to_sheet": true,
"api_key": "your_orchestrator_api_key"
}
```
**Multi-Tenant:**
```
POST /api/orchestrate_all_tenants
{
"model": "gpt-4o",
"include_subfolders": true,
"send_to_sheet": true,
"api_key": "your_orchestrator_api_key"
}
```
## Performance Considerations
- Multi-tenant processing is sequential (one tenant at a time, then one rule at a time)
- Each tenant/rule maintains separate Firebase document tracking
- Large scale deployments may benefit from rule parallelization (future enhancement)
## Managing Rules
### Adding a Rule to an Existing Tenant
You can programmatically add rules to any tenant:
```python
from firebase_tenant_config import (
FirebaseTenantConfigManager,
RuleObject,
)
manager = FirebaseTenantConfigManager()
# Create a new rule
new_rule = RuleObject(
rule_id="new_rule_123",
rule_name="Additional Processing Rule",
source_folder_id="1NEW_folder_id",
target_folder_id="1NEW_destination_id",
target_sheet_id="1NEW_sheet_id",
sheet_tab_name="Sheet Tab Name",
parsing_prompt="Custom parsing instructions...",
is_enabled=True,
)
# Add to tenant
manager.add_rule("my_tenant", new_rule)
```
### Cloning a Rule
Use the provided script to clone the default rule:
```bash
python add_rule_from_default.py new_rule_id "New Rule Name"
```
This creates a new rule in the **default_tenant** with the same configuration (folder IDs, sheet, etc.) as the default rule. Perfect for creating multiple processing pipelines with identical settings.
### Removing a Rule
```python
from firebase_tenant_config import FirebaseTenantConfigManager
manager = FirebaseTenantConfigManager()
manager.remove_rule("my_tenant", "rule_id_to_remove")
```
## Future Enhancements
Potential improvements:
- Parallel rule processing within a tenant
- Rule scheduling (cron-like execution)
- Audit logging for rule changes
- Rule versioning and rollback
- Web UI for rule management