Spaces:
Sleeping
Download MULTI_TENANT_GUIDE.md from breakpointsoftware/document-parser: direct link, hf CLI and curl.
- Browser
- Download file 14.8 kB
-
https://huggingface.co/spaces/breakpointsoftware/document-parser/resolve/main/MULTI_TENANT_GUIDE.md
- Command line
-
hf download hf://spaces/breakpointsoftware/document-parser/MULTI_TENANT_GUIDE.md
-
curl -L -o MULTI_TENANT_GUIDE.md https://huggingface.co/spaces/breakpointsoftware/document-parser/resolve/main/MULTI_TENANT_GUIDE.md
A newer version of the Gradio SDK is available: 6.32.0
Multi-Tenant Document Parser - Configuration & Usage Guide
Overview
The document parser now supports two operational modes:
- Multi-Tenant Mode (NEW) - Load all active tenants from Firebase and process their enabled rules
- Legacy Mode - Single tenant configuration via environment variables (backward compatible)
Architecture
Data Model
All tenant and rule configurations are stored in Firestore under the tenants collection:
TENANTS_COLLECTION (Firestore)
βββ {tenant_id}
β βββ tenant_id: unique tenant ID
β βββ name: "Company/User Name"
β βββ active: boolean (Account Status)
β βββ created_at: timestamp
β βββ updated_at: timestamp
β βββ credentials (object, 1:1 relationship)
β β βββ openai_api_key: "OpenAI Key"
β β βββ google_service_account_json: "Service account JSON"
β βββ rules (subcollection, 1:N relationship)
β βββ {rule_id}
β βββ rule_id: "unique rule ID"
β βββ rule_name: "Invoices to Finance"
β βββ source_folder_id: "Google Drive Source Folder ID"
β βββ target_folder_id: "Google Drive Destination Folder ID"
β βββ target_sheet_id: "Google Sheets Spreadsheet ID"
β βββ sheet_tab_name: "Tab inside Google Sheet"
β βββ parsing_prompt: "Custom LLM Instructions"
β βββ is_enabled: boolean (Rule Toggle)
Processing Flow
Multi-Tenant Mode:
1. orchestrate_all_active_tenants()
ββ> For each active tenant:
ββ> For each enabled rule:
ββ> orchestrate_single_rule()
ββ> Scan rule.source_folder_id
ββ> Parse documents with rule.parsing_prompt
ββ> Move to rule.target_folder_id
ββ> Send to rule.target_sheet_id
Usage
Quick Start (Using .env)
If you have existing .env configuration, multi-tenant mode will automatically create a default tenant:
# Just run it! No Firebase setup needed initially
python document_orchester.py --send
# The system will:
# 1. Check for active tenants in Firebase
# 2. If none found, create "default_tenant" from your .env vars
# 3. Process documents using the default configuration
This is the easiest path if you're migrating from single-tenant mode.
Mode 1: Multi-Tenant Mode (Default)
# Process all active tenants and their enabled rules
python document_orchester.py --model gpt-4o --send
# Optional flags:
# --no-subfolders Only scan direct files (don't recurse)
# --send Send results to Google Sheets
# --output FILE Save JSON summary to file
Mode 2: Legacy Single-Tenant Mode
# Use env var configuration (backward compatible)
python document_orchester.py --legacy --model gpt-4o --send
Gradio Interface
Start the web app:
python app.py
Two orchestrator endpoints are now available:
- Single Tenant (Legacy) - Uses env var configuration
- Multi Tenant - Loads all active tenants from Firebase
Configuration
Automatic Default Tenant (No Firebase Setup Required)
When you run multi-tenant mode without any tenants in Firebase:
- The system checks for active tenants in Firestore
- If none found, it creates a default tenant from your
.envfile - Maps env vars like this:
| .env Variable | Default Tenant |
|---|---|
OPENAI_API_KEY |
credentials.openai_api_key |
GOOGLE_SERVICE_ACCOUNT_JSON |
credentials.google_service_account_json |
GOOGLE_DRIVE_FOLDER_IDS (first) |
rule.source_folder_id |
GOOGLE_DRIVE_INVOICES_ROOT_FOLDER_ID |
rule.target_folder_id |
GOOGLE_SHEETS_SPREADSHEET_ID |
rule.target_sheet_id |
Example: Your existing .env
OPENAI_API_KEY=sk-abc123...
GOOGLE_SERVICE_ACCOUNT_JSON={"type":"service_account","project_id":"my-project",...}
GOOGLE_DRIVE_FOLDER_IDS=1ABC_folder_id_here
GOOGLE_DRIVE_INVOICES_ROOT_FOLDER_ID=1XYZ_destination_folder_id
GOOGLE_SHEETS_SPREADSHEET_ID=18qKL...
Creates this default tenant (in memory, not saved to Firebase):
{
"tenant_id": "default_tenant",
"name": "Default Tenant (from .env)",
"active": true,
"credentials": {
"openai_api_key": "sk-abc123...",
"google_service_account_json": "{...}"
},
"rules": [
{
"rule_id": "default_rule",
"rule_name": "Default Processing Rule",
"source_folder_id": "1ABC_folder_id_here",
"target_folder_id": "1XYZ_destination_folder_id",
"target_sheet_id": "18qKL...",
"sheet_tab_name": "Parsed",
"is_enabled": true
}
]
}
The default tenant provides automatic setup:
- β Created automatically if no Firebase tenants exist
- β Saved to Firebase BEFORE orchestration starts
- β Future runs load from Firebase instead of recreating
- β Try multi-tenant mode without any setup
- β Migrate gradually from single-tenant to multi-tenant
- β Keep existing workflows unchanged
The flow:
- First run: No Firebase tenants β Create default_tenant from .env β Save to Firebase β Process documents
- Second run: Firebase has default_tenant β Load it β Process documents
- Add more tenants anytime in Firebase β They're all processed together
Multi-Tenant Setup (Firebase)
Create a Firestore collection called
tenants(or set viaFIREBASE_TENANTS_COLLECTIONenv var)Add tenant documents with this structure:
{
"name": "Acme Corp",
"active": true,
"created_at": "2024-01-01T00:00:00Z",
"updated_at": "2024-01-01T00:00:00Z",
"credentials": {
"openai_api_key": "sk-...",
"google_service_account_json": "{...full service account json...}"
}
}
- Create a rules subcollection inside the tenant document, and add rules with this structure:
{
"rule_id": "invoices_2024",
"rule_name": "Invoices to Finance",
"source_folder_id": "1ABC...",
"target_folder_id": "1XYZ...",
"target_sheet_id": "18qKL...",
"sheet_tab_name": "Invoices",
"parsing_prompt": "Extract invoice details following these rules...",
"is_enabled": true
}
Environment Variables
Required for both modes:
FIREBASE_SERVICE_ACCOUNT_JSON # Firebase credentials (JSON string or use GOOGLE_SERVICE_ACCOUNT_JSON)
OPENAI_API_KEY # (only for legacy mode, multi-tenant uses tenant credentials)
Optional:
FIREBASE_TENANTS_COLLECTION # Default: "tenants"
FIREBASE_DOCUMENTS_COLLECTION # Default: "processed_documents"
FIREBASE_TRACKER_COLLECTION # Default: "processed_files"
FIREBASE_TRACK_PROCESSED # Default: "true"
OPENAI_MODEL # Default: "gpt-4o"
GOOGLE_DRIVE_INVOICES_BASE_PATH # Default: "Facturas"
ORCHESTRATOR_API_KEY # For API authentication
Firebase Storage Structure
Document Tracking (Multi-Tenant)
Each tenant's processed files are tracked in nested collections:
tenants/{tenant_id}/processed_files/{file_hash}
ββ file_hash: "sha256 hash"
ββ source_file: "filename"
ββ processed_at: "timestamp"
tenants/{tenant_id}/processed_documents/{document_id}
ββ document_id: "drive file id"
ββ parsed_data: {...}
ββ status: "Parsed|Modified|Sent"
Creating Tenants Programmatically
from firebase_tenant_config import (
FirebaseTenantConfigManager,
TenantConfig,
CredentialsObject,
RuleObject,
)
manager = FirebaseTenantConfigManager()
# Create credentials
credentials = CredentialsObject(
openai_api_key="sk-...",
google_service_account_json="..." # full JSON as string
)
# Create a rule
rule = RuleObject(
rule_id="invoices_2024",
rule_name="Invoices to Finance",
source_folder_id="1ABC...",
target_folder_id="1XYZ...",
target_sheet_id="18qKL...",
sheet_tab_name="Invoices",
parsing_prompt="Extract...",
is_enabled=True,
)
# Create tenant
tenant = TenantConfig(
tenant_id="acme-corp",
name="Acme Corp",
active=True,
credentials=credentials,
rules=[rule],
)
# Save to Firebase
manager.save_tenant(tenant)
# List all active tenants
active_tenants = manager.list_active_tenants()
# Get enabled rules for a tenant
enabled_rules = manager.get_enabled_rules("acme-corp")
# Add a rule
manager.add_rule("acme-corp", new_rule)
# Remove a rule
manager.remove_rule("acme-corp", "invoices_2024")
Output Format
Multi-Tenant Orchestration Result
{
"ok": true,
"tenants_processed": 2,
"total_parsed": 15,
"total_modified": 3,
"total_sent": 18,
"total_moved": 18,
"total_corrupted": 2,
"total_errors": 1,
"tenants_results": [
{
"tenant_id": "tenant_123",
"tenant_name": "Acme Corp",
"ok": true,
"rules_processed": 2,
"parsed": 10,
"modified": 2,
"sent": 12,
"moved": 12,
"corrupted": 1,
"errors": 0,
"rules_results": [
{
"ok": true,
"tenant_id": "tenant_123",
"rule_id": "invoices_2024",
"rule_name": "Invoices to Finance",
"scanned": 20,
"to_process": 15,
"skipped": 5,
"parsed": 10,
"modified": 2,
"sent": 12,
"moved": 12,
"corrupted": 1,
"errors": 0,
"processed_items": [...],
"skipped_items": [...]
}
]
}
]
}
Migration from Legacy Mode
No migration needed! The system maintains full backward compatibility:
- Existing env var configuration continues to work with
--legacyflag - New multi-tenant mode coexists without affecting legacy deployments
- To enable multi-tenant: Set up tenants in Firebase and run without
--legacy
Workflow Examples
Simple: Use Existing .env (No Firebase Setup)
The easiest way to get started with multi-tenant mode:
# 1. Ensure your .env has these vars (you probably already do):
# OPENAI_API_KEY
# GOOGLE_SERVICE_ACCOUNT_JSON
# GOOGLE_DRIVE_FOLDER_IDS
# GOOGLE_DRIVE_INVOICES_ROOT_FOLDER_ID
# GOOGLE_SHEETS_SPREADSHEET_ID
# 2. Just run multi-tenant orchestration:
python document_orchester.py --send
# That's it! The system will:
# - Check for tenants in Firebase
# - Find none, create default_tenant from .env
# - Process documents exactly as before
# - Results go to Google Sheets and Drive
No code changes, no Firebase setup, just works!
Advanced: Multiple Tenants in Firebase
For multiple tenants with different configurations:
Step 1: Set Up Tenant in Firebase
# Run this once to initialize a tenant
from firebase_tenant_config import FirebaseTenantConfigManager, TenantConfig, CredentialsObject, RuleObject
manager = FirebaseTenantConfigManager()
credentials = CredentialsObject(
openai_api_key="sk-xxx",
google_service_account_json="json_string_here"
)
rule = RuleObject(
rule_id="rule_1",
rule_name="Invoice Processing",
source_folder_id="drive_folder_id",
target_folder_id="drive_dest_folder_id",
target_sheet_id="sheets_id",
sheet_tab_name="Invoices",
is_enabled=True,
)
tenant = TenantConfig(
tenant_id="my_tenant",
name="My Company",
active=True,
credentials=credentials,
)
manager.save_tenant(tenant)
manager.add_rule("my_tenant", rule)
Step 2: Run Orchestration
# This will automatically:
# 1. Load all active tenants from Firebase (not just one!)
# 2. For each tenant, process all enabled rules
# 3. Track results in tenant-specific collections
python document_orchester.py --model gpt-4o --send
Step 3: Review Results
Results are stored in:
- Google Drive: Documents moved to
{target_folder_id}/{YYYY}{MM}/ - Google Sheets: Parsed data appended to
{target_sheet_id}sheet - Firebase: Document metadata in
tenants/{tenant_id}/processed_documents/
Troubleshooting
"No active tenants found"
- Check that
active: truein Firebase tenant document - Verify
FIREBASE_TENANTS_COLLECTIONenv var is correct - Confirm Firebase credentials are valid
"No enabled rules"
- Check that
is_enabled: truefor at least one rule - Verify rule has valid
source_folder_id
Credentials error
- Ensure
credentials.openai_api_keyandcredentials.google_service_account_jsonare set - Verify JSON strings are properly formatted
Mixing modes
- Don't mix multi-tenant and legacy configurations
- Use either all Firebase-based tenants OR all env vars, not both
- When switching modes, ensure all required env vars/Firebase docs exist
API Endpoints
Gradio Interface
Two REST-compatible endpoints are exposed:
Legacy Single-Tenant:
POST /api/orchestrate_drive_documents
{
"model": "gpt-4o",
"include_subfolders": true,
"send_to_sheet": true,
"api_key": "your_orchestrator_api_key"
}
Multi-Tenant:
POST /api/orchestrate_all_tenants
{
"model": "gpt-4o",
"include_subfolders": true,
"send_to_sheet": true,
"api_key": "your_orchestrator_api_key"
}
Performance Considerations
- Multi-tenant processing is sequential (one tenant at a time, then one rule at a time)
- Each tenant/rule maintains separate Firebase document tracking
- Large scale deployments may benefit from rule parallelization (future enhancement)
Managing Rules
Adding a Rule to an Existing Tenant
You can programmatically add rules to any tenant:
from firebase_tenant_config import (
FirebaseTenantConfigManager,
RuleObject,
)
manager = FirebaseTenantConfigManager()
# Create a new rule
new_rule = RuleObject(
rule_id="new_rule_123",
rule_name="Additional Processing Rule",
source_folder_id="1NEW_folder_id",
target_folder_id="1NEW_destination_id",
target_sheet_id="1NEW_sheet_id",
sheet_tab_name="Sheet Tab Name",
parsing_prompt="Custom parsing instructions...",
is_enabled=True,
)
# Add to tenant
manager.add_rule("my_tenant", new_rule)
Cloning a Rule
Use the provided script to clone the default rule:
python add_rule_from_default.py new_rule_id "New Rule Name"
This creates a new rule in the default_tenant with the same configuration (folder IDs, sheet, etc.) as the default rule. Perfect for creating multiple processing pipelines with identical settings.
Removing a Rule
from firebase_tenant_config import FirebaseTenantConfigManager
manager = FirebaseTenantConfigManager()
manager.remove_rule("my_tenant", "rule_id_to_remove")
Future Enhancements
Potential improvements:
- Parallel rule processing within a tenant
- Rule scheduling (cron-like execution)
- Audit logging for rule changes
- Rule versioning and rollback
- Web UI for rule management