document-parser / MULTI_TENANT_GUIDE.md
segovia
multitenant supprt
028d116
|
Raw History Blame Contribute Delete
14.8 kB

A newer version of the Gradio SDK is available: 6.32.0

Upgrade

Multi-Tenant Document Parser - Configuration & Usage Guide

Overview

The document parser now supports two operational modes:

  1. Multi-Tenant Mode (NEW) - Load all active tenants from Firebase and process their enabled rules
  2. Legacy Mode - Single tenant configuration via environment variables (backward compatible)

Architecture

Data Model

All tenant and rule configurations are stored in Firestore under the tenants collection:

TENANTS_COLLECTION (Firestore)
β”œβ”€β”€ {tenant_id}
β”‚   β”œβ”€β”€ tenant_id: unique tenant ID
β”‚   β”œβ”€β”€ name: "Company/User Name"
β”‚   β”œβ”€β”€ active: boolean (Account Status)
β”‚   β”œβ”€β”€ created_at: timestamp
β”‚   β”œβ”€β”€ updated_at: timestamp
β”‚   β”œβ”€β”€ credentials (object, 1:1 relationship)
β”‚   β”‚   β”œβ”€β”€ openai_api_key: "OpenAI Key"
β”‚   β”‚   └── google_service_account_json: "Service account JSON"
β”‚   └── rules (subcollection, 1:N relationship)
β”‚       └── {rule_id}
β”‚           β”œβ”€β”€ rule_id: "unique rule ID"
β”‚           β”œβ”€β”€ rule_name: "Invoices to Finance"
β”‚           β”œβ”€β”€ source_folder_id: "Google Drive Source Folder ID"
β”‚           β”œβ”€β”€ target_folder_id: "Google Drive Destination Folder ID"
β”‚           β”œβ”€β”€ target_sheet_id: "Google Sheets Spreadsheet ID"
β”‚           β”œβ”€β”€ sheet_tab_name: "Tab inside Google Sheet"
β”‚           β”œβ”€β”€ parsing_prompt: "Custom LLM Instructions"
β”‚           └── is_enabled: boolean (Rule Toggle)

Processing Flow

Multi-Tenant Mode:
1. orchestrate_all_active_tenants()
   └─> For each active tenant:
       └─> For each enabled rule:
           └─> orchestrate_single_rule()
               β”œβ”€> Scan rule.source_folder_id
               β”œβ”€> Parse documents with rule.parsing_prompt
               β”œβ”€> Move to rule.target_folder_id
               └─> Send to rule.target_sheet_id

Usage

Quick Start (Using .env)

If you have existing .env configuration, multi-tenant mode will automatically create a default tenant:

# Just run it! No Firebase setup needed initially
python document_orchester.py --send

# The system will:
# 1. Check for active tenants in Firebase
# 2. If none found, create "default_tenant" from your .env vars
# 3. Process documents using the default configuration

This is the easiest path if you're migrating from single-tenant mode.

Mode 1: Multi-Tenant Mode (Default)

# Process all active tenants and their enabled rules
python document_orchester.py --model gpt-4o --send

# Optional flags:
# --no-subfolders    Only scan direct files (don't recurse)
# --send             Send results to Google Sheets
# --output FILE      Save JSON summary to file

Mode 2: Legacy Single-Tenant Mode

# Use env var configuration (backward compatible)
python document_orchester.py --legacy --model gpt-4o --send

Gradio Interface

Start the web app:

python app.py

Two orchestrator endpoints are now available:

  • Single Tenant (Legacy) - Uses env var configuration
  • Multi Tenant - Loads all active tenants from Firebase

Configuration

Automatic Default Tenant (No Firebase Setup Required)

When you run multi-tenant mode without any tenants in Firebase:

  1. The system checks for active tenants in Firestore
  2. If none found, it creates a default tenant from your .env file
  3. Maps env vars like this:
.env Variable Default Tenant
OPENAI_API_KEY credentials.openai_api_key
GOOGLE_SERVICE_ACCOUNT_JSON credentials.google_service_account_json
GOOGLE_DRIVE_FOLDER_IDS (first) rule.source_folder_id
GOOGLE_DRIVE_INVOICES_ROOT_FOLDER_ID rule.target_folder_id
GOOGLE_SHEETS_SPREADSHEET_ID rule.target_sheet_id

Example: Your existing .env

OPENAI_API_KEY=sk-abc123...
GOOGLE_SERVICE_ACCOUNT_JSON={"type":"service_account","project_id":"my-project",...}
GOOGLE_DRIVE_FOLDER_IDS=1ABC_folder_id_here
GOOGLE_DRIVE_INVOICES_ROOT_FOLDER_ID=1XYZ_destination_folder_id
GOOGLE_SHEETS_SPREADSHEET_ID=18qKL...

Creates this default tenant (in memory, not saved to Firebase):

{
  "tenant_id": "default_tenant",
  "name": "Default Tenant (from .env)",
  "active": true,
  "credentials": {
    "openai_api_key": "sk-abc123...",
    "google_service_account_json": "{...}"
  },
  "rules": [
    {
      "rule_id": "default_rule",
      "rule_name": "Default Processing Rule",
      "source_folder_id": "1ABC_folder_id_here",
      "target_folder_id": "1XYZ_destination_folder_id",
      "target_sheet_id": "18qKL...",
      "sheet_tab_name": "Parsed",
      "is_enabled": true
    }
  ]
}

The default tenant provides automatic setup:

  • βœ… Created automatically if no Firebase tenants exist
  • βœ… Saved to Firebase BEFORE orchestration starts
  • βœ… Future runs load from Firebase instead of recreating
  • βœ… Try multi-tenant mode without any setup
  • βœ… Migrate gradually from single-tenant to multi-tenant
  • βœ… Keep existing workflows unchanged

The flow:

  1. First run: No Firebase tenants β†’ Create default_tenant from .env β†’ Save to Firebase β†’ Process documents
  2. Second run: Firebase has default_tenant β†’ Load it β†’ Process documents
  3. Add more tenants anytime in Firebase β†’ They're all processed together

Multi-Tenant Setup (Firebase)

  1. Create a Firestore collection called tenants (or set via FIREBASE_TENANTS_COLLECTION env var)

  2. Add tenant documents with this structure:

{
  "name": "Acme Corp",
  "active": true,
  "created_at": "2024-01-01T00:00:00Z",
  "updated_at": "2024-01-01T00:00:00Z",
  "credentials": {
    "openai_api_key": "sk-...",
    "google_service_account_json": "{...full service account json...}"
  }
}
  1. Create a rules subcollection inside the tenant document, and add rules with this structure:
{
  "rule_id": "invoices_2024",
  "rule_name": "Invoices to Finance",
  "source_folder_id": "1ABC...",
  "target_folder_id": "1XYZ...",
  "target_sheet_id": "18qKL...",
  "sheet_tab_name": "Invoices",
  "parsing_prompt": "Extract invoice details following these rules...",
  "is_enabled": true
}

Environment Variables

Required for both modes:

FIREBASE_SERVICE_ACCOUNT_JSON    # Firebase credentials (JSON string or use GOOGLE_SERVICE_ACCOUNT_JSON)
OPENAI_API_KEY                   # (only for legacy mode, multi-tenant uses tenant credentials)

Optional:

FIREBASE_TENANTS_COLLECTION      # Default: "tenants"
FIREBASE_DOCUMENTS_COLLECTION    # Default: "processed_documents"
FIREBASE_TRACKER_COLLECTION      # Default: "processed_files"
FIREBASE_TRACK_PROCESSED         # Default: "true"
OPENAI_MODEL                     # Default: "gpt-4o"
GOOGLE_DRIVE_INVOICES_BASE_PATH  # Default: "Facturas"
ORCHESTRATOR_API_KEY             # For API authentication

Firebase Storage Structure

Document Tracking (Multi-Tenant)

Each tenant's processed files are tracked in nested collections:

tenants/{tenant_id}/processed_files/{file_hash}
└─ file_hash: "sha256 hash"
└─ source_file: "filename"
└─ processed_at: "timestamp"

tenants/{tenant_id}/processed_documents/{document_id}
└─ document_id: "drive file id"
└─ parsed_data: {...}
└─ status: "Parsed|Modified|Sent"

Creating Tenants Programmatically

from firebase_tenant_config import (
    FirebaseTenantConfigManager,
    TenantConfig,
    CredentialsObject,
    RuleObject,
)

manager = FirebaseTenantConfigManager()

# Create credentials
credentials = CredentialsObject(
    openai_api_key="sk-...",
    google_service_account_json="..." # full JSON as string
)

# Create a rule
rule = RuleObject(
    rule_id="invoices_2024",
    rule_name="Invoices to Finance",
    source_folder_id="1ABC...",
    target_folder_id="1XYZ...",
    target_sheet_id="18qKL...",
    sheet_tab_name="Invoices",
    parsing_prompt="Extract...",
    is_enabled=True,
)

# Create tenant
tenant = TenantConfig(
    tenant_id="acme-corp",
    name="Acme Corp",
    active=True,
    credentials=credentials,
    rules=[rule],
)

# Save to Firebase
manager.save_tenant(tenant)

# List all active tenants
active_tenants = manager.list_active_tenants()

# Get enabled rules for a tenant
enabled_rules = manager.get_enabled_rules("acme-corp")

# Add a rule
manager.add_rule("acme-corp", new_rule)

# Remove a rule
manager.remove_rule("acme-corp", "invoices_2024")

Output Format

Multi-Tenant Orchestration Result

{
  "ok": true,
  "tenants_processed": 2,
  "total_parsed": 15,
  "total_modified": 3,
  "total_sent": 18,
  "total_moved": 18,
  "total_corrupted": 2,
  "total_errors": 1,
  "tenants_results": [
    {
      "tenant_id": "tenant_123",
      "tenant_name": "Acme Corp",
      "ok": true,
      "rules_processed": 2,
      "parsed": 10,
      "modified": 2,
      "sent": 12,
      "moved": 12,
      "corrupted": 1,
      "errors": 0,
      "rules_results": [
        {
          "ok": true,
          "tenant_id": "tenant_123",
          "rule_id": "invoices_2024",
          "rule_name": "Invoices to Finance",
          "scanned": 20,
          "to_process": 15,
          "skipped": 5,
          "parsed": 10,
          "modified": 2,
          "sent": 12,
          "moved": 12,
          "corrupted": 1,
          "errors": 0,
          "processed_items": [...],
          "skipped_items": [...]
        }
      ]
    }
  ]
}

Migration from Legacy Mode

No migration needed! The system maintains full backward compatibility:

  1. Existing env var configuration continues to work with --legacy flag
  2. New multi-tenant mode coexists without affecting legacy deployments
  3. To enable multi-tenant: Set up tenants in Firebase and run without --legacy

Workflow Examples

Simple: Use Existing .env (No Firebase Setup)

The easiest way to get started with multi-tenant mode:

# 1. Ensure your .env has these vars (you probably already do):
#    OPENAI_API_KEY
#    GOOGLE_SERVICE_ACCOUNT_JSON
#    GOOGLE_DRIVE_FOLDER_IDS
#    GOOGLE_DRIVE_INVOICES_ROOT_FOLDER_ID
#    GOOGLE_SHEETS_SPREADSHEET_ID

# 2. Just run multi-tenant orchestration:
python document_orchester.py --send

# That's it! The system will:
# - Check for tenants in Firebase
# - Find none, create default_tenant from .env
# - Process documents exactly as before
# - Results go to Google Sheets and Drive

No code changes, no Firebase setup, just works!

Advanced: Multiple Tenants in Firebase

For multiple tenants with different configurations:

Step 1: Set Up Tenant in Firebase

# Run this once to initialize a tenant
from firebase_tenant_config import FirebaseTenantConfigManager, TenantConfig, CredentialsObject, RuleObject

manager = FirebaseTenantConfigManager()

credentials = CredentialsObject(
    openai_api_key="sk-xxx",
    google_service_account_json="json_string_here"
)

rule = RuleObject(
    rule_id="rule_1",
    rule_name="Invoice Processing",
    source_folder_id="drive_folder_id",
    target_folder_id="drive_dest_folder_id",
    target_sheet_id="sheets_id",
    sheet_tab_name="Invoices",
    is_enabled=True,
)

tenant = TenantConfig(
    tenant_id="my_tenant",
    name="My Company",
    active=True,
    credentials=credentials,
)

manager.save_tenant(tenant)
manager.add_rule("my_tenant", rule)

Step 2: Run Orchestration

# This will automatically:
# 1. Load all active tenants from Firebase (not just one!)
# 2. For each tenant, process all enabled rules
# 3. Track results in tenant-specific collections
python document_orchester.py --model gpt-4o --send

Step 3: Review Results

Results are stored in:

  • Google Drive: Documents moved to {target_folder_id}/{YYYY}{MM}/
  • Google Sheets: Parsed data appended to {target_sheet_id} sheet
  • Firebase: Document metadata in tenants/{tenant_id}/processed_documents/

Troubleshooting

"No active tenants found"

  • Check that active: true in Firebase tenant document
  • Verify FIREBASE_TENANTS_COLLECTION env var is correct
  • Confirm Firebase credentials are valid

"No enabled rules"

  • Check that is_enabled: true for at least one rule
  • Verify rule has valid source_folder_id

Credentials error

  • Ensure credentials.openai_api_key and credentials.google_service_account_json are set
  • Verify JSON strings are properly formatted

Mixing modes

  • Don't mix multi-tenant and legacy configurations
  • Use either all Firebase-based tenants OR all env vars, not both
  • When switching modes, ensure all required env vars/Firebase docs exist

API Endpoints

Gradio Interface

Two REST-compatible endpoints are exposed:

Legacy Single-Tenant:

POST /api/orchestrate_drive_documents
{
  "model": "gpt-4o",
  "include_subfolders": true,
  "send_to_sheet": true,
  "api_key": "your_orchestrator_api_key"
}

Multi-Tenant:

POST /api/orchestrate_all_tenants
{
  "model": "gpt-4o",
  "include_subfolders": true,
  "send_to_sheet": true,
  "api_key": "your_orchestrator_api_key"
}

Performance Considerations

  • Multi-tenant processing is sequential (one tenant at a time, then one rule at a time)
  • Each tenant/rule maintains separate Firebase document tracking
  • Large scale deployments may benefit from rule parallelization (future enhancement)

Managing Rules

Adding a Rule to an Existing Tenant

You can programmatically add rules to any tenant:

from firebase_tenant_config import (
    FirebaseTenantConfigManager,
    RuleObject,
)

manager = FirebaseTenantConfigManager()

# Create a new rule
new_rule = RuleObject(
    rule_id="new_rule_123",
    rule_name="Additional Processing Rule",
    source_folder_id="1NEW_folder_id",
    target_folder_id="1NEW_destination_id",
    target_sheet_id="1NEW_sheet_id",
    sheet_tab_name="Sheet Tab Name",
    parsing_prompt="Custom parsing instructions...",
    is_enabled=True,
)

# Add to tenant
manager.add_rule("my_tenant", new_rule)

Cloning a Rule

Use the provided script to clone the default rule:

python add_rule_from_default.py new_rule_id "New Rule Name"

This creates a new rule in the default_tenant with the same configuration (folder IDs, sheet, etc.) as the default rule. Perfect for creating multiple processing pipelines with identical settings.

Removing a Rule

from firebase_tenant_config import FirebaseTenantConfigManager

manager = FirebaseTenantConfigManager()
manager.remove_rule("my_tenant", "rule_id_to_remove")

Future Enhancements

Potential improvements:

  • Parallel rule processing within a tenant
  • Rule scheduling (cron-like execution)
  • Audit logging for rule changes
  • Rule versioning and rollback
  • Web UI for rule management