|
Download api/TESTING.md from Text-to-Document-Generation/Docgenie-API: direct link, hf CLI and curl.
- Browser
- Download file 25 kB
-
https://huggingface.co/spaces/Text-to-Document-Generation/Docgenie-API/resolve/main/api/TESTING.md
- Command line
-
hf download hf://spaces/Text-to-Document-Generation/Docgenie-API/api/TESTING.md
-
curl -L -o TESTING.md https://huggingface.co/spaces/Text-to-Document-Generation/Docgenie-API/resolve/main/api/TESTING.md
25 kB
| # Testing Guide: DocGenie API | |
| Complete guide for testing the document generation API endpoints with Google Drive integration. | |
| ## Table of Contents | |
| 1. [Prerequisites](#prerequisites) | |
| 2. [Quick Start](#quick-start) | |
| 3. [Getting Google Drive Token](#getting-google-drive-token) | |
| 4. [Testing Async API](#testing-async-api) | |
| 5. [Testing Sync PDF API](#testing-sync-pdf-api) | |
| 6. [Manual Testing with cURL](#manual-testing-with-curl) | |
| 7. [Frontend Integration Example](#frontend-integration-example) | |
| 8. [Troubleshooting](#troubleshooting) | |
| --- | |
| ## Prerequisites | |
| ### 1. Start Required Services | |
| ```bash | |
| # Terminal 1: Start Redis | |
| ## Option A: Local Redis (Recommended for Development) | |
| # Install Redis (Ubuntu/Debian) | |
| sudo apt-get update && sudo apt-get install redis-server -y | |
| sudo systemctl start redis-server | |
| sudo systemctl enable redis-server | |
| # Verify Redis is running | |
| redis-cli ping # Should return "PONG" | |
| ## Option B: Docker (if Docker is installed) | |
| # docker run -d -p 6379:6379 --name redis redis:7-alpine | |
| # Terminal 2: Start FastAPI Server | |
| cd docgenie/api | |
| python main.py | |
| # Terminal 3: Start RQ Worker | |
| cd docgenie/api | |
| ./start_worker.sh | |
| ``` | |
| ### 2. Configure Environment | |
| Make sure your `api/.env` file has: | |
| ```bash | |
| # Required | |
| ANTHROPIC_API_KEY=your_claude_api_key | |
| SUPABASE_URL=https://your-project.supabase.co | |
| SUPABASE_KEY=your_supabase_key | |
| REDIS_URL=redis://localhost:6379/0 | |
| # Optional (for token refresh) | |
| GOOGLE_CLIENT_ID=your_client_id.apps.googleusercontent.com | |
| GOOGLE_CLIENT_SECRET=your_client_secret | |
| ``` | |
| ### 3. Create Supabase Tables | |
| Run the SQL from [DEPLOYMENT.md](DEPLOYMENT.md#32-create-database-schema) in your Supabase SQL Editor. | |
| --- | |
| ## Quick Start | |
| ### Option 1: Using Test Script (Easiest) | |
| ```bash | |
| # Get Google Drive token first (one-time setup) | |
| python api/test_get_google_token.py \ | |
| --client-id YOUR_CLIENT_ID \ | |
| --client-secret YOUR_CLIENT_SECRET | |
| # Copy the access token, then run test | |
| python api/test_async_api.py --google-token YOUR_ACCESS_TOKEN | |
| ``` | |
| ### Option 2: Using OAuth Playground (Quick Test) | |
| 1. Go to [OAuth Playground](https://developers.google.com/oauthplayground/) | |
| 2. Configure with your credentials | |
| 3. Get access token | |
| 4. Run test script with token | |
| --- | |
| ## Getting Google Drive Token | |
| ### Method 1: Using Helper Script (Recommended) | |
| Our helper script automates the OAuth flow: | |
| ```bash | |
| cd docgenie/api | |
| python test_get_google_token.py \ | |
| --client-id YOUR_GOOGLE_CLIENT_ID \ | |
| --client-secret YOUR_GOOGLE_CLIENT_SECRET | |
| ``` | |
| **What it does:** | |
| 1. Opens browser for Google authorization | |
| 2. Starts local server on port 8080 for callback | |
| 3. Exchanges authorization code for tokens | |
| 4. Displays access token and refresh token | |
| **Output:** | |
| ``` | |
| Access Token: ya29.a0AfH6SMBx... | |
| Refresh Token: 1//0gw... | |
| ``` | |
| ### Method 2: OAuth Playground (No Code) | |
| 1. **Go to**: https://developers.google.com/oauthplayground/ | |
| 2. **Configure Credentials**: | |
| - Click gear icon (β) in top right | |
| - Check "Use your own OAuth credentials" | |
| - Enter your Client ID and Client Secret | |
| 3. **Authorize API**: | |
| - In left panel, scroll to "Drive API v3" | |
| - Select: `https://www.googleapis.com/auth/drive.file` | |
| - Click "Authorize APIs" | |
| - Sign in with your Google account | |
| 4. **Get Token**: | |
| - Click "Exchange authorization code for tokens" | |
| - Copy the "Access token" value | |
| ### Method 3: Manual cURL (For Advanced Users) | |
| **Step 1: Get Authorization Code** | |
| Open this URL in browser (replace YOUR_CLIENT_ID): | |
| ``` | |
| https://accounts.google.com/o/oauth2/v2/auth?client_id=YOUR_CLIENT_ID&redirect_uri=http://localhost:8080&response_type=code&scope=https://www.googleapis.com/auth/drive.file&access_type=offline&prompt=consent | |
| ``` | |
| **Step 2: Exchange Code for Token** | |
| After authorization, you'll be redirected to: | |
| ``` | |
| http://localhost:8080/?code=AUTHORIZATION_CODE | |
| ``` | |
| Exchange the code: | |
| ```bash | |
| curl -X POST https://oauth2.googleapis.com/token \ | |
| -d "code=AUTHORIZATION_CODE" \ | |
| -d "client_id=YOUR_CLIENT_ID" \ | |
| -d "client_secret=YOUR_CLIENT_SECRET" \ | |
| -d "redirect_uri=http://localhost:8080" \ | |
| -d "grant_type=authorization_code" | |
| ``` | |
| Response: | |
| ```json | |
| { | |
| "access_token": "ya29.a0AfH6SMBx...", | |
| "refresh_token": "1//0gw...", | |
| "expires_in": 3600, | |
| "token_type": "Bearer" | |
| } | |
| ``` | |
| --- | |
| ## Testing Async API | |
| The async API (`/generate/async`) is optimized for batch processing with 50% cost savings. Jobs are queued and processed in the background, with status polling. | |
| ### Full Automated Test | |
| ```bash | |
| cd docgenie/api | |
| # Set token as environment variable | |
| export GOOGLE_DRIVE_TOKEN="ya29.a0AfH6SMBx..." | |
| # Run test (generates 2 documents by default) | |
| python test_async_api.py | |
| # Or pass token directly | |
| python test_async_api.py --google-token "ya29.a0AfH6SMBx..." | |
| ``` | |
| **Test Flow:** | |
| 1. β Health check | |
| 2. β Submit async job | |
| 3. β Poll status (every 30 seconds) | |
| 4. β List user jobs | |
| 5. β Display Google Drive link | |
| **Expected Output:** | |
| ``` | |
| ================================================================================ | |
| ASYNC API TEST SUITE | |
| ================================================================================ | |
| Base URL: http://localhost:8000 | |
| User ID: 1 | |
| Documents to Generate: 2 | |
| ================================================================================ | |
| ============================================================ | |
| 1. Testing API Health | |
| ============================================================ | |
| β API is healthy: {'status': 'healthy', 'version': '1.0.0'} | |
| ============================================================ | |
| 2. Submitting Async Job | |
| ============================================================ | |
| Payload: | |
| User ID: 1 | |
| Seed Images: 1 | |
| Num Solutions: 2 | |
| Google Token: ya29.a0AfH6SMBx... | |
| β Job submitted successfully! | |
| Request ID: 550e8400-e29b-41d4-a716-446655440000 | |
| Status: queued | |
| Estimated Time: 10 minutes | |
| Poll URL: /jobs/550e8400-e29b-41d4-a716-446655440000/status | |
| ============================================================ | |
| 3. Polling Job Status | |
| ============================================================ | |
| Polling every 30 seconds (max 60 attempts) | |
| Status flow: queued β processing β generating β completed/failed | |
| [12:00:00] Poll 1/60: QUEUED | |
| [12:00:30] Poll 2/60: PROCESSING - Creating batch request... | |
| [12:01:00] Poll 3/60: GENERATING - Batch submitted to Claude... | |
| [12:08:30] Poll 17/60: GENERATING - Polling batch status... | |
| [12:15:00] Poll 30/60: COMPLETED | |
| ============================================================ | |
| β JOB COMPLETED! | |
| ============================================================ | |
| Download URL: https://drive.google.com/file/d/abc123xyz/view?usp=sharing | |
| File Size: 15.4 MB | |
| Document Count: 2 | |
| Created: 2026-02-28T12:00:00Z | |
| Completed: 2026-02-28T12:15:00Z | |
| ================================================================================ | |
| TEST SUMMARY | |
| ================================================================================ | |
| β ALL TESTS PASSED! | |
| Your documents are available at: | |
| https://drive.google.com/file/d/abc123xyz/view?usp=sharing | |
| Next steps: | |
| 1. Open the Google Drive link in your browser | |
| 2. Download the ZIP file | |
| 3. Extract and verify generated documents | |
| ``` | |
| ### Test Options | |
| ```bash | |
| # Custom number of documents | |
| python test_async_api.py --google-token TOKEN --num-solutions 5 | |
| # Custom API URL (if deployed) | |
| python test_async_api.py --google-token TOKEN --base-url https://api.yourdomain.com | |
| # Different user ID | |
| python test_async_api.py --google-token TOKEN --user-id 42 | |
| # With refresh token | |
| python test_async_api.py \ | |
| --google-token ACCESS_TOKEN \ | |
| --google-refresh-token REFRESH_TOKEN | |
| # Show help for getting token | |
| python test_async_api.py --help-token | |
| ``` | |
| ---Testing Sync PDF API | |
| The sync PDF API (`/generate/pdf`) returns results immediately (20-60s) and supports three modes of operation. Perfect for smaller batch sizes and real-time workflows. | |
| ### Three Operating Modes | |
| **Mode 1: Quick Demo (No Tracking)** | |
| - Returns ZIP immediately | |
| - No Supabase records created | |
| - Perfect for quick testing and demos | |
| - No user_id required | |
| **Mode 2: Demo with Tracking** | |
| - Returns ZIP immediately | |
| - Creates Supabase record for tracking | |
| - Can poll status during generation | |
| - Requires user_id | |
| **Mode 3: Full Production** | |
| - Returns ZIP immediately | |
| - Creates Supabase record | |
| - Uploads to Google Drive in background | |
| - Requires user_id + google_drive_token | |
| - Best for production use | |
| ### Full Automated Test | |
| ```bash | |
| cd docgenie/api | |
| # Mode 1: Quick demo (no tracking) | |
| python test_sync_pdf_api.py | |
| # Mode 2: Demo with tracking | |
| python test_sync_pdf_api.py --user-id 123 | |
| # Mode 3: Full production (tracking + GDrive) | |
| python test_sync_pdf_api.py \ | |
| --user-id 123 \ | |
| --google-token "ya29.a0AfH6SMBx..." \ | |
| --google-refresh-token "1//0gw..." | |
| ``` | |
| **Test Flow for All Modes:** | |
| 1. β Health check | |
| 2. β Test Mode 1: Quick demo (always runs) | |
| 3. β Test Mode 2: With tracking (if user_id provided) | |
| 4. β Test Mode 3: Full production (if user_id + token provided) | |
| 5. β Validate ZIP contents | |
| 6. β Test status polling (Modes 2 & 3) | |
| 7. β Verify GDrive upload (Mode 3) | |
| **Expected Output:** | |
| ``` | |
| ================================================================================ | |
| DocGenie /generate/pdf Endpoint Test Suite | |
| ================================================================================ | |
| ================================================================================ | |
| 1. Testing API Health | |
| ================================================================================ | |
| β API is healthy: {'status': 'healthy', 'version': '1.0.0'} | |
| ================================================================================ | |
| 2. Testing Mode 1: Quick Demo (No Tracking) | |
| ================================================================================ | |
| This mode returns ZIP immediately without creating Supabase records. | |
| Use for quick testing and demos. | |
| Payload: | |
| Seed Images: 1 | |
| Num Solutions: 1 | |
| User ID: None (no tracking) | |
| Google Token: None | |
| β³ Calling /generate/pdf (expect 20-60 seconds)... | |
| β Response received in 42.3 seconds | |
| Response Headers: | |
| Content-Type: application/zip | |
| Content-Disposition: attachment; filename=docgenie_documents.zip | |
| X-Request-ID: NOT SET (expected in mode 1) | |
| X-Status-URL: NOT SET (expected in mode 1) | |
| β ZIP file size: 145.2 KB | |
| β ZIP contains 18 files: | |
| - README.md | |
| - metadata.json | |
| - analysis/document_1.json | |
| - annotations/gt/document_1.json | |
| - bbox/bbox_pdf/word/document_1.json | |
| - html/document_1.css | |
| - html/document_1.html | |
| - img/document_1.png | |
| - pdf/pdf_final/document_1.pdf | |
| - pdf/pdf_initial/document_1.pdf | |
| β Contains metadata.json | |
| β Contains README.md | |
| β Mode 1 (Quick Demo) Test PASSED | |
| β‘ Fast response: 42.3s | |
| π¦ Valid ZIP file | |
| β No tracking overhead | |
| ================================================================================ | |
| 3. Testing Mode 2: Demo with Progress Tracking | |
| ================================================================================ | |
| This mode returns ZIP immediately AND creates Supabase record. | |
| Client can poll /jobs/{request_id}/status during generation. | |
| Payload: | |
| User ID: 123 (tracking enabled) | |
| Seed Images: 1 | |
| Num Solutions: 2 | |
| Google Token: None | |
| β³ Calling /generate/pdf (expect 20-60 seconds)... | |
| β Response received in 58.7 seconds | |
| Response Headers: | |
| Content-Type: application/zip | |
| β X-Request-ID: 550e8400-e29b-41d4-a716-446655440000 | |
| β X-Status-URL: /jobs/550e8400-e29b-41d4-a716-446655440000/status | |
| β ZIP file size: 287.4 KB | |
| β ZIP contains 32 files | |
| β Found 4 PDF files | |
| β³ Testing status polling endpoint... | |
| β Status endpoint working: | |
| Request ID: 550e8400-e29b-41d4-a716-446655440000 | |
| Status: completed | |
| Created: 2026-03-01T10:15:00Z | |
| Updated: 2026-03-01T10:15:58Z | |
| β Job marked as completed | |
| β Mode 2 (Tracking) Test PASSED | |
| β‘ Fast response: 58.7s | |
| π¦ Valid ZIP file | |
| π Progress tracking enabled | |
| β Can poll status during generation | |
| ================================================================================ | |
| 4. Testing Mode 3: Full Production (Tracking + GDrive Upload) | |
| ================================================================================ | |
| This mode returns ZIP immediately AND uploads to Google Drive in background. | |
| Best for production use with full tracking and backup. | |
| Payload: | |
| User ID: 123 | |
| Google Token: ya29.a0AfH6SMBx... | |
| Google Refresh: Yes | |
| Seed Images: 1 | |
| Num Solutions: 1 | |
| β³ Calling /generate/pdf (expect 20-60 seconds)... | |
| β Response received in 45.1 seconds | |
| Response Headers: | |
| β X-Request-ID: 660f9511-f3ac-52e5-b827-557766551111 | |
| β X-Status-URL: /jobs/660f9511-f3ac-52e5-b827-557766551111/status | |
| β ZIP file size: 151.8 KB | |
| β ZIP contains 18 files | |
| β³ ZIP returned immediately, GDrive upload happening in background... | |
| (This doesn't block the response) | |
| β³ Waiting 10 seconds for background GDrive upload... | |
| β Status after background upload: | |
| Status: completed | |
| β GDrive URL: https://drive.google.com/file/d/abc123xyz/view?usp=... | |
| β Background upload completed! | |
| β Mode 3 (Full Production) Test PASSED | |
| β‘ Fast response: 45.1s (GDrive doesn't block) | |
| π¦ Valid ZIP file delivered immediately | |
| π Progress tracking enabled | |
| βοΈ Google Drive backup scheduled | |
| β Production-ready configuration | |
| ================================================================================ | |
| TEST SUMMARY | |
| ================================================================================ | |
| β health: PASSED | |
| β mode_1: PASSED | |
| β mode_2: PASSED | |
| β mode_3: PASSED | |
| 4/4 tests passed | |
| π All tests passed! | |
| ================================================================================ | |
| ``` | |
| ### Test Options | |
| ```bash | |
| # Mode 1 only (default) | |
| python test_sync_pdf_api.py | |
| # Mode 2 with custom user ID | |
| python test_sync_pdf_api.py --user-id 456 | |
| # Mode 3 with custom API URL | |
| python test_sync_pdf_api.py \ | |
| --base-url https://api.yourdomain.com \ | |
| --user-id 123 \ | |
| --google-token TOKEN \ | |
| --google-refresh-token REFRESH_TOKEN | |
| ``` | |
| ### Comparing Sync vs Async | |
| | Feature | Sync (`/generate/pdf`) | Async (`/generate/async`) | | |
| |---------|------------------------|---------------------------| | |
| | **Response Time** | 20-60 seconds | 5-30 minutes | | |
| | **Best For** | 1-3 documents | 5-50+ documents | | |
| | **Cost** | Standard API pricing | 50% cheaper (Batch API) | | |
| | **Result Delivery** | Direct ZIP download | Google Drive upload | | |
| | **Progress Tracking** | Optional (Modes 2 & 3) | Always enabled | | |
| | **Use Case** | Real-time workflows, demos | Bulk generation, scheduled jobs | | |
| **When to use Sync:** | |
| - Generating 1-3 documents | |
| - Need immediate results | |
| - Real-time user interactions | |
| - Quick testing and demos | |
| **When to use Async:** | |
| - Generating 5+ documents | |
| - Cost optimization (50% savings) | |
| - Background/scheduled processing | |
| - Large batch jobs | |
| --- | |
| ## Manual Testing with cURL | |
| ### Async API (`/generate/async`) | |
| #### 1. Submit Async Job | |
| ```bash | |
| curl -X POST http://localhost:8000/generate/async \ | |
| -H "Content-Type: application/json" \ | |
| -d '{ | |
| "user_id": 1, | |
| "google_drive_token": "ya29.a0AfH6SMBx...", | |
| "seed_images": ["https://ocr.space/Content/Images/receipt-ocr-original.webp"], | |
| "prompt_params": { | |
| "language": "English", | |
| "doc_type": "receipts", | |
| "num_solutions": 2, | |
| "enable_handwriting": false, | |
| "enable_visual_elements": false, | |
| "output_detail": "minimal" | |
| } | |
| }' | |
| ``` | |
| **Response:** | |
| ```json | |
| { | |
| "request_id": "550e8400-e29b-41d4-a716-446655440000", | |
| "status": "queued", | |
| "estimated_time_minutes": 10, | |
| "poll_url": "/jobs/550e8400-e29b-41d4-a716-446655440000/status", | |
| "created_at": "2026-02-28T12:00:00Z" | |
| } | |
| ``` | |
| #### 2. Check Job Status | |
| ```bash | |
| curl http://localhost:8000/jobs/550e8400-e29b-41d4-a716-446655440000/status | |
| ``` | |
| **Response (Processing):** | |
| ```json | |
| { | |
| "request_id": "550e8400-e29b-41d4-a716-446655440000", | |
| "status": "processing", | |
| "created_at": "2026-02-28T12:00:00Z", | |
| "updated_at": "2026-02-28T12:02:00Z", | |
| "progress": "Creating batch request..." | |
| } | |
| ``` | |
| **Response (Completed):** | |
| ```json | |
| { | |
| "request_id": "550e8400-e29b-41d4-a716-446655440000", | |
| "status": "completed", | |
| "created_at": "2026-02-28T12:00:00Z", | |
| "updated_at": "2026-02-28T12:15:00Z", | |
| "download_url": "https://drive.google.com/file/d/abc123xyz/view?usp=sharing", | |
| "file_size_mb": 15.4, | |
| "document_count": 2 | |
| } | |
| ``` | |
| #### 3. List User Jobs | |
| ```bash | |
| curl "http://localhost:8000/jobs/user/1?limit=10&offset=0" | |
| ``` | |
| **Response:** | |
| ```json | |
| { | |
| "user_id": 1, | |
| "jobs": [ | |
| { | |
| "request_id": "550e8400-e29b-41d4-a716-446655440000", | |
| "status": "completed", | |
| "created_at": "2026-02-28T12:00:00Z", | |
| "download_url": "https://drive.google.com/file/d/abc123xyz/view" | |
| } | |
| ], | |
| "count": 1, | |
| "limit": 10, | |
| "offset": 0 | |
| } | |
| ``` | |
| ### Sync PDF API (`/generate/pdf`) | |
| #### Mode 1: Quick Demo (No Tracking) | |
| ```bash | |
| curl -X POST http://localhost:8000/generate/pdf \ | |
| -H "Content-Type: application/json" \ | |
| -d '{ | |
| "seed_images": ["https://ocr.space/Content/Images/receipt-ocr-original.webp"], | |
| "prompt_params": { | |
| "language": "English", | |
| "doc_type": "receipts", | |
| "num_solutions": 1, | |
| "enable_handwriting": false, | |
| "enable_visual_elements": false, | |
| "output_detail": "minimal" | |
| } | |
| }' \ | |
| --output documents.zip | |
| ``` | |
| **Response:** | |
| - Returns ZIP file directly (binary) | |
| - No tracking headers | |
| - File saved as `documents.zip` | |
| #### Mode 2: Demo with Tracking | |
| ```bash | |
| curl -X POST http://localhost:8000/generate/pdf \ | |
| -H "Content-Type: application/json" \ | |
| -d '{ | |
| "user_id": 123, | |
| "seed_images": ["https://ocr.space/Content/Images/receipt-ocr-original.webp"], | |
| "prompt_params": { | |
| "language": "English", | |
| "doc_type": "business documents", | |
| "num_solutions": 2, | |
| "enable_handwriting": false, | |
| "output_detail": "minimal" | |
| } | |
| }' \ | |
| --output documents.zip \ | |
| -D headers.txt | |
| ``` | |
| **Response:** | |
| - Returns ZIP file directly (binary) | |
| - Headers saved to `headers.txt` contain: | |
| - `X-Request-ID: 550e8400-e29b-41d4-a716-446655440000` | |
| - `X-Status-URL: /jobs/550e8400-e29b-41d4-a716-446655440000/status` | |
| **Check Status:** | |
| ```bash | |
| # Extract request_id from headers.txt, then: | |
| curl http://localhost:8000/jobs/550e8400-e29b-41d4-a716-446655440000/status | |
| ``` | |
| #### Mode 3: Full Production (Tracking + GDrive) | |
| ```bash | |
| curl -X POST http://localhost:8000/generate/pdf \ | |
| -H "Content-Type: application/json" \ | |
| -d '{ | |
| "user_id": 123, | |
| "google_drive_token": "ya29.a0AfH6SMBx...", | |
| "google_drive_refresh_token": "1//0gw...", | |
| "seed_images": ["https://ocr.space/Content/Images/receipt-ocr-original.webp"], | |
| "prompt_params": { | |
| "language": "English", | |
| "doc_type": "invoices", | |
| "num_solutions": 1, | |
| "enable_handwriting": false, | |
| "output_detail": "dataset" | |
| } | |
| }' \ | |
| --output documents.zip \ | |
| -D headers.txt | |
| ``` | |
| **Response:** | |
| - Returns ZIP file immediately (binary) | |
| - Google Drive upload happens in background | |
| - Wait 10-30 seconds, then check status for GDrive URL: | |
| ```bash | |
| curl http://localhost:8000/jobs/550e8400-e29b-41d4-a716-446655440000/status | |
| ``` | |
| **Response (after background upload):** | |
| ```json | |
| { | |
| "request_id": "550e8400-e29b-41d4-a716-446655440000", | |
| "status": "completed", | |
| "created_at": "2026-03-01T10:00:00Z", | |
| "updated_at": "2026-03-01T10:00:45Z", | |
| "results": { | |
| "download_url": "https://drive.google.com/file/d/abc123xyz/view?usp=sharing", | |
| "file_size_mb": 0.15, | |
| "document_count": 1, | |
| "zip_filename": "docgenie_550e8400-e29b-41d4-a716-446655440000.zip" | |
| } | |
| } | |
| ``` | |
| --- | |
| ## Frontend Integration Example | |
| ### React + TypeScript | |
| ```typescript | |
| import { useState, useEffect } from 'react'; | |
| interface JobStatus { | |
| request_id: string; | |
| status: 'queued' | 'processing' | 'generating' | 'completed' | 'failed'; | |
| download_url?: string; | |
| error_message?: string; | |
| } | |
| function DocumentGenerator() { | |
| const [jobId, setJobId] = useState<string | null>(null); | |
| const [status, setStatus] = useState<JobStatus | null>(null); | |
| const [googleToken, setGoogleToken] = useState<string>(''); | |
| // Step 1: Google OAuth (implement separately) | |
| const handleGoogleAuth = async () => { | |
| // Redirect to Google OAuth | |
| const clientId = 'YOUR_CLIENT_ID'; | |
| const redirectUri = 'https://yourapp.com/auth/callback'; | |
| const scope = 'https://www.googleapis.com/auth/drive.file'; | |
| const authUrl = `https://accounts.google.com/o/oauth2/v2/auth?` + | |
| `client_id=${clientId}&` + | |
| `redirect_uri=${redirectUri}&` + | |
| `response_type=code&` + | |
| `scope=${scope}&` + | |
| `access_type=offline&` + | |
| `prompt=consent`; | |
| window.location.href = authUrl; | |
| }; | |
| // Step 2: Submit job | |
| const handleGenerateDocuments = async () => { | |
| const response = await fetch('http://localhost:8000/generate/async', { | |
| method: 'POST', | |
| headers: { 'Content-Type': 'application/json' }, | |
| body: JSON.stringify({ | |
| user_id: 1, | |
| google_drive_token: googleToken, | |
| seed_images: ['https://example.com/seed.jpg'], | |
| prompt_params: { | |
| language: 'English', | |
| doc_type: 'receipts', | |
| num_solutions: 3 | |
| } | |
| }) | |
| }); | |
| const job = await response.json(); | |
| setJobId(job.request_id); | |
| }; | |
| // Step 3: Poll status | |
| useEffect(() => { | |
| if (!jobId) return; | |
| const interval = setInterval(async () => { | |
| const response = await fetch( | |
| `http://localhost:8000/jobs/${jobId}/status` | |
| ); | |
| const data = await response.json(); | |
| setStatus(data); | |
| if (data.status === 'completed' || data.status === 'failed') { | |
| clearInterval(interval); | |
| } | |
| }, 30000); // Poll every 30 seconds | |
| return () => clearInterval(interval); | |
| }, [jobId]); | |
| return ( | |
| <div> | |
| {!googleToken ? ( | |
| <button onClick={handleGoogleAuth}> | |
| Connect Google Drive | |
| </button> | |
| ) : ( | |
| <button onClick={handleGenerateDocuments}> | |
| Generate Documents | |
| </button> | |
| )} | |
| {status && ( | |
| <div> | |
| <p>Status: {status.status}</p> | |
| {status.status === 'completed' && ( | |
| <a href={status.download_url} target="_blank"> | |
| Download Documents | |
| </a> | |
| )} | |
| {status.status === 'failed' && ( | |
| <p>Error: {status.error_message}</p> | |
| )} | |
| </div> | |
| )} | |
| </div> | |
| ); | |
| } | |
| ``` | |
| --- | |
| ## Troubleshooting | |
| ### Issue: "google_drive_token is required" | |
| **Cause**: No token provided in request | |
| **Solution**: | |
| ```bash | |
| # Make sure you're passing the token | |
| python test_async_api.py --google-token "ya29.a0AfH6SMBx..." | |
| ``` | |
| ### Issue: "Failed to refresh Google Drive token" | |
| **Cause**: Token expired and no refresh token provided | |
| **Solutions**: | |
| 1. Get a new token (tokens expire in ~1 hour) | |
| 2. Include refresh token in request | |
| 3. Frontend should refresh tokens automatically | |
| ### Issue: "Google Drive upload failed: insufficient permissions" | |
| **Cause**: Token doesn't have drive.file scope | |
| **Solution**: Re-authorize with correct scope: | |
| ``` | |
| https://www.googleapis.com/auth/drive.file | |
| ``` | |
| ### Issue: Worker not processing jobs | |
| **Check 1**: Is Redis running? | |
| ```bash | |
| redis-cli ping # Should return "PONG" | |
| ``` | |
| **Check 2**: Is worker running? | |
| ```bash | |
| # Check worker logs | |
| journalctl -u docgenie-worker@1 -f | |
| # Or check RQ info | |
| rq info --url redis://localhost:6379/0 | |
| ``` | |
| **Check 3**: Check failed queue | |
| ```bash | |
| rq info --queue failed --url redis://localhost:6379/0 | |
| ``` | |
| ### Issue: Job stuck in "generating" status | |
| **Cause**: Batch API taking longer than expected | |
| **Solution**: Wait up to 30 minutes for batched requests. Check Anthropic dashboard: | |
| https://console.anthropic.com/settings/batches | |
| ### Issue: Cannot access Google Drive link | |
| **Cause**: File not shared properly | |
| **Solution**: Check worker logs for sharing errors. File should have "anyone with link" permission. | |
| --- | |
| ## Performance Testing | |
| ### Test Batch API Cost Savings | |
| ```bash | |
| # Generate 10 documents | |
| time python test_async_api.py --google-token TOKEN --num-solutions 10 | |
| # Compare with direct API (for reference) | |
| curl -X POST http://localhost:8000/generate \ | |
| -H "Content-Type: application/json" \ | |
| -d '{"seed_images": ["..."], "prompt_params": {"num_solutions": 10}}' | |
| ``` | |
| **Expected Results:** | |
| - **Batched API**: 10-20 minutes, ~$2.50 per 1M tokens | |
| - **Direct API**: 3-5 minutes, ~$5.00 per 1M tokens | |
| - **Cost Savings**: 50% | |
| --- | |
| ## Next Steps | |
| 1. β Test locally with script | |
| 2. β Verify Google Drive upload | |
| 3. β Test with your frontend | |
| 4. β Deploy to production (see [DEPLOYMENT.md](DEPLOYMENT.md)) | |
| 5. β Set up monitoring (see [SCALING.md](SCALING.md)) | |
| --- | |
| ## Additional Resources | |
| - **API Documentation**: http://localhost:8000/docs | |
| - **Deployment Guide**: [DEPLOYMENT.md](DEPLOYMENT.md) | |
| - **Scaling Guide**: [SCALING.md](SCALING.md) | |
| - **Google OAuth Docs**: https://developers.google.com/identity/protocols/oauth2 | |
| - **Anthropic Batch API**: https://docs.anthropic.com/en/docs/batch-api | |